and suppose that $u(\theta),v(\theta)\in\mathcal{H}$ are differentiable with
respect to $\theta$, in the same sense as in Lemma 26. Let
$u^{\prime}_{i}(\theta)=(\partial/\partial\theta_{i})u(\theta)$, and
$v^{\prime}_{i}(\theta)=(\partial/\partial\theta_{i})v(\theta)$. Furthermore,
let $B_{u}$ and $B_{v}$ be linear operators on $\mathcal{H}$, with adjoints
$B_{u}^{*}$ and $B_{v}^{*}$ respectively, such that
$\|B_{v}^{*}B_{u}u(\theta)\|<\infty$ and $\|B_{u}^{*}B_{v}v(\theta)\|<\infty$
for each $\theta\in\Theta$, and $B_{u}^{*}B_{v}v(\theta)$ is continuous with
respect to $\theta$. Then we have

$\frac{\partial}{\partial\theta_{i}}\langle
B_{u}u(\theta),B_{v}v(\theta)\rangle=\langle
B_{u}u^{\prime}_{i}(\theta),B_{v}v(\theta)\rangle+\langle
B_{u}u(\theta),B_{v}v^{\prime}_{i}(\theta)\rangle\,.$

###### Proof of Lemma 27.

Let $\theta_{t}=\theta+te_{i}$, where $e_{i}$ is the $i$’th standard basis
vector of $\mathbb{R}^{b}$. We need to compute $\lim_{t\to 0}\Delta(t)$, where

$\Delta(t)=\frac{1}{t}(\langle
B_{u}u(\theta_{t}),B_{v}v(\theta_{t})\rangle-\langle
B_{u}u(\theta),B_{v}v(\theta)\rangle)\,.$

First, we can show that

$\displaystyle\Delta(t)$ $\displaystyle=\langle
B_{u}(1/t)(u(\theta_{t})-u(\theta)),B_{v}v(\theta_{t})\rangle+\langle
B_{u}u(\theta),B_{v}(1/t)(v(\theta_{t})-v(\theta))\rangle$
$\displaystyle=\langle(1/t)(u(\theta_{t})-u(\theta)),B_{u}^{*}B_{v}v(\theta)\rangle+\langle
B_{v}*B_{u}u(\theta),(1/t)(v(\theta_{t})-v(\theta))\rangle$
$\displaystyle\qquad+\langle(1/t)(u(\theta_{t})-u(\theta)),B_{u}^{*}B_{v}(v(\theta_{t})-v(\theta))\rangle\,.$

Now, given our assumptions, we have that $(1/t)(u(\theta_{t})-u(\theta))\to
u^{\prime}_{i}(\theta)$ and $(1/t)(v(\theta_{t})-v(\theta))\to
v^{\prime}_{i}(\theta)$ as $t\to 0$. Therefore the first two terms above
converge to $\langle u^{\prime}_{i}(\theta),B_{u}^{*}B_{v}v(\theta)\rangle$
and $\langle B_{v}^{*}B_{u}u(\theta),v^{\prime}_{i}(\theta)\rangle$
respectively as $t\to 0$, since by assumption the elements
$B_{v}^{*}B_{u}u(\theta)$ and $B_{u}^{*}B_{v}v(\theta)$ are finite.

Next, by our continuity assumption on $B_{u}^{*}B_{v}v(\theta)$, we have
$\|B_{u}^{*}B_{v}(v(\theta_{t})-v(\theta))\|\to 0$ as $t\to 0$. Furthermore,
$\|(1/t)(u(\theta_{t})-u(\theta))\|\leq\|u^{\prime}_{k}(\theta)\|+\|(1/t)(u(\theta_{t})-u(\theta))-u^{\prime}_{k}(\theta)\|\to\|u^{\prime}_{k}(\theta)\|$
as $t\to 0$. Therefore, by Cauchy Schwartz this third term above vanishes as
$t\to 0$.

Putting the above together, we get

$\displaystyle\Delta(t)$ $\displaystyle\to\langle
u^{\prime}_{i}(\theta),B_{u}^{*}B_{v}v(\theta)\rangle+\langle
B_{v}^{*}B_{u}u(\theta),v^{\prime}_{i}(\theta)\rangle$ $\displaystyle=\langle
B_{u}u^{\prime}_{i}(\theta),B_{v}v(\theta)\rangle+\langle
B_{u}u(\theta),B_{v}v^{\prime}_{i}(\theta)\rangle$

as $t\to 0$, which completes the proof.

∎

######  Lemma 28.

Let the assumptions of Theorem 2 be given, and let $J_{n}$ be defined as in
Eq. 20. Then we have

$\displaystyle\frac{\partial}{\partial\theta_{i}}J_{n}(\theta)$
$\displaystyle=2\langle
B_{n}\frac{\partial}{\partial\theta_{i}}\bar{h}_{n}(\theta),B_{n}\bar{h}_{n}(\theta)\rangle$
$\displaystyle\frac{\partial^{2}}{\partial\theta_{i}\partial\theta_{j}}J_{n}(\theta)$
$\displaystyle=2\langle
B_{n}\frac{\partial}{\partial\theta_{i}}\bar{h}_{n}(\theta),B_{n}\frac{\partial}{\partial\theta_{j}}\bar{h}_{n}(\theta)\rangle+2\langle
B_{n}\frac{\partial^{2}}{\partial\theta_{i}\partial\theta_{j}}\bar{h}_{n}(\theta),B_{n}\bar{h}_{n}(\theta)\rangle\,.$

In addition, let $J$ be defined as in Eq. 21, then we similarly have

$\displaystyle\frac{\partial}{\partial\theta_{i}}J(\theta)$
$\displaystyle=2\langle
B\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta),B\bar{h}(\theta)\rangle$
$\displaystyle\frac{\partial^{2}}{\partial\theta_{i}\partial\theta_{j}}J(\theta)$
$\displaystyle=2\langle
B\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta),B\frac{\partial}{\partial\theta_{j}}\bar{h}(\theta)\rangle+2\langle
B\frac{\partial^{2}}{\partial\theta_{i}\partial\theta_{j}}\bar{h}(\theta),B\bar{h}(\theta)\rangle\,,$

###### Proof.

First we note that by Assumptions 2 and 6, the functions $\rho$,
$\rho^{\prime}_{i}$, and $\rho^{\prime\prime}_{i,j}$ all satisfy the
smoothness assumptions required by Lemma 26, so we have that the elements
$(\partial/\partial\theta_{i})\bar{h}_{n}(\theta)$ and
$(\partial/\partial\theta_{i})\bar{h}(\theta)$ as defined in Eqs. 16 and 17
are the derivatives of $\bar{h}_{n}(\theta)$ and $\bar{h}(\theta)$
respectively with respect to $\theta_{i}$, and the elements
$(\partial^{2}/\partial\theta_{i}\partial\theta_{j})\bar{h}_{n}(\theta)$ and
$(\partial^{2}/\partial\theta_{i}\partial\theta_{j})\bar{h}(\theta)$ as
defined in Eqs. 18 and 19 are the derivatives of
$(\partial/\partial\theta_{i})\bar{h}_{n}(\theta)$ and
$(\partial/\partial\theta_{i})\bar{h}(\theta)$ respectively with respect to
$\theta_{j}$.

Now, consider the case of $J_{n}$ first. Given that $B_{n}$ is bounded by
construction (with norm at most $\alpha_{n}^{-1/2}$), it is clearly the case
that $\|B_{n}^{2}\bar{h}_{n}(\theta)\|<\infty$, and
$\|B_{n}^{2}(\partial/\partial\theta_{i})\bar{h}_{n}(\theta)\|<\infty$ for
each $i\in[b]$. In addition, given Assumptions 2 and 6 the functions
$\bar{h}_{n}(\theta)$ and $(\partial/\partial\theta_{i})\bar{h}_{n}(\theta)$
satisfy the conditions of Lemma 19 for each $i\in[b]$, so therefore
$\bar{h}_{n}(\theta)$ and $(\partial/\partial\theta_{i})\bar{h}_{n}(\theta)$
are Lipschitz-continuous in $\theta$. Then, given the boundedness of $B_{n}$
we have that $B_{n}^{2}\bar{h}_{n}(\theta)$ and
$B_{n}^{2}(\partial/\partial\theta_{i})\bar{h}_{n}(\theta)$ are Lipschitz
continuous in $\theta$ for each $i\in[b]$. Given all of this, it follows from
Lemma 27 that the above equations for
$(\partial/\partial\theta_{i})J_{n}(\theta)$ and
$(\partial^{2}/\partial\theta_{i}\partial\theta_{j})J_{n}(\theta)$ hold.

Finally, consider the case of $J$. Again, by Assumptions 2 and 6 we can appeal
to Lemma 19, which gives us that $\|B^{2}\bar{h}(\theta)\|<\infty$, and
$\|B^{2}(\partial/\partial\theta_{i})\bar{h}(\theta)\|<\infty$ for each
$i\in[b]$. In addition, also by Lemma 19, we have that $B^{2}\bar{h}(\theta)$
and $B^{2}(\partial/\partial\theta_{i})\bar{h}(\theta)$ are Lipschitz
continuous in $\theta$, for each $i\in[b]$. Therefore, applying Lemma 27
again, the above equations for $(\partial/\partial\theta_{i})J(\theta)$ and
$(\partial^{2}/\partial\theta_{i}\partial\theta_{j})J(\theta)$ hold, which
completes the proof.

∎

######  Lemma 29.

Let $J_{n}$ be defined as in Eq. 20, and let the assumptions of Theorem 2 be
given. Then
$\sup_{\theta\in\Theta}\lvert(\partial/\partial\theta_{i})J_{n}(\theta)\rvert=O_{p}(1)$
and
$\sup_{\theta\in\Theta}\lvert(\partial^{2}/\partial\theta_{i}\partial\theta_{j})J_{n}(\theta)\rvert=O_{p}(1)$
for each $i,j\in[b]$.

###### Proof.

Proof of Lemma 29

First, applying the result of Lemma 28, we can bound

$\displaystyle\sup_{\theta\in\Theta}\lvert(\partial/\partial\theta_{i})J_{n}(\theta)\rvert$
$\displaystyle\leq\sup_{\theta\in\Theta}\lvert(\partial/\partial\theta_{i})J(\theta)\rvert+\sup_{\theta\in\Theta}\lvert(\partial/\partial\theta_{i})(J_{n}(\theta)-J(\theta))\rvert$
$\displaystyle\leq\sup_{\theta\in\Theta}\lvert(\partial/\partial\theta_{i})J(\theta)\rvert+2\sup_{\theta\in\Theta}\lvert\langle
B_{n}\frac{\partial}{\partial\theta_{i}}\bar{h}_{n}(\theta)-B\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta),B_{n}\bar{h}_{n}(\theta)\rangle\rvert$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\lvert\langle
B\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta),B_{n}\bar{h}_{n}(\theta)-B\bar{h}(\theta)\rangle\rvert$
$\displaystyle\leq
2\sup_{\theta\in\Theta}\|B\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta)\|\sup_{\theta\in\Theta}\|B\bar{h}(\theta)\|$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\|B_{n}\frac{\partial}{\partial\theta_{i}}\bar{h}_{n}(\theta)-B\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta)\|\sup_{\theta\in\Theta}\|B_{n}\bar{h}_{n}(\theta)\|$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\|B_{n}\bar{h}_{n}(\theta)-B\bar{h}(\theta)\|\sup_{\theta\in\Theta}\|B\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta)\|$
$\displaystyle\leq
2\sup_{\theta\in\Theta}\|B\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta)\|\sup_{\theta\in\Theta}\|B\bar{h}(\theta)\|$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\|B_{n}\frac{\partial}{\partial\theta_{i}}\bar{h}_{n}(\theta)-B\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta)\|\sup_{\theta\in\Theta}\|B\bar{h}(\theta)\|$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\|B_{n}\frac{\partial}{\partial\theta_{i}}\bar{h}_{n}(\theta)-B\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta)\|\sup_{\theta\in\Theta}\|B_{n}\bar{h}_{n}(\theta)-B\bar{h}(\theta)\|$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\|B_{n}\bar{h}_{n}(\theta)-B\bar{h}(\theta)\|\sup_{\theta\in\Theta}\|B\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta)\|\,.$

Now, given Assumptions 2 and 6 the elements $\bar{h}_{n}(\theta)$ and
$\bar{h}(\theta)$, and $(\partial/\partial\theta_{i})\bar{h}_{n}(\theta)$ and
$(\partial/\partial\theta_{i})\bar{h}(\theta)$ satisfy the conditions of Lemma
19, so it follows by Lemma 19 and Lemma 20 that every term appearing in the
final bound above is stochastically bounded, and therefore
$\sup_{\theta\in\Theta}\lvert(\partial/\partial\theta_{i})J_{n}(\theta)\rvert=O_{p}(1)$.

Next, applying the result of Lemma 28 for the second derivatives, and using a
derivation with exactly the same structure as above we have

$\sup_{\theta\in\Theta}\lvert(\partial^{2}/\partial\theta_{i}\partial\theta_{j})J_{n}(\theta)\rvert\leq\xi_{1}+\xi_{2}\,,$

where

$\displaystyle\xi_{1}$
$\displaystyle=2\sup_{\theta\in\Theta}\|B\frac{\partial}{\partial\theta_{j}}\bar{h}(\theta)\|\sup_{\theta\in\Theta}\|B\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta)\|$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\|B_{n}\frac{\partial}{\partial\theta_{j}}\bar{h}_{n}(\theta)-B\frac{\partial}{\partial\theta_{j}}\bar{h}(\theta)\|\sup_{\theta\in\Theta}\|B\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta)\|$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\|B_{n}\frac{\partial}{\partial\theta_{j}}\bar{h}_{n}(\theta)-B\frac{\partial}{\partial\theta_{j}}\bar{h}(\theta)\|\sup_{\theta\in\Theta}\|B_{n}\frac{\partial}{\partial\theta_{i}}\bar{h}_{n}(\theta)-B\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta)\|$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\|B_{n}\frac{\partial}{\partial\theta_{i}}\bar{h}_{n}(\theta)-B\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta)\|\sup_{\theta\in\Theta}\|B\frac{\partial}{\partial\theta_{j}}\bar{h}(\theta)\|\,,$

and

$\displaystyle\xi_{2}$
$\displaystyle=2\sup_{\theta\in\Theta}\|B\frac{\partial^{2}}{\partial\theta_{i}\partial\theta_{j}}\bar{h}(\theta)\|\sup_{\theta\in\Theta}\|B\bar{h}(\theta)\|$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\|B_{n}\frac{\partial^{2}}{\partial\theta_{i}\partial\theta_{j}}\bar{h}_{n}(\theta)-B\frac{\partial^{2}}{\partial\theta_{i}\partial\theta_{j}}\bar{h}(\theta)\|\sup_{\theta\in\Theta}\|B\bar{h}(\theta)\|$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\|B_{n}\frac{\partial^{2}}{\partial\theta_{i}\partial\theta_{j}}\bar{h}_{n}(\theta)-B\frac{\partial^{2}}{\partial\theta_{i}\partial\theta_{j}}\bar{h}(\theta)\|\sup_{\theta\in\Theta}\|B_{n}\bar{h}_{n}(\theta)-B\bar{h}(\theta)\|$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\|B_{n}\bar{h}_{n}(\theta)-B\bar{h}(\theta)\|\sup_{\theta\in\Theta}\|B\frac{\partial^{2}}{\partial\theta_{i}\partial\theta_{j}}\bar{h}(\theta)\|\,.$

Similar to above, given Assumptions 2 and 6 the elements $\bar{h}_{n}(\theta)$
and $\bar{h}(\theta)$, $(\partial/\partial\theta_{i})\bar{h}_{n}(\theta)$ and
$(\partial/\partial\theta_{i})\bar{h}(\theta)$, and elements
$(\partial^{2}/\partial\theta_{i}\partial\theta_{j})\bar{h}_{n}(\theta)$ and
$(\partial^{2}/\partial\theta_{i}\partial\theta_{j})\bar{h}(\theta)$ all
satisfy the conditions of Lemma 19, so it follows by Lemma 19 and Lemma 20
that every term appearing in the final bound above is stochastically bounded.
Given this, we also have
$\sup_{\theta\in\Theta}\lvert(\partial^{2}/\partial\theta_{i}\partial\theta_{j})J_{n}(\theta)\rvert=O_{p}(1)$,
which completes the proof.

∎

######  Lemma 30.

Let the matrices $H_{n}(\theta)$ and $H(\theta)$ be defined according to

$\displaystyle H_{n}(\theta)_{i,j}$
$\displaystyle=\frac{\partial^{2}}{\partial\theta_{i}\partial\theta_{j}}J_{n}(\theta)$
$\displaystyle H(\theta)_{i,j}$
$\displaystyle=\frac{\partial^{2}}{\partial\theta_{i}\partial\theta_{j}}J(\theta)\,.$

Let the assumptions of Theorem 2 be given, and suppose that the sequence
$\theta_{n}$ converges in probability to $\theta_{0}$. Then we have
$H_{n}(\theta_{n})\to H(\theta_{0})$ in probability.

###### Proof of Lemma 30.

We first note that

$\|H_{n}(\theta_{n})-H(\theta_{0})\|\leq\sup_{\theta\in\Theta}\|H_{n}(\theta)-H(\theta)\|+\|H(\theta_{n})-H(\theta_{0})\|\,.$

For the first, of these terms, we note that it is sufficient to argue that

$\sup_{\theta\in\Theta}\lvert\beta^{\top}(H_{n}(\theta)-H(\theta)\beta\rvert\to
0\,,$

for each $\beta\in\mathbb{R}^{b}$. Define the elements
$h_{n}^{\beta}(\theta)$, $h^{\beta}(\theta),g_{n}^{\beta}(\theta)$,
$g^{\beta}(\theta)\in\mathcal{H}$ according to

$\displaystyle h_{n}^{\beta}(\theta)(f)$
$\displaystyle=\mathbb{E}_{n}\left[f(Z)^{\top}\sum_{i=1}^{b}\beta_{i}\rho^{\prime}_{i}(X;\theta)\right]$
$\displaystyle h^{\beta}(\theta)(f)$
$\displaystyle=\mathbb{E}\left[f(Z)^{\top}\sum_{i=1}^{b}\beta_{i}\rho^{\prime}_{i}(X;\theta)\right]$
$\displaystyle g_{n}^{\beta}(\theta)(f)$
$\displaystyle=\mathbb{E}_{n}\left[f(Z)^{\top}\sum_{i,j=1}^{b}\beta_{i}\beta_{j}\rho^{\prime\prime}_{i,j}(X;\theta)\right]$
$\displaystyle g^{\beta}(\theta)(f)$
$\displaystyle=\mathbb{E}\left[f(Z)^{\top}\sum_{i,j=1}^{b}\beta_{i}\beta_{j}\rho^{\prime\prime}_{i,j}(X;\theta)\right]\,.$

Then, by Lemma 28 we have

$\displaystyle\beta^{\top}H_{n}(\theta)\beta$
$\displaystyle=2\|B_{n}h_{n}^{\beta}(\theta)\|^{2}+2\langle
B_{n}g_{n}^{\beta}(\theta),B_{n}\bar{h}_{n}(\theta)\rangle$
$\displaystyle\beta^{\top}H(\theta)\beta$
$\displaystyle=2\|Bh^{\beta}(\theta)\|^{2}+2\langle
Bg^{\beta}(\theta),B\bar{h}(\theta)\rangle\,.$

Given this, we can bound

$\displaystyle\sup_{\theta\in\Theta}$
$\displaystyle\lvert\beta^{\top}(H_{n}(\theta)-H(\theta))\beta\rvert\leq
2\sup_{\theta\in\Theta}\lvert\langle
B_{n}h_{n}^{\beta}(\theta)-Bh^{\beta}(\theta),B_{n}h_{n}^{\beta}(\theta)\rangle\rvert$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\lvert\langle
Bh^{\beta}(\theta),B_{n}h_{n}^{\beta}(\theta)-Bh^{\beta}(\theta)\rangle\rvert$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\lvert\langle
B_{n}g_{n}^{\beta}(\theta)-Bg^{\beta}(\theta),B_{n}\bar{h}_{n}(\theta)\rangle\rvert$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\lvert\langle
Bg^{\beta}(\theta),B_{n}\bar{h}_{n}(\theta)-B\bar{h}(\theta)\rangle\rvert$
$\displaystyle\leq
2\sup_{\theta\in\Theta}\|B_{n}h_{n}^{\beta}(\theta)-Bh^{\beta}(\theta)\|\sup_{\theta\in\Theta}\|B_{n}h_{n}^{\beta}(\theta)\|$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\|B_{n}h_{n}^{\beta}(\theta)-Bh^{\beta}(\theta)\|\sup_{\theta\in\Theta}\|B_{n}h_{n}^{\beta}(\theta)-Bh^{\beta}(\theta)\|$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\|Bh^{\beta}(\theta)\|\sup_{\theta\in\Theta}\|B_{n}h_{n}^{\beta}(\theta)-Bh^{\beta}(\theta)\|$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\|B_{n}g_{n}^{\beta}(\theta)-Bg^{\beta}(\theta)\|\sup_{\theta\in\Theta}\|B\bar{h}(\theta)\|$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\|B_{n}g_{n}^{\beta}(\theta)-Bg^{\beta}(\theta)\|\sup_{\theta\in\Theta}\|B_{n}\bar{h}_{n}(\theta)-B\bar{h}(\theta)\|$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\|Bg^{\beta}(\theta)\|\sup_{\theta\in\Theta}\|B_{n}\bar{h}_{n}(\theta)-B\bar{h}(\theta)\|\,.$

Now, by Assumptions 2 and 6 it follows that the pairs of terms
$\bar{h}_{n}(\theta)$ and $\bar{h}(\theta)$, $h_{n}^{\beta}(\theta)$ and
$h^{\beta}(\theta)$, and $g_{n}^{\beta}(\theta)$ and $g^{\beta}(\theta)$ all
satisfy the conditions of Lemma 19, so therefore by Lemma 20 all the terms of
the form $\sup_{\theta\in\Theta}\|B_{n}h_{n}(\theta)-Bh(\theta)\|$ all
converge in probability to zero (where $(h_{n},h)$ are replaced with either
$(\bar{h}_{n},\bar{h})$, $(h_{n}^{\beta},h^{\beta})$, or
$(g_{n}^{\beta},g^{\beta})$). Also, by Lemma 19, all the terms of the form
$\sup_{\theta\in\Theta}\|Bh(\theta)\|$ are finite (where $h$ is replaced with
either $\bar{h}$, $h^{\beta}$, or $g^{\beta}$), so putting all of this
together we can conclude that
$\sup_{\theta\in\Theta}\lvert\beta^{\top}(H_{n}(\theta)-H(\theta))\beta\rvert\to
0$ in probability for any fixed $\beta$.

Next, consider the term $\|H(\theta_{n})-H(\theta_{0})\|$. As above, it is
sufficient to argue that

$\beta^{\top}(H(\theta_{n})-H(\theta_{0}))\beta\to 0\,,$

in probability for each vector $\beta$. Now, we can bound

$\displaystyle\lvert\beta^{\top}$
$\displaystyle(H(\theta_{n})-H(\theta_{0}))\beta\rvert\leq 2\lvert\langle
Bh^{\beta}(\theta_{n})-Bh^{\beta}(\theta_{0}),Bh^{\beta}(\theta_{n})\rangle\rvert$
$\displaystyle\qquad+2\lvert\langle
Bh^{\beta}(\theta_{0}),Bh^{\beta}(\theta_{n})-Bh^{\beta}(\theta_{0})\rangle\rvert$
$\displaystyle\qquad+2\lvert\langle
Bg^{\beta}(\theta_{n})-Bg^{\beta}(\theta_{0}),B\bar{h}(\theta_{n})\rangle\rvert$
$\displaystyle\qquad+2\lvert\langle
Bg^{\beta}(\theta_{0}),B\bar{h}(\theta_{n})-B\bar{h}(\theta_{0})\rangle\rvert$
$\displaystyle\leq
4\sup_{\theta\in\Theta}\|Bh^{\beta}(\theta)\|\|B(h^{\beta}(\theta_{n})-h^{\beta}(\theta_{0}))\|$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\|B\bar{h}(\theta)\|\|B(g^{\beta}(\theta_{n})-g^{\beta}(\theta_{0}))\|$
$\displaystyle\qquad+2\sup_{\theta\in\Theta}\|Bg^{\beta}(\theta)\|\|B(\bar{h}(\theta_{n})-\bar{h}(\theta_{0}))\|\,.$

Now, as argued above, the pairs of elements $\bar{h}_{n}(\theta)$ and
$\bar{h}(\theta)$, $h_{n}^{\beta}(\theta)$ and $h^{\beta}(\theta)$, and
$g_{n}^{\beta}(\theta)$ and $g^{\beta}(\theta)$ all satisfy the conditions of
Lemma 19, so therefore by Lemma 19 we have that each of the terms of the for
$\sup_{\theta\in\Theta}\|Bh(\theta)\|$ is finite (where $h$ is replaced by
either $\bar{h}$, $h^{\beta}$, or $g^{\beta}$), and each of the terms of the
form $\|B(h(\theta_{n})-h(\theta_{0})\|$ is bounded by
$L^{\prime}\|\theta_{n}-\theta_{0}\|$, for some constant $L^{\prime}$ that
doesn’t depend on $\theta_{n}$. Given this, and the assumption that
$\theta_{n}\to\theta_{0}$ in probability, it instantly follows that
$\beta^{\top}(H(\theta_{n})-H(\theta_{0}))\beta\to 0$ in probability for any
fixed $\beta$.

Therefore, we have bounded both of the terms in the right hand side in our
original bound for $\|H_{n}(\theta_{n})-H(\theta_{0})\|$, so we can conclude.

∎

######  Lemma 31.

Let the assumptions of Theorem 2 be given, and let $\theta_{n}$ be any
sequence such that $J_{n}(\theta_{n})-J_{n}(\theta_{n}^{*})=o_{p}(1/n)$, where
$\theta_{n}^{*}$ is the true minimizer of $J_{n}$. Then we have
$(\partial/\partial\theta_{i})J_{n}(\theta_{n})=o_{p}(n^{-1/2})$ for each
$i\in[b]$.

###### Proof of Lemma 31.

First, applying Taylor’s theorem to $J_{n}$ at $\theta_{n}^{*}$, and noting
that $\nabla J_{n}(\theta_{n}^{*})=0$, we have

$\displaystyle J_{n}(\theta_{n})$
$\displaystyle=J_{n}(\theta_{n}^{*})+(\theta_{n}-\theta_{n}^{*})^{\top}H_{n}(\theta_{n}^{*})(\theta_{n}-\theta_{n}^{*})+\epsilon_{n}(\theta-\theta_{n}^{*})O(\|\theta_{n}-\theta_{n}^{*}\|^{2})$
$\displaystyle=J_{n}(\theta_{n}^{*})+(\theta_{n}-\theta_{n}^{*})^{\top}H(\theta_{0})(\theta_{n}-\theta_{n}^{*})+\epsilon_{n}(\theta-\theta_{n}^{*})O(\|\theta_{n}-\theta_{n}^{*}\|^{2})$
$\displaystyle\qquad+\|H_{n}(\theta_{n}^{*})-H(\theta_{0})\|O(\|\theta_{n}-\theta_{n}^{*}\|^{2})\,,$

where $H_{n}(\theta)$ denotes the Hessian of $J_{n}$ at $\theta$, and $H$
denotes the Hessian of $J$ at $\theta$, and $\epsilon_{n}$ is some function
that satisfies $\lim_{\theta\to 0}\epsilon_{n}(\theta)=0$.

Next, by Lemma 30 we have $\|H_{n}(\theta_{n}^{*})-H(\theta_{0})\|\to 0$ in
probability assuming $\theta_{n}^{*}\to\theta_{0}$ in probability. In
addition, by the assumptions of Theorem 2 and the construction of
$\theta_{n}^{*}$, it is clear that $\theta_{n}^{*}\to\theta_{0}$ in
probability by Theorem 1. Furthermore, by Assumption 7 it is clear that
$H(\theta_{0})$ is strictly positive definite. Therefore, noting that by
assumption we have $J_{n}(\theta_{n})-J_{n}(\theta_{n}^{*})=o_{p}(1/n)$,
putting the above together instantly implies that we must have
$\|\theta_{n}-\theta_{n}^{*}\|=o_{p}(n^{-1/2})$, else we would have a
contradiction, since by our above arguments the
$(\theta_{n}-\theta_{n}^{*})^{\top}H(\theta_{0})(\theta_{n}-\theta_{n}^{*})$
term dominates for small $\theta_{n}-\theta_{n}^{*}$.

Next, applying the mean value theorem, and noting that
$(\partial/\partial\theta_{i})J_{n}(\theta_{n}^{*})=0$, we have

$\frac{\partial}{\partial\theta_{i}}J_{n}(\theta_{n})=\sum_{j=1}^{n}\frac{\partial^{2}}{\partial\theta_{i}\partial\theta_{j}}J_{n}(\theta^{\prime})(\theta_{n}-\theta_{n}^{*})_{j}\,,$

where $\theta^{\prime}$ lies on the line segment between $\theta_{n}$ and
$\theta_{n}^{*}$. However, by Lemma 29 we have that the terms
$\sup_{\theta\in\Theta}\lvert(\partial^{2}/\partial\theta_{i}\partial\theta_{j})J_{n}(\theta^{\prime})\rvert$
are stochastically bounded, and we have already argued that
$\|\theta_{n}-\theta_{n}^{*}\|=o_{p}(n^{-1/2})$. Therefore, we can conclude
that $(\partial/\partial\theta_{i})J_{n}(\theta_{n})=o_{p}(n^{-1/2})$, as
required.

∎

######  Lemma 32.

For each $n\in\mathbb{N}$, let $\alpha_{i}^{(n)}$ be some set of weights, such
that $\alpha_{i}^{(n)}\geq 0$ for each $i$, and
$\sum_{i=1}^{\infty}\alpha_{i}^{(n)}=1$. Furthermore, suppose for each
$n\in\mathbb{N}$ that $\theta_{i}^{(n)}\in\Theta$ are some set of parameter
values, such that $\theta_{i}^{(n)}$ lies on the line segment between
$\hat{\theta}_{n}$ and $\theta_{0}$ for each $i$. Furthermore assume that
$\hat{\theta}_{n}\to\theta_{0}$ in probability, and define

$\displaystyle h_{n}$
$\displaystyle=\sum_{i=1}^{\infty}\alpha_{i}^{(n)}\frac{\partial}{\partial\theta_{i}}\bar{h}_{n}(\theta_{i}^{(n)})$
$\displaystyle h$
$\displaystyle=\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta_{0})\,.$

Then under the assumptions of Theorem 2 we have $B_{n}h_{n}\to Bh$ in
probability under $\mathcal{H}$.

###### Proof of Lemma 32.

We need to show that $\|B_{n}h_{n}-Bh\|\to 0$ in probability. First, since
$B_{n}$ is linear, applying the triangle inequality we can derive the bound

$\displaystyle\|B_{n}h_{n}-Bh\|$
$\displaystyle\leq\sum_{i=1}^{\infty}\|\alpha_{i}^{(n)}B_{n}(\partial/\partial\theta_{i})\bar{h}_{n}(\theta_{i}^{(n)})-\alpha_{i}^{(n)}B(\partial/\partial\theta_{i})\bar{h}(\theta_{0})\|$
$\displaystyle\leq\sum_{i=1}^{\infty}\alpha_{i}^{(n)}\|B_{n}(\partial/\partial\theta_{i})\bar{h}_{n}(\theta_{i}^{(n)})-B(\partial/\partial\theta_{i})\bar{h}(\theta_{i}^{(n)})\|$
$\displaystyle\qquad+\sum_{i=1}^{\infty}\alpha_{i}^{(n)}\|B(\partial/\partial\theta_{i})\bar{h}(\theta_{i}^{(n)})-B(\partial/\partial\theta_{i})\bar{h}(\theta_{0})\|$
$\displaystyle\leq\sup_{\theta\in\Theta}\|B_{n}(\partial/\partial\theta_{i})\bar{h}_{n}(\theta)-B(\partial/\partial\theta_{i})\bar{h}(\theta)\|$
$\displaystyle\qquad+\sup_{\theta\in\Theta_{n}}\|B(\partial/\partial\theta_{i})\bar{h}(\theta)-B(\partial/\partial\theta_{i})\bar{h}(\theta_{0})\|\,,$

where
$\Theta_{n}=\\{\theta\in\Theta:\|\theta-\theta_{0}\|\leq\|\hat{\theta}_{n}-\theta_{0}\|\\}$.
We will proceed by bounding the two terms in the final bound above separately.

In the case of the first term, we note that the collections of functions
$(\partial/\partial\theta_{i})\bar{h}_{n}(\theta)$ and
$(\partial/\partial\theta_{i})\bar{h}(\theta)$ each indexed by
$\theta\in\Theta$ satisfy the conditions of Lemma 19 given Assumption 6, so by
Lemma 20 this first term converges in probability to zero.

For the second term above, by Lemma 22 we have

$\displaystyle\|B(\partial/\partial\theta_{i})\bar{h}(\theta)-B(\partial/\partial\theta_{i})\bar{h}(\theta_{0})\|^{2}$
$\displaystyle\qquad=\mathbb{E}[\mathbb{E}[\rho_{i}^{\prime}(X;\theta)-\rho_{i}^{\prime}(X;\theta_{0})\mid
Z]^{\top}V(Z;\tilde{\theta})^{-1}\mathbb{E}[\rho_{i}^{\prime}(X;\theta)-\rho_{i}^{\prime}(X;\theta_{0})\mid
Z]]\,.$

Now, by Assumption 6 we know that $\rho^{\prime}_{i}(x;\theta)$ is Lipschitz-
continuous in $\theta$ with constant independent of $x$, so therefore the
$L_{2}$ norm of
$\mathbb{E}[\rho_{i}^{\prime}(X;\theta)-\rho_{i}^{\prime}(X;\theta_{0})\mid
Z]$ is bounded by $k\|\theta-\theta_{0}\|$, for some constant $k$ independent
of $\theta_{0}$ and $\theta$. Next, by Assumption 5 we know that
$V(Z;\tilde{\theta})^{-1}$ is a bounded linear operator on $L_{2}$. Let $M$ be
its operator norm. Then we have

$\|B(\partial/\partial\theta_{i})\bar{h}(\theta)-B(\partial/\partial\theta_{i})\bar{h}(\theta_{0})\|^{2}\leq
Mk^{2}\|\theta-\theta_{0}\|^{2}\,.$

Given this and the definition of $\Theta_{n}$, we have

$\sup_{\theta\in\Theta_{n}}\|B(\partial/\partial\theta_{i})\bar{h}(\theta)-B(\partial/\partial\theta_{i})\bar{h}(\theta_{0})\|\leq\sqrt{M}k\|\hat{\theta}_{n}-\theta_{0}\|\,,$

and therefore the second term converges in probability to zero, since by
assumption $\hat{\theta}_{n}\to\theta_{0}$ in probability.

Putting the above together, we have $\|B_{n}h_{n}-Bh\|\to 0$ in probability,
as required.

∎

######  Lemma 33.

Let the assumptions of Theorem 2 be given, and define the random vector
$W^{(n)}$, whose $i$’th entry is given by

$W^{(n)}_{i}=\langle
B_{n}\frac{\partial}{\partial\theta_{i}}\bar{h}_{n}(\hat{\theta}_{n}),\sqrt{n}B_{n}\bar{h}_{n}(\theta_{0})\rangle\,.$

Then $W^{(n)}+\theta_{0}$ is an asymptotically linear and asymptotically
normal estimator for $\theta_{0}$, with covariance matrix $\Delta$, defined as
in the statement of Theorem 2.

###### Proof of Lemma 33.

First, we note that since $B_{n}$ is self-adjoint, we have

$W^{(n)}_{i}=\langle
B_{n}^{2}\frac{\partial}{\partial\theta_{i}}\bar{h}_{n}(\hat{\theta}_{n}),\sqrt{n}\bar{h}_{n}(\theta_{0})\rangle\,.$

Now, by Lemma 19 and Assumption 6, it easily follows that

$\|\frac{\partial}{\partial\theta_{i}}\bar{h}_{n}(\hat{\theta}_{n})-\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta_{0})\|=O_{p}(n^{-1/2})\,.$

In addition, noting that $B^{2}=C^{-1}$, we have

$\displaystyle\|B^{2}\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta_{0})\|$
$\displaystyle=\sup_{\|h\|_{\mathcal{H}}\leq 1}\langle
C^{-1}\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta_{0}),h\rangle$
$\displaystyle=\sup_{\|f\|_{\mathcal{F}}\leq
1}(C^{-1}\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta_{0}))(f)$
$\displaystyle=\sup_{\|f\|_{\mathcal{F}}\leq
1}\mathbb{E}[f(Z)^{\top}V(Z;\tilde{\theta})^{-1}\mathbb{E}[\rho^{\prime}_{i}(X;\theta_{0})\mid
Z]]\,,$

where in the above we apply Lemmas 10 and 18. Now, it easily follows from
Lemma 13 that $\\{f:\|f\|\leq 1\\}$ is bounded in $L_{\infty}$ norm, and in
addition it follows from Assumption 6 that
$\mathbb{E}[\rho^{\prime}_{i}(X)\mid Z]$ has finite $L_{\infty}$ norm.
Furthermore, by Assumption 5 $V(Z;\tilde{\theta})^{-1}$ is a bounded linear
operator on $L_{2}$, and since $L_{2}$ norms are bounded by $L_{\infty}$ norms
it follows that $\|B^{2}(\partial/\partial\theta_{i})\bar{h}(\theta_{0})\|$ is
finite. Given the above, and Lemma 17, it follows from Lemma 21 that
$B_{n}^{2}(\partial/\partial\theta_{i})\bar{h}_{n}(\hat{\theta}_{n})\to
B^{2}(\partial/\partial\theta_{i})\bar{h}(\theta_{0})$ in probability under
$\mathcal{H}$.

Next, note that
$\bar{h}_{n}(\theta_{0})(f)=\mathbb{E}_{n}[f(Z)^{\top}\mathbb{E}[\rho(X;\theta_{0})\mid
Z]]$. Now, by Lemma 14 the function class
$\mathcal{G}=\\{g:g(z)=f(z)^{\top}\mathbb{E}[\rho(X;\theta_{0})\mid
Z=z],\|f\|\leq 1\\}$ is $P$-Donsker in the sense of Kosorok [2007], and thus
it follows from Lemma 15 that $\sqrt{n}\bar{h}_{n}(\theta_{0})$ is
stochastically bounded. Therefore, given the previous convergence result, we
have

$\displaystyle W_{i}^{(n)}$ $\displaystyle=\langle
B^{2}\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta_{0}),\sqrt{n}\bar{h}_{n}(\theta_{0})\rangle+o_{p}(1)$
$\displaystyle=\frac{1}{\sqrt{n}}\sum_{j=1}^{n}\langle
q_{i},h(X_{j};\theta_{0})\rangle+o_{p}(1)\,,$

where $q_{i}=B^{2}(\partial/\partial\theta_{i})\bar{h}(\theta_{0})$.
Furthermore, we note that by the definition of $\theta_{0}$ we have
$\mathbb{E}[\langle q_{i},h(X;\theta_{0})\rangle]=0$, and

$\displaystyle\mathbb{E}[\langle q_{i},h(X;\theta_{0})\rangle\langle
q_{j},h(X;\theta_{0})\rangle]$
$\displaystyle=\mathbb{E}[f_{i}(Z)^{\top}\rho(X;\theta_{0})f_{j}(Z)^{\top}\rho(X;\theta_{0})]$
$\displaystyle=\langle C_{0}q_{i},q_{j}\rangle$
$\displaystyle=\langle(BC_{0}B)B\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta_{0}),B\frac{\partial}{\partial\theta_{j}}\bar{h}(\theta_{0})\rangle$
$\displaystyle=\Delta_{i,j}\,,$

where $f_{i},f_{j}\in\mathcal{F}$ are the dual elements corresponding to
$q_{i},q_{j}\in\mathcal{H}$, and $\Delta$ is defined as in the statement of
Theorem 2 (since $B=C^{-1/2}$). Therefore, we conclude by the central limit
theorem that $W^{(n)}+\theta_{0}$ is an asymptotically linear and
asymptotically normal estimator for $\theta_{0}$, with covariance matrix
$\Delta$, as required.

∎

######  Lemma 34.

Define the matrices $\Omega$ and $\Omega^{\prime}$ according to

$\displaystyle\Omega_{i,j}$
$\displaystyle=\left<B\frac{\partial\bar{h}(\theta_{0})}{\partial\theta_{i}},B\frac{\partial\bar{h}(\theta_{0})}{\partial\theta_{j}}\right>$
$\displaystyle\Omega^{\prime}_{i,j}$
$\displaystyle=\mathbb{E}[\mathbb{E}[\rho^{\prime}_{i}(X;\theta_{0})\mid
Z]^{\top}V(Z;\tilde{\theta})^{-1}\mathbb{E}[\rho^{\prime}_{j}(X;\theta_{0})\mid
Z]]\,.$

Then $\Omega=\Omega^{\prime}$.

###### Proof of Lemma 34.

First, we note that for any arbitrary vector $\alpha$ we have

$\displaystyle\alpha^{\top}\Omega\alpha$
$\displaystyle=\|B\sum_{i=1}^{n}\alpha_{i}(\partial/\partial\theta_{i})\bar{h}(\theta_{0})\|^{2}$
$\displaystyle=\mathbb{E}[\mathbb{E}[\sum_{i=1}^{n}\alpha_{i}\rho^{\prime}_{i}(Z;\theta_{0})]^{\top}V(Z;\tilde{\theta})^{-1}\mathbb{E}[\sum_{i=1}^{n}\alpha_{i}\rho^{\prime}_{i}(Z;\theta_{0})]]$
$\displaystyle=\alpha^{\top}\Omega^{\prime}\alpha\,,$

where the second equality follows from Lemma 22. Now given this, it is clear
that both $\Omega$ and $\Omega^{\prime}$ are symmetric and PSD, since clearly
$\|B\sum_{i=1}^{n}\alpha_{i}(\partial/\partial\theta_{i})\bar{h}(\theta_{0})\|^{2}\geq
0$. However given these matrices are symmetric and PSD, having
$\alpha^{\top}\Omega\alpha=\alpha^{\top}\Omega^{\prime}\alpha$ for every
$\alpha$ ensures that $\Omega=\Omega^{\prime}$, which gives us our required
result.

∎

######  Lemma 35.

Let $f_{n}^{*}(\theta)\in\mathcal{F}$ be the function maximizing
$U_{n}(\theta,f)$ for any given $\theta$. Then under the assumptions of
Theorem 1, there exists some constant $\eta$ such that

$\sup_{\theta\in\Theta}\|f_{n}^{*}(\theta)\|_{\mathcal{F}}\leq\eta+o_{p}(1)\,.$

###### Proof.

Recall that

$\displaystyle U_{n}(\theta,f)$
$\displaystyle=\mathbb{E}_{n}[f(Z)^{t}\rho(X;\theta)]-\frac{1}{4}\mathbb{E}_{n}[f(Z)^{\top}\rho(X;\tilde{\theta}_{n})]-\frac{\alpha_{n}}{4}\|f\|^{2}$
$\displaystyle=\langle\bar{h}_{n}(\theta),h\rangle-\frac{1}{4}\langle(C+\alpha_{n}I)h,h\rangle\,,$

where $h$ is the element of $\mathcal{H}$ corresponding to $f\in\mathcal{F}$
by the duality isomorphism. Now, consider the Gateaux derivative of the above
at $h_{n}^{*}(\theta)$ in the direction of $\epsilon$. This Gateaux derivative
is clearly given by

$\langle\bar{h}_{n}(\theta),\epsilon\rangle-\frac{1}{2}\langle(C+\alpha_{n}I)h_{n}^{*}(\theta),\epsilon\rangle\,.$

For $h_{n}^{*}(\theta)$ to be the optimizing element, we require this
derivative to be zero for every $\epsilon\in\mathcal{H}$, which immediately
gives us

$\displaystyle h_{n}^{*}(\theta)$
$\displaystyle=2(C_{n}+\alpha_{n})^{-1}\bar{h}_{n}(\theta)$
$\displaystyle=2B_{n}^{2}\bar{h}_{n}(\theta)\,.$

Next, define $h^{*}(\theta)=2B^{2}\bar{h}(\theta)$. We note that by Assumption
2, we have that $\bar{h}_{n}(\theta)$ and $\bar{h}(\theta)$ satisfy the
conditions of Lemma 19. Therefore by Lemma 19 we have
$\sup_{\theta\in\Theta}\|B^{2}\bar{h}(\theta)\|<\infty$. Furthermore, by
assumption the conditions of Lemma 17, so therefore by Lemma 21 we have
$\sup_{\theta\in\Theta}\|B_{n}^{2}\bar{h}_{n}(\theta)-B^{2}\bar{h}(\theta)\|=o_{p}(1)$.
Putting this together, we have

$\displaystyle\sup_{\theta\in\Theta}\|f_{n}^{*}(\theta)\|$
$\displaystyle=\sup_{\theta\in\Theta}\|h_{n}^{*}(\theta)\|$
$\displaystyle\leq\sup_{\theta\in\Theta}\|h^{*}(\theta)\|+\sup_{\theta\in\Theta}\|h_{n}^{*}(\theta)-h^{*}(\theta)\|$
$\displaystyle=2\sup_{\theta\in\Theta}\|B^{2}\bar{h}(\theta)\|+2\sup_{\theta\in\Theta}\|B_{n}^{2}\bar{h}_{n}(\theta)-B^{2}\bar{h}(\theta)\|$
$\displaystyle=\eta+o_{p}(1)\,,$

where we define $\eta=2\sup_{\theta\in\Theta}\|B^{2}\bar{h}(\theta)\|$, which
as argued above is finite.

∎

##  Appendix C Omitted Proofs

###### Proof of Lemma 2.

First, note that clearly $C_{n}$ has rank at most $n$, as is therefore
compact. Therefore, applying Lemma 12, we get

$\displaystyle\|($ $\displaystyle
C_{n}+\alpha_{n})^{-1/2}\bar{h}_{n}(\theta)\|^{2}$
$\displaystyle=\sup_{h^{\prime}\in\mathcal{H}}\langle\bar{h}_{n}(\theta),h^{\prime}\rangle-\frac{1}{4}\langle(C_{n}+\alpha_{n}I)h^{\prime},h^{\prime}\rangle$
$\displaystyle=\sup_{h\in\mathcal{H}}\mathbb{E}_{n}[\varphi(h)(Z)^{\top}\rho(X;\theta)]-\frac{1}{4}\mathbb{E}_{n}[(\varphi(h)(Z)^{\top}\rho(X;\tilde{\theta}_{n}))^{2}]-\frac{1}{4}\alpha_{n}\|h^{\prime}\|^{2}\,,$

where in the above we apply Lemma 10, and $\varphi$ maps any element of
$\mathcal{H}$ to its dual element in $\mathcal{F}$. The required result then
immediately follows given the duality between $\mathcal{H}$ and $\mathcal{F}$.

∎

###### Proof of Theorem 1.

Let $J_{n}$ and $J$ be defined as in Eqs. 20 and 21. Now by assumption (LINK)
we have $J_{n}(\hat{\theta}_{n})\leq\inf_{\theta}J_{n}(\theta)+o_{p}(1)$, and
by Lemma 24 we have $\sup_{\theta\in\Theta}\lvert
J_{n}(\theta)-J(\theta)\rvert\to 0$ in probability. Therefore we have

$\displaystyle J(\hat{\theta}_{n})$ $\displaystyle\leq
J_{n}(\hat{\theta}_{n})+o_{p}(1)$
$\displaystyle\leq\inf_{\theta}J_{n}(\theta)+o_{p}(1)$ $\displaystyle\leq
J_{n}(\theta_{0})+o_{p}(1)$ $\displaystyle\leq J(\theta_{0})+o_{p}(1)\,.$

Next, suppose that $\hat{\theta}_{n}$ does not converge in probability to
$\theta_{0}$. This implies that for some $\epsilon>0$, we have that
$P(\|\hat{\theta}_{n}-\theta_{0}\|<\epsilon)$ does not converge to zero. Let
$J^{*}=\inf_{\theta:\|\theta-\theta_{0}\|\geq\epsilon}J(\theta)$. Then it must
be the case that $\limsup_{n\to\infty}J(\hat{\theta}_{n})\geq J^{*}$.
Furthermore, by Lemma 23 $J(\theta)$ is continuous in $\theta$, and by Lemma
25 we know that $\theta_{0}$ is the unique element of $\Theta$ such that
$J(\theta_{0})=0$, so we have that $J^{*}>0$. However, we have already
established that $J(\hat{\theta}_{n})\leq J(\theta_{0})+o_{p}(1)$, so
therefore $\limsup_{n\to\infty}J(\hat{\theta}_{n})=0$. Thus we have a
contradiction, so we conclude that $\hat{\theta}_{n}\to\theta_{0}$ in
probability.

∎

###### Proof of Theorem 2.

First, by assumption we have $J_{n}(\hat{\theta}_{n})=o_{p}(1/n)$. It follows
from Lemma 31 that
$(\partial/\partial\theta_{i})J_{n}(\hat{\theta}_{n})=o_{p}(n^{-1/2})$, for
each $i\in[b]$. Therefore, applying Lemma 28, we have

$\langle
B_{n}\frac{\partial\bar{h}_{n}}{\partial\theta_{i}}(\hat{\theta}_{n}),B_{n}\bar{h}_{n}(\hat{\theta}_{n})\rangle=o_{p}(1/\sqrt{n})\quad\forall
i\in[b].$

Next, by Lemmas 26 and 9, we have

$\bar{h}_{n}(\hat{\theta}_{n})=\bar{h}_{n}(\theta_{0})+\sum_{i=1}^{\infty}\beta_{i}^{(n)}\sum_{j=1}^{b}\frac{\partial}{\partial\theta_{j}}\bar{h}_{n}(\theta_{i}^{(n)})(\hat{\theta}_{n}-\theta_{0})_{j},$

for some sequences $\beta_{i}^{(n)}$ and $\theta_{i}^{(n)}$, where
$\beta_{i}^{(n)}\geq 0$ and
$\theta_{i}^{(n)}\in\text{segment}(\theta_{0},\hat{\theta}_{n})\ \forall i$,
and $\sum_{i}\beta_{i}^{(n)}=1$. Combining together the previous two equations
we obtain

$\langle
B_{n}\nabla\bar{h}_{n}(\hat{\theta}_{n}),B_{n}\bar{h}_{n}(\theta_{0})\rangle+T^{(n)}(\hat{\theta}_{n}-\theta_{0})=o_{p}(1/\sqrt{n}),$

where

$T^{(n)}_{jk}=\langle
B_{n}\frac{\partial\bar{h}_{n}}{\partial\theta_{j}}(\hat{\theta}_{n}),B_{n}\sum_{i=1}^{\infty}\beta_{i}^{(n)}\frac{\partial\bar{h}_{n}}{\partial\theta_{k}}(\theta_{i}^{(n)})\rangle.$

By assumption the conditions of Theorem 1 hold, so we have
$\hat{\theta}_{n}\rightarrow\theta_{0}$ in probability, and thus by Lemma 32
and the continuous mapping theorem it easily follows that $T^{(n)}\to\Omega$
in probability, where

$\Omega_{i,j}=\left<B\frac{\partial\bar{h}(\theta_{0})}{\partial\theta_{i}},B\frac{\partial\bar{h}(\theta_{0})}{\partial\theta_{j}}\right>.$

By Lemma 34, the matrix $\Omega$ takes the same value as in the theorem
statement. Furthermore, for any $v\in\mathbb{R}^{m}$ we have

$\displaystyle v^{\top}\Omega v$
$\displaystyle=\|B\sum_{i=1}^{m}v_{i}\frac{\partial\bar{h}(\theta_{0})}{\partial\theta_{i}}\|^{2}$
$\displaystyle=\mathbb{E}[\mathbb{E}[\sum_{i=1}^{m}v_{i}\rho^{\prime}_{i}(X;\theta_{0})\mid
Z]^{\top}V(Z;\tilde{\theta})^{-1}\mathbb{E}[\sum_{i=1}^{m}v_{i}\rho^{\prime}_{i}(X;\theta_{0})\mid
Z]]\,,$

so it follows from Assumptions 5 and 7 that $v^{\top}\Omega v>0$ for any non-
zero $v$, and therefore $\Omega$ is invertible. Thus, with probability
approaching one, we have

$\displaystyle\sqrt{n}(\hat{\theta}_{n}-\theta_{0})$
$\displaystyle=-(T^{(n)})^{-1}W^{(n)}+o_{p}(1)$
$\displaystyle=-\Omega^{-1}W^{(n)}+(\Omega^{-1}-(T^{(n)})^{-1})W^{(n)}+o_{p}(1)\,,$

where

$W^{(n)}=\left<B_{n}\frac{\partial\bar{h}_{n}}{\partial\theta}(\hat{\theta}_{n}),\sqrt{n}B_{n}\bar{h}_{n}(\theta_{0})\right>\,.$

Now by Lemma 33 it follows that $W^{(n)}$ is asymptotically linear and
asymptotically normal, with mean zero and covariance matrix $\Delta$. This
implies that it is also stochstically bounded, so applying the continuous
mapping theorem to the previous convergence result for $T^{(n)}$ we have

$\sqrt{n}(\hat{\theta}_{n}-\theta_{0})=-\Omega^{-1}W^{(n)}+o_{p}(1)\,.$

Therefore, we have that $\hat{\theta}_{n}$ is an asymptotically linear and
asymptotically normal estimator for $\theta_{0}$, with covariance matrix
$\Omega^{-1}\Delta\Omega^{-1}$, as required.

∎

###### Proof of Theorem 3.

For the first part of the theorem, we note that in the case that
$\tilde{\theta}=\theta_{0}$, we clearly have

$\Delta_{i,j}=\langle(B\frac{\partial}{\partial\theta_{i}}\bar{h}(\theta_{0}),B\frac{\partial}{\partial\theta_{j}}\bar{h}(\theta_{0})\rangle\,,$

since in this case we have $C=C_{0}$. Next, by Lemma 34 it is clear that
$\Delta=\Omega_{0}$, and since $\tilde{\theta}=\theta_{0}$ we have
$\Omega=\Omega_{0}$. Therefore we have that the asymptotic variance of
$\sqrt{n}(\hat{\theta}_{n}-\theta_{0})$ is given by $\Omega_{0}^{-1}$, as
required.

For the second part of the theorem, we first note that this limiting
covariance matches the efficiency bound of Chamberlain [1987]. Furthermore, by
Theorem 2 we know that the estimator is asymptotically linear, so by Kosorok
[2007, Theorem 18.7] we conclucde that it is also regular, and efficient
relative to all possible regular asymptotically linear estimators.

Finally, in order to argue that this estimator is efficient within the class
of VMM estimators with different values of $\tilde{\theta}$, we need to argue
that $\Omega^{-1}\Delta\Omega^{-1}-\Omega_{0}^{-1}$ is PSD for any
$\tilde{\theta}$. This is equivalent to showing that
$\alpha^{\top}\Delta\alpha-\alpha^{\top}\Omega\Omega_{0}^{-1}\Omega\alpha\geq
0$ for every vector $\alpha$. Given the definitions of $\Delta$, $\Omega$, and
$\Omega_{0}$, and defining

$q_{\alpha}=\sum_{i=1}^{m}\alpha_{i}B\frac{\partial}{\partial\theta_{j}}\bar{h}(\theta_{0})\,,$

we have

$\displaystyle\alpha^{\top}\Delta\alpha-\alpha^{\top}\Omega\Omega_{0}^{-1}\Omega\alpha$
$\displaystyle=\langle(BC_{0}B)q_{\alpha},q_{\alpha}\rangle-\langle
q_{\alpha},B\nabla\bar{h}(\theta_{0})\rangle^{\top}\Omega_{0}^{-1}\langle
B\nabla{q_{\alpha}}\bar{h}(\theta_{0}),q_{\alpha}\rangle$
$\displaystyle=\langle(C_{0}Bq_{\alpha},Bq_{\alpha}\rangle-\langle
Bq_{\alpha},\nabla\bar{h}(\theta_{0})\rangle^{\top}\Omega_{0}^{-1}\langle\nabla{q_{\alpha}}\bar{h}(\theta_{0}),Bq_{\alpha}\rangle$
$\displaystyle=\langle(WBq_{\alpha},Bq_{\alpha}\rangle\,,$

where the linear operator $W$ satisfies

$Wh=C_{0}h-\nabla\bar{h}(\theta_{0})^{\top}\Omega_{0}^{-1}\langle\nabla\bar{h}(\theta_{0}),h\rangle\,.$

Now, recall that $\Omega_{0}=\langle C_{0}^{-1/2}\nabla
h(\theta_{0}),C_{0}^{-1/2}\nabla h(\theta_{0})^{\top}\rangle$. Given this, for
any $h$, we can observe that

$\displaystyle(WC_{0}^{-1}W)(h)$
$\displaystyle=W(h-C_{0}^{-1}\nabla\bar{h}(\theta_{0})^{\top}\Omega_{0}^{-1}\langle\nabla\bar{h}(\theta_{0}),h\rangle)$
$\displaystyle=Wh-(C_{0}-Wh)+\nabla\bar{h}(\theta_{0})^{\top}\Omega_{0}^{-1}\langle\nabla\bar{h}(\theta_{0}),C_{0}^{-1}\nabla\bar{h}(\theta_{0})^{\top}\rangle\Omega_{0}^{-1}\langle\nabla\bar{h}(\theta_{0}),h\rangle$
$\displaystyle=2Wh-(C_{0}-\nabla\bar{h}(\theta_{0})^{\top}\Omega_{0}^{-1}\langle\nabla\bar{h}(\theta_{0}),h\rangle)$
$\displaystyle=Wh\,.$

Therefore, we have $W=WC_{0}^{-1}W$. Furthermore, it is trivial to wee that
$W$ is self-adjoint, since $C_{0}$ is self-adjoint and $\Omega_{0}^{-1}$ is
symmetric. Plugging these results into the above, we get

$\displaystyle\alpha^{\top}\Delta\alpha-\alpha^{\top}\Omega\Omega_{0}^{-1}\Omega\alpha$
$\displaystyle=\langle WC_{0}^{-1}WBq_{\alpha},Bq_{\alpha}\rangle$
$\displaystyle=\|C_{0}^{-1/2}WBq_{\alpha}\|^{2}$ $\displaystyle\geq 0\,,$

which concludes the proof.

∎

###### Proof of Lemma 3.

We will prove this by induction. First, in the case that $k=1$,
$\tilde{\theta}_{n}$ is given by a VMM estimate whose prior estimate is a
constant. However, a constant prior estimate satisfies Assumption 4 trivially,
and by assumption the remaining conditions of Theorem 2 hold, so therefore
$\tilde{\theta}_{n}=\theta_{0}+O_{p}(n^{-1/2})$. Given this and Assumption 6,
by the mean value theorem we have

$\rho_{i}(x;\tilde{\theta}_{n})=\rho_{i}(x;\theta_{0})+\nabla\rho_{i}(x;\theta^{\prime}_{n})^{\top}(\tilde{\theta}_{n}-\theta_{0})\,,$

for each $i\in[m]$ and $x\in\mathcal{X}$, where $\theta^{\prime}_{n}$ lies on
the segment between $\theta_{0}$ and $\tilde{\theta}_{n}$ for each $i$ and
$x$. Now, by Assumption 6 $\nabla\rho_{i}(x;\theta)$ is uniformly bounded over
$x$ and $\theta$, so therefore since $\tilde{\theta}_{n}=O_{p}(n^{-1/2})$ it
follows that

$\sup_{x\in\mathcal{X},i\in[m]}\lvert\rho_{i}(x;\tilde{\theta}_{n})-\rho_{i}(x;\theta_{0})\rvert=O_{p}(n^{-1/2})\,,$

so therefore we have established the base case.

For the inductive case, assume that the desired property holds for some $k>1$.
Then, by the inductive hypothesis, $\tilde{\theta}_{n}$ is given by a VMM
estimate whose prior estimate whose prior estimate satisfies Assumption 4 with
$p=1/2$. Furthermore, by assumption the remaining conditions of Theorem 2
hold, so therefore we have $\tilde{\theta}_{n}=\theta_{0}+O_{p}(n^{-1/2})$.
Therefore by the same mean value theorem-based argument as above, again we
have
$\sup_{x\in\mathcal{X},i\in[m]}\lvert\rho_{i}(x;\tilde{\theta}_{n})-\rho_{i}(x;\theta_{0})\rvert=O_{p}(n^{-1/2})$,
which establishes the inductive case.

∎

###### Proof of Lemma 4.

Recall that

$J_{n}(\theta)=\sup_{f\in\mathcal{F}}\mathbb{E}_{n}[f(Z)^{\top}\rho(X;\theta)]-\frac{1}{4}\mathbb{E}_{n}[(f(Z)^{\top}\rho(X;\tilde{\theta}_{n}))^{2}]-\frac{\alpha_{n}}{4}\|f\|^{2}\,.$

Now, by the representer theorem, there must exist an optimal solution to the
supremum over $f\in\mathcal{F}$ that takes the form
$f_{k}(z)=\sum_{i=1}^{n}\beta_{i,k}K_{k}(Z_{i},z)$ for each $k\in[m]$ and
$z\in\mathcal{Z}$, for some vector $\beta\in\mathbb{R}^{n\times m}$. Plugging
in this to the above, we get

$\displaystyle J_{n}(\theta)$ $\displaystyle=\sup_{\beta\in\mathbb{R}^{n\times
m}}\sum_{i,k}\beta_{i,k}\frac{1}{n}\sum_{j=1}^{n}K_{k}(Z_{i},Z_{j})\rho_{k}(X_{j};\theta)$
$\displaystyle\qquad-\frac{1}{4}\sum_{i,k,i^{\prime},k^{\prime}}\beta_{i,k}\beta_{i^{\prime},k^{\prime}}\frac{1}{n}\sum_{j=1}^{n}K_{k}(Z_{i},Z_{j})\rho_{k}(X_{j};\tilde{\theta}_{n})K_{k^{\prime}}(Z_{i^{\prime}},Z_{j})\rho_{k^{\prime}}(X_{j};\tilde{\theta}_{n})$
$\displaystyle\qquad-\frac{\alpha_{n}}{4}\sum_{i,i^{\prime},k}\beta_{i,k}\beta_{i^{\prime},k}K_{k}(Z_{i},Z_{i^{\prime}})$
$\displaystyle=\sup_{\beta\in\mathbb{R}^{n\times
m}}\frac{1}{n}\beta^{\top}L\rho(\theta)-\frac{1}{4}\beta^{\top}(Q(\tilde{\theta}_{n})+\alpha_{n}L)\beta\,.$

Next, taking derivatives of the above with respect to $\beta$, and noting that
$Q(\tilde{\theta}_{n})+\alpha_{n}L$ is clearly positive semi-definite, the
above above supremum is clearly attained when

$\displaystyle\frac{1}{n}L\rho(\theta)-\frac{1}{2}(Q(\tilde{\theta}_{n})+\alpha_{n}L)\beta=0$
$\displaystyle\qquad\iff\beta=\frac{2}{n}(Q(\tilde{\theta}_{n})+\alpha_{n}L)^{-1}L\rho(\theta)\,.$

Plugging this optimal $\beta$ into the above suprumum, and noting that $L$ is
symmetric, gives us

$J_{n}(\theta)=\frac{1}{n^{2}}\rho(\theta)^{\top}L(Q(\tilde{\theta}_{n})+\alpha_{n}L)^{-1}L\rho(\theta)\,,$

as required.

∎

###### Proof of Lemma 5.

First, applying Lemma 4 to this particular problem, we get

$\displaystyle J_{n}(\theta)$
$\displaystyle=(W-\theta(T))^{\top}M(W-\theta(T))$
$\displaystyle=\theta(T)^{\top}M\theta(T)-2W^{\top}M\theta(T)+c\,,$

where $\theta(T)=(\theta(T_{1}),\ldots,\theta(T_{n}))$, and $c$ is a constant
that doesn’t depend on $\theta$. Now, by Lagrangian duality, there must exist
some value $\lambda_{n}\geq 0$ which implicitly depends on $r$, $K_{f}$,
$K_{g}$, $\tilde{\theta}_{n}$, and the observed data, such that minimizing
$J_{n}(\theta)$ over $\Theta$ is equivalent to minimizing
$J_{n}(\theta)+\lambda_{n}\|\theta\|^{2}$ over the entire RKHS given by
$K_{g}$. Now, by the representer theorem, there must exist some optimal
solution to this unconstrained problem that takes the form
$\theta(t)=\sum_{i=1}^{n}\beta_{i}K_{g}(t,T_{i})$ for every treatment value
$t$. Plugging this in to the regularized and unconstrained version of our
problem, and noting that $\theta(T)=L_{g}\beta$, we have that the optimal
vector $\beta^{*}$ satisfies

$\displaystyle\beta^{*}$
$\displaystyle=\operatorname*{arg\,min}_{\beta\in\mathbb{R}^{n}}(L_{g}\beta)^{\top}M(L_{g}\beta)-2W^{\top}ML_{g}\beta+c+\lambda_{n}\beta^{\top}L_{g}\beta$
$\displaystyle=\operatorname*{arg\,min}_{\beta\in\mathbb{R}^{n}}\beta^{\top}(L_{g}ML_{g}+\lambda_{n}L_{g})\beta-2(L_{g}MW)^{\top}\beta\,.$

Taking derivatives with respect to $\beta$ in the above, and noting that
$L_{g}ML_{g}+\lambda_{n}L_{g}$ is clearly positive semi-definite, it follows
that this supremum is attained when

$\displaystyle 2(L_{g}ML_{g}+\lambda_{n}L_{g})\beta^{*}-2L_{g}MW=0$
$\displaystyle\qquad\iff\beta^{*}=(L_{g}ML_{g}+\lambda_{n}L_{g})^{-1}L_{g}MW\,,$

which exactly matches our desired result.

∎

###### Proof of Lemma 6.

For any given $\theta\in\Theta$, let $f_{n}^{*}(\theta)$ be the element of
$\mathcal{F}$ maximizing the adversarial game objective $U_{n}(\theta,f)$, and
for each $i\in[m]$ define
$F_{i,n}=\sup_{\theta\in\Theta}\|f^{*}_{n}(\theta)_{i}\|_{\mathcal{F}}$. Now,
by Lemma 10 and the definitions of $\bar{h}_{n}(\theta)$ and $B_{n}$ we
clearly have

$\displaystyle\langle\bar{h}_{n}(\theta),h^{\prime}\rangle-\frac{1}{4}\langle
B_{n}^{-2}h^{\prime},h^{\prime}\rangle$
$\displaystyle\qquad=\mathbb{E}_{n}[\varphi(f)(Z)^{\top}\rho(X;\theta)]-\frac{1}{4}\mathbb{E}_{n}[(\varphi(h^{\prime})(Z)^{\top}\rho(X;\tilde{\theta}_{n}))^{2}]-\frac{\alpha_{n}}{4}\|\varphi(h^{\prime})\|_{\mathcal{F}}\,,$

so it easily follows that $f_{n}^{*}(\theta)=2B_{n}^{2}\bar{h}_{n}(\theta)$.
Now, by Assumption 2 the functions $\bar{h}_{n}(\theta)$ and $\bar{h}(\theta)$
satisfy the conditions of Lemma 19, so by Lemmas 19 and 21 we have that
$F_{i,n}=O_{p}(1)$ for each $i\in[m]$.

Next, for any integer $k>d_{z}/2$, where $d_{z}$ is the dimensionality of
$\mathcal{Z}$, let $W_{k}$ denote the Sobolev space of functions of the form
$\mathcal{Z}\mapsto\mathbb{R}$ using derivatives up to order $k$, and let
$\|\cdot\|_{W_{k}}$ denote the corresponding Sobolev norm. Then, as argued in
the proof of Cucker and Smale [2002, Theorem D], there exists some constant
$c_{k}$, which may depend on $k$, such that $\|f\|_{W_{k}}\leq
c_{k}\|f\|_{\mathcal{F}}$ for any function $f$. That is,
$\sup_{\theta\in\Theta}\|f_{n}^{*}(\theta)_{i}\|_{W_{k}}\leq c_{k}F_{i,n}$.

Now, by Yarotsky [2017, Theorem 1], there exists a constant $c^{\prime}_{k}$
which depends on $k$, such that for any $\epsilon>0$ and any $f$ satisfying
$\|f\|_{W_{k}}\leq 1$, there exists some ReLU network $f^{\prime}$ with depth
at most $c_{k}(\log(1/\epsilon)+1)$, and at most
$c_{k}\epsilon^{-d_{z}/k}(\log(1/\epsilon)+1)$ computation units in total,
such that $\|f-f^{\prime}\|_{\infty}\leq\epsilon$. It immediately follows that
this function $f^{\prime}$ is guaranteed to exist within a fully connected
ReLU architecture with depth $c_{k}(\log(1/\epsilon)+1)$ and constant width
$c_{k}\epsilon^{-d_{z}/k}(\log(1/\epsilon)+1)$. Putting this result together
with our previous results above, and noting that an $m$-dimensional function
defined by $m$ separate networks with width $W$ and depth $D$ can be defined
within a fixed architecture of width $mW$ and depth $D$, it immediately
follows that as long as $D_{n}=\omega(\log(1/\epsilon)+1)$ and
$W_{n}=\omega(\epsilon^{-d_{z}/k}(\log(1/\epsilon)+1))$, we can ensure that
$\sup_{\theta\in\Theta}\inf_{f\in\mathcal{F}_{n}}\|f^{*}_{n}(\theta)-f\|_{\infty}\leq
c_{k}\max_{i\in[m]}F_{i,n}\epsilon$ for some sufficiently large $n$ that
doesn’t depend on $\epsilon$. Therefore, noting that the choice of $k>d_{z}/2$
was arbitrary, if we choose $D_{n}$ and $W_{n}$ as in Assumption 9, we have
$\sup_{\theta\in\Theta}\inf_{f\in\mathcal{F}_{n}}\|f^{*}_{n}(\theta)-f\|_{\infty}=O_{p}(r_{n})$.

Finally, for each $\theta\in\Theta$, let
$f^{\prime}_{n}(\theta)=\operatorname*{arg\,min}_{f\in\mathcal{F}_{n}}\|f_{n}^{*}(\theta)-f\|_{\infty}$,
and let $U_{n}$ be the game objective for $\hat{\theta}_{n}^{\text{NK-VMM}}$.
Then we have

$\displaystyle\sup_{\theta\in\Theta}\lvert
J_{n}(\theta)-J_{n}^{*}(\theta)\rvert$
$\displaystyle\leq\sup_{\theta\in\Theta}\lvert
U_{n}(\theta,f_{n}^{*}(\theta))-U_{n}(\theta,f_{n}^{\prime}(\theta))\rvert$
$\displaystyle\leq\sup_{\theta\in\Theta}\lvert\mathbb{E}_{n}[(f_{n}^{*}(\theta)-f_{n}^{\prime}(\theta))(Z)^{\top}\rho(X;\theta)]\rvert$
$\displaystyle\qquad+\sup_{\theta\in\Theta}\mathbb{E}_{n}[(f_{n}^{*}(\theta)-f_{n}^{\prime}(\theta))(Z)^{\top}\rho(X;\tilde{\theta}_{n})\rho(X;\tilde{\theta}_{n})^{\top}f_{n}^{*}(\theta)]$
$\displaystyle\qquad+\sup_{\theta\in\Theta}\mathbb{E}_{n}[f_{n}^{\prime}(\theta)(Z)^{\top}\rho(X;\tilde{\theta}_{n})\rho(X;\tilde{\theta}_{n})^{\top}(f_{n}^{*}(\theta)-f_{n}^{\prime}(\theta))(Z)]$
$\displaystyle\qquad+\sup_{\theta\in\Theta}\sum_{k=1}^{m}\lvert((f_{n}^{*})_{k}-(f_{n}^{\prime})_{k})^{\top}K_{k}^{-1}(f_{n}^{*})_{k}\rvert$
$\displaystyle\qquad+\sup_{\theta\in\Theta}\sum_{k=1}^{m}\lvert(f_{n}^{\prime})_{k}^{\top}K_{k}^{-1}((f_{n}^{*})_{k}-(f_{n}^{\prime})_{k})\rvert\,.$

Now, by Assumption 9 we have $r_{n}=o(n^{-1-q}k_{n}^{-1})$, so therefore by
our above result we have
$\sup_{\theta\in\Theta}\|f_{n}^{*}(\theta)-f_{n}^{\prime}(\theta)\|_{\infty}=o_{p}(n^{-1-q}k_{n}^{-1})$,
and
$\sup_{\theta\in\Theta}\|f_{n}^{*}(\theta)-f_{n}^{\prime}(\theta)\|_{2}=o_{p}(n^{-q-1/2}k_{n}^{-1})$.
In addition, $\sup_{\theta\in\Theta}\|f_{n}^{*}(\theta)\|_{\infty}$ and
$\sup_{\theta\in\Theta}\|f_{n}^{\prime}(\theta)\|_{\infty}$ are stochastically
bounded given the above and applying Lemma 13, and therefore
$\sup_{\theta\in\Theta}\|f_{n}^{*}(\theta)\|_{2}$ and
$\sup_{\theta\in\Theta}\|f_{n}^{\prime}(\theta)\|_{2}$ are both in
$O_{p}(\sqrt{n})$, and also by Assumption 8 $K_{k}^{-1}$ has $O_{p}(k_{n})$
operator norm for each $k\in[m]$. Therefore, plugging these results into the
above bound, we get $\sup_{\theta\in\Theta}\lvert
J_{n}(\theta)-J_{n}^{*}(\theta)\rvert=O_{p}(n^{-q})$, as required.

∎


# MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion Recognition

Peihao Xiang  Department of Electrical and Computer Engineering
Florida International University
Miami, USA
<EMAIL_ADDRESS>Chaohao Lin  Department of Electrical and Computer
Engineering
Florida International University
Miami, USA
<EMAIL_ADDRESS>Kaida Wu  Department of Electrical and Computer Engineering
Florida International University
Miami, USA
<EMAIL_ADDRESS>Ou Bai  Department of Electrical and Computer Engineering
Florida International University
Miami, USA
<EMAIL_ADDRESS>
###### Abstract

This paper presents a novel approach to processing multimodal data for dynamic
emotion recognition, named as the Multimodal Masked Autoencoder for Dynamic
Emotion Recognition (MultiMAE-DER). The MultiMAE-DER leverages the closely
correlated representation information within spatio-temporal sequences across
visual and audio modalities. By utilizing a pre-trained masked autoencoder
model, the MultiMAE-DER is accomplished through simple, straightforward fine-
tuning. The performance of the MultiMAE-DER is enhanced by optimizing six
fusion strategies for multimodal input sequences. These strategies address
dynamic feature correlations within cross-domain data across spatial,
temporal, and spatio-temporal sequences. In comparison to state-of-the-art
multimodal supervised learning models for dynamic emotion recognition,
MultiMAE-DER enhances the weighted average recall (WAR) by 4.41% on the
RAVDESS dataset and by 2.06% on the CREMA-D. Furthermore, when compared with
the state-of-the-art model of multimodal self-supervised learning, MultiMAE-
DER achieves a 1.86% higher WAR on the IEMOCAP dataset.

###### Index Terms:

Dynamic Emotion Recognition, Multimodal Model, Self-Supervised Learning, Video
Masked Autoencoder, Vision Transformer

##  I Introduction

With the development of deep learning technology, convolutional neural
networks (CNN) have shown excellent results in static emotion recognition,
especially in facial expression recognition (FER) tasks, as seen in models
like RTCNN [1], DeepEmotion [2], and PAtt-Lite [3]. Dynamic emotion
recognition (DER) is one of the important fields of affective computing.
Psychologists and neuroscientists have been exploring this field for a long
time, while constructing a series of quantitative rules for recognizing
emotional characteristics.

Figure 1: General Multimodal Model vs. MultiMAE-DER. The uniqueness of our
approach lies in the capability to extract features from cross-domain data
using only a single encoder, eliminating the need for targeted feature
extraction for different modalities.

In DER, a new architecture has emerged by combining CNN with recurrent neural
networks (RNN). For example, ResNet-18+LSTM [4] and C3D+LSTM [5] have been
developed to construct dynamic emotion recognition models. More importantly,
with the advent of Transformers [6], there has been further exploration into
the global context correlation. This structure is well-suited for capturing
the spatial and temporal context features in dynamic emotion. Examples of such
models include Former-DFER [7] and STT-DFER [8].

However, CNN, RNN, or Transformers [6], requires a large amount of labeled
data for supervised learning to build high-performance models. This issue was
only alleviated with the advent of self-supervised learning, especially when
applied to natural language processing (NLP), where the BERT [9] model
demonstrated performance beyond supervised learning models.

In the field of computer vision, the introduction of Masked Autoencoder (MAE)
strongly indicated that self-supervised learning would shine in the field of
vision such as ImageMAE [10], VideoMAE [11], and AudioMAE [12]. On the other
hand, while single-modal (visual or audio) inputs have shown good results in
dynamic emotion recognition, as shown in models like HuBERT [13] and MAE-DFER
[14], their performance has not yet reached the level of real human emotion
recognition. Therefore, multimodal (Visual-Audio) input has become a desirable
development trend, as demonstrated by models like AV-LSTM [15] and MSAF [16].

The aim of this study is to explore a new framework for dynamic emotion
recognition model. The proposed approach is a straightforward inheritance and
extension of VideoMAE [11], extending the originally single-modal visual input
into multimodal input encompassing both visual and audio elements.
Simultaneously, the self-supervised learning pre-trained Video Masked
Autoencoder model is used to process visual-audio sequence strategies. This
extended model is named MultiMAE-DER, standing for Multimodal Masked
Autoencoder for Dynamic Emotion Recognition. The MultiMAE-DER is optimized by
investigating the performance of six visual-audio sequence strategies.

##  II RELATED WORK

###  II-A Dynamic Emotion Recognition

Dynamic emotion recognition is a significant challenge in the field of
affective computing that requires the utilization of multimodal feature
extraction, context semantic analysis, and causal reasoning techniques. In
recent years, researchers have addressed the feature extraction task from
multimodal using transfer learning by combining pre-trained models for video
facial expression feature extraction and audio tone emotion feature
extraction, such as MSAF [16] and CFN-SR [17]. However, the above-mentioned
model architecture fails to establish cross-domain associative features
between visual and audio data, leading to the loss of highly correlated
spatio-temporal feature dimensions in the emotion representation information.
Therefore, we propose a MultiMAE-DER framework, a unified feature extraction
pre-trained model that will be employed for visual and audio multimodal data
to find the associated features in visual-audio spatio-temporal emotion
representation information.

###  II-B Masked Autoencoder

Masked Autoencoder is a variant of denoising autoencoder that consists of an
encoder and a decoder. In ImageMAE [10], that is designed as an asymmetric
structure to reconstruct normalized pixels. VideoMAE [11] follows the design
approach of ImageMAE [10] but extends the 2D image input to 3D video input and
incorporates a joint spatio-temporal self-attention mechanism to replace the
vanilla Vision Transformer [18] spatial self-attention mechanism.

For downstream tasks, the decoder of the masked autoencoder is discarded, and
only the pre-trained encoder is used for supervised learning and fine-tuning
on the downstream task dataset. As a leader in self-supervised learning, our
proposal will continue to inherit this characteristic and utilize the pre-
trained encoder model to fine-tune multimodal dynamic emotion data, aiming for
improved emotion recognition performance.

###  II-C Vision Transformer

Vision Transformer (ViT)[18] is a variant of the Transformer [6] applied in
the field of computer vision. The principle is to treat a pixel block (patch)
as an individual “word” to emulate the token sequence input of Transformer
[6], allowing for the serialization of image processing. Furthermore, ViT [18]
serves as the backbone network for VideoMAE [11], which is used to associate
the correlation of spatio-temporal context in video data.

For MultiMAE-DER, ViT [18] will continue to be used as the backbone network
for the encoder. Simultaneously, MultiMAE-DER will inherit the joint spatio-
temporal self-attention mechanism from VideoMAE [11] to replace the spatial
self-attention mechanism in vanilla Vision Transformer [18]. This adaptation
aims to analyze the contextual emotion representation information in visual-
audio data.

Figure 2: Time sequence of video frames and corresponding audio spectrograms.
Figure 3: Six Multimodal Sequence Fusion Strategies.

##  III METHOD

###  III-A Revisiting VideoMAE

To illustrate that this work is an inheritance and extension of VideoMAE [11],
as depicted in Fig. 1, the encoder will directly call the pre-trained model of
VideoMAE [11] encoder as the backbone network. This approach serves to
demonstrate effectiveness as a video representation learner while also
reducing significant training time. The pre-trained encoder model from
VideoMAE [11] is used, which requires consistency with the input size of
$16\times 224\times 224$, where 16 is the number of frames, and the image size
is $224\times 224$. Additionally, the patch size remains $2\times 16\times
16$, resulting in 1568 patches as the input size for the token sequence. The
VideoMAE [11] encoder uses ViT-L [18] as the backbone network with a dimension
of 1024. However, this work primarily analyzes multimodal inputs. Therefore,
several different sequence fusion methods will be optimized while maintaining
the input size and patch size.

###  III-B Multimodal Sequence Strategy

Firstly, 16 frames of video $\textbf{V}\in{R}^{16\times 224\times 224}$ are
uniformly downsampled from the single clip. At the same time, to align with
each video frame, the audio signal is uniformly downsampled to generate 16
spectrograms $\textbf{A}\in{R}^{16\times 224\times 224}$. As shown in Fig. 2,
in the same time sequence, each sampled time point of the video frame
corresponds to the audio signal contained before and after each time point.
This ensures the maximum alignment of partial emotion features between the
audio signal and facial expression in the video.

On the other hand, in order to verify that extracting spatiotemporally highly
correlated dynamic representation information from unified video and audio
signals can help improve the accuracy of emotion recognition. We explore the
following hypotheses from the sequence fusion strategy of spatiotemporal
aspects of audio and video signals:

  * •

From the spatial perspective, video signals and audio signals are spliced or
superimposed to explore the dynamic representation information of the video
and audio signals in spatial integrity under the certain time sequence.

  * •

From the temporal perspective, the video signal and audio signal are sequence
transform to explore the dynamic representation information of the video and
audio signal in time sequence under the condition of spatial integrity.

Therefore, as shown in Fig. 3, this work will propose six different multimodal
sequence fusion strategies to optimize MultiMAE-DER performance based on the
above hypotheses. For strategy 1 (CFAS), 16 frames of video
$\textbf{V}\in{R}^{16\times 112\times 224}$ and corresponding 16 audio
spectrograms $\textbf{A}\in{R}^{16\times 112\times 224}$ will be directly
spliced to explore whether there is a certain correlation in the
spatiotemporal sequence for this multimodal input $\textbf{X}\in{R}^{16\times
224\times 224}$. For strategy 2 (SFAS), 16 frames of video
$\textbf{V}\in{R}^{16\times 224\times 224}$ and corresponding 16 audio
spectrograms $\textbf{A}\in{R}^{16\times 224\times 224}$ will be normalized,
and then directly added to verify the feasibility of spatially fusing audio-
visual signals. For strategy 3 (FFLS), considering input size constraints, the
single clip are uniformly downsampled to 8 frames of video
$\textbf{V}\in{R}^{8\times 224\times 224}$ and corresponding 8 audio
spectrograms $\textbf{A}\in{R}^{8\times 224\times 224}$. They are sorted based
on facial frames first and then audio spectrograms to consider the impact of
solely the temporal sequence in the visual-audio sorting of multimodal input
$\textbf{X}\in{R}^{16\times 224\times 224}$ on emotion recognition results.
The mathematical expressions for these three multimodal sequence fusion
strategies are as follows:

$\textbf{X}_{i}=\text{Concat}(\textbf{V}_{i},\textbf{A}_{i})$ (1)
$\textbf{X}_{i}=\text{Add}(\text{Norm}(\textbf{V}_{i}),\text{Norm}(\textbf{A}_{i}))$
(2)
$\textbf{X}=\text{Seq}(\textbf{V}_{1},\textbf{V}_{2},\ldots,\textbf{V}_{8},\textbf{A}_{1},\textbf{A}_{2},\ldots,\textbf{A}_{8})$
(3)

For strategy 4 (FSLF), following the method of Strategy 3, the order of video
frames $\textbf{V}\in{R}^{8\times 224\times 224}$ and audio spectrograms
$\textbf{A}\in{R}^{8\times 224\times 224}$ are swapped to consider the impact
of different orders in the temporal sequence of visual-audio input
$\textbf{X}\in{R}^{16\times 224\times 224}$ on emotion recognition results.
For strategy 5 (OFOS), following the order of one video frame
$\textbf{V}\in{R}^{8\times 224\times 224}$ followed by one audio spectrogram
$\textbf{A}\in{R}^{8\times 224\times 224}$, to explore the impact of different
permutations of the time sequence on emotion recognition results.

Finally, for strategy 6 (RFAS), randomly sorting 8 frames of video
$\textbf{V}\in{R}^{8\times 224\times 224}$ and corresponding 8 spectrograms
$\textbf{A}\in{R}^{8\times 224\times 224}$ to verify the influence of
unordered temporal sequence on visual-audio multimodal data input
$\textbf{X}\in{R}^{16\times 224\times 224}$ on emotion recognition results.
The mathematical expressions for these three multimodal sequence fusion
strategies are as follows:

$\textbf{X}=\text{Seq}(\textbf{A}_{1},\textbf{A}_{2},\ldots,\textbf{A}_{8},\textbf{V}_{1},\textbf{V}_{2},\ldots,\textbf{V}_{8})$
(4)
$\textbf{X}=\text{Seq}(\textbf{V}_{1},\textbf{A}_{1},\textbf{V}_{2},\textbf{A}_{2},\ldots,\textbf{V}_{8},\textbf{A}_{8})$
(5)
$\textbf{X}=\text{Rand}(\textbf{V}_{1},\textbf{V}_{2},\ldots,\textbf{V}_{8},\textbf{A}_{1},\textbf{A}_{2},\ldots,\textbf{A}_{8})$
(6)

###  III-C MultiMAE-DER: Model Structure

As shown in Fig. 1, uniform downsampling is applied to both video and audio in
the single clip to align them and meet input size requirements. Subsequently,
different sequence fusion strategies from Section III.B are employed as
multimodal input $\textbf{X}\in{R}^{16\times 224\times 224}$ to verify the
assumptions of this work. Like VideoMAE [11], before entering the encoder, the
input data is patched to construct token sequence
$\textbf{S}\in{R}^{1568\times 512}$. In this model, 3D convolutional kernels
are used as embedding layers to perform non-overlapping patching, where the
kernel size is $2\times 16\times 16$. Linear projection (LP) is applied to
each cube patch to flatten the input $\hat{\textbf{X}}\in{R}^{1568\times
1024}$ into the encoder. After processing through the encoder, the output
feature matrix $\textbf{F}\in{R}^{1568\times 1024}$ with the same input size
is obtained. For a given downstream task (dynamic emotion recognition in this
case), a single linear layer (MLP) serves as a specific-task head to process
the feature matrix $\textbf{F}\in{R}^{1568\times 1024}$, yielding the final
emotion result $\textbf{Y}\in{R}^{1\times 7}$ for this clip. The mathematical
expressions for the above process are as follows:

$\textbf{S}=\text{Patching}(\textbf{X})$ (7)
$\hat{\textbf{X}}=\text{LP}(\textbf{S})$ (8)
$\textbf{F}=\text{Encoder}(\hat{\textbf{X}})$ (9)
$\textbf{Y}=\text{MLP}({\textbf{F}})$ (10)

##  IV EXPERIMENTS

###  IV-A Datasets

####  IV-A1 RAVDESS [19]

The Ryerson Audio-Visual Database of Emotional Speech and Song is a dataset
consisting of emotional performances conducted by 24 North American
professional actors in a laboratory environment with North American accent.
The emotions depicted in this dataset include anger, disgust, fear, happiness,
neutral, sadness, and surprise. Each emotion has two intensity variations:
normal and strong. In this work, we utilize only the visual-audio speech
dataset, comprising 1440 video files. Among them, there are 288 videos for the
neutral emotion, and the remaining emotions each have 192 videos. Our model
evaluation is conducted using a 6-fold cross-validation on subjects-
independent of the emotion.

####  IV-A2 CREMA-D[20]

Crowd-sourced Emotional Multimodal Actor Dataset comprises emotional
performances by 91 professional actors from around the world, representing
different countries, and ethnicities. These actors conducted emotional
performances in a laboratory environment with diverse accents. Each actor
presented six different emotions: anger, disgust, fear, happiness, neutral,
and sadness. Additionally, they expressed each emotion at low, medium, high,
and unspecified intensity levels to dialogue. The dataset includes a total of
7,442 video clips, with 1,087 clips for the neutral emotion and 1,271 clips
for each of the remaining emotions. Model evaluation will be conducted using
5-fold cross-validation on subjects-independent of the emotion.

Figure 4: MultiMAE-DER Program Flow-Chart.

####  IV-A3 IEMOCAP[21]

The Interactive Emotional Dyadic Motion Capture Database is a multimodal and
multi-speaker visual-audio dataset featuring improvised emotional binary scene
creations by ten non-professional actors. The dataset includes approximately
12 hours of video data, capturing nine different emotional expressions: anger,
happiness, excitement, sadness, discouragement, fear, surprise, other, and
neutral states. Following the emotion categorization approach of AV-Superb
[22], we conduct a four-emotion classification task (anger, happiness,
sadness, and neutral). To increase the sample size, the data for the excited
state are added to the data for the happy state. The fine-tuned model will be
evaluated on this dataset using 5-fold cross-validation.

###  IV-B Implementation Details

####  IV-B1 Downsampling

As discussed in Section III.B regarding the strategies for multimodal sequence
fusion, downsampling is a crucial step in this proposal. Depending on the
different sequence fusion strategies, we have varying uniform downsampling
steps. Taking the RAVDESS [19] dataset as an example, for strategies 1 and 2,
where facial videos and audio spectrograms have 16 frames per clip, the
downsampling step size is set to 6. For strategies 3, 4, 5, and 6, where
facial videos and audio spectrograms have 8 frames per clip, the downsampling
step size is set to 12. The data preprocessing for the above process is
illustrated in Fig. 4, depicting video preprocessing and audio preprocessing.
For video preprocessing, the cutoff time of a single clip is computed based on
the required frame number and downsampling step size, followed by the
downsampling of the clip. Simultaneously, for audio preprocessing, the audio
signal of the clip is extracted. The time point $T$ corresponding to the
facial frame is determined based on the frame number. Then, the audio signal
is cut at the time point $T-1$ and $T+1$ to perform Short-Time Fourier
Transform (STFT) and Mel-Frequency Cepstral Coefficients (MFCC) to obtain the
audio spectrogram corresponding to the facial frame at time point $T$ .

####  IV-B2 Pre-training

After preprocessing, the obtained facial frames and audio spectrograms
implement sequence fusion, which includes methods such as splice, addition,
and combine. For the pre-trained encoder model, to eliminate domain
differences, the pre-trained encoder model used in our proposal is derived
from MAE-DFER [14]. This encoder is built exclusively upon VideoMAE [11] and
has undergone self-supervised training on the extensive dataset of facial
videos from YouTube, aiming to mitigate domain differences.

####  IV-B3 Fine-tuning

As described in Section III.C, to minimize the interference of inductive bias,
our approach utilizes only single-layer neural network as a specific-task head
for fine-tuning on the downstream task. The fine-tuning employs the AdamW [23]
optimizer with the base learning rate of 1e-2, weight decay of 0.05, sparse
categorical cross-entropy as the loss function, and sparse categorical
accuracy as the metric function. The evaluation metrics used by the model
include unweighted average recall (UAR) and weighted average recall (WAR).

###  IV-C Results

In this section, we first compared MultiMAE-DER with other previous state-of-
the-art multimodal models on the RAVDESS [19]. As shown in Table 1, in the
multimodal model domain, the results exhibit significant variations based on
different learning methods, whether supervised learning or self-supervised
learning. The self-supervised multimodal model VQ-MAE-AV [30] outperforms the
supervised multimodal model AVT [29] by 4% (83.20% vs. 79.20%) in terms of
WAR. Furthermore, our MultiMAE-DER model, employing different multimodal
sequence fusion strategies, generally surpasses the supervised multimodal
model. Particularly, when using strategy 4, MultiMAE-DER-FSLF outperforms the
supervised model by 4.41% (83.61% vs. 79.20%) in terms of WAR. Among the self-
supervised multimodal models, our optimal strategy (i.e., strategy 4 FSLF) of
MultiMAE-DER also outperforms the multimodal model VQ-MAE-AV [30] by 0.41%
(83.61% vs. 83.20%) in terms of WAR.

TABLE I: Results on RAVDESS (7-class). SSL: self-supervised learning method or not. UAR: unweighted average recall. WAR: weighted average recall. Underlined: the best supervised result. Bold: the best result. Method | SSL | Modality | UAR | WAR
---|---|---|---|---
AV-LSTM[15] | $\chi$ | V+A | $-$ | 65.80
AV-Gating[15] | $\chi$ | V+A | $-$ | 67.70
MCBP[24] | $\chi$ | V+A | $-$ | 71.32
MMTM[25] | $\chi$ | V+A | $-$ | 73.12
ERANNs[26] | $\chi$ | V+A | $-$ | 74.80
MSAF[16] | $\chi$ | V+A | $-$ | 74.86
SFN-SR[17] | $\chi$ | V+A | $-$ | 75.76
MATER [27] | $\chi$ | V+A | $-$ | 76.30
MulT [28] | $\chi$ | V+A | $-$ | 76.60
AVT [29] | $\chi$ | V+A | $-$ | 79.20
VQ-MAE-AV [30] | $\checkmark$ | V+A | $-$ | 83.20
MultiMAE-DER | $\checkmark$ | V | $-$ | 74.13
MultiMAE-DER | $\checkmark$ | A | $-$ | 80.55
MultiMAE-DER-RFAS | $\checkmark$ | V+A | $75.97$ | 75.44
MultiMAE-DER-SFAS | $\checkmark$ | V+A | $75.79$ | 76.94
MultiMAE-DER-OFOS | $\checkmark$ | V+A | $77.78$ | 78.61
MultiMAE-DER-CFAS | $\checkmark$ | V+A | $80.65$ | 81.39
MultiMAE-DER-FFLS | $\checkmark$ | V+A | $82.27$ | 83.56
MultiMAE-DER-FSLF | $\checkmark$ | V+A | 83.23 | 83.61

The visual-audio results on the CREMA-D [20] are presented in Table 2. We
observe that the optimal strategy of MultiMAE-DER achieves the WAR result
surpassing the VQ-MAE-AV [30] multimodal self-supervised learning model by
0.96% (79.36% vs. 78.40%). Compared to supervised learning approaches, the
optimal strategy of MultiMAE-DER outperforms the multimodal model RAVER [34]
by 2.06% (79.36% vs. 77.30%) in WAR. In addition, most of the results of other
strategies in MultiMAE-DER outperform state-of-the-art multimodal supervised
learning models. This indicates the effectiveness of multimodal sequence
fusion strategies, which can aid in identifying spatio-temporal correlations
across cross-domain data to some extent.

TABLE II: Results on CREMA-D (6-class). SSL: self-supervised learning method or not. UAR: unweighted average recall. WAR: weighted average recall. Underlined: the best supervised result. Bold: the best result. Method | SSL | Modality | UAR | WAR
---|---|---|---|---
EF-GRU [31] | $\chi$ | V+A | $-$ | 57.06
LF-GRU [31] | $\chi$ | V+A | $-$ | 58.53
TFN [32] | $\chi$ | V+A | $-$ | 63.09
MATER [27] | $\chi$ | V+A | $-$ | 67.20
AuxFormer[33] | $\chi$ | V+A | $-$ | 71.70
AV-LSTM[15] | $\chi$ | V+A | $-$ | 72.90
AV-Gating[15] | $\chi$ | V+A | $-$ | 74.00
RAVER [34] | $\chi$ | V+A | $-$ | 77.30
VQ-MAE-AV [30] | $\checkmark$ | V+A | $-$ | 78.40
MultiMAE-DER | $\checkmark$ | V | $-$ | 77.83
MultiMAE-DER | $\checkmark$ | A | $-$ | 78.45
MultiMAE-DER-RFAS | $\checkmark$ | V+A | $74.62$ | 74.90
MultiMAE-DER-SFAS | $\checkmark$ | V+A | $75.73$ | 75.48
MultiMAE-DER-OFOS | $\checkmark$ | V+A | $76.88$ | 76.54
MultiMAE-DER-CFAS | $\checkmark$ | V+A | $78.24$ | 78.16
MultiMAE-DER-FFLS | $\checkmark$ | V+A | $78.59$ | 78.83
MultiMAE-DER-FSLF | $\checkmark$ | V+A | 79.12 | 79.36

In Table 3, we compare state-of-the-art self-supervised multimodal models on
the IEMOCAP [21]. The results are derived from the evaluation test conducted
by AV-Superb [22]. The results showed that in the field of self-supervised
learning, our MultiMAE-DER optimal strategy model (FSLF) outperforms the
multimodal model AVBERT [37] by 1.86% (63.73% vs. 61.87%) in WAR. Upon
observing the results of multiple strategies in our model and comparing them
with the results of these three self-supervised multimodal models, we notice
that MultiMAE-DER model performs slightly better in handling cross-domain
data. This indirectly reflects the model’s ability to extract representational
information regarding spatio-temporal correlations across cross-domain data.
Moreover, these dynamic features play a crucial role in inferring emotional
context.

TABLE III: Results on IEMOCAP (4-class). SSL: self-supervised learning method or not. UAR: unweighted average recall. WAR: weighted average recall. Bold: the best result. Method | SSL | Modality | UAR | WAR
---|---|---|---|---
AV-HuBERT[35] | $\checkmark$ | V+A | $-$ | 46.45
MAViL[36] | $\checkmark$ | V+A | $-$ | 54.94
AVBERT [37] | $\checkmark$ | V+A | $-$ | 61.87
MultiMAE-DER | $\checkmark$ | V | $-$ | 56.13
MultiMAE-DER | $\checkmark$ | A | $-$ | 58.69
MultiMAE-DER-RFAS | $\checkmark$ | V+A | $58.62$ | 59.98
MultiMAE-DER-SFAS | $\checkmark$ | V+A | $60.39$ | 60.17
MultiMAE-DER-OFOS | $\checkmark$ | V+A | $61.87$ | 61.12
MultiMAE-DER-CFAS | $\checkmark$ | V+A | $61.98$ | 62.25
MultiMAE-DER-FFLS | $\checkmark$ | V+A | $62.92$ | 63.43
MultiMAE-DER-FSLF | $\checkmark$ | V+A | 63.21 | 63.73

In addition, compared with the MultiMAE-DER single-modal model, the MultiMAE-
DER multi-modal optimal strategy model (FSLF) is far superior in performance
to the visual-only model and the audio-only model. The results show that the
WAR on RAVDESS [19] is enhanced by 9.48% (83.61% vs. 74.13%) and 3.06% (83.61%
vs. 80.55%) respectively, the WAR on CREMA-D [20] is enhanced by 1.53% (79.36%
vs. 77.83%) and 0.91% (79.36% vs. 78.45%) respectively, and the WAR on IEMOCAP
[21] is enhanced by 7.60% (63.73% vs. 56.13%) and 5.04% (63.73% vs. 58.69%)
respectively.

Upon analyzing the outcomes across the three datasets, we observed that
MultiMAE-DER exhibits negligible discrepancies in emotion recognition results
between strategy 3 (FFLS) and 4 (FSLF). This aligns with expectations, given
that their distinctions lie solely in the sorting of multimodal sequences.
Furthermore, the effects of these two strategies surpass those of other
strategies, demonstrating enhancements ranging from 2% to 8% in WAR on the
RAVDESS [19], 1% to 5% on the CREMA-D [20], and 1% to 3% on IEMOCAP [21].

According to our analysis, different multimodal sequence fusion strategies
determine the state of the token sequence. As we know, in the ViT [18], the
sorting of the token sequence is constructed by the linear projection of
patches. In our strategy, for strategy 1, the top half of all frames consists
of visual data, while the bottom half consists of audio data. After patching,
every 98 sequences among the 1568 token sequences will undergo cross-domain
sequence exchange. Strategy 2 is the sum of cross-domain data normalization,
which will directly disrupt the overall spatial structure. Strategy 3, which
involves sorting visual data and the corresponding audio data, results in 1568
token sequences where the first 784 sequences are visual, and the latter 784
sequences are audio. Strategy 4 involves swapping the order of visual data and
audio data from strategy 3. In the resulting 1568 token sequences, the first
784 sequences are audio, and the latter 784 sequences are visual. Strategy 5
involves interleaving the order of visual data and corresponding audio data.
In the resulting 1568 token sequences, half of each sequence’s dimensions are
visual features, while the other half consists of audio features. Strategy 6
involves randomly sorting visual data and audio data. The resulting 1568 token
sequences lack coherence, whether visual-audio sequence or spatio-temporal
sequence. Finally, we can draw the following observations:

  * •

Strategy 1 exhibits temporal continuity but lacks spatial continuity.

  * •

Strategy 2 has temporal continuity but disrupts the overall spatial sequence
structure.

  * •

Strategies 3 and 4 demonstrate both spatial and temporal continuity, with a
high concentration of spatio-temporal correlation.

  * •

Strategy 5 shows spatial continuity but disrupts the overall temporal sequence
structure.

  * •

Strategy 6 lacks both spatial and temporal continuity, simultaneously
disrupting the overall spatio-temporal correlation.

##  V CONCLUSIONS AND FUTURE WORK

###  V-A CONCLUSIONS

This work represents a new exploration of the approach to handling multimodal
inputs, establishing a novel framework for the integrated processing of
multimodal data under the conditions of self-supervised learning architecture.
This paper investigates six different multimodal sequence fusion strategies to
explore the diverse interpretations of multimodal representation information
extracted by the same model when exposed to different spatio-temporal sequence
inputs. Results indicate that fusing multimodal data on spatio-temporal
sequences significantly improves the model performance by capturing
correlations between cross-domain data. Furthermore, MultiMAE-DER validates
the performance effectiveness of self-supervised learning as an efficient
learner, contributing to the potential of the masked autoencoder as a unified
framework for solving contextual semantic inference problems.

###  V-B FUTURE WORK

In future work, the framework can be further developed from two aspects. In
terms of depth, the self-supervised learning approach of the multimodal masked
autoencoder involves reconstructing the input of multimodal visual-audio data
to enhance the pre-training encoder. This enhanced model can subsequently
function as a multimodal pre-trained model, substituting the video autoencoder
pre-trained model utilized in this study. In terms of breadth, this work has
covered the aspects of “see” (video) and “listen” (audio) in the model feature
extraction process for input data, which is essential for further
incorporating “read” (text) into the model inference process.

## References

  * [1] Arriaga, Octavio, Matias Valdenegro-Toro, and Paul Plöger. ”Real-time convolutional neural networks for emotion and gender classification.” arXiv preprint arXiv:1710.07557 (2017).
  * [2] Minaee, Shervin, Mehdi Minaei, and Amirali Abdolrashidi. ”Deep-emotion: Facial expression recognition using attentional convolutional network.” Sensors 21.9 (2021): 3046.
  * [3] Ngwe, Jia Le, et al. ”PAtt-Lite: Lightweight Patch and Attention MobileNet for Challenging Facial Expression Recognition.” arXiv preprint arXiv:2306.09626 (2023).
  * [4] Zhang, Yongqiang, et al. ”Dynamic gesture recognition model based on millimeter-wave radar with ResNet-18 and LSTM.” Frontiers in Neurorobotics 16 (2022): 903197.
  * [5] Ouyang, Xi, et al. ”A 3D-CNN and LSTM based multi-task learning architecture for action recognition.” IEEE Access 7 (2019): 40757-40770.
  * [6] Vaswani, Ashish, et al. ”Attention is all you need.” Advances in neural information processing systems 30 (2017).
  * [7] Zhao, Zengqun, and Qingshan Liu. ”Former-dfer: Dynamic facial expression recognition transformer.” Proceedings of the 29th ACM International Conference on Multimedia. 2021.
  * [8] Ma, Fuyan, Bin Sun, and Shutao Li. ”Spatio-temporal transformer for dynamic facial expression recognition in the wild.” arXiv preprint arXiv:2205.04749 (2022).
  * [9] Devlin, Jacob, et al. ”Bert: Pre-training of deep bidirectional transformers for language understanding.” arXiv preprint arXiv:1810.04805 (2018).
  * [10] He, Kaiming, et al. ”Masked autoencoders are scalable vision learners.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.
  * [11] Tong, Zhan, et al. ”Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.” Advances in neural information processing systems 35 (2022): 10078-10093.
  * [12] Huang, Po-Yao, et al. ”Masked autoencoders that listen.” Advances in Neural Information Processing Systems 35 (2022): 28708-28720.
  * [13] Hsu, Wei-Ning, et al. ”Hubert: Self-supervised speech representation learning by masked prediction of hidden units.” IEEE/ACM Transactions on Audio, Speech, and Language Processing 29 (2021): 3451-3460.
  * [14] Sun, Licai, et al. ”Mae-dfer: Efficient masked autoencoder for self-supervised dynamic facial expression recognition.” Proceedings of the 31st ACM International Conference on Multimedia. 2023.
  * [15] Ghaleb, Esam, Mirela Popa, and Stylianos Asteriadis. ”Multimodal and temporal perception of audio-visual cues for emotion recognition.” 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 2019.
  * [16] Su, Lang, et al. ”Msaf: Multimodal split attention fusion.” arXiv preprint arXiv:2012.07175 (2020).
  * [17] Fu, Ziwang, et al. ”A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognition.” arXiv preprint arXiv:2111.02172 (2021).
  * [18] Dosovitskiy, Alexey, et al. ”An image is worth 16x16 words: Transformers for image recognition at scale.” arXiv preprint arXiv:2010.11929 (2020).
  * [19] Livingstone, Steven R., and Frank A. Russo. ”The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English.” PloS one 13.5 (2018): e0196391.
  * [20] Cao, Houwei, et al. ”Crema-d: Crowd-sourced emotional multimodal actors dataset.” IEEE transactions on affective computing 5.4 (2014): 377-390.
  * [21] Busso, Carlos, et al. ”IEMOCAP: Interactive emotional dyadic motion capture database.” Language resources and evaluation 42 (2008): 335-359.
  * [22] Tseng, Yuan, et al. ”Av-superb: A multi-task evaluation benchmark for audio-visual representation models.” arXiv preprint arXiv:2309.10787 (2023).
  * [23] Loshchilov, Ilya, and Frank Hutter. ”Decoupled weight decay regularization.” arXiv preprint arXiv:1711.05101 (2017).
  * [24] Fukui, Akira, et al. ”Multimodal compact bilinear pooling for visual question answering and visual grounding.” arXiv preprint arXiv:1606.01847 (2016).
  * [25] Joze, Hamid Reza Vaezi, et al. ”MMTM: Multimodal transfer module for CNN fusion.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.
  * [26] Verbitskiy, Sergey, Vladimir Berikov, and Viacheslav Vyshegorodtsev. ”Eranns: Efficient residual audio neural networks for audio pattern recognition.” Pattern Recognition Letters 161 (2022): 38-44.
  * [27] Ghaleb, Esam, Jan Niehues, and Stylianos Asteriadis. ”Multimodal attention-mechanism for temporal emotion recognition.” 2020 IEEE International Conference on Image Processing (ICIP). IEEE, 2020.
  * [28] Tsai, Yao-Hung Hubert, et al. ”Multimodal transformer for unaligned multimodal language sequences.” Proceedings of the conference. Association for computational linguistics. Meeting. Vol. 2019. NIH Public Access, 2019.
  * [29] Chumachenko, Kateryna, Alexandros Iosifidis, and Moncef Gabbouj. ”Self-attention fusion for audiovisual emotion recognition with incomplete data.” 2022 26th International Conference on Pattern Recognition (ICPR). IEEE, 2022.
  * [30] Sadok, Samir, Simon Leglaive, and Renaud Séguier. ”A vector quantized masked autoencoder for speech emotion recognition.” 2023 IEEE International conference on acoustics, speech, and signal processing workshops (ICASSPW). IEEE, 2023.
  * [31] Tran, Minh, and Mohammad Soleymani. ”A pre-trained audio-visual transformer for emotion recognition.” ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022.
  * [32] Zadeh, Amir, et al. ”Tensor fusion network for multimodal sentiment analysis.” arXiv preprint arXiv:1707.07250 (2017).
  * [33] Goncalves, Lucas, and Carlos Busso. ”AuxFormer: Robust approach to audiovisual emotion recognition.” ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022.
  * [34] Goncalves, Lucas, and Carlos Busso. ”Robust audiovisual emotion recognition: Aligning modalities, capturing temporal information, and handling missing features.” IEEE Transactions on Affective Computing 13.4 (2022): 2156-2170.
  * [35] Shi, Bowen, et al. ”Learning audio-visual speech representation by masked multimodal cluster prediction.” arXiv preprint arXiv:2201.02184 (2022).
  * [36] Huang, Po-Yao, et al. ”Mavil: Masked audio-video learners.” Advances in Neural Information Processing Systems 36 (2024).
  * [37] Lee, Sangho, et al. ”Parameter efficient multimodal transformers for video representation learning.” arXiv preprint arXiv:2012.04124 (2020).