$\mathbb{E}\|\mathbb{G}_{n}\|_{\mathcal{F}}\leq\mathfrak{C}\pi_{0},$ (E.33)

where $\mathfrak{C}$ is a universal constant. Applying McDiarmid’s inequality
gives us

$\mathbb{P}\bigl{(}\|\mathbb{G}_{n}\|_{\mathcal{F}}-\mathbb{E}\|\mathbb{G}_{n}\|_{\mathcal{F}}\geq
u\bigr{)}\leq\exp\biggl{(}-\frac{2u^{2}}{\sum_{i=1}^{n}c_{i}^{2}}\biggr{)}\leq\exp\biggl{(}-\frac{2u^{2}}{\sum_{i=1}^{n}4\pi_{0}^{2}/n}\biggr{)}=\exp(-u^{2}/2\pi_{0}^{2}),$
(E.34)

where

$\displaystyle c_{i}$
$\displaystyle:=\sup_{\begin{subarray}{c}(x_{i},y_{i},t_{i}),(x_{i}^{\prime},y_{i}^{\prime},t_{i}^{\prime}),\\\
(x_{1},y_{1},t_{1}),\ldots,(x_{n},y_{n},t_{n})\end{subarray}}\sup_{\theta}\sqrt{n}\biggl{|}\frac{1}{n}\sum_{j=1}^{n}\mathbbm{1}\\{t_{j}=0\\}\widehat{\pi}(x_{j})\mathbbm{1}\\{r_{j}\leq\theta\\}-\frac{1}{n}\sum_{j=1,j\neq
i}^{n}\mathbbm{1}\\{t_{j}=0\\}\widehat{\pi}(x_{j})\mathbbm{1}\\{r_{j}\leq\theta\\}$
$\displaystyle\qquad\qquad\qquad-\frac{1}{n}\mathbbm{1}\\{t_{i}^{\prime}=0\\}\widehat{\pi}(x_{i}^{\prime})\mathbbm{1}\\{r_{i}^{\prime}\leq\theta\\}\biggr{|}$
$\displaystyle\leq\sup_{(x_{i},y_{i},t_{i}),(x_{i}^{\prime},y_{i}^{\prime},t_{i}^{\prime})}\sup_{\theta}\sqrt{n}\biggl{|}\frac{1}{n}\mathbbm{1}\\{t_{i}=0\\}\widehat{\pi}(x_{i})\mathbbm{1}\\{r_{i}\leq\theta\\}-\frac{1}{n}\mathbbm{1}\\{t_{i}^{\prime}=0\\}\widehat{\pi}(x_{i}^{\prime})\mathbbm{1}\\{r_{i}^{\prime}\leq\theta\\}\biggr{|}\leq\frac{2\pi_{0}}{\sqrt{n}}.$

Substituting the expectation bound (E.33) in (E.34) and setting the right hand
side of (E.34) to $\delta$ yields for another absolute constant
$\mathfrak{C}^{\prime}$

$\mathbb{P}\biggl{(}\|\mathbb{G}_{n}\|_{\mathcal{F}}\geq\mathfrak{C}^{\prime}\sqrt{\pi_{0}^{2}+\pi_{0}^{2}\log\bigl{(}\frac{1}{\delta}\bigr{)}}\biggr{)}\leq\mathbb{P}\biggl{[}\|\mathbb{G}_{n}\|_{\mathcal{F}}\geq\mathfrak{C}\biggl{\\{}\pi_{0}+\pi_{0}\sqrt{2\log\bigl{(}\frac{1}{\delta}\bigr{)}}\biggr{\\}}\biggr{]}\leq\delta.$
(E.35)

#### Bound on $\sup_{\theta}|\mathcal{R}_{2}(\theta)|$:

We define the class of functions
$\mathcal{F}=\bigl{\\{}f:f_{\theta}(w)=\widehat{m}(\theta,x_{i})(\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\})\bigr{\\}}$
with $F(w)=m_{0}(\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\})$ as its
envelope.

Note that

$\displaystyle\sup_{\theta}|\mathbb{G}_{n}f|$
$\displaystyle=\sup_{\theta}\biggl{|}\mathbb{G}_{n}\Bigl{[}(\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\})\widehat{m}(\theta,x_{i})\Bigr{]}\biggr{|}$
(E.36)
$\displaystyle=\sup_{\theta}\biggl{|}\mathbb{G}_{n}\Bigl{[}(\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\})\int_{0}^{m_{0}}\mathbbm{1}\\{\widehat{m}(\theta,x_{i})\geq
u\\}\mathrm{d}u\Bigr{]}\biggr{|}$
$\displaystyle=\sup_{\theta}\biggl{|}\int_{0}^{m_{0}}\mathbb{G}_{n}\Bigl{[}(\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\})\mathbbm{1}\\{\widehat{m}(\theta,x_{i})\geq
u\\}\mathrm{d}u\Bigr{]}\biggr{|}$
$\displaystyle\stackrel{{\scriptstyle(a)}}{{=}}\sup_{\theta}\biggl{|}\int_{0}^{m_{0}}\mathbb{G}_{n}\Bigl{[}(\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\})\mathbbm{1}\\{h(x_{i},u)\leq\theta\\}\mathrm{d}u\Bigr{]}\biggr{|}\text{,
for some function }h$
$\displaystyle\leq\int_{0}^{m_{0}}\sup_{\theta}\biggl{|}\mathbb{G}_{n}\Bigl{[}(\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\})\mathbbm{1}\\{h(x_{i},u)\leq\theta\\}\Bigr{]}\biggr{|}\mathrm{d}u,$

where equality (a) is from the monotonicity of $\widehat{m}(\theta,x)$ in
$\theta$. Taking the expectation on both sides gives us

$\mathbb{E}\|\mathbb{G}_{n}\|_{\mathcal{F}}\leq\int_{0}^{m_{0}}\mathbb{E}\Bigl{[}\sup_{\theta}\bigl{|}\mathbb{G}_{n}\bigl{[}(\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\})\mathbbm{1}\\{h(x_{i},u)\leq\theta\\}\bigr{]}\bigr{|}\Bigr{]}\mathrm{d}u.$
(E.37)

Applying Lemma 4 for $s(t,x)=\mathbbm{1}\\{t=1\\}-\mathbbm{1}\\{t=0\\}$ gives
us for any fixed $u$,

$\mathbb{E}\sup_{\theta}\Bigl{|}\mathbb{G}_{n}\bigl{[}(\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\})\mathbbm{1}\\{h(x_{i},u)\leq\theta\\}\bigr{]}\Bigr{|}\leq\mathfrak{C},$

where $\mathfrak{C}$ is a universal constant. Plugging this back to (E.37)
gives us

$\mathbb{E}\|\mathbb{G}_{n}\|_{\mathcal{F}}\leq\mathfrak{C}m_{0}.$ (E.38)

Using McDiarmid’s inequality we have

$\mathbb{P}(\|\mathbb{G}_{n}\|_{\mathcal{F}}-\mathbb{E}\|\mathbb{G}_{n}\|_{\mathcal{F}}\geq
u)\leq\exp\biggl{(}-\frac{2u^{2}}{\sum_{i=1}^{n}c_{i}^{2}}\biggr{)}\leq\exp\biggl{(}-\frac{2u^{2}}{\sum_{i=1}^{n}4m_{0}^{2}/n}\biggr{)}=\exp(-u^{2}/2m_{0}^{2}),$
(E.39)

where

$\displaystyle c_{i}$
$\displaystyle:=\sup_{\begin{subarray}{c}(x_{i},y_{i},t_{i}),(x_{i}^{\prime},y_{i}^{\prime},t_{i}^{\prime}),\\\
(x_{1},y_{1},t_{1}),\ldots,(x_{n},y_{n},t_{n})\end{subarray}}\sup_{\theta}\sqrt{n}\biggl{|}\frac{1}{n}\sum_{j=1}^{n}\widehat{m}(\theta,x_{j})(\mathbbm{1}\\{t_{j}=1\\}-\mathbbm{1}\\{t_{j}=0\\})$
$\displaystyle\qquad\qquad\qquad-\frac{1}{n}\sum_{j=1,j\neq
i}^{n}\widehat{m}(\theta,x_{j})(\mathbbm{1}\\{t_{j}=1\\}-\mathbbm{1}\\{t_{j}=0\\})-\frac{1}{n}\widehat{m}(\theta,x_{i}^{\prime})(\mathbbm{1}\\{t_{i}^{\prime}=1\\}-\mathbbm{1}\\{t_{i}^{\prime}=0\\})\biggr{|}$
$\displaystyle\leq\sup_{(x_{i},y_{i},t_{i}),(x_{i}^{\prime},y_{i}^{\prime},t_{i}^{\prime})}\sup_{\theta}\frac{1}{\sqrt{n}}\biggl{|}\widehat{m}(\theta,x_{i})(\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\})-\widehat{m}(\theta,x_{i}^{\prime})(\mathbbm{1}\\{t_{i}^{\prime}=1\\}-\mathbbm{1}\\{t_{i}^{\prime}=0\\})\biggr{|}\leq\frac{2m_{0}}{\sqrt{n}}.$

Substituting the expectation bound (E.38) in (E.39) and setting the right hand
side of (E.39) to $\delta$ yields for another absolute constant
$\mathfrak{C}^{\prime}$,

$\mathbb{P}\biggl{(}\|\mathbb{G}_{n}\|_{\mathcal{F}}\geq\mathfrak{C}^{\prime}m_{0}\sqrt{1+\log\bigl{(}\frac{1}{\delta}\bigr{)}}\biggr{)}\leq\mathbb{P}\biggl{[}\|\mathbb{G}_{n}\|_{\mathcal{F}}\geq\mathfrak{C}m_{0}\biggl{\\{}1+\sqrt{2\log\bigl{(}\frac{1}{\delta}\bigr{)}}\biggr{\\}}\biggr{]}\leq\delta.$
(E.40)

#### Bound on $\mathcal{R}_{3}$:

Because the random variables in the averaging of $\mathcal{R}_{3}$ are i.i.d,
applying Hoeffding’s inequality yields

$\mathbb{P}\biggl{\\{}\frac{1}{n}\sum_{i=1}^{n}\mathbbm{1}\\{t_{i}=1\\}-\mathbb{P}(T=1)\geq
t\biggr{\\}}\leq\exp\biggl{(}-\frac{2t^{2}}{n}\biggr{)}.$ (E.41)

And this leads to

$\mathbb{P}\biggl{(}R_{3}\geq(1-\alpha)\sqrt{\frac{1}{2n}\log\bigl{(}\frac{1}{\delta}\bigr{)}}\biggr{)}\leq\delta.$
(E.42)

Combining (E.35), (E.40) and (E.42) together using the union bound gives the
result that for a universal constant $\mathfrak{C}$,

$\mathbb{P}\biggl{\\{}\sup_{\theta}\bigl{|}\mathcal{R}_{1}(\theta)+\mathcal{R}_{2}(\theta)+\mathcal{R}_{3}\bigr{|}\geq\mathfrak{C}\sqrt{\frac{(m_{0}+\pi_{0}+1-\alpha)^{2}\log\bigl{(}\frac{1}{\delta}\bigr{)}+(m_{0}+\pi_{0})^{2}}{n}}\biggr{\\}}\leq\delta.$
(E.43)

∎

##  S.7 Proof of Theorem 5

###### Proof.

Without loss of generality assume the indexes in $\mathcal{D}^{\mathrm{tr}}$
is $1,\dots,N$ with $N:=|\mathcal{D}^{\mathrm{tr}}|$, and we expand
$\mathbb{P}_{N}\bigl{[}\mathrm{IF}(\theta,\mathcal{D}^{\mathrm{tr}};\widehat{\pi},\widehat{m})\bigr{]}-P\bigl{[}\mathrm{IF}(\theta,X,R,T,\widehat{\pi},\widehat{m})\bigr{]}$
into three parts,

$\displaystyle\mathbb{P}_{N}\bigl{[}\mathrm{IF}(\theta,\mathcal{D}^{\mathrm{tr}};\widehat{\pi},\widehat{m})\bigr{]}-P\bigl{[}\mathrm{IF}(\theta,X,R,T;\widehat{\pi},\widehat{m})\bigr{]}$
(E.44) $\displaystyle=\;$
$\displaystyle\frac{1}{N}\sum_{i=1}^{N}\mathbbm{1}\\{t_{i}=0\\}\widehat{\pi}(x_{i})\mathbbm{1}\\{r_{i}\leq\theta\\}-\mathbb{E}\bigl{[}\mathbbm{1}\\{t_{i}=0\\}\widehat{\pi}(x_{i})\mathbbm{1}\\{r_{i}\leq\theta\\}\bigr{]}$
(E.45)
$\displaystyle\quad+\frac{1}{N}\sum_{i=1}^{N}\widehat{m}(\theta,x_{i})\bigl{(}\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\}\bigr{)}-\mathbb{E}\bigl{[}\widehat{m}(\theta,x_{i})(\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\})\bigr{]}$
(E.46)
$\displaystyle\quad-(1-\alpha)\biggl{[}\frac{1}{N}\sum_{i=1}^{N}\mathbbm{1}\\{t_{i}=1\\}-\mathbb{P}(T=1)\biggr{]}$
(E.47) $\displaystyle=$
$\displaystyle:\;\mathcal{R}_{1}(\widehat{\pi},\theta)+\mathcal{R}_{2}(\widehat{m},\theta)+\mathcal{R}_{3},$
(E.48)

where the three terms will be controlled separately. In particular,
$\sup_{\theta}\mathcal{R}_{1}(\widehat{\pi},\theta)$ and
$\sup_{\theta}\mathcal{R}_{2}(\widehat{m},\theta)$ will be bounded using tools
from empirical processes.

#### Bound on $\sup_{\theta}\mathcal{R}_{1}(\widehat{\pi},\theta)$:

We have a class of functions
$\mathcal{F}=\bigl{\\{}f:f_{\widehat{\pi},\theta}(w)=\mathbbm{1}\\{t=1\\}\widehat{\pi}(x)\mathbbm{1}{\\{R(x,y)\leq\theta\\}},\forall\widehat{\pi}(\cdot)\in\mathcal{F}_{\pi},\theta\in\mathbb{R}\bigr{\\}}$.
Notice that $\forall\theta\in\mathbb{R}$ and
$\widehat{\pi}(\cdot)\in\mathcal{F}_{\pi}$, we have
$|f_{\theta}(w)|\leq\pi_{0}\mathbbm{1}\\{t=1\\}$. Therefore,
$F(w):=\pi_{0}\mathbbm{1}\\{t=1\\}$ is an envelope function of
$\\{f_{\widehat{\pi},\theta}(\cdot):\widehat{\pi}(\cdot)\in\mathcal{F}_{\pi},\theta\in\mathbb{R}\\}$.

Note that because $\widehat{\pi}(\cdot)$ depends on the evaluation data,
$\mathcal{R}_{1}(\widehat{\pi},\theta)$ is not an average of $N$ i.i.d random
variables. However we can still bound this term by
$\sup_{f\in\mathcal{F}}|\mathbb{G}_{N}f|$ where for any fixed
$f\in\mathcal{F},\mathbb{G}_{N}f/\sqrt{N}$ is an average of $N$ i.i.d. random
variables with mean 0.

For any pair of functions
$f_{\widehat{\pi}_{1},\theta_{1}},f_{\widehat{\pi}_{2},\theta_{2}}\in\mathcal{F}$,

$\displaystyle\|f_{\widehat{\pi}_{1},\theta_{1}}-f_{\widehat{\pi}_{2},\theta_{2}}\|_{Q}$
$\displaystyle=\Bigl{[}\sum_{i=1}^{N}\mathbbm{1}^{2}\\{t_{i}=1\\}\bigl{(}\widehat{\pi}_{1}(x_{i})\mathbbm{1}{\\{r_{i}\leq\theta_{1}\\}}-\widehat{\pi}_{2}(x_{i})\mathbbm{1}{\\{r_{i}\leq\theta_{2}\\}}\bigr{)}^{2}Q(w_{i})\Bigr{]}^{1/2}$
(E.49)
$\displaystyle=\frac{\Bigl{[}\sum_{i=1}^{N}\mathbbm{1}\\{t_{i}=1\\}\bigl{(}\widehat{\pi}_{1}(x_{i})\mathbbm{1}{\\{r_{i}\leq\theta_{1}\\}}-\widehat{\pi}_{2}(x_{i})\mathbbm{1}{\\{r_{i}\leq\theta_{2}\\}}\bigr{)}^{2}Q(w_{i})\biggr{]}^{1/2}}{\bigl{[}\sum_{i=1}^{N}\mathbbm{1}^{2}\\{t_{i}=1\\}\
Q(w_{i})\bigr{]}^{1/2}}\times$ (E.50)
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\bigl{[}\sum_{i=1}^{N}\mathbbm{1}^{2}\\{t_{i}=1\\}]Q(w_{i})\bigr{]}^{1/2}$
(E.51)
$\displaystyle=\bigl{\|}\widehat{\pi}_{1}(x_{i})\mathbbm{1}{\\{r_{i}\leq\theta_{1}\\}}-\widehat{\pi}_{2}(x_{i})\mathbbm{1}{\\{r_{i}\leq\theta_{2}\\}}\bigr{\|}_{\widetilde{Q}}\cdot\bigl{[}\sum_{i=1}^{N}\mathbbm{1}\\{t_{i}=1\\}Q(w_{i})\bigr{]}^{1/2},$
(E.52)

where $\widetilde{Q}$ is the new probability measure defined by

$\widetilde{Q}(w_{i}):=\mathbbm{1}\\{t_{i}=1\\}Q(w_{i})/\sum_{i=1}^{N}\mathbbm{1}\\{t_{i}=1\\}Q(w_{i}),i=1,\dots,N.$
(E.53)

Using $F(w)=\pi_{0}\mathbbm{1}\\{t=1\\}$ as the envelope function of the class
$\mathcal{F}$, (E.52) becomes

$\|f_{\widehat{\pi}_{1},\theta_{1}}-f_{\widehat{\pi}_{2},\theta_{2}}\|_{Q}=\bigl{\|}\widehat{\pi}_{1}(x_{i})\mathbbm{1}{\\{r_{i}\leq\theta_{1}\\}}-\widehat{\pi}_{2}(x_{i})\mathbbm{1}{\\{r_{i}\leq\theta_{2}\\}}\bigr{\|}_{\widetilde{Q}}\cdot\|F\|_{Q}/\pi_{0}.$
(E.54)

Define a new class of functions
$\mathcal{F}_{1}:=\bigl{\\{}f:f_{\widehat{\pi},\theta}(z)=\widehat{\pi}(x)\mathbbm{1}\\{R(x,y)\leq\theta\\}\bigr{\\}}$.
Then (E.54) gives the relationship between the covering numbers of the two
classes $\mathcal{F}$ and $\mathcal{F}_{1}$,

$N\bigl{(}\varepsilon\|F\|_{Q},\mathcal{F},L_{2}(Q)\bigr{)}\leq
N\bigl{(}\pi_{0}\varepsilon,\mathcal{F}_{1},L_{2}(\widetilde{Q})\bigr{)}.$
(E.55)

We take the envelope function of $\mathcal{F}_{1}$ to be $F_{1}\equiv\pi_{0}$
and use Theorem 2.6.7 of van der Vaart and Wellner (1996) with $r=2$ to get a
bound on the covering number bound for
$\mathcal{G}=\bigl{\\{}g_{\theta}:g_{\theta}(z)=\mathbbm{1}\\{R(x,y)\leq\theta\\},\forall\theta\in\mathbb{R}\\}$.
Specifically, for any probability measure $Q$, there exists a universal
constant $\mathfrak{C}$ such that

$N\bigl{(}\varepsilon,\mathcal{G},L_{2}(Q)\bigr{)}\leq\frac{\mathfrak{C}}{\varepsilon}.$
(E.56)

Let
$\mathcal{H}:=\\{h:h=\widehat{\pi}(x),\forall\widehat{\pi}\in\mathcal{F}_{\pi}\\}$.
The relationship between covering and bracketing numbers gives us

$N(\varepsilon,\mathcal{H},L_{2}(Q))\leq
N_{[\,]}(2\varepsilon,\mathcal{H},L_{2}(Q))\leq\exp\bigl{(}C(2\pi_{0}\varepsilon)^{-\alpha_{\pi}}\bigr{)}.$
(E.57)

Applying Lemma 5 with $\mathcal{F}_{1}=\mathcal{G}$,
$\mathcal{F}_{2}=\mathcal{H}$ and $C_{1}=1,C_{2}=\pi_{0}$ gives

$N\bigl{(}\pi_{0}\varepsilon,\mathcal{F}_{1},L_{2}(Q)\bigr{)}\leq
N\bigl{(}\varepsilon/2,\mathcal{G},L_{2}(Q)\bigr{)}N\bigl{(}\pi_{0}\varepsilon/2,\mathcal{H},L_{2}(Q)\bigr{)}\leq\frac{2\mathfrak{C}}{\varepsilon}\exp\bigl{(}C(\pi_{0}\varepsilon)^{-\alpha_{\pi}}\bigr{)}.$
(E.58)

Therefore, it holds for some constant $\mathfrak{C}^{\prime}$ such that,

$\log N\bigl{(}\varepsilon\|F_{1}\|_{Q},\mathcal{F}_{1},L_{2}(Q)\bigr{)}\leq
C(\pi_{0}\varepsilon)^{-\alpha_{\pi}}-\log(\varepsilon)+\mathfrak{C}\leq\mathfrak{C}^{\prime}(\pi_{0}\varepsilon)^{-\alpha_{\pi}}.$
(E.59)

Because (E.59) holds for any probability measure $Q$, we choose $Q$ to be
$\widetilde{Q}$ defined in (E.53) to bound the right hand side of (E.55),

$\log N\bigl{(}\varepsilon\|F\|_{Q},\mathcal{F},L_{2}(Q)\bigr{)}\leq\log
N\bigl{(}\pi_{0}\varepsilon,\mathcal{F}_{1},L_{2}(\widetilde{Q})\bigr{)}\leq\mathfrak{C}^{\prime}(\pi_{0}\varepsilon)^{-\alpha_{\pi}}.$
(E.60)

We apply Lemma A.1 of Srebro et al. (2012) so that

$\displaystyle\mathbb{E}\|\mathbb{G}_{N}\|_{\mathcal{F}}/\pi_{0}$
$\displaystyle\lesssim\inf_{\eta}\biggl{\\{}N^{1/2}\eta+\int_{\eta}^{1}\log_{+}^{1/2}N_{[]}\left(\pi_{0}\varepsilon,\mathcal{F},\|\cdot\|_{L_{2}(P)}\right)\mathrm{d}\varepsilon\biggr{\\}}+N^{-1/2}\log_{+}N_{[]}\left(\pi_{0},\mathcal{F},\|\cdot\|_{L_{2}(P)}\right)$
$\displaystyle\leq\inf_{\eta}\biggl{\\{}N^{1/2}\eta+\mathfrak{C}\int_{\eta}^{1}(\pi_{0}\varepsilon)^{-\alpha_{\pi}/2}\mathrm{d}\varepsilon\biggr{\\}}+N^{-1/2}\mathfrak{C}\pi_{0}^{-\alpha_{\pi}}$
$\displaystyle\leq\frac{1}{\pi_{0}}N^{1/2-1/\alpha_{\pi}}\bigl{(}1+\mathbbm{1}\\{\alpha_{\pi}=2\\}\log
N\bigr{)}+N^{-1/2}\mathfrak{C}\pi_{0}^{-\alpha_{\pi}}$
$\displaystyle\leq\frac{\mathfrak{C}}{\pi_{0}}N^{{(1/2-1/\alpha_{\pi})}_{+}}\bigl{(}1+\mathbbm{1}\\{\alpha_{\pi}=2\\}\log
N\bigr{)}\text{, when $N$ is large.}$ (E.61)

Using McDiarmid’s inequality we have

$\mathbb{P}\bigl{(}\|\mathbb{G}_{N}\|_{\mathcal{F}}-\mathbb{E}\|\mathbb{G}_{N}\|_{\mathcal{F}}\geq
u\bigr{)}\leq\exp\biggl{(}-\frac{2u^{2}}{\sum_{i=1}^{N}c_{i}^{2}}\biggr{)}\leq\exp\biggl{(}-\frac{2u^{2}}{\sum_{i=1}^{N}4\pi_{0}^{2}/N}\biggr{)}=\exp(-u^{2}/2\pi_{0}^{2}),$
(E.62)

where

$\displaystyle c_{i}$
$\displaystyle:=\sup_{\begin{subarray}{c}(x_{i},y_{i},t_{i}),(x_{i}^{\prime},y_{i}^{\prime},t_{i}^{\prime}),\\\
(x_{1},y_{1},t_{1}),\ldots,(x_{N},y_{N},t_{N})\end{subarray}}\sup_{\widehat{\pi},\theta}\sqrt{N}\biggl{|}\frac{1}{N}\sum_{j=1}^{N}\mathbbm{1}\\{t_{j}=0\\}\widehat{\pi}(x_{j})\mathbbm{1}\\{r_{j}\leq\theta\\}-\frac{1}{N}\sum_{j=1,j\neq
i}^{N}\mathbbm{1}\\{t_{j}=0\\}\widehat{\pi}(x_{j})\mathbbm{1}\\{r_{j}\leq\theta\\}$
$\displaystyle\qquad\qquad\qquad-\frac{1}{N}\mathbbm{1}\\{t_{i}^{\prime}=0\\}\widehat{\pi}(x_{i}^{\prime})\mathbbm{1}\\{r_{i}^{\prime}\leq\theta\\}\biggr{|}$
$\displaystyle\leq\sup_{(x_{i},y_{i},t_{i}),(x_{i}^{\prime},y_{i}^{\prime},t_{i}^{\prime})}\sup_{\widehat{\pi},\theta}\sqrt{N}\biggl{|}\frac{1}{N}\mathbbm{1}\\{t_{i}=0\\}\widehat{\pi}(x_{i})\mathbbm{1}\\{r_{i}\leq\theta\\}-\frac{1}{N}\mathbbm{1}\\{t_{i}^{\prime}=0\\}\widehat{\pi}(x_{i}^{\prime})\mathbbm{1}\\{r_{i}^{\prime}\leq\theta\\}\biggr{|}\leq\frac{2\pi_{0}}{\sqrt{N}}.$

Substituting the expectation bound (E.61) in (E.62) and setting the right hand
side of (E.62) to $\delta$ yields for another absolute constant
$\mathfrak{C}^{\prime}$,

$\mathbb{P}\biggl{[}\|\mathbb{G}_{n}\|_{\mathcal{F}}\geq\mathfrak{C}^{\prime}\biggl{\\{}N^{{(1/2-1/\alpha_{\pi})}_{+}}\bigl{(}1+\mathbbm{1}\\{\alpha_{\pi}=2\\}\log
N\bigr{)}+\pi_{0}\sqrt{2\log\bigl{(}\frac{1}{\delta}\bigr{)}}\biggr{\\}}\biggr{]}\leq\delta.$
(E.63)

#### Bound on $\sup_{\theta}\mathcal{R}_{2}(\widehat{m},\theta)$:

We define the class of functions
$\mathcal{F}=\\{f:f_{\widehat{m},\theta}(w)=\widehat{m}(\theta,x_{i})(\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\},\forall\widehat{m}(\cdot,\cdot)\in\mathcal{F}_{m}\text{
and }\theta\in\mathbb{R}\\}$ with
$F(w)=m_{0}(\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\})$ as its
envelope. Similar as before we bound this term by
$\sup_{f\in\mathcal{F}}|\mathbb{G}_{N}f|$ and for any fixed
$f\in\mathcal{F},|\mathbb{G}_{N}f|/\sqrt{N}$ is an average of i.i.d. random
variables.

For any pair of functions
$f_{\widehat{m}_{1},\theta_{1}},f_{\widehat{m}_{2},\theta_{2}}\in\mathcal{F}$,

$\displaystyle\|f_{\widehat{m}_{1},\theta_{1}}-f_{\widehat{m}_{2},\theta_{2}}\|_{Q}$
$\displaystyle=\Bigl{[}\sum_{i=1}^{N}\bigl{(}\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\}\bigr{)}^{2}\bigl{(}\widehat{m}_{1}(\theta_{1},x_{i})-\widehat{m}_{2}(\theta_{2},x_{i})\bigr{)}^{2}Q(w_{i})\Bigr{]}^{1/2}$
(E.64)
$\displaystyle=\frac{\Bigl{[}\sum_{i=1}^{N}\bigl{(}\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\}\bigr{)}^{2}\bigl{(}\widehat{m}_{1}(\theta_{1},x_{i})-\widehat{m}_{2}(\theta_{2},x_{i})\bigr{)}^{2}Q(w_{i})\Bigr{]}^{1/2}}{\bigl{[}\sum_{i=1}^{N}\bigl{(}\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\}\bigr{)}^{2}Q(w_{i})\bigr{]}^{1/2}}\times$
(E.65)
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\bigl{[}\sum_{i=1}^{N}\bigl{(}\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\}\bigr{)}^{2}Q(w_{i})\bigr{]}^{1/2}$
(E.66)
$\displaystyle=\bigl{\|}\bigl{(}\widehat{m}_{1}(\theta_{1},x_{i})-\widehat{m}_{2}(\theta_{2},x_{i})\bigr{\|}_{\widetilde{Q}}\cdot\bigl{[}\sum_{i=1}^{n}\bigl{(}\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\}\bigr{)}^{2}Q(w_{i})\bigr{]}^{1/2},$
(E.67)

where $\widetilde{Q}$ is the new probability measure defined by

$\widetilde{Q}(w_{i}):=\bigl{(}\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\}\bigr{)}^{2}Q(w_{i})/\sum_{i=1}^{N}\bigl{(}\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\}\bigr{)}^{2}Q(w_{i}),i=1,\dots,N.$
(E.68)

Using $F(w)=m_{0}(\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\})$ as the
envelope function of the class $\mathcal{F}$, (E.67) becomes

$\|f_{\widehat{m}_{1},\theta_{1}}-f_{\widehat{m}_{2},\theta_{2}}\|_{Q}=\bigl{\|}\widehat{m}_{1}(\theta_{1},x_{i})-\widehat{m}_{2}(\theta_{2},x_{i})\bigr{\|}_{\widetilde{Q}}\cdot\|F\|_{Q}/m_{0}.$
(E.69)

Recall the class of functions
$\mathcal{F}_{m}:=\bigl{\\{}f:f_{\widehat{m},\theta}(z)=\widehat{m}(\theta,x)\bigr{\\}}$
with its envelope $F_{1}\equiv m_{0}$. Then (E.69) gives the relationship
between the covering numbers of the two classes $\mathcal{F}$ and
$\mathcal{F}_{m}$. Along with the bracketing number assumption on
$\mathcal{F}_{m}$ it follows that

$\log N\bigl{(}\varepsilon\|F\|_{Q},\mathcal{F},L_{2}(Q)\bigr{)}\leq\log
N\bigl{(}m_{0}\varepsilon,\mathcal{F}_{m},L_{2}(\widetilde{Q})\bigr{)}\leq\log
N_{[]}\bigl{(}2m_{0}\varepsilon,\mathcal{F}_{m},L_{2}(\widetilde{Q})\bigr{)}\leq
C(2m_{0}\varepsilon)^{-\alpha_{m}}.$ (E.70)

Applying the same technique as we used in (E.61) when bounding
$\sup_{\theta}\mathcal{R}_{1}$ gives us $\forall\alpha_{m}\geq 0$,

$\mathbb{E}\|\mathbb{G}_{N}\|_{\mathcal{F}}\leq\mathfrak{C}N^{{(1/2-1/\alpha_{m})}_{+}}\bigl{(}1+\mathbbm{1}\\{\alpha_{m}=2\\}\log
N\bigr{)}.$ (E.71)

Using McDiarmid’s inequality we have

$\mathbb{P}(\|\mathbb{G}_{N}\|_{\mathcal{F}}-\mathbb{E}\|\mathbb{G}_{N}\|_{\mathcal{F}}\geq
u)\leq\exp\biggl{(}-\frac{2u^{2}}{\sum_{i=1}^{N}c_{i}^{2}}\biggr{)}\leq\exp\biggl{(}-\frac{2u^{2}}{\sum_{i=1}^{N}4m_{0}^{2}/N}\biggr{)}=\exp(-u^{2}/2m_{0}^{2}),$
(E.72)

where

$\displaystyle c_{i}$
$\displaystyle:=\sup_{\begin{subarray}{c}(x_{i},y_{i},t_{i}),(x_{i}^{\prime},y_{i}^{\prime},t_{i}^{\prime}),\\\
(x_{1},y_{1},t_{1}),\ldots,(x_{N},y_{N},t_{N})\end{subarray}}\sup_{\widehat{m},\theta}\sqrt{N}\biggl{|}\frac{1}{n}\sum_{j=1}^{N}\widehat{m}(\theta,x_{j})(\mathbbm{1}\\{t_{j}=1\\}-\mathbbm{1}\\{t_{j}=0\\})$
$\displaystyle\qquad\qquad\qquad-\frac{1}{N}\sum_{j=1,j\neq
i}^{N}\widehat{m}(\theta,x_{j})(\mathbbm{1}\\{t_{j}=1\\}-\mathbbm{1}\\{t_{j}=0\\})-\frac{1}{N}\widehat{m}(\theta,x_{i}^{\prime})(\mathbbm{1}\\{t_{i}^{\prime}=1\\}-\mathbbm{1}\\{t_{i}^{\prime}=0\\})\biggr{|}$
$\displaystyle\leq\sup_{(x_{i},y_{i},t_{i}),(x_{i}^{\prime},y_{i}^{\prime},t_{i}^{\prime})}\sup_{\widehat{m},\theta}\frac{1}{\sqrt{N}}\biggl{|}\widehat{m}(\theta,x_{i})(\mathbbm{1}\\{t_{i}=1\\}-\mathbbm{1}\\{t_{i}=0\\})-\widehat{m}(\theta,x_{i}^{\prime})(\mathbbm{1}\\{t_{i}^{\prime}=1\\}-\mathbbm{1}\\{t_{i}^{\prime}=0\\})\biggr{|}\leq\frac{2m_{0}}{\sqrt{N}}.$

Substituting the expectation bound (E.71) in (E.72) and setting the right hand
side of (E.72) to $\delta$ yields for another absolute constant
$\mathfrak{C}^{\prime}$,

$\mathbb{P}\biggl{[}\|\mathbb{G}_{N}\|_{\mathcal{F}}\geq\mathfrak{C}\biggl{\\{}N^{{(1/2-1/\alpha_{m})}_{+}}\bigl{(}1+\mathbbm{1}\\{\alpha_{m}=2\\}\log
N\bigr{)}+m_{0}\sqrt{2\log\bigl{(}\frac{1}{\delta}\bigr{)}}\biggr{\\}}\biggr{]}\leq\delta.$
(E.73)

#### Bound on $\mathcal{R}_{3}$:

Because the random variables in the averaging of $\mathcal{R}_{3}$ are i.i.d,
applying Hoeffding’s inequality yields

$\mathbb{P}\biggl{\\{}\frac{1}{N}\sum_{i=1}^{N}\mathbbm{1}\\{t_{i}=1\\}-\mathbb{P}(T=1)\geq
t\biggr{\\}}\leq\exp\biggl{(}-\frac{2t^{2}}{N}\biggr{)}.$ (E.74)

And this leads to

$\mathbb{P}\biggl{(}R_{3}\geq(1-\alpha)\sqrt{\frac{1}{2N}\log\bigl{(}\frac{1}{\delta}\bigr{)}}\biggr{)}\leq\delta.$
(E.75)

Combining (E.63), (E.73) and (E.75) together using the union bound gives the
result that for a universal constant $\mathfrak{C}$,

$\displaystyle\mathbb{P}\biggl{\\{}\sup_{\theta}\bigl{(}\mathcal{R}_{1}(\theta)+\mathcal{R}_{2}(\theta)+\mathcal{R}_{3}\bigr{)}\geq$
$\displaystyle\mathfrak{C}\biggl{(}N^{-1/(\alpha_{m}\vee
2)}\bigl{(}1+\mathbbm{1}\\{\alpha_{m}=2\\}\log
N\bigr{)}+N^{-1/(\alpha_{\pi}\vee
2)}\bigl{(}1+\mathbbm{1}\\{\alpha_{\pi}=2\\}\log N\bigr{)}$ (E.76)
$\displaystyle\qquad+\sqrt{\frac{(m_{0}+\pi_{0}+1-\alpha)^{2}\log\bigl{(}\frac{1}{\delta}\bigr{)}}{N}}\biggr{)}\biggr{\\}}\leq\delta.$

∎

##  S.8 Proof of Theorem 1

The first half of the Theorem, specifically (13), is adapted from Lemma S1 of
Qiu et al. (2022). We modify that proof a bit in our setting in the following
and first state a Lemma that is adapted from Theorem 2 and Remark 4 from Shah
and Peters (2020), which will contribute to the proof of Theorem 1.

######  Lemma 3.

(No-free-lunch for conditional independence testing). Assume that $X$ is
continuous, given any $n\in\mathbb{N},\alpha\in(0,1),M\in(0,\infty]$, and any
potentially randomised test $\eta_{n}$ that has valid level $\alpha$ for the
null hypothesis $\mathcal{P}_{0,M}$, we have that
$\mathbb{P}_{Q}\left(\eta_{n}=1\right)\leq\alpha$ for all
$Q\in\mathcal{Q}_{0,M}$. Thus $\eta_{n}$ cannot have power against any
alternative.

Let $\bar{\mathcal{E}}\supseteq\bar{\mathcal{P}}^{0}$ be the space of
distributions for the full data point $\bar{O}$ with the distribution of
$(X,Y)\mid T=0,1$ both dominated by the Lebesgue measure. These distributions
may or may not satisfy the MAR assumption that $Y\perp T\mid X$. For any
$x\in\mathcal{X}$, the prediction set $\widehat{C}(x)$ is the short hand
notation for $C\left(x;O_{1},\ldots,O_{n}\right)$. This notation is helpful to
clarify the dependence of $\widehat{C}$ on the observed training data
$\left(O_{1},\ldots,O_{n}\right)$. Let $\bar{O}_{n+1}$ to denote the full data
point from a future draw.

The subtle difference between $\bar{O}$ and $O$ is that $\bar{O}$ represents
the full but unobserved data $(X,Y,T)$ whereas $O$ corresponds to the observed
data $(X,T,(1-T)Y)$.

Define the randomized test $\eta\left(\bar{O}_{1},\ldots,\bar{O}_{n+1}\right)$
as follows:

$\eta\left(\bar{O}_{1},\ldots,\bar{O}_{n+1}\right)=\begin{cases}\mathbbm{1}\bigl{\\{}Y_{n+1}\notin
C(X_{n+1};O_{1},\ldots,O_{n})\bigr{\\}},\text{ if }T_{n+1}=1;\\\
\begin{cases}1,\text{ w.p. }\alpha,\\\ 0,\text{ w.p.
}1-\alpha,\end{cases}\text{ if }T_{n+1}=0.\end{cases}$ (E.77)

We note that although $\eta$ is a function of $n+1$ full data points
$\left(\bar{O}_{1},\ldots,\bar{O}_{n+1}\right)$, it only relies on $n$
observed training data points $\left(O_{1},\ldots,O_{n}\right)$ and one future
full data point $\bar{O}_{n+1}$. Furthermore, we can see that

$\displaystyle\mathbb{P}_{\bar{P}^{0}}\left(\eta(\bar{O}_{1},\ldots,\bar{O}_{n+1})=1\right)$
$\displaystyle=\mathbb{P}_{\bar{P}^{0}}\left(\eta(\bar{O}_{1},\ldots,\bar{O}_{n+1})=1|T_{n+1}=1\right)\mathbb{P}(T_{n+1}=1)$
$\displaystyle\qquad+\mathbb{P}_{\bar{P}^{0}}\left(\eta(\bar{O}_{1},\ldots,\bar{O}_{n+1})=1|T_{n+1}=1\right)\mathbb{P}(T_{n+1}=1)$
$\displaystyle=\mathbb{P}_{\bar{P}^{0}}\bigl{(}Y\notin\widehat{C}(X)|T=1\bigr{)}\mathbb{P}(T=1)+\mathbb{P}_{\bar{P}^{0}}\bigl{(}\eta=1|T=0\bigr{)}\mathbb{P}(T=0)$
$\displaystyle\leq\alpha\mathbb{P}(T=1)+\alpha\mathbb{P}(T=0)\text{, by
\eqref{eq:finite-marginal-set} and the definition of }\eta$
$\displaystyle\leq\alpha.$

Therefore, $\eta$ can be viewed as a test with level $\alpha$ for the null
hypothesis $Y\perp T\mid X$. Because Theorem 3 states that the power of $\eta$
against the alternative hypothesis is at most $\alpha$, we have that

$\displaystyle\mathbb{P}_{\bar{Q}}\left(\eta\left(\bar{O}_{1},\ldots,\bar{O}_{n+1}\right)=1\right)\leq\alpha\quad\text{
for any distribution }\bar{Q}\in\bar{\mathcal{E}}\text{. }$ (E.78)

Using the fact that $\eta$ is a completely randomized test when when
$T_{n+1})=0$, this gives $\mathbb{P}(\eta=1|T_{n+1}=0)=\alpha$. Therefore,

$\displaystyle\alpha\geq\mathbb{P}_{\bar{Q}}\left(\eta\left(\bar{O}_{1},\ldots,\bar{O}_{n+1}\right)=1\right)$
$\displaystyle=\mathbb{P}_{\bar{Q}}(\eta=1|T_{n+1}=0)\mathbb{P}(T_{n+1}=0)+\mathbb{P}_{\bar{Q}}(\eta=1|T_{n+1}=1)\mathbb{P}(T_{n+1}=1)$
$\displaystyle=\alpha\mathbb{P}(T_{n+1}=0)+\mathbb{P}_{\bar{Q}}(\eta=1|T_{n+1}=1)\mathbb{P}(T_{n+1}=1).$

Therefore,

$\displaystyle\alpha\geq\mathbb{P}_{\bar{Q}}(\eta=1|T_{n+1}=1)=\mathbb{P}_{\bar{Q}}\bigl{(}Y_{n+1}\notin
C(X_{n+1};O_{1},\ldots,O_{n})\mid T_{n+1}=1\bigr{)}.$ (E.79)

For any $x\in\mathcal{X}$, let $D_{x}\subseteq\mathcal{Y}$ be any Lebesgue
measurable set with nonzero finite measure and ${U}_{x}$ be the uniform
distribution on $D_{x}$. Take $\bar{Q}$ to be a distribution such that

  1. (i)

the distribution of $T$ is an arbitrary Bernoulli distribution with success
probability in $(0,1)$;

  2. (ii)

the distributions of $X\mid T=0,1$ satisfy the dominance condition (8), which
is that the source distribution ($X|T=0$) dominates the target distribution
($X|T=1$), and are arbitrary in all other aspects

  3. (iii)

the distribution of $Y\mid X=x,T=0$ is arbitrary and the distribution of
$Y\mid X=x,T=1$ is ${U}_{x}$.

We take $\bar{P}^{0}$ to be the distribution that is identical to $\bar{Q}$,
except that the distribution of $Y\mid X=x,T=1$ is identical to $Y\mid
X=x,T=0$ rather than ${U}_{x}$ under $\bar{P}^{0}$. Note that
$\bar{P}^{0}\in\bar{\mathcal{P}}^{0}$. Since $\widehat{C}$ is trained only on
observed data $\left(O_{1},\ldots,O_{n}\right)$ and $\bar{P}^{0}$ and
$\bar{Q}$ imply the same distribution of the observed data point
$O=(X,T,(1-T)Y)$, following (E.79), we have that

$\displaystyle\mathbb{P}_{\bar{Q}}\bigl{(}Y_{n+1}\notin
C(X_{n+1};O_{1},\ldots,O_{n})\mid T_{n+1}=1\bigr{)}$ $\displaystyle=$
$\displaystyle\int_{y}\mathbb{P}_{\bar{P}_{0}}\bigl{(}y\notin
C(X_{n+1};O_{1},\ldots,O_{n})\mid
T_{n+1}=1\bigr{)}U_{X_{n+1}}(\mathrm{d}y)\leq\alpha,$

where the probability is over training data and possible exogenous randomness
in $C$. Since $D_{x}$ is arbitrary, it follows that the integrand is bounded
by $\alpha$, namely

$\mathbb{P}_{\bar{P}_{0}}\Bigl{(}y\notin C(X_{n+1};O_{1},\ldots,O_{n})\mid
T_{n+1}=1\Bigr{)}\leq\alpha$ (E.80)

for a.e. $y\in\mathcal{Y}$. The desired result (13) follows by replacing the
notations $X_{n+1}$ and $T_{n+1}$ with $X$ and $T$, respectively, and noting
that $C\left(x;O_{1},\ldots,O_{n}\right)=\widehat{C}(x)$ by definition.

Note that (E.80) implies

$\inf_{y\in\mathcal{Y}}\mathbb{P}_{\bar{P}_{0}}(y\in\widehat{C}(X)|T=1)\geq
1-\alpha,$

for any prediction set $\widehat{C}$ that satisfies (12). Define
$L(X)=\inf\\{t:\,t\in\widehat{C}(X)\\}$ and
$U(X)=\sup\\{t:\,t\in\widehat{C}(X)\\}$. Because
$\widehat{C}(X)\subseteq[L(X),\infty)$, we get

$\displaystyle\mathbb{P}_{\bar{P}_{0}}\Bigl{(}\inf_{y\in\mathcal{Y}}y\geq
L(X)\bigm{|}T=1\Bigr{)}=\inf_{y\in\mathcal{Y}}\,\mathbb{P}_{\bar{P}_{0}}(y\geq
L(X)\bigm{|}T=1)=\inf_{y\in\mathcal{Y}}\mathbb{P}_{\bar{P}_{0}}\bigl{(}y\in[L(X),\infty)\bigm{|}T=1\bigr{)}\geq
1-\alpha.$ (E.81)

Similarly, because $\widehat{C}(X)\subseteq(-\infty,U(X)]$,

$\displaystyle\mathbb{P}_{\bar{P}_{0}}\Bigl{(}U(X)\geq\sup_{y\in\mathcal{Y}}y\bigm{|}T=1\Bigr{)}=\inf_{y\in\mathcal{Y}}\mathbb{P}_{\bar{P}_{0}}\bigl{(}y\leq
U(X)\bigm{|}T=1\bigr{)}=\inf_{y\in\mathcal{Y}}\mathbb{P}_{\bar{P}_{0}}\bigl{(}y\in(-\infty,U(X)]\bigm{|}T=1\bigr{)}\geq
1-\alpha.$ (E.82)

Hence with $\mathcal{Y}=\mathbb{R}$ here, then any prediction set with valid
coverage must have at least one of the end points $\infty$ in absolute value
with probability at least $1-\alpha$ and hence must have an infinite diameter
with probability at least $1-\alpha$.

##  S.9 Some useful propositions and lemmas

######  Proposition 1.

(Chernozhukov et al. (2009), Proposition 2) Let the target function
$f_{0}:\mathcal{X}^{d}\rightarrow K$ be weakly increasing and measurable in
$x.$ Let $\widehat{f}:\mathcal{X}^{d}\rightarrow K$ be a measurable function
that is an initial estimate of $f_{0}.$

  1. 1.

For each ordering $\pi$ of $1,\ldots,d$, the $\pi$-rearranged estimate
$\widehat{f}_{\pi}^{*}$ is weakly increasing. Moreover, $\widehat{f}^{*}$, an
average of $\pi$-rearranged estimates, is weakly increasing.

  2. 2.

A $\pi$-rearranged estimate $\widehat{f}_{\pi}^{*}$ of $\widehat{f}$ weakly
reduces the estimation error of $\widehat{f}$ :

$\biggl{\\{}\int_{\mathcal{X}^{d}}\bigl{|}\widehat{f}_{\pi}^{*}(x)-f_{0}(x)\bigr{|}^{p}dx\biggr{\\}}^{1/p}\leq\biggl{\\{}\int_{\mathcal{X}^{d}}\bigl{|}\widehat{f}(x)-f_{0}(x)\bigr{|}^{p}\mathrm{d}x\biggr{\\}}^{1/p}.$

###  S.9.1 Lemma 4 and its proof

######  Lemma 4.

There exists a universal constant $\mathfrak{C}<\infty$ such that for any
functions $s(t,x)\in[-\kappa_{0},\kappa_{0}]$ and $h(x,y)$,

$\mathbb{E}\Bigl{[}\sup_{\theta}|\mathbb{G}_{n}[s(t,x)\mathbbm{1}\\{h(x,y)\leq\theta\\}]|\Bigr{]}\leq\mathfrak{C}\kappa_{0}.$
(E.83)

###### Proof.

We have a class of functions
$\mathcal{F}=\bigl{\\{}f:f_{\theta}(w)=s(t,x)\mathbbm{1}{\\{h(x,y)\leq\theta\\}}\bigr{\\}}$.
Notice that $\forall\theta\in\mathbb{R}$, we have
$|f_{\theta}(w)|\leq|s(t,x)|$. Therefore, $F(w):=|s(t,x)|$ is an envelope
function of $\\{f_{\theta}(\cdot):\theta\in\mathbb{R}\\}$. For any discrete
probability measure $Q$, let $\|f(\cdot)\|_{Q}$ denote the empirical
$L_{2}(Q)$ norm where
$\|f(\cdot)\|_{Q}:=\bigl{(}\sum_{i=1}^{n}f^{2}(x_{i})Q(x_{i})\bigr{)}^{1/2}$.

Let $h_{i}$ denote $h_{i}:=h(x_{i},y_{i})$. For any function $f\in\mathcal{F}$
and $\theta_{1},\theta_{2}\in\mathbb{R}$,

$\displaystyle\|f_{\theta_{1}}-f_{\theta_{2}}\|_{Q}$
$\displaystyle=\Bigl{[}\sum_{i=1}^{n}s(t_{i},x_{i})^{2}\bigl{(}\mathbbm{1}{\\{h_{i}\leq\theta_{1}\\}}-\mathbbm{1}{\\{h_{i}\leq\theta_{2}\\}}\bigr{)}^{2}Q(w_{i})\Bigr{]}^{1/2}$
(E.84)
$\displaystyle=\frac{\Bigl{[}\sum_{i=1}^{n}s(t_{i},x_{i})^{2}\bigl{(}\mathbbm{1}{\\{h_{i}\leq\theta_{1}\\}}-\mathbbm{1}{\\{h_{i}\leq\theta_{2}\\}}\bigr{)}^{2}Q(w_{i})\biggr{]}^{1/2}}{\bigl{[}\sum_{i=1}^{n}s(t_{i},x_{i})^{2}Q(w_{i})\bigr{]}^{1/2}}\times$
(E.85)
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\bigl{[}\sum_{i=1}^{n}s(t_{i},x_{i})^{2}Q(w_{i})\bigr{]}^{1/2}$
(E.86)
$\displaystyle=\bigl{\|}\mathbbm{1}{\\{h_{i}\leq\theta_{1}\\}}-\mathbbm{1}{\\{h_{i}\leq\theta_{2}\\}}\bigr{\|}_{\widetilde{Q}}\cdot\bigl{[}\sum_{i=1}^{n}s(t_{i},x_{i})^{2}Q(w_{i})\bigr{]}^{1/2},$
(E.87)

where $\widetilde{Q}$ is the new probability measure defined by

$\widetilde{Q}(w_{i}):=s(t_{i},x_{i})^{2}Q(w_{i})/\sum_{i=1}^{n}s(t_{i},x_{i})^{2}Q(w_{i}).$
(E.88)

Using the definition of $F(w)$ as the envelope function of the class
$\mathcal{F}$, (E.87) becomes

$\|f_{\theta_{1}}-f_{\theta_{2}}\|_{Q}=\bigl{\|}\mathbbm{1}{\\{h_{i}\leq\theta_{1}\\}}-\mathbbm{1}{\\{h_{i}\leq\theta_{2}\\}}\bigr{\|}_{\widetilde{Q}}\cdot\|F\|_{Q}.$
(E.89)

Define a new class of functions
$\mathcal{F}_{1}:=\bigl{\\{}f:f_{\theta}(z)=\mathbbm{1}\\{h(x,y)\leq\theta\\}\bigr{\\}}$
with its envelope function $F_{1}\equiv 1$. Then (E.89) gives a relationship
between the covering numbers of the two classes $\mathcal{F}$ and
$\mathcal{F}_{2}$,

$N\bigl{(}\varepsilon\|F\|_{Q},\mathcal{F},L_{2}(Q)\bigr{)}\leq
N\bigl{(}\varepsilon,\mathcal{F}_{1},L_{2}(\widetilde{Q})\bigr{)}.$ (E.90)

From Chapter 2.6 of van der Vaart and Wellner (1996), we know that the VC
dimension of the class $\mathcal{F}_{1}$ is $1$. We use Theorem 2.6.7 of van
der Vaart and Wellner (1996) with $r=2$ to get a bound on the covering
numbers. Specifically, for any probability measure $Q$, there exists a
universal constant $\mathfrak{C}$ such that

$N\bigl{(}\varepsilon,\mathcal{F}_{1},L_{2}(Q)\bigr{)}=N\bigl{(}\varepsilon\|F_{1}\|_{Q},\mathcal{F},L_{2}(Q)\bigr{)}\leq\frac{\mathfrak{C}}{\varepsilon}.$
(E.91)

Because (E.91) holds for any probability measure $Q$, we choose $Q$ to be
$\widetilde{Q}$ defined in (E.88) in order to bound the right hand side of
(E.90),

$N\bigl{(}\varepsilon\|F\|_{Q},\mathcal{F},L_{2}(Q)\bigr{)}\leq
N\bigl{(}\varepsilon,\mathcal{F}_{1},L_{2}(\widetilde{Q})\bigr{)}\leq\frac{\mathfrak{C}}{\varepsilon}.$
(E.92)

Next, we obtain an upper bound of the uniform-entropy

$J(\delta,\mathcal{F}):=\sup_{Q}\int_{0}^{\delta}\sqrt{1+\log
N\bigl{(}\varepsilon\|F\|_{Q},\mathcal{F},L_{2}(Q)\bigr{)}}d\varepsilon,$

where the supremum is taken over all discrete probability measures $Q$ with
$\|F\|_{Q}>0.$ Applying $p=1$ to Theorem 2.14.1 of van der Vaart and Wellner
(1996) gives us

$\mathbb{E}\sup_{f\in\mathcal{F}}|\mathbb{G}_{n}f|~{}\leq~{}J(1,\mathcal{F})\|F\|_{Q}~{}\lesssim~{}\|F\|_{Q}\int_{0}^{1}\mathfrak{C}^{1/2}\sqrt{\log(1/\varepsilon)}d\varepsilon~{}\leq~{}\mathfrak{C}_{1}\kappa_{0},$
(E.93)

where the second inequality is from (E.92) and $\mathfrak{C}_{1}$ is a
universal constant. ∎

###  S.9.2 Lemma 5 and its proof

######  Lemma 5.

For any two bounded function classes $\mathcal{F}_{1}$ and $\mathcal{F}_{2}$
with $\sup_{f_{1}\in\mathcal{F}_{1}}\|f_{1}\|_{\infty}\leq C_{1}$ and
$\sup_{f_{2}\in\mathcal{F}_{2}}\|f_{2}\|_{\infty}\leq C_{2}$, the following
holds for the covering number of the class of functions
$\mathcal{F}_{1}\times\mathcal{F}_{2}:=\\{f=f_{1}f_{2},f_{1}\in\mathcal{F}_{1}\text{
and }f_{2}\in\mathcal{F}_{2}\\}$,

$N\bigl{(}\varepsilon,\mathcal{F}_{1}\times\mathcal{F}_{2},L_{2}(Q)\bigr{)}\leq
N\bigl{(}\frac{\varepsilon}{2C_{2}},\mathcal{F}_{1},L_{2}(Q)\bigr{)}N\bigl{(}\frac{\varepsilon}{2C_{1}},\mathcal{F}_{2},L_{2}(Q)\bigr{)},$
(E.94)

where $Q$ is some probability measure.

###### Proof.

For any two positive numbers $\epsilon_{1}$ and $\epsilon_{2}$, let
$\mathcal{G}=\\{g_{1},\dots,g_{K}\\}\subseteq\mathcal{F}_{1}$ be an
$\varepsilon_{1}$-net for $\mathcal{F}_{1}$ and
$\mathcal{H}=\\{h_{1},\dots,h_{L}\\}\subseteq\mathcal{F}_{2}$ be an
$\varepsilon_{2}$-net for $\mathcal{F}_{2}$, where
$K=N(\varepsilon_{1},\mathcal{F}_{1},L_{2}(Q))$ and
$L=N\bigl{(}\varepsilon_{2},\mathcal{F}_{2},L_{2}(Q))$. For any function
$f\in\mathcal{F}_{1}\times\mathcal{F}_{2}$ with $f=f_{1}f_{2}$, where
$f_{1}\in\mathcal{F}_{1}$ and $f_{2}\in\mathcal{F}_{2}$, let $\widetilde{g}$
and $\widetilde{h}$ be the closest element in $\mathcal{G}$ and $\mathcal{H}$
to $f_{1}$ and $f_{2}$ respectively. Then,

$\displaystyle\|f-\widetilde{g}\widetilde{h}\|_{Q}$
$\displaystyle=\|f_{1}f_{2}-\widetilde{g}\widetilde{h}\|_{Q}$
$\displaystyle=\|(f_{1}-\widetilde{g})f_{2}+\widetilde{g}(f_{2}-\widetilde{h})\|_{Q}$
$\displaystyle\leq\|(f_{1}-\widetilde{g})f_{2}\|_{Q}+\|\widetilde{g}(f_{2}-\widetilde{h})\|_{Q}$
$\displaystyle\leq C_{2}\varepsilon_{1}+C_{1}\varepsilon_{2}$
$\displaystyle=\varepsilon,\text{ when we take
}\varepsilon_{1}=\varepsilon/2C_{2}\text{ and
}\varepsilon_{2}=\varepsilon/2C_{1}.$

This way $\\{g_{k}h_{l}:1\leq k\leq K,1\leq l\leq L\\}$ is an
$\varepsilon$-net of $\mathcal{F}_{1}\times\mathcal{F}_{2}$ and the inequality
(E.94) follows. ∎

##  S.10 Some more simulation results

###  S.10.1 Absolute residual score

Figure A.1 and A.2 contain results comparing our doubly robust prediction
method with different splits of data, i.e. we are comparing doubly robust
prediction with full data described in Algorithm 2 and split data (Algorithm
1) with two splits and three splits, and weighted conformal prediction, all
under the absolute residual score as described at the start of Section 7. It
can be seen that DRP performs similarly across different splits and they all
outperform WCP.

Figure A.1: Coverage and width of Double Robust Prediction (DRP) with full
data, two splits, three splits and Weighted Conformal Prediction (WCP) on
synthetic data with the absolute residual score. The width is truncated at 10
for WCP. Figure A.2: Coverage and width of Doubly Robust Prediction (DRP) with
full data, two splits, three splits and Weighted Conformal Prediction (WCP) on
real data with the absolute residual score. The width is truncated at 50 for
WCP.

###  S.10.2 CQR score

Figure A.3 illustrates DRP and WCP’s performance on synthetic data (described
in Section 7.2) under the CQR score. It is found that all the methods achieve
the desired coverage with WCP being overly conservative, i.e. having infinity
width over half of the cases.

Figure A.3: Histograms of coverage and width of Doubly Robust Prediction (DRP)
and Weighted Conformal Prediction (WCP) on synthetic data through CQR score,
where the width is truncated at 300 for WCP.

##  S.11 Proof of Theorem 7

###### Proof of Theorem 7.

By definition of the odds ratio function, we have the following expressions:

$\displaystyle\log\frac{\mathbb{P}(T=0|X=x,Y=y)}{\mathbb{P}(T=1|X=x,Y=y)}=\log\frac{\mathbb{P}(T=0|X=x,Y=0)}{\mathbb{P}(T=1|X=x,Y=0)}+\gamma^{\star}(x,y);$
(E.95)

It follows from Bayes’ rule that

$\displaystyle
f(R|X,T=1)=f(R|X,T=0)\frac{\exp^{-\gamma^{\star}(X,Y)}}{\mathbb{E}\bigl{\\{}\exp^{-\gamma^{\star}(X,Y)|T=0,X\bigr{\\}}}}.$

And this leads to

$\displaystyle\mathbb{E}\Bigl{(}\mathbbm{1}\\{R\leq\theta\\}|X,T=1\Bigr{)}=\frac{\mathbb{E}[\mathbbm{1}\\{R\leq\theta\\}e^{-\gamma^{\star}(X,Y)}\big{|}X,T=0]}{\mathbb{E}[e^{-\gamma^{\star}(X,Y)}\big{|}X,T=0]}.$
(E.96)

  * •

If $\eta(x)$ is correct: This would imply the density ratio is correct, i.e.

$\exp\bigl{\\{}-\eta(x)-\gamma^{\star}(x,y)\bigr{\\}}=\mathbb{P}(T=1|X=x,Y=y)/\mathbb{P}(T=0|X=x,Y=y),$

then we have

$\mathbb{E}\left[\mathbbm{1}\\{T=0\\}\exp\bigl{\\{}-\eta(X)-\gamma^{\star}(X,Y)\bigr{\\}}\big{|}X=x,Y\right]=\mathbb{P}(T=0|X=x,Y)\frac{\mathbb{P}(T=1|X=x,Y)}{\mathbb{P}(T=0|X=x,Y)}=\mathbb{P}(T=1|X=x,Y).$

This implies that

$\displaystyle\mathbb{E}\left[\mathbbm{1}\\{T=0\\}\exp\bigl{\\{}-\eta(X)-\gamma^{\star}(X,Y)\bigr{\\}}\big{\\{}\mathbbm{1}\\{R\leq\theta\\}-m(\theta,X)\big{\\}}\big{|}\theta\right]$
$\displaystyle\quad=\mathbb{E}\Bigl{[}\mathbb{P}(T=1|X,Y)\big{\\{}\mathbbm{1}\\{R\leq\theta\\}-m(\theta,X)\big{\\}}\big{|}\theta\Bigr{]}$
$\displaystyle\quad=\mathbb{E}\Bigl{[}\mathbb{P}(T=1|X,Y)\bigl{\\{}\mathbb{P}(R\leq\theta|X)-m(\theta,X)\bigr{\\}}\big{|}\theta\Bigr{]}.$

Similarly,

$\mathbb{E}\Bigl{[}\mathbbm{1}\\{T=1\\}\left\\{m(\theta,X)-(1-\alpha)\right\\}\big{|}\theta\Bigr{]}=\mathbb{E}\Bigl{[}\mathbb{P}(T=1|X,Y)\bigl{\\{}m(\theta,X)-(1-\alpha)\bigr{\\}}\big{|}\theta\Bigr{]}.$

Hence, if $\exp\bigl{\\{}-\eta(x)-\gamma^{\star}(x,y)\bigr{\\}}$ is the true
density ratio, then

$\displaystyle\mathbb{E}[\mathrm{IF}(\theta,X,Y,R,T;\eta^{\star},m,\gamma^{\star})]$
$\displaystyle=\mathbb{E}\Bigl{[}\mathbb{P}(T=1|X,Y)\bigl{\\{}\mathbb{P}(R\leq\theta|X,\theta)-(1-\alpha)\bigr{\\}}\big{|}\theta\Bigr{]}$
$\displaystyle=\mathbb{E}\Bigl{[}\mathbb{P}(T=1|X,Y)\bigl{\\{}\mathbb{P}(R\leq\theta|X,\theta)\big{|}\theta\Bigr{]}-(1-\alpha)\mathbb{P}(T=1)$
$\displaystyle=\mathbb{E}\Bigl{[}\mathbb{E}(\mathbbm{1}\\{T=1\\}|X,Y)\mathbbm{1}\\{R\leq\theta\\}\big{|}\theta\Bigr{]}-(1-\alpha)\mathbb{P}(T=1)$
$\displaystyle=\mathbb{E}\Bigl{[}\mathbb{E}(1\\{T=1\\}\mathbbm{1}\\{R\leq\theta\\}|X,Y,\theta)\Bigr{]}-(1-\alpha)\mathbb{P}(T=1)$
$\displaystyle=\mathbb{E}\Bigl{[}1\\{T=1\\}\mathbbm{1}\\{R\leq\theta\\}\big{|}\theta\Bigr{]}-(1-\alpha)\mathbb{P}(T=1)$
$\displaystyle=\mathbb{P}(T=1)\Big{\\{}\mathbb{E}\bigl{[}\mathbbm{1}\\{R\leq\theta\\}|T=1,\theta\bigr{]}-(1-\alpha)\Bigr{\\}}.$

Therefore,

$\displaystyle\mathbb{E}[\mathrm{IF}(r_{\alpha},X,Y,R,T;\eta^{\star},m,\gamma^{\star})]=0.$

  * •

If $m$ is correct: In this case,

$\displaystyle\mathbb{E}\left[\mathbbm{1}\\{T=0\\}\exp\bigl{\\{}-\eta(X)-\gamma^{\star}(X,Y)\bigr{\\}}\big{\\{}\mathbbm{1}\\{R\leq\theta\\}-m(\theta,X)\big{\\}}\right]$
$\displaystyle\quad=\mathbb{E}\Bigl{\\{}\exp\\{-\eta(X)\\}\mathbb{E}\Bigl{[}\mathbbm{1}(R\leq\theta)\exp\\{-\gamma^{\star}(X,Y)\\}|T=0,X\Bigr{]}\Bigr{\\}}\mathbb{P}(T=0)$
$\displaystyle\qquad-\mathbb{E}\Bigl{\\{}\exp\\{-\eta(X)\\}\mathbb{P}(R\leq\theta|X,T=1)\mathbb{E}\Bigl{[}\exp\\{-\gamma^{\star}(X,Y)\\}|T=0,X\Bigr{]}\Bigr{\\}}\mathbb{P}(T=0)$
$\displaystyle\quad=0,\text{ by }\eqref{eq:sens-nuisance2}.$

Hence,

$\displaystyle\mathbb{E}[\mathrm{IF}(\theta,X,Y,R,T;\eta,m^{\star},\gamma^{\star})]$
$\displaystyle=\mathbb{E}\Bigl{[}\mathbbm{1}\\{T=1\\}\bigl{\\{}m^{\star}(\theta,X)-(1-\alpha)\bigr{\\}}\big{|}\theta\Bigr{]}$
$\displaystyle=\mathbb{E}\Bigl{[}\mathbbm{1}\\{T=1\\}\bigl{\\{}\mathbb{E}\bigl{(}\mathbbm{1}\\{R\leq\theta\\}|X,T=1,\theta\bigr{)}-(1-\alpha)\bigr{\\}}\big{|}\theta\Bigr{]}$
$\displaystyle=\mathbb{P}(T=1)\Bigl{\\{}\mathbb{E}\Bigl{[}\mathbb{E}\bigl{(}\mathbbm{1}\\{R\leq\theta\\}|X,T=1,\theta\bigr{)}|T=1,\theta\Bigr{]}-(1-\alpha)\Bigr{\\}}$
$\displaystyle=\mathbb{P}(T=1)\Bigl{\\{}\mathbb{E}\bigl{[}\mathbbm{1}\\{R\leq\theta\\}|T=1,\theta\bigr{]}-(1-\alpha)\Bigr{\\}}.$

Therefore,

$\displaystyle\mathbb{E}[\mathrm{IF}(r_{\alpha},X,Y,R,T;\eta,m^{\star},\gamma^{\star})]=0.$

Finally, by the double robustness property of $\mathrm{IF}(\cdots)$, it can be
easily verified that
$\partial_{t}\mathbb{E}[\mathrm{IF}(\cdots;\eta_{t},m^{\star},\gamma^{\star})]/\partial\eta_{t}$
and
$\partial_{t}\mathbb{E}[\mathrm{IF}(\cdots;\eta^{\star},m_{t},\gamma^{\star})]/\partial
m_{t}$ for regular parametric submodels $\eta_{t}$ and $m_{t}$ are both zero
at the truth. And this concludes our claim that $\mathrm{IF}(\cdots)$ is the
efficient influence function up to a proportionality constant. ∎


# Towards Good Practices for Missing Modality Robust Action Recognition

Sangmin Woo, Sumin Lee, Yeonju Park, Muhammad Adi Nugroho, Changick Kim

###### Abstract

Standard multi-modal models assume the use of the same modalities in training
and inference stages. However, in practice, the environment in which multi-
modal models operate may not satisfy such assumption. As such, their
performances degrade drastically if any modality is missing in the inference
stage. We ask: how can we train a model that is robust to missing modalities?
This paper seeks a set of good practices for multi-modal action recognition,
with a particular interest in circumstances where some modalities are not
available at an inference time. First, we study how to effectively regularize
the model during training (_e.g_., data augmentation). Second, we investigate
on fusion methods for robustness to missing modalities: we find that
transformer-based fusion shows better robustness for missing modality than
summation or concatenation. Third, we propose a simple modular network,
Actionmae, which learns missing modality predictive coding by randomly
dropping modality features and tries to reconstruct them with the remaining
modality features. Coupling these good practices, we build a model that is not
only effective in multi-modal action recognition but also robust to modality
missing. Our model achieves the state-of-the-arts on multiple benchmarks and
maintains competitive performances even in missing modality scenarios. Codes
are available at https://github.com/sangminwoo/ActionMAE.

##  1 Introduction

This study aims to answer the underlying question about multi-modal learning
for action recognition in practical situations: How can we train a model that
is robust to missing modalities? Typical multi-modal models assume complete
modalities in both training and inference phases (Bruce, Liu, and Chan 2021;
Bruce et al. 2022). In reality, however, the multi-modal system may be unable
to access particular modalities during the inference phase, despite being able
to access all modalities reliably during the training phase (see Figure 1).
There are a number of potential causes for such circumstances, including
malfunctioning sensors, high data acquisition costs, inaccessibility due to
security or privacy concerns, self-deficiencies, _etc_. This situation affects
the reliability, accuracy, and safety of the model in real-world applications.
Considering the autonomous driving situation, an error caused by insufficient
use of the sensors due to factors such as inclement weather would be life-
threatening. As such, the missing modality problem is critical when multi-
modal models are employed in practice.

Figure 1:  Action recognition with missing modality. Standard multi-modal action recognition assumes that the modalities used in the training stage are complete at inference time: (a) $\rightarrow$ (b). This paper addresses the action recognition problem in situations where such assumption is not established, _i.e_., when modalities are incomplete at inference time: (a) $\rightarrow$ (c). Our goal is to maintain performance in the absence of any input modality. ❄ indicates the weight-frozen.  Dataset |  Train modal∗ |  Test modal |  Acc.† |  $\triangle$ (%p)
---|---|---|---|---
NTU-RGB+D 60 |  R+D+I |  R+D+I |  93.3% |  -
|  R+D+I |  R+I |  59.4% |  $-$33.9
|  R+D+I |  R |  15.6% |  $-$77.7
NW-UCLA |  R+D |  R+D |  91.9% |  -
|  R+D |  R |  53.6% |  $-$38.3
|  R+D |  D |  70.3% |  $-$21.6
*R: RGB, D: depth, I: infrared. $\dagger$ Results based on sum fusion.
Table 1:  Multi-modal models suffer severely in missing modality scenarios.
The rows marked in gray use the same modalities in training and inference
stages, which are standard settings of multi-modal inference. We observe
significant performance drop when the modality is removed one by one from the
complete modality setup in inference stage.

Figure 2:  Towards missing modality robust action recognition. (i) Transformer
fusion is robust than sum fusion. (ii) In contrast to conventional approaches,
our Actionmae well preserves the multi-modal performance even with incomplete
modalities. All models are trained with R+D+I modalities and tested on NTU
RGB+D 60 (Shahroudy et al. 2016).

To address the missing modality scenarios, we require a robust model that
operates reliably even when certain modalities are unavailable in multi-modal
systems. However, conventional multi-modal models are learnt under the premise
that all modalities exist, so that they cannot achieve sufficient performances
when directly applied to missing modality settings. Table 1 shows how much the
conventional multi-modal models suffer from missing modality. Our goal is to
achieve on-par results as when all modalities are available, even if any of
the input modality is missing. We achieve this by designing a set of good
practices for missing modality scenarios in multi-modal action recognition
(see Figure 2): (i) As a preliminary step, we first demonstrate a recipe of
how to effectively build a multi-modal model with several training ingredients
such as architecture, data augmentation, and regularization. (ii) We observe
that the fusion method matters for robustness to missing modality. We compare
the three most widely used methods in the literature among the numerous
methods for multi-modal fusion: summation, concatenation, and transformer.
Among them, we find that transformer-based fusion method is the most robust
against missing modality. (iii) We propose Action Masked Auto Encoder
(Actionmae), a simple modular network that learns missing modality predictive
coding. The model with Actionmae is optimized under the two objectives. It
jointly learns to reconstruct the missing modality features and to predict the
correct action label during training. We can plug-and-play Actionmae on top of
standard space-time encoders. Despite its simplicity, Actionmae achieves a
win-win situation: it not only regularizes the multi-modal model to achieve
better accuracy in complete modality cases, but also maintains accuracy in
missing modality cases.

We report our results on four challenging action recognition benchmarks: NTU
RGB+D 60 (Shahroudy et al. 2016), NTU RGB+D 120 (Liu et al. 2019), NW-UCLA
(Wang et al. 2014), and UWA3D (Rahmani et al. 2016). We set new state-of-the-
art results in both complete and missing modality settings. Furthermore, we
observe some intriguing properties of Actionmae from experiments, including
the following: (i) Missing modality reconstruction yields a nontrivial and
meaningful self-supervision. As such, the results with reconstructed features
are superior than those with the original features extracted directly from
space-time encoder. (ii) Actionmae effectively regularizes the multi-modal
model, hence the model equipped with Actionmae achieves performance gain over
the vanilla model when using complete modalities. (iii) Moreover, Actionmae
alleviates the bias of the multi-modal model toward the dominant modality
(_i.e_., the most contributing modality to the learning objective). (iv) The
robustness of Actionmae is agnostic to the type or the number of missing
modalities. This suggests that it can preserve accuracy even in real-world
environments where we do not know which modality might be missing.

##  2 Preliminary: Multi-modal Action Classifier

We first study how to design a strong multi-modal action recognition model
with several design choices.

###  2.1 Architecture

The standard architecture of multi-modal action recognition is in a form of
space-time encoder followed by a fusion unit. We seek the optimal space and
time encoders for multi-modal action recognition with the fusion unit set to a
simple summation by default. Table 2 compares the accuracies of the simplest
combinations among the numerous candidates: R(2+1)D (Tran et al. 2018), ViT
(Dosovitskiy et al. 2020), ResNet (He et al. 2016), Transformer (Vaswani et
al. 2017).

Space encoder∗ |  Time encoder |  Fusion | Train $\rightarrow$ Test
---|---|---|---
|  |  |  RD $\rightarrow$ RD |  RD $\rightarrow$ R
R(2+1)D |  Sum |  89.2% |  41.2%
ViT-tiny/16† |  Transformer |  |  81.4% |  35.7%
ViT-small/16† |  Transformer |  |  83.0% |  36.3%
ResNet18 |  Transformer |  |  91.9% |  41.9%
ResNet34 |  Transformer |  |  92.6% |  42.5%
*Space encoders are Imagenet initialized.
$\dagger$ Re-implementations from (Wightman 2019).
Table 2:  ResNet34 + Transformer achieves the best accuracy in both full (RD
$\rightarrow$ RD) and missing modality (RD $\rightarrow$ R) cases. Experiments
are conducted on NTU-RGB+D 60.

We choose ResNet34 and Transformer as our space and time encoders,
respectively. The procedure of space-time encoding is formalized as follows.
We sample a sequence of frames $[f_{i}^{m}]_{i=1}^{T}$
($f_{i}^{m}\in\mathbb{R}^{C_{0}\times H\times W}$) from a video
$\mathcal{V}^{m}$, where $i$ indicates the frame index and $m$ indicates the
modality. In practice, we set $T=16$, $C_{0}=3$, $H=W=224$, and
$m\in{\rm\\{R,D,I\\}}$. A frame of modality $m$ is processed with a modality-
specific space encoder $\mathcal{E}^{m}$ followed by an average pooling:

$x_{i}^{m}=\texttt{avgpool}(\mathcal{E}^{m}(f_{i}^{m}))\,.$ (1)

The output feature is $x^{m}_{i}\in\mathbb{R}^{C}$, where $C=512$. The space
encoder operates on every frame and we obtain a sequence of features
$[x^{m}_{i}]_{i=1}^{T}$. Before feeding them to the time encoder, which is a
transformer encoder, a learnable class token cls is prepended to the sequence.
Its representation at the last layer of the encoder serves as the final
representation used by the classification layer (Devlin et al. 2018;
Dosovitskiy et al. 2020; Arnab et al. 2021).

$\mathcal{X}_{(0)}^{m}=[\texttt{cls}^{m},{x}_{1}^{m},...,{x}_{T}^{m}]+\texttt{pos}\,,$
(2)

where, pos is a fixed absolute embedding which represent the temporal
positions 111It is added to the tokens to maintain positional information
since the subsequent self-attention operations in the transformers are
permutation-invariant.. A sequence of tokens is then fed to a transformer
encoder with $L$ layers (we set $L=2$), where each layer $l$ is composed of
Multi-Head Self Attention (MHSA), Layer Normalization (LN), and Feed Forward
Network (FFN) 222We leave an original paper as a reference (Vaswani et al.
2017) for further details on transformer building blocks.:

$\mathcal{Y}_{(l)}^{m}=\textrm{MHSA}(\textrm{LN}(\mathcal{X}_{(l)}^{m}))+\mathcal{X}_{(l)}^{m}\,,$
(3)
$\mathcal{X}_{(l+1)}^{m}=\textrm{FFN}(\textrm{LN}(\mathcal{Y}_{(l)}^{m}))+\mathcal{Y}_{(l)}^{m}\,.$
(4)

The output at the final layer $\mathcal{X}_{(L)}^{m}\in\mathbb{R}^{(T+1)\times
C}$ holds the same dimension as that of the input $\mathcal{X}_{(0)}^{m}$. At
last, we use the final classification token
$\texttt{cls}_{L}^{m}\in\mathbb{R}^{C}$ to classify the action classes with
the fully-connected (FC) layer.

Variant | Train$\rightarrow$Test
---|---
|  RDI $\rightarrow$ RDI |  RD $\rightarrow$ R
Center crop, horizontal flip |  85.9 |  34.8
\+ Imagenet initialization |  89.2 ( $+$3.3) |  39.5 ( $+$4.7)
\+ Shifted center crop |  90.5 ( $+$1.3) |  40.3 ( $+$0.8)
\+ Temporal random sampling |  91.0 ( $+$0.5) |  41.4 ( $+$1.1)
\+ Color jitter (RGB) |  92.2 ( $+$1.2) |  42.0 ( $+$0.6)
\+ Label smoothing ($\alpha=0.1$) |  93.3 ( $+$1.1) |  42.5 ( $+$0.5)
\- Color jitter (0pt) |  91.5 ( $-$1.8) |  -
\- Dropout ($p=0.1$) |  92.6 ( $-$0.7) |  42.5
Table 3:  Progressively adding regularizations helps much in both full and missing modality cases. However, applying color jittering on depth maps and using dropout rather reduced the performance in full modality case. Experiments are conducted on NTU RGB+D 60 with the sum fusion model.  Fusion |  Train modal |  Test modal $\uparrow$ |  $\triangle$(%p) $\uparrow$ |  $|{\bf\bar{\rm\triangle}}|$$\downarrow$
---|---|---|---|---
Sum |  R+D+I |  R+D+I (93.3%) |  - |  34.8
|  |  R+D (89.0%) |  $-$4.3 |
|  |  R+I (59.4%) |  $-$33.9 |
|  |  D+I (89.2%) |  $-$4.1 |
|  |  R (15.6%) |  $-$77.7 |
|  |  D (85.9%) |  $-$7.4 |
|  |  I (12.1%) |  $-$81.2 |
Concat |  R+D+I |  R+D+I (93.2%) |  - |  33.5
|  |  R+D (91.4%) |  $-$1.8 |
|  |  R+I (75.4%) |  $-$17.8 |
|  |  D+I (18.6%) |  $-$74.6 |
|  |  R (53.1%) |  $-$40.1 |
|  |  D (88.2%) |  $-$5.0 |
|  |  I (31.7%) |  $-$61.5 |
Transformer |  R+D+I |  R+D+I (92.6%) |  - |  21.0
|  |  R+D (91.6%) |  $-$1.0 |
|  |  R+I (66.6%) |  $-$26.0 |
|  |  D+I (90.4%) |  $-$2.2 |
|  |  R (56.6%) |  $-$36.0 |
|  |  D (88.9%) |  $-$3.7 |
|  |  I (35.4%) |  $-$57.2 |
Table 4:  Transformer is a missing modality robust fusion unit. Experiments
are conducted on NTU RGB+D 60. $\triangle$ is defined as accuracy discrepancy
between missing modality and full modality cases
($\triangle=\textrm{Acc}_{miss}-\textrm{Acc}_{full}$). While all three fusion
methods show comparable performance when test modalities are complete, they
show different aspects in missing modality inference. The rows marked in gray
are complete modality cases.  $\uparrow$/$\downarrow$: The higher/lower the
better.

Figure 3:  Actionmae framework. (i) Multi-modal training: We first obtain
modality-specific features via their respective encoding stream. Then, we
randomly drop $K$ (here, $K=1$) input tokens from $N$ (=3) modality tokens,
where $K$ ($0\leq K<N$) varies at every iteration. Actionmae learns to
reconstruct missing modality tokens using the remaining $N-K$ modality tokens.
During autoencoding, the memory token, which is also provided as an input,
serves as global memory across varying inputs. The original and reconstructed
tokens are reorganized as an input sequence to the fusion unit. The succeeding
processes follow the standard fusion-and-predict procedure. The overall
training objective is defined in Equation 6. (ii) Missing modality inference:
We drop $K$ (=2) tokens (here, 0pt, IR) and predict the action class by
reconstructing $K$ tokens with remaining $N-K$ (=1) token (RGB). We note that
Actionmae is agnostic to the types or the number of missing modalities.

###  2.2 Tips and Tricks for Training

In our preliminary experiments, we noticed that our models are prone to
overfit, highlighting the necessity for a bag of tricks to train our models
effectively. In order to see the effect of augmentations and regularizations,
we ablate several strategies in Table 3. By default, we employ center crop and
horizontal flip for all modality data, and progressively add more strategies:
Imagenet initialization on space encoder is highly beneficial, implying that
it eases the subsequent optimization of time encoder. It has relatively large
impact in the missing modality case compared to the full modality case,
_i.e_., the strong inductive bias provided by ImageNet initialization can have
a particularly positive effect in the absence of depth information. Instead of
the vanilla center crop, we center crop the larger region ($\times$1.2) and
then random crop ($\times$1) within it. Instead of uniformly sampling frames
from video, we randomly sample frames and arrange them in temporal order.
Color jitter increases divergence of RGB distribution, which has a beneficial
effect. label smoothing (Szegedy et al. 2016) adjusts the label distribution
of the ground truth to encourage the model to produce less confident
predictions. Generally, regularizing the model or increasing the data
diversity were beneficial in both full and missing modality cases. However,
some methods were detrimental in full modality case: Color jitter on depth map
rather harms the consistency of 3D structural information. Dropout (Srivastava
et al. 2014) makes optimization rather difficult. In missing modality case,
Dropout did not have a detrimental effect, but it also did not help regularize
the model. Overall, we achieve a substantial improvement of 7.4%p (7.7%p for
missing modality case) on NTU RGB+D 60. Note that no spatial or temporal
augmentations are used in testing.

###  2.3 Fusion

In Table 4, we examine exemplary fusion methods in terms of multi-modal
inference and missing modality inference. We take three simple yet effective
fusion methods that are commonly used in various multi-modal models:
summation, concatenation, and transformer.

  * •

Sum. Each modality passes through its respective FC layer and then is all
summed up: $\sum\limits_{m}\textrm{FC}^{m}(\texttt{cls}_{L}^{m})\,.$

  * •

Concat. Modalities are channel-wise concatenated (${{\parallel}}$) and passed
through a single FC layer:
$\textrm{FC}(\underset{m}{{{\parallel}}}\texttt{cls}_{L}^{m})\,.$ 333In the
case of concatenation, uni-modal predictions are approximated by splitting the
weight matrix of FC layer into sub-matrices and dividing the bias by the
number of modalities.

  * •

Transformer. Extra class token is prepended to a sequence of modality tokens
$[\texttt{cls},\texttt{cls}_{L}^{R},\texttt{cls}_{L}^{D},\texttt{cls}_{L}^{I}]$,
which is then passed through the transformer (Eqs. 3 and 4). The class token
of the final transformer layer is subsequently passed through the FC layer.

The test accuracies with complete modalities (rows makred in gray) are not
much different between three fusion methods. Among them, the transformer-based
fusion method is the most inferior (lags behind sum by 0.7%p). To our
surprise, however, we observe the opposite tendency in a missing modality
situation. The transformer-based fusion is the most robust to the missing
modalities: it reduces the mean accuracy discrepancy
($|{\bf\bar{\rm\triangle}}|$) by 13.8$\%$p compared to sum fusion.
Nevertheless, we notice the bias in all three fusion methods. They
specifically works well with a certain modality (_e.g_., depth), but poorly
with other modalities (_e.g_., RGB, IR) under the situation of missing
modality.

##  3 Method: ActionMAE

To address the missing modality scenario in action recognition problem, we
propose Actionmae that learns missing modality predictive coding as
illustrated in Figure 3.

Our Actionmae is a simple modular autoencoder that reconstructs the missing
modalities based on partial observation of the remaining modalities. Like
generic autoencoders, our solution is made up of an encoder that maps the
observed modalities to a latent representation and a decoder that reconstructs
the missing modalities from the latent representation. The encoder and decoder
adopted in our Actionmae are simple transformer encoders (Vaswani et al.
2017). By design, it can be plugged into a typical multi-modal action
classifier with a fusion unit. We plug-and-play Actionmae on top of the model
explored in preliminary experiments: ResNet34 + Transformer. During training,
the model equipped with Actionmae simultaneously learns missing modality
information and predicts the correct action class based on these.

###  3.1 Missing Modality Predictive Coding

We define a missing modality scenario as a case in which $N$ modalities are
accessible during training, but $K$ modalities are missing during inference,
_i.e_., only $N-K$ modalities are accessible, where $1\leq K<N$. We use three
modalities in practice, RGB, Depth, and IR (as such, $N=3$), but the number or
type of modalities are not limited to these. From $N$ modality tokens
$[z^{m}]_{m=1}^{N}$ (equivalent to $[\texttt{cls}_{L}^{m}]_{m=1}^{N}$ of Eq.
4), we randomly drop $K$ tokens, _i.e_., only leaving $N-K$ tokens, where
$0\leq K<N$ (if $K=0$, we do not drop any of modality tokens). In addition to
remaining $N-K$ tokens, the Actionmae encoder takes extra learnable token,
_i.e_., memory token, that acts as global memory across varying inputs. It is
worth noting that the input sequence to Actionmae encoder continues to vary
with each iteration. We first obtain a sequence of latent representations via
the encoder. Then, the $K$ dummy tokens are inserted into the sequence at the
position where the modality tokens are missing, and the resulting sequence is
fed to the decoder. As the dummy tokens pass through the decoder, they are
forced to mimic the dropped tokens. The original modality tokens are
reorganized with the reconstructed tokens and passed to the subsequent fusion
unit and prediction layer in turns. Notice that the memory token is used only
for encoding-decoding process, not for prediction.

###  3.2 Training & Inference

Overall training of the model with Actionmae is governed by the two loss
functions: standard classification loss (_i.e_., cross-entropy loss)
$\mathcal{L}_{\rm cls}$ and reconstruction loss $\mathcal{L}_{\rm rec}$. The
decoder aims to convert dummy tokens into original modality tokens. We define
the indices of missing tokens as $j_{1},...,j_{K}$. The reconstruction loss
computes the mean squared error (MSE) between the reconstructed
$[\hat{z}^{j_{1}},...,\hat{z}^{j_{K}}]$ and missing tokens
$[z^{j_{1}},...,z^{j_{K}}]$ in the feature space:

$\mathcal{L}_{rec}=\frac{1}{K}\sum_{i=1}^{K}(z^{j_{i}}-\hat{z}^{j_{i}})^{2}\,.$
(5)

Note that only memory token and dummy tokens are affected by the
reconstruction loss (_i.e_., reconstruction loss is not computed for the
remaining tokens) during training.

Our training objective is jointly optimized upon the two loss functions:

$\mathcal{L}=\lambda_{\rm cls}\mathcal{L}_{\rm cls}+\lambda_{\rm
rec}\mathcal{L}_{rec}\,.$ (6)

In our experiments, we empirically find that balancing hyperparameters
$\lambda_{\rm cls}=\lambda_{\rm rec}=1$ works well.

At inference time, $K$ modalities are dropped from the input stage, and
remaining $N-K$ modalities are encoded by their respective modality encoders.
Actionmae takes only the memory token and $N-K$ encoded modality tokens, and
reconstructs the missing tokens. We progressively drop the modalities to see
whether Actionmae is robust under all circumstances, regardless of which
modality or how many modalities are lost. Experiments demonstrate that our
Actionmae is quite robust to any cases of modality missing.

##  4 Experiments

Model |  Train modal |  Test modal $\uparrow$ |  $\triangle$(%p) $\uparrow$ |  $|{\bf\bar{\rm\triangle}}|$$\downarrow$
---|---|---|---|---
Sum |  R+D+I |  R+D+I (93.3%) |  - |  34.8
Concat |  |  R+D+I (93.2%) |  - |  33.5
Transformer |  |  R+D+I (92.6%) |  - |  21.0
Transformer w/ Actionmae |  R+D+I |  R+D+I (93.0%) |  - |  8.3
|  |  R+D (92.6%) |  $-$0.4 |
|  |  R+I (87.6%) |  $-$5.4 |
|  |  D+I (91.9%) |  $-$1.1 |
|  |  R (83.4%) |  $-$9.6 |
|  |  D (89.7%) |  $-$3.3 |
|  |  I (81.7%) |  $-$11.3 |
Table 5:  Actionmae is a missing modality robust learner. Experiments are
conducted on NTU RGB+D 60. The rows marked in gray are complete modality
cases.  $\uparrow$/$\downarrow$: The higher/lower the better.

|  Model |  Train modal∗ |  Test modal |  Universal† |  NTU60 |  NTU120 |  NWUCLA |  UWA3D
---|---|---|---|---|---|---|---|---
Baseline |  Uni-modal (/w Transformer-fusion) |  R |  R |  ✗ |  86.6% |  84.2% |  85.0% |  73.1%
|  |  D |  D |  ✗ |  92.0% |  88.2% |  92.7% |  81.2%
|  |  I |  I |  ✗ |  85.5% |  85.4% |  - |  -
|  Multi-modal (/w Transformer-fusion) |  R+D |  R+D |  ✗ |  92.3% |  91.7% |  90.9% |  82.5%
|  |  R+D+I |  R+D+I |  ✗ |  92.6% |  92.1% |  - |  -
Complete |  3DFCNN (Sanchez-Caballero et al. 2022) |  D |  D |  ✗ |  78.1% |  - |  83.6% |  66.6%
|  Ren _et al_. (2021) |  R+D |  R+D |  ✗ |  89.7% |  - |  - |  -
|  Deep Bilinear (Hu et al. 2018) |  R+D+S |  R+D+S |  ✗ |  85.4% |  - |  - |  -
|  DMCL-complete (Garcia et al. 2019) |  R+D+F |  R+D+F |  ✗ |  87.3% |  89.7% |  93.9% |  89.8%
Missing |  Hoffman _et al_. (2016) |  R+D |  R |  ✗ |  - |  - |  83.2% |  66.7%
|  Garcia _et al_. (2018) |  R+D |  R |  ✗ |  73.1% |  - |  86.7% |  73.2%
|  ADMD (Garcia, Morerio, and Murino 2019) |  R+D |  R |  ✗ |  73.4% |  - |  93.6% |  78.4%
|  Luo _et al_. (2018) |  R+D+F+S |  R |  ✗ |  89.5% |  - |  - |  -
|  |  |  D |  ✗ |  87.5% |  - |  - |  -
|  DMCL (Garcia et al. 2019) |  R+D+F |  R |  ✗ |  83.6% |  84.3% |  93.6% |  78.4%
|  |  |  D |  ✗ |  80.6% |  82.2% |  83.3% |  81.9%
Ours |  Actionmae (R+D+I) |  R+D+I |  R+D+I |  ✓ |  93.0% |  92.3% |  - |  -
|  |  |  R+D |  ✓ |  92.6% |  91.7% |  - |  -
|  |  |  R+I |  ✓ |  87.6% |  84.8% |  - |  -
|  |  |  D+I |  ✓ |  91.9% |  91.2% |  - |  -
|  |  |  R |  ✓ |  83.4% |  83.1% |  - |  -
|  |  |  D |  ✓ |  90.1% |  86.0% |  - |  -
|  |  |  I |  ✓ |  81.7% |  81.6% |  - |  -
|  Actionmae (R+D) |  R+D |  R+D |  ✓ |  92.5% |  91.5% |  91.0% |  79.8%
|  |  |  R |  ✓ |  84.5% |  84.7% |  84.2% |  70.4%
|  |  |  D |  ✓ |  90.5% |  87.0% |  88.2% |  77.6%
*R: RGB, D: depth, I: infrared, F: optical flow, S: skeleton. $\dagger$ preserves accuracy no matter what kind and how many modalities are missing.

Table 6:  Comparative study on multiple action recognition benchmarks: NTU60
(Shahroudy et al. 2016), NTU120 (Liu et al. 2019), NWUCLA (Wang et al. 2014),
and UWA3D (Rahmani et al. 2016). Here, we report top-1 accuracy. The rows
marked in gray are complete modality cases of ours. By design, our Actionmae
is robust to missing modalities of any type and number.

###  4.1 Implementation

#### Inputs.

The input of each modality network is a 16 frame clip of the corresponding
modality, and their input size is 224$\times$224 pixels. In the spatial
domain, random horizontal flip and shifted center crop are performed on all
modality inputs. RGB frames are applied with color jittering, and the
1-channel depth frames are converted into 3-channels based on a JET color map
with 256 scales. In the temporal domain, we randomly sampled 16 frames from
the raw video, and arrange them in temporal order. At test time, each clip
consists of 16 frames that are uniformly sampled, center cropped, and no data
augmentation techniques are applied.

#### Settings.

We used AdamW optimizer (Loshchilov and Hutter 2017) with an initial learning
rate of $10^{-4}$ and weight decay of $10^{-4}$ for a batch size of 32. The
learning rate is linearly decayed by a factor of 10 every 30 epochs. We used
cross-entropy loss with label smoothing of factor 0.1 (Szegedy et al. 2016).
All space encoders are initialized with imagenet pre-trained weights (Deng et
al. 2009). Actionmae features a symmetrical design with two encoder and
decoder layers.

###  4.2 Datasets

NTU RGB+D 60 (NTU60) (Shahroudy et al. 2016) includes 60 classes and 56,880
video samples. The actions are in three categories: daily actions, mutual
actions, and medical conditions. The dataset is captured by three Kinect V2
cameras concurrently, which contains RGB videos (spatial size: $480\times
270$), depth map sequences ($512\times 424$), infrared ($512\times 424$), and
3D skeletal videos ($25$ joints).

NTU RGB+D 120 (NTU120) (Liu et al. 2019) extends NTU RGB+D 60 by adding an
additional 60 classes and 57,600 video samples, bringing the total to 120
classes and 114,480 samples. It comprises 106 subjects across 155 different
viewpoints and 96 background scenes. Following the convention (Liu et al.
2019), we evaluate our model using the cross-subject protocol in both NTU60
and NTU 120.

NW-UCLA (Wang et al. 2014) contains 10 action classes and 1,475 video samples
collected from 10 different subjects. RGB ($640\times 480$), depth ($320\times
240$), and skeletal videos ($20$ joints) are captured simultaneously from 3
different viewpoints with Kinetic V1. We followed the cross-view protocol
suggested in (Wang et al. 2014), using two views (V1, V2) for training and the
remaining (V3) for testing.

UWA3DII (Rahmani et al. 2016) consists of 30 classes and 1,070 videos: RGB
($320\times 240$), depth ($320\times 240$), and skeleton ($15$ joints). The
actions are performed by 10 subjects in 4 different views: front (V1), left
side (V2), right side (V3), and top view (V4). We used the last two views (V3,
V4) for training and the remaining views (V1, V2) for testing.

###  4.3 ActionMAE

In Table 5, we examined the effect of Actionmae by comparing the model with
and without Actionmae. In preliminary experiments, we observed that complete
modality performance was rather lower when using transformer than much simpler
methods such as sum or concatenation. We hypothesize that this is due to the
lack of inductive bias in transformer, which makes it susceptible to
overfitting. When we attach Actionmae on a transformer-fusion model, the
random drop operation yields nontrivial self-supervisory task, which
effectively regularizes the model. This results in $0.4\%$p performance
improvement under complete modality setting. Generally, Actionmae shows good
robustness to any combination of missing modalities. More importantly, the
missing modality performances, which were particularly poor in accuracy, are
significantly improved upon vanilla transformer baseline (_e.g_., IR:
$35.4\%\rightarrow 81.7\%$). This implies that Actionmae also has a debiasing
effect on the dominant modality (_e.g_., depth). Overall, Actionmae reduces
the mean accuracy discrepancy by $\sim 2.5\times$ compared to the transformer-
fusion baseline, and by $\sim 4\times$ compared to the sum-fusion baseline.

###  4.4 Comparative Study

In Table 6, we benchmark our approach against competitive baselines on
multiple action recognition datasets.

####  Vs. uni-/multi-modal baselines.

We begin by comparing Actionmae with transformer-fusion baselines. With
complete modalities, Actionmae mostly improves performance upon multi-modal
baselines as randomized drop of modalities during training gives a
regularization effect on the overall network analogous to dropout (Srivastava
et al. 2014). While our model is also fairly robust to missing modalities in
both R+D+I and R+D cases, uni-modal models outperform Actionmae when tested
with a single modality. However, in the missing modality situation evaluated
with R+D, our model (92.6%) slightly improves accuracy compared to the multi-
modal baseline (92.3%).

####  Vs. complete modality counterparts.

We compare our approach with recent action recognition methods. These methods
do not drop any modality at the test time (_i.e_., complete modality
inference), yet our method significantly outperforms all listed methods even
in missing modality scenarios.

case |  RD |  R |  D
---|---|---|---
w/o mem |  87.3 |  74.8 |  83.2
w/ mem |  92.5 |  84.5 |  90.5
(a)

$\mathcal{L}_{rec}$:$\mathcal{L}_{cls}$ |  RD |  R |  D
---|---|---|---
1:10 |  91.9 |  83.0 |  90.0
1:1 |  92.5 |  84.5 |  90.5
10:1 |  91.0 |  81.6 |  88.7
(b)

case |  RD |  R |  D
---|---|---|---
w/o pos |  87.2 |  76.3 |  84.7
w/ pos |  92.5 |  84.5 |  90.5
(c)

case |  RD |  R |  D
---|---|---|---
No PT |  92.5 |  84.5 |  90.5
PT (S) |  89.1 |  82.0 |  86.4
PT (S+T) |  88.5 |  81.9 |  86.6

(d)

Table 7:  Ablation experiments on NTU RGB+D 60 with Actionmae (R+D). The
entries marked in gray are our settings.

####  Vs. missing modality counterparts.

Next, we compare our approach with missing modality counterparts. Generally,
as more modalities are used in the training stage, they show better
performance in missing modality situations. In both R+D+I and R+D cases, the
accuracy of Actionmae decreases as more modalities are dropped, albeit shows
competitive performance. When we follow the most common setup in the
literature (train: R+D, test: R or D), Actionmae sets new state-of-the-art
results.

####  Actionmae (R+D+I _vs_. R+D).

In the case of R+D+I, if just one modality is missing at test time (_e.g_.,
R+D), there is a minimal loss of accuracy ($-$0.4%p). However, in the absence
of two modalities (_e.g_., R), mapping from latent representation becomes more
complex as the number of values to be restored increases, thus drops more
($-$9.6%p). On the other hand, even evaluated with the same modality (_e.g_.,
D), R+D performs better than R+D+I since the mapping is simpler.

###  4.5 Ablative Study

From Table 7, we observed several intriguing properties of Actionmae by
ablating it.

#### Memory token.

Memory token contributes significantly to performance improvement, especially
for the missing modality cases: 9.7%p gain for R-only case and 7.3%p gain for
D-only case. We understand this as memory token learns meaningful correlation
between varying inputs and models pair-wise relationships between itself and
modality tokens.

#### Loss hyperparameters.

We investigated a good balance between reconstruction $\mathcal{L}_{rec}$ and
classification loss $\mathcal{L}_{cls}$. From an exploration-exploitation
perspective, when the overall objective is leaned to classification (_i.e_.,
$\mathcal{L}_{rec}:\mathcal{L}_{cls}=1:10$), Actionmae underutilizes the
features extracted from the space-time encoder, resulting in poor
‘exploitation’ of learned information. On the other hand, if the objective is
leaned to reconstruction (_i.e_., 10:1), Actionmae loses the chance of
‘exploration’ to generate more class-discriminative features. We found a ratio
of 1:1 properly balances the two losses.

#### Position embedding.

We observed a significant performance improvements with position embedding
(pos). From the results, we hypothesize that position embedding serves as a
reference for determining which modalities needs to be reconstructed in
Actionmae, _i.e_., position embedding provides guidance on what to
reconstruct.

#### Pre-training.

We examined whether pre-training the space-time encoder would ease the
optimization or rather negatively impact the training of Actionmae. Here, PT
(S) indicates space encoder pre-training, and PT (S+T) indicates pre-training
both space and time encoders. Contrary to our expectations, pre-training any
encoders ahead of Actionmae diminishes accuracy. Even, the performance
degraded when more layers were pre-trained. This suggests that Actionmae can
be effectively learned through self-supervisory reconstruction of diverse
inputs. However, if the space-time encoder is pre-trained, the input will
remain almost constant; hence, Actionmae will likely to find a trivial
solution. Therefore, we only performed imagenet initialization on the space
encoder and trained all networks from scratch.

##  5 Related Work

Multi-modality generally yields reliable results since different modalities
provide complementary information (Huang et al. 2021). For example, RGB
provides rich appearance information, depth provides 3D geometrical structure,
and IR is robust to illumination variation. The missing modality setup assumes
certain modalities available during training are unavailable at test time. As
the typical multi-modal models suffer in such scenarios, there was a large
body of studies to address the missing modality problem (Vapnik and Vashist
2009; Pechyony and Vapnik 2010; Vapnik, Izmailov et al. 2015; Lopez-Paz et al.
2015; Ma et al. 2022; Zhao, Li, and Jin 2021; Alayrac et al. 2020; Ma et al.
2021)

In the context of action recognition (Tran et al. 2015; Wang et al. 2016;
Carreira and Zisserman 2017; Feichtenhofer et al. 2019), there are handful of
studies towards this end (Hoffman, Gupta, and Darrell 2016; Luo et al. 2018;
Garcia, Morerio, and Murino 2018, 2019; Garcia et al. 2019; Stroud et al.
2020), all of which follow transfer learning scheme via knowledge distillation
(Hinton et al. 2015). We rather take inspiration from recent successes of
masked autoencoders (Devlin et al. 2018; He et al. 2022; Feichtenhofer et al.
2022; Bachmann et al. 2022), whose core idea is to remove a portion of the
data and learn to predict the removed data, and show that masked autoencoding
works better than the knowledge distillation approaches. In addition to
accuracy, there are several advantages of Actionmae over the knowledge
distillation approaches: (i) A separate teacher network training, which is a
prerequisite for knowledge distillation, is unnecessary. Therefore, the
training can be completed in an end-to-end manner with only a single step.
(ii) We do not need method-specific architecture. Actionmae can be plugged
into any type of pre-existing space-time encoders. (iii) By design, Actionmae
is universal to the type or the number of missing modalities.

##  6 Discussion

In this paper, we have answered the following questions regarding the missing
modality scenarios in multi-modal action recognition problem: (i) How to train
a strong multi-modal model? We sought the good practices from three
perspectives (see Tables 2 and 3): architecture, data augmentation, and
regularization. (ii) Which fusion should we choose for missing modality action
recognition? Of the three popular choices for fusion (_e.g_., sum,
concatenation, transformer), transformer worked the best (see Table 4), albeit
still far from enough. (iii) Is there more effective way to solve missing
modality problem? We showed that Actionmae is an effective strategy for
missing modality action recognition via extensive experiments (see Tables 5, 6
and 7). In addition, it relieves bias against dominant modality, as well as
effectively regularizes multi-modal model. We hope that our findings will
provide insights into broader missing modality scenarios.

#### Limitations & future work.

There still remain unresolved issues: (i) The uni-modal baselines perform
better than the model equipped with Actionmae when evaluated with a single
modality in missing modality situations. (ii) Even with the same test modality
(_e.g_., R or D), the model with fewer training modalities (R+D) outperforms
the model with more training modalities (R+D+I). We posit these challenges
occur due to the difficulty of mapping from latent representation of
Actionmae. As the number of outputs exceeds the number of inputs, it yields a
sub-optimal solution. We mitigate this problem using memory token, yet more
investigation is needed to find a more effective way. Despite the limitations,
our idea is simple and can be naturally extended to other modalities, such as
vision, language, audio, _etc_. We leave this intriguing challenge to future
work.

## Acknowledgments

This work was supported by the Agency For Defense Development by the Korean
Government (UD190031RD).

## Appendix

##  Appendix A Background: Transformer

The standard transformer (Vaswani et al. 2017) includes multi-head self-
attention layers, residual connections, layer normalization, and feed forward
networks. As our model is built on the transformers, here we briefly explain
how the multi-head self-attention, the core component of transformer, work
internally.

Self-Attention (SA) is a popular yet strong mechanism for neural systems that
enables inputs to interact with each another (“self”) and find out which
element they should pay more attention to (“attention”). The attention
function is essentially a mapping between a query (Q) and a set of key-value
(K-V) pairs. The outputs are aggregations of these interactions and attention
scores. Formally, we compute a weighted sum over all values ${\bf V}$ for each
element in an input sequence ${\bf X}\in\mathbb{R}^{N\times d}$. The attention
weights $A_{ij}$ are pair-wise similarity between two elements of the sequence
and their respective query ${\bf Q}_{i}$ and key ${\bf K}_{j}$
representations.

$[{\bf Q,K,V}]={\bf X}{{\bf W}_{QKV}}+{\bf p},$ (7) $A={\rm
softmax}\left(\frac{{\rm\bf QK}^{T}}{\sqrt{D_{h}}}\right),$ (8) ${\rm SA}({\bf
X})=A{\bf V},$ (9)

where ${{\bf W}_{QKV}}\in\mathbb{R}^{d\times 3d_{h}}$ is projection matrix
that are used to generate different subspace representations of the query,
key, value matrices, and $A\in\mathbb{R}^{N\times N}$ is an attention matrix.
In practice, we add positional encoding ${\bf p}\in\mathbb{R}^{N\times d}$ to
embedded sequence as the transformer is inherently permutation-invariant
w.r.t. input sequence.

Multi-Head Self-Attention (MHSA) is an extension of self-attention where $k$
self-attentions (a.k.a., ”heads”) are performed in parallel, and followed by a
projection of their concatenated outputs. Here, $d_{h}$ is generally set to
$d/k$ to maintain the computation and the number of parameters constant when
$k$ varies.

${\rm MHSA({\bf x})}=[{\rm SA}_{1}({\bf x});{\rm SA}_{2}({\bf x});\cdots;{\rm
SA}_{k}({\bf x});]{{\bf W}_{O}},$ (10)

where [;] denotes channel-wise concatenation and ${{\bf
W}_{O}}\in\mathbb{R}^{k\cdot d_{h}\times d}$ denotes a projection matrix for
the multi-head outputs.

##  Appendix B More Details About Actionmae

Figure 4:  High-level overview of Actionmae encoder-decoder. Actionmae is a
kind of autoencoder that uses an encoder-decoder architecture. This figure
assumes a situation in which a blue modality token is dropped.  Figure 5:
Architectural details of Actionmae. Actionmae is built with two layers of
encoder and two layers of decoder Actionmae has symmetric design, with encoder
and decoder made up of two transformer encoder layers each. The fixed absolute
positional encoding (Vaswani et al. 2017) is added at the beginning of the
encoding or decoding operation. Here, we assume a situation in which the red
and yellow modality tokens are dropped.

Figure 4 shows a high-level overview of Actionmae encoder-decoder, and Figure
5 illustrates a more detailed architecture of Actionmae. Given a sequence of
$N$ modality tokens, $[z^{1},\cdots,z^{N}]$, $K$ tokens are randomly removed
such that $0\leq K<N$, _i.e_., do not remove any token if $K=0$ and leave at
least one token if $K=N-1$, resulting in $[z^{r_{1}},\cdots,z^{r_{N-K}}]$,
where $r_{1},\cdots,r_{N-K}$ are indices of $N-K$ remaining tokens. Note that
$K$ varies with each iteration. A memory token mem is prepended to the
sequence and the resulting sequence,
$[\texttt{mem},z^{r_{1}},\cdots,z^{r_{N-K}}]$, is fed to the Actionmae.
Actionmae, like all other autoencoders, comprises of an encoder and a decoder.
The encoder maps observed modalities to latent representations and the decoder
reconstructs the representations of missing modalities from the latent
representations. During this autoencoding process, the memory token also
serves as a global memory that learns pair-wise relationship between varying
inputs. Formally, the encoder maps an input sequence to latent
representations, $[\texttt{mem}_{e},z^{r_{1}}_{e},\cdots,z^{r_{N-K}}_{e}]$. We
use learnable token, dubbed as dummy token dum, that learns to mimic the
missing token. $K$ dummy tokens, each recovers corresponding missing token,
are inserted into the sequence at the position where the modality tokens were
removed:
$[\texttt{mem}_{e},z^{r_{1}}_{e},\texttt{dum}^{j_{1}},\cdots,\texttt{dum}^{j_{K}},z^{r_{N-K}}_{e}]$
(the positional indices of missing tokens are defined as
$j_{1},\cdots,j_{k}$). This sequence then passes through the Actionmae
decoder, and generates missing tokens: $[\hat{z}^{j-1},\cdots,\hat{z}^{K}]$.
Finally, a class token cls is prepended to reconstructed tokens,
$[\texttt{cls},\hat{z}^{j-1},\cdots,\hat{z}^{K}]$, and the resulting sequence
passes through the fusion unit followed by a prediction head.

In our framework, Actionmae learns to recover missing modality, which is then
used for action classification along with the remaining modality. By design,
unlike previous KD-based methods that require retraining when missing modality
is changed (when R+D $\rightarrow$ R is changed to R+D $\rightarrow$ D),
Actionmae can handle all types of missing modalities (_i.e_., universal) with
just a single training schedule (R+D $\rightarrow$ R or D).

During training, we randomly select modalities to drop (including no drop) at
every iteration:
R$\rightarrow$R+I$\rightarrow$D$\rightarrow$I$\rightarrow$$\cdots$. We trained
the whole framework including Actionmae in an end-to-end manner. During
testing, all testing samples of one or more modalities are dropped.

#### Architecture Specifications.

We set the default diemnsion of Actionmae to 512, and set the hidden dimension
to 2048 in feed forward networks consisting of two linear layers and GeLU
(Hendrycks and Gimpel 2016) activation. The number of encoder and decoder
layers of Actionmae is set to two each, and eight heads are used for self-
attention operations.

## References

  * Alayrac et al. (2020) Alayrac, J.-B.; Recasens, A.; Schneider, R.; Arandjelović, R.; Ramapuram, J.; De Fauw, J.; Smaira, L.; Dieleman, S.; and Zisserman, A. 2020.  Self-supervised multimodal versatile networks.  _Advances in Neural Information Processing Systems_ , 33: 25–37.
  * Arnab et al. (2021) Arnab, A.; Dehghani, M.; Heigold, G.; Sun, C.; Lučić, M.; and Schmid, C. 2021.  Vivit: A video vision transformer.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 6836–6846.
  * Bachmann et al. (2022) Bachmann, R.; Mizrahi, D.; Atanov, A.; and Zamir, A. 2022.  MultiMAE: Multi-modal Multi-task Masked Autoencoders.  _arXiv preprint arXiv:2204.01678_.
  * Bruce, Liu, and Chan (2021) Bruce, X.; Liu, Y.; and Chan, K. C. 2021.  Multimodal fusion via teacher-student network for indoor action recognition.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, 3199–3207.
  * Bruce et al. (2022) Bruce, X.; Liu, Y.; Zhang, X.; Zhong, S.-h.; and Chan, K. C. 2022.  MMNet: A Model-based Multimodal Network for Human Action Recognition in RGB-D Videos.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
  * Carreira and Zisserman (2017) Carreira, J.; and Zisserman, A. 2017.  Quo vadis, action recognition? a new model and the kinetics dataset.  In _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 6299–6308.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255. Ieee.
  * Devlin et al. (2018) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dosovitskiy et al. (2020) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_.
  * Feichtenhofer (2020) Feichtenhofer, C. 2020.  X3d: Expanding architectures for efficient video recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 203–213.
  * Feichtenhofer et al. (2022) Feichtenhofer, C.; Fan, H.; Li, Y.; and He, K. 2022.  Masked Autoencoders As Spatiotemporal Learners.  _arXiv preprint arXiv:2205.09113_.
  * Feichtenhofer et al. (2019) Feichtenhofer, C.; Fan, H.; Malik, J.; and He, K. 2019.  Slowfast networks for video recognition.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 6202–6211.
  * Garcia et al. (2019) Garcia, N. C.; Bargal, S. A.; Ablavsky, V.; Morerio, P.; Murino, V.; and Sclaroff, S. 2019.  Dmcl: Distillation multiple choice learning for multimodal action recognition.  _arXiv preprint arXiv:1912.10982_.
  * Garcia, Morerio, and Murino (2018) Garcia, N. C.; Morerio, P.; and Murino, V. 2018.  Modality distillation with multiple stream networks for action recognition.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 103–118.
  * Garcia, Morerio, and Murino (2019) Garcia, N. C.; Morerio, P.; and Murino, V. 2019.  Learning with privileged information via adversarial discriminative modality distillation.  _IEEE transactions on pattern analysis and machine intelligence_ , 42(10): 2581–2593.
  * He et al. (2022) He, K.; Chen, X.; Xie, S.; Li, Y.; Dollár, P.; and Girshick, R. 2022.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 16000–16009.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Hendrycks and Gimpel (2016) Hendrycks, D.; and Gimpel, K. 2016.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_.
  * Hinton et al. (2015) Hinton, G.; Vinyals, O.; Dean, J.; et al. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2(7).
  * Hoffman, Gupta, and Darrell (2016) Hoffman, J.; Gupta, S.; and Darrell, T. 2016.  Learning with side information through modality hallucination.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 826–834.
  * Hu et al. (2018) Hu, J.-F.; Zheng, W.-S.; Pan, J.; Lai, J.; and Zhang, J. 2018.  Deep bilinear learning for rgb-d action recognition.  In _Proceedings of the European conference on computer vision (ECCV)_ , 335–351.
  * Huang et al. (2021) Huang, Y.; Du, C.; Xue, Z.; Chen, X.; Zhao, H.; and Huang, L. 2021.  What makes multi-modal learning better than single (provably).  _Advances in Neural Information Processing Systems_ , 34: 10944–10956.
  * Liu et al. (2019) Liu, J.; Shahroudy, A.; Perez, M.; Wang, G.; Duan, L.-Y.; and Kot, A. C. 2019.  Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding.  _IEEE transactions on pattern analysis and machine intelligence_ , 42(10): 2684–2701.
  * Lopez-Paz et al. (2015) Lopez-Paz, D.; Bottou, L.; Schölkopf, B.; and Vapnik, V. 2015.  Unifying distillation and privileged information.  _arXiv preprint arXiv:1511.03643_.
  * Loshchilov and Hutter (2017) Loshchilov, I.; and Hutter, F. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Luo et al. (2018) Luo, Z.; Hsieh, J.-T.; Jiang, L.; Niebles, J. C.; and Fei-Fei, L. 2018.  Graph distillation for action detection with privileged modalities.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 166–183.
  * Ma et al. (2022) Ma, M.; Ren, J.; Zhao, L.; Testuggine, D.; and Peng, X. 2022.  Are Multimodal Transformers Robust to Missing Modality?  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 18177–18186.
  * Ma et al. (2021) Ma, M.; Ren, J.; Zhao, L.; Tulyakov, S.; Wu, C.; and Peng, X. 2021.  SMIL: Multimodal learning with severely missing modality.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, 2302–2310.
  * Pechyony and Vapnik (2010) Pechyony, D.; and Vapnik, V. 2010.  On the theory of learnining with privileged information.  _Advances in neural information processing systems_ , 23.
  * Rahmani et al. (2016) Rahmani, H.; Mahmood, A.; Huynh, D.; and Mian, A. 2016.  Histogram of oriented principal components for cross-view action recognition.  _IEEE transactions on pattern analysis and machine intelligence_ , 38(12): 2430–2443.
  * Ren et al. (2021) Ren, Z.; Zhang, Q.; Gao, X.; Hao, P.; and Cheng, J. 2021.  Multi-modality learning for human action recognition.  _Multimedia Tools and Applications_ , 80(11): 16185–16203.
  * Sanchez-Caballero et al. (2022) Sanchez-Caballero, A.; de López-Diz, S.; Fuentes-Jimenez, D.; Losada-Gutiérrez, C.; Marrón-Romera, M.; Casillas-Perez, D.; and Sarker, M. I. 2022.  3dfcnn: Real-time action recognition using 3d deep neural networks with raw depth information.  _Multimedia Tools and Applications_ , 1–25.
  * Shahroudy et al. (2016) Shahroudy, A.; Liu, J.; Ng, T.-T.; and Wang, G. 2016.  Ntu rgb+ d: A large scale dataset for 3d human activity analysis.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1010–1019.
  * Srivastava et al. (2014) Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1): 1929–1958.
  * Stroud et al. (2020) Stroud, J.; Ross, D.; Sun, C.; Deng, J.; and Sukthankar, R. 2020.  D3d: Distilled 3d networks for video action recognition.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 625–634.
  * Szegedy et al. (2016) Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; and Wojna, Z. 2016.  Rethinking the inception architecture for computer vision.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2818–2826.
  * Tran et al. (2015) Tran, D.; Bourdev, L.; Fergus, R.; Torresani, L.; and Paluri, M. 2015.  Learning spatiotemporal features with 3d convolutional networks.  In _Proceedings of the IEEE international conference on computer vision_ , 4489–4497.
  * Tran et al. (2018) Tran, D.; Wang, H.; Torresani, L.; Ray, J.; LeCun, Y.; and Paluri, M. 2018.  A closer look at spatiotemporal convolutions for action recognition.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 6450–6459.
  * Vapnik, Izmailov et al. (2015) Vapnik, V.; Izmailov, R.; et al. 2015.  Learning using privileged information: similarity control and knowledge transfer.  _J. Mach. Learn. Res._ , 16(1): 2023–2049.
  * Vapnik and Vashist (2009) Vapnik, V.; and Vashist, A. 2009.  A new learning paradigm: Learning using privileged information.  _Neural networks_ , 22(5-6): 544–557.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2014) Wang, J.; Nie, X.; Xia, Y.; Wu, Y.; and Zhu, S.-C. 2014.  Cross-view action modeling, learning and recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2649–2656.
  * Wang et al. (2016) Wang, L.; Xiong, Y.; Wang, Z.; Qiao, Y.; Lin, D.; Tang, X.; and Gool, L. V. 2016\.  Temporal segment networks: Towards good practices for deep action recognition.  In _European conference on computer vision_ , 20–36. Springer.
  * Wang et al. (2018) Wang, X.; Girshick, R.; Gupta, A.; and He, K. 2018.  Non-local neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 7794–7803.
  * Wightman (2019) Wightman, R. 2019.  PyTorch Image Models.  https://github.com/rwightman/pytorch-image-models.
  * Zhao, Li, and Jin (2021) Zhao, J.; Li, R.; and Jin, Q. 2021.  Missing modality imagination network for emotion recognition with uncertain missing modalities.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , 2608–2618.