11institutetext: King Abdullah University of Science and Technology (KAUST),
Saudi Arabia 11email<EMAIL_ADDRESS>
# Combating Missing Modalities in Egocentric Videos at Test Time

Merey Ramazanova  Alejandro Pardo  Bernard Ghanem  Motasem Alfarra

###### Abstract

Understanding videos that contain multiple modalities is crucial, especially
in egocentric videos, where combining various sensory inputs significantly
improves tasks like action recognition and moment localization. However, real-
world applications often face challenges with incomplete modalities due to
privacy concerns, efficiency needs, or hardware issues. Current methods, while
effective, often necessitate retraining the model entirely to handle missing
modalities, making them computationally intensive, particularly with large
training datasets. In this study, we propose a novel approach to address this
issue at test time without requiring retraining. We frame the problem as a
test-time adaptation task, where the model adjusts to the available unlabeled
data at test time. Our method, MiDl (Mutual information with self-
Distillation), encourages the model to be insensitive to the specific modality
source present during testing by minimizing the mutual information between the
prediction and the available modality. Additionally, we incorporate self-
distillation to maintain the model’s original performance when both modalities
are available. MiDl represents the first self-supervised, online solution for
handling missing modalities exclusively at test time. Through experiments with
various pretrained models and datasets, MiDl demonstrates substantial
performance improvement without the need for retraining.

###### Keywords:

Missing Modality Test-time Adaptation

##  1 Introduction

Understanding multi-modal data has emerged as a pivotal challenge in various
domains, including foundational model construction [27], emotion recognition
[14], and analysis of egocentric videos for tasks [9, 8, 4] like recognition
[12, 22, 35] and localization [28, 34]. Recent efforts focused on crafting
models harnessing data diverse modalities showing performance gains in
multiple tasks [18]. However, a critical limitation arises: these models
presuppose complete modality availability at test-time, which diverges from
real-world scenarios where data modalities may be incomplete [19]. For
instance, in real-time prediction using wearable devices, portions of
recordings might be redacted to safeguard privacy, or cost constraints may
necessitate using cheaper modalities like audio or IMU [8, 9]. Consequently,
models designed under this assumption demonstrate significant performance
degradation in the face of missing modalities, sometimes even underperforming
unimodal counterparts trained with a single modality [28].

The challenge of addressing missing modalities gained significant attention
from researchers recently, where various strategies have been proposed to
mitigate this issue [23, 20, 19, 29]. Some works have explored architectural
modifications to fuse information from different modalities effectively [19].
Additionally, other approaches have focused on designing effective
regularizers to boost model performance when confronted with missing
modalities [3]. More recently, a promising direction has emerged, wherein
transformer models have been augmented with learnable tokens during training
[15, 29]. These tokens serve to compensate for missing information at test
time, significantly enhancing model robustness against missing modality [29].
Despite these advancements, a common drawback persists: all existing
approaches necessitate expensive retraining of the multi-modal model,
rendering pretrained models obsolete. This poses a substantial challenge,
particularly in applications with extensive training data, where the
retraining process is prohibitively expensive, making the aforementioned
approaches impractical. Consequently, a fundamental question arises:

_Can we develop methods to address missing modalities at test time without
imposing retraining requirements?_

Figure 1: Test-Time Adaptation for missing modalities. The concept of test-
time adaptation in the presence of missing data modalities focuses on a system
where a stream of multi-modal data is input, potentially lacking one or more
modalities (represented by audio and visual symbols). Without adaptation, the
pretrained model $f_{\theta_{0}}$ processes the samples but may not accurately
predict labels due to incomplete data. With test-time adaptation, the model is
dynamically adjusted using the adaptation method $g$, resulting in an adapted
model $f_{\theta_{t}}$, which is designed to handle the missing modalities and
improve over time. The graph on the right illustrates the potential
performance trajectory of models with and without test-time adaptation,
showing the efficacy of adaptation in maintaining higher performance levels
despite the variability in data modality availability.

In this work, we take the initial step of framing the missing modality problem
as a test-time adaptation problem [33, 17, 16]. Specifically, we aim to
establish a new approach wherein pretrained models undergo adaptation at test
time to optimize their performance in the presence of missing modalities. Our
formulation assumes a scenario where a continuous stream of unlabeled data is
fed into the pretrained model during testing, with some segments missing
certain modalities. The objective is to devise an adaptation algorithm capable
of refining the model’s predictions under missing modality in real-time
settings (refer to Figure 1). Based on this formulation, we first evaluate
existing methodologies from the test-time adaptation literature and
demonstrate their limited efficacy in addressing this specific challenge.
Subsequently, we introduce a novel test-time adaptation technique explicitly
tailored to tackle the missing modality problem. Our method revolves around
incentivizing the output of the pretrained model to remain invariant to the
modality source present during testing. To achieve this, we propose minimizing
the mutual information between the model’s output and the modality type of the
incoming unlabeled data at test time in a self-supervised manner. Moreover, to
ensure the preservation of model performance under a complete modality setup,
we integrate our approach with a self-distillation mechanism. Notably, our
method, termed MiDl (Mutual information with Self-Distillation minimization)
is theoretically motivated and agnostic to the choice of pretrained model
architecture, the dataset, and the specific type of missing modality
encountered during testing.

We summarize our contributions in three-fold:

  1. 1.

We redefine the missing modality problem as a test-time adaptation challenge,
pioneering a novel approach where pretrained models are adapted in real-time
to optimize performance in the face of missing modalities. We evaluate the
effectiveness of the current adaptation method under this challenging problem.

  2. 2.

We introduce MiDl (Mutual information with Self-Distillation minimization), a
test-time adaptation method explicitly tailored to combat the missing modality
problem. MiDl incentivizes model outputs to remain invariant to modality
sources during testing, effectively enhancing robustness.

  3. 3.

Our approach offers versatility, being agnostic to factors such as pretrained
model architecture, training dataset, and the specific type of missing
modality encountered during testing, thus providing a comprehensive solution
applicable across diverse scenarios. When combining pretrained models with
MiDl, a significant performance gain is attained (6% on epic sounds and 11% on
Epic Kitchens datasets).

Code to reproduce all our experimental results can be found in this repo.

##  2 Related Wrok

#### Missing Modalities in Multimodal Datasets.

Several works addressed the problem of missing modality in the multimodal
datasets [31, 20, 37, 23, 19]. Most methods addressing the missing modality
problem assume full access to the source (training) data. Some works are based
on the assumption that the training data is modal-complete, and the goal is to
train a model robust to the missing inputs at test time [20, 19]. Other works
tackle the modality distillation task, where the training data is multimodal,
but only one modality is used at test time [26, 7]. Few works assume the
modalities could be missing at train and test times and attempt to train a
robust network [15, 29]. In our work, we explore a more realistic scenario
where we might not have access to the training data or network re-training is
not feasible. We formulate this setup as a test-time adaptation problem, where
the model is experiencing a distribution shift caused by the unavailability of
some modalities at test time. Further, we propose the first test-time
adaptation algorithm tailored to combat the missing modality challenge at
test-time.

#### Test-Time Adaptation.

Test-time Adaptation (TTA) attempts to combat performance gaps that pretrained
models suffer from when exposed to distribution shifts at test-time [21, 13].
This is usually attained through modifying the model’s parameters [17] or its
input [6] by using the incoming unlabeled data at test-time. TTA methods are
practical, as they avoid assumptions on the training phase of a given model
[33]. The first of these approaches adjusts the statistics of the Batch
Normalization (BN) layers [16]. This was followed by more powerful adaptation
methods that involved self-supervised objective functions such as entropy
minimization [33, 24, 25], information maximization [17], and teacher-student
approaches [36]. While such TTA methods made significant progress towards
combating distribution shifts at test-time, they solely focused on simple
covariate shifts such as changes in weather conditions or pixel illumination
[10]. In this work, we extend the problem formulation of test-time adaptation
to a very practical and realistic domain shift: missing modality. In
particular, we adapt the stream setting of online test-time adaptation [1] to
formulate the missing modality problem along with the corresponding evaluation
protocol. Building on this novel view of the missing modality problem, we
analyze the current state-of-the-art TTA methods where we show their limited
impact. Further, we propose a novel TTA method tailored to combat the missing
modality problem.

##  3 Missing Modality as Test Time Adaptation

In this section, we formulate the missing modality problem as a test time
adaptation problem 3.1. We then outline the evaluation protocol of a given
adaptation method in Section 3.2

###  3.1 Problem Formulation

In this work, we focus on the recognition problem. Let
$f_{\theta}:\mathbb{R}^{d}\rightarrow\mathcal{P}(\mathcal{Y})$ be a classifier
that maps a given input $x\in\mathbb{R}^{d}$ into the probability simplex
$\mathcal{P}(\mathcal{Y})$ over the set of labels
$\mathcal{Y}=\\{1,2,\dots,K\\}$111 _e.g._ the output after a softmax layer..
In this work, we assume that an input $x$ is a multi-modal input. However, in
many realistic applications, the input provided to the model might have
missing modalities, thus containing either audio only, visual only, or audio-
visual information. Let $m\in\\{A,V,AV\\}$ denote the type of available
modality for a given input $x$ corresponding to audio only, visual only,
audio-visual, respectively. For simplicity, we fix the dimensionality of the
input $x$ by replacing the missing modality part with either zeros or random
data. Further, we assume that $f_{\theta}$ is pretrained on a training set
$\mathcal{D}$. In this work, we make no assumptions on $f_{\theta}$ (_i.e._
choice of architecture), the dataset $\mathcal{D}$, nor the way the training
process is done.

At test time, $f_{\theta}$ is presented with a stream of _unlabeled_ data
$\mathcal{S}$ with possibly missing modalities. The likelihood with which a
certain modality appears in data revealed from $\mathcal{S}$ is characterized
by a probability mass function; denoted by $\mathbb{P}_{\mathcal{S}}(M=m)$.
For example, if $\mathbb{P}_{\mathcal{S}}(M=V)=0.5$ and
$\mathbb{P}_{\mathcal{S}}(M=AV)=0.5$, then the audio missing rate in the test
stream is $50\%$. This is since 50% of the data comes as video only, thus
missing its audio modality. Let $p_{m}=\mathbb{P}_{\mathcal{S}}(M=m)$, then
the missing rate of different modalities can be equivalently characterized
with $P=\\{p_{A},p_{V},p_{AV}\\}$. Thus, for a stream with 25% missing video,
$P=\\{0.25,0.0,0.75\\}$ (_i.e._ the stream will reveal data with 25%
probability of having only audio and 75% probability of revealing both
modalities). According to this characterization, one can define the missing
rate of at least one modality as $1-p_{AV}$ with $p_{A}$ being the rate of
missing video and $p_{V}$ being the audio missing rate. Next, we discuss the
online evaluation protocol of $f_{\theta}$ under the stream $\mathcal{S}$ of
unlabeled data.

###  3.2 Evaluation Protocol

Given our formulation of missing modality in Section 3.1, we are now ready to
outline the evaluation protocol. Note that an adaptation method is a function
$g(\theta)$ that sequentially adapts the model’s parameters $\theta$ to
enhance the performance under the missing modality setup. Formally and
following the online learning literature [2, 1], we simulate the interaction
between the stream $\mathcal{S}$ characterized by $P$ and the TTA method $g$,
at each time step $t\in\\{0,1,\dots,\infty\\}$, as follows:

1. $\mathcal{S}$ reveals a sample/batch $x_{t}$ with its corresponding modality $m$. 2. $f_{\theta_{t}}$ generates the prediction $\hat{y}_{t}$. 3. $g$ adapts the model parameter $\theta_{t}$ to $\theta_{t+1}$.

where $f_{\theta_{0}}$ is the non-adapted pretrained model. That is, for each
revealed sample/batch $x_{t}$, the model needs to predict its label before
receiving the next data point $x_{t+1}$. The adaptation method $g$ can exploit
the predicted label to improve the model performance on the next revealed
samples. The performance of an adaptation method $g$ is measured in an online
manner by comparing the predicted label $\hat{y}_{t}$ with the ground truth
label $y_{t}$.

##  4 Proposed Solution

Figure 2: Adapting at test-time with MiDl. At test time, the stream reveals a
sample. MiDl uses multimodal samples to adapt by using three copies of the
model and one forward pass for each modality combination. MiDl leverages (KL)
divergence to align the predictions of the adapted model $f_{\theta_{t}}$ with
those of the original model $f_{\theta_{0}}$, ensuring that the adapted model
does not deviate too far from the original model’s predictions. The Mutual-
Information (MI) component uses the prediction from the different modalities
to reduce the dependency on any specific modality, fostering a more
generalized and robust prediction across different modality combinations. MiDl
updates the model for step $t+1$ using the combination of KL and MI in
Equation 2.

In this section, we propose our novel adaptation strategy to combat missing
modalities at test time. Recall that the adaptation method $g$ has to satisfy
the following requirements. First, $g$ has to be fully self-supervised. The
test stream reveals strictly unlabeled data at test time. Second, $g$ has to
conduct the adaptation in an online manner. That is, $g$ should adapt on each
revealed sample/batch of data $x_{t}$ since $\mathcal{S}$ reveals $x_{t+1}$
only after the model predicts $\hat{y}_{t}$ (refer to the interaction between
$\mathcal{S}$ and $g$ in Section 3.1).

To formulate our regularizer, we begin by asking the following question: How
should an optimal $f_{\theta}$ behave under missing modality? We believe that
a robust model against missing modality should satisfy two properties. First,
the prediction of $f_{\theta}$ should not depend on the available modality.
Ideally, $f_{\theta}$ should output the same prediction under both complete
and incomplete modality, hence satisfying the following equality: (i)
$f_{\theta}(x_{i};M=A)=f_{\theta}(x_{i};M=V)=f_{\theta}(x_{i};M=AV)\,\forall
i$. (ii) $f_{\theta}$ should have high accuracy in predicting data with
complete modality which is generally satisfied for $f_{\theta_{0}}$.
Satisfying both properties will result in a model that is accurate (satisfying
(ii)) and robust against missing modality (satisfying (i)). To construct an
adaptation algorithm that satisfies both properties, we propose to solve the
following optimization problem:

$\theta^{*}=\underset{\theta}{\text{arg}\min}\,\underset{{x\sim\mathcal{S}}}{\mathbb{E}}\,\left[\text{MI}\left(f_{\theta}(x;m),m\right)+\text{KL}\left(f_{\theta}(x\>|\>M=AV)||f_{\theta_{0}}(x\>|\>M=AV\right)\right]$
(1)

where $\text{MI}(u,v)$ is the mutual information between $u$ and $v$ and KL is
the KL-Divergence. Note that if the mutual information between two random
variables $\text{MI}(u,v)=0$, then $u$ and $v$ are independent from each
other. That is, minimizing the first term in the objective function in
Equation (1) aims to satisfy property (i). Hence, if
$\text{MI}\left(f_{\theta^{*}}(x;m),m\right)=0$, then the output of the
adapted classifier becomes independent from the available modality at test
time. Furthermore, to ensure that the adapted parameters are still performing
well upon adaptation, we equip the mutual information minimization with a
self-distillation approach through minimizing KL divergence between the
prediction of the adapted model and the original $f_{\theta_{0}}$, satisfying
property (ii).

Although the objective function in Equation (1) is self-supervised, obtaining
$\theta^{*}$ requires accessing all samples from the stream $\mathcal{S}$ to
evaluate the expectation $\mathbb{E}_{x\sim\mathcal{S}}$, which is not
available at test time in the online evaluation protocol. To that end, we
approximate the expected value during adaptation at time $t$ with the samples
$x_{t}$ revealed from the stream. Hence, our Mutual information with self-
Distillation (MiDl) adaptation step at timestamp $t$ can be expressed as the
following gradient descent step:

$\displaystyle\theta_{t+1}=\theta_{t}-\gamma\nabla_{\theta}\mathcal{L}_{\text{MiDl}}|_{\theta=\theta_{t}}=\theta_{t}-\gamma\nabla_{\theta}(\mathcal{L}_{\text{MI}}+\mathcal{L}_{\text{KL}})|_{\theta=\theta_{t}}$
(2)
$\displaystyle\text{with}\,\,\mathcal{L}_{KL}=\text{KL}\left(f_{\theta}(x_{t};M=AV)||f_{\theta_{0}}(x_{t};M=AV\right)$
$\displaystyle\qquad\mathcal{L}_{\text{MI}}=\underbrace{\mathbb{E}_{m}\left[\sum_{i=1}^{K}f_{\theta}^{i}(x_{t};m)\log\left(f_{\theta}^{i}(x_{t};m)\right)\right]}_{\mathcal{L}_{\text{ent}}}-\underbrace{\sum_{i=1}^{K}\hat{f}_{\theta}^{i}(x_{t})\log\left(\hat{f}_{\theta}^{i}(x_{t})\right)}_{\mathcal{L}_{\text{div}}}$

where $\hat{f}_{\theta}(x_{t})=\mathbb{E}_{m}[f_{\theta}(x_{t};m)]$,
$f_{\theta}^{i}(x_{t};m)$ is the $i^{th}$ element in the vector
$f_{\theta}(x_{t};m)$, and $\gamma>0$ is the learning rate of the gradient
descent step. To estimate the expectation
$\mathbb{E}_{m}[f_{\theta}(x_{t};m)]$, we conduct three forward passes for
$x_{t}$ with setting $m\in\\{A,V,AV\\}$ and average their predictions (refer
to Figure 2). Similarly, We average the entropy of those predictions to
calculate $\mathcal{L}_{\text{ent}}$. Note that $\mathcal{L}_{\text{div}}$ is
the entropy of the average prediction across modalities. Note that under
incomplete modality, $\mathcal{L}_{\text{ent}}=\mathcal{L}_{\text{div}}$
resulting in $\mathcal{L}_{\text{MI}}=0$. Hence, $\mathcal{L}_{\text{MI}}\neq
0$ only when $x_{t}$ has complete modalities (refer to the appendix for
further details). Based on that, we propose to conduct our adaptation step
_only_ when $\mathcal{S}$ reveals $x_{t}$ with complete modalities. However,
when $\mathcal{S}$ reveals $x_{t}$ with incomplete modality, we refrain from
adaptation and set $\theta_{t+1}=\theta_{t}$. This makes the interaction
between the stream $\mathcal{S}$ and our proposed adaptation with MiDl at each
time step $t$ take the following form:

1. $\mathcal{S}$ reveals a sample/batch $x_{t}$ with its corresponding modality $m$. 2. $f_{\theta_{t}}$ generates the prediction $\hat{y}_{t}$. 3. If $x_{t}$ is with complete modalities then $g$ adapts the model parameter $\theta_{t}$ to $\theta_{t+1}$ through Equation (2), else set $\theta_{t+1}=\theta_{t}$.

For $x_{t}$ with missing modality, we leverage the most recent adapted model
to perform predictions without adaptation. Next, we validate the effectiveness
of our proposed MiDl experimentally.

##  5 Experiments

We perform a thorough analysis of MiDl’s performance under different missing
modality scenarios. We present our experimental setup in Section 5.1. Then, in
Section 5.2, we compare our method on the most simple test-time adaptation
scenario where the model has to adapt while replying to the stream. Then, in
Section 5.3, we study MiDl’s behaviour under the Long-Term Adaptation (LTA)
setup that happens when the model has been exposed to a long stream of data
$\mathcal{S}$. Finally, Section 5.4 analyzes the scenario, in which one has
access to some unlabeled data from an out-of-domain source before the model is
deployed. Under all these three scenarios, we show that MiDl is the better
alternative to combating missing modalities at test time.

###  5.1 Setup

We follow [29] and use the validation sets of Epic-Kitchens and Epic-Sounds
datasets for all experiments and report action recognition accuracy. We also
use Ego4D-AR videos for the out-of-domain experiments in Section 5.4.

#### Datasets.

Epic-Kitchens [4, 5] contains 100 video hours of people performing cooking
activities recorded with wearable cameras. The dataset is commonly used for
benchmarking audiovisual egocentric action recognition. Each instance is
annotated as a noun and verb pair (_e.g_. cut tomato), with a total of 300
noun and 97 verb classes. The validation set contains 9668 samples. Epic-
Sounds [11] provides sound-based annotations for the same 100 video hours. It
contains 44 classes and 8045 validation set samples.

We assess the effectiveness of the baselines and our proposed MiDl in
combating missing modalities at test time. To do so, we present the pretrained
model with a stream $\mathcal{S}$ of unlabeled validation data where we drop
one modality with a rate of $1-p_{AV}$. We set
$p_{AV}\in\\{0.0,0.25,0.5,0.75,1.0\\}$ resulting in a missing rate of
$\\{100\%,75\%,50\%,25\%,0\%\\}$, respectively. Following [29], for each
dataset, we drop the primary modality (_i.e._ sound for Epic-Sounds and video
for Epic-Kitchens). Thus, for a 75% missing rate (_i.e._ $p_{AV}=0.25$), the
stream containing Epic-Sounds has $P=\\{0.0,0.75,0.25\\}$ and
$P=\\{0.75,0.0,0.25\\}$ for Epic-Kitchens.

#### Architecture.

Unless stated otherwise, we use the stronger architecture Multimodal
Bottleneck Transformer (MBT) [22] as $f_{\theta}$. We use the baselines of
[29] trained on Epic-Sounds and Epic-Kitchens. Nonetheless, we experiment with
the multimodal fusion via vanilla self-attention architecture [32] in Section
6.1.

###  5.2 MiDl Improves Performance

Table 1: Combating missing modalities at Test-time. We showcase different results of different methods when combating missing modalities at test time. The first two rows show the unimodal performance and the Multimodal Baseline with no adaptation. Adapting to missing modalities is critical since the performance of the Baselines degrades quickly under increasing missing modalities. We show three alternative test-time adaptation methods and demonstrate that our proposed MiDl is effective at combating missing modalities at test time, outperforming every other alternative. | Epic-Sounds (%) | Epic-Kitchens (%)
---|---|---
Model $1-p_{AV}$ | 0 | 25 | 50 | 75 | 100 | 0 | 25 | 50 | 75 | 100
Unimodal | 41.4 | 40.0
Baseline | 55.1 | 45.6 | 37.1 | 28.3 | 19.5 | 63.9 | 55.5 | 46.8 | 37.9 | 29.5
+Shot | 55.0 | 45.6 | 37.1 | 28.7 | 20.1 | 63.8 | 56.0 | 48.5 | 41.1 | 34.5
+Tent | 54.9 | 45.0 | 35.9 | 26.3 | 17.7 | 63.8 | 53.7 | 38.6 | 23.6 | 9.8
+ETA | 55.2 | 45.6 | 37.0 | 28.3 | 19.5 | 63.4 | 52.2 | 34.1 | 19.5 | 7.9
+MiDl (ours) | 55.1 | 46.7 | 38.9 | 29.9 | 19.5 | 63.8 | 58.4 | 52.1 | 44.9 | 29.5

Table 1 reports the average accuracy under different missing modality rates on
Epic-Sounds and Epic-Kitchens. In this experiment, we compare our proposed
MiDl against three off-the-shelf test-time adaptation methods. Namely, we
include the information maximization approach, Shot [17], and the entropy
minimization with data selection from ETA [24] (refer to the appendix for
their implementation details). For a fair comparison, we assess the efficacy
of the aforementioned adaptation methods against MiDl (refer to Section 4).

We observe that: (i) our proposed MiDl significantly enhances the performance
of pretrained models under missing modality. We record a significant gain of
5% and 7% on the Epic-Kitchens dataset under missing rates of 50% and 75%,
respectively. On the more challenging Epic-Sounds, MiDl boosts the accuracy of
the baseline from 37.1% and 28.3% to 38.9% and 29.9% under the same missing
rates, respectively. Note that this performance boost comes at no cost of
retraining the model; simply adapting it during test time. This result
demonstrates how our proposed mutual information minimization encourages the
model to become invariant to domain shifts due to missing modality. (ii) MiDl
recovers the performance of the baseline under complete modality (retaining
the 55.1% and 63.8% accuracy under 0% missing rate on Epic-Sounds and Epic-
Kitchens, respectively). That is, our proposed KL divergence regularization
enhances the information retention of the original pretrained model under
complete modality inference. (iii) We also observe that off-the-shelf test-
time adaptation methods are ineffective in this setup. This limitation arises
because all TTA methods are designed to tackle covariate domain shifts and,
thus, are not tailored to enhance performance under this specific type of
domain shift (missing modality). It is noteworthy to mention that MiDl does
not improve nor degrade the performance under 100% missing rate (_i.e._
$p_{AV}=1$). This is because our adaptation method only update the model
parameters under the presence of data with complete modality.

###  5.3 Performance Under Long-Term Adaptation (LTA)

Table 2: Adaptation at Test-time under Long-term Adaptation. We showcase the results of MiDL under the assumption that the stream of data is very long. We use unlabeled data to simulate a longer stream and report results on the validation set of each dataset. Our MiDl benefits from long-term adaptation. Especially at higher missing rates (>75%). | Epic Sounds(%) | Epic Kitchen(%)
---|---|---
Model $1-p_{AV}$ | 0 | 25 | 50 | 75 | 100 | 0 | 25 | 50 | 75 | 100
Unimodal | 41.4 | 40.0
Baseline | 55.1 | 45.6 | 37.1 | 28.3 | 19.5 | 63.9 | 55.5 | 46.8 | 37.9 | 29.5
+MiDl (ours) | 55.1 | 46.7 | 38.9 | 29.9 | 19.5 | 63.8 | 58.4 | 52.1 | 44.9 | 29.5
\+ MiDl - LTA (ours) | 55.1 | 46.9 | 39.5 | 32.6 | 26.2 | 63.8 | 58.4 | 52.4 | 46.7 | 41.4

Next, we analyze the effectiveness of our proposed MiDl under a long stream of
data $\mathcal{S}$. Since MiDl operates at test time, its performance gain can
vary depending on the data revealed at test time. We suspect that if
$\mathcal{S}$ reveals more _unlabeled_ data with complete modality at test
time, MiDl will adapt the model parameters for more iterations enhancing its
invariance properties against missing modality.

To study this interesting setting, we allow MiDl to access unlabeled data with
complete modality in the form of a stream, denoted as
$\mathcal{S}_{\text{in}}$, to perform adaptation. This simulates the case
where the test stream $\mathcal{S}$ is very long where the portion of data
with missing modality is still big enough. We then ask the following question:
how would MiDl perform after this long adaptation? We let
$\mathcal{S}_{\text{in}}$ be a subset of training data, and test the model on
$\mathcal{S}$ being the validation set, following the standard evaluation in
Section 5.2.

Table 2 summarizes the results on Epic-Sounds and Epic-Kitchens datasets at
the same missing rates considered in Section 5.2. We observe: (iv) the longer
the stream is, MiDl provides a bigger performance gain. For example, MiDl
further improves the non-adapted baseline by 4.3% and 8.8% on Epic-Sounds and
Epic-Kitchens, respectively, under a missing rate of 75% (_i.e._
$p_{AV}=0.25$). In addition, even under 100% missing rate, MiDl improves the
accuracy by a notable margin of 6.7% and 11.9% on Epic-Sounds and Epic-
Kitchens, respectively. That is, the adaptation on $\mathcal{S}_{\text{in}}$
unlocks a bigger potential of MiDl for life-long adaptation even when
$\mathcal{S}$ reveals data with a single modality. (v) Unlike MiDl, other
adaptation approaches do not benefit from this long stream setup as their
objective functions do not promote building an invariant model against missing
modality. We present these results in the supplementary material.

###  5.4 Warm-up on Ego4D: Exploiting Out of Domain Adaptation

Table 3: Adaptation at Test-time with Ego4D warm-up. We show another use cause of MiDL, in which the assumption is having access to out-of-domain unlabeled data to adapt before deployment. The results showcase MiDL’s capabilities on leveraging unlabeled-out-of-domain data to combat missing modalities. | Epic-Sounds (%) | Epic-Kitchens (%)
---|---|---
Model $1-p_{AV}$ | 0 | 25 | 50 | 75 | 100 | 0 | 25 | 50 | 75 | 100
Baseline | 55.1 | 45.6 | 37.1 | 28.3 | 19.5 | 63.9 | 55.5 | 46.8 | 37.9 | 29.5
\+ MiDl (ours) | 55.1 | 46.7 | 38.9 | 29.9 | 19.5 | 63.8 | 58.4 | 52.1 | 44.9 | 29.5
\+ Ego4D Warm-up | 55.0 | 46.6 | 38.6 | 30.9 | 19.5 | 63.7 | 58.4 | 52.4 | 46.7 | 37.8

Next, we analyze another practical setup. In particular, we consider the case
when not only the pretrained model $f_{\theta}$ is provided, but unlabeled
data from out-of-domain, denoted as $\mathcal{S}_{\text{out}}$ can also be
accessible. Note that our optimization objective in Equation (1) does not
assume the data distribution we adapt to. This then begs the question: would
adapting with MiDl on a different data distribution $\mathcal{S}_{\text{out}}$
help combat missing modalities in $\mathcal{S}$?

To answer this question, we leverage the recent Ego4D [8] data. Although Ego4D
has egocentric videos, they come from very different environments and
scenarios that deviate from the usual kitchen scene. These changes introduce
additional domain shifts when evaluating on Epic-Sounds and Epic-Kitchens. We
set $\mathcal{S}_{\text{out}}$ to be 25% of the Ego4D-AR and present them to
the model in the form of a stream. It is worth noting that, in this case, we
keep using our self-supervised MiDL objective, and we do not require any
labels from Ego4D-AR.

We replicate our setup from Section 5.3 and perform adaptation on
$\mathcal{S}_{\text{out}}$ followed by evaluation on $\mathcal{S}$ (_i.e._
evaluation on the standard validation set of Epic-Kitchens and Epic-Sounds).
We name the adaptation phase on $\mathcal{S}_{\text{out}}$ as a warm-up phase.

Table 3 summarizes the results where we show the performance of the non-
adapted baseline, our proposed MiDl when adapted solely on $\mathcal{S}$, and
our MiDL equipped with adaptation on $\mathcal{S}_{\text{out}}$. We observe
that (vi) conducting a warm-up generally exerts a positive influence on
overall performance in cases of missing modality. The enhanced version of MiDl
with Ego4D warm-up improves over MiDl by 1% and 1.8% on Epic-Sounds and Epic-
Kitchens, respectively, under a missing rate of 75%. Furthermore, we observe
that even under 100% missing rate, adapting on $\mathcal{S}_{\text{out}}$
enhances the accuracy of MiDl by an impressive 8% on Epic-Kitchens. This
demonstrates the versatility of MiDl, which provides consistent performance
gains under different setups.

#### Takeaway.

In this section, we showcased the effectiveness of our proposed MiDl in
combating the missing modality challenge at test time (Section 5.2). Further,
we analyzed the impact of long-term adaptation where MiDl provided further
performance gain where the relative improvement is up to 30% (Section 5.3). At
last, we showed how adapting with MiDl on out-of-distribution data, mimicking
data scarcity situations, still boosts the model’s accuracy (Section 5.4).

##  6 Analysis on MiDl

In this section, we conduct a comprehensive analysis of MiDl. In particular,
we show how our proposed test-time adaptation is agnostic to the choice of
$f_{\theta}$ (Section 6.1), the missing modality (Section 6.2). We conclude by
analyzing the different components of MiDl in Section 6.3 and its
computational requirements in Section 6.4.

###  6.1 Agnostic to Architecture Choice

Table 4: MiDL performance under a different architecture. We showcase the effectiveness of MiDL when applied to a distinct baseline architecture characterized by multimodal self-attention. MiDL enhances performance across all missing rates, underscoring its robustness and adaptability to various underlying architectures. Model $1-p_{AV}$ | Epic-Sounds (%)
---|---
0 | 25 | 50 | 75 | 100
Self-Attention Baseline | 45.3 | 38.8 | 32.7 | 26.7 | 20.5
+Shot | 45.5 | 39.0 | 32.8 | 26.8 | 20.7
+Tent | 45.3 | 38.6 | 32.3 | 26.0 | 19.8
+ETA | 45.3 | 38.8 | 32.7 | 26.7 | 20.5
+MiDl (ours) | 45.5 | 39.5 | 33.8 | 27.5 | 20.5
+MiDl - LTA (ours) | 45.5 | 39.6 | 34.5 | 29.0 | 23.2

Here, we analyze the robustness of MiDl against the architecture choice
$f_{\theta}$. To this end, we replicate our experimental setup from Sections
5.2 and 5.3 but set $f_{\theta}$ to be the multimodal fusion via vanilla self-
attention architecture [32], as opposed to the MBT architecture [22].

Table 4 summarizes the results under different missing rates. We compare the
non-adapted baseline, Shot and ETA, and our proposed MiDl. We observe that
MiDl provides a consistent performance boost under the self-attention
architecture, similar to our observations with the MBT architecture in Section
5.2. For example, MiDl improves the accuracy of the baseline under 50% missing
rate by a notable 1.1% without affecting the performance under complete
modality. Further, we analyze the performance of MiDl under Long-Term
Adaptation (LTA), following the setup in Section 5.3. Similar to our earlier
findings, LTA unlocks a better potential of MiDl where further performance
gain is attained. Under this setup, MiDl improves the baseline by 2% under 50%
and 75% missing rates, and by 2.7% under 100% missing rates. These results
show that MiDl is agnostic to the choice of architecture.

###  6.2 Agnostic to the Type of Missing Modality

Table 5: Adaptation at Test-time - Other Missing Modalities. In this table we show the results using the complementary modality for each of the datset, _i.e_. video for Epic Sounds and Audio for Epic Kitchens. We observe that MiDL improves consistently under this setup, highlighting its robustness to different types of modalities missing at test time. | Epic Sounds (%) | Epic Kitchens (%)
---|---|---
Model $1-p_{AV}$ | 0 | 25 | 50 | 75 | 100 | 0 | 25 | 50 | 75 | 100
Unimodal | 46.5 | 63.2
Baseline | 55.1 | 53.4 | 51.8 | 50.5 | 48.8 | 63.9 | 61.0 | 58.0 | 54.8 | 52.1
+MiDl | 55.1 | 53.3 | 51.8 | 50.7 | 48.8 | 63.9 | 61.6 | 59.1 | 55.9 | 52.1
+MiDl - LTA | 55.1 | 53.4 | 52.0 | 51.0 | 49.4 | 63.9 | 61.7 | 59.5 | 57.3 | 55.3

In previous sections, we focused our experiments on the effectiveness of MiDl
when the dominant modality is missing. In particular, we analyzed dropping the
audio modality for Epic-Sounds and the visual modality for Epic-Kitchens. In
this section, we attempt to study the robustness of MiDl against dropping the
other modality at test time. To this end, we conduct experiments on Epic-
Sounds, where we drop the visual modality, and on Epic-Kitchens, where we drop
the audio modality. Note that, under this setup, the baseline experiences less
performance degradation, similar to scenarios involving the missing dominant
modality. We replicate our experimental setup from Section 6.1 where we
compare the non-adapted baseline, its equipped version with our proposed MiDl,
and the LTA effect of MiDl.

Table 5 summarizes the results. Our experiments show that MiDl consistently
enhances the performance of the pretrained model irrespective of the type of
missing modality. For example, MiDl improves the baseline by 1% on Epic
Kitchens under missing rates of 50% and 75%. Further, and similar to our
observations in Section 5.3 and 6.1, the LTA further improves MiDl by over 3%
under 100% missing rate (_i.e._ $p_{AV}=0$).

###  6.3 Components of MiDl

Table 6: Analyzing MiDl components. We analyze the different components of MiDL. When the Mutual-Information (MI) component is missing, the model does not have any reason to adapt since the KL divergence is maximized by predicting the same as the base model. When KL is not present, the MI alone deviates from the initial results and performs poorly under higher missing rates. |  |  | Epic Sounds (%) | Epic Kitchens (%)
---|---|---|---|---
Model $1-p_{AV}$ | $\mathcal{L}_{\text{MI}}$ | $\mathcal{L}_{\text{KL}}$ | 0 | 25 | 50 | 75 | 0 | 25 | 50 | 75
Baseline | ✗ | ✗ | 55.2 | 45.6 | 37.1 | 28.3 | 63.9 | 55.5 | 46.8 | 37.9
\+ Dl | ✗ | ✓ | 55.2 | 45.6 | 37.1 | 28.3 | 63.9 | 55.5 | 46.8 | 37.9
\+ Mi | ✓ | ✗ | 40.4 | 39.3 | 36.1 | 29.6 | 53.5 | 50.5 | 47.6 | 45.9
+MiDl (ours) | ✓ | ✓ | 55.1 | 46.7 | 38.9 | 29.9 | 63.8 | 58.4 | 52.1 | 44.9

At last, we ablate the effectiveness of each component of MiDl. Recall from
our adaptation step in Equation (2) that MiDl has two components: (1) mutual
information minimization and self-distillation through minimizing KL-
divergence. We believe that the success of MiDl is attributed to the interplay
between both components.

To analyze the importance of each component, we adapt the pretrained baseline
with each loss function independently and compare the performance under
different missing rates. Table 6 summarizes the results on Epic-Sounds and
Epic-Kitchens datasets with $p_{AV}\in\\{0.25,0.5,0.75,1.0\\}$. We observe
that adapting solely with self-distillation (_i.e._ minimizing
$\mathcal{L}_{\text{KL}}$) results in no adaptation. On the contrary,
adaptation by minimizing only $\mathcal{L}_{\text{MI}}$ results in a
significant performance drop. While minimizing $\mathcal{L}_{\text{MI}}$ can
indeed result in $f_{\theta}$ that is robust against missing modality, it is
not necessarily accurate (_i.e._ satisfying the first (i) property in 4 and
violating the second (ii)). MiDl balances information minimization with the
information retention loss $\mathcal{L}_{\text{KL}}$.

###  6.4 Computational Requirement of MiDl

While our previous results have demonstrated the efficacy of MiDl in providing
performance gains, we note that this improvement comes at a computational
cost. In fact, conducting an adaptation step with MiDl requires conducting a
total of 3 inferences through $f_{\theta_{t}}$ and an inference through the
initial pretrained model $f_{\theta_{0}}$, followed by a single backward pass.
This makes an inference through MiDl 5$\times$ more expensive than doing
inference without any adaptation. In practice, the latency of MiDl is only
2$\times$ slower than the non-adapted model since all the additional 4 forward
passes can be performed in parallel.

#### Takeaway.

In this section, we conducted comprehensive analysis on our proposed MiDl. We
showed that MiDl is agnostic to the choice of architecture (Section 6.1) and
the type of missing modality (Section 6.2). We further analyzed the importance
of both components of MiDl in Section 6.3 and its computational requirements
in Section 6.4.

##  7 Conclusions

In this work, we presented MiDl, a new method for improving how pretrained
video recognition models handle missing modalities at test time. MiDl improves
the model’s ability to give accurate predictions regardless of the
availability of modalities by minimizing mutual information and using self-
distillation. Our experiments show that MiDl can significantly increase
accuracy across various datasets and scenarios and under various missing
rates, making it a practical solution for real-world applications dealing with
incomplete modalities.

## Contribution Statement

We highlight the contribution of each author in this project. Merey Ramazanova
coordinated and refined the project’s focus on missing modality issues in
egocentric videos. She led the implementation and experimental design,
including MiDl and other baselines, managed data preparation, and co-designed
the evaluation protocols. She proposed the Out of Domain Adaptation setup
using Ego4D. Alejandro Pardo refined the code for large-scale experimentation,
assisted in the baseline implementation, conducted the experiments, and
explored various ablations and parameters for each method. He played a
significant role in writing and presenting the experimental section of the
work. Bernard Ghanem guided the project from the beginning, helped shape the
initial ideas, and significantly contributed to the design of the experiments
and the project’s overall development. He also proofread and polished the
manuscript. Motasem Alfarra initiated the project by formulating the missing
modality challenge as a test-time adaptation problem, proposed the MiDl
objective function, co-designed the evaluation protocols, led the writing
efforts, and advised the implementation of MiDl along with the other
baselines.

## References

  * [1] Alfarra, M., Itani, H., Pardo, A., Alhuwaider, S., Ramazanova, M., Pérez, J.C., Cai, Z., Müller, M., Ghanem, B.: Revisiting test time adaptation under online evaluation. arXiv preprint arXiv:2304.04795 (2023)
  * [2] Cai, Z., Sener, O., Koltun, V.: Online continual learning with natural distribution shifts: An empirical study with visual data. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 8281–8290 (2021)
  * [3] Colombo, P., Chapuis, E., Labeau, M., Clavel, C.: Improving multimodal fusion via mutual dependency maximisation. In: Moens, M.F., Huang, X., Specia, L., Yih, S.W.t. (eds.) Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 231–245. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic (Nov 2021). https://doi.org/10.18653/v1/2021.emnlp-main.21, https://aclanthology.org/2021.emnlp-main.21
  * [4] Damen, D., Doughty, H., Farinella, G.M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., Wray, M.: Scaling egocentric vision: The epic-kitchens dataset. In: European Conference on Computer Vision (ECCV) (2018)
  * [5] Damen, D., Doughty, H., Farinella, G.M., Furnari, A., Kazakos, E., Ma, J., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al.: Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision pp. 1–23 (2022)
  * [6] Gao, J., Zhang, J., Liu, X., Darrell, T., Shelhamer, E., Wang, D.: Back to the source: Diffusion-driven test-time adaptation. arXiv preprint arXiv:2207.03442 (2022)
  * [7] Garcia, N.C., Morerio, P., Murino, V.: Learning with privileged information via adversarial discriminative modality distillation. IEEE transactions on pattern analysis and machine intelligence 42(10), 2581–2593 (2019)
  * [8] Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al.: Ego4d: Around the world in 3,000 hours of egocentric video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18995–19012 (2022)
  * [9] Grauman, K., Westbury, A., Torresani, L., Kitani, K., Malik, J., Afouras, T., Ashutosh, K., Baiyya, V., Bansal, S., Boote, B., et al.: Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. arXiv preprint arXiv:2311.18259 (2023)
  * [10] Hendrycks, D., Dietterich, T.: Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations (2019)
  * [11] Huh, J., Chalk, J., Kazakos, E., Damen, D., Zisserman, A.: Epic-sounds: A large-scale dataset of actions that sound. In: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1–5. IEEE (2023)
  * [12] Kazakos, E., Huh, J., Nagrani, A., Zisserman, A., Damen, D.: With a little help from my temporal context: Multimodal egocentric action recognition. In: British Machine Vision Conference (BMVC) (2021)
  * [13] Kojima, T., Matsuo, Y., Iwasawa, Y.: Robustifying vision transformer without retraining from scratch by test-time class-conditional feature alignment. arXiv preprint arXiv:2206.13951 (2022)
  * [14] Lee, H.C., Lin, C.Y., Hsu, P.C., Hsu, W.H.: Audio feature generation for missing modality problem in video action recognition. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 3956–3960. IEEE (2019)
  * [15] Lee, Y.L., Tsai, Y.H., Chiu, W.C., Lee, C.Y.: Multimodal prompting with missing modalities for visual recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14943–14952 (2023)
  * [16] Li, Y., Wang, N., Shi, J., Liu, J., Hou, X.: Revisiting batch normalization for practical domain adaptation. arXiv preprint arXiv:1603.04779 (2016)
  * [17] Liang, J., Hu, D., Feng, J.: Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In: International Conference on Machine Learning. pp. 6028–6039. PMLR (2020)
  * [18] Lin, K.Q., Wang, J., Soldan, M., Wray, M., Yan, R., XU, E.Z., Gao, D., Tu, R.C., Zhao, W., Kong, W., et al.: Egocentric video-language pretraining. Advances in Neural Information Processing Systems 35, 7575–7586 (2022)
  * [19] Ma, M., Ren, J., Zhao, L., Testuggine, D., Peng, X.: Are multimodal transformers robust to missing modality? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18177–18186 (2022)
  * [20] Ma, M., Ren, J., Zhao, L., Tulyakov, S., Wu, C., Peng, X.: Smil: Multimodal learning with severely missing modality. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 2302–2310 (2021)
  * [21] Mancini, M., Karaoguz, H., Ricci, E., Jensfelt, P., Caputo, B.: Kitting in the wild through online domain adaptation. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 1103–1109. IEEE (2018)
  * [22] Nagrani, A., Yang, S., Arnab, A., Jansen, A., Schmid, C., Sun, C.: Attention bottlenecks for multimodal fusion. Advances in Neural Information Processing Systems 34, 14200–14213 (2021)
  * [23] Neverova, N., Wolf, C., Taylor, G., Nebout, F.: Moddrop: adaptive multi-modal gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 38(8), 1692–1706 (2015)
  * [24] Niu, S., Wu, J., Zhang, Y., Chen, Y., Zheng, S., Zhao, P., Tan, M.: Efficient test-time model adaptation without forgetting. In: International conference on machine learning. pp. 16888–16905. PMLR (2022)
  * [25] Niu14, S., Wu, J., Zhang, Y., Wen, Z., Chen, Y., Zhao, P., Tan15, M.: Towards stable test-time adaptation in dynamic wild world. International Conference on Learning Representations (2023)
  * [26] Radevski, G., Grujicic, D., Blaschko, M., Moens, M.F., Tuytelaars, T.: Multimodal distillation for egocentric action recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5213–5224 (2023)
  * [27] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [28] Ramazanova, M., Escorcia, V., Caba, F., Zhao, C., Ghanem, B.: Owl (observe, watch, listen): Audiovisual temporal context for localizing actions in egocentric videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4879–4889 (2023)
  * [29] Ramazanova, M., Pardo, A., Alwassel, H., Ghanem, B.: Exploring missing modality in multimodal egocentric datasets. arXiv preprint arXiv:2401.11470 (2024)
  * [30] Robbins, H., Monro, S.: A stochastic approximation method. The annals of mathematical statistics pp. 400–407 (1951)
  * [31] Tsai, Y.H.H., Liang, P.P., Zadeh, A., Morency, L.P., Salakhutdinov, R.: Learning factorized multimodal representations. arXiv preprint arXiv:1806.06176 (2018)
  * [32] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [33] Wang, D., Shelhamer, E., Liu, S., Olshausen, B., Darrell, T.: Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726 (2020)
  * [34] Wang, H., Mirmehdi, M., Damen, D., Perrett, T.: Centre stage: Centricity-based audio-visual temporal action detection. In: The 1st Workshop in Video Understanding and its Applications (VUA 2023) (2023)
  * [35] Xiao, F., Lee, Y.J., Grauman, K., Malik, J., Feichtenhofer, C.: Audiovisual slowfast networks for video recognition. arXiv preprint arXiv:2001.08740 (2020)
  * [36] Yuan, L., Xie, B., Li, S.: Robust test-time adaptation in dynamic scenarios. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15922–15932 (2023)
  * [37] Zhao, J., Li, R., Jin, Q.: Missing modality imagination network for emotion recognition with uncertain missing modalities. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 2608–2618 (2021)

##  Appendix 0.A Proposed Solution: Extended Discussion

In Section 4, we proposed our novel adaptation strategy; MiDl. First, note
that our intuition of minimizing the mutual information between the output of
the network and the modality source comes from the following observation: Let
$X$ and $Y$ be two random variables, then

If $\displaystyle\text{MI}(X,Y)=0\,\,\rightarrow\,\,X\text{ and }Y\text{ are
independent}.$ Proof: $\displaystyle\text{MI}(X,Y)=\sum_{x\in X}\sum_{y\in
Y}P_{XY}(x,y)\log\left(\frac{P_{XY}(x,y)}{P_{X}(x)P_{Y}(y)}\right)$ If
$\displaystyle\text{MI}(X,Y)=0\,\,\rightarrow\,\,P_{XY}(x,y)=P_{X}(x)P_{Y}(y)$

That is, minimizing the mutual information between the output prediction of
$f_{\theta}$ and the available modality should make the adapted network robust
against missing modality. Second, MiDl adapts the pretrained model
$f_{\theta}$ on the unlabeled data revealed from the stream only if the
revealed data is with complete modality. This is since: (1) the KL-divergence
loss only applies on data with full modality and (2) the information
minimization loss only operates under complete modality. To wit, let $x_{t}$
be revealed with $m=A$. Then the estimate of our $\mathcal{L}_{\text{MI}}$
will be

$\displaystyle\mathcal{L}_{\text{MI}}=$
$\displaystyle\underbrace{\sum_{i=1}^{K}f_{\theta}^{i}(x_{t};M=A)\log\left(f_{\theta}^{i}(x_{t};M=A)\right)}_{\mathcal{L}_{\text{ent}}}$
$\displaystyle-$
$\displaystyle\underbrace{\sum_{i=1}^{K}f_{\theta}^{i}(x_{t};M=A)\log\left(f_{\theta}^{i}(x_{t};M=A)\right)}_{\mathcal{L}_{\text{div}}}=0.$

This is since $\mathbb{E}_{m}$ will be estimated with a single point at $m=A$.
Thus, MiDl adapts the parameters only when $\mathcal{S}$ reveals data with
complete modality at test time.

##  Appendix 0.B Experiments

###  0.B.1 Implementation Details

In Section 5.1, we detailed our experimental setup. Here, and for
reproducibility, we provide the implementation details for MiDl as well as the
baselines. Note that for all considered adaptation methods, we follow the
standard practice in the test-time adaptation literature [33, 17, 24, 25] and
only adapt the learnable parameters of the normalization layers. We freeze the
rest of the model parameters during the update step. Further, for MiDl, we
balance the mutual information loss and the self-distillation loss through:

$\mathcal{L}_{\text{MiDl}}=\lambda_{1}\mathcal{L}_{\text{MI}}+\lambda_{2}\mathcal{L}_{\text{KL}}.$

Note that for all our experiments, we set $\lambda_{1}=\lambda_{2}=3$.
Further, we conduct the adaptation step with an SGD [30] step with a learning
rate of $25\times 10^{-4}$, and a momentum of $0.9$, following [25, 24, 33].

Regarding the considered test-time adaptation baselines, we considered the
entropy minimization approach known as Tent [33], its improved version
equipped with data-selection process ETA [24], and the information
maximization Shot [17]. We followed the official implementation of each method
and used the recommended hyperparamaters.

###  0.B.2 Baselines Under Long Term Adaptation

Table 7: Adaptation at Test-time under Long-term Adaptation. We showcase the results of MiDL under the assumption that the data stream is very long. We use unlabeled data to simulate a longer stream and report results on the validation set of each dataset. Our MiDl benefits from long-term adaptation. Especially at higher missing rates (>75%). | Epic-Sounds (%) | Epic-Kitchens (%)
---|---|---
Model $1-p_{AV}$ | 0 | 25 | 50 | 75 | 100 | 0 | 25 | 50 | 75 | 100
Unimodal | 41.4 | 40.0
Baseline | 55.1 | 45.6 | 37.1 | 28.3 | 19.5 | 63.9 | 55.5 | 46.8 | 37.9 | 29.5
+Shot - LTA | 55.0 | 45.6 | 37.2 | 28.7 | 20.3 | 63.8 | 56.0 | 48.2 | 41.0 | 34.4
+Tent - LTA | 54.5 | 44.6 | 35.5 | 26.1 | 17.7 | 62.7 | 53.5 | 40.0 | 25.8 | 12.3
+ETA - LTA | 55.0 | 45.5 | 37.0 | 28.2 | 19.5 | 60.9 | 48.8 | 33.0 | 19.3 | 7.5
+MiDl - LTA (ours) | 55.1 | 46.9 | 39.5 | 32.6 | 26.2 | 63.8 | 58.4 | 52.4 | 46.7 | 41.4
Table 8: Adaptation at Test-time - Updating all parameters. In this table, we show the results when we unfreeze all network parameters, not only the normalization layer. We observe that there is no significant difference when compared to updating only the normalization layers. | Epic-Kitchens (%)
---|---
Model $1-p_{AV}$ | 0 | 25 | 50 | 75 | 100
Unimodal | 40.0
Baseline | 63.9 | 55.5 | 46.8 | 37.9 | 29.5
+MiDl (all parameters) | 63.6 | 58.4 | 52.4 | 46.3 | 29.5
+MiDl (norm layers) | 63.8 | 58.4 | 52.1 | 44.9 | 29.5
+MiDl - LTA (all parameters) | 63.6 | 58.3 | 52.5 | 47.1 | 42.0
+MiDl - LTA (norm layers) | 63.8 | 58.4 | 52.4 | 46.7 | 41.4

We extend our comparison against the considered baselines under the Long Term
Adaptation (LTA) setup. We replicate our experimental setup in Section 5.3 and
compare MiDl against SHOT, Tent, and ETA. We report the results in Table 7 for
the MBT architecture. We observe consistent findings in Section 5.3, where
naive adaptation baselines do not benefit from this long-term adaptation.
Further, we find that MiDl is the only adaptation method that provides further
performance gains under this LTA setup.

###  0.B.3 Adapting More Layers

At last, we extend our analysis on MiDl to include the effect of adapting more
parameters. In particular, we compare adapting only the learnable parameters
of the normalization layers against adapting the whole network parameters. We
report the result in Table 8 on Epic-Kitchens. We observe that adapting all
network parameters with MiDl results in a minor performance gain. For
instance, under the 75% missing rate, adapting all parameters improves over
adapting only the normalization layers by 0.4% under the LTA setup and by 1.4%
with the test-time adaptation of MiDl. We note here that this comes at a
computational expense as it is faster and more efficient to adapt only the
normalization layers.