11institutetext: Pennsylvania State University 22institutetext: NVIDIA
22email<EMAIL_ADDRESS>
# Reconstruct before Query: Continual Missing Modality Learning with
Decomposed Prompt Collaboration

Shu Zhao 11 Xiaohan Zou 11 Tan Yu 22 Huijuan Xu 11

###### Abstract

Pre-trained large multi-modal models (LMMs) exploit fine-tuning to adapt
diverse user applications. Nevertheless, fine-tuning may face challenges due
to deactivated sensors (e.g., cameras turned off for privacy or technical
issues), yielding modality-incomplete data and leading to inconsistency in
training data and the data for inference. Additionally, continuous training
leads to catastrophic forgetting, diluting the knowledge in pre-trained LMMs.
To overcome these challenges, we introduce a novel task, Continual Missing
Modality Learning (CMML), to investigate how models can generalize when data
of certain modalities is missing during continual fine-tuning. Our preliminary
benchmarks reveal that existing methods suffer from a significant performance
drop in CMML, even with the aid of advanced continual learning techniques.
Therefore, we devise a framework termed Reconstruct before Query (RebQ). It
decomposes prompts into modality-specific ones and breaks them into components
stored in pools accessible via a key-query mechanism, which facilitates
Parameter-Efficient Fine-Tuning111In this paper, fine-tuning denotes
parameter-efficient fine-tuning unless specified otherwise. and enhances
knowledge transferability for subsequent tasks. Meanwhile, our RebQ leverages
extensive multi-modal knowledge from pre-trained LMMs to reconstruct the data
of missing modality. Comprehensive experiments demonstrate that RebQ
effectively reconstructs the missing modality information and retains pre-
trained knowledge. Specifically, compared with the baseline, RebQ improves
average precision from $20.00$ to $50.92$ and decreases average forgetting
from $75.95$ to $8.56$. Code and datasets are available on
https://github.com/Tree-Shu-Zhao/RebQ.pytorch

###### Keywords:

Multi-Modal Learning Missing Modality Continual Learning

Figure 1: Large Multi-Modal Models (LMMs) pre-trained on a huge amount of
multi-modal data often necessitate subsequent Parameter-Efficient Fine-Tuning
(PEFT) to adapt to diverse user applications. However, deactivated sensors
(e.g., cameras turned off for privacy or technical issues) yield modality-
incomplete data, significantly challenging the fine-tuning process.

##  1 Introduction

Large multi-modal Models (LMMs), pre-trained in cloud-based computing
platforms with abundant resources, have powerful understanding and generating
capability for multi-modal data [42, 43, 8, 99, 36, 44, 10, 9, 86, 15, 85, 57,
16, 91, 87]. To adapt to diverse user environments, LMMs often necessitate
subsequent fine-tuning with user data. Normally, it is beneficial to continue
the learning process over the device’s lifespan [41, 56], as illustrated in
Figure 1. However, due to privacy concerns [24], users may resist sending
their data back to the model vendors for fine-tuning. Consequently, model
adaptation should be iteratively conducted on user devices with limited
resources, obviating the transmission of sensitive data externally. Moreover,
deploying all modalities in practical applications might be unfeasible due to
security concerns or technical issues [83, 35, 77, 92, 22]. For instance,
consider a long-term surveillance system that continuously monitors an
environment using multi-modal sensors (e.g., cameras, microphones, and thermal
sensors). Over time, some sensors may be temporarily unavailable, leading to
missing modalities. Additionally, the system needs to adapt to changing
conditions, such as varying lighting, weather, and the introduction of new
objects or activities. Accordingly, LMMs must be robust and adaptive, capable
of operating in dynamic and uncertain environments.

In this work, we define a novel task termed Continual Missing Modality
Learning (CMML) and construct two datasets, UPMC-Food101-CMML and MM-IMDb-
CMML, to benchmark the challenging settings in real-world scenarios. This
paradigm evaluates models in handling non-static data with absent modalities.
We reveal that existing methods for the missing modality problem are
susceptible to severe performance degradation due to catastrophic forgetting
[14]. Even combined with continual learning strategies, these methods struggle
to capture knowledge across modalities and fail to yield satisfactory
outcomes.

Inspired by human learning [2, 63], we devise a novel method named Reconstruct
before Query (RebQ). RebQ utilizes a pre-trained LMM with extensive multi-
modal knowledge against the modality-incomplete data [52, 35], which freezes
the parameters within the pre-trained LMM and leverages prompts to facilitate
parameter-efficient fine-tuning. Specifically, it decomposes unified prompts
into modality-specific ones and then further breaks them into components
stored in pools accessible via a key-query mechanism, significantly enhancing
knowledge transferability for subsequent tasks. However, accessing pools
requires corresponding modality-specific queries, which is not feasible when
the modality-incomplete data comes. Therefore, RebQ explicitly harnesses the
extensive multi-modal knowledge from the pre-trained LMM to reconstruct the
absent query for accessing the relevant components.

The main contributions are outlined as follows:

  * •

To the best of our knowledge, this paper is the first to introduce the
challenging Continual Missing Modality Learning task for non-static modality-
incomplete data. This setting is commonly encountered in real applications but
rarely exploited in academia.

  * •

We construct two datasets to benchmark the CMML task, showing that baselines
suffer from significant performance challenges for missing modality
techniques, even when combined with continual learning approaches.

  * •

We propose Reconstruct before Query (RebQ) to decompose prompts into modality-
specific components and explicitly harness the multi-modal knowledge from a
pre-trained model to reconstruct the absent query for accessing the relevant
components.

##  2 Related Work

###  2.1 Missing Modality

The issue of missing modalities presents a significant challenge in computer
vision [94, 93, 95, 96, 42, 43, 8, 99, 36, 44, 10, 9, 86, 15, 85, 57, 16, 91,
87], particularly when certain data modalities, integral during training,
become inaccessible at the inference stage. A breadth of studies [25, 50, 18,
19, 53, 52, 83, 35, 69, 77, 92, 22] has ventured into resolving this problem.
Some research endeavors [25, 50, 18, 19, 53] have harnessed auxiliary
modalities’ side information during training, empowering models to extrapolate
the absent modality information in the inference phase. This technique notably
improves the model’s effectiveness in missing modality learning. Conversely,
alternative methodologies [69, 77, 92, 22] strive to align embedding spaces
across diverse modalities, thereby allowing incomplete modalities to ascertain
their coordinates within a unified embedding framework during inference.

A recent and notable shift in the landscape has been the adoption of utilizing
tokens within pre-trained transformer architectures to represent the
information of missing modalities [52, 83, 35, 27]. This innovative approach
has gained considerable interest due to its ability to effectively encapsulate
modality information with limited resources. However, these approaches are
trained and tested on fixed datasets, struggling with the non-static data
common in real-world applications. In contrast, in the work, we target to
solve the missing-modality issue in non-static settings. We incorporate prompt
tokens to invoke the intrinsic knowledge within pre-trained models,
subsequently rendering them into concise surrogates for the missing
modalities, which not only streamlines the representation of absent
information but also capitalizes on knowledge in models, thereby enhancing the
robustness and adaptability of models in scenarios confronted with missing
modalities.

###  2.2 Continual Learning

Continual learning is dedicated to the enhancement of a model’s knowledge
through an ongoing series of tasks, crucially maintaining the integrity of
information acquired in earlier stages. This field can be broadly segmented
into four strategies: regularization-based, rehearsal-based, architecture-
based, and prompt-based approaches.

Regularization-based methods [1, 38, 88, 11, 78, 23, 40, 58] introduce
constraints during the training on subsequent tasks, ensuring the modification
of learned weights remains less divergent from established patterns. While
preserving previous knowledge, these methods can inadvertently restrain the
model, potentially undermining performance advancements due to over-emphasis
on stability.

Rehearsal-based methods [68, 17, 48, 60, 28, 46, 73, 74, 5, 100, 39, 45, 51,
97, 4, 65, 66, 90, 72] advocate for retaining a memory buffer composed of
exemplars from previous tasks to replay them in the current task. The efficacy
of such methods is inherently tied to the capacity of the memory buffer.
Nevertheless, privacy concerns may preclude the retention of specific samples,
imposing limitations on their feasibility.

Architecture-based methods devise distinct components for new tasks,
manifesting as network expansion [37, 62, 20, 64, 78, 26, 76, 4, 29, 13] or
the formulation of task-dedicated sub-networks [31, 55, 67, 34, 84]. Despite
achieving satisfactory task-specific precision, these methods need to
predicate the availability of task identity during the testing phase, which is
a condition not guaranteed in more dynamic and realistic task-agnostic
environments.

Prompt-based methods [82, 80, 81, 71, 75, 30, 32] emerge as marks of a
significant evolution in continual learning paradigms. They ingeniously
utilize learnable prompts for each task, extracting and applying knowledge
from pre-trained models. This strategy is not only highly efficient in terms
of parameter usage but also circumvents the need for extensive memory buffers
and elaborate architectural designs. Inspired by the potential of prompt-based
strategies, our study adopts pre-trained models as foundational structures,
capitalizing on prompt tokens to facilitate continual learning in scenarios
plagued by missing modalities.

##  3 Method

Figure 2: Pipeline of the RebQ framework (Text modality is unavailable in the
figure). The parameters of LMMs are frozen, and introducing prompt learning
enables parameter-efficient fine-tuning on user devices. When meeting the
missing modality, the available modality is utilized to generate memory
prompts from a Memory pool via a key-query mechanism, and memory prompts are
inserted into LMMs to reconstruct the missing modality as the modality-
specific query. Then, the two modality-specific queries are utilized to
generate modality-specific prompts from a text pool (Folder) and a visual pool
(Album), which are used to learn downstream tasks.

###  3.1 Problem Definition

In this paper, we focus on multi-modal learning with missing modalities during
continual fine-tuning. Additionally, it is crucial to develop the method
without the need to fine-tune the entire pre-trained model due to the limited
resources in user devices. To simplify but without loss of generality, we
consider a multi-modal dataset with text (t) and visual (v) modalities. Given
a specific task, we denote its training datasets by $\mathcal{D}$ is built
upon three subsets:

$\mathcal{D}=\\{\mathcal{D}^{\mathrm{t}},\mathcal{D}^{\mathrm{v}},\mathcal{D}^{\mathrm{c}}\\}.$
(1)

Specifically, the subset
$\mathcal{D}^{\mathrm{t}}=\\{(\mathbf{t}_{i},\mathbf{y}_{i})\\}_{i=1}^{L}$
contains modality-incomplete data samples with text only,
$\mathcal{D}^{\mathrm{v}}=\\{(\mathbf{v}_{i},\mathbf{y}_{i})\\}_{i=1}^{M}$
contains visual (image)-only data samples, and
$\mathcal{D}^{\mathrm{c}}=\\{(\mathbf{t}_{i},\mathbf{v}_{i},\mathbf{y}_{i})\\}_{i=1}^{N}$
contains modality-complete samples with both text and image, where
$\mathbf{t}_{i}$ denotes the embedding of textual content and $\mathbf{v}_{i}$
corresponds to the embedding of visual content, and
$\mathbf{y}_{i}\in\mathbb{R}^{C}$ is the label vector where $C$ is the number
of classes. Following [35], we assign dummy inputs
$\tilde{\mathbf{t}}/\tilde{\mathbf{v}}$ to represent missing-modality data and
obtain
$\tilde{\mathcal{D}}^{\mathrm{t}}=\\{(\mathbf{t}_{i},\tilde{\mathbf{v}},\mathbf{y}_{i})\\}_{i=1}^{L}$
and
$\tilde{\mathcal{D}}^{\mathrm{v}}=\\{(\tilde{\mathbf{t}},\mathbf{v}_{i},\mathbf{y}_{i})\\}_{i=1}^{M}$.
During the training, data from previous tasks is not available. In the
inference phase, the task identity is unknown.

###  3.2 Modality-Specific Prompt Learning

User devices only have limited resources, which cannot re-train or fine-tune
the entire models with numerous parameters. Thus, we turn our attention to the
prompt-based continual learning methods [82, 81, 71], which retrieve prompts
from a learnable prompt pool by using an image-wise prompt query while keeping
other parameters frozen to maintain the previous knowledge and learn new
skills, significantly reducing the resource requirements.

However, the unimodal prompt pool suffers from capturing knowledge within each
modality and entanglement between different modalities (shown in Section 4.2).
Therefore, for each modality, we create a modality-specific pool for storing
the decomposed modality-specific knowledge. Specifically, we create two
modality-specific prompt pools, Folder for texts and Album for images. Each
modality-specific prompt pool contains learnable attention weights
$\mathbf{A}^{m}\in\mathbb{R}^{D\times K}$, keys
$\mathbf{K}^{m}\in\mathbb{R}^{D\times K}$, where $D$ is the dimension, $K$ is
the size of the pool, and $m$ denotes a specific modality, which can be the
text modality (t) for Folder or the visual modality (v) for Album, _i.e._ ,
$m\in\\{\mathrm{t},\mathrm{v}\\}$. In the meanwhile, we denote the prompt pool
for each specific modality by
$\mathcal{P}^{m}=\\{\mathbf{P}^{m}_{k}\\}_{k=1}^{K}$, where
$\mathbf{P}^{m}_{k}\in\mathbb{R}^{D\times N_{\mathrm{p}}}$ and
$N_{\mathrm{p}}$ is the length of prompts.

We utilized prompt selection [71] for obtaining prompts, which consists of
three steps: 1) query generation, 2) weight computation, and 3) prompt
aggregation. Below we illustrate these three steps, respectively.

Query generation. We convert the text input $\mathbf{t}$ into a sequence of
token embedding
$\mathbf{X}^{\mathrm{t}}=[\mathbf{x}^{\mathrm{t}}_{1},\cdots,\mathbf{x}^{\mathrm{t}}_{P}]$
and crop the image $\tilde{\mathbf{v}}$ into a sequence of patch embedding
$\mathbf{X}^{\mathrm{v}}=[\mathbf{x}_{1}^{\mathrm{v}},\cdots,\mathbf{x}_{Q}^{\mathrm{v}}]$.
We concatenate the $\mathbf{X}^{\mathrm{t}}$ and $\mathbf{X}^{\mathrm{v}}$ as
well as two special tokens $\mathbf{x}_{\mathrm{cls}}^{\mathrm{t}}$ and
$\mathbf{x}_{\mathrm{cls}}^{\mathrm{v}}$ into a long sequence as the input of
an LMM, and generate an output sequence of the same length:

$[\tilde{\mathbf{x}}_{\mathrm{cls}}^{\mathrm{t}},\tilde{\mathbf{X}}^{\mathrm{t}},\tilde{\mathbf{x}}_{\mathrm{cls}}^{\mathrm{v}},\tilde{\mathbf{X}}^{\mathrm{v}}]=\mathrm{LMM}([\mathbf{x}_{\mathrm{cls}}^{\mathrm{t}},\mathbf{X}^{\mathrm{t}},\mathbf{x}_{\mathrm{cls}}^{\mathrm{v}},\mathbf{X}^{\mathrm{v}}]).$
(2)

We set $\tilde{\mathbf{x}}_{\mathrm{cls}}^{\mathrm{t}}$ as
$\mathbf{q}^{\mathrm{t}}$, which is the query embedding for text and
$\tilde{\mathbf{x}}_{\mathrm{cls}}^{\mathrm{v}}$ as $\mathbf{q}^{\mathrm{v}}$,
which is the query embedding for image.

Weight computation. The generated text/image query embedding is further
utilized to generate the weight vector
$\mathbf{w}^{m}=[{w}^{m}_{1},\cdots,{w}^{m}_{K}]$
($m\in\\{\mathrm{t},\mathrm{v}\\}$) as:

$\displaystyle
w^{\mathrm{t}}_{k}={s}(\mathbf{q}^{\mathrm{t}}\odot\mathbf{A}^{\mathrm{t}}_{k},\mathbf{K}^{\mathrm{t}}_{k}),\forall
k\in[1,K],$ (3) $\displaystyle
w^{\mathrm{v}}_{k}=s(\mathbf{q}^{\mathrm{v}}\odot\mathbf{A}^{\mathrm{v}}_{k},\mathbf{K}^{\mathrm{v}}_{k}),\forall
k\in[1,K],$

where $s(\cdot,\cdot)$ measures consine similarity; $\odot$ is Hadamard
product; $\mathbf{A}^{\mathrm{t}}_{k}/\mathbf{A}^{\mathrm{v}}_{k}$ denotes the
$k$-th column of $\mathbf{A}^{\mathrm{t}}/\mathbf{A}^{\mathrm{v}}$.

Prompt Aggreation. Finally, we adopt the computed weights
$\mathbf{w}^{\mathrm{t}}/\mathbf{w}^{\mathrm{v}}$ in Equation (3) to generate
modality-specific prompts by weighted summation over the prompt components:

$\displaystyle\mathbf{P}^{m}=\sum_{k=1}^{K}w^{m}_{k}\cdot\mathbf{P}^{m}_{k},\forall
m\in\\{\mathrm{t},\mathrm{v}\\}.$ (4)

The generated modality-specific prompts $\mathbf{P}^{\mathrm{t}}$ and
$\mathbf{P}^{\mathrm{v}}$ will collaborate in a pre-trained LMM for
facilitating downstream multi-modality understanding tasks. We term this
process as multi-modality prompt collaboration, which we will introduce the
details in Section 3.4.

Note that, the modality-specific prompt learning relies on condition that a
sample contains both textual content $\mathbf{t}$ and visual content
$\mathbf{v}$ for generating text query embedding $\mathbf{q}^{\mathrm{t}}$ and
visual query embedding $\mathbf{q}^{\mathrm{v}}$. Nevertheless, as above
mentioned, in real scenarios, we might have only access to the content of a
single modality and fail to explicitly obtain both $\mathbf{q}^{\mathrm{t}}$
and $\mathbf{q}^{\mathrm{v}}$. To address the modality-missing challenge, we
proposed to reconstruct the query embedding of the missing modality based on a
memory bank $\mathcal{M}$. We will introduce the detailed process of
reconstructing the missing query embedding in Section 3.3.

###  3.3 Missing Query Reconstruction

We create a memory pool $\mathcal{M}=\\{\mathbf{M}_{i}\\}_{i=1}^{R}$
containing multi-modal reconstruction prompts and harness the knowledge in a
pre-trained large multi-modal model (LMM) built upon a stack of Transformer
blocks to reconstruct the missing-modality query $\mathbf{q}^{\mathrm{t}}$ or
$\mathbf{q}^{\mathrm{v}}$.

Let’s consider a sample with only text modality available
$(\mathbf{t},\tilde{\mathbf{v}},\mathbf{y})$ where $\mathbf{t}$ denotes the
text, $\tilde{\mathbf{v}}$ is a dummy image. We convert the text input
$\mathbf{t}$ into a sequence of token embedding
$\mathbf{X}^{\mathrm{t}}=[\mathbf{x}^{\mathrm{t}}_{1},\cdots,\mathbf{x}^{\mathrm{t}}_{P}]$
and crop the image $\tilde{\mathbf{v}}$ into a sequence of patch embedding
$\tilde{\mathbf{X}}^{\mathrm{v}}=[\tilde{\mathbf{x}}_{1}^{\mathrm{v}},\cdots,\tilde{\mathbf{x}}_{Q}^{\mathrm{v}}]$.
Then, we concatenate the $\mathbf{X}^{\mathrm{t}}$ and
$\mathbf{X}^{\mathrm{v}}$ as well as a special token
$\mathbf{x}_{\mathrm{cls}}$ into a long sequence as the input of an LMM, and
generate an output sequence of the same length:

$[\bar{\mathbf{x}}_{\mathrm{cls}},\bar{\mathbf{X}}^{\mathrm{t}},\bar{\mathbf{X}}^{\mathrm{v}}]=\mathrm{LMM}([\mathbf{x}_{\mathrm{cls}},\mathbf{X}^{\mathrm{t}},\tilde{\mathbf{X}}^{\mathrm{v}}]).$
(5)

We use $\bar{\mathbf{x}}_{\mathrm{cls}}$ as the query and conduct prompt
selection based on the memory template pool
$\mathcal{M}=\\{\mathbf{M}_{i}\\}_{i=1}^{M}$ and generate the prompts
$\mathbf{M}=[\mathbf{m}_{1},\cdots,\mathbf{m}_{N_{\mathrm{p}}}]$ using the
same prompt selection process described in Equation (2) (3) (4).

We further concatenate the token embeddings $\mathbf{X}^{\mathrm{t}}$, the
patch embeddings $\tilde{\mathbf{X}}^{\mathrm{v}}$, the prompts $\mathbf{M}$
from the memory pool and the special tokens $\mathbf{x}_{\mathrm{cls}}$ into a
long sequence as the input of the same LMM used in Equation (5):

$[\hat{\mathbf{x}}_{\mathrm{cls}},\hat{\mathbf{M}},\hat{\mathbf{X}}^{\mathrm{t}},\hat{\mathbf{X}}^{\mathrm{v}}]=\mathrm{LMM}([\mathbf{x}_{\mathrm{cls}},\mathbf{M},\mathbf{X}^{\mathrm{t}},\tilde{\mathbf{X}}^{\mathrm{v}}]).$
(6)

We set $\hat{\mathbf{x}}_{\mathrm{cls}}$ in the output as reconstructed visual
query embedding for the downstream task:

$\hat{\mathbf{q}}^{\mathrm{v}}=\hat{\mathbf{x}}_{\mathrm{cls}}.$ (7)

In a similar manner, given a sample with only visual modality available
$(\tilde{\mathbf{t}},{\mathbf{v}},\mathbf{y})$ where $\tilde{\mathbf{t}}$ is a
dummy text, we can reconstruct the text query embedding
$\hat{\mathbf{q}}^{\mathrm{t}}$.

Algorithm 1 Missing Query Reconstruction

Input: Modality-incomplete data
$({\mathbf{t}},\tilde{\mathbf{v}},\mathbf{y})$; Memory pool $\mathcal{M}$;
Large multi-modal model (LMM)
Output: Reconstructed query $\hat{\mathbf{q}}^{\mathrm{v}}$

1:# Generate memory query embedding for modality-incomplete data, described in
Equation (5)

2:$\bar{\mathbf{x}}_{\mathrm{cls}}=\mathrm{LMM}({\mathbf{t}},\tilde{\mathbf{v}})$

3:# Select prompts from the memory pool, described in Equation (3) (4)

4:$\mathbf{M}=\operatorname{select\\_prompt}(\bar{\mathbf{x}}_{\mathrm{cls}},\mathcal{M})$

5:# Reconstruct the missing query with memory prompts, described in Equation
(6) (7)

6:$\hat{\mathbf{q}}^{m}=\mathrm{LMM}(\mathbf{t},\tilde{\mathbf{v}},\mathbf{M})$

Algorithm 2 RebQ Pipeline

Input: Modality-incomplete data
$({\mathbf{t}},\tilde{\mathbf{v}},\mathbf{y})$; Memory prompt pool
$\mathcal{M}$; Text prompt pool $\mathcal{P}^{\mathrm{t}}$; Image prompt pool
$\mathcal{P}^{\mathrm{v}}$; Large multi-modal model (LMM);
Output: Predicted outputs $\hat{\mathbf{y}}$

1:# Generate query embedding for the available text modality, described in
Equation (2)

2:$\mathbf{q}^{\mathrm{t}},\bar{\mathbf{x}}_{\mathrm{cls}}=\mathrm{LMM}(\mathbf{t},\tilde{\mathbf{v}})$

3:# Reconstruct missing vision query embeddings, described in Algorithm 1

4:$\hat{\mathbf{q}}^{\mathrm{v}}=\mathrm{reconstruct}(\textbf{t},\bar{\mathbf{x}}_{\mathrm{cls}},\mathcal{M})$

5:# Select prompts from the pool, described in Equation (3) (4)

6:$\mathbf{P}^{\mathrm{t}}=\operatorname{select\\_prompt}(\mathbf{q}^{\mathrm{t}},\mathcal{P}^{\mathrm{t}})$

7:$\mathbf{P}^{\mathrm{v}}=\operatorname{select\\_prompt}(\hat{\mathbf{q}}^{\mathrm{v}},\mathcal{P}^{\mathrm{v}})$

8:# Predict outputs for downstream tasks, described in Equation (9)

9:$\hat{\mathbf{y}}=\mathrm{LMM}(\mathbf{t},\tilde{\mathbf{v}},\mathbf{P}^{\mathrm{t}},\mathbf{P}^{\mathrm{v}})$

For each modality-complete sample,
$(\mathbf{t}_{i},\mathbf{v}_{i},\mathbf{y}_{i})$, we construct its missing-
modality counterparts by replacing the original embedding
$\mathbf{t}_{i}/\mathbf{v}_{i}$ by the dummy input
$\tilde{\mathbf{t}}_{i}/\tilde{\mathbf{v}}_{i}$ to obtain the visual-only
sample $(\tilde{\mathbf{t}}_{i},{\mathbf{v}}_{i},\mathbf{y}_{i})$ and the
text-only sample $(\mathbf{t}_{i},\tilde{\mathbf{v}}_{i},\mathbf{y}_{i})$.
After that, we use the miss-modality samples to generate the reconstructed
text query embedding $\hat{\mathbf{q}}^{\mathrm{t}}_{i}$ and the reconstructed
visual query embedding $\hat{\mathbf{q}}^{\mathrm{v}}_{i}$. In the meanwhile,
we obtain the ground-truth text/visual query embedding
$\mathbf{q}^{\mathrm{t}}/\mathbf{q}^{\mathrm{v}}$ based on the original
text/visual input as Equation (2). We compute the reconstruction loss to learn
the prompts for assisting reconstruction:

$\displaystyle\mathcal{L}_{r}=\frac{1}{N}\Big{\\{}\sum_{i=1}^{N}\|\mathbf{q}^{\mathrm{t}}_{i}-\hat{\mathbf{q}}^{\mathrm{t}}_{i}\|_{2}^{2}+\|\mathbf{q}^{\mathrm{v}}_{i}-\hat{\mathbf{q}}^{\mathrm{v}}_{i}\|_{2}^{2}\Big{\\}}.$
(8)

###  3.4 Multi-modality Prompt Collaboration

The textual/visual content $\mathbf{t}$/$\mathbf{v}$ and the modality-specific
prompts $\mathbf{P}^{\mathrm{t}}$/$\mathbf{P}^{\mathrm{v}}$ obtained from
Equation (4) serve as the input of the large multi-modal model (LMM) for
target multi-modality understanding task:

$\hat{\mathbf{y}}=\mathrm{LMM}(\mathbf{t},\mathbf{v},\mathbf{P}^{\mathrm{t}},\mathbf{P}^{\mathrm{v}}),$
(9)

where $\hat{\mathbf{y}}\in\mathbb{R}^{C}$ is the predicted logits and $C$
denotes the number of classes. Specifically, to obtain $\hat{\mathbf{y}}$, we
first obtain the embedding of cls token in the output of the LMM,
$\mathbf{x}^{\mathrm{out}}_{\mathrm{cls}}$. After that, we project cls token
embedding into the predicted logits $\hat{\mathbf{y}}$ by a head $h(\cdot)$
with a fully-connect layer:

$\hat{\mathbf{y}}=h(\mathbf{x}^{\mathrm{out}}_{\mathrm{cls}})$ (10)

In this work, we address the classification task and adopt cross-entropy (CE)
loss (binary cross-entropy loss for the multi-label classification task) for
training:

$\mathcal{L}_{\mathrm{c}}=\mathrm{CE}(\hat{\mathbf{y}},\mathbf{y}),$ (11)

where $\mathbf{y}$ denotes the groundtruth class label vector. In the training
process, we freeze the weights of the LMM and learn the memory template pool
$\mathcal{M}$, text/image prompt pools
$\mathcal{P}^{\mathrm{t}}/\mathcal{P}^{\mathrm{v}}$, and the weights of the
prediction head $h(\cdot)$. The objective is jointly optimized on two loss
functions:

$\displaystyle\mathcal{L}=\mathcal{L}_{c}+\lambda\mathcal{L}_{r},$ (12)

where $\lambda$ is a hyper-parameter to balance the two loss values.

The algorithms of reconstruction stage and RebQ are listed in Algorithm 1 and
Algorithm 2, respectively.

##  4 Experiments

Datasets and CMML Setting.  We construct two curated datasets to facilitate
the CMML task: UPMC-Food101-CMML and MM-IMDb-CMML. UPMC-Food101-CMML, derived
from the UPMC-Food101 [79] dataset, containing $101$ food categories and
$61,142$ training, $6,846$ validation, and $22,716$ test image-text pairs. We
eliminated the category with the fewest samples and allocated the categories
into $10$ distinct sessions. The constructed MM-IMDb-CMML dataset is based on
the MM-IMDb [59] dataset. It is a multi-label classification dataset across
$27$ distinct movie genres, consisting of $15,552$ training, $2,608$
validation, and $7,799$ test image-text pairs. Adhering to the multi-label
continual learning paradigm [12], we refine the dataset by excluding $7$ most
infrequently represented categories, subsequently organizing the remaining
into $5$ sessions, each incorporating $4$ categories. To set up the missing
modality, we focus on a more challenging task wherein the absence of modality
can occur in both the training and inference stages. Following [35], we
quantify the missing ratio as $\eta\%$, representing the ratio of data with
incomplete modalities within each session. For scenarios where only one
modality is missing, the proportion of modal-incomplete to modal-complete data
adheres to a ratio of $\eta\%$ to $1-\eta\%$. Besides, in instances where both
image and text modalities are missing, the dataset contains $\frac{\eta}{2}\%$
image-only data and $\frac{\eta}{2}\%$ text-only data, complemented with
$1-\eta\%$ of data retaining both modalities. This configuration highlights
the challenges posed by modality scarcity and underscores the robustness in
continual missing modality learning environments.

Comparison Methods. Because our work introduces a novel task, it presents
unique challenges in direct comparison with existing methods. We have strived
to ensure a comprehensive and fair evaluation of our approach. From the
missing modality perspective, MAP [35] and MSP [27] are selected because they
focus on leveraging pre-trained LMMs to solve the image and text missing
modality during both the training and testing stages. From the continual
learning perspective, considering that our approach uses prompt learning to
harness the rich knowledge in pre-trained LMMs, we choose two prompt-based
continual learning algorithms, L2P [82] and DualPrompt [81], as comparisons.

Table 1: Results on UPMC-Food101-CMML. The evaluation metrics, AP and FG, are
based on the classification accuracy. Missing modality occurs in both training
and testing.

$\eta$ | #Image | #Text | MAP [35] | MSP [27] | L2P [82] | DualPrompt [81] | RebQ
---|---|---|---|---|---|---|---
AP ($\uparrow$) | FG ($\downarrow$) | AP ($\uparrow$) | FG ($\downarrow$) | AP ($\uparrow$) | FG ($\downarrow$) | AP ($\uparrow$) | FG ($\downarrow$) | AP ($\uparrow$) | FG ($\downarrow$)
$10\%$ | $100\%$ | $90\%$ | 20.66 | 82.50 | 21.45 | 80.03 | 34.09 | 5.80 | 59.56 | 3.38 | 68.67 | 9.50
$90\%$ | $100\%$ | 21.53 | 82.20 | 23.29 | 78.98 | 35.21 | 5.51 | 59.90 | 1.72 | 72.46 | 7.64
$95\%$ | $95\%$ | 22.84 | 80.13 | 22.12 | 79.35 | 34.00 | 6.55 | 58.18 | 6.32 | 71.06 | 8.22
$30\%$ | $100\%$ | $70\%$ | 16.57 | 83.77 | 21.37 | 79.95 | 30.40 | 6.53 | 51.86 | 6.46 | 62.06 | 10.76
$70\%$ | $100\%$ | 24.00 | 77.12 | 22.21 | 76.02 | 34.75 | 4.66 | 52.66 | 5.11 | 71.62 | 6.44
$85\%$ | $85\%$ | 20.66 | 79.81 | 21.89 | 79.01 | 29.76 | 6.87 | 48.56 | 8.73 | 66.37 | 8.07
$50\%$ | $100\%$ | $50\%$ | 18.18 | 79.77 | 17.88 | 77.31 | 28.95 | 6.33 | 47.70 | 6.09 | 55.87 | 12.07
$50\%$ | $100\%$ | 23.85 | 75.59 | 21.57 | 77.99 | 32.18 | 4.55 | 50.22 | 5.03 | 69.23 | 5.84
$75\%$ | $75\%$ | 18.66 | 79.80 | 18.10 | 78.04 | 25.30 | 6.17 | 43.67 | 8.52 | 62.40 | 8.24
$70\%$ | $100\%$ | $30\%$ | 17.68 | 77.82 | 19.29 | 78.17 | 26.57 | 5.80 | 43.09 | 8.96 | 50.00 | 12.47
$30\%$ | $100\%$ | 22.48 | 75.21 | 21.22 | 75.21 | 30.43 | 3.99 | 50.28 | 5.19 | 69.41 | 3.73
$65\%$ | $65\%$ | 20.00 | 75.95 | 19.76 | 76.54 | 24.62 | 5.66 | 40.69 | 7.26 | 59.92 | 8.56
$90\%$ | $100\%$ | $10\%$ | 16.92 | 76.60 | 18.62 | 75.14 | 24.94 | 5.60 | 37.26 | 11.46 | 48.15 | 12.76
$10\%$ | $100\%$ | 24.89 | 70.84 | 21.69 | 71.23 | 29.77 | 4.54 | 51.16 | 4.80 | 67.71 | 4.71
$55\%$ | $55\%$ | 18.41 | 75.19 | 20.45 | 74.31 | 24.62 | 5.66 | 40.69 | 7.29 | 54.67 | 8.78

Evaluation Metrics. In the CMML setting, we employ two standard continual
learning metrics [6, 7, 48, 89]: Average Performance (AP) and Average
Forgetting (FG). AP is the average performance calculated from all tasks.
Assume $a_{i,j}$ is the testing performance on the $i$-th task after training
the model on the $j$-th task, $\mathrm{AP}=\frac{1}{T}\sum_{t=1}^{T}a_{t,T}$.
In contrast, FG measures performance degradation in subsequent tasks,
$\mathrm{FG}=\frac{1}{T-1}\sum_{t=1}^{T-1}\max_{z\in\\{t,\cdots,T-1\\}}(a_{t,z}-a_{t,T})$.
We compute Accuracy in UPMC-Food101-CMML and F1-Macro in MM-IMDb-CMML.

Implementation Details. Following [35], we utilize
ViLT222https://huggingface.co/dandelin/vilt-b32-mlm [33] as the pre-trained
LMM. An empty string denotes the dummy text input (i.e.,
$\tilde{\mathbf{t}}$), and an image with all pixel values equal one as the
dummy image input (i.e., $\tilde{\mathbf{v}}$). All the parameters in the LMM
are frozen and a learnable fully-connected layer is added as a classifier
head. For memory, image, and text prompt pools, we set the length of prompts,
the number of prompt layers, and the prompt pool size to $8$, $8$, and $128$,
respectively. All the prompts in pools are randomly initialized and then
optimized during training process. The batch size is $4$. We use the AdamW
[49] optimizer, and the learning rate is set to $1$e-$4$. We adopt the linear
warmup cosine annealing scheduler to adjust the learning rate, and $10\%$ of
the total training steps are used to warm up. $\lambda$ is set to $0.01$. All
experiments are conducted on one NVIDIA A5000 GPU.

Table 2: Results on MM-IMDb-CMML. The evaluation metrics, AP and FG, are based
on the F1-Macro. Missing modality occurs during both training and testing.

$\eta$ | #Image | #Text | MAP [35] | MSP [27] | L2P [82] | DualPrompt [81] | RebQ
---|---|---|---|---|---|---|---
AP ($\uparrow$) | FG ($\downarrow$) | AP ($\uparrow$) | FG ($\downarrow$) | AP ($\uparrow$) | FG ($\downarrow$) | AP ($\uparrow$) | FG ($\downarrow$) | AP ($\uparrow$) | FG ($\downarrow$)
$10\%$ | $100\%$ | $90\%$ | 15.77 | 34.67 | 19.49 | 35.11 | 13.67 | 8.96 | 22.79 | 20.42 | 24.19 | 32.46
$90\%$ | $100\%$ | 16.63 | 33.30 | 19.01 | 34.89 | 13.54 | 8.26 | 24.82 | 24.09 | 27.35 | 28.88
$95\%$ | $95\%$ | 17.24 | 31.28 | 15.69 | 32.17 | 13.90 | 8.42 | 24.19 | 24.04 | 26.90 | 28.26
$30\%$ | $100\%$ | $70\%$ | 19.85 | 32.14 | 20.01 | 34.87 | 10.68 | 8.03 | 22.96 | 23.96 | 24.75 | 25.86
$70\%$ | $100\%$ | 17.19 | 31.09 | 18.31 | 32.45 | 10.98 | 7.27 | 20.98 | 19.67 | 26.68 | 27.89
$85\%$ | $85\%$ | 14.44 | 33.29 | 14.20 | 35.57 | 11.58 | 7.90 | 22.18 | 20.72 | 25.80 | 26.95
$50\%$ | $100\%$ | $50\%$ | 19.19 | 36.18 | 15.93 | 35.29 | 9.54 | 5.54 | 20.04 | 21.48 | 23.89 | 25.27
$50\%$ | $100\%$ | 18.83 | 32.02 | 16.78 | 34.14 | 9.77 | 4.88 | 19.97 | 20.24 | 26.45 | 27.53
$75\%$ | $75\%$ | 17.36 | 29.67 | 16.24 | 31.81 | 10.08 | 5.23 | 20.32 | 19.09 | 25.31 | 25.88
$70\%$ | $100\%$ | $30\%$ | 17.45 | 27.14 | 18.01 | 27.32 | 7.26 | 5.37 | 16.80 | 13.82 | 22.41 | 24.39
$30\%$ | $100\%$ | 16.09 | 34.59 | 14.30 | 35.72 | 7.49 | 5.20 | 18.42 | 15.32 | 26.27 | 27.56
$65\%$ | $65\%$ | 17.20 | 29.39 | 18.27 | 29.34 | 6.93 | 6.23 | 16.19 | 15.76 | 23.77 | 25.26
$90\%$ | $100\%$ | $10\%$ | 18.85 | 25.03 | 18.01 | 29.14 | 5.98 | 5.82 | 15.50 | 11.27 | 21.37 | 23.66
$10\%$ | $100\%$ | 17.79 | 31.65 | 17.88 | 32.19 | 6.29 | 5.09 | 21.22 | 18.38 | 26.22 | 27.50
$55\%$ | $55\%$ | 16.28 | 27.89 | 16.20 | 31.47 | 6.84 | 6.39 | 16.82 | 8.44 | 25.42 | 25.82

Figure 3: Results of each incremental stage on UPMC-Food101-CMML. $\eta$ is
$70\%$.

###  4.1 Main Results

Table 1 summarizes the experimental results of the UPMC-Food101-CMML dataset,
demonstrating that RebQ consistently surpasses the comparative method across
various configurations in both Average Precision (AP) and Forgetting (FG).
Unlike the MAP and MSP, which employ prompt vectors for distinct missing types
(i.e., text-only or image-only) and encounters significant catastrophic
forgetting due to overfitting of prompt vectors on the current task, RebQ
decomposes the unified prompts into modality-specific components stored in
pools and accesses the pools via reconstructed missing queries, markedly
improving knowledge transfer for subsequent tasks. L2P and DualPrompt also
employ the prompt pool, but they cannot access the pool when modality-missing
data arrives, leading to lower AP compared with our proposed RebQ. Note that
L2P and DualPrompt achieve lower FG in some cases. The reason is their
relatively lower precision, which limits the decreasing range, as shown in
Figure 3. Furthermore, a direct correlation is observed between the quantity
of text data and improved performance. We argue that the texts in the UPMC-
Food101-CMML dataset are webpage titles collected from the Internet, which may
explicitly include the class label and introduce a strong classification bias.

Table 2 illustrates the results for the MM-IMDb-CMML dataset, where RebQ
markedly surpasses the baseline methods, demonstrating consistent superiority
even in the challenging multi-label continual missing modality learning
setting. Additionally, we also note a significant correlation between the
quality of text modality and AP. We argue that the UPMC-Food101-CMML dataset
comprises movie posters as visual inputs and plots as text inputs. The latter
contains detailed descriptions, which are beneficial to classify movie genres.

Figure 3 illustrates the results of each incremental stage. The compared
methods suffer from catastrophic forgetting, leading to severe performance
drops. However, our proposed RebQ maintains the performance during incremental
stages, demonstrating the effectiveness of our method.

###  4.2 Ablation Studies

In this section, we explore various RebQ variants to assess the importance of
each component. Experiments are on the UPMC-Food101-CMML dataset with a
missing ratio $\eta$ of $70\%$, where both image and text modalities are
absent.

Table 3: Ablation study on the UPMC-Food101-CMML dataset. Method | AP ($\uparrow$) | FG ($\downarrow$)
---|---|---
RebQ | 59.92 | 8.56
w/o Reconstruction | 39.22 | 16.27
w/o Modality-Specific Query | 51.82 | 11.92
w/o Memory Pool | 53.36 | 12.04
w/o Modality-Specific Pool | 52.07 | 12.29
Figure 4: Prompt designs. (a) The pool size of memory/image/text pool. (b) The
layers that are inserted prompts. (c) The prompt length.

Reconstruction. Vanilla modality-specific prompts assume the data is modality-
complete, and prompts can be optimized during training with fully paired data,
which cannot handle the CMML setting. We remove the reconstruction stage and
directly use the modality-specific queries described in Equation (2) to
generate prompts from pools, denoted as “w/o Reconstruction.” Table 3
illustrates that the performance of “w/o Reconstruction” is significantly
decreasing due to the query embeddings from the absent modality lacking
substantial semantic information, thereby hindering effective prompt
generation. To further assess the impact, we introduce a variant “w/o
Modality-Specific Query” to benchmark the performance using only the available
modalities. Specifically, if only one modality is available, we utilize it to
generate the single-modality prompts and insert them into the pre-trained LMM.
The deteriorating results of the “Unimodal Query” underscore the effectiveness
of multi-modality prompt collaboration. Besides, we argue that missing
modalities can potentially be reconstructed through prompting in pre-trained
LMMs. However, it assumes a consistent data distribution, a condition not met
in continuous fine-tuning settings. Thus, we develop “w/o Memory Pool,”
relying on a singular memory prompt vector instead of a memory pool. This
approach, as the results indicate, leads to catastrophic forgetting. Finally,
we employ t-SNE [54] to analyze the query distribution. By removing one
modality from a modality-complete dataset $\mathcal{D}^{\mathrm{c}}$, we
generated two modality-incomplete datasets $\mathcal{D}^{\mathrm{t}}$ and
$\mathcal{D}^{\mathrm{v}}$. Subsequently, we extracted query embeddings from
$\mathcal{D}^{\mathrm{c}}$ directly using the pre-trained LMM as the “Ground
Truth.” “w/ Reconstruction” refers to the extraction of query embeddings from
$\mathcal{D}^{\mathrm{t}}$ and $\mathcal{D}^{\mathrm{v}}$ inclusive of the
reconstruction stage, while “w/o Reconstruction” denotes extraction without
the reconstruction stage. As depicted in Figure 5, after training with only a
tiny partial of full modal data, the distribution alignment of “Ground Truth”
and “w/ Reconstruction” is notably closer compared to “w/o Reconstruction,”
validating the reconstruction effectiveness of RebQ. The hyper-parameter
$\lambda$ is utilized to balance the classification and reconstruction losses.
Table 6 shows the results of sensitivity.

Table 4: Comparison of different ways of prompt insertion. Attention denotes the prompts inserted into the key and value in MSA layers. Input means the prompts are appended to the input tokens. Image | Text | AP ($\uparrow$) | FG ($\downarrow$)
---|---|---|---
Attention | Attention | 59.92 | 8.56
Attention | Input | 41.61 | 3.32
Input | Attention | 53.36 | 8.13
Input | Input | 37.54 | 2.59
Table 5: Comparison of different ways of prompt generation. Pool denotes the prompts are generated from a prompt pool. Vector means the prompts are a vector. Image | Text | AP ($\uparrow$) | FG ($\downarrow$)
---|---|---|---
Pool | Pool | 59.92 | 8.56
Pool | Vector | 55.41 | 10.14
Vector | Pool | 52.54 | 10.77
Vector | Vector | 46.38 | 11.43

Modality-Specific Prompt. To further assess the effectiveness of the modality-
specific prompt pool, we conducted experiments using a single prompt pool,
referred to as “Unimodal Pool” in Table 3. The results underscore the
significance of modality-specific pools in decomposing knowledge and enhancing
knowledge transferability for successive fine-tuning tasks. The performance is
also influenced by various prompting strategies, including the size of the
prompt pool, the layer at which prompts are inserted, and the length of the
prompts. Figure 4 (a) compares performances across different pool sizes,
revealing that smaller pools are still prone to catastrophic forgetting.
Performance improves up to a certain pool size, beyond which further
enlargement does not yield improvement. Figure 4 (b) illustrates the
performance across different numbers of prompted transformer layers in the
LMM. For instance, a value of $8$ indicates that prompts are inserted into the
first $8$ layers. The results reveal that integrating prompts into too few
layers restricts expressive capacity, whereas incorporating them into an
excessive number of layers often leads to overfitting. Figure 4 (c) presents a
comparison of prompt lengths, where shorter lengths limit learning capacity,
and excessively long lengths result in performance decreasing. Additionally,
we conduct experiments to demonstrate that prompt pools, rather than vectors,
improve the knowledge transfer transferability, illustrated in Table 5. “Pool”
represents the use of a prompt pool for a modality, whereas “Vector” refers to
learnable vectors. The outcomes indicate that prompt pools substantially
mitigate forgetting and maintain average performance. Previous works [35, 81]
have highlighted the criticality of prompt insertion position. We attach the
prompts to key and value layers, denoted as “Attention.” Appending prompts
into the input tokens is denoted as “Input.” The results from Table 4 show
that the “Attention” improves the results in the CMML setting.

Figure 5: t-SNE visualization of Reconstruction. Each point represents a query embedding of dimension $768$. Table 6: Sensitivity of $\lambda$ on UPMC-Food101-CMML. $\eta$ is $70\%$. $\lambda$ | 1e-4 | 1e-3 | 1e-2 | 1e-1 | 1
---|---|---|---|---|---
AP | 58.91 | 59.37 | 59.84 | 59.92 | 59.82

##  5 Conclusion

In this paper, we propose a new task, Continual Missing Modality Learning,
which aims to address the critical issue of fine-tuning large multi-modal
models (LMMs) on resource-constrained devices with modality-incomplete stream
data. Additionally, we devise the Reconstruct before Query (RebQ) framework to
decompose the modal-specific knowledge into prompt pools and then enable
multi-modal knowledge collaboration to enhance knowledge transferability for
subsequent fine-tuning tasks. Moreover, to obtain the missing query for
accessing corresponding pools, we explicitly harness the rich multi-modal
knowledge from pre-trained models to reconstruct the missing query. Our
experiments confirm RebQ’s effectiveness in maintaining performance and
preventing knowledge degradation when faced with modal-incomplete stream data.
Our research not only enhances the adaptability of LMMs in privacy-sensitive
and resource-limited environments but also opens avenues for further research
into more robust and efficient multi-modal learning techniques.

## References

  * [1] Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., Tuytelaars, T.: Memory aware synapses: Learning what (not) to forget. In: ECCV. Lecture Notes in Computer Science, vol. 11207, pp. 144–161. Springer (2018)
  * [2] Alvarez, P., Squire, L.R.: Memory consolidation and the medial temporal lobe: a simple network model. Proceedings of the national academy of sciences 91(15), 7041–7045 (1994)
  * [3] Bai, Y., Xu, X., Liu, Y., Khan, S., Khan, F., Zuo, W., Goh, R.S.M., Feng, C.: Sentence-level prompts benefit composed image retrieval. CoRR abs/2310.05473 (2023)
  * [4] Bhat, P.S., Zonooz, B., Arani, E.: Task-aware information routing from common representation space in lifelong learning. In: The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net (2023), https://openreview.net/pdf?id=-M0TNnyWFT5
  * [5] Cha, S., Cho, S., Hwang, D., Hong, S., Lee, M., Moon, T.: Rebalancing batch normalization for exemplar-based class-incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20127–20136 (2023)
  * [6] Chaudhry, A., Dokania, P.K., Ajanthan, T., Torr, P.H.S.: Riemannian walk for incremental learning: Understanding forgetting and intransigence. In: ECCV. Lecture Notes in Computer Science, vol. 11215, pp. 556–572. Springer (2018)
  * [7] Chaudhry, A., Ranzato, M., Rohrbach, M., Elhoseiny, M.: Efficient lifelong learning with A-GEM. In: ICLR. OpenReview.net (2019)
  * [8] Chen, J., Zhu, D., Shen, X., Li, X., Liu, Z., Zhang, P., Krishnamoorthi, R., Chandra, V., Xiong, Y., Elhoseiny, M.: Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478 (2023)
  * [9] Chen, L., Li, B., Shen, S., Yang, J., Li, C., Keutzer, K., Darrell, T., Liu, Z.: Large language models are visual reasoning coordinators. CoRR abs/2310.15166 (2023)
  * [10] Chen, W., Spiridonova, I., Yang, J., Gao, J., Li, C.: Llava-interactive: An all-in-one demo for image chat, segmentation, generation and editing. CoRR abs/2311.00571 (2023)
  * [11] Dong, J., Liang, W., Cong, Y., Sun, G.: Heterogeneous forgetting compensation for class-incremental learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11742–11751 (2023)
  * [12] Dong, S., Luo, H., He, Y., Wei, X., Cheng, J., Gong, Y.: Knowledge restore and transfer for multi-label class-incremental learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 18711–18720 (2023)
  * [13] Ermis, B., Zappella, G., Wistuba, M., Rawal, A., Archambeau, C.: Memory efficient continual learning with transformers. Advances in Neural Information Processing Systems 35, 10629–10642 (2022)
  * [14] French, R.M.: Catastrophic interference in connectionist networks: Can it be predicted, can it be prevented? In: NIPS. pp. 1176–1177. Morgan Kaufmann (1993)
  * [15] Fu, T., Hu, W., Du, X., Wang, W.Y., Yang, Y., Gan, Z.: Guiding instruction-based image editing via multimodal large language models. CoRR abs/2309.17102 (2023)
  * [16] Fu, T., Yu, L., Zhang, N., Fu, C., Su, J., Wang, W.Y., Bell, S.: Tell me what happened: Unifying text-guided video completion via multimodal masked video generation. In: CVPR. pp. 10681–10692. IEEE (2023)
  * [17] Gao, R., Liu, W.: DDGR: continual learning with deep diffusion-based generative replay. In: ICML. Proceedings of Machine Learning Research, vol. 202, pp. 10744–10763. PMLR (2023)
  * [18] Garcia, N.C., Morerio, P., Murino, V.: Modality distillation with multiple stream networks for action recognition. In: ECCV. Lecture Notes in Computer Science, vol. 11212, pp. 106–121. Springer (2018)
  * [19] Garcia, N.C., Morerio, P., Murino, V.: Learning with privileged information via adversarial discriminative modality distillation. PAMI 42(10), 2581–2593 (2020)
  * [20] Ge, Y., Li, Y., Ni, S., Zhao, J., Yang, M.H., Itti, L.: Clr: Channel-wise lightweight reprogramming for continual learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 18798–18808 (2023)
  * [21] Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K.V., Joulin, A., Misra, I.: Imagebind one embedding space to bind them all. In: CVPR. pp. 15180–15190. IEEE (2023)
  * [22] Gong, X., Mohan, S., Dhingra, N., Bazin, J., Li, Y., Wang, Z., Ranjan, R.: Mmg-ego4d: Multi-modal generalization in egocentric action recognition. In: CVPR. pp. 6481–6491. IEEE (2023)
  * [23] Goswami, D., Schuster, R., van de Weijer, J., Stricker, D.: Attribution-aware weight transfer: A warm-start initialization for class-incremental semantic segmentation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 3195–3204 (2023)
  * [24] Harborth, D., Pape, S.: How privacy concerns, trust and risk beliefs, and privacy literacy influence users’ intentions to use privacy-enhancing technologies: The case of tor. Data Base 51(1), 51–69 (2020)
  * [25] Hoffman, J., Gupta, S., Darrell, T.: Learning with side information through modality hallucination. In: CVPR. pp. 826–834. IEEE Computer Society (2016)
  * [26] Hu, Z., Li, Y., Lyu, J., Gao, D., Vasconcelos, N.: Dense network expansion for class incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11858–11867 (2023)
  * [27] Jang, J., Wang, Y., Kim, C.: Towards robust multimodal prompting with missing modalities. In: ICASSP (2024)
  * [28] Jeeveswaran, K., Bhat, P.S., Zonooz, B., Arani, E.: Birt: Bio-inspired replay in vision transformers for continual learning. In: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, J. (eds.) International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA. Proceedings of Machine Learning Research, vol. 202, pp. 14817–14835. PMLR (2023), https://proceedings.mlr.press/v202/jeeveswaran23a.html
  * [29] Jiang, J., Celiktutan, O.: Neural weight search for scalable task incremental learning. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 1390–1399 (2023)
  * [30] Jung, D., Han, D., Bang, J., Song, H.: Generating instance-level prompts for rehearsal-free continual learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11847–11857 (2023)
  * [31] Ke, Z., Liu, B., Huang, X.: Continual learning of a mixed sequence of similar and dissimilar tasks. In: NeurIPS (2020)
  * [32] Khan, M.G.Z.A., Naeem, M.F., Van Gool, L., Stricker, D., Tombari, F., Afzal, M.Z.: Introducing language guidance in prompt-based continual learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11463–11473 (2023)
  * [33] Kim, W., Son, B., Kim, I.: Vilt: Vision-and-language transformer without convolution or region supervision. In: ICML. Proceedings of Machine Learning Research, vol. 139, pp. 5583–5594. PMLR (2021)
  * [34] Konishi, T., Kurokawa, M., Ono, C., Ke, Z., Kim, G., Liu, B.: Parameter-level soft-masking for continual learning. In: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, J. (eds.) International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA. Proceedings of Machine Learning Research, vol. 202, pp. 17492–17505. PMLR (2023), https://proceedings.mlr.press/v202/konishi23a.html
  * [35] Lee, Y., Tsai, Y., Chiu, W., Lee, C.: Multimodal prompting with missing modalities for visual recognition. In: CVPR. pp. 14943–14952. IEEE (2023)
  * [36] Li, C., Gan, Z., Yang, Z., Yang, J., Li, L., Wang, L., Gao, J.: Multimodal foundation models: From specialists to general-purpose assistants. CoRR abs/2309.10020 (2023)
  * [37] Li, X., Zhou, Y., Wu, T., Socher, R., Xiong, C.: Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In: ICML. Proceedings of Machine Learning Research, vol. 97, pp. 3925–3934. PMLR (2019)
  * [38] Li, Z., Hoiem, D.: Learning without forgetting. In: ECCV. Lecture Notes in Computer Science, vol. 9908, pp. 614–629. Springer (2016)
  * [39] Lin, H., Zhang, B., Feng, S., Li, X., Ye, Y.: Pcr: Proxy-based contrastive replay for online class-incremental continual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 24246–24255 (2023)
  * [40] Lin, S., Yang, L., Fan, D., Zhang, J.: Beyond not-forgetting: Continual learning with backward knowledge transfer. Advances in Neural Information Processing Systems 35, 16165–16177 (2022)
  * [41] Liu, B., Zhu, Y., Gao, C., Feng, Y., Liu, Q., Zhu, Y., Stone, P.: LIBERO: benchmarking knowledge transfer for lifelong robot learning. CoRR abs/2306.03310 (2023)
  * [42] Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning (2023)
  * [43] Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning (2023)
  * [44] Liu, S., Cheng, H., Liu, H., Zhang, H., Li, F., Ren, T., Zou, X., Yang, J., Su, H., Zhu, J., Zhang, L., Gao, J., Li, C.: Llava-plus: Learning to use tools for creating multimodal agents. CoRR abs/2311.05437 (2023)
  * [45] Liu, Y., Schiele, B., Vedaldi, A., Rupprecht, C.: Continual detection transformer for incremental object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 23799–23808 (2023)
  * [46] Liu, Y., Cong, Y., Goswami, D., Liu, X., van de Weijer, J.: Augmented box replay: Overcoming foreground shift for incremental object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11367–11377 (2023)
  * [47] Liu, Z., Sun, W., Hong, Y., Teney, D., Gould, S.: Bi-directional training for composed image retrieval via text prompt learning. CoRR abs/2303.16604 (2023)
  * [48] Lopez-Paz, D., Ranzato, M.: Gradient episodic memory for continual learning. In: NIPS. pp. 6467–6476 (2017)
  * [49] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (Poster). OpenReview.net (2019)
  * [50] Luo, Z., Hsieh, J., Jiang, L., Niebles, J.C., Fei-Fei, L.: Graph distillation for action detection with privileged modalities. In: ECCV. Lecture Notes in Computer Science, vol. 11218, pp. 174–192. Springer (2018)
  * [51] Luo, Z., Liu, Y., Schiele, B., Sun, Q.: Class-incremental exemplar compression for class-incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11371–11380 (2023)
  * [52] Ma, M., Ren, J., Zhao, L., Testuggine, D., Peng, X.: Are multimodal transformers robust to missing modality? In: CVPR. pp. 18156–18165. IEEE (2022)
  * [53] Ma, M., Ren, J., Zhao, L., Tulyakov, S., Wu, C., Peng, X.: SMIL: multimodal learning with severely missing modality. In: AAAI. pp. 2302–2310. AAAI Press (2021)
  * [54] Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of machine learning research 9(11) (2008)
  * [55] Mallya, A., Lazebnik, S.: Packnet: Adding multiple tasks to a single network by iterative pruning. In: CVPR. pp. 7765–7773. Computer Vision Foundation / IEEE Computer Society (2018)
  * [56] Mendez, J.A., Kaelbling, L.P., Lozano-Pérez, T.: Embodied lifelong learning for task and motion planning. CoRR abs/2307.06870 (2023)
  * [57] Narasimhan, M., Yu, L., Bell, S., Zhang, N., Darrell, T.: Learning and verification of task structure in instructional videos. CoRR abs/2303.13519 (2023)
  * [58] Oh, Y., Baek, D., Ham, B.: Alife: Adaptive logit regularizer and feature replay for incremental semantic segmentation. Advances in Neural Information Processing Systems 35, 14516–14528 (2022)
  * [59] Ovalle, J.E.A., Solorio, T., Montes-y-Gómez, M., González, F.A.: Gated multimodal units for information fusion. In: ICLR (Workshop). OpenReview.net (2017)
  * [60] Prabhu, A., Torr, P.H.S., Dokania, P.K.: Gdumb: A simple approach that questions our progress in continual learning. In: ECCV. Lecture Notes in Computer Science, vol. 12347, pp. 524–540. Springer (2020)
  * [61] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: ICML. Proceedings of Machine Learning Research, vol. 139, pp. 8748–8763. PMLR (2021)
  * [62] Rao, D., Visin, F., Rusu, A.A., Pascanu, R., Teh, Y.W., Hadsell, R.: Continual unsupervised representation learning. In: NeurIPS. pp. 7645–7655 (2019)
  * [63] Ratcliff, R.: Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review 97(2), 285 (1990)
  * [64] Rymarczyk, D., van de Weijer, J., Zieliński, B., Twardowski, B.: Icicle: Interpretable class incremental continual learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1887–1898 (2023)
  * [65] Sarfraz, F., Arani, E., Zonooz, B.: Error sensitivity modulation based experience replay: Mitigating abrupt representation drift in continual learning. In: The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net (2023), https://openreview.net/pdf?id=zlbci7019Z3
  * [66] Sarfraz, F., Arani, E., Zonooz, B.: Sparse coding in a dual memory system for lifelong learning. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 9714–9722 (2023)
  * [67] Serrà, J., Suris, D., Miron, M., Karatzoglou, A.: Overcoming catastrophic forgetting with hard attention to the task. In: ICML. Proceedings of Machine Learning Research, vol. 80, pp. 4555–4564. PMLR (2018)
  * [68] Shin, H., Lee, J.K., Kim, J., Kim, J.: Continual learning with deep generative replay. In: NIPS. pp. 2990–2999 (2017)
  * [69] Shvetsova, N., Chen, B., Rouditchenko, A., Thomas, S., Kingsbury, B., Feris, R., Harwath, D., Glass, J.R., Kuehne, H.: Everything at once - multi-modal fusion transformer for video retrieval. In: CVPR. pp. 19988–19997. IEEE (2022)
  * [70] Shvetsova, N., Chen, B., Rouditchenko, A., Thomas, S., Kingsbury, B., Feris, R.S., Harwath, D., Glass, J., Kuehne, H.: Everything at once-multi-modal fusion transformer for video retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20020–20029 (2022)
  * [71] Smith, J.S., Karlinsky, L., Gutta, V., Cascante-Bonilla, P., Kim, D., Arbelle, A., Panda, R., Feris, R., Kira, Z.: Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In: CVPR. pp. 11909–11919. IEEE (2023)
  * [72] Sun, Q., Lyu, F., Shang, F., Feng, W., Wan, L.: Exploring example influence in continual learning. Advances in Neural Information Processing Systems 35, 27075–27086 (2022)
  * [73] Sun, W., Li, Q., Zhang, J., Wang, W., Geng, Y.a.: Decoupling learning and remembering: A bilevel memory framework with knowledge projection for task-incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20186–20195 (2023)
  * [74] Sun, Z., Mu, Y., Hua, G.: Regularizing second-order influences for continual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20166–20175 (2023)
  * [75] Villa, A., Alcázar, J.L., Alfarra, M., Alhamoud, K., Hurtado, J., Heilbron, F.C., Soto, A., Ghanem, B.: PIVOT: prompting for video continual learning. In: CVPR. pp. 24214–24223. IEEE (2023)
  * [76] Wang, F.Y., Zhou, D.W., Liu, L., Ye, H.J., Bian, Y., Zhan, D.C., Zhao, P.: Beef: Bi-compatible class-incremental learning via energy-based expansion and fusion. In: The Eleventh International Conference on Learning Representations (2022)
  * [77] Wang, H., Chen, Y., Ma, C., Avery, J., Hull, L., Carneiro, G.: Multi-modal learning with missing modality via shared-specific feature modelling. In: CVPR. pp. 15878–15887. IEEE (2023)
  * [78] Wang, W., Hu, Y., Chen, Q., Zhang, Y.: Task difficulty aware parameter allocation & regularization for lifelong learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7776–7785 (2023)
  * [79] Wang, X., Kumar, D., Thome, N., Cord, M., Precioso, F.: Recipe recognition with large multimodal food dataset. In: ICME Workshops. pp. 1–6. IEEE Computer Society (2015)
  * [80] Wang, Y., Huang, Z., Hong, X.: S-prompts learning with pre-trained transformers: An occam’s razor for domain incremental learning. In: NeurIPS (2022)
  * [81] Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C., Ren, X., Su, G., Perot, V., Dy, J.G., Pfister, T.: Dualprompt: Complementary prompting for rehearsal-free continual learning. In: ECCV. Lecture Notes in Computer Science, vol. 13686, pp. 631–648. Springer (2022)
  * [82] Wang, Z., Zhang, Z., Lee, C., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J.G., Pfister, T.: Learning to prompt for continual learning. In: CVPR. pp. 139–149. IEEE (2022)
  * [83] Woo, S., Lee, S., Park, Y., Nugroho, M.A., Kim, C.: Towards good practices for missing modality robust action recognition. In: AAAI. pp. 2776–2784. AAAI Press (2023)
  * [84] Xiao, J.W., Zhang, C.B., Feng, J., Liu, X., van de Weijer, J., Cheng, M.M.: Endpoints weight fusion for class incremental semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7204–7213 (2023)
  * [85] Yao, B.M., Chen, Y., Wang, Q., Wang, S., Liu, M., Xu, Z., Yu, L., Huang, L.: AMELI: enhancing multimodal entity linking with fine-grained attributes. CoRR abs/2305.14725 (2023)
  * [86] You, H., Zhang, H., Gan, Z., Du, X., Zhang, B., Wang, Z., Cao, L., Chang, S., Yang, Y.: Ferret: Refer and ground anything anywhere at any granularity. CoRR abs/2310.07704 (2023)
  * [87] Yu, T., Liu, J., Jin, Z., Yang, Y., Fei, H., Li, P.: Multi-scale multi-modal dictionary BERT for effective text-image retrieval in multimedia advertising. In: CIKM. pp. 4655–4660. ACM (2022)
  * [88] Zenke, F., Poole, B., Ganguli, S.: Continual learning through synaptic intelligence. In: ICML. Proceedings of Machine Learning Research, vol. 70, pp. 3987–3995. PMLR (2017)
  * [89] Zhang, X., Zhang, F., Xu, C.: VQACL: A novel visual question answering continual learning setting. In: CVPR. pp. 19102–19112. IEEE (2023)
  * [90] Zhang, Y., Pfahringer, B., Frank, E., Bifet, A., Lim, N.J.S., Jia, Y.: A simple but strong baseline for online continual learning: Repeated augmented rehearsal. Advances in Neural Information Processing Systems 35, 14771–14783 (2022)
  * [91] Zhang, Y., Fei, H., Li, D., Yu, T., Li, P.: Prompting through prototype: A prototype-based prompt learning on pretrained vision-language models. CoRR abs/2210.10841 (2022)
  * [92] Zhang, Y., Doughty, H., Snoek, C.G.M.: Learning unseen modality interaction. In: NeurIPS (2023)
  * [93] Zhao, S., Wu, D., Zhang, W., Zhou, Y., Li, B., Wang, W.: Asymmetric deep hashing for efficient hash code compression. In: ACM MM (2020)
  * [94] Zhao, S., Wu, D., Zhou, Y., Li, B., Wang, W.: Rescuing deep hashing from dead bits problem. In: IJCAI. pp. 1338–1344. ijcai.org (2021)
  * [95] Zhao, S., Xu, H.: Less is more: Toward zero-shot local scene graph generation via foundation models. CoRR abs/2310.01356 (2023)
  * [96] Zhao, S., Xu, H.: NEUCORE: neural concept reasoning for composed image retrieval. CoRR abs/2310.01358 (2023)
  * [97] Zhou, D., Wang, Q., Ye, H., Zhan, D.: A model or 603 exemplars: Towards memory-efficient class-incremental learning. In: The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net (2023), https://openreview.net/pdf?id=S07feAlQHgM
  * [98] Zhu, B., Lin, B., Ning, M., Yan, Y., Cui, J., Wang, H., Pang, Y., Jiang, W., Zhang, J., Li, Z., Zhang, W., Li, Z., Liu, W., Yuan, L.: Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. CoRR abs/2310.01852 (2023)
  * [99] Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 (2023)
  * [100] Zhu, L., Chen, T., Yin, J., See, S., Liu, J.: Continual semantic segmentation with automatic memory sample selection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3082–3092 (2023)

##  Appendix 0.A Comparison of Different Backbones

Table 7: Comparison of different backbones. Backbone | $\eta$ | #Image | #Text | AP ($\uparrow$) | FG ($\downarrow$)
---|---|---|---|---|---
CLIP [61] | $70\%$ | $100\%$ | $30\%$ | 39.69 | 15.26
$30\%$ | $100\%$ | 60.44 | 14.08
$65\%$ | $65\%$ | 53.06 | 6.53
ViLT [33] | $70\%$ | $100\%$ | $30\%$ | 50.00 | 12.47
$30\%$ | $100\%$ | 69.41 | 3.73
$65\%$ | $65\%$ | 59.92 | 8.56

Our proposed RebQ can integrate with various backbones. We employ the ViLT
[33] model as the default backbone in Section 4. ViLT is a fusion-based model
that fuses the visual and text tokens to obtain the fused token. Furthermore,
we also employ the other mainstream vision-language architectures, i.e., the
dual-encoder-based CLIP [61] model, to offer a broader assessment of RebQ.
CLIP is a dual-encoder model, where one encoder processes text and the other
processes images. These encoders transform their respective inputs into a
shared embedding space.

However, the dual-encoder CLIP model with different dimensions within each
encoder brings challenges to integration in RebQ. Specifically, during the
reconstruction stage, the memory prompts from the memory pool are inserted
into the available modality-specific encoder to reconstruct the missing query
embedding. However, the fixed dimension memory prompts (e.g., $512$
dimensions) cannot be inserted and jointly learned with the different
dimensions (e.g., $768$ and $512$ dimensions in the visual and text encoders,
respectively). Therefore, we introduce an extra fully connected layer to
project the memory prompts to the embedding space of the missing modality.
After obtaining the image and text embeddings from corresponding encoders, we
add the two embeddings and feed them into a classifier. The CLIP model in our
experiment is ViT-B-32333https://huggingface.co/openai/clip-vit-base-patch32.

From Table 7, the CLIP model performs worse than the ViLT model, even if
trained on larger datasets. We argue that the CLIP model is trained by the
contrastive learning strategy and aligns the vision and language modality to a
common embedding space. This type of training paradigm is inferior to fusion-
based approaches in scenarios requiring complex multi-modal understanding
skills, which matches the conclusions in recent multi-modal works [3, 47].

##  Appendix 0.B Analysis of Reconstruction

In this section, we will analyze why the pre-trained LMMs can reconstruct the
missing information. Assume the visual modality is missing, we model the
modality-incomplete conditional probability distribution
$\mathrm{Pr}(\tilde{\mathbf{v}}\mid\mathbf{t};\mathcal{K})$:

$\displaystyle\arg\max\mathrm{Pr}(\tilde{\mathbf{v}}\mid\mathbf{t};\mathcal{K})$
$\displaystyle=\frac{\mathrm{Pr}(\tilde{\mathbf{v}},\mathbf{t};\mathcal{K})}{\mathrm{Pr}(\mathbf{t};\mathcal{K})}$
(13)
$\displaystyle\approx\frac{\mathrm{Pr}(\mathbf{v},\mathbf{t};\mathcal{K})}{\mathrm{Pr}(\mathbf{t};\mathcal{K})}$
$\displaystyle=\mathrm{Pr}(\mathbf{v}\mid\mathbf{t};\mathcal{K}),$

where $\mathcal{K}$ is the multi-modal knowledge in LMMs. We assume the
modality-incomplete distribution can be estimated by modality-complete
distribution.

Similarly, when the text modality is missing, we can obtain:

$\displaystyle\arg\max\mathrm{Pr}(\tilde{\mathbf{t}}\mid\mathbf{v};\mathcal{K})$
$\displaystyle=\frac{\mathrm{Pr}(\tilde{\mathbf{t}},\mathbf{v};\mathcal{K})}{\mathrm{Pr}(\mathbf{v};\mathcal{K})}$
(14)
$\displaystyle\approx\frac{\mathrm{Pr}(\mathbf{t},\mathbf{v};\mathcal{K})}{\mathrm{Pr}(\mathbf{v};\mathcal{K})}$
$\displaystyle=\mathrm{Pr}(\mathbf{t}\mid\mathbf{v};\mathcal{K}),$

Equation (13) and (14) indicate that the model can reconstruct the missing
modality by conditioning the existing modality and multi-modal knowledge. They
assume $\\{\mathbf{t},\mathbf{v},\mathbf{y}\\}$ and
$\\{\tilde{\mathbf{t}},\mathbf{v},\mathbf{y}\\}$ are sampled from the same
distribution. In the CMML task, data from sequential tasks has a different
distribution, also suffering from catastrophic forgetting of reconstruction
ability. Therefore, we create a memory pool to tackle this issue, and its
effectiveness is shown in Section 4.2.

##  Appendix 0.C Limitations

In this paper, we introduce a new task, Continual Missing Modality Learning
(CMML), which is commonly encountered in real applications and propose a RebQ
framework to tackle this task. Although we utilize two modalities, including
vision and language, to demonstrate the effectiveness of our proposed
framework, multi-modality contains more modalities, such as depth and audio.
In fact, our framework has the ability to process more modalities. Compared
with MAP [35], which needs $N^{2}-1$ ($N$ is the number of modalities) types
of prompts to represent missing modalities, RebQ only requires $N$ types of
prompt pools, significantly reducing the needed prompt types. On the other
hand, we employ the pre-trained LMM with only two available modalities. To
extend the supported modalities, we can re-train LMMs on large-scale datasets
with more modalities [70], integrate more modalities while keeping the trained
encoders frozen [21, 98], or devise new methods as the future direction.