# Dynamic Transformer Architecture for Continual Learning of Multimodal Tasks

Yuliang Cai and Mohammad Rostami

###### Abstract

Transformer neural networks are increasingly replacing prior architectures in
a wide range of applications in different data modalities. The increasing size
and computational demands of fine-tuning large pre-trained transformer neural
networks pose significant challenges for the widespread adoption of these
models for applications that demand on-edge computing. To tackle this
challenge, continual learning (CL) emerges as a solution by facilitating the
transfer of knowledge across tasks that arrive sequentially for an
autonomously learning agent. However, current CL methods mainly focus on
learning tasks that are exclusively vision-based or language-based. We propose
a transformer-based CL framework focusing on learning tasks that involve both
vision and language, known as Vision-and-Language (VaL) tasks. Due to the
success of transformers in other modalities, our architecture has the
potential to be used in multimodal learning settings. In our framework, we
benefit from introducing extra parameters to a base transformer to specialize
the network for each task. As a result, we enable dynamic model expansion to
learn several tasks in a sequence. We also use knowledge distillation to
benefit from relevant past experiences to learn the current task more
efficiently. Our proposed method, Task Attentive Multimodal Continual Learning
(TAM-CL), allows for the exchange of information between tasks while
mitigating the problem of catastrophic forgetting. Notably, our approach is
scalable, incurring minimal memory and time overhead. TAM-CL achieves state-
of-the-art (SOTA) performance on challenging multimodal tasks111Early partial
results of this work are presented in EMNLP 2023 [9].

###### keywords:

continual learning , multimodal learning , catastrophic forgetting

††journal: Nuclear Physics B

organization=University of Southern California
Los Angeles, CA, USA.

##  1 Introduction

Large pre-trained transformer models have found application in a wide range of
domains, encompassing tasks that involve both vision and language [16, 35, 83,
38, 87, 1]. The transformer architecture is based on encoder and decoder
layers that implement the self-attention mechanism [75]. The self-attention
layers implement an attention mechanism on an input sequence by relating the
positions of tokens in the sequence to the way that a global feature vector is
extracted from the sequence. The idea of self-attention then was adopted for
other modalities such as computer vision tasks [17, 25] and speech processing
tasks [33, 79] based on modeling an image using a sequence of patches as
tokens. Various improved transformer architectures are developed by modifying
the base idea and training procedures for each of the language and image
modalities.

Typically, transformer models undergo pretraining on a substantially large
dataset, followed by fine-tuning to adapt to a specific downstream task, even
with limited task-specific data. This approach enables effective transfer
learning to leverage the generalizable knowledge gained from the large
dataset. However, fine-tuning at the task level can weaken the model’s ability
to generalize because the model learns to become task-specific. It also
requires the retention of a separate version of the base model for each task.
As transformer architectures are persistently becoming larger and larger,
storing many versions of a base architecture is becoming a practically
infeasible approach. To address these issues, continual learning (CL) [32, 84,
80, 47, 19, 66, 45, 41, 78] algorithms have been developed to navigate these
complexities in transformer models. They employ a shared model that leverages
cross-task knowledge transfer to learn tasks that are encountered
sequentially. The goal is to enable the shared model to learn new tasks
without degraded performance on the previously learned tasks.

Continual learning has a rich precedent before the invention of transformers
[64, 13, 56, 55, 74]. The primary obstacle in Continual Learning (CL) is known
as catastrophic forgetting [22], where the model performance on past learned
tasks degrades as the model is updated to learn new tasks. There are several
approaches to address catastrophic forgetting. One category of CL algorithms
tackles this challenge by regularizing a fixed shared model to learn different
tasks through distinct information pathways, or weights [36, 2, 24]. The
fundamental concept of regularization-based methods is to identify a subset of
model parameters crucial for encoding the acquired knowledge of each task and
then consolidate these parameters when updating the model for new tasks.
Another approach involves model expansion [60, 85]. The goal is to enlarge a
base model with a small set of additional weights and applying the network to
learn new tasks using these weights. Lastly, some algorithms employ pseudo-
rehearsal with experience replay [49, 58, 50, 4, 52, 44]. This approach
entails storing a representative sample of training data for each task in a
memory buffer and replaying them alongside the current task’s data to retain
the encoded knowledge of past tasks. Certain methods alleviate the need for a
memory buffer by enabling the model to generate pseudo-samples for previously
learned tasks, which are then employed in experience replay. Despite their
effectiveness, existing CL techniques are designed for unimodal tasks, such as
tasks solely involving vision [57, 40, 18, 29] or language [32, 84], and do
not address the distinct challenges posed by multimodal tasks, such as Vision-
and-Language (VaL) tasks. A major reason behind unimodal approach is that
prior to the transformer architectures, the specific neural network
architectures for each modality were quite different, e.g., convolutional
neural networks vs recurrent neural networks. Because the base idea of
transformers work well in different data modalities, we may be able to address
multimodal learning with a single base unified architecture to fuse inputs
from different modalities.

More specifically, we introduce a novel algorithm for learning tasks involving
both vision and language (VaL) within a CL framework. In our work, we utilize
dynamic model expansion in the context of transformers. To achieve this end,
we utilize the self-attention layers of a foundational bimodal transformer as
a shared encoder across all tasks. Subsequently, we enhance the base model
with task-attention layers [18, 5], which serve to customize the model for
each task by incorporating a task-specific token. Our approach entails a
minimal memory overhead and introduces only a slight increase in inference
time during testing. Furthermore, it operates without the need for extensive
hyper-parameter tuning and remains resilient even in scenarios where the
number of tasks is unknown. Our distinct contributions include:

  * 1.

A dynamically expanding, efficient transformer architecture for multimodal CL
based on task attention layers that make the model task attentive.

  * 2.

A training algorithm to handle diverse, sequentially arriving vision-and-
language tasks such as visual question answering, visual entailment, and
visual reasoning.

  * 3.

Extensive experiments to demonstrate that the proposed model achieves SOTA
performance compared to existing baselines.

##  2 Background and Related Work

##### Transformers for Vision and Language Tasks

Multimodal transformers have been developed for processing tasks that involve
both vision and language (VaL) [68, 72, 35, 14, 12]. The fundamental concept
behind these transformers is to utilize self-attention layers on each modality
to extract relevant features from both the visual and linguistic inputs and
fuse the knowledge to solve the task with more comprehensive understanding
provided by multi-modalities. As knowledge fusion is the key component of
multimodal transformer, several different strategies has been applied to boost
the performance, such as early summation [23, 82], early concatenation [70,
62], and cross attention [46] that fuse the knowledge by different methods and
different stage of feature processing. The features are subsequently fused at
higher layers to derive cross-modal contextualized representations of the
multimodal inputs. The objective is to capture the interaction between the
visual and linguistic inputs, which is crucial for effectively performing VaL
tasks. This approach has also been applied in other domains, such as in the
context of processing video to establish relationships between visual and
speech inputs [3, 70]. Multimodal transformers are typically trained on
extensive datasets and have demonstrated remarkable effectiveness when fine-
tuned for specific tasks downstream. Given the substantial performance gains
they offer, transformers are progressively supplanting older architectural
paradigms.

##### Continual Learning

The primary challenge in continual learning is addressing catastrophic
forgetting [22]. A dominant approach to address this challenge is experience
replay [61, 88, 21, 54, 59, 63, 53, 67]. The main idea for experience replay
is to store and then replay representative samples of past tasks alongside the
current task data to preserve the learned distributions through pseudo-
rehearsal. Weight consolidation using structural plasticity [36, 37, 15] is
another strategy to approximate experience replay. The concept is to identify
important weights that retain knowledge about a task and then consolidate them
based on their relative importance for past tasks in the future [36] and learn
new tasks through weights that are unimportant for past learned tasks. These
approaches are not efficient for transformers due to the large size of these
models. A more practical approach is based on the notion of adapters [34, 20,
19], where most transformer weights remain frozen and a small number of
trainable weights are used to make the model dynamic. In this context,
mechanisms such as experience replay or weight consolidation can be used on
the adapter weights to overcome forgetting effects.

##### Knowledge Distillation for Continual Learning

Despite the impressive success of deep learning, neural networks models fail
to be deployed in limited-sourced devices such as phone and embedded devices
due to the significant computational load of their training. Thus, the
compression and acceleration of the model is urgently needed. First introduce
by Bucilua et al [8], knowledge distillation is widely used to compress the
knowledge of a large and complex model, the teacher model, into a smaller and
efficient ones, the student models. Fitnet [51] deploys the knowledge
distillation between the intermediate representations between teacher and
student model to improve the training of the students. On the other hand, Chen
et al. [11] utilize the output, logit z, of student and teacher model to
compute the knowledge distillation loss. Lopez-Paz and Ranzato [42] developed
one of the first methods to adopt knowledge distillation in to the continual
learning setting. Within the scenario of learning from $\mathcal{T}$
sequential tasks, the copied learnt model on the previous task, $t_{i-1}$, is
used as the teacher model to guide the training of the model on the current
task $t_{i}$. The knowledge distilled from the previous model can efficiently
help preserve the previous knowledge and mitigate catastrophic forgetting,
while at the same time accelerate the forward transfer of the knowledge to
learn the current task efficiently.

##### Continual Learning for Transformer

Fine-tuning transformers inherently sacrifices their ability to generalize
effectively. Employing a separate transformer for each task would result in a
substantial increase in disk memory usage, given the significant growth in
transformer sizes to handle more complex tasks. While CL appears to be a
natural remedy for these challenges, research on CL with transformers remains
relatively scarce. Xin et al. [32] utilize adapters in conjunction with a
hypernetwork [77] to enable CL for language tasks. Alternatively, Yang et al.
[84] propose a transformer calibration module to enhance a transformer’s
adaptability. This module operates independently from the base pre-trained
transformer and aids in adopting it for a specific downstream task. The
Lifelong Vision Transformer [80] integrates information from previous tasks
through an inter-task attention mechanism, thus reducing the rate at which
critical attention weights shift away from older tasks towards the current
one. Douillard et al. [18] propose a CL architecture for vision tasks using
the ViLT model [35]. Pelosin et al. [47] extend this approach to an exemplar-
free setting by distilling the attention-level matrices of transformers. This
enables model adaptability and helps mitigate the effects of forgetting. Ermis
et al. [19] apply the concept of adapters in a vision context. To the best of
our knowledge, there has been no prior exploration of CL for multimodal tasks
using transformer architectures.

##  3 Problem Description

Figure 1: The proposed CL training procedure: (1) A small portion of the data
for previous tasks are randomly selected and stored in a memory buffer. (2)
The current task arrives with $\mathcal{D}^{i}$. (3) The training data
$\mathcal{D}^{i}$ is used as input to the teacher model to compute the
distillation loss. (4) The memory buffer samples are replayed along with the
current task data to train the main model. (5) After learning the current
task, the teacher model of the next task will be a copy of the current model.

Consider a collection of VaL tasks $\\{\mathcal{T}_{i}\\}_{i=1}^{T}$ which are
introduced in a sequential manner, and each task $\mathcal{T}_{i}$ comes with
an annotated training dataset $\mathcal{D}^{i}$ =
$\\{\langle(\bm{I}_{i}^{j},\bm{L}_{i}^{j})^{i},y^{j}_{i}\rangle_{j=1}^{N_{i}}\\}$,
where $\bm{I}_{i}^{j}\in\mathbb{R}^{H\times W\times C}$ represents the image
input, $\bm{L}_{i}^{j}\in\mathbb{R}^{L\times|V|}$ represents the language
input, while $y^{j}_{i}$ is the text-typed discrete label. It’s worth noticing
that the order of these tasks and $T$ are not known in advance and the agent
encounters the task without any advance knowledge. The training data points
for $\mathcal{T}_{i}$ are assumed to be drawn iid from a task-specific joint
distribution $p_{i}^{t}(\cdot,\cdot,\cdot)$. In the context of multimodal
continual learning, our objective is to learn each task at time-step $i$ and
then progress to learn the next tasks. The learned tasks can be encountered at
any time during testing in the future. As a result, we would like to preserve
the performance achieved on the previous learned tasks through the prevention
of catastrophic forgetting.

When learned individually, each of these VaL tasks $\mathcal{T}_{i}$ can be
learned using supervised learning conditioned on selecting the suitable
predictive model $f^{i}_{\theta_{M}}(\cdot,\cdot)$, e.g., a transformer with
trainable parameters $\theta_{M}$, and the discrimination loss
$\mathcal{L}(\cdot)$, e.g., cross-entropy. However, given storage constrains,
we assume a shared model must be employed, as training a separate model per
task is impractical. Moreover, only a limited portion of training data for
each task $\mathcal{T}_{i}$ can be retained post-training because normally the
training data is not stored due to storage limitations, which makes multitask
learning [10, 28, 6] an unfeasible solution. The single task learning strategy
using a shared model is not ideal in CL either. Because when the model is
updated to learn the current tasks, its performance on the past tasks may
suffer from catastrophic forgetting due to updates in the model parameters,
leading to a degradation in performance. On the other hand, single task
learning does not benefit from knowledge transfer because the tasks are
learning in isolation.

Figure 1 provides an overview of our proposed solution to address multimodal
CL. To leverage a shared model across all tasks and facilitate knowledge
transfer across all tasks, we utilize a base transformer model as
$f_{\theta}(\cdot,\cdot)$. The base model remains largely frozen during the
training phase to help maintaining its generalization power. The model is also
made adaptive by incorporating a unique task attention layer after its final
layer. Additionally, we modify the supervised learning loss by introducing a
knowledge distillation loss [26] on the intermediate model layers to enable
positive forward knowledge transfer. We consider the teacher model in the
knowledge distillation formulation to be a copy of
$f^{i-1}_{\theta}(\cdot,\cdot)$ when training the main student model on
$i^{th}$ task $\mathcal{T}_{i}$. The teacher model is served as an
intermediate means of transferring past experiences. Additionally, we employ
pseudo-rehearsal through experience replay [50], utilizing a small memory
buffer to mitigate the effect of catastrophic forgetting [36].

##  4 The Proposed Architecture

Figure 2: The proposed transformer-based architecture: (left) The VaL inputs
are converted into two sequences and then fed into the self-attention layers
to generate a fused global feature vector. The data feature vector is then
concatenated with the learnable task-specific tokens and then fed into the
task attention layer to generate the input for the task-specific classifier
heads. The same VaL inputs are also fed into the teacher model’s transformer
architecture to compute Knowledge Distillation. (right) The task-attention
block architecture.

More specifically, Figure 2 visualizes our transformer-based architecture for
continual learning in multimodal learning scenario. The architecture is
composed of a shared pre-trained replaceable multimodal transformer, a shared
non-pretrained task attention block, and MLP classification headers that are
task-specific. Using task-specific heads allow for learning tasks that do not
share the same output-space. To make the model adaptive to sequentially coming
tasks, The task attention block receives task-specific tokens, estimated
during learning each task, task-specific tokens are initialized and sent to
task attention blocks along with the input instances to generate the expected
output. We provide details about these modules and the strategies that we use
to train it.

###  4.1 Sequential Feature Extraction Block

We refer to our architecture as Task Attentive Multimodal Continual Learning
(TAM-CL). In TAM-CL, we adopt the ViLT [35] feature generation procedure to
encode the visual inputs. To this end, we first break down a given image
$\bm{I}\in\mathbb{R}^{H\times W\times C}$ into a sequence of patches. These
patches are then flattened to create 2D vectors
$\bm{v}\in\mathbb{R}^{N\times(P^{2}\cdot C)}$. In this context,$C$ denotes the
number of channels, $P\times P$ represents the size of each image patch, and
$N=HW/P^{2}$ stands for the number of patches. Next, we apply a trainable
linear projection represented by $\bm{V}\in\mathbb{R}^{(P^{2}\cdot C)\times
H}$. These transformations convert $\bm{v}$ into a sequential representation
$\bm{\overline{v}}\in\mathbb{R}^{N\times H}$ according to the equation:

$\bm{\overline{v}}=[\bm{v}^{\text{class}};\bm{v}_{1}\bm{V};\bm{v}_{2}\bm{V};...;\bm{v}_{N}\bm{V}]+\bm{V}^{\text{pos}}.$
(1)

Here, $\bm{v}^{class}$ corresponds to class-specific vectors, and
$\bm{V}^{pos}$ stands for the position embedding. This process enables the
creation of a sequential representation of the image patches, which can be
subsequently used in the following layers of the model for further processing
and abstract feature extraction.

We also derive word vectors for $\bm{l}\in\mathbb{R}^{L\times|V|}$ to encode
the language input. This task is achieved by applying a word embedding matrix
$\bm{T}\in\mathbb{R}^{|V|\times H}$ and incorporating a position embedding
matrix $\bm{T}^{pos}\in\mathbb{R}^{(L+1)\times H}$. This process results in
the embedded data $\bm{\overline{t}}\in\mathbb{R}^{L\times H}$ according to
the equation:

$\bm{\overline{t}}=[\bm{t}^{\text{class}};\bm{l}_{1}\bm{T};\bm{l}_{2}\bm{T};...;\bm{l}_{L}\bm{T}]+\bm{T}^{\text{pos}}.$
(2)

Here, $\bm{t}^{\text{class}}$ pertains to class-specific vectors, and
$\bm{T}^{\text{pos}}$ denotes the position embedding. Subsequently, we add the
image and text embeddings to their respective independent model-type embedding
vectors, namely $\bm{t}^{type}$ and $\bm{v}^{type}\in\mathbb{R}^{H}$. These
components are then concatenated to form a unified sequence $\bm{s}$ as
follows:

$\bm{s^{0}}=[\bm{v^{\text{type}}}+\bm{\overline{v}};\bm{t^{\text{type}}}+\bm{\overline{t}}]$
(3)

The amalgamated vector $\bm{s}$ is processed through $D$ standard self-
attention layers within the base VaL transformer (ViLT). The output embeddings
from these attention layers are referred to as $\bm{s}^{D}$, and they are
subsequently passed into the task attention layer. The resulting embeddings
from the attention layers are denoted as $\bm{s}^{D+1}$. The transformations
are carried out in two steps, described as:

$\displaystyle\bm{\hat{s}}^{d}=MSA(LN(\bm{s}^{d-1}))+\bm{s}^{d-1},d=1,...,D$
(4)
$\displaystyle\bm{s}^{d}=MLP(LN(\bm{\hat{s}}^{d}))+\bm{\hat{s}}^{d},d=1,...,D$

Despite fusing the multimodal inputs, the model would be a single-task
learning model. To enable learning multiple tasks, we need a task-dependent
block.

###  4.2 Task Attention Block

The essence of our approach hinges on adopting task-level self-attention each
time a new task is learned by introducing task-specific tokens. The notion of
task token has been used before in classic CL to make a base transformer model
adaptive [31]. In contrast to the standard self-attention layer, the task-
attention layer is designed specifically-associated with the current task. For
each new task $i$ within the range of $1$ to $T$ comes in, a trainable task
token $\tau_{i}\in\mathbb{R}^{G\times 1}$ is initialized and applied as the
input of the task-attention layer for the current task, where $G$ represents
the size of the latent space for each self-attention layer. During the
training step for each task, the task token is estimated to generate the
desired output for that task. Much like the self-attention block (SAB), the
task-attention block (TAB) is a module comprising an attention layer, layer
normalization, and a Multi-Layer Perceptron (MLP). However, in this case, the
attention layer is specifically designed as a task-attention layer, deviating
from the standard self-attention, which is task-specific, by using modified
queries.

For task $i$, the task attention block receives two inputs: the batched output
from the previous self-attention blocks $\bm{s}_{ib}^{D}$, where $b$ notes the
specific batch from the training data of $T_{i}$, and the task-specific token
$\tau_{i}$. Notably, the same task token $\tau$ is employed for all instances
in the task-specific training/testing dataset. These two vectors are
concatenated to form an input for the task attention:

$\displaystyle\bm{s}{{}^{\prime}}_{i}^{D+1}=[\tau_{i},\bm{s}_{i}^{D}]\in\mathbb{R}^{(N+1)\times
G},i=1,...,T$ (5)
$\displaystyle\bm{\hat{s}}_{i}^{D+1}=\textit{TA}(LN(\bm{s}{{}^{\prime}}_{i}^{D+1})),i=1,...,T$
$\displaystyle\bm{s}_{i}^{D+1}=\textit{MLP}(LN(\bm{\hat{s}}_{i}^{D+1}))+\bm{\hat{s}}_{i}^{D+1},i=1,...,T$

Here, $LN$ denotes layer normalization, and $TA$ represents the task-attention
layer. This process is performed for each task within the range of $1$ to $T$.

The task attention block is added after the final self-attention block of the
backbone transformer. While it’s possible to incorporate more than one task
attention block, our architecture utilizes a single TAB for simplicity. The
functioning of the task attention layer is described as follows:

$\displaystyle\bm{Q}_{i}=\bm{W}_{q}\times\tau_{i},$ (6)
$\displaystyle\bm{K}_{i}=\bm{W}_{k}\times\bm{s}_{i}^{D+1},$
$\displaystyle\bm{V}_{i}=\bm{W}_{v}\times\bm{s}_{i}^{D+1},$
$\displaystyle\bm{A}_{i}=Softmax(\bm{Q}_{i}\cdot\bm{K}_{i}^{T}/\sqrt{G/h}),$
$\displaystyle\bm{O}_{i}=\bm{W}_{o}\bm{A}_{i}\bm{V}_{i}+\bm{b}_{o}\in\mathbb{R}^{1\times
G},$

where $h$ represents the number of attention heads in the transformer [76].
Additionally, $\bm{W}_{q}$, $\bm{W}_{k}$, $\bm{W}_{v}$ and $\bm{W}_{o}$ are
trainable weight matrices, while $\bm{b}_{o}$ is a bias vector. Subsequently,
the output of the task attention block, $\bm{s}_{i}^{D+1}$, is fed into task-
specific classifier layers:

$\bm{y}_{i}=\textit{Clf}_{i}(\bm{s}_{i}^{D+1}),i=1,...,T,$ (7)

where $\textit{Clf}_{i}$ represents the classifier subnetwork, which is far
smaller than prior blocks in size, specific to task $i$ to generate the final
output for task-specific predictions. Although the model becomes adaptive with
this block, catastrophic forgetting remains a challenge if we use this model
to learn sequential tasks. To address this challenge we relay on knowledge
distillation and experience replay.

###  4.3 Teacher Model Block

Besides the architecture proposed in section 4.1 and 4.2, to further
consolidate the weights that maintains the knowledge of previous tasks, the
teacher model is applied parallel to the main model, noted student model, such
that the same input data can be sent to both the teacher model and student
model for the computation of knowledge distillation loss to enable forward
positive knowledge transfer.

Consider $\mathcal{V}$ as the base transformer ViLT mentioned in section 4.1,
and $\mathcal{V}_{i}$ as model $\mathcal{V}$ after trained on $i^{th}$ task.
Denote $\mathcal{M}_{s}$ as student model and $\mathcal{M}_{t}$ as teacher
model. During the training on task $i$, we have:

$\displaystyle\mathcal{M}_{s}=\mathcal{V}_{i},$ (8)
$\displaystyle\mathcal{M}_{t}=\mathcal{V}_{i-1}.$

where $\mathcal{M}_{t}$ is frozen completely. The detail of our approach
knowledge distillation will be discussed in Section 5.2.

##  5 Training Algorithm

Note that the architecture depicted in Figure 2 provides an overview of our
model, TAM-CL, at a specific time step. This architecture dynamically expands
as additional tasks are learned in succession and more task tokens are
accumulated. However, the addition includes the classification heads and
tokens which makes it significantly more efficient compared to learning
independent models, one per task. We will now elaborate on the appropriate CL
loss functions that we employ for the training process the model and the task
tokens.

###  5.1 Token Expansion

During the training phase, the self-attention blocks of the base transformer,
as well as the task attention block, are shared across all tasks. However, for
each new task, we introduce a new task token, denoted as $\tau\in G\times 1$,
with the same dimension and initialize it uniformly. The token is then updated
to fit well with the new task. Additionally, we initialize a new cumulative
task-specific classifier, $\textit{Clf}_{i}(\cdot)$, for task $i$. The output
dimension of $\textit{Clf}_{i}(\cdot)$ is expanded based on the previous
classifier, $\textit{Clf}_{i-1}(\cdot)$, to include new outputs for the
current task. As more tasks are added, the output dimension of the task-
specific classifier $i$ accumulates as follows:

$\bm{E}_{i}=\bm{E}^{\textit{orig}}_{i}+\bm{E}_{i-1},i=1,\ldots,T.$ (9)

Here, $\bm{E}_{i}$ represents the output dimension for the $i$-th classifier,
while $\bm{E}^{\textit{orig}}_{i}$ signifies the output dimension originally
designed for the $i$-th task.

For the $i$-th task, we combine the $i$-th task token with the updated path
token $\bm{s}^{D}$ from the last self-attention block of the transformer, and
input it into the task attention block, as detailed in Section 4.2. During
this stage, only the $i$-th task token and task-specific classifier are
trainable, while all other task tokens and classifiers remain fixed. Since the
classifier heads are significantly smaller than the base network, storing them
do not lead to a significant memory overhead.

In the testing phase, the task number $i$ of the test data is explicitly
provided. $\bm{s}^{D}$ is combined with the $i$-th learned task token and fed
into the task attention block, along with utilizing its corresponding task-
specific classifier. All other task tokens and classifiers remain inactive.

###  5.2 Loss and Knowledge Distillation

The objective function in TAM-CL consists of three distinct loss components.
First, the cross-entropy loss, $\mathcal{L}_{c}$, serves as the original task-
specific loss function used in single-task learning, with its value varying
depending on the specific task at hand. Second, the knowledge distillation
(KD) loss, $\mathcal{L}_{ikd}$, is computed based on the output of the last
self-attention block $\bm{s}^{D}$, from both the main student model and the
teacher model. This loss term is crucial for transferring knowledge from the
teacher to the student model, effectively constraining distribution shifts and
preventing catastrophic forgetting as well. Finally, the diverse loss,
$\mathcal{L}_{div}$ evaluates the diversity of the data distribution
associated with task tokens, encouraging greater diversity among them to
enable learning task-specific knowledge and mitigates the effect of negative
knowledge transfer [89]. Our final loss function is a weighted combination of
these components:

$\mathcal{L}=(1-\lambda)\mathcal{L}_{c}+\lambda\alpha\mathcal{L}_{ikd}+\beta\mathcal{L}_{div},$
(10)

where $\lambda$ is determined as $\frac{T_{n}-1}{T_{n}}$ and $T_{n}$
represents the total number of tasks encountered thus far. The parameter
$\alpha$ is task-specific and $\beta$ is computed as
$\beta=min(\mathcal{L}_{div},0.1\times((1-\lambda)\mathcal{L}_{c}+\lambda\alpha\mathcal{L}_{ikd}))$.
The intermediate knowledge distillation, $\mathcal{L}_{ikd}$, is a central
aspect of TAM-CL. It enables the transfer of knowledge from the teacher model
to the main student model, effectively limiting the probability shift of pre-
trained parameters. This, in turn, allows subsequent layers to remain flexible
in adapting to new tasks. Notably, TAM-CL introduces this intermediate
knowledge distillation objective function in the context of multimodal
continual learning, marking a significant advancement in the field.

###  5.3 Experience Replay

The described training procedure above allows for the training of a shared
model across tasks, although it necessitates datasets for all tasks, which
challenges a fundamental assumption in CL. Because when a task is learned, we
may not have access to its data when learning future tasks. On the other hand,
catastrophic forgetting is a general issue if we exclude the full dataset for
the previously learned tasks. To address these issues, we rely on the classic
experience replay approach. We employ a memory buffer during the training
phase at each time-step. This buffer retains a small percentage of randomly
selected samples, e.g., approximately 5% in our experiments, of the training
dataset for all previous tasks. When learning the current task with a specific
batch number, the subsequent batch is chosen at random from the memory buffer
and is used to represent the past tasks. This process helps consolidate the
parameter distribution related to previous tasks, effectively mitigating
forgetting effects. We present the Task Attentive Multimodal Continual
Learning (TAM-CL) training procedure in to Algorithms 1 and 2.

Algorithm 1 TAM-CL Train

INPUT: Model M, MemBuffer B, ReplayFreq f

for epoch in num_epoch do

for step, batch in dataloader do

Loss $\leftarrow$ TrainStep(M, batch)

if step % f == 0 then

$\text{batch}_{replay}$ $\leftarrow$ B.getBatch()

$\text{Loss}_{replay}$ $\leftarrow$ TrainStep(M, $\text{batch}_{replay}$)

Algorithm 2 TAM-CL TrainStep

INPUT: Model M, Teacher Model T, Batch, Target t, Token k

p $\leftarrow$ Model(Batch)

$loss$ $\leftarrow$ CrossEntropyLoss(p, t)

ModelSabPre $\leftarrow$ M.SAB(Batch)

TeacherSabPre $\leftarrow$ T.SABs(Batch)

$loss_{ikd}\leftarrow$ KL-Div(ModelSabPre, TeacherSabPre)

$loss_{div}\leftarrow$ CrossEntropyLoss($\textbf{k}_{i},\textbf{k}_{j})\quad
j=1,..,i-1$$loss$ $\leftarrow$ $(1-\lambda)loss$ \+ $\lambda\alpha loss_{ikd}$
\+ $\beta loss_{div}$$loss$.backward()Return loss

##  6 Experiments Result

As a method proposed to solve vision-language tasks, TAM-CL needs to be
evaluated using tasks such as visual question answering, visual understanding
and visual reasoning. To test its ability to prevent catastrophic forgetting,
we choose five independent multimodal datasets for evaluation which are
presented to the architecture in a sequence. Additionally, to show that TAM-CL
is a competitive method, we choose five baseline methods for comparisons.
These methods include fine-tuning as a lowerbound and four continual learning
methods.

###  6.1 Experiments Setup

Name | # Training Examples | # Labels
---|---|---
NLVR2 | $80000$ | $2$
SNLI-VE | $80000$ | $3$
COCOQA | $78736$ | $430$
GQA | $80000$ | $1842$
OKVQA | $18032$ | $2910$
Table 1: Statistics of the VaL datasets used in the experiments.

##### Datasets

we use SNLI-VE [81], an image-sentence pairs dataset whereby a premise is
defined by an image, rather than a natural language sentence as in traditional
Textual Entailment tasks, COCOQA [48], a visual question answering dataset
based on Microsoft COCO image dataset, GQA [30], a compositional question-
answering and visual reasoning dataset leverages scene graph structure, NLVR2
[69], a visual reasoning dataset which takes two images and determine the
correctness of the given sentence, OKVQA [43], a knowledge-based visual
question-answering dataset whhere the image content is not sufficient to
answer questions. Due to the computational limits, we trained all the models
on part of the whole dataset, where the maximum size of the training examples
are 80000. Table 1 provides statistics of these dataset.

##### Base Transformer model

we utilize the ViLT model with pre-trained weights derived from ”BERT-base-
uncased”. To maintain consistency in our comparisons, we employ the same pre-
trained parameters for the ViLT model across all experiments. This
configuration entails 11 self-attention blocks (SAB), each with a
dimensionality of 768 and 12 attention heads. Additionally, we integrate one
task attention block (TAB) following the transformer encoder, which also
features 768 hidden dimensions and 12 attention heads.

The model undergoes training on tasks with entirely distinct datasets. Before
sending it to the linear projection embedding block, we resize the image from
its original dimensions to (384, 640). In the self-attention layers of both
the pre-trained transformer and the task-attention block, the image-text
feature vector has a shape of (batch, 768).

In contrast to tasks like COCOQA, PathVqa, and SNLI-VE that only require a
single input image, NLVR2 necessitates two input images along with a text
hypothesis. During the training phase, the text input is merged with one image
before being fed into the transformer, and the outputs from the two images are
concatenated as one. As a result, the transformer’s output becomes (batch,
1536). However, due to the input size constraints of the task-attention block,
we condense the vector, $\mathbf{V}$, from 1536 to 768 by computing the
average value of adjacent elements:

$\textbf{V}^{\prime}[i]=\frac{\textbf{V}[i]+\textbf{V}[i+1]}{2},i=0,2,4,...,1534,$
(11)

where $\textbf{V}^{\prime}$ is the compressed feature vector. We then feed
$\textbf{V}^{\prime}$ into the task-attention block.

##### Baselines for Comparison

since there is no prior method specifically designed for multimodal CL, we use
extensions of Dytox [18], FDR [73], EWC [36], Experiment Replay [50], and
Direct Fine-Tuning, as five alternatives for comparison. Note that direct
fine-tuning serves as a lowerbound to measure the effect of catastrophic
forgetting and effectiveness of CL.

For EWC, we applied a fisher sample percentage of 0.4, meaning that 40% of the
dataset was used to construct the fisher matrix. During training, we set the
EWC weight to 0.1. As for Experience Replay, the sample size of data in the
memory buffer is set at 5%, and the sample frequency is 100. This choice
indicates that for each task, we randomly select 5% of the data and store it
in the memory buffer. During training, after every 100 batches of the current
dataset are processed, we randomly pick a batch of data from the memory buffer
and train the model on that batch from a randomly chosen previous task. For
TAM-CL, given that the knowledge distillation loss is significantly smaller
compared to other losses, we assign a weight of 5000 to $\mathcal{L}_{ikd}$
for all four tasks. Since TAM-CL also employs experience replay as a training
strategy, we maintain the same settings for the sample size of the memory (5%)
and the sample frequency (100).

##### Evaluation Protocol

after learning each task $\mathcal{T}_{i}$, we assess the forgetting rate on
previous tasks $T_{k}$, where $k$ ranges from 1 to $i-1$. To scrutinize the
influence of task order, we conduct experiments with varying task orders that
correspond to different levels of task difficulty. For each learned task, we
report both the final performance accuracy and the forgetting rates relative
to its highest achieved accuracy. Although the agent does not control the task
order, it is informative to study the effect of task order on the model
performance. To facilitate the analysis of how task order impacts the
experiment’s performance, we establish an intuitive difficulty ranking for the
five tasks based on the metric $\frac{\\#\ Training\ Examples}{\\#\ Labels}$,
as outlined in Table 1. Essentially, the larger this value, the easier the
task is. In this context, we rank the five tasks from most difficult to
easiest as follows OKVQA, GQA, COCOQA, SNLI-VE, and NLVR2, with corresponding
scores of 6.19, 43.43, 183.10, 26666.66, and 40000. To mitigate any potential
bias arising from a specific task order, we perform experiments with several
different task orders. Additional results can be found in the Appendix.

The majority of Continual Learning (CL) algorithms typically focus on
relatively homogeneous tasks. To assess the algorithm’s effectiveness in
preventing catastrophic forgetting on our diverse set of tasks, we employ a
normalization-based metric proposed in [65]:

$\small\mathbb{T}_{\textbf{F}}(j\leftarrow i)=\frac{S^{j}_{A}-S^{j\leftarrow
i}_{A}}{S^{j}_{A}-S^{j}_{R}},$ (12)

where $\mathbb{T}_{\textbf{F}}(j\leftarrow i)$ represents the forgetting rate
of task $j$ after learning task $i$. $S^{j}_{A}$ denotes the accuracy of task
$j$ before introducing new tasks, $\mathbb{T}_{\textbf{F}}(j\leftarrow i)$
represents the accuracy of task $j$ after incorporating task $i$, with $i>j$,
and $S^{j}_{R}$ signifies the accuracy of task $j$ when labels are randomly
selected, computed as $\frac{1}{\\#\ labels}$. Essentially, Eq. (12) allows us
to compare forgetting rates across tasks that may have significant differences
in their nature. This is achieved by evaluating how well the model performs
relative to a baseline of total forgetting for that task.

##### Optimization Method

for all tasks, we implement the AdamW optimizer with the following
hyperparameters: learning rate $l=1e-2$, epsilon $\epsilon=1e-8$,
$\beta_{1}=0.9$ and $\beta_{2}=0.98$. This standardized optimization approach
ensures consistency and facilitates meaningful comparisons between tasks in
our experiments. Regarding the training epochs, we employ a conservative
approach due to computational limitations. Specifically, we conduct training
for a limited number of epochs for each task: 5 epochs for SNLI-VE, 10 epochs
for COCOQA, GQA, and NLVR2, and 20 epochs for OKVQA. In all our experiments,
we utilize a single A100 GPU with a batch size of 16.

###  6.2 Comparative Results

COCOQA $\rightarrow$ NLVR2 $\rightarrow$ OKVQA $\rightarrow$ SNLI-VE
$\rightarrow$ GQA
---
| COCOQA | NLVR2 | OKVQA | SNLI-VE | GQA
TAM-CL | 66.09 | 66.07 | 21.24 | 64.05 | 50.86
Finetune | 40.67 | 53.85 | 8.26 | 53.83 | 51.92
FDR | 48.74 | 55.91 | 11.59 | 59.30 | 50.67
EWC | 51.44 | 60.87 | 16.16 | 57.93 | 49.67
ER | 56.30 | 62.06 | 15.62 | 61.12 | 50.12
Dytox | 60.67 | 65.56 | 10.41 | 62.26 | 11.12
Avg. | 53.98 | 60.72 | 13.88 | 59.75 | 44.06
Table 2: Comparison of the accuracy on one task sequences. For each task
sequence, each reported value is the final accuracy of that task after
learning the last task. The last row represents the average accuracy from
different methods on the specific task.

In Table 2, we present our performance results after training on all five
tasks with the task order of:
$\textbf{COCOQA}\rightarrow\textbf{NLVR2}\rightarrow\textbf{OKVQA}\rightarrow\textbf{SNLI-
VE}\rightarrow\textbf{GQA}$. As anticipated, fine-tuning exhibits the lowest
final accuracy for all tasks except GQA. On the flip side, GQA attains the
highest accuracy among all the methods, which aligns with expectations. This
observation underscores the inherent trade-off between accuracy and forgetting
rate; without any constraints on the forgetting rate, the fine-tuning method
is anticipated to achieve relatively higher accuracy at the current task
because the learning capacity of the network is free to learn the current
task. We observe that for most results, the accuracy results of EWC and FDR
fall below the average accuracy for all previous tasks. For example, we
observe the performance of COCOQA for FDR and EWC are 48.74 and 51.44
respectively, and that for average is 53.98. While EWC and FDR can be
effective with smaller models, in line with prior observations for
transformers used in learning sequential tasks [65, 32, 18], we conclude that
regularization-based CL methods are not suitable for CL with large
transformers. We deduce that transformers are sensitive to weight
consolidation, as their learning capacities are significantly compromised when
a large number of weights are frozen. This observation suggests that the
learning ability of the transformer networks is distributed among most weights
and hence when we use weight consolidation, most of the transformer networks
weight are consolidated and as a result the learning ability of the
transformer is compromised. On the contrary, the results for experience replay
demonstrate more promising outcomes. As shown in Table 2, all the accuracy
results for experience replay surpass the average line. Such a performance
illustrates that replaying the knowledge of previous tasks during the training
of the current one can efficiently prevent the parameter distribution shifting
away from that of previous tasks, and remind the model of its previous trained
results. Although ER is not the best among all the baseline, it still
indicates that experience replay proves to be a stable Continual Learning
method, capable of handling models of various sizes and multimodal data. We
also observe that while Dytox achieves results comparable to our method in
some cases, its performance is notably lower in others. For instance, Dytox
achieves an accuracy of 10.41 on OKVQA, only higher than fine-tuning and even
lower than FDR and EWC. Similarly, the accuracy of Dytox on GQA is only 11.12,
which is significantly lower than the rest of methods. Such performance
further confirming Dytox’s tendency to underfit the current task when trying
to balance the accuracy between the previous tasks and the current tasks.
Given that Dytox is designed primarily for visual-unimodal tasks, our
conclusion is that it is not the most suitable choice for multi-modal tasks.

Comparative experiments demonstrate that TAM-CL has a superior performance
compared to other alternative method listed in Table 2. Notably, it achieves
an 8.93% higher accuracy than the second best method for COCOQA, a remarkable
31.43% higher accuracy than the second best method for OKVQA. Moreover, it
falls only 2.08% below the best accuracy for the last task, GQA. This
experiment showcases TAM-CL’s proficiency in mitigating the forgetting rate of
previous tasks while simultaneously upholding a high accuracy on the current
task.

###  6.3 Ablative Experiments

We conduct ablation experiments to assess the importance of individual
components in our design. We focused on evaluating the impact of the
$\mathcal{L}_{ikd}$ loss function, the training strategy involving experience
replay, and the utilization of the task attention block. These experiments
were conducted using the task sequence order:
$\textbf{OKVQA}\rightarrow\textbf{GQA}\rightarrow\textbf{COCOQA}\rightarrow\textbf{SNLI-
VE}\rightarrow\textbf{NLVR2}$.

Results of this experiment are presented in Table 3. To further study the
significance of each part, besides the final accuracy, we also present both
the forgetting rate of each task which indicates the percentage of dropping
between its original accuracy and final accuracy. Our observation confirms the
crucial role of each component in our TAM-CL approach to obtain optimal
performance. When comparing the full TAM-CL pipeline to ablation tasks, we
consistently observed superior performance in terms of both forgetting rate
and accuracy. This result underscores the indispensability of every element in
our approach for achieving optimal results. Notably, omitting the
$\mathcal{L}_{ikd}$ loss resulted in an average performance drop of 12.74%
across the four tasks, highlighting the critical importance of intermediate
knowledge distillation in knowledge transfer and mitigating the forgetting
effects. The observation that ablating the entire task-attention block, which
encompasses the $\mathcal{L}{ikd}$ loss, results in slightly better forgetting
rates and accuracy performances compared to solely ablating the
$\mathcal{L}{ikd}$ loss may initially seem counterintuitive. However, we posit
that this discrepancy arises from the fact that the $\mathcal{L}_{ikd}$ loss
plays a crucial role in training the task-attention layer, which is the
central component for continual learning in the model. As a result, its impact
on performance is more pronounced.

OKVQA $\rightarrow$ GQA $\rightarrow$ COCOQA $\rightarrow$ SNLI-VE
$\rightarrow$ NLVR2
---
| OKVQA | GQA | COCOQA | SNLI-VE | NLVR2
TAM-CL | 15.09 (54.56%) | 41.91 (27.59%) | 60.86 (18.36%) | 60.47 (29.34%) | 65.86
ablation TAB | 13.97 (57.22%) | 39.95 (28.69%) | 59.92 (19.65%) | 57.35 (37.22%) | 65.71
ablation $\mathcal{L}_{ikd}$ | 11.53 (65.36%) | 37.19 (34.07%) | 58.62 (21.71%) | 58.34 (35.30%) | 65.21
ablation replay | 2.11 (93.69%) | 27.25 (51.84%) | 41.59 (44.33%) | 50.06 (56.40%) | 65.64

Table 3: Ablation experiment: the final accuracy ($\uparrow$) and forgetting
rate ($\downarrow$) for each task in the ablation methods.

As anticipated, our training strategy involving experience replay plays an
important role in achieving optimal performance by mitigating catastrophic
forgetting of previous tasks. We observe that experience replay is
particularly beneficial for maintaining the accuracy of tasks early in the
sequence. For instance, in Table 3, when experience replay is ablated, the
forgetting rate of OKVQA after learning NLVR2 surges to 93.69%. In contrast,
the forgetting rate of GQA after learning NLVR2 is 51.84%, and the forgetting
rate of COCOQA after NLVR2 is 44.33%, both of which are significantly lower
than 93.69%. This highlights the effectiveness of experience replay in
preserving knowledge across tasks. These results demonstrate that the optimal
performance of our method stems from using all the three primary ideas that we
proposed.

###  6.4 Analytic Experiments

In this section, we offer analytic experiments to empirically study the
various aspects of the proposed algorithm.

####  6.4.1 The Effect of the Task Order

To further analyze the performance of TAM-CL, we compare its performance on
four different task sequences and examine the impact of task order on
catastrophic forgetting. While we acknowledge that, in practice, we don’t have
control over the task order as it is determined by the environment, we still
aim to understand the effect of task order assuming it is predefined. This
analysis provides valuable insights into how TAM-CL adapts to different task
sequences.

Tables 4-7 presents the accuracy of current task and the forgetting rate
results of previous tasks for each time step, allowing us to evaluate the
forgetting rate of previous tasks after the training of every individual task.
Upon inspecting the results for these task sequences, we can conclude that the
task order plays a crucial role in the performance of CL methods. For
instance, in table 4, where OKVQA is the third task, its forgetting rate is
22.59% after learning the final task. On the other hand, in 5, where OKVQA is
the fourth task, its forgetting rate increases to 27.29%. Theoretically, as
the tasks accumulates, the earlier task will have higher forgetting rate, but
the forgetting rate of OKVQA as the thrid task in the first task sequence is
lower than that as the fourth task in the second task sequence, which
indicates that the task order does affect the performance of each task.

Based on our intuitive metric of task difficulty described in Section 6.1, we
hypothesize that in table 4, SNLI-VE and NLVR2 are relatively easier tasks,
requiring fewer parameter distribution shifts to achieve high accuracy. As a
result, the parameter distribution for OKVQA is less affected. However, in
table 5, GQA is the second most difficult task, necessitating a larger
distribution shift from the previous task, OKVQA, to achieve higher
performance. We can observe a similar correlation between forgetting rate and
task difficulty in table 5. For example, after training on the two relatively
easier tasks, NLVR2 and SNLI-VE, the forgetting rates for COCOQA are 7.40% and
8.45% respectively, and the forgetting rate increased by only 1.05%. In
contrast, after training on OKVQA and GQA, the forgetting rates for OKVQA are
11.08% and 19.30%, which is a 2.63% and 8.27% incrasing of forgetting rate,
which is significantly higher than after NLVR2 and SNLI-VE. Similarly, in
table 6, where GQA is the first task, the forgetting rate of GQA after learnt
on OKVQA, the third task, is 6.81%, which is 3.59 higher than its previous
task. While SNLI-VE is the fourth task, the forgetting rate of GQA after it is
4.60%, 2.21% lower than the forgetting rate of OKVQA, which again proves the
relative easiness of SNLI-VE, and thus the hypothesis of section 6.1.

We also note an interesting observation: even though NLVR2 is considered a
relatively easier task, in table 4, after training on NLVR2, the forgetting
rates for previous tasks increase by more than twofold. We hypothesize that
while NLVR2 may be easy to train, it possesses a fundamental difference from
the other tasks. Specifically, NLVR2 takes two images as a single input and
performs visual reasoning, whereas all the other tasks only process one image
at a time. This unique property of the NLVR2 task leads to a significant shift
in parameter distribution, though not to the same extent as the difficulty
level of the task.

Furthermore, we have observed an intriguing phenomenon in that after training
on certain specific tasks, the forgetting rate of previous tasks can actually
decrease. In table 4, after training on OKVQA, the forgetting rates for GQA
and COCOQA are 6.81% and 5.81% respectively. However, after training on the
subsequent task, SNLI-VE, the forgetting rates for these tasks decrease to
4.60% and 4.58%. At the same time, in table 7, the forgetting rate of COCOQA
after the second task GQA, is 3.86%, while that after the third task, SNLI-VE,
is 3.37%. This suggests a potential for forward transfer, which is an aspect
we may delve deeper into in future research.

COCOQA $\rightarrow$ NLVR2 $\rightarrow$ OKVQA $\rightarrow$ SNLI-VE
$\rightarrow$ GQA
---
|  COCOQA | NLVR2 | OKVQA
|  |  |  COCOQA |  |  COCOQA |  NLVR2
TAM-CL |  76.09 |  68.88 |  7.89% |  31.63 |  9.45% |  3.72%
Finetune |  75.88 |  68.71 |  70.21% |  32.24 |  19.41% |  45.75%
EWC |  75.89 |  67.93 |  73.65% |  31.39 |  20.95% |  64.20%
FDR |  71.81 |  58.43 |  28.99% |  26.01 |  24.54% |  59.07%
ER |  76.77 |  68.35 |  12.08% |  31.19 |  13.64% |  34.04%
Dytox |  76.68 |  68.41 |  10.37% |  13.93 |  16.15% |  23.35%

| SNLI-VE
---|---
|  |  COCOQA |  NLVR2 |  OKVQA
TAM-CL |  71.32 |  10.25% |  6.98% |  15.19%
Finetune |  71.6 |  31.71% |  41.57% |  71.71%
EWC |  71.03 |  23.55% |  43.49% |  29.75%
FDR |  68.56 |  71.13% |  51.00% |  52.28%
ER |  71.37 |  12.05% |  10.46% |  26.57%
Dytox |  69.82 |  14.30% |  16.25% |  2.15%

| GQA
---|---
|  |  COCOQA |  NLVR2 |  OKVQA |  SNLI-VE
TAM-CL |  50.86 |  13.15% |  14.87% |  22.59% |  19.13%
Finetune |  51.92 |  47.09% |  79.42% |  74.37% |  46.81%
EWC |  49.67 |  33.10% |  42.19% |  49.40% |  20.45%
FDR |  50.67 |  32.12% |  29.89% |  55.44% |  28.26%
ER |  50.12 |  27.20% |  34.27% |  50.10% |  27.19%
Dytox |  11.12 |  20.52% |  15.37% |  25.26% |  20.70%

Table 4: Accuracy and forgetting rate of task order: COCOQA $\rightarrow$
NLVR2 $\rightarrow$ OKVQA $\rightarrow$ SNLI-VE $\rightarrow$ GQA

COCOQA $\rightarrow$ NLVR2 $\rightarrow$ SNLI-VE $\rightarrow$ OKVQA
$\rightarrow$ GQA
---
|  COCOQA | NLVR2 | SNLI-VE
|  |  |  COCOQA |  |  COCOQA |  NLVR2
TAM-CL |  76.73 |  69.99 |  7.40% |  71.12 |  8.45% |  6.58%
Finetune |  76.95 |  67.83 |  74.75% |  71.72 |  62.51% |  38.12%
EWC |  76.69 |  71.67 |  64.88% |  70.84 |  58.61% |  27.77%
FDR |  72.43 |  57.99 |  31.31% |  67.59 |  38.96% |  65.08%
ER |  76.38 |  70.35 |  8.15% |  70.69 |  8.65% |  8.52%
Dytox |  76.63 |  68.89 |  8.50% |  70.45 |  9.43% |  7.94%

| OKVQA
---|---
|  |  COCOQA |  NLVR2 |  SNLI-VE
TAM-CL |  30.47 |  11.08% |  14.57% |  19.23%
Finetune |  32.12 |  19.19% |  59.13% |  22.22%
EWC |  31.68 |  18.83% |  52.07% |  15.44%
FDR |  11.49 |  44.23% |  45.52% |  41.27%
ER |  30.91 |  12.92% |  28.12% |  17.92%
Dytox |  19.12 |  17.16% |  32.55% |  37.06%

| GQA
---|---
|  |  COCOQA |  NLVR2 |  SNLI-VE |  OKVQA
TAM-CL |  56.20 |  19.30% |  23.34% |  24.98% |  27.29%
Finetune |  57.03 |  36.53% |  66.16% |  34.11% |  40.90%
EWC |  57.21 |  33.51% |  60.60% |  32.84% |  42.28%
FDR |  42.67 |  41.08% |  46.41% |  74.08% |  50.17%
ER |  57.04 |  26.04% |  32.89% |  27.97% |  41.33%
Dytox |  6.10 |  21.81% |  38.50% |  30.10% |  34.43%

Table 5: Accuracy and forgetting rate of task order: COCOQA $\rightarrow$
NLVR2 $\rightarrow$ SNLI-VE $\rightarrow$ OKVQA $\rightarrow$ GQA

GQA $\rightarrow$ COCOQA $\rightarrow$ OKVQA $\rightarrow$ SNLI-VE
$\rightarrow$ NLVR2
---
|  GQA | COCOQA | OKVQA
|  |  |  GQA |  |  GQA |  COCOQA
TAM-CL |  56.58 |  74.44 |  3.22% |  30.44 |  6.81% |  5.81%
Finetune |  58.11 |  75.95 |  10.60% |  31.99 |  12.78% |  10.65%
EWC |  58.69 |  76.12 |  10.45% |  31.15 |  13.46% |  7.97%
FDR |  55.50 |  71.42 |  12.70% |  28.39 |  12.17% |  6.21%
ER |  57.61 |  75.17 |  4.17% |  31.21 |  9.58% |  4.26%
Dytox |  56.83 |  75.22 |  1.93% |  29.87 |  11.29% |  7.87%

| SNLI-VE
---|---
|  |  GQA |  COCOQA |  OKVQA
TAM-CL |  71.49 |  4.60% |  4.58% |  8.51%
Finetune |  72.54 |  9.91% |  7.35% |  15.93%
EWC |  72.10 |  10.82% |  6.43% |  11.46%
FDR |  70.17 |  8.29% |  7.11% |  13.43%
ER |  72.39 |  6.00% |  5.03% |  10.47%
Dytox |  72.44 |  8.13% |  5.40% |  10.07%

| NLVR2
---|---
|  |  GQA |  COCOQA |  OKVQA |  SNLI-VE
TAM-CL |  70.04 |  9.43% |  9.72% |  20.72% |  14.98%
Finetune |  70.32 |  55.09% |  63.90% |  77.80% |  53.18%
EWC |  67.57 |  57.47% |  63.77% |  80.89% |  51.90%
FDR |  53.60 |  33.54% |  56.20% |  63.64% |  36.42%
ER |  70.38 |  13.19% |  9.90% |  23.43% |  19.93%
Dytox |  67.34 |  15.86% |  13.90% |  41.45% |  27.61%

Table 6: Accuracy and forgetting rate of task order: GQA $\rightarrow$ COCOQA
$\rightarrow$ OKVQA $\rightarrow$ SNLI-VE $\rightarrow$ NLVR2

COCOQA $\rightarrow$ GQA $\rightarrow$ SNLI-VE $\rightarrow$ OKVQA
$\rightarrow$ NLVR2
---
|  COCOQA | GQA | SNLI-VE
|  |  |  COCOQA |  |  COCOQA |  GQA
TAM-CL |  76.59 |  57.83 |  3.86% |  72.03 |  3.37% |  1.66%
Finetune |  76.84 |  58.72 |  18.86% |  72.09 |  23.57% |  5.64%
EWC |  76.49 |  57.27 |  18.65% |  72.11 |  21.18% |  4.26%
FDR |  72.00 |  53.42 |  14.66% |  68.25 |  20.47% |  2.74%
ER |  76.88 |  58.07 |  17.52% |  72.19 |  9.01% |  1.71%
Dytox |  76.77 |  25.41 |  4.51% |  72.33 |  3.77% |  2.14%

| OKVQA
---|---
|  |  COCOQA |  GQA |  SNLI-VE
TAM-CL |  30.74 |  6.64% |  4.78% |  5.66%
Finetune |  32.77 |  24.37% |  9.89% |  9.27%
EWC |  31.78 |  22.88% |  5.88% |  8.65%
FDR |  26.57 |  26.88% |  8.84% |  4.77%
ER |  33.22 |  11.64% |  6.11% |  5.74%
Dytox |  13.74 |  10.61% |  4.98% |  6.74%

| NLVR2
---|---
|  |  COCOQA |  GQA |  SNLI-VE |  OKVQA
TAM-CL |  69.15 |  10.09% |  11.74% |  14.60% |  16.79%
Finetune |  67.27 |  76.18% |  55.89% |  50.07% |  75.11%
EWC |  68.43 |  78.72% |  54.84% |  54.04% |  79.46%
FDR |  58.73 |  38.38% |  40.97% |  47.51% |  52.12%
ER |  67.63 |  16.08% |  12.26% |  24.12% |  19.80%
Dytox |  69.01 |  18.20% |  23.63% |  26.89% |  29.90%

Table 7: Accuracy and forgetting rate of task order: COCOQA $\rightarrow$ GQA
$\rightarrow$ SNLI-VE $\rightarrow$ OKVQA $\rightarrow$ NLVR2

Overall, we also observe that experience replay has a relatively high accuracy
and low forgetting rates compared to the two regularization-based methods.
However, its capacity to prevent catastrophic forgetting in later tasks is not
as consistent as in early tasks. For example, in Table 4, after training on
the last task, GQA, the forgetting rate of OKVQA for ER is 50.10%, which is
close to FDR and higher than EWC. In contrast, after training on SNLI-VE, the
forgetting rate of OKVQA for ER is 26.57%, considerably better than the
forgetting rate for FDR, which is 52.28%. On the other hand, Dytox exhibits
stable capacity to prevent catastrophic forgetting, which is not significantly
affected by the length of the task sequence. However, it loses competitiveness
due to its lower performance in training the current task. For example, in
Table 4, the accuracy of OKVQA for Dytox is 13.93, while the accuracy for all
the other methods is above 25. Additionally, Dytox only achieves 11.12% in
GQA, whereas the accuracy for all other methods is above 50%. In table 7, the
accuracy of Dytox on OKVQA is only 19.12 and that on GQA is even 6.10, which
are both significantly lower than the performance of other methods.
Ultimately, although TAM-CL may not always lead in every individual forgetting
rate, it surpasses the other methods in 87.5% of the total forgetting rates.
In the remaining exceptional cases, TAM-CL still demonstrates above-average
performance, underscoring its capacity and stability in preventing
catastrophic forgetting. Note that while the accuracy of TAM-CL may not always
be the highest among the six methods, the differences between the top accuracy
and TAM-CL’s accuracy are consistently below 5%. Given that our main focus is
on the improvement of forgetting rate, we consider these slight accuracy
differences to be within an acceptable range.

####  6.4.2 Fine-tuning on Unimodal Tasks

Despite focusing on multimodal learning, the ability for unimodal learning is
important. The reason is that in practice, some of the modalities maybe
occluded. For this reason, we performed experiments on using our approach on
unimodal tasks. We tested our method’s performance on three different unimodal
vision datasets, coco-classification [39], inaturalist [27] and places365
[90], along with other baseline performance shown in 6.2.To apply the
multimodal transformer backbone, ViLT, to unimodal vision-only task, we
created a vision-language pair for every image data, while the language input
is universally “This is an image” for all the images.

We also performed experiments on three different natural language datasets,
including PIQA [7], CommonsenseQA [71], and Hellaswag [86]. To form image-text
pairs for the multimodal transformer input, an averaged image of all MS-COCO
training image dataset is used to pair with every text sample in the dataset.
We note that as ViLT only allows language token with maximum size of 40 as
input which might not be sufficient for language tasks. To deal with this
issue, we first down-sample the meaningless image and thus reduce its token
length from 144 to 16. Then, we extend the available length token by creating
copies of pre-trained ViLT’s language positional embeddings, and concatenate
the embeddings to get the extended positional embeddings.

In table 8, we present our experimental results for vision-only and language-
only unimodal tasks. We report the final accuracy after training the model on
the last task, PLACES365. As shown in the table, although TAM-CL is
specifically designed for multimodal continual learning setting, it’s
performance on vision-only tasks on average remains the top performer. For
COCO classification and iNaturalist datasets, TAM-CL achieves 73.03% and
50.63% higher than all the baseline scores, respectively. Regarding the
accuracy of PLACES365, EWC achieves the best score, 43.64%, and finetuning get
the second high performance. Although that of TAM-CL is not the best
performer, there’s only a 0.4% difference between TAM-CL’s accuracy and the
best performer. Overall, TAM-CL achieves the top performance in vision-only
task compared with those unimodal continual learning methods.

COCO$\rightarrow$iNaturalist$\rightarrow$PLACES365
---
| COCO | iNaturalist | PLACES365 | Avg.
TAM-CL |  73.03 |  50.63 | 43.44 |  55.66
Dytox | 72.56 | 48.88 | 43.29 | 54.91
Replay | 71.91 | 48.79 | 43.18 | 54.62
EWC | 62.22 | 48.71 |  43.64 | 51.52
Finetune | 62.21 | 47.56 | 43.58 | 51.11
COMMONSENSEQA$\rightarrow$PIQA$\rightarrow$HELLASWAG
| COMMON | PIQA | HELLASWAG | Avg.
TAM-CL |  21.43 |  52.45 | 24.98 |  32.95
Dytox | 21.21 | 47.99 |  25.29 | 31.49
Replay | 20.25 | 51.89 | 24.87 | 32.30
EWC | 21.38 | 50.33 | 25.10 | 32.27
Finetune | 21.38 | 49.62 | 24.99 | 31.99

Table 8: Comparison Experiment with Unimodal Tasks. (Upper) The accuracy of
different methods on vision-only task sequence after trained on the last task.
(Lower) The accuracy of different methods on language-only task sequence after
trained on the last task.

As for the language-only unimodal tasks, TAM-CL outperforms other baseline
methods with respect to the non-last tasks. It achieves 21.43% accuracy on
COMMONSENSEQA and 52.45% on PIQA, after trained on HELLASWAG dataset. Dytox,
on the other hand, has the worst performance on PIQA, which is 47.99%. As
Dytox is designed specifically for vision-only task, its underperformance on
language-only tasks is not something unexpected. However, it achieves the best
score on the last task, 25.29% on HELLASWAG, which indicates its capacity of
learning the current task, although it cannot efficiently prevent the
catastrophic forgetting of previous tasks. We conclude that although TAM-CL
does not achieve the same level of performance on unimodal tasks, it still
generally outperforms many SOTA continual learning methods on unimodal tasks.
These results offers an advantage for TAM-CL in that it can perform relatively
well in situations in which one of the inputs is occluded.

####  6.4.3 Hyper-parameter Sensitivity Analysis

We study the sensitivity of the TAM-CL with respect to important
hyperparameters of our approach. To this end, we compared the performance of
our proposed method with variant length of the task token, $\tau$, and the
number of frozen layers of the backbone transformer’s encoder.

COCOQA$\rightarrow$NLVR2$\rightarrow$SNLI-VE$\rightarrow$OKVQA$\rightarrow$GQA
---
| GQA | COCOQA | NLVR2 | SNLI-VE | OKVQA
$\tau\ \times$ 1 | 51.23 | 64.64 |  65.60 |  21.03 | 65.84
$\tau\ \times$ 3 | 50.89 | 64.94 | 61.95 | 19.83 |  67.33
$\tau\ \times$ 5 |  51.66 | 64.85 | 61.95 | 19.47 | 65.34
$\tau\ \times$ 10 | 51.25 |  66.77 | 63.09 | 20.31 | 66.30
Freeze 0 | 49.15 | 58.18 | 62.37 | 21.86 | 66.00
Freeze 3 | 50.47 | 63.14 | 62.25 |  22.48 |  67.20
Freeze 6 |  51.23 |  65.78 |  65.60 | 21.03 | 65.84
Freeze 9 | 50.57 | 65.48 | 63.06 | 17.92 | 62.50
Freeze all | 49.79 | 60.81 | 61.46 | 15.98 | 65.36

Table 9: Experiments of variant of task token length and number of frozen
layers on one multimodal task sequence. The accuracy for each task after
trained on the last task, GQA, is presented. (Upper) The accuracy of different
token length $\tau\times$ N, where N is the integer multiplier. (Lower) The
accuracy of freezing different number of encoder layers.

As task token is a key component that carries the task-specific knowledge and
help preserve the performance of previous tasks during the continual learning,
we want to analyze whether the change of task token size will affect its
capacity to prevent catastrophic forgetting. With all the other hyper-
parameter fixed, we run experiments on varying dimensions for task token,
$\tau$, with size of 768, 2304, 3840 and 7680, respectively. In Table 9, we
observe that the accuracy gap between different $\tau$ size does exist, but
none of the $\tau$ achieves the absolutely leading performance. The default
setting, 768, gets the best score on the accuracy of NLVR2 and SNLI-VE, which
are 65.60 and 21.03 respectively, but under-performed by GQA, COCOQA and OKVQA
compared with other $\tau$ sizes. At the same time, with $\tau$ size of 2304,
which is three times larger than the default, TAM-CL achieve the best accuracy
on OKVQA, 67.33. With $\tau$ size of 3840, TAM-CL achieve the best on the last
task, the GQA dataset. With 10 size of $\tau$, TAM-CL outperforms all the
other on COCOQA, 66.77. Throughout all the experiment results, besides the
performance of default $\tau$ on NLVR2, which outperforms the second best by
3.97%, the leading accuracy of specific $\tau$ size on different tasks
relatively trivial, which indicates that the variant of task token $\tau$ does
not perform the key rule in affecting the performance of TAM-CL as a whole.
This conclusion is important because it shows sensitivity with respect to this
hyper-parameter is not significant and we do not need to fine-tune it
extensively for optimal performance.

We also studied the effect of freezing different number of layers of the
transformer encoder during the training stage on the downstream performance.
We observe that by freezing 0 and 3 layers, the performance on SNLI-VE and
OKVQA would be the best and the second best scores, respectively. Our default
setting, which freezes 6 layers, outperforms all other setting on GQA, COCOQA
and NLVR2. Nevertheless, by freezing 9 and all encoder layers, the performance
of SNLI-VE drops significantly, 17.92% and 15.98% compared with all above
21.00%. We speculate that less frozen layers in the encoder, the model can
learn a single task better. However, its ability of preventing catastrophic
forgetting would decrease. Meanwhile, if most of the encoder layers are
frozen, it harms the capacity of the model to learn the current task. Also,
the preservation of performance on previous tasks is not improved. We conclude
that freezing 6 layers is an optimal choice which on average leads to
acceptable performance on most datasets.

##  7 Conclusion

We developed a multimodal continual learning algorithm when a transformer
architecture is the base model. My approach involves dynamic model expansion,
through task-specific attention, knowledge distillation, and experience replay
to mitigate catastrophic forgetting and enabling positive knowledge transfer.
The experiments demonstrate the effectiveness of your approach, achieving
state-of-the-art performance in terms of both forward transfer and
catastrophic forgetting. The TAM-CL architecture opens up new avenues for
studying continual learning in multimodal settings. Future works include
exploring using adapter weights in architecture to improve the learning
efficiency during the training phase.

## References

  * [1] Anas M Ali, Bilel Benjdira, Anis Koubaa, Walid El-Shafai, Zahid Khan, and Wadii Boulila.  Vision transformers in image restoration: A survey.  Sensors, 23(5):2385, 2023.
  * [2] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars.  Memory aware synapses: Learning what (not) to forget.  In Proceedings of the European conference on computer vision (ECCV), pages 139–154, 2018.
  * [3] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  Vivit: A video vision transformer.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6836–6846, 2021.
  * [4] Craig Atkinson, Brendan McCane, Lech Szymanski, and Anthony Robins.  Pseudo-rehearsal: Achieving deep reinforcement learning without catastrophic forgetting.  Neurocomputing, 428:291–307, 2021.
  * [5] Reza Azad, Moein Heidari, Yuli Wu, and Dorit Merhof.  Contextual attention network: Transformer meets u-net.  In International Workshop on Machine Learning in Medical Imaging, pages 377–386. Springer, 2022.
  * [6] Deblina Bhattacharjee, Tong Zhang, Sabine Süsstrunk, and Mathieu Salzmann.  Mult: An end-to-end multitask learning transformer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12031–12041, 2022.
  * [7] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al.  PIQA: Reasoning about physical commonsense in natural language.  In AAAI Conference on Artificial Intelligence, 2020.
  * [8] Cristian Buciluundefined, Rich Caruana, and Alexandru Niculescu-Mizil.  Model compression.  In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’06, page 535–541, New York, NY, USA, 2006. Association for Computing Machinery.
  * [9] Yuliang Cai, Jesse Thomason, and Mohammad Rostami.  Task-attentive transformer architecture for continual learning of vision-and-language tasks using knowledge distillation.  In EMNLP, 2023.
  * [10] Rich Caruana.  Multitask learning.  Springer, 1998.
  * [11] Guobin Chen, Wongun Choi, Xiang Yu, Tony X. Han, and Manmohan Chandraker.  Learning efficient object detection models with knowledge distillation.  In Neural Information Processing Systems, 2017.
  * [12] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX, pages 104–120. Springer, 2020.
  * [13] Zhiyuan Chen and Bing Liu.  Lifelong machine learning.  Synthesis Lectures on Artificial Intelligence and Machine Learning, 12(3):1–207, 2018.
  * [14] Georgios Chochlakis, Tejas Srinivasan, Jesse Thomason, and Shrikanth Narayanan.  Vault: Augmenting the vision-and-language transformer with the propagation of deep language representations.  arXiv preprint arXiv:2208.09021, 2022.
  * [15] Wei Cong, Yang Cong, Gan Sun, Yuyang Liu, and Jiahua Dong.  Self-paced weight consolidation for continual learning.  IEEE Transactions on Circuits and Systems for Video Technology, 2023.
  * [16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations.
  * [17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2020.
  * [18] Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord.  Dytox: Transformers for continual learning with dynamic token expansion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9285–9295, 2022.
  * [19] Beyza Ermis, Giovanni Zappella, Martin Wistuba, Aditya Rawal, and Cédric Archambeau.  Continual learning with transformers for image classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3774–3781, 2022.
  * [20] Beyza Ermis, Giovanni Zappella, Martin Wistuba, Aditya Rawal, and Cedric Archambeau.  Memory efficient continual learning with transformers.  Advances in Neural Information Processing Systems, 35:10629–10642, 2022.
  * [21] William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney.  Revisiting fundamentals of experience replay.  In International Conference on Machine Learning, pages 3061–3071. PMLR, 2020.
  * [22] Robert M French.  Catastrophic forgetting in connectionist networks.  Trends in cognitive sciences, 3(4):128–135, 1999.
  * [23] Kirill Gavrilyuk, Ryan Sanford, Mehrsan Javan, and Cees G. M. Snoek.  Actor-transformers for group activity recognition, 2020.
  * [24] Honggui Han, Meiting Sun, and Fangyu Li.  Online aware synapse weighted autoencoder for recovering random missing data in wastewater treatment process.  IEEE Transactions on Artificial Intelligence, 2023.
  * [25] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al.  A survey on vision transformer.  IEEE transactions on pattern analysis and machine intelligence, 45(1):87–110, 2022.
  * [26] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [27] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The inaturalist species classification and detection dataset, 2018.
  * [28] Ronghang Hu and Amanpreet Singh.  Unit: Multimodal multitask learning with a unified transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1439–1449, 2021.
  * [29] Zizhao Hu and Mohammad Rostami.  Class-incremental learning using generative experience replay based on time-aware regularization.  arXiv preprint arXiv:2310.03898, 2023.
  * [30] Drew A. Hudson and Christopher D. Manning.  Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019.
  * [31] David Isele, Mohammad Rostami, and Eric Eaton.  Using task features for zero-shot knowledge transfer in lifelong learning.  In Ijcai, volume 16, pages 1620–1626, 2016.
  * [32] Xisen Jin, Bill Yuchen Lin, Mohammad Rostami, and Xiang Ren.  Learn continually, generalize rapidly: Lifelong knowledge accumulation for few-shot learning.  In Findings of Empirical Methods in Natural Language Processing (Findings of EMNLP), 2021.
  * [33] Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, Ryuichi Yamamoto, Xiaofei Wang, et al.  A comparative study on transformer vs rnn in speech applications.  In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 449–456. IEEE, 2019.
  * [34] Zixuan Ke, Bing Liu, Nianzu Ma, Hu Xu, and Lei Shu.  Achieving forgetting prevention and knowledge transfer in continual learning.  Advances in Neural Information Processing Systems, 34:22443–22456, 2021.
  * [35] Wonjae Kim, Bokyung Son, and Ildoo Kim.  Vilt: Vision-and-language transformer without convolution or region supervision.  In International Conference on Machine Learning, pages 5583–5594. PMLR, 2021.
  * [36] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017.
  * [37] Joseph KJ and Vineeth N Balasubramanian.  Meta-consolidation for continual learning.  Advances in Neural Information Processing Systems, 33:14374–14386, 2020.
  * [38] Zihan Li, Yunxiang Li, Qingde Li, Puyang Wang, Dazhou Guo, Le Lu, Dakai Jin, You Zhang, and Qingqi Hong.  Lvit: language meets vision transformer in medical image segmentation.  IEEE transactions on medical imaging, 2023.
  * [39] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft COCO: Common objects in context.  In European Conference on Computer Vision (ECCV), 2014.
  * [40] Zhiqiu Lin, Jia Shi, Deepak Pathak, and Deva Ramanan.  The CLEAR benchmark: Continual learning on real-world imagery.  In Neural Information Processing Systems (NeurIPS), 2021.
  * [41] Yaoyao Liu, Bernt Schiele, Andrea Vedaldi, and Christian Rupprecht.  Continual detection transformer for incremental object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23799–23808, 2023.
  * [42] David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning, 2022.
  * [43] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019.
  * [44] Mehrnoosh Mirtaheri, Mohammad Rostami, and Aram Galstyan.  History repeats: Overcoming catastrophic forgetting for event-centric temporal knowledge graph completion.  In Findings of ACL, 2023.
  * [45] Abdelrahman Mohamed, Rushali Grandhe, KJ Joseph, Salman Khan, and Fahad Khan.  D3former: Debiased dual distilled transformer for incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2420–2429, 2023.
  * [46] Vishvak Murahari, Dhruv Batra, Devi Parikh, and Abhishek Das.  Large-scale pretraining for visual dialog: A simple state-of-the-art baseline, 2020.
  * [47] Francesco Pelosin, Saurav Jha, Andrea Torsello, Bogdan Raducanu, and Joost van de Weijer.  Towards exemplar-free continual learning in vision transformers: an account of attention, functional and weight regularization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3820–3829, 2022.
  * [48] Mengye Ren, Ryan Kiros, and Richard Zemel.  Exploring models and data for image question answering, 2015.
  * [49] Anthony Robins.  Catastrophic forgetting, rehearsal and pseudorehearsal.  Connection Science, 7(2):123–146, 1995.
  * [50] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne.  Experience replay for continual learning.  Advances in Neural Information Processing Systems, 32, 2019.
  * [51] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets, 2015.
  * [52] Mohammad Rostami.  Lifelong domain adaptation via consolidated internal distribution.  Advances in neural information processing systems, 34:11172–11183, 2021.
  * [53] Mohammad Rostami and Aram Galstyan.  Cognitively inspired learning of incremental drifting concepts.  In IJCAI, 2023.
  * [54] Mohammad Rostami and Aram Galstyan.  Overcoming concept shift in domain-aware settings through consolidated internal distributions.  In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 9623–9631, 2023.
  * [55] Mohammad Rostami, David Isele, and Eric Eaton.  Using task descriptions in lifelong machine learning for improved performance and zero-shot transfer.  Journal of Artificial Intelligence Research, 67:673–704, 2020.
  * [56] Mohammad Rostami, Soheil Kolouri, Kyungnam Kim, and Eric Eaton.  Multi-agent distributed lifelong learning for collective knowledge acquisition.  arXiv preprint arXiv:1709.05412, 2017.
  * [57] Mohammad Rostami, Soheil Kolouri, Praveen Pilly, and James McClelland.  Generative continual concept learning.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 5545–5552, 2020.
  * [58] Mohammad Rostami, Soheil Kolouri, and Praveen K Pilly.  Complementary learning for overcoming catastrophic forgetting using experience replay.  In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 3339–3345, 2019.
  * [59] Mohammad Rostami, Leonidas Spinoulas, Mohamed Hussein, Joe Mathai, and Wael Abd-Almageed.  Detection and continual learning of novel face presentation attacks.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 14851–14860, 2021.
  * [60] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  arXiv preprint arXiv:1606.04671, 2016.
  * [61] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver.  Prioritized experience replay.  arXiv preprint arXiv:1511.05952, 2015.
  * [62] Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed.  Learning audio-visual speech representation by masked multimodal cluster prediction, 2022.
  * [63] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim.  Continual learning with deep generative replay.  Neural Information Processing Systems (NeurIPS), 2017.
  * [64] Daniel L Silver, Qiang Yang, and Lianghao Li.  Lifelong machine learning systems: Beyond learning algorithms.  In 2013 AAAI spring symposium series, 2013.
  * [65] Tejas Srinivasan, Ting-Yun Chang, Leticia Leonor Pinto Alva, Georgios Chochlakis, Mohammad Rostami, and Jesse Thomason.  Climb: A continual learning benchmark for vision-and-language tasks.  In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
  * [66] Tejas Srinivasan, Furong Jia, Mohammad Rostami, and Jesse Thomason.  I2i: Initializing adapters with improvised knowledge.  In Conference on Lifelong Learning Agents, 2023.
  * [67] Serban Stan and Mohammad Rostami.  Unsupervised model adaptation for continual semantic segmentation.  In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 2593–2601, 2021.
  * [68] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  In International Conference on Learning Representations, 2019.
  * [69] Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi.  A corpus for reasoning about natural language grounded in photographs, 2018.
  * [70] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid.  Videobert: A joint model for video and language representation learning.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 7464–7473, 2019.
  * [71] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.  CommonsenseQA: A question answering challenge targeting commonsense knowledge.  In North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
  * [72] Hao Tan and Mohit Bansal.  Lxmert: Learning cross-modality encoder representations from transformers.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5100–5111, 2019.
  * [73] Michalis K. Titsias, Jonathan Schwarz, Alexander G. de G. Matthews, Razvan Pascanu, and Yee Whye Teh.  Functional regularisation for continual learning with gaussian processes, 2020.
  * [74] Gido M Van de Ven and Andreas S Tolias.  Three scenarios for continual learning.  arXiv preprint arXiv:1904.07734, 2019.
  * [75] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [76] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2017.
  * [77] Johannes Von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe.  Continual learning with hypernetworks.  arXiv preprint arXiv:1906.00695, 2019.
  * [78] Xin Wang, Yue Liu, Jiapei Fan, Weigao Wen, Hui Xue, and Wenwu Zhu.  Continual few-shot learning with transformer adaptation and knowledge regularization.  In Proceedings of the ACM Web Conference, volume 2023, 2023.
  * [79] Yongqiang Wang, Abdelrahman Mohamed, Due Le, Chunxi Liu, Alex Xiao, Jay Mahadeokar, Hongzhao Huang, Andros Tjandra, Xiaohui Zhang, Frank Zhang, et al.  Transformer-based acoustic modeling for hybrid speech recognition.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6874–6878. IEEE, 2020.
  * [80] Zhen Wang, Liu Liu, Yiqun Duan, Yajing Kong, and Dacheng Tao.  Continual learning with lifelong vision transformer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 171–181, 2022.
  * [81] Ning Xie, Farley Lai, Derek Doran, and Asim Kadav.  Visual entailment: A novel task for fine-grained image understanding, 2019.
  * [82] Peng Xu and Xiatian Zhu.  Deepchange: A large long-term person re-identification benchmark with clothes change, 2022.
  * [83] Peng Xu, Xiatian Zhu, and David A Clifton.  Multimodal learning with transformers: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  * [84] Peng Yang, Dingcheng Li, and Ping Li.  Continual learning for natural language generations with transformer calibration.  In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 40–49, 2022.
  * [85] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang.  Lifelong learning with dynamically expandable networks.  In International Conference on Learning Representations.
  * [86] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  HellaSwag: Can a machine really finish your sentence?  In Association for Computational Linguistics (ACL), 2019.
  * [87] Qiming Zhang, Yufei Xu, Jing Zhang, and Dacheng Tao.  Vitaev2: Vision transformer advanced by exploring inductive bias for image recognition and beyond.  International Journal of Computer Vision, pages 1–22, 2023.
  * [88] Shangtong Zhang and Richard S Sutton.  A deeper look at experience replay.  arXiv preprint arXiv:1712.01275, 2017.
  * [89] Wen Zhang, Lingfei Deng, Lei Zhang, and Dongrui Wu.  A survey on negative transfer.  IEEE/CAA Journal of Automatica Sinica, 10(2):305–329, 2022.
  * [90] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.