# Task Formulation Matters When Learning Continually:
A Case Study in Visual Question Answering

Mavina Nikandrou1, Lu Yu2, Alessandro Suglia1, Ioannis Konstas1, Verena
Rieser1
1 Heriot-Watt University, Edinburgh, United Kingdom
2 Tianjin University of Technology, Tian jin, China
{mn2002,A.Suglia,I.Konstas,V.T.Rieser}<EMAIL_ADDRESS>Work
done at Heriot-Watt University.

###### Abstract

Continual learning aims to train a model incrementally on a sequence of tasks
without forgetting previous knowledge. Although continual learning has been
widely studied in computer vision, its application to Vision+Language tasks is
not that straightforward, as settings can be parameterized in multiple ways
according to their input modalities. In this paper, we present a detailed
study of how different settings affect performance for Visual Question
Answering. We first propose three plausible task formulations and demonstrate
their impact on the performance of continual learning algorithms. We break
down several factors of task similarity, showing that performance and
sensitivity to task order highly depend on the shift of the output
distribution. We also investigate the potential of pretrained models and
compare the robustness of transformer models with different visual embeddings.
Finally, we provide an analysis interpreting model representations and their
impact on forgetting. Our results highlight the importance of stabilizing
visual representations in deeper layers.

##  1 Introduction

The current paradigm to approach Vision+Language (V+L) tasks is to pretrain
large-scale models, which are then finetuned and evaluated on independent and
identically distributed (i.i.d.) data. In practice, the i.i.d. assumption does
not hold: New data becomes available over time, which often results in a shift
in data distribution. One solution is to continuously adapt an existing model
via finetuning. However, this will lead to catastrophic forgetting, i.e.
significant performance degradation on previous data McCloskey and Cohen
(1989); Ratcliff (1990). Continual learning provides a counterpart to i.i.d.
learning: It defines a class of algorithms aiming at incremental learning with
minimal forgetting. This line of work becomes increasingly relevant given the
financial and environmental costs of (re-)training large models Strubell et
al. (2019); Bender et al. (2021), and the limited generalization of static
models Lazaridou et al. (2021).

Figure 1: Predicted answers as the model continuously learns a sequence of
tasks corresponding to different question types. Catastrophic forgetting
causes incorrect predictions for preceding tasks.

While continual learning is widely studied in the computer vision community,
its use within V+L problems remains under-explored. One challenge for applying
continual learning to tasks like Visual Question Answering (VQA) is the lack
of an agreed task definition: Computer vision tasks, like image
classification, often fit in “clear-cut" task, class, or domain incremental
settings Van de Ven and Tolias (2019). This simplification is not always
suitable. First, it rarely holds in real-world scenarios Mi et al. (2020).
Second, it is unsuitable for V+L tasks, which can be parameterized in various
ways according to their different modalities. For example, task definitions
for VQA can either be based on the language reasoning skills (as defined by
the question type, cf. Figure 1) or the visual concepts in the images
Whitehead et al. (2021). Each of these perspectives reflects a different real-
world requirement: New data might be collected with the intention of expanding
the question types or domains to which a VQA system is applied. Similarly,
output spaces are not exclusive. For example, counting questions are
applicable to any visual domain, while binary questions can require different
reasoning skills.

In this work, we provide an in-depth study of task design for Visual Question
Answering (VQA) and its impact when combining pretrained V+L models with
continual learning approaches. We introduce three continual learning settings
based on the VQA-v2 dataset Goyal et al. (2017). Across these settings, we
evaluate several regularization and memory-based continual learning methods.
Our results confirm that algorithmic performance is highly dependent on task
design, order, and similarity, which is in-line with findings for image
classification Van de Ven and Tolias (2019); Yoon et al. (2020); Delange et
al. (2021). We also investigate the potential of pretrained models and their
ability to generalize to unseen tasks in the CL setting. Our results show that
although pretrained representations are more robust than when learning from
scratch, they are still subject to catastrophic forgetting.

In addition, we perform a detailed analysis that relates the amount of
forgetting to task similarity as measured by input embeddings and output
distribution. We find that incremental learning of new question types is the
most challenging setting, as it shows a high divergence in answer
distribution. Figure 1 provides an example where, given the question “What
kind of bird is this?", the last model in the CL task sequence predicts the
incoherent answer “one". To measure more nuanced forgetting, we propose a
novel evaluation metric based on semantic similarity. In the example in Figure
1, changing the answer from “duck" to “seagull" is penalized less.

Finally, we compare two transformer-based models, which use different visual
representations. We find that region features extracted from a fixed object
detection model outperform representations based on image pixels. We track how
representations from each modality change per layer, showing that visual
representations from deeper layers are affected more prominently compared to
language representations.

##  2 Related Work

To the best of our knowledge, this is the first work studying the impact of
task formulation for continual learning in V+L models. The vast majority of
continual learning studies have focused on image classification settings. For
example, previous work has examined the relationship between catastrophic
forgetting and different learning hyper-parameters, such as the activation
function, dropout, and learning rate schedule Goodfellow et al. (2013);
Mirzadeh et al. (2020). Other work has highlighted the important role of task
similarity Ramasesh et al. (2021); Lee et al. (2021) and which properties of
task sequences amplify forgetting Nguyen et al. (2019a).

Continual learning settings are typically categorized as task, class, or
domain incremental Van de Ven and Tolias (2019). In task and class-incremental
settings, new classes are introduced over time, with the difference that task-
incremental settings assume knowledge of the task identity during inference.
In domain-incremental learning, tasks differ in terms of their input
distributions while sharing the same output space.

Previous work on V+L continual learning has studied these settings. For
example, Del Chiaro et al. (2020) and Nguyen et al. (2019b) study continual
learning for domain- and class-incremental image captioning, while Jin et al.
(2020) propose a more flexible setting of “soft task boundaries" for the
masked phrase prediction. More recently, Srinivasan et al. (2022) released a
benchmark that combines V+L task-incremental learning with multimodal and
unimodal transfer. In contrast to these works, we examine the impact of task
specification for V+L on performance and forgetting.

More closely related to our work, Greco et al. (2019) explore the effect of
forgetting in VQA with two question types (‘Wh-’ and binary questions).
Consistent with our findings, they show that task order influences forgetting
and that continual learning methods can alleviate forgetting. However, their
study is limited to only two tasks and does not test the impact of pretrained
models, which has shown potential to mitigate forgetting Mehta et al. (2021).

##  3 Settings for Continual VQA

###  3.1 Problem formulation

In continual learning, model parameters $\bm{\theta}$ are incrementally
updated as new data become available. We assume that samples from tasks
$t=1\dots T$ arrive sequentially as
$D_{t}=\\{\bm{x}_{i},\bm{y}_{i}\\}_{i=1}^{N_{t}}$, where $N_{t}$ is the number
of data for task $t$. Following previous work, VQA is formulated as a multi-
label classification problem with soft targets $\bm{y}_{i}$ Anderson et al.
(2018). Starting from parameters $\bm{\theta}_{t-1}$ of the previous model,
the updated parameters $\bm{\theta}_{t}$ are obtained by training on the new
data $D_{t}$. Some approaches also use a memory $M_{t}$ containing a subset of
samples from previous tasks, e.g. $D_{1},\dots,D_{t-1}$. In our setup, all
tasks share a common output head which is extended with new classes from each
task. This allows inference to be task-agnostic but creates a more challenging
setting than multi-head learning where separate heads are learned for each
task Hussain et al. (2021). At the end of the training sequence, the objective
is to achieve strong performance across all tasks observed so far. This
objective encloses two challenges: 1) minimizing catastrophic forgetting of
tasks seen earlier in training, 2) facilitating positive transfer to improve
performance on new tasks Hadsell et al. (2020).

###  3.2 Task settings

We define three continual learning settings for VQA based on different task
definitions $D_{t}$, as summarized in Table 1. Two of these settings are based
on visual object categories, see Subsection 3.2.1 and one setting is motivated
by language capabilities, see Subsection 3.2.2. Concurrent work Lei et al.
(2022) has followed a similar definition of continual learning settings for
VQA. However, our work focuses on understanding how differences in task
definitions affect the difficulty of the continual learning problem. We study
this problem from the point of view of both the downstream performance as well
as the quality of the learned representations. This is in line with work on
holistic evaluation frameworks for grounded language learning Suglia et al.
(2020).

Setting | Task | Train | Val | Test | Classes
---|---|---|---|---|---
Diverse | Group 1 | 44254 | 11148 | 28315 | 2205
Group 2 | 39867 | 10202 | 22713 | 1874
Group 3 | 37477 | 9386 | 23095 | 1849
Group 4 | 35264 | 8871 | 22157 | 2119
Group 5 | 24454 | 6028 | 14490 | 1777
Taxonomy | Animals | 37270 | 9237 | 22588 | 1331
Food | 26191 | 6612 | 15967 | 1365
Interior | 43576 | 11038 | 26594 | 2096
Sports | 32885 | 8468 | 19205 | 1471
Transport | 41394 | 10280 | 25416 | 1954
Question | Action | 18730 | 4700 | 11008 | 233
Color | 34588 | 8578 | 21559 | 92
Count | 38857 | 9649 | 23261 | 42
Scene | 25850 | 6417 | 14847 | 170
Subcategory | 22324 | 8578 | 21559 | 659

Table 1: Statistics per task within each setting.

####  3.2.1 Visual Settings

We design two settings based on visual object categories, which correspond to
expanding the domain on which the VQA system is applied to. We take advantage
of the fact that images in the VQA-v2 dataset originate from the COCO dataset
Lin et al. (2014) which provides object-level image annotations. Following
previous work in image captioning Del Chiaro et al. (2020), we organize $50$
object categories into five groups. Images with objects from multiple groups
are discarded in order to create clean task splits $D_{t}$ – resulting in a
total of 181K train, 45K validation, and 110K test samples.

For the first setting, Diverse Domains, tasks are defined by grouping the
object categories randomly. Each task is assigned a balanced count of $10$
distinct objects resulting in five tasks. This type of setting corresponds to
common practice of continual learning research within computer vision Rebuffi
et al. (2017); Lomonaco and Maltoni (2017), and reflects a real-world scenario
where sequential data do not necessarily follow a taxonomy.

The second setting, Taxonomy Domains groups objects based on their common
super-category as in Del Chiaro et al. (2020). This results in five tasks:
Animals, Food, Interior, Sports, and Transport. Note that the number of object
classes per task under this definition is unbalanced since splits depend on
the size of the super-category. More details on each task can be found in
Appendix A.

####  3.2.2 Language Setting

We create a third setting Question Types, where each task corresponds to
learning to answer a different category of questions. We use a classification
scheme developed by Whitehead et al. (2021) to form a sequence of five tasks:
Count, Color, Scene-level, Subcategory, and Action recognition. The splits for
Count, Color, and Subcategory questions are obtained from Whitehead et al.
(2021). We create two additional tasks from the remaining questions. In
particular, we cluster question embeddings from Sentence-BERT Reimers and
Gurevych (2019) 111We use the ‘all-MiniLM-L6-v2’ model and Fast Clustering
algorithm from the sentence-transformers package (https://www.sbert.net/). so
that each cluster has at least 15 questions and a minimum cosine similarity of
0.8 between all embeddings. We annotate clusters as ‘scene’, ‘action’ or
‘irrelevant’ question types. Based on a seed of 10K annotated questions, we
retrieve all other questions with similarity above 0.8 and label them using
the K-nearest neighbor algorithm ($K=5$). Question Types have a total of 140K
train, 35K validation and 84K test samples (cf. Table 1). Common question
words and answers per task are presented in the Appendix (Figure 9).

##  4 Experimental Framework

###  4.1 Models

In our experiments, we use two single-stream transformer models, UNITER-base
Chen et al. (2020) and ViLT-base Kim et al. (2021) that differ in terms of how
images are embedded at the input level. UNITER relies on region features
extracted from a frozen pretrained object detector, while ViLT directly embeds
image patches. Both models are pretrained on the same data that include among
others in-domain images for VQA-v2, i.e. COCO captions Lin et al. (2014).

###  4.2 Continual Learning Methods

We benchmark common continual learning algorithms, including regularization-
and replay-based approaches. We investigate two regularization-based
approaches: Learning without Forgetting (LwF) Li and Hoiem (2018), which uses
knowledge distillation Hinton et al. (2015) in order to retain knowledge from
previous tasks, and Elastic Weight Consolidation (EWC) Kirkpatrick et al.
(2017). The EWC regularization term discourages big changes of parameters that
were important for previous tasks, where importance is approximated using the
Fisher information matrix.

We apply three types of replay approaches that allow access to a memory of
past samples. Experience Replay (ER) Chaudhry et al. (2019b) is the most
straightforward approach, as it samples training data from both the current
task and memory at each training step. Average Gradient Episodic Memory
(A-GEM) Lopez-Paz and Ranzato (2017); Chaudhry et al. (2019a) utilizes the
memory of past data to ensure that gradient updates on past and new data are
aligned.

We also experiment with a baseline Pseudo-Replay method for the Question Types
setting. Instead of storing raw data from previous tasks, we use a data
augmentation method, inspired by Kafle et al. (2017); Kil et al. (2021). When
training on task $t$, we augment the data $D_{t}$ by retrieving past questions
based on their shared detected objects classes. For example, if an elephant is
detected on the current picture, we retrieve a past question about an
elephant. We then use the previous model $f_{\theta_{t-1}}$ to generate a
distribution $\tilde{\bm{y}}=f_{\theta_{t-1}}(\tilde{\bm{x}})$ which serves as
soft targets for the new sample $\tilde{\bm{x}}$. By not storing the original
answers, we address privacy and efficiency concerns of replay approaches Van
de Ven and Tolias (2018); Delange et al. (2021).

###  4.3 Evaluation Metrics

After training on task $t$, we compute the VQA accuracy $A_{t,i}$ on data from
the previous task $i$. We report the macro-average accuracy at the end of the
training sequence: $\mathrm{A}=\frac{1}{T}\sum_{i=1}^{T}A_{T,i}$. Following
Riemer et al. (2019), we report the learned accuracy
$\mathrm{LA}=\frac{1}{T}\sum_{i=1}^{T}A_{i,i}$, which measures the ability to
learn the new task $i$. We also compute backward transfer
$\mathrm{BWT}=\frac{1}{T-1}\sum_{i=1}^{T-1}A_{T,i}-A_{i,i}$ Lopez-Paz and
Ranzato (2017), that captures the impact of catastrophic forgetting.

In addition, we introduce a new metric, we term semantic backward transfer
(SBWT), that weights backward transfer with the cosine distance of the
predicted answer embeddings. The motivation for this metric is simply that
some incorrect answers are worse than others. Consider the example in Figure
1, where the ground truth is ‘duck’. After training on subsequent tasks, the
sample gets misclassified as ‘seagull’ which might have a milder impact on the
downstream application than completely unsuited answers such as ‘black and
white’ or ‘one’. More detailed examples are provided in the Appendix Table 10.

For each sample $j=1\dots,N$ of task $i$, we measure the accuracy difference
$\Delta^{Ti}_{j}$ of the answers predicted by the $T$-th and $i$-th models and
weigh it by cosine distance of the two answer embeddings $\bm{e_{Tj}}$ and
$\bm{e_{ij}}$. The final SBWT is computed as :

$\mathrm{SBWT}=\frac{1}{T-1}\sum_{i=1}^{T-1}S_{T,i}$ (1)

where $S_{T,i}$ is the average weighted accuracy difference for task $i$:

$S_{T,i}=\frac{1}{N}\sum_{j=1}^{N}(1-\mathrm{cos}(\bm{e_{Tj}},\bm{e_{ij}}))\cdot\Delta^{Ti}_{j}$
(2)

In our implementation, we use averaged 300-dimensional GloVE embeddings
Pennington et al. (2014), since most answers are single words.

|  | w/o Pretraining | w/ Pretraining
---|---|---|---
Split | Method | Accuracy | LA | BWT | SBWT | Accuracy | LA | BWT | SBWT
Diverse | Fixed Model | 41.60 $\pm$ 0.84 | - | - | - | 57.38 $\pm$ 0.83 | - | - | -
Finetuning | 49.64 $\pm$ 0.78 | 56.69 $\pm$ 0.28 | -8.80 $\pm$ 0.89 | –5.35 $\pm$ 0.61 | 64.59 $\pm$ 0.56 |  67.77 $\pm$ 0.22 | -3.97 $\pm$ 0.59 | -1.93 $\pm$ 0.39
LwF | 50.70 $\pm$ 0.56 | 54.67 $\pm$ 0.42 | -4.96 $\pm$ 0.29 | -2.89 $\pm$ 0.17 | 65.23 $\pm$ 0.42 | 67.62 $\pm$ 0.25 | -3.02 $\pm$ 0.44 | -1.50 $\pm$ 0.28
AGEM | 51.56 $\pm$ 0.78 |  56.72 $\pm$ 0.30 | -6.45 $\pm$ 0.87 | -3.84 $\pm$ 0.60 | 65.65 $\pm$ 0.85 | 67.72 $\pm$ 0.30 | -2.60 $\pm$ 0.71 | -1.22 $\pm$ 0.38
EWC | 52.05 $\pm$ 0.30 | 56.49 $\pm$ 0.22 | -5.55 $\pm$ 0.60 | -3.12 $\pm$ 0.40 | 66.26 $\pm$ 0.55 | 67.58 $\pm$ 0.27 | -1.65 $\pm$ 0.45 | -0.67 $\pm$ 0.29
ER |  54.36 $\pm$ 0.33 | 56.31 $\pm$ 0.51 |  -2.45 $\pm$ 0.49 |  -1.42 $\pm$ 0.26 |  66.66 $\pm$ 0.50 | 67.55 $\pm$ 0.23 |  -1.11 $\pm$ 0.41 |  -0.51 $\pm$ 0.27
Joint | 60.41 $\pm$ 0.03 | - | - | - | 69.76 $\pm$ 0.18 | - | - | -
Taxonomy | Fixed Model | 39.96 $\pm$ 1.05 | - | - | - | 55.00 $\pm$ 0.95 | - | - | -
Finetuning | 47.72 $\pm$ 0.72 | 57.75 $\pm$ 0.24 | -12.53 $\pm$ 0.65 | -8.45 $\pm$ 0.38 | 63.65 $\pm$ 0.63 | 68.77 $\pm$ 0.12 | -6.40 $\pm$ 0.67 | -3.89 $\pm$ 0.53
LwF | 48.05 $\pm$ 0.24 | 55.25 $\pm$ 0.27 | -9.00 $\pm$ 0.38 | -6.13 $\pm$ 0.44 | 64.83 $\pm$ 0.50 | 68.73 $\pm$ 0.17 | -4.88 $\pm$ 0.69 | -2.88 $\pm$ 0.43
AGEM | 50.51 $\pm$ 0.66 |  57.80 $\pm$ 0.25 | -9.10 $\pm$ 0.79 | -5.77 $\pm$ 0.55 | 66.52 $\pm$ 0.34 |  68.86 $\pm$ 0.12 | -2.92 $\pm$ 0.50 | -1.63 $\pm$ 0.33
EWC | 52.17 $\pm$ 0.54 | 57.49 $\pm$ 0.19 | -6.65 $\pm$ 0.44 | -4.33 $\pm$ 0.28 |  67.70 $\pm$ 0.29 | 68.57 $\pm$ 0.16 |  -1.09 $\pm$ 0.33 |  -0.62 $\pm$ 0.19
ER |  54.60 $\pm$ 0.14 | 57.67 $\pm$ 0.28 |  -3.84 $\pm$ 0.42 |  -2.38 $\pm$ 0.27 | 66.76 $\pm$ 0.16 | 68.61 $\pm$ 0.13 | -2.32 $\pm$ 0.16 | -1.22 $\pm$ 0.10
Joint | 60.82 $\pm$ 0.02 | - | - | - | 70.08 $\pm$ 0.18 | - | - | -
Questions | Fixed Model | 18.81 $\pm$ 5.90 | - | - | - | 25.54 $\pm$ 8.75 | - | - | -
Finetuning | 23.30 $\pm$ 8.83 | 65.24 $\pm$ 0.42 | -52.42 $\pm$ 10.88 | -39.86 $\pm$ 12.08 | 48.81 $\pm$ 5.56 | 72.94 $\pm$ 0.20 | -30.17 $\pm$ 7.07 | -22.43 $\pm$ 7.02
LwF | 26.23 $\pm$ 8.56 | 60.69 $\pm$ 1.43 | -43.08 $\pm$ 11.22 | -34.32 $\pm$ 9.94 | 46.61 $\pm$ 3.95 | 72.06 $\pm$ 0.44 | -31.82 $\pm$ 5.42 | -25.13 $\pm$ 5.35
AGEM | 50.73 $\pm$ 1.92 |  65.38 $\pm$ 0.56 | -18.31 $\pm$ 3.04 | -10.02 $\pm$ 1.39 | 68.30 $\pm$ 0.74 | 72.96 $\pm$ 0.24 | -5.83 $\pm$ 1.08 | -2.95 $\pm$ 0.63
EWC | 36.77 $\pm$ 5.01 | 49.05 $\pm$ 3.82 | -15.35 $\pm$ 5.85 | -11.76 $\pm$ 5.41 | 66.77 $\pm$ 3.54 | 70.03 $\pm$ 1.03 |  -4.08 $\pm$ 3.58 | -2.62 $\pm$ 2.28
ER |  59.54 $\pm$ 0.32 | 65.09 $\pm$ 0.52 |  -6.93 $\pm$ 0.71 |  -3.50 $\pm$ 0.35 |  69.18 $\pm$ 0.38 | 72.82 $\pm$ 0.22 | -4.56 $\pm$ 0.56 |  -1.82 $\pm$ 0.34
Joint | 66.35 $\pm$ 0.24 | - | - | - | 72.54 $\pm$ 0.15 | - | - | -

Table 2:  Results from VQA Incremental Learning. We report the average and
standard deviation over five random task orders. LA: Learned Accuracy, BWT:
Backward Transfer, SBWT: Semantic Backward Transfer.

###  4.4 Experimental Setup

We follow a single head setting to allow for task-agnostic inference but
assume knowledge of task boundaries during training. Unless stated otherwise,
memory-based approaches store 500 randomly selected samples per past task. For
further implementation details, please refer to Appendix B.

We consider two baselines: The Fix Model baseline represents the
generalization ability of the model across all tasks after being trained on
only the first task $D_{1}$. The vanilla Finetuning baseline represents the
performance degradation if no measures are taken to prevent forgetting. We
also report the performance of joint training on all the data simultaneously
(Joint) as an upper bound.

##  5 Results

###  5.1 Main Results

##### Task Settings.

Table 2 summarizes the results averaged over five task orders using the UNITER
backbone. The results show an increasing difficulty for the three incremental
learning task definitions, i.e. Diversity Domains $<$ Taxonomy Domains $<$
Question Types, which we will further investigate in Section 6.

Although Question Types has the highest Joint accuracy, naive finetuning shows
poor performance: it has the lowest final accuracy and large negative BWT. The
low Fixed Model accuracy suggests that tasks are dissimilar as a model trained
on a single task fails to generalize.

##### Pretraining.

Our results also confirm that pretraining leads to models that are more robust
to forgetting Mehta et al. (2021): all metrics consistently improve starting
from a pretrained model. Pretraining combined with naive finetuning achieves
on average 58$\%$ relative accuracy improvement over finetuning a model from
scratch. Interestingly, the pretrained Fixed Model is able to generalize
reasonably well to other domains for both image-based settings, and the final
Pretraining+Finetuning accuracy exceeds the Joint accuracy without
pretraining. These results indicate that learning generic V+L representations
via pretraining has persistent benefits. However, pretraining is insufficient
for ensuring continual learning and additional strategies improve the final
accuracy by 8.83% on average.

##### Continual Learning Methods.

Among continual learning methods, LwF offers the smallest gains in terms of
final accuracy and forgetting. For pretrained models in Question Types, it
fails to improve the final accuracy. This can be attributed to the pseudo-
labels generated using the current data becoming too noisy when the answers
for the current and previous tasks differ substantially.

Pretraining+EWC achieves the highest accuracy in the Taxonomy Domains.
However, when dealing with heterogeneous tasks (i.e. within Question Types)
the high regularization weights, which are required to prevent forgetting,
limit the model’s ability to adapt to new tasks. This is reflected in the low
LA of EWC, indicating that the model struggles to learn new tasks. On the
other hand, memory-based approaches have consistently high LA. AGEM performs
reasonably well across settings, but is always outperformed by the
straightforward ER, which shows the best performance with models trained from
scratch and for the challenging setting of Question Types.

##### Measuring Forgetting.

We compare the SBWT metric, which takes semantic similarities into account, to
the standard BWT, which measures absolute forgetting. We observe some notable
differences, which indicate that SBWT favors strong models that forget
gradually.

For instance, EWC w/o pretraining shows lower performance and LA under the
Question Types setting compared to, e.g. AGEM w/o pretraining. However, it
receives a better BWT score. We make similar observations for LwF vs. AGEM in
Taxonomy Domains w/o pretraining, and EWC vs. ER in Taxonomy Domains with
pretraining.

Appendix 10 provides a qualitative analysis with examples from the validation
set. In addition, Table 10 in the Appendix provides an example-based analysis
of our suggested metric, showing that semantically similar answers have higher
SBWT scores.

###  5.2 Experience Replay Ablation

Figure 2: Average accuracy of seen tasks per memory size. Pseudo-Replay
performs competitively up the the third task despite only storing questions.

The above strong performance of the straightforward replay methods suggests
that more advanced strategies for selecting or generating samples
representative of past tasks can yield further improvements. One promising
avenue is to make Experience Relay more efficient. In general, more memory
means less forgetting but at a higher computation and storage cost. We
experiment with a more efficient Pseudo-Replay method which only stores past
questions. Figure 2 shows the average accuracy across training for three
memory sizes. At each step, we compute the average accuracy of the experienced
tasks up to that point. As expected, both methods benefit from access to a
larger memory. Pseudo-Replay shows comparable performance for up to three
tasks, while raw ER replay becomes more advantageous as more tasks are added.
We attribute this convergence in performance to errors accumulated by pseudo-
labeling Tarvainen and Valpola (2017). Despite this limitation, Pseudo-Replay
exceeds the performance of naive finetuning by $18\%$ when storing only 500
samples per task and without requiring access to any past images.

##  6 Task Similarity and Forgetting

###  6.1 Pairwise Task Characterization

To gain further insight into which factors contribute to forgetting, we
measure the correlation between pairwise accuracy drop and task similarity. In
the more widely studied task-incremental learning for image classification,
task similarity refers to the semantic similarity of the old and new classes
Ramasesh et al. (2021). Here, we consider the similarity of the answer
distributions, as well as the image, question and the joint pair
representations.

Diverse Domains
---
Task 1 Task 2 | Group 1 | Group 2 | Group 3 | Group 4 | Group 5
Group 1 | - | -6.58 | -5.21 | -4.84 | -7.09
Group 2 | -4.55 | - | -5.61 | -4.51 | -4.99
Group 3 | -4.64 | -8.39 | - | -7.37 | -11.66
Group 4 | -4.69 | -7.10 | -7.40 | - | -9.63
Group 5 | -4.29 | -5.82 | -6.09 | -3.80 | -
Taxonomy Domains
Task 1 Task 2 | Animals | Food | Interior | Sports | Transport
Animals | - | -8.06 | -3.63 | -5.84 | -4.35
Food | -16.38 | - | -4.29 | -17.08 | -11.94
Interior | -5.75 | -5.19 | - | -7.63 | -2.83
Sports | -11.63 | -18.20 | -9.60 | - | -9.47
Transport | -4.19 | -8.48 | -2.62 | -3.67 | -
Question Types
Task 1 Task 2 | Action | Color | Count | Scene | Subcat.
Action | - | -68.40 | -90.45 | -19.59 | -12.58
Color | -88.89 | - | -99.65 | -27.75 | -62.46
Count | -99.17 | -99.68 | - | -97.52 | -87.00
Scene | -10.91 | -34.40 | -77.73 | - | -15.22
Subcat. | -31.73 | -85.45 | -96.15 | -30.55 | -

Table 3: Task difficulty measured by forgetting in pairwise tasks. Non-
diagonal elements show relative accuracy drop (%) after finetuning on Task 2.

##### Experimental Setup.

We first look into pairwise task relationships following studies in transfer
Zamir et al. (2018) and multitask learning Standley et al. (2020); Lu et al.
(2020). In particular, we measure the extent to which each task is forgotten
after training on a second task. We finetune a pretrained model on Task
$T_{1}$ and compute the accuracy $A_{11}$ on its test set. Then, we finetune
this model on another Task $T_{2}$ and compute the new accuracy $A_{12}$ on
the test set of $T_{1}$. Forgetting is measured as the relative accuracy drop:
$(A_{12}-A_{11})/A_{11}$. Given the varying dataset sizes, we finetune on
$T_{2}$ for a fixed number of 400 steps using a batch size of 512 and learning
rate 5e-5.

Next, we compute the Spearman correlation between the relative accuracy drops
and different factors of task dissimilarity. Here, we consider the answer
distributions , as well as average embeddings of the image, question and the
joint pair. Consider $P$, $Q$ the answer distributions of Tasks $T_{1},T_{2}$
respectively. Since some answers of $T_{1}$ do not appear in $T_{2}$, we
measure the skew divergence Lee (2001) between $P$ and $Q$ as the KL
divergence between $P$ and a mixture distribution $(1-\alpha)P+\alpha Q$ with
$\alpha=0.99$ Ruder and Plank (2017). For the input embeddings, we measure the
cosine distance between the average task representation. As image
representations, we utilize Faster R-CNN features from Anderson et al. (2018),
while questions are embedded using Sentence-BERT. Joint embeddings for image-
question pairs are obtained using the final layer representation of the [CLS]
token of UNITER 222[CLS] is the first token of the input sequence which
aggregates multimodal information. Its representation from the final encoder
layer is passed to the classifier to predict an answer.. The detailed
similarity measures are shown in the Appendix Figure 10.

##### Results.

Table 3 shows the relative accuracy drop for all task pairs. Overall, we
observe that each setting has a distinct pattern. Question Types is evidently
a more challenging setting, where several task combinations show more than
$90\%$ drop. When comparing the visual settings, forgetting in Diverse Domains
fluctuates less depending on the task pairing. This suggests that the task
relationships in Taxonomy Domains might play a more important factor. Although
some relations make sense based on the expected similarity of the visual
scenes, e.g., low forgetting between Food and Interior, others are less
intuitive, e.g., low forgetting between Transport and Interior. Moreover,
certain second tasks seem to consistently affect the amount of forgetting
after finetuning on them. Based on the total number of classes per task as
shown in Table 1, we notice that the model is more robust against forgetting
when Task $T_{2}$ has a wide range of possible answers (e.g., Interior); while
$T_{2}$ with a narrow answer set (e.g., Food, Color, Count) lead to maximum
forgetting.

Dissimilarity | Diverse | Taxonomy | Questions
---|---|---|---
Factor | Domains | Domains | Types
Answer distribution | 0.567* | 0.791* | 0.795*
Image embedding | 0.248 | 0.492* | -0.640*
Question embedding | 0.184 | 0.531* | 0.631*
Joint embedding | 0.220 | 0.622* | -0.223

Table 4: Spearman correlation of pairwise performance drop and task
dissimilarity (* where $p<0.05$).

The correlation results in Table 4 indicate that the more similar two
consecutive tasks are, the less forgetting occurs. The divergence of answer
distributions consistently correlates with forgetting, but does not fully
account for the performance drop. For example, the divergence of Interior from
Animals and Sports answer distributions is the same, however Sports leads to
1.88$\%$ more forgetting. Regarding the embedding distances, image embeddings
show the highest correlation in Taxonomy Domain, meaning that the more
visually similar two domains are, the less severe forgetting is. We observe
the same relationship mirrored in Question Types for question embeddings. We
find no factor to correlate significantly with Diverse Domains, where tasks
are relatively similar to each other (cf. Appendix 10). Looking across
modalities, question and joint similarities in Taxonomy Domains correlate with
forgetting, showing that the shift of the visual domains results in changes of
the referred objects and types of questions per task.333We notice that the
more similar images of two Question Types tasks are, the more forgetting
occurs. A possible explanation is that new questions for similar images
‘overwrite’ previous knowledge. However, all cosine distances of image
embeddings are too low (¡0.05) to lead to any conclusions.

###  6.2 Sensitivity to Task Order

Figure 3: Sensitivity to task order as illustrated for Question Types. Each
bar shows the accuracy of a task sequence ending with a different task. Figure
4: Comparison of Accuracy and Semantic Backward Transfer with UNITER vs ViLT.

w/o Pretraining
---
Method | What animal | What room | What sport
Finetuning | 33.09 $\pm$ 13.38 | 54.38 $\pm$ 32.42 | 25.14 $\pm$ 32.11
EWC | 48.18 $\pm$ 15.67 | 83.48 $\pm$ 7.61 | 62.81 $\pm$ 13.67
ER | 73.11 $\pm$ 0.70 | 89.04 $\pm$ 2.80 | 87.20 $\pm$ 1.84
w/ Pretraining
Method | What animal | What room | What sport
Finetuning | 75.07 $\pm$ 3.54 | 83.26 $\pm$ 12.47 | 69.92 $\pm$ 14.14
EWC | 81.75 $\pm$ 1.42 | 94.32 $\pm$ 0.88 | 90.82 $\pm$ 1.36
ER | 80.73 $\pm$ 0.37 | 94.10 $\pm$ 1.39 | 90.92 $\pm$ 0.71

Table 5: Accuracy and standard deviation of the best performing models on
different sub-questions in Taxonomy Domains.

Previous work on task-incremental learning for image classification Yoon et
al. (2020) has discussed the impact of task order to final performance,
especially when tasks are dissimilar. Similarly, we observe a high standard
deviation in the Question Types results of Table 2. In order to investiagte
this further, we plot the final accuracy of a pretrained model for five
training sequences in Figure 3, each ending with a different task. Our results
show that task order can lead to Finetuning accuracy that varies more than
15$\%$. Although EWC improves the average accuracy, there is still a 10$\%$
fluctuation depending on the order. However, replay-based methods are able to
improve performance and mitigate the sensitivity to task order.

While Table 2 shows low variance in Taxonomy Domains, we find high variance
when examining the performance on specific questions. In particular, we find
that certain question types, such as Animals, Interior, and Sports, have high
variance. Table 5 reveals a standard deviation which is up to 30 times higher
compared to the average results in Table 2. High standard deviation across
randomized task orders is problematic since models can have different behavior
in practice despite similar (aggregated) performance. In other words, the
current task performance will highly depend on the previous task order, even
though the overall accuracy from the randomized trials appears similar.

##  7 Model Representations

As described in Section 4.1, we compare different input representations of two
single-stream transformer models: UNITER-base Chen et al. (2020), which uses
region features extracted from a frozen pretrained object detector; and ViLT-
base Kim et al. (2021), which directly embeds image patches.

###  7.1 Continual Learning Results

Figure 4 shows the performance of ViLT against UNITER when using naive
finetuning, EWC and ER. The compared continual learning strategies perform
similarly with both backbones. However, ViLT shows more forgetting especially
in the case of question types. Although UNITER’s region-based features are
more robust to forgetting, they rely on a frozen pretrained object detector
model. This could limit the model’s applicability to domains with larger
visual distribution shifts. Future work should focus on developing methods
that perform well with V+L models that take image pixels as inputs.

Figure 5: Representation similarity for the [CLS] token of samples from the
first task after each training task. The first row corresponds to UNITER and
the second row to ViLT representations. The columns from left to right refer
to Diverse Domains, Taxonomy Domains, and Question Types. The layers that are
affected are the final layers but also earlier layers (UNITER layer 4 and ViLT
layer 3).

Figure 6: Representation similarity for the average visual (V) and text (T)
tokens of of samples from the first task. The first row corresponds to UNITER
and the second row to ViLT representations. The columns from left to right
refer to Diverse Domains, Taxonomy Domains, and Question Types. The text
representations from both models show decreased representation similarity for
deeper layers. The visual representations of UNITER follow the same trend,
while ViLT visual representations show a large similarity drop for layer 8-11.

###  7.2 Representation Analysis

Finally, we ask how representations from each modality evolve throughout the
training sequence and compare this evolution across our continual learning
settings. We use centered kernel alignment (CKA) Kornblith et al. (2019) to
track the representation similarity of sequentially finetuned models. We
extract representations $X^{1}_{t}$ of the validation data of the first task
after training for each task $t=1\cdots T$, and measure the CKA similarity of
$X^{1}_{t>1}$ to the original representations $X^{1}_{1}$.

Figure 5 shows the evolution of the representation similarity of the sentence-
level representation [CLS] token per layer. Across the three settings, the
representations of different layers change following a similar pattern but at
different magnitudes which agree with the measured amount of forgetting. Our
results echo previous findings Wu et al. (2022) showing that representations
from deeper layers are more affected during continual learning, but there are
also fragile earlier layers (UNITER layer 4, ViLT layer 3).

Figure 6 shows the evolution of the average visual and text token
representations per layer. The representations of question tokens from both
models retain higher similarity than image and [CLS] tokens. In particular,
ViLT visual representations show a large drop in representation similarity for
layers 8-11. Since ViLT uses image patches instead of features extracted from
a separate vision module, it needs to perform both feature extraction and
multimodal alignment. These results suggest that the features extracted from
the visual inputs for VQA are more task-dependent and highlight the importance
of stabilizing visual representations in deeper layers.

##  8 Conclusion

In this work, we provide an in-depth study of task design for VQA and its
impact when combining pretrained V+L models with continual learning
approaches. We empirically investigate the impact of task formulation, i.e.
task design, order and similarity, by evaluating two transformer-based models
and benchmarking several baseline methods. We also propose SBWT as a new
evaluation metric that utilizes the semantic distance of answers. Our results
show that both task order and similarity, especially from the viewpoint of the
answer distribution, highly influence performance.

These results are important for designing continual learning experiments for
real-world settings that take into account how data become available over
time. For example, the Taxonomy Domains resembles applications where data is
continuously collected in different visual surroundings, whereas Question
Types corresponds to ‘teaching’ the system new reasoning capabilities. Our
results suggest that the latter is the most challenging. Our results also
suggest that the easiest and thus ‘best-case’ scenario is the ‘Diverse’ data
collection setup, where the system incrementally learns to recognize new
objects which are randomly sampled from different domains.

In terms of model architectures, we investigated single-stream backbones with
region features and image patches as inputs. Our representation analysis shows
that image and text representations change at different scales. This implies
that regularization-based approaches might be more suited for models with
separate visual and text parameters where different regularization strengths
are applied to each modality.

## References

  * Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018.  Bottom-up and top-down attention for image captioning and visual question answering.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Bender et al. (2021) Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big? In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, page 610–623, New York, NY, USA. Association for Computing Machinery.
  * Chaudhry et al. (2019a) Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2019a.  Efficient lifelong learning with a-GEM.  In _International Conference on Learning Representations_.
  * Chaudhry et al. (2019b) Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Kumar Dokania, Philip H. S. Torr, and Marc’Aurelio Ranzato. 2019b.  Continual learning with tiny episodic memories. _CoRR_ , abs/1902.10486.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020.  Uniter: Universal image-text representation learning.  In _Computer Vision – ECCV 2020_ , pages 104–120, Cham. Springer International Publishing.
  * Del Chiaro et al. (2020) Riccardo Del Chiaro, Bartł omiej Twardowski, Andrew Bagdanov, and Joost van de Weijer. 2020.  Ratt: Recurrent attention to transient tasks for continual image captioning.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 16736–16748. Curran Associates, Inc.
  * Delange et al. (2021) Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. 2021.  A continual learning survey: Defying forgetting in classification tasks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pages 1–1.
  * Goodfellow et al. (2013) Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2013\.  An empirical investigation of catastrophic forgetting in gradient-based neural networks.  _arXiv preprint arXiv:1312.6211_.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017\.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Greco et al. (2019) Claudio Greco, Barbara Plank, Raquel Fernández, and Raffaella Bernardi. 2019\.  Psycholinguistics meets continual learning: Measuring catastrophic forgetting in visual question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3601–3605, Florence, Italy. Association for Computational Linguistics.
  * Hadsell et al. (2020) Raia Hadsell, Dushyant Rao, Andrei A. Rusu, and Razvan Pascanu. 2020.  Embracing change: Continual learning in deep neural networks.  _Trends in Cognitive Sciences_ , 24(12):1028–1040.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_.
  * Hussain et al. (2021) Aman Hussain, Nithin Holla, Pushkar Mishra, Helen Yannakoudakis, and Ekaterina Shutova. 2021.  Towards a robust experimental framework and benchmark for lifelong language learning.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_.
  * Jin et al. (2020) Xisen Jin, Junyi Du, Arka Sadhu, Ram Nevatia, and Xiang Ren. 2020.  Visually grounded continual learning of compositional phrases.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2018–2029, Online. Association for Computational Linguistics.
  * Kafle et al. (2017) Kushal Kafle, Mohammed Yousefhussien, and Christopher Kanan. 2017.  Data augmentation for visual question answering.  In _Proceedings of the 10th International Conference on Natural Language Generation_ , pages 198–202, Santiago de Compostela, Spain. Association for Computational Linguistics.
  * Kil et al. (2021) Jihyung Kil, Cheng Zhang, Dong Xuan, and Wei-Lun Chao. 2021.  Discovering the unknown knowns: Turning implicit knowledge in the dataset into explicit training examples for visual question answering.  _arXiv preprint arXiv:2109.06122_.
  * Kim et al. (2021) Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 5583–5594. PMLR.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526.
  * Kornblith et al. (2019) Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019.  Similarity of neural network representations revisited.  In _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 3519–3529. PMLR.
  * Lazaridou et al. (2021) Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomáš Kočiský, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2021.  Mind the gap: Assessing temporal generalization in neural language models.  In _Advances in Neural Information Processing Systems_.
  * Lee (2001) Lillian Lee. 2001.  On the effectiveness of the skew divergence for statistical language analysis.  In _Proceedings of the Eighth International Workshop on Artificial Intelligence and Statistics_ , volume R3 of _Proceedings of Machine Learning Research_ , pages 176–183. PMLR.  Reissued by PMLR on 31 March 2021.
  * Lee et al. (2021) Sebastian Lee, Sebastian Goldt, and Andrew Saxe. 2021.  Continual learning in the teacher-student setup: Impact of task similarity.  In _2021 International Conference on Machine Learning_.
  * Lei et al. (2022) Stan Weixian Lei, Difei Gao, Jay Zhangjie Wu, Yuxuan Wang, Wei Liu, Mengmi Zhang, and Mike Zheng Shou. 2022.  Symbolic replay: Scene graph as prompt for continual learning on vqa task.  _arXiv preprint arXiv:2208.12037_.
  * Li and Hoiem (2018) Zhizhong Li and Derek Hoiem. 2018.  Learning without forgetting.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 40(12):2935–2947.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _Computer Vision – ECCV 2014_ , pages 740–755, Cham. Springer International Publishing.
  * Lomonaco and Maltoni (2017) Vincenzo Lomonaco and Davide Maltoni. 2017.  Core50: a new dataset and benchmark for continuous object recognition.  In _Proceedings of the 1st Annual Conference on Robot Learning_ , volume 78 of _Proceedings of Machine Learning Research_ , pages 17–26. PMLR.
  * Lopez-Paz and Ranzato (2017) David Lopez-Paz and Marc’Aurelio Ranzato. 2017.  Gradient episodic memory for continual learning.  In _Advances in Neural Information Processing Systems_ , volume 30, pages 6470–6479.
  * Lu et al. (2020) Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 2020.  12-in-1: Multi-task vision and language representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * McCloskey and Cohen (1989) Michael McCloskey and Neal J Cohen. 1989.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_ , volume 24, pages 109–165. Elsevier.
  * Mehta et al. (2021) Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. 2021.  An empirical investigation of the role of pre-training in lifelong learning.  _arXiv preprint arXiv:2112.09153_.
  * Mi et al. (2020) Fei Mi, Lingjing Kong, Tao Lin, Kaicheng Yu, and Boi Faltings. 2020.  Generalized class incremental learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 240–241.
  * Mirzadeh et al. (2020) Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. 2020.  Understanding the role of training regimes in continual learning.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 7308–7320. Curran Associates, Inc.
  * Nguyen et al. (2019a) Cuong V. Nguyen, Alessandro Achille, Michael Lam, Tal Hassner, Vijay Mahadevan, and Stefano Soatto. 2019a.  Toward understanding catastrophic forgetting in continual learning. _CoRR_ , abs/1908.01091.
  * Nguyen et al. (2019b) Giang Nguyen, Tae Joon Jun, Trung Tran, Tolcha Yalew, and Daeyoung Kim. 2019b.  Contcap: A scalable framework for continual image captioning.  _arXiv preprint arXiv:1909.08745_.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global vectors for word representation.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543.
  * Ramasesh et al. (2021) Vinay Venkatesh Ramasesh, Ethan Dyer, and Maithra Raghu. 2021.  Anatomy of catastrophic forgetting: Hidden representations and task semantics.  In _International Conference on Learning Representations_.
  * Ratcliff (1990) Roger Ratcliff. 1990.  Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. _Psychological review_ , 97(2):285.
  * Rebuffi et al. (2017) Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017.  icarl: Incremental classifier and representation learning.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pages 2001–2010.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Riemer et al. (2019) Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, , and Gerald Tesauro. 2019.  Learning to learn without forgetting by maximizing transfer and minimizing interference.  In _International Conference on Learning Representations_.
  * Ruder and Plank (2017) Sebastian Ruder and Barbara Plank. 2017.  Learning to select data for transfer learning with Bayesian optimization.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 372–382, Copenhagen, Denmark. Association for Computational Linguistics.
  * Srinivasan et al. (2022) Tejas Srinivasan, Ting-Yun Chang, Leticia Leonor Pinto Alva, Georgios Chochlakis, Mohammad Rostami, and Jesse Thomason. 2022.  Climb: A continual learning benchmark for vision-and-language tasks.  _arXiv preprint arXiv:2206.09059_.
  * Standley et al. (2020) Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. 2020.  Which tasks should be learned together in multi-task learning? In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 9120–9132. PMLR.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019.  Energy and policy considerations for deep learning in NLP.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3645–3650, Florence, Italy. Association for Computational Linguistics.
  * Suglia et al. (2020) Alessandro Suglia, Ioannis Konstas, Andrea Vanzo, Emanuele Bastianelli, Desmond Elliott, Stella Frank, and Oliver Lemon. 2020.  CompGuessWhat?!: A multi-task evaluation framework for grounded language learning.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7625–7641, Online. Association for Computational Linguistics.
  * Tarvainen and Valpola (2017) Antti Tarvainen and Harri Valpola. 2017.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NEURIPS’17, page 1195–1204, Red Hook, NY, USA. Curran Associates Inc.
  * Van de Ven and Tolias (2018) Gido M Van de Ven and Andreas S Tolias. 2018.  Generative replay with feedback connections as a general strategy for continual learning.  _arXiv preprint arXiv:1809.10635_.
  * Van de Ven and Tolias (2019) Gido M Van de Ven and Andreas S Tolias. 2019.  Three scenarios for continual learning.  _arXiv preprint arXiv:1904.07734_.
  * Whitehead et al. (2021) Spencer Whitehead, Hui Wu, Heng Ji, Rogerio Feris, and Kate Saenko. 2021.  Separating skills and concepts for novel visual question answering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5632–5641.
  * Wu et al. (2022) Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi, and Gholamreza Haffari. 2022.  Pretrained language model in continual learning: A comparative study.  In _International Conference on Learning Representations_.
  * Yoon et al. (2020) Jaehong Yoon, Saehoon Kim, Eunho Yang, and Sung Ju Hwang. 2020.  Scalable and order-robust continual learning with additive parameter decomposition.  In _International Conference on Learning Representations_.
  * Zamir et al. (2018) Amir R. Zamir, Alexander Sax, William Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. 2018.  Taskonomy: Disentangling task transfer learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.

##  Appendix A Data Details

We investigate three continual learning settings based on the VQA-v2 dataset
Goyal et al. (2017), a collection of visual question annotations in English.
Tasks in the Diverse Domains setting are created by grouping $10$ objects from
COCO annotations Lin et al. (2014) as follows:

  * •

Group 1: bird, car, keyboard, motorcycle, orange, pizza, sink, sports ball,
toilet, zebra

  * •

Group 2: airplane, baseball glove, bed, bus, cow, donut, giraffe, horse,
mouse, sheep

  * •

Group 3: boat, broccoli, hot dog, kite, oven, sandwich, snowboard, surfboard,
tennis racket, TV

  * •

Group 4: apple, baseball bat, bear, bicycle, cake, laptop, microwave, potted
plant, remote, train

  * •

Group 5: banana, carrot, cell phone, chair, couch, elephant, refrigerator,
skateboard, toaster, truck

We also provide a few example questions for each task in Question Types:

  * •

Action: What is the cat doing?, Is the man catching the ball?, What is this
sport?

  * •

Color: What color is the ground?, What color is the right top umbrella?

  * •

Count: How many skaters are there?, How many elephants?, How many rooms do you
see?

  * •

Scene: Is the picture taken inside?, Is this photo black and white?, What is
the weather like?

  * •

Subcategory: What type of vehicle is this?, What utensil is on the plate?,
What kind of car is it?

Figures 7-9 show the distribution of the 20 most common question words and
answers for each task. The counts are computed on the combined train and
validation data, excluding stopwords from the question vocabulary. These plots
support our general findings about the characteristics of each task and the
relationships between them. For example, answers in Diverse Domains are highly
similar across tasks, while the most considerable difference of common answers
is observed in Question Types. In addition, frequent nouns in Diverse and
Taxonomy Domains reflect the typical objects from the image annotations of
each task. Common words in Question Types also follow the definition of each
task. For example, top words in Scene such as ‘sunny’, ‘room’, ‘outside’ refer
to the entire image, while Action words such as ‘sport’, ‘playing’, ‘moving’
refer to activities shown in the image.

Figure 7: Most common words (left) and answers (right) per task Diverse
Domains.

Figure 8: Most common words (left) and answers (right) per task Taxonomy
Domains.

Figure 9: Most common words (left) and answers (right) per task in Question
Types.

Dissimilarity | Diverse | Taxonomy | Questions
---|---|---|---
Answers |  0.567 (0.009) |  0.791 (0.000) |  0.795 (0.000)
Image embed. | 0.248 (0.293) |  0.492 (0.028) |  -0.640 (0.002))
Question embed. | 0.184 (0.437) |  0.531 (0.016) |  0.631 (0.003)
Joint embed. | 0.220 (0.350) |  0.622 (0.003) | -0.223 (0.344)

Table 6: Spearman correlation of pairwise performance drop and and different
dissimilarity heuristics. In addition to the results in Table 4, we show in
parentheses the corresponding p-values. We underline statistically significant
results (p < 0.05).

(a) Divergence of answer distributions.

(b) Cosine distance of image embeddings.

(c) Cosine distance of question embeddings.

(d) Cosine distance of joint embeddings.

Figure 10: Dissimilarity measures between task pairs.

##  Appendix B Implementation Details

Our implementation is based on the publicly available PyTorch codebase of
UNITER (https://github.com/ChenRocks/UNITER). For the continual learning
experiments, we train a UNITER-base model (86M parameters) on a cluster of
NVIDIA V100 GPUs using a single node with 4 GPUs. Training on a sequence of 5
tasks requires on average $\sim$ 5 GPU hours. The main experiments (Table 2)
require approximately a total of 200 GPU hours.

We first tune the batch size and learning rate with naive finetuning.
Following previous work on finetuning V+L models, we downscale the learning
rate of the pretrained backbone by 10x. Keeping these hyperparameters fixed,
we then tune the continual learning hyperparameters (EWC, LwF $\lambda$). All
hyperparameters are selected through grid search based on the maximum final
accuracy as shown in Table 7. Initial results with a pretrained model on
Taxonomy Domains showed that best performance is achieved with a mixing ratio
of 3:1 of new and old data per batch. We keep this ratio constant for all
experiments.

| Setting | Batch Size | Learning Rate | LwF $\lambda$ | EWC $\lambda$
---|---|---|---|---|---
UNITER | Diverse | 512 | 8e-5 | 1 | 400
Diverse+PT | 1024 | 8e-5 | 0.7 | 500
Taxonomy | 512 | 8e-5 | 1 | 600
Taxonomy+PT | 1024 | 5e-5 | 0.5 | 500
Questions | 1024 | 1e-4 | 0.9 | 50K
Questions+PT | 512 | 5e-5 | 0.4 | 20K
ViLT | Diverse+PT | 1024 | 1e-5 | - | 500
Taxnomy+PT | 1024 | 1e-5 | - | 700
Questions+PT | 512 | 8e-5 | - | 10K

Table 7: Best hyperparameters for all settings. PT: Initialization from
pretrained checkpoint.

Each experiment is repeated five times with a different random seed and task
order. The task orders used in our experiments are the following:

  * •

Diverse Domains

  * •

group 5, group 3, group 2, group 4, group 1

  * •

group 1, group 2, group 5, group 3, group 4

  * •

group 4, group 3, group 5, group 1, group 2

  * •

group 3, group 1, group 4, group 2, group 5

  * •

group 2, group 5, group 1, group 4, group 3

  * •

Taxonomy Domains

  * •

food, animals, sports, interior, transport

  * •

transport, sports, food, animals, interior

  * •

interior, animals, food, transport, sports

  * •

animals, food, interior, sports, transport

  * •

sports, interior, transport, animals, food

  * •

Question types

  * •

action, count, subcategory, scene, color

  * •

color, subcategory, action, count, scene

  * •

scene, count, action, color, subcategory

  * •

subcategory, color, scene, action, count

  * •

count, scene, color, subcategory, action

##  Appendix C ViLT Results

Table 8 shows the detailed results for ViLT across the three settings.

Split | Method | Accuracy | LA | BWT | SBWT
---|---|---|---|---|---
Diverse | Fixed Model | 51.64 $\pm$ 3.09 | - | - | -
Finetuning | 61.07 $\pm$ 0.41 | 65.03 $\pm$ 1.06 | -5.01 $\pm$ 1.02 | -2.80 $\pm$ 0.68
EWC | 61.80 $\pm$ 0.96 | 63.64 $\pm$ 1.33 | -2.30 $\pm$ 0.62 | -1.14 $\pm$ 0.38
ER | 64.22 $\pm$ 0.10 | 64.74 $\pm$ 0.84 | -0.98 $\pm$ 0.52 | -0.25 $\pm$ 0.71
| Joint | 67.51 $\pm$ 0.10 | - | - | -
Taxonomy | Fixed Model | 50.74 $\pm$ 1.09 | - | - | -
Finetuning | 61.25 $\pm$ 0.50 | 66.51 $\pm$ 0.27 | -6.57 $\pm$ 0.75 | -4.09 $\pm$ 0.41
EWC | 63.69 $\pm$ 0.46 | 64.86 $\pm$ 0.29 | -1.46 $\pm$ 0.40 | -0.92 $\pm$ 0.24
ER | 63.52 $\pm$ 0.20 | 65.59 $\pm$ 0.22 | -2.59 $\pm$ 0.30 | -1.46 $\pm$ 0.20
| Joint | 67.84 $\pm$ 0.09 | - | - | -
Questions | Fixed Model | 23.84 $\pm$ 8.20 | - | - | -
Finetuning | 36.95 $\pm$ 11.09 | 71.06 $\pm$ 0.11 | -42.64 $\pm$ 13.93 | -32.86 $\pm$ 14.25
EWC | 60.25 $\pm$ 2.86 | 68.60 $\pm$ 0.33 | -10.45 $\pm$ 3.52 | -8.19 $\pm$ 2.81
ER | 65.61 $\pm$ 0.76 | 70.77 $\pm$ 0.18 | -6.45 $\pm$ 1.17 | -2.86 $\pm$ 0.62
| Joint | 72.41 $\pm$ 0.12 | - | - | -

Table 8:  ViLT Results from VQA Incremental Learning. We report the average
and standard deviation over five random task orders. LA: Learned Accuracy,
BWT: Backward Transfer, SBWT: Semantic Backward Transfer.

##  Appendix D Qualitative Results

Table 9 shows examples of predicted answers with different approaches. The two
top examples are from two different task orders in Question Types, and the two
bottom examples are from Taxonomy Domains. The model trained from scratch
(column w/o PT) fails to retain knowledge from the corresponding training
task. The pretrained model (column PT) is more resistant to forgetting and we
observe that for the first and third images, it even manages to recover the
correct answer during the training sequence. However, relying only on
pretraining is insufficient, as the model still tends to change the predicted
answer based on the most recent training task. Both EWC and ER combined with
pretraining successfully retain previous knowledge.

|  | What is the horse doing?
---
Task | w/o PT | PT | PT+EWC | PT+ER
Action | jumping | jumping | jumping | jumping
Count | two | one | jumping | jumping
Subcat. | riding | jump | jumping | jumping
Scene | cold | jumping | jumping | jumping
Color | black | black | jumping | jumping

|  | What color is the cow?
---
Task | w/o PT | PT | PT+EWC | PT+ER
Color | black | black | black | black
Subcat | black | black | black | black
Action | zero | yes | cow | black
Count | one | one | black | black
Scene | green | green | black | black

|  | What is orange?
---
Task | w/o PT | PT | PT+EWC | PT+ER
Food | carrots | carrots | carrots | carrots
Animals | birds | carrots | carrots | carrots
Sports | nothing | kites | carrots | carrots
Interior | chair | carrots | carrots | carrots
Transport | nothing | tomato | carrots | carrots

|  | What type of bird is this?
---
Task | w/o PT | PT | PT+EWC | PT+ER
Interior | dog | owl | owl | owl
Animals | pigeon | pigeon | pigeon | pigeon
Food | turkey | pigeon | pigeon | pigeon
Transport | not sure | duck | pigeon | seagull
Sports | zero | seagull | pigeon | seagull

Table 9: Examples of the evolution of predicted answers with different
approaches combined with UNITER. Column Task shows the order of the training
tasks. The bold task corresponds to the task of the sample.

Reference | Compared Answer 1 | Compared Answer 2
---|---|---
Answer | Acc | Answer | Acc | SBWT | Answer | Acc | SBWT
skateboarding | 1 | skateboard | 0 | -0.164 | black | 0 | -0.836
snowboarding | 1 | skiing | 0 | -0.134 | winter | 0 | -0.529
breakfast | 1 | sandwich | 0 | -0.340 | one | 0 | -0.855
food | 1 | meat | 0 | -0.320 | toothbrush | 0 | -0.832
skateboarding | 1 | skateboard | 0.3 | -0.115 | skateboard | 0 | -0.164
carrots | 1 | carrot | 0.3 | -0.093 | three | 0 | -0.818
sheep | 1 | goat | 0.3 | -0.197 | white | 0 | -0.676
cloudy | 1 | overcast | 0.3 | -0.151 | gray | 0 | -0.577
black | 0 | black and white | 1 | 0.136 | brown | 1 | 0.269

Table 10: Comparison of the SBWT metric of two answers with respect to the
same reference answer. We verify that semantically more similar answers have
higher SBWT.

Table 10 presents examples of the SBWT metric. Specifically, it compares SBWT
for two pairs of predicted answers with the same initial reference answer.
When the initial prediction (reference answer) is correct, and both compared
answers are wrong, we observe that SBWT penalizes similar answers less than
unrelated ones (see the first four rows of Table 10). Similarly, when one of
the compared answers is partially correct (rows 5-8) according to the VQA
accuracy metric, SBWT is less punishing compared to BWT, which in our examples
would be $-0.7$. Finally, the last row shows an example of corrected compared
answers, where the accuracy improvement is weighted with the semantic distance
of reference and compared answers.