# 깊은 시각화를 이용한 파괴적 망각 극복

뉴엔  뉴엔  阮  Nguyen

# Overcoming Catastrophic Forgetting by
Deep Visualization

뉴엔  뉴엔  阮  Nguyen

###### Abstract

Explaining the behaviors of deep neural networks, usually considered as black
boxes, is critical especially when they are now being adopted over diverse
aspects of human life. Taking the advantages of interpretable machine learning
(interpretable ML), this work proposes a novel tool called Catastrophic
Forgetting Dissector (or CFD) to explain catastrophic forgetting in continual
learning settings. We also introduce a new method called Critical Freezing
based on the observations of our tool. Experiments on ResNet articulate how
catastrophic forgetting happens, particularly showing which components of this
famous network are forgetting. Our new continual learning algorithm defeats
various recent techniques by a significant margin, proving the capability of
the investigation. Critical freezing not only attacks catastrophic forgetting
but also exposes explainability.

반 지앙 반 지앙 文 江 Van Giang [major]김 대 영Daeyoung Kimsigned [major2]김 대 영Daeyoung
Kimsigned Professor of School of Computing CSengineeringa 20184658 [1]김 대 영
[2]오 혜 연 [3]박 노 성 2020611 2020611 2020

Image captioning, Continual learning, Catastrophic forgetting, Interpretable
ML

###### Contents

  1. 1 Introduction
  2. 2 Background and Related Work
    1. 2.1 Image Captioning
    2. 2.2 Interpretable Machine Learning
      1. 2.2.1 Network Dissection
      2. 2.2.2 Feature Visualization
    3. 2.3 Continual Learning
  3. 3 Approach
    1. 3.1 Partially Freezing for Image Captioning
    2. 3.2 Pseudo-labeling (Learning without Forgeting - LwF)
    3. 3.3 Knowledge Distillation
    4. 3.4 Catastrophic Forgetting Dissector - CFD
    5. 3.5 Critical Freezing
    6. 3.6 Implementation Details
      1. 3.6.1 Feature Extraction
      2. 3.6.2 Language Model
      3. 3.6.3 Training Details
  4. 4 Evaluation
    1. 4.1 Dataset
    2. 4.2 Evaluation Metric
    3. 4.3 Addition of Multiple Classes Sequentially
    4. 4.4 Conclusion and Future Work

###### List of Tables

  1. 4.1 CFD’s result on various instances.
  2. 4.2 Performance when 5 classes arrive sequentially on past tasks and newly added tasks.

###### List of Figures

  1. 1.1 Catastrophic forgetting in continual learning setting.
  2. 1.2 A scalable framework for continual image captioning.
  3. 2.1 A synthetic caption is generated by deep learning.
  4. 2.2 Interpretable ML helps open the black-box of deep learning.
  5. 2.3 Network dissection algorithm workflow.
  6. 2.4 Abstract features in hidden layers of neural networks.
  7. 2.5 PDA highlights pixels that are evidence for (red) and against (blue) a prediction of cat.
  8. 3.1 Freezing a component of the fine-tuned model while the remaining component is trainable.
  9. 3.2 Segmentation ground truth as input of CFD.
  10. 3.3 CFD working on ResNet-50 that includes 5 convolutional blocks.
  11. 3.4 Choosing representatives based on overlap between feature maps and ground truth.
  12. 3.5 (a) IoU value between the segmentation of a train and the positive features. (b) The IoU on the positive features of two representative maps. Red is evidence and blue is against. (c) IoU graph and computer vision of a bird image. Yellow-bounding and blue-bounding boxes are from the old and new model respectively.
  13. 3.6 Choosing representatives based on computer visions of two models.
  14. 4.1 Two-head network for dissecting catastrophic forgetting.
  15. 4.2 Adding 5 classes one by one scenario.
  16. 4.3 Forgetting in feature extractor when using fine-tuning.
  17. 4.4 Feature maps from convolutional blocks of ResNet-50.
  18. 4.5 (a) $IoU_{M,GT}$ of $M_{19}$, $M_{20}$ and $M_{24}$ comparing with GT. (b) $IoU_{MT,MO}$ of model $M_{20}$ and $M_{24}$ comparing with $M_{19}$.
  19. 4.6 Qualitative results of continual learning algorithms.

##  Chapter 1 Introduction

Regarding human evolution, life-long learning has been considered as one of
the most crucial abilities, helping us develop more complicated skills
throughout the lifetime. The idea of this learning strategy is hence deployed
extensively by the deep learning community. Life-long learning (or continual
learning) enables machine learning models to perceive new knowledge while
simultaneously exposing backward-forward transfer, non-forgetting, or few-shot
learning [1]. While the aforementioned properties are the ultimate goals for
life-long learning systems, catastrophic forgetting or semantic drift
naturally occurs in deep neural networks in life-long learning settings
because they are vastly optimized upon gradient descent algorithm [2].

Catastrophic forgetting is defined as when we use a trained model on a given
domain to address a new task. Due to adapting to the new data samples, the
model forgets what it learned before on the old domain. As shown in Fig. 1.1,
if we use fine-tuning for continual learning, the knowledge acquired from the
previous task will be eradicated and the inference fails miserably. In fact,
the description of the cat photo is shifted from “a cat laying on a bed next
to a blanket” to “a person is standing in an orange room”.

Figure 1.1: Catastrophic forgetting in continual learning setting.

Although catastrophic forgetting is tough and undesirable, research on
understanding this problem is rare amongst the deep learning community. The
interest in understanding or measuring catastrophic forgetting does not
commensurate with the number of research to deal with this problem. Kemker et
al. [3] develop new metrics to help compare continual learning techniques
fairly and directly. Nguyen et al. [4] study which properties cause the
hardness for the learning process. By modeling the chosen properties using
task space, they can estimate how much a model forgets in a sequential
learning scenario, shedding light on factors affecting the error rate on a
task sequence. They can not show us what is being forgotten or which
components are forgetting inside the model, but revealing what properties of
tasks trigger catastrophic forgetting. By comparison, our work focuses on
understanding which components of a network are volatile corresponding to a
given sequence of tasks to articulate catastrophic forgetting.

This research, in particular, introduces a novel approach to elaborate
catastrophic forgetting by visualizing hidden layers in class-incremental
learning (considered as the hardest scenario of continual learning). In this
learning paradigm, the use of previous data is prohibited and instances of the
incoming tasks are unseen. We develop a tool named Catastrophic Forgetting
Dissector (or CFD) which automates the dissection of catastrophic forgetting,
exactly pointing out which components, in a model, are causing the forgetting.
We formally adopt Intersection over Union (IoU), a popular evaluation metric
in detection and segmentation tasks which essentially computes the overlapping
ratio between two frames, to measure the forgetting degree of deep neural
networks in this work. The degree of forgetting is objectively measured after
each class is added, thus giving us an intuition of how forgetting happens on
a given part of the network.

To satisfy continual learning settings in image captioning we created a
dataset named Split MS-COCO (a derivative of MS-COCO 2014) in which each task
will come with a new class. For example, with an old task A of classes cat,
dog, and table; the new tasks B will contain samples of class person only.
When training task B, knowledge from task A is leveraged to help the current
model generalize well on all 4 classes: cat, dog, table, and person. We define
terminology “clear image” to represent an image containing only one class out
of 80 classes defined by MS-COCO in its reference captions to manipulate
incremental steps correctly. Split MS-COCO enables us to comprehensively
measure the forgetting on deep neural networks. This dataset is particularized
in Chapter. 4.

We also propose a novel framework (as in Fig. 1.2) that is a scalable
framework to combine encoder-decoder image captioning architecture with
continual learning. The scalability of the framework is maintained as further
continual learning techniques and more sophisticated encoder-decoder
architectures can be smoothly integrated to increase the performance. Fine-
tuning is considered as the baseline for comparison and evaluating the
capability of strategies.

Figure 1.2: A scalable framework for continual image captioning.

From the results of the dissection, we try to infer the plastic components in
the network to preserve the accumulated knowledge. Our new algorithm, critical
freezing, protects these components by simply keeping weights unchanged while
training the new network. The experiments demonstrate catastrophic forgetting
in image captioning and the superiority of the proposed techniques over fine-
tuning. The previously learned tasks are well performed, whilst new
information is also well absorbed. Traditional metrics in image captioning,
such as BLEU, ROUGE-L, CIDEr, and SPICE are calculated for quantitative
assessments. Finally, we provide future directions and discussion over
experiments to elaborate on the results qualitatively and quantitatively.

Contributions: Our main contributions are as follows:

  * •

Our work is the first attempt to address catastrophic forgetting in image
captioning without the need for accessing data from existing tasks.

  * •

We propose a scalable framework (including Split MS-COCO, baselines,
experimental scenarios), which reconciles image captioning and continual
learning in the class-incremental scenario.

  * •

Our method is novel and pioneering in helping explain catastrophic forgetting
in continual learning.

  * •

We introduce a new approach to mitigate catastrophic forgetting based on the
findings.

  * •

Our extensive experiments demonstrate the efficacy of critical freezing and
suggest recondite understanding about catastrophic forgetting.

##  Chapter 2 Background and Related Work

###  2.1 Image Captioning

Given an image, our goal is to generate the most suitable caption describing
the content of an image. This task is rising as one of the most attractive
domains in computer vision because of its exceptionally valuable applications
such as aiding to the visually impaired, social media, digital assistant, or
photo indexing. Image captioning models often follow the encoder-decoder
architecture in which a powerful feature extractor (convolutional neural
network - CNN) incorporates a sequence generator (recurrent neural network -
RNN). Recently, attention mechanism has achieved huge successes in image
captioning. From the introduction of the Neural Image Caption Generator (NIC)
[5], various compelling techniques have been proposed to enhance image
captioning [6, 7, 8]. Experiments are frequently done on three benchmark
datasets: Flickr8 [9], Flickr30 [10], and MS-COCO [11].

The encoder compresses the input image to small images that represent features
of target objects. For years, deep convolutional networks have captured
features very well; as a result, transfer learning is leveraged in image
captioning to produce high-quality immediate features. To do this, last layers
of image classifiers [12, 13] are stripped away to expose immediate features
that are then fed into a long-short term memory network (LSTM). When
generating a sequence, at each time step, the last generated word is taken
into account to produce the next word, repeating until a special token (End of
Sequence - EOS) appears. There is also a mapping process from the number
generated by the network and a pre-built vocabulary. Fig. 2.1 illustrates the
prediction on a cat photo.

Figure 2.1: A synthetic caption is generated by deep learning.

###  2.2 Interpretable Machine Learning

Interpretability is the degree to which human can understand the cause of a
decision. Another one is: Interpretability is the degree to which a person can
consistently predict the model’s result. The higher the interpretability of a
machine learning model, the easier it is for someone to comprehend why certain
decisions or predictions have been made. A model is better interpretable than
another model if its decisions are easier for a human to comprehend than
decisions from the other model. We can use both the terms interpretable and
explainable interchangeably. Despite unbelievable breakthroughs of deep neural
networks, they are notorious for the ambiguity in the decision-making process.
While powerful, deep learning models are difficult to interpret, and thus
often treated as a black-box. Hence, people are reluctant when considering
deep learning in critical applications, such as disease prediction or
autonomous driving. An answer to unveil the myth of deep neural network is
Interpretable Machine Learning (or Interpretable ML), which helps users to
understand neural networks from inside as in Fig. 2.2. More precisely,
contemporary interpretability methods bring us advantages to understand the
decision-making process of deep neural networks [14, 15, 16].

Figure 2.2: Interpretable ML helps open the black-box of deep learning.

####  2.2.1 Network Dissection

Network dissection categorizes network components (e.g. channels) as human
concepts. This line of research is based on a hypothesis that a neural network
learns disentangled concepts. For instance, channel 241 detects red color, and
channel 999 is interested in human eyes. Bau and Zhou et al. [17] use a
trained model and propagate an image up to a target layer, then upscale the
activations to match the original image size and compare with the segmentation
ground-truth (shown in Fig. 2.3).

There are two notable findings that motivate us to study catastrophic
forgetting in neural networks. Many channels detect the same concept and in
transfer learning, the role of a given channel can totally change. For
example, a cat detector becomes a wheel detector.

Figure 2.3: Network dissection algorithm workflow.

####  2.2.2 Feature Visualization

Abstract features and concepts are learned by a network during training; and
thus, in feature visualization approaches, being visualized via activation
maximization. One of the greatest strengths of neural networks is that they
absorb high-level features in hidden layers, wiping out the need for
traditional feature engineering. The deeper a layer is, deep more
sophisticated features it captures. As shown in Fig. 2.4, the complexity of
learned features is increasing from left to right, indicating the depth of
layers.

Figure 2.4: Abstract features in hidden layers of neural networks.

In [18], deconvnets allow us to recognize which features are expected by a
specific part of a network or what properties of image excite a chosen neuron
the most. In stark contrast to feeding an input image to diagnose, Yosinski et
al. [19] attempt to generate an image which maximizes the activation of a
given neuron by gradient descent algorithm. Visualizing the activation of a
neuron or a layer in networks helps us categorize the specific role of each
block, layer, or even a node. It has been proved that the earlier layers
extract local features, such as edges or colors; while deeper layers are
responsible for detecting globally distinctive characteristics. Prediction
Difference Analysis (PDA) [20] \- shown in Fig. 2.5, even more specifically,
highlights pixels that support or counteract a certain class, indicating which
features are positive or negative to a prediction.

Figure 2.5: PDA highlights pixels that are evidence for (red) and against
(blue) a prediction of cat.

However, these tools only provide the computer vision, leaving the conclusion
for users. This manual process can not ensure the quality of the observation
when we may have hundreds or even thousands of feature maps. CFD automatically
detects the forgetting components in a network, entirely leveraging the
generated feature maps from PDA [20].

###  2.3 Continual Learning

Continual learning requires an ability to learn over time without witnessing
catastrophic forgetting (known as semantic drift), and allows neural networks
to incrementally solve new tasks. To achieve these goals of continual
learning, studies mainly focus on addressing catastrophic forgetting problem.
While the community has not yet agreed on a shared categorization for
continual learning strategies, Maltoni and Lomonaco [21] propose a three-way
fuzzy categorization of the most common continual learning strategies:
architecture, regularization, and rehearsal.

Rehearsal approaches [22, 23, 24] demand data from old tasks, leading to
privacy or limited storage budget issues, making them non-scalable if tasks
arrive ceaselessly. Architectural techniques [25, 22, 26, 27], at the same
time, alter the original network architecture for adapting to new tasks
besides preserving old memory. When a new task comes, layers or neurons are
typically added to the previous model. Consequently, the capacity of the
network is increased accordingly. This leads to a storage problem when the
number of new classes is numerous because the final model could be gigantic,
alleviating its portability. The concept of regularization approaches [28, 29,
30] is to keep the model close to its previous state by adding a penalty in
the objective function. Neither accessing old samples nor expanding the old
network is required in regularization methods.

Many works have managed to address the forgetting problem in generative models
[31], image captioning [32, 33, 34], or semantic segmentation [35]. Toy
datasets are preferred to be used when proving the feasibility of continual
learning algorithms. Necessarily, standard datasets are tweaked to fit the
experimental setup. Particularly, continual learning has been experimented on
benchmarks like MNIST variants [29, 36, 37, 26], CIFAR10/100 Split [28], and
ImageNet ILSVRC [38] for image classification. Shmelkov et al. [39] use PASCAL
VOC2007 [40] and COCO datasets [11] for object detection; and PASCAL VOC2012
[41] is selected in [42] for semantic segmentation. Seff et al. [43] suggest
to use an EWC-augmented loss [29] to counteract catastrophic forgetting in
synthetic image generation task while Lifelong GAN [31] leverages knowledge
distillation to transfer knowledge to the new network.

A work from Kemker et al. [3] conducts experiments on state-of-the-art
continual learning techniques that address catastrophic forgetting. It is
demonstrated that although the algorithms work, but only on weak constrains
and unfair baselines, thus the forgetting problem is not fully solved and
witnessed yet. They insist on the infeasibility of using toy datasets, such as
MNIST [29] or CIFAR [28] in continual learning experiments. As a result, this
work inspires us to create a new dataset from MS-COCO 2014 (Split MS-COCO).

Fine-tuning is considered as a baseline in [39, 35, 42, 44]. Freezing either a
few specific layers or a major part of a network is proposed in [42], which
reveals that just simply keeping some parameters unchanged can greatly helps
networks become robust against catastrophic forgetting. Learning without
Forgetting (LwF) [30] utilizes old models to generate pseudo data, which
guides the new model to reach a shared low-error region of problems. Also
leveraging the old networks, knowledge distillation approaches [42, 45] are
formally recognized to facilitate better generalization in life-long learning
via a teacher-student learning strategy. Nevertheless, algorithms are inclined
to rely on external factors (e.g., input or rehearsal data, objective
functions [30, 42]) while ignoring the question of why catastrophic forgetting
internally happens. In this research, our algorithm is derived from
catastrophic forgetting exploration.

##  Chapter 3 Approach

Although research from [4] shows an interest in understanding catastrophic
forgetting, they focus on how task properties influence the hardness of
sequential learning. Hence, they are explaining based on the input data. CFD
approaches the problem from an alternative perspective, trying to explain how
forgetting happens over time based on the computer vision of models. In
comparison, the ultimate goal of this tool is to figure out the most plastic
layers or blocks in a network. Plasticity means a low degree of stiffness or
being easy to change. Although a variety of continual learning techniques have
been proposed to alleviate catastrophic forgetting, none of them takes
advantage of the findings from Interpretable ML. Critical freezing is built on
the top of CFD’s investigation to provide an interpretable and effective
approach to deal with catastrophic forgetting. In the learning process, the
optimal state of the old model is employed to initialize the new network. This
way mimics the working mechanism of the human brain.

Derived from the network structure, we propose to freeze the encoder and
decoder in turns (partially freezing) to moderate semantic drift. Pseudo-
labeling makes use of the previous model to interpret predictions of the
previous model on the coming samples while knowledge distillation utilizes the
previous model as a teacher model, forcing a student to not forget solved
problems.

###  3.1 Partially Freezing for Image Captioning

A heuristic solution to hinder catastrophic forgetting is to freeze a part of
model. As our architecture is divided into an encoder and a decoder, keeping
one component intact can affect directly to the performance on the previous
tasks as well as the new task.

On the one hand, the encoder is frozen and the decoder is updated exploiting
the new dataset. The feature extraction remains compared to the previous model
$M_{A}$ which operates as a fine-tuned model (Task A). The feature extractor
$E$ is constrained, leaving combatting with the new task for the decoder $D$,
we refer as $E_{F}$ (see Fig. 3.1). The vocabulary is expanded to ensure the
naturalness of the prediction, and the decoder is modified according to the
vocabulary size.

Figure 3.1: Freezing a component of the fine-tuned model while the remaining
component is trainable.

On the other hand, we freeze the decoder $D$ while the encoder is trainable
($D_{F}$ as in Fig. 3.1). Because neurons are added to the decoder as by
virtue of expanding vocabulary, the newly attached neurons are the only
trainable part of the decoder. By setting like this, the convolutional network
can better learn unseen features, thus feeding more fine-grained input into
the decoder. In the case of both $E$ and $D$ being frozen, the new model is
the same as the previous model, while fine-tuning is when both $E$ and $D$ are
trainable.

The objective function when applying freezing techniques is the standard loss
function for image captioning:

$\displaystyle\mathcal{L}={}$ $\displaystyle L_{CE}$ (3.1) $\displaystyle={}$
$\displaystyle-\sum_{i=1}^{v}Y_{k}^{i}\log\hat{Y}_{k}^{i}$

In the equation (3.1), $L_{CE}$ is the cross-entropy loss over annotation and
the prediction, $v$ is the size of vocabulary. $Y_{k}$ is the ground truth
(one-hot vector), and $\hat{Y}_{k}$ is the predicted caption (probability
distribution over vocabulary).

###  3.2 Pseudo-labeling (Learning without Forgeting - LwF)

We refer this method as pseudo-labeling since when training, pseudo-labels are
generated to guide the current model to mimic the behavior of the previously
trained model. The procedure of this approach is described as follows:

⬇

At task $k^{th}$,

Start with:

$\theta_{k-1}$ : parameters of the old task

$X_{k}$, $Y_{k}$ : training images and reference captions on the new task

Initialize:

$Y_{k-1}$ $\leftarrow$ $M(X_{k},\theta_{k-1})$ // infer captions by the old

// model on new data

$\theta_{k}\leftarrow\theta_{k-1}$ // initialize new model by old

// parameters

Train:

$\hat{Y}_{k}\leftarrow M(X_{k},\hat{\theta}_{k})$ // new task prediction

$\theta_{k}^{*}\leftarrow\underset{\hat{\theta}_{k}}{argmin}(L(Y_{k},\hat{Y}_{k})+L(Y_{k-1},\hat{Y}_{k}))$

In [30], they use task-specific parameters to generate the pseudo labels while
in our proposed pseudo-labeling approach, we use a unified network for the
whole process. This point eases the training because we do not need to
separate the network into sub-modules. We also do not apply knowledge
distillation in pseudo-labeling approach because we do not expect the logit
values in captioning task. The pseudo labels are fake captions on the new
data.

Pseudo labels $Y_{k-1}$ are acquired by inferring the caption of all input
images in the dataset of the new task using the previous model. Although new
classes appear, the convolutional network architecture stays unchanged during
training since the expected output is textual caption, not the probabilities
for classes as in object classification or semantic segmentation. After being
initialized by the foregoing model, we run training in a supervised manner to
minimize the loss computed based on the ground truth, the predicted caption,
and the pseudo labels $Y_{k-1}$. The loss component from pseudo-labeling
$L_{P}$ is explicitly written as:

$\displaystyle{L_{P}}=-\beta\cdot\sum_{i=1}^{v}Y_{k-1}^{i}\log\hat{Y}_{k}^{i}$
(3.2)

where $\beta$ is considered to be a regulator to accentuate the old or new
task ($\beta=1$ in experiments), and the final objective function for this
approach is:

$\displaystyle\mathcal{L}={}$ $\displaystyle L_{CE}+L_{P}$ (3.3)
$\displaystyle={}$
$\displaystyle-\sum_{i=1}^{v}Y_{k}^{i}\log\hat{Y}_{k}^{i}-\beta\cdot\sum_{i=1}^{v}Y_{k-1}^{i}\log\hat{Y}_{k}^{i}$

###  3.3 Knowledge Distillation

This technique frames the training process in a teacher-student strategy when
we want to deliver the knowledge acquired from one model to another model. By
applying distillation, teacher is the model obtained from the old task while
student is the new model. Once an image $X_{n}$ is passed through both teacher
and student, the mean squared error between outputs of student and teacher
($Y_{k}^{st}$ and $Y_{k}^{tr}$) is added to the loss function of student to
penalize it. We propose to distill the feature learned by the encoder to
provide the student decoder the flexibility adapting to the new task. This
approach is felicitous because the spatial information is global while the
sentence semantics changes constantly over domains. Thus, the decoder is freer
to change, in comparison with the encoder. The distillation term $L_{dis}$ is
computed as:

$\displaystyle{L_{dis}}=\lambda\cdot\left\|Y_{k}^{tr}-Y_{k}^{st}\right\|^{2}$
(3.4)

where $\lambda$ may be increased to encourage student to imitate the behaviors
of teacher more intensively ($\lambda=1$ in experiments). The loss function
is:

$\displaystyle\mathcal{L}={}$ $\displaystyle L_{CE}+L_{dis}$ (3.5)
$\displaystyle={}$
$\displaystyle-\sum_{i=1}^{v}Y_{k}^{i}\log\hat{Y}_{k}^{i}+\lambda\cdot\left\|Y_{k}^{tr}-Y_{k}^{st}\right\|^{2}$

###  3.4 Catastrophic Forgetting Dissector - CFD

To dissect the model, we use the activation of hidden layers like [46] but to
understand the forgetting effect. We initially hypothesized that different
features of the objects could be captured and visualized by particular feature
maps in different layers. By looking into each response map in one
convolutional block (conv block), we realize diverse features, such as eyes,
face shape, car wheels, or background are isolatedly recognized by different
channels, which avers our hypothesis. Unfortunately, finding each feature map
manually by human eyes might be inefficient. To solve this issue, we only
compare the computer vision with the ground truth segmentation rather than
small details. The ground truth is sampled as in Fig. 3.2. In general, we do
not seek for the answer that what features are being forgotten, but which
layers are now forgetting.

Figure 3.2: Segmentation ground truth as input of CFD.

The scheme of CFD is illustrated in Fig. 3.3. The visualization from the tool
and the ground truth segmentation are simultaneously employed to infer the
forgetting blocks. The visualizations of the tool and the ground truth
segmentation are simultaneously employed to infer the forgetting blocks. We
assume that the semantic segmentation label of MS-COCO dataset [11] is what
human eyes perceive. Next, we compare this segmentation with the computer
vision of the model, particularly concentrating on positive evidence for a
prediction to see how supportive features are disregarded.

Figure 3.3: CFD working on ResNet-50 that includes 5 convolutional blocks.
Figure 3.4: Choosing representatives based on overlap between feature maps and
ground truth.

The IoU value between the segmentation and evidence is calculated as shown in
Fig. 3.5 (a). On the right side of Fig. 3.5 (a), we have an input image of a
train, red dots advocate the fact that the output should be “train” while blue
ones contradict this prediction.

Having the m-th feature map ($FM$) in the l-th layer of a model M and the
ground truth segmentation GT, the IoU is computed as in equation (3.6):

$IoU_{M,GT}(l,m)=\frac{FM(l,m)\cap GT}{FM(l,m)\cup GT}$ (3.6)

To select the feature map having the largest overlap with the ground truth in
l-th conv block, the representative feature map (RM) with the best IoU is
$RM_{M,GT}$ :

$RM_{M,GT}(l)=argmax_{m}(IoU_{M,GT}(l,m))$ (3.7)

To understand how the computer vision changes over the training process, we
compare the $RM$s in the new model with the $RM$s of the old model shown in
Fig. 3.5 (b). We compute $IoU_{M_{O},GT}$ by equation (3.6) then achieve
$RM_{M_{O},GT}$ from equation (3.7) ($RM_{M_{O},GT}$ is the representative
feature map of the old model). The forgetting effect of each trained model is
measured by the IoUs between $RM_{M_{O},GT}$ and feature maps of $M_{N}$
($M_{N}$ is the new model):

$IoU_{M_{N},M_{O}}(l,m)=\frac{FM(m)\cap RM_{M_{O},GT}}{FM(m)\cup
RM_{M_{O},GT}}$ (3.8)

|
---|---
Figure 3.5: (a) IoU value between the segmentation of a train and the positive
features. (b) The IoU on the positive features of two representative maps. Red
is evidence and blue is against. (c) IoU graph and computer vision of a bird
image. Yellow-bounding and blue-bounding boxes are from the old and new model
respectively.

Similar to the method of finding out the best map fitting with ground truth,
the feature map representing the best memory of the original feature map is
denoted as $RM_{M_{N},M_{O}}$ in equation (3.9). $RM_{M_{N},M_{O}}$ is
specified as:

$RM_{M_{N},M_{O}}(l)=argmax_{m}(IoU_{M_{N},M_{O}}(l,m))$ (3.9)

For instance, when a new class is added, we compare the computer vision of the
consecutive models to determine which feature maps generated by the new model
should be used to observe the forgetting. As shown in Fig. 3.6, the
representative on the first layer on the new model should have the largest
overlap in comparison to the representative on the same layer of the old
model. The procedure is done similarly on other layers.

Figure 3.6: Choosing representatives based on computer visions of two models.

In the same block of both the old and new model, the role of a filter can be
adjusted. For instance, the $50^{th}$ filter in the $2^{nd}$ block of the old
model detects the eyes, but the same filter in the same block of the new model
may consider the face. Hence, we should not make a comparison based on the
index of a filter.

Algorithm 1 CFD

1: Input: Sample set $S$, segmentation ground truth $GT$, old model $M_{O}$,
new model $M_{N}$, number of blocks $K$

2: Output: Forgetting layer $\mathbb{F}$

3: $i=0$

4: $\L=\emptyset$

5: repeat

6: $I=S[i]$

7: $IoUs=\emptyset$

8: $FM=PDA(I)$

9: for $j=1$ to $K$ do

10: $RM_{M_{O},GT}\leftarrow\textit{FM with highest }{IoU_{M_{O},GT}}$

11: $RM_{M_{N},M_{O}}\leftarrow\textit{FM with highest }{IoU_{M_{N},M_{O}}}$

12: Append($IoUs,max(IoU_{M_{N},M_{O}})$)

13: end for

14: $\boldsymbol{b}\leftarrow\textit{blocks with the highest drop in IoUs}$

15: Append(Ł, $\boldsymbol{b}$)

16: $i=i+1$

17: until $i=\textbf{size}($S$)$

18: $\mathbb{F}$ $\leftarrow$ Most frequent block in Ł

The workflow of CFD is given by Algorithm 1. The sample set $S$ is described
in Section 4, $GT$ is the segmentation ground-truth from MS-COCO dataset,
$M_{O}$ and $M_{N}$ are the old and new model for comparison respectively, and
$K$ is the number of the conv blocks in the network ($K=5$ with ResNets). $\L$
is a list containing the most forgetting layer with respect to all the images
in $S$. By inputting an image $I$ from $S$, we get the visualization of
feature maps ($FM$) over $M_{O}$ and $M_{N}$ by PDA. However, we need to pick
the representative feature map (RM) amongst thousands of feature maps in a
conv block.

To choose the representative feature map $RM$ of the $j^{th}$ conv block in a
model, we define representative feature map $RM$ to be the feature map having
the largest overlap with the $RM$ of the previous model ($RM_{prev}$ in short)
for the same $j^{th}$ block. Particularly, the $RM_{prev}$ of $M_{O}$ is the
ground truth because $M_{O}$ is the starting model, and $RM_{M_{O},GT}$ is the
representative feature map at $j^{th}$ block of $M_{O}$. Likewise, $RM_{prev}$
of $M_{N}$ at $j^{th}$ block is $RM_{M_{O},GT}$, and we obtain
$RM_{M_{N},M_{O}}$ by comparing feature maps of $M_{N}$ and $RM_{M_{O},GT}$.
The IoU values between $RM_{M_{O},GT}$ and $RM_{M_{N},M_{O}}$ are calculated
at each conv block and appended to a list $IoUs$. After calculating IoU drops
through the ResNet blocks and denote the block giving the highest drop as $b$,
we can put $b$ as the block where the most substantial forgetting happens into
a list $\L$, tested on the input image $I$. Finally, we generalize on all
images of $S$ to return the most forgetting component $\mathbb{F}$.

In the juxtaposition of the old and new model, the IoUs are visually drawn to
provide a bird-eye view of the forgetting trend shown in Fig. 3.5 (c). We may
argue that the conv block having the lowest IoU (block 5) is the victim of
catastrophic forgetting. However, this assumption is not asserted because of
the error accumulation in deep neural networks. The computer vision of deeper
layers is directly attributed to earlier conv blocks. Once forgetting occurs
in the first layer, it will be propagated throughout the entire network.

We propose to leverage IoU slopes to find the weakest block $b$. At a point,
if the IoU drops significantly compared to the previous value, it should be
the sign of catastrophic forgetting. In Fig. 3.5 (c), a plummet of the IoU
value is seen between block 2 and block 3 (0.642 to 0.388). The first thought
appearing in our mind was that the $3^{rd}$ is forgetting most
catastrophically. It is true but we need to regard the fact that the worst map
in block 3 is a result of block 2 and block 1. This finding is the foundation
for our technique to prevent catastrophic forgetting.

###  3.5 Critical Freezing

Fine-tuning techniques play a pivotal role in training deep networks if data
distribution evolves. Regarding a pre-trained model, the feature extractor
which captures global information is carefully protected in adaptation. The
output layer may be superseded, or the learning rate should be tweaked to a
tiny number. Another dominant approach is to freeze the weights of the early
layers. They are all effective yet ambiguous because we can not ensure
freezing which layers will give the best result. Using the investigation from
CFD, we freeze the precursors of the most plastic layer in a deep neural
network. If a network has $K$ conv blocks, and we find the $F^{th}$
convolutional layer broken, then we try to freeze earlier layers than the
$F^{th}$ layer. If updating the fragile components is necessary, a learning
rate on those blocks should be thoroughly calibrated. The procedure of
critical freezing is shown in Algorithm 2. The objective function is the
standard cross-entropy loss for image captioning, $\mathbb{V}$ is the size of
the vocabulary, $Y_{k}^{i}$ is the ground truth, and $\hat{Y}_{k}^{i}$ is the
prediction.

Algorithm 2 Critical Freezing

1: Input: Sample set $S$, segmentation ground truth $GT$, old model $M_{O}$,
new model $M_{N}$, number of blocks $K$

2: Output: optimal state $\theta^{*}$

3: $M_{N}$ $\leftarrow$ $M_{O}$ // initialize new model by old parameters

4: $F$ $\leftarrow$ CFD($S$, $GT$, $M_{O}$, $M_{N}$, $K$)

5: for $i=1$ to $F-1$ do

6: grad($M_{N}[i]$, $False$) // freeze the layer $i^{th}$

7: end for

8: $\theta^{*}$ $\leftarrow$
$\underset{\hat{\theta}}{argmin}(-\sum_{i=1}^{\mathbb{V}}Y_{k}^{i}\log\hat{Y}_{k}^{i})$

###  3.6 Implementation Details

####  3.6.1 Feature Extraction

We use the pre-trained model of ResNet-50 [12], removing the final fully
connected layer to obtain the immediate features. Behind the ResNet is an
embedding layer which has an embedding size of 256. This embedding layer is
responsible for bridging the output of the ResNet and the input to the RNN,
followed by a batch normalization layer with $momentum=0.01$.

####  3.6.2 Language Model

The entrance of the language model is an embedding module to embed the
features and labels to feed the LSTM. The RNN contains a single layer LSTM
with the size of hidden state is 512 and the embedding size is 256. A fully
connected layer is placed at the end of the language model to generate the
score for each entry in the vocabulary. Note that when the vocabulary is
increasingly accumulated, the embedding layer and the output layer of the
decoder are also enlarged accordingly.

####  3.6.3 Training Details

To help the model work with images of different sizes, we always resize input
to square images. We shuffle images at each epoch, and the batch size is 16.
Early stopping is used to help models reach the optimal state within 50 epochs
of training. The value $patience$ \- the number of epochs to wait before early
stop if no loss improvement on the validation set, is empirically set to 5.
For updating both encoder and decoder, we choose an Adam optimizer [47] with a
shared learning rate of $5\times 10^{-4}$. Exceptionally, in knowledge
distillation, we apply another Adam optimizer to encoder with a learning rate
of $5\times 10^{-5}$. The parameters of encoder could be changed dramatically
during training time, so we want to slow it down.

All methods but distillation are initialized by the final state of the old
task. While joint training is incorporated only in pseudo-labeling, normal
training procedure is coupled to each remaining technique. Joint training is
when we first freeze the entire model except for the newly added neurons, then
train to converge. Subsequently, we unfreeze the model and perform training
again until convergence (joint-optimization).

The training environment includes one Nvidia TitanX GPU and an Intel(R)
Core(TM) i7 4.0 GHz PC with 16GB RAM, running Ubuntu 16.04 LTS (64-bit).

##  Chapter 4 Evaluation

The captioning model is divided into an encoder followed by a decoder.
Therefore, the original structure should be transformed into a two-head
network so that we can get the prediction of the CNN and the sentence
generated from the decoder simultaneously. We add one more output layer as a
classifier after the encoder, and our proposed architecture is presented in
Fig. 4.1.

Figure 4.1: Two-head network for dissecting catastrophic forgetting.

###  4.1 Dataset

Our experimental results have been recorded on Split MS-COCO, a variant of the
MS-COCO benchmark dataset [11]. The original MS COCO dataset contains 82,783
images in the training split and 40,504 images in the validation split. Each
image of MS COCO dataset belongs to at least one of 80 classes. To create
Split MS-COCO, we carry out processing on MS-COCO as the following steps:

  1. 1.

Split all images into distinguished classes. There are 80 classes in total.

  2. 2.

Resize images to a fixed size of ($224\times 224$) in order to deal with
different sizes of input.

  3. 3.

Discard images including more than one class in its annotations to obtain
“clear” images only.

Taking a clear “dog” image as an example, the reference captions will contain
“dog” in their content and may also have “grass” (not in 80 classes) but NOT
contain any of the 79 remaining classes defined by MS-COCO (cat or frisbee).
The reason that we only choose clear images is that we want to assure when
facing a new task, objects in images of the new task are unseen.

Since MS-COCO test set is not available, we divide the validation set of each
class into two equal parts: one is for validation and the other for testing.
From over 82k images of training set and 40k of validation set from the
original MS-COCO 2014, the new dataset has 47,547 images in the training set;
11,722 images in the validation set; and 11,687 images in the test set.

###  4.2 Evaluation Metric

We report the capacity of the framework using traditional scores for image
captioning, such as BLEU, ROUGE, CIDEr, and SPICE by coco-caption [48]. While
BLEU and ROUGE are variants of overlapping-based metrics and linearly
correlated, CIDEr and SPICE give more weight-age to important terms, thus
giving higher correlation with human judgments. For each scenario, we have a
table correspondingly. The performance of the techniques on the old task is on
the left side while the figure for the new task is located on the right side
of the tables.

###  4.3 Addition of Multiple Classes Sequentially

Initially, we train with 19 classes to acquire the base model, followed by
adding 5 classes sequentially. The sceraio is illustrated in Fig. 4.2.

Figure 4.2: Adding 5 classes one by one scenario.

Predictably, when the model faces unseen classes, the knowledge in old classes
will be forgotten on the feature extractor. Using the two-head model, we can
track both the prediction of the classifier and the computer vision. The model
can predict correctly when a class is added, but when the number of classes
increases, the prediction becomes totally irrelevant (shown in Fig. 4.3).

Figure 4.3: Forgetting in feature extractor when using fine-tuning.

The captioning model is divided into an encoder and a decoder, in which the
encoder is the ResNet-50, and the decoder includes an embedding layer, a
single-layer LSTM, and a fully-connected layer producing a word at a time
step. As CFD works on a single image, running multiple times on different and
diverse input images is needed, helping us to generalize the observation of
forgetting.

We choose a sample set $S$ (bicycle, car, motorcycle, airplane, bus, train,
bird, cat, dog, horse, sheep, and cow) from 19 trained classes.

The results of CFD shown in Table. 4.1 reinforce the fact that the $3^{rd}$
conv block of ResNet is the most plastic component of this famous conv net
given the learning sequence. To evaluate critical freezing, we perform fine-
tuning and various schemes of freezing. In fine-tuning, the old model
initializes the new model, and training is done by minimizing the loss on the
new task. As the network contains two parts, encoder and decoder, we freeze
them separately to specify the best freezing strategy. In addition, we choose
various combinations of layers to be frozen besides a famous baseline in
continual learning experiments called LwF [30]. Two knowledge distillation
techniques from [42] are also taken into comparison. The traditional scores
for image captioning are considered in evaluating the superiority of critical
freezing over the baselines. $BLEU4$ and $ROUGE\\_L$ are essentially word-
overlap based metrics, while CIDEr and SPICE are more trustworthy because they
give more weights on significant terms, such as verbs or nouns. Therefore, we
will give discussion only on CIDEr score for conciseness.

| Category | bicycle | car | motor | airplane | bus | train | bird | cat | dog | horse | sheep | cow
---|---|---|---|---|---|---|---|---|---|---|---|---|---
$\mathbb{F}$ |  $1^{st}$ layer |  |  |  |  |  |  |  |  |  |  |  |
$2^{nd}$ layer |  |  |  |  |  |  |  |  |  |  |  |
$3^{rd}$ layer | ✓ | ✓ | ✓ |  | ✓ | ✓ |  | ✓ |  | ✓ | ✓ |
$4^{th}$ layer |  |  |  | ✓ |  |  | ✓ |  | ✓ |  |  | ✓
$5^{th}$ layer |  |  |  |  |  |  |  |  |  |  |  |
Table 4.1: CFD’s result on various instances. Figure 4.4: Feature maps from
convolutional blocks of ResNet-50.

After adding a new class, we obtain $M_{20}$, and $M_{n}$ is the model when a
total number of $n$ classes are witnessed. In Fig. 4.4, the visualized results
show that the first and second blocks of ResNet-50 can overall capture the
outline of objects. Computer vision turns to represent more detailed features
from the objects and other background features to determine the class of the
input image in the deeper blocks. Also, the IoUs of different blocks in
models, comparing with ground truth, are calculated by equation (3.6) and
equation (3.7). The results reveal that although different models show the
performance of the classification inconsistently, the $IoU_{M_{n},GT}$
($n>19$) values are roughly similar at all the layers, which implies that no
matter the how good the performance is, the level of matching between feature
maps of each model and the human vision is preserved (Fig. 4.5 (a)).

|
---|---
Figure 4.5: (a) $IoU_{M,GT}$ of $M_{19}$, $M_{20}$ and $M_{24}$ comparing with
GT. (b) $IoU_{MT,MO}$ of model $M_{20}$ and $M_{24}$ comparing with $M_{19}$.

To measure how much the forgetting occurs in the ResNet, we compute IoUs by
equation (3.8) and equation (3.9). It is clearly shown in Fig. 4.5 (b) that
the IoUs between $M_{20}$ and $M_{19}$ are roughly equal to the corresponding
figure for $M_{24}$ and $M_{19}$ in every block. For $M_{20}$, the
$IoU_{M_{20},M_{19}}$ is always around 0.8 at the first block, showing that a
marginal forgetting happens here. The $IoU_{M_{20},M_{19}}$ starts to drop
along the blocks because the later feature maps are constructed by the
previous maps. The forgetting effect persists and does not show which block is
forgetting the most.

For $M_{24}$, the first block of the model still gets a high IoU comparing
with $M_{19}$ and the values decrease from the second block. Unlike the
constantly decreasing trend seen in $M_{20}$, the decreasing rate of
$IoU_{M_{24},M_{19}}$ fluctuates through the blocks and a severe drop at block
3 is observed in every testing input, suggesting that the forgetting effect
might happen the most in this block. Iterating this procedure on all images of
the set $S$ reinforces that the worst forgetting happens at block 3.

$Past-task$ $New-task$ BLEU4 ROUGE_L CIDEr SPICE BLEU4 ROUGE_L CIDEr SPICE
Fine-tuning 4.2 32.3 4.9 1.9 10.1 39.6 17.9 5.9 Encoder-Freezing 3.8 31.5 5.2
2.0 8.8 38.4 15.1 5.1 Decoder-Freezing 5.2 33.3 6.8 2.2 10.6 39.8 18.5 5.5
L1-Freezing 5.2 33.5 7.9 2.4 12.0 41.1 23.4 6.4 L2-Freezing 5.8 33.6 7.4 2.1
12.0 41.2 24.0 6.5 L3-Freezing 4.5 32.2 6.8 2.1 10.4 39.6 20.7 6.0 L4-Freezing
5.0 32.6 7.6 2.2 11.4 40.0 23.3 6.2 L5-Freezing 5.4 33.0 7.4 2.3 12.1 40.4
23.3 6.2 LwF[30] 6.4 33.2 9.7 2.6 10.6 38.9 16.1 5.3 KD1 [42] 4.7 33.3 6.7 2.1
11.4 40.4 20.8 6.2 KD2 [42] 3.9 32.4 5.6 2.0 10.6 39.8 18.6 5.9 Critical
Freezing 5.6 33.2 9.5 2.8 12.2 40.7 26.1 6.9

Table 4.2: Performance when 5 classes arrive sequentially on past tasks and
newly added tasks.

While two naive approaches of freezing are also implemented, we devise
critical freezing based on findings, which only freezes critical layers. As
shown in Table. 4.2, precisely freezing helps to learn on both the new and old
tasks much more effectively. Our freezing scheme outperforms the other
approaches on new tasks by a large margin (26.1 CIDEr) while achieving
comparable performance with LwF [30] on past tasks (9.5 CIDEr) although LwF
[30] is far more complicated. Knowledge distillation on intermediate feature
space (KD1) and output layer (KD2) [42] claims 20.8 and 18.6 CIDEr
respectively. We argue that the frozen layers contain global information
derived from past tasks, which is valuable and should be accumulated during
lifetime rather than changing. Fine-tuning optimizes the loss on the new task
without any guidance; as a result, the model may not fall into the low-error
regions of tasks. We try to freeze each layer of the ResNet to fortify the
hypothesis that properly freezing is really better than ambiguous freezing in
fine-tuning schemes. Hence, critical freezing exerts a promising influence on
fine-tuning techniques in deep learning.

Figure 4.6: Qualitative results of continual learning algorithms.

Captions are inferred in every setting to qualitatively evaluate the framework
(See Fig. 4.6). $F$, $E_{F}$, $D_{F}$, and $CF$ are fine-tuning, encoder
freezing, decoder freezing, and critical freezing respectively. The model
misclassifies a cat without using pseudo-labeling or critical freezing when
adding 5 classes sequentially. On an image of new classes, critical freezing
produces the most relevant caption to the content.

###  4.4 Conclusion and Future Work

In this work, we introduce a scalable framework, fusing continual learning and
image captioning, working on a new dataset called Split MS-COCO created from
the standard MS-COCO. We firstly perform image captioning in incremental
schemes and then add techniques from continual learning to weaken catastrophic
forgetting. Working on the most challenging scenario (class incremental)
ensures our framework can perform well in practical schemes while old classes
and new classes can co-occur. The experiments in three settings indicate the
sign of catastrophic forgetting and the effectiveness when integrating
freezing, pseudo-labeling, and distillation.

As the presence of catastrophic forgetting hinders the life-long learning,
understanding how this phenomenon happens in computer vision is imperative. We
introduce CFD to grasp catastrophic forgetting. The investigation of our tool
unearths the mystical question about catastrophic forgetting.

From knowing where the forgetting issue is coming from, a new technique has
been proposed focusing on plastic components of a model to moderate the
information loss. The experiments illustrate the superiority of critical
freezing over various freezing schemes and existing techniques. To the best of
our knowledge, no work has been done for mitigating catastrophic forgetting
under the supervision of Interpretable ML. By knowing which regions are needed
to be kept intact, not only could the performance on the old task be largely
improved, but the new task is also more conquerable. Ultimately, critical
freezing could benefit a variety of fine-tuning schemes and continual learning
approaches.

There are future works following our research. Firstly, scaling this work for
other tasks and deep networks can better validate the feasibility of the
proposed continual learning algorithm. Secondly, RNN is now being overlooked
and not understood fully via interpretability methods. RNN dissection is
arduous but would greatly impact diverse domains (captioning, machine
translation, or text synthesis).

## References

  * [1] Charles X Ling and Tanner Bohn.  A unified framework for lifelong learning in deep neural networks.  arXiv preprint arXiv:1911.09704, 2019.
  * [2] Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio.  An empirical investigation of catastrophic forgetting in gradient-based neural networks.  arXiv preprint arXiv:1312.6211, 2013.
  * [3] Ronald Kemker, Marc McClure, Angelina Abitino, Tyler L Hayes, and Christopher Kanan.  Measuring catastrophic forgetting in neural networks.  In Thirty-second AAAI conference on artificial intelligence, 2018\.
  * [4] Cuong V Nguyen, Alessandro Achille, Michael Lam, Tal Hassner, Vijay Mahadevan, and Stefano Soatto.  Toward understanding catastrophic forgetting in continual learning.  arXiv preprint arXiv:1908.01091, 2019.
  * [5] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.  Show and tell: A neural image caption generator.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156–3164, 2015.
  * [6] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh.  Neural baby talk.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7219–7228, 2018.
  * [7] Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara.  Show, control and tell: A framework for generating controllable and grounded captions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8307–8316, 2019.
  * [8] Jyoti Aneja, Aditya Deshpande, and Alexander G Schwing.  Convolutional image captioning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5561–5570, 2018.
  * [9] Micah Hodosh, Peter Young, and Julia Hockenmaier.  Framing image description as a ranking task: Data, models and evaluation metrics.  Journal of Artificial Intelligence Research, 47:853–899, 2013.
  * [10] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In Proceedings of the IEEE international conference on computer vision, pages 2641–2649, 2015.
  * [11] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [13] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015.
  * [14] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  arXiv preprint arXiv:1312.6034, 2013.
  * [15] Piotr Dabkowski and Yarin Gal.  Real time image saliency for black box classifiers.  In Advances in Neural Information Processing Systems, pages 6967–6976, 2017.
  * [16] Giang Nguyen, Daeyoung Kim, and Anh Nguyen.  The effectiveness of feature attribution methods and its correlation with automatic evaluation scores.  Advances in Neural Information Processing Systems, 34:26422–26436, 2021.
  * [17] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Network dissection: Quantifying interpretability of deep visual representations.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6541–6549, 2017.
  * [18] Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In European conference on computer vision, pages 818–833. Springer, 2014.
  * [19] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson.  Understanding neural networks through deep visualization.  arXiv preprint arXiv:1506.06579, 2015.
  * [20] Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling.  Visualizing deep neural network decisions: Prediction difference analysis.  arXiv preprint arXiv:1702.04595, 2017.
  * [21] Davide Maltoni and Vincenzo Lomonaco.  Continuous learning in single-incremental-task scenarios.  Neural Networks, 116:56–73, 2019.
  * [22] David Lopez-Paz et al.  Gradient episodic memory for continual learning.  In Advances in Neural Information Processing Systems, pages 6467–6476, 2017.
  * [23] Onur Tasar, Yuliya Tarabalka, and Pierre Alliez.  Incremental learning for semantic segmentation of large-scale remote sensing data.  arXiv preprint arXiv:1810.12448, 2018.
  * [24] Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari.  End-to-end incremental learning.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 233–248, 2018.
  * [25] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  arXiv preprint arXiv:1606.04671, 2016.
  * [26] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang.  Lifelong learning with dynamically expandable networks.  arXiv preprint arXiv:1708.01547, 2017.
  * [27] German Ignacio Parisi, Jun Tani, Cornelius Weber, and Stefan Wermter.  Lifelong learning of spatiotemporal representations with dual-memory recurrent self-organization.  Frontiers in neurorobotics, 12:78, 2018.
  * [28] Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3987–3995. JMLR. org, 2017.
  * [29] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  * [30] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017.
  * [31] Mengyao Zhai, Lei Chen, Frederick Tung, Jiawei He, Megha Nawhal, and Greg Mori.  Lifelong gan: Continual learning for conditional image generation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2759–2768, 2019.
  * [32] Giang Nguyen, Shuan Chen, Tae Joon Jun, and Daeyoung Kim.  Explaining how deep neural networks forget by deep visualization.  In International Conference on Pattern Recognition, pages 162–173. Springer, 2021.
  * [33] Giang Nguyen, Shuan Chen, Thao Do, Tae Joon Jun, Ho-Jin Choi, and Daeyoung Kim.  Dissecting catastrophic forgetting in continual learning by deep visualization.  arXiv preprint arXiv:2001.01578, 2020.
  * [34] Giang Nguyen, Tae Joon Jun, Trung Tran, Tolcha Yalew, and Daeyoung Kim.  Contcap: A scalable framework for continual image captioning.  arXiv preprint arXiv:1909.08745, 2019.
  * [35] Onur Tasar, Yuliya Tarabalka, and Pierre Alliez.  Incremental learning for semantic segmentation of large-scale remote sensing data.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(9):3524–3537, 2019.
  * [36] Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung-Tak Zhang.  Overcoming catastrophic forgetting by incremental moment matching.  In Advances in neural information processing systems, pages 4652–4662, 2017.
  * [37] Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner.  Variational continual learning.  arXiv preprint arXiv:1710.10628, 2017.
  * [38] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [39] Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari.  Incremental learning of object detectors without catastrophic forgetting.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3400–3409, 2017.
  * [40] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International journal of computer vision, 88(2):303–338, 2010.
  * [41] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results.  http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html.
  * [42] Umberto Michieli and Pietro Zanuttigh.  Incremental learning techniques for semantic segmentation.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [43] Ari Seff, Alex Beatson, Daniel Suo, and Han Liu.  Continual learning in generative adversarial nets.  arXiv preprint arXiv:1705.08395, 2017.
  * [44] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.  icarl: Incremental classifier and representation learning.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017.
  * [45] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin.  Lifelong learning via progressive distillation and retrospection.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 437–452, 2018.
  * [46] Trung Q Tran, Giang V Nguyen, and Daeyoung Kim.  Simple multi-resolution representation learning for human pose estimation.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 511–518. IEEE, 2021.
  * [47] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [48] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  arXiv preprint arXiv:1504.00325, 2015.