# Overcome Anterograde Forgetting with Cycle Memory Networks

Jian Peng 1, Dingqi Ye 1, Bo Tang2, Yinjie Lei 3, Yu Liu 4, Haifeng Li 1,*

###### Abstract

Learning from a sequence of tasks for a lifetime is essential for an agent
towards artificial general intelligence. This requires the agent to
continuously learn and memorize new knowledge without interference. This paper
first demonstrates a fundamental issue of lifelong learning using neural
networks, named anterograde forgetting, i.e., preserving and transferring
memory may inhibit the learning of new knowledge. This is attributed to the
fact that the learning capacity of a neural network will be reduced as it
keeps memorizing historical knowledge, and the fact that the conceptual
confusion may occur as it transfers irrelevant old knowledge to the current
task. This work proposes a general framework named Cycled Memory Networks
(CMN) to address the anterograde forgetting in neural networks for lifelong
learning. The CMN consists of two individual memory networks to store short-
term and long-term memories to avoid capacity shrinkage. A transfer cell is
designed to connect these two memory networks, enabling knowledge transfer
from the long-term memory network to the short-term memory network to mitigate
the conceptual confusion, and a memory consolidation mechanism is developed to
integrate short-term knowledge into the long-term memory network for knowledge
accumulation. Experimental results demonstrate that the CMN can effectively
address the anterograde forgetting on several task-related, task-conflict,
class-incremental and cross-domain benchmarks.

## Introduction

Humans can continuously acquire and transfer knowledge and skills over time.
This ability, referred as lifelong learning, is also crucial to autonomous
agents when they interact with dynamic real-world environments with non-
stationary data and/or incremental tasks (Hadsell et al. 2020; Parisi et al.
2019), yet given finite resources, e.g., computational power or storage
capacity. To enable machines to possess such human-like lifelong learning
capabilities, it requires agents to be able to learn and accumulate knowledge
sequentially (Chen and Liu 2018). An ideal lifelong learning system is able
to: (i) continuously acquire new knowledge from new tasks (Lopez-Paz and
Ranzato 2017); (ii) preserve previously learned knowledge in old tasks, i.e.,
overcoming catastrophic forgetting (McCloskey and Cohen 1989; Kirkpatrick et
al. 2016; Pfülb and Gepperth 2019); (iii) transfer previous knowledge from the
memory to facilitate the learning of new tasks (Swaroop et al. 2019; Schwarz
et al. 2018); and (iv) use fewer parameters and storage (Pfülb and Gepperth
2019; Farquhar and Gal 2018).

A number of lifelong learning research have been conducted over the past
decades (Parisi et al. 2019; Chen and Liu 2018; McCloskey and Cohen 1989;
Aljundi et al. 2018; Goodfellow et al. 2014.5; Robins 1993). However, few
studies can achieve all above capabilities of an ideal lifelong learning
system. In particular, memory transfer has been long neglected. The
association between memory and learning remains difficult for current lifelong
learning models, leading to the anterograde forgetting. In contrast to the
catastrophic forgetting in which learning of new knowledge could eliminate old
knowledge quickly, the anterograde forgetting describes the phenomenon that
the old knowledge in the memory could prevent the learning of new knowledge.
This issue could occur in both memory retention and memory transfer. On the
one hand, retaining old knowledge continuously occupies model capacity left
for learning new knowledge. Such capacity shrinkage hinders models from
learning new knowledge adequately. On the other hand, transferring irrelevant
old knowledge introduces noise, which leads to conceptual confusions while
learning new knowledge. Our empirical study demonstrates these two scenarios
of the anterograde forgetting, as shown in Fig. 1, where performance
degradation of a lifelong learning model can be observed when it learns a new
task with old knowledge: (i) the model equipped with a memory consolidation
algorithm converges slowly and performs poorly compared to a model trained
without memory retention; and (ii) the model with a memory transfer algorithm
has a lower accuracy than that of the model without memory transfer. Note that
two pairs of models that are compared have the same experimental setting,
i.e., dataset, hyperparameters, and initial parameters, except for the
difference of models with and without memory consolidation and memory
transfer.

Figure 1: Two scenarios of the anterograde forgetting issue in neural networks
for lifelong learning. Preserving (left) and transferring (right) old
knowledge could prevent the model from learning of new knowledge. (Left): The
experiment was conducted on CIFAR-100 for the first two tasks. Models were
trained on the first task and tested on the second task. As a control,
$scratch\\_learning$ trained models from scratch without the influences of
anterograde forgetting. The $weights\\_consolidation$, i.e., EWC (Kirkpatrick
et al. 2016), is one weight consolidation method that adds a regularization
item to remember the previous task’s knowledge. (Right): Similarly, the
experiment was performed on CIFAR-10 for the first two tasks. The
$feature\\_transfer$, i.e., Progressive Neural Networks (Rusu et al. 2016), is
a feature transfer-based method. Compared with $w/o\\_transfer$, it transfers
features from the model of previous tasks to the current task’s learning.

Why are humans capable of learning from a sequence of tasks without
catastrophic and anterograde forgetting? Neuroscience studies suggest that a
short-term memory is first formed when learning a task and is subsequently
transformed into a long-term memory (Trannoy et al. 2011; Dudai, Karni, and
Born 2015; Izquierdo et al. 1999). Relatively, long-term memory retrieves and
reactivates similar memories and facilitates short-term memory formation
(Hulme, Maughan, and Brown 1991). Inspired by the above research, we introduce
the following principles for the development of effective lifelong learning
systems: (i) two individual networks for encoding new and old knowledge
separately to avoid conflicts, (ii) an explicit mechanism to integrate new
knowledge into old knowledge, and (iii) specialized modules to transfer old
knowledge to facilitate the learning of new knowledge.

This work proposes a general lifelong learning framework based on the above
principles, called Cycled Memory Networks (CMN), as shown in Fig. 2. It has a
memory-learning spiral loop: when the CMN learns a task, new knowledge will be
extracted and stored in a short-term memory network, followed by a memory
consolidation process which integrates the short-term memory into a long-term
memory network; when it continuously learns a new task in the future, a
transfer cell is developed to effectively transfer the knowledge in the long-
term memory network to assist learning new knowledge. Extensive experiments
are conducted to evaluate the proposed framework on several benchmarks with
known and unknown task relations. Experimental results show that the proposed
CMN can overcome both anterograde and catastrophic forgetting issues, and
achieve precise knowledge migration between long- and short-term memory
networks. Specifically, the contribution of this paper can be summarized as
follows,

  * •

The anterograde forgetting issue is first demonstrated in lifelong learning.

  * •

A biologically inspired lifelong learning framework is developed through an
association of memory and learning.

  * •

Two individual memory networks: short-term and long-term memory networks are
introduced to separately store new and old knowledge, and a transfer cell is
developed to learn the knowledge transferability from the long-term memory
network to the short-term memory network.

  * •

A memory consolidation mechanism is introduced to integrate the new knowledge
learned in the short-term memory network into the long-term memory network to
achieve an effective knowledge accumulation.

Figure 2: An overview of the proposed CMN. (Left): A memory-learning spiral
loop to enable knowledge transformation and transfer between long- and short-
term memory. (Right): the architecture of CMN, (a) L-S transfer half-cycle,
the process of long-term memory transferring to short-term memory via the
transfer cells; (b) S-L transform half-cycle, the process of short-term memory
transforming to the long-term memory via knowledge distillation.

## Related Works

Memory consolidation for lifelong learning. Lifelong learning, also known as
continual learning (Ring et al. 1994) or sequential learning (Aljundi,
Rohrbach, and Tuytelaars 2018), has been recently receiving increasing
attention in machine learning and artificial intelligence (Chen and Liu 2018).
The majority of previous works focused on addressing the well-known
catastrophic forgetting issue (McCloskey and Cohen 1989). According to the
mechanism of memory consolidation, current approaches are categorized into
three types: (i) Experiential rehearsal-based approaches, which focus on
replaying episodic memory (Robins 1995), and the core of which is to select
representative samples or features from historical data (Rebuffi et al. 2017;
Aljundi et al. 2019; Bang et al. 2021). (ii) Distributed memory representation
approaches (Fernando et al. 2017; Mallya and Lazebnik 2018), which allocate
individual networks for specific knowledge to avoid interference, represented
by Progressive Neural Networks (PNN) (Rusu et al. 2016). While such methods
can well-address the catastrophic forgetting issue, they face the parameter
explosion problem. (iii) Synaptic memory approaches, which optimize the
gradient toward the defined direction by regularization. (Kirkpatrick et al.
2016; Aljundi et al. 2018; Zenke, Poole, and Ganguli 2017; Pan et al. 2020)
selectively retained weights that are highly relevant to the previously
learned knowledge. (Zeng et al. 2019; Tang et al. 2021) kept the parameter
space of the preceding and following tasks orthogonal to avoid interference.
Such methods do not rely on storing additional data or parameters but face the
gradient error accumulation effect. Unlike the above mechanisms, we introduce
a new memory consolidation mechanism, i.e., learning knowledge through a
short-term memory network and then transferring the short-term memory to long-
term memory.

Memory transfer in lifelong learning. Unlike transfer learning, memory
transfer is concerned with preserving previously learned knowledge while
facilitating the learning of new knowledge. This makes traditional transfer
learning methods inapplicable in the lifelong learning setting (Ke, Liu, and
Huang 2020). Current work on memory transfer focus on distributed memory
representations for overcoming catastrophic forgetting, e.g., (Rusu et al.
2016) based on the transferability of features in neural networks, (Yosinski
et al. 2014) transferring features from previous models. (Rajasegaran et al.
2019) reused specific modules of the historical model and extends new modules
to learn new knowledge. (Abraham, Jones, and Glanzman 2019; Cong et al. 2020)
adapted old parameters to the new task by extra adaption networks.
Nevertheless, to the extent of our knowledge, no current approaches consider
adverse effects of memory transfer with a conflict between old and new
knowledge. In particular, these approaches ignore the hindrances of old
knowledge to the learning of new knowledge. In contrast, this work aims to
handle the anterograde forgetting issue through individual networks equipped
with transferring cells, and overcomes the catastrophic forgetting issue
through a hybrid strategy of synaptic memory and long- and short-term memory
transformation.

Distributed frameworks for lifelong learning. Several works start to focus on
distributed frameworks for lifelong learning (Delange et al. 2021), typically
represented by (Rusu et al. 2016), which assign a separate network to each
task. (Ostapenko et al. 2019; Rajasegaran et al. 2019; Yoon et al. 2018; Li et
al. 2019) proposed task-adaptive dynamic networks, and (Schwarz et al. 2018)
built memory-learning coupled frameworks to separately learn and store
knowledge. Unlike these approaches, our CMN does not allocates distinct sub-
networks for each task but encodes long- and short-term networks based on the
cycle of memory. Although (Kemker and Kanan 2018; Delange et al. 2021) also
introduced the concept of long- and short-term memory, we use a memory
consolidation process which distills and transfers condensed knowledge from
the short-term memory network to the long-term memory network.

Similar to our work, P$\&$C (Schwarz et al. 2018) distills new knowledge into
the knowledge base via two separate networks, but there are some differences
to be clarified, (i) CMN designs a novel module called transfer cell to
transfer memory precisely; in contrast, P$\&$C transfer memory by lateral
connection as the same as PNN, without considering the negative transfer; (ii)
CMN utilizes the function-based regularization to preserve memory rather than
the parameter-based one in P$\&$C.

Figure 3: Structure of the transfer cell.It consist of four branches:memory
processing, recall gate, and memory integration.

## Method

### Overview of Cycled Memory Architecture

Since the anterograde forgetting issue comes from two parts: (i) capacity
shrinkage due to memory retention and (ii) conceptual confusion due to
irrelevant memory migration, and motivated by the functional organization
(Alberini 2009; Izquierdo et al. 1999) in the mammalian brain(stated in
principle (i) and (ii), Introduction), we proposed the separate architecture
called CMN, which contains two sub-networks: a Long-Term Memory Network
(L-Net) and a Short-Term Memory network (S-Net), as shown in Fig. 2, to store
old knowledge and learn new knowledge, respectively. The S-Net is independent
of the L-Net to avoid the network’s capacity for the new task being occupied
by the previous task; the transfer cell learns to recall and extract relevant
features from the long-term memory to supplement the short-term memory(L-S
transfer half-cycle). Besides, a memory consolidation strategy(corresponding
to principle (iii)) is developed to allow the integration of new knowledge
learned in the short-term memory network into the long-term memory network
(S-L transform half-cycle). Combined with memory consolidation based on
knowledge distillation, CMN achieves the cycle of long-term memory - short
term memory - long term memory and constructs a sound framework for lifelong
learning (corresponding to lifelong learning requirements).

Denotation preview. For simplicity, we consider a sequence of $T$
classification tasks, in which the $k$-th task contains ${C^{k}}$ classes. We
aim to train a lifelong learning model which can learn the $k$-th task without
forgetting and being disturbed by the old knowledge learned in previous
${k-1}$ tasks. We set the two networks the same architecture except for the
output layer, which is not necessary. Specifically, when it learns the $k$-th
task, the output layer of L-Net contains $\sum_{i=1}^{k-1}C^{i}$ neurons for
predicting all previous $k-1$ tasks; accordingly, S-Net is only responsible
for learning the knowledge of the current task, so its output layer contains
only ${C^{k}}$ neurons. Given a training data set for the $k$-th task
$\mathcal{D}^{k}=\left\\{\textbf{x}_{i},\textbf{y}_{i}\right\\}$,
$i=1,2,\cdots,N_{k}$. We denote the weights of the L-Net for the previous
$k-1$ task as $\theta_{L}^{k-1}$ and the features of layer $l-1$ of the L-Net
as $\check{h}_{l-1}^{k-1}$. The weights of the S-Net are $\varphi_{S}^{k}$,
and the weights of S-Net’s layer $l$ are $W_{l}^{k}$. The features of S-Net’s
layer $l-1$ are $\tilde{h}_{l-1}^{k}$. The feature of L-Net operated by the
Recall Gate is $\vec{h}_{l-1}^{k-1}$. The confluence of the transferred
$\vec{h}_{l-1}^{k-1}$ and $\tilde{h}_{l-1}^{k}$ after the Integration module
is $\hat{h}_{l-1}^{k}$.

L-S transfer half-cycle. Fig. 2(a) shows the process of long-term memory
facilitates learning new tasks via transfer cells. Before training the S-Net,
CMN freezes L-Net’s weights $\theta_{L}^{k-1}$ and initializes the S-Net’s
weights $\varphi_{S}^{k}$. The features $\check{h}_{l-1}^{k-1}$ of the L-Net
flow into the S-Net through the transfer cell and merge with features
$\tilde{h}_{l-1}^{k}$ of S-Net. The S-Net is trained by adjusting
$\varphi_{S}^{k-1}$ only to minimize the following cross-entropy loss:

${\varphi_{S}^{k}}^{*}=\operatorname*{arg\,max}\limits_{\varphi_{S}^{k}}\left\\{-\frac{1}{N_{k}}\sum_{i=1}^{N_{k}}\mathbf{y}_{i}\cdot\log\left(f(\mathbf{x}_{i};\varphi_{S}^{k},\theta_{L}^{k-1})\right)\right\\}$
(1)

where $\cdot$ denotes the inner product and
$f(\mathbf{x}_{i};\varphi_{S}^{k},\theta_{L}^{k-1})$ denotes the output of
S-Net parameterized by $\varphi_{S}^{k}$ for a given input $\mathbf{x}_{i}$
with transferred features from L-Net parameterized by $\theta_{L}^{k-1}$. The
feature transfer from L-Net to S-Net is enabled by a transfer cell, which is
detailed in Section 3.2.

S-L transform half-cycle. Next, in the memory consolidation phase, shown in
Fig. 2(b), CMN freezes the weights $\varphi_{S}^{k}$ of S-Net and the L-Net’s
weights $\theta_{L}^{k-1}$ for previous $k-1$ tasks before training the L-Net.
To integrate the knowledge of historical knowledge and new knowledge, two
distillation losses are utilized. One is to preserve the memory of L-Net, and
the other is to transform the short memory to the L-Net. They adjusts
$\theta_{L}^{k-1}$ to ensure the distribution of the new L-Net consistent with
the one combining both old L-Net for previous $k-1$ tasks and new S-Net for
the current $k$-th task.

Table 1: Comparison of $AF$ on class-incremental and cross-domain learning $AF(\%)$ | tasks | One | Joint | fine | PNN | LwF | EWC | DGR | Hnet | CMN
---|---|---|---|---|---|---|---|---|---|---
CIFAR10 | 2 | \ | \ | $-23.51\pm 2.33$ | $\textbf{20.71}^{2}\pm 1.68$ | $-23.22\pm 1.76$ | $7.44\pm 2.54$ | $-15.86\pm 1.94$ | $\textbf{8.50}^{3}\pm 2.12$ | $\textbf{26.33}^{1}\pm 2.05$
5 | $-7.25\pm 1.10$ | $\textbf{18.14}^{2}\pm 3.13$ | $-64.59\pm 11.34$ | $-5.09\pm 2.22$ | $-55.53\pm 6.24$ | $\textbf{7.36}^{3}\pm 3.03$ | $\textbf{20.24}^{1}\pm 2.90$
CIFAR100 | 2 | \ | \ | $-21.02\pm 0.75$ | $\textbf{17.38}^{1}\pm 2.80$ | $\textbf{-9.31}^{3}\pm 0.87$ | $-31.71\pm 1.58$ | $-39.01\pm 13.18$ | $-73.66\pm 5.12$ | $\textbf{11.64}^{2}\pm 9.48$
5 | $-35.99\pm 1.10$ | $\textbf{22.70}^{1}\pm 2.26$ | $-46.87\pm 0.38$ | $\textbf{-9.88}^{3}\pm 1.35$ | $-40.45\pm 5.79$ | $-21.52\pm 2.75$ | $\textbf{10.45}^{2}\pm 0.69$
10 | $-32.95\pm 1.67$ | $\textbf{24.60}^{2}\pm 0.93$ | $-54.57\pm 11.32$ | $\textbf{-15.21}^{3}\pm 4.75$ | $-72.95\pm 7.52$ | $-37.88\pm 0.32$ | $\textbf{26.40}^{1}\pm 3.80$
20 | $-25.58\pm 0.63$ | $\textbf{29.31}^{2}\pm 1.21$ | $-75.29\pm 0.36$ | $\textbf{-18.99}^{3}\pm 2.77$ | $-94.87\pm 13.50$ | $-21.05\pm 0.30$ | $\textbf{30.39}^{1}\pm 0.43$
Cal/VOC | C-V | \ | \ | $\textbf{-12.85}^{3}\pm 3.45$ | $\textbf{13.61}^{2}\pm 1.48$ | $-45.06\pm 2.71$ | $-25.91\pm 2.02$ | $-49.91\pm 11.68$ | $-23.42\pm 5.81$ | $\textbf{18.70}^{1}\pm 4.57$
V-C | $\textbf{-9.70}^{3}\pm 0.54$ | $\textbf{15.66}^{2}\pm 3.96$ | $-55.98\pm 6.51$ | $-18.76\pm 1.51$ | $-24.14\pm 9.23$ | $-46.69\pm 0.91$ | $\textbf{17.96}^{1}\pm 2.47$

### Transfer Cell

The distinct transfer cell is deployed between the corresponding layers of the
L-Net and the S-Net (Fig. 3). It is consist of three primary modules: (i)
memory processing which maps the L-Net memory into the same space as S-Net,
(ii) retrieval gate, which produces a value ranging from 0 to 1 to control the
amount of knowledge to be transferred, and (iii) knowledge integration which
combines the S/L-Nets’ knowledge using multiplications and additions. In
summary, features of L-Net flow to the S-Net through the transfer cell, in the
order of Memory processing–Recall gate–Memory integration. For illustration,
assume that the S-Net learns the $k$-th task without transfer cell, the
activation of the $l$-th layer is formulated as:

$\tilde{h}_{l}^{k}=\textbf{ReLU}(W_{l}^{k}\tilde{h}_{l-1}^{k})$ (2)

where ReLU is the activation function.

Memory processing. It processes the feature $\check{h}_{l-1}^{k-1}$ of L-Net.
It functions in two ways: (1) the channel attention module (Wang et al. 2020),
denoted as $eca\\_layer_{l-1}^{k-1}$, is used to learn high-relevant features;
(2) the projection layer $P_{l-1}^{k-1}$, is used to map the output of
$eca\\_layer_{l-1}^{k-1}$ to the same space as the features of S-Net to
achieve alignment. The output $\vec{h}_{l-1}^{k-1}$ is calculated by

$\vec{h}_{l-1}^{k-1}=\textbf{ReLU}(P_{l-1}^{k-1}eca\\_layer_{l-1}^{k-1}(\bar{h}_{l-1}^{k-1}))$
(3)

Recall Gate. It calculates the recall gate $g_{l-1}^{k-1}$, controlling the
knowledge from the long-term memory network to be recalled for short-term
memory formation. The recall gate embeds the features extracted from L-Net and
S-Net into a common low-dimensional space, and learns the feature
transferability between long- and short-term memories. $g_{l-1}^{k-1}$ can be
calculated by,

$g_{l-1}^{k-1}=\sigma(\bar{\textbf{E}}_{l-1}^{k-1}{GAP}(\check{h}_{l-1}^{k-1})+\tilde{\textbf{E}}_{l-1}^{k}{GAP}(\tilde{h}_{l-1}^{k})+\textbf{b}_{l-1}^{k-1:k})$
(4)

where $\sigma(\cdot)$ is the sigmoid function used to normalize the values to
$\left(0,1\right)$ . $\bar{\textbf{E}}_{l-1}^{k-1}$ and
$\tilde{\textbf{E}}_{l-1}^{k}$ are linear projection layers to transform
features to a common subspace, corresponding to L-Net and S-Net.
$\textbf{b}_{l-1}^{k-1:k}$ is bias item. In addition, to reduce the
computational cost of the linear layer, the Global Average Pooling (GAP) is
utilized to reduce the dimension of the features $\check{h}_{l-1}^{k-1}$ and
$\tilde{h}_{l-1}^{k}$.

Memory integration. It recalls the processing feature $\vec{h}_{l-1}^{k-1}$
from long-term memory combining with the recall gate, and integrates it with
feature $\tilde{h}_{l-1}^{k}$ of short-term memory. It calculates
$\hat{h}_{l-1}^{k}$ by

$\hat{h}_{l-1}^{k}=\tilde{h}_{l-1}^{k}+g_{l-1}^{k-1}\otimes\vec{h}_{l-1}^{k-1}$
(5)

where $g_{l-1}^{k-1}$ is the recall gate calculated by the memory recall
branch, $\vec{h}_{l-1}^{k-1}$ is the post-processing on the original feature
$\check{h}_{l-1}^{k-1}$ via the memory processing branch, and $\otimes$ is the
element-wise multiplication operator.

Correspondingly, the activation in the $l$-th layer of S-Net with transfer
cell is processed by

$\tilde{h}_{l}^{k}=\textbf{ReLU}(W_{l}^{k}\hat{h}_{l-1}^{k})$ (6)

where $\hat{h}_{l-1}^{k}$ is the output of memory integration branch.

In particular, we found the initialization of the linear layer is significant
and best initialization is constant 1 (details in Fig. 13, Appendix).
Moreover, the results in Fig. 12(Appendix) demonstrate that using Batch
Normalization (Ioffe and Szegedy 2015) does not improve the speed and accuracy
of training, and even has a negative impact. We also discuss the transfer
strategy in Fig. 14, Appendix, and exhibit that the transfer cell based
transfer strategy is the most effective method on the case of negative
transfer and positive transfer.

### Memory Consolidation

The memory consolidation uses two knowledge distillation losses to preserve
and integrate long- and short-term memories. To preserve the long-term memory,
we first want to keep the output distribution of new L-Net $\theta_{L}^{k}$
and the output distribution of previous L-Net $\theta_{L}^{k-1}$ consistent in
the same dimension on $\mathcal{D}^{k}$. The objective function is defined as

$L_{dis\\_long}^{k}=cross\\_entropy(\sigma(\frac{f(\textbf{X}^{k};\theta_{L}^{k})}{T}),\sigma(\frac{f(\textbf{X}^{k};\theta_{L}^{k-1})}{T}))$
(7)

where $T$ is a hyper-parameter to control the concentration of the output
distribution; the larger $T$ is, the more divergent the distribution is, and
vice versa.

For integrating the short-term memory into the L-Net, similar to (Hinton,
Vinyals, and Dean 2015), we define a hybrid objective function as

$\displaystyle
L_{dis\\_short}^{k}=\left(1-\beta\right)cross\\_entropy\left(\sigma(f(\textbf{X}^{k};\theta_{L}^{k})),\textbf{Y}^{k}\right)$
(8) $\displaystyle+\beta
T^{2}cross\\_entropy\left(\sigma(\frac{f(\textbf{X}^{k};\theta_{L}^{k})}{T}),\sigma(\frac{f(\textbf{X}^{k};\varphi_{L}^{k})}{T})\right)$

where $\beta$ is a hyper-parameter to control the degree of knowledge
integration from S-Net. The larger value means more knowledge of S-Net is
integrated in the new L-Net.

In order to integrate S-Net and L-Net, the following total objective function
is minimized to update L-Net:

$L_{distill}^{k}=L_{dis\\_long}^{k}+L_{dis\\_short}^{k}$ (9)

## Experiments and Results

### Experimental Settings

Overview of experiments, dataset, models, and training details. We implemented
two experimental scenarios to investigate (1) the case of task relations
known, including two situations. a. Task conflict, i.e., sequentially learn
noise data and CIFAR-10 (Krizhevsky et al. 2009), to evaluate the ability to
resist irrelevant knowledge, and b. excluding the above case, i.e.,
sequentially learn CIFAR-10 and STL-10 (Coates, Ng, and Lee 2011), to evaluate
the ability to resist capacity shrinkage. (2) The case of task relations
unknown, including class-incremental learning on CIFAR-100 (Krizhevsky et al.
2009) and cross-domain sequential learning on PASCAL VOC2007 (Everingham et
al. 2010) and Caltech-101 (Fei-Fei, Fergus, and Perona 2006), to evaluate the
overall performance of continual learning. In addition, the experiments in
Fig. 1 were re-conducted to verify CMN’s ability of overcoming anterograde
forgetting. Furthermore, in Appendix, we describe (i) the data and pre-
processing in Section Details of dataset and pre-processing; (ii) Basic
classification networks and (iii) Training details.

Metrics. For a comprehensive assessment of the algorithm’s performance to
overcome anterograde forgetting, we designed a new metric called $AF$(details
in appendix), which evaluates the cumulative average of the performance
decline of a single task in the task sequence during continual learning. It
summarizes the cumulative accuracy gap of the continual learning model on the
current task relative to a single-task model learning the same task and the
cumulative accuracy gap of the continual learning model across all seen tasks
to a multi-task model learning the same tasks jointly.

Furthermore, we utilized the evaluation criteria of (Lopez-Paz and Ranzato
2017). Specifically, the average accuracy ($ACC$) was used to evaluate the
overall performance of the approach on a sequence of tasks. The forward
transfer ($FWT$) was used to evaluate the ability of the approach to transfer
memorized knowledge to facilitate the learning of new tasks. Moreover, the
backward transfer ($BWT$) was utilized to evaluate the ability to overcome
forgetting previous tasks’ knowledge. In addition, we utilized the $Params$ to
evaluate the size of the model parameter and designed the  Iteration time  to
evaluate the data storage (details in Table. 5, Appendix).

Figure 4: Performance on task-related(left) and task-conflict(right). left:
Results of transferring knowledge to STL-10, and right: results of
transferring noise to CIFAR-10. Figure 5: Ablation analysis for S-Net and
transfer cells on $AF$, $ACC$, and $FWT$. Figure 6: Robust analysis of hyper-
parameters $T$ and $\beta$. Figure 7: Visualization of features on conv1 and
four blocks for CMN, One, and EWC.

Baseline. We compare our proposed CMN with state of the art approaches,
including experience rehearsal-based method: DGR (Shin et al. 2017);
distributed memory-based: PNN (Rusu et al. 2016), hnet (von Oswald et al.
2020); and synaptic memory-based method: EWC (Kirkpatrick et al. 2016), LwF
(Li and Hoiem 2017). We also compared with some traditional methods, including
One (individual training model for each task), Joint (jointly training all
tasks at once), fine-tuning (fine-tuning on the previous model without
constraints).

### Results

Class-incremental and cross-domain. We first repeated the experiments in Fig.
1 with CMN. In Fig. 8(Appendix), results show that CMN can significantly
reduce anterograde forgetting. In memory retention (left), CMN’s S-Net
undergoes almost no anterograde forgetting because its network capacity for
the new task is constant; although CMN’s L-Net cannot wholly overcome
anterograde forgetting, it still dramatically slows down anterograde
forgetting compared to the EWC algorithm. The results on memory transfer
(right) are also consistent with our findings.

We then compared our CMN with the above methods on standard tasks, including
class-incremental and cross-domain, in Table 1. The average results running
five times of operation demonstrate that CMN achieves the highest score on
$AF$ in most cases, significantly better than other algorithms. It achieves
the second score in some cases. In Table 2 and Table 3, Appendix, we further
compared the performance on $ACC$ and $FWT$, which partly evaluate algorithms’
ability to address the two cases of anterograde forgetting. The results
demonstrate that CMN performs closely to PNN in several cases and even
outperforms PNN and other approaches on both $ACC$ and $FWT$ when learning
long sequence of tasks. Although PNN performs close to our algorithm and even
outperforms CMN in a few cases, it faces the parameter explosion problem. As
shown in Table 5 (Appendix), PNN has a significantly higher number of
parameters than the other algorithms. Moreover, its parameter size increases
rapidly with the number of tasks (Fig. 9 in Appendix). In Fig. 10 (Appendix),
we exhibit PNN’s performance declines rapidly with a constant number of
parameters on $AF$, $ACC$, $BWT$. In contrast, CMN uses only 1/5 of the
testing parameters and 1/2 training parameters of PNN, but achieves or even
outperforms PNN in the case of a fixed number of parameters. Note that CMN’s
parameters are constant instead of dramatically increasing with tasks’ length
like PNN. In addition, compared the Iteration time with the baseline, CMN
costs less time compared with DGR but achieves better performance.

We future found that CMN may benefit from the separate network architecture.
The confusion matrix in Fig. 11 (Appendix) exhibits that CMN preserves a
higher accuracy on the earlier learned categories than EWC when learning
categories increase. The quantitative result in Table 4 (Appendix) supports
this conclusion. CMN achieves a dramatic score on $BWT$, closing to the usual
upper limit PNN.

Task-related and task-conflict. Fig. 4(left) compares these methods in task-
related cases, where a learning model is sequentially trained on CIFAR-10 and
STL-10. On $AF$, PNN and CMN obtain the best results compared to fine-tuning,
a typical transfer learning approach. The fine-tuning, PNN, and CMN exceed One
by around 15%. Fig. 4(right) compares the performance of these methods in
resisting irrelevant knowledge transfer. We assumed the extreme case of
transferring knowledge from noisy data to the learning of CIFAR-10. The
results show that CMN is the only method among these methods to address the
negative migration effect of irrelevant memory. CMN ranks first on $AF$ and is
the only method with a positive value on $FWT$. Compared to CMN, all other
baselines on the new task are disturbed by the introduction of noise.

### Analysis

Component ablation analysis. To investigate the contribution of each component
of the CMN, we performed ablation experiments on CIFAR-100 with the first two
tasks for two crucial components: the use of short-term memory network and the
transfer cell. Fig. 5 presents the results of algorithms on $AF$, $ACC$, and
$FWT$ after removing different components. It indicates that both short-time
memory network and transfer cells enhance the model’s performance, but the
former contributes more. The model’s performance sharply drops when removing
the short-term memory network, even below One, i.e.,11.0% for $AF$, 3.2% for
$ACC$, and 7.8% for $FWT$. The model’s performance decreases when removing the
transfer cell but is still higher than One, i.e., $AF$ decreases by 2.5%,
$ACC$ decreases by 0.5%, and $FWT$ decreases by 2.2%.

Stable to hyper-parameters. Fig.6 investigates the proposed approach’s
stability for two significant hyper-parameters, beta distill $\beta$ and
temperature $T$, in various ranges. The $\beta$ controls the amount of
knowledge transferred from S-Let to L-Net, and the larger its value, the more
knowledge is distilled from S-Net into L-Net. The $T$ controls the temperature
of knowledge distillation, and the larger its value, the more divergent the
distribution of distilled soft label is. The two parameters collaborate to
reach a balance between the old and new memories. The results indicate that
the model is tolerant to $T$. The model’s performance increases with the
increase of $T$. In contrast, the model is more sensitive to the value of
$\beta$, which reaches the best performance at a value of 0.6. Overall, it is
acceptable, the $AF$ floats within 3% when $\beta$ varies in the range of
$\left[0.1,1\right]$.

Feature learning visualization. We analyzed the features of various layers of
networks on Caltech-101 and compared them with One and EWC by Grad-CAM
(Selvaraju et al. 2017). In Fig. 7, compared with One and EWC, CMN grasps more
representative characteristics, especially on the higher level. For example,
on block 4, S-Net and L-Net capture features in a few critical local regions
to represent the concept of a motorcycle. In contrast, One learns fewer local
vital features, and EWC learns features of the most discrete areas. S-Net
learns richer features than L-Net, mainly because L-Net selectively integrates
features as it needs to balance old and new knowledge when integrating short-
term memory. Furthermore, this phenomenon is not coincidental, and we obtained
consistent findings on other samples, as shown in Fig. 15(Appendix).

## Conclusion

Learning and memory are the core of lifelong learning, one essential step
towards an open world intelligence. However, in the past years, we ignored the
interference effect of memory on learning, i.e., anterograde forgetting issue.
Inspired by the human brain’s memory mechanisms, this paper suggests the
designing principle for the functional architecture of lifelong learning. In
particular, we propose a cycled memory network to overcome the anterograde
forgetting. It transforms short-term memory to long-term memory through a
memory consolidation mechanism and transfers long-term memory to learning
through transfer cells. Comprehensive experiments have been conducted to
verify the effectiveness of CMN. While our results show that using two
individual short-term and long-term memory networks can effectively address
the anterograde forgetting, this architecture could be further explored.
Besides, the proposed framework focuses on the classification task and does
not consider the network’s evolution strategy when long-time memory is
saturated. It is of great significance to explore ways to address these issues
by drawing on the brain’s memory mechanisms. In future work, we will explore
the properties and corresponding structures of different memories and the
engram of memory storage, such as meta-knowledge. In addition, we will
consider integrating the mechanisms about synaptic memory and sleep
consolidation memory in neuroscience into the proposed memory-learning
framework.

## References

  * Abraham, Jones, and Glanzman (2019) Abraham, W. C.; Jones, O. D.; and Glanzman, D. L. 2019.  Is plasticity of synapses the mechanism of long-term memory storage?  _NPJ science of learning_ , 4(1): 1–10.
  * Alberini (2009) Alberini, C. M. 2009.  Transcription factors in long-term memory and synaptic plasticity.  _Physiological reviews_ , 89(1): 121–145.
  * Aljundi et al. (2018) Aljundi, R.; Babiloni, F.; Elhoseiny, M.; Rohrbach, M.; and Tuytelaars, T. 2018\.  Memory aware synapses: Learning what (not) to forget.  In _Proceedings of the European Conference on Computer Vision_ , 139–154.
  * Aljundi et al. (2019) Aljundi, R.; Lin, M.; Goujaud, B.; and Bengio, Y. 2019.  Gradient based sample selection for online continual learning.  _arXiv preprint arXiv:1903.08671_.
  * Aljundi, Rohrbach, and Tuytelaars (2018) Aljundi, R.; Rohrbach, M.; and Tuytelaars, T. 2018.  Selfless sequential learning.  _arXiv preprint arXiv:1806.05421_.
  * Bang et al. (2021) Bang, J.; Kim, H.; Yoo, Y.; Ha, J.-W.; and Choi, J. 2021.  Rainbow Memory: Continual Learning with a Memory of Diverse Samples.  In _CVPR_.
  * Chen and Liu (2018) Chen, Z.; and Liu, B. 2018.  Lifelong machine learning.  _Synthesis Lectures on Artificial Intelligence and Machine Learning_ , 12(3): 1–207.
  * Coates, Ng, and Lee (2011) Coates, A.; Ng, A.; and Lee, H. 2011.  An analysis of single-layer networks in unsupervised feature learning.  In _Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics_ , 215–223.
  * Cong et al. (2020) Cong, Y.; Zhao, M.; Li, J.; Wang, S.; and Carin, L. 2020.  GAN Memory with No Forgetting.  _Advances in Neural Information Processing Systems_ , 33.
  * Delange et al. (2021) Delange, M.; Aljundi, R.; Masana, M.; Parisot, S.; Jia, X.; Leonardis, A.; Slabaugh, G.; and Tuytelaars, T. 2021.  A continual learning survey: Defying forgetting in classification tasks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
  * Dudai, Karni, and Born (2015) Dudai, Y.; Karni, A.; and Born, J. 2015.  The consolidation and transformation of memory.  _Neuron_ , 88(1): 20–32.
  * Everingham et al. (2010) Everingham, M.; Van Gool, L.; Williams, C. K.; Winn, J.; and Zisserman, A. 2010\.  The pascal visual object classes (voc) challenge.  _International journal of computer vision_ , 88(2): 303–338.
  * Farquhar and Gal (2018) Farquhar, S.; and Gal, Y. 2018.  Towards robust evaluations of continual learning.  _arXiv preprint arXiv:1805.09733_.
  * Fei-Fei, Fergus, and Perona (2006) Fei-Fei, L.; Fergus, R.; and Perona, P. 2006.  One-shot learning of object categories.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 28(4): 594–611.
  * Fernando et al. (2017) Fernando, C.; Banarse, D.; Blundell, C.; Zwols, Y.; Ha, D.; Rusu, A. A.; Pritzel, A.; and Wierstra, D. 2017.  Pathnet: Evolution channels gradient descent in super neural networks.  _arXiv preprint arXiv:1701.08734_.
  * Goodfellow et al. (2014.5) Goodfellow, I. J.; Mirza, M.; Xiao, D.; Courville, A.; and Bengio, Y. 2014.5.  An empirical investigation of catastrophic forgetting in gradient-based neural networks.  _International Conference on Learning Representations_.
  * Hadsell et al. (2020) Hadsell, R.; Rao, D.; Rusu, A. A.; and Pascanu, R. 2020.  Embracing Change: Continual Learning in Deep Neural Networks.  _Trends in Cognitive Sciences_.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 770–778.
  * Hinton, Vinyals, and Dean (2015) Hinton, G.; Vinyals, O.; and Dean, J. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_.
  * Hulme, Maughan, and Brown (1991) Hulme, C.; Maughan, S.; and Brown, G. D. 1991.  Memory for familiar and unfamiliar words: Evidence for a long-term memory contribution to short-term memory span.  _Journal of memory and language_ , 30(6): 685–701.
  * Ioffe and Szegedy (2015) Ioffe, S.; and Szegedy, C. 2015.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _International conference on machine learning_ , 448–456. PMLR.
  * Izquierdo et al. (1999) Izquierdo, I.; Medina, J. H.; Vianna, M. R.; Izquierdo, L. A.; and Barros, D. M. 1999.  Separate mechanisms for short-and long-term memory.  _Behavioural brain research_ , 103(1): 1–11.
  * Ke, Liu, and Huang (2020) Ke, Z.; Liu, B.; and Huang, X. 2020.  Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks.  _Advances in Neural Information Processing Systems_ , 33.
  * Kemker and Kanan (2018) Kemker, R.; and Kanan, C. 2018.  FearNet: Brain-Inspired Model for Incremental Learning.  In _International Conference on Learning Representations_.
  * Kirkpatrick et al. (2016) Kirkpatrick, J.; Pascanu, R.; Rabinowitz, N.; Veness, J.; Desjardins, G.; Rusu, A. A.; Milan, K.; Quan, J.; Ramalho, T.; and Grabskabarwinska, A. 2016.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the National Academy of Sciences of the United States of America_ , 114(13): 3521.
  * Krizhevsky et al. (2009) Krizhevsky, A.; et al. 2009.  Learning multiple layers of features from tiny images.  _Computer Science Department, University of Toronto,Tech. Rep_.
  * Li et al. (2019) Li, X.; Zhou, Y.; Wu, T.; Socher, R.; and Xiong, C. 2019.  Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting.  In _International Conference on Machine Learning_ , 3925–3934. PMLR.
  * Li and Hoiem (2017) Li, Z.; and Hoiem, D. 2017.  Learning without forgetting.  _IEEE transactions on pattern analysis and machine intelligence_ , 40(12): 2935–2947.
  * Lopez-Paz and Ranzato (2017) Lopez-Paz, D.; and Ranzato, M. 2017.  Gradient episodic memory for continual learning.  _Advances in neural information processing systems_ , 30: 6467–6476.
  * Mallya and Lazebnik (2018) Mallya, A.; and Lazebnik, S. 2018.  Packnet: Adding multiple tasks to a single network by iterative pruning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 7765–7773.
  * McCloskey and Cohen (1989) McCloskey, M.; and Cohen, N. J. 1989.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of Learning and Motivation_ , volume 24, 109–165. Elsevier.
  * Ostapenko et al. (2019) Ostapenko, O.; Puscas, M.; Klein, T.; Jahnichen, P.; and Nabi, M. 2019.  Learning to remember: A synaptic plasticity driven framework for continual learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 11321–11329.
  * Pan et al. (2020) Pan, P.; Swaroop, S.; Immer, A.; Eschenhagen, R.; Turner, R. E.; and Khan, M. E. 2020.  Continual deep learning by functional regularisation of memorable past.  _arXiv preprint arXiv:2004.14070_.
  * Parisi et al. (2019) Parisi, G. I.; Kemker, R.; Part, J. L.; Kanan, C.; and Wermter, S. 2019.  Continual lifelong learning with neural networks: A review.  _Neural Networks_ , 113: 54–71.
  * Pfülb and Gepperth (2019) Pfülb, B.; and Gepperth, A. 2019.  A comprehensive, application-oriented study of catastrophic forgetting in dnns.  _arXiv preprint arXiv:1905.08101_.
  * Rajasegaran et al. (2019) Rajasegaran, J.; Hayat, M.; Khan, S.; Khan, F. S.; and Shao, L. 2019.  Random path selection for incremental learning.  _Advances in Neural Information Processing Systems_.
  * Rebuffi et al. (2017) Rebuffi, S.-A.; Kolesnikov, A.; Sperl, G.; and Lampert, C. H. 2017.  icarl: Incremental classifier and representation learning.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2001–2010.
  * Ring et al. (1994) Ring, M. B.; et al. 1994.  _Continual learning in reinforcement environments_.  Ph.D. thesis, University of Texas at Austin Austin, Texas 78712.
  * Robins (1993) Robins, A. 1993.  Catastrophic forgetting in neural networks: the role of rehearsal mechanisms.  In _Proceedings 1993 The First New Zealand International Two-Stream Conference on Artificial Neural Networks and Expert Systems_ , 65–68. IEEE.
  * Robins (1995) Robins, A. 1995.  Catastrophic forgetting, rehearsal and pseudorehearsal.  _Connection Science_ , 7(2): 123–146.
  * Rusu et al. (2016) Rusu, A. A.; Rabinowitz, N. C.; Desjardins, G.; Soyer, H.; Kirkpatrick, J.; Kavukcuoglu, K.; Pascanu, R.; and Hadsell, R. 2016.  Progressive neural networks.  _arXiv preprint arXiv:1606.04671_.
  * Schwarz et al. (2018) Schwarz, J.; Czarnecki, W.; Luketina, J.; Grabska-Barwinska, A.; Teh, Y. W.; Pascanu, R.; and Hadsell, R. 2018.  Progress & compress: A scalable framework for continual learning.  In _International Conference on Machine Learning_ , 4528–4537. PMLR.
  * Selvaraju et al. (2017) Selvaraju, R. R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; and Batra, D. 2017.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , 618–626.
  * Shin et al. (2017) Shin, H.; Lee, J. K.; Kim, J.; and Kim, J. 2017.  Continual learning with deep generative replay.  _arXiv preprint arXiv:1705.08690_.
  * Swaroop et al. (2019) Swaroop, S.; Nguyen, C. V.; Bui, T. D.; and Turner, R. E. 2019.  Improving and understanding variational continual learning.  _arXiv preprint arXiv:1905.02099_.
  * Tang et al. (2021) Tang, S.; Chen, D.; Zhu, J.; Yu, S.; and Ouyang, W. 2021.  Layerwise optimization by gradient decomposition for continual learning.  _arXiv preprint arXiv:2105.07561_.
  * Trannoy et al. (2011) Trannoy, S.; Redt-Clouet, C.; Dura, J.-M.; and Preat, T. 2011.  Parallel processing of appetitive short-and long-term memories in Drosophila.  _Current Biology_ , 21(19): 1647–1653.
  * von Oswald et al. (2020) von Oswald, J.; Henning, C.; Sacramento, J.; and Grewe, B. F. 2020.  Continual learning with hypernetworks.  In _International Conference on Learning Representations_.
  * Wang et al. (2020) Wang, Q.; Wu, B.; Zhu, P.; Li, P.; Zuo, W.; and Hu, Q. 2020.  ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks, 2020 IEEE.  In _CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE_.
  * Yoon et al. (2018) Yoon, J.; Yang, E.; Lee, J.; and Hwang, S. J. 2018.  Lifelong Learning with Dynamically Expandable Networks.  ICLR.
  * Yosinski et al. (2014) Yosinski, J.; Clune, J.; Bengio, Y.; and Lipson, H. 2014.  How transferable are features in deep neural networks?  In Ghahramani, Z.; Welling, M.; Cortes, C.; Lawrence, N.; and Weinberger, K., eds., _Advances in Neural Information Processing Systems 27 (NIPS ’14)_ , 3320–3328. Curran Associates, Inc.
  * Zeng et al. (2019) Zeng, G.; Chen, Y.; Cui, B.; and Yu, S. 2019.  Continual learning of context-dependent processing in neural networks.  _Nature Machine Intelligence_ , 1(8): 364–372.
  * Zenke, Poole, and Ganguli (2017) Zenke, F.; Poole, B.; and Ganguli, S. 2017.  Continual learning through synaptic intelligence.  In _International Conference on Machine Learning_ , 3987–3995. PMLR.

##  Appendix A Basic classification networks

Considering the complexity of tasks, we utilized the resnet-34 (He et al.
2016) in the cross-domain experiments considering, and resnet-18 (He et al.
2016) in other experiments. Besides, we utilized similar architectures for
L-Net and S-Net, except for the different units in the output layer.

##  Appendix B Training details

All experiments used the optimization strategy of stochastic gradient descent
with an initialized learning rate $\left\\{1,0.1,0.001\right\\}$. The momentum
was 0.9, and weight decay was $1\times 10^{-5}$. The grid search was utilized
to get the optimal result. The training epoch was 100. Besides, all models’
weights were initialized with $kaiming\\_uniform$. The mini-batch size was set
to 54 in the experiment of cross-domain and set to 1024 in other experiments.
In Appendix, we details the hyper-parameters setting and computational
resources in Table. 6 and Computational resources.

Figure 8: Overcome anterograde forgetting with CMN corresponding to the experiments in Fig. 1. Figure 9: Comparison of testing and training parameter size with tasks increase. left, testing Params comparison and right, training Params comparison Figure 10: Comparison between CMN and PNN under identical parameter size setting Table 2: Comparison of $ACC$ on class-incremental and cross-domain learning $ACC(\%)$ | tasks | One | Joint | fine | PNN | LwF | EWC | DGR | Hnet | CMN
---|---|---|---|---|---|---|---|---|---|---
CIFAR10 | 2 | $\textbf{88.6}^{3}\pm 2.31$ | $68.9\pm 0.7$ | $47.91\pm 0.22$ | $\textbf{89.19}^{2}\pm 0.32$ | $61.68\pm 4.30$ | $75.75\pm 0.62$ | $64.60\pm 4.24$ | $87.58\pm 0.60$ | $\textbf{91.20 }^{1}\pm 0.14$
5 | $\textbf{92.8}^{3}\pm 4.60$ | $67.23\pm 0.41$ | $\textbf{93.30}^{2}\pm 0.19$ | $34.38\pm 3.14$ | $64.78\pm 6.00$ | $25.06\pm 2.56$ | $88.56\pm 3.17$ | $\textbf{94.43}^{1}\pm 0.30$
CIFAR100 | 2 | $\textbf{68.9}^{3}\pm 0.54$ | $57.0\pm 0.7$ | $35.45\pm 0.24$ | $\textbf{71.53}^{1}\pm 1.45$ | $55.63\pm 0.35$ | $37.60\pm 1.30$ | $47.64\pm 3.58$ | $28.61\pm 1.02$ | $\textbf{69.30}^{2}\pm 0.33$
5 | $\textbf{62.5}^{3}\pm 3.58$ | $15.63\pm 0.53$ | $\textbf{70.51}^{1}\pm 0.48$ | $27.18\pm 0.52$ | $38.02\pm 0.48$ | $16.1\pm 2.28$ | $49.55\pm 1.04$ | $\textbf{63.87}^{2}\pm 0.24$
10 | $\textbf{75.2}^{3}\pm 1.50$ |  | $16.82\pm 0.36$ | $\textbf{78.00}^{1}\pm 2.02$ | $25.01\pm 5.02$ | $37.32\pm 4.78$ | $7.37\pm 0.67$ | $43.77\pm 0.34$ | $\textbf{77.86}^{2}\pm 3.46$
20 | $\textbf{84.3}^{1}\pm 0.21$ |  | $21.62\pm 0.26$ | $\textbf{82.33}^{2}\pm 0.38$ | $16.46\pm 0.28$ | $37.05\pm 6.38$ | $2.90\pm 0.52$ | $53.56\pm 0.35$ | $\textbf{82.29}^{3}\pm 0.37$
Cal/VOC | C-V | $\textbf{45.0}^{3}\pm 4.34$ | $43.0\pm 0.3$ | $22.26\pm 0.56$ | $\textbf{49.75}^{1}\pm 0.90$ | $30.64\pm 0.18$ | $17.98\pm 0.32$ | $11.25\pm 3.79$ | $25.59\pm 6.21$ | $\textbf{45.90 }^{2}\pm 4.34$
V-C | $32.61\pm 0.53$ | $\textbf{55.58}^{1}\pm 2.86$ | $29.53\pm 3.61$ | $28.86\pm 0.32$ | $27.24\pm 6.61$ | $20.77\pm 0.24$ | $\textbf{53.98}^{2}\pm 2.58$
Table 3: Comparison of $FWT$ on class-incremental and cross-domain learning $FWT(\%)$ | tasks | One | Joint | fine | PNN | LwF | EWC | DGR | Hnet | CMN
---|---|---|---|---|---|---|---|---|---|---
CIFAR10 | 2 | 0 | \ | $-2.55\pm 1.78$ | $\textbf{0.40}^{3}\pm 1.46$ | $-11.40\pm 4.71$ | $\textbf{0.56 }^{2}\pm 1.67$ | $-11.59\pm 3.09$ | $-10.21\pm 1.73$ | $\textbf{4.0}^{1}\pm 1.77$
5 | 0 | $-2.18\pm 1.08$ | $\textbf{2.35}^{2}\pm 5.19$ | $-27.84\pm 10.94$ | $\textbf{0.75}^{3}\pm 3.40$ | $-17.93\pm 6.89$ | $-4.41\pm 2.77$ | $\textbf{3.77}^{1}\pm 5.15$
CIFAR100 | 2 | 0 | \ | $\textbf{0.53}^{3}\pm 0.78$ | $\textbf{2.85}^{2}\pm 1.73$ | $-7.94\pm 0.36$ | $-12.3\pm 0.90$ | $-29.64\pm 9.95$ | $-41.83\pm 2.05$ | $\textbf{4.50}^{1}\pm 1.91$
5 | 0 | $-0.41\pm 0.32$ | $\textbf{12.51}^{1}\pm 4.32$ | $-23.03\pm 0.41$ | $\textbf{3.69}^{3}\pm 1.16$ | $-14.50\pm 4.88$ | $-11.28\pm 0.97$ | $\textbf{7.93}^{2}\pm 0.18$
10 | 0 | $-0.038\pm 0.48$ | $\textbf{4.62}^{2}\pm 0.34$ | $-34.41\pm 7.73$ | $\textbf{0.32}^{3}\pm 3.40$ | $-36.53\pm 4.59$ | $-26.16\pm 0.16$ | $\textbf{8.44}^{1}\pm 0.94$
20 | 0 | $\textbf{0.006}^{3}\pm 0.34$ | $\textbf{2.18}^{2}\pm 0.27$ | $-49.65\pm 0.36$ | $-3.13\pm 0.97$ | $-51.31\pm 15.85$ | $-22.00\pm 0.17$ | $\textbf{4.68}^{1}\pm 0.30$
Cal/VOC | C-V | 0 | \ | $\textbf{7.92}^{2}\pm 2.94$ | $\textbf{6.88}^{3}\pm 1.72$ | $-32.67\pm 2.56$ | $-0.87\pm 2.08$ | $-18.14\pm 8.33$ | $-5.99\pm 7.08$ | $\textbf{16.22}^{1}\pm 3.63$
V-C | $\textbf{0.61}^{3}\pm 0.38$ | $\textbf{3.00}^{2}\pm 1.02$ | $-42.59\pm 2.75$ | $-4.69\pm 0.70$ | $-8.35\pm 5.90$ | $-24.55\pm 0.79$ | $\textbf{6.90}^{1}\pm 0.54$
Table 4: Comparison of $BWT$ on class-incremental and cross-domain learning $AF(\%)$ | tasks | One | Joint | fine | PNN | LwF | EWC | DGR | Hnet | CMN
---|---|---|---|---|---|---|---|---|---|---
CIFAR10 | 2 | \ | \ | $-23.51\pm 2.33$ | $\textbf{20.71}^{2}\pm 1.68$ | $-23.22\pm 1.76$ | $7.44\pm 2.54$ | $-15.86\pm 1.94$ | $\textbf{8.50}^{3}\pm 2.12$ | $\textbf{26.33}^{1}\pm 2.05$
5 | $-7.25\pm 1.10$ | $\textbf{18.14}^{2}\pm 3.13$ | $-64.59\pm 11.34$ | $-5.09\pm 2.22$ | $-55.53\pm 6.24$ | $\textbf{7.36}^{3}\pm 3.03$ | $\textbf{20.24}^{1}\pm 2.90$
CIFAR100 | 2 | \ | \ | $-21.02\pm 0.75$ | $\textbf{17.38}^{1}\pm 2.80$ | $\textbf{-9.31}^{3}\pm 0.87$ | $-31.71\pm 1.58$ | $-39.01\pm 13.18$ | $-73.66\pm 5.12$ | $\textbf{11.64}^{2}\pm 9.48$
5 | $-35.99\pm 1.10$ | $\textbf{22.70}^{1}\pm 2.26$ | $-46.87\pm 0.38$ | $\textbf{-9.88}^{3}\pm 1.35$ | $-40.45\pm 5.79$ | $-21.52\pm 2.75$ | $\textbf{10.45}^{2}\pm 0.69$
10 | $-32.95\pm 1.67$ | $\textbf{24.60}^{2}\pm 0.93$ | $-54.57\pm 11.32$ | $\textbf{-15.21}^{3}\pm 4.75$ | $-72.95\pm 7.52$ | $-37.88\pm 0.32$ | $\textbf{26.40}^{1}\pm 3.80$
20 | $-25.58\pm 0.63$ | $\textbf{29.31}^{2}\pm 1.21$ | $-75.29\pm 0.36$ | $\textbf{-18.99}^{3}\pm 2.77$ | $-94.87\pm 13.50$ | $-21.05\pm 0.30$ | $\textbf{30.39}^{1}\pm 0.43$
Cal/VOC | C-V | \ | \ | $\textbf{-12.85}^{3}\pm 3.45$ | $\textbf{13.61}^{2}\pm 1.48$ | $-45.06\pm 2.71$ | $-25.91\pm 2.02$ | $-49.91\pm 11.68$ | $-23.42\pm 5.81$ | $\textbf{18.70}^{1}\pm 4.57$
V-C | $\textbf{-9.70}^{3}\pm 0.54$ | $\textbf{15.66}^{2}\pm 3.96$ | $-55.98\pm 6.51$ | $-18.76\pm 1.51$ | $-24.14\pm 9.23$ | $-46.69\pm 0.91$ | $\textbf{17.96}^{1}\pm 2.47$
Table 5: Comparison of Parameters on each methods.  | One | Joint | fine | PNN | LwF | EWC | DGR | Hnet | CMN
---|---|---|---|---|---|---|---|---|---
test Params($M$) | 11.326 | 11.315 | 11.326 | $\textbf{57.366}^{1}$ | $\textbf{11.829}^{3}$ | 11.188 | 11.295 | $\textbf{18.763}^{2}$ | 11.788
training Params($M$) | 11.326 | 11.315 | 11.326 | $\textbf{57.366}^{1}$ | 11.829 | 11.188 | 12.471 | $\textbf{18.763}^{3}$ | $\textbf{23.575}^{2}$
Iteration time | 10 | $\textbf{50}^{1}$ | 10 | 10 | 10 | 10 | $\textbf{20}^{2}$ | 10 | 10

##  Appendix C Details of dataset and pre-processing

In task-conflict experiments, we used CIFAR-10 and Gaussian noise. In task-
related experiments, we used STL-10, a benchmark of few-shot learning, and
resized the sample as the same as CIFAR-10. In class-incremental, we used
CIFAR-10 and CIFAR-100. CIFAR-10 was uniformly divided into 2 and 5 sub-
datasets (each sub-dataset contains 2 or 5 classes). Similarly, CIFAR-100 was
divided into 2,5,10, and 20 sub-datasets. In the cross-domain experiments,
considering the effect of learning order, PASCAL VOC2007 and Caltech-101 were
learned from two directions, and all images were first resized to
256$\times$256, and then randomly cropped to 224$\times$224.

##  Appendix D Definition of metric AF (anterograde forgetting)

Given $T$ task, denoting the accuracy on $i$-th task for the model trained on
the $i$-th task as $P_{i,i}$, the average accuracy for the model trained on
the $i$ task as $ACC_{i}$, the accuracy on the $i$-th task through the joint
learning as $n_{i,i}$, and the accuracy on the $i$-th task through the one
learning as $m_{i}$. We define AF metric as

$AF=\frac{1}{T-1}\sum_{i=2}^{T}\left(ACC_{i}-n_{i,i}\right)+\frac{1}{T-1}\sum_{i=2}^{T}\left(P_{i,i}-m_{i}\right)$
(10)

##  Appendix E Definition of metric Iteration time for data storage

We believe that low data storage is essential in lifelong learning. However,
the currently commonly used training epochs do not truly reflect the model’s
storage cost. We believe that calculating the number of snapshots of the model
observation samples is more representative of the data storage required to
train the model. For instance, although all algorithms in the experiment have
the same training epoch settings, the number of sample snapshots for DGR and
Joint differs from the other methods because they require replaying historical
data. Denoting the number of images in a single training session as $N_{p}$,
Iteration time is the number of batches needed to complete one epoch, and can
be defined as

$\textit{ Iteration time }=N_{p}/\text{ batchsize }$ (11)

A larger values of  Iteration time  indicates a higher consumption of
resources in a single training epoch. For example, we have 5 tasks with 20
classes, and each class has 500 training images. And the $batchsize$ of all
methods are 1024. The  Iteration time  is $500*20/1024$.

We compares the model parameters size and  Iteration time  in 5 tasks with 20
classes per task on CIFAR100 experiments, as shown in Table. 5.

Figure 11: Visualization of confusion matrix on CIFAR-100 with task num of 10.
(top): The result of EWC. (bottom): The result of CMN.

##  Appendix F Comparison between CMN and PNN under identical parameter size
setting

Fig. 10 shows CMN and PNN in 10 tasks with 10 classes for CIFAR100
experiments. All methods in this part of the experiment use ResNet-18 as the
backbone network.

##  Appendix G Visualization of precision confusion matrix

Fig. 11 investigate the ability to overcome catastrophic forgetting. We find
that this advantage becomes more significant as the number of learning tasks
increases. At task number 5, CMN achieves higher accuracy than EWC on the
previously learned categories. Moreover, this advantage dramatically increases
when the number of tasks is 10. CMN significantly performs higher accuracy on
previous classes, while EWC can barely identify the previous classes.

Figure 12: Comparison of transfer cells with batch normalization. Figure 13:
Analysis of weights initialization for forgetting gate.

##  Appendix H Utilization of batch normalization after the transfer cell

We compare the effect of using batch normalization in the last layer of the
transfer cell. Fig. 12 shows the experimental results on CIFAR-100 with task
number 2. It indicates that the BN layer utilization in the transfer cell
reduces the convergence speed and generalization accuracy on new tasks.
Compared to the w/o_BN, the performance of the model configured with w_BN
reduces by about 3% on $ACC$ and 8% on $FWT$.

Figure 14: Analysis of transfer strategy for memory transfer.

##  Appendix I Initialization of embedding layer in the forgetting gate

We investigate the effect of weights initialization on CMN and find that the
embedding layer of the forgetting gate is susceptible to the initialization.
Fig. 13(a) shows the model’s performance on a CIFAR-100 with a task number of
2 using different initialization methods. The results indicate that it
achieves the highest test accuracy and convergent fastest on the current task
when the embedding layer’s parameter weights are initialized to constant 1. In
comparison, the accuracy is the lowest, and convergence is the slowest when
using kaiming_normal. The xavier_normal and orthogonal are in the middle of
the two. It is intuitive to leave all features to contribute equally in the
initial stage during training.

##  Appendix J Transfer strategy in the transfer cell.

We investigate the effect of transfer strategies on memory transfer. We
consider two scenarios, the positive transfer case and the negative transfer
case. Moreover, four transfer strategies, i combination of transfer cell and
pre-trained S-Net (w_transfer cell); ii using a pre-trained model on S-Net
(w/o_transfer), which does not introduce adverse transfer effects; iii using
transfer matrix (transfer_matrix), a transfer strategy proposed by PNN; iv
directly fusing features from the same layer of L-Net (direct_transfer), a
common way of feature fusion. Fig. 14(a) shows the results of negative
transfer. The transfer cell avoids the negative transfer effect throughout the
training phase, while neither direct_transfer nor transfer_matrix can handle
the negative migration, especially direct_transfer. These methods produce
degradation in both accuracy and convergence speed. Fig. 14(b) shows the
results under positive transfer. The combination of transfer cell and pre-
trained model is far superior to the other strategies in terms of convergence
speed and accuracy throughout the training session. The pre-trained model
approach is more effective than the feature transfer-based approach, e.g.,
w/o_transfer has higher accuracy than transfer_matrix and direct_transfer in
the late training phase. The transfer_matrix has an 18% higher accuracy than
direct_transfer. It indicates that the features in the two networks are not in
the same space, and direct fusion has a negative impact instead.

##  Appendix K Grad CAM Visualization with CMN, EWC, and ONE

We randomly selected samples from 6 classes (cat, dog, sheep, airplane, car,
motorbike) from the PASCAL-VOC 2007. We chose these categories because the
background of their images is simple. The results on some samples suggest that
S-Net is better at learning key features than L-Net. Specifically, for block1
and conv1, the L-Net (bottom left) response is more divergent than the S-Net,
which even responds to background regions. A similar phenomenon is observed at
higher levels (block3), and even at block4, L-Net responds to the local region
of the background. It is comprehensible because L-Net requires a trade-off
between maintaining knowledge of the previous task and knowledge of the
current task (although CMN largely mitigates the anterograde forgetting caused
by memory retention compared to EWC). Moreover, the features learned by CMN
(L-Net and S-Net) are significantly more representative than ONE and EWC. We
attached the visualization of others in Fig. 15.

##  Appendix L Hyper-parameters of CMN on experiments.

We list the hyper-parameters of CMN on our experiments in Table. 6. Noting
that, we used the uniform template $T$ value of 2, although the best $T$ is 10
in the analysis of hyper-parameters.

Table 6: Hyper-parameters of CMN on experiments.  Hyper-parameters | tasks | batch_size | long_lr | short_lr | beta_distll | temp
---|---|---|---|---|---|---
CIFAR10 | 2 | 1024 | 0.1 | 0.01 | 0.8 | 2
5 | 1024 | 0.1 | 0.01 | 0.8 | 2
CIFAR100 | 2 | 1024 | 0.1 | 0.01 | 0.8 | 2
5 | 1024 | 0.1 | 0.01 | 0.8 | 2
10 | 1024 | 0.1 | 0.01 | 0.8 | 2
20 | 1024 | 0.1 | 0.01 | 0.8 | 2
DOMAIN | C-V | 54 | 0.1 | 0.01 | 0.8 | 2
V-C
few-shot | / | 512 | 0.1 | 0.01 | 0.8 | 2
conflict-task | / | 512 | 0.1 | 0.01 | 0.8 | 2

##  Appendix M Computational resources.

All experiments were implemented on three deep learning stations (one is a DGX
A100 Station with 4 * 80G GPU, one is with 8 * 40G A100 GPU, and one is with 2
* 48G A6000 GPU). The deep learning framework PyTorch 1.8.1 was used for the
experiments.

Figure 15: Grad CAM Visualization with CMN, EWC, and ONE.