#  Disentangling and mitigating the impact of task similarity for continual
learning

Naoki Hiratani
Department of Neuroscience
Washington University in St Louis
St Louis, MO 63110
<EMAIL_ADDRESS>

###### Abstract

Continual learning of partially similar tasks poses a challenge for artificial
neural networks, as task similarity presents both an opportunity for knowledge
transfer and a risk of interference and catastrophic forgetting. However, it
remains unclear how task similarity in input features and readout patterns
influences knowledge transfer and forgetting, as well as how they interact
with common algorithms for continual learning. Here, we develop a linear
teacher-student model with latent structure and show analytically that high
input feature similarity coupled with low readout similarity is catastrophic
for both knowledge transfer and retention. Conversely, the opposite scenario
is relatively benign. Our analysis further reveals that task-dependent
activity gating improves knowledge retention at the expense of transfer, while
task-dependent plasticity gating does not affect either retention or transfer
performance at the over-parameterized limit. In contrast, weight
regularization based on the Fisher information metric significantly improves
retention, regardless of task similarity, without compromising transfer
performance. Nevertheless, its diagonal approximation and regularization in
the Euclidean space are much less robust against task similarity. We
demonstrate consistent results in a permuted MNIST task with latent variables.
Overall, this work provides insights into when continual learning is difficult
and how to mitigate it.

##  1 Introduction

Artificial neural networks surpass human capabilities in various domains, yet
struggle with continual learning. These networks tend to forget previously
learned tasks when trained sequentially—a problem known as catastrophic
forgetting [42, 15, 22, 31]. This phenomenon affects not only supervised
training of feedforward networks but also extends to recurrent neural networks
[10], reinforcement learning tasks [29], and fine-tuning of large language
models [39]. Many algorithms for mitigating catastrophic forgetting have been
developed previously, including rehearsal techniques [46, 47, 57], weight
regularization [29, 35, 65], and activity-gating methods [14, 52, 40, 54],
among others [55, 48, 61, 20]. However, these methods often hinder forward and
backward knowledge transfer [27, 38, 28], and thus it remains unclear how to
achieve knowledge transfer and retention simultaneously.

A key factor for continual learning is task similarity. If two subsequent
tasks are similar, there is a potential for a knowledge transfer from one task
to another, but the risk of interference also becomes high [45, 27, 34, 11,
38]. The impact of task similarity on transfer and retention performance is
particularly complicated because two tasks can be similar in different manners
[63, 31]. Sometimes familiar input features need to be associated with novel
output patterns, but at other times, novel input features need to be
associated with familiar output patterns. Previous works observed that these
two scenarios influence continual learning differently [34], yet the impact of
the input and output similarity on knowledge transfer and retention has not
been well understood. Moreover, it remains unknown how the task similarity
interacts with algorithms for continual learning such as activity gating or
weight regularization.

To gain insight into these questions, in this work, we investigate how
transfer and retention performance depend on task similarity, task-dependent
gating, and weight regularization in analytically tractable teacher-student
models. Teacher-student models are simple, typically linear, neural networks
in which the generative model of data is specified explicitly by the teacher
network [17, 64, 3]. These models have provided tremendous insights into
generalization property [53, 43, 18, 1], convergence rate [59, 56, 37], and
learning dynamics [49, 50, 4, 24] of neural networks, due to their analytical
tractability. Several works also studied continual learning using teacher-
student settings [2, 34, 26, 11, 19, 36, 12] (see Related works section for
details).

We develop a linear teacher-student model with a low-dimensional latent
structure and analyze how the similarity of input features and readout
patterns between tasks affect continual learning. We show analytically that a
combination of low feature similarity and high readout similarity is
relatively benign for continual learning, as the retention performance remains
high and the transfer performance remains non-negative. However, the opposite,
a combination of high feature similarity and low readout similarity is
harmful. In this regime, both transfer and retention performance become below
the chance level even when the two subsequent tasks are positively correlated.
Furthermore, transfer performance depends on the feature similarity non-
monotonically, such that, beyond a critical point, the higher the feature
similarity is, the lower the transfer performance becomes.

We further analyze how common algorithms for continual learning, activity and
plasticity gating [14, 40, 44], activity sparsification [55], and weight
regularization [29, 35, 65], interact with task similarity in our problem
setting, deriving several non-trivial conclusions. Activity gating improves
retention at the cost of transfer when the gating highly sparsifies the
activity, but helps both transfer and retention on average if the activity is
kept relatively dense. Plasticity gating and activity sparsification, by
contrast, do not influence either transfer or retention performance at the
over-parameterized limit. Lastly, weight regularization in the Fisher
information metric helps retention without affecting knowledge transfer and
achieves perfect retention regardless of task similarity in the presence of
low-dimensional latent. However, its diagonal approximation and the
regularization in the Euclidean metric are much less robust against both task
similarity and regularizer amplitude.

Furthermore, we test our key predictions numerically in a permuted MNIST task
with a latent structure. When the input pixels are permuted from one task to
the next, the retention performance remains high. However, when the mapping
from the latent to the target output is changed, both the retention and
transfer performance go below the chance level, as predicted. Random task-
dependent gating of input and hidden layer activity improves retention at the
cost of knowledge transfer, but adaptive gating mitigates this tradeoff. We
also show that in a fully-connected feedforward network, there exists an
efficient layer-wise approximation of the weight regularization in the Fisher
information metric, which outperforms its diagonal approximation and the
regularization in the Euclidean metric. Nevertheless, the performance of the
diagonal approximation is much closer to the layer-wise approximation of the
Fisher information metric than to the Euclidean weight regularization.

Our theory thus reveals when continual learning is difficult, and how
different algorithms mitigate these challenging situations, providing a basic
framework for analyzing continual learning in artificial and biological neural
networks.

##  2 Related works

Previous works on continual learning in linear teacher-student models found
that forgetting is most prominent at the intermediate task similarity [11, 19,
12] as observed empirically [45]. However, these works did not address the
tradeoff between forgetting and knowledge transfer, and these simple settings
did not disentangle the effect of the similarity in input feature and readout
pattern. Forward and backward transfer performance in continual learning were
also analyzed in linear and deep-linear networks [32, 8], yet its relationship
with catastrophic forgetting has not been well characterized. Lee et al. [34]
analyzed dynamics of both forgetting and forward transfer in one-hidden layer
nonlinear network under a multi-head continual learning setting. However,
their analysis of readout similarity and its comparison to feature similarity
were conducted numerically and it did not address the effect of common
heuristics, such as gating or weight regularization, either.

By comparison, we introduce a low-dimensional latent structure into a linear
teacher-student model, which enables us to decouple the influence of feature
and readout similarity on knowledge transfer and retention. Moreover, this
low-dimensionality assumption on the latent enables us to evaluate the
transfer and retention performance analytically even in the presence of gating
or weight regularization in the Fisher information metric.

Continual learning has been studied from many theoretical frameworks beyond
teacher-student modeling, including neural tangent kernel [5, 9, 26], PAC
learning [6, 58], and computational complexity [30]. Learning of low-
dimensional latent representation has also been studied in the context of
multi-task learning [41, 60]. In addition, several works investigated the
effect of weight regularization on continual learning in analytically
tractable settings [29, 11, 23].

##  3 Teacher-student model with low-dimensional latent variables

Let us consider task-incremental continual learning of regression tasks. For
analytical tractability, we consider a teacher-student setting where the
target outputs are generated by the teacher network. We define the student
network, which learns the task, as a linear projection from a potentially
nonlinear transformation of the input, $\bm{y}=W\psi(\bm{x})$, where
$\bm{x}\in\mathbb{R}^{N_{x}}$ is the input, $\bm{y}\in\mathbb{R}^{N_{y}}$ is
the output, $W\in\mathbb{R}^{N_{y}\times N_{x}}$ is the trainable weight
matrix, and $\psi(\bm{x}):\mathbb{R}^{N_{x}}\to\mathbb{R}^{N_{x}}$ is an input
transformation. We use $\psi(\bm{x})=\bm{x}$ for the vanilla and weight
regularization models, $\psi(\bm{x})=\bm{g}\odot\bm{x}$ for the task-dependent
gating model, and
$\psi(\bm{x})=sgn(\bm{x})\odot\max\\{0,\lvert\bm{x}\rvert-\bm{h}\\}$ for the
soft-thresholding model, where $\bm{g}$ and $\bm{h}$ are gating and
thresholding vectors, respectively. Here, $\bm{g}\odot\bm{x}$ is an element-
wise multiplication, and $sgn(x)$ is a function that returns the sign of the
input. Throughout the paper, we use bold-italic letters for vectors, capital-
italic letters for matrices.

We generate the input $\bm{x}$ and target output $\bm{y}^{*}$ of the $\mu$-th
task using a latent variable $\bm{s}\in\mathbb{R}^{N_{s}}$:

$\displaystyle\bm{s}\leftarrow\mathcal{N}\left(0,I_{s}\right),\quad\bm{x}=A_{\mu}\bm{s},\quad\bm{y}^{*}=B_{\mu}\bm{s},$
(1)

for $\mu=1,2$, where $A_{\mu}\in\mathbb{R}^{N_{x}\times N_{s}}$ and
$B_{\mu}\in\mathbb{R}^{N_{y}\times N_{s}}$ are mixing matrices hidden from the
student network, and $I_{s}$ is the size $N_{s}$ identity matrix. We introduce
this latent structure, $\bm{s}$, to decouple the effect of feature and readout
similarity on the transfer and retention performance. Below, we set the latent
space to be low-dimensional compared to the input space (i.e., $N_{s}\ll
N_{x}$). This is motivated by the presence of low-dimensional latent structure
in many machine learning datasets [62, 7] and the tasks used in neuroscience
experiments [16, 25], but also aids analytical tractability.

Figure 1: Transfer and retention performance of the vanilla model. (A)
Schematic of task similarity. (B) Illustration of $\Delta\epsilon_{TF}$ and
$\Delta\epsilon_{RT}$. Red and blue lines represent the error on task 1 and
task 2, respectively. Here, the model was trained on task 1 for 100 iterations
and then trained on task 2 for another 100 iterations. (C, D) Transfer
performance under various task similarity. Points in panel C are numerical
results (the means and the standard deviations over ten random seeds), while
solid lines are analytical results (Eq. 4). (E-H) Retention performance under
various task similarity. Panel H magnifies the $0.9\leq\rho_{b}\leq 1.0$
region of panel F, and the white dashed line in panel H represents local
minima/maxima.

We generate the mixing matrices for the first task, $A_{1}$ and $B_{1}$, by
sampling elements independently from a Gaussian distribution with mean zero
and variance $\tfrac{1}{N_{s}}$. The subsequent task matrices $A_{2}$ and
$B_{2}$ are also generated randomly, but with element-wise correlation with
the previous matrices $A_{1}$ and $B_{1}$ (see Appendix A for the details). We
denote the element-wise correlation between $A_{1}$ and $A_{2}$ by $\rho_{a}$
and refer it as feature correlation, because $A_{1}$ and $A_{2}$ specify the
input features. Similarly, we denote the correlation between $B_{1}$ and
$B_{2}$, the readout correlation, by $\rho_{b}$. Fig. 1A illustrates the
difference between $\rho_{a}$ and $\rho_{b}$. When $\rho_{a}=1$ and
$\rho_{b}=0$, the new task requires the network to associate familiar input
features to novel outputs (middle row). By contrast, when $\rho_{a}=0$ and
$\rho_{b}=1$, the network needs to associate novel input features to familiar
readouts (bottom row). Below, we focus on the scenario when tasks have non-
negative correlation in terms of both input features and readout (i.e.,
$0\leq\rho_{a},\rho_{b}\leq 1$). This is because when either $\rho_{a}$ or
$\rho_{b}$ is negative (i.e., in a reversal learning setting), the network
fails to achieve knowledge transfer rather trivially.

We measure the performance of the student network with weight $W$ on the
$\mu$-th task by mean-squared error
$\epsilon_{\mu}[W]\equiv\tfrac{1}{N_{y}}\left\langle\left\lVert
B_{\mu}\bm{s}-W\psi(A_{\mu}\bm{s})]\right\rVert^{2}\right\rangle_{\bm{s}}$,
where $\left\langle\cdot\right\rangle_{\bm{s}}$ is the expectation over latent
variable $\bm{s}$. The transfer and retention performance are defined by

$\displaystyle\Delta\epsilon_{TF}\equiv\left\langle\epsilon_{2}[W_{o}]-\epsilon_{2}[W_{1}]\right\rangle_{A,B},\quad\Delta\epsilon_{RT}\equiv\left\langle\epsilon_{1}[W_{o}]-\epsilon_{1}[W_{2}]\right\rangle_{A,B}.$
(2)

Here, $\left\langle\cdot\right\rangle_{A,B}$ is the expectation over randomly
generated task matrices $A_{1},A_{2},B_{1}$ and $B_{2}$ under a given feature
and readout correlation $\rho_{a},\rho_{b}$. As illustrated in Fig. 1B, the
transfer performance $\Delta\epsilon_{TF}$ measures how much performance the
model achieves on task 2 by learning task 1, whereas the retention performance
$\Delta\epsilon_{RT}$ measures how well the model can perform task 1 after
learning task 2 (here, the task switch occurs at the 100th iteration). Below,
we study how knowledge transfer and retention, the two key objectives of
continual learning, depend on task similarity, and how to optimize the
performance through gating and weight regularization.

##  4 Impact of task similarity on knowledge transfer and retention

Let us first investigate the vanilla model without gating or weight
regularization, to examine how task similarity affects knowledge transfer and
retention. Given $\psi(\bm{x})=\bm{x}$, at the infinite sample limit, learning
by gradient descent follows $\dot{W}=-(WA_{\mu}-B_{\mu})A_{\mu}^{T}$. The
fixed point of this dynamics, as detailed in Appendix B.1, is:

$\displaystyle W_{\mu}=W_{\mu-1}(I-U_{\mu}U_{\mu}^{T})+B_{\mu}A_{\mu}^{+},$
(3)

where $W_{\mu-1}$ is the weight after learning of the previous task, $U_{\mu}$
is the semi-orthogonal matrix from singular value decomposition (SVD) of
$A_{\mu}=U_{\mu}\Lambda_{\mu}V_{\mu}^{T}$, and $A_{\mu}^{+}$ is the pseudo-
inverse of matrix $A_{\mu}$. Inserting Eq. 3 into Eq. 2 and taking the
expectation over randomly generated tasks $A_{1},B_{1},A_{2},B_{2}$, the
transfer and retention performance are written as (see Appendix B.1)

$\displaystyle\Delta\epsilon_{TF}=\rho_{a}(2\rho_{b}-\rho_{a}),\quad\Delta\epsilon_{RT}=1-\rho_{a}^{2}(\rho_{a}^{2}-2\rho_{a}\rho_{b}+1).$
(4)

Recall that $\rho_{a}$ is the feature similarity defined by the correlation
between $A_{1}$ and $A_{2}$, while $\rho_{b}$ is the readout similarity,
representing the correlation between $B_{1}$ and $B_{2}$. The derivation of
the above equations relies on (correlated) random generation of
$A_{1},B_{1},A_{2},B_{2}$ and the low-rank latent assumption: $N_{s}\ll
N_{x}$. These two equations, despite their simplicity, capture the transfer
and retention performance in numerical simulations well (Figs. 1C, E, and G;
see Appendix E.1 for the details of numerical estimation). Furthermore, they
reveal asymmetric and non-monotonic impact of the feature and readout
similarity on the performance.

Fig. 1D depicts the knowledge transfer from task 1 to task 2,
$\Delta\epsilon_{TF}$, under various $(\rho_{a},\rho_{b})$ conditions. As
expected, $\Delta\epsilon_{TF}=0$ when the two tasks are independent (i.e.,
$(\rho_{a},\rho_{b})=(0,0)$), and $\Delta\epsilon_{TF}=1$ when the tasks are
identical (i.e., $(\rho_{a},\rho_{b})=(1,1)$). At intermediate levels of
similarity, there is clear asymmetry in the influence of feature and readout
similarities on transfer performance; particularly, a combination of high
feature similarity and low readout similarity leads to negative transfer,
while the opposite scenario results in a modest positive transfer (lower-right
vs upper-left of Fig. 1D).

Notably, under a fixed readout similarity $\rho_{b}$, the knowledge transfer
depends non-monotonically on the feature similarity $\rho_{a}$. When
$\rho_{a}<\rho_{b}$, the higher the feature similarity the better transfer
becomes, as implied from Eq. 4. However, once the feature similarity exceeds
the readout similarity, counter-intuitively, high feature similarity worsens
the transfer performance (Figs. 1C and D). This is because, when the feature
similarity is high, inputs are aligned with learned features, resulting in a
large output regardless of readout similarity. Particularly, under a low
readout similarity, performance becomes below zero because extracted features
are mostly projected to the incorrect directions.

The retention performance also exhibits asymmetric dependence on feature and
readout similarity. When feature similarity $\rho_{a}$ is low, the network
barely forgets the previous task regardless of readout similarity (Fig. 1F
left). By contrast, when the feature similarity is high, the retention
performance can either be positive or negative depending on the readout
similarity. Moreover, in the high readout similarity regime, the retention
performance is the lowest at an intermediate feature similarity (Figs. 1G and
H). This is because high retention is possible when either interference is
low, or similarity between two tasks is high. Note that, this last point on
the non-monotonic dependence on feature similarity under $\rho_{b}=1$ has been
investigated both empirically [45] and analytically [34, 11, 12]. Indeed, at
$\rho_{b}=1$, $\Delta\epsilon_{RT}$ in Eq. 4 coincides with Eq. (5) in [12].

Therefore, the knowledge transfer and retention performance depend on feature
and readout similarities in an asymmetric and non-monotonic manner.
Specifically, a combination of high feature similarity and low readout
similarity is detrimental to continual learning, resulting in negative
transfer and retention performance, even in the presence of a positive
correlation between the two tasks in both input features and readout patterns.
Moreover, under a fixed readout similarity, the knowledge transfer performance
depends on the feature similarity in a non-monotonic manner. Thus, the
continual learning in the vanilla model is sensitive to task similarity and
limited in performance. These results make us wonder whether we can mitigate
the sensitivity to task similarity and improve the overall transfer and
retention performance by modifying the learning algorithm. To this end, we
next investigate how task-dependent gating and weight regularization methods,
two popular strategies in continual learning, alleviate knowledge transfer and
retention.

Figure 2: Random task-dependent activity gating model. (A) Knowledge transfer
performance under $\rho_{a}=1.0$. The gating level $\alpha$ is defined as the
fraction of active input neurons (i.e., $\alpha=\Pr[g_{i}=1]$). (B) The
transfer performance under the optimal gating level
$\alpha^{*}=\min\\{\tfrac{\rho_{b}}{\rho_{a}},1\\}$. (C) Retention performance
under $\rho_{a}=1.0$. (D) Average transfer and retention performance over
uniform prior on $0\leq\rho_{a},\rho_{b}\leq 1$. Horizontal dashed lines are
the performance of the vanilla model, while solid lines are the performance of
the random gating model. Points are numerical estimations.

##  5 Task-dependent gating

###  5.1 Random activity gating

One popular method for mitigating forgetting in continual learning is activity
gating [14, 52, 40, 20, 54]. With gating of input activity, the network is
written as $\bm{y}=W(\bm{g}_{\mu}\odot\bm{x})$, where
$\bm{g}_{\mu}\in\\{0,1\\}^{N_{x}}$ is a binary gating vector for task $\mu$.
We first consider a random gating scenario where elements of $\bm{g}_{\mu}$
are randomly sampled from a Bernoulli distribution with rate $\alpha$ which we
denote as the gating level. All units are active at $\alpha=1$, while no units
are active at $\alpha\to 0$ limit.

From a parallel argument with the vanilla model, when $N_{s}\ll\alpha N_{x}$,
the transfer and retention performance are estimated as (see Appendix B.1):

$\displaystyle\Delta\epsilon_{TF}$
$\displaystyle=\alpha\rho_{a}(2\rho_{b}-\alpha\rho_{a}),$ (5a)
$\displaystyle\Delta\epsilon_{RT}$
$\displaystyle=1-\alpha^{2}\rho_{a}^{2}(\alpha^{2}\rho_{a}^{2}-2\alpha\rho_{a}\rho_{b}+1).$
(5b)

Thus, random gating scales the feature similarity from $\rho_{a}$ to
$\alpha\rho_{a}$. This scaling lowers the knowledge transfer if
$\rho_{b}\geq\rho_{a}$ because random gating reduces the fraction of input
neurons active in two subsequent tasks (lime line in Fig. 2A). However, if
$\rho_{b}<\rho_{a}$, gating with $\alpha\geq\tfrac{\rho_{b}}{\rho_{a}}$
enhances the transfer by reducing the effective feature similarity (blue lines
in Fig. 2A; also compare Fig. 2B with Fig. 1D). The optimal gating level
$\alpha^{*}$ for knowledge transfer is
$\min\\{\tfrac{\rho_{b}}{\rho_{a}},1\\}$, indicating that the input activity
typically needs to be dense. By contrast, the retention performance is
optimized at $\alpha\to 0$ limit where tasks do not interfere with each others
(Fig. 2C). At this limit, we have $\Delta\epsilon_{RT}\to 1$. Note that, in
reality, $\alpha$ has to be non-zero to optimize the retention (Eq. 5b holds
only when $\alpha\gg\tfrac{N_{s}}{N_{x}}$).

These results indicate that random activity gating improves the retention at
the cost of forward knowledge transfer, as suggested previously [27, 38, 28].
This tradeoff between knowledge transfer and retention is especially critical
when the network doesn’t know the task similarities. Suppose that the task
similarity is distributed uniformly over $0\leq\rho_{a},\rho_{b}\leq 1$. Then,
the average transfer performance is maximized at $\alpha=\tfrac{3}{4}$, while
the average retention performance decreases monotonically as a function of
$\alpha$ (Fig. 2D). Thus, a moderate gating ($\alpha\approx\tfrac{3}{4}$)
benefits both transfer and retention on average. However, retention
performance in this regime is significantly below one, implying that the
network cannot reliably achieve high retention performance without sacrificing
the transfer performance.

###  5.2 Adaptive activity gating

Figure 3: Transfer and retention performance of the adaptive activity gating
(A, B), random plasticity gating (C), and input soft-thresholding (D) models.
Dashed and solid lines in panels A and B represent the performance of the
random and adaptive activity gating models, respectively.

One way to overcome the tradeoff between transfer and retention performance is
to consider adaptive gating. Let us introduce a probe trial at the beginning
of the task 2, in which the model tests how well it performs if it uses the
gating for task 1, $\bm{g}_{1}$, for task 2 as well. If the error in the probe
trial is small, the model should keep using the same gating vectors to achieve
a good knowledge transfer. Otherwise, it should resample the gating vector for
retaining the knowledge on task 1. Using the probe error $\epsilon_{pb}$, we
set the probability of using the same gating as
$\rho_{g}=1-\epsilon_{pb}/\epsilon_{o}$. This adaptive gating indeed improves
the average transfer performance compared to the random gating (solid vs
dashed lines in Fig. 3A) with only a relatively small reduction in the
retention performance (Fig. 3B).

###  5.3 Plasticity gating and input soft-thresholding

Previous works have also explored other gating mechanisms, such as plasticity
gating [44] and activity sparsification [55]. However, their benefits over the
activity gating discussed above have not been fully understood. We implement
task-dependent plasticity gating into our problem setting by making only the
synaptic weights projected from a subset of input neurons plastic.
Unexpectedly, we found that both transfer and retention performance are
independent of the fraction of plastic synapses in the $N_{s}\ll N_{x}$ limit,
potentially due to over-parameterization (Fig. 3C; see Appendix B.2 for
details).

Similarly, when the input activity is sparsified using soft-thresholding
$\varphi(x)=sgn(x)\max\\{0,|x|-h\\}$, with a fixed threshold
$h=\sqrt{2}\text{erfc}^{-1}(\alpha)$, the transfer and retention performance
remain independent of the resultant input sparsity $\alpha$ (Fig. 3D; see
Appendix B.3 for the details). These results imply that plasticity gating and
input sparsification are not effective in our model, which operates in an
over-parameterized regime (i.e., $N_{s}\ll N_{x}$), but do not exclude their
potential benefits for many continual learning tasks.

##  6 Weight regularization

###  6.1 Weight regularization in Euclidean metric

Figure 4: Performance of weight regularization in Euclidean metric. (A,B)
Transfer (A) and retention (B) performance. The amplitude of the weight
regularization scales with $\tfrac{1}{\gamma}-1$. (C) Regularizer coefficient
$\gamma$ that optimizes the retention performance. (D) Average performance
over uniform task similarity distribution in $0\leq\rho_{a},\rho_{b}\leq 1$.
Horizontal dashed lines are the performance of the vanilla model.  Figure 5:
Weight regularization in the Fisher information metric. (A,B) The retention
performance under various task similarities and regularizer coefficients.
(C,D) Average transfer and retention performance under the regularization with
the exact Fisher information metric (C) and its diagonal approximation (D).

Another popular method is weight regularization, which keeps the synaptic
weights close to those learned from previous tasks [29, 35, 65]. Let us first
consider regularization of the Euclidean distance between the current weight
$W$ and the weight learned in the previous task $W_{\mu-1}$:

$\displaystyle\ell_{\mu}=\tfrac{1}{2}\left\lVert
B_{\mu}-WA_{\mu}\right\rVert_{F}^{2}+\tfrac{N_{x}}{2N_{s}}(\tfrac{1}{\gamma}-1)\left\lVert
W-W_{\mu-1}\right\rVert_{F}^{2}.$ (6)

Here, we parameterize the regularizer amplitude by
$\tfrac{N_{x}}{N_{s}}(\frac{1}{\gamma}-1)$ using a non-negative parameter
$\gamma$ for brevity. $\gamma=1$ corresponds to zero regularization, while
$\gamma=0$ is the infinite regularization limit. Under this parameterization,
if $N_{s}\ll N_{x}$, the transfer and retention performance follow simple
expressions as below (see Appendix C.1):

$\displaystyle\Delta\epsilon_{TF}$
$\displaystyle=\gamma\rho_{a}(2\rho_{b}-\gamma\rho_{a}),$ (7a)
$\displaystyle\Delta\epsilon_{RT}$
$\displaystyle=1-\gamma^{2}\rho_{a}^{2}(1-2\gamma\rho_{a}\rho_{b}+\gamma^{2}\rho_{a}^{2})+2\gamma\rho_{a}(1-\gamma)(\rho_{b}-\gamma\rho_{a})-(1-\gamma)^{2}.$
(7b)

The expression of the transfer performance is equivalent to Eq. 5a if we
change $\gamma$ to $\alpha$. Thus, the Euclidean weight regularization with
amplitude $\tfrac{N_{x}}{N_{s}}(\frac{1}{\gamma}-1)$ is mathematically
equivalent with random activity gating with sparsity $\gamma$, in terms of
knowledge transfer (compare Fig. 4A with Fig. 2A). By contrast, the expression
of the retention performance contains two additional terms compared to 5b. In
particular, the last term, $(1-\gamma)^{2}$, indicates that strong weight
regularization not only prevent forgetting, but also impairs task acquisition.
Thus, the retention performance is optimized at an intermediate regularizer
strength (Figs. 4B and C).

Because strong regularization (i.e., small $\gamma$) impairs both retention
and transfer, unlike the random activity gating model, there isn’t a tradeoff
between knowledge transfer and retention in terms of the average performance
over $0\leq\rho_{a},\rho_{b}\leq 1$ (Fig. 4D). However, the retention
performance at the optimal parametric regime is relatively low (compare Fig.
4D with Fig. 3B).

###  6.2 Weight regularization in Fisher information metric

Previous works proposed that the synaptic weights should be regularized in the
Fisher information metric to allow flexibility in the non-overlapping weight
space [29, 65]. Applying it to our model setting, the regularizer term in Eq.
6 instead becomes
$\tfrac{1}{2}(\tfrac{1}{\gamma}-1)\left\lVert(W-W_{\mu-1})A_{\mu-1}\right\rVert_{F}^{2}$
(see Appendix C.2). If $\rho_{a},\gamma<1$ and $N_{s}\ll N_{x}$, optimization
of the weight under this regularization yields

$\displaystyle
W_{\mu}=W_{\mu-1}\left(I-\tfrac{N_{s}}{N_{x}(1-\rho_{a}^{2})}A_{\mu}\left[A_{\mu}^{T}-\rho_{a}A_{\mu-1}^{T}\right]\right)+\tfrac{N_{s}}{N_{x}(1-\rho_{a}^{2})}B_{\mu}\left(A_{\mu}^{T}-\rho_{a}A_{\mu-1}^{T}\right),$
(8)

where $W_{\mu-1}$ is the weight after the previous task learning. Notably, the
weight becomes independent of the regularizer’s amplitude $\gamma$.
Furthermore, the retention performance under arbitrary task similarity
$(\rho_{a},\rho_{b})$ is derived as
$\Delta\epsilon_{RT}=1-\mathcal{O}(\tfrac{N_{s}}{N_{x}(1-\rho_{a}^{2})})$.
Thus, under the Fisher information metric, the retention performance is
perfect as long as the condition $N_{s}\ll N_{x}(1-\rho_{a}^{2})$ is satisfied
(Figs. 5A-C). Intuitively, assuming over-parameterization ($N_{s}\ll N_{x}$),
the network can freeze the weight changes in a low-dimensional subspace, while
maintaining sufficient plasticity for new tasks, unless the two tasks share
the same feature. Notably, if the first task is learned with weight
regularization in an orthogonal direction, the transfer performance remains
the same with the vanilla model (Fig. 5C).

Importantly, this invariance no longer holds when the Fisher information
metric is approximated by its diagonal component (compare Figs. 5D vs 5C), as
is done in the elastic weight methods [29]. This is because the diagonal
approximation makes the metric full-rank even when the true metric has a low-
rank structure. Thus, weight regularization in the Fisher information metric
robustly helps retention without harming transfer, but diagonal approximation
attenuates its robustness.

##  7 Numerical experiments

Figure 6: Permuted MNIST with latent variables. (A,B) Transfer and retention
performance of the vanilla model. (C,D) Performance of random (dashed lines)
and adaptive (solid lines) activity gating models. (E-H) Performance of the
weight regularization in the Euclidean metric, and approximated Fisher
information metrics (red: layer-wise approximation; orange: synapse-
wise/diagonal approximation). Here, lines are linear interpolations and error
bars are standard errors over random seeds, not standard deviations.

Our theory so far has revealed when continual learning is difficult and when
activity gating and weight regularization can ameliorate the transfer and
retention in a simple problem setting. Let us next examine how much these
insights are applicable to more realistic datasets and neural architectures.

To this end, we consider the permuted MNIST dataset [33, 21], but with a
latent structure (see Appendix E.2). For each output label, we constructed a
four-dimensional latent variable $\bm{s}$ using the binary representation of
the corresponding digit (e.g., $\bm{s}_{9}=[1,0,0,1]^{T}$). The target output
was then generated by a random fixed projection of the latent variable:
$\bm{y}^{*}=B_{\mu}(\bm{s}-\tfrac{1}{2})$. We controlled the readout
similarity between tasks using the element-wise correlation between two
matrices $B_{1}$ and $B_{2}$. The feature similarity was controlled by
permuting a subset of input pixels, as in the previous permuted MNIST tasks.
We used a one-hidden-layer fully-connected network with rectified-linear
nonlinearity at the hidden layer, and trained the network using stochastic
gradient descent.

As predicted from our theory, both transfer and retention performance,
measured by the test error, exhibited asymmetric and non-monotonic dependence
on the task similarity. When $(\rho_{a},\rho_{b})=(1.0,0.0)$, both transfer
and retention performance were below zero (lower-right of Figs. 6A and B vs.
Figs. 1D and F). By contrast, the model achieved high retention and zero
transfer under $(\rho_{a},\rho_{b})=(0.0,1.0)$ as expected (upper-left).
Notably, around $\rho_{b}=0.5$, the transfer performance exhibited a non-
monotonic dependence on $\rho_{a}$ in accordance with the theory. The
retention performance also exhibited non-monotonic dependence on $\rho_{a}$
under $\rho_{b}\approx 1$, but the dependence became monotonic after a long
training (see Figs. 9C and D in Appendix).

Random activity gating, implemented in both input and hidden layers, improved
the retention performance at the cost of low knowledge transfer as predicted
(compare Fig. 6D and Fig. 3B). Moreover, adaptive gating based on a probe
trial improved the transfer performance under a low gating level, especially
when the readout similarity is high (solid vs dashed lines in Figs. 6C and D).

In a deep neural network, weight regularization using the Fisher information
metric is typically computationally expensive. However, an efficient layer-
wise approximation exists for fully-connected networks (see Appendix E.2).
This layer-wise approximation enabled high retention performance across a wide
range of regularization amplitudes and consistently outperformed weight
regularization in the Euclidean metric under various task similarities (red
vs. gray lines in Figs. 6F-H). Nevertheless, the retention performance under
the diagonal approximation was closer to that of the layer-wise approximation
of the Fisher information metric and was slightly better when both methods
performed poorly (red vs. orange lines). This is because the sparsity of
hidden layer weights and activity makes the diagonal components sparse.

##  8 Discussion

Here, we analyzed the impact of task similarity on continual learning in a
linear teacher-student model with low-dimensional latent structures. We showed
that, a combination of high feature similarity and low readout similarity lead
to poor outcomes in both retention and transfer, unlike the opposite
combination. We further explored how continual learning algorithms such as
gating mechanisms, activity sparsification, and weight regularization interact
with task similarity. Results indicate that weight regularization in the
Fisher information metric significantly aids retention regardless of task
similarity. Numerical experiments in a permuted MNIST task with latent
supported these findings.

Our findings on the interaction between task similarity and continual learning
algorithms have several implications. Firstly, when tasks are known to have
high feature similarity and low readout similarity, adaptive activity gating
and weight regularization in the Fisher information metric help retention
without sacrificing transfer (Figs. 3, 5, 6). In the context of neuroscience,
our results indicate that simple non-adaptive activity or plasticity gating
mechanisms may not be sufficient for good continual learning performance,
especially when familiar sensory stimuli need to be associated with novel
motor actions. Indeed, a previous study reported catastrophic forgetting among
rats learning timing estimation tasks [13]. Lastly, it also implies the
potential importance of studying wider benchmarks beyond permuted image
recognition tasks for continual learning, because image permutation belongs to
a class of relatively harmless task similarity according to our theory. As a
simple extension, here we developed an image recognition task with a latent
variable and showed that a vanilla deep network exhibits more forgetting under
readout remapping than under input permutation (Figs. 6A and B).

##### Limitations

Our theoretical results from a teacher-student model come with limitation in
direct applicability. The presence of low-dimensional latent generating both
input and target output is a reasonable assumption for many real-world tasks
[7, 25], but the linear projection assumption is not. While we replicated most
key results in a deep nonlinear network solving permuted MNIST task, we found
that the diagonal approximation of Fisher information metric performs much
better than the prediction from the teacher-student model. This is potentially
because sparse activity at the hidden layer effectively makes the
regularization low-rank even under the diagonal approximation. More generally,
the presence of hidden layers should enable a distinctive adaptation for
feature and readout similarity, which is an important future direction. It is
also important to apply our theoretical framework for analyzing continual
learning in more complicated neural architectures and datasets.

## Acknowledgments and Disclosure of Funding

The author thanks Liu Ziyin and Ziyan Li for discussion. This work was
partially supported by the McDonnell Center for Systems Neuroscience.

## References

  * Advani et al. [2020] M. S. Advani, A. M. Saxe, and H. Sompolinsky.  High-dimensional dynamics of generalization error in neural networks.  _Neural Networks_ , 132:428–446, 2020.
  * Asanuma et al. [2021] H. Asanuma, S. Takagi, Y. Nagano, Y. Yoshida, Y. Igarashi, and M. Okada.  Statistical mechanical analysis of catastrophic forgetting in continual learning with teacher and student networks.  _Journal of the Physical Society of Japan_ , 90(10):104001, 2021.
  * Bahri et al. [2020] Y. Bahri, J. Kadmon, J. Pennington, S. S. Schoenholz, J. Sohl-Dickstein, and S. Ganguli.  Statistical mechanics of deep learning.  _Annual Review of Condensed Matter Physics_ , 11:501–528, 2020.
  * Baity-Jesi et al. [2018] M. Baity-Jesi, L. Sagun, M. Geiger, S. Spigler, G. B. Arous, C. Cammarota, Y. LeCun, M. Wyart, and G. Biroli.  Comparing dynamics: Deep neural networks versus glassy systems.  In _International Conference on Machine Learning_ , pages 314–323. PMLR, 2018.
  * Bennani et al. [2020] M. A. Bennani, T. Doan, and M. Sugiyama.  Generalisation guarantees for continual learning with orthogonal gradient descent.  _arXiv preprint arXiv:2006.11942_ , 2020.
  * Chen et al. [2022] X. Chen, C. Papadimitriou, and B. Peng.  Memory bounds for continual learning.  In _2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS)_ , pages 519–530. IEEE, 2022.
  * Cohen et al. [2020] U. Cohen, S. Chung, D. D. Lee, and H. Sompolinsky.  Separability and geometry of object manifolds in deep neural networks.  _Nature communications_ , 11(1):746, 2020.
  * Dhifallah and Lu [2021] O. Dhifallah and Y. M. Lu.  Phase transitions in transfer learning for high-dimensional perceptrons.  _Entropy_ , 23(4):400, 2021.
  * Doan et al. [2021] T. Doan, M. A. Bennani, B. Mazoure, G. Rabusseau, and P. Alquier.  A theoretical analysis of catastrophic forgetting through the ntk overlap matrix.  In _International Conference on Artificial Intelligence and Statistics_ , pages 1072–1080. PMLR, 2021.
  * Ehret et al. [2020] B. Ehret, C. Henning, M. R. Cervera, A. Meulemans, J. Von Oswald, and B. F. Grewe.  Continual learning in recurrent neural networks.  _arXiv preprint arXiv:2006.12109_ , 2020.
  * Evron et al. [2022] I. Evron, E. Moroshko, R. Ward, N. Srebro, and D. Soudry.  How catastrophic can catastrophic forgetting be in linear regression?  In _Conference on Learning Theory_ , pages 4028–4079. PMLR, 2022.
  * Evron et al. [2024] I. Evron, D. Goldfarb, N. Weinberger, D. Soudry, and P. Hand.  The joint effect of task similarity and overparameterization on catastrophic forgetting–an analytical model.  _arXiv preprint arXiv:2401.12617_ , 2024.
  * French and Ferrara [1999] R. French and A. Ferrara.  Modelling time perception in rats: evidence for catastrophic interference in animal learning.  In _21st Congress of Cognitive Sciences_ , 1999.
  * French [1991] R. M. French.  Using semi-distributed representations to overcome catastrophic forgetting in connectionist networks.  In _Proceedings of the 13th annual cognitive science society conference_ , volume 1, pages 173–178, 1991.
  * French [1999] R. M. French.  Catastrophic forgetting in connectionist networks.  _Trends in cognitive sciences_ , 3(4):128–135, 1999.
  * Gao et al. [2017] P. Gao, E. Trautmann, B. Yu, G. Santhanam, S. Ryu, K. Shenoy, and S. Ganguli.  A theory of multineuronal dimensionality, dynamics and measurement.  _BioRxiv_ , page 214262, 2017.
  * Gardner and Derrida [1989] E. Gardner and B. Derrida.  Three unfinished works on the optimal storage capacity of networks.  _Journal of Physics A: Mathematical and General_ , 22(12):1983, 1989.
  * Ghorbani et al. [2019] B. Ghorbani, S. Mei, T. Misiakiewicz, and A. Montanari.  Limitations of lazy training of two-layers neural network.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Goldfarb and Hand [2023] D. Goldfarb and P. Hand.  Analysis of catastrophic forgetting for random orthogonal transformation tasks in the overparameterized regime.  In _International Conference on Artificial Intelligence and Statistics_ , pages 2975–2993. PMLR, 2023.
  * Golkar et al. [2019] S. Golkar, M. Kagan, and K. Cho.  Continual learning via neural pruning.  _arXiv preprint arXiv:1903.04476_ , 2019.
  * Goodfellow et al. [2013] I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio.  An empirical investigation of catastrophic forgetting in gradient-based neural networks.  _arXiv preprint arXiv:1312.6211_ , 2013.
  * Hadsell et al. [2020] R. Hadsell, D. Rao, A. A. Rusu, and R. Pascanu.  Embracing change: Continual learning in deep neural networks.  _Trends in cognitive sciences_ , 24(12):1028–1040, 2020.
  * Heckel [2022] R. Heckel.  Provable continual learning via sketched jacobian approximations.  In _International Conference on Artificial Intelligence and Statistics_ , pages 10448–10470. PMLR, 2022.
  * Hiratani et al. [2022] N. Hiratani, Y. Mehta, T. Lillicrap, and P. E. Latham.  On the stability and scalability of node perturbation learning.  _Advances in Neural Information Processing Systems_ , 35:31929–31941, 2022.
  * Jazayeri and Ostojic [2021] M. Jazayeri and S. Ostojic.  Interpreting neural computations by examining intrinsic and embedding dimensionality of neural activity.  _Current opinion in neurobiology_ , 70:113–120, 2021.
  * Karakida and Akaho [2021] R. Karakida and S. Akaho.  Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting.  In _International Conference on Learning Representations_ , 2021.
  * Ke et al. [2020] Z. Ke, B. Liu, and X. Huang.  Continual learning of a mixed sequence of similar and dissimilar tasks.  _Advances in neural information processing systems_ , 33:18493–18504, 2020.
  * Ke et al. [2023] Z. Ke, B. Liu, W. Xiong, A. Celikyilmaz, and H. Li.  Sub-network discovery and soft-masking for continual learning of mixed tasks.  _arXiv preprint arXiv:2310.09436_ , 2023.
  * Kirkpatrick et al. [2017] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526, 2017.
  * Knoblauch et al. [2020] J. Knoblauch, H. Husain, and T. Diethe.  Optimal continual learning has perfect memory and is np-hard.  In _International Conference on Machine Learning_ , pages 5327–5337. PMLR, 2020.
  * Kudithipudi et al. [2022] D. Kudithipudi, M. Aguilar-Simon, J. Babb, M. Bazhenov, D. Blackiston, J. Bongard, A. P. Brna, S. Chakravarthi Raja, N. Cheney, J. Clune, et al.  Biological underpinnings for lifelong learning machines.  _Nature Machine Intelligence_ , 4(3):196–210, 2022.
  * Lampinen and Ganguli [2018] A. K. Lampinen and S. Ganguli.  An analytic theory of generalization dynamics and transfer learning in deep linear networks.  _arXiv preprint arXiv:1809.10374_ , 2018.
  * LeCun et al. [1998] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.
  * Lee et al. [2021] S. Lee, S. Goldt, and A. Saxe.  Continual learning in the teacher-student setup: Impact of task similarity.  In _International Conference on Machine Learning_ , pages 6109–6119. PMLR, 2021.
  * Lee et al. [2017] S.-W. Lee, J.-H. Kim, J. Jun, J.-W. Ha, and B.-T. Zhang.  Overcoming catastrophic forgetting by incremental moment matching.  _Advances in neural information processing systems_ , 30, 2017.
  * Li et al. [2023] C. Li, Z. Huang, W. Zou, and H. Huang.  Statistical mechanics of continual learning: Variational principle and mean-field potential.  _Physical Review E_ , 108(1):014309, 2023.
  * Li et al. [2020] Y. Li, T. Ma, and H. R. Zhang.  Learning over-parametrized two-layer neural networks beyond ntk.  In _Conference on learning theory_ , pages 2613–2682. PMLR, 2020.
  * Lin et al. [2022] S. Lin, L. Yang, D. Fan, and J. Zhang.  Beyond not-forgetting: Continual learning with backward knowledge transfer.  _Advances in Neural Information Processing Systems_ , 35:16165–16177, 2022.
  * Luo et al. [2023] Y. Luo, Z. Yang, F. Meng, Y. Li, J. Zhou, and Y. Zhang.  An empirical study of catastrophic forgetting in large language models during continual fine-tuning.  _arXiv preprint arXiv:2308.08747_ , 2023.
  * Masse et al. [2018] N. Y. Masse, G. D. Grant, and D. J. Freedman.  Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization.  _Proceedings of the National Academy of Sciences_ , 115(44):E10467–E10475, 2018.
  * Maurer et al. [2016] A. Maurer, M. Pontil, and B. Romera-Paredes.  The benefit of multitask representation learning.  _Journal of Machine Learning Research_ , 17(81):1–32, 2016.
  * McCloskey and Cohen [1989] M. McCloskey and N. J. Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_ , volume 24, pages 109–165. Elsevier, 1989.
  * Ndirango and Lee [2019] A. Ndirango and T. Lee.  Generalization in multitask deep neural classifiers: a statistical physics approach.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Özgün et al. [2020] S. Özgün, A.-M. Rickmann, A. G. Roy, and C. Wachinger.  Importance driven continual learning for segmentation across domains.  In _Machine Learning in Medical Imaging: 11th International Workshop, MLMI 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 4, 2020, Proceedings 11_ , pages 423–433. Springer, 2020.
  * Ramasesh et al. [2020] V. V. Ramasesh, E. Dyer, and M. Raghu.  Anatomy of catastrophic forgetting: Hidden representations and task semantics.  _arXiv preprint arXiv:2007.07400_ , 2020.
  * Robins [1995] A. Robins.  Catastrophic forgetting, rehearsal and pseudorehearsal.  _Connection Science_ , 7(2):123–146, 1995.
  * Rolnick et al. [2019] D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne.  Experience replay for continual learning.  _Advances in neural information processing systems_ , 32, 2019.
  * Rusu et al. [2016] A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell.  Progressive neural networks.  _arXiv preprint arXiv:1606.04671_ , 2016.
  * Saad and Solla [1995] D. Saad and S. A. Solla.  On-line learning in soft committee machines.  _Physical Review E_ , 52(4):4225, 1995.
  * Saxe et al. [2013] A. M. Saxe, J. L. McClelland, and S. Ganguli.  Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.  _arXiv preprint arXiv:1312.6120_ , 2013.
  * Saxe et al. [2019] A. M. Saxe, J. L. McClelland, and S. Ganguli.  A mathematical theory of semantic development in deep neural networks.  _Proceedings of the National Academy of Sciences_ , 116(23):11537–11546, 2019.
  * Serra et al. [2018] J. Serra, D. Suris, M. Miron, and A. Karatzoglou.  Overcoming catastrophic forgetting with hard attention to the task.  In _International conference on machine learning_ , pages 4548–4557. PMLR, 2018.
  * Seung et al. [1992] H. S. Seung, H. Sompolinsky, and N. Tishby.  Statistical mechanics of learning from examples.  _Physical review A_ , 45(8):6056, 1992.
  * Sezener et al. [2021] E. Sezener, A. Grabska-Barwińska, D. Kostadinov, M. Beau, S. Krishnagopal, D. Budden, M. Hutter, J. Veness, M. Botvinick, C. Clopath, et al.  A rapid and efficient learning rule for biological neural circuits.  _BioRxiv_ , pages 2021–03, 2021.
  * Srivastava et al. [2013] R. K. Srivastava, J. Masci, S. Kazerounian, F. Gomez, and J. Schmidhuber.  Compete to compute.  _Advances in neural information processing systems_ , 26, 2013.
  * Tian [2017] Y. Tian.  An analytical formula of population gradient for two-layered relu network and its applications in convergence and critical point analysis.  In _International conference on machine learning_ , pages 3404–3413. PMLR, 2017.
  * Van de Ven et al. [2020] G. M. Van de Ven, H. T. Siegelmann, and A. S. Tolias.  Brain-inspired replay for continual learning with artificial neural networks.  _Nature communications_ , 11(1):4069, 2020.
  * Wang et al. [2023] L. Wang, X. Zhang, Q. Li, M. Zhang, H. Su, J. Zhu, and Y. Zhong.  Incorporating neuro-inspired adaptability for continual learning in artificial intelligence.  _Nature Machine Intelligence_ , 5(12):1356–1368, 2023.
  * Werfel et al. [2003] J. Werfel, X. Xie, and H. Seung.  Learning curves for stochastic gradient descent in linear feedforward networks.  _Advances in neural information processing systems_ , 16, 2003.
  * Xu and Tewari [2022] Z. Xu and A. Tewari.  On the statistical benefits of curriculum learning.  In _International Conference on Machine Learning_ , pages 24663–24682. PMLR, 2022.
  * Yoon et al. [2017] J. Yoon, E. Yang, J. Lee, and S. J. Hwang.  Lifelong learning with dynamically expandable networks.  _arXiv preprint arXiv:1708.01547_ , 2017.
  * Yu et al. [2017] X. Yu, T. Liu, X. Wang, and D. Tao.  On compressing deep models by low rank and sparse decomposition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7370–7379, 2017.
  * Zamir et al. [2018] A. R. Zamir, A. Sax, W. Shen, L. J. Guibas, J. Malik, and S. Savarese.  Taskonomy: Disentangling task transfer learning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3712–3722, 2018.
  * Zdeborová and Krzakala [2016] L. Zdeborová and F. Krzakala.  Statistical physics of inference: Thresholds and algorithms.  _Advances in Physics_ , 65(5):453–552, 2016.
  * Zenke et al. [2017] F. Zenke, B. Poole, and S. Ganguli.  Continual learning through synaptic intelligence.  In _International conference on machine learning_ , pages 3987–3995. PMLR, 2017.

##  Appendix A Problem setting: Linear teacher-student model with a latent
variable

Let us consider a continuous learning of $N_{sess}$ tasks. We generate the
input $\bm{x}\in\mathbb{R}^{N_{x}}$ and the target output
$\bm{y}^{*}\in\mathbb{R}^{N_{y}}$ of the $\mu$-th task by

$\displaystyle\bm{s}\leftarrow\mathcal{N}\left(0,I_{s}\right),\quad\bm{x}=A_{\mu}\bm{s},\quad\bm{y}^{*}=B_{\mu}\bm{s},$
(9)

where $\bm{s}\in\mathbb{R}^{N_{s}}$ is the latent variable, $I_{s}$ is the
size $N_{s}$ identity matrix, and $A_{\mu}\in\mathbb{R}^{N_{x}\times N_{s}}$
and $B_{\mu}\in\mathbb{R}^{N_{y}\times N_{s}}$ are mixing matrices that map
the latent variable $\bm{s}$ into the input space and output space,
respectively. Throughout the paper, the vector $\bm{x}$ is defined as a column
vector, and its transpose $\bm{x}^{T}$ is a row vector.

We consider sequential learning of these tasks with a neural network defined
by $\bm{y}=W\psi(\bm{x})$, where $W\in\mathbb{R}^{N_{y}\times N_{x}}$ is the
plastic weight. We set the function
$\psi:\mathbb{R}^{N_{x}}\to\mathbb{R}^{N_{x}}$ to $\psi(\bm{x})=\bm{x}$ in the
vanilla and weight regularization models, $\psi(\bm{x})=\bm{g}\odot\bm{x}$ in
activity and plasticity gating models, and
$\psi(\bm{x})=sgn(\bm{x})\odot\max\\{0,\lvert\bm{x}\rvert-\bm{h}\\}$ in the
soft-thresholding model. Here, $\bm{g}\odot\bm{x}$ represents an element-wise
multiplication of two vectors, and $sgn(x)$ is a function that returns the
sign of the input $x$. The goal of the student network,
$\bm{y}=W\psi(\bm{x})$, is to mimic the teacher network that generates both
input and target output [17, 64, 3]. Below, we focus on learning of
$N_{sess}=2$ tasks. To analyze the transfer and retention performance, we
introduce the following two assumptions on task structure.

##### Assumption I: Random task assumption

We consider random generation of task matrices $A_{\mu},B_{\mu}$ but with
correlation between subsequent tasks. We sample elements of the mixing
matrices for the first task, $A_{1}$ and $B_{1}$, from a Gaussian distribution
with mean zero and variance $\tfrac{1}{N_{s}}$ independently. For the
subsequent tasks, we generate mixing matrices $A_{\mu}$ by

$\displaystyle A_{\mu,ij}=\begin{cases}A_{\mu-1,ij}&(\text{with probability
}\rho_{a})\\\ \widetilde{A}_{\mu,ij}&(\text{otherwise})\end{cases}$ (10)

where $\widetilde{A}_{\mu}$ is a random matrix whose elements are sampled
independently from a Gaussian distribution with mean zero and variance
$\tfrac{1}{N_{s}}$. We generate $B_{\mu}$ in the same manner, but with
correlation $\rho_{b}$.

##### Assumption II: Low-dimensional latent assumption

The second assumption is the relative low dimensionality of the latent space
with respect to the input space: $N_{s}\ll N_{x}$. This assumption of over-
parameterization in the input space simplifies the analysis (see Appendix D).

##### Evaluation of the transfer and retention performance

We evaluate the task performance of a student model with weight $W$ on task
$\mu$ by mean-squared error:

$\displaystyle\epsilon_{\mu}[W]\equiv\frac{1}{N_{y}}\left\langle\left\lVert
B_{\mu}\bm{s}-W\psi(A_{\mu}\bm{s})]\right\rVert^{2}\right\rangle_{\bm{s}}.$
(11)

Here, $\left\langle\cdot\right\rangle_{\bm{s}}$ is the expectation over latent
$\bm{s}\sim\mathcal{N}(0,I_{s})$. Using $\epsilon_{\mu}[W]$, the degree of
forward knowledge transfer and retention are evaluated by

$\displaystyle\Delta\epsilon^{TF}_{\mu}$
$\displaystyle\equiv\left\langle\epsilon_{\mu}[W_{o}]-\epsilon_{\mu}[W_{\mu-1}]\right\rangle_{A,B},$
(12a) $\displaystyle\Delta\epsilon^{RT}_{\mu}$
$\displaystyle\equiv\left\langle\epsilon_{\mu}[W_{o}]-\epsilon_{\mu}[W_{\mu+1}]\right\rangle_{A,B},$
(12b)

where $W_{\mu}$ is the weight after training on the $\mu$-th task. Note that,
the retention performance is defined by the difference with the initial error
$\epsilon_{\mu}[W_{o}]$. This means that, if the network fails to learn the
task in the first place (i.e., if $\epsilon_{\mu}[W_{\mu}]>0$), the retention
performance becomes sub-optimal even if the network doesn’t forget the learned
task. We used this definition for our problem setting because the network is
able to learn the task perfectly unless strong regularization is added to the
network.

##### Learning procedures

We consider the gradient descent learning from infinite samples at the
gradient flow limit and analyze the performance at the convergence. Below, we
investigate how $\Delta\epsilon^{TF}_{\mu}$ and $\Delta\epsilon^{RT}_{\mu}$
depend on the task similarity and hyper-parameters under various algorithms
for continual learning.

##  Appendix B Task-dependent gating model

###  B.1 Activity gating

Let us first consider task-dependent activity gating model. The student
network (the network that learns the task) is given by:

$\displaystyle\bm{y}=W[\bm{g}_{\mu}\odot\bm{x}],$ (13)

where $\bm{g}_{\mu}$ is the gating vector that depends on the task id $\mu$,
but independent of input $\bm{x}$. Note that, by setting
$\bm{g}_{\mu}=\bm{1}$, we recover the vanilla model $\bm{y}=W\bm{x}$.

The performance of this network on the $\mu$-th task is written as

$\displaystyle\epsilon_{\mu}[W]\equiv\frac{1}{N_{y}}\left\langle\left\lVert
B_{\mu}\bm{s}-W[\bm{g}_{\mu}\odot(A_{\mu}\bm{s})]\right\rVert^{2}\right\rangle_{\bm{s}}=\frac{1}{N_{y}}\left\lVert
B_{\mu}-WD_{\mu}A_{\mu}\right\rVert_{F}^{2},$ (14)

where $D_{\mu}\equiv\text{diag}[\bm{g}_{\mu}]$ is a diagonal matrix whose
$(i,i)$-th element is $g_{\mu,i}$, and $\left\lVert\cdot\right\rVert_{F}$ is
the Frobenius norm.

##### Solution of gradient descent learning

Under the gradient descent learning, the weight dynamics follows

$\displaystyle\dot{W}(t)=-\eta\frac{\partial\epsilon_{\mu}[W]}{\partial
W}=-\frac{2\eta}{N_{y}}\left(WD_{\mu}A_{\mu}-B_{\mu}\right)\left(D_{\mu}A_{\mu}\right)^{T}.$
(15)

From singular value decomposition (SVD), $D_{\mu}A_{\mu}$ is rewritten as

$\displaystyle D_{\mu}A_{\mu}=U_{\mu}\Sigma_{\mu}V_{\mu}^{T},$ (16)

where $U_{\mu}\in\mathbb{R}^{N_{x}\times N_{o}}$ and
$V_{\mu}\in\mathbb{R}^{N_{s}\times N_{o}}$ are semi-orthonormal matrices
(i.e., $U_{\mu}^{T}U_{\mu}=V_{\mu}^{T}V_{\mu}=I_{o}$),
$\Sigma_{\mu}\in\mathbb{R}^{No\times N_{o}}$ is a positive diagonal matrix,
$N_{o}$ is the number of non-zero singular values of $D_{\mu}A_{\mu}$. Using
this decomposition, the gradient descent dynamics is rewritten as

$\displaystyle\dot{W}(t)=-\frac{2\eta}{N_{y}}\left(WU_{\mu}\Sigma_{\mu}-B_{\mu}V_{\mu}\right)\Sigma_{\mu}U_{\mu}^{T}.$
(17)

Thus, the weight matrix $W(t)$ at arbitrary $t$ is written as

$\displaystyle W(t)=W(t=0)+Q(t)U_{\mu}^{T},$ (18)

where $Q\in\mathbb{R}^{N_{y}\times N_{o}}$ is a time-dependent matrix. At the
fixed point of this learning dynamics, we have

$\displaystyle\left(\left[W_{\mu-1}+QU_{\mu}^{T}\right]U_{\mu}\Sigma_{\mu}-B_{\mu}V_{\mu}\right)\Sigma_{\mu}U_{\mu}^{T}=O,$
(19)

where $O$ is the zero matrix and $W_{\mu-1}\equiv W(t=0)$ is the weight after
learning of the previous task. Because $\Sigma_{\mu}$ is a positive diagonal
matrix, the equation above has a unique solution:

$\displaystyle Q$
$\displaystyle=\left(B_{\mu}V_{\mu}-W_{\mu-1}U_{\mu}\Sigma_{\mu}\right)\Sigma_{\mu}^{-1},$
(20a) $\displaystyle W$
$\displaystyle=W_{\mu-1}\left(I_{x}-U_{\mu}U_{\mu}^{T}\right)+B_{\mu}(D_{\mu}A_{\mu})^{+},$
(20b)

where $(D_{\mu}A_{\mu})^{+}=V_{\mu}\Sigma_{\mu}^{-1}U_{\mu}^{T}$ is the
pseudo-inverse of $D_{\mu}A_{\mu}$, and $I_{x}$ is the size $N_{x}$ identity
matrix.

From Eqs. 20b and 14, we have

$\displaystyle\epsilon_{\mu}[W_{\mu}]=\frac{1}{N_{y}}\left\lVert
B_{\mu}\left(V_{\mu}V_{\mu}^{T}-I_{s}\right)\right\rVert_{F}^{2}.$ (21)

Moreover, by setting the initial weight $W_{o}$ (the weight before the first
tasks) to zero, we have

$\displaystyle\epsilon_{\mu}[W_{o}]=\frac{1}{N_{y}}\left\lVert
B_{\mu}\right\rVert_{F}^{2},$ (22)

and $W_{1}$ and $W_{2}$ follow

$\displaystyle W_{1}=B_{1}(D_{1}A_{1})^{+},\quad
W_{2}=B_{1}(D_{1}A_{1})^{+}(I_{x}-U_{2}U_{2}^{T})+B_{2}(D_{2}A_{2})^{+}.$ (23)

The error in task 2 after learning of task 1 is thus written as

$\displaystyle\epsilon_{2}[W_{1}]=\frac{1}{N_{y}}\left\lVert
B_{2}-B_{1}(D_{1}A_{1})^{+}D_{2}A_{2}\right\rVert_{F}^{2}.$ (24)

Similarly, the error on task 1 after learning of task 2 is

$\displaystyle\epsilon_{1}[W_{2}]=\frac{1}{N_{y}}\left\lVert
B_{1}\left(I_{s}-(D_{1}A_{1})^{+}(I_{x}-U_{2}U_{2}^{T})D_{1}A_{1}\right)-B_{2}(D_{2}A_{2})^{+}D_{1}A_{1}\right\rVert_{F}^{2}$
(25)

Note that the results so far does not rely on Assumptions I and II, making
them applicable to arbitrary task matrices $(A,B)$.

The weight matrix after learning depends on the pseudo-inverse,
$(DA)^{+}=V\Sigma^{-1}U^{T}$, which is typically a complicated function of the
original matrix $DA$. However, when $\alpha N_{x}\gg N_{s}$, it can be
approximated by a scaled transpose, $\frac{N_{s}}{\alpha N_{x}}(DA)^{T}$ (see
Eq. 101).

##### Random and adaptive activity gating

In the adaptive activity gating model, we generate input gating units
$\\{g^{\mu}_{i}\\}$ randomly, but with correlations between subsequent tasks.
We sample the gating units for the first task with
$g^{1}_{i}\leftarrow\text{Bernoulli}(\alpha)$, where $\alpha$ is the sparsity
of the gating units. The gating units for the $\mu+1$ task,
$\\{g^{\mu+1}_{i}\\}$, are then generated by

$\displaystyle g^{\mu+1}_{i}=\begin{cases}g^{\mu}_{i}&(\text{with probability
}\rho^{g}_{\mu+1})\\\ \text{Bernoulli}(\alpha)&(\text{otherwise})\end{cases}$
(26)

Here, $\rho^{g}_{\mu+1}$ is the correlation between the gating units for the
$\mu$-th and $\mu+1$-th tasks. The gating correlation $\rho^{g}_{\mu}$ is set
to zero in the case of random task-dependent gating, whereas $\rho^{g}_{\mu}$
is adjusted based on the model performance right after a model switch in the
adaptive task-dependent gating. To this end, we introduce a probe trial where
the model solves the new task using the gating vector for the old task. The
error on the probe trial is

$\displaystyle\epsilon_{pb}\equiv\frac{1}{N_{y}}\left\lVert
B_{2}-W_{1}D_{1}A_{2}\right\rVert_{F}^{2}.$ (27)

We then set the probability of using the same gating elements by

$\displaystyle\rho_{g}=\max\left\\{0,1-\tfrac{\epsilon_{pb}}{\epsilon_{o}}\right\\}$
(28)

where $\epsilon_{o}\equiv\tfrac{1}{N_{y}}\left\lVert
B_{2}\right\rVert_{F}^{2}$ is the baseline error on the task 2. It keeps the
gating the same if $\epsilon_{pb}=0$, while it resamples all gating elements
in case $\epsilon_{pb}\geq\epsilon_{o}$.

##### Transfer performance

The average transfer performance $\Delta\epsilon_{TF}$ from the first to the
second tasks is:

$\displaystyle\Delta\epsilon_{TF}$
$\displaystyle=\frac{1}{N_{y}}\left\langle\left\lVert
B_{2}\right\rVert_{F}^{2}\right\rangle-\frac{1}{N_{y}}\left\langle\left\lVert
B_{2}-B_{1}\left(D_{1}A_{1}\right)^{+}D_{2}A_{2}\right\rVert_{F}^{2}\right\rangle$
$\displaystyle=\frac{1}{N_{y}}\left\langle
2\text{tr}\left[B_{2}^{T}B_{1}(D_{1}A_{1})^{+}D_{2}A_{2}\right]-\left\lVert
B_{1}(D_{1}A_{1})^{+}D_{2}A_{2}\right\rVert_{F}^{2}\right\rangle$
$\displaystyle=\frac{2\rho_{b}}{N_{s}}\left\langle\text{tr}\left[(D_{1}A_{1})^{+}D_{2}A_{2}\right]\right\rangle-\frac{1}{N_{s}}\left\langle\left\lVert(D_{1}A_{1})^{+}D_{2}A_{2}\right\rVert_{F}^{2}\right\rangle.$
(29)

In the last line, we took the expectation over the correlated random matrices
$B_{1}$ and $B_{2}$. Using the approximation
$(DA)^{+}\approx\frac{N_{s}}{\alpha N_{x}}(DA)^{T}$ (Eq. 101), and then taking
the expectation over $A_{1}$, $A_{2}$, $D_{1}$, and $D_{2}$, the first term
becomes

$\displaystyle\left\langle\text{tr}[(D_{1}A_{1})^{+}D_{2}A_{2}]\right\rangle$
$\displaystyle\approx\tfrac{N_{s}}{\alpha
N_{x}}\left\langle\text{tr}[A_{1}^{T}D_{1}D_{2}A_{2}]\right\rangle$
$\displaystyle=\tfrac{N_{s}}{\alpha
N_{x}}\sum_{i=1}^{N_{s}}\sum_{j=1}^{N_{x}}\left\langle
a^{(1)}_{ji}g^{(1)}_{j}g^{(2)}_{j}a^{(2)}_{ji}\right\rangle=\tilde{\alpha}\rho_{a}N_{s},$
(30)

where $a^{(1)}_{ji}$ represents the $(j,i)$-th element of $A_{1}$, and
$\tilde{\alpha}$ is defined by

$\displaystyle\tilde{\alpha}\equiv\rho_{g}+(1-\rho_{g})\alpha.$ (31)

The second term of Eq. B.1 follows

$\displaystyle\left\langle\left\lVert(D_{1}A_{1})^{+}D_{2}A_{2}\right\rVert_{F}^{2}\right\rangle$
$\displaystyle\approx\left(\tfrac{N_{s}}{\alpha
N_{x}}\right)^{2}\left\langle\left\lVert
A_{1}^{T}D_{1}D_{2}A_{2}\right\rVert_{F}^{2}\right\rangle$
$\displaystyle=\left(\tfrac{N_{s}}{\alpha
N_{x}}\right)^{2}\sum_{i,k=1}^{N_{s}}\sum_{j,l=1}^{N_{x}}\left\langle
g^{(1)}_{j}g^{(2)}_{j}g^{(1)}_{l}g^{(2)}_{l}a^{(1)}_{ji}a^{(2)}_{jk}a^{(2)}_{lk}a^{(1)}_{li}\right\rangle$
$\displaystyle=\frac{1}{\alpha^{2}N_{x}^{2}}\sum_{i,k=1}^{N_{s}}\sum_{j,l=1}^{N_{x}}\left\langle
g^{(1)}_{j}g^{(2)}_{j}g^{(1)}_{l}g^{(2)}_{l}[\delta_{ik}\rho_{a}^{2}+\delta_{jl}+\delta_{ik}\delta_{jl}\rho_{a}^{2}]\right\rangle$
$\displaystyle=N_{s}\left(\tilde{\alpha}^{2}\rho_{a}^{2}+\frac{\tilde{\alpha}N_{s}}{\alpha
N_{x}}\left(1+\tfrac{\rho_{a}^{2}}{N_{s}}\right)\right).$ (32)

The third line follows from Isserlis’ theorem, from which we can decompose the
higher-order correlation as below:

$\displaystyle\left\langle
a^{(1)}_{ji}a^{(2)}_{jk}a^{(2)}_{lk}a^{(1)}_{li}\right\rangle$
$\displaystyle=\left\langle a^{(1)}_{ji}a^{(2)}_{jk}\right\rangle\left\langle
a^{(2)}_{lk}a^{(1)}_{li}\right\rangle+\left\langle
a^{(1)}_{ji}a^{(1)}_{li}\right\rangle\left\langle
a^{(2)}_{jk}a^{(2)}_{lk}\right\rangle+\left\langle
a^{(1)}_{ji}a^{(2)}_{lk}\right\rangle\left\langle
a^{(2)}_{jk}a^{(1)}_{li}\right\rangle$
$\displaystyle=\tfrac{\rho_{a}^{2}}{N_{s}^{2}}\delta_{ik}+\tfrac{1}{N_{s}^{2}}\delta_{jl}+\tfrac{\rho_{a}^{2}}{N_{s}^{2}}\delta_{ik}\delta_{jl}.$
(33)

Summing up the equations above, at $\tfrac{N_{s}}{\alpha N_{x}}\to 0$ limit,
we have

$\displaystyle\Delta\epsilon_{TF}=\tilde{\alpha}\rho_{a}\left(2\rho_{b}-\tilde{\alpha}\rho_{a}\right).$
(34)

##### Retention performance

Let us next analyze the retention performance,
$\Delta\epsilon_{RT}\equiv\left\langle\epsilon_{1}[W_{o}]-\epsilon_{1}[W_{2}]\right\rangle_{A,B}$,
which characterizes how well the network performs the task 1 after learning
task 2. At $\alpha N_{x}\gg N_{s}$ regime,
$\left\langle\epsilon_{1}[W_{o}]\right\rangle_{A,B}=1$. Inserting Eq. 23 into
Eq. 14, we get

$\displaystyle\epsilon_{1}[W_{2}]$ $\displaystyle=\frac{1}{N_{y}}\left\lVert
B_{1}-\left[B_{1}(D_{1}A_{1})^{+}(I_{x}-U_{2}U_{2}^{T})+B_{2}(D_{2}A_{2})^{+}\right]D_{1}A_{1}\right\rVert_{F}^{2}$
$\displaystyle=\frac{1}{N_{y}}\left\lVert
B_{1}(D_{1}A_{1})^{+}U_{2}U_{2}^{T}D_{1}A_{1}-B_{2}(D_{2}A_{2})^{+}D_{1}A_{1}\right\rVert_{F}^{2}.$
(35)

Thus, $\Delta\epsilon_{RT}$ follows

$\displaystyle\Delta\epsilon_{RT}$
$\displaystyle=\frac{1}{N_{y}}\left\langle\left\lVert
B_{1}\right\rVert_{F}^{2}\right\rangle-\frac{1}{N_{y}}\left\langle\left\lVert
B_{1}(D_{1}A_{1})^{+}U_{2}U_{2}^{T}D_{1}A_{1}\right\rVert_{F}^{2}\right\rangle-\frac{1}{N_{y}}\left\langle\left\lVert
B_{2}(D_{2}A_{2})^{+}D_{1}A_{1}\right\rVert_{F}^{2}\right\rangle$
$\displaystyle\quad+\frac{2}{N_{y}}\left\langle\text{tr}\left[B_{2}^{T}B_{1}(D_{1}A_{1})^{+}U_{2}U_{2}^{T}D_{1}A_{1}(D_{1}A_{1})^{T}\left((D_{2}A_{2})^{+}\right)^{T}\right]\right\rangle$
$\displaystyle=1-\frac{1}{N_{s}}\left\langle\left\lVert(D_{1}A_{1})^{+}U_{2}U_{2}^{T}D_{1}A_{1}\right\rVert_{F}^{2}\right\rangle-\frac{1}{N_{s}}\left\langle\left\lVert(D_{2}A_{2})^{+}D_{1}A_{1}\right\rVert_{F}^{2}\right\rangle$
$\displaystyle\quad+\frac{2\rho_{b}}{N_{s}}\left\langle\text{tr}\left[(D_{1}A_{1})^{+}U_{2}U_{2}^{T}D_{1}A_{1}(D_{1}A_{1})^{T}\left((D_{2}A_{2})^{+}\right)^{T}\right]\right\rangle.$
(36)

Because $D_{2}A_{2}A_{2}^{T}D_{2}=U_{2}\Sigma_{2}^{2}U_{2}^{T}$, at the
$N_{s}\ll\alpha N_{x}$ limit, $U_{2}U_{2}^{T}$ is approximated by (see Eq.
102)

$\displaystyle U_{2}U_{2}^{T}\approx\frac{N_{s}}{\alpha
N_{x}}D_{2}A_{2}A_{2}^{T}D_{2}.$ (37)

Using this approximation and $(DA)^{+}\approx\frac{N_{s}}{\alpha
N_{x}}(DA)^{T}$ (Eq. 101), we can obtain an analytical expression for Eq. B.1.
Taking the expectation over $D$ and $A$, the second term of Eq. B.1 becomes

$\displaystyle\frac{1}{N_{s}}\left(\frac{N_{s}}{\alpha
N_{x}}\right)^{4}\left\langle\left\lVert
A_{1}^{T}D_{1}D_{2}A_{2}A_{2}^{T}D_{2}D_{1}A_{1}\right\rVert_{F}^{2}\right\rangle$
$\displaystyle=\sum_{ijkl}\sum_{mnpq}\left\langle
g^{(1)}_{j}g^{(1)}_{l}g^{(1)}_{n}g^{(1)}_{q}g^{(2)}_{j}g^{(2)}_{l}g^{(2)}_{n}g^{(2)}_{q}a^{(1)}_{ji}a^{(2)}_{jk}a^{(2)}_{lk}a^{(1)}_{lm}a^{(1)}_{nm}a^{(2)}_{np}a^{(2)}_{qp}a^{(1)}_{qi}\right\rangle$
$\displaystyle=\frac{1}{\alpha^{4}N_{s}N_{x}^{4}}\sum_{ijkl}\sum_{mnpq}\left\langle
g^{(1)}_{j}g^{(1)}_{l}g^{(1)}_{n}g^{(1)}_{q}g^{(2)}_{j}g^{(2)}_{l}g^{(2)}_{n}g^{(2)}_{q}\right\rangle$
$\displaystyle\quad\times\left(\left[\delta_{ik}\delta_{km}\delta_{mp}+\delta_{ik}\delta_{lq}\delta_{mp}+\delta_{km}\delta_{jn}\delta_{ip}\right]\rho_{a}^{4}+\left[\delta_{ik}\delta_{km}\delta_{nq}+\delta_{km}\delta_{mp}\delta_{jq}+\delta_{mp}\delta_{pi}\delta_{jl}+\delta_{pi}\delta_{ik}\delta_{ln}\right]\rho_{a}^{2}\right)$
$\displaystyle\quad+\mathcal{O}\left(\tfrac{1}{\alpha N_{x}}\right)$
$\displaystyle=\tilde{\alpha}^{4}\rho_{a}^{4}+\tfrac{N_{s}}{\alpha
N_{x}}\tilde{\alpha}^{3}(4\rho_{a}^{2}+2\rho_{a}^{4})+\mathcal{O}\left(\tfrac{1}{\alpha
N_{x}}\right),$ (38)

Here, we applied Isserlis’ theorem again, then retained the terms up to the
next-to-leading order with respect to $\tfrac{N_{s}}{\alpha N_{x}}$.
Similarly, the third term of Eq. B.1 becomes

$\displaystyle\frac{1}{N_{s}}\left(\frac{N_{s}}{\alpha
N_{x}}\right)^{2}\left\langle\text{tr}\left[A_{1}^{T}D_{1}D_{2}A_{2}A_{2}^{T}D_{2}D_{1}A_{1}\right]\right\rangle=\tilde{\alpha}^{2}\rho_{a}^{2}+\frac{\tilde{\alpha}N_{s}}{\alpha
N_{x}}\left(1+\tfrac{\rho_{a}^{2}}{N_{s}}\right).$ (39)

Lastly, the cross-term is evaluated as

$\displaystyle\frac{1}{N_{s}}\left(\frac{N_{s}}{\alpha
N_{x}}\right)^{3}\left\langle\text{tr}\left[A_{2}^{T}D_{2}D_{1}A_{1}A_{1}^{T}D_{1}D_{2}A_{2}A_{2}^{T}D_{2}D_{1}A_{1}\right]\right\rangle$
$\displaystyle=\frac{1}{\alpha^{3}N_{s}N_{x}^{3}}\sum_{ijk}\sum_{lmn}\left\langle
g^{(1)}_{j}g^{(1)}_{l}g^{(1)}_{n}g^{(2)}_{j}g^{(2)}_{l}g^{(2)}_{n}\right\rangle\left(\delta_{ik}\delta_{km}\rho_{a}^{3}+\delta_{ik}\delta_{ln}\rho_{a}+\delta_{im}\delta_{jl}\rho_{a}+\delta_{mk}\delta_{jn}\rho_{a}^{3}\right)+\mathcal{O}\left(\tfrac{1}{\alpha
N_{x}}\right)$
$\displaystyle=\tilde{\alpha}^{3}\rho_{a}^{3}+\tfrac{N_{s}}{\alpha
N_{x}}\tilde{\alpha}^{2}(2\rho_{a}+\rho_{a}^{3})+\mathcal{O}\left(\tfrac{1}{\alpha
N_{x}}\right).$ (40)

Therefore, up to the leading order, the retention performance follows

$\displaystyle\Delta\epsilon_{RT}=1-\tilde{\alpha}^{2}\rho_{a}^{2}\left(\tilde{\alpha}^{2}\rho_{a}^{2}-2\tilde{\alpha}\rho_{a}\rho_{b}+1\right)+\mathcal{O}\left(\tfrac{N_{s}}{\alpha
N_{x}}\right).$ (41)

Figure 7: The gating level dependence of the transfer and retention
performance (A) Phase diagram of the gating level dependence. (B-D) Transfer
and retention performance as a function of the gating level at a
representative point of each phase.

##### Vanilla model

In the vanilla model, we have $\tilde{\alpha}=1$. Therefore, at $N_{s}\ll
N_{x}$ limit, from Eqs. 34 and 41, the transfer and retention performance
follow

$\displaystyle\Delta\epsilon_{TF}$
$\displaystyle=\rho_{a}(2\rho_{b}-\rho_{a}),$ (42a)
$\displaystyle\Delta\epsilon_{RT}$
$\displaystyle=1-\rho_{a}^{2}(\rho_{a}^{2}-2\rho_{a}\rho_{b}+1).$ (42b)

##### Random activity gating

Under the random task-dependent activity gating, the gating level
$\tilde{\alpha}=\alpha$ is a constant. If $N_{s}\ll\alpha N_{x}$, the transfer
and retention performance are written by

$\displaystyle\Delta\epsilon_{TF}$
$\displaystyle=\alpha\rho_{a}(2\rho_{b}-\alpha\rho_{a}),$ (43a)
$\displaystyle\Delta\epsilon_{RT}$
$\displaystyle=1-\alpha^{2}\rho_{a}^{2}(\alpha^{2}\rho_{a}^{2}-2\alpha\rho_{a}\rho_{b}+1).$
(43b)

Note that, this expression does not hold at the sparse limit
$\alpha<\frac{N_{s}}{N_{x}}$.

Depending on ($\rho_{a}$, $\rho_{b}$) combination, the gating level influences
the transfer and retention performance in three different manners (Fig. 7).
When $\rho_{b}>\rho_{a}$, unless both $\rho_{a}$ and $\rho_{b}$ are large,
there is a monotonic tradeoff between the transfer and retention performance
(Fig. 7B). In the red region where
$\rho_{b}\geq\tfrac{2\rho_{a}}{3}+\tfrac{1}{3\rho_{a}}$ or
$\rho_{b}\geq\tfrac{2\sqrt{2}}{3}\wedge\rho_{a}\geq\tfrac{1}{\sqrt{2}}$, a
large gating level above a threshold improves the retention performance (Fig.
7C). When $\rho_{b}<\rho_{a}$, high gating level,
$\alpha>\tfrac{\rho_{b}}{\rho_{a}}$, lower both transfer and retention
performance, thus there is no benefit of choosing large $\alpha$ (Fig. 7D).

##### Adaptive activity gating

In the adaptive task-dependent activity gating model, we introduced a probe
trial at the beginning of the task 2 to test the model performance for the new
task. Under an adaptive gating with
$\rho_{g}=\max\left\\{0,1-\tfrac{\epsilon_{pb}}{\epsilon_{o}}\right\\}$, the
probe error $\epsilon_{pb}$ follows:

$\displaystyle\left\langle\epsilon_{pb}\right\rangle_{A,B,g}$
$\displaystyle=\left\langle\frac{1}{N_{y}}\left\lVert
B_{2}-W_{1}D_{1}A_{2}\right\rVert_{F}^{2}\right\rangle_{A,B,g}$
$\displaystyle\approx\left\langle\frac{1}{N_{y}}\left\lVert
B_{2}-\frac{N_{s}}{\alpha
N_{x}}B_{1}(D_{1}A_{1})^{T}D_{1}A_{2}\right\rVert_{F}^{2}\right\rangle_{A,B,g}$
$\displaystyle=1-2\rho_{a}\rho_{b}+\rho_{a}^{2}+\mathcal{O}\left(\frac{N_{s}}{\alpha
N_{x}}\right).$ (44)

Thus, at $N_{s}\ll\alpha N_{x}$ region,
$\rho_{g}=\max\left\\{0,\rho_{a}(2\rho_{b}-\rho_{a})\right\\}$, and the
effective gating level becomes

$\displaystyle\widetilde{\alpha}=\begin{cases}\alpha+(1-\alpha)\rho_{a}\left(2\rho_{b}-\rho_{a}\right)&(\text{if}\,\,2\rho_{b}\geq\rho_{a})\\\
\alpha&(\text{otherwise})\end{cases}$ (45)

meaning that if the transfer performance is positive in the vanilla model,
there should be an overlap in the gating, but otherwise, the gating vector for
the next task should be sampled independently.

##### Average error under a uniform prior on task similarity

Gating influences the performance differently depending on the feature and
readout similarity. Thus, ideally the model should adjust the gating level
based on the task similarity, but in most practical settings, the model
doesn’t know the similarity a priori. Therefore, it is important to analyze
how it performs on average under a randomly chosen task similarity. To this
end, here we measure the average transfer and retention performance assuming a
uniform prior on $0\leq\rho_{a},\rho_{b}\leq 1$.

In the vanilla model, we have

$\displaystyle\Delta\bar{\epsilon}_{TF}$
$\displaystyle=\int_{0}^{1}d\rho_{a}\int_{0}^{1}d\rho_{b}\left(\rho_{a}(2\rho_{b}-\rho_{a})\right)=\frac{1}{6},$
(46a) $\displaystyle\Delta\bar{\epsilon}_{RT}$
$\displaystyle=\int_{0}^{1}d\rho_{a}\int_{0}^{1}d\rho_{b}\left(1-\rho_{a}^{2}\left[\rho_{a}^{2}-2\rho_{a}\rho_{b}+1\right]\right)=\frac{43}{60}.$
(46b)

Red and blue dash lines in Figs. 2D, 3C, 3D, 4D, 5C, and 5D represent these
baseline performance of the vanilla model:
$\Delta\bar{\epsilon}_{TF}=\tfrac{1}{6}$ and
$\Delta\bar{\epsilon}_{RT}=\tfrac{43}{60}$. By contrast, under a random task-
dependent gating,

$\displaystyle\Delta\bar{\epsilon}_{TF}=\frac{\alpha}{2}-\frac{\alpha^{2}}{3},\quad\Delta\bar{\epsilon}_{RT}=1-\frac{\alpha^{2}}{3}+\frac{\alpha^{3}}{4}-\frac{\alpha^{4}}{5}.$
(47)

Solid lines in Fig. 2D and dashed lines in Fig. 3B plot the result above. In
the case of adaptive gating, the expression of the average performance becomes
complicated, yet still numerically tractable.

Note that the average transfer performance is expected to be lower than the
retention performance, because forward knowledge transfer requires a high
similarity. Even if we use the optimal gating level for transfer,
$\alpha^{*}(\rho_{a},\rho_{b})=\min(1,\tfrac{\rho_{b}}{\rho_{a}})$, the
average transfer performance becomes

$\displaystyle\int_{0}^{1}d\rho_{a}\int_{0}^{1}d\rho_{b}\alpha^{*}(\rho_{a},\rho_{b})\rho_{a}(2\rho_{b}-\alpha^{*}(\rho_{a},\rho_{b})\rho_{a})=\frac{1}{4}.$
(48)

###  B.2 Plasticity gating

An alternative strategy is to gate plasticity at certain synapses in a task-
dependent manner while keeping the activity intact. We implement this method
by making synapses from only a subset of input neurons plastic for each task.
Given a linear regression model $\bm{y}=W\bm{x}$, the learning dynamics
follows:

$\displaystyle\dot{W}(t)=-\eta(WA_{\mu}-B_{\mu})(D_{\mu}A_{\mu})^{T},$ (49)

where $D_{\mu}=\text{diag}(\bm{g}_{\mu})$ is a diagonal matrix specifying
which synapses are gated. As in the activity gating, $\bm{g}_{\mu}$ is a
gating vector whose elements take either one or zero in a task-dependent
manner. We sample elements of $\bm{g}_{\mu}$ from a Bernoulli distribution
with probability $\alpha$ as before.

Let us denote SVD of $A_{\mu}$ by $A_{\mu}=U_{\mu}\Lambda_{\mu}V_{\mu}^{T}$.
Then, from a parallel argument with Eqs. 18 and 19, we have

$\displaystyle
W_{\mu}=W_{\mu-1}+(B_{\mu}V_{\mu}\Lambda_{\mu}^{-1}-W_{\mu-1}U_{\mu})(U_{\mu}^{T}D_{\mu}U_{\mu})^{-1}U_{\mu}^{T}D_{\mu}.$
(50)

Notably, unlike Eq. 20b, gating term $D_{\mu}$ appears both inside and outside
of the inverse term ($(U_{\mu}^{T}D_{\,u}U_{\mu})^{-1}$ and
$U_{\mu}^{T}D_{\mu}$). Thus, up to the leading order, gating level $\alpha$
does not affect the transfer and retention performance.

###  B.3 Input soft-thresholding

Input-dependent gating is an another method proposed for efficient continual
learning. In a regression setting, it corresponds to soft-thresholding of the
inputs. Let us introduce a soft-thresholding function $\varphi(x)$ by

$\displaystyle\varphi(x)=sgn(x)\max\\{0,|x|-h\\},$ (51)

where $sgn(x)$ represents the sign of $x$. This nonlinearity makes the
analysis difficult, but if $N_{s}\gg 1$ in addition to Assumptions I and II,
the error becomes tractable. Note that $N_{s}\gg 1$ assumption is required
only in this subsection. Under the student network defined by

$\displaystyle\hat{\bm{y}}=W\varphi(\bm{x}),$ (52)

the mean-squared error follows

$\displaystyle\epsilon[W]=\frac{1}{N_{y}}\left\langle\left\lVert
B\bm{s}\right\rVert^{2}\right\rangle_{\bm{s}}+\frac{1}{N_{y}}\left\langle\left\lVert
W\varphi(A\bm{s})\right\rVert^{2}\right\rangle_{\bm{s}}-\frac{2}{N_{y}}\left\langle\text{tr}\left[B\bm{s}\varphi(A\bm{s})^{T}W^{T}\right]\right\rangle_{\bm{s}}.$
(53)

Let us denote $m_{i}\equiv\sum_{j=1}^{N_{s}}A_{ij}s_{j}$, then we have

$\displaystyle\left\langle m_{i}\right\rangle=0,\quad\left\langle
m_{i}^{2}\right\rangle=\sum_{j=1}^{N_{s}}A_{ij}^{2}\approx 1,\quad\left\langle
s_{j}m_{i}\right\rangle=A_{ij}.$ (54)

Thus, $s_{j}$ and $m_{i}$ approximately follows a joint Gaussian distribution:

$\displaystyle\begin{pmatrix}s_{j}\\\
m_{i}\end{pmatrix}\sim\mathcal{N}\left(\begin{pmatrix}0\\\
0\end{pmatrix},\begin{pmatrix}1&A_{ji}\\\ A_{ji}&1\end{pmatrix}\right).$ (55)

Denoting $A_{ji}=\rho$ for brevity, we have

$\displaystyle\left\langle s_{j}\varphi(m_{i})\right\rangle$
$\displaystyle=\int_{-\infty}^{\infty}ds\left(\int_{-\infty}^{-h}dm(m+h)+\int_{h}^{\infty}dm(m-h)\right)\frac{s}{2\pi\sqrt{1-\rho^{2}}}\exp\left(-\frac{1}{2(1-\rho^{2})}\left[s^{2}+m^{2}-2\rho
sm\right]\right)$
$\displaystyle=\int_{-\infty}^{\infty}ds\left(\int_{-\infty}^{-h}dm(m+h)+\int_{h}^{\infty}dm(m-h)\right)\frac{s}{2\pi}\left[1+\rho
sm+\tfrac{\rho^{2}}{2}(s^{2}-1)(m^{2}-1)+\mathcal{O}(\rho^{3})\right]$
$\displaystyle\quad\times\exp\left(-\frac{s^{2}+m^{2}}{2}\right)$
$\displaystyle=\text{erfc}\left[\tfrac{h}{\sqrt{2}}\right]\rho+\mathcal{O}(\rho^{3}).$
(56)

In the third line, we performed a Taylor expansion around $\rho=0$. Similarly,
the expectation of $\varphi(m_{j})\varphi(m_{i})$ over $\bm{s}$ is written as

$\displaystyle\left\langle\varphi(m_{j})\varphi(m_{i})\right\rangle$
$\displaystyle=\frac{1}{2\pi}\left(\int_{-\infty}^{-h}dm(m+h)+\int_{h}^{\infty}dm(m-h)\right)\left(\int_{-\infty}^{-h}dm^{\prime}(m^{\prime}+h)+\int_{h}^{\infty}dm^{\prime}(m^{\prime}-h)\right)$
$\displaystyle\qquad\times\left[1+\rho_{c}mm^{\prime}+\tfrac{\rho_{c}^{2}}{2}(m^{2}-1)(m^{\prime
2}-1)+\mathcal{O}(\rho_{c}^{3})\right]\exp\left(-\frac{m^{2}+m^{\prime
2}}{2}\right)$
$\displaystyle=\left(\text{erfc}\left[\tfrac{h}{\sqrt{2}}\right]\right)^{2}\rho_{c}+\mathcal{O}(\rho_{c}^{3}),$
(57)

where $\rho_{c}=\sum_{k=1}^{N_{s}}A_{ik}A_{jk}$. Therefore, if $|\rho|\ll 1$
and $|\rho_{c}|\ll 1$, the error $\epsilon[W]$ is written as

$\displaystyle\epsilon[W]=\frac{1}{N_{y}}\left\lVert
B\right\rVert_{F}^{2}+\frac{\alpha^{2}}{N_{y}}\text{tr}\left[WAA^{T}W^{T}\right]-\frac{2\alpha}{N_{y}}\text{tr}[BA^{T}W^{T}],$
(58)

where the input activity sparseness $\alpha$ follows
$\alpha=\text{erfc}\left[\tfrac{h}{\sqrt{2}}\right]$. Under this loss
function, gradient descent from $W=W_{o}$ converges to

$\displaystyle W=W_{o}(I-UU^{T})+\frac{1}{\alpha}BA^{+},$ (59)

where $A=U\Lambda V^{T}$ is the singular value decomposition of $A$.
Therefore, from $W_{o}=O$, the weights after task 1 and task 2 become

$\displaystyle W_{1}=\frac{1}{\alpha}\widetilde{W}_{1},\quad
W_{2}=\frac{1}{\alpha}\widetilde{W}_{2},$ (60)

where

$\displaystyle\widetilde{W}_{1}\equiv
B_{1}A_{1}^{+},\quad\widetilde{W}_{2}\equiv
B_{1}A_{1}^{+}\left[I-U_{2}U_{2}^{T}\right]+B_{2}A_{2}^{+}.$ (61)

This means that for both tasks ($\mu,\nu=1,2$)

$\displaystyle\epsilon_{\mu}[W_{\nu}]=\frac{1}{N_{y}}\left\lVert
B_{\mu}\right\rVert_{F}^{2}+\frac{1}{N_{y}}\text{tr}\left[\widetilde{W}_{\nu}A_{\mu}A_{\mu}^{T}\widetilde{W}_{\nu}^{T}\right]-\frac{2}{N_{y}}\text{tr}[B_{\mu}A_{\mu}^{T}\widetilde{W}_{\nu}^{T}],$
(62)

implying that the error is invariant against input sparseness $\alpha$. Thus,
in our problem setting, input sparsification via soft-thresholding doesn’t
influence knowledge transfer or retention. It also implies that from energy
efficiency perspective, soft-thresholding is beneficial because it reduces the
number of active neurons while preserving its learning proficiency.

##  Appendix C Weight regularization

###  C.1 Weight regularization in Euclidean metric

Let us next consider a weight regularization approach by introducing L2
regularization with respect to the weight learned in the previous task. The
loss function $\ell_{\mu}$ for the $\mu$-th task is then given by

$\displaystyle\ell_{\mu}=\frac{1}{2}\left\lVert
B_{\mu}-WA_{\mu}\right\rVert_{F}^{2}+\frac{\lambda}{2}\left\lVert
W-W_{\mu-1}\right\rVert_{F}^{2}.$ (63)

When $\lambda>0$, there exists a unique solution:

$\displaystyle W=(B_{\mu}A_{\mu}^{T}+\lambda
W_{\mu-1})(A_{\mu}A_{\mu}^{T}+\lambda I_{x})^{-1}.$ (64)

Thus, from zero-initialization, $W_{1}$ and $W_{2}$ become

$\displaystyle W_{1}$
$\displaystyle=B_{1}A_{1}^{T}\left(A_{1}A_{1}^{T}+\lambda I_{x}\right)^{-1}$
(65a) $\displaystyle W_{2}$ $\displaystyle=\left(B_{2}A_{2}^{T}+\lambda
B_{1}A_{1}^{T}[A_{1}A_{1}^{T}+\lambda
I_{x}]^{-1}\right)(A_{2}A_{2}^{T}+\lambda I_{x})^{-1}.$ (65b)

Under $N_{x}\gg N_{s}$, the inverse term is approximated by

$\displaystyle\left(AA^{T}+\lambda I_{x}\right)^{-1}$
$\displaystyle=\frac{1}{\lambda}I_{x}-\frac{1}{\lambda^{2}}A\left(I_{s}+\frac{1}{\lambda}A^{T}A\right)^{-1}A^{T}$
$\displaystyle\approx\frac{1}{\lambda}\left(I_{x}-\frac{N_{s}}{N_{x}+\lambda
N_{s}}AA^{T}\right).$ (66)

In the first line, we used Woodbury matrix identity, and in the second line,
we used $A^{T}A\approx\tfrac{N_{x}}{N_{s}}I_{s}$ (see Eq. 99). Let us
introduce normalized regularization amplitude $\gamma$ and $\tilde{\gamma}$ by

$\displaystyle\gamma\equiv\frac{N_{x}}{N_{x}+\lambda
N_{s}},\quad\widetilde{\gamma}\equiv\frac{N_{s}}{N_{x}+\lambda N_{s}}.$ (67)

Under these approximations, $W_{1}$ and $W_{2}$ simplify to

$\displaystyle W_{1}$
$\displaystyle\approx\tfrac{1}{\lambda}B_{1}A_{1}^{T}\left(1-\widetilde{\gamma}A_{1}A_{1}^{T}\right)\approx\widetilde{\gamma}B_{1}A_{1}^{T},$
(68a) $\displaystyle W_{2}$
$\displaystyle\approx\tfrac{1}{\lambda}\left(B_{2}A_{2}^{T}+\lambda\widetilde{\gamma}B_{1}A_{1}^{T}\right)\left(I_{x}-\widetilde{\gamma}A_{2}A_{2}^{T}\right)\approx\widetilde{\gamma}\left(B_{1}A_{1}^{T}+B_{2}A_{2}^{T}\right)-\widetilde{\gamma}^{2}B_{1}A_{1}^{T}A_{2}A_{2}^{T}.$
(68b)

Here, we further applied $A^{T}A\approx\tfrac{N_{x}}{N_{s}}I_{s}$. Note that,
unlike Eq. 20b, the result above holds only under Assumptions I & II.

Figure 8: Weight regularization in Euclidean metric. (A,B) Optimal regularizer
coefficient $\gamma$ that maximizes the transfer performance (A), and the
maximum performance at the optimal $\gamma$ (B) under various ($\rho_{a}$,
$\rho_{b}$) pairs. (C,D) Optimal regularizer coefficient for retention (C),
and the resultant performance (D). Panel C is the same with Fig. 4C
(replicated for completeness).

##### Transfer performance

The transfer performance $\Delta\epsilon_{TF}$ follows

$\displaystyle\Delta\epsilon_{TF}$
$\displaystyle=\frac{1}{N_{y}}\left\langle\left\lVert
B_{2}\right\rVert_{F}^{2}\right\rangle-\frac{1}{N_{y}}\left\langle\left\lVert
B_{2}-\widetilde{\gamma}B_{1}A_{1}^{T}A_{2}\right\rVert_{F}^{2}\right\rangle$
$\displaystyle=\frac{2\widetilde{\gamma}}{N_{y}}\left\langle\text{tr}[B_{2}^{T}B_{1}A_{1}^{T}A_{2}]\right\rangle-\frac{\widetilde{\gamma}^{2}}{N_{y}}\left\langle\left\lVert
B_{1}A_{1}^{T}A_{2}\right\rVert_{F}^{2}\right\rangle$
$\displaystyle=\frac{2\widetilde{\gamma}\rho_{b}}{N_{s}}\left\langle\text{tr}[A_{1}^{T}A_{2}]\right\rangle-\frac{\widetilde{\gamma}^{2}}{N_{s}}\left\langle\left\lVert
A_{1}^{T}A_{2}\right\rVert_{F}^{2}\right\rangle$
$\displaystyle=\gamma\rho_{a}\left(2\rho_{b}-\gamma\rho_{a}\right)+\mathcal{O}\left(\tfrac{N_{s}}{N_{x}}\right).$
(69)

In the last line, we used Eq. B.1 to estimate $\left\langle\left\lVert
A_{1}^{T}A_{2}\right\rVert_{F}^{2}\right\rangle$. Notably, the expression of
the transfer performance, $\Delta\epsilon_{TF}$, coincides with Eq. 5a under
$\gamma\to\alpha$, indicating that in terms of transfer performance, the
weight regularization with amplitude
$\tfrac{N_{x}}{N_{s}}\left(\tfrac{1}{\alpha}-1\right)$ is equivalent to random
activity gating with gating level $\alpha$. Figs. 8A and B show the optimal
regularizer coefficient $\gamma$ that maximizes transfer performance and the
resultant performance.

##### Retention performance

The average error on task 1 after learning task 2 is

$\displaystyle\left\langle\epsilon_{1}[W_{2}]\right\rangle_{A,B}$
$\displaystyle=\left\langle\frac{1}{N_{y}}\left\lVert
B_{1}-\left(\widetilde{\gamma}[B_{1}A_{1}^{T}+B_{2}A_{2}^{T}]-\widetilde{\gamma}^{2}B_{1}A_{1}^{T}A_{2}A_{2}^{T}\right)A_{1}\right\rVert_{F}^{2}\right\rangle$
$\displaystyle=\frac{1}{N_{y}}\left\langle\left\lVert
B_{1}(I-\widetilde{\gamma}A_{1}^{T}A_{1})\right\rVert_{F}^{2}\right\rangle+\frac{2\widetilde{\gamma}}{N_{y}}\left\langle\text{tr}\left[(I-\widetilde{\gamma}A_{1}^{T}A_{1})B_{1}^{T}(\widetilde{\gamma}B_{1}A_{1}^{T}A_{2}-B_{2})A_{2}^{T}A_{1}\right]\right\rangle$
$\displaystyle\quad+\frac{\widetilde{\gamma}^{2}}{N_{y}}\left\langle\left\lVert(\widetilde{\gamma}B_{1}A_{1}^{T}A_{2}-B_{2})A_{2}^{T}A_{1}\right\rVert_{F}^{2}\right\rangle.$
(70)

Taking the expectation over $A_{1},A_{2},B_{1},B_{2}$, up to the leading order
with respect to $\frac{N_{s}}{N_{x}}$, we have

$\displaystyle\frac{1}{N_{y}}\left\langle\left\lVert
B_{1}(I-\widetilde{\gamma}A_{1}^{T}A_{1})\right\rVert_{F}^{2}\right\rangle$
$\displaystyle=\frac{1}{N_{y}}\left\langle\left\lVert
B_{1}\right\rVert_{F}^{2}-2\widetilde{\gamma}\text{tr}[B_{1}^{T}B_{1}A_{1}^{T}A_{1}]+\widetilde{\gamma}^{2}\left\lVert
B_{1}A_{1}^{T}A_{1}\right\rVert_{F}^{2}\right\rangle$
$\displaystyle=1-\frac{2\widetilde{\gamma}}{N_{s}}\left\langle\text{tr}[A_{1}^{T}A_{1}]\right\rangle+\frac{\widetilde{\gamma}^{2}}{N_{s}}\left\langle\left\lVert
A_{1}^{T}A_{1}\right\rVert_{F}^{2}\right\rangle$
$\displaystyle=\left(1-\gamma\right)^{2}+\mathcal{O}\left(\tfrac{N_{s}}{N_{x}}\right).$
(71)

Similarly, we have

$\displaystyle\frac{\widetilde{\gamma}}{N_{y}}\left\langle\text{tr}\left[(I-\widetilde{\gamma}A_{1}^{T}A_{1})B_{1}^{T}(\widetilde{\gamma}B_{1}A_{1}^{T}A_{2}-B_{2})A_{2}^{T}A_{1}\right]\right\rangle$
$\displaystyle=\frac{\widetilde{\gamma}}{N_{s}}\left\langle\text{tr}\left[(I-\widetilde{\gamma}A_{1}^{T}A_{1})(\widetilde{\gamma}A_{1}^{T}A_{2}-\rho_{b}I)A_{2}^{T}A_{1}\right]\right\rangle$
$\displaystyle=\rho_{a}\gamma(1-\gamma)(\gamma\rho_{a}-\rho_{b})+\mathcal{O}\left(\tfrac{N_{s}}{N_{x}}\right),$
(72)

and

$\displaystyle\frac{1}{N_{y}}\widetilde{\gamma}^{2}\left\langle\left\lVert(\widetilde{\gamma}B_{1}A_{1}^{T}A_{2}-B_{2})A_{2}^{T}A_{1}\right\rVert_{F}^{2}\right\rangle$
$\displaystyle=\frac{\widetilde{\gamma}^{2}}{N_{s}}\left\langle\text{tr}[A_{1}^{T}A_{2}A_{2}^{T}A_{1}]-2\widetilde{\gamma}\rho_{b}\text{tr}[A_{1}^{T}A_{2}A_{1}^{T}A_{2}A_{2}^{T}A_{1}]+\widetilde{\gamma}^{2}\text{tr}[A_{1}^{T}A_{2}A_{2}^{T}A_{1}A_{1}^{T}A_{2}A_{2}^{T}A_{1}]\right\rangle$
$\displaystyle=\rho_{a}^{2}\gamma^{2}\left(1-2\rho_{a}\rho_{b}\gamma+\rho_{a}^{2}\gamma^{2}\right)+\mathcal{O}\left(\tfrac{N_{s}}{N_{x}}\right).$
(73)

Therefore, the error is written as

$\displaystyle\left\langle\epsilon_{1}[W_{2}]\right\rangle=(1-\gamma)^{2}-2\gamma\rho_{a}(1-\gamma)(\rho_{b}-\gamma\rho_{a})+\gamma^{2}\rho_{a}^{2}(1-2\gamma\rho_{a}\rho_{b}+\gamma^{2}\rho_{a}^{2}).$
(74)

The optimal $\gamma$ for each task similarity $(\rho_{a},\rho_{b})$ and the
resultant retention performance are plotted in Figs. 8C and D.

###  C.2 Elastic weight regularization

In the previous section, we regularized the change in the weight matrix in the
Euclidean space. However, previous works indicate that the weight should be
regularized using the Fisher information matrix (FIM) as the metric [29, 65].
Let us construct an inference model by $\bm{y}=W\bm{x}+\sigma\bm{\xi}$, where
$\bm{\xi}$ is a zero mean Gaussian random variable. Given input-target pair
$\bm{x},\bm{y}$, the likelihood of weight $W$ is

$\displaystyle p(W|\bm{x},\bm{y})\propto
p(\bm{x},\bm{y}|W)p(W)\propto\exp\left(-\frac{1}{2\sigma^{2}}\left\lVert
W\bm{x}-\bm{y}\right\rVert^{2}\right).$ (75)

Thus, $(ij,kl)$-th component of FIM becomes

$\displaystyle\left\langle\frac{\partial^{2}}{\partial
w_{ij}w_{kl}}\left(-\log
p\left(W|\bm{x},\bm{y}\right)\right)\right\rangle_{\bm{s}}$
$\displaystyle=\frac{1}{2\sigma^{2}}\delta_{ik}\left\langle\sum_{m}\sum_{n}a_{jm}a_{ln}s_{m}s_{n}\right\rangle_{\bm{s}}=\frac{1}{2\sigma^{2}}\delta_{ik}\sum_{n}a_{jn}a_{ln},$
(76)

and the quadratic weight regularization in the Fisher information metric
follows

$\displaystyle\sum_{ij}\sum_{kl}\left(\delta_{ik}\sum_{n}a_{jn}a_{ln}\right)(w_{ij}-w^{o}_{ij})(w_{kl}-w^{o}_{kl})=\left\lVert(W-W_{o})A\right\rVert_{F}^{2}.$
(77)

Therefore, the loss function for this weight regularizer is written as

$\displaystyle\ell_{\mu}=\frac{1}{2}\left\lVert
B_{\mu}-WA_{\mu}\right\rVert_{F}^{2}+\frac{\lambda}{2}\left\lVert(W-W_{\mu-1})A_{\mu-1}\right\rVert_{F}^{2}.$
(78)

As we will see, this loss function has different forms of solution depending
on whether $\rho_{a}<1$ or $\rho_{a}=1$. We thus consider these two conditions
separately below.

####  C.2.1 Variable features ($\rho_{a}<1$)

Taking gradient with respect to $W$, we have

$\displaystyle\frac{\partial\ell_{\mu}}{\partial
W}=-(B_{\mu}-WA_{\mu})A_{\mu}^{T}+\lambda(W-W_{\mu-1})A_{\mu-1}A_{\mu-1}^{T},$
(79)

indicating that $W$ is written as $W=W_{\mu-1}+Q\widetilde{A}^{T}$ where $Q$
is a $N_{y}\times 2N_{s}$ matrix and $\widetilde{A}\equiv[A_{\mu-1},A_{\mu}]$
is an $N_{x}\times 2N_{s}$ matrix. Solving $\frac{\partial\ell_{\mu}}{\partial
W}=0$, we get

$\displaystyle Q\begin{pmatrix}\lambda
A_{\mu-1}^{T}A_{\mu-1}&A_{\mu-1}^{T}A_{\mu}\\\ \lambda
A_{\mu}^{T}A_{\mu-1}&A_{\mu}^{T}A_{\mu}\end{pmatrix}\begin{pmatrix}A_{\mu-1}^{T}\\\
A_{\mu}^{T}\end{pmatrix}=\begin{pmatrix}O&B_{\mu}-W_{\mu-1}A_{\mu}\end{pmatrix}\begin{pmatrix}A_{\mu-1}^{T}\\\
A_{\mu}^{T}\end{pmatrix},$ (80)

Multiplying both sides with $(A_{\mu-1}\,\,A_{\mu})$ from the right side,

$\displaystyle Q\begin{pmatrix}\lambda
A_{\mu-1}^{T}A_{\mu-1}&A_{\mu-1}^{T}A_{\mu}\\\ \lambda
A_{\mu}^{T}A_{\mu-1}&A_{\mu}^{T}A_{\mu}\end{pmatrix}\begin{pmatrix}A_{\mu-1}^{T}A_{\mu-1}&A_{\mu-1}^{T}A_{\mu}\\\
A_{\mu}^{T}A_{\mu-1}&A_{\mu}^{T}A_{\mu}\end{pmatrix}$
$\displaystyle=\begin{pmatrix}O&B_{\mu}-W_{\mu-1}A_{\mu}\end{pmatrix}\begin{pmatrix}A_{\mu-1}^{T}A_{\mu-1}&A_{\mu-1}^{T}A_{\mu}\\\
A_{\mu}^{T}A_{\mu-1}&A_{\mu}^{T}A_{\mu}\end{pmatrix}$ (81)

If $\lambda\neq 0$ and $\rho_{a}<1$, two square matrices in the above
equations are almost surely invertible under assumptions I & II. Therefore, we
have

$\displaystyle
Q=\begin{pmatrix}O&B_{\mu}-W_{\mu-1}A_{\mu}\end{pmatrix}\begin{pmatrix}\lambda
A_{\mu-1}^{T}A_{\mu-1}&A_{\mu-1}^{T}A_{\mu}\\\ \lambda
A_{\mu}^{T}A_{\mu-1}&A_{\mu}^{T}A_{\mu}\end{pmatrix}^{-1}$ (82)

Under $N_{x}\gg N_{s}$, from Eq. 99, the inverse matrix is approximated by

$\displaystyle\begin{pmatrix}\lambda
A_{\mu-1}^{T}A_{\mu-1}&A_{\mu-1}^{T}A_{\mu}\\\ \lambda
A_{\mu}^{T}A_{\mu-1}&A_{\mu}^{T}A_{\mu}\end{pmatrix}^{-1}\approx\frac{N_{s}}{N_{x}}\begin{pmatrix}\lambda
I_{s}&\rho_{a}I_{s}\\\
\lambda\rho_{a}I_{s}&I_{s}\end{pmatrix}^{-1}=\frac{N_{s}}{N_{x}(1-\rho_{a}^{2})}\begin{pmatrix}\tfrac{1}{\lambda}I_{s}&-\tfrac{\rho_{a}}{\lambda}I_{s}\\\
-\rho_{a}I_{s}&I_{s}\end{pmatrix}$ (83)

Therefore, $W=W_{\mu-1}+Q\widetilde{A}^{T}$ follows

$\displaystyle
W=W_{\mu-1}\left(I-\frac{N_{s}}{N_{x}(1-\rho_{a}^{2})}A_{\mu}\left[A_{\mu}^{T}-\rho_{a}A_{\mu-1}^{T}\right]\right)+\frac{N_{s}}{N_{x}(1-\rho_{a}^{2})}B_{\mu}\left(A_{\mu}^{T}-\rho_{a}A_{\mu-1}^{T}\right).$
(84)

Notably, the equation above doesn’t depend on the regularizer amplitude
$\lambda$, except for $\lambda\neq 0$ condition. At $\lambda\to 0$ limit, the
inverse matrix term in Eq. 82 effectively becomes singular, and thus the
equation above no longer holds.

Let us suppose the first task is learned without any weight regularization, or
learned with weight regularization in the Fisher information metric imposed by
an uncorrelated task. Then, we have
$W_{1}=\tfrac{N_{s}}{N_{x}}B_{1}A_{1}^{T}$. Hence, the transfer performance
becomes the same with the vanilla model. The weight after the second task
becomes

$\displaystyle
W_{2}=\frac{N_{s}}{N_{x}}B_{1}A_{1}^{T}\left(I-\frac{N_{s}}{N_{x}(1-\rho_{a}^{2})}A_{2}[A_{2}^{T}-\rho_{a}A_{1}^{T}]\right)+\frac{N_{s}}{N_{x}(1-\rho_{a}^{2})}B_{2}(A_{2}^{T}-\rho_{a}A_{1}^{T}).$
(85)

Thus, the retention performance under $W_{2}$ follows

$\displaystyle\Delta\epsilon_{RT}$
$\displaystyle=\frac{1}{N_{y}}\left\langle\left\lVert
B_{1}\right\rVert_{F}^{2}\right\rangle$
$\displaystyle\quad-\frac{1}{N_{y}}\left\langle\left\lVert
B_{1}\left(I-\tfrac{N_{s}}{N_{x}}A_{1}^{T}A_{1}\right)+\tfrac{N_{s}}{N_{x}(1-\rho_{a}^{2})}\left(\tfrac{N_{s}}{N_{x}}B_{1}A_{1}^{T}A_{2}-B_{2}\right)(A_{2}^{T}-\rho_{a}A_{1}^{T})A_{1}\right\rVert_{F}^{2}\right\rangle.$
(86)

Taking expectation over $B_{1}$ and $B_{2}$, $\Delta\epsilon_{RT}$ is
rewritten as

$\displaystyle\Delta\epsilon_{RT}$
$\displaystyle=1-\tfrac{1}{N_{s}}\left\langle\left\lVert
I-\tfrac{N_{s}}{N_{x}}A_{1}^{T}A_{1}\right\rVert_{F}^{2}\right\rangle$
$\displaystyle\quad-\tfrac{2}{N_{x}(1-\rho_{a}^{2})}\left\langle\text{tr}\left[(I-\tfrac{N_{s}}{N_{x}}A_{1}^{T}A_{1}^{T})\left(\tfrac{N_{s}}{N_{x}}A_{1}^{T}A_{2}-\rho_{b}I\right)\left(A_{2}^{T}-\rho_{a}A_{1}^{T}\right)A_{1}\right]\right\rangle$
$\displaystyle\quad-\tfrac{1}{N_{s}}\left(\tfrac{N_{s}}{N_{x}(1-\rho_{a}^{2})}\right)^{2}\left\langle\left\lVert\tfrac{N_{s}}{N_{x}}A_{1}^{T}A_{2}(A_{2}^{T}-\rho_{a}A_{1}^{T})A_{1}\right\rVert_{F}^{2}+\left\lVert(A_{2}^{T}-\rho_{a}A_{1}^{T})A_{1}\right\rVert_{F}^{2}\right\rangle$
$\displaystyle\quad+\tfrac{2\rho_{b}}{N_{x}}\left(\tfrac{N_{s}}{N_{x}(1-\rho_{a}^{2})}\right)^{2}\left\langle\text{tr}\left[A_{1}^{T}(A_{2}-\rho_{a}A_{1})A_{2}^{T}A_{1}(A_{2}^{T}-\rho_{a}A_{1}^{T})A_{1}\right]\right\rangle.$
(87)

The second term is rewritten as

$\displaystyle\tfrac{1}{N_{s}}\left\langle\left\lVert
I-\tfrac{N_{s}}{N_{x}}A_{1}^{T}A_{1}\right\rVert_{F}^{2}\right\rangle=1-\tfrac{2}{N_{x}}\left\langle\text{tr}[A_{1}^{T}A_{1}]\right\rangle+\tfrac{1}{N_{s}}\left(\tfrac{N_{s}}{N_{x}}\right)^{2}\left\langle\text{tr}[A_{1}^{T}A_{1}A_{1}^{T}A_{1}]\right\rangle=\tfrac{N_{s}+1}{N_{x}}.$
(88)

Moreover, $(A_{2}^{T}-\rho_{a}A_{1}^{T})A_{1}$ term also cancels out up to the
leading order term because

$\displaystyle\tfrac{1}{N_{s}}\left(\tfrac{N_{s}}{N_{x}(1-\rho_{a}^{2})}\right)^{2}\left\langle\left\lVert(A_{2}^{T}-\rho_{a}A_{1}^{T})A_{1}\right\rVert_{F}^{2}\right\rangle$
$\displaystyle=\tfrac{1}{N_{s}}\left(\tfrac{N_{s}}{N_{x}(1-\rho_{a}^{2})}\right)^{2}\sum_{ijkl}\left\langle
a^{(1)}_{ji}(a^{(2)}_{jk}-\rho_{a}a^{(1)}_{jk})(a^{(2)}_{lk}-\rho_{a}a^{(1)}_{lk})a^{(1)}_{li}\right\rangle$
$\displaystyle=\tfrac{1}{N_{s}}\left(\tfrac{N_{s}}{N_{x}(1-\rho_{a}^{2})}\right)^{2}\sum_{ijkl}\left\langle
a^{(1)}_{ji}a^{(1)}_{li}\right\rangle\left\langle(a^{(2)}_{jk}-\rho_{a}a^{(1)}_{jk})(a^{(2)}_{lk}-\rho_{a}a^{(1)}_{lk})\right\rangle$
$\displaystyle=\tfrac{1}{N_{s}}\left(\tfrac{N_{s}}{N_{x}(1-\rho_{a}^{2})}\right)^{2}\sum_{ijkl}\delta_{jl}\tfrac{1-\rho_{a}^{2}}{N_{s}^{2}}=\tfrac{N_{s}}{N_{x}(1-\rho_{a}^{2})}$
(89)

Similarly, the trace terms are also cancelled out up to the leading order
term:

$\displaystyle\tfrac{1}{N_{x}}\left(\tfrac{N_{s}}{N_{x}(1-\rho_{a}^{2})}\right)^{2}\left\langle\text{tr}[A_{1}^{T}(A_{2}^{T}-\rho_{a}A_{1})A_{2}^{T}A_{1}(A_{2}^{T}-\rho_{a}A_{1}^{T})A_{1}]\right\rangle$
$\displaystyle=\tfrac{1}{N_{x}}\left(\tfrac{N_{s}}{N_{x}(1-\rho_{a}^{2})}\right)^{2}\sum_{ijklmn}\left\langle
a^{(1)}_{ji}(a^{(2)}_{jk}-\rho_{a}a^{(1)}_{jk})a^{(2)}_{lk}a^{(1)}_{lm}(a^{(2)}_{nm}-\rho_{a}a^{(1)}_{nm})a^{(1)}_{ni}\right\rangle$
$\displaystyle=\tfrac{1}{N_{x}}\left(\tfrac{N_{s}}{N_{x}(1-\rho_{a}^{2})}\right)^{2}\sum_{ijklmn}\left\langle(a^{(2)}_{jk}-\rho_{a}a^{(1)}_{jk})(a^{(2)}_{nm}-\rho_{a}a^{(1)}_{nm})\right\rangle\left\langle
a^{(1)}_{ji}a^{(2)}_{lk}a^{(1)}_{lm}a^{(1)}_{ni}\right\rangle$
$\displaystyle=\tfrac{1}{N_{x}}\left(\tfrac{N_{s}}{N_{x}(1-\rho_{a}^{2})}\right)^{2}\sum_{ijklmn}\tfrac{1}{N_{s}}\delta_{jn}\delta_{km}(1-\rho_{a}^{2})\tfrac{1}{N_{s}^{2}}\left(\rho_{a}+\delta_{jl}\delta_{ik}2\rho_{a}\right)$
$\displaystyle=\tfrac{N_{s}\rho_{a}}{N_{x}(1-\rho_{a}^{2})}\left(1+\tfrac{2}{N_{x}N_{s}}\right),$
(90)

and

$\displaystyle\tfrac{1}{N_{x}(1-\rho_{a}^{2})}\left\langle\text{tr}[(I-\tfrac{N_{s}}{N_{x}}A_{1}^{T}A_{1})(\tfrac{N_{s}}{N_{x}}A_{1}^{T}A_{2}-\rho_{b}I)(A_{2}^{T}-\rho_{a}A_{1}^{T})A_{2}]\right\rangle$
$\displaystyle=\tfrac{1}{N_{x}(1-\rho_{a}^{2})}\sum_{ijkl}\left\langle\left(\delta_{ij}-\tfrac{N_{s}}{N_{x}}\sum_{m}a^{(1)}_{mi}a^{(1)}_{mj}\right)\left(\tfrac{N_{s}}{N_{x}}\sum_{n}a^{(1)}_{nj}a^{(2)}_{nk}-\rho_{b}\delta_{jk}\right)\left(a^{(2)}_{lk}-\rho_{a}a^{(1)}_{lk}\right)a^{(2)}_{li}\right\rangle=0.$
(91)

Thus, we get
$\Delta\epsilon_{RT}=1-\mathcal{O}\left(\tfrac{N_{s}}{N_{x}(1-\rho_{a}^{2})}\right)$.
Therefore, if $N_{s}\ll(1-\rho_{a}^{2})N_{x}$, we have
$\Delta\epsilon_{RT}\approx 1$ regardless of task similarity $\rho_{a}$ and
$\rho_{b}$.

####  C.2.2 Fixed features ($\rho_{a}=1$)

When $\rho_{a}=1$, $A_{\mu-1}=A_{\mu}=A$. The gradient follows

$\displaystyle\frac{\partial\ell_{\mu}}{\partial
W}=\left[-\left(B_{\mu}-WA\right)+\lambda(W-W_{\mu-1})A\right]A^{T},$ (92)

and the weight $W$ is written as $W=W_{\mu-1}+QU^{T}$, where $U$ is defined by
SVD of A: $A=U\Lambda V^{T}$. Solving $\frac{\partial\ell_{\mu}}{\partial
W}=0$, we get

$\displaystyle
W=W_{\mu-1}\left(I-\frac{1}{1+\lambda}UU^{T}\right)+\frac{1}{1+\lambda}B_{\mu}A^{+}$
(93)

Thus, the weight after task 1 and 2 follow

$\displaystyle W_{1}=\frac{1}{1+\lambda}B_{1}A^{+},\quad
W_{2}=\frac{1}{1+\lambda}B_{1}A^{+}\left(I-\frac{1}{1+\lambda}UU^{T}\right)+\frac{1}{1+\lambda}B_{2}A^{+}.$
(94)

Therefore, the transfer performance is

$\displaystyle\Delta\epsilon_{TF}$
$\displaystyle=\frac{1}{N_{y}}\left\langle\left\lVert
B_{2}\right\rVert^{2}\right\rangle-\frac{1}{N_{y}}\left\langle\left\lVert
B_{2}-W_{1}A_{2}\right\rVert^{2}\right\rangle$
$\displaystyle=\frac{2\rho_{b}}{1+\lambda}-\frac{1}{(1+\lambda)^{2}}=\gamma_{f}\left(2\rho_{b}-\gamma_{f}\right).$
(95)

In the last line, we defined $\gamma_{f}$ by
$\gamma_{f}\equiv\frac{1}{1+\lambda}$. Similarly, the retention performance is
written as

$\displaystyle\Delta\epsilon_{RT}$
$\displaystyle=\frac{1}{N_{y}}\left\langle\left\lVert
B_{1}\right\rVert_{F}^{2}\right\rangle-\frac{1}{N_{y}}\left\langle\left\lVert
B_{1}-W_{2}A\right\rVert_{F}^{2}\right\rangle$
$\displaystyle=\frac{1}{N_{y}}\left\langle\left\lVert
B_{1}\right\rVert_{F}^{2}\right\rangle-\frac{1}{N_{y}}\left\langle\left\lVert\left(1-\frac{\lambda}{(1+\lambda)^{2}}\right)B_{1}-\frac{1}{1+\lambda}B_{2}\right\rVert_{F}^{2}\right\rangle$
$\displaystyle=2\gamma_{f}(1-\gamma_{f})^{2}-\gamma_{f}^{4}+2\gamma_{f}\left[(1-\gamma_{f})^{2}-\gamma_{f}^{2}\right]\rho_{b}.$
(96)

If the first task is learned without weight regularization, the retention
performance instead becomes:

$\displaystyle\Delta\epsilon_{RT}=\frac{1}{N_{y}}\left\langle\left\lVert
B_{1}\right\rVert_{F}^{2}\right\rangle-\frac{1}{N_{y}}\left\langle\left\lVert\frac{1}{1+\lambda}(B_{1}-B_{2})\right\rVert_{F}^{2}\right\rangle=1-2\gamma_{f}^{2}(1-\rho_{b}).$
(97)

##  Appendix D Properties of very tall zero-mean random matrices

Our simple analytical results rely on the assumption that matrix
$A\in\mathbb{R}^{N_{x}\times N_{s}}$ is very tall (i.e., $N_{x}\gg N_{s}$),
and each element of $A$ is sampled independently from a normal distribution
with mean zero and a finite variance.

Given $A_{ij}\sim\mathcal{N}(0,\tfrac{1}{N_{s}})$, we have
$\left\langle\frac{1}{N_{x}}A^{T}A\right\rangle_{A}=\tfrac{1}{N_{s}}I_{s}$.
The element-wise deviation from the mean is evaluated as

$\displaystyle\left\langle\left(\left[\tfrac{1}{N_{x}}A^{T}A-\tfrac{1}{N_{s}}I_{s}\right]_{ij}\right)^{2}\right\rangle_{A}=\frac{1+\delta_{ij}}{N_{x}N_{s}^{2}},$
(98)

implying that
$\left\langle\left\lVert\tfrac{1}{N_{x}}A^{T}A-\tfrac{1}{N_{s}}I_{s}\right\rVert_{F}^{2}\right\rangle_{A}=\tfrac{1}{N_{x}}\left(1+\tfrac{1}{N_{s}}\right)$.
Therefore, at $N_{x}\gg N_{s}$ limit, $\tfrac{1}{N_{x}}A^{T}A$ approximately
follows

$\displaystyle\tfrac{1}{N_{x}}A^{T}A\approx\tfrac{1}{N_{s}}I_{s}.$ (99)

Even in the presence of random gating, if the effective matrix is still tall
(i.e., $\alpha N_{x}\gg N_{s}$), similar approximations hold. Given a diagonal
matrix $D$ whose diagonal components are sampled independently from a
Bernoulli distribution with rate $\alpha$, at $\alpha N_{x}\gg N_{s}$ limit,
we have

$\displaystyle\tfrac{1}{\alpha N_{x}}(DA)^{T}DA\approx\tfrac{1}{N_{s}}I_{s}.$
(100)

This result implies that,

$\displaystyle(DA)^{+}\approx\tfrac{N_{s}}{\alpha N_{x}}(DA)^{T}.$ (101)

Moreover, denoting the SVD of $DA$ by $DA=U\Sigma V^{T}$, at $\alpha N_{x}\gg
N_{s}$ limit, we have

$\displaystyle UU^{T}\approx\tfrac{N_{s}}{\alpha N_{x}}DAA^{T}D.$ (102)

This is because, at $\alpha N_{x}\gg N_{s}$ limit, from Eq. 100, the
eigenspectrum of matrix $\tfrac{1}{\alpha N_{x}}DA(DA)^{T}$ is concentrated at
$\lambda=\tfrac{1}{N_{s}}$, and thus,

$\displaystyle\tfrac{N_{s}}{\alpha N_{x}}DAA^{T}D=\tfrac{N_{s}}{\alpha
N_{x}}U\Sigma^{2}U^{T}\approx UU^{T}.$ (103)

##  Appendix E Numerical methods

Numerical experiments were conducted in standard laboratory GPUs and CPUs.
Source codes for all numerical results are made publicly available at
https://github.com/nhiratani/transfer_retention_model.

###  E.1 Implementation of linear teacher-student models

Numerical results in Figs. 1-5 and 7-8 were implemented as below. We set the
latent variable dimensionality $N_{s}=30$, the input width $N_{x}=3000$, and
the output width $N_{y}=10$. The student weight $W$ was initialized as the
zero matrix, and updated with the full gradient descent with learning rate
$\eta=0.001$. We trained the network with the first task for $N_{epoch}$
epochs, and then retrained it with the second task for another $N_{epoch}$
epochs. We used $N_{epoch}=100$ for the vanilla model and the activity gating
models, but we used $N_{epoch}=500$ for other models. Unless stated otherwise,
error bars represent standard deviation over 10 random seeds. Average
performances in Figs. 2C, 3B-D, 4D, and 5CD were estimated by taking average
over 100 pairs of ($\rho_{a},\rho_{b}$), sampled uniformly from
$\rho_{a},\rho_{b}\in[0,1]$.

In the input soft-thresholding model, we estimated the gradient in a sample-
based manner because the exact gradient is not tractable due to nonlinearity.
We estimated the gradient from 10000 random samples at each iterations, then
updated the model for 5000 iterations with learning $\eta=0.01$.

###  E.2 Implementation of permuted MNIST with latent

Figure 9: Permuted MNIST with a latent variable. (A) Learning curves under
$(\rho_{a},\rho_{b})=(0.8,0.2)$. (B) Knowledge transfer under various task
similarities. (C) Comparison of the retention performance measured after 10
epochs (black) and 100 epochs (gray) of training. (D) Retention performance
after 100 epochs of training with task 2. After 100 epochs of training, the
system still exhibits strong asymmetric dependence on the feature and readout
similarities, but the non-monotonic dependence on the feature similarity under
$\rho_{b}\approx 1$ region disappears. (E,F) Retention performance under the
weight regularization in the Fisher information metric (FIM) with a layer-wise
approximation (panel E) and a diagonal approximation (panel F), under
$(\rho_{a},\rho_{b})=(0.5,0.5)$. Unlike Figs. 6F-H in the main figure, we
adjusted the regularizer amplitude of two layers independently. As in Fig. 6,
error bars in panel A-C represent standard errors over 10 random seeds.

##### Data generation

We used permuted MNIST dataset [33, 21], a common benchmark for continual
learning, but with addition of the latent space. We constructed a four
dimensional latent space $\bm{s}$ using the binary representation of digits,
$\bm{s}_{0}=[0,0,0,0]^{T}$, $\bm{s}_{1}=[0,0,0,1]^{T}$, and so on. The target
output was generated by a projection of the latent variable $\bm{s}$ to a ten-
dimensional space, $\bm{y}^{*}=B(\bm{s}-\tfrac{1}{2}\bm{1})$, where $B$ is a
$10\times 4$ matrix generated randomly. The elements of matrix $B_{1}$ for the
first task were sampled independently from a Gaussian distribution with mean
zero and variance one. For the second task, we introduced readout similarity
by keeping some of the elements while resampling other elements. We defined
the readout similarity $\rho_{b}$ by the fraction of the elements kept the
same between two tasks.

Feature similarity was introduced by permuting the input pixels as done
previously [21]. We used the vanilla MNIST images for the first task and
permuted some of the pixels in the second task depending on the feature
similarity $\rho_{a}$.

##### Model implementation

We used one hidden layer neural network with ReLU non-linearity in the hidden
layer:
$\bm{y}=\bm{b}_{2}+W_{2}\text{ReLU}\left[\bm{b}_{1}+W_{1}\bm{x}\right]$. We
set the hidden layer width to $N_{h}=1500$. The input and output widths were
set to $N_{x}=784$ and $N_{y}=10$. We initialized the first weight $W_{1}$ and
the bias $\bm{b}_{1}$ by sampling each weight independently from a Gaussian
distribution with mean zero and variance $\tfrac{1}{N_{h}^{2}}$. We
initialized the second weight $W_{2}$ and the bias $\bm{b}_{2}$ in the same
manner, but with variance $\tfrac{1}{N_{y}^{2}}$. Here, we set the amplitude
of the initial weights small to operate the learning in the rich regime [51].
We set the mini-batch size to $300$ and the learning rate to $\eta=0.01$, and
trained the network for $100$ epochs per task. The retention performance was
measured after $10$ epochs of training with task 2 except for Figs. 9 C and D.
In the learning of task 2, the test error typically drops significantly in the
first 10 epochs and shifts to gradual improvement afterward (red line in Fig.
9A). Notably, the asymmetric task similarity dependence of the retention
performance was observed consistently both after a short and long training on
the second task (Figs. 6B and 9D).

##### Weight regularization in the Fisher information metric

From the same argument made in section C.2, FIM of a noise-free model is
approximated by the Hessian of the mean squared error. Given
$\bm{y}=\bm{b}_{2}+W_{2}\phi(\bm{b}_{1}+W_{1}\bm{x})$ and
$\ell=\tfrac{1}{2}\left\lVert\bm{y}-\bm{y}^{*}\right\rVert^{2}$, the Hessian
is estimated as

$\displaystyle\frac{\partial^{2}\ell}{\partial w^{(2)}_{ij}\partial
w^{(2)}_{kl}}$ $\displaystyle=\delta_{ik}\phi_{j}\phi_{l},$ (104a)
$\displaystyle\frac{\partial^{2}\ell}{\partial w^{(2)}_{ij}\partial
w^{(1)}_{kl}}$
$\displaystyle=w^{(2)}_{ik}\phi^{\prime}_{k}x_{l}\phi_{j}+\delta_{jk}(y_{i}-y^{*}_{i})\phi^{\prime}_{j}x_{l},$
(104b) $\displaystyle\frac{\partial^{2}\ell}{\partial w^{(1)}_{ij}\partial
w^{(1)}_{kl}}$
$\displaystyle=\sum_{m}w^{(2)}_{mi}\phi^{\prime}_{i}x_{j}w^{(2)}_{mk}\phi^{\prime}_{k}x_{l}+\delta_{ik}\sum_{m}(y_{m}-y_{m}^{*})w^{(2)}_{mi}\phi^{\prime\prime}_{i}x_{j}x_{l}.$
(104c)

Under the ReLU activation, the second-order derivative $\phi^{\prime\prime}$
is effectively negligible, thus the weight regularization in the Fisher
information metric is written as

$\displaystyle R[W_{1},W_{2};(\bm{x},\bm{y}^{*})]$
$\displaystyle=\sum_{ijkl}\tfrac{\partial^{2}\ell}{\partial
w^{(2)}_{ij}\partial w^{(2)}_{kl}}\delta w^{(2)}_{ij}\delta
w^{(2)}_{kl}+2\sum_{ijkl}\tfrac{\partial^{2}\ell}{\partial
w^{(2)}_{ij}\partial w^{(1)}_{kl}}\delta w^{(2)}_{ij}\delta
w^{(1)}_{kl}+\tfrac{\partial^{2}\ell}{\partial w^{(1)}_{ij}\partial
w^{(1)}_{kl}}\delta w^{(1)}_{ij}\delta w^{(1)}_{kl}$
$\displaystyle=\left\lVert\delta
W_{2}\phi\right\rVert^{2}+2\text{tr}\left[\delta
W_{2}^{T}W_{2}\nabla\phi\delta
W_{1}\bm{x}\phi^{T}\right]+2\text{tr}\left[\delta W_{2}\nabla\phi\delta
W_{1}\bm{x}\delta\bm{y}^{T}\right]+\left\lVert W_{2}\nabla\phi\delta
W_{1}\bm{x}\right\rVert^{2},$ (105)

where $\delta W_{1}\equiv W_{1}-W_{1}^{o}$ and $\delta W_{2}\equiv
W_{2}-W_{2}^{o}$ are the deviations from the target weights,
$\delta\bm{y}=\bm{y}-\bm{y}^{*}$, and $\nabla\phi$ is a diagonal matrix where
diagonal components are derivative of $\phi$. In the layer-wise approximation,
we omitted the trace-terms originated from the cross terms, which yields

$\displaystyle R_{lw}[W_{1},W_{2}]=\frac{1}{2}\left\langle\left\lVert\delta
W_{2}\phi\right\rVert^{2}\right\rangle+\frac{1}{2}\left\langle\left\lVert
W_{2}\nabla\phi\delta W_{1}\bm{x}\right\rVert^{2}\right\rangle,$ (106)

where the expectation is taken over training data $(\bm{x},\bm{y}^{*})$
generated from the previous task. Thus, the derivative with respect to $\delta
W_{1}$ and $\delta W_{2}$ follows

$\displaystyle\frac{\partial R_{lw}}{\partial(\delta W_{1})}$
$\displaystyle=\left\langle\nabla\phi W_{2}^{T}W_{2}\nabla\phi\delta
W_{1}\bm{x}\bm{x}^{T}\right\rangle\approx\left\langle\nabla\phi
W_{2}^{T}W_{2}\nabla\phi\right\rangle\delta
W_{1}\left\langle\bm{x}\bm{x}^{T}\right\rangle,$ (107a)
$\displaystyle\frac{\partial R_{lw}}{\partial(\delta W_{2})}$
$\displaystyle=\delta W_{2}\left\langle\phi\phi^{T}\right\rangle.$ (107b)

In the first line, we further decoupled the expectations for computational
tractability. In the numerical implementation, we scaled the regularization
term on the weight changes $\Delta W_{1}$ and $\Delta W_{2}$ by
$Z_{1}=\left\lVert
W_{2}W_{2}^{T}\right\rVert_{2}\left\lVert\left\langle\bm{x}\bm{x}^{T}\right\rangle\right\rVert_{2}$
and $Z_{2}=\left\lVert\left\langle\phi\phi^{T}\right\rangle\right\rVert_{2}$,
respectively, where $\left\lVert\cdot\right\rVert_{2}$ is the spectral norm
and the expectation was calculated at the last weight of the previous task. We
did not impose weight regularization on the bias parameters $\bm{b}_{1}$ and
$\bm{b}_{2}$.

In the case of diagonal approximation, we used the following regularizer:

$\displaystyle
R_{diag}[W_{1},W_{2}]=\frac{1}{2Z_{1}}\sum_{ij}\sum_{k}\left(w^{(2)}_{ki}\phi^{\prime}_{i}x_{j}\right)^{2}\left(\delta
w^{(1)}_{ij}\right)^{2}+\frac{1}{2Z_{2}}\sum_{ij}\phi_{j}^{2}\left(\delta
w^{(2)}_{ij}\right)^{2},$ (108)

where the scaling factors $Z_{1}$ and $Z_{2}$ were defined by
$Z_{1}=\max_{ij}\left\langle\sum_{k}(w^{(2)}_{ki}\phi^{\prime}_{i}x_{j})^{2}\right\rangle$
and $Z_{2}=\max_{j}\left\langle\phi_{j}^{2}\right\rangle$.

To see the potential effect of different normalization methods used for the
layer-wise and diagonal approximations, in Figs. 9 E and F, we independently
changed the regularizer amplitude for $W_{1}$ and $W_{2}$ and measured the
retention performance under the two approximation methods. Even in this
setting, the layer-wise approximation of the Fisher-information metric
robustly outperformed the diagonal approximation, when the regulazier
amplitude for $W_{2}$ was sufficiently large.