# Elastic Multi-Gradient Descent
for Parallel Continual Learning

Fan Lyu∗, , Wei Feng∗, , Yuepan Li, Qing Sun,
Fanhua Shang, , Liang Wan, , Liang Wang F. Lyu and L. Wang is with the Center
for Research on Intelligent Perception and Computing (CRIPAC) Institute of
Automation, Chinese Academy of Sciences (CASIA), Beijing, 100190 China.
Contacts<EMAIL_ADDRESS><EMAIL_ADDRESS>Feng, Y. Li, Q.
Sun, F. Shang, L. Wan are with the College of Intelligence and Computing,
Tianjin University, Tianjin, 300350 China. Contacts: ({yuepanli, sssunqing,
fhshang<EMAIL_ADDRESS>wfeng@ieee.org)*Equal Contribution.This study was
done when F. Lyu was working for his PhD in Tianjin University.The
corresponding author is Wei Feng.Manuscript received ** **, 20**; revised
August **, **.

###### Abstract

The goal of Continual Learning (CL) is to continuously learn from new data
streams and accomplish the corresponding tasks. Previously studied CL assumes
that data are given in sequence nose-to-tail for different tasks, thus indeed
belonging to Serial Continual Learning (SCL). This paper studies the novel
paradigm of Parallel Continual Learning (PCL) in dynamic multi-task scenarios,
where a diverse set of tasks is encountered at different time points. PCL
presents challenges due to the training of an unspecified number of tasks with
varying learning progress, leading to the difficulty of guaranteeing effective
model updates for all encountered tasks. In our previous conference work, we
focused on measuring and reducing the discrepancy among gradients in a multi-
objective optimization problem, which, however, may still contain negative
transfers in every model update. To address this issue, in the dynamic multi-
objective optimization problem, we introduce task-specific elastic factors to
adjust the descent direction towards the Pareto front. The proposed method,
called Elastic Multi-Gradient Descent (EMGD), ensures that each update follows
an appropriate Pareto descent direction, minimizing any negative impact on
previously learned tasks. To balance the training between old and new tasks,
we also propose a memory editing mechanism guided by the gradient computed
using EMGD. This editing process updates the stored data points, reducing
interference in the Pareto descent direction from previous tasks. Experiments
on public datasets validate the effectiveness of our EMGD in the PCL setting.

###### Index Terms:

parallel continual learning, catastrophic forgetting, training conflict,
multi-objective optimization, Pareto optimum.

##  I Introduction

How to learn a single model for multiple tasks from multiple data sources has
been a long-standing research topic [53, 39, 9], with applications in various
fields, such as recommendation systems [85] and machine translation [81]. Two
major paradigms (i.e., Multi-Task Learning (MTL) and Continual Learning (CL))
have emerged for learning a unified model, and they differ in terms of their
approach to data access. Fig. 1(a) and (b) provide an overview of MTL and CL.

Figure 1:  Comparisons of Multi-Task Learning (MTL), Serial Continual Learning
(SCL) and Parallel Continual Learning (PCL). (a) MTL relies on a fixed number
of different tasks without adaptive incremental learning for new tasks. (b)SCL
learns from sequential multiple tasks, where new tasks have to wait for the
completion of previous training. (c) PCL enables training of multiple tasks
based on their access time, allowing new tasks to be incorporated immediately.

MTL [76, 57, 31] receives a fixed number of data pipes (from one to many) for
different tasks, where all tasks are trained in parallel at the same time. The
challenge of MTL is the competition between tasks, i.e., task conflict. To
address the task conflict, traditional MTL solutions can be mainly grouped
into feature-based [55, 74] and parameter-based [72, 10, 36] approaches. In
recent years, some studies focus on reweighting multi-task training for task
balancing, which can be categorized into three kinds, including learning-based
methods [19, 45], solving-based methods [66], and calculating-based methods
[46, 20, 79, 33, 42].

Different from MTL, CL [37, 40, 50] aims to continuously learn new knowledge
from a sequence of tasks with non-overlapping data streams over a lifelong
time. Because the data of each task comes in sequence, the challenge of CL is
catastrophic forgetting of old tasks or finished tasks. The existing CL
methods are grouped into three types[22], i.e., regularization-based methods
[37, 15, 24, 80, 1, 28, 27], architecture-based methods [54, 77, 65, 64]. and
rehearsal-based methods [50, 17, 34, 6, 68, 58, 52]. Specifically, rehearsal
has been proven effective against forgetting, which stores a small sample from
old tasks and retrains them together with new tasks. In fact, both MTL and CL
achieve the goal of a single model for multiple tasks, where the difference
between them is the data form as shown in Fig. 1.

Despite the success of MTL and traditional CL in learning a unified model for
multiple tasks, they are still insufficient for learning in a changing
environment with multiple data sources. On one hand, classical MTL never
considers new tasks or environments access in sequence or incrementally,
leading to poor adaptation to new knowledge. On the other hand, CL can hardly
achieve an immediate response to new task demands, new task data streams can
only be learned after the previous task training is finished. We refer to this
type of continual learning as _Serial Continual Learning_ (SCL). To address
the limitations of MTL and CL, we propose a more general and practical
scenario, where new tasks can be accessed and trained at any time, together
with any in-training tasks.

In our previous conference work [51], we investigated _Parallel Continual
Learning_ (PCL) that cuts off the one-pipe data stream setting in SCL and
makes incremental adaption in MTL. In PCL, a new task demand can arise at any
time, allowing for instant response to new task demands without waiting for
pending tasks. PCL satisfies the requirements of continual learning and
immediate response by enabling new tasks to be trained immediately and in
parallel with other in-training tasks, as depicted in Fig.1(c). PCL faces
challenges inherited from both MTL and SCL: task conflict and catastrophic
forgetting. The diverse training processes across different tasks can lead to
the task conflict issue [66], in which back-propagation may not guarantee all
in-training tasks to converge to their respective optimum. Meanwhile, the
catastrophic forgetting issue [32, 37] arises due to the unavailability of
finished task data [7]. Notably, these two issues are more complex than those
encountered in traditional MTL and SCL, as the access time for new tasks is
uncertain and may occur at any time.

Nonetheless, our previous study [51] only tries to balance multi-tasks in an
intuitive way, and ignores the negative transfer in updates, which can not
satisfy the Pareto direction.  In this paper, we address the challenges of
catastrophic forgetting and training conflict in PCL by formulating it as a
dynamic MTL problem, and seek to constrain the update towards the Pareto
front. To this end, we propose a simple yet effective approach called Elastic
Multi-Gradient Descent (EMGD). EMGD builds on the well-established steepest
descent method [30] in Multi-Objective Optimization (MOO) and adds an elastic
factor for each task. The elastic factor is utilized to mitigate the
discrepancy between the gradient of each task and the under-optimized
gradient, and can be evaluated by gradient magnitude changes and gradient
similarities. EMGD can be easily solved in its dual form with quadratic
programming. We also prove that EMGD can converge to a Pareto critical point
for long-term training. Additionally, we propose to edit the stored data with
the guidance of the Pareto descent direction obtained from EMGD. By performing
such memory editing, the potential task conflict between old and new tasks can
be reduced, and the memory can be utilized in subsequent training.

Our previous conference work [51] focused on measuring the discrepancy among
multiple gradients. We proposed an explicit measurement for the learning from
gradient discrepancy in PCL, named asymmetric gradient distance, which
considers gradient magnitude ratios and directions, and sets a tolerance for
smaller gradients. Moreover, an effective optimization strategy was proposed
for minimizing the gradient discrepancy to avoid self-interference. We name
the strategy Maximum Discrepancy Optimization (MaxDO), which minimizes the
maximum discrepancy from each gradient to the others. In comparison with our
previous work, this work aims to further require a Pareto descent direction
achieved from the varied gradients. To achieve this, in this paper, we
transform the PCL task into a dynamic MOO problem, and propose a simple yet
effective elastic multi-gradient descent method, which guarantees the training
of new tasks while reducing the catastrophic forgetting of previous methods by
maintaining the Pareto descent direction in each update. Compared to the
previous work [51], the proposed EMGD method has the following three
advantages: (1) Pareto descent direction in MOO. This paper is inspired by the
classic MGDA and proposes to control the Pareto descent direction, which will
never worsen any training task. (2) Better rehearsal using Pareto descent
direction. This paper proposes to edit the stored memory for better rehearsal,
which mitigates the task conflicts within the stored samples, and reduces the
catastrophic fgetting. (3) Better performance. We show more experimental
comparisons with MTL, SCL and our previous work [51] and achieve new state-of-
the-art results in PCL. In our experiments, to evaluate the proposed method,
we construct multiple parallel data streams using three popular image
classification datasets, where access time for each task varied. Our
observations show that EMGD can ensure both new task training and old task
retention when a new task is introduced.

Our contributions are three-fold:

  1. 1.

We propose PCL, a general scenario to achieve a single model for multiple data
sources and tasks. In contrast to MTL and SCL, PCL conducts adaptive training
based on the continuously received multiple data streams. At any time, PCL
trains an unfixed number of tasks using available data from these streams

  2. 2.

We present EMGD, a solution to resolve dynamic task conflicts in PCL, which
transforms the PCL task into a dynamic MOO problem. To solve this problem, we
introduce task-specific elastic factors. EMGD ensures both new task training
and old task preservation in PCL.

  3. 3.

We propose a memory editing technique that relies on the target gradient
obtained from EMGD, which edits the pixel-level information to mitigate task
conflicts within the stored memory, as well as catastrophic forgetting.

##  II Related Work

###  II-A Multi-Task Learning

Multi-Task Learning (MTL) [76, 57] is inspired by the human ability to learn
multiple tasks simultaneously, addressing sharing information about different
problems in the same [43, 66, 48, 25] or different [70, 56, 71, 60] domains.
In most situations of MTL, the multiple tasks refer to different outputs of
the model, such as different classifiers in classification tasks or
segmentation and detection tasks in computer vision. These tasks may have
inter-dependencies that can benefit each other. In such scenarios, multi-task
learning allows tasks to reuse knowledge from other tasks, reducing the cost
of manual annotation.

MTL is based on the fundamental assumption that multiple tasks, when learned
together, can mutually reinforce or constrain each other, ultimately leading
to the achievement of complex multi-task recognition capabilities [84]. In
MTL, simultaneously learning multiple tasks can be more challenging. Unlike
single-task learning, there may be task conflicts among multiple tasks,
manifesting as each task in multi-task learning not performing as well as it
would in isolated single-task training. In such cases, improving the
performance of one task may lead to a decline in the performance of other
tasks, a phenomenon referred to as Negative Transfer [83]. Mitigating the
impact of negative transfer and reducing training conflicts are among the
primary objectives in multi-task learning. In summary, traditional MTL
solutions can be mainly grouped into three kinds [84] as follows.

(1) Feature-based methods generally assume that multiple tasks are related,
with the core focus being on learning shared representations for different
tasks [55, 74]. These shared representations encapsulate the shared knowledge
among multiple tasks. By leveraging this shared knowledge for each task, it
strengthens its representation capabilities, leading to improved performance.
Based on the relationship between the original feature representation and the
learned feature representation, feature-based multi-task learning can be
further divided into two categories [84]: feature transformation methods and
feature selection methods. Feature transformation methods [41, 69, 5, 47]
suggest that shared features among multiple tasks should be transformed
through a model, while feature selection methods [44, 38] assume that there is
redundancy in the feature information and aim to learn appropriate
combinations as new feature representations for all tasks.

(2) Parameter-based methods utilize model parameters from any task to assist
in learning the model parameters for other tasks. These methods can be
primarily categorized into four approaches [84]: low-rank methods, task
clustering methods, task relationship learning methods, and decomposition
methods. Low-rank methods interpret the correlation between multiple tasks as
a low-rank representation of their parameter matrices [4, 82]. Task clustering
methods group tasks into clusters based on similarities, where each task
cluster contains similar tasks [72, 78]. Task relationship learning methods
aim to automatically learn quantitative relationships between tasks from the
data. Decomposition methods break down the model parameters for all tasks into
multiple sub-models, and each sub-model is subject to different
regularizations [11, 14]. These approaches are designed to leverage the
interplay between task parameters to improve the overall performance of MTL.

(3) Weighting-based approaches formulate the MTL problem into finding an
optimal gradient via weighting sum for updating, and can be categorized into
three types. The learning-based methods [19, 45] design extra models to learn
the appropriate weights for each task. The solving-based methods [66]
formulate the MTL into a multi-objective optimization problem and obtain the
Pareto descent direction. The calculating-based methods [46, 20, 79, 33, 42]
project conflict gradients to other spaces to reduce the conflicts.

In this paper, we only consider the multi-source MTL, i.e., each task has its
specific data stream without sharing with other tasks. However, multi-source
MTL can only learn from data within a fixed number of domains for a fixed
number of tasks, i.e., without the ability to evolve. Besides, MTL assumes the
data are always totally available, which is not realistic for long-term
learning.

###  II-B Continual Learning

Continual Learning (CL), also known as incremental learning, receives data
from new domains in order to satisfy new task demands. CL imitates the
learning way of the human brain, which is manifested as adaptation to the
environment. In the real world, people encounter new things and acquire new
knowledge every day, which requires the human brain to have the ability of
continual learning, that is, to keep learning new knowledge while ensuring
that the old knowledge is not forgotten. Similarly, the CL based on neural
networks often uses the gradient descent method to update the model. When
learning new knowledge, the gradient containing only new knowledge will
overwrite the old knowledge within the model parameter, resulting in
catastrophic forgetting [32]. According to the incremental carrier of new
knowledge, CL can be task-incremental, class-incremental or other forms. In
this paper, we focus on task-incremental and class-incremental CL in our
experiments. The traditional CL has the training tasks in sequence, that is,
nose-to-tail. We name this kind of CL Serial CL (SCL). The main solutions so
far to CL can be roughly categorized into three types [22].

(1) Rehearsal-based methods [50, 17, 34, 6, 68, 58] save previous knowledge,
which will be used to remind model together with the current training.
According to different knowledge storage forms, rehearsal-based methods can be
divided into [21] three categories. First, rehearsal on raw data, which
retrains directly on the stored raw data, and restricts the update of the
model parameters [63, 50, 17, 18]. Second, rehearsal on features constrains
the parameter update after the feature extracted layer but does not affect the
previous neural network layers [61, 16, 12]. Third, rehearsal on synthesizing
data from generative models, which stores knowledge in the form of model
instead of data [75, 49].

(2) Regularization-based methods [37, 15, 24, 80, 1] leverage extra
regularization terms in the loss function to consolidate previous knowledge
when learning from new data. Kirkpatrick et al. [37] propose elastic weight
consolidation, which uses Laplace approximation to estimate the posterior of
previous knowledge and uses the Fisher information matrix to constrain the
update. Aljundi et al. [1] propose another approximation for parameter
importance by adding perturbation on model parameters. Zenke et al. [80] adopt
a similar idea, considering the influence of the loss change can be
represented by gradient influence.

(3) Architecture-based methods [54, 77, 64] allocate task-specific parameters
or grow new branches for new tasks to bring in new knowledge. According to the
operation on structure, these methods can be divided into expansion and
routing. The expansion-based methods directly expand the model structure, and
the new task is trained on the expanded parameters, so as to avoid the damage
of the parameters related to the old task [65, 26]. The routing-based methods
seek to make different tasks use different parameter paths, which avoid the
overlap of important parameters by new tasks [29, 67].

However, existing CL methods assume that the new task demand appears only
after the previous training is done, which is not realistic enough in real-
world applications. In this paper, we consider a more general scenario that
the data w.r.t.  a new task can access at any time. That is, different tasks
can be trained with data from different domains at the same time, what we call
the Parallel Continual Learning (PCL).

##  III Parallel Continual Learning

###  III-A Problem Definition: Parallel Continual Learning

Suppose we have a sequence of $T$ tasks corresponding to $T$ parallel data
streams $\\{\mathcal{D}_{1},\cdots,\mathcal{D}_{T}\\}$, in which each data
stream can be accessed and suspended at any time. Data streams are composed by
non-overlapping sequences of data tuples $(\mathbf{x},\mathbf{y})$, where
$\mathbf{x}$ is an input data point, $\mathbf{y}$ is a one-hot label and
$C_{t}$ is the class number of task $t$. For simplicity, we assume that each
data stream is stationary, which means that the data are drawn from a fixed,
unknown probability distribution.

We define the start and end time of the $t$-th data stream as $(s_{t},e_{t})$.
To simplify matters, we assume that $s_{t-1}\leq s_{t}\leq
1+\max(e_{1},e_{2},\cdots,e_{t-1})$ for any task $t>1$. This means that tasks
are accessed in order from $1$ to $T$, and no blank in the timeline where no
data stream is active. It is worth noting that traditional SCL represents an
extreme case of PCL, where all tasks are executed in sequence, i.e., $\forall
t>1$, $s_{t}=e_{t-1}+1$.

To adapt to incremental tasks, a PCL model is composed of a task-agnostic
backbone with parameter $\bm{\theta}$ and dynamically incremental number of
task-specific classifiers with parameters $\bm{\theta}_{t}$. When a new task
is accessed, a new corresponding task-specific classifier is constructed. The
function of both networks is given by
$f_{\bm{\theta}}:\mathbf{x}\mapsto\mathbf{v}\in\mathbb{R}^{1\times D}$ and
$f_{\bm{\theta}_{t}}:\mathbf{v}\in\mathbb{R}^{1\times
D}\mapsto\mathbf{p}\in\mathbb{R}^{1\times C_{t}}$, where
$\mathbf{x},\mathbf{v},\mathbf{p}$ denote the input data point, the task-
agnostic feature and the predicted probability, respectively.

Based on the above concepts, we define Parallel Continual Learning (PCL) as
follows.

######  Definition 1 (Parallel Continual Learning).

Parallel Continual Learning (PCL) refers to the adaptive training of a model
that continuously receives multiple data streams based on their access period.
At any time, PCL trains an unfixed number of tasks using available data from
these streams.

###  III-B Dynamic multi-task learning with rehearsal

To prevent a task from being forgotten when its data stream is finished, we
employ the Rehearsal strategy [50, 17, 34, 6, 68, 58], which involves storing
a small amount of training data and creating an additional data stream for
retraining completed tasks. Specifically, a portion of the data from each
completed task is stored in a fixed set $\mathcal{D}_{0}$. At time $t$, PCL
training, together with the memory stream $\mathcal{D}_{0}$, yields the
following formulation for multi-objective empirical risk minimization:

$\mathop{\min}_{\bm{\theta},\\{\bm{\theta}_{i}|{i\in\mathcal{T}_{t}}\\}}\quad\left\\{\ell_{i}\left(\mathcal{D}_{i}\right)|i\in\mathcal{T}_{t}\right\\}$
(1)

where $\mathcal{T}_{t}$ is the task index set with activated data streams at
time $t$. According to the rehearsal strategy, if at least one finished task
exists, we include $0$ in $\mathcal{T}_{t}$ and use $\ell_{0}$ to represent
the loss for all finished tasks for experience replay.

A naive solution to this dynamic MOO problem is to apply the Average Gradient
(AvgGrad), in which the model is updated with the following rules

$\displaystyle\bm{\theta}\leftarrow\bm{\theta}-\gamma\cdot\frac{1}{|\mathcal{T}_{t}|}\sum\nolimits_{i\in\mathcal{T}_{t}}\nabla_{\bm{\theta}}\ell_{i},$
(2)
$\displaystyle\bm{\theta}_{i}\leftarrow\bm{\theta}_{i}-\gamma_{i}\cdot\nabla_{\bm{\theta_{i}}}\ell_{i},~{}~{}\forall
i\in\mathcal{T}_{t}$ (3)

where $\gamma$ and $\gamma_{i}$ are the step sizes for optimization. Since the
task-specific classifiers are updated by their own gradients, we focus on the
update of the shared backbone $\bm{\theta}$, and let the negative gradient
$\mathbf{g}_{i}=-\nabla_{\bm{\theta}}\ell_{i}$. However, treating each task
equally in the AvgGrad method is naive and does not lead to Pareto optimality
[13].

######  Definition 2 (Pareto Optimality).

(1) (_Pareto Dominate_) Let $\bm{\theta}_{a}$, $\bm{\theta}_{b}$ be two
solutions for Problem (1), $\bm{\theta}_{a}$ is said to dominate
$\bm{\theta}_{b}$ ($\bm{\theta}_{a}\prec\bm{\theta}_{b}$) if and only if
$\ell_{i}(\bm{\theta}_{a})\leq\ell_{i}(\bm{\theta}_{b})$, $\forall
i\in\mathcal{T}$ and $\ell_{i}(\bm{\theta}_{a})<\ell_{i}(\bm{\theta}_{b})$,
$\exists i\in\mathcal{T}$.
(2) (_Pareto Critical_) $\bm{\theta}$ is called Pareto critical if no other
solution in its neighborhood can have better values in all objective
functions.
(3) (_Pareto Descent Direction_) If $\bm{\theta}_{1}$ is not Pareto critical
and can be updated to $\bm{\theta}_{2}$ by gradient $\mathbf{g}$. If
$\bm{\theta}_{2}\prec\bm{\theta}_{1}$, say $\mathbf{g}$ is a Pareto descent
direction.

An elegant solution for achieving Pareto optimality in MOO is the Steepest
Descent Method (SDM) [30]. SDM obtains an optimal descent direction
$\mathbf{d}^{*}$ that satisfies

$\displaystyle\mathbf{d}^{*},\alpha^{*}=\arg\min_{\mathbf{d},\alpha}$
$\displaystyle\quad\alpha+\frac{1}{2}\left\|\mathbf{d}\right\|^{2},$ (4) s.t.
$\displaystyle\quad\mathbf{g}_{i}^{\top}\mathbf{d}\leq\alpha,\quad\forall
i\in\mathcal{T}$

where the constraints ensure that each task does not conflict with the
gradient $\mathbf{d}$. Considering the Karush–Kuhn–Tucker (KKT) condition, the
dual problem solved by the Multi-Gradient Descent Algorithm (MGDA) [66] is
presented as

$\displaystyle\bm{\lambda}^{*}=\arg\min_{\bm{\lambda}}*\quad\Big{\|}\sum\nolimits_{i}\lambda_{i}\mathbf{g}_{i}\Big{\|}^{2},$
(5)
$\displaystyle\text{s.t.}*\quad\sum\nolimits_{i}\lambda_{i}=1~{}~{}\text{and}~{}~{}\lambda_{i}\geq
0,\forall i.$

The objective of MGDA is $0$ and the resulting point satisfies the KKT
conditions, or the solution gives a Pareto descent direction that improves all
tasks [23].

Figure 2: Two synthetic functions are optimized in the PCL setting, where the
new task access at iter 500.

However, both AvgGrad and MGDA ignore the training variations between any two
parallel tasks that may differ a lot. In Fig.2, we conduct a toy PCL
experiment with two synthetic functions 111The two functions are
$f_{1}(x,y)=\log(1+x^{2})+0.8\left(1-e^{x}\cdot\sin(y)\right)^{2}$ and
$f_{2}(x,y)=\log(1+y^{2})+0.004\left(0.1+e^{y}\cdot\cos(x)\right)^{2}$. The
values of $(x,y)$ are initialized to $(3,3)$. The optimizer is the steepest
gradient descent, and the step size is set to $2e$-$5$. The total number of
iterations is 1,500. In PCL, we let $f_{1}(x,y)$ be the old task, and the new
task $f_{2}(x,y)$ is set to access at the 500-th iteration.. As we observed,
AvgGrad is biased towards the new task with larger gradients, which hinders
the optimization of old tasks. In contrast, MGDA is inclined to update with a
conservative Pareto descent direction, which may compromise the new task
learning. To strike a balance among old and new tasks, we propose a novel yet
simple Elastic Multi-Gradient Descent (EMGD) method. EMGD considers the
difference in the learning process among tasks and obtains a Pareto descent
direction at each iteration that ensures both new task training and old task
preservation.

##  IV Elastic Multi-Gradient Descent

###  IV-A Elastic constraint for steepest gradient method

Instead of using direct weighting or solving methods, we seek controllable
constraints in PCL. Here, we define an elastic solution for Eq. (4) with an
elastic factor $\sigma>0$ for $\bm{\theta}$, such that if $\bm{\theta}$ is not
Pareto critical, the objective satisfies:

$\alpha(\bm{\theta})+\frac{1}{2}||\mathbf{d}(\bm{\theta})||^{2}\leq\sigma\alpha^{*}(\bm{\theta})$
(6)

where $\alpha^{*}(\bm{\theta})$ is the optimum of Problem (4). For $\sigma=1$
only the exact solution satisfies the above inequality. To further consider
the angle between each $\mathbf{g}_{i}$ and $\mathbf{d}$, we generalize the
elasticity to each task, and rewrite the Problem (4) as follows:

$\displaystyle\mathbf{d}^{*},\alpha^{*}=\arg\min_{\mathbf{d},\alpha}$
$\displaystyle\quad\alpha+\frac{1}{2}||\mathbf{d}||^{2},$ (7) s.t.
$\displaystyle\quad\mathbf{g}_{i}^{\top}\mathbf{d}\leq\sigma_{i}\alpha,\quad\forall
i\in\mathcal{T}.$

The only difference of Eq. (7) from SDM is the task-specific elastic factors
$\sigma_{i}\in(0,1]$.

######  Lemma 1.

Let $\mathbf{d}^{*}(\bm{\theta})$ and $\alpha^{*}(\bm{\theta})$ be the
solution of Eq. (7) under the paramter $\bm{\theta}$.
$\mathbf{d}^{*}(\bm{\theta})$ and $\alpha^{*}(\bm{\theta})$ hold the features:
(1) if $\bm{\theta}$ is Pareto critical, it has
$\mathbf{d}^{*}(\bm{\theta})=\mathbf{0}$ and $\alpha^{*}(\bm{\theta})=0$;
(2) if $\bm{\theta}$ is not Pareto critical, it has
$\alpha^{*}(\bm{\theta})\leq-\frac{1}{2}||\mathbf{d}^{*}(\bm{\theta})||^{2}<0$
and
$\mathbf{g}_{i}^{\top}\mathbf{d}^{*}(\bm{\theta})\leq\sigma_{i}\alpha^{*}(\bm{\theta}),~{}~{}\forall
i$.

Proof. We prove the two features in Lemma 1 with the definition of Pareto
critical. Reviewing the definition of Pareto optimality in Definition 2

_Feature_ 1): If the solution $\bm{\theta}$ of PCL is Pareto critical, the
solution to Problem (7) (EMGD) is $\mathbf{d}^{*}(\bm{\theta})=\mathbf{0}$,
then there exists no direction make all objectives get better. Due to the
constraints, we have
$\sigma_{i}\alpha^{*}(\bm{\theta})\geq\mathbf{g}_{i}^{\top}\mathbf{d}^{*}(\bm{\theta}),~{}\forall
i$, thus $\alpha^{*}(\bm{\theta})=0$.

_Feature_ 2): If the solution $\bm{\theta}$ of PCL is not Pareto critical, it
means that $\mathbf{d}(\bm{\theta})=\mathbf{0}$ and $\alpha(\bm{\theta})=0$
are the feasible solutions to Problem (7). That is, we have

$\alpha^{*}(\bm{\theta})+\frac{1}{2}\|\mathbf{d}^{*}(\bm{\theta})\|^{2}<\alpha(\bm{\theta})+\frac{1}{2}\|\mathbf{d}(\bm{\theta})\|^{2}=0.$

Thus,

$\alpha^{*}(\bm{\theta})\leq\frac{1}{2}\|\mathbf{d}^{*}(\bm{\theta})\|^{2}<0.$

Moreover, referring to the constraints, we have

$\mathbf{g}_{i}^{\top}\mathbf{d}^{*}(\bm{\theta})\leq\sigma_{i}\alpha^{*}(\bm{\theta}),~{}~{}\forall
i.$

All claims of Lemma 1 are clear. $\hfill\blacksquare$

The task-specific elastic factors, which satisfy $0<\sigma<1$, relax the
constraint to varying degrees for each task. By the Lagrange multipliers for
the linear inequality constraints, the dual problem can be rewritten as
follows:

$\displaystyle\bm{\lambda}^{*}=\min_{\bm{\lambda}}$
$\displaystyle\quad\Big{\|}\sum\nolimits_{i}\lambda_{i}\mathbf{g}_{i}\Big{\|}^{2},$
(8) s.t.
$\displaystyle\quad\sum\nolimits_{i}\lambda_{i}\sigma_{i}=1~{}~{}\text{and}~{}~{}\lambda_{i}\geq
0,\forall i.$

The decision space is reduced to the number of in-training tasks, which is
significantly smaller than the original problem. As a quadratic programming
problem, this can be easily solved. The main challenge is to determine the
appropriate elastic factor $\sigma$ for each task. If so, new tasks with large
gradients can be trained quickly, while old tasks or memory tasks with small
gradients can be sustained well.

Figure 3:  The schematic of the proposed method (one-step). Together with the
memory stream, each activated data stream is put into the model and used to
compute the task-specific gradient. These gradients are sent to EMGD to obtain
a Pareto descent direction. Then, we use the optimal gradient from EMGD to
guide the editing on the memory.

###  IV-B Evaluating elastic factor

Reviewing the diverse training process of PCL, we consider the following two
goals to set appropriate elastic factors for Problem (7): 1) ensuring that new
tasks converge as quickly as possible; 2) minimizing the impact on old tasks
as much as possible. To evaluate whether a task satisfies both goals, we
propose using two metrics: gradient magnitude change and gradient
similarities.

(1) _Gradient Magnitude Change_ (GMC): The magnitude of the gradient can serve
as an indicator to evaluate the degree of convergence or the possibility of
being affected by other tasks. Therefore, we first propose to compute elastic
factors by evaluating the gradient norm change for each task as follows:

$\sigma_{i}=\cfrac{\exp({m_{i}})}{\sum_{j}\exp(m_{j})}$ (9)

where $m_{i}$ is the gradient norm with momentum, i.e.,
$m^{t}_{i}=\epsilon_{1}\cdot
m^{t-1}_{i}+\epsilon_{2}\cdot||\mathbf{g}^{t}_{i}||$, where $\epsilon_{1}$ and
$\epsilon_{2}$ are empirically set to 0.9 and 0.1, respectively. The momentum
acceleration in mini-batch stochastic gradient descent helps to reduce the
instability of the gradient norm.

(2) _Gradient Similarity_ (GS): Gradient similarity between multiple gradients
represents the impact of each gradient on other tasks. Hence, we use the
similarity between any two gradients to compute the elastic factors

$\sigma_{i}=\frac{\exp(\sum_{k}\cos(g^{t}_{i},g^{t}_{k}))}{\sum_{j}\exp(\sum_{k}\cos(g^{t}_{j},g^{t}_{k}))}$
(10)

Compared to GMC, which only considers the self-change of each gradient, GS
takes the relative impact of each gradient on others into account. We show the
details of the full algorithm in Algorithm 1.

Input: Random-initialized $\bm{\theta}$, $\bm{\theta}_{1:T}$; Step sizes
$\gamma$, $\gamma_{1:T}$

Output: $\bm{\theta}$, $\bm{\theta}_{1:T}$

1

2for  _$t$ in timeline_ do

3 $\mathcal{T}_{t}\leftarrow$ in-training task index;

4 for  _$i\in\mathcal{T}_{t}$_ do

5 $\mathcal{B}_{i}\sim\mathcal{D}_{i}$;

6
$\bm{\theta}_{i}\leftarrow\bm{\theta}_{i}-\gamma_{i}\nabla_{\bm{\theta_{i}}}\ell_{i}\left(\mathcal{B}_{i}\right)$;

7 $\mathbf{g}_{i}=-\nabla_{\bm{\theta}}\ell\left(\mathcal{B}_{i}\right)$;

8 end for

9 $\mathcal{B}_{\mathrm{m}}\sim\mathcal{M}$;

10 $\\{\sigma_{i}|\forall i\in\mathcal{T}_{t}\\}\leftarrow$ Compute elastic
factors using GMC (Eq. (9)) or GS (Eq. (10));

11 $\bm{\lambda}^{*}\leftarrow$ Solving Problem (8) with all $\sigma$;

12
$\bm{\theta}\leftarrow\bm{\theta}+\gamma\cdot\sum_{i}\lambda_{i}\mathbf{g}_{i}$;

13 for  _$\mathbf{x}\in\mathcal{B}_{\mathrm{m}}$_ do

14
$\mathbf{x}\leftarrow\mathbf{x}-\alpha\nabla_{\mathbf{x}}\left\|\mathbf{g}(\mathbf{x})-\sum_{i}\lambda_{i}\mathbf{g}_{i}\right\|^{2}$

15 end for

16

17 end for

Algorithm 1 PCL using EMGD

##  V Gradient-guided Memory Editing

As mentioned above, our proposed method, EMGD, aims to balance the training of
old and new tasks by taking the difference in gradient magnitudes or
similarities into account. Notably, the issue of catastrophic forgetting in
rehearsal is rooted in the quality of the stored memory. In the case of SCL,
existing rehearsal methods are typically based on the assumption that the
stored data points are used solely to represent the current task. However,
this approach may result in a diminishing utility of the stored memory over
time. To overcome this limitation, the Gradient-based Memory EDiting (GMED)
[62] method focuses on refreshing the memory by evaluating the interference
between old and new tasks. It achieves this by evaluating loss difference on
the same memory data point $\mathbf{x}$ before and after training on the new
task.

Given a model with parameter $\bm{\theta}$, GMED needs to first store the
model $\bm{\theta}^{\prime}$ updated with a target gradient $\mathbf{d}$. To
reduce the interference, one can edit the input data $\mathbf{x}$ as follows:

$\mathbf{x}=\mathbf{x}-\alpha\nabla_{\mathbf{x}}\left\|\ell(\mathbf{x},\bm{\theta})-\ell(\mathbf{x},\bm{\theta}^{\prime})\right\|^{2}$
(11)

where $\bm{\theta}^{\prime}=\bm{\theta}-\alpha\mathbf{d}$. In this way, it is
possible that the difference in loss values between the old and new tasks is
negligible, but the update gradients point in completely different directions.
This discrepancy makes it inaccurate to edit the memory based solely on the
loss increase.

Figure 4: Compared to MGDA [66], EMGD can control the ratio of $\lambda_{2}$
and $\lambda_{1}$ in two task scenario. The ratio highly depends on the value
of $\sigma$.

However, in PCL, simply refreshing the memory based on the loss difference is
not sufficient. Since PCL utilizes parallel data streams, it is important to
accurately edit the memory to minimize both forgetting and conflicts between
old and new tasks. To achieve this, we propose to edit the memory based on the
gradient difference, which can be computed using the optimal gradient
$\mathbf{d}$ as a reference:

$\mathbf{x}=\mathbf{x}-\alpha\nabla_{\mathbf{x}}\left\|\mathbf{g}(\mathbf{x})-\mathbf{d}\right\|^{2}$
(12)

where

$\nabla_{\mathbf{x}}\left\|\mathbf{g}(\mathbf{x})-\mathbf{d}\right\|^{2}=2\left(\frac{\partial\ell(\mathbf{x},\bm{\theta})}{\partial\bm{\theta}}-\mathbf{d}\right)^{\top}\frac{\partial^{2}\ell(\mathbf{x},\bm{\theta})}{\partial\bm{\theta}\partial\mathbf{x}}.$
(13)

As shown in Algorithm 1 (L12-14) and Fig. 3, the proposed memory editing based
on gradient guidance is easy to implement with a second-order derivate.

###  V-A Convergence analysis

In this subsection, we prove that our method can converge to a Pareto critical
point.

######  Theorem 1.

The accumulation point of the continuous optimization by Algorithm 1 is Pareto
Critical.

Proof. Let the solution at the $k$-th iteration be $\bm{\theta}_{k}$, and the
corresponding objective of Problem (7) is
$\alpha(\bm{\theta}_{k})+\frac{1}{2}||\mathbf{d}(\bm{\theta}_{k})||^{2}$.
Suppose an accumulation point is $\hat{\bm{\theta}}$ and

$\lim_{k\rightarrow\infty}\ell(\bm{\theta}_{k})=\ell(\hat{\bm{\theta}}).$ (14)

Thus,

$\lim_{k\rightarrow\infty}|\ell(\bm{\theta}_{k+1})-\ell(\bm{\theta}_{k})|=0.$
(15)

For each task, according to the Taylor Expansion, we have

$\lim_{k\rightarrow\infty}\ell_{i}(\bm{\theta}_{k+1})-\ell_{i}(\bm{\theta}_{k})\leq\gamma\cdot\mathbf{g}_{i}^{\top}\mathbf{d}(\bm{\theta}_{k})\leq
0,\quad\forall i$ (16)

where $\gamma$ is the step size for optimization. Since
$\ell_{i}(\bm{\theta}_{k+1})-\ell_{i}(\bm{\theta}_{k})$ serves as the lower
bound of $\gamma\cdot\mathbf{g}_{i}^{\top}\mathbf{d}$, we have

$\lim_{k\rightarrow\infty}\mathbf{g}_{i}^{\top}\mathbf{d}(\bm{\theta}_{k})=0,\quad\forall
i.$ (17)

According to Lemma 1, we have
$\mathbf{g}_{i}^{\top}\mathbf{d}(\bm{\theta}_{k})\leq\sigma_{i}{\alpha}(\bm{\theta}_{k})$
and
$\alpha(\bm{\theta}_{k})\leq-\frac{1}{2}||\mathbf{d}(\bm{\theta}_{k})||^{2}$,
thus

$\lim_{k\rightarrow\infty}{\alpha}(\bm{\theta}_{k})=0,\quad\lim_{k\rightarrow\infty}\mathbf{d}(\bm{\theta}_{k})=\mathbf{0}.$
(18)

Since $\hat{\bm{\theta}}$ is the accumulated point of $\bm{\theta}^{k}$,
$\hat{\bm{\theta}}$ is Pareto critical. $\hfill\blacksquare$

###  V-B Discussion

####  V-B1 The limitation of MGDA

Suppose we have two gradients $\mathbf{g}_{1}$ and $\mathbf{g}_{2}$ w.r.t. two
tasks 1 and 2, and $||\mathbf{g}_{1}||>||\mathbf{g}_{2}||$. We build the
objective of MGDA

$\displaystyle\lambda_{1}^{*},\lambda_{2}^{*}=\min_{\lambda_{1},\lambda_{2}}$
$\displaystyle\big{\|}\
\lambda_{1}\mathbf{g}_{1}+\lambda_{2}\mathbf{g}_{2}\big{\|}^{2},$ (19) s.t.
$\displaystyle\lambda_{1}+\lambda_{2}=1~{}~{}\text{and}~{}~{}\lambda_{i}\geq
0,\forall i\in\\{1,2\\}.$

It is easy to obtain the optimal $\lambda_{1}^{*},\lambda_{2}^{*}$ by solving
the above problem, and we have

$\lambda_{1}^{*}=\max\left(\min\left(\cfrac{\mathbf{g}_{2}^{\top}\mathbf{g}_{2}-\mathbf{g}_{1}^{\top}\mathbf{g}_{2}}{\left\|\mathbf{g}_{1}-\mathbf{g}_{2}\right\|^{2}},1\right),0\right),\quad\lambda_{2}^{*}=1-\lambda_{1}^{*}.$
(20)

It is easy to know that Eq. (20) can be piecewise as

$\lambda_{1}^{*},\lambda_{2}^{*}=\left\\{\begin{aligned}
&1,0,~{}~{}~{}~{}~{}\text{if}~{}~{}\mathbf{g}_{1}^{\top}\mathbf{g}_{2}\geq\mathbf{g}_{1}^{\top}\mathbf{g}_{1}\\\
&0,1,~{}~{}~{}~{}~{}\text{if}~{}~{}\mathbf{g}_{1}^{\top}\mathbf{g}_{2}\geq\mathbf{g}_{2}^{\top}\mathbf{g}_{2}\\\
&\frac{\mathbf{g}_{2}^{\top}\mathbf{g}_{2}-\mathbf{g}_{1}^{\top}\mathbf{g}_{2}}{\left\|\mathbf{g}_{1}-\mathbf{g}_{2}\right\|^{2}},\frac{\mathbf{g}_{1}^{\top}\mathbf{g}_{1}-\mathbf{g}_{1}^{\top}\mathbf{g}_{2}}{\left\|\mathbf{g}_{1}-\mathbf{g}_{2}\right\|^{2}},~{}~{}~{}\text{otherwise}.\end{aligned}\right.$
(21)

Since $||\mathbf{g}_{1}||>||\mathbf{g}_{2}||$, we have
$\mathbf{g}_{1}^{\top}\mathbf{g}_{1}>\mathbf{g}_{2}^{\top}\mathbf{g}_{2}$ and
$\mathbf{g}_{1}^{\top}\mathbf{g}_{1}>\mathbf{g}_{1}^{\top}\mathbf{g}_{2}$.

If
$\mathbf{g}_{1}^{\top}\mathbf{g}_{1}>\mathbf{g}_{1}^{\top}\mathbf{g}_{2}\geq\mathbf{g}_{2}^{\top}\mathbf{g}_{2}$,
we have $\lambda_{1}^{*}=0,\lambda_{2}^{*}=1$ that
$\lambda_{1}^{*}<\lambda_{2}^{*}$.

If
$\mathbf{g}_{1}^{\top}\mathbf{g}_{1}>\mathbf{g}_{2}^{\top}\mathbf{g}_{2}\geq\mathbf{g}_{1}^{\top}\mathbf{g}_{2}$,
we have
$\lambda_{1}^{*}=\cfrac{\mathbf{g}_{2}^{\top}\mathbf{g}_{2}-\mathbf{g}_{1}^{\top}\mathbf{g}_{2}}{\left\|\mathbf{g}_{1}-\mathbf{g}_{2}\right\|^{2}},\lambda_{2}^{*}=\cfrac{\mathbf{g}_{1}^{\top}\mathbf{g}_{1}-\mathbf{g}_{1}^{\top}\mathbf{g}_{2}}{\left\|\mathbf{g}_{1}-\mathbf{g}_{2}\right\|^{2}}$
that $\lambda_{1}^{*}<\lambda_{2}^{*}$.

To sum up, if we have $||\mathbf{g}_{1}||>||\mathbf{g}_{2}||$ in MGDA, then
$\lambda_{1}<\lambda_{2}$ and the update
$\mathbf{d}=-\lambda_{1}\mathbf{g}_{1}-\lambda_{2}\mathbf{g}_{2}$ has
preference towards the small gradient. And this preference depends on the
gradient norm difference. A similar phenomenon can be promoted to multiple
tasks. $\hfill\blacksquare$

TABLE I:  Task-incremental PCL final performance comparisons (avg $\pm$ std).

Method | Type | PS-EMNIST | PS-CIFAR-100 | PS-ImageNet-TINY
---|---|---|---|---
(Rehearsal) |  $A_{\bar{e}}$ (%) |  $F_{\bar{e}}$ (%) |  $A_{\bar{e}}$ (%) |  $F_{\bar{e}}$ (%) |  $A_{\bar{e}}$ (%) |  $F_{\bar{e}}$ (%)
AvgGrad | MTL | $89.344\pm 0.231$ | $-5.287\pm 0.216$ | $47.579\pm 0.089$ | $23.691\pm 0.432$ | $38.392\pm 0.076$ | $0.764\pm 0.139$
MGDA [23] | MTL | $84.887\pm 0.469$ | $-4.301\pm 0.818$ | $48.957\pm 0.451$ | $25.088\pm 0.203$ | $38.058\pm 0.740$ | $1.465\pm 0.490$
GradNorm [19] | MTL | $87.888\pm 0.158$ | $-6.197\pm 0.183$ | $47.210\pm 1.323$ | $23.474\pm 0.688$ | $38.226\pm 0.769$ | $1.647\pm 0.667$
DWA [46] | MTL | $88.405\pm 0.322$ | $-5.452\pm 0.338$ | $44.969\pm 0.378$ | $22.682\pm 0.537$ | $34.290\pm 1.099$ | $-1.053\pm 0.866$
PCGrad [79] | MTL | $89.698\pm 0.164$ | $-4.921\pm 0.121$ | $47.026\pm 0.538$ | $23.244\pm 0.740$ | $39.427\pm 1.275$ | $2.017\pm 0.769$
RLW [42] | MTL | $89.288\pm 0.218$ | $-5.226\pm 0.204$ | ${47.574}\pm 0.349$ | $23.833\pm 0.117$ | $38.531\pm 1.610$ | $1.332\pm 1.148$
AGEM [17] | SCL | $87.022\pm 0.519$ | $-7.646\pm 0.483$ | $27.379\pm 0.585$ | $5.416\pm 0.851$ | $28.530\pm 0.994$ | $-7.070\pm 1.410$
GMED [62] | SCL | $85.471\pm 0.324$ | $-8.875\pm 0.335$ | $49.094\pm 1.792$ | $18.356\pm 1.345$ | $34.495\pm 1.568$ | $-0.640\pm 1.799$
ER [18] | SCL | $89.106\pm 0.315$ | $-5.525\pm 0.207$ | $47.324\pm 0.584$ | $23.330\pm 0.762$ | $35.950\pm 0.763$ | $-0.767\pm 0.591$
MEGA [34] | SCL | $86.262\pm 0.582$ | $-4.753\pm 0.295$ | ${47.745}\pm 0.460$ | $23.718\pm 0.832$ | $38.386\pm 1.234$ | $2.655\pm 0.824$
MDMTR [52] | SCL | $73.115\pm 0.382$ | $-16.359\pm 0.178$ | $38.865\pm 1.691$ | $18.297\pm 1.052$ | $23.422\pm 0.273$ | $-1.544\pm 0.389$
DER++ [12] | SCL | $89.654\pm 0.451$ | $-4.979\pm 0.293$ | $49.276\pm 0.480$ | $22.634\pm 0.165$ | $40.039\pm 0.039$ | $2.601\pm 0.643$
MaxDO [51] | PCL | $90.189\pm 0.314$ | $-4.258\pm 0.311$ | $50.203\pm 0.978$ | $24.510\pm 0.092$ | $40.770\pm 0.354$ | $3.119\pm 0.450$
EMGD(GMC) | PCL | $90.442\pm 0.144$ | $-\textbf{4.142}\pm 0.182$ | $52.798\pm 0.768$ | $\textbf{26.858}\pm 0.426$ | $46.666\pm 0.095$ | $8.934\pm 0.350$
EMGD(GMC)+ME | PCL | $90.402\pm 0.102$ | $-4.217\pm 0.093$ | $\textbf{56.611}\pm 0.102$ | ${23.970}\pm 0.424$ | $48.746\pm 0.413$ | $6.086\pm 0.499$
EMGD(GS) | PCL | $90.348\pm 0.157$ | $-4.265\pm 0.187$ | $52.389\pm 1.109$ | $26.530\pm 0.524$ | $46.213\pm 0.334$ | $7.830\pm 0.715$
EMGD(GS)+ME | PCL | $\textbf{90.524}\pm 0.038$ | $-{4.501}\pm 0.060$ | $56.378\pm 0.244$ | $22.760\pm 0.399$ | $\textbf{49.269}\pm 0.285$ | $\textbf{7.744}\pm 0.481$

TABLE II:  Class-incremental PCL final performance comparisons (avg $\pm$
std).

Method | Type | PS-EMNIST | PS-CIFAR-100 | PS-ImageNet-TINY
---|---|---|---|---
(Rehearsal) |  $A_{\bar{e}}$ (%) |  $F_{\bar{e}}$ (%) |  $A_{\bar{e}}$ (%) |  $F_{\bar{e}}$ (%) |  $A_{\bar{e}}$ (%) |  $F_{\bar{e}}$ (%)
AvgGrad | MTL | $43.823\pm 0.566$ | $-44.484\pm 0.596$ | $8.921\pm 0.276$ | $-4.985\pm 0.702$ | $7.511\pm 0.464$ | $-15.825\pm 0.228$
MGDA [23] | MTL | $52.823\pm 0.201$ | $-7.753\pm 0.661$ | $11.323\pm 0.282$ | $-4.065\pm 0.725$ | $8.318\pm 0.155$ | $-13.118\pm 0.739$
GradNorm [19] | MTL | $43.292\pm 0.590$ | $-44.824\pm 0.688$ | $9.477\pm 0.222$ | $-4.403\pm 1.236$ | $6.395\pm 0.130$ | $-15.869\pm 0.336$
DWA [46] | MTL | $42.734\pm 0.211$ | $-38.944\pm 0.581$ | $3.519\pm 0.116$ | $-7.470\pm 0.244$ | $3.003\pm 0.139$ | $-12.774\pm 0.158$
PCGrad [79] | MTL | $45.035\pm 0.273$ | $-43.309\pm 0.176$ | $10.140\pm 0.253$ | $-5.149\pm 0.553$ | $7.789\pm 0.237$ | $-15.860\pm 0.318$
RLW [42] | MTL | ${44.595}\pm 0.480$ | $-43.424\pm 0.325$ | $9.957\pm 0.135$ | $-5.173\pm 0.613$ | $7.419\pm 0.258$ | $-16.079\pm 0.255$
AGEM [17] | SCL | $26.249\pm 0.511$ | $-62.480\pm 0.527$ | $3.374\pm 0.098$ | $-10.462\pm 0.543$ | $3.379\pm 0.149$ | $-18.514\pm 0.516$
GMED [62] | SCL | $22.694\pm 0.153$ | $-65.805\pm 0.255$ | $4.941\pm 0.407$ | $-12.837\pm 0.710$ | $3.386\pm 0.348$ | $-19.060\pm 0.682$
ER [18] | SCL | $43.474\pm 0.860$ | $-44.597\pm 0.861$ | $9.317\pm 0.339$ | $-5.337\pm 0.626$ | $6.126\pm 0.277$ | $-13.686\pm 0.290$
MEGA [34] | SCL | ${51.797}\pm 0.607$ | $-5.358\pm 0.865$ | $12.245\pm 0.388$ | $0.380\pm 0.503$ | $10.082\pm 0.302$ | $-2.375\pm 0.119$
MDMTR [52] | SCL | $42.199\pm 1.100$ | $-32.140\pm 0.996$ | $5.525\pm 0.260$ | $-3.454\pm 0.145$ | $2.793\pm 0.042$ | $-12.992\pm 0.082$
DER++ [12] | SCL | $50.039\pm 0.520$ | $-43.422\pm 0.439$ | $11.022\pm 0.642$ | $-1.653\pm 0.120$ | $9.190\pm 0.429$ | $-14.049\pm 0.227$
MaxDO [51] | PCL | $53.139\pm 0.156$ | $-11.903\pm 0.476$ | $12.237\pm 0.176$ | $-2.280\pm 0.270$ | $9.532\pm 0.363$ | $-12.511\pm 0.610$
EMGD(GMC) | PCL | $53.441\pm 0.341$ | $-13.250\pm 0.904$ | $13.705\pm 0.633$ | $0.367\pm 1.299$ | $11.512\pm 0.216$ | $-11.748\pm 0.331$
EMGD(GMC)+ME | PCL | $\textbf{56.891}\pm 0.062$ | $\textbf{-18.515}\pm 0.211$ | $\textbf{16.362}\pm 0.294$ | $\textbf{-1.654}\pm 0.808$ | $12.641\pm 0.921$ | $-9.890\pm 0.743$
EMGD(GS) | PCL | $53.425\pm 0.279$ | $-25.981\pm 0.587$ | $13.601\pm 0.374$ | $1.530\pm 0.524$ | $11.588\pm 0.508$ | $-11.324\pm 1.072$
EMGD(GS)+ME | PCL | $56.031\pm 0.254$ | $-17.826\pm 0.296$ | $16.189\pm 1.343$ | $-2.937\pm 1.490$ | $\textbf{12.796}\pm 0.398$ | $\textbf{-9.093}\pm 0.831$

(a) EMGD (b) EMGD+ME

Figure 5: Performance comparisons with different memory size on PS-CIFAR-100.

####  V-B2 Comparing MGDA with EMGD

In this subsection, we illustrate why EMGD can control the Pareto descent
direction against MGDA. Let us consider the situation of two tasks at some
steps, two gradients $\mathbf{g}_{1}$ and $\mathbf{g}_{2}$ are obtained and
$||\mathbf{g}_{1}||>||\mathbf{g}_{2}||$. For MGDA, by solving Eq. (5), we have
$\lambda_{1}<\lambda_{2}$, which means that MGDA prefers to keep the objective
of tasks with a small gradient norm.

In the scenario with only two tasks, the objective of EMGD is represented by

$\displaystyle\lambda_{1}^{*},\lambda_{2}^{*}=$
$\displaystyle\arg\min_{\lambda_{1},\lambda_{2}}\quad\big{\|}\lambda_{1}\mathbf{g}_{1}+\lambda_{2}\mathbf{g}_{2}\big{\|}^{2},$
(22)
$\displaystyle\text{s.t.}\quad\lambda_{1}\sigma_{1}+\lambda_{2}\sigma_{2}=1,\lambda_{1}\geq
0,~{}~{}\lambda_{2}\geq 0.$

If
$\sigma_{1}\mathbf{g}_{2}^{\top}\mathbf{g}_{2}<\sigma_{2}\mathbf{g}_{1}^{\top}\mathbf{g}_{2}$,
we have $\lambda_{1}^{*}=0$ and $\lambda_{2}^{*}=\frac{1}{\sigma_{2}}$. If
$\sigma_{2}\mathbf{g}_{1}^{\top}\mathbf{g}_{1}<\sigma_{1}\mathbf{g}_{1}^{\top}\mathbf{g}_{2}$,
we have $\lambda_{1}^{*}=\frac{1}{\sigma_{1}}$ and $\lambda_{2}^{*}=0$.
Otherwise, we have

$\lambda_{1}^{*}=\frac{\sigma_{1}\mathbf{g}_{2}^{\top}\mathbf{g}_{2}-\sigma_{2}\mathbf{g}_{1}^{\top}\mathbf{g}_{2}}{\big{\|}\sigma_{2}\mathbf{g}_{1}-\sigma_{1}\mathbf{g}_{2}\big{\|}^{2}},\quad\lambda_{2}^{*}=\frac{\sigma_{2}\mathbf{g}_{1}^{\top}\mathbf{g}_{1}-\sigma_{1}\mathbf{g}_{1}^{\top}\mathbf{g}_{2}}{\big{\|}\sigma_{2}\mathbf{g}_{1}-\sigma_{1}\mathbf{g}_{2}\big{\|}^{2}}.$
(23)

In this situation,

$\frac{\lambda_{1}^{*}}{\lambda_{2}^{*}}=\frac{\sigma_{1}\mathbf{g}_{2}^{\top}\mathbf{g}_{2}-\sigma_{2}\mathbf{g}_{1}^{\top}\mathbf{g}_{2}}{\sigma_{2}\mathbf{g}_{1}^{\top}\mathbf{g}_{1}-\sigma_{1}\mathbf{g}_{1}^{\top}\mathbf{g}_{2}}=\frac{||\mathbf{g}_{2}||\cos\langle\sigma_{2}\mathbf{g}_{1}-\sigma_{1}\mathbf{g}_{2},\mathbf{g}_{2}\rangle}{||\mathbf{g}_{1}||\cos\langle\sigma_{1}\mathbf{g}_{2}-\sigma_{2}\mathbf{g}_{1},\mathbf{g}_{1}\rangle}.$
(24)

As shown in Fig. 4, the relative ratio between $\lambda_{1}^{*}$ and
$\lambda_{2}^{*}$ is equivalent to the ratio of gradient projection from two
tasks ($\mathbf{g}_{1}$ and $\mathbf{g}_{2}$) to
$\sigma_{2}\mathbf{g}_{1}-\sigma_{1}\mathbf{g}_{2}$. Note that in MGDA,
$\sigma_{1}=\sigma_{2}=1$, the ratio depends highly on the norm of gradient.
Instead, EMGD adjusts the ratio via elastic factors.

##  VI Experiment

###  VI-A Dataset construction

In our experiments, we conduct transformations on three image recognition
datasets to suit the requirements of PCL. To ensure effective transformation,
we need to fulfill several requirements: (1) Random label sets for each task:
Each data stream had a different label space, with randomly assigned labels
for each task; (2) Random timeline for each label set: The debut of each task
could occur at any time between the first access of the previous and
subsequent tasks. To simplify the process, we omitted any periods where no
data streams were present. The three datasets are as follows:

  * •

Parallel Split EMNIST (PS-EMNIST): We split EMNIST (62 classes) into 5 tasks
and randomly generate 3 label sets for each task and 3 timelines for each
label set (say 9 different situations). The size of the label set for each
task, i.e., the number of classes, is set to larger than 2 while no more than
15.

  * •

Parallel Split CIFAR-100 (PS-CIFAR-100): We split CIFAR-100 into 20 tasks and
randomly generate 3 label sets for each task and 3 timelines for each label
set. The size of the label set for each task is set to larger than 2 while no
more than 15.

  * •

Parallel Split ImageNet-TINY (PS-ImageNet-TINY): We split it into 20 tasks
w.r.t. random 3 label sets, and each label set has 3 randomly generated
timelines. The size of the label set for each task is set to larger than 5
while no more than 20.

Note that for each task, we randomly generated 3 label sets and 3 timelines.
To ensure fairness in our comparisons, we conducted each experiment using 3
fixed seeds ranging from 1,234 to 1,236. This resulted in a total of 27
different experiments for each dataset. In our experiments, each subset of
data was used only once and then discarded. We considered two types of
incremental scenarios in continual learning: task-incremental and class-
incremental. Task-incremental learning involved having task IDs available
during both the training and testing phases. On the other hand, class-
incremental learning did not include task IDs, making it a more challenging
scenario.

###  VI-B Experiment details

The input batch size is set to 128 for all three datasets. We take a 2-layer
MLP as the backbone network for PS-EMNIST, and a Resnet-18 [35] for PS-
CIFAR-100 and PS-ImageNet-Tiny, respectively. In the rehearsal strategy, we
store 30 samples per class for PS-CIFAR-100, 5 for PS-EMNIST and 15 for PS-
ImageNet-Tiny.

To evaluate PCL, we compute the average accuracy and the forgetting following
previous continual learning studies such as [50, 17, 62]. Suppose
$\bar{e}=\max(e_{1},e_{2},\cdots,e_{T})$, the two metrics are computed as
follows:

$\displaystyle
A_{\bar{e}}=\frac{1}{T}\sum\nolimits_{t=1}^{T}a_{\bar{e}}^{t},,\quad
F_{\bar{e}}=\frac{1}{T}\sum\nolimits_{t=1}^{T}a^{t}_{\bar{e}}-a_{e_{t}}^{t}$
(25)

where $a^{j}_{k}$ is the mean testing accuracy of task $j$ at time $k$. The
$A_{\bar{e}}$ denotes the final average accuracy on all the tasks, and the
$F_{\bar{e}}$ (also known as backward transfer) means the final performance
drop compared to each task was first trained.

(a) PS-EMNIST (b) PS-CIFAR-100 (c) PS-ImageNet-TINY

Figure 6: Visualization of editing gradients. Left: Stored images, Right:
Normalized gradient.

(a) PS-EMNIST (b) PS-CIFAR (c) PS-ImageNet-TINY

Figure 7: Learning process comparisons on three datasets.

###  VI-C Main Results

We compare EMGD with MTL methods including MGDA [66], GradNorm [19], DWA [46],
GradDrop [20], PCGrad [79] and RLW [42]. Though the MTL methods are not
designed for PCL, we treat any time as an MTL subunit and the connection of
all these subunits can be used to train PCL. We also compare our method with
AvgGrad as illustrated in Sec. III-B. We also compare EMGD with SCL methods
including AGEM [17], GMED [62], MEGA [34] and MDMTR [52]. To adapt to the SCL
methods, we merge batches from every in-training task to mimic a naive
sequence learning that only a batch from the current task and a batch from the
memory buffer. For our proposed method, we use EMGD(GMC) and EMGD(GS) to
represent EMGD using gradient magnitude change or gradient similarity to
determine elastic factors.

As shown in Tables I and II, we compare our method with MTL and SCL methods on
three datasets in the task- and class-incremental PCL scenarios, respectively.
We have several major observations. First, the final accuracy $A_{\bar{e}}$ of
EMGD is better than that of the compared methods on three datasets, which
shows the superiority of EMGD in solving task conflicts and catastrophic
forgetting. Second, with the proposed gradient-guided Memory Editing (ME), the
performance of EMGD can be further improved, which means that with the
guidance of a good gradient, the memory may reduce its negative influence on
future training conflicts. Third, similar to SCL, task-incremental PCL has
less forgetting than class-incremental PCL. We think that this is because
task-incremental has specific task IDs for training, which may reduce the
knowledge negative transfer for old tasks. However, for class-incremental PCL,
we find that forgetting may be more important than training conflict. The
third finding is AvgGrad is good enough compared to other methods in the PCL
setting. This means the compared MTL and SCL methods focus on only studying
one of the balanced training and less forgetting, which is not appropriate for
the PCL setting. Moreover, MGDA has poor performance in task-incremental PCL
but good in class-incremental PCL, which also shows that task-incremental PCL
focuses more on training conflict and class-incremental PCL should first solve
forgetting. Our EMGD considers the diverse training process in PCL and designs
to obtain a Pareto descent direction, which balances the training between new
task learning and old task keeping. Memory editing helps the training towards
better balance in future training. Finally, we compare two metrics, GMC and
GS, and observe that in task-incremental PCL, GS outperforms GMC. However, in
class-incremental PCL, GMC performs better than GS. This indicates that in
task-incremental PCL, the availability of task IDs allows the model to focus
more on individual tasks. On the other hand, in class-incremental PCL,
considering task correlations becomes more crucial for improving performance.

###  VI-D Reherasal analysis of EMGD

We evaluate PCL with different sizes of rehearsal coreset on PS-EMNIST and PS-
CIFAR-100 datasets. The results are shown in Fig. 5. We have the following
observations. First, with the size growth of the memory buffer, EMGD has a
better $A_{\bar{e}}$, which means the retraining on rehearsal data streams can
clearly reduce the catastrophic forgetting in PCL. As the memory size
increases, the rate of performance improvement in PCL slows down, and in some
cases, it may even lead to a slight decline in performance. Larger memory
sizes have a more pronounced impact on performance enhancement with GS, while
GMC demonstrates better performance at smaller memory sizes.

###  VI-E Visualization of edited gradient

In Fig. 6, we conducted visualization on gradient editing (normalized
gradients in Eq. (12)) using two datasets: PS-EMNIST and PS-CIFAR-100.
Brighter pixels mean more edits. We observe a phenomenon where the ease of
editing gradients was influenced by the information density of the data. On
the PS-EMNIST dataset, we find that the gradients were more easily edited in
regions with lower information density. This implies that the areas of the
background are more susceptible to being edited. On the other hand, in the PS-
CIFAR-100 dataset, which contains complex images from various categories, we
noticed that gradients associated with higher information density pixels are
more prone to be edited. These pixels contained more intricate features,
textures, and patterns, making them more likely to be altered. These
experimental results suggest that on simple datasets, where the task
information is more explicit and the density of information is higher,
rehearsal performs better. However, on more complex datasets, where there is
greater uncertainty in the data, it is necessary to selectively edit the key
elements of the original data to enhance its compatibility with future tasks.
This targeted editing helps to improve the data’s ability to support
subsequent tasks by reducing uncertainty and enhancing relevant information.

Figure 8: Softmax temperature selection for three datasets. TABLE III:  Multi-
epoch experiments on PS-CIFAR-100.

Method | Task increment | Class increment
---|---|---
$A_{\text{first}}$ (%) |  $F_{\text{final}}$ (%) |  $A_{\text{first}}$ (%) |  $F_{\text{final}}$ (%)
AvgGrad | 59.600 $\pm$ 0.437 | -5.440 $\pm$ 0.271 | 19.390 $\pm$ 0.366 | -17.990 $\pm$ 0.650
MGDA | 58.780 $\pm$ 6.844 | -2.770 $\pm$ 6.250 | 21.057 $\pm$ 1.269 | -14.886 $\pm$ 0.810
PCGrad | 56.720 $\pm$ 1.712 | -6.210 $\pm$ 0.962 | 17.610 $\pm$ 0.105 | -24.000 $\pm$ 1.422
AGEM | 49.949 $\pm$ 1.533 | -14.084 $\pm$ 1.779 | 7.408 $\pm$ 0.411 | -35.003 $\pm$ 0.896
ER | 59.322 $\pm$ 0.349 | -6.774 $\pm$ 0.619 | 16.030 $\pm$ 0.787 | -21.300 $\pm$ 0.058
MEGA | 60.710 $\pm$ 0.386 | -2.779 $\pm$ 0.518 | 22.710 $\pm$ 0.317 | -11.470 $\pm$ 0.731
MaxDO | 60.400 $\pm$ 1.862 | -0.997 $\pm$ 2.326 | 22.940 $\pm$ 0.727 | -19.020 $\pm$ 2.976
EMGD(GMC) | 63.754 $\pm$ 1.080 | -3.488 $\pm$ 0.862 | 26.984 $\pm$ 0.328 | -11.944 $\pm$ 1.238
EMGD(GMC)+ME | 65.222 $\pm$ 0.420 | -2.586 $\pm$ 0.403 |  30.657 $\pm$ 0.263 |  -3.483 $\pm$ 0.707
EMGD(GS) | 64.462 $\pm$ 0.524 | -2.255 $\pm$ 0.734 | 25.885 $\pm$ 0.069 | -8.741 $\pm$ 0.660
EMGD(GS)+ME |  67.260 $\pm$ 0.926 |  -1.470 $\pm$ 0.143 | 29.445 $\pm$ 0.281 | -2.607 $\pm$ 0.973

(a) TASK 1 - early (EMGD)  (b) Task 1 - final (ER) (c) Task 1 - final
(EMGD+ME)

Figure 9: t-SNE visualization on PS-EMNIST. We omit the “Task 1 - early (ER)”
cause it is quite similar to (a) TASK 1 - early (EMGD) at the early of PCL.
TABLE IV:  Blurring PCL Evaluation on CIFAR-100.

Method | Task increment | Class increment
---|---|---
$A_{\text{first}}$ (%) |  $F_{\text{final}}$ (%) |  $A_{\text{first}}$ (%) |  $F_{\text{final}}$ (%)
AvgGrad | 31.365 $\pm$ 0.767 | 18.070 $\pm$ 0.532 | 8.539 $\pm$ 0.253 | -1.156 $\pm$ 0.169
MGDA | 31.492 $\pm$ 0.095 | 18.088 $\pm$ 0.337 | 11.220 $\pm$ 0.684 | 1.827 $\pm$ 0.749
PCGrad | 31.732 $\pm$ 1.636 | 18.276 $\pm$ 1.436 | 8.874 $\pm$ 0.510 | -1.042 $\pm$ 0.490
AGEM | 18.502 $\pm$ 0.868 | 7.584 $\pm$ 0.244 | 9.477 $\pm$ 0.526 | 4.431 $\pm$ 0.293
ER | 31.044 $\pm$ 0.625 | 17.744 $\pm$ 0.056 | 8.142 $\pm$ 0.263 | -1.511 $\pm$ 0.040
MEGA | 30.809 $\pm$ 0.336 | 18.074 $\pm$ 0.278 | 11.567 $\pm$ 0.414 | 4.051 $\pm$ 0.665
MaxDO | 32.391 $\pm$ 1.519 | 18.382 $\pm$ 1.305 | 10.912 $\pm$ 0.321 | -0.271 $\pm$ 0.747
EMGD(GMC) | 34.860 $\pm$ 0.975 | 20.244 $\pm$ 0.881 | 12.473 $\pm$ 0.115 | 1.785 $\pm$ 0.272
EMGD(GMC)+ME |  45.139 $\pm$ 1.032 |  24.593 $\pm$ 1.428 | 18.227 $\pm$ 0.466 | 1.887 $\pm$ 0.264
EMGD(GS) | 34.811 $\pm$ 1.685 | 20.212 $\pm$ 1.546 | 12.861 $\pm$ 0.904 | 2.089 $\pm$ 0.930
EMGD(GS)+ME | 44.719 $\pm$ 0.840 | 24.321 $\pm$ 0.509 |  18.912 $\pm$ 0.793 |  2.742 $\pm$ 0.581

###  VI-F PCL learning process

As shown in Fig. 7, we evaluate the performance on seen tasks along the whole
PCL training process. The point $i$ in Fig. 7 means that the task $i$ just
finished its training, and the corresponding value denotes the average testing
accuracy on all seen tasks (i.e., tasks in training and finished). Our
experiment effectively enhanced overall performance in three datasets and two
PCL settings (task-incremental and class-incremental PCL). Our method
consistently outperforms baseline models across datasets, validating its
robustness and generalizability. It demonstrated remarkable consistency in
performance, ensuring effectiveness throughout the entire PCL training
process. These findings show that our EMGD method contributes to PCL problem,
providing a reliable solution for optimizing performance. Moreover, we compare
two kinds of computing of elastic factors. We find that GMC has better
performance on the early training tasks while GS has better performance on the
last training period. We think this is because, in the early training period,
the number of seen tasks is small, and the training of each task considers
more about itself. While in the last training period, more tasks need to be
considered, the gradient similarity is important to adjust the conflict.

###  VI-G Hyperparameter selection

In Fig. 8, we show the major hyperparameter selection for Softmax temperature
in GMC (Eq. (9)) and GS (Eq. (10)). The results show that the trends of
different strategy are similar, and the two strategies have similar
sensitivity to temperature. Therefore, it is feasible to select the same
hyperparameter for the two strategies by tuning once.

###  VI-H Visualization of feature space

As depicted in Fig. 9, we employed the t-SNE [73] tool to visualize the
feature space and demonstrate the effectiveness of our proposed memory editing
technique. The visualizations clearly indicate that the proposed memory
editing method successfully maintains representation and uniformity. In
contrast, due to semantic drift, the traditional ER method struggles to
maintain uniformity during the PCL process. Furthermore, we computed the
average cosine similarity between rehearsal samples and class means. After
task 1 is fully trained, there is a considerable similarity between randomly
selected samples and class means. However, as all tasks complete their
training, the similarity in ER decreases significantly due to semantic drift,
where some classes have their rehearsal samples out of distributions. In
contrast, our memory editing method effectively preserves the similarity,
demonstrating that our rehearsal approach mitigates forgetting to a greater
extent.

###  VI-I Multi-epoch comparisons

In major experiments, we evaluate in the 1-epoch online setting for each task.
In Table III, we evaluate in 20-epoch for each task without changing the
training order. The results show that the 20-epoch setting provides better
final accuracy than the single-epoch setting because each task can be trained
more sufficiently toward convergence. Moreover, we find that forgetting
becomes harder than before because more new task training is implemented after
the old tasks are finished. Compared to the other methods, our 20-epoch
performances are also better than them and even further expand the advantages.
The results show that our methods are also effective in off-line or multi-
epoch scenarios.

###  VI-J EMGD in Blurring PCL

Blurring CL [8] is different from traditional CL setting, where the label
spaces are disjoint for any two tasks. Blurring CL has overlap label spaces to
meet the real-world needs. We also implement blurring PCL on CIFAR-100.
Specifically, we split the 100 classes of CIFAR-100 into 9 overlap tasks, each
of which has 20 classes. The label space has 50% overlap between any two
neighbouring tasks. The results are shown in Table IV, which shows that our
EMGD is also effective on blurring PCL. For the two metrics of evaluation
elastic factors, we find that GMC is more effective on task-incremental
blurring PCL setting while GS is more effective on class-incremental blurring
PCL. This is because that no taks-id in class-incremental setting, the
gradient similarity is useful to adjust the influence from one task to another
task. The results also show that the proposed memory editing is useful for
rehearsal-based blurring PCL.

TABLE V:  Comparisons on SCL dataset Split-CIFAR-10, Split-CIFAR-100 and
Split-Tiny-ImageNet.

Method | Split-CIFAR-10 | Split-CIFAR-100 | Split-Tiny-Imagenet
---|---|---|---
$A_{\bar{e}}$ (%) |  $F_{\bar{e}}$ (%) |  $A_{\bar{e}}$ (%) |  $F_{\bar{e}}$ (%) |  $A_{\bar{e}}$ (%) |  $F_{\bar{e}}$ (%)
iCarL [61] | $74.917\pm 1.579$ | $0.013\pm 0.938$ | $31.960\pm 0.715$ | $0.740\pm 0.639$ | - | -
GEM [50] | $73.027\pm 3.571$ | $0.167\pm 4.344$ | $39.380\pm 1.914$ | $4.407\pm 1.434$ | - | -
AGEM [17] | $71.700\pm 2.644$ | $-3.533\pm 2.796$ | $32.607\pm 2.790$ | $-2.960\pm 2.779$ | $20.733\pm 0.682$ | $-8.853\pm 0.439$
GDUMB [59] | $50.937\pm 1.254$ | - | $17.720\pm 0.732$ | - | $6.878\pm 0.955$ | -
GSS [3] | $68.663\pm 2.761$ | $-5.603\pm 3.141$ | $32.757\pm 0.526$ | $-8.753\pm 1.673$ | $27.000\pm 1.578$ | $-9.419\pm 0.943$
MIR [2] | $74.900\pm 3.034$ | $-1.163\pm 1.480$ | $43.140\pm 3.840$ | $\mathbf{6.833}\pm 3.456$ | $27.961\pm 0.310$ | $-1.983\pm 0.192$
GMED [62] | $75.983\pm 1.121$ | ${0.853}\pm 1.351$ | $44.697\pm 4.934$ | $2.930\pm 4.574$ | $27.911\pm 1.015$ | $-0.936\pm 2.124$
MaxDO [51] | $74.324\pm 2.402$ | $-2.835\pm 2.588$ | $44.809\pm 1.910$ | $4.332\pm 1.004$ | $28.380\pm 0.839$ | $-0.610\pm 0.536$
EMGD(GMC) | ${76.983}\pm 0.268$ | $-0.970\pm 1.967$ | ${47.037}\pm 0.397$ | $3.920\pm 1.204$ | ${31.067}\pm 0.707$ | ${0.508}\pm 0.661$
EMGD(GMC)+ME | ${77.356}\pm 0.789$ | $0.234\pm 1.198$ | $\mathbf{49.500}\pm 0.876$ | $5.123\pm 0.456$ | $\mathbf{33.587}\pm 0.319$ | $\mathbf{0.745}\pm 0.862$
EMGD(GS) | ${76.438}\pm 0.597$ | $-0.675\pm 1.256$ | ${47.932}\pm 0.104$ | $3.879\pm 0.619$ | ${31.403}\pm 0.758$ | ${0.531}\pm 0.287$
EMGD(GS)+ME | $\mathbf{77.695}\pm 0.374$ | $\mathbf{0.922}\pm 1.569$ | ${49.218}\pm 0.746$ | $4.385\pm 0.654$ | ${33.421}\pm 0.879$ | ${0.593}\pm 0.127$

TABLE VI:  MTL Comparisons on Split-CIFAR-10, Split-CIFAR-100, and Split-Tiny-
ImageNet.

Method | Split-CIFAR10 | Split-CIFAR100 | Split-Tiny
---|---|---|---
AvgGrad | 76.127 $\pm$ 0.502 | 23.303 $\pm$ 1.626 | 37.840 $\pm$ 1.789
MGDA [66] | 73.213 $\pm$ 1.071 | 23.320 $\pm$ 2.015 | 35.923 $\pm$ 1.007
GradNorm [19] | 76.127 $\pm$ 0.799 | 21.897 $\pm$ 4.882 | 36.890 $\pm$ 0.178
DWA [46] | 76.217 $\pm$ 0.515 | 23.417 $\pm$ 1.568 | 36.420 $\pm$ 2.841
GradDrop [20] | 75.810 $\pm$ 0.470 | 23.403 $\pm$ 1.460 | 38.110 $\pm$ 0.760
PCGrad [79] | 76.237 $\pm$ 0.592 | 22.910 $\pm$ 1.762 | 37.743 $\pm$ 1.497
RLW [42] | 76.100 $\pm$ 0.155 | 23.243 $\pm$ 1.589 | 39.253 $\pm$ 0.993
MaxDO [51] | 76.299 $\pm$ 0.443 | 23.546 $\pm$ 1.671 | 39.297 $\pm$ 0.764
EMGD(GMC) | 76.373 $\pm$ 1.472 | 24.147 $\pm$ 2.110 | 39.467 $\pm$ 0.722
EMGD(GS) |  76.986 $\pm$ 1.304 |  24.780 $\pm$ 2.456 |  39.842 $\pm$ 0.635

###  VI-K EMGD in SCL

Although EMGD is designed for PCL where multiple tasks are trained together.
In this subsection, we also show that EMGD can be used in rehearsal-based SCL,
where the rehearsal plays the role of finished task. That is, we will always
have two parallel tasks from the second task. To verify this, in the SCL
setting, we compare EMGD with 7 rehearsal-based methods (iCarL [61], GDUMB
[59], GEM [50], AGEM [17], GSS [3], MIR [2] and GMED [62]). We evaluate on two
popular public SCL datasets, i.e., Split-CIFAR-10 [80] and Split-CIFAR-100
[80]. We store 300 and 3,000 in total for Split-CIFAR-10 and Split-CIFAR-100,
respectively. Each data point in the online fashion will pass only once. We
also use $A_{\bar{e}}$ and $F_{\bar{e}}$ to evaluate the capacity of models.
We show the results in Table V. EMGD outperforms most of the compared methods,
especially in $A_{\bar{e}}$. This means that EMGD can well balance the memory
task and the new task in SCL process. Another observation is that online
fashion may lead the training insufficient, and continual training on memory
may further improve the performance then drop after convergence.

###  VI-L EMGD in MTL

In this subsection, we show that the proposed EMGD can also be applied to
common Multi-Task Learning (MTL). We compare EMGD with SOTA MTL methods
including MGDA [66], GradNorm [19], DWA [46], GradDrop [20], PCGrad [79], IMTL
[45] and RLW [42] in MTL setting. In our experiments, we also use the Split-
CIFAR-10, Split-CIFAR-100 and Split-Tiny-Imagenet. The results are shown in
Table VI. It is clear that even though MTL suffers less from the diverse
training process challenges, gradients have small differences at each training
step. EMGD captures the small gradient differences and adjusts the Pareto
descent direction and achieves better performance than other methods.

##  VII Conclusion and Future Work

In this paper, we compared traditional Multi-Task Learning (MTL) and Serial
Continual Learning (SCL) that train a single model for multi-task on multi-
source data. MTL lacks the ability to adaption to new knowledge while SCL can
not respond to new tasks immediately. In this paper, we proposed a novel and
practical continual learning problem, called Parallel Continual Learning
(PCL), in which the new task can be trained with previous tasks parallelly
without pending. To optimize the model among dynamic multiple tasks, we treat
PCL as a dynamic multi-objective optimization problem. Based on the well-
proven steepest descent method, we propose to set appropriate elastic factors
to constrain the training of each task. We named our method Elastic Multi-
Gradient Descent (EMGD). To reduce the task conflict inside memory, we also
proposed a gradient-guided memory editing method. The experimental results on
three image recognition clearly show the effectiveness of the proposed
benchmark.

At any time, the multiple tasks in PCL are trained in parallel. Although our
proposed method achieved clear performance improvement compared to other
methods, we assume the training is only one one client or machine. In our
view, PCL has latent relationship with parallel computing and distributed
learning. It is foreseeable to further accelerate PCL training via combining
these techniques to meet the requirements from real-world applications.

## References

  * [1] R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars.  Memory aware synapses: Learning what (not) to forget.  In ECCV, 2018.
  * [2] R. Aljundi, E. Belilovsky, T. Tuytelaars, L. Charlin, M. Caccia, M. Lin, and L. Page-Caccia.  Online continual learning with maximal interfered retrieval.  In NeurIPS, 2019.
  * [3] R. Aljundi, M. Lin, B. Goujaud, and Y. Bengio.  Gradient based sample selection for online continual learning.  In NeurIPS, 2019.
  * [4] R. K. Ando, T. Zhang, and P. Bartlett.  A framework for learning predictive structures from multiple tasks and unlabeled data.  JMLR, 6(11):1817–1853, 2005.
  * [5] A. Argyriou, T. Evgeniou, and M. Pontil.  Convex multi-task feature learning.  Machine learning, 73(3):243–272, 2008.
  * [6] C. Atkinson, B. McCane, L. Szymanski, and A. Robins.  Pseudo-rehearsal: Achieving deep reinforcement learning without catastrophic forgetting.  arXiv preprint arXiv:1812.02464, 2018.
  * [7] B. Babcock, S. Babu, M. Datar, R. Motwani, and J. Widom.  Models and issues in data stream systems.  In SIGMOD-SIGACT-SIGART, 2002.
  * [8] J. Bang, H. Kim, Y. Yoo, J.-W. Ha, and J. Choi.  Rainbow memory: Continual learning with a memory of diverse samples.  In CVPR, 2021.
  * [9] P. Barham, A. Chowdhery, J. Dean, S. Ghemawat, S. Hand, D. Hurt, M. Isard, H. Lim, R. Pang, S. Roy, et al.  Pathways: Asynchronous distributed dataflow for ml.  In MLSys, 2022.
  * [10] A. Barzilai and K. Crammer.  Convex multi-task learning by clustering.  In AISTATS, 2015.
  * [11] E. V. Bonilla, K. Chai, and C. Williams.  Multi-task gaussian process prediction.  In NeurIPS, 2007.
  * [12] P. Buzzega, M. Boschini, A. Porrello, D. Abati, and S. Calderara.  Dark experience for general continual learning: a strong, simple baseline.  In NeurIPS, 2020.
  * [13] Y. Censor.  Pareto optimality in multiobjective problems.  Applied Mathematics and Optimization, 4(1):41–59, 1977.
  * [14] K. Chai.  Generalization errors and learning curves for regression with multi-task gaussian processes.  In NeurIPS, 2009.
  * [15] A. Chaudhry, P. K. Dokania, T. Ajanthan, and P. H. Torr.  Riemannian walk for incremental learning: Understanding forgetting and intransigence.  In ECCV, 2018.
  * [16] A. Chaudhry, A. Gordo, P. Dokania, P. Torr, and D. Lopez-Paz.  Using hindsight to anchor past knowledge in continual learning.  In AAAI, 2021.
  * [17] A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny.  Efficient lifelong learning with a-gem.  In ICLR, 2019.
  * [18] A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. Torr, and M. Ranzato.  On tiny episodic memories in continual learning.  arXiv preprint arXiv:1902.10486, 2019.
  * [19] Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich.  Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks.  In ICML, 2018.
  * [20] Z. Chen, J. Ngiam, Y. Huang, T. Luong, H. Kretzschmar, Y. Chai, and D. Anguelov.  Just pick a sign: Optimizing deep multitask models with gradient sign dropout.  In NeurIPS, 2020.
  * [21] M. De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars.  Continual learning: A comparative study on how to defy forgetting in classification tasks.  arXiv preprint arXiv:1909.08383, 2019.
  * [22] M. De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  IEEE TPAMI, 44(7):3366–3385, 2021.
  * [23] J.-A. Désidéri.  Multiple-gradient descent algorithm (mgda) for multiobjective optimization.  Comptes rendus mathematique, 350(5):313–318, 2012.
  * [24] P. Dhar, R. V. Singh, K.-C. Peng, Z. Wu, and R. Chellappa.  Learning without memorizing.  In CVPR, 2019.
  * [25] C. Doersch and A. Zisserman.  Multi-task self-supervised visual learning.  In ICCV, 2017.
  * [26] A. Douillard, A. Ramé, G. Couairon, and M. Cord.  Dytox: Transformers for continual learning with dynamic token expansion.  In CVPR, 2022.
  * [27] K. Du, L. Li, F. Lyu, F. Hu, Z. Xia, and F. Xu.  Class-incremental lifelong learning in multi-label classification.  arXiv preprint arXiv:2207.07840, 2022.
  * [28] K. Du, F. Lyu, F. Hu, L. Li, W. Feng, F. Xu, and Q. Fu.  Agcn: augmented graph convolutional network for lifelong multi-label image recognition.  In ICME, 2022.
  * [29] C. Fernando, D. Banarse, C. Blundell, Y. Zwols, D. Ha, A. A. Rusu, A. Pritzel, and D. Wierstra.  Pathnet: Evolution channels gradient descent in super neural networks.  arXiv preprint arXiv:1701.08734, 2017.
  * [30] J. Fliege and B. F. Svaiter.  Steepest descent methods for multicriteria optimization.  Mathematical methods of operations research, 51:479–494, 2000.
  * [31] D. Fourure, R. Emonet, E. Fromont, D. Muselet, N. Neverova, A. Trémeau, and C. Wolf.  Multi-task, multi-domain learning: application to semantic segmentation and pose regression.  Neurocomputing, 251:68–80, 2017.
  * [32] R. M. French.  Catastrophic forgetting in connectionist networks.  Trends in cognitive sciences, 3(4):128–135, 1999.
  * [33] R. Groenendijk, S. Karaoglu, T. Gevers, and T. Mensink.  Multi-loss weighting with coefficient of variations.  In WACV, 2021.
  * [34] Y. Guo, M. Liu, T. Yang, and T. Rosing.  Learning with long-term remembering: Following the lead of mixed stochastic gradient.  arXiv preprint arXiv:1909.11763, 2019.
  * [35] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [36] A. Jalali, S. Sanghavi, C. Ruan, and P. Ravikumar.  A dirty model for multi-task learning.  In NeurIPS, pages 964–972, 2010.
  * [37] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  PNAS, 114(13):3521–3526, 2017.
  * [38] S. Lee, J. Zhu, and E. Xing.  Adaptive multi-task lasso: with application to eqtl detection.  In NeurIPS, 2010.
  * [39] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.  Gshard: Scaling giant models with conditional computation and automatic sharding.  arXiv preprint arXiv:2006.16668, 2020.
  * [40] Z. Li and D. Hoiem.  Learning without forgetting.  IEEE TPAMI, 40(12):2935–2947, 2017.
  * [41] X. Liao and L. Carin.  Radial basis function network for multi-task learning.  In NeurIPS, pages 792–802, 2005.
  * [42] B. Lin, F. Ye, and Y. Zhang.  A closer look at loss weighting in multi-task learning.  arXiv preprint arXiv:2111.10603, 2021.
  * [43] X. Lin, H.-L. Zhen, Z. Li, Q.-F. Zhang, and S. Kwong.  Pareto multi-task learning.  In NeurIPS, 2019.
  * [44] J. Liu, S. Ji, and J. Ye.  Multi-task feature learning via efficient l2, 1-norm minimization.  In UAI, 2012.
  * [45] L. Liu, Y. Li, Z. Kuang, J. Xue, Y. Chen, W. Yang, Q. Liao, and W. Zhang.  Towards impartial multi-task learning.  In ICLR, 2021.
  * [46] S. Liu, E. Johns, and A. J. Davison.  End-to-end multi-task learning with attention.  In CVPR, 2019.
  * [47] W. Liu, T. Mei, Y. Zhang, C. Che, and J. Luo.  Multi-task deep visual-semantic embedding for video thumbnail selection.  In CVPR, 2015.
  * [48] X. Liu, P. He, W. Chen, and J. Gao.  Multi-task deep neural networks for natural language understanding.  arXiv preprint arXiv:1901.11504, 2019.
  * [49] X. Liu, C. Wu, M. Menta, L. Herranz, B. Raducanu, A. D. Bagdanov, S. Jui, and J. v. de Weijer.  Generative feature replay for class-incremental learning.  In CVPR Workshop, 2020.
  * [50] D. Lopez-Paz and M. Ranzato.  Gradient episodic memory for continual learning.  In NeurIPS, 2017.
  * [51] F. Lyu, Q. Sun, F. Shang, L. Wan, and W. Feng.  Measuring asymmetric gradient discrepancy in parallel continual learning.  In ICCV, 2023.
  * [52] F. Lyu, S. Wang, W. Feng, Z. Ye, F. Hu, and S. Wang.  Multi-domain multi-task rehearsal for lifelong learning.  In AAAI, 2021.
  * [53] J. Ma, Z. Zhao, X. Yi, J. Chen, L. Hong, and E. H. Chi.  Modeling task relationships in multi-task learning with multi-gate mixture-of-experts.  In ACM SIGKDD, 2018.
  * [54] A. Mallya, D. Davis, and S. Lazebnik.  Piggyback: Adapting a single network to multiple tasks by learning to mask weights.  In ECCV, 2018.
  * [55] A. Maurer, M. Pontil, and B. Romera-Paredes.  Sparse coding for multitask and transfer learning.  In ICML, 2013.
  * [56] H. Nam and B. Han.  Learning multi-domain convolutional neural networks for visual tracking.  In CVPR, 2016.
  * [57] N. Peng and M. Dredze.  Multi-task domain adaptation for sequence tagging.  In Workshop on Representation Learning for NLP, 2017.
  * [58] J. Pomponi, S. Scardapane, and A. Uncini.  Pseudo-rehearsal for continual learning with normalizing flows.  arXiv preprint arXiv:2007.02443, 2020.
  * [59] A. Prabhu, P. Torr, and P. Dokania.  Gdumb: A simple approach that questions our progress in continual learning.  In ECCV, 2020.
  * [60] S.-A. Rebuffi, H. Bilen, and A. Vedaldi.  Efficient parametrization of multi-domain deep neural networks.  In CVPR, 2018.
  * [61] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert.  icarl: Incremental classifier and representation learning.  In CVPR, 2017.
  * [62] L. Risheng, L. Yaohua, Z. Shangzhi, and J. Zhang.  Gradient-based editing of memory examples for online task-free continual learning.  In NeurIPS, 2021.
  * [63] A. Robins.  Catastrophic forgetting, rehearsal and pseudorehearsal.  Connection science, 7(2):123–146, 1995.
  * [64] A. Rosenfeld and J. K. Tsotsos.  Incremental learning through deep adaptation.  IEEE TPAMI, 42(3):651–663, 2018.
  * [65] A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell.  Progressive neural networks.  arXiv preprint arXiv:1606.04671, 2016.
  * [66] O. Sener and V. Koltun.  Multi-task learning as multi-objective optimization.  In NeurIPS, 2018.
  * [67] J. Serra, D. Suris, M. Miron, and A. Karatzoglou.  Overcoming catastrophic forgetting with hard attention to the task.  In ICML, 2018.
  * [68] H. Shah, K. Javed, and F. Shafait.  Distillation techniques for pseudo-rehearsal based incremental learning.  arXiv preprint arXiv:1807.02799, 2018.
  * [69] D. L. Silver, R. Poirier, and D. Currie.  Inductive transfer with context-sensitive neural networks.  Machine Learning, 73(3):313–336, 2008.
  * [70] H. Tang and K. Jia.  Discriminative adversarial domain adaptation.  In AAAI, 2020.
  * [71] S. Tars and M. Fishel.  Multi-domain neural machine translation.  arXiv preprint arXiv:1805.02282, 2018.
  * [72] S. Thrun and J. O’Sullivan.  Discovering structure in multiple learning tasks: The tc algorithm.  In ICML, 1996.
  * [73] L. Van der Maaten and G. Hinton.  Visualizing data using t-sne.  JMLR, 9:2579–2605, 2008.
  * [74] J. Wang and J. Ye.  Safe screening for multi-task feature learning with multiple data matrices.  In ICML, 2015.
  * [75] C. Wu, L. Herranz, X. Liu, J. van de Weijer, B. Raducanu, et al.  Memory replay gans: Learning to generate new categories without forgetting.  In NeurIPS, 2018.
  * [76] Y. Yang and T. M. Hospedales.  A unified perspective on multi-domain and multi-task learning.  In ICLR, 2014.
  * [77] J. Yoon, E. Yang, J. Lee, and S. J. Hwang.  Lifelong learning with dynamically expandable networks.  arXiv preprint arXiv:1708.01547, 2017.
  * [78] K. Yu, V. Tresp, and A. Schwaighofer.  Learning gaussian processes from multiple tasks.  In ICML, 2005.
  * [79] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn.  Gradient surgery for multi-task learning.  In NeurIPS, 2020.
  * [80] F. Zenke, B. Poole, and S. Ganguli.  Continual learning through synaptic intelligence.  In ICML, 2017.
  * [81] B. Zhang, A. Bapna, R. Sennrich, and O. Firat.  Share or not? learning to schedule language-specific capacity for multilingual translation.  In ICLR, 2020.
  * [82] J. Zhang, Z. Ghahramani, and Y. Yang.  Learning multiple related tasks using latent independent component analysis.  In NeurIPS, 2005.
  * [83] W. Zhang, L. Deng, and D. Wu.  Overcoming negative transfer: A survey.  arXiv preprint arXiv:2009.00909, 2020.
  * [84] Y. Zhang and Q. Yang.  A survey on multi-task learning.  IEEE TKDE, 34(12):5586–5609, 2021.
  * [85] Z. Zhao, L. Hong, L. Wei, J. Chen, A. Nath, S. Andrews, A. Kumthekar, M. Sathiamoorthy, X. Yi, and E. Chi.  Recommending what video to watch next: a multitask ranking system.  In ACM CRS, 2019.