P[1]¿p#1

# Adaptive Energy Regularization for Autonomous Gait Transition and Energy-
Efficient Quadruped Locomotion

Boyuan Liang*, Lingfeng Sun*, Xinghao Zhu*, Bike Zhang, Ziyin Xiong, Chenran
Li,
Koushil Sreenath, Masayoshi Tomizuka * Equal contributions. These three
authors are ordered alphabetically.Department of Mechanical Engineering,
University of California, Berkeley, California, USA, 94720. Corresponding
author: Lingfeng Sun (email<EMAIL_ADDRESS>
###### Abstract

In reinforcement learning for legged robot locomotion, crafting effective
reward strategies is crucial. Pre-defined gait patterns and complex reward
systems are widely used to stabilize policy training. Drawing from the natural
locomotion behaviors of humans and animals, which adapt their gaits to
minimize energy consumption, we propose a simplified, energy-centric reward
strategy to foster the development of energy-efficient locomotion across
various speeds in quadruped robots. By implementing an adaptive energy reward
function and adjusting the weights based on velocity, we demonstrate that our
approach enables ANYmal-C and Unitree Go1 robots to autonomously select
appropriate gaits—such as four-beat walking at lower speeds and trotting at
higher speeds—resulting in improved energy efficiency and stable velocity
tracking compared to previous methods using complex reward designs and prior
gait knowledge. The effectiveness of our policy is validated through
simulations in the IsaacGym simulation environment and on real robots,
demonstrating its potential to facilitate stable and adaptive locomotion.
Videos and more details are at
https://sites.google.com/berkeley.edu/efficient-locomotion

##  I Introduction

Humans and animals exhibit various locomotion behaviors at different speeds,
optimizing for their energy efficiency. For instance, humans typically walk at
low speeds and run at higher speeds, rarely opting for jumping. Prior research
demonstrated through optimal control on planar models the correlation between
speed and optimal gait choices concerning the cost of transport (CoT). For
quadrupeds, the optimal gaits were four-beat walking111Four-beat walking, two-
beat walking, trotting and galloping are typical gaits for quadruped robots
defined in [1] based on feet contact schedule. at low speeds, trotting at
intermediate speeds, and trotting/galloping at high speeds [1].

Due to the rich information in the gaits, using a gait as guidance for
locomotion policies is popular among lots of reinforcement learning (RL) based
methods [2, 3]. However, crafting a versatile and robust locomotion policy
that can adapt to and transition between multiple speeds while generalizing
across different platforms poses substantial challenges. One of the main
challenges here is the reward design. Gait reference can be used as extended
state or extra regularization terms in reward functions to provide more
supervision. Previous works [3, 4, 5] trained on different quadruped robots
within simulation environments like IsaacGym [6, 7] and successfully
transferred these policies to physical hardware. However, they often
necessitate intricate reward designs and weight tuning. Apart from gait
information, reward terms like feet-air time and contact force penalizing [4]
were also used to encourage specific behaviors and help stabilize the
training. While these additional reward components are aimed at inducing or
preventing specific behavioral traits, they inadvertently align with the
broader objective of reducing energy costs. This convergence prompts a
reconsideration of our reward strategy: could a more straightforward, energy-
centric reward term be used to replace the specifically designed terms used in
prior policy training? Such a term would encapsulate the core objective of
reducing energy consumption and fostering stable and efficient locomotion.

Figure 1: Compared to legged-gym baseline [4], our single policy (from one-
time RL training) autonomously adopted different energy-efficient gaits (four-
beat walking and trotting). It achieved lower energy consumption (low leg-
swing) at varying speeds ($0.1m/s$ interval; mean and variance from 50
independent runs).

Building on the concept that energy-efficient gaits correlate with speed [1]
and aligning with prior work emphasizing that energy minimization at pre-
selected speed results in the emergence of specific gaits [8], this study
investigates a more streamlined reward formulation for energy-efficient
locomotion. By focusing on energy minimization without intricately designed
reward components, we aim to verify if such a simplified approach can yield
stable and effective velocity-tracking in quadruped robots across various
speeds. Instead of generating multiple velocity-specific energy optimal
policies [8], we focus on getting a single energy-optimal policy across all
target velocities via RL training.

In our research, we examine the influence of energy regularization weights on
policy performance, identifying that both excessively low and high weights can
lead to unnatural movements or immobility. Recognizing that energy terms have
different scales across velocities and require adaptive velocity-conditioned
weights, we first design a non-negative energy reward function and then find
an adaptive reward form by interpolating the maximum energy weights at
selected speeds to facilitate effective velocity tracking.

Employing this adaptive reward structure within IsaacGym enables the training
of robust policies for the ANYmal-C [9] and Unitree Go1 [10] quadruped robots.
Our methodology, illustrated in Figure 1, identifies appropriate gaits, such
as four-beat walking at lower speeds and trotting at higher speeds, without
predefined gait knowledge—outperforming baseline approaches [4] in energy
efficiency. Our policy also significantly improves velocity tracking and
energy consumption performance compared to a policy trained with fixed-weight
energy rewards. The trained single policy is deployed on a real Go1 robot to
verify its stable moving and transition locomotion skills in the real world.

The main contribution of this paper includes:

  * •

Introduction of a streamlined reward formula integrating basic velocity-
tracking and adaptive energy minimization to foster stable, velocity-sensitive
locomotion policies.

  * •

Demonstration that the derived policies autonomously adopt different energy-
efficient gaits at varying speeds without preset gait knowledge.

  * •

Evaluation of the velocity-tracking and energy efficiency across reward
structures and weight tunings for ANYmal-C and Unitree Go1, culminating in the
real-world application of these policies on a Go1 robot, affirming their
efficacy in stable locomotion and gait transition.

##  II Related Works

###  II-A Reinforcement Learning for Locomotion Skills

After deep RL demonstrated its capability to fit general policies in an
unsupervised manner, researchers actively sought its potential to be deployed
on legged locomotion. Hwangbo et al. [11] achieved RL-generated walking
policies on a plane ground using a pre-trained actuator net to reduce the sim-
to-real gap. Further research also achieved training the policy with adaptive
actuator net [12] or motor control parameters [13, 14]. In Hwangbo’s follow-up
works, locomotion on complicated terrains using a similar approach was also
accomplished [15, 16, 17]. With modern GPU-accelerated simulators [6, 7], more
time-efficient training frameworks were proposed [4, 18]. Due to the intensive
reward engineering in these approaches, the model-free RL tends to converge to
a single gait, usually trotting gait, which may not be the most efficient for
all terrains and target velocities [19].

Many efforts were made to overcome this limitation by promoting the behavioral
diversity of legged robots. Researchers in [20] proposed a hierarchical
framework that pre-specifies a set of gait primitives and allows an RL model
to choose from them. In [21, 22], gait primitives were parameterized using the
contact schedule and RL policy was trained to select these parameters. These
works used model predictive control (MPC) as the lower-level controller, which
demands preliminary knowledge of locomotion and contact modeling [23]. In [3],
behavioral-related arguments such as body height, step frequency, and phase
are directly added to the RL model input for end-to-end training. Although
various two-beat gaits (where legs touch the ground in pairs) were realized,
they cannot generate four-beat gaits (where legs touch the ground in orders)
and require manual gait specification in different scenarios. It is more
desirable if the quadruped robot can select the most suitable behavior by
itself. In [8], a pipeline was proposed to output different gaits under
different velocities via minimizing energy consumption. However, this pipeline
relies on training and distillation of several velocity-specific RL policies.

###  II-B Energy Studies on Locomotion

The energetic economy of legged robots has always been an important concern
for researchers. The cost of transport, namely the amount of energy used per
distance traveled, was introduced in [24, 25], where the minimization of CoT
was connected to the choice of speeds [26] and step lengths [27] in human
locomotion behaviors. The optimal velocity varies for different gaits,
allowing animals to transit between different gaits to move at various speeds.

Conceptual legged models of bipedal [28, 29], and quadruped [30, 31, 1] robots
are later designed by researchers to search for energetically optimal motions
on various gaits. For quadruped robots, genetic algorithms were used in [30]
to study potential gaits at different speeds. Our research is mainly inspired
by [1], where optimal control problems are formulated on realistic robot
models considering the effects of leg mass, plastic collisions, and damping
losses. This work uses an unbiased search on energy-efficient locomotion
patterns at various velocities and demonstrates the energy vs. velocity curve
for different gaits. Inspired by the results, we believe there exists a policy
that can transit between gaits at different velocities with an energy-
optimization reward. Among previous works, [8] is close to ours in utilizing
relations between energy and gaits. It generated multiple policies, with each
policy velocity-specific and energy-optimal. In contrast, our work focuses on
generating a single energy-optimal policy across various speeds, and it aims
to replace the complex-designed reward terms in RL.

Figure 2: Top row: Gait plot generated from the ANYmal-C simulation where the
legged robot is first commanded to move at $0.5$ m/s from $0$ to $2$ seconds
and demonstrated four-beat walking (four legs touch the ground one by one in
order, refer to [1] for details), then at $1.0$ m/s from $2$ to $4$ seconds
and demonstrated a transition gait between walking and trotting, finally at
$2.0$ m/s for $4$ to $6$ seconds and demonstrated trotting (legs touch the
ground in pairs). Middle row: Snapshots from the ANYmal-C simulation taken
approximately at the vertical purple lines in the gait plot. Bottom row:
Snapshots of Go1 moving on a playground. Its feet contact state approximately
corresponds to the vertical purple line in the gait plot.

##  III Locomotion Reward Design

A general form of energy regularized locomotion reward takes the following
form:

$R=R_{motion}+R_{energy}+R_{others}$ (1)

where $R_{motion}$ encourages accurate velocity tracking, $R_{energy}$
discourages energy consumption and $R_{others}$ includes other necessary
rewards to stabilize training. In previous work [8], motion rewards include
penalty on linear and angular velocity tracking errors; energy rewards include
penalty on motor power with a fixed weight; survival bonus is also added. In
experiments, we found the training process unstable potentially due to the
negative nature of tracking and energy rewards. Besides, each energy reward
weight usually only works within a very narrow range of reference speeds. It
is hard to find a single energy reward weight value that works for all
reference velocities without knowing more simulation settings and training
details. As a result, we proposed the following reward function to promote the
automatic generation of energy-efficient behavior of legged robots under
various reference velocities.

$R=\frac{1}{Z(\hat{v}_{x})}\Big{[}R_{lin}+\alpha_{ang}R_{ang}+\alpha(\hat{v}_{x})R_{en}\Big{]}$
(2)

where $\alpha_{ang}=0.5$, $\hat{v}_{x}$ is the user-specified reference
velocity, $\alpha(\hat{v}_{x})$ is the adaptive energy reward weight,
$Z(\hat{v}_{x})$ is the normalizing index for total reward, $R_{lin},R_{ang}$
are velocity tracking rewards, and $R_{en}$ is the energy reward. The
remaining section elaborates on each component in (2).

####  III-1 Motion Rewards

$R_{lin}$ and $R_{ang}$ respectively encourage the legged robot to track the
linear reference velocities in two directions $\hat{v}_{x},\hat{v}_{y}$ and
angular reference velocities $\hat{\omega}_{z}$.

$\begin{split}R_{lin}&=\exp\Big{(}-\frac{|v_{x}-\hat{v}_{x}|^{2}+|v_{y}-\hat{v}_{y}|^{2}}{\sigma_{v}}\Big{)}\\\
R_{ang}&=\exp\Big{(}-\frac{|\omega_{z}-\hat{\omega}_{z}|^{2}}{\sigma_{\omega}}\Big{)}\end{split}$
(3)

$\hat{v}_{y}$ and $\hat{\omega}_{z}$ are not user-specified commands, but
randomly sampled during training as explained in section IV. $\sigma_{v}$ and
$\sigma_{\omega}$ are scaling factors depending on the training velocity
range. The structure of motion rewards, the coefficient $\alpha_{ang}=0.5$ for
angular velocity tracking, and the scaling coefficients follow the default
setting in legged-gym [4].

####  III-2 Energy Rewards

$R_{en}$ rewards the system for consuming less energy while moving.

$R_{en}=\exp\Big{(}-\frac{\sum_{i}|\tau_{i}||\dot{q}_{i}|}{\sigma_{en}}\Big{)}$
(4)

The exponential form guarantees a positive reward. $\tau$ are each joint’s
actuated torques, and $\dot{q}$ are joint velocities. We multiply the absolute
values of each entry of $\tau$ with each entry of $\dot{q}$ and sum them up in
(4) to follow the fact that a motor does not get charged back even when the
applied torque is opposite to the motion [32]. $\sigma_{en}$ is an energy
scaling constant.

####  III-3 Adaptive Energy Weight

$\alpha(\hat{v}_{x})$ is the weight of the energy reward terms. Previous works
take this value as a constant [8, 32], but we argue that it should be adaptive
to the reference velocity $\hat{v}_{x}$ to achieve suitable behaviors directly
from RL. To verify this, we first run RL under fixed sampled $\hat{v}_{x}$
using pre-selected $\sigma_{en}$ and try various values of
$\alpha(\hat{v}_{x})$. When $\alpha(\hat{v}_{x})$ is too large, the robot
tends to stay unmoved to save energy, neglecting the velocity tracking task.
When $\alpha(\hat{v}_{x})$ is too small, the robot tends to use a highly
inefficient and unnatural way to walk.

To find an adaptive weight, we first collect the largest $\alpha(\hat{v}_{x})$
when the velocity tracking error is smaller than a pre-defined small threshold
$\delta$ for a set of speeds $\hat{v}_{x}$. After collecting velocity-weight
sample pairs $(\hat{v}_{x,j},\alpha(\hat{v}_{x,j}))_{j=1}^{M}$ for a range of
velocities, we see a clear trend (see Figure 3): with velocity increasing, the
maximum allowed $\alpha$ decreases. This trend corresponds to the fact that
the kinetic energy increases quadratically with the velocity. While the motion
rewards are expected to converge close to zero, the energy reward has variant
optimal values across velocities. Therefore, we can set a larger weight for
lower-speed training but need to decrease it as the velocity increases. When
training velocity-conditioned locomotion policies, we linearly interpolate
between selected pairs to acquire the velocity-conditioned energy weight
$\alpha(\hat{v}_{x})$ for the current sampled velocity. We will show in
experiments that this is a simple but effective way of training energy-
effective policies across different velocities.

####  III-4 Normalization Index

$Z(\hat{v}_{x})$ is an adaptive normalization index. Due to the adaptive
$\alpha(\hat{v}_{x})$, the reward scale under different command velocities
varies. This often leads to instability in RL training. As such, for each
velocity-weight sample pair $(\hat{v}_{x,j},\alpha(\hat{v}_{x,j}))$, we also
record the final achieved reward $Z(\hat{v}_{x,j})$ and analogously use linear
interpolation to get the normalization index curve $Z(\hat{v}_{x})$ to
stabilize training.

##  IV Locomotion Skill Training Details

Section III focused on the reward design of the proposed method, which
constitutes the most essential part of RL training. However, the energy reward
is not a stand-alone one that can be used directly for RL training, and we
have to use it together with the basic locomotion rewards. Across quadruped
locomotion baselines, the default training settings and locomotion rewards
slightly differ. This section explains the basic RL training and simulation
settings other than energy regularization. These basic settings can also
generate a naive locomotion policy without energy regularization, but these
policies usually have low energy efficiency and might have undeployable
abnormal behavior. The two baselines we use in this research are legged-gym
[4] for ANYmal-C and walk-these-ways [3] for Unitree Go1.

###  IV-A ANYmal-C Settings

We utilized the robot model and PPO training package in [4]. The system
outputs the position command of the 12 joints in the next time step. The
system inputs include the linear and angular velocities of the trunk,
projected gravity in the robot frame, the commanded x-y velocities, the
commanded yaw rate, each joint’s position and velocity, as well as the action
at previous time step. The commanded y-velocity and yaw rate are fixed at zero
here; only the commanded x-velocity will be set. The training episode will
reset after 1000 time steps or if any part of the robot except its feet
touches the floor.

The policy was trained on a flat ground with the coefficient of friction
randomized between $[0.0,1.5]$. We also disturbed the mass of the robot with a
uniform random value in $[-5.0,5.0]$ kg. A uniformly distributed noise was
added to the observation. A random push with x-y velocity uniformly sampled
between $[-1,1]$ m/s lasting 15 seconds was exerted on the robot. All these
randomized domain parameters are renewed every time the training episode
resets, except observation noise is resampled after every time step.

In motion rewards (3), during fixed velocity training (to get results in
Figure 3) and when the $\hat{v}_{x}$ is large, the policy may fail to converge
to a desirable tracking accuracy. Hence, we use
$\sigma_{v}(\hat{v}_{x})=|\hat{v}_{x}|^{2}/3$ to overcome this difficulty and
$\sigma_{\omega}$ is fixed at $0.25$. In energy rewards (4), the energy
scaling constant $\sigma_{en}$ is fixed at $800$. To obtain the energy weight
curve $\alpha(\hat{v}_{x})$ and the normalization index $Z(\hat{v}_{x})$, we
trained fixed-velocity policies by setting $\hat{v}_{x}$ at $0.5$ to $2.0$
m/s, with $0.1$ common difference. For each fixed reference velocity, we
trained $11$ policies using different $\alpha(\hat{v}_{x})$ values ranging
from $0.5$ to $4.5$. Figure 3 summarizes the corresponding
$\alpha(\hat{v}_{x})$ and $Z(\hat{v}_{x})$ for each reference velocity. These
dense parameter pairs are exhibited to visualize the $\alpha$-$\hat{v}_{x}$
relation; in real experiments, only a few pairs of parameters at
representative speeds are required.

Figure 3: $\alpha(\hat{v}_{x})$ and $Z(\hat{v}_{x})$ corresponding to each
reference velocities. These two values are defined in (2). For each reference
velocity, fixed-velocity trainings were conducted across various
$\alpha(\hat{v}_{x})$, then we select the largest $\alpha(\hat{v}_{x})$ whose
velocity tracking error is smaller than a threshold.

(a) Comparison of energy consumption

(b) Comparison of velocity tracking error

Figure 4: Ablation study in ANYmal-C simulation. Reference velocities are
chosen from $0.5$ to $2.0$ m/s with $0.1$ common gap. Results of each velocity
are obtained from $50$ independent runs. The solid line indicates the average
and the shadow indicates the variance. Left: Energy consumption per unit
moving distance under different reference velocities. We can see that adaptive
energy regularization generates the most energy-efficient policy compared to
fixed energy regularization. The $\alpha=0$ policy is neglected because it
consumes an uncontrollable amount of energy. Right: Velocity tracking error
under different reference velocities. Among all fixed energy regularization
policies, only $\alpha=0.9$ has a comparable tracking error. However, the left
figure shows that $\alpha=0.9$ policy is considerably less energy efficient.

###  IV-B Go1 Settings

We mainly inherited the training methods released by [3]. Similar to [4], the
system also outputs the position command of the 12 joints, but its input
excludes the linear and angular velocities of the trunk. In addition, the
inputs of the previous 30 time steps are also given to the RL system. Compared
to ANYmal-C, we found that the energy reward $R_{en}$ alone is insufficient to
regularize Go1’s behavior, which is likely due to the lighter weight compared
to its motor power. Thus, following the settings in [3], we further add a
fixed auxiliary reward $R_{aux}$ to (2) as an amendment to the normalization
index $Z(\hat{v}_{x})$. This auxiliary reward is derived mainly from safety
concerns, such as penalizing limb-ground collision, out-of-range joint
position, and high frequency joint action. The details of $R_{aux}$ can be
found on the project website. The training episode will reset after 1000 time
steps or if the trunk touches the floor. Similar domain randomization in
section IV-A is applied here. Compared to [3], we did not include any gait-
related rewards.

We also discovered that curriculum technique is essential even for fixed
velocity training. Given a reference velocity $\hat{v}_{x}$, the sampling
range of x-velocity starts with
$[-\min\\{\hat{v}_{x},1\\},\min\\{\hat{v}_{x},1\\}]$ m/s. the sampling range
increases when the total reward achieves a certain threshold, and the maximal
sampling range is set at $[-\hat{v}_{x}-0.1,\hat{v}_{x}+0.1]$ m/s. The fixed
velocity training is considered a success if the x-velocity sampling range
expanded to $[-\hat{v}_{x}-0.1,\hat{v}_{x}+0.1]$ m/s in the end and the
tracking error for speed $\hat{v}_{x}$ is smaller than a threshold $\delta$.

In motion rewards (3), both $\sigma_{v}(\hat{v}_{x})$ and $\sigma_{\omega}$
were fixed at $0.25$. In energy rewards (4), the energy scaling constant
$\sigma_{en}$ is fixed at $300$. We trained fixed-velocity policies by setting
$\hat{v}_{x}$ from $0.5$ to $2.5$ m/s, with a 0.5 common difference. For each
reference velocity, we trained eight policies using different $\alpha$ values
ranging from $0.7$ to $2.1$, then fit the $\alpha(\hat{v}_{x})$ curve with the
method in section III.

##  V Experiments

The experiments were designed to show the following after the legged robot was
trained using the adaptive energy-regularized reward shown in Equation (2).

  * •

With adaptive $\alpha(\hat{v}_{x})$ and $Z(\hat{v}_{x})$, the legged robot
automatically selects suitable behaviors to move with the reference velocity.
We also demonstrate that the trained policy is deployable to a real quadruped
robot.

  * •

If $\alpha(\hat{v}_{x})$ and $Z(\hat{v}_{x})$ are constants, the legged robot
may not be able to find an energy-efficient walking policy for all target
velocities.

  * •

Our method can generate a more energy-efficient walking policy compared to
established baselines.

Figure 5: Three policies walking at $0.5$ m/s. Built-in MPC and walk-these-
ways (WTW) [3] swing the legs to a redundant height, while our policy only
makes necessary lifting of the leg. This demonstrates the energy-efficiency of
our policy.

###  V-A One Policy with Different Gaits at Different Velocities

To demonstrate natural emergence of efficient locomotion gaits, we evaluate
the trained walking policy under different reference velocities. Fig. 2
demonstrates a trial run where the legged robot was commanded to move at
$\hat{v}_{x}=0.5$, $1.0$ and then $2.0$ m/s. Each commanded velocity lasts for
two seconds. We plot the gait recorded from the ANYmal-C simulation and show
the snapshots for simulated ANYmal-C and real world Go1. We can see that our
trained policy exhibits four-beat walking at low speed ($0.5$ m/s) by moving
one leg each time in the order of right hind, right front, left hind and left
front. At medium speed ($1.0$ m/s), the policy exhibits an intermediate gait
between walking and trotting. At this gait, the right hind and left front legs
move at the same time, while the right front and left hind legs have a
displacement in their motions. At high speed ($1.0$ m/s), the trained policy
exhibits a standard trotting gaits, where the right hind and left front legs
move together, and the right front and left hind legs move together. This gait
transition is endorsed by previous works [1, 21] that four-beat walking and
trotting are respectively the most energy-efficient gaits under low and high
speeds.

A closer look at the snapshots in Figure (2) uncovers that the swing ratio was
also regularized. The legged robot only lifts up its leg to a height necessary
to reach the reference velocity $\hat{v}_{x}$ to avoid wasting energy. At low
speed, it only mildly lifts up its feet. As the reference velocity increases,
the robot also lifts its feet higher, but only to a necessary height. Without
considering energy-efficiency, the trained policy usually tends to lift the
feet redundantly high. This will be argued in more detail in section V-C.

Finally, Figure 5 also showcases a successful hardware deployment of our
policy. Videos can be found in our project website stated in the abstract.

###  V-B Ablation Studies with Fixed Energy Rewards

In ANYmal-C simulation, when the reference velocity is fixed at $1.0$ and
$2.0$ m/s, we found that velocity tracking error was reasonably small when
$\alpha$ was set as $3.0$ and $0.9$. Therefore, we run ablation experiments by
fixing $\alpha$ at these values and $0.0$ to compare with varying
$\alpha(\hat{v}_{x})$.

Figure 4(a) shows the energy consumption for adaptive $\alpha$, $\alpha=0.9$
and $\alpha=3.0$. The policy with $\alpha=0.0$ is dropped because it consumes
multiple orders more energy. We observe that adaptive $\alpha$ reaches the
lowest energy consumption. Figure 4(b) shows the velocity tracking error. Only
the $\alpha=0.9$ policy has comparable tracking accuracy with adaptive
$\alpha$, but Figure 4(a) shows that $\alpha=0.9$ policy consumes considerable
more energy. These observations conclude the non-triviality of finding a
constant energy reward scale $\alpha$ and elaborates the usefulness of
$\alpha(\hat{v}_{x})$ varying with reference velocity.

###  V-C Comparison to Other Methods on Go1

We also compare our method with built-in MPC and walk-these-ways on real world
Go1 robot. All policies are commanded to move at $0.5$ m/s. Such low velocity
requires only mild movement of each leg for Go1. Figure 5 shows snapshots of
each policy. It can be seen that both built-in MPC and walk-these-ways over-
swing the legs, which squanders energy. Our policy swings the leg only to a
necessary height, which can be visually recognized as the most energy-
efficient.

##  VI Discussions

This paper focuses on developing energy-efficient locomotion strategies for
quadruped robots, employing a simple yet effective reinforcement learning
approach. However, there are some inherent limitations which provide avenues
for future research.

####  VI-1 Limitations

The core limitation of the presented approach lies in the requirement for pre-
running experiments to determine appropriate energy regularization weights.
This necessity stems from our method’s reliance on empirical observations to
calibrate the weights, which, while effective, does not afford the flexibility
in a fully adaptive reinforcement learning system. Although the policy
developed is applicable across different speeds post-training, it cannot be
obtained with only one training for a new quadruped platform. Moreover, we
only tested its generalizability across speeds. We did not verify the adaptive
capability of energy rewards in different environments, which would be crucial
for deploying these robots in real-world scenarios where they might encounter
multiple operational challenges.

####  VI-2 Future extensions

Future research could address these limitations to develop methodologies for
automatically tuning energy regularization weights within one single
reinforcement learning training. This would enable the system to dynamically
adjust its strategy in response to multi-task RL [33] or cross-embodiment
settings [34, 35, 36].

Moreover, while this study concentrated on locomotion tasks, the underlying
principle of leveraging energy efficiency to drive behavior selection holds
broader potential. Future work could explore applying this energy-centric
approach across different robotic tasks. For instance, manipulation and
interaction tasks could also benefit from strategies prioritizing energy
efficiency, potentially finding natural, efficient behaviors analogous to
those observed in biological systems [37, 38]. Such a framework would align
robotic systems more closely with sustainability principles and environmental
consciousness.

##  VII Conclusions

This paper presented a novel approach to energy-efficient locomotion in
quadruped robots through the implementation of a simplified, energy-centric
reward strategy within a reinforcement learning framework. Our method
demonstrated that quadruped robots, specifically ANYmal-C and Unitree Go1,
could autonomously develop and transition between various gaits across
different velocities without relying on predefined gait patterns or intricate
reward designs. The adaptive energy reward function, adjusted based on
velocity, enabled these robots to select the most energy-efficient locomotion
strategies naturally. Our policy showed energy-efficient behaviors and gait
transitions in both simulation experiments (ANYmal-C) and hardware experiments
(Go1). We also demonstrated the usefulness of adaptive energy regularization
via ablation studies.

## Acknowledgements

This project was supported by the Agency of Science, Technology and Research
of Singapore via the National Science Scholarship, the FANUC Advanced Research
Laboratory, and the InnoHK of the Government of the Hong Kong Special
Administrative Region via the Hong Kong Centre for Logistics Robotics.

## References

  * [1] W. Xi, Y. Yesilevskiy, and C. D. Remy, “Selecting gaits for economical locomotion of legged robots,” _The International Journal of Robotics Research_ , vol. 35, no. 9, pp. 1140–1154, 2016.
  * [2] J. Siekmann, Y. Godse, A. Fern, and J. Hurst, “Sim-to-real learning of all common bipedal gaits via periodic reward composition,” in _IEEE International Conference on Robotics and Automation_ , 2021.
  * [3] G. B. Margolis and A. Pulkit, “Walk these ways: Tuning robot control for generalization with multiplicity of behavior,” in _Conference on Robot Learning_ , 2023.
  * [4] N. Rudin, D. Hoeller, P. Reist, and M. Hutter, “Learning to walk in minutes using massively parallel deep reinforcement learning,” in _Conference on Robot Learning_ , 2021.
  * [5] G. Feng, H. Zhang, Z. Li, X. B. Peng, B. Basireddy, L. Yue, Z. Song, L. Yang, Y. Liu, K. Sreenath, _et al._ , “Genloco: Generalized locomotion controllers for quadrupedal robots,” in _Conference on Robot Learning_. PMLR, 2023.
  * [6] J. Liang, V. Makoviychuk, A. Handa, N. Chentanez, M. Macklin, and D. Fox, “Gpu-accelerated robotic simulation for distributed reinforcement learning,” in _Conference on Robot Learning_ , 2018.
  * [7] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State, “Isaac gym: High performance GPU based physics simulation for robot learning,” in _Conference on Neural Information Processing Systems_ , 2021.
  * [8] Z. Fu, A. Kumar, J. Malik, and D. Pathak, “Minimizing energy consumption leads to the emergence of gaits in legged robots,” in _Conference on Robot Learning_ , 2021.
  * [9] M. Hutter, C. Gehring, A. Lauber, F. Gunther, C. D. Bellicoso, V. Tsounis, P. Fankhauser, R. Diethelm, S. Bachmann, M. Bloesch, H. Kolvenbach, M. Bjelonic, L. Isler, and K. Meyer, “Anymal - toward legged robots for harsh environments,” _Advanced Robotics_ , vol. 31, no. 17, pp. 918–931, 2017.
  * [10] “Unitree robotics, go1,” https://www.unitree.com/products/go1, online; accessed Jun. 2022.
  * [11] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter, “Learning agile and dynamic motor skills for legged robots,” _Science Robotics_ , vol. 4, no. 26, p. eaau5872, 2019.
  * [12] G. Ji, J. Mun, H. Kim, and J. Hwangbo, “Concurrent training of a control policy and a state estimator for dynamic and robust legged locomotion,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 4630–4637, 2022.
  * [13] Z. Li, X. Cheng, X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath, “Reinforcement learning for robust parameterized locomotion control of bipedal robots,” in _IEEE International Conference on Robotics and Automation_ , 2021.
  * [14] S. Chen, B. Zhang, M. W. Mueller, A. Rai, and K. Sreenath, “Learning torque control for quadrupedal locomotion,” in _IEEE-RAS International Conference on Humanoid Robots_ , 2023.
  * [15] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning robust perceptive locomotion for quadrupedal robots in the wild,” _Science Robotics_ , vol. 7, no. 62, p. eabk2822, 2022.
  * [16] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning quadrupedal locomotion over challenging terrain,” _Science Robotics_ , vol. 5, no. 47, p. eabc5986, 2020.
  * [17] S. Choi, G. Ji, J. Park, H. Kim, J. Mun, J. H. Lee, and J. Hwangbo, “Learning quadrupedal locomotion on deformable terrain,” _Science Robotics_ , vol. 8, no. 74, p. eade2256, 2023.
  * [18] G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal, “Rapid locomotion via reinforcement learning,” _The International Journal of Robotics Research_ , vol. 43, no. 4, pp. 572–587, 2024.
  * [19] M. H. Raibert, “Trotting, pacing and bounding by a quadruped robot,” _Journal of Biomechanics_ , vol. 23, pp. 79–98, 1990.
  * [20] X. Da, Z. Xie, D. Hoeller, B. Boots, A. Anandkumar, Y. Zhu, B. Babich, and A. Garg, “Learning a contact-adaptive controller for robust, efficient legged locomotion,” in _Conference on Robot Learning_ , 2020.
  * [21] Y. Yang, T. Zhang, E. Coumans, J. Tan, and B. Boots, “Fast and efﬁcient locomotion via learned gait transitions,” in _Conference on Robot Learning_ , 2021.
  * [22] H. Duan, A. Malik, J. Dao, A. Saxena, K. Green, J. Siekmann, A. Fern, and J. Hurst, “Sim-to-real learning of footstep-constrained bipedal dynamic walking,” in _IEEE International Conference on Robotics and Automation_ , 2022.
  * [23] S. Le Cleac’h, T. A. Howell, S. Yang, C.-Y. Lee, J. Zhang, A. Bishop, M. Schwager, and Z. Manchester, “Fast contact-implicit model predictive control,” _IEEE Transactions on Robotics_ , vol. 40, pp. 1617–1629, 2024.
  * [24] G. Gabrielli, “What price speed? specific power required for propulsion of vehicles,” _Mechanical Engineering-CIME_ , vol. 133, no. 10, pp. 4–5, 2011.
  * [25] V. Tucker, “The energetic cost of moving about: Walking and running are extremely inefficient forms of locomotion. much greater efficiency is achieved by birds, fish, and bicyclists.” _American Scientist_ , vol. 63, no. 4, pp. 413–419, 1975.
  * [26] H. Ralston, “Energy-speed relation and optimal speed during level walking.” _Internationale Zeitschrift für angewandte Physiologie einschließlich Arbeitsphysiologie_ , vol. 17, no. 4, pp. 277–283, 1958.
  * [27] B. Umberger and P. Martin, “Mechanical power and efficiency of level walking with different stride rates.” _Journal of Experimental Biology_ , vol. 210, no. 18, pp. 3255–3265, 2007.
  * [28] C. Chevallereau and Y. Aoustin, “Optimal reference trajectories for walking and running of a biped robot.” _Robotica_ , vol. 19, no. 5, pp. 557–569, 2001.
  * [29] C. D. Remy, “Optimal exploitation of natural dynamics in legged locomotion.” Ph.D. dissertation, Eidgenossische Technische Hochschule., 2011.
  * [30] K. Kiguchi, Y. Kusumoto, K. Watanabe, I. K, and F. T, “Energy-optimal gait analysis of quadruped robots.” _Artificial Life and Robotics._ , vol. 6, no. 3, pp. 120–125, 2002.
  * [31] A. Muraro, C. Chevallereau, and Y. Aoustin, “Optimal trajectories for a quadruped robot with trot, amble and curvet gaits for two energetic criteria.” _Multibody System Dynamics_ , vol. 9, no. 1, pp. 39–62, 2003.
  * [32] Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwertfeger, C. Finn, and D. Pathak, “Robot parkour learning,” in _Conference on Robot Learning_ , 2023.
  * [33] L. Sun, H. Zhang, W. Xu, and M. Tomizuka, “Paco: Parameter-compositional multi-task reinforcement learning,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 21 495–21 507, 2022.
  * [34] K. Zakka, A. Zeng, P. Florence, J. Tompson, J. Bohg, and D. Dwibedi, “Xirl: Cross-embodiment inverse reinforcement learning,” in _Conference on Robot Learning_ , 2022, pp. 537–546.
  * [35] L. Sun, H. Zhang, W. Xu, and M. Tomizuka, “Efficient multi-task and transfer reinforcement learning with parameter-compositional framework,” _Robotics and Automation Letters_ , 2023.
  * [36] Z.-H. Yin, L. Sun, H. Ma, M. Tomizuka, and W.-J. Li, “Cross domain robot imitation with invariant representation,” in _IEEE International Conference on Robotics and Automation_ , 2022, pp. 455–461.
  * [37] X. Cheng, A. Kumar, and D. Pathak, “Legs as manipulator: Pushing quadrupedal agility beyond locomotion,” in _IEEE International Conference on Robotics and Automation_ , 2023.
  * [38] E. Vollenweider, M. Bjelonic, V. Klemm, N. Rudin, J. Lee, and M. Hutter, “Advanced skills through multiple adversarial motion priors in reinforcement learning,” in _IEEE International Conference on Robotics and Automation_ , 2023.


# Mitigating Catastrophic Forgetting in Task-Incremental Continual Learning
with Adaptive Classification Criterion

Yun Luo 1 $\\#$, Xiaotian Lin 2 $\\#$, Zhen Yang 3, Fandong Meng 3, Jie Zhou
3, Yue Zhang 1,4 *
1 School of Engineering, Westlake University, Hangzhou, 310024, P.R. China.
2 Guangdong University of Foreign Studies. Guangzhou
3 Pattern Recognition Center, WeChat AI, Tencent Inc, Beijing, China.
4 Institute of Advanced Technology, Westlake Institute for Advanced Study,
Hangzhou, 310024, P.R. China.
{luoyun<EMAIL_ADDRESS>
{zieenyang, fandongmeng<EMAIL_ADDRESS>
###### Abstract

Task-incremental continual learning refers to continually training a model in
a sequence of tasks while overcoming the problem of catastrophic forgetting
(CF). The issue arrives for the reason that the learned representations are
forgotten for learning new tasks, and the decision boundary is destructed.
Previous studies mostly consider how to recover the representations of learned
tasks. It is seldom considered to adapt the decision boundary for new
representations and in this paper we propose a Supervised Contrastive learning
framework with adaptive classification criterion for Continual Learning
(SCCL), In our method, a contrastive loss is used to directly learn
representations for different tasks and a limited number of data samples are
saved as the classification criterion. During inference, the saved data
samples are fed into the current model to obtain updated representations, and
a k Nearest Neighbour module is used for classification. In this way, the
extensible model can solve the learned tasks with adaptive criteria of saved
samples. To mitigate CF, we further use an instance-wise relation distillation
regularization term and a memory replay module to maintain the information of
previous tasks. Experiments show that SCCL achieves state-of-the-art
performance and has a stronger ability to overcome CF compared with the
classification baselines.

##  1 Introduction

Continual learning aims to continually train models with new tasks without
forgetting previously learned tasks Ke and Liu (2022); De Lange et al. (2022).
It has become a promising direction for NLP models to incrementally learn new
tasks/domains/classes as humans do Ke and Liu (2022). A typical scenario aims
to enable NLP models to solve various tasks in an incremental manner, namely
the task-incremental continual learning scenario, which is our study setting
in this paper. A salient challenge for continual learning is that continually
learned models usually suffer from cartographic forgetting (CF), i.e. the
performance on previously learned tasks decreases after training on the new
one Lopez-Paz and Ranzato (2017).

Figure 1: Illustration of representations after contrastive continual learning
on a task before and after learning a new task.

Various training strategies have been proposed to mitigate CF Li and Hoiem
(2017); Kirkpatrick et al. (2017); Lopez-Paz and Ranzato (2017). Under the
fixed model structure, regularization-based methods design regularization
terms to control the shift of representations learned from previous tasks Li
and Hoiem (2017); Kirkpatrick et al. (2017); Aljundi et al. (2018). Rehearsal-
based methods save the data samples from previous tasks into a memory buffer
and re-train the model to recover knowledge during training on the current
task Riemer et al. (2019); Lopez-Paz and Ranzato (2017); de Masson D’Autume et
al. (2019). However, most continual learning methods are designed to recover
the learned knowledge or mitigate the representation of forgetting.

Seldom considers adapting the classification criterion for the newly learned
representations. For example, in supervised contrastive learning, the
contrastive objective is designed to pull the data representations with the
same labels together and push representations with different labels away Chen
et al. (2020a); Gao et al. (2021); Zhang et al. (2021); Neelakantan et al.
(2022); Zhao et al. (2022). Representations of the training samples can be
saved as a classification criterion, after which an instance-based method such
as a k Nearest Neighbor (kNN) module can be leveraged for inference Kassner
and Schütze (2020); Khandelwal et al. (2020). After learning the new task, we
can feed the saved sample into models for new classification criteria and
mitigate the problem of CF. For example in Figure 1, although the
representations have decayed for learning the new task, the saved samples
adapt to serve as the classification criterion in kNN modules.

Inspired by the above motivation, we investigate the use of supervised
contrastive learning for task-incremental continual learning (SCCL). After
supervised contrastive learning on each task, we use a K-means module to
select several samples and save them into a memory buffer while maintaining
the learned representation distribution. In addition, to mitigate the
representation drift when training the model for new tasks, we use an
instance-wise relation distillation (IRD) term Fang et al. (2020); Cha et al.
(2021) and a memory replay module de Masson D’Autume et al. (2019) to maintain
the learned knowledge. During inference, the saved samples are fed into the
trained model to obtain updated representations and a kNN module is used for
classification.

Experimental results show that our proposed model can achieve state-of-the-art
performance compared with standard cross-entropy-based (CE) baselines. We
additionally extend different continual learning strategies Kirkpatrick et al.
(2017); Aljundi et al. (2018); Li and Hoiem (2017) to the supervised
contrastive continual learning framework, which gives stronger results than
corresponding CE-based methods, showing the advantage of contrastive learning
with a kNN classifier in continual learning scenarios. We further analyze the
effectiveness of each module in our paper through ablation studies. To our
knowledge, we are the first to propose a supervised contrastive learning
framework for task-incremental continual learning, without any augmented
parameters. The code will be released when accepted.

Figure 2: The model framework of SCCL contains four main modules: (1) the
supervised contrastive learning for each task; (2) the explicit control of
catastrophic forgetting with IRD knowledge distillation and memory replay; (3)
the selection of learned representations; (4) a kNN inference module.

##  2 Related Work

Continual Learning Various continual learning methods have been proposed to
mitigate the problem of CF. The methods can be broadly divided into
architecture-based methods Yoon et al. (2018); Serra et al. (2018),
regularization-based methods Li and Hoiem (2017); Kirkpatrick et al. (2017),
and rehearsal-based methods Rolnick et al. (2019). Under the fixed model
structure, regularization-based methods Kirkpatrick et al. (2017); Aljundi et
al. (2018); Li and Hoiem (2017) optimize network parameters on the current
task while constraining the representation drift. For example, Li and Hoiem
(2017) propose learning without forgetting (LwF) to tackle this problem, which
regularizes the model output of current data close to those trained for the
previous model. Another category of fixed-structure strategies (rehearsal-
based) stores a limited subset of samples from previous tasks to mitigate CF
such as ER Rolnick et al. (2019), RM Bang et al. (2021), and iCaRL Rebuffi et
al. (2017).

Contrastive Learning Contrastive learning is initially introduced in self-
supervised settings and proved to subsume or significantly outperform
traditional contrastive losses such as triplet loss Chen et al. (2020b); Wu et
al. (2018); Gao et al. (2021). For example, Khosla et al. (2020) first propose
the idea of self-supervised contrastive learning and prove that the method is
more robust to natural corruptions, stable to hyper-parameter settings, and
has strong transfer performance. Luo et al. (2022) uses supervised contrastive
learning combined with a kNN inference module for cross-domain sentiment
analysis, showing a stronger generalization ability compared with standard CE-
based methods.

Cha et al. (2021) propose a contrastive continual learning method, Co2L, for
class-incremental continual learning. The method uses an asymmetric supervised
contrastive loss to enlarge the distance between representations of previous
and new tasks. However, there are significant differences between our model
and Co2L. First, the asymmetric contrastive loss of Co2L is unsuitable for
task-incremental continual learning, because a representation can be predicted
as different labels according to task objectives. Second, Co2L uses a
decoupled classification layer for inference, i.e. it learns representations
first and then learns a linear classifier separately, causing low
extensibility and high complexity of the model. In contrast, we use a kNN
module as the classification criteria to enhance the extensibility of the
model. Third, Co2L only considers the representation drift from the view of
regularization. But we also mitigate the problem of representation drift by
feeding memory data into the current model to obtain updated classification
criteria.

##  3 Method

The overall SCCL framework is illustrated in Figure 2, consisting of four
parts. First, we introduce the contrastive learning objective of SCCL in
Section 3.1. Second, the selection of learned representations is shown in
Section 3.2. Third, an instance-wise distillation module and a memory replay
module are introduced to preserve learned knowledge in Section 3.3 and 3.4,
respectively. Fourth, the kNN inference procedure is shown in 3.5,
respectively. The training algorithm is shown in Algorithm 1.

Formally, a model learns several tasks denoted as $\\{T^{i}\\},i=1,2,...,n$
($i$ is the number of tasks). Each task $T^{i}$ contains a limited set of
labels $C^{i}$. During the training of the task $T^{i}$, only the
corresponding data $D^{i}=\\{(x^{i}_{j},y^{i}_{j})\\}$ are available, where
$x^{i}_{j}$ is the input text and $y^{i}_{j}\in C^{i}$ is the corresponding
label. In the scenario of task-incremental continual training, the task id can
be observed when carrying out inference, and for generality, we consider the
label set $C^{j}\cap C^{k}=\emptyset,$ if $i\neq j$.

###  3.1 Supervised Contrastive Continual Training (SCCL)

During the learning on the task $T^{i}$, we first feed the input $x^{i}_{j}$
into a pre-trained language model to obtain hidden states. The hidden states
of a special token $[CLS]$ (the beginning token of the pre-trained language
model) are regarded as the representation of the input sequence:

${h}^{i}_{j}=Norm(LM^{i}(x^{i}_{j})[CLS]),$ (1)

where $Norm(\cdot)$ refers to normalization, $LM^{i}$ is the language model
encoder trained for the task $T^{i}$, and $LM^{0}$ is the initial pre-trained
language model.

We denote the data samples in a mini-batch as $A$ (we omit the corner mark $i$
during task $T^{i}$ for simplicity). For each data sample $j$, we denote
$\mathcal{N}(j)\equiv A/\\{j\\}$, and the positive neighbor set of it as
$P(j)=\\{u|y_{u}=y_{j}\ and\ u\in N(j)\\}$. To push the representations with
different labels away, and pull them with the same labels together, we use
supervised contrastive learning objective following Khosla et al. (2020):

$\mathcal{L}_{cl}=\sum_{j\in A}\frac{-1}{|P(j)|}\sum_{p\in
P(j)}log\frac{exp({h}^{i}_{j}\cdot{h}^{i}_{p}/\kappa)}{\sum_{a\in
N(j)}exp({h}^{i}_{j}\cdot{h}^{i}_{a}/\kappa)}$ (2)

where $\kappa$ is the hyper-parameter of temperature.

###  3.2 Sample Selection

After training on each task $T^{i}$, we select $m$ samples from training data
of $D^{i}$ to keep the representation distribution with respect to the labels
(Algorithm 1 (18-22)). In particular, we adopt a K-means module to aggregate
the data $D^{i}(c)$ of each label ($c\in C^{i}$) to clusters. Then we randomly
select samples according to the data density to keep representation
distribution, which can be formulated as:

$\mathcal{M}^{c}=Sample(Kmeans(D^{i}(c)),c,\frac{m}{|C^{i}|}).$ (3)

The selected samples for task $T^{i}$ are the union of selected data for each
label $c$ that $\mathcal{M}^{i}=\cup_{c\in C^{i}}\mathcal{M}^{c}$.
$\mathcal{M}^{i}$ is saved in the memory buffer and serves as the
classification criteria for task $T^{i}$ in the continual learning process.

###  3.3 Instance-wise Relation Distillation (IRD)

To preserve the knowledge learned for previous tasks, inspired by Fang et al.
(2020) and Cha et al. (2021), we use an instance-wise relation distillation
term to control representation drift (Algorithm 1 (7-9)). During the learning
on task $T^{i},i>1$, the normalized instance-wise similarity in the mini-batch
$A$ is calculated as:

$s_{j,p}^{i}=\frac{exp({h}^{i}_{j}\cdot{h}^{i}_{p}/\tau)}{\sum_{a\in
N(j)}exp({h}^{i}_{j}\cdot{h}^{i}_{a}/\tau)},$ (4)

where $\mathcal{N}(j)\equiv A/\\{j\\}$, the representations are encoded by the
model $LM^{i}$ and $\tau$ is the hyper-parameter temperature. Then the IRD
regularization term follows:

$\mathcal{L}_{IRD}=\frac{1}{|A|^{2}}\sum_{j}\sum_{p}s_{j,p}^{i-1}\ log\
s_{j,p}^{i}.$ (5)

The IRD regularization term aims to estimate the discrepancy of current
representations to those learned in the previous model, and mitigate the
representation drift through optimization. In this way, the knowledge of
previous models is preserved and the CF problem can be mitigated.

The overall training objective can be denoted as follows:

$\mathcal{L}=\mathcal{L}_{cl}+\mathcal{L}_{IRD}.$ (6)

Algorithm 1 SCCL Training

0: A set of training task $\\{T^{i}\\}^{n}$, the corresponding data set
$\\{D^{i}\\}^{n}$, sets of disjoint classes $\\{C^{i}\\}^{n}$. Training steps
$S$ and memory replay frequency $f$. Memory buffer size $m$. Initial pre-
trained language model $LM^{0}$.

0: Trained language model encoder $LM^{n}$ and memory buffer $\mathcal{M}$.

1: Load pre-trained language model $LM^{0}$;

2: $\mathcal{M}=[]$

3: for $i=1,...,n$ do

4: for $t=1,...,S$ do

5: Draw mini-batch $A$ from $D^{i}$;

6: Calculate $\mathcal{L}_{cl}$ of $A$ with $LM^{i}$ (Eq (1-2));

7: if $i>1$ then

8: Calculate $\mathcal{L}_{IRD}$ of $A$ (Eq (5));

9: $\mathcal{L}=\mathcal{L}_{cl}+\mathcal{L}_{IRD}$;

10: else

11: $\mathcal{L}=\mathcal{L}_{cl}$;

12: end if

13: Update model parameters with $\mathcal{L}$;

14: if $i\ \%\ f==0$ then

15: Update model parameters with memory relay;

16: end if

17: end for

18: for $c\in C^{i}$ do

19: Obtain k-means clusters of data with label $c$;

20: $\mathcal{M}^{c}=Sample(Kmeans(D^{i}(c)),c,\frac{m}{|C^{i}|})$;

21: $\mathcal{M}^{i}=\mathcal{M}^{i}\cup\mathcal{M}^{c}$;

22: end for

23: $\mathcal{M}$ = $\mathcal{M}+\mathcal{M}^{i}$;

24: end for

###  3.4 Memory Replay (MR)

To make full use of the memory buffer saved during training, we use a memory
replay module de Masson D’Autume et al. (2019) to further recover the
knowledge learned in the previous tasks (Algorithm 1 (14-16)). In the training
on the task $T^{i},i>1$, we revisit the samples in the memory buffer and train
the model with the same loss in Eq (2) after training every $f$ step on the
current task.

###  3.5 Inference

After learning the task $T^{i}$, we can obtain the model $LM^{i}$. During the
inference for previous tasks $T^{u},\ u<=i$, we feed each test data
$x^{u}_{j}$ into $LM^{i}$ and obtain the corresponding representation
${h}^{u}_{j}$. Then we retrieve the $k$ buffered data from $\mathcal{M}^{u}$
whose cosine similarity with ${h}^{u}_{j}$ is the largest. Note that the
representations of buffered data are obtained using the current model, which
can adapt to the representation drift for parameter update. We denote the $k$
nearest neighbors as ${({h}^{u}_{k},y^{u}_{k})\in\mathcal{K}^{u}_{j}}$. The
retrieved set is converted to a probability distribution over the labels by
applying a softmax with temperature $T$ to the similarity. Using the
temperature $T>1$ can flatten the distribution, and prevent over-fitting to
the most similar searches Khandelwal et al. (2020). The probability
distribution on the labels is expressed as follows:

$p_{k}(y_{j})\propto\sum_{{(h^{u}_{k},y^{u}_{k})\in\mathcal{K}^{u}_{j}}}\mathds{1}_{y_{j}=y^{u}_{k}}\cdot
exp(\frac{{h}_{j}^{u}\cdot{h}_{k}^{u}}{T}),$ (7)

and the label with the largest probability is taken as the prediction result.

##  4 Experimental Setting

###  4.1 Tasks

We adopt classification tasks from the benchmark GLUE Wang et al. (2018) and
those from MBPA++ Huang et al. (2021); de Masson D’Autume et al. (2019). We
select dissimilar tasks to form the task sequences, i.e. there are no overlap
labels between each task. The tasks contain 1) CoLA Warstadt et al. (2019),
requiring the model to determine whether a sentence is linguistically
acceptable; 2) MNLI Williams et al. (2017) containing 433k sentence pairs
annotated with textual entailment information; 3)
QNLI111https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs,
requiring deciding whether the answer answers the question; 4) QQP, (parsed
from SQuAD Rajpurkar et al. (2016)), testing whether a pair of Quora questions
are synonymous; 5) Yelp Zhang et al. (2015), requiring detecting the sentiment
of a sentence; 6) AG Zhang et al. (2015), requiring to classify the topics of
the news.

The sequences can be divided into 2 types with respect to the task lengths: 1)
a sequence of 4 classification tasks containing AG, Yelp, QNLI, and MRPC; 2) a
sequence of 6 classification tasks containing AG, MRPC, MNLI, CoLA, Yelp, and
QNLI. Without losing generality the orders are randomly selected and the task
orders for experiments are shown in Table 1.

Orders
---
1 | AG $\rightarrow$ Yelp $\rightarrow$ QNLI$\rightarrow$ MRPC
2 | MRPC $\rightarrow$ QNLI $\rightarrow$ Yelp $\rightarrow$AG
3 | QNLI $\rightarrow$Yelp $\rightarrow$MRPC$\rightarrow$AG
4 | AG$\rightarrow$MRPC $\rightarrow$CoLA$\rightarrow$MNLI $\rightarrow$Yelp$\rightarrow$ QNLI
5 | QNLI $\rightarrow$Yelp $\rightarrow$MNLI$\rightarrow$CoLA $\rightarrow$MRPC$\rightarrow$ AG
6 | MNLI $\rightarrow$ AG $\rightarrow$QNLI$\rightarrow$ MRPC $\rightarrow$Yelp$\rightarrow$ CoLA
Table 1: Different task orders for our experiments. Model | Order 1 | Order 2 | Order 3 | Order 4 | Order 5 | Order 6
---|---|---|---|---|---|---
| ACC | BWT | ACC | BWT | ACC | BWT | ACC | BWT | ACC | BWT | ACC | BWT
Joint | 83.09 | - | 83.09 | - | 83.09 | - | 83.06 | - | 83.06 | - | 83.06 | -
1-13[2pt/4pt] CE | 54.74 | -36.96 | 59.43 | -31.11 | 51.73 | -41.22 | 56.05 | -29.71 | 49.70 | -32.32 | 48.64 | -39.96
CL | 68.10 | -14.74 | 59.95 | -23.13 | 60.13 | -22.72 | 62.35 | -24.28 | 58.65 | -27.26 | 63.36 | -18.7
1-13[2pt/4pt] CE-MAS | 59.91 | -23.53 | 61.21 | -24.45 | 61.75 | -19.25 | 62.52 | -19.25 | 54.77 | -17.26 | 58.30 | -32.58
CL-MAS | 68.99 | -1.37 | 71.83 | -3.55 | 73.61 | -6.94 | 69.95 | -2.89 | 69.27 | -6.27 | 68.93 | -2.44
CE-EWC | 66.11 | -22.83 | 71.44 | -14.56 | 69.66 | -16.47 | 62.95 | -22.42 | 59.22 | -25.79 | 61.87 | -22.81
CL-EWC | 73.19 | -0.59 | 75.89 | -2.04 | 73.52 | -5.97 | 66.74 | -3.65 | 68.83 | -6.27 | 68.56 | -4.00
CE-LwF | 72.09 | -13.26 | 72.13 | -14.39 | 73.54 | -12.36 | 68.23 | -13.84 | 63.15 | -22.72 | 67.92 | -17.38
CL-LwF | 76.53 | -0.33 | 79.15 | -3.71 | 79.58 | -3.23 | 68.24 | -5.34 | 72.39 | -9.83 | 71.48 | -5.97
1-13[2pt/4pt] CE-ER | 76.83 | -9.13 | 76.60 | -10.39 | 76.90 | -12.59 | 75.08 | -10.05 | 76.51 | -6.39 | 76.15 | -14.61
IDBR | 75.70 | -3.16 | 73.62 | -7.43 | 75.11 | -3.65 | 65.40 | -10.56 | 69.94 | -5.96 | 66.30 | -10.86
Co2L | 70.58 | -2.07 | 74.02 | -7.52 | 74.10 | -7.68 | 64.31 | -3.57 | 65.04 | -10.62 | 64.94 | -15.65
1-13[2pt/4pt] SCCL | 79.20 | -2.93 | 80.05 | -3.07 | 80.24 | -3.51 | 78.36 | 0.57 | 79.00 | -3.39 | 78.55 | -3.75
w/o MR | 77.19 | -5.34 | 78.63 | -4.59 | 80.27 | -2.91 | 75.65 | -2.91 | 71.22 | -11.62 | 74.64 | -7.30
w/o IRD | 77.57 | -5.33 | 79.73 | -2.48 | 79.48 | -3.33 | 73.87 | -6.62 | 76.89 | -4.10 | 74.32 | -4.03
Table 2: Continual Learning results on 6 different tasks. ‘CE’ refers to the
standard cross-entropy-based methods, and ‘CL’ refers to extended contrastive-
learning-based methods with continual learning strategies. ‘-’ for not
acquirable. All the results are averaged on 5 different random seeds.

###  4.2 Evaluation Metrics

We adopt the metrics of average accuracy (ACC) and backward transfer (BWT) to
evaluate the performance of the continual learning model Lopez-Paz and Ranzato
(2017). The model trained after the task $T^{i}$ is evaluated on the test set
of earlier tasks $T^{j}$ ($j<=i$), and the test accuracy is denoted as
$R^{i}_{j}$. The metrics are shown as follows:

$ACC=\frac{1}{n}\sum_{i=1}^{n}R^{n}_{i}$ (8)

$BWT=\frac{1}{n-1}\sum_{i=1}^{n-1}R^{n}_{i}-R^{i}_{i},$ (9)

where the former evaluates the overall performance of the final trained model,
and the latter calculates the knowledge forgetting during the continual
training procedure.

###  4.3 Baselines

We not only compare our model with several CE-based continual learning methods
but extend training strategies of them to our contrastive learning framework
(i.e. training with contrastive learning and inferring with kNN) to verify the
effectiveness of contrastive learning in mitigating CF. We also compare our
model with the competitive models IRDB Huang et al. (2021) and Co2L Cha et al.
(2021). The shared hyper-parameters are kept the same as SCCL in baselines.
The model details are as follows:

  * •

Fine-tune (CE, CL) Yogatama et al. (2019) modifies the parameters of the pre-
trained language model to adapt to a new task without any augmented strategies
and additional loss.

  * •

Experience Replay (ER) Riemer et al. (2019) stores a small subset of samples
from previous tasks and replays those to prevent models from forgetting past
knowledge.

  * •

Elastic Weight Consolidation (EWC) Kirkpatrick et al. (2017) slows down the
updates of the optimal parameters for previous tasks by extending the loss
function with a regularization term.

  * •

Memory Aware Synapses (MAS) Aljundi et al. (2018) slows down the update
according to the importance weight of each parameter in the network, i.e. the
sensitivity of the output function to a parameter change.

  * •

Learning Without Forgetting (LwF) Li and Hoiem (2017); Yu et al. (2020a) aims
to keep the model output of current data close to those of the previous model.

  * •

IDBR Huang et al. (2021) uses information disentanglement regularization to
encode task-specific information and general information individually, which
are jointly considered for classification.

  * •

Co2L Cha et al. (2021) uses an asymmetric supervised contrastive learning
method to learn representations and trains a decoupled layer for inference.

  * •

Multi-task Training (Joint) Yu et al. (2020b) trains on all the tasks
simultaneously, i.e. the data of different tasks are mixed up for training. It
does not suffer from catastrophic forgetting and represents an upper bound on
model performance.

###  4.4 Implementation Details

We adopt the officially released roberta-base from HuggingFace
222https://huggingface.co/ as our backbone network. We train our model on 1
GPU (A100 80G) using the Adam optimizer Kingma and Ba (2014). For all the
models, the batch size is 96, the learning rate is 3e-5, and the scheduler is
set linear. We train our model 10 epochs for each task. Following Huang et al.
(2021), we select 4,000 samples for each label in training. The hyper-
parameters of temperatures $\kappa$ is 0.2, $\tau^{*}$ is 0.2, and $T$ is 5,
and the number of nearest neighbors $k$ is 10. The memory size for each task
is set to 200 (2.5% of the training data) and the memory replay frequency $f$
is 100. Through the training of our model, no development set is applied to
find the best checkpoints, but stop until the training step is reached.

##  5 Results

###  5.1 Overall Results

The overall results of our experiments are shown in Table 2. First, our model
SCCL achieves ACCs of 79.20%, 80.05%, 80.24%, 78.36%, 79.00%, and 78.55% in
Order 1-6, respectively, which are 2.37%, 0.9%, 0.66%, 3.28%, 2.49%, and 2.40%
higher than the second-best performance of the continual learning baselines.
It shows that the performance of the continually learned model is well-
maintained in SCCL, but the problem of CF still exists. SCCL achieves state-
of-the-art ACCs compared with the baseline models, indicating the
effectiveness of our proposed framework. We also observe that the performance
variance is small in the SCCL model for different orders, which implies that
our models are not sensitive to the order of task sequences.

Second, the results of BWT range from -3.75% to 0.57% in SCCL for Orders 1-6,
which demonstrates knowledge forgetting during the continual learning
procedure. The results of SCCL are relatively higher than CE-based models,
indicating that SCCL suffers from a milder impact of CF. Note that the BWT of
SCCL is 0.57% in Order 4, which indicates that SCCL can even backward transfer
the knowledge from the current tasks to previous tasks. But compared with CL-
LwF, CL-MAS, and CL-EWC, the values of ACCs in SCCL are higher, but BWTs are
adverse. It implies that using the regularization-based strategies, the fine-
tuning performance is destructed for explicit control of representations. In
this way, BWTs become low since the fine-tuning performance on downstream
tasks is relatively weak.

Figure 3: Detailed results during continual learning procedure for different
strategies in Order 3. Figure 4: Comparisons of SCCL and Co2L with ablation
studies.

Third, the extended CL-based models achieve stronger performance than
corresponding standard CE-based models. For example, the model CL-LwF achieves
ACCs of 76.53%, 79.15%, 79.58%, 68.24%, 72.39%, and 71.48%, which are 4.44%,
7.02%, 6.04%, 0.01%, 9.24% and 3.56% higher than those of CE-LwF. The results
of CL, CL-MAS, and CL-EWC are in a similar pattern. The results reflect that
contrastive learning with a kNN classifier for continual learning has a
stronger ability to overcome CF. But we observe that Co2L achieves relatively
low performance compared with our model, which proves that Co2L is not
effective for task-incremental learning. It can be explained that Co2L keeps
the knowledge of classes and separate the tasks with clear boundary, by using
asymmetric supervised contrastive loss, which makes it difficult to
distinguish a representation for different task purposes.

Finally, we observe a significant variance in the results of different task
orders for regularization-based methods. For example, ACCs of CL-EWC range
from 75.89% to 66.74%. But in CE-ER or SCCL the variance is less drastic, such
as CE-ER ranging from 76.90% to 75.08% and SCCL ranging from 80.24% to 78.55%.
The phenomenon may result from that knowledge forgetting of previous tasks
increases step by step for information los, but no samples help recover such
information in regularization-based methods.

###  5.2 Ablation Study

We show the ablation study of memory replay and IRD in the last two rows in
Table 2. The ACCs of the models w/o memory replay range from 71.22%, to 80.27%
for Order 1-6, which are 2.01%, 1.42%, -0.03%, 2.71%, 7.78%, and 3.91% lower
than SCCL, respectively. It shows the effectiveness of memory replay, without
which ACC also becomes less robust to task orders. Then ACCs of the models w/o
IRD are 1.63%, 0.32%, 0.76%, 4.49%, 2.11%, and 4.23% lower than SCCL for Order
1-6, respectively. We observe that the models w/o IRD are more robust to task
orders, which implies that rehearsal-based methods are less sensitive to task
sequences. Comparing the model w/o IRD with CE-ER, the model performance are
also higher than those of CE-ER, which uses almost the same training strategy.
The phenomenon demonstrates the effectiveness of contrastive learning in
overcoming CF.

We also compare our model with Co2L in ablation studies (Figure 4). First, we
replace the kNN module of SCCL with a decoupled linear classifier like Cha et
al. (2021) (SCCL-CLS), where ACCs are slightly smaller than SCCL. It indicates
that the kNN module in SCCL can achieve satisfactory performance without
additional training on the final representations of contrastive learning. Then
we replace the decoupled linear classifier of Co2L with our kNN module
(Co2L-kNN), and we observe an increase in performance. It implies that the
representations learned by Co2L are not separated clearly in the feature
space, thus a trained linear layer is less effective for classification. But
K-means selection of the samples and kNN inference module can estimate the
representation distribution more precisely, resulting in better performance.
Note that the results of SCCL are also stronger than Co2L-kNN, which indicates
the effectiveness of our model on task-incremental continual learning.

Figure 5: t-SNE visualization of the representations of QNLI samples learned
based on the different continual learning methods in Order 5. ‘E’ refers to
the representations at the end of continual learning.

###  5.3 Detailed Results

As an example, we show the detailed results of Order 3 in several models
(Figure 3). First, in the model of CE, we observe that the test accuracies of
QNLI decrease from 87.78% to 50.21% step by step with the continual training
on the task QNLI, Yelp, MRPC, and AG. The accuracies of Yelp and AG are also
in a similar pattern, where the final performance is nearly random. It
indicates that using standard CE for continual learning suffers from CF
significantly. But in the method CL, the final performance of QNLI, Yelp, and
MRPC is still stronger than a random prediction, indicating that contrastive
learning with the kNN module can maintain learned knowledge in each training
step and results in satisfactory performance at the end.

The model CE-ER can also mitigate CF compared with CE and CL, but the
performance still decreases a large margin in the task of QNLI, Yelp, and
MRPC. The accuracy of QNLI decreases by 17.38%, that of Yelp decreases by
11.59%, and that of MRPC decreases by 14.84%. As for our model SCCL, we
observe that the test performance is 88.70%, 87.24%, 86.24% and 85.68% after
training on tasks QNLI, Yelp, MRPC, and AG, respectively. It shows that the
performance of SCCL decreases as the training precedes, but within a small
range (3.02%). The results on Yelp and MRPC are in a similar pattern. It
demonstrates that our model has a strong ability to overcome CF.

###  5.4 Visualization

We use t-SNE to visualize the representations of QNLI in Order 3 of the
training models, CE, CE-ER, CL, and SCCL (Figure 5). As we observe in CE the
representations of the test data are clearly separated into two clusters after
training on the task QNLI. When finishing the continual learning, the
representations become nearly uniformly distributed on the feature space and
the model only achieves an accuracy of 50.21%. It demonstrates that
catastrophic forgetting is significant due to representation drift. In the
model CL, the representations drift severely as well, but the distribution is
less uniform compared with CE. Typically, we can clearly at the upper right of
the distribution, there are more memory samples with label 1, and the test
samples with label 1 also gather in the position, indicating correct
classification based on kNN. The test performance achieves 61.74%, but is
still 26.94% lower than the initial model. The phenomenon shows
representations during continual learning drift less significantly and the
saved samples (the classification criterion) also drift, which maintains some
correct inferences. But CF is still a salient problem in contrastive learning.

But in CE-ER, the boundary of the representations becomes indistinct, and the
accuracy of QNLI decreases from 87.79% to 70.11% after continual learning. It
indicates that the representations are less effective compared with the
initially trained, i.e. CF is significant in CE-ER. But the representations in
SCCL are still clearly divided into two parts according to the labels. The
representations of the memory samples are among the according clusters,
implying the performance on the task QNLI is well-maintained. Correspondingly,
the accuracy at the end of learning is 85.68% based on SCCL, only 3.02% lower
than the initial performance. It shows that in SCCL the representation drift
slightly and the classification criterion is well-maintained, resulting in a
satisfactory performance.

##  6 Conclusion

In this paper, we proposed a supervised contrastive learning model for task-
incremental continual learning (SCCL) to boost the extensibility of continual
learning. The model used contrastive learning to learn representations and a
kNN module was adopted for inference, together with an instance-wise
distillation and a memory replay module to maintain previously learned
knowledge. With extensive experiments, our model achieved state-of-the-art
performance compared with standard CE-based methods. Ablation studies and
visualizations also proved the effectiveness of our model in solving the
problem of CF.

##  7 Limitations

Our model SCCL is specific for task-incremental continual learning scenarios,
but not suitable for class-incremental scenarios. In class-incremental
scenarios, the representations of current classes should be designed to be far
away from previous ones. For simplicity, we do not consider data augmentation
in our model, so the batch size should be large enough to contain positive
pairs for each label. But data augmentation (such as two different dropout
representations Gao et al. (2021)) is a plug-and-play module for our model if
there are plenty of labels in each task.

## References

  * Aljundi et al. (2018) Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. 2018.  Memory aware synapses: Learning what (not) to forget.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 139–154.
  * Bang et al. (2021) Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Ha, and Jonghyun Choi. 2021.  Rainbow memory: Continual learning with a memory of diverse samples.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8218–8227.
  * Cha et al. (2021) Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. 2021.  Co2l: Contrastive continual learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9516–9525.
  * Chen et al. (2020a) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020a.  A simple framework for contrastive learning of visual representations.
  * Chen et al. (2020b) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020b.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR.
  * De Lange et al. (2022) Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2022.  A continual learning survey: Defying forgetting in classification tasks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(7):3366–3385.
  * de Masson D’Autume et al. (2019) Cyprien de Masson D’Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019\.  Episodic memory in lifelong language learning.  _Advances in Neural Information Processing Systems_ , 32.
  * Fang et al. (2020) Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. 2020.  Seed: Self-supervised distillation for visual representation.  In _International Conference on Learning Representations_.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Huang et al. (2021) Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. 2021.  Continual learning for text classification with information disentanglement based regularization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2736–2746.
  * Kassner and Schütze (2020) Nora Kassner and Hinrich Schütze. 2020.  BERT-kNN: Adding a kNN search component to pretrained language models for better QA.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3424–3430, Online. Association for Computational Linguistics.
  * Ke and Liu (2022) Zixuan Ke and Bing Liu. 2022.  Continual learning of natural language processing tasks: A survey.  _arXiv preprint arXiv:2211.12701_.
  * Khandelwal et al. (2020) Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020\.  Nearest neighbor machine translation.  _arXiv preprint arXiv:2010.00710_.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020.  Supervised contrastive learning.  _Advances in Neural Information Processing Systems_ , 33:18661–18673.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526.
  * Li and Hoiem (2017) Zhizhong Li and Derek Hoiem. 2017.  Learning without forgetting.  _IEEE transactions on pattern analysis and machine intelligence_ , 40(12):2935–2947.
  * Lopez-Paz and Ranzato (2017) David Lopez-Paz and Marc’Aurelio Ranzato. 2017.  Gradient episodic memory for continual learning.  _Advances in neural information processing systems_ , 30.
  * Luo et al. (2022) Yun Luo, Fang Guo, Zihan Liu, and Yue Zhang. 2022.  Mere contrastive learning for cross-domain sentiment analysis.  _arXiv preprint arXiv:2208.08678_.
  * Neelakantan et al. (2022) Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. 2022\.  Text and code embeddings by contrastive pre-training.  _arXiv preprint arXiv:2201.10005_.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Rebuffi et al. (2017) Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. 2017.  icarl: Incremental classifier and representation learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Riemer et al. (2019) Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. 2019.  Learning to learn without forgetting by maximizing transfer and minimizing interference.  In _International Conference on Learning Representations_. International Conference on Learning Representations, ICLR.
  * Rolnick et al. (2019) David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. 2019.  Experience replay for continual learning.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Serra et al. (2018) Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. 2018.  Overcoming catastrophic forgetting with hard attention to the task.  In _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 4548–4557. PMLR.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355.
  * Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641.
  * Williams et al. (2017) Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017.  A broad-coverage challenge corpus for sentence understanding through inference.  _arXiv preprint arXiv:1704.05426_.
  * Wu et al. (2018) Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. 2018.  Unsupervised feature learning via non-parametric instance discrimination.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3733–3742.
  * Yogatama et al. (2019) Dani Yogatama, Cyprien de Masson d’Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. 2019.  Learning and evaluating general linguistic intelligence.  _arXiv preprint arXiv:1901.11373_.
  * Yoon et al. (2018) Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. 2018.  Lifelong learning with dynamically expandable networks.  In _6th International Conference on Learning Representations, ICLR 2018_. International Conference on Learning Representations, ICLR.
  * Yu et al. (2020a) Lu Yu, Bartłomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer. 2020a.  Semantic drift compensation for class-incremental learning.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6980–6989.
  * Yu et al. (2020b) Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020b.  Gradient surgery for multi-task learning.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 5824–5836. Curran Associates, Inc.
  * Zhang et al. (2021) Dejiao Zhang, Shang-Wen Li, Wei Xiao, Henghui Zhu, Ramesh Nallapati, Andrew O. Arnold, and Bing Xiang. 2021.  Pairwise supervised contrastive learning of sentence representations.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5786–5798, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  _Advances in neural information processing systems_ , 28.
  * Zhao et al. (2022) Kang Zhao, Hua Xu, Jiangong Yang, and Kai Gao. 2022.  Consistent representation learning for continual relation extraction.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3402–3411, Dublin, Ireland. Association for Computational Linguistics.

##  Appendix A Data Statistics

We show the data statistics in Table 3.

Task | Type | #Train | #Test | #Labels
---|---|---|---|---
AG | News | 16000 | 7600 | 4
QNLI | Q & A | 8000 | 5266 | 2
Yelp | Sentiment | 20000 | 7600 | 5
CoLA | Linguistics | 6527 | 1042 | 2
MNLI | Inference | 12000 | 9815 | 3
MRPC | Paraphrase | 4074 | 1725 | 2
Table 3: Statistics for different classification tasks.

##  Appendix B kNN Sensitivity

We show the sensitivity of SCCL to the number of $k$ in the kNN module (Figure
6). We find that the performance of our model fluctuates from 80.24% to 80.28%
(a significantly small range) in our method, indicating the representations in
our model cluster well in the feature space and are robust to the hyper-
parameter $k$. But the performance in CL fluctuates more severely, ranging
from 59.02% to 60.12%. The best performance is achieved when $k=10$ and
decreases with the increase of $k$, which means the representations drift
significantly and the clusters become less reliable. The experiment
demonstrates the effectiveness of IRD regularization term and the memory
replay module in maintaining the representation distribution, and without them
the representations drift significantly, suffering from the CF problem.

Figure 6: Test results with respect to different numbers of $k$ for Order 3.