# Enhancing Visual Continual Learning with Language-Guided Supervision

Bolin Ni${}^{1,2^{\star}}$, Hongbo Zhao${}^{1,2^{\star}}$, Chenghao Zhang1,2,
Ke Hu2
Gaofeng Meng1,2,3†, Zhaoxiang Zhang1,2,3, Shiming Xiang1,2
1State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute
of Automation, Chinese Academy of Sciences.
2School of Artificial Intelligence, University of Chinese Academy of Sciences.
3Centre for Artificial Intelligence and Robotics, HK Institute of Science &
Innovation, Chinese Academy of Sciences.
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Continual learning (CL) aims to empower models to learn new tasks without
forgetting previously acquired knowledge. Most prior works concentrate on the
techniques of architectures, replay data, regularization, _etc_. However, the
category name of each class is largely neglected. Existing methods commonly
utilize the one-hot labels and randomly initialize the classifier head. We
argue that the scarce semantic information conveyed by the one-hot labels
hampers the effective knowledge transfer across tasks. In this paper, we
revisit the role of the classifier head within the CL paradigm and replace the
classifier with semantic knowledge from pretrained language models (PLMs).
Specifically, we use PLMs to generate semantic targets for each class, which
are frozen and serve as supervision signals during training. Such targets
fully consider the semantic correlation between all classes across tasks.
Empirical studies show that our approach mitigates forgetting by alleviating
representation drifting and facilitating knowledge transfer across tasks. The
proposed method is simple to implement and can seamlessly be plugged into
existing methods with negligible adjustments. Extensive experiments based on
eleven mainstream baselines demonstrate the effectiveness and generalizability
of our approach to various protocols. For example, under the class-incremental
learning setting on ImageNet-100, our method significantly improves the Top-1
accuracy by 3.2% to 6.1% while reducing the forgetting rate by 2.6% to 13.1%.

Figure 1: We introduce LingoCL, a simple yet effective continual learning
paradigm leveraging language-guided supervision, which can be integrated into
most existing approaches seamlessly. (a) Overview of the typical methods which
are supervised only by one-hot labels. (b) Overview of the proposed LingoCL
which is supervised by semantic targets generated from the pretrained language
model. (c) LingoCL is versatile, which significantly enhances the performance
of mainstream methods in class-, task- and domain-incremental scenarios.

††footnotetext: ⋆Equal contribution. †Corresponding author.

##  1 Introduction

The main challenge in continual learning (CL) is catastrophic forgetting,
where models experience significant performance degradation on earlier tasks
when new tasks are introduced. To address this, researchers have developed
various strategies, including architecture-based [28, 43, 53, 29], replay-
based [3, 38, 44], distillation-based [41, 13, 17], and regularization-based
methods [24, 7, 56], alongside other notable contributions [51, 22].

However, most existing approaches overlook the significance of the semantic
knowledge contained in category names. The prevailing trend in prior work
leans towards using one-hot labels, coupled with the randomly initialized
classifier head, and optimizing the encoder and classifier head jointly. Such
a methodology is de facto paradigm for stationary environments. Nevertheless,
in the CL scenarios, this practice presents two issues. Firstly, the problem
of representation drifting emerges. When the model encounters new tasks, the
feature space could drift or even be overwritten, compromising the stability
of models. This drift arises because the optimization of the semantic
target††Each row of the classifier’s weights represents the semantic target
for its corresponding class. of each class is narrowly focused on its current
task. Due to the limited access to old data and the unpredictability of new
data, the model struggles to be compatible with the previous and future
classes. For example, as shown in Fig. 1(a), the potential future class
“chimpanzee” may erase the feature space of the learned class “plane”,
exacerbating the forgetting of old tasks. Secondly, this particularity of data
in CL also results in inefficient knowledge transfer. Since the semantic
targets in the classifier are randomly initialized without any prior
knowledge, and are then optimized within individual tasks, it struggles to
capture the semantic correlation across all tasks. This incompleteness in
semantic correlations impedes the model’s knowledge transfer, thereby
affecting its plasticity.

In this work, we study how to enhance CL performance by leveraging the
semantic knowledge in category names from a classifier perspective. Inspired
by the impressive generalization capabilities of pretrained language models
(PLMs) [39, 5], we propose a simple yet effective approach, language-guided
supervision for CL (LingoCL), which employs PLM to generate the semantic
targets. Specifically, for the incoming task, we first use the category name
of each class as input to the language model and take the outputs as the
weights in the classifier. Then, the classifier is kept frozen during CL
training, guiding the learning of the encoder. Our approach is motivated by
the rich knowledge and strong generalization abilities of PLMs. Even with the
limitations in previous and future data, PLMs ensure that each generated
semantic target implicitly considers the semantic correlations between all
classes. Therefore, these targets can be used to direct the learning of the
encoder. For instance, as illustrated in Fig. 1(b), PLMs can provide the prior
knowledge that the “eagle” in current tasks shares a similar semantic target
with the learned “parrot” class, facilitating the knowledge transfer between
the learned classes and new classes. We explore two types of language models
in this work: self-supervised models on unimodal data and vision-supervised
models on multimodal data. Our results demonstrate that both types of models
can serve as excellent classifier heads, constantly improving performance.
Moreover, the analysis in Sec. 3.3 demonstrates that the improvements come
from alleviating the representation drift and facilitating knowledge transfer,
instead of the individual gains at each task.

Without loss of generality, we choose eleven methods as baselines and
incorporate the text-supervised classifiers for them. Comprehensive
experiments demonstrate the proposed methods are generally effective. In
particular, under the class-incremental learning setting, LingoCL can improve
the accuracy on ImageNet-100 by $3.2\%$ to $6.1\%$, and reduce the forgetting
rate by $2.6\%$ to $13.1\%$. In task- and domain-incremental learning, LingoCL
improves the accuracy by $3.9\%$ to $9.7\%$ and $1.2\%$ to $4.0\%$,
respectively.

The contributions can be summarized as follows:

  * •

We point out that the semantic knowledge in category names is largely
neglected by existing methods when initializing classifiers, which could have
two issues, _i.e_., representation drifting and insufficient knowledge
transfer.

  * •

We propose LingoCL, a new CL paradigm with language-guided supervision. With
the rich semantic knowledge in PLMs, we alleviate the abovementioned issues
and thus enhance the performance of mainstream CL methods.

  * •

The proposed LingoCL has several key advantages: 1) computation efficiency; 2)
orthogonality to existing methods; 3) flexibility with various PLMs; and 4)
versatility in diverse CL scenarios. Extensive experiments are conducted to
systematically examine our method.

##  2 Related Work

Continual learning. To alleviate catastrophic forgetting, researchers have
explored various routes. Regulation-based methods [24, 56, 2, 7] aim to
prevent catastrophic forgetting by penalizing the changes of network
parameters when learning current tasks. Replay-based methods entail selecting
a subset of data from previous tasks [3, 38, 58] or using generative models to
produce synthetic data [19, 21, 37] as “replayed” data to preserve the
knowledge of previous tasks. Distillation-based methods take the model trained
on the previous task as the teacher to supervise the learning of the current
model. These methods can be divided into logits distillation [41, 52], feature
distillation [13, 17], and relational distillation [45, 46]. Architecture-
based methods involve dynamic allocation of different parameters for each task
through architecture expansion [28, 43, 14, 53] or mask operation [31, 32].
Rectification-based methods analyzes the abnormal behaviors in CL models
compared to oracle models and tries to rectify them. These methods usually
focus on the imbalance in the feature embedding [4, 29, 44] or network weights
[52, 6, 57].

Most existing methods commonly use one-hot labels coupled with randomly
initialized classifiers, ignoring the category names seriously. In contrast to
them, our work studies whether and how to improve CL by leveraging the
semantic information contained in the category names.

Cross-modality adaptation. In recent years, transferring language knowledge to
visual modeling has emerged as a new paradigm. For example, contrastive
language-image pretraining demonstrates impressive “zero-shot” transfer and
generalization capacities [39, 20, 55]. Moreover, some works explore how to
model the vision input using pretrained language models in order to transfer
the ability of language models [1, 47, 26]. Another line of work focuses on
how to improve the vision encoder with the guidance of the language
information [35]. For instance, Tex [49] proposes to use language models to
reduce the bias in the classifier of fine-tuned visual models. DUET [8]
integrates the latent semantic knowledge from PLMs to vision models for better
zero-shot recognition ability. Additionally, Lei et al. [27] designed a suite
of evaluation tasks across various perception aspects and showed that language
models can learn visual features from vast amounts of data, including shape,
texture, and color, and that vision supervision can enhance the comprehension
of visual concepts. In this work, we are pioneering the exploration of how to
transfer knowledge in language models to address the catastrophic forgetting
issue in continual learning.

##  3 Methodology

###  3.1 Revisiting Classifier in Existing CL Paradigms

We first review the typical CL paradigms from a classifier perspective. In CL
scenarios, the vision encoder, denoted by $g_{V}$, is sequentially optimized
over tasks. Each task typically requires an individual classifier head. For
$t$-th task, we symbolize the training dataset as
$\mathcal{D}_{t}=\\{(\boldsymbol{x}_{t},y_{t})\\}$ that contains $C_{t}$
disjoint classes and the classifier head as
$\mathbf{W}_{t}\in\mathbb{R}^{C_{t}\times d}$. The vision encoder $g_{V}$ and
semantic targets in $\mathbf{W}_{t}$ are optimized jointly, following the
learning objective in the stationary environment:

${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}g^{*}_{V}},{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\mathbf{W}_{t}^{*}}=\underset{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\Theta_{V}},{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\mathbf{W}_{t}}}{\mathrm{argmin}}~{}{\mathbb{E}}_{{\boldsymbol{x}_{t},y_{t}\sim\mathcal{D}_{t}}}\big{[}\mathcal{L}(\mathrm{sim}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\mathbf{W}_{t}},{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}g_{V}}(\boldsymbol{x}_{t})),y_{t})\big{]},$
(1)

where $\mathbf{W}_{t}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{d})$. The function
$\mathrm{sim}(\mathbf{W},g_{V}(\boldsymbol{x}_{t}))$ calculates the similarity
between the image embedding $g_{\theta}(\boldsymbol{x})$, and every semantic
target in $\mathbf{W}_{t}$, using the inner product.

However, this approach encounters challenges specific to CL. Firstly, CL
benefits from knowledge transfer across tasks, such as forward and backward
transfer, but randomly initialized classifiers struggle to capture the
semantic similarity among classes across tasks, resulting in inefficient
knowledge transfer. Secondly, the optimization of each semantic target is
confined to its current task. This narrow focus overlooks the broader
compatibility between semantic targets spanning all tasks. Such a shortsighted
learning approach can cause conflicts between the semantic targets of
different tasks, inducing representation drifting or erasure in the feature
space.

###  3.2 Our Proposed Language-Guided Supervision

To address these issues, we utilize the rich semantic knowledge contained in
pretrained language models to guide the learning process for each task.
Specifically, for an incoming task $t$, the procedure is as follows:

  1. (i)

Gathering the category names
$[\boldsymbol{l}_{1},\cdots,\boldsymbol{l}_{C_{t}}]$ of task $t$.

  2. (ii)

Feeding these category names into PLM to generate the semantic targets for the
classifier $\tilde{\mathbf{W}}_{t}$:

${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\tilde{\mathbf{W}}_{t}}={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}g_{T}}([\boldsymbol{l}_{1},\cdots,\boldsymbol{l}_{C_{t}}]),$
(2)

  3. (iii)

Optimizing the vision encoder $g_{V}$, while keeping the classifier
$\tilde{\mathbf{W}}_{t}$ frozen:

${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}g^{*}_{V}}=\underset{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\Theta_{V}}}{\mathrm{argmin}}~{}{\mathbb{E}}_{{\boldsymbol{x}_{t},y_{t}\sim\mathcal{D}_{t}}}\big{[}\mathcal{L}(\mathrm{sim}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\tilde{\mathbf{W}}_{t}},{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}g_{V}}(\boldsymbol{x}_{t})),y_{t})\big{]}.$
(3)

The classifier $\tilde{\mathbf{W}}_{t}$ is kept frozen to preserve the
semantic knowledge from being disturbed or forgotten in the CL process. As
these generated semantic targets are optimized with sufficient data and
concepts, they effectively serve as supervision signals, directing the vision
encoder’s optimization.

In light of the outlined methodology, LingoCL has several key advantages: 1)
it is computation-efficient; leveraging category names requires only a single
forward propagation, with a negligible cost comparison to overall training; 2)
it provides flexibility to utilize knowledge from various language models,
promoting easy integration of the latest PLM advancements; 3) it is orthogonal
to most of existing CL methods, allowing for seamless integration; 4) it is
versatile, and compatible with diverse CL scenarios such as class-, task- and
domain-IL.

###  3.3 Quantitative Analysis

Next, we examine our method to answer the two questions mentioned above: 1)
Does our method alleviate representation drifting, and 2) Does it facilitate
knowledge transfer?

To answer the first question, we perform a subspace analysis [40] on
challenging class-incremental learning protocol. Given the same input, let
$\mathbf{F}_{t},\mathbf{F}_{t^{\prime}}\in\mathbb{R}^{n\times d}$ denote the
output of the encoder after the $t$-th task and after the $t^{\prime}$-th task
($t^{\prime}>t$), respectively. $\mathbf{V}_{k,t}$ and
$\mathbf{V}_{k,t^{\prime}}$ are the top-$k$ principal directions of
$\mathbf{F}_{t}$ and $\mathbf{F}_{t^{\prime}}$, respectively. The
representation drifting from the $t$-th task to the $t^{\prime}$-th can be
defined as:

$\mathrm{RepreDrift}_{k}(\mathbf{F}_{t},\mathbf{F}_{t^{\prime}})=1-\frac{1}{k}\|\mathbf{V}_{k,t}^{T}\mathbf{V}_{k,t^{\prime}}\|_{F}^{2},$
(4)

where a smaller value indicates less representation drifting. We adopt LUCIR
[17] as the baseline. Fig. LABEL:fig:drifting shows the evolution of the first
task’s representations as the training progresses. LingoCL significantly
reduces the representation drifting, demonstrating the capacity to enhance the
stability of the CL model.

For the second question, we sample the first 18 classes in ImageNet-100 and
calculate the inter-class correlation of the embeddings produced by the
encoder with vanilla classifier and LingoCL. Note that these classes are
scattered among different tasks. Fig. LABEL:fig:corr shows that LingoCL
exhibits certain inter-class correlations, indicating that a well-considered
semantic target can facilitate knowledge transfer and improve the performance
of CL. The above analyses offer an initial demonstration of the effectiveness
of LingoCL. A more in-depth exploration of these issues is presented in Tab.
LABEL:tab:component.

##  4 Experiments

###  4.1 Experimental Setup

Continual learning protocols. We evaluate LingoCL on four common CL protocols,
including: class-incremental learning (CIL), general few-shot class-
incremental learning, task-incremental learning, and domain incremental
learning.

Datasets. We use CIFAR100 [25] for task-IL, both CIFAR100 and ImageNet-100
[41] for class-IL, and OfficeHome [48] for domain-IL. More details about
datasets are shown in the _supplementary material._

Architecture. We employ MobileNetV2 for BiMeCo [36] and ResNet18 [16] for
other CNN-based methods. For ViT-based methods, such as DyTox [14], we follow
the original implementation and use ConViT [15]. As for the pretrained
language model, we utilize the text transformer in CLIP-B/32 [39] pretrained
on WIT-400M [39]. Results about more language models are explored in Tab.
LABEL:tab:language.

Metrics. Following [10, 34], LingoCL is extensively evaluated by three
metrics: last-step accuracy (Last), average incremental accuracy (Avg), and
forgetting rate ($\mathcal{F}$).

Baselines. We comprehensively evaluate the effectiveness of the proposed
method on eleven baselines, spanning various continual learning approaches.
These include distillation-based methods such as LUCIR [17] and BiC [52],
architecture-based methods like DyTox [14] and AANet [29], rehearsal-based
methods such as GEM [30], regularization-based methods including EWC [24], MAS
[2], and SI [56], and rectification-based methods like IL2M [4] and CwD [44].
To ensure a fair comparison, we implement all the baseline methods using their
officially released code or the widely recognized CL library [18, 33] in the
research community and keep their original hyperparameters unchanged.

###  4.2 Class-incremental Learning Experiments

Benchmark protocol. We denote the number of classes in the initial task by $B$
and the number of new classes learned per task after the initial one by $C$.
We adopt two popular protocols [41, 52]: 1) $B=50$: where the initial task
covers half of the total number of classes and the remaining classes are
equally divided among the subsequent tasks, and (2) $C=B$, where each task
within the data stream involves an equal number of classes. The memory size
for each class is set to 20 in all datasets. All approaches are evaluated
under the same class order [41, 29, 17] for fair comparison.

Implementation details. We follow the original hyperparameters of all methods.
We select exemplars as memory based on the herding strategy following previous
works [41]. Detailed hyperparameters are in the _supplementary material_.

Results. We conduct extensive experiments by incorporating our method into
various baselines of different routes. Tab. LABEL:tab:i100 and Tab.
LABEL:tab:cifar present the results on ImageNet-100 and CIFAR100,
respectively, which illustrate that our method consistently and significantly
improves all metrics. Taking LUCIR on ImageNet-100 as an example, our method
improves the average accuracy by $3.2\%\sim 6.1\%$ across different settings.
Importantly, our method barely impacts the performance of the oracle model,
implying that the performance gains stem from reducing forgetting instead of
the individual gains at each task. This is evidenced by the significant
reductions in forgetting rate by $2.6\%\sim 13.1\%$.

Moreover, we observe that the gains in last accuracy are usually larger than
that in average accuracy, indicating that our method benefits more on long
task sequences which are commonly more challenging. This observation is also
supported by the experimental results. For instance, when the number of task
increases from 6 to 11 (B=50,C=10 to B=50,C=5), the gains of CwD and LUCIR
increase from $1.4\%$ and $3.2\%$ to $2.6\%$ and $3.8\%$, respectively.

Finally, in addition to the quantitative results, Fig. LABEL:fig:lucir
displays the accuracy and forgetting rate curves for LUCIR in long sequence
settings. Our method achieves a smoother forgetting rate curve, with a gradual
and consistent improvement in accuracy at each task. More importantly, our
method even achieves negative forgetting rate, indicating that learning the
later tasks helps to improve the accuracy of the previous ones. We attribute
these gains to the fact that our method utilizes the semantic similarity among
classes to guide the CL process, which promotes backward knowledge transfer.

###  4.3 General Few-shot Class-IL Experiments

Benchmark protocol. General few-shot CIL is a more realistic setting where the
initial task has sufficient training data to initialize the model, while the
subsequent tasks only have $K$ training samples per class. No rehearsal buffer
is available. In this study, we use ImageNet-100 as the benchmark dataset with
$B=50$ and $C=10$ settings.

Implementation details. $K$ is set to 4/8/16/32 in our experiments. We choose
LUCIR [17] as the baseline.

Results. Due to the scarcity of data, few-shot CIL requires the model to not
only overcome forgetting but also transfer as much learned knowledge as
possible. As shown in Fig. LABEL:fig:fewshot, our proposed method shows
greater improvements in this challenging setting. Specifically, when $K$ is
32, our method achieves an improvement in accuracy of $14.0\%$ and a reduction
in the forgetting rate of $15.8\%$. This demonstrates that our method achieves
more effective knowledge transfer from the initial well-learned task by pre-
allocating the semantic target for each class. We also observe that when $K$
is 4, although the gain of accuracy is marginal, the forgetting rate is
reduced by $10.6\%$. It demonstrates that our method can alleviate the
representation drifting in the feature space. See supplementary material for
more results.

###  4.4 Task-incremental Learning Experiments

Benchmark protocol. The benchmark in this study is 10-split-CIFAR100, which
involves dividing CIFAR100 into 10 tasks with non-overlapping classes.
Following [50, 51], the last accuracy and forgetting rate are reported.

Implementation details. The learning rate is 1e-4 and epochs is 80. More
detailed hyperparameters are shown in the _supplementary material_.

Results. As shown in Tab. LABEL:table:task_domain_exp, our method improves the
accuracy by $3.9\%\sim 9.7\%$ and reduces the forgetting by $2.2\%\sim 9.6\%$.
Additionally, we observe that rehearsal-free methods can be competitive with
rehearsal-based methods. SI [56] with our method achieves $51.1\%$ accuracy,
surpassing the Rehearsal baseline by $3.0\%$. This finding suggests that our
approach facilitates the effective use of learned knowledge, thus mitigating
the reliance on old data.

###  4.5 Domain-incremental Learning Experiments

Benchmark protocol. OfficeHome [48] comprises four different domains, each
treated as a distinct task. As the label set is consistent across all tasks, a
shared classifier is utilized. We report the last accuracy and forgetting
rate.

Implementation details. The regularization coefficients of EWC, MAS, SI and
GEM are set to 100, 0.1, 0.3 and 5, respectively. More details are in the
_supplementary material_.

Results. Tab. LABEL:table:task_domain_exp reports that our method improves
accuracy by $1.2\%\sim 4.0\%$, while simultaneously reducing the forgetting
rate by $3.8\%\sim 7.6\%$. Due to the variability of the image domains, the
semantic targets often shift or are limited to the current domains only. In
contrast, the semantic targets generated by PLMs can utilize the rich source
of domain knowledge in PLMs, ensuring a more representative distribution of
these targets.

###  4.6 Analysis and Ablation

In this subsection, we conduct comprehensive ablation studies and analyses to
systematically examine LingoCL. Unless stated otherwise, the experiments are
based on LUCIR [17] and ImageNet-100 (B=10, C=10).

Analysis of freezing the language-guided classifier. We delve into two pivotal
components in the design of the language-guided classifier: 1) freezing the
weights, and 2) the semantic correlation in the weights. We first analyze the
effect of freezing the weights in Tab. LABEL:tab:frozen. The comparison
between updating and freezing weights reveals that updates lead to a decrease
in accuracy (from $61.7\%$ to $60.3\%$). This performance drop is attributed
to catastrophic forgetting in semantic targets, triggered by updating weights
for each task. It highlights the necessity of preserving the semantic
knowledge sourced from pretrained language models. However, it’s also notable
that even with updated weights, performance exceeds that of random
initialization, suggesting that strong initialization with rich semantics
plays a crucial role in CL.

Analysis of the semantic correlation in the classifier. Furthermore, we ablate
the semantic correlation in the classifier. By orthogonalizing the semantic
targets output of the pretrained language model, we construct a classifier
that removes semantic correlations among classes. The orthogonal classifier is
kept frozen during training. As indicated in Tab. LABEL:tab:sim, the removal
of semantic correlation leads to a $2.0\%$ decrease in accuracy and a $4.0\%$
increase in the forgetting rate. Nevertheless, the orthogonal classifier still
surpasses traditional vanilla classifiers by $3.1\%$ in accuracy. This
suggests that the frozen, orthogonal targets help to reduce interference
between different tasks, thereby diminishing feature drift in the feature
space. On the other hand, the absence of semantic correlation appears to
impede knowledge transfer across tasks. These findings underscore the dual
significance of maintaining a frozen state and preserving semantic correlation
in the classifier.

Comparison with oracle classifier. To thoroughly assess the impact of our
language-guided supervision, we introduce an oracle classifier as a benchmark
for oracle supervision. Initially, an idealized oracle model is trained with
data from all tasks, typically considered the performance upper bound in CL.
Subsequently, we replace the baseline model’s vanilla classifier with this
oracle classifier, which remains frozen during training. As shown in Tab.
LABEL:tab:signal, our language-guided classifier not only matches but
surpasses the oracle classifier in both average accuracy and forgetting rate.
This superiority is likely attributable to the fact that the dataset for
pretraining language models is conceptually more diverse and sufficient than
that used for the oracle model, providing semantically richer targets for each
class. These results highlight the exceptional efficacy of our approach.

Comparison with logits rectification-based methods. Tab. LABEL:tab:rectify
presents a comparison of LingoCL with other methods that modify the classifier
to address anomalies. We use a simple CIL baseline with rehearsal and
distillation on CIFAR100 under the setting of B=50, C=10. BiC [52] addresses
classifier bias by adding an extra linear layer, while EEIL [6] finetunes the
classifier using balanced data. Divergence head [14] utilizes an additional
classifier to separate the features of old and new tasks to preserve the
feature space for future classes. Existing methods mainly focus on addressing
the compatibility with old tasks using statistical corrections, whereas our
method stands out by considering the semantic correlation among all classes,
including the past and the future. Notably, LingoCL does not entirely conflict
with these methods; in fact, LingoCL can complement it to further enhance
performance, It is evidenced in Tab. LABEL:tab:cifar and Tab. LABEL:tab:i100,
where LingoCL notably enhances the efficacy of BiC.

Ablation on different language models. In Tab. LABEL:tab:language, we explore
two types of language models: multimodal pretraining models and unimodal
pretraining models. The overall results indicate that the multimodal
pretraining language models perform better, which we attribute to the
pretraining aligned with images allowing the language models to learn more
semantic information from visual cues. Although the semantic targets generated
by the unimodal pretraining models are not aligned with images, they still can
be easily fitted with trainable vision encoders. Furthermore, we found that
increasing the amount of pretraining data can effectively improve performance
($67.5\%\rightarrow 68.0\%$, $66.6\%\rightarrow 67.0\%$), as the language
model learns more concepts.

Effect of the number of exemplars for replay. We investigate the effect of the
number of old exemplars on model performance. The results in Fig.
LABEL:fig:rehearsal show that LingoCL can consistently improve the accuracy
and reduce the forgetting rate under all settings, especially when the number
of reserved exemplars is quite small. Notably, integrated with LingoCL, the
baseline with reserving only 2 exemplars per class is comparable to the
vanilla version by utilizing 20 exemplars per class ($69.6\%$ v.s. $70.2\%$),
further verifying the power of the proposed LingoCL.

Impact of the number of classes in the initial task. In this ablation, we
discuss the effect of the number of classes learned in the initial task. As
shown in Fig. LABEL:fig:classes, the X-axis represents the number of classes
in the initial task, and the remaining classes are incremented with 10 classes
per task. We can observe that LingoCL can bring a $+3.2\%\sim 5.1\%$ acreage
accuracy improvement and reduce the forgetting rate by $+5.9\%\sim 13.1\%$,
which illustrates the effectiveness and robustness of our method.

##  5 Conclusion

In this work, we present a new perspective on CL, _i.e_., how to utilize the
semantic knowledge in category names. Specifically, we use pretrained language
models to generate the semantic target for each class. Empirical study shows
that our method alleviates the representation drifting and facilitates
knowledge transfer. Extensive experiments across various scenarios demonstrate
the effectiveness of our method.

Acknowledgements. This work was supported partially by the National Natural
Science Foundations of China (Grants No.62376267, 62076242), the Pre-Research
Project on Civil Aerospace Technologies (No.D030312), the National Defense
Basic Scientific Research Program of China(No.JCKY2021203B063) and the innoHK
project.

## References

  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  In _NeurIPS_ , 2022.
  * Aljundi et al. [2018] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars.  Memory aware synapses: Learning what (not) to forget.  In _ECCV_ , pages 139–154, 2018.
  * Aljundi et al. [2019] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio.  Gradient based sample selection for online continual learning.  In _NeurIPS_ , 2019.
  * Belouadah and Popescu [2019] Eden Belouadah and Adrian Popescu.  Il2m: Class incremental learning with dual memory.  In _CVPR_ , pages 583–592, 2019.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _NeurIPS_ , 33:1877–1901, 2020.
  * Castro et al. [2018] Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari.  End-to-end incremental learning.  In _ECCV_ , pages 233–248, 2018.
  * Chaudhry et al. [2018] Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr.  Riemannian walk for incremental learning: Understanding forgetting and intransigence.  In _ECCV_ , pages 532–547, 2018.
  * Chen et al. [2023] Zhuo Chen, Yufeng Huang, Jiaoyan Chen, Yuxia Geng, Wen Zhang, Yin Fang, Jeff Z Pan, and Huajun Chen.  Duet: Cross-modal semantic grounding for contrastive zero-shot learning.  In _AAAI_ , pages 405–413, 2023.
  * Cherti et al. [2023] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev.  Reproducible scaling laws for contrastive language-image learning.  In _CVPR_ , 2023.
  * De Lange et al. [2021] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  _IEEE TPAMI_ , 44(7):3366–3385, 2021.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _CVPR_ , 2009.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL_ , 2019.
  * Douillard et al. [2020] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle.  Podnet: Pooled outputs distillation for small-tasks incremental learning.  In _ECCV_ , 2020.
  * Douillard et al. [2022] Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord.  Dytox: Transformers for continual learning with dynamic token expansion.  In _CVPR_ , 2022.
  * d’Ascoli et al. [2021] Stéphane d’Ascoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun.  Convit: Improving vision transformers with soft convolutional inductive biases.  In _ICML_ , pages 2286–2296. PMLR, 2021.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * Hou et al. [2019] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin.  Learning a unified classifier incrementally via rebalancing.  In _CVPR_ , pages 831–839, 2019.
  * Hsu et al. [2018] Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira.  Re-evaluating continual learning scenarios: A categorization and case for strong baselines.  In _NeurIPS Workshop_ , 2018.
  * Hu et al. [2019] Wenpeng Hu, Zhou Lin, Bing Liu, Chongyang Tao, Zhengwei Tao Tao, Dongyan Zhao, Jinwen Ma, and Rui Yan.  Overcoming catastrophic forgetting for continual learning via model adaptation.  In _ICLR_ , 2019.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _ICML_ , pages 4904–4916. PMLR, 2021.
  * Kemker and Kanan [2018] Ronald Kemker and Christopher Kanan.  Fearnet: Brain-inspired model for incremental learning.  In _ICLR_ , 2018.
  * Khan et al. [2023] Muhammad Gul Zain Ali Khan, Muhammad Ferjad Naeem, Luc Van Gool, Didier Stricker, Federico Tombari, and Muhammad Zeshan Afzal.  Introducing language guidance in prompt-based continual learning.  In _ICCV_ , pages 11463–11473, 2023.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kirkpatrick et al. [2017] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  _PNAS_ , 114(13):3521–3526, 2017.
  * Krizhevsky et al. [2009] Alex Krizhevsky et al.  Learning multiple layers of features from tiny images.  2009\.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023.
  * Li et al. [2022] Lei Li, Jingjing Xu, Qingxiu Dong Dong, Ce Zheng, Qi Liu, Lingpeng Kong, and Xu Sun.  What does vision supervision bring to language models? a case study of clip.  _OpenReview_ , 2022.
  * Li et al. [2019] Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong.  Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting.  In _ICML_ , pages 3925–3934. PMLR, 2019.
  * Liu et al. [2021] Yaoyao Liu, Bernt Schiele, and Qianru Sun.  Adaptive aggregation networks for class-incremental learning.  In _CVPR_ , pages 2544–2553, 2021.
  * Lopez-Paz and Ranzato [2017] David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  In _NeurIPS_ , 2017.
  * Mallya and Lazebnik [2018] Arun Mallya and Svetlana Lazebnik.  Packnet: Adding multiple tasks to a single network by iterative pruning.  In _CVPR_ , pages 7765–7773, 2018.
  * Mallya et al. [2018] Arun Mallya, Dillon Davis, and Svetlana Lazebnik.  Piggyback: Adapting a single network to multiple tasks by learning to mask weights.  In _ECCV_ , pages 67–82, 2018.
  * Masana et al. [2022a] Marc Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer.  Class-incremental learning: Survey and performance evaluation on image classification.  _IEEE TPAMI_ , pages 1–20, 2022a.
  * Masana et al. [2022b] Marc Masana, Xialei Liu, Bartłomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer.  Class-incremental learning: survey and performance evaluation on image classification.  _IEEE TPAMI_ , 45(5):5513–5533, 2022b.
  * Ni et al. [2022] Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling.  Expanding language-image pretrained models for general video recognition.  In _European Conference on Computer Vision_ , pages 1–18. Springer, 2022.
  * Nie et al. [2023] Xing Nie, Shixiong Xu, Xiyan Liu, Gaofeng Meng, Chunlei Huo, and Shiming Xiang.  Bilateral memory consolidation for continual learning.  In _CVPR_ , pages 16026–16035, 2023.
  * Ostapenko et al. [2019] Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi.  Learning to remember: A synaptic plasticity driven framework for continual learning.  In _CVPR_ , pages 11321–11329, 2019.
  * Prabhu et al. [2020] Ameya Prabhu, Philip HS Torr, and Puneet K Dokania.  Gdumb: A simple approach that questions our progress in continual learning.  In _ECCV_ , pages 524–540, 2020.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , pages 8748–8763. PMLR, 2021.
  * Ramasesh et al. [2021] Vinay V Ramasesh, Ethan Dyer, and Maithra Raghu.  Anatomy of catastrophic forgetting: Hidden representations and task semantics.  In _ICLR_ , 2021.
  * Rebuffi et al. [2017] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, G. Sperl, and Christoph H. Lampert.  icarl: Incremental classifier and representation learning.  In _CVPR_ , pages 5533–5542, 2017.
  * Robbins and Monro [1951] Herbert Robbins and Sutton Monro.  A stochastic approximation method.  _The annals of mathematical statistics_ , pages 400–407, 1951.
  * Rusu et al. [2016] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  _arXiv preprint arXiv:1606.04671_ , 2016.
  * Shi et al. [2022] Yujun Shi, Kuangqi Zhou, Jian Liang, Zihang Jiang, Jiashi Feng, Philip HS Torr, Song Bai, and Vincent YF Tan.  Mimicking the oracle: an initial phase decorrelation approach for class incremental learning.  In _CVPR_ , pages 16722–16731, 2022.
  * Tao et al. [2020a] Xiaoyu Tao, Xinyuan Chang, Xiaopeng Hong, Xing Wei, and Yihong Gong.  Topology-preserving class-incremental learning.  In _ECCV_ , pages 254–270, 2020a.
  * Tao et al. [2020b] Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong.  Few-shot class-incremental learning.  In _CVPR_ , pages 12183–12192, 2020b.
  * Tsimpoukelli et al. [2021] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill.  Multimodal few-shot learning with frozen language models.  In _NeurIPS_ , pages 200–212, 2021.
  * Venkateswara et al. [2017] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In _CVPR_ , pages 5018–5027, 2017.
  * Wang et al. [2023] Junyang Wang, Yuanhong Xu, Juhua Hu, Ming Yan, Jitao Sang, and Qi Qian.  Improved visual fine-tuning with natural language supervision.  In _ICCV_ , 2023.
  * Wang et al. [2021] Shipeng Wang, Xiaorong Li, Jian Sun, and Zongben Xu.  Training networks in null space of feature covariance for continual learning.  In _CVPR_ , pages 184–193, 2021.
  * Wang et al. [2022] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister.  Learning to prompt for continual learning.  In _CVPR_ , pages 139–149, 2022.
  * Wu et al. [2019] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu.  Large scale incremental learning.  In _CVPR_ , pages 374–382, 2019.
  * Yan et al. [2021] Shipeng Yan, Jiangwei Xie, and Xuming He.  Der: Dynamically expandable representation for class incremental learning.  In _CVPR_ , pages 3014–3023, 2021.
  * Yang et al. [2019] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.  Xlnet: Generalized autoregressive pretraining for language understanding.  _NeurIPS_ , 32, 2019.
  * Yuan et al. [2021] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.  Florence: A new foundation model for computer vision.  _arXiv preprint arXiv:2111.11432_ , 2021.
  * Zenke et al. [2017] Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In _ICML_ , pages 3987–3995, 2017.
  * Zhao et al. [2020] Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shutao Xia.  Maintaining discrimination and fairness in class incremental learning.  In _CVPR_ , pages 13205–13214, 2020.
  * Zhao et al. [2024] Hongbo Zhao, Bolin Ni, Haochen Wang, Junsong Fan, Fei Zhu, Yuxi Wang, Yuntao Chen, Gaofeng Meng, and Zhaoxiang Zhang.  Continual forgetting for pre-trained vision models.  _arXiv preprint arXiv:2403.11530_ , 2024.
  * Zhong et al. [2018] Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu.  Practical block-wise neural network architecture generation.  In _CVPR_ , pages 2423–2432, 2018.

Supplementary Material

In this supplementary material, we provide additional details regarding the
main manuscript. More specifically:

  * •

In Sec. 6, we provide further explanation of the datasets, protocols and
metrics.

  * •

In Sec. 7, we provide the detailed hyperparameters of different continual
learning settings.

  * •

In Sec. 8, we provide additional experiments and results.

##  6 Datasets, Protocols and Metrics

In Sec. 6.1, we present the statistical information for the datasets used in
our experiments. In Sec. 6.2, we describe the continual learning protocols
that are commonly used in the literature. Finally, in Sec. 6.3, we introduce
the evaluation metrics used to measure the performance comprehensively.

###  6.1 Datasets statistics

  * •

Split-CIFAR-100 (Task-IL). The CIFAR100 dataset [25] comprises 60,000 32×32
images belonging to 100 classes. In task-incremental learning setting, Split-
CIFAR-100 splits the original CIFAR-100 [25] into 10 tasks, 10 disjoint
classes per task.

  * •

CIFAR-100 (Class-IL). In the class-incremental learning setting, we divide the
classes into mutually exclusive sets. The first task consists of $B$ classes,
and each subsequent task consists of $C$ classes.

  * •

ImageNet-100 (Class-IL). ImageNet-100 [41] is the subset of ImageNet1000 [11]
containing 100 classes [41]. These classes are selected from the first 100
classes after a random shuffle with seed 1,993 [59]. Each image is represented
by 224×224 pixels.

  * •

OfficeHome (Domain-IL). OfficeHome [48] consists of images from four different
domains: Artistic images, Clip Art, Product images and Real-World images. For
each domain, the dataset contains images of 65 object categories found
typically in Office and Home settings. Each image is represented by 224×224
pixels.

We use the official categories provided by the respective dataset creators for
all datasets, which can be accessed through the dataset resources [25, 11,
48]. These categories are also presented in Fig. 2, Fig. 4, and Fig. 3 for
CIFAR100, ImageNet100, and OfficeHome, respectively.

###  6.2 Continual Learning Protocols

In continual learning (CL), the model is trained in a task-by-task manner. We
define a sequence of tasks denoted by
$\mathcal{D}=\\{\mathcal{D}_{1},\cdots,\mathcal{D}_{T}\\}$. The $t$-th task,
denoted by
$\mathcal{D}_{t}=\\{(\boldsymbol{x}^{t}_{i},y^{t}_{i})\\}_{i=1}^{n_{t}}$,
comprises tuples consisting of an input sample
$\boldsymbol{x}^{t}\in\mathcal{X}_{t}$ and its corresponding label
$y^{t}\in\mathcal{Y}_{t}$. Depending on the target set and the number of
training samples, CL protocols can be divided into four common categories:

  * •

Task-incremental learning where the target set of test sample
$\boldsymbol{x}^{t}$ is $\mathcal{Y}_{t}$.

  * •

Class-incremental learning where the target set of test sample
$\boldsymbol{x}^{t}$ is $\cup_{i=1}^{t}\mathcal{Y}_{i}$.

  * •

Few-shot Class-incremental learning where the target set of test sample
$\boldsymbol{x}^{t}$ is $\cup_{i=1}^{t}\mathcal{Y}_{i}$ and the $n_{t}(t>1)$
of training set is limited.

  * •

Domain-incremental learning where each task shares the same target set,
_i.e_., $\mathcal{Y}_{1}=\mathcal{Y}_{2}=\cdots=\mathcal{Y}_{T}$.

###  6.3 Evaluation Metrics

Formally, suppose the model is conducted for $N$ tasks and let $A_{i.j}$
denote the classification accuracy evaluated on the test set of the task $i$
after the incremental learning of the $j$-th task is $A_{i.j}$. Our method is
extensively evaluated by three commonly used metrics:

  * •

Last-step accuracy (Last) which measures the overall performance at last:

$\displaystyle Last$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{N}A_{i,N}$ (5)

  * •

Average incremental accuracy (Avg) which measure the performance evolution
along the learning trajectory:

$\displaystyle Avg$
$\displaystyle=\frac{1}{N}\sum_{j=1}^{N}(\frac{1}{j}\sum_{i=1}^{j}A_{i,j})$
(6)

  * •

Forgetting rate (Forget) which measures the degree of forgetting on learned
tasks:

$\displaystyle Forget$
$\displaystyle=\frac{1}{N-1}\sum_{i=1}^{N-1}\max\\{A_{i,1},\cdots,A_{i,N-1}\\}-A_{i,N}$
(7)

Besides, following [40], we perform a subspace similarity analysis to measure
the representation drifting. Given the input from the same task, let
$\mathbf{F}_{t},\mathbf{F}_{t^{\prime}}\in\mathbb{R}^{n\times d}$ denote the
output of the encoder after the $t$-th task and after the $t^{\prime}$-th task
($t^{\prime}>t$), respectively. We compute the PCA decomposition of
$\mathbf{F}_{t}$, _i.e_., the eigenvectors $(v_{1},v_{2},\cdots)$ of
$\mathbf{F}_{t}^{\top}\mathbf{F}_{t}$. Let $\mathbf{V}_{k,t}$ are the top-$k$
principal directions of $\mathbf{F}_{t}$, and $\mathbf{V}_{k,t^{\prime}}$ the
corresponding matrix for $\mathbf{F}_{t^{\prime}}$. The representation
drifting from the $t$-th task to the $t^{\prime}$-th can be defined as:

$\mathrm{RepreDrift}_{k}(\mathbf{F}_{t},\mathbf{F}_{t^{\prime}})=1-\frac{1}{k}\|\mathbf{V}_{k,t}^{T}\mathbf{V}_{k,t^{\prime}}\|_{F}^{2}.$
(8)

$\frac{1}{k}\|\mathbf{V}_{k,t}^{T}\mathbf{V}_{k,t^{\prime}}\|_{F}^{2}$
measures the similarity of the subspaces spanned by $\mathbf{F}_{t}$ and
$\mathbf{F}_{t^{\prime}}$. The smaller the similarity between the subspaces at
task $t$ and $t^{\prime}$, the greater the representation drifting.

##  7 Hyperparameter details

We provide the detailed hyperparameters of class-incremental learning, task-
incremental learning, and domain-incremental experiments in Sec. 7.1, Sec. 7.2
and Sec. 7.3, respectively.

###  7.1 Class-incremental learning

For CNN-based methods [29, 44, 17, 52, 4], we employ the SGD optimizer [42]
with an initial learning rate of 0.1, a momentum of 0.9, and a batch size of
128. In the experiments performed on CIFAR100, all models are trained for 160
epochs within each task, with the learning rate decreased by a factor of 10 at
the 80-th and 120-th epochs. For ImageNet100, all models are trained for 90
epochs within each task, with the learning rate reduced by a factor of 10 at
the 30-th and 60-th epochs.

For ViT-based methods such as DyTox [14], we follow the original
hyperparameters. We train the model for 500 epochs per task with Adam [23]
with a learning rate of 5e-4, including 5 epochs of warmup. At the end of each
task (except the first), we finetune the model for 20 epochs with a learning
rate of 5e-5 on a balanced dataset.

###  7.2 Task-incremental learning

The learning rate starts from 1e-4 and decays at epochs 30 and 60 with a
multiplier of 0.1. The total epochs are 80. The batch size is set to 32. The
regularization coefficient of EWC [24], MAS [2] and SI [56] are set to 100,
0.1 and 10, respectively.

###  7.3 Domain-incremental learning

We use the Adam [23] optimizer with an initial learning rate 0.001, and a
batch size of 128. The epochs are 80 and the learning rate is decay by 10 at
the 40-th and 60-th epochs. The regularization coefficients of EWC [24], MAS
[2], SI [56] and GEM [30] are set to 100, 0.1, 0.3 and 5, respectively.

##  8 Additional Experiments Analysis

In Sec. 8.1 and Sec. 8.2, we present additional results on class-incremental
learning and few-shot class-incremental learning experiments, respectively.
Moreover, in Sec. 8.3, we offer an analysis of the prompting technique.

###  8.1 Class-incremental Learning

In the main manuscript, Table 1 presents the results of class-incremental
learning (CIL) experiments on CIFAR100 under the setting where the number of
base classes ($B$) equals the number of incremental classes ($C$). To provide
further insights, we supplement additional results under the setting where
$B=50$ in Tab. LABEL:tab:suppl_cifar. The results show that our proposed
method consistently and significantly improves the performance across all
metrics under the $B=50$ setting. These results provide further evidence of
the effectiveness of our approach in various CIL settings.

###  8.2 Few-shot Class-incremental Learning

The results of few-shot class-incremental learning are displayed in Figure 7
of the main manuscript. To provide more quantitative results, we also present
them in Tab. LABEL:tab:suppl_few. It is evident that our proposed approach
consistently achieves significant performance gains, with or without buffers.
These findings provide further evidence that our method facilitates effective
knowledge transfer from the initial well-learned task.

###  8.3 Prompting Technique

Prompting [39] is a widely used technique to transfer knowledge from
pretrained language models. In Tab. LABEL:tab:suppl_temp, we compare three
different settings for prompting. In setting #1, we used the category name as
input without any additional templates. In setting #2, we used the template a
photo of a {object}. Finally, in setting #3 [39], we averaged the results of
80 different templates. Our results show that the use of templates can
slightly ease the forgetting, which we attribute to the fact that the template
ensemble enhances the stability of the generated features and reduces the
effect of noise. These findings highlight the robustness and generalizability
of our approach.

apple
aquarium fish
baby
bear
beaver
bed
bee
beetle
bicycle
bottle
bowl
boy
bridge
bus
butterfly
camel
can
castle
caterpillar
cattle
chair
chimpanzee
clock
cloud
cockroach
couch
crab
crocodile
cup
dinosaur
dolphin
elephant
flatfish
forest
fox
girl
hamster
house
kangaroo
computer keyboard
lamp
lawn mower
leopard
lion
lizard
lobster
man
maple tree
motorcycle
mountain
mouse
mushroom
oak tree
orange
orchid
otter
palm tree
pear
pickup truck
pine tree
plain
plate
poppy
porcupine
possum
rabbit
raccoon
ray
road
rocket
rose
sea
seal
shark
shrew
skunk
skyscraper
snail
snake
spider
squirrel
streetcar
sunflower
sweet pepper
table
tank
telephone
television
tiger
tractor
train
trout
tulip
turtle
wardrobe
whale
willow tree
wolf
woman
worm

Figure 2: The categories of CIFAR100.

Alarm Clock
Backpack
Batteries
Bed
Bike
Bottle
Bucket
Calculator
Calendar
Candles
Chair
Clipboards
Computer
Couch
Curtains
Desk Lamp
Drill
Eraser
Exit Sign
Fan
File Cabinet
Flipflops
Flowers
Folder
Fork
Glasses
Hammer
Helmet
Kettle
Keyboard
Knives
Lamp Shade
Laptop
Marker
Monitor
Mop
Mouse
Mug
Notebook
Oven
Pan
Paper Clip
Pen
Pencil
Postit Notes
Printer
Push Pin
Radio
Refrigerator
Ruler
Scissors
Screwdriver
Shelf
Sink
Sneakers
Soda
Speaker
Spoon
TV
Table
Telephone
ToothBrush
Toys
Trash Can
Webcam

Figure 3: The categories of OfficeHome.

eastern hog-nosed snake
rooster
wardrobe
corkscrew
isopod
beaver
acorn
goldfinch
Siamese cat
chiffonier
bittern bird
screw
Cairn Terrier
valley
lens cap
Brittany dog
Appenzeller Sennenhund
entertainment center
Greater Swiss Mountain Dog
Band-Aid
dhole
sea anemone
ice cream
threshing machine
bell or wind chime
sunglasses
can opener
microphone
quail
brussels griffon
computer keyboard
hand-held computer
eel
Norwegian Elkhound
mailbox
leopard
mitten
Cocker Spaniel
split-rail fence
dowitcher
tennis ball
Afghan Hound
parking meter
snow leopard
spiny lobster
monarch butterfly
hook
drumstick
toilet paper
sawmill
silver salmon
remote control
chain mail
swim trunks / shorts
white stork
teddy bear
moped
horse chestnut seed
holster
ping-pong ball
purse
indigo bunting
wolf spider
lighthouse
sturgeon
toaster
Arctic fox
doormat
southern black widow
high-speed train
vending machine
cricket insect
longhorn beetle
African rock python
red wine
assault rifle
carbonara
CRT monitor
candy store
academic gown
cannon
music speaker
African wild dog
farm plow
koala
crutch
Groenendael dog
Norwich Terrier
cardboard box / carton
combination lock
candle
Windsor tie
pan flute
rose hip
small white butterfly
space shuttle
Chow Chow
wool
ring binder
alligator lizard

Figure 4: The categories of ImageNet100.