# Adapt Your Teacher: Improving Knowledge Distillation for Exemplar-free
Continual Learning

Filip Szatkowski corresponding author, email<EMAIL_ADDRESS>Warsaw University of Technology  IDEAS NCBR  Mateusz Pyla  IDEAS NCBR
Jagiellonian University, Faculty of Mathematics and Computer Science
Jagiellonian University, Doctoral School of Exact and Natural Sciences  Marcin
Przewięźlikowski  IDEAS NCBR  Jagiellonian University, Faculty of Mathematics
and Computer Science  Jagiellonian University, Doctoral School of Exact and
Natural Sciences
Sebastian Cygert  IDEAS NCBR  Gdańsk University of Technology  Bartłomiej
Twardowski  IDEAS NCBR  Autonomous University of Barcelona  Computer Vision
Center  Tomasz Trzciński  Warsaw University of Technology  IDEAS NCBR
Tooploox

###### Abstract

In this work, we investigate exemplar-free class incremental learning (CIL)
with knowledge distillation (KD) as a regularization strategy, aiming to
prevent forgetting. KD-based methods are successfully used in CIL, but they
often struggle to regularize the model without access to exemplars of the
training data from previous tasks. Our analysis reveals that this issue
originates from substantial representation shifts in the teacher network when
dealing with out-of-distribution data. This causes large errors in the KD loss
component, leading to performance degradation in CIL models. Inspired by
recent test-time adaptation methods, we introduce Teacher Adaptation (TA), a
method that concurrently updates the teacher and the main models during
incremental training. Our method seamlessly integrates with KD-based CIL
approaches and allows for consistent enhancement of their performance across
multiple exemplar-free CIL benchmarks. The source code for our method is
available at https://github.com/fszatkowski/cl-teacher-adaptation.

##  1 Introduction

Continual learning aims to create machine learning models capable of acquiring
new knowledge and adapting to evolving data distributions over time. One of
the most challenging continual learning scenarios is _class incremental
learning_ (CIL) [50, 32], where the model is trained to classify objects
incrementally from the sequence of tasks, without forgetting the previously
learned ones.

Figure 1: Enhancement of vanilla Knowledge Distillation approach used in
Continual Learning with our method of Teacher Adaptation. When training the
student model on the new task, we allow the teacher model to continuously
update its batch normalization statistics, which reduces the divergence
between the representations in both models. Our method leads to lower
knowledge distillation loss and an overall more stable model.

A simple and effective method of reducing forgetting is by leveraging
_exemplars_ [41, 21, 6, 37] of previously encountered training examples, e.g.
by replaying them or using them for regularization. However, this approach
presents challenges, particularly in terms of high storage needs and privacy
concerns. These problems can affect edge devices, due to their limited storage
capacity, and medical data, given their sensitive nature.

Figure 2:  Applying our teacher adaptation (TA) method reduces knowledge
distillation (KD) loss and improves stability throughout continual learning.
(left) KD loss and cross-entropy (CE) loss of training the model with and
without TA. Our method leads to more consistent representation, as visualized
by the CKA [24] between the representations of the new data obtained in the
teacher and student models while learning the second task (middle). KD with TA
leads to better task-agnostic accuracy (right). We conduct the experiments on
CIFAR100 split into 10 tasks.

A common approach for exemplar-free CIL is knowledge distillation (KD), where
the current model (student) is trained on the new data with a regularization
term that minimizes the output difference with the previous model (teacher),
which is kept frozen. This approach was introduced by LwF [29] and has been
extended by many other methods. However, most of these methods use exemplars,
such as iCaRL [41], EEIL [8], LUCIR [18], PodNET [14], SSIL [2], or rely on
external datasets [61, 28].

Exemplar-free CIL still remains challenging [47] for KD methods due to the
possibility of significant distribution drift in subsequent tasks. Such drift
leads to large errors during training with KD loss, causing more undesired
changes in the main model and harming the overall performance of the CIL
training. This raises the question: Can we adjust the teacher model to better
transfer knowledge from earlier tasks?

Motivated by the recent domain adaptation methods [52, 45], we examine the
role of batch normalization statistics in CIL training. We conjecture that in
standard KD methods, the KD loss between models with different normalization
statistics may introduce unwanted model updates due to the data distribution
shifts. To avoid this, we propose to continuously adapt them to the new data
for the teacher model while training the student.

We show that adapting the teacher’s batch normalization statistics to the new
task can significantly lower KD loss without affecting the CE loss, which
reduces changes in the model’s representations (Figure 2). We note that, while
the idea of changing the teacher model was explored in the standard KD
settings [64, 30], our approach is the first application of this idea to CIL
scenario, where the teacher and the model are trained on non-overlapping data.
Moreover, our method works differently by exploiting the batch normalization
statistics. We apply our method on top of different distillation strategies in
CIL and show consistent improvements across various settings.

In summary, we make the following contributions:

  1. 1.

We revisit the KD-based class-incremental learning (CIL) framework and study
the negative impact of regularization using out-of-distribution data. We are
the first to highlight the need for adjusting the teacher model in an
exemplar-free situation, where it is usually kept frozen.

  2. 2.

We propose a simple yet highly effective technique called Teacher Adaptation
(TA), that enhances KD for exemplar-free CIL.

  3. 3.

Through extensive experiments, we demonstrate that TA can be seamlessly
integrated with various KD approaches, leading to significant improvements
over the baselines across a wide range of continual learning scenarios for
various datasets. We show that those improvements hold even when using
pretrained models or in the presence of substantial distributional shifts
between consecutive domains.

##  2 Related works

Class Incremental Learning (CIL) [50, 32] is a subfield of continual learning,
where the aim is to learn incrementally from a stream of tasks, without the
task identifier. There exist several families of approaches to CIL:

Memory-based methods store either exemplars or features from the previous
tasks in the buffer [41, 21, 6, 37] and use them during training the new task
to consolidate previously learned knowledge. Those methods usually perform
well, but their practical applications are limited due to privacy concerns and
memory requirements that arise when storing the data. Architectural approaches
focus on modifying the structure of the model, often by allocating certain
parameters to corresponding tasks [54, 53]. Finally, regularization-based
methods aim to preserve the knowledge in the network by imposing constraints
on the changes in model weights [23] or activations during learning the new
task [29]. Many CIL methods often also combine the above approaches [8, 55,
42], for instance using both memory and regularizaion [41, 29, 2].

Regularization methods for continual learning offer a way to prevent
forgetting with constant memory usage and no privacy issues. There are two
main types of regularization methods: (i) parameter regularization and (ii)
functional regularization. The first type of methods regularizes the model
weights, for example using the Fisher Information Matrix [23], synaptic
saliency [59] or the gradient inspection [3]. On the other hand, functional
regularization methods employ knowledge distillation (KD) techniques to
regularize model activations. KD was originally proposed by Hinton et al. [17]
to transfer the knowledge from a larger model to a smaller one. In CL, KD was
first applied in Learning without Forgetting (LwF) [29], where the model is
discouraged from drifting too far from the model from previous tasks. We
describe KD methods in detail in Section 3.1.

Functional regularization has been widely used in CL since the introduction of
LwF [40, 18, 41, 37] and numerous variants of KD have been proposed.
Particularly, SSIL [2] uses task-wise knowledge distillation, while PODNET
[14] regularizes using spatial-based distillation loss applied throughout the
model.

Multi-level knowledge distillation [13] uses the current model to distill the
knowledge from pruned snapshots of all previous models, while ANCL [22]
distills simultaneously from the previous task model and the model learned
specifically for the new task. Moreover, DMC [60] uses knowledge distillation
on an auxiliary dataset to consolidate the knowledge from previous tasks.

However, most of those methods use a memory buffer and their performance
depends on it heavily, which makes them impractical for exemplar-free
settings. Recently, several works explored the idea of modifying the teacher
model through meta-learning for better knowledge transfer in standard KD
setting [64, 30]. Similarly, works on KD [16] suggest that updating the
normalization statistics of the teacher model on the data used to train the
student improves the performance. However, to our knowledge, our method is the
first one that explores different approaches to teacher adaptation, such as
updating the normalization statistics, in the context of CIL.

Batch Normalization (BN) [20] is widely used in deep learning models, but can
be problematic for settings where the data distribution changes over time.
Alternative approaches such as LayerNorm [5] or GroupNorm [57] do not rely on
the batch-wise statistics, but directly replacing BN layers with them was
shown to often decrease the performance of the models. Several domain
adaptation methods achieve domain transfer through the use of normalization
statistics [52, 45]. Recent work on efficient finetuning of large language
models using only normalization layers [38] also suggests that the
normalization layers play a crucial role in training deep neural networks. In
CL, it was shown that BN can cause a discrepancy between the training and
testing phases of BN, as the testing data is normalized using the statistics
biased towards the current task, which results in higher forgetting of older
tasks [44]. Several works have attempted to address this issue by CL-specific
modifications to BN [36, 9]. However, those approaches are not suited for
exemplar-free settings.

##  3 Method

In class-incremental learning setup, the model learns tasks sequentially. Each
task contains several classes which are disjoint with the classes in other
tasks. During training task $t$, we only have access to the data $D^{t}$ from
task $t$ which contains images $\mathbf{x}_{i}\in X^{t}$ with class labels
$y_{i}\in C^{t}$. Thus an incremental learning problem $\mathcal{T}$ with $n$
tasks can be formulated as:
$\mathcal{T}=\left\\{D^{1},D^{2},...,D^{t},...,D^{n}\right\\}$, where after
training $n$ tasks we evaluate the model on all classes $C^{1}\cup\ldots\cup
C^{n}$, without knowing the task label at inference time (this is different
than task-incremental learning, where task id can be used).

Below, we first introduce standard KD-based methods for exemplar-free CIL.
Then we outline a problem of diverging batch normalization statistics between
the teacher and student model caused by the shifts in training data between
subsequent tasks. Finally, we propose to address this issue with a method that
we call _Teacher Adaptation_ \- a simple, yet effective solution that allows
the teacher model to continuously update its normalization statistics
alongside the student when training on the new data. The method in comparison
to standard LwF is presented in Figure 1.

###  3.1 Knowledge Distillation in Continual Learning

Knowledge distillation is one of the most popular techniques employed to
reduce forgetting between subsequent tasks in incremental learning. Continual
learning methods that use knowledge distillation save the model $\Theta_{t}$
(_teacher_) trained after each task $t$ and use it during learning the model
$\Theta_{t+1}$ (_student_) on new task $t+1$. The learning objective for task
$t+1$ then becomes:

$L=L_{CE}+\lambda L_{KD},$ (1)

where $L_{CE}$ is the cross-entropy loss for classification on new data,
$L_{KD}$ is the knowledge distillation loss computed using $\Theta_{t}$ and
$\Theta_{t+1}$, and $\lambda$ is the coefficient that controls the trade-off
between stability and plasticity. The general formula for knowledge
distillation loss can include either output from the final layer of the model
[29, 41, 2], or also representations from intermediate model layers [12, 14].
In practice, most exemplar-free methods that use knowledge distillation
compute knowledge distillation loss using only the final layer outputs, and
various methods that use intermediate representations usually only perform
well with exemplars [47].

Multiple variants of knowledge distillation loss were proposed for continual
learning. In exemplar-free CIL, KD loss is usually computed with the logits
$y_{o}$, $\hat{y_{o}}$ returned by the current and previous models
respectively. Following Li et al. [29], we denote that the loss uses logits
corresponding to previously seen classes with a subscript $o$. Ahn et al. [2]
classify KD methods into general KD (GKD), which aggregates together logits
belonging to the classes from all the previous tasks, and task-wise KD (TKD),
which treats classes within each task separately.

GKD loss appears in several works [28, 56, 62] and usually uses cross-entropy:

$\displaystyle\mathcal{L}_{GKD}(\mathbf{y}_{o},\mathbf{\hat{y}}_{o})$
$\displaystyle=-\sum_{i=1}^{|C_{t}|}p_{o}^{(i)}\log\hat{p}_{o}^{(i)},$ (2)

where $p_{o}^{(i)}$ is the probability of the $i$-th class and $|C_{t}|$ is
the number of classes learned by previous model $\Theta_{t}$. Probabilities
$p_{o}^{(i)}$, $\hat{p}_{o}^{(i)}$ are computed with temperature parameter $T$
as follows:

$p_{o}^{(i)}=\dfrac{e^{y_{o}/T}}{\sum_{j}e^{y_{o}/T}},\quad\hat{p_{o}}^{(i)}=\dfrac{e^{\hat{y_{o}}/T}}{\sum_{j}e^{\hat{y_{o}}/T}}$
(3)

Comparatively, TKD loss, which was also used in several works [29, 8, 2], sums
of the separately computed losses for each task:

$\displaystyle\mathcal{L}_{TKD}(\mathbf{y}_{o},\mathbf{\hat{y}}_{o})$
$\displaystyle=\sum_{i=1}^{t}\mathcal{D}_{KL}(p_{o}^{(i)}\log\hat{p}_{o}^{(i)}),$
(4)

where $\mathcal{D}_{KL}$ is Kullback–Leibler divergence and $p_{o}^{(i)}$,
$\hat{p}_{o}^{(i)}$ are computed task-wise across the classes that belong to
task $i$ as in Equation 3.

Rebuffi et al. [41] proposed another distinct variant of KD for multi-class
incremental learning, where the loss is computed element-wise for each class
(MKD):

$\displaystyle\mathcal{L}_{MKD\ }(\mathbf{y}_{o},\mathbf{\hat{y}}_{o})$
$\displaystyle=-\sum_{i=1}^{|C_{t}|}\sigma(y_{o}^{i})\log\sigma(\hat{y}_{o}^{i}),$
(5)

where $\sigma$ is a sigmoid function.

Additionally, a more recent KD-based method, Auxiliary Network Continual
Learning (ANCL) [22], explores the idea of multi-teacher KD for continual
learning. ANCL trains an auxiliary network trained only for the current task
and combines standard GKD loss with the KD loss computed between outputs for
the current task and outputs of this auxiliary network.

In this work, we investigate the aforementioned KD techniques (GKD, MKD, TKD,
ANCL).

###  3.2 Teacher Adaptation

Most models used in class incremental learning for vision tasks are
convolutional neural networks such as ResNet [15]. Those models typically use
batch normalization layers and keep the parameters and statistics of those
layers in the teacher model $\Theta_{t}$ fixed during learning $\Theta_{t+1}$.
However, with the changing distribution of the data in the new task, batch
normalization statistics in the student and teacher models quickly diverge,
which leads to higher KD loss. Gradient updates in this case not only
regularize the model towards the stable performance on previous tasks but also
compensate for the changes in normalization statistics, which needlessly
overwrites the knowledge stored in the model and harms the learning process.

Inspired by test-time adaptation methods [52], we propose to reduce this
negative interference with a simple method that we label _Teacher Adaptation
(TA)_. Our method updates batch normalization statistics of both models
simultaneously on new data while learning the new task. As shown in Figure 2,
it allows for significantly reduced KD loss over learning from sequential
tasks in CIL, which improves the overall model stability. We provide
additional analysis on why TA improves learning with KD in Section 4.4 and in
the Appendix.

|  | Equal split of classes | Start from half of the classes
---|---|---|---
|  | 10 tasks | 20 tasks | 11 tasks | 26 tasks
|  | $Acc_{Inc}\uparrow$ | $Forg_{Inc}\downarrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Inc}\downarrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Inc}\downarrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Inc}\downarrow$
a) CIFAR100 | GKD | 42.52$\pm$0.76 | 22.26$\pm$0.31 | 31.89$\pm$0.45 | 34.68$\pm$1.87 | 41.69$\pm$1.18 | 18.09$\pm$0.88 | 17.64$\pm$0.93 | 9.67$\pm$0.26
+ours | 45.25$\pm$1.02 | 19.87$\pm$0.34 | 37.11$\pm$0.64 | 24.87$\pm$1.04 | 46.27$\pm$1.09 | 13.98$\pm$0.98 | 26.15$\pm$0.94 | 8.73$\pm$0.85
MKD | 39.36$\pm$0.70 | 42.74$\pm$0.52 | 32.89$\pm$0.42 | 32.01$\pm$1.36 | 41.04$\pm$0.93 | 15.37$\pm$0.33 | 19.14$\pm$1.36 | 8.76$\pm$0.35
+ours | 44.85$\pm$0.80 | 30.08$\pm$0.57 | 36.79$\pm$0.70 | 21.84$\pm$0.33 | 44.19$\pm$1.17 | 12.56$\pm$0.53 | 26.10$\pm$0.75 | 9.17$\pm$0.20
TKD | 43.74$\pm$0.84 | 23.65$\pm$0.79 | 34.58$\pm$0.34 | 21.13$\pm$1.17 | 40.44$\pm$1.40 | 12.20$\pm$0.46 | 14.64$\pm$0.33 | 6.02$\pm$0.54
+ours | 46.21$\pm$0.86 | 20.45$\pm$0.57 | 36.26$\pm$0.71 | 17.01$\pm$0.89 | 44.22$\pm$1.08 | 11.79$\pm$0.57 | 22.00$\pm$0.97 | 9.36$\pm$0.68
ANCL | 43.15$\pm$0.49 | 32.78$\pm$1.52 | 34.32$\pm$0.41 | 36.74$\pm$1.38 | 45.16$\pm$0.32 | 20.21$\pm$0.30 | 21.84$\pm$1.33 | 11.79$\pm$0.51
+ours | 46.73$\pm$0.20 | 26.86$\pm$1.17 | 38.48$\pm$0.94 | 27.99$\pm$1.20 | 48.25$\pm$0.33 | 16.11$\pm$0.19 | 29.67$\pm$1.13 | 10.98$\pm$0.56
b) TinyImageNet200 | GKD | 32.12$\pm$0.57 | 21.21$\pm$1.31 | 25.75$\pm$0.65 | 27.55$\pm$1.81 | 34.32$\pm$1.96 | 11.41$\pm$2.80 | 23.30$\pm$1.59 | 12.63$\pm$2.21
+ours | 33.90$\pm$0.78 | 17.70$\pm$1.48 | 27.85$\pm$0.90 | 21.65$\pm$1.43 | 37.50$\pm$1.84 | 10.14$\pm$1.10 | 28.82$\pm$1.95 | 11.44$\pm$0.61
MKD | 31.04$\pm$1.00 | 16.90$\pm$0.75 | 25.22$\pm$0.88 | 25.80$\pm$2.46 | 33.75$\pm$2.11 | 10.30$\pm$1.63 | 23.42$\pm$1.95 | 10.41$\pm$1.97
+ours | 32.22$\pm$0.95 | 11.94$\pm$1.07 | 27.39$\pm$1.33 | 16.74$\pm$1.52 | 32.99$\pm$1.98 | 6.42$\pm$0.80 | 25.75$\pm$1.74 | 8.35$\pm$0.82
TKD | 33.15$\pm$0.45 | 21.05$\pm$0.39 | 27.29$\pm$0.76 | 23.20$\pm$2.46 | 37.31$\pm$1.36 | 11.83$\pm$1.31 | 23.94$\pm$2.14 | 10.20$\pm$1.09
+ours | 34.58$\pm$0.96 | 17.27$\pm$0.54 | 28.71$\pm$1.06 | 17.37$\pm$1.78 | 38.41$\pm$1.52 | 9.16$\pm$0.62 | 28.10$\pm$1.97 | 9.56$\pm$0.65
ANCL | 32.84$\pm$0.78 | 27.24$\pm$1.47 | 26.98$\pm$0.60 | 32.45$\pm$1.18 | 37.74$\pm$0.60 | 17.28$\pm$2.59 | 27.95$\pm$1.98 | 20.91$\pm$0.92
+ours | 34.59$\pm$0.75 | 23.64$\pm$0.76 | 29.18$\pm$0.71 | 26.50$\pm$1.34 | 40.10$\pm$1.39 | 15.12$\pm$1.01 | 32.60$\pm$1.45 | 17.94$\pm$0.39
c) ImageNet100 | GKD | 54.62$\pm$0.52 | 25.95$\pm$0.11 | 42.82$\pm$0.58 | 35.39$\pm$0.88 | 57.94$\pm$0.90 | 14.47$\pm$0.83 | 21.91$\pm$0.06 | 9.29$\pm$0.69
+ours | 55.82$\pm$0.61 | 20.52$\pm$0.24 | 45.88$\pm$0.79 | 23.25$\pm$0.62 | 57.18$\pm$0.45 | 17.24$\pm$0.39 | 22.31$\pm$0.64 | 11.28$\pm$0.98
MKD | 54.01$\pm$0.01 | 28.19$\pm$0.61 | 43.39$\pm$0.66 | 34.25$\pm$0.81 | 56.18$\pm$0.90 | 14.94$\pm$0.17 | 26.07$\pm$0.29 | 16.00$\pm$0.06
+ours | 56.02$\pm$0.20 | 18.60$\pm$0.76 | 46.18$\pm$0.54 | 19.14$\pm$0.79 | 52.05$\pm$0.24 | 14.23$\pm$0.46 | 22.25$\pm$0.13 | 11.94$\pm$1.23
TKD | 55.70$\pm$0.49 | 23.55$\pm$0.35 | 44.75$\pm$0.28 | 32.16$\pm$0.14 | 54.72$\pm$0.86 | 10.16$\pm$0.34 | 19.32$\pm$0.23 | 9.67$\pm$0.61
+ours | 56.23$\pm$0.70 | 18.09$\pm$0.26 | 46.45$\pm$0.42 | 19.55$\pm$0.30 | 53.85$\pm$0.39 | 13.15$\pm$0.16 | 22.55$\pm$0.83 | 9.96$\pm$0.28
ANCL | 55.81$\pm$0.41 | 27.13$\pm$0.50 | 44.94$\pm$0.63 | 34.24$\pm$1.14 | 60.97$\pm$0.62 | 15.72$\pm$0.14 | 29.19$\pm$0.76 | 16.12$\pm$0.27
+ours | 57.41$\pm$0.22 | 21.54$\pm$0.50 | 47.49$\pm$0.35 | 22.61$\pm$0.40 | 59.22$\pm$0.26 | 17.34$\pm$0.53 | 29.64$\pm$0.50 | 14.67$\pm$0.84

Table 1: Standard Knowledge Distillation (KD) techniques with and without our
Teacher Adaptation (TA) method on different splits of a) CIFAR100, b)
TinyImageNet200 and c) ImageNet100. TA is generally beneficial to the CIL
process, and the improvements occur most consistently in scenarios with a long
sequence of equally sized tasks, where the initial model learns a weaker
feature extractor.

##  4 Experiments

###  4.1 Experimental setup

We conduct experiments on common continual learning benchmarks, such as
CIFAR100 [26], TinyImageNet200 [1] and ImageNet-Subset [11]. We measure
models’ adaptability to large shifts in data distributions on DomainNet [35]
dataset. Additionally, following FACIL [32] we construct fine-grained
classification benchmark using Oxford Flowers [34], MIT Indoor Scenes [39],
CUB-200-2011 Birds [51], Stanford Cars [25], FGVC Aircraft [31], and Stanford
Actions [58]. Finally, we also introduce a corrupted CIFAR100 setting where
data in every other task contains noise of varying severity, which allows us
to measure the impact of TA under varying and controllable degrees of data
shift.

We create CIL scenarios by splitting the classes in each dataset into disjoint
tasks. We experiment with two particular types of settings: the first type of
setting is built by splitting the classes in the dataset into tasks containing
an equal number of classes, while the other simulates pretaining the network
and uses half of the classes as a larger first task, with subsequent tasks
composed of the evenly split remaining classes.

For all experiments, we use FACIL framework provided by Masana et al. [32].
For experiments on CIFAR100, we keep the class order from iCaRL [41] and we
use ResNet32 [15]. For TinyImageNet, following [22], we rescale images to
32x32 pixels and also use ResNet32. For the other datasets, we use ResNet18
[15]. We always use the same hyperparameters for all variants within the
single KD method unless stated otherwise, we report the results averaged over
three runs with different random seeds.

In every setup, we train the network on each new task for 200 epochs with
batch size 128. We use SGD optimizer without momentum or weight decay, with a
learning rate scheduler proposed by Zhou et al. [63], where the initial
learning rate of 0.1 is decreased 10x after 60th, 120th and 160th epoch. For
experiments conducted on CIFAR100 and TinyImageNet200 in Table 1 we also
employ a warmup phase [27] for the new classification head. In the Appendix,
we provide the ablation study of the warmup with different benchmarks,
alongside the details of its implementation and discussion on the method.
Additionally, we provide the evaluation of our method with different model
architectures and batch sizes.

Evaluation metrics. The average incremental accuracy at task $k$ is defined as
$A_{k}\\!=\\!\frac{1}{k}\sum_{j=1}^{k}a_{k,j}$, where ${a_{k,j}}\in[0,1]$ be
the accuracy of the $j$-th task ($j\leq k$) after training the network
sequentially for $k$ tasks [4]. Overall average incremental accuracy
$Acc_{Inc}$ is the mean value from all tasks. We also report average
forgetting as defined in [10], while the $Forg_{Inc}$ is similarly the mean
value from all tasks. We provide results with additional metrics such as final
accuracy $Acc_{Final}$ and final forgetting $Forg_{Final}$ in the Appendix.

| 6 tasks, 20 classes each | 12 tasks, 10 classes each | 24 tasks, 5 classes each
---|---|---|---
| $Acc_{Inc}\uparrow$ | $Forg_{Inc}\downarrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Inc}\downarrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Inc}\downarrow$
GKD |  56.03$\pm$3.75 |  30.23$\pm$0.72 |  44.03$\pm$1.53 |  42.70$\pm$2.32 |  30.14$\pm$4.00 |  51.95$\pm$2.28
+ours |  57.24$\pm$1.79 |  33.12$\pm$1.76 |  50.80$\pm$1.27 |  37.58$\pm$3.01 |  42.86$\pm$1.82 |  39.01$\pm$1.55
MKD |  60.12$\pm$1.59 |  26.42$\pm$1.60 |  45.19$\pm$2.91 |  43.81$\pm$1.56 |  32.61$\pm$0.55 |  55.91$\pm$1.77
+ours |  55.77$\pm$2.23 |  31.43$\pm$2.28 |  51.16$\pm$1.84 |  34.89$\pm$2.60 |  45.49$\pm$1.82 |  34.48$\pm$0.89
TKD |  56.69$\pm$3.36 |  33.86$\pm$1.36 |  46.28$\pm$1.42 |  43.38$\pm$0.82 |  32.17$\pm$2.71 |  51.72$\pm$1.95
+ours |  57.99$\pm$1.88 |  33.79$\pm$1.80 |  51.66$\pm$1.63 |  35.43$\pm$2.75 |  43.95$\pm$2.60 |  33.28$\pm$1.86
Table 2: Average task-agnostic accuracy and forgetting for KD-based CL methods
on fine-grained classification datasets.

| 6 tasks | 6 tasks, pretrained | 12 tasks | 12 tasks, pretrained
---|---|---|---|---
| $Acc_{Inc}\uparrow$ | $Forg_{Inc}\downarrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Inc}\downarrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Inc}\downarrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Inc}\downarrow$
GKD | 18.63$\pm$0.27 | 23.27$\pm$0.26 | 43.27$\pm$0.10 | 36.83$\pm$0.88 | 14.45$\pm$0.25 | 29.04$\pm$0.37 | 35.98$\pm$0.96 | 43.00$\pm$1.66
+ours | 19.55$\pm$0.42 | 27.22$\pm$0.21 | 43.52$\pm$0.17 | 34.98$\pm$0.19 | 16.25$\pm$0.46 | 33.03$\pm$0.19 | 38.89$\pm$0.52 | 41.10$\pm$0.42
TKD | 19.12$\pm$0.26 | 26.66$\pm$0.38 | 42.42$\pm$0.10 | 40.83$\pm$1.11 | 16.31$\pm$0.55 | 37.32$\pm$0.87 | 38.15$\pm$0.22 | 42.28$\pm$0.50
+ours | 19.57$\pm$0.10 | 29.96$\pm$0.05 | 42.75$\pm$0.14 | 41.79$\pm$0.65 | 16.74$\pm$0.55 | 33.32$\pm$0.47 | 39.06$\pm$0.33 | 40.19$\pm$0.85
MKD | 18.74$\pm$0.52 | 19.10$\pm$0.40 | 45.70$\pm$0.30 | 27.48$\pm$0.60 | 13.45$\pm$0.53 | 27.23$\pm$0.98 | 39.14$\pm$0.21 | 36.53$\pm$1.00
+ours | 18.04$\pm$0.16 | 22.70$\pm$0.25 | 42.91$\pm$0.04 | 29.43$\pm$0.04 | 15.30$\pm$0.35 | 28.71$\pm$0.26 | 37.84$\pm$0.35 | 34.09$\pm$0.39
ANCL | 19.58$\pm$0.46 | 25.63$\pm$0.20 | 42.90$\pm$0.84 | 37.28$\pm$1.86 | 14.82$\pm$0.41 | 33.46$\pm$0.40 | 33.34$\pm$0.55 | 49.05$\pm$0.65
+ours | 20.34$\pm$0.40 | 30.73$\pm$0.44 | 42.67$\pm$0.51 | 38.56$\pm$1.24 | 17.19$\pm$0.06 | 37.40$\pm$0.38 | 35.81$\pm$0.18 | 43.84$\pm$0.23

Table 3: Average task-agnostic accuracy and forgetting for KD and TA under
significant semantic drift on DomainNet. We test scenarios with 6 tasks of 50
classes and 12 tasks of 25 classes, both when training from scratch and
starting from pretrained model. Aside from MKD, TA generally leads to better
results.

###  4.2 Standard CIL benchmarks

We evaluate knowledge distillation approaches described in Section 3.1 on the
standard CIL benchmarks CIFAR100, TinyImageNet200 and ImageNet100, using
different class splits. We present the results in Table 1 a), b) and c)
respectively. We also provide results for more settings and ablation study of
our method on those datasets in the Appendix.

In most settings, we observe that our method improves upon the baseline
knowledge distillation. We notice that the improvement TA is generally more
significant in settings with a larger number of tasks and an equal split of
the classes. In settings with half the classes presented in the first task,
the gains from TA are sometimes not that visible, as in this case the initial
model already learns a good feature extractor, and the distribution of its
normalization statistics after the first task is a better approximation of the
statistics for the whole dataset. TA sometimes underperforms with MKD, which
might be caused by the fact that the loss formula of MKD uses sigmoid
function, and the differences between the probabilities for KD loss are
insignificant if the values of logits are not small and centered around zero,
which is not guaranteed without imposing additional learning constraints.

###  4.3 TA under severe distribution shifts

Motivated by the continual learning settings in which the data distribution
changes significantly across the tasks, we conduct a series of experiments to
empirically verify the benefits of our method.

####  4.3.1 Fine-grained classification datasets

We evaluate TA on fine-grained classification tasks using six datasets:
Stanford Actions, FGVC Aircraft, Stanford Cars, CUB-200-2011 Birds, MIT Indoor
Scenes and Oxford Flowers. We create CIL tasks by randomly sampling a subset
of classes from each dataset, in the above-mentioned order. We sample the
classes without replacement, and to obtain the settings with 12 or 24 tasks we
repeat the procedure. For this set of experiments, we start from ResNet18
checkpoint pretrained on ImageNet.

We conduct experiments using splits of 24 tasks with 5 classes each, 12 tasks
with 10 classes each, and 6 tasks with 20 classes each. We show the results in
Table 2. Consistently with the results from Section 4.2, our method generally
improves upon the base KD, with the improvements being more visible on the
longer tasks.

We omit ANCL method from our analysis, as we were unable to obtain
sufficiently good results with its official implementation. We provide the
results of additional experiments conducted with reverse order of datasets and
with the full datasets used as a single task in the Appendix.

####  4.3.2 Large domain shifts with DomainNet

Figure 3:  Average incremental accuracy for standard KD and our method of TA
under varying strength of data shift on splits of CIFAR100. As the noise
strengthens, the gap between TA and standard KD widens, indicating that our
method leads to more robust learning in case of data shifts. We obtain data
shifts by adding noise of varying strength to every other task, using the
Gaussian noise and noise severity scale proposed by Michaelis et al. [33].

To verify the effectiveness of teacher adaptation for continual learners under
significant data distribution shifts, we use DomainNet [35] as our evaluation
dataset. DomainNet consists of images from 6 domains and 345 classes. We
select the first 50 classes and create each task from a different domain,
resulting in more severe data drift between tasks in CIL. This allows us to
measure how well the models can adapt to new data distributions. We use
ResNet18 and compare two settings: training from scratch and from starting
from the model pretrained on ImageNet. Table 3 shows the results of our
experiments. Consistently with the results from previous Sections, we find
that, aside from MKD, TA generally performs better than the baselines, and the
differences are more visible when training on 12 tasks, where the model is
exposed to more changes in the data distribution.

####  4.3.3 Varying the strength of the distribution shift

We create CIL settings with controllable levels of data distribution shift
between subsequent tasks by corrupting every other task. We split CIFAR100
into 5, 10, and 20 tasks of equal size and add Gaussian noise to every other
task, so that in subsequent tasks the data distribution changes from clean to
noisy or vice versa. We obtain varying strength of distribution shift by using
different levels of noise severity, following the methodology of Michaelis et
al. [33].

We show the results of this experiment in Figure 3. We see that as the noise
severity increases, the gap between standard KD and TA widens, indicating that
our method is better suited to more challenging scenarios of learning under
extreme data distribution shifts.

###  4.4 Detailed analysis

####  4.4.1 Alternatives to batch normalization

We conduct a series of ablation experiments on CIFAR100 split into 10 tasks to
justify the validity of our method over other potential solutions for
adaptation of batch normalization layers. The results of those experiments are
shown in Table 4. We compare the following settings: 1) standard training with
batch normalization statistics from the previous task fixed in the teacher
model, but updated in the student model, 2) batch normalization layers
removed, 3) batch normalization statistics fixed in both models after learning
the first task, 4) batch normalization layers replaced with LayerNorm [5] or
5) GroupNorm [57] layers, and finally 6) our solution of Teacher Adaptation.

$clip=100$ | $\lambda=5$ | $\lambda=10$
---|---|---
| $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$
1) GKD | 25.47$\pm$0.57 | 41.59$\pm$0.32 | 27.96$\pm$0.34 | 42.28$\pm$0.67
2) -BN | 0.33$\pm$1.15 | 2.01$\pm$2.67 | 0.33$\pm$1.15 | 2.85$\pm$3.81
3) fix BN | - | - | - | -
4) -BN +LN | 21.94$\pm$0.95 | 34.7$\pm$0.48 | 22.76$\pm$1.05 | 34.48$\pm$0.15
5) -BN +GN | 21.92$\pm$0.46 | 32.15$\pm$0.16 | 22.01$\pm$0.82 | 31.71$\pm$0.35
6) +TA | 31.39$\pm$0.17 | 44.98$\pm$0.38 | 31.85$\pm$0.10 | 44.06$\pm$0.69
$clip=1$ | $\lambda=5$ | $\lambda=10$
| $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$
1) GKD | 20.80$\pm$0.56 | 34.28$\pm$0.24 | 27.96$\pm$0.34 | 42.28$\pm$0.67
2) -BN | 19.47$\pm$0.18 | 29.83$\pm$0.53 | 0.33$\pm$1.15 | 2.85$\pm$3.81
3) fix BN | 20.21$\pm$0.31 | 32.07$\pm$0.20 | - | -
4) -BN +LN | 18.49$\pm$1.41 | 30.39$\pm$0.72 | 22.76$\pm$1.05 | 34.48$\pm$0.15
5) -BN +GN | 16.17$\pm$0.89 | 32.15$\pm$0.16 | 15.73$\pm$1.01 | 25.07$\pm$1.10
6) +TA | 24.19$\pm$0.90 | 36.13$\pm$0.24 | 31.85$\pm$0.10 | 44.06$\pm$0.69

Table 4: Results for different solutions to the problem of diverging batch
normalization layers when using knowledge distillation in continual learning.
We use GKD with different $\lambda$ and gradient clipping values. We compare
the baseline with variants without batch normalization layers, with batch
normalization statistics fixed after the first task and with batch
normalization layers replaced with LayerNorm or GroupNorm. "-" indicates that
training crashes due to instability. TA is the only solution that improves
upon the baseline.

Fixing or removing BatchNorms leads to unstable training. This can be
partially fixed by setting a high gradient clipping value or lowering the
lambda parameter, but both solutions lead to much worse network performance.
Different normalization layers enable stable training, but ultimately converge
to much worse solutions than the network with BatchNorm. Our solution is the
only one that improves over different values of $\lambda$ and does not require
controlling the gradients by clipping the high values.

####  4.4.2 Alternative methods of teacher adaptation.

We study alternative methods of adapting the teacher model and try pretraining
($P$) or continuously training ($CT$) the teacher model. For pertaining, we
train the teacher on the new data in isolation for a few epochs before the
training of the main model. During continuous training, we update the teacher
alongside the main model using the same batches of new data. With both
approaches, we set a lower learning rate for the teacher. We conduct those
experiments by training either the full teacher model ($FM$) or only its batch
normalization layers ($BN$). Finally, to isolate the impact of changing batch
normalization statistics and training model parameters, we repeat all the
experiments with fixed batch normalization statistics ($fix\ BN$).

We conduct our experiment on CIFAR100 split into 10 tasks and present the
results in Table 5. Alternative solutions perform within the standard
deviation of TA with tuned hyperparameters, but the values of the
hyperparameters for those models (described in the Appendix) are also very
small, indicating that the teacher model doesn’t change much. Upon closer
study, we find that it’s mostly batch normalization statistics that change
throughout the training. Therefore, knowing that other successful methods from
test-time adaptation [52] use a similar approach, we continue with Teacher
Adaptation based on batch normalization layers, as it does not require any
hyperparameter tuning or additional pretraining epochs.

Method | $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$
---|---|---|---|---
Base | 27.53$\pm$0.15 | 42.22$\pm$0.38 | 31.28$\pm$1.64 | 23.11$\pm$1.58
P-FM | 31.54$\pm$0.67 | 43.46$\pm$0.72 | 24.18$\pm$1.17 | 20.80$\pm$1.51
+fix BN | 28.02$\pm$0.60 | 42.33$\pm$0.53 | 29.91$\pm$1.27 | 22.66$\pm$0.95
P-BN | 31.16$\pm$0.54 | 43.64$\pm$0.77 | 24.44$\pm$0.96 | 20.13$\pm$0.75
+fix BN | 27.62$\pm$0.48 | 42.12$\pm$0.38 | 29.95$\pm$1.64 | 22.50$\pm$0.95
CT-FM | 31.37$\pm$0.94 | 43.38$\pm$0.77 | 24.34$\pm$1.37 | 20.93$\pm$1.58
+fix BN | 28.17$\pm$0.49 | 42.29$\pm$0.42 | 29.79$\pm$1.02 | 22.55$\pm$0.67
CT-BN | 31.35$\pm$0.63 | 43.69$\pm$0.76 | 24.29$\pm$0.61 | 20.23$\pm$0.59
+fix BN | 27.33$\pm$0.50 | 42.09$\pm$0.45 | 30.20$\pm$1.73 | 22.50$\pm$0.85
TA | 32.15$\pm$0.12 | 44.31$\pm$0.26 | 23.55$\pm$0.51 | 19.85$\pm$0.93

Table 5: Ablation study of different ways to adapt the teacher model. Our
method achieves the best results while requiring no additional
hyperparameters. We try teacher adaptation during pretraining (P) and
continuous training (CT). We train either full model (FM) or only batch
normalization layers (BN). _fix BN_ indicates fixed BN statistics.

##  5 Conclusions

We propose Teacher Adaptation, a simple yet effective method to improve the
performance of knowledge distillation-based methods in exemplar-free class-
incremental learning. Our method continuously updates the teacher network by
adjusting batch normalization statistics during learning a new task both for
the currently learning model and the teacher model saved after learning the
previous tasks. This mitigates the changes in the model caused by knowledge
distillation loss that arise as the current learner is continuously trying to
compensate for the modified normalization statistics. We further improve the
stability of the model by introducing a warm-up phase at the beginning of the
task, where a new classification head is trained in isolation before
finetuning the whole model. The warm-up phase ensures that the initialization
of the weights is not random in the initial phases of training, and reduces
the gradient updates to the whole model. We conduct experiments with Teacher
Adaptation on several class-incremental benchmarks and show that it
consistently improves the results for different knowledge distillation-based
methods in an exemplar-free setting. Moreover, our method can be easily added
to the existing class-incremental learning solutions and induces only a slight
computational overhead.

##### Discussion

Since the introduction of Learning without Forgetting, KD-based methods have
emerged as effective solutions to mitigate forgetting in CIL models. Several
approaches, such as iCaRL, EEIL, BiC, LUCIR, and SSIL, have integrated KD with
exemplars, which helps maintain a balanced discrepancy between the teacher and
student models. In scenarios where a sufficient number of exemplars are
available, teacher adaptation may not be required, as their presence in the
training data mitigates the divergence between the normalization statistics of
the subsequent tasks. Nevertheless, our research is dedicated exclusively to
the exemplar-free setting, in which we investigate techniques that do not rely
on storing exemplars. To the best of our knowledge, we are the first to
propose the adaptation of the teacher model within the context of KD-based
exemplar-free CIL.

##### Impact

Our method focuses on exemplar-free scenarios, and therefore we alleviate the
issues with storing potentially confidential, private, or sensitive data.
However, we recognize that machine learning algorithms can be harmful if
applied carelessly, and we encourage practitioners to carefully check training
data and the models to ensure that the results of their work do not perpetuate
biases or discriminate against any minority.

All our work was conducted using publicly available datasets and open-source
code. To allow other researchers to build on our work and validate the
results, we will share the code for the experiments in this paper on GitHub
upon acceptance.

## Acknowledgements

Filip Szatkowski and Tomasz Trzcinski are supported by National Centre of
Science (NCP, Poland) Grant No. 2022/45/B/ST6/02817. Tomasz Trzciński is also
supported by NCP Grant No. 2020/39/B/ST6/01511.

## References

  * [1] Zoheb Abai and Nishad Rajmalwar.  Densenet models for tiny imagenet classification.  arXiv preprint arXiv:1904.10429, 2019.
  * [2] Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang, Hyojun Kim, and Taesup Moon.  Ss-il: Separated softmax for incremental learning.  In Proceedings of the IEEE/CVF International conference on computer vision, pages 844–853, 2021.
  * [3] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars.  Memory aware synapses: Learning what (not) to forget, 2018.
  * [4] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars.  Expert gate: Lifelong learning with a network of experts.  In CVPR, pages 3366–3375, 2017.
  * [5] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [6] Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Ha, and Jonghyun Choi.  Rainbow memory: Continual learning with a memory of diverse samples.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8218–8227, 2021.
  * [7] Tudor Berariu, Wojciech Czarnecki, Soham De, Jorg Bornschein, Samuel Smith, Razvan Pascanu, and Claudia Clopath.  A study on the plasticity of neural networks.  arXiv preprint arXiv:2106.00042, 2021.
  * [8] Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari.  End-to-end incremental learning.  In Proceedings of the European conference on computer vision (ECCV), pages 233–248, 2018.
  * [9] Sungmin Cha, Sungjun Cho, Dasol Hwang, Sunwon Hong, Moontae Lee, and Taesup Moon.  Rebalancing batch normalization for exemplar-based class-incremental learning, 2023.
  * [10] Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr.  Riemannian walk for incremental learning: Understanding forgetting and intransigence.  In ECCV, pages 532–547, 2018.
  * [11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248–255, 2009.
  * [12] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa.  Learning without memorizing.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5138–5146, 2019.
  * [13] Fei Ding, Yin Yang, Hongxin Hu, Venkat Krovi, and Feng Luo.  Multi-level knowledge distillation via knowledge alignment and correlation, 2021.
  * [14] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle.  Podnet: Pooled outputs distillation for small-tasks incremental learning.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX 16, pages 86–102. Springer, 2020.
  * [15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [16] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi.  A comprehensive overhaul of feature distillation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1921–1930, 2019.
  * [17] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [18] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin.  Learning a unified classifier incrementally via rebalancing.  In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 831–839, 2019.
  * [19] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al.  Searching for mobilenetv3.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314–1324, 2019.
  * [20] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. pmlr, 2015.
  * [21] Ahmet Iscen, Thomas Bird, Mathilde Caron, Alireza Fathi, and Cordelia Schmid.  A memory transformer network for incremental learning, 2022.
  * [22] Sanghwan Kim, Lorenzo Noci, Antonio Orvieto, and Thomas Hofmann.  Achieving a better stability-plasticity trade-off via auxiliary networks in continual learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11930–11939, 2023.
  * [23] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  * [24] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton.  Similarity of neural network representations revisited.  In International Conference on Machine Learning, pages 3519–3529. PMLR, 2019.
  * [25] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In 2013 IEEE International Conference on Computer Vision Workshops, pages 554–561, 2013.
  * [26] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  2009\.
  * [27] Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang.  Fine-tuning can distort pretrained features and underperform out-of-distribution.  In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
  * [28] Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee.  Overcoming catastrophic forgetting with unlabeled data in the wild.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 312–321, 2019.
  * [29] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017.
  * [30] Xinge Ma, Jin Wang, Liang-Chih Yu, and Xuejie Zhang.  Knowledge distillation with reptile meta-learning for pretrained language model compression.  In Proceedings of the 29th International Conference on Computational Linguistics, pages 4907–4917, 2022.
  * [31] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi.  Fine-grained visual classification of aircraft.  Technical report, 2013.
  * [32] Marc Masana, Xialei Liu, Bartłomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer.  Class-incremental learning: survey and performance evaluation on image classification.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  * [33] Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander S. Ecker, Matthias Bethge, and Wieland Brendel.  Benchmarking robustness in object detection: Autonomous driving when winter is coming.  arXiv preprint arXiv:1907.07484, 2019.
  * [34] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722–729, 2008.
  * [35] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1406–1415, 2019.
  * [36] Quang Pham, Chenghao Liu, and Steven Hoi.  Continual normalization: Rethinking batch normalization for online continual learning.  arXiv preprint arXiv:2203.16102, 2022.
  * [37] Ameya Prabhu, Philip HS Torr, and Puneet K Dokania.  Gdumb: A simple approach that questions our progress in continual learning.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 524–540. Springer, 2020.
  * [38] Wang Qi, Yu-Ping Ruan, Yuan Zuo, and Taihao Li.  Parameter-efficient tuning on layer normalization for pre-trained language models.  arXiv preprint arXiv:2211.08682, 2022.
  * [39] Ariadna Quattoni and Antonio Torralba.  Recognizing indoor scenes.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 413–420, 2009.
  * [40] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi.  Efficient parametrization of multi-domain deep neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8119–8127, 2018.
  * [41] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.  icarl: Incremental classifier and representation learning.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017.
  * [42] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne.  Experience replay for continual learning.  Advances in Neural Information Processing Systems, 32, 2019.
  * [43] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [44] Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry.  How does batch normalization help optimization?, 2019.
  * [45] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge.  Improving robustness against common corruptions by covariate shift adaptation.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 11539–11551. Curran Associates, Inc., 2020.
  * [46] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [47] James Seale Smith, Junjiao Tian, Shaunak Halbe, Yen-Chang Hsu, and Zsolt Kira.  A closer look at rehearsal-free continual learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2409–2419, 2023.
  * [48] Leslie N Smith and Nicholay Topin.  Super-convergence: Very fast training of neural networks using large learning rates.  In Artificial intelligence and machine learning for multi-domain operations applications, volume 11006, pages 369–386. SPIE, 2019.
  * [49] Michael Sun, Ananya Kumar, Divyam Madaan, and Percy Liang.  Improving representational continuity via continued pretraining.  arXiv preprint arXiv:2302.13289, 2023.
  * [50] Gido M Van de Ven and Andreas S Tolias.  Three scenarios for continual learning.  arXiv preprint arXiv:1904.07734, 2019.
  * [51] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * [52] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno A. Olshausen, and Trevor Darrell.  Tent: Fully test-time adaptation by entropy minimization.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  * [53] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al.  Dualprompt: Complementary prompting for rehearsal-free continual learning.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVI, pages 631–648. Springer, 2022.
  * [54] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister.  Learning to prompt for continual learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139–149, 2022.
  * [55] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu.  Large scale incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [56] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu.  Large scale incremental learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 374–382, 2019.
  * [57] Yuxin Wu and Kaiming He.  Group normalization, 2018.
  * [58] Bangpeng Yao, Xiaoye Jiang, Aditya Khosla, Andy Lai Lin, Leonidas Guibas, and Li Fei-Fei.  Human action recognition by learning bases of action attributes and parts.  In 2011 International Conference on Computer Vision, pages 1331–1338, 2011.
  * [59] Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence, 2017.
  * [60] Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C. C. Jay Kuo.  Class-incremental learning via deep model consolidation, 2020.
  * [61] Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry P. Heck, Heming Zhang, and C.-C. Jay Kuo.  Class-incremental learning via deep model consolidation.  In IEEE Winter Conference on Applications of Computer Vision, WACV 2020, Snowmass Village, CO, USA, March 1-5, 2020, pages 1120–1129. IEEE, 2020.
  * [62] Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia.  Maintaining discrimination and fairness in class incremental learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13208–13217, 2020.
  * [63] Da-Wei Zhou, Fu-Yun Wang, Han-Jia Ye, and De-Chuan Zhan.  Pycil: A python toolbox for class-incremental learning, 2023.
  * [64] Wangchunshu Zhou, Canwen Xu, and Julian J. McAuley.  BERT learns to teach: Knowledge distillation with meta learning.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7037–7049. Association for Computational Linguistics, 2022.

Appendix

This document contains additional experimental results and analysis.

##  Appendix A ANCL implementation with TA

We follow the provided implementation of ANCL [22] based on the FACIL [32]
framework. We only apply the adaptation to the main teacher network fitted on
the previous data, and, in cases where we use warmup, we only apply it to the
new head of the main network. The auxiliary network does not undergo the
teacher adaptation process, as it is already fit for the current task, and we
do not see a reason to apply the warmup to this network. We keep the same
training schedule and hyperparameters as described in Section 4.1.

##  Appendix B Warmup phase

As mentioned in Section 4.1, we experimented with applying the warmup phase, a
technique that recently appeared in several works [27, 7, 49]. With this
method, before training the full student model on the new data, we first
finetune the classification head added for this new task in isolation, while
the rest of the network is kept frozen. Compared to the standard practice of
finetuning the full student model on the new task, starting with a warmup
phase reduces the initial cross-entropy loss at the start of the new task, and
avoids overwriting the knowledge from the previous tasks with large gradient
updates caused by random classification head initialization.

For training the new head during the warmup phase, we use SGD optimizer and
OneCycle scheduler [48] with cosine annealing, maximum learning rate of 0.1
and the number of epochs of increasing learning rate set as 40. We train the
new head for 200 epochs with early stopping and freeze updates of batch
normalization statistics in the model during the training.

In Appendix D, we evaluate different CL benchmarks with TA and warmup
separately. We generally find that the warmup works complementary to our
method when we use smaller networks such as ResNet32 and train on smaller
datasets. We hypothesize that for the settings with a larger dataset and a
larger network better initialization of the new head does not translate to
better performance in CIL because the overall magnitude of updates required to
train the model in such settings is greater and leads to overwriting the
initialization over the course of the training.

##  Appendix C Ablation studies of Teacher Adaptation

###  C.1 Teacher Adaptation with different batch sizes.

Larger batch sizes should lead to better estimation of the global statistics
in the batch normalization layer, which, in theory, should lead to reduced
effectiveness of TA. To investigate the impact of the batch size on our
method, we train the model on CIFAR100 split into 10 tasks with different
batch sizes, tuning the learning rate and $\lambda$ for each batch size. We
present the results in Table 6. Regardless of the batch size, TA constantly
improves upon standard training, and the difference between the two approaches
does not scale with batch size.

Batch size | $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$
---|---|---|---|---
32 | 28.94$\pm$0.29 | 44.24$\pm$0.27 | 32.57$\pm$0.92 | 23.89$\pm$1.19
+TA | 32.58$\pm$0.80 | 45.69$\pm$0.35 | 25.44$\pm$0.94 | 20.92$\pm$0.53
64 | 28.92$\pm$0.57 | 43.89$\pm$0.27 | 30.48$\pm$1.11 | 21.70$\pm$0.39
+TA | 32.15$\pm$1.24 | 44.71$\pm$0.57 | 23.43$\pm$1.97 | 19.16$\pm$0.67
128 | 28.27$\pm$0.44 | 42.52$\pm$0.76 | 29.59$\pm$0.92 | 22.26$\pm$0.31
+TA | 31.92$\pm$0.86 | 44.09$\pm$0.97 | 22.65$\pm$1.32 | 19.41$\pm$0.60
256 | 25.13$\pm$0.46 | 40.44$\pm$0.11 | 40.06$\pm$0.38 | 31.39$\pm$0.98
+TA | 30.56$\pm$0.68 | 43.28$\pm$0.23 | 28.24$\pm$0.64 | 23.94$\pm$0.59
512 | 23.01$\pm$0.89 | 37.44$\pm$0.34 | 40.57$\pm$1.20 | 32.03$\pm$0.69
+TA | 27.52$\pm$1.03 | 39.86$\pm$0.45 | 29.31$\pm$0.86 | 25.03$\pm$0.55

Table 6: Teacher adaptation with different batch sizes. Our method improves
upon the standard training regardless of the batch size.

###  C.2 Different CNN architectures with Teacher Adaptation.

To test the robustness of our method, in addition to experiments with ResNets,
we test it with other standard convolutional neural networks. We evaluate our
method with MobileNetV2 [43], MobileNetV3-small [19] and VGG11 [46]. We use
CIFAR100 and ImageNet100 split into 10 equally sized tasks and train the
networks as described in Section 4.1. We show the results of those experiments
in the Table 7. The final results obtained with tested networks are lower than
with our standard baselines, as do not tune their hyperparameters extensively,
but we still notice that using TA constantly improves the results, regardless
of the architecture we use.

CIFAR100
---
Network | $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$
MobilenetV2 | 12.47$\pm$0.69 | 25.71$\pm$1.05 | 49.86$\pm$0.61 | 41.23$\pm$1.25
+TA | 20.62$\pm$1.01 | 31.91$\pm$0.57 | 26.66$\pm$1.34 | 22.31$\pm$0.56
MobilenetV3-s | 16.59$\pm$0.09 | 29.86$\pm$0.06 | 38.89$\pm$0.42 | 30.99$\pm$0.12
+TA | 21.22$\pm$0.91 | 32.24$\pm$0.60 | 20.98$\pm$0.13 | 16.84$\pm$1.19
ResNet32 | 28.27$\pm$0.44 | 42.52$\pm$0.76 | 29.59$\pm$0.92 | 22.26$\pm$0.31
+TA | 31.92$\pm$0.86 | 44.09$\pm$0.97 | 22.65$\pm$1.32 | 19.41$\pm$0.60
VGG11 | 17.10$\pm$0.12 | 34.34$\pm$0.39 | 62.36$\pm$0.62 | 52.99$\pm$0.97
+TA | 24.87$\pm$0.35 | 42.50$\pm$0.15 | 43.96$\pm$0.62 | 32.92$\pm$0.51
ImageNet100
Network | $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$
ResNet18 | 40.99$\pm$0.45 | 54.82$\pm$0.56 | 31.99$\pm$0.87 | 25.90$\pm$0.65
+TA | 42.69$\pm$0.53 | 55.84$\pm$0.51 | 25.26$\pm$0.23 | 20.43$\pm$0.33
VGG11 | 24.35$\pm$0.32 | 44.72$\pm$0.50 | 61.62$\pm$0.10 | 46.73$\pm$0.31
+TA | 30.67$\pm$0.31 | 48.81$\pm$0.29 | 50.04$\pm$0.76 | 37.60$\pm$0.56

Table 7: Teacher Adaptation with different network architectures on CIFAR100
and ImageNet100 split into 10 equally sized tasks.

###  C.3 Batch normalization statistics investigation.

Figure 4: TA impact on batch normalization statistics. Left: KLD between the
normalization statistics of the teacher and the student at the end of the
training of all the tasks. Right: KLD between the normalization statistics of
the model at the end of each task and the model trained on the first task.

We investigate the batch normalization statistics in the teacher and student
model throughout CIL on CIFAR100 split into 10 equally sized tasks. To measure
the divergence between the normalization statistics in the models, we compute
the average Kullback-Leibler divergence (KLD) between the distributions of
normalization statistics in every batch normalization layer. We show the
results of this analysis in the Figure 4. Specifically, we measure KLD between
the final teacher and student model at the end of each task, as well as the
difference between the final student model trained for each task and the model
initially learned after the first task. Applying TA leads to reduced KLD
between the statistics in both teacher and student, and also student and
initial model. This proves that our method leads to more stable
representations throughout the training.

Figure 5: Task confusion matrix after learning all ten tasks on CIFAR100/10
for (left) base GKD and (right) GKD+TA. We see that TA leads to a model that
is better at distinguishing between the first tasks and exhibits lower recency
bias.

###  C.4 Hyperparameters for alternative adaptation methods ablation

The best results reported in Table 5 were obtained with teacher learning rate
of 1e-7. For variants with teacher pertaining, the best results were obtained
with training for 5 epochs. We keep $\lambda$ fixed at 10 for all methods
reported in the Table. While all variants that allow adaptation of batch norm
statistics improve upon the baseline, the values of best hyperparameters are
generally small, suggesting that the main source of improvement in all the
methods comes from changes in normalization. This is further supported by the
fact that all improvements vanish when normalization statistics are fixed.

###  C.5 Task recency bias with TA

We also conduct additional analysis of our method of Teacher Adaptation (TA)
to understand the mechanism with which it improves upon the standard knowledge
distillation. At Figure 5, we analyze task confusion matrices of standard
knowledge distillation (LwF) and its extension with TA. We find that applying
TA results in a model that is better at distinguishing between the tasks, and
generally exhibits lower recency bias.

We hypothesize that the lower KD loss that we observe when using TA results in
smaller updates to the model, so the difference between the magnitudes of
logits learned for different tasks is smaller. Therefore, Teacher Adaptation
helps to alleviate the recency bias in class-incremental learning.

##  Appendix D Additional experiments for different CL benchmarks

We provide the results of additional experiments conducted with our method and
warmup. To shorten the notation, we denote the total number of tasks,
including the first pretraining task (if present), with T, and the number of
classes in the first task by S. For example, for CIFAR100 T10S10 is a setting
composed of 10 tasks with 10 classes each, while T11S50 is a setting where the
first task contains 50 classes and the next 10 tasks contain 5 classes each.

###  D.1 Standard benchmarks

In addition to Section 4.2, we conduct more experiments on CIFAR100,
TinyImageNet200 and ImageNet100, adding two settings with a smaller number of
tasks. We report final accuracy and forgetting in addition to incremental
accuracy and forgetting. Additionally, we include the results for the warmup
phase (WU) applied in isolation, without the Teacher Adaptation.

We report the results for CIFAR100, TinyImageNet200 and ImageNet100 in Table
8, Table 9, and Table 10 respectively.

| T5S20 | T6S50
---|---|---
| $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$ | $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$
GKD | 37.63$\pm$0.52 | 48.80$\pm$0.36 | 23.13$\pm$2.28 | 19.30$\pm$2.37 | 40.74$\pm$0.72 | 51.93$\pm$1.24 | 25.87$\pm$0.75 | 18.90$\pm$0.29
+TA | 40.84$\pm$0.23 | 50.08$\pm$0.26 | 16.76$\pm$1.68 | 16.66$\pm$1.25 | 43.18$\pm$1.66 | 52.22$\pm$1.28 | 18.73$\pm$1.61 | 14.09$\pm$0.66
+WU | 38.82$\pm$0.06 | 49.69$\pm$0.26 | 22.08$\pm$0.61 | 19.34$\pm$1.24 | 43.19$\pm$0.95 | 53.42$\pm$0.50 | 24.54$\pm$2.17 | 18.94$\pm$2.69
+TA+WU | 41.50$\pm$0.60 | 50.70$\pm$0.27 | 16.85$\pm$0.74 | 17.16$\pm$1.24 | 44.66$\pm$0.54 | 53.40$\pm$0.73 | 18.73$\pm$1.73 | 14.53$\pm$1.56
MKD | 34.41$\pm$0.16 | 47.42$\pm$0.32 | 23.74$\pm$1.63 | 23.51$\pm$1.75 | 39.33$\pm$0.72 | 50.28$\pm$1.12 | 23.54$\pm$0.62 | 18.19$\pm$0.31
+TA | 37.03$\pm$0.82 | 48.63$\pm$0.17 | 14.64$\pm$0.22 | 17.63$\pm$0.73 | 39.45$\pm$1.29 | 49.42$\pm$1.38 | 13.97$\pm$0.89 | 11.02$\pm$0.28
+WU | 35.37$\pm$0.19 | 47.96$\pm$0.15 | 24.63$\pm$0.81 | 24.35$\pm$0.44 | 40.89$\pm$1.52 | 51.46$\pm$0.68 | 24.45$\pm$2.59 | 19.53$\pm$2.86
+TA+WU | 38.26$\pm$0.24 | 49.58$\pm$0.06 | 15.18$\pm$0.41 | 17.96$\pm$0.65 | 41.42$\pm$0.78 | 51.04$\pm$1.17 | 15.00$\pm$1.07 | 12.03$\pm$1.32
TKD | 38.33$\pm$0.70 | 49.56$\pm$0.48 | 24.78$\pm$2.58 | 25.04$\pm$2.55 | 41.19$\pm$0.42 | 52.07$\pm$1.35 | 17.31$\pm$1.20 | 15.02$\pm$0.39
+TA | 41.12$\pm$0.35 | 50.87$\pm$0.15 | 18.12$\pm$1.55 | 21.09$\pm$1.39 | 41.36$\pm$0.89 | 51.88$\pm$0.80 | 12.84$\pm$1.25 | 11.42$\pm$0.64
+WU | 40.03$\pm$0.66 | 50.50$\pm$0.17 | 22.79$\pm$0.73 | 24.19$\pm$1.61 | 43.16$\pm$0.26 | 53.51$\pm$0.39 | 18.40$\pm$1.73 | 16.03$\pm$2.77
+TA+WU | 42.07$\pm$0.67 | 51.47$\pm$0.33 | 17.77$\pm$1.37 | 21.21$\pm$1.34 | 43.08$\pm$0.35 | 53.12$\pm$0.60 | 13.64$\pm$0.89 | 12.13$\pm$1.37
ANCL | 37.13$\pm$0.33 | 48.60$\pm$0.27 | 32.41$\pm$0.26 | 31.71$\pm$0.81 | 43.41$\pm$0.62 | 53.40$\pm$0.49 | 23.54$\pm$1.29 | 18.08$\pm$0.40
+TA | 41.04$\pm$0.22 | 50.36$\pm$0.43 | 24.07$\pm$0.81 | 26.18$\pm$0.51 | 44.64$\pm$1.65 | 53.33$\pm$0.66 | 18.43$\pm$3.12 | 14.44$\pm$1.29
+WU | 38.94$\pm$0.57 | 49.26$\pm$0.61 | 29.84$\pm$0.88 | 30.98$\pm$1.98 | 44.74$\pm$0.30 | 54.25$\pm$0.29 | 22.42$\pm$1.75 | 17.49$\pm$1.56
+TA+WU | 42.08$\pm$0.53 | 50.59$\pm$0.21 | 22.55$\pm$1.15 | 26.09$\pm$1.11 | 45.64$\pm$0.87 | 54.05$\pm$0.42 | 18.12$\pm$1.58 | 14.28$\pm$1.06
| T10S10 | T11S50
GKD | 28.27$\pm$0.44 | 42.52$\pm$0.76 | 29.59$\pm$0.92 | 22.26$\pm$0.31 | 30.79$\pm$1.62 | 41.69$\pm$1.18 | 26.84$\pm$2.12 | 18.09$\pm$0.88
+TA | 31.92$\pm$0.86 | 44.09$\pm$0.97 | 22.65$\pm$1.32 | 19.41$\pm$0.60 | 33.20$\pm$0.76 | 44.05$\pm$1.12 | 18.90$\pm$0.19 | 12.97$\pm$0.43
+WU | 31.11$\pm$0.58 | 43.95$\pm$0.65 | 25.26$\pm$0.93 | 21.95$\pm$0.74 | 33.17$\pm$0.86 | 44.09$\pm$1.26 | 23.85$\pm$1.29 | 17.25$\pm$0.74
+TA+WU | 33.76$\pm$0.78 | 45.25$\pm$1.02 | 20.87$\pm$0.13 | 19.87$\pm$0.34 | 34.72$\pm$0.77 | 46.27$\pm$1.09 | 19.13$\pm$1.77 | 13.98$\pm$0.98
MKD | 25.41$\pm$1.07 | 39.36$\pm$0.70 | 44.04$\pm$0.98 | 42.74$\pm$0.52 | 31.31$\pm$1.22 | 41.04$\pm$0.93 | 22.56$\pm$0.94 | 15.37$\pm$0.33
+TA | 29.97$\pm$1.44 | 43.55$\pm$0.96 | 32.37$\pm$2.17 | 32.61$\pm$0.80 | 30.90$\pm$0.69 | 41.67$\pm$1.35 | 14.76$\pm$0.25 | 10.51$\pm$0.36
+WU | 27.32$\pm$1.28 | 40.78$\pm$0.95 | 37.89$\pm$0.22 | 39.03$\pm$0.53 | 31.63$\pm$1.16 | 42.45$\pm$0.78 | 22.57$\pm$1.57 | 16.98$\pm$1.23
+TA+WU | 31.94$\pm$0.23 | 44.85$\pm$0.80 | 26.83$\pm$1.01 | 30.08$\pm$0.57 | 32.79$\pm$0.55 | 44.19$\pm$1.17 | 16.44$\pm$0.63 | 12.56$\pm$0.53
TKD | 30.05$\pm$0.81 | 43.74$\pm$0.84 | 24.53$\pm$0.23 | 23.65$\pm$0.79 | 28.38$\pm$1.46 | 40.44$\pm$1.40 | 15.68$\pm$0.84 | 12.20$\pm$0.46
+TA | 31.80$\pm$0.67 | 45.29$\pm$1.02 | 18.59$\pm$0.90 | 19.42$\pm$0.85 | 28.50$\pm$0.39 | 41.68$\pm$1.03 | 11.58$\pm$0.38 | 9.29$\pm$0.75
+WU | 31.23$\pm$0.94 | 44.59$\pm$0.72 | 23.21$\pm$0.35 | 23.73$\pm$0.32 | 30.88$\pm$1.16 | 42.45$\pm$1.31 | 17.94$\pm$0.39 | 14.30$\pm$0.99
+TA+WU | 33.14$\pm$1.03 | 46.21$\pm$0.86 | 17.55$\pm$0.66 | 20.45$\pm$0.57 | 30.60$\pm$0.37 | 44.22$\pm$1.08 | 14.65$\pm$0.49 | 11.79$\pm$0.57
ANCL | 29.77$\pm$1.10 | 43.15$\pm$0.49 | 36.37$\pm$1.48 | 32.78$\pm$1.52 | 29.77$\pm$1.10 | 43.15$\pm$0.49 | 36.37$\pm$1.48 | 32.78$\pm$1.52
+TA | 33.47$\pm$0.17 | 45.67$\pm$0.14 | 28.04$\pm$1.61 | 26.71$\pm$1.71 | 33.47$\pm$0.17 | 45.67$\pm$0.14 | 28.04$\pm$1.61 | 26.71$\pm$1.71
+WU | 31.41$\pm$0.54 | 44.28$\pm$0.08 | 34.70$\pm$0.25 | 32.06$\pm$1.07 | 31.41$\pm$0.54 | 44.28$\pm$0.08 | 34.70$\pm$0.25 | 32.06$\pm$1.07
+TA+WU | 34.17$\pm$0.30 | 46.73$\pm$0.20 | 28.67$\pm$0.63 | 26.86$\pm$1.17 | 34.17$\pm$0.30 | 46.73$\pm$0.20 | 28.67$\pm$0.63 | 26.86$\pm$1.17
| T20S5 | T26S50
GKD | 15.59$\pm$0.32 | 31.89$\pm$0.45 | 43.28$\pm$0.56 | 34.68$\pm$1.87 | 10.10$\pm$0.71 | 17.64$\pm$0.93 | 15.29$\pm$0.27 | 9.67$\pm$0.26
+TA | 19.55$\pm$0.24 | 35.99$\pm$0.79 | 30.38$\pm$2.08 | 23.32$\pm$1.79 | 11.99$\pm$0.66 | 19.37$\pm$1.73 | 9.05$\pm$0.63 | 8.31$\pm$0.68
+WU | 19.22$\pm$0.80 | 34.67$\pm$0.55 | 37.05$\pm$1.03 | 31.14$\pm$1.41 | 10.41$\pm$1.01 | 21.87$\pm$0.16 | 17.92$\pm$0.48 | 11.02$\pm$0.65
+TA+WU | 21.52$\pm$0.62 | 37.11$\pm$0.64 | 30.34$\pm$0.78 | 24.87$\pm$1.04 | 18.11$\pm$0.52 | 26.15$\pm$0.94 | 10.11$\pm$0.97 | 8.73$\pm$0.85
MKD | 17.61$\pm$0.51 | 32.89$\pm$0.42 | 33.36$\pm$0.80 | 32.01$\pm$1.36 | 11.55$\pm$0.86 | 19.14$\pm$1.36 | 14.04$\pm$0.71 | 8.76$\pm$0.35
+TA | 19.07$\pm$0.65 | 35.36$\pm$0.85 | 21.27$\pm$1.08 | 20.96$\pm$1.31 | 13.91$\pm$0.80 | 20.99$\pm$1.53 | 9.52$\pm$0.26 | 8.20$\pm$0.98
+WU | 18.96$\pm$0.82 | 34.30$\pm$0.94 | 30.45$\pm$0.70 | 29.75$\pm$1.56 | 11.48$\pm$0.36 | 22.42$\pm$0.76 | 17.42$\pm$1.70 | 11.51$\pm$1.19
+TA+WU | 20.52$\pm$0.85 | 36.79$\pm$0.70 | 21.88$\pm$0.81 | 21.84$\pm$0.33 | 18.15$\pm$0.70 | 26.10$\pm$0.75 | 11.64$\pm$1.04 | 9.17$\pm$0.20
TKD | 19.39$\pm$0.41 | 34.58$\pm$0.34 | 22.06$\pm$0.46 | 21.13$\pm$1.17 | 7.88$\pm$0.08 | 14.64$\pm$0.33 | 7.96$\pm$0.47 | 6.02$\pm$0.54
+TA | 18.30$\pm$0.50 | 34.62$\pm$0.92 | 15.22$\pm$1.25 | 14.72$\pm$1.28 | 9.05$\pm$0.64 | 16.66$\pm$1.66 | 7.17$\pm$0.53 | 6.88$\pm$0.36
+WU | 20.77$\pm$0.59 | 35.72$\pm$0.09 | 21.14$\pm$0.88 | 20.93$\pm$1.62 | 9.55$\pm$0.53 | 18.01$\pm$0.59 | 12.63$\pm$1.04 | 9.90$\pm$0.99
+TA+WU | 20.24$\pm$0.97 | 36.26$\pm$0.71 | 17.98$\pm$1.00 | 17.01$\pm$0.89 | 13.34$\pm$0.73 | 22.00$\pm$0.97 | 10.80$\pm$1.03 | 9.36$\pm$0.68
ANCL | 19.21$\pm$0.75 | 34.32$\pm$0.41 | 39.97$\pm$1.81 | 36.74$\pm$1.38 | 11.04$\pm$0.40 | 21.84$\pm$1.33 | 18.31$\pm$1.64 | 11.79$\pm$0.51
+TA | 20.63$\pm$1.04 | 37.52$\pm$0.78 | 31.07$\pm$1.00 | 26.77$\pm$0.49 | 18.96$\pm$0.91 | 26.47$\pm$0.41 | 12.69$\pm$1.49 | 10.33$\pm$0.57
+WU | 21.53$\pm$0.38 | 36.06$\pm$0.73 | 37.34$\pm$0.53 | 35.34$\pm$1.52 | 11.81$\pm$0.24 | 23.81$\pm$0.88 | 20.05$\pm$0.95 | 13.18$\pm$0.34
+TA+WU | 22.54$\pm$0.33 | 38.48$\pm$0.94 | 31.87$\pm$0.92 | 27.99$\pm$1.20 | 20.79$\pm$0.28 | 29.67$\pm$1.13 | 13.84$\pm$0.45 | 10.98$\pm$0.56

Table 8: Additional results for CIFAR100

| T5S40 | T6S100
---|---|---
| $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$ | $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$
GKD | 27.51$\pm$0.81 | 37.53$\pm$1.28 | 18.22$\pm$0.55 | 14.58$\pm$0.56 | 32.92$\pm$1.40 | 38.74$\pm$2.77 | 13.77$\pm$2.12 | 9.60$\pm$2.04
+TA | 28.36$\pm$1.13 | 37.60$\pm$1.49 | 15.55$\pm$1.79 | 12.63$\pm$0.39 | 33.79$\pm$0.89 | 38.56$\pm$1.73 | 10.94$\pm$0.71 | 8.28$\pm$0.73
+WU | 28.31$\pm$0.66 | 37.74$\pm$1.00 | 17.32$\pm$0.54 | 15.04$\pm$0.93 | 33.80$\pm$0.79 | 39.33$\pm$1.84 | 13.04$\pm$2.02 | 9.44$\pm$1.29
+TA+WU | 29.36$\pm$0.81 | 38.21$\pm$1.10 | 14.05$\pm$1.38 | 12.50$\pm$0.60 | 34.68$\pm$1.06 | 39.45$\pm$1.73 | 10.40$\pm$0.73 | 8.36$\pm$0.78
MKD | 24.09$\pm$0.94 | 34.93$\pm$1.39 | 11.53$\pm$0.17 | 10.22$\pm$1.34 | 29.65$\pm$0.77 | 35.48$\pm$2.06 | 9.27$\pm$1.86 | 6.66$\pm$1.30
+TA | 24.17$\pm$0.77 | 34.67$\pm$1.58 | 6.57$\pm$0.26 | 6.51$\pm$0.67 | 28.28$\pm$0.79 | 34.07$\pm$1.63 | 4.31$\pm$0.47 | 4.14$\pm$0.60
+WU | 24.55$\pm$0.73 | 35.28$\pm$1.08 | 11.10$\pm$0.07 | 10.79$\pm$1.45 | 30.40$\pm$1.05 | 36.43$\pm$2.08 | 9.79$\pm$2.19 | 7.10$\pm$1.53
+TA+WU | 24.33$\pm$0.46 | 35.05$\pm$1.37 | 6.28$\pm$0.70 | 6.92$\pm$1.04 | 28.53$\pm$1.58 | 34.81$\pm$1.95 | 4.49$\pm$0.34 | 4.42$\pm$0.63
TKD | 28.18$\pm$0.92 | 37.43$\pm$0.86 | 19.76$\pm$1.11 | 20.37$\pm$0.78 | 34.17$\pm$0.72 | 40.11$\pm$1.59 | 12.33$\pm$1.05 | 11.59$\pm$1.22
+TA | 28.95$\pm$0.93 | 38.07$\pm$1.03 | 16.36$\pm$1.91 | 17.23$\pm$0.56 | 34.42$\pm$0.61 | 40.20$\pm$1.31 | 10.13$\pm$1.03 | 10.15$\pm$1.03
+WU | 29.07$\pm$0.54 | 37.73$\pm$0.71 | 17.35$\pm$1.16 | 19.68$\pm$0.80 | 34.22$\pm$0.79 | 40.44$\pm$1.63 | 12.09$\pm$1.68 | 11.48$\pm$1.58
+TA+WU | 29.57$\pm$0.46 | 38.29$\pm$0.74 | 14.69$\pm$1.34 | 16.93$\pm$0.86 | 34.63$\pm$1.07 | 40.64$\pm$1.38 | 9.55$\pm$0.60 | 10.01$\pm$1.07
LwF(A) | 28.33$\pm$0.36 | 37.63$\pm$0.83 | 21.80$\pm$1.09 | 20.57$\pm$0.85 | 34.48$\pm$0.78 | 40.85$\pm$1.49 | 19.51$\pm$0.94 | 15.69$\pm$1.43
+TA | 29.37$\pm$0.69 | 38.31$\pm$1.06 | 19.34$\pm$1.20 | 18.17$\pm$0.74 | 36.01$\pm$1.02 | 41.22$\pm$1.46 | 15.83$\pm$0.35 | 13.75$\pm$1.12
+WU | 28.59$\pm$0.63 | 37.69$\pm$0.73 | 21.70$\pm$0.96 | 20.87$\pm$1.02 | 34.64$\pm$0.38 | 41.11$\pm$1.38 | 20.23$\pm$1.55 | 15.70$\pm$1.97
+TA+WU | 29.98$\pm$0.86 | 38.45$\pm$0.84 | 18.66$\pm$1.61 | 18.35$\pm$0.98 | 36.20$\pm$0.73 | 41.54$\pm$1.21 | 16.64$\pm$1.02 | 13.85$\pm$1.43
| T10S20 | T11S100
GKD | 20.76$\pm$0.96 | 32.12$\pm$0.57 | 26.87$\pm$2.16 | 21.21$\pm$1.31 | 27.17$\pm$0.73 | 34.32$\pm$1.96 | 17.53$\pm$3.99 | 11.41$\pm$2.80
+TA | 22.94$\pm$0.70 | 33.12$\pm$0.53 | 21.39$\pm$2.14 | 17.50$\pm$1.76 | 30.65$\pm$0.55 | 37.01$\pm$1.84 | 12.85$\pm$1.60 | 10.05$\pm$0.87
+WU | 21.94$\pm$0.71 | 33.00$\pm$0.59 | 24.84$\pm$1.49 | 20.37$\pm$0.65 | 28.77$\pm$0.81 | 36.54$\pm$1.45 | 18.11$\pm$1.90 | 11.66$\pm$1.83
+TA+WU | 23.72$\pm$1.34 | 33.90$\pm$0.78 | 21.29$\pm$2.19 | 17.70$\pm$1.48 | 31.34$\pm$0.49 | 37.50$\pm$1.84 | 13.25$\pm$2.11 | 10.14$\pm$1.10
MKD | 19.37$\pm$0.85 | 31.04$\pm$1.00 | 17.04$\pm$0.66 | 16.90$\pm$0.75 | 26.11$\pm$0.72 | 33.75$\pm$2.11 | 16.22$\pm$2.18 | 10.30$\pm$1.63
+TA | 20.74$\pm$0.88 | 31.78$\pm$0.83 | 9.89$\pm$1.58 | 11.42$\pm$1.23 | 25.59$\pm$1.25 | 32.23$\pm$2.28 | 6.18$\pm$0.99 | 5.91$\pm$0.66
+WU | 19.93$\pm$0.67 | 31.32$\pm$0.95 | 16.81$\pm$1.12 | 16.97$\pm$0.27 | 26.36$\pm$0.80 | 33.91$\pm$1.39 | 17.12$\pm$2.33 | 11.17$\pm$1.71
+TA+WU | 21.42$\pm$0.75 | 32.22$\pm$0.95 | 10.13$\pm$1.25 | 11.94$\pm$1.07 | 26.79$\pm$1.56 | 32.99$\pm$1.98 | 6.78$\pm$0.92 | 6.42$\pm$0.80
TKD | 22.88$\pm$0.29 | 33.15$\pm$0.45 | 19.17$\pm$0.02 | 21.05$\pm$0.39 | 28.32$\pm$1.18 | 37.31$\pm$1.36 | 14.34$\pm$0.78 | 11.83$\pm$1.31
+TA | 23.92$\pm$0.49 | 33.94$\pm$0.70 | 14.91$\pm$0.98 | 17.32$\pm$0.97 | 30.18$\pm$0.92 | 37.97$\pm$1.47 | 7.93$\pm$0.77 | 9.02$\pm$0.65
+WU | 23.32$\pm$0.61 | 33.71$\pm$0.97 | 18.77$\pm$1.33 | 20.33$\pm$0.41 | 27.90$\pm$2.05 | 37.22$\pm$1.60 | 14.43$\pm$1.16 | 11.48$\pm$0.86
+TA+WU | 24.69$\pm$0.89 | 34.58$\pm$0.96 | 15.09$\pm$0.92 | 17.27$\pm$0.54 | 30.80$\pm$0.75 | 38.41$\pm$1.52 | 8.55$\pm$1.06 | 9.16$\pm$0.62
ANCL | 22.58$\pm$0.83 | 32.84$\pm$0.78 | 28.76$\pm$2.52 | 27.24$\pm$1.47 | 27.78$\pm$0.96 | 37.74$\pm$0.60 | 25.24$\pm$2.77 | 17.28$\pm$2.59
+TA | 23.98$\pm$0.68 | 33.85$\pm$0.55 | 25.23$\pm$1.31 | 24.02$\pm$1.39 | 31.34$\pm$0.34 | 39.50$\pm$1.18 | 20.36$\pm$1.54 | 14.88$\pm$1.41
+WU | 23.20$\pm$0.57 | 33.33$\pm$0.81 | 28.66$\pm$1.18 | 26.86$\pm$0.20 | 29.05$\pm$0.75 | 38.86$\pm$1.29 | 25.35$\pm$0.64 | 17.11$\pm$0.89
+TA+WU | 24.78$\pm$0.86 | 34.59$\pm$0.75 | 25.24$\pm$1.47 | 23.64$\pm$0.76 | 32.36$\pm$0.60 | 40.10$\pm$1.39 | 20.31$\pm$1.73 | 15.12$\pm$1.01
| T20S10 | T26S100
GKD | 13.83$\pm$0.25 | 25.75$\pm$0.65 | 33.70$\pm$0.54 | 27.55$\pm$1.81 | 14.33$\pm$1.42 | 23.30$\pm$1.59 | 23.27$\pm$2.88 | 12.63$\pm$2.21
+TA | 16.15$\pm$0.83 | 27.16$\pm$0.87 | 27.52$\pm$0.51 | 22.62$\pm$2.03 | 19.70$\pm$0.79 | 26.88$\pm$1.62 | 14.57$\pm$1.00 | 11.08$\pm$0.47
+WU | 15.20$\pm$0.46 | 26.98$\pm$1.05 | 30.78$\pm$1.84 | 25.57$\pm$1.97 | 16.10$\pm$1.03 | 25.59$\pm$1.76 | 21.45$\pm$1.77 | 12.66$\pm$1.65
+TA+WU | 17.65$\pm$0.16 | 27.85$\pm$0.90 | 25.60$\pm$0.40 | 21.65$\pm$1.43 | 21.86$\pm$1.34 | 28.82$\pm$1.95 | 15.56$\pm$0.77 | 11.44$\pm$0.61
MKD | 13.81$\pm$0.53 | 25.22$\pm$0.88 | 24.61$\pm$1.91 | 25.80$\pm$2.46 | 15.99$\pm$1.72 | 23.42$\pm$1.95 | 18.15$\pm$3.02 | 10.41$\pm$1.97
+TA | 16.66$\pm$0.59 | 27.05$\pm$1.11 | 14.46$\pm$0.75 | 17.55$\pm$2.36 | 17.65$\pm$0.32 | 23.29$\pm$1.61 | 8.57$\pm$1.70 | 7.61$\pm$1.18
+WU | 13.97$\pm$0.62 | 25.76$\pm$0.73 | 24.33$\pm$2.37 | 24.76$\pm$2.29 | 16.74$\pm$1.44 | 24.42$\pm$1.72 | 19.95$\pm$2.21 | 12.29$\pm$2.42
+TA+WU | 17.31$\pm$0.48 | 27.39$\pm$1.33 | 13.66$\pm$0.39 | 16.74$\pm$1.52 | 20.29$\pm$1.27 | 25.75$\pm$1.74 | 9.48$\pm$1.04 | 8.35$\pm$0.82
TKD | 16.13$\pm$0.45 | 27.29$\pm$0.76 | 21.34$\pm$2.13 | 23.20$\pm$2.46 | 14.77$\pm$1.50 | 23.94$\pm$2.14 | 14.54$\pm$2.43 | 10.20$\pm$1.09
+TA | 18.34$\pm$0.25 | 28.51$\pm$0.96 | 14.67$\pm$0.26 | 17.38$\pm$2.44 | 18.20$\pm$0.34 | 26.56$\pm$1.62 | 8.11$\pm$1.01 | 8.05$\pm$0.36
+WU | 16.16$\pm$0.29 | 28.02$\pm$0.63 | 21.28$\pm$1.94 | 22.07$\pm$1.48 | 15.71$\pm$1.15 | 24.74$\pm$2.16 | 16.75$\pm$1.81 | 12.14$\pm$2.04
+TA+WU | 18.99$\pm$0.50 | 28.71$\pm$1.06 | 14.58$\pm$0.94 | 17.37$\pm$1.78 | 20.07$\pm$0.87 | 28.10$\pm$1.97 | 9.74$\pm$0.90 | 9.56$\pm$0.65
ANCL | 15.75$\pm$0.24 | 26.98$\pm$0.60 | 35.24$\pm$1.66 | 32.45$\pm$1.18 | 15.42$\pm$1.49 | 27.95$\pm$1.98 | 29.72$\pm$3.14 | 20.91$\pm$0.92
+TA | 17.91$\pm$0.33 | 28.42$\pm$0.48 | 30.13$\pm$1.18 | 27.26$\pm$0.98 | 21.04$\pm$1.79 | 31.60$\pm$1.89 | 23.59$\pm$1.71 | 18.10$\pm$0.55
+WU | 16.40$\pm$0.39 | 27.54$\pm$0.44 | 34.51$\pm$1.06 | 31.76$\pm$0.91 | 16.47$\pm$1.58 | 28.80$\pm$1.59 | 29.62$\pm$2.49 | 20.74$\pm$1.10
+TA+WU | 19.02$\pm$0.34 | 29.18$\pm$0.71 | 28.78$\pm$0.40 | 26.50$\pm$1.34 | 22.32$\pm$1.25 | 32.60$\pm$1.45 | 23.21$\pm$1.01 | 17.94$\pm$0.39

Table 9: Additional results for TinyImageNet200.

| T5S20 | T6S50
---|---|---
| $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$ | $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$
GKD | 51.06$\pm$0.59 | 61.95$\pm$0.50 | 27.57$\pm$0.79 | 23.16$\pm$0.88 | 57.30$\pm$1.40 | 66.36$\pm$0.75 | 19.19$\pm$0.53 | 12.92$\pm$0.59
+TA | 52.29$\pm$0.28 | 62.89$\pm$0.32 | 23.40$\pm$0.31 | 18.94$\pm$0.43 | 57.86$\pm$0.68 | 66.20$\pm$0.40 | 16.81$\pm$1.00 | 12.70$\pm$0.36
+WU | 51.04$\pm$0.53 | 60.24$\pm$0.23 | 26.26$\pm$0.63 | 27.22$\pm$0.29 | 59.52$\pm$0.27 | 67.63$\pm$0.39 | 18.26$\pm$0.76 | 14.45$\pm$0.47
+TA+WU | 51.27$\pm$0.18 | 61.01$\pm$0.12 | 23.98$\pm$0.36 | 23.51$\pm$0.29 | 58.40$\pm$0.86 | 66.86$\pm$0.31 | 18.87$\pm$0.75 | 15.29$\pm$0.69
MKD | 50.17$\pm$0.25 | 61.48$\pm$0.35 | 23.21$\pm$0.78 | 21.01$\pm$0.35 | 54.70$\pm$1.02 | 65.82$\pm$0.24 | 27.21$\pm$0.70 | 18.25$\pm$0.32
+TA | 50.92$\pm$0.42 | 62.18$\pm$0.45 | 16.00$\pm$0.61 | 14.71$\pm$0.35 | 57.26$\pm$0.83 | 66.03$\pm$0.32 | 19.00$\pm$1.04 | 14.34$\pm$0.11
+WU | 50.45$\pm$0.60 | 61.45$\pm$0.47 | 23.20$\pm$0.50 | 21.42$\pm$1.21 | 56.78$\pm$0.61 | 66.36$\pm$0.31 | 25.14$\pm$0.24 | 19.54$\pm$0.67
+TA+WU | 51.59$\pm$0.24 | 62.37$\pm$0.59 | 16.66$\pm$0.72 | 15.56$\pm$1.62 | 57.82$\pm$0.37 | 66.60$\pm$0.14 | 19.86$\pm$0.60 | 16.32$\pm$0.62
TKD | 53.73$\pm$0.25 | 62.91$\pm$0.44 | 20.77$\pm$0.58 | 20.30$\pm$0.80 | 58.90$\pm$1.02 | 67.49$\pm$0.42 | 17.38$\pm$0.80 | 14.15$\pm$0.13
+TA | 53.29$\pm$0.04 | 63.04$\pm$0.30 | 18.28$\pm$0.72 | 17.26$\pm$0.89 | 58.80$\pm$0.48 | 67.00$\pm$0.23 | 15.07$\pm$0.66 | 13.48$\pm$0.23
+WU | 52.14$\pm$0.16 | 60.83$\pm$0.24 | 22.90$\pm$0.70 | 25.45$\pm$0.55 | 59.85$\pm$0.95 | 67.77$\pm$0.46 | 18.40$\pm$0.63 | 16.35$\pm$0.64
+TA+WU | 51.69$\pm$0.19 | 61.09$\pm$0.28 | 21.68$\pm$0.15 | 23.15$\pm$0.59 | 58.45$\pm$0.41 | 66.94$\pm$0.26 | 18.31$\pm$0.51 | 16.40$\pm$0.60
ANCL | 54.50$\pm$0.28 | 63.96$\pm$0.32 | 20.26$\pm$0.36 | 17.56$\pm$0.40 | 58.24$\pm$1.28 | 66.98$\pm$0.70 | 16.34$\pm$0.36 | 11.40$\pm$0.48
+TA | 54.39$\pm$0.64 | 63.92$\pm$0.42 | 18.04$\pm$0.33 | 15.29$\pm$0.37 | 59.16$\pm$0.63 | 66.92$\pm$0.56 | 15.03$\pm$0.91 | 11.91$\pm$0.10
+WU | 54.84$\pm$0.20 | 64.25$\pm$0.37 | 18.71$\pm$0.18 | 16.55$\pm$0.51 | 60.14$\pm$0.66 | 67.78$\pm$0.41 | 14.04$\pm$0.27 | 10.56$\pm$0.42
+TA+WU | 54.39$\pm$0.60 | 64.08$\pm$0.40 | 17.81$\pm$0.53 | 15.00$\pm$0.59 | 59.72$\pm$0.56 | 67.00$\pm$0.20 | 14.62$\pm$0.32 | 12.12$\pm$0.51
| T10S10 | T11S50
GKD | 40.33$\pm$0.37 | 54.62$\pm$0.52 | 32.72$\pm$0.09 | 25.95$\pm$0.11 | 43.34$\pm$0.85 | 57.94$\pm$0.90 | 26.04$\pm$2.37 | 14.47$\pm$0.83
+TA | 43.17$\pm$1.06 | 55.82$\pm$0.61 | 25.10$\pm$1.03 | 20.52$\pm$0.24 | 48.80$\pm$0.21 | 57.18$\pm$0.45 | 17.56$\pm$0.53 | 17.24$\pm$0.39
+WU | 41.64$\pm$0.32 | 53.12$\pm$0.61 | 28.09$\pm$0.62 | 27.10$\pm$0.55 | 47.81$\pm$1.13 | 60.93$\pm$0.31 | 23.24$\pm$1.20 | 14.88$\pm$0.68
+TA+WU | 42.37$\pm$1.01 | 54.23$\pm$0.99 | 24.94$\pm$0.76 | 23.31$\pm$0.65 | 50.79$\pm$0.11 | 59.56$\pm$0.21 | 19.53$\pm$0.13 | 18.56$\pm$0.28
MKD | 40.35$\pm$0.43 | 54.01$\pm$0.01 | 30.94$\pm$1.01 | 28.19$\pm$0.61 | 43.01$\pm$0.94 | 56.18$\pm$0.90 | 26.99$\pm$0.88 | 14.94$\pm$0.17
+TA | 42.35$\pm$0.50 | 56.02$\pm$0.20 | 19.37$\pm$1.18 | 18.60$\pm$0.76 | 43.80$\pm$0.49 | 52.05$\pm$0.24 | 13.67$\pm$1.09 | 14.23$\pm$0.46
+WU | 40.49$\pm$0.52 | 53.79$\pm$0.54 | 29.89$\pm$1.41 | 27.98$\pm$0.27 | 43.61$\pm$1.63 | 57.32$\pm$1.60 | 27.71$\pm$1.83 | 16.09$\pm$3.04
+TA+WU | 42.94$\pm$0.88 | 55.39$\pm$0.25 | 20.62$\pm$0.53 | 21.11$\pm$0.38 | 45.33$\pm$3.15 | 54.05$\pm$3.32 | 15.21$\pm$2.99 | 15.60$\pm$2.54
TKD | 43.19$\pm$0.16 | 55.70$\pm$0.49 | 24.84$\pm$0.35 | 23.55$\pm$0.35 | 40.56$\pm$1.30 | 54.72$\pm$0.86 | 15.39$\pm$1.01 | 10.16$\pm$0.34
+TA | 43.93$\pm$0.72 | 56.23$\pm$0.70 | 17.89$\pm$0.14 | 18.09$\pm$0.26 | 42.83$\pm$0.61 | 53.85$\pm$0.39 | 10.13$\pm$0.20 | 13.15$\pm$0.16
+WU | 42.71$\pm$1.29 | 53.51$\pm$0.56 | 24.84$\pm$0.96 | 26.84$\pm$0.28 | 45.44$\pm$0.98 | 58.94$\pm$0.45 | 23.85$\pm$1.16 | 18.09$\pm$0.76
+TA+WU | 43.05$\pm$0.55 | 54.40$\pm$0.64 | 21.32$\pm$0.73 | 22.50$\pm$0.12 | 49.48$\pm$0.92 | 58.86$\pm$0.38 | 17.65$\pm$1.12 | 19.07$\pm$0.75
ANCL | 44.01$\pm$0.19 | 55.81$\pm$0.41 | 29.56$\pm$0.51 | 27.13$\pm$0.50 | 46.94$\pm$0.67 | 60.97$\pm$0.62 | 26.30$\pm$0.84 | 15.72$\pm$0.14
+TA | 45.75$\pm$0.90 | 57.41$\pm$0.22 | 23.71$\pm$1.08 | 21.54$\pm$0.50 | 50.35$\pm$0.45 | 59.22$\pm$0.26 | 18.07$\pm$1.14 | 17.34$\pm$0.53
+WU | 45.08$\pm$0.08 | 56.13$\pm$0.40 | 27.27$\pm$0.34 | 26.13$\pm$0.20 | 50.18$\pm$0.87 | 63.15$\pm$0.12 | 23.10$\pm$0.73 | 13.29$\pm$0.35
+TA+WU | 46.37$\pm$0.36 | 57.20$\pm$0.32 | 22.79$\pm$0.74 | 21.70$\pm$0.29 | 52.70$\pm$0.25 | 61.18$\pm$0.34 | 17.93$\pm$0.82 | 16.92$\pm$0.26
| T20S5 | T26S50
GKD | 24.14$\pm$0.91 | 42.82$\pm$0.58 | 45.53$\pm$0.88 | 35.39$\pm$0.88 | 14.82$\pm$0.85 | 21.91$\pm$0.06 | 17.99$\pm$0.53 | 9.29$\pm$0.69
+TA | 31.89$\pm$1.63 | 45.88$\pm$0.79 | 27.74$\pm$1.03 | 23.25$\pm$0.62 | 17.16$\pm$0.84 | 22.31$\pm$0.64 | 12.71$\pm$1.11 | 11.28$\pm$0.98
+WU | 28.27$\pm$1.19 | 43.92$\pm$0.38 | 37.93$\pm$1.38 | 32.37$\pm$0.66 | 17.85$\pm$0.27 | 30.15$\pm$0.56 | 23.05$\pm$1.17 | 15.86$\pm$0.91
+TA+WU | 32.63$\pm$0.69 | 45.75$\pm$0.61 | 29.33$\pm$0.37 | 25.37$\pm$0.04 | 24.90$\pm$0.24 | 34.19$\pm$0.28 | 20.82$\pm$1.10 | 18.97$\pm$1.23
MKD | 26.27$\pm$1.28 | 43.39$\pm$0.66 | 40.22$\pm$1.04 | 34.25$\pm$0.81 | 15.83$\pm$1.18 | 26.07$\pm$0.29 | 29.85$\pm$0.95 | 16.00$\pm$0.06
+TA | 31.90$\pm$0.34 | 46.18$\pm$0.54 | 18.92$\pm$0.14 | 19.14$\pm$0.79 | 16.12$\pm$0.25 | 22.25$\pm$0.13 | 11.65$\pm$0.66 | 11.94$\pm$1.23
+WU | 27.50$\pm$1.10 | 43.73$\pm$0.43 | 38.01$\pm$1.61 | 33.47$\pm$0.76 | 16.15$\pm$0.27 | 28.42$\pm$1.65 | 32.35$\pm$3.78 | 18.39$\pm$4.16
+TA+WU | 33.55$\pm$1.15 | 46.12$\pm$0.64 | 22.06$\pm$1.28 | 21.56$\pm$0.80 | 19.85$\pm$6.72 | 25.79$\pm$6.22 | 13.35$\pm$3.06 | 13.72$\pm$3.01
TKD | 29.39$\pm$0.47 | 44.75$\pm$0.28 | 35.96$\pm$0.95 | 32.16$\pm$0.14 | 10.99$\pm$0.22 | 19.32$\pm$0.23 | 13.55$\pm$0.88 | 9.67$\pm$0.61
+TA | 32.71$\pm$0.27 | 46.45$\pm$0.42 | 19.12$\pm$1.02 | 19.55$\pm$0.30 | 13.90$\pm$0.52 | 22.55$\pm$0.83 | 7.24$\pm$0.71 | 9.96$\pm$0.28
+WU | 30.79$\pm$0.60 | 45.29$\pm$0.38 | 33.50$\pm$0.33 | 31.09$\pm$0.94 | 15.48$\pm$1.06 | 27.81$\pm$0.55 | 29.99$\pm$0.49 | 23.07$\pm$0.97
+TA+WU | 33.73$\pm$0.68 | 46.45$\pm$0.27 | 23.74$\pm$0.63 | 22.98$\pm$0.53 | 22.59$\pm$0.66 | 32.46$\pm$0.52 | 21.63$\pm$1.15 | 20.65$\pm$0.99
ANCL | 28.83$\pm$0.71 | 44.94$\pm$0.63 | 41.59$\pm$1.13 | 34.24$\pm$1.14 | 17.92$\pm$0.24 | 29.19$\pm$0.76 | 28.50$\pm$0.64 | 16.12$\pm$0.27
+TA | 33.92$\pm$0.68 | 47.49$\pm$0.35 | 25.89$\pm$0.82 | 22.61$\pm$0.40 | 23.52$\pm$0.73 | 29.64$\pm$0.50 | 15.06$\pm$0.78 | 14.67$\pm$0.84
+WU | 31.66$\pm$0.44 | 46.28$\pm$0.56 | 37.07$\pm$0.77 | 32.07$\pm$1.20 | 19.79$\pm$0.47 | 32.25$\pm$0.15 | 28.02$\pm$0.46 | 15.11$\pm$0.46
+TA+WU | 35.79$\pm$0.39 | 47.81$\pm$0.30 | 27.14$\pm$0.10 | 24.93$\pm$0.89 | 27.49$\pm$1.10 | 33.31$\pm$0.58 | 15.23$\pm$0.58 | 14.68$\pm$0.35

Table 10: Additional results for ImageNet100

###  D.2 Fine-grained classification using full datasets

We conduct experiments using the same datasets as in the Section 4.3.1, but
without sampling classes, so each of the 6 datasets is treated as a single
task. Additionally, we add the results for experiments with a reversed order
of tasks to ensure that the ordering does not affect our results. We report
final and incremental accuracy and forgetting in Table 11. We observe that
Teacher Adaptation (TA) improves the final accuracy, but applying warmup (WU)
in this setting results in significant drops in performance regardless of the
order of tasks.

| Basic order | Reverse order
---|---|---
| $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$ | $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$
GKD | 26.71$\pm$0.53 | 46.74$\pm$0.94 | 46.57$\pm$0.89 | 37.47$\pm$1.07 | 26.61$\pm$1.07 | 54.88$\pm$0.62 | 52.71$\pm$1.01 | 39.34$\pm$0.41
+TA | 31.16$\pm$0.75 | 46.38$\pm$0.74 | 44.09$\pm$0.68 | 37.84$\pm$0.98 | 33.61$\pm$0.40 | 54.97$\pm$0.35 | 41.89$\pm$0.64 | 36.63$\pm$0.58
+WU | 23.96$\pm$1.53 | 36.37$\pm$0.95 | 34.78$\pm$1.80 | 46.72$\pm$1.22 | 23.60$\pm$1.33 | 43.02$\pm$0.56 | 39.25$\pm$1.36 | 51.69$\pm$0.76
+TA+WU | 24.74$\pm$0.95 | 37.74$\pm$1.03 | 40.44$\pm$0.87 | 44.85$\pm$0.74 | 27.06$\pm$0.44 | 44.68$\pm$0.39 | 37.26$\pm$0.77 | 47.10$\pm$0.61

Table 11: Additional results for training on full fine-grained datasets in
standard and reversed order.

###  D.3 Fine-grained classification with reverse order of datasets

We show results for the experiments conducted in Section 4.3.1, extended with
additional metrics. Additionally, we add the results for the results with
reversed order of tasks, as in Section D.2. Similar to the results in the main
paper, the impact of our method is more visible in settings with a larger
number of tasks.

| 6 tasks, 20 classes each, base order | 6 tasks, 20 classes each, reverse order
---|---|---
| $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$ | $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$
GKD | 38.12$\pm$2.00 | 56.03$\pm$3.75 | 45.18$\pm$3.15 | 30.23$\pm$0.72 | 37.10$\pm$3.40 | 67.30$\pm$1.99 | 55.82$\pm$4.80 | 34.60$\pm$3.77
+TA | 43.11$\pm$0.64 | 57.24$\pm$1.79 | 39.58$\pm$0.80 | 33.12$\pm$1.76 | 46.97$\pm$3.94 | 67.19$\pm$4.81 | 41.20$\pm$4.55 | 33.94$\pm$7.78
+WU | 31.56$\pm$6.58 | 44.06$\pm$5.43 | 42.48$\pm$1.08 | 49.24$\pm$2.65 | 36.70$\pm$3.30 | 54.88$\pm$1.53 | 42.92$\pm$4.58 | 51.30$\pm$3.39
+TA+WU | 42.85$\pm$2.34 | 52.96$\pm$1.20 | 35.97$\pm$2.75 | 38.77$\pm$2.45 | 44.05$\pm$1.94 | 58.92$\pm$2.47 | 38.27$\pm$2.75 | 45.16$\pm$4.57
MKD | 41.32$\pm$0.81 | 60.12$\pm$1.59 | 40.30$\pm$3.98 | 26.42$\pm$1.60 | 43.96$\pm$3.89 | 70.65$\pm$1.67 | 42.83$\pm$5.11 | 25.52$\pm$3.43
+TA | 39.93$\pm$1.95 | 55.77$\pm$2.23 | 36.22$\pm$1.07 | 31.43$\pm$2.28 | 46.94$\pm$4.42 | 67.15$\pm$4.48 | 37.01$\pm$6.67 | 31.07$\pm$8.38
+WU | 34.65$\pm$2.72 | 48.01$\pm$1.97 | 44.86$\pm$0.61 | 47.07$\pm$2.40 | 35.49$\pm$5.51 | 56.83$\pm$2.79 | 46.14$\pm$6.67 | 48.15$\pm$5.95
+TA+WU | 44.91$\pm$2.51 | 56.41$\pm$1.04 | 35.53$\pm$2.19 | 34.47$\pm$0.71 | 44.62$\pm$2.85 | 61.25$\pm$3.28 | 38.31$\pm$3.91 | 41.48$\pm$6.24
TKD | 38.70$\pm$3.61 | 56.69$\pm$3.36 | 44.99$\pm$1.04 | 33.86$\pm$1.36 | 40.23$\pm$4.35 | 66.84$\pm$2.79 | 50.63$\pm$5.56 | 35.77$\pm$4.85
+TA | 42.34$\pm$0.77 | 57.99$\pm$1.88 | 39.97$\pm$0.52 | 33.79$\pm$1.80 | 46.86$\pm$5.55 | 67.18$\pm$5.28 | 39.90$\pm$6.90 | 33.67$\pm$8.84
+WU | 32.71$\pm$5.27 | 45.64$\pm$5.00 | 44.43$\pm$2.13 | 49.06$\pm$3.58 | 37.53$\pm$2.65 | 56.12$\pm$1.34 | 44.06$\pm$4.80 | 50.70$\pm$3.59
+TA+WU | 43.04$\pm$1.30 | 53.68$\pm$1.56 | 37.14$\pm$1.52 | 39.73$\pm$2.92 | 44.52$\pm$2.14 | 59.37$\pm$2.39 | 38.32$\pm$2.53 | 45.65$\pm$4.61
| 12 tasks, 10 classes each, base order | 12 tasks, 10 classes each, reverse order
| $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$ | $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$
GKD | 23.66$\pm$2.75 | 44.03$\pm$1.53 | 58.93$\pm$1.75 | 42.70$\pm$2.32 | 26.84$\pm$4.11 | 51.68$\pm$2.03 | 59.80$\pm$5.93 | 47.19$\pm$2.43
+TA | 36.46$\pm$0.94 | 50.80$\pm$1.27 | 43.53$\pm$1.96 | 37.58$\pm$3.01 | 41.15$\pm$5.50 | 59.51$\pm$4.40 | 42.61$\pm$6.36 | 36.14$\pm$6.20
+WU | 11.02$\pm$1.11 | 29.93$\pm$2.68 | 50.32$\pm$3.34 | 54.06$\pm$0.60 | 19.55$\pm$0.93 | 39.99$\pm$0.23 | 46.40$\pm$0.80 | 54.24$\pm$0.27
+TA+WU | 36.00$\pm$0.89 | 49.46$\pm$2.09 | 35.49$\pm$1.90 | 35.81$\pm$2.02 | 44.79$\pm$4.28 | 52.88$\pm$5.06 | 26.81$\pm$5.14 | 40.33$\pm$3.03
MKD | 25.35$\pm$4.36 | 45.19$\pm$2.91 | 55.04$\pm$2.71 | 43.81$\pm$1.56 | 31.81$\pm$5.29 | 53.63$\pm$2.31 | 52.38$\pm$6.99 | 43.86$\pm$3.05
+TA | 35.10$\pm$1.92 | 51.16$\pm$1.84 | 37.96$\pm$1.84 | 34.89$\pm$2.60 | 41.95$\pm$5.36 | 60.11$\pm$3.87 | 36.16$\pm$6.75 | 33.18$\pm$5.64
+WU | 15.61$\pm$6.67 | 32.93$\pm$3.65 | 49.31$\pm$7.17 | 54.64$\pm$1.52 | 26.19$\pm$8.31 | 42.21$\pm$1.31 | 44.49$\pm$6.28 | 54.20$\pm$1.67
+TA+WU | 34.65$\pm$4.00 | 50.40$\pm$3.57 | 36.79$\pm$1.04 | 35.74$\pm$3.45 | 45.91$\pm$4.64 | 54.05$\pm$4.55 | 27.74$\pm$4.92 | 40.08$\pm$3.32
TKD | 24.69$\pm$3.39 | 46.28$\pm$1.42 | 56.57$\pm$2.87 | 43.38$\pm$0.82 | 31.50$\pm$3.26 | 54.13$\pm$2.24 | 52.51$\pm$5.19 | 43.47$\pm$2.21
+TA | 35.06$\pm$1.42 | 51.66$\pm$1.63 | 39.72$\pm$2.08 | 35.43$\pm$2.75 | 41.40$\pm$4.78 | 59.22$\pm$3.67 | 37.10$\pm$5.97 | 34.27$\pm$5.88
+WU | 15.04$\pm$1.20 | 33.32$\pm$1.74 | 50.10$\pm$1.89 | 53.59$\pm$1.52 | 26.10$\pm$4.04 | 42.34$\pm$1.31 | 43.50$\pm$3.29 | 52.66$\pm$0.59
+TA+WU | 35.81$\pm$1.27 | 50.12$\pm$2.31 | 34.73$\pm$0.52 | 34.64$\pm$1.99 | 43.91$\pm$5.45 | 52.38$\pm$5.54 | 26.77$\pm$4.76 | 40.35$\pm$3.60
| 24 tasks, 5 classes each, base order | 24 tasks, 5 classes each, reverse order
| $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$ | $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$
GKD | 9.41$\pm$2.78 | 30.14$\pm$4.00 | 60.16$\pm$4.76 | 51.95$\pm$2.28 | 15.33$\pm$2.59 | 33.65$\pm$0.76 | 64.43$\pm$3.19 | 57.18$\pm$2.13
+TA | 26.92$\pm$2.31 | 42.86$\pm$1.82 | 44.80$\pm$3.25 | 39.01$\pm$1.55 | 29.52$\pm$2.33 | 48.34$\pm$1.51 | 48.32$\pm$2.43 | 41.58$\pm$3.30
+WU | 7.24$\pm$1.03 | 24.91$\pm$1.44 | 43.78$\pm$2.98 | 51.18$\pm$1.15 | 6.15$\pm$1.91 | 25.45$\pm$2.08 | 39.88$\pm$3.29 | 46.36$\pm$1.88
+TA+WU | 27.30$\pm$1.97 | 45.28$\pm$2.25 | 34.75$\pm$4.58 | 33.67$\pm$2.48 | 34.14$\pm$9.24 | 42.65$\pm$1.21 | 19.92$\pm$11.23 | 32.38$\pm$8.24
MKD | 11.57$\pm$2.70 | 32.61$\pm$0.55 | 63.70$\pm$2.58 | 55.91$\pm$1.77 | 15.53$\pm$2.96 | 34.85$\pm$1.40 | 62.28$\pm$3.64 | 56.57$\pm$2.11
+TA | 27.88$\pm$1.58 | 45.49$\pm$1.82 | 37.65$\pm$1.16 | 34.48$\pm$0.89 | 31.82$\pm$1.51 | 49.47$\pm$1.16 | 38.89$\pm$1.18 | 35.98$\pm$2.07
+WU | 7.52$\pm$4.24 | 24.62$\pm$1.56 | 44.85$\pm$6.75 | 52.59$\pm$2.59 | 4.57$\pm$3.36 | 21.36$\pm$4.30 | 40.82$\pm$4.75 | 51.73$\pm$0.78
+TA+WU | 26.71$\pm$2.29 | 45.80$\pm$2.48 | 33.52$\pm$4.66 | 32.48$\pm$2.15 | 30.63$\pm$6.78 | 41.09$\pm$4.04 | 20.99$\pm$4.34 | 31.64$\pm$3.87
TKD | 9.99$\pm$0.47 | 32.17$\pm$2.71 | 60.29$\pm$3.39 | 51.72$\pm$1.95 | 16.28$\pm$2.59 | 34.68$\pm$1.62 | 59.14$\pm$4.12 | 54.83$\pm$1.69
+TA | 26.24$\pm$3.62 | 43.95$\pm$2.60 | 36.28$\pm$2.17 | 33.28$\pm$1.86 | 30.81$\pm$1.61 | 48.95$\pm$0.95 | 38.10$\pm$1.53 | 35.45$\pm$1.59
+WU | 3.79$\pm$1.21 | 22.99$\pm$1.33 | 36.45$\pm$3.12 | 42.30$\pm$4.03 | 8.62$\pm$4.87 | 24.91$\pm$6.55 | 39.22$\pm$7.44 | 49.71$\pm$2.49
+TA+WU | 20.24$\pm$4.08 | 37.22$\pm$6.11 | 23.39$\pm$2.94 | 24.53$\pm$3.01 | 25.21$\pm$3.57 | 47.46$\pm$5.21 | 28.40$\pm$4.04 | 26.15$\pm$2.93

Table 12: Additional results for the fine-grained classification datasets,
extended by the experiments with reversed order of the datasets.

###  D.4 DomainNet

| 6 tasks 50 classes with $\lambda=10$
---|---
| Trained from scratch | Pre-trained on Imagenet
| $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$ | $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$
GKD | 3.43$\pm$0.45 | 18.63$\pm$0.27 | 31.05$\pm$0.42 | 23.27$\pm$0.26 | 27.69$\pm$0.09 | 43.27$\pm$0.10 | 41.61$\pm$0.71 | 36.83$\pm$0.88
+TA | 6.73$\pm$0.82 | 19.55$\pm$0.42 | 36.26$\pm$0.55 | 27.22$\pm$0.21 | 30.12$\pm$0.23 | 43.52$\pm$0.17 | 43.38$\pm$0.39 | 38.33$\pm$0.41
+TA+WU | 5.83$\pm$0.70 | 18.68$\pm$0.36 | 33.00$\pm$0.59 | 31.12$\pm$0.38 | 28.83$\pm$0.26 | 39.00$\pm$0.12 | 42.64$\pm$0.56 | 49.44$\pm$0.37
ANCL | 7.33$\pm$0.91 | 19.58$\pm$0.46 | 33.28$\pm$0.18 | 25.63$\pm$0.20 | 25.58$\pm$0.62 | 42.90$\pm$0.84 | 45.01$\pm$1.19 | 37.28$\pm$1.86
+TA | 8.85$\pm$0.25 | 20.34$\pm$0.40 | 39.70$\pm$0.38 | 30.73$\pm$0.44 | 26.95$\pm$0.07 | 42.67$\pm$0.51 | 41.62$\pm$0.52 | 38.56$\pm$1.24
+TA+WU | 9.65$\pm$0.40 | 20.39$\pm$0.09 | 37.57$\pm$0.12 | 32.83$\pm$0.66 | 29.15$\pm$0.98 | 42.44$\pm$0.80 | 34.23$\pm$0.85 | 37.09$\pm$1.33
| 12 tasks 25 classes with $\lambda=10$
| Trained from scratch | Pre-trained on Imagenet
| $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$ | $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$
GKD | 2.48$\pm$0.41 | 14.45$\pm$0.25 | 29.01$\pm$0.63 | 29.04$\pm$0.37 | 17.69$\pm$0.84 | 35.98$\pm$0.96 | 50.12$\pm$1.00 | 43.00$\pm$1.66
+TA | 4.88$\pm$0.46 | 16.25$\pm$0.46 | 36.47$\pm$1.06 | 33.03$\pm$0.19 | 28.02$\pm$0.68 | 38.89$\pm$0.52 | 40.87$\pm$0.64 | 41.10$\pm$0.42
+TA+WU | 6.37$\pm$1.39 | 16.22$\pm$0.45 | 27.45$\pm$0.70 | 32.63$\pm$0.11 | 27.97$\pm$0.38 | 34.24$\pm$0.42 | 30.86$\pm$0.33 | 44.17$\pm$0.37
ANCL | 3.97$\pm$0.50 | 14.82$\pm$0.41 | 32.80$\pm$0.61 | 33.46$\pm$0.40 | 14.13$\pm$1.35 | 33.34$\pm$0.55 | 55.72$\pm$0.16 | 49.05$\pm$0.65
+TA | 6.54$\pm$0.51 | 17.19$\pm$0.06 | 39.77$\pm$0.76 | 37.40$\pm$0.38 | 23.42$\pm$1.03 | 35.81$\pm$0.18 | 40.86$\pm$0.85 | 43.84$\pm$0.23
+TA+WU | 8.06$\pm$0.47 | 18.12$\pm$0.07 | 36.07$\pm$0.13 | 36.71$\pm$0.05 | 27.83$\pm$1.36 | 36.73$\pm$1.78 | 31.00$\pm$0.39 | 39.29$\pm$0.49
| 6 tasks 50 classes with $\lambda=1$
| Trained from scratch | Pre-trained on Imagenet
| $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$ | $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$
GKD | 5.52$\pm$2.11 | 18.80$\pm$0.75 | 39.48$\pm$1.12 | 32.74$\pm$0.60 | 19.93$\pm$0.47 | 36.56$\pm$0.12 | 58.83$\pm$0.96 | 56.75$\pm$0.89
+TA | 11.42$\pm$0.30 | 20.84$\pm$0.24 | 42.92$\pm$0.49 | 34.98$\pm$0.19 | 25.50$\pm$0.86 | 39.01$\pm$0.60 | 56.75$\pm$0.90 | 54.99$\pm$0.93
TKD | 4.87$\pm$0.97 | 18.94$\pm$0.36 | 32.95$\pm$0.78 | 26.60$\pm$0.29 | 27.44$\pm$0.15 | 42.18$\pm$0.41 | 42.47$\pm$0.65 | 41.23$\pm$1.05
+TA | 6.76$\pm$1.03 | 19.67$\pm$0.19 | 37.43$\pm$0.57 | 29.91$\pm$0.09 | 28.96$\pm$0.32 | 42.51$\pm$0.43 | 43.91$\pm$0.16 | 42.00$\pm$0.59
+TA+WU | 5.88$\pm$0.82 | 18.65$\pm$0.27 | 33.50$\pm$0.96 | 32.33$\pm$0.49 | 29.55$\pm$0.30 | 40.62$\pm$0.07 | 42.08$\pm$0.55 | 46.86$\pm$0.39
MKD | 3.24$\pm$1.10 | 18.74$\pm$0.52 | 26.83$\pm$0.83 | 19.10$\pm$0.40 | 30.41$\pm$0.47 | 45.70$\pm$0.30 | 32.96$\pm$0.29 | 27.48$\pm$0.60
+TA | 3.99$\pm$0.38 | 18.04$\pm$0.16 | 29.16$\pm$0.29 | 22.70$\pm$0.25 | 27.47$\pm$0.22 | 42.91$\pm$0.04 | 31.05$\pm$0.31 | 29.43$\pm$0.04
+TA+WU | 6.26$\pm$0.02 | 19.18$\pm$0.13 | 31.36$\pm$1.04 | 27.81$\pm$0.49 | 30.10$\pm$1.07 | 42.20$\pm$0.65 | 34.50$\pm$0.41 | 38.01$\pm$0.40
| 12 tasks 25 classes with $\lambda=1$
| Trained from scratch | Pre-trained on Imagenet
| $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$ | $Acc_{Final}\uparrow$ | $Acc_{Inc}\uparrow$ | $Forg_{Final}\downarrow$ | $Forg_{Inc}\downarrow$
GKD | 2.88$\pm$0.25 | 14.38$\pm$0.82 | 38.89$\pm$3.42 | 39.78$\pm$2.23 | 9.62$\pm$0.56 | 27.99$\pm$0.40 | 64.74$\pm$0.21 | 59.93$\pm$1.13
+TA | 6.80$\pm$0.28 | 17.02$\pm$0.19 | 47.37$\pm$0.82 | 43.44$\pm$0.16 | 19.00$\pm$0.83 | 33.90$\pm$0.62 | 58.20$\pm$0.71 | 54.97$\pm$0.71
TKD | 3.39$\pm$0.43 | 12.28$\pm$0.99 | 41.22$\pm$0.28 | 33.32$\pm$0.33 | 22.66$\pm$1.59 | 37.56$\pm$1.03 | 41.65$\pm$1.73 | 41.21$\pm$1.97
+TA | 5.10$\pm$0.47 | 16.84$\pm$0.42 | 34.03$\pm$0.61 | 33.46$\pm$0.41 | 29.39$\pm$0.37 | 38.85$\pm$0.31 | 34.51$\pm$0.54 | 39.91$\pm$0.39
+TA+WU | 6.27$\pm$0.52 | 16.40$\pm$0.56 | 26.51$\pm$0.57 | 32.87$\pm$0.34 | 29.26$\pm$2.05 | 36.81$\pm$0.97 | 30.00$\pm$1.48 | 41.98$\pm$0.95
MKD | 2.32$\pm$0.39 | 13.45$\pm$0.53 | 23.92$\pm$1.02 | 27.23$\pm$0.98 | 24.55$\pm$0.20 | 39.14$\pm$0.21 | 38.09$\pm$1.02 | 36.53$\pm$1.00
+TA | 3.60$\pm$0.51 | 15.30$\pm$0.35 | 27.32$\pm$0.37 | 28.71$\pm$0.26 | 27.49$\pm$0.94 | 37.84$\pm$0.35 | 27.99$\pm$1.16 | 34.09$\pm$0.39
+TA+WU | 5.46$\pm$1.20 | 15.89$\pm$0.96 | 22.23$\pm$1.98 | 30.62$\pm$0.48 | 30.34$\pm$1.32 | 37.79$\pm$1.11 | 24.81$\pm$1.05 | 37.00$\pm$1.84

Table 13: Additional results for DomainNet.

We present the full experimental results for DomainNet in Table 3. In addition
to incremental accuracy and forgetting, we also add final accuracy and
forgetting to the metrics in the experiments. Additionally, we analyze the
random initialization and impact of starting from pretrained checkpoint on
ImageNet. For task-wise knowledge distillation (TKD) we apply $\lambda$ equal
to 1, for global knowledge distillation we notice that $\lambda=10$ works
better. We set the learning rate to $0.01$ and use the scheduler with LR decay
after 60, 120 and 180 epochs.