# Continual Learning in the Presence of Repetition

Hamed Hemati  University of St. Gallen, Switzerland  Lorenzo Pellegrini
University of Bologna, Italy  Xiaotian Duan* University of Chicago, United
States  Zixuan Zhao* University of Chicago, United States  Fangfang Xia*
University of Chicago, United States
Marc Masana* Graz University of Technology, Austria  Benedikt Tscheschner*
Graz University of Technology, Austria  Eduardo Veas* Graz University of
Technology, Austria  Yuxiang Zheng* Nanjing University of Aeronautics and
Astronautics, China  Shiji Zhao* Nanjing University of Aeronautics and
Astronautics, China
Shao-Yuan Li* Nanjing University of Aeronautics and Astronautics, China
Sheng-Jun Huang* Nanjing University of Aeronautics and Astronautics, China
Vincenzo Lomonaco  University of Pisa, Italy  Gido M. van de Ven  KU Leuven,
Belgium

###### Abstract

Continual learning (CL) provides a framework for training models in ever-
evolving environments. Although re-occurrence of previously seen objects or
tasks is common in real-world problems, the concept of _repetition_ in the
data stream is not often considered in standard benchmarks for CL. Unlike with
the rehearsal mechanism in buffer-based strategies, where sample repetition is
controlled by the strategy, repetition in the data stream naturally stems from
the environment. This report provides a summary of the CLVision challenge at
CVPR 2023, which focused on the topic of repetition in class-incremental
learning. The report initially outlines the challenge objective and then
describes three solutions proposed by finalist teams that aim to effectively
exploit the repetition in the stream to learn continually. The experimental
results from the challenge highlight the effectiveness of ensemble-based
solutions that employ multiple versions of similar modules, each trained on
different but overlapping subsets of classes. This report underscores the
transformative potential of taking a different perspective in CL by employing
repetition in the data stream to foster innovative strategy design. ***Please
refer to Appendix A for details about author contributions.
$\,~{}\,~{}~{}~{}~{}^{@}$Correspondence to<EMAIL_ADDRESS>(H. Hemati)
and<EMAIL_ADDRESS>(G.M. van de Ven)

_Keywords_ Continual learning $\cdot$ Class-incremental learning $\cdot$
Repetition $\cdot$ Competition

##  1 Introduction

Designing systems that can learn and accumulate knowledge over time in non-
stationary environments is an important objective in artificial intelligence
research [1, 2, 3]. In traditional machine learning, however, the common
practice is to build and train models based on statistical learning
assumptions. Under these assumptions, a model has access to a static dataset
with samples that are independent and identically distributed (IID). This
implies that both the training set and test set come from the same
distribution and remain unchanged throughout the training and evaluation
processes. In reality, these assumptions are often violated, and the model is
exposed to different forms of shift in the data. To address such shifts,
continual learning (CL) offers a framework to simulate the “never-ending”
learning setting [4]. In CL, a model is exposed to a data stream consisting of
a potentially unbounded sequence of experiences [5, 6, 7]. Unlike under the
statistical learning assumptions, in CL, the model does not have full access
to a static dataset. Instead, it gets non-IID access to the data distribution
in the form of a data stream, as data become partially available in an
incremental manner. In a specific variant of CL, referred to as batch
continual learning or task-based continual learning, the model gets locally
IID access to part of the data distribution. These locally IID parts of the
data stream have been referred to as tasks, contexts, or experiences.

A central objective in CL is to design models that mitigate catastrophic
forgetting [8, 9]. Catastrophic forgetting refers to the phenomenon where a
model’s performance on a previously learned task rapidly and drastically
declines as the model continues to train on data from other tasks. To tackle
the issue of forgetting in neural networks, researchers have proposed various
strategies. Some strategies add regularization terms to the loss to penalize
changes to parts of the model important for past tasks [10, 11, 12]. Other
strategies replay samples from previous experiences by either employing an
external buffer to store raw samples or by using a generative model to
synthesize samples similar to those seen previously [13, 14, 15, 16, 17].
While replay strategies often yield good performance, they come with certain
drawbacks, such as data privacy concerns and high costs in terms of
computation or memory, which can become prohibitive for extended streams. In
another CL strategy, parameters are added to increase the capacity of the
model to ease the learning of new tasks while avoiding interference with older
tasks [18, 19, 20].

An aspect that is not often considered in the CL literature is the repetition
of previously seen concepts over time. CL research often uses “academic
scenarios” in which tasks, domains or classes are presented in a strictly
sequential manner without repetition of previously seen ones [21, 22]. (The
terms task-, domain- and class-incremental learning learning are sometimes
even interpreted as excluding the possibility of repetition, but see [23].)
However, in real-world problems, the re-occurrence of previously encountered
concepts is common. For example, in a practical scenario where an agent
explores a large environment consisting of many areas or rooms over time and
learns about new concepts through observation, the agent may revisit similar
concepts in different ways as it explores the world. Consequently, the
repetition of concepts can naturally happen during the lifetime of the agent,
which may lead to an improvement in understanding the concept as it re-appears
in different contexts.

Adding the dimension of repetition to a CL problem raises questions such as
‘How does the amount of repetition in a stream affect the behavior of
different strategies?’ or ‘What type of strategies are better for streams with
repetition?’. Considering repetition can also reveal new potential for
developing CL strategies. For example, it can lead to the development of
solutions that are more efficient when deployed for settings where repetition
of patterns follows some regularities. It is important to point out that
repetition in the data stream is fundamentally different from using a memory
buffer to replay past experiences. When using a memory buffer, the strategy
controls the repetition of samples or concepts, and typically, samples from
all previous experiences are replayed alongside new data. However, in streams
with repetition, it is the environment or the underlying generation factors of
the stream that enforce the recurrence of a concept. Thus, the model might
only be presented with a subset of the previously encountered concepts. Some
previous works, such as [24, 25], have started to empirically evaluate CL
methods in streams with a large number of tasks, where repetition can happen.
The role and importance of repetition for learning have also been investigated
from a neuroscience perspective. For instance, [26] explores how repetition
can affect memory performance and brain activity for associative memory over
time. In other studies [27, 28], the authors focus on the spacing effect,
which concerns how spaced repetition can enhance memory retention compared to
massed repetition.

In recent years, several workshops on CL have organized challenges. For
example, the first edition of the CLVision challenge at CVPR 2020 introduced
new benchmarks based on the CORe50 dataset [29]. Although the ‘New Instances
and Classes’ scenario [30] that was used in that challenge includes some form
of repetition of past classes, it lacks a principled approach to the study of
the problem. In the 2021 edition of the CLVision challenge, the focus was
mainly on supervised class-incremental classification and continual
reinforcement learning [31]. In that same year, in a challenge at ICCV 2021,
there was a track on domain-incremental object classification and detection
based on a benchmark for autonomous driving called CLAD [32]. Finally, the
CLVision challenge at CVPR 2022 [33] emphasized the design of strategies for
object detection and classification using the large EgoObjects dataset [34].

For the CLVision challenge at CVPR 2023, the primary goal was to explore the
role of repetition in CL, a topic not often explored by the community. To do
so, the participants were asked to design strategies that could exploit the
repetition inherent in the stream to boost knowledge transfer and reduce
forgetting without storing raw samples. This report provides a summary of this
challenge. Details about the design of the challenge are provided in section
2. Solutions proposed by finalist teams are presented in sections 3 to 5.
Section 6 presents the results of the challenge, and section 7 discusses the
main lessons learned.

##  2 Challenge Details

Participants in the challenge were asked to design strategies for a class of
CL problems referred to as class-incremental with repetition (CIR). CIR
encompasses a variety of streams with two key characteristics:

(i) New classes can appear over time.

(ii) Previously encountered classes can re-occur with varying repetition
patterns.

Since many existing strategies in the CL literature have only been tested in
repetition-free settings, their efficacy and performance in the presence of
repetition remain unclear. To investigate the influence of repetition in the
data stream on the relative effectiveness of different strategies, a set of
CIR benchmarks was generated using a sampling-based data stream generator
controlled by four interpretable parameters (see subsection 2.1). Participants
were tasked with developing strategies that, upon completing training on the
entire stream, could achieve high average accuracy on a test set containing
unseen samples from the same classes present in the stream. To allow
participants flexibility in strategy design without the need for significant
computational resources, we used CIFAR-100 [35], a relatively small-scale
dataset but still containing natural images, as the base dataset for the
stream generator.

The competition consisted of two phases:

#### Pre-selection Phase

The pre-selection phase took place between the 20th of March 2023 and the 20th
of May 2023. For this phase, there were three challenge data streams, which
were released on the GitHub
repository†††https://github.com/ContinualAI/clvision-challenge-2023 of the
challenge. The participants were asked to submit their solutions in the form
of model predictions on the test set after training on each stream. Appendix B
provides details on the participation over time in the pre-selection phase.

#### Final Evaluation Phase

The final phase started from the 20th of May 2023. In this phase, the five
highest-ranking participants from the pre-selection phase were asked to send
the source code of their strategies. Their strategies were tested on three new
data streams, and the average accuracy on the test set after training on each
of these three new streams was used as the metric for the final ranking.
Participants were allowed to make changes to their strategies between the pre-
selection phase and the final evaluation phase.

###  2.1 Stream Generator

The sampling-based generator developed by Hemati et al. [36] was used to
generate the data streams for this challenge using four control parameters
with clear interpretations:

  * •

Stream length ($N$): Number of experiences in the stream.

  * •

Experience size ($S$): Number of samples in each experience.

  * •

First occurrence distribution ($P_{f}$): A discrete probability distribution
over the experiences in the stream that determines the first occurrence of
each class.

  * •

Repetition probability ($P_{r}$): Per-class repetition probabilities that
control the likelihood of each class re-appearing in future experiences in the
stream after its first occurrence.

Figure 1: Illustration of transitioning from “standard Class-IL” to
“cumulative Class-IL” by increasing the probability of repetition for seen
classes, which is set by control parameter $P_{r}$. Class-IL: class-
incremental learning.

By default, an equal number of samples is assigned to all classes present in
each experience. $P_{f}$, which controls the “first occurrence” property of
the stream, determines when dataset classes make their initial appearance. For
instance, in one stream, all classes might appear for the first time at the
outset, whereas in another stream, new classes might appear gradually. Once a
class has appeared for the first time, it re-appears based on a per-class
repetition probability, which is controlled by $P_{r}$. Figure 1 illustrates
how increasing the repetition probability from $0$ to $1$ corresponds to a
transition from “standard” class-incremental learning, where each class is
present in only one experience, to “cumulative” class-incremental learning,
where each class re-appears in each experience after its first occurrence.
While the repetition probability can be dynamically set based on a time-
variant probability mass function, in this challenge, the repetition
probability for each class remains fixed according to a specific pattern. This
means that, in this challenge, $P_{r}$ is a list of probability values, with
one value assigned to each class in the dataset.

In Figure 2, three different examples of streams generated using the generator
are shown, with a dataset containing 30 classes and $N$ set to $50$. In these
examples, $P_{f}$ is varied while $P_{r}$ is kept constant to illustrate the
effect of first occurrence on the generated streams.

(a) $P_{f}:\text{Geom}(0.01)$, $P_{r}:\text{Zipf}(0.5)$

(b) $P_{f}:\text{Geom}(0.3)$, $P_{r}:\text{Zipf}(0.5)$

(c) $P_{f}:\text{Geom}(0.9)$, $P_{r}:\text{Zipf}(0.5)$

Figure 2: Examples of generated streams with a Geometric distribution over the
first occurrences of the dataset’s classes, and a Zipfian distribution for the
repetition of classes along the stream.

#### Selected Challenge Stream Parameters

For the challenge, the primary focus was on the effect of $P_{f}$ and $P_{r}$
in the learning process of a strategy. The settings used for $P_{f}$ and
$P_{r}$ in each of the challenge data streams are provided in Appnedix C. The
number of experiences ($N$) in each stream was fixed as $50$, and the number
of samples in each experience ($S$) was set to $2000$. Following the default
setting, the number of samples in each experience was equally distributed
across the classes present in the experience.

###  2.2 Participation

Participation was in the form of teams. Team registrations and solution
submissions were handled using the CodaLab platform [37]. Each team was
allowed to create only one account on the platform to prevent parallel
submissions. Participants were provided with a DevKit based on the Avalanche
library [38]: https://github.com/ContinualAI/clvision-challenge-2023/. The
DevKit contains all necessary scripts and codes to load the benchmarks for the
challenge configurations. Researchers who are interested in evaluating the
performance of their own strategies can use the DevKit for both pre-selection
and final phase streams.

###  2.3 Rules and Restrictions

Submission: Each submission in the pre-selection phase consisted of three sets
of predictions. Each set of predictions was made by testing the model against
the challenge’s test set, after training the designed strategy on one of the
three challenge streams.

Strategy Design: Within each experience in a data stream, users had full
access to the training data from that experience, but not to the data from
other experiences. In the default settings of the DevKit, the number of epochs
per experience was set to 20. Participants had the flexibility to tweak and
tailor the epoch iterations and dataset loading. For instance, a possible
strategy would be to iterate for more epochs in the initial experiences and
fewer in the final ones.

Model Architecture: All participants were required to utilize the
(Slim-)ResNet-18 provided in the DevKit as the base architecture for their
models. However, they were allowed to use multiple copies and to add
additional modules, e.g. gating modules, provided they adhered to the maximum
GPU memory usage (4000 MB) allowed for the competition during training. The
GPU memory limit is set to almost eight times the memory size needed to train
the Slim-ResNet-18 backbone with a batch size of 64. This allows for
considerable flexibility in adding additional modules to the backbone.

Replay Buffer: Using a replay buffer to store dataset samples was not allowed.
However, buffers could be used to store any form of data representation, such
as the model’s internal representations of data samples. Irrespective of the
buffer type, the buffer size (i.e., the total number of stored exemplars) was
not allowed to exceed 200.

Hardware Limitations: To encourage a comparison between strategies that use
similar computational resources, participants were limited to using one GPU
for training, and the maximum training time for each run was capped at 500
minutes. No computing limitations were specified for the inference time.

##  3 Strategy 1: HAT-CIR

The strategy proposed by team _xduan7_ is called HAT-CIR. This strategy
combines the strengths of Hard Attention to the Task (HAT) [39] and Supervised
Contrastive Learning (SupCon) [40], along with other elements, such as network
replicas and test-time decision-making, to achieve the best performance in CIR
streams. The implementation is publicly available on GitHub:
https://github.com/xduan7/clvis-chlg-2023/.

###  3.1 Motivation and Related Work

In CIR, as more generally with class-incremental learning, comparing model
output logits across different classes is difficult because images from
different classes are not jointly used for training. Potential solutions
include: (i) using a buffer and replaying samples to calibrate the logits (but
due to restrictions on the use of replay, this approach is not well suited for
this challenge), and (ii) employing out-of-distribution (OOD) detection
methods for “task-agnostic inference”.

Additionally, strategies for mitigating forgetting, such as parameter
isolation techniques, offer promising complementary approaches. Parameter-
isolation methods divide the neural network into segments, each responsible
for a specific task or experience, preventing the neural network from
forgetting knowledge acquired from prior experience. For example, HAT employs
trainable binary masks to partition the network based on the experience
identifier. This has proven effective, particularly in task-incremental
learning problems where the experience identifier is provided during both
training and test phases. However, it encounters challenges when applied to
the CIR setting, where experience identifiers are unavailable during testing.

To address the experience identifier limitation in parameter-isolation methods
within the CIR setting, one might employ OOD detection techniques. These can
help to detect samples that are not from the current experience, as for
example demonstrated in [41] and [42]. On the other hand, representation
learning methods like SupCon can be employed to make the output logits more
comparable across different classes. This combination has been explored in
previous works (e.g., [41]) and has been shown to be effective when applied to
the class-incremental learning setting.

In light of these existing works, and considering the specific challenges
posed by the competition, the subsequent sections will elaborate on how to
adapt and integrate these strategies to propose a novel solution for the CIR
setting.

###  3.2 Method Description

The proposed method comprises three core components: (i) structural design,
featuring HAT-based partitioning and network replicas, (ii) a two-phase
training strategy, including supervised contrastive learning and
classification, as illustrated in Figure 3, and (iii) a momentum-based
inference mechanism for test-time decision making.

Figure 3: The two training phases of HAT-CIR. Top: The supervised contrastive
learning phase, where the network is trained to learn better feature
representations. Bottom: The classification phase, where the network is
trained to classify the samples in the current experience.

#### HAT-based Partitioning

Training sequentially over multiple experiences presents challenges due the
risk of catastrophic forgetting. To mitigate this challenge, HAT is employed,
isolating network parameters based on experience IDs. However, the original
HAT method suffers from slow training and hyperparameter sensitivity when
handling a large number of experiences. To overcome this, HAT-CL [43] is
adapted, which initializes masks to ones and uses a cosine mask scaling curve
to facilitate better alignment with network weights. By using the cosine mask
scaling curve, each training epoch is divided into three phases:

  1. 1.

Train weights while masks are mostly ones

  2. 2.

Train masks and weights together to make masks more sparse

  3. 3.

Fine-tune weights while masks are mostly ones again

These changes significantly improve the training speed and stability, and
performance of HAT. Furthermore, the effect of the regularization term for the
masks is gradually reduced. This step ensures full network capacity
utilization and provides an accounting for the number of classes in each
experience through variable regularization terms. It is important to note that
the HAT-based partitioning was only used in the pre-selection phase; for the
final phase, only network replicas were used, which led to higher performance.

#### Network Replicas

Two types of network replicas, fragments and ensembles, are used to increase
the model capacity and improve performance. Fragments refer to network
replicas trained on different experiences. For example, if there are ten
experiences, five fragments might be trained, each on two experiences.
Ensembles are replicas trained on the same experiences. For each experience,
multiple ensemble replicas will be trained on the same samples with different
initializations and data augmentation. Thus, their predictions are averaged
for final decision-making. These two replica types are complementary and can
be combined with or without HAT-based partitioning. Importantly, adding
fragment replicas does not increase the computational costs during training,
while adding ensemble replicas does. Due to the challenge’s restriction on
memory usage and training time, the final version of the model used for the
experiments employed 50 fragments and two ensembles. It is important to note
that during the pre-selection phase, network replicas were not yet used.

#### Supervised Contrastive Learning

At the beginning of each experience, the network is trained with supervised
contrastive learning to learn better feature representations. The objective is
to maximize the similarity between feature vectors of the same class and
minimize those of different classes with the following loss function:

$L=\sum_{i=1}^{N}\max\left(0,D(f(x_{i}^{a}),f(x_{i}^{p}))-D(f(x_{i}^{a}),f(x_{i}^{n}))+\alpha\right)$

In this equation, $f(x)$ produces the embedded feature vector for input $x$,
$D(x,y)$ denotes a distance function, $x_{i}^{a}$, $x_{i}^{p}$, and
$x_{i}^{n}$ represent the anchor, positive, and negative samples,
respectively, and $\alpha$ is a margin parameter. The number of samples in a
batch is denoted by $N$. It is also worth noting that, the training of hard
attention masks occurs exclusively in this phase due to their sensitivity to
learning rates and epoch numbers.

#### Momentum-based Test-time Decision Making

During inference, each input is processed by multiple model “snapshots” that
were originally trained on data consisting of the same class as the input.
Therefore, a specific weighted average of logits is calculated for each class,
aggregating information from its last $N$ appearances and giving higher weight
to more recent appearances. This proposed approach, called “momentum-based
decision making”, achieved strong performance in this challenge by setting
$N=3$, using the weights $[1,2,3]$ for the last three experiences in which
each class appeared.

#### Other Details

Standard techniques such as data augmentation, early stopping, and learning
rate scheduling were explored. Notably, it was found that data augmentation
during both training and inference was crucial for improving performance. For
the final submission, a random combination of RandomResizedCrop,
RandomHorizontalFlip, and ColorJitter transformations were used.

Further details, consisting of an ablation study and an analysis of the
influence of the number of network replicas, can be found in Appendix D.

###  3.3 Discussion

In this section, the limitations and possible improvements for the method are
critically assessed, as well as the implications of the findings for future
research.

#### Limitations

A significant drawback of the approach arises when the number of classes is
small in the initial experiences. In such cases, the network fails to learn
effective class representations due to the limited diversity. This negatively
impacts the performance of the momentum-based test-time decision-making
strategy. Another inherent limitation is related to the use of HAT. The
rigidity of the network structure requires careful hyperparameter tuning to
match the expected total number of experiences, which is not always known in
advance in CIR settings. As a result, suboptimal parameter allocation might be
achieved, leading to compromised performance.

#### Future Directions

The experiments show that the method’s performance scales positively with the
number of replicas, implying that network capacity plays a significant role.
Utilizing larger networks, possibly incorporating different structures with
pre-trained weights, might offer avenues for further improvement in
performance.

##  4 Strategy 2: Horde

The strategy proposed by team _mmasana_ is called Horde. The core idea of this
strategy relies on learning an ensemble of feature extractors (FEs) on
selected experiences, which should provide robust features useful for
discriminating between seen and unseen downstream classes. After training, the
learned FEs are frozen to obtain a zero-forgetting ensemble. However, since
not all classes are present in each experience, the outputs need alignment to
balance them effectively. The strategy uses pseudo-feature projection on the
outputs to retain as much discrimination between classes as possible; while
learning a unified head capable of discriminating between all classes seen so
far. To further facilitate the pseudo-feature projection, FEs are trained with
both the usual cross-entropy loss and an additional metric learning loss,
which promotes alignment among the learned classes within each feature space.
Furthermore, due to the competition setting, where not all classes are
available or known from the beginning, this strategy requires to manage FEs
across all the experiences sequence. This includes the addition and removal of
FEs from the total budget, while balancing their learning of the unified head.

###  4.1 Motivation and Related Work

The motivation for the proposed approach stems from the attempt to balance the
stability-plasticity dilemma [44]. This balance involves retaining useful
knowledge from previous experiences while learning new ones from the sequence.

The most support for stability is provided by zero-forgetting methods, which
freeze the feature extractor part of model after training, thus retaining all
knowledge up to that point [6, 45]. As new experiences are learned, these
methods usually reduce catastrophic forgetting by extending the model [19, 18,
46] or by applying masks to the parameters or the intermediate representations
[47, 48, 49]. The set up of the challenge excludes the use of the experience
ID at test time and the use of pretrained models, which limits the direct
application of existing methods. However, the enforcement of the stability
component via freezing is preserved for Horde and represented with the use of
an ensemble of FEs that retain all learned knowledge.

Robust and discriminative representations [50] are also important for
plasticity, as they help adapt to new classes and promote generalization (or
forward transfer). Therefore, the proposed approach attempts to strike a
balance between both stability and plasticity, with a priority on retaining
useful knowledge, in an attempt to limit excessive model growth as
representations from the ensemble cover the space better.

Initial inspiration was found by identifying some of these characteristics in
both Feature Translation for Exemplar-Free Class-Incremental Learning (FeTrIL)
[51] and Ternary Feature Masks (TFM) [49]. FeTrIL [51] uses a pretrained
frozen backbone to benefit from the stability of having zero-forgetting for
that part of the model, while the plasticity is introduced via the learning of
a unified head. This method has a pseudo-feature generator that uses the
representations from a single FE and applies a geometric translation to align
both past and new classes. The pretrained backbone could be replaced by
learning on the first experience, although, the method heavily relies on the
initial training covering a majority of classes such that the resulting
feature extractor is expressive and discriminative enough. TFM [49] freezes
the existing model and extends it at each experience while reusing the
representations from previous experiences. This method applies masking on the
outputs of each layer to control the flow of gradients so that previous
knowledge can be used but its corresponding weights are not modified. However,
the dynamic architecture requires the experience ID at test time. Both methods
break the challenge rules in multiple ways when used as proposed in their
original works.

Horde is inspired by these two methods by having a dynamic zero-forgetting
architecture for the ensemble and making use of a pseudo-feature generation
approach for learning the unified head. The approach is adapted to the
challenge by extending the pseudo-feature generation with the usage of the
standard deviation, and using the contrastive loss [52, 53] to improve the
learned shape of the representations, and avoiding the issues for the pseudo-
feature generation. Furthermore, the proposed pseudo-feature generation method
is adapted to the challenge scenarios that contain class repetition.

###  4.2 Method Description

The proposed strategy Horde combines the feature representations of individual
FEs into a single unified head capable of predicting all classes seen so far.
This is achieved through a two-step process: first, the learning of an FE
(based on stability), and second, the pseudo-feature alignment for adapting
the unified head (based on plasticity). Each individual FE is an expert model
trained on a single experience and thus possesses knowledge only of its
corresponding classes. Therefore, during the first step, to ensure full
stability (zero-forgetting), each of these expert models is frozen after
training and added to the ensemble. In the second step, data are passed
through all the ensemble’s models, and only the unified head is fine-tuned.
Training the unified head involves using representations directly from the
expert FEs familiar with a class and the pseudo-feature projections from the
representations of FEs not trained in that class.

Figure 4: The panels indicate for stream 1 (left) and stream 2 (right) in
which experiences Horde added a new FE (indicated by bright colors), while
comparing them based on three factors: the total number of classes in the
experience (blue), the number of classes in the experience on which no FE had
been trained yet (yellow), and the number of new, unseen classes in the
experience (red). The black line represents the total amount of classes seen
so far. Figure 5: Feature extractor training in Horde is done by using both a
metric learning and a cross-entropy head. Figure 6: Using pseudo-feature
alignment, the unified head in Horde is trained to discriminate between all
seen classes based on the features from the ensemble.

When to add a new FE to the ensemble: For the specific setting of the
competition, two constraints (heuristics) were designed to determine when to
add an FE to the ensemble. First, to constrain the presence of overly
overfitted FEs and to limit the size of the ensemble, experiences with fewer
than five classes are not considered. Second, the addition of FEs to the
ensemble is stopped after 85% of the classes have been seen, because once
robust features for most classes have been learned, good performance on the
remaining ones is expected. Additionally, an FE is always trained on the first
experience. With these constraints, the proposed approach learns feature
extractors based on the highlighted experiences shown in Figure 4.

Feature extractor training: When an FE is trained on the current experience,
learning occurs with the usual cross-entropy loss on a fully connected head
with as many outputs as classes. In order to promote the learning of features
in a more similarly distributed space, a contrastive loss with emphasis on the
hard negative pairs is also included on a separate head, as shown in Figure 5.
Both losses are balanced with an adaptive alpha, which is automatically
computed based on the energy of each loss. After learning that experience, the
heads are removed and the backbone is frozen and added to the rest of the
ensemble.

Pseudo-feature projection: Regardless of an FE being trained and added to the
ensemble for the current experience, the unified head that uses all the
representations is always trained from all FEs and learns to discriminate
between all seen classes (Figure 6). The proposed pseudo-feature projection
extends feature translation [51] with the corresponding class standard
deviation to allow a better sampling of the dimensions. It translates a
representation from a class into a projected representation of a different
class. Let $a_{i}$ be the concatenated representation of all FEs outputs for a
sample from the current experience belonging to class $i$. The proposed
projection is then defined as

$\hat{a}_{j,i}=\mu_{j}+\frac{a_{i}-\mu_{i}}{\sigma_{i}}\cdot\sigma_{j}\text{,}$

where $\hat{a}_{j,i}$ is the estimated projection from class $i$ to class $j$.
This projection is applied to each sample in the training batch while learning
the unified head, and not used during evaluation. The target class $j$ is
chosen at random from previously learned classes, and both the original
representation and the projected one are added onto the loss.

The class prototypes (i.e., the mean $\mu_{i}$ and standard deviation
$\sigma_{i}$) are always updated before the unified head is trained by
calculating the statistics over the available class data. However, since
multiple FEs exist, access might not be available to the mean and standard
deviation for each class, depending on when they were learned, or if those
classes have appeared since that time. Therefore, if the class statistics are
unknown for a feature extractor $e$, estimates are needed for
$\hat{\mu}_{c,e}$ and $\hat{\sigma}_{c,e}$. We fix the estimation of the
standard deviation to $\hat{\sigma}=1$, as we do not have any information
about the class shape for this feature extractor. For the estimation of the
unknown mean statistic of a specific FE output, $\hat{\mu}_{c,e}$, we resort
to a simple heuristic: using the original representation of the current sample
without modification $\hat{\mu}_{c,e}=a_{i,e}$.

Pseudocode for Horde as well as further experimental analysis is presented in
Appendix E. The implementation is publicly available on GitHub:
https://github.com/mmasana/clvision-chlg-2023/.

##  5 Strategy 3: DWGRNet

The strategy proposed by team _pddbend_ is called Dynamic Weighted Gated
Representation Network (DWGRNet). This strategy creates independent branches
for each experience, and uses gating units to control which branches are
active. During training, the branch corresponding to the current experience is
activated by its gating unit, facilitating learning. In contrast, branches
from previous experiences remain inactive. To improve the model’s
generalization and robustness, data augmentation techniques, such as AugMix
[54], are used to enhance the data samples. In the testing phase, the gating
units control the sequence of predictions. However, it is important to note
that using predictions from all branches, as done in [55], may not always
yield optimal results. This is because many branches may not have been exposed
to a particular class, leading to overconfident and potentially inaccurate
predictions. This issue resembles an open-set recognition problem. To mitigate
this, DWGRNet is proposed, which assigns weights based on entropy, feature
norm, and the number of classes experienced by each branch. Specifically, the
entropy of each branch’s predicted probability distribution is assessed. High
entropy indicates the possibility of the sample being an open-set item for
that branch. Similarly, the feature norm is computed. A higher feature norm
suggests that the sample is likely to be an open-set sample. Finally, it is
posited that experiences with a larger number of classes will render the
model’s prediction more reliable. As a result, weighting can also adjusted
based on the number of classes in each experience.

###  5.1 Motivation and Related Work

To achieve a good balance between stability and plasticity in class-
incremental learning, [55] introduced a method called Dynamically Expandable
Representation (DER). DER decouples the adaptivity of feature representation
from the classification procedure. When faced with a new task that contains
novel categories, the DER method first freezes the previously learned feature
extractor. Then, it incorporates a new feature extractor to expand the main
feature extractor network. Ultimately, the features extracted by all the
extractors are combined and forwarded to the classifier module for predicting
the category. This strategy exhibits notable efficiency in preserving existing
knowledge while providing sufficient flexibility to capture new information.

However, the DER architecture cannot be directly employed in the context of
this competition. This is due to the fact that DER constructs a model ensemble
that exploits replay buffers for raw sample storage. Its incompatibility
arises from several limitations, with the most notable being the simultaneous
execution of predictions across all branches, which has high computational
complexity. In practice, DER applies predictions from each branch without
distinction. A problem with this is that certain branches produce
overconfident predictions for unseen classes, leading to sub-optimal
performance.

To fix the limitations in DER, the proposed method implements an open-set
approach [56, 57, 58]. Within this framework, each model is restricted to be
exposed to a limited number of classes. This approach is inspired by recent
advancements in open-set recognition literature that have elucidated the
relationships between logit entropy, feature norm, and out-of-distribution
(OOD) samples. Capitalizing on these findings, a scoring system is designed
for OOD samples.

This new formulation based on the idea of an open-set problem, demands the
evaluation of each model to determine whether a test sample falls within the
out-of-distribution category, prior to the ensembling process. This evaluation
allows two possible options: (1) to either suppress the OOD outputs or (2) to
harness the OOD scores to assign weight to each sample. Therefore, it serves
to alleviate the constraints of the DER method, with the overarching aim of
augmenting both its performance and applicability in the realm of class-
incremental learning.

###  5.2 Method Description

To address restrictions such as raw samples storage in a buffer, gated units
are designed to control the activation of each branch. A new branch is added
with each new experience and then activated, while the model parameters in the
old branches are kept frozen. As illustrated in Figure 7, during the training
phase, no specific loss functions or replay buffers are used. Instead, the
naive cross-entropy loss is utilized to train the model. Another important
change is that AugMix is used to enhance the model’s generalization and
robustness. AugMix combines different data augmentation techniques. During the
testing phase, the gate units could activate each model one by one to avoid
the need for a large GPU memory. Their outputs are collected and then used to
make the final prediction. This process is illustrated in Figure 8.

Figure 7: Schematic showing the training phase of the DWGRNet strategy. The
blue modules are frozen. Figure 8: Testing phase of the DWGRNet strategy. The
outputs of each module are collected for the final prediction.

This method is fundamentally equivalent to a model ensemble. It is designed in
a way that both complies with the competition rules and also speeds up
training time and reduces memory space during training and inference. On the
other hand, the “naive” model ensembling approach may not work properly as
each model only learns limited knowledge. Therefore, the open-set idea is
employed. If the problem is treated as an open-set problem, and asks each
model whether a test sample is an OOD sample before ensembling, it is possible
to either mask the OOD outputs or use the OOD scores to weigh each sample.

The method relies solely on the concept of “entropy” to make the entire
strategy work. The assumption is that the entropy of a model’s outputs can
indicate their confidence. For an in-distribution sample, the model’s outputs
should have low entropy since the model has seen the class before and is
confident in its prediction. For an OOD sample, the model’s outputs should
have a high entropy since the model has not seen this class before, and lacks
confidence in its output. While this phenomenon is not always accurate, it
holds true in many circumstances.

Additionally, the aim is to further enhance the performance of the model
ensemble. First, the reliability of each model is to be evaluated. The
hypothesis is that the more classes a model encounters, the more reliable it
becomes. Therefore, the number of classes ($N_{C}$) is used to weight each
model’s output. Finally, inspired by the idea of “a good closed-set classifier
is all you need” in [58], feature norms are used to verify whether the model
perceives the sample as an in-distribution or OOD sample. The OOD samples tend
to have a lower feature norm.

To calculate the final logit, the $k$-th output logit of the model ensemble is
calculated as:

$\text{ensemble\\_logit}_{k}=\max_{i\in
M}{(\frac{\text{logit}_{i,k}}{\text{entropy}_{i,k}}*{N_{C}}^{(i)}*\text{feature\\_norm}_{i,k})},$
(1)

where $M$ is the set of saved model branches, and $k$ denotes the $k$-th test
sample. An ablation study to investigate the role of each module can be found
in Appendix F.

##  6 Results

###  6.1 Pre-selection Phase

In the pre-selection phase of the challenge, the teams were ranked based on
their average accuracy. The average accuracy for each team was computed as the
mean accuracy on the test set after training on each of the three streams
released for the first phase, namely $S_{1}$, $S_{2}$, and $S_{3}$. The top
ten teams in the pre-selection phase are shown in Table 1. The top five teams
proceeded to the final phase.

Team | Average Accuracy $\uparrow$ | Accuracy $S_{1}$ | Accuracy $S_{2}$ | Accuracy $S_{3}$
---|---|---|---|---
1\. pddbend | $44.77$ | $52.32$ | $40.31$ | $41.67$
2\. linzz | $44.08$ | $50.88$ | $41.83$ | $39.52$
3\. shelley | $42.53$ | $45.21$ | $41.80$ | $40.58$
4\. xduan7 | $41.37$ | $44.99$ | $40.80$ | $38.31$
5\. mmasana | $40.52$ | $42.47$ | $34.29$ | $44.81$
6\. zys_tjut | $34.35$ | $45.34$ | $30.43$ | $27.28$
7\. Vanixxz | $31.72$ | $33.33$ | $30.01$ | $31.83$
8\. M1andy | $31.01$ | $33.32$ | $28.97$ | $30.74$
9\. rainstar | $30.88$ | $36.01$ | $27.77$ | $28.87$
10\. flyfishzzz | $29.97$ | $31.31$ | $31.31$ | $27.28$
Table 1: Results from the pre-selection phase for the top ten teams.

###  6.2 Final Phase

In the final phase of the challenge, three streams, namely $S_{4}$, $S_{5}$,
and $S_{6}$, with configurations different from those in the pre-selection
phase streams, were used to evaluate the solutions submitted by the finalist
teams. The details of the challenge stream configurations are given in Table
C.1 in the Appendix C. The final phase demonstrated substantial variations in
performance among the solutions of the finalist teams, as presented in Table
2. Team xduan7 (Section 3) was selected as the challenge winner with an
average accuracy of $62.75\%$, thereby substantially outperforming the other
finalists. Team linzz earned the second spot with an average accuracy of
$45.02\%$. However, they decided not to contribute their solution to this
report. The other two teams, mmasana (Section 4) and pddbend (Section 5),
achieved average accuracies of $41.11\%$ and $40.91\%$, respectively. Recorded
video presentation of the submitted solutions by the finalist teams can be
accessed through
https://sites.google.com/view/clvision2023/challenge/challenge-results.

Team | Average Accuracy $\uparrow$ | Accuracy $S_{4}$ | Accuracy $S_{5}$ | Accuracy $S_{6}$
---|---|---|---|---
1\. xduan7 | $62.75$ | $63.64$ | $68.04$ | $56.57$
2\. linzz | $45.02$ | $46.21$ | $47.27$ | $41.58$
3\. mmasana | $41.11$ | $40.82$ | $45.79$ | $36.72$
4\. pddbend | $40.91$ | $42.07$ | $44.44$ | $36.23$
Table 2: Results from the final phase for the finalist teams. Team shelley is
not included since they decided not to participate in the final phase after
realizing that their solution violated the challenge rules.

##  7 Discussion

The challenge of class-incremental learning with repetition necessitates a
careful balance between learning new information and not forgetting previous
knowledge. This report has described three solutions proposed by the finalists
of the CLVision challenge at CVPR 2023. These solutions reflect diverse
methodologies to address the main objective of the challenge: learning
continually in the presence of repetition. Interestingly, an approach that is
shared by all finalist solutions is the use of separate (sub-)networks for
different experiences. This commonality suggests that the relatively simple
approach of having separate (sub-)networks per experience might, from a
practical perspective, be a very useful strategy, especially in settings such
as those explored in this challenge. In recent years, this approach of
combining multiple task- or experience-specific networks has been explored
under the names of representation ensembling [59, 60] and dynamically
expandable representations [55]. Aspects of the challenge setting that might
have been important for the success of this ensemble-based strategy include
the discouragement of the use of replay (storing raw samples was not allowed,
and only a very limited amount of latent data representations could be stored)
and the restrictions that were placed on the computational budget for
training, combined with the absence of such restrictions for testing. The
repetition in the data stream is also likely important for the success of
ensemble-based strategies, because the repetition allows each class to be
observed in multiple experiences and in different pairings with other classes.
This diversity of observed combinations of classes might allow for the
development of a “rich” set of feature extractors, leading to more robust
representations.

The suitability of ensemble-based strategies for this challenge is best
illustrated by the winning strategy  _HAT-CIR_. During the pre-selection phase
of the challenge, this strategy used a single network with a HAT-based
partitioning approach, which can be interpreted as a relatively soft version
of an ensemble-based strategy. In the final phase of the challenge this
strategy switched to using network replicas, a more explicit version of an
ensemble-based strategy, which substantially improved its performance. The
second strategy discussed in this report, Horde, trained an ensemble of
feature extractors, where a new feature extractor was added only when needed
based on a heuristic. It also used a unified head to discriminate between
classes that are observed in the current step of training. The third strategy,
DWGRNet, used independent branches for each experience, activated by gating
units during training and incorporating data augmentation techniques for
robustness. This strategy further approached the problem from an open-set
problem perspective, and to address open-set recognition issues, it employed a
weighting strategy based on factors such as entropy, feature norm, and the
number of classes in each experience.

Overall, the proposed solutions offer diverse yet converging approaches that
prioritize a balanced retention of old and new knowledge during the learning
process. These strategies are mostly based on the idea of ensemble learning,
and additionally leverage specialized discrimination techniques, data
augmentation and weighting strategies. This report provides multiple avenues
for future research to further refine and combine these approaches for more
effective and robust class-incremental learning strategies when repetition is
present in the data stream.

## Declaration of Generative AI and AI-assisted technologies in the writing
process

During the preparation of this work, some of the authors used ChatGPT (GPT-4)
in order to improve the readability and language of certain sections. After
using this tool, the authors reviewed and edited the content as needed and
take full responsibility for the content of the publication.

## Acknowledgements

This work has been supported by the Exascale Computing Project, a
collaborative effort of the U.S. Department of Energy Office of Science and
the National Nuclear Security Administration [grant number 17-SC-20-SC]; by
TU-Graz SAL DES Lab, part of the “University SAL Labs” initiative of Silicon
Austria Labs (SAL) and its Austrian partner universities for applied
fundamental research for electronic based systems; by the National Science and
Technology Major Project [grant number 2022ZD0114801]; by a senior
postdoctoral fellowship from the Research Foundation – Flanders (FWO) [grant
number 1266823N]; and by a Marie Skłodowska-Curie fellowship under the
European Union’s Horizon Europe programme [grant number 101067759].

## References

  * [1] Sebastian Thrun and Tom M. Mitchell.  Lifelong robot learning.  Robotics and Autonomous Systems, 15(1):25–46, 1995.
  * [2] Zhiyuan Chen and Bing Liu.  Lifelong Machine Learning, Second Edition.  Synthesis Lectures on Artificial Intelligence and Machine Learning. Springer Cham, 2018.
  * [3] Eli Verwimp, Rahaf Aljundi, Shai Ben-David, Matthias Bethge, Andrea Cossu, Alexander Gepperth, Tyler L Hayes, Eyke Hüllermeier, Christopher Kanan, Dhireesha Kudithipudi, Christoph H Lampert, et al.  Continual learning: Applications and the road forward.  Transactions on Machine Learning Research, 2024.
  * [4] Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, Bishan Yang, Justin Betteridge, Andrew Carlson, Bhavana Dalvi, Matt Gardner, Bryan Kisiel, et al.  Never-ending learning.  Communications of the ACM, 61(5):103–115, 2018.
  * [5] German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter.  Continual lifelong learning with neural networks: A review.  Neural Networks, 113:54–71, 2019.
  * [6] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3366–3385, 2022.
  * [7] Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Díaz-Rodríguez.  Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges.  Information Fusion, 58:52–68, 2020.
  * [8] Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In Psychology of Learning and Motivation, volume 24, pages 109–165. Elsevier, 1989.
  * [9] Roger Ratcliff.  Connectionist models of recognition memory: constraints imposed by learning and forgetting functions.  Psychological Review, 97(2):285–308, 1990.
  * [10] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017.
  * [11] Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In International Conference on Machine Learning, pages 3987–3995. PMLR, 2017.
  * [12] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935–2947, 2017.
  * [13] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne.  Experience replay for continual learning.  Advances in Neural Information Processing Systems, 32, 2019.
  * [14] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc’Aurelio Ranzato.  On tiny episodic memories in continual learning.  arXiv preprint arXiv:1902.10486, 2019.
  * [15] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.  icarl: Incremental classifier and representation learning.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017.
  * [16] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim.  Continual learning with deep generative replay.  Advances in Neural Information Processing Systems, 30, 2017.
  * [17] Gido M van de Ven, Hava T Siegelmann, and Andreas S Tolias.  Brain-inspired replay for continual learning with artificial neural networks.  Nature Communications, 11:4069, 2020.
  * [18] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  arXiv preprint arXiv:1606.04671, 2016.
  * [19] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars.  Expert gate: Lifelong learning with a network of experts.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3366–3375, 2017.
  * [20] Ju Xu and Zhanxing Zhu.  Reinforced continual learning.  Advances in Neural Information Processing Systems, 31, 2018.
  * [21] Gido M van de Ven and Andreas S Tolias.  Three continual learning scenarios.  In NeurIPS Continual Learning Workshop, 2018.
  * [22] Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira.  Re-evaluating continual learning scenarios: A categorization and case for strong baselines.  arXiv preprint arXiv:1810.12488, 2018.
  * [23] Gido M van de Ven, Tinne Tuytelaars, and Andreas S Tolias.  Three types of incremental learning.  Nature Machine Intelligence, 4(12):1185–1197, 2022.
  * [24] Stefan Stojanov, Samarth Mishra, Ngoc Anh Thai, Nikhil Dhanda, Ahmad Humayun, Chen Yu, Linda B Smith, and James M Rehg.  Incremental object learning from contiguous views.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8777–8786, 2019.
  * [25] Timothée Lesort, Oleksiy Ostapenko, Diganta Misra, Md Rifat Arefin, Pau Rodríguez, Laurent Charlin, and Irina Rish.  Scaling the number of tasks in continual learning.  arXiv preprint arXiv:2207.04543, 2022.
  * [26] Lexia Zhan, Dingrong Guo, Gang Chen, and Jiongjiong Yang.  Effects of repetition learning on associative recognition over time: Role of the hippocampus and prefrontal cortex.  Frontiers in Human Neuroscience, 12:277, 2018.
  * [27] Christopher D Smith and Damian Scarf.  Spacing repetitions over long timescales: a review and a reconsolidation explanation.  Frontiers in Psychology, 8:962, 2017.
  * [28] Kanyin Feng, Xiao Zhao, Jing Liu, Ying Cai, Zhifang Ye, Chuansheng Chen, and Gui Xue.  Spaced learning enhances episodic memory by increasing neural pattern similarity across repetitions.  Journal of Neuroscience, 39(27):5351–5360, 2019.
  * [29] Vincenzo Lomonaco, Lorenzo Pellegrini, Pau Rodriguez, Massimo Caccia, Qi She, Yu Chen, Quentin Jodelet, Ruiping Wang, Zheda Mai, David Vazquez, et al.  Cvpr 2020 continual learning in computer vision competition: Approaches, results, current challenges and future directions.  Artificial Intelligence, 303:103635, 2022.
  * [30] Davide Maltoni and Vincenzo Lomonaco.  Continuous learning in single-incremental-task scenarios.  Neural Networks, 116:56–73, 2019.
  * [31] Continual Learning Challenge, CVPR 2021.  https://eval.ai/web/challenges/challenge-page/829/overview, 2021\.  Accessed: 2024-03-29.
  * [32] Eli Verwimp, Kuo Yang, Sarah Parisot, Lanqing Hong, Steven McDonagh, Eduardo Pérez-Pellitero, Matthias De Lange, and Tinne Tuytelaars.  CLAD: A realistic continual learning benchmark for autonomous driving.  Neural Networks, 161:659–669, 2023.
  * [33] Lorenzo Pellegrini, Chenchen Zhu, Fanyi Xiao, Zhicheng Yan, Antonio Carta, Matthias De Lange, Vincenzo Lomonaco, Roshan Sumbaly, Pau Rodriguez, and David Vazquez.  3rd continual learning workshop challenge on egocentric category and instance level object understanding.  arXiv preprint arXiv:2212.06833, 2022.
  * [34] Chenchen Zhu, Fanyi Xiao, Andrés Alvarado, Yasmine Babaei, Jiabo Hu, Hichem El-Mohri, Sean Chang, Roshan Sumbaly, and Zhicheng Yan.  EgoObjects: A large-scale egocentric dataset for fine-grained object understanding.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
  * [35] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, University of Toronto, 2009.
  * [36] Hamed Hemati, Andrea Cossu, Antonio Carta, Julio Hurtado, Lorenzo Pellegrini, Davide Bacciu, Vincenzo Lomonaco, and Damian Borth.  Class-incremental learning with repetition.  In Sarath Chandar, Razvan Pascanu, Hanie Sedghi, and Doina Precup, editors, Proceedings of The 2nd Conference on Lifelong Learning Agents, volume 232 of Proceedings of Machine Learning Research, pages 437–455. PMLR, 22–25 Aug 2023.
  * [37] Adrien Pavao, Isabelle Guyon, Anne-Catherine Letournel, Dinh-Tuan Tran, Xavier Baro, Hugo Jair Escalante, Sergio Escalera, Tyler Thomas, and Zhen Xu.  Codalab competitions: An open source platform to organize scientific challenges.  Journal of Machine Learning Research, 24(198):1–6, 2023.
  * [38] Antonio Carta, Lorenzo Pellegrini, Andrea Cossu, Hamed Hemati, and Vincenzo Lomonaco.  Avalanche: A pytorch library for deep continual learning.  Journal of Machine Learning Research, 24(363):1–6, 2023.
  * [39] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou.  Overcoming catastrophic forgetting with hard attention to the task.  In International Conference on Machine Learning, pages 4548–4557. PMLR, 2018.
  * [40] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan.  Supervised contrastive learning.  Advances in Neural Information Processing Systems, 33:18661–18673, 2020.
  * [41] Gyuhak Kim, Sepideh Esmaeilpour, Changnan Xiao, and Bing Liu.  Continual learning based on OOD detection and task masking.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3856–3866, 2022.
  * [42] Gyuhak Kim, Bing Liu, and Zixuan Ke.  A multi-head model for continual learning via out-of-distribution replay.  In Sarath Chandar, Razvan Pascanu, and Doina Precup, editors, Proceedings of The 1st Conference on Lifelong Learning Agents, volume 199 of Proceedings of Machine Learning Research, pages 548–563. PMLR, 22–24 Aug 2022.
  * [43] Xiaotian Duan.  HAT-CL: A hard-attention-to-the-task pytorch library for continual learning.  arXiv preprint arXiv:2307.09653, 2023.
  * [44] Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin.  The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects.  Frontiers in Psychology, 2013.
  * [45] Marc Masana, Xialei Liu, Bartłomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer.  Class-incremental learning: survey and performance evaluation on image classification.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513–5533, 2023.
  * [46] Jathushan Rajasegaran, Munawar Hayat, Salman H Khan, Fahad Shahbaz Khan, and Ling Shao.  Random path selection for continual learning.  Advances in Neural Information Processing Systems, 32, 2019.
  * [47] Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra.  PathNet: Evolution channels gradient descent in super neural networks.  arXiv preprint arXiv:1701.08734, 2017.
  * [48] Arun Mallya and Svetlana Lazebnik.  PackNet: Adding multiple tasks to a single network by iterative pruning.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765–7773, 2018.
  * [49] Marc Masana, Tinne Tuytelaars, and Joost van de Weijer.  Ternary feature masks: Zero-forgetting for task-incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 3570–3579, 2021.
  * [50] Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic.  Learning and transferring mid-level image representations using convolutional neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1717–1724, 2014.
  * [51] Grégoire Petit, Adrian Popescu, Hugo Schindler, David Picard, and Bertrand Delezoide.  Fetril: Feature translation for exemplar-free class-incremental learning.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3911–3920, 2023.
  * [52] Brian Kulis.  Metric learning: A survey.  Foundations and Trends in Machine Learning, 5(4):287–364, 2013\.
  * [53] Raia Hadsell, Sumit Chopra, and Yann LeCun.  Dimensionality reduction by learning an invariant mapping.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1735–1742, 2006.
  * [54] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan.  Augmix: A simple data processing method to improve robustness and uncertainty.  arXiv preprint arXiv:1912.02781, 2019.
  * [55] Shipeng Yan, Jiangwei Xie, and Xuming He.  DER: Dynamically expandable representation for class incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3014–3023, 2021.
  * [56] Chuanxing Geng, Sheng-jun Huang, and Songcan Chen.  Recent advances in open set recognition: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10):3614–3631, 2020.
  * [57] Yiyou Sun, Chuan Guo, and Yixuan Li.  ReAct: Out-of-distribution detection with rectified activations.  Advances in Neural Information Processing Systems, 34:144–157, 2021\.
  * [58] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman.  Open-set recognition: a good closed-set classifier is all you need?  In International Conference on Learning Representations, 2022.
  * [59] Joshua T Vogelstein, Jayanta Dey, Hayden S Helm, Will LeVine, Ronak D Mehta, Tyler M Tomita, Haoyin Xu, Ali Geisa, Qingyang Wang, et al.  Representation ensembling for synergistic lifelong learning with quasilinear complexity.  arXiv preprint arxiv:2004.12908, 2020.
  * [60] Timm Hess, Eli Verwimp, Gido M van de Ven, and Tinne Tuytelaars.  Knowledge accumulation in continually learned representations and the issue of feature forgetting.  arXiv preprint arXiv:2304.00933, 2023.

##  Appendix A Author Contributions

All authors have made substantial contributions to the writing, design and/or
interpretation of this research paper. The following authors have been
responsible for the development and description of the different strategies
reported on in this paper:

  * •

Strategy 1: HAT-CIR developed by Xiaotian Duan, Zixuan Zhao, and Fangfang Xia.

  * •

Strategy 2: Horde developed by Marc Masana, Benedikt Tscheschner, and Eduardo
Veas.

  * •

Strategy 3: DWGRNet developed by Yuxiang Zheng, Shiji Zhao, Shao-Yuan Li, and
Sheng-Jun Huang.

##  Appendix B Participation Over Time

The graphs in Figure B.1 show the number of submissions per day by all teams,
and the maximum average accuracy achieved on each day. In summary,
participants began with a low average accuracy of $\sim 10\%$. Over time, the
highest average accuracy showed gradual improvement, peaking and stabilizing
at $\sim 40\%$ towards the end. The challenge saw a steady increase in
engagement, with daily submissions starting from as low as one to three in the
beginning, and reaching above $30$ towards the end.

(a) Highest average accuracy per day.

(b) Number of submissions per day.

Figure B.1: Participation over time in pre-selection phase. Accuracy values
are rounded by the competition platform.

##  Appendix C Stream Configurations

The configurations of the streams used for both pre-selection and final phases
are given in Table C.1.

Stream | First Occurrence $P_{f}$ | Repetition Dist. $P_{r}$
---|---|---
$S1$ | Geometric: $p=0.5$ | Zipfian: $e=0.7$
$S2$ | Geometric: $p=0.001$ | Zipfian: $e=0.8$
$S3$ | Geometric: $p=0.2$ | Fixed at $0.04$
$S4$ | Geometric: $p=0.6$ | Zipfian: $e=0.8$
$S5$ | Geometric: $p=0.001$ | Zipfian: $e=0.6$
$S6$ | Geometric: $p=0.1$ | Fixed at $0.05$
Table C.1: Configurations of the challenge streams.

##  Appendix D Appendix – HAT-CIR

###  D.1 Ablation Study for Single-model Settings

The ablation study, presented in Table D.1, indicates that various components
of the proposed method were important for achieving robust performance in the
pre-selection phase, when network replicas were not yet used. Notably, the
modified HAT-CL technique significantly outperforms the traditional HAT method
in the context of CIR, thus confirming its efficacy. The supervised
contrastive learning phase also plays a crucial role in learning better
feature representations, as its absence led to a marked decline in
performance.

Method | Accuracy $S_{4}$ | Accuracy $S_{5}$ | Accuracy $S_{6}$ | Avg. Decrease from Baseline
---|---|---|---|---
Baseline | 39.08% | 42.89% | 34.59% | -
With Original HAT | 23.89% | 26.05% | 19.47% | 15.71%
Without SupCon | 35.39% | 28.93% | 22.16% | 10.02 %
Without Momentum & TTA | 30.67% | 35.16% | 26.61% | 8.04%
Table D.1: Ablation study results for HAT-CIR when using a single model
without using network replicas. The accuracy metrics for each data stream from
the final phase of the challenge are presented. The “Avg. Decrease from
Baseline” column indicates how much performance is lost relative to the
baseline (i.e., the version of HAT-CIR that was used in the pre-selection
phase of the challenge). The term “Original HAT” refers to the original hard
attention method, “SupCon” stands for supervised contrastive learning, and
“Momentum & TTA” are abbreviations for momentum-based test-time decision-
making and test-time augmentation, respectively.

###  D.2 Influence of Number of Network Replicas

The results presented in Table D.2 demonstrate that both increases in ensemble
replicas and increases in fragment replicas enhance performance. Yet, by
adding more replicas, the added benefit starts to diminish. This suggests
there is a sweet spot in balancing the number of fragments and ensembles for
optimal performance without excessive training and/or test time. Importantly,
adding ensemble replicas comes with the trade-off of both longer training and
test time, while adding fragment replicas only comes with the trade-off of
longer test time. Because the rules of the challenge predominantly put
restrictions on computational resources during training, the final version of
the model that was selected predominantly used fragment replicas.

# of Replicas | # of Fragments | # of Ensembles | Accuracy $S_{4}$ | Accuracy $S_{5}$ | Accuracy $S_{6}$
---|---|---|---|---|---
1 | 1 | 1 | 39.08% | 42.89% | 34.59%
2 | 2 | 1 | 40.20% | 43.03% | 35.90%
2 | 1 | 2 | 43.90% | 46.17% | 39.11%
10 | 5 | 2 | 45.96% | 49.42% | 42.18%
10 | 2 | 5 | 48.06% | 48.50% | 44.69%
50 | 10 | 5 | 51.14% | 53.22% | 46.37%
50 | 5 | 10 | 50.70% | 52.58% | 47.92%
100 | 10 | 10 | 52.74% | 54.13% | 46.87%
100 | 50 | 2 | $\mathbf{63.64}\%$ | $\mathbf{68.04}\%$ | $\mathbf{56.57}\%$
Table D.2: Effect of the number and types of network replicas on the
performance of HAT-CIR on the data streams from the final phase of the
challenge. The table shows the accuracy achieved for each data stream with
different numbers of fragments and ensembles.

##  Appendix E Appendix – Horde

###  E.1 Algorithm

Inspired by zero-forgetting methods, Horde promotes stability on each task-
specific Feature Extractor (FE) while leveraging class repetition for
balancing plasticity by learning an aligned unified representation. Each
extractor provides a discriminative representation for the seen classes of the
task where it is learned. Then, pseudo-feature projection is used to achieve
alignment on the unified head. A description of the pseudo-code of Horde is
provided in Algorithm 1. Additionally, the feature extractor training steps
are shown in Algorithm 2.

Algorithm 1 Horde Algorithm

1:for experience do

2: if Training Condition then

3: Train FE and add to ensemble $\triangleright$ See Alg. 2

4: end if

5: Calculate $\mathbf{\mu}_{c}$ and $\mathbf{\sigma}_{c}$ for all classes $c$

6: for training epoch do $\triangleright$ Only Unified head

7: for Batch do

8: Generate $\mathbf{\hat{a}_{j,i}}$ from $\mathbf{a}_{i}$ and old classes

9: Calculate $\mathcal{L}_{CE}$ with both $\mathbf{a_{i}}$ and
$\mathbf{\hat{a}_{j,i}}$

10: Backprop $\mathcal{L}_{CE}$

11: end for

12: end for

13:end for

Algorithm 2 FE Training

1:Initialize CE and ML Head

2:Initialize new or transfer learning model

3:for training epoch do

4: for Batch do

5: Calculate $\mathcal{L}_{CE}$

6: Calculate $\mathcal{L}_{ML}$

7: Backprop $(1-\alpha)\cdot\mathcal{L}_{CE}+\alpha\cdot\mathcal{L}_{ML}$

8: end for

9: Update $\alpha$ from avg. $\mathcal{L}_{CE}$ and $\mathcal{L}_{ML}$

10:end for

11:Remove CE and ML head

12:Freeze FE

###  E.2 Additional Experimental Analysis

To further study the effects of the ensemble size, the Horde strategy is
evaluated with different number of maximum ensemble size. The results are
presented in Figure E.1. Stream 1 seems to be invariant to the ensemble size,
probably due to the first experience having 51 classes and, therefore,
providing a robust representation from the beginning. This can be exploited by
heavily enforcing stability and limited number of FEs, thus becoming
unnecessary to extend the ensemble after enough classes are seen. Streams 2
and 3 seem to show a benefit of extending the ensemble over time, which
accommodates the incorporation of new seen classes without the need of
replacing older FEs.

Figure E.1: Impact of the maximum number of feature extractors allowed to be
trained. The heuristic to decide when feature extractors are trained is not
changed. Results are shown on the three streams from the challenge’s initial
phase.

Another interesting direction to investigate is the different possibilities
for estimating unknown class statistics for a feature extractor. Since
multiple FEs exist, access might not be available to the mean and standard
deviation for each class, depending on when they were learned, or if those
classes have reappeared since last time. The complete class statistics
$\mu_{c}$ and $\sigma_{c}$ can thus be written as a concatenation of the
individual FEs where $n$ denotes the number of feature extractors.

$\mu_{c}=(\mu_{c,1},\,\dots\,,\mu_{c,n})$
$\sigma_{c}=(\sigma_{c,1},\,\dots\,,\sigma_{c,n})$

For the pseudo-feature projection step, given a class $c$ and a feature
extractor $e$, estimates are needed for $\hat{\mu}_{c,e}$ and
$\hat{\sigma}_{c,e}$. In cases we do not have any information about the class
shape for this feature extractor, we fix the estimation of the standard
deviation to $\hat{\sigma}=1$. For the estimation of $\hat{\mu}_{c,e}$ we
propose three heuristics:

  1. 1.

zeros: setting all $\hat{\mu}_{c,e}$ estimations to $0$.

  2. 2.

random: randomly sample $\hat{\mu}_{c,e}$ from a multivariate normal
distribution $\mathcal{N}(0,1)$.

  3. 3.

original features: estimate $\hat{\mu}_{c,e}$ with the original representation
of the current sample without modification.

Results are presented in Figure E.2, showing that estimating with the original
representations provides a clear advantage. Both zeros and random heuristics
seem to perform similarly, and clearly underperform compared to the original
features one.

Figure E.2: Impact of estimating unknown class statistics under different
heuristics. Zeros and random heuristics had almost the same performance.
Results are shown on the three data streams from the initial phase of the
challenge.

##  Appendix F Appendix – DWGRNet

###  F.1 Additional Experimental Analysis

An ablation study was conducted to investigate the role of each module. The
results are presented in Table F.1. It can be seen that entropy plays the most
significant role in the final accuracy. Adjusting each model’s logits using
the number of classes and the number of features also leads to improved
accuracy. This verifies the effectiveness of each module.

Module | Average Accuracy $\uparrow$ | Accuracy $S_{1}$ | Accuracy $S_{2}$ | Accuracy $S_{3}$
---|---|---|---|---
Only mean | $18.34$ | $20.22$ | $12.08$ | $22.71$
\+ Entropy | $41.61$ | $49.16$ | $38.14$ | $37.52$
\+ $N_{C}$ | $42.79$ | $50.11$ | $38.62$ | $39.64$
\+ feature_norm | $44.77$ | $52.32$ | $40.31$ | $41.67$
Table F.1: Result from the ablation study for the DWGRNet strategy.