# Forgetting Order of Continual Learning:
Examples That are Learned First are Forgotten Last

Guy Hacohen
KU Leuven, Leuven, Belgium
<EMAIL_ADDRESS>
&Tinne Tuytelaars
KU Leuven, Leuven, Belgium
<EMAIL_ADDRESS>

###### Abstract

Catastrophic forgetting poses a significant challenge in continual learning,
where models often forget previous tasks when trained on new data. Our
empirical analysis reveals a strong correlation between catastrophic
forgetting and the learning speed of examples: examples learned early are
rarely forgotten, while those learned later are more susceptible to
forgetting. We demonstrate that replay-based continual learning methods can
leverage this phenomenon by focusing on mid-learned examples for rehearsal. We
introduce Goldilocks, a novel replay buffer sampling method that filters out
examples learned too quickly or too slowly, keeping those learned at an
intermediate speed. Goldilocks improves existing continual learning
algorithms, leading to state-of-the-art performance across several image
classification tasks.

##  1 Introduction

The surge in deploying deep models in real-world scenarios has heightened the
need for models capable of adjusting to changing environments. Continual
learning (see extensive reviews, e.g [10, 19, 37]), which involves training
deep learning models on new data even after mastering previous tasks, is
crucial in this context. However, continual learning encounters a significant
obstacle called catastrophic forgetting [12, 24, 32]: as models learn from new
data, they tend to forget previous tasks, resulting in decreased performance
on those tasks. Despite extensive research efforts, catastrophic forgetting
persists as a central challenge in continual learning.

We begin by empirically examining catastrophic forgetting, identifying which
samples deep models tend to forget when trained on new tasks. Our findings
reveal a last-in-first-out pattern, where examples learned later are forgotten
more readily than those learned earlier. Fig. 1 visualizes this by showing
which examples are likely to be forgotten or remembered over time. This
phenomenon is connected to the simplicity bias [48, 50] of neural networks,
where networks learn simple examples faster than complex ones. However,
complex examples are more likely to be forgotten, while simple examples are
more likely to be remembered.

This observation has an important consequence: by measuring how fast examples
are being learned by the model, we can predict in advance which examples are
more susceptible to forgetting. This holds true regardless of the distribution
of the new task introduced to the model. Furthermore, we demonstrate that
models showing varying degrees of forgetting do so consistently: models with
less catastrophic forgetting recall most of the examples retained by those
with more forgetting, with the additional ability to remember slightly more
complex examples.

Replay-based continual learning methods use a memory buffer to address
catastrophic forgetting, storing a small subset of original task examples.
Typically, this buffer is uniformly sampled from previous tasks, and different
continual learning techniques vary in how they utilize it. Our findings
indicate that while a replay buffer helps mitigate forgetting, a similar
pattern emerges: examples learned later are forgotten sooner. Increasing
buffer size or improving the underlying model’s architecture can alleviate
more of the forgetting to some extent, but the fundamental phenomenon
persists. This observation implies that when strengthening a continually
trained model, we can predict in advance which examples it will better
remember compared to its weaker version.

This raises the question: can one exploit this phenomenon and reduce
forgetting by cleverly selecting samples for the replay buffer? Our
observations reveal that including in the replay buffer more examples that the
model can potentially remember given more practice can improve a large range
of continual learning methods. Given the fixed buffer size, these examples
replace the fastest-to-learn examples, which the model is likely to remember
anyway, and the slowest-to-learn examples, which it is likely to forget.
Leveraging these insights, we introduce a practical sampling method named
Goldilocks. In constructing the replay buffer, Goldilocks eliminates examples
learned too quickly or too slowly, ensuring the buffer comprises examples
learned at just the right speed.

Integrating Goldilocks sampling into different competitive continual learning
algorithms is computationally negligible, and results in a significant
improvement across a diverse range of image classification tasks, surpassing
state-of-the-art performance in many cases. Unlike prior replay-buffer
sampling methods, which often exhibit effectiveness only within specific
contexts or with particular algorithms, Goldilocks consistently performs well
across all evaluated settings.

Figure 1: Test examples that networks tend to either remember or forget.
Networks were trained on 2 tasks from CIFAR-100. Examples from the first task
that were learned quickest (left) and slowest (right) are shown. Slowest
examples were often forgotten, while the quickest were seldom forgotten.

###  1.1 Our contribution

  1. 1.

We demonstrate consistent catastrophic forgetting across diverse continual
learning approaches, highlighting certain examples’ susceptibility to
forgetting regardless of the specific continual learning framework.

  2. 2.

Networks are more likely to forget examples learned during later stages of the
learning than examples learned early on. This observation allows us to predict
during training the likelihood a given example will be forgotten.

  3. 3.

We introduce Goldilocks, a novel and efficient replay buffer sampling method.
Goldilocks selectively samples examples learned midway through training,
discarding examples that are too simple or too complex for the model.
Goldilocks reduces catastrophic forgetting and improves the performance of a
large variety of continual learning methods.

###  1.2 Related work

Existing studies primarily focus on mitigating catastrophic forgetting [25,
28, 29, 45, 47], rather than its characterization. Some works have explored
catastrophic forgetting from a model perspective: [36, 42] showed differences
in forgetting patterns across model layers, while [34, 40] demonstrated the
impact of training hyper-parameters. [35] identified task complexity as a
factor influencing forgetting rates. Our study differs by examining the
relationship between learning speed and forgettability of examples,
emphasizing a data-centric perspective.

Simplicity bias. Empirical evidence supports the hypothesis that neural
networks initially learn simple models, gaining complexity over time [7, 14,
15, 20, 21, 22, 23, 38, 49, 54]. This phenomenon, known as simplicity bias
[11, 48], is extended in our work to catastrophic forgetting, illustrating a
reverse simplicity bias where complex examples are forgotten before simpler
ones.

Forgetting dynamics. Several studies have directly explored the dynamics of
forgetting. [31, 53] revealed that, even in non-continual settings, certain
examples tend to be less forgettable during training. [33] demonstrated that
the forgetting dynamics of noisy examples differ from those of clean labels
and proposed adjusting the rehearsal process accordingly. Our results align
with these findings, further delineating which examples are more prone to
forgetting and retention, establishing connections between catastrophic
forgetting and simplicity bias, and proposing a method to identify examples
likely to be forgotten proactively. Additionally, our work illustrates how to
leverage these insights to enhance the sampling of the replay buffer for
various continual learning approaches.

Replay buffer sampling functions. Replay-based methods mitigate catastrophic
forgetting by storing examples from previous tasks in a memory buffer of fixed
size. Despite the availability of various sampling strategies [1, 3, 6, 44,
52, 55], their effectiveness often depends on specific conditions or
algorithms. As a result, many replay-based methods resort to uniform sampling
[5, 16, 25, 30, 41, 43, 46] in practice. Unlike previous methods, Goldilocks
can improve replay-based methods across diverse tasks and scenarios.

##  2 Empirical analysis of catastrophic forgetting

In this section, we empirically study catastrophic forgetting from a data-
centric perspective, drawing a connection between the speed at which a model
learns examples, and its likelihood to forget them.

###  2.1 Definitions

We examine the continual learning classification setting, where the training
dataset $\mathcal{D}=\\{\mathcal{X},\mathcal{Y}\\}$ consists of examples
$x\in\mathcal{X}$ and their corresponding labels $y\in\mathcal{Y}$. This
dataset is divided into $T>1$ tasks, with each task $t\in[T]$ represented as
$\mathcal{D}_{t}=\\{\mathcal{X}_{t},\mathcal{Y}_{t}\\}$, and
$\mathcal{D}=\bigcup_{t=1}^{T}{D_{t}}$. There is no overlap between the data
of different tasks. A deep model $f:\mathcal{X}\rightarrow\mathcal{Y}$ is
sequentially trained on each task $\mathcal{D}_{t}$ from $t=1$ to $t=T$, with
$E\in\mathbb{N}$ epochs per task. Each task also has a separate test set from
the same distribution. During training on a task $\mathcal{D}_{t}$, the model
$f$ is provided with a replay buffer $B$ of fixed-memory size, containing
examples from previous tasks ${1,...,t-1}$. Notably, $|B|\ll|\mathcal{D}|$,
meaning only a fraction of the data can be stored in the replay buffer. It’s
worth mentioning that the buffer may be empty ($B=\emptyset$), representing
training $f$ without a memory buffer.

Learning speed. Neural networks demonstrate a trend in learning examples,
where certain examples are consistently learned before others across various
deep neural models [2, 9, 17]. This orderliness is often connected to the
simplicity bias phenomenon [18], as examples learned earlier tend to be
simpler than those learned later. Rather than measuring the example’s
complexity, which can be defined in many different ways, we use the speed at
which examples are learned by the deep model, which can be measured
efficiently and directly. Given a model $f$ and an example $(x,y)$ either in
the train or test data, we define the _learning speed_ of the example $(x,y)$
to be:

$learning\\_speed(x,y)=\frac{1}{E}\sum_{e=1}^{E}\mathds{1}[f_{t}^{e}(x)=y]$
(1)

Here, $f_{t}^{e}$ denotes the intermediate model $f$ after training for
$e\in[E]$ epochs on task $\mathcal{D}_{t}$. This definition is based on the
_accessibility score_ introduced in [17], adapted to be computed by a single
model rather than an ensemble, enabling efficient computation throughout the
training of $f$. Intuitively, this score indicates how rapidly the model
learns an example: an example correctly classified from the early stages of
learning will sustain $f_{t}^{e}(x)=y$ for more epochs, resulting in a higher
_learning speed_.

Computing the learning speed. To calculate the learning speed of each example,
we maintain a boolean matrix, called the epoch-wise classification matrix,
$M\in\left\\{0,1\right\\}^{E\times|\mathcal{D}_{t}|}$ throughout $f$’s
training on task $\mathcal{D}_{t}$, indicating whether the model correctly
classified each example during learning. Specifically, for example
$(x_{i},y_{i})\in\mathcal{D}_{t}$, we have
$M_{e,i}=\mathds{1}[f_{t}^{e}(x_{i})=y_{i}]$. An example’s learning speed is
then the average of the $M$ across epochs. This matrix can be computed during
the forward pass of the network, incurring minimal to no additional
computation overhead. Furthermore, memory usage can be further reduced by
directly computing the matrix’s mean during training.

###  2.2 Preliminaries

Class incremental vs. task incremental learning. While similar, task
incremental learning distinguishes itself from class incremental learning by
providing the task identity for each example during testing, enabling the use
of the specific classifier of the classes within the task. As a result,
performance in class incremental learning is typically much lower. Due to
space constraints, our main paper exclusively presents results for task
incremental learning. However, findings for class incremental learning, which
align qualitatively with the task incremental case, are available in App. A.

Datasets. We investigated various image continual learning classification
tasks using split versions of several image datasets, including CIFAR-10,
CIFAR-100 [26], and TinyImagenet [27]. The data is split into $T$ tasks by
partitioning the classes into $T$ equal-sized subsets. This partitioning is
denoted as dataset-T. For example, splitting CIFAR-10 into 5 classes is
denoted as CIFAR-10-5, comprising 5 tasks, each with 2 distinct classes.
Unless otherwise specified, classes are divided into tasks according to the
original order they appeared in the dataset (often alphabetically).

Architectures and training hyper-parameters. In our experiments without a
replay buffer, unless stated otherwise, we trained ResNet-18 for $E=100$
epochs per task. We employed a base learning rate of $0.1$ with a cosine
scheduler, SGD optimizer, momentum of $0.9$, and weight decay of $0.0005$. All
networks were trained on NVIDIA TITAN X. These hyperparameters were chosen
arbitrarily based on their performance during joint dataset training and were
consistent across all experiments. We anticipate consistent qualitative
results despite potential variations in these hyperparameters. When
introducing a replay buffer, we employed an experience replay strategy [46],
alternating batches of data from the new task and the replay buffer. When
comparing against other replay buffer sampling methods, we maintained the same
ResNet-18 settings. When integrating with other continual learning algorithms,
we evaluated all methods within the framework of [4, 5], using the best
hyperparameters suggested in the original papers, while only altering the
replay buffer sampling method.

###  2.3 Correlating learning speed with catastrophic forgetting

(a) Test accuracy

(b) $M$ train

(c) $M$ test

(d) Speed vs. forgetting

Figure 2:  Forgetting as a function of the learning speed. We trained $10$
networks on CIFAR-100-2, without a replay buffer. (a) Mean test accuracy of
each task, where the dashed line marks the task switch. The models forget much
of the first task after the switch. (b-c), the first task’s binary epoch-wise
classification matrices $M$ for train and test data. Rows denote examples, and
columns denote epochs, indicating correct classifications by a majority of the
networks. Rows are sorted by learning speed. Faster-learned examples from the
first task are less likely to be forgotten at the end of the second task. (d)
The mean learning speed of examples in the first task vs. the $\%$ of networks
that remember them at the end of the second task. Networks forget more
examples learned slowly.

We begin by exploring catastrophic forgetting in a simplistic case, without
using replay buffer ($B=\emptyset$), and with only $T=2$ tasks. In Fig. 2(a),
we plot the mean test accuracy of each task when training $10$ networks on
CIFAR-100-2. The dashed black line marks the transition from the first to the
second task. Consistent with prior research, the networks exhibit significant
catastrophic forgetting, evidenced by a sharp decline in the accuracy of the
first task during training on the second task. Our objective is to
characterize those examples where the network successfully classified after
the first task but failed after the second task.

To simultaneously track learning speed and forgetting, we maintain the epoch-
wise boolean classification matrix $M$ for each network (formally defined in
§2.2). Throughout the training of both tasks, we record, for each epoch,
whether the network correctly classified each example from the first task. The
initial $E=100$ epochs represent classification during the first task,
enabling us to observe the learning speed of the different examples, while the
subsequent $E=100$ epochs allow us to monitor the forgetting of the first task
during training of the second task.

In Figs. 2(b-c), we examine these epoch-wise classification matrices for both
the train and test sets, respectively. To aid visualization, instead of
plotting the epoch-wise classification matrix for each network, we aggregate
together the networks from all runs using majority voting. The rows are sorted
by the learning speed of the data samples. Notably, we observe a correlation
between learning speed and catastrophic forgetting: examples learned more
quickly during the first task tend to remain correctly classified throughout
the second task, whereas slower-learned examples are more prone to immediate
misclassification upon task transition, making them susceptible to forgetting.
Essentially, slower learning speeds of examples correlate with a higher
probability of forgetting by deep models.

To quantify this relationship, we define an example in the first task as
"remembered" by the network if it is classified correctly at the conclusion of
both the first and second tasks. In Fig. 2(d), we plot, for each example in
the test set, the $\%$ of networks that remembered it vs. its mean learning
speed across all $10$ networks. For visualization, we group examples
remembered by a similar percentage of networks and average their mean learning
speeds. We observe a strong correlation ($r=0.995,p\leq 10^{-10}$), indicating
that networks are more likely to remember quickly learned examples.

While the results presented here pertain to a simplified case, this phenomenon
is robust across diverse datasets, architectures, and hyperparameters. For
additional experiments, please refer to App. B.

###  2.4 How the replay buffer changes the catastrophic forgetting dynamics

(a) Small buffer, $4\%$

(b) Medium buffer, $12\%$

(c) Large buffer, $40\%$

(d) Buffer size vs. speed

Figure 3: Impact of the buffer size on forgetting dynamics. (a-c) Similarly to
Fig. 2(c), the first task’s binary matrices $M$, when adding replay buffer of
varying buffer sizes ($4\%$, $12\%$, and $40\%$). Rows denote examples, and
columns denote epochs, indicating correct classifications by the majority of
$10$ networks. Rows are sorted by learning speed. In all cases, examples that
were learned faster are more likely to be remembered. With bigger buffers, the
networks can remember gradually slower-to-learn examples. (d) The mean
learning speed of remembered examples of models with different buffer sizes.
Models with bigger replay buffers remember slower-to-learn examples.

To evaluate replay-based methods, we incorporate a replay buffer in our
experiments. At the end of the first task, we uniformly sample examples to a
fixed-memory replay buffer. During training on the subsequent task, we adopt
the experience replay technique [46], alternating batch sampling between the
new data and the contents of the replay buffer.

Similarly to §2, we trained $10$ networks on CIFAR-100-2, with a replay
buffers of varying sizes: $1000$, $3000$, and $10000$ samples corresponding to
$4\%$, $12\%$, and $40\%$ of the data size, respectively. In Figs. 3(a-c), we
visualize the aggregated epoch-wise classification matrix of the test data for
each case. The rows in each matrix are sorted by the learning speed of the
examples. As expected, integrating a replay buffer mitigates some catastrophic
forgetting, boosting the model performance on the first task, with larger
buffers yielding better results. Nevertheless, similar to the scenario without
a replay buffer $B=\emptyset$, a relationship exists between learning speed
and catastrophic forgetting: networks tend to remember fast-learned examples
while forgetting those learned later even when using a uniformly sampled
replay buffer. Notably, models with larger buffer sizes remember most of the
examples that models with smaller buffer sizes do, while additionally
remembering gradually slower-learned examples. Similar results occur when
changing the network architecture (see App.B)

We quantify the relationship between the buffer size and remembered examples’
learning speed. Training $10$ networks on CIFAR-100-2 with various buffer
sizes, we plot the mean learning speed of examples remembered by the models
for each buffer size. A robust correlation emerges ($r=0.966$, $p\leq
10^{-6}$): models with smaller buffer sizes remember quickly learned examples,
whereas larger models with larger buffers enable the model also to remember
slower-learned. These results are plotted in Fig. 3(d). Similar results were
achieved across diverse datasets, architectures, hyperparameters, and
continual learning algorithms, see App. B.

##  3 Effects of different replay buffer compositions on catastrophic
forgetting

Replay-based continual learning methods typically sample the replay buffer
uniformly, including both quickly and slowly learned examples. In this
section, we analyze various replay buffer compositions and find that medium-
speed learned examples are the most beneficial for the replay process. These
examples, which the model has only a moderate chance of remembering, prove
particularly important. One possible explanation is that increasing the buffer
size impacts the catastrophic forgetting of these examples the most,
suggesting that focusing on them is advantageous.

###  3.1 The Goldilocks approach – methodology and definitions

To explore different buffer compositions, we remove a combination of the
quickest and slowest learned examples from the data and then sample the buffer
uniformly from the remaining examples. This allows us to add a bias to the
replay buffer toward examples learned at certain speeds. We propose
Goldilocks, a novel sampling method, used both in our empirical analysis and
as a practical replay buffer sampling technique. The name Goldilocks reflects
the method’s focus on examples learned at an optimal speed, akin to the story
where Goldilocks prefers things that are "just right."

Formally, Goldilocks is defined by two hyperparameters, $quick$ and $slow$,
which specify the percentage of examples learned too quickly or too slowly to
be removed from dataset $\mathcal{D}_{t}$. During training of the model $f$ on
$\mathcal{D}_{t}$, we maintain the epoch-wise classification matrix $M$ (see §
2.2). At the end of task $t$, examples in $\mathcal{D}_{t}$ are sorted by
learning speed, and the top $\%quick$ and bottom $\%slow$ are removed. The
replay buffer is then uniformly sampled from the remaining examples. Notably,
for tasks beyond the initial two, when reducing the replay buffer size,
examples can be uniformly sampled out. The pseudocode for Goldilocks sampling
is provided in Algorithm 1.

Algorithm 1 Training continual learning method $CL$ with Goldilocks sampling.

Input: task $\mathcal{D}_{t}$, sample size $|B|$, epochs $E$, amount of
quick/slow learned examples to remove $q,s$. Output: buffer of size $|B|$

1:$Classification\\_Matrix$ $\leftarrow$ ${0}^{E\times|\mathcal{D}_{t}|}$

2:for $e=1,...,E$ do

3: Train the model $f$ one epoch using the selected continual learning method
$CL$

4: for $(x_{i},y_{i})\in\mathcal{D}_{t}$ do $\triangleright$ Can be calculated
efficiently in the forward pass

5: Classification_Matrix$[e,i]\leftarrow\mathds{1}[f(x_{i})=y_{i}]$

6: end for

7:end for

8:$Learning\\_Speed$ $\leftarrow$ Mean($Classification\\_Matrix$, axis=0)

9:$\mathcal{D}_{t}^{{}^{\prime}}\leftarrow$ remove $q\%$ highest and $s\%$
lowest $Learning\\_Speed$ examples from $\mathcal{D}_{t}$

10:return $|B|$ examples sampled uniformly from
$\mathcal{D}_{t}^{{}^{\prime}}$

###  3.2 Which examples should appear in the replay buffer

(a) CIFAR-100-2, $4\%$ buffer

(b) CIFAR-100-2, $40\%$ buffer

(c) TinyImagenet-2, $20\%$ buffer

Figure 4: Comparing different replay buffer compositions. Each matrix compares
the mean final test accuracy of 10 networks trained with different replay
buffer compositions achieved by modifying Goldilocks’ $quick$ and $slow$
hyperparameters. The ($quick$, $slow$) = ($0\%$, $0\%$) case, marked by a
black box, corresponds to uniform sampling. (a) CIFAR-100-2 with a $4\%$
buffer. Removing faster-to-learn examples is more beneficial. (b) CIFAR-100-2
with a $40\%$ buffer. Removing slower-to-learn examples is more beneficial.
(c) TinyImageNet-2 with a $40\%$ buffer. Across all scenarios, focusing on
examples learned in the middle of the learning process is most beneficial.
Smaller buffers exhibit greater benefits from prioritizing faster-to-learn
examples.

(d) (A)–>(B)

(e) (A)–>(C)

Figure 5: Comparing replay buffer compositions when training the same original
task, followed by different subsequent tasks. We consider 3 tasks, A, B, and
C, training networks on task A followed by either B or C. All tasks are
subsets of CIFAR-100, where Task A is classes 0-24, B is classes 25-49 and C
is classes 50-74. (a) training on task A followed by B. (b) training on task A
followed by C. The results format is similar to Fig. 3.2. Despite changes in
the subsequent tasks, effective and ineffective buffer compositions for the
original task A remain consistent.

(a) GEM, CIFAR-100-2, $1\%$ buffer Figure 6:  Comparing different replay
buffer compositions for GEM with a buffer of $1\%$. The experimental setup
replicates the one done in Fig. 3.2. Focusing on examples learned midway
through the learning process is most beneficial. Due to the smaller buffer,
removing more easier examples is better.

We investigate the effect of constructing the replay buffer with examples
learned at different stages of the learning process. This is achieved by
adjusting the $slow$ and $fast$ parameters in Goldilocks (see §3.1).
Regardless of these parameters’ values, the replay buffer maintains a constant
size but is sampled from different data distributions. We evaluate a wide
range of $slow$ and $fast$ values by training 10 networks per parameter set on
various datasets and buffer sizes. The mean final test accuracy across all
tasks is plotted in a grid-like heatmap, in Fig. 3.2. To aid visualization,
standard errors are plotted separately, see App. C. Notably, the point
$(slow,fast)=(0,0)$ (marked by a black box) represents training with a
uniformly sampled replay buffer, as is done in the original works.

In Figs. 3.2(a-b), we compare replay buffer compositions for networks trained
on CIFAR-100-2. We consider a small buffer of size $1000$ ($4\%$ of data size)
and a large buffer of size $10000$ ($40\%$ of data size). Blank entries denote
infeasible parameters where too much data is removed, rendering a $40\%$
buffer impossible. In both cases, removing both slowly- and quickly-learned
examples proves beneficial, with parameters mutually influencing each other.
Focusing on examples learned midway through the learning process, where the
network can either remember or forget, is most effective in reducing
catastrophic forgetting. When the buffer is smaller, optimal results are
achieved with a bias toward fast-learned examples, as models struggle to
remember simpler examples in such scenarios.

In Fig. 4(c), we replicate this experiment with TinyImagenet-2, with a buffer
size of $10000$ ($20\%$ of the data). Similarly to the CIFAR-100-2 case,
focusing on examples learned midway through learning yields the best
performances. It’s worth noting that across all cases, the range of
hyperparameters that enhance performance is broad and continuous, suggesting
that improvements are not confined to specific hyperparameter sets; instead,
various methods of focusing on midway-learned examples yield positive
outcomes. Similar results with class incremental learning can be found in App.
A.

###  3.3 Independency of the optimal buffer composition on subsequent tasks

We empirically examine how subsequent tasks impact the ideal composition of a
replay buffer for a given task. We find that regardless of the similarity or
dissimilarity between subsequent tasks and the original task, the optimal
replay buffer composition remains largely independent and consistent. To show
this, we conduct experiments with three distinct tasks denoted A, B, and C. By
training the model on task A and subsequently introducing either task B or
task C while keeping the original task unchanged, we analyze the ideal replay
buffer composition under varying subsequent tasks.

We initially focus on tasks from the same datasets. We pick task A to be the
first 25 classes of CIFAR-100, task B to be classes 26 to 50, and task C to be
classes 51 to 75. In Fig. 5, we depict the mean final accuracy of 10 networks
across various replay buffer compositions when using either task B or task C
as the subsequent task after task A. Despite some quantitative variations in
performance, the qualitative results consistently indicate that good buffer
compositions for one case remain effective for the other, and vice versa.
These results suggest that the ideal replay buffer composition of task A is
robust to differences between tasks B and C in this context.

Further, we explore scenarios involving either changing the classification
task or replacing it with memorization by training on random labels. We adopt
the RotNet approach [13] to change the classification task. We pick tasks A
and B to be different subsets of CIFAR-100, while task C involves a rotation
classification task, where each example in task A is randomly rotated between
$90\degree,180\degree$, or $270\degree$ degrees, with the task being to
classify these rotations regardless of the original label. For the random
case, tasks A and B remain unchanged, while task C involves the same subset of
task B, but with labels chosen uniformly at random, as suggested in [57]. In
both scenarios, we observe consistent results: the ideal replay buffer
composition of task A remains robust to differences between tasks B and C in
both contexts. Further details and heatmaps for the results are available in
App. D.

Choosing hyperparameters. Implementing Goldilocks practically requires
identifying the optimal $quick$ and $slow$ hyperparameters without extra data.
While simple heuristics can often suffice due to the broad range of
hyperparameters yielding good results, our findings propose an alternative
approach for hyperparameter fine-tuning. By conducting a grid search for both
parameters, training the network on the original task, and using the rotated
version as the subsequent task, efficient parameter selection can be achieved.
The smooth impact of parameters on the results implies that a coarse grid
search is often adequate, reducing computational burden.

###  3.4 Goldilocks sampling improves various continual learning methods

Many replay-based continual learning algorithms sample the replay buffer
uniformly. In this section, we investigate the impact of replacing uniform
sampling with Goldilocks in various such algorithms, including DER [5], DER++
[5], X-DER [4], AGEM [8], ER [46], GEM [30], RPC [39], and FDR [51]. Our
results demonstrate that Goldilocks consistently improves the performance of
all tested continual algorithms, thereby advancing the state of the art in
multiple image classification tasks.

We replicate the experimental setup outlined in §3.2, exploring various
compositions of the replay buffer in a grid-like fashion. In Fig. 6 we plot a
heatmap example for GEM trained on CIFAR-100-2, employing a buffer size of 500
($2\%$ of the dataset size). We focus on small buffers, as the original works
often mainly report results for such buffers. Our findings mirror our previous
findings on the experience replay case: a large set of hyperparameters
improves the algorithm’s performance, while prioritizing the removal of slowly
learned examples proves advantageous given the buffer’s small size. Comparable
outcomes across different algorithms, datasets, and class incremental learning
scenarios are detailed in Apps. A,E.

Next, we compare continual learning methods with Goldilocks on popular
continual learning benchmarks, including CIFAR-10-5, CIFAR-100-5, and
TinyImageNet-10 datasets. The $quick$ and $slow$ hyperparameters are
determined through RotNet training, as outlined in §3.3. Although these
parameters might vary per task when $T>2$, we maintain consistent values
across tasks to mitigate an excessive number of hyperparameters. Goldilocks
consistently improves various continual learning algorithms, indicating its
compatibility with diverse methods. These results are depicted in Fig. 7.

###  3.5 Comparing Goldilocks to other replay buffer sampling methods

While numerous replay buffer sampling methods have been proposed over the
years, uniform sampling remains prevalent in practical implementation within
continual learning algorithms. This preference may stem from the specific
settings or algorithmic modifications required for other sampling methods to
work well. These methods do not consistently improve different continual
learning algorithms across datasets and buffer sizes. In contrast, Goldilocks
sampling improves a wide range of continual learning algorithms across various
buffer sizes and datasets.

We explored various replay buffer sampling strategies, including uniform
sampling, max entropy, IPM [56], GSS [1], and Herding [44]. We trained $10$
networks on CIFAR-10-2 and CIFAR-100-2 datasets with buffer sizes of $1000$
and $10000$ (equivalent to $4\%$ and $40\%$ of the dataset size, respectively)
using experience replay. Table 3.5 presents the mean test accuracy across all
tasks for each case.

Goldilocks consistently outperforms all other sampling strategies across all
scenarios, surpassing them in all four cases. In contrast, other sampling
methods exhibit efficacy only in specific scenarios. For instance, Herding,
which prioritizes examples with mean features close to class mean features,
performs better than uniform sampling with smaller buffer sizes but worse with
larger buffers. This might be because this preference for typical examples,
which are learned early on, is more beneficial with smaller buffers.
Conversely, GSS, which selects examples based on gradient distances,
outperforms uniform sampling in scenarios with large buffers but performs
suboptimally with smaller budgets. This bias towards selecting examples with
large gradients favors examples learned slowly, which is more beneficial with
large buffers.

(a) CIFAR-100-5

(b) TinyImagenet-10

Figure 7: Goldilocks vs. uniform sampling with various continual learning
methods. Each bar shows the mean final test accuracy across all tasks for each
method, with a uniform sampled buffer (traditionally done) in blue, and
Goldilocks in orange. Error bars denote the standard error. (a) CIFAR-100-5
with a buffer size of $5\%$. (b) TinyImagenet-10 with a buffer of size $10\%$.
Goldilocks significantly improves performance across all methods and datasets.

Table 1: Comparing replay buffer sampling methods. Each entry is the mean test
accuracy across all tasks of 10 networks trained using experience replay on
CIFAR-100-2. Bold denotes the method with the highest accuracy.

| CIFAR-10-2 | CIFAR-100-2
---|---|---
Buffer size | $4\%$ | $40\%$ | $4\%$ | $40\%$
Random | 87.67 | 95.03 | 69.03 | 79.24
Max entropy | 84.44 | 94.63 | 64.27 | 77.91
IPM | 85.9 | 94.76 | 64.16 | 75.73
GSS | 85.28 | 95.55 | 64.15 | 80.11
Herding | 88.41 | 94.53 | 71.07 | 78.97
Goldilocks | 88.99 | 96.03 | 71.53 | 80.62

##  4 Conclusion

Limitations. Goldilocks necessitates maintaining an epoch-wise classification
matrix to compute the _learning speed_ of each example. This estimation’s
accuracy increases with the number of epochs completed. Consequently, in
stream-based approaches, where each data point is encountered only once,
Goldilocks cannot be applied. In such cases, alternative methods for
approximating example complexity may be viable but fall beyond the scope of
this study, leaving them for future work.

Another limitation arises in the potential lengthy computation time when fine-
tuning hyperparameters in multi-task settings, as the $quick$ and $slow$
hyperparameters may differ between each subsequent task. However, in practice,
these hyperparameters can be selected using simple heuristics and kept
constant across tasks, as demonstrated in our training on multiple tasks (see
Fig. 7).

Discussion. We show a correlation between the speed at which examples are
learned and the likelihood of models forgetting these examples when learning
new data. This correlation aligns closely with the simplicity bias, suggesting
that models tend to forget more complex examples. Although measuring _learning
speed_ is a simple proxy for example complexity, its simplicity underscores
the potential for developing more sophisticated methods to mitigate
catastrophic forgetting, offering avenues for future research.

While most replay-based continual learning works focus on better utilization
of the replay buffer, our study demonstrates the potential for buffer
improvement, extending the scope of continual learning research beyond buffer
utilization alone. By showing that Goldilocks can be implemented with minimal
computational overhead, we highlight its applicability to future advancements
in continual learning. Furthermore, this work underscores the mutual benefits
of concurrent progress in better sampling of the replay buffer and
improvements in buffer utilization, suggesting promising directions for future
research.

## References

  * [1] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio.  Gradient based sample selection for online continual learning.  Advances in neural information processing systems, 32, 2019.
  * [2] Robert Baldock, Hartmut Maennel, and Behnam Neyshabur.  Deep learning through the lens of example difficulty.  Advances in Neural Information Processing Systems, 34:10876–10889, 2021.
  * [3] Beatrix Benkő.  Example forgetting and rehearsal in continual learning.  Pattern Recognition Letters, 179:65–72, 2024.
  * [4] Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara.  Class-incremental continual learning into the extended der-verse.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  * [5] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara.  Dark experience for general continual learning: a strong, simple baseline.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 15920–15930. Curran Associates, Inc., 2020.
  * [6] Pietro Buzzega, Matteo Boschini, Angelo Porrello, and Simone Calderara.  Rethinking experience replay: a bag of tricks for continual learning.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 2180–2187. IEEE, 2021.
  * [7] Yuan Cao, Zhiying Fang, Yue Wu, Ding-Xuan Zhou, and Quanquan Gu.  Towards understanding the spectral bias of deep learning.  In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 2205–2211. ijcai.org, 2021.
  * [8] Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny.  Efficient lifelong learning with A-GEM.  In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
  * [9] Leshem Choshen, Guy Hacohen, Daphna Weinshall, and Omri Abend.  The grammar-learning trajectories of neural language models.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8281–8297, 2022.
  * [10] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  IEEE transactions on pattern analysis and machine intelligence, 44(7):3366–3385, 2021.
  * [11] Kamaludin Dingle, Chico Q Camargo, and Ard A Louis.  Input–output maps are strongly biased towards simple outputs.  Nature communications, 9(1):1–7, 2018.
  * [12] Robert M French.  Catastrophic forgetting in connectionist networks.  Trends in cognitive sciences, 3(4):128–135, 1999.
  * [13] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
  * [14] Daniel Gissin, Shai Shalev-Shwartz, and Amit Daniely.  The implicit bias of depth: How incremental learning drives generalization.  In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
  * [15] Suriya Gunasekar, Jason D. Lee, Daniel Soudry, and Nati Srebro.  Implicit bias of gradient descent on linear convolutional networks.  In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31, pages 9482–9491, 2018.
  * [16] Yunhui Guo, Mingrui Liu, Tianbao Yang, and Tajana Rosing.  Improved schemes for episodic memory-based lifelong learning.  Advances in Neural Information Processing Systems, 33:1023–1035, 2020.
  * [17] Guy Hacohen, Leshem Choshen, and Daphna Weinshall.  Let’s agree to agree: Neural networks share classification order on real datasets.  In International Conference on Machine Learning, pages 3950–3960. PMLR, 2020.
  * [18] Guy Hacohen and Daphna Weinshall.  Principal components bias in over-parameterized linear models, and its manifestation in deep neural networks.  Journal of Machine Learning Research, 23(155):1–46, 2022.
  * [19] Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu.  Embracing change: Continual learning in deep neural networks.  Trends in cognitive sciences, 24(12):1028–1040, 2020.
  * [20] Reinhard Heckel and Mahdi Soltanolkotabi.  Denoising and regularization via exploiting the structural bias of convolutional generators.  In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
  * [21] Wei Hu, Lechao Xiao, Ben Adlam, and Jeffrey Pennington.  The surprising simplicity of the early-time learning dynamics of neural networks.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33, 2020.
  * [22] Hui Jin and Guido Montúfar.  Implicit bias of gradient descent for mean squared error regression with two-layer wide neural networks.  J. Mach. Learn. Res., 24:137:1–137:97, 2023.
  * [23] Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin L. Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang.  SGD on neural networks learns functions of increasing complexity.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32, pages 3491–3501, 2019.
  * [24] Ronald Kemker, Marc McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan.  Measuring catastrophic forgetting in neural networks.  In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  * [25] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  * [26] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  Online, 2009.
  * [27] Ya Le and Xuan Yang.  Tiny imagenet visual recognition challenge.  CS 231N, 7(7):3, 2015.
  * [28] Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung-Tak Zhang.  Overcoming catastrophic forgetting by incremental moment matching.  Advances in neural information processing systems, 30, 2017.
  * [29] Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong.  Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting.  In International conference on machine learning, pages 3925–3934. PMLR, 2019.
  * [30] David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  Advances in neural information processing systems, 30, 2017.
  * [31] Pratyush Maini, Saurabh Garg, Zachary Lipton, and J Zico Kolter.  Characterizing datapoints via second-split forgetting.  Advances in Neural Information Processing Systems, 35:30044–30057, 2022.
  * [32] Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989.
  * [33] Monica Millunzi, Lorenzo Bonicelli, Angelo Porrello, Jacopo Credi, Petter N Kolm, and Simone Calderara.  May the forgetting be with you: Alternate replay for learning with noisy labels.  2023\.
  * [34] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh.  Understanding the role of training regimes in continual learning.  Advances in Neural Information Processing Systems, 33:7308–7320, 2020.
  * [35] Cuong V Nguyen, Alessandro Achille, Michael Lam, Tal Hassner, Vijay Mahadevan, and Stefano Soatto.  Toward understanding catastrophic forgetting in continual learning.  arXiv preprint arXiv:1908.01091, 2019.
  * [36] Giang Nguyen, Shuan Chen, Thao Do, Tae Joon Jun, Ho-Jin Choi, and Daeyoung Kim.  Dissecting catastrophic forgetting in continual learning by deep visualization.  arXiv preprint arXiv:2001.01578, 2020.
  * [37] German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter.  Continual lifelong learning with neural networks: A review.  Neural networks, 113:54–71, 2019.
  * [38] Guillermo Valle Pérez, Chico Q. Camargo, and Ard A. Louis.  Deep learning generalizes because the parameter-function map is biased towards simple functions.  In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
  * [39] Federico Pernici, Matteo Bruni, Claudio Baecchi, Francesco Turchini, and Alberto Del Bimbo.  Class-incremental learning with pre-allocated fixed classifiers.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 6259–6266. IEEE, 2021.
  * [40] Benedikt Pfülb and Alexander Gepperth.  A comprehensive, application-oriented study of catastrophic forgetting in dnns.  In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
  * [41] Ameya Prabhu, Philip HS Torr, and Puneet K Dokania.  Gdumb: A simple approach that questions our progress in continual learning.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 524–540. Springer, 2020.
  * [42] Vinay Venkatesh Ramasesh, Ethan Dyer, and Maithra Raghu.  Anatomy of catastrophic forgetting: Hidden representations and task semantics.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  * [43] Rahul Ramesh and Pratik Chaudhari.  Model zoo: A growing brain that learns continually.  In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
  * [44] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.  icarl: Incremental classifier and representation learning.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017.
  * [45] Hippolyt Ritter, Aleksandar Botev, and David Barber.  Online structured laplace approximations for overcoming catastrophic forgetting.  Advances in Neural Information Processing Systems, 31, 2018.
  * [46] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne.  Experience replay for continual learning.  Advances in neural information processing systems, 32, 2019.
  * [47] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou.  Overcoming catastrophic forgetting with hard attention to the task.  In International conference on machine learning, pages 4548–4557. PMLR, 2018.
  * [48] Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli.  The pitfalls of simplicity bias in neural networks.  Advances in Neural Information Processing Systems, 33:9573–9585, 2020.
  * [49] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro.  The implicit bias of gradient descent on separable data.  The Journal of Machine Learning Research, 19(1):2822–2878, 2018.
  * [50] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
  * [51] Michalis K. Titsias, Jonathan Schwarz, Alexander G. de G. Matthews, Razvan Pascanu, and Yee Whye Teh.  Functional regularisation for continual learning with gaussian processes.  In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
  * [52] Rishabh Tiwari, Krishnateja Killamsetty, Rishabh Iyer, and Pradeep Shenoy.  Gcr: Gradient coreset based replay buffer selection for continual learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 99–108, 2022.
  * [53] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon.  An empirical study of example forgetting during deep neural network learning.  In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
  * [54] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.  Deep image prior.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9446–9454, 2018.
  * [55] Felix Wiewel and Bin Yang.  Entropy-based sample selection for online continual learning.  In 2020 28th European Signal Processing Conference (EUSIPCO), pages 1477–1481. IEEE, 2021.
  * [56] Alireza Zaeemzadeh, Mohsen Joneidi, Nazanin Rahnavard, and Mubarak Shah.  Iterative projection and matching: Finding structure-preserving representatives and its application to computer vision.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5414–5423, 2019.
  * [57] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning (still) requires rethinking generalization.  Communications of the ACM, 64(3):107–115, 2021.

## Appendix

##  Appendix A Class incremental learning

In continual learning, a distinction is made between class incremental
learning and task incremental learning. The key difference lies in test time:
in task incremental learning, the model knows the task identity of the given
examples, allowing it to use a specialized classifier that differentiates
between classes within that task. Conversely, in class incremental learning,
the model lacks task identity information at test time and must identify both
the task and the class of the example, typically resulting in lower
performance. Due to space constraints, we report only the task-incremental
results in the main paper, as it is the more common benchmark. However, we
also tested for class incremental settings in all our experiments. While the
performance is lower, the qualitative outcomes remain consistent: buffer
compositions that are good for task incremental learning are also good for
class incremental settings and vice versa, and Goldilocks significantly
enhances a variety of continual learning algorithms in both scenarios.

In Fig.8(a), we extend the results from Fig.4(b) to class incremental
settings. We examine different buffer compositions by adjusting the $quick$
and $slow$ hyperparameters of Goldilocks and plot the mean final test accuracy
across all tasks for 10 networks trained on CIFAR-100-2 with a $40\%$ buffer,
using experience replay. The qualitative results are consistent: the same
buffer compositions are effective for both learning scenarios. In Fig.8(b), we
replicate Fig.6, applying the same experiment with GEM on CIFAR-100-2 using a
$1\%$ buffer, confirming that the results align with those reported in the
main paper.

Finally, in Fig.9, we reproduce the results from Fig.7, showing the
performance of competitive continual learning algorithms when changing only
the buffer sampling strategy to Goldilocks, across CIFAR-100-5, CIFAR-10-5,
and TinyImageNet-10. In all datasets and with all continual learning
algorithms, Goldilocks significantly improves the performance of the original
methods by a significant margin.

(a) CIFAR-100-2, $40\%$ buffer

(b) GEM, CIFAR-100-2, $1\%$ buffer

Figure 8: Comparing different replay buffer compositions in the class
incremental case. Each matrix compares the mean final test accuracy of 10
networks trained with different replay buffer compositions achieved by
modifying Goldilocks’ $quick$ and $slow$ hyperparameters. The ($quick$,
$slow$) = ($0\%$, $0\%$) case, marked by a black box, corresponds to uniform
sampling. (a) Replicating Fig. 4(b) with class incremental learning (b)
Replicating Fig. 6 with class incremental learning. In both cases, the results
mirror the task-incremental case, although with lower performances.

(a) CIFAR-100-5

(b) CIFAR-10-5

(c) TinyImagenet-10

Figure 9: Goldilocks vs. uniform sampling with various continual learning
methods, in class incremental settings. Each bar shows the mean final test
accuracy across all tasks for each method, with a uniform sampled buffer
(traditionally done) in blue, and Goldilocks in orange. Error bars denote the
standard error.

(a) CIFAR-100-2

(b) CIFAR-10-2

(c) TinyImagenet-2

(d) CIFAR-100-2

(e) CIFAR-10-2

(f) TinyImagenet-2

Figure 10: Extendeing Fig.2 to CIFAR-100-2, Cifar-10-2 and a subset of
TinyImagenet-2. (top) The first task’s binary epoch-wise classification
matrices $M$ for the test data of each dataset. Rows denote examples, and
columns denote epochs, indicating correct classifications by a majority of the
networks. Rows are sorted by _learning speed_. Faster-learned examples from
the first task are less likely to be forgotten at the end of the second task.
(bottom) The mean _learning speed_ of examples in the first task vs. the
percentage of networks that remember them at the end of the second task.
Networks forget more examples learned slowly across all 3 datasets. (a)
CIFAR-10-2 Figure 11: Extending Fig. 3(d) to CIFAR-10-2. We plot the mean
_learning speed_ of remembered examples by $10$ models trained with different
buffer sizes. Models with bigger replay buffers remember slower-to-learn
examples.

(a) ResNet-18

(b) Small ResNet

(c) Tiny ResNet

Figure 12: Comparing the extended epoch-wise classification matrix for
different architectures, rows represent examples, and columns represent
epochs, indicating correct classifications by a majority of 10 networks. In
(a), we train ResNet-18. In (b), we train ResNet-18 with both width and depth
reduced by half. In (c), we train ResNet-18 with both width and depth reduced
by a factor of four. In all cases, rows are sorted by _learning speed_.
Examples learned quickly in the first task are less likely to be forgotten
after the second task.

(a) DER

(b) DER++

(c) GEM

(d) AGEM

(e) RPC

(f) X-DER

Figure 13: Comparing different replay buffer compositions for various
continual learning methods, with a buffer of $1\%$. The experimental setup
replicates the one done in Fig. 3.2. Like the experience replay case, focusing
on examples learned midway through the learning process is most beneficial.
Due to the smaller buffer, removing more easier examples is better.

##  Appendix B Correlations between learning speed and catastrophic forgetting
in different datasets

In Section 2, we established a link between the _learning speed_ of examples
and catastrophic forgetting, primarily utilizing ResNet-18 networks trained on
CIFAR-100-2. Here, we expand upon these findings by demonstrating similar
phenomena across different datasets, architectures, and task numbers.

Other Datasets. We first explore other datasets using the experimental setup
detailed in Fig. 2. We extend our analysis to CIFAR-10-2, CIFAR-100-2, and a
subset of TinyImagenet-2 (comprising the initial 40 classes). The extended
epoch-wise classification matrices for each task and the correlation between
_learning speed_ and the percentage of networks remembering each example are
plotted in Fig. 10. Consistently across all three datasets, we observe a
robust correlation between _learning speed_ and the likelihood of example
retention during continual training. Notably, faster-learned examples exhibit
lower rates of catastrophic forgetting, as evidenced by both quantitative
correlations and visual inspection of the extended epoch-wise classification
matrices.

Further, in Fig. 11, we expand upon Fig.3 to include CIFAR-10-2. This
extension reveals that as the buffer size increases, continual models tend to
retain slower-to-learn examples. Analogous to the results observed in
CIFAR-100-2, as depicted in Fig.3, we also observe a near-perfect negative
correlation between the buffer size and the mean learning speed of remembered
examples in CIFAR-10-2.

Other Architectures. We extend the results from Fig. 2 to architectures beyond
ResNet-18 by creating two new, smaller variants: Small-ResNet and Tiny-ResNet.
These are formed by reducing both the width and depth of ResNet-18 by a factor
of 2 and 4 respectively. We replicate the experimental setup from Fig.2,
training 10 models of each architecture on CIFAR-100-2 without a replay
buffer, and storing the extended epoch-wise classification matrix. Each matrix
row is sorted by the _learning speed_ of the examples and plotted in Fig. 12.
All matrices depict the classification of the test dataset. In all
architectures, _learning speed_ is highly correlated with catastrophic
forgetting: networks forget more examples learned later in training while
retaining an almost perfect recollection of those learned early on.
Additionally, stronger architectures, similar to larger buffers, can remember
slower-to-learn examples. This reinforces the motivation for Goldilocks, as it
shows that the examples selected by Goldilocks are those the model could learn
independently if it had a stronger architecture.

Other continual learning algorithms. In §3.4, we demonstrated the benefits of
non-uniform buffer compositions for various continual learning methods. We
extend these findings, initially shown for GEM in Fig. 6, to additional
algorithms including DER, DER++, GEM, A-GEM, RPC, and X-DER. For each
algorithm, we trained 10 networks with different buffer compositions by
varying the $quick$ and $slow$ hyperparameters of Goldilocks. For each
algorithm, we trained 10 networks on different buffer compositions, achieved
by varying the $quick$ and $slow$ hyperparameters of Goldilocks. In all cases,
we used the hyperparameters and architectures suggested in the original works.
We used a buffer of size $500$ ($1\%$ of the data), due to its popularity in
previous works. Consistent with the GEM results, we found that replay buffers
focused on examples learned mid-way through the training process were most
beneficial. Additionally, in all cases, removing slower-to-learn examples
proved advantageous, given the small buffer size. These results can be found
in Fig. 13.

Different number of tasks.

In most experiments presented in the paper, we focused on continual learning
with 2 tasks. When considering a larger number of tasks, additional
hyperparameters emerge, as the $quick$ and $slow$ parameters apply to each
intermediate task. These can be tuned similarly to the two-task case, either
through grid-search with a rotation task or using simple heuristics. To avoid
excessive hyperparameter tuning when training on more than two tasks, we tuned
the $quick$ and $slow$ values for the first task and kept them constant for
all subsequent tasks. Results from training on CIFAR-10-5, CIFAR-100-5, and
TinyImageNet-10 are shown in Fig. 17. We observe that Goldilocks improves over
uniform sampling in these cases as well. Notably, further tuning of all
Goldilocks hyperparameters could yield even better results.

##  Appendix C standard errors

In the main paper, we omitted the standard errors for visualization porpuses
from Figs. 3.2,5,6 and Table 3.5. In both cases, these were usually very small
and did not affect the qualitative results. For completeness, we report the
omitted values in this section. The standard error for Figs. 3.2,5,6 can be
found in Figs. C,15,16 respectively. A replication of Table 3.5, with the
standard errors, can be found in Table C.

(a) CIFAR-100-2, $4\%$ buffer

(b) CIFAR-100-2, $40\%$ buffer

(c) TinyImagenet-2, $20\%$ buffer

Figure 14: Standard errors for Fig. 3.2. The error is taken over 10
repetitions in each experiment.

(d) (A)–>(B)

(e) (A)–>(C)

Figure 15: Standard errors for Fig. 5. The error is taken over 10 repetitions
in each experiment.

(a) GEM, CIFAR-100-2, $1\%$ buffer Figure 16: Standard errors for Fig. 6. The
error is taken over 10 repetitions in each experiment.

Table 2: Same as Table 3.5, but with the addition of standard errors. Each
entry is the mean test accuracy across all tasks of 10 networks trained using
experience replay on CIFAR-100-2. Bold denotes the method with the highest
accuracy.

| CIFAR-10-2 | CIFAR-100-2
---|---|---
Buffer size | $4\%$ | $40\%$ | $4\%$ | $40\%$
Random | 87.67 $\pm$ 0.06 | 95.03 $\pm$ 0.05 | 69.03 $\pm$ 8.89 | 79.24 $\pm$ 1.82
Max entropy | 84.44 $\pm$ 0.23 | 94.63 $\pm$ 0.06 | 64.27 $\pm$ 2.11 | 77.91 $\pm$ 0.91
IPM | 85.9 $\pm$ 0.16 | 94.76 $\pm$ 0.13 | 64.16 $\pm$ 4.13 | 75.73 $\pm$ 1.43
GSS | 85.28 $\pm$ 0.13 | 95.55 $\pm$ 0.08 | 64.15 $\pm$ 5.01 | 80.11 $\pm$ 0.81
Herding | 88.41 $\pm$ 0.09 | 94.53 $\pm$ 0.12 | 71.07 $\pm$ 9.49 | 78.97 $\pm$ 0.1
Goldilocks | 88.99 $\pm$ 0.03 | 96.03 $\pm$ 0.1 | 71.53 $\pm$ 7.31 | 80.62 $\pm$ 1.11

(a) CIFAR-100-5

(b) CIFAR-10-5

(c) TinyImagenet-10

Figure 17: Goldilocks vs. uniform sampling with various continual learning
methods. Each bar shows the mean final test accuracy across all tasks for each
method, with a uniform sampled buffer (traditionally done) in blue, and
Goldilocks in orange. Error bars denote the standard error. (a) CIFAR-100-5
with a buffer size of $5\%$. (b) CIFAR-10-5 with a buffer of size $5\%$ (c)
TinyImagenet-10 with a buffer of size $10\%$. Goldilocks significantly
improves performance across all methods and datasets.

(a) CIFAR-100 –> CIFAR-100

(b) CIFAR-100 –> rotated

(c) CIFAR-100 –> random labels

Figure 18: Comparing replay buffer compositions when training the same
original task, followed by different subsequent tasks. Each entry in the
matrices denotes the mean final test accuracy of $10$ networks, trained with
experience replay on 2 tasks, with a buffer size of $40\%$. This Figure
replicated the same experimental setup as Fig. 5. In (a) we train the models
on the 50 first classes of CIFAR-100, followed by the last 50 classes of
CIFAR-100. In (b) we train the models on the 50 first classes of CIFAR-100,
followed by a rotation classification task on the same examples (see text). In
(c) we train the models on the 50 first classes of CIFAR-100, followed by the
last 50 classes of CIFAR-100, but with random labels. In all cases, the same
buffer compositions of the first task give the same qualitative results,
suggesting that the optimal composition of the buffer of the first task is
independent of the subsequent task.

##  Appendix D Independency of the optimal buffer composition when changing
the subsequent tasks

We empirically examine how subsequent tasks impact the ideal composition of a
replay buffer for a given task. We find that regardless of the similarity or
dissimilarity between subsequent tasks and the original task, the optimal
replay buffer composition remains largely independent and consistent. To show
this, we conduct experiments with three distinct tasks denoted A, B, and C. By
training the model on task A and subsequently introducing either task B or
task C while keeping the original task unchanged, we analyze the ideal replay
buffer composition under varying subsequent tasks.

In Section 3.3, we demonstrate that when selecting tasks A, B, and C from the
same dataset, the optimal buffer composition of task A remains unaffected by
tasks B or C. We observe consistent quantitative behavior across different
$quick$ and $slow$ parameters. We further explore task variations by setting A
as the first 50 classes of CIFAR-100, B as the last 50 classes of CIFAR-100,
and C as a rotation classification [13] task. In the rotation classification
task, examples from A are randomly rotated by angles of
$\\{90\degree,180\degree,270\degree\\}$, with labels adjusted accordingly. We
evaluate the performance of 10 networks trained continuously with different
buffer compositions first on task A and then on task B (Fig.18(a)), and on
task A followed by task C (rotation) (Fig.18(b)). Consistent with previous
findings, the performance of various buffer compositions remains consistent,
independent of the choice of B or C. This property enables us to utilize the
rotation task for hyper-parameter grid search without requiring additional
label data.

We further modify task C to comprise examples from the last 50 classes of
CIFAR-100 but with random labels [57]. Training on random labels forces the
network to memorize the examples, as no generalization is possible,
eliminating any potential overlap between the subsequent tasks. The results of
training on task A followed by the random label task are depicted in Fig.
18(c). Analogous to the rotation classification scenario, we observe that the
same buffer compositions of task A remain effective, further suggesting that
the optimal buffer composition of A does not depend on the subsequent task.

##  Appendix E Goldilocks with different continual learning methods

Here, we compare Goldilocks to uniform sampling with different continual
learning algorithms. We tested several different algorithms on CIFAR-100-5,
CIFAR-10-5, and TinyImagenet-10. The results, which extend Fig. 7, are plotted
in Fig. 17.