# FOCIL: Finetune-and-Freeze for Online Class Incremental Learning
by Training Randomly Pruned Sparse Experts

Murat Onur Yildirim  Elif Ceren Gok Yildirim  Decebal Constantin Mocanu
Joaquin Vanschoren

###### Abstract

Class incremental learning (CIL) in an online continual learning setting
strives to acquire knowledge on a series of novel classes from a data stream,
using each data point only once for training. This is more realistic compared
to offline modes, where it is assumed that all data from novel class(es) is
readily available. Current online CIL approaches store a subset of the
previous data which creates heavy overhead costs in terms of both memory and
computation, as well as privacy issues. In this paper, we propose a new online
CIL approach called FOCIL. It fine-tunes the main architecture continually by
training a randomly pruned sparse subnetwork for each task. Then, it freezes
the trained connections to prevent forgetting. FOCIL also determines the
sparsity level and learning rate per task adaptively and ensures (almost) zero
forgetting across all tasks without storing any replay data. Experimental
results on 10-Task CIFAR100, 20-Task CIFAR100, and 100-Task TinyImagenet,
demonstrate that our method outperforms the SOTA by a large margin. The code
is publicly available at https://github.com/muratonuryildirim/FOCIL.

Machine Learning, ICML

##  1 Introduction

Current artificial intelligence systems are prone to forgetting previously
learned information when confronted with new tasks, commonly known as
catastrophic forgetting (McCloskey & Cohen, 1989). Continual learning (CL)
tries to tackle this phenomenon by striving to acquire knowledge from a
nonstationary data stream, presenting a potential path for emulating human
cognitive abilities. Class incremental learning (CIL) (Van de Ven & Tolias,
2019) is a challenging scenario of CL where models learn a sequence of tasks
and the task identifiers at inference are not accessible.

Traditional CIL involve retaining all training data for the current task, and
the model undergoes multiple epochs of training with repeated shuffles.
However, constraints such as privacy and memory may limit access to all data.

This study focuses on online CIL, a more demanding and realistic scenario in
which samples from each task can only be used once and access to previous
batches is not available. To avoid catastrophic forgetting in online CIL,
various methods are proposed, and most of which -except one (Guo et al.,
2022a)\- are built on replay-based CL in which a subset of the previous
training data is stored to train the model on both the new data and stored
data together to adjust the features and decision boundaries for all classes.
However, storing a subset of the data (i) brings overhead costs in terms of
both memory and computation, (ii) violates privacy concerns, and (iii)
complicates data retrieval and training.

In this paper, we aim to learn continually in an online manner without storing
any exemplars. To this end, we present Finetune-and-Freeze for Online Class
Incremental Learning by Training Randomly Pruned Sparse Experts (FOCIL). It
has a very simple working principle: For every task, it prunes an
overparameterized neural network to obtain a task-specific sparse subnetwork
(or expert), then trains the new connections that belong to this expert only.
Once trained, it freezes them to prevent catastrophic forgetting. Our
contributions are summarized as follows:

  * •

To our knowledge, FOCIL is the first approach that utilizes random network
pruning and sparse subnetworks trained from scratch in online class
incremental learning.

  * •

FOCIL only fine-tunes the backbone continually without any complicated
training procedure while adaptively determining the sparsity level and
learning rate per task.

  * •

FOCIL secures (almost) zero forgetting across all tasks without storing any
replay data. This reduces memory and computation expenses, as well as privacy
concerns.

  * •

Experiments show that FOCIL surpasses SOTA methods by a large margin: For
example, it remarkably improves upon the top baseline almost 4 times in terms
of accuracy on 100-Task TinyImageNet.

Figure 1: During training (left), FOCIL starts with an overparameterized
network. Based on the search space, which includes the sparsity level and the
learning rate, it applies random unstructured pruning to the network to find a
distinct task-specific sparse subnetwork, which we call an expert. Then, it
trains the expert’s weights that have not been assigned to prior ones. In
other words, it freezes the parameters of already assigned connections to
prevent catastrophic forgetting. However, it allows using the frozen
parameters in the subsequent tasks to facilitate the knowledge transfer which
is particularly important in large-scale incremental learning. This is
repeated for every task the network encounters. During inference (right),
FOCIL only necessitates a single batch to identify the corresponding expert.

##  2 Related Work

#### Continual Learning.

Many approaches proposed for CL require and assume that the whole training
data is readily available for the task at hand and train the model multiple
times with repeated shuffling. These methods include regularization-based
techniques with consolidation (Kirkpatrick et al., 2017; Zenke et al., 2017;
Chaudhry et al., 2018; Aljundi et al., 2018; Lee et al., 2020) or knowledge
distillation (Li & Hoiem, 2017; Hou et al., 2018; Kang et al., 2022b) to
optimize model stability, replay-based methods with storing (Rebuffi et al.,
2017; Lopez-Paz & Ranzato, 2017; Wu et al., 2019; Zhao et al., 2020; Bang et
al., 2021; Liu et al., 2021b; Arani et al., 2022; Sarfraz et al., 2023) or
generating (Mocanu et al., 2016b; Shin et al., 2017; He et al., 2018; Hu et
al., 2019; Petit et al., 2023) samples to retain past experiences,
architecture expansions (Aljundi et al., 2017; Liu et al., 2021a; Pham et al.,
2021; Yan et al., 2021; Wang et al., 2022a, 2023; Douillard et al., 2022; Zhou
et al., 2023) to optimize model plasticity, and parameter isolation with mask
learning (Mallya et al., 2018), iterative pruning (Mallya & Lazebnik, 2018;
Golkar et al., 2019; Wortsman et al., 2020; Dekhovich et al., 2023) and
dynamic sparse training (Sokar et al., 2021; Gurbuz & Dovrolis, 2022; Kang et
al., 2022a; Wang et al., 2022b; Sokar et al., 2023; Kang et al., 2023) to
maintain the key parameters.

However, accessing the whole data at once is not always possible, and
performing multiple epochs of training imposes a significant computational
expense.

#### Online Class Incremental Learning.

Online CIL assumes that the data for each task comes gradually in a stream and
can only be trained once. AGEM (Chaudhry et al., 2019a) modifies the loss
function of GEM (Lopez-Paz & Ranzato, 2017) to accelerate training with small
episodic memory. GSS (Aljundi et al., 2019b) improves learning by diversifying
gradients within the memory buffer, allowing the model to extract a wider
range of information from its stored experiences. ER (Chaudhry et al., 2019b)
selects samples for the replay data randomly. MIR (Aljundi et al., 2019a)
chooses replay samples characterized by maximal increases in loss magnitude.
GDumb (Prabhu et al., 2020) employs a greedy sampling strategy and maintains a
balanced class representation, utilizing all samples in memory while training
the learner. ASER (Shim et al., 2021) suggests buffer management using Shapley
values. SCR (Mai et al., 2021) employs supervised contrastive loss in the
process of representation learning and adopts the nearest class mean
classifier. CoPE (De Lange & Tuytelaars, 2021) relies on prototypes with a
high momentum-based update strategy for every observed batch. DVC (Gu et al.,
2022) improves online class-incremental learning through gradient-based sample
selection and maximization of mutual information between training image views.
OCM (Guo et al., 2022b) prevents forgetting by maximizing mutual information.
OnPro (Wei et al., 2023) introduces an online prototype equilibrium to
alleviate bias and class confusion, along with an adaptive prototypical
feedback mechanism to improve boundary definitions.

Except for AOP (Guo et al., 2022a), which leverages a pre-trained backbone,
online CIL approaches rely on replay-based CL which requires storing a subset
of seen data in a memory bank. At each time step of the task, the model
receives a mini-batch data $X_{t}\cup X_{b}$ for training, where $X_{t}$ and
$X_{b}$ are drawn from the i.i.d. distribution streaming data $D_{t}$ for task
$t$ and the memory bank $M$, respectively.

However, storing a subset of the data for a replay may be restricted due to
constraints such as privacy, memory, and computation. These factors impose
some limitations on current online CIL approaches.

#### Pruning at Initialization.

In offline supervised learning, pruning at initialization approaches remove
some connections before training, and only a subset of parameters undergoes
the training process. The naive approach is pruning each layer uniformly with
the same pruning ratio, known as uniform pruning or initialization (Mariet &
Sra, 2015; He et al., 2017; Gale et al., 2019). Instead of uniform pruning, ER
(Mocanu et al., 2018) and ERK (Evci et al., 2020) introduced a more advanced
layer-wise pruning approach from the field of graph theory with strong
results.

Instead of a predefined rule-based sparsity ratio, many works proposed to
adjust the layerwise sparsity ratios before training. SNIP (Lee et al., 2019)
only keeps the connections with a large weight and gradient which are obtained
by a single batch iteration and prunes the rest of the connections. GraSP
(Wang et al., 2020) prunes connections by computing the Hessian gradient
product. SynFlow (Tanaka et al., 2020) proposes a way to iteratively prune the
network to avoid layer collapse without ever looking at the data.

From a more theoretical perspective, Mocanu et al. (2016a) showed that pruning
a restricted boltzmann machine randomly at initialization performs as well as
its dense counterpart. Liu et al. (2022) demonstrated that randomly pruned
backbones are effective enough as long as the backbone is wide enough and
Gadhikar et al. (2023) proved that randomly pruned random networks are
sufficiently expressive.

However, to our knowledge, none of the existing studies have considered
pruning at initialization within the context of continual learning or (online)
class incremental learning.

##  3 FOCIL

Figure 1 illustrates the FOCIL method. In this section, we provide motivation,
preliminaries, and optimization objectives, and explain our expert
identification approach as well as the learning rate and sparsity level
adaptation strategy.

#### Motivation.

Deep Neural Networks are typically over-parameterized and exhibit the
capability of maintaining multiple subnetworks effectively (Frankle & Carbin,
2019; Liu et al., 2020).

This characteristic allows the learning of more than one expert in a single
model, akin to the brain where only a few neurons respond to any given
stimulus (Babadi & Sompolinsky, 2014). It underscores that different parts of
the brain have distinct responsibilities or functions.

In offline supervised learning, Liu et al. (2022) demonstrated that randomly
pruned sparse networks perform as well as dense counterparts, particularly in
the context of wide neural networks. Furthermore, Gadhikar et al. (2023)
showed that randomly pruned networks are sufficiently expressive and can
effectively approximate any target network if they are wider by a certain
factor.

In offline class incremental learning, Mirzadeh et al. (2022a, b) proved that
increasing the parameter count does not invariably correlate with enhanced
performance unless such a parameter increase specifically targets the width of
the network.

Notably, many models developed for offline supervised learning such as AlexNet
and ResNet tend to be inherently very wide for continual learning tasks. For
instance, in a 20-task CIFAR100 scenario, the data for each task is 20 times
smaller than the original dataset, which makes many models sufficiently wide.

Algorithm 1 FOCIL Training Algorithm

Input: Model $\theta$

Tasks $\mathcal{T}$

Trials $\mathcal{N}$

Search space for learning rate $\eta_{space}$

Search space for sparsity level $s_{space}$

for $t$ in $\mathcal{T}$ do

for $n$ in $\mathcal{N}$ do

$s\leftarrow s_{space}$

$\eta\leftarrow\eta_{space}$

Use Eq. 1 to randomly prune $\theta$ with sparsity level $s$

Use Eq. 2 to finetune $\theta$ by training $\theta_{s}$ with $\eta$

end for

return $\theta_{s}^{*}$

end for

Drawing inspiration from these intuitions and previous findings, we propose
training a not wide deep neural network model by performing random pruning at
each task initialization to find sparse experts that are responsible for
different tasks in online class incremental learning scenarios.

#### Preliminaries.

Online CIL involves updating a neural network where the training data for each
task come gradually as a stream with a sequence of learning tasks
$\mathcal{T}_{1:t}=(\mathcal{T}_{1},\mathcal{T}_{2},...,\mathcal{T}_{t})$,
each with a corresponding dataset
$\mathcal{D}_{t}~{}=~{}{(x_{i,t},y_{i,t})^{n_{t}}}$ consisting of $n_{t}$
instances per task. In a new learning task, a deep network is optimized to map
the input instance to the classifier space, denoted as
$f_{\theta}:\mathcal{X}_{t}\rightarrow\mathcal{Y}_{t}$, where $\theta$
represents the network parameters. Note that the learning tasks are mutually
exclusive, i.e., $\mathcal{Y}_{t-1}\cap\mathcal{Y}_{t}=~{}\emptyset,~{}\forall
t\in\mathcal{T}$.

During online class-incremental learning, it is necessary to have a unified
model that can make accurate predictions for both new and previously
encountered classes. To achieve this, we use a single-head classifier that is
expanded each time the model is trained with a new task. This is necessary
because, during inference, a unified classifier must choose labels from all
the classes that have been observed since task identifiers are not available.
Note that this is different from task incremental learning which requires task
identifiers.

#### Pruning.

To find an expert for each task, we apply random unstructured pruning to the
neural network using Erdős-Rényi Kernel (ERK) (Evci et al., 2020) that has
been proven to perform more effectively in CL (Yildirim et al., 2023). While
keeping the total sparsity at the level of $s$, ERK scales the sparsity of the
layers proportional to

$\displaystyle 1-\frac{(n^{l-1}+n^{l}+w^{l}+h^{l})}{n^{l-1}n^{l}w^{l}h^{l}}$
(1)

where $n^{l}$ denotes the number of neurons or feature maps at layer $l$, and
$w^{l}$ and $h^{l}$ indicate the width and height of the $l^{th}$
convolutional kernel. It allocates higher sparsities to layers with more
parameters and lower sparsities to layers with fewer parameters to protect the
information flow. The sparsity of the fully connected layers scales as in the
original Erdős-Rényi (Mocanu et al., 2018).

#### Objective.

FOCIL aims to effectively learn an expert per task in an online manner by
solving the optimization problem that minimizes the following function for
each task:

$\displaystyle\small\theta_{s}^{*}$
$\displaystyle=\arg\min\sum_{i=1}^{n_{t}}[CE(f_{\theta_{s}}(x_{i,t}),y_{i,t})]$
(2)

Here, $CE$ is the Cross-Entropy Loss and $\theta_{s}$ is the output of random
pruning, and the optimal $\theta_{s}^{*}$ is determined by solving this
optimization problem (Eq. 2). After finding $\theta_{s}^{*}$, FOCIL freezes
these connections to avoid catastrophic forgetting. However, it allows shared
connections between experts. Hence, the overall usage of the backbone is task-
dependent and highly related to the extent of shared knowledge among tasks,
although the sparsity level per task is predetermined.

We describe a detailed flow and optimization procedure of FOCIL in Algorithm
1.

#### Expert Selection.

Similar to (Rajasegaran et al., 2020; Wortsman et al., 2020; Dekhovich et al.,
2023), we posit that the model’s predictive confidence relies on accurate
alignment with the corresponding expert. This prompts us to expect that the
entropy of the signal within the most relevant expert should demonstrate a
lower magnitude compared to alternative experts. By leveraging these nuanced
entropy dynamics, we identify the responsible expert with a single batch of
streaming data by aiming for the lowest entropy and highest confidence.

#### Adaptivity.

FOCIL treats the learning rate ($\eta$) and the sparsity level ($s$) as
variables to be searched and optimized, given a search space $\eta_{space}$
and $s_{space}$ with a predefined number of trials
$\mathcal{N}_{1:n}=(\mathcal{N}_{1},\mathcal{N}_{2},...,\mathcal{N}_{n})$.
Despite the availability of more sophisticated search algorithms like Bayesian
Optimization (Bergstra et al., 2011), FOCIL opts for a simple Random Search
(RS) (Bergstra & Bengio, 2012) along with Asynchronous Successive Halving (Li
et al., 2018) which accelerates the search. These choices are driven by their
practicality in an online setup; they are simple, parallelizable, robust to
noisy objectives, and have been proven to work well in practice (Hutter et
al., 2019).

##  4 Experiments

In this section, we describe our experimental setup, present our findings and
results, and provide an ablation study.

###  4.1 Experimental Setup

#### Backbones.

We use ResNet18 (He et al., 2016) in our main experiments. It has an initial
convolutional layer with a 7x7 kernel and 64 feature maps, followed by four
residual blocks, each containing two convolutional layers with 3x3 kernels,
and the number of feature maps doubling from 64 to 512 in each successive
block. To perform additional experiments on a different architecture, we also
use a simple multilayer perceptron (MLP) with two hidden layers which have 300
and 100 neurons respectively before the classifier head and share its results
in Section A.4.

#### Datasets.

We use CIFAR100 (Krizhevsky et al., 2009) and TinyImageNet (Le & Yang, 2015)
for ResNet18, MNIST (LeCun et al., 2010) and Fashion-MNIST (Xiao et al., 2017)
for MLP; with different scenarios. 10-Task CIFAR-100 has 10 disjoint tasks and
each task has 10 disjoint classes with 5000 samples for training and 1000
samples for testing. 20-Task CIFAR-100 has 20 disjoint tasks and each task has
5 disjoint classes with 2500 samples for training and 500 samples for testing.
100-Task TinyImageNet has 100 disjoint tasks and each task has 2 disjoint
classes with 1000 samples for training and 100 samples for testing. 5-Task
MNIST and 5-Task Fashion-MNIST have 5 disjoint tasks and each task has 2
disjoint classes with 12000 samples for training and 2000 samples for testing.

Table 1: Average accuracy [%] (higher is better) of our memory-free (M=0)
approach FOCIL compared to baselines with different memory sizes M on
different benchmarks. We report the mean and standard deviation across ten
independent runs for all experiments conducted on FOCIL and baselines in the
20-Task CIFAR100. The average accuracy of the baselines for 10-Task CIFAR100
and 100-Task TinyImageNet are reported performances from (Wei et al., 2023).
The best results are highlighted in bold.

Method | 10-Task CIFAR100 | 20-Task CIFAR100 | 100-Task TinyImageNet
---|---|---|---
M=0.5k | M=1k | M=2k | M=0.5k | M=1k | M=2k | M=1k | M=2k | M=4k
AGEM (Chaudhry et al., 2019a) | 5.8±0.1 | 5.9±0.1 | 5.8±0.1 | 3.1±0.3 | 3.1±0.2 | 3.2±0.2 | 0.8±0.1 | 0.8±0.1 | 0.8±0.1
GSS (Aljundi et al., 2019b) | 8.1±0.2 | 9.4±0.5 | 10.1±0.8 | 6.6±0.5 | 9.1±0.5 | 11.4±1.9 | 1.1±0.1 | 1.5±0.1 | 2.4±0.4
ER (Chaudhry et al., 2019b) | 8.7±0.3 | 9.9±0.5 | 10.7±0.8 | 6.6±0.4 | 9.2±0.6 | 12.7±0.9 | 1.2±0.1 | 1.5±0.2 | 2.0±0.2
MIR (Aljundi et al., 2019a) | 9.7±0.3 | 11.2±0.4 | 13.0±0.7 | 6.4±0.4 | 9.1±0.5 | 13.9±0.6 | 1.4±0.1 | 1.9±0.2 | 2.9±0.3
GDumb (Prabhu et al., 2020) | 8.2±0.2 | 11.0±0.4 | 15.3±0.3 | 7.9±0.3 | 11.2±0.4 | 15.5±0.5 | 4.6±0.3 | 6.6±0.2 | 10.0±0.3
ASER (Shim et al., 2021) | 11.0±0.3 | 13.5±0.3 | 17.6±0.4 | 8.7±0.5 | 11.6±0.1 | 17.1±0.9 | 2.2±0.1 | 4.2±0.6 | 8.4±0.7
SCR (Mai et al., 2021) | 19.3±0.6 | 26.5±0.5 | 32.7±0.3 | 11.7±0.3 | 13.8±0.4 | 15.2±0.4 | 8.9±0.3 | 14.7±0.3 | 19.5±0.3
CoPE (De Lange & Tuytelaars, 2021) | 11.6±0.7 | 14.6±1.3 | 16.8±0.9 | 8.5±0.6 | 11.5±0.3 | 15.5±0.6 | 2.1±0.3 | 2.3±0.4 | 2.5±0.3
DVC (Gu et al., 2022) | 15.4±0.7 | 20.3±1.0 | 25.2±1.6 | 13.3±0.9 | 15.9±0.9 | 17.6±1.6 | 4.9±0.6 | 7.5±0.5 | 10.9±1.1
OCM (Guo et al., 2022b) | 19.7±0.5 | 27.4±0.3 | 34.4±0.5 | 22.2±0.4 | 30.1±0.4 | 38.2±0.3 | 10.8±0.4 | 15.4±0.4 | 20.9±0.7
OnPro (Wei et al., 2023) | 22.7±0.7 | 30.0±0.4 | 35.9±0.6 | 22.3±0.7 | 29.7±0.9 | 36.9±0.5 | 11.9±0.3 | 16.9±0.4 | 22.1±0.4
FOCIL ($M=0$) | 45.1 ± 0.6 | 58.8 ± 0.8 | 57.9 ± 1.3

#### Compared Baselines.

We use 11 recent online CIL baselines: AGEM (Chaudhry et al., 2019a), GSS
(Aljundi et al., 2019b), ER (Chaudhry et al., 2019b), MIR (Aljundi et al.,
2019a), GDumb (Prabhu et al., 2020), ASER (Shim et al., 2021), SCR (Mai et
al., 2021), CoPE (De Lange & Tuytelaars, 2021), DVC (Gu et al., 2022), OCM
(Guo et al., 2022b), and OnPro (Wei et al., 2023).

#### Evaluation Metrics.

We use Average Accuracy and Average Forgetting (Shim et al., 2021; Gu et al.,
2022) to measure the performance in online CIL.

Average Accuracy is measured by learning from the data stream of all tasks,
and then testing the final model using the test data of all tasks. It can
formally defined as: $\frac{1}{T}\sum_{j=1}^{T}a_{T,j}$ where $a_{i,j}$ is the
accuracy on task $j$ after the model is trained from task 1 to $i$.

Average Forgetting measures how much the model forgets after being trained on
the final task and can be defined as: $\frac{1}{T-1}\sum_{j=1}^{T-1}f_{T,j}$
where $f_{i,j}=\max_{k\in(1,2,...,i-1)}a_{k,j}-a_{i,j}$.

#### Implementation Details.

We train from scratch with a batch size of 10, following (Shim et al., 2021;
Guo et al., 2022b; Wei et al., 2023), and SGD optimizer with weight decay of
$5\times 10^{-4}$. We use Random Search with $20$ trials to find $\eta$ and
$s$ per task. The search space for $\eta$ is $[0.05,0.3]$ with a step size of
$0.05$, and it is $[0.90,0.98]$ with a step size of $0.02$ for $s$. We did not
search for an optimal $s$ in 100-Task TinyImageNet since the number of tasks
is large enough to set it to the highest $s$ in our search space, which is
$0.98$. We report the average of ten runs for the experiments. For a fair
comparison, we use the same backbone and batch size on the baselines. Please
refer to Section A.5 for details.

Figure 2: Average accuracy after each learning task on 20-Task CIFAR100. Our
memory-free approach FOCIL consistently outperforms all baselines that store
replay buffers by a large margin.

###  4.2 Results and Analysis

#### Performance of average accuracy.

In Table 1, the comprehensive evaluation of average incremental accuracy
scores after learning all tasks is presented across three benchmarks,
considering various memory bank sizes (M), and comparing the performance of
baseline models to our memory-free approach, FOCIL. The results consistently
demonstrate the superior performance of FOCIL over the baseline methods across
all benchmarks. Particularly, substantial performance improvement is observed
in FOCIL, especially when the number of tasks increases which is a critical
attribute in any continual learning scenario.

Table 2: Average forgetting [%] (lower is better) of our memory-free (M=0)
approach FOCIL compared to baselines with different memory sizes M on
different benchmarks. We report the mean and standard deviation across ten
independent runs for all experiments conducted on FOCIL and baselines in the
20-Task CIFAR100. The average forgetting of the baselines for 10-Task CIFAR100
and 100-Task TinyImageNet are reported performances from (Wei et al., 2023).
The best results are highlighted in bold.

Method | 10-Task CIFAR100 | 20-Task CIFAR100 | 100-Task TinyImageNet
---|---|---|---
M=0.5k | M=1k | M=2k | M=0.5k | M=1k | M=2k | M=1k | M=2k | M=4k
AGEM (Chaudhry et al., 2019a) | 41.7±0.8 | 41.8±0.7 | 41.7±0.6 | 55.9±0.9 | 56.6±0.8 | 56.3±0.9 | 73.9±0.7 | 73.1±0.7 | 72.9±0.5
GSS (Aljundi et al., 2019b) | 48.7±0.8 | 46.7±1.3 | 44.7±1.1 | 56.4±0.8 | 52.4±1.2 | 49.2±2.5 | 78.9±0.7 | 77.0±0.5 | 75.2±0.7
ER (Chaudhry et al., 2019b) | 47.0±1.0 | 46.4±0.8 | 44.7±1.5 | 54.7±1.1 | 50.3±1.1 | 46.1±1.1 | 79.1±0.6 | 77.7±0.6 | 76.3±0.5
MIR (Aljundi et al., 2019a) | 45.7±0.9 | 44.2±1.3 | 42.3±1.0 | 52.8±1.1 | 48.4±1.6 | 42.1±1.2 | 75.3±0.9 | 71.5±1.0 | 66.8±0.8
GDumb (Prabhu et al., 2020) | 25.0±0.4 | 23.2±0.4 | 20.7±0.3 | 13.6±0.5 | 15.1±0.4 | 16.8±0.8 | 22.7±0.3 | 18.4±0.2 | 17.0±0.2
ASER (Shim et al., 2021) | 52.8±0.8 | 50.4±0.9 | 46.8±0.7 | 56.1±0.3 | 52.1±0.5 | 42.9±1.1 | 78.9±0.5 | 75.4±0.7 | 68.2±1.1
SCR (Mai et al., 2021) | 29.3±0.9 | 20.4±0.6 | 11.5±0.6 | 17.6±0.7 | 16.8±0.8 | 16.5±0.9 | 44.8±0.6 | 26.8±0.5 | 20.1±0.4
CoPE (De Lange & Tuytelaars, 2021) | 25.6±0.9 | 17.8±1.3 | 14.4±0.8 | 63.8±0.7 | 58.8±0.6 | 52.1±0.4 | 11.9±0.6 | 10.9±0.4 | 9.7±0.4
DVC (Gu et al., 2022) | 32.0±0.9 | 32.7±2.0 | 28.0±2.2 | 42.2±0.8 | 40.1±1.1 | 39.2±0.9 | 59.8±2.2 | 52.9±1.3 | 45.1±1.9
OCM (Guo et al., 2022b) | 18.3±0.9 | 15.2±1.0 | 10.8±0.6 | 27.2±0.5 | 20.4±1.1 | 10.9±0.6 | 23.6±0.5 | 26.2±0.5 | 23.8±1.0
OnPro (Wei et al., 2023) | 15.0±0.8 | 10.4±0.5 | 6.1±0.6 | 25.3±1.5 | 20.4±0.9 | 14.2±0.6 | 21.3±0.5 | 17.4±0.4 | 16.8±0.4
FOCIL ($M=0$) | 0.3 ± 0.3 | 0.5 ± 0.3 | 0.6 ± 0.4

For instance, in 100-Task TinyImageNet, FOCIL remarkably exhibits almost 4
times improvement compared to the top performer baseline OnPro, in terms of
average accuracy. These results signify the scalability and efficacy of FOCIL,
positioning it as a top-performing solution in all scenarios.

#### Performance after each task.

In Figure 2, the average accuracy performance is presented, showcasing the
accuracy over all seen tasks at each incremental step within the context of
20-Task CIFAR100, with a buffer size set to 500 (i.e. M=0.5k) for the baseline
models. Notably, FOCIL consistently outperforms the baseline models,
underscoring its seamless ability to learn each incoming task. In contrast,
the baseline models fall short of achieving a comparable level of continual
learning. The observed performance of FOCIL highlights its unique capability
to learn with agility.

The allocation and utilization of different experts for each task allows for
the acquisition of more representative and discriminative features, even in
the absence of a memory buffer. This emphasizes the robustness and
effectiveness of our memory-free approach in handling class incremental
learning strategies that demand scalable and efficient solutions. This further
establishes FOCIL as a promising solution for real-world applications. For the
other benchmarks, please refer to Section A.1.

#### Performance of average forgetting.

In Table 2, the average forgetting rates for both our FOCIL model and all
baseline approaches are reported across three distinct benchmarks. Notably,
FOCIL consistently exhibits an almost zero forgetting rate across all
evaluated scenarios. That means that FOCIL does not forget any of the
previously learned experiences.

Figure 3: Selected learning rate and sparsity levels for each task, on 20-Task
CIFAR100. Our approach FOCIL picks them diversely which indicates the
necessity of adaptivity in CIL.

This remarkable performance is attributed to the allocation of different
experts for each task. In the training, these experts undergo random pruning,
training, and subsequent freezing, except batch normalization layers, which
cannot be task-specifically assigned and are therefore left unfrozen.

Our task-specific subnetwork allocation strategy ensures that knowledge gained
from previous tasks is retained. FOCIL stands in stark contrast to baseline
approaches and highlights its incomparable ability to overcome the challenges
associated with catastrophic forgetting. The results affirm the practicality
and efficacy of FOCIL in continual learning scenarios, where maintaining
knowledge across diverse tasks is of paramount importance.

#### Selected learning rates and sparsity levels.

Figure 3 illustrates the chosen learning rates ($\eta$) and sparsity levels
($s$) tailored for each task within the context of 20-Task CIFAR100.
Specifically, our proposed approach FOCIL exhibits a diverse selection of
learning rates and sparsity levels across tasks, underscoring the crucial
importance of adaptivity in the realm of continual learning.

The varied choices made by FOCIL in terms of learning rates and sparsity
levels emphasize the nuanced nature of the learning process when confronted
with a multitude of tasks. The adaptability in our approach signifies its
ability to dynamically adjust hyperparameters based on the specific
requirements of each task, thereby enhancing performance. This adaptivity is
particularly essential in class incremental learning scenarios, where the
model must continually assimilate new information without compromising
previously acquired knowledge. For the selected values on other benchmarks,
please refer to Section A.2.

#### Backbone capacity.

Although the backbone’s capacity might be a concern, it is important to note
that our approach effectively addresses this. Table 3 breaks down how FOCIL
uses the backbone capacity, showing that there is still plenty of room to take
on new tasks. Our adaptive method stands out here by dynamically picking the
sparsity levels based on the data’s complexity.

Table 3: Used backbone capacities in different benchmarks. FOCIL strategically harnesses the backbone by allowing shared connections within tasks. This not only optimizes the utilization of the backbone but also enables knowledge transfer between tasks. Dataset |  Used Capacity [%]
---|---
10-Task CIFAR100 |  45 ± 6
20-Task CIFAR100 |  59 ± 8
100-Task TinyImageNet |  74 ± 1

For example, on 10-Task CIFAR100, the model uses only 45% of its capacity and
goes up to 59% in the 20-Task CIFAR100. This scaling shows the efficiency of
FOCIL, in terms of handling a large number of tasks without a direct and
proportional increase in backbone usage.

Furthermore, in the 100-task TinyImageNet scenario, with a fixed sparsity
level of 98%, the overall capacity reaches only 74% after training 100 tasks.
However, in a conventional backbone utilization, it would reach its full
capacity after 50 tasks. This indicates that FOCIL is robust and well-suited
for an expanding range of tasks.

These findings underscore the efficiency of our approach by allowing
connection sharing within tasks. It does not only enable knowledge transfer
from previous tasks but also retains 55%, 41%, and 26% of the backbone
capacity in 10-Task CIFAR100, 20-Task CIFAR100, and 100-Task TinyImageNet
respectively.

#### Training time.

FOCIL stands out for its training time as well. Unlike the baselines, it does
not store previous data, making the learning process faster. By training a
randomly pruned subnetwork for each task, we have managed to speed up the
training while maintaining high accuracy. This is especially crucial in online
learning environments where being responsive to new information is essential.
FOCIL not only excels in performance but also offers a practical solution for
scenarios where timely updates matter.

For example, our proposed method trains 20-task CIFAR100 approximately in 8
minutes using just a single NVIDIA A100 GPU. This remarkable speed advantage
positions FOCIL as a standout solution by significantly outpacing its closest
competitors which require around 20 to 25 minutes for the same benchmark as
seen in Figure 4.

Figure 4: Training time and average accuracy on the 20-Task CIFAR100. Our
FOCIL stands out by its remarkable average accuracy while completing the
training in approximately 8 minutes when executed on a single NVIDIA A100 GPU.
In contrast, the top baselines necessitate around 20 to 25 minutes to
accomplish while returning lower average accuracy on the same benchmark.

###  4.3 Ablation Study

In Table 4, we systematically ablate the components of FOCIL, shedding light
on the impact of key elements such as the searcher, the search space, and the
number of trials. Please refer to Section A.3 for the batch size ablation to
identify an expert for a streaming task.

Despite the Tree Parzen Estimator (TPE) (Bergstra et al., 2011) with 20 trials
emerging as a slightly better choice, we acknowledge the necessity of a search
strategy that is not only highly effective, but also responsive to the demands
of online CIL; being fast, easily parallelizable, and cost-efficient.
Therefore, to meet those demands, we opt for RS (Bergstra & Bengio, 2012) with
20 trials, which secures highly competitive results and aligns seamlessly with
the pragmatic requirements of the online setting.

Table 4: Ablation Study of FOCIL on 20-Task CIFAR100 benchmark. We compare
Average (Avg) Accuracy and Average Forgetting for adaptivity components such
as the number of trials, search space, searchers of TPE, and RS as well as
FOCIL without HPO.

Method | 20-Task CIFAR100
---|---
Avg Accuracy [%] | Avg Forgetting [%]
FOCIL w/o HPO | 54.5±0.3 | 0.4±0.2
FOCIL w/ TPE (10-trial) | 58.1±1.3 | 0.4±0.2
FOCIL w/ TPE (20-trial) | 59.1±0.2 | 0.3±0.1
FOCIL w/ RS (5-trial, only $\eta$) | 57.5±0.3 | 0.3±0.2
FOCIL w/ RS (5-trial, only $s$) | 57.9±0.4 | 0.4±0.3
FOCIL w/ RS (10-trial) | 55.1±1.6 | 0.5±0.4
FOCIL w/ RS (20-trial, ours) | 58.8±0.8 | 0.5±0.3

Experiments with partial search space, focusing solely on learning rate or
sparsity level, still yield performance improvements. However, the combined
optimization of both parameters proves to be more promising, emphasizing the
impact of tuning these elements together.

Figure 5: Average accuracy after each task on 20-Task CIFAR100 with different
searchers and the number of trials.

To measure the effect of adaptivity, we run FOCIL without hyperparameter
optimization (HPO). Here, we set the learning rate to $0.1$ and the sparsity
level to 95%. Remarkably, FOCIL demonstrates significant potential, achieving
a noteworthy 54.5% average accuracy after learning all tasks even without the
adaptivity component. Although this figure falls short by 4.4% compared to
optimized parameters, it’s crucial to highlight that it still outperforms all
existing baselines by a large margin, demonstrating its inherent strength.

This exploration of FOCIL’s components and optimization strategies not only
reinforces its competitive edge, but also provides valuable insights for
tailoring the algorithm to diverse requirements. The results underscore the
versatility of FOCIL, further solidifying its position as a leading solution
in the landscape of online class incremental learning.

Figure 5 offers additional insight into the impact of hyperparameter tuning at
each task in the FOCIL approach. What becomes apparent is the substantial
assistance provided by hyperparameter optimization not only in achieving
higher final average accuracy but also in enhancing performance on every task.

This observation underscores the positive influence of HPO throughout the
learning process, indicating a consistent and beneficial effect on FOCIL’s
ability to adapt and improve in each incremental phase.

##  5 Conclusion

In this paper, we propose Finetune-and-Freeze for Online Class Incremental
Learning by Training Randomly Pruned Sparse Experts (FOCIL) which improves the
top baseline almost 4 times in terms of average accuracy while returning
nearly zero forgetting. Besides, it trains almost 3 times faster. These
achievements signify a substantial leap forward in the landscape of online
class incremental learning.

In contrast to existing methods, FOCIL innovatively opts for training a
network from scratch without the reliance on storing replay data in online
class incremental learning. It creates task-specific sparse experts by
training a randomly pruned subnetwork for each task, allowing fine-tuning of
the backbone continually while adaptively selecting the learning rate and
sparsity level per task, ensuring dynamic adjustments to enhance the
performance. Subsequently, the trained connections are frozen in post-training
of each task to safeguard against forgetting.

Beyond its immediate application in online scenarios, the versatility of FOCIL
suggests potential adaptability to many continual learning scenarios. Our
future research will delve into exploring more effective ways for expert
identification of a given data or task. In addition, extending FOCIL to an
offline setting as well as to other continual learning scenarios such as
domain incremental learning would be promising further advancements in the
field. Overall, the success of FOCIL sets a new standard in online class
incremental learning and opens avenues for refining incremental learning
methodologies, with the potential to reshape approaches in both online and
offline continual learning scenarios.

## Broader Impact

This paper presents work whose goal is to advance the field of Machine
Learning, especially on the subject of Online Class Incremental Learning.
Besides the advancements in the field, it eliminates the need to store data,
thereby improving privacy.

## References

  * Aljundi et al. (2017) Aljundi, R., Chakravarty, P., and Tuytelaars, T.  Expert gate: Lifelong learning with a network of experts.  In _CVPR_ , 2017.
  * Aljundi et al. (2018) Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., and Tuytelaars, T.  Memory aware synapses: Learning what (not) to forget.  In _ECCV_ , 2018.
  * Aljundi et al. (2019a) Aljundi, R., Belilovsky, E., Tuytelaars, T., et al.  Online continual learning with maximal interfered retrieval.  In _NeurIPS_ , 2019a.
  * Aljundi et al. (2019b) Aljundi, R., Lin, M., Goujaud, B., and Bengio, Y.  Gradient based sample selection for online continual learning.  In _NeurIPS_ , 2019b.
  * Arani et al. (2022) Arani, E., Sarfraz, F., and Zonooz, B.  Learning fast, learning slow: A general continual learning method based on complementary learning system.  In _ICLR_ , 2022.
  * Babadi & Sompolinsky (2014) Babadi, B. and Sompolinsky, H.  Sparseness and expansion in sensory representations.  _Neuron_ , 83(5), 2014.
  * Bang et al. (2021) Bang, J., Kim, H., Yoo, Y., Ha, J.-W., and Choi, J.  Rainbow memory: Continual learning with a memory of diverse samples.  In _CVPR_ , 2021.
  * Bergstra & Bengio (2012) Bergstra, J. and Bengio, Y.  Random search for hyper-parameter optimization.  _Journal of machine learning research_ , 13(2), 2012.
  * Bergstra et al. (2011) Bergstra, J., Bardenet, R., Bengio, Y., and Kégl, B.  Algorithms for hyper-parameter optimization.  In _NeurIPS_ , 2011.
  * Chaudhry et al. (2018) Chaudhry, A., Dokania, P. K., Ajanthan, T., and Torr, P. H. S.  Riemannian walk for incremental learning: Understanding forgetting and intransigence.  In _ECCV_ , 2018.
  * Chaudhry et al. (2019a) Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M.  Efficient lifelong learning with a-gem.  In _ICLR_ , 2019a.
  * Chaudhry et al. (2019b) Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P. K., Torr, P. H., and Ranzato, M.  On tiny episodic memories in continual learning.  _arXiv:1902.10486_ , 2019b.
  * De Lange & Tuytelaars (2021) De Lange, M. and Tuytelaars, T.  Continual prototype evolution: Learning online from non-stationary data streams.  In _ICCV_ , 2021.
  * Dekhovich et al. (2023) Dekhovich, A., Tax, D. M., Sluiter, M. H., and Bessa, M. A.  Continual prune-and-select: class-incremental learning with specialized subnetworks.  _Applied Intelligence_ , 2023.
  * Douillard et al. (2022) Douillard, A., Ramé, A., Couairon, G., and Cord, M.  Dytox: Transformers for continual learning with dynamic token expansion.  In _CVPR_ , 2022.
  * Evci et al. (2020) Evci, U., Gale, T., Menick, J., Castro, P. S., and Elsen, E.  Rigging the lottery: Making all tickets winners.  In _ICML_ , 2020.
  * Frankle & Carbin (2019) Frankle, J. and Carbin, M.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  In _ICLR_ , 2019.
  * Gadhikar et al. (2023) Gadhikar, A. H., Mukherjee, S., and Burkholz, R.  Why random pruning is all we need to start sparse.  In _ICML_ , 2023.
  * Gale et al. (2019) Gale, T., Elsen, E., and Hooker, S.  The state of sparsity in deep neural networks.  _arXiv:1902.09574_ , 2019.
  * Golkar et al. (2019) Golkar, S., Kagan, M., and Cho, K.  Continual learning via neural pruning.  In _NeurIPS_ , 2019.
  * Gu et al. (2022) Gu, Y., Yang, X., Wei, K., and Deng, C.  Not just selection, but exploration: Online class-incremental continual learning via dual view consistency.  In _CVPR_ , 2022.
  * Guo et al. (2022a) Guo, Y., Hu, W., Zhao, D., and Liu, B.  Adaptive orthogonal projection for batch and online continual learning.  In _AAAI_ , 2022a.
  * Guo et al. (2022b) Guo, Y., Liu, B., and Zhao, D.  Online continual learning through mutual information maximization.  In _ICML_ , 2022b.
  * Gurbuz & Dovrolis (2022) Gurbuz, M. B. and Dovrolis, C.  Nispa: Neuro-inspired stability-plasticity adaptation for continual learning in sparse networks.  In _ICML_ , 2022.
  * He et al. (2018) He, C., Wang, R., Shan, S., and Chen, X.  Exemplar-supported generative reproduction for class incremental learning.  In _BMVC_ , 2018.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * He et al. (2017) He, Y., Zhang, X., and Sun, J.  Channel pruning for accelerating very deep neural networks.  In _ICCV_ , 2017.
  * Hou et al. (2018) Hou, S., Pan, X., Loy, C. C., Wang, Z., and Lin, D.  Lifelong learning via progressive distillation and retrospection.  In _ECCV_ , 2018.
  * Hu et al. (2019) Hu, W., Lin, Z., Liu, B., Tao, C., Tao, Z. T., Zhao, D., Ma, J., and Yan, R.  Overcoming catastrophic forgetting for continual learning via model adaptation.  In _ICLR_ , 2019.
  * Hutter et al. (2019) Hutter, F., Kotthoff, L., and Vanschoren, J.  _Automated machine learning: methods, systems, challenges_.  Springer Nature, 2019.
  * Kang et al. (2022a) Kang, H., Mina, R. J. L., et al.  Forget-free continual learning with winning subnetworks.  In _ICML_ , 2022a.
  * Kang et al. (2023) Kang, H., Yoon, J., Madjid, S. R. H., Hwang, S. J., and Yoo, C. D.  On the soft-subnetwork for few-shot class incremental learning.  In _ICLR_ , 2023.
  * Kang et al. (2022b) Kang, M., Park, J., and Han, B.  Class-incremental learning by knowledge distillation with adaptive feature consolidation.  In _CVPR_ , 2022b.
  * Kirkpatrick et al. (2017) Kirkpatrick, J. et al.  Overcoming catastrophic forgetting in neural networks.  _PNAS_ , 2017.
  * Krizhevsky et al. (2009) Krizhevsky, A., Hinton, G., et al.  Learning multiple layers of features from tiny images.  2009\.
  * Le & Yang (2015) Le, Y. and Yang, X.  Tiny imagenet visual recognition challenge.  2015\.
  * LeCun et al. (2010) LeCun, Y., Cortes, C., and Burges, C.  Mnist handwritten digit database.  _ATT Labs_ , 2, 2010.
  * Lee et al. (2020) Lee, J., Hong, H. G., Joo, D., and Kim, J.  Continual learning with extended kronecker-factored approximate curvature.  In _CVPR_ , 2020.
  * Lee et al. (2019) Lee, N., Ajanthan, T., and Torr, P.  Snip: Single-shot network pruning based on connection sensitivity.  In _ICLR_ , 2019.
  * Li et al. (2018) Li, L., Jamieson, K., Rostamizadeh, A., Gonina, E., Hardt, M., Recht, B., and Talwalkar, A.  Massively parallel hyperparameter tuning.  _arXiv:1810.05934_ , 2018.
  * Li & Hoiem (2017) Li, Z. and Hoiem, D.  Learning without forgetting.  _TPAMI_ , 2017.
  * Liu et al. (2020) Liu, S., Van der Lee, T., Yaman, A., Atashgahi, Z., Ferraro, D., Sokar, G., Pechenizkiy, M., and Mocanu, D. C.  Topological insights into sparse neural networks.  In _ECML PKDD_ , 2020.
  * Liu et al. (2022) Liu, S., Chen, T., Chen, X., Shen, L., Mocanu, D. C., Wang, Z., and Pechenizkiy, M.  The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training.  In _ICLR_ , 2022.
  * Liu et al. (2021a) Liu, Y., Schiele, B., and Sun, Q.  Adaptive aggregation networks for class-incremental learning.  In _CVPR_ , 2021a.
  * Liu et al. (2021b) Liu, Y., Sun, Q., and Sun, Q.  Rmm: Reinforced memory management for class-incremental learning.  In _NeurIPS_ , 2021b.
  * Lopez-Paz & Ranzato (2017) Lopez-Paz, D. and Ranzato, M.  Gradient episodic memory for continual learning.  In _NeurIPS_ , 2017.
  * Mai et al. (2021) Mai, Z., Li, R., Kim, H., and Sanner, S.  Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning.  In _CVPR Workshops_ , 2021.
  * Mallya & Lazebnik (2018) Mallya, A. and Lazebnik, S.  Packnet: Adding multiple tasks to a single network by iterative pruning.  In _CVPR_ , 2018.
  * Mallya et al. (2018) Mallya, A., Davis, D., and Lazebnik, S.  Piggyback: Adapting a single network to multiple tasks by learning to mask weights.  In _ECCV_ , 2018.
  * Mariet & Sra (2015) Mariet, Z. and Sra, S.  Diversity networks: Neural network compression using determinantal point processes.  _arXiv:1511.05077_ , 2015.
  * McCloskey & Cohen (1989) McCloskey, M. and Cohen, N. J.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_. Elsevier, 1989.
  * Mirzadeh et al. (2022a) Mirzadeh, S. I., Chaudhry, A., Yin, D., Hu, H., Pascanu, R., Gorur, D., and Farajtabar, M.  Wide neural networks forget less catastrophically.  In _ICML_ , 2022a.
  * Mirzadeh et al. (2022b) Mirzadeh, S. I., Chaudhry, A., Yin, D., Nguyen, T., Pascanu, R., Gorur, D., and Farajtabar, M.  Architecture matters in continual learning.  _arXiv:2202.00275_ , 2022b.
  * Mocanu et al. (2016a) Mocanu, D. C., Mocanu, E., Nguyen, P. H., Gibescu, M., and Liotta, A.  A topological insight into restricted boltzmann machines.  _Machine Learning_ , 104:243–270, 2016a.
  * Mocanu et al. (2016b) Mocanu, D. C., Vega, M. T., et al.  Online contrastive divergence with generative replay: Experience replay without storing data.  _arXiv:1610.05555_ , 2016b.
  * Mocanu et al. (2018) Mocanu, D. C. et al.  Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science.  _Nature communications_ , 2018.
  * Petit et al. (2023) Petit, G., Popescu, A., Schindler, H., Picard, D., and Delezoide, B.  Fetril: Feature translation for exemplar-free class-incremental learning.  In _WACV_ , 2023.
  * Pham et al. (2021) Pham, Q., Liu, C., and Hoi, S.  Dualnet: Continual learning, fast and slow.  In _NeurIPS_ , 2021.
  * Prabhu et al. (2020) Prabhu, A., Torr, P. H., and Dokania, P. K.  Gdumb: A simple approach that questions our progress in continual learning.  In _ECCV_ , 2020.
  * Rajasegaran et al. (2020) Rajasegaran, J., Khan, S., Hayat, M., Khan, F. S., and Shah, M.  itaml: An incremental task-agnostic meta-learning approach.  In _CVPR_ , 2020.
  * Rebuffi et al. (2017) Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H.  icarl: Incremental classifier and representation learning.  In _CVPR_ , 2017.
  * Sarfraz et al. (2023) Sarfraz, F., Arani, E., and Zonooz, B.  Error sensitivity modulation based experience replay: Mitigating abrupt representation drift in continual learning.  In _ICLR_ , 2023.
  * Shim et al. (2021) Shim, D., Mai, Z., Jeong, J., Sanner, S., Kim, H., and Jang, J.  Online class-incremental continual learning with adversarial shapley value.  In _AAAI_ , 2021.
  * Shin et al. (2017) Shin, H., Lee, J. K., Kim, J., and Kim, J.  Continual learning with deep generative replay.  In _NeurIPS_ , 2017.
  * Sokar et al. (2021) Sokar, G., Mocanu, D. C., and Pechenizkiy, M.  Spacenet: Make free space for continual learning.  _Neurocomputing_ , 2021.
  * Sokar et al. (2023) Sokar, G., Mocanu, D. C., and Pechenizkiy, M.  Avoiding forgetting and allowing forward transfer in continual learning via sparse networks.  In _ECML PKDD_ , 2023.
  * Tanaka et al. (2020) Tanaka, H., Kunin, D., Yamins, D. L., and Ganguli, S.  Pruning neural networks without any data by iteratively conserving synaptic flow.  In _NeurIPS_ , 2020.
  * Van de Ven & Tolias (2019) Van de Ven, G. M. and Tolias, A. S.  Three scenarios for continual learning.  _arXiv:1904.07734_ , 2019.
  * Wang et al. (2020) Wang, C., Zhang, G., and Grosse, R.  Picking winning tickets before training by preserving gradient flow.  In _ICLR_ , 2020.
  * Wang et al. (2022a) Wang, F.-Y., Zhou, D.-W., Ye, H.-J., and Zhan, D.-C.  Foster: Feature boosting and compression for class-incremental learning.  In _ECCV_ , 2022a.
  * Wang et al. (2023) Wang, W., Hu, Y., Chen, Q., and Zhang, Y.  Task difficulty aware parameter allocation & regularization for lifelong learning.  In _CVPR_ , 2023.
  * Wang et al. (2022b) Wang, Z., Zhan, Z., Gong, Y., Yuan, G., Niu, W., Jian, T., Ren, B., Ioannidis, S., Wang, Y., and Dy, J.  Sparcl: Sparse continual learning on the edge.  In _NeurIPS_ , 2022b.
  * Wei et al. (2023) Wei, Y., Ye, J., Huang, Z., Zhang, J., and Shan, H.  Online prototype learning for online continual learning.  In _ICCV_ , 2023.
  * Wortsman et al. (2020) Wortsman, M., Ramanujan, V., Liu, R., Kembhavi, A., Rastegari, M., Yosinski, J., and Farhadi, A.  Supermasks in superposition.  In _NeurIPS_ , 2020.
  * Wu et al. (2019) Wu, Y., Chen, Y., Wang, L., Ye, Y., Liu, Z., Guo, Y., and Fu, Y.  Large scale incremental learning.  In _CVPR_ , 2019.
  * Xiao et al. (2017) Xiao, H., Rasul, K., and Vollgraf, R.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
  * Yan et al. (2021) Yan, S., Xie, J., and He, X.  Der: Dynamically expandable representation for class incremental learning.  In _CVPR_ , 2021.
  * Yildirim et al. (2023) Yildirim, M. O., Yildirim, E. C. G., Sokar, G., Mocanu, D. C., and Vanschoren, J.  Continual learning with dynamic sparse training: Exploring algorithms for effective model updates.  In _CPAL_ , 2023.
  * Zenke et al. (2017) Zenke, F., Poole, B., and Ganguli, S.  Continual learning through synaptic intelligence.  In _ICML_ , 2017.
  * Zhao et al. (2020) Zhao, B., Xiao, X., Gan, G., Zhang, B., and Xia, S.-T.  Maintaining discrimination and fairness in class incremental learning.  In _CVPR_ , 2020.
  * Zhou et al. (2023) Zhou, D.-W., Wang, Q.-W., Ye, H.-J., and Zhan, D.-C.  A model or 603 exemplars: Towards memory-efficient class-incremental learning.  In _ICLR_ , 2023.

##  Appendix A Appendix

###  A.1 Average Accuracy Performance After Each Task.

In this section, we present the performance results of FOCIL with and without
hyperparameter tuning on the benchmarks of 10-Task CIFAR100, 20-Task CIFAR100,
and 100-Task TinyImageNet, depicted in Figure A, Figure B, and Figure C
respectively.

Figure A: Average accuracy after each task on 10-Task CIFAR100 with and
without HPO.

Figure B: Average accuracy after each task on 20-Task CIFAR100 with and
without HPO.

Figure C: Average accuracy after each task on 100-Task TinyImageNet with and
without HPO.

###  A.2 More on Adaptivity.

In this section, we provide selected hyperparameters for learning rate $\eta$
and sparsity level $s$ per task for 10-Task CIFAR100 and 100-Task TinyImageNet
benchmarks.

In Figure D selected hyperparameters, the learning rate $\eta$ and the
sparsity level $s$ are given for 10-Task CIFAR100. We shared only the selected
learning rate $\eta$ in Figure E for 100-Task TinyImageNet since the sparsity
level $s$ was predetermined and fixed to 98% per task.

For all benchmarks, selected learning rates $\eta$ and sparsity levels $s$
fluctuate between tasks which indicates their sensitivity and need for tuning
to adapt to the tasks which in the end increases the performance.

Figure D: Selected hyperparameters for 10-Task CIFAR100.

Figure E: Selected hyperparameters for 100-Task TinyImageNet.

###  A.3 Different Batch Sizes to Identify an Expert.

In the section, we also ablate various batch sizes to identify the
corresponding experts for streaming data during the inference on Figure F. We
found that only a single observation is not feasible to identify the
corresponding expert. When we increase the batch size to 2 or 5, we spot a
huge jump in terms of average accuracy since FOCIL starts to find the related
expert reasonably better. However, after the batch size of 10, the performance
becomes stable and FOCIL selects the correct expert for all streaming data.

Figure F: Ablation of different batch sizes (bs) for expert identification
with Average Accuracy on 20-Task CIFAR100.

###  A.4 Experiments on MLP.

To show FOCIL’s versatility on different network architectures, we also
conducted experiments on a simple MLP. We share the results of 5-Task MNIST
and 5-Task Fashion-MNIST in Table A and Figure G.

Figure G: Average Accuracy after each task for 5-Task MNIST and 5-Task
Fashion-MNIST. Table A: Average Accuracy and Average Forgetting after the last
task for 5-Task MNIST and 5-Task Fashion-MNIST.

Dataset | Avg Accuracy [%] | Avg Forgetting [%]
---|---|---
5-task MNIST | 91.2 ± 2.3 | 0.1 ± 0.1
5-task Fashion-MNIST | 91.5 ± 3.4 | 0.1 ± 0.1

###  A.5 Implementation Details about Baselines.

For the baselines, we follow their original papers and default settings. We
set the random seed to 0 and run the experiments 10 times. To reproduce them,
we refer to Table B.

For AGEM and ER, we use SGD optimizer and set the learning rate to 0.1. We
uniformly randomly select samples to update the memory buffer and replay.

For GSS, we use SGD optimizer and set the learning rate to 0.1. The number of
batches randomly sampled from the memory buffer to estimate the maximal
gradients cosine similarity score is set to 64, and the random sampling batch
size for calculating the score is set to 64.

For MIR, we use SGD optimizer and set the learning rate to 0.1. The number of
subsamples is set as 100.

For GDumb, we use SGD optimizer and set the learning rate to 0.1. The value
for gradient clipping is set to 10. The minimal learning rate is set to
0.0005, and the epochs to train for the memory buffer are 70.

For ASER, we use SGD optimizer and set the learning rate to 0.1. The nearest
neighbor is set to 3. We use mean values of Adversarial SV and Cooperative SV
and set the maximum number of samples per class for random sampling to 1.5. We
use the SV-based methods for memory update and retrieval as in the original
paper.

For SCR, we use SGD optimizer and set the learning rate to 0.1. We set the
temperature to 0.07 and employed a linear layer with a hidden size of 128 as
the projection head.

For CoPE, we use SGD optimizer and set the learning rate to 0.005. We set the
temperature to 0.05. The momentum of the moving average updates for the
prototypes is set to 0.9.

For DVC, we use SGD optimizer and set the learning rate to 0.1. The number of
candidate samples for retrieval is set to 50. We set loss weights
$\lambda_{1}=\lambda_{2}=1$, and $\lambda_{3}=4$.

For OCM, we use Adam optimizer and set the learning rate to 0.001 with a
weight decay of 0.0001. We set the temperature to 0.07 and employ a linear
layer with a hidden size of 128 as the projection head. $\lambda$ is set to
0.5. We set $\alpha$ to 1 and $\beta$ to 2 for contrastive loss and set
$\alpha$ to 0 and $\beta$ to 2 for supervised contrastive loss as in the
original paper.

For OnPro, we use Adam optimizer and set the learning rate of 0.0005. The
weight decay is set as 0.0001. We employ a linear layer with a hidden size of
128 as the projection head.

Table B: Baselines with source code links. Baseline | Link
---|---
AGEM | https://github.com/RaptorMai/online-continual-learning
GSS | https://github.com/RaptorMai/online-continual-learning
ER | https://github.com/RaptorMai/online-continual-learning
MIR | https://github.com/RaptorMai/online-continual-learning
GDumb | https://github.com/RaptorMai/online-continual-learning
ASER | https://github.com/RaptorMai/online-continual-learning
SCR | https://github.com/RaptorMai/online-continual-learning
CoPE | https://github.com/Mattdl/ContinualPrototypeEvolution
DVC | https://github.com/YananGu/DVC
OCM | https://github.com/gydpku/OCM
OnPro | https://github.com/weilllllls/OnPro