# Representational Continuity for
Unsupervised Continual Learning

Divyam Madaan1 Jaehong Yoon${}^{2,3\leavevmode\nobreak\
\color[rgb]{0.6,0,0}{\dagger}}$ Yuanchun Li5,6 Yunxin Liu5,6 Sung Ju Hwang2,4
New York University1 KAIST2 Microsoft Research3 AITRICS4
Institute for AI Industry Research (AIR)5 Tsinghua University6
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding
author. † The work was done while the author was an intern at Microsoft
Research.

###### Abstract

Continual learning (CL) aims to learn a sequence of tasks without forgetting
the previously acquired knowledge. However, recent CL advances are restricted
to supervised continual learning (SCL) scenarios. Consequently, they are not
scalable to real-world applications where the data distribution is often
biased and unannotated. In this work, we focus on _unsupervised continual
learning (UCL)_ , where we learn the feature representations on an unlabelled
sequence of tasks and show that reliance on annotated data is not necessary
for continual learning. We conduct a systematic study analyzing the learned
feature representations and show that unsupervised visual representations are
surprisingly more robust to catastrophic forgetting, consistently achieve
better performance, and generalize better to out-of-distribution tasks than
SCL. Furthermore, we find that UCL achieves a smoother loss landscape through
qualitative analysis of the learned representations and learns meaningful
feature representations. Additionally, we propose Lifelong Unsupervised Mixup
(Lump), a simple yet effective technique that interpolates between the current
task and previous tasks’ instances to alleviate catastrophic forgetting for
unsupervised representations. We release our code online.

##  1 Introduction

Recently continual learning (Thrun, 1995) has gained a lot of attention in the
deep learning community due to its ability to continually learn on a sequence
of non-stationary tasks (Kumar & Daume III, 2012; Li & Hoiem, 2016) and close
proximity to the human learning process (Flesch et al., 2018). However, the
inability of the learner to prevent forgetting of the knowledge learnt from
the previous tasks has been a long-standing problem (McCloskey & Cohen, 1989;
Goodfellow et al., 2013). To address this problem, a large body of methods
(Rusu et al., 2016; Zenke et al., 2017; Yoon et al., 2018; Li et al., 2019;
Aljundi et al., 2019; Buzzega et al., 2020) have been proposed; however, all
these methods focus on the supervised learning paradigm, but obtaining high-
quality labels is expensive and often impractical in real-world scenarios. In
contrast, CL for unsupervised representation learning has received limited
attention in the community. Although Rao et al. (2019) instantiated a
continual unsupervised representation learning framework (Curl), it is not
scalable for high-resolution tasks, as it is composed of MLP encoders/decoders
and a simple MoG generative replay. This is evident in their limited empirical
evaluation using digit-based gray-scale datasets.

Meanwhile, a set of directions have shown huge potential to tackle the
representation learning problem without labels (He et al., 2020; Chen et al.,
2020a; Grill et al., 2020; Chen et al., 2020b; Chen & He, 2021; Zbontar et
al., 2021) by aligning contrastive pairs of training instances or maximizing
the similarity between two augmented views of each image. However, a common
assumption for existing methods is the availability of a large amount of
unbiased and unlabelled datasets to learn the feature representations. We
argue that this assumption is not realistic for most of the real-time
applications, including self-driving cars (Bojarski et al., 2016), medical
applications (Kelly et al., 2019) and conversational agents (Li et al., 2020).
The collected datasets are often limited in size during the initial training
phase (Finn et al., 2017), and datasets/tasks change continuously with time.

To accommodate such continuous shifts in data distributions, representation
learning models need to increment the knowledge without losing the
representations learned in the past.

|
---|---
Supervised Continual Learning (SCL) | Unsupervised Continual Learning (UCL)

Figure 1: Illustration of supervised and unsupervised continual learning. The
objective of SCL is to learn the ability to classify labeled images in the
current task while preserving the past tasks’ knowledge, where the tasks are
non-iid to each other. On the other hand, UCL aims to learn the representation
of images without the presence of labels and the model learns general-purpose
representations during sequential training.

With this motivation, we attempt to bridge the gap between unsupervised
representation learning and continual learning to address the challenge of
learning the representations on a sequence of tasks. Specifically, we focus on
_unsupervised continual learning (UCL)_ , where the goal of the continual
learner is to learn the representations from a stream of unlabelled data
instances without forgetting (see Figure 1). To this end, we extend various
existing SCL strategies to the unsupervised continual learning framework and
analyze the performance of current state-of-the-art representation learning
techniques: _SimSiam_ (Chen & He, 2021) and _BarlowTwins_ (Zbontar et al.,
2021) for UCL. Surprisingly, we observe that the unsupervised representations
are comparatively more robust to catastrophic forgetting across all datasets
and simply finetuning on the sequence of tasks can outperform various state-
of-the-art continual learning alternatives. Furthermore, we show that UCL
generalize better to various out of distribution tasks and outperforms SCL for
few-shot training scenarios (Section 5.2).

We demystify the robustness of unsupervised representations by investigating
the feature similarity, measured by centered kernel alignment (CKA) (Kornblith
et al., 2019) between two independent UCL and SCL models and between UCL and
SCL models. We notice that two unsupervised model representations have a
relatively high feature similarity compared to two supervised representations.
Furthermore, in all cases, two models have high similarity in lower layers
indicating that they learn similar low-level features. Further, we measure the
$\ell_{2}$ distance between model parameters (Neyshabur et al., 2020) and
visually compare the feature representations learned by different SCL and UCL
strategies. We observe that UCL obtains human perceptual feature patterns for
previous tasks, demonstrating their effectiveness to alleviate catastrophic
forgetting (Section 5.3). We conjecture that this is due to their
characteristic ability to learn general-purpose features (Doersch et al.,
2020), which makes them transfer better and comparatively more robust to
catastrophic forgetting.

To gain further insights, we visualize the loss landscape (Li et al., 2018) of
the UCL and SCL models and observe that UCL obtains a flatter and smoother
loss landscape than SCL. Additionally, we propose a simple yet effective
technique coined Lifelong Unsupervised Mixup (Lump), which utilizes mixup
(Zhang et al., 2018) for unlabelled training instances. In particular, Lump
interpolates between the current task examples and examples from previous
instances to minimize catastrophic forgetting. We emphasize that Lump is easy
to implement, does not require additional hyperparameters, and simply trains
on the interpolated instances. To this end, Lump requires little, or no
modification to existing rehearsal-based methods effectively minimizes
catastrophic forgetting even with uniformly selecting the examples from replay
buffer. We show that Lump with UCL outperforms the state-of-the-art supervised
continual learning methods across multiple experimental settings with
significantly lower catastrophic forgetting. In summary, our contributions are
as follows:

  * •

We attempt to bridge the gap between continual learning and representation
learning and tackle the two crucial problems of continual learning with
unlabelled data and representation learning on a sequence of tasks.

  * •

Systematic quantitative analysis shows that UCL achieves better performance
over SCL with significantly lower catastrophic forgetting on Sequential
CIFAR-10, CIFAR-100, and Tiny-ImageNet. Additionally, we evaluate out-of-
distribution tasks and few-shot training demonstrating the expressive power of
unsupervised representations.

  * •

We provide visualization of the representations and loss landscapes, which
show that UCL learns discriminative, human perceptual patterns and achieves a
flatter and smoother loss landscape. Furthermore, we propose Lifelong
Unsupervised Mixup (Lump) for UCL, which effectively alleviates catastrophic
forgetting and provides better qualitative interpretations.

##  2 Related work

Continual learning. We can partition the existing continual learning methods
into three categories. The _regularization_ approaches (Li & Hoiem, 2016;
Zenke et al., 2017; Schwarz et al., 2018; Ahn et al., 2019) impose a
regularization constraint to the objective that mitigates catastrophic
forgetting. The _architectural_ approaches (Rusu et al., 2016; Yoon et al.,
2018; Li et al., 2019) avoid this problem by including task-specific
parameters and allowing the expansion of the network during continual
learning. The  _rehearsal_ approaches (Rebuffi et al., 2017; Rolnick et al.,
2019; Aljundi et al., 2019) allocate a small memory buffer to store and replay
the examples from the previous task. However, all these methods are confined
to supervised learning, which limits their application in real-life problems.
Rao et al. (2019); Smith et al. (2021) tackled the problem of continual
unsupervised representation learning; however, their methods are restricted to
simple low-resolution tasks and not scalable to large-scale continual learning
datasets.

Representational learning. A large number of works have addressed the
unsupervised learning problem in the standard machine learning framework.
Specifically, contrastive learning frameworks (He et al., 2020; Chen et al.,
2020a; Grill et al., 2020; Chen et al., 2020b; c) that learn the
representations by measuring the similarities of positive and negative pairs
have gained a lot of attention in the community. However, all these methods
require large batches and negative sample pairs, which restrict the
scalability of these networks. Chen & He (2021) tackled these limitations and
proposed _SimSiam_ , that use standard Siamese networks (Bromley et al., 1994)
with the stop-gradient operation to prevent the collapsing of Siamese networks
to a constant. Recently, Zbontar et al. (2021) formulated an objective that
pushes the cross-correlation matrix between the embeddings of distorted
versions of a sample closer to the identity matrix. However, all these methods
assume the presence of large datasets for pre-training, which is impractical
in real-world applications. In contrast, we tackle the problem of incremental
representational learning and learn the representations sequentially while
maximizing task adaptation and minimizing catastrophic forgetting.

##  3 Preliminaries

###  3.1 Problem Setup

We consider the continual learning setting, where we learn on a continuum of
data consisting of $T$ tasks
$\mathcal{T}_{1:T}=\left(\mathcal{T}_{1}\ldots\mathcal{T}_{T}\right)$. In
supervised continual learning, each task consists a task descriptor
$\tau\in\\{1\ldots T\\}$ and its corresponding dataset
$\mathcal{D}_{\tau}=\\{\left(\boldsymbol{x}_{i,\tau},y_{i,\tau}\right)_{i=1}^{n_{\tau}}\\}$
with $n_{\tau}$ examples. Each input-pair
$(\boldsymbol{x}_{i,\tau},y_{i,\tau})\in\mathcal{X}_{\tau}\times\mathcal{Y}_{\tau}$,
where $(\mathcal{X}_{\tau},\mathcal{Y}_{\tau})$ is an unkown data
distribution. Let us consider a network
$f_{\Theta}:\mathcal{X}_{\tau}\rightarrow\mathbb{R}^{D}$ parametrized by
$\Theta=\\{\boldsymbol{w}_{l}\\}_{l=1}^{l=L}$, where $\mathbb{R}^{D}$ and $L$
denote $D$-dimensional embedding space and number of layers respectively. The
classifier is denoted by
$h_{\psi}:\mathbb{R}^{D}\rightarrow\mathcal{Y}_{\tau}$. The network error
using cross entropy loss (CE) for SCL with finetuning can be formally defined
as:

$\mathcal{L}_{\textsc{SCL}}^{\textsc{Finetune}}=\textsc{CE}\left(h_{\psi}\left(f_{\Theta}\left(\boldsymbol{x}_{i,\tau}\right),\tau\right),y_{i,\tau}\right).$
(1)

In this work, we assume the absence of label supervision during training and
focus on unsupervised continual learning. In particular, each task consists of
$\mathcal{U}_{\tau}=\\{\left(\boldsymbol{x}_{i,\tau}\right)_{i=1}^{n_{\tau}}\\}$,
$\boldsymbol{x}_{i,\tau}\in\mathcal{X}_{\tau}$ with $n_{\tau}$ examples. Our
aim is to learn the representations
$f_{\Theta}:\mathcal{X}_{\tau}\rightarrow\mathbb{R}^{D}$ on a sequence of
tasks while preserving the knowledge of the previous tasks. We introduce the
representation learning framework and propose Lump in Section 4 to learn
unsupervised representations while effectively mitigating catastrophic
forgetting.

###  3.2 Learning Protocol and Evaluation Metrics

Currently, the traditional continual learning strategies follow the standard
training protocol, where we learn the network representations
$f_{\Theta}:\mathcal{X}_{\tau}\rightarrow\mathcal{Y}_{\tau}$ on a sequence of
tasks. In contrast, our goal is to learn the feature representations
$f_{\Theta}:\mathcal{X}_{\tau}\rightarrow\mathbb{R}^{D}$, so we follow a two-
step learning protocol to obtain the model predictions. First, we pre-train
the representations on a sequence of tasks
$T_{1:T}=\left(\mathcal{T}\ldots\mathcal{T}_{T}\right)$ to obtain the
representations. Next, we evaluate the quality of our pre-trained
representations using a K-nearest neighbor (KNN) classifier (Wu et al., 2018)
following the setup in Chen et al. (2020a); Chen & He (2021); Zbontar et al.
(2021).

To validate knowledge transfer of the learned representations, we adopt the
metrics from the SCL literature (Chaudhry et al., 2019b; Mirzadeh et al.,
2020). Let $a_{\tau,i}$ denote the test accuracy of task $i$ after learning
task $\mathcal{T}_{\tau}$ using a KNN on frozen pre-trained representations on
task $\mathcal{T}_{\tau}$. More formally, we can define the metrics to
evaluate the continually learned representations as follow:

  1. 1.

Average accuracy is the average test accuracy of all the tasks completed until
the continual learning of task $\tau$:
$A_{\tau}=\frac{1}{\tau}\sum_{i=1}^{\tau}a_{\tau,i}$

  2. 2.

Average Forgetting is the average performance decrease of each task between
its maximum accuracy and accuracy at the completion of training:
$F\leavevmode\nobreak\ =\leavevmode\nobreak\
\frac{1}{T-1}\sum_{i=1}^{T-1}\max_{{\tau\in\\{1,\ldots,T\\}}}\left(a_{\tau,i}-a_{T,i}\right)$

##  4 Unsupervised Continual Learning

###  4.1 Continuous representation learning with sequential tasks

To learn feature representations, contrastive learning (Chen et al., 2020a; b;
He et al., 2020) maximizes the similarity of representations between the
images of the same views (positive pairs) and minimizes the similarity between
images of different views (negative pairs). However, these methods require
large batches, negative sample pairs (Chen et al., 2020a; b), or architectural
modifications (He et al., 2020; Chen et al., 2020c), or non-differentiable
operators (Caron et al., 2020), which makes their application difficult for
continual learning scenarios. In this work, we focus on SimSiam (Chen & He,
2021) and BarlowTwins (Zbontar et al., 2021), which tackle these limitations
and achieve state-of-the-art performance on standard representation learning
benchmarks.

SimSiam (Chen & He, 2021) uses a variant of Siamese networks (Bromley et al.,
1994) for learning input data representations. It consists of an encoder
network $f_{\Theta}$, which is composed of a backbone network, and is shared
across a projection MLP and prediction MLP head $h(\cdot)$. Specifically,
SimSiam minimizes the cosine-similarity between the output vectors of the
projector and the predictor MLP across two different augmentations for an
image. Initially, we consider Finetune, which is a a naive CL baseline that
minimizes the cosine-similarity between the projector output
$(z_{i,\tau}^{1}=f_{\Theta}(x_{i,\tau}^{1}))$ and the predictor output
$(p_{i,\tau}^{2}=h(f_{\Theta}(x_{i,\tau}^{2}))$ on a sequence of tasks as
follows:

$\displaystyle\mathcal{L}_{\textsc{UCL}}^{\textsc{Finetune}}=\frac{1}{2}D(p_{i,\tau}^{1},\texttt{stopgrad}(z_{i,\tau}^{2}))+\frac{1}{2}D($
$\displaystyle p_{i,\tau}^{2},\texttt{stopgrad}(z_{i,\tau}^{1})),$ (2)
$\displaystyle\text{where}\leavevmode\nobreak\ \leavevmode\nobreak\
D(p_{i,\tau}^{1},z_{i,\tau}^{2})$ $\displaystyle=-\frac{p_{i,\tau}^{1}}{\lVert
p_{i,\tau}^{2}\rVert_{2}}\cdot\frac{z_{i,\tau}^{2}}{\lVert
z_{i,\tau}^{2}\rVert_{2}},$

$x_{i,\tau}^{1}$ and $x_{i,\tau}^{2}$ are two randomly augmented views of an
input example $x_{i,\tau}\in\mathcal{T}_{\tau}$ and $\lVert\cdot\rVert_{2}$
denotes the $\ell_{2}$-norm. Note that, the stopgrad is a crucial component in
SimSiam to prevent the trivial solutions obtained by Siamese networks. Due to
its simplicity and effectiveness, we chose Simsiam to analyze the performance
of unsupervised representations for continual learning.

BarlowTwins (Zbontar et al., 2021) minimizes the redundancy between the
embedding vector components of the distorted versions of an instance while
conserving the maximum information inspired from Barlow (1961). In particular,
the objective function eliminates the SimSiam stopgrad component and instead
makes the cross-correlation matrix computed between the outputs of two
identical networks closer to the identity matrix. Let $\mathcal{C}$ be the
cross-correlation matrix between the outputs of two Siamese branches along the
batch dimension and $Z_{1}$ and $Z_{2}$ denote the batch embeddings of the
distorted views for all images of a batch from the current task
$(x_{\tau}\in\mathcal{U}_{\tau})$. The objective function for UCL with
finetuning and BarlowTwins can then be defined as:

$\displaystyle\mathcal{L}_{\textsc{UCL}}^{\textsc{Finetune}}=\sum_{i}(1-\mathcal{C}_{ii})^{2}+\leavevmode\nobreak\
\leavevmode\nobreak\ \lambda\cdot\sum_{i}\sum_{j\neq
i}{\mathcal{C}_{ij}}^{2},\leavevmode\nobreak\ \leavevmode\nobreak\
\text{where}\leavevmode\nobreak\ \leavevmode\nobreak\
\mathcal{C}_{ij}=\frac{\sum_{\mathcal{B}}z^{1}_{\mathcal{B},i}z^{2}_{\mathcal{B},j}}{\sqrt{\sum_{\mathcal{B}}{(z^{1}_{\mathcal{B},i})}^{2}}\sqrt{\sum_{\mathcal{B}}{(z^{2}_{\mathcal{B},j})}^{2}}}.$
(3)

$\lambda$ is a positive constant trading off the importance of the invariance
and redundancy reduction terms of the loss, $i$ and $j$ denote the network’s
output vector dimensions. Similar to SimSiam, BarlowTwins is simple, easy to
implement, and can be applied to existing continual learning strategies with
little or no modification.

###  4.2 Preserving representational continuity: A view of existing SCL
methods

Learning feature representations from labelled instances on a sequence of
tasks has been long studied in continual learning. However, the majority of
these learning strategies are not directly applicable to UCL. To compare with
the regularization-based strategies, we extend _Synaptic Intelligence (SI)_
(Zenke et al., 2017) to UCL and consider the online per-synapse consolidation
during the entire training trajectory of the unsupervised representations. For
architectural-based strategies, we investigate _Progressive Neural Networks
(PNN)_ (Rusu et al., 2016) and learn the feature representations progressively
using the representations learning frameworks proposed in Section 4.1.

We also formulate _Dark Experience Replay (DER)_ (Buzzega et al., 2020) for
UCL. DER for SCL alleviates catastrophic forgetting by matching the network
logits across a sequence of tasks during the optimization trajectory. Notably,
the loss for SCL-DER can be defined as follow:

$\mathcal{L}_{\textsc{SCL}}^{\textsc{Der}}=\mathcal{L}_{\textsc{SCL}}^{\textsc{Finetune}}+\leavevmode\nobreak\
\alpha\cdot\mathbb{E}_{(x,p)\sim\mathcal{M}}\big{[}\left\lVert\mathrm{softmax}(p)-\mathrm{softmax}(h_{\psi}(x_{i,\tau}))\right\rVert^{2}_{2}\big{]},$
(4)

where $p=h_{{\psi}_{\tau}(x)}$,
$\mathcal{L}_{\textsc{SCL}}^{\textsc{Finetune}}$ denotes the cross-entropy
loss on the current task (see Equation 1) and random examples are selected
using reservoir sampling from the replay-buffer $\mathcal{M}$. Since, we do
not have access to the labels for UCL, we cannot minimize the aforementioned
objective.

Instead, we utilize the output of the projected output by the backbone network
to preserve the knowledge of the past tasks over the entire training
trajectory. In particular, DER for UCL consists of a combination of two terms.
The first term learns the representations using SimSiam from Equation 2 or
BarlowTwins from Equation 3 and the second term minimizes the Euclidean
distance between the projected outputs to minimize catastrophic forgetting.
More formally, UCL-DER minimizes the following loss:

$\mathcal{L}_{\textsc{UCL}}^{\textsc{Der}}=\mathcal{L}_{\textsc{UCL}}^{\textsc{Finetune}}+\leavevmode\nobreak\
\alpha\cdot\mathbb{E}_{(x)\sim\mathcal{M}}\big{[}\left\lVert
f_{\Theta_{\tau}}(x)-f_{\Theta}(x_{i,\tau})\right\rVert^{2}_{2}\big{]}$ (5)

However, the performance of the rehearsal-based methods is sensitive to the
choice of $\alpha$ and often requires supervised training setup, task
identities, and boundaries. To tackle this issue, we propose Lifelong
Unsupervised Mixup in the subsequent subsection, which interpolates between
the current and past task instances to mitigate catastrophic forgetting
effectively.

###  4.3 Lifelong unsupervised mixup

The standard Mixup (Zhang et al., 2018) training constructs virtual training
examples based on the principle of Vicinal Risk Minimization . In particular,
let $(x_{i},y_{i})$ and $(x_{j},y_{j})$ denote two random feature-target pairs
sampled from the training data distribution and let $(\tilde{x},\tilde{y})$
denote the interpolated feature-target pair in the vicinity of these examples;
mixup then minimizes the following objective:

$\displaystyle\mathcal{L}^{\textsc{Mixup}}(\tilde{x},\tilde{y})$
$\displaystyle=\textsc{CE}\left(h_{\psi}\left(f_{\Theta}\left(\tilde{x}\right)\right),\tilde{y}\right),$
(6) $\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\
\text{where}\leavevmode\nobreak\ \leavevmode\nobreak\ \tilde{x}$
$\displaystyle=\lambda\cdot x_{i}+(1-\lambda)\cdot x_{j}\leavevmode\nobreak\
\leavevmode\nobreak\ \text{and}\leavevmode\nobreak\ \leavevmode\nobreak\
\tilde{y}=\lambda\cdot y_{i}+(1-\lambda)\cdot y_{j}.$

$\lambda\sim\text{Beta}(\alpha,\alpha)$, for $\alpha\in(0,\infty)$. In this
work, we focus on lifelong self-supervised learning and propose Lifelong
Unsupervised Mixup (Lump) that utilizes mixup for UCL by incorporating the
instances stored in the replay-buffer from the previous tasks into the vicinal
distribution. In particular, Lump interpolates between the examples of the
current task ($x_{i,\tau})\in\mathcal{U}_{\tau}$ and random examples selected
using uniform sampling from the replay buffer, which encourages the model to
behave linearly across a sequence of tasks. More formally, Lump minimizes the
objective in Equation 2 and Equation 3 on the following interpolated instances
$\tilde{x}_{i,\tau}$ for the current task $\tau$:

$\tilde{x}_{i,\tau}=\lambda\cdot x_{i,\tau}+(1-\lambda)\cdot
x_{j,\mathcal{M}},$ (7)

where $x_{j,\mathcal{M}}\sim\mathcal{M}$ denotes the example selected using
uniform sampling from replay buffer $\mathcal{M}$. The interpolated examples
not only augments the past tasks’ instances in the replay buffer but also
approximates a regularized loss minimization (Zhang et al., 2021). During UCL,
Lump enhances the robustness of learned representation by revisiting the
attributes of the past task that are similar to the current task. Recently,
Kim et al. (2020); Lee et al. (2021); Verma et al. (2021); Shen et al. (2022)
also employed mixup for contrastive learning. Our work is different from these
existing works in that our objective is different, and we focus on
unsupervised continual learning. To this end, Lump successively mitigates
catastrophic forgetting and learns discriminative & human-perceptual features
over the current state-of-the-art SCL strategies (see Table 1 and Figure 4).

##  5 Experiments

###  5.1 Experimental setup

Baselines. We compare with multiple supervised and unsupervised continual
learning baselines across different categories of continual learning methods.

  1. 1.

Supervised continual learning. Finetune is a vanilla supervised learning
method trained on a sequence of tasks without regularization or episodic
memory and Multitask optimizes the model on complete data. For regularization-
based CL methods, we compare against SI (Zenke et al., 2017) and AGEM
(Chaudhry et al., 2019a). We include PNN (Rusu et al., 2016) for architecture-
based methods. Lastly, we consider GSS (Aljundi et al., 2019) that populates
the replay-buffer using solid-angle minimization and DER (Buzzega et al.,
2020) matches the network logits sampled through the optimization trajectory
for rehearsal during continual learning.

  2. 2.

Unsupervised continual learning. We consider the unsupervised variants of
various SCL baselines to show the utility of the unsupervised representations
for sequential learning. Specifically, we use SimSiam (Chen & He, 2021) and
BarlowTwins (Zbontar et al., 2021), which are the state-of-the-art
representational learning techniques for learning the unsupervised continual
representations. We compare with Finetune and Multitask following the
supervised learning baselines, and Si (Zenke et al., 2017), Pnn (Rusu et al.,
2016) for unsupervised regularization and architecture CL methods
respectively. For rehearsal-based method, we compare with the UCL variant of
Der (Buzzega et al., 2020) described in Section 4.2

Datasets. We compare the performance of SCL and UCL on various continual
learning benchmarks using single-head ResNet-18 (He et al., 2016)
architecture. Split CIFAR-10 (Krizhevsky, 2012) consists of two random classes
out of the ten classes for each task. Split CIFAR-100 (Krizhevsky, 2012)
consists of five random classes out of the 100 classes for each task. Split
Tiny-ImageNet is a variant of the ImageNet dataset (Deng et al., 2009)
containing five random classes out of the 100 classes for each task with the
images sized 64 $\times$ 64 pixels.

Training and evaluation setup. We follow the hyperparameter setup of Buzzega
et al. (2020) for all the SCL strategies and tune them for the UCL
representation learning strategies. All the learned representations are
evaluated with KNN classifier (Wu et al., 2018) across three independent runs.
Further, we use the hyper-parameters obtained by SimSiam for training UCL
strategies with BarlowTwins to analyze the sensitivity of UCL to hyper-
parameters and for a fair comparison between different methods. We train all
the UCL methods for 200 epochs and evaluate with the KNN classifier (Wu et
al., 2018). We provide the hyper-parameters in detail in Table A.5.

###  5.2 Quantitative results

Evaluation on SimSiam. Table 1 shows the evaluation results for supervised and
unsupervised representations learnt by SimSiam (Chen & He, 2021) across
various continual learning strategies. In all cases, continual learning with
unsupervised representations achieves significantly better performance than
supervised representations with substantially lower forgetting. For instance,
Si with UCL obtains better performance and $68\%$, $54\%$, and $44\%$ lower
forgetting relative to the best-performing SCL strategy on Split CIFAR-10,
Split CIFAR-100, and Split Tiny-ImageNet, respectively. Surprisingly, Finetune
with UCL achieves higher performance and significantly lower forgetting in
comparison to all SCL strategies except Der. Furthermore, Lump improves upon
the UCL strategies: $2.8\%$ and $5.9\%$ relative increase in accuracy and
$15\%$ and $57.1\%$ relative decrease in forgetting on Split CIFAR-100 and
Split Tiny-ImageNet, respectively.

Evaluation on BarlowTwins. To verify that unsupervised representations are
indeed more robust to catastrophic forgetting, we train BarlowTwins (Zbontar
et al., 2021) on a sequence of tasks. We notice that the representations
learned with BarlowTwins substantially improve the accuracy and forgetting
over SCL: $71.4\%$, $69.7\%$ and $73.2\%$ decrease in forgetting with Finetune
on Split CIFAR-10, Split CIFAR-100 and Split Tiny-ImageNet respectively.
Similarly, we observe that Si, and Der are more robust to catastrophic
forgetting; however, PNN underperforms on complicated tasks since feature
accumulation using adaptor modules is insufficient to construct useful
representations for current task adaptation. Interestingly, representations
learnt with BarlowTwins achieve lower forgetting for Finetune, Der and Lump
than SimSiam with comparable accuracy across all the datasets.

Table 1: Accuracy and forgetting of the learnt representations on Split
CIFAR-10, Split CIFAR-100 and Split Tiny-ImageNet on Resnet-18 architecture
with KNN classifier (Wu et al., 2018). All the values are measured by
computing mean and standard deviation across three trials. The best and
second-best results are highlighted in bold and underline respectively.

| Method | Split CIFAR-10 | Split CIFAR-100 | Split Tiny-ImageNet
---|---|---|---|---
|  | Accuracy | Forgetting | Accuracy | Forgetting | Accuracy | Forgetting
| Supervised Continual Learning
| Finetune | 82.87 ($\pm$ 0.47) | 14.26 ($\pm$ 0.52) | 61.08 ($\pm$ 0.04) | 31.23 ($\pm$ 0.41) | 53.10 ($\pm$ 1.37) | 33.15 ($\pm$ 1.22)
|  PNN (Rusu et al., 2016) | 82.74 ($\pm$ 2.12) | $-$ | 66.05 ($\pm$ 0.86) | $-$ | 64.38 ($\pm$ 0.92) | $-$
|  Si (Zenke et al., 2017) | 85.18 ($\pm$ 0.65) | 11.39 ($\pm$ 0.77) | 63.58 ($\pm$ 0.37) | 27.98 ($\pm$ 0.34) | 44.96 ($\pm$ 2.41) | 26.29 ($\pm$ 1.40)
|  A-gem (Chaudhry et al., 2019a) | 82.41 ($\pm$ 1.24) | 13.82 ($\pm$ 1.27) | 59.81 ($\pm$ 1.07) | 30.08 ($\pm$ 0.91) | 60.45 ($\pm$ 0.24) | 24.94 ($\pm$ 1.24)
|  Gss (Aljundi et al., 2019) | 89.49 ($\pm$ 1.75) |  7.50 ($\pm$ 1.52) | 70.78 ($\pm$ 1.67) | 21.28 ($\pm$ 1.52) | 70.96 ($\pm$ 0.72) | 14.76 ($\pm$ 1.22)
|  Der (Buzzega et al., 2020) | 91.35 ($\pm$ 0.46) |  5.65 ($\pm$ 0.35) | 79.52 ($\pm$ 1.88) | 12.80 ($\pm$ 1.47) | 68.03 ($\pm$ 0.85) | 17.74 ($\pm$ 0.65)
| Multitask | 97.77 ($\pm$ 0.15) | $-$ | 93.89 ($\pm$ 0.78) | $-$ | 91.79 ($\pm$ 0.46) | $-$
| Unsupervised Continual Learning
SimSiam | Finetune | 90.11 ($\pm$ 0.12) |  5.42 ($\pm$ 0.08) | 75.42 ($\pm$ 0.78) | 10.19 ($\pm$ 0.37) | 71.07 ($\pm$ 0.20) | 9.48 ($\pm$ 0.56)
PNN (Rusu et al., 2016) | 90.93 ($\pm$ 0.22) | $-$ | 66.58 ($\pm$ 1.00) | $-$ | 62.15 ($\pm$ 1.35) | $-$
Si (Zenke et al., 2017) | 92.75 ($\pm$ 0.06) |  1.81 ($\pm$ 0.21) | 80.08 ($\pm$ 1.30) |  5.54 ($\pm$ 1.30) | 72.34 ($\pm$ 0.42) | 8.26 ($\pm$ 0.64)
Der (Buzzega et al., 2020) | 91.22 ($\pm$ 0.30) |  4.63 ($\pm$ 0.26) | 77.27 ($\pm$ 0.30) |  9.31 ($\pm$ 0.09) | 71.90 ($\pm$ 1.44) | 8.36 ($\pm$ 2.06)
Lump | 91.00 ($\pm$ 0.40) |  2.92 ($\pm$ 0.53) | 82.30 ($\pm$ 1.35) |  4.71 ($\pm$ 1.52) | 76.66 ($\pm$ 2.39) | 3.54 ($\pm$ 1.04)
Multitask | 95.76 ($\pm$ 0.08) | $-$ | 86.31 ($\pm$ 0.38) | $-$ | 82.89 ($\pm$ 0.49) | $-$
BarlowTwins | Finetune | 87.72 ($\pm$ 0.32) |  4.08 ($\pm$ 0.56) | 71.97 ($\pm$ 0.54) |  9.45 ($\pm$ 1.01) | 66.28 ($\pm$ 1.23) | 8.89 ($\pm$ 0.66)
PNN (Rusu et al., 2016) | 87.52 ($\pm$ 0.33) | $-$ | 57.93 ($\pm$ 2.98) | $-$ | 48.70 ($\pm$ 2.59) | $-$
Si (Zenke et al., 2017) | 90.21 ($\pm$ 0.08) |  2.03 ($\pm$ 0.22) | 75.04 ($\pm$ 0.63) |  7.43 ($\pm$ 0.67) | 56.96 ($\pm$ 1.48) | 17.04 ($\pm$ 0.89)
Der (Buzzega et al., 2020) | 88.67 ($\pm$ 0.24) |  2.41 ($\pm$ 0.26) | 73.48 ($\pm$ 0.53) |  7.98 ($\pm$ 0.29) | 68.56 ($\pm$ 1.47) |  7.87 ($\pm$ 0.44)
Lump | 90.31 ($\pm$ 0.30) |  1.13 ($\pm$ 0.18) | 80.24 ($\pm$ 1.04) |  3.53 ($\pm$ 0.83) | 72.17 ($\pm$ 0.89) |  2.43 ($\pm$ 1.00)
Multitask | 95.48 ($\pm$ 0.14) | $-$ | 87.16 ($\pm$ 0.52) | $-$ | 82.42 ($\pm$ 0.74) | $-$

Evaluation on Few-shot training. Figure 2 compares the effect of few-shot
training on UCL and SCL, where each task has a limited number of training
instances. Specifically, we conduct the experimental evaluation using $100$,
$200$, $500$, and $2500$ training instances for each task in split CIFAR-100
dataset. Surprisingly, we observe that the gap in average accuracy between SCL
and UCL methods widens with a decrease in the number of training instances.
Note that UCL decreases the accuracy by $15.78\%p$ on average with lower
forgetting when the number of training instances decreases from $2500$ to
$100$; whereas, SCL obtains a severe $32.21\%p$ deterioration in accuracy. We
conjecture that this is an outcome of the discriminative feature embeddings
learned by UCL, which discriminates all the images in the dataset and captures
more than class-specific information as also observed in Doersch et al.
(2020). Furthermore, Lump improves the performance over all the baselines with
a significant margin across all few-shot experiments.

|
---|---

Figure 2: Evaluation on Few-shot training for Split CIFAR-100 across different
number of training instances per task. The results are measured across three
independent trials.

|  |
---|---|---
(a) Finetune |  (b) Si |  (c) Der

Figure 3: CKA Feature similarity between two independent UCL models (red), two
independent SCL models (blue), and UCL and SCL model (green) for different
strategies on Split CIFAR-100 test distribution.

Evaluation on OOD datasets. We evaluate the learnt representations on various
out-of-distribution (OOD) datasets in Table 2 to measure their generalization
to unseen data distributions. In particular, we conduct the OOD evaluation on
MNIST (LeCun, 1998), Fashion-MNIST (FMNIST) (Xiao et al., 2017), SVHN (Netzer
et al., 2011), CIFAR-10 and CIFAR-100 (Krizhevsky, 2012) using a KNN
classifier (Wu et al., 2018). We observe that unsupervised representations
outperform the supervised representations in all cases across all the
datasets. In particular, the UCL representations learned with Simsiam, and Si
on Split-CIFAR-10 improves the absolute performance over the best-performing
SCL strategy by 4.58%, 6.09%, 15.26%, and 17.07% on MNIST, FMNIST, SVHN, and
CIFAR-100 respectively. Further, Lump trained on Split-CIFAR-100 outperforms
Si across all datasets and obtains comparable performance with Split CIFAR-10
dataset.

###  5.3 Qualitative analysis

Similarity in feature and parameter space. We analyze the similarity between
the representations learnt between (i) Two independent UCL models, (ii) Two
independent SCL models (iii) SCL and UCL models using centered kernel
alignment (CKA) (Kornblith et al., 2019) in Figure 3, which provides a score
between 0 and 1 measuring the similarity between a pair of hidden
representations. For two representations
$\Theta_{1}:\mathcal{X}\rightarrow\mathbb{R}^{d_{1}}$ and
$\Theta_{2}:\mathcal{X}\rightarrow\mathbb{R}^{d_{1}}$,
$\text{CKA}(\Theta_{1},\Theta_{2})=\frac{||\text{Cov}(\Theta_{1}(x),\Theta_{2}(x))||_{F}^{2}}{||\text{Cov}(\Theta_{1}(x))||_{F}\cdot||\text{Cov}(\Theta_{2}(x))||_{F}}$,
where covariances are with respect to the test distribution. Additionally, we
measure the $\ell_{2}$ distance (Neyshabur et al., 2020) between the
parameters of two independent UCL models (see Table 3) and two independent SCL
models (see Table 4). First, we observe that the representations learned by
two independent UCL methods have a high feature similarity and lower
$\ell_{2}$ distance compared to the two independent SCL methods, demonstrating
UCL representations’ robustness. Second, we note that the representations
between any two independent models are highly similar in the lower layers
indicating that they learn similar high-level features, including edges and
shapes; however, the features are dissimilar for the higher modules. Lastly,
we see that the representations between a UCL and SCL model are similar in the
lower layers but diverge in the higher layers across all CL strategies.

Table 2: Comparison of accuracy on out of distribution datasets using a KNN
classifier (Wu et al., 2018) on pre-trained SCL and UCL representations. We
consider MNIST (LeCun, 1998), Fashion-MNIST (FMNIST) (Xiao et al., 2017), SVHN
(Netzer et al., 2011) as out of distribution for Split CIFAR-100 and Split
CIFAR-10. All the values are measured by computing mean and standard deviation
across three trials. The best and second-best results are highlighted in bold
and underline respectively.

| In-class | Split CIFAR-10 | Split CIFAR-100 |
---|---|---|---|---
| Out-of-class | MNIST | FMNIST | SVHN | CIFAR-100 | MNIST | FMNIST | SVHN | CIFAR-10 |
| Supervised Continual Learning |
| Finetune | 86.42 ($\pm$ 1.11) | 74.47 ($\pm$ 0.84) | 41.00 ($\pm$ 0.85) | 17.42 ($\pm$ 0.96) | 75.02 ($\pm$ 3.97) | 62.37 ($\pm$ 3.20) | 38.05 ($\pm$ 0.73) | 39.18 ($\pm$ 0.83) |
|  Si (Zenke et al., 2017) | 87.08 ($\pm$ 0.79) | 76.41 ($\pm$ 0.81) | 42.62 ($\pm$ 1.31) | 19.14 ($\pm$ 0.91) | 79.96 ($\pm$ 2.63) | 63.71 ($\pm$ 1.36) | 40.92 ($\pm$ 1.64) | 40.41 ($\pm$ 1.71) |
|  A-gem (Chaudhry et al., 2019a) | 86.07 ($\pm$ 1.94) | 74.74 ($\pm$ 3.21) | 37.77 ($\pm$ 3.49) | 16.11 ($\pm$ 0.38) | 77.56 ($\pm$ 3.21) | 64.16 ($\pm$ 2.29) | 37.48 ($\pm$ 1.73) | 37.91 ($\pm$ 1.33) |
|  Gss (Aljundi et al., 2019) | 70.36 ($\pm$ 3.54) | 69.20 ($\pm$ 2.51) | 33.11 ($\pm$ 2.26) | 18.21 ($\pm$ 0.39) | 76.54 ($\pm$ 0.46) | 65.31 ($\pm$ 1.72) | 35.72 ($\pm$ 2.37) | 49.41 ($\pm$ 1.81) |
|  Der (Buzzega et al., 2020) | 80.32 ($\pm$ 1.91) | 70.49 ($\pm$ 1.54) | 41.48 ($\pm$ 2.76) | 17.72 ($\pm$ 0.25) | 87.71 ($\pm$ 2.23) | 75.97 ($\pm$ 1.29) | 50.26 ($\pm$ 0.95) | 59.07 ($\pm$ 1.06) |
| multitask | 88.79 ($\pm$ 1.13) | 79.50 ($\pm$ 0.52) | 41.26 ($\pm$ 1.95) | 27.68 ($\pm$ 0.66) | 92.29 ($\pm$ 3.37) | 86.12 ($\pm$ 1.87) | 54.94 ($\pm$ 1.77) | 54.04 ($\pm$ 3.68) |
| Unsupervised Continual Learning |
SimSiam | Finetune | 89.23 ($\pm$ 0.99) | 80.05 ($\pm$ 0.34) | 49.66 ($\pm$ 0.81) | 34.52 ($\pm$ 0.12) | 85.99 ($\pm$ 0.86) | 76.90 ($\pm$ 0.11) | 50.09 ($\pm$ 1.41) | 57.15 ($\pm$ 0.96) |
Si (Zenke et al., 2017) |  93.72 ($\pm$ 0.58) |  82.50 ($\pm$ 0.51) |  57.88 ($\pm$ 0.16) |  36.21 ($\pm$ 0.69) | 91.50 ($\pm$ 1.26) | 80.57 ($\pm$ 0.93) |  54.07 ($\pm$ 2.73) | 60.55 ($\pm$ 2.54) |
Der (Buzzega et al., 2020) | 88.35 ($\pm$ 0.82) | 79.33 ($\pm$ 0.62) | 48.83 ($\pm$ 0.55)) | 30.68 ($\pm$ 0.36) | 87.96 ($\pm$ 2.04) | 76.21 ($\pm$ 0.63) | 47.70 ($\pm$ 0.94) | 56.26 ($\pm$ 0.16) |
Lump | 91.03 ($\pm$ 0.22) | 80.78 ($\pm$ 0.88) | 45.18 ($\pm$ 1.57) | 31.17 ($\pm$ 1.83) |  91.76 ($\pm$ 1.17) |  81.61 ($\pm$ 0.45) | 50.13 ($\pm$ 0.71) |  63.00 ($\pm$ 0.53) |
Multitask | 90.69 ($\pm$ 0.13) | 80.65 ($\pm$ 0.42) | 47.67 ($\pm$ 0.45) | 39.55 ($\pm$ 0.18) | 90.35 ($\pm$ 0.24) | 81.11 ($\pm$ 1.86) | 52.20 ($\pm$ 0.61) | 70.19 ($\pm$ 0.15) |
BarlowTwins | Finetune | 86.86 ($\pm$ 1.62) | 78.37 ($\pm$ 0.74) | 44.64 ($\pm$ 2.39) | 28.03 ($\pm$ 0.52) | 76.08 ($\pm$ 2.86) | 76.82 ($\pm$ 0.83) | 42.95 ($\pm$ 0.90) | 53.12 ($\pm$ 0.13) |
Si (Zenke et al., 2017) | 90.31 ($\pm$ 0.69) | 80.58 ($\pm$ 0.68) | 49.18 ($\pm$ 0.51) | 31.80 ($\pm$ 0.4) | 85.24 ($\pm$ 0.99) | 78.82 ($\pm$ 0.67) | 45.18 ($\pm$ 1.37) | 53.99 ($\pm$ 0.56) |
Der (Buzzega et al., 2020) | 85.15 ($\pm$ 2.19) | 77.96 ($\pm$ 0.59) | 45.68 ($\pm$ 0.93) | 27.83 ($\pm$ 0.86) | 78.08 ($\pm$ 1.95) | 76.67 ($\pm$ 0.68) | 44.58 ($\pm$ 1.01) | 53.24 ($\pm$ 0.82) |
Lump | 88.73 ($\pm$ 0.54) | 81.69 ($\pm$ 0.45) | 51.53 ($\pm$ 0.41) | 31.53 ($\pm$ 0.36) |  90.22 ($\pm$ 1.39) |  81.28 ($\pm$ 0.91) | 50.24 ($\pm$ 0.95) |  60.76 ($\pm$ 0.87) |
Multitask | 88.63 ($\pm$ 1.38) | 79.49 ($\pm$ 0.29) | 49.24 ($\pm$ 2.44) | 36.33 ($\pm$ 0.29) | 86.98 ($\pm$ 1.70) | 79.40 ($\pm$ 1.10) | 50.19 ($\pm$ 0.81) | 49.50 ($\pm$ 0.38) |

Visualization of feature space. Next, we visualize the learned features to
dissect further the representations learned by UCL and SCL strategies. Figure
4 shows the visualization of the latent feature maps for tasks
$\mathcal{T}_{0}$ and $\mathcal{T}_{13}$ after the completion of continual
learning. For $\mathcal{T}_{0}$, we observe that the SCL methods are prone to
catastrophic forgetting, as the features appear noisy and do not have coherent
patterns. In contrast, the features learned by UCL strategies are perceptually
relevant and robust to catastrophic forgetting, with Lump learning the most
distinctive features. Similar to $\mathcal{T}_{0}$, we observe that the UCL
features are more relevant and distinguishable than SCL for
$\mathcal{T}_{13}$. Note that we randomly selected the examples and feature
maps for all visualizations.

Loss landscape visualization.  To gain further insights, we visualize the loss
landscape of task $\mathcal{T}_{0}$ after the completion of training on task
$\mathcal{T}_{0}$ and $\mathcal{T}_{19}$ for various UCL and SCL strategies in
Figure 5. We measure the cross-entropy loss for all methods with a randomly
initialized linear classifier for a fair evaluation of two different
directions. We use the visualization tool from Li et al. (2018) that searches
the task loss surface by repeatedly adding random perturbations to model
weights. We observe that the loss landscape after $\mathcal{T}_{0}$ looks
quite similar across all the strategies since the forgetting does not exist
yet. However, after training $\mathcal{T}_{19}$, there is a clear difference
with the UCL strategies obtaining a flatter and smoother loss landscape
because UCL methods are more stable and robust to the forgetting, which hurts
the loss landscapes of past tasks for SCL. It is important to observe that
Lump obtains a smoother landscape than other UCL strategies, demonstrating its
effectiveness. We defer further analyses for feature and loss landscape
visualization to Section A.2.

Table 3: $\ell_{2}$ distance between UCL parameters after completion of
training.

Model | Finetune | Si | Der | MultiTask
---|---|---|---|---
Finetune | 60.00 ($\pm$ 1.70) |  |  |
Si | 76.46 ($\pm$ 0.48) | 92.35 ($\pm$ 0.61) |  |
der | 55.60 ($\pm$ 1.42) | 75.54 ($\pm$ 0.97) | 48.76 ($\pm$ 1.54) |
Multitask | 61.32 ($\pm$ 0.59) | 79.95 ($\pm$ 0.40) | 57.90 ($\pm$ 0.86) | 61.42 ($\pm$ 0.78 )

Table 4: $\ell_{2}$ distance between SCL paraneters after completion of
training.

Model | Finetune | Si | Der | MultiTask
---|---|---|---|---
Finetune | 183.31 ($\pm$ 0.10) |  |  |
Si | 206.16 ($\pm$ 0.28) | 226.05 ($\pm$ 0.13) |  |
der | 202.61 ($\pm$ 0.46) | 224.78 ($\pm$ 0.75) | 219.06 ($\pm$ 0.27) |
Multitask | 258.12 ($\pm$ 0.26) | 277.30 ($\pm$ 0.69) | 271.48 ($\pm$ 0.45) | 314.84 ($\pm$ 0.92)

|  |  |  |  |  |  |
---|---|---|---|---|---|---|---
Apple ($\mathcal{T}_{0}$) |  | Scl-Finetune
---
Acc: $54.7\pm 0.2$
| Scl-Si
---
Acc: $58.9\pm 0.2$
| Scl-Gss
---
Acc: $78.4\pm 1.8$
| Scl-Der
---
Acc: $73.1\pm 0.4$
| Ucl-Finetune
---
Acc: $70.8\pm 0.4$
| Ucl-Si
---
Acc: $76.4\pm 1.6$
| Lump (Ours)
---
Acc: $76.6\pm 2.7$

|  |  |  |  |  |  |
---|---|---|---|---|---|---|---
Raccoon ($\mathcal{T}_{13}$) |  | Scl-Finetune
---
Acc: $50.6\pm 1.4$
| Scl-Si
---
Acc: $48.4\pm 1.0$
| Scl-Gss
---
Acc: $59.9\pm 2.2$
| Scl-Der
---
Acc: $76.4\pm 2.2$
| Ucl-Finetune
---
Acc: $74.6\pm 0.5$
| Ucl-Si
---
Acc: $78.0\pm 1.6$
| Lump (Ours)
---
Acc: $80.8\pm 0.5$

Figure 4: Visualization of feature maps for the second block representations
learnt by SCL and UCL strategies
(with Simsiam) for ResNet-18 architecture after the completion of CL for Split
CIFAR-100 dataset ($n=20$).

$\mathcal{T}_{0}$ |  |  |  |  |
---|---|---|---|---|---
$\mathcal{T}_{19}$ |  |  |  |  |
| Scl-Finetune
---
Acc:$54.73\pm 0.25$
| Scl-Si
---
Acc: $58.59\pm 0.20$
| Scl-Der
---
Acc: $73.13\pm 0.38$
| Ucl-Finetune
---
Acc: $70.80\pm 0.40$
| Ucl-Si
---
Acc: $76.39\pm 1.56$
| Lump (Ours)
---
Acc:$76.60\pm 2.70$

Figure 5: Loss landscape visualization of $\mathcal{T}_{0}$ after the
completion of training on task $\mathcal{T}_{0}$ (top) and $\mathcal{T}_{19}$
(bottom) for Split CIFAR-100 dataset on ResNet-18 architecture. We use Simsiam
for UCL methods.

##  6 Discussion and Conclusion

This work attempts to bridge the gap between unsupervised representation
learning and continual learning. In particular, we establish the following
findings for unsupervised continual learning.

Surpassing supervised continual learning. Our empirical evaluation across
various CL strategies and datasets shows that UCL representations are more
robust to catastrophic forgetting than SCL representations. Furthermore, we
notice that UCL generalizes better to OOD tasks and achieves stronger
performance on few-shot learning tasks. We propose _Lifelong unsupervised
mixup ( Lump)_, which interpolates the unsupervised instances between the
current task and past task and obtains higher performance with lower
catastrophic forgetting across a wide range of tasks.

Dissecting the learned representations. We conduct a systematic analysis to
understand the differences between the representations learned by UCL and SCL
strategies. By investigating the similarity between the representations, we
observe that UCL and SCL strategies have high similarities in the lower layers
but are dissimilar in the higher layers. We also show that UCL representations
learn coherent and discriminative patterns and smoother loss landscape than
SCL.

Limitations and future work. In this work, we do not consider the high-
resolution tasks for CL. We intend to evaluate the forgetting of the learnt
representations on ImageNet (Deng et al., 2009) in future work, since UCL
shows lower catastrophic forgetting and representation learning has made
significant progress on ImageNet over the past years. In follow-up work, we
intend to conduct further analysis to understand the behavior of UCL and
develop sophisticated methods to continually learn unsupervised
representations under various setups, such as class-incremental or task-
agnostic CL.

## Acknowledgements

We thank the anonymous reviewers for their insightful comments and
suggestions. This work was supported by Microsoft Research Asia, the
Engineering Research Center Program through the National Research Foundation
of Korea (NRF) funded by the Korean Government MSIT (NRF-2018R1A5A1059921),
Institute of Information & communications Technology Planning & Evaluation
(IITP) grant funded by the Korea government (MSIT) (No.2019-0-00075,
Artificial Intelligence Graduate School Program (KAIST) and 2021-0-01696). Any
opinions, findings, and conclusions or recommendations expressed in this
material are those of the authors and do not necessarily reflect the views of
the funding agencies.

## Author Contributions

Divyam Madaan conceived of the presented idea, developed the experimental
framework, carried out OOD evaluation, CKA visualization and took the lead in
writing the manuscript. Jaehong Yoon performed the hyperparameter search,
carried out the visualization of loss landscape and feature maps and performed
the few-shot training analysis. Yuanchun Li, Yunxin Liu, and Sung Ju Hwang
supervised the project.

## References

  * Ahn et al. (2019) Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon.  Uncertainty-based continual learning with adaptive regularization.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * Aljundi et al. (2019) Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio.  Gradient based sample selection for online continual learning.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * Barlow (1961) Horace Barlow.  Possible principles underlying the transformations of sensory messages.  1961\.
  * Bojarski et al. (2016) Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba.  End to end learning for self-driving cars.  _arXiv preprint arXiv:1604.07316_ , 2016.
  * Bromley et al. (1994) Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah.  Signature verification using a "siamese" time delay neural network.  1994\.
  * Buzzega et al. (2020) Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara.  Dark experience for general continual learning: a strong, simple baseline.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Caron et al. (2020) Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Chaudhry et al. (2019a) Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny.  Efficient lifelong learning with a-gem.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2019a.
  * Chaudhry et al. (2019b) Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and M Ranzato.  Continual learning with tiny episodic memories.  _arXiv preprint arXiv:1902.10486_ , 2019b.
  * Chen et al. (2020a) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2020a.
  * Chen et al. (2020b) Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton.  Big self-supervised models are strong semi-supervised learners.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020b.
  * Chen & He (2021) Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Chen et al. (2020c) Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  _arXiv preprint arXiv:2003.04297_ , 2020c.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2009.
  * Doersch et al. (2020) Carl Doersch, Ankush Gupta, and Andrew Zisserman.  Crosstransformers: spatially-aware few-shot transfer.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2017.
  * Flesch et al. (2018) Timo Flesch, Jan Balaguer, Ronald Dekker, Hamed Nili, and Christopher Summerfield.  Comparing continual task learning in minds and machines.  _Proceedings of the National Academy of Sciences_ , 2018.
  * Goodfellow et al. (2013) Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio.  An empirical investigation of catastrophic forgetting in gradient-based neural networks.  _arXiv preprint arXiv:1312.6211_ , 2013.
  * Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko.  Bootstrap your own latent - a new approach to self-supervised learning.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Kelly et al. (2019) Christopher J Kelly, Alan Karthikesalingam, Mustafa Suleyman, Greg Corrado, and Dominic King.  Key challenges for delivering clinical impact with artificial intelligence.  _BMC medicine_ , 2019.
  * Kim et al. (2020) Sungnyun Kim, Gihun Lee, Sangmin Bae, and Se-Young Yun.  Mixco: Mix-up contrastive learning for visual representation.  _arXiv preprint arXiv:2010.06300_ , 2020.
  * Kornblith et al. (2019) Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton.  Similarity of neural network representations revisited.  In _Proceedings of the International Conference on Machine Learning (ICML)_. PMLR, 2019.
  * Krizhevsky (2012) Alex Krizhevsky.  Learning multiple layers of features from tiny images.  _University of Toronto_ , 05 2012.
  * Kumar & Daume III (2012) Abhishek Kumar and Hal Daume III.  Learning task grouping and overlap in multi-task learning.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2012.
  * LeCun (1998) Yann LeCun.  The mnist database of handwritten digits.  _http://yann. lecun. com/exdb/mnist/_ , 1998.
  * Lee et al. (2021) Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee.  i-mix: A domain-agnostic strategy for contrastive representation learning.  In _ICLR_ , 2021.
  * Li et al. (2018) Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein.  Visualizing the loss landscape of neural nets.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2018.
  * Li et al. (2019) Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong.  Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2019.
  * Li et al. (2020) Yuanpeng Li, Liang Zhao, Kenneth Church, and Mohamed Elhoseiny.  Compositional language continual learning.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2020.
  * Li & Hoiem (2016) Zhizhong Li and Derek Hoiem.  Learning without forgetting.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2016.
  * McCloskey & Cohen (1989) Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_. 1989.
  * Mirzadeh et al. (2020) Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh.  Understanding the role of training regimes in continual learning.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Netzer et al. (2011) Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  2011\.
  * Neyshabur et al. (2020) Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang.  What is being transferred in transfer learning?  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Rao et al. (2019) Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell.  Continual unsupervised representation learning.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Rebuffi et al. (2017) Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.  icarl: Incremental classifier and representation learning.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Rolnick et al. (2019) David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne.  Experience replay for continual learning.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * Rusu et al. (2016) Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  _arXiv preprint arXiv:1606.04671_ , 2016.
  * Schwarz et al. (2018) Jonathan Schwarz, Jelena Luketina, Wojciech M Czarnecki, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell.  Progress & compress: A scalable framework for continual learning.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2018.
  * Shen et al. (2022) Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, Trevor Darrell, and Eric Xing.  Un-mix: Rethinking image mixtures for unsupervised visual representation learning.  In _Proceedings of the AAAI National Conference on Artificial Intelligence (AAAI)_ , 2022.
  * Smith et al. (2021) James Smith, Cameron Taylor, Seth Baer, and Constantine Dovrolis.  Unsupervised progressive learning and the stam architecture.  In _Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)_ , 2021.
  * Thrun (1995) Sebastian Thrun.  _A Lifelong Learning Perspective for Mobile Robot Control_.  Elsevier, 1995.
  * Verma et al. (2021) Vikas Verma, Thang Luong, Kenji Kawaguchi, Hieu Pham, and Quoc Le.  Towards domain-agnostic contrastive learning.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2021.
  * Wu et al. (2018) Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Xiao et al. (2017) Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  _arXiv preprint arXiv:1708.07747_ , 2017.
  * Yoon et al. (2018) Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang.  Lifelong learning with dynamically expandable networks.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2018.
  * Zbontar et al. (2021) Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2021.
  * Zenke et al. (2017) Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2017.
  * Zhang et al. (2018) Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2018.
  * Zhang et al. (2021) Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou.  How does mixup help with robustness and generalization?  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2021.

##  Appendix A Supplementary Material

Organization. In the supplementary material, we provide the implementation
details followed by the hyper-parameter configurations in Section A.1.
Further, we show the other experiments we conducted and additional
visualizations and results in Section A.2.

###  A.1 Experimental Details

Implementations. We use the DER (Buzzega et al., 2020) open-source
codebase111https://github.com/aimagelab/mammoth for all the experiments. In
particular, we reproduce all their experimental results for supervised
continual learning and use various models with their set of hyper-parameters
as our baselines. We follow the original representations for
SimSiam222https://github.com/facebookresearch/simsiam and
BarlowTwins333https://github.com/facebookresearch/barlowtwins for unsupervised
continual learning. We verify our implementation by reproducing the reported
results on CIFAR-10 in the original paper, where we train the representations
on the complete CIFAR-10 dataset and evaluate on the test-set using KNN
classifier (Wu et al., 2018). In particular, (Wu et al., 2018) stores the
features for each instance in the task-level training set in a discrete memory
bank. The optimal feature-level embeddings are then learned by instance-level
discrimination, which maximally scatters the features of the training samples.
Following prior works in representation learning, we use the task-level
training set without any augmentation in the task-incremental setup for the
supervised and unsupervised KNN evaluation.

Hyperparameter configurations. We use the tuned hyper-parameters reported by
Buzzega et al. (2020) for all the SCL experiments. On the other hand, we tune
the hyper-parameters for continual learning strategies for UCL. We provide the
hyper-parameters setup for UCL for different datasets in Table A.5. We train
all the UCL methods with a batch size of $256$ for $200$ epochs, while
training the SCL methods with a batch size of $32$ for $50$ epochs following
Buzzega et al. (2020). We observed that training the SCL methods further lead
to a degredation in performance for all the methods. We use the same set of
augmentations for both SCL and UCL except that we use RandomResizedCrop with
scale in $[0.2,1.0]$ for UCL (Wu et al., 2018; Chen & He, 2021) and RandomCrop
for SCL. For rehearsal-based methods, we use the buffer size $200$ for Split
CIFAR-10, Split CIFAR-100 and $256$ for Split Tiny-ImageNet dataset. We use a
learning rate of $0.03$ for SGD optimizer with weight decay $5$e-$4$ and
momentum $0.9$.

Table A.5: Hyperparameter configurations for all the datasets on ResNet-18 architecture.  Method | Split CIFAR-10 | Split CIFAR-100 | Seq. Tiny-ImageNet
---|---|---|---
Si |  c $:100$ $\xi:1$ |  c $:0.1$ $\xi:1$ |  c $:0.01$ $\xi:1$
Pnn |  wd $:64$ |  wd $:12$ |  wd $:8$
DER |  $\alpha:0.1$ |  $\alpha:0.1$ |  $\alpha:0.01$
Lump |  $\lambda:0.1$ |  $\lambda:0.1$ |  $\lambda:0.4$

###  A.2 Additional Experiments

We provide additional loss landscape on Split CIFAR-100 in Figure A.6 and
Figure A.7, Figure A.8 show the second and third block feature visualizations
on Split CIFAR-100 respectively. Figure A.9 shows the feature visualizations
for Split Tiny-ImageNet on ResNet-18 architecture.

| $\mathcal{T}_{0}$ | $\mathcal{T}_{17}$ | $\mathcal{T}_{18}$ | $\mathcal{T}_{19}$
---|---|---|---|---
| Scl-Finetune
---
Acc: $54.73\pm 0.25$
|  |  |
| Scl-Si
---
Acc: $58.59\pm 0.20$
|  |  |
| Scl-Der
---
Acc: $73.13\pm 0.38$
|  |  |
| Ucl-Finetune
---
Acc: $70.80\pm 0.40$
|  |  |
| Ucl-Si
---
Acc: $76.39\pm 1.59$
|  |  |
| Lump (Ours)
---
Acc: $76.60\pm 2.70$
|  |  |

Figure A.6: Loss landscape visualization of $\mathcal{T}_{0}$ after the
completion of training on task $\mathcal{T}_{0}$,
$\mathcal{T}_{17},\mathcal{T}_{18}$, and $\mathcal{T}_{19}$ for Split
CIFAR-100 dataset on ResNet-18 architecture. We use Simsiam for UCL methods.

| Raw Images
---
$\mathcal{T}_{0},\leavevmode\nobreak\ \mathcal{T}_{1},\leavevmode\nobreak\
\mathcal{T}_{2},\leavevmode\nobreak\ \mathcal{T}_{13}$
|  |  |
| Scl-Finetune
---
Acc:$61.08\pm 0.04$
|  |  |
| Scl-Si
---
Acc:$63.58\pm 0.37$
|  |  |
| Scl-Gss
---
Acc:$70.78\pm 1.67$
|  |  |
| Scl-Der
---
Acc:$79.52\pm 1.88$
|  |  |
| Ucl-Finetune
---
Acc:$75.42\pm 0.78$
|  |  |
| Ucl-Si
---
Acc:$80.08\pm 1.30$
|  |  |
| Lump (Ours)
---
Acc:$82.30\pm 1.35$
|  |  |

Figure A.7: Visualization of feature maps for the second block representations
learnt by SCL and UCL strategies (with Simsiam) for Resnet-18 architecture
after the completion of continual learning for Split CIFAR-100 dataset
($n=20$). The accuracy is the mean across three runs for the corresponding
task.

| Raw Images
---
$\mathcal{T}_{0},\leavevmode\nobreak\ \mathcal{T}_{1},\leavevmode\nobreak\
\mathcal{T}_{2},\leavevmode\nobreak\ \mathcal{T}_{13}$
|  |  |
| Scl-Finetune
---
Acc:$61.08\pm 0.04$
|  |  |
| Scl-Si
---
Acc:$63.58\pm 0.37$
|  |  |
| Scl-Gss
---
Acc:$70.78\pm 1.67$
|  |  |
| Scl-Der
---
Acc:$79.52\pm 1.88$
|  |  |
| Ucl-Finetune
---
Acc:$75.42\pm 0.78$
|  |  |
| Ucl-Si
---
Acc:$80.08\pm 1.30$
|  |  |
| Lump (Ours)
---
Acc:$82.30\pm 1.35$
|  |  |

Figure A.8: Visualization of feature maps for the third block representations
learnt by SCL and UCL strategies (with Simsiam) for Resnet-18 architecture
after the completion of continual learning for Split CIFAR-100 dataset
($n=20$). The accuracy is the mean across three runs for the corresponding
task.

| Raw Images
---
$\mathcal{T}_{0},\leavevmode\nobreak\ \mathcal{T}_{1},\leavevmode\nobreak\
\mathcal{T}_{2},\leavevmode\nobreak\ \mathcal{T}_{10}$
|  |  |
| Scl-Finetune
---
Acc:$53.10\pm 1.37$
|  |  |
| Scl-Si
---
Acc:$44.96\pm 2.41$
|  |  |
| Scl-Gss
---
Acc:$70.96\pm 0.72$
|  |  |
| Scl-Der
---
Acc:$68.03\pm 0.85$
|  |  |
| Ucl-Finetune
---
Acc:$71.07\pm 0.20$
|  |  |
| Ucl-Si
---
Acc:$72.34\pm 0.42$
|  |  |
| Lump (Ours)
---
Acc:$76.66\pm 2.39$
|  |  |

Figure A.9: Visualization of feature maps for the second block representations
learnt by SCL and UCL strategies (with Simsiam) for Resnet-18 architecture
after the completion of continual learning for Split Tiny-ImageNet dataset
($n=20$). The accuracy is the mean across three runs for the corresponding
task.