# Tackling Catastrophic Forgetting and Background Shift in Continual Semantic
Segmentation

Arthur Douillard1,2, Yifu Chen1, Arnaud Dapogny3, and Matthieu Cord1,4 1
Sorbonne Université, 2 Heuritech, 3 Datakalab, 4 Valeo.aiCorresponding author:
Arthur Douillard<EMAIL_ADDRESS>
###### Abstract

Deep learning approaches are nowadays ubiquitously used to tackle computer
vision tasks such as semantic segmentation, requiring large datasets and
substantial computational power. Continual learning for semantic segmentation
(CSS) is an emerging trend that consists in updating an old model by
sequentially adding new classes. However, continual learning methods are
usually prone to catastrophic forgetting. This issue is further aggravated in
CSS where, at each step, old classes from previous iterations are collapsed
into the background. In this paper, we propose Local POD, a multi-scale
pooling distillation scheme that preserves long- and short-range spatial
relationships at feature level. Furthermore, we design an entropy-based
pseudo-labelling of the background w.r.t. classes predicted by the old model
to deal with background shift and avoid catastrophic forgetting of the old
classes. Finally, we introduce a novel rehearsal method that is particularly
suited for segmentation. Our approach, called PLOP, significantly outperforms
state-of-the-art methods in existing CSS scenarios, as well as in newly
proposed challenging benchmarks.

###### Index Terms:

Continual Learning, Semantic Segmentation, Knowledge Distillation, Rehearsal
Learning

##  1 Introduction

Computer vision went this last decade under a dramatic change from hand-
crafted features [1, 2] and machine learning models [3] to deep convolutional
neural networks (DCNNS) [4]. Among the existing tasks involving computer
vision, semantic segmentation aims to assign a label to each pixel of an
image. It allows the prediction of multiple objects in the same image, and
moreover their exact position and shape. This task recently florished [5, 6,
7] with larger datasets with thousand of fully annotated images [8, 9],
increased computational power, and larger attention [10]. Unfortunatly, the
recent research in this area is often impracticable for real-life
applications: they mostly need fully annotated data and require to be
retrained from scratch if a new class is added to the dataset. Idealy, one
would wish to regularly expand a dataset, only adding and labelling new
classes and updating the model in accordance. This setup, referred here as
Continual Semantic Segmentation (CSS), has emerged very recently for
specialized applications [11, 12, 13] before being proposed for general
segmentation datasets [14, 15, 16].

In particular, in this paper, we argue that two problems arise when performing
CSS with DCNNs. The first one, inherited from continual learning, is called
catastrophic forgetting [17, 18, 19], and points to the fact that neural
networks tend to completely and abruptly forget previously learned knowledge
when learning new information [20]. Catastrophic forgetting presents a real
challenge for continual learning applications based on deep learning methods.
For some applications, storing previously seen data is not allowed for privacy
reasons. This is a real challenge because the model can no longer be trained
on the old categories with labeled data. The worse part is that the previously
learned categories may not even exist in the future data, which requires the
model to never forget these classes in the case of learning only once. In the
case where storage of past data is allowed, we can use the past data to
prevent forgetting. However, given the size of the dataset, fine-tuning over
all past data is extremely costly in memory and time. Therefore, rehearsal
methods are usually employed, which consist of storing some of the past
examples and replaying them while learning new information. This problem is
exacerbated in continual segmentation where the storage cost is high and
images are partially labeled.

The second challenge, specific to CSS, is the semantic shift of the background
class. In a traditional semantic segmentation setup, all object categories are
predefined, and the ”background” class contains pixels that do not belong to
any of these classes. However, in CSS, the background contains pixels that do
nott belong to any of the current classes. Thus, for a specific learning step,
the background can contain both future classes, not yet seen by the model, as
well as old classes. Thus, if nothing is done to distinguish pixels belonging
to the real background class from old class pixels, this background shift
phenomenon risks exacerbating the catastrophic forgetting even further [15].
This issue also has an impact on the selection of the old data we want to
store. Because some currently learned classes are annotated as background in
the old data, this may degrade the performance of these classes if one naively
treat them as background to fine-tune the current model.

We tackle the first challenge of catastrophic forgetting by designing a
constraint enforcing a similar behavior between the old and current model.
Specifically, we leverage intermediary representations of the convolutional
networks to ensure that similar patterns are extracted through time. This
feature-based constraints, called Local POD, fully exploits the global and
local scale necessary to semantic segmentation through a multi-scale design.
The second challenge, background shift, is greatly allievated a confidence-
based pseudo-labeling strategy to retrieve old class pixels within the
background. For instance, if a current ground truth mask only distinguishes
pixels from class sofa and background, our approach allows to assign old
classes to background pixels, e.g. classes person, dog or background (the
semantic class). We name PLOP the model exploiting those two contributions. We
then propose an extention called PLOPLong that aims to excel on long continual
learning scenarios. This extention exploits cosine normalization to adapt the
classifier and the Local POD resulting in an improving robustness to the
discrepancy between old and new classes. Moreover, PLOPLong features a
modified batch normalization which reduces the sensitivity of the model to
moving statistics seen across tasks in continual learning. Finally, we
investigate for the first time rehearsal learning for CSS. We propose an
initial version based on rehearsing complete images which unfortunatly is
memory intensive. We then create the Object Rehearsal, a rehearsal method as
performant but significantly more memory efficient.

From a practical point of view, our proposed methods (PLOP, PLOPLong, and
Object Rehearsal) showed three important results. First, we achieve the state-
of-the-art performance on several challenging datasets. Secondly, we propose
several novel scenarios to further quantify the performances of CSS methods
when it comes to long term learning, class presentation order and domain
shift. Last but not least, we show that our model contributions largely
outperform every CSS approach in these scenarios.

To sum it up, our contributions are four-folds:

  * •

We propose a multi-scale spatial distillation loss to better retain knowledge
through the continual learning steps, by preserving long- and short-range
spatial statistics, avoiding catastrophic forgetting.

  * •

We introduce a confidence-based pseudo-labeling strategy to identify old
classes for the current background pixels and deal with background shift.

  * •

We propose PLOPLong, a carefully designed refinement of our method for dealing
with long CSS scenarios. The extention comes from an adaptation of PLOP’s
classifier and Local POD distillation as well as batch re-normalization for
better handling of both catastrophic forgetting and background shift,
respectively.

  * •

We design a novel memory-efficient Object rehearsal learning procedure that
consists in storing and carefully pasting objects through selective erasing of
foreground objects.

Additionaly, We show that PLOP significantly outperforms state-of-the-art
approaches in existing scenarios and datasets for CSS, as well as in several
newly proposed challenging benchmarks. Furthermore, we show that PLOPLong
leads to superior performances on longer CSS scenarios. Last but not least, we
show that performance of CSS models can be greatly improved where rehearsal
learning is an option. In such cases, the proposed Object rehearsal allows to
reach high accuracies with low memory footprint.

In this paper, we expand on our previous work published in conference venue
[16]. The updated PLOP includes both model and experimental improvements.
While keeping the losses introduced with PLOP, we incorporate PLOP with the
batch renormalization technique [21] and a cosine classifier [22]. We
structurally modify part of the network with these methods, which further
mitigates forgetting on long stream of tasks compared to the original PLOP. We
also investigate rehearsal learning for continual segmentation. Taking into
account the background shift problem for replying and also to minimise the
external memory needed to store past data, we propose a novel memory-efficient
rehearsal method based on objects pasting. We extend our experiments with a
class-incremental version of Cityscapes, which proved to be challenging, and
we show how our model modifications handle better this dataset. Finally, we
propose more ablations and compare rehearsal-based methods against a semi-
supervised method using an external unlabeled dataset.

##  2 Related Work

Continual Semantic Segmentation is a relativery young field that started
getting traction following [14, 15]. However, this field is at the
intersection of many popular topics. Therefore, we start this section with an
overview of recent advances in segmentation, continual learning, and a
particular aspect of the latter: rehearsal learning. We then follow with a
more in-depth discussion of existing approaches to CSS.

Semantic Segmentation methods based on Fully Convolutional Networks (FCN) [23,
24] have achieved impressive results on several segmentation benchmarks [25,
26, 8, 27]. These methods improve the segmentation accuracy by incorporating
more spatial information or exploiting contextual information specifically.
Atrous convolution [28, 29] and encoder-decoder architecture [30, 31, 32] are
the most common methods for retaining spatial information. Examples of recent
works exploiting contextual information include attention mechanisms [33, 34,
35, 36, 37, 5, 6], and fixed-scale aggregation [38, 28, 7, 39].

Continual Learning models suffers from catastrophic forgetting where old
classes are forgotten, leading to a major drop of performance [17, 19, 18].
Multiple approaches exist to reduce this forgetting, among them some adapt the
architecture continually to integrate new knowledge [40, 41, 42] or train
together co-existing sub-networks [43] each specialized in one specific task
[44, 45, 46]. It can be also possible to fix the knowledge drift in the
classifier by recalibration [47, 48, 49, 50]. A popular approach consists in
designing constraints/distillations that enforce the new model to exhibit a
similar behavior as the previous model. These constraints can be defined
directly on the weights [51, 52, 53, 54], the gradients [55, 56], the output
probabilities [57, 58, 59, 15], or even intermediary representations [60, 61,
62, 63].

Rehearsal Learning [17] allows to replay a limited amount of samples from
previous tasks in order to reduce forgetting. While popular for continual
image classification [58, 59, 64, 65, 66, 67, 68, 69, 64] or continual
reinforcement learning [70], this approach has been fewly explored in
continual segmentation. It can be explained in part because of the prohibitive
memory cost of the large images used in segmentation (up to $1024\times 2048$
vs $224\times 224$). Another drawback comes the nature of continual
segmentation where images at each step are partially labeled. This partial
labelling will be different for each rehearsed image, leading to a damaging
concept shift [71, 72]. Huang et al. [73] proposed after our PLOP a pseudo-
rehearsal method for segmentation that while effective, incurs an important
time overhead (up to $3\times$ longer).

Continual Semantic segmentation: Despite enormous progress in the two
aforementioned areas respectively, segmentation algorithms are mostly used in
an offline setting, while continual learning methods generally focus on image
classification. Recent works extend existing continual learning methods [57,
60] for specialized applications [11, 12, 13] and general semantic
segmentation [14]. The latter considers that the previously learned categories
are properly annotated in the images of the new dataset. This is an
unrealistic assumption that fails to consider the background shift: pixels
labeled as background at the current step are semantically ambiguous, in that
they can contain pixels from old classes (including the real semantic
background class, which is generally deciphered first) as well as pixels from
future classes. Cermelli et al. [15] propose a novel classification and
distillation losses. Both handle the background shift by summing respectively
the old logits with the background logits and the new logits with the
background. We argue that a distillation loss applied to the model output is
not strong enough for catastrophic forgetting in CSS. Furthermore, their
classification loss doesn’t preserve enough discriminative power w.r.t the old
classes when learning new classes under background shift. We introduce our
PLOP framework that solves more effectively those two aspects. Yu et al. [74]
proposed to exploit an external unlabeled dataset in order to self-training
through a pseudo-labeling loss; we show that our model while not designed with
this assumption in mind can outperform their performance. Cermelli et al. [75]
creates a novel setting of continual few-shots segmentation, we implement
their method in our setting and draw inspiration from it to further improve
PLOP. Michielli et al.[76] draws inspiration from the metric learning
litterature to conceive a model for continual segmentation that exploits
prototypes updated with an exponential moving average of the mean batch
features.

Positioning:  Contrary to previous works in continual segmentation [14, 15]
which reduced slightly forgetting through distillation of the probabilities,
we propose a stronger constraint based on global and local statistics
extracted from intermediary features. Moreover, background is often not
considered [14] or only weakly tackled [15], while we propose to eliminate it
through segmentation maps completion with pseudo-labeling. Finally, none of
the work proposed rehearsal methods for CSS, while we propose a non-trivial
based on image rehearsal and further improve it with a more data-efficient
method based with object rehearsal.

##  3 Model

Figure 1: Local POD details and the complete PLOP strategy. (a) Local POD
consists in POD embeddings compute at multiple scale. The global scale
aggregates statistics across the whole features maps while the local scale
focuses on finer details. (b) The model incrementally learns new classes (e.g.
car, dog, person). Only the current class (person) is labeled while previous
classes are folded into the background. We use the previous model
$g^{t-1}\circ f^{t-1}$ to generate pseudo-labels $\hat{\mathcal{S}}^{t-1}$
regarding the old classes to alleviate this ambiguity, and complete the labels
$\mathcal{S}^{t}$ which are then used as ground-truth in
$\mathcal{L}_{\text{pseudo}}$. The Local POD distillation is applied at
multiple levels of the features extractors $f^{t-1}$ and $f^{t}$.

The model description is organized as follows: we first details the continual
protocol and the notations. Then, we tackle the issue of catastrophic
forgetting by designing an adapted distillation loss and we allievate the
background shift by proposing a uncertainty-based pseudo-labeling. Drawing
ideas from the continual learning litterature, we propose an extention of PLOP
specialized for long-range continual training that we nickname PLOPLong.
Finally, we detail the limits of rehearsal learning in segmentation, propose a
naive adaptation to the problem, and then deliver our carefully designed
method.

###  3.1 Framework and notations

TABLE I: Notations used in this paper. Total number of tasks | $T$
---|---
Current task | $t$
Current classes | $\mathcal{C}^{t}$
Previous classes | $\mathcal{C}^{1:t-1}$
Future classes | $\mathcal{C}^{t+1:T}$
Cardinality of a set of classes | $\mathcal{N}^{t}=\operatorname{card}(\mathcal{C}^{t})-1$
Image and Segmentation maps at task $t$ |  $\mathcal{I}^{t}$ and $\mathcal{S}^{t}$
Feature extractor at task $t$ | $f^{t}(\cdot)$
Classifier at task $t$ | $g^{t}(\cdot)$
Learnable parameters at task $t$ | $\Phi^{t}$
Predicted segmentation maps | $\hat{\mathcal{S}}^{t}$
Intermediary features at level $l$ | $f^{t}_{l},\,l\in\\{1,\dots,L\\}$

In Continual Semantic Segmentation (CSS), a model learns a dataset
$\mathcal{D}$ in $t=1\dots T$ steps. At each step $t$, the model has to learn
a new set of classes $\mathcal{C}^{t}$, for which only a subset of data
$\mathcal{D}_{t}$, that consists in a set of pairs
$(\mathcal{I}^{t},\mathcal{S}^{t})$, is available. $\mathcal{I}^{t}$ denotes
an RGB input image of size $W\times H\times 3$ and $\mathcal{S}^{t}$ the
corresponding one-hot encoded ground-truth segmentation maps of size $W\times
H\times\mathcal{C}^{t}$. If a pixel of $\mathcal{I}^{t}$ belongs to either one
of the previous $\mathcal{C}^{1:t-1}$ or future classes $\mathcal{C}^{t+1:T}$,
it is not labeled as such in $\mathcal{S}^{t}$ and shall instead be considered
as belonging to the background class, denoted $c_{\text{bg}}$. However, at
test time, a model at step $t$ must be able to discriminate between all the
classes that have been seen so far, i.e. $\mathcal{C}^{1:t}$. This leads us to
identify two major pitafalls in CSS: the first one, inherited from continual
learning, is catastrophic forgetting [17, 18]. It suggests that a network will
completely forget the old classes $\mathcal{C}^{1:t-1}$ when learning the new
ones $\mathcal{C}^{t}$. Furthermore, catastrophic forgetting is aggravated by
the second pitfall, specific to CSS, that we call background shift: at step
$t$, the pixels labeled as background are indeed ambiguous, as they may
contain either old (including the real background class, predicted in
$\mathcal{C}^{1}$) or future classes.

We define our model at step $t$ as a composition of a feature extractor
$f^{t}(\cdot)$ (a ResNet 101 [77] backbone and a classifier $g^{t}(\cdot)$.
The output predicted segmentation map can then be written
$\hat{\mathcal{S}}^{t}=g^{t}\circ f^{t}(\mathcal{I})$. We denote the
intermediate features at each layer of the feature extractor
$f_{l}^{t}(\cdot)\,,\,l\in\\{1,\dots L\\}$. Finally, we denote the set of
learnable parameters of $f^{t}$ and $g^{t}$ as $\Theta^{t}$. The notations
introduced in this paper are summarized in Table I.

###  3.2 Overcoming catastrophic forgetting in CSS with local distillation

In this section, we propose to tackle the issue of catastrophic forgetting in
continual learning in general and in CSS in particular. An effective method
for doing so involves setting constraints between the old ($g^{t-1}\circ
f^{t-1}$) and current ($g^{t}\circ f^{t}$) models. These constraints aim at
enforcing a similar behavior between both models and in turn reduce the loss
of performance on old classes. A common such contraint is based on applying
knowledge distillation [78, 57] between the predicted probabilities of both
models. When applied to CSS, such distillation loss must be carefully balanced
to find a good trade-off between rigidity (i.e. too strong constraints,
resulting in not being able to learn new classes) and plasticity (i.e.
enforcing loose constraints, which can lead to catastrophic forgetting of the
old classes).

In previous work [63] we designed POD, short for Pooled Output Distillation.
Rather than solely constraining the output model probabilities, POD enforces
consistenty between intermediary statistics of both models. In practice, for a
feature map $\mathbf{x}$, we define a POD embedding $\Phi(\cdot)$ as:

$\Phi(\mathbf{x})=\left[\frac{1}{W}\sum_{w=1}^{W}\mathbf{x}[:,w,:]\bigg{\|}\frac{1}{H}\sum_{h=1}^{H}\mathbf{x}[h,:,:]\right]\in\mathbb{R}^{(H+W)\times
C}\,,$ (1)

where $[\cdot\,\|\,\cdot]$ denotes concatenation over the channel axis. The
POD embedding is thus computed as the concatenation of the $H\times C$ width-
pooled slices and the $W\times C$ height-pooled slices of $\mathbf{x}$ and
captures long-range statistics across the whole features maps. The POD
distillation loss then consists in minimizing the $\mathcal{L}_{2}$ distance
between POD embeddings computed at several layers $l\in\\{1,\dots,L\\}$, w.r.t
the current model parameters $\Theta^{t}$:

$\mathcal{L}_{\text{pod}}(\Theta^{t})=\frac{1}{L}\sum_{l=1}^{L}\left\|\Phi(f^{t}_{l}(\mathcal{I}))-\Phi(f^{t-1}_{l}(\mathcal{I}))\right\|^{2}\,.$
(2)

POD yields state-of-the-art results in continual learning for image
classification, especially when large numbers of tasks are considered, a case
where the aforementioned plasticity-rigidity trade-off becomes even more
crucial. Another interest arises in the context of CSS: the long-range
statistics computed across an entire axis (horizontal or vertical) which
reminds recent work on attention for segmentation [10, 79, 80] which aim to
enlarge the receptive field through global attention/statistics [10].

In the frame of classification, it is, to a certain extent, necessary to
discard spatial information through global pooling. However, conversely,
semantic segmentation requires preservation of both long-range and short-range
statistics, making a distillation loss such as POD suboptimal for that
purpose.

Following this reflexion, and inspired by the multi-scale litterature [81,
82], we design a distillation loss, called Local POD that retain the long-
range spatial statistics while also preserving the local information. The
proposed Local POD consists in computing the width and height statistics at
different scales $\\{1/2^{s}\\}_{s=0\dots S}$, as illustrated in Fig. 1 (a).
At a given level $l$ of the features extractor, $s^{2}$ POD embeddings are
computed per scale $s$ and concatenated:

$\Psi^{s}(\mathbf{x})=\left[\Phi(\mathbf{x}^{s}_{0,0})\|\dots\|\Phi(\mathbf{x}^{s}_{s-1,s-1})\right]\in\mathbb{R}^{S\times(H+W)\times
C}\,,$ (3)

where $\forall i=0\dots s-1$, $\forall j=0\dots s-1$,
$\mathbf{x}^{s}_{i,j}=\mathbf{x}[iH/s:(i+1)H/s,jW/s:(j+1)W/s,:]$ is a sub-
region of the embedding tensor $\mathbf{x}$ of size $W/s\times H/s$. We then
concatenate (along the channel axis) the Local POD embeddings
$\Psi^{s}(\mathbf{x})$ of each scale $s$ to form the final embedding:

$\Psi(\mathbf{x})=\left[\Psi^{1}(\mathbf{x})\|\dots\|\Psi^{S}(\mathbf{x})\right]\in\mathbb{R}^{(1+\dots+S)\times(H+W)\times
C}\,.$ (4)

Similarly to POD, we compute Local POD embeddings for every layer
$l\in\\{1,\dots,L\\}$ of both the old and current models. The resulting loss
is thus:

$\mathcal{L}_{\scriptstyle\text{LocalPod}}(\Theta^{t})=\frac{1}{L}\sum_{l=1}^{L}\left\|{\Psi}(f^{t}_{l}(I))-{\Psi}(f^{t-1}_{l}(I))\right\|^{2}\,.$
(5)

Thus, notice that the first scale of Local POD ($1/2^{0}$) is similar to the
original POD and models long-range dependencies across the entire image. The
subsequent scales ($s=1/2^{1},1/2^{2}\dots$), enforce short-range
dependencies. Thus, the proposed Local POD tackles the problem of catastrophic
forgetting by modeling and preserving long and short-range statistics between
the old and current models, throughout the CSS steps.

###  3.3 Pseudo-labeling to fix background shift

In addition to catastrophic forgetting, a successful CSS approach shall handle
the background shift problem, thus shall take into account the ambiguity of
pixels labelled as background at each step. We propose a pseudo-labeling
strategy that “completes” the ambiguous background labels. Pseudo-labeling
[83] is commonly used in domain adaption for semantic segmentation [84, 85,
86, 87] where a model is trained to match both the labels of a source dataset
and the pseudo-labels (usually obtained using the same predictive model, in a
self-training fashion) of an unlabeled target dataset. In this case, the
knowledge acquired on the source dataset helps the model to generate labels
for the target dataset. In the frame of CSS, at each step, we use the
predictions of the old model to decipher previously seen classes among the
ambiguous background pixels, as illustrated in Fig. 1. The pseudo-labeling
rely on the previous model which can be uncertain for some pixels due to
inherent bias to the optimization and because of the forgetting. Therefore in
order to avoid propagate errors through incorrect pseudo-labels, we filter out
the most uncertain ones based on a adaptive entropy-based threshold.

Formally, let $\mathcal{N}^{t}=\operatorname{card}(\mathcal{C}^{t})-1$ the
cardinality of the current classes excluding the background class. Let
$\hat{\mathcal{S}}^{t}\in\mathbb{R}^{W,H,1+\mathcal{N}^{1}+\dots+\mathcal{N}^{t}}$
denotes the predictions of the current model (which includes the real
background class, all the old classes as well as the current ones). We define
$\tilde{\mathcal{S}}^{t}\in\mathbb{R}^{W,H,1+\mathcal{N}^{1}+\dots+\mathcal{N}^{t}}$
the target as step $t$, computed using the one-hot ground-truth segmentation
map $\mathcal{S}^{t}\in\mathbb{N}^{W,H,1+\mathcal{N}^{t}}$ at step $t$ as well
as pseudo-labels extracted using the old model predictions
$\hat{\mathcal{S}}^{t-1}\in\mathbb{R}^{W,H,1+\mathcal{N}^{1}+\dots+\mathcal{N}^{t-1}}$
as follows:

$\footnotesize\tilde{\mathcal{S}}^{t}\left(w,h,c\right)=\mkern-5.0mu\left\\{\begin{array}[]{ll}\mkern-10.0mu1\mkern-27.0mu&\text{
if }S^{t}(w,h,c_{bg})=0\text{ and
}c=\operatorname*{argmax}\limits_{c^{\prime}\in\mathcal{C}^{t}}\mathcal{S}^{t}(w,h,c^{\prime})\\\
\mkern-10.0mu1\mkern-27.0mu&\text{ if }\mathcal{S}^{t}(w,h,c_{bg})=1\text{ and
}c=\mkern-8.0mu\operatorname*{argmax}\limits_{c^{\prime}\in\mathcal{C}^{1:t-1}}\mkern-6.0mu\hat{\mathcal{S}}^{t-1}(w,h,c^{\prime})\\\
\mkern-10.0mu0\mkern-27.0mu&\text{ otherwise }\\\ \end{array}\right.$ (6)

In other words, in the case of non-background pixels we simply copy the ground
truth label. Otherwise, we use the class predicted by the old model
$g^{t-1}(f^{t-1}(\cdot))$. This pseudo-label strategy allows to assign each
pixel labelled as background his real semantic label if this pixel belongs to
any of the old classes. However, pseudo-labeling all background pixels can be
unproductive, e.g. on uncertain pixels where the old model is likely to fail.
Therefore we only keep pseudo-labels where the old model is deemed “confident”
enough. Eq. 6 is modified to take into account this uncertainty:

$\scriptsize\tilde{\mathcal{S}}^{t}\left(w,h,c\right)\mkern-5.0mu=\mkern-5.0mu\left\\{\begin{array}[]{ll}\mkern-14.0mu1\mkern-27.0mu&\mkern-8.0mu\text{
if }\mathcal{S}^{t}\mkern-4.0mu(w,h,c_{bg})\mkern-4.0mu=\mkern-4.0mu0\text{
and
}c\mkern-4.0mu=\mkern-4.0mu\operatorname*{argmax}\limits_{c^{\prime}\in\mathcal{C}^{t}}\mathcal{S}^{t}\mkern-4.0mu(w,h,c^{\prime})\\\
\mkern-14.0mu1\mkern-27.0mu&\mkern-8.0mu\text{ if
}\mathcal{S}^{t}\mkern-4.0mu(w,h,c_{bg})\mkern-4.0mu=\mkern-4.0mu1\text{ and
}c\mkern-4.0mu=\mkern-12.0mu\operatorname*{argmax}\limits_{c^{\prime}\in\mathcal{C}^{1:t-1}}\mkern-6.0mu\hat{\mathcal{S}}^{t-1}\mkern-4.0mu(w,h,c^{\prime})\text{
and }u\mkern-4.0mu<\mkern-4.0mu\tau_{c}\\\
\mkern-14.0mu0\mkern-27.0mu&\mkern-4.0mu\,\text{ otherwise\,, }\\\
\end{array}\right.$ (7)

By notation abuse, $u$ is function $u(\mathcal{S}^{t}(w,h))$ that measures the
uncertainty of the current model given a pixel $\mathcal{I}(w,h)$. $\tau_{c}$
denotes a class-specific uncertainty threshold. Hence, in the case where the
old model is uncertain ($u\geq\tau_{c}$) about some pixels, they will be
ignored in the final classification loss. Our framework is agnostic to the
type of uncertainty used, but in practice we define it as the entropy.
Therefore we use for $u$ the current model’s per-pixel entropy
$u(\mathcal{S}^{t}(w,h))=-\sum_{c\in
C^{1:t}}\mathcal{S}^{t}(w,h,c)\log\mathcal{S}^{t}(w,h,c)$. Likewise, the
class-specific threshold $\tau_{c}$ is computed from the median entropy of the
old model over all pixels of $\mathcal{D}^{t}$ predicted the class $c$ for all
$c\in\mathcal{C}^{1:t-1}$ as proposed in [87]. Consequently, the cross-entropy
loss with pseudo-labeling of the old classes can be written as:

$\mathcal{L}_{\text{pseudo}}(\Theta^{t})=-\frac{\nu}{WH}\sum_{w,h}^{W,H}\sum_{c\in\mathcal{C}^{t}}\tilde{\mathcal{S}}\left(w,h,c\right)\log\hat{\mathcal{S}}^{t}\left(w,h,c\right)\,.$
(8)

We reduce the normalization factor $WH$ by as much discarded pixels. To avoid
giving disproportional importance to the pixels belonging to new classes
(which aren’t discarded), we introduce in Eq. 8 an daptive factor $\nu$, which
is the ratio of accepted old classes pixels over the total number of such
pixels. i.e. if most if most of the image is uncertain, the overall importance
of the image relative to other images in the batch is reduced. The overall
behavior of our pseudo-labeling is illustrated in Fig. 1.

We call our final model PLOP (standing for Pseudo-labeling and LOcal Pod).
PLOP’s final loss is a weighted combination of Eq. 5 and Eq. 8:

$\mathcal{L}(\Theta^{t})=\underbrace{\mathcal{L}_{\text{pseudo}}(\Theta^{t})}_{\text{classification}}+\lambda\underbrace{\mathcal{L}_{\text{localPod}}(\Theta^{t})}_{\text{distillation}}\,,$
(9)

with $\lambda$ an hyperparameter. PLOP, while already very competitive, can
face difficulties when dealing with long continual settings, i.e. for which
the number of steps grows larger. For this reason, we propose PLOPLong, an
extension of PLOP for dealing with such cases.

###  3.4 PLOPLong: a specialization for long settings

A first axis of improvement consists in correcting the forgetting located at
the classifier level. Indeed, the classifier weights tend to be specialized
for the last classes to the detriment of older classes [60]. Several means
exist to correct this bias [47, 49, 48, 22] and we choose the cosine
normalization. In practice, we replace the classifier by a cosine classifier
[22], where the final inner product is discarded in favor of a cosine
similarity. By doing so, all class weights –both old and new– have a constant
magnitude of 1, which reduces drastically the bias towards new classes. The
classifier $g^{t}$ is in segmentation a pointwise ($1\times 1$ kernel)
convolution which doesn’t alter the spatial organisation but maps the $ch$
features channels to $\mathcal{C}^{1:t}$ channels, one per class to predict.
This pointwise convolution can be seen as a fully-connected layer that is
apply independentely to each pixel. Therefore, the classifier $g^{t}$ has
$\\{\boldsymbol{\theta}_{c}^{t}\in\mathbb{R}^{ch}|\forall
c\in\mathcal{C}^{1:t}\\}$. The cosine normalization can then be express as:

$\hat{\mathcal{S}}(w,h,c)=\frac{\alpha\langle\boldsymbol{\theta}^{t}_{c},\mathbf{h}(w,h)\rangle}{\|\boldsymbol{\theta}^{t}_{c}\|^{2}\|\mathbf{h}(w,h)\|^{2}}\,$
(10)

with $\alpha$ a learned scalar parameter initialized to 1 and helping the
convergence, and $\mathbf{h}\in\mathbb{R}^{W\times H\times ch}$ the final
features embedding before the classifier. First used in continual learning by
Hou et al. [60], this classifier has more recently be adopted by continual
few-shot segmentation [75] or multi-modes continual learning [63]. New cosine
classifier weights for new classifier can then be initialized with weight
imprinting [88] as recently adapted for segmentation by Cermelli et al. [75].

As second improvement we propose to exploit the cosine normalization for
intermediary features: the comparison between the Local POD embeddings of the
previous model $f^{t-1}$ with the current $f^{t}$ can be too constrained. We
relax the constraints by imposing not a low L2 distance between both
embeddings, but rather a high cosine similarity. We simply need to alter Eq. 5
by replacing the $\Phi(\cdot)$ operator by
$\bar{\Phi}(\cdot)=\nicefrac{{\Phi(\cdot)}}{{|\Phi(\cdot)\|^{2}}}$. However
note that the features from level $l$ given to level $l+1$ are not
normalization. The normalization only happens for the Local POD embeddings.

Figure 2: Our Object Rehearsal strategy. In task $t-1$, we select from
$\\{\mathcal{I}^{t-1},\dots\\}$ a limited amount of objects (here bus, bird,
and dog), which will be then mixed in the images $\\{\mathcal{I}^{t},\dots\\}$
from the current task $t$; after pasting, the other present objects are
erased. Finally the current model $g^{t}\circ f^{t}$ will be given the
concatenation of the the original images and the augmented images
$\\{\mathcal{I}^{\prime\prime t},\dots\\}$.

The third axis of improvement focuses the Batch Normalization [89], an almost
ubiquitous component to deep computer vision models. It normalizes internal
representation with the batch mean $\mu_{\mathbb{B}}$ and batch standard
deviation $\sigma_{\mathbb{B}}$ (resp. running mean $\mu$ and running std
$\sigma$) during training (resp. testing). Formally for a all batch
normalization layer taking an input $\mathbf{x}$ and producing an output
$\mathbf{y}$:

$\mathbf{y}=\frac{\mathbf{x}-\mu_{\mathbb{B}}}{\sigma_{\mathbb{B}}}\cdot\gamma+\beta$
(11)

with $\gamma$ and $\beta$ two learned parameters. The drawback of this
normalization layeris its assumption that the data is sampled i.i.d. which is
not the case in continual segmentation where multiple shifts [71, 72, 90],
happen in the training tasks, different from the testing tasks. Lomonaco et
al. [91] and Cermelli et al. [75] proposed to replace the batch normalization
by the batch renormalization [21]:

$\mathbf{y}=\frac{\mathbf{x}-\mu_{\mathbb{B}}}{\sigma_{\mathbb{B}}}\cdot
r+d,\,\text{where}\,r=\frac{\sigma_{\mathbb{B}}}{\sigma},\,d=\frac{\mu_{\mathbb{B}}-\mu}{\sigma}\,$
(12)

Intuitively, batch renormalization avoids the discrepancy between training and
testing of the batch normalization. Furthermore, following [75], we freeze
during training the statistics ($\mu$ and $\sigma$) after the first task to
avoid harmful statistics drifts.

All three improvements further increase performance for long series of tasks:
(1) Cosine classifer reduces the increased bias between recent and old
classes. (2) A cosine-based Local POD relax the constraints in order to learn
correctly the bigger number of new classes. (3) Frozen BatchReNorm reduces the
inherent drift of statistics that grew larger as the number of tasks increase.

###  3.5 Rehearsing previous data for CSS

Neither the proposed PLOP or PLOPLong did make use of previously seen data
$\\{\mathcal{D}^{1},\dots,\mathcal{D}^{t-1}\\}$ when considering step $t$. In
this section, we explore how to further improve CSS performance if a model is
now allowed to rehearse a limited amount of previous data. We first consider a
traditional approach, namely Image Rehearsal, that we adapt to the CSS
setting, and then propose a more efficient method, that we call Object
Rehearsal.

####  3.5.1 Image Rehearsal

We first consider rehearsing a limited amount of images from previous tasks
during the current task. Contrary to image classification, in CSS, images can
have multiple labels if several objects (e.g. car, sky) are present in the
image. We therefore propose to select $M$ images for each class $c\in C^{t}$,
resulting in $M\times|C^{t}|$. We ensure that the selected images are unique,
thus because multiple classes can co-exist in the same image, the resulting
amount of sampled classes may be above $M\times|C^{t}|$. We select the images
with a simple random selection, which has been proved to generally be as good
as more elaborate methods [59]. The memory footprint will grow until all tasks
but the last are seen, resulting in a total amount of $M\times|C^{1:T-1}|$
stored images. Note however that a naive rehearsal will only bring marginal
gains because the stored ground truth segmentation maps are partially labeled.
Indeed, images stored from a previous task $t_{o}$ ($t_{o}<t$, with $t$ the
current task) will only have the classes $C^{t_{o}}$ labeled, and the other
classes folded into the background. Here again, pseudo-labeling (see Sec. 3.3)
can complete the segmentation maps with both even older classes
$C^{1:t_{o}-1}$ and newer classes $C^{t_{o}+1:t}$.

####  3.5.2 Object Rehearsal

The main drawback of image rehearsal is that CSS images are usually large
($512\times 512$ to $1024\times 2048$) yet they are sparsely informative, as a
significant part of the images consists in background pixels [92] (e.g. 63% of
Pascal-VOC [25] pixels) or belongs to a majoritary class (e.g. 32% of
Cityscapes [26] pixels are roads).

To address both problems, instead of storing whole images from the previous
tasks $\\{1,...,t-1\\}$, we propose to store an informative portion that we
will mix with the images of the current task $t$. Image mixing is popular for
classification [93, 94, 95, 96, 97, 98] yet, to the best of our knowledge,
sees limited use for semantic segmentation [99, 100, 101, 102, 103], and has
never been considered to design memory-efficient rehearsal learning systems.
Formally, given an image $I$ and the corresponding ground truth segmentation
maps $S^{t}$, we define a binary mask $\Pi_{c}$ such that $\forall c\in
C^{t}$:

$O_{c}=I\odot\Pi_{c}\,\text{where}\,\Pi_{c}(x,y)=\begin{cases}1&\text{ if
}S^{t}(x,y)=c\\\ 0&\text{ otherwise }\end{cases},$ (13)

with $O_{c}$ the selected object for class $c$. By nature, this patch is
extremely sparse and can be efficiently stored on disk by modern compression
algorithms [104]. The total memory footpring at task t is thus
$M\times|C^{1:t}|$ objects.

Then, when learning a task $t>1$, the model will learn on both the task
dataset $\cup_{i}^{n}(I^{t}_{i},S^{t}_{i})$ and the object memory $\cup_{c\in
C^{1:t-1}}\cup_{i}^{M}O_{c,i}$. The latter will augment the former through
object pasting. We augment each object by applying an affine transformation
matrix [99]:

$\mathbf{T}=\left[\begin{array}[]{ccc}z\cos\alpha&z\sin\alpha&0\\\
-z\sin\alpha&z\cos\alpha&0\\\ 0&0&1\end{array}\right]\,,$ (14)

with $z$ a zoom factor, and $\alpha$ an in-plane rotation angle. Note that we
don’t translate the object as its original position is often a good prior:
indeed, objects are usually located at the same location (e.g. pedestrian on
the left and right sidewalk, car on the middle road, etc.) We abuse the
notation by denoting $T(.)$ the application of this transformation matrix.
Once the data augmentation in done on the object (and its mask), we paste it
on an image $I^{t}$ that refers to the current task:

${I^{\prime}}^{t}=I^{t}\odot(\mathbf{1}-T(\Pi_{c}))+T(O_{c})\odot
T(\Pi_{c})\,.$ (15)

The pasting can result in local incoherence where the pixel of a cow is pasted
on top or next to the pixel of a television. Naively the object borders can be
smoothed into the image with a Gaussian filter. Unfortunatly it results in
inprecise countours which are important in segmentation [105].

This approach has already been envisoned as a form of data augmentation in
semantic segmentation, although the gains were small and to avoid the noise
induced by this pasting, the batch size is prohibitively large (up to 512)
[103]. We propose to reduce interference between the pasted object and the
destination images by selective erasing of the surrounding pixels. Indeed,
given a binary matrix $\Xi(\Pi,S)$ of the same dimension as $I$ and $\Pi$:

${I^{\prime\prime}}^{t}={I^{\prime}}^{t}\odot(\mathbf{1}-\Xi)+\kappa\odot\Xi\,.$
(16)

We replace the pixels erased according to the mask $\Xi$ with a RGB color
$\kappa\in\mathbb{R}^{3}$. This RGB vector could be chosen through in-painting
[99] or be random noise, but in practice we choose a constant color gray. We
also update accordingly the segmentation maps likewise:

${S^{\prime\prime}}^{t}={S^{\prime\prime}}^{t}\odot(\mathbf{1}-\Xi)+255\odot\Xi\,,$
(17)

where 255 is the default value used in segmentation to ignore some pixel
labels which won’t be counted in the classification loss Eq. 8. We illustrate
our rehearsal strategy in Fig. 2.

TABLE II: Description of the three datasets considered in this paper. For datasets without explicit background class, one is created based on unlabeled pixels. Dataset | # classes | Background? | # train | # test | Image size
---|---|---|---|---|---
Pascal-VOC [25] | 20 | ✓ | 10k | 1.5k | $512\times 512$
Cityscapes [26] | 19 | ✗ | 3k | 0.5k | $512\times 1024$
ADE20k [8] | 150 | ✗ | 20k | 2.0k | $512\times 512$
TABLE III: Description of the 12 different benchmarks evaluated in this paper. For some of these, each task brings new classes while, for others, it comes with new domains. Dataset | Setting | Mode | # tasks | # base classes | # classes / inc. task
---|---|---|---|---|---
Pascal-VOC | 19-1 | class | 2 | 19 | 1
15-5 | class | 2 | 15 | 5
15-1 | class | 6 | 15 | 1
10-1 | class | 11 | 10 | 1
Cityscapes | 14-1 | class | 6 | 14 | 1
ADE20k | 100-50 | class | 2 | 100 | 50
50-50 | class | 2 | 50 | 50
100-10 | class | 6 | 100 | 10
100-5 | class | 11 | 100 | 5
TABLE IV: Continual Semantic Segmentation results on Pascal-VOC 2012 in Mean IoU (%). $\dagger$: results excerpted from [15], $\diamond$ from [76]. Other results comes from re-implementation. |  19-1 (2 tasks) |  15-5 (2 tasks) |  15-1 (6 tasks)
---|---|---|---
Method | 0-19 | 20 | all | avg | 0-15 | 16-20 | all | avg | 0-15 | 16-20 | all | avg
$\text{Fine Tuning}^{\dagger}$ |  6.80 | 12.90 |  7.10 |  |  2.10 | 33.10 |  9.80 |  |  0.20 |  1.80 |  0.60 |
$\text{PI}^{\dagger}$ [54] |  7.50 | 14.00 |  7.80 |  |  1.60 | 33.30 |  9.50 |  |  0.00 |  1.80 |  0.50 |
$\text{EWC}^{\dagger}$ [51] | 26.90 | 14.00 | 26.30 |  | 24.30 | 35.50 | 27.10 |  |  0.30 |  4.30 |  1.30 |
$\text{RW}^{\dagger}$ [53] | 23.30 | 14.20 | 22.90 |  | 16.60 | 34.90 | 21.20 |  |  0.00 |  5.20 |  1.30 |
$\text{LwF}^{\dagger}$ [57] | 51.20 |  8.50 | 49.10 |  | 58.90 | 36.60 | 53.30 |  |  1.00 |  3.90 |  1.80 |
$\text{LwF-MC}^{\dagger}$ [58] | 64.40 | 13.30 | 61.90 |  | 58.10 | 35.00 | 52.30 |  |  6.40 |  8.40 |  6.90 |
$\text{ILT}^{\dagger}$ [14] | 67.10 | 12.30 | 64.40 |  | 66.30 | 40.60 | 59.90 |  |  4.90 |  7.80 |  5.70 |
ILT [14] | 67.75 | 10.88 | 65.05 | 71.23 | 67.08 | 39.23 | 60.45 | 70.37 |  8.75 |  7.99 |  8.56 | 40.16
$\text{MiB}^{\dagger}$ [15] | 70.20 | 22.10 | 67.80 |  | 75.50 | 49.40 | 69.00 |  | 35.10 | 13.50 | 29.70 |
MiB [15] | 71.43 | 23.59 | 69.15 | 73.28 | 76.37 | 49.97 | 70.08 | 75.12 | 34.22 | 13.50 | 29.29 | 54.19
$\text{SDR}^{\diamond}$ [76] | 71.30 | 23.40 | 69.0 |  | 76.30 | 50.20 | 70.10 |  | 47.30 | 14.70 | 39.50 |
GIFS [75] | 57.88 | 32.82 | 56.69 | 67.05 | 23.61 | 16.43 | 21.90 | 50.97 | 59.36 | 13.89 | 48.53 | 61.43
PLOP | 75.35 | 37.35 | 73.54 | 75.47 | 75.73 | 51.71 | 70.09 | 75.19 | 65.12 | 21.11 | 54.64 | 67.21
PLOPLong | 74.75 | 39.68 | 73.08 | 74.32 | 75.95 | 48.31 | 69.37 | 73.58 | 72.00 | 26.66 | 61.20 | 70.02

##  4 Experiments

###  4.1 Datasets, Protocols, and Baselines

To ensure fair comparisons with state-of-the-art approaches, we follow the
experimental setup of [15] for datasets, protocol, metrics, and baseline
implementations. ALthough, we also propose to evaluate on new datasets and on
more challenging protocols. Furthermore, we explore in advanced experiments,
for the first time, how rehearsal can improve performance in CSS.

Datasets:  We evaluate our model on three datasets, summarized in Table II:
Pascal-VOC [25], Cityscapes [26] and ADE20k [8]. VOC contains 20 classes,
10,582 training images, and 1,449 testing images. Cityscapes contains 2975 and
500 images for train and test, respectively. Those images represent 19 classes
and were taken from 21 different cities. ADE20k has 150 classes, 20,210
training images, and 2,000 testing images. All ablations and hyperparameters
tuning were done on a validation subset of the training set made of 20% of the
images. For all datasets, we use random resize and crop augmentation (scale
from 80% to 110%), as well as random horizontal flip during training time. The
final image size for Pascal-VOC and ADE20k is $512\times 512$ while it is
$512\times 1024$ for Cityscapes.

CSS protocols:  [15] introduced the Overlapped setting where only the current
classes are labeled vs. a background class $\mathcal{C}^{t}$. Moreover, pixels
can belong to any classes
$\mathcal{C}^{1:t-1}\cup\mathcal{C}^{t}\cup\mathcal{C}^{t+1:T}$ (old, current,
and future). Those two constraints make this setting both challenging and
realistic, as in a real setting there isn’t any oracle method to exclude
future classes from the background. In all our experiments, we respect the
Overlapped setting. While the training images are only labeled for the current
classes, the testing images are labeled for all seen classes. We evaluate
several CSS protocols for each dataset, e.g. on VOC 19-1, 15-5, and 15-1
respectively consists in learning 19 then 1 class ($T=2$ steps), 15 then 5
classes ($2$ steps), and 15 classes followed by five times 1 class ($6$
steps). The last setting is the most challenging due to its higher number of
steps. Similarly, on Cityscapes 14-1 means 14 followed by five times 1 class
($6$ steps) and on ADE 100-50 means 100 followed by 50 classes ($2$ steps). We
provide a summary of all these settings in Table III.

Metrics:  we compare the different models using traditional mean Intersection
over Union (mIoU). Specifically, we compute mIoU after the last step $T$ for
the initial classes $\mathcal{C}^{1}$, for the incremented classes
$\mathcal{C}^{2:T}$, and for all classes $\mathcal{C}^{1:T}$ (all). These
metrics respectively reflect the robustness to catastrophic forgetting (the
model rigidity), the capacity to learn new classes (plasticity), as well as
its overall performance (trade-of between both). We also introduce a novel avg
metric (short for average), which measures the average of mIoU scores measured
step after step, integrating performance over the whole continual learning
process. We stress that all four metrics are important and we shouldn’t
disregard one for the other: a model suffering no forgetting but not able to
learn anything new is useless.

Baselines:  We benchmark our model against the latest state-of-the-arts CSS
methods ILT [14], MiB [15], GIFS [75], and SDR [76]. Note that while GIFS was
created by Cermelli et al. [75] for continual few-shots segmentation, we adapt
it for the more general task of CSS. Unless stated otherwise, all the results
are excerpted from the corresponding papers. We also evaluate general
continual models based on weight constraints (PI [54], EWC [51], and RW [53])
and knowledge distillation (LwF [57] and LwF-MC [58]). Moreover, unless
explicitely stated otherwise, all the models (ours included), do not use
rehearsal learning.

Implementation Details: As in [15], we use a Deeplab-V3 [106] architecture
with a ResNet-101 [77] backbone pretrained on ImageNet [107] for all
experiments and models but SDR [76] which used a Deeplab-V3+ [108]. For all
datasets, we set a maximum threshold for the uncertainty measure of Eq. 7 to
$\tau=1e-3$. We train our model for 30, 60, and 30 epochs per CSS step on
Pascal-VOC, ADE20k, and Cityscapes, respectively, with an initial learning
rate of $1e-2$ for the first CSS step, and $1e-3$ for all the following ones.
Note that for Cityscapes, the first step is longer with 50 epochs. We reduce
the learning rate exponentially with a decay rate of $9e-1$. We use SGD
optimizer with $9e-1$ Nesterov momentum. The Local POD weighting
hyperparameter $\lambda$ is set to $1e-2$ and $5e-4$ for intermediate feature
maps and logits, respectively. Moreover, we multiply this factor by the
adaptive weighting $\sqrt{\nicefrac{{|C^{1:t}|}}{{|C^{t}|}}}$ introduced by
[60] that increases the strength of the distillation the further we are into
the continual process. For all feature maps, Local POD is applied before ReLU,
with squared pixel values, as in [109, 63]. We use 3 scales for Local POD:
$1$, $\nicefrac{{1}}{{2}}$, and $\nicefrac{{1}}{{4}}$, as adding more scales
experimentally brought diminishing returns. For PLOPLong only, we L2-normalize
all POD embeddings before distilling them as also done on [63]. Furthermore
for PLOPLong, the gradient norm-clipping is set at $1.0$ for Pascal-VOC and
$2.0$ for Cityscapes. We use a batch size of 24 distributed on two 12Go Titan
Xp GPUs. Contrary to many continual models, we don’t have access to any task
id in inference, therefore our setting/strategy has to predict a class among
the set of all seen classes.

###  4.2 Quantitative Evaluation

Figure 3: mIoU evolution on Pascal-VOC 2012 15-1. While MiB’s mIoU quickly
deteriorates, PLOP and PLOPLong’s mIoU remains high, due to improved
resilience to catastrophic forgetting.

####  4.2.1 Pascal VOC 2012

Table IV shows quantitative experiments on VOC 19-1, 15-5, and 15-1. both the
proposed PLOP and PLOPLong outperform state-of-the-art approaches, MiB [15],
SDR [76], and GIFS [75] on all evaluated settings by a significant margin. For
instance, on 19-1, the forgetting of old classes (1-19) is reduced by 4.39
percentage points (p.p ) while performance on new classes is greatly improved
(+13.76 p.p ), as compared to the best performing method so far, MIB [15]. On
15-5, our model is on par with our re-implementation of MiB, and surpasses the
original paper scores [15] by 1 p.p . On the most challenging 15-1 setting,
general continual models (EWC and LwF-MC) and ILT all have very low mIoU.
While MiB shows significant improvements, PLOP still outperforms it by a wide
margin. Furthermore, PLOP also significantly outperform the best performing
approach on this setting, GIFS [75] (+6.11 p.p ). Furthermore, mIoU for the
joint model is $77.40\%$, thus PLOP narrows the gap between CSS and joint
learning on every CSS scenario. The average mIoU is also improved (+5.78 p.p )
compared to GIFS, indicating that each CSS step benefits from the improvements
related to our method. This is echoed by Fig. 3, which shows that while mIoU
for both ILT and MiB deteriorates after only a handful of steps, PLOP ’s mIoU
remains very high throughout, indicating improved resilience to catastrophic
forgetting and background shift. Last but not least, while PLOP performs
better than PLOPLong on short setups (e.g. 19-1, 15-5), PLOPLong performs
better, however, on longer CSS benchmarks such as 15-1 (+2.81 p.p ).

TABLE V: Continual Semantic Segmentation results on Cityscapes 14-1 in Mean IoU (%). |  14-1 (6 tasks)
---|---
Method |  1-14 |  15-19 | all | avg
MiB [15] |  55.11 |  12.91 | 44.56 | 49.76
PLOP |  56.59 |  13.07 | 45.71 | 51.28
PLOPLong |  58.60 |  15.04 | 47.71 | 54.31
TABLE VI: Continual Semantic Segmentation results on ADE20k in Mean IoU (%). |  100-50 (2 tasks) |  50-50 (3 tasks) |  100-10 (6 tasks)
---|---|---|---
Method | 0-100 | 101-150 | all | avg | 0-50 | 51-150 | all | avg | 0-100 | 101-150 | all | avg
ILT [14] | 18.29 | 14.40 | 17.00 | 29.42 |  3.53 | 12.85 |  9.70 | 30.12 |  0.11 |  3.06 |  1.09 | 12.56
MiB [15] | 40.52 | 17.17 | 32.79 | 37.31 | 45.57 | 21.01 | 29.31 | 38.98 | 38.21 | 11.12 | 29.24 | 35.12
PLOP | 41.87 | 14.89 | 32.94 | 37.39 | 48.83 | 20.99 | 30.40 | 39.42 | 40.48 | 13.61 | 31.59 | 36.64

####  4.2.2 Cityscapes

We also validate our method on Cityscapes in Table V. In order to design a
setting similar to Pascal-VOC 15-1 (6 tasks), we design the 14-1 setting.
Also, we simulate a background class by folding together the unlabeled
classes. Here again, PLOP performs slightly better than MIB (+2.48 p.p on the
old classes, +0.16 p.p on the new ones, +1.15 p.p on average). Moreover,
PLOPLong performs significantly better (+3.59 p.p on the old classes, +2.13
p.p on the new ones, +3.15 p.p on average), indicating better robustness to
both catastrophic forgetting and background shift, especially when considering
long continual learning setups. To more precisely assess this phenomenon, we
investigate the performance of both methods when dealing with longer task
learning sequences.

####  4.2.3 ADE20k

Table VI shows experiments on ADE 100-50, 100-10, and 50-50. This dataset is
notoriously hard, as the joint model baseline mIoU is only 38.90%. ILT has
poor performance in all three scenarios. PLOP shows comparable performance
with MiB on the short setting 100-50 (only 2 tasks), improves by 1.09 p.p on
the medium setting 50-50 (3 tasks), and significantly outperforms MiB with a
wider margin of 2.35 p.p on the long setting 100-10 (6 tasks). In addition to
being better on all settings, PLOP showcased an increased performance gain on
longer CSS (e.g. 100-10) scenarios, due to increased robustness to
catastrophic forgetting and background shift.

TABLE VII: Mean IoU on Pascal-VOC 2012 10-1. |  VOC 10-1 (11 tasks)
---|---
Method | 0-10 | 11-20 | all | avg
ILT [14] |  7.15 |  3.67 |  5.50 | 25.71
MiB [15] | 12.25 | 13.09 | 12.65 | 42.67
PLOP | 44.03 | 15.51 | 30.45 | 52.32
PLOPLong | 61.06 | 18.56 | 40.83 | 58.62
TABLE VIII: Mean IoU on ADE20k 100-5. |  ADE 100-5 (11 tasks)
---|---
Method | 0-100 | 101-150 | all | avg
ILT [14] |  0.08 |  1.31 |  0.49 |  7.83
MiB [15] | 36.01 |  5.66 | 25.96 | 32.69
PLOP | 39.11 |  7.81 | 28.75 | 35.25

####  4.2.4 Longer Continual Learning

We argue that CSS experiments should push towards more steps [110, 91, 63, 59]
to quantify the robustness of approaches w.r.t. catastrophic forgetting and
background shift. We introduce two novel and much more challenging settings
with 11 tasks, almost twice as many as the previous longest setting. We report
results for VOC 10-1 in Table VII (10 classes followed by 10 times 1 class)
and ADE 100-5 in Table VIII (100 classes followed by 10 times 5 classes). The
second previous State-of-the-Art method, ILT, has a very low mIoU ($<6$ on VOC
10-1 and practically null on ADE 100-5). Furthermore, the gap between PLOP and
MiB is even wider compared with previous benchmarks (e.g. $\times$3.6 mIoU on
VOC for mIoU of base classes 1-10), which confirms the superiority of PLOP
when dealing with long continual processes. Furthermore, in such a case,
PLOPlong really shines, bringing significant improvements (+17.03 p.p on the
old classes, +3.05 p.p on the new classes, +10.42 p.p on average) due to
combination of cosine normalization (both on the classifier and Local POD) and
its frozen BatchReNormalization.

###  4.3 Object Rehearsal

TABLE IX: Comparison of rehearsal-based methods on Pascal-VOC 2012 15-1 overlap in Mean IoU (%). We only consider the time overhead spent after the first task whose computation overhead is similar for all methods. |  15-1 (6 tasks)
---|---
Method | Rehearsal |  Memory (Mb) $\downarrow$ |  Time (Hours) $\downarrow$ | 0-15 | 16-20 | all | avg
PLOP | — | 0 | 1.8 | 65.12 | 21.11 | 54.64 | 67.21
PLOPLong | — | 0 | 1.8 | 72.00 | 26.66 | 61.20 | 70.02
Yu et al.[74] | Unlabeled COCO | 20,000 | 7.0 | 71.40 | 40.00 | 63.60 |
PLOP | Unlabeled COCO | 20,000 | 1.4 | 72.57 | 45.08 | 66.03 | 71.85
PLOP | Unlabeled VOC | 2,000 | 1.4 | 75.32 | 52.59 | 69.91 | 75.21
PLOPLong | Partial VOC | 2.2 | 2.6 | 74.14 | 38.87 | 65.74 | 72.02
PLOPLong | Partial VOC | 22 | 2.6 | 74.18 | 43.22 | 66.81 | 72.48
PLOPLong | Object VOC | 0.26 | 2.7 | 73.32 | 42.86 | 66.07 | 72.21
PLOPLong | Object VOC | 2.6 | 2.7 | 73.79 | 45.78 | 67.12 | 72.42
Joint model | — | — | — | 79.10 | 72.60 | 77.40 | —

Pascal-VOC 15-1:  We now allow CSS models to store information from the
previous steps and classes: in such a case, the overall model efficiency can
be understood as to what extent it allows to find good trade-off between its
accuracy (as measured by the aforementionned standard metrics) and the memory
footprint of the stored images or objects. Methods such as HRHF [73] can not
easily be understood in these terms as they do not store data, strictly
speaking, but rather use deep inversion [111] techniques to generate synthetic
data, which comes at a high time requirement: thus, we exclude this method
from our comparisons. We first consider the challenging Pascal-VOC 15-1
setting in Table IX. We use PLOP and PLOPLong as baselines with 0 memory
overhead, as both models only use data from the current task. Moreover, we
compare with [74], where an unlabeled external dataset such as COCO [112] is
used through pseudo-labelling to improve performance on Pascal-VOC, as both
datasets present significant overlap in terms of classes and domains. We
reimplemented their method and also considered PLOP in this configuration.
Using the external COCO provide PLOP an important gain of mIoU for both old
classes (+7 p.p.) and new classes (+24 p.p.). Furthermore, PLOP with COCO is
significantly more performant than Yu et al. model (+5.43 p.p.) despite the
latter was designed explicitly to use such unlabeled external dataset. Notice
also how PLOPLong, without any kind of rehearsal, remains equivalent to [74]
in terms of mIoU on 0-15 (72.00% vs 71.40%), despite the latter using a large
pool of data (+20Go). Perhaps counter-intuitively the gain is located on the
new classes (26.66% vs 40.00%) as the rehearsal effect has a regularizing
effect leading to less over-predicting of the recent classes. A drawback of
using COCO is that the visual domain is not exactly the same as VOC, therefore
we also considered using PLOP with a rehearsal of the unlabeled VOC. Without
surprise, it results in a much better overall performance (+3.88 p.p. compared
to using COCO). Another setup that we consider is the image rehearsal
paradigm, where, at each step, we keep a number of (randomly selected) images
along with their segmentation map. These segmentation maps are however
incomplete, due to the nature of the CSS problem (see Sec. 3.5): hence, we
refer to this setting as partial VOC. We consider two amounts of images to
keep: 10 images per class (22 Mb) and 1 image per class (2.2 Mb). PLOPLong
largely benefits from this rehearsal, most notably on new classes (16-20) with
a gain up to 16.56 p.p.. Finally, we compare our novel Object Rehearsal
denoted by “object VOC” where we store either one object per class (0.26 Mb)
or 10 objects per class (2.6 Mb). As shown on Table IX, The proposed object
rehearsal, in addition to being significantly more memory efficient than whole
image rehearsal (8.5$\times$ less space used), is equivalent or better in
terms of mIoU especially for new classes (2.56 p.p.). The ensemble of our
experiments proves that rehearsal, when the partial or missing labeling is
carefully handled, can provide important performance gain. Furthermore, our
novel object rehearsal manages to strike the best trade-off with a minimal
memory overhead without impacting its performance.

Cityscapes:  We propose more experiments with Object Rehearsal in Table X
where we apply our model on Cityscapes 14-1. For the rehearsal, we sample
either 10 images or object per class. Cityscapes images are particularly large
(even resized to $512\times 1024$) and ”empty” (most of it is road and sky).
Consequently the memory overhead is extremely important compared object
rehearsal (117 vs 0.8). We show that both our novel image and object rehearsal
improve performance of the already competitive PLOPLong, and with object
rehearsal providing the large gain (+9 p.p. in all).

TABLE X: Continual Semantic Segmentation results on Cityscapes 14-1 overlap in Mean IoU (%). |  14-1 (6 tasks)
---|---
Method |  Rehearsal |  Memory | 1-14 | 15-19 | all | avg
PLOPLong |  — |  0 | 58.60 | 15.04 | 47.71 | 54.31
PLOPLong |  Partial |  117.0 | 58.93 | 19.55 | 49.09 | 55.74
PLOPLong |  Object |  0.8 | 57.82 | 23.13 | 49.15 | 54.80

###  4.4 Model Introspection

In this section, we perform component by component ablation study of PLOP, but
also aim at understanding the behavior of the model, as well as comparing
different rehearsal alternatives. Lastly we propose qualitative visualizations
of the models predictions.

####  4.4.1 PLOP Ablations

We compare several distillation and classification losses on VOC 15-1 to
stress the importance of the components of PLOP and report results in Table
XI. All comparisons are evaluated on a val set made with 20% of the train set,
therefore results are slightly different from the main experiments.

Distillation comparisons: XI(a) compares different distillation losses when
combined with our pseudo-labeling loss. As such, UNKD introduced in [15]
performs better than the Knowledge Distillation (KD) of [78], but not at every
step (as indicated by the avg. value), which indicates instability during the
training process. POD, proposed in [63], improves the results on the old
classes, but not on the new classes (16-20). In fact, due to too much
plasticity, POD model likely overfits and predicts nothing but the new
classes, hence a lower mIoU. Finally, Local POD leads to superior performance
(+20 p.p ) w.r.t. all metrics, due to its integration of both long and short-
range dependencies. This final row represents our full PLOP strategy.

Classification comparisons: XI(b) compares different classification losses
when combined with our Local POD distillation loss. Cross-Entropy (CE)
variants perform poorly, especially on new classes. UNCE, introduced in [15],
improves by merging the background with old classes, however, it still
struggles to correctly model the new classes, whereas our pseudo-labeling
propagates more finely information of the old classes, while learning to
predict the new ones, dramatically enhancing the performance in both cases.
This penultimate row represents our full PLOP strategy. Also notice that the
performance for pseudo-labeling is very close to Pseudo-Oracle (where the
incorrect pseudo-labels are removed), which may constitute a performance
ceiling of our uncertainty measure. A comparison between these two results
illustrates the relevance of our entropy-based uncertainty estimate.

####  4.4.2 Robutness to class ordering

Continual learning methods may be prone to instability. It has already been
shown in related contexts [113] that class ordering can have a large impact on
performance. Unfortunatly, in real-world settings, the optimal class order can
never be known beforehand: thus, the performance of an ideal CSS method should
be as invariant to class order as possible. In all experiments done so far,
this class order has been kept constant, as defined in [15]. We report results
in Fig. 4 as boxplots obtained by applying 20 random permutations of the class
order on VOC 15-1. We report in Fig. 4 (from left to right) the mIoU for the
old, new classes, all classes, and average over CSS steps. In all cases, PLOP
surpasses MiB in term of avg mIoU. Furthermore, the standard deviation (e.g.
10% vs 5% on all) is always significantly lower, showing the excellent
stability of PLOP compared with existing approaches. Furthermore, PLOPLong,
while having more variance on the new classes, performs better overall as
compared to PLOP, not to mention MiB. This is due to the fact that PLOPLong,
due to improved distillation loss (at prediction and Local POD levels) and
batch re-normalization that allows to better retain information of the old
classes, which again is more conspicuous on longer CSS scenarios.

TABLE XI: Comparison studies on Pascal-VOC 2012 15-1 on a validation subset of
20% of the training set.

(a) Pseudo loss (Eq. 8) with different distillation losses. Distillation loss | 0-15 | 16-20 | all | avg
---|---|---|---|---
Knowledge Distillation | 29.72 |  4.42 | 23.69 | 49.18
UNKD | 34.85 |  5.26 | 27.80 | 46.39
POD | 43.94 |  4.82 | 34.62 | 53.35
Local POD (Eq. 5) | 63.06 | 17.92 | 52.31 | 65.71

(b) Local POD loss (Eq. 5) with different classification losses. Classification loss | 0-15 | 16-20 | all | avg
---|---|---|---|---
CE only on new | 12.95 |  2.54 | 10.47 | 47.02
CE | 33.80 |  4.67 | 26.87 | 50.79
UNCE | 48.46 |  4.82 | 38.62 | 53.19
Pseudo (Eq. 8) | 63.06 | 17.92 | 52.31 | 65.71
Pseudo-Oracle | 63.69 | 23.35 | 54.09 | 66.05

Figure 4: Boxplots of the mIoU of initial classes (1-15), new (16-20), all,
and average for 20 random class orderings. PLOP is significantly better and
more stable than MiB. PLOPLong further improves upon PLOP by better retaining
old class information.

####  4.4.3 Rehearsal alternatives

Table XII draws a comparison between several rehearsal alternatives in CSS. We
split methods according to three criterions: “type” denotes whether we store a
whole image, an object, or a patch For both object and patch, we only rehearse
a small amount of the images: in object, only the pixel’s objects are used
while for patch, we use the pixel’s object and the close surrounding pixels
that contain background information. We insert the reheasal data according to
“mixing”, either by pasting (see Sec. 3.5), by mixing pixels according to
mixup [93] rule, or in the case of image rehearsal no mixing is done. Finally
we consider two erasing methods: either all pixels are erased (All), or only
pixels belonging to non-background classes are erased (Foreground). Note that,
for the latter method, it includes classes detected via pseudo-labeling. For
all methods, we randomly select 10 images/patches/objects per class for
rehearsal. Without surprise, Patch (III-V) and Object-based (VI-X) rehearsal
are more data-efficient than images (I and II) (4.50 and 2.60 vs 22.20 Mb).
Mixing the rehearsed data with mixup (VI and VII) is less efficient than
direct pasting (III, IV, and VIII-X) because segmentation, contrary to
classification, requires sharp boundaries [105]. We considered rehearsing each
patch/object without mixing, but because their shape may variate no batching
is possible which would be much slower. Therefore, we investigate mixing
patches/objects while erasing all others pixels (III and VIII). This doesn’t
work, which is probably linked to the altered statistics of the batch
normalization. On the other hand, we found that a selective erasing that
remove foreground objects (V, VII, and X) while keeping the background of the
destination image proved to be very effective (63.12% to 67.12% all mIoU for
object). This confirmed our intuition that pasting in segmentation is a
delicate operation that may lead to confusion in the network, particularly on
object boundaries. The erased pixels are replaced by gray pixels. We
considered more complicated approaches such as in-painting [99] or textures
filling [114], but were slightly less effective than our simpler solution.
Overall, through careful design, we manage to get the best performance using
our novel object rehearsal which was also the most memory-efficient.

TABLE XII: Rehearsal alternative on Pascal-VOC 2012 in Mean IoU (%). Object/Patch-based methods with 10 objects/patches per class, and Image-based with 10 images per class. All experiments done with PLOPLong . |  |  |  |  15-1 (6 tasks) |
---|---|---|---|---|---
Type | Mixing | Erase |  Memory $\downarrow$ | all | avg |
Image | Mixup | — | 22.20 | 61.77 | 69.88 | I
— | — | 66.81 | 72.48 | II
Patch | Pasting | All | 4.50 | 55.45 | 66.35 | III
Pasting | — | 63.41 | 70.75 | IV
Pasting | Foreground | 66.28 | 71.66 | V
Object | Mixup | — | 2.60 | 63.25 | 70.91 | VI
Mixup | Foreground | 64.45 | 71.65 | VII
Pasting | All | 52.26 | 65.97 | VIII
Pasting | — | 63.12 | 70.52 | IX
Pasting | Foreground | 67.12 | 72.42 | X
Figure 5: Distribution of the pixel per class during the $5^{\text{th}}$ step
of Pascal-VOC 15-1. Figure 6: Visualization of the predictions of MiB, PLOP,
PLOPLong, and PLOPLong with Object Rehearsal on three test images at the
$6^{\text{th}}$ and final step on VOC 15-1 scenario. The first image contains
car, cow, and person; the second plane and bus, and the third bicycle and
person. While MiB doesn’t manage to predict the correct classes, and tends to
overpredict the most recent ones (e.g. train in light green). PLOP mostly
grasp the correct classes, though sometimes with imprecision. PLOPLong and a
fortiori PLOPLong + Object rehearsal captures all existing classes, with the
latter predicting almost perfect segmentation masks, compared to the ground-
truth. Figure 7: Visualization of the predictions of MiB, PLOP, PLOPLong, and
PLOPLong with Object Rehearsal across time in VOC 15-1 on a test set image. At
steps 1-4 only class person has been seen. At step 5, the class train is
introduced, causing dramatic background shift. While MiB overfits on the new
class and forget the old class, PLOP is able to predict both classes
correctly. PLOPLong and PLOPLong + Object Rehearsal futher refine the
predicted masks, resulting in much sharper boundaries.

Rehearsal allievates pseudo-labeling’s limitation:  All previous methods, PLOP
and PLOPLong included, didn’t consider rehearsal-learning. While popular in
image classification, it has fewly been explored for continual segmentation.
An attentive reader may wonder if rehearsal learning can have benefit given
the fact that our pseudo-labeling can uncover “hidden” old classes in the
images. Even a perfect pseudo-labeling (considered in XI(b)) may fail in some
particular data situation where the class distribution of a step is almost
dirac. We show in Fig. 5 the distribution of pixels per class at the
$5^{\text{th}}$ step of Pascal-VOC 15-1. The new class to learn, train, is
abundant but also almost alone but the class person. In this case, no amount
of pseudo-labeling can uncover previous classes like bicycle or cat. Our
proposed rehearsal learning is complementary to our pseudo-labeling loss where
the former can reduce the weakness for the latter in some cases.

####  4.4.4 Visualization

Fig. 6 shows the predictions at the $6^{\text{th}}$ and final step of VOC 15-1
for four different models: MiB, PLOP, PLOPLong, and PLOPLong with Object
Rehearsal. MiB struggles to even find the correct classes: in the first image
the car is predicted as train, in the second the plane as train, and in the
third no classes are detected. On the other hand, PLOP always manages to pick
up the present classes, although sometimes with imperfection. PLOPLong further
refines results, and the addition of Object Rehearsal produces a sharp and
near perfect masks compared to the ground-truths (GTs). We showcased the
harmful effect of background shift in Fig. 7 where the class person is learned
at $1^{\text{st}}$ step and the class train at $5^{\text{th}}$ step. MiB
completly forgets the former class and overpredict the latter. The background
shift is efficiently mitigated with pseudo-labeling as showed by the third row
of PLOP. Our efficient proposition of object rehearsal allows further
refinement of the predicted masks resulting in sharper boundaries.

##  5 Conclusion

Continual Semantic Segmentation (CSS) is an emerging but challenging computer
vision domain. In this paper, we highlited two major issues in CSS:
catastrophic forgetting and background shift. To deal with the former, we
proposed Local POD, a multi-scale distillation scheme that preserves both long
and short-range spatial statistics between pixels. This lead to to an
effective balance between rigidity and plasticity for CSS, which in turns
alleviate catastrophic forgetting. We then tackled background shift with an
efficient uncertainty-based pseudo-labeling loss. It completes the partially-
labeled segmentation maps, allowing the network to efficiently retain
previously learned knowledge. Afterwards, We showed that carefully designed
structural changes to the model could improve performance on long CSS
scenarios, namely a cosine normalization adaptation of the classifier and
Local POD followed by a modified batch normalization. Finally, we proposed to
introduce rehearsal learning to CSS, one based on partially-labeled whole
image rehearsal, and the other –much more memory-efficient– consisting in
object rehearsal. The latter further refining our already effective model
performance and allowing real world application of CSS with a stringent memory
constraint. We evaluated the proposed PLOPLong, with or without object
rehearsal, on three datasets and over twelve different benchmarks. In each, we
showed that our model performs significantly better than all existing
baselines. Finally, we qualitatively validate our model through extensive
ablations in order to better understand the performance gain.

## Acknowledgments

This work was granted access to the HPC resources of IDRIS under the
allocation AD011011706 made by GENCI. We thank Fabio Cermelli for the
insightful discussions and his help with the BatchRenormalization code. We
also thank Estelle Thou and Thomas Robert for the invaluable discussions and
support.

## References

  * [1] D. G. Lowe, “Object recognition from local scale-invariant features,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 1999.
  * [2] F. Perronnin and C. Dancer, “Fisher kernels on visual vocabularies for image categorization,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2007.
  * [3] C. Cortes and V. Vapnik, “Support-vector networks,” _Machine Learning_ , 2015\.
  * [4] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2012.
  * [5] A. Tao, K. Sapra, and B. Catanzaro, “Hierarchical multi-scale attention for semantic segmentation,” in _arXiv preprint library_ , 2020.
  * [6] H. Zhang, C. Wu, Z. Zhang, Y. Zhu, Z. Zhang, H. Lin, Y. Sun, T. He, J. Muller, R. Manmatha, M. Li, and A. Smola, “Resnest: Split-attention networks,” in _arXiv preprint library_ , 2020.
  * [7] L. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proceedings of the IEEE European Conference on Computer Vision (ECCV)_ , 2018\.
  * [8] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba, “Scene parsing through ade20k dataset,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * [9] G. Neuhold, T. Ollmann, S. Rota Bulò, and P. Kontschieder, “The mapillary vistas dataset for semantic understanding of street scenes,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2017.
  * [10] H. Wang, Y. Zhu, B. Green, H. Adam, A. Yuille, and L.-C. Chen, “Axial-deeplab: Stand-alone axial-attention for panoptic segmentation,” in _Proceedings of the IEEE European Conference on Computer Vision (ECCV)_ , 2020.
  * [11] F. Ozdemir, P. Fuernstahl, and O. Goksel, “Learn the new, keep the old: Extending pretrained models with new anatomy and images,” in _International Conference on Med- ical Image Computing and Computer-Assisted Intervention_ , 2018.
  * [12] F. Ozdemir and O. Goksel, “Extending pretrained segmentation networks with additional anatomical structures,” in _International journal of computer assisted radiology and surgery_ , 2019.
  * [13] O. Tasar, Y. Tarabalka, and P. Alliez, “Incremental learning for semantic segmentation of large-scale remote sensing data,” in _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 2019.
  * [14] U. Michieli and P. Zanuttigh, “Incremental learning techniques for semantic segmentation,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshop_ , 2019.
  * [15] F. Cermelli, M. Mancini, S. Rota Bulò, E. Ricci, and B. Caputo, “Modeling the background for incremental learning in semantic segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [16] A. Douillard, Y. Chen, A. Dapogny, and M. Cord, “Plop: Learning without forgetting for continual semantic segmentation,” in _arXiv preprint library_ , 2020.
  * [17] A. Robins, “Catastrophic forgetting, rehearsal and pseudorehearsal,” _Connection Science_ , 1995.
  * [18] R. French, “Catastrophic forgetting in connectionist networks,” _Trends in cognitive sciences_ , 1999.
  * [19] S. Thrun, “Lifelong learning algorithms,” in _Springer Learning to Learn_ , 1998.
  * [20] R. Kemker, M. McClure, A. Abitino, T. L. Hayes, and C. Kanan, “Measuring catastrophic forgetting in neural networks,” in _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , 2018.
  * [21] S. Ioffe, “Batch renormalization: Towards reducing minibatch dependence in batch-normalized models,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017.
  * [22] C. Luo, J. Zhan, X. Xue, L. Wang, R. Ren, and Q. Yang, “Cosine normalization: Using cosine similarity instead of dot product in neural networks,” in _International Conference on Artificial Neural Networks_ , 2018.
  * [23] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015.
  * [24] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, “Overfeat: Integrated recognition, localization and detection using convolutional networks,” in _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2014.
  * [25] M. Everingham, S. M. Ali Eslami, L. Van Gool, C. K. I. Williams, J. M. Winn, and A. Zisserman, “The pascal visual object classes challenge: A retrospective,” in _International Journal of Computer Vision (IJCV)_ , 2015\.
  * [26] M. Cordts, M. Omran, S. Ramos, T. Reheld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [27] H. Caesar, J. R. R. Uijlings, and V. Ferrari, “Coco-stuff: Thing and stuff classes in context,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * [28] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” in _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , 2018.
  * [29] S. Mehta, M. Rastegari, A. Caspi, L. G. Shapiro, and H. Hajishirzi, “Efficient spatial pyramid of dilated convolutions for semantic segmentation,” in _Proceedings of the IEEE European Conference on Computer Vision (ECCV)_ , 2018\.
  * [30] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI)_ , 2015.
  * [31] H. Noh, S. Hong, and B. Han, “Learning deconvolution network for semantic segmentation,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2015.
  * [32] V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , 2017.
  * [33] Y. Yuan and J. Wang, “Ocnet: Object context network for scene parsing,” in _arXiv preprint library_ , 2018.
  * [34] H. Zhao, Y. Zhang, S. Liu, J. Shi, C. C. Loy, D. Lin, and J. Jia, “Psanet: Point-wise spatial attention network for scene parsing,” in _Proceedings of the IEEE European Conference on Computer Vision (ECCV)_ , 2018\.
  * [35] J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu, “Dual attention network for scene segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [36] Z. Huang, X. Wang, L. Huang, C. Huang, Y. Wei, and W. Liu, “Ccnet: Criss-cross attention for semantic segmentation,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019.
  * [37] Y. Yuan, X. Chen, and J. Wang, “Object-contextual representations for semantic segmentation,” in _Proceedings of the IEEE European Conference on Computer Vision (ECCV)_ , 2020.
  * [38] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * [39] H. Zhang, K. J. Dana, J. Shi, Z. Zhang, X. Wang, A. Tyagi, and A. Agrawal, “Context encoding for semantic segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * [40] J. Yoon, E. Yang, J. Lee, and S. J. Hwang, “Lifelong learning with dynamically expandable networks,” in _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2018.
  * [41] X. Li, Y. Zhou, T. Wu, R. Socher, and C. Xiong, “Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting,” _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2019.
  * [42] S. Yan, J. Xie, and X. He, “Der: Dynamically expandable representation for class incremental learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * [43] J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,” in _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2019.
  * [44] C. Fernando, D. Banarse, C. Blundell, Y. Zwols, D. Ha, A. A. Rusu, A. Pritzel, and D. Wierstra, “PathNet: Evolution Channels Gradient Descent in Super Neural Networks,” _arXiv preprint library_ , 2017.
  * [45] S. Golkar, M. Kagan, and K. Cho, “Continual learning via neural pruning,” _Advances in Neural Information Processing Systems (NeurIPS) Workshop_ , 2019\.
  * [46] S. C. Hung, C.-H. Tu, C.-E. Wu, C.-H. Chen, Y.-M. Chan, and C.-S. Chen, “Compacting, picking and growing for unforgetting continual learning,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * [47] Y. Wu, Y. Chen, L. Wang, Y. Ye, Z. Liu, Y. Guo, and Y. Fu, “Large scale incremental learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [48] B. Zhao, X. Xiao, G. Gan, B. Zhang, and S. Xia, “Maintaining discrimination and fairness in class incremental learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [49] E. Belouadah and A. Popescu, “Il2m: Class incremental learning with dual memory,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019.
  * [50] ——, “Scail: Classifier weights scaling for class incremental learning,” in _Proceedings of the IEEE Winter Conference on Application of Computer Vision (WACV)_ , 2020.
  * [51] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, “Overcoming catastrophic forgetting in neural networks,” _Proceedings of the National Academy of Sciences_ , 2017\.
  * [52] R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars, “Memory aware synapses: Learning what (not) to forget,” in _Proceedings of the IEEE European Conference on Computer Vision (ECCV)_ , 2018.
  * [53] A. Chaudhry, P. Dokania, T. Ajanthan, and P. H. S. Torr, “Riemannian walk for incremental learning: Understanding forgetting and intransigence,” _Proceedings of the IEEE European Conference on Computer Vision (ECCV)_ , 2018\.
  * [54] F. Zenke, B. Poole, and S. Ganguli, “Continual learning through synaptic intelligence,” in _International Conference on Machine Learning (ICML)_ , 2017.
  * [55] D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for continual learning,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., 2017.
  * [56] A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny, “Efficient lifelong learning with a-gem,” in _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2019.
  * [57] Z. Li and D. Hoiem, “Learning without forgetting,” _Proceedings of the IEEE European Conference on Computer Vision (ECCV)_ , 2016.
  * [58] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * [59] F. M. Castro, M. J. Marín-Jiménez, N. Guil, C. Schmid, and K. Alahari, “End-to-end incremental learning,” in _Proceedings of the IEEE European Conference on Computer Vision (ECCV)_ , 2018.
  * [60] S. Hou, X. Pan, C. Change Loy, Z. Wang, and D. Lin, “Learning a unified classifier incrementally via rebalancing,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [61] P. Dhar, R. V. Singh, K.-C. Peng, Z. Wu, and R. Chellappa, “Learning without memorizing,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [62] P. Zhou, L. Mai, J. Zhang, N. Xu, Z. Wu, and L. S. Davis, “M2kd: Multi-model and multi-level knowledge distillation for incremental learning,” _arXiv preprint library_ , 2019.
  * [63] A. Douillard, M. Cord, C. Ollion, T. Robert, and E. Valle, “Podnet: Pooled outputs distillation for small-tasks incremental learning,” in _Proceedings of the IEEE European Conference on Computer Vision (ECCV)_ , 2020\.
  * [64] A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. Torr, and M. Ranzato, “On tiny episodic memories in continual learning,” in _International Conference on Machine Learning (ICML) Workshop_ , 2019.
  * [65] T. L. Hayes, K. Kafle, R. Shrestha, M. Acharya, and C. Kanan, “Remind your neural network to prevent catastrophic forgetting,” in _Proceedings of the IEEE European Conference on Computer Vision (ECCV)_ , 2020.
  * [66] A. Iscen, J. Zhang, S. Lazebnik, and C. Schmid, “Memory-efficient incremental learning through feature adaptation,” in _Proceedings of the IEEE European Conference on Computer Vision (ECCV)_ , 2020.
  * [67] R. Kemker and C. Kanan, “Fearnet: Brain-inspired model for incremental learning,” in _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2018.
  * [68] H. Shin, J. K. Lee, J. Kim, and J. Kim, “Continual learning with deep generative replay,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017.
  * [69] Y. Liu, Y. Su, A.-A. Liu, B. Schiele, and Q. Sun, “Mnemonics training: Multi-class incremental learning without forgetting,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020\.
  * [70] R. Traoré, H. Caselles-Dupré, T. Lesort, T. Sun, G. Cai, N. Díaz-Rodríguez, and D. Filliat, “Discorl: Continual reinforcement learning via policy distillation,” in _Advances in Neural Information Processing Systems (NeurIPS) Workshop_ , 2019.
  * [71] J. G. Moreron-Torresa, T. Raeder, R. Alaiz-Rodríguez, N. V. Chawla, and F. Herrera, “A unifying view on dataset shift in classification,” in _Pattern Recognition_ , 2012.
  * [72] T. Lesort, M. Caccia, and I. Rish, “Understanding continual learning settings with data distribution drift analysis,” in _Findings of CLVISION at IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop_ , 2021\.
  * [73] Z. Huang, W. Hao, X. Wang, M. Tao, J. Huang, W. Liu, and X.-S. Hua, “Half-real half-fale distillation for class-incremental semantic segmentation,” in _arXiv preprint library_ , 2021.
  * [74] L. Yu, X. Liu, and J. van de Weijer, “Self-training for class-incremental semantic segmentation,” in _arXiv preprint library_ , 2020.
  * [75] F. Cermelli, M. Mancini, Y. Xian, Z. Akata, and B. Caputo, “A few guidelines for incremental few-shot segmentation,” in _arXiv preprint library_ , 2020\.
  * [76] U. Michieli and P. Zanuttigh, “Continual semantic segmentation via repulsion-attraction of sparse and disentangled latent representations,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * [77] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [78] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in _Advances in Neural Information Processing Systems (NeurIPS) Workshop_ , 2015.
  * [79] Z. Huang, X. Wang, Y. Wei, L. Huang, H. Shi, W. Liu, and T. S. Huang, “Ccnet: Criss-cross attention for semantic segmentation,” in _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , 2020.
  * [80] S. Park and Y. S. Heo, “Knowledge distillation for semantic segmentation using channel and spatial correlations and adaptive cross entropy,” in _Sensors_ , 2020.
  * [81] S. Lazebnik, C. Schmid, and J. Ponce, “Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories,” _Object Categorization: Computer and Human Vision Perspectives, Cambridge University Press_ , 2006.
  * [82] K. He, X. Zhang, S. Ren, and J. Sun, “Spatial pyramid pooling in deep convolutional networks for visual recognition,” in _Proceedings of the IEEE European Conference on Computer Vision (ECCV)_ , 2014.
  * [83] D.-H. Lee, “Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks,” in _International Conference on Machine Learning (ICML) Workshop_ , 2013.
  * [84] T.-H. Vu, H. Jain, M. Bucher, M. Cord, and P. Pérez, “Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [85] Y. Li, L. Yuan, and N. Vasconcelos, “Bidirectional learning for domain adaptation of semantic segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [86] Y. Zou, Z. Yu, B. Vijaya Kumar, and J. Wang, “Unsupervised domain adaptation for semantic seg- mentation via class-balanced self-training,” in _Proceedings of the IEEE European Conference on Computer Vision (ECCV)_ , 2018\.
  * [87] A. Saporta, T.-H. Vu, M. Cord, and P. Pérez, “Esl: Entropy-guided self-supervised learning for domain adaptation in semantic segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop_ , 2020.
  * [88] H. Qi, M. Brown, and D. G. Lowe, “Low-shot learning with imprinted weights,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * [89] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _International Conference on Machine Learning (ICML)_ , 2015.
  * [90] A. Douillard and T. Lesort, “Continuum: Simple management of complex continual learning scenarios,” in _Findings of CLVISION at IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop_ , 2021.
  * [91] V. Lomonaco, D. Maltoni, and L. Pellegrini, “Rehearsal-free continual learning over small non-i.i.d. batches,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop_ , 2020.
  * [92] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2017.
  * [93] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2018.
  * [94] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019.
  * [95] A. Dabouei, S. Soleymani, F. Taherkhani, and N. M. Nasrabadi, “Supermix: Supervising the mixing data augmentation,” in _arXiv preprint library_ , 2020\.
  * [96] V. Verma, A. Lamb, C. Beckham, A. Najafi, I. Mitliagkas, A. Courville, D. Lopez-Paz, and Y. Bengio, “Manifold mixup: Better representations by interpolating hidden states,” in _International Conference on Machine Learning (ICML)_ , 2019.
  * [97] B. Li, F. Wu, S.-N. Lim, S. Belongie, and K. Q. Weinberger, “On feature normalization and data augmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * [98] A. Rame, R. Sun, and M. Cord, “Mixmo: Mixing multiple inputs for multiple outputs via deep subnetworks,” in _arxiv_ , 2021.
  * [99] H.-S. Fang, J. Sun, R. Wang, M. Gou, Y.-L. Li, and C. Lu, “Instaboost: Boosting instance segmentation via probability map guided copy-pasting,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019.
  * [100] V. Olsson, W. Tranheden, J. Pinto, and L. Svensson, “Classmix: Segmentation-based data augmentation for semi-supervised learning,” in _Proceedings of the IEEE Winter Conference on Application of Computer Vision (WACV)_ , 2021.
  * [101] J. Zhang, Y. Zhang, and X. Xu, “Objectaug: Object-level data augmentation for semantic image segmentation,” in _arXiv preprint library_ , 2021.
  * [102] W. Tranheden, V. Olsson, J. Pinto, and L. Svensson, “Dacs: Domain adaptation via cross-domain mixed sampling,” in _Proceedings of the IEEE Winter Conference on Application of Computer Vision (WACV)_ , 2021.
  * [103] G. Ghiasi, Y. Cui, A. Srinivas, R. Qian, T.-Y. Lin, E. D. Cubuk, Q. V. Le, and B. Zoph, “Simple copy-paste is a strong data augmentation method for instance segmentation,” in _arXiv preprint library_ , 2020.
  * [104] “Information technology — digital compression and coding of continuous-tone still images: Requirements and guidelines,” International Organization for Standardization, Geneva, CH, Standard, Feb. 1994.
  * [105] Y. Chen, A. Dapogny, and M. Cord, “Semeda: Enhancing segmentation precision with semantic edge aware loss,” in _Pattern Recognition_ , 2020.
  * [106] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, “Rethinking atrous convolution for semantic image segmentation,” in _arXiv preprint library_ , 2017.
  * [107] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2009.
  * [108] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proceedings of the IEEE European Conference on Computer Vision (ECCV)_ , 2018\.
  * [109] S. Zagoruyko and N. Komodakis, “Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer,” _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2016.
  * [110] M. Wortsman, V. Ramanujan, R. Liu, A. Kembhavi, M. Rastegari, J. Yosinski, and A. Farhadi, “Supermasks in superposition for continual learning,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * [111] H. Yin, P. Molchanov, Z. Li, J. M. Alvarez, A. Mallya, D. Hoiem, N. K. Jha, and J. Kautz, “Dreaming to distill: Data-free knowledge transfer via deepinversion,” in _Proceedings of the IEEE Winter Conference on Application of Computer Vision (WACV)_ , 2019.
  * [112] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and L. Zitnick, “Microsoft coco: Common objects in context,” in _Proceedings of the IEEE European Conference on Computer Vision (ECCV)_ , 2014\.
  * [113] D. Kim, J. Bae, Y. Jo, and J. Choi, “Incremental learning with maximum entropy regularization: Rethinking forgetting and intransigence,” _arXiv preprint library_ , 2019.
  * [114] P. Mallikarjuna, A. Tavakoli Targhi, M. Fritz, E. Hayman, B. Caputo, and J.-O. Eklundh, “The kth-tips2 database,” in _Computational Vision and Active Perception Laboratory, Stockholm, Sweden_ , 2006.

|  Arthur Douillard is a second year PhD student at Sorbonne University in the
MLIA / LIP6 team under the supervision of Prof. Matthieu Cord. He is also a
research scientist at the fashion tech startup Heuritech, and a lecturer at
the engineering school EPITA. He received a M.Eng. degree in Computer Science
from EPITA in 2018. His research interests include continual learning, metric
learning, segmentation, and more broadly computer vision.
---|---
|  Yifu Chen received his Ph.D. in 2020 from the Computer Science Laboratory
(LIP6) of Sorbonne University, under the supervision of Prof. Matthieu Cord.
His thesis is on deep learning methods for semantic image segmentation. His
research also focuses on the visualization and explanation of deep neural
networks, as well as on continual learning.
---|---
|  Arnaud Dapogny is a computer vision researcher at Datakalab in Paris. He
obtained the Engineering degree from the Supélec engineering School in 2011
and the Masters degree from Sorbonne University, Paris, in 2013 with high
honors. He also obtained his PhD at Institute for Intelligent Systems and
Robotics (ISIR) in 2016 and worked as a post-doctoral fellow at LIP6. His
works concern deep learning for computer vision and its application to
automatic facial behavior as well as gesture analysis.
---|---
|  Matthieu Cord is full professor at Sorbonne University. He is also part-
time principal scientist at Valeo.ai. His research expertise includes computer
vision, machine learning and artificial intelligence. He is the author of more
150 pub- lications on image classification, segmentation, deep learning, and
multimodal vision and lan- guage understanding. He is an honorary member of
the Institut Universitaire de France and served from 2015 to 2018 as an AI
expert at CNRS and ANR (National Research Agency).
---|---