# PLOP: Learning without Forgetting for Continual Semantic Segmentation

Arthur Douillard1,2 Yifu Chen1 Arnaud Dapogny3 Matthieu Cord1,4
1Sorbonne Université  2Heuritech  3Datakalab  4valeo.ai
<EMAIL_ADDRESS>{yifu.chen<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Deep learning approaches are nowadays ubiquitously used to tackle computer
vision tasks such as semantic segmentation, requiring large datasets and
substantial computational power. Continual learning for semantic segmentation
(CSS) is an emerging trend that consists in updating an old model by
sequentially adding new classes. However, continual learning methods are
usually prone to catastrophic forgetting. This issue is further aggravated in
CSS where, at each step, old classes from previous iterations are collapsed
into the background. In this paper, we propose Local POD, a multi-scale
pooling distillation scheme that preserves long- and short-range spatial
relationships at feature level. Furthermore, we design an entropy-based
pseudo-labelling of the background w.r.t. classes predicted by the old model
to deal with background shift and avoid catastrophic forgetting of the old
classes. Our approach, called PLOP, significantly outperforms state-of-the-art
methods in existing CSS scenarios, as well as in newly proposed challenging
benchmarks111Code is available at
https://github.com/arthurdouillard/CVPR2021_PLOP.

##  1 Introduction

Figure 1: Our two-part strategy aims at learning a segmentation network in a
continual learning framework, where old class pixels are collapsed into the
background at current stage. We generate pseudo labels from old predictions
(blue) to deal with the background shift, and retain short- and long-range
spatial dependencies by Local POD distillation (red) to prevent catastrophic
forgetting.

Semantic segmentation is a fundamental problem of computer vision, that aims
at assigning a label to each pixel of an image. In recent years, the
introduction of Convolutional Neural Networks (CNNs) has addressed semantic
segmentation in a traditional framework, where all classes are known
beforehand and learned at once [67, 80, 12]. This setup, however, is quite
limited for practical applications. In a more realistic scenario, the model
should be able to continuously learn new classes without retraining from
scratch. This setup, referred here as Continual Semantic Segmentation (CSS),
has emerged very recently for medical applications [56, 57] before being
proposed for general segmentation datasets [54, 8].

Deep learning approaches that deal with CSS face two main challenges. The
first one, inherited from continual learning, is called catastrophic
forgetting [61, 23, 68], and points to the fact that neural networks tend to
completely and abruptly forget previously learned knowledge when learning new
information [38]. Catastrophic forgetting presents a real challenge for
continual learning applications based on deep learning methods, especially
when storing previously seen data is not allowed for privacy reasons.

The second issue, CSS specific, is the semantic shift of the background class.
In a traditional semantic segmentation setup, the background contains pixels
that don’t belong to any other class. However, in CSS, the background contains
pixels that don’t belong to any of the current classes. Thus, for a specific
learning step, the background can contain both future classes, not yet seen by
the model, as well as old classes. Thus, if nothing is done to distinguish
pixels belonging to the real background class from old class pixels, this
background shift phenomenon risks exacerbating the catastrophic forgetting
even further [8].

In this paper, we propose a deep learning strategy to address these two
challenges in CSS. Instead of reusing old images, our approach, called PLOP ,
standing for Pseudo-label and LOcal POD leverages the old model in two
manners, as illustrated on Fig. 1. First, we propose a feature-based multi-
scale distillation scheme to alleviate catastrophic forgetting. Second, we
employ a confidence-based pseudo-labeling strategy to retrieve old class
pixels within the background. For instance, if a current ground truth mask
only distinguish pixels from class sofa and background, our approach allows to
assign old classes to background pixels, e.g. classes person, dog or
background (the semantic class).

We thoroughly validate PLOP on several datasets, showcasing significant
performance improvements compared to the state-of-the-art methods in existing
CSS scenarios. Furthermore, we propose several novel scenarios to further
quantify the performances of CSS methods when it comes to long term learning,
class presentation order and domain shift. Last but not least, we show that
PLOP largely outperforms every CSS approach in these scenarios. To sum it up,
our contributions are three-folds:

  * •

We propose a multi-scale spatial distillation loss to better retain knowledge
through the continual learning steps, by preserving long- and short-range
spatial relationships, avoiding catastrophic forgetting.

  * •

We introduce a confidence-based pseudo-labeling strategy to identify old
classes for the current background pixels and deal with background shift.

  * •

We show that PLOP significantly outperforms state-of-the-art approaches in
existing scenarios and datasets for CSS, as well as in several newly proposed
challenging benchmarks.

##  2 Related Work

CSS is a relatively new field where only a few recent papers addressed this
specific problem. We thus start this section with a brief overview of the
recent advances in semantic segmentation as well as continual learning and
follow with a more in-depth discussion of existing approaches to CSS.

Semantic Segmentation methods based on Fully Convolutional Networks (FCN) [51,
65] have achieved impressive results on several segmentation benchmarks [20,
15, 84, 6]. These methods improve the segmentation accuracy by incorporating
more spatial information or exploiting contextual information specifically.
Atrous convolution [13, 53] and encoder-decoder architecture [63, 55, 2] are
the most common methods for retaining spatial information. Examples of recent
works exploiting contextual information include attention mechanisms [76, 83,
24, 32, 75, 67, 80], and fixed-scale aggregation [82, 13, 12, 79]. More
recently, Strip Pooling [30] consists in pooling along the width or height
dimensions similarly to POD [18] as a complement to a spatial pyramid pooling
[27] to capture both global and local statistics.

Continual Learning models generally face the challenge of catastrophic
forgetting of the old classes [61, 68, 23]. Several solutions exist to address
this problem: for instance, rehearsal learning consists in keeping a limited
amount of training data from old classes either as raw images [61, 60, 7, 11],
compressed features [26, 35], or generated training data [37, 66, 48]. Other
works focus on adaptive architectures that can extend themselves to integrate
new classes [74, 45] or dynamically re-arrange co-existing sub-networks [22]
each specialized in one specific task [21, 25, 34], or to explicitly correct
the classifier drift [73, 81, 3, 4] that happens with continually changing
class distributions. Last but not least, distillation-based methods aim at
constraining the model as it changes, either directly on the weights [40, 1,
9, 78], the gradients [52, 10], the output probabilities [47, 60, 7, 8],
intermediary features [31, 17, 85, 18], or combinations thereof.

Continual Semantic segmentation: Despite enormous progress in the two
aforementioned areas respectively, segmentation algorithms are mostly used in
an offline setting, while continual learning methods generally focus on image
classification. Recent works extend existing continual learning methods [47,
31] for medical applications [56, 57] and general semantic segmentation [54].
The latter considers that the previously learned categories are properly
annotated in the images of the new dataset. This is an unrealistic assumption
that fails to consider the background shift: pixels labeled as background at
the current step are semantically ambiguous, in that they can contain pixels
from old classes (including the real semantic background class, which is
generally deciphered first) as well as pixels from future classes. To the best
of our knowledge, Cermelli et al. [8] are the first to address this background
shift problem along with catastrophic forgetting. To do so, they apply two
loss terms at the output level. First, they use a knowledge distillation loss
to reduce forgetting. However, only constraining the output of the network
with a distillation term is not enough to preserve the knowledge of the old
classes, leading to too much plasticity and, ultimately, catastrophic
forgetting. Second, they propose to modify the traditional cross-entropy loss
for background pixels to propagate only the sum probability of old classes
throughout the continual learning steps. We argue that this constraint is not
strong enough to preserve a high discriminative power w.r.t. the old classes
when learning new classes under background shift. On the contrary, in what
follows, we introduce our PLOP framework and show how it enables learning
without forgetting for CSS.

##  3 PLOP Segmentation Learning Framework

###  3.1 Continual semantic segmentation framework

CSS aims at learning a model in $t=1\dots T$ steps. For each step, we present
a dataset $\mathcal{D}_{t}$ that consists in a set of pairs $(I^{t},S^{t})$,
where $I^{t}$ denotes an input image of size $W\times H$ and $S^{t}$ the
corresponding ground truth segmentation mask. The latter only contains the
labels of current classes $\mathcal{C}^{t}$, and all other labels (e.g. old
classes $\mathcal{C}^{1:t-1}$ or future classes $\mathcal{C}^{t+1:T}$) are
collapsed into the background class $c_{\text{bg}}$. However, the model at
step $t$ shall be able to predict all the classes seen over time
$\mathcal{C}^{1:t}$. Consequently, we identify two major pitfalls in CSS: the
first one, catastrophic forgetting [61, 23], suggests that the network will
completely forget the old classes $\mathcal{C}^{1:t-1}$ when learning
$\mathcal{C}^{t}$. Furthermore, catastrophic forgetting is aggravated by the
second pitfall, the background shift: at step $t$, the pixels labeled as
background are indeed ambiguous, as they may contain either old (including the
real background class, predicted in $\mathcal{C}^{1}$) or future classes. Fig.
2 (top row) illustrates background shift.

Classically, a deep model at step $t$ can be written as the composition of a
feature extractor $f^{t}(\cdot)$ and a classifier $g^{t}(\cdot)$. Features can
be extracted at any layer $l$ of the former
$f_{l}^{t}(\cdot)\,,l\in\\{1,...L\\}$. We denote $\hat{S}^{t}=g^{t}\circ
f^{t}(I)$ the output predicted segmentation mask and $\Theta^{t}$ the set of
learnable parameters for the current network at step $t$.

###  3.2 Multi-scale local distillation with Local POD

A common solution to alleviate catastrophic forgetting in continual learning
consists of using a distillation loss between the predictions of the old and
current models [47]. This distillation loss should constitute a suitable
trade-off between too much rigidity (i.e. enforcing too strong constraints,
resulting in not being able to learn new classes) and too much plasticity
(i.e. enforcing loose constraints, which leads to catastrophic forgetting of
the old classes).

Among existing distillation schemes based on intermediate features [18, 77,
62, 17, 85, 31], POD [18] consists in matching global statistics at different
feature levels between the old and current models. Let $\mathbf{x}$ denote an
embedding tensor of size $H\times W\times C$. Extracting a POD embedding
$\Phi$ consists in concatenating the $H\times C$ width-pooled slices and the
$W\times C$ height-pooled slices of $\mathbf{x}$:

$\Phi(\mathbf{x})=\left[\frac{1}{W}\sum_{w=1}^{W}\mathbf{x}[:,w,:]\bigg{\|}\frac{1}{H}\sum_{h=1}^{H}\mathbf{x}[h,:,:]\right]\in\mathcal{R}^{(H+W)\times
C}\,,$ (1)

where $[\cdot\,\|\,\cdot]$ denotes concatenation over the channel axis. In our
case, this embedding is computed at several layers, for both the old and
current model. Then the POD loss consists in minimizing the L2 distance
between the two sets of embeddings over the current network parameters
$\Theta^{t}$:

Figure 2: Background shift example in ground truth masks (top row). At step
$2$ background pixels contain old (person) and future classes (bottle). The
model’s target (middle row) is the union of the ground-truth and the pseudo-
labels (with transparent filtered uncertain pixels) generated by the previous
model. The latter helps the current model predictions (bottom row) to retain
information of the old classes (table).

$\mathcal{L}_{\text{pod}}(\Theta^{t})=\frac{1}{L}\sum_{l=1}^{L}\left\|\Phi(f^{t}_{l}(I))-\Phi(f^{t-1}_{l}(I))\right\|^{2}\,.$
(2)

Due to its ability to constraint spatial statistics instead of raw pixel
values, this approach yields state-of-the-art results in the context of
continual learning for classification. In the frame of CSS, another interest
arises: its ability to model long-range dependencies across a whole axis
(horizontal or vertical). However, while spatial information is discarded by
global pooling in classification, semantic segmentation requires a higher
degree of spatial precision. Therefore, modeling statistics across the whole
width or height leads to blurring local statistics important for smaller
objects.

Hence, a suitable distillation scheme for CSS shall retain both long-range and
short-range spatial relationships. Thus, inspired from the multi-scale
literature [43, 27], we propose a novel Local POD feature distillation scheme,
that consists in computing width and height-pooled slices on multiple regions
extracted at different scales $\\{1/2^{s}\\}_{s=0\dots S}$, as shown on Fig.
3. For an embedding tensor $\mathbf{x}$ of size $H\times W\times C$, and at
scale $1/2^{s}$, the Local POD embedding $\Psi^{s}(\mathbf{x})$ at scale $s$
is computed as the concatenation of $s^{2}$ POD embeddings:

$\Psi^{s}(\mathbf{x})=\left[\Phi(\mathbf{x}^{s}_{0,0})\|\dots\|\Phi(\mathbf{x}^{s}_{s-1,s-1})\right]\in\mathcal{R}^{(H+W)\times
C}\,,$ (3)

where $\forall i=0\dots s-1$, $\forall j=0\dots s-1$,
$\mathbf{x}^{s}_{i,j}=\mathbf{x}[iH/s:(i+1)H/s,jW/s:(j+1)W/s,:]$ is a sub-
region of the embedding tensor $\mathbf{x}$ of size $W/s\times H/s$. We then
concatenate (along channel axis) the Local POD embeddings
$\Psi^{s}(\mathbf{x})$ of each scale $s$ to form the final embedding:

Figure 3: Illustration of local POD. An embedding of size $W\times H\times C$
is pooled at $S$ scales with POD with a spatial-pyramid scheme. Here applying
local POD with $S=2$ and scales 1 and 1/2 respectively produces 1, and 4 POD
embeddings making $S\times C\times(H+W)$ dimensions total.

$\Psi(\mathbf{x})=\left[\Psi^{1}(\mathbf{x})\|\dots\|\Psi^{S}(\mathbf{x})\right]\in\mathcal{R}^{(H+W)\times
C\times S}\,.$ (4)

We provide in the supplementary materials the complete algorithm of Local POD
embedding extraction. We compute Local POD embeddings for several layers of
both old and current models. The final Local POD loss is:

$\mathcal{L}_{\scriptstyle\text{LocalPod}}(\Theta^{t})=\frac{1}{L}\sum_{l=1}^{L}\left\|{\Psi}(f^{t}_{l}(I))-{\Psi}(f^{t-1}_{l}(I))\right\|^{2}\,.$
(5)

Note that while the first scale of Local POD ($1/2^{0}$) is equivalent to POD
and models long-range dependencies, which are important for segmentation [70,
33, 58, 30], the subsequent scales ($s=1/2^{1},1/2^{2}\dots$) enforce short-
range dependencies. This constrains the old and current models to have similar
statistics over more local regions. Thus, Local POD allows retaining both
long-range and short-range spatial relationships, thus alleviating
catastrophic forgetting.

###  3.3 Solving background shift with pseudo-labeling

As described above, the pixels labelled as background at step $t$ can belong
to either old (including the semantic background class) or future classes.
Thus, treating them as background would result in aggravating catastrophic
forgetting. Rather, we address background shift with a pseudo-labeling
strategy for background pixels. Pseudo-labeling [44] is commonly used in
domain adaptation for semantic segmentation [69, 46, 86, 64], where a model is
trained on the union of real labels of a source dataset and pseudo labels
assigned to an unlabeled target dataset. In our case, we use predictions of
the old model for background pixels as clues regarding their real class, most
notably if they belong to any of the old classes, as illustrated on Fig. 2
(middle row). Formally, let $C^{t}=card(\mathcal{C}^{t})-1$ the cardinality of
the current classes excluding the background class. Let
$\hat{S}^{t}\in\mathcal{R}^{W,H,1+C^{1}+\dots+C^{t}}$ denote the predictions
of the current model (which include the real background class, all the old
classes as well as the current ones). We define
$\tilde{S}^{t}\in\mathcal{R}^{W,H,1+C^{1}+\dots+C^{t}}$ the target as step
$t$, computed using the one-hot ground-truth segmentation map
$S^{t}\in\mathcal{R}^{W,H,1+C^{t}}$ at step $t$ as well as pseudo-labels
extracted using the old model predictions
$\hat{S}^{t-1}\in\mathcal{R}^{W,H,1+C^{1}+\dots+C^{t-1}}$ as follows:

$\footnotesize\tilde{S}^{t}\left(w,h,c\right)=\mkern-5.0mu\left\\{\begin{array}[]{ll}\mkern-10.0mu1\mkern-27.0mu&\text{
if }S^{t}(w,h,c_{bg})=0\text{ and
}c=\operatorname*{argmax}\limits_{c^{\prime}\in\mathcal{C}^{t}}S^{t}(w,h,c^{\prime})\\\
\mkern-10.0mu1\mkern-27.0mu&\text{ if }S^{t}(w,h,c_{bg})=1\text{ and
}c=\mkern-8.0mu\operatorname*{argmax}\limits_{c^{\prime}\in\mathcal{C}^{1:t-1}}\mkern-6.0mu\hat{S}^{t-1}(w,h,c^{\prime})\\\
\mkern-10.0mu0\mkern-27.0mu&\text{ otherwise }\\\ \end{array}\right.$ (6)

In other words, in the case of non-background pixels we copy the ground truth
label. Otherwise, we use the class predicted by the old model
$g^{t-1}(f^{t-1}(\cdot))$. This pseudo-label strategy allows to assign each
pixel labelled as background his real semantic label if this pixel belongs to
any of the old classes. However pseudo-labeling all background pixels can be
unproductive, e.g. on uncertain pixels where the old model is likely to fail.
Therefore we only retain pseudo-labels where the old model is “confident”
enough. Eq. 6 can be modified to take into account this uncertainty:

$\scriptsize\tilde{S}^{t}\left(w,h,c\right)\mkern-5.0mu=\mkern-5.0mu\left\\{\begin{array}[]{ll}\mkern-14.0mu1\mkern-27.0mu&\mkern-8.0mu\text{
if }S^{t}\mkern-4.0mu(w,h,c_{bg})\mkern-4.0mu=\mkern-4.0mu0\text{ and
}c\mkern-4.0mu=\mkern-4.0mu\operatorname*{argmax}\limits_{c^{\prime}\in\mathcal{C}^{t}}S^{t}\mkern-4.0mu(w,h,c^{\prime})\\\
\mkern-14.0mu1\mkern-27.0mu&\mkern-8.0mu\text{ if
}S^{t}\mkern-4.0mu(w,h,c_{bg})\mkern-4.0mu=\mkern-4.0mu1\text{ and
}c\mkern-4.0mu=\mkern-12.0mu\operatorname*{argmax}\limits_{c^{\prime}\in\mathcal{C}^{1:t-1}}\mkern-6.0mu\hat{S}^{t-1}\mkern-4.0mu(w,h,c^{\prime})\text{
and }u\mkern-4.0mu<\mkern-4.0mu\tau_{c}\\\
\mkern-14.0mu0\mkern-27.0mu&\mkern-4.0mu\,\text{ otherwise\,, }\\\
\end{array}\right.$ (7)

Table 1: Continual Semantic Segmentation results on Pascal-VOC 2012 in Mean IoU (%). $\dagger$: results excerpted from [8]. Other results comes from re-implementation. |  19-1 (2 tasks) |  15-5 (2 tasks) |  15-1 (6 tasks)
---|---|---|---
Method | 0-19 | 20 | all | avg | 0-15 | 16-20 | all | avg | 0-15 | 16-20 | all | avg
$\text{EWC}^{\dagger}$ [40] | 26.90 | 14.00 | 26.30 |  | 24.30 | 35.50 | 27.10 |  |  0.30 |  4.30 |  1.30 |
$\text{LwF-MC}^{\dagger}$ [60] | 64.40 | 13.30 | 61.90 |  | 58.10 | 35.00 | 52.30 |  |  6.40 |  8.40 |  6.90 |
$\text{ILT}^{\dagger}$ [54] | 67.10 | 12.30 | 64.40 |  | 66.30 | 40.60 | 59.90 |  |  4.90 |  7.80 |  5.70 |
ILT [54] | 67.75 | 10.88 | 65.05 | 71.23 | 67.08 | 39.23 | 60.45 | 70.37 |  8.75 |  7.99 |  8.56 | 40.16
$\text{MiB}^{\dagger}$ [8] | 70.20 | 22.10 | 67.80 |  | 75.50 | 49.40 | 69.00 |  | 35.10 | 13.50 | 29.70 |
MiB [8] | 71.43 | 23.59 | 69.15 | 73.28 | 76.37 | 49.97 | 70.08 | 75.12 | 34.22 | 13.50 | 29.29 | 54.19
PLOP | 75.35 | 37.35 | 73.54 | 75.47 | 75.73 | 51.71 | 70.09 | 75.19 | 65.12 | 21.11 | 54.64 | 67.21
Table 2: Continual Semantic Segmentation results on ADE20k in Mean IoU (%). |  100-50 (2 tasks) |  50-50 (3 tasks) |  100-10 (6 tasks)
---|---|---|---
Method | 0-100 | 101-150 | all | avg | 0-50 | 51-150 | all | avg | 0-100 | 101-150 | all | avg
ILT [54] | 18.29 | 14.40 | 17.00 | 29.42 |  3.53 | 12.85 |  9.70 | 30.12 |  0.11 |  3.06 |  1.09 | 12.56
MiB [8] | 40.52 | 17.17 | 32.79 | 37.31 | 45.57 | 21.01 | 29.31 | 38.98 | 38.21 | 11.12 | 29.24 | 35.12
PLOP | 41.87 | 14.89 | 32.94 | 37.39 | 48.83 | 20.99 | 30.40 | 39.42 | 40.48 | 13.61 | 31.59 | 36.64

where $u$ represents the uncertainty of pixel $(w,h)$ and $\tau_{c}$ is a
class-specific threshold. Thus, we discard all the pixels for which the old
model is uncertain ($u\geq\tau_{c}$) in Eq. 7 and decrement the normalization
factor $WH$ by one. We use entropy as the uncertainty measurement $u$.
Specifically, before learning task $t$, we compute the median entropy for the
old model over all pixels of $\mathcal{D}^{t}$ predicted as $c$ for all the
previous classes $c\in C^{1:t-1}$, which provides in thresholds $\tau_{c}\in
C^{1:t-1}$, as proposed in [64]. The cross-entropy loss with pseudo-labeling
of the old classes can be written as:

$\mathcal{L}_{\text{pseudo}}(\Theta^{t})=-\frac{\nu}{WH}\sum_{w,h}^{W,H}\sum_{c\in\mathcal{C}^{t}}\tilde{S}\left(w,h,c\right)\log\hat{S}^{t}\left(w,h,c\right)\,,$
(8)

where $\nu$ is the ratio of accepted old classes pixels over the total number
of such pixels. This ponderation allows to adaptively weight the importance of
the pseudo-labeling within the total loss. We call PLOP (standing for Pseudo-
labeling and LOcal Pod) the proposed approach, that uses both Local POD to
avoid catastrophic forgetting, and our uncertainty-based pseudo-labeling to
address background shift. To sum it up, the total loss in PLOP is:

$\mathcal{L}(\Theta^{t})=\underbrace{\mathcal{L}_{\text{pseudo}}(\Theta^{t})}_{\text{classification}}+\lambda\underbrace{\mathcal{L}_{\text{localPod}}(\Theta^{t})}_{\text{distillation}}\,,$
(9)

with $\lambda$ an hyperparameter.

##  4 Experiments

###  4.1 Datasets, Protocols, and Baselines

To ensure fair comparisons with state-of-the-art approaches, we follow the
experimental setup of [8] for datasets, protocol, metrics, and baseline
implementations.

Datasets:  we evaluate PLOP on 3 segmentation datasets: Pascal-VOC 2012 [20]
(20 classes), ADE20k [84] (150 classes) and CityScapes [15] (19 classes from
21 different cities). Full details are in the supplementary materials.

CSS protocols:  [8] describes two different CSS settings: Disjoint and
Overlapped. In both, only the current classes are labeled vs. a background
class $C^{t}$. However, in the former, images of task $t$ only contain pixels
$C^{1:t-1}\cup C^{t}$ (old and current), while, in the latter, pixels can
belong to any classes $C^{1:t-1}\cup C^{t}\cup C^{t+1:T}$ (old, current, and
future). Thus, the Overlapped setting is the most challenging and realistic,
as in a real setting there isn’t any oracle method to exclude future classes
from the background. Therefore, in our experiments, we focus on Overlapped CSS
but more results for Disjoint CSS can be found in the supplementary materials.
While the training images are only labeled for the current classes, the
testing images are labeled for all seen classes. We evaluate several CSS
protocols for each dataset, e.g. on VOC 19-1, 15-5, and 15-1 respectively
consists in learning 19 then 1 class ($T=2$ steps), 15 then 5 classes ($2$
steps), and 15 classes followed by five times 1 class ($6$ steps). The last
setting is the most challenging due to its higher number of steps. Similarly,
on ADE 100-50 means 100 followed by 50 classes ($2$ steps), 100-10 means 100
followed by 5 times 10 classes ($6$ steps), and so on.

Metrics:  we compare the different models using traditional mean Intersection
over Union (mIoU). Specifically, we compute mIoU after the last step $T$ for
the initial classes $C^{1}$, for the incremented classes $C^{2:T}$, and for
all classes $C^{1:T}$ (all). These metrics respectively reflect the robustness
to catastrophic forgetting (the model rigidity), the capacity to learn new
classes (plasticity), as well as its overall performance (trade-of between
both). We also introduce a novel avg metric (short for average), which
measures the average of mIoU scores measured step after step, integrating
performance over the whole continual learning process.

Baselines: We benchmark our model against the latest state-of-the-arts CSS
methods ILT [54] and MiB [8]. We also evaluate general continual models based
on weight constraints (EWC [40]) and knowledge distillation (LwF-MC [60]).
More baselines are available in the supplementary materials. All models, ours
included, don’t use rehearsal learning [61, 60, 11] where a limited quantity
of previous tasks data can be rehearsed. Finally, we also compare with a
reference model learned in a traditional semantic segmentation setting (“Joint
model” without continual learning), which may constitute an upper bound for
CSS methods.

Implementation Details: As in [8], we use a Deeplab-V3 [14] architecture with
a ResNet-101 [28] backbone pretrained on ImageNet [16] for all experiments.
Full details are provided in the supplementary materials.

###  4.2 Quantitative Evaluation

First, we compare PLOP with state-of-the-art methods.

Pascal VOC 2012:  Table 1 shows quantitative experiments on VOC 19-1, 15-5,
and 15-1. PLOP outperforms its closest contender, MiB [8] on all evaluated
settings by a significant margin. On 19-1, the forgetting of old classes
(1-19) is reduced by 4.39 percentage points ( p.p) while performance on new
classes is greatly improved (+13.76 p.p). On 15-5, our model is on par with
our re-implementation of MiB, and surpasses the original paper scores [8] by 1
p.p. On the most challenging 15-1 setting, general continual models (EWC and
LwF-MC) and ILT all have very low mIoU. While MiB shows significant
improvements, PLOP still outperforms it by a wide margin: +86% on all classes,
+90% on old classes, and +56% on new classes. Also, the joint model mIoU is
$77.40\%$, thus PLOP narrows the gap compared to state-of-the-art approaches
on every CSS scenario. The average mIoU is also improved by +24% compared to
MiB, indicating that each CSS step benefits from the improvements related to
our method. This is echoed by Fig. 4, which shows that while mIoU for both ILT
and MiB deteriorates after only a handful of steps, PLOP ’s mIoU remains very
high throughout, indicating improved resilience to catastrophic forgetting and
background shift.

ADE20k:  Table 2 shows experiments on ADE 100-50, 100-10, and 50-50. This
dataset is notoriously hard, as the joint model baseline mIoU is only 38.90%.
ILT has poor performance in all three scenarios. PLOP shows comparable
performance with MiB on the short setting 100-50 (only 2 tasks), improves by
1.09 p.p on the medium setting 50-50 (3 tasks), and significantly outperforms
MiB with a wider margin of 2.35 p.p on the long setting 100-10 (6 tasks). In
addition to being better on all settings, PLOP showcased an increased
performance gain on longer CSS (e.g. 100-10) scenarios, due to increased
robustness to catastrophic forgetting and background shift. To further
validate this robustness, we propose harder novel CSS scenarios.

###  4.3 New Protocols and Evaluation

Table 3: Mean IoU on Pascal-VOC 2012 10-1. |  VOC 10-1 (11 tasks)
---|---
Method | 0-10 | 11-20 | all | avg
ILT [54] |  7.15 |  3.67 |  5.50 | 25.71
MiB [8] | 12.25 | 13.09 | 12.65 | 42.67
PLOP | 44.03 | 15.51 | 30.45 | 52.32
Table 4: Mean IoU on ADE20k 100-5. |  ADE 100-5 (11 tasks)
---|---
Method | 0-100 | 101-150 | all | avg
ILT [54] |  0.08 |  1.31 |  0.49 |  7.83
MiB [8] | 36.01 |  5.66 | 25.96 | 32.69
PLOP | 39.11 |  7.81 | 28.75 | 35.25

Longer Continual Learning:  We argue that CSS experiments should push towards
more steps [72, 50, 18, 7] to quantify the robustness of approaches w.r.t.
catastrophic forgetting and background shift. We introduce two novel and much
more challenging settings with 11 tasks, almost twice as many as the previous
longest setting. We report results for VOC 10-1 in Table 3 (10 classes
followed by 10 times 1 class) and ADE 100-5 in Table 4 (100 classes followed
by 10 times 5 classes). The second previous State-of-the-Art method, ILT, has
a very low mIoU ($<6$ on VOC 10-1 and practically null on ADE 100-5).
Furthermore, the gap between PLOP and MiB is even wider compared with previous
benchmarks (e.g. $\times$3.6 mIoU on VOC for mIoU of base classes 1-10), which
confirms the superiority of PLOP when dealing with long continual processes.

Figure 4: mIoU evolution on Pascal-VOC 2012 15-1. While MiB’s mIoU quickly
deteriorates, PLOP’s mIoU remains high, due to improved resilience to
catastrophic forgetting. Figure 5: Boxplots of the mIoU of initial classes
(1-15), new (16-20), all, and average for 20 random class orderings. PLOP is
significantly better and more stable than MiB.

Stability w.r.t. class ordering: We already showed that existing continual
learning methods may be prone to instability. It has already been shown in
related contexts [39] that class ordering can have a large impact on
performance. However, in real-world settings, the optimal class order can
never be known beforehand: thus, the performance of an ideal CSS method should
be as class order-invariant as possible. In all experiments done so far, this
class order has been kept constant, as defined in [8]. We report results in
Fig. 5 under the form of boxplots obtained by applying 20 random permutations
of the class order on VOC 15-1. We report in Fig. 5 (from left to right) the
mIoU for the old, new classes, all classes, and average over CSS steps. In all
cases, PLOP surpasses MiB in term of avg mIoU. Furthermore, the standard
deviation (e.g. 10% vs 5% on all) is always significantly lower, showing the
excellent stability of PLOP compared with existing approaches.

Table 5: Final mIoU for Continual-domain Cityscapes. Method |  11-5 (3 tasks) |  11-1 (11 tasks) |  1-1 (21 tasks)
---|---|---|---
ILT [54] | 59.14 | 57.75 | 30.11
MiB [8] | 61.51 | 60.02 | 42.15
PLOP | 63.51 | 62.05 | 45.24

Domain Shift:  The previous experimental setups mainly assess the capacity of
CSS methods to integrate new classes, i.e. to deal with catastrophic
forgetting and background shift at a semantic level. However, a domain shift
can also happen in CSS scenarios. Thus, we propose a novel benchmark on
Cityscapes to quantify robustness to domain shift, in which all 19 classes
will be known from the start and, instead of adding new classes, each step
brings a novel domain (e.g. a new city), similarly to the NI setting of [49]
for image classification. Table 5 compares the performance of ILT, MiB, and
PLOP on CityScapes 11-5, 11-1, and 1-1, making 3, 11 and 21 steps of 11 + 2
times 5 cities, 11 + 10 times 1 city, and 1 + 20 times 1 city respectively.
PLOP performs better by a significant margin in every such scenario compared
with ILT and MiB which, in this setting, is equivalent to a simple cross-
entropy plus basic knowledge distillation [29]. Our Local POD, however,
retains better domain-related information by modeling long and short-range
dependencies at different representation levels.

###  4.4 Model Introspection

We compare several distillation and classification losses on VOC 15-1 to
stress the importance of the components of PLOP and report results in Table 6.
All comparisons are evaluated on a val set made with 20% of the train set,
therefore results are slightly different from the main experiments.

Distillation comparisons: 6(a) compares different distillation losses when
combined with our pseudo-labeling loss. As such, UNKD introduced in [8]
performs better than the Knowledge Distillation (KD) of [29], but not at every
step (as indicated by the avg. value), which indicates instability during the
training process. POD, proposed in [18], improves the results on the old
classes, but not on the new classes (16-20). In fact, due to too much
plasticity, POD model likely overfits and predicts nothing but the new
classes, hence a lower mIoU. Finally, Local POD leads to superior performance
(+20 p.p) w.r.t. all metrics, due to its integration of both long and short-
range dependencies. This final row represents our full PLOP strategy.

Classification comparisons: 6(b) compares different classification losses when
combined with our Local POD distillation loss. Cross-Entropy (CE) variants
perform poorly, especially on new classes. UNCE, introduced in [8], improves
by merging the background with old classes, however, it still struggles to
correctly model the new classes, whereas our pseudo-labeling propagates more
finely information of the old classes, while learning to predict the new ones,
dramatically enhancing the performance in both cases. This penultimate row
represents our full PLOP strategy. Also notice that the performance for
pseudo-labeling is very close to Pseudo-Oracle (where the incorrect pseudo-
labels are removed), which may constitute a performance ceiling of our
uncertainty measure. A comparison between these two results illustrates the
relevance of our entropy-based uncertainty estimate.

Table 6: Comparison studies on Pascal-VOC 2012 15-1 on a validation subset of
20% of the training set.

(a) Pseudo loss (Eq. 8) with different distillation losses. Distillation loss | 0-15 | 16-20 | all | avg
---|---|---|---|---
Knowledge Distillation | 29.72 |  4.42 | 23.69 | 49.18
UNKD | 34.85 |  5.26 | 27.80 | 46.39
POD | 43.94 |  4.82 | 34.62 | 53.35
Local POD (Eq. 5) | 63.06 | 17.92 | 52.31 | 65.71

(b) Local POD loss (Eq. 5) with different classification losses. Classification loss | 0-15 | 16-20 | all | avg
---|---|---|---|---
CE only on new | 12.95 |  2.54 | 10.47 | 47.02
CE | 33.80 |  4.67 | 26.87 | 50.79
UNCE | 48.46 |  4.82 | 38.62 | 53.19
Pseudo (Eq. 8) | 63.06 | 17.92 | 52.31 | 65.71
Pseudo-Oracle | 63.69 | 23.35 | 54.09 | 66.05

Figure 6: Visualization of MiB and PLOP predictions across time in VOC 15-1
for two test images. MiB quickly forgets the initial 15 classes (row 1: person
and table, row 3: bird) in favor of new classes (plant, sheep, sofa, train)
and is biased towards new classes. PLOP, however, barely suffers from
catastrophic forgetting (rows 2+4). Figure 7: Visualization of MiB and PLOP
predictions across time in VOC 15-1 on a test set image. At steps 1-4 only
class person has been seen. At step 5, the class train is introduced, causing
dramatic background shift. While MiB overfits on the new class and forget the
old class, PLOP is able to predict both classes correctly.

Vizualisation:  Fig. 6 shows the predictions for both MiB and PLOP on VOC 15-1
across time. At first, both models output equivalent predictions. However, MiB
quickly forgets the previous classes and becomes biased towards new classes.
On the other hand, PLOP predictions are much more stable on old classes while
learning new classes, thanks to Local POD alleviating catastrophic forgetting
by spatially constraining representations, and pseudo-labeling dealing with
background shift. Fig. 7 more closely highlights this phenomenon: at first,
the ground-truth only contains the class person. At step $5$, the class train
is introduced. As a result, MiB overfits on train and forgets person. PLOP,
instead, manages to avoid forgetting person and predicts decent segmentation
for both classes.

##  5 Conclusion

In this paper, we paved the way for future research on Continual Semantic
Segmentation, which is an emerging domain in computer vision. We highlighted
two main challenges in Continual Semantic Segmentation (CSS), namely
catastrophic forgetting and background shift. To deal with the former, we
proposed Local POD, a multi-scale pooling distillation scheme that allows
preserving long and short-range spatial relationships between pixels, leading
to a suitable trade-off between rigidity and plasticity for CSS and,
ultimately, alleviating catastrophic forgetting. The proposed method is
general enough to be used in other related distillation settings, where
preserving spatial information is a concern. In addition, we introduced a new
strategy to address the background shift based on an efficient pseudo-labeling
method. We validate our PLOP framework, on several existing CSS scenarios
involving multiple datasets. In addition, we propose novel experimental
scenarios to assess the performance of future CSS approaches in terms of long
term learning capacity and stability. We showed that PLOP performs
significantly better than all existing baselines in every such CSS benchmark.

Acknowledgments: This work was granted access to the HPC resources of IDRIS
under the allocation AD011011706 made by GENCI.

##  Appendix A Appendix

###  A.1 Further Work

In our CSS setting, pixels of task $T$ can belong to old $C^{1:t-1}$, current
$C^{t}$, and future classes $C^{t+1:T}$. In this paper we cover how to better
handle old and current classes. Further works should investigate how to
exploit the already present future information with Zeroshot [42, 41] as
already done in semantic segmentation [36, 5] and explored for continual
classification [71, 19].

###  A.2 Algorithm view of Local POD

In Algo. 1, we summarize the algorithm for the proposed Local POD. The
algorithm consists in three functions. First, Distillation, loops over all $L$
layers onto which we apply Local POD. Second, LocalPOD, computes the L2
distance (L.26) between POD embeddings of the current (L.19) and old (L.20)
models. It loops over $S$ different scales (L.14) and $\Phi$ computes the POD
embedding given two features maps subsets (L.19-20) as defined in Eq. 1. $\|=$
denotes an in-place concatenation.

Algorithm 1 Local POD algorithm

1:function Distillation($f^{t}$, $f^{t-1}$, $x$, $S$)

2: $loss\leftarrow 0$

3: for $l\leftarrow 0$; $l<L$; $l{+}{+}$ do

4: $\mathbf{h}^{t}_{l}\leftarrow f^{t}_{l}(\mathbf{x})$

5: $\mathbf{h}^{t-1}_{l}\leftarrow f^{t-1}_{l}(\mathbf{x})$

6: $loss\leftarrow
loss+\operatorname{LocalPOD}(\mathbf{h}^{t}_{l},\mathbf{h}^{t-1}_{l},S)$

7: end for

8: return $\frac{loss}{L}$

9:end function

10:

11:function LocalPOD($\mathbf{h}^{t}$, $\mathbf{h}^{t-1},S$)

12: $\mathbf{P}^{t}\leftarrow[\,]$

13: $\mathbf{P}^{t-1}\leftarrow[\,]$

14: for $s\leftarrow 0$; $s<S$; $s{+}{+}$ do $\triangleright$ Eq. 3

15: $w\leftarrow\nicefrac{{W}}{{2^{s}}}$

16: $h\leftarrow\nicefrac{{H}}{{2^{s}}}$

17: for $i\leftarrow 0$; $i<W-w$; $i{+}=w$ do

18: for $j\leftarrow 0$; $j<H-h$; $j{+}=h$ do

19: $\mathbf{p}^{t}\leftarrow\operatorname{\Phi}(\mathbf{h}^{t}\texttt{[i:i+w,
j:j+h]})$ $\triangleright$ Eq. 1

20:
$\mathbf{p}^{t-1}\leftarrow\operatorname{\Phi}(\mathbf{h}^{t-1}\texttt{[i:i+w,
j:j+h]})$

21: $\mathbf{P}^{t}\|=\mathbf{p}^{t}$

22: $\mathbf{P}^{t-1}\|=\mathbf{p}^{t-1}$

23: end for

24: end for

25: end for

26: return $\left\|\mathbf{P}^{t}-\mathbf{P}^{t-1}\right\|^{2}$
$\triangleright$ Eq. 5

27:end function

###  A.3 Reproducibility

Datasets:  We evaluate our model on three datasets Pascal-VOC [20], ADE20k
[84], and Cityscapes [15]. VOC contains 20 classes, 10,582 training images,
and 1,449 testing images. ADE20k has 150 classes, 20,210 training images, and
2,000 testing images. Cityscapes contains 2975 and 500 images for train and
test, respectively. Those images represent 19 classes and were taken from 21
different cities. All ablations and hyperparameters tuning were done on a
validation subset of the training set made of 20% of the images. For all
datasets, we resize the images to $512\times 512$, with a center crop. An
additional random horizontal flip augmentation is applied at training time.

Implementation details:  For all experiments, we use a Deeplab-V3 [14]
architecture with a ResNet-101 [28] backbone pretrained on ImageNet [16], as
in [8]. For all datasets, we set a maximum threshold for the uncertainty
measure of Eq. 7 to $\tau=1e-3$. We train our model for 30 and 60 epochs per
CSS step on Pascal VOC and ADE, respectively, with an initial learning rate of
$1e-2$ for the first CSS step, and $1e-3$ for all the following ones. We
reduce the learning rate exponentially with a decay rate of $9e-1$. We use SGD
optimizer with $9e-1$ Nesterov momentum. The Local POD factor $\lambda$ is set
to $1e-2$ and $5e-4$ for intermediate feature maps and logits, respectively.
Moreover, we multiply this factor by the adaptive weighting
$\sqrt{\nicefrac{{|C^{1:t}|}}{{|C^{t}|}}}$ introduced by [31] that increases
the strength of the distillation the further we are into the continual
process. For all feature maps, Local POD is applied before ReLU, with squared
pixel values, as in [77, 18]. We use 3 scales for Local POD: $1$,
$\nicefrac{{1}}{{2}}$, and $\nicefrac{{1}}{{4}}$, as adding more scales
experimentally brought diminishing returns. We use a batch size of 24
distributed on two GPUs. Contrary to many continual models, we don’t have
access to any task id in inference, therefore our setting/strategy has to
predict a class among the set of all seen classes —a realist setting.

Classes ordering details:  For all quantitative experiments on Pascal-VOC 2012
and ADE20k, the same class ordering was used across all evaluated models. For
Pascal-VOC 2012 it corresponds to [1, 2, …, 20] and ADE20k to [1, 2, …, 150]
as defined in [8]. For continual-domain cityscapes, the order of the
domains/cities is the following: aachen, bremen, darmstadt, erfurt, hanover,
krefeld, strasbourg, tubingen, weimar, bochum, cologne, dusseldorf, hamburg,
jena, monchengladbach, stuttgart, ulm, zurich, frankfurt, lindau, and munster.

In the main paper we showcased a boxplot featuring 20 different class orders
for Pascal-VOC 2012 15-1. For the sake of reproducibility, we provide details
on these orders:

⬇ [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
[12, 9, 20, 7, 15, 8, 14, 16, 5, 19, 4, 1, 13, 2, 11, 17, 3, 6, 18, 5]  [9,
12, 13, 18, 2, 11, 15, 17, 10, 8, 4, 5, 20, 16, 6, 14, 19, 1, 7, 3]  [13, 19,
15, 17, 9, 8, 5, 20, 4, 3, 10, 11, 18, 16, 7, 12, 14, 6, 1, 2]  [15, 3, 2, 12,
14, 18, 20, 16, 11, 1, 19, 8, 10, 7, 17, 6, 5, 13, 9, 4]  [7, 13, 5, 11, 9, 2,
15, 12, 14, 3, 20, 1, 16, 4, 18, 8, 6, 10, 19, 17]  [12, 9, 19, 6, 4, 10, 5,
18, 14, 15, 16, 3, 8, 7, 11, 13, 2, 20, 17, 1]  [13, 10, 15, 8, 7, 19, 4, 3,
16, 12, 14, 11, 5, 20, 6, 2, 18, 9, 17, 1]  [3, 14, 13, 1, 2, 11, 15, 17, 7,
8, 4, 5, 9, 16, 19, 12, 6, 18, 10, 20]  [1, 14, 9, 5, 2, 15, 8, 20, 6, 16, 18,
7, 11, 10, 19, 3, 4, 17, 12, 13]  [16, 13, 1, 11, 12, 18, 6, 14, 5, 3, 7, 9,
20, 19, 15, 4, 2, 10, 8, 17]  [10, 7, 6, 19, 16, 8, 17, 1, 14, 4, 9, 3, 15,
11, 12, 2, 18, 20, 13, 5]  [7, 5, 3, 9, 13, 12, 14, 19, 10, 2, 1, 4, 16, 8,
17, 15, 18, 6, 11, 20]  [18, 4, 14, 17, 12, 10, 7, 3, 9, 1, 8, 15, 6, 13, 2,
5, 11, 20, 16, 19]  [5, 4, 13, 18, 14, 10, 19, 15, 7, 9, 3, 2, 8, 16, 20, 1,
12, 11, 6, 17]  [9, 12, 13, 18, 7, 1, 15, 17, 10, 8, 4, 5, 20, 16, 6, 14, 19,
11, 2, 3]  [3, 14, 13, 18, 2, 11, 15, 17, 10, 8, 4, 5, 20, 16, 6, 12, 19, 1,
7, 9]  [7, 5, 9, 1, 15, 18, 14, 3, 20, 10, 4, 19, 11, 17, 16, 12, 8, 6, 2, 13]
[3, 14, 6, 1, 2, 11, 12, 17, 7, 20, 4, 5, 9, 16, 19, 15, 13, 18, 10, 8]  [1,
2, 12, 14, 6, 19, 18, 17, 5, 20, 8, 4, 9, 16, 10, 3, 15, 13, 11, 7]

In the 15-1 setting, we first learn the first fifteen classes, then increment
the five remaining classes one by one. Note that the special class background
(0) is always learned during the first task.

Hardware and Code:  For each experiment, we used two Titan Xp GPUs with 12 Go
of VRAM each. The initial step $t=1$ for each setting is common to all models,
therefore we re-use the weights trained on this step. All models took less
than 2 hours to train on Pascal-VOC 2012 15-1, and less than 16 hours on
ADE20k 100-10. We distributed the batch size equally on both GPUs. All models
are implemented in PyTorch [59] and runned with half-precision for efficiency
reasons with Nvdia’s APEX library (https://github.com/NVIDIA/apex) using O1
optimization level. Our code base is based on [8]’s code
(https://github.com/fcdl94/MiB) that we modified to implement our strategy. It
is available at https://github.com/arthurdouillard/CVPR2021_PLOP.

###  A.4 Additional Experiments

Model ablation:  Table 7 shows the construction of our model component by
component on Pascal-VOC 2012 in 15-5 and 15-1. For this experiment, we train
our model on 80% of the training set and evaluate on the validation set made
of the remaining 20%. We report the mIoU at the final task (“all”) and the
average of the mIoU after each task (“avg”). We start with a crude baseline
made of solely cross-entropy (CE). Pseudo-labeling by itself increases by a
large margin performance (eg. 3.99 to 19.74 for 15-1). Applying Local POD
reduces drastically the forgetting leading to a massive gain of performance
(eg. 19.74 to 50.41 for 15-1). Finally our adaptive factor $\nu$ based on the
ratio of accepted pseudo-labels over the number of background pixels further
increases our overall results (eg. 50.41 to 52.31 for 15-1). The interest of
$\nu$ arises when PLOP faces hard images where few pseudo-labels will be
created due to an overall high uncertainty. In such a case, current classes
will be over-represented, which can in turn lead to strong bias towards new
classes (i.e. the model will have a tendency to predict one of the new classes
for every pixel). The $\nu$ factor therefore decreases the overall
classification loss on such images, and empirical results confirm its
effectiveness.

Table 7: Ablations of PLOP on the Pascal-VOC 2012 dataset in 15-5 and 15-1. Scores are measured on a validation subset made of 20% of the training set. |  15-5 (2 tasks) |  15-1 (6 tasks)
---|---|---
Model | all | avg | all | avg
CE | 13.85 | 46.91 |  3.99 | 19.37
Pseudo | 66.19 | 73.07 | 19.74 | 44.48
Pseudo + Local POD | 70.29 | 75.13 | 50.41 | 64.95
$\nu$Pseudo + Local POD | 71.43 | 75.70 | 52.31 | 65.71

Pascal-VOC 2012 Disjoint:  In the main paper, we reported results on Pascal-
VOC 2012 Overlap. For reasons mentioned previously, Overlap is a more realist
setting than Disjoint. Nevertheless, for the sake of comparison, we also
provide results in Table 8 in the Disjoint setting. While PLOP has similar
performance to MiB in 15-5 (the differences are not significant), it
significantly outperforms previous state-of-the-art methods in both 19-1 and
15-1.

Table 8: Mean IoU on the Pascal-VOC 2012 dataset for different incremental class learning scenarios, all in Disjoint. $\dagger$ denotes results from Cermelli et al.[8]. |  19-1 (2 tasks) |  15-5 (2 tasks) |  15-1 (6 tasks)
---|---|---|---
Method | 0-19 | 20 | all | avg | 0-15 | 16-20 | all | avg | 0-15 | 16-20 | all | avg
$\text{Fine Tuning}^{\dagger}$ |  5.80 | 12.30 |  6.20 |  |  1.10 | 33.60 |  9.20 |  |  0.20 |  1.80 |  0.60 |
$\text{PI}^{\dagger}$ [78] |  5.40 | 14.10 |  5.90 |  |  1.30 | 34.10 |  9.50 |  |  0.00 |  1.80 |  0.40 |
$\text{EWC}^{\dagger}$ [40] | 23.20 | 16.00 | 22.90 |  | 26.70 | 37.70 | 29.40 |  |  0.30 |  4.30 |  1.30 |
$\text{RW}^{\dagger}$ [9] | 19.40 | 15.70 | 19.20 |  | 17.90 | 36.90 | 22.70 |  |  0.20 |  5.40 |  1.50 |
$\text{LwF}^{\dagger}$ [47] | 53.00 |  9.10 | 50.80 |  | 58.40 | 37.40 | 53.10 |  |  0.80 |  3.60 |  1.50 |
$\text{LwF-MC}^{\dagger}$ [60] | 63.00 | 13.20 | 60.50 |  | 67.20 | 41.20 | 60.70 |  |  4.50 |  7.00 |  5.20 |
$\text{ILT}^{\dagger}$ [54] | 69.10 | 16.40 | 66.40 |  | 63.20 | 39.50 | 57.30 |  |  3.70 |  5.70 |  4.20 |
$\text{MiB}^{\dagger}$ [8] | 69.60 | 25.60 | 67.40 |  | 71.80 | 43.30 | 64.70 |  | 46.20 | 12.90 | 37.90 |
PLOP | 75.37 | 38.89 | 73.64 | 75.71 | 71.00 | 42.82 | 64.29 | 72.05 | 57.86 | 13.67 | 46.48 | 62.67

Pascal-VOC 2012 Overlap with more baselines:  In Table 9, we report results on
Pascal-VOC 2012 Overlap with more baselines. In addition to the models
presented in the main paper, we add a naive Fine Tuning, two continual models
based on weights constraints (PI [78] and RW [9]), and one continual model
based on knowledge distillation (LwF [47]). PLOP surpasses these methods in
all CSS scenarios.

Table 9: Mean IoU on the Pascal-VOC 2012 dataset for different incremental class learning scenarios, all in Overlap. $\dagger$ denotes results from Cermelli et al. [8], all other results are from us. |  19-1 (2 tasks) |  15-5 (2 tasks) |  15-1 (6 tasks)
---|---|---|---
Method | 0-19 | 20 | all | avg | 0-15 | 16-20 | all | avg | 0-15 | 16-20 | all | avg
$\text{Fine Tuning}^{\dagger}$ |  6.80 | 12.90 |  7.10 |  |  2.10 | 33.10 |  9.80 |  |  0.20 |  1.80 |  0.60 |
$\text{PI}^{\dagger}$ [78] |  7.50 | 14.00 |  7.80 |  |  1.60 | 33.30 |  9.50 |  |  0.00 |  1.80 |  0.50 |
$\text{EWC}^{\dagger}$ [40] | 26.90 | 14.00 | 26.30 |  | 24.30 | 35.50 | 27.10 |  |  0.30 |  4.30 |  1.30 |
$\text{RW}^{\dagger}$ [9] | 23.30 | 14.20 | 22.90 |  | 16.60 | 34.90 | 21.20 |  |  0.00 |  5.20 |  1.30 |
$\text{LwF}^{\dagger}$ [47] | 51.20 |  8.50 | 49.10 |  | 58.90 | 36.60 | 53.30 |  |  1.00 |  3.90 |  1.80 |
$\text{LwF-MC}^{\dagger}$ [60] | 64.40 | 13.30 | 61.90 |  | 58.10 | 35.00 | 52.30 |  |  6.40 |  8.40 |  6.90 |
$\text{ILT}^{\dagger}$ [54] | 67.10 | 12.30 | 64.40 |  | 66.30 | 40.60 | 59.90 |  |  4.90 |  7.80 |  5.70 |
ILT [54] | 67.75 | 10.88 | 65.05 | 71.23 | 67.08 | 39.23 | 60.45 | 70.37 |  8.75 |  7.99 |  8.56 | 40.16
$\text{MiB}^{\dagger}$ [8] | 70.20 | 22.10 | 67.80 |  | 75.50 | 49.40 | 69.00 |  | 35.10 | 13.50 | 29.70 |
MiB [8] | 71.43 | 23.59 | 69.15 | 73.28 | 76.37 | 49.97 | 70.08 | 75.12 | 34.22 | 13.50 | 29.29 | 54.19
PLOP | 75.35 | 37.35 | 73.54 | 75.47 | 75.73 | 51.71 | 70.09 | 75.19 | 65.12 | 21.11 | 54.64 | 67.21

## References

  * [1] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars.  Memory aware synapses: Learning what (not) to forget.  In Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2018.
  * [2] V. Badrinarayanan, A. Kendall, and R. Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2017.
  * [3] Eden Belouadah and Adrian Popescu.  Il2m: Class incremental learning with dual memory.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
  * [4] Eden Belouadah and Adrian Popescu.  Scail: Classifier weights scaling for class incremental learning.  In Proceedings of the IEEE Winter Conference on Application of Computer Vision (WACV), 2020.
  * [5] Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez.  Zero-shot semantic segmentation.  In Advances in Neural Information Processing Systems (NeurIPS), 2019\.
  * [6] Holger Caesar, Jasper R. R. Uijlings, and Vittorio Ferrari.  Coco-stuff: Thing and stuff classes in context.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [7] Francisco M. Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari.  End-to-end incremental learning.  In Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2018.
  * [8] Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulò, Elisa Ricci, and Barbara Caputo.  Modeling the background for incremental learning in semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [9] Arslan Chaudhry, Puneet Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr.  Riemannian walk for incremental learning: Understanding forgetting and intransigence.  Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2018.
  * [10] Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny.  Efficient lifelong learning with a-gem.  In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
  * [11] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H.S. Torr, and Marc’Aurelio Ranzato.  On tiny episodic memories in continual learning.  In International Conference on Machine Learning (ICML) Workshop, 2019.
  * [12] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2018.
  * [13] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  In IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2018.
  * [14] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  In arXiv preprint library, 2017.
  * [15] M. Cordts, M. Omran, S. Ramos, T. Reheld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [16] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
  * [17] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa.  Learning without memorizing.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [18] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle.  Podnet: Pooled outputs distillation for small-tasks incremental learning.  In Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2020.
  * [19] Arthur Douillard, Eduardo Valle, Charles Ollion, and Matthieu Cord.  Insights from the future for continual learning.  In arXiv preprint library, 2020.
  * [20] Mark Everingham, S. M. Ali Eslami, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman.  The pascal visual object classes challenge: A retrospective.  In International Journal of Computer Vision (IJCV), 2015.
  * [21] Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra.  PathNet: Evolution Channels Gradient Descent in Super Neural Networks.  arXiv preprint library, 2017.
  * [22] Jonathan Frankle and Michael Carbin.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
  * [23] Robert French.  Catastrophic forgetting in connectionist networks.  Trends in cognitive sciences, 1999.
  * [24] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu.  Dual attention network for scene segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [25] Siavash Golkar, Michael Kagan, and Kyunghyun Cho.  Continual learning via neural pruning.  Advances in Neural Information Processing Systems (NeurIPS) Workshop, 2019.
  * [26] Tyler L. Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan.  Remind your neural network to prevent catastrophic forgetting.  In Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2020.
  * [27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Spatial pyramid pooling in deep convolutional networks for visual recognition.  In Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2014.
  * [28] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [29] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean.  Distilling the knowledge in a neural network.  In Advances in Neural Information Processing Systems (NeurIPS) Workshop, 2015.
  * [30] Qibin Hou, Li Zhang, Ming-Ming Cheng, and Jiashi Feng.  Strip pooling: Rethinking spatial pooling for scene parsing.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [31] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin.  Learning a unified classifier incrementally via rebalancing.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [32] Z. Huang, X. Wang, L. Huang, C. Huang, Y. Wei, and W. Liu.  Ccnet: Criss-cross attention for semantic segmentation.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
  * [33] Zilong Huang, Xinggang Wang, Yunchao Wei, Lichao Huang, Humphrey Shi, Wenyu Liu, and Thomas S. Huang.  Ccnet: Criss-cross attention for semantic segmentation.  2020\.
  * [34] Steven C.Y. Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen.  Compacting, picking and growing for unforgetting continual learning.  In Advances in Neural Information Processing Systems (NeurIPS), 2019\.
  * [35] Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, and Cordelia Schmid.  Memory-efficient incremental learning through feature adaptation.  In Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2020.
  * [36] Naoki Kato, Toshihiko Yamasaki, and Kiyoharu Aizawa.  Zero-shot semantic segmentation via variational mapping.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshop, 2019.
  * [37] Ronald Kemker and Christopher Kanan.  Fearnet: Brain-inspired model for incremental learning.  In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
  * [38] Ronald Kemker, Marc McClure, Angelina Abitino, Tyler L. Hayes, and Christopher Kanan.  Measuring catastrophic forgetting in neural networks.  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2018.
  * [39] Dahyun Kim, Jihwan Bae, Yeonsik Jo, and Jonghyun Choi.  Incremental learning with maximum entropy regularization: Rethinking forgetting and intransigence.  arXiv preprint library, 2019.
  * [40] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the National Academy of Sciences, 2017.
  * [41] Vinay Kumar Verma, Gundeep Arora, Ashish Mishra, and Piyush Rai.  Generalized zero-shot learning via synthesized examples.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [42] C. H. Lampert, H. Nickisch, and S. Hermeling.  Learning to detect unseen object classes by between-class attribute transfer.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
  * [43] Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce.  Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories.  Object Categorization: Computer and Human Vision Perspectives, Cambridge University Press, 2006.
  * [44] Dong-Hyun Lee.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In International Conference on Machine Learning (ICML) Workshop, 2013.
  * [45] Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong.  Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting.  Proceedings of the International Conference on Learning Representations (ICLR), 2019.
  * [46] Yunsheng Li, Lu Yuan, and Nuno Vasconcelos.  Bidirectional learning for domain adaptation of semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [47] Z. Li and D. Hoiem.  Learning without forgetting.  Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2016.
  * [48] Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun.  Mnemonics training: Multi-class incremental learning without forgetting.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [49] Vincenzo Lomonaco and Davide Maltoni.  Core50: a new dataset and benchmark for continuous object recognition.  In Annual Conference on Robot Learning, 2017.
  * [50] Vincenzo Lomonaco, Davide Maltoni, and Lorenzo Pellegrini.  Rehearsal-free continual learning over small non-i.i.d. batches.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop, 2020.
  * [51] J. Long, E. Shelhamer, and T. Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [52] David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems (NeurIPS), 2017.
  * [53] Sachin Mehta, Mohammad Rastegari, Anat Caspi, Linda G. Shapiro, and Hannaneh Hajishirzi.  Efficient spatial pyramid of dilated convolutions for semantic segmentation.  In Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2018.
  * [54] Umberto Michieli and Pietro Zanuttigh.  Incremental learning techniques for semantic segmentation.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshop, 2019.
  * [55] H. Noh, S. Hong, and B. Han.  Learning deconvolution network for semantic segmentation.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015.
  * [56] Firat Ozdemir, Philipp Fuernstahl, and Orcun Goksel.  Learn the new, keep the old: Extending pretrained models with new anatomy and images.  In International Conference on Med- ical Image Computing and Computer-Assisted Intervention, 2018.
  * [57] Firat Ozdemir and Orcun Goksel.  Extending pretrained segmentation networks with additional anatomical structures.  In International journal of computer assisted radiology and surgery, 2019.
  * [58] Sangyong Park and Yong Seok Heo.  Knowledge distillation for semantic segmentation using channel and spatial correlations and adaptive cross entropy.  In Sensors, 2020.
  * [59] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  In Advances in Neural Information Processing Systems (NeurIPS) Workshop, 2017.
  * [60] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert.  icarl: Incremental classifier and representation learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [61] Anthony Robins.  Catastrophic forgetting, rehearsal and pseudorehearsal.  Connection Science, 1995.
  * [62] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  arXiv preprint library, 2014.
  * [63] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), 2015.
  * [64] Antoine Saporta, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez.  Esl: Entropy-guided self-supervised learning for domain adaptation in semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop, 2020.
  * [65] Pierre Sermanet, David Eigen, Xiang Zhang, Michaë Mathieu, Rob Fergus, and Yann LeCun.  Overfeat: Integrated recognition, localization and detection using convolutional networks.  In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
  * [66] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim.  Continual learning with deep generative replay.  In Advances in Neural Information Processing Systems (NeurIPS), 2017\.
  * [67] Andrew Tao, Karan Sapra, and Bryan Catanzaro.  Hierarchical multi-scale attention for semantic segmentation.  In arXiv preprint library, 2020.
  * [68] Sebastian Thrun.  Lifelong learning algorithms.  In Springer Learning to Learn, 1998.
  * [69] Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez.  Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [70] Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen.  Axial-deeplab: Stand-alone axial-attention for panoptic segmentation.  In Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2020.
  * [71] Kai Wang, Luis Herranz, Anjan Dutta, and Joost van de Weijer.  Bookworm continual learning: beyond zero-shot learning and continual learning.  In Proceedings of the IEEE European Conference on Computer Vision (ECCV) Workshop, 2020.
  * [72] Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi.  Supermasks in superposition for continual learning.  In Advances in Neural Information Processing Systems (NeurIPS), 2020\.
  * [73] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu.  Large scale incremental learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [74] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang.  Lifelong learning with dynamically expandable networks.  In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
  * [75] Yuhui Yuan, Xilin Chen, and Jingdong Wang.  Object-contextual representations for semantic segmentation.  In Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2020.
  * [76] Yuhui Yuan and Jingdong Wang.  Ocnet: Object context network for scene parsing.  In arXiv preprint library, 2018.
  * [77] Sergey Zagoruyko and Nikos Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  Proceedings of the International Conference on Learning Representations (ICLR), 2016.
  * [78] Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In International Conference on Machine Learning (ICML), 2017.
  * [79] Hang Zhang, Kristin J. Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi, and Amit Agrawal.  Context encoding for semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [80] Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Zhi Zhang, Haibin Lin, Yue Sun, Tong He, Jonas Muller, R. Manmatha, Mu Li, and Alexander Smola.  Resnest: Split-attention networks.  In arXiv preprint library, 2020.
  * [81] Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shutao Xia.  Maintaining discrimination and fairness in class incremental learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [82] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia.  Pyramid scene parsing network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [83] Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen Change Loy, Dahua Lin, and Jiaya Jia.  Psanet: Point-wise spatial attention network for scene parsing.  In Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2018.
  * [84] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [85] Peng Zhou, Long Mai, Jianming Zhang, Ning Xu, Zuxuan Wu, and Larry S. Davis.  M2kd: Multi-model and multi-level knowledge distillation for incremental learning.  arXiv preprint library, 2019.
  * [86] Yang Zou, Zhiding Yu, BVK Vijaya Kumar, and Jinsong Wang.  Unsupervised domain adaptation for semantic seg- mentation via class-balanced self-training.  In Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2018.