# Continual Learning for Image Segmentation with Dynamic Query

Weijia Wu∗, Yuzhong Zhao∗, Zhuang Li, Lianlei Shan, Hong Zhou†, Mike Zheng
Shou W. Wu and H. Zhou are with the Zhejiang University, Hangzhou, 310058,
China. (e-mail<EMAIL_ADDRESS>zhouh@mail.bme.zju.edu.cn).Y. Zhao and L.
Shan are with the School of Computer Science and Technology, University of
Chinese Academy of Sciences, Beijing, 101408, China.Z. Li is with the Kuaishou
Technology, Beijing, China.M. Shou is with the National University of
Singapore, Singapore.$*$Equal contribution.$\dagger$Corresponding author (Z.
Hong).Copyright © 2023 IEEE. Personal use of this material is permitted.
However, permission to use this material for any other purposes must be
obtained from the IEEE by sending an email to pubs-permissionsieee.org.

###### Abstract

Image segmentation based on continual learning exhibits a critical drop of
performance, mainly due to catastrophic forgetting and background shift, as
they are required to incorporate new classes continually. In this paper, we
propose a simple, yet effective Continual Image Segmentation method with
incremental Dynamic Query (CiSDQ), which decouples the representation learning
of both old and new knowledge with lightweight query embedding. CiSDQ mainly
includes three contributions: 1) We define dynamic queries with adaptive
background class to exploit past knowledge and learn future classes naturally.
2) CiSDQ proposes a class/instance-aware Query Guided Knowledge Distillation
strategy to overcome catastrophic forgetting by capturing the inter-class
diversity and intra-class identity. 3) Apart from semantic segmentation, CiSDQ
introduce the continual learning for instance segmentation in which instance-
wise labeling and supervision are considered. Extensive experiments on three
datasets for two tasks (_i.e.,_ continual semantic and instance segmentation
are conducted to demonstrate that CiSDQ achieves the state-of-the-art
performance, specifically, obtaining 4.4% and 2.9% mIoU improvements for the
ADE 100-10 (6 steps) setting and ADE 100-5 (11 steps) setting.

###### Index Terms:

Image Segmentation, Continual Learning, Dynamic Query, Dense matching
strategy, Transformer.

##  I Introduction

Image segmentation, including semantic segmentation and instance segmentation,
is a fundamental task in computer vision. In recent years, data-driven
segmentation networks [1, 2] have made extraordinary progress with fully-
supervised learning of fixed data, where all classes are fixed and known
beforehand and learned at once. However, in a real-world system, it is
preferable that one model can dynamically update its knowledge and extend to
segment new classes without retraining from scratch. To reach the setup,
previous works [3, 4, 5] propose to introduce the class incremental learning
for semantic segmentation, named continual semantic segmentation (CSS).
Different from these works that focus on semantic segmentation, in this paper,
we try to establish a generic continual image segmentation task, which
includes semantic segmentation and instance segmentation.

Continual learning-based image segmentation (or continual image segmentation,
CIS) is a task that requires one model incrementally learn and segment newly
arriving class objects while not catastrophically forgetting the past learned
classes (old classes). Similar to continual semantic segmentation, continual
image segmentation also faces two main challenges. The first one is the
catastrophic forgetting [6, 7], where the model quickly fits the new data
distribution and loses the old discriminative representation. During continual
learning, the new training data and class annotation is only used, and the old
classes are not available usually be treated as background. The network
usually tends to catastrophically and abruptly forget previously learned
knowledge (old classes) when learning new information (new classes). Existing
continual semantic segmentation methods [3, 5] typically adopt inefficient and
explicit multiple strategies to tackle the problem. For instance, PLOP [3]
naively adopts a spatial distillation loss without fine-grained relationship
learning of intra-class, inter-class, each pixel. SSUL [5] adopts three
strategies, _i.e.,_ pseudo-labeling, exemplar memory, and model freezing to
cover the challenge, but pseudo-labeling usually shows extremely instability
and exemplar memory requires the extra cost memory of the network.

Figure 1:  Illustration of CiSDQ. Dynamic query decouples the representation
learning of old and new knowledge with dynamically increasing lightweight
queries. $Q_{t}$ , $C_{t}$ and $M_{t}$ refers to the Query group, Classes, and
Masks in $t$-th step, respectively.

The second challenge, inherited from continual semantic segmentation, is the
background shift [3, 5]. Different from the fixed and certain background
pixels of traditional image segmentation, the background pixel for continual
semantic segmentation belongs to three categories: potential future object
classes, past object classes, and the true background. For instance, if an
image contains three object classes, _i.e.,_ sofa, person, dog, where only
class sofa mask annotation is available and class person and dog belong to the
past classes without annotation. If the model naively treats all unknown
background pixels to the true background pixel class, the old and future
knowledge will be further damaged and forgotten. Some works [3, 5] try to
solve background shift with pseudo-labeling from the old model or other off-
the-shelf saliency-map detectors. However, the predicted pseudo-labeling from
the model usually present instability and worse performance, and it is
impossible to predict all potential objects for future classes.

In this paper, we propose a simple, yet efficient Continual Image Segmentation
method with incremental Dynamic Query (CiSDQ), which decouples the
representation learning of both old and new knowledge with class/instance-
aware lightweight query embedding. Inspired by query-based modeling for vision
tasks [2, 8, 9, 10, 11], we propose a new concept dynamic query, enabling the
model dynamically update its knowledge and extend to segment new classes via
dynamically increasing query embedding of new classes or instances. As shown
in Fig. 1, we adopt one query embedding to represent one class or instance and
dynamically increase the number of query embeddings to extend new classes
without retraining from scratch. CiSDQ main includes two advantages: 1)
Different queries are used to represent different classes at each step to
handle catastrophic forgetting problem. At step $t$, queries
$\\{Q_{0},Q_{1}...Q_{t-1}\\}$ will be used to retain the past knowledge of old
classes (_i.e.,_ $\\{\mathcal{C}_{0},\mathcal{C}_{1}...\mathcal{C}_{t-1}\\}$),
newly added queries $\\{Q_{t}\\}$ are used to learn new appeared knowledge
(classes) $\\{\mathcal{C}_{t}\\}$. 2) Adaptive background concept for each
query of each step is proposed to solve background shift problem. Different
background definitions
$\\{\mathcal{B}_{0},\mathcal{B}_{1}...\mathcal{B}_{t-1}\\}$ for different
queries $\\{Q_{0},Q_{1}...Q_{t-1}\\}$ are used to cover different background
representation at each step. CNN-based multi-classification per-pixel in one
feature map [1] is transformed to a set of binary-classification in multi-maps
[2], thus there are $t$ different definitions of background for $t$ predicted
masks each associated with a single category. For instance, at step $t-1$, if
only class sofa mask annotation is available, person, dog pixel region as the
unknown region is defined as the background $\mathcal{B}_{t-1}$. But at step
$t$, person as newly added class is defined as the positive sample, but sofa
mask is not available, belonging to the background class $\mathcal{B}_{t}$.
Thus the adaptive background does not give confusing supervision, while there
is no background shift.

Besides, we propose a query-guided knowledge distillation (Query G-KD) to
further alleviate catastrophic forgetting. Different from previous knowledge
distillation of semantic segmentation, Query G-KD can achieve more precise
class/instance-aware knowledge distillation via capturing the inter-class
diversity and intra-class identity, while each query is responsible for one
class. Similarly, the knowledge of inter-instance diversity and intra-instance
identity can be retained with Query G-KD for continual instance segmentation
tasks. To summarize, our contributions are four-folds:

  * •

We propose a simple, yet efficient unified Continual Image Segmentation method
with incremental Dynamic Query, namely CiSDQ, including a adaptive background
concept for each class/instance, which decouples the representation learning
of old and new knowledge with dynamically increasing lightweight query
embedding.

  * •

CiSDQ introduces a Query Guided Knowledge Distillation (Query G-KD) strategy
to overcome catastrophic forgetting via capturing the inter-class diversity
and intra-class identity.

  * •

CiSDQ introduces continual learning for instance segmentation in which
instance-wise labeling and supervision are considered. We also solve semantic
and instance segmentation in a unified framework.

  * •

Experiments are conducted on continual semantic segmentation and continual
instance segmentation on three datasets, respectively. And CiSDQ achieves
state-of-the-art performance with up to $10\%$ improvements than previous
works.

##  II Related Work

###  II-A Image Segmentation

Image Segmentation mainly includes two tasks: semantic and instance
segmentation tasks. For semantic segmentation, Some early works attempted [12,
13, 14, 15, 16] to explore the use of certain image priors, such as Gaussian
Mixture Modeling, to segment images or videos. Fully Convolutional Networks
(FCN) [17] is the first deep learning based work to perform pixel-to-pixel
semantic classification in an end-to-end manner. After that, researchers
focused on different aspects for improving semantic segmentation, _e.g.,_
contextual relationships [18, 19, 20, 21], Spatial pyramid pooling [22, 23,
24], and image pyramid [25, 26]. In recent years, several works [27, 28] try
to adopt transformer architectures for semantic segmentation. Segformer [27]
design a novel hierarchically structured Transformer encoder to output multi-
scale features. For instance segmentation [29], Mask R-CNN [30], as the
representation, extend Faster R-CNN [31] by adding a branch for predicting an
object mask in parallel with the existing branch for bounding box recognition.
DETR [32] proposed to segment instances with mask attention, which is more
natural. Maskformer [28] proposed mask classification modeling, which solves
semantic- and instance-level segmentation tasks in a unified manner. GenPromp
[33], DiffuMask [34] and DatasetDM [35] proposed to use diffusion model for
enhancing semantic segmentation tasks, supporting open-set segmentation. In
this paper, we try to utilize the advantage of a transformer for continual
image segmentation.

###  II-B Class Continual Learning

Class continual learning method [36, 37, 38, 39, 40, 41, 42] mainly focuses on
the classification task, and alleviates catastrophic forgetting [43], which is
caused by the domain distribution change from the training data. To solve this
problem, most works [44, 45, 46] try to maintain the performance of old
classes with knowledge distillation [47, 48, 49], and adversarial training
[50, 51]. In recent years, inspired by the success of transformer architecture
[52, 32] in computer vision, some works try to solve class incremental
learning with transformer [53, 54, 55, 56]. MEta-ATtention [53] proposed to
use a pre-trained ViT to new tasks without sacrificing performance on already
learned tasks. DyTox [55] designs a shared transformer encoder and decoder to
cover incremental learning with a dynamic expansion of special tokens. TwF
[56] proposed a hybrid approach building upon a fixed pre-trained sibling
network, which continuously propagates the knowledge inherent from the
previous task through a layer-wise loss term. However, no one tries to adopt
transformer to solve continual semantic and instance segmentation.

###  II-C Continual Image Segmentation

Semantic Segmentation. Except for catastrophic forgetting, continual semantic
segmentation also faces another challenge, _i.e.,_ background shift [3, 5].
Most existing methods solve the two challenges with rehearsal-based [57, 58],
pesudo label [59, 3, 60, 61], and knowledge distillation [3, 62]. MiB [62]
propose a new objective function and introduce a specific classifier
initialization strategy to solve the background shift. PLOP [3] proposed a
multi-scale pooling distillation and entropy-based pseudo-labelling of the
background to deal with the two problems. RCIL [4] proposed a structural re-
parameterization to decouple the representation learning of both old and new
knowledge for solving catastrophic forgetting. SSUL [5] solves background
shift and catastrophic forgetting with three strategies, _i.e.,_ unknown
classes in background class, freeze backbone, and exemplar memory. To avoid
forgetting old knowledge, MicroSeg [63] first splits the given image into
hundreds of segment proposals with a proposal generator. Those segment
proposals with strong objectness from the background are then clustered and
assigned newly-defined labels during the optimization. Based on Segformer
[27], SATS [64] design a knowledge distillation for transformer-based semantic
segmentation. However, the above method still can not achieve preferable
performance, while they can not solve the two challenges entirely. Different
from previous methods, We propose an incremental dynamic query and query-
guided knowledge distillation to decouple the representation learning of old
and new knowledge.

Instance Segmentation. Despite enormous progress in the continual learning and
instance segmentation tasks, almost no work try to solve continual problem in
instance segmentation, while continual instance segmentation is also important
And there exists a related task, _i.e.,_ incremental few-shot instance
segmentation. iMTFA [65] design the first approach for incremental few-shot
instance segmentation, which match these class embeddings at the RoI-level
using cosine similarity. iFS-RCNN [66] leverages Bayesian learning to address
a paucity of training examples of new classes. But the setting of incremental
few-shot instance segmentation is different with continual instance
segmentation in two parts: 1). Continual instance segmentation requires model
to learn feature of new class during multi-steps, while incremental few-shot
instance segmentation only includes two-steps. 2). Continual instance
segmentation provides abundant data of new classes, while incremental few-shot
instance segmentation only training on a few data. Besides, different from the
above two methods, the proposed dynamic query and query-guided knowledge
distillation can decouple the representation learning of old and new
knowledge, which can deal with catastrophic forgetting and background shift
challenges effectively.

(a) Dynamic Query for Continual Image Segmentation.

(b) Class/Instance-Aware Query-Guided KD.

Figure 2: Illustration for Dynamic Query and Query-Guided Knowledge
Distillation. (a) Incremental Dynamic Query decouples the representation
learning of old and new knowledge with dynamically increasing lightlweight
query embedding; (b) Compared with previous knowledge distillation [3, 5, 4]
over the whole feature distribution, the proposed Query-Guided KD develop
class/instance level distillation to overcome catastrophic forgetting
challenge.

##  III Notations and Problem Setting

Similar to previous works [3, 5], for the continual learning scenario, each
training on the newly added classes and dataset as a _step_ , where existing
$T$ steps. For $t$-th step, given a base model $f^{t-1}_{\bm{\theta}}$ with
parameter $\theta_{t-1}$ trained on
$\\{\mathcal{D}_{0},\mathcal{D}_{1}...\mathcal{D}_{t-1}\\}$ with
$\\{\mathcal{C}_{0},\mathcal{C}_{1}...\mathcal{C}_{t-1}\\}$ classes, the model
is expected to segment $\sum_{i=0}^{t}\mathcal{C}_{i}$ classes after training
on the newly added dataset $\mathcal{D}_{t}$ with extra $\mathcal{C}_{t}$ new
classes, where the training data of old classes are not accessible. The
network usually quickly fits the new data distribution with extra
$\mathcal{C}_{t}$ new classes, and cause serious performance drop for old
classes. The challenge in continual image segmentation is named catastrophic
forgetting. In this paper, existing $T$ different query groups
$\\{Q_{0},Q_{1}...Q_{T}\\}$ are responsible for
$\\{\mathcal{C}_{0},\mathcal{C}_{1}...\mathcal{C}_{T}\\}$ classes at each
step. Therefore, we can decouple the representation learning of old and new
knowledge for step $t$ from two aspects: 1) Query groups
$\\{Q_{0},Q_{1}...Q_{t-1}\\}$ will be frozen to retain the past knowledge of
old classes (_i.e.,_ $\sum_{i=0}^{t-1}\mathcal{C}_{i}$), newly added queries
$\\{Q_{t}\\}$ are used to learn new appeared knowledge (classes)
$\\{\mathcal{C}_{t}\\}$; 2) Knowledge distillation [3, 4, 49] is a commonly
used technique to overcome the challenge of catastrophic forgetting, where it
typically involves directly distilling and aligning the features between
$t$-th step and $(t-1)$-th to avoid a serious performance drop for old classes
when learning new classes. Different from previous knowledge distillation,
query-guided knowledge distillation is more precise to transfer the knowledge
from $f_{\bm{\theta}}^{t-1}$ to $f^{t}_{\bm{\theta}}$ at each class/instance
level with query groups $\sum_{i=0}^{t-1}Q_{i}$, where the inter-
class/instance diversity and intra-class/instance identity can be captured, as
shown in Fig. 2. Specifically, existing knowledge distillation methods often
directly extract features from the backbone and decoder for distillation.
However, features at $t-1$ step only contain characteristics of old classes,
when features at step $t-1$ contain characteristics of both old and new
classes. Forcing distillation on these two features can undermine the learning
of new class features because features at step $t-1$ do not encompass the
characteristics of new classes.

For $t$-th step, we use $\mathcal{D}_{t}$ = $\\{\bm{x}_{t},\bm{y}_{t}\\}$ to
denote the current training set, where $\bm{x}_{t}\in\mathcal{X}$ denotes the
input image, and the $\bm{y}_{t}\in\mathcal{Y}_{t}$ denotes the corresponding
ground-truth (GT) pixel labels. The label space
$\mathcal{Y}_{t}=\mathcal{B}_{t}\cup\mathcal{C}_{t}$ consists of the current
newly added classes $\mathcal{C}_{t}$ and the dummy background class
$\mathcal{B}_{t}$. The $\mathcal{B}_{t}$ label may includes past classes
$\mathcal{C}_{0:t-1}$, the future classes $\mathcal{C}_{t+1:T}$, or the true
background pixels $\mathcal{\widetilde{B}}_{t}$. Therefore, the dummy
background class will cause a serious performance drop, while the potential
foreground classes ( $\mathcal{C}_{0:t-1}$, $\mathcal{C}_{t+1:T}$) is
uncertain or misleading. The challenge in continual image segmentation is
named background shift. All previous works try to predict the potential
foreground classes, _i.e.,_ $\mathcal{C}_{0:t-1}$, $\mathcal{C}_{t+1:T}$ with
the pseudo label from old model [62, 3] or other off-the-shelf saliency-map
detector [5]. But the predicted pseudo-labeling from the model usually
presents instability and worse performance, and it is impossible to predict
all potential objects for future classes. In our work, we introduce a new
concept, “Adaptive” background class, set
$\mathcal{B}_{t}=\\{c_{t}\\}\cup\mathcal{\widetilde{B}}_{t}$, where
$\\{c_{t}\\}$ refer to the potential foreground classes and is considered as
background for training at $t$-th step. $c_{t}$ is a variable and different
for each class in a different step. For instance, at step $t-1$, if only class
sofa mask annotation is available, person, dog pixel region as the unknown
regions belong to $\\{c_{t-1}\\}$. But at step $t$, person as newly added
class is defined as the positive sample, but sofa mask is not available,
belonging to the ‘background’ class $\\{c_{t}\\}$. The adaptive background
concept can completely solve the negative impact from background shift, while
we do not need to give a certain definition for what is real background in
each step.

##  IV Approach

###  IV-A Dynamic Query

Fig. 2 presents the whole framework for incremental Dynamic Query, which
describes how to decouple the retaining of old knowledge and learning of new
knowledge. All previous CNN-based works [3, 5] all adopt Deeplab v3 [1] as the
base framework, which views segmentation task as the multi-classification per-
pixel problem. It is difficult to learn new classes and keep discrimination
for old classes simultaneously for such a framework, where all representations
are stored in the same embedding space. Different from these works, we adopt
query-based mask classification architecture, Mask2former [2], which uses a
set of $C$-dimensional feature vectors (_i.e.,_ “query”) to predict a set of
binary masks each associated with a single category. Therefore, different
representations from different categories or instances can be decoupled into
different feature embedding with a different query.

####  IV-A1 Incremental Dynamic Query Queue

To enable continual learning in such architecture, we modify the fixed set of
learned queries of Transformer decoder [2] to dynamic incremental increasing
sets. For the standard setting, given a input image $I$ and the base model
$f_{\bm{\theta}}$ with parameter $\theta$, the predicted classes is denoted as
$\\{\mathcal{C},\mathcal{B}\\}=f_{\bm{\theta}}(Q|I)$, where $Q$ is a fixed set
of queries, _i.e.,_ $N$ $C$-dimensional learnable feature vectors.
$\mathcal{C}$ and $\mathcal{B}$ are the predicted foreground classes and
background class.

The fixed set of queries is transformed to a dynamic incremental increasing
query queue of $C$-dimensional vectors. Specially, in step $0$, $Q_{0}$
include $n^{q}_{0}$ $C$-dimensional vectors, representing the base class set
$\mathcal{C}_{0}$ with $n^{c}_{0}$ classes, where $n^{q}_{0}$ $>$ $n^{c}_{0}$.
And we model the current background class as
$\mathcal{B}_{0}=\\{c_{0}\\}\cup\mathcal{\widetilde{B}}_{0}$, where
$\widetilde{B}_{0}$ is the true background pixel, and $c_{0}$ is the potential
foreground pixel defined as ‘background’, _e.g.,_ the region of past class
person. For the subsequent learning steps, the model is supposed to segment
newly added classes
$\\{\mathcal{C}_{1},\mathcal{C}_{2},\mathcal{C}_{3}...\\}$. Corresponding,
newly query sets $\\{Q_{1},Q_{2},Q_{3}...\\}$ will be added to guide the
representation learning of newly added classes. In the training phase of step
$t$, query embedding sets $Q_{0:t-1}$ will be frozen to avoid forgetting old
knowledge. And $Q_{t}$ is used to guide the network to learn new knowledge of
newly added classes. Finally, we model the continual learning at $t$-th step
as:

$\displaystyle\\{\mathcal{C}_{0:t},\mathcal{B}_{0:t}\\}=f_{\bm{\theta}}({{\color[rgb]{0.46484375,0.53515625,0.6015625}\definecolor[named]{pgfstrokecolor}{rgb}{0.46484375,0.53515625,0.6015625}Q_{0:t-1}}},Q_{t}|I)\,.$
(1)

Actually, only $\\{\mathcal{C}_{t},\mathcal{B}_{t}\\}$ are supervised by
manual annotation, while other annotations of
$\\{\mathcal{C}_{0:t-1},\mathcal{B}_{0:t-1}\\}$ are not available.
Class/instance-aware query guided knowledge distillation (Section IV-B) is
used to further retain the old knowledge of past classes.

####  IV-A2 Adaptive Background for Different Query Set

Section IV-A1 have provided basic information concerning the adaptive
backgrounds $\\{\mathcal{B}_{0},\mathcal{B}_{1}...\mathcal{B}_{T}\\}$ for
different step. Actually, as shown in Fig. 2, different from foreground
classes $\mathcal{C}_{0:T}$, we do not use query to guide predict background
mask, where a pixel belongs to background region $\mathcal{B}_{t}$ if it is
not in foreground region $\mathcal{C}_{t}$ at step $t$. During continual
learning steps, the definition of background will change adaptively with the
changing of foreground classes. Therefore, the background shift problem can be
solved naturally, while it is unnecessary to give a certain definition that
which region is the true background, and no confused supervision.

####  IV-A3 Independent Matching for Each Query Set

Independent bipartite matching is designed for each query set at different
learning step, which includes advantages: 1) Avoiding the mutual interference
between old and new knowledge. 2) Enabling adaptive background.

Standard bipartite matching between ground truth $\bm{y}^{i}$ and a
predictions $\hat{\bm{y}}^{\sigma(i)}$ with index $\sigma(i)$ computes:

$\hat{\sigma}=\operatorname*{arg\,min}_{\sigma\in\mathfrak{S}_{M}}\sum_{i}^{M}{\mathcal{L}}_{\rm
match}(\bm{y}^{i},\hat{\bm{y}}^{\sigma(i)}),$ (2)

where $M$ is the number of predictions, the same as the number of queries.
${\mathcal{L}}_{\rm match}(\bm{y}^{i},\hat{\bm{y}}^{\sigma(i)})$ is a pair-
wise matching cost [32]. For step $t$, we modify the matching to:

$\hat{\sigma}=\operatorname*{arg\,min}_{\sigma\in\mathfrak{S}_{M_{t}}}\sum_{i}^{M_{t}}{\mathcal{L}}_{\rm
match}(\bm{y}^{i}_{t},\hat{\bm{y}}^{\sigma(i)}_{t}),$ (3)

where $M_{t}$, $\bm{y}^{i}_{t}$, and $\hat{\bm{y}}^{\sigma(i)}_{t}$ refer to
the number of newly added query, the ground truth of new classes, and the
corresponding prediction. Therefore, only the prediction of the newly added
classes is supervised with annotation for the network. For the knowledge of
old classes, we adopt freezing query embedding and query guided knowledge
distillation (Sec. IV-B) to retain them.

Figure 3: Illustration of Query Guided Knowledge Distillation. Query G-KD
mainly includes three parts: 1) Multi-Scale KD with local POD [3, 49]
$\mathcal{L}_{pod}$ for feature from backbone and pixel-decoder; 2)
Class/Instance-Aware Query KD
$\\{\mathcal{L}_{q}^{1},\mathcal{L}_{q}^{2},\dots,\mathcal{L}_{q}^{L}\\}$ for
query embedding vectors from multi-layers of transformer decoder; 3)
Class/Instance-Aware Prediction KD for network prediction, _i.e.,_ class
$\mathcal{L}_{c}$ and mask $\mathcal{L}_{m}$.

###  IV-B Query Guided Knowledge Distillation

As shown in Fig. 2, existing knowledge distillation methods [3, 4, 49] try to
solve catastrophically forgetting via matching global statistics distribution
at different feature levels between the old and current models. Given an
embedding tensor ${\bm{x}}$, whose size is $H\times W\times C$. These methods
usually focus on how to extract a better representation of the embedding
tensor with a mapping function $\Phi$ (_e.g.,_ , PCKD [49], POD [4]), then
train the model $f^{t}_{\bm{\theta}}$ via minimizing the L2 distance between
the two distribution:

$\mathcal{L}(\Theta^{t})=\frac{1}{L}\sum_{l=1}^{L}\left\|\Phi({\bm{x}}^{t}_{l})-\Phi({\bm{x}}^{t-1}_{l})\right\|^{2}\,,$
(4)

where $L$ denotes the number of embedding tensors. Actually, the global
statistics match paradigm is crude and unreasonable with two drawbacks: 1)
Directly matching will damage the representation learning of new classes,
while ${\bm{x}}^{t-1}_{l}$ from the previous model $f^{t-1}_{\bm{\theta}}$ do
not contain the representation. 2) Inter-class diversity and intra-class
identity can not be captured for indiscriminate matching.

To solve the two drawbacks, Class/Instance-Aware Query Guided Knowledge
Distillation (Query G-KD) is proposed. Fig. 3 presents the whole illustration
of Query G-KD, which includes three KD methods, _i.e.,_ Multi-Scale KD with
local POD, Class/Instance-Aware Query KD, and Class/Instance-Aware Prediction
KD.

####  IV-B1 Class/Instance-Aware Query KD

At step $t$, given a $C$-dimensional learnable query embedding
${\bm{q}}\in\\{Q_{0},Q_{1},...,Q_{t-1}\\}$ that is responsible for a single
old category $\mathcal{C}_{0:t-1}$, we can obtain $L$ query embedding vectors
$\\{{\bm{q}}_{1},{\bm{q}}_{2}...{\bm{q}}_{L}\\}$ from the output of each layer
at transformer decoder for all $L$ layers, as show in Fig. 3. In the training
stage, The original query embedding ${\bm{q}}$ is frozen and extracted query
embedding vectors $\\{{\bm{q}}_{1},{\bm{q}}_{2}...{\bm{q}}_{L}\\}$ are used
for distillation. Then, the query knowledge distillation loss function
$\mathcal{L}_{q}$ for the intermediate layers can be denoted as

$\mathcal{L}_{q}(\Theta^{t})=\frac{1}{M}\frac{1}{L}\sum_{j=0}^{M}\sum_{l=1}^{L}\left\|{\bm{q}}^{t}_{j,l}-{\bm{q}}^{t-1}_{j,l}\right\|^{2}\,,$
(5)

where $M$ is the number of all query embeddings $Q_{0:t-1}$ for old classes.
And ${\bm{q}}^{t}_{j,l}$ and ${\bm{q}}^{t-1}_{j,l}$ refer to the student and
teacher model from step $t$ and $t-1$, respectively.

####  IV-B2 Class/Instance-Aware Prediction KD

To further retain the knowledge of old classes, prediction knowledge
distillation is proposed for match classification and mask prediction between
the student and teacher models. Given $t-1$ query groups
$\\{Q_{0},Q_{1},...,Q_{t-1}\\}$ for old classes, we can obtain $t-1$ predicted
class distribution vectors
$\\{{\bm{c}}_{0},{\bm{c}}_{1},...,{\bm{c}}_{t-1}\\}$ from the output of class
head before Softmax activation function. Therefore, we can match the
distribution of old classes between two models with Kullback-Leibler (KL)
divergence:

$\mathcal{L}_{c}(\Theta^{t})=\sum_{i=0}^{t-1}{\bm{c}}_{i}^{t-1}\log\frac{{\bm{c}}_{i}^{t-1}}{{\bm{c}}_{i}^{t}}\,,$
(6)

where ${\bm{c}}_{i}^{t}$ and ${\bm{c}}_{i}^{t-1}$ refer to $i$-th class
distribution prediction associated with query set
$Q_{i}\in\\{Q_{0},Q_{1},...,Q_{t-1}\\}$ from step $t-1$ model (_i.e.,_
teacher) and $t$ model (_i.e.,_ student), respectively. Similarly, the
knowledge distillation of the predicted mask can be computed with the binary
cross-entropy loss [28] $\mathcal{L}_{ce}$ and dice loss [67]
$\mathcal{L}_{dice}$:

$\displaystyle\mathcal{L}_{m}(\Theta^{t})$
$\displaystyle=\lambda_{c}\frac{1}{M}\sum_{j=0}^{M}\mathcal{L}_{\text{ce}}({\bm{m}}_{j}^{t},{\bm{m}}_{j}^{t-1})$
(7)
$\displaystyle+\lambda_{d}\frac{1}{M}\sum_{j=0}^{M}\mathcal{L}_{\text{dice}}({\bm{m}}_{j}^{t},{\bm{m}}_{j}^{t-1})\,,$

where $M$ is the number of all query embeddings $Q_{0:t-1}$ for old classes.
${\bm{m}}_{j}^{t}$ and ${\bm{m}}_{j}^{t-1}$ denote the $j$-th predicted mask
from step $t-1$ model (_i.e.,_ teacher) and $t$ model (_i.e.,_ student),
respectively. $\lambda_{c}$ and $\lambda_{d}$ are two weight parameters,
similar to Mask2former [2].

Algorithm 1 Training pipeline of CisDQ

1:$\mathcal{C}_{0},\mathcal{I}_{0}$: the base class set and corresponding
image set.

2:$\\{(\mathcal{C}_{1},\mathcal{I}_{1}),(\mathcal{C}_{2},\mathcal{I}_{2}),\cdots,(\mathcal{C}_{T},\mathcal{I}_{T})\\}$
: the novel class set and corresponding image set for each incremental step.

3:$\alpha$ : the learning rate.

4:Initialize model parameters $\bm{\theta}$ and the base query set $Q_{0}$ for
CisDQ model $f$

5:Optimize $\\{\bm{\theta},Q_{0}\\}$ with the Mask2former loss on image set
$\mathcal{I}_{0}$

6:for incremental step $t\in\\{1,\cdots,T\\}$ do

7: $\triangleright$ $Q=\\{Q_{i}\\}_{i=0}^{t-1}\cup Q_{t}$, Update the query
set with random

8: initialized novel query

9: $\triangleright$ Freeze $\\{Q_{i}\\}_{i=0}^{t-1}$

10: for each training step do

11: $\triangleright$ Sample a mini-batch of training images $I$,
$I\in\mathcal{I}_{t}$

12: $\triangleright$ $\hat{\bm{y}}=f_{\bm{\theta}}(Q_{0:t}|I)$, get network
predictions $\hat{\bm{y}}$

13: $\triangleright$ Match the predictions $\hat{\bm{y}}$ to the ground truth
$\bm{y}\in\mathcal{C}_{t}$

14: with Equ. 3

15: $\triangleright$ Calculate loss $\mathcal{L}$ with Equ. 8:

16:
$\quad\quad\mathcal{L}=\mathcal{L}_{\text{new}}+\lambda_{1}\mathcal{L}_{q}+\lambda_{2}\mathcal{L}_{c}+\lambda_{3}\mathcal{L}_{m}+\lambda_{4}\mathcal{L}_{pod}$

17: $\triangleright$ Update $\\{\bm{\theta},Q_{t}\\}$ with back propagation
algorithm:

18:
$\quad\quad\\{\bm{\theta},Q_{t}\\}=\\{\bm{\theta},Q_{t}\\}-\alpha\nabla\mathcal{L}$

19: end for

20:end for

21:The trained CisDQ model $f$ with weights $\\{\bm{\theta},Q_{0:T}\\}$

Settings | Step Number | Few Shot for New Class
---|---|---
iFSIS | Two-Step (Step 0, 1) | ✓
CIS | Multi-Step (Step 0, 1, …, n) |
TABLE I: Comparison of Related Task. ‘iFSIS’ and ‘CIS’ refer to incremental few-shot instance segmentation and continual instance segmentation, respectively. Seq. | 80 Instance Classes on COCO 2017
---|---
1-8 | person, bicycle, car, motorcycle, airplane, bus, train, truck
9-16 | boat, traffic light, fire hydrant, stop sign, parking meter, bench, bird, cat
17-24 | dog, horse, sheep, cow, elephant, bear, zebra, giraffe
25-32 | backpack, umbrella, handbag, tie, suitcase, frisbee, skis, snowboard
33-40 | sports ball, kite, baseball bat, baseball glove, skateboard, surfboard, tennis racket, bottle
41-48 | wine glass, cup, fork, knife, spoon, bowl, banana, apple
49-56 | sandwich, orange, broccoli, carrot, hot dog, pizza, donut, cake
57-64 | chair, couch, potted plant, bed, dining table,toilet, tv, laptop
65-72 | mouse, remote, keyboard, cell phone, microwave, oven, toaster, sink
72-80 | refrigerator, book, clock, vase, scissors, teddy bear, hair drier, toothbrush
TABLE II: Class Order, for Instance, Segmentation with 80 Classes on COCO 2017 . Seq. | 100 Instance Classes on ADE 20k
---|---
1-10 | bed, windowpane, cabinet, person, door, table, curtain, chair, car, painting,
11-20 | sofa, shelf, mirror, armchair, seat, fence, desk, wardrobe, lamp, bathtub,
21-30 | railing, cushion, box, column, signboard, chest of drawers, counter, sink, fireplace, refrigerator,
31-40 | stairs, case, pool table, pillow, screen door, bookcase, coffee table, toilet, flower, book ,
41-50 | bench, countertop, stove, palm, kitchen island, computer, swivel chair, boat, arcade machine, bus,
51-60 | towel, light, truck, chandelier, awning, streetlight, booth, television receiver, airplane, apparel,
61-70 | pole, bannister, ottoman, bottle, van, ship, fountain, washer, plaything, stool,
71-80 | barrel, basket, bag, minibike, oven, ball, food, step, trade name, microwave,
81-90 | pot, animal, bicycle, dishwasher, screen, sculpture, hood, sconce, vase, traffic light,
91-100 | tray, ashcan, fan, plate, monitor, bulletin board, radiator, glass, clock, flag
TABLE III: Class Order, for Instance, Segmentation with 100 Classes on ADE 20k. Method | Base Network | Backbone | ADE 100-50 (2 steps) | ADE 100-10 (6 steps) | ADE 50-50 (3 steps) | ADE 100-5 (11 steps)
---|---|---|---|---|---|---
1-100 | 101-150 | all | 1-100 | 101-150 | all | 1-50 | 51-150 | all | 1-100 | 101-150 | all
Joint Training | Deeplab V3 | ResNet-101 | 44.3 | 28.2 | 38.9 | 44.3 | 28.2 | 38.9 | 51.1 | 33.3 | 38.9 | 44.3 | 28.2 | 38.9
ILT [68] | Deeplab V3 | ResNet-101 | 18.3 | 14.4 | 17.0 | 0.1 | 3.1 | 1.1 | 3.5 | 12.9 | 8.7 | 0.1 | 1.3 | 0.5
MiB [62] | Deeplab V3 | ResNet-101 | 40.5 | 17.2 | 32.8 | 38.2 | 11.1 | 29.2 | 45.6 | 21.0 | 29.3 | 36.0 | 5.7 | 26.0
PLOP [3] | Deeplab V3 | ResNet-101 | 41.9 | 14.9 | 32.9 | 40.5 | 13.6 | 31.6 | 48.8 | 21.0 | 30.4 | 39.1 | 7.8 | 28.8
RCIL [4] | Deeplab V3 | ResNet-101 | 42.3 | 18.8 | 34.5 | 39.3 | 17.6 | 32.1 | 48.3 | 25.0 | 32.5 | 38.5 | 11.5 | 29.6
MiB+EWF [69] | Deeplab V3 | ResNet-101 | 41.2 | 21.3 | 34.6 | 41.5 | - | 33.2 | - | - | - | 41.4 | 13.4 | 32.1
SSUL [5] | Deeplab V3 | ResNet-101 | 41.3 | 18.0 | 33.6 | 40.2 | 18.8 | 33.1 | 48.4 | 20.2 | 29.6 | 39.9 | 17.4 | 32.5
Joint Training ⋆ | Mask2former | ResNet-50 | 49.8 | 36.9 | 45.5 | 49.8 | 36.9 | 45.5 | 55.9 | 40.3 | 45.5 | 49.8 | 36.9 | 45.5
CoMFormer [70] | Mask2former | ResNet-101 | 44.7 | 26.2 | 38.4 | 40.6 | 15.6 | 32.3 | - | - | - | 39.5 | 13.6 | 30.9
MiB ⋆ | Mask2former | ResNet-50 | 47.6 | 27.8 | 41.0 | 40.6 | 16.3 | 32.5 | 52.7 | 30.8 | 38.1 | 38.0 | 12.2 | 29.4
PLOP ⋆ | Mask2former | ResNet-50 | 48.2 | 28.0 | 41.5 | 44.3 | 18.6 | 35.7 | 55.5 | 32.0 | 39.9 | 39.2 | 13.2 | 30.5
MicroSeg [63] | Mask2former | ResNet-101 | 40.2 | 18.8 | 33.1 | 41.5 | 21.6 | 34.9 | 48.6 | 24.8 | 32.9 | 40.4 | 20.5 | 33.8
MicroSeg-M [63] | Mask2former | ResNet-101 | 43.4 | 20.9 | 35.9 | 43.7 | 22.2 | 36.6 | 49.8 | 22.0 | 31.4 | 43.6 | 22.4 | 36.6
CiSDQ (ours) | Mask2former | ResNet-50 | 48.9 | 28.2 | 42.0 | 47.8 | 24.6 | 40.1 | 55.7 | 33.6 | 41.0 | 46.2 | 17.6 | 36.7
TABLE IV: The final mIoU(%) of Semantic Segmentation on the ADE20K dataset. ⋆ refers to our re-implementation performance. In gray denotes joint training with all classes or using extra model (_i.e.,_ SSUL using DSS [71] pretrained on MSRA-B dataset [72]). Method | Base Network | Backbone | VOC 15-1 (6 steps) | VOC 15-5 (2 steps) | VOC 10-1 (11 steps)
---|---|---|---|---|---
0-15 | 16-20 | all | 0-15 | 16-20 | all | 0-10 | 11-20 | all
Joint Training (offline) | Deeplab V3 | ResNet-101 | 79.8 | 72.3 | 77.4 | 79.8 | 72.4 | 77.4 | 78.4 | 76.4 | 77.4
LwF-MC [73] | Deeplab V3 | ResNet-101 | 6.4 | 8.4 | 6.9 | 58.1 | 35.0 | 52.3 | 4.7 | 5.9 | 5.0
ILT [68] | Deeplab V3 | ResNet-101 | 8.8 | 8.0 | 8.6 | 67.1 | 39.2 | 60.5 | 7.2 | 3.7 | 5.5
MiB [62] | Deeplab V3 | ResNet-101 | 35.1 | 13.5 | 29.7 | 75.5 | 49.4 | 69.0 | 12.2 | 13.1 | 12.6
PLOP [3] | Deeplab V3 | ResNet-101 | 65.1 | 21.1 | 54.6 | 75.7 | 51.7 | 70.1 | 44.0 | 15.5 | 30.5
RCIL [4] | Deeplab V3 | ResNet-101 | 70.6 | 23.7 | 59.4 | 78.8 | 52.0 | 72.4 | 55.4 | 15.1 | 34.3
SSUL [5] | Deeplab V3 | ResNet-101 | 77.3 | 36.6 | 67.6 | 77.8 | 50.1 | 71.2 | 71.3 | 46.0 | 59.3
Joint Training (offline) ⋆ | Mask2former | ResNet-50 | 80.0 | 75.2 | 78.8 | 80.0 | 75.2 | 78.8 | 78.9 | 78.8 | 78.8
PLOP ⋆ | Mask2former | ResNet-50 | 71.3 | 12.5 | 57.3 | 78.2 | 28.4 | 66.3 | 56.9 | 14.5 | 36.7
CiSDQ (ours) | Mask2former | ResNet-50 | 77.9 | 13.2 | 62.5 | 80.5 | 49.3 | 73.1 | 73.2 | 15.5 | 45.7
CiSDQ (ours) | Mask2former | ResNet-101 | 78.2 | 13.6 | 62.8 | 80.6 | 49.5 | 73.2 | 73.9 | 15.8 | 46.2
CiSDQ (ours) | Mask2former | Swin-B | 79.7 | 14.9 | 64.3 | 84.0 | 55.1 | 77.1 | 74.6 | 15.9 | 46.7
TABLE V: The mIoU(%) of Semantic Segmentation on the Pascal VOC 2012 dataset.
⋆ refer to our re-implementation performance. In gray denotes joint training
with all classes or using extra model (_i.e.,_ SSUL using DSS [71] pretrained
on MSRA-B dataset [72]).

###  IV-C Loss Function

The proposed pipeline mainly contains two losses, _i.e.,_ manual annotation
supervision loss, and knowledge distillation loss. The whole loss function can
be formulated as Equation 8:

$\mathcal{L}=\underbrace{\mathcal{L}_{\text{new}}}_{\text{New
Classes}}+\underbrace{\lambda_{1}\mathcal{L}_{q}+\lambda_{2}\mathcal{L}_{c}+\lambda_{3}\mathcal{L}_{m}+\lambda_{4}\mathcal{L}_{pod}\,\,}_{\text{Distillation
for Old Classes}}\\!\\!\,,$ (8)

where $\mathcal{L}_{\text{new}}$ denotes the supervision from the available
annotation of newly added classes, the same as that of Mask2former [2].
$\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$, $\lambda_{4}$ are the weight
parameters, which are set to $1$, $5$, $300$, $100$, respectively.
$\mathcal{L}_{pod}$ is the local POD loss [49], the same as PLOP [3], which is
used to distill the features from backbone and pixel decoders.

###  IV-D Pseudo Code

We describe the training pipeline of CisDQ in Algorithm 1. We first train the
model on the image set of the base classes (line 1-2). Then, for each
incremental step, we add a set of new queries for the novel classes to the
model and freeze the queries of the old classes (line 4-6). After that, we
optimize the model with the proposed matching strategy (line 10) and loss
function (line 12-15). Finally, we get the CisDQ model that is capable to
segment objects of all classes.

##  V Experiments

###  V-A Experimental Setups

####  V-A1 Semantic Segmentation

Datasets. Similar to previous works [3, 5], Pascal-VOC 2012 [74] (20 classes)
and ADE20k [75] (150 classes) are used to evaluate our CiSDQ. Protocols.
Following works [3, 5, 76], the model is trained to segment new classes in
multiple steps continually. And only new classes in the current step are
labeled. And there are two different CSS settings: Disjoint and Overlapped.
For step $t$, images of Disjoint only contain classes
$\mathcal{C}^{1:t-1}\cup\mathcal{C}^{t}$ (old and new), while that of
Overlapped can includes any classes
$\mathcal{C}^{1:t-1}\cup\mathcal{C}^{t}\cup\mathcal{C}^{t+1:T}$ (old, new, and
future). Therefore, the Overlapped setting is more challenging and realistic.
In our experiments, we only focus on the result concerning Overlapped CSS.
During the testing phase, the model need to segment all classes. Protocol
setting following previous works [3, 4], _e.g.,_ ADE 100-50, 100-10, 50-50,
and 100-5, which consists in learning $100$ classes followed by 50 class ($2$
steps), $100$ classes followed by five times $10$ classes ($6$ steps), $50$
classes followed by two times $50$ classes ($3$ steps), and $100$ classes
followed by ten times $10$ classes ($11$ steps).

Figure 4: More visualization of CiSDQ for semantic segmentation on ADE 20k
val. CiSDQ presents high-quality results.

####  V-A2 Instance Segmentation

Datasets. Following general instance segmentation [2, 30], COCO [77] (80
classes) and ADE20k [75] (100 classes) are used to evaluate our CiSDQ.
Protocols. Different from CCS, Continual Instance Segmentation (CIS), as one
new task, need the first definition for experiment settings. As for COCO [77],
we give three-step settings, _i.e.,_ 40-40 (2 steps), 40-8 (6 steps), and 40-4
(11 steps). For ADE20k, similar to the class split of CCS, we provide three
step settings concerning two steps (50-50), six steps (50-10), and eleven
steps (50-5). The detailed classes split for each step and related experiments
for ADE20k are provided in in supplementary material. Similar to CCS, we focus
on Overlapping setting, which is more realistic and challenging.

####  V-A3 Implementation Details.

The most experiment setting of the experiments all follow Mask2former [2],
_i.e.,_ backbone, transformer, pixel decoder, AdamW optimizer with an initial
learning rate of 0.0001. And 8 Tesla V100 GPUs are used for the experiment.
For semantic segmentation, the number of the query is the same as the that of
class during continual learning, _e.g.,_ ADE 100-50 (2 steps) require 100
queries to learn 1-100 classes at step 0, then increasing 50 queries to learn
101-150 classes at step 1. For instance segmentation, the number of the query
is set to $2.5$ times of that of class during continual learning, COCO
requires 200 queries no matter which settings. COCO 40-20 (3 steps) requires
100, 50, 50 queries for $0$-th, $1$-th, $2$-th steps, respectively.

####  V-A4 Compared with iFSIS.

Tab. I compares continual instance segmentation (CIS) with other related
problem, _i.e.,_ incremental few-shot instance segmentation [65, 66].
Different from continual segmentation, iFSIS only incremental learning on base
and novel classes with just two steps, and the training data for new classes
is limited and few.

####  V-A5 Protocols for Continual Instance Segmentation Datasets.

COCO 2017. Different from continual semantic segmentation, Continual Instance
Segmentation (CIS), as one new task, needs the first definition for experiment
settings. As for COCO 2017 [77], we give three-step settings, _i.e.,_ 40-40 (2
steps), 40-8 (6 steps), and 40-4 (11 steps). And Table. II provides the
corresponding class order. For ADE20k, similar to the class split of CCS, we
provide three-step settings concerning two steps (50-50), six steps (50-10),
and eleven steps (50-5). The detailed class order for splitting is provided in
Table. III. Similar to CCS, all experiments concerning instance segmentation
task focus on Overlapping setting, which is more realistic and challenging.

Method | COCO 40-40 (2 steps) | COCO 40-8 (6 steps) | COCO 40-4 (11 steps)
---|---|---|---
1-40 | 41-80 | mAP | 1-40 | 41-48 | 49-56 | 57-64 | 65-72 | 73-80 | mAP | 1-40 | 41-80 | mAP
Joint Training (offline) | 35.6 | 51.8 | 43.7 | 35.6 | 47.0 | 59.2 | 53.8 | 52.6 | 46.2 | 43.7 | 35.6 | 51.8 | 43.7
PLOP ∗ | 28.0 | 40.1 | 34.1 | 24.4 | 16.4 | 28.0 | 22.3 | 25.1 | 16.3 | 23.0 | 16.5 | 11.3 | 13.9
CiSDQ | 28.8 | 41.8 | 35.3 | 24.2 | 17.9 | 29.4 | 24.8 | 27.6 | 19.8 | 24.1 | 18.2 | 11.7 | 15.0
TABLE VI: The final AP(%) of Instance Segmentation on COCO val2017 with 80 categories. Mask2former with ResNet50 is used as the base network. ‘offline’ refers to training with all data and classes. ⋆ refers to our re-implementation performance. Method | ADE 50-50 (2 steps) | ADE 50-10 (6 steps) | ADE 50-5 (11 steps)
---|---|---|---
1-50 | 51-100 | all | 1-50 | 51-60 | 61-70 | 71-80 | 81-90 | 91-100 | all | 1-50 | 51-100 | all
Joint Training (offline) | 32.2 | 20.6 | 26.4 | 32.2 | 24.6 | 14.2 | 16.9 | 26.7 | 20.5 | 26.4 | 32.2 | 20.6 | 26.4
PLOP ⋆ | 23.4 | 11.8 | 17.6 | 13.1 | 6.4 | 2.1 | 7.2 | 11.1 | 4.9 | 9.7 | 12.9 | 2.5 | 7.7
CiSDQ (ours) | 31.2 | 13.8 | 23.7 | 26.2 | 7.1 | 3.9 | 7.9 | 9.8 | 5.0 | 16.4 | 18.1 | 2.9 | 10.5
TABLE VII: The final AP(%) of Instance Segmentation on the ADE20K dataset. ⋆
refers to our re-implementation performance. ResNet50 as the backbone is used.

###  V-B Continual Semantic Segmentation

PASCAL VOC 2012. Table. V presents the experimental results of the last step
for three continual learning settings. Compared with previous methods based on
deeplab v3, our CiSDQ achieves obvious mIoU improvements, especially for long-
term steps setting, _e.g.,_ VOC 10-1 (11 steps) and 15-1 (6 steps). For a fair
comparison, the performance of SSUL [5] is not the main reference, while it
uses the extra Salient Object Segmentation model pre-trained on MSRA-B dataset
[72]). Actually, the Salient object segmentation model usually can segment and
obtain competitive pseudo label effectively, especially for VOC 2012, where
the image domain is simple and only support 20 classes. And we also re-
implementation PLOP [3] with Mask2former framework. We do not re-
implementation RCIL [4] and SSUL [5] on Mask2former architecture due to the
mismatched re-parameterization on transformer architecture from RCIL and using
extract model of SSUL. Overall, our CiSDQ achieves state-of-the-art
performance with obvious improvements no matter for the old or new classes.
Besides, we also provide more high-quality visualization results for semantic
segmentation tasks on ADE 20k, as shown in Fig. 4.

ADE20k. Table. IV presents the experimental results of the last step for four
continual learning settings on ADE 20K. On different continual learning tasks,
_i.e.,_ 100-50, 100-10 and 50-50, 100-5, our method all achieves state-of-the-
art results, especially for the long step tasks, _i.e.,_ 100-10 (6 steps) and
100-5 (11 steps), achieving $4.4\%$ and $2.9\%$ mIoU improvement over that of
previous SOTA method, respectively. Our algorithm demonstrates significant
improvements compared to the ten methods based on Deeplab v3 or Mask2Former,
substantiating the effectiveness of our approach. For fair comparison, the
performance of SSUL [5] and MicroSeg-M [63] is not the main reference, while
it uses the extra Salient Object Segmentation model pre-trained on MSRA-B
dataset [72]). And we also re-implementation MiB [62] and PLOP [3] with
Mask2former framework. We do not re-implementation RCIL [4] and SSUL [5] on
Mask2former due to the mismatched re-parameterization on transformer
architecture of RCIL and the extra model of SSUL.

Figure 5: Visualization of PLOP and CiSDQ for 100-5 (11 steps) on ADE 20k val.
CiSDQ presents more robust performance for remaining the knowledge of old
classes ( _e.g.,_ buffet), while PLOP [3] suffer from the catastrophic
forgetting. Figure 6: Visualization of CiSDQ for continual instance
segmentation on COCO 2017 val. CiSDQ presents robust performance on learning
new classes and remaining old knowledge.

###  V-C Continual Instance Segmentation

COCO2017. To verify the effectiveness of CiSDQ for instance segmentation, we
provide corresponding experiments on COCO [77] for three different continual
tasks, _i.e.,_ COCO 40-40, 40-8, and 40-4, as shown in Table. VI. CiSDQ
presents competitive performance, around $1.0\%$ mAP improvement over previous
methods. It is worth noting that CiSDQ presents a better performance for newly
added classes, power from more reasonable KD for class/instance level.

ADE 20k. We also provide the results for continual instance segmentation with
three continual settings on ADE 20k, as shown in Fig. VII. We implement PLOP
[3] on Mask2former architecture for continual instance segmentation, while
there is no one continual instance segmentation method for comparison. CiSDQ
achieves obvious improvements over PLOP, with $6.1\%$, $6.7\%$, and $2.8\%$
mAP for the three different settings, respectively. Besides, we also provide
some visualizations for continual instance segmentation on COCO 2017, as shown
in Fig. 6.

Figure 7: Compared with PLOP [3] for semantic segmentation task on ADE 20k
100-5 (11 steps). CiSDQ presents better performance.

Incremental Query | Independent Match | 1-100 | 101-150 | all
---|---|---|---|---
|  | 39.2 | 13.2 | 30.5
✓ |  | 39.6 (+0.4) | 15.3 (+2.1) | 31.5(+1.0)
✓ | ✓ | 46.2 (+6.6) | 17.6 (+2.3) | 36.7 (+5.2)
(a) Dynamic Query Queue. Experiments are conducted on 100-5 (11 steps) setting
for semantic segmentation (mAP) task on ADE20k val.

Method | Base Network | Backbone | Params | Flops | FPS | mIoU
---|---|---|---|---|---|---
SSUL | Deeplab V3 | ResNet-101 | 62.7 | 255.1 | 14.1 | 33.1
CiSDQ | Mask2former | ResNet50 | 44.0 | 71.0 | 9.7 | 40.1
(b) Base Network. Experiments are conducted on 100-10 setting for semantic
segmentation (mIoU) task on ADE20K val.

Prediction-KD | Pod-KD | Query-KD | 1-100 | 101-150 | all
---|---|---|---|---|---
|  |  | 0.7 | 2.7 | 1.2
✓ |  |  | 35.5 | 15.1 | 28.7
✓ |  | ✓ | 41.3 | 17.9 | 33.5
✓ | ✓ |  | 38.7 | 15.6 | 31.0
✓ | ✓ | ✓ | 46.2 | 17.6 | 36.7
(c) Query-KD and Prediction-KD. Experiments are conducted for 100-5 setting on
ADE20k val.

Query Frozen | 1-100 | 101-150 | all
---|---|---|---
| 44.2 | 17.3 | 35.2
✓ | 46.2 | 17.6 | 36.7
(d) Frozen for Old Query Embedding. Experiments are conducted on 100-5 (6
steps) overlapped setting for semantic segmentation (mIoU) task on ADE20k val.

100-5 | Queries | Params | FPS | mIoU
---|---|---|---|---
Step 0 (1-100) | 100 | 43.915 | 11.0 | 32.2
Step 1 (1-110) | 110 | 43.916 | 10.8 | 32.3
Step 5 (1-150) | 150 | 43.920 | 9.7 | 36.7
(e) Increasing Params for Dynamic Query. Experiments are conducted on 100-5
setting for semantic segmentation (mIoU) task on ADE20K val.

TABLE VIII: CiSDQ ablations. We perform ablations on ADE20k val for continual semantic segmentation task.  Class | Train Set | Test Set | ADE 50-50 (3 steps,mIoU/%) | ADE 100-10 (6 steps,mIoU/%)
---|---|---|---|---
$0$-th step |  $1$-th step |  $2$-th step |  $0$-th step |  $1$-th step |  $2$-th step |  $3$-th step |  $4$-th step |  $5$-th step
Person | ADE20K | VOC 2012 | 76.7 | 76.4 | 75.6 | 77.2 | 76.3 | 76.7 | 76.8 | 75.5 | 75.0
Car | ADE20K | VOC 2012 | 74.1 | 75.5 | 75.1 | 75.6 | 75.8 | 75.7 | 78.6 | 79.4 | 79.5
Bus | ADE20K | VOC 2012 | 0 | 68.7 | 68.2 | 62.0 | 61.3 | 60.1 | 52.3 | 50.0 | 50.8
Boat | ADE20K | VOC 2012 | 0 | 33.4 | 37.7 | 26.0 | 26.5 | 27.2 | 28.7 | 27.7 | 27.4
TABLE IX: The ablation study for model generalization during continual
learning. We conducted cross-dataset validation to investigate the
generalization capability of the model. In the ADE 50-50 (3 steps) setting,
the ’bus’ and ’boat’ categories are considered as new classes, requiring the
model to learn them in the $1$-th step.

###  V-D Ablation Study

Incremental query queue and independent bipartite match, as two indispensable
points, enable our dynamic query and adaptive background. Table. VIII(a)
presents the effect of the two components. Without the independent match, the
incremental query just increases the number of queries, which can not decouple
the learning of old and new classes. Therefore, with the only incremental
query, CiSDQ achieves a little improvement, around $1.0\%$ mIoU. By contrast,
the combination of two components brings obvious improvement, up to $5.2\%$
mIoU over the baseline.

Query-KD and Prediction-KD, two key components for our Query Guided Knowledge
Distillation. Table. VIII(c) presents the effect for the two components.
Baseline, without the three KDs, training with new classes directly, presents
an unacceptable result $1.2\%$ mIoU. Prediction-KD, Pod-KD, and Query-KD bring
different gains for the final results, _i.e.,_ $27.5\%$, $2.3\%$, $5.7\%$,
respectively. Although Pod-KD damages the representation learning of new
classes due to the unbalanced matching in Fig. 2, the positive impact for
remaining the knowledge of old classes is not neglectable. Unlike our Query-
Guided KD, Pod-KD forces on the features from the backbone and pixel decoder,
thus we adopt it in Equ. 8. Actually, any KDs will damage the performance of
newly added classes. Compared with Pod-KD, Query-KD shows better performance
on newly added classes (101-150 classes), where $17.9\%$ $v.s$ $15.6\%$ mIou
on 3-th and 4-th lines. Meanwhile, Query-KD also gives a competitive
performance on old classes, as shown in Fig. 5.

Frozen for Query of Old Class. Except for knowledge distillation, query
embedding frozen also help to remain the performance of old classes, as shown
in Table. VIII(d). With query frozen, the performance of old classes (1-100
classes) shows an obvious improvement with $1.5\%$ mIoU.

Figure 8: Visualization of the output tensor of the decoder for 50 randomly
sampled images on the VOC 2012 dataset. Throughout the continual learning
process, the model exhibits activations for novel classes, and there is no
forgetting of the feature for old classes.

Computation Cost from Increasing Query Embedding. We also study the extra
computation cost from increasing queries in Table. VIII(e). For 100-5 (11
steps) setting, the number of queries will increase from 100 to 150 to
represent the newly added classes. And the increased 50 queries only bring
around $0.005$ M extra parameters, while the performance shows an obvious
improvement. Compared with Deeplab V3 based methods, mask2former-based
framework (ours) shows a big advantage with low computation cost (fewer
parameters, smaller flops, and faster speed) and better performance.

Base Framework. Table. VIII(b) gives a comparison for the two different base
networks. Mask2former is a better choice for us to apply continual learning to
image segmentation, while the parameters, flops, and FPS are all acceptable.

Comparison between PLOP. Fig. 7 presents the comparison between PLOP [3] and
CiSDQ for semantic segmentation task on ADE 20k 100-5 setting. As for the
learning of newly added classes, CiSDQ shows better performance over PLOP,
which is mainly caused by two aspects: 1) Different queries decouple the
learning of new classes and remaining old knowledge; 2) Query-Guided KD
performs a more precise distillation matching without damaging the data
distribution of new classes. Similarly, CiSDQ presents less performance loss
for old classes during the continual learning of newly added classes process.

Ablation Study for Model Generalization. Table IX present the ablation study
for model generalization during continual learning. We conducted cross-dataset
evaluation to study the model’s generalization. Due to variations in the
number and distribution of categories across different datasets, such as the
inclusion of the ’refrigerator’ category in ADE 20K, which is not present in
the VOC 2012 dataset, we opted to explore specific categories. In fact, the
generalization variations of the model differ for different categories during
the continual learning process. For example, the generalization of the
’person’ category tends to decrease, while the ’car’ category shows
improvement. We attribute these changes to a balance between the forgetting of
knowledge related to old classes and the decrease in false positives. In the
continual learning process, the forgetting of knowledge related to old classes
is a recognized challenge that can lead to a decline in the performance of
these classes. However, as the number of categories increases during continual
learning, it can also assist an old class in excluding certain regions that
are prone to false positives. As a result, different categories and
experimental settings may yield varied results. Overall, the generalization
remains stable. It is essential to emphasize that continual learning is not
aimed at enhancing the model’s generalization but rather enabling the model to
dynamically update its knowledge and extend its ability to segment new
classes.

Visualization of the output tensor. In order to gain a deeper understanding of
the continual learning process and the changes in the model, we randomly
selected 50 images from the VOC 2012 test set. Then we extracted and
visualized the multiplication of the pixel decoder and transformer decoder
outputs. Given 50 images, we obtain corresponding tensor outputs of size
$50\times 20\times w\times h$, where $20$ refers to the number of category
(Excluding the background category). For ease of visualization and examining
activations for each category, we first use average pooling to transform the
tensor from $50\times 20\times w\times h$ to $50\times 20\times 1\times 1$.
Subsequently, we normalize the tensor and visualize the results as shown in
Figure 8. At $0$-th step, it is evident that the tensor lacks activations for
the last five classes, such as ’pottedplant’ and ’sheep.’ However, as
continual learning progresses and the model learns new classes, the tensor
exhibits activations for the new classes, allowing for predictions of these
novel categories.

##  VI Limitations and Future Works

Although CisDQ can achieve good performance on continual semantic segmentation
(CSS), its performance on continual instance segmentation (CIS) is still not
satisfactory. Compared to CSS, CIS requires more queries as instances from the
same class can only be distinguished by different queries. In our future work,
we will explore a way to assign queries to the old and novel classes more
efficiently, so that classes from each incremental step can get enough queries
to achieve high performance on CIS.

##  VII Conclusion

In this paper, we propose a simple, yet effective Continual Image Segmentation
method with incremental Dynamic Query, named CiSDQ, which decouples the
representation learning of both old and new knowledge with lightweight query
embedding. CiSDQ also includes a novel class/instance-aware Query Guided
Knowledge Distillation strategy to overcome catastrophic forgetting by
capturing the inter-class diversity and intra-class identity. Meanwhile, we
introduce the continual learning to instance segmentation task, which is more
challenging. Experimental results show that our CiSDQ achieves SOTA
performance, with up to $10\%$ mIoU improvement over previous methods.

##  VIII Acknowledgements

This work is supported by the National Key Research and Development Program of
China (# 2022YFC3602601), and the Key Research and Development Program of
Zhejiang Province of China (# 2021C02037). M. Shou’s participation was
supported by Mike Zheng Shou’s Start-Up Grant from NUS and the Ministry of
Education, Singapore, under the Academic Research Fund Tier 1 (FY2022).

## References

  * [1] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, “Rethinking atrous convolution for semantic image segmentation,” _arXiv preprint arXiv:1706.05587_ , 2017.
  * [2] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar, “Masked-attention mask transformer for universal image segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 1290–1299.
  * [3] A. Douillard, Y. Chen, A. Dapogny, and M. Cord, “Plop: Learning without forgetting for continual semantic segmentation,” _arXiv preprint arXiv:2011.11390_ , 2020.
  * [4] C.-B. Zhang, J.-W. Xiao, X. Liu, Y.-C. Chen, and M.-M. Cheng, “Representation compensation networks for continual semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 7053–7064.
  * [5] S. Cha, Y. Yoo, T. Moon _et al._ , “Ssul: Semantic segmentation with unknown label for exemplar-based class-incremental learning,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 10 919–10 930, 2021.
  * [6] A. Robins, “Catastrophic forgetting, rehearsal and pseudorehearsal,” _Connection Science_ , vol. 7, no. 2, pp. 123–146, 1995.
  * [7] R. M. French, “Catastrophic forgetting in connectionist networks,” _Trends in cognitive sciences_ , vol. 3, no. 4, pp. 128–135, 1999.
  * [8] W. Wu, D. Zhang, Y. Fu, C. Shen, H. Zhou, Y. Cai, and P. Luo, “End-to-end video text spotting with transformer,” _arXiv preprint arXiv:2203.10539_ , 2022.
  * [9] W. Wu, D. Zhang, Y. Cai, S. Wang, J. Li, Z. Li, Y. Tang, and H. Zhou, “A bilingual, openworld video text dataset and end-to-end video text spotter with transformer,” in _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.
  * [10] Y. Zhao, Y. Cai, W. Wu, and W. Wang, “Explore faster localization learning for scene text detection,” in _2023 IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, 2023, pp. 156–161.
  * [11] C. Gong, Z. Yang, Y. Bai, J. Shi, A. Sinha, B. Xu, D. Lo, X. Hou, and G. Fan, “Curiosity-driven and victim-aware adversarial policies,” in _Proceedings of the 38th Annual Computer Security Applications Conference_ , 2022, pp. 186–200.
  * [12] M. S. Allili, D. Ziou, N. Bouguila, and S. Boutemedjet, “Image and video segmentation by combining unsupervised generalized gaussian mixture modeling and feature selection,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 20, no. 10, pp. 1373–1377, 2010.
  * [13] L. Salgado, N. Garcia, J. M. Menendez, and E. Rendon, “Efficient image segmentation for region-based motion estimation and compensation,” _IEEE transactions on circuits and systems for video technology_ , vol. 10, no. 7, pp. 1029–1039, 2000.
  * [14] T. Ida and Y. Sambonsugi, “Image segmentation using fractal coding,” _IEEE transactions on circuits and systems for video technology_ , vol. 5, no. 6, pp. 567–570, 1995.
  * [15] H. Lu, J. C. Woods, and M. Ghanbari, “Binary partition tree for semantic object extraction and image segmentation,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 17, no. 3, pp. 378–383, 2007.
  * [16] S. Sun, D. R. Haynor, and Y. Kim, “Semiautomatic video object segmentation using vsnakes,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 13, no. 1, pp. 75–82, 2003.
  * [17] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3431–3440.
  * [18] A. Arnab, S. Jayasumana, S. Zheng, and P. H. Torr, “Higher order conditional random fields in deep neural networks,” in _European conference on computer vision_. Springer, 2016, pp. 524–540.
  * [19] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du, C. Huang, and P. H. Torr, “Conditional random fields as recurrent neural networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1529–1537.
  * [20] D. Zhang, H. Zhang, J. Tang, X.-S. Hua, and Q. Sun, “Causal intervention for weakly-supervised semantic segmentation,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 655–666, 2020.
  * [21] ——, “Self-regulation for semantic segmentation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 6953–6963.
  * [22] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Semantic image segmentation with deep convolutional nets and fully connected crfs,” _arXiv preprint arXiv:1412.7062_ , 2014.
  * [23] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2881–2890.
  * [24] W. Liu, A. Rabinovich, and A. C. Berg, “Parsenet: Looking wider to see better,” _arXiv preprint arXiv:1506.04579_ , 2015.
  * [25] D. Eigen and R. Fergus, “Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 2650–2658.
  * [26] C. Farabet, C. Couprie, L. Najman, and Y. LeCun, “Learning hierarchical features for scene labeling,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 35, no. 8, pp. 1915–1929, 2012.
  * [27] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 12 077–12 090, 2021.
  * [28] B. Cheng, A. Schwing, and A. Kirillov, “Per-pixel classification is not all you need for semantic segmentation,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 17 864–17 875, 2021.
  * [29] B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik, “Simultaneous detection and segmentation,” in _European conference on computer vision_. Springer, 2014, pp. 297–312.
  * [30] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969.
  * [31] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _Advances in neural information processing systems_ , vol. 28, 2015.
  * [32] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _European conference on computer vision_. Springer, 2020, pp. 213–229.
  * [33] Y. Zhao, Q. Ye, W. Wu, C. Shen, and F. Wan, “Generative prompt model for weakly supervised object localization,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2023, pp. 6351–6361.
  * [34] W. Wu, Y. Zhao, M. Z. Shou, H. Zhou, and C. Shen, “Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models,” _Proc. Int. Conf. Computer Vision (ICCV 2023)_ , 2023.
  * [35] W. Wu, Y. Zhao, H. Chen, Y. Gu, R. Zhao, Y. He, H. Zhou, M. Z. Shou, and C. Shen, “Datasetdm: Synthesizing data with perception annotations using diffusion models,” _arXiv preprint arXiv:2308.06160_ , 2023.
  * [36] D. Yu, M. Zhang, M. Li, F. Zha, J. Zhang, L. Sun, and K. Huang, “Contrastive correlation preserving replay for online continual learning,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2023.
  * [37] X. Fu, J. Xiao, Y. Zhu, A. Liu, F. Wu, and Z.-J. Zha, “Continual image deraining with hypergraph convolutional networks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [38] Z. Le, J. Huang, H. Xu, F. Fan, Y. Ma, X. Mei, and J. Ma, “Uifgan: An unsupervised continual-learning generative adversarial network for unified image fusion,” _Information Fusion_ , vol. 88, pp. 305–318, 2022.
  * [39] M. Zhou, J. Xiao, Y. Chang, X. Fu, A. Liu, J. Pan, and Z.-J. Zha, “Image de-raining via continual learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 4907–4916.
  * [40] H. Xu, J. Ma, J. Jiang, X. Guo, and H. Ling, “U2fusion: A unified unsupervised image fusion network,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 1, pp. 502–518, 2020.
  * [41] R. Yan, L. Xie, X. Shu, and J. Tang, “Interactive fusion of multi-level features for compositional activity recognition,” _arXiv preprint arXiv:2012.05689_ , 2020.
  * [42] R. Yan, L. Xie, J. Tang, X. Shu, and Q. Tian, “Social adaptive module for weakly-supervised group activity recognition,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VIII 16_. Springer, 2020, pp. 208–224.
  * [43] M. McCloskey and N. J. Cohen, “Catastrophic interference in connectionist networks: The sequential learning problem,” in _Psychology of learning and motivation_. Elsevier, 1989, vol. 24, pp. 109–165.
  * [44] J. Bang, H. Kim, Y. Yoo, J.-W. Ha, and J. Choi, “Rainbow memory: Continual learning with a memory of diverse samples,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 8218–8227.
  * [45] E. Belouadah and A. Popescu, “Il2m: Class incremental learning with dual memory,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 583–592.
  * [46] A. Chaudhry, A. Gordo, P. Dokania, P. Torr, and D. Lopez-Paz, “Using hindsight to anchor past knowledge in continual learning,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 8, 2021, pp. 6993–7001.
  * [47] A. Chaudhry, P. K. Dokania, T. Ajanthan, and P. H. Torr, “Riemannian walk for incremental learning: Understanding forgetting and intransigence,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 532–547.
  * [48] A. Cheraghian, S. Rahman, P. Fang, S. K. Roy, L. Petersson, and M. Harandi, “Semantic-aware knowledge distillation for few-shot class-incremental learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 2534–2543.
  * [49] A. Douillard, M. Cord, C. Ollion, T. Robert, and E. Valle, “Podnet: Pooled outputs distillation for small-tasks incremental learning,” in _European Conference on Computer Vision_. Springer, 2020, pp. 86–102.
  * [50] Y. Xiang, Y. Fu, P. Ji, and H. Huang, “Incremental learning using conditional adversarial networks,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 6619–6628.
  * [51] S. Ebrahimi, F. Meier, R. Calandra, T. Darrell, and M. Rohrbach, “Adversarial continual learning,” in _European Conference on Computer Vision_. Springer, 2020, pp. 386–402.
  * [52] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [53] M. Xue, H. Zhang, J. Song, and M. Song, “Meta-attention for vit-backed continual learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 150–159.
  * [54] A. Ashok, K. Joseph, and V. Balasubramanian, “Class-incremental learning with cross-space clustering and controlled transfer,” _arXiv preprint arXiv:2208.03767_ , 2022.
  * [55] A. Douillard, A. Ramé, G. Couairon, and M. Cord, “Dytox: Transformers for continual learning with dynamic token expansion,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 9285–9295.
  * [56] M. Boschini, L. Bonicelli, A. Porrello, G. Bellitto, M. Pennisi, S. Palazzo, C. Spampinato, and S. Calderara, “Transfer without forgetting,” _arXiv preprint arXiv:2206.00388_ , 2022.
  * [57] Z. Huang, W. Hao, X. Wang, M. Tao, J. Huang, W. Liu, and X.-S. Hua, “Half-real half-fake distillation for class-incremental semantic segmentation,” _arXiv preprint arXiv:2104.00875_ , 2021.
  * [58] S. Yan, J. Zhou, J. Xie, S. Zhang, and X. He, “An em framework for online incremental learning of semantic segmentation,” in _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021, pp. 3052–3060.
  * [59] F. Cermelli, M. Mancini, S. R. Bulo, E. Ricci, and B. Caputo, “Modeling the background for incremental learning in semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9233–9242.
  * [60] H. Zhao, F. Yang, X. Fu, and X. Li, “Rbc: Rectifying the biased context in continual semantic segmentation,” in _European Conference on Computer Vision_. Springer, 2022, pp. 55–72.
  * [61] E. Zheng, Q. Yu, R. Li, P. Shi, and A. Haake, “A continual learning framework for uncertainty-aware interactive image segmentation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 7, 2021, pp. 6030–6038.
  * [62] F. Cermelli, M. Mancini, S. R. Bulo, E. Ricci, and B. Caputo, “Modeling the background for incremental learning in semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9233–9242.
  * [63] Z. Zhang, G. Gao, Z. Fang, J. Jiao, and Y. Wei, “Mining unseen classes via regional objectness: A simple baseline for incremental segmentation,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 24 340–24 353, 2022.
  * [64] Y. Qiu, Y. Shen, Z. Sun, Y. Zheng, X. Chang, W. Zheng, and R. Wang, “Sats: Self-attention transfer for continual semantic segmentation,” _arXiv preprint arXiv:2203.07667_ , 2022.
  * [65] D. A. Ganea, B. Boom, and R. Poppe, “Incremental few-shot instance segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 1185–1194.
  * [66] K. Nguyen and S. Todorovic, “ifs-rcnn: An incremental few-shot instance segmenter,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 7010–7019.
  * [67] F. Milletari, N. Navab, and S.-A. Ahmadi, “V-net: Fully convolutional neural networks for volumetric medical image segmentation,” in _2016 fourth international conference on 3D vision (3DV)_. IEEE, 2016, pp. 565–571.
  * [68] U. Michieli and P. Zanuttigh, “Incremental learning techniques for semantic segmentation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_ , 2019, pp. 0–0.
  * [69] J.-W. Xiao, C.-B. Zhang, J. Feng, X. Liu, J. van de Weijer, and M.-M. Cheng, “Endpoints weight fusion for class incremental semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 7204–7213.
  * [70] F. Cermelli, M. Cord, and A. Douillard, “Comformer: Continual learning in semantic and panoptic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 3010–3020.
  * [71] Q. Hou, M.-M. Cheng, X. Hu, A. Borji, Z. Tu, and P. H. Torr, “Deeply supervised salient object detection with short connections,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 3203–3212.
  * [72] T. Liu, Z. Yuan, J. Sun, J. Wang, N. Zheng, X. Tang, and H.-Y. Shum, “Learning to detect a salient object,” _IEEE Transactions on Pattern analysis and machine intelligence_ , vol. 33, no. 2, pp. 353–367, 2010.
  * [73] Z. Li and D. Hoiem, “Learning without forgetting,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 40, no. 12, pp. 2935–2947, 2017.
  * [74] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” _International journal of computer vision_ , vol. 88, no. 2, pp. 303–338, 2010.
  * [75] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba, “Scene parsing through ade20k dataset,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 633–641.
  * [76] F. Cermelli, M. Mancini, S. R. Bulo, E. Ricci, and B. Caputo, “Modeling the background for incremental learning in semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9233–9242.
  * [77] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.