ge short=GE, long=General Encoder, foreign-plural= gs short=ContinualSeg,
long=Continual Segment, foreign-plural= fov short=FOV, long=field of view,
foreign-plural= fbkg short=FBkg, long=fake background, foreign-plural= kd
short=KD, long=knowledge distillation, foreign-plural=

# Continual Segment: Towards a Single, Unified and Non-forgetting Continual
Segmentation Model of 143 Whole-body Organs in CT Scans

Zhanghexuan Ji1,2† Dazhou Guo1† Puyang Wang1 Ke Yan1,3 Le Lu1 Minfeng Xu1,3
Jingren Zhou1,3 Qifeng Wang4 Jia Ge5 Mingchen Gao2 Xianghua Ye5∗ Dakai Jin1∗

1DAMO Academy, Alibaba Group 2 University at Buffalo 3Hupan Lab, 310023,
Hangzhou, China
4Sichuan Cancer Hospital 5The First Affiliated Hospital of Zhejiang University
$\dagger$ ZJ and DG contribute equally. $*$ For correspondence, please contact
XY<EMAIL_ADDRESS>and DJ (dakai.jin@alibaba-inc.com).
This paper is accepted by ICCV 2023

###### Abstract

Deep learning empowers the mainstream medical image segmentation methods.
Nevertheless, current deep segmentation approaches are not capable of
efficiently and effectively adapting and updating the trained models when new
segmentation classes are incrementally added. In the real clinical
environment, it can be preferred that segmentation models could be dynamically
extended to segment new organs/tumors without the (re-)access to previous
training datasets due to obstacles of patient privacy and data storage. This
process can be viewed as a continual semantic segmentation (CSS) problem,
being understudied for multi-organ segmentation. In this work, we propose a
new architectural CSS learning framework to learn a single deep segmentation
model for segmenting a total of 143 whole-body organs. Using the
encoder/decoder network structure, we demonstrate that a continually trained
then frozen encoder coupled with incrementally-added decoders can extract
sufficiently representative image features for new classes to be subsequently
and validly segmented, while avoiding the catastrophic forgetting in CSS. To
maintain a single network model complexity, each decoder is progressively
pruned using neural architecture search and teacher-student based knowledge
distillation. Finally, we propose a body-part and anomaly-aware output merging
module to combine organ predictions originating from different decoders and
incorporate both healthy and pathological organs appearing in different
datasets. Trained and validated on 3D CT scans of 2500+ patients from four
datasets, our single network can segment a total of 143 whole-body organs with
very high accuracy, closely reaching the upper bound performance level by
training four separate segmentation models (i.e., one model per dataset/task).

Figure 1: Illustration of the continual multi-organ segmentation. At each
continual learning step, only the previously trained model is available (green
arrow). Previous datasets are not accessible. We allow organs from different
datasets to have overlaps, and these datasets may also contain diseased organs
(with tumors).

##  1 Introduction

Multi-organ segmentation has been extensively studied in medical imaging
because of its core importance for many downstream tasks, such as quantitative
disease analysis [27, 17], computer-aided diagnosis [51, 7], and cancer
radiotherapy planning [31, 68, 29]. With the emergence of many dedicated
labeled organ datasets [2] and the fast developments in deep learning
segmentation techniques [26], deep segmentation networks trained on specific
datasets achieve comparable performance with human observers [59, 68, 56].
However, this setup can have serious limitations in practical deployment for
clinical applications. These trained models are pre-trained to segment a fixed
number of organs, while in real clinical practice, it is desirable that
segmentation models can be dynamically extended to enable segmenting new
organs without the (re-)access to previous training datasets or without
training from scratch. In this way, patient privacy and data storage issues
can be solved, and model development and deployment can be much more
efficient. This clinically preferred process can be viewed as continual
semantic segmentation (CSS), which is emerging very recently in the natural
image domain [42, 6, 13, 72, 43, 65, 5] but has been only scarcely studied for
medical imaging [45, 37]. Notably, if all labeled datasets are simultaneously
accessible, it simplifies to a federated learning [50, 55] or partial label
learning [16, 57] problem. However, labeled datasets are always sequentially
built over time by annotating different organs of interest according to
various clinical tasks.

Multi-organ CSS faces several major challenges. First, since old datasets are
not accessible when training on the new dataset, deep networks may easily
forget the previously learned knowledge if no additional constraints are
added, which is the most prominent issue (known as catastrophic forgetting
[60, 32]) in continual learning. Second, in contrast to natural image datasets
that are often completely labeled [15, 74], fully annotated medical image
datasets are rare, especially for comprehensive multi-organ datasets. For
example, concerning both necessity and cost, labeling 143 organs for all
datasets is simply infeasible or impossible. These partially labeled datasets
bring up the label conflict issue (semantic shift of the background class
[6]), meaning a labeled organ in dataset-1 may become unlabeled background in
dataset-2. Third, domain incremental learning is common in multi-organ CSS,
since different datasets may contain overlapped yet “style-different” organs.
Appropriately tackling these domain gaps is non-trivial. E.g., dataset-1 is
made up of healthy subjects with normal esophagus annotated, while dataset-2
is a dedicated esophageal cancer dataset where esophagus with tumor is
labeled.

There are several recent CSS work in computer vision [42, 6, 13, 72, 43]. MiB
loss is often applied to handle the background-label conflicting issue [6,
13]. Regularization-based methods are mostly adopted to reduce the forgetting
of old knowledge while learning new classes. However, since network parameters
are updated on the training of new classes, it is extremely difficult to
achieve high performance on both old and new classes. There are few previous
works of CSS in medical imaging [45, 37]. Ozdemir et al. employed only 9
patients with 2 labels to develop a regularization-based CSS preliminary model
[45]. The most recent work [37] used MiB loss and prototype matching to
continually segment a small number of 5 abdominal organs focusing only on the
abdomen CT. When involving a large number of organs (e.g., $\geq$ 100 classes)
affiliated with a variety of body parts, such as in whole-body CT scans for
practical considerations, this strategy becomes non-scalable and suffers
severe performance degradation (as demonstrated in our experiments later).

A most recent continual classification work [64] has empirically shown that a
base classification model trained with a sufficiently large number of classes
(e.g., 800) is capable of extracting representative features even for new
classes. Hence, freezing most part of its parameters and incrementally fine-
tuning the newly added last convolutional block for each new task leads to an
almost non-forgetting continual classification model, whose performance is
close to the joint learning upper bound for both old and new classes.

Motivated by the observation in continual classification, in this work, we
propose a novel architecture-based continual multi-organ segmentation
framework. On the basis of the common encoder + decoder architecture of
segmentation networks, we demonstrate that its encoder is capable of
extracting representative deep features (non-specific to organ or body part)
for the new data. Hence, we can freeze the encoder and incrementally add a
separate decoder for each new learning task. Under this scheme, when adding a
new task, organs learned in previous tasks will never be forgotten because the
encoder is frozen, and previous decoders are independent of the new task. In
addition, the new decoder is trained separately to segment a fixed number of
foreground organs using only the new dataset. Hence, it avoids the background-
label conflict with previous datasets during training. Yet, this scheme can
lead to a swelling model as tasks expand. To make it scalable, a progressive
trimming method using neural architectural search (NAS) and teacher-student-
based knowledge distillation (KD) is exploited to maintain the overall model
complexity and inference time comparable to the original single network.
Finally, to merge organ predictions originating from different decoders and
incorporate both healthy and pathological organs appearing in different
datasets, we propose a body-part and anomaly-aware output merging scheme using
automated body part and tumor predictions.

Figure 2: Overall framework of the proposed continual multi-organ
segmentation, which is composed of a General Encoder, multiple optimized and
pruned decoders (one for each learning step), and a body-part and anomaly-ware
output merging module. After training the base encoding/decoding segmentation
network using $D_{1}$, the General Encoder is frozen afterward, and separate
trainable decoders are incrementally added to continually learn new datasets,
which leads to a non-forgetting architecture. Decoder optimization and pruning
are applied at each learning step to maintain a reasonable model complexity.
Finally, the merging module is designed to combine organs from all decoders.

In summary, the main contributions are as follows:

  * •

We are the first to comprehensively study the multi-organ continual semantic
segmentation (CSS) problem with a clinically desirable number of organs (143
organs) across different body parts (head & neck, chest, abdomen) to more
sufficiently and efficiently support medical diagnosis and treatment planning
purposes.

  * •

We propose the first (pure) architecture-based multi-organ continual
segmentation framework. Consisting of a general encoder, continually expanded
and pruned decoders, and a body-part and anomaly-aware output merging module,
the proposed network avoids the notorious catastrophic forgetting in CSS while
being scalable (maintaining the model complexity similar to other types of CSS
approaches).

  * •

Continually trained and validated on 3D CT scans of 2500+ patients compiled
from four different datasets, our scalable unified model can segment total of
143 whole-body organs with very high accuracy, closely reaching the upper
bound performance level of four well-trained individual models (i.e., nnUNet
[26]).

##  2 Related Work

Multi-Organ Segmentation. Automated multi-organ segmentation (MOS) is a
challenging task in medical imaging with a long study history. The early
registration-based atlas approach faces difficulty when large organ variation,
tumor growth, or image acquisition differences exist. Recently, deep learning-
based methods [26, 70, 31, 20, 21, 28, 48] have achieved great success when
working on specific datasets with the same set of labeled organs. However, in
practice, there are often partially labeled datasets, each with only one or a
few labeled organs. Several recent works explore training a joint single model
leveraging on multiple partially labeled datasets [75, 16, 57, 46, 73]. To
address the major issue of background label conflicts, the marginal loss is
often adopted to merge all unlabeled organs with the background [16, 57].
Different from these previous works that require all training datasets to be
available/accessible at once, we train a single multi-organ segmentation model
incrementally on multi-center partially-labeled datasets, with no access to
previous datasets during the sequential process.

Continual Learning. Continual Learning aims to update a model from a sequence
of new tasks and datasets without catastrophic forgetting [19, 32]. There are
three main categories [11]. Rehearsal-based methods store a limited amount of
training exemplars from old classes as raw images [49, 23, 38, 10, 3],
embedded features [22, 25] or generators [44, 58]. However, it may be
impracticable in real-world practice when data privacy is concerned, e.g.,
medical scans across multiple hospital sites are inaccessible. Regularization-
based methods constrain the model plasticity either through regularization on
weights [1, 8, 33, 71, 34] and gradients [39, 9], or knowledge distillation on
output logits [36, 53, 49, 4] and intermediate features [12, 14, 76, 77].
Although without storing exemplars, they cannot guarantee desirable
performance on challenging tasks. Architecture-based methods aim at either
dynamically dividing task-specific partial network [18, 24, 41, 54], which
suffers from running out of trainable parameters or expanding the network by
freezing the old model and adding new parameters for new tasks [52, 35, 61,
69, 64, 62, 40], which guarantee no-forgetting performance but result in
gradually growing/swelling model sizes. Our work falls into the expanding
category, and we perform network pruning for each new task to control the
overall model complexity.

Continual Semantic Segmentation. Continual semantic segmentation (CSS) is an
emerging research topic with limited previous studies. Besides catastrophic
forgetting, CSS faces the same challenge as partially labeled segmentation
known as background shift [6]. ILT [42] proposes a CIS setting with a simple
knowledge distillation solution. MiB [6] adapts marginal loss for both
classification and distillation to solve background shift. A local-pooling-
based distillation is applied to intermediate features in PLOP [13]. CSWKD
[47] weights the distillation loss based on the old and new class similarity.
SDR [43] propose to regularize the latent feature space using prototype
matching and contrastive learning. Other than knowledge distillation, RCIL
[72] designs a two-branch module for decoupling the representation learning of
old and new classes. In multi-organ segmentation, only one study LISMO [37]
applies CSS, based on MiB and prototype matching adapted from SDR [43], to
segment five abdominal organs, which is an easy setting merely focusing on a
single body part (abdomen). Our work is generalized for significantly more
organ classes that are located in a large range of body parts (head & neck,
chest, abdomen, hip & thigh).

##  3 Method

Problem Formulation. We aim to sequentially and continuously learn a single
multi-organ segmentation model from several partially-labeled datasets one by
one. Let $D=\\{D_{1},\dots,D_{T}\\}$ denote a sequence of data. When training
on $D_{t}$, all previous training data $\\{D_{p},p<t\\}$ are not accessible.
For the $t^{th}$ dataset $D_{t}=\\{{X_{i}}^{t},{Y_{i}}^{t}\\}_{i=1}^{n_{t}}$
with $C_{t}$ organ classes, let $X^{t}$ and $Y^{t}$ denote the input image and
the corresponding organ label in the $t^{th}$ dataset, the prediction map for
voxel location, $j$, and output class $c^{t}$:

$\displaystyle\hat{Y}^{t}(j)$
$\displaystyle=f_{d}\left(Y^{t}(j)=c^{t}|f_{e}\left(X^{t};W_{e}\right);W_{d}\right),$
(1) $\displaystyle\hat{\mathbf{Y}}$
$\displaystyle=\bigcup_{t=1}^{T}{\hat{Y}^{t}},$ (2)

where $f_{e}$, $f_{d}$, $W_{e}$, and $W_{d}$ denote the CNN functions and the
corresponding parameters for the encoding and decoding paths, respectively.
The final prediction $\hat{\mathbf{Y}}$ is the union (with possible class
overlapping) of all previous predictions.

Overall Training Process. Figure 2 illustrates the proposed multi-organ
continual segmentation framework, which is composed of an encoder, multiple
optimized and pruned decoders (one for each $D_{t}$), a body-part, and
anomaly-ware output merging module. It starts from training a base
encoding/decoding segmentation network using a comprehensive dataset $D_{1}$.
We hypothesize that the well-trained encoder on $D_{1}$, represented as a
General Encoder, is capable of extracting representative features (universal
to all organs and datasets) to facilitate the subsequent learning tasks.
Hence, this General Encoder is fixed afterward, and separate trainable
decoders are incrementally added at the future learning steps, which leads to
a non-forgetting architecture. Decoder optimization and pruning are also
conducted at each learning step to maintain the model complexity comparable to
a single network. Finally, by merging predictions from all decoders, we obtain
a single unified model that can segment all organs of interest.

###  3.1 General Encoder Training

Ideally, for whole-body multi-organ segmentation, we expect to construct a
sufficiently representative and universal General Encoder that extracts deep
image features to capture and encode all visual information inside the full
human body. Compared to the image statistics of broad natural image databases,
medical images exist in a much more confined semantic domain, i.e., the human
body is anatomically structured and composed of distinct body parts, no matter
with or without diseases. This makes it feasible to learn a strong universal
General Encoder competently capturing the holistic human body CT imaging
statistics using large or not-so-limited multi-organ datasets. Sharing a
similar idea, a very recent continual classification work [64] has empirically
shown that a base classification model trained with a sufficiently large
number of classes (e.g., 800) in ImageNet is capable of extracting
representative features even for new classes. Here, our goal is to build a
single unified segmentation model to accurately and continually segment up to
143 whole-body organs in CT scans (appeared in multiple datasets of both
healthy subjects and diseased patients).

To train the General Encoder for multi-organ continual segmentation, we
recommend starting with the publicly available TotalSegmentator [63] dataset
as $D_{1}$, which consists of 1204 CT scans with a total of 103 labeled whole-
body organs. These are routine diagnostic CT scans of different body parts
with various scanning protocols. Besides this comprehensive dataset, we also
supplement the General Encoder with auxiliary body-part segmentation and
abnormal/tumor segmentation tasks. The body part labels can be obtained based
on axial CT slice scores predicted by an automated body part regression
algorithm [66]. As the slice score is monotonously correlated with the
patient’s anatomic height, slices with key landmarks can be determined to
divide the whole body into four major regions, i.e., head & neck, chest,
abdomen, and hip & thigh. The abnormal/tumor segmentation head is trained
using dedicated tumor datasets. By involving these additional tasks, the
General Encoder explicitly recognizes each pixel’s anatomy region (body part)
and potential abnormal tissues, which may be beneficial for learning better
pixel representations. Moreover, the body part and tumor segmentation results
can be further utilized in the output merging step to combine outputs from all
decoders and reduce potential distal false positives from different decoders.
For implementation, light-weighted body parts and tumor segmentation heads are
added to the General Encoder using only the FCN8-like projection layers
($0.04\times$ size of a regular decoder) [30].

###  3.2 Decoder Optimization & Pruning

As the continual segmentation step extends, the proposed model complexity may
escalate. Therefore, after initially training the decoder at each continual
step, we further apply a progressive optimization and pruning procedure to
scale down the decoder complexity with the least possible performance drop.

Decoder Optimization via Neural Architectural Search (NAS). We first conduct
NAS to optimize the decoder’s segmentation performance and possibly reduce the
decoder’s parameters. Let $\phi\left(\cdot;\omega_{x\times y\times z}\right)$
denote a composition function of consecutive operations: batch normalization,
a rectified linear unit, and a convolution layer with an $x\times y\times z$
dimension kernel. Inspired by previous work [78, 20], different convolutional
layers may require various 2D/3D kernel types to segment 3D organs. Hence, we
search for a set of possible convolutional kernels tailored to our problem:
projection convolution $\phi\left(\cdot;\omega_{1\times 1\times 1}\right)$, 2D
convolution $\phi\left(\cdot;\omega_{3\times 3\times 1}\right)$, pseudo-3D
(P3D) convolution $\phi\left(\phi\left(\cdot;\omega_{3\times 3\times
1}\right);\omega_{1\times 1\times 3}\right)$, and 3D convolution
$\phi\left(\cdot;\omega_{3\times 3\times 3}\right)$. To simplify the searching
process, we use only one type of convolutional kernel to build each decoding
block. At the end of the search, we determine the architecture of each block
by choosing the $\phi$ corresponding to the largest weight value. Besides the
optimized decoder performance, the searched 2D and P3D kernel parameters are
only 1/3 and 4/9 of the 3D one, which also trims down the network parameter
numbers.

Figure 3: Illustration of the decoder optimization and pruning via neural
architectural search and knowledge distillation.

Decoder Compression via Knowledge Distillation (KD). After NAS, we further
prune the decoder by designing a convolution block-wise teacher-student-based
Knowledge Distillation (KD) method. Each convolutional block is fixed and used
as the teacher block. Next, we pair each teacher block with a projection block
(i.e., a convolutional block with projection layers with kernel size 1),
aiming to replace the teacher block with this projection block. The mean-
square error loss is adopted to match the feature maps of the teacher block to
the student block. Note that the student blocks have no path connection (hence
no gradient back-propagation). To reduce the optimization difficulty, the
deeper level of the decoding blocks is optimized first. Once the KD training
of the deeper blocks is saturated, we freeze them and progressively move to
the shallower ones. Figure 3 illustrates the pruning method. After this
process, there are $2^{5}$ decoding paths when choosing between the original
and the projection convolutional block, where all possible combinations are
enumerated, and the corresponding segmentation performance and decoding
parameter numbers are recorded. We use the decreased segmentation Dice score
(%) to select the most possibly pruned decoding path. This decreased Dice
score is defined by a performance drop tolerance parameter $\tau$. In ablation
experiments, we use $\tau\in\\{1\%,3\%,5\%\\}$ to inspect the model
compression results. The final results are reported using $\tau=1\%$. For the
detailed distillation training process, please refer to the supplementary
materials.

###  3.3 Body-part & Anomaly-aware Output Merging

We exploit the body part and anomaly predictions from two auxiliary tasks and
propose a simple yet effective rule-based approach to combine the predictions
from all decoders. Specifically, for each dataset/task, we pre-compute the
merged bounding boxes of all labeled organs. Next, we calculate the average
body part distribution map $P^{t}$ for each dataset $t$ by overlapping the
averaged bounding box to the body part labels. Let $\hat{Y}^{\epsilon}$ denote
the distinct tumor prediction, $\odot$ denote the element-wise multiplication,
and $J$ denote the matrix of ones, the weighting map $M^{t}$ is calculated
using Eq. (3), i.e., only when $\hat{Y}^{\epsilon}\rightarrow 0$ and
$P^{t}\rightarrow 1$ s.t. the $M^{t}\rightarrow 1$, whereas $M^{t}\rightarrow
0.5$ for the rest states. We use the entropy function Eq. (4) to compute the
confidence map.

$\displaystyle M^{t}$
$\displaystyle=J-\frac{1}{2}\left(J-P^{t}+\hat{Y}^{\epsilon}\odot
P^{t}\right)$ (3) $\displaystyle H^{t}$
$\displaystyle=-\left(M^{t}\odot\hat{Y}^{t}\right)\log\left(M^{t}\odot\hat{Y}^{t}\right),$
(4) $\displaystyle\mathbf{H}(j)$
$\displaystyle=\bigcup_{\forall\hat{Y}(j)^{t}\neq
0}{H^{t}(j)},t\in\\{1,\dots,T\\},$ (5) $\displaystyle\hat{\mathbf{Y}}(j)$
$\displaystyle=\hat{Y}^{\operatorname*{arg\,min}(\mathbf{H}(j))}(j)$ (6)

For each voxel, we collect a set $\mathbf{H}(j)$, for all $\hat{Y}(j)^{t}\neq
0$. Depicted in Eq. (6), the final output class $\hat{\mathbf{Y}}(j)$ is
determined using the prediction $\hat{Y}^{t}(j)$, of which with the smallest
$H^{t}(j)$. For the detailed merging setups, please refer to the supplementary
Sec. LABEL:S-supp:implement.

##  4 Experiments

Table 1: Continual multi-organ segmentation final results on two orders of our
datasets. Dataset names are followed by their class numbers. Mean DSC (%,
$\uparrow$), HD95 (mm, $\downarrow$) and ASD (mm, $\downarrow$) are evaluated
on each dataset as well as all classes (All). ‘Params #’: decoder(s) parameter
number of the final model (# (MB)) and the relative number (Rel #) compared to
the original nnUNet decoder. $\dagger$: ILT is reimplemented using a frozen
encoder setting and the unbiased loss from MiB for better performance.

| TotalSeg (103) | ChestOrgan (31) | HNOrgan (13) | EsoOrgan (1) | All (143) | Params #
---|---|---|---|---|---|---
Methods | DSC | HD95 | ASD | DSC | HD95 | ASD | DSC | HD95 | ASD | DSC | HD95 | ASD | DSC | HD95 | ASD | # (MB) | Rel #
| Order A: TotalSeg $\rightarrow$ ChestOrgan $\rightarrow$ HNOrgan $\rightarrow$ EsoOrgan |
MiB [6] | 7.65 | 119.66 | 67.41 | 19.24 | 37.14 | 8.34 | 6.37 | 7.40 | 2.38 | 86.92 | 4.33 | 1.09 | 8.51 | 98.98 | 51.98 |  |
$\textbf{ILT}^{\dagger}$ [42] | 10.87 | 192.23 | 116.20 | 27.87 | 36.93 | 7.41 | 6.39 | 4.04 | 0.81 | 85.75 | 4.57 | 1.17 | 11.99 | 148.96 | 86.34 |  |
PLOP [13] | 37.30 | 53.71 | 23.33 | 51.74 | 35.36 | 8.71 | 25.38 | 16.12 | 9.24 | 82.90 | 6.21 | 1.62 | 39.01 | 46.63 | 18.48 |  |
LISMO [37] | 10.82 | 129.82 | 76.92 | 28.24 | 36.33 | 9.08 | 6.30 | 12.93 | 4.14 | 87.12 | 4.24 | 1.05 | 12.11 | 96.89 | 54.71 |  |
RCIL [72] | 42.58 | 48.28 | 23.24 | 57.76 | 33.95 | 9.12 | 27.96 | 16.88 | 8.59 | 84.72 | 5.95 | 1.16 | 42.43 | 44.89 | 18.67 | 15.068 | 1.00
Ours | 92.98 | 4.09 | 0.98 | 78.26 | 9.17 | 1.82 | 83.97 | 2.22 | 0.59 | 86.94 | 5.04 | 1.11 | 88.74 | 5.28 | 1.14 | 14.669 | 0.98
| Order B: TotalSeg $\rightarrow$ HNOrgan $\rightarrow$ ChestOrgan $\rightarrow$ EsoOrgan |  |
MiB [6] | 10.35 | 136.77 | 63.51 | 65.63 | 14.37 | 1.94 | 6.29 | 24.83 | 7.22 | 86.79 | 4.31 | 1.08 | 20.00 | 68.82 | 29.87 |  |
$\textbf{ILT}^{\dagger}$ [42] | 13.12 | 201.66 | 106.51 | 67.28 | 14.21 | 1.88 | 6.18 | 3.12 | 0.95 | 85.52 | 4.80 | 1.25 | 22.31 | 115.23 | 59.34 |  |
PLOP [13] | 30.82 | 62.07 | 23.14 | 70.18 | 13.05 | 2.36 | 15.77 | 11.09 | 3.84 | 83.41 | 6.11 | 1.54 | 36.49 | 44.78 | 16.01 |  |
LISMO [37] | 14.04 | 90.17 | 47.81 | 67.19 | 14.88 | 1.93 | 6.15 | 9.13 | 1.44 | 86.87 | 4.18 | 1.03 | 22.92 | 57.71 | 28.22 |  |
RCIL [72] | 35.24 | 59.81 | 24.20 | 70.74 | 12.98 | 2.22 | 18.43 | 11.81 | 3.65 | 84.17 | 6.14 | 1.09 | 39.85 | 45.52 | 15.07 | 15.068 | 1.00
Ours | 92.98 | 4.09 | 0.98 | 78.26 | 9.17 | 1.82 | 83.97 | 2.22 | 0.59 | 86.94 | 5.04 | 1.11 | 88.74 | 5.28 | 1.14 | 14.669 | 0.98
Upper bound | 93.24 | 3.29 | 0.83 | 78.45 | 8.16 | 1.83 | 84.35 | 2.38 | 0.60 | 87.15 | 4.44 | 0.98 | 89.02 | 4.41 | 1.06 | 15.07$\times$4 | 1.0$\times$4

Figure 4: The mean DSCs over all learned classes at each step of Order A
(left, solid line) and Order B (right, dashed line).

Datasets: We evaluated our method using 2500+ patients from one public and
three private partially labeled multi-organ datasets. TotalSegmentator [63]
consists of 1204 CT scans of different body parts with a total of 103 labeled
anatomical structures (26 major organs, 59 bone instances, 10 muscles, and 8
vessels). Note that the face label is removed as it is an artificially created
label for patient de-identification purposes after blurring the facial area.
In the in-house collection, the ChestOrgan dataset contains 292 chest CT
scans, most of which come from early esophageal or lung cancer patients. For
the ChestOrgan dataset, 31 chest anatomical structures are labeled, among
which 4 overlapped with organs in TotalSegmentator (esophagus, trachea, SVC,
pulmonary artery). Another dataset includes 447 head & neck CT scans (denoted
as HNOrgan dataset), where 13 organs are annotated as organs at risk (OARs) in
radiation therapy and do not have class overlap with all other datasets. The
fourth dataset is a dedicated cancer dataset validating the domain change of
CSS, containing 640 diagnostic CT scans of advanced esophageal cancer patients
where only the esophagus is labeled (denoted as the EsoOrgan dataset). For the
detailed organ list, please refer to the supplementary Sec.
LABEL:S-supp:dataset. By combining all datasets, we have a total of
$103+27+13=143$ organ classes from 2583 unique patients. For each of these
four datasets, $20\%$ is randomly chosen as an independent testing set, while
the rest is used as training + validation in each continual learning step.

In addition, for the purpose of training and validating our abnormality
segmentation module, we further collect CT scans from 304 esophageal (private)
and 625 lung cancer (public with labels) patients where the 3D tumor masks are
segmented.

Overall CSS Training Process: In our CSS experiment, the model is trained to
segment organs sequentially at multiple steps. At each step $t$, the model is
trained on the specific dataset $D_{t}$ without access to any other datasets.
Specifically, at step-1, $D_{1}$ is first used to train both the General
Encoder and the associated decoder, where the decoder is further optimized and
pruned using $D_{1}$. After that, $D_{1}$ cannot be accessed in any future
steps. This process is repeated for step 2, $...$, $T$, except that at each
step-$t$, $D_{t}$ is only used to train, optimize and prune $D_{t}$ dedicated
decoder keeping the General Encoder always frozen.

CSS Protocols: We examine two CSS orders with four learning steps. Order A
goes as: TotalSegmentator $\rightarrow$ ChestOrgan $\rightarrow$ HNOrgan
$\rightarrow$ EsoOrgan. Order B goes as: TotalSegmentator $\rightarrow$
HNOrgan $\rightarrow$ ChestOrgan $\rightarrow$ EsoOrgan, which exchanges the
ChestOrgan with HNOrgan to demonstrate the effect of different body parts in
CSS. All methods (including ours) are trained and evaluated in both orders. To
report the final results in CSS setting, we compute segmentation metrics after
the last learning step for all the previous datasets. For reporting the
results in any intermediate step $t$, these metrics are calculated after the
learning step $t$ for all the datasets $i\leq t$.

Metrics: We report the Dice similarity coefficient (DSC), $95\%$ Hausdorff
distance (HD95) and average surface distance (ASD) to quantify the organ
segmentation results.

###  4.1 Implementation Details

A $[-1024,1024]$ HU CT windowing is applied to every CT image. We resample all
CT scans to the same resolution: $0.75\times 0.75\times 3.0$mm. The ratio
between the training and validation set is 4:1. “3d-full res” version (+
“moreDA” data augmentation) of nnUNet [26] with DSC+CE losses is adopted for
all model training with a batch size of 2. The training patch size is
$128\times 128\times 64$. We set 8000 epochs for training General Encoder and
the associated decoder using the TotalSegmentator dataset in step-1, and 1000
epochs for training the dedicated decoder at each future step-$t$.

NAS Setting: At a learning step $t,1\leq t\leq T$, after initially training
the decoder, we further exploit NAS to search for the optimal network
architecture for the associate decoder. For NAS training, the dataset $D_{t}$
(training+validation) is initially divided into 1) $60\%$ for network
training, 2) $30\%$ for NAS training, and 3) $10\%$ for validation evaluation.
The initial learning rate is set to 0.01. We first fix the NAS parameters and
train the network for 400 epochs. Then we alternatively update the NAS and
network parameters for additional 600 epochs. The batch size is set to 4 in
NAS training. Only the validation set is used for updating NAS parameters.
After NAS training, we follow the same ‘moreDA’ data augmentation scheme and
retrain the searched decoding architecture from scratch using $D_{t}$
(training+validation) with a re-divided ‘training-validation’ ratio of 4:1.

Pruning Setting: After NAS, we perform a block-wise teacher-student KD to
compress the decoder by replacing the searched convolutional kernels with the
projection kernels. The initial learning rate is 0.01. We fix the teacher
networks and train the paired student network for another 500 epochs. MSE loss
is used for teacher-student feature map matching. After the pruning is
completed, we replace the selected teacher blocks with the student blocks and
finetune the trimmed network for 500 epochs with a learning rate of 0.001. All
models are developed using PyTorch and trained on one NVIDIA A100 GPU. Please
refer to the supplementary Sec. LABEL:S-supp:implement for more implementation
details.

Comparing Methods: We compare our method with five latest leading CSS works,
including four regularization-based methods (ILT [42], MiB [6], PLOP [13],
RCIL [72]) and a hybrid of regularization and rehearsal-based method (LISMO
[37]). To ensure comparisons’ fairness, we re-implement ILT, MiB, LISMO, and
PLOP in the nnUNet framework to guarantee consistent data pre-processing and
data augmentation (Re-implementation details are fully disclosed in the
supplementary Sec. LABEL:S-supp:impl_comp). Noted that all four datasets in
our experiment are partially labeled, hence, it is not straightforward to
compute the upper bound performance using a single model. In this work, we
train a separate nnUNet [26] model for each dataset, the results of which can
serve as the CSS performance upper bound for each dataset.

Table 2: Segmentation performance under two 1-step continual learning
scenarios with and without freezing the General Encoder. Mean DSC (%,
$\uparrow$), HD95 (mm, $\downarrow$) and ASD (mm, $\downarrow$) are evaluated.

Settings | TotalSeg $\rightarrow$ ChestOrgan | TotalSeg $\rightarrow$ HNOrgan
---|---|---
TotalSeg | ChestOrgan | TotalSeg | HNOrgan
DSC | HD95 | DSC | HD95 | DSC | HD95 | DSC | HD95
unfreezing | 51.42 | 26.52 | 78.45 | 8.16 | 2.90 | 162.09 | 84.35 | 2.38
freezing | 92.98 | 4.09 | 77.91 | 8.37 | 92.98 | 4.09 | 84.14 | 2.39

###  4.2 Comparison to Leading CSS Methods

Overall Performance: Table 1 and Figure 4 show final segmentation results
after continually learning on two orders (each with four steps) of our
datasets. Our proposed method significantly outperforms other leading methods
on the previously learned three datasets as well as the total 143 organs in
both CSS orders. The second best performing method RCIL [72] still experiences
catastrophic forgetting and has a mean DSC of 41.43% and 39.85% in CSS order A
and B, far less than our mean DSC of 88.74%. Similar performance gaps are
noticed on HD95 and ASD metrics (e.g., $\sim$45mm vs. $\sim$5mm in terms of
HD95). Our proposed method achieves very similar performance to the upper
bound with a 0.28% marginal decrease in DSC and a 0.08mm increase in ASD (see
Figure 5 for qualitative results). In model complexity, the overall parameter
number of our four pruned decoders (14.7 MB) is 98% of an original nnUNet
decoder (15.1 MB), which is only 24% size of the decoders required by
achieving the upper bound performance. The running time of the proposed
framework (segmenting 143 organs) is slightly longer (+$12\%$) than the
running time of a single nnUNet to segment 103 organs. For our detailed
results of individual organs or organ groups, please refer to the
supplementary Sec. LABEL:S-supp:organ.

Two CSS Orders: Table 1 also demonstrates the segmentation results under two
CSS orders (order A and B). Because the proposed framework consists of a
frozen General Encoder, independent decoders (each for one continual learning
step), and a unified output merging module, our method is order invariant if
the base dataset for training General Encoder is the same. On the other hand,
the continual learning order may significantly affects the comparison methods.
E.g., LISMO has a mean DSC of 28.24% v.s. 67.19% on ChestOrgan dataset in
order A and B, respectively.

The significant performance drop of the comparing methods could be caused by
the catastrophic forgetting induced from the bodypart-related domain gap. In
our experiments, we observe that the comparing methods generally work well if
new and old datasets share similar domains/body-parts, e.g. ChestOrgan
$\rightarrow$ EsoOrgan (second from the left plot of bottom row in the
supplementary Figure C.1). However, in whole-body organ continual
segmentation, different datasets may cover various body parts with limited
overlaps, which causes a large gap in the image domain and significantly
deteriorates the performance, e.g. ChestOrgan $\rightarrow$ HNOrgan (third
from the left plot of bottom row in the supplementary Figure C.1). In
contrast, when learning new tasks, our framework keeps previously learned
parameters unchanged and avoids knowledge forgetting. Please refer to the
supplementary Sec. LABEL:S-supp:ablation for more detailed results and
discussion on the step-wise results of our method and comparing methods.

Table 3: Multi-organ segmentation results using decoder optimization &
pruning. We report the number of decoder parameters and the relative size
percentage compared to the original nnUNet decoder when the DSC (%) is dropped
by $\tau\in\\{1\%,3\%,5\%\\}$.

| DSC Drop
---|---
1% | 3% | 5%
TotalSeg | DSC | 92.98 | 90.72 | 88.83
#(MB) | 6.53 | 4.50 | 3.28
Rel # | 0.43 | 0.30 | 0.22
ChestOrgan | DSC | 78.26 | 77.16 | 74.88
#(MB) | 3.39 | 2.85 | 1.23
Rel # | 0.23 | 0.19 | 0.08
HNOrgan | DSC | 83.97 | 82.24 | 80.27
#(MB) | 4.18 | 4.04 | 1.88
Rel # | 0.28 | 0.27 | 0.12
EsoOrgan | DSC | 86.94 | 85.97 | –
#(MB) | 0.67 | 0.57 | –
Rel # | 0.04 | 0.04 | –

###  4.3 Ablation Study Results

Effectiveness of General Encoder: To demonstrate the importance of freezing
the General Encoder when learning subsequent tasks, we compare the
segmentation performance with and without freezing the General Encoder when
continually learning on new datasets (using two CSS orders with two learning
steps). Results are summarized in Table 2. First, it is observed that without
freezing the General Encoder, the model has catastrophic forgetting, e.g.,
segmentation DSC of the old dataset in TotalSegmentator $\rightarrow$
ChestOrgan decreases from 93.24% to 51.42% as compared to that with the frozen
encoder. Second, the performance for segmenting the new dataset is similar
regardless of the encoder status (freezing or trainable). For instance, 84.14%
vs. 84.35% DSC of HNOrgan dataset is achieved in TotalSegmentator
$\rightarrow$ HNOrgan. The experimental results demonstrate that a well-
trained and subsequently frozen General Encoder could generalize well to
support specialized tasks.

Effectiveness of Decoder Pruning: Table 3 shows the detailed decoder pruning
results. Several conclusions can be drawn. First, the proposed decoder pruning
method achieves a good trade-off between model complexity and accuracy
reduction. For example, for the TotalSegmentator decoder, with 1% DSC
decrease, the number of parameters is reduced from 15.07 MB to 6.53 MB with a
relative 43% of the original decoder size. As the larger performance drop is
allowed, e.g., 3% and 5% DSC decrease, the size of pruned decoder decreases to
30% and 22% of the original decoder, respectively. Second, as the number of
segmented organs becomes smaller, a higher compressed ratio can be achieved.
With 1% DSC performance decrease, the pruned ChestOrgan decoder (segmenting 31
organs) has 3.39 MB parameters as compared to 6.53 MB of pruned
TotalSegmentator decoder. Third, the EsoOrgan decoder has the highest model
compression ratio with only 0.67 MB parameters (4% of original decoder size).
This indicates that domain-incremental segmentation may be an easier task as
compared to class-incremental continual segmentation.

Effectiveness of Merging Module: Table 4 presents the segmentation results
using two merging methods. It is observed that a simple ensemble-based merging
method exhibits decreased performance in all metrics on all datasets. The
proposed anomaly-aware output merging significantly boosts the performance on
the EsoOrgan dataset (DSC: 80.22% to 86.94%, HD95: 7.62 to 5.04mm, ASD: 1.92
to 1.11mm). This demonstrates the effectiveness and importance of the abnormal
detection module. The proposed merging module can identify the esophageal
tumor and subsequently generate a high confidence score for the EsoOrgan
decoder suitable for segmenting advanced esophageal cancer patients. In
contrast, the ensemble method could not differentiate if there exists
abnormality in an image. Hence, averaging the esophagus predictions from three
decoders that predict the esophagus leads to significantly decreased
performance.

Figure 5: The qualitative comparison of our method with MiB [6] and PLOP [13].
All the segmentation results are from the last step of CSS order A. Seven
quality examples are shown covering different body parts and organ groups.
Yellow arrows indicate missing/wrong predictions. MiB and PLOP experience
severe forgetting in the head, abdomen and pelvis regions, since these body
parts only appear in one or two tasks/datasets; while less forgetting is
observed in neck and upper chest regions, which appear in all four
tasks/datasets (more suitable for CSS by the MiB and PLOP methods). (For
visualization purpose, not all the organs are shown in each example.) Table 4:
Quantitative results of using different output merging methods. Mean DSC (%),
HD95 (mm) and ASD (mm) are evaluated. Better performance is indicated in bold.

| Ensemble | Anomaly-aware merging
---|---|---
| DSC | HD95 | ASD | DSC | HD95 | ASD
TotalSeg | 88.59 | 4.41 | 1.09 | 92.98 | 4.09 | 0.98
ChestOrgan | 76.78 | 9.44 | 1.89 | 78.26 | 9.17 | 1.82
HNOrgan | 77.84 | 2.65 | 0.67 | 83.97 | 2.22 | 0.59
EsoOrgan | 80.22 | 7.62 | 1.92 | 86.94 | 5.04 | 1.11

Alternative Training Dataset for the General Encoder: We recommend starting
with TotalSegentator as $D_{1}$ to train the General Encoder as it covers most
body parts with a large set of labeled organs for comprehensive feature
extraction. However, this is not a hard requirement. Alternatively, other
datasets can also be used as the starting dataset to train the General
Encoder. When training the General Encoder using the ChestOrgan dataset with
much less training scans (292 vs. 1204 CT scans) and organ classes (31 vs. 103
anatomical structures), a tolerable performance drop ($<$1% Dice) of our
method is observed in the final results of CSS Order A. The assumed reason is
that CT scans in the ChestOrgan dataset covers most of the torso region with
diverse anatomies, which allows the General Encoder to learn sufficient
representative features. Hence, General Encoder trained with ChestOrgan
exhibits similar performance as the one trained using TotalSegmentator.

##  5 Conclusion

In this work, we propose a new CSS framework to continually segment a total of
143 whole-body organs from four partially labeled datasets. With the trained
and frozen General Encoder and continually-added and architecturally optimized
decoders, our model avoids catastrophic forgetting while effectively
segmenting new organs with high accuracy. We further propose a body-part and
anomaly-aware output merging module to combine organ predictions originating
from different decoders and incorporate both healthy and pathological organs
appearing in different datasets. Continually trained and validated on 3D CT
scans of 2500+ patients of four datasets, our single network can segment 143
whole-body organs with very high accuracy, closely reaching the upper bound
performance level by training four separate segmentation models.

## References

  * [1] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars.  Memory aware synapses: Learning what (not) to forget.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 139–154, 2018.
  * [2] Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers, et al.  The medical segmentation decathlon.  Nature communications, 13(1):1–13, 2022.
  * [3] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara.  Dark experience for general continual learning: a strong, simple baseline.  Advances in neural information processing systems, 2020.
  * [4] Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari.  End-to-end incremental learning.  In Proceedings of the European conference on computer vision (ECCV), pages 233–248, 2018.
  * [5] Fabio Cermelli, Matthieu Cord, and Arthur Douillard.  Comformer: Continual learning in semantic and panoptic segmentation.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3010–3020, 2023.
  * [6] Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulo, Elisa Ricci, and Barbara Caputo.  Modeling the background for incremental learning in semantic segmentation.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9233–9242, 2020.
  * [7] Chun-Hung Chao, Zhuotun Zhu, Dazhou Guo, Ke Yan, Tsung-Ying Ho, Jinzheng Cai, Adam P Harrison, Xianghua Ye, Jing Xiao, Alan Yuille, et al.  Lymph node gross tumor volume detection in oncology imaging via relationship learning using graph neural network.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 772–782. Springer, 2020.
  * [8] Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr.  Riemannian walk for incremental learning: Understanding forgetting and intransigence.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 532–547, 2018.
  * [9] Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny.  Efficient lifelong learning with a-gem.  arXiv preprint arXiv:1812.00420, 2018.
  * [10] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc’Aurelio Ranzato.  On tiny episodic memories in continual learning.  International Conference on Machine Learning (ICML) Workshop, 2019\.
  * [11] Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2021.
  * [12] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa.  Learning without memorizing.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5138–5146, 2019.
  * [13] Arthur Douillard, Yifu Chen, Arnaud Dapogny, and Matthieu Cord.  Plop: Learning without forgetting for continual semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4040–4050, 2021.
  * [14] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle.  Podnet: Pooled outputs distillation for small-tasks incremental learning.  In European Conference on Computer Vision, pages 86–102, 2020.
  * [15] Mark Everingham, SM Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes challenge: A retrospective.  International journal of computer vision, 111(1):98–136, 2015.
  * [16] Xi Fang and Pingkun Yan.  Multi-organ segmentation over partially labeled datasets with multi-scale feature abstraction.  IEEE Transactions on Medical Imaging, 39(11):3619–3629, 2020.
  * [17] Elise MN Ferré, Timothy J Break, Peter D Burbelo, Michael Allgäuer, David E Kleiner, Dakai Jin, Ziyue Xu, Les R Folio, Daniel J Mollura, Muthulekha Swamydas, et al.  Lymphocyte-driven regional immunopathology in pneumonitis caused by impaired central immune tolerance.  Science translational medicine, 11(495):eaav5597, 2019.
  * [18] Siavash Golkar, Michael Kagan, and Kyunghyun Cho.  Continual learning via neural pruning.  Advances in Neural Information Processing Systems (NeurIPS) Workshop, 2019.
  * [19] Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio.  An empirical investigation of catastrophic forgetting in gradient-based neural networks.  Proceedings of the International Conference on Learning Representations (ICLR), 2014.
  * [20] Dazhou Guo, Dakai Jin, Zhuotun Zhu, Tsung-Ying Ho, Adam P Harrison, Chun-Hung Chao, Jing Xiao, and Le Lu.  Organ at risk segmentation for head and neck cancer using stratified learning and neural architecture search.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4223–4232, 2020.
  * [21] Dazhou Guo, Xianghua Ye, Jia Ge, Xing Di, Le Lu, Lingyun Huang, Guotong Xie, Jing Xiao, Zhongjie Lu, Ling Peng, et al.  Deepstationing: thoracic lymph node station parsing in ct scans using anatomical context encoding and key organ auto-search.  In Medical Image Computing and Computer Assisted Intervention, pages 3–12. Springer, 2021.
  * [22] Tyler L Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan.  Remind your neural network to prevent catastrophic forgetting.  In European Conference on Computer Vision, pages 466–483, 2020\.
  * [23] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin.  Learning a unified classifier incrementally via rebalancing.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 831–839, 2019.
  * [24] Ching-Yi Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen.  Compacting, picking and growing for unforgetting continual learning.  Advances in Neural Information Processing Systems, 2019.
  * [25] Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, and Cordelia Schmid.  Memory-efficient incremental learning through feature adaptation.  In European Conference on Computer Vision, pages 699–715. Springer, 2020.
  * [26] Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein.  nnu-net: a self-configuring method for deep learning-based biomedical image segmentation.  Nature methods, 18(2):203–211, 2021.
  * [27] Krishna S Iyer, John D Newell Jr, Dakai Jin, Matthew K Fuld, Punam K Saha, Sif Hansdottir, and Eric A Hoffman.  Quantitative dual-energy computed tomography supports a vascular etiology of smoking-induced inflammatory lung disease.  American journal of respiratory and critical care medicine, 193(6):652–661, 2016.
  * [28] Zhanghexuan Ji, Yan Shen, Chunwei Ma, and Mingchen Gao.  Scribble-based hierarchical weakly supervised learning for brain tumor segmentation.  In Medical Image Computing and Computer Assisted Intervention, pages 175–183. Springer, 2019.
  * [29] Dakai Jin, Dazhou Guo, Jia Ge, Xianghua Ye, and Le Lu.  Towards automated organs at risk and target volumes contouring: Defining precision radiation therapy in the modern era.  Journal of the National Cancer Center, 2022.
  * [30] Dakai Jin, Dazhou Guo, Tsung-Ying Ho, Adam P Harrison, Jing Xiao, Chen-Kan Tseng, and Le Lu.  Accurate esophageal gross tumor volume segmentation in pet/ct using two-stream chained 3d deep network fusion.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 182–191. Springer, 2019.
  * [31] Dakai Jin, Dazhou Guo, Tsung-Ying Ho, Adam P Harrison, Jing Xiao, Chen-Kan Tseng, and Le Lu.  Deeptarget: Gross tumor and clinical target volume segmentation in esophageal cancer radiotherapy.  Medical Image Analysis, 68:101909, 2021.
  * [32] Ronald Kemker, Marc McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan.  Measuring catastrophic forgetting in neural networks.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [33] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  * [34] Abhishek Kumar, Sunabha Chatterjee, and Piyush Rai.  Bayesian structural adaptation for continual learning.  In International Conference on Machine Learning, pages 5850–5860. PMLR, 2021.
  * [35] Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong.  Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting.  In International Conference on Machine Learning, pages 3925–3934. PMLR, 2019.
  * [36] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017.
  * [37] Pengbo Liu, Xia Wang, Mengsi Fan, Hongli Pan, Minmin Yin, Xiaohong Zhu, et al.  Learning incrementally to segment multiple organs in a CT image.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 714–724. Springer, 2022.
  * [38] Yaoyao Liu, Bernt Schiele, and Qianru Sun.  Rmm: Reinforced memory management for class-incremental learning.  Advances in Neural Information Processing Systems, 34, 2021.
  * [39] David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  Advances in neural information processing systems, 30, 2017.
  * [40] Chunwei Ma, Zhanghexuan Ji, Ziyun Huang, Yan Shen, Mingchen Gao, and Jinhui Xu.  Progressive voronoi diagram subdivision enables accurate data-free class-incremental learning.  In The Eleventh International Conference on Learning Representations, 2022.
  * [41] Arun Mallya and Svetlana Lazebnik.  Packnet: Adding multiple tasks to a single network by iterative pruning.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765–7773, 2018.
  * [42] Umberto Michieli and Pietro Zanuttigh.  Incremental learning techniques for semantic segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [43] Umberto Michieli and Pietro Zanuttigh.  Continual semantic segmentation via repulsion-attraction of sparse and disentangled latent representations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1114–1124, 2021.
  * [44] Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi.  Learning to remember: A synaptic plasticity driven framework for continual learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11321–11329, 2019.
  * [45] Firat Ozdemir, Philipp Fuernstahl, and Orcun Goksel.  Learn the new, keep the old: Extending pretrained models with new anatomy and images.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 361–369. Springer, 2018.
  * [46] Olivier Petit, Nicolas Thome, and Luc Soler.  Iterative confidence relabeling with deep convnets for organ segmentation with partial labels.  Computerized Medical Imaging and Graphics, 91:101938, 2021.
  * [47] Minh Hieu Phan, Son Lam Phung, Long Tran-Thanh, Abdesselam Bouzerdoum, et al.  Class similarity weighted knowledge distillation for continual semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16866–16875, 2022.
  * [48] Ashwin Raju, Zhanghexuan Ji, Chi Tung Cheng, Jinzheng Cai, Junzhou Huang, Jing Xiao, et al.  User-guided domain adaptation for rapid annotation from user interactions: a study on pathological liver segmentation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 457–467. Springer, 2020.
  * [49] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.  icarl: Incremental classifier and representation learning.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017.
  * [50] Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletari, Holger R Roth, Shadi Albarqouni, Spyridon Bakas, Mathieu N Galtier, Bennett A Landman, Klaus Maier-Hein, et al.  The future of digital health with federated learning.  NPJ digital medicine, 3(1):1–7, 2020.
  * [51] Holger R Roth, Le Lu, Jiamin Liu, Jianhua Yao, Ari Seff, Kevin Cherry, Lauren Kim, and Ronald M Summers.  Improving computer-aided detection using convolutional neural networks and random view aggregation.  IEEE transactions on medical imaging, 35(5):1170–1181, 2015.
  * [52] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  arXiv preprint arXiv:1606.04671, 2016.
  * [53] Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell.  Progress & compress: A scalable framework for continual learning.  In International Conference on Machine Learning, pages 4528–4537, 2018.
  * [54] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou.  Overcoming catastrophic forgetting with hard attention to the task.  In International Conference on Machine Learning, pages 4548–4557. PMLR, 2018.
  * [55] Yan Shen, Jian Du, Han Zhao, Zhanghexuan Ji, Chunwei Ma, and Mingchen Gao.  Fedmm: A communication efficient solver for federated adversarial domain adaptation.  In International Conference on Autonomous Agents and Multiagent Systems, pages 1808–1816, 2023.
  * [56] Feng Shi, Weigang Hu, Jiaojiao Wu, Miaofei Han, Jiazhou Wang, Wei Zhang, Qing Zhou, Jingjie Zhou, Ying Wei, Ying Shao, et al.  Deep learning empowered volume delineation of whole-body organs-at-risk for accelerated radiotherapy.  Nature Communications, 13(1):1–13, 2022.
  * [57] Gonglei Shi, Li Xiao, Yang Chen, and S Kevin Zhou.  Marginal loss and exclusion loss for partially supervised multi-organ segmentation.  Medical Image Analysis, 70:101979, 2021.
  * [58] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim.  Continual learning with deep generative replay.  Advances in neural information processing systems, 30, 2017.
  * [59] Hao Tang, Xuming Chen, Yang Liu, Zhipeng Lu, Junhua You, Mingzhou Yang, Shengyu Yao, Guoqi Zhao, Yi Xu, Tingfeng Chen, et al.  Clinically applicable deep learning framework for organs at risk delineation in ct images.  Nature Machine Intelligence, 1(10):480–491, 2019.
  * [60] Sebastian Thrun.  Lifelong learning algorithms.  In Learning to learn, pages 181–209. Springer, 1998.
  * [61] Yu-Xiong Wang, Deva Ramanan, and Martial Hebert.  Growing a brain: Fine-tuning by increasing model capacity.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2471–2480, 2017.
  * [62] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister.  Learning to prompt for continual learning.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139–149, 2022.
  * [63] Jakob Wasserthal, Manfred Meyer, Hanns-Christian Breit, Joshy Cyriac, Shan Yang, and Martin Segeroth.  Totalsegmentator: robust segmentation of 104 anatomical structures in ct images.  arXiv preprint arXiv:2208.05868, 2022.
  * [64] Tz-Ying Wu, Gurumurthy Swaminathan, Zhizhong Li, Avinash Ravichandran, Nuno Vasconcelos, Rahul Bhotika, and Stefano Soatto.  Class-incremental learning with strong pre-trained models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9601–9610, 2022.
  * [65] Jia-Wen Xiao, Chang-Bin Zhang, Jiekang Feng, Xialei Liu, Joost van de Weijer, and Ming-Ming Cheng.  Endpoints weight fusion for class incremental semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7204–7213, 2023.
  * [66] Ke Yan, Le Lu, and Ronald M. Summers.  Unsupervised body part regression via spatially self-ordering convolutional neural networks.  In IEEE ISBI, pages 1022–2025, 2018.
  * [67] Ke Yan, Xiaosong Wang, Le Lu, and Ronald M Summers.  Deeplesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning.  Journal of medical imaging, 5(3):036501, 2018.
  * [68] Xianghua Ye, Dazhou Guo, Jia Ge, Senxiang Yan, Yi Xin, Yuchen Song, Yongheng Yan, Bing-shen Huang, et al.  Comprehensive and clinically accurate head and neck cancer organs-at-risk delineation on a multi-institutional study.  Nature communications, 13(1):1–15, 2022.
  * [69] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang.  Lifelong learning with dynamically expandable networks.  Proceedings of the International Conference on Learning Representations (ICLR), 2017.
  * [70] Qihang Yu, Dong Yang, Holger Roth, Yutong Bai, Yixiao Zhang, Alan L Yuille, and Daguang Xu.  C2fnas: Coarse-to-fine neural architecture search for 3d medical image segmentation.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4126–4135, 2020.
  * [71] Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In International Conference on Machine Learning, pages 987–995, 2017.
  * [72] Chang-Bin Zhang, Jia-Wen Xiao, Xialei Liu, Ying-Cong Chen, and Ming-Ming Cheng.  Representation compensation networks for continual semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7053–7064, 2022.
  * [73] Jianpeng Zhang, Yutong Xie, Yong Xia, and Chunhua Shen.  Dodnet: Learning to segment multi-organ and tumors from multiple partially labeled datasets.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1195–1204, 2021.
  * [74] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In IEEE conference on computer vision and pattern recognition, pages 633–641, 2017.
  * [75] Yuyin Zhou, Zhe Li, Song Bai, Chong Wang, Xinlei Chen, Mei Han, Elliot Fishman, and Alan L Yuille.  Prior-aware neural network for partially-supervised multi-organ segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10672–10681, 2019.
  * [76] Fei Zhu, Xu-Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu.  Prototype augmentation and self-supervision for incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5871–5880, 2021.
  * [77] Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, and Zheng-Jun Zha.  Self-sustaining representation expansion for non-exemplar class-incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9296–9305, 2022.
  * [78] Zhuotun Zhu, Chenxi Liu, Dong Yang, Alan Yuille, and Daguang Xu.  V-nas: Neural architecture search for volumetric medical image segmentation.  In 2019 International conference on 3d vision (3DV), pages 240–248, 2019.

##

## Appendix

##  Appendix A Dataset Details

We describe the dataset details (one public and three private multi-organ
datasets) used in our experiment here. For the public dataset TotalSegmentator
[63], it consists of 1204 CT scans of different body parts with total 103
labeled anatomical structures (26 major organs, 59 bone instances, 10 muscles,
and 8 vessels). Note that the face label is removed as it is an artificially
created label for patient de-identification purpose after blurring the facial
area. The detailed organ 103 organ instance list can be found in the link
https://github.com/wasserth/TotalSegmentator. For the three in-house multi-
organ datasets, they are head & neck organ dataset (denoted as HNOrgan), chest
organ dataset (denoted as ChestOrgan) and dedicated esophageal cancer dataset
(denoted as EsoOrgan). In HNOrgan, each of the 447 head and neck CT scans has
13 head and neck organs labeled: brainstem, eye (left and right), lens (left
and right), optic nerve (left and right), optic chiasm, parotid (left and
right), spinal cord, temporomandibular joint (TMJoint, left and right).
ChestOrgan contains 292 chest CT scans with 31 chest anatomical structures
annotated including major organs, muscles, arteries and veins. The detailed
list is as follow: esophagus, sternum, thyroid left, thyroid right, trachea,
bronchus left, bronchus right, anterior cervical muscle, scalenus muscle,
scalenus anterior muscle, sternocleidomastoid muscle, ascending aorta,
descending aorta, aorta arch, common carotid artery left, common carotid
artery right, pulmonary artery, subclavian artery left, subclavian artery
right, vertebral artery left, vertebral artery right, azygos vein,
brachiocephalic vein left, brachiocephalic vein right, internal jugular vein
left, internal jugular vein right, pulmonary vein, subclavian vein left,
subclavian vein right, inferior vena cava, superior vena cava. There are four
organs in ChestOrgan that are overlapped with organs in TotalSegmentator
(esophagus, pulmonary artery, superior vena cava, trachea). The EsoOrgan
collects 640 diagnostic CT scans of advanced esophageal cancer patient where
only the esophagus is labeled. By combining all datasets, we have total
103+27+13 organ classes from 2583 unique patients. For each of these four
datasets, $20\%$ are randomly chosen as the testing set, while the rest is
used as training + validation. Detailed training/validation dataset split in
the decoder optimization module can be found in the Implementation Details
section of this supplementary material.

In addition, for the purpose of training and validating our abnormality
detection module, we further collect CT scans from 304 esophageal (private)
and 625 lung cancer (public with tumor labels) patients where the 3D tumor
masks are delineated at the pixel level. These combine as the lung/esophageal
tumor classes from additional 929 patients.

##  Appendix B Implementation Details

The default nnUNet backbone in 3D full resolution setting is adopted in our
work, including a 6-block encoding path and a 5-block decoding path. Each
encoding block consists of the following consecutive operations: a
convolution, an instance normalization, a Leaky ReLU unit, followed by max-
pooling operator.

The total training epoch for the baseline TotalSegmentator is 8000 with 250
iterations per epoch, and the training epoch for each of the in-house datasets
(served as performance upper bound) is 1000 with 250 iterations per epoch. The
batch size is 2. The optimizer is stochastic gradient descent with a
Polynomial learning rate policy. The initial learning rate is 0.01 with a
Nesterov momentum of 0.99. Default “moreDA” data augmentation is adopted,
e.g., horizontal flipping, random rotation in the x-y plane with $\pm 10$
degrees, intensity scaling with a ratio between $[0.75,1.25]$, adding Gaussian
noise with zero mean and $[0,0.1]$ variance. The total average training time
is 2.5 GPU days per 1000 epochs. For model inference, the average running time
for the proposed framework, before the decoding path optimization, is
approximately 15 minutes per patient. After the decoding path optimization,
the average inference time is less than 5 minutes per patient. All models are
developed using PyTorch and trained on one NVIDIA A100 GPU.

Table B.1: The detailed auto-searched and pruned decoding architecture based
on nnUNet. Note that decoder block 5 refers to the deepest decoding block.

|  | Decoder Block 5 | Decoder Block 4 | Decoder Block 3 | Decoder Block 2 | Decoder Block 1
---|---|---|---|---|---|---
TotalSeg | NAS | P3D | P3D | 2D | P3D | 2D
Pruned | P3D | P3D | 2D | Projection | 2D
ChestOrgan | NAS | 3D | P3D | P3D | 2D | 2D
Pruned | Projection | P3D | P3D | 2D | 2D
HNOrgan | NAS | 3D | P3D | 3D | P3D | P3D
Pruned | Projection | P3d | 3D | P3D | P3D
EsoOrgan | NAS | P3D | 2D | Projection | 2D | Projection
Pruned | Projection | Projection | Projection | Projection | Projection

Decoder Optimization – NAS Setting. For NAS, we divide each dataset into 1)
$60\%$ for network training, 2) $30\%$ for NAS training, and 3) $10\%$ for
validation and ablation evaluation. To train the learnable weight for
selecting the architecture of each decoding block, we first fix the
convolution kernel selection weight to $\frac{1}{C_{t}}$ for $40\%$ of the
total epochs, where $C_{t}$ denotes the number of classes of the $t^{th}$
task. Then we alternatively update the convolution kernel selection weight and
the decoder parameters for the rest epochs. The initial learning rate is set
to 0.01 for all tasks. The learning rate is decayed following the Polynomial
learning rate policy. After NAS training is complete, we follow the same
‘moreDA’ data augmentation scheme and retrain the searched decoding path from
scratch using the re-divided dataset in a 4:1 ‘training-validation’ ratio. The
searched decoding blocks for each task are shown in Table B.1.

Decoder Optimization – Pruning Setting. We perform a block-wise teacher-
student kd, aiming to further compress the decoder by replacing the searched
convolutional kernels with the projection kernels. The mean-square error loss
is used to match the feature map outputs of the teacher block to the student
block. To ease the optimization difficulty, we first distill the deeper
decoding blocks (lower resolution), then move to shallower blocks. Once the kd
training of the deeper block is saturated, we freeze the deeper student blocks
and move to the shallower ones. When conducting this block-wise KD, the
shallower convolutional block needs to choose if to receive output feature
maps from the deeper teacher blocks or those from the student block. Under
this setting, if the feature map difference from previous teacher and student
block is large, it would affect the subsequent feature response in the next
shallower block causing the degenerated segmentation. To conquer this, we
employ a simple yet effective approach: Once the distillation of the student
block is finished, we use a smaller learning rate (e.g., $0.1\times$) and
finetune the shallower teacher block using the deeper student block’s output
feature maps. We monitor the before-and-after performance drop. If the drop is
less than $1\%$ in terms of Dice score, we keep the deeper student block,
otherwise, we restore the teacher block. To speed up the training process,
block-wise side supervision is also used during training. The pruned decoder
for each task is demonstrated in Table B.1.

Body-part and Anomaly-aware Decoder Merging. To generate the final 143-class
organ segmentation output, we need to combine and merge the predictions from
all decoders. There are two major issues in this step. First, since each
training dataset/task often covers a specific body part, the task-specific
decoder might generate false positives in body parts that are not covered in
this specific dataset (because that decoder never sees other anatomic
regions). We propose a straightforward yet effective solution to reduce these
false positives due to the body part coverage: for each decoder, we pre-
compute the body part coverage rate using all the data in this dataset/task.
In this way, for a specific decoder, voxels outside the covered body parts
would have a lower/zero weight, and we can use this weight to significantly
decrease the confidence score of the false positive segmentations out of the
covered body parts. Specifically, we first generate the body part map using an
automated body part regression algorithm [66]. Then, by overlapping the
bounding box of all labeled organs to the body part map, we compute the
volume-wise overlapping ratio between the bounding box and the body part map.
Then, a body part distribution map is generated for a patient, e.g., 80% in
the chest, and 20% in the abdomen. This calculation is repeated for all
patients in the dataset, and finally averaged to get a pre-computed body part
distribution map $\hat{Y}^{t}_{\beta}$. The detailed body part distributions
for the three in-house datasets are illustrated in Table B.2.

Table B.2: The body part distributions of the in-house datasets.

| Head | Chest | Abdomen | Pelvic
---|---|---|---|---
ChestOrgan | 8.2% | 89.4% | 2.4% | 0%
HNOrgan | 96.5% | 3.5% | 0% | 0%
EsoOrgan | 0% | 96.3% | 3.7% | 0%

The second issue is that some decoders might not see the patients with
abnormalities (e.g., tumors). Hence, the predictions may have false
segmentation in the anomaly region. To resolve this problem, we first
supplement the framework with an anomaly segmentation head. In our work, we
use the esophageal and lung tumor dataset to train this head as an
illustration. More abnormality datasets can be utilized, such as DeepLesion
[67]. Then, we exploit the tumor predictions $\hat{Y}^{\epsilon}$ to generate
an anomaly weighting map. The averaged tumor size $p^{\epsilon}$ is pre-
calculated using the annotated tumor mask and used as the standard deviation
of a Gaussian filter of zero-mean to further smooth the tumor prediction
$\hat{Y}^{\epsilon}$. Here, we assume that, at location $j$, the prediction
$\hat{Y}^{t}(j)$ of the $t^{th}$ decoder is less confident if
$\hat{Y}^{\epsilon}(j)$ is of high value.

To combine the predictions, we perform a voxel-wise selection by choosing the
most confident prediction from all decoding heads, considering both the body
part distribution map and the anomaly map. As the entropy function produces
the highest value when the input closes to 0.5 (the most uncertain
prediction), we could find the most confident prediction when the input closes
to 0 or 1. Eq. (3) in the main text is used to combine the body part map
$\hat{Y}^{t}_{\beta}$ and anomaly distribution map $\hat{Y}^{\epsilon}$. When
there is no tumor prediction $\hat{Y}^{\epsilon}(j)=0$ and the organ
prediction is within the decoder’s body part distribution range
$\hat{Y}^{t}_{\beta}(j)=1$, the output score is considered as confident and
sets $M^{t}(j)=0$. On the other hand, the other states are considered
uncertain and set $M^{t}(j)=0.5$. The confidence map is generated using Eq.
(4) of the main text. For a voxel at location $j$, using Eq. (5), we collect
the confidence values from all tasks whose foreground prediction is not in the
background. The final output class $\hat{\mathbf{Y}}(j)$ is determined using
the prediction $\hat{Y}^{t}(j)$, of which with the smallest $H^{t}(j)$.

##  Appendix C Additional Ablation Results and Analysis

Table C.1: Mean DSC (%, $\uparrow$) and forgetting rate (%, $\downarrow$) of
our method and other comparison methods in each dataset/step of two continual
segmentation orders. The last column ‘All Learned Classes’ lists the mean DSC
and average forgetting rate [8] over all learned organs/classes at each step.
The DSC in TotalSegmentator at step 1 of all comparison methods is the upper
bound 93.24%, while DSC for our method is slightly lower with 92.98% at step 1
due to the decoder compression/pruning. (Sub-figures in Fig.
LABEL:M-fig:step_all from left to right are corresponding to the numeric
results in column ‘All Learned Classes’, ‘TotalSeg’, ‘ChestOrgan’ and
‘HNOrgan’ of this table.)

|  | TotalSeg (103) | ChestOrgan (31) | HNOrgan (13) | EsoOrgan (1) | All Learned Classes
---|---|---|---|---|---|---
Methods | Step | DSC | Forget | DSC | Forget | DSC | Forget | DSC | Forget | DSC | Avg. Forget
upper bound |  | 93.24 | — | 78.45 | — | 84.35 | — | 87.15 | — | 89.02 | —
| Order A: TotalSeg $\rightarrow$ ChestOrgan $\rightarrow$ HNOrgan
$\rightarrow$ EsoOrgan
| 2 | 45.80 | 50.87 | 78.40 | — |  [HTML]D9D9D9 |  [HTML]D9D9D9 |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 50.56 | 50.87
| 3 | 11.68 | 87.48 | 25.66 | 67.27 | 84.22 | — |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 20.86 | 77.37
MiB [6] | 4 | 7.65 | 91.80 | 19.24 | 75.46 | 6.37 | 92.43 | 86.92 | — | 8.51 | 86.56
| 2 | 48.50 | 47.98 | 77.78 | — |  [HTML]D9D9D9 |  [HTML]D9D9D9 |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 54.40 | 47.98
| 3 | 13.68 | 85.33 | 28.74 | 63.04 | 84.21 | — |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 23.08 | 74.19
$\text{ILT}^{\dagger}$ [42] | 4 | 10.87 | 88.34 | 27.87 | 64.17 | 6.39 | 92.42 | 85.75 | — | 11.99 | 81.64
| 2 | 59.13 | 36.59 | 76.52 | — |  [HTML]D9D9D9 |  [HTML]D9D9D9 |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 62.40 | 36.59
| 3 | 39.46 | 57.68 | 49.19 | 35.72 | 83.17 | — |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 45.10 | 46.70
PLOP [13] | 4 | 37.30 | 59.99 | 51.74 | 32.38 | 25.38 | 69.48 | 82.90 | — | 39.01 | 53.95
| 2 | 52.57 | 43.62 | 78.48 | — |  [HTML]D9D9D9 |  [HTML]D9D9D9 |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 57.74 | 43.62
| 3 | 13.59 | 85.42 | 29.05 | 62.99 | 84.36 | — |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 22.86 | 74.21
LISMO [37] | 4 | 10.82 | 88.40 | 28.24 | 64.02 | 6.30 | 92.54 | 87.12 | — | 12.11 | 81.65
| 2 | 92.98 | 0.00 | 78.26 | — |  [HTML]D9D9D9 |  [HTML]D9D9D9 |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 88.27 | 0.00
| 3 | 92.98 | 0.00 | 78.26 | 0.00 | 83.97 | — |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 87.88 | 0.00
Ours | 4 | 92.98 | 0.00 | 78.26 | 0.00 | 83.97 | 0.00 | 86.94 | — | 87.87 | 0.00
| Order B: TotalSeg $\rightarrow$ HNOrgan $\rightarrow$ ChestOrgan
$\rightarrow$ EsoOrgan
| 2 | 21.96 | 76.45 |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 84.49 | — |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 29.42 | 76.45
| 3 | 10.72 | 88.50 | 78.46 | — | 6.38 | 92.45 |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 23.85 | 90.48
MiB [6] | 4 | 10.35 | 88.90 | 65.63 | 16.35 | 6.29 | 92.55 | 86.79 | — | 20.20 | 65.94
| 2 | 21.62 | 76.81 |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 84.25 | — |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 29.09 | 76.81
| 3 | 14.10 | 84.88 | 78.02 | — | 8.48 | 89.93 |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 26.13 | 87.41
$\text{ILT}^{\dagger}$ [42] | 4 | 13.12 | 85.93 | 67.28 | 13.76 | 6.18 | 92.66 | 85.52 | — | 22.44 | 64.12
| 2 | 45.11 | 51.62 |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 83.59 | — |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 49.70 | 50.87
| 3 | 31.90 | 65.79 | 76.13 | — | 17.56 | 78.99 |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 38.99 | 72.39
PLOP [13] | 4 | 30.82 | 66.95 | 70.18 | 7.81 | 15.77 | 81.13 | 83.41 | — | 36.63 | 51.96
| 2 | 24.36 | 73.88 |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 84.35 | — |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 31.51 | 73.88
| 3 | 15.08 | 83.83 | 78.47 | — | 7.85 | 90.69 |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 26.84 | 87.26
LISMO [37] | 4 | 14.04 | 84.94 | 67.19 | 14.37 | 6.15 | 92.71 | 86.87 | — | 23.09 | 64.01
| 2 | 92.98 | 0.00 |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 83.97 | — |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 91.27 | 0.00
| 3 | 92.98 | 0.00 | 78.26 | — | 83.97 | 0.00 |  [HTML]D9D9D9 |  [HTML]D9D9D9 | 87.88 | 0.00
Ours | 4 | 92.98 | 0.00 | 78.26 | 0.00 | 83.97 | 0.00 | 86.94 | — | 87.87 | 0.00

Figure C.1: The mean DSCs over each dataset at each step of Order A (solid
line) and Order B (dashed line).

In our main text, we have briefly demonstrated and discussed the final
segmentation performance and forgetting curve of our method and other
comparison methods at each continual learning step in Table LABEL:M-tab:main
and Fig LABEL:M-fig:step_all (Section 4.3). Here, we show the detailed numeric
results of Fig LABEL:M-fig:step_all in Fig C.1 and Table C.1 and provide more
in-depth discussion on results achieved by our model and other comparison
methods.

First, the mean DSC and the forgetting rate in each dataset/step of two CSS
orders are detailed in Table C.1. The mean DSC and average forgetting rate
over all 143 organs at the last step is also shown. Besides the observations
discussed in the main text, several additional findings can be noticed. First,
the slight decrease of DSC of our method in the sub-figures of Fig
LABEL:M-fig:step_all (mean DSC over all learned organs at the current step) is
not due to our model forgetting, but simply because the achieved DSC values
are lower in new datasets/steps. E.g., the upper bound mean DSC in ChestOrgan
dataset is only 78.45% (as compared to 93.24% in TotalSegmentator dataset). As
shown the last three rows of two CSS order in Table C.1, our method completely
avoids forgetting of old knowledge when continually learning new dataset/task
because of our proposed framework (frozen general encoder, light-weighted
decoders, and body-part and anomaly-aware merging). In contrast, other
distillation-based CSS methods all experienced severe forgetting with more
than 50% forgetting rate at the last step.

Second, regarding the step-wise results of comparison methods. It is observed
that methods based on the output-level knowledge distillation and MiB losses
(MiB [6], ILT [42] and LISMO [37]) suffer from catastrophic forgetting after
the last step (overall mean DSC $<25\%$ and forgetting rate $>64\%$). In
contrast, although PLOP [13] also has a large forgetting rate (about 50%), the
overall performance is noticeably better as compared to the other three
methods. For instance, the overall mean DSC is 39.01% for PLOP in CSS order A,
which is at least 3 times higher than that for MiB, ILT and LISMO. The
increased ability to keep old knowledge in PLOP might come from the applied
entropy-based pseudo-labeling and the knowledge distillation on intermediate
features in both encoder and decoder.

Third, regarding two continual segmentation orders, the main difference is the
learning order of ChestOrgan and HNOrgan: order A first learns ChestOrgan in
step-2, then HNOrgan in step-3, while order B exchanges the dataset in step-2
and 3. It is observed in Table C.1 and Fig C.1 that the comparison methods
forgetting rate at learning step-2 for the TotalSegmentator dataset is higher
in order B than that in order A. E.g., TotalSegmentator DSC of MiB [6] at
step-2 is decreased from 93.24% $\rightarrow$ 21.96% in order B vs. from
93.24% $\rightarrow$ 45.80% in order A. Notice that order B learns the HNOrgan
right after TotalSegmentator at step-2, and HNOrgan contains CT images only
focusing in the head and neck region, where TotalSegmentator has organs mostly
labeled in the chest, abdomen, and pelvic regions. As a result, MiB can no
longer see the chest, abdomen, and pelvic regions at step-2, which causes
catastrophic forgetting in these body parts resulting in a significant
performance drop. Instead, order A learns the ChestOrgan at step-2, and
ChestOrgan covers all the chest and neck regions as well as most parts of the
abdomen. Hence, MiB is still able to rehearse some old knowledge over these
overlapping body parts so that the forgetting rate is reduced as compared to
that in order B. A similar trend can be found in the forgetting curves of
ChestOrgan and HNOrgan. These findings show that for the multi-organ continual
segmentation, the forgetting rate of other comparison methods is closely
related to the overlapping range of body parts in each dataset/step. In
contrast, our proposed architectural-based method is learning-order and body-
part invariant, which facilitates the model deployment in clinical practice.

Last, we evaluate the impact of Alternative General Encoders. We recommended
starting with TotalSegentator as it covers most body parts for comprehensive
feature extraction. Alternatively, other datasets can be used as the starting
dataset to train General Encoder. We trained the General Encoder using the
ChestOrgan dataset. A tolerable performance drop ($<$1% Dice) is observed in
the CSS Order A final results. The assumed reason is that the torso region
includes diverse anatomies and covers most of the body parts, and hence
exhibits similar performance as the one trained using the TotalSegmentator
dataset. Yet, when using the HNOrgan dataset to train the General Encoder, we
notice a markedly $3\%$ Dice drop in the final results.

##  Appendix D Detailed Results of Individual Organs

We provide detailed organ segmentation results of our final model as well as
the upper bound nnUNet performance trained and evaluated on TotalSegmentator,
ChestOrgan, HNOrgan datasets (shown in Table D.2, D.3, D.4). The final
performance on the EsoOrgan dataet has been reported in the Table 1 of the
main text. For organs in TotalSegmentator, due to the large amount of organs,
we choose to group the 103 whole body organs into eight anatomical groups
(Table D.1) and report the average scores of each group (see Table D.2). As
shown in those tables, our final model performs closely to the upper bound
accuracy when training separate nnUNet models on each dataset. There are no
organs experiencing a large performance drop. The overall slightly drop in DSC
and increasing in HD95/ASD is because of the decoder pruning process.

Table D.1: TotalSegmentator [63] label list of each organ group.

TotalSeg Organ Group | TotalSeg Organ Labels
---|---
Main Chest Organs | 13—17, 42—48
Cardiovascular Vessels | 7, 8, 9, 49, 51, 52, 53, 54
Excretory Organs | 2, 3, 55, 57, 104
Main Abdomen Organs | 1, 4, 5, 6, 10, 11, 12, 56
Head | 50 (‘face’-93 is exluded)
Vertebraes | 18—41
Ribs | 58—81
Other Bones | 82—92
Muscles | 94—103

Table D.2: Mean DSC (%, $\uparrow$), HD95 (mm, $\downarrow$) and ASD (mm,
$\downarrow$) of 8 anatomical organ groups in TotalSeg (total 103 full body
organs) of upper bound model and our final model.

TotalSeg Organ Group | Upper Bound | Ours
---|---|---
DSC | HD95 | ASD | DSC | HD95 | ASD
Chest Main Organ | 96.66 | 1.71 | 0.35 | 95.62 | 2.54 | 0.43
Cardiovascular Vessels | 91.75 | 2.33 | 0.55 | 91.97 | 2.96 | 0.74
Excretory System | 93.28 | 4.24 | 1.22 | 93.53 | 4.45 | 1.28
Abdomen Main Organ | 89.44 | 3.45 | 0.80 | 91.05 | 3.96 | 0.88
Head | 94.51 | 2.44 | 0.62 | 94.61 | 3.02 | 0.69
Vertebraes | 92.94 | 2.01 | 0.48 | 92.65 | 2.92 | 0.68
Ribs | 91.54 | 4.24 | 1.06 | 91.49 | 5.15 | 1.20
Other Bones | 95.02 | 6.94 | 2.08 | 93.14 | 7.85 | 2.27
Muscles | 96.09 | 2.17 | 0.38 | 95.92 | 2.98 | 0.54

Table D.3: Mean and standard deviation of DSC (%, $\uparrow$), HD95 (mm,
$\downarrow$) and ASD (mm, $\downarrow$) of each ChestOrgan organ (total 31
chest organs) of upper bound model and our final model. _L and _R refer to the
left and right.

ChestOrgan | Upper Bound | Ours
---|---|---
DSC | HD95 | ASD | DSC | HD95 | ASD
Esophagus | 85.41$\pm$4.12 | 4.93$\pm$3.32 | 0.68$\pm$0.42 | 85.22$\pm$4.30 | 5.78$\pm$2.81 | 0.67$\pm$0.34
Sternum | 90.30$\pm$3.27 | 5.07$\pm$5.00 | 1.16$\pm$1.07 | 90.26$\pm$3.64 | 5.83$\pm$5.33 | 1.18$\pm$2.07
Thyroid_L | 84.13$\pm$4.71 | 2.87$\pm$1.36 | 0.66$\pm$0.36 | 84.10$\pm$3.98 | 3.54$\pm$1.23 | 0.73$\pm$0.33
Thyroid_R | 82.79$\pm$6.28 | 3.39$\pm$2.34 | 0.79$\pm$0.41 | 82.75$\pm$5.32 | 4.18$\pm$1.89 | 0.88$\pm$0.38
Trachea | 93.74$\pm$2.19 | 4.74$\pm$3.45 | 1.09$\pm$0.88 | 93.68$\pm$1.83 | 5.45$\pm$2.76 | 1.03$\pm$0.69
Bronchus_L | 86.53$\pm$4.39 | 4.84$\pm$2.55 | 0.48$\pm$0.33 | 86.34$\pm$5.15 | 5.45$\pm$2.29 | 0.44$\pm$0.29
Bronchus_R | 75.86$\pm$13.79 | 9.18$\pm$7.51 | 2.35$\pm$2.87 | 75.88$\pm$13.41 | 10.42$\pm$7.67 | 2.41$\pm$2.70
Anterior cervical muscle | 69.23$\pm$8.05 | 6.31$\pm$5.67 | 1.34$\pm$1.22 | 69.02$\pm$6.79 | 7.08$\pm$4.54 | 1.39$\pm$1.20
Scalenus muscle | 74.26$\pm$4.42 | 5.83$\pm$3.79 | 0.83$\pm$0.38 | 74.24$\pm$5.18 | 6.98$\pm$3.86 | 0.79$\pm$0.38
Scalenus anterior muscle | 77.89$\pm$6.34 | 4.37$\pm$3.46 | 0.95$\pm$1.03 | 77.82$\pm$6.89 | 5.04$\pm$3.17 | 0.83$\pm$0.83
sternocleidomastoid muscle | 82.17$\pm$3.92 | 4.01$\pm$2.89 | 1.16$\pm$1.11 | 82.10$\pm$3.35 | 4.89$\pm$3.23 | 1.06$\pm$0.81
common carotid artery_L | 78.09$\pm$7.65 | 9.33$\pm$17.74 | 2.17$\pm$4.33 | 77.97$\pm$8.21 | 9.98$\pm$16.47 | 2.28$\pm$4.33
common carotid artery_R | 73.92$\pm$11.08 | 12.43$\pm$16.29 | 2.84$\pm$4.72 | 73.77$\pm$10.63 | 13.37$\pm$17.83 | 2.90$\pm$4.81
Pulmonary artery | 90.11$\pm$3.64 | 6.64$\pm$3.09 | 1.28$\pm$0.67 | 89.93$\pm$5.46 | 7.11$\pm$3.97 | 1.13$\pm$0.77
Subclavian artery_L | 71.79$\pm$9.26 | 20.26$\pm$20.29 | 2.37$\pm$3.40 | 71.78$\pm$9.50 | 23.15$\pm$22.32 | 2.28$\pm$3.46
Subclavian artery_R | 80.03$\pm$5.87 | 11.93$\pm$13.24 | 2.04$\pm$2.11 | 79.97$\pm$6.07 | 12.14$\pm$13.72 | 2.19$\pm$2.90
Vertebral artery_L | 47.72$\pm$19.66 | 20.25$\pm$21.22 | 6.65$\pm$11.83 | 46.62$\pm$20.32 | 21.27$\pm$20.94 | 6.51$\pm$11.73
Vertebral artery_R | 45.28$\pm$18.38 | 19.90$\pm$21.07 | 6.33$\pm$11.35 | 42.66$\pm$17.08 | 20.81$\pm$17.43 | 6.40$\pm$11.12
Ascending aorta | 93.08$\pm$2.54 | 5.41$\pm$2.40 | 1.39$\pm$0.77 | 93.05$\pm$2.17 | 6.25$\pm$2.50 | 1.34$\pm$0.97
Descending aorta | 97.13$\pm$1.75 | 3.21$\pm$2.21 | 0.72$\pm$0.37 | 97.05$\pm$0.94 | 3.73$\pm$1.98 | 0.67$\pm$0.35
Aorta arch | 92.12$\pm$9.08 | 4.23$\pm$2.72 | 1.25$\pm$1.42 | 92.11$\pm$7.53 | 4.93$\pm$2.80 | 1.27$\pm$1.42
Azygos vein | 73.29$\pm$11.53 | 8.99$\pm$14.07 | 1.68$\pm$3.58 | 73.25$\pm$10.02 | 9.79$\pm$14.16 | 1.65$\pm$3.53
brachiocephalic vein_L | 85.80$\pm$5.57 | 3.27$\pm$2.35 | 0.35$\pm$0.29 | 85.73$\pm$5.00 | 4.13$\pm$2.76 | 0.35$\pm$0.26
brachiocephalic vein_R | 85.83$\pm$5.07 | 4.71$\pm$1.88 | 0.87$\pm$0.54 | 85.82$\pm$5.70 | 5.30$\pm$2.37 | 0.89$\pm$0.57
internal jugular vein_L | 74.66$\pm$14.57 | 12.77$\pm$16.31 | 2.88$\pm$4.51 | 74.63$\pm$14.41 | 14.19$\pm$14.63 | 2.90$\pm$3.81
internal jugular vein_R | 78.28$\pm$8.73 | 12.03$\pm$16.86 | 2.95$\pm$3.68 | 78.24$\pm$8.19 | 15.86$\pm$13.63 | 3.02$\pm$3.52
Pulmonary vein | 70.62$\pm$8.24 | 7.04$\pm$2.80 | 1.53$\pm$0.67 | 70.57$\pm$7.83 | 7.81$\pm$2.91 | 1.49$\pm$0.76
Subclavian vein_L | 63.32$\pm$14.72 | 9.64$\pm$6.69 | 1.99$\pm$1.90 | 63.32$\pm$15.21 | 10.14$\pm$7.88 | 2.00$\pm$1.71
Subclavian vein_R | 60.59$\pm$12.21 | 11.62$\pm$8.13 | 2.75$\pm$2.35 | 60.60$\pm$12.92 | 13.89$\pm$10.63 | 2.83$\pm$2.20
Inferior vena cava | 82.51$\pm$6.23 | 7.95$\pm$5.58 | 1.69$\pm$1.13 | 82.45$\pm$5.82 | 9.01$\pm$5.56 | 1.71$\pm$1.25
Superior vena cava | 85.38$\pm$3.83 | 5.85$\pm$3.55 | 1.36$\pm$0.88 | 85.29$\pm$3.85 | 6.60$\pm$3.45 | 1.33$\pm$0.75

Table D.4: Mean and standard deviation of DSC (%, $\uparrow$), HD95 (mm,
$\downarrow$) and ASD (mm, $\downarrow$) of each HNOrgan organ (total 13 head-
neck organs) of upper bound model and our final model. _L and _R refer to the
left and right.

HNOrgan | Upper Bound | Ours
---|---|---
DSC | HD95 | ASD | DSC | HD95 | ASD
BrainStem | 91.42$\pm$2.47 | 2.93$\pm$1.15 | 0.76$\pm$0.32 | 90.89$\pm$2.38 | 2.65$\pm$1.37 | 0.74$\pm$0.36
Eye_L | 92.32$\pm$1.73 | 1.57$\pm$0.60 | 0.36$\pm$0.11 | 92.25$\pm$1.69 | 1.50$\pm$0.48 | 0.33$\pm$0.12
Eye_R | 91.99$\pm$1.49 | 1.68$\pm$0.54 | 0.40$\pm$0.10 | 91.95$\pm$1.19 | 1.63$\pm$0.92 | 0.49$\pm$0.21
Lens_L | 81.49$\pm$10.64 | 1.64$\pm$0.90 | 0.53$\pm$0.47 | 80.33$\pm$8.81 | 1.55$\pm$0.89 | 0.47$\pm$0.50
Lens_R | 84.16$\pm$8.35 | 1.39$\pm$0.70 | 0.40$\pm$0.30 | 82.46$\pm$6.33 | 1.23$\pm$1.12 | 0.36$\pm$0.36
Optic Chiasm | 67.04$\pm$13.34 | 3.73$\pm$1.66 | 1.03$\pm$0.65 | 66.60$\pm$13.29 | 3.59$\pm$1.91 | 0.96$\pm$0.71
Optic Nerve_L | 74.34$\pm$6.77 | 3.05$\pm$2.75 | 0.56$\pm$0.27 | 74.72$\pm$6.93 | 2.96$\pm$3.27 | 0.61$\pm$0.51
Optic Nerve_R | 75.15$\pm$6.47 | 2.64$\pm$0.80 | 0.51$\pm$0.30 | 73.64$\pm$7.07 | 2.49$\pm$1.18 | 0.56$\pm$0.35
Parotid_L | 91.32$\pm$3.01 | 3.13$\pm$1.72 | 0.75$\pm$0.33 | 91.09$\pm$3.08 | 2.90$\pm$2.10 | 0.76$\pm$0.32
Parotid_R | 90.93$\pm$2.87 | 3.22$\pm$1.94 | 0.86$\pm$0.46 | 90.94$\pm$2.97 | 3.03$\pm$2.10 | 0.86$\pm$0.67
TMJ_L | 81.55$\pm$9.42 | 2.10$\pm$1.06 | 0.56$\pm$0.47 | 82.14$\pm$9.33 | 1.87$\pm$1.19 | 0.52$\pm$0.65
TMJ_R | 84.81$\pm$8.57 | 1.85$\pm$1.06 | 0.43$\pm$0.39 | 84.70$\pm$9.28 | 1.69$\pm$0.93 | 0.42$\pm$0.50
Spinal cord | 90.01$\pm$2.35 | 2.03$\pm$0.61 | 0.65$\pm$0.22 | 89.93$\pm$2.19 | 1.83$\pm$0.81 | 0.64$\pm$0.31

##  Appendix E Re-implementation of Comparison Methods

For all comparison methods, we start with the same pretrained nnUNet model on
TotalSegmentator dataset, which has been trained using 3D nnUNet setting for
8000 epochs, with 250 iterations per epoch and initial learning rate 0.01.
After that, the model is finetuned sequentially on continual segmentation
tasks (ChestOrgan, HNOrgan and EsoOrgan), where each dataset are finetuned for
500 epochs, with 250 iterations per epoch and initial learning rate 0.005. All
the other nnUNet settings, such as data augmentation, remain the same as our
implementation. Moverover, since our segmentation datasets/tasks are 3D CT
scans (different from the previous continual segmentation works in natural
images), adjustments to these comparison methods are required (extending 2D
methods to 3D), as well as transferring their implementations to the nnUNet
framework. We describe the detailed re-implementation of previous methods,
especially our modifications, in the following subsections.

MiB. MiB [6] proposes two marginal losses, or unbiased losses to solve the
background shift issue in continual segmentation in their original paper:
unbiased cross-entropy (UNCE) loss, which merges the probabilities of old
classes to the background label, and unbiased knowledge distillation (UNKD)
loss, which merges the probabilities of all new classes (belonging unseen
classes of the old model) to the background label. Notice that, the original
unbiased loss assumes that new classes from the current dataset are completely
disjoint with all the old classes, however, this assumption is not holding in
our datasets. E.g., TotalSegmentator and ChestOrgan contain four overlapping
organs: inferior vena cava, trachea, esophagus and pulmonary artery.
Therefore, in order to re-implement MiB losses in the nnUNet framework and
make them compatibility with our datasets, we slightly modifies and
generalizes both unbiased losses to handle overlapping labels in the continual
learning setting. The modified UNCE loss is as follows:

$\ell^{\theta^{t}}_{ce}(x,y)=-\frac{1}{\lvert\mathcal{I}\rvert}\sum_{i\in\mathcal{I}}\log\tilde{q}_{x}^{t}(i,y_{i})$
(7)

where:

$\tilde{q}_{x}^{t}(i,c)=\begin{cases}q_{x}^{t}(i,c)&\text{if $c\neq b$}\\\
\sum_{k\in{\mathcal{Y}}^{t-1}{\color[rgb]{1,0,0}\setminus{\mathcal{C}}^{t}_{p}}}q_{x}^{t}(i,k)&\text{if
$c=b$}\end{cases}$ (8)

Here, same notations referred to the original paper is used, except
${\mathcal{C}}^{t}_{p}={\mathcal{Y}}^{t-1}\cup{\mathcal{C}}^{t}-b$, which
indicates the overlapping classes (excluding background label) between current
dataset ${\mathcal{C}}^{t}$ and all the previous classes ${\mathcal{Y}}^{t-1}$
at the learning step $t$. When calculating UNCE loss, we merge all the old
labels to the background except the overlapping classes.

Similarly, we adapt UNKD loss as:

$\ell^{\theta^{t}}_{kd}(x,y)=-\frac{1}{\lvert\mathcal{I}\rvert}\sum_{i\in\mathcal{I}}\sum_{c\in{\mathcal{Y}}^{t-1}{\color[rgb]{1,0,0}\setminus{\mathcal{C}}^{t}_{p}}}q_{x}^{t-1}(i,c)\log\hat{q}_{x}^{t}(i,y_{i})$
(9)

where:

$\hat{q}_{x}^{t}(i,c)=\begin{cases}q_{x}^{t}(i,c)&\text{if $c\neq b$}\\\
\sum_{k\in{\mathcal{C}}^{t}}q_{x}^{t}(i,k)&\text{if $c=b$}\end{cases}$ (10)

In the above formula, overlapping organs from the old class set are excluded
so that the knowledge distillation works on the real old classes that cannot
be learned from the current dataset.

Using two modified losses, we always train the model with the latest labels
and ignore the previously learned overlapping labels when overlapping organs
occur. Thus, overlapping labels are trained directly using the cross-entropy
loss and merged to the background in the knowledge distillation loss. In
addition, we use the same hyperparameters as the MiB setting: the weight of
UNKD loss are set as 10 with balanced classifier initialization strategy.

ILT. ILT [42] originally first proposes the continual semantic segmentation
(CSS) protocol and provides a naive solution using an output-level knowledge
distillation on the old classes (${\mathcal{L}}_{D}^{\prime}$) and a feature-
level knowledge distillation on the intermediate features from encoder
(${\mathcal{L}}_{D}^{\prime\prime}$). This method leads to inferior
performance and experienced severe forgetting as compared to MiB and other CSS
methods on multiple natural image benchmarks [13, 43]. In order to improve ILT
performance on our datasets/tasks, we modifies the original ILT setting and
losses as follows: (1) ILT uses a frozen encoder setting ($E_{F}$) together
with ${\mathcal{L}}_{D}^{\prime}$, which is similar to our general encoder
method, therefore, we re-implement ILT using this frozen encoder setting, as
mentioned in the main paper; (2) since original ILT losses do not alleviate
the background shift issue and have a large bias towards new classes
(experiencing severe forgetting even with the frozen encoder), we additionally
apply the MiB loss (Eq.8,10) to reinforce the decoder to preserve more old
knowledge. In short, our re-implemented ILT can be treated as a frozen encoder
version of MiB ($E_{F}+{\mathcal{L}}_{\text{mib}}$). Although leading to an
improved performance as compared to the original ILT, this frozen encoder ILT
version still has obvious knowledge forgetting as shown in Table 1 of the main
text. This indicates that the frozen encoder with unbiased output-level
knowledge distillation is not sufficient to preserve the old knowledge in CSS.
In contrast, our proposed framework (general encoder + light-weighted decoder)
can performance at the accuracy for the first time with real non-forgetting in
CSS.

PLOP. PLOP [13] is originally implemented for 2D images, especially its multi-
scale local distillation loss based on local POD. Local POD is a multi-scale
feature pooling strategy consisting of computing width and height-pooled
slices on multi-scale regions, which aims to better retain both global and
local spatial knowledge from the old model. However, since our data are all 3D
CT scans with an additional depth dimension, we specifically extend the local
POD to higher dimensions when re-implementing the method. Two pooling
strategies can be adopted for the 3D cases: (1) pooling 3D feature map along
each single dimension and extracting three 2D projections along each axis:

$\displaystyle\begin{split}\Phi({\mathbf{x}})=&\left(\frac{1}{H}\sum_{h=1}^{H}{\mathbf{x}}\left[h,:,:,:\right]\middle\|\frac{1}{W}\sum_{w=1}^{W}{\mathbf{x}}\left[:,w,:,:\right]\right.\\\
&\left.\middle\|\frac{1}{D}\sum_{d=1}^{D}{\mathbf{x}}\left[:,:,d,:\right]\right)\in{\mathcal{R}}^{(WD+HD+HW)\times
C}\end{split}$ (11)

where notations follow the original PLOP paper. This pooling method can
preserve enough spatial information meanwhile providing some level of
plasticity to the model. (2) Pooling 3D feature map on two dimensions and only
extract 1D projection along the remaining axis:

$\displaystyle\begin{split}\Phi({\mathbf{x}})=&\left(\frac{1}{HW}\sum_{h=1}^{H}\sum_{w=1}^{W}{\mathbf{x}}\left[h,w,:,:\right]\middle\|\frac{1}{WD}\sum_{w=1}^{W}\sum_{d=1}^{D}{\mathbf{x}}\left[:,w,d,:\right]\right.\\\
&\left.\middle\|\frac{1}{HD}\sum_{h=1}^{H}\sum_{d=1}^{D}{\mathbf{x}}\left[h,:,d,:\right]\right)\in{\mathcal{R}}^{(H+W+D)\times
C}\end{split}$ (12)

This pooling strategy has similar feature shape, however, when pooling on two
axes together, most of the spatial information are lost and POD loss cannot
retain the old knowledge. After comparing the performance using two
strategies, we select the former one, which better handles the trade-off
between model rigidity and plasticity.

For hyperparameters, the original paper uses the pod weighting factor of 0.01,
which is too large for the 3D pooling case. Because the L2 norm of 3D pooled
features is more than 10 times larger than that of 2D pooled features. In our
experiments, we set this pod factor to 0.001. Other hyperparameters are
consistent with those used in the original paper.

LISMO. The original LISMO [37] is designed based on nnUNet framework, so we
are able to directly re-implement this method. We would like to mention
several important differences between our datasets and those used in LISMO. In
LISMO [37], it has a slightly improved result than MiB when segmenting five
large abdominal organs in their experiment (using 3D low resolution of
nnUNet). Under this setup, all five abdominal organs could be seen in each 3D
training patch most of the time, which could frequently reinforce and
rehearsal the model’s ability on unseen organs in the current dataset through
their memory module and prototype matching loss. However, this is not the case
in our experiments, since many old organs are no longer able to observe in the
new dataset due to non-overlapping body parts. E.g. abdominal organs cannot
appear in the HNOrgan dataset. Moreover, the high resolution nnUNet version is
used to meet the high segmentation accuracy required in practice and there are
over 100 target organs spreading among the whole body range, so our patch size
is impossible to cover most organs within each patch. Under this situation,
the prototype matching loss is not able to compute on non-existing organs and
the contrastive loss is not sufficient to constraint the feature distributions
of these organs, which results in severe forgetting for the unobserved organs
in our experiment.