# CoinSeg: Contrast Inter- and Intra- Class Representations
for Incremental Segmentation

Zekang Zhang1, Guangyu Gao1, Jianbo Jiao2, Chi Harold Liu1, Yunchao Wei3,4

1 School of Computer Science, Beijing Institute of Technology
2 School of Computer Science, University of Birmingham
3 WEI Lab, Institute of Information Science, Beijing Jiaotong University
4 Beijing Key Laboratory of Advanced Information Science and Network
<EMAIL_ADDRESS>
Work done during an intern at WEI Lab of Beijing Jiaotong
University.Corresponding author<EMAIL_ADDRESS>
###### Abstract

Class incremental semantic segmentation aims to strike a balance between the
model’s stability and plasticity by maintaining old knowledge while adapting
to new concepts. However, most state-of-the-art methods use the freeze
strategy for stability, which compromises the model’s plasticity. In contrast,
releasing parameter training for plasticity could lead to the best performance
for all categories, but this requires discriminative feature representation.
Therefore, we prioritize the model’s plasticity and propose the Contrast
inter- and intra-class representations for Incremental Segmentation (CoinSeg),
which pursues discriminative representations for flexible parameter tuning.
Inspired by the Gaussian mixture model that samples from a mixture of Gaussian
distributions, CoinSeg emphasizes intra-class diversity with multiple
contrastive representation centroids. Specifically, we use mask proposals to
identify regions with strong objectness that are likely to be diverse
instances/centroids of a category. These mask proposals are then used for
contrastive representations to reinforce intra-class diversity. Meanwhile, to
avoid bias from intra-class diversity, we also apply category-level pseudo-
labels to enhance category-level consistency and inter-category diversity.
Additionally, CoinSeg ensures the model’s stability and alleviates forgetting
through a specific flexible tuning strategy. We validate CoinSeg on Pascal VOC
2012 and ADE20K datasets with multiple incremental scenarios and achieve
superior results compared to previous state-of-the-art methods, especially in
more challenging and realistic long-term scenarios. Code is available at
https://github.com/zkzhang98/CoinSeg.

##  1 Introduction

Figure 1: Comparison of Simple Prototype Modeling and Multiple Distribution
Modeling. (a): Simple prototype modeling to represent a category with global
average pooling; (b): Multiple Gussian distribution modeling to represent a
category with its regional objectness, which is discovered with the guidance
of mask proposals.

In recent years, deep learning based methods have achieved satisfactory
performance on various recognition tasks, with the assumption of fixed or
stable data distribution [17]. However, real-world data is typically a
continuous stream with an unstable distribution, making it difficult for
models to retain old knowledge while acquiring new concepts, known as
catastrophic forgetting [3, 24, 30]. To tackle this problem, incremental
learning is proposed to adapt to changing data streams for new concepts, but
also avoid forgetting old knowledge, especially for the classification task,
_i.e_., Class-Incremental Learning (CIL) [16, 18, 46].

Class Incremental Semantic Segmentation(CISS) aims to assign an image with the
pixel-wise label of the CIL setting. In semantic segmentation tasks involving
dense predictions, the problem of catastrophic forgetting typically becomes
more challenging. Most recent works [4, 5, 12] have struggled to alleviate
this problem, and the freeze strategy [5] (_i.e_., freezing most of the
parameters during all the incremental steps after learning base classes) was
shown to be the most efficient way in the state-of-the-art CISS methods [5,
49].

However, while the freeze strategy effectively alleviates catastrophic
forgetting, it is a compromise for the model’s plasticity, meaning the model
becomes hard to adapt to new classes. Furthermore, when considering the
lifelong learning setting with an infinite number of novel classes, the
plasticity of the incremental learner will be crucial. Therefore, ideally, the
optimal solution should be fine-tuning all the parameters for new classes,
while catastrophic forgetting needs to be handled properly, rather than a
freeze strategy.

Thus, to address the above-mentioned issues in CISS, especially the
limitations of the freeze strategy, we prioritize the model’s plasticity by a
flexible parameters tuning strategy, and pursue the more discriminative
feature representation for the balance to stability. An intuitive idea for
discriminative representation is to compute prototypes (category centroids)
for each category and apply contrastive learning to improve the diversity
among categories [26, 44]. However, as mentioned in the Gaussian Mixture Model
(GMM) [33], the natural samples from the same category should come from the
mixture of multiple Gaussian distributions. Furthermore, several prior works
[28, 29, 33, 43] have also claimed that the representation of categories in
the feature space should be multiple activations, as shown in Fig. 1.

To this end, we propose the Contrast inter- and intra-class representations
for Incremental Segmentation (CoinSeg), by pursuing discriminative
representations. Although the idea of adapting contrastive learning to CISS is
intuitive, it is worth studying and critical to choose the appropriate areas,
to contrast inter- and intra-class representations for incremental
segmentation. Firstly, the CoinSeg emphasizes intra-class diversity with
multiple contrastive representation centroids. Specifically, we identify
regions with strong objectness using mask proposals, which are more likely to
be instances/centroids of a particular category, as shown in Fig. 1 (b). We
contrast these regional objectness to reinforce intra-class diversity and
robust representation of the model. In order to mitigate the potential bias
from intra-class diversity, we incorporated category-level pseudo-labels to
augment category-level consistency and inter-category diversity. Meanwhile, we
apply Swin Transformer [19] to better extract the local representation of the
samples. Additionally, CoinSeg ensures the model’s stability and alleviates
forgetting through a specific Flexible Tuning strategy. In this strategy, a
better balance between plasticity and stability is achieved by designing an
initial learning rate schedule and regularization terms.

Finally, the CoinSeg outperforms prior methods in multiple benchmarks,
especially in realistic and hard long-term scenarios VOC 10-1 and 2-2, where
our approaches show significant performance gains of 6.7% and 17.8%, comparing
with previous state-of-the-art, respectively.

##  2 Related work

###  2.1 Class Incremental Learning

Class incremental learning (CIL) is incremental learning focusing on defying
catastrophic forgetting in classification tasks. Replay-based approaches were
proposed to store a sampler of historical data [10, 20, 32, 34, 38], which can
be used for future training to prevent forgetting. The historical data can
also be obtained with web search engine [23] or generation models [27, 39,
41]. Another intuitive thought to tackle CIL task is based on parameter
isolation [2, 22, 35, 36, 37, 46]. Parameter isolation methods assign
dedicated model parameters for each specific task, while bringing the
continually increasing number of parameters with task increases.
Regularization-based methods, such as knowledge distillation [1, 15, 16, 51,
14] and restricting model training with networks trained at previous tasks
[18, 31, 47], are also effective in tackling the catastrophic forgetting
problem in incremental learning. These methods allow the model to transfer
knowledge from previous tasks to new tasks, which can help prevent forgetting.

###  2.2 Class Incremental Semantic Segmentation

Recently, there is a growing interest in the field of incremental learning for
semantic segmentation(_i.e_., Class incremental semantic segmentation, CISS),
and researchers are proposing various approaches to tackle CISS. Modeling-the-
Background (MiB) [4] first remodeling the background (dummy label) in the
ground truth, and designs a distillation-based framework in CISS. Douillard et
al. [12] proposed the approach of PLOP to define a pseudo label for CISS and
proposes a local distillation method as an extended constraint based on MiB.
The SSUL [5] first introduces replay-based approaches to the CISS task and
maintains a memory bank of historical samples for future model training.
Besides, the SSUL prevents the model from forgetting by freezing model
parameters. RCIL [45] proposes a dual-branch architecture, in which one is
freeze and the other is trainable, and introduces a channel-wise feature
distillation loss. MicroSeg [49] introduces mask proposals to CISS and further
clarifies image labels to tackle background shifts.

###  2.3 Vision Transformer

Transformers for computer vision (_i.e_., Vision Transformer) have attracted
more and more attention. ViT [11] is the first widely known vision
transformer, which transposes transformer directly to image classification
tasks, achieving comparable performances with CNN-based methods. Since then,
researchers have explored modifications and optimizations to ViT, proposing
numerous designs [9, 40] for Vision Transformers that differ from those used
in NLP. Swin Transformer [19] proposes shifted window-based self-attention,
bringing greater efficiency for vision transformer. Other works have explored
the use of transformers for image segmentation. Segformer [42] consists of
hierarchically structured transformer layers to deal with semantic
segmentation. Mask2Former [7] and MaskFormer [8] propose universal
transformer-based architectures for semantic, instance and panoptic
segmentation.

##  3 Method

###  3.1 Task Definition

We define the task of Class Incremental Semantic Segmentation (CISS) according
to the common definition in previous works [4, 5, 12, 26, 45]. CISS is
composed of a series of incremental learning steps, as $t=1,\dots,T$. Each
learning step has a sub-dataset $\bm{\mathcal{D}}^{t}$ and a corresponding
class set of samples $\bm{\mathcal{C}}^{t}$. For any pair
$(\bm{x}^{t},\bm{y}^{t})$ within $\bm{\mathcal{D}}^{t}$, $\bm{x}^{t}$ and
$\bm{y}^{t}$ denote the input image and its ground-truth mask, respectively.
We follow the definition of classes set learned in CISS as in MicroSeg [49]:
In each learning step of class incremental semantic segmentation, the current
classes set can be represented as the union of the classes to be learned,
denoted by $\bm{\mathcal{C}}^{t}$, and a special class $c_{u}$ denotes “areas
does not belong to current foreground classes”. From the perspective of each
current learning step, the class $c_{u}$ can be interpreted as the background
class, and its composition varies across different learning steps.

The goal of the CISS model $f_{t,\bm{\theta}}$ with parameters $\bm{\theta}$
at the $t_{th}$ learning step is to assign a probability to each class for
every pixel in $\bm{x}^{t}$. The CISS model $f_{t,\bm{\theta}}$ is a composite
of a feature extractor $g_{t,\bm{\theta}_{1}}$ and a classifier
$h_{t,\bm{\theta}_{2}}$, which classifies each category in the union of
$\bm{\mathcal{C}}^{1:t}={\textstyle\bigcup_{i=1}^{t}\bm{\mathcal{C}}^{i}}$ and
the unseen class $c_{u}$ (i.e., $\mathbb{C}^{t}=\bm{\mathcal{C}}^{1:t}\cup
c_{u}$). After the $t$-th learning step, the model also needs to provide a
prediction for all seen classes $\mathbb{C}^{t}$. The prediction of CISS with
model $f_{t,\bm{\theta}}$ can be expressed as
$\hat{\bm{y}}_{t}=\arg\max_{c\in\mathbb{C}^{t}}f_{t,\bm{\theta}}^{c}(\bm{x})$.

###  3.2 Contrast inter- and intra-class Representations

Figure 2: Illustration of the inter- & intra-class diversity. Only the
contrastive learning with class “bird” is shown for example.

Previous studies [5, 49] have highlighted the advantages of the freeze
strategy in CISS. However, while parameter freezing can be beneficial, it may
also restrict the model’s plasticity and block further exploration of CISS.
Unlike previous methods, our approach prioritizes model plasticity and allows
fine-tuning to address this concern. With the application of fine-tuning, it
is possible to train the model to obtain more robust and discriminative
representations. Thus, we designed two contrastive losses for enhancing the
ability of model representation learning to improve the model plasticity,
_i.e_., pseudo label-guided representation learning for inter-class diversity
and mask proposal-guided representation learning for intra-class diversity, as
depicted in Fig. 2.

####  3.2.1 Contrast inter-class representations

We apply pseudo-label-guided representation learning to enhance inter-class
diversity. The approach also highlights category-level consistency to avoid
bias from intra-class diversity, which introduces in Sec. 3.2.2.

##### Pseudo labels.

Due to the limitation of data acquisition in CISS, the current ground truth
$\bm{y}^{t}$ is only annotated with classes at the current learning step,
_i.e_., $\bm{\mathcal{C}}^{t}$. So we need to extend the available ground
truth $\bm{y}^{t}$ to include predictions from $f_{t-1}$, creating a more
informative label for contrastive learning. Specifically, we get the label
prediction $\bm{\hat{y}}^{t-1}\in\mathbb{R}^{|\bm{\mathcal{C}}^{1:t-1}\cup
c_{u}|\times H\times W}$ with the inference of $f_{t-1}(\bm{x})$ and the
confidence $\bm{{s}}^{t-1}\in\mathbb{R}^{H\times W}$ of prediction. Formally:

$\displaystyle\bm{\hat{y}}^{t-1}=\arg\max_{\bm{\mathcal{C}}^{1:t-1}\cup
c_{u}}f_{t-1}(\bm{x}),$ (1)
$\displaystyle\bm{{s}}^{t-1}=\max_{\bm{\mathcal{C}}^{1:t-1}\cup
c_{u}}\sigma(f_{t-1}(\bm{x})),$

where $\sigma(\cdot)$ denotes Sigmoid function. With the ground truth mask
$\bm{y}^{t}=\\{\bm{y}^{t}_{i}\\}$ in current step, we mix the supervision
label $\tilde{\bm{y}}^{t}_{i}$ of pixel $i$ according to the following rules:

$\begin{aligned} \tilde{\bm{y}}^{t}_{i}=\begin{cases}{\bm{y}}^{t}_{i}&\text{
where }{\bm{y}}^{t}_{i}\in\bm{\mathcal{C}}^{t}\text{ or
}\bm{y}^{t}_{i}=c_{u}\wedge\bm{{s}}^{t-1}_{i}<\tau\\\
\bm{\hat{y}}^{t-1}_{i}&\text{ where
}\bm{y}^{t}_{i}=c_{u}\wedge\bm{{s}}^{t-1}_{i}\geq\tau\\\
\end{cases},\end{aligned}$

(2)

where threshold $\tau=0.7$, ‘$\wedge$’ represents the co-taking of conditions.

##### Inter-class contrastive loss.

With the guidance of pseudo label, CoinSeg gets the prototypes of classes
(_i.e_., class centroids), and sets up contrastive loss to better represent
inter-class diversity.

Given the feature maps $\bm{M}^{t}=g_{t}(\bm{x})$ and binary masks
$\tilde{\bm{y}}^{t}\in\\{0,1\\}^{|\mathbb{C}^{t}|\times h\times w}$ from the
pseudo labels, CoinSeg applies masked average pooling (MAP) [48] to obtain
prototypes of each foreground class as $\bm{P}_{int}^{t}$ for contrast inter-
class representation:

$\bm{P}_{int}^{t}=MAP(\bm{M}^{t},\tilde{\bm{y}}^{t})=\frac{\sum_{i=1,j=1}^{h,w}(\tilde{\bm{y}}^{t,i,j}*{M}^{t,i,j})}{\sum_{i=1,j=1}^{h,w}\tilde{\bm{y}}^{t,i,j}}$.

(3)

We abbreviate the operation as
$\bm{P}_{int}^{t}=MAP(\bm{M}^{t},\tilde{\bm{y}}^{t})$. $\bm{P}_{int}^{t}$ is a
series of vectors as prototypes, _i.e_., discriminative representation of each
classes. At learning step $t>1$, CoinSeg adapts model $f_{t-1}$ from learning
step $t-1$ as guidance to train the current model $f_{t}$, and we note
$\bm{M}^{t-1}=g_{t-1}(\bm{x})$. $\bm{P}^{t-1}_{int}$ can also be obtained with
feature maps of previous step $\bm{M}^{t-1}$ through a similar operation.
CoinSeg assigns prototypes clustered from the same pseudo label as positive
pairs in contrastive learning, and prototypes from different labels as
negative pairs.

The distance of prototypes is measured with the inner product
$Dot(\cdot,\cdot)$, and the contrastive loss with the guidance of the pseudo
label can be expressed as:

$\begin{aligned}
\mathcal{L}_{int}=-\frac{1}{K}\sum_{i=1}^{K}\log\frac{\exp(Dot(\bm{P}^{t,i}_{int},\bm{P}^{t-1,i}_{int}))}{\sum_{j=1,j\neq
i}^{2K}\exp(Dot(\bm{P}^{t,i}_{int},\bm{P}^{*,j}_{int}))},\end{aligned}$

(4)

where $\bm{P}^{*}_{int}=\bm{P}^{t}_{int}\cup\bm{P}^{t-1}_{int}$,
$K=|\bm{P}^{t}_{int}|$ is the number of inter-class prototypes, thus
$2K=|\bm{P}^{t}_{int}|+|\bm{P}^{t-1}_{int}|$ .

Figure 3: The pipeline of contrastive learning.

Generally, as shown in Fig. 3, we abbreviate the contrastive loss by previous
operations with the guidance of mask $\tilde{\bm{y}}^{t}$ as:

$\mathcal{L}_{int}=CON(\tilde{\bm{y}}^{t},M^{t},M^{t-1}).$ (5)

####  3.2.2 Contrast intra-class representations

Inspired by the Gaussian mixture models, we focus on intra-class diversity
benefits to more robust representation learning. Thus we mine potential
regional objectness within categories, and emphasize intra-class diversity
through mask proposals-guided representation learning.

##### Mask proposals.

CoinSeg adapts a set of class-agnostic binary mask
$\bm{B}\in\\{0,1\\}^{N\times H\times W}$ as proposals (_i.e_., mask
proposals), where $N$ denotes the number of mask proposals. Following the
practice of [49], mask proposals are generated with Mask2Former [7]. Note each
pixel in an image belongs and only belongs to one of the mask proposals.

##### Intra-class contrastive loss.

Mask proposals discover regional objectness in images, which are likely to be
diverse instances or centroids of a category, and benefit the construction of
intra-class contrastive learning. For $\bm{M}^{t}=g_{t}(\bm{x})$, CoinSeg
obtains prototypes of each mask proposal by $\bm{P}^{t}_{itr}=MAP(\bm{M}^{t}$,
$\bm{B})$. Namely, $\bm{P}^{t}_{itr}$ is a series of mask proposal-based
prototypes with size of $N\times C$. $\bm{P}^{t-1}_{itr}$ can also be obtained
with the similar operation. To better characterize the intra-class diversity
by contrastive learning, CoinSeg assigns prototypes from the same mask
proposal as positive pairs, and prototypes from different mask proposals as
negative pairs. The contrastive loss with the guidance of mask proposals can
be expressed as:

$\begin{aligned} \mathcal{L}_{itr}&=CON(\bm{B},\bm{M}^{t},\bm{M}^{t-1})\\\
&=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp(Dot(\bm{P}^{t,i}_{itr},\bm{P}^{t-1,i}_{itr}))}{\sum_{j=1,j\neq
i}^{2N}\exp(Dot(\bm{P}^{t,i}_{itr},\bm{P}^{*,j}_{itr}))}.\end{aligned}$

(6)

####  3.2.3 Summary

As explained in Sec. 3.1, due to the incremental learning task, there is a
limitation in the acquisition of category labels in the ground truth, which
leads to the fact that it is difficult for the CISS model to distinguish
between categories. To this end, we enhance the model’s category
discrimination ability by emphasizing inter- and intra-class diversity.
Moreover, contrastive learning overcomes catastrophic forgetting by the design
of positive contrast pairs, _i.e_., constraining the corresponding prototypes
from feature map $\bm{M}^{t}$ and $\bm{M}^{t-1}$ to be consistent. This
knowledge distillation-like mechanism helps to alleviate forgetting and
improve performance in CISS.

Finally, the total loss for the learning of the Contrast inter- and intra-
class Representations is:

$\displaystyle\mathcal{L}_{ct}=\mathcal{L}_{int}+\mathcal{L}_{itr}.$ (7)

###  3.3 Flexible Tuning Strategy

As mentioned before, releasing parameter training for plasticity could lead to
the best performance, but more discriminative feature representation is
needed. Although we have designed the Contrast inter- and intra-class
Representations for the discriminative representation, some more specific
parameter tuning strategy is necessary to ensure stability (_i.e_., handling
catastrophic forgetting) as well. Therefore, we introduce the Flexible Tuning
(FT) strategy as that, which allows for training the model while mitigating
the effects of forgetting, achieving a balance between stability and
plasticity.

Figure 4: Comparison of Freeze and Flexible Tuning strategy for learning step
$t>1$. Best viewed in color.

Fig. 4 shows the comparison of the freeze strategy and flexible tuning
strategy. The freeze strategy involves keeping the parameters of the feature
extractor and classifier for historical classes fixed for learning step $t>1$.
In contrast, the flexible tuning strategy uses a lower learning rate and
regularization constraints to allow for more flexible adjustments of these
parameters.

##### Flexible initial learning rate.

The segmentation model $f_{\bm{\theta}}$ is represented with a feature
extractor $g_{\bm{\theta}_{1}}$ and a classifier $h_{\bm{\theta}_{2}}$, with
parameters set $\bm{\theta}_{1}$ and $\bm{\theta}_{2}$ accordingly. Unlike the
freeze strategy [5], which sets the learning rate to zero for both
$\bm{\theta}_{1}$ and $\bm{\theta}_{2}$ of historical classes, the FT strategy
applies a initial learning rate schedule of each learning step to these
parameters. Specifically, during the first learning step ($t=1$), the initial
learning rate $lr$ will be kept as original $lr_{0}$. For subsequent learning
steps ($t>1$), the initial learning rate is gradually reduced to better
preserve the knowledge of the historical categories. So the initial $lr$ of
step $t$ is expressed as:

$lr^{t}=\begin{cases}lr_{0}&\text{ if }t=1\\\ e^{-t}\lambda_{lr}\cdot
lr_{0}&\text{ if }t>1\end{cases},$ (8)

where $\lambda_{lr}$ is a hyper-parameter, and $e^{-t}$ refers to the
exponential decay of the learning rate.

##### Regularization constraints.

When the flexible learning schedule allows the model to adapt more to new
concepts, regularization constraints for mitigating forgetting become even
more crucial than before. To better alleviate forgetting and ensure stability
when the model adapts to new categories, CoinSeg applies some regularization
constraints. To be more specific, for a sample $\bm{x}$, CoinSeg extract
feature map $\bm{M}^{t-1}\in\mathbb{R}^{C\times H^{\prime}\times W^{\prime}}$
by the feature extractor $g_{t-1}$. _i.e_., $\bm{M}^{t-1}=g_{t-1}(\bm{x})$,
and $C$ denotes the number of channels of $\bm{M}_{t-1}$. Similarly,
$\bm{M}^{t}=g_{t}(\bm{x})$. Then, CoinSeg constrains the consistency of
$\bm{M}^{t}$ and $\bm{M}^{t-1}$ with the Mean Square Error (MSE) as:

$\displaystyle\mathcal{L}^{F}_{kd}$
$\displaystyle=MSE(\bm{M}^{t},\bm{M}^{t-1})$ (9)
$\displaystyle=\frac{1}{C}\frac{1}{H^{\prime}W^{\prime}}\sum_{j=1}^{C}\sum_{i=1}^{H^{\prime}W^{\prime}}(\bm{M}^{t}_{i,j}-\bm{M}^{t-1}_{i,j})^{2}.$

Furthermore, CoinSeg also adapts knowledge distillation of logits $z^{t}$ and
$z^{t-1}$. Logits is the output of the model, _i.e_., $z^{t}=f_{t}(x)$. Note
$z^{t}\in\mathbb{R}^{|\bm{\mathcal{C}}^{1:t}\cup c_{u}|\times H\times W}$,
where
$\bm{\mathcal{C}}^{1:t}={\textstyle\bigcup_{i=1}^{t}\bm{\mathcal{C}}^{i}}$ is
all seen categories of step $t$, including historical and present classes. As
for logits $z^{t-1}\in\mathbb{R}^{|\bm{\mathcal{C}}^{1:t-1}\cup c_{u}|\times
H\times W}$, while $\bm{\mathcal{C}}^{t}$ is not exist for previous model
$f_{t-1}$. Following the common practice [4, 12, 26, 45], CoinSeg adopts the
following approaches to remodeling the logits $\hat{z}^{t}$ for pixel $i$ with
class $c$ :

$\hat{z}^{t}_{i,c}=\begin{cases}{z}^{t}_{i,c}&\text{ if }c\neq c_{u}\\\
\sum_{j\in\bm{\mathcal{C}}^{t}}{z}^{t}_{i,j}&\text{ if }c=c_{u}\end{cases}.$
(10)

Then cross entropy loss $L^{z}_{kd}$ is adapted to distill logits:

$\begin{aligned} \mathcal{L}^{z}_{kd}&=CE(\hat{z}^{t},z^{t-1})\\\
&=-\frac{1}{|\mathbb{C}^{t-1}|}\frac{1}{H^{\prime}W^{\prime}}\sum_{j=1}^{|\mathbb{C}^{t-1}|}\sum_{i=1}^{H^{\prime}W^{\prime}}(\hat{z}^{t}_{i,j}\log
z^{t-1}_{i,j}),\end{aligned},$ (11)

where we assign $|\mathbb{C}^{t-1}|=|\bm{\mathcal{C}}^{1:t-1}\cup c_{u}|$.

In summary, the regularization constraints in the flexible tuning strategy
are:

$\begin{aligned}
\mathcal{L}_{reg}=\mathcal{L}^{F}_{kd}+\mathcal{L}^{z}_{kd}\end{aligned}.$
(12)

###  3.4 Objective Function

Following the practice of previous methods[49], CoinSeg adapt common Binary
Cross-Entropy (BCE) loss as supervised segmentation loss $\mathcal{L}_{BCE}$
with the augmented label $\tilde{\bm{y}}$, and the final objective function of
CoinSeg is:

$\mathcal{L}=\mathcal{L}_{BCE}+\lambda_{c}\cdot\mathcal{L}_{ct}+\lambda_{r}\cdot\mathcal{L}_{reg}.$
(13)

##  4 Experiment

###  4.1 Experimental Setups

##### Dataset.

We have evaluated our approach on datasets of Pascal VOC 2012 [13] and ADE20K
[50]. Pascal VOC 2012 contains 10,582 training images and 1449 validation
images, with a total of 20 foreground classes and one background class. ADE20K
contains 20,210 training images and 2,000 validation images with 100 thing
classes and 50 stuff classes.

##### Protocols.

Following the conventions of previous work [5, 12, 49], we mainly evaluate our
approach for CISS with overlapped experimental setup, which is more realistic
than the disjoint setting that was studied by [4, 26, 45]. For each benchmark
dataset, we examine our approach under multiple incremental scenarios. We
abbreviate each incremental scenario in the form of $X-Y$, where $X$ means the
initial number of base classes, and $Y$ refers to the incremental number of
classes in each step. For instance, the 10-1 scenario of the VOC dataset (VOC
10-1) signifies 10 classes learned at the first learning step, and 1
incremental class learned in each subsequent step, _i.e_., VOC 10-1 scenario
takes a total of 11 steps to learn the entire dataset.

Table 1:  Comparison with state-of-the-art methods on Pascal VOC 2012. †: Re-
implemented with Swin-B backbone; Joint is the upperbound.

Method Backbone VOC 10-1 (11 steps) VOC 15-1 (6 steps) VOC 19-1 (2 steps) VOC
15-5 (2 steps) VOC 2-2 (10 steps) 0-10 11-20 all 0-15 16-20 all 0-19 20 all
0-15 16-20 all 0-2 3-20 all Joint Resnet101 82.1 79.6 80.9 82.7 75.0 80.9 81.0
79.1 80.9 82.7 75.0 80.9 76.5 81.6 80.9 LwF-MC [18] Resnet101 4.7 5.9 4.9 6.4
8.4 6.9 64.4 13.3 61.9 58.1 35.0 52.3 3.5 4.7 4.5 ILT [25] Resnet101 7.2 3.7
5.5 8.8 8.0 8.6 67.8 10.9 65.1 67.1 39.2 60.5 5.8 5.0 5.1 MiB [4] Resnet101
12.3 13.1 12.7 34.2 13.5 29.3 71.4 23.6 69.2 76.4 50.0 70.1 41.1 23.4 25.9 SDR
[26] Resnet101 32.1 17.0 24.9 44.7 21.8 39.2 69.1 32.6 67.4 57.4 52.6 69.9
13.0 5.1 6.2 PLOP [12] Resnet101 44.0 15.5 30.5 65.1 21.1 54.6 75.4 37.4 73.5
75.7 51.7 70.1 24.1 11.9 13.7 RCIL [45] Resnet101 55.4 15.1 34.3 70.6 23.7
59.4 68.5 12.1 65.8 78.8 52.0 72.4 28.3 19.0 19.4 SSUL [5] Resnet101 71.3 46.0
59.3 77.3 36.6 67.6 77.7 29.7 75.4 77.8 50.1 71.2 62.4 42.5 45.3 MicroSeg [49]
Resnet101 72.6 48.7 61.2 80.1 36.8 69.8 78.8 14.0 75.7 80.4 52.8 73.8 61.4
40.6 43.5 Joint† Swin-B 82.4 83.0 82.7 83.8 79.3 82.7 82.6 84.4 82.7 83.8 79.3
82.7 75.8 83.9 82.7 SSUL†[5] Swin-B 74.3 51.0 63.2 78.1 33.4 67.5 80.8 31.5
78.4 79.7 55.3 73.9 60.3 40.6 44.0 MicroSeg† [49] Swin-B 73.5 53.0 63.8 80.5
40.8 71.0 79.0 25.3 76.4 81.9 54.0 75.2 64.8 43.4 46.5 CoinSeg (Ours) Swin-B
80.1 60.0 70.5 82.7 52.5 75.5 81.5 44.8 79.8 82.1 63.2 77.6 70.1 63.3 64.3

##### Implementation details.

In accordance with the established practice [4, 5, 26], we use DeepLabv3 [6]
as the segmentation network. CoinSeg chooses Swin Transformer-base (Swin-B)
[19] pretrained on ImageNet-1K as the backbone. Swin Transformer provides a
better feature representation of the local patches, which is concerned in our
CoinSeg. For the segmentation head, we apply the dual-head architecture from
MicroSeg, including dense prediction branch and proposal classification branch
[49]. We optimize the model by ADAMW [21] with an learning rate of
$lr_{0}=10^{-4}$. The batch size is 16 for Pascal VOC 2012, and 12 for ADE20K.
The window size for Swin Transformer is 12. Data augmentation [5] are applied
for all samples. To ensure a fair comparison with previous methods, we apply
the same class-agnostic mask proposals with MicroSeg [49]. Specifically, mask
proposals are generated with parameters-fixed Mask2Former [7] pre-trained on
MS-COCO, with $N=100$ for all experiments. To prevent information leakage, the
Mask2Former is not fine-tuned on any benchmark dataset [49]. All experiments
are implemented with PyTorch on two NVIDIA GeForce RTX 3090 GPUs. Hyper-
parameters $\lambda_{lr}=10^{-3}$, $\lambda_{c}=0.01$ and $\lambda_{r}=0.1$
are set for all experiments.

##### Baselines.

We evaluate our CoinSeg on multiple incremental scenarios. The performance of
CoinSeg is compared with some representative approaches in CIL, including Lwf-
MC [18] and ILT [25], which are applied to the experimental setup of CISS.
Besides, we provide the results of the prior state-of-the-art CISS methods,
including MiB [4], SDR [26], PLOP [12], SSUL [5], RCIL [45], and MicroSeg
[49]. To ensure a fair comparison with recent state-of-the-art CISS methods
[5, 49] that employ different backbones, we produced results by replacing
their backbones with Swin-B. These methods are re-implemented with their
official codes. Methods with suffix ‘M’ (for example, SSUL-M) are with memory
sampling strategy [5], which maintains a memory bank of historical samples for
rehearsal at future learning steps. Besides, we also provide the experimental
results of joint training (training all classes together), on both Resnet101
and Swin-B backbones. The result of joint training is usually regarded as the
upper bound of incremental learning [4, 18], _i.e_., offline training. The
mean Intersection-over-Union (mIoU) is applied as the evaluation metric for
all experiments and analyses. For each method in CISS, three perspectives are
presented: the performance of base classes, novel classes, and total
performance, respectively. Please refer to the supplementary material for more
details.

###  4.2 Experimental Results

##### Comparison on VOC.

Figure 5:  Performance comparisons for VOC. Illustration of the change of miou
with learning step in (a) VOC 15-1, (b) VOC 2-2. Note we calculate mean IoU
for all seen classes until current learning step. _e.g_., the mIoU of classes
0-10 for VOC 2-2 step 5. And (c) Average performance comparisons with 20
different incremental orders of VOC 15-1.

On the Pascal VOC 2012 dataset, we evaluate CoinSeg on various incremental
scenarios, including long-term scenarios with lots of learning steps (10-1,
15-1), with a large number of base classes (19-1, 15-5) and an equally-divided
long-term scenario (2-2). The performance comparison of our approach with
classical CIL methods and prior CISS methods is presented in Tab. 1. Our
CoinSeg exhibits a significant performance advantage in all incremental
scenarios, even when compared to the state-of-the-art methods using Swin
Transformer as the backbone.

In particular, in very long-term incremental scenarios with few base classes,
like 10-1 and 2-2, our CoinSeg brings huge performance gaps of 6.7% and 17.8%,
respectively, in comparison to MicroSeg. The freeze strategy in SSUL and
MicroSeg prevented these methods from performing well in incremental scenarios
with few base classes. In such scenarios, the base classes contain fewer
concepts, which can easily introduce bias for the representation learning of
novel classes. Our approach addresses this challenge by allowing the model to
adapt to novel classes with the design of ‘contrast inter- and intra-class
representations’. In such incremental scenarios, the base classes contain
fewer concepts and can easily introduce bias for the representation learning
of novel classes. Besides, we concern the scenarios with a large number of
base classes (_i.e_., 19-1 and 15-5), in which, CoinSeg achieves state-of-the-
art with a performance gain of 3.4% and 2.4%, compared with MicroSeg.

Table 2:  Comparison with state-of-the-art methods on ADE20K. †: Re-
implemented with Swin-B backbones; Joint is the upperbound.

Method Backbone ADE 100-5 (11 steps) ADE 100-10 (6 steps) ADE 100-50 (2 steps)
ADE 50-50 (3 steps) 0-100 101-150 all 0-100 101-150 all 0-100 101-150 all 0-50
51-150 all Joint Resnet101 43.8 28.9 38.9 43.8 28.9 38.9 43.8 28.9 38.9 50.7
32.8 38.9 ILT [25] Resnet101 0.1 1.3 0.5 0.1 3.1 1.1 18.3 14.4 17.0 3.5 12.9
9.7 MiB [4] Resnet101 36.0 5.7 26.0 38.2 11.1 29.2 40.5 17.2 32.8 45.6 21.0
29.3 PLOP [12] Resnet101 39.1 7.8 28.8 40.5 13.6 31.6 41.9 14.9 32.9 48.8 21.0
30.4 SSUL [5] Resnet101 39.9 17.4 32.5 40.2 18.8 33.1 41.3 18.0 33.6 48.4 20.2
29.6 RCIL [45] Resnet101 38.5 11.5 29.6 39.3 17.7 32.1 42.3 18.8 34.5 48.3
24.6 32.5 MicroSeg [49] Resnet101 40.4 20.5 33.8 41.5 21.6 34.9 40.2 18.8 33.1
48.6 24.8 32.9 Joint † Swin-B 43.5 30.6 39.2 43.5 30.6 39.2 43.5 30.6 39.2
50.2 33.7 39.2 SSUL † [5] Swin-B 41.3 16.0 32.9 40.7 19.0 33.5 41.9 20.1 34.6
49.5 21.3 30.7 MicroSeg † [49] Swin-B 41.2 21.0 34.5 41.0 22.6 34.8 41.1 24.1
35.4 49.8 23.9 32.5 CoinSeg Swin-B 43.1 24.1 36.8 42.1 24.5 36.2 41.6 26.7
36.6 49.0 28.9 35.6

Besides, Fig. 5 (a,b) depicts the variations of mIoU of all seen classes by
learning steps during incremental learning, in two long-term incremental
scenarios, _i.e_., VOC 15-1 and VOC 2-2, respectively. The performance of each
step depends on two factors: 1) the forgetting of historical classes, and 2)
the ability to learn new classes. Fig. 5 (a) shows the results of VOC 15-1,
which involves a large number of base classes. The performance of our CoinSeg
exhibits the least decrease with increasing training steps, suggesting that
CoinSeg causes less forgetting of previously learned knowledge than prior
methods. In contrast, while VOC 2-2 consists of 18 novel classes, results of
Fig. 5 (b) clearly demonstrate that our CoinSeg is more adaptable to learning
new classes, compared to previous methods. Additionally, our CoinSeg even
shows a significant performance improvement while learning new classes in some
steps (steps 3, 4, and 8). Besides, to demonstrate the robustness of CoinSeg,
we provide experimental results of VOC 15-1 with 20 different class
incremental orders, including average and standard variance of mIoU,
illustrate as Fig. 5 (c). The results indicate that our approach performs
better and is more robust, with a lower standard variance.

Table 3:  Comparisons of CoinSeg using memory sampling strategy. †: Re-
implemented with Swin-B backbone.

Method Backbone VOC 10-1 (11 steps) VOC 15-1 (6 steps) VOC 19-1 (2 steps) VOC
15-5 (2 steps) VOC 2-2 (10 steps) 0-10 11-20 all 0-15 16-20 all 0-19 20 all
0-15 16-20 all 0-2 3-20 all SSUL-M [5] Resnet101 74.0 53.2 64.1 78.4 49.0 71.4
77.8 49.8 76.5 78.4 55.8 73.0 58.8 45.8 47.6 MicroSeg-M [49] Resnet101 77.2
57.2 67.7 81.3 52.5 74.4 79.3 62.9 78.5 82.0 59.2 76.6 60.0 50.9 52.2 SSUL-M†
[5] Swin-B 75.3 54.1 65.2 78.8 49.7 71.9 78.5 50.0 77.1 79.3 55.1 73.5 61.1
47.5 49.4 MicroSeg-M† [49] Swin-B 78.9 59.2 70.1 82.0 47.3 73.7 81.0 62.4 80.0
82.9 60.1 77.5 62.7 51.4 53.0 CoinSeg (Ours) Swin-B 80.0 63.4 72.5 82.7 52.5
75.5 81.5 44.8 79.8 82.1 63.2 77.6 70.1 63.3 64.3 CoinSeg-M (Ours) Swin-B 81.3
64.4 73.7 84.1 65.6 79.6 82.7 52.6 81.3 84.1 69.9 80.8 68.4 65.6 66.0

Table 4:  Ablation Studies for our proposed methods. Coin: contrast inter- and
intra-class representations, Fz: Freeze strategy, FLR: flexible initial
learning rate. Numbers in the brackets (): gains w.r.t. the preceding row.

Backbone parameter strategy Coin VOC 15-1 (6 steps) LR $\mathcal{L}^{F}_{kd}$
$\mathcal{L}^{z}_{kd}$ $\mathcal{L}_{int}$ $\mathcal{L}_{itr}$ 0-15 16-20 all
ResNet101 Fz ✗ ✗ ✗ ✗ 74.9 26.4 63.3 Swin-B Fz ✗ ✗ ✗ ✗ 79.5 42.4 70.5 Swin-B
FLR ✗ ✗ ✗ ✗ 73.6 44.0 66.7 Swin-B FLR ✓ ✗ ✗ ✗ 78.9 42.7 70.3 (+3.6) Swin-B FLR
✓ ✓ ✗ ✗ 80.4 43.7 71.6 (+1.3) Swin-B FLR ✓ ✓ ✓ ✗ 80.8 45.9 72.4 (+0.8) Swin-B
FLR ✓ ✓ ✓ ✓ 82.7 52.5 75.5 (+3.1)

##### Comparison on ADE.

For the more challenging ADE dataset, we choose incremental scenarios that
have been widely compared in past methods, including 100-5, 100-10, 100-50,
and 50-50, shown in Tab. 2. Our method, CoinSeg, outperforms previous state-
of-the-art once again, which shows that our method is not specific to a
particular dataset and is effective under multiple benchmarks.

###  4.3 Ablation Studies

##### Overall ablation of CoinSeg.

In this section, we present an evaluation of each specific design of the
proposed CoinSeg, including the Swin Transformer, Flexible Tuning (FT)
strategy, and Contrast inter- and intra-class Representations (Coin). As shown
in Tab. 4, we can find that all these proposed designs benefit the performance
of CISS. By replacing the backbone with Swin-Transformer (row 1 _vs_ row 2),
it improves the ability of feature representation of the feature extractor,
which is especially beneficial for the representation of local features, and
necessary for CoinSeg. The comparison between row 2 and row 3 reflects the
experiment of replacing the freeze strategy with the FT strategy, showing that
the FT enhances model plasticity, allowing adaptation to novel classes, and
slightly improving the performance. The application of Coin (row3 _vs_ row4)
significantly improves the performance by imposing explicit constraints on
intra-class and inter-class diversity, thereby enhancing the model’s ability
to represent data. Overall, these components enable CoinSeg to achieve state-
of-the-art performance.

##### Number of proposal in $\mathcal{L}_{itr}$.

We also investigated the influence of hyperparameter $N$ on the method’s
performance concerning contrast intra-class diversity. Specifically, we
explored how altering the number of proposals would affect the proposed
CoinSeg. The results are presented in Tab. 5. It can be observed that with $N$
increases, there is a slight improvement in performance, albeit with
diminishing returns as the cardinality grows. Consequently, considering a
trade-off between performance and computational complexity, we choose $N=100$
in our method.

Table 5:  Ablations to # of proposals ($N$) in $\mathcal{L}_{itr}$ (Left), and pseudo-labeling in $\mathcal{L}_{int}$ (Right). GT: ground truth, PL: pseudo label.  Method | $N$ | VOC 15-1 (6 steps)
---|---|---
0-15 | 16-20 | all
CoinSeg | 50 | 81.4 | 51.1 | 74.2
100 | 82.7 | 52.5 | 75.5
200 | 83.1 | 53.8 | 76.1

###  4.4 Qualitative Analysis

We have conducted a qualitative analysis using two examples in Fig. 6. In the
first example (rows 1,3 & 5), during incremental learning, CoinSeg is able to
retain knowledge about past classes and accurate predictions for them, whereas
prior methods exhibit forgetting and misclassification, which demonstrated the
stability of CoinSeg. The second example (rows 2, 4 & 6), demonstrates the
plasticity of our method, which refers to the ability to learn new classes.
For example, while prior methods predicted wrong bounds for the class ‘train’
in incremental learning, our method is better adapted to these new classes
making appropriate predictions.

Figure 6: Qualitative analysis for VOC, comparing with prior CISS methods.
Text points out the uncorrectly predicted areas.

###  4.5 Expansibility of CoinSeg

##### Flexible tuning on prior methods.

Table 6:  The performance of Flexible tuning for prior freeze-strategy-based
method MicroSeg, ‘FT’ denotes flexible tuning .

Method Backbone parameter VOC 15-1 (6 steps) strategy 0-15 16-20 all MicroSeg
Resnet101 Freeze 80.5 40.8 71.0 Resnet101 FT 80.9 41.5 72.3 (+1.3) Swin-B
Freeze 80.1 36.8 69.8 Swin-B FT 79.8 40.2 70.4 (+0.6)

As claimed, the FT strategy flexibly releases parameter training for
plasticity and provides some more specific parameter tuning ways to ensure
stability as well. To better validate the effectiveness of the FT strategy, we
applied the FT strategy to the state-of-the-art method MicroSeg, and compared
the results to its original freeze strategy, as shown in Tab. 6. The results
demonstrate that on both backbones, applying the FT strategy leads to better
performance, particularly, for the new classes, which proves that model
plasticity is significantly enhanced by the FT strategy. It means that the FT
strategy can be also extended to other CISS methods for better performance,
especially performance relying on model plasticity.

##### CoinSeg with memory sampling.

The memory sampling strategy alleviates forgetting by rehearsing samples from
past learning steps, and significantly improves performance [5, 49]. Thus, we
produce the results of CoinSeg with this strategy, denoted as CoinSeg-M, for
comparison with prior methods with the same strategy. As shown in Tab. 3, our
CoinSeg (the 5th row) achieves better performance in almost all incremental
scenarios, even when compared with previous work using the sampling strategy.
When equipped with the memory sampling method, _i.e_., CoinSeg-M, it
undoubtedly achieves state-of-the-art performance.

##  5 Conclusion

In this work, we studied class incremental semantic segmentation and proposed
an effective method CoinSeg. Inspired by the Gaussian mixture model, we
proposed Coin to better characterize samples with explicitly constrain inter-
and intra- class diversity. Furthermore, we proposed a flexible tuning
strategy, to keep the stability of the model and alleviate forgetting by the
flexible initial learning rate and regularization constraints. Extensive
experimental evaluations show the effectiveness of our method. CoinSeg
outperforms prior state-of-the-art CISS methods, especially on more
challenging long-term incremental scenarios.

##### Acknowledgment.

This work was supported in part by the National Key R&D Program of China
(No.2021ZD0112100), the National Natural Science Foundation of China (No.
61972036), the Fundamental Research Funds for the Central Universities (No.
K22RC00010).

## References

  * [1] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars.  Memory aware synapses: Learning what (not) to forget.  In Proc. European Conference on Computer Vision, pages 139–154, 2018.
  * [2] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars.  Expert gate: Lifelong learning with a network of experts.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 3366–3375, 2017.
  * [3] Gert Cauwenberghs and Tomaso Poggio.  Incremental and decremental support vector machine learning.  Advances in neural information processing systems, 13, 2000.
  * [4] Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulo, Elisa Ricci, and Barbara Caputo.  Modeling the background for incremental learning in semantic segmentation.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 9233–9242, 2020.
  * [5] Sungmin Cha, YoungJoon Yoo, Taesup Moon, et al.  Ssul: Semantic segmentation with unknown label for exemplar-based class-incremental learning.  Advances in neural information processing systems, 34, 2021.
  * [6] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE Transactions on Pattern Recognition and Machine Intelligence, 40(4):834–848, 2017.
  * [7] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-attention mask transformer for universal image segmentation.  arXiv preprint arXiv:2112.01527, 2021.
  * [8] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov.  Per-pixel classification is not all you need for semantic segmentation.  Advances in neural information processing systems, 34, 2021.
  * [9] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen.  Conditional positional encodings for vision transformers.  arXiv preprint arXiv:2102.10882, 2021.
  * [10] Matthias De Lange and Tinne Tuytelaars.  Continual prototype evolution: Learning online from non-stationary data streams.  In Proc. IEEE International Conference on Computer Vision, pages 8250–8259, 2021.
  * [11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [12] Arthur Douillard, Yifu Chen, Arnaud Dapogny, and Matthieu Cord.  Plop: Learning without forgetting for continual semantic segmentation.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 4040–4050, 2021.
  * [13] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International Journal on Computer Vision, 88(2):303–338, 2010.
  * [14] Kunyang Han, Yong Liu, Jun Hao Liew, Henghui Ding, Jiajun Liu, Yitong Wang, Yansong Tang, Yujiu Yang, Jiashi Feng, Yao Zhao, and Yunchao Wei.  Global knowledge calibration for fast open-vocabulary segmentation, 2023.
  * [15] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2(7), 2015.
  * [16] Heechul Jung, Jeongwoo Ju, Minju Jung, and Junmo Kim.  Less-forgetting learning in deep neural networks.  arXiv preprint arXiv:1607.00122, 2016.
  * [17] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  nature, 521(7553):436–444, 2015.
  * [18] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  IEEE Transactions on Pattern Recognition and Machine Intelligence, 40(12):2935–2947, 2017.
  * [19] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.
  * [20] David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  Advances in neural information processing systems, 30, 2017.
  * [21] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [22] Arun Mallya and Svetlana Lazebnik.  Packnet: Adding multiple tasks to a single network by iterative pruning.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 7765–7773, 2018.
  * [23] Andrea Maracani, Umberto Michieli, Marco Toldo, and Pietro Zanuttigh.  Recall: Replay-based continual learning in semantic segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7026–7035, 2021.
  * [24] Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989.
  * [25] Umberto Michieli and Pietro Zanuttigh.  Incremental learning techniques for semantic segmentation.  In Proc. IEEE International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [26] Umberto Michieli and Pietro Zanuttigh.  Continual semantic segmentation via repulsion-attraction of sparse and disentangled latent representations.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 1114–1124, 2021.
  * [27] Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi.  Learning to remember: A synaptic plasticity driven framework for continual learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11321–11329, 2019.
  * [28] Haim Permuter, Joseph Francos, and Ian Jermyn.  A study of gaussian mixture models of color and texture features for image classification and segmentation.  Pattern recognition, 39(4):695–706, 2006.
  * [29] Franz Pernkopf and Djamel Bouchaffra.  Genetic-based em algorithm for learning gaussian mixture models.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(8):1344–1348, 2005.
  * [30] Robi Polikar, Lalita Upda, Satish S Upda, and Vasant Honavar.  Learn++: An incremental learning algorithm for supervised neural networks.  IEEE transactions on systems, man, and cybernetics, part C (applications and reviews), 31(4):497–508, 2001.
  * [31] Amal Rannen, Rahaf Aljundi, Matthew B Blaschko, and Tinne Tuytelaars.  Encoder based lifelong learning.  In Proc. IEEE International Conference on Computer Vision, pages 1320–1328, 2017.
  * [32] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.  icarl: Incremental classifier and representation learning.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017.
  * [33] Douglas A Reynolds et al.  Gaussian mixture models.  Encyclopedia of biometrics, 741(659-663), 2009.
  * [34] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne.  Experience replay for continual learning.  Advances in neural information processing systems, 32, 2019.
  * [35] Amir Rosenfeld and John K Tsotsos.  Incremental learning through deep adaptation.  IEEE Transactions on Pattern Recognition and Machine Intelligence, 42(3):651–663, 2018.
  * [36] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  arXiv preprint arXiv:1606.04671, 2016.
  * [37] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou.  Overcoming catastrophic forgetting with hard attention to the task.  In Proc. International Conference on Machine Learning, pages 4548–4557, 2018.
  * [38] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim.  Continual learning with deep generative replay.  Advances in neural information processing systems, 30, 2017.
  * [39] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim.  Continual learning with deep generative replay.  Advances in neural information processing systems, 30, 2017.
  * [40] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In International conference on machine learning, pages 10347–10357. PMLR, 2021.
  * [41] Chenshen Wu, Luis Herranz, Xialei Liu, Joost Van De Weijer, Bogdan Raducanu, et al.  Memory replay gans: Learning to generate new categories without forgetting.  Advances in Neural Information Processing Systems, 31, 2018.
  * [42] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  Advances in Neural Information Processing Systems, 34:12077–12090, 2021.
  * [43] Boyu Yang, Chang Liu, Bohao Li, Jianbin Jiao, and Qixiang Ye.  Prototype mixture models for few-shot semantic segmentation.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VIII 16, pages 763–778. Springer, 2020.
  * [44] Guanglei Yang, Enrico Fini, Dan Xu, Paolo Rota, Mingli Ding, Moin Nabi, Xavier Alameda-Pineda, and Elisa Ricci.  Uncertainty-aware contrastive distillation for incremental semantic segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  * [45] Chang-Bin Zhang, Jia-Wen Xiao, Xialei Liu, Ying-Cong Chen, and Ming-Ming Cheng.  Representation compensation networks for continual semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7053–7064, 2022.
  * [46] Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei.  Slca: Slow learner with classifier alignment for continual learning on a pre-trained model.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
  * [47] Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo.  Class-incremental learning via deep model consolidation.  In Proc. IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1131–1140, 2020.
  * [48] Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S Huang.  Sg-one: Similarity guidance network for one-shot semantic segmentation.  IEEE Transactions on Cybernetics, 50(9):3855–3865, 2020.
  * [49] Zekang Zhang, Guangyu Gao, Zhiyuan Fang, Jianbo Jiao, and Yunchao Wei.  Mining unseen classes via regional objectness: A simple baseline for incremental segmentation.  Advances in neural information processing systems, 35, 2022.
  * [50] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 633–641, 2017.
  * [51] Hongguang Zhu, Yunchao Wei, Xiaodan Liang, Chunjie Zhang, and Yao Zhao.  Ctp: Towards vision-language continual pretraining via compatible momentum contrast and topology preservation, 2023.


# Beyond One-Size-Fits-All: Multi-Domain, Multi-Task Framework for Embedding
Model Selection

Vivek Khetan
<EMAIL_ADDRESS>
###### Abstract

This position paper proposes a systematic approach towards developing a
framework to help select the most effective embedding models for natural
language processing (NLP) tasks, addressing the challenge posed by the
proliferation of both proprietary and open-source encoder models.

Beyond One-Size-Fits-All: Multi-Domain, Multi-Task Framework for Embedding
Model Selection

Vivek Khetan<EMAIL_ADDRESS>

##  1 Introduction

The field of Natural Language Processing (NLP) has been revolutionized by the
advent of Transformer-based models Vaswani et al. (2017), notably through
their capacity for generating contextually relevant text and encoding textual
information into dense vector representations. This evolution has led to a
deluge of related research as well as the availability of encoder models, both
proprietary and open-source. This presents a critical challenge: selecting the
most effective encoder for specific NLP tasks, particularly under varied
client requirements. The primary question this position paper addresses is how
to develop a framework for the decision-making process in choosing an
embedding model that best fits a client’s needs.

##  2 Assumptions

In this position paper, we plan to focus on text modality only. Our focus will
be on text encoders designed for the English language, considering their
prevalence and the available resources. While custom training of embedding
models is a viable path, this position paper is focused on evaluating and
selecting the optimal pre-existing embedding model from the various available
options.

However, it is important to note that in many scenarios, custom training may
not be necessary. One of the pivotal considerations in this context is the
cost versus benefit analysis. The substantial resources and time required for
training bespoke models may not always justify the incremental gains in
performance, especially when pre-trained models with extensive coverage and
capabilities are readily available. This efficiency and practicality aspect
underscores why, in most cases, leveraging existing embedding models could be
a more strategic and resourceful choice.

Our goal is to choose the best model and the cost consideration will not be
taken into account right now.

##  3 Experimental Hypothesis

We hypothesize that a strategically selected embedding model, tailored to the
client’s specific data, domain, and downstream tasks, can provide superior
performance by capturing nuanced semantic meanings more effectively than a
generic model.

We aim to establish a methodical approach for embedding model selection that
leverages existing models’ strengths in line with specific downstream task
requirements.

##  4 Related work

Previous research has transitioned from focusing on word-level embeddings,
such as GloVe Pennington et al. (2014), to more sophisticated context-aware
embeddings from Transformer models. These advancements have significantly
pushed the boundaries of natural language understanding, incorporating
strategies like utilizing the [CLS] token of BERT Devlin et al. (2019) for
sentence-level representations, leveraging Sentence-BERT Reimers and Gurevych
(2019) for improved sentence embeddings and embedding based on decoder models
and their variant Black et al. (2022); Neelakantan et al. (2022).

Furthermore, proprietary embedding systems like Ada and more recent text-
embedding-3-(small/large)111https://platform.openai.com/docs/guides/embeddings/what-
are-embeddings from OpenAI have introduced unique capabilities into the NLP
landscape. Despite significant advancements, the application of these models
outside of general web domains remains under-explored. There have been efforts
to measure embedding performance on specific tasks such as Agirre et al.
(2012) for STS and Thakur et al. (2021) for IR tasks. Efforts like the MTEB
benchmarking Wang et al. (2023) initiative have sought to evaluate text
embeddings across diverse tasks and languages, yet no consensus has emerged on
a universally superior embedding approach. MTEB aims to rigorously assess the
capabilities of publicly available embeddings across a variety of tasks,
covering 8 different tasks across a total of 58 datasets and in 112 languages.
Yet, despite such comprehensive efforts, the field has not reached a consensus
on a universally superior embedding approach that excels across all tasks and
languages. This is primarily because NLP tasks such as sentiment analysis,
entity recognition, or machine translation, come with distinct requirements
and complexities. Additionally, each domain, be it medical, legal, or
financial, has unique linguistic features and terminologies. Consequently, an
embedding model’s performance may vary significantly across different
contexts.

This position paper seeks to motivate researchers to build upon existing
foundations, identifying gaps in current methodologies, especially in domain-
specific and task-specific embedding model selection.

##  5 Proposed Experiments

To address the identified challenge, we propose need of a series of
experiments across three distinct scenarios:

###  5.1 Scenario 1: Limited Domain Understanding

In such a scenario, researchers would need to employ metadata analysis and
clustering techniques on client-provided text data to evaluate how well
different embedding models represent data points in latent space, aiming to
identify models that minimize clustering errors. This will involve mapping
each piece of text to a point in latent space and then applying clustering
algorithms to group these data points based on their proximity and similarity.
Examining the distribution of data points within clusters and assessing the
coherence of cluster tags (if pre-existing tags are not available, synthetic
tags or categories derived from external sources like newsrooms could be used
as a proxy), one can evaluate the effectiveness of each embedding model in
capturing the semantic relationships inherent in the text data for a variety
of industry applications and related dataset not readily available for
academic research.

###  5.2 Scenario 2: General Domain with Varied End Tasks

Building upon the insights gained from Scenario 1 - in Scenario 2, researchers
would need to first select a subset of embedding models that showed promise in
accurately representing client data in latent space. The next step involves a
thorough evaluation of these models against a set of tasks that are either
common across clients or unique to specific clients’ needs. This is informed
by the findings that an embedding model’s effectiveness is not universally
applicable across all tasks; what performs well for one application may not
yield the same results for another. To address this, they will need to conduct
task-specific assessments. For tasks that align with a common academic dataset
(or client dataset - if we have access to the defined tasks), they could
leverage publicly available datasets as benchmark comparisons A good starting
point could be the dataset in MTEB benchmark Muennighoff et al. (2022) and
BEIR benchmark Thakur et al. (2021).

###  5.3 Scenario 3: Diverse Domains and Tasks

Scenario 1 and scenario 2 only discuss dealing with general web domains. In
this scenario, researchers may have a variety of domains including legal,
medical, HR, web3, etc. Dealing with such a scenario involves repeating the
process for each domain (e.g., legal, medical) and specific tasks. This
scenario represents the combination of our framework, developed in Scenario 1
(metadata analysis and clustering for each domain) and Scenario 2 (task-
specific performance assessment) based on aggregation of various client
requirements. This multi-domain, multi-task approach paves the way for the
development of a Multi-Domain-Multi-Task MTEB (Multilingual Text Embeddings
Benchmark) benchmarking framework - an extension of the current MTEB
Muennighoff et al. (2022) benchmark for industry applications. Such a
framework would not only extend the existing MTEB initiative but also enrich
it by incorporating a broader spectrum of domains and tasks.

Through these experiments, this position paper aims to develop a framework for
embedding model selection that is both robust and adaptable to various client
requirements and domains.

##  6 Conclusion

This position paper motivates the researchers to bridge the gap in current NLP
practices by providing a systematic approach to embedding model selection. By
aligning model capabilities with specific application needs, we anticipate not
only enhancing task performance but also contributing valuable insights to the
ongoing discourse in the NLP community. Success in this endeavor has the
potential to significantly streamline the development of NLP solutions in
industry settings, enabling practitioners to make informed decisions quickly.

## References

  * Agirre et al. (2012) Eneko Agirre, Daniel Matthew Cer, Mona T. Diab, and Aitor Gonzalez-Agirre. 2012.  Semeval-2012 task 6: A pilot on semantic textual similarity.  In _International Workshop on Semantic Evaluation_.
  * Black et al. (2022) Sid Black, Stella Biderman, Eric Hallahan, Quentin G. Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Martin Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Benqi Wang, and Samuel Weinbach. 2022.  Gpt-neox-20b: An open-source autoregressive language model.  _ArXiv_ , abs/2204.06745.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _North American Chapter of the Association for Computational Linguistics_.
  * Muennighoff et al. (2022) Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2022.  Mteb: Massive text embedding benchmark.  In _Conference of the European Chapter of the Association for Computational Linguistics_.
  * Neelakantan et al. (2022) Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas A. Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David P. Schnurr, Felipe Petroski Such, Kenny Sai-Kin Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welinder, and Lilian Weng. 2022.  Text and code embeddings by contrastive pre-training.  _ArXiv_ , abs/2201.10005.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global vectors for word representation.  In _Conference on Empirical Methods in Natural Language Processing_.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Conference on Empirical Methods in Natural Language Processing_.
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Ruckl’e, Abhishek Srivastava, and Iryna Gurevych. 2021.  Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.  _ArXiv_ , abs/2104.08663.
  * Vaswani et al. (2017) Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Neural Information Processing Systems_.
  * Wang et al. (2023) Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023.  Improving text embeddings with large language models.  _ArXiv_ , abs/2401.00368.


intensity). NL means that the corresponding transition was not observed and
therefore not fitted. All the fitted transitions are labelled with the letter
F. The symbol $\sim$ is representing lines that were fitted at an important
percentage, around 40–70%, and usually without confusion with other lines.
Blending lines, in terms of integrated intensity contamination percentage,
based on preliminary results from synthetic spectra in region 5 (Martín et al.
in prep.) are shown in the last column when surpassing 5%.

Table 8: A-type methanol lines used as input for our LTE model.

Transition |  $\nu$ [GHz] | Type | R1 | R2 | R3 | R4 | R5 | R6 | R7 | R8 | R9 | R10 | Blending lines
---|---|---|---|---|---|---|---|---|---|---|---|---|---
2${}_{1}\rightarrow$11 | 95.914310 |  A+ | N | N | N | F | F | $\sim$ | N | N | N | $\sim$ |  C2H5OH
2${}_{1}\rightarrow$11 | 97.582798 |  A+ | N | N | N | F | F | $\sim$ | N | N | N | F | –
4${}_{1}\rightarrow$31 | 191.810503 |  A+ | N | $\sim$ | F | F | F | F | N | F | N | F | –
4${}_{1}\rightarrow$31 | 195.146790 |  A- | N | $\sim$ | $\sim$ | $\sim$ | F | $\sim$ | N | N | N | N |  CH3OCHO
1${}_{1}\rightarrow$20 | 205.791270 |  A+ | $\sim$ | F | F | F | $\sim$ | F | F | F | F | $\sim$ |  C2H5OH
4${}_{2}\rightarrow$51 | 234.683370 |  A- | $\sim$ | F | F | F | $\sim$ | F | F | $\sim$ | $\sim$ | $\sim$ |  C2H5OH, H2CS
5${}_{1}\rightarrow$41 | 239.746219 |  A+ | F | F | F | F | F | F | N | $\sim$ | $\sim$ | $\sim$ | –
4${}_{3}\rightarrow$42 | 251.866524 |  A-+ | F | F | F | $\sim$ | N | F | F | F | F | N | –
5${}_{3}\rightarrow$52 | 251.890886 |  A+- | F | F | F | $\sim$ | N | F | F | F | F | F | –
6${}_{3}\rightarrow$62 | 251.895728 |  A+- | F | F | F | $\sim$ | N | F | F | F | F | F | –
4${}_{3}\rightarrow$42 | 251.900452 |  A+- | F | F | F | $\sim$ | N | F | F | F | F | F | –
3${}_{3}\rightarrow$32 | 251.905729 |  A-+ | F | F | F | $\sim$ | N | F | F | F | F | F | –
3${}_{3}\rightarrow$32 | 251.917065 |  A+- | F | F | F | $\sim$ | N | F | F | F | F | F | –
7${}_{3}\rightarrow$72 | 251.923701 |  A+- | F | F | F | $\sim$ | N | F | F | F | F | F | –
8${}_{3}\rightarrow$82 | 251.984837 |  A+- | F | F | F | $\sim$ | N | F | F | F | F | F |
6${}_{1}\rightarrow$51 | 287.670767 |  A+ | F | $\sim$ | F | F | F | $\sim$ | $\sim$ | F | F | F | –
1${}_{1}\rightarrow$10 | 303.366921 |  A-+ | F | $\sim$ | $\sim$ | $\sim$ | – | F | F | $\sim$ | $\sim$ | $\sim$ |  H2CN
2${}_{1}\rightarrow$20 | 304.208348 |  A-+ | $\sim$ | $\sim$ | F | $\sim$ | – | F | F | $\sim$ | $\sim$ | $\sim$ |  C2H5OH
3${}_{1}\rightarrow$30 | 305.473491 |  A-+ | $\sim$ | $\sim$ | $\sim$ | $\sim$ | – | $\sim$ | F | $\sim$ | $\sim$ | F |  c–C3H2, CH3SH
4${}_{1}\rightarrow$40 | 307.165924 |  A-+ | F | $\sim$ | $\sim$ | N | – | $\sim$ | F | $\sim$ | F | F |  CH3OCHO
5${}_{1}\rightarrow$50 | 309.290360 |  A-+ | F | $\sim$ | $\sim$ | $\sim$ | – | $\sim$ | F | F | F | F | –
6${}_{1}\rightarrow$60 | 311.852612 |  A-+ | F | F | F | $\sim$ | $\sim$ | F | F | F | F | $\sim$ | –
7${}_{1}\rightarrow$70 | 314.859528 |  A-+ | $\sim$ | F | F | $\sim$ | $\sim$ | F | $\sim$ | $\sim$ | N | $\sim$ | –
2${}_{2}\rightarrow$31 | 335.133570 |  A- | N | $\sim$ | F | F | F | F | F | F | F | $\sim$ |  CH3COOH
7${}_{1}\rightarrow$61 | 335.582017 |  A+ | $\sim$ | $\sim$ | $\sim$ | $\sim$ | F | $\sim$ | F | F | $\sim$ | $\sim$ | –
7${}_{1}\rightarrow$61 | 341.415615 |  A- | F | N | $\sim$ | F | F | F | F | F | F | $\sim$ | –
1${}_{1}\rightarrow$00 | 350.905100 |  A+ | F | F | F | F | F | F | $\sim$ | F | F | F | –
181818Same as for Table 7, but for A-type methanol. Lines denoted with the
symbol – were not attempted to fit in Region 5 (see Section. 4.2.1).

In this Appendix, we present a couple of more sophisticated models, aiming to
characterize in a more refined way both regions well characterized by LTE
conditions and regions with strong maser emission. The former are being
located toward the nucleus of NGC 253, while the latter occupy the outskirts
of the CMZ of NGC 253, in a bi-symmetric configuration.

We performed a model (hereafter model (1)) to fit the spectrum of region 4
(see Table 1), which shows the best agreement between E- and A-type methanol
in our rotation diagrams (see Table 6), both in terms of excitation
temperatures and column densities. In this model we used two physical
components (both with slab geometry) for each methanol type.

In CASSIS it is possible to use the MCMC method to explore the space of
parameters and fit the line profiles of both methanol types together by
adjusting the ISO parameter, which corresponds to the column density ratio of
E–CH3OH over A–CH3OH. Unfortunately, this two-component approach is
constrained to a fixed single excitation temperature per component. Thus, our
region with the lowest difference in $T_{\rm ex}$ (see Fig. 6) between
methanol types, i.e. region 4, is the best candidate to perform such analysis
(model (1)).

The other model (2), addressing region 8, presents the highest difference
between A- and E-methanol excitation temperatures (Table 6), in addition of
having all maser lines but the one at 198.4 GHz observed through the rotation
diagram method (see Table 2). For model (2) we found better results by fitting
both methanol types together, as for our model (1), with two components each.
This is not in contradiction with the rotation diagrams, as they only
accounted for one gaseous component while here we are using two of them. It is
then possible to have one component related to each slope in Fig. 3.

The resulting parameters of the two models are shown in Table 9 after 600
iterations for each model. While model (1) describes the full LTE conditions,
model (2) is mostly non-LTE: the component with the highest column density
shows gas densities of $\sim$6$\times$105 cm-3, far below the critical
densities of methanol lines used in Sect. 4.1. The LTE synthetic spectrum
obtained for region 4 is shown in Fig. 14 for selected lines covering the
entire ALCHEMI frequency range. We did not find important over-fits to the
observed spectra in this two-component LTE modelling. Although some lines are
still underpredicted at frequencies below $\sim$150 GHz, in general there is a
better agreement in comparison to the previous model (single-component)
results in terms of line shape (in blue in Fig. 14).

Summing over column densities of each component in model (1) we obtain column
densities of 1.86$\pm 0.024\times$1015 and 8.29$\pm 0.17\times$1014cm-2 for
E–CH3OH and A–CH3OH, respectively (model uncertainties only, the calibration
uncertainty (15%, see Sect. 2) should be added). These values are slightly
smaller than in our single-component approach (see Table 3). That is probably
due to the avoidance of over-fitting in some line transitions when the
spectrum becomes crowded due to nearby methanol lines, like our panels
centered at 157.3, 241.8, 290.2 and 338.5 GHz presented in Fig. 14.

Our non-LTE modelling allows us to obtain better results than the LTE model
for region 8. This is remarkable in the case of the
$J_{K}\rightarrow(J-1)_{K}$ transitions (for both methanol symmetric types).
Namely, the $2_{K}\rightarrow 1_{K}$ line series ($K$=-1 … 1) centered around
96.7 GHz, the $3_{K}\rightarrow 2_{K}$ series ($K=$-2 … 2) at $\sim$145.1 GHz;
the $4_{K}\rightarrow 3_{K}$ series ($K=$-3 … 3) at $\sim$193.5 GHz, the
$5_{K}\rightarrow 4_{K}$ series ($K=$-4 … 4) at $\sim$241.8 GHz, and the
$6_{K}\rightarrow 5_{K}$ series ($K=$-5 … 5) at $\sim$290.1 GHz, as can be
seen in Fig. 15. Our only concern regarding this non-LTE approach is the
overestimation of the $J_{0}\rightarrow J_{-1}-E$ line series at about 157.3
GHz, with $J$ from 1 to 3. From this model, it can also be observed that the
line at 108 GHz is underpredicted by far, similar to the case of the maser
candidates, although the optical depth for this transition is positive.

From model (2) we have obtained the optical depths ($\tau$) for each methanol
transition, and we use them to discriminate between maser line candidates and
pure outliers in both our rotation diagrams and radiative transfer models
(Tables 2 and 4, respectively). In this way, methanol maser candidates can be
limited to transitions with negative $\tau$ only. The general departure from
LTE observed in the outliers of the CMZ by the two methods outlined in
Sects.4.1 and 4.2 can be explained by blending lines, non-LTE conditions, or
an extra colder and more extended component, when they have positive $\tau$.

Table 9: Best E-type methanol fit parameters from our detailed two components
modelling.

Region | Component |  $N$(Sp) | $T_{\rm kin}$ | FWHM | $V_{\rm LSR}$ | ISO | $n_{\rm H_{2}}$
---|---|---|---|---|---|---|---
|  |  [cm-2] | [K] |  [km s-1] |  [km s-1] |  |  [cm-3]
4 | 1 |  7.67$\times$10${}^{14}\pm$1.20$\times$1013 |  26.68$\pm$0.21 |  84.78$\pm$0.32 | 252 |  1.516$\pm$0.025 | –
| 2 |  4.19$\times$10${}^{14}\pm$1.19$\times$1013 |  32.32$\pm$0.39 |  30.66$\pm$0.25 | 248 |  1.295$\pm$0.061 | –
8 | 1 |  1.35$\times$10${}^{14}\pm$1.29$\times$1013 |  24.76$\pm$0.80 |  75.59$\pm$0.37 | 205 |  1.984$\pm$0.121 |  2.777($\pm$0.101)$\times$108
| 2 |  1.34$\times$10${}^{15}\pm$5.00$\times$1013 |  14.84$\pm$0.32 |  62.42$\pm$0.68 | 200 |  3.141$\pm$0.578 |  5.949($\pm$0.279)$\times$105
191919Uncertainties correspond to 3$\sigma$. Velocity uncertainties (not
shown) correspond to less than 10% of the instrumental uncertainty ($\sim$8–9
km s-1) . Column density ($N$(Sp)) values for A-type methanol can be obtained
dividing the them by the ISO number (column density ratio of E- over A-type
methanol, see Appendix C), while the rest of the parameters are the same for
both methanol symmetric types. The minus symbol (-) indicates that no $n_{\rm
H_{2}}$ was derived as the model is in LTE.

Figure 14: Two-component LTE model for region 4 (in red) over-plotted to the
rest-frame spectra (in black). The common velocity range (in km s-1, after
applying the radio convention and subtracting the $v_{\rm LSR}$ velocity of
the region, as in Fig. 6, namely 252 km s-1, see Table 1) is depicted at the
bottom left panel while line frequencies are indicated above each panel in GHz
and by green dashed lines inside the panels. Previous single-component LTE
modelling (see Fig. 7) is shown in blue for comparison. Figure 15: Two-
component RADEX non-LTE model for region 8 (in red) over-plotted on the rest-
frame spectra (in black). The common velocity range is depicted at the bottom
left panel and was obtained in the same manner as in Fig. 14 but this time
subtracting a $v_{\rm LSR}$ velocity of 205 km s-1 (see Table 1). Line
frequencies are indicated above each panel in GHz and by green dashed lines
inside the panels. Previous single-component LTE modelling (see Fig. 7) is
shown in blue for comparison.

##  Appendix D Outliers identified through LTE modelling

As mentioned in Sect. 4.2.1, there are methanol transitions not reproduced by
our synthetic spectra in LTE conditions. Without considering the two maser
candidates at 95.2 and 278.3 GHz detected by our radiative transfer modelling,
there are still a number of transitions not reproduced (see Table 4). Some
intriguing cases will be described below, ordered by frequency.

The $\mathbf{3_{1}\rightarrow 4_{0}-A^{+}}$ transition line at 107.0 GHz
($E_{\rm up}/$k$=$28.3 K) has intensities surpassing the LTE models in regions
3, 4, 6, and 7. This line is not identified as a maser candidate through the
rotation diagram method, presenting only a couple of values slightly above the
LTE fit of the rotation diagrams by a factor ($N_{\rm up,107\,GHz}$ – $N_{\rm
up,LTE}$) of 0.25 to 0.46 (in regions 6 and 4, respectively), while in other
regions it falls below the fit, like in the case of region 7, where it appears
below the expected value by a factor of 1.1. In the LTE modelling its
intensity is not strongly underpredicted (see Fig. 8), in contrast to the case
of the Class I methanol masers in the $J_{-1}\rightarrow(J-$ 1)${}_{0}-E$ and
$J_{0}\rightarrow(J-$ 1)${}_{1}-A^{+}$ line series (see Fig. 9).

Methanol transitions, frequently observed as masers in interstellar space have
sometimes the property that they show, in case of absence of a proper
stimulating environment, anti-inversion. A characteristic case is the 12.1 GHz
$2_{0}\rightarrow 3_{-1}-E$ transition, often seen as a Class II maser line in
massive star forming regions (e.g. Breen et al. 2011). This line has been
observed in absorption toward dark clouds, thus indicating a typical
excitation temperature below 2.7 K in this class of objects (Walmsley et al.
1988). Furthermore, from Galactic data, Class I and Class II maser regions are
not well correlated (Kang et al. 2015). The 3${}_{1}\rightarrow 4_{0}-A^{+}$
transition at 107.0 GHz may be a related case. The line shows weak absorption
in regions 1 and 9, where class I emission is observed. This leads us to
relate this line to a Class II maser. Another possibility is that this
transition comes from colder gas, following an LTE behavior in the rotation
diagrams but requiring an extra component to be fitted by synthetic spectra.
The presence of an extra component is in line with recent observations
performed by Holdship et al. (2021), based on C2H observations from the same
ALCHEMI dataset. They found an extended and relatively cold component seen
predominantly at lower frequencies (mainly in the $J$=1–0 transition at 87
GHz), where C2H has a gas density ($n_{\rm H2}$) in the range of
(0.35–1.7)$\times$104 cm-3.

When this line is not in emission, it shows either absorption or a low signal.
This happens in regions 1, 2, 5, 8, 9, and 10 (Fig. 8), the same regions where
Class I masers are detected in emission in the rotation diagrams (see Sect.
4.1.1), with the only exception of region 7 and the previously avoided region
5. Masers related to both classes might arise in region 7 due to our limited
angular resolution, that cannot disentangle individual maser spots. Coming
back to the rotation diagram method, within all regions where the 107 GHz line
is detected in emission, the only one where its intensity is lower than
expected by the LTE behavior is region 7. This opens the possibility that this
line is suffering absorption not detectable by the LTE models due to a colder
gas component that may be contaminating our spectra below 156 GHz.

We cannot be confident about the nature of this line, but if it is maser
emission, that will indicate a first detection of methanol Class II maser
emission in NGC 253 and the farthest detection of such a maser (107 GHz) so
far. We note that the strongest Class II line in our Galaxy is the
5${}_{1}\rightarrow 6_{0}-A^{+}$ line at 6.7 GHz (Leurini et al. 2016), first
reported by Menten (1991), which belongs to the same family as the 107.0 GHz
line detected in the present study. Absorption might then be expected for this
methanol line in regions populated by Class I methanol masers. We highlight,
however, that based on a non-LTE analysis of region 8, this line has positive
optical depths (Appendix C).

The $\mathbf{0_{0}\rightarrow 1_{-1}-E}$ transition at 108.9 GHz ($E_{\rm
up}/$k$=5.2$ K) was not reproduced by our models, although it follows LTE
conditions according to the rotation diagrams, where it is placed slightly
above the LTE fit in most regions (see Fig. 3). Its intensity cannot be
reached by our non-LTE approximation either (see Appendix C) and,
interestingly, its line shape is similar to that observed in our maser
candidates at 84.5 and 132.9 GHz. We do not expect negative optical depths for
this line (see Table 4) but also no clear blending lines, either from
preliminary model results (Martín et al. in prep.), or from our current
inspection (see Table 5, last column). Masers in the 0${}_{0}\rightarrow
1_{-1}-E$ transition are expected to be pumped by radiation of hot dust rather
than by ultra-compact H ii regions and to be thermalized faster than other
Class II masers (Kalenskii et al. 2002).

Figure 7 shows that there are plenty of confirmed LTE lines spanning a
frequency range between 86 and 364 GHz. As can be noted toward regions around
the core of the CMZ (regions 3 to 6), the $J_{1}\rightarrow J_{0}-A^{+}$
series of lines at 304.2, 305.5, 307.2, and 309.3 GHz is slightly
underestimated by our synthetic spectra and is weakly misaligned from the LTE
behavior in the rotation diagrams. Those lines belong to the same family as
the line at 318.3 GHz, previously observed to depart from the rotation
diagrams (see Table 2). Since the difference between predicted and observed
line strengths is mostly not remarkable, we do not account for them as new
maser candidates. In addition, based on a non-LTE model performed for region 8
(Appendix C), only positive optical depths are calculated for these lines.

The $\mathbf{J_{K}\rightarrow(J-1)_{K}-X}$ line series, with $X$ being either
$E$, $A^{+}$, or $A^{-}$ are successfully reproduced in region 8 by a mixture
between LTE and non-LTE models (see Appendix C), being placed out of LTE but
with positive optical depth, indicating a quasi-thermal nature. We have
therefore decided to discard them as maser candidates.

The $\mathbf{5_{3}\rightarrow 5_{2}-A^{-+}}$ line at 251.81 GHz is the only
among 13 transitions in the $J_{K}\rightarrow J_{K}-A$ line series (251–252
GHz) which shows a clear departure from our LTE modelling. The intensities of
this line are much higher than expected in the central regions of NGC 253. We
performed an LTE model for SO (1000 iterations in region 6 only, not shown)
that can account for $\sim$64% of the emission at 251.83 GHz, and therefore we
do not expect an important contribution for the methanol line, assuming that
other species can be also contaminating. However, as this line is not fully
reproduced, we add this as an outlier in our Table 4.

##  Appendix E Maser line distribution at 146.6 GHz

As mentioned in Sect 4.1.3, the $9_{0}\rightarrow 8_{1}-A^{+}$ line at 146.6
GHz is observed to depart from LTE in regions 1, 7, 8, and 9, with intensities
$\geq$8.4$\pm$1.2 stronger than predicted, indicating a maser behaviour
according to our threshold of 3.3 for this factor. Following its cousins at
lower frequencies in the $J_{0}\rightarrow(J-1)_{1}-A^{+}$ series, at 44.1
($J=7$) and 95.2 GHz ($J=8$), it belongs to the Class I maser category (e.g.
Leurini et al. 2016; Yang et al. 2020, and references therein).

It can be noted in Fig. 16 that, contrary to what is observed for methanol
masers in the $J_{-1}\rightarrow(J-1)_{-1}-E$ family of lines (except the last
covered transition at 278.3 GHz), the $9_{1}\rightarrow 8_{0}-A^{+}$ methanol
line at 146.6 GHz peaks in region 6. That might be an indication that this
line is a better tracer of the highest densities instead of cloud-cloud
collisions, that dominate the outskirts of the CMZ of NGC 253.

Figure 16: Same as for Figure 4 but for the $9_{1}\rightarrow 8_{0}-A^{+}$
methanol line at 146.6 GHz.

##  Appendix F Equivalency between intensity and column density ratios

In Fig. 13 we present line intensity ratios between the
5${}_{-1}\rightarrow$4${}_{0}-E$ methanol transition at 84.5 GHz and the
average of a couple of E-type methanol lines that are always found to follow
LTE conditions (according to our rotation diagrams; see Sect. 5.2). It is
important to note a certain equivalency between LTE departure, in terms of
$N_{\rm up,maser}$/$N_{\rm up,LTE}$ (as in Fig. 6) and line intensity ratios
between the 84.5 GHz line and the proposed LTE average. They are not
necessarily the same, especially considering optically thick transitions and
negative optical depths and excitation temperatures present in masers.
Therefore, we reproduce in Fig. 17 the first panel of Fig. 6 with the values
on the leftmost y-axis along with the intensity ratios averaged inside the
beam size (1$\aas@@fstack{\prime\prime}$6 diameter) presented in Fig. 13. The
results are in agreement in terms of distribution: regions where the
5${}_{-1}\rightarrow$4${}_{0}-E$ line is observed in LTE through the rotation
diagram method (regions 3, 4, and 6) present the lowest integrated intensity
ratios. We note that region 5 was avoided when performing the rotation
diagrams and that region 10 is not available for all the thermal lines used to
obtain the LTE average. We found that a threshold of 3.3 in terms of upper
level column density ratios is equivalent to a factor of 0.1244 in terms of
integrated intensity ratios, at least within the studied regions.

Figure 17: Magenta stars are scaled according to the left y-axis, and are
equivalent to the magenta stars in the first panel of Fig. 6. Blue dots are
scaled with respect to the right y-axis and correspond to integrated
intensities of the 5${}_{-1}\rightarrow$4${}_{0}-E$ methanol transition at
84.5 GHz over an LTE approximation (see Sect. 5.2) averaged inside a beam size
aperture (1$\aas@@fstack{\prime\prime}$6 diameter) for each of the studied
regions in this work (see Table 3).