11institutetext: VCIP, CS, Nankai University 22institutetext: NKIARI, Shenzhen
Futian
22email<EMAIL_ADDRESS>
22email<EMAIL_ADDRESS>33institutetext: SICE, UESTC

# Early Preparation Pays Off: New Classifier Pre-tuning for Class Incremental
Semantic Segmentation

Zhengyuan Xie 11 0000-0003-2522-285X Haiquan Lu 11 0009-0003-2115-9782  Jia-
wen Xiao 11 Enguang Wang 11 0009-0002-5161-913X
Le Zhang 33 Xialei Liu (🖂) 1122 0000-0001-8534-3026

###### Abstract

Class incremental semantic segmentation aims to preserve old knowledge while
learning new tasks, however, it is impeded by catastrophic forgetting and
background shift issues. Prior works indicate the pivotal importance of
initializing new classifiers and mainly focus on transferring knowledge from
the background classifier or preparing classifiers for future classes,
neglecting the flexibility and variance of new classifiers. In this paper, we
propose a new classifier pre-tuning (NeST) method applied before the formal
training process, learning a transformation from old classifiers to generate
new classifiers for initialization rather than directly tuning the parameters
of new classifiers. Our method can make new classifiers align with the
backbone and adapt to the new data, preventing drastic changes in the feature
extractor when learning new classes. Besides, we design a strategy considering
the cross-task class similarity to initialize matrices used in the
transformation, helping achieve the stability-plasticity trade-off.
Experiments on Pascal VOC 2012 and ADE20K datasets show that the proposed
strategy can significantly improve the performance of previous methods. The
code is available at https://github.com/zhengyuan-xie/ECCV24_NeST.

###### Keywords:

Class incremental learning Semantic segmentation

##  1 Introduction

Nowadays, segmentation models driven by deep neural networks have achieved
notable success in various fields [33, 10, 9]. However, these models are
usually trained and tested in a static environment [20], which is contrary to
real-world scenarios [47]. When facing a continuous data stream [24], a model
needs to handle several novel classes. Blending new and previous data to
retrain the model, or called _Joint Training_ , ensures the performance of the
model. However, the computational cost becomes prohibitive when dealing with
large datasets. Simply fine-tuning the model may lead to the loss of
previously acquired knowledge, a phenomenon known as catastrophic forgetting
[36, 29]. In particular, for semantic segmentation, it also faces the problem
of background shift [5] (i.e., pixels labeled as background in the current
step may belong to previous or future classes).

Figure 1:  Different classifier initialization methods for class incremental
semantic segmentation. MiB [5] directly uses background classifiers to
initialize new classifiers. Some methods [6, 2] train an auxiliary classifier
for future classes. AWT [21] selects the most relevant weights from the
background classifier for new classifiers’ initialization by gradient-based
attribution. Our new classifier pre-tuning method learns a transformation from
all old classifiers to generate new classifiers for initialization.

Class incremental semantic segmentation (CISS) [5, 16, 55, 6] has been
proposed to address the challenges of catastrophic forgetting and background
shift. It requires the segmentation model to learn concepts of new classes
while preserving old knowledge. If no restrictions are imposed, parameters
crucial to old classes may be updated in the wrong direction [28], causing the
model to forget previously learned knowledge. On the contrary, only focusing
on memorizing old knowledge may limit the model to learning new classes. Thus,
the key issue to continual learning is actually the trade-off between
stability and plasticity [22, 28].

As far as stability is concerned, existing methods rely on old models learned
from the previous step for knowledge transfer [5, 16, 55, 50], _e.g_. _weight
initialization_. Yet the old model does not have the ability to recognize new
classes, finding a proper way to initialize the newly added classifier is
crucial. Random initialization may cause the misalignment between new
classifiers and features, leading to training instabilities [5]. In accordance
with this situation, as Fig. 1 shows, MiB [5] utilizes the background
classifier to initialize new classifiers, as future classes may appear in
current data, labeled as the background. However, it may cause the model to
make incorrect predictions for true background pixels [21]. AWT [21] selects
the most relevant weight for initialization from the old background classifier
via a gradient-based attribution technique, yet it neglects other old
classifiers and brings a huge memory cost. Some methods train an auxiliary
classifier to initialize new classifiers [6, 2], while there is still a bias
between the auxiliary classifier and true future classifiers because there is
not any future data with ground truth. What’s more, the above initialization
methods treat each new classifier equally, ignoring the differences between
new classes.

From the above observations, we propose New claSsifier pre-Tuning (NeST)
method to make new classifiers better align with the backbone and adapt to the
training data, preventing drastic changes in the feature extractor caused by
unstable training process. To better utilize the old knowledge, instead of
directly tuning the parameters of new classifiers, we tune a linear
transformation from all old classifiers to generate new classifiers.
Specifically, before the formal training process at the current step, we
assign two transformation matrices, _i.e_., an importance matrix, and a
projection matrix, to each new class. As trainable parameters, the importance
matrix learns a weighted score associated with each channel of the old
classifier pertinent to the new class, while the projection matrix learns a
linear combination from the weighted old classifiers to generate the new
classifier. Subsequently, we use data from the current step to learn the
transformation from old classifiers to each new classifier. Finally, we employ
learned importance matrices and projection matrices to generate new
classifiers from old ones, and then leverage the derived weights for
classifier initialization of the formal training process. By using these two
matrices, new classifiers are generated via old knowledge and are different
from each other.

To achieve the stability-plasticity trade-off, we also find it crucial for the
initial values of these transformation matrices and propose a strategy for
initialization by considering cross-task class similarities between old and
new classes, thus facilitating the learning of new classifiers.

To summarize, the main contributions of our paper are threefold:

  1. 1.

We propose a new classifier pre-tuning (NeST) method that learns a
transformation from all old classifiers to generate new classifiers with
previous knowledge before the formal training process.

  2. 2.

We further optimize the initialization of transformation matrices, striking a
balance between stability and plasticity.

  3. 3.

We conduct experiments on Pascal VOC 2012 and ADE20K. Results show that NeST
can be readily integrated into other approaches and significantly enhance
their performance.

##  2 Related Work

Semantic Segmentation. In recent years, the development of deep learning
techniques has facilitated the performance of semantic segmentation models.
Fully Convolutional Networks (FCN) [34] pioneers semantic segmentation, and a
series of convolution-based segmentation networks have achieved high
performance in many benchmarks [41, 8, 57, 48, 23, 19]. More recently, the
Transformer architecture has become ever more popular and reached remarkable
performance [51, 13, 11, 40, 59, 44, 32]. Thus, we conduct experiments on two
different backbones: ResNet and Swin-Transformer.

Class Incremental Learning. Class incremental learning tackles the
catastrophic forgetting of the task with ever-increasing categories. The whole
training process is divided into several steps and in each step, the model is
required to learn one or more classes, which is the most challenging. Existing
methods can be categorized into three groups. The _model expansion_ methods
[54, 27, 26, 52] enlarge the model size in incremental steps to learn new
knowledge while preserving old knowledge in fixed parameters. The _rehearsal
based_ methods store a series of exemplars [3, 45] or prototypes along the
task sequence [61], or using generative networks to maintain old knowledge
[43, 35]. The _parameter regularization_ methods either constrain the learning
of some parameters [30, 7] or use knowledge distillation techniques [17, 49,
1, 16].

Class Incremental Semantic Segmentation. CISS requires a segmentation model to
recognize all learned classes in continuous learning steps. Different from
classification tasks, segmentation has its own background shift [5] issue. MiB
[5] first proposes unbiased cross entropy and distillation to address
background shift. PLOP [16] uses pseudo-label and intermediate features to
transfer old knowledge. SDR [38] proposes a contrastive-learning-based method,
minimizing intra-class feature distances. RCIL [55] introduces
reparameterization to continual semantic segmentation with a complementary
network structure. GSC [14] explores incremental semantic segmentation
considering the gradient and semantic compensation. EWF [50] fuses old and new
knowledge with a weight fusion strategy. Many methods have explored the
classifier initialization in CISS. SSUL [6] employs auxiliary data to train an
‘unknown’ classifier and use it to initialize new classifiers. DKD [2]
proposes a decomposed knowledge distillation, improving the rigidity and
stability. AWT [21] applies gradient-based attribution to transfer the
relevant weight of old classifiers to new classifiers. Different from the
method above, we propose a new classifier pre-tuning method, achieving the
goal of the trade-off between plasticity and stability.

##  3 Method

###  3.1 Preliminaries

Problem Definition. Following [5, 16], CISS contains several steps
$\\{t\\}_{t=1}^{n}$ to receive sequential data stream
$\\{\mathcal{D}_{t}\\}_{t=1}^{n}$ with classes
$\\{\mathcal{C}_{t}\\}_{t=1}^{n}$. In each step $t$, the model is required to
learn $\left|\mathcal{C}_{t}\right|$ new classes, while old training data is
not available. For an image in the current step, pixels belonging to
$\mathcal{C}_{t}$ are labeled as their ground truth classes, leaving other
pixels labeled as background. Finally, after the last step, the model will be
tested on the data of all learned classes.

At step $t$, the segmentation model is composed of a feature extractor
$f_{\theta}^{t}$ and a classifier $h_{\phi}^{t}$. $\phi_{t}$ is newly added
and weights of previous classifiers are $\phi_{1:t-1}$. In this paper, our
main concern is the initialization of $\phi_{t}$.

Figure 2:  Illustration of our new classifier pre-tuning (NeST) method. The
left side of the figure is an iteration of the new classifier pre-tuning
process. The right side of the figure represents the cross-task class
similarity-based initialization of importance matrices and projection matrices
before the pre-tuning process.

Revisiting Previous Initialization Methods. In this part, we illustrate the
shortcomings of previous initialization methods and further clarify our
motivation. As mentioned before, classes in $\mathcal{C}_{t}$ will appear as
background in previous steps, which is known as the _background shift_.
According to this premise, previous methods either try to find a better way of
utilizing the background classifier [5, 21], or train an auxiliary classifier
for future classes, both neglecting the differences between new classes [6,
2]. Meanwhile, there will also be a mismatching issue between new classifiers
and the feature extractor as the methods above do not have training processes
with data from new classes. This may lead to drastic parameter changes in the
feature extractor when training with new data, thus undermining previous
knowledge.

###  3.2 New Classifier Pre-Tuning

Considering the observations above, we propose a simple yet effective method,
NeST, to improve the performance of CISS. In the pre-tuning stage before the
formal training process, we first learn to generate each new classifier via
old classifiers and then tune it with new data in each forward pass.

New Classifier Generation. As shown in Fig. 2, for each new class
$c\in\mathcal{C}_{t}$, we assign an importance matrix and a projection matrix,
learning the transformation from all old classifiers in each forward pass as
follows:

$\displaystyle\mathbf{w}_{c}$
$\displaystyle=(\mathbf{M}_{c}\odot\mathbf{W}_{old})\mathbf{P}_{c}\,,$ (1)

where $\mathbf{w}_{c}\in\mathbb{R}^{d}$ denotes the weight of new class $c$,
$\mathbf{M}_{c}\in\mathbb{R}^{d\times n_{old}}$ denotes the importance matrix
of class $c$ and $n_{old}=\left|\bigcup_{i=1}^{t-1}\mathcal{C}_{i}\right|+1$
denotes the number of old classes including the background, $\odot$ denotes
the operation of Hadamard product, $\mathbf{W}_{old}\in\mathbb{R}^{d\times
n_{old}}$ denotes the weight matrix of all old classifiers,
$\mathbf{P}_{c}\in\mathbb{R}^{n_{old}\times 1}$ denotes the projection matrix
of class $c$, and $d$ denotes the output dimension of $f_{\theta}^{t-1}$.
$\mathbf{M}_{c}$ reflects the importance of each channel of the old
classifiers for the new class $c$, and $\mathbf{P}_{c}$ combines weighted old
classifiers, completing weight generation for the new class. After generating
new weights, we concatenate them as the parameter of $\phi_{t}$ as follows:

$\displaystyle\phi_{t}=\rm{concat}[\mathbf{w}_{n_{old}},...,\mathbf{w}_{n_{old}+\left|\mathcal{C}_{t}\right|-1}]\,,$
(2)

where $\text{concat}[]$ is the concatenation operation. Specifically, as the
old background class may see new classes in previous steps and the score of
background in the pre-tuning process may be high, preventing the model from
learning new classes, we also learn a transformation from the old background
classifier to the new background classifier during the pre-tuning process as
follows:

$\displaystyle\hat{\mathbf{w}}_{0}$
$\displaystyle=(\mathbf{M}_{0}\odot\mathbf{w}_{0})\mathbf{P}_{0}\,,$ (3)

where $\mathbf{w}_{0}\in\mathbb{R}^{d}$ denotes the previous background
classifier, $\mathbf{M}_{0}\in\mathbb{R}^{d\times 1}$ and
$\mathbf{P}_{0}\in\mathbb{R}^{1\times 1}$ are matrices specifically for the
background, and $\hat{\mathbf{w}}_{0}\in\mathbb{R}^{d}$ is new background
classifier prepared for current task.

Pre-tuning and Initialization. In each forward pass, as Fig. 2 shows, after
generating weights of new classifiers, we feed training data from the current
step to the old model equipped with generated classifiers, using the output to
calculate the loss and then backpropagating the loss to update learnable
parameters. The loss function used in the pre-tuning is unbiased cross entropy
$\mathcal{L}_{unce}$ [5], as it can help to avoid the overfitting problem
[39]:

$\displaystyle\mathcal{L}_{unce}=-\frac{1}{\left|\mathcal{I}\right|}\sum_{i\in\mathcal{I}}\log\tilde{q}_{x}^{t}(i,y_{i})\,,$
(4)

where $\mathcal{I}$ denotes the pixel set of an image, $y_{i}$ denotes the
ground-truth label of pixel $i$ at current step, $\tilde{q}_{x}^{t}$ denotes
the modified output of the current model. It should be noted that during the
whole pre-tuning process, the importance matrices and projection matrices are
learnable and will be updated, while other components such as the feature
extractor remain frozen. After the pre-tuning process, we use importance
matrices and projection matrices to generate weights of each new classifier
for initialization, and additional parameters will be removed.

###  3.3 Cross-Task Class Similarity-based Transformation Matrix
Initialization

We found it crucial to initialize importance matrices and projection matrices.
Random initialization disregards the importance of different channels among
old classifiers and results in poor performance. AWT [21] reveals that only a
few channels of the background classifier contribute to the classification of
new classes, redundant channels may hinder the learning of new classifiers.
Meanwhile, for different new classes, the important channels of old
classifiers should be different. Hence, we introduce a cross-task class
similarity-based transformation matrix initialization method. The core idea is
that if an old class is more similar to a new class, then during the pre-
tuning process, the contribution of this old classifier should be more
significant, and thus, greater initial weights should be assigned to
corresponding positions in the importance matrix and projection matrix.

Hence, we employ the predictions made by the old model as cross-task class
similarity scores for each new class pixel, aiming to assess the resemblance
between the new class and every old class. As shown in Fig. 2, before learning
transformation, we feed each training image in the current step to the old
model and get the corresponding output of the layer before $h_{\phi}^{t-1}$.
Then considering the classification process of a new class pixel embedding
$\mathbf{p}_{u}\in\mathbb{R}^{d}$ extracted from the old model, assuming that
the total number of old classes is $n_{old}$, the matrix multiplication can be
decomposed into an element-wise multiplication (also called Hadamard product)
between $\mathbf{p}_{u}$ and the old classifier weight
$\mathbf{W}_{old}\in\mathbb{R}^{d\times n_{old}}$ and a sum-softmax operation,
as follows:

$\begin{split}\mathbf{H}_{u}&=(\mathbf{W}_{old}\odot\mathbf{p}_{u}^{\prime}),\\\
\mathbf{s}_{u}&={\rm softmax}({\rm sum}(\mathbf{H}_{u}))^{\top}\,,\end{split}$
(5)

where $\mathbf{p}_{u}^{\prime}\in\mathbb{R}^{d\times n_{old}}$ denotes
broadcasting $\mathbf{p}_{u}$ by $n_{old}$ times,
$\mathbf{H}_{u}\in\mathbb{R}^{d\times n_{old}}$ denotes the result of the
Hadamard product, ${\rm sum}$ denotes summing $\mathbf{H}_{u}$ along the
channel dimension, and $\mathbf{s}_{u}\in\mathbb{R}^{n_{old}}$ denotes the
classification scores of $\mathbf{p}_{u}$. We consider that for each column
(corresponding to an old class $i$) of $\mathbf{H}_{u}$, the positive elements
contribute to $\mathbf{p}_{u}$ being classified into class $i$. Thus, to
select relevant channels, we set all positions with positive values to $1$ and
get $\mathbf{H}_{u}^{mask}$, which can be regarded as a binary mask, as
follows:

$\mathbf{H}_{u}^{mask}(i,j)=\left\\{\begin{aligned}
&1,\mathbf{H}_{u}(i,j)>0\\\ &0,otherwise\,.\end{aligned}\right.$ (6)

Finally, by utilizing ground-truth masks of the current step, for each pixel
embedding belonging to new class $\boldsymbol{c_{new}}$, we calculate the
Hadamard product between its corresponding $\mathbf{H}_{mask}$ and predicted
score, then averaged the results to get the importance matrix weight
$\mathbf{M}_{c_{new}}$ of class $c_{new}$, as follows:

$\displaystyle\mathbf{M}_{c_{new}}=\frac{1}{N}\sum_{\mathbf{p}_{u}}\mathbf{H}_{u}^{mask}\odot\mathbf{s}_{u}^{\prime}\,,$
(7)

where $\mathbf{p}_{u}$ denotes the pixel embedding belonging to the new class
$c_{new}$, $\mathbf{s}_{u}^{\prime}\in\mathbb{R}^{d\times n_{old}}$ denotes
broadcasting the predicted score $\mathbf{s}_{u}$ of $\mathbf{p}_{u}$ by $d$
times, $N$ denotes the number of pixels belonging to new class $c_{new}$. Here
we use $\mathbf{s}_{u}^{\prime}$ because the old class with a very small score
will not contribute too much to the initialization of the new class, as the
similarity between these two classes is relatively low. For projection matrix
weight $\mathbf{P}_{c_{new}}$, we sum up $\mathbf{M}_{c_{new}}$ along the
channel dimension and apply the softmax function to get the weight score for
old classes, as follows:

$\displaystyle\mathbf{P}_{c_{new}}=({\rm softmax}({\rm
sum}(\mathbf{M}_{c_{new}})))^{\top}\,.$ (8)

Then we use $\mathbf{P}_{c_{new}}\in\mathbb{R}^{n_{old}\times 1}$ to
initialize the projection matrix of class $c_{new}$ as these scores reflect
the degree of similarity between the new class and old classes, deciding which
old class the model should focus on for knowledge transfer. We provide the
pseudo-code of our NeST in Algorithm 1.

Algorithm 1 Pseudo-code of the proposed NeST.

1:Training samples $\mathcal{D}_{t}=\\{(x_{i},y_{i})\\}_{t}$ of current task
$t$, feature extractor $f_{\theta}^{0}$, classifier $h_{\phi}^{0}$, task
number $T$, new class set $\mathcal{C}_{t}$.

2:initial weight of new classifier $\phi_{t}$

3:for $t\in$ $\\{1,2,...,T\\}$ do

4: for $i\in$ $\mathcal{C}_{t}$ do $\triangleright$ Initializing
Transformation matrices. Sec. 3.3

5: $(\mathbf{M}_{i},\mathbf{P}_{i})$ ←
$Init(\mathcal{D}_{t},h_{\phi}^{t-1},f_{\theta}^{t-1})$

6: end for

7: while not converged do $\triangleright$ Pre-tuning new classifiers. Sec.
3.2

8: train $(\mathbf{M},\mathbf{P})$ by minimizing $\mathcal{L}_{unce}$

9: end while

10: $\phi_{t}$ ← ${\rm Transform}(\mathbf{M},\mathbf{P},\phi_{1:t-1})$
$\triangleright$ Initializing new classifiers.

11: $f_{\theta}^{t}$ ← $f_{\theta}^{t-1}$

12: $h_{\phi}^{t}$ ← $(h_{\phi}^{t-1},\phi_{t})$

13: while not converged do $\triangleright$ Formal training process.

14: train $f_{\theta}^{t}$ and $h_{\phi}^{t}$

15: end while

16:end for

##  4 Experiments

###  4.1 Experimental setup

Protocols. In CISS, the whole training process contains $T$ steps, and the
task of each step may contain one or more classes. In step $t$, pixels
belonging to previous steps are labeled as _background_. In evaluation, the
model needs to identify all learned classes. There are two settings:
_Disjoint_ and _Overlapped_ [5]. _Disjoint_ setting assumes that in the
current step, images do not contain any pixel belonging to classes that will
be learned in the future. _Overlapped_ setting is more realistic, as future
classes may appear in images from the current step, and we mainly evaluate our
method on _Overlapped_ setting.

Datasets. Pascal VOC 2012 dataset [18] contains 20 classes including 10,582
images for training and 1449 images for validation. ADE20K dataset [60]
contains 150 classes including 20,210 images for training and 2000 images for
validation. In CISS, the training setting can be described as $X-Y$, $X$ means
the number of classes trained in the initial step and $Y$ means the number of
classes trained in incremental steps. We conduct experiments with _10-1_ ,
_15-1_ , _15-5_ and _19-1_ settings on Pascal VOC 2012 dataset [18]. For
ADE20K dataset [60], we validate our method on _100-50_ , _100-10_ , _100-5_
and _50-50_ settings.

Implementation Details. Following previous methods [5, 16, 55], we use
DeepLabV3 [8] with ResNet-101 [25] as our segmentation model. We also use
Swin-B [32] as the Transformer-based backbone. Following [5, 16], we use
random crop and horizontal flip for data augmentation. We train the model with
a batch size of 24 on 4 GPUs for all experiments. We use SGD as the optimizer
in our experiments. The learning rate is set to 0.02 for the initial step and
0.001 for incremental steps, both for Pascal VOC 2012 and ADE20K. We also use
_poly_ schedule as the weight decay strategy. For Pascal VOC 2012, we pre-tune
for 5 epochs with a learning rate of 0.001. For ADE20K, pre-tuning lasts 15
epochs, with a learning rate of 0.1 for experiments involving RCIL and Swin-B,
and 0.5 for other experiments. Further details are provided in the
supplementary material.

Table 1: The mIoU (%) of the last step on the Pascal VOC dataset for four
different CISS scenarios. * implies results from the re-implementation of the
official code.

Method | Backbone | 10-1(11 steps) | 15-1(6 steps) | 15-5(2 steps) | 19-1(2 steps)
---|---|---|---|---|---
0-10 | 11-20 | all | 0-15 | 16-20 | all | 0-15 | 16-20 | all | 0-19 | 20 | all
Joint | Res101 | 78.8 | 77.7 | 78.3 | 79.8 | 73.4 | 78.3 | 79.8 | 73.4 | 78.3 | 78.2 | 80.0 | 78.3
ILT [37] | Res101 | 7.2 | 3.7 | 5.5 | 9.6 | 7.8 | 9.2 | 67.8 | 40.6 | 61.3 | 68.2 | 12.3 | 65.5
SDR [38] | Res101 | 32.4 | 17.1 | 25.1 | 44.7 | 21.8 | 39.2 | 75.4 | 52.6 | 69.9 | 69.1 | 32.6 | 67.4
PLOP+UCD [53] | Res101 | 42.3 | 28.3 | 35.3 | 66.3 | 21.6 | 55.1 | 75.0 | 51.8 | 69.2 | 75.9 | 39.5 | 74.0
SPPA [31] | Res101 | - | - | - | 66.2 | 23.3 | 56.0 | 78.1 | 52.9 | 72.1 | 76.5 | 36.2 | 74.6
MiB+AWT [21] | Res101 | 33.2 | 18.0 | 26.0 | 59.1 | 17.2 | 49.1 | 77.3 | 52.9 | 71.5 | - | - | -
ALIFE [39] | Res101 | - | - | - | 64.4 | 34.9 | 57.4 | 77.2 | 52.5 | 71.3 | 76.6 | 49.4 | 75.3
GSC [14] | Res101 | 50.6 | 17.3 | 34.7 | 72.1 | 24.4 | 60.8 | 78.3 | 54.2 | 72.6 | 76.9 | 42.7 | 75.3
MiB [5] | Res101 | 12.2 | 13.1 | 12.6 | 38.0 | 13.5 | 32.2 | 76.4 | 49.4 | 70.0 | 71.2 | 22.1 | 68.9
MiB* [5] | Res101 | 10.4 | 9.9 | 10.2 | 45.2 | 15.7 | 38.2 | 76.8 | 49.1 | 70.2 | 71.6 | 28.6 | 69.6
MiB+NeST (Ours) | Res101 | 52.3 | 21.0 | 37.4 | 61.7 | 20.4 | 51.8 | 77.1 | 50.1 | 70.7 | 71.7 | 28.2 | 69.7
PLOP [16] | Res101 | 44.0 | 15.5 | 30.5 | 65.1 | 21.1 | 54.6 | 75.7 | 51.7 | 70.1 | 75.4 | 37.4 | 73.5
PLOP* [16] | Res101 | 45.9 | 17.1 | 32.2 | 66.8 | 22.3 | 56.2 | 77.0 | 50.9 | 70.8 | 75.7 | 39.4 | 74.0
PLOP+NeST (Ours) | Res101 | 54.2 | 17.8 | 36.9 | 72.2 | 33.7 | 63.1 | 77.6 | 55.8 | 72.4 | 77.0 | 49.1 | 75.7
RCIL [55] | Res101 | 55.4 | 15.1 | 34.3 | 70.6 | 23.7 | 59.4 | 78.8 | 52.0 | 72.4 | 77.0 | 31.5 | 74.7
RCIL* [55] | Res101 | 47.8 | 17.0 | 33.1 | 69.9 | 23.9 | 58.9 | 78.8 | 52.4 | 72.5 | 76.8 | 28.9 | 74.5
RCIL+NeST (Ours) | Res101 | 51.4 | 20.9 | 36.8 | 71.9 | 28.0 | 61.4 | 79.0 | 52.8 | 72.8 | 77.0 | 33.3 | 74.9
Joint | Swin-B | 80.4 | 79.7 | 80.1 | 81.1 | 76.7 | 80.1 | 81.1 | 76.7 | 80.1 | 80.0 | 80.7 | 80.1
MiB* [5] | Swin-B | 11.4 | 18.9 | 15.0 | 35.0 | 43.2 | 36.9 | 80.7 | 66.5 | 77.3 | 79.2 | 60.2 | 78.3
MiB+NeST (Ours) | Swin-B | 65.2 | 35.8 | 51.2 | 77.0 | 53.3 | 71.4 | 81.2 | 67.4 | 77.9 | 79.7 | 60.0 | 78.8
PLOP* [16] | Swin-B | 37.8 | 23.1 | 30.8 | 74.1 | 52.1 | 68.9 | 80.1 | 68.1 | 77.2 | 77.0 | 65.8 | 76.4
PLOP+NeST (Ours) | Swin-B | 64.3 | 28.3 | 47.2 | 76.8 | 57.2 | 72.2 | 80.5 | 70.8 | 78.2 | 79.6 | 70.2 | 79.1

###  4.2 Results

In this section, we apply NeST to three classic methods, MiB [5], PLOP [16]
and RCIL [55] with different backbones.

Pascal VOC 2012. We conduct experiments on _10-1_ , _15-1_ , _15-5_ and _19-1_
settings. As shown in Tab. 1, with NeST, MiB [5], PLOP [16] and RCIL [55] can
achieve huge performance gains. On the _15-1_ setting with Deeplab, by simply
using the proposed new classifier strategy, we can improve the performance of
MiB, PLOP, and RCIL by 13.6%, 6.9%, and 2.5%, respectively. In the more
difficult _10-1_ scenario, NeST can boost the performance of MiB by a large
margin, achieving an improvement of 27.2%. Meanwhile, results with Swin-B show
that NeST is also suitable for Transformer-based models. On MiB with Swin-B,
NeST can improve the performance by 34.5% on the _15-1_ setting and 36.2% on
the _10-1_ setting. In Tab. 1, the results across all settings for old and new
classes demonstrate that NeST significantly enhances stability by aligning new
classifiers with the existing backbone through the pre-tuning process, thereby
mitigating the forgetting of old knowledge. We also report the mIoU of our
method and _baselines_ at each step. As shown in Fig. 3, The performance of
our method surpasses _baselines’_ performance during the whole training
process.

(a) _15-1_ on PASCAL VOC 2012

(b) _10-1_ on PASCAL VOC 2012

Figure 3:  The mIoU (%) at each step for the setting _15-1_ (a) and _10-1_
(b).  Table 2: The mIoU (%) of the last step on the ADE20K dataset for four
different CISS scenarios. * implies results from the re-implementation of the
official code.

Method | Backbone | 100-50(2 steps) | 100-10(6 steps) | 100-5(11 steps) | 50-50(3 steps)
---|---|---|---|---|---
0-100 | 101-150 | all | 0-100 | 101-150 | all | 0-100 | 101-150 | all | 0-50 | 51-150 | all
Joint [55] | Res101 | 44.3 | 28.2 | 38.9 | 44.3 | 28.2 | 38.9 | 44.3 | 28.2 | 38.9 | 51.1 | 33.3 | 38.9
ILT [37] | Res101 | 18.3 | 14.8 | 17.0 | 0.1 | 2.9 | 1.1 | 0.1 | 1.3 | 0.5 | 13.6 | 6.2 | 9.7
PLOP+UCD [53] | Res101 | 42.1 | 15.8 | 33.3 | 40.8 | 15.2 | 32.3 | - | - | - | 47.1 | 24.1 | 31.8
SPPA [31] | Res101 | 42.9 | 19.9 | 35.2 | 41.0 | 12.5 | 31.5 | - | - | - | 49.8 | 23.9 | 32.5
MiB+AWT [21] | Res101 | 40.9 | 24.7 | 35.6 | 39.1 | 21.3 | 33.2 | 38.6 | 16.0 | 31.1 | 46.6 | 26.9 | 33.5
ALIFE [39] | Res101 | 42.2 | 23.1 | 35.9 | 41.0 | 22.8 | 35.0 | - | - | - | 49.0 | 25.7 | 33.6
GSC [14] | Res101 | 42.4 | 19.2 | 34.8 | 40.8 | 17.6 | 32.6 | 39.5 | 11.2 | 30.2 | 46.2 | 26.2 | 33.0
MiB [5] | Res101 | 40.5 | 17.2 | 32.8 | 38.2 | 11.1 | 29.2 | 36.0 | 5.6 | 25.9 | 45.6 | 21.0 | 29.3
MiB* [5] | Res101 | 40.5 | 23.5 | 34.9 | 37.8 | 12.1 | 29.3 | 35.8 | 6.0 | 25.9 | 45.9 | 23.9 | 31.3
MiB+NeST (Ours) | Res101 | 40.3 | 24.6 | 35.1 | 40.2 | 20.6 | 33.7 | 39.9 | 18.0 | 32.7 | 45.6 | 26.8 | 33.2
PLOP [16] | Res101 | 41.9 | 14.9 | 32.9 | 40.5 | 13.6 | 31.6 | 39.1 | 7.8 | 28.7 | 48.8 | 21.0 | 30.4
PLOP* [16] | Res101 | 42.2 | 15.3 | 33.3 | 41.0 | 13.7 | 32.0 | 39.6 | 8.2 | 29.2 | 48.5 | 20.8 | 30.2
PLOP+NeST (Ours) | Res101 | 42.2 | 24.3 | 36.3 | 40.9 | 22.0 | 34.7 | 39.3 | 17.4 | 32.0 | 48.7 | 27.7 | 34.8
RCIL [55] | Res101 | 42.3 | 18.8 | 34.5 | 39.3 | 17.6 | 32.1 | 38.5 | 11.5 | 29.6 | 48.3 | 25.0 | 32.5
RCIL* [55] | Res101 | 42.3 | 16.5 | 33.8 | 39.9 | 15.7 | 31.9 | 39.1 | 12.2 | 30.2 | 48.2 | 23.6 | 31.9
RCIL+NeST (Ours) | Res101 | 42.3 | 22.8 | 35.8 | 40.7 | 19.0 | 33.5 | 39.4 | 15.5 | 31.5 | 48.2 | 27.4 | 34.4
Joint | Swin-B | 43.4 | 31.9 | 39.6 | 43.4 | 31.9 | 39.6 | 43.4 | 31.9 | 39.6 | 50.7 | 33.9 | 39.6
MiB* | Swin-B | 42.7 | 26.1 | 37.2 | 40.2 | 15.0 | 31.8 | 39.1 | 8.6 | 29.0 | 48.3 | 26.8 | 34.1
MiB+NeST (Ours) | Swin-B | 42.8 | 27.8 | 37.9 | 41.8 | 23.8 | 35.9 | 40.5 | 19.9 | 33.7 | 49.7 | 29.3 | 36.2
PLOP* | Swin-B | 43.4 | 17.1 | 34.7 | 41.4 | 17.7 | 33.6 | 39.7 | 13.6 | 31.0 | 50.5 | 24.1 | 33.0
PLOP+NeST (Ours) | Swin-B | 43.5 | 26.5 | 37.9 | 41.7 | 24.2 | 35.9 | 39.7 | 18.3 | 32.6 | 50.6 | 28.9 | 36.2

ADE20K. To further evaluate NeST, we conduct experiments on the more
challenging settings of the ADE20K dataset. Results of _100-50_ , _100-10_ ,
_100-5_ and _50-50_ settings are shown in Tab. 2. NeST outperforms other
competing methods. On the most difficult _100-5_ setting, NeST improves MiB
[5], PLOP [16] and RCIL [55] by 6.8%, 2.8% and 1.3%, which indicates that NeST
is also applicable to scenarios with a large number of classes.

(a) feature map similarity

(b) training loss

Figure 4: The feature map similarity and training loss on Pascal VOC 2012
_15-1_ overlapped setting.

Effectiveness of Our Method. It is crucial for pre-tuning new classifiers
before the formal training step, not only for learning new classes but also
for bridging the _stability gap_ [15], which means that the model will undergo
a short period of forgetting when the task changes and then gradually recover.
This phenomenon is also discovered in CISS scenarios [2, 50]. At the beginning
of each formal training step, a well-tuned new classifier can make the loss
smaller, thus there will also be less impact on other model parameters as the
gradient values become smaller. As the model’s parameters are inherited from
the old model, the old knowledge learned from previous steps can be preserved.
We conduct experiments on the _15-1_ setting with the _baseline_ MiB [5] and
MiB+NeST. For a fair and plain comparison, we plot losses and cosine feature
similarities by the mean and standard deviation of each epoch at the formal
training process of the second step, while in the first step, we use the
original MiB for all experiments. As shown in Fig. 4(a), the feature
similarity of NeST exceeds that of MiB throughout the entire recovery process,
demonstrating the ability of NeST to help bridge the stability gap in the
model. Fig. 4(b) shows that during the formal training process, the loss of
NeST is lower than the loss of MiB, helping the model converge faster and
learn better.

###  4.3 Ablation Study

Different Classifier Initialization Methods. We compare NeST with other
initialization ways. As shown in Tab. 3, random initialization gets the worst
performance because of the misalignment in the early stages of training.
Initialization by the background classifier can get better performance.
Directly tuning the weight of new classifiers initialized by the background
classifier before the formal training step can bring a slight increase in
performance. Unlike the aforementioned approaches, NeST enables new
classifiers to adapt to training data and enhances the performance of the
_baseline_ by 16.5% on old classes and 4.7% on new classes.

Table 3: Ablation study of different classifier initialization strategies. The performance is reported on Pascal VOC 2012 15-1 overlapped setting with MiB [5]. Method | 0-15 | 16-20 | all
---|---|---|---
Random | 43.5 | 4.2 | 34.1
Background [5] | 45.2 | 15.7 | 38.2
Two-Stage | 46.0 | 15.3 | 38.7
NeST (Ours) | 61.7 | 20.4 | 51.8
Table 4: Comparison between different transformation matrix initialization methods. The performance is reported on Pascal VOC 2012 15-1 overlapped setting.  Method | 0-15 | 16-20 | all
---|---|---|---
Random Matrix Initialization | 53.8 | 7.5 | 42.8
NeST (Ours) | 61.7 | 20.4 | 51.8

Transformation Matrices Initialization. According to Tab. 4, if we use random
initialization instead of our designed strategy, the overall performance will
still be higher than the performance of _baseline_ [5]. However, the mIoU of
new classes learned in incremental steps is significantly lower than
_baseline_ ’s performance. It means that if we random initialize importance
matrices and projection matrices, it will only help to maintain the stability
of the model. The second row in Tab. 4 shows that after initializing
importance matrices and projection matrices using our proposed strategy
increases the performance of new classes to 20.4%, while also improving the
performance of the old classes. The design of initial values for matrices used
in the transformation can facilitate the model’s learning of new classes.
Thus, by initializing matrices with our strategy, NeST can achieve the trade-
off between stability and plasticity.

Figure 5:  Qualitative comparisons on Pascal VOC 2012 _15-1_ overlapped
setting.

Component-Level Ablation Study. We conduct a component-level ablation study on
two kinds of matrices, as shown in Tab. 5. Eliminating the importance matrix
means that we set values in the importance matrix to 1 and freeze it, _i.e_.,
only learning a combination of old classifiers to get new classifiers.
Similarly, eliminating the projection matrix indicates that we average the
weighted old classifiers to generate new classifiers. Experimental results
demonstrate that with equally treated channels, the performance dramatically
drops due to the unawareness of different channels.

Table 5: Component level ablation study on Pascal VOC 2012 _15-1_ overlapped setting. Method | Projection Matrix | Importance Matrix | _0-15_ | _16-20_ | _all_
---|---|---|---|---|---
_baseline_ |  |  | 45.2 | 15.7 | 38.2
Variants | ✓ |  | 53.3 | 15.4 | 44.3
|  | ✓ | 59.7 | 19.2 | 50.1
| ✓ | ✓ | 61.7 | 20.4 | 51.8

Computational Costs. The number of extra parameters can be represented as
$n_{new}\times n_{old}\times(d+1)+d+n_{new}+1$ and $n_{old}$ increases
linearly with steps while $n_{new}$ is always fixed. The additional FLOPs
(about 15.6K) and parameters (about 5.4K) are negligible during the pre-tuning
process of Step 5 on the _15-1_ setting. The whole training process of
MiB+NeST takes an additional 2.7% (3.7 minutes) of the total training time
(2.3 hours). Note that after pre-tuning, extra parameters are removed and the
formal training process is the same as MiB.

Discussion on AWT v.s. NeST. AWT [21] utilizes the new training data and old
background classifier to generate new classifiers. Here we discuss the
differences between AWT and NeST. AWT employs gradient-based attribution to
transfer the most relevant weights to new classifiers. However, it treats each
new class equally, neglecting the role of other old classifiers and the
differences between new classes. Meanwhile, the gradient-based attribution
technique introduces a huge memory cost, AWT can not run on RTX 3090 even if
the batch size is set to 1. NeST is based on learning, which can better make
the generated classifiers’ weight align with the backbone. Moreover, as shown
in Tab. 6, the memory cost is acceptable, and NeST is a little faster than
AWT.

Table 6: The memory costs and additional training time on Pascal VOC 2012 _15-1_ overlapped setting.  Method | Memory per GPU | Additional Time
---|---|---
MiB+AWT | $>24{\rm GB}$ | $6\%$
MiB+NeST (Ours) | $\boldsymbol{6.47{\rm GB}}$ | $\boldsymbol{2.7\%}$

Robustness of Different Class Orders. In CISS scenarios, the class order is
crucial as the learning process of a class may boost or damage the performance
of another class. To validate the robustness of NeST, we conduct experiments
on the _15-1_ setting with five different class orders. As shown in Tab. 7,
the performance of NeST is higher than the performance of _baseline_.

Table 7: The average performance of five different class orders on Pascal VOC 2012 _15-1_ overlapped setting.  Method | A | B | C | D | E | avg
---|---|---|---|---|---|---
MiB | 38.2 | 28.2 | 38.5 | 38.0 | 52.0 | 39.0
MiB+NeST (Ours) | 51.8 | 43.4 | 50.4 | 60.0 | 59.8 | 51.5

###  4.4 Visualization

In Fig. 5, we show the visualization results of our proposed method based on
MiB [5]. Samples are selected from step 0 and from step 5 to show the
stability and plasticity of our NeST. In the first two rows, classes _person_
and _horse_ are learned in step 0, then in the following steps, the _baseline_
MiB [5] gradually forget concepts learned in step 0, while NeST helps the
model to preserve old knowledge. In the last two rows, class _chair_ is
learned in step 0 and class _tv/monitor_ is learned in step 5. After learning
class _train_ , MiB has almost forgotten the knowledge of class _chair_ ,
while NeST can give correct predictions for most of the pixels. In the last
step, NeST can help the model learn new class _tv/monitor_ better than MiB,
showing the plasticity of our method.

##  5 Conclusions

In this work, we propose a simple yet effective new classifier pre-tuning
method that can enhance the CISS ability by learning a transformation from old
classifiers to new classifiers. We further find a way to initialize matrices
by utilizing the information of cross-task class similarities between old
classes and new classes, helping the model achieve the stability-plasticity
trade-off. Experiments on two datasets show that NeST can significantly
improve the performance of _baselines_ , and it can be easily applied to other
CISS methods. While our approach can lead to huge performance gains, the
limitation is that it introduces additional computational overhead. In future
work, we will try our method in other continual learning tasks.

## Acknowledgments

This work is funded by NSFC (NO. 62206135), Key Program for International
Cooperation of Ministry of Science and Technology, China (NO. 2024YFE0100700),
Young Elite Scientists Sponsorship Program by CAST (NO. 2023QNRC001), Tianjin
Natural Science Foundation (NO. 23JCQNJC01470), and the Fundamental Research
Funds for the Central Universities (Nankai University). Computation is
supported by the Supercomputing Center of Nankai University.

## References

  * [1] Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., Tuytelaars, T.: Memory aware synapses: Learning what (not) to forget. In: Eur. Conf. Comput. Vis. pp. 139–154 (2018)
  * [2] Baek, D., Oh, Y., Lee, S., Lee, J., Ham, B.: Decomposed knowledge distillation for class-incremental semantic segmentation. In: Adv. Neural Inform. Process. Syst. (2022)
  * [3] Bang, J., Kim, H., Yoo, Y., Ha, J.W., Choi, J.: Rainbow memory: Continual learning with a memory of diverse samples. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 8218–8227 (2021)
  * [4] Cermelli, F., Cord, M., Douillard, A.: Comformer: Continual learning in semantic and panoptic segmentation. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 3010–3020 (2023)
  * [5] Cermelli, F., Mancini, M., Bulo, S.R., Ricci, E., Caputo, B.: Modeling the background for incremental learning in semantic segmentation. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 9233–9242 (2020)
  * [6] Cha, S., Yoo, Y., Moon, T., et al.: Ssul: Semantic segmentation with unknown label for exemplar-based class-incremental learning. In: Adv. Neural Inform. Process. Syst. vol. 34, pp. 10919–10930 (2021)
  * [7] Chaudhry, A., Dokania, P.K., Ajanthan, T., Torr, P.H.: Riemannian walk for incremental learning: Understanding forgetting and intransigence. In: Eur. Conf. Comput. Vis. pp. 532–547 (2018)
  * [8] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 834–848 (2017)
  * [9] Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Eur. Conf. Comput. Vis. pp. 801–818 (2018)
  * [10] Chen, Y.H., Chen, W.Y., Chen, Y.T., Tsai, B.C., Frank Wang, Y.C., Sun, M.: No more discrimination: Cross city adaptation of road scene segmenters. In: Int. Conf. Comput. Vis. pp. 1992–2001 (2017)
  * [11] Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 1290–1299 (2022)
  * [12] Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 1290–1299 (2022)
  * [13] Cheng, B., Schwing, A., Kirillov, A.: Per-pixel classification is not all you need for semantic segmentation. In: Adv. Neural Inform. Process. Syst. vol. 34, pp. 17864–17875 (2021)
  * [14] Cong, W., Cong, Y., Dong, J., Sun, G., Ding, H.: Gradient-semantic compensation for incremental semantic segmentation. IEEE Trans. Multimedia (2023)
  * [15] De Lange, M., van de Ven, G., Tuytelaars, T.: Continual evaluation for lifelong learning: Identifying the stability gap. arXiv preprint arXiv:2205.13452 (2022)
  * [16] Douillard, A., Chen, Y., Dapogny, A., Cord, M.: Plop: Learning without forgetting for continual semantic segmentation. In: IEEE Conf. Comput. Vis. Pattern Recog. (2021)
  * [17] Douillard, A., Cord, M., Ollion, C., Robert, T., Valle, E.: Podnet: Pooled outputs distillation for small-tasks incremental learning. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 86–102. Springer (2020)
  * [18] Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html (2012)
  * [19] Gao, S.H., Cheng, M.M., Zhao, K., Zhang, X.Y., Yang, M.H., Torr, P.: Res2net: A new multi-scale backbone architecture. IEEE Trans. Pattern Anal. Mach. Intell. 43(2), 652–662 (2019)
  * [20] Geng, C., Huang, S.j., Chen, S.: Recent advances in open set recognition: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 43(10), 3614–3631 (2020)
  * [21] Goswami, D., Schuster, R., van de Weijer, J., Stricker, D.: Attribution-aware weight transfer: A warm-start initialization for class-incremental semantic segmentation. In: WACV. pp. 3195–3204 (2023)
  * [22] Grossberg, S.T.: Studies of mind and brain: Neural principles of learning, perception, development, cognition, and motor control, vol. 70. Springer Science & Business Media (2012)
  * [23] Guo, M.H., Lu, C.Z., Hou, Q., Liu, Z., Cheng, M.M., Hu, S.M.: Segnext: Rethinking convolutional attention design for semantic segmentation. In: Adv. Neural Inform. Process. Syst. vol. 35, pp. 1140–1156 (2022)
  * [24] Hadsell, R., Rao, D., Rusu, A.A., Pascanu, R.: Embracing change: Continual learning in deep neural networks. Trends in cognitive sciences 24(12), 1028–1040 (2020)
  * [25] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conf. Comput. Vis. Pattern Recog. (2016)
  * [26] Hu, Z., Li, Y., Lyu, J., Gao, D., Vasconcelos, N.: Dense network expansion for class incremental learning. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 11858–11867 (2023)
  * [27] Hung, C.Y., Tu, C.H., Wu, C.E., Chen, C.H., Chan, Y.M., Chen, C.S.: Compacting, picking and growing for unforgetting continual learning. In: Adv. Neural Inform. Process. Syst. vol. 32 (2019)
  * [28] Kim, D., Han, B.: On the stability-plasticity dilemma of class-incremental learning. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 20196–20204 (2023)
  * [29] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al.: Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114(13), 3521–3526 (2017)
  * [30] Li, Z., Hoiem, D.: Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell. 40(12), 2935–2947 (2017)
  * [31] Lin, Z., Wang, Z., Zhang, Y.: Continual semantic segmentation via structure preserving and projected feature alignment. In: Eur. Conf. Comput. Vis. pp. 345–361. Springer (2022)
  * [32] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Int. Conf. Comput. Vis. pp. 10012–10022 (2021)
  * [33] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 3431–3440 (2015)
  * [34] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 3431–3440 (2015)
  * [35] Maracani, A., Michieli, U., Toldo, M., Zanuttigh, P.: Recall: Replay-based continual learning in semantic segmentation. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 7026–7035 (2021)
  * [36] McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: The sequential learning problem. In: Psychology of learning and motivation, vol. 24, pp. 109–165. Elsevier (1989)
  * [37] Michieli, U., Zanuttigh, P.: Incremental learning techniques for semantic segmentation. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 0–0 (2019)
  * [38] Michieli, U., Zanuttigh, P.: Continual semantic segmentation via repulsion-attraction of sparse and disentangled latent representations. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 1114–1124 (2021)
  * [39] Oh, Y., Baek, D., Ham, B.: Alife: Adaptive logit regularizer and feature replay for incremental semantic segmentation. In: Adv. Neural Inform. Process. Syst. vol. 35, pp. 14516–14528 (2022)
  * [40] Ranftl, R., Bochkovskiy, A., Koltun, V.: Vision transformers for dense prediction. In: Int. Conf. Comput. Vis. pp. 12179–12188 (2021)
  * [41] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI. pp. 234–241. Springer (2015)
  * [42] Shang, C., Li, H., Meng, F., Wu, Q., Qiu, H., Wang, L.: Incrementer: Transformer for class-incremental semantic segmentation with knowledge distillation focusing on old class. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 7214–7224 (2023)
  * [43] Shin, H., Lee, J.K., Kim, J., Kim, J.: Continual learning with deep generative replay. In: Adv. Neural Inform. Process. Syst. vol. 30 (2017)
  * [44] Strudel, R., Garcia, R., Laptev, I., Schmid, C.: Segmenter: Transformer for semantic segmentation. In: Int. Conf. Comput. Vis. pp. 7262–7272 (2021)
  * [45] Verwimp, E., De Lange, M., Tuytelaars, T.: Rehearsal revealed: The limits and merits of revisiting samples in continual learning. In: Int. Conf. Comput. Vis. pp. 9385–9394 (2021)
  * [46] Vinogradova, K., Dibrov, A., Myers, G.: Towards interpretable semantic segmentation via gradient-weighted class activation mapping (student abstract). In: AAAI. pp. 13943–13944 (2020)
  * [47] Wang, E., Peng, Z., Xie, Z., Yang, F., Liu, X., Cheng, M.M.: Unlocking the multi-modal potential of clip for generalized category discovery (2024), https://arxiv.org/abs/2403.09974
  * [48] Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., et al.: Deep high-resolution representation learning for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 43(10), 3349–3364 (2020)
  * [49] Wu, Y., Chen, Y., Wang, L., Ye, Y., Liu, Z., Guo, Y., Fu, Y.: Large scale incremental learning. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 374–382 (2019)
  * [50] Xiao, J.W., Zhang, C.B., Feng, J., Liu, X., van de Weijer, J., Cheng, M.M.: Endpoints weight fusion for class incremental semantic segmentation. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 7204–7213 (2023)
  * [51] Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: Segformer: Simple and efficient design for semantic segmentation with transformers. Adv. Neural Inform. Process. Syst. 34, 12077–12090 (2021)
  * [52] Yan, S., Xie, J., He, X.: Der: Dynamically expandable representation for class incremental learning. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 3014–3023 (2021)
  * [53] Yang, G., Fini, E., Xu, D., Rota, P., Ding, M., Nabi, M., Alameda-Pineda, X., Ricci, E.: Uncertainty-aware contrastive distillation for incremental semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 45(2), 2567–2581 (2022)
  * [54] Yoon, J., Yang, E., Lee, J., Hwang, S.J.: Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547 (2017)
  * [55] Zhang, C.B., Xiao, J.W., Liu, X., Chen, Y.C., Cheng, M.M.: Representation compensation networks for continual semantic segmentation. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 7053–7064 (2022)
  * [56] Zhang, Z., Gao, G., Jiao, J., Liu, C.H., Wei, Y.: Coinseg: Contrast inter-and intra-class representations for incremental segmentation. In: Int. Conf. Comput. Vis. pp. 843–853 (2023)
  * [57] Zhang, Z., Zhang, X., Peng, C., Xue, X., Sun, J.: Exfuse: Enhancing feature fusion for semantic segmentation. In: Eur. Conf. Comput. Vis. pp. 269–284 (2018)
  * [58] Zhao, B., Xiao, X., Gan, G., Zhang, B., Xia, S.T.: Maintaining discrimination and fairness in class incremental learning. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 13208–13217 (2020)
  * [59] Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P.H., et al.: Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 6881–6890 (2021)
  * [60] Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 633–641 (2017)
  * [61] Zhu, F., Zhang, X.Y., Wang, C., Yin, F., Liu, C.L.: Prototype augmentation and self-supervision for incremental learning. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 5871–5880 (2021)

##  Appendix 0.A Baseline Details

In this section, we introduce the _baselines_ used in experiments.

MiB. In MiB [5], two kinds of loss are used to model the background, _i.e_.,
$\mathcal{L}_{unce}$ and $\mathcal{L}_{unkd}$:

$\displaystyle\mathcal{L}_{unce}=-\frac{1}{\left|\mathcal{I}\right|}\sum_{i\in\mathcal{I}}\log\tilde{q}_{x}^{t}(i,y_{i})\,,$
(9)
$\displaystyle\mathcal{L}_{unkd}=-\frac{1}{\left|\mathcal{I}\right|}\sum_{i\in\mathcal{I}}\sum_{c\in\bigcup_{j=1}^{t-1}\mathcal{C}_{j}}q_{x}^{t-1}(i,c)\log\hat{q}_{x}^{t}(i,c)\,,$

where $\mathcal{I}$ denotes the pixel set of an image, $y_{i}\in
c_{bg}\cup\mathcal{C}_{t}$ denotes the ground-truth label of pixel $i$,
$q_{x}^{t}$ denotes the output of the model at step $t$, $\tilde{q}_{x}^{t}$
and $\hat{q}_{x}^{t}$ denotes the modified output of the current model,
considering the old classes for the cross entropy loss and new classes for the
knowledge distillation loss.

PLOP. Different from MiB [5], PLOP [16] utilizes pseudo-labeling to address
the issue of background shift, as follows:

$\displaystyle\mathcal{L}_{pseudo}=-\frac{\nu}{WH}\sum_{w,h}^{W,H}\sum_{c\in\mathcal{C}_{t}}\tilde{S}(w,h,c)\log\hat{S}^{t}(w,h,c)\,,$
(10)

where $\hat{S}$ denotes the prediction of the model and $\tilde{S}$ denotes
the pseudo-labels generated by the old model in the previous step.

It also distills intermediate features by Local POD, as follows:

$\displaystyle\mathcal{L}_{LocalPod}=\frac{1}{L}\sum_{l=1}^{L}\left|\left|\Phi(f_{l}^{t}(I))-\Phi(f_{l}^{t-1}(I))\right|\right|\,,$
(11)

where $L$ denotes the number of layers, $\Phi$ denotes the operation of Local
POD embedding extraction, $f_{l}^{t}(I)$ denotes the output feature from the
layer $l$ with the input $I$.

RCIL. RCIL [55] decouple the remembering of old knowledge and the learning of
new knowledge by adding a parallel module composed of a convolution layer and
a normalization layer for each $3\times 3$ convolution module. At step 0, all
parameters are trainable. At the beginning of each incremental step, two
branches of the old model are fused into one frozen branch to memorize the old
knowledge, while the other branch is learnable. A drop path strategy is also
used when fusing the outputs of two branches, which can be denoted as:

$\displaystyle x_{out}=\eta\cdot x_{1}+(1-\eta)\cdot x_{2}\,,$ (12)

where $x_{out}$ denotes the fused output, $x_{1}$ and $x_{2}$ denotes the
outputs from two branches, and $\eta$ denotes a channel-wise weight vector.
For training process, $\eta$ is sampled from the set $\\{0,0.5,1\\}$ and for
evaluation $\eta$ is set to 0.5. RCIL also proposed a Pooled Cube Knowledge
Distillation, using average pooling operation on spatial and channel
dimensions.

##  Appendix 0.B Results and Analysis of Disjoint Settings

In _Disjoint_ settings, at each step, the _bg_ classifier will not see any
future class, leading to MiB’s initialization struggling in these settings,
while our NeST leverages semantic knowledge from old classifiers to generate
new classifiers for initialization, the pre-tuning process also benefits the
stability of the model. Results of NeST and _baselines_ on _15-1 Disjoint_ and
_10-1 Disjoint_ settings are shown in Tab. 8, indicating that NeST can
significantly improve the performance of previous methods in _Disjoint_
settings.

Table 8: Results of disjoint settings on Pascal VOC 2012 dataset. Method | _15-1 Disjoint_ | _10-1 Disjoint_
---|---|---
MiB | 38.6 | 2.0
MiB+NeST | 41.0 | 20.5
PLOP | 40.7 | 12.6
PLOP+NeST | 52.7 | 22.4

##  Appendix 0.C Experiments of COCO-Stuff 10K

To prove the ability to apply our NeST in scenarios with more classes, we
introduce another dataset for class incremental semantic segmentation, COCO-
Stuff 10K, to evaluate the effectiveness of our method. COCO-Stuff 10K
includes 80 thing classes and 91 stuff classes, which is a subset of the
original COCO-Stuff dataset. We evaluate our NeST on the _80-91_ (2 Steps)
overlapped setting, which contains more classes than ADE20K and Pascal VOC
2012. As shown in Tab. 9, our method can handle scenarios with more classes in
one step.

Table 9: Results of _80-91_ overlapped setting on COCO-Stuff 10K dataset. Method | _0-80_ | _81-171_ | _all_
---|---|---|---
MiB | 40.2 | 18.6 | 28.9
MiB+NeST | 41.9 | 20.7 | 30.7
PLOP | 46.1 | 17.0 | 30.8
PLOP+NeST | 46.0 | 18.8 | 31.6

##  Appendix 0.D Comparisons with Transformer-based SOTA Methods

In recent years, many Transformer-based CISS methods have emerged, here we
briefly discuss the differences between NeST and these methods. Comformer [4]
uses a universal segmentation model Mask2Former [12] to do mask classification
for continual panoptic segmentation and continual semantic segmentation.
CoinSeg [56] introduces a pretrained Mask2Former model as class-agnostic mask
generator. Incrementer [42] sequentially adds tokens of new classes and
performs dot production between features and updated class tokens to generate
segmentation prediction results. The _baseline_ is based on a simple per-pixel
classification model SETR with ViT-B as the backbone, while equipped with
NeST, it can achieve SOTA performances, as shown in Tab. 10. Moreover, NeST
has the potential to be integrated into these transformer-based methods, and
we leave it as our future work.

Table 10: Comparisons with Transformer-base SOTA methods. Method | Backbone | Model | _15-1_ | _15-5_ | _10-1_
---|---|---|---|---|---
CoinSeg | Swin-B | Deeplab+Mask2Former | 75.5 | 77.6 | 70.5
Incrementer | ViT-B | Segmenter | 75.5 | 79.9 | 70.2
MiB | ViT-B | SETR | 53.3 | 80.2 | 25.5
MiB+NeST (Ours) | ViT-B | SETR | 76.5 | 80.3 | 71.9

##  Appendix 0.E More Implementation Details

Weight Align. To prevent the new classifiers’ weight from being too large
during the pre-tuning process, we apply Weight Aligning (WA) [58] as follows:

$\displaystyle\hat{w}_{new}=w_{new}\cdot\frac{Mean(Norm_{old})}{Mean(Norm_{new})}$
(13)

where $Norm_{old}$ and $Norm_{new}$ denote norms of old and new classifiers’
weights, $Mean(\cdot)$ denotes the operation of calculating mean values.
Relevant experiment results in Tab. 11 show that WA can correct the biased
weight thus boosting the performance of NeST.

Table 11:  Ablation study of Weight Align for NeST. All performances are reported on the _15-1_ setting.  Method | 0-15 | 16-20 | all
---|---|---|---
MiB+NeST w/o WA | 58.4 | 10.9 | 47.1
MiB+NeST w/ WA | 61.7 | 20.4 | 51.8
PLOP+NeST w/o WA | 72.5 | 32.4 | 62.9
PLOP+NeST w/ WA | 72.2 | 33.7 | 63.1

Fix old classifiers.  We find that the pseudo-labeling strategy may change the
geometric structure of old classifiers severely, which has a detrimental
impact on our method. This phenomenon is particularly obvious on the Pascal
VOC 2012 dataset. To preserve the old knowledge learned in previous steps,
following EWF [50], we fix old classifiers in the formal training steps on
settings of Pascal VOC 2012. Relevant experiment results are shown in Tab. 12.

Table 12:  Ablation study of fixing previous classifiers for our method based on PLOP [16]. All performances are reported on the _15-1_ setting.  Method | 0-15 | 16-20 | all
---|---|---|---
PLOP w/ fix | 56.9 | 11.3 | 46.0
PLOP+NeST w/o fix | 66.8 | 20.2 | 55.7
PLOP+NeST w/ fix | 72.2 | 33.7 | 63.1

##  Appendix 0.F Further Analysis

Figure 6:  Visualization of the importance matrix on ADE20K _100-5_ step1.

Effectiveness of the importance matrix. For plasticity, we learn to generate a
new classifier with relevant old classifiers. In particular, the importance
matrix can capture the semantic relationship between old and new classifiers
on the channel level. To verify this, we visualize the importance matrix
$M\in\mathbb{R}^{100\times 256}$ of the new class _van_ on ADE20K _100-5_
step1. We normalized the absolute values to [0, 1] and a lighter color means a
higher value. As shown in Fig. 6, the bg class (row: 0) and car class (row:
21) make the largest contributions. It is intuitive, as the class van may
appear in old data, labeled as bg, and van and old class car are closer in
semantic relationship.

Figure 7:  More qualitative results. All experiments are conducted on the
_15-1_ setting.  Figure 8:  More qualitative results. All experiments are
conducted on the _15-1_ setting.

Different class orders. To evaluate the effectiveness of our method, following
PLOP [16], we use five different class orders of Pascal VOC 2012 _15-1_
overlapped setting, as follows:

$\displaystyle A:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]\,,$
(14) $\displaystyle
B:[0,12,9,20,7,15,8,14,16,5,19,4,1,13,2,11,17,3,6,18,10]\,,$ $\displaystyle
C:[0,13,19,15,17,9,8,5,20,4,3,10,11,18,16,7,12,14,6,1,2]\,,$ $\displaystyle
D:[0,15,3,2,12,14,18,20,16,11,1,19,8,10,7,17,6,5,13,9,4]\,,$ $\displaystyle
E:[0,7,5,3,9,13,12,14,19,10,2,1,4,16,8,17,15,18,6,11,20]\,.$

More qualitative results. More qualitative results are shown in Fig. 7 and
Fig. 8. By applying the pre-tuning process, our method can help the model
preserve old knowledge.

Moreover, to validate the effectiveness of the matrix initialization, we also
visualize the class activation map for the last class _tv/ monitor_. To
visualize segmentation CAMs, we adopt the method proposed in [46]. As shown in
Fig. 9, with the designed matrix initialization strategy, the model can pay
more attention to areas of new classes.

Figure 9:  Class activation maps for the last class _tv/monitor_ on the _15-1_
setting $w\\!/\\!o$ (the top row) and $w\\!/\\!$ (the bottom row) our matrix
initialization strategy.


# Histogram Layers for Synthetic Aperture Sonar Imagery
††thanks: This material is based upon work supported by the National Science
Foundation Graduate Research Fellowship under Grant No. DGE-1842473 and by the
Office of Naval Research Grant N00014-16-1-2323. The views and opinions of
authors expressed herein do not necessarily state or reflect those of the
United States Government or any agency thereof. Distribution Statement A:
Approved for public release: distribution is unlimited.

Joshua Peeples1, Alina Zare2, Jeffrey Dale3,4, James Keller3 4Naval Surface
Warfare Center Panama City Division, 110 Vernon Ave, Panama City, FL 32407,
USA
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>3Department of Computer Science and Electrical
Engineering, University of Missouri, Columbia, MO, USA
2Department of Electrical and Computer Engineering, University of Florida,
Gainesville, FL, USA
1Department of Electrical and Computer Engineering, Texas A&M University,
College Station, TX, USA

###### Abstract

Synthetic aperture sonar (SAS) imagery is crucial for several applications,
including target recognition and environmental segmentation. Deep learning
models have led to much success in SAS analysis; however, the features
extracted by these approaches may not be suitable for capturing certain
textural information. To address this problem, we present a novel application
of histogram layers on SAS imagery. The addition of histogram layer(s) within
the deep learning models improved performance by incorporating statistical
texture information on both synthetic and real-world datasets.

###### Index Terms:

deep learning, histograms, texture analysis, SAS imagery

##  I Introduction

Synthetic aperture sonar (SAS) produces high resolution images of the seafloor
[1]. The imagery generated through SAS can be used to perform important tasks
such as automatic target recognition [2], environmental segmentation [3], and
automated scene understanding [4]. Machine learning methods have been applied
to SAS imagery to improve the accuracy and efficiency for these different
applications. A critical aspect of the machine learning methods is the
representation (i.e., features) of the SAS imagery. As a result, several works
investigate feature extraction approaches using a) handcrafted [5] and/or b)
deep learning [6] features. Deep learning models, particularly convolutional
neural networks (CNNs), have led to great strides in research for processing
SAS imagery [2, 4, 3, 6]. A main reason for this success is the ability of
deep learning frameworks to automate the feature extraction process and
perform follow-on tasks (e.g., classification, segmentation). Literature has
suggested that CNNs are biased towards texture features rather than shape [7,
8]. For SAS-related efforts, texture information is crucial to represent the
data [5]. CNNs are biased towards a particular type of texture (i.e.,
structural texture) [9, 10]. To improve the information extracted within these
deep learning models, other types of texture features can be incorporated into
the network.

Texture information is a common type of image feature that has led to much
success [11]. For example, texture can be used to provide environmental
context in sonar imagery [12, 13, 14] that can be used in various tasks such
as identifying potential hazards. In Figure 1, there are several difficult
aspects of the environment that texture features can capture. For example, the
seafloor types blend (e.g., sand ripple patterns change, overlap between
different seafloor types) and texture approaches can model these gradients and
regions of transition that occur within the SAS imagery. Texture is always
present within images [15]; therefore, the representation of texture content
within an image is an important component of texture analysis. Additionally,
the need for informative, domain-specific (e.g., SAS) texture information has
led to several methods for texture feature extraction, particularly histogram-
based approaches [16, 12, 17].

Figure 1: SAS image example for understanding the environmental context that
can be used in different tasks (e.g., object recognition). The seafloor
changes between different textures types and texture features can help capture
this information. The objects of interest in the image are marked by red
rectangles.

As a result of the success of histogram-based, handcrafted features, histogram
layer(s) have been proposed [9, 18, 19, 20, 21]. These methods bridge the gap
between handcrafted and deep learning approaches to improve the statistical
texture context within the model. Structural textures consist of defining a
set of texture examples and an order of spatial positions for each exemplar
[22]. On the other hand, statistical textures represent the data through
parameters that characterize the distributions and correlation between the
intensity and/or feature values in an image, as opposed to understanding the
structure of each texture [23]. Methods that focused on extracting statistical
texture, particularly those computing measures between pairs of pixels (i.e.,
second order), have outperformed the other texture analysis approaches [24,
25, 26]. To illustrate the difference between structural and statistical
textures, we show an example in Figure 2. The structural textures are a
checkboard, cross, and stripe displayed along the vertical axis of the figure.
The statistical textures are shown along the horizontal axis where the
foreground pixels values were sampled from a multinomial, binomial, and
constant distributions. CNNs would have difficulty learning weights that could
account for both the spatial ordering of the pixels (i.e., structural changes)
and individual pixel changes (i.e., statistical changes). However, histogram
layer(s) could be used to learn the statistical distributions to further
enhance the features learned by the model to correctly classify the textures
[9].

Inspired by the utility of statistical texture features, histogram layer(s)
can be used to provide a powerful representation of the data more suitable for
SAS imagery. We hypothesize that incorporating histogram layer(s) within deep
learning models will improve performance on SAS-related tasks. The
contributions of this work are the following:

  * •

First application of histogram layer(s) for SAS imagery

  * •

Investigation into model configuration for histogram layer(s)

  * •

Thorough analysis of statistical and structural textures for SAS imagery

Figure 2: Examples of different types of structural and statistical textures.
The structural and statistical textures vary along the vertical and horizontal
axis respectively.

##  II Method

###  II-A Histogram Layer Review

We briefly summarize the local, radial basis function (RBF) histogram layer
that was previously introduced [9]. The histogram layer can be applied to
several data modalities with spatial information (e.g., grayscale, RGB, and
convolutional feature maps). Histograms generally require three
hyperparameters: bin centers ($\gamma_{bd}$), bin widths ($\mu_{bd}$), and the
number of bins ($B$). With the histogram layer, the bin centers and widths are
learned by the network instead of setting these values a priori. Given input
data, $\mathbf{X}\in\mathbb{R}^{M\times N\times D}$, where $M$ and $N$ are the
spatial dimensions while $D$ is the feature dimensionality, the output tensor
of the local histogram layer, $\mathbf{Y}\in\mathbb{R}^{R\times C\times
B\times D}$ with spatial dimensions $R$ and $C$ after applying a histogram
layer with kernel size $S\times T$ is shown in Equation 1:

$Y_{rcbd}=\cfrac{1}{ST}\sum_{s=1}^{S}\sum_{t=1}^{T}e^{-\gamma_{bd}^{2}\left(x_{r+s,c+t,d}-\mu_{bd}\right)^{2}}.$
(1)

Each input feature dimension is treated independently; therefore, $BD$
histogram features maps are extracted (e.g., three input channels and three
bins would result in nine histogram feature maps). The histogram layer is
constructed using pre-existing layers that add flexibility and ease of
implementation.

###  II-B Network Architectures

(a) Resnet18 Backbone

(b) Parallel (c) Series

Figure 3: Convolutional backbone and histogram model configurations used for
multi-site dataset experiments. Resnet18 was used as the backbone architecture
to extract convolutional features. In Figure 3a, the convolutional, ReLU, and
max pooling layers are represented with the light yellow, orange, and purple
colors respectively. The local features maps produced by the histogram layer
are $2\times 2\times 128$. The histogram features are reshaped to be the same
size (i.e., $1\times 1\times 512$) as the feature vector produced by the
global average pooling layer (GAP).

We implemented a) shallow and b) deep networks to evaluate the statistical and
structural features of the SAS imagery. Following [9], the shallow models were
composed of a feature extractor (histogram or convolutional layer), global
average pooling layer (GAP), and output classification layer. The backbone
architecture for the deep networks was Resnet18 (Figure 3a). In this work, we
investigated series and parallel configuration (previously used in [9, 19]) as
shown in Figures 3b and 3c respectively. The series configuration would encode
only statistical texture information before the output classification layer.
The parallel configuration would consist of both statistical and structural
texture features. As a result, we are able to observe the impact of
structural, statistical, and combination of these texture features through the
performance of the baseline (i.e., ResNet18), series histogram, and parallel
histogram models respectively for SAS data.

##  III Experimental Procedure

###  III-A Dataset Description

#### Statistical PISAS Dataset

(a) T1_S1

(b) T1_S2

(c) T1_S3

(d) T10_S1

(e) T10_S2

(f) T10_S3

Figure 4: Example of sand ripple (“T1”) and rocky (“T10”) images from the
PISAS dataset with different statistical distributions. The same statistical
classes (multinomial, binomial, and constant) and two SAS-related structures
(i.e., sand ripple and rocky) were used to generate a total of six distinct
classes. The binomial, multinomial, and constant textures are referenced as
“S1”, “S2”, and “S3” respectively. The subset of PISAS consisted of a total of
$14,208$ images.

The Pseudo Image SAS (PISAS) dataset [4] contains ten classes of sand ripple
and rocky textures. Instead of the simple structures of cross, checkerboard,
and stripe as shown in Figure 2, one class of sand ripple and rocky textures
were selected to represent more SAS-related structures. Examples of sand
ripple and rocky textures in the three different statistical classes are shown
in Figure 4. The statistical PISAS dataset consisted of 14,208 images.

#### Multi-site Dataset

(a) Craters

(b) Flat

(c) Rocky

(d) Sandripple

Figure 5: Example of images from multi-site dataset [27]. The dataset
consisted of real-world SAS imagery pure texture regions from multiple
locations on the seafloor. Four seafloor textures types were identified to
train, validate and test the model: craters, flat, rocky, and sand ripple. The
multi-site dataset contained 118 total images.

Example images for the multi-site dataset [27] are shown in Figure 5. The
dataset consisted of 94 training, 12 validation, and 12 test images (118
images total) of pure textures from multiple locations along the seafloor.
Random patches were sampled from the validation and test splits five times to
generate 60 images for each set. A total of four classes comprised the
dataset: flat, rocky, sand ripple, and craters.

###  III-B Experimental Design

#### Shallow Networks

We followed a similar training procedure as in [9] for the synthetic dataset
experiments where each model was evaluated as a local feature extractor. The
network architecture consisted of a feature extractor (convolutional or
histogram layer), global average pooling layer, and fully connected output
layer. The “CNN” model has a convolutional layer with ReLU as the feature
extractor while the “Hist” model used the histogram layer as the feature
extractor. The experimental settings for each model was the following:
70/10/20 training, validation, and testing splits, 300 epochs, and early
stopping by checking if the validation loss did not decrease after 10
consecutive epochs. The input image

(a) CNN ($55.65\pm 21.0$)

(b) Hist ($81.11\pm 0.76$)

Figure 6: Average confusion matrices for the convolutional and histogram layer
models on the PISAS dataset. “T1” and “T10” are sand ripple and rocky textures
respectively. “S1”, “S2”, and “S3” are the binomial, multinomial, and constant
distributions respectively.

patches were $128\times 128$. The model parameters were adjusted so that the
kernel size was $7\times 7$ with a $3\times 3$ stride, and a batch size of
$128$ was used to train each model. The number of bins for the RBF histogram
layer model (i.e., Hist) and convolutional kernels for the CNN model were set
to three. The histogram layer bins were initialized to be equally spaced
between the range of the input data (i.e., $0$ and $1$). A total of three
experimental runs were performed.

#### Deep Networks

The HistRes_$B$ model [9] was applied to the multi-site dataset, and two
configurations were investigated: parallel and series. The training details
were the following: 100 epochs, learning rate of $0.001$, Adam optimization
[28], mini batch size of 16, $224\times 224$ input image patches, and random
crop data augmentation. A total of five experimental runs of random
initialization were completed to evaluate the stability of each method. The
number of bins was varied from $4$, $8$, and $16$.

###  III-C Results and Discussion

(a) CNN ($77.70$)

(b) Hist ($82.18$)

Figure 7: t-SNE projections for the convolutional and histogram layer models
on the PISAS dataset. The test classification accuracy is shown in
parenthesis. Example images from each class are shown. The frames around each
image represent the statistical class of the PISAS dataset. Red, green, and
blue frames represent the multinomial, constant, and binomial distributions
respectively. The CNN model appears to map structural textures near one
another in the projected space (i.e., sand ripple images are grouped to the
left region of the projection and the rocky textures are grouped in the right
of the projection). On the other hand, the histogram layer model maps
statistical textures close to one another. The binomial statistical textures
(blue frames) are grouped in the left of the projection. The multinomial (red
frames) and constant (green frames) statistical textures are clustered in the
center and right of the projection respectively.

(a) ResNet18 ($95.00$)

(b) Series ($86.67$)

(c) Parallel ($100.00$)

Figure 8: t-SNE projections for the best baseline ResNet18, HistRes_16 Series,
and HistRes_16 Parallel. The test classification accuracy for the model is
shown in parenthesis. The baseline ResNet18 did not have the histogram layer;
as a result, the model will focus on structural textures. The projection of
the ResNet18 features on the test data highlight that images with similar
structures are in the mapped near one another in the lower dimensional space.
Sand ripple images are to the left of the projection while there is a
transition to rocky and crater textures in the middle with flat textures
projected to the far right. The contrary is shown for the Series histogram
model. Textures with similar statistics (i.e., pixel intensities) are
projected near one another. The parallel model consist of both statistical and
structural textures information; therefore, we observe a mix of the different
textures in the projected space.

#### CNN vs. Histogram Layer Models

TABLE I: The average test accuracy for each model per class type is shown. The best average performance is bolded. The statistical labels relay the classification performance for predicting the distribution of the foreground pixels, while the structural labels show each model’s prediction for sand ripple or rocky textures. Three experimental runs were conducted, and error values are reported with $\pm 1$ standard deviation. Label | CNN | Hist
---|---|---
Both | 55.65$\pm$21.0 | 81.11$\pm$0.76
Statistical | 57.37$\pm$18.8 | 99.90$\pm$0.07
Structural | 94.28$\pm$7.51 | 81.21$\pm$0.73
TABLE II: The average Calinski-Harabasz index [29] for each model per class type is shown. The best average performance is bolded (larger values indicate better separability and compactness). The statistical labels demonstrate the feature quality for predicting the distribution of the foreground pixels, while the structural labels show the feature quality for distinguishing between the sand ripple or rocky textures. Three experimental runs were conducted, and error values are reported with $\pm 1$ standard deviation. Label | CNN | Hist
---|---|---
Both | 3,333.88$\pm$2,163.06 | 26,129.02$\pm$236.77
Statistical | 40.55$\pm$32.50 | 50,516.00$\pm$460.19
Structural | 15,399.12$\pm$9951.00 | 134.28$\pm$3.92

The results of each shallow network on the PISAS test data are shown in Table
I. Similar trends in the results follow for the SAS structures as observed by
the simple structures in [9]. The convolutional model can easily distinguish
the sand ripple and rocky textures, but struggles with the statistical
differences in comparison to the histogram layer. The histogram layer can be
important in SAS application when the statistics of the data may change due to
environmental conditions such as speckle noise [30]. The confusion matrices
also highlight the differences between the two models, as shown in Figure 6.
The histogram layer model has the most difficulty with the structural changes
between the sand ripple and rocky images, since there is equal weighting on
spatial contribution of neighbors due to the average pooling operation.
However, the CNN has more degradation in performance when observing
statistical changes, particularly for the rocky texture (“T10”).

Interesting observations can also be seen qualitatively in the t-SNE
projection of the training data, shown in Figure 7. The t-SNE projections used
the same initialization and random seed. The only difference is the features
that were projected into the lower dimensional space (convolutional or
histogram features). For the CNN, the sand ripple and rocky images are
projected near each other. However, the statistical textures are mapped to
similar regions of the feature space after the t-SNE projection. The clusters
are also appeared tighter for the histogram layer model, indicating that the
histogram layer may be able to handle small intra-class variations in the data
through the soft binning operations to characterize images in the same class.

Additional analysis was needed to verify the plausible conclusions discussed
above to ensure the difference is a result of the higher dimensional structure
and not an artifact of the

Figure 9: t-SNE projections of SAS images from various locations of the
seafloor passed through the best parallel HistRes_16 model. The colors
indicate images collected from different regions. The best parallel model was
trained on a small subset of labelled SAS imagery (i.e., multi-site dataset).
The model was able to learn features that generalized to new images from
different regions. In the projected space, we see meaningful groupings such as
sand ripple textures in the far right of the projection. The model was also
able to extract features that grouped images containing cross talk, as shown
in the bottom left-hand corner of the projection.

t-SNE mapping. Therefore, we used a cluster validity metric to assess the
quality of the features learned by each method with regard to the labels. The
Calinski-Harabsz index [29] considers both intra- and inter-cluster distances.
Larger values indicate more compact and separated classes in the feature
space. We observed that the metrics matched our analysis of the lower
dimensional space, as shown in Table II. The histogram layer model
outperformed the CNN for statistical textures, while the CNN was superior for
structural textures. Overall, we noted that the histogram features had the
best features across all classes. Experiments on additional SAS images in the
Parallel vs. Series Configuration section further demonstrated the power of
statistical features for deep learning models.

#### Parallel vs. Series Configuration

TABLE III: The average test accuracy for each model on the SAS dataset is shown. The best average performance is bolded. The baseline ResNet18 model achieved a test classification accuracy of 84.66$\pm$11.42. Five experimental runs were conducted, and error values are reported with $\pm 1$ standard deviation. Configuration | HistRes_4 | HistRes_8 | HistRes_16
---|---|---|---
Parallel | 90.33$\pm$4.00 | 82.67$\pm$8.27 | 88.33$\pm$9.60
Series | 77.33$\pm$5.01 | 73.33$\pm$5.06 | 81.00$\pm$4.03

The test performance on the multi-site dataset for the model configuration
experiments is shown in Table III. The parallel and series models were robust
to the number of bins, as there was not a significant difference in
performance (i.e., overlapping error bars). The parallel model outperformed
both the baseline and series models. This further validated the utility of
using both statistical and structural information to maximize texture
classification. In addition to the overall test performance, a qualitative
analysis was performed using t-SNE (Figure 8). The convolutional model mapped
textures with similar structures near each other, while the histogram models
mapped points with similar statistics near one another. For the parallel
configuration, there was a mix of similar statistical and structural SAS
images near one another in the projected space.

Once training and evaluation was completed on the subset of labeled SAS
images, the model was applied to hold-out SAS images from multiple locations
on the seafloor. The t-SNE projection is shown in Figure 9. The model
successfully extracted features from the hold-out images that showed
meaningful groupings in the lower dimensional projection space. The parallel
HistRes_16 model can also potentially be used to identify images with cross-
talk (bottom left-hand corner of Figure 9). The results in this section
demonstrated the use of the histogram layer for various real-world
applications for SAS texture classification.

##  IV Conclusion

We presented a novel application of histogram layer(s) for SAS image
classification. The results presented in this work show the impact of
statistical texture features within deep learning frameworks. For SAS imagery,
histogram layer(s) provide a powerful feature representation that can jointly
be tuned with CNNs to optimize performance. Future work includes exploring
other convolutional backbones, investigating other binning functions,
performing semantic segmentation, and extensive comparisons with other SAS
image analysis approaches.

## References

  * [1] M. P. Hayes and P. T. Gough, “Synthetic aperture sonar: a review of current status,” _IEEE journal of oceanic engineering_ , vol. 34, no. 3, pp. 207–224, 2009.
  * [2] G. Richard, J.-M. Le Caillec, J. Habonneau, and D. Gueriot, “A deep sas atr explainability framework assessment,” in _OCEANS 2021: San Diego–Porto_. IEEE, 2021, pp. 1–5.
  * [3] Y.-C. Sun, I. D. Gerg, and V. Monga, “Iterative, deep synthetic aperture sonar image segmentation,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–15, 2022.
  * [4] D. Stewart, S. Johnson, and A. Zare, “Image-to-height domain translation for synthetic aperture sonar,” _arXiv preprint arXiv:2112.06307_ , 2021.
  * [5] D. P. Williams, “Unsupervised seabed segmentation of synthetic aperture sonar imagery via wavelet features and spectral clustering,” in _2009 16th IEEE International Conference on Image Processing (ICIP)_. IEEE, 2009, pp. 557–560.
  * [6] D. Kohntopp, B. Lehmann, D. Kraus, and A. Birk, “Seafloor classification for mine countermeasures operations using synthetic aperture sonar images,” in _OCEANS 2017-Aberdeen_. IEEE, 2017, pp. 1–5.
  * [7] R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel, “Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness,” in _International Conference on Learning Representations_ , 2018.
  * [8] K. Hermann, T. Chen, and S. Kornblith, “The origins and prevalence of texture bias in convolutional neural networks,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 19 000–19 015, 2020.
  * [9] J. Peeples, W. Xu, and A. Zare, “Histogram layers for texture analysis,” _IEEE Transactions on Artificial Intelligence_ , vol. 3, no. 4, pp. 541–552, 2022.
  * [10] L. Zhu, D. Ji, S. Zhu, W. Gan, W. Wu, and J. Yan, “Learning statistical texture for semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 12 537–12 546.
  * [11] L. Liu, J. Chen, P. Fieguth, G. Zhao, R. Chellappa, and M. Pietikäinen, “From bow to cnn: Two decades of texture representation for texture classification,” _International Journal of Computer Vision_ , vol. 127, no. 1, pp. 74–109, 2019.
  * [12] J. Peeples, M. Cook, D. Suen, A. Zare, and J. Keller, “Comparison of possibilistic fuzzy local information c-means and possibilistic k-nearest neighbors for synthetic aperture sonar image segmentation,” in _Detection and Sensing of Mines, Explosive Objects, and Obscured Targets XXIV_ , vol. 11012. International Society for Optics and Photonics, 2019, p. 110120T.
  * [13] A. Zare, N. Young, D. Suen, T. Nabelek, A. Galusha, and J. Keller, “Possibilistic fuzzy local information c-means for sonar image segmentation,” in _2017 IEEE Symposium Series on Computational Intelligence (SSCI)_. IEEE, 2017, pp. 1–8.
  * [14] D. P. Williams, “Fast unsupervised seafloor characterization in sonar imagery using lacunarity,” _IEEE transactions on Geoscience and Remote Sensing_ , vol. 53, no. 11, pp. 6022–6034, 2015.
  * [15] R. M. Haralick, K. Shanmugam _et al._ , “Textural features for image classification,” _IEEE Transactions on systems, man, and cybernetics_ , no. 6, pp. 610–621, 1973.
  * [16] Z. Zhu, X. Xu, L. Yang, H. Yan, S. Peng, and J. Xu, “A model-based sonar image atr method based on sift features,” in _OCEANS 2014-TAIPEI_. IEEE, 2014, pp. 1–4.
  * [17] J. T. Cobb, K. C. Slatton, and G. J. Dobeck, “A parametric model for characterizing seabed textures in synthetic aperture sonar images,” _IEEE Journal of Oceanic Engineering_ , vol. 35, no. 2, pp. 250–266, 2010\.
  * [18] V. Sedighi and J. Fridrich, “Histogram layer, moving convolutional neural networks towards feature-based steganalysis,” _Electronic Imaging_ , vol. 2017, no. 7, pp. 50–55, 2017.
  * [19] Z. Wang, H. Li, W. Ouyang, and X. Wang, “Learnable histogram: Statistical context features for deep neural networks,” in _European Conference on Computer Vision_. Springer, 2016, pp. 246–262.
  * [20] I. Yusuf, G. Igwegbe, and O. Azeez, “Differentiable histogram with hard-binning,” _arXiv preprint arXiv:2012.06311_ , 2020.
  * [21] H. Sadeghi and A.-A. Raie, “Histnet: Histogram-based convolutional neural network with chi-squared deep metric learning for facial expression recognition,” _Information Sciences_ , vol. 608, pp. 472–488, 2022.
  * [22] A. Materka, M. Strzelecki _et al._ , “Texture analysis methods–a review,” _Technical university of lodz, institute of electronics, COST B11 report, Brussels_ , pp. 9–11, 1998.
  * [23] A. Humeau-Heurtier, “Texture feature extraction methods: A survey,” _IEEE Access_ , vol. 7, pp. 8975–9000, 2019.
  * [24] J. S. Weszka, C. R. Dyer, and A. Rosenfeld, “A comparative study of texture measures for terrain classification,” _IEEE transactions on Systems, Man, and Cybernetics_ , no. 4, pp. 269–285, 1976.
  * [25] R. M. Haralick _et al._ , “Statistical and structural approaches to texture,” _Proceedings of the IEEE_ , vol. 67, no. 5, pp. 786–804, 1979\.
  * [26] T. Ojala, M. Pietikäinen, and D. Harwood, “A comparative study of texture measures with classification based on featured distributions,” _Pattern recognition_ , vol. 29, no. 1, pp. 51–59, 1996.
  * [27] S. Walker, J. Peeples, J. Dale, J. Keller, and A. Zare, “Explainable systematic analysis for synthetic aperture sonar imagery,” in _2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS_ , 2021, pp. 2835–2838.
  * [28] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _International Conference on Learning Representations (ICLR)_ , 2015.
  * [29] T. Caliński and J. Harabasz, “A dendrite method for cluster analysis,” _Communications in Statistics-theory and Methods_ , vol. 3, no. 1, pp. 1–27, 1974.
  * [30] A. Abu and R. Diamant, “Robust image denoising for sonar imagery,” in _2018 OCEANS-MTS/IEEE Kobe Techno-Oceans (OTO)_. IEEE, 2018, pp. 1–5.