# Efficient Self-Ensemble for Semantic Segmentation

Walid Bousselham
Guillaume Thibault
Lucas Pagano
Archana Machireddy
Joe Gray
Young Hwan Chang

Oregon Health and Science University
Portland, OR, USA
Xubo Song

###### Abstract

Ensemble of predictions is known to perform better than individual predictions
taken separately. However, for tasks that require heavy computational
resources, e.g. semantic segmentation, creating an ensemble of learners that
needs to be trained separately is hardly tractable. In this work, we propose
to leverage the performance boost offered by ensemble methods to enhance the
semantic segmentation, while avoiding the traditional heavy training cost of
the ensemble. Our self-ensemble approach takes advantage of the multi-scale
features set produced by feature pyramid network methods to feed independent
decoders, thus creating an ensemble within a single model. Similar to the
ensemble, the final prediction is the aggregation of the prediction made by
each learner. In contrast to previous works, our model can be trained end-to-
end, alleviating the traditional cumbersome multi-stage training of ensembles.
Our self-ensemble approach outperforms the current state-of-the-art on the
benchmark datasets Pascal Context and COCO-Stuff-10K for semantic segmentation
and is competitive on ADE20K and Cityscapes. Code is publicly available at
github.com/WalBouss/SenFormer.

##  1 Introduction

Semantic segmentation is the task of assigning each pixel of an image with a
semantic category, and therefore is close to the task of image classification.
Its many applications include robotics, autonomous cars, medical application,
augmented reality and more. Most segmentation methods follow an Encoder-
Decoder scheme. The encoder extracts the relevant features of the image to
characterize each pixel, a process usually involving down-sampling the feature
maps to increase the receptive field of the model. The decoder up-samples the
feature maps to both recover the spatial information and produce a per-pixel
classification. In [35] the authors extended this procedure to fully
convolutional network (FCN), which paved the way for later work to achieve
impressive results on various segmentation datasets and has since dominated
the field of semantic segmentation, let it be for medical [40, 20], self-
driving cars [42] or robotics applications [21]. Follow up work mainly focused
on enhancing FCN to mitigate the inherent locality of the convolution
operation. Some examples are the atrous convolution that introduces holes in
convolution kernel [4, 5], the pyramid pooling module (PPM) that aggregates
context information using different kernel pooling layers, and [55] that
combine the PPM and the feature pyramid network (FPN) [31] to capture context
information at different resolutions.

The starting observation of this paper was that the combined use of a backbone
and an FPN-like method [31, 33, 17, 44, 52, 23] allows extracting multiple
features sets at different scales for a single image with a unique forward
pass. Furthermore, in [31] the authors show that these features are both
semantically and spatially strong at each level of the feature pyramid.
Consequently, one has access to multiple features representations of the same
image that carry different contextual information at different scale and are
loosely correlated (as shown in[52]). This raises questions about the optimal
way to use these multi-scales features. A canonical use is UperNet [55], which
concatenates the multi-scale feature maps before feeding them to a decoder.
However, this paper argues that the ”features fusing” strategy consisting of
merging the different sets of features maps and letting the model decide which
one is important is sub-optimal and often computationally expensive. Indeed,
in UperNet, the four pyramid levels are concatenated and merged by a
convolution, which by its own involves 155G FLOPs, thus making the ”features
fusing” strategy FLOPs intensive. Moreover, we hypothesize that a single
decoder cannot fully take advantage of the multi-scale features that contain
different views of the same objects of interest. Hence, the model may focus on
one view and overlook valuable features. This ”multi-view” hypothesis is
indeed supported by a recent study: in [1] the authors argue that in vision
datasets, objects can be recognized using multiple views and show that in the
context of image classification, for a given weight initialization, a model
will learn to focus on particular views while discarding others.

To overcome the limitations of ”features fusion” strategies, we propose and
study a different approach to exploit the FPN multi-scale features. Our
approach feeds independent decoders with features coming from different levels
of the feature pyramid, and then combine the segmentation maps together, hence
avoiding expensive features fusion operations. Since the inputs to the
learners (i.e., decoders) come from different levels of the feature pyramid
that differ in scale and contain different spatial and semantic information
and that the learners are independent, our method can be interpreted as a form
of self-ensemble segmentation. Usually, the learners of an ensemble must be
trained independently. In this work, we show that, in the context of semantic
segmentation, this condition can be relaxed and imposed solely to the
decoders. Our experiments show that – all else being equal – this strategy
improves UperNet performance. However, increasing the number of
decoders/learners inevitably increases parameters number. Overall, our
observations on self-ensemble performance effectiveness but parameter burden,
lead us to design a transformer-based model: SenFormer (Self-ensemble
segmentation transFormer). Our motivation for using transformer-based learners
is that besides transformers’ ability to capture long-range dependencies, it
has been observed [25, 10, 28] that recursively applying the same transformer
block to the same input features can produce similar – if not better – results
than using different blocks while reducing the number of parameters and
overfitting. Ultimately, our method has fewer parameters and FLOPs than
UperNet and performs better.

Overall, our SenFormer approach achieves excellent results on various
benchmark datasets. Specifically, it outperforms similar architectures [55]
that use ”feature fusion” strategy, suggesting that our self-ensemble approach
effectively leverages the expressive power of ensemble methods. In particular,
SenFormer achieves 51.5 mIoU on the benchmark dataset COCO-Stuff-10K [2] and
64.0 mIoU on Pascal-Context [37], outperforming the previous state-of-the-art
by a large margin of 6 mIoU and 3.0 mIoU respectively. SenFormer is also on
par with state-of-the-art methods on Ade20K [60] and Cityscapes [9]. To
summarize, our contributions are two fold:

  * •

We propose an innovative way to leverage the multi-scale features produced by
the FPN to form an ensemble of learners inside a single model.

  * •

We develop a light-weight transformer-based decoder that is used as a learner
in our self-ensemble.

Figure 1: SenFormer architecture. (Left): The features extracted by the
backbone $\\{C_{2},C_{3},C_{4},C_{5}\\}$ are enhanced in a feature pyramid to
produce spatially and semantically strong features maps at every level of the
pyramid $\\{P_{2},P_{3},P_{4},P_{5}\\}$. Each set of features is decoded by a
different learner in the ensemble and the learners’ predictions are merged.
(Right): architecture of the transformer block.

##  2 Related Works

Semantic Segmentation. Since fully convolutional networks (FCN) have been
introduced in the seminal work [35], it has dominated the field of semantic
segmentation. However, due to the inherent locality of the convolution
operation, architectures solely based on convolutions struggle to capture
long-range dependencies, making it difficult to deal with large and occulted
objects. Follow up methods to alleviate the locality issue include but are not
limited to atrous convolution [4, 5], pyramid pooling module [58] or the use
of FPN [55, 26].

More recently, motivated by the stupendous success of transformer-based
architecture for image classification [11, 46], multiple works proposed to
leverage the self-attention operation to improve segmentation performance of
FCN scheme or even completely replace it. Transformer-based architectures can
be used as a drop-in replacement of traditional CNN backbones to enhance the
extracted features supplied to the decoder [34, 46, 51, 12, 50]. It has been
observed that transformer backbones that produce a hierarchical feature
representation [34, 50] are the most suited for segmentation tasks.
Additionally, following the original Encoder-Decoder Transformer [47] used in
NLP, recent works proposed to formulate the problem of semantic segmentation
as a sequence-to-sequence problem [59, 43, 56], freeing the architecture from
any inductive prior biases.

Alternatively, motivated by the success of DETR [3] which used transformer for
object detection, MaX-DeepLab [49] and MaskFormer [7] treated semantic (and
panoptic) segmentation no longer as a per-pixel classification but as a mask
classification problem. These architectures first generate a set of candidate
masks that are then classified. Shifting the semantic segmentation paradigm
from a per-pixel classification to a mask classification problem helps bridge
the gap between detection/panoptic segmentation methods and semantic
segmentation. It also involves the computation of an assignment score between
each generated mask and every class, therefore transferring a part of the
training burden to the loss calculation. Since our investigation focuses on
the efficacy of features fusion and self-ensemble, we will limit our
comparisons to per-pixel classification-based architectures.

Efficient Ensemble. A major limiting factor for building an ensemble of deep
learning models is the computational cost during training and testing. Diverse
methods were proposed to tackle this issue. By repeatedly applying dropout at
inference on an already trained model, Monte Carlo Dropout [15] allows getting
many predictions from a single model, ultimately improving its accuracy.
BatchEnsemble [53] significantly lower ensemble cost by defining each
learner’s weights to be the Hadamard product between a shared matrix and a
rank-one matrix per learner. Snapshot [22] train a single model to converge to
several local minima by leveraging cyclic learning rate scheduling. Other
methods for classification include MIMO [18], hyper-batch ensemble [54], late-
phase weights [48] or FGE [16]. For segmentation, [45] improves the widely
used multi-scale inference by learning relative attention between the scales
during training and is used at test-time to greatly improve the performance.
However, these methods still require several forward passes of the same image,
let it be for training or testing. Perhaps most related to our work is TreeNet
[29], which uses multiple classifier branches that share their early layers.
Nevertheless, besides being for classification, unlike to our work, all the
learners receive the same input, limiting the depth of the shared part.
Moreover, in SenFormer, the parameter cost of the ensemble is further reduced
through weight sharing within a learner.

##  3 Method

In this section, we first present the general framework of our method based on
self-ensemble as shown in figure 1. Then we detail the different merging
strategies. Finally, we describe learners’ architecture and the different
weight sharing strategies.

Following notations in [19, 31, 55], we denote
$C_{i}\in\mathcal{R}^{d_{i}\times\frac{H}{2^{i}}\times\frac{W}{2^{i}}}$ the
output of the i-th stage of the bottom-up network (i.e. backbone) which has
stride of $2^{i}$ pixels with respect to the input image, where $H\times W$ is
the spatial dimension of the input image and $d_{i}$ the number of channels.
Similarly, we denote
$P_{i}\in\mathcal{R}^{d\times\frac{H}{2^{i}}\times\frac{W}{2^{i}}}$ the output
of the i-th stage of the top-down network (i.e. output of the FPN), where $d$
is the numbers of channels in all the feature maps of the FPN. We denote $N$
the number of class.

###  3.1 Self-Ensemble

In this paper, we approach the problem of semantic segmentation as that of a
per-pixel classification. Therefore, learners predictions and the merging
strategies will be described for an arbitrary pixel and can easily generalize
to the whole segmentation map.

An ensemble traditionally consists of $M$ independently trained models called
learners. For a given pixel, let denote $X_{i}\in\mathcal{R}^{N}$ the random
variable parameterized by the output of the i-th learner for that particular
pixel, which can be decomposed in:

$\displaystyle X_{i}$ $\displaystyle=Y+\epsilon_{i}$ (1)

where $Y$ is the target and $\epsilon_{i}$ is the prediction error of the i-th
learner.

The most straightforward way to merge different learners’ predictions is by
averaging them. It is well known [39] [61] [27] that the ensemble performance
is usually better than the individual learners.

Classical statistics suggest that when the predictions are roughly
independent, the last term in equation 2 is close to zero and therefore
averaging greatly reduces the noise.

$\displaystyle
Var(\frac{1}{M}\sum_{i=1}^{M}\epsilon_{i})=\frac{1}{M^{2}}\sum_{i=1}^{M}Var(\epsilon_{i})+\frac{2}{M^{2}}\sum_{i<j}Cov(\epsilon_{i},\epsilon_{j}).$
(2)

On another note, a recent study suggests that this hypothesis might not hold
in the context of deep learning. In [1], Allenzhu et. al, acknowledge that for
the task of image classification, the different learners learn to detect
different views/features of the object of interest depending on their weight
initialization. However, there are some images taken from a particular angle
where the learned features may be missing. Hence, when the ensemble is large
enough, all possible views are captured, thus increasing the model’s accuracy.
Note, however, that it is not clear in [1] if this result also holds for
semantic segmentation. Either way, a key requirement is that the learners’
predictions must be independent, let it be for the variance reduction or the
multi-view hypothesis.

Total independence of the predictions implies tediously training multiple
independent models. In this paper, we aim at relaxing the independence
hypothesis to reduce the training cost, while maintaining the performance
benefits of ensemble. To do so, the learners/decoders share the same backbone
but receive input features coming from different levels of the feature
pyramid, i.e., $\\{P_{2},P_{3},P_{4},P_{5}\\}$, as shown in figure 1.

Nevertheless, it is observed that if one trains the different learners of an
ensemble altogether (i.e., applying the loss on the merged prediction), the
performance boost offered by the ensemble disappears [1]. However, we show in
our experiments that it is not the case in our setting. We hypothesize that it
is because each learner is independently initialized (as in ensemble) and
receives different inputs, therefore alleviating the need for separate
training. In this manner, several segmentation predictions can be obtained
with only a single forward pass of the input image.

###  3.2 Merging strategies

We describe the different methods considered to merge the different learner
predictions (during inference).

Averaging. It is the most commonly used method for prediction merging as no
additional trainable parameters are required. The merged prediction $X_{avg}$
of $M$ learners is obtained by:

$\displaystyle X_{avg}=\frac{1}{M}\sum_{i}X_{i}$ (3)

Product. The predicted probability for each pixel is multiplied rather than
average. That way, more weight is given to learners with high confidence.The
merged prediction $X_{prod}$ of $M$ is given by:

$\displaystyle X_{prod}=\prod_{i=1}^{M}X_{i}$ (4)

Majority vote. Each learner assigns a vote to the class with the largest
confidence. The merged prediction $X_{maj}$ is obtained by:

$\displaystyle
X_{maj}=\frac{1}{M}\sum_{i}\delta_{c}(X_{i})\quad\text{where}\quad
c=\underset{1\leq j\leq N}{argmax}\ X_{i}^{j}$ (5)

where $\forall j\leq
N,\delta_{c}(X_{i})^{j}=\left\\{\begin{array}[]{ll}X_{i}^{j}&\mbox{if }j=c\\\
0&\mbox{else.}\end{array}\right.$

Hierarchical Attention. We borrow the ”attention module” from [45] that is
used to learn a relative attention mask between adjacent scales. The module
consists of $(3\times
3\,conv)\rightarrow(BatchNorm)\rightarrow(ReLU)\rightarrow(3\times
3\,conv)\rightarrow(BatchNorm)\rightarrow(ReLU)\rightarrow(1\times
1\,conv)\rightarrow(Sigmoid)$, where the last convolution output a single
(attention) map. In the original paper, the module is fed with the same input
features maps of the decoder. Another variant would be to use the segmentation
logits (decoder’s output) instead. In our experiment, we tried both and found
the latter to work better with SenFormer. Since SenFormer has four learners,
we need 3 ”attention modules” to predict the relative attention maps.

Explicit Attention. We used the same ”attention module” as for Hierarchical
Attention [45], but trained it to predict a dense mask for each scale rather
than a relative mask.

Surprisingly, our experiments found the simple ”averaging” strategy to perform
better than others, except for the ”hierarchical attention” (Table 11).
However, given the performance boost of the ”attention module” is limited, it
does not justify the overhead complexity. Therefore, SenFormer uses the
”averaging” as the default merging strategy since it yields high performance
without requiring additional parameters.

###  3.3 Learner architectures

Hereafter, we described the architecture of a single learner/decoder.

As depicted in Figure 1, the $i^{th}$ decoder branch takes as input the
features coming from the corresponding level of the FPN (with stride $s_{i}$)
$P_{i}\in\mathcal{R}^{d\times\frac{H}{2^{i}}\times\frac{W}{2^{i}}}$, as well
as a set of $N$ learnable embeddings termed as class embeddings,
$\textbf{cls}_{i}=[cls^{1}_{i},\dots,cls^{N}_{i}]\in\mathcal{R}^{N\times d}$,
where $N$ is the number of class. In this respect, there is one learnable
class embedding $cls^{k}_{i}$ per segmentation class and per level in the
feature pyramid.

Each decoder is a transformer composed of $L$ layers whose architecture is
inspired by the traditional transformer [47]. Note however that a ”pre-norm”
strategy is used in place of ”post-norm” for the placement of Layer
Normalization (LN), i.e., the skip connections inside each transformer block
are not affected by the LN [38] (see ablation study in the Annex).

In a nutshell, a single Transformer Decoder block consists of three successive
operations: Cross-Attention, Self-Attention and Multi-Layer Perceptron layers.
In the Cross-Attention operation the feature map $P_{i}$ is used as key and
value while the class embedding $\textbf{cls}_{i}$ is used a query. The Self-
Attention and MLP are applied only to the class embeddings.

Finally, each decoder/learner is composed of L layers of decoder block and its
prediction is obtained via a dot product between the class embeddings
$\textbf{cls}_{i}$ and the corresponding feature pyramid feature $P_{i}$ – see
the Annex for more details. However, using multiple decoders greatly increases
the number of parameters. To mitigate this, we explore several weight-sharing
strategies.

###  3.4 Weight sharing

Weight sharing is a commonly used technique to reduce the number of parameters
[25, 10, 28], while also regularizing the optimization by reducing the degree
of freedom which mitigates overfitting. However, in the context of Ensemble,
special care regarding the kind of weight sharing used must be given.

Two types of weight sharing can be used: inter-learner and intra-learner
sharing. The former involves sharing parameters between the different
learners, while latter within the learner. Figure 2 depicting the different
sharing methods can be found in Annex.

Repeated block. A given learner is composed of a single decoder block
recursively used $L$ times. It is a form of ”intra-learner sharing” since no
parameters are shared between the different learners.

Decoder sharing. The different learners share the same decoder but have their
own class embedding. It is a form of ”inter-learner sharing”.

Class embeddings sharing. The same learnable class embeddings cls is used for
all the learners. It is also a form of ”inter-learner sharing”.

Table 2 shows that any ”inter-learner sharing” strategy significantly degrades
the segmentation performance, confirming the importance of keeping the
different learners as independent as possible. Conversely, the ”repeated
block” strategy performs better than when no sharing is used, while
significantly reducing the number of parameters. Hence, SenFormer uses the
”repeated block” as the default weight sharing policy.

##  4 Experiments

Datasets. We evaluate our model performance using four semantic segmentation
benchmark datasets, ADE20K [60], Pascal Context [37], COCO-Stuff-10K [2] and
Cityscapes [9]. We use ADE20K, which is a challenging scene parsing dataset
consisting of 20,210 training images and 2,000 validation images and covers
$150$ fine-grained labeled classes, for the ablation studies. Please see the
Annex for detailed descriptions of all used datasets.

Evaluation metric. We report the mean Intersection over Union (mIoU), a
standard metric for semantic segmentation.

Baseline model. To demonstrate that the performance improvement of our method
is genuinely a result of self-ensemble instead of feature fusion, we introduce
a simple decoder baseline module that borrows the features fusion strategy
from UperNet [55], but uses our transformer decoder– see Figure 3. This way,
the FeatureFusionBaseline and SenFormer only differ by the multi-scale fusion
strategy. Following [55], the baseline multi-scale fusion strategy is as
follow: we first resize (through bilinear interpolation) all the features
$\\{P_{2},P_{3},P_{4},P_{5}\\}$ to match $P_{2}$ dimension (i.e. $1/4$ of the
input image) and concatenate them. We then apply a $3\times 3$ convolution
followed by a batch normalization layer and a ReLU activation. Note that this
baseline is only used for ablation purposes and SenFormer is thereafter also
compared to state of the art methods in Section 4.3.

Table 1: Performances by using different learner combinations, where ✓/✗ indicates whether the learner is used for the prediction. $d_{2}$ | $d_{3}$ | $d_{4}$ | $d_{5}$ | mIoU
---|---|---|---|---
✓ | ✗ | ✗ | ✗ | 42.90
✗ | ✓ | ✗ | ✗ | 42.08
✗ | ✗ | ✓ | ✗ | 41.40
✗ | ✗ | ✗ | ✓ | 38.23
✓ | ✓ | ✓ | ✗ | 44.12
✓ | ✓ | ✓ | ✓ | 44.38
Table 2: Performance comparisons on ADE20K validation of different weight sharing settings for SenFormer.♣ indicates SenFormer’s default setting. weight sharing | # blocks | mIoU | #params. | FLOPs
---|---|---|---|---
setting
decoder shared | 6 | 42.69 | 68M | 179G
cls embeddings | 6 | 42.91 | 67M | 179G
repeated♣ | 6 | 44.38 | 55M | 179G
none | 1 | 43.12 | 55M | 111G
none | 6 | 44.68 | 144M | 179G
UperNet | 42.05 | 67M | 238G

###  4.1 Implementation and training details

Backbones. Since SenFormer uses the FPN to build a multi-scale set of
features, it is compatible with any backbone architecture. In our experiments
we use both convolutional ResNet50 and ResNet101 [19] and the different size
of the transformer-based backbones Swin-Transformers[34].

FPN. The channel dimension of the feature pyramid $d$ is set to $512$. For
small backbones, we find it beneficial in the FPN to replace the traditional
$3\times 3$ convolution by a Window-based Transformer Block from [34]. Since
it introduces a marginal computation overhead, we applied it to all backbones.
Please see the appendix for detailed ablation of the FPN.

Decoders. Each learner is independently supervised with a cross-entropy loss.
In addition, we apply a standard cross-entropy loss on the final ensemble
prediction.

Training setting. We use mmsegmentation [8] library as codebase and follow the
standard training practice for each dataset. Moreover, we apply common data
augmentation for semantic segmentation, which include left-right flipping,
standard random color jittering, random resize with ratio $0.5-2$ and random
cropping.

For the optimizer, we use AdamW [36]. As common practice for segmentation, we
use ”poly” learning rate scheduler. Following [55, 7], we set the initial
learning rate to $10^{-4}$ and weight decay to $10^{-4}$ for ResNet backbones,
an initial learning rate of $6\cdot 10^{-6}$ and a weight decay of $10^{-2}$
for Swin-Transformer. We also use gradient clipping of $3$ to help stabilizing
the training, and for ResNet backbones a learning rate multiplier of $0.1$ is
applied.

During training, the input images are cropped to a size of $512\times 512$ for
ADE20K [60] and COCO-Stuff-10K [2], $512\times 512$ for Pascal Context [37]
and $512\times 1024$ for Cityscapes [9], unless stated otherwise. All the
models are trained on 8 V100 GPUs with a batch size of 8 for Cityscapes and 16
for the others (see Annex for details). The segmentation performance is
reported using single-scale inference. Finally, all the backbones are
pretrained on ImageNet-1K [41] unless stated otherwise.

###  4.2 Self-ensemble vs Features Fusion

Features at different levels of the pyramid carry different scale of
contextual information, and our experiments support that self-ensemble is
capable of capturing and integrating such information.

Table 3: Experiments with different number of decoder blocks. # block | mIoU | #params. | FLOPs
---|---|---|---
1 | 42.44 | 55M | 111G
3 | 43.25 | 55M | 139M
6 | 43.6 | 55M | 179G
9 | 43.7 | 55M | 220G
Table 4: Comparisons of the features fusion and self-ensemble strategies.♣indicates self-ensemble. method | mIoU | #params. | FLOPs
---|---|---|---
UperNet | 42.02 | 67M | 238G
SenUperNet♣ | 42.8 | 70M | 135G
FeaturesFusionBaseline | 43.1 | 52M | 307G
SenFormer♣ | 44.3 | 55M | 179G

Ensemble effect. We first analyze the output produced by each decoder and
assess their performance. Table 1 outlines the mIoU scores of independent
prediction of each decoder as well as for the ensemble. Notably, the ensemble
mIoU score is $+3.5$ better than the mean score of the learners taken
separately with $\frac{1}{4}\sum_{i=2}^{5}\textit{mIoU}(d_{i})=41.15$. More
surprisingly, even though $d_{5}$ taken separately performs significantly
worse than the others – due to its low-resolution inputs – it positively
contributes to the ensemble, consistent with traditional ensemble methods
where even weak learners can be combined to enhance the overall prediction.

Does the performance boost really comes from self-ensemble? To rule out the
performance gain brought by the use of transformer-based decoders rather than
convolution, we compare SenFormer and the FeaturesFusionBaseline, since they
only differ in the multi-scale fusion strategy (_features fusion vs. self-
ensemble_). In Table 4, we observe that SenFormer is +$2$ mIoU better than the
baseline. Conversely, we applied the self-ensemble method to UperNet [55] by
using the same convolution-based decoder at each level of the feature pyramid
rather than merging the features. Likewise, the self-ensemble version
(SenUperNet) performs better than the vanilla UperNet, suggesting that our
self-ensemble approach is the main driver for improvement.

Table 5: Self-ensemble SenFormer vs features fusion UperNet on ADE20K validation. Backbones pre-trained on ImageNet-22K are marked with ‡. | method | backbone | crop size | #params. | FLOPs | mIoU
---|---|---|---|---|---|---
CNN | UperNet | ResNet-50 | 512$\times$512 | 67M | 238G | 42.05
SenFormer | ResNet-50 | 512$\times$512 | 55M | 179G | 44.38
| UperNet | ResNet-101 | 512$\times$512 | 86M | 257G | 43.82
| SenFormer | ResNet-101 | 512$\times$512 | 79M | 199G | 46.93
Transformer | UperNet | Swin-T | 512$\times$512 | 60M | 236G | 44.41
SenFormer | Swin-T | 512$\times$512 | 59M | 179G | 46.0
UperNet | Swin-S | 512$\times$512 | 81M | 259G | 47.72
SenFormer | Swin-S | 512$\times$512 | 81M | 202G | 49.2
| UperNet | Swin-B‡ | 640$\times$640 | 121M | 471G | 50.04
| SenFormer | Swin-B‡ | 640$\times$640 | 120M | 371G | 52.21
| UperNet | Swin-L‡ | 640$\times$640 | 234M | 647G | 52.05
| SenFormer | Swin-L‡ | 640$\times$640 | 233M | 546G | 53.08

SenFormer vs UperNet. We compare SenFormer with UperNet architecture for a
variety of CNN- and transformer-based backbones. As we can see from Table 5,
when using the same standard Swin-Transformer backbone, SenFormer consistently
outperforms UperNet regardless of the backbone size. The performance gap is
even larger when using convolutional backbones (+$3$ mIoU), suggesting that
our transformer-based decoder successfully captures the long-range
dependencies missed by the CNN-based backbones.

Thanks to its weight sharing strategy, SenFormer has fewer parameters than
UperNet. Furthermore, since SenFormer avoids the computationally expensive
features merging operation, it also has substantially fewer FLOPs.

###  4.3 Comparison to state-of-the-art

In this section we further compare SenFormer to state-of-the-art methods on
ADE20K and additional benchmark datasets.

Table 6: Benchmark on ADE20K validation set. method | backbone | mIoU | +MS
---|---|---|---
DeepLabV3+ [6] | R50 | 44.0 | 44.9
PerPixelBaseline+[7] | R50 | 41.9 | 42.9
MaskFormer [7] | R50 | 44.5 | 46.7
SenFormer | R50 | 44.4 | 45.2
OCRNet [57] | R101 | - | 45.3
DeepLabV3+ [6] | R101 | 45.5 | 46.4
MaskFormer [7] | R101 | 45.5 | 47.2
SenFormer | R101 | 46.9 | 47.9
SETR-L MLA [59] | ViT-L | - | 50.3
Segmenter [43] | ViT-L | 50.71 | 52.25
Segmenter-Mask[43] | ViT-L | 51.82 | 53.63
SegFormer [56] | MiT-B5 | 51.0 | 51.8
UperNet [34] | Swin-L | 52.05 | 53.5
SenFormer | Swin-L | 53.08 | 54.2
MaskFormer [7] | Swin-L | 54.1 | 55.6
Table 7: Benchmark on Pascal Context test. ‡/blue indicates previous/new SOTA. method | backbone | mIoU | +MS
---|---|---|---
DANet [14] | R50 | - | 50.5
EMANet [30] | R50 | - | 50.5
CAA [24] | R50 | 50.23 | -
SenFormer | R50 | 53.18 | 54.3
DANet [14] | R101 | - | 52.6
EMANet [30] | R101 | - | 53.1
DeepLabV3+ [6] | R101 | 53.2 | 54.67
OCRNet [57] | R101 | - | 54.8
CAA [24] | R101 | - | 55.0
SenFormer | R101 | 54.6 | 56.6
OCRNet [57] | HRNet | - | 56.2
CAA [24] | EN-B7 | 58.40 | 60.5‡
SETR-L MLA[59] | ViT-L | 54.9 | 55.8
Segmenter [43] | ViT-L | 58.1 | 59.0
SenFormer | Swin-L | 63.1 | 64.5

ADE20K. In Table 6 we compare SenFormer to a variety of FCN- and transformer-
based decoders using both CNN- and transformer-based backbones. When using
standard ResNet backbones, SenFormer outperforms all other methods. The same
can be said for per-pixel classification-based models when using transformer-
based backbones, where SenFormer even outperforms recently introduced
transformer-based decoders like SETR [59], Segmenter [43] and SegFormer [56].
Note however that MaskFormer [7] is doing better than SenFormer when using
transformer-based backbones. Indeed, MaskFormer introduces a new approach for
semantic segmentation that is based on mask classification (rather than
traditional per-pixel classification) and that greatly improves segmentation
performances. In fact, MaskFormer [7] significantly outperforms
PerPixelBaseline+ [7] while sharing the same architecture and only differing
by the problem formulation (per-pixel vs mask classification). We plan to
formulate SenFormer as mask classification in our future work, as it has
significant potential to improve segmentation.

Pascal Context. In Table 7 we compare SenFormer to current state-of-the-art
methods on Pascal Context test dataset, which is obtained by CAA [24] using
EfficientNet-B7(EN-B7) as backbone, with a mIoU of $60.5$. SenFormer
outperforms previous FCN methods when using standard ResNet backbones, as well
as recent transformer-based methods. SenFormer outperforms the current state-
of-the-art (CAA) when using the same ResNet-101 backbone, showing the benefit
of our approach. Moreover, we reach a score of 64.0 mIoU when using Swin-L as
backbone. Overall, our approach shows a significant improvement of +3.5 mIoU
over the previous state-of-the-art.

Table 8: Benchmark on COCO-Stuff-10K test. ‡/blue indicates previous/new SOTA. method | backbone | mIoU | +MS
---|---|---|---
EMANet [30] | R50 | - | 37.6
PerPixelBaseline+[7] | R50 | 34.2 | 35.8
MaskFormer [7] | R50 | 37.1 | 38.9
SenFormer | R50 | 40.0 | 41.3
DANet [14] | R101 | - | 39.7
EMANet [30] | R101 | - | 39.9
OCRNet [57] | R101 | - | 39.5
CAA [24] | R101 | - | 41.2
MaskFormer [7] | R101 | 38.1 | 39.8
SenFormer | R101 | 41.0 | 42.1
OCRNet [57] | HRNet | - | 40.5
CAA [24] | EN-B7 | - | 45.4‡
SenFormer | Swin-L | 49.8 | 51.5

COCO-Stuff-10K. Table 8 compares SenFormer to state-of-the-art methods on
COCO-Stuff-10K test dataset, which is obtained by CAA [24] using
EfficientNet-B7(EN-B7) as backbone, with a mIoU of $45.4$. When using standard
ResNet backbones, SenFormer outperforms previous FCN methods, as well as the
transformer-based method MaskFormer. Moreover, we obtain 51.5 mIoU when using
Swin-L as backbone, establishing a new SOTA by a substantial margin of +6 mIoU
over previous methods on COCO-Stuff-10K.

Cityscapes. Table 9 compare SenFormer to state-of-the-art methods on
Cityscapes validation dataset. We observe that SenFormer performs on par with
the best FCN and transformer-based methods. We hypothesis that since
Cityscapes dataset has only 19 classes, the object classification aspect of
the segmentation is easier and therefore SenFormer cannot benefit as much from
its class embeddings, as it does with datasets where the number of classes is
larger.

Table 9: Benchmark on Cityscapes val. method | backbone | mIoU | +MS
---|---|---|---
MaskFormer [7] | R50 | 78.5 | -
SenFormer | R50 | 78.8 | 80.1
DeepLabV3+[6] | R50 | 78.97 | 80.46
MaskFormer [7] | R101 | 79.7 | 81.4
SenFormer | R101 | 79.9 | 81.4
OCRNet [57] | R101 | - | 82.0
DeepLabV3+ [6] | R101 | 80.9 | 82.03
SETR-L PUP [59] | ViT-L | - | 82.2
Segmenter [43] | ViT-L | - | 80.7
Segmenter-Mask[43] | ViT-L | 79.1 | 81.3
SenFormer | Swin-L | 82.8 | 84.0
SegFormer [56] | MiT-B5 | 82.4 | 84.0

###  4.4 Ablation studies

For ablation studies, we solely use ResNet-50 pre-trained on ImageNet-1K [41]
as backbone and trained all models for 100k iterations.

Number of blocks. Table 3 show the results of SenFormer trained with varying
number of decoder block per learner. The mIoU improves with more decoder
blocks added. Note that using a single decoder block leads to significantly
poorer performance, suggesting that all the information in $P_{i}$ can not be
transfered to the class embeddings $\textbf{cls}_{i}$ in one operation. We
choose to use 6 decoder blocks per learner as it offers a good
complexity/performance trade-off.

Weight sharing. Table 2 compares different weight sharing approaches for
SenFormer. First, we observe that policies that involve sharing weights
between the learners (”decoder sharing” and ”cls embeddings”) lead to a
significant drop in performance, even when the number of parameters is not
reduced. For example, ”cls embedding” is 1.2mIoU lower than the base setting
while having 89M more parameters, confirming that independence between
learners is a key component in SenFormer. Furthermore, recursively applying
the same decoder block leads to non trivial performance boost of +1mIoU, while
having the same parameter number.

Table 10: Merging strategies. merging strategy | mIoU
---|---
averaging | 44.4
product | 40.28
majority vote | 39.89
hierarchical att. | 44.5
explicit att. | 39.7

Table 11: Ensemble and learners variance on Ade20K validation. Output | var. $(10^{-3})$
---|---
ensemble | 56.6
$d_{2}$ | 56.0
$d_{3}$ | 56.8
$d_{4}$ | 56.3
$d_{5}$ | 55.2

Table 12: Effect of increasing the number of learners. # learners | total | mIoU
---|---|---
per scale | # learner
1 | 4 | 44.3
2 | 8 | 44.2

##  5 Discussion

Variance reduction. A common explanation for the better performance of the
ensemble over its composing elements is that by averaging the variance over
the merged prediction is reduced. To test this assumption, for each input
image in the Ade20K validation set, we computed for the ensemble and for each
learner the variance over the segmentation map prediction for each pixel
(i.e., the variance along the channel axis). We then averaged over the entire
validation set. As shown in Table 11, the ensemble variance is not
significantly smaller than the variance of the individual learners.
Consequently, the variance reduction interpretation may not apply in the
context of self-ensemble, and more broadly for deep learning models [1].

Multi-view approach. A more recent explanation for the success of Ensemble is
that the different learners capture multi-views present in the data [1].
However, since the mutli-scale inputs of the learners come from the same
backbone, it is very unlikely that they focus on different views of the
objects of interest. We rather hypothesize that in SenFormer the boost in
performance does not emerge from the different random initialization of the
learners that will learn to focus on specific views of the input image, but
rather from the different scale information captured by the FPN. Consequently,
using more than one learner per level in the feature pyramid will not yield
better results. It is indeed confirmed by results in Table 12 where SenFormer
performances do not improve with additional learners.

##  6 Conclusions

This paper introduces our self-ensemble approach for semantic segmentation, a
simple methodology that benefits from ensemble learning while avoiding the
inconvenience and cost of training multiple times the same model. We leveraged
the multi-scale feature set produced by FPN-like methods to build an ensemble
of decoders within a _single model_ , where learners in the ensemble are fed
with features coming from different levels of the feature pyramid. We also
developed a transformer-based architecture for the learner/decoders.

Our approach outperforms current state-of-the-art on Pascal Context and COCO-
Stuff-10K datasets and is competitive on Ade20K and Cityscapes datasets for
semantic segmentation. It is more efficient in terms of FLOPs and limit the
number of parameter thanks to weight sharing. It is part of our future work to
investigate ”mask classification” formulation of semantic segmentation.

## Acknowledgements

We thank Romain Fabre for insightful discussion without which this paper would
not be possible. This work was partially supported by the National Institutes
of Health (NIH), National Cancer Institute (NCI) Human Tumor Atlas Network
(HTAN) Research Center (U2C CA233280), and and a NIH/NCI Cancer Systems
Biology Consortium Center (U54 CA209988).

## References

  * [1] Zeyuan Allen-Zhu and Yuanzhi Li.  Towards understanding ensemble, knowledge distillation and self-distillation in deep learning, 2021.
  * [2] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari.  Coco-stuff: Thing and stuff classes in context, 2018.
  * [3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers, 2020.
  * [4] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs, 2017.
  * [5] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation, 2017.
  * [6] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proceedings of the European conference on computer vision (ECCV), pages 801–818, 2018.
  * [7] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov.  Per-pixel classification is not all you need for semantic segmentation, 2021.
  * [8] MMSegmentation Contributors.  MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark.  https://github.com/open-mmlab/mmsegmentation, 2020.
  * [9] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding, 2016.
  * [10] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser.  Universal transformers, 2019.
  * [11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
  * [12] Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jegou.  Xcit: Cross-covariance image transformers, 2021.
  * [13] Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International Journal of Computer Vision, 88:303–338, 2009.
  * [14] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu.  Dual attention network for scene segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3146–3154, 2019.
  * [15] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In international conference on machine learning, pages 1050–1059. PMLR, 2016.
  * [16] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson.  Loss surfaces, mode connectivity, and fast ensembling of dnns.  Advances in neural information processing systems, 31, 2018.
  * [17] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V Le.  Nas-fpn: Learning scalable feature pyramid architecture for object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7036–7045, 2019.
  * [18] Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew Mingbo Dai, and Dustin Tran.  Training independent subnetworks for robust prediction.  In ICLR, 2021.
  * [19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * [20] Mohammad Hesam Hesamian, Wenjing Jia, Xiangjian He, and Paul J. Kennedy.  Deep learning techniques for medical image segmentation: Achievements and challenges.  Journal of Digital Imaging, 32:582 – 596, 2019.
  * [21] Zhang-Wei Hong, Chen Yu-Ming, Shih-Yang Su, Tzu-Yun Shann, Yi-Hsiang Chang, Hsuan-Kung Yang, Brian Hsi-Lin Ho, Chih-Chieh Tu, Yueh-Chuan Chang, Tsu-Ching Hsiao, Hsin-Wei Hsiao, Sih-Pin Lai, and Chun-Yi Lee.  Virtual-to-real: Learning to control in visual semantic segmentation, 2018\.
  * [22] Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hopcroft, and Kilian Q Weinberger.  Snapshot ensembles: Train 1, get m for free.  arXiv preprint arXiv:1704.00109, 2017.
  * [23] Shihua Huang, Zhichao Lu, Ran Cheng, and Cheng He.  Fapn: Feature-aligned pyramid network for dense image prediction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 864–873, 2021.
  * [24] Ye Huang, Di Kang, Wenjing Jia, Xiangjian He, and Liu Liu.  Channelized axial attention for semantic segmentation – considering channel relation within spatial attention for semantic segmentation, 2021.
  * [25] Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira.  Perceiver: General perception with iterative attention, 2021.
  * [26] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár.  Panoptic feature pyramid networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6399–6408, 2019.
  * [27] Sotiris B Kotsiantis, Ioannis D Zaharakis, and Panayiotis E Pintelas.  Machine learning: a review of classification and combining techniques.  Artificial Intelligence Review, 26(3):159–190, 2006.
  * [28] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations, 2020.
  * [29] Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra.  Why m heads are better than one: Training a diverse ensemble of deep networks, 2015.
  * [30] Xia Li, Zhisheng Zhong, Jianlong Wu, Yibo Yang, Zhouchen Lin, and Hong Liu.  Expectation-maximization attention networks for semantic segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9167–9176, 2019.
  * [31] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection, 2017.
  * [32] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár.  Microsoft coco: Common objects in context, 2015.
  * [33] Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia.  Path aggregation network for instance segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8759–8768, 2018.
  * [34] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows, 2021.
  * [35] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  CoRR, abs/1411.4038, 2014.
  * [36] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization, 2019.
  * [37] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille.  The role of context for object detection and semantic segmentation in the wild.  In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 891–898, 2014.
  * [38] Toan Q Nguyen and Julian Salazar.  Transformers without tears: Improving the normalization of self-attention.  arXiv preprint arXiv:1910.05895, 2019.
  * [39] David Opitz and Richard Maclin.  Popular ensemble methods: An empirical study.  Journal of artificial intelligence research, 11:169–198, 1999.
  * [40] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [41] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  Imagenet large scale visual recognition challenge, 2015.
  * [42] Mennatullah Siam, Sara Elkerdawy, Martin Jagersand, and Senthil Yogamani.  Deep semantic segmentation for automated driving: Taxonomy, roadmap and challenges, 2017.
  * [43] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid.  Segmenter: Transformer for semantic segmentation, 2021.
  * [44] Mingxing Tan, Ruoming Pang, and Quoc V Le.  Efficientdet: Scalable and efficient object detection.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10781–10790, 2020.
  * [45] Andrew Tao, Karan Sapra, and Bryan Catanzaro.  Hierarchical multi-scale attention for semantic segmentation, 2020.
  * [46] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention, 2021.
  * [47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2017.
  * [48] Johannes Von Oswald, Seijin Kobayashi, Joao Sacramento, Alexander Meulemans, Christian Henning, and Benjamin F Grewe.  Neural networks with late-phase weights.  In International Conference on Learning Representations, 2020.
  * [49] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen.  Max-deeplab: End-to-end panoptic segmentation with mask transformers, 2021\.
  * [50] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions, 2021.
  * [51] Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofei He, and Wei Liu.  Crossformer: A versatile vision transformer hinging on cross-scale attention, 2021.
  * [52] Xinjiang Wang, Shilong Zhang, Zhuoran Yu, Litong Feng, and Wei Zhang.  Scale-equalizing pyramid convolution for object detection.  CVPR, 2020.
  * [53] Yeming Wen, Dustin Tran, and Jimmy Ba.  Batchensemble: an alternative approach to efficient ensemble and lifelong learning.  In International Conference on Learning Representations, 2019.
  * [54] Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton.  Hyperparameter ensembles for robustness and uncertainty quantification.  Advances in Neural Information Processing Systems, 33:6514–6527, 2020.
  * [55] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun.  Unified perceptual parsing for scene understanding, 2018.
  * [56] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers, 2021.
  * [57] Yuhui Yuan, Xiaokang Chen, Xilin Chen, and Jingdong Wang.  Segmentation transformer: Object-contextual representations for semantic segmentation, 2021.
  * [58] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network, 2017.
  * [59] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H. S. Torr, and Li Zhang.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers, 2021.
  * [60] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Semantic understanding of scenes through the ade20k dataset, 2018.
  * [61] Zhi-Hua Zhou, Jianxin Wu, and Wei Tang.  Ensembling neural networks: many could be better than all.  Artificial intelligence, 137(1-2):239–263, 2002.

Figure 2: Weight sharing policies. Different color indicates different set of
parameters.

## Supplementary Materials

We first provide more information about the datasets used to evaluate
SenFormer performances (Section A). In an attempt to gain more insights about
our self-ensemble approach, additional experiments and discussions on
SenFormer are presented. Eventually, we provide qualitative segmentation
results of SenFormer.

Figure 3: FeaturesFusionBaseline architecture.

##  Appendix A Datasets Descriptions

ADE20K [60] is a scene parsing dataset built from ADE20K-Full dataset, where
$150$ classes were selected to constitute SceneParse150 challenge. It consists
of $20,210$ training images, $2,000$ validation images, and covers $150$ fine-
grained labeled classes. Models are trained for $160$k iterations, with a
batch size of $16$ and a crop size of $640\times 640$ pixels when using Swin-B
and Swin-L as backbone; otherwise a crop size of $512\times 512$ is used.

COCO-Stuff-10K [2] is a subset of the COCO dataset [32] for semantic
segmentation. It consists of $9$k images for training and $1$k images for
testing, covering $171$ semantic-level categories. For training, all SenFormer
models were trained for $80$k iterations, with a batch size of $16$ and a crop
size of $512\times 512$.

Pascal Context [37] training set contains $4,996$ images covering $59$ classes
and the testing set contains $5,104$ images. The data come from the PASCAL VOC
2010 contest [13], where annotations for the whole scene have been added. All
SenFormer models were trained for $40$k iterations, with a batch size of $16$
and a crop size of $480\times 480$.

Cityscapes [9] is a high-resolution dataset of $5,000$ street-view images with
$19$ semantic classes. Conventionally, the dataset is split into a training
set of $2,975$ images and a validation set of $500$ images. All SenFormer
models were trained for $100$k iterations, with a batch size of $8$ and a crop
size of $512\times 1024$.

##  Appendix B Detailed Learners Architecture

Hereafter, we detail the architecture of the learners. Each decoder is a
transformer composed of $L$ layers. In a nutshell, a single Transformer
Decoder block consists of three successive operations: Cross-Attention (CA),
Self-Attention and Multi-Layer Perceptron layers.

First, the multi-scales features $\\{P_{2},P_{3},P_{4},P_{5}\\}$ are reshaped
into a set of tokens $\\{z_{2},z_{3},z_{4},z_{5}\\}$ where
$z_{i}\in\mathcal{R}^{n_{i}\times d}$, $n_{i}=\frac{HW}{2^{i}}$ is the number
of token and $d$ is the numbers of channels in all the feature maps of the
FPN.

In the CA layer, the FPN’s token features $z_{i}$’s are linearly transformed
through matrix multiplication to acts as the keys and values, and the class
embedding $\textbf{cls}_{i}$’s as the queries. For $i\in\\{2,3,4,5\\}$ the CA
operation is as follows:

$\displaystyle K=z_{i}W_{K},V=z_{i}W_{V},Q=LN(\textbf{cls}_{i})W_{Q},$ (6)
$\displaystyle\text{where }W_{K},W_{V},W_{Q}\in\mathcal{R}^{D\times D}$
$\displaystyle
CA(Q_{i},P_{i})=\textbf{cls}_{i}+softmax(\frac{QK^{T}}{\sqrt{D}})V$

where softmax denotes the softmax function applied along the last dimension
and LN denotes the ”Layer Normalization” operation. This operation aims at
distilling the knowledge gained by the backbone contained in the features
$P_{i}$ into the class embeddings $\textbf{cls}_{i}$. This way, the model will
retain the information of _what is it for a feature token $z_{i}$ to represent
a specific class._ Indeed, since the class embedding vector is used to get the
final prediction via a dot product with the features tokens, during the back-
propagation the class embedding tensor $cls_{i}^{k}$ (that represents the
$k^{th}$ class) will be encouraged to become similar to the tokens of $z_{i}$
that correspond to the $k^{th}$ class, according to the ground truth
segmentation maps.

Then, the Self-Attention layer enables sharing the information acquired during
the Cross-Attention across the class embedding vectors. Eventually, an Multi-
Layer Perceptron layer is used to propagate the information across the channel
dimensions.

Overall, each decoder is composed of $L$ layers of decoder blocks, and its
prediction is obtained via a dot product between the class embeddings
$\textbf{cls}_{i}$ and the corresponding feature pyramid feature $P_{i}$.

###  B.1 Feature Pyramid Network

The set of features extracted by the backbone $\\{C_{2},C_{3},C_{4},C_{5}\\}$,
is enhanced by the FPN [31] to obtain a feature pyramid that has strong
spatial and semantics at all scales. To do so, $\\{C_{2},C_{3},C_{4},C_{5}\\}$
undergo a linear projection to set the channel dimension of each scale to a
fixed size denoted as $d$(fixed to $512$). Consecutive levels are then
upsampled to the same size and merged by element-wise addition. Eventually,
the merged features are processed by a $3\times 3$ convolution to alleviate
the aliasing effect of the upsampling. In sum, the output set of features of
the FPN $\\{P_{2},P_{3},P_{4},P_{5}\\}$ is obtained by:

$\displaystyle P_{i}^{\prime}$ $\displaystyle=\textit{Conv}_{1\times
1}\left(C_{i}\right),\quad i\in\\{2,3,4,5\\}$ (7) $\displaystyle P_{5}$
$\displaystyle=P_{5}^{\prime}$ $\displaystyle P_{i}$
$\displaystyle=\textit{Conv}_{3\times
3}\left(P_{i}^{\prime}+\textit{Upsample}(P_{i+1}^{\prime})\right),\quad
i\in\\{2,3,4\\}$

with $\textit{Conv}_{\cdot\times\cdot}$ being a convolution with
$\cdot\times\cdot$ kernel and Upsample being the nearest neighbor upsampling
operation.

####  B.1.1 Feature Pyramid Network Transformer

We empirically found that while introducing marginal changes to the
implementation and a minimal computational cost, replacing the $3\times 3$
convolution by a transformer block in the FPN increases the segmentation
performance for SenFormer, see Table 13.a. In practice, we use the window-
based transformer block (denoted as WTB) of [34] to limit the memory footprint
overhead. We name this enhanced FPN version as Feature Pyramid Network
Transformer-enhanced (FPNT), where

$\displaystyle P_{i}^{\prime}$ $\displaystyle=\textit{Conv}_{1\times
1}\left(C_{i}\right),\quad i\in\\{2,3,4,5\\}$ (8) $\displaystyle P_{5}$
$\displaystyle=P_{5}^{\prime}$ $\displaystyle P_{i}$
$\displaystyle=\textit{WTB}\left(P_{i}^{\prime}+\textit{Upsample}(P_{i+1}^{\prime})\right),\quad
i\in\\{2,3,4\\}$

Table 13: Ablation studies related to SenFormer architectural choices. Models
are trained on ADE20K validation for 100k iterations with a ResNet-50 backbone
pretrained on ImageNet-1K [41]

method | mIoU
---|---
none | 38.15
FPN | 42.7
FPNT | 43.6
(a)

method | mIoU
---|---
Post-Norm | 42.63
Pre-Norm | 43.6
(b)

##  Appendix C Additional Experiments

Table 14: Comparison of the ”repeated” sharing strategy vs no weight sharing (none) on multiple benchmark datasets.  backbone | sharing | params. | ADE20k | Pascal | COCO | Cityscapes
---|---|---|---|---|---|---
ResNet-50 | none | 144M | 44.6 | 53.2 | 39.0 | 78.8
repeated | 55M | 44.3 | 53.2 | 40.0 | 78.8
ResNet-101 | none | 163M | 46.5 | 55.1 | 39.6 | 80.3
repeated | 79M | 46.9 | 54.6 | 41.0 | 79.9
Swin-L | none | 314M | 53.1 | 62.4 | 49.1 | 82.2
repeated | 233M | 53.1 | 63.1 | 49.8 | 82.8

Weight sharing. In Table 14 we compare on multiple datasets SenFormer
performance when no weight sharing is used (none) and the default weight
sharing setting, where the same decoder block is recursively used $L$ times
(repeated). Despite having significantly fewer parameters, our weight sharing
strategy has similar performance to when no sharing is used.

Multi-scales feature generation. In Table 13.a we demonstrate the benefit of
the feature pyramid network transformer (FPNT) over the traditional FPN for
SenFormer. We observe that FPNT is $0.9$ mIoU better than the FPN baseline
(with ResNet50 as backbone). Also, not using any FPN-like method significantly
reduces the performance, which can be explained by the fact that each learner
must receive semantically and spatially strong features.

##  Appendix D Additional discussions

This section further discusses the impact of the learners’ architectural
choices on their predictions. In particular, we study the behavior of the
class embeddings after convergence. Furthermore, we show how the ”pre-norm”
strategy may help the class embeddings act as a memory bank.

Class embeddings acting like a memory bank. In SenFormer, for a given learner,
each class embedding vector represents a unique class and is used to produce
the learner’s prediction for that given class. Hence, we expect the class
embedding to retain specific information about that class. Accordingly, at the
end of the training, the different learners should converge to different
vectors (as they represent different classes).

Figure 4 shows the distribution density of the cosine similarity between the
different class embeddings of SenFormer trained on ADE20K with ResNet-50 as
backbone; i.e. for the $i$-th learner the distribution of the following set
$\left\\{\cos(cls_{i}^{k},cls_{i}^{l}),\;(k,l)\in\\{1,..,N_{c}-1\\}\times\\{k,..,N_{c}\\}\right\\}$.

Figure 4: Distribution of the cosine similarity between the different class
embeddings for each learner in SenFormer.

As expected, although ADE20K has a large number of classes, the density curves
are close to the origin, suggesting that the different class embeddings
converge to different vectors. Hence, for a given learner $i$, the class
embeddings $\textbf{cls}_{i}=[cls^{1}_{i},\dots,cls^{N_{c}}_{i}]$ of SenFormer
effectively acts as a _memory bank_ that is used by the decoder to assess how
likely a given feature token of $P_{i}$ represents a certain class.

Pre-Norm vs Post-Norm. Early versions of transformers [47] as well as recent
applications to object detection and segmentation [3, 7] applied layer
normalization after the skip connection (post-norm), while recent
implementations tend toward using pre-normalization setting. Table 13.b shows
that a pre-normalization strategy performs best for our architecture. We
believe that, by leaving the skip connection pathway unaltered, the pre-norm
setting ease the information flow from the ground truth supervision to the
input class embeddings of the learner, therefore fostering the _”memory bank”_
mechanism described above. It may also explain why [3, 7] do not benefit from
the use of pre-norm. Indeed, in DETR and MaskFormer, each query embedding
vector (the equivalent of our class embedding) does not correspond to a unique
class, but is rather dynamically routed to a class by using a bipartite
matching, therefore the learned embeddings do not act as a ”memory bank”.

##  Appendix E Visualization

Figure 5, we visualize sample segmentation predictions of SenFormer with
Swin-L backbone on ADE20K validation.

Figure 5: Visualization of SenFormer segmentation predictions on ADE20K
validation with Swin-L backbone.


# Fast Approximate Spectral Normalization for Robust Deep Neural Networks

Zhixin Pan and Prabhat Mishra  Department of Computer & Information Science &
Engineering
University of Florida, Gainesville, Florida, USA

###### Abstract

Deep neural networks (DNNs) play an important role in machine learning due to
its outstanding performance compared to other alternatives. However, DNNs are
not suitable for safety-critical applications since DNNs can be easily fooled
by well-crafted adversarial examples. One promising strategy to counter
adversarial attacks is to utilize spectral normalization, which ensures that
the trained model has low sensitivity towards the disturbance of input
samples. Unfortunately, this strategy requires exact computation of spectral
norm, which is computation intensive and impractical for large-scale networks.
In this paper, we introduce an approximate algorithm for spectral
normalization based on Fourier transform and layer separation. The primary
contribution of our work is to effectively combine the sparsity of weight
matrix and decomposability of convolution layers. Extensive experimental
evaluation demonstrates that our framework is able to significantly improve
both time efficiency (up to 60%) and model robustness (61% on average)
compared with the state-of-the-art spectral normalization.

###### Index Terms:

Deep learning, adversarial attack, approximate algorithms, spectral
normalization

##  I Introduction

Deep neural networks (DNNs) are are widely used in machine learning field.
They can express or simulate a wide variety of intrinsic functionalities
including classification, regression, reconstruction, etc. The flexibility of
DNNs also enables their different variations to be successfully employed in
diverse applications. The adversarial attack proposed by Szegedy et al.[1]
revealed the vulnerability of most existing neural networks against
adversarial examples. The difference between adversarial samples and the
original sample can hardly be distinguished by naked eyes, but will lead the
model to make an incorrect prediction with high confidence. As shown in Figure
1, a human-invisible noise was added to input traffic sign image. While a pre-
trained network can successfully recognize the original input as a stop sign,
the same network will incorrectly classify it as a yield sign if the input is
perturbed with well-crafted noise. There are many such real-world examples of
malicious (adversarial) attacks. In order to design robust DNNs, it is
critical defend against adversarial attacks.

Figure 1: Example adversarial attack on autonomous driving

While there are many promising defense strategies, most of them are designed
for specific attack algorithms, which severely restricts their applicability.
Spectral normalization [2] has gained significant attention because it is
algorithm-agnostic and can reduce DNNs’ sensitivity to input perturbation. A
major challenge with this method is that it provides a trade-off between
computation time and numerical accuracy for computing the spectral norm of
DNNs’ weight matrix. Specifically, it applies one iteration of power method
for each individual layer, which achieves poor accuracy in most cases.
Moreover, it introduces high computation overhead when dealing with large
convolution kernels. In reality, power iteration may not numerically converge
to the desired result in some specific scenarios.

In this paper, we propose a fast and dependable acceleration approach for
spectral normalization. We have evaluated the effectiveness of our proposed
approach using several benchmark deep neural networks along with datasets.
Specifically, this paper makes the following three important contributions.

  1. 1.

To the best of our knowledge, we are the first to utilize spatial separation
of convolution layers for regularized training.

  2. 2.

We develop a fast approximate spectral norm computing scheme for convolution
layers, which is based on Fourier transform and can be easily extended to
fully-connected layers.

  3. 3.

We demonstrate the feasibility of our approach with experiments on a variety
of classic network structures and dataset. Experimental results demonstrate
the improvement of DNNs’ robustness against adversarial attack through
extremely low attack success rate for bounded attack and high distortion for
unbounded attack.

The rest of this paper is organized as follows: We firstly provide background
on adversarial attack and spectral norm regularization in Section 2. Section 3
describes our proposed method where we will demonstrate detailed algorithm
workflow. Experimental results are provided in Section 4, and we conclude the
paper in Section 5.

##  II Background and Related Work

In this section, we provide relevant background before describing related
efforts. First, we describe three types of layers in a DNN. Next, we define a
given model’s stability using linear algebra concepts. Finally, we discuss
related efforts to motivate the need for our proposed approach.

###  II-A Three Types of Layers in DNNs

DNN consists of the following three types of layers: fully connected layer,
convolution layer and activation functions.

Linear Layer:  In this layer, each output node is nothing but a weighted sum
of inputs as shown in Figure 2. If we consider all equations and view it as a
linear system, it can be represented in the form of matrix multiplication.

Figure 2: Fully connected layer in DNNs

Convolution Layer: Convolution layers are widely applied in computer vision
tasks, where a convolution kernel $K$ will slide along the surface of input
feature map, and each weighted sum is stored into an entry of the output
feature map. We can represent it as matrix multiplication between the input
vector and a doubly block circulant matrix (which is called the convolution
matrix of $K$). Figure 3 shows a trivial example to illustrate this idea.

Figure 3: Convolution layer in DNNs

Activation Functions:  Activation functions are inserted between consecutive
network layers to induce nonlinearity to allow DNNs complete nontrivial tasks.
They are usually selected as piecewise linear functions, such as ReLU, maxout,
and maxpooling. Since adversarial examples always reside in the close
neighborhood of a given input, activation function can be approximately
considered as a linear function in this case, which is compatible with matrix
representation.

###  II-B Stability of DNNs

Since we have already shown the close relationship between DNNs’ layer and
matrix multiplication, we can interpret adversarial attacks into the language
of linear algebra. Assume that our model’s interpreted matrix (we will call
this weight matrix in the rest of this paper) is $W\in\mathbb{R}^{m\times n}$,
and there is a well-crafted noise $\xi\in\mathbb{R}^{n}$ added to our input
vector $x\in\mathbb{R}^{n}$, then we measure change of output by the following
relative error $\epsilon\in\mathbb{R}$:

$\epsilon=\frac{||W\cdot(x+\xi)-W\cdot
x||_{2}}{||\xi||_{2}}=\frac{||W\cdot\xi||_{2}}{||\xi||_{2}}$

Notice the spectral norm $\sigma(W)$ of matrix $W$ is defined as:

$\sigma(W)\triangleq\max\limits_{\xi\neq\textbf{0}}\frac{||W\cdot\xi||_{2}}{||\xi||_{2}}$

Therefore, spectral norm of matrix $W$ gives a tight upperbound of given
layer’s stability, and a small value of $\sigma(W)$ indicates this model’s
insensitivity to the perturbation of input $x$.

###  II-C Related Work

Based on matrix representation of DNN’ forward pass, recent research efforts
have developed several countermeasures, but many of them failed to protect
against strong attacks [3, 4, 5, 6, 7]. Spectral Normalization (SN) [2] is one
of the most promising defense strategies because it is algorithm-agnostic
approach and is able to enhance DNNs’ robustness against adversarial attacks.
During model training, the key idea of spectral normalization is to append the
spectral norm of each layer to loss function as penalty term, so that it can
reduce DNNs’ sensitivity to input perturbation by minimizing every layer’s
spectral norm. The loss function is:

$J=\frac{1}{N}\sum\limits_{i=1}^{N}L(f(x_{i}),y_{i})+\frac{\lambda}{2}\sum\limits_{k=1}^{K}\sigma(W^{k})^{2}$

Here, $x_{i}$ is a training sample, the label of $x_{i}$ is denoted as
$y_{i}$, the total number of training samples in a batch is denoted as $N$,
$L$ is the dissimilarity measurement and is frequently selected to be cross
entropy or squared $l_{2}$ distance, $\sigma(W^{k})$ is the spectral norm of
$k$-th layer, $K$ is the total number of layers, and
$\lambda\in\mathbb{R}^{+}$ is a regularization factor.

The state-of-the-art method in [2] applied power iteration to approximate
$\sigma(W^{k})$, then perform stochastic gradient descent (SGD) to train the
DNN, as shown in Algorithm 1.

Algorithm 1 State-of-the-art: Spectral Normalization (SN)

1:for each iteration of SGD do

2: for $k=1$ to $K$ do

3: for a sufficient number of iterations do

4: Apply power iteration to compute $\sigma(W^{k})$

5: Add $\sigma(W^{k})$ to loss function

6: end for

7: end for

8: Compute gradient of modified loss function

9: Update parameters by back propagation

10:end for

This spectral normalization (SN) algorithm suffers from high computation
complexity. Consider a convolution layer with $a$ input channels, $b$ output
channels, and a convolution kernel in a size of $w\times h$. The corresponding
weight matrix will be at least in a size of $b\times awh$. The exact
computation of its spectral norm requires SVD decomposition, whose complexity
is $\bm{O}(min(m^{2}n,n^{2}m))$ for a $m\times n$ matrix, Therefore the time
complexity in our task is $\bm{O}(min(a^{2}bw^{2}h^{2},ab^{2}wh))$, which is
infeasible to apply during network training. In order to make it run in a
reasonable time, the authors in [2] use only a few iterations of power
iteration method to approximate the spectral norms which leads to very coarse
approximation. As a result, it significantly compromises its robustness goal.
Our proposed method provides a fast approximate algorithm for spectral
normalization through synergistic integration of layer separation and Fourier
transform as described in the next section.

##  III Fast Approximate Spectral Normalization

Figure 4 shows an overview of our proposed framework for fast spectral
normalization (FSN). The major steps of our framework is outlined in Algorithm
3. Our proposed framework consists of two strategies to accelerate training
process: layer separation and Fourier transform. The remainder of this section
describes these strategies in detail.

Figure 4: Comparison of our proposed framework with the state-of-the-art. We
utilize layer separation to decompose large and multi-dimension layers into
1-D layers, then apply Fourier transform on each of the kernels.

###  III-A Layer Separation

The reason for computing spectral norm being so expensive is the size of
weight matrix. Here we apply layer separation to reduce the time complexity
for this task. First, a 2-D filter kernel $K$ is said to be $separable$ if it
can be expressed as the outer product of one row vector $\bm{r}$ and one
column vector $\bm{c}$. But vectors are special cases of matrix, and their
outer product is equivalent to their convolution. Then due to the
associativity of convolution:

$\begin{split}&K=\bm{r}\times\bm{c}=\bm{r}*\bm{c}\\\
\Rightarrow{}&A*K=A*(\bm{r}*\bm{c})=(A*\bm{r})*\bm{c}\end{split}$

A famous example will be the Sobel kernel, which is widely used to approximate
the gradient value of image brightness function:

$\begin{bmatrix}-1&0&1\\\ -2&0&2\\\ -1&0&1\end{bmatrix}=\begin{bmatrix}1\\\
2\\\ 1\end{bmatrix}\times\begin{bmatrix}-1&0&1\\\ \end{bmatrix}$

Once the given convolution layer is separable ($\operatorname{rank}(W)=1$), we
can always decompose it into 2 1-D layers but achieve the same effect. This
time we have 2 consecutive convolution layers, and their kernel size will be
$w\times 1$ and $1\times h$. We can compute each layer’s spectral norm
separately and then append them to the loss function together.

What about inseparable layers? In this case, our method utilize the SVD
decomposition of kernel filter to transfer it into its low-rank approximation.
Here, we are performing SVD decomposition on kernel matrix instead of the
doubly block circulant matrix. Kernel matrix encountered in modern CNNs are
usually in the form of $3\times 3$ or $5\times 5$, time cost for SVD
decomposition on these tiny matrices is trivial. Also, applying low-rank
approximation of kernel filter is reasonable when it comes to real situations.
In CNN applications, especially in computer vision areas, many frequently used
feature extraction kernels like edge detection filters are trained to have one
dominant eigen direction, where the largest singular value is far larger than
the inferior ones, one extreme example is the Sobel kernel mentioned above.
Under this circumstance, the deviation induced by low-rank approximation can
be neglected.

###  III-B Fourier Transform

It is a well known fact that the spectral norm of a 2D convolution kernel $K$
is exactly the largest eigenvalue of $A^{\top}A$, where $A$ is the
corresponding convolution matrix of $K$. In the baseline method illustrated in
Algorithm 1, power iteration method was applied. But power method, as a
fundamental numerical method for computing eigenvalues, usually takes too many
iterations until acceptable accuracy is obtained. Also, it assumes the weight
matrix has an eigenvalue that is strictly greater in magnitude than the
others, and the initial random vector should contain an nonzero component in
the direction of a dominant eigenvector. If the above assumptions fails, the
power method may not converge! Instead, we apply the following theorem [8] to
calculate the spectral norm of a convolution kernel fast.

Theorem 1: For any convolution matrix formed by kernel $K$, the eigenvalues of
the convolution matrix are the entries of 2D Fourier transform of $K$, and its
singular values are their magnitudes.

Proof: Assume $K$ be a kernel matrix, and $A$ is the convolution matrix of $K$
(as described in Figure 3). Now the task is to determine the singular values
of a doubly block circulant matrix $A$. First, we define

$Q\triangleq\frac{1}{n}(F\otimes F)$

where $F$ is the Fourier matrix. To complete the proof, we use the following
lemma.

Lemma 1: For any doubly block circulant matrix $A$, the eigenvectors of $A$
are the columns of $Q$, and $Q$ is unitary. (J.Toriwaki(1989) in [9])

Based on the above lemma, we know $A$ can always be decomposed into its
eigenvalue decomposition $A=Q*DQ$ where D is a diagonal matrix. Now we show
that $A$ is normal:

$\begin{split}&AA^{\top}=AA^{*}=Q^{*}DQQD^{*}Q\\\
=&Q^{*}DD^{*}Q=Q^{*}D^{*}DQ=Q^{*}D^{*}QQ^{*}DQ\\\
=&A^{*}A=A^{\top}A\end{split}$

Since the singular values of any normal matrix are the magnitudes of its
eigenvalues (Johnson (2012), page 158 [10]) and we have shown that $A$ is
normal, by applying the Lemma 1 we complete the proof. $\blacksquare$

Therefore, we can calculate the spectral norm of target convolution kernel by
Fourier transform, and this approach is depicted in Algorithm 2.

Algorithm 2 Algorithm for computing spectral norms

Input: convolution kernel $K$
Output: $\sigma$, the spectral norm respected to $K$.

1:$k\leftarrow$ 2D Fourier transform of $K$

2:$k^{\prime}\leftarrow$ set all the entries of $k$ to zero except the one
with maximum absolute value

3:$K^{\prime}\leftarrow$ 2D inverse Fourier transform of $k^{\prime}$

4:$\sigma\leftarrow$ largest entry inside $K^{\prime}$.

###  III-C Activation Functions

By exploiting layer separation and Fourier transform, the computation of
spectral norm for linear and convolution layers can be solved efficiently as
shown in Algorithm 2. When it comes to activation layers, there is no inherent
way to represent activation layers in matrix format. Activation functions are
deployed in neural networks to induce nonlinearity, therefore it is impossible
to view an activation function as a linear transformation.

To perform our method with activation functions, instead of parsing it into
matrix and compute spectral norm, we take their Lipschitz constants into
consideration. A function $f$ defined on X is said to be $K-$Lipschitz if

$\forall x_{1},x_{2}\in\textbf{X},\,|f(x_{1})-f(x_{2})|\leq K|x_{1}-x_{2}|$

The smallest possible constant $K$ for $f$ is also called Lipschitz norm,
which reflects how expansive function $f$ is. The Lipschitz norm upper bounds
the relationship between input perturbation and output variation for a given
distance. Notice the similarity between the definition of spectral norm and
Lipschitz norm. Actually, spectral norm of matrix $W$ is essentially the
Lipschitz norm of function $f$ if $f(x)=Wx$.

It has been proven that most activation functions such as ReLU, Leaky ReLU,
SoftPlus, Tanh, Sigmoid, ArcTan or Softsign, as well as max-pooling, are short
maps [11], i.e, they have a Lipschitz constant equal to 1. We refer the reader
to [12] for detailed proof on this subject. As a result, the output variation
induced by activation layers is already restricted by a constant number.
Therefore, there is no need to spare extra effort to append regularization
terms for activation functions to loss function, since it will not get changed
during training. Moreover, the constant number is 1, which means activation
functions will neither expand or contract the variation of layer outputs. In
other words, we can omit activation layers when evaluating the stability of
DNNs.

###  III-D Complexity Analysis

The complete framework for approximate spectral normalization is depicted in
Algorithm 3. For our target problem, based on the discussion in Section II-C,
we decompose the $b\times awh$ weight matrix into 2 matrices, $b\times aw$ and
$b\times ah$ for each. Then we apply Fourier transform on each separated
matrix. In our experiment fast Fourier transform (FFT) is applied with
$\bm{O}(mnlog(mn))$ time complexity for $m\times n$ matrix. So the proposed
method’s complexity is $O(abw\log(abw)+abh\log(abh)))$.

Algorithm 3 Proposed: Fast Spectral Normalization (FSN)

1:for each iteration of SGD do

2: Compute the gradient of general loss function as usual

3: for $k=1$ to $K$ do

4: if Convolution Layer then

5: Perform layer separation

6: Form the corresponding convolution matrix

7: end if

8: if Linear Layer then

9: Form the corresponding weight matrix

10: end if

11: Perform Algorithm2 to compute $\sigma(W^{k})$

12: Add $\sigma(W^{k})$ to loss function

13: end for

14: Update parameters using modified gradient

15:end for

##  IV Experiments

We evaluated the effectiveness of our optimized spectral norm regularization
framework to confirm its time-efficiency and ability of enhancing DNNs’
robustness.

###  IV-A Experiment setup

Experiments were conducted on a host machine with Intel i7 3.70GHz CPU, 32 GB
RAM and RTX 2080 256-bit GPU. We developed code using Python for model
training. We used PyTorch as the machine learning library. For adversarial
attack algorithms, we utilized the Adversarial Robustness 360 Toolbox (ART)
[13]. Based on this environment, we considered the following two settings for
our experimental evaluation:

  1. 1.

The VGG16 classifier for German Traffic Sign Benchmark (GTSB) dataset [GTSB].

  2. 2.

The Lenet-5 network for Udacity Self-Driving (USD) dataset [USD].

For each setting we compare the following three approaches:

  1. 1.

Normal: Ordinary training without regularization, which is considered as
control group.

  2. 2.

SN: State-of-the-art approach (Algorithm 1) with spectral normalization [2].

  3. 3.

FSN: Our proposed approach (Algorithm 2) with fast and robust spectral
normalization.

As for result evaluation, we first evaluated the functionality of them by
reporting their accuracy and training time. Next, both bounded and unbounded
adversarial attacks were deployed to test their robustness. Finally, we tuned
the regularizer factor $\lambda$ for FSN to demonstrate its stability across a
wide variety of different $\lambda$ values.

###  IV-B Case Study: German Traffic Sign Benchmark (GTSB)

We train VGG16 on GTSB dataset. GTSB is a large and reliable benchmark which
contains more than 50000 traffic sign image samples over 43 classes. VGG16
[Alex] contains 16 hidden layers (13 convolution layers and 3 fully connected
layers). One advantage of VGG16 is the replacement of larger convolution
kernels ($11\times 11,7\times 7,5\times 5$) with consecutive $3\times 3$
convolution kernels, which makes it coincidentally suitable for our framework
since small size of convolution kernel implies less time cost for low-rank
approximation. We choose a mini-batch size of 64. For each setting, we train
the network with 200 epochs. For every 10 epochs, we randomly shuffle and
split $80\%$ as training set and $20\%$ as test set, and then report the
training time along with test performance.

Table I: Training time and test accuracy of VGG16 Methods |  Time(s/epoch) |  Best Test Accuracy(%)
---|---|---
Normal | 14.5053 | 83.45
SN | 29.1126 | 91.72
FSN | 18.1345 | 95.46

Table I indicates the basic functionality performance of three training
methods. We also plot the training curve for accuracy updates in Figure 5. In
general, all three methods possess good training accuracy. The Normal approach
achieved the fastest speed but when it comes to test accuracy, it lagged
behind the other two due to spectral normalization’s ability to enhance
model’s generalizability [2]. SN promote the test accuracy by $8.27\%$ but
also yields the highest time cost. FSN further improves the test accuracy by
$3.66\%$ and reduce average training time by $60.5\%$ compared with to state-
of-the-art (SN).

Figure 5: Accuracy curve of VGG16 on GTSB

Table II shows the performance of three models against bounded attacks. We
choose four different attack algorithms: FGSM [6], CW-$L_{2}$ [Carlini], JSMA
[14] and DeepFool [4]. For consistency $\lambda=0.01$ for both SN and FSN
here. The hyperparameters for attacks are provided in the table.

Table II: Bounded Attack on GTSB Methods |  N/A |  FGSM ($\varepsilon$ = .1) |  CW-$L_{2}$ (c = .01) |  JSMA $(\theta,\gamma=.1,1)$ |  DeepFool $\varepsilon=1e-6$
---|---|---|---|---|---
Normal | 83.45 | 61.04 | 45.62 | 77.58 | 40.06
SN | 91.72 | 88.25 | 76.31 | 88.23 | 69.30
FSN | 95.46 | 92.92 | 87.05 | 91.57 | 88.94

As we can see, our proposed method (FSN) provides the best robustness. The
Normal and SN method appeared fragile in the face of powerful attacks like
CW-$L_{2}$ and DeepFool, while FSN still retained an acceptable accuracy. For
lightweight attacks, especially gradient-based ones (like FGSM), FSN is almost
unaffected. For unbounded attack, we applied unbounded incremental
$\varepsilon$ value in FGSM from 0 to 1. It starts to break detection
(accuracy $<$ 50%) as presented in Figure 6(a), but the bisection parameter
value of FSN($\varepsilon=.64$) has to be nearly three times large of that for
Normal method($\varepsilon=.21$). Finally, FSN’s broad robust performance
across a wide range of hyperparameter $\lambda$ is demonstrated in Figure 6(b)
where we plot accuracy under all attacks with $\lambda$ varying from 0.01 to
0.1. The average accuracy here is $91.46\%$ with standard deviation of 0.0729.
In the worst case, model trained with FSN still performed well with accuracy
above $85\%$.

(a) Performance of three models towards unbounded FGSM attack

(b) FSN’s performance with different $\lambda$ and attack algorithm

Figure 6: Performance with varying hyperparameter (GTSB)

###  IV-C Case Study: Udacity Self-Driving (USD)

We trained Lenet-5 on USD’s steering angle image dataset which has more than
5000 autonomous vehicle’s real-time screenshot along with labels indicating
the corresponding “Left”, “Right” or “Center” prediction for next wheel
steering operation. Compared with VGG16, Lenet-5 is a lightweight network with
only five convolution layers and one fully connected layer. The training
process consists of 500 epochs with a mini-batch size of 128. For every 50
epochs, we randomly shuffle sample and split $80\%$ as training set and $20\%$
as test.

Table III: Training time and test accuracy of Lenet-5 Methods |  Time(s/epoch) |  Best Test Accuracy(%)
---|---|---
Normal | 0.13369 | 99.71
SN | 0.25870 | 98.57
FSN | 0.18284 | 99.43

Table III and Figure 7 present performance results. Again, Normal method
provides the fastest training speed, while FSN is nearly $41.4\%$ faster than
SN. Note that the improvement of speed here is not as high as we got from GTSB
case, and we consider this difference of acceleration amplitude is caused by
more convolution layers involved in VGG16 than Lenet-5. For accuracy, due to
Lenet-5’s excellent capability in image recognition, all three models achieved
high level of accuracy.

Figure 7: Accuracy curve of Lenet-5 on USD

The robustness is evaluated against four attack algorithms. As expected,
Normal method behaved extremely vulnerable. Even in $\varepsilon=0.1$ case, a
$8.3\%$ accuracy against FGSM attack was obtained. FSN refined it to $71.4\%$
which gives an 61.1% average improvement on accuracy. When it comes to
unbounded attack, the Normal method’s outcome immediately drops below $50\%$
when $\varepsilon=0.1$, while SN and FSN extended it to $0.4$ and $0.5$,
respectively. Figure 8(b) shows the accuracy after applying various
regularization factor of $\lambda$. The average accuracy is 62.63% with
standard deviation of 0.2252. If we consider only CW-$L_{2}$ and DeepFool, the
average drops to $41.99\%$ with standard deviation of $0.0713$.

Table IV: Bounded Attack on USD Methods |  N/A |  FGSM ($\varepsilon$ = .1) |  CW-$L_{2}$ (c = .01) |  JSMA $(\theta,\gamma=.1,1)$ |  DeepFool $\varepsilon=1e-6$
---|---|---|---|---|---
Normal | 99.71 | 8.3 | 5.62 | 17.58 | 4.06
SN | 98.57 | 69.9 | 34.34 | 78.23 | 29.30
FSN | 99.43 | 71.4 | 40.05 | 82.34 | 34.94

(a) Performance of three models under unbounded FGSM attack

(b) FSN’s performance with different $\lambda$ and attack algorithm

Figure 8: Performance with varying hyperparameters (USD)

##  V Conclusion

Adversarial attacks are vital threats towards deep neural networks. In this
paper, we investigated adversarial attack scenarios and proposed a novel
regularized training strategy, the fast spectral normalization (FSN). Such a
neural network regularization technique made three important contributions.
(1) FSN utilized the spatial separation of convolution layers as well as
Fourier transform to drastically (up to $60\%$) reduce training time compared
with spectral normalization. (2) FSN is algorithm-agnostic, easy to implement,
and applicable across a wide variety of datasets. (3) Experimental evaluation
using three popular benchmarks demonstrated that models trained with FSN
provide significant improvement in robustness for bounded, unbounded as well
as transferred threats, which can protect applications from various
adversarial attacks. Compared with the state-of-the-art, our approach ensures
that the model works correctly unless the adversarial samples are drastically
different from the original samples. Our proposed regularized training
approach is expected to play a crucial role in improving the robustness of
future systems against adversarial attacks.

## References

  * [1] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” 12 2013.
  * [2] Y. Yoshida and T. Miyato, “Spectral norm regularization for improving the generalizability of deep learning,” 05 2017.
  * [3] N. Carlini and D. A. Wagner, “Towards evaluating the robustness of neural networks,” _CoRR_ , vol. abs/1608.04644, 2016.
  * [4] S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “Deepfool: a simple and accurate method to fool deep neural networks,” _CVPR_ , 11 2016.
  * [5] N. Papernot, P. D. McDaniel, and I. J. Goodfellow, “Transferability in machine learning: from phenomena to black-box attacks using adversarial samples,” _CoRR_ , vol. abs/1605.07277, 2016.
  * [6] I. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” 01 2015, pp. 1–10.
  * [7] A. Kurakin _et al._ , “Adversarial examples in the physical world,” in _5th International Conference on Learning Representations_ , 2017.
  * [8] H. Sedghi, V. Gupta, and P. M. Long, “The singular values of convolutional layers,” 2018.
  * [9] J. Toriwaki and H. Yoshida, _Fundamentals of Three-dimensional Digital Image Processing_. Springer, 2009.
  * [10] R. A. Horn and C. R. Johnson, _Matrix Analysis, 2nd Ed_. Cambridge University Press, 2012.
  * [11] A. Virmaux and K. Scaman, “Lipschitz regularity of deep neural networks: analysis and efficient estimation,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 3839–3848.
  * [12] Y. LeCun, _et al._ , “Deep learning,” _Nature_ , vol. 521, no. 7553, pp. 436–444, 2015.
  * [13] M.-I. Nicolae _et al._ , “Adversarial robustness toolbox v1.0.1,” _CoRR_ , vol. 1807.01069, 2018. [Online]. Available: https://arxiv.org/pdf/1807.01069
  * [14] R. Wiyatno and A. Xu, “Maximal jacobian-based saliency map attack,” _CoRR_ , vol. abs/1808.07945, 2018.