# Feature boosting with efficient attention for scene parsing

Vivek Singh<EMAIL_ADDRESS>Shailza Sharma<EMAIL_ADDRESS>Fabio Cuzzolin<EMAIL_ADDRESS>School of Engineering, Computing
and Mathematics, University of Plymouth, UK  School of Earth and Environment,
University of Leeds, UK  School of Engineering, Computing and Mathematics,
Oxford Brookes University, UK

###### Abstract

The complexity of scene parsing grows with the number of object and scene
classes, which is higher in unrestricted open scenes. The biggest challenge is
to model the spatial relation between scene elements while succeeding in
identifying objects at smaller scales. This paper presents a novel feature-
boosting network that gathers spatial context from multiple levels of feature
extraction and computes the attention weights for each level of representation
to generate the final class labels. A novel ‘channel attention module’ is
designed to compute the attention weights, ensuring that features from the
relevant extraction stages are boosted while the others are attenuated. The
model also learns spatial context information at low resolution to preserve
the abstract spatial relationships among scene elements and reduce computation
cost. Spatial attention is subsequently concatenated into a final feature set
before applying feature boosting. Low-resolution spatial attention features
are trained using an auxiliary task that helps learning a coarse global scene
structure. The proposed model outperforms all state-of-the-art models on both
the ADE20K and the Cityscapes datasets.

###### keywords:

Semantic segmentation , Convolutional neural network , Spatial attention ,
Channel attention , Scene parsing

##  1 Introduction

|  |
---|---|---
|  |
Figure 1: Samples images from the ADE20K dataset Zhou et al. [2017] to reflect
the complexity of unrestricted natural scenes.

Scene parsing is the task of breaking an image into its constituent components
and is crucial to reach a complete understanding of a visual scene. Recently,
semantic image segmentation has gained even more traction within the
artificial intelligence community because of the huge scope for a commercial
application, e.g., robotic vision, visual scene understanding, aerial image
analysis, autonomous driving, etc. Taghanaki et al. [2019], Liu et al. [2019].
The complexity of the segmentation task, however, can vary as a function of
the set of categories being considered for the segmentation and of how fine-
grained the description needs to be (e.g., exact vehicle model vs generic
’vehicle’ class, or breed of dog vs ’dog’). For an input image $I$ of size
$H\times W$ the segmentation output will retain the same size (i.e., $H\times
W$), but with each location encoding a label (represented by a number from $0$
to $L-1$) instead of color information.

The most recent scene segmentation methods are based on Fully Convolutional
Networks (FCNs) Long et al. [2015] which only use convolutional layers for
making the prediction where output contains class-wise probability
distribution for each input pixel. The biggest advantage of FCN architectures
is that they are scale-independent and computationally very efficient. FCN-
based models have been successful at tackling a number of semantic
segmentation task Lin et al. [2017a], Poudel et al. [2019], Wu et al. [2019],
especially when number of object classes are limited and scene complexity is
manageable. However, as the number of classes increases and unrestricted
natural scenes are considered, where dozens of objects are present at various
scales and orientations, even state-of-the-art methods struggle to perform
reliably. This is the case, for example, with the ADE20K dataset Zhou et al.
[2017]. Fig. 1 shows some images from ADE20K to illustrate the variability
exhibited by natural scenes in both indoor and outdoor settings.

The neighbourhood of each pixel plays a key role in identifying the class
label for that pixel. A key issue with images is that, due to perspective
projection, the same object can appear in different sizes or shapes. This, in
turn, greatly affects the shape and size of the ‘context window’ to be used
when considering neighbourhoods. A fixed-depth network with fixed-size
convolutional kernels simply cannot extract relationships between pixels when
objects can be present at different scale and orientations.

To address this problem, we propose an architecture that gathers multi-level
representations from different stages of feature extraction, providing rich
contextual information at different levels of abstraction. The proposed model
uses two attention modules: a Channel Attention Module (CAM) and a Spatial
Attention Module (SAM). SAM generates spatial attention maps which identify
the importance of each feature inside single feature map. Whereas, CAM
identifies importance of feature across channels and is applied at the last
stage of feature extraction. Before applying channel attention, features from
different stages of feature extraction are gathered ($f_{i}(x,y)$) which
provide feature vector ($f_{i}$) with multi-scale neighbour for each pixel
location $(x,y)$. Additionally, an auxiliary task is used to train the SAM to
learn the low-resolution semantic structure of the scene. The proposed model
outperforms all state-of-the-art approaches on the ADE20K Zhou et al. [2017]
and Cityscapes Cordts et al. [2016] benchmark datasets.

In summary, our main contributions are:

  * 1.

A novel feature extraction approach designed to learn and exploit _multi-scale
spatial context_ for scene parsing.

  * 2.

A novel _channel attention module_ is geared to learn the individual
contribution of each feature to the final semantic label. Additionally, a
simplified spatial attention module efficiently extracts the relevant
attention matrix.

  * 3.

A new learning mechanism which uses low-resolution semantic maps in an
auxiliary task to improve the training of the spatial attention module.

  * 4.

The proposed models shows significantly better performance with much lower
parameter count.

##  2 Related work

The image segmentation literature broadly focuses onto either model
architectures or spatial context learning mechanisms. The former group of
papers focus on techniques for extracting more task-relevant features to
improve model generalisation, whereas the latter group considers the relevance
of spatial features across adjacent pixels.

###  2.1 Segmentation architectures

The first successful application of Fully Convolutional Neural Network for
semantic segmentation is by Lon et al. Long et al. [2015]. Convolution was
used there to extract spatial features and pooling was used to reduce the size
of the feature maps. The paper used a deconvolution (up-sampling) operation on
the extracted feature to restore model output to the original input’s size.
Subsequent architectures used the encoder-decoder approach proposed in Noh et
al. [2015] for semantic segmentation. In these FCN-based networks the encoder
block extracts the features from the image while the decoder uses these
features to generate a semantic map. Architectures such as like U-net
Ronneberger et al. [2015], SegNet Badrinarayanan et al. [2017] and V-net
Milletari et al. [2016] all use this encoder-decoder concept as the backbone
of their designs. Both U-net Ronneberger et al. [2015] and V-net Milletari et
al. [2016] use skip connections between encoder and decoder stages to gather
multi-level feature representation. In SegNet Badrinarayanan et al. [2017],
rather than training an upsampling layer, the authors apply an ‘un-pooling’
operation in which feature indexes from the max pooling layers are used to
upsample the features.

Pyramid Scene Parsing Networks Zhao et al. [2017] use a Pyramidal Pooling
Module (PPM) to learn multi-scale feature representations, idea originally
proposed by Long et al. [2015]. The PPM applies average pooling on the final
feature maps with kernels of different sizes. Later outputs of average pooling
operations are concatenated together to provide a global prospective. Much
recent work uses PPMs as part their architecture Yu et al. [2018], Li et al.
[2018], Yang et al. [2018]. The recent DeepLabV3+ approach Chen et al. [2018]
proposes the use of atrous convolution in the encoder at different dilation
rates to generate multi-level context information, resulting in a very
effective way of improving model performance. The work in Hu et al. [2018]
incorporated a ‘Squeeze and Excite’ (SE) block in their architecture to add
global context but for classification and detection task. The method re-
calibrates the channel-wise feature response of each layer by explicitly
modeling the interdependence among different channels. RefineNet Lin et al.
[2017a], on its side, extracts features at different scales throughout the
network’s stages and uses this multiscale feature representation to predict
the final segmentation labels. Another successful architecture, termed UPerNet
Xiao et al. [2018], uses a unified perceptual parsing approach which tries to
learn as many visual concepts as possible from the given data. This work
leverages multiple dataset such as ADE20k Zhou et al. [2017], Pascal-Context
Mottaghi et al. [2014], Pascal-Part Chen et al. [2014], OpenSurfaces Bell et
al. [2013], and so on, as all these datasets can provide complementary
information about the images such as surface texture, objects, or parts. There
are other methods that use neural architecture search Zhang et al. [2021] or
model adaptation Fleuret et al. [2021].

###  2.2 Spatial context modeling

Spatial relationship modeling considers the dependencies among image pixels
and how these these dependencies affect the object structure and class label
for the given pixel. Both Wang et al. Wang et al. [2017] and Zhao et al. Zhao
et al. [2018] introduced a spatial attention method for image segmentation.
The model in Zhao et al. [2018], in particular, uses two attention branches to
capture both local and global spatial attention across features. Zhang et
al.’s EncNet Zhang et al. [2018], instead, uses context learning for modeling
spatial relationships. The paper also uses an auxiliary task to learn image-
level classes as, according to authors, this helps improve model performance.
A self attention-based context-learning approach was proposed by Yuan et al.
[2019] which led to state-of-the-art performance on the ADE20k dataset Zhou et
al. [2017]. A dual attention architecture was instead proposed in Fu et al.
[2019] which uses two distinct modules for spatial and channel attention. Both
attention modules are applied in parallel on the output of backbone model.
Attention is applied in Chen et al. [2016] to the scale of the input, using
different levels of scaling on the input image. The scaled versions of the
output are then merged together by the attention mechanism. Finally, Hu et al.
Li et al. [2019] used stack-based attention for CT scan images, in which each
slice of the stack gives global contextual information about the ground truth.

The proposed model is novel in both its architecture as well context modeling.
The architecture is designed with memory requirements in mind (as shown in
Figure 4b). Hence, SAM is applied on low resolution features and auxiliary
task is also used at same level. For context modeling, we gather mid-level
representation from multiple levels of feature extraction and merge them with
SAM output. Later, a novel and efficient CAM is applied to identify relevance
of each feature for label prediction.

##  3 Methodology

Figure 2: Complete architecture of the proposed Feature Boosting Network
(FBNet).

In semantic scene segmentation, the goal is to predict for each pixel of an
input image $I$ a label from a predefined set of classes $\\{0,...,L-1\\}$.
All recent methods learn semantic maps at lower resolution (generally 1/8 of
the input image size Fu et al. [2019], Yuan et al. [2019], Zhao et al.
[2017]), whereas previous approaches used to perform semantic segmentation at
the original resolution (which is generally detrimental to the quality of the
output). However, when output maps have lower resolution it becomes difficult
to correctly segment smaller objects/parts. To address this, our model learns
semantic structures at two resolutions: 1/8th and 1/4th of the original image.

###  3.1 Overall architecture

All semantic segmentation methods employ a backbone model to extract good-
quality feature representations. The most recent architectures use residual
learning-based backbones, such as ResNet He et al. [2016], ResNeSt Zhang et
al. [2020] and ResNeXt Xie et al. [2017]. The proposed architecture (Figure 2)
uses a ResNeSt backbone due to its superior feature learning capability.

Backbone feature extraction. The pooling operations in last three residual
blocks of ResNest are adjusted to ensure all feature maps are of the same
size. Hence, feature maps produced by the first residual block are 1/4th of
the original image size, and feature maps produced by the remaining three
blocks are 1/8th of the original (see Fig. 2). Additionally, a dilation rate
of 2 and 4 is used in the last two residual blocks to obtain a wider receptive
field. The final outputs of the residual blocks in the backbone are denoted by
$F_{1}^{1}$, $F_{2}^{1}$, $F_{3}^{1}$ and $F_{4}^{1}$ in Fig. 2, respectively,
in the order of their depth. The mid-level feature representations
($F_{1}^{1}$, $F_{2}^{1}$ and $F_{3}^{1}$) are transformed into more useful
abstract scene-level information by applying separate convolutional blocks.
Each of these block uses two sequential convolutional layers, each with $256$
kernels.

Spatial attention. Deeper features encode more abstract information about the
scene. Hence, it is reasonable to hypothesize that features from the last
residual block ($F_{4}^{1}$) are particularly suitable for learning spatial
context. In our model we apply a novel _Spatial Attention Module_ (SAM,
presented in Section 3.3), inspired by self-attention Lin et al. [2017b],
Vaswani et al. [2017], to feature map $F_{4}^{1}$. The module is designed to
reduce the computational cost and preserve the efficiency of the overall
network. In our experiments we found that increasing the complexity of the SAM
does not translate into higher model performance. Thus, in our model we
compute spatial attention at lower feature resolution using a smaller
attention module. This is computationally efficient while preserving spatial
invariance.

Our feature boosting mechanism is the composition of two operations: fusion
and channel attention. Namely, the feature maps outputted by the last three
residual blocks ($F_{2}^{2}$, $F_{3}^{2}$ and $F_{4}^{2}$) are up-sampled to
match the resolution of the first (shallower) feature map ($F_{1}^{2}$). As
these features are extracted from different depth levels of the backbone, they
represent multiple scales of receptive field, with the shallower map encoding
‘local’ feature relationships and the deeper feature maps representing
‘global’ relationships between pixels. All such feature maps are concatenated
together along with SAM output. The first three ($F_{1}^{2}$, $F_{2}^{3}$ and
$F_{3}^{3}$) have 256 channels each, while the output of the SAM ($F_{4}^{3}$)
has 512 channels. A higher channel count signifies a greater contribution to
the final feature vector. The final feature map ($F_{l}$) has 1280 channels
and assembles contributions from each feature extraction stage.

The gathered features ($F_{l}$, Fig. 2) are forwarded to our novel _Channel
Attention Module_ (CAM, presented in Section 3.2): see Figure 3. The latter
computes the importance of each channel feature for the final prediction. As
contributions from different feature extraction levels are present there, CAM
helps identify what features are more effective for scale-invariant
representation. The output of CAM is transferred to a classification block
which uses softmax to compute the class probabilities of each pixel.

Figure 3: The proposed Channel Attention Module (CAM) used in FBNet.

Finally, an auxiliary task is used to improve the feature representation in
the backbone model. The auxiliary task is applied to the output of SAM
($F_{4}^{2}$, see bottom of Fig. 2). It learns same semantic segmentation
targets but at half the resolution of the final output, preserving
computation. This process helps the SAM learn spatial context in a robust
fashion. The low-resolution auxiliary output also ensures that the spatial
attention module focuses on the global scene context rather than overfit the
local fine-grain information.

###  3.2 Channel attention

Let us denote the last set of feature maps after upsampling and concatenation
by $y(k,i,j)\in\mathbb{R}^{C\times H\times W}$, assuming the output of the
last convolutional layer has $C$ feature maps of size $H\times W$ each. In
scene segmentation tasks, each feature location in each of the maps is in
correspondence with a location in the image. Hence, for each location $(i,j)$,
the $C$-dimensional vector $y(\cdot,i,j)$ can be considered as the feature
vector for pixel $(i,j)$ (see Figure 3). The proposed CAM considers all
feature locations as independent, so that pixels belonging to the same class
must be represented by a similar attention vector.

In our model, channel attention is computed by applying a fully-connected
neural block composed by two layers (Fig. 3, top). First, pixel-specific
feature vectors are compressed to develop a more robust and invariant
representation. This is subsequently inflated to the same size as the original
vector using a second fully connected layer. The operation is applied to each
spatial location independently, generating an output of the same size
($\mathbb{R}^{C\times H\times W}$). A softmax activation function is used to
compute the relative weight of each individual component of the vector along
the $C$ dimension. This operation is performed on all locations (i.e.,
$H\times W$ times). Final output of CAM is the sum of the original feature
vectors and attention values. Formally, the overall CAM transformation can be
represented as:

$F_{c}(k,i,j)=\frac{\exp(f_{c}(y(k,i,j)))}{\sum_{k=1}^{C}\exp(f_{c}(y(k,i,j)))}+y(k,i,j),$
(1)

where $f_{c}$ is a function representing the fully connected layer. Softmax is
computed along the channel dimension ($k$) and the result is added to the
original features ($y$). This is a simple yet effective way of boosting the
vector representation.

As explained, the feature maps passed to CAM are obtained by merging feature
representations computed at different stages of the architecture (see Fig. 2).
Hence, each component of the feature vector for a given location $(i,j)$
represents context information for the given pixel captured using different
‘context windows’.

###  3.3 Spatial attention

The spatial attention module receives the feature maps
$x(k,i,j)\in\mathbb{R}^{\bar{C}\times\bar{H}\times\bar{W}}$ extracted by the
final layer of the backbone, which are half the size of the final feature maps
(i.e., $\bar{H}$ and $\bar{W}$ are 1/8 of image height and width,
respectively). This, low resolution representation of the image, is designed
to encode a ‘bigger picture’ view of the scene. Here, a simple yet effective
attention mechanism inspired by _self-attention_ Lin et al. [2017b], Vaswani
et al. [2017] is used.

First, key and query values are computed by applying a key function $f_{k}(.)$
and a query function $f_{q}(.)$ to $x(k,i,j)$. These can be represented as:

$s_{k}(k,i,j)=f_{k}(x(k,i,j)),\quad s_{q}(k,i,j)=f_{q}(x(k,i,j)),$

where $k$, $i$ and $j$ represent the channel, height and width dimensions of
the feature map, respectively. Both functions are computed using a 2D
convolutional layer with a $1\times 1$ kernel. The representation for both
$s_{k}$ and $s_{q}$ are then transformed as in
$\mathbb{R}^{\bar{C}\times\bar{H}\times\bar{W}}\rightarrow\mathbb{R}^{\hat{C}\times
N}$. Spatial attention ($S_{a}\in\mathbb{R}^{N\times N}$) can then be
expressed as the softmax of the dot product of key and query values:

$S_{a}=\frac{\exp(s_{q}^{T}\cdot s_{k})}{\sum_{N}\exp(s_{q}^{T}\cdot s_{k})}.$

A value function $f_{v}(.)$ is then computed from the feature map $x(k,i,j)$
as in Equation (2), resulting in a mapping
$\mathbb{R}^{\bar{C}\times\bar{H}\times\bar{W}}\rightarrow\mathbb{R}^{\bar{C}\times
N}$, with $N=\bar{H}\times\bar{W}$:

$s_{v}(k,i,j)=f_{v}(x(k,i,j)).$ (2)

The final self-attention output is computed by taking the dot product of the
attention matrix $S_{a}$ and the output $s_{v}$ of the value function, adding
the initial feature map $x(k,i,j)$:

$S_{f}=s_{v}^{T}\cdot S_{a}+x.$ (3)

Note that, before performing the addition in (3), the matrix
$\left[s_{v}^{T}\cdot S_{a}\right]$ is reshape into the shape of the input
feature map $x$ via a mapping $\mathbb{R}^{\bar{C}\times
N}\rightarrow\mathbb{R}^{\bar{C}\times\bar{H}\times\bar{W}}$.

The addition of self-attention components to the original feature maps $x$
helps attenuate or amplify their respective values and is generally more
effective than using only attention values or simple concatenation. The
computational approach to attention proposed here is very simple and does not
resort to an excessive number of parameters. In our tests we found that
increasing the complexity of the spatial attention module does not
significantly improve the performance of the model.

##  4 Experiments

Datasets. The proposed model is validated on two of the most commonly used
benchmarks for scene parsing: ADE20K Zhou et al. [2017] and Cityscapes Cordts
et al. [2016]. ADE20K is the largest scene parsing dataset with 150 object and
scene classes covering a wide range of natural scenes. The dataset contains
20,210 images for training and 2,000 images for evaluation. Cityscapes
contains images of roads, vehicles and surrounding scenes with a total of
2,975 training images and 500 images for validation, with for 19 semantic
classes.

Experimental setup. For a fair and unbiased evaluation of the proposed model
against the state-of-the-art, training and evaluation processes have been
standardised for all models. We use a base size of 520 to resize the smaller
side of image, and a crop size of 480 is chosen to generate a random crop from
the base image. During evaluation, all images are resized to the base size.
All models are trained for 180 epochs on the ADE20K dataset and 240 epochs on
the Cityscapes dataset. For augmentation, we use random flipping and scaling
in the interval $[0.5,2]$. Categorical cross entropy is used as the loss
function for both the main and the auxiliary input. Stochastic Gradient
Descent (SGD) is used for optimisation with momentum value of $0.9$ and weight
decay value of $10^{-4}$. Initial learning rates of $0.02$ and $0.04$ are used
for ADE20K and Cityscapes, respectively. Additionally, we reduce the learning
rate by $1/10$ for the pre-trained backbone. A polynomial-based learning rate
reduction policy is used at training time Chen et al. [2017]. The learning
rate at any given iteration is obtained by scaling the base learning rate by
factor of $\left(1-\frac{\text{iter}}{\text{total\\_iter}}\right)^{0.9}$. All
models are trained on eight NVIDIA RTX 6000 GPUs. Models are trained using
Synchronized BN Zhang et al. [2018].

Performance Metric. Mean intersection over union (mIoU) score and pixel
accuracy are used as the evaluation metrics. mIoU is the measure of overlap
between the predicted object shape and ground-truth shape. Pixel accuracy is
ratio of pixel correctly classified by the model to the total image pixels.
Additionally, we use epoch to convergence (Epochs) to provide the insight on
how fast each model achieves its best performance.

###  4.1 Ablation studies

Table 1: Ablation study evaluating the performance of FBNet under different combinations of its structural components. Model | Strategy | mIoU | Accuracy | Epochs
---|---|---|---|---
FBNet | SAM | 47.13 | 81.43 | 164
FBNet | CAM | 47.21 | 80.98 | 150
FBNet | FF | 47.39 | 81.16 | 156
FBNet | CAM+ SAM (parallel) | 47.55 | 81.15 | 156
FBNet | CAM+ SAM (series) | 47.95 | 81.57 | 136
FBNet | FF+ SAM+ CAM | 48.71 | 81.89 | 168

Significance of architectural components. The first ablation study analyses
how different components of the proposed architecture contribute to the final
performance. The first two results in Table 1 show the performance of model
when only the SAM or the CAM modules are applied to the output of the
backbone’s final layer ($F_{4}^{1}$ in Fig. 2) and No feature fusion is used.
The SAM model achieves a 47.13 mIoU, while CAM achieves 47.21. Hence, even
without feature fusion, the CAM module is good at feature selection for
correct prediction. Additionally, the model is faster at converging and only
takes 150 epochs to reach the best mIoU. We also tested the combination of SAM
and CAM modules in series and parallel, respectively, applied in both cases
only to the final layer feature maps ($F_{4}^{1}$ in Fig. 2). Interestingly,
the series combination performs better than the individual modules as well as
the parallel combination. In the series combination SAM follows CAM, whereas
in parallel, the same feature map is fed to both modules and their outputs are
added to generate the final attention maps.

In the last experiment, we use a feature fusion strategy where mid-level
features from the backbone are combined with the output of SAM. The proposed
CAM is applied to the resulting maps. The resulting model achieves
significantly better performance than any of the independent modules. This
further supports the hypothesis that, when applied to multi-level feature
representation, CAM is able to identify the most relevant features and
generate a scale invariant representation leading to much better predictions.

Table 2: Ablation study showing the performance of the proposed model with different state-of-the-art backbones. Model | Backbone | MIoU | Accuracy | Epochs
---|---|---|---|---
FBNet | ResNet-101 He et al. [2016] | 43.52 | 79.53 | 170
FBNet | ResNeXt-101 Xie et al. [2017] | 43.35 | 80.09 | 180
FBNet | ResNeSt-101 Zhang et al. [2020] | 46.71 | 80.89 | 145
FBNet | ResNet-152 He et al. [2016] | 44.70 | 79.89 | 135
FBNet | ResNeSt-200 Zhang et al. [2020] | 48.71 | 81.89 | 168

Effect of different backbones. In the second ablation study (Table 2), we
analyse how the performance of FBNet changes when different state-of-the-art
backbones are employed. The three most commonly used types of residual
learning-based architectures, i.e., ResNet He et al. [2016], ResNeXt Xie et
al. [2017] and ResNeSt Zhang et al. [2020], are considered. All such backbone
models are pre-trained on Imagenet and later fine-tuned as part of the
proposed architecture, using a learning rate that is 1/10th of that of the
whole model. When comparing models of similar expressive power (i.e.,
ResNet-101, ResNeXt-101 and ResNeSt-101), it is clear that ResNeSt-101 is the
best choice for the backbone. The model achieves a 46.71 mIoU score whereas
the much larger ResNet-152 only achieves 44.70 mIoU. Furthermore, ResNeSt-101
take significantly less time to train. Nevertheless, the best results are
achieved when using a ResNeSt-200 backbone, with the caveat that this model
takes many folds longer to train (FBNet with ResNeSt-200 takes around 90
hours).

Analysis of model parameters. Figure 4(a) plots the mIoU values achieved (for
each backbone) against the number of model parameters. All models on top left
side of the plot have good performance and are computational efficient (FBNet
with both ResNeSt-101 and ResNeSt-200 backbones). On the other hand, models on
bottom right corner have very high computational cost and lower performance
(FBNet with a ResNeXt-101 backbone). Figure 4(b) shows the comparision of
parameters in proposed FBNet and state-of-the-art models. All models are
compared with ResNeSt-200 backbone. The proposed model only has 86.48 million
parameters which is significantly lower than others. FBNet can be seen as
better choice when comparing mIoU to parameter ratio.

|
---|---
(a) | (b)
Figure 4: Plot of (a) mIOU achieved versus number of parameters for different
backbones in Table 2; (b) mIoU value against number of parameters for all the
models in Table 4.

Attention maps for SAM and CAM . To further elucidate the contribution of each
module, we have provided attention maps for both the spatial attention module
(SAM) and the channel attention module (CAM) for the model trained on the
ADE20K dataset (refer Fig. 5). The attention maps are remarkably sparse, with
each map exhibiting a distinct region of focus. Certain maps concentrate on
people and shops, while others highlight background structures. Notably, the
spatial attention maps are half the size of the channel attention map, making
the mechanism computationally highly efficient. This also enables effective
translation of attention maps at higher resolutions.

Figure 5: Attention maps for the FBNet trained on ADE20K dataset. First and
second rows show attention maps for SAM (size: 64x86) and CAM (size: 128x171),
respectively.

###  4.2 Comparison with the state-of-the-art

Experiments on ADE20K. For unbiased evaluation we evaluated all previous
state-of-the-art methods using a ResNeSt-200 backbone, even when other types
of backbone were used in the original papers, as our experiments show that
ResNeSt-200 Zhang et al. [2020] is a better backbone than ResNet-101 He et al.
[2016]. The results for all the models are presented in Table 3. The proposed
FBNet model is able to achieve a mIoU of 48.71, outperforming all the other
methods. The proposed model not only achieves the best mIoU but also the best
pixel accuracy, evidence of the model being able not just to predict the
correct object/scene class but also to achieve a good contextual understanding
of an object’s neighbourhood. The second highest performance is achieved by
EncNet Zhang et al. [2018], when trained using both auxiliary and SE losses.

Figure 6 shows some sample predictions made using FBNet against the ground
truth labels. Figure contains images from both indoor and outdoor scenes with
complex scene structure. All the images contain a complex scene structure
where it is difficult to correctly identify pixel association in the object.
The proposed model successfully predicts the scene structure that very closely
resembles to the ground truth.

Table 3: Comparison of performance against state-of-the-art segmentation models on the ADE20k dataset. Model | Backbone | MIoU | Accuracy | Epochs
---|---|---|---|---
PSPNet Zhao et al. [2017] | ResNeSt-200 | 47.32 | 81.11 | 132
EncNet Zhang et al. [2018] | ResNeSt-200 | 47.80 | 81.13 | 128
DANet Fu et al. [2019] | ResNeSt-200 | 47.75 | 80.66 | 112
OCNet Yuan & Wang [2018] | ResNeSt-200 | 47.38 | 81.48 | 176
FBNet (ours) | ResNeSt-200 | 48.71 | 81.89 | 168

Experiments on Cityscapes. The evaluation results on the popular Cityscapes
benchmark are shown in Table 4. When computing the mIoU we used the mean over
the samples rather than the mean over the classes, as arguably a more
expressive way of portraying true performance. The proposed model is again
able to beat the previous art by a small but significant margin. FBNet
achieves an mIoU of $81.38$, whereas the second-best model (DANet) only
achieves an mIoU value of $80.49$. Concerning pixel classification accuracy,
again FBNet achieves the best value ($96.87$) with a significant margin from
the competitors.

Table 4: Comparison of performance against state-of-the-art segmentation models on the Cityscapes dataset. Model | Backbone | MIoU | Accuracy | Epochs
---|---|---|---|---
PSPNet Zhao et al. [2017] | ResNeSt-200 | 79.65 | 96.46 | 234
EncNet Zhang et al. [2018] | ResNeSt-200 | 79.47 | 96.22 | 174
DANet Fu et al. [2019] | ResNeSt-200 | 80.49 | 96.47 | 240
OCNet Yuan & Wang [2018] | ResNeSt-200 | 80.11 | 96.33 | 198
FBNet (ours) | ResNeSt-200 | 81.38 | 96.87 | 228
|  |  |  |
---|---|---|---|---
|  |  |  |
|  |  |  |
Figure 6: Prediction results from the proposed FBNet against the ground truth
labels for a number of sample images from validation set of ADE20K dataset
Zhou et al. [2017]. The first row shows the original image, followed by the
output of FBNet. The last row shows the ground truth label image.

##  5 Conclusions

In this paper we proposed a novel feature boosting network for scene parsing.
The model gathers mid-level feature representations from different feature
extraction stages. Subsequently a novel channel attention module is applied
which computes the relevance of each feature towards computing the final class
label. Additionally, a simplified self attention module is used to extract
spatial contextual information before applying feature fusion. An auxiliary
task is used to learn semantic features at lower resolution to achieve a
better feature representation in the spatial attention module. The ablation
studies are conducted structural component in the proposed model. We also
studied how each backbone affects the performance of proposed model. FBNet was
evaluated against other state-of-the-art models on the ADE20K and Cityscapes
benchmark datasets, outperforming all the considered approaches.

## References

  * Badrinarayanan et al. [2017] Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017).  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  IEEE transactions on pattern analysis and machine intelligence, 39, 2481–2495.
  * Bell et al. [2013] Bell, S., Upchurch, P., Snavely, N., & Bala, K. (2013).  Opensurfaces: A richly annotated catalog of surface appearance.  ACM Transactions on graphics (TOG), 32, 1–17.
  * Chen et al. [2017] Chen, L.-C., Papandreou, G., Schroff, F., & Adam, H. (2017).  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587, .
  * Chen et al. [2016] Chen, L.-C., Yang, Y., Wang, J., Xu, W., & Yuille, A. L. (2016).  Attention to scale: Scale-aware semantic image segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3640–3649).
  * Chen et al. [2018] Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018).  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proceedings of the European conference on computer vision (ECCV) (pp. 801–818).
  * Chen et al. [2014] Chen, X., Mottaghi, R., Liu, X., Fidler, S., Urtasun, R., & Yuille, A. (2014).  Detect what you can: Detecting and representing objects using holistic models and body parts.  In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1971–1978).
  * Cordts et al. [2016] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016).  The cityscapes dataset for semantic urban scene understanding.  In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  * Fleuret et al. [2021] Fleuret, F. et al. (2021).  Uncertainty reduction for model adaptation in semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9613–9623).
  * Fu et al. [2019] Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., & Lu, H. (2019).  Dual attention network for scene segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3146–3154).
  * He et al. [2016] He, K., Zhang, X., Ren, S., & Sun, J. (2016).  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778).
  * Hu et al. [2018] Hu, J., Shen, L., & Sun, G. (2018).  Squeeze-and-excitation networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7132–7141).
  * Li et al. [2019] Li, H., Li, J., Lin, X., & Qian, X. (2019).  Pancreas segmentation via spatial context based u-net and bidirectional lstm.  arXiv preprint arXiv:1903.00832, .
  * Li et al. [2018] Li, H., Xiong, P., An, J., & Wang, L. (2018).  Pyramid attention network for semantic segmentation.  arXiv preprint arXiv:1805.10180, .
  * Lin et al. [2017a] Lin, G., Milan, A., Shen, C., & Reid, I. (2017a).  Refinenet: Multi-path refinement networks for high-resolution semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1925–1934).
  * Lin et al. [2017b] Lin, Z., Feng, M., Santos, C. N. d., Yu, M., Xiang, B., Zhou, B., & Bengio, Y. (2017b).  A structured self-attentive sentence embedding.  arXiv preprint arXiv:1703.03130, .
  * Liu et al. [2019] Liu, X., Deng, Z., & Yang, Y. (2019).  Recent progress in semantic image segmentation.  Artificial Intelligence Review, 52, 1089–1106.
  * Long et al. [2015] Long, J., Shelhamer, E., & Darrell, T. (2015).  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3431–3440).
  * Milletari et al. [2016] Milletari, F., Navab, N., & Ahmadi, S.-A. (2016).  V-net: Fully convolutional neural networks for volumetric medical image segmentation.  In 2016 Fourth International Conference on 3D Vision (3DV) (pp. 565–571).  IEEE.
  * Mottaghi et al. [2014] Mottaghi, R., Chen, X., Liu, X., Cho, N.-G., Lee, S.-W., Fidler, S., Urtasun, R., & Yuille, A. (2014).  The role of context for object detection and semantic segmentation in the wild.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 891–898).
  * Noh et al. [2015] Noh, H., Hong, S., & Han, B. (2015).  Learning deconvolution network for semantic segmentation.  In Proceedings of the IEEE international conference on computer vision (pp. 1520–1528).
  * Poudel et al. [2019] Poudel, R. P., Liwicki, S., & Cipolla, R. (2019).  Fast-scnn: fast semantic segmentation network.  arXiv preprint arXiv:1902.04502, .
  * Ronneberger et al. [2015] Ronneberger, O., Fischer, P., & Brox, T. (2015).  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention (pp. 234–241).  Springer.
  * Taghanaki et al. [2019] Taghanaki, S. A., Abhishek, K., Cohen, J. P., Cohen-Adad, J., & Hamarneh, G. (2019).  Deep semantic segmentation of natural and medical images: A review.  arXiv preprint arXiv:1910.07655, .
  * Vaswani et al. [2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017).  Attention is all you need.  arXiv preprint arXiv:1706.03762, .
  * Wang et al. [2017] Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., & Tang, X. (2017).  Residual attention network for image classification.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3156–3164).
  * Wu et al. [2019] Wu, Z., Wang, X., Gonzalez, J. E., Goldstein, T., & Davis, L. S. (2019).  Ace: Adapting to changing environments for semantic segmentation.  In Proceedings of the IEEE International Conference on Computer Vision (pp. 2121–2130).
  * Xiao et al. [2018] Xiao, T., Liu, Y., Zhou, B., Jiang, Y., & Sun, J. (2018).  Unified perceptual parsing for scene understanding.  In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 418–434).
  * Xie et al. [2017] Xie, S., Girshick, R., Dollár, P., Tu, Z., & He, K. (2017).  Aggregated residual transformations for deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1492–1500).
  * Yang et al. [2018] Yang, G., Li, G., Pan, T., Kong, Y., Wu, J., Shu, H., Luo, L., Dillenseger, J.-L., Coatrieux, J.-L., Tang, L. et al. (2018).  Automatic segmentation of kidney and renal tumor in ct images based on 3d fully convolutional neural network with pyramid pooling module.  In 2018 24th International Conference on Pattern Recognition (ICPR) (pp. 3790–3795).  IEEE.
  * Yu et al. [2018] Yu, B., Yang, L., & Chen, F. (2018).  Semantic segmentation for high spatial resolution remote sensing images based on convolution neural network and pyramid pooling module.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11, 3252–3261.
  * Yuan et al. [2019] Yuan, Y., Chen, X., & Wang, J. (2019).  Object-contextual representations for semantic segmentation.  arXiv preprint arXiv:1909.11065, .
  * Yuan & Wang [2018] Yuan, Y., & Wang, J. (2018).  Ocnet: Object context network for scene parsing.  arXiv preprint arXiv:1809.00916, .
  * Zhang et al. [2018] Zhang, H., Dana, K., Shi, J., Zhang, Z., Wang, X., Tyagi, A., & Agrawal, A. (2018).  Context encoding for semantic segmentation.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (pp. 7151–7160).
  * Zhang et al. [2020] Zhang, H., Wu, C., Zhang, Z., Zhu, Y., Zhang, Z., Lin, H., Sun, Y., He, T., Mueller, J., Manmatha, R. et al. (2020).  Resnest: Split-attention networks.  arXiv preprint arXiv:2004.08955, .
  * Zhang et al. [2021] Zhang, X., Xu, H., Mo, H., Tan, J., Yang, C., Wang, L., & Ren, W. (2021).  Dcnas: Densely connected neural architecture search for semantic image segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13956–13967).
  * Zhao et al. [2017] Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017).  Pyramid scene parsing network.  In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2881–2890).
  * Zhao et al. [2018] Zhao, H., Zhang, Y., Liu, S., Shi, J., Change Loy, C., Lin, D., & Jia, J. (2018).  Psanet: Point-wise spatial attention network for scene parsing.  In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 267–283).
  * Zhou et al. [2017] Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., & Torralba, A. (2017).  Scene parsing through ade20k dataset.  In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 633–641).