# Technical Report on Subspace Pyramid Fusion Network for Semantic
Segmentation

Mohammed A. M. Elhassan1, Chenhui Yang2, Chenxi Huang2 and Tewodros Legesse
Munea2
1School of Computer Science, Zhejiang Normal University,688 Yingbin
Road,Jinhua,Zhejiang,China
2School of Informatics, Xiamen University,Xiamen, China,361005 Email address:
mohammedac29@zjnu.edu.cn(Mohammed A. M. Elhassan).

###### Abstract

The following is a technical report to test the validity of the proposed
Subspace Pyramid Fusion Module (SPFM) to capture multi-scale feature
representations, which is more useful for semantic segmentation. In this
investigation, we have proposed the Efficient Shuffle Attention Module(ESAM)
to reconstruct the skip-connections paths by fusing multi-level global context
features. Experimental results on two well-known semantic segmentation
datasets, including Camvid and Cityscapes, show the effectiveness of our
proposed method. The source code is available at
111https://github.com/mohamedac29/SPFNet

##  I Introduction

SSemantic Semantic segmentation is an essential high-level topic in computer
vision and has been widely used in various challenging problems such as,
medical diagnosis, [1, 2], autonomous vehicles [3], and scene analysis [4, 5].
Unlike image classification, which aims to classify the whole image, semantic
segmentation predicts the per-pixel class for each image content. The current
frontiers in semantic segmentation methods are driven by the success of deep
convolution neural networks, specifically the fully convolution network (FCN)
Framework [6]. In the original FCN approach to obtain larger receptive fields,
they increase the depth of the network with more convolutional and
downsampling layers. However, simply increasing the number of downsampling
operations leads to reduced feature resolution, which causes spatial
information loss. On the other hand, the increase in the number of
convolutional layers adds more challenges to network optimization.

Figure 1: Accuracy-Speed performance comparison on the Cityscapes test set.
Our methods are presented in red dots while other methods are presented in
blue dots. Our approaches achieve state-of-the-art speed-accuracy trade-off

Over the years, various approaches have been proposed to tackle the drawbacks
of the typical FCN-based architecture. Here we listed three categories of
methodologies as follows: (i) increasing the resolution of feature maps or
maintaining a high-resolution feature across stages, e.g., through decoder
network [1, 7, 8], dilated convolutions [9, 10] or high resolution [11]. (ii)
improve the segmentation performance with the subsequent approaches, e.g.,
PSPNet [5], DeepLab [10, 12], PAN [13] DenseASPP [14] , RefineNet [15], PPANet
[16], and ParseNet [17]. These previous networks enlarge the receptive field
to exploit the rich contextual information. Nevertheless, the latter solution
is infeasible for real-time applications because it requires heavy networks
and expensive computation. (iii) some networks such as [18, 19] have added
spatial encoding path to preserve the spatial details To perform fast
segmentation with light computation cost, small size and satisfactory
segmentation accuracy, there are certain design philosophy that could be
followed: (i) incorporate a pre-trained lightweight classification
networks[20, 21, 22, 23] as a backbone to construct segmentation architecture
such as BiSeNet [19], STDC-Seg [24]. (ii) design a building block that is
suitable for low computation cost [25, 26]. (iii) put into consideration the
downsampling strategy and the depth of the network [27]. (iv) combine the low-
level and high-level features using the multi-path framework [19, 28, 18]. in
addition to that, restricting the input image size plays an important part in
increasing the inference speed.

(a) Encoder-Decoder Structure

(b) Encoder-Decoder with Atrous Conv

(c) Encoder-Decoder with SPFM

Figure 2: A comparison of an important semantic segmentation architectures.
(a) encoder-decoder structure.(b) encoder-decoder structure with atrous
convolutions.(c) our network, encoder-decoder structure with atrous
convolutions and context aggregation module.

Multi-scale information is the most crucial problem that should be considered
to improve the performance of semantic segmentation networks. Typically, a
class of targets may exist at different scales in the image, and a well-
designed model should be able to extract this property, which leads to better
semantic segmentation accuracy. In this paper, we propose a Subspace Pyramid
Fusion Module (SPFM) to improve the multi-scale feature learning in semantic
segmentation. In particular, the SPFM generates global contextual information
in each feature subspace based on parallel Reduced Pyramid Pooling submodule
(RPP). RPP submodule constructed from three components: dilated convolution or
atrous convolution, subpixel convolution, and average pooling. Two dilated
convolutions were used for each reduced pyramid pooling level to capture
object features with various receptive fields, followed by subpixel
convolution instead of the original upsampling method to unify the feature
resolution. To address the problems of resolution loss via downsampling in the
encoding part, we develop the efficient shuffle attention module (ESAM) with
two branches. Through the lower branch, i.e., ESAM integrates the shuffle
attention block to facilitate cross-channel information communication for
feature maps from different encoder levels and reduces channel redundancy.
Finally, a Decoder-based Subpixel Convolution (DSC) is proposed to improve
category pixel localization and recover object details. The SPFM module is
embedded in the center between the encoder and the decoder to help select the
suitable receptive field for objects with different scales by fusing the
multi-scale context information. Based on these descriptions, we design the
Subspace Pyramid Fusion Network (SPFNet), see Figure3. The proposed SPFNet is
validated with two challenging semantic segmentation benchmarks, Camvid [29]
and Cityscapes [30]. Figure. 1 shows the accuracy and the speed comparison of
different methods on Cityscapes dataset.

Furthermore, many existing frameworks [31, 32, 33, 34, 35] have been developed
based on FPN [35] to exploit the inherent multi-scale feature representation
of deep convolutional networks. More specifically, FPN-based architectures
combine large-receptive field features, low-resolution- with small-receptive-
field features, high-resolution to capture objects at different scales. For
instance, [34, 35] utilize the lateral path to fuse adjacent features in a
top-down manner. This way FPN strengthen the learning of multi-scale
representation.Until now, feature pyramid network based approaches have
achieved state-of-arts performance in object detection. Our main contributions
are summarized as follows:

  1. 1.

A novel Subspace Pyramid Fusion (SPFM) module is proposed to learn multi-scale
representations.

  2. 2.

We introduce the Efficient Shuffle Attention Module(ESAM) module, which
utilizes channel shuffle operation to integrate the complementary global
context

  3. 3.

We proposed an architecture based on SPFM and ESAM modules that can be easily
applied for road scene understanding. Furthermore, these modules can be
plugged in any network

Figure 3: An overview of the proposed Subspace Pyramid Fusion Network
(SPFNet).

##  II Related Work

###  II-A Efficient Network Designs

In recent literature, real-time model design has become an essential part of
computer vision research development. Several lightweight architectures were
introduced as a trade-off between accuracy and latency SqueezeNet [20],
MobileNet [21, 36], ShuffleNet [37, 22],and Xception [23]. SqueezeNet reduces
the number of parameters through the squeeze and expansion module. MobileNetV1
implemented depth-wise separable convolutions to reduce the number of
parameters. MobileNetV2 proposed an inverted residual block to alleviate the
effect of depth-wise separable convolutions. ShuffleNet and Xception utilize
group convolution to reduce FLOPs. ResNet [38] adopted residual blocks to
obtain outstanding performance. These models were designed primarily for image
classification tasks. Later on, incorporated to work as backbones for semantic
segmentation networks to achieve real-time semantic segmentation.

###  II-B Real-time Segmentation

Real-time semantic segmentation methods aim to predict dense pixels accurately
while maintaining high inference speed. Over the years there are different
approaches have been proposed in this regard. ENet [39], ERFNet [26], and
ESPNet [40, 41] proposed compact semantic segmentation networks that utilize
lightweight backbones. LEDNet [42], DABNet [25] introduced a lightweight model
to address the computation burden that limits the usage of deep convolutional
neural networks in mobile devices. The former used asymmetric encoder-decoder-
based architecture and implemented channel split and shuffle operations to
reduce the computation cost. The latter structured a depth-wise symmetric
building block to jointly extract local and global context information.
FDDWNet [43] utilized factorized depth-wise separable convolutions to reduce
the number of parameters and the dilation to learn multi-scale feature
representations. ICNet [44] proposes a compressed version of PSPNet based on
an image cascade network to increase the semantic segmentation speed. DFANet
[27] obtains the multi-scale feature propagation by utilizing a sub-stage and
sub-network aggregation.Tow-pathway networks such as BiSeNetV2 [28] and DSANet
[18] Fast-SCNN [45] introduce a shallow detailed branch to extract the spatial
details and a deep semantic branch to extract the semantic context. This paper
propose a lightweight model that utilizes (ResNet18 or ResNet34) as backbones.

##  III Methodology

In this section, we first present an overview of the SPFNet, and then
elaborate the mechanism of different modules that used to construct this
network.

###  III-A Overview

Figure 3 illustrates the architectural framework of SPFNet, designed within a
conventional encoder-decoder paradigm. The encoder is responsible for
downsampling the input data to acquire diverse multi-scale features, while the
decoder focuses on reconstructing high-level features from low-level ones,
facilitating pixel-level semantic predictions. SPFNet initiates the process by
taking an input image and encoding its features through a widely-used multi-
scale backbone, specifically ResNet34 [38] ,with hierarchical representations
$[F_{1}^{b},F_{2}^{b},F_{3}^{b},F_{4}^{b},F_{5}^{b}]$. The notation
$[F_{2}^{egca},F_{3}^{egca},F_{4}^{egca},F_{5}^{egca}]$ denotes the rich
context information extracted from the feature hierarchy within the Efficient
Shuffle Attention Module (ESAM) (see section III-C).

(a) Subspace pyramid fusion module (SPFM).

(b) Reduced pyramid pooling module (RPP).

Figure 4: Illustration of the (a) Reduced pyramid pooling module (RPP). (b)
Subspace Pyramid Fusion (SPFM) module. SPFM extracts the multi-scale
information in each sub-features using RPP module, then transmit the
information to decoder.

###  III-B Subspace Pyramid Fusion Module (SPFM)

In the realm of semantic segmentation research [10, 46, 13], it has been
substantiated that augmenting receptive fields is advantageous for the
semantic segmentation task. A conventional encoder-decoder architecture
involves the extraction of global context information by the encoder from the
input image, encompassing the adjacent and class characteristics of the
object. However, the transmission of information to shallower layers is known
to attenuate the extraction of context information owing to downsampling
processes. Motivated by this observation and drawing inspiration from prior
works such as [10] and [47], we introduce a pioneering Subspace Pyramid Fusion
Module (SPFM) in this subsection.

Within the framework of SPFM, each subspace feature undergoes a singular
reduced pyramid pooling (RPP) treatment. In contrast to methodologies
presented in [10] and [48], SPFM, illustrated in Figure 4, diverges in its
approach to collecting multi-scale contextual information. Rather than
directly fusing dilated convolutions, SPFM initiates the generation of multi-
scale features for each split using a reduced pyramid pooling module. The RPP
is constructed with only two parallel dilated convolutions, employing dilation
rates of (4,8). Subsequently, pixel shuffle is employed to enhance feature
resolution. Additionally, adaptive average pooling is integrated with the
output from two parallel convolutions to capture complex contextual
information. The resulting RPP blocks are concatenated to form the SPFM. In
comparison with SPP and ASPP, our module demonstrates superior information
acquisition through subpixel convolution and stacked RPP blocks.

Diverging from pyramid-based multi-scale learning approaches such as SPP and
ASPP, our SPFM module acquires heightened multi-scale information by learning
multiple Reduced Pyramid Pooling blocks (RPP) for each feature map. These RPP
blocks are then amalgamated within the Subspace Pyramid Fusion Module. A
distinctive characteristic of SPFM, as opposed to [47], lies in the approach
to information aggregation. While [47] integrates attention across different
stages in pre-existing compact backbones to facilitate global information
learning, SPFM achieves an increase in receptive fields through dilated
convolutions and adaptive average pooling to capture global context.

Given the feature map $F_{5}^{b}\in\mathbb{R}^{H\times W\times C}$ The
objective of our study is to efficiently capture multi-scale features from the
feature map $F_{5}^{b}\in\mathbb{R}^{H\times W\times C}$ obtained from the
backbone, where H and W represent the spatial dimensions of the feature maps,
and C denotes the number of channels. In Figure 4.4(a), we introduce the
proposed Subspace Pyramid Fusion Module (SPFM) and its submodule Reduced
Pyramid Pooling (RPP) in Figure 4.4(b). SPFM segments the input feature map
$F_{5}^{b}$into s splits along the channel dimension. Here are the details of
SPFM and RPP modules.
Reduced Pyramid Pooling (RPP): Each RPP block takes one split from SPFM.
First, the input features $F_{i}^{p}$ enter into two identical pathways, each
path consists of dilated convolution with kernel size of 3 followed by batch
normalization(BN) and Parametric Rectified Linear Unit (PReLU) to generate
multi-scale context information from these feature maps. Subsequently, we
apply pixel shuffle to obtain $P_{i,1}^{p}$ as in Eq. 1 and $P_{i,2}^{p}$ Eq.
2. Then, we concatenate the output features $P_{i,c}^{p}$ Eq. 3 and refined it
with three consecutive convolutions with kernel sizes ($1\times 1$), ($3\times
3$), and ($1\times 1$) respectively, to reduce aliasing effect. Finally, the
output is added with feature map of adaptive average pooling and convolution
Eq. 4 to produce the final reduced pyramid pooling $P_{\tilde{s},i}^{p}$ as
shown in Eq 5.

$P_{i,1}^{p}=\mathbb{PS}(\mathbb{D}_{conv}@4(F_{i}^{p}))$ (1)

$P_{i,2}^{p}=\mathbb{PS}(\mathbb{D}_{conv}@8(F_{i}^{p}))$ (2)

$P_{i,c}^{p}=Concat([P_{i,}^{p},P_{i,2}^{p}])$ (3)

$A_{i}^{p}=f^{1}(APool(F_{i}^{p}))$ (4)

where Concat([.]) is the element-wise concatenation. The reduced pyramid
fusion for each split $P_{\tilde{s},i}^{p}$ is computed by the following
equation.

$P_{\tilde{s},i}^{p}=A_{i}^{p}\oplus P_{i,c}^{p}$ (5)

In Eq. 1, and Eq. 2, $\mathbb{D}_{conv}@$ refers to dilated convolution layer
with kernel size of $3\times 3$. In Eq4 APool(.), is the adaptive average
pooling, $f^{1}$ represents convolution layer with kernel size of $1\times 1$.
In Eq. 1 and Eq. 2, $\mathbb{PS}(.)$ is the PixelShuffle with upsample scale
factor of 2.
Subspace Pyramid Fusion Module (SPFM): The final output for SPFM module
$\hat{P}_{out}$ is obtained by concatenating RPP of all splits (Eq. 6).

$\hat{P}_{out}=f_{1}(Cancat([\hat{P}_{\tilde{s},1},\hat{P}_{\tilde{s},2},...,\hat{P}_{\tilde{s},s}]))$
(6)

Figure 5: An overview of the proposed Global Context Aggregation module. It
utilize channel split/shuffle to process features in each group.

###  III-C Efficient Shuffle Attention Module(ESAM)

The significance of local and global context information in semantic
segmentation and various computer vision tasks has been acknowledged, even in
the traditional machine learning era [49]. The Efficient Shuffle Attention
Module (ESAM) is introduced to amalgamate channel attention via the shuffle
unit operator, thereby enhancing cross-feature interaction among sub-features.
In this context, the input features from the backbone are represented as
$F_{j}^{b}$, where j denotes the level in the encoder (refer to Figure 3).
Considering the distinct semantic content across different levels, ESAM is
applied before upsampling to a common resolution using bilinear interpolation.
Subsequently, features from all encoder scales and the final decoder-based
subpixel convolution outputs are concatenated to formulate a multi-scale
feature map. This process facilitates the fusion of information from both
shallow and deeper layers.

Given the $j^{th}$ feature $F_{j}^{b}\in\mathbb{R}^{H\times W\times C}$ from
the backbone, where H and W are spatial dimensions and $C$ is the channel
dimension. The output $F_{j}^{egca}$ of efficient shuffle attention module is
computed using the following equation:

${F}_{j}^{egca}=\widehat{F}_{j}\oplus F_{j}^{u}$ (7)

Where $\oplus$ represents the element-wise summation, $\widehat{F}_{j}$ is the
output of shuffle attention part, and $F_{j}^{u}$ represents the output
feature of the upper part. At first, the input feature $F_{j}^{b}$ is used to
process information into upper branch and the shuffle attention branch (see
Figure. 5). More details are given as follow:
Part I: Upper branch
in this branch $F_{j}^{b}$ is passed through depthwise separable convolutions,
leading to feature map of $F_{j}^{u}\in\mathbb{R}^{H\times W\times C}$ as
shown in Eq. 8.

$F_{j}^{u}=f_{DW}^{3}(f^{1}(F_{j}^{b}))$ (8)

where $f_{DW}^{3}(.)$ represents depthwise convolution with kernel $3\times
3$.
Part II: Lower branch- shuffle attention
The second branch uses shuffle channel attention to generate sub-features that
gradually capture multi-scale semantic responses. In this branch, the input
$F_{j}^{b}$ is divided into two groups $F_{j,1}\in\mathbb{R}^{H\times W\times
C/2}$, and $F_{j,2}\in\mathbb{R}^{H\times W\times C/2}$ along the channel
dimensions.

$A_{j,1}=f_{PW}^{1}(\mathbb{MP}(f_{DW}^{1}(F_{j,1})))$ (9)

$A_{j,2}=f_{PW}^{1}(\mathbb{MP}(f_{DW}^{1}(F_{j,2})))$ (10)

$A_{j,1}\in\mathbb{R}^{H\times W\times C}$ ,and $A_{j,1}\in\mathbb{R}^{H\times
W\times C}$, have the same operations. Each of them starts with depth-wise
convolution followed by Max Pooling and Point-wise. The two part are combined
to form an attention map $A_{j}$ for each sub-group to captures the long-range
dependencies.

$A_{j}=Softmax(A_{j,1}\oplus A_{j,2})$ (11)

The output $\widehat{F}_{j}$ is computed using Eq. 12.

$\widehat{F}_{j}=Shuffle((A_{j}\odot F_{j}^{b})\oplus F_{j}^{b})$ (12)

Where $\odot$ is the element-wise multiplication. $f_{DW}^{1}(.)$ represents a
depth-with convolution with kernel size of $1\times 1$, $\mathbb{MP}(.)$ is a
Max Pooling and Point-wise $f_{PW}^{1}(.)$.

The obtained output undergoes normalization through the softmax function, as
expressed in Equation 11. Subsequently, element-wise multiplication and
summation operations are implemented within residual-like connections
involving the input feature $A_{j}$. Additionally, we incorporate a channel
shuffle operator, akin to ShuffleNet [22], to facilitate cross-group
information communication. The resulting channel attention output, as
delineated in Equation 12, is then summated with the upper branch to compose
the Efficient Shuffle Attention Module (ESAM), as outlined in Equation 7.
Importantly, the final output dimensions of the ESAM module align with those
of $F_{i}^{b}$, rendering ESAM a straightforward and adaptable component that
can be seamlessly integrated into diverse stages of the encoder. This
flexibility empowers the module to acquire multi-scale features with distinct
semantic attributes. Consequently, the ESAM module proves conducive to
effortless integration into contemporary semantic segmentation architectures
and analogous tasks. The impact of the proposed efficient context aggregation
module is further illustrated in Table LABEL:tab:egca_stages, where
evaluations encompass floating-point operations per second (FLOPS),
parameters, processing speed, and mean intersection over union.

TABLE I: THE PER-CLASS, CLASS, AND CATEGORY IOU EVALUATION ON THE CITYSCAPES
TEST SET. ”-” INDICATES THE CORRESPONDING RESULT IS NOT REPORTED BY THE
METHODS

Method Road Sidewalk Building Wall Fence Pole Traffic light Traffic sign
Vegetation Terrain Sky Person Rider Car Truck Bus Train Motor Bicyclist mIoU
CRF-RNN[50] 96.3 73.9 88.2 47.6 41.3 35.2 49.5 59.7 90.6 66.1 93.5 70.4 34.7
90.1 39.2 57.5 55.4 43.9 54.6 62.5 FCN[6] 97.4 78.4 89.2 34.9 44.2 47.4 60.1.5
65.0 91.4 69.3 93.9 77.1 51.4 92.6 35.3 48.6 46.5 51.6 66.8 65.3 DeepLabv2[10]
- - - - - - - - - - - - - - - - - - - 70.4 Dilation10[51] 97.6 79.2 89.9 37.3
47.6 53.2 58.6 65.2 91.8 69.4 93.7 78.9 55.0 93.3 45.5 53.4 47.7 52.2 66.0
67.1 AGLNet[52] 97.8 80.1 91.0 51.3 50.6 58.3 63.0 68.5 92.3 71.3 94.2 80.1
59.6 93.8 48.4 68.1 42.1 52.4 67.8 70.1 BiSeNetV2/BiSeNetV2_L[28] - - - - - -
- - - - - - - - - - - - - 73.2 LBN-AA[53] 98.2 84.0 91.6 50.7 49.5 60.9 69.0
73.6 92.6 70.3 94.4 83.0 65.7 94.9 62.0 70.9 53.3 62.5 71.8- 73.6 Our(SPFNet)
98.5 85.2 92.6 48.4 55.8 67.0 74.5 77.8 93.4 70.7 95.2 85.7 68.8 95.2 53.3
73.0 59.7 67.2 75.4 75.7 Our(SPFNet) 98.4 84.9 92.2 47.6 54.1 65.9 73.6 76.2
93.0 66.4 94.8 85.3 68.5 94.6 47.6 53.9 28.8 65.7 75.0 71.9

##  IV Experiments

In this section, we conduct a comprehensive set of experiments to assess the
efficacy of the proposed SPFNet. The evaluations are performed on the Camvid
[29] and Cityscapes [30] datasets. The experimental results showcase SPFNet’s
state-of-the-art performance on the Camvid dataset and comparable results to
other methods on the Cityscapes dataset. The subsequent sections detail the
datasets and implementation specifics, followed by a series of ablation
studies conducted on the Camvid and Cityscapes datasets to elucidate the
impact of each component in SPFNet. Finally, we present a comparative analysis
of our method with other state-of-the-art networks on both datasets.

###  IV-A Experiments Settings

####  IV-A1 Camvid

The Camvid dataset [29] comprises 376 training examples, 101 evaluation
images, and 233 testing images. To ensure a fair comparison with other state-
of-the-art models, we assessed our model across 11 classes, including
building, sky, tree, car, and road. The 12th class was designated as an ignore
class to accommodate unlabelled data. The dataset’s small size and imbalanced
label distribution pose significant challenges.

####  IV-A2 Cityscapes

The Cityscapes dataset [30] serves as an urban scene understanding benchmark,
featuring 5000 high-resolution images of dimensions 2048 x 1024 pixels with
fine annotations from various cities. The dataset is divided into 2975 images
for training, 500 images for validation, and the remaining 1525 images for
testing. The evaluation spans 19 semantic segmentation classes.

The subsequent sections delve into the ablation studies, exploring the impact
of each constituent in SPFNet on the Camvid and Cityscapes datasets. The
experimental design aims to comprehensively assess the model’s performance
across diverse scenarios. Finally, a comparative analysis is presented,
juxtaposing SPFNet against other state-of-the-art networks on the two
datasets.

####  IV-A3 Implementation Details

To implement our model, we use Adam optimizer [54] with weight decay $5\times
10^{-6}$, power 0.9, and the starting learning rate $lr_{init}$ for Camvid and
Cityscapes datasets is set to $3\times 10^{-4}$. The network optimization
utilized the Poly learning rate policy, as described in Equation 13, a
strategy consistent with prior works such as [10, 5]. All experiments were
conducted using the PyTorch framework [55] on an NVIDIA 3090 RTX GPU, with
training extending over 150 epochs for Camvid and 500 epochs for Cityscapes.
Notably, BatchNorm layers in SPFNet, excluding those in the encoder, were
substituted with InplaceBN-Sync [56] to curtail memory usage, enhance speed,
and facilitate FLOPS analysis, which was conducted using an Nvidia GTX1080
GPU. Consistent data augmentation was applied across all experiments,
involving random scaling between 0.75 and 2.0, random horizontal flips, and
random cropping of $512\times 1024$ and $360\times 480$ image patches for
training Cityscapes and Camvid, respectively. Weighted cross-entropy loss was
employed for training on both datasets.To assess the experimental results of
our model, comprehensive comparisons were conducted on the Cityscapes
validation and testing sets against current state-of-the-art segmentation
models. Further detailed analyses are presented in subsequent sections.

$lr=lr_{init}\times(1-\frac{iter}{max\\_iter})^{power}$ (13)

Where ${max\\_iter}$ is the maximum number of iterations.

###  IV-B Ablation Study

In this subsection, a series of experiments is conducted to assess the
individual contributions of components within the proposed SPFNet. Initially,
a baseline network is established, employing an encoder-decoder architecture
consisting of a ResNet-based encoder and a decoder incorporating subpixel
convolution. This baseline serves as the foundation for the subsequent
ablation study.

####  IV-B1 Ablation for Efficient Shuffle Attention Module(ESAM)

To assess the impact of the Efficient Shuffle Attention Module (ESAM) at each
stage, we seamlessly integrate our proposed ESAM into a baseline U-shaped
network architecture, denoted as ResNet34H (encoder) and DSC (decoder). The
experimental results, presented in Table LABEL:tab:egca_stages, encompass
evaluations in terms of FLOPS, parameters, processing speed, and mean
Intersection over Union (mIoU). This integration enables the model to acquire
multi-level cross-feature communication and mitigate the semantic gap among
features from different levels. Importantly, the Subspace Pyramid Fusion
Module (SPFM) is not incorporated into the network at this juncture. The
introduction of ESAM leads to a marginal increase in computational
requirements, as evident from the FLOPS results. However, the corresponding
improvement in segmentation accuracy, as demonstrated in Table
LABEL:tab:egca_stages, substantiates the efficacy of ESAM. Notably,
integrating ESAM in the second stage results in a slightly higher FLOPS
compared to other stages. In the configuration utilizing ESAM across stages 2
to 5, the mIoU experiences an improvement of over 2.84% compared to the
baseline. It is noteworthy that ESAM incurs a reduction in processing speed by
5.7 FPS compared to the baseline..

TABLE II: EVALUATION OF ESAM EFFECT AT DIFFERENT STAGES ON CITYSCAPES
VALIDATION SET.FLOPS, SPEED ARE ESTIMATED FOR AN INPUT SIZE OF 512,1024.

Baseline $Stage_{2}$ $Stage_{3}$ $Stage_{4}$ $Stage_{5}$ FLOPS (G) Params( M)
Speed (FPS) mIoU(%) ✓ 268.6 37.7 19.3 72.16 ✓ ✓ 280.3 37.8 16.2 72.8 ✓ ✓ ✓
287.0 37.9 15.2 73.19 ✓ ✓ ✓ ✓ 293.6 38.0 14.2 74.01 ✓ ✓ ✓ ✓ ✓ 300.2 38.7 13.6
75.0

####  IV-B2 Ablation on Subspace Pyramid Fusion Module (SPFM)

In our hyperparameter analysis on the Subspace Pyramid Fusion Module (SPFM)
applied to the Camvid dataset, we systematically examine the impact of the
hyperparameter s, representing the number of splits used in each Reduced
Pyramid Pooling (RPP) module within the SPFM. The evaluations are conducted on
both the Camvid and Cityscapes datasets, utilizing an Nvidia GTX1080 GPU.
Specifically, we investigate four scenarios with ResNet18 (SPFNet18L) as the
backbone and another four scenarios with ResNet34 (SPFNet23L), considering
different values for s, namely$s={2,4,8,16}$. Initially, to establish a
baseline, we replace the SPFM with a $1\times 1$ Convolution layer, and the
results for the Camvid dataset are reported in Table III. When $s=2$, the SPFM
fuses two Reduced Pyramid Fusion Modules to generate multi-scale features. As
evident in Table III, leveraging SPFM with varying numbers of splits
demonstrates higher accuracy, measured in terms of mean Intersection over
Union (mIoU), compared to the baseline. Moreover, models with larger values of
s exhibit improved segmentation accuracy, indicating that higher s enhances
the network’s ability to extract complex multi-scale features for both
SPFNet34L and SPFNet18L, albeit at the expense of speed (Frames Per Second,
FPS). Table III and Figure 6.6(b) reveal a performance drop in SPFM when $s=8$
for both backbones in comparison to the baseline. Interestingly, utilizing 2
splits adds more parameters and Floating-Point Operations per Second (FLOPS)
than larger s options, resulting in faster inference. We further compared SPFM
with four splits in terms of FLOPS, number of parameters, and speed. As
outlined in Table III, a larger s introduces fewer FLOPS and parameters, yet
it diminishes speed. Considering both efficiency and segmentation accuracy,
our analysis concludes with the selection of $s=4$ for SPFM.

TABLE III: THE EVALUATION AND ANALYSIS OF THE HYPERPARAMETER S ON THE SPFM
MODULE. THE INPUT SIZE FOR THE BASELINE ON THE CAMVID TEST SET IS $360\times
840$.THE INPUT SIZE FOR SPFM IS $512\times 12\times 15$.

Baseline SPFNet-34L(ResNet34 as backbone) Network FLOPS (G) Params (M) Speed
(FPS) IoU (%) Overall SPFM Overall SPFM Overall SPFM Baseline 22.8 37.7 122.8
69.2 SPFM (s=2) 22.8 3.0 37.7 6.7 122.8 553.2 70.0 SPFM (s=4) 22.8 1.6 37.7
3.4 122.8 398.7 70.3 SPFM (s=8) 22.8 0.896 37.7 1.9 122.8 214.0 70.1 SPFM
(s=16) 22.8 0.544 37.7 1.0 122.8 109.0 70.6 Baseline SPFNet-18L(ResNet18+DSC
Module) Baseline 16.4 27.6 163.9 68.5 SPFM (s=2) 16.4 3.0 27.6 6.7 163.9 553.2
69.7 SPFM (s=4) 16.4 1.6 27.6 3.4 163.9 398.7 69.8 SPFM (s=8) 16.4 0.896 27.6
1.9 163.9 214.0 69.1 SPFM (s=16) 16.4 0.544 27.6 1.0 163.9 109.0 69.9

(a) Hyperparameter vs accuracy.

(b) Hyperparameter vs FPS.

Figure 6: (a) The accuracy obtained by the proposed SPFM with different values
of the parameter s on the Camvid test set. (b) The inference speed with
different s values.

Hyperparameter analysis on SPFM Module on the Cityscapes dataset: We conducted
similar analysis to examine the hyperparameter s of the SPFM on Cityscapes
dataset in terms of the efficiency and accuracy. For this analysis we run the
experiments for 150 epochs. From Table. IV and Figure. 7 Experimental results
and evaluation shows that the hyperparameter s follows the same patterns as on
Camvid dataset.

TABLE IV: THE EVALUATION AND ANALYSIS OF THE HYPERPARAMETER S ON THE SPFM
MODULE.THE INPUT SIZE FOR THE BASELINE ON THE CITYSCAPES TEST SET IS
$512\times 1024$.INPUT FOR THE SPFM IS $512\times 16\times 32$.

Baseline SPFNet-34L(ResNet34 as backbone) Network FLOPS (G) Params (M) Speed
(FPS) mIoU (%) Overall SPFM Overall SPFM Overall SPFM Baseline 68.1 37.7 65.2
71.4 SPFM (s=2) 68.1 8.6 37.7 6.7 65.2 355.6 72.5 SPFM (s=4) 68.1 4.6 37.7 3.4
65.2 266.2 73.7 SPFM (s=8) 68.1 2.5 37.7 1.9 65.2 204.4 73.1 SPFM (s=16) 68.1
1.5 37.7 1.0 65.2 107.6 73.7 Baseline SPFNet-18L(ResNet18+DSC Module) Baseline
48.7 27.6 83.7 71.1 SPFM (s=2) 48.7 8.6 27.6 6.7 83.7 355.6 72.1 SPFM (s=4)
48.7 4.6 27.6 3.4 83.7 266.2 73.4 SPFM (s=8) 48.7 2.56 27.6 1.9 83.7 204.4
72.9 SPFM (s=16) 48.7 1.5 27.6 1.0 83.7 109.0 73.6

(a) Hyperparameter vs accuracy.

(b) Hyperparameter vs FPS.

Figure 7: (a) The accuracy obtained by the proposed SPFM with different values
of the parameter s on the Cityscapes validation set. (b) The inference speed
with different s values.

####  IV-B3 Ablation of SPFM and Other Multi-scale Modules

To comprehensively investigate the impact of various modules within our
proposed network architecture, we initially configured a modified ResNet34 by
abstaining from downsampling in the second stage of the encoder. Subsequently,
we integrated the Dilated Spatial Convolution (DSC) module and the Efficient
Shuffle Attention Module (ESAM) module as baselines to assess different multi-
scale feature extractor modules. These ablation experiments were conducted
over 500 epochs.

In particular, we performed a series of experiments to compare our proposed
Subspace Pyramid Fusion Module (SPFM) with other multi-scale pyramid modules,
namely Atrous Spatial Pyramid Pooling (ASPP) from Deeplab [10], Vortex Pooling
[48], and DenseASPP [14]. In these experiments, we opted for a somewhat
lighter backbone, employing the aforementioned ResNet18 and a modified
ResNet34. The results, summarized in Table V, reveal that all the multi-scale
modules enhance the baseline. Notably, the baseline model incorporating our
proposed SPFM outperforms those with ASPP (75.64% mIoU), Vortex Pooling
(75.72% mIoU), and DenseASPP (75.21% mIoU), achieving the highest accuracy
with a 78.04 mIoU. Moreover, the SPFM variant requires fewer parameters and
exhibits a speed profile closer to ASPP. Additionally, SPFM incurs lower
computational cost, measured in terms of parameters and Floating-Point
Operations per Second (FLOPS), compared to both DenseASPP and Vortex Pooling.
This superiority could be attributed to the SPFM module’s efficacy in
extracting complex multi-scale semantic information more efficiently. SPFM
concurrently learns multiple Reduced Pyramid Pooling (RPP) blocks, utilizing
small dilation rates to generate feature maps of different resolutions,
thereby minimizing noise and enhancing its overall performance.

TABLE V: PERFORMANCE COMPARISON OF SPFM WITH STATE-OF-THE-ART MULTI-SCALE
EXTRACTOR METHODS SUCH AS ASPP, DENSEASPP, AND VORTEX POOLING ON THE
CITYSCAPES VALIDATION. FLOPS, SPEED ARE ESTIMATED FOR AN INPUT SIZE OF
512,1024. UNDER THE SPFNET34H SETTING.

Method FLOPS Params FPS mIoU(%) ResNet34+DSC baseline 268.6 37.7 19.3 72.16
ResNet34+DSC+ESAM 300.0 38.7 13.6 75.0 ResNet34+DSC+SPFM 286.6 40.8 16.3 75.8
ResNet34+DSC+ESAM+DenseASPP 380.7 77.9 8.7 75.21 ResNet34+DSC+ESAM+ASPP 304.8
41.0 12.3 75.64 ResNet34+DSC+ESAM+Vortex Pooling 322.0 49.4 11.1 75.72
ResNet34+DSC+ESAM+SPFM 317.0 41.8 12.4 78.04

###  IV-C Results on Cityscapes Dataset

Table VI presents a comprehensive comparison of the proposed SPFNet against
other state-of-the-art models on the Cityscapes dataset, considering factors
such as Floating-Point Operations per Second (FLOPS), parameters, and speed.
The experiments were conducted on both the training and validation sets,
utilizing a reduced input resolution of 2048 × 1024 2048×1024 to enhance
training efficiency. Subsequently, model segmentation accuracy was evaluated
on the test set, and the results were submitted to the Cityscapes dataset
online server footnotehttps://www.cityscapes-dataset.com/submit/ to obtain
benchmark results. The comparison encompasses both smaller networks such as
ICNet [44], BiSeNet [57], and larger networks including PSPNet [5], RefineNet
[15], and Deeplab [58]. Notably, SPFNet34H achieves a mean Intersection over
Union (mIoU) of 75.7%, featuring 41.9 million parameters and an inference
speed of 12.7 frames per second (FPS). On the other hand, SPFNet18L achieves a
mIoU of 71.9%, with 31.7 million parameters and an inference speed of 46.5
FPS. It is worth highlighting that SPFNet employs ResNet34 and ResNet18, which
have fewer parameters compared to many larger models in Table VI, yet achieve
comparable results. Importantly, our models do not employ multi-scale testing
or multi-crop evaluation, techniques often utilized to enhance accuracy by
other practitioners. Figure 8 illustrates visual examples of SPFNet’s
performance on the Cityscapes validation set, providing qualitative insights
into its segmentation capabilities.

TABLE VI: COMPARISON BETWEEN THE PROPOSED METHOD SPFNET AND OTHER STATE-OF-
THE-ARTS METHODS ON THE CITYSCAPES TEST DATASET.

Method Backbone Resolution Params (M) FLOPS (G) Speed(FPS) test set mIoU CRF-
RNN[50] VGG16 512$\times$1024 - - 1.4 ✓ 62.5 Deeplab[58] VGG16 512$\times$1024
262.1 457.8 0.25 ✓ 63.1 FCN[6] VGG16 512$\times$1024 134.5 136 2 ✓ 65.3
Dilation10[51] VGG16 512$\times$1024 140.8 - - ✓ 67.1 RefineNet[15] ResNet101
512$\times$1024 118.1 526 9.1 ✓ 73.6 PSPNet[5] ResNet101 713$\times$713 250
412.2 0.78 ✓ 78.4 SegNet[7] BiseNet18 640$\times$360 29.5 286 16.7 ✓ 57
TD4-Bise18[59] BiseNet18 1024$\times$2048 - ✓ 74.9 BiSeNet1[19] Xception39
768$\times$1536 5.8 14.8 72.3 ✓ 68.4 BiSeNet2[19] ResNet18 768$\times$1536
49.0 54.0 45.7 ✓ 74.7 ICNet[44] PSPNet50 1024$\times$2048 26.5 28.3 30.3 ✓
69.5 LBN-AA[53] LBN-AA+MobileNetV2 488$\times$896 6.2 49.5 51.0 ✓ 74.4
BiSeNetV2L[28] No 512$\times$1024 47.3 ✓ 75.3 SPFNet34H ResNet34M
512$\times$1024 41.9 317 12.7 ✓ 75.7 SPFNet18L ResNet18 512$\times$1024 31.7
61.0 46.5 ✓ 71.9

Figure 8: Visual results of our method SPFNet on Cityscapes dataset. The first
row is the original image,the second row is the ground truth while the last
row is represents the model performance.

###  IV-D Results on Camvid Dataset

Table VIII presents the segmentation accuracy of our proposed method on the
Camvid test set, offering a comparative analysis with other state-of-the-art
methods. The model was trained with an input resolution of $360\times 480$
pixels, utilizing only the training and validation samples for training and
validation, respectively. The testing samples were reserved to obtain results
for comparison with other state-of-the-art models. Notably, our model exhibits
superior accuracy across various configurations. The SPFNet34H, leveraging
ResNet34 as its backbone, achieves a significant 75.1% mean Intersection over
Union (mIoU) on the test set, accompanied by an inference speed of 33.0 frames
per second (FPS), representing a well-balanced trade-off between speed and
accuracy. The SPFNet43L, incorporating downsampling at stage 2, achieves a
71.4% mIoU with a faster inference speed of 79.3 FPS. Additionally, we
evaluate our methods with two configurations of ResNet18. SPFNet18H attains a
72.2% mIoU with an inference speed of 39.7 FPS, while SPFNet18L achieves a
70.3% mIoU with a higher inference speed of 109 FPS. These experiments
validate the efficacy of the proposed architecture design. As demonstrated in
Table VIII, training the model with extra data enhances performance for some
methods. For instance, BiSeNetV2* [28], when trained without extra data,
achieves a 72.4% mIoU, and its performance improves to 76.7% when the model is
initially trained on Cityscapes data. Similarly, BiSeNetV2-L* improves from
73.2% mIoU, without extra data, to 78.5% with the same settings. Individual
category results on the Camvid test set are reported in Table VIII. Finally,
we showcase visual results of our SPFNet method on the Camvid test set.

TABLE VII: THE ACCURACY, SPEED AND PARAMETERS COMPARISON OF THE PROPOSED
METHOD AGAINST OTHER SEMANTIC SEGMENTATION METHODS ON THE CAMVID TEST SET. ∗
INDICATES THE MODEL PRE-TRAINED ON CITYSCAPES. ”-” INDICATES THE CORRESPONDING
RESULT IS NOT REPORTED BY THE METHODS.

Method Year Resolution Params (M) Speed (FPS) mIoU Deeplabv2[10] 2017
$720\times 960$ 262.1 4.9 61.6 PSPNet[5] 2017 $720\times 960$ 250 5.4 69.1
DenseDecoder[57] 2018 $720\times 960$ - - 70.9 Dilation8[9] 2016 $720\times
960$ - 4.4 65.3 SegNet[7] 2015 $360\times 480$ 29.5 - 55.6 ENet[39] 2016
$360\times 480$ 0.37 - 51.3 DFANet-A[27] 2019 $720\times 960$ 7.8 120 64.7
DFANet-B[27] 2019 $720\times 960$ 4.8 160 59.3 BiSeNet1[19] 2018 $720\times
960$ 5.8 175 65.7 BiSeNet2[19] 2018 $720\times 960$ 49.0 116.3 68.7 ICNet[44]
2018 $720\times 960$ 26.5 27.8 67.1 DABNet[25] 2019 $360\times 480$ 0.76 66.4
CAS[60] 2020 $720\times 960$ - 169 71.2 GAS[61] 2020 $720\times 960$ - 153.1
72.8 AGLNet[52] 2020 $360\times 480$ 1.12 90.1 69.4 CGNet[62] 2020 $360\times
480$ 0.5 - 65.6 NDNet45-FCN8-LF[63] 2020 $360\times 480$ 1.1 - 57.5 LBN-AA[53]
2020 $720\times 960$ 6.2 39.3 68.0 BiSeNetV2/BiSeNetV2L[28] 2021 $720\times
960$ - 124.5/32.7 72.4/73.2 BiSeNetV2∗/BiSeNetV2L∗[28] 2021 $720\times 960$ -
124.5/32.7 76.7/78.5 SPFNet34H(ours) $360\times 480$ 41.8 33 75.1
SPFNet34L(ours) $360\times 480$ 41.8 79.3 71.4 SPFNet18H(ours) $360\times 480$
31.7 39.7 72.2 SPFNet18L(ours) $360\times 480$ 31.7 109 70.3

TABLE VIII: INDIVIDUAL CATEGORY RESULTS ON CAMVID TEST SET IN TERMS OF MIOU FOR 11 CLASSES. ”-” INDICATES THE CORRESPONDING RESULT IS NOT REPORTED BY THE METHODS. Method | Building | Tree | Sky | Car | Sign | Road | Ped | Fence | Pole | Sidewalk | Bicyclist | mIoU
---|---|---|---|---|---|---|---|---|---|---|---|---
SegNet[7] | 88.8 | 87.3 | 92.4 | 82.1 | 20.5 | 97.2 | 57.1 | 49.3 | 27.5 | 84.4 | 30.7 | 65.2
BiSeNet1[19] | 82.2 | 74.4 | 91.9 | 80.8 | 42.8 | 93.3 | 53.8 | 49.7 | 25.4 | 77.3 | 50.0 | 65.6
BiSeNet2[19] | 83.0 | 75.8 | 92.0 | 83.7 | 46.5 | 94.6 | 58.8 | 53.6 | 31.9 | 81.4 | 54.0 | 68.7
AGLNet[52] | 82.6 | 76.1 | 91.8 | 87.0 | 45.3 | 95.4 | 61.5 | 39.5 | 39.0 | 83.1 | 62.7 | 69.4
LBN-AA[53] | 83.2 | 70.5 | 92.5 | 81.7 | 51.6 | 93.0 | 55.6 | 53.2 | 36.3 | 82.1 | 47.9 | 68.0
BiSeNetV2/BiSeNetV2L[28] | - | - | - | - | - | - | - | - | - | - | - | 72.4/73.2
SPFNet34H(ours) | 86.4 | 79.2 | 92.6 | 90.3 | 56.3 | 95.9 | 68.1 | 54.5 | 42.3 | 85.4 | 74.2 | 75.1
SPFNet34L(ours) | 84.6 | 77.2 | 91.6 | 90.2 | 49.9 | 95.6 | 61.5 | 47.9 | 35.4 | 85.0 | 66.1 | 71.4
SPFNet18H(ours) | 84.7 | 78.0 | 92.5 | 89.9 | 52.0 | 95.4 | 66.0 | 43.1 | 40.7 | 84.1 | 67.7 | 72.2
SPFNet18L(ours) | 83.7 | 76.7 | 91.8 | 89.6 | 49 | 95.4 | 59.7 | 45.2 | 34.4 | 83.5 | 63.7 | 70.3

Figure 9: Visual results of our method SPFNet on Camvid test set. The first
row is the image,the second row is the prediction while the last row is ground
truth.

##  V Conclusion

This paper introduces the Subspace Pyramid Fusion Module (SPFM), a novel
method designed for learning multi-scale context information by partitioning
input feature maps into distinct subspaces. Additionally, we propose the
Efficient Shuffle Attention Module (ESAM), which leverages the channel-shuffle
operator to facilitate enhanced information communication across various sub-
features. Combining these innovations, we present a semantic segmentation
architecture, termed SPFNet, specifically tailored to address the challenges
of multi-scale feature fusion. Ablation studies conducted on the Cityscapes
dataset underscore the effectiveness of the proposed SPFM and ESAM modules.
The performance evaluation on the Camvid dataset demonstrates the efficacy of
SPFNet across various configurations. Specifically, SPFNet34H achieves a mean
Intersection over Union (mIoU) of 75.1%, SPFNet34L achieves 71.4% mIoU,
SPFNet18H achieves 72.2% mIoU, and SPFNet18L achieves 70.3% mIoU. Furthermore,
SPFNet34H and SPFNet18L exhibit competitive results on the Cityscapes test
sets, achieving mIoUs of 75.7% and 71.9%, respectively. These findings
underscore the potential of SPFNet in addressing semantic segmentation tasks
with improved accuracy and efficiency.

## References

  * [1] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [2] M. Saha and C. Chakraborty, “Her2net: A deep framework for semantic segmentation and classification of cell membranes and nuclei in breast cancer evaluation,” _IEEE Transactions on Image Processing_ , vol. 27, no. 5, pp. 2189–2200, 2018.
  * [3] M. Siam, M. Gamal, M. Abdel-Razek, S. Yogamani, M. Jagersand, and H. Zhang, “A comparative study of real-time semantic segmentation for autonomous driving,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2018, pp. 587–597.
  * [4] H. Zhao, Y. Zhang, S. Liu, J. Shi, C. C. Loy, D. Lin, and J. Jia, “Psanet: Point-wise spatial attention network for scene parsing,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 267–283.
  * [5] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2881–2890.
  * [6] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3431–3440.
  * [7] V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 12, pp. 2481–2495, 2017.
  * [8] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, “Unet++: A nested u-net architecture for medical image segmentation,” in _Deep learning in medical image analysis and multimodal learning for clinical decision support_. Springer, 2018, pp. 3–11.
  * [9] F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” _arXiv preprint arXiv:1511.07122_ , 2015.
  * [10] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 4, pp. 834–848, 2017.
  * [11] K. Sun, B. Xiao, D. Liu, and J. Wang, “Deep high-resolution representation learning for human pose estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 5693–5703.
  * [12] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 801–818.
  * [13] H. Li, P. Xiong, J. An, and L. Wang, “Pyramid attention network for semantic segmentation,” _arXiv preprint arXiv:1805.10180_ , 2018.
  * [14] M. Yang, K. Yu, C. Zhang, Z. Li, and K. Yang, “Denseaspp for semantic segmentation in street scenes,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 3684–3692.
  * [15] G. Lin, A. Milan, C. Shen, and I. Reid, “Refinenet: Multi-path refinement networks for high-resolution semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1925–1934.
  * [16] M. A. Elhassan, Y. Chen, Y. Chen, C. Huang, J. Yang, X. Yao, C. Yang, and Y. Cheng, “Ppanet: Point-wise pyramid attention network for semantic segmentation,” _Wireless Communications and Mobile Computing_ , vol. 2021, 2021.
  * [17] W. Liu, A. Rabinovich, and A. C. Berg, “Parsenet: Looking wider to see better,” _arXiv preprint arXiv:1506.04579_ , 2015.
  * [18] M. A. Elhassan, C. Huang, C. Yang, and T. L. Munea, “Dsanet: Dilated spatial attention for real-time semantic segmentation in urban street scenes,” _Expert Systems with Applications_ , vol. 183, p. 115090, 2021.
  * [19] C. Yu, J. Wang, C. Peng, C. Gao, G. Yu, and N. Sang, “Bisenet: Bilateral segmentation network for real-time semantic segmentation,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 325–341.
  * [20] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, “Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size,” _arXiv preprint arXiv:1602.07360_ , 2016.
  * [21] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” _arXiv preprint arXiv:1704.04861_ , 2017.
  * [22] N. Ma, X. Zhang, H.-T. Zheng, and J. Sun, “Shufflenet v2: Practical guidelines for efficient cnn architecture design,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 116–131.
  * [23] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1251–1258.
  * [24] M. Fan, S. Lai, J. Huang, X. Wei, Z. Chai, J. Luo, and X. Wei, “Rethinking bisenet for real-time semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 9716–9725.
  * [25] G. Li, I. Yun, J. Kim, and J. Kim, “Dabnet: Depth-wise asymmetric bottleneck for real-time semantic segmentation,” _arXiv preprint arXiv:1907.11357_ , 2019.
  * [26] E. Romera, J. M. Alvarez, L. M. Bergasa, and R. Arroyo, “Erfnet: Efficient residual factorized convnet for real-time semantic segmentation,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 19, no. 1, pp. 263–272, 2017.
  * [27] H. Li, P. Xiong, H. Fan, and J. Sun, “Dfanet: Deep feature aggregation for real-time semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 9522–9531.
  * [28] C. Yu, C. Gao, J. Wang, G. Yu, C. Shen, and N. Sang, “Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation,” _International Journal of Computer Vision_ , pp. 1–18, 2021.
  * [29] G. J. Brostow, J. Fauqueur, and R. Cipolla, “Semantic object classes in video: A high-definition ground truth database,” _Pattern Recognition Letters_ , vol. 30, no. 2, pp. 88–97, 2009.
  * [30] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 3213–3223.
  * [31] G. Ghiasi, T.-Y. Lin, and Q. V. Le, “Nas-fpn: Learning scalable feature pyramid architecture for object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7036–7045.
  * [32] S. Liu, L. Qi, H. Qin, J. Shi, and J. Jia, “Path aggregation network for instance segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8759–8768.
  * [33] M. Tan, R. Pang, and Q. V. Le, “Efficientdet: Scalable and efficient object detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 10 781–10 790.
  * [34] A. Kirillov, R. Girshick, K. He, and P. Dollár, “Panoptic feature pyramid networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 6399–6408.
  * [35] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2117–2125.
  * [36] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 4510–4520.
  * [37] X. Zhang, X. Zhou, M. Lin, and J. Sun, “Shufflenet: An extremely efficient convolutional neural network for mobile devices,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 6848–6856.
  * [38] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [39] A. Paszke, A. Chaurasia, S. Kim, and E. Culurciello, “Enet: A deep neural network architecture for real-time semantic segmentation,” _arXiv preprint arXiv:1606.02147_ , 2016.
  * [40] S. Mehta, M. Rastegari, A. Caspi, L. Shapiro, and H. Hajishirzi, “Espnet: Efficient spatial pyramid of dilated convolutions for semantic segmentation,” in _Proceedings of the european conference on computer vision (ECCV)_ , 2018, pp. 552–568.
  * [41] S. Mehta, M. Rastegari, L. Shapiro, and H. Hajishirzi, “Espnetv2: A light-weight, power efficient, and general purpose convolutional neural network,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 9190–9200.
  * [42] Y. Wang, Q. Zhou, J. Liu, J. Xiong, G. Gao, X. Wu, and L. J. Latecki, “Lednet: A lightweight encoder-decoder network for real-time semantic segmentation,” in _2019 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2019, pp. 1860–1864.
  * [43] J. Liu, Q. Zhou, Y. Qiang, B. Kang, X. Wu, and B. Zheng, “Fddwnet: a lightweight convolutional neural network for real-time semantic segmentation,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 2373–2377.
  * [44] H. Zhao, X. Qi, X. Shen, J. Shi, and J. Jia, “Icnet for real-time semantic segmentation on high-resolution images,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 405–420.
  * [45] R. P. Poudel, S. Liwicki, and R. Cipolla, “Fast-scnn: Fast semantic segmentation network,” _arXiv preprint arXiv:1902.04502_ , 2019.
  * [46] Z. Gu, J. Cheng, H. Fu, K. Zhou, H. Hao, Y. Zhao, T. Zhang, S. Gao, and J. Liu, “Ce-net: Context encoder network for 2d medical image segmentation,” _IEEE transactions on medical imaging_ , vol. 38, no. 10, pp. 2281–2292, 2019\.
  * [47] R. Saini, N. K. Jha, B. Das, S. Mittal, and C. K. Mohan, “Ulsam: Ultra-lightweight subspace attention module for compact convolutional neural networks,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2020, pp. 1627–1636.
  * [48] C.-W. Xie, H.-Y. Zhou, and J. Wu, “Vortex pooling: Improving context representation in semantic segmentation,” _arXiv preprint arXiv:1804.06242_ , 2018.
  * [49] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, “Contour detection and hierarchical image segmentation,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 33, no. 5, pp. 898–916, 2010.
  * [50] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du, C. Huang, and P. H. Torr, “Conditional random fields as recurrent neural networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1529–1537.
  * [51] F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions (2015),” _arXiv preprint arXiv:1511.07122_ , 2016.
  * [52] Q. Zhou, Y. Wang, Y. Fan, X. Wu, S. Zhang, B. Kang, and L. J. Latecki, “Aglnet: Towards real-time semantic segmentation of self-driving images via attention-guided lightweight network,” _Applied Soft Computing_ , vol. 96, p. 106682, 2020.
  * [53] G. Dong, Y. Yan, C. Shen, and H. Wang, “Real-time high-performance semantic image segmentation of urban street scenes,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 22, no. 6, pp. 3258–3274, 2020.
  * [54] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [55] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” in _NIPS 2017 Workshop on Autodiff_ , 2017. [Online]. Available: https://openreview.net/forum?id=BJJsrmfCZ
  * [56] S. R. Bulo, L. Porzi, and P. Kontschieder, “In-place activated batchnorm for memory-optimized training of dnns,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 5639–5647.
  * [57] P. Bilinski and V. Prisacariu, “Dense decoder shortcut connections for single-pass semantic segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 6596–6605.
  * [58] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Semantic image segmentation with deep convolutional nets and fully connected crfs,” _arXiv preprint arXiv:1412.7062_ , 2014.
  * [59] P. Hu, F. Caba, O. Wang, Z. Lin, S. Sclaroff, and F. Perazzi, “Temporally distributed networks for fast video semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 8818–8827.
  * [60] Y. Zhang, Z. Qiu, J. Liu, T. Yao, D. Liu, and T. Mei, “Customizable architecture search for semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 11 641–11 650.
  * [61] P. Lin, P. Sun, G. Cheng, S. Xie, X. Li, and J. Shi, “Graph-guided architecture search for real-time semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 4203–4212.
  * [62] T. Wu, S. Tang, R. Zhang, J. Cao, and Y. Zhang, “Cgnet: A light-weight context guided network for semantic segmentation,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 1169–1179, 2021.
  * [63] Z. Yang, H. Yu, M. Feng, W. Sun, X. Lin, M. Sun, Z.-H. Mao, and A. Mian, “Small object augmentation of urban scenes for real-time semantic segmentation,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 5175–5190, 2020.