WGMwgm QEqe EPep PMSpms BECbec DEde

[1]This work is supported by the National Natural Science Foundation of China
(Nos.62272418,62102058), Basic public welfare research program of Zhejiang
Province(No.LGG18E050011).

[cor1]Corresponding authors
<EMAIL_ADDRESS>(Mohammed A. M. Elhassan)
<EMAIL_ADDRESS>(Changjun Zhou)
<EMAIL_ADDRESS>(Amina Benabid)
abuzar@cqupt.edu.cn(Abuzar B. M. Adam)

# P2AT: Pyramid Pooling Axial Transformer for Real-time Semantic Segmentation

Mohammed A. M. Elhassana,c Changjun Zhoua,* Amina Benabidb,c Abuzar B. M.
Adamd School of Computer Science, Zhejiang Normal
University,Jinhua,321004,Zhejiang,P.R.China  School of Mathematical Medicine,
Zhejiang Normal University,Jinhua,321004,Zhejiang,P.R.China  Zhejiang
Institute of Photoelectronics & Zhejiang Institute for Advanced Light Source,
Zhejiang Normal University  School of Communications and Information
Engineering, Chongqing University of Posts and Telecommunications,
Chongqing,40065,P.R.China

###### Abstract

Recently, Transformer-based models have achieved promising results in various
vision tasks, due to their ability to model long-range dependencies. However,
transformers are computationally expensive, which limits their applications in
real-time tasks such as autonomous driving. In addition, an efficient local
and global feature selection and fusion are vital for accurate dense
prediction, especially driving scene understanding tasks. In this paper, we
propose a real-time semantic segmentation architecture named Pyramid Pooling
Axial Transformer (P2AT). The proposed P2AT takes a coarse feature from the
CNN encoder to produce scale-aware contextual features, which are then
combined with the multi-level feature aggregation scheme to produce enhanced
contextual features. Specifically, we introduce a pyramid pooling axial
transformer to capture intricate spatial and channel dependencies, leading to
improved performance on semantic segmentation. Then, we design a Bidirectional
Fusion module (BiF) to combine semantic information at different levels.
Meanwhile, a Global Context Enhancer is introduced to compensate for the
inadequacy of concatenating different semantic levels. Finally, a decoder
block is proposed to help maintain a larger receptive field. We evaluate P2AT
variants on three challenging scene-understanding datasets. In particular, our
P2AT variants achieve state-of-art results on the Camvid dataset 80.5%, 81.0%,
81.1% for P2AT-S, P2AT-M, and P2AT-L, respectively. Furthermore, our
experiment on Cityscapes and Pascal VOC 2012 have demonstrated the efficiency
of the proposed architecture, with results showing that P2AT-M, achieves 78.7%
on Cityscapes. The source code will be available at
111https://github.com/mohamedac29/P2AT.

###### keywords:

Vision TransformerAxial attentionPyramid poolingMulti-scale feature
fusionReal-time semantic segmentationlightweight neural network.

##  1 Introduction

Perception is a vital task of any intelligent driving system, which collects
the necessary information about the surrounding environment of the moving
vehicle. As an inseparable part of automatic driving, visual perception is
being explored and researched by major mainstream automobile manufacturers,
enterprises, universities, and scientific research institutes. The large-scale
application of artificial intelligence in the automotive industry accelerates
the development in this field. High precision and speed architectures are
crucial for the future development of advanced driver assistance systems and
autonomous vehicles. The Research in visual perception algorithms based on
deep learning is a very important part of landing industrial technology
applications because deep learning methods have unique abilities for
constructing robust intelligent driving algorithms in many research
directions, such as traffic sign recognition [1, 2], lane detection [3, 4],
target detection [5, 6], driving free space recognition and semantic
segmentation [7, 8]. Fast and accurate semantics segmentation and target
detection are prerequisites for safe, intelligent driving.

Figure 1: The inference speed and accuracy trade-off for real-time models on
the Cityscapes [9] test set. red color refers to our models, while black
represents others.

As the complexity of image content increases, the task of semantic
segmentation becomes progressively more challenging due to intricate
structures and variations in color, texture, and scale. In recent years, deep
learning has significantly influenced the development of various semantic
segmentation approaches, emerging as the dominant framework. Many state-of-
the-art methods for semantic segmentation, such as those described in [7, 10,
11, 12], utilize Fully Convolutional Networks (FCNs) [13] as fundamental
components. Notably, PSPNet [10] and Deeplab [7] introduce specialized modules
for capturing multi-scale contextual information, namely the Pyramid Pooling
Module (PPM) and the Atrous Spatial Pyramid Module (ASPP), respectively.
Despite these advancements, challenges persist, particularly when handling
complex image content, as existing approaches tend to generate imprecise
masks.

In recent years, the remarkable performance of Vision Transformer (ViT) in
image classification [14, 15] has spurred efforts to extend its application to
semantic segmentation tasks. These endeavors have significantly improved over
previous semantic segmentation convolutional neural networks (CNNs) [16, 17,
18, 19]. However, implementing pure transformer models for semantic
segmentation has a considerable computational cost, particularly when dealing
with large input images. To address this issue, [16] has introduced
hierarchical Vision Transformers, which offer a more computationally efficient
alternative. SegFormer [18] has proposed refined design for the encoder and
decoder, resulting in an efficient semantic segmentation ViT. Nevertheless,
one concern with SegFormer is its heavy reliance on increasing the model
capacity of the encoder as the primary means of improving performance,
potentially limiting its overall efficiency.

Unlike the aforementioned methods that introduced pure Transformer for dense
pixel prediction, we propose a hybrid architecture for better and more
efficient semantic segmentation of autonomous driving. Specifically, since
contextual information is crucial for semantic segmentation, we exploit the
Pyramid Pooling Axial Transformer in CNN to capture global context information
effectively. To fully utilize the merit of Transformer and CNN, a
bidirectional fusion module is proposed to integrate the feature from the
network encoder and global context information, which is then refined using
the global context enhancer. As a hybrid ConvNet-Transformer framework, our
P2AT can accurately segment objects in the scene for autonomous driving with
faster inference speed.

Our main contributions are summarized as follows:

  1. 1.

We introduce a novel pyramid pooling Axial Transformer framework (P2AT) for
real-time semantic segmentation. To achieve an accuracy/speed trade-off, four
modules, including, Scale-aware context aggregator module, multi-level feature
fusion module, a decoder , and feature refinement module are designed, leading
to the following contributions.

  2. 2.

We encapsulate pyramid pooling to the Axial Transformer to extract contextual
features, leading to powerful architecture that is easier to train in small
datasets.

  3. 3.

We introduce a multi-level fusion module to fuse encoded detailed
representations and deep semantic features. Specifically, a bidirectional
fusion (BiF) module based on semantic feature upsampler (SFU) and local
feature refinement (LFR) is designed to obtain efficient feature fusion.

  4. 4.

We introduce a global context enhancer (GCE) module to compensate for the
inadequacy of concatenating different semantic levels.

  5. 5.

We propose an efficient decoder based on enhanced ConvNext and a feature
refinement module proposed to remove noises to enhance the final prediction.

  6. 6.

We evaluate P2AT on three challenging scene understanding datasets: Camvid,
Cityscapes, and PASCAL VOC 2012. The results show that P2AT achieves state-of-
the-art results.

Figure 2: The architecture of P2AT. (a) Encoder based on pre-trained ResNet,
(b) Transformer Layers to extract contextual information, (c) Multi-stage
Feature Fusion Block (d) Decoder Block (e) Feature Refinement Block.

##  2 Related Work

###  2.1 Semantic Segmentation

Semantic segmentation has witnessed significant advancements in the deep
learning era [20, 21, 22]. The introduction of Fully Convolutional Networks
(FCNs) [13] revolutionized semantic segmentation by enabling end-to-end pixel-
to-pixel classification. Building upon FCNs, researchers have explored various
avenues to enhance semantic segmentation performance. Efforts have been
dedicated to enlarging the receptive field [10, 23, 24, 21, 7, 25],
incorporating boundary information [26, 27, 28], refining contextual
information [29, 30, 31, 32, 33], integrating attention modules [34, 35, 36,
12, 37], and leveraging AutoML technologies [38, 39, 40] to design optimized
models for scene parsing. These approaches have substantially improved the
accuracy of semantic segmentation. More recently, Transformer-based
architectures have demonstrated their efficacy in semantic segmentation [19,
41]. Nevertheless, these methods still require significant computational
resources, which limits their applications in real-time tasks such as
autonomous driving. To speed up the segmentation and reduce the computational
cost, ICNet [42] proposes a cascade network with multi-resolution input image.
DFANet [43] utilizes a lightweight backbone to speed up its network and
proposes a cross-level feature aggregation to boost accuracy. SwiftNet [44]
uses lateral connections as the cost-effective solution to restore the
prediction resolution while maintaining the speed. BiSeNet [45], ContextNet
[46], GUN [47], and DSANet [34] introduce spatial and semantic paths to reduce
computation. SFNet [48] aligns feature maps from adjacent levels and further
enhances the feature maps using a feature pyramid framework. ESPNet [49] save
computation by decomposing standard convolution into point-wise convolution
and spatial pyramid of dilated convolutions.

Vision Transformer: Originally introduced in Natural Language Processing (NLP)
tasks, have gained significant traction in computer vision. These models,
relying on self-attention mechanisms, excel at capturing long-range
dependencies among tokens in sentences. Moreover, transformers possess
inherent parallelization capabilities, enabling efficient training on
extensive datasets. Inspired by the success of transformers in NLP, several
methodologies have emerged in computer vision that combines Convolutional
Neural Networks (CNNs) with various forms of self-attention to tackle diverse
tasks such as object detection, semantic segmentation, panoptic segmentation,
video processing, and few-shot classification. Vision Transformer (ViT) [14]
presents a convolution-free transformer model for image classification. ViT
processes input images as sequences of patch tokens, revolutionizing the
traditional approach. Although ViT necessitates training on large-scale
datasets to achieve optimal performance, DeiT [15] proposes a token-based
distillation strategy that leverages a CNN as a teacher model, resulting in a
competitive vision transformer trained on the ImageNet-1k dataset [50].
Building upon this foundation, concurrent research has extended transformer-
based models to various domains, including video classification [51, 52] and
semantic segmentation [16, 19]. SETR [19] combines a ViT backbone with a
standard CNN decoder. In contrast, Swin Transformer [16] adopts a variant of
ViT that employs local windows shifted across layers. These advancements
further demonstrate the versatility and effectiveness of Transformer-based
architectures in computer vision tasks.

In this work, we present P2AT, a novel hybrid CNN-Transformer based encoder-
decoder architecture, designed for semantic image segmentation. Our method
leverages the powerful CNN backbone and integrates a Pyramid Pooling Axial
Transformer as a global context information aggregator. Through extensive
evaluations of widely adopted image segmentation benchmarks, our proposed
method demonstrates competitive performance, showcasing its efficacy and
potential for advancing real-time semantic image segmentation.

##  3 Methodology

###  3.1 Overall Architecture of P2AT

Figure 2 illustrates a diagram of the proposed P2AT. First, we present an
overview of the proposed P2AT for real-time semantic image segmentation. Then,
we analyze in detail the importance of several key elements that construct the
model, including: (a) an encoder based on pre-trained ResNet [21] (b) the
pyramid pooling axial attention which is one of the main building blocks of
the method (c) the bidirectional fusion module which is used to fuse features
of different stages efficiently, (d) decoder block.
Given an input image $I\in\mathbb{R}^{H\times W\times C}$ with channels C and
spatial resolution W, H, we first utilize ResNet[36] to generate high-level
features, and then integrate the proposed Transformer layers to complement the
CNN on modeling the contextual features. Then, fed these features to a decoder
that is introduced to maintain the global contextual information. After that,
these high semantic features are fused with the features from low-level
features through the bidirectional fusion module; BiF is efficient at
combining features of different semantics. Finally, we enhanced the output
feature using a global context enhancer module and refined them before the
final prediction.

###  3.2 Global and Local Feature Importance

A typical encoder-decoder framework usually utilizes the shallow layers to
encode the high-resolution feature maps that carry target object detail
information and the deeper layers to encode the higher semantics. However,
simple upsampling strategies such as bilinear interpolation and deconvolution
are incapable of collecting global context and restoring the missing
information during the downsampling process. In this part of our work, we aim
to improve semantic segmentation performance by designing a network capable of
overcoming some of the problems of the encoder-decoder architecture. For this
purpose, several modules and blocks have been developed and combined to
construct the P2AT.

###  3.3 Bidirectional Fusion Module

To efficiently combine the encoded features representations from a low-level
encoder and high semantic features of the decoding module, we propose a new
bidirectional fusion module (Figure 3) that integrates both channel attention
which used to transform the low-level features through the local feature
refinement block, a semantic feature injection, and multi-stage multi-level
fusion mechanisms. In equation 3, $F_{BiF}$ is the multi-level fusion
function.

$B=F_{BiF}(D,L,F_{s})$ (1)

Where D is the semantic descriptor, L denotes the detailed object features,
and $F_{s}$ is the feature of the stage.

Figure 3: Efficient Bidirectional fusion Module.

Semantic Feature Upsampler (SFU) block: SFU is proposed to gather semantic
features, as shown in Figure 3.(a). $D\in\mathbb{R}^{H\times W\times C}$
denote the output of layer $D_{5}$, and $D_{4}$ of the decoder. Feature
Pyramid Network [53] is a simple architecture to propagate semantic features
into rich detail features in the lower levels. By fusing semantic features
with multi-scale features, the performance has been improved substantially in
object detection and semantic segmentation. However, the process of reducing
channels over stages causes a loss of important information. This paper
introduces the semantic feature upsampler, a simple yet efficient upsampling
that uses an attention mechanism to selectively inject global features into
the BiF module. We formulated the semantic feature upsampler
$S\in\mathbb{R}^{H\times W\times C}$ as follows:

$S=\alpha(D;W_{\alpha})\odot softmax(\beta(D;W_{\beta})$ (2)

Where different $1\times 1$ convolutional layers ($\alpha$ and $\beta$) are
used to map the input D, $\odot$ denotes the Hadamard product.

Local Feature Refinement block The proposed architecture 2 employs a
bidirectional fusion design to facilitate the flow of information from
different stages during the training process. In order to maintain the fusion
of consistent semantic features, we introduce channel attention (Figure 3.b)
to gather global information. In parallel to that, a spatial filter is
integrated to suppress irrelevant information and enhance local details, as
low-level encoder features could be noisy. the local feature refinement block
is formulated as follows:

$F_{g}=sigmoid(\eta(F;W_{\eta})$ (3)

Where $\eta$ refers to a convolutional layer with kernels of $1\times 1$,

$L=\propto(F_{g}\oplus\gamma(G(F);W_{\gamma});W_{\propto})$ (4)

Where ($\gamma$, $\propto$) represent convolution with kernels of $1\times 1$,
G denotes the global average pooling.

Global Context Enhancer (GCE) is introduced to compensate for the inadequacy
of concatenating different semantic levels. Given the input feature $F_{B}iF$,
the global context enhancer module first applies global average pooling to
gather global semantic information on and uses a gating mechanism to
selectively choose the informative high semantic descriptors, which help
remove the noise that can be introduced into the BiF by the shallower stages
of the encoder.

Figure 4: Global Context Enhancer Module.

###  3.4 Feature Decoding and Refinement

The decoder block (refer to Figure.4.(b))constructs of depth-wise convolution
with kernel sizes of 3, 5 and 7 for stages 5, 4, and 3, respectively, followed
by batch normalization. Then, we employ two point-wise convolution layers to
enrich the local representation and help maintain object context. Unlike
ConvNeXt [54], which has used Layer Normalization and Gaussian error Unit
activation, we have used Hardswish activation [55] for non-linear feature
mapping. Finally, a skip connection is added to facilitate the information
flow across the network hierarchy. This decoder can be represented as follows:

Figure 5: Illustrates the detail of the (a) feature refinement module and (b)
feature decoding block.

$D_{i+1}=D_{i}+f_{L}^{H}(f_{L}(f^{DW}_{k\times k}(D_{i})))$ (5)

where $D_{i}$ is an input feature maps of shape $H\times W\times C$,
$f_{L}^{H}$ denotes the point-wise convolution layer followed by Hardswish,
$f^{DW}$ is a depth-wise convolution with kernel size of $k\times k$, and
$D_{i+1}$ denotes the output feature maps of the decoder block.

The refinement block (Figure. 5.b is introduced to filter the noisy features
that produced by the decoder more accurate per-pixel classification and
localization.

Figure 6: Feature visualization of the refinement module. Left to right are
original image, ground truth, output of decoder block, and the maps of the
refined feature.

###  3.5 Scale-aware Semantic Aggregation Block

The scale-aware semantic aggregator consists of L number of stacked Pyramid
Pooling Axial Transformer blocks. Each Transformer comprises a pyramid pooling
axial attention module and Feed-Forward Network (FFN). The output of the
$l-th$ $(l\in[1,2,..,L])$ Axial Transformer blocks can be presented as
follows:

$\acute{z}_{l}=P2A2(z_{l-1})+z_{l-1}$ (6)

$z_{l}=MLP(\acute{z}_{l})+\acute{z}_{l}$ (7)

Where $z$, $z_{l}$, and $z_{l}$ denote the input, output of the axial
attention and the output of the Transformer block, respectively. P2A2 is the
abbreviation of Pyramid Pooling Axial Attention.

Pyramid Pooling Axial Attention Here we present the proposed pyramid pooling
axial attention. The main structure is illustrated in Figure 7. First, the
input features are fed into a pyramid pooling submodule, which captures global
context information by performing pooling operations with different kernel
sizes, Equation 12. Next, the axial attention module is applied to the pooled
features to capture spatial dependencies in both the vertical and horizontal
axes. This module leverages positional embedding [56] to encode spatial
information and generate attention maps that highlight relevant regions in the
image. The attention maps are then combined with residual blocks to enhance
the spatial details, allowing the model to generate informative contextual
information that enables our architecture to obtain high accuracy on a very
small dataset.

$X=\theta(F_{in};W_{\theta})$ (8) $\textbf{P}_{1}=AvgPool_{3}(X)$ (9)
$\textbf{P}_{2}=AvgPool_{5}(\textbf{P}_{1})$ (10)
$\textbf{P}_{3}=AvgPool_{7}(\textbf{P}_{2})$ (11)

Where $\textbf{P}_{3}$, $\textbf{P}_{5}$ , $\textbf{P}_{7}$ is the generated
pyramid features. Next, we sum up the pyramid feature maps and employ a
convolutional operation on top of it.

$\textbf{P}^{pp}=W_{3,3}^{DW}([P_{3},P_{5},P_{7}])$ (12)

Figure 7: Right: Illustration of the proposed Scale-aware Semantic Aggregation
Block including a Pyramid Pooling Axial attention and a Feed-Forward Network
(FFN). Left is the Pyramid Pooling Axial Attention, Pyramid Pooling and Axial
Attention.

Axial Attention [57, 58] has been introduced to reduce the computation cost of
the attention network. Since then, it has been integrated in many semantic
segmentation frameworks [59, 60].

Table 1: The Per-class, class, and category IoU evaluation on the Cityscapes
Test set. List of classes from left to right: Road, Sidewalk, Building, wall,
Fence, Pole, Traffic light, Traffic sign, Vegetation, Terrain, Sky,
Pedestrian, Rider, Car, Truck, Bus, Train, Motorbike, and Bicycle."cla->mIoU".
”-” indicates the corresponding result is not reported by the methods

Method Road Sidewalk Building Wall Fence Pole Traffic light Traffic sign
Vegetation Terrain Sky Person Rider Car Truck Bus Train Motor Bicyclist mIoU
CRF-RNN[61] 96.3 73.9 88.2 47.6 41.3 35.2 49.5 59.7 90.6 66.1 93.5 70.4 34.7
90.1 39.2 57.5 55.4 43.9 54.6 62.5 FCN[13] 97.4 78.4 89.2 34.9 44.2 47.4
60.1.5 65.0 91.4 69.3 93.9 77.1 51.4 92.6 35.3 48.6 46.5 51.6 66.8 65.3
DeepLabv2[7] - - - - - - - - - - - - - - - - - - - 70.4 Dilation10[62] 97.6
79.2 89.9 37.3 47.6 53.2 58.6 65.2 91.8 69.4 93.7 78.9 55.0 93.3 45.5 53.4
47.7 52.2 66.0 67.1 AGLNet[63] 97.8 80.1 91.0 51.3 50.6 58.3 63.0 68.5 92.3
71.3 94.2 80.1 59.6 93.8 48.4 68.1 42.1 52.4 67.8 70.1
BiSeNetV2/BiSeNetV2_L[45] - - - - - - - - - - - - - - - - - - - 73.2 LBN-
AA[64] 98.2 84.0 91.6 50.7 49.5 60.9 69.0 73.6 92.6 70.3 94.4 83.0 65.7 94.9
62.0 70.9 53.3 62.5 71.8- 73.6 P2AT-S 98.4 84.4 92.9 54.4 56.6 67.7 75.2 78.1
93.5 71.9 95.6 85.7 69.2 95.5 62.2 81.1 76.3 64.7 74.9 77.8 P2AT-M 98.5 85.7
93.0 58.6 58.3 67.3 74.7 77.7 93.5 72.2 95.6 85.3 69.2 95.6 68.1 83.0 78.3
65.9 75.1 78.7 P2AT-L 98.5 85.3 92.6 53.1 57.4 66.6 74.7 78.1 93.3 69.8 95.1
86.0 70.5 95.8 67.8 83.3 72.2 67.0 74.8 78.0

##  4 Experimental Results

In this section, we present the results of comprehensive experiments that were
conducted to evaluate the effectiveness of P2AT on three challenging
benchmarks: Camvid [65], Cityscapes [9], and PASCAL VOC 2012 [66]. We first
present the datasets and implementation details. Then, the ablation studies
performed to ensure the validity of each component in P2AT. Finally, we
compare our method with other state-of-the-art networks using the standards
metrics: number of trainable parameters, floating point operations (flops),
and class mean intersection over union (mIoU)

###  4.1 Experiments Settings

####  4.1.1 Camvid

Camvid is a challenging road scene understanding dataset consisting of 376
training images, 101 validation images, and 233 test images. The dataset is
small, with only 376 training images, and the distribution of labels is
unbalanced. This makes it difficult for models to learn to segment images
accurately. To make a fair comparison with other state-of-the-art models, we
evaluated our model on 11 classes, including building, sky, tree, car, and
road. The class 12th was marked as ignore class to hold the unlabelled data..

####  4.1.2 Cityscapes

The Cityscapes dataset is a benchmark for urban scene understanding. It
contains 5000 high-resolution images of 2048x1024 pixels with fine annotations
captured from different cities. The dataset is divided into 2975 training
images, 500 validation images, and 1525 test images. We evaluated our model on
19 semantic segmentation classes, including road, sidewalk, building,
vegetation, and sky. Despite the challenges, the Cityscapes dataset is a
valuable resource for researchers and practitioners working on urban scene
understanding. The dataset provides a large and diverse set of images that can
be used to train and evaluate models.

####  4.1.3 Implementation Details

Our training settings are closely aligned with previous works [7, 10].
Specifically, we have employed SGD optimizer with a poly-learning rate
strategy. Additionally, we have incorporated various data augmentation
techniques that have been used by other methods for a fair comparison,
including random horizontal flipping, random cropping, and random scaling
within the range of 0.5 to 2.0. For the Cityscapes, Camvid, and PASCAL VOC
datasets, we have established the following key training parameters: 500
epochs, an initial learning rate of $1e-2$, weight decay of $5e-4$, cropped
image size of $1024\times 1024$, and a batch size of 12 for Cityscapes; 140
epochs, an initial learning rate of $5e-3$, weight decay of $5e-4$, cropped
image size of $960\times 720$, and a batch size of 12 for Camvid; and 200
epochs, an initial learning rate of $1e-3$, weight decay of $1e-4$, cropped
image size of $512\times 512$, and a batch size of 16 for PASCAL VOC. During
the inference stage, prior to testing, our models underwent training using
both the training and validation sets for the Cityscapes and Camvid datasets.
To evaluate the inference speed, we conducted measurements using a platform
equipped with a single RTX 3090 GPU, PyTorch [67] 1.8, CUDA 11.7, cuDNN 8.0,
and an Ubuntu environment. We conducted thorough evaluations of the inference
speed to ensure the robustness and validity of our results.

Figure 8: Feature maps visualization of our proposed architecture on the
Cityscapes validation dataset. From left to right: (a) original input images;
(b) The ground truth; (c) Scale-aware Semantic Aggregation block ; (d)
bidirectional fusion (stage4) maps; (e) global context enhancer (stage 4); (f)
the feature maps of the segmentation head.

###  4.2 Ablation Study

We perform ablation studies on Camvid and Pascal VOC2012 datasets to evaluate
the impact of each component in the proposed P2AT. The results on Camvid
reported on the testing set, while Pascal VOC2012 results reported on the
validation set.

###  4.3 Results on Pascal VOC2012

PASCAL VOC 2012 [66] consists of images with different resolutions,
representing 21 classes, including the background class. The dataset comprised
of 1,464 training, 1,449 validation, and 1,456 test images. The training set
was extended to a total of 10,582 images [68]. While this dataset is not
commonly used for evaluating real-time segmentation methods, its low-
resolution images can be utilized to run ablation studies, making it suitable
for initial tests. Table 2 presents backbone, accuracy, flops, and parameters
for our model compared to existing work. We specifically chose methods that
reported results on the PASCAL VOC validation set without employing inference
strategies such as horizontal mirroring and multi-scale testing. The results
indicate that our medium size of our method P2AT-M achieves competitive
accuracy of 79.6%mIoU

Table 2: PERFORMANCE COMPARISON OF P2AT VARIANTS WITH STATE-OF-THE-ART METHODS
ON PASCAL VOC 2012 VALIDATION SET.

Method Backbone Params Flops mIoU(%) DeepLabV3[69] ResNet-101 58.6 249.2 78.5
Auto-DeepLab-L[40] - 44.42 79.3 73.6 DeepLabV3Plus[70] Xception-71 43.48 177
80.0 DFN[71] ResNet-101 - - 79.7 SDN[72] DenseNet-161 238.5 - 79.9
HyperSeg-L[73] EfficientNet-B1 39.6 8.21 80.6 P2AT-M ResNet-34 41.7 37.5 79.6

###  4.4 Ablation on Camvid and Cityscapes

In this subsection, we conduct a series of experiments to explore different
variants of our proposed method.

Table 3: COMPARISON BETWEEN P2AT-S, P2AT-M, AND P2AT-L IN TERMS OF PARAMETERS,
FLOPS, FRAMES PER SECOND (FPS), AND ACCURACY ON CITYSCAPES AND CAMVID DATASETS

Method Backbone Params Cityscapes Camvid Flops FPS mIoU % Flops FPS mIoU %
P2AT-S ResNet-18 12.6 80.2 80.1 78.0 73 113.6 80.5 P2AT-M ResNet-34 22.7 118
61.8 79.8 96 86.6 81.0 P2AT-L ResNet-50 37.5 166 40.6 78.8 112 56.1 81.1

Table 3 presents the results of the ablation study on variants of P2AT models
for semantic segmentation in Camvid and Cityscapes datasets. P2AT-S has a more
compact model architecture than the other variants, with only 0.8% mIoU less
than P2AT-L and 1.8 mIoU than the variant with the highest accuracy P2AT-M.
The different settings of P2AT have achieved very high performance on the
Camvid test set and competitive results on the cityscapes and Pascal VOC2012
datasets.

Model complexity vs. performance: As the model complexity increases (from
P2AT-S to P2AT-L), the number of parameters and flops increases. However, the
impact on performance varies. P2AT-M achieves higher mIoU than P2AT-S,
indicating a better trade-off between model complexity and accuracy. P2AT-L,
despite having higher parameters and flops, achieves a slightly lower mIoU
than P2AT-M, suggesting diminishing returns in performance improvement with
increased model complexity or needing more optimization to utilize the whole
parameters. Overall, the ablation study provides insights into the trade-offs
between model complexity, computational efficiency, and accuracy in semantic
segmentation for road scene understanding

###  4.5 Results on Cityscapes Dataset

Table 4 provides a comprehensive comparison between three variants of P2AT
(P2AT-S, P2AT-M, P2AT-L) and other state-of-the-art (SOTA) approaches on the
Cityscapes test dataset. The comparison includes various aspects such as
backbone architecture, input resolution, GPU type, number of parameters, flops
(floating-point operations), evaluation split(val, test), achieved accuracy
(mIoU), and inference speed (FPS). Firstly, it is evident that the proposed
P2AT-S, P2AT-M, and P2AT-L models obtain a competitive result on the
Cityscapes dataset. P2AT-S, based on ResNet-18, achieves an accuracy of 77.8%
mIoU on the test set, outperforming many existing methods such as
SwiftNetRN-18, SFNet (DF1), FANet-18, LBN-AA, and AGLNet. P2AT-M, using
ResNet-34, further improves the performance, obtaining an accuracy of 78.7%
mIoU, surpassing several methods, including transformer-based methods such as
SeaFormer-S, and SegFormer, as well as other plane methods, PP-LiteSeg-T2, and
HyperSeg-M. P2AT-L, utilizing ResNet50, achieves a mIoU of 78%, comparable to
other top-performing methods like HyperSeg-S and BiSeNetV1. Compare to the
SOTA methods, it is notable that our proposed P2AT models demonstrate
competitive accuracy while maintaining reasonable computational efficiency.
P2AT-S achieves comparable accuracy to SegFormer, SwiftNetRN-18, and SFNet
(DF1), while offering faster inference speeds. P2AT-M exhibits improved
accuracy compared to SwiftNetRN-18, PP-LiteSeg-T2, and HyperSeg-M, while
maintaining a comparable inference speed. P2AT-L achieves competitive accuracy
similar to HyperSeg-S and BiSeNetV1, although at a slightly lower inference
speed. Additionally, the proposed P2AT models exhibit favorable trade-offs
between accuracy and efficiency compared to other existing methods. For
instance, P2AT-M achieves a higher mIoU than SeaFormer-S, SeaFormer-B, and
SwiftNetRN-18 ens, while being more computationally efficient in terms of
parameters and flops. Similarly, P2AT-L achieves comparable accuracy to
HyperSeg-S and BiSeNetV1 with a significantly lower number of parameters and
flops. These observations highlight the effectiveness of the proposed models
in achieving competitive performance while maintaining computational
efficiency. Furthermore, it is important to note that the proposed P2AT models
leverage different backbone architectures (ResNet-18, ResNet-34, and
ResNet-50). The use of deeper backbones (P2AT-M and P2AT-L) contributes to
improved performance, as evident from their higher mIoU values compared to
P2AT-S. This demonstrates the importance of feature representation capacity in
achieving better accuracy, but a well optimized training is needed to benefit
from the larger backbones. In terms of input resolution, all P2AT models adopt
a fixed resolution of 1024x1024. Despite using a relatively lower resolution
compared to some methods like SwiftNetRN-18 (1024x2048) and SFNet (DF1)
(1024x2048), the proposed models achieve competitive accuracy. This suggests
that the integration of pyramid pooling axial attention, and proposed feature
fusion modules in the P2AT architecture enables effective multi-scale feature
extraction, compensating for the lower input resolution. In conclusion, the
proposed P2AT variants exhibit competitive performance on the Cityscapes
dataset compared to other SOTA methods. They achieve high accuracy while
maintaining reasonable computational efficiency.

Table 4: COMPARISON BETWEEN THE PROPOSED METHOD S2-FPN AND THE OTHER SOTA
METHODS ON THE CITYSCAPES TEST DATASET. WE REPORT THE BACKBONE, INPUT
RESOLUTION, GPU TYPE, NUMBER OF PARAMETERS (M), FLOPS (G), EVALUATION SPLIT
(SET), ACHIEVED ACCURACY (MIOU), AND THE INFERENCE SPEED (FPS)

Method Backbone Resolution GPU #Params #GFLOPs #FPS mIoU Val Test
SeaFormer-S[56] SeaFormer-S 1024$\times$1024 GTX 1080Ti - 8.0 - 76.1 75.9
SeaFormer-B[56] SeaFormer-B 1024$\times$1024 GTX 1080Ti - 13.7 77.7 77.5
SegFormer MiT-B0 1024$\times$1024 Tesla V100 3.8 125.5 15.2 - 76.2
SwiftNetRN-18[44] ResNet-18 1024$\times$2048 GTX 1080Ti 11.8 104.0 39.9 75.5
75.4 SwiftNetRN-18 ens[44] ResNet-18 1024$\times$2048 GTX 1080Ti 24.7 218.0
18.4 - 76.5 PP-LiteSeg-T2[74] STDC1 768$\times$1536 GTX 1080Ti - - 143.6 76.0
74.9 PP-LiteSeg-B2[74] STDC2 768$\times$1536 GTX 1080Ti - - 102.6 78.2 77.5
ENet[61] No 640 $\times$360 TitanX 0.4 3.8 135.4 - 58.3 ICNet[23] PSPNet50
1024$\times$2048 TitanX 26.5 28.3 30.3 - 69.5 DABNet[13] No 512$\times$1024
GTX 1080Ti 0.76 10.4 104 - 70.1 DFANet-A[43] XceptionA 1024$\times$1024 Titan
X 7.8 3.4 100 - 71.3 DFANet-B[43] XceptionB 1024$\times$1024 Titan X 4.8 2.1
120 - 67.1 FasterSeg[39] No 1024$\times$2048 GTX 1080Ti - - 163.9 71.5
TD4-Bise18[75] BiseNet18 1024$\times$2048 Titan Xp - - - - 74.9 SFNet(DF1)[48]
DF1 1024$\times$2048 GTX 1080Ti GPU 9.03 - 74 - 74.5 SFNet(DF2)[48] DF2
1024$\times$2048 GTX 1080Ti GPU 10.53 - 53.0 - 77.8 FANet-18[76] ResNet-18
1024$\times$2048 Titan X - 49 72 74.4 FANet-34[76] ResNet-34 1024$\times$2048
Titan X - 65 58 - 75.5 LBN-AA[64] LBN-AA+MobileNetV2 488$\times$896 Titan X
6.2 49.5 51.0 - 73.6 AGLNet[63] No 512$\times$1024 GTX 1080Ti 1.12 13.88 71.3
52.0 HMSeg[77] No 768$\times$1536 GTX 1080Ti 2.3 - 74.3 - 83.2 TinyHMSeg[77]
No 768$\times$1536 GTX 1080Ti 0.7 - 71.4 - 172.4 BiSeNetV1[78] ResNet-18
768$\times$1536 GTX 1080Ti 49.0 55.3 65.5 74.8 74.7 BiSeNetV2[45] No
512$\times$1024 GTX 1080Ti - - 156 73.4 72.6 BiSeNetV2-L[45] No
512$\times$1024 GTX 1080Ti - 118.5 47.3 75.8 75.3 STDC1-Seg50[79] STDC1
512$\times$1024 GTX 1080Ti 8.4 - 250.4 72.2 71.9 STDC2-Seg50[79] STDC2
512$\times$1024 GTX 1080Ti 12.5 - 188.6 74.2 73.4 STDC1-Seg75[79] STDC1
768$\times$1536 GTX 1080Ti 8.4 - 126.7 74.5 75.3 STDC2-Seg75[79] STDC2
768$\times$1536 GTX 1080Ti 12.5 - 97.0 77.0 76.8 HyperSeg-S[73]
EfficientNet-B1 768$\times$1536 GTX 1080Ti 10.2 17.0 16.1 78.2 78.1
HyperSeg-M[73] EfficientNet-B1 512$\times$1024 GTX 1080Ti 10.1 7.5 36.9 76.2
75.8 P2AT-S ResNet-18 1024$\times$1024 RTX 3090 12.6 80.2 80.1 78.0 77.8
P2AT-M ResNet-34 1024$\times$1024 RTX 3090 22.7 118 61.8 79.8 78.7 P2AT-L
ResNet-50 1024$\times$1024 RTX 3090 73.5 166 40.6 78.8 78.0

Figure 9: Visual results of our method P2AT-M on Cityscapes dataset. The first
row is the original image, the second row is the ground truth, and the last
row represents the model performance.

###  4.6 Results on Camvid Dataset

In this section, we analyze and discuss the performance of the proposed
methods, namely P2AT-S, P2AT-M, and P2AT-L, when compared to other state-of-
the-art (SOTA) methods on the Camvid test dataset. The results are presented
in Table 5. First, let’s consider the performance of the SOTA methods on the
Camvid dataset. Deeplab [7] and PSPNet [10] achieved an accuracy of 61.6% mIoU
and 69.1% mIoU, respectively. These methods utilized different approaches,
with Deeplab employing dilated convolutions and PSPNet pyramid pooling
modules. DFANet-A and DFANet-B [43], introduced in 2019, adopted lightweight
architectures with 7.8 and 4.8 million parameters, respectively. DFANet-A
achieved 64.7% mIoU with an inference speed of 120 FPS, while DFANet-B
achieved 59.3% mIoU with a faster speed of 160 FPS. These methods demonstrated
a higher inference speed but their accuracy is very low, and they have been
further improved upon by subsequent approaches. GAS [80] introduced a graph-
guided architecture and achieved 72.8% mIoU with a processing speed of 153.1
FPS. LBN-AA [64], on the other hand, utilized an attention aggregation module
and obtained 68.0% mIoU at a speed of 39.3 FPS. BiSeNetV2-L, an extended
version of BiSeNetV2, further improved the mIoU to 78.5% by incorporating
larger receptive fields. However, the computational speed for BiSeNetV2-L is
not provided. STDC1-Seg and STDC2-Seg [79] utilized the STDC1 and STDC2
architectures, respectively. STDC1-Seg achieved 73% mIoU and 73.9%. HyperSeg-S
and HyperSeg-L [73] employed EfficientNet-B1 as the backbone network and
achieved mIoU scores of 78.4% and 79.1%, respectively. These methods utilized
a resolution of 720$\times$960 and demonstrated better performance while
maintaining reasonable computational efficiency. HyperSeg-S operated at 38.0
FPS, while HyperSeg-L operated at 16.6 FPS. These methods demonstrated
competitive performance but with varying levels of computational efficiency.
Our proposed method P2AT-S, P2AT-M, and P2AT-L utilized ResNet-18, ResNet-34,
and ResNet-50 backbones with resolutions of 720$\times$960\. We pretrained
P2AT on the Cityscapes dataset. P2AT-S* utilized a ResNet-18 backbone and
surpasses HyperSeg and BiSeNetV2-L in terms of accuracy with an impressive
test accuracy of 80.5% at a speed of 120 FPS. P2AT-M* with ResNet-34 achieved
of 81.0% at a speed of 103 FPS. P2AT-L*, utilizing the heavy ResNet-50
backbone, achieved an accuracy of 81.1% mIoU at a speed of 89 FPS. The
pretrained versions of P2AT-S, P2AT-M, and P2AT-L outperformed most of the
other SOTA methods in terms of mIoU, highlighting their effectiveness and
potential for various real-world applications. These findings validate the
effectiveness of the proposed methods in semantic segmentation tasks and their
potential for practical applications in various domains.

Table 5: THE COMPARISON OF P2AT VARIOUS VERSIONS ON CAMVID DATASET.∗ INDICATES
THE MODELS PRE-TRAINED ON CITYSCAPES DATASET.

Method backbone Resolution Params Speed (FPS) mIoU% Deeplab[7] ResNet-101 2017
262.1 4.9 61.6 PSPNet [10] ResNet-101 2017 250.8 5.4 69.1 DFANet-A [43]
XceptionA 2019 7.8 120 64.7 DFANet-B [43] XceptionB 2019 4.8 160 59.3 GAS [80]
no 720$\times$960 - 153.1 72.8 LBN-AA [64] 360$\times$480 6.2 39.3 68.0
BiSeNetV1 [78] Xception39 720$\times$960 5.8 175 65.7 BiSeNetV2 [78] ResNet-18
720$\times$960 49.0 116.3 68.7 BiSeNetV2[45] no 720$\times$960 - 124.5 76.7
BiSeNetV2-L*[45] no 720$\times$960 32.7 - 78.5 STDC1-Seg [79] STDC1
720$\times$960 8.4 197.6 73.0 STDC2-Seg [79] STDC2 720$\times$960 12.5 152.2
73.9 HyperSeg-S[73] EfficientNet-B1 720$\times$960 9.9 38.0 78.4 HyperSeg-L
[73] EfficientNet-B1 720$\times$960 10.2 16.6 79.1 P2AT-S* ResNet-18
720$\times$960 12.6 113.6 80.5 P2AT-M* ResNet-34 720$\times$960 22.7 86.6 81.0
P2AT-L* ResNet-50 720$\times$960 37.5 56.1 81.1

Table 6: INDIVIDUAL CATEGORY RESULTS ON CAMVID TEST SET IN TERMS OF MIOU FOR
11 CLASSES. ”-” INDICATES THE CORRESPONDING RESULT IS NOT REPORTED BY THE
METHODS.

Method Building Tree Sky Car Sign Road Ped Fence Pole Sidewalk Bicyclist mIoU
SegNet[81] 88.8 87.3 92.4 82.1 20.5 97.2 57.1 49.3 27.5 84.4 30.7 65.2
BiSeNet1[78] 82.2 74.4 91.9 80.8 42.8 93.3 53.8 49.7 25.4 77.3 50.0 65.6
BiSeNet2[78] 83.0 75.8 92.0 83.7 46.5 94.6 58.8 53.6 31.9 81.4 54.0 68.7
AGLNet[63] 82.6 76.1 91.8 87.0 45.3 95.4 61.5 39.5 39.0 83.1 62.7 69.4 LBN-
AA[64] 83.2 70.5 92.5 81.7 51.6 93.0 55.6 53.2 36.3 82.1 47.9 68.0
BiSeNetV2/BiSeNetV2L[45] - - - - - - - - - - - 72.4/73.2 P2AT-S 91.6 81.4 93.3
95.0 55.8 96.7 77.1 75.7 50.4 90.4 74.7 80.5 P2AT-M 91.0 81.4 93.4 95.3 55.9
97.6 77.5 73.8 49.3 92.6 77.6 81.0 P2AT-L 91.0 81.2 93.2 95.3 57.3 97.3 77.6
74.0 48.9 92.0 78.9 81.1

Figure 10: Visual results of our method P2AT on Camvid test set. The first row
is the image, the second row is the prediction, and the last row is the ground
truth.

Figure 11: The feature maps visualization of our proposed architecture on the
Camvid validation dataset. From left to right: (a) original input images; (b)
The ground truth; (c) The Scale-aware Semantic Aggregation ; (d) the
bidirectional fusion (stage4); (e) the global context enhancer (stage 4); (f)
the feature refinement before the final segmentation.

##  5 Conclusion

In this work, we have presented P2AT, a real-time semantic segmentation
network. The scale-aware context aggregator of P2At is capable of extracting
rich contextual information, which is established on our proposed pyramid
pooling axial attention. Additionally, we have designed the Bidirectional
Fusion (BiF) module that efficiently integrates semantic information at
different levels and a global context enhancer module to address limitations
in concatenating different semantic levels. Notably, P2AT-S surpasses existing
models on the Camvid dataset, achieving an accuracy of 81.0% mIoU. Moreover,
our experiments on Cityscapes and PASCAL VOC 2012 demonstrate the efficiency
and effectiveness of the proposed architecture, with P2AT-S and P2AT-L
achieving an accuracy of 77.8% and 78.0% on Cityscapes, respectively. In
summary, our contributions encompass the novel P2AT framework, which addresses
the challenge of achieving accurate scene understanding in real-time tasks,
such as autonomous driving, while considering computational efficiency.

## References

  * Min et al. [2022] W. Min, R. Liu, D. He, Q. Han, Q. Wei, Q. Wang,  Traffic sign recognition based on semantic scene understanding and structural traffic sign location,  IEEE Transactions on Intelligent Transportation Systems 23 (2022) 15794–15807.
  * Tian et al. [2019] Y. Tian, J. Gelernter, X. Wang, J. Li, Y. Yu,  Traffic sign detection using a multi-scale recurrent attention network,  IEEE transactions on intelligent transportation systems 20 (2019) 4466–4475.
  * Huang et al. [2023] S. Huang, Z. Shen, Z. Huang, Z.-h. Ding, J. Dai, J. Han, N. Wang, S. Liu,  Anchor3dlane: Learning to regress 3d anchors for monocular 3d lane detection,  in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 17451–17460.
  * Wang et al. [2023] R. Wang, J. Qin, K. Li, Y. Li, D. Cao, J. Xu,  Bev-lanedet: An efficient 3d lane detection based on virtual camera via key-points,  in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 1002–1011.
  * Carion et al. [2020] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, S. Zagoruyko,  End-to-end object detection with transformers,  in: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, Springer, 2020, pp. 213–229.
  * Han et al. [2019] B. Han, Y. Wang, Z. Yang, X. Gao,  Small-scale pedestrian detection based on deep neural network,  IEEE transactions on intelligent transportation systems 21 (2019) 3046–3055.
  * Chen et al. [2017] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. L. Yuille,  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,  IEEE transactions on pattern analysis and machine intelligence 40 (2017) 834–848.
  * Paszke et al. [2016] A. Paszke, A. Chaurasia, S. Kim, E. Culurciello,  Enet: A deep neural network architecture for real-time semantic segmentation,  arXiv preprint arXiv:1606.02147 (2016).
  * Cordts et al. [2016] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, B. Schiele,  The cityscapes dataset for semantic urban scene understanding,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 3213–3223.
  * Zhao et al. [2017] H. Zhao, J. Shi, X. Qi, X. Wang, J. Jia,  Pyramid scene parsing network,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2881–2890.
  * Bilinski and Prisacariu [2018] P. Bilinski, V. Prisacariu,  Dense decoder shortcut connections for single-pass semantic segmentation,  in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 6596–6605.
  * Zhao et al. [2018] H. Zhao, Y. Zhang, S. Liu, J. Shi, C. C. Loy, D. Lin, J. Jia,  Psanet: Point-wise spatial attention network for scene parsing,  in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 267–283.
  * Long et al. [2015] J. Long, E. Shelhamer, T. Darrell,  Fully convolutional networks for semantic segmentation,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3431–3440.
  * Dosovitskiy et al. [2020] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.,  An image is worth 16x16 words: Transformers for image recognition at scale,  arXiv preprint arXiv:2010.11929 (2020).
  * Touvron et al. [2021] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, H. Jégou,  Training data-efficient image transformers & distillation through attention,  in: International conference on machine learning, PMLR, 2021, pp. 10347–10357.
  * Liu et al. [2021] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, B. Guo,  Swin transformer: Hierarchical vision transformer using shifted windows,  in: Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10012–10022.
  * Liu et al. [2022] Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang, L. Dong, et al.,  Swin transformer v2: Scaling up capacity and resolution,  in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 12009–12019.
  * Xie et al. [2021] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, P. Luo,  Segformer: Simple and efficient design for semantic segmentation with transformers,  Advances in Neural Information Processing Systems 34 (2021) 12077–12090.
  * Zheng et al. [2021] S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. Torr, et al.,  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,  in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 6881–6890.
  * Krizhevsky et al. [2017] A. Krizhevsky, I. Sutskever, G. E. Hinton,  Imagenet classification with deep convolutional neural networks,  Communications of the ACM 60 (2017) 84–90.
  * He et al. [2016] K. He, X. Zhang, S. Ren, J. Sun,  Deep residual learning for image recognition,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * Simonyan and Zisserman [2014] K. Simonyan, A. Zisserman,  Very deep convolutional networks for large-scale image recognition,  arXiv preprint arXiv:1409.1556 (2014).
  * Chen et al. [2014] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. L. Yuille,  Semantic image segmentation with deep convolutional nets and fully connected crfs,  arXiv preprint arXiv:1412.7062 (2014).
  * Yu and Koltun [2015] F. Yu, V. Koltun,  Multi-scale context aggregation by dilated convolutions,  arXiv preprint arXiv:1511.07122 (2015).
  * Yang et al. [2018] M. Yang, K. Yu, C. Zhang, Z. Li, K. Yang,  Denseaspp for semantic segmentation in street scenes,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3684–3692.
  * Ding et al. [2019] H. Ding, X. Jiang, A. Q. Liu, N. M. Thalmann, G. Wang,  Boundary-aware feature propagation for scene segmentation,  in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 6819–6829.
  * Bertasius et al. [2016] G. Bertasius, J. Shi, L. Torresani,  Semantic segmentation with boundary neural fields,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 3602–3610.
  * Li et al. [2020] X. Li, X. Li, L. Zhang, G. Cheng, J. Shi, Z. Lin, S. Tan, Y. Tong,  Improving semantic segmentation via decoupled body and edge supervision,  in: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVII 16, Springer, 2020, pp. 435–452.
  * Yuan et al. [2018] Y. Yuan, L. Huang, J. Guo, C. Zhang, X. Chen, J. Wang,  Ocnet: Object context network for scene parsing,  arXiv preprint arXiv:1809.00916 (2018).
  * Yu et al. [2020] C. Yu, J. Wang, C. Gao, G. Yu, C. Shen, N. Sang,  Context prior for scene segmentation,  in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 12416–12425.
  * Yuan et al. [2020] Y. Yuan, X. Chen, J. Wang,  Object-contextual representations for semantic segmentation,  in: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16, Springer, 2020, pp. 173–190.
  * Lin et al. [2017] G. Lin, A. Milan, C. Shen, I. Reid,  Refinenet: Multi-path refinement networks for high-resolution semantic segmentation,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1925–1934.
  * Wu et al. [2021] T. Wu, S. Tang, R. Zhang, J. Cao, Y. Zhang,  Cgnet: A light-weight context guided network for semantic segmentation,  IEEE Transactions on Image Processing 30 (2021) 1169–1179.
  * Elhassan et al. [2021] M. A. Elhassan, C. Huang, C. Yang, T. L. Munea,  Dsanet: Dilated spatial attention for real-time semantic segmentation in urban street scenes,  Expert Systems with Applications 183 (2021) 115090.
  * Fu et al. [2019] J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, H. Lu,  Dual attention network for scene segmentation,  in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 3146–3154.
  * Huang et al. [2019] Z. Huang, X. Wang, L. Huang, C. Huang, Y. Wei, W. Liu,  Ccnet: Criss-cross attention for semantic segmentation,  in: Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 603–612.
  * Li et al. [2018] H. Li, P. Xiong, J. An, L. Wang,  Pyramid attention network for semantic segmentation,  arXiv preprint arXiv:1805.10180 (2018).
  * Shaw et al. [2019] A. Shaw, D. Hunter, F. Landola, S. Sidhu,  Squeezenas: Fast neural architecture search for faster semantic segmentation,  in: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2019, pp. 0–0.
  * Chen et al. [2019] W. Chen, X. Gong, X. Liu, Q. Zhang, Y. Li, Z. Wang,  Fasterseg: Searching for faster real-time semantic segmentation,  arXiv preprint arXiv:1912.10917 (2019).
  * Liu et al. [2019] C. Liu, L.-C. Chen, F. Schroff, H. Adam, W. Hua, A. L. Yuille, L. Fei-Fei,  Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation,  in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 82–92.
  * Yan et al. [2022] H. Yan, C. Zhang, M. Wu,  Lawin transformer: Improving semantic segmentation transformer with multi-scale representations via large window attention,  arXiv preprint arXiv:2201.01615 (2022).
  * Zhao et al. [2018] H. Zhao, X. Qi, X. Shen, J. Shi, J. Jia,  Icnet for real-time semantic segmentation on high-resolution images,  in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 405–420.
  * Li et al. [2019] H. Li, P. Xiong, H. Fan, J. Sun,  Dfanet: Deep feature aggregation for real-time semantic segmentation,  in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 9522–9531.
  * Orsic et al. [2019] M. Orsic, I. Kreso, P. Bevandic, S. Segvic,  In defense of pre-trained imagenet architectures for real-time semantic segmentation of road-driving images,  in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 12607–12616.
  * Yu et al. [2021] C. Yu, C. Gao, J. Wang, G. Yu, C. Shen, N. Sang,  Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation,  International Journal of Computer Vision (2021) 1–18.
  * Poudel et al. [2018] R. P. Poudel, U. Bonde, S. Liwicki, C. Zach,  Contextnet: Exploring context and detail for semantic segmentation in real-time,  arXiv preprint arXiv:1805.04554 (2018).
  * Mazzini [2018] D. Mazzini,  Guided upsampling network for real-time semantic segmentation,  arXiv preprint arXiv:1807.07466 (2018).
  * Li et al. [2020] X. Li, A. You, Z. Zhu, H. Zhao, M. Yang, K. Yang, S. Tan, Y. Tong,  Semantic flow for fast and accurate scene parsing,  in: European Conference on Computer Vision, Springer, 2020, pp. 775–793.
  * Mehta et al. [2018] S. Mehta, M. Rastegari, A. Caspi, L. Shapiro, H. Hajishirzi,  Espnet: Efficient spatial pyramid of dilated convolutions for semantic segmentation,  in: Proceedings of the european conference on computer vision (ECCV), 2018, pp. 552–568.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei,  Imagenet: A large-scale hierarchical image database,  in: 2009 IEEE conference on computer vision and pattern recognition, Ieee, 2009, pp. 248–255.
  * Arnab et al. [2021] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić, C. Schmid,  Vivit: A video vision transformer,  in: Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 6836–6846.
  * Bertasius et al. [2021] G. Bertasius, H. Wang, L. Torresani,  Is space-time attention all you need for video understanding?,  in: ICML, volume 2, 2021, p. 4.
  * Lin et al. [2017] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, S. Belongie,  Feature pyramid networks for object detection,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2117–2125.
  * Liu et al. [2022] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, S. Xie,  A convnet for the 2020s,  in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 11976–11986.
  * Howard et al. [2019] A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, et al.,  Searching for mobilenetv3,  in: Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1314–1324.
  * Wan et al. [2023] Q. Wan, Z. Huang, J. Lu, G. Yu, L. Zhang,  Seaformer: Squeeze-enhanced axial transformer for mobile semantic segmentation,  arXiv preprint arXiv:2301.13156 (2023).
  * Ho et al. [2019] J. Ho, N. Kalchbrenner, D. Weissenborn, T. Salimans,  Axial attention in multidimensional transformers,  arXiv preprint arXiv:1912.12180 (2019).
  * Wang et al. [2020] H. Wang, Y. Zhu, B. Green, H. Adam, A. Yuille, L.-C. Chen,  Axial-deeplab: Stand-alone axial-attention for panoptic segmentation,  in: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV, Springer, 2020, pp. 108–126.
  * Lou et al. [2022] A. Lou, S. Guan, H. Ko, M. H. Loew,  Caranet: context axial reverse attention network for segmentation of small medical objects,  in: Medical Imaging 2022: Image Processing, volume 12032, SPIE, 2022, pp. 81–92.
  * Huang et al. [2022] Y. Huang, D. Kang, W. Jia, L. Liu, X. He,  Channelized axial attention–considering channel relation within spatial attention for semantic segmentation,  in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 2022, pp. 1016–1025.
  * Zheng et al. [2015] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du, C. Huang, P. H. Torr,  Conditional random fields as recurrent neural networks,  in: Proceedings of the IEEE international conference on computer vision, 2015, pp. 1529–1537.
  * Yu and Koltun [2016] F. Yu, V. Koltun,  Multi-scale context aggregation by dilated convolutions (2015),  arXiv preprint arXiv:1511.07122 (2016).
  * Zhou et al. [2020] Q. Zhou, Y. Wang, Y. Fan, X. Wu, S. Zhang, B. Kang, L. J. Latecki,  Aglnet: Towards real-time semantic segmentation of self-driving images via attention-guided lightweight network,  Applied Soft Computing 96 (2020) 106682.
  * Dong et al. [2020] G. Dong, Y. Yan, C. Shen, H. Wang,  Real-time high-performance semantic image segmentation of urban street scenes,  IEEE Transactions on Intelligent Transportation Systems 22 (2020) 3258–3274.
  * Brostow et al. [2009] G. J. Brostow, J. Fauqueur, R. Cipolla,  Semantic object classes in video: A high-definition ground truth database,  Pattern Recognition Letters 30 (2009) 88–97.
  * Everingham et al. [2010] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, A. Zisserman,  The pascal visual object classes (voc) challenge,  International journal of computer vision 88 (2010) 303–338.
  * Paszke et al. [2017] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, A. Lerer,  Automatic differentiation in pytorch,  in: NIPS 2017 Workshop on Autodiff, 2017\. URL: https://openreview.net/forum?id=BJJsrmfCZ.
  * Hariharan et al. [2011] B. Hariharan, P. Arbeláez, L. Bourdev, S. Maji, J. Malik,  Semantic contours from inverse detectors,  in: 2011 international conference on computer vision, IEEE, 2011, pp. 991–998.
  * Chen et al. [2017] L.-C. Chen, G. Papandreou, F. Schroff, H. Adam,  Rethinking atrous convolution for semantic image segmentation,  arXiv preprint arXiv:1706.05587 (2017).
  * Chen et al. [2018] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, H. Adam,  Encoder-decoder with atrous separable convolution for semantic image segmentation,  in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 801–818.
  * Yu et al. [2018] C. Yu, J. Wang, C. Peng, C. Gao, G. Yu, N. Sang,  Learning a discriminative feature network for semantic segmentation,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 1857–1866.
  * Fu et al. [2019] J. Fu, J. Liu, Y. Wang, J. Zhou, C. Wang, H. Lu,  Stacked deconvolutional network for semantic segmentation,  IEEE Transactions on Image Processing (2019).
  * Yuval Nirkin [2021] T. H. Yuval Nirkin, Lior Wolf,  Hyperseg: Patch-wise hypernetwork for real-time semantic segmentation,  in: Conf. on Computer Vision and Pattern Recognition (CVPR), 2021. URL: https://talhassner.github.io/home/publication/2021_CVPR_2.
  * Peng et al. [2022] J. Peng, Y. Liu, S. Tang, Y. Hao, L. Chu, G. Chen, Z. Wu, Z. Chen, Z. Yu, Y. Du, et al.,  Pp-liteseg: A superior real-time semantic segmentation model,  arXiv preprint arXiv:2204.02681 (2022).
  * Hu et al. [2020a] P. Hu, F. Caba, O. Wang, Z. Lin, S. Sclaroff, F. Perazzi,  Temporally distributed networks for fast video semantic segmentation,  in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020a, pp. 8818–8827.
  * Hu et al. [2020b] P. Hu, F. Perazzi, F. C. Heilbron, O. Wang, Z. Lin, K. Saenko, S. Sclaroff,  Real-time semantic segmentation with fast attention,  IEEE Robotics and Automation Letters 6 (2020b) 263–270.
  * Li et al. [2020] P. Li, X. Dong, X. Yu, Y. Yang,  When humans meet machines: Towards efficient segmentation networks.,  in: BMVC, 2020.
  * Yu et al. [2018] C. Yu, J. Wang, C. Peng, C. Gao, G. Yu, N. Sang,  Bisenet: Bilateral segmentation network for real-time semantic segmentation,  in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 325–341.
  * Fan et al. [2021] M. Fan, S. Lai, J. Huang, X. Wei, Z. Chai, J. Luo, X. Wei,  Rethinking bisenet for real-time semantic segmentation,  in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 9716–9725.
  * Lin et al. [2020] P. Lin, P. Sun, G. Cheng, S. Xie, X. Li, J. Shi,  Graph-guided architecture search for real-time semantic segmentation,  in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 4203–4212.
  * Badrinarayanan et al. [2017] V. Badrinarayanan, A. Kendall, R. Cipolla,  Segnet: A deep convolutional encoder-decoder architecture for image segmentation,  IEEE transactions on pattern analysis and machine intelligence 39 (2017) 2481–2495.