© Higher Education Press and Springer-Verlag Berlin Heidelberg 2012 Received
month dd, yyyy; accepted month dd, yyyy

# Graph-Segmenter: Graph Transformer with Boundary-aware Attention for
Semantic Segmentation

Zizhang Wu1 Yuanzhu Gan1 Tianhao Xu1,2 Fan Wang1 $1$ Computer Vision
Perception Department of ZongMu Technology, Shanghai, 201203, China.
$2$ Faculty of Electronics, Information Technology, Physics, Technical
University of Braunschweig,
Braunschweig, 38106, Germany.

###### Abstract

The transformer-based semantic segmentation approaches, which divide the image
into different regions by sliding windows and model the relation inside each
window, have achieved outstanding success. However, since the relation
modeling between windows was not the primary emphasis of previous work, it was
not fully utilized. To address this issue, we propose a Graph-Segmenter,
including a Graph Transformer and a Boundary-aware Attention module, which is
an effective network for simultaneously modeling the more profound relation
between windows in a global view and various pixels inside each window as a
local one, and for substantial low-cost boundary adjustment. Specifically, we
treat every window and pixel inside the window as nodes to construct graphs
for both views and devise the Graph Transformer. The introduced boundary-aware
attention module optimizes the edge information of the target objects by
modeling the relationship between the pixel on the object’s edge. Extensive
experiments on three widely used semantic segmentation datasets (Cityscapes,
ADE-20k and PASCAL Context) demonstrate that our proposed network, a Graph
Transformer with Boundary-aware Attention, can achieve state-of-the-art
segmentation performance.

###### doi:

††articletype: REVIEW ARTICLE

Graph Transformer, Graph Relation Network, Boundary-aware, Attention, Semantic
Segmentation

##  1 Introduction

Semantic segmentation [1, 2] is a primary task in the field of computer
vision, with the goal of labeling each picture pixel with a category that
corresponds to it. As a result, intense research attention has lately been
focused on it since it has the potential to improve a wide range of downstream
applications [3, 4, 5, 6], including geographic information systems,
autonomous driving, medical picture diagnostics, and robotics. Modern semantic
segmentation models almost follow the same paradigm[7, 8, 9, 10]: they consist
of a basic backbone for feature extraction and a head for pixel-level
classification tasks in the current deep learning era. Improving the
performance of the backbone network and the head of the segmentation model are
two of the most controversial subjects in current semantic segmentation work.

Natural language processing (NLP) has been dominated by transformers for a
long time [11, 12], leading to a current spike of interest in exploring the
prospect of using transformers in vision tasks, including significant
advancements in semantic segmentation. Using the vision transformer [13],
images are divided into a number of non-overlapping windows/patches, and some
subsequent research investigated methods to enhance connections between
windows/patches. It has the benefit of improving the modeling power of the
system [14, 15, 16].

Figure 1: An illustration of the proposed Graph-Segmenter with boundary-aware
attention for semantic segmentation. The top shows the segmentation results of
the previous transformer-based semantic segmentation methods (e.g., Swin
[14]). The bottom shows the actual segmentation results of our proposed Graph-
Segmenter, which achieves promising boundary segmentation via the hierarchical
level graph reasoning and efficient boundary adjustment requiring no
additional annotation.

Swin employs [14] a shifted window approach, however, the direction in which
it interacts with other windows is fixed and unavoidably unresponsive.
Segformer utilizes [15] the overlapping patch merging approach, however it
focuses primarily on the local continuity between patches rather than the
overall continuity between patches. The modeling of long-distance interactions
between windows/patches is not investigated in depth by these methodologies,
which is unfortunate. In addition to current work on the optimization of the
backbone, a number of research have contributed to the design of the head in
order to facilitate further optimization [17, 18, 19]. However, following the
majority of them is computationally costly due to the fact that boundary
prediction-based approaches need extra object boundary labeling for
segmentation boundary optimization.

We propose a novel relation modeling method acting on sliding windows, using
graph convolutions to establish relationships between windows and pixels
inside each window, which enhanced the backbone to address the issues above.
In particular, we regard each window or the pixels inside as nodes for the
graph network and use the visual similarity between nodes to establish the
edges between nodes. After that, we use the graph network further to update
the nodes and edges of the graph. So that different nodes can adaptively
establish connections and update information in network transmission to
realize the nonlinear relationship modeling between different windows and
different pixels inside. In brief, the network’s overall feature learning and
characterization capabilities are further improved by enhancing the long-
distance nonlinear modeling capabilities between different windows and
different pixels inside, as shown in Figure 1, which leads to an evident rise
in performance.

Furthermore, we introduce an efficient boundary-aware attention-enhanced
segmentation head that optimizes the boundary of objects in the semantic
segmentation task, allowing us to reduce the labeling cost even further while
simultaneously improving the accuracy of the semantic segmentation in the
boundary of the objects under consideration. To put it another way, we develop
a lightweight local information-aware attention module that allows for
improved boundary segmentation. By determining the weights of the pixels
around an object’s border and applying various attention coefficients to
distinct pixels via local perception, it is possible to reinforce the
important pixels that are critical in categorization while weakening the
interfering pixels. The attention module used in this study has just a few
common CNN layers, which makes it efficient for segmentation boundary
adjustments when considering the size, floating point operations, and latency
time of the segmentation data.

We investigate the atypical interaction between windows that are arranged
hierarchically on a vision transformer. Additionally, we improve the boundary
of target instances using an efficient and lightweight boundary optimization
approach that does not need any extra annotation information and can
adaptively alter the segmentation boundary for a variety of objects. We have
demonstrated in this paper that our Graph-Segmenter can achieve state-of-the-
art performance on semantic segmentation tasks using extensive experiments on
three standard semantic segmentation datasets (Cityscapes [20], ADE-20k [21],
and PASCAL Context [22]), as demonstrated by extensive experiments on three
standard semantic segmentation datasets.

##  2 Related works

###  2.1 CNN-based Semantic Segmentation

Convolutional neural networks (CNNs) based methods serve as the standard
approaches throughout the semantic segmentation [23, 24, 25, 26, 27, 28] task
due to apparent advantages compared with traditional methods. FCN [23] started
the era of end-to-end semantic segmentation, introducing dense prediction
without any fully connected layer. Subsequently, many FCN-based methods [25,
26, 29] have been proposed to promote image semantic segmentation. DNN-based
methods usually need to expand the receptive field through the superposition
of convolutional layers. Besides that, the receptive field issue was explored
by several approaches, such as Pyramid Pooling Module (PPM) [25] in PSPNet and
Atrous Spatial Pyramid Pooling (ASPP) in DeepLabv3 [26], which expand the
receptive field and capture multiple-range information to enhance the
representation capabilities. Recent CNN-based methods place a premium on
effectively aggregating the hierarchical features extracted from a pre-trained
backbone encoder using specifically developed modules: DANet [27] applies a
different form of the non-local network; SFNet [30] addresses the misalignment
problem through semantic flow by using the Flow Alignment Module (FAM); CCNet
[31] present a Criss-Cross network which gathers contextual information
adaptively throughout the criss-cross path; GFFNet [32]; APCNet [33] proposes
Adaptive Pyramid Context Network for semantic segmentation, which creates
multi-scale contextual representations adaptively using several well-designed
Adaptive Context Modules (ACMs). [28] divides the feature map into different
regions to extract regional features separately, and the bidirectional edges
of the directed graph are used to represent the affinities between these
regions, in order to help model region dependencies and alleviate unrealistic
results. [34] learns boundaries as an additional semantic category, so the
network can be aware of the layout of the boundaries. It also proposes
unidirectional acyclic graphs (UAGs) to simulate the function of undirected
cyclic graphs (UCGs) to structure the image. Boundaryaware feature propagation
(BFP) module can acquire and propagate boundary local features of regions to
create strong and weak connections between regions in the UAG.

###  2.2 Transformer and Self-Attention

The attention mechanism for image classification was first seen in [35], and
then [36] employed similar attention on translation and alignment
simultaneously in the machine translation task, which was the first
application of the attention mechanism to the NLP field. Inspired by the
dominant performance in the NLP field later, the application of self-attention
and transformer architectures in computer vision are widely explored in recent
years. Self-attention layers as a replacement for spatial convolutional layers
achieved rises in terms of robustness and performance-cost trade-off [37].
Moreover, Vision Transformer (ViT) [13] shown transformer structure with
simple non-overlap patches could surpass state-of-the-art results, and the
impressive result led to a trend of vision transformers. The issue that ViT is
unsuitable as a general backbone for semantic segmentation was further
addressed by several approaches: [38] proposed DEtection TRansformer (DETR)
for direct set prediction based on transformers and bipartite matching loss;
based on DETR, [39] suggested Deformable DETR with attention modules that
focus only on a limited number of critical sampling locations around a
reference; after that, [40] investigated the video instance segmentation (VIS)
task using vision transformers, termed VisTR, which view the VIS task as a
direct end-to-end parallel sequence decoding/prediction problem. [41] expands
the idea of DETR to multi-camera 3D object detection to make great progress.

In addition, the transformer-based semantic segmentation methods [42, 43, 14,
15] have drawn much attention. [42] relies on a ViT [13] backbone and
introduces a mask decoder inspired by DETR [38]. [43] reformulates the image
semantic segmentation problem from a transformer-based learning perspective.
[14] uses a shifted windowing scheme to break the limited self-attention
computation into non-overlapping local windows. [15] proposes a powerful
semantic segmentation framework with lightweight multilayer perceptron
decoders. They adopt the Overlapped Patch Merging module to preserve local
continuity. However, these methods didn’t look at the relationship between
windows very well, which led this study to use hierarchical-level graph
reasoning and efficient boundary adjustment. Our method puts more focus on the
graph modeling between windows within the transformer blocks, which is the
first work for the semantic segmentation task.

###  2.3 Graph Model

Graph-based convolutional networks (GCNs) [44, 45, 46, 47] can exploit the
interactions of non-Euclidean data, which was initially proposed in [48]. As
graphs can be irregular, GCNs led to an increasing interest in recent years,
and many excellent works further explored GCNs from various perspectives such
as graph attention networks [49] for guided learning, and dynamic graph [50]
to learn features more adaptively. GCNs also have achieved great performance
in kinds of computer vision tasks, such as human pose estimation [51], image
captioning [52], action recognition [53], and so on. [51] adopts graph atrous
convolution and transformer layers to extract multi-scale context and long-
range information for human pose estimation. [52] proposes an image captioning
model based on a dual-GCN and transformer combination. [54] proposes an
adaptive graph convolutional network with attention graph clustering for image
co-saliency detection. Moreover, GCNs can propagate information globally and
model contextual information efficiently for semantic segmentation [55, 56,
44, 57], point cloud segmentation [58, 59] and instance segmentation [60].
[55] performs graph reasoning directly in the original feature space to model
long-range context. [56] introduces the class-wise dynamic graph convolution
module to capture long-range contextual information adaptively. [44] captures
the global context along the spatial and channel dimension by the graph-
convolutional module, respectively. [57] combines adjacency graphs and KSC-
graphs by affinity nodes of multi-scale superpixels for better segmentation.
[58] proposes a hierarchical attentive pooling graph network for point cloud
segmentation to enhance the local modeling. [61, 62] provide more detailed
surveys of GCNs.

###  2.4 Discussion

Different from Swin [14], first of all, our proposed graph transformer focuses
on the nonlinear relationship modeling of sliding windows, including the
modeling of high-order (greater than or equal to 2-order) relationship between
sliding windows and among different feature points inside each sliding window.
Secondly, our graph transformer module is a lightweight pluggable sliding
window modeling module, which can be embedded into some sliding window-based
transfomer models such as Swin [14] to improve the feature modeling and
expression ability of the original model. Finally, the graph transformer
modules plus the boundary-aware attention modules together form our Graph-
Segmenter network, which achieves state-of-the-art performance compared to
other recent works on the existing segmentation benchmarks.

Figure 2: An overview of the proposed Graph-Segmenter with efficient boundary
adjustment for semantic segmentation, which includes Global Relation Modeling,
Local Relation Modeling, and Boundary-aware Attention. “GR” denotes the global
window-aware relation module and “LR” denotes the local window-aware relation
module. “GR” or “LR” consists of $1\times 1$ convolution, softmax,
$\bm{\mathrm{W}}$ and $1\times 1$ convolution. $\bm{\mathrm{W}}$ is the
learnable weighting matrix, corresponding to $\bm{\mathrm{W}}^{(l)}$ in
Equation 6, and also equivalent to $\mathcal{C}^{gr}$ in “GR” or
$\mathcal{C}^{lr}$ in “LR”.

##  3 Methods

This section will elaborate on the design of our devised Graph-Segmenter,
which includes the Graph Transformer module and Boundary-aware attention
module.

###  3.1 The Overview

As shown in Figure 2, our proposed coarse-to-fine-grained boundary modeling
network, Graph-Segmenter, includes a coarse-grained graph transformer module
in the backbone and the fine-grained boundary-aware attention module in the
head of the semantic segmentation framework. Assume that the input image
passes through some CNN layers and then obtains the positional embedding
enhanced feature maps $\bm{\mathrm{X}}\in\mathbb{R}^{C\times H\times W}$,
where $C$ denotes the number of channels, $H$ and $W$ denote height and width
in the spatial dimension of the image, respectively. In order to make better
use of the Transformer mechanism, inspired by [14, 16], we first divide
feature maps $\bm{\mathrm{X}}$ into $M\times N$ window patches and define each
window as
$\\{\bm{\mathrm{x}}_{m,n}\in\mathbb{R}^{C\times\frac{H}{M}\times\frac{W}{N}}|m\in\\{1,2,...,M\\},n\in\\{1,2,...,N\\}\\}$.
And then $\bm{\mathrm{X}}$ passes through Graph Transformer (GT) Network
module, generating the relation-enhanced features $\bm{\mathrm{Y}}$ after GT
module in the backbone network. After this, we use the boundary-aware
attention (BA) module to adjust the segmentation boundary on the feature maps
$\bm{\mathrm{Y}}$ and obtain the object boundary optimal feature
$\bm{\mathrm{Z}}$. At last, we use these features $\bm{\mathrm{Z}}$ to
formulate the corresponding segmentation loss function in each pixel. In the
following subsections, we will elaborate on our proposed window-aware relation
transformer network module and boundary-aware attention module. To facilitate
the following description, we first introduce the efficient graph relation
network, which plays a central role in the graph transformer block and is
capable of efficiently and effectively modeling the nonlinear relations for
abstract graph nodes.

###  3.2 Efficient Graph Relation Network

In this section, we first introduce the preliminary work on graph convolution
networks (GCN), which introduces convolutional parameters that can be
optimized compared to graph neural networks (GNN). As shown in [49], a GCN can
be defined as

$\bm{\mathrm{X}}=\bm{\mathrm{R}}\bm{\mathrm{X}}\bm{\mathrm{W}},$ (1)

where $\bm{\mathrm{X}}$, $\bm{\mathrm{R}}$ and $\bm{\mathrm{W}}$ are the input
node matrix (here we still denote the output of the above equation as
$\bm{\mathrm{X}}$), the adjacency matrix reflecting the relationship between
nodes and the learnable convolution parameter matrix respectively.

Based on above preliminary work, we introduce an efficient graph relation
network, which was inspired by [49, 63], to model the non-linear relation of
higher-order among the sliding windows and inside each window. Given the input
$\bm{\mathrm{X}}$, we define the relation function $\mathcal{R}$ as follows

$\bm{\mathrm{R}}=\mathcal{R}(\bm{\mathrm{X}}).$ (2)

As shown in Equation (2), to establish the relationship $\mathcal{R}$ between
different input data $\bm{\mathrm{x}}_{i}$ and $\bm{\mathrm{x}}_{j}$, where
$\bm{\mathrm{x}}_{i},\bm{\mathrm{x}}_{j}\in\bm{\mathrm{X}}$,
$i,j\in\\{1,2,\dots,K\\}$, and $K$ is the number of nodes in
$\bm{\mathrm{X}}$, mathematically, we can establish a simple linear
relationship using linear functions or a more complex nonlinear relationship
using nonlinear functions. In general, linear relations are poor in terms of
robustness, so here we design a quadratic multiplication operation that
establishes a nonlinear relationship between different input nodes.

$r_{i,j}=\frac{\bm{\mathrm{x}}_{i}\cdot\bm{\mathrm{x}}_{j}}{||\bm{\mathrm{x}}_{i}\cdot\bm{\mathrm{x}}_{j}||},$
(3)

where $r_{i,j}\in\bm{\mathrm{R}}$ is the learned relation between
$\bm{\mathrm{x}}_{i}$ and $\bm{\mathrm{x}}_{j}$. $||\cdot||$ represents a two-
norm operation. Because
$\bm{\mathrm{x}}_{i}\cdot\bm{\mathrm{x}}_{j}\leq||\bm{\mathrm{x}}_{i}\cdot\bm{\mathrm{x}}_{j}||$,
$|r_{i,j}|\leq 1$, where $|\cdot|$ denotes the absolute value operation, we do
not need to normalize the $r_{i,j}$ as in [64].

With the connection relation matrix $\bm{\mathrm{R}}$, we can define the node
update function $\mathcal{U}$ of the graph as follows

$\bm{\mathrm{X}}^{(l+1)}=\mathcal{U}(\bm{\mathrm{X}}^{(l)}).$ (4)

Now that we have $r_{i,j}$ and also to design an efficient graph relational
network, we want this graph network node propagation update operation to be a
sparse matrix operation. To achieve efficient computation, we use the
indicative function and the relational matrix $\bm{\mathrm{R}}$. Equation (4)
can be rewritten into a form that satisfies the sparse matrix operation as
follows:

$\bm{\mathrm{x}}_{i}^{(l+1)}=\sum\limits_{\bm{\mathrm{x}}_{j}^{(l)}\in\delta(\bm{\mathrm{x}}_{i}^{(l)})}\mathbb{I}(r_{i,j}>\theta)\cdot\bm{\mathrm{x}}_{j}^{(l)},\\\
\\\ i,j\in\\{1,2,\dots,K\\},$ (5)

where $\mathbb{I}(\cdot)$ is the indicative function which takes $r_{ij}$ if
the condition holds, or 0 otherwise. $\bm{\mathrm{x}}_{j}^{(l)}$ is the $j$th
graph node in layer $l$ in the graph, which is generated in the layer-by-layer
transfer process of graph convolutional network.
${\delta(\bm{\mathrm{x}}_{i}^{(l)})}$ denotes the set of neighboring nodes of
node $\bm{\mathrm{x}}_{i}^{(l)}$ in layer $l$. Finally, in order to accomplish
the graph convolutional network defined in Equation (1), we use the
convolutional function $\mathcal{C}$ to convolute the node information as
follows

$\bm{\mathrm{X}}^{(l+1)}=\mathcal{C}(\bm{\mathrm{X}}^{(l)})=\bm{\mathrm{X}}^{(l)}\bm{\mathrm{W}}^{(l)},$
(6)

where $\bm{\mathrm{W}}^{(l)}$ is the learnable weighting matrix.

###  3.3 Graph Transformer

Sliding windows-based methods [14, 16] model the local relation inside each
window via transformer [13], most of which does not take into account the
global modeling of a nonlinear relation between windows. In order to obtain
more robust relation-enhanced features, we propose Graph Transformer (GT) to
model the relationship between different windows and within each window. We
use the visual similarity among different windows to learn the relation of
different windows while using the visual similarity between different pixels
to model the local relation inside each sliding window. Based on the above-
devised efficient graph relation network, the detailed design of the global
relation modeling module and the local one will be further elaborated on
below.

####  3.3.1 Global Relation Modeling

We propose a Global Relation (GR) Modeling module to model the global relation
among different sliding windows and make the pixel-level classification more
accurate. Specifically, we consider each window as one node and use the
efficient graph relation network to build the relation of the graph nodes. As
a result, we can obtain the coarse relation of the objects in each image via
the relation modeling of windows.

Concretely, given the partitioned $M\times N$ sliding windows in feature maps
$\bm{\mathrm{X}}\in\mathbb{R}^{C\times H\times W}$, we look at each sliding
window
$\bm{\mathrm{x}}_{i,j}\in\mathbb{R}^{C\times\frac{H}{M}\times\frac{W}{N}}$ as
a node to establish a graph network. For the sake of narrative convenience, we
denote $\bm{\mathrm{x}}_{m,n}$ as $\bm{\mathrm{x}}_{i}$, where $i=(m-1)\times
N+n$.

We first define the connection relation matrix of nodes
${\bm{\mathrm{R}}}^{gr}=\\{r_{i,j}^{gr}|i,j\in\\{1,2,\cdots,M\times N\\}\\}$,
here we use the matrix multiplication-based visual similarity to model the
connection relation between nodes. From Equation (2), the relation matrix
${\bm{\mathrm{R}}}^{gr}=\mathcal{R}(\bm{\mathrm{X}})$. Simultaneously, in
order to alleviate the complexity of visual similarity computation, we devise
a function $\mathcal{A}^{gr}$ to reduce the channel dimension

(a) Series

(b) Parallel

Figure 3: Three designs of fusion. ”GR” denotes the global relation modeling
module and ”LR” denotes the local relation modeling module. (a) The two
relation modeling modules are connected in two different series connections.
(b) Two relation modeling modules are connected in parallel.

$\displaystyle\bm{\mathrm{\hat{x}}}_{i}={\mathcal{A}^{gr}}(\bm{\mathrm{x}}_{i}),$
(7)

where
$\bm{\mathrm{\hat{X}}}=\\{\bm{\mathrm{\hat{x}}}_{i}\in\mathbb{R}^{\frac{C}{r^{gr}}\times\frac{H}{M}\times\frac{W}{N}}|i\in\\{1,2,\cdots,M\times
N\\}\\}$ is the hidden vector used to compute the connectivity of the graph,
mainly to save computation and to enhance the learnability of the module.
$r^{gr}$ is the channel compression ratio. Thus, this allows us to derive a
formula for calculating the connection relation that saves significant
computational complexity, as shown below

$\displaystyle\bm{\mathrm{R}}^{gr}=\mathcal{R}^{gr}(\bm{\mathrm{\hat{X}}}),$
(8)

where, for ease of understanding, we rewrite the relational modeling function
$\mathcal{R}$ in Equation (2) here as $\mathcal{R}^{gr}$. With the connection
relation matrix $\bm{\mathrm{R}}^{gr}$ (we rewrite $\bm{\mathrm{R}}$ as
$\bm{\mathrm{R}}^{gr}$ in layer $l$ of the graph neural network for simplicity
in the following section), based on Equation (4) and Equation (6), we can
define the node update function $\mathcal{U}^{gr}$ and the convolutional
function $\mathcal{C}^{gr}$ for global window-aware relation network as
follows

$\bm{\mathrm{\hat{X}}}^{(l+1)}=\mathcal{C}^{gr}\circ\mathcal{U}^{gr}(\bm{\mathrm{\hat{X}}}^{(l)}),$
(9)

where $\circ$ denotes the composition operations on multiple functions,
generating a compound function. In order to keep the input and output
dimensions unchanged, we define $(\mathcal{A}^{gr})^{-1}$, the inverse
transformation of $\mathcal{A}^{gr}$, to restore the dimension of
$\bm{\mathrm{\hat{X}}}$ in layer $(l+1)$, as shown below

$\bm{\mathrm{X}}^{(l+1)}=(\mathcal{A}^{gr})^{-1}(\bm{\mathrm{\hat{X}}}^{(l+1)}).$
(10)

####  3.3.2 Local Relation Modeling

In order to learn the local relation inside each sliding window, we conduct a
Local Relation (LR) Modeling module to build relations inside each sliding
window. Similar to the global relation modeling module, we consider each pixel
as a node and exploit the efficient graph relation network to model the local
relation among different pixels within each window.

Specifically, given the sliding window
$\bm{\mathrm{x}}_{i,j}\in\mathbb{R}^{C\times\frac{H}{M}\times\frac{W}{N}}$, we
need to learn the relationships among these feature points inside each window
$\bm{\mathrm{x}}_{i,j}$. Thus, we can similarly define the entire process as
follows

$\bm{\mathrm{x}}_{i,j}^{(l+1)}=(\mathcal{A}^{lr})^{-1}\circ\mathcal{C}^{lr}\circ\mathcal{U}^{lr}\circ\mathcal{R}^{lr}\circ\mathcal{A}^{lr}(\bm{\mathrm{x}}_{i,j}^{(l)}),$
(11)

where $\bm{\mathrm{x}}_{i,j}^{(l)}$ and $\bm{\mathrm{x}}_{i,j}^{(l+1)}$ are
the feature maps in layer $l$ and layer $l+1$ of the local window-aware
module, respectively.

####  3.3.3 Module Structure

As shown in Figure 2, our presented graph transformer can be implemented by
two $1\times 1$ Conv, a normalization function (Softmax) and other basic
elements (Self-Attention and Multilayer Perceptron) in ViT [13]. Specifically,
$\mathcal{A}^{gr}$ and $\mathcal{A}^{lr}$ can be completed by a $1\times 1$
Conv, which is used to reduce the computational complexity in the relation
modeling process. Similarly, $(\mathcal{A}^{gr})^{-1}$ and
$(\mathcal{A}^{lr})^{-1}$ can be realized by another $1\times 1$ Conv, aiming
to resume the channel dimension for module integration in the corresponding
relation modeling network. In order to accomplish $\mathcal{R}^{gr}$ and
$\mathcal{R}^{lr}$, we use a matrix multiplication and a softmax function to
norm the results and obtain relation matrix $\bm{\mathrm{R}}^{gr}$ and
$\bm{\mathrm{R}}^{lr}$ respectively. The node update function
$\mathcal{U}^{gr}$ and $\mathcal{U}^{lr}$ can be accomplished by a matrix
multiplication simply.

####  3.3.4 Fusion

Given a global relation modeling module and a local one, we fuse these two
different relation modeling modules to obtain better performance. As shown in
Figure 3, there are three fusion types, namely, global then local series and
local then global series connection (Figure 3a), parallel connection (Figure
3b). We will discuss in detail the impact of different fusion methods on
segmentation performance in the experimental section.

Figure 4: Some examples of the three datasets: Cityscapes (first row), ADE-20k
(second row), and PASCAL Context (third row).

###  3.4 Boundary-aware Attention

A robust feature extraction backbone network has been constructed, as
indicated above. We further devise boundary-aware attention (BA) network to
optimize the boundary region of the corresponding object for a better overall
semantic segmentation effect.

As we all know, the edge information adjustment of objects is generally local,
and is greatly affected by local features. Based on this, we design a local-
aware attention module, namely the BA module, which further improves the
boundary segmentation accuracy of the model by enhancing and weakening the
weight information of some local feature points (or pixels, here we do not
distinguish between pixels and feature points), some of which are the
boundaries of objects. Concretely, we perceive the relation between local
pixels by the local perception module and perform local relation modeling,
based on which we learn to obtain the attention coefficients of each
surrounding pixel. Finally, we use the attention coefficients to weigh the
pixels, strengthen the key pixels, and weaken the inessential ones to achieve
accurate boundary segmentation.

Given the input
$\bm{\mathrm{Y}}\in\mathbb{R}^{\widetilde{C}\times\widetilde{H}\times\widetilde{W}}$,
where $\widetilde{C}$, $\widetilde{H}$ and $\widetilde{W}$ denotes the
dimension of channel, height and width respectively, we use the channel
squeezing function $\bm{\mathcal{\widetilde{A}}}$ to reduce the dimension of
input features, and then use the local modeling function $\bm{\mathcal{G}}$ to
learning the local relation, where the pixels around the boundary of the
object is adjusted. Besides, we add non-linear function $\bm{\mathcal{H}}$
inside the boundary-aware module to learn the more robust relation and use the
inverse function $(\bm{\mathcal{\widetilde{A}}})^{-1}$ to resume the input
channel dimension. At last, we normalize the learned attention coefficient via
the normalization function $\bm{\mathcal{N}}$, so the whole formulation can
be:

$\bm{\mathrm{Z}}=\bm{\mathcal{N}}\circ(\bm{\mathcal{\widetilde{A}}})^{-1}\circ\bm{\mathcal{H}}\circ\bm{\mathcal{G}}\circ\bm{\mathcal{\widetilde{A}}}(\bm{\mathrm{Y}}).$
(12)

####  3.4.1 Module Structure

As shown in Figure 2, boundary-aware attention module compose of two layers of
$1\times 1$ Conv, a layer of GELU and a layer of Sigmoid. Similar to the
design of the graph transformer module, we use $1\times 1$ Conv to accomplish
the channel squeezing and resume functions $\bm{\mathcal{\widetilde{A}}}$ and
$(\bm{\mathcal{\widetilde{A}}})^{-1}$. Here we simply use $7\times 7$ Conv to
implement the local relation modeling function $\bm{\mathcal{G}}$. Finally,
function $\bm{\mathcal{H}}$ and $\bm{\mathcal{N}}$ can be completed through
the common non-linear layer (GELU) and normalization layer (Sigmoid),
respectively.

Table 1: The complexity comparison of our Graph-Segmenter model and the original Swin [14] model with Swin-L as a backbone in terms of the number of model parameters (#param) and the amount of multiplication computation (GMac). ∗ refers to reproduced results by us. Method | # param. | GMac
---|---|---
Swin∗ [14] | 233.66 | 191.45
Graph-Segmenter (Ours) | 283.46 | 195.63

###  3.5 Discussion

We discuss the complexity of the proposed model in this section. The analysis
of our designed lightweight pluggable Graph-Segmenter, consisting of a Graph
Transformer and Boundary-aware Attention, is shown in Table 1. From the table,
we can conclude that the increase of our proposed Graph-Segmenter model with
the original Swin [14] model in terms of the number of model parameters
(#param) and the multiplicative computation is very small. In particular, in
terms of multiplication computation, our model only increases the computation
of 2.18% compared to the original model.

##  4 Experiments

In this section, we evaluate our proposed Graph-Segmenter on three standard
semantic segmentation datasets: Cityscapes [20], ADE-20k [21] and PASCAL
Context [22].

###  4.1 Datasets and metrics

Cityscapes [20]. The dataset covers a range of 19 semantic classes from 50
different cities. It has 5,000 images in total, with 2,975 for training, 500
for validation, and another 1,525 for testing.

ADE-20k [21]. The dataset annotates 150 categories in challenging scenes. It
contains more than 25,000 finely annotated images, split into 20,210, 2,000
and 3,352 for training, validation and testing respectively.

PASCAL Context [22]. The dataset is an extension of the PASCAL VOC 2010, which
includes 4998 and 5105 photos for training and validation, respectively, and
gives pixel-by-pixel semantic labels. It contains more than 400 classes from
three major categories (stuff, hybrids and objects). However, since most
classes are too sparse, we only analyze the most common 60 classes, which is
the approach previous works did [43].

Figure 4 shows some examples of the three datasets.

Metrics. We adopt mean Intersection over Union (mIoU) to evaluate semantic
segmentation models.

###  4.2 Implementation Details

The entire model is implemented based on the Swin [14] codes that have been
made publicly available. Training is separated into two phases: we first
establish the backbone network using the model parameters pre-trained on
ImageNet, and then we train the backbone network with the Graph-Segmenter that
we proposed. The trained network parameters are utilized to initialize and
train the network integrated with the segmentation boundary optimization
module, which is then used to optimize the segmentation boundary.

###  4.3 Comparison with State-of-the-Art

We compare our proposed Graph-Segmenter with the state-of-the-art models,
which contains CNN (e.g., ResNet-101, ResNeSt-200) based methods [26, 27, 29,
65], and latest transformer based methods [14, 43, 42] on Cityscapes, ADE-20k
and PASCAL Context datasets.

Table 2: Multi-scale inference results of semantic segmentation on the Cityscapes validation dataset compared with state-of-the-art methods. ∗ refers to reproduced results. Method | Backbone | val mIoU
---|---|---
FCN [23] | ResNet-101 | 76.6
Non-local [66] | ResNet-101 | 79.1
DLab.v3+ [26] | ResNet-101 | 79.3
DNL [65] | ResNet-101 | 80.5
DenseASPP [67] | DenseNet | 80.6
DPC [68] | Xception-71 | 80.8
CCNet [31] | ResNet-101 | 81.3
DANet [27] | ResNet-101 | 81.5
Panoptic-Deeplab [69] | Xception-71 | 81.5
Strip Pooling [70] | ResNet-101 | 81.9
Seg-L-Mask/16 [42] | ViT-L | 81.3
SETR-PUP [43] | ViT-L | 82.2
Swin∗ [14] | Swin-L | 82.3
Graph-Segmenter (Ours) | Swin-L | 82.9
Table 3: Results of semantic segmentation on the Cityscapes test dataset compared with state-of-the-art methods. ∗ refers to reproduced results by us. Method | Backbone | test mIoU
---|---|---
PSPNet [25] | ResNet-101 | 78.4
BiSeNet [71] | ResNet-101 | 78.9
PSANet [72] | ResNet-101 | 80.1
OCNet [73] | ResNet-101 | 80.1
BFP [34] | ResNet-101 | 81.4
DANet [27] | ResNet-101 | 81.5
CCNet [31] | ResNet-101 | 81.9
SETR-PUP [43] | ViT-L | 81.1
Swin∗ [14] | Swin-L | 80.6
Graph-Segmenter (Ours) | Swin-L | 81.9
Table 4: Multi-scale inference results of semantic segmentation on the ADE-20k validation dataset compared with state-of-the-art methods. ∗ refers to reproduced results by us. Method | Backbone | val mIoU
---|---|---
FCN [23] | ResNet-101 | 41.4
UperNet [74] | ResNet-101 | 44.9
DANet [27] | ResNet-101 | 45.3
OCRNet [29] | ResNet-101 | 45.7
ACNet [75] | ResNet-101 | 45.9
DNL [65] | ResNet-101 | 46.0
DLab.v3+ [26] | ResNeSt-101 | 47.3
DLab.v3+ [26] | ResNeSt-200 | 48.4
SETR-PUP [43] | ViT-L | 50.3
SegFormer-B5 [15] | SegFormer | 51.8
Seg-L-Mask/16 [42] | ViT-L | 53.6
Swin∗ [14] | Swin-L | 53.1
Graph-Segmenter (Ours) | Swin-L | 53.9
Table 5: Results of semantic segmentation on the ADE-20k test dataset compared with state-of-the-art methods. ∗ refers to reproduced results by us. Method | Backbone | test score
---|---|---
ACNet [75] | ResNet-101 | 38.5
DLab.v3+ [26] | ResNeSt-101 | 55.1
OCRNet [29] | ResNet-101 | 56.0
DNL [65] | ResNet-101 | 56.2
SETR-PUP [43] | ViT-L | 61.7
Swin∗ [14] | Swin-L | 61.5
Graph-Segmenter (Ours) | Swin-L | 62.4

Cityscapes. Table 2 and Table 3 demonstrate the results on Cityscapes dataset.
It can be seen that our Graph-Segmenter is +0.6 mIoU higher (82.9 vs. 82.3)
than transformer-based method Swin-L [14] in Cityscapes validation dataset and
+0.8 mIoU higher (81.9 vs. 81.1) than transformer-based SETR-PUP [43] in
Cityscapes test dataset. In addition, we further show the accuracy of our
proposed Graph-Segmenter in each category, as shown in Figure 6. From the
figure, we can see that Graph-Segmenter almost obtains the best performance in
all categories compared to the Swin model, and even for some more difficult
categories (e.g., train, rider, wall) our model still achieves a very large
performance improvement. This phenomenon shows that global and local window
relation modeling and boundary-aware attention modeling have further improved
the ability of the network for difficult-to-segment categories.

ADE-20k. Table 4 and Table 5 demonstrate the results on ADE-20k dataset. Our
Graph-Segmenter achieves 53.9 mIoU and 62.4 test score on the ADE-20k val and
test set, surpassing the previous best transformer-based model by +0.3 mIoU
(53.6 mIoU by Seg-L-Mask/16 [42]) and +0.7 test score (61.7 by SETR-PUP [43]).
Our approach still can perform excellently in the challenging ADE-20k dataset
with more categories, which confirms our modules’ robustness. A more
qualitative comparison is shown in Figure 5.

PASCAL Context. To further demonstrate the effectiveness of our proposed
Graph-Segmenter module, we conduct more experimental results on PASCAL Context
dataset for UperNet [76] and “UperNet + CAR” [77] models. As shown in Table 6,
we can observe that different models equipped with our Graph-Segmenter module
can gain a consistent performance improvement. All in all, the very recent
model “UperNet + CAR” embedding Graph-Segmenter achieves the best mIoU
accuracy.

Figure 5: Qualitative performance comparison of our proposed Graph-Segmenter with DeepLabV3+ [26] and Swin Tranformer [14] for semantic segmentation. Our Graph-Segmenter can obtain a better segmentation boundary. Table 6: Results of semantic segmentation on the Pascal Context dataset compared with state-of-the-art methods. Method | Backbone | mIoU
---|---|---
FCN[23] | ResNet-101 | 45.74
PSPNet [25] | ResNet-101 | 47.80
DANet [27] | ResNet-101 | 52.60
EMANet [78] | ResNet-101 | 53.10
SVCNet [79] | ResNet-101 | 53.20
BFP [34] | ResNet-101 | 53.6
Strip pooling [70] | ResNet-101 | 54.50
GFFNet [80] | ResNet-101 | 54.20
APCNet [33] | ResNet-101 | 54.70
GRAr [28] | ResNet-101 | 55.70
SETR-PUP[43] | ViT-L | 55.83
UperNet [76] | Swin-L | 57.48
Graph-Segmenter + UperNet (Ours) | Swin-L | 57.80
Graph-Segmenter + UperNet + CAR (Ours) | Swin-L | 59.01

Qualitative results. We further visualized and analyzed our devised Graph-
segmenter network, as shown in Figure 5. As shown in Figure 5, our Graph-
Segmenter can obtain a better segmentation boundary compared with DeepLabV3+
[26] and Swin Tranformer [14].

Table 7: mIoU performance of different connection types on Cityscapes dataset. “$\cup$” denotes parallel connection. “-T” denotes Tiny. Connection Type | Backbone | val mIoU
---|---|---
GR $\cup$ LR | Swin-T | 76.70
LR followed by GR | Swin-T | 76.44
GR followed by LR | Swin-T | 76.99
Table 8: mIoU performance of different $\theta$ on Cityscape dataset. We set $v=\frac{1}{K^{2}}\displaystyle\sum_{i}^{K}\sum_{j}^{K}r_{i,j}$ for the convenience of expression. “-T” denotes Tiny. $\theta$ | Backbone | val mIoU
---|---|---
$2v$ | Swin-T | 76.44
$v$ | Swin-T | 76.27
$\frac{1}{2}v$ | Swin-T | 76.07
$\frac{1}{4}v$ | Swin-T | 77.64
$\frac{1}{8}v$ | Swin-T | 77.05
Table 9: mIoU performance of different components on Cityscape dataset. “-T” denotes Tiny. Method | Backbone | val mIoU
---|---|---
Swin [14] | Swin-T | 75.82
Swin-GT | Swin-T | 76.99
Swin-BA | Swin-T | 75.94
Swin-GTBA | Swin-T | 77.32

###  4.4 Ablation Study

In this section, we deeply investigate the performance of each sub-module of
the model and conduct experiments on the Cityscapes dataset.

####  4.4.1 Sparsity Analysis

We first analyze the effect of different sparsity of the relation matrix
$\bm{\mathrm{R}}$ on the experimental results. In order to eliminate the
influence of other factors, we do not add the boundary-aware attention module
in our experiments. By a simple setting of
$\theta=\frac{1}{K^{2}}\displaystyle\sum_{i}^{K}\sum_{j}^{K}r_{i,j}$, the
accuracy of the model with the sparse constraint on the Cityscapes dataset
could achieve 0.59 rise than the case without it. It can be concluded that for
a given node, not all surrounding nodes can provide positive feature
enhancement for the node. On the contrary, by properly filtering out some
nodes, the model can achieve better segmentation performance. To further
discuss the effect of the value of $\theta$ on the experimental results, we
set $\theta$ as different thresholds, respectively, as shown in Table 8. From
the table, we can observe that different thresholds still have a certain
degree of influence on the final segmentation effect of the model, and it is
found that the model can achieve the best results when
$\theta=\frac{1}{4K^{2}}\displaystyle\sum_{i}^{K}\sum_{j}^{K}r_{i,j}$.
Besides, the experimental results further prove the effectiveness of the
sparse setting, which can not only simplify the calculation, but also improve
the segmentation accuracy of the model to a certain extent.

Figure 6: Val mIoU comparison with Swin Tranformer [14] in each class on the
Cityscapes val set. Our Graph-Segmenter generally achieves better performance.
"sw" denotes "sidewalk", "bd" denotes "building", "tl" denotes "traffic
light", "ts" denotes "traffic sign", "vg" denotes "vegetation", "mc" denotes
"motorcycle" and "bc" denotes "bicycle".

####  4.4.2 Different Connection Type Analysis

We analyze the influence of the different sparseness of the relation matrix R
on the experimental results. As Table 7 shows, we can see that the connection
in the serial is better than in parallel. Meanwhile, the best performance is
obtained for serial connections by placing global window-aware attention
before local window-aware attention. The phenomenon further indicates that
better segmentation results can be achieved by large-scale coarse-grained
relation modeling among windows first and following fine-grained modeling
within each window.

Table 10: mIoU performance of two components in the graph transformer module on Cityscape dataset. “-T” denotes Tiny.  Method | Backbone | val mIoU
---|---|---
Swin [14] | Swin-T | 75.82
Swin-GR | Swin-T | 76.24
Swin-LR | Swin-T | 76.28
Swin-GT | Swin-T | 76.99
Table 11: mIoU performance of Graph-Segmenter (only with graph transformer module) on Cityscapes dataset for various channel compression ratio $r$. r | r=2 | r=4 | r=8 | r=16 | r=32
---|---|---|---|---|---
mIoU | 76.65 | 76.15 | 76.29 | 76.99 | 76.35

####  4.4.3 Graph Transformer and Boundary-aware Attention Modules Analysis

In order to analyze the specific role of each component, we compare various
sub-modules to analyze the experimental results separately. As Table 9 shows,
the model equipped with a graph transformer (Swin-GT) module or equipped with
boundary-aware attention (Swin-BA) module can obtain a noticeable improvement
compared to the original Swin model. Besides, we can observe that BA can
improve a lot over Swin-GT, although it only gains marginal improvement over
Swin. This phenomenon implies that after modeling the global and local
relationship, the BA module can achieve more accurate classification based on
more robust features than without Swin-GT enhancement. At the same time, we
can see that Swin embeds all of the proposed modules (GT+BA) and obtains the
best performance compared to the model embedding only some of our modules.

For analyzing the effect of two components in the graph transformer module, we
compare the sub-module results, as shown in Table 10. The model with the
Global Relation Modeling module (Swin-GR) or with the Local Relation Modeling
module (Swin-LR) can obtain a visible improvement compared to the original
model. When combining both kinds of sub-modules (Swin-GT), the model can
achieve the best performance.

####  4.4.4 Different Channel Compression Ratio $r$ Analysis

We further analyze the performance of different compression ratios in the
graph transformer module. As indicated in Table 11, we can see that the model
obtains the best semantic segmentation accuracy when the compression ratio $r$
(here we simply set $r^{gr}$ and $r^{lr}$ to the same value and mark them as
$r$) is 16, which indicates that either too large or too small compression
rate will have an impact on the recognition progress of the model, so as shown
in the table, we choose the same compression rate of 16 in this paper.

##  5 Conclusion

This paper presents Graph-Segmenter, which enhanced the vision transformer
with hierarchical level graph reasoning and efficient boundary adjustment
requiring no additional annotation. Graph-Segmenter achieves state-of-the-art
performance on Cityscapes, ADE-20k and PASCAL Context semantic segmentation,
noticeably surpassing previous results. Extensive experiments demonstrate the
effectiveness of multi-level graph modeling, features with unstructured
associations leading to evident rise; and the effectiveness of the Boundary-
aware Attention (BA) module, minor refinements around boundary gave smoother
masks than the previous state-of-the-art.

## References

  * [1] Hongjia Ruan, Huihui Song, Bo Liu, Yong Cheng, and Qingshan Liu.  Intellectual property protection for deep semantic segmentation models.  Frontiers of Computer Science, 17(1):1–9, 2023.
  * [2] Di Zhang, Yong Zhou, Jiaqi Zhao, Zhongyuan Yang, Hui Dong, Rui Yao, and Huifang Ma.  Multi-granularity semantic alignment distillation learning for remote sensing image semantic segmentation.  Frontiers of Computer Science, 16(4):1–3, 2022.
  * [3] Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu.  A survey of deep learning techniques for autonomous driving.  Journal of Field Robotics, 37(3):362–386, 2020.
  * [4] Di Feng, Christian Haase-Schütz, Lars Rosenbaum, Heinz Hertlein, Claudius Glaeser, Fabian Timm, Werner Wiesbeck, and Klaus Dietmayer.  Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges.  IEEE Transactions on Intelligent Transportation Systems, 22(3):1341–1360, 2020.
  * [5] Joel Janai, Fatma Güney, Aseem Behl, Andreas Geiger, et al.  Computer vision for autonomous vehicles: Problems, datasets and state of the art.  Foundations and Trends in Computer Graphics and Vision, 12(1–3):1–308, 2020.
  * [6] Eduardo Arnold, Omar Y Al-Jarrah, Mehrdad Dianati, Saber Fallah, David Oxtoby, and Alex Mouzakitis.  A survey on 3d object detection methods for autonomous driving applications.  IEEE Transactions on Intelligent Transportation Systems, 20(10):3782–3795, 2019.
  * [7] Panqu Wang, Pengfei Chen, Ye Yuan, Ding Liu, Zehua Huang, Xiaodi Hou, and Garrison Cottrell.  Understanding convolution for semantic segmentation.  In Winter Conference on Applications of Computer Vision, pages 1451–1460. IEEE, 2018.
  * [8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [9] Li Wang, Dong Li, Yousong Zhu, Lu Tian, and Yi Shan.  Dual super-resolution learning for semantic segmentation.  In Computer Vision and Pattern Recognition, pages 3774–3783. IEEE, 2020.
  * [10] Changqian Yu, Jingbo Wang, Changxin Gao, Gang Yu, Chunhua Shen, and Nong Sang.  Context prior for scene segmentation.  In Computer Vision and Pattern Recognition, pages 12416–12425. IEEE, 2020.
  * [11] Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap.  Compressive transformers for long-range sequence modelling.  arXiv preprint arXiv:1911.05507, 2019.
  * [12] Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh.  Set transformer: A framework for attention-based permutation-invariant neural networks.  In International Conference on Machine Learning, pages 3744–3753. PMLR, 2019.
  * [13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations. OpenReview.net, 2021.
  * [14] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  arXiv preprint arXiv:2103.14030, 2021.
  * [15] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  arXiv preprint arXiv:2105.15203, 2021.
  * [16] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen.  Twins: Revisiting the design of spatial attention in vision transformers.  arXiv preprint arXiv:2104.13840, 2021.
  * [17] Jiemin Fang, Lingxi Xie, Xinggang Wang, Xiaopeng Zhang, Wenyu Liu, and Qi Tian.  Msg-transformer: Exchanging local spatial information by manipulating messenger tokens.  arXiv preprint arXiv:2105.15168, 2021.
  * [18] Pichao Wang, Xue Wang, Fan Wang, Ming Lin, Shuning Chang, Wen Xie, Hao Li, and Rong Jin.  Kvt: k-nn attention for boosting vision transformers.  arXiv preprint arXiv:2106.00515, 2021.
  * [19] Xiangxiang Chu, Bo Zhang, Zhi Tian, Xiaolin Wei, and Huaxia Xia.  Do we really need explicit position encodings for vision transformers?  arXiv preprint arXiv:2102.10882, 2021.
  * [20] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Computer Vision and Pattern Recognition, pages 3213–3223. IEEE, 2016.
  * [21] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Semantic understanding of scenes through the ade20k dataset.  International Journal of Computer Vision, 127(3):302–321, 2019\.
  * [22] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille.  The role of context for object detection and semantic segmentation in the wild.  In Computer Vision and Pattern Recognition, pages 891–898. IEEE, 2014.
  * [23] Jonathan Long et al.  Fully convolutional networks for semantic segmentation.  In Computer Vision and Pattern Recognition, pages 3431–3440. IEEE, 2015.
  * [24] Ying Shen, Heye Zhang, Yiting Fan, Alex Puiwei Lee, and Lin Xu.  Smart health of ultrasound telemedicine based on deeply represented semantic segmentation.  IEEE Internet of Things Journal, 8(23):16770–16778, 2021.
  * [25] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In Computer Vision and Pattern Recognition, pages 2881–2890. IEEE, 2017.
  * [26] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In European Conference on Computer Vision, pages 833–851. Springer, 2018.
  * [27] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu.  Dual attention network for scene segmentation.  In Computer Vision and Pattern Recognition, pages 3146–3154. IEEE, 2019.
  * [28] Henghui Ding, Hui Zhang, Jun Liu, Jiaxin Li, Zijian Feng, and Xudong Jiang.  Interaction via bi-directional graph of semantic region affinity for scene parsing.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15848–15858, 2021.
  * [29] Yuhui Yuan, Xiaokang Chen, Xilin Chen, and Jingdong Wang.  Segmentation transformer: Object-contextual representations for semantic segmentation.  In European Conference on Computer Vision. Springer, 2020.
  * [30] Xiangtai Li, Ansheng You, Zhen Zhu, Houlong Zhao, Maoke Yang, Kuiyuan Yang, Shaohua Tan, and Yunhai Tong.  Semantic flow for fast and accurate scene parsing.  In European Conference on Computer Vision, pages 775–793. Springer, 2020.
  * [31] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu.  Ccnet: Criss-cross attention for semantic segmentation.  In International Conference on Computer Vision, pages 603–612. IEEE, 2019.
  * [32] Xiangtai Li, Houlong Zhao, Lei Han, Yunhai Tong, Shaohua Tan, and Kuiyuan Yang.  Gated fully fusion for semantic segmentation.  In AAAI Conference on Artificial Intelligence, pages 11418–11425. AAAI, 2020.
  * [33] Junjun He, Zhongying Deng, Lei Zhou, Yali Wang, and Yu Qiao.  Adaptive pyramid context network for semantic segmentation.  In Computer Vision and Pattern Recognition, pages 7519–7528. IEEE, 2019.
  * [34] Henghui Ding, Xudong Jiang, Ai Qun Liu, Nadia Magnenat Thalmann, and Gang Wang.  Boundary-aware feature propagation for scene segmentation.  In International Conference on Computer Vision, pages 6819–6829. IEEE, 2019.
  * [35] Volodymyr Mnih, Nicolas Heess, Alex Graves, et al.  Recurrent models of visual attention.  In Conference on Neural Information Processing Systems, pages 2204–2212. MIT Press, 2014.
  * [36] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473, 2014.
  * [37] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran.  Image transformer.  In International Conference on Machine Learning, pages 4055–4064. PMLR, 2018.
  * [38] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European Conference on Computer Vision, pages 213–229. Springer, 2020.
  * [39] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable DETR: Deformable transformers for end-to-end object detection.  In International Conference on Learning Representations. OpenReview.net, 2021.
  * [40] Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia.  End-to-end video instance segmentation with transformers.  In Computer Vision and Pattern Recognition, pages 8741–8750. IEEE, 2021.
  * [41] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon.  Detr3d: 3d object detection from multi-view images via 3d-to-2d queries.  In Conference on Robot Learning, pages 180–191. PMLR, 2022.
  * [42] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid.  Segmenter: Transformer for semantic segmentation.  In IEEE International Conference on Computer Vision. IEEE, 2021\.
  * [43] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.  In Computer Vision and Pattern Recognition. IEEE, 2021.
  * [44] Li Zhang, Xiangtai Li, Anurag Arnab, Kuiyuan Yang, Yunhai Tong, and Philip HS Torr.  Dual graph convolutional network for semantic segmentation.  arXiv preprint arXiv:1909.06121, 2019.
  * [45] Shun-Yi Pan, Cheng-You Lu, Shih-Po Lee, and Wen-Hsiao Peng.  Weakly-supervised image semantic segmentation using graph convolutional networks.  In International Conference on Multimedia and Expo, pages 1–6. IEEE, 2021.
  * [46] Hao Wang, Liyan Dong, and Minghui Sun.  Local feature aggregation algorithm based on graph convolutional network.  Frontiers of Computer Science, 16(3):1–3, 2022.
  * [47] Jiancan Wu, Xiangnan He, Xiang Wang, Qifan Wang, Weijian Chen, Jianxun Lian, and Xing Xie.  Graph convolution machine for context-aware recommender system.  Frontiers of Computer Science, 16(6):1–12, 2022.
  * [48] Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun.  Spectral networks and locally connected networks on graphs.  In International Conference on Learning Representations. OpenReview.net, 2014.
  * [49] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio.  Graph attention networks.  In International Conference on Learning Representations. OpenReview.net, 2018.
  * [50] Li Zhang, Dan Xu, Anurag Arnab, and Philip HS Torr.  Dynamic graph message passing networks.  In Computer Vision and Pattern Recognition, pages 3726–3735. IEEE, 2020.
  * [51] Yiran Zhu, Xing Xu, Fumin Shen, Yanli Ji, Lianli Gao, and Heng Tao Shen.  Posegtac: Graph transformer encoder-decoder with atrous convolution for 3d human pose estimation.  International Joint Conference on Artificial Intelligence, 2021\.
  * [52] Xinzhi Dong, Chengjiang Long, Wenju Xu, and Chunxia Xiao.  Dual graph convolutional networks with transformer and curriculum learning for image captioning.  In ACM International Conference on Multimedia, pages 2615–2624, 2021.
  * [53] Sijie Yan, Yuanjun Xiong, and Dahua Lin.  Spatial temporal graph convolutional networks for skeleton-based action recognition.  In AAAI Conference on Artificial Intelligence. AAAI, 2018.
  * [54] Tengpeng Li, Kaihua Zhang, Shiwen Shen, Bo Liu, Qingshan Liu, and Zhu Li.  Image co-saliency detection and instance co-segmentation using attention graph clustering based graph convolutional network.  IEEE Transactions on Multimedia, 2021.
  * [55] Xia Li, Yibo Yang, Qijie Zhao, Tiancheng Shen, Zhouchen Lin, and Hong Liu.  Spatial pyramid based graph reasoning for semantic segmentation.  In Computer Vision and Pattern Recognition, pages 8950–8959. IEEE, 2020.
  * [56] Hanzhe Hu, Deyi Ji, Weihao Gan, Shuai Bai, Wei Wu, and Junjie Yan.  Class-wise dynamic graph convolution for semantic segmentation.  In European Conference on Computer Vision, pages 1–17. Springer, 2020.
  * [57] Yang Zhang, Moyun Liu, Jingwu He, Fei Pan, and Yanwen Guo.  Affinity fusion graph-based framework for natural image segmentation.  IEEE Transactions on Multimedia, 2021.
  * [58] Chaofan Chen, Shengsheng Qian, Quan Fang, and Changsheng Xu.  Hapgn: Hierarchical attentive pooling graph network for point cloud segmentation.  IEEE Transactions on Multimedia, 23:2335–2346, 2020.
  * [59] Yanfei Su, Weiquan Liu, Zhimin Yuan, Ming Cheng, Zhihong Zhang, Xuelun Shen, and Cheng Wang.  Dla-net: Learning dual local attention features for semantic segmentation of large-scale building facade point clouds.  Pattern Recognition, 123:108372, 2022.
  * [60] Yiding Liu, Siyu Yang, Bin Li, Wengang Zhou, Jizheng Xu, Houqiang Li, and Yan Lu.  Affinity derivation and graph merge for instance segmentation.  In European Conference on Computer Vision, pages 686–703, 2018\.
  * [61] Ziwei Zhang, Peng Cui, and Wenwu Zhu.  Deep learning on graphs: A survey.  IEEE Transactions on Knowledge and Data Engineering, 2020.
  * [62] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip.  A comprehensive survey on graph neural networks.  IEEE transactions on neural networks and learning systems, 32(1):4–24, 2020.
  * [63] William L Hamilton, Rex Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  In Conference on Neural Information Processing Systems, pages 1025–1035. MIT Press, 2017.
  * [64] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In International Conference on Learning Representations. OpenReview.net, 2017.
  * [65] Minghao Yin, Zhuliang Yao, Yue Cao, Xiu Li, Zheng Zhang, Stephen Lin, and Han Hu.  Disentangled non-local neural networks.  In European Conference on Computer Vision, pages 191–207. Springer, 2020.
  * [66] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In Computer Vision and Pattern Recognition, pages 7794–7803. IEEE, 2018.
  * [67] Maoke Yang, Kun Yu, Chi Zhang, Zhiwei Li, and Kuiyuan Yang.  Denseaspp for semantic segmentation in street scenes.  In Computer Vision and Pattern Recognition, pages 3684–3692. IEEE, 2018.
  * [68] Liang-Chieh Chen, Maxwell D Collins, Yukun Zhu, George Papandreou, Barret Zoph, Florian Schroff, Hartwig Adam, and Jonathon Shlens.  Searching for efficient multi-scale architectures for dense image prediction.  arXiv preprint arXiv:1809.04184, 2018.
  * [69] Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang-Chieh Chen.  Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation.  In Computer Vision and Pattern Recognition, pages 12475–12485. IEEE, 2020.
  * [70] Qibin Hou, Li Zhang, Ming-Ming Cheng, and Jiashi Feng.  Strip pooling: Rethinking spatial pooling for scene parsing.  In Computer Vision and Pattern Recognition, pages 4003–4012. IEEE, 2020.
  * [71] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang.  Bisenet: Bilateral segmentation network for real-time semantic segmentation.  In European Conference on Computer Vision, pages 325–341. Springer, 2018.
  * [72] Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen Change Loy, Dahua Lin, and Jiaya Jia.  Psanet: Point-wise spatial attention network for scene parsing.  In European Conference on Computer Vision, pages 267–283. Springer, 2018.
  * [73] Yuhui Yuan, Lang Huang, Jianyuan Guo, Chao Zhang, Xilin Chen, and Jingdong Wang.  Ocnet: Object context network for scene parsing.  arXiv preprint arXiv:1809.00916, 2018.
  * [74] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun.  Unified perceptual parsing for scene understanding.  In European Conference on Computer Vision, pages 418–434. Springer, 2018.
  * [75] Jun Fu, Jing Liu, Yuhang Wang, Yong Li, Yongjun Bao, Jinhui Tang, and Hanqing Lu.  Adaptive context network for scene parsing.  In IEEE International Conference on Computer Vision, pages 6748–6757. IEEE, 2019.
  * [76] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun.  Unified perceptual parsing for scene understanding.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 418–434, 2018.
  * [77] Ye Huang, Di Kang, Liang Chen, Xuefei Zhe, Wenjing Jia, Xiangjian He, and Linchao Bao.  Car: Class-aware regularizations for semantic segmentation.  arXiv preprint arXiv:2203.07160, 2022.
  * [78] Xia Li, Zhisheng Zhong, Jianlong Wu, Yibo Yang, Zhouchen Lin, and Hong Liu.  Expectation-maximization attention networks for semantic segmentation.  In Computer Vision and Pattern Recognition, pages 9167–9176. IEEE, 2019.
  * [79] Henghui Ding, Xudong Jiang, Bing Shuai, Ai Qun Liu, and Gang Wang.  Semantic correlation promoted shape-variant context for segmentation.  In Computer Vision and Pattern Recognition, pages 8885–8894. IEEE, 2019.
  * [80] Xiangtai Li, Houlong Zhao, Lei Han, Yunhai Tong, Shaohua Tan, and Kuiyuan Yang.  Gated fully fusion for semantic segmentation.  In AAAI Conference on Artificial Intelligence, volume 34, pages 11418–11425, 2020.

bio/Zizhang_Wu.jpgZizhang Wu received the B.Sc. and M.Sc. degree in Pattern
Recognition and Intelligent Systems from Northeastern University in 2010 and
2012, respectively. He is currently a perception algorithm manager in the
Computer Vision Perception Department of ZongMu Technology. He is mainly
responsible for the development of core perception algorithms, optimizing
algorithms, and model effects, and driving business development with
technology.width=3cm,height=3.8cm

bio/Yuanzhu_Gan.jpgYuanzhu Gan received the M.Sc. degree in Pattern
Recognition and Artificial Intelligence from Nanjing University, Jiangsu,
China, in 2021. He is now an algorithm engineer at ZongMu Technology,
Shanghai, China. His current interests include 3D object detection for
autonomous driving.width=3cm,height=3.8cm

bio/Tianhao_Xu.jpgTianhao Xu received the B.E. degree in Vehicle Engineering
from Jilin University, Jilin, China, in 2017. He is currently pursuing the
M.Sc. degree in Electric Mobility at Technical University of Braunschweig,
Braunschweig, Germany. His current research interests include computer vision
and microstructural analysis in machine learning.width=3cm,height=3.8cm

bio/Fan_Wang.jpgFan Wang received the B.Sc. and M.Sc. degree in Computer
Science and Artificial Intelligence from Northwestern Polytechnical
University, Xi’an, China, in 1997 and 2000, respectively. He is currently with
ZongMu Technology as a Vice President and Chief Technology Officer. His
current research interests include Computer Vision, Sensor Fusion, Automatic
Parking, Planning & Control, and $L2/L3/L4$ Autonomous
Driving.width=3cm,height=3.8cm