# AsymFormer: Asymmetrical Cross-Modal Representation Learning for Mobile
Platform Real-Time RGB-D Semantic Segmentation

Siqi Du1$\nmid$, Weixi Wang1$\nmid$, _Member, IEEE_ , Renzhong Guo1 and
Shengjun Tang1*, _Member, IEEE_ $\nmid$ indicates equal contribution.*
corresponding author.This work was supported by the Research Program of
Shenzhen S and T Innovation Committee grant( (Grant No. JCYJ20210324093012033)
and the Natural Science Foundation of Guangdong Province grant (Grant No.
2121A1515012574) and the National Natural Science Foundation of China (Project
Nos. 71901147, 41971354, and 41971341), Shenzhen Key Laboratory of Digital
Twin Technologies for Cities (Project No: ZDSYS20210623101800001)1Siqi Du,
Weixi Wan, Renzhong Guo, and Shengjun Tang is with Research Institute for
Smart Cities, School of Architecture and Urban Planning, Shenzhen University

###### Abstract

In the realm of robotic intelligence, achieving efficient and precise RGB-D
semantic segmentation is a key cornerstone. State-of-the-art multimodal
semantic segmentation methods, primarily rooted in symmetrical skeleton
networks, find it challenging to harmonize computational efficiency and
precision. In this work, we propose AsymFormer, a novel network for real-time
RGB-D semantic segmentation, which targets the minimization of superfluous
parameters by optimizing the distribution of computational resources and
introduces an asymmetrical backbone to allow for the effective fusion of
multimodal features. Furthermore, we explore techniques to bolster network
accuracy by redefining feature selection and extracting multi-modal self-
similarity features without a substantial increase in the parameter count,
thereby ensuring real-time execution on robotic platforms. Additionally, a
Local Attention-Guided Feature Selection (LAFS) module is used to selectively
fuse features from different modalities by leveraging their dependencies.
Subsequently, a Cross-Modal Attention-Guided Feature Correlation Embedding
(CMA) module is introduced to further extract cross-modal representations.
This method is evaluated on NYUv2 and SUNRGBD datasets, with AsymFormer
demonstrating competitive results with 52.0% mIoU on NYUv2 and 49.1% mIoU on
SUNRGBD. Notably, AsymFormer achieves an inference speed of 65 FPS and after
implementing mixed precision quantization, it attains an impressive inference
speed of 79 FPS on RTX3090. This significantly outperforms existing multi-
modal methods, thereby demonstrating that AsymFormer can strike a balance
between high accuracy and efficiency for RGB-D semantic segmentation. Code:
https://github.com/Fourier7754/AsymFormer

##  I INTRODUCTION

Real-time semantic segmentation methods are important for robots as they rely
on obtaining semantic information in real-time to support navigation and task
decision-making. Current Real-Time semantic segmentation networks achieve
competitive results and reach over 100 FPS inference speed on simple outdoor
environments (Cityscapes 19 classes) [3, 4, 2, 1]. However, these networks
perform poorly on complex indoor environments, such as NYU Depth v2 (40
classes) [5] and SUNRGBD (37 classes) [6]. Several studies have explored how
to improve indoor scene semantic segmentation performance by integrating RGB-D
information [7, 8, 9, 10], which is widely available from RGB-D sensors on
indoor robots. RGB-D information consists of RGB (color, texture and shape)
and Depth (boundaries and relative location) features, which are somewhat
complementary [7, 8].

At present, RGB-D semantic segmentation methods have achieved state-of-the-art
performance by employing dual-branch backbone for feature extraction and
attention mechanisms for feature selection [8]. However, there is a dearth of
discourse on whether these operations can be executed with less redundancy and
higher efficiency. Primarily, most methodologies utilize two symmetrical,
computationally intensive backbones, which effectively doubles the overall
computational complexity [12]. Secondly, numerous methods concentrate on
designing complex feature fusion modules to enhance accuracy, but do not
exhibit significant advantages than some methods that employ simpler attention
modules like ESANet [9], which indicates that as more computational resources
are invested, the incremental improvement in accuracy decreases. Lastly, the
use of cross-attention modules to jointly utilize multi-modal features has
been demonstrated to be effective in improving network accuracy [8, 28].
However, most methods do not discuss how to implement cross-attention more
efficiently. Therefore, the reduction of redundancy in dual-branch networks
and the construction of more efficient attention modules remain significant
topics for discussion.

To address this issue, this study suggests a balanced approach between
computational efficiency and feature richness in multi-modal representation
learning. RGB images are found to contain more information than Depth images,
thus it is not necessary to use the same size backbone for extracting Depth
information. The performance improvements achieved by using attention
mechanisms for various tasks come with minimal increase in computational cost.
While existing modules like SE [34] are efficient for feature selection, they
lack the ability to model correlations in spatial dimensions. Increasing
parameters can facilitate spatial dimension feature selection, but careful
design is needed to maintain computational efficiency. For feature extraction,
the commonly used Multi-Head Self Attention (MHSA) [22] is effective for
single-modal features despite its small parameter count. Extending this
efficient feature extraction to multiple modalities is feasible, but
consideration should be given to feature fusion while maintaining
computational efficiency.

This paper introduces AsymFormer, a high-performance real-time network for
RGB-D integration semantic segmentation that employs an asymmetrical backbone
design. This includes a larger parameter backbone for RGB features and a
smaller backbone for the Depth branch. Regarding framework selection, Vision
Transformer are always performance better, but slower than same computational
complexity CNNs due to a lack of hardware optimization [40]. In order to speed
up the main branch, this paper employ a hardware-friendly CNN [16] for the RGB
branch and a Transformer [17] with fewer parameters but similar performance
for the Depth branch to further compress the parameters. This paper also
discuss feature selection modeling and construct the Local Attention-Guided
Feature Selection (LASF) module. This module computes the spatial-channel
attention weights in parallel to enhance inference speed. Moreover, this paper
establishes a novel approach to model spatial attention. This module can
estimate the differences in features from different modalities by utilizing
learnable channel weights and calculating spatial attention weights for each
pixel. Additionally, a Cross-Modal Attention (CMA) module is introduced to
embed cross-modal integration learning information into pixel-wise fused
features. Finally, we employ a lightweight MLP-Decoder[17] to decode semantic
information from shallow features.

This paper evaluates AsymFormer on two classic indoor scene semantic
segmentation datasets: NYUv2 and SUNRGBD. Meanwhile, the inference speed test
is also preformed on Nvidia RTX 3090 platform. The AsymFormer achieves 52.0%
mIoU on NYUv2 and 49.1 mIoU on SUNRGBD, with 65 FPS inference speed. After
performing mixed precision quantization on the model using the TensorRT
framework, AsymFormer achieves an impressive inference speed of 79 FPS on RTX
3090, which significantly surpasses existing methods. Our experiments
highlight AsymFormer’s ability to acquire high accuracy and efficiency at the
same time. Our main contributions are summarized as follows:

  * $\bullet$

We employed an asymmetric backbone that compressed the parameters of the Depth
feature extraction branch, thus reducing redundancy.

  * $\bullet$

We introduced the LAFS module for feature selection, utilizing learnable
feature weights to calculate spatial attention weights.

  * $\bullet$

We introduce a novel efficient cross-modal attention (CMA) for modeling of
self-similarity in multi-modal features, validating its capability to enhance
network accuracy with minimal additional model parameters.

##  II Related Works

RGB-D Representation Learning. One of the earliest works on RGB-D semantic
segmentation, FCN [35], treated RGB-D information as a single input and
processed it with a single backbone. However, subsequent works have recognized
the need to extract features from RGB and Depth information separately, as
they have different properties. Therefore, most of them have adopted two
symmetric backbones for RGB and Depth feature extraction [36, 37, 9, 11, 27,
8]. Further, the ACNet [25] proposed a three-stream backbone to process RGB,
Depth and Fusion features independently. Moreover, some works have considered
the variability of RGB-D features and used asymmetric backbones for their
feature extraction. For example, TSNet [39] used ResNet [31] for RGB feature
extraction and VGG (without residual connections) [38] for Depth feature
extraction. The PSCNet [12] also used an asymmetric backbone and reduced the
redundant computational cost by cutting the parameters of the Depth branch.

RGB-D feature fusion. The design of RGB-D backbones for feature extraction has
been mostly based on two or three stream architectures [8]. However, the
performance of different frameworks depends largely on how they fuse RGB and
Depth features. Some early works, such as FuseNet[36] and RedNet [37], fused
RGB and Depth feature maps pixel-wise in the backbone. Later works, such as
ACNet [25], ESANet [9] and EMSANet[11], proposed channel attention to select
features from different channels, as RGB and Depth feature maps may not align
well on the corresponding channels. PSCNet [12] further extended channel
attention to both spatial and channel directions and achieved better
performance. Recently, more complex models have been proposed to exploit
cross-modal information and select features for RGB-D fusion. For example,
SAGate [27] proposed a gated attention mechanism that can leverage cross-modal
information for feature selection. CANet [28] extended non-local attention
[20] to cross-modal semantic information and achieved significant improvement.
CMX [8] extended SA-Gate to spatial and channel directions and proposed a
novel cross-modal attention with global receptive field, which can enhance
cross-modal complementarity using the similarity of two modalities. However,
integrating cross-modal information and learning cross-modal similarity is
still an open question in vision tasks.

##  III Method

Figure 1: Overview of AsymFormer.

###  III-A Framework Overview

This paper proposes AsymFormer, a novel network for RGB-D integration semantic
segmentation. The network employs a ConvNext [16] based backbone to extract
features from RGB images and a Mix-Transformer [17] based backbone to process
RGB-D fused features. We introduced the LAFS module for feature selection,
utilizing learnable feature weights to calculate spatial attention weights. We
integrated modeling of self-similarity in multi-modal features, validating its
capability to enhance network accuracy with minimal additional model
parameters. The overall framework of AsymFormer is illustrated in Figure 1.

###  III-B Local Attention-Guided Feature Selection

Attention mechanisms have been shown to be effective in selecting
complementary features from RGB and Depth features, thus improving efficiency
and performance[25, 1, 12, 8]. Current spatial attention methods extract
global information using pixel maximum or average values, neglecting the
differences in channels from multiply modalities. In this work, we extend the
previous spatial attention by proposing a more flexible spatial attention
mechanism. The core design of LAFS is a trainable spatial attention weight.
This weight is learned through an MLP in a similar manner to SE [34].

In specific, Figure 1 (b)-1 illustrates the details of LAFS. The LAFS receives
input features of the concatenation of RGB feature $RGB$ with dimension
$(C_{1},H,W)$ and current layer Depth feature $Depth$ with dimension
$(C_{2},H,W)$.

For the computation of the attention weight, we first extract global
information vectors $Avg$ of each feature map by adaptive AvgPool. For the
channel attention, we process the global information by a feed-forward network
with a squeeze-and-excitation structure.

$W_{C}=\text{Sigmoid}(\text{FFN}(Avg))$ (1)

For the spatial attention, we process the global information $Avg$ by another
feed-forward network with a squeeze-and-excitation structure. The output
vector is split into $R_{Max}$ and $R_{Avg}$, which represent two different
descriptions of pixel spatial-similarity.

$R_{Avg}=\text{FFN}(Avg)$ (2)

Furthermore, the global spatial information $I_{S}$ of feature map can be
extracted by calculating the inner product similarity from $R_{Avg}$ and the
$Input$ (In computation, this is equivalent to the weighted sum of the
features from each channel). Then, the spatial attention weight $W_{S}$ is
computed by sigmoid normalization.

$W_{S}=\text{Sigmoid}(\frac{\text{Dot}(Input.\text{Reshape}(C,H\times
W)^{T},R_{Avg})}{C^{2}})$ (3)

Where, the results all divide $\sqrt{C}$ for preventing sigmoid overflow.
Finally, the input features are selected by weight in spatial$W_{S}$ and
channel $W_{C}$ direction:

$Output=W_{C}\times W_{S}\times Input$ (4)

###  III-C Cross Modal Attention-Guided Feature Correlation Embedding

In this paper, we introduce a novel cross-modal attention (CMA) module that
incorporates cross-modal self-similarity information into pixel-wise fused
features. Current MHSA [21] is limited to learning the self-similarity of a
single modality, whereas in multi-modal processing, our goal is to jointly use
information from multiple modalities for representation learning. To achieve
this, CMA is proposed to learn the self-similarity of multiple modalities. The
key insight of CMA is independently learning the self-similarity of two
modalities and embeds the sum of the results into the fusion feature, which is
different than standard MHSA in theoretical. Specifically, for each pixel
$(i,j)$ in feature map, we separately embed its RGB and Depth feature:

$\displaystyle Key_{RGB}=(Kr_{1,i,j},Kr_{2,i,j},...,Kr_{N,i,j})$ (5)
$\displaystyle Query_{RGB}=(Qr_{1,i,j},Qr_{2,i,j},...,Qr_{N,i,j})$
$\displaystyle Key_{Depth}=(Kd_{1,i,j},Kd_{2,i,j},...,Kd_{N,i,j})$
$\displaystyle Query_{Depth}=(Qd_{1,i,j},Qd_{2,i,j},...,Kd_{N,i,j})$

The cross-modal similarity of pixel $(i_{0},j_{0})$ and other pixel in feature
map are defined as:

$W(i,j)=\sum_{n=1}^{N}(Kr_{n,i,j}\cdot
Qr_{n,i_{0},j_{0}})+\sum_{n=1}^{N}(Kd_{n,i,j}\cdot Qd_{n,i_{0},j_{0}})\\\ $
(6)

Moreover, the CMA embeds cross-modal similarity into the fused feature $Value$
with same operation as[21, 20, 22]. Following above idea, the CMA first
embedding $Key$, $Query$ and $Value$ independently for three modalities
features. Then, we extend self-attention to the multimodal case, which contain
two design: 1. Information mixing and redistribution of $Key$ and $Query$; 2.
Learning two different different representation subspaces for the same
position. The overall framework of CMA is demonstrated in Figure

Independent linear embedding: According to our definition, the $Key$, $Query$
and $Value$ should be embedded independently. Assuming the input $RGB$ feature
has dimension ${}^{(}C_{0},H,W)$; the Enhanced Depth feature $Depth_{En.}$ has
dimension ${}^{(}C_{1},H,W)$; and the previous fused feature $Fus_{1}$ has
dimension $(C_{2},H,W)$. We first normalize the RGB feature and Depth feature.
Then, we use six independent convolution layers to generate self-attention
keys and queries from RGB and Depth feature:

$\displaystyle
RGB_{emb}=\left\\{\begin{array}[]{c}K_{RGB}=\text{Conv}_{f\times f}(RGB)\\\
Q_{RGB}=\text{Conv}_{1\times 1}(RGB)\\\ \end{array}\right.$ (7) $\displaystyle
Depth_{emb}=\left\\{\begin{array}[]{c}K_{Depth}=\text{Conv}_{f\times
f}(Depth_{En.})\\\ Q_{Depth}=\text{Conv}_{1\times 1}(Depth_{En.})\\\
\end{array}\right.$ (8) $\displaystyle
Fused_{emb}=\left\\{\begin{array}[]{c}V=\text{Conv}_{f\times f}(Fused_{1})\\\
V_{1},V2=\text{Split}(V,2)\\\ \end{array}\right.$ (9)

The dimensions of $K_{RGB}$ and $K_{Depth}$ are
$(\frac{C_{1}}{4},\frac{H}{f},\frac{W}{f})$, where $f$ is the down-sample rate
and equals the convolution kernel-size and stride. This reduces the feature
map resolution to $\frac{1}{f}$. The dimensions of $Q_{RGB}$ and $Q_{Depth}$
are $(\frac{C_{1}}{4},H,W)$. The $V_{1}$ and $V_{2}$ is reshaped into
dimension $(\frac{C_{2}}{4},\frac{H\times W}{f^{2}})$.

Information mixing and redistribution: We concatenate the cross-modal keys and
queries to obtain the $Key$ and $Query$:

$\displaystyle
Key,Query=\left\\{\begin{array}[]{c}Key=\text{Cat}[K_{RGB},K_{Depth}]\\\
Query=\text{Cat}[Q_{RGB},Q_{Depth}]\\\ \end{array}\right.$ (10)

The first $\frac{C_{2}}{4}$ channels of $Key$ and $Query$ contain RGB
information. Similarly, the second half of the $Key$ and $Query$ channels
contain Depth information. To obtain the $K1$, $K2$ and $Q1$, $Q2$ for
integrating RGB-D information computing the two feature subspaces $W_{1}$,
$W_{2}$, we perform a channel shuffle of $Key$ and $Query$ and split them into
required vector:

$\displaystyle K_{1},K_{2}=\text{Split}(\text{Shuffle}(Key),2)$ (11)
$\displaystyle Q_{1},Q_{2}=\text{Split}(\text{Shuffle}(Query),2)$

Representation subspaces learning: The $Q_{1}$, $Q_{2}$ are reshaped into
dimension $(\frac{C_{1}}{4},H\times W)$ and the $K_{1}$, $k_{2}$ are reshaped
into dimension $(\frac{C_{1}}{4},\frac{H\times W}{f^{2}})$. Then, the two
representation subspaces $W_{1}$ and $W_{2}$ are computed by dot product of
$K$ and $Q$ respectively:

$\displaystyle W_{1}=\text{Softmax}(\frac{Q_{1}\cdot
K_{1}^{T}}{\sqrt{C_{1}/4}})$ (12) $\displaystyle
W_{2}=\text{Softmax}(\frac{Q_{2}\cdot K_{2}^{T}}{\sqrt{C_{1}/4}})$

The $W_{1}$ and $W_{2}$ are embedded into $V_{1}$ and $V_{2}$ by dot product.
The fused feature $Fused_{2}$ is the concatenation of $V_{1}$ and $V_{2}$:

$Fused_{2}=\text{Cat}[W_{1}\cdot V_{1},W_{2}\cdot V_{2}]\\\ $ (13)

Finally, the $Fused_{2}$ is adapted into output channel $C_{2}$ and added with
residual connection $Fused_{1}$.

$Fused_{2}=\text{ConvBN}_{1\times 1}(Fused_{2})+Fuesed_{1}$ (14)

TABLE I: Ablation experiment results for different multi-modal feature fusion method. Model | Feature Fusion Method | Metric
---|---|---
Cat | SE+MHSA-H2 | LAFS | CMA-H2 | Params/M | mIoU(%) | Inf.Speed(FPS)
Baseline | ✓ |  |  |  | 31.9 | 47.0 | 77.5
| ✓ |  |  | 32.6(+0.7M) | 49.9 (+2.9%) | 65.7
Ours |  |  | ✓ |  | 32.4 (+0.5M) | 49.1 (+2.1%) | 75.7
|  |  | ✓ | 32.5 (+0.6M) | 49.6 (+2.6%) | 67.4
|  | ✓ | ✓ | 33.0 (+1.1M) | 52.0 (+5.0%) | 65.5

##  IV EXPERIMENT RESULTS AND ANALYSIS

###  IV-A Implementation Details

To evaluate our Real-Time semantic segmentation network design, we conduct a
series of experiments on two widely-used datasets: NYUv2[5] (795 training and
654 testing RGB-D images) and SUNRGBD[6] (5825 training and 5050 testing RGB-D
images).

We conduct the model training and testing on different platforms. For the
training, we use Nvidia A100-40G GPU. For the evaluation and inference speed
testing, we use Nvidia RTX 3090 GPU, Ubuntu 20.04, CUDA 12.0 and Pytorch
2.0.1. We apply data augmentation to all datasets by randomly flipping
(p=0.5), srandom scales [1.0,2.0], random crop 480$\times$640 and random HSV.
We adopt Mix Transformer[17] and ConvNext[16] backbones pretrained on
ImageNet-1k[24]. In the augmented setting, both backbones are pretrained on
ADE20k[23]. For the CMA implementation, we set down-sample rate $f$=1 in CMA-1
and $f$=2 in CMA-2,3. The MLP-decoder in AsymFormer has the same structure as
Segformer and an embedding dimension of 256. We choose AdamW optimizer with a
weight decay of 0.01. The initial learning rate is $5e^{-5}$ and we use a poly
learning rate schedule $(1-\frac{iter}{max_{i}ter})^{0.9r}$ with a warm-up of
10 epochs. We train with a batch size of 8 for NYUv2 (500 epochs) and SUNRGBD
(200 epochs). We employ cross-entropy as the loss function and do not use any
auxiliary loss during training process. The evaluation metric is mean
Intersection over Union (mIoU).

###  IV-B Model Assessment

This paper proposes two different models based on the AsymFormer framework
with different backbones. The AsymFormer uses ConvNext-T[16] for RGB
representation learning and MiT-B0[17] for fused feature processing. This
model has 33.0 million parameters and 36.0 GFLOPs computational cost, and it
can achieve 65 FPS inference speed on RTX 3090. Particularly, with mixed
precision quantization, AsymFormer achieves a real-time inference speed of 79
FPS on the RTX 3090 GPU. It also achieves a real-time inference speed of 29.4
FPS on the Tesla T4 GPU, which has a more limited computation resources (65
TFLOPs FP16).

###  IV-C Ablation Experiment

We conduct a series of ablation experiments on NYUv2 dataset to evaluate the
effectiveness of the LAFS and CMA module. We set two common feature fusion
methods as our comparative baseline: 1\. Cat: This method directly
concatenates two features and then uses convolution layers to adjust the
channel numbers. Essentially, it is a pixel-wise fusion without feature
selection. 2\. SE+MHSA-H2: This method combines the popular SE attention [34]
and MHSA attention [21] for feature fusion. Here, SE is used for feature
selection in the channel direction, while MHSA is employed for further feature
extraction on the fused features. Specifically, we use an MHSA with 2 heads to
align with CMA-H2.

In our experiments, the Cat fusion method, used as a baseline, achieved a
segmentation accuracy of 47.0 mIoU and an inference speed of 77.5 FPS. When
using LAFS for feature selection alone, we achieved a performance improvement
of 2.1% while sacrificing only 1.8 FPS of inference speed. This demonstrates
that LAFS is a highly efficient feature selection method that provides
performance gains without significantly impacting inference speed. In
comparison to the other baseline, Cat+MHSA-H2, which resulted in an 11.8 FPS
reduction in inference speed, only a 2.9% improvement in segmentation accuracy
was achieved. This further highlights the efficiency of LAFS.

Furthermore, we conducted experiments to investigate whether incorporating the
mining of self-similarity within the two modalities after feature selection
could further improve segmentation accuracy. When using CMA-H2 alone, we
observed a 2.6% improvement in segmentation accuracy but encountered a
significant decrease in inference speed of 10.1 FPS. Compared to LAFS, CMA
showed a more noticeable reduction in inference speed. In comparison to
SE+MHSA-H2, CMA-H2 achieved similar accuracy, but it is important to note that
no feature selection was performed in this case, indicating that relying
solely on mining multi-modal self-similarity can achieve accuracy similar to
the traditional feature fusion baseline SE+MHSA.

Finally, we combined LAFS with CMA (LAFS+CMA). Since LAFS had minimal impact
on inference speed and served a different purpose than CMA, the network’s
inference speed only decreased by 2 FPS while achieving a significant
improvement of 5.0% compared to the baseline Cat. At this point, the inference
speed of LAFS+CMA was similar to SE+MHSA, but with a 2.1% performance
improvement. This validates our experimental hypothesis: by re-modeling
feature selection and mining cross-modal self-similarity, we can enhance the
segmentation performance of the network without sacrificing inference speed
compared to existing models. This demonstrates that we have indeed improved
the efficiency of the network.

###  IV-D Comparison With State-of-The-Arts

####  IV-D1 NYUv2 Comparison Results

Table II presents the results of our network on the NYUv2 dataset with
different backbone sizes and pretraining settings. Even without ImageNet-1k
pretraining, which other methods use for a fair comparison, our method still
achieves leading scores. Our Real-Time efficient method, AsymFormer achieves
52.0 $\%$ mIoU, demonstrating competitive accuracy compared to those high-
performance heavy designs. AsymFormer also has faster inference speed than
other methods. For instance, AsymFormer outperforms PSCNet-T[12] by 6.6% mIoU
and about 30% inference speed improvement. Similarly, AsymFormer is two times
faster than ESANet[9] with the same parameters and performance. In particular,
AsymFormer has the two times smaller parameters than CMX-B2[8], but has about
three times faster than CMX-B2 with a little performance degradation. In terms
of semantic segmentation accuracy, AsymFormer does not show a significant
disadvantage compared to the methods included in the comparison. Additionally,
it demonstrates a substantial lead over other methods in terms of inference
speed. This validates the effectiveness of our various efforts in reducing
network redundancy parameters and improving inference speed.

TABLE II: Comparison Results on NYUv2. * denotes ADE20k pretrain. The inference speed is tested on RTX 3090 platform, (480 $\times$ 640) inputs. Method | Year | Backbone | Params/M | Pixel Acc. (%) | mIoU (%) | Inf Speed/FPS | Inf Speed (%)
---|---|---|---|---|---|---|---
PSCNet-T | 2022 | Res50 | 40 | - | 45.4 | 47 | 72.3%
PSCNet-L | 2022 | Res50 | 52 | - | 46.2 | 30 | 46.2%
ACNet | 2019 | Res50 | 118 | - | 48.3 | 28 | 43.1%
RDFNet | 2017 | Res152 | 450 | 76 | 49.1 | 15 | 23.1%
SA-Gate | 2021 | Res50 | 65 | 76.8 | 50.2 | 35 | 53.8%
ESANet | 2022 | Res34-Nbt1D | 34 | - | 51.6 | 32 | 49.2%
MSFNet | 2022 | Res50 | 49 | - | 52.2 | 25 | 38.5%
FRNet | 2022 | Res34 | 86 | 77.6 | 53.6 | 39 | 60.0%
PGDENet | 2022 | Res34 | 101 | 78.1 | 53.7 | 32 | 49.2%
CMX-B2 | 2022 | Segformer-B2 | 67 | 78.7 | 54.1 | 24 | 36.9%
AsymFormer | 2023 | B0+T | 33 | 77.0 | 52.0 | 65 | 100.0%
AsymFormer (FP16) | 2023 | B0+T | 33 | 77.0 | 52.0 | 79 | 121.5%

####  IV-D2 SUNRGBD Comparison Results

Table III reports the performance of AsymFormer on the SUNRGBD dataset.
AsymFormer achieves competitive accuracy with 49.1 % mIoU. Since the input
image size is the same as NYUv2, all methods have the same inference speed as
shown in Table II. However, the advantage of AsymFormer is not as significant
as in NYUv2 experiment. For example, AsymFormer improves 1.8 mIoU over SA-
Gate[27] in NYUv2 dataset (52.0% vs 50.2% mIoU), but decreases 0.3 mIoU in
SUNRGBD dataset (49.1% vs 49.4% mIoU). A similar performance degradation can
be observed in CMX-B2 result which also uses Transformer based backbone. We
conjecture that this phenomenon may be caused by low quality depth images in
SUNRGBD dataset. The aim of our research is not to construct a state-of-the-
art method that has a marginal mIoU improvement over other methods, but to
construct a method that has a better performance-speed balance and is more
suitable for robot platform. Given that AsymFormer still has faster inference
speed than other methods, we consider this performance acceptable for
AsymFormer.

TABLE III: Comparison Results on SUNRGBD. # denotes multi-scale test. Method | Pixel Acc. (%) | mIoU (%)
---|---|---
RDFNet[26] | 81.5 | 47.7
ESANet[9] | - | 48.0
ACNet[25] | - | 48.1
SA-Gate[27] | 82.5 | 49.4
CMX-B2[8] | 82.8 | 49.7
MSFNet[30] | - | 50.3
FRNet[10] | 87.4 | 51.8
PGDENet[29] | 87.7 | 51.0
AsymFormer | 81.9 | 49.1

###  IV-E Visualization

1) LAFS Attention Map: As shown in Figure 2, to demonstrate that LAFS performs
better than CBAM [18] in selecting features in the spatial dimension, we
visualized the spatial attention weights of both methods. It can be observed
that LAFS provides better coverage of informative regions in the image while
maintaining consistency within objects and preserving the integrity of edges.

Figure 2: Difference between CBAM and LAFS’s spatial attention map.

2) Semantic Segmentation Results: The Figure 3 demonstrates the segmentation
results of AsymFormer on the NYUv2 dataset. As observed, while maintaining a
significantly faster inference speed compared to other methods, AsymFormer
achieves comparable semantic segmentation accuracy to mainstream approaches.

Figure 3: Visualization of AsymFormer Semantic Segmentation Results.

##  V CONCLUSIONS

In this work, we proposed AsymFormer, which aims to construct a less-redundant
real-time indoor scene understanding system. To enhance efficiency and reduce
redundant parameters, we implemented the following improvement: 1. We employed
an asymmetric backbone that compressed the parameters of the Depth feature
extraction branch, thus reducing redundancy. 2. We introduced the LAFS module
for feature selection, utilizing learnable feature weights to calculate
spatial attention weights. This resulted in a significant performance
improvement while almost maintaining the inference speed. 3. Expanding on
feature selection, we integrated modeling of self-similarity in multi-modal
features, validating its capability to enhance network accuracy with minimal
additional model parameters. The experiments demonstrated that the AsymFormer
surpassed existing SOTA methods in terms of speed, demonstrating our method
achieved a balance between accuracy and speed. Moving forward, we will
continue to optimize the modules and address issues such as self-supervised
pre-training of the model, aiming for further improvements.

## References

  * [1] C. Yu, C. Gao, J. Wang, G. Yu, C. Shen, and N. Sang, “Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation,” _International Journal of Computer Vision_ , vol. 129, pp. 3051–3068, 2021\.
  * [2] C. Yu, J. Wang, C. Peng, C. Gao, G. Yu, and N. Sang, “Bisenet: Bilateral segmentation network for real-time semantic segmentation,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 325–341.
  * [3] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2881–2890.
  * [4] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, “Rethinking atrous convolution for semantic image segmentation,” _arXiv preprint arXiv:1706.05587_ , 2017.
  * [5] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images.” _ECCV (5)_ , vol. 7576, pp. 746–760, 2012.
  * [6] S. Song, S. P. Lichtenberg, and J. Xiao, “Sun rgb-d: A rgb-d scene understanding benchmark suite,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 567–576.
  * [7] J. Cao, H. Leng, D. Lischinski, D. Cohen-Or, C. Tu, and Y. Li, “Shapeconv: Shape-aware convolutional layer for indoor rgb-d semantic segmentation,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 7088–7097.
  * [8] H. Liu, J. Zhang, K. Yang, X. Hu, and R. Stiefelhagen, “Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers,” _arXiv preprint arXiv:2203.04838_ , 2022.
  * [9] D. Seichter, M. Köhler, B. Lewandowski, T. Wengefeld, and H.-M. Gross, “Efficient rgb-d semantic segmentation for indoor scene analysis,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 13 525–13 531.
  * [10] W. Zhou, E. Yang, J. Lei, and L. Yu, “Frnet: Feature reconstruction network for rgb-d indoor scene parsing,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 16, no. 4, pp. 677–687, 2022.
  * [11] D. Seichter, S. B. Fischedick, M. Köhler, and H.-M. Groß, “Efficient multi-task rgb-d scene analysis for indoor environments,” in _2022 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2022, pp. 1–10.
  * [12] S. Du, S. Tang, W. Wang, X. Li, Y. Lu, and R. Guo, “Pscnet: Efficient rgb-d semantic segmentation parallel network based on spatial and channel attention.” _ISPRS Annals of Photogrammetry, Remote Sensing & Spatial Information Sciences_, no. 1, 2022.
  * [13] V. Nekrasov, C. Shen, and I. Reid, “Light-weight refinenet for real-time semantic segmentation,” _arXiv preprint arXiv:1810.03272_ , 2018.
  * [14] P. Hu, F. Caba, O. Wang, Z. Lin, S. Sclaroff, and F. Perazzi, “Temporally distributed networks for fast video semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 8818–8827.
  * [15] V. Nekrasov, T. Dharmasiri, A. Spek, T. Drummond, C. Shen, and I. Reid, “Real-time joint semantic segmentation and depth estimation using asymmetric annotations,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 7101–7107.
  * [16] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 11 976–11 986.
  * [17] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 12 077–12 090, 2021.
  * [18] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “Cbam: Convolutional block attention module,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 3–19.
  * [19] Q. Hou, D. Zhou, and J. Feng, “Coordinate attention for efficient mobile network design,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 13 713–13 722.
  * [20] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7794–7803.
  * [21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [22] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [23] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba, “Scene parsing through ade20k dataset,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 633–641.
  * [24] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [25] X. Hu, K. Yang, L. Fei, and K. Wang, “Acnet: Attention based network to exploit complementary features for rgbd semantic segmentation,” in _2019 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2019, pp. 1440–1444.
  * [26] S.-J. Park, K.-S. Hong, and S. Lee, “Rdfnet: Rgb-d multi-level residual feature fusion for indoor semantic segmentation,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 4980–4989.
  * [27] X. Chen, K.-Y. Lin, J. Wang, W. Wu, C. Qian, H. Li, and G. Zeng, “Bi-directional cross-modality feature propagation with separation-and-aggregation gate for rgb-d semantic segmentation,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI_. Springer, 2020, pp. 561–577.
  * [28] H. Zhou, L. Qi, H. Huang, X. Yang, Z. Wan, and X. Wen, “Canet: Co-attention network for rgb-d semantic segmentation,” _Pattern Recognition_ , vol. 124, p. 108468, 2022.
  * [29] W. Zhou, E. Yang, J. Lei, J. Wan, and L. Yu, “Pgdenet: Progressive guided fusion and depth enhancement network for rgb-d indoor scene parsing,” _IEEE Transactions on Multimedia_ , 2022.
  * [30] S. Jiang, Y. Xu, D. Li, and R. Fan, “Multi-scale fusion for rgb-d indoor semantic segmentation,” _Scientific Reports_ , vol. 12, no. 1, p. 20305, 2022\.
  * [31] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [32] X. Ding, Y. Guo, G. Ding, and J. Han, “Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 1911–1920.
  * [33] W. Yu, M. Luo, P. Zhou, C. Si, Y. Zhou, X. Wang, J. Feng, and S. Yan, “Metaformer is actually what you need for vision,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 10 819–10 829.
  * [34] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.
  * [35] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3431–3440.
  * [36] C. Hazirbas, L. Ma, C. Domokos, and D. Cremers, “Fusenet: Incorporating depth into semantic segmentation via fusion-based cnn architecture,” in _Computer Vision–ACCV 2016: 13th Asian Conference on Computer Vision, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part I 13_. Springer, 2017, pp. 213–228.
  * [37] J. Jiang, L. Zheng, F. Luo, and Z. Zhang, “Rednet: Residual encoder-decoder network for indoor rgb-d semantic segmentation,” _arXiv preprint arXiv:1806.01054_ , 2018.
  * [38] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [39] W. Zhou, J. Yuan, J. Lei, and T. Luo, “Tsnet: Three-stream self-attention network for rgb-d indoor semantic segmentation,” _IEEE intelligent systems_ , vol. 36, no. 4, pp. 73–78, 2020.
  * [40] J. Li, X. Xia, W. Li, H. Li, X. Wang, X. Xiao, R. Wang, M. Zheng, and X. Pan, “Next-vit: Next generation vision transformer for efficient deployment in realistic industrial scenarios,” _arXiv preprint arXiv:2207.05501_ , 2022\.