# Collaborative Auto-encoding for Blind Image Quality Assessment

###### Abstract

Blind image quality assessment (BIQA) is a challenging problem with important
real-world applications. Recent efforts attempting to exploit powerful
representations by deep neural networks (DNN) are hindered by the lack of
subjectively annotated data. This paper presents a novel BIQA method which
overcomes this fundamental obstacle. Specifically, we design a pair of
collaborative autoencoders (COAE) consisting of a content autoencoder (CAE)
and a distortion autoencoder (DAE) that work together to extract content and
distortion representations, which are shown to be highly descriptive of image
quality. While the CAE follows a standard codec procedure, we introduce the
CAE-encoded feature as an extra input to the DAE’s decoder for reconstructing
distorted images, thus effectively forcing DAE’s encoder to extract distortion
representations. The self-supervised learning framework allows the COAE
including two feature extractors to be trained by almost unlimited amount of
data, thus leaving limited samples with annotations to finetune a BIQA model.
We will show that the proposed BIQA method achieves state-of-the-art
performance and has superior generalization capability over other learning
based models. The codes are available at: https://github.com/Macro-Zhou/NRIQA-
VISOR/.

Index Terms—  blind image quality assessment, autoencoder, self-supervised
learning

##  1 Introduction

Image quality assessment (IQA) is an indispensable tool in the modern digital
media era. IQA models can be divided as full reference (FR), reduced reference
(RR), and no reference (NR) according to whether full, partial, and no
reference image is available. Compared with FR IQA and RR IQA, NR IQA, also
known as blind image quality assessment (BIQA), is much more challenging and
potentially more useful.

In BIQA, the key is to extract visual quality-related features. Many
traditional methods [1, 3, 2] extract such features based on natural scene
statistics (NSS). However, due to the complexity of distortion and diversity
of image content, these hand-crafted features have limited visual quality
representation capabilities. Recently, deep neural networks (DNNs) have been
employed in BIQA. However, the problem of insufficient annotated data, e.g.,
not enough images with mean opinion scores (MOS), is hindering the development
of deep learning-based BIQA solutions. To address the problem, one approach is
to develop end-to-end models using various learning strategies including
patch-wise learning [4], multi-task learning [5], transfer learning [22],
meta-learning [6], and so on. Despite progress, most existing DNN-based BIQA
methods generally suffer from the “blackbox” problem where it is often unknown
what features are extracted and what role a feature plays. Another solution is
to use supervised learning to extract relevant features through: (i) manually
designing different learning targets such as pseudo MOS / similarity maps [7]
and ranking labels [8], or (ii) performing quality-related tasks such as image
restoration [21] and primary content prediction [20]. However, their
effectiveness heavily depends on the designers’ experience. Therefore, the
problem of lacking annotated data remains a fundamental obstacle to developing
deep learning-based BIQA.

In this paper, we tackle these issues by designing an autoencoder (AE) system
through self-supervised learning, thus overcoming the difficulty of having to
annotate large amount of training data. Using self-supervised learning means
that there is almost unlimited amount of data available for training highly
representative models to extract quality-related features, thus leaving the
limited MOS annotated samples to train a light-weight quality predictor. The
core of our novel BIQA method is the introduction of a COllaborative
AutoEncoder (COAE) to extract image content and distortion representations.
The COAE consists of a content autoencoder (CAE) and a distortion autoencoder
(DAE) working collaboratively and individually. While the CAE is a standard AE
trained with high quality undistorted images for extracting content-related
representations, the design of DAE is unique and key to the success of the new
BIQA model. By introducing the CAE-encoded content feature as an external
input to the decoder of the DAE and requiring the DAE’s decoder to reconstruct
distorted input images, the DAE’s encoder is forced to extract distortion-
aware features. As image quality is not only related to distortion but is also
content dependent, the content features from the CAE and the distortion
features from the DAE are together sent to a quality score regressor to assess
the input image’s quality without using a reference.

Fig. 1: The proposed collaborative autoencoder.

The main contributions of this paper can be summarized as follows:

(1) We propose a self-supervised learning strategy to extract quality-related
features for developing deep learning-based BIQA models, thus overcoming the
fundamental problem of lacking sufficient annotated data.

(2) We introduce an innovative collaborative autoencoder (COAE) architecture
and design a strategy to enable a pair of autoencoders to work collaboratively
and individually to extract content feature and distortion-aware feature
respectively. We will show that these features are highly descriptive of image
quality.

(3) We have developed a BIQA method termed Visual qualIty aSsessment based on
cOllaborative autoencodeRs (VISOR) which uses the content feature and the
distortion-aware feature of the COAE to construct a blind image quality
predictor. We will show that VISOR achieves state-of-the-art performance and
has superior generalization capability over other learning based models.

##  2 Collaborative Autoencoder

Image quality mainly depends on two attributes: the image content and
distortions [23]. However, when taking image as a signal, the content
information mostly dominants the signal energy, while the distortion
information is much smaller and regarded as disturbance. Therefore, when
training a single model in a normal self-supervised way (e.g., an AE), it
tends to encode the main content information even if it is trained on
distorted images. When using such features for quality assessment, the
distortion information is usually difficult to make enough contributions. To
settle that, a novel self-supervised architecture, termed as collaborative
autoencoder (COAE), is introduced to separately extract content- and
distortion-related features, as shown in Fig. 1. The new architecture consists
of two AEs, the CAE and the DAE. The CAE is trained as a standard AE using
pristine images, aiming at extracting main information of an image, i.e., the
content features. However, when training a DAE, our idea is to externally
injecting its decoder with sufficient content information to force its encoder
to concentrate on image distortions. If trained on distorted images, the
mission of the decoder is to better reconstruct the distorted input images.
With the content information from CAE, the DAE’s encoder only needs to focus
on the distortion information to enable the DAE’s decoder to make accurate
reconstruction. In this way, the DAE can effectively learn the distortion
representation. In the following subsections, we detail the architecture of
COAE.

###  2.1 The Content Autoencoder

It has been shown in many vision tasks that the AEs can extract content
features of the input images in a self-supervised way. Thus, it is reasonable
to use a standard AE to extract main content information. Specifically, the
CAE consists of an encoder $\mathbb{E}_{c}$ employed to extract content
features $\textbf{F}_{c}$ from the input image $\textbf{I}_{cin}$, and a
decoder $\mathbb{D}_{c}$ used to reproduce the input image from
$\textbf{F}_{c}$. As shown in the Fig. 1(a) and (b), the architecture of
$\mathbb{E}_{c}$ has three convolutional layers with the appliance of batch
normalization and ReLU activation, which are subsequently followed by four
residual blocks. The architecture of $\mathbb{D}_{c}$ is dual with
$\mathbb{E}_{c}$: four residual blocks are followed by three deconvolutional
layers. The feature channel of $\textbf{F}_{c}$ is set to 256 since the image
contents are diversed.

To ensure $\textbf{F}_{c}$ will not be disturbed by distortions, only pristine
images are utilized to train the CAE using the following loss function:

$L_{overall}=L_{recon}+L_{percep},$ (1)

where $L_{recon}=\left\|\textbf{I}_{cin}-\textbf{I}_{cout}\right\|_{F}$ is the
pixel-wise loss, $\left\|.\right\|_{F}$ is the Frobenius norm, and
$L_{percep}=LPIPS(\textbf{I}_{in},\textbf{I}_{out})$ is the Learned Perceptual
Image Patch Similarity [9] between $\textbf{I}_{in}$ and $\textbf{I}_{out}$
representing the perceptual loss, which is commonly used in reconstruction
tasks [20].

###  2.2 The Distortion Autoencoder

Different from the CAE, the DAE’s decoder has an extra input, i.e., the
content features $\textbf{F}_{c}$ from the CAE’s encoder. When the input
images of the CAE and the DAE share the same image content, the
$\textbf{F}_{c}$ will contain adequate content information for DAE’s decoder
to reconstruct DAE’s input image. By this way, when the output of the DAE’s
encoder is restricted to a low dimension, the encoder will strive to abandon
most of the content information and extract distortion information to enable
the decoder to make accurate reconstruction. Specifically, a distortion
encoder $\mathbb{E}_{d}$ is employed to extract the distortion-aware features
$\textbf{f}_{d}$ from the input $\textbf{I}_{din}$. Using both
$\textbf{f}_{d}$ and $\textbf{F}_{c}$, a distortion decoder $\mathbb{D}_{d}$
is designed to reproduce $\textbf{I}_{din}$. The procedure of the DAE can be
described as:

$\textbf{I}_{dout}=\mathbb{D}_{d}(\textbf{F}_{c},\textbf{f}_{d})=\mathbb{D}_{d}(\mathbb{E}_{c}(\textbf{I}_{cin}),\mathbb{E}_{d}(\textbf{I}_{din})),$
(2)

where $\textbf{I}_{din}$ is a distorted version of $\textbf{I}_{cin}$ and
$\textbf{I}_{dout}$ is the output of the DAE.

The architecture of the distortion encoder is illustrated in Fig. 1(c). As
distortions may affect local, regional and global structures, features
extracted from multiple layers are included. Specifically, four feature maps
are extracted from four different layers. To further learn multi-scale
information, each map is fed to a Spatial Pyramid Pooling (SPP) net to
generate a fixed-length representation. The SPP net is a simple network
consisting of a convolutional layer, an SPP operation [10], and an FC layer.
After obtaining $\textbf{f}_{d}^{j}$ from multiple layers, they are
concatenated to form the distortion-aware feature:
$\textbf{f}_{d}=concat(\textbf{f}_{d}^{1},\textbf{f}_{d}^{2},\textbf{f}_{d}^{3},\textbf{f}_{d}^{4})\in\mathcal{R}^{256\times
1}$. The distortion decoder is shown in Fig. 1(d). It has two inputs, the
content feature $\textbf{F}_{c}$ and the distortion-aware feature
$\textbf{f}_{d}$. Using an effective technique similar to feature modulation
[11], we treat $\textbf{f}_{d}$ as a modulation signal to rectify
$\textbf{F}_{c}$. Specifically, $\textbf{f}_{d}$ is first fed into an FC layer
to generate an extended feature $\textbf{s}\in\mathcal{R}^{1024\times 1}$,
which is then split evenly into four features
{$\textbf{s}_{k}\in\mathcal{R}^{256\times 1},k=1,2,3,4$} before feeding into
four Sub-Modulation Residual Blocks (SMResBlocks) to modulate $\textbf{F}_{c}$
in sequence. In each SMResBlock, $\textbf{s}_{k}$ is spatially extended into
feature maps with the same width and height as $\textbf{F}_{c}$ through simple
copying. After being modulated four times with four SMResBlocks, the modulated
feature $\textbf{F}_{m}$ is obtained from $\textbf{F}_{c}$ and
$\textbf{f}_{d}$. After feature modulation, we further employ three
deconvolutional layers with the same kernel size and stride as those in the
content decoder $\mathbb{D}_{c}$ to transform $\textbf{F}_{m}$ into
$\textbf{I}_{dout}$. The same loss function $L_{overall}$ as described in Eq.
(1) is adopted to optimize the DAE’s reconstruction where $\textbf{I}_{cin}$
and $\textbf{I}_{cout}$ are respectively replaced by $\textbf{I}_{din}$ and
$\textbf{I}_{dout}$.

Fig. 2: The proposed BIQA model VISOR.

###  2.3 Training Procedure of the COAE

The training of the COAE has two stages. In the first stage, the CAE is
trained with a large number of pristine images to provide pure
$\textbf{F}_{c}$. In the second stage, the parameters of the CAE are frozen so
that the $\textbf{F}_{c}$ can be stable. Then a number of distorted images are
utilized to train the DAE, when CAE’s inputs are still pristine images that
share the same content with the distorted ones. After all the learnable
parameters in the COAE are determined, we obtain two feature extractors: the
content encoder $\mathbb{E}_{c}$ and the distortion encoder $\mathbb{E}_{d}$.
Several experiments will be conducted to demonstrate the effectiveness of
these two feature extractors.

Table 1: Performance comparisons on individual IQA datasets.

Method | LIVE | CSIQ | TID2013 | KADID-10k | LIVEC | KonIQ-10k | Weigthed Avg.
---|---|---|---|---|---|---|---
SRCC | PLCC | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC
BRISQUE | 0.939 | 0.942 | 0.750 | 0.829 | 0.573 | 0.651 | 0.422 | 0.491 | 0.607 | 0.585 | 0.624 | 0.624 | 0.552 | 0.590
NFERM | 0.941 | 0.946 | 0.769 | 0.811 | 0.584 | 0.660 | 0.502 | 0.583 | 0.536 | 0.581 | 0.590 | 0.656 | 0.569 | 0.638
ILNIQE | 0.902 | 0.865 | 0.807 | 0.808 | 0.519 | 0.640 | 0.533 | 0.571 | 0.430 | 0.510 | 0.507 | 0.523 | 0.537 | 0.574
HOSA | 0.948 | 0.949 | 0.781 | 0.842 | 0.688 | 0.764 | 0.613 | 0.651 | 0.660 | 0.680 | 0.805 | 0.813 | 0.714 | 0.743
WaDIQaM | 0.960 | 0.972 | 0.852 | 0.844 | 0.835 | 0.855 | 0.885 | 0.890 | 0.606 | 0.601 | 0.710 | 0.738 | 0.800 | 0.815
DIQA | 0.975 | 0.977 | 0.884 | 0.915 | 0.825 | 0.850 | / | / | 0.703 | 0.704 | / | / | / | /
DBCNN | 0.968 | 0.971 | 0.937 | 0.938 | 0.816 | 0.865 | 0.851 | 0.856 | 0.844 | 0.851 | 0.864 | 0.872 | 0.858 | 0.869
AIGQA | 0.960 | 0.957 | 0.927 | 0.952 | 0.871 | 0.893 | 0.864 | 0.863 | 0.751 | 0.761 | / | / | / | /
VCRNet | 0.973 | 0.974 | 0.945 | 0.955 | 0.846 | 0.875 | 0.853 | 0.849 | 0.856 | 0.865 | 0.894 | 0.909 | 0.866 | 0.876
CLRIQA | 0.977 | 0.980 | 0.915 | 0.938 | 0.837 | 0.863 | 0.837 | 0.843 | 0.832 | 0.866 | 0.831 | 0.846 | 0.841 | 0.855
VISOR | 0.973 | 0.978 | 0.961 | 0.967 | 0.905 | 0.922 | 0.928 | 0.930 | 0.846 | 0.866 | 0.896 | 0.910 | 0.911 | 0.921

##  3 The COAE based BIQA Model

Since the image quality mainly depends on two attributes: the primary content
and distortions [22, 20, 24], both $\mathbb{E}_{c}$ and $\mathbb{E}_{d}$ are
utilized to build an effective BIQA model. The overview of the COAE based BIQA
model VISOR is shown in Fig. 2. Since the CAE is trained on pristine images
only, $\textbf{F}_{c}$ from the distorted input images will be contaminated by
distortion. To reduce the effect of the distortion on $\textbf{F}_{c}$, we
first reduce the dimension of $\textbf{F}_{c}$ to output a content-aware
feature $\textbf{f}_{c}\in\mathcal{R}^{16\times 1}$ by feeding it into a
simple pooling module consisting of one convolutional layer with a 1×1 kernel
(for channel-wise pooling) and one global average pooling (GAP) layer. Then
$\textbf{f}_{c}$ and $\textbf{f}_{d}$ are concatenated as the input
$\textbf{f}_{q}\in\mathcal{R}^{272\times 1}$ to a regression network
consisting of three FC layers to simulate the interaction of the two
attributes and infer the quality score of the input image.

When training the VISOR, the parameters of $\mathbb{E}_{c}$ and
$\mathbb{E}_{d}$ are FROZEN. Therefore, only few parameters need to be learned
in the VISOR, which can be fully trained on existing IQA datasets with
subjective annotations. The VISOR is optimized by minimizing the mean square
error (MSE) between the predicted scores and the ground-truth labels.

##  4 Experiments

###  4.1 Implementation Details and Data

Implementation. The proposed COAE and VISOR is implemented in Pytorch on a
machine of 4 NVIDIA Tesla P40 GPUs. When training the COAE, all the training
images are cropped into 256×256 patches before fed into the network. When
training the VISOR, the images as well as their subjective scores in existing
IQA datasets are utilized, and the parameters of $\mathbb{E}_{c}$ and
$\mathbb{E}_{d}$ are frozen. The 224×224 patches are cropped from the input
images to fit the mini-batch training, while the full-size images can be
directly used as the inputs in the testing phases owing to the introduction of
SPP module.

Data Preparation for COAE. To fully train the COAE, we have prepared a large
amount of data. Specifically, we collect and pick out 10, 000 high-quality
images from ImageNet 111https://image-net.org/ and MSCOCO
222https://cocodataset.org/ as pristine images. Afterwards, these images are
destructed with 25 distortion types at 5 levels, which cover most distortions
that might occur in the real world. Totally, 10, 000 pristine images and 1,
250, 000 distorted images are employed to train the COAE.

IQA Datasets and Criteria. Four synthetic IQA datasets including LIVE [13],
CSIQ [14], TID2013 [12], and KADID-10K [17], and two authentic IQA datasets
including LIVEC [15] and KonIQ-10K [16], are used to evaluate the performance
of the proposed VISOR method. To evaluate the performance of the BIQA models,
two commonly-used criteria, the Spearman Rank Order Correlation Coefficient
(SRCC) and the Pearson Linear Correlation Coefficient (PLCC) are employed to
evaluate the monotonicity and linearity between the predictions and the
labels.

###  4.2 Performance Comparisons

We test the performance of VISOR on six IQA datasets and compare it with 10
classical and state-of-the-art BIQA methods. The competitors include 4
traditional BIQA methods (BRISQUE [1], NFERM [3], ILNIQE [2], and HOSA [19])
and 6 DNN-based BIQA methods (WaDIQaM [4], DIQA [7], DBCNN [22], AIGQA [20],
CLRIQA [8], and VCRNet [21]). When evaluating on each dataset, all the images
are randomly divided into 80% for training and 20% for testing [24]. For the
synthetic datasets, each dataset is divided according to the reference image
to ensure that there is no overlapping image content between the training and
testing sets. The experiments are conducted in 10 sessions and the median
SRCC, PLCC results are reported to avoid the training bias.

Table 1 lists the SRCC and PLCC results on different datasets. The best and
second-best results of SRCC and PLCC are respectively highlighted in bold and
underline. As can be seen, the proposed VISOR achieves the bes or second-best
results in 11 of 12 comparisons across six IQA datasets. Note that the VISOR
achieves three best results and one second-best result on two authentic
datasets LIVEC and KonIQ-10k, even though its feature extractors were only
trained with synthetic data. We attribute this to the implementation of
several distortion types that simulate real-world distortions (such as motion
blur, under- and over-exposure, etc.) when training the COAE, and the separate
representations of content and distortion information. The results on six
datasets and the overall weighted average performance indicate that the VISOR
shows its superiority over the other methods.

Table 2: Cross-Dataset Evaluation Train | TID2013 | KonIQ-10k
---|---|---
Test | LIVE | CSIQ | LIVE | LIVEC
BRISQUE [1] | 0.814 | 0.612 | 0.336 | 0.525
NFERM [3] | 0.634 | 0.551 | 0.502 | 0.493
WaDIQaM [4] | 0.792 | 0.690 | 0.277 | 0.646
DBCNN [22] | 0.891 | 0.807 | 0.676 | 0.749
VCRNet [21] | 0.822 | 0.712 | 0.401 | 0.616
VISOR | 0.909 | 0.870 | 0.715 | 0.754

###  4.3 Cross-dataset Evaluations

To analyze the generalization ability of the VISOR, cross-dataset evaluations
are conducted. All the compared methods are trained on one specific dataset
and tested on different datasets. The SRCC and PLCC results are listed in
Table 2. As we can see, from the overall performance of both synthetic and
authentic datasets, the generalization ability of VISOR outperforms others.

###  4.4 Ablation Studies

Here we conduct ablation studies of the VISOR on TID2013 and LIVEC. First, to
demonstrate the importance of the collaborative design of the COAE framework,
we train the CAE and the DAE independently instead of collaboratively (denoted
as S-CAE and S-DAE) with only distorted images. By this case, both of them can
only extract mixed features. We build 3 BIQA models using the S-CAE’s feature,
the S-DAE’s feature, and the concatenated S-CAE and S-DAE feature (denoted as
S-CAE(C)S-DAE), respectively, by sending them to a regression network. The
performance is listed at the top part of the Table 3. It can be seen that
without the collaboration of the two autoencoders, the performance of the
extracted features drop dramatically, indicating the importance of the
innovative design of the COAE framework.

Table 3: IQA Performance with different settings Setting | TID2013 | LIVEC
---|---|---
SRCC | PLCC | SRCC | PLCC
S-CAE | 0.766 | 0.799 | 0.719 | 0.745
S-DAE | 0.709 | 0.729 | 0.736 | 0.762
S-CAE(C)S-DAE | 0.769 | 0.801 | 0.740 | 0.765
w/o SPP in DAE | 0.787 | 0.812 | 0.713 | 0.689
w/o ML in DAE | 0.755 | 0.777 | 0.720 | 0.740
Full DAE | 0.890 | 0.896 | 0.771 | 0.781
CAE(C)DAE | 0.905 | 0.922 | 0.846 | 0.866

Second, to demonstrate the importance of the SPP nets and the multi-level
feature extraction in DAE for efficiently capturing distortion information, we
remove all the SPP nets in $\mathbb{E}_{d}$ and directly concatenate the
distortion feature maps with the content features together inside
$\mathbb{D}_{d}$ when training the COAE, and using the GAP results of the
feature maps when training the VISOR. Then we conduct another study by only
leaving the last layer output of the $\mathbb{E}_{d}$ instead of multi-level
extraction. The results are listed at the middle part of the Table 3, which
shows the necessity of the SPP nets and multi-level features extraction.
Besides, the introduction of the content features further improves the
performance, especially on authentic dataset LIVEC where the image contents
are diverse, indicating the importance of the content information to quality
assessment.

Fig. 3: t-SNE results of the $\textbf{f}_{d}$. (a) Scatter points colorized
according to the distortion types of images. (b) Scatter points colorized
according to the DMOS of images.

###  4.5 Feature Analysis

Distortion-aware Features: To analyse the effectiveness of distortion-aware
features, we use t-SNE [18] to visualize them. Specifically, $\textbf{f}_{d}$
are extracted from the distorted and pristine images in LIVE dataset, and then
t-SNE algorithm is utilized to map the features onto a 2D visualization space
as shown in Fig. 3. Fig. 3(a) shows the features for different distortions. It
is seen that almost all the images are grouped together according to their
distortion types. Fig. 3(b) shows the features according to the images’ DMOS.
It is interesting to observe that pristine images are grouped together in the
region near the origin. Besides, images with lower DMOS (higher quality) are
closer to this area, whereas those with higher DMOS (lower quality) are
further away from the origin. From the above analysis we can conclude that the
distortion-aware features extracted by the novel COAE framework can
effectively represent not only the characteristics of distortions but also the
visual quality of the images.

Content-aware Features: The content feature extracted by the CAE’s encoder
which is trained by pristine images is processed by a simple convolution and a
GAP operations as shown in Fig. 2 to force it to discard most of the
distortion information. To demonstrate the effectiveness of this scheme, the
averaged cosine similarity between the $\textbf{F}_{c}$ of distorted images
and their pristine versions for over one million samples in our COAE training
data is calculated and the result is 0.974. The same calculation is also
conducted on the LIVE, CSIQ, TID2013, and KADID-10k datasets, where the
pristine versions of distorted images are available, and the averaged result
is 0.964. The above results show that the content-aware features can indeed
effectively represent the main information of the image content and are
insensitive to distortions.

##  5 Concluding Remarks

In this work, we first propose a novel self-supervised framework based on a
collaborative autoencoder (COAE) to extract effective quality-related
features. Through employing a large-scale unlabeled data to fully train the
COAE, effective content encoder and distortion encoder are obtained and then
deployed to develop a BIQA model. Extensive experiments show the effectiveness
of the proposed feature extraction framework and the excellent performance of
the new blind image quality assessment model.

## References

  * [1] A. Mittal, A. K. Moorthy, and A. C. Bovik,  “No-reference image quality assessment in the spatial domain,”  IEEE Trans. Image Process., vol. 21, no. 12, pp. 4695–4708, 2012\.
  * [2] L. Zhang, L. Zhang, and A. C. Bovik,  “A feature-enriched completely blind image quality evaluator,”  IEEE Trans. Image Process., vol. 24, no. 8, pp. 2579–2591, 2015\.
  * [3] K. Gu, G. Zhai, X. Yang, and W. Zhang,  “Using free energy principle for blind image quality assessment,”  IEEE Trans. Multimedia, vol. 17, no. 1, pp. 50–63, 2015.
  * [4] S. Bosse, D. Maniry, K.-R. Müller, T. Wiegand, and W. Samek,  “Deep neural networks for no-reference and full-reference image quality assessment,”  IEEE Trans. Image Process., vol. 27, no. 1, pp. 206–219, 2018.
  * [5] K. Ma, W. Liu, K. Zhang, Z. Duanmu, Z. Wang, and W. Zuo,  “End-to-end blind image quality assessment using deep neural networks,”  IEEE Trans. Image Process., vol. 27, no. 3, pp. 1202–1213, 2018\.
  * [6] H. Zhu, L. Li, J. Wu, W. Dong, and G. Shi,  “Generalizable no-reference image quality assessment via deep meta-learning,”  IEEE Trans. Circuits Syst. Video Technol., vol. 32, no. 3, pp. 1048–1060, 2022.
  * [7] J. Kim, A.-D. Nguyen, and S. Lee,  “Deep cnn-based blind image quality predictor,”  IEEE Trans. Neur. Net. Learn. Syst., vol. 30, no. 1, pp. 11–24, 2019.
  * [8] F.-Z. Ou, Y.-G. Wang, J. Li, G. Zhu, and S. Kwong,  “A novel rank learning based no-reference image quality assessment method,”  IEEE Trans. Multimedia, p. In Press, 2022.
  * [9] R. Zhang, P. Isola, A. A Efros, E. Shechtman, and O. Wang,  “The unreasonable effectiveness of deep features as a perceptual metric,”  in Proc. IEEE Conf. on Comput. Vision and Pattern Recognit., 2018, pp. 586–595.
  * [10] K. He, X. Zhang, S. Ren, and J. Sun,  “Spatial pyramid pooling in deep convolutional networks for visual recognition,”  IEEE Trans. Pattern Anal. Mach. Intell., vol. 37, no. 9, pp. 1904–1916, 2015.
  * [11] H.-Y. Lee, H.-Y. Tseng, J.-B. Huang, M. Singh, and M.-H. Yang,  “Diverse image-to-image translation via disentangled representations,”  in Proc. Euro. Conf. Comput. Vision, September 2018, pp. 35–51.
  * [12] N. Ponomarenko, L. Jin, O. Ieremeiev, V. Lukin, K. Egiazarian, J. Astola, B. Vozel, K. Chehdi, M. Carli, F. Battisti, and C.-C.J. Kuo,  “Image database tid2013: Peculiarities, results and perspectives,”  Signal Process. Image Commun., vol. 30, pp. 57–77, 2015.
  * [13] H.R. Sheikh, M.F. Sabir, and A.C. Bovik,  “A statistical evaluation of recent full reference image quality assessment algorithms,”  IEEE Trans. Image Process., vol. 15, no. 11, pp. 3440–3451, 2006\.
  * [14] E. C. Larson and D. M. Chandler,  “Most apparent distortion: Full-reference image quality assessment and the role of strategy,”  Journal of Electronic Imaging, vol. 19, no. 1, pp. 011006, 2010\.
  * [15] D. Ghadiyaram and A. C. Bovik,  “Massive online crowdsourced study of subjective and objective picture quality,”  IEEE Trans. Image Process., vol. 25, no. 1, pp. 372–387, 2016.
  * [16] V. Hosu, H. Lin, T. Sziranyi, and D. Saupe,  “Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment,”  IEEE Trans. Image Process., vol. 29, pp. 4041–4056, 2020.
  * [17] H. Lin, V. Hosu, and D. Saupe,  “Deepfl-iqa: Weak supervision for deep iqa feature learning,”  arXiv preprint arXiv:2001.08113, 2020.
  * [18] L. Van der Maaten and G. Hinton,  “Visualizing data using t-sne,”  J. Mach. Learn. Research, vol. 9, no. 11, pp. 2579–2605, 2008.
  * [19] J. Xu, P. Ye, Q. Li, H. Du, Y. Liu, and D. Doermann,  “Blind image quality assessment based on high order statistics aggregation,”  IEEE Trans. Image Process., vol. 25, no. 9, pp. 4444–4457, 2016\.
  * [20] J. Ma, J. Wu, L. Li, W. Dong, X. Xie, G. Shi, and W. Lin,  “Blind image quality assessment with active inference,”  IEEE Trans. Image Process., vol. 30, pp. 3650–3663, 2021.
  * [21] Z. Pan, F. Yuan, J. Lei, Y. Fang, X. Shao, and S. Kwong,  “Vcrnet: Visual compensation restoration network for no-reference image quality assessment,”  IEEE Trans. Image Process., vol. 31, pp. 1613–1627, 2022.
  * [22] W. Zhang, K. Ma, J. Yan, D. Deng, and Z. Wang,  “Blind image quality assessment using a deep bilinear convolutional neural network,”  IEEE Trans. Circuits Syst. Video Technol., vol. 30, no. 1, pp. 36–47, 2020.
  * [23] D. Li, T. Jiang, W. Lin, and M. Jiang,  “Which has better visual quality: The clear blue sky or a blurry animal?,”  IEEE Trans. Multimedia, vol. 21, no. 5, pp. 1221–1234, 2019.
  * [24] J. Kim, H. Zeng, D. Ghadiyaram, S. Lee, L. Zhang, and A. C. Bovik,  “Deep convolutional neural models for picture-quality prediction: Challenges and solutions to data-driven image quality assessment,”  IEEE Signal Process. Mag., vol. 34, no. 6, pp. 130–141, 2017.


# Pixel Difference Convolutional Network for RGB-D Semantic Segmentation

$\text{Jun Yang}^{\dagger}$, $\text{Lizhi Bai}^{\dagger}$, $\text{Yaoru
Sun}^{*}$, $\text{Chunqi Tian}^{*}$, Maoyu Mao, Guorun Wang Jun Yang, Lizhi
Bai, Yaoru Sun, Chunqi Tian, Maoyu Mao, Guorun Wang are with Department of
Computer Science and Technology, Tongji University, Shanghai, China (e-mail:
{junyang, bailizhi, yaoru, maomy<EMAIL_ADDRESS>tianchunqi@163.com).
Jun Yang is also with the Laboratoire des Systèmes Perceptifs (UMR8248), École
normale supérieure, PSL Research University, Paris, France.

###### Abstract

RGB-D semantic segmentation can be advanced with convolutional neural networks
due to the availability of Depth data. Although objects cannot be easily
discriminated by just the 2D appearance, with the local pixel difference and
geometric patterns in Depth, they can be well separated in some cases.
Considering the fixed grid kernel structure, CNNs are limited to lack the
ability to capture detailed, fine-grained information and thus cannot achieve
accurate pixel-level semantic segmentation. To solve this problem, we propose
a Pixel Difference Convolutional Network (PDCNet) to capture detailed
intrinsic patterns by aggregating both intensity and gradient information in
the local range for Depth data and global range for RGB data, respectively.
Precisely, PDCNet consists of a Depth branch and an RGB branch. For the Depth
branch, we propose a Pixel Difference Convolution (PDC) to consider local and
detailed geometric information in Depth data via aggregating both intensity
and gradient information. For the RGB branch, we contribute a lightweight
Cascade Large Kernel (CLK) to extend PDC, namely CPDC, to enjoy global
contexts for RGB data and further boost performance. Consequently, both modal
data’s local and global pixel differences are seamlessly incorporated into
PDCNet during the information propagation process. Experiments on two
challenging benchmark datasets, $i.e.$, NYUDv2 and SUN RGB-D reveal that our
PDCNet achieves state-of-the-art performance for the semantic segmentation
task.

###### Index Terms:

Semantic segmentation, pixel-difference convolution, cascade large kernel

†† $*$ Corresponding author.
$\dagger$ Equal contribution. Figure 1: Illustration of our PDCNet. In the RGB
image, pixels A and D are the edges of pillow; B and C belong to other
categories. RGB images reflect overall spatial information. In the Depth
image, G is the junction of the bed and the pillow. While the pillow and bed
have similar visual characteristics in the RGB image, they can be
distinguished in Depth. Based on this, we propose PDC to capture local subtle
pixel differences in RGB and CPDC to focus on long-range pixel dependence for
RGB.

##  I Introduction

Semantic segmentation infers semantic labels of every pixel in a scene. With
the help of 3D sensors, RGB-D data boosts the advancement of RGB-D semantic
segmentation. Specifically, RGB image captures photometric appearance
properties in projected image space, while Depth map provides local geometry
appearance cues that are complementary to RGB images [69, 70].

However, the convolution operator that is widely adopted for consuming RGB
data might be suboptimal for processing the Depth data. For example, pillows
on a bed with similar color of bed in Fig.1. It is evident that it is not
feasible to distinguish pillows based solely on 2-dimensional features such as
colors and shapes. In contrast with RGB images, without being limited by the
similar indiscernible appearance, Depth maps have a foreground object that is
distinct from the background, which theoretically leads to better segmentation
performance, compared to RGB images only. In fact, Depth contours are cleaner
than RGB contours and tend to better describe the edge information of objects
for better separation of objects [62]. In contrast, RGB images are frequently
utilized for providing global information [14]. As a result, it is vital to
take advantage of the two complementary modalities to improve semantic
segmentation performance.

Recently, researchers make remarkable efforts on RGB-D semantic segmentation
tasks [4, 6]. They conduct fusion models which leverage RGB images and Depth
images for the better understanding of the two complementary modalities. In
some research, Depth maps are applied as an extra channel for the RGB images
in the early stage of the model [4, 5]. Similarly, the fusion method can also
be designed by fusing the features extracted from the RGB and Depth data [6,
72]. However, these fusion methods either lack the non-linear feature fusion
ability or treat both modal images equally by engaging the same contribution
from both modal data, regardless of the intrinsic distribution gap or
information loss, which is obviously not appropriate.

Furthermore, due to the fixed grid kernel structure, convolutional networks
are limited by their lack of ability to encode detailed gradient information
and therefore cannot focus on pixel difference so as to accurately achieve
pixel-level segmentation. Incorporating the geometric information from Depth
images into CNN is essential yet challenging.

To mitigate the problems mentioned above, we propose a Pixel Difference
Convolutional Network (PDCNet) for RGB-D segmentation. Two mechanisms, namely
Pixel Difference Convolution (PDC) and Cascade Large Kernel (CLK)-based PDC
(CPDC), are introduced to improve the ability of two modal data interactions
in this task.

The PDC augments the standard convolution with a pixel difference term while
aggregating local intensity and gradient information of objects in Depth
images. According to the principle of PDC, this pixel difference operation can
also solve the problem of noise in the depth map. The CLK, consisting of three
small convolutions to obtain a field of perception of $21\times 21$, has a
significantly lower computational cost than the corresponding convolution of
size 21. CLK not only involves the advantages of long-range dependence in 2D
structure, but also achieves channel-wise adaptability. After extending PDC
with CLK, the CPDC combines the advantage of capturing local structure
information of PDC and long-range dependence of CLK. The CPDC is employed for
RGB data to enjoy global contextual information. As a result, the PDC and CPDC
exploit the characteristics of the two modal data and complement each other
effectively. In summary, the contributions of this paper are fourfold as
below:

$\bullet$ We propose a PDC for considering the subtle intensity and gradient
information in local regions for Depth data, which integrates the pixel
difference operators into the convolutional operations. The principle of PDC
also shows its ability to overcome the noise of the depth map.

$\bullet$ We propose a CLK to achieve long-range receptive fields. The
combination of CLK and PDC, namely CPDC, combines both advantages to extend
PDC to enjoy the global information of RGB images.

$\bullet$ Both PDC and CPDC are seamlessly plugged and play in existing CNNs
to form a two-branch PDCNet. PDCNet takes advantage of the characteristics of
the two modal data, generating a more robust modeling capacity.

$\bullet$ Our proposed model achieves a new state-of-art performance on NYUDv2
and SUN RGB-D datasets.

##  II Related Work

###  II-A Context information Aggregation

Contextual information is commonly utilized in semantic segmentation networks
to enhance feature representation. Non-local network [14] focuses on spatial
information distribution, which guarantees that a pixel at any position
perceives contextual information from all pixels. Atrous spatial pyramid
pooling (ASPP) is proposed in Deeplabv2 [8] to enhance contextual information
utilizing different dilation convolutions. Further, DenseASPP [63] brings the
dense connections into ASPP to encode the multi-scale features. CCNet [64]
captures full-image contextual information when introducing less computation
and memory cost. PSA [65] builds an attention map that aggregates contextual
information for individual points in a specific and adaptive manner. Nie et
al. [73] propose an encoder-decoder architecture and a MRIF module to generate
high-resolution segmentation results by considering both details and boundary
information. Gao et al. [74] propose a bottleneck residual unit where the
convolutions have different dilation rates to achieve rich contextual
information.

Our PDCNet captures contextual information by aggregating both intensity and
gradient subtle pixel-wise difference/similarity in local range for Depth data
and global range for RGB data, respectively.

###  II-B RGB-D Semantic Segmentation

Recent advances in semantic segmentation benefit a lot from combining such two
complementary [19, 20, 21, 50, 22, 23]. Many works simply concatenate the
features of RGB and Depth images to enhance the semantic information of each
pixel [20, 19]. Typically, the RGB-D fusion methods are classified into early,
middle, and late fusion. As an early stage fusion method, Cao et al. [50]
firstly concatenate the RGB and Depth data and then decompose them into a
shape and a base component for extracting semantic information by individual
corresponding learnable weights. However, two types of data contain
inconsistent features and cannot be processed by shared network feature
extractors. Jiao et al. [21] apply the encoder-decoder architecture to make
full use of the both 2D appearance and 3D geometry information in late stage
by only taking one single RGB image as input. In this method, the interaction
between both modal data is insufficient and the learned embeddings are
gradually compressed and even lost. The middle stage fusion strategy
outperforms the aforementioned methods, which overcomes the drawback of early
stage and late stage fusion strategies by interacting the intermediate
information of the two different modalities. Chen et al. [54] present a cross-
modality guided encoder to fuse each pair of feature maps by the SA-Gate and
propagate to the next stage for further feature transformation. This method
reduces the distribution gap in the middle stage and achieves ample
interaction from multi-modal features. Chen et al. [53] a spatial information-
guided convolution is proposed to enhance the spatial adaptability, which
incorporates geometric information into the feature learning process by
generating spatially adaptive convolutional weights. Zhang et al. [71] propose
a coarse-to-fine decoder to generate the feature based on the transformer
mechanism, which achieves pixel-level multi-task prediction.

Our PDCNet applies a two-branch structure, where each branch focuses on
extracting modality-specific features. Specifically, the Depth branch focuses
on local geometric and illumination-independent features from Depth images by
PDC module. RGB branch extracts global color and texture information by CPDC
module. Attentive RGB and contextual features generated by the CPDC and PDC
are fused at each resolution stage in the encoders.

##  III Methodology

In this section, we first present the formulation of our proposed Pixel
Difference Convolution (PDC), then present the Cascade Large Kernel (CLK) and
the combined CPDC to implicitly improve the semantic segmentation performance.
At last, we present the Pixel Difference Convolutional Network (PDCNet) for
our task.

###  III-A Overview

The discrepancy of adjacent pixels is usually subtle and occurs in local
regions, which is not easy to be captured by vanilla convolutional networks.
To fill this gap, we propose a pixel difference convolution to capture subtle
pixel difference, which is a straightforward solution for refined RGB-D
semantic segmentation.

The existing methods either treat RGB and Depth images equally or mainly
consider the Depth data. We believe that the depth map is more responsive to
object boundary information due to the geometry characteristics, while RGB
better reflects the global image information. In our paper, the PDC is applied
to capture local information such as edges and boundaries for Depth data. On
the contrast, we propose a CLK embedded into our PDC, namely CPDC, to achieve
a long-range pixel-wise relationship for RGB images. Both PDC and CPDC apply
$depthwise$ $separable$ $convolution$ [32] structure to reduce computation and
memory.

Figure 2: The structure of PDC1 (left) and PDC2 (right). Without loss of
generality, we show the $3\times 3$ convolution kernel for the PDC1 and the
$3\times 3$ convolution kernel with dilation of 2 for the PDC2. The operation
combining PDC1 and PDC2 contributes the CPDC.

###  III-B Pixel Difference Convolution

The sense should be emphasized that the pixels with the same semantic labels
have more substantial pixel-wise similarity [76]. As shown in Fig. 3, the
pixel-wise difference can be applied to force pixels with more consistent
geometry to make more contributions to the corresponding output. This
relationship between different points is obtained by our PDC.

####  III-B1 Vanilla Convolution

An ordinary 2D convolution computes the weighted sum of the local grid. There
are two main steps in the 2D convolution: 1) sampling local receptive field
region $\mathbb{R}$ over the input feature map $\mathbf{x}$; 2) aggregating
sampled values through weighted summation. For each pixel location
$\mathbf{p}_{0}$, the output feature map $\mathbf{y}$ is:

$\mathbf{y}(\mathbf{p}_{0})=Conv(\mathbf{w},\mathbf{p}_{n})=\sum_{\mathbf{p}_{n}\in\mathbb{R}}\mathbf{w}(\mathbf{p}_{n})\cdot\mathbf{x}(\mathbf{p}_{0}+\mathbf{p}_{n}),$
(1)

where $\mathbf{w}$ is the convolution kernel, $\mathbf{p}_{0}$ indicates
current location on both input and output feature maps while $\mathbf{p}_{n}$
enumerates the locations in $\mathbb{R}$.

####  III-B2 Vanilla Convolution Meets Pixel Difference

To exploit the correlation between pixels, we introduce pixel difference into
vanilla convolution to enhance its representation and generalization capacity.
The new convolution formulation also consists of two main steps, where the
aggregation step is different from that in vanilla convolution. As illustrated
in the left part of Fig. 2, the pixel difference convolution operation, when
applied, focuses on aggregating the gradient information from surrounding
pixel points of a given pixel point. We directly apply subtraction operation
to find original pixel difference, Eq.(1) henceforth becomes:

$\begin{split}\mathbf{y}(\mathbf{p}_{0})&=Conv(\mathbf{w},\Delta\mathbf{p})\\\
&=\sum_{\mathbf{p}_{n}\in\mathbb{R}}\mathbf{w}(\mathbf{p}_{n})\cdot\left(\mathbf{x}(\mathbf{p}_{0}+\mathbf{p}_{n})-\mathbf{x}(\mathbf{p}_{0})\right).\end{split}$
(2)

Eq.(2) supports that the pixels with the same semantic labels have more
substantial pixel-wise similarity. We make use of this similarity to force
pixels with more consistent geometry to make more contributions to the
corresponding output.

Figure 3: In pixel-difference calculations, intra-class compactness and inter-
class dispersion are motivated by pulling closer pixels within a class and
pushing away pixels within another class.

####  III-B3 Details of Pixel difference convolution

It is obvious that both fine-grained gradient and intensity semantic
information are vital for RGB-D semantic segmentation task. Based on this, we
aim to combine the vanilla convolution with pixel difference convolution to
provide a more robust feature capture capacity. For this purpose, we
generalize the pixel difference convolution as:

$\begin{split}\mathbf{y}(\mathbf{p}_{0})&=\alpha\cdot
Conv(\mathbf{w},\Delta\mathbf{p})+(1-\alpha)\cdot
Conv(\mathbf{w},\mathbf{p}_{n})\\\
&=\alpha\cdot\underbrace{\sum_{\mathbf{p}_{n}\in\mathbb{R}}\mathbf{w}(\mathbf{p}_{n})\cdot\left(\mathbf{x}(\mathbf{p}_{0}+\mathbf{p}_{n})-\mathbf{x}(\mathbf{p}_{0})\right)}_{\text{pixel
difference term}}\\\
&+(1-\alpha)\cdot\underbrace{\sum_{\mathbf{p}_{n}\in\mathbb{R}}\mathbf{w}(\mathbf{p}_{n})\cdot\mathbf{x}(\mathbf{p}_{0}+\mathbf{p}_{n})}_{\text{vanilla
term}}.\end{split}$ (3)

In this context, $\alpha$ is used to adjust the contribution of fine-grained
gradient-level changes and intensity-level changes for semantic information.
The higher the $\alpha$ value, the more important of pixel differences are,
and vice versa. In this paper, we set $\alpha$ as the learnable parameter.
Additionally, we also set the $\alpha$ as default configurations, $i.e.$,
varying from 0 to 1, to show the effectiveness of pixel difference term. The
impact of $\alpha$ is further explained in the ablation studies.

We name the formulation of Pixel Difference Convolution (PDC) displayed in
Eq.(3) as PDC for short. We choose the kernel size of $5\times 5$ with
dilation of 1 for PDC to capture local fine-grained information.

####  III-B4 Implementation for PDC

To better implement PDC, we follow the form in Pytorch [66]. After rewriting
the form of Eq.(3) through simple compose and merge operation, the PDC can be
easily implemented in CNNs. Eq.(3) becomes:

$\begin{split}\mathbf{y}(\mathbf{p}_{0})=&-\alpha\cdot\underbrace{\sum_{\mathbf{p}_{n}\in\mathbb{R}}\mathbf{w}(\mathbf{p}_{n})\cdot\mathbf{x}(\mathbf{p}_{0})}_{\text{pixel
difference term}}\\\
&+\underbrace{\sum_{\mathbf{p}_{n}\in\mathbb{R}}\mathbf{w}(\mathbf{p}_{n})\cdot\mathbf{x}(\mathbf{p}_{0}+\mathbf{p}_{n})}_{\text{vanilla
term}}.\end{split}$ (4)

It should be noted that $\mathbf{w}(\mathbf{p}_{n})$ is shared between the
pixel difference term and vanilla convolution term. Compared with Eq.(3),
Eq.(4) can be easily performed. Subsequently, given an input feature map
$\mathbf{F}\in\mathbb{R}^{H\times W\times C}$, the output utilizing PDC is
generated as:

$\mathcal{O}_{\text{pdc}}=Conv_{1\times
1}\big{(}\text{PDC}(\mathbf{F})\big{)}\otimes\mathbf{F},$ (5)

where $Conv_{1\times 1}$ is a channel convolution for achieving the channel-
wise adaptability.

(a)

(b)

(c)

Figure 4: Illustration of parallel mode and cascade mode. (a) indicates
$Conv_{5\times 5}$, (b) indicates parallel mode: $Conv_{5\times
5}(\cdot)+Conv_{7\times 7}(\cdot)$ and (c) indicates cascade mode:
$Conv_{7\times 7}(Conv_{5\times 5}(\cdot)$). The color bar indicates the
number of times each pixel point in the feature map is used.

###  III-C Cascade Large Kernel

As we mentioned before, different from Depth maps, RGB images are better
suited to provide contextual information and spatial adaptability within the
global field of view. To this purpose, we propose a CLK to provide long-range
receptive fields. We do not utilize the CLK directly; instead, the proposed
CLK is the key component applied to extend the PDC for RGB data.

####  III-C1 Implementation for CLK

Our CLK consists of three convolutions, that is, a depth-wise local
convolution ($Conv_{5\times 5}$, dilation of 1), a depth-wise long-range
convolution ($Conv_{7\times 7}$, dilation of 3, and a point-wise convolution
($1\times 1$ convolution) to approximate a $21\times 21$ convolution. Under
this setting, CLK can effectively achieve both local information as well as
long-range dependence. Notably, the dilation of $7\times 7$ convolution can be
filled by the $5\times 5$ convolution, which means the CLK does not cause any
information loss compared to other long-ranged feature extraction methods
using ordinary dilated convolution, such as PPM [44] and ASPP [8].

####  III-C2 Cascade mode vs. Parallel mode

To prove the effectiveness of CLK, we compare two common modes, that is,
parallel mode and cascade mode. Since the channel convolution do not change
the receptive field, we just show the receptive field of the two modes in Fig.
4. The output feature maps generated by the two modes can be written as:

$\begin{split}&\mathcal{O}_{\text{parallel}}\rightarrow Conv_{{5\times
5}}(\mathbf{F})+Conv_{7\times 7}(\mathbf{F}),\\\
&\mathcal{O}_{\text{cascade}}\rightarrow Conv_{7\times 7}\left(Conv_{5\times
5}(\mathbf{F})\right).\end{split}$ (6)

Here, $\mathbf{F}\in\mathbb{R}^{H\times W\times C}$ is an input feature map.
Obviously, compared with the receptive field of parallel mode, cascade mode
has a large receptive field. Additionally, parallel mode loses a large portion
of local information due to the dilation. This proves that the cascade mode we
apply is better. We name the cascade mode as Cascade Large Kernel (CLK).

####  III-C3 CLK vs. large kernels

Our CLK differs from ordinary large kernel convolution in two ways. Firstly,
the CLK achieves the receptive field of $21\times 21$ utilizing a $5\times 5$
convolution, a $7\times 7$ convolution and a $1\times 1$ convolution, the
computational cost significantly less than the large kernel with size of 21.
Notably, the dilation of $7\times 7$ convolution can be filled by the $5\times
5$ convolution, which means the CLK do not lacks any information like ordinary
dilated convolution. Secondly, the CLK also achieves channel-wise adaptability
shown in Eq. 5 compared with an ordinary large kernel. The proposed CLK is
further employed to extend PDC to build CPDC for RGB data.

###  III-D CPDC: Extend PDC with CLK.

Since PDC focuses on both fine-grained gradient-level and intensity-level
semantic information in local regions from Depth data, we further apply CLK to
extend PDC to enjoy global information for RGB images.

From Eq.(4), the field size of convolution is input dependent, we hence
directly replace it with our CLK. The combination of PDC and CLK, $i.e.,$ CPDC
is demonstrated in Fig. 2, and the formulation of CPDC can be summarized as:

$\mathbf{F}_{out}=\text{PDC}_{7\times 7}\left(\text{PDC}_{5\times
5}(\mathbf{F})\right).$ (7)

Notably, the PDC formulation in Eq.(3) applies the a convolution with size of
$5\times 5$ with dilation of 1. Here, $\text{PDC}_{5\times 5}$ denotes the PDC
operation with $Conv_{5\times 5}$: $5\times 5$ convolution, dilation of 1 and
PDC7×7 denotes the PDC operation with $Conv_{7\times 7}$: $7\times 7$
convolution, dilation of 3, respectively.

Similar to Eq.(5), the corresponding output is formulated as:

$\mathcal{O}_{\text{cpdc}}=Conv_{1\times
1}(\mathbf{F}_{out})\otimes\mathbf{F},$ (8)

where $Conv_{1\times 1}$ is a channel convolution for achieving the channel-
wise adaptability. When embedded into the PDC, CPDC enjoys the advantages of
both fine-grained gradient-level and intensity-level semantic information in a
satisfied receptive field.

Figure 5: The overview of PDCNet. The network is a two-branch structure, which
consists of two ResNet-101 encoders. The CPDC and PDC are equipped after each
block of ResNet-101 encoder in the RGB branch and Depth branch, respectively.
During training, output feature maps of RGB layer and Depth layer are fused
through a ECF module and then fed into the next stage.

###  III-E PDCNet Architecture

####  III-E1 The Overall Architecture

We adopt the DeepLabv3+ [34] to train our baseline model, where the encoder is
ResNet-101 [33] and keep its original DeepLabv3+ settings. We apply a two-
branch structure in our PDCNet, one for RGB (green part in Fig. 5) and another
for Depth data (yellow part in Fig. 5). Instead of treating RGB and Depth data
equally, we apply the PDC for the Depth branch and CPDC for the RGB branch.
The proposed PDC and CPDC are generalized approaches that are easily plugged
into each block of backbone layers in the two different branches. The feature
representations of both branches are fused in each stage of the backbone
network. In addition, we also employ multi-scale feature fusion strategy,
where low-level and high-level semantic features are combined to produce rich
maps (purple part in Fig. 5). The architecture of our PDCNet is shown in Fig.
5.

####  III-E2 Efficient Cross-modal Fusion (ECF)

Firstly, the PDC is employed for Depth data in each ResNet-101 block in Depth
branch, while CPDC is employed for RGB data in RGB branch. Secondly, for
cross-modal fusion, the complementary geometric information generated by PDC
is fused into the RGB encoder at each stage of the ResNet-101.

Assume that the RGB and Depth features of $l$-th layer are
$\mathbf{F}_{\text{rgb}}^{(l)}\in\mathbb{R}^{H\times W\times C}$ and
$\mathbf{F}_{\text{Depth}}^{(l)}\in\mathbb{R}^{H\times W\times C}$,
respectively. The contribution on RGB and Depth data as follows:

$\hat{\mathbf{F}}_{\text{Depth}}^{(l)}=\text{PDC}\left(\mathbf{F}_{\text{Depth}}^{(l)}\right),\hat{\mathbf{F}}_{\text{rgb}}^{(l)}=\text{CPDC}\left(\mathbf{F}_{\text{rgb}}^{(l)}\right).$
(9)

We set learnable parameters $\eta$ and $\lambda$ for the better RGB and Depth
features fusion. The two parameters $\eta$ and $\lambda$ are not directly
related as it is not reasonable to directly control the fusion ratio of the
two modal data. Consequently, the RGB-D fusion map of $l$-th layer are
generated as:

$\begin{split}\mathbf{F}_{\text{fusion}}^{(l)}&=\eta\left(Conv_{1\times
1}(\hat{\mathbf{F}}_{\text{rgb}}^{(l)})\otimes\hat{\mathbf{F}}_{\text{rgb}}^{(l)}\oplus\mathbf{F}_{\text{rgb}}^{(l)})\right)\\\
&+\lambda\left(Conv_{1\times
1}(\hat{\mathbf{F}}_{\text{Depth}}^{(l)})\otimes\hat{\mathbf{F}}_{\text{Depth}}^{(l)}\oplus\mathbf{F}_{\text{Depth}}^{(l)})\right),\end{split}$
(10)

where $Conv_{1\times 1}$ is a channel convolution for achieving the channel-
wise adaptability.

####  III-E3 Loss Function

Our proposed model is trained on the training test

$\begin{split}\\{(\mathbf{X}_{rgb}^{i},\mathbf{X}_{Depth}^{i},\mathbf{Y}^{i})|\mathbf{X}_{rgb}&\in\mathbb{R}^{H\times
W\times 3},\mathbf{X}_{Depth}\in\mathbb{R}^{H\times W},\\\ &\mathbf{Y}^{i}\in
L^{H\times W},i=\\{1,\cdots,M\\}\\},\end{split}$ (11)

where $\mathbf{Y}$ represents the label and $M$ represents the number of
classes.

The loss function is cross-entropy, which is defined as:

$\displaystyle\mathcal{L}$
$\displaystyle=\frac{-1}{N}\sum_{i}^{N}\ell(yi,\hat{y}i),$ (12)
$\displaystyle\ell(yi,\hat{y}i)$ $\displaystyle=-\sum_{i}^{M}y_{n}log(p_{n}),$
(13)

where $y_{i}$ and $\hat{y}i$ indicates the true label and the predictive label
of samples. $N$ denotes the pixels in the training data and $p_{n}$ denotes
the possibility when the current pixel belongs to class n.

TABLE I: Performance comparison with the state-of-the-art methods on NYUDv2
dataset. IS: Information source. ‘*’: multi-scale strategy.

Method | Backbone | IS. | Pixel Acc. | mIoU | Method | Backbone | IS. | Pixel Acc. | mIoU
---|---|---|---|---|---|---|---|---|---
FCN [45] | VGG16 | HHA | 65.4 | 34.0 | RDF [51]* | ResNet152 | HHA | 76.0 | 50.1
LSD-GF [46] | VGG16 | HHA | 71.9 | 45.9 | DeepLab-LFOV [8]* | VGG16 | Depth | 70.3 | 39.4
D-CNN [47] | VGG16 | Depth | - | 48.4 | SGNet [53]* | ResNet50 | Depth | 76.8 | 51.1
MMAF-Net [48] | ResNet101 | Depth | 72.2 | 44.8 | SA-Gate [54]* | ResNet101 | Depth | 77.9 | 52.4
ACNet [49] | ResNet50 | Depth | - | 48.3 | InverseForm [55]* | Transformer | Depth | 78.1 | 53.1
ShapeConv [50] | ResNet101 | HHA | 75.8 | 50.2 | ShapeConv [50]* | ResNet101 | HHA | 76.4 | 51.3
ADSD [67] | ResNet50 | Depth | 77.5 | 52.5 | M2.5D [52]* | ResNet101 | Depth | 76.9 | 50.9
Ours | ResNet101 | Depth | 77.7 | 52.7 | Ours* | ResNet101 | Depth | 78.4 | 53.5

TABLE II: Performance comparison with the state-of-the-art methods on SUN
RGB-D datasets. IS: Information source. ‘*’: multi-scale strategy.

Method | Backbone | IS | Pixel Acc. | mIoU | Method | Backbone | IS | Pixel Acc. | mIoU
---|---|---|---|---|---|---|---|---|---
3DGNN [58] | VGG16 | HHA | - | 44.1 | 3DGNN [58]* | VGG16 | HHA | - | 45.9
D-CNN [47] | VGG16 | HHA | - | 42.0 | DeepLab-LFOV [8]* | VGG16 | Depth | 71.9 | 32.1
MMAF-Net [48] | ResNet50 | Depth | 81.0 | 47.0 | RDF [51]* | ResNet152 | HHA | 81.5 | 47.7
SGNet [53] | ResNet101 | Depth | 81.0 | 47.5 | SGNet [53]* | ResNet101 | Depth | 82.0 | 47.6
ShapeConv [50] | ResNet101 | HHA | 82.0 | 47.6 | ShapeConv [50]* | ResNet101 | HHA | 82.2 | 48.6
ACNet [49] | ResNet50 | HHA | - | 48.1 | RefineNet [61]* | ResNet152 | Depth | - | 48.1
ESANet-R34-NBt1D [68] | ResNet50 | Depth | - | 48.2 | CRF [60]* | ResNet152 | Depth | - | 48.1
Ours | ResNet101 | Depth | 82.4 | 49.2 | Ours* | ResNet101 | Depth | 83.3 | 49.6

TABLE III: Single-scale testing performance comparison with different baseline
methods on NYUDv2 test set.

Method | Backbone | Setting | Pixel Acc. | mIoU
---|---|---|---|---
Deeplabv3+ [34] | ResNet-101 | baseline | 75.1 | 47.4
ours | 77.7 | 52.7
$\uparrow(\%)$ | 2.6 | 5.3
ResNet-50 | baseline | 74.5 | 46.5
ours | 76.5 | 51.1
$\uparrow(\%)$ | 2.0 | 4.6
Deeplabv3 [42] | ResNet-101 | baseline | 73.3 | 45.4
ours | 75.9 | 50.2
$\uparrow(\%)$ | 2.6 | 4.8
ResNet-50 | baseline | 72.7 | 45.2
ours | 75.6 | 49.3
$\uparrow(\%)$ | 2.9 | 4.1
PSPNet [43] | ResNet-101 | baseline | 72.8 | 44.3
ours | 74.9 | 49.1
$\uparrow(\%)$ | 2.1 | 4.8
ResNet-50 | baseline | 72.2 | 43.6
ours | 75.1 | 48.5
$\uparrow(\%)$ | 2.9 | 3.9
FPN [44] | ResNet-101 | baseline | 74.4 | 46.5
ours | 76.4 | 50.1
$\uparrow(\%)$ | 2.0 | 4.6
ResNet-50 | baseline | 74.1 | 46.2
ours | 75.9 | 50.1
$\uparrow(\%)$ | 1.8 | 3.9

TABLE IV: The effectiveness of PDC and CPDC on NYUDv2 dataset under single-
scale testing. Note: Our proposed method applies PDC for Depth branch and CPDC
for RGB branch, and ‘Place’ indicates whether the two modules are placed in
the right position.

Method | Depth branch | RGB branch | Place | Pixel Acc. | mIoU
---|---|---|---|---|---
$\mathfrak{a}$ | Baseline | Baseline | - | 75.1 | 47.4
$\mathfrak{b}$ | Baseline + PDC | Baseline | yes | 76.9 | 51.8
$\mathfrak{c}$ | Baseline | Baseline + CPDC | yes | 76.8 | 51.6
$\mathfrak{d}$ | Baseline + CPDC | Baseline | no | 75.7 | 50.7
$\mathfrak{e}$ | Baseline | Baseline + PDC | no | 76.2 | 50.9
$\mathfrak{f}$ (Ours) | Baseline + PDC | Baseline + CPDC | yes | 77.7 | 52.7

TABLE V: Comparison with other context aggregation methods on NYUDv2 dataset.
The four methods are for RGB data to capture long-range dependence and no
operations are performed on the Depth data.

Method | PLK | NL | CLK | CPDC (CLK+PDC) | Pixel Acc. | mIoU
---|---|---|---|---|---|---
$\mathfrak{a}$ | ✓ |  |  |  | 75.1 | 47.8
$\mathfrak{b}$ |  | ✓ |  |  | 75.5 | 49.5
$\mathfrak{c}$ |  |  | ✓ |  | 75.8 | 50.3
$\mathfrak{d}$ |  |  |  | ✓ | 76.8 | 51.6

##  IV Experiments

###  IV-A Dataset and Metrics

NYUDv2 [20] consists of 1,449 densely labeled pairs of RGB-D images. We keep
the 40-class settings, where 795 images are used for training and the
remaining 654 images are used for testing.

SUN RGB-D [38] contains 10,335 RGB-D images with 37 categories. We follow the
common setting, where 5,285 images are used as training set and 5,050 are
utilized as testing set.

We apply two common metrics, $i.e.,$ Pixel Accuracy (Pixel Acc.), and Mean
Intersection Over Union (mIoU) to report the results. The two metrics are
defined as follows:

$\text{Pixel Acc.}=\sum_{i}\frac{p_{ii}}{g},$ (14)
$\text{mIoU}=\frac{1}{M}\sum_{i}\frac{p_{ii}}{g_{i}+\sum_{j}p_{ji}-p_{ii}}.$
(15)

where $p_{ij}$ is the number of pixels that are predicted as class $j$ with
ground truth class $i$, $M$ is the number of classes and $g_{i}$ is the number
of pixels with ground truth class $i$, the total number of all pixels is
$g=\sum_{i}g_{i}$.

RGB

Depth

Baseline

Ours

Ground truth

Figure 6: Visual comparison of scene semantic segmentation on NYUDv2 dataset.

RGB

Depth

Baseline

Ours

Ground truth

Figure 7: Visual comparison of scene semantic segmentation on SUN RGB-D
dataset.

Figure 8: Visualizations of feature map response. We compare the response map
of Depth data with/without PDC and RGB data with/without CPDC.

###  IV-B Implementation Details

####  IV-B1 Training Setting

We implement our network using the PyTorch deep learning framework [39], and
train the models on two NVIDIA Tesla V100 GPUs. The initial learning rate is
set to 8e-3 with the “poly” policy [40]. The input images are cropped to
$480\times 480$ and the batch size is $8$. The SGD is used as our optimizer,
with momentum of 0.9 and a weight decay of 1e-4.

####  IV-B2 Comparison Protocol

For fair comparisons, we compare the baseline (two original parallel
DeepLabv3+, ResNet-101) [34] with two improved encoders (equipped with the PDC
and CPDC, respectively), without changing other settings in DeepLabv3+. This
is to verify that the model effectiveness comes from the proposed PDC and
CPDC, not other factors.

For a fair comparison with other state-of-the-arts, we employ both single-
scale and multi-scale testing strategies. If not otherwise noted, the
experiments are evaluated under single-scale testing, and ‘$*$’ in tables
indicates the multi-scale strategy.

###  IV-C Comparisons with State-of-the-arts

####  IV-C1 NYUDv2

As shown in Tab. I, we achieve the Pixel Acc and mIoU of 77.7% and 52.7% under
single-scale testing, outperforming the contemporary work, ShapeConv [50] by
1.9% in and 2.5% in Pixel Acc. and mIoU, respectively. Moreover, our
segmentation performance is 0.3% Pixel Acc. and 0.4% mIoU more than
InverseForm [55] using Transformer as the backbone. On multi-scale testing,
our PDCNet also achieves leading performance.

####  IV-C2 SUN RGB-D

Tab. II shows the testing results on SUN RGB-D dataset. Our PDCNet outperforms
other methods under both single-scale and multi-scale testing. For example,
under multi-scale testing, PDCNet outperforms the contemporary work, SGNet
[53] by 1.3% in Pixel Acc and 2.0% in mIoU, respectively. The competitive
performances benefit from our proposed PDC and CPDC, which can make use of
subtle pixel-wise gradient-level and intensity-level information in local and
global range simultaneously. In other words, the PDC and CPDC exploit the
characteristics of the two modal data and complement each other effectively.

###  IV-D Experiments on Different Architectures

To confirm the generalization performance of our method, we equip the PDC and
CPDC modules with other classic semantic segmentation architectures, including
Deeplabv3+ [34], Deeplabv3 [42], PSPNet [43] and FPN [44] with (ResNet-50,
ResNet-101 [33]) on NYUDv2 dataset. As shown in Tab. III, under all settings,
our method achieves significant performance improvements, demonstrating its
generalization ability. For example, when applying Deeplabv3+ as the baseline,
the results reveal that our model achieves significant improvements in
performance (+2.6% Pixel Acc and +5.3% mIoU).

###  IV-E Ablation Study

####  IV-E1 Effectiveness of PDC and CPDC

Tab. IV shows the indispensability of our model on the NYUDv2 dataset by
adopting the proposed PDC and CPDC modules. As shown in Tab. IV, we can
conclude that 1) comparing our model with $\mathfrak{a}$ to $\mathfrak{c}$,
both the PDC and CPDC modules improve model performance and work best when
used in combination. 2) Comparing our model with $\mathfrak{d}$ and
$\mathfrak{e}$ illustrate that our original setting, $i.e.,$ PDC and CPDC are
suitable for Depth and RGB data, respectively. This also explains that Depth
maps are more suitable for providing intrinsic fine-grained patterns of
objects due to their Depth continuity, while RGB images effectively provide a
global view.

In this regard, the PDC and CPDC are complementary and can function as a
unified unit to mine the intra-domain inherent context within the data. The
model is enabled to explore the inherent intra-domain context by explicitly
regularizing the feature space via aggregation of gradient pixel-wise
differences and intensity pixel-wise differences.

####  IV-E2 Comparison with Alternatives

One may argue that our self-attention based modules can replace our CLK to
capture long-range contextual information. Here, we compare the performance of
our CLK with a classic self-attention module, Non-local neural networks [14],
denoted as ‘NL’. We compare the long-range context aggregation abilities in
the RGB branch, while Depth branch utilizes original baseline backbone. As can
be observed in Tab. V $\mathfrak{b}$ and $\mathfrak{c}$, CLK outperforms ‘NL’
by 0.3% Pixel Acc. and 0.8% mIoU. Combing CLK and PDC, $i.e.,$ CPDC, achieves
the best results.

In fact, self-attention ignores the fundamental difference between sequence-
based NLP tasks and the 2D structure, which destroys the crucial 2D structure
of images. Moreover, it is worth noting that channels-wise information often
represents different objects in CNNs [37, 36, 75] and the channel-wise
adaptability is also crucial for segmentation tasks, which is lacking in self-
attention. With depth-wise convolution, each input channel undergoes a
convolution separately, while point-wise convolution is performed to combine
the outputs from depth-wise convolution. Compared with ‘NL’, CLK not only
involves the advantages of long-range dependence in 2D structure, but also
achieves channel-wise adaptability.

We also verify the performance of our CLK by comparing it with the Parallel
mode large kernel shown in Eq. 6, denoted as ‘PLK’. As illustrated in
$\mathfrak{a}$ and $\mathfrak{c}$ of Tab. V, compared with ‘PLK’, the CLK has
a larger receptive field without losing any local information and henceforce
achieves higher Pixel Acc and mIoU. Notably, the dilation of $7\times 7$
convolution can be filled by the $5\times 5$ convolution, which means the CLK
do not lacks any information.

####  IV-E3 Impact of $\alpha$ in PDCNet

As discussed in Eq.(3) that $\alpha$ adjusts the contribution of fine-grained
gradient-level changes and intensity-level changes for semantic information.
As illustrated in Tab. LABEL:impact, when $\alpha$ is set as a learnable
parameter, both Pixel Acc. and mIoU achieves the best score. Additionally, we
also set the $\alpha$ as default configurations, $i.e.$, varying from 0 to 1
to test the importance of the pixel difference term in PDC. It proves that
pixel difference term shown in Eq. 3 is more crucial than the vanilla term,
which means integrating pixel difference operation into CNNs is an effective
solution.

TABLE VI: Impact of $\alpha$ in PDCNet tested on NYUDv2 dataset under multi-
scale testing.

$\alpha$ | Pixel Acc. | mIoU
---|---|---
0.1 | 77.2 | 51.3
0.2 | 77.4 | 51.5
0.3 | 77.6 | 52.1
0.4 | 77.8 | 52.5
0.5 | 78.0 | 52.8
0.6 | 78.1 | 53.0
0.7 | 78.1 | 53.2
0.8 | 78.2 | 53.4
0.9 | 78.0 | 52.8
learnable | 78.4 | 53.5

TABLE VII: Impact of kernel size in PDC tested on NYUDv2 dataset under multi-
scale testing.

Kernel size | Pixel Acc. | mIoU
---|---|---
$3\times 3$ | 78.2 | 53.1
$5\times 5$ | 78.4 | 53.5
$7\times 7$ | 78.4 | 53.0
$9\times 9$ | 78.1 | 52.9

####  IV-E4 Impact of Kernel Size in PDC

Compared with the long-range receptive field in CPDC, our PDC applies the
convolution kernel of $5\times 5$, dilation 1 to capture fine-grained
gradient-level and intensity-level semantic information in local regions from
Depth data. We conduct the ablation study to compare the effectiveness of
convolution kernels of different sizes. As shown in Tab.LABEL:kernel_size, PDC
achieves the best performance when applying the kernel size of $5\times 5$.

##  V Visualization of PDCNet

To illustrate the results of the model more intuitively, we display the
qualitative results of the NYUDv2 and SUN RGB-D datasets in Fig. 6 and Fig. 7,
respectively. From the results, we confirm that the local pixel difference in
Depth image and global dependence are well enhanced by our PDCNet. For
example, in the first row of Fig. 6, the pillows have very similar patterns to
the bed, which cannot be easily discriminated by just the 2D appearance, while
they can be well separated by the local pixel difference in Depth.
Furthermore, the global dependence makes sure that the rest of the bed is not
misidentified as a pillow. Indeed, this is what we find: since CPDC captures
features in high contrast areas from RGB images while PDC separates objects by
Depth rather than lighting conditions.

We also visualize the feature maps from the RGB branch and Depth branch of our
proposed model as well as the feature maps of the original Deeplabv3+ model.
As illustrated in Fig. 8, the PDC and CPDC focus more on pixel-level subtle
information, such as edges and boundaries. Moreover, the PDC tends to show the
overall outline of the object better. Notably, we believe that our PDC
overcomes image noise, which can be analyzed from the principle of PDC: If a
data point in Depth map is noisy, the pixel differences between this point and
other points are very large, which can be easily captured. This is expected,
it is also the pixel difference convolution matters.

##  VI Conclusion

We present a state-of-the-art two-branch PDCNet by introducing two plug-and-
play modules: PDC and CPDC. PDC considers subtle geometric information (pixel
difference) that occurs in local regions in Depth data. We also propose CLK,
with which PDC is easily extended to CPDC to absorb the advantage of capturing
local pixel-wise features of PDC and long-range dependence of CLK, which
further captures refined pixel discrepancy in the satisfying receptive field.
Consequently, both intensity and gradient information from 2D patterns and 3D
geometric maps are seamlessly incorporated into our PDCNet. We also conduct
extensive experiments on NYUDv2 and SUN RGB-D to evaluate the superiority of
our proposed model. Empirical results illustrate that PDCNet exhibits
substantial improvements over the state-of-the-arts.

## Acknowledgement

This work was supported by the grants from the National Key R&D Program of
China (2019YFC1906201), the National Natural Science Foundation of China
(91748122). Jun Yang thanks for China Scholarship Council (CSC) for supporting
him to study at École normale supérieure and also thanks for Yerkebulan
Massalim, a PH.D student in École normale supérieure, for his help in English
improvement.

## References

  * [1] L. Qi, L. Jiang, S. Liu, X. Shen, and J. Jia, “Amodal instance segmentation with kins dataset,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3014–3023.
  * [2] G. Lin, C. Shen, A. Van Den Hengel, and I. Reid, “Efficient piecewise training of deep structured models for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 3194–3203.
  * [3] S. Gupta, R. Girshick, P. Arbeláez, and J. Malik, “Learning rich features from rgb-d images for object detection and segmentation,” in _European conference on computer vision_. Springer, 2014, pp. 345–360.
  * [4] Y. He, W.-C. Chiu, M. Keuper, and M. Fritz, “Std2p: Rgbd semantic segmentation using spatio-temporal data-driven pooling,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 4837–4846.
  * [5] F. Husain, H. Schulz, B. Dellen, C. Torras, and S. Behnke, “Combining semantic and geometric features for object class segmentation of indoor scenes,” _IEEE Robotics and Automation Letters_ , vol. 2, no. 1, pp. 49–55, 2016.
  * [6] J. Jiang, L. Zheng, F. Luo, and Z. Zhang, “Rednet: Residual encoder-decoder network for indoor rgb-d semantic segmentation,” _arXiv preprint arXiv:1806.01054_ , 2018.
  * [7] V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 12, pp. 2481–2495, 2017.
  * [8] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 4, pp. 834–848, 2017.
  * [9] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [10] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [11] Y. Cao, J. Xu, S. Lin, F. Wei, and H. Hu, “Gcnet: Non-local networks meet squeeze-excitation networks and beyond,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_ , 2019, pp. 0–0.
  * [12] J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu, “Dual attention network for scene segmentation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 3146–3154.
  * [13] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “Cbam: Convolutional block attention module,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 3–19.
  * [14] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7794–7803.
  * [15] Z. Zhu, M. Xu, S. Bai, T. Huang, and X. Bai, “Asymmetric non-local neural networks for semantic segmentation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 593–602.
  * [16] H. Zhao, J. Jia, and V. Koltun, “Exploring self-attention for image recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 10 076–10 085.
  * [17] A. Vaswani, P. Ramachandran, A. Srinivas, N. Parmar, B. Hechtman, and J. Shlens, “Scaling local self-attention for parameter efficient visual backbones,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 12 894–12 904.
  * [18] P. Ramachandran, N. Parmar, A. Vaswani, I. Bello, A. Levskaya, and J. Shlens, “Stand-alone self-attention in vision models,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [19] X. Ren, L. Bo, and D. Fox, “Rgb-(d) scene labeling: Features and algorithms,” in _2012 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2012, pp. 2759–2766.
  * [20] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in _European conference on computer vision_. Springer, 2012, pp. 746–760.
  * [21] J. Jiao, Y. Wei, Z. Jie, H. Shi, R. W. Lau, and T. S. Huang, “Geometry-aware distillation for indoor semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2869–2878.
  * [22] S. Gupta, P. Arbelaez, and J. Malik, “Perceptual organization and recognition of indoor scenes from rgb-d images,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2013, pp. 564–571.
  * [23] S. H. Khan, M. Bennamoun, F. Sohel, R. Togneri, and I. Naseem, “Integrating geometrical context for semantic labeling of indoor scenes using rgbd images,” _International Journal of Computer Vision_ , vol. 117, no. 1, pp. 1–20, 2016.
  * [24] M. Lin, Q. Chen, and S. Yan, “Network in network,” _arXiv preprint arXiv:1312.4400_ , 2013.
  * [25] V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in _Icml_ , 2010.
  * [26] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” _The journal of machine learning research_ , vol. 15, no. 1, pp. 1929–1958, 2014.
  * [27] T. Ojala, M. Pietikainen, and T. Maenpaa, “Multiresolution gray-scale and rotation invariant texture classification with local binary patterns,” _IEEE Transactions on pattern analysis and machine intelligence_ , vol. 24, no. 7, pp. 971–987, 2002.
  * [28] H. Mei, B. Dong, W. Dong, P. Peers, X. Yang, Q. Zhang, and X. Wei, “Depth-aware mirror segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 3044–3053.
  * [29] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena, “Self-attention generative adversarial networks,” in _International conference on machine learning_. PMLR, 2019, pp. 7354–7363.
  * [30] F. Wang, M. Jiang, C. Qian, S. Yang, C. Li, H. Zhang, X. Wang, and X. Tang, “Residual attention network for image classification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 3156–3164.
  * [31] J. Hu, L. Shen, S. Albanie, G. Sun, and A. Vedaldi, “Gather-excite: Exploiting feature context in convolutional neural networks,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [32] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1251–1258.
  * [33] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [34] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 801–818.
  * [35] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.
  * [36] L. Chen, H. Zhang, J. Xiao, L. Nie, J. Shao, W. Liu, and T.-S. Chua, “Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 5659–5667.
  * [37] X. Qin, Z. Wang, Y. Bai, X. Xie, and H. Jia, “Ffa-net: Feature fusion attention network for single image dehazing,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 07, 2020, pp. 11 908–11 915.
  * [38] S. Song, S. P. Lichtenberg, and J. Xiao, “Sun rgb-d: A rgb-d scene understanding benchmark suite,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 567–576.
  * [39] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [40] W. Liu, A. Rabinovich, and A. C. Berg, “Parsenet: Looking wider to see better,” _arXiv preprint arXiv:1506.04579_ , 2015.
  * [41] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _International conference on machine learning_. PMLR, 2015, pp. 448–456.
  * [42] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, “Rethinking atrous convolution for semantic image segmentation,” _arXiv preprint arXiv:1706.05587_ , 2017.
  * [43] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2117–2125.
  * [44] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2881–2890.
  * [45] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3431–3440.
  * [46] Y. Cheng, R. Cai, Z. Li, X. Zhao, and K. Huang, “Locality-sensitive deconvolution networks with gated fusion for rgb-d indoor semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 3029–3037.
  * [47] W. Wang and U. Neumann, “Depth-aware cnn for rgb-d segmentation,” _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 135–150.
  * [48] F. Fooladgar and S. Kasaei, “Multi-modal attention-based fusion model for semantic segmentation of rgb-depth images,” _arXiv preprint arXiv:1912.11691_ , 2019.
  * [49] X. Hu, K. Yang, L. Fei, and K. Wang, “Acnet: Attention based network to exploit complementary features for rgbd semantic segmentation,” in _2019 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2019, pp. 1440–1444.
  * [50] J. Cao, H. Leng, D. Lischinski, D. Cohen-Or, C. Tu, and Y. Li, “Shapeconv: Shape-aware convolutional layer for indoor rgb-d semantic segmentation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 7088–7097.
  * [51] S.-J. Park, K.-S. Hong, and S. Lee, “Rdfnet: Rgb-d multi-level residual feature fusion for indoor semantic segmentation,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 4980–4989.
  * [52] Y. Xing, J. Wang, and G. Zeng, “Malleable 2.5 d convolution: Learning receptive fields along the depth-axis for rgb-d scene parsing,” in _European Conference on Computer Vision_. Springer, 2020, pp. 555–571.
  * [53] L.-Z. Chen, Z. Lin, Z. Wang, Y.-L. Yang, and M.-M. Cheng, “Spatial information guided convolution for real-time rgbd semantic segmentation,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 2313–2324, 2021.
  * [54] X. Chen, K.-Y. Lin, J. Wang, W. Wu, C. Qian, H. Li, and G. Zeng, “Bi-directional cross-modality feature propagation with separation-and-aggregation gate for rgb-d semantic segmentation,” in _European Conference on Computer Vision_. Springer, 2020, pp. 561–577.
  * [55] S. Borse, Y. Wang, Y. Zhang, and F. Porikli, “Inverseform: A loss function for structured boundary-aware segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 5901–5911.
  * [56] H. Ye and D. Xu, “Inverted pyramid multi-task transformer for dense scene understanding,” _arXiv preprint arXiv:2203.07997_ , 2022.
  * [57] Y. Wang, X. Chen, L. Cao, W. Huang, F. Sun, and Y. Wang, “Multimodal token fusion for vision transformers,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 12 186–12 195.
  * [58] X. Qi, R. Liao, J. Jia, S. Fidler, and R. Urtasun, “3d graph neural networks for rgbd semantic segmentation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 5199–5208.
  * [59] H. Ding, X. Jiang, B. Shuai, A. Q. Liu, and G. Wang, “Semantic segmentation with context encoding and multi-path decoding,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 3520–3533, 2020.
  * [60] D. Lin, G. Chen, D. Cohen-Or, P.-A. Heng, and H. Huang, “Cascaded feature network for semantic segmentation of rgb-d images,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 1311–1319.
  * [61] G. Lin, A. Milan, C. Shen, and I. Reid, “Refinenet: Multi-path refinement networks for high-resolution semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1925–1934.
  * [62] X. Zhao, Y. Pang, L. Zhang, H. Lu, and X. Ruan, “Self-supervised pretraining for rgb-d salient object detection,” in _AAAI Conference on Artificial Intelligence_ , vol. 3, 2022.
  * [63] M. Yang, K. Yu, C. Zhang, Z. Li, and K. Yang, “Denseaspp for semantic segmentation in street scenes,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 3684–3692.
  * [64] Z. Huang, X. Wang, L. Huang, C. Huang, Y. Wei, and W. Liu, “Ccnet: Criss-cross attention for semantic segmentation,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 603–612.
  * [65] H. Zhao, Y. Zhang, S. Liu, J. Shi, C. C. Loy, D. Lin, and J. Jia, “Psanet: Point-wise spatial attention network for scene parsing,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 267–283.
  * [66] Z. Yu, C. Zhao, Z. Wang, Y. Qin, Z. Su, X. Li, F. Zhou, and G. Zhao, “Searching central difference convolutional networks for face anti-spoofing,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 5295–5305.
  * [67] Y. Zhang, Y. Yang, C. Xiong, G. Sun, and Y. Guo, “Attention-based dual supervised decoder for rgbd semantic segmentation,” _arXiv preprint arXiv:2201.01427_ , 2022.
  * [68] D. Seichter, M. Köhler, B. Lewandowski, T. Wengefeld, and H.-M. Gross, “Efficient rgb-d semantic segmentation for indoor scene analysis,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 13 525–13 531.
  * [69] T. Huang, H. Zou, J. Cui, J. Zhang, X. Yang, L. Li, and Y. Liu, “Adaptive recurrent forward network for dense point cloud completion,” _IEEE Transactions on Multimedia_ , 2022.
  * [70] J. Liu, J. Guo, and D. Xu, “Geometrymotion-transformer: An end-to-end framework for 3d action recognition,” _IEEE Transactions on Multimedia_ , 2022.
  * [71] X. Zhang, S. Zhang, Z. Cui, Z. Li, J. Xie, and J. Yang, “Tube-embedded transformer for pixel prediction,” _IEEE Transactions on Multimedia_ , 2022\.
  * [72] H. Zhu, J. Deng, Y. Zhang, J. Ji, Q. Mao, H. Li, and Y. Zhang, “Vpfnet: Improving 3d object detection with virtual point based lidar and stereo data fusion,” _IEEE Transactions on Multimedia_ , 2022.
  * [73] J. Nie, C. Wang, S. Yu, J. Shi, X. Lv, and Z. Wei, “Mign: Multiscale image generation network for remote sensing image semantic segmentation,” _IEEE Transactions on Multimedia_ , 2022.
  * [74] G. Gao, G. Xu, J. Li, Y. Yu, H. Lu, and J. Yang, “Fbsnet: A fast bilateral symmetrical network for real-time semantic segmentation,” _IEEE Transactions on Multimedia_ , 2022.
  * [75] L. Ma, H. Xie, C. Liu, and Y. Zhang, “Learning cross-channel representations for semantic segmentation,” _IEEE Transactions on Multimedia_ , 2022.
  * [76] M. Zhang and Q. Ling, “Supervised pixel-wise gan for face super-resolution,” _IEEE Transactions on Multimedia_ , vol. 23, pp. 1938–1950, 2020.