(2cm,0.5cm)This work has been submitted to the IEEE for possible publication.
Copyright may be transferred without notice, after which this version may no
longer be accessible.

# Towards Dynamic and Small Objects Refinement for Unsupervised Domain
Adaptative Nighttime Semantic Segmentation

Jingyi Pan1, Sihang Li1, Yucheng Chen1, Jinjing Zhu1 and Lin
Wang${}^{1,2}{{}^{*}}$ ∗Corresponding author1J. PAN, S. Li, Y. Chen, J. Zhu
are with the AI Thrust, HKUST(GZ), Guangdong, 511458, China. Email: {jpan305,
sli886, ychen208, jzhu706}@connect.hkust-gz.edu.cn1,2L. Wang is with AI/CMA
Thrust, HKUST(GZ) and Dept. of CSE, HKUST, Hong Kong SAR, China, Email:
<EMAIL_ADDRESS>
###### Abstract

Nighttime semantic segmentation is essential for various applications, e.g.,
autonomous driving, which often faces challenges due to poor illumination and
the lack of well-annotated datasets. Unsupervised domain adaptation (UDA) has
shown potential for addressing the challenges and achieved remarkable results
for nighttime semantic segmentation. However, existing methods still face
limitations in 1) their reliance on style transfer or relighting models, which
struggle to generalize to complex nighttime environments, and 2) their
ignorance of dynamic and small objects like vehicles and traffic signs, which
are difficult to be directly learned from other domains. This paper proposes a
novel UDA method that refines both label and feature levels for dynamic and
small objects for nighttime semantic segmentation. First, we propose a dynamic
and small object refinement module to complement the knowledge of dynamic and
small objects from the source domain to target nighttime domain. These dynamic
and small objects are normally context-inconsistent in under-exposed
conditions. Then, we design a feature prototype alignment module to reduce the
domain gap by deploying contrastive learning between features and prototypes
of the same class from different domains, while re-weighting the categories of
dynamic and small objects. Extensive experiments on four benchmark datasets
demonstrate that our method outperforms prior arts by a large margin for
nighttime segmentation. Project page: https://rorisis.github.io/DSRNSS/.

##  I Introduction

Semantic segmentation plays an important role in many applications, e.g.,
autonomous driving [1, 2, 3] and robots [4, 5]. Recently, with the development
of deep neural networks (DNNs), semantic segmentation has achieved remarkable
progress. However, existing methods predominantly target the daytime images
[6, 7, 8]. Thus their performance drastically drops in the challenging visual
conditions, e.g., nighttime [9, 10, 11], especially to dynamic and small
objects. The bottleneck of semantic segmentation at nighttime lies in the
decreased performance resulting from inadequate illumination and the scarcity
of annotated datasets.

To tackle the problem, unsupervised domain adaptation (UDA) methods have been
developed to adapt the model learning unlabeled target domain (i.e.,
nighttime) images from the model trained with the source domain (i.e.,
daytime) images. The prevailing methods can be categorized into three types:
1) Some methods, e.g., [9, 12, 13, 10, 14], use the style transfer models,
e.g., CycleGAN [15], to generate daytime or nighttime images, which serve as
an intermediate domain to connect the source and target domains. However,
these methods are cumbersome as they require multi-stage learning, and the
performance can’t be guaranteed if style transfer fails. 2) Other methods
utilize twilight images with the coarsely aligned day-night image pairs in the
target domain, to progressively adapt from the daytime to nighttime domains
[16, 9, 10]. 3) The rest methods leverage prior GPS knowledge or static loss
to reduce the influence of coarsely aligned day-night image pairs, which
improves the quality of pseudo labels [11, 17, 18, 19, 20]. However, they pay
less attention to the dynamic and small objects, e.g., vehicles and traffic
signs, in nighttime images, making it difficult to achieve effective domain
alignment in low-exposure conditions. Also, existing UDA methods employing
patch-wise or pixel-wise contrastive learning [21, 22] face challenges in
acquiring effective semantic contexts within patches or pixels in the under-
exposed conditions.

Figure 1: Left: Our UDA method addresses the difficulty of transferring
knowledge about dynamic and small objects from other domains for nighttime
segmentation. Right: Compared with DANNet [11], our method significantly
improves the performance of the dynamic and small objects.

Motivation: In this paper, we propose a novel one-stage UDA framework for
nighttime semantic segmentation paying more attention to the dynamic and small
objects. Specifically, our framework consists of two key technical modules:
label-level dynamic and small objects refinement (DSR) module and feature
prototype alignment (FPA) module. Firstly, the DSR module generates a
composite mask from the ground truth of the source domain, emphasizing dynamic
and small objects, and randomly selected classes. We also introduce a memory
bank to store the long-tailed objects across images, which complements these
low-occurring categories from different images (Sec. III-B). We observe that
the mixup operation forms a new domain—the mixed domain that has a domain
shift with the source daytime and target nighttime domain. Therefore, we
propose the FPA module to align the source domain with the mixed and target
nighttime domain, respectively (Sec. III-C). FPA extracts reliable prototypes
from labels in the source domain and pseudo-labels in the mixed domain while
pulling the features with the same classes from another domain as positive
pairs closer, and pushing other negative pairs away via a contrastive loss.
Note that the same operation is conducted in the source and target nighttime
domain. We further design an adaptive re-weighting mechanism to improve the
attention of some dynamic and small categories in the FPA module. Extensive
experiments on the Dark Zurich [9] and the Nighttime Driving [16] datasets
outperform the state-of-the-art (SoTA) in most categories, particularly in
identifying dynamic and small objects. Notably, our approach achieved
remarkable mean intersection over union (mIoU) scores of 86.8% for the ‘car’
category and 25.2% for the ’bus’ category, which represent a significant
improvement of 3.0% and 20.2% absolute performance, respectively, over the
SoTA method.

In summary, the main contributions of this work are three-fold: (I) We study a
crucial problem of focusing more on improving the performance of dynamic and
small objects and reducing the domain shifts caused by illumination and style
differences. (II) We propose a novel UDA framework for nighttime semantic
segmentation framework by designing a label-level dynamic and small objects
refinement (DSR) module and feature prototype alignment (FPA) module. (III)
Extensive experiments on the Dark Zurich and Nighttime Driving datasets verify
that our method achieves a new SoTA performance for nighttime semantic
segmentation.

Figure 2: Overview of our proposed framework. $F_{s}$ and $F_{t}$ denote the
student network and teacher network. 1) In the image-level mixup stage, we mix
dynamic and small classes in source images into target nighttime images
according to the ground truth of source images. 2) In the FPA module, we
calculate prototypes ($\rho_{m}$, $\rho_{s},\rho_{n}$) of each class to
regularize the pixel embedding space ($f_{s}$, $f_{m}$, $f_{n}$) with
contrastive learning. 3) In the label-level mixup stage, the refined target
results obtained from the holistic refinement and source ground truth are
mixed in the same classes as the image-level mixup into the mixed pseudo label
for focusing the dynamic and small objects refinement.

##  II Related work

###  II-A UDA for semantic segmentation

####  II-A1 Daytime semantic segmentation

can be divided into three categories. The first type of approach leverages
adversarial learning to reduce the domain gap [23, 24]. The second type of
method employs style transfer models, e.g., CycleGAN [15], to build an
intermediate domain to bridge the domain gap[25, 26, 27]. The rest line of the
methods imposes extra constraints on data distribution to align source and
target domains for domain adaptation. [28] utilizes the NCE loss on a vector
which is converted from the feature to obtain the global semantic
relationship. On the other hand, Liu et al.[29] brings the distribution closer
to each other by BN statistical information. However, these daytime semantic
segmentation models are hard to be applied in nighttime scenarios due to the
poor illumination. Thus, we focus on improving the performance of nighttime
semantic segmentation, especially for dynamic and small objects.

####  II-A2 Nighttime semantic segmentation

[30, 9, 10, 12, 11, 17] utilize style transfer models to enhance the nighttime
appearance to resemble daytime lighting. Among them, DANNet [11] designs a
relighting network to push the intensity distribution of daytime and nighttime
images closer. Bi-Mix [17] proposes a bidirectional mixing framework and
exploits day-night image pairs to improve the quality of relighting. However,
these methods render the segmentation performance to be highly dependent on
the style transfer model. Besides, several works propose to construct an
intermediate domain to reduce the domain gap. Dai et al. [16, 10, 9]
progressively adapt the daytime-trained model to model learning nighttime
images, assisted by the intermediate twilight domain. Also, some methods use
other sensors to assist nighttime segmentation [31, 32]. However, these
methods ignore the dynamic and small objects, e.g., vehicles and traffic
signs, in the domain alignment process, and the capacity to address the
inherent domain shifts between datasets caused by illumination and style
differences.

###  II-B Mixup

It is a strategy to augment data to improve the robustness of DNNs. It has
been widely used to augment samples between the source and target domains to
reduce the domain gap. [33] [34] [35] exploit the mixup ratio combined with
randomly sampled values from the beta distribution to achieve domain
adaptation. However, mixup samples are inclined to have local ambiguity and
artifacts, which result in model confusion. Cutmix [36] is similar to general
mixup while the difference is that it removes pixels and adds removed regions
with regions from another image. After that, Walawalkar et al.[37] propose
attentive cutmix to enhance the cutmix strategy for better generalization.
Classmix [38] is a generalization of cutmix, which utilizes a binary mask to
mix randomly sampled images. Unlike previous mixup strategies, we propose a
dynamic and small object refinement strategy, which enables the nighttime
domain to enhance these weak categories from the source domain.

##  III Method

###  III-A Overview

An overview of our proposed framework is depicted in Fig. 2. The proposed
approach involves a labeled source domain $S_{d}$ and two coarsely aligned,
unlabeled target domains $T_{d}$ and $T_{n}$. In the source domain, we have
pixel-level annotated source data $X_{s}\in S_{d}$ and extract feature
representations $f_{s}$ using the student model $F_{s}$. We then obtain
segmentation predictions $y_{s}$ supervised by cross-entropy loss
$\mathcal{L}{sup}$. Additionally, the target daytime $X_{d}\in T_{d}$ and
nighttime images $X_{n}\in T_{n}$ are unlabeled. We obtain segmentation
predictions $y_{d}$ and $y_{n}$ using the teacher network $F_{t}$ and align
the large static regions present in both daytime and nighttime target images
through holistic refinement [39]. By replacing the aligned regions in
nighttime predictions with those of daytime predictions, we obtain a refined
nighttime pseudo label $y^{\prime}_{n}$. Our aim is to learn a target model
$F^{\prime}_{s}$ that obtains more knowledge on the dynamic and small objects
from labeled source data and unlabeled target daytime domain.

The lack of emphasis on dynamic and small objects in previous works has
negatively impacted the nighttime segmentation performance. Moreover, domain
shifts caused by illumination and style differences among different domains
have further complicated the issue. Thus, we first propose a label-level
dynamic and small objects refinement (DSR) module (Sec. III-B) that employs a
class-aware mixup function to generate a composite mask from ground truth in
the source domain, and then adds small and dynamic objects from the source
images to nighttime images, which creates a mixed domain. This mixed domain
aims to provide complementary supervision for these classes in nighttime
images (Sec. III-B1). To further enhance the effectiveness of our mixup, we
obtain more reliable pseudo-labels for nighttime images using the holistic
refinement. We then use label-level mixup to generate mixed pseudo labels to
effectively supervise the mixed images from image-level mixup (Sec. III-B2).
Additionally, we introduce the memory bank mechanism further to complement
rare dynamic and small objects across different images. Moreover, we propose
the feature prototype alignment (FPA) module among the source domain, mixed
domain, and target nighttime domain to decrease inherent domain shifts. We
also introduce a re-weighting strategy to pay more attention to the dynamic
and small classes (Sec. III-C). Detailed descriptions of these modules are
provided below.

###  III-B Label-level Dynamic and Small Object Refinement

####  III-B1 Image-level Mixup

Figure 3: An illustration of our DSR module. At the image level, two images
are obtained from $S_{d}$ and $T_{n}$, respectively. We generate a composite
mask from $Y_{s}$ in the source domain, including randomly selected, dynamic
and small classes. This mask is then used to mix the two images while also
mixing $Y_{s}$ and $y^{\prime}_{n}$. Furthermore, the long-tailed memory bank
mechanism introduces a few occurrence regions into the mixed image and label.

Due to the coarse alignment between target nighttime and daytime images, there
needs to be more supervision on these dynamic and small objects in nighttime
images that we hard to directly obtain accurate pseudo label from daytime
domain. To address these challenges, we first leverage the labels of the
source images to mix up the regions of dynamic and small objects from the
source domain into the nighttime images for label-level dynamic and small
object refinement (See Fig. 3). By doing so, we are able to provide accurate
predictions on these previously overlooked objects. Overall, our proposed
approach effectively addresses the limitations of previous methods by
improving segmentation performance on dynamic and small objects in target
nighttime images.

Based on the labels $Y_{s}$ of source images, we define the mixed mask as:

$M(h,w)=\left\\{\begin{array}[]{ll}1,&\mbox{if }Y_{s}(h,w)\in c\\\
0,&\mbox{otherwise }\end{array}\right.,$ (1)

where $M$ is a binary mask, and $c$ is the selected class. The parameters
$h\in H$ and $w\in W$ represent the height and width of the image,
respectively.

We further generate a composite mask, denoted as $M_{c}$, including the
randomly selected class mask $M_{r}$ and the dynamic and small object mask
$M_{m}$ that contains classes of small and dynamic objects in the source
images.

$M_{c}=M_{r}\cup M_{m}.$ (2)

Based on the composite mask $M_{c}$, we augment the nighttime images with
complementary objects from source domain, especially for dynamic and small
objects. For image-level mixup, we utilize the composite mask to mixup
corresponding regions of source images and target images as follow:

$X_{m}=M_{c}\odot X_{n}+(1-M_{c})\odot X_{n},$ (3)

here, $\odot$ represents element-wise multiplication.

####  III-B2 Label-level Mixup

To further provide the supervision for the mixed image domain, reliable mixed
pseudo labels from label-level alignment by the composite mask in Eq. 2 is
necessary. Thus, we mixup the labels of source and nighttime images
corresponding to the image-level mixup as:

$Y_{m}=M_{c}\odot Y_{s}+(1-M_{c})\odot y^{\prime}_{n},$ (4)

where $y^{\prime}_{n}$ is the refined pseudo labels of nighttime images
generated using target daytime images.

In addition to the challenge of long-tailed classes, e.g., bus, bicycle,
motorcycle, for further improving the performance, we incorporate long-tailed
classes from diverse source images into the target nighttime domain.
Therefore, we first design three memory banks which are an image bank $B_{i}$,
a mask bank $B_{m}$, and a label bank $B_{l}$ to store some of the information
of long-tailed classes across different images. These memory banks are the
first-in-first-out queues, which stores a certain number of content. Next, we
mixup the regions of long-tailed classes from the source domain to the above-
mixed images ($X_{m},Y_{m}$) as follows:

$\begin{split}X^{\prime}_{m}=B_{m}\odot B_{i}+(1-B_{m})\odot X_{m},\\\
Y^{\prime}_{m}=B_{m}\odot B_{l}+(1-B_{m})\odot Y_{m}.\end{split}$ (5)

Further, we utilize the student model $F_{s}$ to make predictions
$y^{\prime}_{m}=F_{s}(X^{\prime}_{m})$ of mixed image $X^{\prime}_{m}$. To
ensure consistency between the mixed predictions from $F_{s}$ and the mixed
pseudo labels from $F_{t}$, we define a mixup loss as follows:

$\mathcal{L}_{mix}=-\sum^{C}_{c=0}\sum^{W}_{w=0}\sum^{H}_{h=0}Y^{\prime
c,h,w}_{m}log(y^{\prime c,h,w}_{m}).$ (6)

The weights $\phi^{\prime}_{t}$ of teacher model during the training are
updated at every t step by student’s weights $\phi_{t}$:

$\phi^{\prime}_{t}=\lambda\dot{\phi}^{\prime}_{t-1}+(1-\lambda)\dot{\phi}_{t},$
(7)

where $\lambda$ is the EMA decay of the smoothing coefficient, and
$\lambda\in[0,1]$.

###  III-C Feature Prototype Alignment

In the context of DSR module, the mixup operation generates a new domain,
referred to as the mixed domain. However, the mixed domain introduces a new
domain shift that needs to be addressed for the effective alignment of this
domain with the source and target nighttime domains. To overcome this issue,
certain constraints must be imposed. In this study, we propose a feature
prototype alignment approach to align the mixed domain with the source and
target nighttime domain. Note that we also align the source domain and target
nighttime domain in the same way.

To learn domain-invariant features, we aggregate pixel-level features for the
same object class across different domains. Firstly, we obtain prototypes of
each class by using the ground truth $Y_{s}$ from the source domain and mixed
pseudo label $Y^{\prime}_{m}$ from the mixed domain:

$\rho_{s}^{c}=\frac{\sum\limits_{h}^{H}\sum\limits_{w}^{W}f_{s}^{h,w}Y_{s}^{c,h,w}}{\sum\limits_{h}^{H}\sum\limits_{w}^{W}Y_{s}^{c,h,w}},\rho_{m}^{c}=\frac{\sum\limits_{h}^{H}\sum\limits_{w}^{W}f_{m}^{h,w}Y^{\prime
c,h,w}_{m}}{\sum\limits_{h}^{H}\sum\limits_{w}^{W}Y^{\prime c,h,w}_{m}},$ (8)

where $\rho_{s}^{c}$ and $\rho_{m}^{c}$ represent the prototypes of the source
domain and mixed domain for class $c$, while $f_{s}$ and $f_{m}$ denote the
source and mixed domain extracted by the student model $F_{s}$, respectively.
We use these prototypes to compute the cross-domain contrastive loss with the
features in another domain. Specifically, we select pixel-wise features and
the prototype of same classes between the source domain and mixed domain as
positive pairs, while others are set as negative pairs. This helps us to
maximize the pixel-prototype similarity within the same classes:

TABLE I: Comparison with the state-of-the-art methods and baseline models on
the Dark Zurich-test set. The best results are presented in bold, and the
second results are presented in underline.

Method |  road |  sidewallk |  building |  wall |  fence |  pole |  light |  sign |  vegetation |  terrain |  sky |  person |  rider |  car |  truck |  bus |  train |  motocycle |  bicycle | mIoU
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
RefineNet [40]-Cityscapes | 68.8 | 23.2 | 46.8 | 20.8 | 12.6 | 29.8 | 30.4 | 26.9 | 43.1 | 14.3 | 0.3 | 36.9 | 49.7 | 63.6 | 6.8 | 0.2 | 24.0 | 33.6 | 9.3 | 28.5
GCMA [9] | 81.7 | 46.9 | 58.8 | 22.0 | 20.0 | 41.2 | 40.5 | 41.6 | 64.8 | 31.0 | 32.1 | 53.5 | 47.5 | 75.5 | 39.2 | 0.0 | 49.6 | 30.7 | 21.0 | 42.0
MGCDA [10] | 80.3 | 49.3 | 66.2 | 7.8 | 11.0 | 41.4 | 38.9 | 39.0 | 64.1 | 18.0 | 55.8 | 52.1 | 53.5 | 74.7 | 66.0 | 0.0 | 37.5 | 29.1 | 22.7 | 42.5
DANNet [11]-RefineNet | 90.0 | 54.0 | 74.8 | 41.0 | 21.1 | 25.0 | 26.8 | 30.2 | 72.0 | 26.2 | 84.0 | 47.0 | 33.9 | 68.2 | 19.0 | 0.3 | 66.4 | 38.3 | 23.6 | 44.3
Bi-Mix [17] | 89.2 | 59.4 | 75.8 | 41.7 | 19.2 | 39.0 | 31.9 | 31.5 | 70.9 | 30.1 | 81.9 | 44.9 | 41.8 | 66.3 | 34.2 | 1.0 | 61.1 | 47.4 | 14.6 | 46.5
CCDistill [14] | 89.6 | 58.1 | 70.6 | 36.6 | 22.5 | 33.0 | 27.0 | 30.5 | 68.3 | 33.0 | 80.9 | 42.3 | 40.1 | 69.4 | 58.1 | 0.1 | 72.6 | 47.7 | 21.3 | 47.5
Ours (RefineNet) | 93.9 | 68.2 | 77.4 | 32.3 | 11.9 | 50.4 | 45.4 | 44.2 | 72.9 | 32.4 | 85.7 | 42.7 | 30.4 | 75.8 | 92.9 | 2.5 | 59.8 | 15.6 | 28.4 | 50.7
DeepLab-v2 [24]-Cityscapes | 79.0 | 21.8 | 53.0 | 13.3 | 11.2 | 22.5 | 20.2 | 22.1 | 43.5 | 10.4 | 18.0 | 37.4 | 33.8 | 64.1 | 6.4 | 0.0 | 52.3 | 30.4 | 7.4 | 28.8
DANNet [11]-DeepLab-v2 | 88.6 | 53.4 | 69.8 | 34.0 | 20.0 | 25.0 | 31.4 | 35.9 | 69.5 | 32.2 | 82.3 | 44.2 | 43.7 | 54.1 | 22.0 | 0.1 | 40.9 | 36.0 | 24.1 | 42.5
SePiCo [41] | 91.2 | 61.3 | 67.0 | 28.5 | 15.5 | 44.7 | 44.3 | 41.3 | 65.4 | 22.5 | 80.4 | 41.3 | 52.4 | 71.2 | 39.3 | 0.0 | 39.6 | 27.5 | 28.8 | 45.4
Ours (DeepLab-v2) | 94.3 | 68.9 | 77.4 | 37.4 | 17.5 | 46.6 | 45.6 | 41.2 | 67.6 | 35.4 | 82.3 | 53.7 | 47.7 | 79.6 | 71.3 | 6.7 | 71.7 | 40.8 | 34.5 | 53.7
DAFormer [42]-Cityscapes | 93.5 | 65.5 | 73.3 | 39.4 | 19.2 | 53.3 | 44.1 | 44.0 | 59.5 | 34.5 | 66.6 | 53.4 | 52.7 | 82.1 | 52.7 | 9.5 | 89.3 | 50.5 | 38.5 | 53.8
Refign [39]-DAFormer | 91.8 | 65.0 | 80.9 | 37.9 | 25.8 | 56.2 | 45.2 | 51.0 | 78.7 | 31.0 | 88.9 | 58.8 | 52.9 | 77.7 | 51.8 | 6.1 | 90.8 | 40.2 | 37.1 | 56.2
Ours (DAFormer) | 93.0 | 65.5 | 81.0 | 44.3 | 26.3 | 56.1 | 40.6 | 54.8 | 78.6 | 32.8 | 89.4 | 49.7 | 50.7 | 77.4 | 75.7 | 3.0 | 89.0 | 37.9 | 35.1 | 56.9
HRDA [43]-Cityscapes | 90.4 | 56.3 | 72.0 | 39.5 | 19.5 | 57.8 | 52.7 | 43.1 | 59.3 | 29.1 | 70.5 | 60.0 | 58.6 | 84.0 | 75.5 | 11.2 | 90.5 | 51.6 | 40.9 | 55.9
MIC [44]-HRDA | 94.8 | 75.0 | 84.0 | 55.1 | 28.4 | 62.0 | 35.5 | 52.6 | 59.2 | 46.8 | 70.0 | 65.2 | 61.7 | 82.1 | 64.2 | 18.5 | 91.3 | 52.6 | 44.0 | 60.2
Refign [39]-HRDA | 95.4 | 76.1 | 86.9 | 53.7 | 37.1 | 58.0 | 57.2 | 63.7 | 79.4 | 42.0 | 91.3 | 63.8 | 56.6 | 83.8 | 85.6 | 5.0 | 91.7 | 47.4 | 39.5 | 63.9
Ours (HRDA) | 95.2 | 75.1 | 85.0 | 52.0 | 36.2 | 60.9 | 56.9 | 63.5 | 80.2 | 38.8 | 91.8 | 62.5 | 57.3 | 86.8 | 85.6 | 25.2 | 92.1 | 51.6 | 42.1 | 65.2

Figure 4: Qualitative comparison between our approach and the state-of-the-art
methods on the Dark Zurich-val set.

$S_{m\rightarrow s}^{c}=(s(f_{m}^{h,w},\rho_{s}^{c,h,w})/\tau)\cdot W^{c},$
(9)

where $s(\cdot,\cdot)$ denotes cosine similarity, and $\tau$ is a temperature
parameter, while $W^{c}$ is the similarity weight of class $c$. We can
calculate the contrastive loss as follows.

$\mathcal{L}_{m\rightarrow s}=-\sum_{c}^{C}\sum_{h}^{H}\sum_{w}^{W}y^{\prime
c,h,w}_{n}log\frac{exp(S_{m\rightarrow s}^{c})}{\sum_{c}exp(S_{m\rightarrow
s}^{c})},$ (10)

here, $\mathcal{L}_{m\rightarrow s}$ represents the contrastive loss that
measures the distance between the source prototypes and mixup features.
Additionally, the prototypes of the mixed domain can effectively capture
positive pairs belonging to the same class in the source domain while
simultaneously repelling negative pairs.

$\begin{split}S_{s\rightarrow
m}^{c}=(s(f_{s}^{h,w},\rho_{m}^{c,h,w})/\tau)\cdot W^{c},\\\
\mathcal{L}_{s\rightarrow
m}=-\sum_{c}^{C}\sum_{h}^{H}\sum_{w}^{W}Y_{s}^{c,h,w}log\frac{exp(S_{s\rightarrow
m}^{c})}{\sum_{c}exp(S_{s\rightarrow m}^{c})}.\end{split}$ (11)

To address the issue of class imbalance, we propose an adaptive re-weighting
algorithm specifically for cases where the overlapping regions contain dynamic
and small objects. By assigning weights to these prototypes, we can adjust the
corresponding pixel-wise features accordingly. We set the weights of cosine
similarity as:

$W^{c}=\left\\{\begin{array}[]{ll}1,&\mbox{if }c\in C_{o}\\\ s+1/s&\mbox{if
}c\in C_{l}\\\ 0,&\mbox{otherwise }\end{array}\right.$ (12)

where $W$ denotes the weight of the contrastive loss, and $s$ represents the
number of overlapping classes. The set $C_{o}$ contains the overlapping
classes while not including the long-tailed classes, whereas $C_{l}$ comprises
the long-tailed classes that are also part of the overlapping classes. The
total prototype contrastive loss is:

$\mathcal{L}_{proto}=\mathcal{L}_{n\rightarrow s}+\mathcal{L}_{s\rightarrow
n}+\mathcal{L}_{m\rightarrow s}+\mathcal{L}_{s\rightarrow m}.$ (13)

###  III-D Objective functions

In addition to the loss functions described in Sec. III-B and Sec. III-C, we
also utilize cross-entropy loss to supervise our segmentation model with
labeled source images.

$\mathcal{L}_{sup}=-\sum^{H}_{h=0}\sum^{W}_{w=0}\sum^{C}_{c=0}\mathds{1}(Y_{s}^{c,h,w})log(y_{s}^{c,h,w}),$
(14)

where $\mathds{1}(\cdot)$ is the one-hot encoding operation.

The objective of the entire network is formulated as:

$\mathcal{L}=\mathcal{L}_{sup}+\alpha{\mathcal{L}_{mix}}+\beta{\mathcal{L}_{proto}}.$
(15)

where $\alpha$ and $\beta$ are hyper-parameters to balance three terms. In
this work, we set the values as follows: $\alpha=1.0$, $\beta=0.1$.

##  IV Experiments

###  IV-A Datasets

For all experiments, we use the mean of category-wise Intersection-over-union
(mIoU) as the evaluation metric. The following datasets are utilized for model
training and performance evaluation:

Cityscapes [45] This dataset contains 5,000 frames with pixel-level
annotations of 19 categories from street scenes in 50 cities. The images have
a resolution of 2,048 x 1,024 pixels. It consists of 2,975 training images,
500 validation images, and 1,525 testing images.

Dark Zurich [9] The Dark Zurich dataset includes 3,041 daytime images, 2,920
twilight images, and 2,416 nighttime images captured with coarse aligned in
three conditions. We utilize 2,416 day-night image pairs for training, 151 for
testing and 50 for validation.

Additionally, we evaluate our nighttime segmentation model on Nighttime
Driving test set [16], ACDC-night val set [46], and BDD100k-night dataset
[47].

###  IV-B Implementation details

We implement our method by using Pytorch-lightning on single Nvidia RTX A6000
GPU. We choose HRDA [43], DAFormer [42], DeepLabv2 [24] and RefineNet [40] as
our based methods. Both our models are trained by the AdamW [48] optimizer
with a weight decay of $1\times 10^{-2}$, and the initial learning rate is set
as $6\times 10^{-4}$, and linear learning rate warmup with warmup ratio
$10^{-6}$. Then the learning rate is decreased with the poly learning rate
policy with a power of 0.9. The batch size is set to 2. The total training
iterations are set to 40k. An EMA factor is $\alpha=0.999$. At the inference
time, there is no change to be introduced to the final model $F_{s}$. The
training resolution follows the used UDA methods (half resolution for
DAFormer, DeepLabv2, and RefineNet and full resolution for HRDA).

TABLE II: Comparison with state-of-the-art methods on Nighttime Driving test
sets and Dark Zurich-val sets. The best results are presented in bold. ‘D’,
means DeepLab-v2[24], ’R’ means RefineNet[40], and ’T’ means HRDA[43].

Method | Backbone | mIoU
---|---|---
Nighttime Driving | Dark Zurich-val | ACDC-night | BDD100k-night
RefineNet [40]-Cityscapes | R | 32.8 | 15.2 | 20.3 | 20.4
DeepLab-v2 [24]-Cityscapes | D | 25.4 | 12.1 | 16.3 | 17.3
GCMA [9] | R | 45.6 | 26.7 | - | 33.2
MGCDA [10] | R | 49.4 | 26.1 | - | 34.9
DANNet [11]-RefineNet | R | 42.4 | 30.0 | 37.0 | 30.3
CCDistill [14] | R | 46.2 | - | 37.3 | 33.0
Refign [39]-HRDA | T | 58.0 | 47.5 | 53.2 | 40.6
Ours (RefineNet) | R | 50.8 | 34.5 | 40.3 | 26.7
Ours (DeepLabv2) | D | 52.4 | 37.7 | 43.6 | 24.2
Ours (HRDA) | T | 58.4 | 48.5 | 53.8 | 40.3

###  IV-C Comparison with state-of-the-art methods

Comparison on Dark Zurich. We compare our proposed method with some existing
state-of-the-art methods, incuding DMAda [16], GCMA [9], MGCDA [10], DANNet
[11], Bi-Mix [17], CCDistill [14], and several other domain adaptation methods
[42, 43, 41, 44, 39]. In order to maintain a fair comparison, we employ a
consistent backbone when comparing each method and present the results of
corresponding source-only models. The outcomes are reported in Tab. I.

Our method delivers an increase in mIoU of approximately 1.3%, surpassing the
state-of-the-art method Refign (HRDA) [39]. Furthermore, our method
demonstrates significant performance improvements over other methods in
several categories, particularly in misaligned regions such as vegetation,
sky, car, truck, bus, train, and bicycle, where bus and bicycle are categories
with relatively fewer occurrences. While scores for other categories differ
slightly from the best scores, we achieve competitive performance in each
backbone. This significant performance with different backbones can also
verify the effectiveness of our method. We present qualitative results on Dark
Zurich-val in Fig. 4, which corroborate our quantitative results. The
qualitative results reveal that our method can accurately predict intricate
details of small and dynamic objects, such as the predictions of a person’s
legs in the first and second rows, and the motorcycle’s details in the third
row.

Comparison on Other Datasets. To confirm the generality of our approach, we
present the performance of our method and compare results with previous
methods on the Nighttime driving test set, ACDC-night val set, and
BDD100k-night set in Tab. II. Our method with HRDA achieves a 0.4% and 0.6%
increase in mIoU over the best score in Nighttime driving test set and ACDC-
night val set, respectively, while the performance of our method outperforms
other methods with the same backbone. In Fig 5 and Fig. 6, we provide
qualitative results of Nightime driving test set and ACDC-night val set that
clearly demonstrate the enhanced accuracy of our method in predicting the
details of dynamic and small objects. It should be noted that the
BDD100k-night test set has some limitations, including the absence of certain
classes and significant differences from our training datasets. Nevertheless,
our method still achieved the second-highest performance improvement on this
dataset. The qualitative results of BDD100k-night are present in Fig 7. It is
worth noting that the Nighttime driving test set has coarse annotation.
Despite this limitation, our method still achieves significant performance
improvements with different backbones.

Figure 5: Qualitative comparison between our approach and some existing state-
of-the-art methods on the Nighttime driving test set. Figure 6: Qualitative
comparison between our approach and the state-of-the-art methods on the ACDC-
night val set. Figure 7: Qualitative comparison between our approach and the
state-of-the-art methods on the BDD100k-night dataset.

###  IV-D Ablation study

In this section, we train several model variants for 40k iterations and test
them on Dark Zurich-test due to their fine-grained pixel-wise annotation and
completed categories.

Effectiveness of each component. we conduct extensive experiments to validate
the effectiveness of different components of our method with the same settings
and hyperparameters. The results are presented in Tab. III. Adding the
composite mask $M_{c}$ for input and label-level dynamic and small objects
refinement improved performance to 63.2 mIoU. Incorporating the long-tailed
bank further increased performance by 0.5%. The prototype loss $L_{proto}$ and
re-weighting strategy successfully addressed domain shifts and improved
alignment between domains.

TABLE III: The ablation study of each component.

Baseline [43] | DSR(w/o long-tailed bank) | DSR(full) | FPA(w/o re-weighting) | FPA(full) | mIoU
---|---|---|---|---|---
✓ |  | 55.9
✓ | ✓ |  | 63.2
✓ | ✓ | ✓ |  | 63.7
✓ | ✓ | ✓ | ✓ |  | 64.8
✓ | ✓ | ✓ | ✓ | ✓ | 65.2

Effectiveness of the DSR selected classes. To demonstrate the effectiveness of
our DSR module, we select different regions in our method and train them with
the same implementation for a fair comparison. The results are presented in
Tab. IV. Our method, combining randomly selected classes and misaligned
regions, achieved a +5.2% mIoU improvement for the baseline and +2.7% mIoU
improvement for the randomly selected classes method. This validates the
effectiveness of our domain adaptive nighttime semantic segmentation approach.

TABLE IV: The ablation results of selected mixup classes in DSR module.

Method |  | mIoU |  $\Delta$ mIoU ($\%$)
---|---|---|---
Baseline + FPA |  | 60.0 | -
\+ random classes [38] |  | 62.5 | +2.5
\+ random classes + small classes |  | 63.7 | +3.7
\+ random classes + dynamic classes |  | 64.1 | +4.1
Ours |  | 65.2 | +5.2

Effectiveness of the loss weight. We perform a sensitivity analysis on our
method’s hyperparameters $\alpha$ and $\beta$ to evaluate their impact on
performance, as presented in Tab. V. By varying the values of $\alpha$ and
$\beta$, we measure their effect on the mIoU. The results indicate that the
mIoU remains relatively stable when $\beta$ remains fixed and $\alpha$
increases significantly. Based on these results, we select the initial weights
of $\alpha=1$ and $\beta=0.1$. Additional experiments can be found in the
suppl. mat (project page).

TABLE V: Ablation results on loss weights. $\alpha$ and $\beta$.

$\beta$ $\alpha$ | 0.75 | 1.00 | 1.25 | 1.50
---|---|---|---|---
0.075 | 62.94 | 63.66 | 63.41 | 63.93
0.100 | 63.99 | 65.18 | 64.77 | 64.78
0.125 | 62.37 | 64.91 | 63.21 | 62.65

##  V Conclusion

In this paper, we proposed a novel UDA framework for nighttime segmentation by
focusing on dynamic and small object refinements and inherent domain shifts.
The DSR module combines the random selection of classes with dynamic and small
objects while introducing a long-tailed bank to store long-tailed objects to
augment the dynamic and small data, which are hard to segment due to poor
illumination. The FPA module further constrains the mixup process through
feature-level prototype alignment to reduce the domain shift in different
domains while improving the weights of dynamic and small objects during the
alignment process. Experimental results demonstrated the effectiveness of our
method.

## References

  * [1] M. Hofmarcher, T. Unterthiner, J. Arjona-Medina, G. Klambauer, S. Hochreiter, and B. Nessler, “Visual scene understanding for autonomous driving using semantic segmentation,” _Explainable AI: Interpreting, Explaining and Visualizing Deep Learning_ , pp. 285–296, 2019.
  * [2] M. Siam, M. Gamal, M. Abdel-Razek, S. Yogamani, M. Jagersand, and H. Zhang, “A comparative study of real-time semantic segmentation for autonomous driving,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2018, pp. 587–597.
  * [3] H. Blum, P.-E. Sarlin, J. Nieto, R. Siegwart, and C. Cadena, “Fishyscapes: A benchmark for safe semantic segmentation in autonomous driving,” in _proceedings of the IEEE/CVF international conference on computer vision workshops_ , 2019, pp. 1–10.
  * [4] M. Kerkech, A. Hafiane, and R. Canals, “Vine disease detection in uav multispectral images using optimized image registration and deep learning segmentation approach,” _Computers and Electronics in Agriculture_ , vol. 174, p. 105446, 2020.
  * [5] S. Liu, J. Cheng, L. Liang, H. Bai, and W. Dang, “Light-weight semantic segmentation network for uav remote sensing images,” _Ieee Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 14, pp. 8287–8296, 2021.
  * [6] Y. Zou, Z. Yu, B. Kumar, and J. Wang, “Unsupervised domain adaptation for semantic segmentation via class-balanced self-training,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 289–305.
  * [7] R. P. Poudel, S. Liwicki, and R. Cipolla, “Fast-scnn: Fast semantic segmentation network,” _arXiv preprint arXiv:1902.04502_ , 2019.
  * [8] V. Guizilini, J. Li, R. Ambruș, and A. Gaidon, “Geometric unsupervised domain adaptation for semantic segmentation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 8537–8547.
  * [9] C. Sakaridis, D. Dai, and L. V. Gool, “Guided curriculum model adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 7374–7383.
  * [10] C. Sakaridis, D. Dai, and L. Van Gool, “Map-guided curriculum domain adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [11] X. Wu, Z. Wu, H. Guo, L. Ju, and S. Wang, “Dannet: A one-stage domain adaptation network for unsupervised nighttime semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 15 769–15 778.
  * [12] L. Sun, K. Wang, K. Yang, and K. Xiang, “See clearer at night: towards robust nighttime semantic segmentation through day-night image conversion,” in _Artificial Intelligence and Machine Learning in Defense Applications_ , vol. 11169. SPIE, 2019, pp. 77–89.
  * [13] Q. Xu, Y. Ma, J. Wu, C. Long, and X. Huang, “Cdada: A curriculum domain adaptation for nighttime semantic segmentation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 2962–2971.
  * [14] H. Gao, J. Guo, G. Wang, and Q. Zhang, “Cross-domain correlation distillation for unsupervised domain adaptation in nighttime semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 9913–9923.
  * [15] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2223–2232.
  * [16] D. Dai and L. Van Gool, “Dark model adaptation: Semantic image segmentation from daytime to nighttime,” in _2018 21st International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2018, pp. 3819–3824.
  * [17] G. Yang, Z. Zhong, H. Tang, M. Ding, N. Sebe, and E. Ricci, “Bi-mix: Bidirectional mixing for domain adaptive nighttime semantic segmentation,” _arXiv preprint arXiv:2111.10339_ , 2021.
  * [18] H. Lee, C. Han, and S.-W. Jung, “Gps-glass: Learning nighttime semantic segmentation using daytime video and gps data,” _arXiv preprint arXiv:2207.13297_ , 2022.
  * [19] W. Liu, W. Li, J. Zhu, M. Cui, X. Xie, and L. Zhang, “Improving nighttime driving-scene segmentation via dual image-adaptive learnable filters,” _arXiv preprint arXiv:2207.01331_ , 2022.
  * [20] F. Shen, Z. Pataki, A. Gurram, Z. Liu, H. Wang, and A. Knoll, “Loopda: Constructing self-loops to adapt nighttime semantic segmentation,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2023, pp. 3256–3266.
  * [21] I. Alonso, A. Sabater, D. Ferstl, L. Montesano, and A. C. Murillo, “Semi-supervised semantic segmentation with pixel-level contrastive learning from a class-wise memory bank,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 8219–8228.
  * [22] W. Liu, D. Ferstl, S. Schulter, L. Zebedin, P. Fua, and C. Leistner, “Domain adaptation for semantic segmentation via patch-wise contrastive learning,” _arXiv preprint arXiv:2104.11056_ , 2021.
  * [23] J. Hoffman, D. Wang, F. Yu, and T. Darrell, “Fcns in the wild: Pixel-level adversarial and constraint-based adaptation,” _arXiv preprint arXiv:1612.02649_ , 2016.
  * [24] Y.-H. Tsai, W.-C. Hung, S. Schulter, K. Sohn, M.-H. Yang, and M. Chandraker, “Learning to adapt structured output space for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7472–7481.
  * [25] J. He, X. Jia, S. Chen, and J. Liu, “Multi-source domain adaptation with collaborative learning for semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 11 008–11 017.
  * [26] H. Ma, X. Lin, Z. Wu, and Y. Yu, “Coarse-to-fine domain adaptive semantic segmentation with photometric alignment and category-center regularization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 4051–4060.
  * [27] T. Isobe, X. Jia, S. Chen, J. He, Y. Shi, J. Liu, H. Lu, and S. Wang, “Multi-target domain adaptation with collaborative consistency learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 8187–8196.
  * [28] W. Wang, T. Zhou, F. Yu, J. Dai, E. Konukoglu, and L. Van Gool, “Exploring cross-image pixel contrast for semantic segmentation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 7303–7313.
  * [29] Y. Liu, W. Zhang, and J. Wang, “Source-free domain adaptation for semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 1215–1224.
  * [30] E. Romera, L. M. Bergasa, K. Yang, J. M. Alvarez, and R. Barea, “Bridging the day and night domain gap for semantic segmentation,” in _2019 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2019, pp. 1312–1318.
  * [31] J. Vertens, J. Zürn, and W. Burgard, “Heatnet: Bridging the day-night domain gap in semantic segmentation with thermal images,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 8461–8468.
  * [32] R. Xia, C. Zhao, M. Zheng, Z. Wu, Q. Sun, and Y. Tang, “Cmda: Cross-modality domain adaptation for nighttime semantic segmentation,” _arXiv preprint arXiv:2307.15942_ , 2023.
  * [33] X. Mao, Y. Ma, Z. Yang, Y. Chen, and Q. Li, “Virtual mixup training for unsupervised domain adaptation,” _arXiv preprint arXiv:1905.04215_ , 2019\.
  * [34] M. Xu, J. Zhang, B. Ni, T. Li, C. Wang, Q. Tian, and W. Zhang, “Adversarial domain adaptation with domain mixup,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 34, no. 04, 2020, pp. 6502–6509.
  * [35] Y. Wu, D. Inkpen, and A. El-Roby, “Dual mixup regularized learning for adversarial domain adaptation,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIX 16_. Springer, 2020, pp. 540–555.
  * [36] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 6023–6032.
  * [37] D. Walawalkar, Z. Shen, Z. Liu, and M. Savvides, “Attentive cutmix: An enhanced data augmentation approach for deep learning based image classification,” _arXiv preprint arXiv:2003.13048_ , 2020.
  * [38] V. Olsson, W. Tranheden, J. Pinto, and L. Svensson, “Classmix: Segmentation-based data augmentation for semi-supervised learning,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2021, pp. 1369–1378.
  * [39] D. Bruggemann, C. Sakaridis, P. Truong, and L. Van Gool, “Refign: Align and refine for adaptation of semantic segmentation to adverse conditions,” _arXiv preprint arXiv:2207.06825_ , 2022.
  * [40] G. Lin, A. Milan, C. Shen, and I. Reid, “Refinenet: Multi-path refinement networks for high-resolution semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1925–1934.
  * [41] B. Xie, S. Li, M. Li, C. H. Liu, G. Huang, and G. Wang, “Sepico: Semantic-guided pixel contrast for domain adaptive semantic segmentation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [42] L. Hoyer, D. Dai, and L. Van Gool, “Daformer: Improving network architectures and training strategies for domain-adaptive semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 9924–9935.
  * [43] ——, “Hrda: Context-aware high-resolution domain-adaptive semantic segmentation,” in _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXX_. Springer, 2022, pp. 372–391.
  * [44] L. Hoyer, D. Dai, H. Wang, and L. Van Gool, “Mic: Masked image consistency for context-enhanced domain adaptation,” _arXiv preprint arXiv:2212.01322_ , 2022\.
  * [45] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 3213–3223.
  * [46] C. Sakaridis, D. Dai, and L. Van Gool, “Acdc: The adverse conditions dataset with correspondences for semantic driving scene understanding,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 10 765–10 775.
  * [47] F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell, “Bdd100k: A diverse driving dataset for heterogeneous multitask learning,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 2636–2645.
  * [48] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” _arXiv preprint arXiv:1711.05101_ , 2017.


# Digging Into Normal Incorporated Stereo Matching

Zihua Liu  Tokyo Institute of TechnologyTokyoJapan<EMAIL_ADDRESS>,
Songyan Zhang  CAD Research Center, Tongji University.ShanghaiChina
<EMAIL_ADDRESS>,  Zhicheng Wang  CAD Research Center, Tongji
University.ShanghaiChina<EMAIL_ADDRESS>and  Masatoshi Okutomi
Tokyo Institute of TechnologyTokyoJapan<EMAIL_ADDRESS>
(2022)

###### Abstract.

Despite the remarkable progress facilitated by learning-based stereo-matching
algorithms, disparity estimation in low-texture, occluded, and bordered
regions still remains a bottleneck that limit the performance. To tackle these
challenges, geometric guidance like plane information is necessary as it
provides intuitive guidance about disparity consistency and affinity
similarity. In this paper, we propose a normal incorporated joint learning
framework consisting of two specific modules named non-local disparity
propagation(NDP) and affinity-aware residual learning(ARL). The estimated
normal map is first utilized for calculating a non-local affinity matrix and a
non-local offset to perform spatial propagation at the disparity level. To
enhance geometric consistency, especially in low-texture regions, the
estimated normal map is then leveraged to calculate a local affinity matrix,
providing the residual learning with information about where the correction
should refer and thus improving the residual learning efficiency. Extensive
experiments on several public datasets including Scene Flow, KITTI 2015, and
Middlebury 2014 validate the effectiveness of our proposed method. By the time
we finished this work, our approach ranked 1st for stereo matching across
foreground pixels on the KITTI 2015 dataset and 3rd on the Scene Flow dataset
among all the published works.

Surface Normal, Stereo Matching, Residual Learning, Spatial Propagation

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
30th ACM International Conference on Multimedia ; October 10–14, 2022; Lisboa,
Portugal.††booktitle: Proceedings of the 30th ACM International Conference on
Multimedia (MM ’22), October 10–14, 2022, Lisboa, Portugal††price:
15.00††isbn: 978-1-4503-9203-7/22/10††doi: 10.1145/3503161.3548312††ccs:
Computing methodologies††ccs: Computing methodologies Matching

Figure 1. Visualization of predicted disparity and surface normal on SceneFlow
and KITTI 2015 datasets.Our proposed NINet successfully preserves detailed
structure and consistency even in the challenging bordered, low-texture and
occluded regions.

##  1\. Introduction

Stereo matching has been a fundamental task in computer vision for decades of
years. Given a pair of rectified stereo images, the core of stereo matching is
to find the corresponding pixels along the epipolar line so that the disparity
can be calculated, which has been closely related to various applications like
robotic navigation, autonomous driving, and VR/AR.

Recently, the application of convolutional neural networks significantly
facilitates the development of stereo matching (Chang and Chen, 2018; Kendall
et al., 2017; Xu and Zhang, 2020; Zhang et al., 2021; Mayer et al., 2016).
Most of the learning-based methods can be divided into two categories
according to the construction of cost volume which is an essential step for
stereo matching as mentioned in (Scharstein et al., 2001). The pioneer works
(Mayer et al., 2016; Dosovitskiy et al., 2015) employ the inner production for
measuring the similarity between pixels and propose a lightweight correlation
layer for building the cost volume which is widely adopted in the following
works (Xu and Zhang, 2020; Wang et al., 2020; Liang et al., 2018). The
correlation layer takes advantage of efficient 2D convolution networks but
suffers from the loss of context information is inevitable as only a single
channel of feature is preserved. GCNet proposes to build a 4D cost volume
where the complete stereo features are directly concatenated along the
disparity dimension for abundant context information. Most of the top-
performance methods like (Guo et al., 2019; Zhang et al., 2019; Xu et al.,
2022) leverage this concatenation-based cost volume which follows 3D
convolution networks to improve stereo matching. Even though that learning-
based methods help the disparity estimation reach a new peak, ambiguous
matching still exists especially in bordered, occluded, and low-texture
regions, which remains a challenge. EDNet(Zhang et al., 2021) proposes a novel
method to learn an error-aware attention map which is utilized to guide the
residual learning concentrate on the inaccurate regions. However, there still
exists a limitation that the model only has intuitive information about where
the error occurs but lacks guidance about where the correction should be
learned from. Some previous works manage to improve the disparity estimation
with stronger geometric constraints (Qi et al., 2018; Kusupati et al., 2020;
Yin et al., 2019) by conducting a joint learning framework with the surface
normal, but most of them just simply enforce the consistency between normal
estimation and disparity estimation. The normal regression is usually
independent of the stereo matching which means the plane information fails to
be fully utilized by stereo matching.

To tackle the challenge of stereo matching in bordered, occluded, and low-
texture regions as well as explicitly leverage normal information, in this
paper, we propose a normal incorporated joint learning network named NINet.
Our idea lies that the relatively easier task of normal estimation can provide
abundant plane information as well as pattern affinity. And disparities within
the same plane should be consistent. To realize this, we propose two specific
modules which are the non-local disparity propagation(NDP) module and
affinity-aware residual learning(ARL) module, respectively. The former NDP
module leverages the estimated normal and computes a non-local affinity matrix
and a corresponding location offset indicating the sampled points for spatial
propagation at the disparity level. This operation can efficiently improve the
disparity estimation performance, especially in bordered and occluded regions.
Moreover, to mitigate the effect of low-texture regions, we provide another
ARL module to further correct the disparity estimation. Following EDNet (Zhang
et al., 2021), we introduce error based attention method with the modification
of utilizing the normal estimation to calculate a local affinity matrix. Using
the reweighted feature map based on the local affinity matrix to regress the
disparity residual. To sum up, our contributions can be summarized as follows:

  * •

We propose a normal incorporated joint learning framework to explicitly
leverage normal estimation and exploit plane information to provide intuitive
geometric guidance for disparity estimation.

  * •

We propose two specific modules: non-local spatial propagation (NDP) at the
disparity level and affinity-aware residual learning (ARL) at the feature
level. The former module efficiently improves the disparity estimation in both
bordered and occluded regions while the latter module primarily improves the
performance in low-texture regions.

  * •

We propose a novel method to generate dense pseudo normal ground truth on
sparsely annotated KITTI 2015 datasets which enables the application of our
method in real-world scenes.

  * •

We conduct extensive experiments on several public datasets including Scene
Flow, KITTI 2015, and Middlebury 2014. Our proposed NINet outperforms most of
the previous top-performance approaches and ranks 1st on KITTI 2015
leaderboard for foreground pixels. Our method demonstrates remarkable
generalizability as well, which further validates the effectiveness of our
method.

Figure 2. An overview of our proposed NINet. Our model is mainly composed of
two modules named ARL and NDP. Note that some skip-connection operations are
omitted here for simplifying the visualization.

##  2\. Related Works

In this section, we briefly review some pioneer learning-based works for
disparity estimation as well as giving an introduction about surface normal
incorporated stereo matching methods. Finally, we summarize the spatial
propagation mechanism utilized in this field.

###  2.1. Learning-based Stereo Matching

The pioneer learning-based work DispNetC(Mayer et al., 2016) follows the
previous work (Dosovitskiy et al., 2015) which introduces a correlation layer
to calculate the inner product of the corresponding pixels for measuring the
similarity. Many real-time approaches like (Wang et al., 2020; Xu and Zhang,
2020) takes this method for constructing the cost volume. To preserve abundant
context information which may be lost in the correlation operation, GCNet
(Kendall et al., 2017) employs the concatenation of the left and right
features along the disparity dimension to construct a 4D cost volume. The
concatenation-based cost volume following 3D convolution networks for
aggregation are widely adopted in many state-of-the-art works like (Zhang et
al., 2019; Wang et al., 2021b; Shen et al., 2021; Gu et al., 2020).
PSMNet(Chang and Chen, 2018) proposes a spatial pyramid pooling module for
incorporating global context information into image features along with
stacked hourglass 3D convolution networks for extending the context
information. As 3D convolution networks are highly memory-consuming, StereoNet
(Khamis et al., 2018) decreases the layer number of 3D convolution networks
and improves the performance by using RGB images for edge refinement. To
incorporate both the advantages of correlation-based cost volume and
concatenation-based volume, GwcNet(Guo et al., 2019) proposes the group-wise
correlation volume to form a 4D correlation-based cost volume which is then
concatenated with the concatenation cost volume. The recent remarkable work
ACVNet(Xu et al., 2022) introduces an attention-based strategy to suppress
redundant information in the cost volume which achieves top performance and
real-time efficiency.

###  2.2. Normal Assisted Depth Estimation

Many indoor depth estimation utilizes the normal as an assisted geometry
constraints (Zhang et al., 2019; Kusupati et al., 2020; Wang et al., 2015; Qi
et al., 2018; Rossi et al., 2020). GeoNet (Qi et al., 2018) proposes a normal-
to-depth branch and a depth-to-normal branch for jointly learning. VNL (Yin et
al., 2019) designs a loss term named virtual normal directions to impose
geometric constraints. The work (Kusupati et al., 2020) is also a joint
learning-based model which proposes a consistency loss to refine the depth
from depth/normal pairs. An energy optimization-based approach is introduced
in (Zhao et al., 2021) for multi-view stereo matching. IDNSolver (Zhao et al.,
2021) proposes an iterative energy optimization strategy which is a novel view
for multi-view stereo matching. The ASNDepth (Long et al., 2021) introduces an
adaptive surface normal constraint to effectively correlate the depth
estimation with geometric consistency. Most related works follow the pipeline
of enforcing the consistency between the depth and the normal to impose more
implicit geometric constraints.

###  2.3. Spatial Propagation for Disparity

Spatial propagation is a popular mechanism in depth completion whose input
data is a sparse depth map generated by the sensor of LiDAR. Representative
works are (Wang et al., 2021b; Park et al., 2020; Xu et al., 2020). When it
comes to the depth estimation task, PAPNet(Zhang et al., 2019) conducts two
types of propagation which are task-specific propagation and cross-task
propagation, respectively. The cross-task propagation integrates affinity from
different tasks including semantic segmentation, surface normal estimation,
and depth estimation while the task-specific propagation performs an iterative
diffusion in the feature space to widely spread the cross-task affinity
patterns. The recent work PatchMatchNet (Wang et al., 2021a) introduces an
iterative multi-scale patchmatch algorithm to learn adaptive propagation for
multi-view stereo matching. HitNet (Tankovich et al., 2021) imitates its
predecessor (Michael Bleyer and Rother, 2011) and performs spatial propagation
with the assistance of predicted slanted plane hypotheses.

##  3\. Methodology

In this section, we will provide a thorough introduction to our proposed
normal incorporated stereo matching network(NINet) in the following order: the
architecture of our model is demonstrated in Subsection 3.1 following the
instruction of normal estimation subnet in Subsection 3.2. Details about our
proposed two modules can be found in Subsection 3.3 and Subsection 3.4. We
describe our multi-scale training mechanism in Subsection3.5.

Figure 3. Illustration of local spatial propagation (a), non-local spatial
propagation (b). (c) shows sampled points in low-texture regions. (d)
demonstrates sampled points at edges. (e) shows sampled points in occluded
regions. Red points indicated the selected ones for propagating disparity to
the targeted white/green point. It’s obvious that our method successfully
learns to dynamically sample points for propagation according to different
patterns.

###  3.1. Network Architecture

The overview architecture of our proposed NINet is shown in Figure 2. We
proposed a normal incorporated joint learning framework for surface normal and
disparity estimation. A weight-sharing 2D residual blocks are used for feature
extraction from stereo images which are then fed into two parallel branches
for normal estimation and disparity estimation, respectively. The normal
estimation will be discussed in Subsection 3.2. As for the stereo matching
process, we imitate (Wang et al., 2020; Mayer et al., 2016) and employ inner
production for calculating the similarity between pixels. Given a pair of
stereo feature maps, $i.e.$ $f_{l}$ and $f_{r}$ $\in R^{H\times W\times C}$,
the correlation is computed as :

(1) $\displaystyle Coor(d,x,y)=\frac{1}{C}<f_{l}(x-d,y),f_{r}(x,y)>,$

where $<A,B>$ indicates the inner production between $A$ and $B$. The
aggregation part takes a shallow UNet-like network which follows our proposed
NDP and ARL modules for disparity estimation at multi scales.

###  3.2. Normal Estimation Sub-Net

Since there is a certain degree of the geometric relationship between
disparity and surface normal, we directly use the network structure of
DispNetC(Mayer et al., 2016) to estimate the surface normal and use the
coarse-to-fine strategy to obtain the surface normal prediction results at
different scales with a normalization at the feature dimension. For the reason
that Sceneflow and KITTI datasets do not provide ground truth for surface
normal, we use a Sobel operator to obtain plane information as the supervision
information. See Section 4.2 for the different processing methods of different
datasets.

###  3.3. Non-Local Disparity Propagation

Inspired by the spatial propagation mechanism utilized in the depth completion
task[4], we introduce a normal incorporated non-local disparity propagation
module in which we hub NDP to generate non-local affinities and offsets for
spatial propagation at the disparity level. The motivation lies that the
sampled pixels for edges and occluded regions are supposed to be selected. The
propagation process aggregates disparities via plane affinity relations, which
alleviates the phenomenon of parallax blurring at object edges due to frontal
parallel windows. And the disparities in occluded areas are also be optimized
at the same time by being propagated from non-occluded areas where the
predicted disparities with high confidence. The architecture of NDP module can
be seen in Figure 2.

In the NDP module, the coarse disparity and its corresponding warped error are
firstly utilized to obtain the disparity confidence map through the confidence
estimation branch. A sigmoid activation function is used to generate the
disparity confidence map. This confidence map can reflect the reliability of
the current disparity which benefits the spatial propagation process.

For the disparity spatial propagation, previous works such as (Wang et al.,
2021b) use a local fixed $3\times 3$ window to consider all the possible
directions of neighbor pixels for disparity aggregation. Its process can be
shown in Figure3.a. For a specific location $p$, we have its 8-closest
neighbor $N_{p}$:

(2) $\displaystyle
N_{p}=\left\\{p+p_{n}|p_{n}\in\left\\{(-1,-1),(-1,0)...(1,0),(1,1)\right\\}\right\\}$

After we obtain the affinity map $A\in R^{H\times W\times 8}$ and confidence
map $C\in R^{H\times W\times 1}$. The local spatial propagation for a specific
position $p$ can be described as follows:

(3) $\displaystyle\tilde{d}_{p}=\sum_{n\in N_{p}}(A_{p}(n)\times c_{n}\times
d_{n})+(1-\sum_{n\in N_{p}}(A_{p}(n)\times c_{n})\times d_{p})$

where $n$ enumerates the locations in $N_{p}$. $A_{p}(n)$, $c_{n}$ is the
corresponding affinity and confidence. $d_{p}$ is the initial disparity, and
$\tilde{d_{p}}$ is the refined disparity.

Unlike conventional spatial propagation using fixed propagation window, in
NDP, we directly use the concatenation of pixel-level information to provide
the guidance to learn an additional offset $\Delta p$ and its affinities for
each 8 closest neighbors. By applying the non-local offset, the propagation
kernel can break through the limitation of the fixed window, and thus enhances
the ability to determine the position of the propagation neighborhood
adaptively which can be seen in Figure3.b. The new 8 neighbors can be defined
as:

(4) $\displaystyle\tilde{N}_{p}=\\{n+\Delta p_{n}|n\in N_{p},\Delta
p_{n}=f_{\theta}(I_{l},I_{r},S,E)\\}$

Here $\tilde{N}_{p}$ is the non-local 8 closest neighbour, $f$ is the residual
block in Figure 3 with a learnable parameter $\theta$. The network takes left
image $I_{l}$, right image $I_{r}$, estimated surface normal $S$, and warped
error $E$ as the input to provide the guidance information. Here the surface
normal indicates the geometry consistency relations, and the warped error $E$
reveals the locations of the inaccurate and occluded areas, which is
beneficial to increase the directivity and efficiency of the spatial
propagation process. Besides, the residual block also predicts the non-local
affinities to manifest the similarities between non-local neighbor’s
disparities and the target disparity. To enhance the stability of the
propagation, we apply confidence incorporated affinity normalization by
multiplying the predicted non-local affinity values with their corresponding
confidence value and then be divided by the absolute sum of all 8 non-local
affinities at the target position. For one specific location $p$, the
normalized affinity $\tilde{A}_{p}^{norm}$ is:

(5)
$\displaystyle\tilde{A}_{p}^{norm}=\left\\{\frac{a_{i}c_{i}}{\sum_{i=1}^{8}|a_{i}c_{i}|}|a_{i}\in\tilde{A}_{p},c_{i}\in
C\right\\}$

By using the non-local neighbour $\tilde{N}$ and normalized non-local affinity
$\tilde{A}_{p}^{norm}$, we apply non-local disparity propagation to refine the
disparities. Compared with Equation 3 ,the non-local propagation process
becomes:

(6)
$\displaystyle\tilde{d}_{p}=\sum_{n\in\tilde{N}_{p}}(\tilde{A}_{p}^{norm}(n)\times
d_{n})+(1-\sum_{n\in\tilde{N}_{p}}\tilde{A}_{p}^{norm}(n))\times d_{p}$

As it shows that we also assign a dynamic weight to the initial disparity
which equals 1 minus the summation of the affinities of its 8 non-local
neighboring points.

Figure 3 (c)(d)(e) show how our proposed NDP module decides non-local neighbor
of specific positions when dealing with edges, low texture, and occluded
regions with the guidance of surface normal and warped error. Thus, the
proposed NDP module can refine the disparities in these ill positions by non-
local propagation.

###  3.4. Affinity-Aware Residual Learning

The non-local disparity propagation module primarily concerns the disparity
estimation in bordered and occluded regions, but stereo matching in low-
texture regions remains a bottleneck. To mitigate this problem, we imitate the
idea from EDNet(Zhang et al., 2021) to apply error based spatial attention
strategy for efficient residual learning and obtain the error concentrated
feature map $\hat{F_{e}}\in R^{H\times W\times C}$. Given the concatenation of
stereo images, warped error predicted disparity, and surface normal map, a
block of 2D convolution network is introduced to obtain an error-aware feature
map $F_{e}$. A sigmoid function is then used for calculating an error-based
spatial attention map. The error concentrated feature map $\hat{F}_{e}$ is
computed as:

(7) $\displaystyle\hat{F_{e}}=F\odot(\sigma{F_{e}}),$

where $F$ is the original feature map, $\odot$ represents element-wise
product, and $\sigma()$ is sigmoid function.

However, it’s insufficient to know where mismatching occurs as the model still
lacks intuitive guidance about where to refer for better refinement.
Therefore, with the concatenation of stereo images, predicted disparity,
warped error map, and normal estimation, we conduct a residual block to
calculate a normalized local affinity matrix $A\in R^{H\times W\times k\times
k}$ within a $k\times k$ neighboring region for each pixel. The error-
concentrated feature map is further re-weighted with this affinity matrix. The
final local affinity-aware feature map is calculated as follows:

(8)
$\displaystyle\tilde{F}_{i,j}=\sum_{a,b=-(k-1)/2}^{(k-1)/2}\hat{F}_{i+a,j+b}\odot
A_{i,j}(a,b),$

where $\odot$ represents the element-wise product. Assisted by this normal
guided local affinity matrix, those mismatched pixels in low-texture regions
can be corrected by referring to their neighboring regions with similar
affinity. The re-weighted feature map is then fed into a deformable hourglass
network for further aggregation and get an affinity-aware disparity $r_{a}$ to
obtain the updated disparity estimation.

###  3.5. Multi-Scale Loss Function

Our network is trained in an end-to-end manner supervised by ground truth
disparity and generated ground truth surface normal at multi scales. Disparity
loss $L_{s}^{d}$ and surface normal loss $L_{s}^{n}$ at scale $s$ are computed
as:

(9) $\displaystyle
L_{s}^{d}=\frac{1}{N}\sum_{i=1}^{N}{SmoothL1(d_{i}^{s}-\hat{d_{i}^{s}})}$ (10)
$\displaystyle
L_{s}^{n}=\frac{1}{N}\sum_{i=1}^{N}{SmoothL1(n_{i}^{s}-\hat{n_{i}^{s}})}$

where $N$ is the number of elements for the predicted map. $\hat{d_{i}^{s}}$
and $\hat{n_{i}^{s}}$ indicate the $i^{th}$ element of the estimated disparity
$\hat{d^{s}}$ and estimated surface normal $\hat{n^{s}}$ while $d^{s}$ and
$n^{s}$ represent the ground truth for disparity and surface normal,
respectively. The utilized Smooth L1 loss is defined as follows:

(11) $\displaystyle SmoothL1(x)=\begin{cases}0.5x^{2},&if\lvert x\rvert<1\\\
\lvert x\rvert-0.5,&otherwise\end{cases}$

To supervise the confidence $c$ used in the NDP module, we apply the loss
function in (Tankovich et al., 2021):

(12) $\displaystyle
L_{s}^{c}(c)=max(1-c,0)_{\tau\lvert{d_{i}^{s}-\hat{d_{i}^{s}}}\rvert<C_{1}}+max(c,0)_{\tau\lvert{d_{i}^{s}-\hat{d_{i}^{s}}}\rvert>C_{2}}$

where $\tau$ is an indicator function which equals 1 when the condition is
satisfied and 0 otherwise. The function aims at increasing the confidence if
the distance between predicted disparity and ground truth is smaller that a
threshold $C_{1}$ and decreasing the confidence if the distance is larger than
a threshold $C_{2}$. In our work, we set $C_{1}=0.5$ and $C_{2}=1.5$.

The final loss function is a combination of losses for the disparity, surface
normal, and confidence for all pixels across 4 scales:

(13) $\displaystyle
L^{total}=\sum_{s=0}^{3}\lambda_{s}(\lambda_{d}L_{s}^{d}+\lambda_{n}L_{s}^{n}+\lambda_{c}L_{s}^{c})$

where $\lambda_{s}$ is a scalar for adjusting the loss weight at scale $s$.
$\lambda_{d}$, $\lambda_{n}$ and $\lambda_{c}$ are coefficients for disparity,
surface normal and confidence losses, respectively. More details are provided
in Section 4.3.

##  4\. Experiment

In this section, we will first introduce the training datasets as well as the
detailed training scheme. Then we provide a thorough quantitative analysis of
our proposed model on the corresponding datasets.

###  4.1. Datasets and Evaluation Metrics

We validate the effectiveness on three public datasets which are Scene Flow
(Mayer et al., 2016), KITTI 2015 (Menze and Geiger, 2015), and Middlebury
(Scharstein et al., 2014). The Scene Flow dataset consists of 39,824 pairs of
synthetic stereo RGB images with a full resolution of 960$\times$540 among
which 35,454 pairs are split as the training set and the rest 4,370 pairs are
split as the testing set. To facilitate the research of autonomous driving,
the KITTI 2015 collects 400 pairs of real scenes with sparse depth ground
truth annotated by LiDAR, where 200 pairs are for training and the other 200
pairs are utilized for testing. Only 33 pairs of stereo images are available
for the MiddleBurry2014 dataset so we conduct a generalization evaluation on
it. The standard end-point error (EPE) is reported on the Scene Flow dataset
and we follow the standard evaluation protocol to submit our predicted results
to the KITTI Benchmark from where the D1 error is obtained.

###  4.2. Normal Estimation Supervision

For the Scene Flow dataset, the provided disparity labels are pixel-wise
annotated which makes it possible to directly introduce the Sobel kernel for
calculating gradient along $x$ and $y$ axis, respectively. The primary
challenge comes from the KITTI datasets because of the sparse disparity
annotations. To tackle this problem, we first leverage the state-of-the-art
model LEAStereo (Cheng et al., 2020) to generate the dense pseudo disparity
labels following the Sobel kernel to calculate the pseud normal ground truth
$N_{pseudo}$. Then we search for the local region where the disparity ground
truth is available for the neighboring 8 pixels to obtain the normal ground
truth $N_{sparse}$ as well as the ground truth position mask $M_{sparse}$. The
final normal ground truth $N_{gt}$ is then calculated as follows:

(14) $\displaystyle N_{gt}=N_{pseudo}*(1-M_{sparse})+N_{sparse}*M_{sparse}$

To fully exploit the limited sparse normal information and alleviate the
influence of pseudo disparity of low quality, we further calculate the end-
point error between the pseudo disparity labels and the sparse disparity
ground truth to obtain an index $E_{index}$ where the EPE is greater than 1
disparity. The loss of normal estimation $L_{N}$ is computed as follows:

(15) $\displaystyle L_{N}=$ $\displaystyle S(N_{gt},\hat{N})*M_{sparse}*1.5$
$\displaystyle+S(N_{gt},\hat{N})*(1-M_{sparse})*E_{index}*1.0$
$\displaystyle+S(N_{gt},\hat{N})*(1-M_{sparse})*(1-E_{index})*0.5,$

where $S(x,y)$ denotes the SmoothL1 loss function mentioned before and
$\hat{N}$ represents the predicted normal. Our generated surface normal ground
truth is shown in Figure 4

Figure 4. Visualization of our generated surface normal ground truth on KITTI
2015. Row one is the left image, row two is the gt sparse disparity, row three
is the sparse surface normal generated with neighbour-existing disparities.The
last row is our generated disparity with the assistance of pseudo disparity.

###  4.3. Implementation Details

We implemented our NINet by PyTorch(Paszke et al., 2019) and trained our model
with Adam(momentum=0.9, beta=0.999) as the optimizer. We randomly cropped the
input to 320$\times$640 on the KITTI 2015 $\&$ 2012 and the Scene Flow
datasets. The whole training process is performed on 2 NVIDIA RTX 2080ti GPUs
with a batch size of 8. The max disparity search range is set as 192 pixels.
The coefficients of four disparity losses at each scale are correspondingly
set as $\lambda_{0}=1.0$, $\lambda_{1}=\lambda_{2}=0.8$, and
$\lambda_{3}=0.6$. The weights for disparity, surface normal and confidence
are set as $\lambda_{d}=5$, $\lambda_{n}=50$, $\lambda_{n}=1$.

For the Scene Flow dataset, we trained the networks for 150 epochs in total
with an initial learning rate of 1e-3. After the first 20 epochs of training,
we decrease our learning rate to 4e-4,1e-4,4e-5 at the epoch of 20, 40, 100.
Note that to enhance the reliability of spatial propagation guided by normal,
the normal estimation branch is trained independently for 40 epochs before the
joint learning of disparity and normal.

For the KITTI 2015 and the KITTI 2012 datasets, we trained our networks based
on the model pre-trained on the Scene Flow dataset. We first mixed these two
datasets with $300$ pairs of images in total for the first $200$ epochs of
training to obtain the model with the best performance on the validation set.
Another $200$ epochs of fine-tuning are employed on the KITTI 2015 and the
KITTI 2012 datasets respectively. The learning rate is constantly set as 1e-5
for the whole training process.

Table 1. Ablation study of our proposed method on the Scene Flow dataset. The best performance is stressed in bold. Our proposed affinity-aware residual learning and non-local spatial propagation modules significantly improve the disparity estimation. The best results are stressed in bold. Our proposed modules efficiently improve the stereo matching performance. Method | NR Branch | Affinity-aware Residual | Spatial Propagation | EPE | P1 Error | P3 Error
---|---|---|---|---|---|---
attention | dynamic filtering | local | non-local
Baseline |  |  |  |  |  | 0.96 | 12.65$\%$ | 5.94$\%$
Baseline-R | ✓ |  |  |  |  | 0.73 | 9.58$\%$ | 4.72$\%$
Baseline-RD | ✓ |  | ✓ |  |  | 0.66 | 8.06$\%$ | 4.42$\%$
Baseline-RA | ✓ | ✓ |  |  |  | 0.63 | 7.90$\%$ | 4.18$\%$
Baseline-ARL | ✓ | ✓ | ✓ |  |  | 0.60 | 7.42$\%$ | 4.00$\%$
Baseline-LDP | ✓ |  |  | ✓ |  | 0.65 | 8.02$\%$ | 4.20$\%$
Baseline-NDP | ✓ |  |  |  | ✓ | 0.60 | 7.39$\%$ | 3.98$\%$
Baseline-ARL-LDP | ✓ | ✓ | ✓ | ✓ |  | 0.54 | 6.65$\%$ | 3.71$\%$
Baseline-ARL-NDP | ✓ | ✓ | ✓ |  | ✓ | 0.51 | 6.17$\%$ | 3.53$\%$
Table 2. Comparison of our proposed method and other remarkable works on the Scene Flow dataset. Among all the competing methods, our NINet ranks the 3rd for end-point error. The best performance is emphasized in bold. Model | DispNetC(Mayer et al., 2016) | PSMNet(Chang and Chen, 2018) | GwcNet(Guo et al., 2019) | EDNet(Zhang et al., 2021) | HITNet(Tankovich et al., 2021) | LeaStereo(Cheng et al., 2020) | CSPN(Wang et al., 2021b) | ACVNet(Xu et al., 2022) | NINet (Ours)
---|---|---|---|---|---|---|---|---|---
EPE | 1.68 | 1.09 | 0.76 | 0.63 | 0.41 | 0.78 | 0.78 | 0.48 | 0.51
Table 3. Benchmark results on KITTI 2015 test sets. “Noc” and “All” indicate the percentage of outliers averaged over ground truth pixels of non-occluded and all regions, respectively. “fg” and “all” indicate the percentage of outliers averaged over the foreground and all ground truth pixels, respectively. Our NINet achieves the best performance on the foreground pixels with a competitive inference speed. Method | Noc (%) | All (%) | Time (s)
---|---|---|---
fg | all | fg | all
GANet (Zhang et al., 2019) | 3.37 | 1.73 | 3.82 | 1.93 | 0.36
GCNet (Kendall et al., 2017) | 5.58 | 2.61 | 6.16 | 2.87 | 0.9
PSMNet (Chang and Chen, 2018) | 4.31 | 2.14 | 4.62 | 2.32 | 0.41
GwcNet (Guo et al., 2019) | 3.49 | 1.92 | 3.93 | 2.11 | 0.32
SegStereo (Yang et al., 2018) | 3.70 | 2.08 | 4.07 | 2.25 | 0.6
HD3 (Yin et al., 2019) | 3.43 | 1.87 | 3.63 | 2.02 | 0.14
Bi3D (Badki et al., 2020) | 3.11 | 1.79 | 3.48 | 1.95 | 0.48
AANet (Xu and Zhang, 2020) | 4.93 | 2.32 | 5.39 | 2.55 | 0.075
DispNetC (Mayer et al., 2016) | 3.72 | 4.05 | 4.41 | 4.34 | 0.06
FADNet (Wang et al., 2020) | 3.07 | 2.59 | 3.50 | 2.82 | 0.05
EDNet(Zhang et al., 2021) | 3.33 | 2.31 | 3.88 | 2.53 | 0.05
ACVNet(Xu et al., 2022) | 2.84 | 1.52 | 3.07 | 1.65 | 0.20
CSPN(Wang et al., 2021b) | 2.67 | 1.61 | 2.88 | 1.74 | 1.0
HITNet (Tankovich et al., 2021) | 2.72 | 1.74 | 3.20 | 1.98 | 0.02
LEAStereo(Cheng et al., 2020) | 2.65 | 1.51 | 2.91 | 1.65 | 0.30
NINet (Ours) | 2.53 | 1.83 | 2.80 | 1.98 | 0.17
Figure 5. Comparison among our proposed methods with different settings on the
Scene Flow testing set as well as other SOTA works. It is obvious that our
proposed modules significantly facilitate the speed of convergence. Figure 6.
Visualization of the ablation study on Scene Flow dataset. With the assistance
of our proposed modules, thin structures and more consistent disparity can be
preserved.

###  4.4. Ablation Study

Based on the performance shown in Table 1, it can be directly analyzed that
our proposed non-local disparity propagation and affinity-aware residual
learning modules effectively improve the performance. The joint learning
scheme with surface normal improves the baseline with an obvious margin. The
employment of the affinity-aware residual learning module which combines both
attention and dynamic filtering operations significantly decreases the EPE to
0.60. The disparity level non-local propagation further improves the result as
reflected by the EPE of baseline-ARL-NDP, which is 0.51. The second column
provides straightforward evidence that our NINet successfully preserves the
detailed outline of the shelf with consistent disparity while other settings
fail to reach this result. To further provide a more comprehensive
interpretation of the superiority of our method, we visualize the converging
speed of our methods as well as other competing approaches as shown in Figure
5. With the assistance of the proposed modules, our method not only achieves
the lower initial EPE but outperforms other works at the corresponding epochs
as well. Note that, the competitive result of GwcNet is mainly attributed to
the application of 3D convolution which is discarded in our work.

Table 4. Evaluation for bordered and occluded regions on the Scene Flow dataset with our proposed modules. Method | Bordered EPE | Occluded EPE
---|---|---
Baseline | 1.95 | 2.31
+ARL | 1.63 | 1.92
+NDP | 1.44 | 1.81
+NDP+ARL | 1.31 | 1.71
Table 5. Generalization evaluation on the MiddleBurry 2014 dataset. Our NINet achieves the best generalizability. Method | Bad2.0 | Bad4.0 | RMSE | AvgErr
---|---|---|---|---
Occ | Non | Occ | Non | Occ | Non | Occ | Non
AANet | 38.6 | 42.5 | 25.3 | 28.9 | 23.3 | 21.4 | 12.4 | 10.7
EDNet | 69.5 | 66.8 | 57.1 | 54.0 | 21.9 | 20.4 | 44.2 | 41.2
GwcNet | 34.9 | 30.8 | 22.7 | 19.4 | 27.0 | 25.4 | 13.4 | 11.8
FADNet | 31.3 | 27.0 | 20.9 | 17.1 | 21.8 | 19.6 | 11.2 | 9.5
NINet(Ours) | 24.5 | 20.2 | 15.8 | 12.5 | 18.8 | 18.3 | 8.0 | 7.8

###  4.5. Performance Evalution

In this subsection, we will mainly focus on the comparison between our method
and other top-performing methods.

Scene Flow As shown in Table 2, our method outperforms most of the superior
approaches including LEAStereo (Cheng et al., 2020) and ranks 3rd among all
the competing works. As illustrated in Figure 7, our method produces sharper
and more detailed disparity estimation. A representative instance comes from
the third column that our NINet achieves consistent disparity estimation for
the plank while results from other methods are less convincing. Table 4
specifically reveals the effectiveness of proposed modules in both bordered
and occluded regions.

KITTI 2015 Table 3 demonstrates the performance of several remarkable methods
in the KITTI 2015 leaderboard. When evaluating disparity estimation in
foreground areas, our approach even outperforms the HITNet by 6.99$\%$ in non-
occluded regions and ranks 1st among all the published works, which strongly
validates the effectiveness of our model.

MiddleBurry 2014 Generalizability is also a necessary indicator when
comprehensively evaluating learning-based stereo matching methods and thus we
conduct a generalization experiment on MiddleBurry 2014 dataset. As shown in
Table 5, our proposed method obtains the best performance when transferring to
a new data domain without finetuning. Compared with 3D convolution-based
GwcNet, our method decreases the average error from 13.4 to 8.0 and 11.8 to
7.8 in occluded regions and non-occluded, respectively.

Figure 7. SceneFlow performance compares with other superior works. Figure 8.
Generalization ability on MiddlyBurry 2014 dataset.

##  5\. Conclusion

In this paper, we propose a normal incorporated joint learning framework named
NINet to explicitly leverage the plane information for intuitive geometric
guidance. A non-local spatial propagation (NDP) module is also introduced to
dynamically select points with a similar affinity for further propagation at
the disparity level. An affinity-aware residual learning (ARL) module is then
designed to impose consistency within the same plane according to a local
affinity matrix. Extensive experiments on several datasets have demonstrated
the effectiveness as well as the generalization ability of our proposed
mothed. By the time we finished it, our method ranked 1st on the KITTI 2015
dataset for foreground objects.

## References

  * (1)
  * Badki et al. (2020) A. Badki, A. Troccoli, K. Kim, J. Kautz, P. Sen, and O. Gallo. 2020.  Bi3D: Stereo Depth Estimation via Binary Classifications. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 1597–1605.
  * Chang and Chen (2018) J. Chang and Y. Chen. 2018.  Pyramid Stereo Matching Network. In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5410–5418.
  * Cheng et al. (2020) Xuelian Cheng, Yiran Zhong, Mehrtash Harandi, Yuchao Dai, Xiaojun Chang, Hongdong Li, Tom Drummond, and Zongyuan Ge. 2020.  Hierarchical Neural Architecture Search for Deep Stereo Matching. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 22158–22169.  https://proceedings.neurips.cc/paper/2020/file/fc146be0b230d7e0a92e66a6114b840d-Paper.pdf
  * Dosovitskiy et al. (2015) A. Dosovitskiy, P. Fischer, E. Ilg, P. Häusser, C. Hazirbas, V. Golkov, P. v. d. Smagt, D. Cremers, and T. Brox. 2015.  FlowNet: Learning Optical Flow with Convolutional Networks. In _2015 IEEE International Conference on Computer Vision (ICCV)_. 2758–2766.
  * Gu et al. (2020) Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan. 2020.  Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 2492–2501.  https://doi.org/10.1109/CVPR42600.2020.00257
  * Guo et al. (2019) X. Guo, K. Yang, W. Yang, X. Wang, and H. Li. 2019.  Group-Wise Correlation Stereo Network. In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 3268–3277.
  * Kendall et al. (2017) A. Kendall, H. Martirosyan, S. Dasgupta, P. Henry, R. Kennedy, A. Bachrach, and A. Bry. 2017.  End-to-End Learning of Geometry and Context for Deep Stereo Regression. In _2017 IEEE International Conference on Computer Vision (ICCV)_. 66–75.
  * Khamis et al. (2018) Sameh Khamis, Sean Fanello, Christoph Rhemann, Adarsh Kowdle, Julien Valentin, and Shahram Izadi. 2018.  StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction. In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Kusupati et al. (2020) Uday Kusupati, Shuo Cheng, Rui Chen, and Hao Su. 2020.  Normal Assisted Stereo Depth Estimation. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 2186–2196.  https://doi.org/10.1109/CVPR42600.2020.00226
  * Liang et al. (2018) Z. Liang, Y. Feng, Y. Guo, H. Liu, W. Chen, L. Qiao, L. Zhou, and J. Zhang. 2018.  Learning for Disparity Estimation Through Feature Constancy. In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2811–2820.
  * Long et al. (2021) Xiaoxiao Long, Cheng Lin, Lingjie Liu, Wei Li, Christian Theobalt, Ruigang Yang, and Wenping Wang. 2021.  Adaptive Surface Normal Constraint for Depth Estimation. In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_. 12829–12838.  https://doi.org/10.1109/ICCV48922.2021.01261
  * Mayer et al. (2016) N. Mayer, E. Ilg, P. Häusser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. 2016.  A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation. In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 4040–4048.
  * Menze and Geiger (2015) Moritz Menze and Andreas Geiger. 2015.  Object scene flow for autonomous vehicles. In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 3061–3070.  https://doi.org/10.1109/CVPR.2015.7298925
  * Michael Bleyer and Rother (2011) Christoph Rhemann Michael Bleyer and Carsten Rother. 2011.  PatchMatch Stereo - Stereo Matching with Slanted Support Windows. In _Proceedings of the British Machine Vision Conference_. 14.1–14.11.  http://dx.doi.org/10.5244/C.25.14.
  * Park et al. (2020) Jinsun Park, Kyungdon Joo, Zhe Hu, Chi-Kuei Liu, and In So Kweon. 2020.  Non-local Spatial Propagation Network for Depth Completion. In _Computer Vision – ECCV 2020_ , Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 120–136.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library. In _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc., 8026–8037.  https://proceedings.neurips.cc/paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf
  * Qi et al. (2018) Xiaojuan Qi, Renjie Liao, Zhengzhe Liu, Raquel Urtasun, and Jiaya Jia. 2018.  GeoNet: Geometric Neural Network for Joint Depth and Surface Normal Estimation. In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 283–291.  https://doi.org/10.1109/CVPR.2018.00037
  * Rossi et al. (2020) Mattia Rossi, Mireille El Gheche, Andreas Kuhn, and Pascal Frossard. 2020.  Joint Graph-Based Depth Refinement and Normal Estimation. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 12151–12160.  https://doi.org/10.1109/CVPR42600.2020.01217
  * Scharstein et al. (2014) D. Scharstein, H. Hirschmuller, Y. Kitajima, G. Krathwohl, N. Nesic, and P. Westling X Wang. 2014.  High-resolution stereo datasets with subpixel-accurate ground truth.. In _German Conference on Pattern Recognition (GCPR)_.
  * Scharstein et al. (2001) D. Scharstein, R. Szeliski, and R. Zabih. 2001.  A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. In _Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001)_.
  * Shen et al. (2021) Zhelun Shen, Yuchao Dai, and Zhibo Rao. 2021.  CFNet: Cascade and Fused Cost Volume for Robust Stereo Matching. In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 13901–13910.  https://doi.org/10.1109/CVPR46437.2021.01369
  * Tankovich et al. (2021) Vladimir Tankovich, Christian Häne, Yinda Zhang, Adarsh Kowdle, Sean Fanello, and Sofien Bouaziz. 2021.  HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching. In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 14357–14367.  https://doi.org/10.1109/CVPR46437.2021.01413
  * Wang et al. (2021a) Fangjinhua Wang, Silvano Galliani, Christoph Vogel, Pablo Speciale, and Marc Pollefeys. 2021a.  PatchmatchNet: Learned Multi-View Patchmatch Stereo. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 14194–14203.
  * Wang et al. (2020) Qiang Wang, Shaohuai Shi, Shizhen Zheng, Kaiyong Zhao, and Xiaowen Chu. 2020.  FADNet: A Fast and Accurate Network for Disparity Estimation. In _2020 IEEE International Conference on Robotics and Automation (ICRA 2020)_. 101–107.
  * Wang et al. (2021b) Tianyuan Wang, Can Ma, Haoshan Su, and Weiping Wang. 2021b.  CSPN: Multi-Scale Cascade Spatial Pyramid Network for Object Detection. In _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. 1490–1494.  https://doi.org/10.1109/ICASSP39728.2021.9414883
  * Wang et al. (2015) Xiaolong Wang, David F. Fouhey, and Abhinav Gupta. 2015.  Designing deep networks for surface normal estimation. In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 539–547.  https://doi.org/10.1109/CVPR.2015.7298652
  * Xu et al. (2022) Gangwei Xu, Junda Cheng, Peng Guo, and Xin Yang. 2022.  ACVNet: Attention Concatenation Volume for Accurate and Efficient Stereo Matching.  _arXiv e-prints_ , Article arXiv:2203.02146 (March 2022), arXiv:2203.02146 pages.  arXiv:2203.02146 [cs.CV]
  * Xu and Zhang (2020) H. Xu and J. Zhang. 2020.  AANet: Adaptive Aggregation Network for Efficient Stereo Matching. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 1956–1965.
  * Xu et al. (2020) Zheyuan Xu, Hongche Yin, and Jian Yao. 2020.  Deformable Spatial Propagation Networks For Depth Completion. In _2020 IEEE International Conference on Image Processing (ICIP)_. 913–917.  https://doi.org/10.1109/ICIP40778.2020.9191138
  * Yang et al. (2018) Guorun Yang, Hengshuang Zhao, Jianping Shi, Zhidong Deng, and Jiaya Jia. 2018.  SegStereo: Exploiting Semantic Information for Disparity Estimation. In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Yin et al. (2019) Wei Yin, Yifan Liu, Chunhua Shen, and Youliang Yan. 2019.  Enforcing Geometric Constraints of Virtual Normal for Depth Prediction. In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_. 5683–5692.  https://doi.org/10.1109/ICCV.2019.00578
  * Yin et al. (2019) Z. Yin, T. Darrell, and F. Yu. 2019.  Hierarchical Discrete Distribution Decomposition for Match Density Estimation. In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 6037–6046.
  * Zhang et al. (2019) F. Zhang, V. Prisacariu, R. Yang, and P. H. S. Torr. 2019.  GA-Net: Guided Aggregation Net for End-To-End Stereo Matching. In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zhang et al. (2021) Songyan Zhang, Zhicheng Wang, Qiang Wang, Jinshuo Zhang, Gang Wei, and Xiaowen Chu. 2021.  EDNet: Efficient Disparity Estimation with Cost Volume Combination and Attention-based Spatial Residual. In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 5429–5438.  https://doi.org/10.1109/CVPR46437.2021.00539
  * Zhang et al. (2019) Zhenyu Zhang, Zhen Cui, Chunyan Xu, Yan Yan, Nicu Sebe, and Jian Yang. 2019.  Pattern-Affinitive Propagation Across Depth, Surface Normal and Semantic Segmentation. In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 4101–4110.  https://doi.org/10.1109/CVPR.2019.00423
  * Zhao et al. (2021) Wang Zhao, Shaohui Liu, Yi Wei, Hengkai Guo, and Yong-Jin Liu. 2021.  A Confidence-based Iterative Solver of Depths and Surface Normals for Deep Multi-view Stereo. In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_. 6148–6157.  https://doi.org/10.1109/ICCV48922.2021.00611

##  Appendix A Normal Estimation Sub-Net

Our Normal Estimation Sub-Net is illustrated in Figure 11. We directly use the
shared encoder part to provide the feature and adopt the network structure of
DispNetC[1] to estimate the surface normal in a coarse-to-fine manner. For
residual learning(Residual Blocks in blue ), we use a shallow U-Net
architecture to take features at the current scale together with upsampled
predicted surface normal at the previous scale and left-right image pairs as
input to predict the surface normal residual at the current scale. Besides, we
normalized the output of each scale directly on the feature channels(3),
making it more in line with the definition of the surface normal. Figure 9
illustrate visualization results on SceneFlow and KITTI 2015 and MiddleBurry
2014 datasets.

##  Appendix B Non-Local Disparity Propagation at different scales.

In this section, by visualizing the disparities estimation results at
different resolutions, we show how our proposed NDP module improves the
quality of the predicted disparity through spatial propagation, especially in
thin structures, edges, and occluded regions. As is shown in Figure 12, the
non-local disparity propagation based on surface normal and other semi-context
information performs well at thin structures and edges especially at low
scales to alleviate the disparity discontinuity and blurring issues. After
spatial propagation, our disparity preserves more structural details, which
facilitates more accurate depth results for downstream tasks and thus
constructs more accurate AR/VR applications.

##  Appendix C Spatial attention and affinities distribution in ARL module

As mentioned in the main paper, we have proposed an ARL module that manages to
optimize disparity estimation results at the feature level. There are two main
tricks in the ARL module for efficient residual learning. One is to apply the
dynamic spatial attention mechanism to generate attention maps at different
refinement scales. The attention maps imply where should be highlighted as
well as where should be ignored. It promotes the network to efficiently
optimize the disparity estimation results for different locations at different
scales.

As shown in Figure 10, the dynamic spatial attention mechanism tends to assist
residual learning at different levels and focus on regions with different
attributes. The attention distribution at the 1/4 scale mainly lies in the
background objects, while the distribution of attention at 1/2 resolution is
highly consistent with that of the occlusion mask. This enables the ARL module
to be occlusion-aware when refining the disparity. Besides, the attention map
at full scale shows bigger attention to foreground object and planes that is
texture-less, which benefit the refinement in such regions.

The other trick of the ARL module is the local affinity filtering module whose
intermediate affinity visualization result of the local 8 neighbors can be
seen in Figure 13. As the figure illustrated, the red arrow in reference
images shows the plane direction, as we can see that the affinity value at the
left-top value is bigger than the other direction, which indicates the
direction of the feature aggregation should be. By applying the local affinity
filtering module, our ARL module can make full use of surface normal
information for better feature aggregation.

Figure 9. Normal Estimation Result on SceneFlow,KITTI2015 and MiddleBurry
testing set. Figure 10. Attention Maps at different scales(Range from 1/4 full
scale). Different scales pay different attention to different regions, such as
foreground, texture-less planes, occluded areas and background. Figure 11.
Normal Estimation Sub-Net Architecture. The feature encoder is shared with the
disparity estimation branch. The whole Sub-Net adopts the DispNetC (Mayer et
al., 2016) architecture with a residual learning module. We normalized the
predicted disparity to satisfy the geometry constrain. Figure 12. Disparity
refinement at different scales. It clearly shows that our proposed non-local
propagation witness a clear edges and structures at different disparity
scales, which alleviates the blurring and breakage issues at the edges of the
image. Figure 13. A sample of affinity distribution of local 8 neighbors in
ARL module.