# USegScene: Unsupervised Learning of Depth, Optical Flow and Ego-Motion with
Semantic Guidance and Coupled Networks

Johan Vertens  Wolfram Burgard Johan Vertens is with the University of
Freiburg, Germany. Wolfram Burgard is with the University of Technology,
Nuremberg, Germany. Corresponding author<EMAIL_ADDRESS>
###### Abstract

In this paper we propose USegScene, a framework for semantically guided
unsupervised learning of depth, optical flow and ego-motion estimation for
stereo camera images using convolutional neural networks. Our framework
leverages semantic information for improved regularization of depth and
optical flow maps, multimodal fusion and occlusion filling considering dynamic
rigid object motions as independent SE(3) transformations. Furthermore,
complementary to pure photometric matching, we propose matching of semantic
features, pixel-wise classes and object instance borders between the
consecutive images. In contrast to previous methods, we propose a network
architecture that jointly predicts all outputs using shared encoders and
allows passing information across the task-domains, e.g the prediction of
optical flow can benefit from the prediction of the depth. Furthermore, we
explicitly learn the depth and optical flow occlusion maps inside the network,
which are leveraged in order to improve the predictions in the respective
regions. We present results on the popular KITTI dataset and show that our
approach outperforms previous methods by a large margin.

##  I Introduction

The estimation of optical flow, depth and ego-motion from stereo-images are
fundamental computer vision problems with an enormous amount of robotic
applications. In the field of autonomous driving it is used for solving
problems like mapping [1], object tracking [2, 3], or localization [4],
enabling the systems to reason about their surrounded environment.

In previous years researchers mainly approached the tasks with supervised
learning, leveraging datasets like KITTI [5] or synthetic datasets such as
Synthia [6] or FlyingChairs [7]. However, real world datasets with a high
amount of ground-truth data are still rare, since expensive laserscanners are
required and complicated procedures need to be carried out to gather such
data. Furthermore, networks trained for depth, optical flow, and visual
odometry on these datasets tend to overfit to the datasets specific camera-
setup and thus have limited flexibility.

Recent works propose to tackle the individual problems with unsupervised
methods using photometric consistency to trigger the learning [8, 9, 10, 11].
Due to the pure usage of photometric cues these approaches have problems with
largely untextured regions or significant lighting changes between the
consecutive frames.

---
|
|
|
Figure 1: Example output of our method. From top to bottom: Aggregated (20
frames) semantically segmented point-cloud, color image of left camera (left),
motion segmentation map (right), optical flow (left), predicted occlusion
between consecutive images (right), predicted disparity (left), predicted
occlusion between left-right image pair (right). The yellow arrow illustrates
the camera-pose related to all predictions.

Furthermore, most of the previous works struggle with occlusions between the
consecutive images and use simple edge-based smoothing techniques that do not
involve context information.

Other works [12, 13, 14, 15, 16] couple the tasks of estimating depth, optical
flow and ego-motion by exploiting problem-dependencies within the loss
function. This way the tasks are not longer learned independently but are
constrained by each other in the learning process. However, these proposed
approaches do not leverage context information, such as semantic segmentation
that we show can help the network to improve the predictions. Additionally,
these approaches tackle the individual tasks with independent and largely
decoupled networks ,making the complete system less efficient, and do not
handle occlusions that occur due to dynamic objects.

In this paper, we propose an unsupervised learning framework for semantic
aware estimation of depth, optical flow and ego-motion using stereo cameras.
We jointly infer all predictions using two consecutive stereo-pairs and their
corresponding semantic features in a coupled network with shared encoders. The
unified nature of the network enables leveraging complementary information
across the output-domains, e.g the ego-motion estimation can profit from the
estimation of depth and optical flow.

Complementary to direct photometric losses we propose semantic feature
matching that matches feature maps from a semantic segmentation network,
improving predictions on textureless regions. This pre-trained segmentation
network is trained to predict the class-wise semantic segmentation and
additionally regresses the borders of distinct rigid instances of objects in
the scene.

For flow and disparity regularization we propose a novel weighted semantic
patch smoothing loss (WSPS) which smooths the optical flow or disparity map
only within certain semantic classes, considering the output of the semantic
network.

To cope with occlusions we further propose a method that fills these regions
with information from estimated SE(3) transformations of the rigid ego-motion
and independent moving rigid objects. In contrast to previous works, our
network also predicts the occluded regions directly without the need of
multiple inference passes and predicts correction for these regions.

We evaluate our method on the popular real world dataset KITTI [17], achieving
state-of-the-art accuracies.

##  II Related Work

As our method comprises of the estimation of disparity, ego-motion and optical
flow we list related work for each field of research individually. Finally we
give an overview on previous works that focus on the combination of these
different tasks. We mainly focus on works that leverage convolutional neural
networks.

###  II-A Disparity Estimation

Traditionally, disparity estimation algorithms follow the paradigm of
computing costs, cost aggregation, global optimization and depth refinement
[18]. Published works propose to learn parts of this pipeline with neural
networks. LeCun _et al._[19] propose to learn the matching costs by training a
siamese-network to differentiate between small similar and dissimilar image
patches. This training is done in a supervised fashion where a binary
classification dataset yields the basis as training data. Similarly Shaked et.
al [20] learn the matching costs with a new highway network architecture that
leverages multilevel residual shortcut connections. Additionally, they propose
a global disparity network to estimate the disparity confidence. As more real
world [17, 21] and synthetic datasets [7, 22, 6] for depth estimation became
available, many approaches have been proposed to tackle the problem in an end-
to-end way. DispNet [23] learns to predict the disparity directly using a two-
stream network with explicit correlation layers.

In the work from Kendall _et al._[24] a new way to construct the cost volume,
where all pixel-wise costs according to a specific disparity value are
aggregated and then convolved with 3D convolutions.

Chang _et al._[25] present a network architecture that aggregates global
context information via a spatial pyramid pooling module for improving the
disparity accuracy in regions with difficult appearance or lighting.

Other works [10, 26] learn to predict depth in an unsupervised fashion. They
use photometric cues and forward-backward warping in order to train the
network without the need of any labeled data.

Additionally, some methods [27, 28] have been proposed that solve disparity
estimation and semantic segmentation within a single network. The authors show
that the learned semantic embedding helps to improve the accuracy.

###  II-B Optical Flow Estimation

In recent years CNNs became increasingly popular in the field of optical flow
estimation, superseding previous methods. The authors of FlowNet [29] propose
to learn pixel-wise optical flow using a convolutional neural network and
synthetic data. Further works present cascaded [30] and pyramid network [31]
architectures that make the predictions more accurate and the system more
efficient.

Hui _et al._[32] propose a fast network and uses flow regularization for
further speeding up inference time while maintaining accurate and sharp
predictions.

In order to get rid of the dependence of ground-truth data, Yu _et al._[8]
propose to learn optical flow in an unsupervised fashion using photometric
losses and smoothing terms. The authors of UnFlow [9] further propose a loss
that is robust to illumination changes across the consecutive images.

Recent advances [33, 34, 35] in unsupervised optical flow estimation cope with
the problem of occlusion. They mask out faulty photometric losses [33], or
handle occlusions by training a student network from a teacher network that
has more information about occluded regions [34].

###  II-C Odometry Estimation

Visual odometry is a well studied field of computer vision where many
efficient and robust methods exist [1, 36, 37]. However, researchers [38, 39]
have been investigating learning visual odometry with neural networks in order
to overcome problems with texture-less environments or changing lighting
conditions. Additionally these methods do not rely on apriori handcrafted
feature descriptors but learn them themselves.

In recent years many works [40, 11, 41] learn visual odometry in an
unsupervised way. Beside an odometry network, a depth estimator is trained
that predicts the scene geometry. The predicted geometry and the ego-motion
estimate can then be used to synthesize adjacent images and a photometric loss
is applied to train the networks jointly.

###  II-D Combined Approaches

Leveraging all advances in the individual fields of disparity, optical flow
and ego-motion estimation, many works emerged that combine all tasks in a
single unsupervised framework [13, 16, 15, 42, 14]. In GeoNet [13] firstly
monocular depth and the ego-motion is learned. A rigid flow field is then
computed by shifting the 3D geometry by the predicted ego-motion and
projecting the motion vectors on the image plane. Since this flow prediction
is only valid for static parts of the scene, they also propose a residual
optical flow module that corrects for the dynamic regions. In the work of Zou
_et al._[16] the rigid optical flow constraints the learning of a pixel-wise
optical flow module. The constraint is not applied on dynamic regions, which
they detect with forward-backward flow consistency checking. The authors of
EPC++ [15] additionally add consistency constraints between consecutive depth
maps and segment dynamic regions using scene flow vectors. Wang _et al._[14]
present a unified system for learning optical flow, visual odometry and stereo
depth estimation. Instead of directly using the ego-motion estimate from a
network they refine the ego-motion using the flow and depth cues.

All combined approaches exploit the relations between the individual tasks and
show that joint training of all depth, optical flow and visual odometry yields
overall significantly better results in each domain. However, previous works
do not couple the tasks for dynamic regions while exploiting semantic cues.
Additionally, none of the previous works couple all problems in the network
structure directly but rather use stand-alone networks for each individual
task independently. Furthermore, all previous methods do not handle regions
that are occluded and dynamic. We, instead propose a system which improves the
prediction quality in these regions using rigid object-instance level SE(3)
transforms.

##  III Technical Approach

In the following we describe our approach for unsupervised learning of depth,
optical flow and ego-motion. We first present our unified network
architectures that solves all tasks within a single network which gets
additional semantic features as input. We then present semantic feature
matching which enforces semantic consistency across the consecutive frames.
Furthermore, we present a loss for semantic aware regularization which guides
the network to predict smooth output maps within equal classes, while being
sharp at class boundaries. Using a similar technique as for the semantic aware
regularization we propose to constrain occluded depth regions by neighboring
non-occluded regions with the same semantic class. We also present a method to
distinguish between static and dynamic objects and propose novel 3D
consistency losses which couples optical flow, disparity and ego-motion
directly in the 3D space. Finally we propose a loss that enforces the network
to fill occluded regions in optical flow maps using ego-motion and independent
motions of rigid dynamic objects.

###  III-A Unified 3D Geometry and Motion Network

Figure 2: Overview of our USegScene architecture. Using two consecutive stereo
image-pairs we predict optical flow $F$, two consecutive disparity maps $D$,
ego-motion $T_{\mathrm{ego}}$ and occlusions $O$ in respective estimators at
multiple levels $k^{1},k^{2},k^{3},k^{4},k^{5}$ with different resolutions.
Each estimator additionally leverages complementary information from a
semantic segmentation network, which predicts pixel-wise class labels $S$ and
a regression of instance-borders $B$.

While previous methods predict optical flow, disparity and ego-motion with
independent networks we present a neural network architecture that predict
these outputs jointly. In general, our architecture can be divided into seven
different modules namely a semantic segmentation network, a shared feature
encoder, a disparity estimator, an optical flow estimator, an occlusion
refinement module, an ego-motion estimator and a context-module.

Our network takes advantage from the fact that the predictions of optical
flow, disparity and ego-motion, leverage similar feature representations that
are well suited for dense matching. Thus, our structure does not predict this
redundant information for optical flow, disparity and ego-motion individually
but computes them once and feeds it to the modules in a shared manner. In
contrast to other works, the output of the disparity module gets passed to the
optical flow and the ego-motion module, while the ego-motion module also
benefits from the output of the optical flow module.

To cope with texture-less regions and occlusions, we additionally feed
semantic cues from a semantic segmentation network into each of the sub-
modules, making our modules context aware. The high-level semantic information
is then used in the matching process, making the predictions more robust.

Our network is structured as a multi-scale pyramid neural network similar to
PWCNet [31], following a course-to-fine scheme. In total we use 5 different
levels $k_{1}...k_{5}\in K$ where in each level a optical flow map, a
disparity map and the ego-motion gets estimated. After the last level $k_{5}$
we further pass the disparity and optical flow predictions through a context-
network.

####  III-A1 Feature Encoder

We adapt the feature encoder from PWCNet [31] and add a SPP module [25] to it,
which showed significant improvements in disparity estimation. Our feature
encoder extracts features $f_{l_{t}}$, $f_{l_{t+1}}$, $f_{r_{t}}$ and
$f_{r_{t+1}}$ corresponding to the consecutive input stereo image pairs
$(I_{l_{t}}$, $I_{r_{t}})$, $(I_{l_{t+1}}$, $I_{r_{t+1}})$. The feature maps
are predicted for the different scales $\frac{1}{2}$, $\frac{1}{4}$,
$\frac{1}{8}$, $\frac{1}{16}$ and $\frac{1}{32}$ and then fed into the 5
levels of the depth, optical flow and ego-motion estimators.

####  III-A2 Semantic Segmentation Network

We use a ResNext50 [43] architecture with ASPP [44] and skip connections in
order to predict pixel-wise semantic segmentation maps. We choose the classes
in a way that they represent classes with distinct geometric complexity and
rigidity characteristics. Our classes are: Road, Sidewalk, Building,
Construction, Fence, Poles, Vegetation, Various Terrain, Sky,
Pedestrian/Rider, Vehicle and Background. Beside the segmentation map $S$ our
semantic network also predicts a regression of borders of rigid object
instances $B=e^{-\frac{d}{\sigma}}$ where $d$ is the distance to the instance
borders in pixels and $\sigma$ a shaping constant for the exponential, which
we set to $15$ during all experiments. Fig. 3 shows an example output of our
segmentation network. We feed both, the segmentation maps $S_{l_{t}}$,
$S_{l_{t+1}}$ and the border regression maps $B_{l_{t}}$, $B_{l_{t+1}}$,
corresponding to the consecutive left images $I_{l_{t}}$, $I_{l_{t+1}}$, into
every level of our disparity, optical flow and ego-motion modules. In order to
be compatible to the corresponding scales for each level we simply re-size the
segmentation and border maps bi-linearly to match with the input resolution of
the respective modules.

---
Figure 3: Output prediction examples of our semantic segmentation network.
From top to bottom: Color image of left camera, pixel-wise semantic
segmentation, regression of instance-wise object borders, features from the
second block of the ResNext architecture of the segmentation network. For
better visualization we sum the features along the channel-dimension and apply
a jet-colorcoding.

####  III-A3 Depth Estimator

Figure 4: This figure illustrates the module GM-Net, which comprises of the
depth estimator, optical flow estimator and ego-motion estimator. The GM-Net
is applied to every level within the USegScene architecture predicting the
disparities $D_{l_{t}}^{k}$, $D_{l_{t+1}}^{k}$, optical flow $F_{l_{t}}^{k}$
and ego-motion $T_{\mathrm{ego}}^{k}$ for the current level, given the
previous predictions $D_{l_{t}}^{k-1}$, $D_{l_{t+1}}^{k-1}$,
$F_{l_{t}}^{k-1}$, $T_{\mathrm{ego}}^{k-1}$ and the features from the shared
encoder $f_{l_{t}}^{k}$, $f_{l_{t+1}}^{k}$, $f_{r_{t}}^{k}$,
$f_{r_{t+1}}^{k}$. In the GM-Net all sub-modules leverage the predicted
consecutive semantic class and instance-border information $S$, $B$.
Additionally we apply our OCC-Net to each disparity and optical flow
estimator, which estimates the respective occlusion maps
$O_{D_{l_{t}}}$,$O_{D_{l_{t+1}}}$,$O_{F_{l_{t}}}$ and applies corrections in
those regions. In our architecture the optical flow estimator can benefit from
the consecutive disparity maps and the rigid flow. Complementary to the
features from the shared encoder the ego-motion estimator also takes the
consecutive disparity maps $D_{l_{t}}^{k}$, $D_{l_{t+1}}^{k}$ and the
predicted optical flow $F_{l_{t}}$ as input.

In order to predict the 3D geometry as two consecutive disparity maps
$D_{l_{t}}^{k}$ and $D_{l_{t+1}}^{k}$, we adapt the estimator of PWCNet [31]
with dense blocks that was originally designed to predict optical flow. In
contrast to previous works such as [25], which builds a memory expensive cost
volume and use 3D convolutions, this estimator only uses 2D convolutions and
fast correlation modules, resulting in a faster network with smaller memory
footprint. In addition to the features of the left and right image
$f_{l_{t}}^{k}$ and $f_{r_{t}}^{k}$ from the shared feature encoder of the
respective level and the previous disparity estimate $D_{l_{t}}^{k-1}$ from
the previous level, we concatenate the re-sized segmentation maps
$S_{l_{t}}^{k}$, $S_{r_{t}}^{k}$ and border maps $B_{l_{t}}^{k}$,
$B_{r_{t}}^{k}$ to the features after the correlation layer, thus making our
disparity estimator semantic aware.

####  III-A4 Optical Flow Estimator

Similar to the depth estimator our optical flow estimator follows the
estimator architecture of PWCNet [31], but further leverages complementary
geometric and semantic cues from the depth estimators, the semantic
segmentation network and the ego-motion estimators. More precisely, our
optical flow estimator at level $k\in K$ takes the feature maps
$f_{l_{t}}^{k}$ and $f_{l_{t+1}}^{k}$ from the shared feature encoder, the
disparity maps $D_{l_{t}}^{k}$, $D_{l_{t+1}}^{k}$, the semantic and border
maps $S_{l_{t}}^{k}$, $S_{l_{t+1}}^{k}$, $B_{l_{t}}^{k}$, $B_{l_{t+1}}^{k}$
and the optical flow estimate from the previous level $F_{l_{t}}^{k-1}$ in
order to predict the next optical flow estimate $F_{l_{t}}^{k}$. Furthermore,
using the disparity $D_{l_{t}}^{k}$ of the current level and the ego-motion
estimate $T_{\mathrm{ego}_{t\rightarrow t+1}}^{k-1}$ of the previous level,
the rigid flow ${F_{l_{t}}^{R}}^{k}$, which refers to an optical flow map that
is valid only for the static regions, is calculated and additionally
concatenated to the inputs of the optical flow estimator. For more information
on how to render the rigid flow please refer to chapter III-B. Thus, our
multi-modal optical flow branch is able to learn to match consecutive features
of the consecutive frames but also 3D geometry and semantic information. Since
we pass the rigid flow to the estimator as prior information about the optical
flow in static regions, the full prediction of the optical flow is eased,
which we show in our experiments in section IV-D.

####  III-A5 Occlusion Refinement Network

Figure 5: This figure illustrates the architecture of our OCC-Net module.
Given an initial disparity or optical flow map and high-level features from
the respective estimator, we first estimate the occluded regions. Following,
we predict corrections that are gated by the occlusion and added to the
initial disparity/optical flow map.

Since pixel-wise disparity and optical flow estimation is dependent on
consistently observable image features, handling the occluded pixels in the
consecutive stereo image pairs plays an important role. We therefore propose a
novel module in our network that is capable to learn to detect and correct
occluded regions in the disparity and optical flow maps. First, features
$f_{F}^{k}$ or $f_{D}^{k}$ and the corresponding initial optical flow
$\bar{D}^{k}$ or disparity map $\bar{F}^{k}$ are passed into a 4-layer
convolutional occlusion detection module that predicts the likelihood of a
pixel being occluded or not. Here, $f_{F}^{k}$ and $f_{D}^{k}$ are the
features taken from before the last layer of the respective optical flow or
disparity estimator modules at level $k$. We use a sigmoid function at the end
of the occlusion detection module in order to have a output range between 0
and 1, where 0 means that a pixel is occluded and 1 means that the pixel is
not occluded. Second, an occlusion correction network takes $f_{F}^{k}$ or
$f_{D}^{k}$, the estimated occlusion map $O_{F}^{k}$ or $O_{D}^{k}$, and the
optical flow/disparity map $\bar{D}^{k}$ or $\bar{F}^{k}$ in order to predict
corrections $R_{O}$ for the occluded parts of the input maps. Finally, the
corrections are gated by the estimated occlusion map and added to the input
optical flow/disparity map resulting in the final maps $F^{k}$ or $D^{k}$.
Fig. 5 shows the schematic of the module, which we refer to as OCC-Net. We
apply OCC-Net after each disparity and optical flow estimator.

####  III-A6 Ego-Motion Estimator

Figure 6: This figure illustrates our ego-motion estimator. First, we compile
the rigid flow using the disparity and the previous ego-motion estimate. We
then use the predicted rigid flow to warp features related to the next left
image to the view of the previous left image. Following, the warped features
are correlated with the features of the previous left image. The correlation
result is fed together with the disparities, disparity change, optical flow,
rigid optical flow, semantic information and learned high-level features to
the SE(3) estimator. The final ego-motion is obtained by applying global
average pooling and adding the ego-motion of the previous level.

In previous works [12, 13, 11] for unsupervised learning of ego-motion the
problem is approached with dense photometric matching while the respective
networks do not use 3D information in their inputs. We propose a CNN
architecture that additionally leverages 3D geometry given by the consecutive
disparity maps $D_{l_{t}}^{k}$ and $D_{l_{t+1}}^{k}$. In each level $k$ we use
$D_{l_{t}}^{k}$ and the ego-motion of the previous level
$T_{\mathrm{ego}_{t\rightarrow t+1}}^{k-1}$ to warp $f_{l_{t+1}}^{k}$ to the
reference of the first frame $I_{l_{t}}$. We then correlate the warped
features $\hat{f_{l_{t}}^{k}}$ with $f_{l_{t}}^{k}$ and feed the correlation
result together with the semantic maps $S_{l_{t}}^{k}$, $S_{l_{t+1}}^{k}$,
border maps $B_{l_{t}}^{k}$, $B_{l_{t+1}}^{k}$, the optical flow
$F_{l_{t}}^{k}$ and the consecutive disparity maps
$D_{l_{t}}^{k},D_{l_{t+1}}^{k}$ in a convolutional network block. Considering
a ego-motion estimator at level $k$ our estimator consists of $5+(k-1)$
layers, where each additional layer has a stride of two, so that its output of
every level has the same dimension as the lowest level $k_{1}$.Similar to
previous works [12] we apply global average pooling at the output of each ego-
motion estimator that predicts the rigid body transformation
$T_{\mathrm{ego}_{t\rightarrow t+1}}=\begin{pmatrix}R&t\\\ 0&1\end{pmatrix}$
where $R\in\mathit{SO}(3)$ and $t\in\mathbb{R}^{3}$. We predict our
transformation in lie-algebraic exponential coordinates
$\xi=(v^{T}\>\omega^{T})\in\mathfrak{se}(3)$ and use the exponential map with
small-angle approximations [45] to map from $\mathfrak{se}(3)$ to
$\mathit{SE}(3)$. In every level we add the previous lie-algebraic exponential
coordinates from the previous level to the new result. Thus, higher levels
only predict residuals that further refine the ego-motion prediction. We
illustrate the ego-motion estimator architecture on Fig. 6.

The complete network architecture is illustrated in Fig. 2, where GM-Net
denotes the network branches for optical flow, disparity and odometry in each
level. For brevity, we do not present all architecture parameters, but depict
the general scheme of our network. Following PWCNet [31], as there are no
previous predictions for optical flow, disparity or ego-motion in the lowest
level $k_{1}$ we directly feed the feature representations into the network
without having a correlation before.

###  III-B Unsupervised Training

For training our network architecture we formulate the optical flow, disparity
and ego-motion estimation as a photometric optimization problem similar to
previous works [14, 15, 13]. Given the optical flow $F_{\mathscr{T}}$ or
disparity $D_{\mathscr{T}}$ corresponding to a target image $I_{\mathscr{T}}$,
we can warp a source Image $I_{\mathscr{S}}$ to $I_{\mathscr{T}}$ by using
spatial transformer networks [46].

For synthesizing $I_{\mathscr{T}}$ from $I_{\mathscr{S}}$ given the ego-motion
$T_{ego_{\mathscr{T}\rightarrow\mathscr{S}}}$, we use the depth
$Z_{\mathscr{T}}=\frac{fb}{D_{\mathscr{T}}}$, where $f$ is the camera focal-
length and $b$ is the baseline of the stereo rig, and back-project the pixel
locations of $I_{\mathscr{T}}$ to 3D. In order to get the pixel
correspondences in $I_{\mathscr{S}}$ we then transform the points by
$T_{ego_{\mathscr{T}\rightarrow\mathscr{S}}}$ and project them back on the
image plane using the camera intrinsic matrix $K$ , resulting in the
corresponding pixel coordinates of $I_{\mathscr{T}}$ in $I_{\mathscr{S}}$.

The pixel correspondences for the synthesizing operations can be found with:

$\displaystyle
p_{\mathscr{S}}^{F_{\mathscr{T}}}=p_{\mathscr{T}}+F_{\mathscr{T}}(p_{t})$ (1)
$\displaystyle
p_{\mathscr{S}}^{D_{\mathscr{T}}}=p_{\mathscr{T}}-D_{\mathscr{T}}(p_{\mathscr{T}})$
$\displaystyle
p_{\mathscr{S}}^{F_{\mathscr{T}}^{R}}=\phi(K\>T_{ego_{\mathscr{T}\rightarrow\mathscr{S}}}\gamma(p_{\mathscr{T}}\mid
K,D_{\mathscr{T}}))$

where $p_{\mathscr{S}}^{F_{\mathscr{T}}}$ denote the pixel-locations in
$I_{\mathscr{S}}$ corresponding to $p_{\mathscr{T}}$ mapped by the optical
flow $F_{\mathscr{T}}$, $p_{\mathscr{S}}^{D_{\mathscr{T}}}$ denote the pixel-
locations in $I_{\mathscr{S}}$ corresponding to $p_{\mathscr{T}}$ mapped by
the disparity $D_{\mathscr{T}}$, and $p_{\mathscr{S}}^{F_{\mathscr{T}}^{R}}$
denote the pixel-locations in $I_{\mathscr{S}}$ corresponding to
$p_{\mathscr{T}}$ mapped by the disparity $D$ and the ego-motion
$T_{ego_{\mathscr{T}\rightarrow\mathscr{S}}}$. Here,
$\gamma(p_{\mathscr{T}}\mid
K,D_{\mathscr{T}})=D_{\mathscr{T}}(p_{\mathscr{T}})K^{-1}h(p_{\mathscr{T}})$
means the back-projection of pixel coordinate $p_{\mathscr{T}}$ to the 3D
space, where $h(p_{\mathscr{T}})$ converts $p_{\mathscr{T}}$ into the
homogeneous vector form. The function $\phi(x)$ divides a vector x by it’s
last element.

Leveraging the calculated target pixel-locations we denote a synthesized
target image as $\hat{I_{\mathscr{T}}}=W(I_{\mathscr{S}},M_{\mathscr{T}})$,
where $W$ is the warping function that synthesizes the target bi-linearly
using a displacement map $M$, such as the optical flow or disparity map.

We further denote the pixel-displacement vectors between
$p_{\mathscr{S}}^{F_{\mathscr{T}}^{R}}$ and $p_{\mathscr{T}}$ as the rigid
optical flow map
$F_{\mathscr{T}}^{R}=p_{\mathscr{S}}^{F_{\mathscr{T}}^{R}}-p_{\mathscr{T}}$,
which is valid in completely static scenes.

Given the pixel correspondences, we define a loss function that calculates
direct photometric differences and the structure similarity [47] between the
synthesized target image $\hat{I_{\mathscr{T}}}$ and the observed target image
$I_{\mathscr{T}}$:

$\displaystyle\mathcal{L}_{p}^{M_{\mathscr{T}}}=O_{M_{\mathscr{T}}}[\alpha_{p}\mid
I_{\mathscr{T}}-W(I_{\mathscr{S}},M_{\mathscr{T}})\mid+\beta_{p}\mathrm{SSIM}(I_{\mathscr{T}},W(I_{\mathscr{S}},M_{\mathscr{T}}))]$
(2)

Here, $O_{M_{\mathscr{T}}}$ denotes an occlusion mask that indicates which
pixels are matchable between the corresponding frames, the weights
$\alpha_{p}$ and $\beta_{p}$ are empirically set to $0.6$ and $0.4$
respectively.

####  III-B1 Estimating Occlusions

We follow [13] to estimate occluded regions $O_{M}$, corresponding to any flow
or disparity map $M$, using forward-backward consistency checks on the
disparity and optical flow maps. Pixels where forward and backward optical
flow/disparity contradict are assumed to be occluded.

Specifically, while training, the optical flow and disparity modules infer the
output in both directions, meaning a flow or disparity map gets calculated
that maps from a source image $I_{\mathscr{S}}$ to a target image
$I_{\mathscr{T}}$ and also from $I_{\mathscr{T}}$ to $I_{\mathscr{S}}$.

We define the binary occlusion map as:

$\displaystyle O_{M_{\mathrm{fb}}}=(\mid\mid\Delta
M\mid\mid_{2}<\max\\{\alpha,\beta\mid\mid\Delta M\mid\mid_{2}\\})$ (3)

where $\Delta M$ is the difference between the aligned forward and backward
inference of $M$: $\Delta
M=M_{\mathscr{T}}-W(M_{\mathscr{S}},M_{\mathscr{T}})$.

We use the calculated binary occlusion maps in turn for self-supervised
training of the occlusion detectors within the OCC-Nets. Since the forward-
backward consistency checks are only needed for training, at testing time our
network is able to predict occlusions without the need of running the optical
flow or disparity estimators in forward and backward mode, saving computation
time. While training the occlusion detectors the loss function is simply:

$\displaystyle\mathcal{L}_{\mathrm{occ}}^{M}=\mid\mid
O_{M}-O_{M_{\mathrm{fb}}}\mid\mid_{2}$ (4)

Here, $O_{M}$ is the predicted occlusion map generated by the respective OCC-
Net. We set $\alpha=0.5$ and $\beta=2e^{-3}$ for inferring optical flow
occlusions and $\alpha=1.0$ and $\beta=4e^{-2}$ for inferring disparity
occlusions respectively.

####  III-B2 Semantic Feature Matching

Since pure photometric losses do not deliver meaningful information in ill
posed conditions, such as low-lighting, textureless regions, transparent areas
or reflections, we propose a loss function that matches complementary semantic
feature representations from our semantic segmentation network. Specifically,
we match source softmax segmentation values $f_{s}$, the border regression map
$B$, as well as features $f_{\mathrm{seg}}$ from the encoder of the
segmentation network, to the target view $\mathscr{T}$. In our case, $f_{s}$
correspond to features that are inferred from consecutive images
$I_{\mathscr{S}}$ and $I_{\mathscr{T}}$ and are matched by $M_{\mathscr{T}}$.
As semantic features $f_{\mathrm{seg}}$ we take the feature maps after the
second block of the ResNext-architecture of our semantic segmentation network
and apply a sigmoid activation function to squeeze the values to the range
between $0$ and $1$.

We formulate our semantic feature matching loss as:

$\mathcal{L}_{sm}^{M_{\mathscr{T}}}=\sum_{f\in{(f_{s},B,f_{\mathrm{seg}})}}O_{M_{\mathscr{T}}}(\mid
f_{\mathscr{T}}-W(f_{\mathscr{S}},M_{\mathscr{T}})\mid)$ (5)

An example of each semantic modality $S$, $B$, $f_{\mathrm{seg}}$ is shown in
Fig. 3.

####  III-B3 Weighted Semantic Patch Smoothing (WSPS)

In image regions where neither photometric nor semantic cues exist, we guide
the optical flow and disparity estimation by information from neighboring
pixels. Many works [10, 15, 48] use smoothing techniques, which minimize the
local second derivative while down-weighting the smoothness effect at
positions with high gradient in the corresponding color image. Instead, we
propose a smoothing method that considers larger neighborhoods and additional
semantic cues.

For every pixel location $p_{m}\in P_{M}$ of a displacement map $M$ we define
$\mathcal{N}(p_{m})$ as the set of pixel locations within a quadratic
neighborhood around the location $p_{m}$ where the semantic class at location
$p_{m}$ is equal to $p_{n}\in\mathcal{N}(p_{m})$, e.g $S(p_{n})=S(p_{m})$.
Within this neighborhood, we minimize the difference of the gradient of the
flow and disparity map at the center pixel $\nabla M(p_{m})$ with all other
gradients of other pixels $\nabla M(p_{n})$ of the same class.

We formulate our weighted semantic patch smoothing WSPS loss as follows:

$\displaystyle\mathcal{L}_{\mathrm{WSPS}}^{M}=\sum_{p_{m}\in
P_{M}}\frac{1}{N}\alpha_{c}\sum_{p_{n}\in\mathcal{N}(p_{m})}\omega(p_{n},p_{m},Z(p_{m}))\cdot$
(6) $\displaystyle\rho(\mid\nabla M(p_{n})-\nabla M(p_{m})\mid)$

where:

$\omega(u_{n},u_{m},z)=e^{-\frac{\mid\mid
p_{m}-p_{n}\mid\mid_{2}}{\eta\theta(z)}}$ (7)

$\theta(z)=\frac{1}{1+\frac{z^{2}}{bf}}$ (8)

The function $\omega$ weights the differences inversely proportional to the
euclidean distance between the locations $p_{n}$ and $p_{m}$ and the depth of
the center pixel. Thus, differences of gradients that are far away from each
other are less penalized. Further, if the center pixel has far depth $z$ this
effect gets strengthened since the covered real area of the neighborhood
becomes bigger with high distances to the camera, due to the projection-
characteristics. The regularization is weighted class-dependently by
$\alpha_{c}$, which we empirically set to $0.1$ for classes with high
structural variance: Construction, Fence, Poles, Vegetation, Various Terrain,
Pedestrian/Rider, $1.0$ for classes : Building, Various Terrain, Sky and $8.0$
for occluded pixels or classes with low structural variance or less textural
cues: Vehicle, Road, Sidewalk. The parameter $\eta$ defines how much pixels
$p_{n}$ that are far away from $p_{m}$ are influenced by the loss. We set
$\eta$ empirically to $10$ for all experiments.

To avoid oversmoothing between distinct object instances, we enhance our
semantic segmentation map with instance information of the types Vehicle and
Pedestrian/Rider using Mask-RCNN [49]. Note that Mask-RCNN is however not
needed at testing time. Consequently, we consider every object instance as a
individual class focusing the smoothing inside the instance. Additionally, by
using a robust loss function $\rho$ [50] (with parameters: $\alpha=0.5,c=0.1$)
we allow outliers within a class patch up to a certain level, accounting for
intra-class breaks.

In comparison to standard local smoothing [10], our loss improves predictions
in regions with challenging conditions, such as difficult lighting or
overexposure. Pixels that lie in ill-posed regions are not just coupled to
their direct neighbors, but over a large patch, regularizing over longer
distances. Instead of just down-weighting the loss at pixels with high color
gradient, our loss explicitly avoids oversmoothing between different semantic
objects. Furthermore, our loss offers the possibility to adjust the smoothing
strength over the different classes, e.g allowing less smoothing on classes
with high structural variance, such as vegetation.

We illustrate WSPS on Fig. 7.

Figure 7:  Weighted Semantic Patch Smoothing (WSPS) regularizes either
disparity or optical flow maps. We enforce consistency between a
disparity/flow gradient $\nabla p_{m}$ with a fixed position and all other
gradients $\nabla p_{n}$ that lie within a certain neighborhood and have equal
semantic class/instance (visualized with distinct colors). Each gradient
difference is weighted by a function, which is dependent of pixel distance and
depth. Red means high weight, while green means low weight.

####  III-B4 Semantic Depth Occlusion Filling

Figure 8: We guide the learning of occluded disparity regions (dotted regions)
by values of non-occluded regions of the same semantic class (distinct
colors). A disparity/flow vector $p_{m}$ with a fixed position gets
constrained by all other vectors $p_{n}$ within a certain neighborhood.
Constrains between $p_{m}$ and $p_{n}$ that have a larger y-distance are
weighted less (red means high weight, green means low weight), since
corresponding disparity values should lie on the x-axis due to image
rectification.

In previous works [28], occluded disparity values get post-processed by
assigning the nearest not occluded value to the occluded ones. This method,
however does not consider the context and thus fails if the nearest valid
values are not belonging to the same object that is further or nearer.

Instead, we guide the disparity predictions inside occluded areas using the
semantic information at training time, similar to WSPS. Thus, we do not rely
on any post-processing at testing time. We formulate our loss for guiding
occluded depth pixels as:

$\mathcal{L}_{D_{\mathrm{occ}}}^{D}=\\\ \sum_{p_{m}\in
P_{O_{D}}}\alpha_{c}\frac{\sum_{p_{n}\in\mathcal{N}(p_{m})}\omega(p_{n},p_{m},Z(p_{m}))\rho(\mid
D(p_{n})-D(p_{m})\mid)}{\sum_{p_{n}\in\mathcal{N}(p_{m})}\omega(p_{n},p_{m},Z(p_{m}))}\\\
$

where:

$\omega(p_{n},p_{m},z)=e^{-\frac{\mid
p_{m_{y}}-p_{n_{y}}\mid}{\eta\theta(z)}}$ (9)

For every pixel location $p_{m}\in P_{O_{D}}$ that is occluded by $O_{D}$ we
minimize the absolute difference between $D(p_{m})$ and the value $D(p_{n})$,
where $p_{n}$ is a pixel location that is in $\mathcal{N}(p_{m})$, which is
the set of pixel locations within a square neighborhood around $p_{m}$ that
have the same semantic class as $p_{n}$ and that are not occluded. Since all
stereo pairs are rectified, the stereo matches are located along the
horizontal axes. We therefore down-weight the differences between $p_{m}$ and
$p_{n}$ by the distance of their y-coordinates $p_{m_{y}}$ and $p_{n_{y}}$.
Further, we down-weight the influence of the loss proportional to the depth
$z$ at $p_{m}$ and a constant factor $\eta$ similar to WSPS. As in WSPS, we
leverage the robust loss function $\rho$ [50], where we set $\alpha$ to $0.5$
and $c$ to $0.1$. We illustrate the method in Fig. 8

####  III-B5 Sky Regularization

Since sky has infinite depth and weak visual features, previous methods often
predict unreasonable optical flow and disparity values in these regions or
foreground-predictions ’bleed’ into the sky regions. To overcome this problem,
we propose to minimize the optical flow or disparity in regions that belong to
the class sky using the segmentation map from our semantic network.
Specifically we formulate our loss as:

$\mathcal{L}_{\mathrm{sky}}^{M}=\frac{1}{N}\sum_{p_{m}\mid
S_{M}(p_{m})=\mathrm{sky}}\mid\mid M(p_{m})\mid\mid_{2})$ (10)

where $N$ is the number of pixels belonging to the class sky.

####  III-B6 3D Coupling

While other works [14, 16, 15] propose a minimization of the difference
between the rigid flow and the optical flow within rigid parts of the image,
we instead constrain the network directly in 3D space.

First, a source and a target depth map $Z_{\mathscr{S}}$ and $Z_{\mathscr{T}}$
are back-projected to 3D, resulting in the point-clouds $P_{\mathscr{S}}$ and
$P_{\mathscr{T}}$. Second, we synthesize $\hat{P_{\mathscr{T}}}$ using the
predicted optical flow $F_{{\mathscr{T}}}$ and $P_{\mathscr{S}}$. Third, we
transform the synthesized pointcloud $\hat{P_{\mathscr{T}}}$ by the ego-motion
$T_{ego_{{\mathscr{S}}\rightarrow{\mathscr{T}}}}$ in order to align it with
the target pointcloud $P_{\mathscr{T}}$. We then formulate a loss function
that minimizes the euclidean distance between the two pointclouds for the
static parts of the scene, which are not occluded by $O_{F_{\mathscr{T}}}$.

We define our loss as:

$\displaystyle\mathcal{L}_{\mathrm{3D_{\mathrm{rigid}}}}^{{\mathscr{S}}\rightarrow{\mathscr{T}}}=\theta(Z_{\mathscr{T}})O_{F_{\mathscr{T}}}(1-C_{\mathscr{T}})(\mid\mid
P_{\mathscr{T}}-T_{ego_{{\mathscr{S}}\rightarrow{\mathscr{T}}}}\hat{P_{\mathscr{T}}}\mid\mid_{2})$
(11)

Here, $C_{\mathscr{T}}$ denotes a map which is 1 at locations that are dynamic
and 0 otherwise. For far points the depth tends to be inaccurate due to the
limited resolution of the cameras. Therefore, with
$\theta(Z)=\frac{z^{2}}{bf}$ we weight the euclidean loss inversely
proportional to an approximated quadratic depth error as in Eq. 8, making our
loss function numerically stable.

####  III-B7 Dynamic Detection and Instance 3D Loss

We segment dynamic objects using the predicted disparities
$D_{l_{t}},D_{l_{t+1}}$, optical flow $F_{l_{t}}$ and semantic instances
$\mathscr{I}_{l_{t}}$, $\mathscr{I}_{l_{t+1}}$ corresponding to the left
consecutive images. As in previous sections, we define a source or target map
as either being related to time-step $t$ or the consecutive $t+1$. As for the
WSPS term, we infer the instances with Mask-RCNN [49]. For every instance $i$
of the vehicle class we calculate the SE(3) transformation
$T_{{\mathscr{S}}\rightarrow{\mathscr{T}}}^{i}$, which represents the rigid
motion of the corresponding dynamic object. Therefore, we compose the total
motion that is needed to transform the aligned instance pointcloud
$\hat{P_{{\mathscr{T}}}^{i}}=W(P_{{\mathscr{S}}}^{i},F_{{\mathscr{T}}})$ to
the corresponding target instance pointcloud $P_{{\mathscr{T}}}^{i}$, by
combining the predicted ego-motion
$T_{\mathrm{ego}_{\mathscr{S}}\rightarrow{\mathscr{T}}}$ with the dynamic
motion of the individual object
$T_{{\mathscr{S}}\rightarrow{\mathscr{T}}}^{i}$.

We define the instance-object motions as a optimization problem as follows:

$\displaystyle
T_{{\mathscr{S}}\rightarrow{\mathscr{T}}}^{i}=\underset{T}{\mathrm{argmin}}[Q\mid\mid
TT_{\mathrm{ego}_{{\mathscr{S}}\rightarrow{\mathscr{T}}}}\hat{P_{{\mathscr{T}}}^{i}}-P_{{\mathscr{T}}}^{i}\mid\mid_{2}]$
(12)

where:

$Q=e^{-(\mid\mid\hat{P_{{\mathscr{T}}}^{i}}-P_{{\mathscr{T}}}\mid\mid_{2}-\mathrm{median}(\mid\mid\hat{P_{{\mathscr{T}}}^{i}}-P_{\mathscr{T}}\mid\mid_{2}))}$
(13)

Here, $Q$ is a weighting term reduces the influence of outliers. We solve the
optimization problem with a differentiable SVD [51].

Using the estimated motion of the rigid instances we then present a loss
function that complements our 3D coupling loss for rigid regions with
additional constraints for dynamic rigid instances:

$\mathcal{L}_{3D_{\mathrm{dyn}}}^{{\mathscr{S}}\rightarrow{\mathscr{T}}}=\sum_{i\in\mathscr{I}}\theta(D_{\mathscr{T}}^{i})\mid\mid
T_{a}^{i}\hat{P_{\mathscr{T}}^{i}}-P_{\mathscr{T}}^{i}\mid\mid_{2}$ (14)

where $D_{\mathscr{T}}^{i}$ refers to the average disparity of object instance
$i$ and
$T_{a}^{i}=T_{{\mathscr{S}}\rightarrow{\mathscr{T}}}^{i}T_{\mathrm{ego}_{{\mathscr{S}}\rightarrow{\mathscr{T}}}}$
is the apparent instance motion corresponding to instance $i$. For solving the
data association of the target and source instance masks, we use the optical
flow predictions to warp the source instance masks to the target instance
masks and calculate the IoU score for all combinations between source and
target instances. Finally we use the hungarian algorithm [52] to assign the
best association. Associations with an IoU score under 0.5 are not considered
for the loss function.

Given all instance transformations
$T_{{\mathscr{S}}\rightarrow{\mathscr{T}}}^{\mathscr{I}}$, we compile the
dynamic object segmentation map $C_{\mathscr{T}}$ where all pixels are set to
1 that belong to instances that have a speed over $15$ $km/h$. All other
pixels are set to 0. Objects that belong to a non-rigid class such as person
are set always to dynamic during training.

Combining both 3D losses for rigid and dynamic regions we define the overall
3D loss as:

$\mathcal{L}_{3D}^{{\mathscr{S}}\rightarrow{\mathscr{T}}}=\mathcal{L}_{\mathrm{3D_{\mathrm{rigid}}}}^{{\mathscr{S}}\rightarrow{\mathscr{T}}}+\mathcal{L}_{3D_{\mathrm{dyn}}}^{{\mathscr{S}}\rightarrow{\mathscr{T}}}$
(15)

####  III-B8 Occlusion Filling within Optical Flow Maps using 3D Geometry and
Motion

For filling occluded areas in the optical flow maps previous works [15]
propose to fill occluded pixels with the information of the rigid flow
$F^{R}$. This method however, does handle partially occluded dynamic moving
rigid objects. We propose an enhanced version that leverages all predicted
instance motions $T^{\mathscr{I}}$ to create rigid flow maps for all pixels.
We formulate our occlusion loss as:

$\displaystyle\mathcal{L}_{\mathrm{F_{occ}}}^{{\mathscr{S}}\rightarrow{\mathscr{T}}}=\gamma(1-O_{F_{{\mathscr{T}}}})(1-C_{\mathscr{T}})\mid
F_{{\mathscr{T}}}-F_{{\mathscr{T}}}^{R}\mid+(1-O_{F_{{\mathscr{T}}}})C_{\mathscr{T}}$
(16)
$\displaystyle\cdot\sum_{i\in\mathscr{I}}\sigma(T_{a}^{i}\hat{P_{\mathscr{T}}^{i}},P_{\mathscr{T}}^{i})\mid\mid(KT_{a}^{i}\hat{P_{\mathscr{T}}^{i}}-KP_{\mathscr{T}}^{i})-F_{{\mathscr{T}}}\mid\mid_{1}$

where
$\sigma({P_{\mathscr{S}}^{i}},P_{\mathscr{T}}^{i})=e^{-\mid\mid{P_{\mathscr{S}}^{i}}-P_{\mathscr{T}}^{i}\mid\mid_{2}}$
denotes the reconstruction error between the transformed source pointcloud and
the target pointcloud and serves as a measure of the quality of
$T_{{\mathscr{S}}\rightarrow{\mathscr{T}}}^{i}$. The constant factor $\gamma$
is set to $0.1$ in all experiments. We present a schematic of our flow
occlusion filling loss on Fig. 9.

Figure 9: This figure illustrates our proposed rigid optical flow occlusion
filling loss. Leveraging the depth, optical flow, occlusion, ego-motion and
object-instance estimations we compute SE(3) transformations for all
individual instances. We then compile a optical flow from all corresponding
rigid transformations and use it as guidance for the predicted flow in
occluded regions. We employ a L1 loss in which the occluded object instance
pixels are weighted by a quality measure that is based on the 3D
reconstruction errors.

####  III-B9 Total Loss

Our final loss function is composed by a weighted combination of all
previously presented losses:

$\displaystyle\mathcal{L}=\lambda_{1}\mathcal{L}_{p}^{M_{1}}+\lambda_{2}\mathcal{L}_{s}^{M_{1}}+\lambda_{3}\mathcal{L}_{\mathrm{WSPS}}^{M_{1}}+\lambda_{4}\mathcal{L}_{\mathrm{sky}}^{M_{1}}+\lambda_{5}\mathcal{L}_{\mathrm{D_{occ}}}^{M_{2}}$
(17)
$\displaystyle+\lambda_{6}(\mathcal{L}_{3D}^{t\rightarrow(t+1)}+\mathcal{L}_{3D}^{(t+1)\rightarrow
t})+$
$\displaystyle+\lambda_{7}(\mathcal{L}_{\mathrm{Focc}}^{t\rightarrow(t+1)}+\mathcal{L}_{\mathrm{F_{occ}}}^{(t+1)\rightarrow
t})+\lambda_{8}(\mathcal{L}_{\mathrm{occ}}^{M_{1}})$

Here, $M_{1}$ means that the loss is applied with $F_{l_{t}}$,
$F_{l_{(t+1)}}$, $F_{l_{t}}^{R}$, $F_{l_{(t+1)}}^{R}$, $D_{l_{t}}$ and
$D_{r_{t}}$, while $M_{2}$ means that the loss applied on $D_{l_{t}}$ and
$D_{r_{t}}$. We apply $\mathcal{L}_{p}$, $\mathcal{L}_{s}$ and
$\mathcal{L}_{\mathrm{sky}}$ on all available levels. For computational
efficiency the smoothing loss $\mathcal{L}_{WSPS}$ and depth occlusion
guidance loss $\mathcal{L}_{\mathrm{Docc}}$ are only applied to the last two
levels 4 and 5, while the 3D reconstruction loss $\mathcal{L}_{3D}$ as well as
the flow occlusion filling loss $\mathcal{L}_{\mathrm{Focc}}$ are applied only
on the last level 5. We weight the individual loss terms by the
hyperparameters $\lambda_{1}...\lambda_{8}$.

##  IV Experimental Results

|  Left/Right Image |  Ours |  UnOS [14] |  DispSegNet [28] |  DispNetC [23]
---|---|---|---|---|---
Disparity |  |  |  |  |
|  |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_1/depth_useg_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$D1_{\mathrm{all}}=5.81$}} \end{overpic} |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_1/depth_unos_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$D1_{\mathrm{all}}=6.01$}} \end{overpic} |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_1/depth_dispsegnet_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$D1_{\mathrm{all}}=6.38$}} \end{overpic} |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_1/depth_dispnet_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$D1_{\mathrm{all}}=5.62$}} \end{overpic}
|  Left/Right Image |  Ours |  UnOS [14] |  EPC++ [15] |  FlowNet2 [30]
Flow |  |  |  |  |
|  |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_1/flow_useg_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$F1_{\mathrm{all}}=16.98$}} \end{overpic} |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_1/flow_unos_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$F1_{\mathrm{all}}=27.42$}} \end{overpic} |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_1/flow_epc_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$F1_{\mathrm{all}}=27.42$}} \end{overpic} |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_1/flow_flownet2_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$F1_{\mathrm{all}}=18.36$}} \end{overpic}
|  Left/Right Image |  Ours |  UnOS [14] |  DispSegNet [28] |  DispNetC [23]
Disparity |  |  |  |  |
|  |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_2/depth_useg_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$D1_{\mathrm{all}}=7.94$}} \end{overpic} |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_2/depth_unos_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$D1_{\mathrm{all}}=9.80$}} \end{overpic} |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_2/depth_dispsegnet_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$D1_{\mathrm{all}}=10.58$}} \end{overpic} |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_2/depth_dispnet_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$D1_{\mathrm{all}}=5.86$}} \end{overpic}
|  Left/Right Image |  Ours |  UnOS [14] |  EPC++ [15] |  FlowNet2 [30]
Flow |  |  |  |  |
|  |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_2/flow_useg_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$F1_{\mathrm{all}}=6.79$}} \end{overpic} |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_2/flow_unos_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$F1_{\mathrm{all}}=10.45$}} \end{overpic} |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_2/flow_epc_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$F1_{\mathrm{all}}=13.63$}} \end{overpic} |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_2/flow_flownet2_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$F1_{\mathrm{all}}=5.67$}} \end{overpic}
|  Left/Right Image |  Ours |  UnOS [14] |  DispSegNet [28] |  DispNetC [23]
Disparity |  |  |  |  |
|  |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_3/depth_useg_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$D1_{\mathrm{all}}=2.84$}} \end{overpic} |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_3/depth_unos_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$D1_{\mathrm{all}}=5.74$}} \end{overpic} |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_3/depth_dispsegnet_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$D1_{\mathrm{all}}=2.52$}} \end{overpic} |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_3/depth_dispnet_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$D1_{\mathrm{all}}=1.83$}} \end{overpic}
|  Left/Right Image |  Ours |  UnOS [14] |  EPC++ [15] |  FlowNet2 [30]
Flow |  |  |  |  |
|  |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_3/flow_useg_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$F1_{\mathrm{all}}=5.19$}} \end{overpic} |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_3/flow_unos_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$F1_{\mathrm{all}}=20.56$}} \end{overpic} |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_3/flow_epc_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$F1_{\mathrm{all}}=22.31$}} \end{overpic} |  \begin{overpic}[width=420.61192pt,tics=10]{publications/USegScene/figures/comparison_3/flow_flownet2_error.png} \put(60.0,25.0){{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tiny$F1_{\mathrm{all}}=10.53$}} \end{overpic}
Figure 10: Qualitative comparison to other state-of-the-art methods for
optical flow and disparity prediction. We compare our method to previous
unsupervised methods UnOS, DispSegNet and EPC++, and to the supervised methods
DispNetC and FlowNet2.

###  IV-A Implementation Details

We implement our method using the pytorch [53] framework. We train our network
in two stages with two distinct sets of hyperparameters
$\lambda_{1}...\lambda_{8}$. First, we train our network for 50k iterations
with $[10,1,0.001,0.1,0,4,0,10]$ and a initial learning rate $5e^{-05}$. In
the second stage we increase the weight of the WSPS, the depth occlusion
filling and the flow occlusion filling terms, which yield a total parameter
set as $[10,1,0.1,0.1,0.5,4,2,10]$. We train the second stage for 150k
iterations with a initial learning rate of $1e^{-5}$. During all trainings we
set the batch size to 2 and half the learning rate every 25k iterations. Adam
[54] is used as the optimizer.

We apply random cropping as data augmentation and set the crop size to
$1024x256$. Note that our odometry module needs the full and non-cropped image
to work. We thus pass the cropped and the full input in two copies of the
network, resulting in two different outputs. We compute all losses on the
outputs of the network with cropped input, but take the ego-motion estimate of
the network with full input. We fix all parameters except the ones of the
odometry estimator modules in the network with full-input. Thus, all training
of the feature encoder, depth estimator and optical flow estimator is done on
with cropped input, while optimizing the parameters of the ego-motion
estimator is done with full image input.

###  IV-B Datasets

We use the KITTI raw dataset [5] to train our network for the disparity,
optical flow and visual odometry tasks and adopt the corresponding training
and testing sets for evaluation. In order to be comparable to other works [14,
15, 28, 34, 16] we train our network on different splits which are described
in the following task-dependent sections. The semantic segmentation network is
trained on the mapillary vistas dataset [55] which comprises of 25k labelled
images that are taken from various types of cameras and perspectives. While
training all other parts of our network, the weights of the semantic
segmentation network stay fixed. Other than for the semantic network we do not
apply any form of network pretraining.

###  IV-C Disparity Evaluation

TABLE I: Comparison of disparity estimation performance with state-of-the-art approaches on the KITTI 2015 dataset. |  | Test split | Train split
---|---|---|---
Type |  Approach |  D1-bg |  D1-fg |  D1-all | D1-all
S |  DispNetC [23] |  $4.32$ |  $4.41$ |  $4.34$ | -
U |  Godard et al. [10] |  - |  - |  - | $9.19$
UnOS [14] |  $5.10$ |  $14.551$ |  $6.67$ | $5.94$
DispSegNet [28] |  $4.20$ |  $16.97$ |  $6.33$ | $6.32$
SegStereo [27] |  - |  - |  - | $10.03$
USegScene (ours) |  $\mathbf{4.12}$ |  $\mathbf{6.58}$ |  $\mathbf{4.53}$ | $\mathbf{4.50}$
TABLE II: Model ablation study for optical flow and depth on the Kitti 2015
dataset. All models are trained on the multi-view and evaluated on the
training split.

|  |  |  |  |  |  | Flow | Depth
---|---|---|---|---|---|---|---|---
Nr. |  Depth |  Semantic |  Semantic |  WSPS |  3D |  Rigid | All | Noc | All
|  Input |  Input |  Matching |  |  Coupling |  Occlusion Filling |  F1 |  EPE |  F1 |  EPE |  D1
1 |  ✓ |  ✓ |  ✗ |  ✗ |  ✗ |  ✗ |  $85.81$ |  $4.10$ |  $74.99$ |  $10.69$ |  $9.66$
2 |  ✓ |  ✓ |  ✓ |  ✗ |  ✗ |  ✗ |  $86.56$ |  $3.85$ |  $75.55$ |  $10.54$ |  $9.50$
3 |  ✓ |  ✓ |  ✓ |  ✓ |  ✗ |  ✗ |  $88.28$ |  $3.26$ |  $77.49$ |  $9.52$ |  $6.71$
4 |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  ✗ |  $88.37$ |  $3.26$ |  $77.54$ |  $9.41$ |  $5.84$
5 |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  $88.93$ |  $2.29$ |  $82.29$ |  $3.80$ |  $6.00$
6 |  ✓ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  $89.30$ |  $2.34$ |  $86.96$ |  $3.15$ |  $6.30$
7 |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  $\mathbf{90.62}$ |  $\mathbf{2.03}$ |  $\mathbf{88.56}$ |  $\mathbf{2.79}$ |  $\mathbf{5.73}$

Nr. |  Qualitative Flow |  Qualitative Depth
---|---|---
1 |  |
2 |  |
3 |  |
4 |  |
5 |  |
6 |  |
7 |  |

In order to evaluate the disparity estimations we train our network on the
KITTI raw dataset and remove all scenes that contain images of the KITTI 2012
or 2015 optical flow/ disparity sets, following [14, 28, 15]. In Table I we
show the D1 error [17] on the training and testing splits of KITTI and compare
it to various state-of-the-art methods. It can be seen that our method
outperforms other unsupervised approaches by a large margin. Our approach
performs on par with the fully supervised method DispNetC [23], which is
trained on the ground-truth data from KITTI 2015.

Fig. 10 illustrates multiple comparisons to other unsupervised and
unsupervised methods and reveals that our network predicts significantly
smoother disparity maps while being sharp at depth boundaries. Additionally,
our disparity predictions are significantly less blurred out in occluded
regions. Since we regularize the disparity predictions in regions that belong
to sky, we do not get artifacts that occur in previous works. In contrast to
other works, our network directly predicts the occlusions. An example of a
occlusion prediction can be seen in Fig. 1.

In Table. II we further present different configurations of our approach. We
observe that the all presented losses consistently improve the results, while
the WSPS and 3D coupling losses yield the highest gains of $41.6\%$ and
$14.9\%$ respectively. If the semantic inputs are neglected in the disparity
estimator modules the performs drops by $9.9\%$, showing the benefit of
complementary semantic inputs.

###  IV-D Optical Flow Evaluation

TABLE III: Comparison of optical flow estimation performance with state-of-the-art approaches on the KITTI 2015 dataset. |  | Test split | Train split
---|---|---|---
|  | F1 | EPE | F1
Approach |  Split |  bg-all |  fg-all |  all |  all |  all
UnFlow-CSS [9] |  raw |  - |  - |  - |  $8.10$ |  -
GeoNet [13] |  raw |  - |  - |  - |  $10.81$ |  -
DF-Net [16] |  raw |  - |  - |  $25.70$ |  $8.98$ |  -
Competitive-Collaboration-uft [48] |  raw |  - |  - |  $25.27$ |  $5.66$ |  -
EPC++ (stereo) [15] |  raw |  $19.24$ |  $26.93$ |  $20.52$ |  $5.43$ |  -
UnOS [14] |  raw |  $16.93$ |  $23.34$ |  $18.00$ |  $5.58$ |  -
Multi-frame [56] |  mv |  $22.67$ |  $24.27$ |  $22.94$ |  $6.59$ |  -
DSTFlow [57] |  mv |  - |  - |  $39.00$ |  $16.79$ |  -
OccAwareFlow [33] |  mv |  - |  - |  $31.20$ |  $8.88$ |  -
DDFlow [35] |  mv |  $13.08$ |  $20.40$ |  $14.29$ |  $5.72$ |  -
SelFlow [34] |  mv |  $12.68$ |  $21.74$ |  $14.19$ |  $4.84$ |  -
USegScene (ours) |  raw |  - |  - |  - |  $4.78$ |  $14.47$
USegScene (ours) |  mv |  $\mathbf{11.69}$ |  $\mathbf{20.23}$ |  $\mathbf{13.11}$ |  $\mathbf{2.79}$ |  $\mathbf{11.97}$

As in the disparity evaluation we train our network on the raw dataset without
the scenes that hold images of the 2015/2012 optical flow training/testing
sets that we use for evaluation. This enables us to be comparable to UnOS
[14], EPC++ [15], DF-Net[16], Competitive-Collaboration [48], GeoNet [13] and
UnFlow [9].

Since most of the dynamic object instances of the KITTI dataset are contained
in the corresponding scenes of the optical flow/disparity training/testing
sets, many recent works [34, 35, 57, 33, 56] leverage the multi-view
extensions of those sets for training. Here, all images with ground-truth
maps, and their neighbors, are removed for training.

We train our model on both sets and name the sets ’raw’ and ’mv’ respectively.
In table III, we compare our optical flow predictions to all other methods
using the average end-point error as well as the F1 error [5].

We found that training on the multi-view dataset yields consistently better
results, which may be due to higher occurrences of dynamic objects in the
dataset.

Our method outperforms other methods by a significant margin.

The qualitative results, shown in Fig. 10 reveal that our method shows
especially great performance in occluded regions and regions with high
displacement vectors.

As in the disparity evaluation section, we also present an ablation study for
the optical flow estimation in Fig. II. We observe that semantic feature
matching improves the EPE in the non-occluded benchmark by $6.5\%$, WSPS by
another $18.1\%$ and rigid occlusion filling by further $60.5\%$. In contrast
to the disparity case we see only little improvements when utilizing the 3D
coupling loss. When looking at the EPE errors in occluded and non-occluded
regions one can note that our rigid occlusion filling loss improves the EPE
from $9.41$ to $2.79$.

In order to better understand the influence of our novel network architecture
we additionally conduct experiments where we neglect the semantic inputs $S$
and $B$ to all modules. We can note that removing the semantic inputs yields
to $1.8\%$ performance decrease. Furthermore, we note a $7.6\%$ decrease in
the performance of the optical flow prediction when neglecting the depth and
depth-dependent inputs $D_{l_{t}}$ and $F_{l_{t}}^{R}$ at the input of the
optical flow estimator modules. This confirms that geometric inputs ease the
optical flow estimation tasks.

In Fig. 11 we show a example optical flow prediction with and without rigid
optical flow occlusion filling. Here, rigid optical flow occlusion filling not
only drastically reduces the error in occluded static regions but also in
occluded rigid dynamic regions, as we guide the prediction by individual SE(3)
transformations of the dynamic rigid object.

|
---|---
|
|
Figure 11:  This figure shows the utility of the optical flow rigid occlusion
filling loss. It can be observed that the model, which is trained with rigid
occlusion filling predicts much more accurate optical flow in occluded static
as well as dynamic regions. Top: consecutive rgb images. Middle: optical flow
prediction and corresponding error-map without rigid flow occlusion filling.
Bottom: optical flow prediction and corresponding error-map with rigid flow
occlusion filling.

###  IV-E Ego-Motion Estimation

We follow [15, 14] and use the KITTI odometry sequences 00-08 as training and
sequences 09, 10 as validation datasets. Table IV shows a comparison to other
unsupervised state-of-the-art methods using the metrics described in [17].
Note that we only use 2-frames to predict the ego-motion, while others [12,
13, 58] require more. The numbers suggest that our method outperforms other
approaches by a large margin, showing the effectiveness of our method.

TABLE IV: Odometry evaluation on the KITTI odometry dataset. We compare our approach to other state-of-the-art methods on sequence 9 and 10. Approach | Seq. 09 | Seq. 10
---|---|---
|  $t_{err}\%$ |  $r_{err}(\frac{\degree}{100m})$ |  $t_{err}\%$ |  $r_{err}(\frac{\degree}{100m})$
Zhou et al. [12] |  $30.75$ |  $11.41$ |  $44.22$ |  $12.42$
GeoNet [13] |  $39.43$ |  $14.30$ |  $28.99$ |  $8.85$
EPC++ [15] |  $8.84$ |  $3.34$ |  $8.86$ |  $3.18$
UnOS [14] |  $5.21$ |  $1.80$ |  $5.20$ |  $2.18$
Uscene (ours) |  $\mathbf{2.71}$ |  $\mathbf{0.52}$ |  $\mathbf{3.43}$ |  $\mathbf{0.72}$

##  V Conclusion

In this paper we propose a novel unsupervised framework for learning depth,
optical flow and ego-motion. We propose a new architecture that solves all
individual tasks jointly, using shared network components and leveraging
semantic maps as complementary information. We present novel loss functions
for complementary semantic feature matching, semantic regularization,
3D-constraints and occlusion handling. To quantify our work, we present state-
of-the-art quantitative and qualitative results on the KITTI benchmark [17].

## References

  * [1] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos.  Orb-slam: a versatile and accurate monocular slam system.  IEEE transactions on robotics, 31(5):1147–1163, 2015.
  * [2] Thao Dang, Christian Hoffmann, and Christoph Stiller.  Fusing optical flow and stereo disparity for object tracking.  In Proceedings. The IEEE 5th International Conference on Intelligent Transportation Systems, pages 112–117. IEEE, 2002.
  * [3] Wongun Choi.  Near-online multi-target tracking with aggregated local flow descriptor.  In Proceedings of the IEEE international conference on computer vision, pages 3029–3037, 2015.
  * [4] Igor Cvišić and Ivan Petrović.  Stereo odometry based on careful feature selection and tracking.  In 2015 European Conference on Mobile Robots (ECMR), pages 1–6. IEEE, 2015.
  * [5] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  International Journal of Robotics Research (IJRR), 2013.
  * [6] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez.  The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3234–3243, 2016.
  * [7] N.Mayer et al.  A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation.  In CVPR, 2016.
  * [8] J Yu Jason, Adam W Harley, and Konstantinos G Derpanis.  Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness.  In European Conference on Computer Vision, pages 3–10. Springer, 2016.
  * [9] Simon Meister, Junhwa Hur, and Stefan Roth.  Unflow: Unsupervised learning of optical flow with a bidirectional census loss.  In Thirty-Second AAAI Conference on Artificial Intelligence, 2018\.
  * [10] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow.  Unsupervised monocular depth estimation with left-right consistency.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 270–279, 2017.
  * [11] Ruihao Li, Sen Wang, Zhiqiang Long, and Dongbing Gu.  Undeepvo: Monocular visual odometry through unsupervised deep learning.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7286–7291. IEEE, 2018.
  * [12] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe.  Unsupervised learning of depth and ego-motion from video.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6612–6619, July 2017.
  * [13] Zhichao Yin and Jianping Shi.  Geonet: Unsupervised learning of dense depth, optical flow and camera pose.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1983–1992, 2018.
  * [14] Yang Wang, Peng Wang, Zhenheng Yang, Chenxu Luo, Yi Yang, and Wei Xu.  Unos: Unified unsupervised optical-flow and stereo-depth estimation by watching videos.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8071–8081, 2019.
  * [15] Chenxu Luo, Zhenheng Yang, Peng Wang, Yang Wang, Wei Xu, Ram Nevatia, and Alan L. Yuille.  Every pixel counts ++: Joint learning of geometry and motion with 3d holistic understanding.  CoRR, abs/1810.06125, 2018.
  * [16] Yuliang Zou, Zelun Luo, and Jia-Bin Huang.  Df-net: Unsupervised joint learning of depth and flow using cross-task consistency.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 36–53, 2018.
  * [17] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
  * [18] Daniel Scharstein and Richard Szeliski.  A taxonomy and evaluation of dense two-frame stereo correspondence algorithms.  International journal of computer vision, 47(1-3):7–42, 2002.
  * [19] Jure Zbontar, Yann LeCun, et al.  Stereo matching by training a convolutional neural network to compare image patches.  Journal of Machine Learning Research, 17(1-32):2, 2016.
  * [20] Amit Shaked and Lior Wolf.  Improved stereo matching with constant highway networks and reflective confidence learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4641–4650, 2017.
  * [21] Daniel Scharstein, Heiko Hirschmüller, York Kitajima, Greg Krathwohl, Nera Nešić, Xi Wang, and Porter Westling.  High-resolution stereo datasets with subpixel-accurate ground truth.  In German conference on pattern recognition, pages 31–42. Springer, 2014.
  * [22] Adrien Gaidon, Qiao Wang, Yohann Cabon, and Eleonora Vig.  Virtual worlds as proxy for multi-object tracking analysis.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4340–4349, 2016.
  * [23] Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox.  A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4040–4048, 2016.
  * [24] Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry.  End-to-end learning of geometry and context for deep stereo regression.  In Proceedings of the IEEE International Conference on Computer Vision, pages 66–75, 2017.
  * [25] Jia-Ren Chang and Yong-Sheng Chen.  Pyramid stereo matching network.  CoRR, abs/1803.08669, 2018.
  * [26] Yiran Zhong, Yuchao Dai, and Hongdong Li.  Self-supervised learning for stereo matching with self-improving ability.  arXiv preprint arXiv:1709.00930, 2017.
  * [27] Guorun Yang, Hengshuang Zhao, Jianping Shi, Zhidong Deng, and Jiaya Jia.  Segstereo: Exploiting semantic information for disparity estimation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 636–651, 2018.
  * [28] Junming Zhang, Katherine A Skinner, Ram Vasudevan, and Matthew Johnson-Roberson.  Dispsegnet: Leveraging semantics for end-to-end learning of disparity estimation from stereo imagery.  IEEE Robotics and Automation Letters, 4(2):1162–1169, 2019.
  * [29] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox.  Flownet: Learning optical flow with convolutional networks.  In Proceedings of the IEEE international conference on computer vision, pages 2758–2766, 2015.
  * [30] E. Ilg et al.  Flownet 2.0: Evolution of optical flow estimation with deep networks.  arXiv:1612.01925, Dec 2016.
  * [31] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.  Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume.  CoRR, abs/1709.02371, 2017.
  * [32] Tak-Wai Hui, Xiaoou Tang, and Chen Change Loy.  A lightweight optical flow cnn–revisiting data fidelity and regularization.  arXiv preprint arXiv:1903.07414, 2019.
  * [33] Yang Wang, Yi Yang, Zhenheng Yang, Liang Zhao, Peng Wang, and Wei Xu.  Occlusion aware unsupervised learning of optical flow.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4884–4893, 2018.
  * [34] Pengpeng Liu, Michael Lyu, Irwin King, and Jia Xu.  Selflow: Self-supervised learning of optical flow.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4571–4580, 2019.
  * [35] Pengpeng Liu, Irwin King, Michael R Lyu, and Jia Xu.  Ddflow: Learning optical flow with unlabeled data distillation.  arXiv preprint arXiv:1902.09145, 2019.
  * [36] Christian Forster, Matia Pizzoli, and Davide Scaramuzza.  Svo: Fast semi-direct monocular visual odometry.  In 2014 IEEE international conference on robotics and automation (ICRA), pages 15–22. IEEE, 2014.
  * [37] Jakob Engel, Vladlen Koltun, and Daniel Cremers.  Direct sparse odometry.  IEEE transactions on pattern analysis and machine intelligence, 40(3):611–625, 2017.
  * [38] Sen Wang, Ronald Clark, Hongkai Wen, and Niki Trigoni.  Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks.  In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 2043–2050. IEEE, 2017.
  * [39] Sen Wang, Ronald Clark, Hongkai Wen, and Niki Trigoni.  End-to-end, sequence-to-sequence probabilistic visual odometry through deep neural networks.  The International Journal of Robotics Research, 37(4-5):513–542, 2018.
  * [40] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe.  Unsupervised learning of depth and ego-motion from video.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1851–1858, 2017.
  * [41] Ganesh Iyer, J Krishna Murthy, Gunshi Gupta, Madhava Krishna, and Liam Paull.  Geometric consistency for self-supervised end-to-end visual odometry.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 267–275, 2018.
  * [42] Zhe Cao, Abhishek Kar, Christian Hane, and Jitendra Malik.  Learning independent object motion from unlabelled stereoscopic videos.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5594–5603, 2019.
  * [43] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500, 2017.
  * [44] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587, 2017.
  * [45] Ethan Eade.  Lie groups for 2d and 3d transformations.
  * [46] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and koray kavukcuoglu.  Spatial transformer networks.  In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2017–2025. Curran Associates, Inc., 2015.
  * [47] Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [48] Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J Black.  Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12240–12249, 2019.
  * [49] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [50] Jonathan T Barron.  A general and adaptive robust loss function.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4331–4339, 2019.
  * [51] Mike Giles.  An extended collection of matrix derivative results for forward and reverse mode automatic differentiation.  2008\.
  * [52] Harold W Kuhn.  The hungarian method for the assignment problem.  Naval research logistics quarterly, 2(1-2):83–97, 1955.
  * [53] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in PyTorch.  In NIPS Autodiff Workshop, 2017.
  * [54] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [55] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulò, and Peter Kontschieder.  The mapillary vistas dataset for semantic understanding of street scenes.  In International Conference on Computer Vision (ICCV), 2017.
  * [56] Joel Janai, Fatma Guney, Anurag Ranjan, Michael Black, and Andreas Geiger.  Unsupervised learning of multi-frame optical flow with occlusions.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 690–706, 2018.
  * [57] Zhe Ren, Junchi Yan, Bingbing Ni, Bin Liu, Xiaokang Yang, and Hongyuan Zha.  Unsupervised deep learning for optical flow estimation.  In Thirty-First AAAI Conference on Artificial Intelligence, 2017\.
  * [58] Reza Mahjourian, Martin Wicke, and Anelia Angelova.  Unsupervised learning of depth and ego-motion from monocular video using 3d geometric constraints.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5667–5675, 2018.