# Two-in-One Depth: Bridging the Gap Between Monocular and Binocular Self-
supervised Depth Estimation

Zhengming Zhou and Qiulei Dong ✉
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA
School of Artificial Intelligence, UCAS
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Monocular and binocular self-supervised depth estimations are two important
and related tasks in computer vision, which aim to predict scene depths from
single images and stereo image pairs respectively. In literature, the two
tasks are usually tackled separately by two different kinds of models, and
binocular models generally fail to predict depth from single images, while the
prediction accuracy of monocular models is generally inferior to binocular
models. In this paper, we propose a Two-in-One self-supervised depth
estimation network, called TiO-Depth, which could not only compatibly handle
the two tasks, but also improve the prediction accuracy. TiO-Depth employs a
Siamese architecture and each sub-network of it could be used as a monocular
depth estimation model. For binocular depth estimation, a Monocular Feature
Matching module is proposed for incorporating the stereo knowledge between the
two images, and the full TiO-Depth is used to predict depths. We also design a
multi-stage joint-training strategy for improving the performances of TiO-
Depth in both two tasks by combining the relative advantages of them.
Experimental results on the KITTI, Cityscapes, and DDAD datasets demonstrate
that TiO-Depth outperforms both the monocular and binocular state-of-the-art
methods in most cases, and further verify the feasibility of a two-in-one
network for monocular and binocular depth estimation. The code is available at
https://github.com/ZM-Zhou/TiO-Depth_pytorch.

††footnotetext: ✉ corresponding author

##  1 Introduction

Figure 1: Diagrams of three kinds of self-supervised depth estimation models
trained with stereo pairs: (a) Monocular model is tested with a single image
but needs stereo pairs during training. (b) Binocular model is trained and
tested with stereo pairs, but could not predict depths from a single image;
(c) TiO-Depth could be tested with both single images and stereo pairs.

With the development of deep learning techniques, deep-neural-network-based
methods have shown their effectiveness for handling both the monocular and
binocular depth estimation tasks, which pursue depths from single images and
stereo image pairs respectively [5, 14, 16, 62]. Since it is time-consuming
and labor-intensive to obtain abundant high-quality ground truth scene depths,
monocular and binocular self-supervised depth estimation methods, which do not
require ground truth depths for training, have attracted increasing attention
in recent years [17, 20, 56, 60].

It is noted that the above two tasks are closely related, as shown in Fig. 1:
both the monocular and binocular methods output the same type of results
(_i.e_., depth maps), and some self-supervised monocular methods [7, 19, 58]
use the same type of training data (_i.e_., stereo pairs) as the binocular
models. Their main difference is that the monocular task is to predict depths
from a single image, while the binocular task is to predict depths from a
stereo pair. Due to this difference, the two tasks have been handled
separately by two different kinds of models (_i.e_., monocular and binocular
models) in literature. Compared with the monocular models that learn depths
from single image features, the binocular models focus on learning depths from
the geometric features (_e.g_. cost volumes [60]) generated with stereo pairs,
and consequently, they generally perform better than the monocular models but
could not predict depth from a single image. Moreover, it is found in [7] that
although the whole performances of the monocular models are poorer than the
binocular ones, the monocular models still perform better on some special
local regions, _e.g_., the occluded regions around objects which could only be
seen at a single view. Inspired by this finding, some monocular (or binocular)
models employed a separate binocular (or monocular) model to boost their
performances in their own task [1, 7, 9, 15, 40, 42, 49]. All the above issues
naturally raise the following problem: Is it feasible to explore a general
model that could not only compatibly handle the two tasks, but also improve
the prediction accuracy?

Obviously, a general model has the following potential advantages in
comparison to the separate models: (1) Flexibility: This model could
compatibly deal with both the monocular and binocular tasks, and it would be
of great benefit to the platforms with a binocular system in the real
application, where one camera in the binocular system might be occasionally
occluded or even broken down. (2) High Efficiency: This model has the
potential to perform better than both monocular and binocular models, while
the number of its parameters is less than that of two separate models.

Addressing the aforementioned problem and potential advantages of a general
depth estimation model, in this paper, we propose a Two-in-One model for both
monocular and binocular self-supervised depth estimations, called TiO-Depth.
TiO-Depth employs a monocular model as a sub-network of a Siamese
architecture, so that the whole architecture could take stereo images as
input. Considering that the two sub-networks extract image features
independently, we design a monocular feature matching module to fuse features
from the two sub-networks for binocular prediction. Then, a multi-stage joint-
training strategy is proposed for training TiO-Depth in a self-supervised
manner and boosting its accuracy in the two tasks by combining their relative
advantages and alleviating their disadvantages.

In sum, our main contributions include:

  * •

We propose a novel self-supervised depth estimation model called TiO-Depth,
which could handle both the monocular and binocular depth estimation tasks.

  * •

We design a dual-path decoder with the monocular feature matching modules for
aggregating the features from either single images or stereo pairs, which may
provide new insights into the design of the self-supervised depth estimation
network.

  * •

We propose a multi-stage joint-training strategy for training TiO-Depth, which
is helpful for improving the performances of TiO-Depth in the two tasks.

Figure 2: Architecture of TiO-Depth. TiO-Depth employs a Siamese architecture
and each sub-network is comprised of a Monocular Feature Encoder and a dual-
path decoder. The features extracted by the encoder are passed through the
decoder via different paths for handling different tasks. $\\{P_{m},P_{s}\\}$
denote the probability volumes predicted by the monocular and binocular paths
respectively, while $\\{D_{m},D_{s}\\}$ are the corresponding depth maps. The
superscripts ‘l’ and ‘r’ denote the left and right views respectively.

##  2 Related work

###  2.1 Self-supervised monocular depth estimation

Self-supervised monocular depth estimation methods take multi-view images as
training data and learn to estimate the depth from a single input image with
the image reconstruction. The existing methods could be categorized into two
groups according to the training data: video training methods and stereo
training methods.

The methods trained with video sequences [6, 8, 20, 27, 33, 35, 46, 50, 61,
63, 28] needed to estimate scene depths and camera poses simultaneously. Zhou
_et al_. [63] proposed an end-to-end framework which is comprised of two
separate networks for predicting depths and camera poses. Godard _et al_. [20]
designed a per-pixel minimum reprojection loss with an auto-mask and a full-
resolution sampling for training the model to learn more accurate depths. SD-
SSMDE [46] utilized a self-distillation framework where a student network was
trained by the absolute depth pseudo labels generated with a teacher network.
Several methods [8, 27, 33, 35] used extra semantic information for improving
the performance, and the frameworks explored in [6, 61] jointly learnt depth,
camera pose and optical flow. Additionally, the multi-frame monocular depth
estimation was handled in [26, 59], which predicted more accurate depths by
taking two frames of a monocular video as input.

The methods trained with stereo image pairs [3, 7, 9, 17, 19, 21, 45, 47, 52,
58, 67, 65, 64] generally predicted scene depths by estimating the disparity
between the stereo pair. Godard _et al_. [19] designed a left-right disparity
consistency loss to improve its robustness. Zhu _et al_. [67] proposed an edge
consistency loss between the depth map and the semantic segmentation map,
while a stereo occlusion mask was proposed for alleviating the influence of
the occlusion problem during training. An indirect way of learning depths was
proposed in [3, 21, 22], where the model outputted a probability volume of a
set of discrete disparities for depth prediction. The self-distillation
technique [24] was incorporated in [45, 65] to boost the performance of the
model by using the reliable results predicted by itself. Considering that the
stereo pairs were available at the training stage, Watson _et al_. [58]
proposed to utilize the disparities generated with Semi Global Matching [29]
as the ‘Depth Hints’ to improve the accuracy. The frameworks that trained a
monocular depth estimation network with the pseudo labels selected from the
results of a binocular depth estimation network were proposed in [9, 7].

###  2.2 Self-supervised binocular depth estimation

Binocular depth estimation (so called as stereo matching) aims to estimate
depths by taking stereo image pairs as input [4, 5, 29, 62]. Recently, self-
supervised binocular depth estimation methods [63, 60, 56, 38, 55, 31, 1] were
proposed for overcoming the limitation of the ground truth. Zhou _et al_. [63]
proposed a framework for learning stereo matching in an iterative manner,
which was guided by the left-right check. UnOS [56] and Flow2Stereo [38] were
proposed for predicting optical flow and binocular depth simultaneously, where
the geometrical consistency between the two types of the predicted results was
used to improve the accuracy of them. Wang _et al_. [55] proposed a parallax-
attention mechanism to learn the stereo correspondence. H-Net [31] was
proposed to learn binocular depths with a Siamese network and an epipolar
attention mechanism.

##  3 Methodology

In this section, we firstly introduce the architecture of the proposed TiO-
Depth, including the details of the dual-path decoder and the Monocular
Feature Matching (MFM) module. Then, we describe the multi-stage joint-
training strategy and the loss functions for training TiO-Depth.

###  3.1 Overall architecture

Since TiO-Depth is to handle both monocular and binocular depth estimation
tasks, it should be able to predict depths from both single image features and
geometric features, while the binocular and monocular models could only
estimate depths from one type of the features respectively. To this end, TiO-
Depth utilizes a Siamese architecture as shown in Fig. 2, and each of the two
sub-networks is used as a monocular model. They predict the monocular depth
$D_{m}$ from a single image $I\in\mathbb{R}^{3\times H\times W}$ for avoiding
the model learning depths only based on the geometric features, where
$\\{H,W\\}$ denote the height and width of the image. The parameters of the
two sub-networks are shared, and they consist of a monocular feature encoder
and a decoder. For effectively extracting geometric features from available
stereo pairs for the binocular task, the dual-path decoder is proposed as the
decoder part of the sub-networks, where a binocular path is added to the path
for the monocular task (called monocular path). In the binocular path, the MFM
modules are added to learn the geometric features by matching the monocular
features extracted by the two sub-networks from a stereo pair and integrate
them into the input features. Accordingly, the full TiO-Depth is used to
predict binocular depths $\\{D^{l}_{s},D^{r}_{s}\\}$.

Specifically, a modified Swin-transformer [39] is adopted as the encoder as
done in [65], which extracts 4 image features $\\{C_{i}\\}^{4}_{i=1}$ with the
resolutions of $\\{\frac{H}{2^{i}}\times\frac{W}{2^{i}}\\}^{4}_{i=1}$. We
detail the dual-path decoder and the MFM module as following.

###  3.2 Dual-path decoder

As shown in Fig. 2, the dual-path decoder is used to gradually aggregate the
extracted image features for depth prediction, which consists of three Self-
Distilled Feature Aggregation (SDFA) blocks [65], one decoder block [20],
three monocular feature matching (MFM) modules, and two $3\times 3$
convolutional layers used as the output layers. The features could be passed
through different modules via different paths for the monocular and binocular
tasks.

For monocular depth estimation, the multi-scale features
$\\{C_{i}\\}^{4}_{i=1}$ are gradually aggregated by the SDFA blocks and the
decoder block, which is defined as the monocular path. The SDFA block was
proposed in [65] for aggregating the features with two resolutions and
maintaining the contextual consistency, which takes a low resolution decoder
feature $F_{i+1}$ (Specifically, $F_{5}$ = $C_{4}$) and a high resolution
encoder feature $C_{i-1}$, outputting a new decoder feature with the same
shape as $C_{i-1}$. The decoder block is comprised of two $3\times 3$
convolutional layers with the ELU activation [10] and an upsample operation
for generating a high resolution feature $F_{i}$ from the output of the last
block. The output layer is to generate a discrete disparity volume
$V\in\mathbb{R}^{N\times H\times W}$ from the last decoder feature $F_{1}$,
where $N$ is the number of the discrete disparity levels.

It is noted that two volumes (defined as the auxiliary volume $V_{a}$ and the
final volume $V_{m}$) could be generated for monocular depth estimation by
using different offset learning branches in SDFA blocks at the training stage,
which would be trained with the photometric loss and the distilled loss at
different steps respectively. More details would be described in Sec. 3.4.
Accordingly, the branches in SDFA used to generate the two volumes are called
auxiliary branch and the final branch. Since $V_{a}$ is only used at the
training stage, it is not illustrated in Fig. 2, and the depth calculated
based on $V_{m}$ is the final monocular result.

For binocular depth estimation, the dual-path decoders in the two sub-networks
are utilized for processing left and right image features via the binocular
path. In this path, MFM modules take the decoder features
$\\{F^{l}_{i},F^{r}_{i}\\}^{4}_{i=2}$ outputted by the SDFA blocks (where the
auxiliary branch is used) for generating the corresponding stereo features
$\\{{F^{l}_{i}}^{\prime},{F^{r}_{i}}^{\prime}\\}^{4}_{i=2}$ by incorporating
the stereo knowledge. The left and right stereo discrete disparity volumes
$\\{V^{l}_{s},V^{r}_{s}\\}$ are obtained by passing the last decoder features
$\\{F^{l}_{1},F^{r}_{1}\\}$ to another output layer in each decoder.

For obtaining the depth map from the discrete disparity volume $V$, as done in
[2, 65], a set of discrete disparity levels $\\{b_{n}\\}_{n=0}^{N-1}$ is
generated with the mirrored exponential disparity discretization by given the
maximum and minimum disparities $[b_{\rm min},b_{\rm max}]$. Then, a
probability volume $P$ is obtained by normalizing $V$ through a softmax
operation along the first (_i.e_. channel) dimension, and a disparity map is
calculated by weighted summing of $\\{b_{n}\\}_{n=0}^{N-1}$ with the
corresponding channels in $P$:

$d=\sum_{n=0}^{N-1}P_{n}\odot b_{n}\quad,$ (1)

where $P_{n}$ denotes the $n^{\rm th}$ channel of $P$ and ‘$\odot$’ is the
element-wise multiplication. Given the baseline length $B$ of the stereo pair
and the horizontal focal length $f_{x}$ of the camera, the depth map is
calculated via $D=\frac{Bf_{x}}{d}$.

Figure 3: Architecture of Monocular Feature Matching (MFM) module. ‘ⓒ’ denotes
the concatenation operation and ‘SE.’ is the SE convolutional layer [30].

###  3.3 Monocular Feature Matching (MFM) module

Given the features $\\{F^{l},F^{r}\\}\in\mathbb{R}^{C\times H^{\prime}\times
W^{\prime}}$ obtained from the two decoders of the two sub-networks, MFM
utilizes the cross-attention mechanism [54] for generating the cost volume at
the left (or right) view and integrates it into the corresponding feature for
outputting a stereo feature that has the same shape of input the feature.
$\\{C,H^{\prime},W^{\prime}\\}$ are the channel, height, and width of the
features. Without loss of generality, as shown in Fig. 3, for obtaining the
stereo feature at the left-view ${F^{l}}^{\prime}$, MFM firstly applies two
$1\times 1$ convolutional layers to generate the left-view query feature
$Q^{l}$ and the right-view key feature $K^{r}$ from $\\{F^{l},F^{r}\\}$
respectively. As done in [26], the left-view cost volume is generated based on
the attention scores between $Q^{l}$ and a set of shifted $K^{r}$, where each
score map $S^{l}_{n}\in\mathbb{R}^{1\times H^{\prime}\times W^{\prime}}$ is
calculated between $Q^{l}$ and $K^{r}$ shifted with $b_{n}^{\prime}$, which is
formulated as:

$S^{l}_{n}=\frac{{\rm sum}(Q^{l}\odot K_{n}^{r})}{\sqrt{C}}\quad,$ (2)

where $K_{n}^{r}$ denotes the $K^{r}$ shifted with $b_{n}^{\prime}$, and
‘${\rm sum}(\cdot)$’ is a sum operation along the first dimension. Then, the
cost volume $A^{l}\in\mathbb{R}^{N\times H^{\prime}\times W^{\prime}}$ is
obtained by concatenating $S^{l}_{n}$ generated with all the disparity levels
$\\{b_{n}^{\prime}=\frac{W^{\prime}}{W}b_{n}\\}^{N-1}_{n=0}$ and normalizing
it with a softmax operation along the first dimension:

$A^{l}={\rm softmax}\left([\\{S^{r}_{n}\\}^{N-1}_{n=0}]\right)\quad,$ (3)

where ‘$[\cdot]$’ denotes the concatenation operation. For integrating the
stereo knowledge in the cost volume into the decoder feature to obtain the
stereo feature ${F^{l}}^{\prime}$, $F^{l}$ and $A^{l}$ are concatenated and
passed through a $3\times 3$ SE convolutional layer [30] with the ELU
activation:

${F^{l}}^{\prime}={\rm SE}\left([A^{l},F^{l}]\right)\quad.$ (4)

Figure 4: Multi-stage joint-training strategy. There are three steps in each
training iteration, where TiO-Depth is trained for different tasks. The
training at the current step could be benefited from the results generated at
the previous steps. The modules that do not optimized in each step are denoted
by grey and the _italic font_.

###  3.4 Multi-stage joint-training strategy

TiO-Depth is trained with stereo image pairs in a self-supervised manner.
Considering the motivation of the architecture of TiO-Depth and the different
advantages and constraints of the two tasks, we design the multi-stage
training strategy as shown in Fig. 4. There are three stages in the strategy,
where the training iterations are divided into one, two and three steps
respectively. At the last two stages, the training at the current step could
be benefited from the results generated at the previous steps. We detail the
three steps as following.

Step (1). TiO-Depth is trained for learning monocular depth estimation under
monocular constraints at this step. The discrete depth constraint [64, 2] is
used to generate a left-view reconstructed image $\hat{I}^{l}_{a}$ with the
right-view auxiliary volume $V^{r}_{a}$ (generated with the auxiliary branches
in SDFAs as mentioned in Sec. 3.2) and the right-view real image $I^{r}$. As
done in [2, 65], the monocular loss $L_{M}$ for training TiO-Depth contains a
reconstruction loss $L_{rec1}$ for reflecting the difference between
$\hat{I}^{l}_{a}$ and $I^{l}$, and an edge-aware smoothness loss $L_{smo1}$:

$L_{M}=L_{rec1}+\lambda_{1}L_{smo1},$ (5)

where $\lambda_{1}$ is a preset weight parameters. All the parameters in TiO-
Depth except MFMs are optimized at this step.

Step (2). TiO-Depth is trained for learning binocular depth estimation under
binocular constraints and some monocular results obtained at step (1). The
continuous depth constraint [64, 7] is used to reconstruct a left-view image
$\tilde{I}^{l}_{s}$ by taking the right-view image $I^{r}$ and the predicted
left-view depth map $D^{l}_{s}$ as the input. Then, a stereo loss is adopted
to train the network, which consists of the following terms:

The stereo reconstruction loss term $L_{rec2}$ is formulated as a weighted sum
of the $L_{1}$ loss and the structural similarity (SSIM) loss [57] as done in
[7, 20]. Considering the relative advantage of the monocular results on the
occluded regions, the occluded pixels in $I^{l}$ are replaced by the
corresponding pixels in a monocular reconstructed image $\tilde{I}^{l}_{a}$
calculated with the auxiliary monocular depth map $D^{l}_{a}$:

$L_{rec2}=\alpha\left\|\tilde{I}^{l}_{s}-{I^{l}}^{\prime}\right\|_{1}+(1-\alpha){\rm
SSIM}{(\tilde{I}^{l}_{s},{I^{l}}^{\prime})}\quad,$ (6)
${I^{l}}^{\prime}=M^{l}_{occ}\odot
I^{l}+(1-M^{l}_{occ})\odot\tilde{I}^{l}_{a}\quad,$ (7)

where $\alpha$ is a balance parameter and ‘$\|\cdot\|_{1}$’ denotes the
$L_{1}$ norm. $M^{l}_{occ}$ is an occlusion mask generated with the auxiliary
monocular disparity $d^{l}_{a}$ as done in [67], where the values are zeros in
the occluded regions, and ones otherwise.

The cost volume loss term $L_{cos}$ is adopted to guide the cost volumes
$\\{A^{l}_{i}\\}^{3}_{i=1}$ generated in MFMs through the auxiliary monocular
probability volume $P^{l}_{a}$, which is formulated as:

$L_{cos}=\sum^{3}_{i=1}{\frac{1}{\Omega_{i}}\sum_{\left\|A^{l}_{i}(x)-P^{l}_{a}\left<x\right>\right\|_{1}>t_{1}}\left\|A^{l}_{i}(x)-P^{l}_{a}\left<x\right>\right\|_{1}},$
(8)

where $\Omega_{i}$ denotes the number of the valid coordinates $x$ in $A_{i}$,
and $t_{1}$ is a predefined threshold. ‘$\left<\cdot\right>$’ denotes the
bilinear sampling operation for getting the element at the corresponding
coordinate of $x$ in a different resolution volume.

The disparity guidance loss term $L_{gui}$ leverages both the gradient
information and the edge region values in the auxiliary monocular disparity
map $d^{l}_{a}$ for improving the quality of the binocular result:

$\displaystyle L_{gui}$
$\displaystyle=\left\|\partial_{x}d^{l}_{a}-\partial_{x}d^{l}_{s}\right\|_{1}+\left\|\partial_{y}d^{l}_{a}-\partial_{y}d^{l}_{s}\right\|_{1}$
$\displaystyle+M^{l}_{out}\odot\left\|d^{l}_{a}-d^{l}_{s}\right\|_{1}\quad,$
(9)

where ‘$\partial_{x}$’, ‘$\partial_{y}$’ are the differential operators in the
horizontal and vertical directions respectively, $M^{l}_{out}$ denotes a
binary mask [41] where the pixels whose reprojected coordinates are out of the
image are ones, and zeros otherwise. Accordingly, the stereo loss is
formulated as:

$L_{S}=L_{rec2}+\lambda_{2}L_{smo2}+\lambda_{3}L_{cos}+\lambda_{4}L_{gui}\quad,$
(10)

where $\\{\lambda_{2},\lambda_{3},\lambda_{4}\\}$ are preset weight
parameters, and $L_{smo2}$ is the edge-aware smoothness loss [20]. At this
step, only the parameters in the dual-path decoder are optimized.

Method | PP. | Sup. | Resolution |  Abs. Rel. $\downarrow$ |  Sq. Rel. $\downarrow$ |  RMSE $\downarrow$ |  logRMSE $\downarrow$ |  A1 $\uparrow$ |  A2 $\uparrow$ |  A3 $\uparrow$
---|---|---|---|---|---|---|---|---|---|---
R-MSFM6 [66] |  | M |  320$\times$1024 | 0.108 | 0.748 | 4.470 | 0.185 | 0.889 | 0.963 | 0.982
PackNet [25] |  | M |  384$\times$1280 | 0.107 | 0.802 | 4.538 | 0.186 | 0.889 | 0.962 | 0.981
SGDepth [35] |  | M(Se.) |  384$\times$1280 | 0.107 | 0.768 | 4.468 | 0.186 | 0.891 | 0.963 | 0.982
SD-SSMDE [46] |  | M |  320$\times$1024 | 0.098 | 0.674 | 4.187 | 0.170 | 0.902 | 0.968 | 0.985
monoResMatch [52] | ✓ | S(SGM) |  384$\times$1280 | 0.111 | 0.867 | 4.714 | 0.199 | 0.864 | 0.954 | 0.979
Monodepth2 [20] | ✓ | S |  320$\times$1024 | 0.105 | 0.822 | 4.692 | 0.199 | 0.876 | 0.954 | 0.977
DepthHints [58] | ✓ | S(SGM) |  320$\times$1024 | 0.096 | 0.710 | 4.393 | 0.185 | 0.890 | 0.962 | 0.981
SingleNet [7] | ✓ | S(S.T.) |  320$\times$1024 | 0.094 | 0.681 | 4.392 | 0.185 | 0.892 | 0.962 | 0.981
FAL-Net [21] | ✓ | S |  384$\times$1280 | 0.093 | 0.564 | 3.973 | 0.174 | 0.898 | 0.967 | 0.985
Edge-of-depth [67] | ✓ | S(SGM, Se.) |  320$\times$1024 | 0.091 | 0.646 | 4.244 | 0.177 | 0.898 | 0.966 | 0.983
PLADE-Net [22] | ✓ | S |  384$\times$1280 | 0.089 | 0.590 | 4.008 | 0.172 | 0.900 | 0.967 | 0.985
EPCDepth [45] | ✓ | S(SGM) |  320$\times$1024 | 0.091 | 0.646 | 4.207 | 0.176 | 0.901 | 0.966 | 0.983
OCFD-Net [64] | ✓ | S |  384$\times$1280 | 0.090 | 0.563 | 4.005 | 0.172 | 0.903 | 0.967 | 0.984
SDFA-Net [65] | ✓ | S |  384$\times$1280 | 0.089 | 0.531 | 3.864 | 0.168 | 0.907 | 0.969 | 0.985
_TiO-Depth_ |  | S |  384$\times$1280 | 0.085 | 0.544 | 3.919 | 0.169 | 0.911 | 0.969 | 0.985
_TiO-Depth_ | ✓ | S |  384$\times$1280 | 0.083 | 0.521 | 3.864 | 0.167 | 0.912 | 0.970 | 0.985
DepthFormer (2F.) [26] |  | M |  192$\times$640 | 0.090 | 0.661 | 4.149 | 0.175 | 0.905 | 0.967 | 0.984
ManyDepth (2F.) [59] |  | M |  320$\times$1024 | 0.087 | 0.685 | 4.142 | 0.167 | 0.920 | 0.968 | 0.983
H-Net (Bino.) [31] |  | S |  192$\times$640 | 0.076 | 0.607 | 4.025 | 0.166 | 0.918 | 0.966 | 0.982
_TiO-Depth (Bino.)_ |  | S |  384$\times$1280 | 0.063 | 0.523 | 3.611 | 0.153 | 0.943 | 0.972 | 0.985
Table 1: Quantitative comparison on the KITTI Eigen test set.
$\downarrow/\uparrow$ denotes that lower / higher is better. The best and the
second best results are in bold and underlined under each metric. The methods
marked with ‘2F.’ predict depths by taking 2 frames from a monocular video as
input, while the methods with ‘Bino.’ predict depths by taking stereo pairs as
input. ‘PP.’ means using the post-processing step. The methods marked with
‘Se.’, ‘SGM’, and ‘S.T.’ are trained with the semantic segmentation label, the
depth generated with SGM [29], and the depth predicted by a binocular teacher
network respectively.

Step (3). TiO-Depth is trained in a distilled manner by utilizing the results
obtained at step (1)&(2) as the teacher for further improving monocular
prediction. A distilled loss $L_{dis}$ is used to constrain the final
monocular probability volume $P^{l}_{m}$ (generated with the final branches in
SDFAs) with the stereo probability volume $P^{l}_{s}$ and the auxiliary
monocular probability volume $P^{l}_{a}$. Considering the relative advantages
of the monocular and stereo results, a hybrid probability volume $P^{l}_{h}$
is generated by fusing them weighted by a half-object-edge map $M^{l}_{hoe}$:

$P^{l}_{h}=(1-M^{l}_{hoe})\odot P^{l}_{s}+M^{l}_{hoe}\odot P^{l}_{a}\quad.$
(11)

$M^{l}_{hoe}$ is a grayscale map for indicating the flat areas and the areas
on one side of the object, where the binocular results are more accurate
experimentally:

$M^{l}_{hoe}=M^{l}_{occ^{\prime}}\odot{\rm min}(\frac{{\rm
maxpool}(\|k*D^{l}_{s}\|_{1})}{t_{2}},1)\quad,$ (12)

where ‘${\rm maxpool}(\cdot)$’ denotes a $3\times 3$ max pooling layer with
stride 1, ‘$*$’ denotes the convolutional operation, $k$ is a $3\times 3$
Laplacian kernel, and $t_{2}$ is a predefined threshold.
$M^{l}_{occ^{\prime}}$ is an opposite occlusion mask obtained by treating the
left-view disparity map as the right-view one during calculating the occlusion
mask. KL divergence is employed to reflect the similarity between the final
monocular probability volume $P^{l}_{m}$ and $P^{l}_{h}$, which is formulated
as:

$L_{dis}={\rm KL}(P^{l}_{h}||P^{l}_{m})\quad.$ (13)

Only the parameters in the SDFA blocks, the decoder block and the output layer
are optimized at this step. Please see the supplemental material for more
details about the training strategy and losses.

##  4 Experiments

In this section, we train TiO-Depth on the KITTI dataset [18], and the
evaluations are conducted on the KITTI, Cityscapes [11], and DDAD [25]
datasets. For monocular depth estimation, the Eigen split [14] of KITTI is
utilized, which consists of a training set with 22600 stereo pairs and a test
set with 697 images. For binocular depth estimation, a training set with 28968
stereo pairs collected from KITTI is used for training as done in [7, 37, 56],
while the training set of the KITTI 2015 stereo benchmark [43] is used for the
evaluation, which consists of 200 image pairs. For exploring the generation
ability of TiO-Depth, Cityscapes and DDAD are used for conducting an
additional evaluation. Please see the supplemental material for more details
about the datasets and metrics.

Input Images EPCDepth [45] SDFA-Net [65] TiO-Depth (Mono.) TiO-Depth (Bino.)

Figure 5: Visualization results of EPCDepth [45], SDFA-Net [65] and our TiO-Depth on KITTI. The input stereo pairs are shown in the first column, where the left-view images are used for monocular depth estimation. The predicted depth maps with the corresponding ‘Abs. Rel.’ error maps calculated on an improved Eigen test set [53] are shown in the following columns. For the error maps, red indicates larger error, and blue indicates smaller error as shown in the color bars. Method | Sup. | Resolution |  Abs. Rel. $\downarrow$ |  Sq. Rel. $\downarrow$ |  RMSE $\downarrow$ |  logRMSE $\downarrow$ |  A1 $\uparrow$ |  A2 $\uparrow$ |  A3 $\uparrow$ |  EPE-all$\downarrow$ |  D1-all$\downarrow$
---|---|---|---|---|---|---|---|---|---|---|---
MonoDepth [19] | S |  256$\times$512 | 0.068 | 0.835 | 4.392 | 0.146 | 0.942 | 0.978 | 0.989 | - | 9.194
UnOS (Stereo-only) [56] | S |  256$\times$832 | 0.060 | 0.833 | 4.187 | 0.135 | 0.955 | 0.981 | 0.990 | - | 7.073
UnOS (Full) [56] | MS |  256$\times$832 | 0.049 | 0.515 | 3.404 | 0.121 | 0.965 | 0.984 | 0.992 | - | 5.943
Liu _et al_. [37] | S |  256$\times$832 | 0.051 | 0.532 | 3.780 | 0.126 | 0.957 | 0.982 | 0.991 | 1.520 | 9.570
Flow2Stereo [38] | MS |  384$\times$1280 | - | - | - | - | - | - | - | 1.340 | 6.130
StereoNet [7] | S |  320$\times$1024 | 0.052 | 0.558 | 3.733 | 0.123 | 0.961 | 0.984 | 0.992 | - | -
StereoNet-D [7] | S* |  320$\times$1024 | 0.048 | 0.482 | 3.393 | 0.105 | 0.969 | 0.989 | 0.994 | - | -
_TiO-Depth_ | S |  384$\times$1280 | 0.050 | 0.434 | 3.239 | 0.104 | 0.967 | 0.987 | 0.994 | 1.282 | 6.647
SingleNet (Mono.) [7] | S(S.T.) |  320$\times$1024 | 0.083 | 0.688 | 4.464 | 0.154 | 0.904 | 0.972 | 0.990 | - | -
_TiO-Depth (Mono.)_ | S |  384$\times$1280 | 0.075 | 0.458 | 3.717 | 0.130 | 0.925 | 0.979 | 0.992 | 2.203 | 17.860
_TiO-Depth (Mono.)+PP._ | S |  384$\times$1280 | 0.073 | 0.439 | 3.680 | 0.128 | 0.925 | 0.980 | 0.993 | 2.158 | 17.570
Table 2: Quantitative comparison on KITTI 2015 training set. The methods
marked with ‘Mono.’ predict depths by taking single image as input, while
other methods predict depths with stereo pairs. ‘S*’ denotes the method is
jointly trained with a separate monocular model.

###  4.1 Implementation details

TiO-Depth is implemented with the PyTorch [44] framework. The tiny size
modified Swin-transformer [39, 65] used as the monocular feature encoder is
pretrained on the ImageNet dataset [48]. We set the minimum and the maximum
disparities to $b_{\min}=2,b_{\max}=300$ for the discrete disparity volume,
and the number of the discrete disparity levels is set to $N=49$. The weight
parameters for the loss function are set to
$\lambda_{1}=0.0008,\lambda_{2}=0.008,\lambda_{3}=0.01$, and
$\lambda_{4}=0.01$, while we set $\alpha=0.15$ $t_{1}=1$, and $t_{2}=0.13$.
The Adam optimizers [34] with $\beta_{1}=0.5$ and $\beta_{2}=0.999$ are used
to train TiO-Depth for 50 epochs. The learning rate is firstly set to
$10^{-4}$, and is downgraded by half at the 20, 30, 40, 45 epochs. At both the
training and testing stages, the images are resized into the resolution of
$384\times 1280$, while we assume that the intrinsics of all the images are
identical. The on-the-fly data augmentations are performed in training,
including random resizing (from 0.67 to 1.5) and cropping (256$\times$832),
random horizontal flipping, and random color augmentation.

###  4.2 Comparative evaluation

For monocular depth estimation, we firstly evaluate TiO-Depth on the KITTI
Eigen test set [14] in comparison to 4 methods trained with monocular video
sequences (M) and 10 methods trained with stereo image pairs (S). The
corresponding results by all the referred methods are cited from their
original papers and reported in Tab. 1.

It can be seen that TiO-Depth with a post-processing as done in [65]
outperforms all the comparative methods in most cases, including the methods
trained with the depth pseudo labels generated by additional algorithms or
networks (SGM, S.T.). Since _the same_ TiO-Depth model could handle the
binocular task by using the binocular path, we give its performance in
binocular depth estimation (‘Bino.’) in comparison with 3 methods. As seen
from Tab. 1, TiO-Depth gets the top performance among all the comparative
multi-frame (2F.) and binocular methods. Several visualization results of TiO-
Depth as well as two comparative methods: EPCDepth [45] and SDFA-Net [65] are
given in Fig. 5. As shown in the figure, the depth maps predicted by TiO-Depth
are more accurate and contain more delicate geometric details, while the
performance of TiO-Depth is further improved by taking the stereo pairs as
input. These results demonstrate that the TiO-Depth could predict accurate
depths by taking both monocular and binocular inputs.

For binocular depth estimation, we evaluate TiO-Depth on the KITTI 2015
training set [43] in comparison to 5 self-supervised binocular depth
estimation methods. It is noted that all of the comparative methods could not
handle the monocular task. As seen from the corresponding results shown in
Tab. 2, TiO-Depth outperforms all the methods trained with stereo pairs (S) or
stereo videos (MS) in most cases, and it achieves comparable performance with
StereoNet-D [7] benefited from an additional monocular depth estimation model,
while the performance of TiO-Depth is boosted by itself. The monocular depth
estimation results of _the same_ TiO-Depth model are also given in Tab. 2,
which show that it effectively handling the monocular task at the same time,
further indicating the effectiveness of TiO-Depth as a two-in-one model.

Furthermore, we train TiO-Depth on KITTI [18] and evaluate it on DDAD [25] and
Cityscapes [11] for testing its cross-dataset generalization ability. The
corresponding results of TiO-Depth and 6 comparative methods are reported in
Tab. 3. As shown in the table, TiO-Depth not only performs best in comparison
to the methods evaluated in a cross-dataset manner, but also achieves a
competitive performance with the methods trained and tested on the same
dataset. When the stereo pairs are available, TiO-Depth could predict more
accurate binocular depths by taking the image pairs. These results demonstrate
the generalization ability of TiO-Depth on the unseen dataset. Please see the
supplemental material for the additional exponential results.

Method | train | test |  Abs. Rel. $\downarrow$ |  Sq. Rel. $\downarrow$ |  RMSE $\downarrow$ |  A1 $\uparrow$
---|---|---|---|---|---|---
PackNet [25] | D | D | 0.173 | 7.164 | 14.363 | 0.835
ManyDepth (2F.) [59] | D | D | 0.146 | 3.258 | 14.098 | 0.822
DepthFormer (2F.) [26] | D | D | 0.135 | 2.953 | 12.477 | 0.836
_TiO-Depth_ | K | D | 0.144 | 2.664 | 14.273 | 0.808
MonoDepth2 [20] | C | C | 0.129 | 1.569 | 6.876 | 0.849
Li _et al_. [36] | C | C | 0.119 | 1.290 | 6.980 | 0.846
ManyDepth (2F.) [59] | C | C | 0.114 | 1.193 | 6.223 | 0.875
SD-SSMDE [46] | C | C | 0.114 | 1.017 | 5.949 | 0.870
MonoDepth2 [20] | K | C | 0.153 | 1.785 | 8.590 | 0.774
SD-SSMDE [46] | K | C | 0.143 | 1.635 | 8.441 | 0.789
_TiO-Depth_ | K | C | 0.120 | 1.176 | 7.157 | 0.850
_TiO-Depth (Bino.)_ | K | C | 0.066 | 0.423 | 4.070 | 0.961
Table 3: Quantitative comparison on DDAD and Cityscapes. ‘C’, ‘K’, and ‘D’ denote the methods are trained or tested on the Cityscapes, KITTI and DDAD datasets respectively. Methods |  Abs. Rel. $\downarrow$ |  Sq. Rel. $\downarrow$ |  A1 $\uparrow$ |  EPE $\downarrow$ |  D1 $\downarrow$
---|---|---|---|---|---
w. Cat module (321) | 0.069 | 0.505 | 0.947 | 2.074 | 15.952
w. Attn module (321) | 0.053 | 0.439 | 0.965 | 1.377 | 7.421
w. MFM (1) | 0.054 | 0.423 | 0.960 | 1.483 | 8.784
w. MFM (21) | 0.052 | 0.445 | 0.965 | 1.305 | 7.077
TIO-Depth | 0.051 | 0.429 | 0.966 | 1.281 | 6.684
w/o. $L_{gui}$ | 0.053 | 0.506 | 0.966 | 1.292 | 6.984
w/o. $L_{gui}$, $L_{cos}$ | 0.053 | 0.522 | 0.965 | 1.326 | 6.755
w/o. $L_{gui}$, $L_{cos}$, $M_{occ}$ | 0.054 | 0.565 | 0.963 | 1.345 | 7.159
Table 4: Binocular depth estimation results on KITTI 2015 training set in the ablation study. The numbers in the name of methods mean the indexes of the used modules as shown in Fig. 2. All the results are evaluated after training 30 epochs. Steps | $L_{dis}$ | FB. |  Abs. Rel. $\downarrow$ |  Sq. Rel. $\downarrow$ |  RMSE $\downarrow$ |  A1 $\uparrow$
---|---|---|---|---|---|---
1 | - | - | 0.088 | 0.556 | 4.093 | 0.904
1+2 | - | - | 0.088 | 0.557 | 4.067 | 0.906
1+2+3 | $P^{l}_{s}$ | ✓ | 0.086 | 0.590 | 4.021 | 0.911
1+2+3 | $P^{l}_{h}$ | ✓ | 0.085 | 0.544 | 3.919 | 0.911
1+2+3 | $P^{l}_{h}$ | - | 0.098 | 0.695 | 4.367 | 0.892
Table 5: Monocular depth estimation results on the KITTI Eigen test set in the
ablation study. ’FB.’ denotes using the final branches.

###  4.3 Ablation studies

This subsection verifies the effectiveness of each key element in TiO-Depth by
conducting ablation studies on the KITTI dataset [18].

Dual-path decoder. We firstly replace the proposed Monocular Feature Matching
(MFM) modules with the concatenation-based modules (Cat module) and the cross-
attention-based modules without the SE layer (Attn modules), respectively. The
corresponding results are shown in the first part of Tab. 4, which show that
TiO-Depth (with MFM (321)) performs best compared to the models with other
modules. Then, the impact of the number of MFMs is shown in the second part of
Tab. 4. It can be seen that the binocular performances are gradually improved
by using more MFMs in most cases. The monocular depth estimation results of
TiO-Depth with/without the ‘final branch (FB.)’ in the SDFA modules are shown
in the last two rows of Tab. 5, where the performance of TiO-Depth with the
final branches is much better than that of the model without these branches.
We notice that the switchable branches are important for TiO-Depth to improve
the monocular results, but the SDFA block is not a necessary choice. Please
see the supplemental material for more experimental results and discussions.
Considering that the three MFMs only contain 1.7M parameters in total, these
results indicate the effectiveness of the dual-path decoder with MFMs in the
two tasks.

Multi-stage joint-training strategy. We firstly analyze the impact of each
term in the stereo loss $L_{S}$ in binocular depth estimation by sequentially
taking out the disparity guidance loss term $L_{gui}$, the cost volume loss
term $L_{cos}$ and the occlusion mask $M_{occ}$ used in $L_{rec2}$. The
corresponding results in the third part of Tab. 4 show that the performances
of the model are dropped by removing the loss terms and the mask. Then we
train TiO-Depth with different numbers of step(s) and pseudo labels to
validate the effectiveness of the training strategy in monocular depth
estimation in Tab. 5. As shown in the table, the monocular performance could
not be improved by just training TiO-Depth for learning the two tasks without
distillation (_i.e_., with ‘1+2’ steps), but it is improved in most cases by
training with three steps. Compared with using the stereo probability volume
$P^{l}_{s}$, the accuracy of the monocular results could be consistently
improved by using the hybrid probability volume $P^{l}_{h}$ in the distilled
loss $L_{dis}$. These results demonstrate that our training strategy is
helpful for TiO-Depth to learn more accurate monocular and binocular depths.

##  5 Conclusion

In this paper, we propose TiO-Depth, a two-in-one depth prediction model for
both the monocular and binocular self-supervised depth estimation tasks, while
a multi-stage joint-training strategy is explored for training. The full TiO-
Depth is used to predict depths from stereo pairs, while the partial TiO-Depth
by closing the duplicate parts could predict depths from single images. The
experimental results in monocular and binocular depth estimations not only
prove the effectiveness of TiO-Depth but also indicate the feasibility of
bridging the gap between the two tasks.

Acknowledgements. This work was supported by the Strategic Priority Research
Program of the Chinese Academy of Sciences (Grant No. XDA27040811), the
National Natural Science Foundation of China (Grant Nos. 61991423, U1805264),
the Beijing Municipal Science and Technology Project (Grant No.
Z211100011021004).

## References

  * [1] Filippo Aleotti, Fabio Tosi, Li Zhang, Matteo Poggi, and Stefano Mattoccia.  Reversing the cycle: self-supervised deep stereo through enhanced monocular distillation.  In European Conference on Computer Vision, pages 614–632. Springer, 2020.
  * [2] Juan Luis Gonzalez Bello and Munchurl Kim.  Forget about the lidar: Self-supervised depth estimators with med probability volumes.  Advances in Neural Information Processing Systems, 33, 2020.
  * [3] Juan Luis Gonzalez Bello and Munchurl Kim.  Self-supervised deep monocular depth estimation with ambiguity boosting.  IEEE TPAMI, 2021.
  * [4] Michael Bleyer, Christoph Rhemann, and Carsten Rother.  Patchmatch stereo-stereo matching with slanted support windows.  In Bmvc, volume 11, pages 1–11, 2011.
  * [5] Jia-Ren Chang and Yong-Sheng Chen.  Pyramid stereo matching network.  In CVPR, pages 5410–5418, 2018.
  * [6] Yuhua Chen, Cordelia Schmid, and Cristian Sminchisescu.  Self-supervised learning with geometric constraints in monocular video: Connecting flow, depth, and camera.  In ICCV, pages 7063–7072, 2019.
  * [7] Zhi Chen, Xiaoqing Ye, Wei Yang, Zhenbo Xu, Xiao Tan, Zhikang Zou, Errui Ding, Xinming Zhang, and Liusheng Huang.  Revealing the reciprocal relations between self-supervised stereo and monocular depth estimation.  In ICCV, pages 15529–15538, 2021.
  * [8] Bin Cheng, Inderjot Singh Saggu, Raunak Shah, Gaurav Bansal, and Dinesh Bharadia.  S3 net: Semantic-aware self-supervised depth estimation with monocular videos and synthetic data.  In ECCV, pages 52–69, 2020.
  * [9] Hyesong Choi, Hunsang Lee, Sunkyung Kim, Sunok Kim, Seungryong Kim, Kwanghoon Sohn, and Dongbo Min.  Adaptive confidence thresholding for monocular depth estimation.  In ICCV, pages 12808–12818, 2021.
  * [10] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter.  Fast and accurate deep network learning by exponential linear units (elus).  arXiv preprint arXiv:1511.07289, 2015.
  * [11] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016.
  * [12] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In Proceedings of the IEEE international conference on computer vision, pages 764–773, 2017.
  * [13] David Eigen and Rob Fergus.  Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture.  In Proceedings of the IEEE international conference on computer vision, pages 2650–2658, 2015.
  * [14] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  Advances in neural information processing systems, 27, 2014.
  * [15] José M Fácil, Alejo Concha, Luis Montesano, and Javier Civera.  Single-view and multi-view depth fusion.  IEEE Robotics and Automation Letters, 2(4):1994–2001, 2017.
  * [16] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao.  Deep ordinal regression network for monocular depth estimation.  In CVPR, pages 2002–2011, 2018.
  * [17] Ravi Garg, Vijay Kumar BG, Gustavo Carneiro, and Ian Reid.  Unsupervised cnn for single view depth estimation: Geometry to the rescue.  In ECCV, pages 740–756, 2016.
  * [18] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In CVPR, pages 3354–3361, 2012.
  * [19] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow.  Unsupervised monocular depth estimation with left-right consistency.  In CVPR, pages 270–279, 2017.
  * [20] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow.  Digging into self-supervised monocular depth estimation.  In ICCV, pages 3828–3838, 2019.
  * [21] Juan Luis GonzalezBello and Munchurl Kim.  Forget about the lidar: Self-supervised depth estimators with med probability volumes.  Advances in Neural Information Processing Systems, 33:12626–12637, 2020.
  * [22] Juan Luis GonzalezBello and Munchurl Kim.  Plade-net: Towards pixel-level accuracy for self-supervised single-view depth estimation with neural positional encoding and distilled matting loss.  In CVPR, pages 6851–6860, 2021.
  * [23] Ariel Gordon, Hanhan Li, Rico Jonschkowski, and Anelia Angelova.  Depth from videos in the wild: Unsupervised monocular depth learning from unknown cameras.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8977–8986, 2019.
  * [24] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao.  Knowledge distillation: A survey.  IJCV, 129(6):1789–1819, 2021.
  * [25] Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos, and Adrien Gaidon.  3d packing for self-supervised monocular depth estimation.  In CVPR, pages 2485–2494, 2020.
  * [26] Vitor Guizilini, Rareș Ambruș, Dian Chen, Sergey Zakharov, and Adrien Gaidon.  Multi-frame self-supervised depth with transformers.  In CVPR, pages 160–170, 2022.
  * [27] Vitor Guizilini, Rui Hou, Jie Li, Rares Ambrus, and Adrien Gaidon.  Semantically-guided representation learning for self-supervised monocular depth.  In International Conference on Learning Representations (ICLR), 2020\.
  * [28] Mu He, Le Hui, Yikai Bian, Jian Ren, Jin Xie, and Jian Yang.  Ra-depth: Resolution adaptive self-supervised monocular depth estimation.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVII, pages 565–581. Springer, 2022.
  * [29] Heiko Hirschmuller.  Accurate and efficient stereo processing by semi-global matching and mutual information.  In CVPR, volume 2, pages 807–814. IEEE, 2005.
  * [30] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.
  * [31] Baoru Huang, Jian-Qing Zheng, Stamatia Giannarou, and Daniel S Elson.  H-net: Unsupervised attention-based stereo depth estimation leveraging epipolar geometry.  In CVPR, pages 4460–4467, 2022.
  * [32] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In ECCV, pages 694–711, 2016.
  * [33] Hyunyoung Jung, Eunhyeok Park, and Sungjoo Yoo.  Fine-grained semantics-aware representation enhancement for self-supervised monocular depth estimation.  In ICCV, pages 12642–12652, 2021.
  * [34] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [35] Marvin Klingner, Jan-Aike Termöhlen, Jonas Mikolajczyk, and Tim Fingscheidt.  Self-supervised monocular depth estimation: Solving the dynamic object problem by semantic guidance.  In ECCV, pages 582–600, 2020.
  * [36] Hanhan Li, Ariel Gordon, Hang Zhao, Vincent Casser, and Anelia Angelova.  Unsupervised monocular depth learning in dynamic scenes.  In Conference on Robot Learning, pages 1908–1917. PMLR, 2021.
  * [37] Liang Liu, Guangyao Zhai, Wenlong Ye, and Yong Liu.  Unsupervised learning of scene flow estimation fusing with local rigidity.  In IJCAI, pages 876–882, 2019.
  * [38] Pengpeng Liu, Irwin King, Michael R Lyu, and Jia Xu.  Flow2stereo: Effective self-supervised learning of optical flow and stereo matching.  In CVPR, pages 6648–6657, 2020.
  * [39] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In ICCV, pages 10012–10022, 2021.
  * [40] Yangqi Long, Huimin Yu, and Biyang Liu.  Two-stream based multi-stage hybrid decoder for self-supervised multi-frame monocular depth.  IEEE Robotics and Automation Letters, 7(4):12291–12298, 2022.
  * [41] Reza Mahjourian, Martin Wicke, and Anelia Angelova.  Unsupervised learning of depth and ego-motion from monocular video using 3d geometric constraints.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5667–5675, 2018.
  * [42] Diogo Martins, Kevin Van Hecke, and Guido De Croon.  Fusion of stereo and still monocular depth estimates in a self-supervised learning context.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 849–856. IEEE, 2018.
  * [43] Moritz Menze, Christian Heipke, and Andreas Geiger.  Joint 3d estimation of vehicles and scene flow.  In ISPRS Workshop on Image Sequence Analysis (ISA), 2015.
  * [44] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32:8026–8037, 2019.
  * [45] Rui Peng, Ronggang Wang, Yawen Lai, Luyang Tang, and Yangang Cai.  Excavating the potential capacity of self-supervised monocular depth estimation.  In ICCV, pages 15560–15569, 2021.
  * [46] Andra Petrovai and Sergiu Nedevschi.  Exploiting pseudo labels in a self-supervised learning framework for improved monocular depth estimation.  In CVPR, pages 1578–1588, 2022.
  * [47] Andrea Pilzer, Stephane Lathuiliere, Nicu Sebe, and Elisa Ricci.  Refine and distill: Exploiting cycle-inconsistency and knowledge distillation for unsupervised monocular depth estimation.  In CVPR, pages 9768–9777, 2019.
  * [48] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, and Michael Bernstein.  Imagenet large scale visual recognition challenge.  International Journal of Computer Vision, 11(3):211–252, 2015.
  * [49] Ashutosh Saxena, Jamie Schulte, Andrew Y Ng, et al.  Depth estimation using monocular and stereo cues.  In IJCAI, volume 7, pages 2197–2203, 2007.
  * [50] Chang Shu, Kun Yu, Zhixiang Duan, and Kuiyuan Yang.  Feature-metric loss for self-supervised learning of depth and egomotion.  In ECCV, pages 572–588, 2020.
  * [51] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [52] Fabio Tosi, Filippo Aleotti, Matteo Poggi, and Stefano Mattoccia.  Learning monocular depth estimation infusing traditional stereo knowledge.  In CVPR, pages 9799–9809, 2019.
  * [53] Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger.  Sparsity invariant cnns.  In 2017 international conference on 3D Vision (3DV), pages 11–20, 2017.
  * [54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [55] Longguang Wang, Yulan Guo, Yingqian Wang, Zhengfa Liang, Zaiping Lin, Jungang Yang, and Wei An.  Parallax attention for unsupervised stereo correspondence learning.  IEEE TPAMI, 2020.
  * [56] Yang Wang, Peng Wang, Zhenheng Yang, Chenxu Luo, Yi Yang, and Wei Xu.  Unos: Unified unsupervised optical-flow and stereo-depth estimation by watching videos.  In CVPR, pages 8071–8081, 2019.
  * [57] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE TIP, 13(4):600–612, 2004.
  * [58] Jamie Watson, Michael Firman, Gabriel J Brostow, and Daniyar Turmukhambetov.  Self-supervised monocular depth kints.  In ICCV, pages 2162–2171, 2019.
  * [59] Jamie Watson, Oisin Mac Aodha, Victor Prisacariu, Gabriel Brostow, and Michael Firman.  The temporal opportunist: Self-supervised multi-frame monocular depth.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1164–1174, 2021.
  * [60] Guorun Yang, Hengshuang Zhao, Jianping Shi, Zhidong Deng, and Jiaya Jia.  Segstereo: Exploiting semantic information for disparity estimation.  In ECCV, pages 636–651, 2018.
  * [61] Zhichao Yin and Jianping Shi.  Geonet: Unsupervised learning of dense depth, optical flow and camera pose.  In CVPR, pages 1983–1992, 2018.
  * [62] Feihu Zhang, Victor Prisacariu, Ruigang Yang, and Philip HS Torr.  Ga-net: Guided aggregation net for end-to-end stereo matching.  In CVPR, pages 185–194, 2019.
  * [63] Chao Zhou, Hong Zhang, Xiaoyong Shen, and Jiaya Jia.  Unsupervised learning of stereo matching.  In ICCV, pages 1567–1575, 2017.
  * [64] Zhengming Zhou and Qiulei Dong.  Learning occlusion-aware coarse-to-fine depth map for self-supervised monocular depth estimation.  In Proceedings of the 30th ACM International Conference on Multimedia, pages 6386––6395, 2022.
  * [65] Zhengming Zhou and Qiulei Dong.  Self-distilled feature aggregation for self-supervised monocular depth estimation.  In ECCV, pages 709–726. Springer, 2022.
  * [66] Zhongkai Zhou, Xinnan Fan, Pengfei Shi, and Yuanxue Xin.  R-msfm: Recurrent multi-scale feature modulation for monocular depth estimating.  In ICCV, pages 12777–12786, 2021.
  * [67] Shengjie Zhu, Garrick Brazil, and Xiaoming Liu.  The edge of depth: Explicit constraints between segmentation and depth.  In CVPR, pages 13116–13125, 2020.
  * [68] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai.  Deformable convnets v2: More deformable, better results.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9308–9316, 2019.

Supplemental Material

##  Appendix A Multi-stage joint-training strategy

###  A.1 Image reconstruction

As mentioned in Sec. 3.4 of the main paper, the discrete depth constraint [2,
22, 3, 65] is used for helping TiO-Depth learn monocular depth estimation at
step (1), which assumes that the depth of each pixel is inversely proportional
to a weighted sum of a set of discrete disparities determined by the visual
consistency between the input training stereo images [64]. A left-view
reconstructed image $\hat{I}^{l}_{a}\in\mathbb{R}^{3\times H\times W}$ is
obtained with the right-view real image $I^{r}\in\mathbb{R}^{3\times H\times
W}$ and the predicted right-view auxiliary volume
$V^{r}_{a}\in\mathbb{R}^{N\times H\times W}$ under the discrete depth
constraint, where $N$ is the number of the discrete disparity levels and
$\\{H,W\\}$ are the height and width of the image. Specifically, a left-view
auxiliary volume $\hat{V}^{l}_{a}\in\mathbb{R}^{N\times H\times W}$ is firstly
generated by shifting the $n^{\rm th}$ channel of $V^{r}_{a}$ with the
corresponding disparity value $b_{n}$ generated with the mirrored exponential
disparity discretization [2]. Then, $\hat{V}^{l}_{a}$ is passed thought a
softmax operation along the first dimension to obtain the corresponding
probability volume $\hat{P}^{l}_{a}$. Accordingly, the left-view reconstructed
image $\hat{I}^{l}_{a}$ is obtained by calculating a weighted sum of the
shifted $N$ versions of the right image $I^{r}$ with $\hat{P}^{l}_{a}$:

$\hat{I}^{l}=\sum_{n=0}^{N-1}{\hat{P}^{l}_{an}\odot I^{r}_{n}}\quad,$ (14)

where $\hat{P}^{l}_{an}\in\mathbb{R}^{1\times H\times W}$ is the $n^{\rm th}$
channel of $\hat{P}^{l}_{a}$, ‘$\odot$’ denotes the element-wise
multiplication, and $I^{r}_{n}$ is the left-view image shifted with $b_{n}$.

The continuous depth constraint [19, 20, 7] is used for helping TiO-Depth
learn binocular depth estimation at step (2), which assumes that the depth of
each pixel is a continuous variable determined by the visual consistency
between the input training stereo images [64]. A left-view image
$\tilde{I}^{l}_{s}$ is obtained with the right-view real image $I^{r}$ and the
predicted left-view depth map $D^{l}_{s}\in\mathbb{R}^{1\times H\times W}$
under the continuous depth constraint. Specifically, for an arbitrary pixel
coordinate $p\in\mathbb{R}^{2}$ in the left-view image, its corresponding
coordinate $p^{\prime}$ in the right image could be calculated with
$D^{l}_{s}$:

$p^{\prime}=p-\left[\frac{Bf_{x}}{D^{l}_{s}(p)},0\right]^{\top}\quad,$ (15)

where $B$ is the baseline length of the stereo pair and $f_{x}$ is the
horizontal focal length of the camera. Accordingly, the reconstructed left-
view image $\tilde{I}^{l}_{s}$ is obtained by assigning the RGB value of the
right image pixel $p^{\prime}$ to the pixel $p$ of $\tilde{I}^{l}_{s}$.

###  A.2 Monocular loss

The monocular loss used in step (1) contains a monocular reconstruction loss
$L_{rec1}$ and an edge-aware smoothness loss $L_{smo1}$. Specifically,
$L_{rec1}$ consists a $L_{1}$ loss term and a perceptual loss [32] term for
measuring the similarity between the left-view reconstructed image
$\hat{I}^{l}_{a}$ and the left-view real image $I^{l}$ as done in [2, 65]:

$L_{rec1}=\left\|\hat{I}^{l}_{a}-I^{l}\right\|_{1}+\beta\sum_{i=1,2,3}\left\|\phi_{i}(\hat{I}^{l}_{a})-\phi_{i}(I^{l})\right\|_{2}\quad,$
(16)

where ‘$\|\cdot\|_{1}$’ and ‘$\|\cdot\|_{2}$’ denote the $L_{1}$ and $L_{2}$
norms, $\phi_{i}(\cdot)$ represents the output of $i^{\rm th}$ pooling layer
of a pretrained VGG19 [51], and $\beta=0.01$ is a balance parameter. The edge-
aware smoothness loss $L_{smo1}$ is employed for constraining the continuity
of the auxiliary disparity map $d^{r}_{a}$ as done in [19, 65, 2, 7]:

$L_{smo1}=\left\|\partial_{x}d^{r}_{a}\right\|_{1}e^{-\gamma\left\|\partial_{x}I^{r}\right\|_{1}}+\left\|\partial_{y}d^{r}_{a}\right\|_{1}e^{-\gamma\left\|\partial_{y}I^{r}\right\|_{1}}\
\quad,$ (17)

where ‘$\partial_{x}$’, ‘$\partial_{y}$’ are the differential operators in the
horizontal and vertical directions respectively, and $\gamma=2$ is a parameter
for adjusting the degree of edge preservation.

Method | PP. | Sup. | Resolution |  Abs Rel $\downarrow$ |  Sq Rel $\downarrow$ |  RMSE $\downarrow$ |  logRMSE $\downarrow$ |  A1 $\uparrow$ |  A2 $\uparrow$ |  A3 $\uparrow$
---|---|---|---|---|---|---|---|---|---|---
DepthHints [58] | ✓ | S(SGM) |  320$\times$1024 | 0.074 | 0.364 | 3.202 | 0.114 | 0.936 | 0.989 | 0.997
FAL-Net [21] | ✓ | S |  384$\times$1280 | 0.071 | 0.281 | 2.912 | 0.108 | 0.943 | 0.991 | 0.998
PLADE-Net [22] | ✓ | S |  384$\times$1280 | 0.066 | 0.272 | 2.918 | 0.104 | 0.945 | 0.992 | 0.998
OCFD-Net [64] | ✓ | S |  384$\times$1280 | 0.069 | 0.262 | 2.785 | 0.103 | 0.951 | 0.993 | 0.998
SDFA-Net [65] | ✓ | S |  384$\times$1280 | 0.074 | 0.228 | 2.547 | 0.101 | 0.956 | 0.995 | 0.999
_TiO-Depth_ |  | S |  384$\times$1280 | 0.066 | 0.229 | 2.597 | 0.096 | 0.961 | 0.995 | 0.999
_TiO-Depth_ | ✓ | S |  384$\times$1280 | 0.065 | 0.218 | 2.558 | 0.094 | 0.962 | 0.995 | 0.999
DepthFormer (2F.) [26] |  | M |  320$\times$1024 | 0.055 | 0.265 | 2.723 | 0.092 | 0.959 | 0.992 | 0.998
ManyDepth (2F.) [59] |  | M |  352$\times$1216 | 0.055 | 0.305 | 2.945 | 0.094 | 0.963 | 0.992 | 0.997
_TiO-Depth (Bino.)_ |  | S |  384$\times$1280 | 0.033 | 0.078 | 1.583 | 0.050 | 0.996 | 0.999 | 1.000
Table 6: Quantitative comparison on the improved KITTI Eigen test set. $\downarrow/\uparrow$ denotes that lower / higher is better. The best and the second best results are in bold and underlined under each metric. The methods marked with ‘2F.’ predict depths by taking 2 frames from a monocular video as input, while the methods with ‘Bino.’ predict depths by taking stereo pairs as input. ‘PP.’ means using the post-processing step. The methods marked with ‘SGM’ are trained with the the depth generated with SGM [29].  Method | train | test |  Abs. Rel. $\downarrow$ |  Sq. Rel. $\downarrow$ |  RMSE $\downarrow$ |  logRMSE $\downarrow$ |  A1 $\uparrow$ |  A2 $\uparrow$ |  A3 $\uparrow$
---|---|---|---|---|---|---|---|---|---
PackNet [25] | D | D | 0.173 | 7.164 | 14.363 | 0.249 | 0.835 | - | -
ManyDepth (2F.) [59] | D | D | 0.146 | 3.258 | 14.098 | - | 0.822 | - | -
DepthFormer (2F.) [26] | D | D | 0.135 | 2.953 | 12.477 | - | 0.836 | - | -
_TiO-Depth_ | K | D | 0.144 | 2.664 | 14.273 | 0.242 | 0.808 | 0.933 | 0.970
MonoDepth2 [20] | C | C | 0.129 | 1.569 | 6.876 | 0.187 | 0.849 | 0.957 | 0.983
Li _et al_. [36] | C | C | 0.119 | 1.290 | 6.980 | 0.190 | 0.846 | 0.952 | 0.982
ManyDepth (2F.) [59] | C | C | 0.114 | 1.193 | 6.223 | 0.170 | 0.875 | 0.967 | 0.989
SD-SSMDE [46] | C | C | 0.114 | 1.017 | 5.949 | 0.169 | 0.870 | 0.967 | 0.990
MonoDepth2 [20] | K | C | 0.153 | 1.785 | 8.590 | 0.234 | 0.774 | 0.926 | 0.976
SD-SSMDE [46] | K | C | 0.143 | 1.635 | 8.441 | 0.221 | 0.789 | 0.931 | 0.980
_TiO-Depth_ | K | C | 0.120 | 1.176 | 7.157 | 0.187 | 0.850 | 0.958 | 0.987
_TiO-Depth (Bino.)_ | K | C | 0.066 | 0.423 | 4.070 | 0.106 | 0.961 | 0.992 | 0.997
Table 7: Quantitative comparison on DDAD [25] and Cityscapes [11] (Tab. 3 in
the main paper). ‘C’, ‘K’, and ‘D’ denote the methods are trained or tested on
the Cityscapes, KITTI and DDAD datasets respectively.

###  A.3 Details of the training

Since the predicted depth results are not reliable at the early training
epochs, which lack the ability to effectively guide the following steps, the
second and third steps are enabled after $E_{1}=20$ and $E_{2}=30$ training
epochs respectively. Thus, the multi-stage joint-training strategy contains
three stages, where the training iterations are divided into one, two and
three steps respectively as mentioned in Sec. 3.4 of the main paper.
Considering that the second and the third steps are enabled after $E_{1}$ and
$E_{2}$ epochs respectively and different parameters are optimized at these
steps, we use three Adam optimizers [34] at the three steps for training. The
learning rate of each optimizer is set to $10^{-4}$ when the corresponding
training step is firstly enabled, and which is downgraded by half as described
in Sec. 4.1 of the main paper. Since there are several parameters are trained
only at one step (_e.g_., the parameters in the monocular feature matching
modules), while other parameters are trained at multiple steps (_e.g_., the
parameters in the decoder block), we multiply the learning rates of the
parameters that have optimized at the previous steps by 0.1.

Input Images EPCDepth [45] SDFA-Net [65] TiO-Depth (Mono.) TiO-Depth (Bino.)

Figure 6: Visualization results of EPCDepth [45], SDFA-Net [65] and our TiO-
Depth on KITTI. The input stereo pairs are shown in the first column, where
the left-view images are used for monocular depth estimation. The predicted
depth maps with the corresponding ‘Abs. Rel.’ error maps calculated on the
improved Eigen test set are shown in the following columns. For the error
maps, red indicates larger error, and blue indicates smaller error as shown in
the color bars.

##  Appendix B Dataset and metric

TiO-Depth is trained on the KITTI dataset [18] and evaluated on the KITTI,
Cityscapes [11], and DDAD [25] datasets as mentioned in Sec. 4 of the main
paper.

In addition to the Eigen split [14] and the KITTI 2015 stereo benchmark [43]
which are employed for training and testing, an improved Eigen test set [53]
comprised of 652 images with high-quality depth labels is also used for
evaluation. The test set of Cityscapes [11] which contains 1525 stereo pairs
with the disparity maps provided by SGM [29] and the validation set of DDAD
which contains 3950 single images and the aligned LiDAR depth labels are used
for evaluating the cross-dataset generalization ability of TiO-Depth,

The following seven metrics are used to evaluate the performances of monocular
and binocular depth estimations on all the datasets:

  * •

Abs Rel:
$\frac{1}{N}\sum_{i}{\frac{\left|\hat{D}_{i}-D^{gt}_{i}\right|}{D^{gt}_{i}}}$

  * •

Sq Rel:
$\frac{1}{N}\sum_{i}{\frac{\left|\hat{D}_{i}-D^{gt}_{i}\right|^{2}}{D^{gt}_{i}}}$

  * •

RMSE: $\sqrt{\frac{1}{N}\sum_{i}{\left|\hat{D}_{i}-D^{gt}_{i}\right|^{2}}}$

  * •

logRMSE:
$\sqrt{\frac{1}{N}\sum_{i}{\left|\log\left(\hat{D}_{i}\right)-\log\left(D^{gt}_{i}\right)\right|^{2}}}$

  * •

Threshold (A$j$): $\%\quad
s.t.\quad\max{\left(\frac{\hat{D}_{i}}{D^{gt}_{i}},\frac{D^{gt}_{i}}{\hat{D}_{i}}\right)}<a^{j}$

where $\\{\hat{D}_{i},D^{gt}_{i}\\}$ are the predicted depth and the ground-
truth depth at pixel $i$, and $N$ denotes the total number of the pixels with
the ground truth. In practice, we use $a^{j}=1.25,1.25^{2},1.25^{3}$, which
are denoted as A1, A2, and A3 in all the tables. EPE and D1 metrics are also
adopted for the evaluation of binocular depth estimation as done in [37, 38]:

  * •

EPE: $\frac{1}{N}\sum_{i}{\left|\hat{d}_{i}-d^{gt}_{i}\right|}$

  * •

D1: $\%\quad
s.t.\left(\left|\hat{d}_{i}-d^{gt}_{i}\right|>3\right)\vee\left(\frac{\left|\hat{d}_{i}-d^{gt}_{i}\right|}{d^{gt}_{i}}>0.05\right)$

where $\\{\hat{d}_{i},d^{gt}_{i}\\}$ are the predicted disparity and the
ground-truth disparity at pixel $i$.

Methods |  Abs. Rel. $\downarrow$ |  Sq. Rel. $\downarrow$ |  RMSE $\downarrow$ |  logRMSE $\downarrow$ |  A1 $\uparrow$ |  A2 $\uparrow$ |  A3 $\uparrow$ |  EPE $\downarrow$ |  D1 $\downarrow$
---|---|---|---|---|---|---|---|---|---
w. Cat module (321) | 0.069 | 0.505 | 3.442 | 0.123 | 0.947 | 0.983 | 0.992 | 2.074 | 15.952
w. Attn module (321) | 0.053 | 0.439 | 3.214 | 0.106 | 0.965 | 0.987 | 0.994 | 1.377 | 7.421
w. MFM (1) | 0.054 | 0.423 | 3.211 | 0.109 | 0.960 | 0.986 | 0.993 | 1.483 | 8.784
w. MFM (21) | 0.052 | 0.445 | 3.268 | 0.107 | 0.965 | 0.987 | 0.994 | 1.305 | 7.077
TIO-Depth | 0.051 | 0.429 | 3.137 | 0.105 | 0.966 | 0.988 | 0.994 | 1.281 | 6.684
w/o. $L_{gui}$ | 0.053 | 0.506 | 3.378 | 0.108 | 0.966 | 0.987 | 0.993 | 1.292 | 6.984
w/o. $L_{gui}$, $L_{cos}$ | 0.053 | 0.522 | 3.404 | 0.110 | 0.965 | 0.986 | 0.993 | 1.326 | 6.775
w/o. $L_{gui}$, $L_{cos}$, $M_{occ}$ | 0.054 | 0.565 | 3.637 | 0.121 | 0.963 | 0.984 | 0.992 | 1.345 | 7.159
Table 8: Binocular depth estimation results on KITTI 2015 training set in the ablation study (Tab. 4 in the main paper). The numbers in the name of methods mean the indexes of the used modules as shown in Fig. 2 of the main paper. All the results are evaluated after training 30 epochs. Steps | $L_{dis}$ | FB. |  Abs. Rel. $\downarrow$ |  Sq. Rel. $\downarrow$ |  RMSE $\downarrow$ |  logRMSE $\downarrow$ |  A1 $\uparrow$ |  A2 $\uparrow$ |  A3 $\uparrow$
---|---|---|---|---|---|---|---|---|---
1 | - | - | 0.088 | 0.556 | 4.093 | 0.173 | 0.904 | 0.967 | 0.984
1+2 | - | - | 0.088 | 0.557 | 4.067 | 0.172 | 0.906 | 0.968 | 0.984
1+2+3 | $P^{l}_{s}$ | ✓ | 0.086 | 0.590 | 4.021 | 0.169 | 0.911 | 0.969 | 0.985
1+2+3 | $P^{l}_{h}$ | ✓ | 0.085 | 0.544 | 3.919 | 0.169 | 0.911 | 0.969 | 0.985
1+2+3 | $P^{l}_{h}$ | - | 0.098 | 0.695 | 4.367 | 0.183 | 0.892 | 0.964 | 0.983
Table 9: Monocular depth estimation results predicted by TiO-Depth on the
KITTI Eigen test set in the ablation study (Tab. 5 in the main paper). ’FB.’
denotes using the final branches.

For the evaluation on the raw and improved KITTI Eigen test sets [14, 53], we
use the center crop proposed in [17] and the standard cap of 80m. For the
evaluation on the KITTI 2015 training set, all the ground truth disparities
are used for calculating D1 and EPE metrics, while other metrics are
calculated with the cap of 80m as done in [7]. For the evaluation on the DDAD
dataset [25], the cap of 200m is used, while the input images are resized into
the resolution of $384\times 640$ as done in [25]. For the evaluation on the
Cityscapes dataset [11], we use the center crop and the standard cap of 80m as
done in [59, 23, 36], while the input images are cropped and resized into the
resolution of $192\times 512$ as done in [59]. All the cross-dataset results
of TiO-Depth are calculated after the median scaling [13].

(a)

(b)

Figure 7: (a) Architecture of the Self-Distilled Feature Aggregation (SDFA) block cited from [65]. (b) Architecture of the switchable feature aggregation block inspired by the deformable convolution [12, 68]. Steps | $L_{dis}$ | FB. |  Abs. Rel. $\downarrow$ |  Sq. Rel. $\downarrow$ |  RMSE $\downarrow$ |  logRMSE $\downarrow$ |  A1 $\uparrow$ |  A2 $\uparrow$ |  A3 $\uparrow$
---|---|---|---|---|---|---|---|---|---
1 | - | - | 0.094 | 0.579 | 4.155 | 0.178 | 0.896 | 0.966 | 0.984
1+2 | - | - | 0.094 | 0.582 | 4.165 | 0.177 | 0.896 | 0.966 | 0.984
1+2+3 | $P^{l}_{h}$ | ✓ | 0.086 | 0.551 | 3.967 | 0.170 | 0.907 | 0.969 | 0.985
1+2+3 | $P^{l}_{h}$ | - | 0.103 | 0.688 | 4.367 | 0.181 | 0.890 | 0.966 | 0.984
Table 10: Monocular depth estimation results predicted by the variant of TiO-Depth on the KITTI Eigen test set in the ablation study. Method | Resolution |  Abs Rel$\downarrow$ |  Sq Rel$\downarrow$ |  RMSE$\downarrow$ |  logRMSE$\downarrow$ |  A1$\uparrow$ |  A2$\uparrow$ |  A3$\uparrow$
---|---|---|---|---|---|---|---|---
TiO-Depth |  192$\times$640 | 0.091 | 0.625 | 4.179 | 0.174 | 0.902 | 0.968 | 0.984
TiO-Depth |  320$\times$1024 | 0.087 | 0.566 | 3.970 | 0.170 | 0.910 | 0.969 | 0.985
TiO-Depth |  384$\times$1280 | 0.085 | 0.544 | 3.919 | 0.169 | 0.911 | 0.969 | 0.985
TiO-Depth (Bino.) |  192$\times$640 | 0.065 | 0.572 | 3.767 | 0.157 | 0.940 | 0.971 | 0.984
TiO-Depth (Bino.) |  320$\times$1024 | 0.064 | 0.526 | 3.594 | 0.153 | 0.943 | 0.973 | 0.985
TiO-Depth (Bino.) |  384$\times$1280 | 0.063 | 0.523 | 3.611 | 0.153 | 0.943 | 0.972 | 0.985
Table 11: Depth estimation results with different input image resolutions on
the KITTI Eigen test set in the ablation study.

##  Appendix C Comparative Results

As done in [58, 21, 22, 64, 65], we evaluate TiO-Depth on the improved KITTI
Eigen test set [53] and the corresponding results are shown in Tab. 6. It can
be seen that TiO-Depth outperforms all the comparative methods in most cases
in both monocular and binocular (multi-frame) tasks. Additional visualization
results are given in Fig. 5. These results further demonstrate the
effectiveness of TiO-Depth as a two-in-one model.

In Tab. 3, the monocular and binocular depth estimation results of TiO-Depth
and 6 comparison methods [36, 20, 25, 26, 46, 59] on the DDAD [25] and
Cityscapes [11] datasets under all the seven metrics are given, which
demonstrate the generalization ability of TiO-Depth on the unseen datasets.

##  Appendix D Ablation Study

We have verified the effectiveness of each key element in TiO-Depth by
conducting ablation studies on the KITTI dataset [18] in Sec. 4.3 of the main
paper. Tab. 8 shows the binocular depth estimation results in the ablation
study under all of the nine metrics, which demonstrate the effectiveness of
the dual-path decoder and the stereo loss $L_{S}$ on the binocular task.

The monocular depth estimation results in the ablation study under all of the
seven metrics are shown in Tab. 9, which indicate the effectiveness of the
multi-stage joint-training strategy. Furthermore, the results also prove the
significance of the final branches in the Self-Distilled Feature Aggregation
(SDFA) [65] blocks (as shown in Fig. 7(a) where the raw data path in blue is
used as the auxiliary branch and the distilled branch in red is used as the
final branch) for the monocular task.

To further explore the effect of such switchable branches on learning more
accurate monocular depths, a variant of TiO-Depth is built by replacing the
three SDFA blocks in the dual-path decoder by the switchable aggregation
blocks shown in Fig. 7(b). The switchable aggregation block is inspired by the
deformable convolution [12, 68] and is built based on the basic decoder block
described in Sec. 3.2 of the main paper. In comparison to the basic decoder
block, it employs two additional $3\times 3$ convolutional layers as the
switchable ‘final branches’ to learn the spatial offsets for the kernels of
the convolutional layers in the basic decoder block. Accordingly, the standard
convolutional layers in the basic block are converted to the deformable
convolutions when the final branches are used. We train this variant with the
multi-stage joint-training strategy and conduct the ablation studies. The
corresponding results are shown in Tab. 10. It can be seen that the whole
performances of the variant TiO-Depth are poorer than that of TiO-Depth shown
in Tab. 5, mainly because the SDFA blocks could aggregate the features more
effectively than the basic decoder layers. However, using the switchable final
branches significantly improves the performance of the model in comparison to
that without the final branches. These results further demonstrate that the
potential of TiO-Depth for employing a more general architecture.

Finally, we conduct the ablation study on the input image resolution. As seen
from Tab. 11, TiO-Depth still performs well under the two low resolutions.