# Formatting Instructions for CoLLAs 2024
Conference Submissions

Amir El-Ghoussani
Friedrich-Alexander-Universität Erlangen-Nürnberg
<EMAIL_ADDRESS>&Julia Hornauer
Ulm University
<EMAIL_ADDRESS>
Gustavo Carneiro
University of Surrey
<EMAIL_ADDRESS>&Vasileios Belagiannis
Friedrich-Alexander-Universität Erlangen-Nürnberg
<EMAIL_ADDRESS>
# Consistency Regularisation for Unsupervised Domain Adaptation in Monocular
Depth Estimation

Amir El-Ghoussani
Friedrich-Alexander-Universität Erlangen-Nürnberg
<EMAIL_ADDRESS>&Julia Hornauer
Ulm University
<EMAIL_ADDRESS>
Gustavo Carneiro
University of Surrey
<EMAIL_ADDRESS>&Vasileios Belagiannis
Friedrich-Alexander-Universität Erlangen-Nürnberg
<EMAIL_ADDRESS>
###### Abstract

In monocular depth estimation, unsupervised domain adaptation has recently
been explored to relax the dependence on large annotated image-based depth
datasets. However, this comes at the cost of training multiple models or
requiring complex training protocols. We formulate unsupervised domain
adaptation for monocular depth estimation as a consistency-based semi-
supervised learning problem by assuming access only to the source domain
ground truth labels. To this end, we introduce a pairwise loss function that
regularises predictions on the source domain while enforcing perturbation
consistency across multiple augmented views of the unlabelled target samples.
Importantly, our approach is simple and effective, requiring only training of
a single model in contrast to the prior work. In our experiments, we rely on
the standard depth estimation benchmarks KITTI and NYUv2 to demonstrate state-
of-the-art results compared to related approaches. Furthermore, we analyse the
simplicity and effectiveness of our approach in a series of ablation studies.
The code is available at https://github.com/AmirMaEl/SemiSupMDE.

##  1 Introduction

Monocular depth estimation plays an active role in computer vision with many
applications such as 3D pose estimation, semantic segmentation, and 3D
reconstruction. Obtaining accurate depth predictions from a single image,
though, is an ill-posed problem, especially in complex indoor and outdoor
scenes, due to the ambiguous nature of the inverse depth problem. While the
standard supervised learning paradigm, using large annotated depth datasets,
have achieved promising performance (Eigen et al., 2014; Laina et al., 2016;
Agarwal & Arora, 2023), collecting such data requires extensive resources
(Saxena et al., 2009; Gaidon et al., 2016; Handa et al., 2015).

Input KITTI images.Predictions after pretrainingin $D_{S}$.Predictions after
proposedconsistency-based training.

Figure 1:  Following pretraining on the source domain using CutMix (Yun et
al., 2019) data augmentation, we observe that the resulting depth predictions
show some fidelity but could benefit from further refinement in certain areas.
Specifically, they appear overly fragmented or ”edgy” in localised regions
(highlighted in the figure). Based on this observation we carefully design our
consistency-based approach for domain adaptation in monocular depth
estimation, to particularly smooth localised and fragmented regions.

To relax the data annotation requirement, unsupervised learning approaches
exploit geometric constraints between multiple views of a scene. For example,
stereo-based methods leverage left-right consistency between rectified stereo
image pairs to train without explicit depth supervision (Godard et al., 2017;
Park & Park, 2022). Such methods require calibrated stereo cameras during both
training and inference. Another line of work utilises temporal consistency in
video sequences to derive implicit depth cues from camera motion (Zhou et al.,
2017; Mahjourian et al., 2018; Zou et al., 2018). While these approaches do
not require labelled training data, they are limited to scenarios where stereo
imagery or video sequences are available. Furthermore, unsupervised learning
methods struggle in situations involving occlusion and object motion (Wang et
al., 2018; Sun et al., 2023). In contrast, semi-supervised learning methods
utilise a subset of labelled images to directly supervise the model on the
depth prediction task (Kuznietsov et al., 2017; Amiri et al., 2019). This
guidance is stronger than the indirect cues from multi-view constraints alone
in unsupervised learning. More importantly, semi-supervised approaches can
leverage unlabelled data along with a limited amount of labelled data, even if
stereo or video data is scarce or unavailable in the target domain. However,
semi-supervised monocular depth estimation techniques often rely on labelled
and unlabelled samples being drawn from the same underlying distribution
(Kuznietsov et al., 2017). In practical domain adaptation scenarios, this
assumption may not always hold since the labelled source and unlabelled target
domains exhibit distribution shifts (Akada et al., 2022; Lopez-Rodriguez &
Mikolajczyk, 2020).

To address this issue, recent approaches in monocular depth estimation have
explored unsupervised domain adaptation which leverages available labelled
source data together with unlabelled target domain images (Yen et al., 2022;
Kundu et al., 2018; Zheng et al., 2018; Zhao et al., 2019; Lopez-Rodriguez &
Mikolajczyk, 2020).

In particular, unsupervised domain adaptation techniques leverage labelled
source data to guide prediction in unlabelled target domains by aligning
representations across domains (Qiu & Yuille, 2016; Shrivastava et al., 2017).
These are the key capabilities that we also aim to explore in this work.
Existing domain adaptation approaches for monocular depth estimation mainly
rely on self-supervised or unsupervised techniques using consistency-based
(Zheng et al., 2018; Xie et al., 2016; Zhou et al., 2017) or geometry-based
(Yen et al., 2022; Yin et al., 2018) losses. While consistency methods provide
a general constraint using simple transformations, geometry models require
additional metadata, e.g., camera intrinsics, that may be unavailable for new
unlabelled target domains. Similarly, our approach also falls into the first
category of consistency-based approaches. In contrast to prior consistency-
based works that focused only on target-domain consistency regularisation, we
incorporate supervised information by matching predictions on the target and
source images simultaneously. Additionally, we employ consistency
regularisation for the purpose of unsupervised domain adaptation by requiring
target predictions to remain coherent when propagated through multiple
augmentation streams. To our knowledge this represents the first application
of a technique using both multi-stream target consistency and source-based
regularisation for domain adaptive monocular depth estimation.

Furthermore, the existing domain adaptation approaches often involve complex
multi-stage optimising various geometric and photometric losses modelling
camera motion or stereo correspondences (Zou et al., 2018; Mahjourian et al.,
2018), leading to optimisation challenges. For example, external models such
as semantic segmentation models are used to enforce semantic consistency
(Lopez-Rodriguez & Mikolajczyk, 2020) or 3D point cloud models (Yen et al.,
2022) are employed for pseudo label generation, ultimately increasing
dependencies. In contrast, we present an approach to train only the depth
estimation model and require no additional trainable models, keeping the
training procedure simple. Additionally, target and source data are typically
optimised separately for the above methods without an explicit linking of
predictions across domains. In comparison, we propose to simultaneously
optimise the supervised source objective and the unsupervised target objective
in a single training stage.

In this work, we formulate the problem of unsupervised domain adaptation for
monocular depth estimation as a consistency-based semi-supervised learning
problem. We present a semi-supervised approach for training a single model by
assuming access only to the labels of the source domain data. Initially, we
conduct supervised pre-training using heavily augmented labelled source data
to obtain preliminary depth predictions. Afterwards, depth maps predicted on
unlabelled target domain data are used as pseudo-labels. Our approach is
motivated by the fact that after the initial pre-training in the source
domain, the depth predictions appear somewhat faithful but are still edgy and
fragmented in certain areas (see Fig. 1). This suggests that additional
refinement of object boundaries and detection of previously missing objects
could improve the overall depth map quality. To that end, we introduce a
pairwise loss term that regularises the predictions on the source domain to be
consistent with the ground truth depth distribution in the source domain,
while simultaneously enforcing multiple perturbation consistency during
pseudo-labelling on the target domain. We assess the performance of our method
in two standard domain adaptation benchmarks with indoor and outdoor datasets,
where Virtual KITTI (Gaidon et al., 2016) / SceneNet (Handa et al., 2015) are
the source domains and KITTI (Geiger et al., 2013) / NYU (Nathan Silberman &
Fergus, 2012) are the corresponding target domains.

Our main contributions can be summarised as follows:

  * •

We present a semi-supervised approach for unsupervised domain adaptation by
introducing a pairwise loss that regularises the predictions on the source
domain, while simultaneously enforcing perturbation consistency across
multiple augmented views of unlabelled target data.

  * •

Importantly, we do not require any additional trainable models, making the
proposed method simple and effective. To the best of our knowledge, we are the
first to develop domain adaptation by simultaneous optimisation using source
and target data in one training stage.

  * •

In the extensive evaluation based on the standard depth estimation benchmarks
KITTI (Geiger et al., 2013) and NYU (Nathan Silberman & Fergus, 2012), our
proposed method demonstrates state-of-the-art performance in the domain
adaptation task.

##  2 Related Work

###  2.1 Monocular Depth Estimation

Monocular depth estimation consists of predicting the depth information of a
scene from a single image. With the rapid development of deep neural networks,
monocular depth estimation with the help of deep learning has been widely
studied. Various supervised learning approaches have been proposed in recent
years for monocular depth estimation. Eigen et al. (Eigen et al., 2014)
propose a CNN-based approach to directly regress the depth. Liu et al. (Liu et
al., 2016) propose utilising a fully-connected CRF as a post-processing step
to refine monocular depth predictions. Other methods have extended the CNN-
based network by changing the regression loss to a classification loss (Cao et
al., 2016; Fu et al., 2018; Peng Wang et al., 2015; Yin et al., 2019; Bhat et
al., 2021) or change the architecture of the depth estimation network entirely
(Liu et al., 2023; Rudolph et al., 2022). In addition, a number of self-
supervised techniques have been proposed to train monocular depth estimation
models without relying on ground truth labels. GeoNet (Yin et al., 2018)
enforces epipolar geometry consistency, and DF-Net (Zou et al., 2018) models
rigid scene flow. SfMLearner (Zhou et al., 2017) explores photometric
alignment. Godard et al. (2017) use stereo consistency between synchronised
camera views for unsupervised monocular depth estimation. More recently,
attention-based architectures have achieved new performance milestones in
monocular depth estimation. DepthFormer adopted Vision Transformers for strong
multi-scale representations (Lin et al., 2020). Similarly, DPT fused CNNs and
self-attention in a hybrid network (Ranftl et al., 2021). Neural Window Fully
Connected CRFs (Yuan et al., 2022) introduced CRF modelling in a neural
window, leveraging pixel-level affinity and semantic connections to refine
initial depth maps. Agarwal & Arora (2023) propose a novel architecture that
utilises skip attention to refine pixel-level queries and improve depth
predictions. While supervised learning approaches have achieved considerable
success in monocular depth estimation tasks, acquiring large annotated depth
datasets at scale remains challenging. Self-supervised methods provide an
alternative, but traditional techniques depend on camera parameters and frame
geometries, which often do not hold strictly in real-world settings.
Furthermore, the lack of direct supervision signals hinders the ability of
self-supervised models to fully leverage available visual cues.

###  2.2 Domain adaptation for depth estimation

Domain adaptation techniques leverage synthetic data as a labelled source
domain and unlabelled real data as the target domain for monocular depth
estimation (Chen et al., 2019; Kundu et al., 2018; Lopez-Rodriguez &
Mikolajczyk, 2020; PNVR et al., 2020; Zhao et al., 2019; Zheng et al., 2018;
Yen et al., 2022). Most prior approaches treat depth estimation as a
regression task while relying on style- or image translation for pixel-level
adaptation (Atapour Abarghouei & Breckon, 2018), adversarial learning for
feature-level adaptation (Kundu et al., 2018; Hornauer et al., 2021), or a
combination of both (Zheng et al., 2018; Zhao et al., 2019). For instance,
AdaDepth (Kundu et al., 2018) aligns source and target distributions in
features and predictions. $T^{2}Net$ (Zheng et al., 2018) extends this idea by
adopting both synthetic-to-real translation network and feature alignment in
the task network. However, aligning features were found to be ineffective in
outdoor datasets despite promising improvements observed when training on real
stylised images (Zheng et al., 2018). 3D-PL (Yen et al., 2022) combines domain
adaptation, pseudo-labelling and 3D-awareness to improve the depth estimation
performance in a target domain. These methods leverage the geometric
information present in depth maps to generate more accurate pseudo-labels.
Other follow-up methods have explored bidirectional translation (real-to-
synthetic and synthetic-to-real) and utilise depth consistency loss on the
predictions between real and real-to-synthetic images (Zhao et al., 2019; Chen
et al., 2021). GASDA (Zhao et al., 2019) incorporates additional information:
it utilises stereo pairs to encourage geometry consistency and align stereo
images for depth estimation. Similarly, SharinGAN (PNVR et al., 2020) maps
domains to a shared space with geometry constraints. Moreover, DESC (Lopez-
Rodriguez & Mikolajczyk, 2020) adopts segmentation for pseudo-labeling based
on instance properties. Our approach to domain adaptation for depth estimation
offers several advantages compared to existing methods. Specifically, our
method provides direct pseudo-labels on real data in a simple and
straightforward manner, without the need for any external information, such as
semantic information (Lopez-Rodriguez & Mikolajczyk, 2020). Additionally we
leverage labeled source data by simultaneously optimizing on target and source
domain images.

###  2.3 Semi-supervised learning

There are two main methodologies used in semi-supervised learning, namely
entropy minimisation and consistency regularisation. Entropy minimisation
involves self-training on pseudo labels for unlabelled data. Conversely,
consistency regularisation assumes predictions should remain stable under
perturbations. Among the consistency regularisation methods, FixMatch (Sohn et
al., 2020) proposes using strong perturbations for unlabelled images and
supervising the training process with predictions from weakly perturbed
images, combining the benefits of both approaches. Recent advancements like
FlexMatch (Zhang et al., 2021) and FreeMatch (Wang et al., 2022) consider
class-wise confidence thresholds. In Yang et al. (2023), UniMatch is proposed,
which aims to improve the performance of semi-supervised semantic segmentation
by incorporating extra information and imposing constraints on real images.
Our work takes inspiration from these semi-supervised learning techniques that
leverage consistency regularisation principles. However, these approaches have
primarily focused on classification and segmentation tasks with discrete label
spaces. As monocular depth estimation produces continuous-valued depth maps,
it poses a regression problem rather than classification or segmentation.
Inspired by FixMatch (Sohn et al., 2020), we propose a framework for semi-
supervised monocular depth estimation. Specifically, we formulate depth
prediction as regression and design augmentations and consistency losses
suitable for continuous outputs.

##  3 Method

Let $f(\mathbf{x};\theta)$ denote a neural network parameterised by $\theta$
that maps the input image $\mathbf{x}\in\mathbb{R}^{h\times w\times 3}$ to the
depth prediction $\hat{\mathbf{y}}\in\mathbb{R}^{h\times w\times 1}$. The
depth estimation model $f(\cdot)$ is trained on the source domain $D_{S}$
containing images $\mathbf{x}_{s}$ with corresponding ground truth depth maps
$\mathbf{y}_{s}$, and the target domain $D_{T}$ of images $\mathbf{x}_{u}$
without accompanying depth labels. Directly optimising $f(\cdot)$ based solely
on supervised losses in the source domain $D_{S}$ would lead to degraded
performance when applied to the unlabelled target domain $D_{T}$, owing to the
domain gap between the source and the target data distributions.

To address this issue, we propose a semi-supervised learning approach that
facilitates direct supervision on the target image $\mathbf{x}_{u}$, thereby
effectively reducing the domain gap. Initially, we train the depth prediction
model $f(\cdot)$ using source domain data. After this pre-training, we train
our model by enforcing perturbation consistency across augmented views in
$D_{T}$ while also optimising on supervised source data using a new pairwise
loss. Critically, we simultaneously apply both loss terms during training,
allowing source supervision to guide pseudo-label refinement while the
proposed consistency across target perturbations induces adaptation.

Figure 2: Overview of the approach. Initially, we sample two independent
source domain images $\mathbf{x}_{s,1}$ and $\mathbf{x}_{s,2}$ along with one
target domain image $\mathbf{x}_{u}$. This target domain image is then fed
into two perturbation streams, in which independent augmentations are applied
to the target image, denoted as $\tilde{\mathbf{x}}_{u,1}$ and
$\tilde{\mathbf{x}}_{u,2}$. In total five samples are concatenated and fed
into the depth estimation model $f(\cdot)$. Afterwards, the predictions are
chunked back into their initial shapes. The loss on the supervised domain is
computed by enforcing consistency between the sum of predictions and the sum
of the two ground truth samples. Finally, the unsupervised loss is calculated
by enforcing consistency between generated perturbations. Yellow and blue
colors correspond to the supervised source and the unsupervised target domain,
respectively.

#### Preliminaries

We derive inspiration by FixMatch (Sohn et al., 2020) in which each unlabelled
sample undergoes simultaneous perturbations by two operators: a weak
perturbation such as cropping and a strong perturbation like colour jittering.
Weak perturbations are generally augmentations, that minimally alter the scene
content, while strong perturbations introduce more distortion and artefacts.
The overall objective function is a combination of supervised loss
$\mathcal{L}^{s}$ and unsupervised loss $\mathcal{L}^{u}$, given by:

$\mathcal{L}=\frac{1}{2}(\mathcal{L}^{s}+\mathcal{L}^{u})$ (1)

Typically the supervised term $\mathcal{L}_{s}$ is a cross-entropy loss
between the model predictions and the labels. The unsupervised loss
$\mathcal{L}_{u}$ serves to regularise the prediction of the sample under
strong perturbations to be the same as that under weak perturbations.

Based on Eq. 1, we present a formulation for the monocular depth estimation
setting.

We propose to replace the supervised loss $\mathcal{L}^{s}$ with a pairwise
loss $\mathcal{L}_{\Sigma}^{s}$, applied to source domain samples in $D_{S}$,
while also extending $\mathcal{L}^{u}$ with perturbation consistency across
three augmented views in the target domain $D_{T}$.

###  3.1 Perturbation consistency

We seek to minimise the differences between predictions on augmented versions
of the same input image. Due to the unsupervised domain adaptation setting
where no depth annotation for the target domain is available, we rely on
enforcing consistency between multi-stream perturbed image predictions in the
target domain. For each input sample, we apply the RandAugment (Cubuk et al.,
2019) algorithm to generate multiple perturbed views, or ”streams”, of the
data. Each augmentation chain is applied independently to a target sample
$\mathbf{x}_{u}$ to produce uniquely transformed versions of the input. In
total, we generate two augmented versions of the image, in addition to the
initial unlabelled target domain image $\mathbf{x}_{u}$:

$\tilde{\mathbf{x}}_{u,1}=RA_{Depth,1}(\mathbf{x}_{u}),$ (2)
$\tilde{\mathbf{x}}_{u,2}=RA_{Depth,2}(\mathbf{x}_{u}),$ (3)

where $RA_{Depth,1}$ and $RA_{Depth,2}$ represent independent RandAugment
operations. The multi-stream perturbed images $\tilde{\mathbf{x}}_{u,1}$ and
$\tilde{\mathbf{x}}_{u,2}$ along with an initial sample $\mathbf{x}_{u}$ are
then fed through the depth estimation model $f(\cdot)$ to obtain three
predicted depth maps. All three depth predictions should be consistent;
therefore, we compute the $\mathcal{L}_{1}$ loss between the predicted depth
maps to enforce consistency:

$\displaystyle\mathcal{L}_{cons}^{u}=\sum_{i=1}^{N}\lVert
f(\mathbf{x}_{u,i})-f(\tilde{\mathbf{x}}_{u,1,i})\rVert_{1}+\lVert
f(\mathbf{x}_{u,i})-f(\tilde{\mathbf{x}}_{u,2,i})\rVert_{1},$

where $N$ is the batch size. We empirically observed that two augmented
versions of each unlabelled sample provided the best trade-off between
boosting performance via view redundancy and managing memory costs. Moreover,
prior work on semi-supervised learning has shown that generating multiple
perturbed views of unlabelled data strengthens consistency regularisation
(Berthelot et al., 2019; Caron et al., 2020).

###  3.2 Pairwise loss

While the unsupervised perturbation consistency adapts the model to the target
domain without source supervision, there is no way to directly assess the
quality and accuracy of pseudo-labels
$f(\mathbf{x}_{u}),f(\tilde{\mathbf{x}}_{u,1})$ and
$f(\tilde{\mathbf{x}}_{u,2})$. Errors can accumulate during training as the
network learns potentially incorrect target predictions. For that reason, we
regularise the training process, leveraging a pairwise loss
$\mathcal{L}^{s}_{\Sigma}$ in the source domain $D_{S}$. Specifically, for
each source domain batch we sample image pairs
$(\mathbf{x}_{s,1},\mathbf{x}_{s,2})$ along with their corresponding depth
annotations $(\mathbf{y}_{s,1},\mathbf{y}_{s,2})$, and then compute the
$\mathcal{L}_{1}$ distance between the sum of predictions and labels over a
batch of size $N$:

$\hat{\mathbf{y}}_{s,1}=f(\mathbf{x}_{s,1}),\quad\hat{\mathbf{y}}_{s,2}=f(\mathbf{x}_{s,2})$
(4)
$\hat{\mathbf{y}}_{\Sigma,s}=\hat{\mathbf{y}}_{s,1}+\hat{\mathbf{y}}_{s,2}$
(5) $\mathbf{y}_{\Sigma,s}=\mathbf{y}_{s,1}+\mathbf{y}_{s,2}$ (6)
$\mathcal{L}_{\Sigma}^{s}=\sum_{i=1}^{N}||\mathbf{y}_{\Sigma,s,i}-\hat{\mathbf{y}}_{\Sigma,s,i}||_{1}.$
(7)

A key observation from our experiments is that the proposed pairwise source
regularisation approach from the source domain labels requires using an
extremely low learning rate for stable optimisation. Since already seen images
are combined in new configurations during each summation, essentially, this
approach ends up augmenting the training data online.

###  3.3 Complete loss function

In total, we propose a loss formulation that combines the multiple
perturbation consistency constraints imposed on unlabelled target samples with
the proposed summation-based pairwise loss computed on the labelled source
samples. It is given by:

$\mathcal{L}_{total}=\frac{1}{2}\sum_{i}^{N}({\mathcal{L}^{s}_{\Sigma,i}+\mathcal{L}^{u}_{cons,i})\Big{|}_{r=2}}.$
(8)

where $N$ denotes the batch size. Additionally, we introduce the batch ratio
hyperparameter $r=\frac{n_{supervised}}{n_{unsupervised}}$ to balance the
amount of supervised vs. unsupervised training data. A ratio parameter $2$
indicates that twice as many supervised as unsupervised samples were used
during one forward pass (an exact calculation of the total amount of samples
in one forward pass with $N=12$ and $r=2$ is shown in the appendix, in Eq.
15). An ablation (see Table 4, in Sec. 4.3) examines various $r$ values,
revealing a static value of 2 outperforms other schemes.

###  3.4 Model Pretraining

During model pre-training, images from one single batch are mixed with one
another using CutMix (Yun et al., 2019) augmentations. This prevents the depth
estimation model from relying too heavily on local image details, as it forces
the model to consider information from different regions. We apply CutMix to
all source samples $\mathbf{x}_{s}$ and their corresponding annotations
$\mathbf{y}_{s}$ during pretraining. CutMix randomly combines two different
images by cutting out a random patch from one image and replacing it with the
same-sized patch from another image. The corresponding ground truth labels are
also mixed in proportion to the area of the patches, defined by a
hyperparameter $\alpha$. Fig. 3 illustrates this procedure for the virtual
KITTI source dataset (Gaidon et al., 2016). We then compute the
$\mathcal{L}_{1}$-distance between depth predictions of these augmented
samples $\tilde{\mathbf{x}}_{cm,s}$ and their corresponding ground truth depth
annotation $\tilde{\mathbf{y}}_{cm,s}$ over the batch of size $N$:

$\mathbf{x}_{cm,s}$$\mathbf{y}_{cm,s}$

Figure 3: CutMix augmented input images $\mathbf{x}_{cm,s}$ in the source
domain along with their corresponding augmented ground truth depth annotation
$\mathbf{y}_{cm,s}$. We choose $\alpha=0.5$, controlling the patch size of the
CutMix augmentation.

$\mathcal{L}^{s}=\sum_{i=1}^{N}||f(\tilde{\mathbf{x}}_{cm,s,i})-\tilde{\mathbf{y}}_{cm,s,i}||_{1}.$
(9)

##  4 Experiments

We assess our method on two challenging benchmarks, one indoors and one
outdoors. Afterwards, we conduct detailed ablation studies on each benchmark
to analyse the importance of the different components in our approach.

###  4.1 Datasets

#### Outdoor

We utilise the virtual KITTI (vKITTI) (Gaidon et al., 2016) dataset, with
21,260 synthetic image-depth pairs as our labeled source domain, and the real
KITTI dataset (Geiger et al., 2013), containing 22,600 images as the unlabeled
target domain. As vKITTI and KITTI contain outdoor street scenes, we
preprocessed both and clip maximum depth values to 80 m to ensure consistency
when applying our approach during training, similar to (Zheng et al., 2018).
Following prior works (Zheng et al., 2018; Yen et al., 2022; Zhao et al.,
2019), we evaluate the performance on the Eigen test split (Eigen et al.,
2014). For the final test evaluation, we use the Garg crop (Garg et al., 2016)
and clamp the ground truth depth to 50 m and 80 m following prior work (Yen et
al., 2022).

#### Indoor

For our second experiment, we employ the large-scale SceneNet synthetic
dataset (Handa et al., 2015) as the source and evaluated the real-world NYUv2
(Nathan Silberman & Fergus, 2012) indoor dataset as the target domain.
Additionally, we apply a filtering procedure to the SceneNet dataset to better
match the characteristics of NYUv2. Specifically, we aim to resemble the depth
distribution of NYUv2, as SceneNet is considerably larger. After filtering,
our source SceneNet set contains 4,737 samples. We combine this filtered
SceneNet set with a randomly selected subset of 5,000 samples from the raw
NYUv2 data as our target domain samples. By selecting a target domain subset
of similar size to our source domain set, we aim to keep the ratio of
unsupervised to supervised samples close to $1:1$ during training.

Given these datasets capture indoor room layouts and objects, significantly
shorter maximum depths are present. Therefore, we clip the maximum depth to 8
m for this indoor adaptation problem, following prior work (Zheng et al.,
2018). Our final evaluation is then performed on the Eigen split (Eigen et
al., 2014) of the NYU depth data, clipped to 8 m.

#### Implementation details

We adopt a U-Net like (Ronneberger et al., 2015) architecture similar to Zheng
et al. (2018) as our depth prediction model $f(\cdot)$. The model is trained
on an NVIDIA GTX 6000 GPU using the Adam optimizer (Kingma & Ba, 2014),
pretraining is done with a learning rate of $4*10^{-3}$ for $250$ epochs while
our domain adaptation is applied at learning rate of $4*10^{-8}$ for $10$
epoch with a linear decay after 4 epochs. In total pretraining takes about six
hours for the outdoor benchmarks and about one hour for the indoor benchmarks.

#### Evaluation metrics

Consistent with previous research (Yen et al., 2022; Zheng et al., 2018; Kundu
et al., 2018; Zhao et al., 2019), we assess unsupervised domain adaptation for
monocular depth prediction. To evaluate the adapted depth estimation model, we
utilize common metrics, including absolute relative error (AbsRel), root mean
squared error (RMSE), squared relative error (SqRel), logarithm of root mean
squared error (RMSE Log), and accuracy under different threshold values ($A1$
at threshold $\delta<1.25$, $A2$ at $\delta<1.25^{2}$ and $A3$ at
$\delta<1.25^{3}$).

#### Comparison to Related Work

For a fair evaluation, we compare to prior work using analogous experimental
protocols. Specifically, we benchmark against fully-supervised methods
training on single domains (Eigen et al., 2014; Liu et al., 2016). We also
compare to baselines trained on separate source/target distributions without
adaptation. Critically, we evaluate against prominent unsupervised domain
adaptation methods (Kundu et al., 2018; Zheng et al., 2018; Yen et al., 2022)
which similarly utilise unlabelled target data through depth completion or
translation. These prior works employ architectures and datasets matching our
setup, allowing an effective and fair assessment of our approach. The works of
SharingGAN (PNVR et al., 2020) and GASDA (Zhao et al., 2019) report results
when leveraging stereo imagery under the assumption that epipolar geometry
constraints can be imposed. Consequently we do not compare our approaches to
theirs.

Table 1: Results on the KITTI Eigen split (80m cap) for single image depth estimation methods. The column ’supervised’ determines whether an approach has been trained with target image ground truth samples. ’K’ refers to KITTI, ’S’ to the vKITTI and ’CS’ to Cityscapes data. ’DA’ refers to domain adaptation approaches. Method | Supervised | Dataset | AbsRel$\downarrow$ | SqRel$\downarrow$ | RMSE$\downarrow$ | RMSE log$\downarrow$ | A1$\uparrow$ | A2$\uparrow$ | A3$\uparrow$
---|---|---|---|---|---|---|---|---|---
Eigen et al. (2014) | Yes | K | 0.203 | 1.548 | 6.307 | 0.282 | 0.702 | 0.890 | 0.958
Liu et al. (2016) | Yes | K | 0.202 | 1.614 | 6.523 | 0.275 | 0.678 | 0.895 | 0.965
Zhou et al. (2017) | No | K | 0.208 | 1.768 | 6.856 | 0.283 | 0.678 | 0.885 | 0.957
Zhou et al. (2017) | No | K+CS | 0.198 | 1.836 | 6.565 | 0.275 | 0.718 | 0.901 | 0.960
All synthetic | No | S | 0.253 | 2.303 | 6.953 | 0.328 | 0.635 | 0.856 | 0.937
All real | No | K | 0.158 | 1.151 | 5.285 | 0.238 | 0.811 | 0.934 | 0.970
Kundu et al. (2018) | No | K+S(DA) | 0.214 | 1.932 | 7.157 | 0.295 | 0.665 | 0.882 | 0.950
Zheng et al. (2018) | No | K+S(DA) | 0.182 | 1.611 | 6.216 | 0.265 | 0.749 | 0.898 | 0.959
Yen et al. (2022) | No | K+S(DA) | 0.169 | 1.371 | 6.037 | 0.256 | 0.759 | 0.904 | 0.961
Ours | No | K+S(DA) | 0.168 | 1.347 | 6.002 | 0.254 | 0.769 | 0.907 | 0.961
Table 2: Results on the KITTI Eigen split capped to 50m. ’K’ refers to KITTI, ’S’ to the vKITTI and ’CS’ to Cityscapes data. ’DA’ refers to domain adaptation approaches.  Method | Dataset | AbsRel$\downarrow$ | SqRel$\downarrow$ | RMSE$\downarrow$ | RMSE log $\downarrow$ | A1$\uparrow$ | A2$\uparrow$ | A3$\uparrow$
---|---|---|---|---|---|---|---|---
Garg et al. (2016) | K | 0.169 | 1.080 | 5.104 | 0.273 | 0.740 | 0.904 | 0.962
All synthetic | S | 0.244 | 1.771 | 5.354 | 0.313 | 0.647 | 0.866 | 0.943
All real | K | 0.151 | 0.856 | 4.043 | 0.227 | 0.824 | 0.940 | 0.973
Kundu et al. (2018) | K+S(DA) | 0.203 | 1.734 | 6.251 | 0.284 | 0.687 | 0.899 | 0.958
Zheng et al. (2018) | K+S(DA) | 0.168 | 1.199 | 4.674 | 0.243 | 0.772 | 0.912 | 0.966
Yen et al. (2022) | K+S(DA) | 0.162 | 1.049 | 4.463 | 0.239 | 0.776 | 0.916 | 0.968
Ours | K+S(DA) | 0.161 | 1.028 | 4.449 | 0.237 | 0.785 | 0.918 | 0.968
Table 3: Results of our approach on the NYU Eigen split capped at 8m. ’N’ refers to NYU data, ’S’ refers to SceneNet data and ’DA’ corresponds to domain adaptation. Method | Dataset | AbsRel$\downarrow$ | SqRel$\downarrow$ | RMSE$\downarrow$ | RMSE log $\downarrow$ | A1$\uparrow$ | A2$\uparrow$ | A3$\uparrow$
---|---|---|---|---|---|---|---|---
all-real | N | 0.201 | 0.252 | 0.789 | 0.288 | 0.609 | 0.862 | 0.956
all-synthetic | S | 0.432 | 0.720 | 1.183 | 0.439 | 0.395 | 0.686 | 0.859
Zheng et al. (2018) | N+S(DA) | 0.394 | 0.613 | 1.156 | 0.417 | 0.407 | 0.707 | 0.878
Ours | N+S | 0.417 | 0.601 | 1.055 | 0.385 | 0.426 | 0.725 | 0.890

###  4.2 Results

#### Outdoor experiments

Table 1 shows quantitative results where we compare our approach to various
baselines and state-of-the-art approaches with ground truth depth labels
clipped to 80 m. In addition, we compare to ’all-synthetic’ and ’all-real’
baselines, referring to the approaches where the model is trained on synthetic
or real data only. These can be seen as the upper and lower bound,
respectively. We also provide quantitative results capped to 50 m in Table 2.
In this setup, we compare to the supervised approach proposed by Garg et al.
(Garg et al., 2016). In comparison to the standard supervised training
baseline ’all-synthetic’, our method outperforms this baseline by a
significant margin. We further compare to three previous state-of-the-art
semi-supervised depth estimation techniques. Notably, our approach achieves a
significant improvement in $A1$-accuracy compared to prior work and
outperforms 3D-PL (Yen et al., 2022) in all metrics.

#### Indoor experiments

To further assess model generalisation, we conduct additional experiments in
an indoor domain transfer setting. Specifically, we benchmark our approach
against a baseline experiment ’all-synthetic’, trained only on supervised
source data, and T2Net (Zheng et al., 2018). However, as our experimental
protocol evaluates a novel SceneNet-to-NYUv2 domain shift scenario not
explored in the original work, it was necessary to retrain T2Net’s framework
for direct comparison. As shown in Table 3, our approach significantly
outperforms training the ’all-synthetic’ baseline, trained without leveraging
unlabelled NYUv2 data. Remarkably, we achieve gains over the prior work (Zheng
et al., 2018) across most metrics. We outperform Zheng et al. (2018)’s
approach in every metric except the Absolute Relative Error.

Input image $\mathbf{x}_{u}$Ground truthZheng et al. (2018)Yen et al.
(2022)Ours

Figure 4: Qualitative results on KITTI (Geiger et al., 2013) with models
trained on vKITTI-KITTI. Ground truth depth is linearly interpolated for
visualization.

#### Qualitative results

In this setting, we compare our approach trained on the vKITTI-KITTI domain
adaptation task and test it on the KITTI Eigen split (Eigen et al., 2014) with
previous state-of-the-art methods (Yen et al., 2022; Zheng et al., 2018). Fig.
4 shows that our approach better recovers fine-grained depth details in
several examples. In particular, in the upper row our approach produces more
accurate results for the lamp post. Similarly, in the second row our approach
is capable of detecting the cyclist entirely, compared to 3D-PL.

#### Limitations

A key limitation of our approach is its reliance on synthetic depth maps
possessing well-defined edges in the source domain. Due to the discontinuities
at object boundaries present in these rendered depth maps, they differ from
real-world captured depth maps which typically exhibit smoother transitions at
edges. The edge characteristics between our synthetic source depth maps and
target domain data, which lacks such discontinuities, introduces a domain
shift that our method seeks to overcome via target-domain consistency
regularisation and source-based regularisation. A key assumption of our
approach is therefore that the edge characteristics of depth maps in the
target and source domains differ sufficiently.

Another key limitation of the proposed approach stems from its computational
demands (see also Tab. 9). The framework enforces consistency across multiple
augmented views of target images as well as a supervised pairwise loss in the
source domain. This joint optimisation comes at the cost of increased
computational complexity. Scaling the framework to high-resolution imagery or
large datasets would require more expensive GPU hardware.

###  4.3 Ablation studies

Next, we analyse several key hyper-parameters involved in our proposed
methodology. First, we explore the effect of varying the ratio parameter $r$.

Next, we evaluate different data augmentation techniques to understand their
impact on our model’s ability when no source domain samples are provided.

#### Importance of the ratio $r$

The balance between unlabelled target consistency training samples and
labelled source supervision samples is determined by the ratio $r$. To
evaluate the importance of this hyperparameter, we conduct an ablation study
examining different constant sampling ratios. The quantitative results
presented in Table 4 indicate the sensitivity of our approach to the ratio
hyperparameter $r$. Peak performance was observed for a ratio of $r=2/1$,
where twice as many supervised source images as unsupervised target images
were sampled within each batch. Hyperparameters were chosen based on the
obtained uncertainty $u$ (For a more detailed explanation please refer to
A.3). Additionally, the effect of the ratio $r$ on the level of regularisation
can be observed. A ratio $r\approx 1$ leads to weaker regularisation, as
evidenced by the higher absolute and squared relative errors achieved with
lower $r$ values. This is understandable, as a lower $r$ places less
importance on the paired source loss term during training, therefore providing
less regularisation pressure.

Table 4: Ablation study of our approach with ratio hyperparameter $r$. All approaches are capped to 50m. ratio $r$ | AbsRel$\downarrow$ | SqRel$\downarrow$ | RMSE$\downarrow$ | RMSE log $\downarrow$ | A1$\uparrow$ | A2$\uparrow$ | A3$\uparrow$ | Uncertainty $u$
---|---|---|---|---|---|---|---|---
${7}/{4}$ | 0.170 | 1.092 | 4.406 | 0.241 | 0.783 | 0.918 | 0.967 | 216.83
${9}/{2}$ | 0.169 | 1.115 | 4.437 | 0.240 | 0.783 | 0.918 | 0.967 | 198.83
${6}/{5}$ | 0.183 | 1.441 | 4.635 | 0.252 | 0.775 | 0.912 | 0.963 | 219.29
${8}/{15}$ | 0.176 | 1.153 | 4.414 | 0.243 | 0.779 | 0.915 | 0.966 | 210.15
$2/1$ | 0.161 | 1.028 | 4.449 | 0.237 | 0.785 | 0.918 | 0.968 | 170.73

#### Comparison with different loss functions

Next, we conducted an additional experimental comparison using the virtual
KITTI (vKITTI) (Gaidon et al., 2016) and real KITTI (Geiger et al., 2013)
datasets (see Tab. 5). We compare our method to a standard baseline inspired
by FixMatch (Sohn et al., 2020) that only leverages data augmentations within
the target KITTI domain, i.e., without considering our multi-stream
perturbation consistency losses. Note: all experiments in Tab. 5 are conducted
without using the pairwise source loss $\mathcal{L}_{\sum}^{s}$ except from
the experiment ’Ours’ in the last row. In particular, we evaluate a setup
using strong color-space augmentations only, and compare to RandAugment’s
geometric transformations. No weak augmentations were applied in all these
setups.

The ablation study allowed us to assess the benefits of including geometric
augmentations into our approach. Our experimental findings indicate that semi-
supervised techniques for depth estimation consistently outperform direct
pretraining (see the ”all synthetic” experiment in Tab. 2) without additional
adaptation or regularisation. However, incorporating geometric data
augmentations into the consistency regulariser provides a substantial boost in
performance. Overall, our complete approach shows considerable improvements on
the absolute relative error and squared relative error - two key metrics for
assessing depth estimation accuracy.

Table 5: Ablation study: comparison with FixMatch (Sohn et al., 2020) baselines. All approaches are capped to 50m and all experiments are subjected to pre-training. Method | AbsRel$\downarrow$ | SqRel$\downarrow$ | RMSE$\downarrow$ | RMSE log $\downarrow$ | A1$\uparrow$ | A2$\uparrow$ | A3$\uparrow$
---|---|---|---|---|---|---|---
Colorspace perturbations in $D_{T}$ | 0.183 | 1.393 | 4.697 | 0.250 | 0.775 | 0.912 | 0.964
RandAugment perturbations in $D_{T}$ | 0.172 | 1.159 | 4.475 | 0.239 | 0.781 | 0.916 | 0.965
Ours with $\mathcal{L}_{\sum}^{s}$ | 0.161 | 1.028 | 4.449 | 0.237 | 0.785 | 0.918 | 0.968

#### Significance of combining $\mathcal{L}_{\sum}^{s}$ and
$\mathcal{L}_{cons}^{t}$

To determine the effectiveness of our proposed approach combining source loss
$\mathcal{L}_{\sum}^{s}$ and target loss $\mathcal{L}_{cons}^{t}$ we conduct
an ablation study using the losses separately (see Tab. 6).

Table 6: Comparison of our approach when combining and separating both the source loss $\mathcal{L}_{\sum}^{s}$ and the target loss $\mathcal{L}_{cons}^{t}$. Losses | AbsRel$\downarrow$ | SqRel$\downarrow$ | RMSE$\downarrow$ | RMSE log$\downarrow$ | A1$\uparrow$ | A2$\uparrow$ | A3$\uparrow$
---|---|---|---|---|---|---|---
with $\mathcal{L}_{\sum}^{s}$, $\mathcal{L}_{cons}^{t}$ | 0.161 | 1.028 | 4.449 | 0.237 | 0.785 | 0.918 | 0.968
with $\mathcal{L}_{cons}^{t}$ | 0.171 | 1.178 | 4.544 | 0.248 | 0.781 | 0.912 | 0.965
with $\mathcal{L}_{\sum}^{s}$ | 0.178 | 1.356 | 5.040 | 0.258 | 0.779 | 0.912 | 0.961
Only pretraining | 0.173 | 1.413 | 5.062 | 0.264 | 0.767 | 0.906 | 0.957

These results reveal that the primary driver of the model’s enhanced
performance is the unsupervised consistency loss enforced on target
perturbations (i.e. $\mathcal{L}_{cons}^{t}$). Specifically, the pairwise loss
on the source domain (i.e. $\mathcal{L}_{\sum}^{s}$) alone provides only
minimal regularisation benefits compared to the baseline. However, when
combining both proposed objectives, we observe more noticeable gains according
to certain error metrics such as absolute relative error (Abs Rel) and root
mean squared error of the logarithm (RMSE log). This suggests that jointly
leveraging source supervision together with unsupervised consistency across
target transformations produces a regularisation effect superior to utilising
either loss separately.

##  5 Conclusion

We formalised the problem of unsupervised domain adaptation for monocular
depth estimation as consistency-based semi-supervised learning. Our key
contributions are the simplicity and effectiveness of our method for training
a single model to produce high quality depth estimates in the unlabelled
target domain, based on the consistency of the predictions under different
input augmentations. The proposed joint optimisation of the source and target
domains with the proposed perturbation consistency loss and the pairwise
source loss functions simplifies the training process compared to previous
work that relies on complex training protocols and uses more than a single
model. In our experiments, we showed promising results on the KITTI and NYUv2
datasets, which are considered standard for the domain adaptation task in
monocular depth estimation.

##  6 Acknowledgements

Part of the research leading to these results is funded by the German Research
Foundation (DFG) within the project 458972748. The authors would like to thank
the foundation for the successful cooperation.

Additionally the authors gratefully acknowledge the scientific support and HPC
resources provided by the Erlangen National High Performance Computing Center
(NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU). The
hardware is funded by the German Research Foundation (DFG).

## References

  * Agarwal & Arora (2023) Ashutosh Agarwal and Chetan Arora.  Attention attention everywhere: Monocular depth prediction with skip attention.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 5861–5870, 2023.
  * Akada et al. (2022) Hiroyasu Akada, Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka.  Self-supervised learning of domain invariant features for depth estimation.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 3377–3387, 2022.
  * Amiri et al. (2019) Ali Jahani Amiri, Shing Yan Loo, and Hong Zhang.  Semi-supervised monocular depth estimation with left-right consistency using deep neural network.  In _2019 IEEE International Conference on Robotics and Biomimetics (ROBIO)_ , pp. 602–607, 2019.  doi: 10.1109/ROBIO49542.2019.8961504.
  * Atapour Abarghouei & Breckon (2018) Amir Atapour Abarghouei and Toby Breckon.  Real-time monocular depth estimation using synthetic data with domain adaptation via image style transfer.  pp. 2800–2810, 06 2018.  doi: 10.1109/CVPR.2018.00296.
  * Berthelot et al. (2019) David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel.  Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring.  In _International Conference on Learning Representations_ , 2019.
  * Bhat et al. (2021) Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka.  Adabins: Depth estimation using adaptive bins.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 4009–4018, 2021.
  * Cao et al. (2016) Yuanzhouhan Cao, Zifeng Wu, and Chunhua Shen.  Estimating Depth from Monocular Images as Classification Using Deep Fully Convolutional Residual Networks, May 2016.  URL https://arxiv.org/abs/1605.02305v3.
  * Caron et al. (2020) Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  _Advances in neural information processing systems_ , 33:9912–9924, 2020.
  * Chen et al. (2021) Xiaotian Chen, Yuwang Wang, Xuejin Chen, and Wenjun Zeng.  S2r-depthnet: Learning a generalizable depth-specific structural representation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 3034–3043, 2021.
  * Chen et al. (2019) Yun-Chun Chen, Yen-Yu Lin, Ming-Hsuan Yang, and Jia-Bin Huang.  Crdoco: Pixel-level domain transfer with cross-domain consistency.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 1791–1800, 2019.
  * Cubuk et al. (2019) Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le.  RandAugment: Practical automated data augmentation with a reduced search space, November 2019.  URL http://arxiv.org/abs/1909.13719.  arXiv:1909.13719 [cs].
  * Eigen et al. (2014) David Eigen, Christian Puhrsch, and Rob Fergus.  Depth Map Prediction from a Single Image using a Multi-Scale Deep Network, June 2014.  URL http://arxiv.org/abs/1406.2283.  arXiv:1406.2283 [cs].
  * Fu et al. (2018) Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao.  Deep Ordinal Regression Network for Monocular Depth Estimation, June 2018.  URL https://arxiv.org/abs/1806.02446v1.
  * Gaidon et al. (2016) Adrien Gaidon, Qiao Wang, Yohann Cabon, and Eleonora Vig.  Virtual Worlds as Proxy for Multi-Object Tracking Analysis, May 2016.  URL http://arxiv.org/abs/1605.06457.  arXiv:1605.06457 [cs, stat].
  * Garg et al. (2016) Ravi Garg, Vijay Kumar Bg, Gustavo Carneiro, and Ian Reid.  Unsupervised cnn for single view depth estimation: Geometry to the rescue.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14_ , pp. 740–756. Springer, 2016.
  * Geiger et al. (2013) Andreas Geiger, P Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: the KITTI dataset.  _The International Journal of Robotics Research_ , 32:1231–1237, September 2013.  doi: 10.1177/0278364913491297.
  * Godard et al. (2017) Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow.  Unsupervised monocular depth estimation with left-right consistency.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 270–279, 2017.
  * Handa et al. (2015) Ankur Handa, Viorica Patraucean, Vijay Badrinarayanan, Simon Stent, and Roberto Cipolla.  SceneNet: Understanding Real World Indoor Scenes With Synthetic Data, November 2015.  URL http://arxiv.org/abs/1511.07041.  arXiv:1511.07041 [cs].
  * Hornauer & Belagiannis (2022) Julia Hornauer and Vasileios Belagiannis.  Gradient-based Uncertainty for Monocular Depth Estimation, August 2022.  URL http://arxiv.org/abs/2208.02005.  arXiv:2208.02005 [cs].
  * Hornauer et al. (2021) Julia Hornauer, Lazaros Nalpantidis, and Vasileios Belagiannis.  Visual domain adaptation for monocular depth estimation on resource-constrained hardware.  _CoRR_ , abs/2108.02671, 2021.  URL https://arxiv.org/abs/2108.02671.
  * Kingma & Ba (2014) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980, 2014.  URL https://api.semanticscholar.org/CorpusID:6628106.
  * Kundu et al. (2018) Jogendra Kundu, Phani Krishna Uppala, Anuj Pahuja, and R. Babu.  Adadepth: Unsupervised content congruent adaptation for depth estimation.  03 2018.
  * Kuznietsov et al. (2017) Yevhen Kuznietsov, Jorg Stuckler, and Bastian Leibe.  Semi-supervised deep learning for monocular depth map prediction.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 6647–6655, 2017.
  * Laina et al. (2016) Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab.  Deeper depth prediction with fully convolutional residual networks.  In _2016 Fourth international conference on 3D vision (3DV)_ , pp. 239–248. IEEE, 2016.
  * Lin et al. (2020) Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn.  Space: Unsupervised object-oriented scene representation via spatial attention and decomposition.  _arXiv preprint arXiv:2001.02407_ , 2020.
  * Liu et al. (2023) Ce Liu, Suryansh Kumar, Shuhang Gu, Radu Timofte, and Luc Van Gool.  Va-depthnet: A variational approach to single image depth prediction, 2023\.
  * Liu et al. (2016) Fayao Liu, Chunhua Shen, Guosheng Lin, and Ian Reid.  Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 38(10):2024–2039, October 2016.  ISSN 0162-8828, 2160-9292.  doi: 10.1109/TPAMI.2015.2505283.  URL http://arxiv.org/abs/1502.07411.  arXiv:1502.07411 [cs].
  * Lopez-Rodriguez & Mikolajczyk (2020) Adrian Lopez-Rodriguez and Krystian Mikolajczyk.  DESC: Domain Adaptation for Depth Estimation via Semantic Consistency, September 2020.  URL http://arxiv.org/abs/2009.01579.  arXiv:2009.01579 [cs].
  * Mahjourian et al. (2018) Reza Mahjourian, Martin Wicke, and Anelia Angelova.  Unsupervised learning of depth and ego-motion from monocular video using 3d geometric constraints.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 5667–5675, 2018.
  * Nathan Silberman & Fergus (2012) Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  In _ECCV_ , 2012.
  * Park & Park (2022) Hyeseung Park and Seungchul Park.  An unsupervised depth-estimation model for monocular images based on perceptual image error assessment.  _Applied Sciences_ , 12(17), 2022.  ISSN 2076-3417.  doi: 10.3390/app12178829.  URL https://www.mdpi.com/2076-3417/12/17/8829.
  * Peng Wang et al. (2015) Peng Wang, Xiaohui Shen, Zhe Lin, Scott Cohen, Brian Price, and Alan Yuille.  Towards unified depth and semantic prediction from a single image.  In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2800–2809, Boston, MA, USA, June 2015\. IEEE.  ISBN 978-1-4673-6964-0.  doi: 10.1109/CVPR.2015.7298897.  URL http://ieeexplore.ieee.org/document/7298897/.
  * PNVR et al. (2020) Koutilya PNVR, Hao Zhou, and David Jacobs.  SharinGAN: Combining Synthetic and Real Data for Unsupervised Geometry Estimation, June 2020.  URL http://arxiv.org/abs/2006.04026.  arXiv:2006.04026 [cs].
  * Qiu & Yuille (2016) Weichao Qiu and Alan Yuille.  Unrealcv: Connecting computer vision to unreal engine.  In _Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14_ , pp. 909–916. Springer, 2016.
  * Ranftl et al. (2021) René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.  Vision transformers for dense prediction.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 12179–12188, 2021.
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  Lecture Notes in Computer Science, pp. 234–241, Cham, 2015. Springer International Publishing.  ISBN 978-3-319-24574-4.  doi: 10.1007/978-3-319-24574-4˙28.
  * Rudolph et al. (2022) Michael Rudolph, Youssef Dawoud, Ronja Güldenring, Lazaros Nalpantidis, and Vasileios Belagiannis.  Lightweight monocular depth estimation through guided decoding, 2022.
  * Saxena et al. (2009) Ashutosh Saxena, Min Sun, and Andrew Ng.  Make3d: Learning 3d scene structure from a single still image.  _IEEE transactions on pattern analysis and machine intelligence_ , 31:824–40, 06 2009.  doi: 10.1109/TPAMI.2008.132.
  * Shrivastava et al. (2017) Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb.  Learning from simulated and unsupervised images through adversarial training.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2107–2116, 2017.
  * Sohn et al. (2020) Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  _Advances in neural information processing systems_ , 33:596–608, 2020.
  * Sun et al. (2023) Zitang Sun, Shin’ya Nishida, and Zhengbo Luo.  Unsupervised learning optical flow in multi-frame dynamic environment using temporal dynamic modeling.  _arXiv preprint arXiv:2304.07159_ , 2023.
  * Wang et al. (2018) Yang Wang, Yi Yang, Zhenheng Yang, Liang Zhao, Peng Wang, and Wei Xu.  Occlusion aware unsupervised learning of optical flow.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 4884–4893, 2018.
  * Wang et al. (2022) Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, et al.  Freematch: Self-adaptive thresholding for semi-supervised learning.  _arXiv preprint arXiv:2205.07246_ , 2022.
  * Xie et al. (2016) Junyan Xie, Linli Xu, and Enhong Chen.  Deep3d: Fully automatic 2d-to-3d video conversion with deep convolutional neural networks.  In _AAAI_ , pp. 542–548, 2016.
  * Yang et al. (2023) Lihe Yang, Lei Qi, Litong Feng, Wayne Zhang, and Yinghuan Shi.  Revisiting weak-to-strong consistency in semi-supervised semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7236–7246, 2023.
  * Yen et al. (2022) Yu-Ting Yen, Chia-Ni Lu, Wei-Chen Chiu, and Yi-Hsuan Tsai.  3D-PL: Domain Adaptive Depth Estimation with 3D-aware Pseudo-Labeling, September 2022.  URL http://arxiv.org/abs/2209.09231.  arXiv:2209.09231 [cs].
  * Yin et al. (2019) Wei Yin, Yifan Liu, Chunhua Shen, and Youliang Yan.  Enforcing geometric constraints of virtual normal for depth prediction, July 2019.  URL https://arxiv.org/abs/1907.12209v2.
  * Yin et al. (2018) Zhichao Yin, Jianping Shi, Hui Li, Xiaogang Dai, and Xinchao Lin.  Geonet: Unsupervised learning of dense depth, optical flow and camera pose.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 1983–1992, 2018.
  * Yuan et al. (2022) Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan.  New crfs: Neural window fully-connected crfs for monocular depth estimation.  _arXiv preprint arXiv:2203.01502_ , 2022.
  * Yun et al. (2019) Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features, August 2019.  URL http://arxiv.org/abs/1905.04899.  arXiv:1905.04899 [cs].
  * Zhang et al. (2021) Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki.  Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling.  _Advances in Neural Information Processing Systems_ , 34:18408–18419, 2021.
  * Zhao et al. (2019) Shanshan Zhao, Huan Fu, Mingming Gong, and Dacheng Tao.  Geometry-Aware Symmetric Domain Adaptation for Monocular Depth Estimation, April 2019.  URL http://arxiv.org/abs/1904.01870.  arXiv:1904.01870 [cs].
  * Zheng et al. (2018) Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai.  T2Net: Synthetic-to-Realistic Translation for Solving Single-Image Depth Estimation Tasks, August 2018.  URL http://arxiv.org/abs/1808.01454.  arXiv:1808.01454 [cs].
  * Zhou et al. (2017) Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe.  Unsupervised learning of depth and ego-motion from video.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 6619–6627, 2017.
  * Zou et al. (2018) Yuliang Zou, Zelun Luo, and Jia-Bin Huang.  Df-net: Unsupervised joint learning of depth and flow using cross-task consistency.  In _Proceedings of the European conference on computer vision (ECCV)_ , pp. 36–53, 2018.

##  Appendix A Appendix

###  A.1 Metrics

We use the common evaluation metrics used for several depth estimation
benchmarks and established by Eigen et al. (2014). These include:

  * •

Root Mean Squared Error (RMSE):

$RMSE=\sqrt{\frac{1}{N}\sum_{i=1}^{N}||d_{i}-\hat{d}_{i}||^{2}}$ (10)

  * •

RMSE log:

$RMSE_{log}=\sqrt{\frac{1}{N}\sum_{i=1}^{N}||\log{d_{i}}-\log{\hat{d}_{i}}||^{2}}$
(11)

  * •

Squared Relative Error (Sq Rel):

$SqRel=\frac{1}{N}\sum_{i=1}^{N}\frac{||d_{i}-\hat{d}_{i}||^{2}}{d_{i}^{2}}$
(12)

  * •

Absolute Relative Error (Abs Rel):

$AbsRel=\frac{1}{N}\sum_{i=1}^{N}\frac{|d_{i}-\hat{d}_{i}|}{d_{i}}$ (13)

  * •

Accuracy under threshold $\delta$:

$Accuracy_{\delta}=\frac{1}{N}\sum_{i=1}^{N}I(|\hat{d}_{i}-d_{i}|<(d_{i}*\delta))$
(14)

Where $d_{i}$ is the ground truth depth, $\hat{d}_{i}$ is the estimated depth,
$N$ is the total number of samples, and $I(\cdot)$ is an indicator function
that returns 1 if the condition holds, 0 otherwise. In the experimental
results $A1$ refers to $\delta<1.25$,$A2$ to $\delta<1.25^{2}$ and $A3$ to
$\delta<1.25^{3}$.

###  A.2 Pretraining

During pretraining, we solely use source domain data $D_{S}$ to train the
depth estimation model $f(\cdot)$. The following sections explain
hyperparameter adjustment for the proposed pretraining approach and provide
more information on the used augmentations during pretraining.

#### CutMix Hyperparameter $\alpha$

We use a hyperparameter of $\alpha=0.5$ to balance the information between the
original and augmented images during CutMix (Yun et al., 2019). This ratio was
chosen so that the model would not be overwhelmed by drastic augmentations, as
monocular depth estimation requires an understanding of the surrounding scene
context. With $\alpha=0.5$, patches from the two input images are mixed
approximately equally, providing additional cues about depth relationships
while still retaining most of the original scene structure.

#### Other augmentations

In addition to CutMix augmentations during the pre-training, we also employ
standard color and geometric data augmentation techniques. Color
augmentations, including random jitter, are applied to expose the model to
variations in colorspace. Furthermore, geometric transformations such as
random rotations are utilised. This can be seen in Fig. 5.

Figure 5: CutMix augmented images with random jitter and rotation
augmentations in $D_{S}$ for images in vKITTI.

Input image $\mathbf{x}_{u}$Zheng et al. (2018)Yen et al.
(2022)$f(\cdot)$after refinement

Figure 6: Additional qualitative results on KITTI eigen test samples $x_{u}$
between other state-of-the-art models.

Input image $\mathbf{x}_{u}$pretrained $f(\cdot$)$f(\cdot)$after refinement

Figure 7: Comparison of depth predictions on KITTI eigen test samples $x_{u}$,
after pretraining (middle column) and after refinement (right column).

#### Quantitative results

Our findings demonstrate that models pre-trained using the augmentation
pipeline incorporating CutMix, randomised colorspace jittering, and geometric
rotations produce improved depth predictions compared to networks exclusively
optimised on the labeled vKITTI source data without augmentations (see Tab.
7). However, quantitative evaluation reveals the proposed approach does not
achieve a performance similar to state-of-the-art methods for monocular depth
estimation in unsupervised domain adaptation.

#### Qualitative results

Further qualitative results in the pretraining stage, where the model
$f(\cdot)$ is only trained on heavily augmented source data in $D_{S}$, can be
seen in Fig. 7. A comparison between the pretrained $f(\cdot)$ and $f(\cdot)$
after the proposed approach shows a clear smoothing effect of our approach.
Sharp discontinuities present in the boundary regions of the pretrained
$f(\cdot)$ maps are noticeably attenuated after refinement. This indicates the
model $f(\cdot)$ learns to produce depth outputs that align more smoothly
across pixel neighbours after leveraging both labelled source data and
unlabelled target images during refinement.

Table 7: Depth estimation performance of our model $f(\cdot)$ after the pretraining in $D_{S}$. $f(\cdot)$ is trained on vKITTI only and tested on KITTI (50m cap). Different augmentation pipelines are compared. Augmentations | AbsRel$\downarrow$ | SqRel$\downarrow$ | RMSE$\downarrow$ | RMSE log$\downarrow$ | A1$\uparrow$ | A2$\uparrow$ | A3$\uparrow$
---|---|---|---|---|---|---|---
None | 0.244 | 1.771 | 5.354 | 0.313 | 0.647 | 0.866 | 0.943
CutMix, rotations, jitter | 0.173 | 1.413 | 5.062 | 0.264 | 0.767 | 0.906 | 0.957
Rotations, jitter | 0.187 | 1.538 | 5.104 | 0.266 | 0.756 | 0.901 | 0.958

#### Training process

During refinement, both the source ($D_{S}$) and target ($D_{T}$) datasets are
jointly used to update the monocular depth estimation model $f(\cdot)$. To
augment the training data, we apply a combination of various techniques, which
will be discussed in the following section. Subsequently, we provide more
qualitative results. Finally, we present details on the computational
resources required for the refinement process.

###  A.3 Further ablation studies

#### Number of perturbations in $\mathcal{L}_{cons}^{t}$

To explore the effect of increasing target domain perturbations, we conduct an
experiment varying the number of augmentation streams applied to target
samples (see Tab. 8). Specifically, we evaluate model performance when
employing two, three and four independent RandAugment (Cubuk et al., 2019)
augmentations of target samples simultaneously as multi-stream perturbations
during training, in addition to the pairwise supervised source loss
$\mathcal{L}_{\sum}^{s}$.

Table 8: Comparison of required GPU memory and performance metrics for two,three and four perturbations of target images for the target loss $\mathcal{L}_{cons}^{t}$. # of perturbations | required GPU memory | Abs Rel$\downarrow$ | A1$\uparrow$
---|---|---|---
2 | 30GB | 0.165 | 0.779
3 | 35GB | 0.161 | 0.785
4 | 40GB | 0.161 | 0.786

It should be noted that we calculate the number of perturbations by counting
each augmented sample alongside its corresponding original, non-augmented
counterpart. For example, to obtain a total of three perturbations, we employ
one baseline image paired with two transformed views. Our experiments revealed
that the quantity of perturbations applied to target domain data has a
substantial impact on the efficacy of the proposed approach. While gains were
observed for all perturbation counts investigated, performance improvements
became more pronounced with more than two augmented samples. Peak results were
attained with three perturbations, as evidenced by our key evaluation metrics.
As incorporating additional perturbations linearly increases computational
overhead and memory requirements, we opted to use the three-perturbation
setting due to its trade-off between achieved accuracy and demanded resources.

#### Training time in comparison to other approaches

To facilitate quantitative comparison of computational requirements, we
conducted an ablation analysis reporting training time and number of models
for our method and relevant baselines. Since training duration scales with the
number of models optimized as well as their architectures, we record both the
total time spent training and the additional models employed beyond our
approach. Across all experiments, we log two performance metrics(see Tab. 9).

Table 9: Comparison of the proposed approach with other methods in terms of the total training time, the number of additionally trained models and two performance metrics. Method | Total training time | # of trainable models | Abs Rel$\downarrow$ | A2$\uparrow$
---|---|---|---|---
Ours |  $\approx$6h | 1 | 0.161 | 0.918
Zheng et al. (2018) |  $\approx$8h | 3 | 0.168 | 0.912
Yen et al. (2022) | - | 2 | 0.162 | 0.916
Lopez-Rodriguez & Mikolajczyk (2020) |  $\approx$18h | 3 | 0.161 | 0.915

Due to the unavailability of training code from 3D-PL (Yen et al., 2022), we
could not directly replicate their model optimization procedure and thus
cannot report their total training time. While DESC (Lopez-Rodriguez &
Mikolajczyk, 2020) attains markedly better results than other referenced
techniques, it employs a multi-stage pipeline incorporating additional models
for object detection, sim-to-real translation, and depth estimation.
Remarkably, our proposed method outperforms all prior work across standard
evaluation metrics for this task while retaining a highly efficient single-
model formulation.

#### Significance of $\mathcal{L}_{\sum}^{s}$

We conduct additional ablation experiments to further evaluate the impact of
our proposed pairwise source loss $\mathcal{L}_{\sum}^{s}$ (see Tab. 10).
First, we compare the standard per-sample $\mathcal{L}_{1}$ loss
$(\mathcal{L}_{1}^{s}(\hat{y}_{s,1},y_{s,1}))$ against our pairwise
formulation. Secondly, we independently apply the pairwise loss
($\mathcal{L}_{\sum,sep}^{s}=\mathcal{L}_{1}(\hat{y}_{s,1},y_{s,1})+\mathcal{L}_{1}(\hat{y}_{s,2},y_{s,2}))$
without jointly optimising the samples.
All experiments utilise three target domain perturbations for consistency
regularisation, to isolate the effects of varying only the source supervision
approach.

Table 10: Comparison of different implementations for the source loss. source loss | AbsRel$\downarrow$ | SqRel$\downarrow$ | RMSE$\downarrow$ | RMSE log$\downarrow$ | A1$\uparrow$ | A2$\uparrow$ | A3$\uparrow$
---|---|---|---|---|---|---|---
no source loss | 0.171 | 1.178 | 4.544 | 0.248 | 0.781 | 0.912 | 0.965
$\mathcal{L}_{\sum}^{s}$ | 0.161 | 1.028 | 4.449 | 0.237 | 0.785 | 0.918 | 0.968
$\mathcal{L}_{1}^{s}$ | 0.167 | 1.143 | 4.468 | 0.241 | 0.783 | 0.913 | 0.965
$\mathcal{L}_{\sum,sep}^{s}$ | 0.164 | 1.109 | 4.491 | 0.241 | 0.781 | 0.912 | 0.966

Across all experiments utilising the source loss $\mathcal{L}^{s}$, we
observed a regularising effect relative to omitting source supervision
entirely. However, upon closer inspection, the Abs Rel and Sq Rel error
metrics revealed a notable performance gap. Specifically, the pairwise loss
${\mathcal{L}_{\sum}}^{s}$ substantially outperformed the standard per-sample
${\mathcal{L}1}^{s}$ and independently applied pairwise
${\mathcal{L}_{\sum,sep}}^{s}$ variants.

#### Hyperparameter selection

Given the lack of depth annotations in the target domain, we required an
unsupervised method to assess prediction performance without ground truth
labels. Gradient-based uncertainty estimation (Hornauer & Belagiannis, 2022)
allows for such an evaluation under this unlabelled setting.
By approximating a model’s uncertainty through perturbations and gradients,
this technique enables quantification of a prediction’s ambiguity even without
a labelled target. As our goal was to validate the quality and reliability of
our estimated depth maps for the unlabelled target domain, we opted to apply
this approach.
The hyperparameters ($r$, learning rate, number of epochs) were optimised via
predictive uncertainty estimation of the trained models. Models trained with
various hyperparameter values were evaluated, model selection was then based
on minimizing average estimated uncertainty.
Specifically, we generated reference depth predictions by augmenting inputs
through horizontal flipping (Hornauer & Belagiannis, 2022). To approximate
uncertainty, gradients of the convolutional blocks within the UNet-based depth
estimation network, particularly in the expansion phase, were extracted.
We performed gradient-based uncertainty analysis on the KITTI Eigen validation
split comprising 888 to evaluate outdoor performance. For indoors, we applied
the uncertainty estimation to a held-out 1000 image NYUv2 subset, both unseen
during training.

###  A.4 Training process in $D_{T}$

To gain deeper insights, ablation experiments are also conducted investigating
the impact of perturbations applied via RandAugment (Cubuk et al., 2019)
mutli-stream consistency regularisation. RandAugment utilises randomly
composed augmentations parameterised by the augmentation set $S$, depth $n$
and severity $m$. Specifically, we analyse variations in these hyperparameters
which control the RandAugment transformations enforcing predictive consistency
between augmented unlabelled target samples. The parameters studied include
the augmentation set $S$ from which augmentations are selected, as well as the
ranges $n$ and $m$ defining the permissible number and strength of
augmentations in a random augmentation chain. Our default augmentation set is
similar to the configuration presented in the FixMatch (Sohn et al., 2020)
literature, though we additionally evaluate removing geometric transforms to
analyse their contribution. The official augmentation set $s_{fm}$ used in
FixMatch are:

    s_fm = [AutoContrast(),
        Brightness(),
        Color(),
        Contrast(),
        Equalize(),
        Identity(),
        Posterize(),
        Rotate(),
        Sharpness(),
        ShearX(),
        ShearY(),
        Solarize(),
        TranslateX(),
        TranslateY()]

In our ablation studies we removed all geometric augmentation, such that
$s_{geo}$:

    s_geo = [AutoContrast(),
        Brightness(),
        Color(),
        Contrast(),
        Equalize(),
        Identity(),
        Posterize(),
        Sharpness(),
        Solarize()]

In Tab. 11 the performance of the depth estimation model $f(\cdot)$ is
compared in mulitple variations of the hyperparameters $s$,$n$ and $m$. The
results indicate that the augmentation set $s_{geo}$ is outperformed by
$s_{fm}$. The best performance is given for the parameter combination
$s=s_{fm}$,$n=1$ and $m=7$ although performance gaps in general are minor for
parameters within $s_{fm}$. During our analysis of the augmented data, we
observed that for the parameter setting of the depth parameter $n=1$, the
transformations sometimes did not perturb the input at all, limiting their
regularisation effect. To address this, we introduce a static CutOut
augmentation applied to the input images prior to the other transformations.

Table 11: Comparison of the refinement process trained on vKITTI-KITTI and tested on the KITTI eigen split (capped to 50m), depending on the augmentation set $s$, depth $n$ and severity $m$ of the augmentation in $D_{T}$. $s$ | $n$ | $m$ | AbsRel$\downarrow$ | SqRel$\downarrow$ | RMSE$\downarrow$ | RMSE log$\downarrow$ | A1$\uparrow$ | A2$\uparrow$ | A3$\uparrow$
---|---|---|---|---|---|---|---|---|---
$s_{geo}$ | 1 | 5 | 0.172 | 1.164 | 4.490 | 0.244 | 0.781 | 0.916 | 0.966
$s_{fm}$ | 1 | 5 | 0.169 | 1.087 | 4.428 | 0.239 | 0.783 | 0.917 | 0.967
$s_{fm}$ | 5 | 7 | 0.173 | 1.123 | 4.41 | 0.242 | 0.781 | 0.916 | 0.966
$s_{fm}$ | 1 | 7 | 0.161 | 1.028 | 4.449 | 0.237 | 0.785 | 0.918 | 0.968

#### Qualitative results

To further evaluate our approach on more complex KITTI scenes, Fig. 6 compares
our refined model $f(\cdot)$ to other state-of-the-art methods using denser
samples from the KITTI eigen test split. Qualitatively, our method produces
depth maps that better preserve fine structural details throughout the scenes.

Input image $\mathbf{x}_{u}$Zheng et al. (2018)$f(\cdot)$after refinement

Figure 8: Qualitative results on NYUv2 eigen test samples $x_{u}$ to another
state-of-the-art model. Both models were trained in the indoor domain
adaptation setting (SceneNet-NYUv2) and are evaluated on NYUv2 eigen test
samples.

Prior work tends to oversmooth object boundaries and thin structures such as
street signs. In contrast, $f(\cdot)$ trained with our domain adaptation
framework more sharply smoothes objects at boundaries while still maintaining
consistent predictions. This indicates its capability to balance high-
frequency content with regularizing across image domains $D_{S}$ and $D_{T}$.

To further demonstrate the effectiveness of our approach in indoor scenes, we
present qualitative evaluations on the NYUv2 (Nathan Silberman & Fergus, 2012)
eigen test set (Eigen et al., 2014) in Fig. 8. Similar to the outdoor KITTI
results, our method preserves fine-scale structural details that prior methods
often overlook.

For example, as shown in the first row of Fig. 8, our predicted depth maps are
able to sharply identify objects like toilet seats with thin structures.
Additionally, Fig. 8 shows that our approach accurately infers the scene
layout while also capturing small objects such as a desk lamp against complex
indoor backgrounds.

#### Hardware requirements

Empirically, we found a ratio parameter $r=2$ performed best across indoor and
outdoor scenes. With a batch size $N$ of 12, the concatenated inputs contain
$N_{concat}=42$ images as:

$N_{concat}=\underbrace{3\times
N\times\frac{1}{r}}_{[x_{u},\tilde{x}_{u,1},\tilde{x}_{u,2}]\in
D_{T}}+\underbrace{2\times N\times r}_{[x_{s,1},x_{s,2}]\in
D_{S}}\bigg{|}_{r=2,b=12}$ (15)

For KITTI resized to 640x192, this requires 35GB of GPU memory. With NYUv2 at
256x192, memory usage is 20GB.

Higher resolutions increase memory cost due to redundant feature
representations. We also tested varying $N$, but larger batches exceed GPU
limits whereas smaller $N$ weakened the overall regularisation effect. Given
hardware constraints, the above configuration provides the optimal trade-off
between batch size, memory requirements, and regularisation effect during our
refinement approach.