# NeRFmentation: NeRF-based Augmentation for Monocular Depth Estimation

###### Abstract

The capabilities of monocular depth estimation (MDE) models are limited by the
availability of sufficient and diverse datasets. In the case of MDE models for
autonomous driving, this issue is exacerbated by the linearity of the captured
data trajectories. We propose a NeRF-based data augmentation pipeline to
introduce synthetic data with more diverse viewing directions into training
datasets and demonstrate the benefits of our approach to model performance and
robustness. Our data augmentation pipeline, which we call “NeRFmentation”,
trains NeRFs on each scene in the dataset, filters out subpar NeRFs based on
relevant metrics, and uses them to generate synthetic RGB-D images captured
from new viewing directions. In this work, we apply our technique in
conjunction with three state-of-the-art MDE architectures on the popular
autonomous driving dataset, KITTI, augmenting its training set of the Eigen
split. We evaluate the resulting performance gain on the original test set, a
separate popular driving set, and our own synthetic test set.

††*Authors contributed equally

##  1 Introduction

Figure 1: NeRFmentation. We develop a novel dataset augmentation pipeline that
utilizes NeRF-generated data rendered from poses unseen in the original
dataset to improve the robustness of Monocular Depth Estimation networks. The
method is intended to improve generalization capability in cases where the
initial dataset provides limited spatial variability. The RGB image is taken
from Waymo Open Dataset [35]

As the field of computer vision continues to advance at a rapid pace, the
pursuit of safer and more reliable autonomous driving systems remains
paramount. Within this context, monocular depth estimation (MDE) plays an
important role, offering a key solution to the complex challenge of depth
perception in dynamic road environments. Although it offers a solution to
these particular situations, it comes with extra challenges compared to stereo
or in general multi-camera depth estimation. Accurately estimating the depth
from a single image remains a challenging problem due to the inherent scale
ambiguity in 2D projections of 3D scenes, which leads to an infinite number of
geometrically plausible 3D reprojections for every 2D image. This means that
monocular depth estimation is an underdetermined problem that relies on
extracting and interpreting visual cues in images correctly. To address this
challenge, numerous approaches leverage large-scale datasets to train deep
neural networks, allowing them to learn to interpret visual cues and transform
them into complex mappings between image features and corresponding depth
information.

Figure 2: Overview of our proposed pipeline. The pipeline consists of four
distinct stages. We train a NeRF for each scene in the MDE training dataset,
reserving a portion of the images and poses per scene to evaluate NeRF
reconstruction quality. Subpar scene NeRFs are filtered out. Then, we render
novel views of the scene by systematically perturbing the original poses and
querying the NeRF with these poses. Lastly, we combine the novel views with
the original views of the scene to create a NeRFmented training dataset for
the MDE network. Here, the source dataset is KITTI. [12]

This mapping is implicitly learned from the data that is being fed into the
system during training time and a common approach to making the model learn
these mappings is to implement sophisticated modules to extend the underlying
neural network architecture.

While improving the model architecture is a promising approach to this
problem, it also introduces the need for more training data due to an
increased model complexity. This need for more training data can be dealt with
by adding an existing dataset or by recording a new one. The former is common
practice [22], although there are only finitely many real-world datasets.
Recording a new dataset, however, is in the majority of cases infeasible since
this introduces massive costs. This infeasibility of recording new datasets
and the limited availability of real-world datasets leads to the need for
different data sources. For that, one can resort to using synthetic datasets
[30, 21, 23, 26]. This could lead to an unwanted domain shift which in turn
can be compensated by using techniques from domain adaptation [39, 11, 25] to
modify the data to match the training distribution. Domain adaptation by
itself brings its own limitations as there is no guarantee that it matches the
target distribution.

In recent years, however, Neural Radiance Field (NeRF) methods that can encode
complete 3D scenes have emerged. These NeRFs have the ability to generate
photorealistic and geometrically consistent, dense RGB-D images from novel
viewpoints with remarkable accuracy.

We propose to use the capabilities of NeRFs to generate high-quality RGB-D
images from novel viewpoints in order to enlarge and diversify existing real-
world datasets. This method is able to match the appearance distribution of
the source dataset while generating novel viewpoints that would be arduous and
costly to capture by recording a new dataset. Thus, we alleviate the need for
combining synthetic datasets and domain adaptation as well as making MDE
models more robust to images taken from out-of-distribution poses.

These benefits come with the extra advantage that the rendering of novel views
can be automated and is very fast, while only relying upon reasonable
computing power which is needed to train MDE networks anyway.

Moreover, this approach does not require designing a new model architecture,
as we show that our method makes existing MDE network architectures more
robust while keeping comparable performance or improving slightly with respect
to the baseline.

Our main contributions are as follows:

  * •

We propose a novel dataset augmentation scheme for Monocular Depth Estimation
that leverages NeRFs to augment a limited-scale dataset via scene
reconstruction and virtual view RGB-D image generation for supervised training
of data-driven MDE networks.

  * •

We conduct extensive experiments on multiple outdoor datasets, showing its
robust performance gains on the target dataset, while keeping comparable
performance on the source dataset which validates the effectiveness of our
approach, showcasing the benefits of our scheme to MDE networks.

##  2 Related work

Depth-Nerfacto | Depth-Nerfacto-Huge
---|---
RGB |
Depth |
Figure 3: Qualitative comparison of Depth-Nerfacto vs. Depth-Nerfacto-Huge
reconstruction on KITTI [12]. The figure shows reconstructions of training
images using both Depth-Nerfacto and Depth-Nerfacto-Huge. Depth-Nerfacto-Huge
outputs exhibit higher levels of detail and better accuracy in both the
generated RGB and depth images compared to Depth-Nerfacto.

#### Monocular Depth Estimation.

The task of Monocular Depth Estimation (MDE) involves predicting the depth
values for each pixel given a monocular 2D image. Due to the inherent
difficulty in predicting the correct 3D reprojection to a 2D image, well-
performing models employ sophisticated data-driven methods to infer the depth
values. Initially, CNN-based encoder-decoder architectures [24, 9] that
interpreted the MDE task as a pure regression task were employed. With the
introduction of transformers [38] and their subsequent introduction into
Computer Vision as vision transformers (ViT) [8], methods that employed a
combination of CNNs and ViTs emerged, such as [6, 15, 16]. AdaBins [6] is a
popular example of this class of models which interprets MDE as an ordinal
regression problem similar to [10] but with the addition of learning adaptive
bin centers and sizes. For this, it includes a ViT [8] into the CNN itself. In
contrast to AdaBins [6], DepthFormer [16] treats MDE as a pure regression
problem and employs the ViT [8] to directly encode images, purportedly
exploiting the ViT’s strengths. Additionally, Li _et al_. introduce the
hierarchical aggregation and heterogeneous interaction (HAHI) module, to
enhance ViT features and leverage the affinity between ViT and convolution
features [16]. BinsFormer [15] follows the idea of combining ordinal
regression on binned depth and transformer networks. We do not develop a novel
MDE architecture, but instead, use AdaBins [6], DepthFormer [16], and
BinsFormer [15] to show the transferability and versatility of our method.

#### Neural Radiance Fields (NeRFs).

NeRFs are a scene representation method introduced in the seminal work [18] by
Mildenhall  _et al_. Using a sparse set of input views, they optimize an
underlying volumetric scene function, taking advantage of a differentiable
rendering pipeline. Novel views as well as depth data can then be synthesized
by querying the network with novel camera poses. Nerfstudio [37] is a
framework that facilitates the design, training, and testing of a wide variety
of NeRF-based architectures through a modular approach. Nerfacto is an
extension of the original NeRF developed by the Nerfstudio team. This model
leverages the modularity of Nerfstudio to introduce new features proposed in
recent NeRF literature such as camera pose refinement, per-image appearance
conditioning, proposal sampling, scene contraction, and hash encoding. This
combination of features leads Nerfacto to achieve near-SOTA novel view quality
at a significantly faster speed than competitors such as MipNeRF [3]. Among
further extensions of Nerfacto is depth-Nerfacto, extending this model with
depth supervision, by providing a new loss term as proposed by Deng _et al_.
[7]. Finally, Nerfacto-huge is the largest in a series of Nerfacto models
offering different variants with different model sizes and capacities,
offering the highest fidelity scene representations among all Nerfacto
variants. When choosing an appropriate NeRF variant in our work, we once again
leverage the modularity and customizability exposed by Nerfstudio to combine
the aforementioned extensions of Nerfacto to benefit from both their
advantages.

In the spirit of using NeRFs not only for novel view synthesis of RGBs but
also other properties, MuVieNeRF [42] extends Neural Radiance Fields to
surface normal prediction, shading estimation, edges detection, keypoints
detection, and semantic labeling. The authors argue that exploiting the pose
data for predictions leads to better results than when using traditional CNNs.

Method | $\delta_{1}\uparrow$ | $\delta_{2}\uparrow$ | $\delta_{3}\uparrow$ |  REL $\downarrow$ |  SQ. REL $\downarrow$ |  RMS $\downarrow$ |  RMS LOG $\downarrow$
---|---|---|---|---|---|---|---
AdaBins | 0.561 | 0.851 | 0.933 | 0.224 | 2.201 | 8.027 | 0.328
+Reconstructed | 0.658 | 0.879 | 0.944 | 0.204 | 2.056 | 7.397 | 0.292
+Interpolated | 0.681 | 0.874 | 0.940 | 0.200 | 2.156 | 7.389 | 0.286
+Angled $\pm 3^{\circ}$ | 0.632 | 0.855 | 0.929 | 0.215 | 2.237 | 7.835 | 0.313
DepthFormer | 0.545 | 0.885 | 0.958 | 0.230 | 1.920 | 7.776 | 0.313
+Reconstructed | 0.607 | 0.907 | 0.969 | 0.199 | 1.723 | 7.730 | 0.275
+Interpolated | 0.669 | 0.926 | 0.975 | 0.184 | 1.542 | 6.755 | 0.252
+Angled $\pm 3^{\circ}$ | 0.602 | 0.909 | 0.970 | 0.199 | 1.711 | 7.286 | 0.274
BinsFormer | 0.571 | 0.901 | 0.966 | 0.208 | 1.782 | 7.547 | 0.289
+Reconstructed | 0.702 | 0.924 | 0.973 | 0.183 | 1.598 | 6.714 | 0.249
+Interpolated | 0.707 | 0.928 | 0.975 | 0.181 | 1.556 | 6.563 | 0.244
+Angled $\pm 3^{\circ}$ | 0.680 | 0.925 | 0.975 | 0.185 | 1.573 | 6.866 | 0.251
Table 1: AdaBins, DepthFormer, and BinsFormer trained on augmented KITTI
dataset evaluated on the Waymo dataset. We showcase the performance of AdaBins
[6], DepthFormer [16], and BinsFormer [15] models trained on the KITTI Eigen
[9, 12] train split and compare it to the NeRFmented (Ours) versions of the
same models evaluated against the unseen Waymo Open Dataset. Our method
greatly increases the robustness to unseen data, as demonstrated by the
increased performance of NeRFmented models on the Waymo dataset in comparison
to the baseline models which were trained without introducing novel views. The
best results are in bold dark green. Second best results are underlined light
green.

#### Data augmentation.

Data augmentation is a well-established regularization technique in Machine
Learning that generally enables the training of increasingly complex models by
applying software-based label-preserving transforms to scarce data. These
augmentations for Convolutional Neural Networks (CNNs) and derived models
started off with simple transformations such as random translations,
rotations, or flips [32, 13]. Recent advancements in generative modeling have
enabled the creation of photo-realistic synthetic data through increasingly
sophisticated data augmentation techniques. These methods are based on
Generative Adversarial Networks (GANs) [27, 5] or diffusion models [2, 1].
Sandfort _et al_. employ CycleGAN [43], a GAN that enforces cycle-consistency,
to augment images for medical CT segmentation tasks [27] and successfully
improve the performance even for out-of-distribution samples. While Rajpal _et
al_. show in [21] that it is possible to improve monocular depth estimation
performance on real-world data using a fully synthetic custom dataset, we aim
to show that real-world datasets can be extended synthetically using NeRFs to
improve the performance. Yen-Chen _et al_. [40] use NeRFs for robotic action
planning by building a 3D scene representation and subsequently using novel
view synthesis of orthographic images. The authors are able to show that
compared to images produced by physical cameras, these synthetic orthographic
images allow for better pixel-wise action values correlated with picking and
placing success of objects by the robot.

Mancini _et al_. [17] argue for the need for generalization of MDE networks
for autonomous mobile robots, to “guarantee robust operation in real-world
scenarios”. To achieve this, they train on two synthetic datasets, one urban
environment and one forest environment. They then test the models trained on
the synthetic datasets on KITTI, where they achieve results comparable to
Eigen _et al_. [9]. They also test on two datasets they collected themselves
to test generalization, one is of countryside roads, and one is of a forest.
They find that the network trained on both urban and forest environments
performs better than networks trained only on one of the scenarios.

Bauer _et al_. [4] use novel view synthesis to improve depth prediction. They
use three blocks: one network to predict depth, a block that transforms views
from one direction to another, by using the available depth data, and one
network which reconstructs the image obtained from the previous block, since
it is imperfect. They manage to beat the state of the art in one metric on the
KITTI dataset.

##  3 Methods

We propose a dataset augmentation technique for Monocular Depth Estimation
which is, to the best of our knowledge, novel. We leverage the fact that many
real-world image datasets have a sequence-like structure, meaning that the
same environment is captured from many slightly different viewpoints.
Furthermore, pose data for each frame is often either provided or can be
inferred through structure-from-motion, for example using COLMAP [29, 28].
Using this data, we train NeRFs to encode 3D scenes and subsequently render
novel RGB-D images from unobserved poses.

We apply our proposed augmentation method to the KITTI Eigen training split
[12, 9] and create a ”NeRFmented” version of it to show the diversity of
scenes to which our method can be applied. We train the off-the-shelf MDE
models AdaBins [6], DepthFormer [16], and BinsFormer [15] on the NeRFmented
dataset and evaluate them on the original dataset against their vanilla-
trained model to show the transferability between different architectures.
Finally, we evaluate the NeRFmented models vs the vanilla-trained models on a
different dataset that the models have not seen during training to show the
added robustness and diversity that our method introduces. The ablation
dataset is the Waymo Open Dataset [35].

###  3.1 Proposed pipeline

For each scene in the source dataset, we train a separate NeRF using RGB-D
images and their corresponding poses to encode a 3D representation. To ensure
a high reconstruction quality for each NeRF, we withhold a small part of the
input data from the training. Once the training has concluded, we use the
withheld data as a small validation split to potentially filter out scenes
with subpar image- or depth reconstruction quality. We use the retained NeRFs
to subsequently render RGB-D images from novel viewpoints, which are typically
obtained by perturbing the original poses either through translation or
rotation. Finally, we combine the RGB-D images from our novel views with the
source dataset to form an augmented dataset and train the downstream monocular
depth estimation network on it. The overall pipeline is depicted in Figure 2.

###  3.2 Scene representation model

For the NeRFs of our pipeline we use the off-the-shelf Nerfstudio framework
[37], which provides convenient access to various NeRF extensions. Because our
method requires high-quality depth reconstruction and image reconstruction, we
base our model on a combination of the depth-Nerfacto and Nerfacto-huge
architectures, which were both introduced by Tancik _et al_. in [37]. This
way, we benefit from depth-supervision that is included in the depth-Nerfacto
training while increasing the learning capacity of the network by replacing
the MLP of depth-Nerfacto with the larger MLP of Nerfacto-huge. Figure 3 shows
the qualitative performance gain of our depth-Nerfacto-huge compared to the
vanilla depth-Nerfacto. We implement a filtering step which eliminates NeRFs
that fail to meet a specified dataset-dependent reconstruction quality
standard. To achieve this, we utilize the trained NeRF for each scene to
render the RGB-D images from the poses of the small withheld validation split
and compare them to the corresponding ground-truth RGB-D images. This involves
computing the absolute relative error to assess the depth reconstruction
quality and the LPIPS score [41] to evaluate the image reconstruction quality.
We define dataset-specific thresholds for both metrics and any NeRF that falls
below these thresholds is discarded.

RGB |
---|---
AdaBins [6] |
NeRFmented AdaBins [6] |
DepthFormer [16] |
NeRFmented DepthFormer [16] |
Figure 4: Qualitative Results on the Waymo [36] dataset. We show the performance of NeRFmentation (Ours) compared to vanilla-trained AdaBins [6] and DepthFormer [16] leading to improved predictions by using NeRF augmented data for introducing additional viewpoints in the training images. Both the NeRFmented models and vanilla-trained models have been trained on the KITTI Eigen [9, 12] train split and evaluated on the unseen Waymo Open Dataset. Color scale: 0 (purple) to 80 meters (yellow). RGB |  AdaBins [6] |  NeRFmented AdaBins [6]
---|---|---
|  |
|  |
|  |
Figure 5: Qualitative results on the Waymo [36] dataset, focusing on close-up
details. We show the qualitative close-ups of the performance of the vanilla-
trained AdaBins [6] vs our proposed NeRFmented AdaBins, demonstrating the
capability of our method to recover fine grain details in the prediction that
the baseline is not able to predict. Color scale: 0 (purple) to 80 meters
(yellow).

##  4 Experiments

###  4.1 Datasets

#### KITTI [12].

The KITTI dataset is an outdoor dataset designed for diverse autonomous
driving scenarios, featuring stereo-image pairs and annotated LiDAR ground-
truth depth data. The KITTI Eigen split [9] is a popular split of the dataset
for monocular depth estimation with approximately 23,000 stereo-image pairs
for training which are separated into 32 scenes for training and validation.
The corresponding validation split consists of 888 stereo-image pairs. The
test split contains 697 images from 29 different scenes.

#### Waymo Open Dataset [35].

The Waymo Open Dataset (Waymo) dataset is a large-scale outdoor dataset
collected and developed by Waymo LLC for various autonomous driving tasks. The
dataset incorporates data from five high-resolution pinhole cameras and five
LiDARs and other sensors, which offers a wider range of camera viewpoints,
denser depth maps, and a more thorough overview of the surroundings compared
to datasets such as KITTI [12]. This enables more robust training of models
used for autonomous driving tasks. To evaluate the robustness of our
NeRFmented models, we test them against the Waymo test dataset. We choose to
use only the subset of the test set coming from the front, front-left, and
front-right facing cameras. We do this to avoid too large of a domain shift
from the standard MDE datasets based on images taken by forward-facing
cameras, such as KITTI.

###  4.2 Monocular Depth Estimation models

To show that our augmentation technique is model-agnostic, we show it in
combination with three popular MDE models: AdaBins [6], DepthFormer [16], and
BinsFormer [15]. We train and evaluate using the Monocular-Depth-Estimation-
Toolbox [14], an open-source monocular depth estimation toolbox aimed at
collecting several state-of-the-art MDE models into a single environment. Due
to computational constraints, we can not use the original batch size used for
DepthFormer and BinsFormer. Hence, we decrease the learning rate and increase
the number of iterations accordingly.

For the evaluation of the MDE models, we use the same metrics as in [4]. For
their definitions we refer to the supplementary material.

###  4.3 Training of NeRFs

We train our depth-Nerfacto-huge model on each scene of the KITTI Eigen [12,
9] train split and retain 10 % of the training data as a validation split. As
we experimentally find that the reconstruction quality of NeRFs drastically
reduces as the size of a scene increases, the varying size of the scenes in
the KITTI dataset requires splitting them into sub-scenes. We do not want to
decrease the scene size too much though, as this would increase the
computational complexity and bring no additional gain when rendering novel
views from challenging poses. In our case, we ensure that no pose in each sub-
scene should be more than 50m away from the first pose in that sub-scene to
solve the mentioned trade-off. For each sub-scene, we train a separate NeRF.

For each NeRF, we tune the weight of the depth loss and whether the poses are
refined or not by running a grid search for 10,000 steps. We observe that when
using NeRFs for 3D reconstruction, there is a strong trade-off between RGB
quality and accuracy of the geometry, which in turn affects the accuracy of
the depth maps. Since we require both to be of high quality, we use the small
validation split that we withheld from the training and re-render the RGB-D
images for each of their poses using all NeRFs from our grid search. Then, we
select the NeRF that minimizes a trade-off validation measure

$T_{\text{RGBD}}=\sqrt{\alpha\cdot\text{Abs. Rel}^{2}+\mathrm{LPIPS}^{2}}$ (1)

where we use the absolute relative error of the reprojected depth maps as a
quality measure of the geometry and the LPIPS score [41] as a quality measure
of the RGB reconstruction. We set $\alpha=10$ for this experiment.

Finally, we continue training the best run of the hyperparameter tuning of
each NeRF until we reach 30,000 iterations. Then, we apply the filtering step
described in Section 3.2 using the same validation split where we discard all
sub-scenes and their corresponding NeRF, where the NeRF did not achieve an
LPIPS score of less than $0.22$ or an absolute relative error of less than
$0.05$ on the 3D reconstructions. We choose the threshold for LPIPS
empirically by observing which value produces minimally visually acceptable
RGB images. The threshold for depth absolute relative error is chosen because
it is an approximate lower bound of state-of-the-art MDE networks. We filter
any scenes with a higher score so as not to introduce extraneous noise into
the training dataset.

Figure 6 shows the images reconstructed by NeRFs alongside ground-truth images
present in the KITTI dataset. Apart from a low LPIPS score, visual inspection
of the reconstructed RGB images also suggests that they are of comparable
quality to the original. Additionally, the reconstructed depth maps are dense,
whereas the ground-truth ones are sparse. The low absolute relative error of
the reconstructed depth maps indicates a high agreement between the original
and reconstructed depth maps as well.

###  4.4 Novel view synthesis

We propose and evaluate three data augmentation strategies: (i)
Reconstruction: Re-rendering the exact train poses used to train the NeRF.
This completes the sparse depth from the source dataset. (ii) Interpolated:
Creating novel poses by interpolating between pairs of training poses as these
poses inject additional data while benefiting from good depth and RGB
supervision. (iii) Angled: Rendering two additional views for each train pose,
rotated $\pm 3^{\circ}$ horizontally as this injects the most diverse data
while simulating a plausible motion performed by a driving car. For the KITTI
Eigen train split [9, 12] this results in dataset sizes of (i) 29,836 (+28%),
(ii) 29,744 (+28%), and (iii) 32,132 images (+39%).

The motivation behind these data augmentation strategies is to evaluate
whether it is sufficient to use NeRFs as depth completion networks or whether
certain novel views of training scenes can improve performance on evaluation
datasets.

GT |
---|---
Ours |
GT |
Ours |
Figure 6: Qualitative NeRF reconstruction results on KITTI [12]. We show
original images from the KITTI dataset and the corresponding images generated
using the trained and filtered NeRFs for the same camera poses. It is evident
that reconstructed RGB images are very close to their real counterparts. The
NeRFs also complete the sparse ground truth depth maps.

###  4.5 Training on KITTI

We train the described off-the-shelf MDE networks on the original KITTI Eigen
[12, 9] train split and on our NeRFmented version of it and compare the
results based on the standard MDE metrics. Our models are trained from scratch
using the same hyper-parameters as described in the corresponding papers to
show the influence of our augmentation method compared to the baseline without
additional factors contributing to the performance change.

Among the three MDE architectures trained with NeRFmented variants of the
KITTI Eigen train split, in Table 2 we see that the DepthFormer models surpass
the baseline model trained on the standard KITTI Eigen train split. The angled
$\pm 3^{\circ}$ NeRFmentation yields an improvement in the majority of metrics
we evaluate for, breaking even for the rest. The interpolated NeRFmentation is
not far behind, yielding better results for 3/7 metrics while the
reconstructed NeRFmentation is behind both, suggesting that the introduction
of diverse poses through angle variation and interpolation introduces novel
information that the MDE visibly benefits from.

Augmentation | $\delta_{1}$ | $\delta_{2}$ | $\delta_{3}$ | REL | SQ. REL | RMS | $\underset{\textrm{log}}{\textrm{RMS}}$
---|---|---|---|---|---|---|---
| $\uparrow$ | $\uparrow$ | $\uparrow$ | $\downarrow$ | $\downarrow$ | $\downarrow$ | $\downarrow$
AdaBins | 0.964 | 0.995 | 0.999 | 0.058 | 0.190 | 2.360 | 0.088
+Reconstructed | 0.961 | 0.995 | 0.999 | 0.060 | 0.204 | 2.434 | 0.092
+Interpolated | 0.960 | 0.995 | 0.999 | 0.61 | 0.204 | 2.408 | 0.092
+Angled $\pm 3^{\circ}$ | 0.961 | 0.995 | 0.999 | 0.061 | 0.204 | 2.412 | 0.093
DepthFormer | 0.975 | 0.997 | 0.999 | 0.052 | 0.158 | 2.143 | 0.079
+Reconstructed | 0.974 | 0.997 | 0.999 | 0.052 | 0.152 | 2.109 | 0.098
+Interpolated | 0.976 | 0.997 | 0.999 | 0.052 | 0.150 | 2.072 | 0.079
+Angled $\pm 3^{\circ}$ | 0.974 | 0.997 | 0.999 | 0.051 | 0.150 | 2.072 | 0.078
BinsFormer | 0.974 | 0.997 | 0.999 | 0.052 | 0.151 | 2.098 | 0.079
+Reconstructed | 0.974 | 0.997 | 0.999 | 0.054 | 0.155 | 2.104 | 0.081
+Interpolated | 0.973 | 0.997 | 0.999 | 0.055 | 0.157 | 2.125 | 0.081
+Angled $\pm 3^{\circ}$ | 0.973 | 0.997 | 0.999 | 0.055 | 0.159 | 2.164 | 0.082
Table 2: Comparison of performances on the KITTI Eigen [12, 9] test split. We
showcase the performance of AdaBins [6], DepthFormer [16] and BinsFormer [15]
models trained on the KITTI Eigen [12, 9] train split and compare it to the
NeRFmented versions of the same models, using different novel view generation
strategies. NeRFmented models stay very close to baseline performance. The
best results are in bold dark green. The second best results are underlined
light green.

###  4.6 Zero-shot dataset transfer to Waymo

Evaluation on a different dataset provides a good insight into generalization
capabilities. In this comparison, the models are strictly trained on KITTI and
KITTI-based NeRFmented images. Table 1 shows that the NeRFmented runs deliver
massive uplifts across the board, beating baseline performance in a large
majority of metrics for every single augmentation strategy. Notably,
interpolation outperforms reconstruction and angling most of the time, likely
as it strikes a good balance between providing novel viewpoints while staying
close to train data distribution.

The quantitative success is mirrored by the qualitative results displayed in
Figures 4 and 5. As a result of the densification of the depth maps, the MDE
architecture gains supervision in the upper part of the image. This effect can
be seen in the increased clarity in the upper parts of the predicted depth
maps. Additionally, an increase in fine detail prediction in street signs,
stoplights, street lamps, and foliage is clearly visible. Car outlines become
sharper and background building details are much improved as well.

###  4.7 Ablation on NeRF-generated KITTI dataset

To demonstrate that NeRFmented MDE architectures generalize better on
disturbed non-ideal test sets, we generate our own dataset by using novel
views. More specifically, we train NeRFs on 111 sub-scenes from the KITTI
Eigen test split. Then to filter out subpar scene representations, we set the
LPIPS threshold to 0.3 and absolute relative error to 0.05, leaving 15 scenes
for rendering. To render novel views, poses are disturbed randomly in the x
and y directions and rotated randomly along each axis. This yields a test set
with 1,218 dense RGB-D pairs.

In table 3 we see significant improvement in all metrics for all evaluated
depth architectures using NeRFmented data. The trend of interpolation
resulting in the best performance can be observed here as well.

Augmentation | $\delta_{1}$ | $\delta_{2}$ | $\delta_{3}$ | REL | SQ. REL | RMS | $\underset{\textrm{log}}{\textrm{RMS}}$
---|---|---|---|---|---|---|---
| $\uparrow$ | $\uparrow$ | $\uparrow$ | $\downarrow$ | $\downarrow$ | $\downarrow$ | $\downarrow$
AdaBins | 0.761 | 0.961 | 0.988 | 0.183 | 0.987 | 4.248 | 0.211
+Reconstructed | 0.901 | 0.985 | 0.995 | 0.128 | 0.645 | 3.424 | 0.143
+Interpolated | 0.903 | 0.986 | 0.995 | 0.121 | 0.603 | 3.359 | 0.139
+Angled $\pm 3^{\circ}$ | 0.892 | 0.983 | 0.994 | 0.132 | 0.682 | 3.541 | 0.147
DepthFormer | 0.791 | 0.978 | 0.992 | 0.171 | 0.840 | 3.804 | 0.193
+Reconstructed | 0.917 | 0.989 | 0.996 | 0.112 | 0.504 | 3.140 | 0.130
+Interpolated | 0.922 | 0.989 | 0.996 | 0.106 | 0.457 | 3.009 | 0.126
+Angled $\pm 3^{\circ}$ | 0.917 | 0.989 | 0.996 | 0.112 | 0.512 | 3.098 | 0.129
BinsFormer | 0.807 | 0.979 | 0.992 | 0.168 | 0.831 | 3.743 | 0.189
+Reconstructed | 0.924 | 0.990 | 0.997 | 0.111 | 0.492 | 3.045 | 0.128
+Interpolated | 0.925 | 0.990 | 0.996 | 0.105 | 0.459 | 2.994 | 0.125
+Angled $\pm 3^{\circ}$ | 0.914 | 0.988 | 0.996 | 0.118 | 0.562 | 3.235 | 0.133
Table 3: Comparison of performances on the disturbed densified KITTI dataset.
This table shows the performance of baseline and NeRFmented models evaluated
on a dataset of images generated using NeRFs trained on scenes not part of the
KITTI Eigen [9, 12] train split. Our method improves upon baseline performance
independent of the architecture used. The best results are in bold dark green.
The second best results are underlined light green.

##  5 Discussion

In this paper, we explore the idea of increasing the accuracy and robustness
of monocular depth estimators by synthesizing new data from KITTI and
introducing new views to train on. Our experiments show that for evaluation on
sparse depth, we are able to achieve a performance close or better to the one
when training on the vanilla dataset. For evaluation on disturbed densified
depth we achieve better performance with all of the MDE networks we used as
baselines. This shows that NeRFmented images are desirable if the sharpness of
the depth maps is of interest. We also find that introducing new views greatly
increases robustness since performance is much better when testing on an
unseen dataset than without.

#### Limitations.

Our proposed method increases the accuracy and robustness of monocular depth
estimators by remedying a lack of dense depth maps, limited viewing angles,
and a general lack of scene diversity. However, given a dataset already rich
with dense depth maps, varied viewing angles, and diverse scenes,
NeRFmentation shows no improvement in comparison with baseline methods. The
NYU-Depth V2 (NYU) dataset [31] is an example of this. It is an indoor dataset
of various commercial and residential scenes captured using Microsoft Kinect.
As such, the depth maps in the NYU dataset are dense. Furthermore, the dataset
was recorded by manually panning the Kinect sensor. As a result, the scenes
are captured from various viewpoints. We conjecture that the dense depth maps
and varied poses constrain NeRFmentation’s potential for performance and
robustness improvement. We provide experimental evidence for this in the
supplementary material.

#### Future work.

NeRFs are good at compressing and interpolating data. However, they lack the
ability to model regions that were unobserved during training. Thus, when we
render novel views where parts of the image were unobserved during the
training process, no constructive prediction can be made by the NeRF. As such
these parts of the image are filled with noise, unnecessarily punishing the
MDE. A masking strategy based on knowledge of observed areas of the scene is
delivered in the supplementary material. Dynamic objects such as cars and
people are incorrectly reconstructed due to the static world assumption made.
Dynamic NeRF methods such as [20, 33, 19] could remedy this shortcoming,
allowing for even higher fidelity data synthesis.

## Acknowledgements

One of the authors is funded by 2023/00027/001 under the “Campaign financed by
the European Union Next Generation - Recovery Plan - Competent Ministry -
Spanish Government”.

## References

  * Azizi et al. [2023] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet.  Synthetic Data from Diffusion Models Improves ImageNet Classification, 2023.  arXiv:2304.08466 [cs].
  * Baranchuk et al. [2022] Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko.  Label-Efficient Semantic Segmentation with Diffusion Models, 2022.  arXiv:2112.03126 [cs].
  * Barron et al. [2021] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields, 2021.
  * Bauer et al. [2021] Zuria Bauer, Zuoyue Li, Sergio Orts-Escolano, Miguel Cazorla, Marc Pollefeys, and Martin R. Oswald.  NVS-MonoDepth: Improving Monocular Depth Prediction with Novel View Synthesis.  In _2021 International Conference on 3D Vision (3DV)_ , pages 848–858, London, United Kingdom, 2021. IEEE.
  * Besnier et al. [2019] Victor Besnier, Himalaya Jain, Andrei Bursuc, Matthieu Cord, and Patrick Pérez.  This dataset does not exist: training models from generated images, 2019.  arXiv:1911.02888 [cs, eess].
  * Bhat et al. [2021] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka.  AdaBins: Depth Estimation using Adaptive Bins.  In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4008–4017, 2021.  arXiv:2011.14141 [cs].
  * Deng et al. [2022] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan.  Depth-supervised NeRF: Fewer Views and Faster Training for Free.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12872–12881, New Orleans, LA, USA, 2022. IEEE.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2021.
  * Eigen et al. [2014] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network, 2014.
  * Fu et al. [2018] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao.  Deep ordinal regression network for monocular depth estimation.  _CoRR_ , abs/1806.02446, 2018.
  * Gatys et al. [2016] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge.  Image style transfer using convolutional neural networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * Geiger et al. [2013] A Geiger, P Lenz, C Stiller, and R Urtasun.  Vision meets robotics: The KITTI dataset.  _The International Journal of Robotics Research_ , 32(11):1231–1237, 2013.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  ImageNet Classification with Deep Convolutional Neural Networks.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2012.
  * Li [2022] Zhenyu Li.  Monocular depth estimation toolbox.  https://github.com/zhyever/Monocular-Depth-Estimation-Toolbox, 2022.
  * Li et al. [2022] Zhenyu Li, Xuyang Wang, Xianming Liu, and Junjun Jiang.  Binsformer: Revisiting adaptive bins for monocular depth estimation, 2022.
  * Li et al. [2023] Zhenyu Li, Zehui Chen, Xianming Liu, and Junjun Jiang.  Depthformer: Exploiting long-range correlation and local information for accurate monocular depth estimation.  _Machine Intelligence Research_ , pages 1–18, 2023.
  * Mancini et al. [2017] Michele Mancini, Gabriele Costante, Paolo Valigi, Thomas Ciarfuglia, Jeffrey Delmerico, and Davide Scaramuzza.  Toward domain independence for learning-based monocular depth estimation.  _IEEE Robotics and Automation Letters_ , PP:1–1, 2017.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, 2020.  arXiv:2003.08934 [cs].
  * Park et al. [2023] Sungheon Park, Minjung Son, Seokhwan Jang, Young Chun Ahn, Ji-Yeon Kim, and Nahyup Kang.  Temporal interpolation is all you need for dynamic neural radiance fields, 2023.
  * Pumarola et al. [2020] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes, 2020.
  * Rajpal et al. [2023] Aakash Rajpal, Noshaba Cheema, Klaus Illgner-Fehns, Philipp Slusallek, and Sunil Jaiswal.  High-Resolution Synthetic RGB-D Datasets for Monocular Depth Estimation, 2023.  arXiv:2305.01732 [cs].
  * Ranftl et al. [2020] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer, 2020.
  * Richter et al. [2016] Stephan R. Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun.  Playing for data: Ground truth from computer games, 2016.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015_ , pages 234–241, Cham, 2015. Springer International Publishing.
  * Sakaridis et al. [2020] Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  Map-guided curriculum domain adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * Sakaridis et al. [2021] Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  ACDC: The adverse conditions dataset with correspondences for semantic driving scene understanding.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * Sandfort et al. [2019] Veit Sandfort, Ke Yan, Perry J. Pickhardt, and Ronald M. Summers.  Data augmentation using generative adversarial networks (CycleGAN) to improve generalizability in CT segmentation tasks.  _Scientific Reports_ , 9(1):16884, 2019.  Number: 1 Publisher: Nature Publishing Group.
  * Schonberger and Frahm [2016] Johannes L. Schonberger and Jan-Michael Frahm.  Structure-from-Motion Revisited.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4104–4113, Las Vegas, NV, USA, 2016. IEEE.
  * Schönberger et al. [2016] Johannes L. Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys.  Pixelwise View Selection for Unstructured Multi-View Stereo.  In _Computer Vision – ECCV 2016_ , pages 501–518, Cham, 2016. Springer International Publishing.
  * Shah et al. [2017] Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor.  Airsim: High-fidelity visual and physical simulation for autonomous vehicles, 2017.
  * Silberman et al. [2012] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  In _Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12_ , pages 746–760. Springer, 2012.
  * Simard et al. [2003] P.Y. Simard, D. Steinkraus, and J.C. Platt.  Best practices for convolutional neural networks applied to visual document analysis.  In _Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings._ , pages 958–963, 2003.
  * Song et al. [2023] Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger.  Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields, 2023.
  * Straub et al. [2019] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe.  The Replica dataset: A digital replica of indoor spaces.  _arXiv preprint arXiv:1906.05797_ , 2019.
  * Sun et al. [2020a] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov.  Scalability in perception for autonomous driving: Waymo open dataset.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020a.
  * Sun et al. [2020b] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov.  Scalability in Perception for Autonomous Driving: Waymo Open Dataset.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2443–2451, Seattle, WA, USA, 2020b. IEEE.
  * Tancik et al. [2023] Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa.  Nerfstudio: A Modular Framework for Neural Radiance Field Development.  In _Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Proceedings_ , pages 1–12, 2023.  arXiv:2302.04264 [cs].
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2017.
  * Yang and Soatto [2020] Yanchao Yang and Stefano Soatto.  Fda: Fourier domain adaptation for semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Yen-Chen et al. [2022] Lin Yen-Chen, Pete Florence, Andy Zeng, Jonathan T. Barron, Yilun Du, Wei-Chiu Ma, Anthony Simeonov, Alberto Rodriguez Garcia, and Phillip Isola.  MIRA: Mental Imagery for Robotic Affordances, 2022.  arXiv:2212.06088 [cs].
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric.  In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 586–595, Salt Lake City, UT, 2018. IEEE.
  * Zheng et al. [2023] Shuhong Zheng, Zhipeng Bao, Martial Hebert, and Yu-Xiong Wang.  Multi-task View Synthesis with Neural Radiance Fields, 2023.  arXiv:2309.17450 [cs].
  * Zhu et al. [2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _Computer Vision (ICCV), 2017 IEEE International Conference on_ , 2017.

Supplementary Material

##  Appendix A Evaluation Metrics for MDE

We utilize the MDE evaluation metrics used in [4]. The metrics are defined as
follows, given prediction $\hat{y}_{i}$ and ground truth value $y_{i}$:
Relative Error (REL): $\frac{1}{N}\sum^{N}_{i=1}\frac{\lvert
y_{i}-\hat{y}_{i}\rvert}{y_{i}}$
Squared Relative Error (SQ. REL): $\frac{1}{N}\sum^{N}_{i=1}\frac{\lvert
y_{i}-\hat{y}_{i}\rvert^{2}}{y_{i}}$
Root Mean Squared Error (RMSE):
$\sqrt{\frac{1}{N}\sum^{N}_{i=1}(y_{i}-\hat{y}_{i})^{2}}$
Log RMSE: $\sqrt{\frac{1}{N}\sum^{N}_{i=1}(\log y_{i}-\log\hat{y}_{i})^{2}}$
Threshold Accuracy ($\delta_{j}$):
$\max(\frac{y_{i}}{\hat{y}_{i}},\frac{\hat{y}_{i}}{y_{i}})=\delta<1.25^{j}$

##  Appendix B Qualitative comparison with state-of-the-art methods on the
KITTI and Waymo dataset

Figures B.1 and B.2 provide additional qualitative results compared to the
state-of-the-art methods AdaBins [6], Depthformer [16] and BinsFormer [15] on
the KITTI [12] and Waymo [36] datasets. Qualitative improvements on KITTI are
similar to those found on Waymo, our proposed method is able to predict
increased details in objects such as trees and street signs, which can be
especially noted in the top third part of the image.

##  Appendix C Qualitative comparison between the Reconstructed, Interpolated,
and Angled

We compare the three proposed data augmentation strategies in conjunction with
DepthFormer in Figure C.1. The differences between the different NeRFmented
depth predictions on the Waymo dataset show that they all perform similarly.
This suggests that the depth completion offered by NeRFs is the significant
driver behind the performance gains.

##  Appendix D Scene mesh of NeRFs trained on KITTI

Using the built-in poisson mesh export in Nerfstudio [37] we generate a mesh
of the NeRF scene. Figure D.1 shows a bird-eye’s view and a front-facing view
of a 50-meter KITTI sub-scene. This 3D representation allows us to verify that
the NeRF depth map predictions are sound and consistent.

|
---|---
Bird’s-eye view | Front-facing view
Figure D.1: Mesh generated with NeRF that was trained on KITTI [12] sub-scene.
One can see that the underlying scene geometry is accurately learned. The
bird’s-eye view shows what a single sub-scene looks like after the scene has
been split into smaller 50-meter sub-scenes.

##  Appendix E Discussion: NeRFmentation on NYU-Depth V2

Given the promising results produced using NeRFmentation on the task of MDE on
outdoor scenes, investigating the efficacy of this method on MDE on indoor
scenes becomes an obvious follow-up question. The tasks of indoor and outdoor
MDE have differing constraints, use cases, and available resources. The lack
of diverse poses in outdoor scene trajectories, for instance, may not
translate to indoor scenes. Here a trajectory may be captured by a device like
the Microsoft Kinect, which is able to operate with a wide range of viewing
angles and short physical distances between captured frames. This is because a
Kinect is not limited by the lanes of a road or the speed of a vehicle. Thus,
the inherent problems present in outdoor MDE and the resulting issue of data
scarcity that NeRFmentation aims to tackle may have limited value for the
indoor setting. Our following investigation applies the same experiments and
procedures previously followed for the outdoor KITTI Dataset to the indoor
NYU-Depth V2 Dataset. The performance of NeRFmentation is evaluated using
AdaBins [6] and DepthFormer [16] models.

#### NYU-Depth V2 Dataset [31].

NYU-Depth V2 is an indoor dataset consisting of various commercial and
residential scenes captured using Microsoft Kinect. The dataset comprises a
smaller subset with dense segmentation labels and preprocessed depth as well
as the much larger raw output of the Kinect sensor in the form of raw RGB,
depth, and accelerometer data. The raw data is used for the purposes of our
work, including a training set of approximately 120K samples and a test set of
654 samples [9]. As the RGB and depth images are not synchronized, they are
preprocessed using the toolbox provided by the dataset authors. The images are
additionally center cropped as described by Eigen _et al_. [9].

###  E.1 Training on NYU-Depth V2

The aforementioned MDE networks are trained on the original NYU-Depth V2 train
split and NeRFmented versions of it using the approach described in Section
4.5. The relevant results are provided in Table E.1. NeRFmentation appears to
hinder performance for both models and all novel view synthesis strategies.
Unlike in the case of KITTI, for NYU-Depth V2, the introduction of synthetic
images may not improve generalization to a more diverse set of poses since the
original data is not particularly lacking in pose diversity. Instead, the
introduction of noisy images that reduce the data quality becomes the main
factor determining the net outcome of the augmentation. Additionally, our
experiments on the KITTI dataset suggest that dense supervision coming from
NeRFmentation is a large factor in why NeRFmentation improves on vanilla
training. With NYU-Depth V2, the vanilla dataset already provides dense
supervision. Therefore, this advantage of NeRFmentation is lost when using the
NYU-Depth V2 dataset.

###  E.2 Zero-shot data transfer to Replica Dataset

To evaluate performance on a different indoor dataset, the Replica Dataset
[34] is used. Replica consists of high-quality 3D indoor scene
reconstructions, of which we choose the two validation scenes, and the one
test scene from Bauer _et al_. [4] as our validation split. We modify the
source code of the provided ReplicaRenderer and ReplicaViewer [34] to be able
to render customized sequences of synthetic RGB-D images following a natural
trajectory through each scene. NeRFmentation deteriorates generalization to
the unseen data distribution of synthetic images from the Replica dataset, as
seen in Table E.2 and Figure E.3. These results together with the NYU test set
results show that the training data becomes too noisy for potential
NeRFmentation benefits in generalization to unseen data distributions to be
possible.

###  E.3 Ablation on NeRF-generated NYU-Depth V2 dataset

Similar to the procedure applied in the case of KITTI-based NeRFmented MDE
architectures, we aim to demonstrate generalization to perturbed poses for the
NYU-Depth V2 dataset. The same pose disturbance scheme as in KITTI is used,
and the scheme is applied to NeRFs trained on the test scenes remaining after
the same filtration process as in KITTI experiments to eliminate subpar
scenes. To create a test set of approximately the same size as the original
NYU-Depth V2 test set, up to 10 frames per scene are selected at equal
intervals along the scene sequence, and new images are rendered using
corresponding perturbed poses. With this approach, the perturbed test set
consists of 700 images covering 73 scenes, capturing a diverse set of indoor
environments. As demonstrated in Table E.3, the NeRFmented MDE models surpass
the baseline AdaBins [6] models in the case of NeRFmentation using
reconstructed views. In contrast, the DepthFormer [16] models do not exhibit a
conclusive advantage for either the NeRFmented or the vanilla-trained models.

Augmentation | $\delta_{1}$ | $\delta_{2}$ | $\delta_{3}$ | REL | RMS | $\log_{10}$
---|---|---|---|---|---|---
| $\uparrow$ | $\uparrow$ | $\uparrow$ | $\downarrow$ | $\downarrow$ | $\downarrow$
AdaBins | 0.903 | 0.984 | 0.997 | 0.103 | 0.364 | 0.044
+Reconstructed | 0.877 | 0.977 | 0.994 | 0.118 | 0.414 | 0.049
+Interpolated | 0.898 | 0.983 | 0.996 | 0.106 | 0.375 | 0.045
+Angled $\pm 3^{\circ}$ | 0.884 | 0.981 | 0.996 | 0.112 | 0.387 | 0.047
DepthFormer | 0.921 | 0.989 | 0.998 | 0.096 | 0.339 | 0.041
+Reconstructed | 0.909 | 0.987 | 0.997 | 0.101 | 0.350 | 0.043
+Interpolated | 0.909 | 0.987 | 0.997 | 0.101 | 0.353 | 0.043
+Angled $\pm 3^{\circ}$ | 0.908 | 0.987 | 0.998 | 0.102 | 0.350 | 0.043
Table E.1: Comparison of performances on the NYU-Depth-v2 [31] dataset. This table shows the performance of AdaBins [6] and DepthFormer [16] models trained on the NYU-Depth V2 train split [9, 31] and compare it to the NeRFmented versions of the same models, using different novel view generation strategies. Both vanilla-trained and NeRFmented models were evaluated against the NYU-Depth V2 test split [9]. Our method yields reduced performance in most metrics for both models. The best results are in bold dark green. The second best results are underlined light green. Augmentation | $\delta_{1}$ | $\delta_{2}$ | $\delta_{3}$ | REL | RMS | $\log_{10}$
---|---|---|---|---|---|---
| $\uparrow$ | $\uparrow$ | $\uparrow$ | $\downarrow$ | $\downarrow$ | $\downarrow$
AdaBins | 0.512 | 0.921 | 0.982 | 0.271 | 0.821 | 0.103
+Reconstructed | 0.214 | 0.576 | 0.895 | 0.526 | 1.572 | 0.179
+Interpolated | 0.336 | 0.772 | 0.952 | 0.398 | 1.284 | 0.140
+Angled $\pm 3^{\circ}$ | 0.300 | 0.713 | 0.916 | 0.418 | 1.297 | 0.157
DepthFormer | 0.410 | 0.904 | 0.978 | 0.325 | 0.937 | 0.116
+Reconstructed | 0.150 | 0.606 | 0.943 | 0.526 | 1.540 | 0.175
+Interpolated | 0.115 | 0.542 | 0.908 | 0.582 | 1.634 | 0.190
+Angled $\pm 3^{\circ}$ | 0.109 | 0.486 | 0.908 | 0.604 | 1.752 | 0.196
Table E.2: Comparison of performances on the Replica Dataset [34]. Models trained on the NYU-Test V2 train split [9, 31] are compared to the NeRFmented (Ours) versions of the same models evaluated against a subset of the Replica Dataset [34], a 3D scene reconstruction dataset. Our method impairs performance considerably on unseen synthetic indoor sequences, invalidating the possibility of improved generalization. The best results are in bold dark green. The second best results are underlined light green. Augmentation | $\delta_{1}$ | $\delta_{2}$ | $\delta_{3}$ | REL | RMS | $\log_{10}$
---|---|---|---|---|---|---
| $\uparrow$ | $\uparrow$ | $\uparrow$ | $\downarrow$ | $\downarrow$ | $\downarrow$
AdaBins | 0.706 | 0.907 | 0.960 | 0.794 | 0.750 | 0.094
+Reconstructed | 0.732 | 0.912 | 0.964 | 0.622 | 0.691 | 0.083
+Interpolated | 0.315 | 0.804 | 0.939 | 1.235 | 1.159 | 0.147
+Angled $\pm 3^{\circ}$ | 0.311 | 0.804 | 0.936 | 1.347 | 1.129 | 0.148
DepthFormer | 0.773 | 0.933 | 0.974 | 0.882 | 0.650 | 0.077
+Reconstructed | 0.791 | 0.935 | 0.979 | 0.916 | 0.742 | 0.074
+Interpolated | 0.797 | 0.934 | 0.978 | 0.948 | 0.731 | 0.072
+Angled $\pm 3^{\circ}$ | 0.796 | 0.935 | 0.979 | 0.953 | 0.732 | 0.072
Table E.3: Comparison of performances on the perturbed NYU-Depth v2 [31] dataset. This table shows the performance of baseline and NeRFmented models evaluated on a dataset of images generated using NeRFs trained on scenes not part of the NYU-Depth v2 train split [31, 9]. Images are generated using random disturbations to original image poses. Our method does not yield an architecture-independent performance increase, rendering the robustness ablation for indoor MDEs inconclusive. The best results are in bold dark green. The second best results are underlined light green. GT |
---|---
Ours |
GT |
Ours |
Figure E.1: Qualitative NeRF reconstruction results on the NYU-Depth V2 Dataset [31] We show original images from the NYU-Depth V2 dataset and the corresponding images generated using the trained and filtered NeRFs for the same camera poses. The reconstructed RGB images appear very similar to their real counterparts. The reconstructed depth maps are also very close to the ground truth depth maps. However, there is some noise apparent on the edges of objects in the reconstructed depth maps. Color scale: 0 (purple) to 10 meters (yellow). RGB |
---|---
GT depth |
AdaBins [6] |
NeRFmented AdaBins [6] |
DepthFormer [16] |
NeRFmented DepthFormer [16] |
Figure E.2: Qualitative Results on the NYU-Depth V2 Dataset [31] We show the performance of NeRFmentation (Ours) compared to vanilla-trained AdaBins [6] and DepthFormer [16]. Both the NeRFmented models and vanilla-trained models have been trained on the NYU-Depth V2 train split and evaluated on the test split [9]. There is little discernible visual difference between the predictions of the vanilla-trained models and NeRFmented models. This is in line with the scores presented in Table E.1. Color scale: 1 (purple) to 6.5 meters (yellow). RGB |
---|---
GT Depth |
AdaBins [6] |
NeRFmented AdaBins [6] |
DepthFormer [16] |
NeRFmented DepthFormer [16] |
Figure E.3: Qualitative Results on the Replica Dataset [35]. We show the
performance of NeRFmentation (Ours) compared to vanilla-trained AdaBins [6]
and DepthFormer [16]. Both the NeRFmented models and vanilla-trained models
were trained on the NYU-Depth V2 train split [9, 31] and evaluated on a subset
of the unseen Replica Dataset [35]. The NeRFmented models produce higher noise
around edges of objects than the vanilla-trained models. This could be
attributed to the noise introduced by NeRF-generated training data, as
presented in Figure E.1. Color scale: 1 (purple) to 8 meters (yellow).

##  Appendix F Future Works: Masking

Due to the sparsity of depth maps of datasets that were recorded using a LiDAR
sensor such as KITTI, NeRFs that are trained on these datasets have to inter-
and extrapolate data that they have never seen before. Interpolating data for
a NeRF is mostly unproblematic. However, NeRFs struggle at the scene borders
of the training data as they have to extrapolate in these regions. Due to the
fact that the level of supervision approaches zero in these regions, the NeRF
learns to predict noise in these regions without any guarantee that the data
there is useful. In the best case, the data corresponds to the actual real-
world geometry, but in the worst case, it will predict values that could hurt
the overall training procedure and thus need to be taken care of. This issue
limits us from rendering more diverse poses for our NeRFmentation method, as
any strong rotations or extreme translations will lead to noisy, but dense
renders that we cannot mask out trivially without any extra steps.

In the following, we describe our proposed method to generate non-trivial
binary occupancy masks that can be used to mask out unseen regions of NeRF-
rendered depth maps during the training of a MDE network. These masks are
designed to prevent the network from getting penalized for predicting
incorrect values in pixels corresponding to these unseen regions.

#### Point cloud generation.

First, we need to obtain the 3D point cloud of each sub-scene $s$ in Cartesian
coordinates. For that we must first obtain the homogeneous point cloud
$\text{PC}_{s,h}$ using the following Equation (2):

$\displaystyle\forall
d_{k}\in\text{DM}_{\text{train},s}\subseteq\text{DM}_{\text{train}}\subseteq\mathcal{D}_{\text{train}},\quad
d_{k}\in\mathbb{R}^{U\times V},$ (2)
$\displaystyle\text{PC}_{s,h}=\left\\{P_{W,s,h}\in\mathbb{R}^{4}\mid d_{k}\ni
d_{k,uv}>0\right\\},$ where $\displaystyle
P_{W,s,h}=T_{C_{k}}^{W}\cdot\begin{bmatrix}(K_{C}^{-1}\cdot p_{k,uv})^{T}\cdot
d_{k,uv}\\\ d_{k,uv}\end{bmatrix},$ $\displaystyle
K_{C}^{-1}\in\mathbb{R}^{3\times 3},\quad T_{C_{k}}^{W}\in\mathbb{R}^{4\times
4},\quad\text{PC}_{s,h}\in\mathbb{R}^{N\times 4},$ $\displaystyle
p_{c,uk}=p_{c}(u,v)\in\mathbb{R},\quad N\leq U\cdot
V\cdot\lvert\text{DM}_{\text{train},s}\rvert.$

where $d_{k}$ is the k-th depth map of the training split
$\text{DM}_{\text{train},s}$ of sub-scene $s$ which is a subset of the
complete training dataset $\mathcal{D}_{\text{train}}$, and $P_{W,s,h}$ is the
4-dimensional homogeneous point in the world frame. For each sub-scene $s$, we
first unproject each valid 2D point of each sparse depth map $d_{k}$ that was
part of the NeRF training split $\text{DM}_{\text{train},s}$ into the camera
reference frame $C$ using homogeneous coordinates, the depth value $d_{k,uv}$
at every pixel index $u\in[0,U-1],v\in[0,V-1]$ and the inverse of the given
intrinsic camera matrix $K_{C}\in\mathbb{R}^{3x3}$. Then we transform each
homogeneous point into the world coordinate system $W$ using the given
extrinsic matrices $T_{C_{k}}^{W}\in\mathbb{R}^{4x4}$. After this step, the 3D
point cloud $\text{PC}_{s}$ is derived by dividing each point in
$\text{PC}_{s,h}$ by its fourth homogeneous coordinate

$\text{PC}_{s}=\frac{1}{\text{PC}_{s,h}[3]}\cdot\begin{bmatrix}\text{PC}_{s,h}[0]\\\
\text{PC}_{s,h}[1]\\\ \text{PC}_{s,h}[2]\end{bmatrix},$ (3)

where $\text{PC}_{s,h}[i]$ denotes the $i$-th element of the homogeneous point
cloud $\text{PC}_{s,h}$.

#### Mask generation.

When we generate an RGB-D image from a novel view, we store the pose used for
each RGB-D image and split it into a rotation matrix $R_{W}^{NV_{k}}$ and a
translation vector $t_{W}^{NV_{k}}$ to project the point cloud into the
reference frame of the novel view $NV$ and using the intrinsic camera matrix
$K_{C}$, we can then project the point-cloud into the image as a 2D projection
and set the value for each pixel to 1 if it is occupied by data from at least
one 3D point. This operation is the inverse of the point cloud generation
process. It is described in the following equation 4:

Depth |  |
---|---|---
RGB |  |
Mask |  |
Masked Depth |  |
Masked RGB |  |
Figure F.1: NeRF reconstruction results with 15 degrees rotation and occupancy
masks on KITTI [12]. We show novel views from KITTI that were generated by
rotating 15 degrees to the left or right, starting from a training pose. We
show the rendered, dense depth map and the rendered RGB image which clearly
show artifacts in the border regions. Using the occupancy masks, it is
possible to filter out these artifacts, but at the same time sparsify the
depth maps.

$\displaystyle\forall
NV_{k}\in\mathcal{D}_{\text{nerfmentation},s}\subseteq\mathcal{D}_{\text{nerfmentation}},$
(4) $\displaystyle\text{mask}_{NV_{k}}(u,v)=\mathds{1}\\{d_{NV_{k},uv}>0\\},$
where $\displaystyle
d_{NV_{k},h}=K_{C}\cdot(R_{W}^{NV_{k}}\cdot\text{PC}_{s}+t_{W}^{NV_{k}}),$
$\displaystyle K_{C}\in\mathbb{R}^{3\times 3},\quad
R_{W}^{NV_{k}}\in\mathbb{R}^{3\times 3},\quad
t_{W}^{NV_{k}}\in\mathbb{R}^{3},$
$\displaystyle\text{PC}_{s}\in\mathbb{R}^{N\times 3}.$

where $NV_{k}$ is the novel view which is defined by the rotation matrix
$R_{W}^{NV_{k}}\mathbb{R}^{3\times 3}$ and the translation vector
$t_{W}^{NV_{k}}\in\mathbb{R}^{3}$. The depth map of novel view $NV_{k}$
retrieved from the ground-truth point cloud $PC_{s}$ might have multiple depth
values per pixel $u,v$ which is irrelevant in our case since we only need to
consider occupancy which is unaffected by multi-occupancy.

For future work, we plan to investigate the impact of this proposed method.
Our proposed method sparsifies each RGB-D image again and excludes the sky, a
step that is already done during the evaluation on KITTI. As a result, we
expect this method to not yield a significant improvement over our proposed
vanilla NeRFmentation. However, we do expect substantial benefits for renders
captured from extreme poses. Since these renders would deviate strongly from
the data distribution of KITTI and would be too sparse for Waymo, we expect a
notable performance loss on KITTI and no gain on Waymo. As a result, our
extended method might not be beneficial for current datasets.

Figure F.1 shows rendered RGB-D images from challenging poses and how the
occupancy masks can effectively remove these artifacts.

RGB |
---|---
AdaBins [6] |
NeRFmented AdaBins [6] |
DepthFormer [16] |
NeRFmented DepthFormer [16] |
BinsFormer [15] |
NeRFmented BinsFormer [15] |
Figure B.1: Qualitative Results on the KITTI [12] dataset. We show the
performance of NeRFmentation (Ours Interpolated) compared to vanilla-trained
AdaBins [6], DepthFormer [16] and BinsFormer [15] leading to improved
predictions by using NeRF augmented data for introducing additional viewpoints
in the training images. Both the NeRFmented models and vanilla-trained models
have been trained on the KITTI Eigen [9, 12] train split and evaluated on test
split. Color scale: 0 (purple) to 80 meters (yellow).

RGB |
---|---
AdaBins [6] |
NeRFmented AdaBins [6] |
DepthFormer [16] |
NeRFmented DepthFormer [16] |
BinsFormer [15] |
NeRFmented BinsFormer [15] |
Figure B.2: Qualitative predictions on the Waymo [35] dataset. We show the
performance of NeRFmentation (Ours) compared to AdaBins [6], DepthFormer [16]
and BinsFormer [15]. The color scale goes from 0 (purple) to 80 meters
(yellow). Best viewed on a monitor zoomed in.

RGB | DepthFormer |  NeRFmented DepthFormer (Reconstructed) |  NeRFmented DepthFormer (Interpolated) |  NeRFmented DepthFormer (Angled)
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
Figure C.1: Qualitative DepthFormer [16] results on the Waymo [36] dataset,
focusing on close-up details. Greatly improved detail compared to the
baseline, but no apparent differences between reconstructed, interpolated, and
angled data augmentation strategies. Color scale: 0 (purple) to 80 meters
(yellow).