# SCADE: NeRFs from Space Carving with Ambiguity-Aware Depth Estimates

Mikaela Angelina Uy1,2 Ricardo Martin-Brualla2 Leonidas Guibas1,2 Ke Li2,3
1Stanford University 2Google 3Simon Fraser University

###### Abstract

Neural radiance fields (NeRFs) have enabled high fidelity 3D reconstruction
from multiple 2D input views. However, a well-known drawback of NeRFs is the
less-than-ideal performance under a small number of views, due to insufficient
constraints enforced by volumetric rendering. To address this issue, we
introduce SCADE, a novel technique that improves NeRF reconstruction quality
on sparse, unconstrained input views for in-the-wild indoor scenes. To
constrain NeRF reconstruction, we leverage geometric priors in the form of
per-view depth estimates produced with state-of-the-art monocular depth
estimation models, which can generalize across scenes. A key challenge is that
monocular depth estimation is an ill-posed problem, with inherent ambiguities.
To handle this issue, we propose a new method that learns to predict, for each
view, a continuous, multimodal distribution of depth estimates using
conditional _Implicit Maximum Likelihood Estimation_ (cIMLE). In order to
disambiguate exploiting multiple views, we introduce an original space carving
loss that guides the NeRF representation to fuse multiple hypothesized depth
maps from each view and distill from them a common geometry that is consistent
with all views. Experiments show that our approach enables higher fidelity
novel view synthesis from sparse views. Our project page can be found at
scade-spacecarving-nerfs.github.io.

Figure 1: SCADE Overview. We present SCADE, a novel technique for NeRF
reconstruction under _sparse, unconstrained views_ for in-the-wild indoor
scenes. We leverage on generalizable monocular depth priors and address to
represent the inherent ambiguities of monocular depth by exploiting our
ambiguity-aware depth estimates (left). Our approach accounts for
_multimodality_ of both distributions using our novel _space carving loss_
that seeks to _disambiguate_ and find the common mode to _fuse_ the
information between different views (middle). SCADE enables better photometric
reconstruction especially in highly ambiguous scenes, _e.g_. non-opaque
surfaces (right).

##  1 Introduction

Neural radiance fields (NeRF) [26] have enabled high fidelity novel view
synthesis from dozens of input views. Such number of views are difficult to
obtain in practice, however. When given only a small number of sparse views,
vanilla NeRF tends to struggle with reconstructing shape accurately, due to
inadequate constraints enforced by the volume rendering loss alone.

Shape priors can help remedy this problem. Various forms of shape priors have
been proposed for NeRFs, such as object category-level priors [14], and
handcrafted data-independent priors [27]. The former requires knowledge of
category labels and is not applicable to scenes, and the latter is agnostic to
the specifics of the scene and only encodes low-level regularity (e.g., local
smoothness). A form of prior that is both scene-dependent and category-
agnostic is per-view monocular depth estimates, which have been explored in
prior work [6, 34]. Unfortunately, monocular depth estimates are often
inaccurate, due to estimation errors and inherent ambiguities, such as albedo
vs. shading (cf. check shadow illusion), concavity vs. convexity (cf. hollow
face illusion), distance vs. scale (cf. miniature cinematography), etc. As a
result, the incorrect or inconsistent priors imposed by such depth estimates
may mislead the NeRF into reconstructing incorrect shape and produce artifacts
in the generated views.

In this paper, we propose a method that embraces the uncertainty and
ambiguities present in monocular depth estimates, by modelling a probability
distribution over depth estimates. The ambiguities are retained at the stage
of monocular depth estimation, and are only resolved once information from
multiple views are fused together. We do so with a principled loss defined on
probability distributions over depth estimates for different views. This loss
selects the subset of modes of probability distributions that are consistent
across all views and matches them with the modes of the depth distribution
along different rays as modelled by the NeRF. It turns out that this operation
can be interpreted as a probabilistic analogue of classical depth-based space
carving [19]. For this reason, we dub our method _Space Carving with
Ambiguity-aware Depth Estimates_ , or SCADE for short.

Compared to prior approaches of depth supervision [6, 34] that only supervise
the moments of NeRF’s depth distribution (rather than the modes), our key
innovation is that we supervise the modes of NeRF’s depth distribution. The
supervisory signal provided by the former is weaker than the latter, because
the former only constrains the value of an integral aggregated along the ray,
whereas the latter constrains the values at different individual points along
the ray. Hence, the supervisory signal provided by the former is 2D (because
it integrates over a ray), whereas the supervisory signal provided by the our
method is 3D (because it is point-wise) and thus can be more fine-grained.

An important technical challenge is modelling probability distributions over
depth estimates. Classical approaches use simple distributions with closed-
form probability densities such as Gaussian or Laplace distributions.
Unfortunately these distributions are not very expressive, since they only
have a single mode (known as “unimodal”) and have a fixed shape for the tails.
Since each interpretation of an ambiguous image should be a distinct mode,
these simple unimodal distributions cannot capture the complex ambiguities in
depth estimates. Naïve extensions like a mixture of Gaussians are also not
ideal because some images are more ambiguous than others, and so the number of
modes needed may differ for the depth estimates of different images. Moreover,
learning such a mixture requires backpropagating through E-M, which is
nontrivial. Any attempt at modifying the probability density to make it
capable of handling a variable number of modes can easily run into an
intractable partition function [10, 4, 33], which makes learning difficult
because maximum likelihood requires the ability to evaluate the probability
density, which is a function of the partition function.

Figure 2: Ambiguities from a single image. We show samples from our ambiguity-
aware depth estimates that is able to handle different types of ambiguities.
(Top) An image of a chair taken from the top-view. Without context of the
scene, it is unclear that it is an image of a chair. We show different samples
from our ambiguity-aware depth estimates that are able to capture different
degrees of convextiy. (Bottom) An image of a cardboard under bad lighting
conditions that capture the albedo vs shading ambiguity that is also
represented by our different samples.

To get around this conundrum, we propose representing the probability
distribution with a set of samples generated from a neural network. Such a
distribution can be learned with a conditional GAN; however, because GANs
suffer from mode collapse, they cannot model multiple modes [12] and are
therefore unsuited to modelling ambiguity. Instead, we propose leveraging
conditional Implicit Maximum Likelihood Estimation (cIMLE) [21, 30] to learn
the distribution, which is designed to avoid mode collapse.

We consider the challenging setting of leveraging out-of-domain depth priors
to train NeRFs on real-world indoor scenes with sparse views. Under this
setting, the depth priors we use are trained on a different dataset (e.g.,
Taskonomy) from the scenes our NeRFs are trained on (e.g., ScanNet). This
setting is more challenging than usual due to domain gap between the dataset
the prior is trained on and the scenes NeRF is asked to reconstruct. We
demonstrate that our method outperforms vanilla NeRF and NeRFs with
supervision from depth-based priors in novel view synthesis.

In summary, our key contributions include:

  * •

An method that allows the creation of NeRFs in unconstrained indoor settings
from only a modest number of 2D views by introducing a sophisticated way to
exploit ambiguity-aware monocular depth estimation.

  * •

A novel way to sample distributions over image depth estimates based on
conditional Implicit Maximum Likelihood Estimation that can represent depth
ambiguities and capture a variable number of discrete depth modes.

  * •

A new space-carving loss that can be used in the NeRF formulation to optimize
for a mode-seeking 3D density that helps select consistent depth modes across
the views and thus compensate for the under-constrained photometric
information in the few view regime.

##  2 Related Work

##### Novel View Synthesis and 3D Reconstruction.

Reconstructing the structure of a scene from a few views is a long standing
problem in computer vision with a long literature. Neural Radiance Fields or
NeRF [26] revolutionized the field by showing how to use weights of an MLP to
represent a scene that is rendered using volume rendering [25]. A key
innovation was the use of positional encoding [39, 40] to increase the
effective capacity of the MLPs that model a emitted radiance and density as a
function of position and viewing direction. Extensions of NeRF include works
on unconstrained photo collections [24], dynamic scenes [22], deformable
scenes [29], and reflective materials [41, 1].

A critical shortcoming of NeRF is its reliance on having many input views of a
scene. Several approaches have been proposed, including adding patch
likelihood losses [27], data-driven priors [50, 38], semantic consistency
prior [13], image features [43], or surface [27, 41], occupancy [28], and
depth [45, 16, 34] priors. DS-NeRF [6] uses the sparse point reconstructions
recovered during Structure-from-Motion, to supervise the depth of sparse
points in the recovered NeRF. In the same spirit, DDP [34] uses a depth
completion network, that takes as input sparse point cloud projected to one an
input view, and produces a depth estimate. Both works [16, 34] model depth
with uncertainty but only supervise with _moments_ of NeRF’s depth
distribution. In contrast, our work is able to represent _multimodal_ depth
estimates, which can handle the inherit ambiguities of depth estimation, and
is able to seek the modes for each depth estimate to make a consistent
prediction of the scene structure.

Inspired by traditional multi-view stereo, MVS-NeRF [3] and RC-MVSNet [2]
incorporate the use cost volumes [9] to NeRF. To construct the cost volumes,
these works look for agreement between features to look for correspondences,
which is difficult under the setting of large variations in appearance or
viewpoint. In contrast, our approach introduces a novel space carving loss
that does not rely on feature correspondences and instead directly defines the
loss in 3D.

Another line of works focus on geometry reconstruction [42, 47, 51] by using
the volumetric rendering scheme to learn a neural SDF representation. These
works tackle a different problem on modeling geometry reconstruction, unlike
NeRFs whose focus is modeling appearance for novel view synthesis, where our
work falls under.

##### Depth Estimation from Single View.

Depth estimation from a single image is a complex task due multiple
ambiguities, including scene scale and shading / albedo ambiguities. Early
efforts used MRFs to compute predictions on superpixels [35]. The advent of
consumer depth cameras like Kinect [55] enabled the acquisition of larger
scale indoor 3D datasets [37], leading to methods that used deep neural
networks to predict depth from a single color image [7]. Nonetheless, scaling
datasets for depth estimation remained a challenge. Deep3D [46] enabled the
creation of stereo views by training on a large dataset of stereo movies,
while [8] used the self-supervision of left-right consistency to learn depth
from stereo views from driving cars. This approach was extended to videos
where the egomotion is also estimated and self-supervision happens across time
[56]. MegaDepth [23] uses the depth from SfM reconstructions of internet
photos, together with semantic segmentation that conveys ordinal depth
supervision cues, i.e. transient objects must be in front of the static
scenes. Most recent, multi-task learning has shown promise in training depth
estimators that work well in out-of-domain data [32].

In some cases, surface normal estimation has fewer pitfalls than direct depth
estimation, as it suffers somewhat less from scale ambiguities. GeoNet [31]
jointly estimates depth and surface normals, then uses the geometric relation
between them to refine the estimates. LeRes [48] uses a second geometry
reasoning module that refines a focal length estimate and the depth itself.
Our depth estimation approach is derived from LeReS without the second stage,
as the focal length is already estimated during SfM for NeRF scenes.

Most relevant is the work of Kendall and Gal [16], that learns depth
estimation in a Bayesian setting, where their objective maximizes the
likelihood under a predicted distribution. In our work, we use instead a
multimodal depth estimation technique built on cIMLE [21], which makes our
prior robust to ambiguities in depth prediction.

##  3 Background

###  3.1 Neural Radiance Fields (NeRF)

A neural radiance field [26], or NeRF for short, represents a field in 3D
space, where each point represents an infinitesimal particle with a certain
opacity that emits varying amounts of light along different viewing
directions. The opacity at a point is represented as a volume density, and the
amount of emitted light is represented as a colour. Mathematically, a NeRF is
represented as two parameterized functions, a volume density
$\sigma_{\theta}:\mathbb{R}^{3}\to\mathbb{R}_{\geq 0}$ and a colour
$\mathbf{c}_{\psi}:\mathbb{R}^{3}\times S^{2}\to[0,255]^{3}$. The former maps
a 3D coordinate $\mathbf{x}\in\mathbb{R}^{3}$ to a volume density
$\sigma\in\mathbb{R}_{\geq 0}$, and the latter maps a 3D coordinate
$\mathbf{x}\in\mathbb{R}^{3}$ and a viewing direction $\mathbf{d}\in S^{2}$ to
a colour $\mathbf{c}\in[0,255]^{3}$.

To render a NeRF from a view, we shoot rays from each pixel on the camera
sensor and integrate over the product of colour, volume density and visibility
along the ray to arrive at each pixel value. Visibility at a point is
represented with transmittance, which accumulates the exponentiated negative
volume density multiplicatively up to the point. The higher transmittance at a
point, the more visible the point is from the camera. Mathematically, if the
camera ray is $\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$, where $\mathbf{o}$ is
the camera centre and $\mathbf{d}$ is the ray direction, the pixel value
$\hat{I}_{\theta,\psi}(\mathbf{o},\mathbf{d})$ we would have is:

$\begin{split}&\hat{I}_{\theta,\psi}(\mathbf{o},\mathbf{d}):=\int_{t_{n}}^{t_{f}}T_{\theta,\mathbf{o},\mathbf{d}}(t)\sigma_{\theta}(\mathbf{o}+t\mathbf{d})\mathbf{c}_{\psi}(\mathbf{o}+t\mathbf{d},\mathbf{d})\>\mathrm{d}t\\\
&\text{where
}\;T_{\theta,\mathbf{o},\mathbf{d}}(t)=\exp(-\int_{t_{n}}^{t}\sigma_{\theta}(\mathbf{o}+s\mathbf{d})\>\mathrm{d}s).\end{split}$
(1)

In the expressions above, $t_{n}$ and $t_{f}$ denote the points the ray
intersects with the near plane and far plane respectively.

###  3.2 Inverse Rendering with NeRF

Given a set of real-world images from known views, inverse rendering aims to
find the scene whose rendered images match the real-world images. If we use
$I(\mathbf{o},\mathbf{d})$ to denote the pixel value for the ray
$\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$, this problem can be cast as an
optimization problem, namely:

$\begin{split}\min_{\theta,\psi}\sum_{\mathbf{o}}\sum_{\mathbf{d}}\|\hat{I}_{\theta,\psi}(\mathbf{o},\mathbf{d})-I(\mathbf{o},\mathbf{d})\|_{2}^{2}\end{split}$
(2)

If $\hat{I}_{\theta,\psi}(\mathbf{o},\mathbf{d})$ is differentiable w.r.t. the
parameters $\theta,\psi$, this problem can be tackled straightforwardly with
gradient-based optimization. To enable this, the volume density function
$\sigma_{\theta}$ and the colour function $\mathbf{c}_{\psi}$ are chosen to be
neural networks. As a result, inverse rendering amounts to training the NeRF
to reconstruct the scene in a way that is consistent with the images that are
given. Once the NeRF is trained, novel view synthesis can be achieved by
rendering the NeRF from new, unseen views.

Since this optimization problem is underconstrained, in general many views are
needed to reconstruct the scene accurately. Yet, in practical applications,
typically only a few views are available, hence priors are needed to provide
sufficient constraints. In this paper, we consider priors in the form of
monocular depth estimates from each view. The monocular depth estimates are
trained on different datasets from the dataset NeRF is trained on to simulate
real-world conditions where there is a domain gap between what the prior was
trained on and what the NeRF is trained on.

###  3.3 Ray Termination Distance

In order to leverage depth priors, a natural way is to use them to constrain
the ray termination distance. Because NeRF can represent non-opaque surfaces,
there is no single ray termination distance. Instead, there is a distribution
of ray termination distances. The cumulative distribution function (CDF) of
this distribution represents the probability that the ray terminates before
reaching a given point. It turns out that the probability density of this
distribution $f_{\theta,\mathbf{o},\mathbf{d}}(t)$ is given by [6]:

$\begin{split}f_{\theta,\mathbf{o},\mathbf{d}}(t)=T_{\theta,\mathbf{o},\mathbf{d}}(t)\sigma_{\theta}(\mathbf{o}+t\mathbf{d})\end{split}$
(3)

##  4 Method

###  4.1 Supervising Ray Termination Distance

If all surfaces were opaque and the ground truth depth were given, supervising
NeRF with the ground truth depth would be straightforward. All that is needed
is to make the NeRFs distribution of ray termination distances as close as
possible to a delta distribution centred at the ground truth depth. However,
when the ground truth depth is not known, we have to use monocular depth
estimates as supervision. The challenge is that unlike ground truth depth,
monocular depth estimates may not be stereo-consistent, due to estimation
error and inherent ambiguity. As a result, it may not be possible to find a
scene that is consistent with the depth estimates from every view. Therefore,
in order to reconstruct the scene, we must model the uncertainty and ambiguity
in the depth estimates, which can be mathematically characterized by
distributions. Such ambiguities cannot be resolved from a single view and can
often be resolved from multiple views. So we need an method that can fuse
together uncertain depth estimates from different views.

A natural way of representing uncertainty is through a probability
distribution. When there are ambiguities, the distribution of depth estimates
are typically _multimodal_ , i.e., their probability densities have disjoint
peaks, where each mode represents one interpretation of the scene structure.
For example, albedo vs shading ambiguity as shown in Figure 2 can result in
multiple plausible interpretations and make the distribution multimodal.

An added challenge arises when non-opaque surfaces are present. In this case,
there is no single ground truth depth, and so the distribution of ground truth
depths is multimodal. So even after training the NeRF, the distribution of ray
termination distances induced by NeRF should be multimodal, as illustrated in
the glass walls in Figure 1.

Prior methods compute moments (i.e., mean and/or variance) of the distribution
of depth estimates or ray termination distance induced by NeRF. For example,
DS-NeRF [6] predicts the mean depth estimate and uses reprojection error as a
proxy for the variance. It then minimizes the KL divergence from the
distribution of ray termination distances induced by NeRF to a Gaussian whose
moments match those of the depth estimates. DDP [34] fits a Gaussian to
distribution of ray termination distances and maximizes the likelihood of the
depth estimate under the Gaussian.

Distilling complex distributions to moments has the drawback of ignoring their
possible multimodality. Because both the distribution of depth estimates and
ray termination distances are potentially multimodal, it is important to
handle the multimodality.

Figure 3: Network Architecture for our Ambiguity-aware Depth Estimates.

###  4.2 SCADE

We now present our novel method _Space Carving with Ambiguity-aware Depth
Estimates_ (SCADE). Our contributions addresses the existing problems and
drawbacks mentioned in the previous section. First, SCADE accounts for the
_multimodality_ of both the distributions of monocular depth estimates and ray
termination distances that arise due to inherent ambiguities (Sec 4.2.1) and
non-opaque surfaces. Second, SCADE is also able to resolve these ambiguities
by fusing together information from multiple views through our space carving
loss (Sec 4.2.3) that uses the reverse cross entropy loss. Because this a loss
on the distribution rather than moments of the distribution (which was the
paradigm in prior work [6, 34]), it achieves supervision in 3D rather than 2D.
Third, our loss formulation is sample-based (Sec 4.2.2) and so is
computationally efficient to optimize.

####  4.2.1 Our Ambiguity-aware Depth Estimates

We handle multimodality of monocular depth distributions by introducing our
ambiguity-aware depth estimation module to account for ambiguities in depth
estimation from a single view (Figure 1, 2). In contrast to existing monocular
depth estimation networks [49] that predict single point estimates for depth,
we model the inherent uncertainty in monocular depth estimation by
representing the distribution as a _set of samples_ generated by a neural
network. We propose leveraging conditional Implicit Maximum Likelihood
estimation (cIMLE) [21] to learn a multimodal distribution of depth estimates.
We chose cIMLE rather than conditional GANs [15] in order to avoid mode
collapse, which can lead to a unimodal distribution.

Figure 3 shows our network architecture. Concretely, we combine cIMLE with a
state-of-the-art monocular depth estimation network, LeReS [49]. Our
ambiguity-aware depth estimation module ($G$) takes an input image ($I$) and a
latent code $z\sim\mathcal{N}(0,\mathbf{I})$ and outputs a conditional depth
sample $G(I,z)$ for the input image $I$. To inject randomness into the
network, we follow the technique used in [15] and incorporate AdaIn layers
into the network backbone that predicts a scale and shift to the intermediate
network features. Specifically, we added four AdaIn layers to the encoder of
the depth estimation backbone.

####  4.2.2 Sample-based Losses on Distributions

We desire to achieve an agreement between the distributions from our
ambiguity-aware prior and the ray termination distance from NeRF to constrain
the NeRF optimization under the sparse view set-up using our learned prior. As
distributions are continuous, taking their analytical integral is
computationally intractable. Thus, in order to define a differentiable loss
function for our optimization problem, we result to a sample-based loss on
these two distributions.

##### Samples from depth prior.

As previously described, we leverage cIMLE to sample from our ambiguity-aware
depth prior. Sampling $z_{1},...,z_{M}\sim\mathcal{N}(0,\mathbf{I})$, we get
depth map samples $G(I,z_{1}),...,G(I,z_{M})$ for input image $I$. Hence we
denote corresponding samples to _estimate_ and represent the distribution for
ray $\mathbf{r}$ as $y_{1},y_{2},...,y_{M}\sim G_{\mathbf{o},\mathbf{d}}$.

##### Samples from NeRF.

For the distribution given by the ray termination distances modelled by NeRF,
we sample from its probability density function
$f_{\theta,\mathbf{o},\mathbf{d}}$ (Eq. 3) with inverse transform sampling.
Concretely, we define the probability mass function for ray $\mathbf{r}$ as
$f_{\theta,\mathbf{o},\mathbf{d}}(t_{i})$as given in Eq. 3 with samples
$t_{1},...,t_{K}$, where $t_{i}\in[t_{n},t_{f}]\forall i$. By inverse
transform sampling, we first compute the cumulative distribution function
(CDF) of the ray termination distribution at samples $t_{i}$, which we denote
as
$F_{\theta,\mathbf{o},\mathbf{d}}(t_{i})=\sum_{j<i}f_{\theta,\mathbf{o},\mathbf{d}}(t_{i})$.
Thus our samples $x_{1},x_{2},...,x_{N}$ of ray termination distances from
NeRF are then given by:

$\begin{split}x_{i}=F^{-1}_{\theta,\mathbf{o},\mathbf{d}}(u_{i}),\\\
\text{where }u_{i}\sim\mathcal{U}(0,1)\end{split}$ (4)

Figure 4: Our space carving loss supervises the ray termination in 3D as
opposed to existing approaches that supervise on 2D expected depth. As shown,
this clears out clouds of dust in space. Moreover supervising in 3D allows for
wide baseline input views and still finds the common mode that correctly snaps
the surfaces.

####  4.2.3 Our Space Carving Loss

We now introduce our novel space carving loss utilizes the samples from both
multimodal distributions to constrain the NeRF optimization. The goal is to
find a subset of modes captured in the monocular depth distributions from each
view that are globally consistent. In doing so, we can fuse together the
information from the different views, since the inherent ambiguities are only
resolved given information from multiple views. This would allow us to find a
common shape that is consistent across all views and “snap” objects surfaces
together.

We thus desire a loss that has mode seeking behavior, that is the modes of the
distribution being trained should be a subset of the modes of the distribution
that provides supervision. One such loss is the reverse cross entropy, which
is the cross entropy from the NeRF ray termination distance distribution to
the distribution of our ambiguity-aware depth estimates:

$H(f_{\theta,\mathbf{o},\mathbf{d}},G_{\mathbf{o},\mathbf{d}})=-\mathbb{E}_{f_{\theta,\mathbf{o},\mathbf{d}}}[\log
G_{\mathbf{o},\mathbf{d}}].$ (5)

As shown in the IMLE papers [21, 30], this is equivalent to penalizing the
minimum of the L2 norm between the samples from the two distributions, please
see [20] for the full proof. Hence, our space carving loss is given by

$\mathcal{L}_{\text{space\\_carving}}(\mathbf{r})=\sum_{i\in[N]}\min_{j\in[M]}||x_{i}-y_{j}||^{2}_{2}.$
(6)

Because our novel space carving loss operates on distributions rather than
moments of distributions, we can have different supervision targets for
different points along the ray. In contrast, if the loss were to only to
operate on moments, which are integrals along the ray, this only yields the
same supervision target for all points along the ray. Therefore, our loss
allows us to supervise in 3D, unlike prior methods [6, 34] (which use losses
on moments) that supervise in 2D. 3D supervision allows us to clear out dust
in space, since in order to move a sample from the ray termination distance
distribution farther in depth, the loss would have to decrease the probability
density of _all_ the points in front of it. These advantages are highlighted
in Figure 4.

Our total loss to optimize and train our NeRF model is given by

$\mathcal{L}=\mathcal{L}_{\text{photometric}}+\lambda\mathcal{L}_{\text{space\\_carving}},$
(7)

where $\mathcal{L}_{\text{photometric}}$ is the standard MSE loss on the
predicted and ground truth image. See Fig. 1 for our overall pipeline.

##  5 Results

In this section, we present our experimental evaluation to demonstrate the
advantages of SCADE.

###  5.1 Datasets and Evaluation Metrics

We evaluate our method on ScanNet [5] and an in-the-wild dataset we collected
111Please also see supplement for results on the Tanks and Temples dataset
[18].. We use the sparse-view ScanNet data used by DDP [34] in their
evaluations, which comprises of three sample scenes each with 18 to 20 train
images and 8 test images. To further test the robustness of our method, we
further evaluated our method on three in-the-wild scenes collected using an
iPhoneX. We captured sparse views in three different scenes – basement,
kitchen and lounge, where each scene has 18 to 23 train images and 8 test
images. Following similar data preprocessing steps as DDP [34], we ran SfM
[36] on all images to obtain camera poses for NeRF training. For quantitative
comparison, we follow the original NeRF [26] paper and report the PSNR, SSIM
[44] and LPIPS [54] on the novel test views.

###  5.2 Implementation Details

We train our ambiguity-aware prior on the Taskonomy [52] dataset. We
initialize the weights with the pretrained LeReS [49] model. We use $M=20$
depth estimates in our experiments. Please see supplement for additional
implementation details.

Figure 5: Depth and Fusion Comparison. Depth map and fusion comparison between SCADE and DDP. |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
---|---|---|---
Vanilla NeRF [26] |  19.03 |  0.670 |  0.398
NerfingMVS [49] |  16.29 |  0.626 |  0.502
IBRNet [43] |  13.25 |  0.529 |  0.673
MVSNeRF [3] |  15.67 |  0.533 |  0.635
DS-NeRF [6] |  20.85 |  0.713 |  0.344
DDP [34] |  19.29 |  0.695 |  0.368
SCADE (Ours) |  21.54 |  0.732 |  0.292
Table 1: ScanNet Results. Results for DS-NeRF and NerfingMVS follow what was
reported in prior literature [34]. Because our setting requires out-of-domain
priors, the results for DDP are with out-of-domain priors. The results of DDP
with in-domain priors are (20.96, 0.737, 0.236) for PSNR, SSIM and LPIPS,
respectively. Figure 6: Qualitative results on ScanNet.

###  5.3 Experiments on ScanNet

We evaluate on ScanNet following the evaluation from DDP [34]. We compare to
the original NeRF [26] (Vanilla NeRF) and the recent state-of-the-art NeRF
with depth prior-based supervision, Dense Depth Prior [34] (DDP). Table 1
shows SCADE quantitatively outperforming the baselines. Because we are
interested in simulating real-world conditions with a domain gap between the
prior and the NeRF, our setting requires the use of out-of-domain priors.
Since DDP uses a in-domain prior, we retrain DDP’s depth completion network
using their official code and hyperparameters on Taskonomy [52], _i.e_. the
same out-of-domain dataset that our prior is trained on.

Figure 6 shows our qualitative results. As shown, compared to the baselines,
SCADE is able to avoid producing the clouds of dust that are present in the
results of baselines (first, third and last column). Moreover, SCADE is also
able to snap and recover objects in the scene such as the details on the
blinds (second column), the back of the chair (fourth column) and the legs of
the piano stool (fifth column). Moreover, Fig. 5 shows rendered depthmaps and
fusion results using [53] on a Scannet scene. Notice that SCADE is able to
recover better geometry compared to DDP – see corner of the calendar, cabinets
and office chair in the right image.

###  5.4 Experiments on In-the-Wild Data

To further test the robustness of SCADE, we further evaluate on data collected
in-the-wild with a standard phone camera (iPhoneX). The intrinsics for
different views are different and are not known to the algorithm. As shown in
Table 2, SCADE with the same out-of-domain prior (trained on Taskonomy)
outperforms the baselines on in-the-wild captured data, which demonstrates its
robustness. Figure 7 shows qualitative examples. Interestingly, we are able to
recover the room behind the glass better the baselines, whose results are more
blurry (first, third and fourth column). We are also able to recover objects
behind the glass such as the monitor and chair (first column). As discussed in
Sect. 4.1, non-opaque surfaces are the most challenging because both the
ground truth distribution of ray termination distances induced by NeRF and the
distribution of estimated depths from the prior must be multimodal. The fact
that we do well validates the ability of our model to capture the multimodal
distributions.

Similar to the ScanNet results, we are able to recover crisp shapes of objects
such as the thin table leg and the white chair (second and last column), and
we are also able to clear up dust near the microwave and printer compared to
the baselines (fifth and last column).

Figure 7: Qualitative results on In-the-Wild data. |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
---|---|---|---
Vanilla NeRF [26] |  20.46 |  0.713 |  0.398
DDP [34] |  21.28 |  0.727 |  0.366
SCADE (Ours) |  22.82 |  0.743 |  0.347
Table 2: In-the-wild Results.

###  5.5 Ablation Study

##### Supervision on full distribution vs moments

We first validate the importance of using our novel space carving loss that
supervises the full distribution of ray termination distance rather than just
its moments. The latter integrates along the ray, and so provides one target
value for the entire ray, which effectively makes it 2D supervision. The
former provides different target values for different samples along the same
ray, which makes it 3D supervision. We adapt our method to use 2D supervision
proposed in the recent MonoSDF [51], which computes the expected ray
termination distance and aligns the output depth map with a monocular depth
estimation prior using the MiDaS [32] loss. We use our learned prior as the
monocular depth estimate. Table 3 validates the effectiveness of our space
carving loss.

##### Multimodal prior vs unimodal prior.

Our prior models does not constrain the depth distribution to a particular
form and allows it to be multimodal. In contrast, DDP’s depth completion prior
assumes a Gaussian, which is unimodal. We ablate our prior against DDP’s by
sampling from their Gaussian distribution. We train with our space carving
loss using both a single sample, as well as $M$ samples drawn from DDP’s
prior. As shown in Table 3, while both perform better than the 2D supervision
provided by MonoSDF, but they are subpar compared to using our multimodal
prior.

##### Multiple vs single sample.

Finally, we also ablate on using a single sample vs multiple samples for our
space carving loss. For the single sample set-up, we supervise using the
sample mean of the depth estimates, which is equivalent to the maximum
likelihood estimate of the depth under Gaussian likelihood. Results in Table 3
show the importance of using multiple samples for our space carving loss.

|  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
---|---|---|---
MonoSDF supervision |  20.13 |  0.710 |  0.332
DDP prior - single sample |  20.85 |  0.712 |  0.320
DDP prior - multiple samples |  21.00 |  0.718 |  0.316
Our prior - single sample |  21.22 |  0.714 |  0.318
SCADE (Ours) |  21.54 |  0.732 |  0.292
Table 3: Ablation Study. Results on the Scannet dataset. MonoSDF supervision
refers to supervising with MonoSDF loss on expected ray termination distance
using our prior.

##### Sparsity.

We further show addt’l results under varying number of views in Tab 4. Note
that in general sparsity is not fully reflected by the absolute number of
views, because all else being equal, a larger scene requires more views to
attain complete coverage. Following prior work [34], we also report the
average number of views that see the same point as another measure of
sparsity. As shown, our setting is at the edge of the theoretical lower limit
of 2 for general-purpose reconstruction, which shows view sparsity.

ave. #views visible | 1.87 | 1.98 | 2.01 | 2.2
---|---|---|---|---
absolute #views | 18 | 20 | 22 | 24
DDP / SCADE |  19.35/ 21.66 |  22.4/ 23.67 |  23.10/ 24.00 |  23.56/ 24.84
Table 4: PSNR for DDP [34]/SCADE on scene781 of Scannet. Figure 8: Depth Mode
Discussion. a) Train images from Taskonomy [52] and their labels. Notice that
non-opaque surfaces are labelled differently. b) Output of our prior on
reflective surfaces.

###  5.6 Discussion on our Ambiguity-Aware Prior

We provide addt’l details on our ambiguity-aware prior on i) how and why it
works, ii) its performance on reflective surfaces, which is a common failure
case for depth estimate, and iii) provide an intuition on how what our depth
modes look like. i) We achieve variable depth modes by exploiting
inconsistently labeled training data. As shown in Fig 8-a, in Taskonomy [52],
different training images with non-opaque surfaces label depth differently:
shooting through the glass (left), on the glass (middle), or a mixture of both
(right). Despite multiple possible depth labels, each image only has one
ground truth label. Training with cIMLE allows our prior to model these
multiple possible (ambiguous) outputs through sampling222See supplement for
more qualitative examples on our ambiguity-aware depth estimates., even when
given only one label per image. Interestingly when testing our prior on a test
image with a mirror, we find that it is able to capture variable modes on
reflective surfaces, including the “correct" flat surface as shown in Fig 8-b.
An intuition of what the depth modes look like is for example if a ray
intersects $n-1$ non-opaque surfaces, the $n$ modes are intersections with the
non-opaque surfaces and the terminating point on the opaque surface.

##  6 Conclusion

In this paper, we present a new approach towards NeRF reconstruction that can
work with a modest number of in-the-wild views of an indoor scene. We address
the under-constrained nature of this problem by regularizing the NeRF
optimization with additional depth estimates for each view. Our key technical
contribution is to model multimodality in the depth estimates, which can
capture inherent ambiguities in monocular depth estimation as well as the
possible presence of non-opaque surfaces. We resolve ambiguities using a novel
space carving loss that fuses the multimodal depth estimates from different
views and seeks the modes that are consistent across views so as to arrive at
a globally consistent 3D reconstruction. The improved recovery of shape and
appearance enables higher fidelity novel view synthesis from sparse views.

##### Limitations and Future Work.

The performance of our method is constrained by the quality of the monocular
depth priors. While we found that our prior generalizes well across domains,
if the domain gap is too great, the performance of our method will degrade. A
future direction would to be detect when this happens and dynamically adjust
the strength of depth supervision in response.

##### Acknowledgements.

We sincerely thank the Teleportation and CCI team in Google for all the
insightful discussions during the summer. We also thank Mirko Visontai for
internship logistics, Guandao Yang for experiment set-up assist, and Weicheng
Kuo for looking over the paper.

## References

  * [1] Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi.  Neural reflectance fields for appearance acquisition.  arXiv preprint arXiv:2008.03824, 2020.
  * [2] Di Chang, Aljaž Božič, Tong Zhang, Qingsong Yan, Yingcong Chen, Sabine Süsstrunk, and Matthias Nießner.  Rc-mvsnet: Unsupervised multi-view stereo with neural rendering.  ECCV 2022, 2022.
  * [3] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14124–14133, 2021.
  * [4] Kyung Hyun Cho, Tapani Raiko, and Alexander Ilin.  Gaussian-bernoulli deep boltzmann machine.  In The 2013 International Joint Conference on Neural Networks (IJCNN), pages 1–7. IEEE, 2013.
  * [5] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017.
  * [6] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan.  Depth-supervised nerf: Fewer views and faster training for free.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12882–12891, 2022.
  * [7] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  Advances in neural information processing systems, 27, 2014.
  * [8] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow.  Unsupervised monocular depth estimation with left-right consistency.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 270–279, 2017.
  * [9] Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan.  Cascade cost volume for high-resolution multi-view stereo and stereo matching.  2019\.
  * [10] Geoffrey E Hinton.  A practical guide to training restricted boltzmann machines.  In Neural networks: Tricks of the trade, pages 599–619. Springer, 2012.
  * [11] Xun Huang and Serge Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  In Proceedings of the IEEE international conference on computer vision, pages 1501–1510, 2017.
  * [12] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * [13] Ajay Jain, Matthew Tancik, and Pieter Abbeel.  Putting nerf on a diet: Semantically consistent few-shot view synthesis.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5885–5894, October 2021.
  * [14] Wonbong Jang and Lourdes Agapito.  Codenerf: Disentangled neural radiance fields for object categories.  In ICCV, 2021.
  * [15] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  * [16] Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  Advances in neural information processing systems, 30, 2017.
  * [17] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [18] Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun.  Tanks and temples: Benchmarking large-scale scene reconstruction.  ACM Transactions on Graphics, 36(4), 2017.
  * [19] K.N. Kutulakos and S.M. Seitz.  A theory of shape by space carving.  In ICCV, 1999.
  * [20] Ke Li and Jitendra Malik.  Implicit maximum likelihood estimation.  arXiv preprint arXiv:1809.09087, 2018.
  * [21] Ke Li*, Shichong Peng*, Tianhao Zhang*, and Jitendra Malik.  Multimodal image synthesis with conditional implicit maximum likelihood estimation.  International Journal of Computer Vision, May 2020.
  * [22] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang.  Neural scene flow fields for space-time view synthesis of dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498–6508, 2021.
  * [23] Zhengqi Li and Noah Snavely.  Megadepth: Learning single-view depth prediction from internet photos.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2041–2050, 2018.
  * [24] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth.  NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections.  In CVPR, 2021.
  * [25] Nelson Max.  Optical models for direct volume rendering.  IEEE Transactions on Visualization and Computer Graphics, 1(2):99–108, 1995.
  * [26] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I, pages 405–421, 2020.
  * [27] Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, and Noha Radwan.  Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs.  In CVPR, 2022.
  * [28] Michael Oechsle, Songyou Peng, and Andreas Geiger.  Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction.  In International Conference on Computer Vision (ICCV), 2021.
  * [29] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  ICCV, 2021.
  * [30] Shichong Peng, Seyed Alireza Moazenipourasil, and Ke Li.  Chimle: Conditional hierarchical imle.  In NeurIPS, 2022.
  * [31] Xiaojuan Qi, Renjie Liao, Zhengzhe Liu, Raquel Urtasun, and Jiaya Jia.  Geonet: Geometric neural network for joint depth and surface normal estimation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 283–291, 2018.
  * [32] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  IEEE transactions on pattern analysis and machine intelligence, 2020\.
  * [33] Marc’Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun.  A unified energy-based framework for unsupervised learning.  In Artificial Intelligence and Statistics, pages 371–379. PMLR, 2007.
  * [34] Barbara Roessle, Jonathan T Barron, Ben Mildenhall, Pratul P Srinivasan, and Matthias Nießner.  Dense depth priors for neural radiance fields from sparse input views.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12892–12901, 2022.
  * [35] Ashutosh Saxena, Min Sun, and Andrew Y Ng.  Make3d: Learning 3d scene structure from a single still image.  IEEE transactions on pattern analysis and machine intelligence, 31(5):824–840, 2008.
  * [36] Johannes Lutz Schönberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [37] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  In European conference on computer vision, pages 746–760. Springer, 2012.
  * [38] Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng.  Learned initializations for optimizing coordinate-based neural representations.  In CVPR, 2021.
  * [39] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng.  Fourier features let networks learn high frequency functions in low dimensional domains.  Advances in Neural Information Processing Systems, 33:7537–7547, 2020.
  * [40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [41] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan.  Ref-NeRF: Structured view-dependent appearance for neural radiance fields.  CVPR, 2022.
  * [42] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  NeurIPS, 2021.
  * [43] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser.  Ibrnet: Learning multi-view image-based rendering.  In CVPR, 2021.
  * [44] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [45] Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, and Jie Zhou.  Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo.  In ICCV, 2021.
  * [46] Junyuan Xie, Ross Girshick, and Ali Farhadi.  Deep3d: Fully automatic 2d-to-3d video conversion with deep convolutional neural networks.  In European conference on computer vision, pages 842–857. Springer, 2016.
  * [47] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman.  Volume rendering of neural implicit surfaces.  In NeurIPS, 2021.
  * [48] Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Simon Chen, Yifan Liu, and Chunhua Shen.  Towards accurate reconstruction of 3d scene shape from a single monocular image.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [49] Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Long Mai, Simon Chen, and Chunhua Shen.  Learning to recover 3d scene shape from a single image.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. (CVPR), 2021.
  * [50] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelNeRF: Neural radiance fields from one or few images.  In CVPR, 2021.
  * [51] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger.  Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction.  arXiv preprint arXiv:2206.00665, 2022.
  * [52] Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese.  Taskonomy: Disentangling task transfer learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3712–3722, 2018.
  * [53] Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser.  3dmatch: Learning local geometric descriptors from rgb-d reconstructions.  In CVPR, 2017.
  * [54] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
  * [55] Zhengyou Zhang.  Microsoft kinect sensor and its effect.  IEEE multimedia, 19(2):4–10, 2012.
  * [56] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe.  Unsupervised learning of depth and ego-motion from video.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1851–1858, 2017.

##  A Additional Results

###  A.1 Experiments on Tanks and Temples

We conduct further experiments to test the robustness of SCADE. We evaluate on
three scenes from the Tanks and Temples [18] dataset, namely three large
indoor rooms - Church, Courtroom and Auditorium scenes. The training set
consists of 21, 26 and 21 sparse views for the Church, Courtroom and
Auditorium scenes respectively, and the test set consists of 8 sparse views,
so the amount of data is similar to that used in prior work [34]. We also
followed similar data preprocessing steps as prior work [34] and ran SfM [36]
on all images to obtain camera poses for training.

As shown in Table A5, SCADE trained with the same out-of-domain prior that we
used for the other datasets (which was trained on Taskonomy [52]) outperforms
the baselines on the Tanks and Temples dataset as well. Moreover, Figure A11
shows qualitative results. As shown, SCADE is able to recover objects better
than the baselines such as the table in the Church, the group of chairs in the
Courtroom (second column), and the rows of seats in the Auditorium (clearer in
the side-view seats on the second column). Moreover, results also show that
SCADE avoids clouds of dust such as the lights on the wall of the Church
(second column), painting on the wall of the Courtroom (last column) and
details on the repetitive seats of the auditorium.

|  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
---|---|---|---
Vanilla NeRF [26] |  17.19 |  0.559 |  0.457
DDP [34] |  19.18 |  0.651 |  0.361
SCADE |  20.13 |  0.662 |  0.358
Table A5: Quantitative results for the Tanks and Temples [18] dataset.

###  A.2 Video Demo

Our project page scade-spacecarving-nerfs.github.io shows a video trajectory
from each of the three datasets in our experiments. As shown on the Scannet
scene, SCADE is able to better recover and crisp up the black chair; on the
In-the-Wild data scene, the room and objects behind the glass wall are better
captured, and finally, on the Tanks and Temples scene, the table on the center
is more solid and also clear up dust on the wall and aisle of the church.

|  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
---|---|---|---
$M=1\text{ }$ |  21.22 |  0.714 |  0.318
$M=5\text{ }$ |  21.05 |  0.722 |  0.304
$M=10\text{ }$ |  21.41 |  0.729 |  0.296
$M=20\text{ }$ |  21.54 |  0.732 |  0.292
$M=40\text{ }$ |  21.61 |  0.729 |  0.293
$M=80\text{ }$ |  21.58 |  0.729 |  0.295
Table A6: Ablation on $M$. We ablate on the number of depth estimates used
from our ambiguity-aware prior to train SCADE.

###  A.3 Ablation on Number of Hypotheses $M$

We further ablate on the number of estimates $M$ from our ambiguity-aware
prior for training SCADE. Table A6 shows the quantative results for the
Scannet dataset. As shown, the results in general improve as we increase the
number of depth estimates as this gives us a better approximation of the depth
distribution. We observe that the improvement is marginal as we increase the
number of depth estimates beyond $20$. Hence, we use $M=20$ in our
experiments.

##  A Implementation Details

We train our ambiguity-aware prior with a batch size of 16 and use a learning
rate of 0.001 for the base model, and 0.0001 for the MLP layers before AdaIn
[11]. We use a latent code dimension of $32$, and we follow the standard cIMLE
training strategy [21] and sample 20 latent codes per image and resample every
10 epochs.

To train our NeRF model, we use a batch size of 1024 rays for 500k iterations.
We use the Adam optimizer [17] with a learning rate of $5e^{-4}$ decaying to
$5e^{-5}$ in the last 100k iterations. We use the same architecture as the
original NeRF [26], which samples 64 points the coarse network and an
additional 128 points for the fine network. Because the depth estimates are in
relative scale, we directly optimize for the scale and shift for each input
image. We directly optimize a 2-dim variable for _each input image_ that
scales and shifts depth hypotheses. These variables are jointly optimized with
the NeRF for the first 400k iters (learning rate of 1e-6), and are then kept
frozen for the last 100k iters.

##  A Ambiguity-Aware Depth Estimates

Figure A9: Depth Estimate Samples. Here we show two examples of train images
from scenes used in our experiments that show the ambiguity in (top) different
degrees of convexity and (b) albedo vs shading ambiguity on the door frame and
possible existence of an object inside the bookshelf. Please see Fig. A12 for
more examples.

We show some samples from the depth distribution of our multimodal prior from
scenes in ScanNet and our in-the-wild data in Figures A10 and A9. We see that
depth from a single input image is ambiguous as captured by our multimodal
prior. In Figure A10, we are able to capture the multimodality in ray
termination distance caused by non-opaque glass surfaces. In Figure A9 (top
row) we are able to capture different degrees of concavity of the sofa as well
as the ambiguity in the depth of the far wall and floor. In Figure A9 (bottom
row), we have ambiguity on the presence of a dark colored object on the boxed
shelf and the depth of the door w.r.t. the door frame due to albedo vs shading
ambiguity. Figure A12 shows more samples of our multimodal depth estimates on
train images for the different scenes used in our experiments. Note that the
depth map visualizations are normalized per image, i.e. the colors represent
per image relative depth.

###  A.1 Adaptation of the cIMLE [21] proof for our Ambiguity-Aware Prior

Here we show an adaptation of the proof provided in IMLE [20] in the context
of learning our ambiguity-aware depth estimates. Recall that we are given a
set of input images $\\{I_{1},I_{2},...,I_{n}\\}$ each with a corresponding
ground truth depth map $D_{1},D_{2},...,D_{n}$. As we know that monocular
depth estimation is inherently ambiguous, we desire to learn a _multimodal
distribution of depth estimates_ conditioned on an input image given only one
ground truth lable (_i.e_. depth map).

Thus, we want to learn the network parameters $\phi$ for conditional
distribution $G$ such that $G_{\phi}(I,z)$ models the distribution of depth
estimates for a given input image $I\in\\{I_{1},I_{2},...,I_{n}\\}$, where
$z\sim\mathcal{N}(0,\mathbf{I})$ are latent codes sampled from a normal
distribution.

Unlike GAN’s [15] that optimize that each sample is similar to a ground truth
data point, cIMLE [21] prevents mode collapse by instead enforcing that all
ground truth data points are explained by at least one generated sample.
Hence, in order to learn $\phi$, the objective function that we want to
optimize is maximizing the sum of the likelihoods at the training examples.

Consider our ambiguity-aware prior $G_{\phi,i}$, an implicit generative model,
the likelihood induced by this model $P_{\phi,i}$ is computationally
intractable to compute as it cannot be expressed in closed form. In this
proof, we show that maximizing this likelihood is equivalent to optimizing a
sample-based objective, making it tractable. We first i) rewrite the desired
objective function (Sec A.1.1), ii) show its equivalence to the loss function
used in training (Sec A.1.2), then finally iii) show its equivalence to
maximizing the sum of the likelihoods at the training examples, _i.e_. the
single ground truth depth maps associated with each image (Sec A.1.3).

####  A.1.1 Objective function

Let’s consider the following objective function:

$\begin{split}\max_{\phi}\mathcal{L}_{\\{\delta_{i}\\}_{i}}(\phi):=\max_{\phi}\mathbb{E}_{\\{y_{i,j}\sim
P_{\phi,i}\\}_{i,j}}\Big{[}\frac{1}{n}\sum_{i=1}^{n}\frac{1}{w_{i}}\Big{(}\delta_{i}-\\\
\frac{1}{M}\sum_{j=1}^{M}\Phi_{\delta_{i}}(d(y_{i,j},D_{i})\Big{)}\Big{]}\end{split}$
(8)

$y_{i,j}$ is a sampled depth estimate, _i.e_. $y_{i,j}=G(I_{i},z_{j})$, $M$ is
the number of samples drawn, $d(y_{i,j},D_{i})$ denotes the distance between
the sampled depth estimate and the given ground truth depth map for image
$I_{i}$.

$\delta_{i}>0$ denotes the threshold of the radius of the largest neighborhood
that we are interested in, _i.e_. the neighborhood around the ground truth
data points (depth maps) where we are interested in having generated depth
estimate samples at. This radius is dependent on the training example (hence
the subscript $i$) as some examples may have a larger/smaller neighborhood of
interest than others. $\Phi_{\delta_{i}}$ is a function we choose, which we
will define below, and $w_{i}$ is a weighting factor that is also dependent on
the training example.

Note that here, we reuse $\mathcal{L}$ to denote the likelihood, and it should
not be confused with the notation for the loss functions in the main paper.

Choosing $\Phi_{\delta}$.

For $\delta>0$ (threshold on the radius), $\Phi_{\delta}$ is chosen as

$\begin{split}\Phi_{\delta}(t)=\begin{cases}t&0\leq t\leq\delta\\\
\delta&t>\delta\end{cases},\end{split}$ (9)

Intuitively, this assigns the random variable t, which is our case will be the
the distance $d(\cdot)$ between the ground truth depth and a sampled depth
estimate, to a value depending on the radius threshold $\delta$. Any distance
larger than $\delta$, _i.e_. is the sampled estimate is far enough, is set to
$\delta$.

Consequently, the chosen antiderivative is shown below

$\Phi_{\delta}^{\prime}(t)=\begin{cases}1&0\leq t\leq\delta\\\
0&t>\delta\end{cases}$

Relating to model distribution $P_{\phi,i}$
Three lemmas written below tie together the likelihood $\mathcal{L}_{\delta}$
to the objective function.

######  Lemma 1.

Let $Y$ be a non-negative random variable and $f$ be a continuous function on
$[0,\infty)$, and $f^{\prime}$ to denote a function whose antiderivative is
$f$.

$\mathbb{E}\left[f(Y)\right]=f(0)+\int_{0}^{\infty}f^{\prime}(t)\mathrm{Pr}(Y\geq
t)dt$

###### Proof.

$\displaystyle f(0)+\int_{0}^{\infty}f^{\prime}(t)\mathrm{Pr}(Y\geq t)dt$
$\displaystyle=f(0)+\int_{0}^{\infty}\int_{t}^{\infty}f^{\prime}(t)p(y)dydt$
$\displaystyle=f(0)+\int_{\\{y\geq t,t\geq
0\\}}f^{\prime}(t)p(y)d\left(\begin{array}[]{c}y\\\ t\end{array}\right)$
$\displaystyle=f(0)+\int_{0}^{\infty}\int_{0}^{y}f^{\prime}(t)p(y)dtdy$
$\displaystyle=f(0)+\int_{0}^{\infty}\left(\int_{0}^{y}f^{\prime}(t)dt\right)p(y)dy$
$\displaystyle=f(0)+\int_{0}^{\infty}\left(f(y)-f(0)\right)p(y)dy\quad\text{(2nd
FTC)}$
$\displaystyle=f(0)+\int_{0}^{\infty}f(y)p(y)dy-\int_{0}^{\infty}f(0)p(y)dy$
$\displaystyle=f(0)+\mathbb{E}\left[f(Y)\right]-f(0)$
$\displaystyle=\mathbb{E}\left[f(Y)\right]$

∎

######  Lemma 2.

With the chosen $\Phi_{\delta}(\cdot)$ and $\Phi_{\delta}^{\prime}(\cdot)$
shown previously, $\mathbb{E}_{\\{y_{i,j}\sim
P_{\phi,i}\\}_{i,j}}\left[\Phi_{\delta_{i}}(d(y_{i,j},D_{i}))\right]=\delta_{i}-\int_{0}^{\delta_{i}}\mathrm{Pr}(d(y_{i,j},D_{i})<t)dt$.

###### Proof.

By definition, $\Phi_{\delta_{i}}(0)=0$.

$\displaystyle\mathbb{E}_{\\{y_{i,j}\sim
P_{\phi,i}\\}_{i,j}}\left[\Phi_{\delta_{i}}(d(y_{i,j},D_{i})\right]$
$\displaystyle=\Phi_{\delta_{i}}(0)+\int_{0}^{\infty}\Phi_{\delta_{i}}^{\prime}(t)\mathrm{Pr}(d(y_{i,j},D_{i})\geq
t)dt\quad$ (From Lemma 1)
$\displaystyle=\int_{0}^{\delta_{i}}1\cdot\mathrm{Pr}(d(y_{i,j},D_{i})\geq
t)dt$
$\displaystyle+\int_{\delta_{i}}^{\infty}0\cdot\mathrm{Pr}(d(y_{i,j},D_{i})\geq
t)dt$ $\displaystyle=\int_{0}^{\delta_{i}}\mathrm{Pr}(d(y_{i,j},D_{i})\geq
t)dt$
$\displaystyle=\int_{0}^{\delta_{i}}\left(1-\mathrm{Pr}(d(y_{i,j},D_{i})<t)\right)dt$
$\displaystyle=\delta_{i}-\int_{0}^{\delta_{i}}\mathrm{Pr}(d(y_{i,j},D_{i})<t)dt$

∎

######  Lemma 3.

The likelihood above is equivalent to
$\mathcal{L}_{\\{\delta_{i}\\}_{i}}(\phi)=\frac{1}{n}\sum_{i=1}^{n}\frac{1}{Mw_{i}}\sum_{j=1}^{M}\int_{0}^{\delta_{i}}\mathrm{Pr}(d(y_{i,j},D_{i})<t)dt$.

###### Proof.

$\displaystyle\mathcal{L}_{\\{\delta_{i}\\}_{i}}(\phi)$
$\displaystyle=\mathbb{E}_{\\{y_{i,j}\sim
P_{\phi,i}\\}_{i,j}}\left[\frac{1}{n}\sum_{i=1}^{n}\frac{1}{w_{i}}\left(\delta_{i}-\frac{1}{M}\sum_{j=1}^{M}\Phi_{\delta_{i}}(d(y_{i,j},D_{i}))\right)\right]$
$\displaystyle=\frac{1}{n}\sum_{i=1}^{n}\frac{1}{w_{i}}\left(\delta_{i}-\frac{1}{M}\sum_{j=1}^{M}\mathbb{E}_{\\{y_{i,j}\sim
P_{\phi,i}\\}_{i,j}}\left[\Phi_{\delta_{i}}(d(y_{i,j},D_{i})\right]\right)$
$\displaystyle=\frac{1}{n}\sum_{i=1}^{n}\frac{1}{w_{i}}$
$\displaystyle\left(\delta_{i}-\frac{1}{M}\sum_{j=1}^{M}\left(\delta_{i}-\int_{0}^{\delta_{i}}\mathrm{Pr}(d(y_{i,j},D_{i})<t)dt\right)\right)\quad$
(From Lemma 2)
$\displaystyle=\frac{1}{n}\sum_{i=1}^{n}\frac{1}{Mw_{i}}\sum_{j=1}^{M}\int_{0}^{\delta_{i}}\mathrm{Pr}(d(y_{i,j},D_{i})<t)dt$

∎

####  A.1.2 Equivalence to loss function for training

Here shows the equivalence of a tractable sample-based loss function used for
training.

Radius Threshold $\delta_{i}$ Lemma 3 shows that the likelihood computes the
probability the model $P_{p}hi$ assigns to the neighborhood of the training
sample, which is controlled by the radius threshold $\delta_{i}$. To maximize
the likelihood, a small neighborhood is desired, hence a small value of
$\delta_{i}$ is desirable. However, if $\delta_{i}$ is “too small", then by
the chosen $\Phi_{\delta_{i}}$, if $d(y_{i,j},D_{i})>\delta_{i}$, for all $j$,
then $d(y_{i,j},D_{i})=\delta_{i}\forall j$, which leads to
$\frac{1}{n}\sum_{i=1}^{n}\frac{1}{w_{i}}(\delta_{i}-\frac{1}{M}\sum_{j=1}^{M}\Phi_{\delta_{i}}(d(y_{i,j},D_{i})=0$.
This leads to having no gradients w.r.t. to $\phi$ since it is constant, which
does not allow for network training. Thus the smallest $\delta_{i}$ that can
have such that the expression’s value is not constant and allows for gradients
is $\min_{j\in[M]}d(y_{i,j},D_{i})$. The likelihood objective then becomes:

$\displaystyle\mathcal{L}_{\\{\delta_{i}\\}_{i}}(\theta)=\mathbb{E}_{\\{y_{i,j}\sim
P_{\phi,i}\\}_{i,j}}$
$\displaystyle\left[\frac{1}{n}\sum_{i=1}^{n}\frac{1}{w_{i}}\left(\delta_{i}-\frac{M-1}{M}\delta_{i}-\frac{1}{M}\min_{j\in[M]}d(y_{i,j},D_{i})\right)\right]$
$\displaystyle=\mathbb{E}_{\\{y_{i,j}\sim
P_{\phi,i}\\}_{i,j}}\left[\frac{1}{n}\sum_{i=1}^{n}\frac{1}{w_{i}}\left(\frac{1}{M}\delta_{i}-\frac{1}{M}\min_{j\in[M]}d(y_{i,j},D_{i})\right)\right]$
$\displaystyle=\mathbb{E}_{\\{y_{i,j}\sim
P_{\phi,i}\\}_{i,j}}\left[\frac{1}{nM}\sum_{i=1}^{n}\frac{1}{w_{i}}\left(\delta_{i}-\min_{j\in[m]}d(y_{i,j},D_{i})\right)\right]$

The sample-based loss function then becomes equivalent to the objective of
maximizing the likelihood as follows:

$\displaystyle\arg\max_{\phi}\mathcal{L}_{\\{\phi_{i}\\}_{i}}(\phi)$
$\displaystyle=\arg\max_{\phi}\mathbb{E}_{\\{y_{i,j}\sim P_{\phi,i}\\}_{i,j}}$
$\displaystyle\left[\frac{1}{nM}\sum_{i=1}^{n}\frac{1}{w_{i}}\left(\delta_{i}-\min_{j\in[M]}d(y_{i,j},D_{i})\right)\right]$
$\displaystyle=\arg\max_{\phi}\mathbb{E}_{\\{y_{i,j}\sim
P_{\phi,i}\\}_{i,j}}\left[\sum_{i=1}^{n}\frac{\delta_{i}}{w_{i}}-\frac{1}{w_{i}}\min_{j\in[M]}d(y_{i,j},D_{i})\right]$
$\displaystyle=\arg\max_{\phi}\mathbb{E}_{\\{y_{i,j}\sim
P_{\phi,i}\\}_{i,j}}\left[-\sum_{i=1}^{n}\frac{1}{w_{i}}\min_{j\in[M]}d(y_{i,j},D_{i})\right]$
$\displaystyle=\arg\min_{\phi}\mathbb{E}_{\\{y_{i,j}\sim
P_{\phi,i}\\}_{i,j}}\left[\sum_{i=1}^{n}\frac{1}{w_{i}}\min_{j\in[M]}d(y_{i,j},D_{i})\right]$

which is the sample-based loss function, _i.e_. taking the minimum loss for
the set of drawn samples. In our case, $w_{i}=0\forall i$, and for each
training data point, we sample $M=20$ estimates by drawing
$z_{j}\sim\mathcal{N}(0,\mathbf{I})$, and taking the minimum loss w.r.t. to
the corresponding single ground truth depth map $D_{i}$ for the training data
point. This allows us to learn multimodal depth distributions to capture the
inherent ambiguities in monocular depth estimation.

####  A.1.3 Equivalence to maximizing the sum of the likelihoods.

For completeness, here shows the equivalence of the objective function to
maximizing the sum of the likelihood as proven in IMLE [20]. The learning of
$\phi$ involves solving a sequence of optimization problems at current values
for $\delta_{i}$, and as optimization progresses later into the sequence,
$\delta_{i}$ becomes smaller and smaller and eventually converges to the
maximum likelihood.

######  Lemma 4.

$\lim_{\\{\delta_{i}\to
0^{+}\\}_{i}}\mathcal{L}_{\\{\delta_{i}\\}_{i}}(\phi)=\frac{1}{n}\sum_{i=1}^{n}p_{\delta}(D_{i})$.

###### Proof.

$\displaystyle\mathcal{L}_{\\{\delta_{i}\\}_{i}}(\phi)$
$\displaystyle=\frac{1}{n}\sum_{i=1}^{n}\frac{1}{Mw_{i}}\sum_{j=1}^{M}\int_{0}^{\tau_{i}}\mathrm{Pr}(d(y_{i,j},D_{i})<t)dt\quad$
(From Lemma 3)
$\displaystyle=\frac{1}{nM}\sum_{i=1}^{n}\sum_{j=1}^{M}\frac{1}{w_{i}}\int_{0}^{\tau_{i}}\int_{B_{t}(D_{i})}p_{\phi,i}(\mathbf{y})d\mathbf{y}dt$
$\displaystyle=\frac{1}{nM}\sum_{i=1}^{n}\sum_{j=1}^{M}\frac{\int_{0}^{\delta_{i}}\int_{B_{t}(D_{i})}P_{\phi,i}(\mathbf{y})d\mathbf{y}dt}{\int_{0}^{\delta_{i}}\int_{B_{t}(D_{i})}d\mathbf{y}dt}$

$\displaystyle\lim_{\\{\delta_{i}\to
0^{+}\\}_{i}}\mathcal{L}_{\\{\delta_{i}\\}_{i}}(\phi)$
$\displaystyle=\frac{1}{nM}\sum_{i=1}^{n}\left(\lim_{\delta_{i}\to
0^{+}}\left(\sum_{j=1}^{M}\frac{\int_{0}^{\delta_{i}}\int_{B_{t}(D_{i})}p_{\phi,i}(\mathbf{y})d\mathbf{y}dt}{\int_{0}^{\delta_{i}}\int_{B_{t}(D_{i})}d\mathbf{y}dt}\right)\right)$
$\displaystyle=\frac{1}{nM}\sum_{i=1}^{n}\sum_{j=1}^{M}\left(\lim_{\delta_{i}\to
0^{+}}\frac{\int_{0}^{\delta_{i}}\int_{B_{t}(D_{i})}p_{\phi,i}(\mathbf{y})d\mathbf{y}dt}{\int_{0}^{\delta_{i}}\int_{B_{t}(D_{i})}d\mathbf{y}dt}\right)$
$\displaystyle=\frac{1}{nM}\sum_{i=1}^{n}\sum_{j=1}^{M}\left(\lim_{\delta_{i}\to
0^{+}}\frac{\int_{B_{\delta_{i}}(D_{i})}p_{\phi,i}(\mathbf{y})d\mathbf{y}}{\int_{B_{\delta_{i}}(D_{i})}d\mathbf{y}}\right)\quad$
(L’Hôpital and 2nd FTC)
$\displaystyle=\frac{1}{nM}\sum_{i=1}^{n}\sum_{j=1}^{M}\left(\lim_{\delta_{i}\to
0^{+}}\frac{\int_{0}^{\delta_{i}}\int_{\\{\mathbf{y}|d(\mathbf{y},D_{i})=r\\}}p_{\phi,i}(\mathbf{y})d\mathbf{y}dr}{\int_{0}^{\delta_{i}}\int_{\\{\mathbf{y}|d(\mathbf{y},D_{i})=r\\}}d\mathbf{y}dr}\right)$
$\displaystyle=\frac{1}{nM}\sum_{i=1}^{n}\sum_{j=1}^{M}\left(\lim_{\delta_{i}\to
0^{+}}\frac{\int_{\\{\mathbf{y}|d(\mathbf{y},D_{i})=\delta_{i}\\}}p_{\phi,i}(\mathbf{y})d\mathbf{y}}{\int_{\\{\mathbf{y}|d(\mathbf{y},D_{i})=\delta_{i}\\}}d\mathbf{y}}\right)\quad$
(L’Hôpital and 2nd FTC)
$\displaystyle=\frac{1}{nM}\sum_{i=1}^{n}\sum_{j=1}^{M}p_{\phi,i}(D_{i})$
$\displaystyle=\frac{1}{n}\sum_{i=1}^{n}p_{\phi,i}(D_{i})$

∎

##  A Training Images Samples

We also show samples of train images from the three scenes in each of the
three datasets in our experiments. Samples are shown in Figure A13.

Figure A10: Ambiguity-Aware Depth Estimates. Hypothesis from two input views
with non-opaque surfaces. This figure shows that in both cases, our ambiguity-
aware prior is able to recover a distribution of depth estimates that is
multimodal. These multimodal distributions allow to capture the room as well
as the recover the objects behind the glass. Given the multiple views with
multimodal distributions, NeRF is able to find the mode that is consistent,
hence allowing for less blurry and better photometric reconstruction. Please
view the attached video demo for the results on this In-the-Wild scene.

‘

Figure A11: Qualitative Results for the Tanks and Temples [18] dataset. Figure
A12: Samples from our Ambiguity-Aware Depth Estimates on train images of the
different scenes used in our experiments. Ambiguity is shown in [Left; right]:
(a) How far the back wall is relative to the chair as well as the width of the
cabinet and how far it is relative to the desk; whether the door is at a
different compared to the wall and the relative depth of the the second chair
w.r.t. to the nearer chair and the wall. (b) Objects on the desk have varying
depths, e.g. it is unclear from a single view whether the papers have a
thickness or not; relative depth of the chair w.r.t. the wall and the camera
(c) Depth of the bookshelf; albedo v.s. shading of the door w.r.t to the door
frame. (d) Depth of the curtain, whether it is flat on the wall or not, and
without scene context, it can also be interpreted as painted texture on the
wall; relative depths of the different cluttered objects. (e) Relative depths
of the barrier, the seats and the far back wall with a cabinet; depth of the
far back corner of the room w.r.t. the desk and chair and the camera. (f)
Whether the painting is flat on the wall or the frame protrudes it out;
relative depths of the chairs and the far back wall. (g) Whether the painted
texture is convex or is flat (i.e. just painted) on the wall; whether there is
a far back door or is just a texture on the wall. (h) Both are similar to g.
left but on different viewpoints and on the opposite side of the room. (i)
Non-opaque surface ambiguity due to the glass cabinet; glass door behind the
sofa is also non-opaque.  Figure A13: Samples of training images from the
three scenes from the three datasets - Scannet [5], In-the-Wild and Tanks and
Temples [18].