# UNISURF: Unifying Neural Implicit Surfaces and
Radiance Fields for Multi-View Reconstruction

Michael Oechsle1,2,3 Songyou Peng1,4 Andreas Geiger1,2
1Max Planck Institute for Intelligent Systems, Tübingen 2University of
Tübingen
3ETAS GmbH, Stuttgart 4ETH Zurich
<EMAIL_ADDRESS>
###### Abstract

Neural implicit 3D representations have emerged as a powerful paradigm for
reconstructing surfaces from multi-view images and synthesizing novel views.
Unfortunately, existing methods such as DVR or IDR require accurate per-pixel
object masks as supervision. At the same time, neural radiance fields have
revolutionized novel view synthesis. However, NeRF’s estimated volume density
does not admit accurate surface reconstruction. Our key insight is that
implicit surface models and radiance fields can be formulated in a unified
way, enabling both surface and volume rendering using the same model. This
unified perspective enables novel, more efficient sampling procedures and the
ability to reconstruct accurate surfaces without input masks. We compare our
method on the DTU, BlendedMVS, and a synthetic indoor dataset. Our experiments
demonstrate that we outperform NeRF in terms of reconstruction quality while
performing on par with IDR without requiring masks.

##  1 Introduction

Figure 1:  Illustration. Implicit models based on surface rendering [61, 38]
require input masks and radiance fields [34] do not optimize implicit surfaces
directly. UNISURF provides a principled unified formulation, enabling accurate
surface reconstruction from images without input masks.

Capturing the geometry and appearance of 3D scenes from a set of images is one
of the cornerstone problems in computer vision. Towards this goal, coordinate-
based neural models have emerged as a powerful tool for 3D reconstruction of
geometry and appearance within the last years.

Many recent methods employ continuous implicit functions parameterized with
neural networks as 3D representations of geometry [31, 8, 41, 57, 47, 32, 12,
37, 3, 43] or appearance [47, 38, 39, 52, 40, 34, 61]. These neural 3D
representations have shown impressive performance on geometry reconstruction
and novel view synthesis from multi-view images. Besides the choice of the 3D
representation (e.g., occupancy field, unsigned or signed distance field), one
key element for neural implicit multi-view reconstruction is the rendering
technique. While some of these works represent the implicit surface as level
set and hence render the appearance from surfaces [38, 52, 61], others
integrate densities by drawing samples along the viewing rays [34, 22, 49].

In existing work, surface rendering techniques have shown impressive
performance in 3D reconstruction [38, 61]. However, they require per-pixel
object masks as input and an appropriate network initialization since surface
rendering techniques only provide gradient information locally where a surface
intersects with a ray. Intuitively speaking, optimizing wrt. local gradients
can be seen as an iterative deformation procedure applied to an initial neural
surface which is often initialized as a sphere. Additional constraints such as
mask supervision are necessary for converging to a valid surface, see Fig. 2
for an illustration. Due to their reliance on masks, surface rendering methods
are limited to object-level reconstruction and do not scale to larger scenes.

|  |
---|---|---
Random Init. | Sphere Init. | Sphere Init.
No Masks | No Masks111We remark that this result w/o masks from IDR [61] was the only successful case out of several failure cases. | With Masks
Figure 2: Surface Rendering (IDR Results). State-of-the-art methods like IDR
[61] require object masks and careful initialization for capturing accurate
geometry.

On contrary, volume rendering methods like NeRF [34] have shown impressive
results for novel view synthesis, also for larger scenes. However, surfaces
extracted as level sets of the underlying volume density are usually non-
smooth and contain artifacts due to the flexibility of the radiance field
representation which does not sufficiently constrain the 3D geometry in the
presence of ambiguities, see Fig. 3.

|  |
---|---|---
$\sigma=1$ | $\sigma=50$ | $\sigma=500$
Figure 3: Volume Rendering (NeRF Results). We show level sets of the recovered
density volume for a trained NeRF model [34] using different density
thresholds $\sigma$.

Contributions: In this paper, we propose UNISURF (UNIfied Neural Implicit
SUrface and Radiance Fields) a principled unified framework for implicit
surfaces and radiance fields, with the goal of reconstructing solid (i.e.,
non-transparent) objects from a set of RGB images. Our framework combines the
benefits of surface rendering with those of volume rendering, enabling the
reconstruction of accurate geometry from multi-view images without masks. By
recovering implicit surfaces, we are able to gradually decrease the sampling
region for volume rendering during optimization. Starting with large sampling
regions enables capturing coarse geometry and resolving ambiguities during
early iterations. At a later stage, we draw samples closer to the surface
which improves reconstruction accuracy. We show that our approach enables
capturing accurate geometry without mask supervision on the _DTU MVS_ dataset
[1], attaining results competitive with state-of-the-art implicit neural
reconstruction methods like IDR [61] which use strong mask supervision.
Moreover, we also demonstrate our method on scenes from the _BlendedMVS_
dataset [60] as well as synthetic indoor scenes from SceneNet [29]. Code is
available at https://github.com/autonomousvision/unisurf.

##  2 Related Work

In this section, we first discuss related work from the domain of 3D
reconstruction from multi-view images. Next, we provide an overview of recent
works on neural implicit representations as well as differentiable rendering.

3D Reconstruction from Multi-View Images: Reconstructing 3D geometry from
multiple images has been a longstanding computer vision problem [14]. Before
the era of deep learning, classic multi-view stereo (MVS) methods [2, 4, 5,
20, 7, 20, 48, 50, 51] focus on either matching features across views [4, 48]
or representing shapes with a voxel grid [27, 2, 5, 7, 50, 20, 42, 54, 55].
The former approaches usually have a complex pipeline requiring additional
steps like fusing depth information [9, 30] and meshing [18, 19], while the
latter ones are limited to low resolution due to cubic memory requirements. In
contrast, neural implicit representations for 3D reconstruction do not suffer
from discretization artifacts as they represent surfaces by the level set of a
neural network with continuous outputs.

Recent learning-based MVS methods attempt to replace some parts of the classic
MVS pipeline. For instance, some works learn to match 2D features[15, 21, 26,
56, 62], fuse depth maps [11, 46], or infer depth maps [16, 58, 59] from
multi-view images. Contrary to these learning-based MVS approaches, our method
only requires weak 2D supervision during optimization. Moreover, our method
yields high-quality 3D geometry and synthesizes photorealistic and consistent
novel views.

Neural Implicit Representations: Recently, neural implicit functions have
emerged as an effective representation of 3D geometry [31, 8, 41, 57, 47, 32,
12, 37, 3, 43] and appearance [47, 38, 39, 24, 52, 40, 34, 22, 49] as they
represent 3D content continuously and without discretization while
simultaneously having a small memory footprint. Most of these methods require
3D supervision. However, several recent works [38, 52, 61, 34, 23]
demonstrated differentiable rendering for training directly from images [38,
52, 61, 34, 23]. We divide these methods into two groups: surface rendering
and volume rendering.

Surface rendering approaches, including DVR [38] and IDR [61], determine the
radiance directly on the surface of an object and provide a differentiable
rendering formulation using implicit gradients. This allows for optimizing
neural implicit surfaces from multi-view images. Conditioning on the viewing
direction allows IDR to capture a high level of detail, even in the presence
of non-lambertian surfaces. However, both DVR and IDR require pixel-accurate
object masks for all views as input. In contrast, our method leads to similar
reconstructions without requiring masks.

NeRF [34] and follow-ups [63, 53, 28, 49, 36, 35, 44, 6, 45] use volume
rendering by learning alpha-compositing of a radiance field along rays. This
method has shown impressive results on novel view synthesis and does not
require mask supervision. However, the recovered 3D geometry is far from
satisfactory, see Fig. 3. Several follow-up works (Neural Body [44] D-NeRF
[45] and NeRD [6]) extract meshes using the volume density from NeRF, but none
of them considers optimizing surfaces directly. Unlike these works, we aim at
capturing accurate geometry and propose a volume rendering formulation that
provably approaches surface rendering in the limit.

##  3 Background

The two main ingredients for learning neural implicit 3D representations from
multi-view images are the 3D representation and the rendering technique
linking the 3D representation and the 2D observations. This section provides
the relevant background on implicit surface and volumetric radiance
representations which we unify in this paper for the case of solid (non-
transparent) objects and scenes.

Implicit Surface Models: Occupancy Networks [31, 38] represent surfaces as the
decision boundary of a binary occupancy classifier, parameterized by a neural
network

$o_{\theta}(\mathbf{x}):\mathbb{R}^{3}\rightarrow[0,1]$ (1)

where $\mathbf{x}\in\mathbb{R}^{3}$ is a 3D point and $\theta$ are the model
parameters. The surface is defined as the set of all 3D points where the
occupancy probability is one half:
$\mathcal{S}=\\{\mathbf{x}_{s}|o_{\theta}(\mathbf{x}_{s})=0.5\\}$. To
associate a color with every 3D point $\mathbf{x}_{s}$ on the surface, a color
field $c_{\theta}(\mathbf{x}_{s})$ can be learned jointly with the occupancy
field $o_{\theta}(\mathbf{x})$. The color for a particular pixel/ray
$\mathbf{r}$ is thus predicted as follows

$\displaystyle\hat{C}(\mathbf{r})$ $\displaystyle=c_{\theta}(\mathbf{x}_{s})$
(2)

where $\mathbf{x}_{s}$ is retrieved by root finding along ray $\mathbf{r}$,
see [38] for details. The parameters222For convenience, we use the same symbol
$\theta$ for all model parameters. $\theta$ of the occupancy field
$o_{\theta}$ and the color field $c_{\theta}$ are determined by optimizing a
reconstruction loss via gradient descent as described in [38, 24, 61].

While surface rendering allows for accurately estimating geometry and
appearance, existing approaches strongly rely on supervision with object masks
as surface rendering methods are only able to reason about rays that intersect
a surface.

Volumetric Radiance Models: In contrast to implicit surface models, NeRF [34]
represents scenes as colorized volume densities and integrates radiance along
rays via alpha blending [25, 34]. More specifically, NeRF uses a neural
network to map a 3D location $\mathbf{x}\in\mathbb{R}^{3}$ and a viewing
direction $\mathbf{d}\in\mathbb{R}^{3}$ to a volume density
$\sigma_{\theta}(\mathbf{x})\in\mathbb{R}^{+}$ and a color value
$c_{\theta}(\mathbf{x},\mathbf{d})\in\mathbb{R}^{3}$. Conditioning on the
viewing direction $\mathbf{d}$ allows for modeling view-dependent effects such
as specular reflections [40, 34] and improves reconstruction quality in case
the Lambertian assumption is violated [61]. Let $\mathbf{o}$ denote the
location of the camera center. Given $N$ samples $\\{\mathbf{x}_{i}\\}$ along
ray $\mathbf{r}=\mathbf{o}+t\mathbf{d}$, NeRF approximates the color of
pixel/ray $\mathbf{r}$ using numerical quadrature:

$\displaystyle\hat{C}(\mathbf{r})$
$\displaystyle=\sum_{i=1}^{N}T_{i}\,\left(1-\exp\left(-\sigma_{\theta}(\mathbf{x}_{i})\,\delta_{i}\right)\right)\,c_{\theta}(\mathbf{x}_{i},\mathbf{d})$
(3) $\displaystyle T_{i}$
$\displaystyle=\exp\left(-\sum_{j<i}\sigma_{\theta}(\mathbf{x}_{j})\,\delta_{j}\right)$
(4)

Here, $T_{i}$ is the accumulated transmittance along the ray and
$\delta_{i}={|\mathbf{x}_{i+1}-\mathbf{x}_{i}|}$ is the distance between
adjacent samples. As Eq. (3) is differentiable, the parameters $\theta$ of the
density field $\sigma_{\theta}$ and the color field $c_{\theta}$ can be
estimated by optimizing a reconstruction loss. We refer to [34] for details.

While NeRF does not require object masks for training due to its volumetric
radiance representation, extracting the scene geometry from the volume density
requires careful tuning of the density threshold and leads to artifacts due to
the ambiguity present in the density field, see Fig. 3.

##  4 Method

Figure 4: Concept and Notation. Our rendering consists of two steps: First, we
seek the surface $\mathbf{x}_{s}$ (green) in the occupancy field $o_{\theta}$.
Second, we define an interval around the surface to sample points
$\\{\mathbf{x}_{i}\\}$ (red) for volume rendering.

We now describe our main contribution. In contrast to NeRF which is also
applicable to non-solid scenes (e.g., fog, smoke), we restrict our focus to
solid objects that can be represented by 3D surfaces and view-dependent
surface colors. Our method exploits both, the power of volumetric radiance
representations to learn coarse scene structure without mask supervision as
well as surface rendering which acts as an inductive bias to represent objects
by a set of precise 3D surfaces, leading to accurate reconstructions.

###  4.1 Unifying Surface and Volume Rendering

We start by noting that Eq. (3) can be rewritten as333We drop dependencies on
the model parameters for clarity.

$\hat{C}(\mathbf{r})=\sum_{i=1}^{N}\alpha_{i}(\mathbf{x}_{i})\prod_{j<i}\bigl{(}1-\alpha_{j}(\mathbf{x}_{j})\bigr{)}\,c(\mathbf{x}_{i},\mathbf{d})$
(5)

with alpha values
$\alpha_{i}(\mathbf{x})=1-\exp\left(-\sigma(\mathbf{x})\,\delta_{i}\right)$.
Assuming solid objects, $\alpha$ becomes a discrete occupancy indicator
variable $o\in\\{0,1\\}$ which either takes $o=0$ in free space and $o=1$ in
occupied space as value:

$\hat{C}(\mathbf{r})=\sum_{i=1}^{N}o(\mathbf{x}_{i})\prod_{j<i}\bigl{(}1-o(\mathbf{x}_{j})\bigr{)}\,c(\mathbf{x}_{i},\mathbf{d})$
(6)

We recognize this expression as the image formation model for solid objects
[55] where the term
$o(\mathbf{x}_{i})\prod_{j<i}\bigl{(}1-o(\mathbf{x}_{j})\bigr{)}$ evaluates to
$1$ for the first occupied sample $\mathbf{x}_{i}$ along ray $\mathbf{r}$ and
to $0$ for all other samples. $\prod_{j<i}\bigl{(}1-o(\mathbf{x}_{j})\bigr{)}$
is an indicator for visibility which is $1$ if there exists no occupied sample
$\mathbf{x}_{j}$ with $j<i$ before sample $\mathbf{x}_{i}$. Thus,
$\hat{C}(\mathbf{r})$ takes the color $c(\mathbf{x}_{i},\mathbf{d})$ of the
first occupied sample along ray $\mathbf{r}$.

To unify implicit surface and volumetric radiance models, we parameterize $o$
directly by a continuos occupancy field $o_{\theta}$ (1) as opposed to
predicting volume density $\sigma$. Following [61], we condition the color
field $c_{\theta}$ on the surface normal $\mathbf{n}$ and a feature vector
$\mathbf{h}$ of the geometry network which empirically induces a useful bias
as also observed in [61] for the case of implicit surfaces. Importantly, our
unified formulation allows for both volume and surface rendering

$\displaystyle\hat{C}_{v}(\mathbf{r})$
$\displaystyle=\sum_{i=1}^{N}o_{\theta}(\mathbf{x}_{i})\prod_{j<i}\bigl{(}1-o_{\theta}(\mathbf{x}_{j})\bigr{)}c_{\theta}(\mathbf{x}_{i},\mathbf{n}_{i},\mathbf{h}_{i},\mathbf{d})$
(7) $\displaystyle\hat{C}_{s}(\mathbf{r})$
$\displaystyle=c_{\theta}(\mathbf{x}_{s},\mathbf{n}_{s},\mathbf{h}_{s},\mathbf{d})$
(8)

where $\mathbf{x}_{s}$ is retrieved by root-finding along ray $\mathbf{r}$ and
$\mathbf{n}_{s}$, $\mathbf{h}_{s}$ denote the normal and geometry features at
$\mathbf{x}_{s}$, respectively. Note that $\mathbf{x}_{s}$ depends on the
occupancy field $o_{\theta}$, but we have dropped this dependency here for
clarity. For further details, we refer the reader to the supplementary
material.

The advantage of this unified formulation is that it allows for both rendering
on the surface directly and rendering throughout the entire volume which
enables gradually removing ambiguities during optimization. As evidenced by
our experiments, it is indeed critical to combine both for obtaining accurate
reconstructions without mask supervision. Being able to quickly recover the
surface $\mathcal{S}$ via root-finding enables more effective volume
rendering, successively focusing on and refining the object surfaces as we
will describe in Section 4.3. Furthermore, surface rendering enables faster
novel view synthesis as illustrated in Fig. 5.

###  4.2 Loss Function

We optimize the following regularized loss function

$\mathcal{L}=\mathcal{L}_{rec}+\lambda\,\mathcal{L}_{reg}$ (9)

with $\ell_{1}$ reconstruction loss and $\ell_{2}$ surface regularization
which encourages the normal of a surface point $\mathbf{x}_{s}$ and a point
sampled in its neighborhood to be similar:

$\displaystyle\mathcal{L}_{rec}$
$\displaystyle=\sum_{\mathbf{r}\in\mathcal{R}}{\lVert\hat{C}_{v}(\mathbf{r})-C(\mathbf{r})\rVert}_{1}$
(10) $\displaystyle\mathcal{L}_{reg}$
$\displaystyle=\sum_{\mathbf{x}_{s}\in\mathcal{S}}{\lVert\mathbf{n}(\mathbf{x}_{s})-\mathbf{n}(\mathbf{x}_{s}+\boldsymbol{\epsilon})\rVert}_{2}$
(11)

Here, $\mathcal{R}$ denotes the set of all pixels/rays in the minibatch,
$\mathcal{S}$ is the set of corresponding surface points, $C(\mathbf{r})$ is
the observed color for pixel/ray $\mathbf{r}$ and $\boldsymbol{\epsilon}$ is a
small random uniform 3D perturbation. The normal at $\mathbf{x}_{s}$ is given
by

$\mathbf{n}(\mathbf{x}_{s})=\frac{\nabla_{\mathbf{x}_{s}}o_{\theta}(\mathbf{x}_{s})}{\lVert\nabla_{\mathbf{x}_{s}}o_{\theta}(\mathbf{x}_{s})\rVert_{2}}$
(12)

which can be computed using double backpropagation [38].

###  4.3 Optimization

The key hypothesis of implicit surface models [38, 61] is that only the region
at the first intersection point with the surface contributes to the rendering
equation. However, this assumption is not true during early iterations where
the surface is not well defined. Consequently, existing methods [38, 61]
require strong mask supervision. Conversely, during later iterations,
knowledge of the approximate surface is valuable for drawing informative
samples when evaluating the volume rendering equation in Eq. (7). Therefore,
we utilize a training schedule with a monotonically decreasing sampling
interval for drawing samples during volume rendering, as visualized in Fig. 4.
In other words, during early iterations, the samples $\\{\mathbf{x}_{i}\\}$
cover the entire optimization volume, effectively bootstrapping the
reconstruction process using volume rendering. During later iterations, the
samples $\\{\mathbf{x}_{i}\\}$ are drawn closer around the estimated surface.
As the surface can be estimated directly from the occupancy field $o_{\theta}$
via root-finding [38], this eliminates the need for hierarchical two-stage
sampling as in NeRF. Our experiments demonstrate that this procedure is
particularly effective for estimating accurate geometry, while it allows for
resolving ambiguities during early iterations.

More formally, let $\mathbf{x}_{s}=\mathbf{o}+t_{s}\mathbf{d}$. We obtain
samples $\mathbf{x}_{i}=\mathbf{o}+t_{i}\mathbf{d}$ by drawing $N$ depth
values $t_{i}$ using stratified sampling within the interval
$[t_{s}-\Delta,t_{s}+\Delta]$ centered at $t_{s}$:

$t_{i}\sim\mathcal{U}\left[t_{s}+\left(\frac{2i-2}{N}-1\right)\Delta,t_{s}+\left(\frac{2i}{N}-1\right)\Delta\right]$
(13)

During training, we start with a large interval $\Delta_{\text{max}}$ and
gradually decrease $\Delta$ for more accurate sampling and optimization of the
surface using the following decay schedule

$\Delta_{k}=\max(\Delta_{\text{max}}\exp(-k\,\beta),\Delta_{\text{min}})$ (14)

where $k$ denotes the iteration number and $\beta$ is a hyperparameter. In
fact, it can be shown that for $\Delta\rightarrow 0$ and $N\rightarrow\infty$,
volume rendering (7) indeed approaches surface rendering (8):
$\hat{C}_{v}(\mathbf{r})\rightarrow\hat{C}_{s}(\mathbf{r})$. A formal proof of
this limit is provided in the supplementary material.

As evidenced by our experiments, the decay schedule in (14) is critical for
capturing detailed geometry as it combines volume rendering of large and
uncertain volumes in the beginning of training with surface rendering towards
the end of training. To reduce free space artifacts, we combine these samples
with points sampled randomly between the camera and the surface. For rays
without surface intersection, we use stratified sampling on the entire ray.

###  4.4 Implementation Details

Architecture: Similar to Yariv et al. [61], we use an 8-layer MLP with a
Softplus activation function and a hidden dimension of $256$ for the occupancy
field $o_{\theta}$. We initialize the network such that the decision boundary
is a sphere [13]. In contrast, the radiance field $c_{\theta}$ is
parameterized as a 4-layer ReLU MLP. We encode the 3D location $\mathbf{x}$
and the viewing direction $\mathbf{d}$ using Fourier features [34] at $k$
octaves. We empirically found $k=6$ for the 3D location $\mathbf{x}$ and $k=4$
for the viewing direction $\mathbf{d}$ to work best.

Optimization: In all experiments, we fit our model to multi-view images of a
single scene. During optimization of the model parameters, we first randomly
sample a view and then $M$ pixels/rays $\mathcal{R}$ from this view based on
the camera intrinsics and extrinsics. Next, we render all rays to compute the
loss function in Eq. (9). For root-finding, we use $256$ uniform sampled
points and apply the secant method with $8$ steps [31]. For our rendering
procedure, we use $N=64$ query points inside the interval and $32$ in the free
space between the camera and the lower bound of the interval. The interval
decay parameters are $\beta=1.5e-5$, $\Delta_{\text{min}}=0.05$ and
$\Delta_{\text{max}}=1.0$. We use Adam with a learning rate of $0.0001$ and
optimize for $M=1024$ pixels per iteration with two decay steps after $200$k
and $400$k iterations. In total, we train our models for $450$k iterations.

|  |
---|---|---
GT | VR ($178\,\text{s}$)444We used a single NVIDIA V100 GPU to measure the inference time. | SR ($84\,\text{s}$)
| $32/0.79/0.33$ | $30/0.78/0.32$
Figure 5: Volume vs. Surface Rendering. We compare images generated using
volume rendering (VR, $\Delta=\Delta_{\text{min}}$) and surface rendering
(SR), reporting three image metrics
(PSNR$\uparrow$/SSIM$\uparrow$/LPIPS$\downarrow$). Both approaches yield
similar results while surface rendering is twice as fast.

Inference: Our method allows for inferring 3D shapes as well as for
synthesizing novel view images. For synthesizing images, we can render our
representation in two different ways, we can either use volume rendering or
surface rendering. In Fig. 5, we show that both rendering approaches lead to
similar results. However, we observe that surface rendering is faster than
volume rendering.

To extract meshes, we apply the Multiresolution IsoSurface Extraction (MISE)
algorithm from [31]. We use $64^{3}$ as the initial resolution and up-sample
the mesh in 3 steps without gradient-based refinement.

##  5 Experimental Evaluation

We conduct experiments on multi-view 3D reconstruction to assess our method.
First, we provide a qualitative and quantitative comparison of our approach to
existing methods (IDR [61], NeRF [34], COLMAP [48]) on the widely used _DTU
MVS_ dataset [17]. Second, we show a qualitative comparison for samples from
the _BlendedMVS_ dataset [60] and synthetic renderings of scenes from the
_SceneNet_ dataset [29]. Third, we analyze our rendering procedure and loss
function in an ablation study. In the supplementary, we provide results on the
LLFF dataset [33].

###  5.1 Baselines

To validate the effectiveness of our method, we compare it to three different
baselines.

COLMAP [48]: We consider COLMAP [48] as a classical MVS baseline, as it shows
strong performance on multi-view reconstruction and is widely used in related
works [38, 61]. We reconstruct a mesh from the output of COLMAP using screened
Poisson Surface reconstruction (sPSR) [19]. Following [38, 61], we show
results for the quantitatively best trim parameter $7$ and for the setting
that results in watertight meshes (trim parameter $0$).

NeRF [34]: Although NeRF targets novel view synthesis, its volume density
admits the extraction of geometry. To extract meshes from NeRF, we define a
density threshold of 50. We validate this choice in the supplementary.

IDR [61]: IDR is the state-of-the-art multi-view reconstruction method for
neural implicit surfaces. IDR reconstructs surfaces with an impressively high
level of detail and handles specular surfaces, but requires input masks. We do
not compare to DVR [38] as IDR’s view-dependent modeling has been demonstrated
to be superior to DVR, see [61].

###  5.2 Datasets

_DTU MVS_ Dataset [17]: The _DTU MVS_ dataset contains 49 to 64 images at a
resolution of $1200\times 1600$ as well as extrinsic and intrinsic camera
parameters for all views. The dataset consists of objects with different
shapes and appearances. Non-lambertian appearance effects make some of the
objects particularly challenging. For each scan, ground truth 3D shapes, as
well as the official evaluation procedure, are
available555https://roboimagedata.compute.dtu.dk. Like previous works [38,
61], we use the ”Surface” method of the evaluation script, and evaluate all
methods on meshes cleaned with the respective masks. The official evaluation
procedure calculates the Chamfer distance between sampled points of the
predicted shapes and ground truth shapes provided in the dataset. For
evaluating IDR [61], we use the pixel-accurate masks for all images which are
provided by the authors of IDR.

_BlendedMVS_ Dataset [60]: The _BlendedMVS_ dataset is a large-scale dataset
containing multi-view images with respective camera extrinsics and intrinsics.
We use examples from the _BlendedMVS_ low-res set with an image resolution of
$768\times 576$. These examples contain 24 to 64 different views of unmasked
images. We define the scene as such that the object is in the center and the
closest camera lies near the unit sphere.

_SceneNet_ Dataset [29]: For testing our model on complex indoor scenes, we
take two scenes from [29] for evaluation. _BlenderProc_ [10] is applied for
rendering images of a part of the scenes containing multiple objects. The
first scene is a bedroom scene with a bed, a lamp and a bedside table. The
other scene is a living room scene with a sofa, a curtain and a round table.
We use 83 and 40 images, respectively.

###  5.3 Comparison on DTU

| COLMAP | IDR | COLMAP | NeRF | Ours
---|---|---|---|---|---
masks | ✗ | ✓ | ✗ | ✗ | ✗
trim | 7 | - | 0 | - | -
scan24 | 0.45 | 1.63 | 0.81 | 1.90 | 1.32
scan37 | 0.91 | 1.87 | 2.05 | 1.60 | 1.36
scan40 | 0.37 | 0.63 | 0.73 | 1.85 | 1.72
scan55 | 0.37 | 0.48 | 1.22 | 0.58 | 0.44
scan63 | 0.90 | 1.04 | 1.79 | 2.28 | 1.35
scan65 | 1.00 | 0.79 | 1.58 | 1.27 | 0.79
scan69 | 0.54 | 0.77 | 1.02 | 1.47 | 0.80
scan83 | 1.22 | 1.33 | 3.05 | 1.67 | 1.49
scan97 | 1.08 | 1.16 | 1.40 | 2.05 | 1.37
scan105 | 0.64 | 0.76 | 2.05 | 1.07 | 0.89
scan106 | 0.48 | 0.67 | 1.00 | 0.88 | 0.59
scan110 | 0.59 | 0.90 | 1.32 | 2.53 | 1.47
scan114 | 0.32 | 0.42 | 0.49 | 1.06 | 0.46
scan118 | 0.45 | 0.51 | 0.78 | 1.15 | 0.59
scan122 | 0.43 | 0.53 | 1.17 | 0.96 | 0.62
mean | 0.65 | 0.90 | 1.36 | 1.49 | 1.02

Table 1: Quantitative Comparison on DTU [17]. We show a quantitative
comparison against the baselines for the reconstructed geometry on 15 scans
from the DTU dataset. Our method performs on par with IDR, even though IDR
uses masks as input. Bold numbers are only considering methods without mask
supervision and trimming.

In Table 1, we quantitatively compare our method to the baselines on the _DTU
MVS_ dataset. While the COLMAP baseline with trim parameter $\zeta=7$ shows
the best performance on the Chamfer distance, it produces non-watertight
meshes with incomplete surfaces. Our method performs nearly on par with the
state-of-the-art neural implicit model IDR while not relying on strong mask
supervision. NeRF and COLMAP ($\zeta=0$) also do not use input masks but
exhibit worse performance in terms of Chamfer distance.

In Fig. 6, we show qualitative results for our method and the baselines. While
COLMAP provides detailed reconstructions, it leads to incomplete geometry due
to trimming. For NeRF, holes and noise artifacts can be observed in the
reconstructions. In contrast, our approach and IDR (with input masks) produce
accurate surfaces with high-quality details. We remark that our model captures
the overall spatial arrangement of the scene accurately while being able to
also capture geometric details, e.g., the teeth of the skull and other surface
details.

|  | Without Input Masks |  | With Input Masks
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
GT view | COLMAP [48] | NeRF [34] | Ours | IDR [61]
| $(\zeta=7)$ |  |  |
Figure 6: Reconstructed 3D Shapes for DTU. We show a qualitative comparison of reconstructed surfaces. We use COLMAP [48], NeRF [34] and IDR [61] as baselines. While IDR requires pixel-accurate object masks for all images, the other methods reason about the full scene extent. Our method performs visually on par with IDR while not requiring masks. |  |  |  |  |
---|---|---|---|---|---
| SR | Uniform VR | HVR | Ours w/o Surf Reg | Ours
Chamfer Distance | (not converging) | 3.70 | 3.22 | 0.48 | 0.46
Figure 7: Ablation Study. We compare our method to different rendering procedures: Surface rendering in isolation (SR), volume rendering with uniformly sampled points (Uniform VR) and hierachical volume rendering (HVR). This analysis on the Buddha scene (DTU) shows that our full method performs best, both visually and quantitatively. Moreover, our model without surface regularizer (Ours w/o Surf Reg) leads to non-smooth regions, especially at the textureless flat table. |  |  |
---|---|---|---
|  |  |
|  |  |
|  |  |
GT view | COLMAP [48] | NeRF [34] | Ours
Figure 8: Reconstructions for Indoor Scenes [29] and BlendedMVS [60]. We show
a qualitative comparison of reconstructed surfaces on indoor scenes (1st and
2nd row) and BlendedMVS (3rd, 4th and 5th row).

###  5.4 Comparison on BlendedMVS and SceneNet

To show our model’s capabilities on more diverse scenes, we use samples of the
BlendedMVS dataset and indoor scenes from SceneNet. As there exist no object
masks for these scenes, we only consider COLMAP and NeRF as baseline methods.
While we have tried running IDR, none of the scenes converged, resulting in
degenerate outputs. For scenes with complex backgrounds, we use a background
model that learns to capture appearance information outside of our region
interest, we refer the reader to the supplementary material for more details.

Our qualitative results in Fig. 8 provide evidence that, unlike existing
implicit surface models, our method is able to reconstruct plausible geometry
for complex scenes with multiple objects and backgrounds. While COLMAP works
well on indoor scenes, it shows artifacts on uniformly colored regions (e.g.,
round table in the second row). As for the BlendedMVS experiment, NeRF can
reason about the overall spatial structure but shows less accurate surfaces
with significantly higher levels of noise compared to UNISURF. More results
can be found in the supplementary material.

###  5.5 Ablation study

We finally investigate the impact of rendering design choices and show
ablations of the loss function.

Rendering Procedure: We argue for our choices of the rendering procedure by
comparing different variations in Fig. 7: First, we consider a baseline which
only uses surface rendering during optimization (SR). We use an $\ell_{1}$
reconstruction loss on the surface color and backpropagate through implicit
differentiation following [38]. A second baseline uses uniform volume
rendering with 96 query points (Uniform VR). Third, we use NeRF’s hierarchical
volume sampling (HVR) [34] with 64+64 sample points for querying the occupancy
field. While surface rendering does not converge, Uniform VR and HVR result in
overly smooth and bloated shapes with missing details. This provides evidence
that the proposed unified model leads to more accurate reconstructions
compared to the baselines.

Losses: In Fig. 7, we also show an ablation study of our surface
regularization term in (9). Without this regularizer surfaces become less
smooth in flat and ambiguous areas, e.g., at the table. This regularization
term is particularly useful for regions that are observed less frequently, as
it incorporates an inductive bias towards smooth surfaces.

##  6 Discussion and Conclusion

This work presents UNISURF, a unified formulation of implicit surfaces and
radiance fields for capturing high-quality implicit surface geometry from
multi-view images without input masks. We believe that neural implicit
surfaces and advanced differentiable rendering procedures play a key role in
future 3D reconstruction methods. Our unified formulation shows a path towards
optimizing implicit surfaces in a more general setting than possible before.

Limitations: By design, our model is limited to represent solid, non-
transparent surfaces. Overexposed and textureless regions are also limiting
factors that lead to inaccuraries and non-smooth surfaces. Furthermore, the
reconstructions are less accurate at rarely visible regions in the images.
Limitations are discussed in more detail in the supplementary.

In future work, for resolving ambiguities from rarely visible and texture-less
regions, a prior is necessary for reconstruction. While we incorporate an
explicit smoothness prior during optimization, learning a probabilistic neural
surface model which captures regularities and uncertainty across objects would
help to resolve ambiguities, leading to more accurate reconstructions.

#### Acknowledgement

This work was supported by an NVIDIA research gift, the ERC Starting Grant
LEGO-3D (850533) and the DFG EXC number 2064/1 - project number 390727645.
Songyou Peng is supported by the Max Planck ETH Center for Learning Systems.

## References

  * [1] Henrik Aanæs, Rasmus Ramsbøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl.  Large-scale data for multiple-view stereopsis.  International Journal of Computer Vision (IJCV), 2016.
  * [2] Motilal Agrawal and Larry S Davis.  A probabilistic framework for surface reconstruction from multiple images.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2001.
  * [3] Matan Atzmon, Niv Haim, Lior Yariv, Ofer Israelov, Haggai Maron, and Yaron Lipman.  Controlling neural level sets.  In Advances in Neural Information Processing Systems (NIPS), 2019\.
  * [4] Michael Bleyer, Christoph Rhemann, and Carsten Rother.  Patchmatch stereo - stereo matching with slanted support windows.  In Proc. of the British Machine Vision Conf. (BMVC), 2011.
  * [5] JS De Bonet and Paul Viola.  Poxels: Probabilistic voxelized volume reconstruction.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 1999.
  * [6] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch.  Nerd: Neural reflectance decomposition from image collections.  arXiv.org, 2020.
  * [7] A Broadhurst, T W Drummond, and R Cipolla.  A probabilistic framework for space carving.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2001.
  * [8] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [9] Brian Curless and Marc Levoy.  A volumetric method for building complex models from range images.  In ACM Trans. on Graphics, 1996.
  * [10] Maximilian Denninger, Martin Sundermeyer, Dominik Winkelbauer, Youssef Zidan, Dmitry Olefir, Mohamad Elbadrawy, Ahsan Lodhi, and Harinandan Katam.  Blenderproc.  arXiv.org, 2019.
  * [11] Simon Donne and Andreas Geiger.  Learning non-volumetric depth fusion using successive reprojections.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [12] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser.  Learning shape templates with structured implicit functions.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019.
  * [13] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman.  Implicit geometric regularization for learning shapes.  In Proc. of the International Conf. on Machine learning (ICML), 2020\.
  * [14] Richard Hartley and Andrew Zisserman.  Multiple View Geometry in Computer Vision.  Cambridge university press, 2003.
  * [15] Wilfried Hartmann, Silvano Galliani, Michal Havlena, Luc Van Gool, and Konrad Schindler.  Learned multi-patch similarity.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2017.
  * [16] Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Jia-Bin Huang.  Deepmvs: Learning multi-view stereopsis.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [17] Rasmus Ramsbøl Jensen, Anders Lindbjerg Dahl, George Vogiatzis, Engil Tola, and Henrik Aanæs.  Large scale multi-view stereopsis evaluation.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
  * [18] Michael M. Kazhdan, Matthew Bolitho, and Hugues Hoppe.  Poisson surface reconstruction.  In Proceedings of the Fourth Eurographics Symposium on Geometry Processing, 2006.
  * [19] Michael M. Kazhdan and Hugues Hoppe.  Screened poisson surface reconstruction.  ACM Trans. on Graphics, 2013.
  * [20] Kiriakos N. Kutulakos and Steven M. Seitz.  A theory of shape by space carving.  International Journal of Computer Vision (IJCV), 2000.
  * [21] Vincent Leroy, Jean-Sébastien Franco, and Edmond Boyer.  Shape reconstruction using volume sweeping and learned photoconsistency.  In Proc. of the European Conf. on Computer Vision (ECCV), 2018.
  * [22] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt.  Neural sparse voxel fields.  In Advances in Neural Information Processing Systems (NeurIPS), 2020\.
  * [23] Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li.  Learning to infer implicit surfaces without 3d supervision.  In Advances in Neural Information Processing Systems (NIPS), 2019\.
  * [24] Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, and Zhaopeng Cui.  Dist: Rendering deep implicit signed distance function with differentiable sphere tracing.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [25] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh.  Neural volumes: Learning dynamic renderable volumes from images.  In ACM Trans. on Graphics, 2019.
  * [26] W. Luo, A. Schwing, and R. Urtasun.  Efficient deep learning for stereo matching.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [27] David Marr and Tomaso Poggio.  Cooperative computation of stereo disparity.  Science, 1976.
  * [28] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [29] John McCormac, Ankur Handa, Stefan Leutenegger, and Andrew J. Davison.  Scenenet RGB-D: can 5m synthetic images beat generic imagenet pre-training on indoor segmentation?  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2017.
  * [30] Paul Merrell, Amir Akbarzadeh, Liang Wang, Philippos Mordohai, Jan-Michael Frahm, Ruigang Yang, David Nistér, and Marc Pollefeys.  Real-time visibility-based fusion of depth maps.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2007.
  * [31] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [32] Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson.  Implicit surface representations as layers in neural networks.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019.
  * [33] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  ACM Trans. on Graphics, 2019.
  * [34] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  NeRF: Representing scenes as neural radiance fields for view synthesis.  In Proc. of the European Conf. on Computer Vision (ECCV), 2020.
  * [35] Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Chakravarty R. Alla Chaitanya, Anton Kaplanyan, and Markus Steinberger.  Donerf: Towards real-time rendering of neural radiance fields using depth oracle networks.  arXiv.org, 2021.
  * [36] Michael Niemeyer and Andreas Geiger.  Giraffe: Representing scenes as compositional generative neural feature fields.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [37] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.  Occupancy flow: 4d reconstruction by learning particle dynamics.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019.
  * [38] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.  Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [39] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger.  Texture fields: Learning texture representations in function space.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019.
  * [40] Michael Oechsle, Michael Niemeyer, Christian Reiser, Lars Mescheder, Thilo Strauss, and Andreas Geiger.  Learning implicit surface light fields.  In Proc. of the International Conf. on 3D Vision (3DV), 2020.
  * [41] Jeong Joon Park, Peter Florence, Julian Straub, Richard A. Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [42] Despoina Paschalidou, Ali Osman Ulusoy, Carolin Schmitt, Luc van Gool, and Andreas Geiger.  Raynet: Learning volumetric 3d reconstruction with ray potentials.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [43] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger.  Convolutional occupancy networks.  In Proc. of the European Conf. on Computer Vision (ECCV), 2020.
  * [44] Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou.  Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [45] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  arXiv.org, 2020.
  * [46] Gernot Riegler, Ali Osman Ulusoy, Horst Bischof, and Andreas Geiger.  OctNetFusion: Learning depth fusion from data.  In Proc. of the International Conf. on 3D Vision (3DV), 2017.
  * [47] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019.
  * [48] Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm.  Pixelwise view selection for unstructured multi-view stereo.  In Proc. of the European Conf. on Computer Vision (ECCV), 2016.
  * [49] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  Graf: Generative radiance fields for 3d-aware image synthesis.  In Advances in Neural Information Processing Systems (NeurIPS), 2020\.
  * [50] S.M. Seitz and C.R. Dyer.  Photorealistic scene reconstruction by voxel coloring.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 1997.
  * [51] Steven M. Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski.  A comparison and evaluation of multi-view stereo reconstruction algorithms.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2006.
  * [52] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  In Advances in Neural Information Processing Systems (NIPS), 2019\.
  * [53] Pratul Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T. Barron.  NeRV: Neural reflectance and visibility fields for relighting and view synthesis.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [54] Shubham Tulsiani, Tinghui Zhou, Alexei A Efros, and Jitendra Malik.  Multi-view supervision for single-view reconstruction via differentiable ray consistency.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [55] Ali Osman Ulusoy, Andreas Geiger, and Michael J. Black.  Towards probabilistic volumetric reconstruction using ray potentials.  In Proc. of the International Conf. on 3D Vision (3DV), 2015.
  * [56] Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy Ilg, Alexey Dosovitskiy, and Thomas Brox.  Demon: Depth and motion network for learning monocular stereo.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [57] Weiyue Wang, Xu Qiangeng, Duygu Ceylan, Radomir Mech, and Ulrich Neumann.  DISN: Deep implicit surface network for high-quality single-view 3d reconstruction.  Advances in Neural Information Processing Systems (NIPS), 2019.
  * [58] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan.  Mvsnet: Depth inference for unstructured multi-view stereo.  Proc. of the European Conf. on Computer Vision (ECCV), 2018.
  * [59] Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang, and Long Quan.  Recurrent mvsnet for high-resolution multi-view stereo depth inference.  Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [60] Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan.  Blendedmvs: A large-scale dataset for generalized multi-view stereo networks.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [61] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman.  Multiview neural surface reconstruction by disentangling geometry and appearance.  Advances in Neural Information Processing Systems (NeurIPS), 2020\.
  * [62] Sergey Zagoruyko and Nikos Komodakis.  Learning to compare image patches via convolutional neural networks.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [63] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.  Nerf++: Analyzing and improving neural radiance fields.  arXiv.org, 2020.