#  LOLNeRF: Learn from One Look

Daniel Rebain 1,3 Mark Matthews3 Kwang Moo Yi1 Dmitry Lagun3 Andrea
Tagliasacchi2,3
1University of British Columbia 2Simon Fraser University 3Google Research

###### Abstract

We present a method for learning a generative 3D model based on neural
radiance fields, trained solely from data with only single views of each
object. While generating realistic images is no longer a difficult task,
producing the corresponding 3D structure such that they can be rendered from
different views is non-trivial. We show that, unlike existing methods, one
does not need multi-view data to achieve this goal. Specifically, we show that
by reconstructing many images aligned to an approximate canonical pose with a
single network conditioned on a shared latent space, you can learn a space of
radiance fields that models shape and appearance for a class of objects. We
demonstrate this by training models to reconstruct object categories using
datasets that contain only one view of each subject without depth or geometry
information. Our experiments show that we achieve state-of-the-art results in
novel view synthesis and high-quality results for monocular depth prediction.
https://lolnerf.github.io

Figure 1:  Teaser – Our method learns a space of shape and appearance by
reconstructing a large collection of single-view images using a single neural
network conditioned on a shared latent space (left). This allows for a
volumetric 3D model to be lifted from the image, and rendered from novel
viewpoints (right). https://lolnerf.github.io

##  1 Introduction

A long-standing challenge in computer vision is the extraction of 3D geometric
information from images of the real world [37]. Understanding 3D geometry is
critical to understanding the physical and semantic structure of objects and
scenes, but achieving it remains a very challenging problem. Work in this area
has mainly focused either on deriving geometric understanding from more than
one view [1, 62, 25], or by using known geometry to supervise the learning of
geometry from single views [43, 14, 10, 18]. Here, we take a more ambitious
approach and aim to derive equivalent 3D understanding in a generative model
from only single views of objects, and without relying on explicit geometric
information like depth or point clouds. Deriving such 3D understanding is,
however, non trivial. While Neural Radiance Field (NeRF)-based methods [44,
13] have shown great promise in geometry-based rendering, they focus on
learning a single scene from multiple views.

Existing NeRF works [44, 52, 16] all require supervision from more than one
viewpoint, as without it, NeRF methods are prone to collapse to a flat
representation of the scene, because they have no incentive to create a
volumetric representation; see Figure 2 (left). This serves as a major
bottleneck, as multiple-view data is hard to acquire. Thus, architectures have
been devised to workaround this that combine NeRF and Generative Adversarial
Networks (GANs) [57, 9, 47], where the multi-view consistency is enforced
through a discriminator to avoid the need for multi-view training data.

\begin{overpic}[width=195.12767pt]{fig/mainpoint.png}
\put(-7.0,40.0){\footnotesize\rotatebox{90.0}{train}}
\put(-7.0,11.0){\footnotesize\rotatebox{90.0}{test}}
\put(7.0,-7.0){\footnotesize{(left)}} \put(37.0,-7.0){\footnotesize{(middle)}}
\put(74.0,-7.0){\footnotesize{(right)}} \end{overpic} Figure 2:

In this work we show that – surprisingly – having only single views of a class
of objects is enough to train NeRF models without adversarial supervision, as
long as a shared generative model is trained, and approximate camera poses are
provided. In a nutshell, the multi-view constraint of existing works no longer
necessarily needs to be enforced, and cameras do not have to be accurate to
achieve compelling results; see Figure 2 (right). Specifically, we roughly
align all images in the dataset to a canonical pose using predicted 2D
landmarks, which is then used to determine from which view the radiance field
should be rendered to reproduce the original image. For the generative model
we employ an auto-decoder framework [51]. To improve generalization, we
further train two models, one for the foreground – the common object class of
the dataset – and one for the background, since the background is often
inconsistent throughout the data, hence unlikely to be subject to the
3D-consistency bias. We also encourage our model to model shapes as solid
surfaces (i.e. sharp outside-to-inside transitions), which further improves
the quality of predicted shapes; see the improvements from Figure 2 (middle)
to Figure 2 (right).

A noteworthy aspect of our method is that we do not require rendering of
entire images, or even patches, while training. In the auto-decoder framework,
we train our models to reconstruct images from datasets, and at the same time
find the optimal latent representations for each image – an objective that can
be enforced on individual pixels. Hence, our method can be trained with
arbitrary image sizes without any increase in memory requirement during
training. In contrast, existing methods that utilize GANs [57, 9, 47]
supervise on inter-pixel relationships through their discriminators, greatly
limiting or outright preventing them from being able to scale with respect to
training image resolution.

\begin{overpic}[width=433.62pt]{fig/geometry.png}
\put(0.0,3.0){\footnotesize\rotatebox{90.0}{Our Method}}
\put(0.0,27.0){\footnotesize\rotatebox{90.0}{Wu et al.\leavevmode\nobreak\
\cite[cite]{[\@@bibref{}{Wu2020ulp}{}{}]}}}
\put(0.0,51.0){\footnotesize\rotatebox{90.0}{$\pi$-GAN\leavevmode\nobreak\
\cite[cite]{[\@@bibref{}{Chan21cvpr_piGAN}{}{}]}}} \end{overpic} Figure 3:
Geometry recovery – Our model is able to capture reasonable coarse geometry
from a single view as well as topologically complex detail like glasses that
can’t be encoded in a depth map.

In summary, we:

  * •

propose a method for learning 3D reconstruction of object categories from
single-view images which decouples training complexity from image resolution;

  * •

show that single views are enough to learn high-quality prediction of geometry
(e.g. depth) without any geometric supervision (Figure 3);

  * •

show that our method exceeds adversarial methods in representing appearance of
objects from the learned category by reconstructing held-out images and novel
views.

##  2 Related Work

There are three main research topics related to ours: \raisebox{-.6pt}{1}⃝
classic 3DMMs, \raisebox{-.6pt}{2}⃝ neural implicit shape representations, and
\raisebox{-.6pt}{3}⃝ shape estimation from single view. We also review NeRF
and auto-decoders, on which we base our research.

#### 3D Morphable Models (3DMM)

Classical approaches to building shape spaces have focused on parameterized 3D
mesh representations, with much work devoted to human faces [5, 54, 19], as
surveyed by Egger et al.[15]. These models are typically built from geometric
information sources like 3D scans or depth sensors, which are highly accurate,
but require significant investment (e.g. the proprietary Disney Medusa capture
rig[55, 20] or the commercially available solutions provided by 3DMD[66]). The
use of mesh representations also limits these models for applications like
novel view synthesis where phenomena like hair are not well reproduced. By
contrast, our approach is relatively unconstrained, as it derives flexible
volumetric representations directly from images taken in uncontrolled
environments.

#### Neural Implicit Representations

Representing scenes as 3D implicit fields has proven successful for a number
of tasks. These models can take a number of forms, including representations
of distance [51], occupancy [43, 11], learned scene features [62], and light
fields [61]. One representation in particular has been highly successful in
simultaneously modeling shape and appearance: radiance fields. Neural Radiance
Field models, known as NeRFs [44], use fields for density and radiance, and
are particularly effective in learning 3D scene structure from images alone. A
substantial number of extensions to NeRF have been proposed [13], some notable
examples being: per-view appearance codes [39], multi-resolution training [3],
camera co-optimization [69, 35, 47], hard surface priors[49], deformable
scenes [52], variable topology [53] and foreground-background
decompositions[74]. The single-scene formulation of NeRF has also been
extended to general object classes with [25, 71], and hybridized with GAN
methods as in GNeRF[42], GIRAFFE [48], and StyleNeRF[22].

#### Shape from Single View

A long-standing objective in computer vision has been to understand the 3D
structure of scenes and objects from a single image. Many works have
approached this problem by encoding a relationship between appearance and
structure using prior knowledge of that structure as supervision [18, 14, 43,
23]. Until recently however, the problem of deriving such a model from only
single-view observations has remained very difficult. Wu et al. [70]
demonstrate how shape can be inferred for categories of objects which are
approximately symmetric.

The most relevant to our work however, has been the development of GAN-based
methods which learn a space of shapes that when rendered produce a
distribution of images is indistinguishable from a training distribution [57,
9, 47, 50]. This approach is effective in producing models with plausible
structure, as they enforce an implicit multi-view constraint by requiring that
renders from any viewpoint appear realistic. Unfortunately, this requires the
use of discriminator networks which are very inefficient when combined with 3D
representations that use volumentric representations. To avoid this
limitation, we reconstruct images directly with a more efficient and scalable
stochastic sampling process.

#### Neural Radiance Fields (NeRF)

Neural Radiance Fields [44] use classical volume rendering [28] to compute
radiance values for each pixel $\mathbf{p}$ from samples taken at points
$\mathbf{x}$ along the associated ray. These samples are computed using a
learned _radiance field_ which maps $\mathbf{x}$, as well as the ray direction
$\mathbf{d}$, to radiance values $c$ and density values $\sigma$. The volume
rendering equation takes the form a weighted sum of the radiance values at
each sample point $\mathbf{x}_{i}$:

$C_{\textrm{NeRF}}(\mathbf{p})=\sum_{i=1}^{N}w_{i}\cdot
c(\mathbf{x}_{i},\mathbf{d})\vspace{-0.8em}$ (1)

with the weights $w_{i}$ being derived from an accumulation of the
transmittance along the view ray $\mathbf{x}_{i}$:

$w_{i}=\left(1-\mathrm{exp}(-\sigma(\mathbf{x}_{i})\delta_{i})\right)\,\cdot\,\mathrm{exp}\left(-\sum_{j=1}^{i-1}\sigma(\mathbf{x}_{j})\delta_{j}\right)\vspace{-0.8em}$
(2)

where $\delta_{i}$ is the sample spacing at the $i$-th point. Note here that
we denote the product of the accumulated transmittance and sample opacity as
$w$, as this value determines the contribution of a single sample to the final
pixel value. These weights can also be used to compute other values such as
surface depth (by replacing the per sample radiance values with sample depth
$0pt(\mathbf{x}_{i})$, or the overall pixel opacity:

$D(\mathbf{p})=\sum_{i=1}^{N}w_{i}\cdot
0pt(\mathbf{x}_{i})\quad\quad\alpha(\mathbf{p})=\sum_{i=1}^{N}w_{i}\cdot
1\vspace{-0.8em}$ (3)

#### Auto-decoders

Auto-decoders [51, 64, 60], also known as Generative Latent Optimization (GLO)
[6, 39, 51, 59, 2], are a family of generative models that learn without the
use of either an encoder or discriminator. The method works similarly to an
auto-_encoder_ , in that a decoder network maps a latent code to a final
output. However the method differs in how these latent codes are found: auto-
decoders learn the codes directly by allocating a table of codes with a row
for each distinct element in the training dataset. These codes are co-
optimized with the rest of the model parameters as learnable variables.

\begin{overpic}[width=433.62pt,trim=0.0pt 185.69374pt 0.0pt
0.0pt,clip]{fig/novel.png}
\put(2.0,5.0){\footnotesize\rotatebox{90.0}{$\pi$-GAN\leavevmode\nobreak\
\cite[cite]{[\@@bibref{}{Chan21cvpr_piGAN}{}{}]}}} \end{overpic}

\begin{overpic}[width=433.62pt,trim=0.0pt 0.0pt 0.0pt
185.69374pt,clip]{fig/novel.png}
\put(2.0,5.0){\footnotesize\rotatebox{90.0}{Our Method}} \end{overpic}

Figure 4: Novel view synthesis – For each method we show an example
appropriate to the training method: for $\pi$-GAN [9] a latent code sampled
from the training distribution, and for ours a learned latent code which
reconstructs a training image. Our method recovers more sharp detail due to
training on higher resolution images. For a comparison of novel views of the
same image reconstructed by both methods, see Figure 8.  Figure 5:
Architecture – Our method learns a per-image table of latent codes alongside
foreground and background NeRFs. Volumetric rendering output is subject to a
per-ray RGB loss against each training pixel, and alpha value against an image
segmenter. Camera alignments are derived from a least-squares fit of 2D
landmarker outputs to class-specific canonical 3D keypoints.
Legend: input data learned fixed function

##  3 Method

We visualize our architecture in Figure 5. We train our network parameters and
latent codes $\mathbf{Z}$ by minimizing the weighted sum of three losses:

$\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{rgb}}+\underbrace{\lambda_{\textrm{mask}}\mathcal{L}_{\text{mask}}}_{\text{Section\leavevmode\nobreak\
\ref{sec:mask}}}+\underbrace{\lambda_{\textrm{hard}}\mathcal{L}_{\text{hard}}}_{\text{Section\leavevmode\nobreak\
\ref{sec:hard}}}\vspace{-0.8em}$ (4)

where the first term is the standard L2 photometric reconstruction loss over
pixels $\mathbf{p}$ from the training images $I_{k}$:

$\mathcal{L}_{\text{rgb}}=\mathbb{E}_{k\in\\{1..K\\},\mathbf{p}\in
I_{k}}\left[(C(\mathbf{p}|\mathbf{z}_{k})-C_{\mathrm{GT}}^{k}(\mathbf{p}))^{2}\right]$
(5)

We extend the “single-scene” (i.e. overfitting/memorization) formulation of
NeRF to support learning a latent space of shapes by incorporating an auto-
decoder architecture. In this modified architecture, the main NeRF backbone
network is conditioned on a per-object latent code
$\mathbf{z}\in\mathbb{R}^{D}$, as well as the $L$-dimensional positional
encoding $\mathbf{\gamma}^{L}(x)$ as in [44]. Mathematically, the density and
radiance functions are then of the form $\sigma(\mathbf{x}|\mathbf{z})$ and
$c(\mathbf{x}|\mathbf{z})$; note we consider a formulation where radiance is
not a function of view direction $\mathbf{d}$. These latent codes are rows
from the latent table $\mathbf{Z}{\in}\mathbb{R}^{K\times D}$, which we
initialize to $\textbf{0}^{K\times D}$, where $K$ is the number of images.
This architecture makes it possible to accurately reconstruct training
examples without requiring significant extra computation and memory for an
encoder model, and avoids requiring a convolutional network to extract 3D
information from the training images [67, 72]. Training this model follows the
same procedure as single-scene NeRF, but draws random rays from all $K$ images
in the dataset, and associates each ray with the latent code that corresponds
to the object in the image it was sampled from.

###  3.1 Foreground-Background Decomposition

Similar to [74, 47, 17], we use a separate model to handle the generation of
background details. We use a lower-capacity model
$C_{\textrm{bg}}(\mathbf{d}|\mathbf{z})$ for the background that predicts
radiance on a per-ray basis. We then render by combining the background and
foreground colors using a transparency value derived from the NeRF density
function:

$\displaystyle C(\mathbf{p}|\mathbf{z})$
$\displaystyle=\alpha(\mathbf{p}|\mathbf{z})\cdot
C_{\textrm{NeRF}}(\mathbf{p}|\mathbf{z})$ (6)
$\displaystyle+(1-\alpha(\mathbf{p}|\mathbf{z}))\cdot
C_{\textrm{bg}}(\mathbf{d}_{\mathbf{p}}|\mathbf{z})$

In practice supervising the foreground/background separation is not always
necessary; see the SRN Cars [62] results in Figure 10 which learn a foreground
decomposition naturally from solid background color and 360∘ camera
distribution. When a pre-trained module is available for predicting the
foreground segmentation of the training images, we also apply an additional
loss to encourage the transparency of the NeRF volume to be consistent with
this prediction:

$\mathcal{L}_{\text{mask}}=\mathbb{E}_{k\in\\{1..K\\},\mathbf{p}\in
I_{k}}\left[(\alpha(\mathbf{p}|\mathbf{z}_{k})-S_{I}(\mathbf{p}))^{2}\right]$
(7)

where $S_{I}(\cdot)$ is the pre-trained image segmenter applied to image
$I_{k}$ and sampled at pixel $\mathbf{p}$. When training on face datasets, we
employ the MediaPipe Selfie Segmentation [41] for the pre-trained module in
(7) and $\lambda_{\textrm{mask}}{=}1.0$.

###  3.2 Hard Surfaces

NeRF does not explicitly enforce that the learned volumetric function strictly
model a hard surface. With enough input images, and sufficiently textured
surfaces, multi-view consistency will favor the creation of hard transitions
from empty to solid space. Unfortunately, this property does not hold in the
single view case. Because the field function that corresponds to each latent
code is only supervised from one viewpoint, this often results in blurring of
the surface along the view direction; see Figure 2. To counter this, we impose
a prior on the probability of the weights $w$ to be distributed as a mixture
of Laplacian distributions, one with mode around weight zero, and one with
mode around weight one:

$\text{I\kern-1.49994ptP}(w)\propto e^{-|w|}+e^{-|1-w|}$ (8)

Note the distribution is peaky, and will encourage a sparse solution where any
values of $w$ in the open interval $(0,1)$ to be discouraged. We convert this
prior into a loss via:

$\mathcal{L}_{\text{hard}}=-\log(\text{I\kern-1.49994ptP}(w))$ (9)

The magnitude of $\sigma(\mathbf{x})$ which satisfies this constraint depends
on the sampling density. Equation (9) encourages the density to produce a step
function that saturates sampling weight over at least one sampling interval,
which, by construction, is appropriate for the scale of scene being modelled.
We employ $\lambda_{\textrm{hard}}=0.1$ in our experiments.

Figure 6: Landmarker and Segmenter – (Samples outputs from the landmarker and
the segmenter networks for two input identities. Blue circles ($\bullet$)
denote the identified landmarks.

###  3.3 Camera Parameters

Volume rendering requires camera parameters that associate each pixel with a
ray used to compute sample locations. In classic NeRF, cameras are estimated
by structure-from-motion on the input image dataset. For our single-view use
case, this is not possible due to depth ambiguity. To make our method
compatible with single-view images, we we employ the MediaPipe Face Mesh[40]
pre-trained network module to extracts 2D landmarks that appear in consistent
locations for the object class being considered. Figure 6 shows example
network output of the five landmarks used for human faces.

These landmark locations are then aligned with projections of canonical 3D
landmark positions with a “shape matching” least-squares optimization to
acquire a rough estimate of camera parameters; see the supplementary material
for more detail on this process.

\begin{overpic}[width=433.62pt]{fig/ffhq_fits.pdf}
\put(-2.0,38.0){\scriptsize\rotatebox{90.0}{FFHQ Target}}
\put(-2.0,22.0){\scriptsize\rotatebox{90.0}{$\pi$-GAN Fit}}
\put(-2.0,4.0){\scriptsize\rotatebox{90.0}{LOLNeRF Fit}} \end{overpic} Figure
7:  Conditional generation – we show examples of both our method (trained on
CelebA-HQ@$256^{2}$) and $\pi$-GAN (trained on CelebA@$128^{2}$) fitting to
images not seen during training. Our method produces much sharper
reconstructions, especially for non-frontal views and appearances that are not
well represented in the training set.
\begin{overpic}[width=390.25534pt,trim=50.18748pt 0.0pt 0.0pt
0.0pt,clip]{fig/fitting.png} \put(-2.0,15.0){\footnotesize\rotatebox{90.0}{Our
Method}}
\put(-2.0,62.0){\footnotesize\rotatebox{90.0}{$\pi$-GAN\leavevmode\nobreak\
\cite[cite]{[\@@bibref{}{Chan21cvpr_piGAN}{}{}]}}} \end{overpic} Figure 8:
Novel view comparison – Comparison of novel views on held-out image fitting
with our method and $\pi$-GAN [9]. Our method captures more fine detail and
produces sharper results at large angles.

###  3.4 Conditional Generation

Given a pre-trained model, we can find a latent code $z$ which reconstructs an
image which was not present in the training set. As the latent table is
learned in parallel with the NeRF model parameters, we can treat this process
as a fine-tuning optimization for an additional row in the latent table. This
row is initialized to the mean $\mu_{\mathbf{Z}}$ over the existing rows of
the latent table, and is optimized using the same losses and optimizer as the
main model. Results for this fitting method are shown in Section 4.

###  3.5 Unconditional Generation

To sample novel objects from the space learned by our model, we sample latent
codes from the empirical distribution $\mathcal{Z}$ defined by the rows of the
latent table $\mathbf{Z}$. We model $\mathcal{Z}$ as a multivariate Gaussian
with mean $\mu_{\mathbf{Z}}$ and covariance $\chi_{\mathbf{Z}}$ found by
performing principal component analysis on the rows of $\mathbf{Z}$. Similar
to other generative models which use a Gaussian prior on their latent
variables, we observe a trade-off between diversity and quality of samples
when sampling further away from the mean of the distribution. Thus, we employ
the “truncation trick” commonly used in the GAN literature[30, 38, 8, 32] to
control this trade-off.

##  4 Results

We visualize images rendered from our models trained on the CelebA-HQ [29],
FFHQ [30], AFHQ [12], and SRN Cars [62] datasets in Figure 10. We include
additional qualitative results from these datasets in the supplementary
material. To provide quantitative evaluation of our method and comparison to
state of the art we perform a number of experiments, described in the
following subsections.

Method |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$ | Res.
---|---|---|---|---
$\pi$-GAN [9] (CelebA) | 23.5 | 0.858 | 0.226 | $128^{2}$
Ours (FFHQ) | 29.0 | 0.913 | 0.199
Ours (CelebA-HQ) | 29.1 | 0.914 | 0.197

Table 1:  Reconstructions of training images – Metrics on a subset of 200
images from the $\pi$-GAN [9] training set. Our model achieves significantly
higher reconstruction quality, regardless of whether it is trained on (FFHQ)
or (CelebA-HQ).

Method |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$ | Res.
---|---|---|---|---
$\pi$-GAN [9] (CelebA) | 21.8 | 0.796 | 0.412 | $256^{2}$
Ours (CelebA-HQ) | 26.2 | 0.856 | 0.363
$\pi$-GAN [9] (CelebA) | 20.9 | 0.795 | 0.522 | $512^{2}$
Ours (CelebA-HQ) | 25.1 | 0.831 | 0.501
Ours (FFHQ) | 25.3 | 0.836 | 0.491

Table 2:  Reconstructions of test images – Reconstruction quality (rows 1 and
2) of models trained on (CelebA) and (CelebA-HQ) on images from a 200-image
subset of FFHQ, and (rows 3-5) of models trained at $256^{2}$ (Ours) and
$128^{2}$ ($\pi$-GAN [9]) on high resolution $512^{2}$ versions of the test
images.

###  4.1 Image Reconstruction – Table 1 and Table 2

As LOLNeRF is trained with an image reconstruction metric, we first perform
experiments to evaluate how well images from the training dataset are
reconstructed. In Table 1, we show the average image reconstruction quality of
both our method and $\pi$-GAN [9] for a 200-image subset of the $\pi$-GAN [9]
training set (CelebA [36]), as measured by peak signal to noise ratio (PSNR),
structural similarity index measure (SSIM), and learned perceptual image patch
similarity (LPIPS). To compare against $\pi$-GAN [9], which does not learn
latent codes corresponding to training images, we use the procedure included
with the original $\pi$-GAN [9] implementation for fitting images through
test-time latent optimization. Because this assumes perfectly forward facing
pose, to make the comparison fair, we augment it with our camera fitting
method to improve its results on profile-view images. We also perform a more
direct comparison of image fitting by testing on a set of held out images not
seen by the network during training. To do this, we sample a set of 200 images
from the FFHQ dataset and use the latent optimization procedure described in
Section 3.4 to produce reconstructions using a model trained on CelebA images.
We show the reconstruction metrics for these images using LOLNeRF and
$\pi$-GAN [9] in Table 2 and examples of the reconstructed images in Figure 7.

Method |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  Mask-LPIPS$\downarrow$
---|---|---|---
$\pi$-GAN [9] (CelebA) | 24.5 | 0.918 | 0.102
Ours (CelebA-HQ) | 26.6 | 0.930 | 0.0989
Ours (FFHQ) | 27.0 | 0.931 | 0.0975
Table 3:  Novel view synthesis – We sample pairs of images from one frame for
each subject in the HUMBI dataset [73] and use them as query/target pairs. The
query image is used to optimize a latent representation of the subject’s face,
which is then rendered from the target view. To evaluate how well the models
have learned the 3D structure of faces, we then evaluate image reconstruction
metrics for the face pixels of the predicted and target images after applying
a mask computed from face landmarks.

###  4.2 Novel View Synthesis – Table 3

To evaluate the accuracy of the learned 3D structure, we perform image
reconstruction experiments for synthesized novel views. We render these novel
views by performing image fitting on single frames from a synchronized multi-
view face dataset, Human Multiview Behavioural Imaging (HUMBI) [73], and
reconstructing images using the camera parameters from other ground truth
views of the same person. The results of this experiment for LOLNeRF and
$\pi$-GAN [9] are given in Table 3. We find that our model achieves
significantly better reconstruction from novel views, indicating that our
method has indeed learned a better 3D shape space than $\pi$-GAN [9] – a shape
space that is capable of generalizing to unseen data and does more than simply
reproducing the query image from the query view. We also show qualitative
examples of novel views rendered by LOLNeRF and $\pi$-GAN [9] in Figures 4 and
8.

Method |  Depth Correlation$\uparrow$
---|---
Ground Truth | 66
AIGN (supervised) [68] | 50.81
DepthNetGAN (supervised) [45] | 58.68
MOFA [65] | 15.97
DepthNet [45] | 35.77
UnSup3D [70] | 54.65
Ours (CelebA-HQ) | 50.18
Table 4:  Depth prediction – Correlation between predicted and true keypoint
depth values on 3DFAW. We compare to results from supervised and unsupervised
methods as reported in [70] and [45].

###  4.3 Depth Prediction

We also evaluate our shape model by predicting depth values for images where
ground truth depth is available. We use the 3DFAW dataset [34, 75, 26, 21, 27]
which provides ground truth 3D keypoint locations. For this task we fit latent
codes from our model on the 3DFAW images and sample the predicted depth values
for each image-space landmark location. We use the same procedure as [70] to
compute the correlation of the predicted and ground truth depth values, which
is recorded in in Table 4. While our score is not as high as the best
performing unsupervised method ([70]), it outperforms several supervised and
unsupervised methods specifically designed for depth prediction.

###  4.4 High Resolution Image Synthesis

To demonstrate the benefits of being able to train directly on high-resolution
images, we quantitatively and qualitatively compare high-resolution renders
from LOLNeRF trained on 256$\times$256 FFHQ and CelebA-HQ images to those of
$\pi$-GAN [9] trained on 128$\times$128 CelebA images (the largest feasible
size used due to compute constraints). These results are shown in Table 2 and
the supplementary material. We find that for this task our models do much
better at reproducing high-resolution detail, even though both methods are
capable of producing “infinite resolution” images in theory.

##  5 Discussion

###  5.1 Limitations and Future Work

Figure 9:  Failure cases – For some images, the segmentation module fails to
remove foreground objects that are not the subject, or erroneously removes
parts of the subject (Top). Our camera fitting also sometimes produces
unreasonable camera estimates with focal lengths that are too short, or angles
that are far from correct, resulting in distorted reconstruction of that
subject during training (Bottom). Both failure modes occur in a small fraction
our data, but the model still learns a space of 3D shape and appearance that
performs well on structure-sensitive tasks, evident by our novel view
synthesis (Table 3) and depth prediction (Table 4) results.

While our method produces very high-quality results from training on in-the-
wild data, it is still reliant on other methods for extracting semantic
information (i.e. landmarks) about the objects being observed. As shown in
Figure 9, this dependence can lead to failure cases for objects where the
estimated pose or segmentation is incorrect. Finding a method to achieve
alignment without any prior knowledge remains an open research question.

Also, while our auto-decoder framework has many advantages over GANs, it does
not provide the same ability to maximize ”plausibility” of the rendered
images, which can result in some loss of detail. A possible direction for
future work would be to augment our method with adversarial training to
further improve the perceptual quality of images rendered from novel latent
codes.

###  5.2 Ethical Considerations

Our research on image generation focuses on socially beneficial use cases and
applications. When developed correctly, generative models can do that in a
number of ways including simulating a diverse population of users (fairness)
and amplify the effectiveness of personal data thus reducing the need for
large scale data collection (privacy). However, we acknowledge the potential
for misuse and importance of acting responsibly. To that end, we will only
release the code for reproducibility purposes, but will not release any
trained generative model.

Figure 10:  Qualitative renders – Outputs from our method on all datasets.
Rows 1-2: CelebA-HQ [29], Rows 3-5: FFHQ [30], Rows 6-7: AFHQ Cats [12], Rows
8-9: SRN Cars [62].

###  5.3 Conclusions

We have presented LOLNeRF, a method for learning spaces of 3D shape and
appearance from datasets of single-view images. Our experiments have shown
that this method is able to learn effectively from unstructured, ”in-the-wild”
data, without incurring the high cost of a full-image discriminator, and while
avoiding problems such as mode-dropping that are inherent to adversarial
methods.

## Acknowledgements

We thank Matthew Brown, Kevin Swersky, David Fleet, and Viral Carpenter for
their helpful technical insights and feedback. Also Danica Matthews for her
assistance with feline data collection. This work was partly supported by the
Natural Sciences and Engineering Research Council of Canada (NSERC) and by
Compute Canada.

## References

  * [1] Alex M Andrew.  Multiple view geometry in computer vision.  Kybernetes, 2001.
  * [2] Idan Azuri and Daphna Weinshall.  Generative latent implicit conditional optimization when learning from small sample.  In ICPR, 2020.
  * [3] Jonathan Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul Srinivasan.  Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.  In ICCV, October 2021.
  * [4] Mikołaj Bińkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton.  Demystifying mmd gans.  In International Conference on Learning Representations, 2018.
  * [5] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3d faces.  In Siggraph, 1999.
  * [6] Piotr Bojanowski, Armand Joulin, David Lopez-Pas, and Arthur Szlam.  Optimizing the Latent Space of Generative Networks.  In Proceedings of the 35th International Conference on Machine Learning, pages 600–609. PMLR, July 2018.  ISSN: 2640-3498.
  * [7] Ali Borji.  Pros and cons of gan evaluation measures.  Computer Vision and Image Understanding, 179:41–65, 2019.
  * [8] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale GAN training for high fidelity natural image synthesis.  In International Conference on Learning Representations, 2019.
  * [9] Eric Chan, Marco Monteiro, Peter Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis.  In CVPR, 2021.
  * [10] Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang.  BSP-Net: Generating Compact Meshes via Binary Space Partitioning.  In CVPR, 2020.
  * [11] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In CVPR, pages 5939–5948, 2019.
  * [12] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  Stargan v2: Diverse image synthesis for multiple domains.  In CVPR, pages 8188–8197, 2020.
  * [13] Frank Dellaert and Yen-Chen Lin.  Neural volume rendering: Nerf and beyond.  CoRR, abs/2101.05204, 2021.
  * [14] Boyang Deng, Kyle Genova, Sofien Bouaziz, Geoffrey Hinton, Andrea Tagliasacchi, and Soroosh Yazdani.  CvxNet: Learnable Convex Decomposition.  In CVPR, 2020.
  * [15] Bernhard Egger, William AP Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, et al.  3d morphable face models—past, present, and future.  ACM Transactions on Graphics (TOG), 39(5):1–38, 2020.
  * [16] Guy Gafni, Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction.  In CVPR, 2021.
  * [17] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang.  Dynamic View Synthesis from Dynamic Monocular Video.  In ICCV, October 2021.
  * [18] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William Freeman, and Thomas Funkhouser.  Learning Shape Templates with Structured Implicit Functions.  In ICCV, 2019.
  * [19] Thomas Gerig, Andreas Morel-Forster, Clemens Blumer, Bernhard Egger, Marcel Luthi, Sandro Schönborn, and Thomas Vetter.  Morphable face models-an open framework.  In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pages 75–82. IEEE, 2018.
  * [20] Paulo Gotardo, Jérémy Riviere, Derek Bradley, Abhijeet Ghosh, and Thabo Beeler.  Practical dynamic facial appearance modeling and acquisition.  In Siggraph Asia, 2018.
  * [21] Ralph Gross, Iain Matthews, Jeffrey Cohn, Takeo Kanade, and Simon Baker.  Multi-pie.  Image and vision computing, 28(5):807–813, 2010.
  * [22] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt.  Stylenerf: A style-based 3d aware generator for high-resolution image synthesis.  In International Conference on Learning Representations, 2022.
  * [23] Philipp Henzler.  Unsupervised Learning of 3D Object Categories from Videos in the Wild.  In CVPR, 2021.
  * [24] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [25] Wongbong Jang and Lourdes Agapito.  CodeNeRF: Disentangled Neural Radiance Fields for Object Categories.  In ICCV, October 2021.
  * [26] László A. Jeni, Jeffrey F. Cohn, and Takeo Kanade.  Dense 3D face alignment from 2D video for real-time use.  Image and Vision Computing, 58:13–24, 2016.  doi:10.1016/j.imavis.2016.05.009.
  * [27] László A. Jeni, Sergey Tulyakov, Lijun Yin, Nicu Sebe, and Jeffrey F. Cohn.  The First 3D Face Alignment in the Wild (3DFAW) Challenge.  In ECCVW, 2016.
  * [28] James T. Kajiya and Brian P. Von Herzen.  Ray tracing volume densities.  In Siggraph, 1984.
  * [29] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive Growing of GANs for Improved Quality, Stability, and Variation.  In ICLR, 2018.
  * [30] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In CVPR, 2019.
  * [31] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR (Poster), 2015.
  * [32] Durk P Kingma and Prafulla Dhariwal.  Glow: Generative Flow with Invertible 1x1 Convolutions.  In NeurIPS, 2018.
  * [33] Kenneth Levenberg.  A method for the solution of certain non-linear problems in least squares.  Quarterly of applied mathematics, 2(2):164–168, 1944.
  * [34] Lijun Yin, Xiaochen Chen, Yi Sun, Tony Worm, and Michael Reale.  A High-Resolution 3D Dynamic Facial Expression Database.  In The 8th International Conference on Automatic Face and Gesture Recognition (FGR08), Amsterdam, The Netherlands, 2008.
  * [35] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey.  BARF: Bundle-Adjusting Neural Radiance Fields.  In ICCV, October 2021.
  * [36] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In CVPR, 2015.
  * [37] Jitendra Malik, Pablo Arbeláez, João Carreira, Katerina Fragkiadaki, Ross Girshick, Georgia Gkioxari, Saurabh Gupta, Bharath Hariharan, Abhishek Kar, and Shubham Tulsiani.  The three R’s of computer vision: Recognition, reconstruction and reorganization.  Pattern Recognition Letters, 72:4–14, 2016.
  * [38] Marco Marchesi.  Megapixel size image creation using generative adversarial networks.  arXiv preprint arXiv:1706.00082, 2017.
  * [39] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan Barron, Alexey Dosovitskiy, and Daniel Duckworth.  NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections.  In CVPR, 2021.
  * [40] MediaPipe Face Mesh.  https://google.github.io/mediapipe/solutions/face_mesh.html.  Accessed: 2021-10-20.
  * [41] MediaPipe Selfie Segmentation.  https://google.github.io/mediapipe/solutions/selfie_segmentation.html.  Accessed: 2021-10-20.
  * [42] Quan Meng.  GNeRF: GAN-Based Neural Radiance Field Without Posed Camera.  In ICCV, October 2021.
  * [43] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy Networks: Learning 3D Reconstruction in Function Space.  In CVPR, 2019.
  * [44] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.  In ECCV, pages 405–421. Springer, 2020.
  * [45] Joel Ruben Antony Moniz, Christopher Beckham, Simon Rajotte, Sina Honari, and Chris Pal.  Unsupervised Depth Estimation, 3D Face Rotation and Replacement.  In NeurIPS, 2018.
  * [46] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang.  Hologan: Unsupervised learning of 3d representations from natural images.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7588–7597, 2019.
  * [47] Michael Niemeyer and Andreas Geiger.  CAMPARI: Camera-aware decomposed generative neural radiance fields.  In International Conference on 3D Vision (3DV), 2021.
  * [48] Michael Niemeyer and Andreas Geiger.  GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields.  In CVPR, 2021.
  * [49] Michael Oechsle, Songyou Peng, and Andreas Geiger.  UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction.  In ICCV, October 2021.
  * [50] Xingang Pan, Xudong Xu, Chen Change Loy, Christian Theobalt, and Bo Dai.  A shading-guided generative implicit model for shape-accurate 3d-aware image synthesis.  In Advances in Neural Information Processing Systems (NeurIPS), 2021\.
  * [51] Jeong Joon Park, Pete Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation.  In CVPR, 2019.
  * [52] Keunhong Park, Utkarsh Sinha, Jonathan Barron, Sofien Bouaziz, Dan Goldman, Steven Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable Neural Radiance Fields.  In ICCV, October 2021.
  * [53] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz.  Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields.  ACM Trans. Graph., 2021.
  * [54] Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter.  A 3d face model for pose and illumination invariant face recognition.  In IEEE International Conference on Advanced Video and Signal Based Surveillance, 2009.
  * [55] Jérémy Riviere, Paulo Gotardo, Derek Bradley, Abhijeet Ghosh, and Thabo Beeler.  Single-shot high-quality facial geometry and skin appearance capture.  In Siggraph, 2020.
  * [56] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  Advances in neural information processing systems, 29:2234–2242, 2016.
  * [57] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis.  In Adv. Neural Inform. Process. Syst., 2020.
  * [58] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  Graf: Generative radiance fields for 3d-aware image synthesis.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 20154–20166. Curran Associates, Inc., 2020.
  * [59] Sidak Pal Singh, Angela Fan, and Michael Auli.  GLOSS: Generative latent optimization of sentence representations.  arXiv preprint arXiv:1907.06385, 2019.
  * [60] Vincent Sitzmann, Eric R. Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein.  MetaSDF: Meta-learning signed distance functions.  In NeurIPS, 2020.
  * [61] Vincent Sitzmann, Semon Rezchikov, William T. Freeman, Joshua B. Tenenbaum, and Fredo Durand.  Light field networks: Neural scene representations with single-evaluation rendering.  In Proc. NeurIPS, 2021.
  * [62] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein.  Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations.  In Adv. Neural Inform. Process. Syst., 2019.
  * [63] Supasorn Suwajanakorn, Noah Snavely, Jonathan J Tompson, and Mohammad Norouzi.  Discovery of latent 3d keypoints via end-to-end geometric reasoning.  Advances in Neural Information Processing Systems, 31, 2018.
  * [64] Shufeng Tan and Michael L. Mayrovouniotis.  Reducing data dimensionality through optimizing neural network inputs.  AIChE Journal, 41(6):1471–1480, 1995.
  * [65] Ayush Tewari, Michael Zollhofer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Christian Theobalt.  Mofa: Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 1274–1283, 2017.
  * [66] Products - 3dMD.  https://3dmd.com/products/.  Accessed: 2021-10-25.
  * [67] Alex Trevithick and Bo Yang.  GRF: Learning a General Radiance Field for 3D Scene Representation and Rendering.  In ICCV, October 2021.
  * [68] Hsiao-Yu Fish Tung, Adam W Harley, William Seto, and Katerina Fragkiadaki.  Adversarial inverse graphics networks: Learning 2d-to-3d lifting and image-to-image translation from unpaired supervision.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 4364–4372. IEEE, 2017.
  * [69] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu.  NeRF–: Neural Radiance Fields Without Known Camera Parameters.  https://arxiv.org/abs/2102.07064, 2021.
  * [70] Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi.  Unsupervised learning of probably symmetric deformable 3d objects from images in the wild.  In CVPR, 2020.
  * [71] Christopher Xie, Keunhong Park, Ricardo Martin-Brualla, and Matthew Brown.  FiG-NeRF: Figure-ground neural radiance fields for 3d object category modelling.  arXiv preprint arXiv:2104.08418, 2021.
  * [72] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelNeRF: Neural Radiance Fields from One or Few Images.  In CVPR, 2021.
  * [73] Zhixuan Yu, Jae Shin Yoon, In Kyu Lee, Prashanth Venkatesh, Jaesik Park, Jihun Yu, and Hyun Soo Park.  HUMBI: A large multiview dataset of human body expressions.  In CVPR, 2020.
  * [74] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.  NERF++: Analyzing and improving neural radiance fields.  https://arxiv.org/abs/2010.07492, 2020.
  * [75] Xing Zhang, Lijun Yin, Jeffrey F. Cohn, Shaun Canavan, Michael Reale, Andy Horowitz, and Peng Liu.  A high-resolution spontaneous 3D dynamic facial expression database.  Image and Vision Computing (special issue of The Best of Face and Gesture 2013), 32:692–706, 2014.

LOLNeRF: Learn from One Look
Supplementary Material

##  Appendix A Qualitative Results – Figures 11, 12

We provide additional qualitative comparisons with $\pi$-GAN [9] in Figures 11
and 12, which show novel views at high angles and high resolution renders,
respectively.

Please also see the project website (https://lolnerf.github.io) for animations
which demonstrate the high-quality 3D structure and novel views that our
method produces.

\begin{overpic}[width=433.62pt]{fig/rebuttal_angles.png}
\put(8.0,34.0){\footnotesize{$0^{\circ}$}}
\put(22.0,34.0){\footnotesize{$30^{\circ}$}}
\put(40.0,34.0){\footnotesize{$60^{\circ}$}}
\put(58.0,34.0){\footnotesize{$0^{\circ}$}}
\put(72.0,34.0){\footnotesize{$30^{\circ}$}}
\put(90.0,34.0){\footnotesize{$60^{\circ}$}}
\put(20.0,-3.0){\footnotesize{LOLNeRF}}
\put(70.0,-3.0){\footnotesize{$\pi$-GAN}} \end{overpic} Figure 11:  High view
angles – both methods show degraded quality at high angles, but ours maintains
better sharpness and view-consistency.
\begin{overpic}[width=433.62pt]{fig/highres.png}
\put(71.0,-2.0){\footnotesize\rotatebox{0.0}{$\pi$-GAN\leavevmode\nobreak\
\cite[cite]{[\@@bibref{}{Chan21cvpr_piGAN}{}{}]}}}
\put(20.0,-2.0){\footnotesize\rotatebox{0.0}{Our Method}} \end{overpic} Figure
12:  High resolution renders – Our method is trained natively to reconstruct
high resolution images and can reconstruct sharp details, while $\pi$-GAN [9]
reconstructions lack detail beyond its training resolution and contain
artifacts that become easily visible.

##  Appendix B Camera Fitting Procedure

For a class-specific landmarker which provides estimates for $M$ 2D landmarks
$\mathbf{\ell}\in\mathbb{R}^{M\times 2}$, we estimate the extrinsics
$\mathbf{T}$ and (optionally) intrinsics $\mathbf{K}$ of a camera which
minimizes the reprojection error between $\mathbf{\ell}$ and a set of
canonical 3D positions $\mathbf{p}\in\mathbb{R}^{M\times 3}$. We achieve this
by solving the following least-squares optimization:

$\operatorname*{arg\,min}_{\mathbf{T},\mathbf{K}}||\mathbf{\ell}-P(\mathbf{p}|\mathbf{T},\mathbf{K})||^{2}$
(10)

where $P(\mathbf{x}|\mathbf{T},\mathbf{K})$ represents the projection
operation for a world-space position vector $\mathbf{x}$ into image space. We
perform this optimization using the Levenberg–Marquardt algorithm [33]. The
canonical positions $\mathbf{p}$ may be either manually specified or derived
from data. For human faces we use a predetermined set of positions which
correspond to the known average geometry of the human face. For AFHQ, we
perform a version of the above optimization jointly across all images where
$\mathbf{p}$ is also a free variable, and constrained only to obey symmetry.

In our experiments we predict camera intrinsics for human face data, but use
fixed intrinsics for AFHQ where the landmarks are less effective in
constraining the focal length. For SRN cars, we use the camera intrinsics and
extrinsics provided with the dataset, though we note that semantically
consistent landmarkers do exist for this class of data [63].

##  Appendix C Dataset Size Ablation – Figure 13

\begin{overpic}[width=433.62pt]{fig/data_ablation.png}
\put(7.0,-4.0){\footnotesize 10} \put(21.0,-4.0){\footnotesize 50}
\put(34.0,-4.0){\footnotesize 100} \put(48.0,-4.0){\footnotesize 500}
\put(63.0,-4.0){\footnotesize 1k} \put(77.0,-4.0){\footnotesize 5k}
\put(90.0,-4.0){\footnotesize 10k} \end{overpic} Figure 13:  Dataset size
ablation – we show the behaviour of the method as the size of the training
dataset varies. Rows 1, 3, and 5 show the learned reconstruction of training
images from the predicted view for that image. Rows 2, 4, and 6 show the
models rendered from a novel view. The columns show the results as the total
number of training images is increased from ten to ten thousand. Note that
this ablation is performed with lower-capacity networks (256 neurons per
layer), both to avoid wasting energy in training and to more clearly show how
reconstruction quality changes with dataset size.

To quantify our method’s dependence on large amounts of data, we perform an
ablation study in which we train models with subsets of the full dataset. We
then show the novel view synthesis quality from these networks as a way of
determining how well they have generalized to reconstructing different views.
The results of this experiment are shown in Figure 13. We find a clear trade-
off in quality of the training image reconstruction and quality of the learned
3D structure as the dataset size increases. Very small datasets reconstruct
their training images with high accuracy, but produce completely unreasonable
geometry and novel views. As the number of training images increases, the
accuracy of reconstruction slowly decreases, but the predicted structure
generalizes to become much more consistent and geometrically reasonable.

##  Appendix D Unconditional Sampling – Figure 14

Method |  FID$\downarrow$ |  KID$\downarrow$ |  IS$\uparrow$
---|---|---|---
HoloGAN [46] | 39.7 | 2.91 | 1.89
GRAF [58] | 41.1 | 2.29 | 2.34
$\pi$-GAN [9] | 14.7 | 0.39 | 2.62
Ours | 128.2 | 0.11 | 2.34
Table 5: Unconditional Sampling Quality – Perceptual image distribution
quality metrics on CelebA for our model and baselines. The results for HoloGAN
and GRAF are taken from [9].

\begin{overpic}[width=214.63977pt,trim=0.0pt 0.0pt 694.59499pt
0.0pt,clip]{fig/unconditional.png} \put(34.0,-8.0){\footnotesize{Our Method}}
\end{overpic}

\begin{overpic}[width=214.63977pt,trim=694.59499pt 0.0pt 0.0pt
0.0pt,clip]{fig/unconditional.png}
\put(34.0,-8.0){\footnotesize{$\pi$-GAN\leavevmode\nobreak\
\cite[cite]{[\@@bibref{}{Chan21cvpr_piGAN}{}{}]}}} \end{overpic}

Figure 14:  Unconditional generation – Both methods produce samples that
resemble the training distribution in appearance and shape. Ours produce
sharper details in the central face region where the data is more consistent,
while $\pi$-GAN [9] produces more plausible hair and backgrounds due to its
adversarial training.

To evaluate the quality of unconditional samples generated by our PCA-based
sampling method, we compute three standard quality metrics for generative
image models on these renders: Frechet Inception Distace (FID) [24], Kernel
Inception Distance, (KID) [4], and Inception Score (IS) [56], the results of
which are shown in Table 5. We find that our method achieves an inception
score competitive with other 3D-aware GAN methods, indicating that we are able
to model a variety of facial appearances. Our result for the distribution
distance metrics, FID and KID, however, show opposing results with our method
doing far worse in FID but better in KID. The reason for this is not entirely
clear, but FID has been shown to be sensitive to noise [7], and details in the
peripheral areas of our generated images show more noise-like artifacts than
$\pi$-GAN [9]. Regardless, we do not necessarily expect to outperform
$\pi$-GAN [9] for these metrics as it is trained to produce images with high
perceptual quality as determined by a CNN, which is a better proxy for these
distribution metrics than our reconstruction loss.

##  Appendix E Architecture Details

Our architecture uses a standard NeRF backbone architecture as described in
[44] with a few modifications. In addition to the standard positional encoding
we condition the network on an additional latent code by concatenating it
alongside the positional encoding. For SRN cars and AFHQ we use the standard
256 neuron network width and 256-dimensional latents for this network, but we
increase to 1024 neurons and 2048-dimensional latents for our high-resolution
CelebA-HQ and FFHQ models. For our background model we use a 5-layer,
256-neuron relu MLP in all cases. During training, we use 128 samples per ray
for volume rendering with no hierarchical sampling.

##  Appendix F Training Details

We train each model for 500k iterations using a batch size of 32 pixels per
image, with a total of 4096 images included in each batch. For comparison,
with 2562 images, this compute budget would allow just 2 images per batch for
a GAN-based method which renders entire frames.

We train with an ADAM [31] optimizer using exponential decay for the learning
rate from $5\times 10^{-4}$ to $1\times 10^{-4}$. We run each training job
using 64 v4 Tensor Processing Unit chips, taking approximately 36 hours to
complete for our high resolution models.


# Depth-Regularized Optimization for 3D Gaussian Splatting in Few-Shot Images

Jaeyoung Chung Jeongtaek Oh Kyoung Mu Lee
ASRI, Department of ECE, Seoul National University, Seoul, Korea
{robot0321, ohjtgood<EMAIL_ADDRESS>
###### Abstract

In this paper, we present a method to optimize Gaussian splatting with a
limited number of images while avoiding overfitting. Representing a 3D scene
by combining numerous Gaussian splats has yielded outstanding visual quality.
However, it tends to overfit the training views when only a small number of
images are available. To address this issue, we introduce a dense depth map as
a geometry guide to mitigate overfitting. We obtained the depth map using a
pre-trained monocular depth estimation model and aligning the scale and offset
using sparse COLMAP feature points. The adjusted depth aids in the color-based
optimization of 3D Gaussian splatting, mitigating floating artifacts, and
ensuring adherence to geometric constraints. We verify the proposed method on
the NeRF-LLFF dataset with varying numbers of few images. Our approach
demonstrates robust geometry compared to the original method that relies
solely on images.

##  1 Introduction

Figure 1: The efficacy of depth regularization in a few-shot setting We
optimize Gaussian splats with a limited number of images, avoiding overfitting
through the geometry guidance estimated from the images. Please note that we
utilized only two images to create this 3D scene.

Reconstruction of three-dimensional space from images has long been a
challenge in the computer vision field. Recent advancements show the
feasibility of photorealistic novel view synthesis [3, 31], igniting research
into reconstructing a complete 3D space from images. Driven by progress in
computer graphics techniques and industry demand, particularly in sectors such
as virtual reality [14] and mobile [11], research on achieving high-quality
and high-speed real-time rendering has been ongoing. Among the recent notable
developments, 3D Gaussian Splatting (3DGS) [23] stands out through its
combination of high quality, rapid reconstruction speed, and support for real-
time rendering. 3DGS employs Gaussian attenuated spherical harmonic splats
[38, 12] with opacity as primitives to represent every part of a scene. It
guides the splats to construct a consistent geometry by imposing a constraint
on the splats to satisfy multiple images at the same time.

The approach of aggregating small splats for a scene provides the capability
to express intricate details, yet it is prone to overfitting due to its local
nature. 3DGS [24] optimizes independent splats according to multi-view color
supervision without global structure. Therefore, in the absence of a
sufficient quantity of images that can offer a global geometric cue, there
exists no precaution against overfitting. This issue becomes more pronounced
as the number of images used for optimizing a 3D scene is small. The limited
geometric information from a few number of images leads to an incorrect
convergence toward a local optimum, resulting in optimization failure or
floating artifacts as shown in Figure 1. Nevertheless, the capability to
reconstruct a 3D scene with a restricted number of images is crucial for
practical applications, prompting us to tackle the few-shot optimization
problem.

One intuitive solution is to supplement an additional geometric cue such as
depth. In numerous 3D reconstruction contexts [6], depth proves immensely
valuable for reconstructing 3D scenes by providing direct geometric
information. To obtain such robust geometric cues, depth sensors aligned with
RGB cameras are employed. Although these devices offer dense depth maps with
minimal error, the necessity for such equipment also presents obstacles to
practical applications.

Hence, we attain a dense depth map by adjusting the output of the depth
estimation network with a sparse depth map from the renowned Structure-from-
Motion (SfM), which computes the camera parameters and 3D feature points
simultaneously. 3DGS also uses SfM, particularly COLMAP [41], to acquire such
information. However, the SfM also encounters a notable scarcity in the
available 3D feature points when the number of images is few. The sparse
nature of the point cloud also makes it impractical to regularize all Gaussian
splats. Hence, a method for inferring dense depth maps is essential. One of
the methods to extract dense depth from images is by utilizing monocular depth
estimation models. While these models are able to infer dense depth maps from
individual images based on priors obtained from the data, they produce only
relative depth due to scale ambiguity. Since the scale ambiguity leads to
critical geometry conflicts in multi-view images, we need to adjust scales to
prevent conflicts between independently inferred depths. We show that this can
be done by fitting a sparse depth, which is a free output from COLMAP [41] to
an estimated dense depth map.

In this paper, we propose a method to represent 3D scenes using a small number
of RGB images leveraging prior information from a pre-trained monocular depth
estimation model [5] and a smoothness constraint. We adapt the scale and
offset of the estimated depth to the sparse COLMAP points, solving the scale
ambiguity. We use the adjusted depth as a geometry guide to assist color-based
optimization, reducing floating artifacts and satisfying geometry conditions.
We observe that even the revised depth helps guide the scene to geometrically
optimal solution despite its roughness. We prevent the overfitting problem by
incorporating an early stop strategy, where the optimization process stops
when the depth-guide loss starts to rise. Moreover, to achieve more stability,
we apply a smoothness constraint, ensuring that neighbor 3D points have
similar depths. We adopt 3DGS as our baseline and compare the performance of
our method in the NeRF-LLFF [30] dataset. We confirm that our strategy leads
to plausible results not only in terms of RGB novel-view synthesis but also 3D
geometry reconstruction. Through further experiments, we demonstrate the
influence of geometry cues such as depth and initial points on Gaussian
splatting. They significantly influence the stable optimization of Gaussian
splatting.

In summary, our contributions are as follows:

$\bullet$

We propose depth-guided Gaussian Splatting optimization strategy which enables
optimizing the scene with a few images, mitigating over-fitting issue. We
demonstrate that even an estimated depth adjusted with a sparse point cloud,
which is an outcome of the SfM pipeline, can play a vital role in geometric
regularization.

$\bullet$

We present a novel early stop strategy: halting the training process when
depth-guided loss suffers to drop. We illustrate the influence of each
strategy through thorough ablation studies.

$\bullet$

We show that the adoption of a smoothness term for the depth map directs the
model to finding the correct geometry. Comprehensive experiments reveal
enhanced performance attributed to the inclusion of a smoothness term.

##  2 Related Work

#### Novel view synthesis

Structure from motion (SfM) [46] and Multi-view stereo (MVS) [45] are
techniques for reconstructing 3D structures using multiple images, which have
been studied for a long time in the computer vision field. Among the
continuous developments, COLMAP [41] is a widely used representative tool.
COLMAP performs camera pose calibration and finds sparse 3D keypoints using
the epipolar constraint [22] of multi-view images. For more dense and
realistic reconstruction, deep learning based 3D reconstruction techniques
have been mainly studied. [21, 51, 31] Among them, Neural radiance fields
(NeRF) [31] is a representative method that uses a neural network as a
representation method. NeRF creates realistic 3D scenes using an MLP network
as a 3D space expression and volume rendering, producing many follow-up papers
on 3D reconstruction research. [44, 18, 3, 54, 47, 4] In particular, to
overcome slow speed of NeRF, many efforts continues to achieve real-time
rendering by utilizing explicit expression such as sparse voxels[27, 56, 43,
16], featured point clouds[52], tensor [10], polygon [11]. These
representations have local elements that operate independently, so they show
fast rendering and optimization speed. Based on this idea, various
representations such as Multi-Level Hierarchies [32, 33], infinitesimal
networks [19, 39], triplane [9] have been attempted. Among them, 3D Gaussian
splatting [23] presented a fast and efficient method through alpha-blending
rasterization instead of time-consuming volume rendering. It optimizes a 3D
scene using multi-million Gaussian attenuated spherical harmonics with opacity
as a primitive, showing easy and fast 3D reconstruction with high quality.

Figure 2: Overview. We optimize the 3D Gaussian splatting [23] using dense
depth maps adjusted with the point clouds obtained from COLMAP [41]. By
incorporating depth maps to regulate the geometry of the 3D scene, our model
successfully reconstructs scenes using a limited number of images.

#### Few-shot 3D reconstruction

Since an image contains only partial information about the 3D scene, 3D
reconstruction requires a large number of multi-view images. COLMAP uploads
feature points matched between multiple images onto 3D space, so the more
images are used, the more reliable 3D points and camera poses can be
obtained.[41, 17] NeRF also optimizes the color and geometry of a 3D scene
based on the pixel colors of a large number of images to obtain high-quality
scenes. [48, 57] However, the requirements for many images hindered practical
application, sparking research on 3D reconstruction using only a few number of
images. Many few-shot 3D reconstruction studies utilize depth to provide
valuable geometric cue for creating 3D scenes. Depth helps reduce the effort
of inferring geometry through color consensus in multiple images by
introducing a surface smoothness constraint [25, 35], supervising the sparse
depth obtained from COLMAP [13, 49], using the dense depth obtained from
additional sensors [2, 7, 15], or exploiting estimated dense depth from
pretrained network. [37, 40, 34] These studies regularize geometry based on
the globality of the neural network, so it is difficult to apply them to
representations with large locality such as sparse voxel [16] or feature point
[52]. Instead, they attempted to establish connectivity between local elements
in a 3D space through the total variation (TV) loss [59, 16, 53], but it
requires exhaustive hyperparameter tuning of total variation which varies on
the scene and location. 3D Gaussian splatting [23] generates floating
artifacts with a few number of images, due to its strong locality. The sparse
COLMAP feature points that can be obtained from the Gaussian splat subprocess
are a free depth guide that can be obtained without additional information
[40], but the number of sparse points obtained from a small number of images
is so small that it cannot guide all Gaussian splats with strong locality. We
use a coarse geometry guide for optimization through a pretrained depth
estimation model [5, 58, 29]. Even if they do not have an exact fine-detailed
depth, they provide a rough guide to the location of splats, which greatly
contributes to optimization stability in few-shot situations and helps
eliminate floating artifacts that occur in random locations.

##  3 Method

Our method facilitates the optimization from a small set of images
$\\{I_{i}\\}_{i=0}^{k-1},I_{i}\in[0,1]^{H\times W\times 3}$. As a
preprocessing, we run SfM (such as COLMAP[41]) pipeline and get the camera
pose $\mathrm{R_{i}}\in\mathbb{R}^{3\times
3},\mathrm{t_{i}}\in\mathbb{R}^{3}$, intrinsic parameters
$K_{i}\in\mathbb{R}^{3\times 3}$, and a point cloud $P\in\mathbb{R}^{n\times
3}$. With those informations, we can easily obtain a sparse depth map for each
image, by projecting all visible point to pixel space:

$\displaystyle p={P_{\text{homog}}}[\mathrm{R_{i}}|\mathrm{t_{i}}],$ (1)
$\displaystyle\text{and}\quad D_{sparse,i}=p_{z}\in[0,\infty]^{H\times W}.$
(2)

Our approach builds upon 3DGS [23]. They optimize the Gaussian splats based on
the rendered image with a color loss $\mathcal{L}_{color}$ and D-SSIM loss
$\mathcal{L}_{D-SSIM}$. Prior to the 3DGS optimization, we estimate a depth
map for each image using a depth estimation network and fit the sparse depth
mapSection 3.1). We render a depth from the set of Gaussian splattings
leveraging the color rasterization process and add a depth constraint using
the dense depth prior (Section 3.2). We add an additional constraint for
smoothness between depths of adjacent pixels (Section 3.3) and refine
optimization options for few-shot settings (Section 3.4).

###  3.1 Preparing Dense Depth Prior

With the goal of guiding the splats into plausible geometry, we require to
provide global geometry information due to the locality of Gaussian splats.
The density depth is one of the promising geometry prior, but there is a
challenge in constructing it. The density of SfM points depends on the number
of images, so the number of valid points are too small to directly estimate
dense depth in a few-shot setting. (For example, SfM reconstruction from 19
images creates a sparse depth map with 0.04% valid pixels on average. [40])
Even the latest depth completion models fail to complete dense depth due to
the significant information gap.

When designing the depth prior, it is important to note that even rough depth
significantly aids in guiding the splats and eliminating artifacts resulting
from splats trapped in incorrect geometry. Hence, we employ a state-of-the-art
monocular depth estimation model and scale matching to provide a coarse dense
depth guide for optimization. From a train image $I$, the monocular depth
estimation model $F_{\theta}$ outputs dense depth $D_{\text{dense}}$,

$\displaystyle D_{\text{dense}}=s\cdot F_{\theta}(I)+t.$ (3)

To resolve the scale ambiguity in the estimated dense depth
$D_{\text{dense}}$, we adjust the scale $s$ and offset $t$ of estimated depth
to sparse SfM depth $D_{\text{sparse}}$:

$\displaystyle s^{\ast},t^{\ast}=\arg\min\limits_{s,t}\sum_{p\in
D_{\text{sparse}}}{\begin{Vmatrix}w(p){\cdot}D_{\text{sparse}}(p)-D_{\text{dense}}(p;s,t)\end{Vmatrix}^{2}},$
(4)

where $w\in[0,1]$ is a normalized weight presenting the reliability of each
feature points calculated as the reciprocal of the reprojection error from
SfM. Finally, we use the adjusted dense depth
$D^{\ast}_{\text{dense}}=s^{\ast}\cdot F_{\theta}(I)+t^{\ast}$ to regularize
the optimization loss of Gaussian splatting.

###  3.2 Depth Rendering through Rasterization

3D Gaussian splatting utilizes a rasterization pipeline [1] to render the
disconnected and unstructured splats leveraged on the parallel architecture of
GPU. Based on differentiable point-based rendering techniques [50, 55, 26],
they render an image by rasterizing the splats through $\alpha$-blending.
Point-based approaches exploit a similar equation to NeRF-style volume
rendering, rasterizing a pixel color with ordered points that cover that
pixel,

$\displaystyle C=\sum_{i\in\textit{N}}{c_{i}\alpha_{i}T_{i}}$ (5)
$\displaystyle\text{where}\quad{T_{i}=\prod_{j=1}^{i-1}(1-\alpha_{j})},$

$C$ is the pixel color, $c$ is the color of splats, and $\alpha$ here is
learned opacity multiplied by the covariance of 2D Gaussian. This formulation
prioritizes the color $c$ of opaque splat positioned closer to the camera,
significantly impacting the final outcome $C$. Inspired by the depth
implementation in NeRF, we leverage the rasterization pipeline to render the
depth map of Gaussian splats,

$\displaystyle D=\sum_{i\in\textit{N}}{d_{i}\alpha_{i}T_{i}},$ (6)

where $D$ is the rendered depth and $d_{i}=(R_{i}p_{i}+T_{i})_{z}$ is the
depth of each splat from the camera. Eqn. (6) enables the direct utilization
of $\alpha_{i}$ and $T_{i}$ calculated in Eqn. (5), facilitating rapid depth
rendering with minimal computational load. Finally, we guide the rendered
depth to the estimated dense depth using L1 distance,

$\displaystyle\mathcal{L}_{depth}=\begin{Vmatrix}D-D^{\ast}_{\text{dense}}\end{Vmatrix}_{1}$
(7)

###  3.3 Unsupervised Smoothness Constraint

Even though each independently estimated depth was fitted to the COLMAP
points, conflicts often arise. We introduce an unsupervised constraint for
geometry smoothness inspired by [20] to regularize the conflict. This
constraint implies that points in similar 3D positions have similar depths on
the image plane. We utilize the Canny edge detector [8] as a mask to ensure
that it does not regularize the area with significant differences in depth
along the boundaries. For a depth $d_{i}$ and its adjacent depth $d_{j}$, we
regularize the difference between them:

$\displaystyle\mathcal{L}_{smooth}=\sum_{d_{j}\in\text{adj}(d_{i})}\mathbbm{1}_{ne}{(d_{i},d_{j})}{\cdot}\begin{Vmatrix}d_{i}-d_{j}\end{Vmatrix}^{2}$
(8)

where $\mathbbm{1}_{ne}$ is a indicator function that signifies whether both
depths are not in edge.

We conclude the final loss terms by incorporating the depth loss from Eqn. (7)
and smoothness loss and smoothness loss from Eqn. (8) with their own
hyperparameters $\lambda_{depth}$ and $\lambda_{smooth}$:

$\displaystyle\mathcal{L}=(1{-}\lambda_{ssim})\mathcal{L}_{color}$
$\displaystyle+\lambda_{ssim}\mathcal{L}_{D-SSIM}$ (9)
$\displaystyle+\lambda_{depth}\mathcal{L}_{depth}$
$\displaystyle+\lambda_{smooth}\mathcal{L}_{smooth}$

where the preceding two loss terms $\mathcal{L}_{color},\mathcal{L}_{D-SSIM}$
correspond to the original 3D Gaussian splatting losses. [23]

| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---
2-view | 3-view | 4-view | 5-view | 2-view | 3-view | 4-view | 5-view | 2-view | 3-view | 4-view | 5-view
NeRF-LLFF[31] | Fern | 3DGS | 13.03 | 14.29 | 16.73 | 18.59 | 0.336 | 0.408 | 0.517 | 0.603 | 0.476 | 0.389 | 0.296 | 0.217
Ours | 17.59 | 19.13 | 19.91 | 20.55 | 0.516 | 0.588 | 0.616 | 0.642 | 0.286 | 0.232 | 0.203 | 0.167
Oracle | 18.18 | 20.30 | 20.78 | 21.81 | 0.524 | 0.636 | 0.654 | 0.701 | 0.278 | 0.201 | 0.185 | 0.157
Flower | 3DGS | 14.90 | 17.75 | 19.71 | 21.39 | 0.351 | 0.508 | 0.605 | 0.671 | 0.406 | 0.257 | 0.190 | 0.146
Ours | 15.92 | 17.80 | 19.15 | 20.45 | 0.395 | 0.445 | 0.538 | 0.576 | 0.414 | 0.376 | 0.323 | 0.293
Oracle | 19.71 | 22.16 | 23.26 | 24.65 | 0.570 | 0.673 | 0.714 | 0.760 | 0.250 | 0.163 | 0.128 | 0.097
Fortress | 3DGS | 13.87 | 15.98 | 19.26 | 19.98 | 0.363 | 0.492 | 0.609 | 0.631 | 0.389 | 0.283 | 0.201 | 0.191
Ours | 19.80 | 21.85 | 23.07 | 23.72 | 0.567 | 0.655 | 0.724 | 0.740 | 0.232 | 0.191 | 0.162 | 0.144
Oracle | 23.07 | 24.51 | 26.39 | 26.73 | 0.654 | 0.728 | 0.787 | 0.797 | 0.159 | 0.130 | 0.100 | 0.093
Horns | 3DGS | 11.43 | 12.48 | 13.76 | 14.75 | 0.264 | 0.339 | 0.433 | 0.498 | 0.531 | 0.464 | 0.395 | 0.350
Ours | 15.91 | 16.22 | 18.09 | 18.39 | 0.420 | 0.466 | 0.527 | 0.565 | 0.362 | 0.349 | 0.306 | 0.296
Oracle | 18.56 | 20.08 | 20.88 | 22.52 | 0.568 | 0.644 | 0.668 | 0.725 | 0.259 | 0.212 | 0.199 | 0.168
Leaves | 3DGS | 12.33 | 12.36 | 12.49 | 12.26 | 0.260 | 0.275 | 0.298 | 0.297 | 0.412 | 0.397 | 0.397 | 0.401
Ours | 13.04 | 13.63 | 13.97 | 14.13 | 0.235 | 0.270 | 0.283 | 0.297 | 0.460 | 0.445 | 0.440 | 0.438
Oracle | 13.52 | 14.23 | 14.78 | 14.85 | 0.287 | 0.353 | 0.377 | 0.397 | 0.380 | 0.348 | 0.341 | 0.356
Orchids | 3DGS | 11.78 | 13.94 | 15.41 | 16.08 | 0.182 | 0.320 | 0.416 | 0.460 | 0.426 | 0.310 | 0.245 | 0.219
Ours | 12.88 | 14.71 | 15.40 | 16.13 | 0.216 | 0.297 | 0.343 | 0.391 | 0.462 | 0.383 | 0.366 | 0.352
Oracle | 14.89 | 16.45 | 17.42 | 18.45 | 0.365 | 0.471 | 0.525 | 0.576 | 0.303 | 0.237 | 0.200 | 0.174
Room | 3DGS | 10.18 | 11.51 | 11.59 | 12.21 | 0.404 | 0.494 | 0.510 | 0.552 | 0.606 | 0.559 | 0.556 | 0.515
Ours | 17.21 | 18.11 | 18.87 | 19.63 | 0.668 | 0.719 | 0.732 | 0.757 | 0.352 | 0.360 | 0.326 | 0.295
Oracle | 20.66 | 22.31 | 23.80 | 24.59 | 0.758 | 0.801 | 0.839 | 0.864 | 0.217 | 0.188 | 0.160 | 0.156
Trex | 3DGS | 10.72 | 11.72 | 13.11 | 14.14 | 0.322 | 0.417 | 0.492 | 0.548 | 0.520 | 0.446 | 0.394 | 0.351
Ours | 14.90 | 15.90 | 16.75 | 17.37 | 0.480 | 0.537 | 0.567 | 0.625 | 0.358 | 0.362 | 0.348 | 0.305
Oracle | 17.76 | 19.58 | 20.84 | 22.83 | 0.591 | 0.669 | 0.714 | 0.786 | 0.284 | 0.226 | 0.192 | 0.134
Mean | 3DGS | 12.25 | 13.75 | 15.26 | 16.17 | 0.306 | 0.407 | 0.485 | 0.533 | 0.471 | 0.388 | 0.334 | 0.299
Ours | 15.94 | 17.17 | 18.15 | 18.74 | 0.439 | 0.497 | 0.541 | 0.571 | 0.365 | 0.337 | 0.309 | 0.288
Oracle | 18.29 | 19.95 | 21.02 | 22.05 | 0.539 | 0.622 | 0.660 | 0.701 | 0.266 | 0.213 | 0.188 | 0.167

Table 1: Quantitative results in NeRF-LLFF [30] dataset. The best performance
except oracle is bolded.

###  3.4 Modification for Few-Shot Learning

We modify two optimization techniques from the original paper to create 3D
scenes with limited images. The techniques employed in 3DGS were designed
under the assumption of utilizing a substantial number of images, potentially
hindering convergence in a few-shot setting. Through iterative experiments, we
confirm this and modify the techniques to suit the few-shot setting. Firstly,
we set the maximum degree of spherical harmonics (SH) to 1. This prevents
overfitting of spherical harmonic coefficients responsible for high
frequencies due to insufficient information. Secondly, we implement an early-
stop policy based on depth loss. We configure Eqn. (9) to be primarily driven
by color loss, while employing the depth loss and the smoothness loss as
guiding factors. Hence, overfitting gradually emerges due to the predominant
influence of color loss. We use a moving averaged depth loss to halt
optimization when the splats start to deviate from the depth guide. Lastly, we
remove the periodic reset process. We observe that resetting the opacity
$\alpha$ of all splats leads to irreversible and detrimental consequences. Due
to a lack of information from the limited images, the inability to restore the
opacity of splats led to scenarios where either all splats were removed or
trapped in local optima, causing unexpected outcomes and optimization
failures. As a result of the aforementioned techniques, we achieve stable
optimization in few-shot learning.

##  4 Experiment

###  4.1 Experiment settings

#### Datasets.

We evaluate our method on NeRF-LLFF [30] dataset. NeRF-LLFF includes 8 scenes
with forward-facing cameras, and we split the images of each scene into train
and test sets. We use the image outer edge of the camera group as the train
set based on the convex hull algorithm [36] due to the forward-facing camera
distribution. For each experiment, we optimize the scene with k-shot
(k=2,3,4,5) images randomly selected from the train set and evaluate on the
same test set. We use ten randomly selected seeds and report the average of
ten experiments.

|  (iv) Horns |  (iii) Fortress |  (ii) Room |  (i) Fern
---|---|---|---|---

|  5-view |  2-view |  5-view |  2-view |  5-view |  2-view |  5-view |  2-view
---|---|---|---|---|---|---|---|---

(a)

(b)

(c)

(d)

(e)

(a)

(b)

(c)

(d)

(e)

(a)

(b)

(c)

(d)

(e)

(a)

(b)

(c)

(d)

(e)

(a)

(b)

(c)

(d)

(e)

(a)

(b)

(c)

(d)

(e)

(a)

(b)

(c)

(d)

(e)

(a)

(b)

(c)

(d)

(e)

Figure 3: Qualitative comparison in NeRF-LLFF [30] dataset. We visualize the
distinction between 3D Gaussian Splatting (3DGS) [23] and our method in both
2-view and 5-view settings. Driven primarily by color loss, 3DGS struggled to
achieve desirable geometry. Our approach consistently established plausible
geometric structures with depth guidance, resulting in superior reconstruction
outcomes.

(a)

(b)

(c)

(d)

Figure 4: Details in cropped patches. (a) Input View (b) 3DGS [23] (c) Ours
(d) Ground Truth. Our method produces superior reconstruction results compared
to 3DGS [23], leveraging additional geometric cues. Our method establishes
stable geometry, outperforming 3DGS in reconstruction quality.

#### Implementation details.

For a fair comparison among different options, it is essential to use unified
coordinates in each scene and standardize the evaluation values. We achieve
this by processing the entire images of a scene through COLMAP to obtain
consistent camera poses and feature points, selecting those relevant to each
k-shot experiment. We select $k$ cameras from the train set and extract the
feature points that are visible in at least three out of the $k$ cameras. We
use these feature points as depth guidance $D_{\text{sparse}}$ in Eqn. (4) and
initial points for Gaussian splatting optimization. In the baseline(3DGS), we
use the same $k$ camera poses and the same filtered initial points, reporting
the evaluation values at 30k iterations like in the original setup. For the
oracle, we aim to illustrate the effectiveness of precise depth. We create
pseudo-GT depth by optimizing the entire images of both the train and test. We
replace the estimated depth from our method with pseudo-GT depth and report
the results in the oracle. Lastly, we implement the differentiable depth
rasterizer outlined in Eqn. (6) based on CUDA.

###  4.2 Experiment results

We present the comparison results of 3DGS, our method, and oracle for NeRF-
LLFF scenes in Table 1. Across all methods and scenes, a decrease in the
number of used images consistently results in lower visual quality. Our method
typically demonstrates superior results compared to 3DGS, particularly when
the number of images is limited. Figure 3 visualize the difference between
3DGS and our method. The depth map highlights the geometry failure of 3DGS in
the few-shot case. For instance, in the 2-view of the Fern, it displays
entirely erroneous geometry compared to the similarity in RGB. In har
conditions like the 2-view scenario, 3DGS often fails to form appropriate
geometry. In contrast, our method forms plausible geometry while generating an
attractive image. We present additional examples in Figure 4. The cropped
patches demonstrate that our method achieves better results through depth
guidance. Hence, we confirm that the geometric cues provided by depth become
significantly beneficial for the reconstruction in Gaussian splatting,
especially when the number of images is limited. This fact is reaffirmed by
the remarkably high performance of the oracle, which employs accurate
geometry. The example images of the oracle demonstrate the effectiveness of
accurate depth, as depicted in Figure 5. The rich information provided by
pseudo-GT depth enables the creation of detailed and reliable results even
with a limited number of images.

An important observation to note is the substantial reliance of our approach
on the pre-trained monocular depth estimation model. We exploit the pre-
trained model of ZoeDepth [5] trained on the indoor dataset NYU Depth v2 [42]
and urban dataset KITTI [28]. As a result, our model reports relatively higher
performance in indoor scenes (Fortress, Room, Fern) and comparatively worse
results for natural scenes (Orchids, Flower). Note that the Leaves presents
challenges for COLMAP, leading to generally unsuccessful Gaussian splatting
training.

Method | 2-views | 5-views
---|---|---
PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
w/o Adjustment | 7.86 | 0.319 | 0.740 | 10.01 | 0.346 | 0.761
(Section 3.1)
w/o $\mathcal{L}_{depth}$ | 11.49 | 0.344 | 0.533 | 12.97 | 0.506 | 0.418
(Section 3.2)
w/o $\mathcal{L}_{smooth}$ | 14.75 | 0.415 | 0.391 | 17.79 | 0.561 | 0.297
(Section 3.3)
w/o early stop | 13.99 | 0.345 | 0.433 | 17.28 | 0.494 | 0.333
(Section 3.4)
Ours | 15.91 | 0.420 | 0.362 | 18.39 | 0.565 | 0.296

Table 2: Ablations. We describe the ablation studies on each element of the
proposed method. We also present an experiment supervising with sparse depth
from COLMAP instead of dense depth. The reported values are evaluated in
Horns.

Initialization | 2-views | 5-views
---|---|---
points | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
COLMAP from | 19.80 | 0.567 | 0.232 | 23.72 | 0.740 | 0.144
sparse-view (Ours)
Unprojected | 16.39 | 0.457 | 0.281 | 19.15 | 0.569 | 0.222
from $D^{\ast}_{\text{dense}}$
COLMAP | 21.18 | 0.681 | 0.200 | 24.09 | 0.778 | 0.127
from all-view

Table 3: Comparison of Initialization Methods. We describe the ablation
studies on each element of the proposed method. We also present an experiment
supervising with sparse depth from COLMAP instead of dense depth. The reported
values are evaluated in Fortress.  Figure 5: Example results utilizing pseudo-
GT depth (oracle). Accurate depth facilitates high-quality 3D reconstruction,
even with a limited number of images. Fine details are perceptible in both RGB
and depth.

###  4.3 Ablations

We present ablation studies for each component of our proposed method in Table
2. The first and second rows demonstrate the necessity of absolute depth
guidance. Without the adjustment process in Section 3.1, the dense Depth
$D_{\text{dense}}$ has an incorrect scale from the monocular depth estimation
model. The depth is misaligned with the camera intrinsic parameters from
COLMAP, leading to complete training failure. We also observed optimization
failure when solely utilizing unsupervised smooth constraints without depth
supervision introduced in Section 3.2. The application of smoothness
constraints without absolute geometry supervision yields worse results
compared to the baseline. The third and fourth rows of Table 2 demonstrate the
degree of performance enhancement from additional techniques. With the depth
supervision $D^{\ast}_{\text{dense}}$, the smoothness constraints in Section
3.3 contribute to performance improvement by providing additional geometric
cues. Notably, the early stop mechanism introduced in Section 3.4 plays a
pivotal role in preventing performance degradation within our approach. By
leveraging depth loss, it scrutinizes the divergence of splats from the
prescribed geometry guide, effectively halting potential instances of
overfitting.

In Table 3, we compared the performances between the different Gaussian
splatting initializations. The second row illustrates the outcomes when
utilizing a point cloud produced by unprojecting dense depth
$D^{\ast}_{\text{dense}}$ as initialization points. The numerous initial
points generated through unprojection are not effectively merged or pruned,
resulting in lower performance compared to the sparse COLMAP initialization.
In contrast, the outcomes depicted in the third row assume the utilization of
all COLMAP points. Employing a multitude of favorable initial points
unattainable with k images contributes to the enhancement of outcomes via
dense depth adjustment and Gaussian splatting initialization.

##  5 Limitation and Future Work

Our approach demonstrated the feasibility of Gaussian splatting optimization
in a few-shot setting through depth guidance, yet it has limitations. Firstly,
it heavily relies on the estimation performance of the monocular depth
estimation model. Moreover, this model’s depth estimation performance can vary
based on the learned data domain, consequently affecting the performance of
Gaussian splatting optimization. Additionally, relying on fitting the
estimated depth to COLMAP points means a dependency on COLMAP’s performance,
rendering it incapable of handling textureless plains or challenging surfaces
where COLMAP might fail. We leave as future work the optimization of 3D scenes
by interdependent estimated depths rather than COLMAP points. Also, exploring
methods to regularize geometry across various datasets, including areas where
depth estimation, such as the sky, might be challenging, is another avenue for
future work.

##  6 Conclusion

In this work, we introduce Depth-Regularized Optimization for 3D Gaussian
Splatting in Few-Shot Images, a model for learning 3D Gaussian splatting with
a small number of images. Our model regularizes the splats using depth,
demonstrating the effectiveness of such geometric guidance. To acquire dense
depth guidance, we exploit a monocular depth estimation model and adjust the
depth scale based on SfM points. We examined the effectiveness of our proposed
depth loss, unsupervised smooth constraint, and early stop technique in the
NeRF-LLFF dataset. Our method outperforms 3D Gaussian splatting in a few-shot
setting, creating plausible geometry. Finally, we demonstrated through
additional experiments that improved depth and initialization points
significantly enhance the performance of Gaussian splatting-based 3D
reconstruction.

## References

  * Akenine-Möller et al. [2019] Tomas Akenine-Möller, Eric Haines, and Naty Hoffman.  _Real-time rendering_.  Crc Press, 2019.
  * Azinović et al. [2022] Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies.  Neural rgb-d surface reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Barron et al. [2021] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021.
  * Barron et al. [2022] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Bhat et al. [2023] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller.  Zoedepth: Zero-shot transfer by combining relative and metric depth.  _arXiv preprint arXiv:2302.12288_ , 2023.
  * Bian et al. [2023] Wenjing Bian, Zirui Wang, Kejie Li, Jiawang Bian, and Victor Adrian Prisacariu.  Nope-nerf: Optimising neural radiance field with no pose prior.  2023\.
  * Cai et al. [2022] Hongrui Cai, Wanquan Feng, Xuetao Feng, Yan Wang, and Juyong Zhang.  Neural surface reconstruction of dynamic scenes with monocular rgb-d camera.  _Advances in Neural Information Processing Systems_ , 2022.
  * Canny [1986] John Canny.  A computational approach to edge detection.  _IEEE Transactions on pattern analysis and machine intelligence_ , 1986.
  * Chan et al. [2022] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In _CVPR_ , 2022.
  * Chen et al. [2022] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.  Tensorf: Tensorial radiance fields.  In _ECCV_ , 2022.
  * Chen et al. [2023] Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi.  Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures.  In _CVPR_ , 2023.
  * Crawfis and Max [1993] Roger A Crawfis and Nelson Max.  Texture splats for 3d scalar and vector field visualization.  In _Proceedings Visualization’93_ , 1993.
  * Deng et al. [2022a] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan.  Depth-supervised nerf: Fewer views and faster training for free.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022a.
  * Deng et al. [2022b] Nianchen Deng, Zhenyi He, Jiannan Ye, Budmonde Duinkharjav, Praneeth Chakravarthula, Xubo Yang, and Qi Sun.  Fov-nerf: Foveated neural radiance fields for virtual reality.  _IEEE Transactions on Visualization and Computer Graphics_ , 2022b.
  * Dey et al. [2022] Arnab Dey, Yassine Ahmine, and Andrew I Comport.  Mip-nerf rgb-d: Depth assisted fast neural radiance fields.  _arXiv preprint arXiv:2205.09351_ , 2022.
  * Fridovich-Keil et al. [2022] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  In _CVPR_ , 2022.
  * Furukawa et al. [2015] Yasutaka Furukawa, Carlos Hernández, et al.  Multi-view stereo: A tutorial.  _Foundations and Trends® in Computer Graphics and Vision_ , 2015.
  * Gao et al. [2022] Kyle Gao, Yina Gao, Hongjie He, Dening Lu, Linlin Xu, and Jonathan Li.  Nerf: Neural radiance field in 3d vision, a comprehensive review.  _arXiv preprint arXiv:2210.00379_ , 2022.
  * Garbin et al. [2021] Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin.  Fastnerf: High-fidelity neural rendering at 200fps.  In _ICCV_ , 2021.
  * Godard et al. [2017] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow.  Unsupervised monocular depth estimation with left-right consistency.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017.
  * Han et al. [2019] Xian-Feng Han, Hamid Laga, and Mohammed Bennamoun.  Image-based 3d object reconstruction: State-of-the-art and trends in the deep learning era.  _PAMI_ , 2019.
  * Hartley and Zisserman [2003] Richard Hartley and Andrew Zisserman.  _Multiple view geometry in computer vision_.  Cambridge university press, 2003.
  * Kerbl et al. [2023a] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.  3d gaussian splatting for real-time radiance field rendering.  _TOG_ , 2023a.
  * Kerbl et al. [2023b] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.  3d gaussian splatting for real-time radiance field rendering.  _ACM Transactions on Graphics_ , 2023b.
  * Kim et al. [2022] Mijeong Kim, Seonguk Seo, and Bohyung Han.  Infonerf: Ray entropy minimization for few-shot neural volume rendering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Kopanas et al. [2021] Georgios Kopanas, Julien Philip, Thomas Leimkühler, and George Drettakis.  Point-based neural rendering with per-view optimization.  In _Computer Graphics Forum_ , 2021.
  * Liu et al. [2020] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt.  Neural sparse voxel fields.  _NIPS_ , 2020.
  * Menze and Geiger [2015] Moritz Menze and Andreas Geiger.  Object scene flow for autonomous vehicles.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015.
  * Mertan et al. [2022] Alican Mertan, Damien Jade Duff, and Gozde Unal.  Single image depth estimation: An overview.  _Digital Signal Processing_ , 2022.
  * Mildenhall et al. [2019] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  _ACM Transactions on Graphics (TOG)_ , 2019.
  * Mildenhall et al. [2021] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Communications of the ACM_ , 2021.
  * Müller et al. [2021] Thomas Müller, Fabrice Rousselle, Jan Novák, and Alexander Keller.  Real-time neural radiance caching for path tracing.  _arXiv preprint arXiv:2106.12372_ , 2021.
  * Müller et al. [2022] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  _TOG_ , 2022.
  * Neff et al. [2021] Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H Mueller, Chakravarty R Alla Chaitanya, Anton Kaplanyan, and Markus Steinberger.  Donerf: Towards real-time rendering of compact neural radiance fields using depth oracle networks.  In _Computer Graphics Forum_ , 2021.
  * Niemeyer et al. [2022] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan.  Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Preparata and Shamos [2012] Franco P Preparata and Michael I Shamos.  _Computational geometry: an introduction_.  Springer Science & Business Media, 2012.
  * Prinzler et al. [2023] Malte Prinzler, Otmar Hilliges, and Justus Thies.  Diner: Depth-aware image-based neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Reed [2012] Michael Reed.  _Methods of modern mathematical physics: Functional analysis_.  Elsevier, 2012.
  * Reiser et al. [2021] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger.  Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps.  In _ICCV_ , 2021.
  * Roessle et al. [2022] Barbara Roessle, Jonathan T Barron, Ben Mildenhall, Pratul P Srinivasan, and Matthias Nießner.  Dense depth priors for neural radiance fields from sparse input views.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Schonberger and Frahm [2016] Johannes L Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In _CVPR_ , 2016.
  * Silberman et al. [2012] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  In _Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12_ , 2012.
  * Sun et al. [2022] Cheng Sun, Min Sun, and Hwann-Tzong Chen.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction.  In _CVPR_ , 2022.
  * Tewari et al. [2022] Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Wang Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, et al.  Advances in neural rendering.  In _Computer Graphics Forum_ , 2022.
  * Tomasi and Kanade [1992] Carlo Tomasi and Takeo Kanade.  Shape and motion from image streams under orthography: a factorization method.  _International journal of computer vision_ , 1992.
  * Ullman [1979] Shimon Ullman.  The interpretation of structure from motion.  _Proceedings of the Royal Society of London. Series B. Biological Sciences_ , 1979.
  * Wang et al. [2021a] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  _arXiv preprint arXiv:2106.10689_ , 2021a.
  * Wang et al. [2021b] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu.  Nerf–: Neural radiance fields without known camera parameters.  _arXiv preprint arXiv:2102.07064_ , 2021b.
  * Wei et al. [2021] Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, and Jie Zhou.  Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021.
  * Wiles et al. [2020] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson.  Synsin: End-to-end view synthesis from a single image.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * Xie et al. [2022] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar.  Neural fields in visual computing and beyond.  In _Computer Graphics Forum_ , 2022.
  * Xu et al. [2022] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann.  Point-nerf: Point-based neural radiance fields.  In _CVPR_ , 2022.
  * Yang et al. [2023] Chen Yang, Peihao Li, Zanwei Zhou, Shanxin Yuan, Bingbing Liu, Xiaokang Yang, Weichao Qiu, and Wei Shen.  Nerfvs: Neural radiance fields for free view synthesis via geometry scaffolds.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Yariv et al. [2021] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman.  Volume rendering of neural implicit surfaces.  _Advances in Neural Information Processing Systems_ , 2021.
  * Yifan et al. [2019] Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine-Hornung.  Differentiable surface splatting for point-based geometry processing.  _ACM Transactions on Graphics (TOG)_ , 2019.
  * Yu et al. [2021] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.  Plenoctrees for real-time rendering of neural radiance fields.  In _ICCV_ , 2021.
  * Zhang et al. [2020] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.  Nerf++: Analyzing and improving neural radiance fields.  _arXiv preprint arXiv:2010.07492_ , 2020.
  * Zhao et al. [2023] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu.  Unleashing text-to-image diffusion models for visual perception.  _arXiv preprint arXiv:2303.02153_ , 2023.
  * Zhou et al. [2017] Chao Zhou, Hong Zhang, Xiaoyong Shen, and Jiaya Jia.  Unsupervised learning of stereo matching.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 2017.