# InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering

Mijeong Kim1 Seonguk Seo1 Bohyung Han1,2
1ECE & 1ASRI & 1,2IPAI, Seoul National University
{mijeong.kim, seonguk<EMAIL_ADDRESS>
###### Abstract

We present an information-theoretic regularization technique for few-shot
novel view synthesis based on neural implicit representation. The proposed
approach minimizes potential reconstruction inconsistency that happens due to
insufficient viewpoints by imposing the entropy constraint of the density in
each ray. In addition, to alleviate the potential degenerate issue when all
training images are acquired from almost redundant viewpoints, we further
incorporate the spatial smoothness constraint into the estimated images by
restricting information gains from additional rays with slightly different
viewpoints. The main idea of our algorithm is to make reconstructed scenes
compact along individual rays and consistent across rays in the neighborhood.
The proposed regularizers can be plugged into most of existing neural volume
rendering techniques based on NeRF in a straightforward way. Despite its
simplicity, we achieve consistently improved performance compared to existing
neural view synthesis methods by large margins on multiple standard
benchmarks. Our codes and models are available in the project
website111http://cvlab.snu.ac.kr/research/InfoNeRF.

Figure 1:  Qualitative comparison of InfoNeRF (ours) with other NeRF-based
models [21, 11, 38] on the Lego, Materials, and Ficus scenes of Realistic
Synthetic 360∘ dataset in 4-view settings. Existing works often suffer from
noise (b), color distortion (c), or blur effect (d), while InfoNeRF (ours)
achieves outstanding quality of rendered images with only a few input views.
The last column (f) visualizes depth maps estimated by InfoNeRF (ours), which
provide clear boundaries and fine details of the objects.

##  1 Introduction

Understanding 3D structure of a natural scene is a critical step for various
high-level computer vision applications including object recognition,
photorealistic rendering, autonomous driving, virtual reality, and many
others. Recent advance of deep learning capacitates high-fidelity 3D
reconstruction and recognition, but learning with 3D data is inherently more
difficult than its counterpart based on 2D images due to unstructured nature
of data format, high memory requirement, and lack of principled architectures.
Hence, many researchers investigate the standard models with appropriate
training algorithms and the methods for reducing their computational costs,
and attempt to solve various challenging tasks.

Novel view synthesis based on neural implicit representations is one of the 3D
learning tasks that draws a lot of attention these days since Neural Radiance
Field (NeRF) [21] has been introduced. NeRF delivers accurate 3D
reconstruction results without explicit modeling of 3D scene structures, but
the requirement of many images captured from multiple calibrated cameras
hampers the applicability of the method. Therefore, several recent approaches
aim to reduce the high computational cost and alleviate the constraints
related to datasets [7, 11, 38].

In this line of research, we explore the few-shot prior-free novel view
synthesis task, where only a limited number of training images are accessible
and other prior information, such as object categories and semantic structures
of target scenes, are unavailable. There exist several prior works for this
task, but they either work barely with few examples [11] or require narrow
baseline assumption to find correspondences using an external module [7].
Other approaches rely on prior knowledge of scenes such as object classes or
features. For example, PixelNeRF [38] takes advantage of the features
extracted from seen images to compensate for missing information in unseen
views while [24, 14] focus on a particular object class, _e.g._ , human, in
novel view synthesis.

We address the fundamental drawbacks of existing few-shot novel view synthesis
methods: inconsistent reconstruction, which generates noise, blur, or
artifacts in rendered images, and overfitting to seen views, which leads to
degenerate or trivial solutions. The proposed approach, referred to as
InfoNeRF, alleviates the reconstruction inconsistency by imposing the sparsity
on the estimated scene, which is achieved by entropy minimization in each ray.
The overfitting issues are handled by enforcing the smoothness of the
reconstruction with respect to viewpoint changes, which is controlled by
minimizing information gains from a pair of slightly different viewpoints.
Figure 1 illustrates the outstanding quality of rendered images and depth maps
estimated by our model, which delineates clear object boundaries and fine
structures using only 4 input views with wide baselines.

Overall, the main contributions and benefits of our algorithm are summarized
as follows:

  * •

We propose a novel information-theoretic approach, InfoNeRF, for the
regularization of the neural implicit representations for volume rendering.
Our method points out key drawbacks of the existing few-shot novel view
synthesis techniques, and introduces two effective regularization schemes, ray
entropy minimization and ray information gain reduction.

  * •

Since InfoNeRF is a generic regularization technique and does not require any
other external data structures, _e.g._ , voxels or meshes, or additional
learnable parameters, it can be applied to various neural volume rendering
algorithms with and without scene prior.

  * •

The proposed regularization technique turns out to be effective to alleviate
reconstruction inconsistency across multiple views and prevent degenerate
solutions by overfitting despite its simplicity. We demonstrate outstanding
performance of InfoNeRF on several standard benchmarks for few-shot novel view
synthesis.

  * •

To our knowledge, InfoNeRF is the first NeRF variant that performs few-shot
novel view synthesis on wide-baseline image datasets without prior
information.

##  2 Related Work

###  2.1 Novel View Synthesis

Novel view synthesis aims to render realistic images via geometric and
photometric understanding of a 3D scene given a set of training images. To
address this problem, light fields [15, 30, 36] or image-based rendering [6,
3, 5, 28, 4] approaches have been employed traditionally, and the approaches
based on deep learning [40, 41, 8, 9, 20] have recently received growing
attention. In particular, NeRF [21] achieves photo-realistic rendering results
by applying multi-layer perceptrons to differentiable volume rendering
successfully. The following works attempt to extend NeRF in various aspects,
such as dynamic view synthesis [16], self-calibrated view synthesis [34],
real-time rendering [37, 35, 22], relighting [29], and anti-aliasing [1].
Although NeRF-based models have achieved impressive performance, they have a
common drawback; they require dense scene sampling, making them difficult to
be applied in real-world scenarios. We address the few-shot volume rendering
task to improve applicability by reducing the need for many images captured by
calibrated cameras.

###  2.2 Few-shot Novel View Synthesis

To synthesize novel views of a scene given sparse observations, some
algorithms estimate depth maps from images since depth is valuable source for
view synthesis and 3D reconstruction. Depth information plays a crucial role
in depth-guided image interpolation [9, 25], multi-plane image prediction
[32], and learned geometry regularization in complicated scenes [7] for few-
shot view synthesis. However, these methods require training images with depth
supervision or external depth estimation modules, _e.g._ , multiview stereo or
COLMAP SfM [26], and are susceptible to large projection errors due to
incorrect depth predictions.

To overcome the limitation, several approaches exploit multi-view feature
semantics by introducing an image encoder for NeRF to estimate color and
opacity [38, 24, 14] or achieve semantic consistency between seen images and
rendered novel views [11]. These strategies facilitate learning the semantic
prior, and allow us to synthesize novel views with only few-shot images.
Unlike the aforementioned works, the proposed algorithm does not rely on any
prior information or additional pretrained encoders. Our regularization
technique is orthogonal to the other methods discussed above and can be
integrated into existing few-shot volume rendering approaches
straightforwardly.

Some explicit representation methods [37, 17] incorporate sparsity constraints
for neural volume rendering, which may also be useful for few-shot novel-view
synthesis, although they do not directly address the task. Yu  _et al._ [37]
adopt a variation of the octree structure with a sparsity prior loss to remove
a subset of nodes in the tree corresponding to unobserved regions. On the
other hand, Lombardi  _et al._ [17] conduct ray marching through voxel grids
for volume rendering and regularize the total variation of voxel opacities by
enforcing sparse spatial gradient. Our method also employ sparsity via entropy
constraints, but is more generic than the explicit methods because our
algorithm does not require external data structures and suffer from memory
bound to store explicit representations.

##  3 Preliminaries: NeRF

(a) Rendered RGB and depth images estimated by NeRF [21] on the Realistic
Synthetic 360∘ dataset in a 4-view setting.

(b) Rendered RGB images for two slightly different viewpoints (left: seen,
right: unseen) estimated by NeRF [21] on the DTU dataset in a 3-view setting.

Figure 2: Typical drawbacks of NeRF in the few-shot volume rendering. We are
motivated by the following two observations and alleviate the limitations by
adding two regularizers in this work. (a) Few-shot NeRF suffers from
significant noise in 3D depth estimation, which leads to noisy rendering. (b)
Few-shot NeRF provides completely different rendering results between slightly
different viewpoints because it extremely overfits to seen input views.

Neural Radiance Fields (NeRF) [21] is a novel framework to represent a 3D
scene with a neural implicit function, where a neural network
$f(\cdot,\cdot)$, typically given by an MLP, maps a 3D point
$\textbf{x}=(x,y,z)$ and a unit viewing direction $\textbf{d}=(\theta,\phi)$
to a volume density $\sigma$ as well as an emitted RGB color
$\textbf{c}=(r,g,b)$. Following the classical volume rendering theory [19],
the rendered RGB color of a target pixel is obtained by integrating colors and
densities along a ray. In practice, since the output value
$(\mathbf{c},\sigma)$ of all continuous points on a ray is not observable, a
subset of points are sampled and the rendered color of the ray is approximated
by using the quadrature rule as follows:

$\hat{C}(\mathbf{r})=\sum_{i=1}^{N}T_{i}(1-\exp(-\sigma_{i}\delta_{i}))\mathbf{c}_{i},$
(1)

where $\mathbf{r}$ denotes a ray, $N$ is the number of samples, and
$\delta_{i}$ is the distance between the $i^{\text{th}}$ point and its
adjacent sample. Note that $T_{i}$ indicates the accumulated transmittance
along the ray until the $i^{\text{th}}$ point, which is given by

$T_{i}=\sum_{j=1}^{i-1}\sigma_{j}\delta_{j}$ (2)

The points on the ray are sampled in a two-stage hierarchical manner to
increase rendering efficiency. In the first stage, the points are sampled
uniformly while, in the second stage, the importance sampling is performed
based on the density estimated in the first stage. Since all processes are
fully differentiable, the neural networks encoding radiance fields are
optimized with the following objective:

$\mathcal{L}_{\text{RGB}}=\frac{1}{\left|\mathcal{R}\right|}\sum_{\mathbf{r}\in\mathcal{R}}\left\|C(\mathbf{r})-\hat{C}(\mathbf{r})\right\|^{2}_{2}$
(3)

where $\mathcal{R}$ denotes a set of rays. Note that the positional encoding
is also employed before the MLP to map an input coordinate
$(\textbf{x},\textbf{d})$ onto a higher dimensional space, which is helpful to
represent high-frequency scenes.

Although NeRF [21] achieves outstanding photorealistic view synthesis results,
it requires a lot of images densely captured by calibrated cameras in general.
To alleviate this data acquisition issue, we propose a simple yet effective
few-shot novel view synthesis approach.

##  4 Proposed Method

Our approach learns a robust neural volume rendering model based only on a few
input images without any prior knowledge about a scene. We focus on how to
alleviate the reconstruction inconsistency given by lack of input views and
the degeneracy induced by overfitting. This section discusses the proposed
information-theoretic regularizations imposed on NeRF-based models to achieve
our goals.

###  4.1 Motivation

Due to the small number of views available, few-shot 3D reconstruction and
volume rendering are inherently prone to result in noisy estimations and
degenerate solutions. For example, Figure 2(a) shows that NeRF fails to
reconstruct the 3D scene accurately, mainly due to insufficient viewpoints. In
addition, the learned model is severely overfit to seen images and converges
to a degenerate solution, especially when the viewpoints of training images
are similar to each other. Consequently, the rendering fails with only a
slight change of view as illustrated in Figure 2(b).

###  4.2 Regularization by Ray Entropy Minimization

To alleviate the reconstruction inconsistency, we impose the sparsity
constraint on the reconstructed scene, which is achieved by minimizing the
entropy of each ray density function using additional regularization terms.
This constraint is reasonable because only a small subset of sampled points
along a ray hit objects or background in a scene and the rest of points are
likely to observe noise.

##### Ray density

Before discussing the ray entropy, we first define the normalized ray density
denoted by $p(\mathbf{r})$ as follows:

$\displaystyle
p(\mathbf{r}_{i})=\frac{\alpha_{i}}{\sum_{j}\alpha_{j}}=\frac{1-\exp{(-\sigma_{i}\delta_{i}})}{\sum_{j}1-\exp{(-\sigma_{j}\delta_{j}})},$
(4)

where $\mathbf{r}_{i}$ ($i=1,\dots,N$) is a sampled point in a ray,
$\sigma_{i}$ is the observed density at $\mathbf{r}_{i}$, $\delta_{i}$ is a
sampling interval around $\mathbf{r}_{i}$, and $\alpha_{i}\equiv
1-\exp(-\sigma_{i}\delta_{i})$ is the opacity at $\mathbf{r}_{i}$. The points
on the ray are sampled in a two-stage manner, which are drawn from the uniform
distribution followed by from the distribution of opacity as in [21]. Note
that we actually utilize the opacity $\alpha_{i}$ to compute
$p(\mathbf{r}_{i})$ in (4), instead of the density $\sigma_{i}$, to consider
the irregular sampling interval $\delta_{i}$.

##### Ray entropy

Following Shannon Entropy [27], we define the entropy of a discrete ray
density function given by

$\displaystyle
H(\mathbf{r})=-\sum_{i=1}^{N}p(\mathbf{r}_{i})\log{p(\mathbf{r}_{i})}.$ (5)

Because $\sigma$ and $\delta$ values are already calculated to conduct volume
rendering procedure in (1), the computation of ray entropy incurs only
negligible additional cost.

##### Disregarding non-hitting rays

One issue in ray entropy minimization is that some rays are enforced to have
low entropy though they do not hit any objects in the scene. To prevent the
potential artifacts induced by this issue, we simply disregard the rays with
low density for the entropy minimization. Formally, we employ a mask variable
$M(\cdot$) to indicate the rays that have sufficient observations of the
scene, which is based on the opacity as follows:

$M(\mathbf{r})=\begin{cases}1&\text{if~{}}Q(\mathbf{r})>\epsilon\\\
0&\text{otherwise}\end{cases},$ (6)

where

$\displaystyle Q(\mathbf{r})=\sum_{i=1}^{N}1-\exp{(-\sigma_{i}\delta_{i}})$
(7)

denotes the cumulative ray density.

##### Ray entropy loss

Based on the ray entropy computed in (5), our ray entropy minimization loss is
defined as follows:

$\mathcal{L}_{\text{entropy}}=\frac{1}{\left|\mathcal{R}_{s}\right|+\left|\mathcal{R}_{u}\right|}\sum_{\mathbf{r}\in{\mathcal{R}_{s}\cup\mathcal{R}_{u}}}M(\mathbf{r})\odot
H(\mathbf{r}),$ (8)

where $\mathcal{R}_{s}$ denotes a set of rays from training images,
$\mathcal{R}_{u}$ denotes a set of rays from randomly sampled unseen images,
and $\odot$ indicates element-wise multiplication. Note that, NeRF-based
models are unavailable to use rays from the unseen images due to the lack of
their ground-truths of pixel colors, while our model can utilize them since
entropy regularization does not require the ground-truths. We observe that it
is beneficial to utilize the rays even from the unobserved viewpoint for
better scene reconstruction.

##### Comparison with existing methods

There exist only a few prior works that impose constraints on scene
representations or models, but their objectives are different from ours [17,
37], _e.g._ , focusing on improving reconstruction quality and/or achieving
real-time processing without considering few-shot training scenarios. In
addition, since they rely on 3D volume entropy based on voxel representations,
they have to draw a large number of samples to estimate 3D density or
occupancy map, resulting in heavy computational cost in terms of both space
and time complexities. On the other hand, InfoNeRF employs the entropy
minimization along a ray via 1D sampling and consequently, it runs very
efficiently compared to the methods based on 3D volume entropy.

###  4.3 Regularization by Information Gain Reduction

According to our observations, when the training images have sufficiently
diverse viewpoints, the proposed entropy regularization is very helpful to
improve the quality of both rendered images and 3D depth estimation in the
few-shot setting. However, if all the training images have similar viewpoints
to each other, the models are prone to overfit to seen images and fail to
generalize to unseen views. This is probably because the lack of diverse
observations makes the trained models find degenerate and trivial solutions.

To alleviate the aforementioned limitations, we introduce an additional
regularization term to assure the consistent density distribution across rays
in the neighborhood. Given an observed ray $\mathbf{r}$, we sample another ray
with a slightly different viewpoint, denoted by $\tilde{\mathbf{r}}$, and
minimize the KL-divergence between the density functions of the two rays. The
motivation of this objective is to make the observations from two similar
viewpoints sufficiently consistent so that the model is generalized to a
nearby viewpoint, which is achieved by enforcing smoothness to reconstruction
results over spatial view perturbations.

The regularization loss for information gain reduction is given by

$\displaystyle\mathcal{L}_{\text{KL}}$
$\displaystyle=D_{\text{KL}}\Big{(}P(\mathbf{r})||{P}(\tilde{\mathbf{r}})\Big{)}=\sum_{i=1}^{N}p(\mathbf{r}_{i})\log\frac{p(\mathbf{r}_{i})}{p(\tilde{\mathbf{r}_{i}})},$
(9)

where $\tilde{\mathbf{r}}_{i}$ is a sampled point for observation in ray
$\tilde{\mathbf{r}}$. In our implementation, we obtain $\tilde{\mathbf{r}}$ by
slightly rotating the camera pose of $\mathbf{r}$ in the range from
$-5^{\circ}$ to $5^{\circ}$.

###  4.4 Overall Objective

The total loss function to train a neural implicit model for few-shot neural
volume rendering is given by

$\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{RGB}}+\lambda_{1}\mathcal{L}_{\text{entropy}}+\lambda_{2}\mathcal{L}_{\text{KL}},$
(10)

where $\lambda_{1}$ and $\lambda_{2}$ are balancing terms for our
regularization terms. As mentioned in (3), the reconstruction loss, denoted by
$\mathcal{L}_{\text{RGB}}$, is given by

$\mathcal{L}_{\text{RGB}}=\frac{1}{\left|\mathcal{R}_{s}\right|}\sum_{\mathbf{r}\in\mathcal{R}_{s}}\left\|C(\mathbf{r})-\hat{C}(\mathbf{r})\right\|^{2}_{2},$
(11)

which uses only a set of rays from training images with pixel-level ground-
truth, unlike $\mathcal{L}_{\text{entropy}}$ and $\mathcal{L}_{\text{KL}}$
which utilize the rays even from unobserved viewpoints.

Table 1: Experimental results of few-shot novel view synthesis on the
Realistic Synthetic 360∘ dataset in the 4-view setting. Our approach
outperforms all other existing methods by significant margins in all image
quality metrics. The asterisk ($\ast$) denotes that the model is pretrained on
an external training dataset with dense input views and finet-uned on this
dataset with 4 input views. We run all experimental five times with different
viewpoint samples and the same hyperparameters, and compute the average scores
and their standard deviations.

Method | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | FID $\downarrow$ | KID $\downarrow$
---|---|---|---|---|---
NeRF, 100 views | 31.01 | 0.947 | 0.081 | 42.83 | 0.002
PixelNeRF∗ [38] | 16.09$\pm$0.78 | 0.738$\pm$0.012 | 0.390$\pm$0.030 | 265.25$\pm$6.73 | 0.127$\pm$0.006
NeRF [21] | 15.93$\pm$1.06 | 0.780$\pm$0.014 | 0.320$\pm$0.049 | 215.16$\pm$2.32 | 0.074$\pm$0.012
DietNeRF [11] | 16.06$\pm$1.13 | 0.793$\pm$0.019 | 0.306$\pm$0.050 |  197.02$\pm$12.87 | 0.065$\pm$0.004
InfoNeRF (ours) | 18.65$\pm$0.18 | 0.811$\pm$0.008 | 0.230$\pm$0.008 | 181.47$\pm$4.97 | 0.062$\pm$0.004

Table 2: Average PSNRs and standard deviations of individual scenes on the
Realistic Synthetic 360∘ dataset in the 4-view setting.

Method | Lego | Chair | Drums | Ficus | Hotdog | Materials | Mic | Ship | Avg.
---|---|---|---|---|---|---|---|---|---
NeRF, 100 views | 32.54 | 33.00 | 25.01 | 30.13 | 36.18 | 29.62 | 32.91 | 28.65 | 31.01
PixelNeRF∗ [38] | 15.14$\pm$0.75 | 18.87$\pm$1.38 | 15.10$\pm$0.63 | 16.60$\pm$0.70 | 19.37$\pm$1.78 | 12.31$\pm$1.02 | 16.35$\pm$0.97 | 14.96$\pm$0.75 | 16.09$\pm$0.78
NeRF [21] | 15.61$\pm$4.53 | 18.57$\pm$1.64 | 12.50$\pm$0.98 | 16.37$\pm$2.24 | 19.64$\pm$2.26 | 15.65$\pm$4.16 | 14.78$\pm$2.37 | 14.30$\pm$4.04 | 15.93$\pm$1.06
DietNeRF [11] | 17.13$\pm$4.77 | 19.37$\pm$3.12 | 13.74$\pm$1.55 | 15.76$\pm$3.56 | 18.24$\pm$5.28 | 15.00$\pm$5.18 | 17.71$\pm$1.55 | 11.51$\pm$4.27 | 16.06$\pm$1.13
InfoNeRF (ours) | 18.92$\pm$0.51 | 20.06$\pm$1.11 | 14.33$\pm$0.62 | 19.41$\pm$0.07 | 21.30$\pm$2.31 | 18.34$\pm$0.88 | 18.55$\pm$1.71 | 18.27$\pm$0.71 | 18.65$\pm$0.18

##  5 Experiments

We demonstrate the effectiveness of the proposed approach, referred to as
InfoNeRF, on the standard benchmarks. This section also discusses the
characteristics of our algorithm based on the experiment results.

###  5.1 Datasets

We describe the details of three benchmarks employed to evaluate our
algorithm, which include the Realistic Synthetic 360∘ [21], ZJU-MoCap [24],
and DTU [12] datasets.

##### Realistic Synthetic 360∘

This benchmark is common for neural volume rendering, which contains 8
synthetic scenes with view-dependent light transport effects. Each scene has
an object at the center and 400 rendered images from inward-facing virtual
cameras with different viewpoints. For few-shot training, we randomly sample 4
viewpoints out of 100 training images in each scene, and use the 200 testing
images for evaluation.

##### ZJU-MoCap

This dataset consists of multi-view videos capturing human motion from 23
calibrated cameras. Following [24], we sample 4 uniformly distributed
viewpoints to construct a training set and use the remaining images for
testing.

##### DTU MVS Dataset (DTU)

The images in this dataset contain complex and real-world scenes captured by
the calibrated cameras in controlled environments. All the collected images
have similar viewpoints and face only one side of a scene. We conduct
experiments on 15 scenes, where we optimize the model with 3 images out of 49
views while testing with the remaining 46 views.

###  5.2 Implementation and Evaluation

##### Implementation details

Our implementation is based on PyTorch [23]. We use Adam optimizer [13] with
the initial learning rate of $5\times 10^{-4}$, which decays exponentially by
a factor of 10 at every 250,000 iterations. The balancing term for
$\mathcal{L}_{\text{KL}}$ is decayed by a factor of 2 at every 5,000
iterations. We set the number of rays from seen and unobserved views, denoted
respectively by $|\mathcal{R}_{s}|$ and $|\mathcal{R}_{u}|$, identically to
1024, and our experiments are conducted with a single NVIDIA Titan XP GPU.

##### Metrics

We evaluate the novel view rendering quality based on the standard image
quality metrics, peak signal-to-noise ratio (PSNR) and structural similarity
(SSIM) [33]. We also use perceptual metrics, learned perceptual image patch
similarity (LPIPS) [39], Frèchet inception distance (FID) [10], and kernel
inception distance (KID) [2]. LPIPS estimates normalized features distance
between image pair while FID and KID compute the distance in Inception
representations [31] between two sets of images.

###  5.3 Results

####  5.3.1 Realistic Synthetic 360∘

We compare our approach with NeRF [21], DietNeRF [11], and PixelNeRF [38] on
Realistic Synthetic 360∘ dataset. NeRF, DietNeRF, and InfoNeRF (ours) are
trained with randomly sampled 4 views from scratch. Unlike others, PixelNeRF
is pretrained on the DTU [12] dataset with dense input views, and we fine-tune
the model with 4 sampled views to handle the domain shift issue between the
two datasets.

Table G presents overall quantitative results, where InfoNeRF consistently
outperforms the baseline algorithms in terms of all metrics with considerable
margins while having lower standard deviations. Table G breaks down the PSNU
scores into 8 individual scenes, where InfoNeRF obviously achieves significant
gains for all scenes. Refer to our supplementary document for the results of
SSIM and LPIPS, which have the same tendencies.

Figure 1 demonstrates the qualitative results on the novel viewpoints, where
InfoNeRF shows outstanding quality in the rendered images compared to all the
compared methods. As illustrated in Figure 1(f), the quality of the depth maps
estimated by InfoNeRF looks impressive while we notice that all the compared
algorithms often fail to reconstruct 3D structures accurately and DietNeRF
even has color distortion due to its high-level semantic consistency loss.

(a) Ground-truth

(b) NeRF [21]

(c) NV [17]

(d) InfoNeRF (ours)

(e) NB [24]

Figure 3: Qualitative comparison on the ZJU-MoCap dataset in 4-view setting.
We visualize the rendering results of prior-free algorithms (b-d) including
ours, and prior-based algorithm (e). While existing prior-free algorithms
(b-c) often suffer from inconsistent reconstruction and missing parts of the
human body, InfoNeRF manages to render most of the human body comparable to
prior-based algorithm (e).

####  5.3.2 ZJU-MoCap

Table 3: Quantitative comparison on the ZJU-MoCap dataset in the 4-view
setting. NB [24] has the geometric prior by exploiting the pretrained human
body model (SMPL).

Method | Prior | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
---|---|---|---|---
NB [24] | ✓ | 24.18 | 0.888 | 0.182
NeRF [21] |  | 20.19 | 0.794 | 0.309
NV [17] | 21.74 | 0.827 | 0.253
InfoNeRF (ours) | 22.88 | 0.838 | 0.242

For the ZJU-MoCap dataset, InfoNeRF is evaluated in comparison with NeRF [21],
Neural Volume (NV) [17], and Neural Body (NB) [24], where all algorithms are
trained with 4 images. Note that, since NB employs a pretrained human body
model denoted by SMPL [18] as its prior, the performance of NB can be regarded
as the upper-bound of all other methods.

Table 3 summarizes the experimental results on the ZJU-MoCap dataset, where
InfoNeRF achieves the best performance among the methods without using the
prior in terms of all the tested metrics. Figure 3 demonstrates the
qualitative results of all the compared methods, and the reconstruction result
given by InfoNeRF is particularly accurate.

####  5.3.3 DTU MVS Dataset (DTU)

Contrary to the other two datasets, DTU has substantially different
characteristics because the images in each scene have similar viewpoints.
PixelNeRF takes advantage of this property and learns the scene-agnostic model
successfully while the original NeRF exhibits poor generalization performance
in this dataset with few-shot learning.

We compare our algorithms with NeRF and PixelNeRF in this dataset. We train
InfoNeRF and NeRF from scratch without exploiting any scene prior, so it is
not possible to reconstruct invisible parts of the scene. Therefore, the naïve
evaluation of the algorithms without scene prior is not desirable and we use
the mask corresponding to the visible parts of each scene for the evaluation
performance. Table 4 presents the experimental results on the DTU dataset,
where our algorithm achieves outstanding performance compared to NeRF. Note
that PixelNeRF achieves the highest performance because it exploits the
dataset prior by pretraining on the training split of DTU with dense input
views.

Table 4: Quantitative comparison on the DTU dataset in 3-view setting.
PixelNeRF [38] has the dataset prior by pretraining on other scenes of DTU
with dense input views.

Method | Prior | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
---|---|---|---|---
PixelNeRF [38] | ✓ | 19.55 | 0.724 | 0.286
NeRF [21] |  |  8.50 | 0.426 | 0.611
InfoNeRF (ours) |  | 11.23 | 0.445 | 0.543

###  5.4 Analysis

##### Effect of unseen view sampling

To verify the effectiveness of sampling from unseen viewpoints for entropy
minimization, we run our algorithm by varying the number of rays from unseen
views on the Chair scene of Realistic Synthetic 360∘ with the images reduced
to half. Table 5 presents that increasing the number of rays from the unseen
views achieves gradual improvement by alleviating the reconstruction
inconsistency but its benefit is saturated when the number of rays is larger
than 1,024. Figure 4 visualizes the benefit of sampling additional rays for
unseen views; the use of the rays is helpful for noise reduction in this
example.

Table 5: Impact of the number of rays sampled from unseen viewpoints on the
Chair scene of Realistic Synthetic 360∘ in a 4-view setting. We fix the number
of rays for seen views to 1,024 and vary the number of rays for unseen views.
Bold and underline indicate the first and second place among the results,
respectively.

$\\#$ of seen rays |  $\\#$ of unseen rays | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
---|---|---|---|---
1024 | 0 | 20.14 | 0.834 | 0.225
256 | 20.97 | 0.844 | 0.197
512 | 21.11 | 0.851 | 0.188
1024 | 21.37 | 0.853 | 0.185
2048 | 21.33 | 0.855 | 0.167

(a) seen view

(b) seen view + unseen view

Figure 4: Benefit of sampling rays for unseen viewpoints. We visualize the
rendered images on the Chair scene of Realistic Synthetic 360∘ dataset, where
the noise in the background area is removed completely after adding the rays
from unseen viewpoints.

(a) Ground-truth

(b) NeRF [21]

(c) InfoNeRF w/o $\mathcal{L}_{\text{KL}}$

(d) InfoNeRF

Figure 5:  Qualitative comparisons of our method with its ablative models on
the DTU dataset in the 3-view setting. We visualize the image synthesis (left)
and the depth estimation (right) results for each algorithm. While NeRF
suffers from noises, blurs, and artifacts to reconstruct 3D structure, our two
loss terms, $\mathcal{L}_{\text{entropy}}$ and $\mathcal{L}_{\text{KL}}$,
contribute to outstanding rendering quality and fine depth estimation results.

##### Benefit of regularization

We analyze the impact of the proposed regularization schemes, ray entropy
minimization loss, $\mathcal{L}_{\text{entropy}}$, and ray information gain
reduction loss, $\mathcal{L}_{\text{KL}}$. Table 6 shows the ablative results
of InfoNeRF on the DTU dataset. The entropy minimization loss successfully
improves PSNR, but the SSIM value gets worse on this dataset. This is because
the models are prone to overfit to the seen images when all the training
images have similar viewpoints to each other. However, thanks to our
information gain reduction loss that enforces smoothness to reconstruction
results over spatial view perturbations, our full model helps alleviate the
overfitting issue and prevent degenerate solutions. Note that, in the other
two datasets with substantial viewpoint variations, the entropy minimization
loss works well while the information gain reduction loss makes minor
contribution in general. Figure 5 visualizes the rendering quality of InfoNeRF
in comparison to its ablative models on the DTU dataset. Although InfoNeRF
without the information gain reduction loss tends to generate crisp images,
there exists a lot of noise in the outputs and inconsistency in the depth
maps. On the other hand, our full algorithm manages to reconstruct the overall
shape and geometry of the scene more accurately. Refer to Table C of the
supplementary document for more detailed ablation results on the Realistic
Synthetic 360∘ dataset.

Table 6: Ablative results of our regularization schemes on the DTU dataset in
a 3-view setting.

Method | $\mathcal{L}_{\text{entropy}}$ | $\mathcal{L}_{\text{KL}}$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
---|---|---|---|---|---
NeRF |  |  | 8.50 | 0.426 | 0.611
InfoNeRF w/o $\mathcal{L}_{\text{entropy}}$ |  | ✓ | 8.91 | 0.439 | 0.581
InfoNeRF w/o $\mathcal{L}_{\text{KL}}$ | ✓ |  | 10.54 | 0.418 | 0.561
InfoNeRF | ✓ | ✓ | 11.23 | 0.445 | 0.543

Figure 6: PSNR with respect to the number of views for training on the
Realistic Synthetic 360∘ dataset.

##### Robustness to the number of views

Figure 6 illustrates performance of InfoNeRF by varying the number of views
for training on the Realistic Synthetic 360∘. Compared to NeRF, InfoNeRF
illustrates improved results in terms of all metrics until 8 views, but its
merit is saturated as the number of views increases. This is partly because
the uncertainty of the reconstructed scene decreases as the number of training
views increases, weakening the importance of entropy regularization. See Table
D of our supplementary document for SSIM and LPIPS results.

Figure 7:  Qualitative comparison on the Drums scene in the Realistic
Synthetic 360∘ dataset. InfoPixelNeRF (right) reduces blur significantly
compared to PixelNeRF (left), which demonstrates the effectiveness of our
regularization schemes.

##### Integration into PixelNeRF

To demonstrate the generality of our method, we incorporate the proposed
regularization method to PixelNeRF [38], and refer to this version of our
model as InfoPixelNeRF. Figure 7 illustrates qualitative comparisons between
both models, where InfoPixelNeRF reduces blur in the rendered image
significantly. Table E of our supplementary file presents detailed
comparisons.

##  6 Conclusion

We proposed an information-theoretic regularization technique for few-shot
novel view synthesis. Existing few-shot view synthesis methods suffer from
inconsistent reconstruction, which often generates noise, blur, or artifacts
in rendered images, and overfitting to seen views, which leads to degenerate
solutions. To address these issues, we introduced two effective regularization
schemes, ray entropy minimization and ray information gain reduction. Despite
its simplicity, the proposed method turns out to be effective to alleviate
reconstruction inconsistency across views. We demonstrated outstanding
performance of our method on multiple standard benchmarks, and also conducted
a detailed analysis of our approach via extensive analysis.

##### Acknowledgments

This work was supported in part by the IITP grants [2021-0-02068, AI
Innovation Hub; 2021-0-01343, Interdisciplinary Program in AI (SNU)] and the
Bio & Medical Technology Development Program of NRF [2021M3A9E4080782] funded
by the Korean government (MSIT) and by Samsung Advanced Institute of
Technology.

## References

  * [1] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In ICCV, 2021.
  * [2] Mikołaj Bińkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton.  Demystifying mmd gans.  In ICLR, 2018.
  * [3] Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen.  Unstructured lumigraph rendering.  In ACM SIGGRAPH, 2001.
  * [4] Gaurav Chaurasia, Sylvain Duchene, Olga Sorkine-Hornung, and George Drettakis.  Depth synthesis and local warps for plausible image-based navigation.  In ACM TOG, 2013.
  * [5] Gaurav Chaurasia, Olga Sorkine, and George Drettakis.  Silhouette-aware warping for image-based rendering.  In Computer Graphics Forum, volume 30, 2011.
  * [6] Paul E Debevec, Camillo J Taylor, and Jitendra Malik.  Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach.  In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, 1996.
  * [7] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan.  Depth-supervised nerf: Fewer views and faster training for free.  arXiv preprint arXiv:2107.02791, 2021.
  * [8] John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker.  Deepview: View synthesis with learned gradient descent.  In CVPR, 2019.
  * [9] John Flynn, Ivan Neulander, James Philbin, and Noah Snavely.  Deepstereo: Learning to predict new views from the world’s imagery.  In CVPR, 2016.
  * [10] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In NeurIPS, 2017.
  * [11] Ajay Jain, Matthew Tancik, and Pieter Abbeel.  Putting nerf on a diet: Semantically consistent few-shot view synthesis.  In ICCV, 2021.
  * [12] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs.  Large scale multi-view stereopsis evaluation.  In CVPR, 2014.
  * [13] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [14] Youngjoong Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs.  Neural human performer: Learning generalizable radiance fields for human performance rendering.  In NeurIPS, 2021.
  * [15] Marc Levoy and Pat Hanrahan.  Light field rendering.  In ACM SIGGRAPH, 1996.
  * [16] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang.  Neural scene flow fields for space-time view synthesis of dynamic scenes.  In CVPR, 2021.
  * [17] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh.  Neural volumes: Learning dynamic renderable volumes from images.  ACM TOG.
  * [18] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  Smpl: A skinned multi-person linear model.  ACM TOG, 2015.
  * [19] Nelson Max.  Optical models for direct volume rendering.  IEEE TVCG, 1995.
  * [20] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  ACM TOG, 2019.
  * [21] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [22] Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Chakravarty R Alla Chaitanya, Anton Kaplanyan, and Markus Steinberger.  Donerf: Towards real-time rendering of neural radiance fields using depth oracle networks.  In EGSR, 2021.
  * [23] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  2019\.
  * [24] Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou.  Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans.  In CVPR, 2021.
  * [25] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In ICCV, 2019.
  * [26] Johannes L Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In CVPR, 2016.
  * [27] Claude Elwood Shannon.  A mathematical theory of communication.  The Bell system technical journal, 1948.
  * [28] Sudipta Sinha, Drew Steedly, and Rick Szeliski.  Piecewise planar stereo for image-based rendering.  2009\.
  * [29] Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron.  Nerv: Neural reflectance and visibility fields for relighting and view synthesis.  In CVPR, 2021.
  * [30] Pratul P Srinivasan, Tongzhou Wang, Ashwin Sreelal, Ravi Ramamoorthi, and Ren Ng.  Learning to synthesize a 4d rgbd light field from a single image.  In ICCV, 2017.
  * [31] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In CVPR, 2016.
  * [32] Richard Tucker and Noah Snavely.  Single-view view synthesis with multiplane images.  In CVPR, 2020.
  * [33] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE TIP, 2004.
  * [34] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu.  Nerf–: Neural radiance fields without known camera parameters.  arXiv preprint arXiv:2102.07064, 2021.
  * [35] Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn.  Nex: Real-time view synthesis with neural basis expansion.  In CVPR, 2021.
  * [36] Daniel N Wood, Daniel I Azuma, Ken Aldinger, Brian Curless, Tom Duchamp, David H Salesin, and Werner Stuetzle.  Surface light fields for 3d photography.  In ACM SIGGRAPH, 2000.
  * [37] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.  Plenoctrees for real-time rendering of neural radiance fields.  In ICCV, 2021.
  * [38] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In CVPR, 2021.
  * [39] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.
  * [40] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely.  Stereo magnification: Learning view synthesis using multiplane images.  ACM TOG, 2018.
  * [41] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros.  View synthesis by appearance flow.  In ECCV, 2016.

##  A Validation for $\mathcal{L}_{\text{entropy}}$ and
$\mathcal{L}_{\text{KL}}$

To analyze the behavior of our two regularization losses, we measure the
values of ray entropy and information gain of NeRF models after training with
different settings. Figure A illustrates that both ray entropy and information
gain values of the few-shot NeRF model are much higher than those of the NeRF
model with 100 views. This observation implies that the rendered images by the
few-shot NeRF are noisier and the reduction of the two losses is desirable.
Note that InfoNeRF with 4 views has almost similar values to the original NeRF
model with 100 views, which presents that the proposed algorithm provides
compact and consistent reconstruction results even with a small number of
views.

(a) Ray entropy values after training

(b) Information gain values after training

Figure A:  Quantitative comparisons of ray entropy and information gain
computed in three different models. Although the few-shot NeRF model gives
higher ray entropy and information gain than the NeRF with 100 views, the
proposed few-shot InfoNeRF model reduces both values, which motivates the two
regularization loss employed in our approach.

##  B Experiments on More Complex Scenarios

To validate our algorithm in more challenging scenarios, we test InfoNeRF on a
more complex real-world dataset, LLFF [20], which contains 8 natural scenes
captured by a handheld cellphone. Each scene has 20 to 62 images, and we hold
out $\nicefrac{{1}}{{8}}$ of the images as test sets following the standard
protocol [20, 21]. For training, we sample 2 views in each scene. Table A
presents that our algorithm still outperforms the baseline by large margins
and Figure B illustrates InfoNeRF provides better visual quality with less
blurring artifact.

Table A: Results on a complex dataset, LLFF [20] in the 2-view setting.

| PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
---|---|---|---
NeRF [21] | 12.93 | 0.267 | 0.554
InfoNeRF | 14.37 | 0.349 | 0.457

(a) Ground-truth

(b) NeRF [21]

(c) InfoNeRF

(d) Ground-truth

(e) NeRF [21]

(f) InfoNeRF

Figure B: Qualitative comparisons on the Flower, T-Rex, Orchids, and Fern
scenes of the LLFF dataset in the 2-view setting.

##  C Role of $\mathcal{L}_{\text{KL}}$ in Narrow-Baseline Data

InfoNeRF employs $\mathcal{L}_{\text{KL}}$ to prevent overfitting, especially
for training with narrow-baseline images, and this loss term is particularly
helpful for DTU. To confirm the proposition, we create two sets of views in
the Lego scene of Realistic Synthetic 360∘ with narrow and wide baselines as
illustrated in Figure C. Table B presents the full results in both settings;
the combination of $\mathcal{L}_{\text{KL}}$ and
$\mathcal{L}_{\text{entropy}}$ is effective for the narrow-baseline setting,
as in the results on the DTU dataset in the Table 6 of the main paper, while
degrading accuracy slightly for the wide-baseline setting. Also,
$\mathcal{L}_{\text{KL}}$ does not work well without
$\mathcal{L}_{\text{entropy}}$ because incorporating the smoothness constraint
between neighborhood rays is not helpful for noisy reconstructed scenes using
the model trained without $\mathcal{L}_{\text{entropy}}$.

Table B: Effects of the $\mathcal{L}_{\text{KL}}$ with respect to viewpoint
variations on the Lego scene of Realistic Synthetic 360∘ in the 4-view
setting.

Method | $\mathcal{L}_{\text{entropy}}$ | $\mathcal{L}_{\text{KL}}$ | narrow-baseline | wide-baseline
---|---|---|---|---
PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS
NeRF [21] |  |  | 13.16 | 0.753 | 0.350 | 18.03 | 0.763 | 0.248
InfoNeRF w/o $\mathcal{L}_{\text{entropy}}$ |  | ✓ | 13.33 | 0.750 | 0.349 | 17.96 | 0.753 | 0.254
InfoNeRF w/o $\mathcal{L}_{\text{KL}}$ | ✓ |  | 16.74 | 0.767 | 0.241 | 19.51 | 0.792 | 0.189
InfoNeRF | ✓ | ✓ | 18.41 | 0.775 | 0.199 | 19.28 | 0.789 | 0.190

(a) Sampled images in the narrow-baseline training set.

(b) Sampled images in the wide-baseline training set.

Figure C: Training set with the narrow and wide baselines sampled from the
Lego scene of the Realistic Synthetic 360∘ dataset in the 4-view setting.

(a) Ground-truth

(b) NeRF [21]

(c) InfoNeRF w/o $\mathcal{L}_{\text{entropy}}$

(d) InfoNeRF w/o $\mathcal{L}_{\text{KL}}$

(e) InfoNeRF

Figure D: Qualitative comparison between the reconstructed images for the Lego
scene of the Realistic Synthetic 360∘ dataset in the narrow-baseline setting
with 4 views.

##  D Robustness to the Number of Training Views

Figure E demonstrates PSNR, SSIM, and LPIPS results of our model and the
baseline by varying the number of training views on the Realistic Synthetic
360∘ dataset. These graphs supplement Figure 6 of the main paper, showing the
consistent tendency for all metrics.

(a) PSNR

(b) SSIM

(c) LPIPS

Figure E: PSNR, SSIM and LPIPS results with respect to the number of training
views on the Realistic Synthetic 360∘ dataset.

(a) Ground-truth

(b) NeRF [21]

(c) InfoNeRF

Figure F: Qualitative comparison with the HotDog scene of the Realistic
Synthetic 360∘ dataset in a 2-view setting.

##  E Integration into Other NeRF-based Models

To demonstrate the generality of our method, we incorporate the proposed
regularization technique to PixelNeRF [38] and mipNeRF [1], and refer to these
versions of our model as InfoPixelNeRF and InfoMipNeRF, respectively. Tables C
presents the quantitative results of both models together with their
baselines, where we train all models from scratch with 4 views. PixelNeRF and
mipNeRF outperform NeRF, partly due to their semantic prior and cone-based
positional encoding, respectively, but InfoPixelNeRF and InfoMipNeRF still
achieve relative gains in terms of all metrics compared to their baselines.

Table C: Experimental results of InfoPixelNeRF and InfoMipNeRF in comparisons
with their baselines in the 4-view setting on the Realistic Synthetic 360∘
dataset, where all results are based on 5 runs. The asterisk ($\ast$) denotes
that the model is pretrained on an external training dataset with dense input
views and fine-tuned on this dataset with 4 views.

Method | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | FID $\downarrow$ | KID $\downarrow$
---|---|---|---|---|---
NeRF [21] | 15.93$\pm$1.06 | 0.780$\pm$0.014 | 0.320$\pm$0.049 | 215.16$\pm$2.32 | 0.0740$\pm$0.0123
PixelNeRF∗ [38] | 16.09$\pm$0.78 | 0.738$\pm$0.012 | 0.390$\pm$0.030 | 265.25$\pm$6.73 | 0.1274$\pm$0.0063
InfoPixelNeRF∗ (ours) | 16.30$\pm$0.80 | 0.745$\pm$0.015 | 0.372$\pm$0.030 | 264.33$\pm$6.21 | 0.1263$\pm$0.0058
mipNeRF [1] | 19.04$\pm$0.02 | 0.814$\pm$0.000 | 0.219$\pm$0.000 | 142.96$\pm$1.89 | 0.0422$\pm$0.0010
InfoMipNeRF (ours) | 19.33$\pm$0.04 | 0.818$\pm$0.000 | 0.213$\pm$0.001 | 142.95$\pm$0.54 | 0.0418$\pm$0.0004

##  F Per-Scene Breakdown on the Realistic Synthetic 360∘ Dataset

Supplementing Table 1 of the main paper, we show the experimental results from
individual scenes in terms of PSNR, SSIM, and LPIPS in Table G, G, and G,
respectively. Table G is the same as Table 1 of the main paper, added for
reference. As shown in the tables, InfoNeRF achieves consistent and non-
trivial improvement over its baselines.

Table D: Experimental results of few-shot novel view synthesis on the
Realistic Synthetic 360∘ dataset in the 4-view setting. Our approach
outperforms all other existing methods by significant margins in all image
quality metrics. The asterisk ($\ast$) denotes that the model is pretrained on
an external training dataset with dense input views and finet-uned on this
dataset with 4 input views. We run all experimental five times with different
viewpoint samples and the same hyperparameters, and compute the average scores
and their standard deviations.

Method | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | FID $\downarrow$ | KID $\downarrow$
---|---|---|---|---|---
NeRF, 100 views | 31.01 | 0.947 | 0.081 | 42.83 | 0.002
PixelNeRF∗ [38] | 16.09$\pm$0.78 | 0.738$\pm$0.012 | 0.390$\pm$0.030 | 265.25$\pm$6.73 | 0.127$\pm$0.006
NeRF [21] | 15.93$\pm$1.06 | 0.780$\pm$0.014 | 0.320$\pm$0.049 | 215.16$\pm$2.32 | 0.074$\pm$0.012
DietNeRF [11] | 16.06$\pm$1.13 | 0.793$\pm$0.019 | 0.306$\pm$0.050 |  197.02$\pm$12.87 | 0.065$\pm$0.004
InfoNeRF (ours) | 18.65$\pm$0.18 | 0.811$\pm$0.008 | 0.230$\pm$0.008 | 181.47$\pm$4.97 | 0.062$\pm$0.004

Table E: Average PSNRs and standard deviations of individual scenes on the
Realistic Synthetic 360∘ dataset in the 4-view setting.

Method | Lego | Chair | Drums | Ficus | Hotdog | Materials | Mic | Ship | Avg.
---|---|---|---|---|---|---|---|---|---
NeRF, 100 views | 32.54 | 33.00 | 25.01 | 30.13 | 36.18 | 29.62 | 32.91 | 28.65 | 31.01
PixelNeRF∗ [38] | 15.14$\pm$0.75 | 18.87$\pm$1.38 | 15.10$\pm$0.63 | 16.60$\pm$0.70 | 19.37$\pm$1.78 | 12.31$\pm$1.02 | 16.35$\pm$0.97 | 14.96$\pm$0.75 | 16.09$\pm$0.78
NeRF [21] | 15.61$\pm$4.53 | 18.57$\pm$1.64 | 12.50$\pm$0.98 | 16.37$\pm$2.24 | 19.64$\pm$2.26 | 15.65$\pm$4.16 | 14.78$\pm$2.37 | 14.30$\pm$4.04 | 15.93$\pm$1.06
DietNeRF [11] | 17.13$\pm$4.77 | 19.37$\pm$3.12 | 13.74$\pm$1.55 | 15.76$\pm$3.56 | 18.24$\pm$5.28 | 15.00$\pm$5.18 | 17.71$\pm$1.55 | 11.51$\pm$4.27 | 16.06$\pm$1.13
InfoNeRF (ours) | 18.92$\pm$0.51 | 20.06$\pm$1.11 | 14.33$\pm$0.62 | 19.41$\pm$0.07 | 21.30$\pm$2.31 | 18.34$\pm$0.88 | 18.55$\pm$1.71 | 18.27$\pm$0.71 | 18.65$\pm$0.18

Table F: Average SSIMs and standard deviations of individual scenes on the
Realistic Synthetic 360∘ dataset in the 4-view setting.

Method | Lego | Chair | Drums | Ficus | Hotdog | Materials | Mic | Ship | Avg.
---|---|---|---|---|---|---|---|---|---
NeRF, 100 views | 0.961 | 0.967 | 0.925 | 0.964 | 0.974 | 0.949 | 0.980 | 0.856 | 0.947
PixelNeRF∗ [38] | 0.703$\pm$0.014 | 0.802$\pm$0.026 | 0.699$\pm$0.016 | 0.802$\pm$0.017 | 0.836$\pm$0.023 | 0.644$\pm$0.027 | 0.767$\pm$0.021 | 0.655$\pm$0.014 | 0.738$\pm$0.012
NeRF [21] | 0.739$\pm$0.065 | 0.818$\pm$0.019 | 0.721$\pm$0.022 | 0.833$\pm$0.030 | 0.863$\pm$0.024 | 0.768$\pm$0.070 | 0.826$\pm$0.043 | 0.675$\pm$0.047 | 0.780$\pm$0.014
DietNeRF [11] | 0.766$\pm$0.079 | 0.846$\pm$0.022 | 0.750$\pm$0.021 | 0.812$\pm$0.046 | 0.851$\pm$0.070 | 0.789$\pm$0.050 | 0.879$\pm$0.028 | 0.649$\pm$0.057 | 0.793$\pm$0.019
InfoNeRF (ours) | 0.788$\pm$0.008 | 0.840$\pm$0.011 | 0.730$\pm$0.015 | 0.851$\pm$0.001 | 0.871$\pm$0.027 | 0.799$\pm$0.052 | 0.883$\pm$0.012 | 0.723$\pm$0.012 | 0.811$\pm$0.008

Table G: Average LPIPS’s and standard deviations of individual scenes on the
Realistic Synthetic 360∘ dataset in the 4-view setting.

Method | Lego | Chair | Drums | Ficus | Hotdog | Materials | Mic | Ship | Avg.
---|---|---|---|---|---|---|---|---|---
NeRF (100 views) | 0.050 | 0.046 | 0.091 | 0.044 | 0.121 | 0.063 | 0.028 | 0.206 | 0.081
PixelNeRF∗ [38] | 0.410$\pm$0.031 | 0.325$\pm$0.065 | 0.462$\pm$0.026 | 0.326$\pm$0.039 | 0.266$\pm$0.050 | 0.490$\pm$0.034 | 0.395$\pm$0.033 | 0.446$\pm$0.037 | 0.390$\pm$0.030
NeRF [21] | 0.318$\pm$0.149 | 0.284$\pm$0.047 | 0.452$\pm$0.057 | 0.224$\pm$0.089 | 0.232$\pm$0.049 | 0.294$\pm$0.178 | 0.351$\pm$0.094 | 0.412$\pm$0.095 | 0.320$\pm$0.049
DietNeRF [11] | 0.285$\pm$0.178 | 0.314$\pm$0.152 | 0.304$\pm$0.094 | 0.315$\pm$0.173 | 0.229$\pm$0.113 | 0.324$\pm$0.200 | 0.210$\pm$0.059 | 0.464$\pm$0.137 | 0.306$\pm$0.050
InfoNeRF (ours) | 0.182$\pm$0.020 | 0.196$\pm$0.016 | 0.374$\pm$0.026 | 0.148$\pm$0.011 | 0.188$\pm$0.044 | 0.218$\pm$0.073 | 0.207$\pm$0.038 | 0.324$\pm$0.015 | 0.230$\pm$0.008

##  G Additional Qualitative Results

###  G.1 Realistic Synthetic 360∘

(a) Ground-truth

(b) NeRF [21]

(c) DietNeRF [11]

(d) PixelNeRF [38]

(e) InfoNeRF

(f) InfoNeRF (depth)

Figure G:  Qualitative comparison of our method with other NeRF-based models
on the Lego, Chair, Drums, Ficus, HotDog and Materials scenes of the Realistic
Synthetic 360∘ dataset in the 4-view setting. Existing works often suffer from
noise (b), color distortion (c), or blur effect (d), while InfoNeRF provides
distinguished rendering quality. The column (f) visualizes depth maps
estimated by InfoNeRF, which provide clear boundaries and fine details.

(a) Ground-truth

(b) NeRF [21]

(c) DietNeRF [11]

(d) PixelNeRF [38]

(e) InfoNeRF

(f) InfoNeRF (depth)

Figure H:  Additional qualitative comparisons of our method with other NeRF-
based models on the Mic and Ship scenes of Realistic Synthetic 360∘ in the
4-view setting. As in Figure G, InfoNeRF provides clear boundaries and fine
details compared to other works.

###  G.2 ZJU-MoCap

(a) Ground-truth

(b) NeRF [21]

(c) NV [17]

(d) InfoNeRF

(e) NB [24]

Figure I: Qualitative comparison on the ZJU-MoCap dataset in the 4-view
setting. We visualize the rendering results of prior-free algorithms (b)–(d),
including ours, and a prior-based algorithm (e). While existing prior-free
algorithms, NeRF and NV, often suffer from inconsistent reconstructions and
missing parts of the human body, InfoNeRF render most human bodies with
competitive quality to the prior-based algorithm, NB.

##  H Potential Negative Societal Impact & Limitations

The proposed few-shot view synthesis algorithm works well with a very small
number of views, and it is more vulnerable to adversarial attacks, which is
problematic in real-world scenarios, _e.g._ , AR or VR systems. InfoNeRF shows
outstanding results on the few-shot volume rendering, but is still struggling
from the need for calibrated cameras albeit few in number, which we leave as
our future work.


irmct]School of Automation Science and Electrical Engineering, Beihang
University, Beijing 100191, China nus]Department of Mechanical Engineering,
National University of Singapore, Singapore astar]SRO department, Institute
for Infocomm Research, 1 Fusionopolis Way, Singapore

# Sparse LiDAR Assisted Self-supervised Stereo Disparity Estimation

Xiaoming Zhaoirmct  Weihai Chenirmct  Xingming Wuirmct  Peter C. Y. Chennus
and Zhengguo Liastar  [<EMAIL_ADDRESS>[<EMAIL_ADDRESS>[
<EMAIL_ADDRESS>
###### Abstract

Deep stereo matching has made significant progress in recent years. However,
state-of-the-art methods are based on expensive 4D cost volume, which limits
their use in real-world applications. To address this issue, 3D correlation
maps and iterative disparity updates have been proposed. Regarding that in
real-world platforms, such as self-driving cars and robots, the Lidar is
usually installed. Thus we further introduce the sparse Lidar point into the
iterative updates, which alleviates the burden of network updating the
disparity from zero states. Furthermore, we propose training the network in a
self-supervised way so that it can be trained on any captured data for better
generalization ability. Experiments and comparisons show that the presented
method is effective and achieves comparable results with related methods.

###### keywords:

Stereo, Lidar, Disparity, Self-Supervise, Depth Estimation

00footnotetext: This work was supported by the National Nature Science
Foundation of China under Grant No. 61620106012, Beijing Natural Science
Foundation under Grant No. 4202042, and the Key Research and Development
Program of Zhejiang Province under Grant No. 2020C01109.

##  1 Introduction

Estimating disparity/depth from stereo images is a common task in many
applications. In recent years, the deep disparity estimation has get much
progress, and yields better results than traditional handcrafted stereo
algorithms. The state-of-the-art disparity estimation networks [1, 2, 3]
usually have four steps: 1) image feature extraction, 2) 4D cost volume
generation, 3) cost aggregation, and 4) disparity regression. In this
pipeline, the 4D cost volume is constructed by concatenating deep features on
possible disparities, and computational expensive 3D convolutions are used to
aggregate the 4D cost volume. Although simper aggregation operators are
proposed [4], the high computational cost on 4D cost volume and the
requirement for large labeled data still limit their use on practical
platforms.

In this paper, we seek a solution to relieve the computational cost and
labeled data requirements for stereo estimation networks. Given that Lidar is
very likely to be installed on real-world platforms, collecting stereo images
and Lidar data can be quick and easy. So we concentrate on self-supervised
stereo disparity estimation with Lidar data.

A straight-forward way to improve stereo estimation with Lidar data is to fuse
them into estimated stereo depth [5, 6]. However, the Lidar data cannot be
used during stereo matching in this post-fusion strategy. To incorporate the
Lidar data into the stereo matching, some researches [7, 8] modify the cost
volume based on Lidar data, while others [9, 10] fuse the Lidar feature into
image feature. Nonetheless, they still rely on the expensive 4D cost volume.
Recent works [11, 12] inspired by the correlation and iterative pipeline in
optical-flow estimation [13], in which the 3D correlation map is constructed
by correlating features on possible disparities, are very efficient and
effective. In this paper, we propose incorporating Lidar points into iterative
updates based on this pipeline. Specifically, the sparse disparity from the
Lidar are regard as seminal states for the iteration. We adopt the
convolutional spatial propagation network (CSPN) [2] to estimate the pixel
affinity, and propagate the sparse disparity with the affinity to a wider
region after each iteration. This strategy allows the iterative steps to be
reduced while improving the inference accuracy, thus reducing the running
time.

Figure 1: The proposed network takes a stereo image pair and left sparse
disparity as input and outputs left dense disparity.

Another limitation for supervised disparity estimation is the requirement of
dense ground-truth disparity/depth label. Because acquiring dense depth data
is costly and limited by the environment. The sun light would interfere with
most dense depth sensors, thus we can only capture dense depth in the indoor
environment. Therefore, only indoor or synthetic datasets can be used to train
the network, which greatly limited the generalization ability. Fortunately,
Lidar can operate without these constraints, thus, self-supervised training
with stereo image and Lidar data is essential. We adopt advances of self-
supervised optical flow estimation [14, 15] to train the proposed network.

To summarize, we develop a sparse Lidar-assisted stereo disparity estimation
network by incorporating sparse depth into an iterative disparity update
pipeline. As a result, the update steps can be reduced to improve running
performance. In addition, self-supervised losses are used for network
training. Therefore, the network can be trained on any stereo-Lidar data for
improved generalization.

##  2 Overview

The goal of this paper is to estimate dense disparity with stereo images
$\mathbf{I}_{l}$, $\mathbf{I}_{r}$ and the sparse Lidar depth
$\mathbf{dep}_{l}^{s}$ of left image. For convenience, the sparse depth
$\mathbf{dep}_{l}^{s}$ is first converted to sparse disparity
$\mathbf{d}_{l}^{s}=fB/\mathbf{dep}_{l}^{s}$, where $f$ and $B$ denote the
camera focal length and stereo baseline respectively. The proposed network
takes $\mathbf{I}_{l}$, $\mathbf{I}_{r}$ and $\mathbf{d}_{l}^{s}$ as inputs
and outputs the dense disparity of left image $\hat{\mathbf{d}}_{l}$. In the
following, we will detail the network architecture and the training losses.

##  3 Sparse assisted stereo estimation network

The overall architecture of the propose network is shown in Fig. 1, it mainly
contains three stages: 1) the image feature and context extraction, 2) the
correlation pyramid construction, and 3) the iterative disparity update.

###  3.1 Image feature and context extraction

The feature network extracts deep feature maps $\mathbf{f}_{l},\
\mathbf{f}_{r}\in\mathbb{R}^{H\times W\times 128}$ for the left and right
images $\mathbf{I}_{l}$, $\mathbf{I}_{r}$. For computational considerations,
the feature maps are 1/4 or 1/8 of the original image resolution. They are
then used to construct the correlation pyramid. The context network has the
same architecture with the feature network, but with different network
parameters. In contrast to RAFT-Stereo [12], we use an additional Residual
block [16] for image-dependent affinity estimation to optimize the estimated
disparity with sparse input. The affinity $\mathbf{A}$ is a pixel by pixel
kernels map for disparity propagation and optimization. In this paper, the
kernel size is $3\times 3$, and the shape of this affinity map is $H\times
W\times 8$.

###  3.2 Correlation pyramid construction

In cost volume based method, the 3D feature map $\mathbf{f}_{l},\
\mathbf{f}_{r}\in\mathbb{R}^{H\times W\times C}$ are concatenated at different
possible disparities. For example, when the maximum disparity is $D$, the
shape of cost volume is $H\times W\times 2C\times D$, and the process of this
4D cost volume is computational expensive. On the other hand, the RAFT-Stereo
[12] shows that the 4D is not necessary, and the 3D correlation map is enough
for disparity estimation. By computing the dot product of features at
horizontal rows of stereo images (the “C” in Fig. 1), the correlation map is
constructed as

$\mathbf{C}(i,j,k)=\sum_{h}\mathbf{f}_{l}(i,j,h)\cdot\mathbf{f}_{r}(i,k,h),\quad\mathbf{C}\in\mathbb{R}^{H\times
W\times W}.$ (1)

Following the RAFT-Stereo [12], the correlation pyramid is generated by 1D
average pooling the last dimension of the correlation map with stride of 2.
Thus the correlation map at level $k$ is of size $H\times W\times W/2^{k}$. In
this pyramid, the top level provides a wide range of similarity, while the
bottom level provides detailed local similarity.

###  3.3 Iterative disparity update

To update the disparity, the network first lookup (the “L” in Fig. 1) into the
correlation pyramid at current disparity estimation, and obtain the
similarities at each pyramid level in a radius $r_{L}$ with bilinear sampling.
Then the similarity and current disparity are encoded as motion features. With
the context from context network, the delta disparity $\Delta_{k}$ is estimate
from the multi-scale convolutional GRU after $k$-th iteration [12], and the
disparity is updated as $\mathbf{d}_{k}=\mathbf{d}_{k-1}+\Delta_{k}$. To
introduce the sparse disparity, we adopt the convolution spatial propagation
(CSPN) [2] before the first and after each GRU iteration

$\small\mathbf{d}_{k}=\operatorname{CSPN}(\mathbf{d}_{k-1}+\Delta_{k}\ |\
\mathbf{A}),$ (2)

where $\operatorname{CSPN}(\mathbf{d}\ |\ \mathbf{A})$ denotes the convolution
spatial propagation on $\mathbf{d}\in\mathbb{R}^{H\times W}$ with the affinity
map $\mathbf{A}\in\mathbb{R}^{H\times W\times 8}$ estimated from context
network. On the affinity map, the last dimension denotes the eight
neighborhood affinities of a location. Thus it can be reviewed as a $H\times
W\times 3\times 3$ tensor with the center pixel is zero. Then the disparity
$\mathbf{d}$ is updated using the following equation

$\displaystyle\mathbf{d}_{t+1}(i,j)=$
$\displaystyle\hat{\mathbf{A}}(i,j,0,0)\odot\mathbf{d}_{0}(i,j)+$ (3)
$\displaystyle\sum_{a,b=-1}^{a,b=1}\hat{\mathbf{A}}(i,j,a,b)\odot\mathbf{d}_{t}(i+a,j+b),$

where

$\displaystyle\hat{\mathbf{A}}(i,j,a,b)=\frac{\mathbf{A}_{(}i,j,a,b)}{\sum_{a,b=-1;a,b\neq
0}^{a,b=1}\left|\mathbf{A}(i,j,a,b)\right|}$ (4)
$\displaystyle\hat{\mathbf{A}}(i,j,0,0)=\mathbf{1}-\sum_{a,b=-1;a,b\neq
0}^{a,b=1}\hat{\mathbf{A}}(i,j,a,b),$

and $t$ is the update times of convolution spatial propagation, $0<i\leq H$
and $0<j\leq W$ denote the pixel location, $0<a\leq 3$ and $0<b\leq 3$
represent index of local $3\times 3$ kernel, and $\odot$ is an element-wise
product. After each propagation, the sparse disparity is then applied to
guarantee that the disparities at valid positions $\mathbf{V}_{l}^{s}$ stay
the same

$\small\mathbf{d}_{t+1}(i,j)=\left(1-\mathbf{V}_{l}^{s}(i,j)\mathbf{d}_{t+1}(i,j)\right)+\mathbf{V}_{l}^{s}(i,j)\mathbf{d}^{s}(i,j).$
(5)

At last, the convex upsampling [12] upsamples the low resolution disparity map
to the original image resolution.

##  4 Self-supervised loss functions

Current stereo estimation networks frequently use different network parameters
for different datasets, indicating a lack of generalizability. One approach to
address this issue is to train on as many datasets as possible. The
requirements for dense disparity ground-truth, on the other hand, make
training on different datasets difficult. With the help of sparse Lidar input,
we use self-supervised network training, which eliminates the need of dense
disparity ground-truth.

###  4.1 Appearance loss

This loss is based on the image reconstruction difference. Considering the
estimated dense disparity $\hat{\mathbf{d}}_{l}$ and $\hat{\mathbf{d}}_{r}$
for left and right image. The left image can be reconstructed with the right
image $\mathbf{I}_{r}$

$\small\hat{\mathbf{I}}_{l}\left(i,j\right)=\operatorname{Bisample}\left(\mathbf{I}_{r}\left(i,j-\hat{\mathbf{d}}_{l}\left(i,j\right)\right)\right),$
(6)

where $\operatorname{Bisample}\left(\mathbf{x}(i,j)\right)$ denotes the
bilinear sampling at $(i,j)$ on image $\mathbf{x}$. The right image
$\hat{\mathbf{I}}_{r}$ can also be reconstruct in the same way. Taking the
left image as example, the appearance loss of left image [17] is given as

$\small L_{ap}^{l}=\frac{1}{N}\sum_{i,j\in
1-\mathbf{O}_{l}}\left(\alpha\frac{1-\operatorname{SSIM}\left(\mathbf{I}_{l},\hat{\mathbf{I}}_{l}\right)}{2}+(1-\alpha)\left\|\mathbf{I}_{l}-\hat{\mathbf{I}}_{l}\right\|\right),$
(7)

where $\mathbf{O}_{l}$ is the occlusion map of left image based on range map
[15], $N$ denotes the number of non-occlusion pixels of left image, and
$\operatorname{SSIM}\left(\mathbf{x},\mathbf{y}\right)$ computes the
Structural Similarity (SSIM) [18] of image $\mathbf{x}$ and $\mathbf{y}$, and
$\alpha=0.85$ is a hyper-parameter to balance two loss terms [17].

###  4.2 Sparse disparity loss

The sparse but accurate disparity can be used to not only help obtain the
dense disparity, but also to supervise the disparity estimation. The sparse
disparity supervision loss is deﬁned as follows:

$\small L_{sp}^{l}=\frac{1}{M}\sum_{i,j;\mathbf{V}_{l}^{s}(i,j)\neq
0}\left\|\mathbf{d}_{l}^{s}(i,j)-\hat{\mathbf{d}}_{l}(i,j)\right\|$ (8)

where $\mathbf{V}_{l}^{s}$ denotes the boolean valid map of sparse disparity
$\mathbf{d}_{l}^{s}$, and $M$ denotes the number of valid points in
$\mathbf{V}_{l}^{s}$.

###  4.3 Left-Right consistency loss

To guarantee that the estimated left disparity and right disparity are
consistent, we apply the left-right disparity consistency loss. Taking the
left disparity $\hat{\mathbf{d}}_{l}$ as example, right disparity
$\hat{\mathbf{d}}_{r}$ is warped to left image with

$\small\hat{\mathbf{d}}_{r2l}=\operatorname{Bisample}\left(\hat{\mathbf{d}}_{r}\left(i,j-\hat{\mathbf{d}}_{l}\left(i,j\right)\right)\right).$
(9)

Then, the left-right consistency of left disparity is

$\small L_{lr}^{l}=\frac{1}{N}\sum_{i,j\in
1-\mathbf{O}_{l}}\left\|\hat{\mathbf{d}}_{r2l}(i,j)-\hat{\mathbf{d}}_{l}(i,j)\right\|.$
(10)

where $\mathbf{O}_{l}$ and $N$ are defined in equation (7). The consistency
loss forces the corresponding positions between image pairs has the same
disparity.

###  4.4 Smooth loss

Training with only appearance and sparsity supervise may result in noisy
disparity estimation, which is non-smooth and inaccurate. By considering the
structure of image, the smooth loss is introduced to alleviate this problem

$\small L_{sm}^{l}=\frac{1}{N}\sum_{i,j\in
1-\mathbf{O}_{l}}\left(\left|\nabla_{x}^{2}\hat{\mathbf{d}}_{l}\right|e^{-\left|\nabla_{x}^{2}\mathbf{I}_{l}\right|}+\left|\nabla_{y}^{2}\hat{\mathbf{d}}_{l}\right|e^{-\left|\nabla_{y}^{2}\mathbf{I}_{l}\right|}\right).$
(11)

With all the losses presented, the total loss function for left image is given
as

$\small
L^{l}=\beta_{ap}L_{ap}^{l}+\beta_{sp}L_{sp}^{l}+\beta_{lr}L_{lr}^{l}+\beta_{sm}L_{sm}^{l},$
(12)

where $\beta$ represents the weight of each loss terms. It is notable that
predicting zero disparities is a trivial solution for left-right consistency
loss and smooth loss. Therefore, their weight should not be too large, so as
to avoid affecting appearance loss and sparse disparity loss, which play their
main supervisory role. In practice, the loss of right image is also
considered, thus the overall loss is $L=(L^{l}+L^{r})/2$.

##  5 Experiments

###  5.1 Datasets

SceneFlow [19] is a synthetic image dataset with 34,801 training frames that
includes stereo image pairs, dense disparity, optical flow, . It contains
three subsets: “FlyingThings3D”, “Monkaa’, and “Driving”. It also provides
4248 test frames on ‘FlyingThings3D’ subset. This dataset is used for network
ablation studies and network pre-training.

KITTI [20] depth completion dataset provides stereo images, sparse Lidar
depth, and semi-dense ground truth depth by accumulating successive 11 Lidar
frames. The dataset consists of 42,949 training and 3,426 validation frames.
This dataset also provides a selection of 1000 images for test, which are
parts of the validation set. As the KITTI stereo benchmark dose not have Lidar
data, and depth completion benchmark has no stereo image, so we did not submit
the result. Following previous works [9, 10] the tests are conducted on the
depth completion task.

Figure 2: Ablation studies of network structure on FlyingThings3D test set
[19]. Top and bottom are the end-point-error (EPE) and D1 ($>1$px) with
respect to GRU iteration numbers, respectively.

###  5.2 Implementation details

The network is implemented with Pytorch and trained with one-cycle learning
schedule [21] with a maximum learning rate $2\times 10^{-4}$. During the
training, input images are cropped to $360\times 720$, and are augmented with
saturation augmentation (0 1.4). In the loss calculation, the $\beta_{ap}=1$,
$\beta_{sp}=0.5$, $\beta_{lr}=0.01$, and $\beta_{sm}=0.01$. The details of
ablation studies and comparisons are given below.

Ablation studies are conducted on the SceneFlow [19] dataset to verify the
proposed network architecture and self-supervise losses. In the training
process, 500 random points are sampled from the ground truth depth to simulate
the sparse Lidar data, the number of GRU and CSPN iterations are set to 10
times, and the batch size is 6.

The comparisons with related works are conducted on the KITTI [20] depth
completion task. The training batch size is 8 in the comparisons, and both the
GRU and CSPN iterations are 20 times. All LiDAR points are used as network
inputs in supervised training, whereas in the self-supervised training, 250
points are network inputs and the remaining are used for sparse loss
computation. During self-supervised training, the network is initialized with
a self-supervised model trained on the SceneFlow [19]. The RMSE (root mean
square error), MAE (mean absolute error), iRMSE (inverse root mean square
error in millimeter), and iMAE (inverse mean absolute error in millimeter) of
estimated depth and ground-truth semi-dense depth are evaluated and compared.

###  5.3 Ablation studies of network architecture

We use the RAFT-Stereo [12] as the baseline model. By introducing sparse
disparity points and CSPN [2] propagation module, four variations of networks
are studied. To eliminate the effects of loss functions, we use supervised
training of the network. We evaluate the end-point-error (EPE) of the
estimated disparity and the percentage of disparity outliers of one pixel in
first frame (D1) at 20K steps on the FlyingThings3D test set [19]. The
evaluation results are shown in Fig. 2. The most obvious first phenomenon is
that both the EPE and the D1 decrease as the number of iterations increases
for all variations. However, the EPE and D1 of the baseline network can only
be reduced to about 1.75 and 20% respectively. By incorporating the CSPN [2]
propagation and sparse disparity, the EPE and D1 comes to 0.66 and 7%
respectively. This indicates that they are both helpful for disparity
estimation. Morever, one can also use much lesser iterations to obtain
equivalent disparity accuracy with the CSPN [2] propagation and sparse
disparity, which is important for efficient applications. Without the LiDAR
measurements, extracting accurate sparse keypoints [22] and matching them [23]
would also be possible to act as the accurate sparse input.

###  5.4 Ablation studies of self-supervise training

Table 1: Ablation studies of loss functions on FlyingThings3D test set [19]. The “App” and “Sps” denote appearance and sparse loss respectively. The “+CSPN” and “+Sparse” represent network variations with CSPN and sparse respectively. And the “-half1” and “-half2” are detailed in Section 5.4. Network | App | Sps | EPE |  D1($>$1px)
---|---|---|---|---
Baseline | ✓ |  | 11.48 | 26.51 [t]
Baseline | ✓ | ✓ | 6.05 | 26.18 [b]
Baseline+CSPN | ✓ |  | 10.06 | 25.58 [t]
Baseline+CSPN+Sparse | ✓ | ✓ | 6.06 | 25.87
Baseline+CSPN+Sparse-half1 | ✓ | ✓ | 1.57 | 13.56
Baseline+CSPN+Sparse-half2 | ✓ | ✓ | 1.24 |  12.15 [b]
Baseline+CSPN+Sparse-Supervise | - | - | 0.66 |  7.00 [t]
Figure 3: Visualization of evaluation results on FlyingThings3D [19] test set.
(a) left image, (b) right image, (c) left ground-truth disparity, (d)-(h)
estimated left disparity of supervised baseline, self-supervised baseline
without sparse, self-supervised baseline+CSPN with sparse, self-supervised
baseline+CSPN with sparse-half1, and self-supervised baseline+CSPN with
sparse-half2, respectively.

We investigated appearance and sparsity disparity loss on different network
variations while keeping the weights of left-right consistency and smooth loss
constant at 0.01. The test results are shown in Table 1 and Fig. 3. First, the
baseline network is tested for appearance and sparsity disparity loss. As
shown in Table 1, adding sparse disparity improves the EPE (from 11.48 to
6.05) but has limited effect on the D1 (26.51% and 26.18%). When the sparse
disparity is added to the baseline+CSPN variation, the EPE falls from 10.06 to
6.06, but the D1 remains high above 25%. The visualization of estimated
disparity in Fig. 3 (e) and (f) also shows that the network has difficulty
fitting large disparities. After careful examination, we discover that the
disparity update in Equation (5) produces the same disparity and ground-truth
disparity of the corresponding sparse positions in the output dense disparity
map (equivalent to directly replacing the disparity of effective sparse points
with ground-truth disparity). As a result, the sparsity loss is masked out
during network back propagation. To avoid this problem, we tested two self-
supervised strategies with the sparse disparities:

  * $\bullet$

Sparse-half1: Half of the sparse disparities is randomly selected as the
network input, and all the sparse disparities is used to calculate the sparse
loss;

  * $\bullet$

Sparse-half2: Half of the sparse disparities is randomly selected as the
network input, and the remaining sparse disparities is used to calculate the
sparse loss.

The EPE and D1 are reduced to 1.57 and 13.56%, respectively, with the Sparse-
half1 training strategy, as shown in Table 1. But it is inferior to the
Sparse-half2 training strategy because half of the sparse points are still
invalid in network back propagation and would affect affinity training. The
Sparse-half2 training strategy reduces the EPE and D1 to 1.24 and 12.15%,
respectively, which is much closer to supervised training results. In
addition, the visualization in Fig. 3 confirms the efficacy of the proposed
strategy.

Table 2: Comparisons with related supervised methods on KITTI depth completion validation set. “M+L” and “S+L” denote “Mono+LiDAR” and “Stereo+LiDAR” respectively. The best and second best of “S+L” are ITALIC and BOLD. Modality | Method | RMSE | MAE | iRMSE |  iMAE
---|---|---|---|---|---
M+L | Sparsetodense [24] | 814.70 | 249.90 | 2.80 | 1.21 [t]
M+L | RGB_certainty [25] | 772.80 | 215.00 | 2.19 | 0.93
M+L | Spade [26] | 1035.29 | 248.32 | 2.60 | 0.98
M+L | CSPN [2] | 1019.64 | 279.46 | 2.93 | 1.15
M+L | Guidenet [27] | 777.78 | 221.59 | 2.39 | 1.00
M+L | NLSPN [28] | 771.80 | 197.30 | 2.00 | 0.80
M+L | PENet [29] | 757.20 | 209.00 | 2.22 | 0.92
M+L | ACMNet [30] | 790.75 | 217.34 | 2.39 | 0.97 [b]
S+L | Park et al. [5] | 2021.20 | 500.50 | 3.39 | 1.38 [t]
S+L | CCVN [31] | 749.30 | 252.50 | 1.40 | 0.81
S+L | LiStereo [9] | 832.16 | 283.91 | 2.19 | 1.10
S+L | Ours | 775.66 | 210.26 | 2.23 |  0.86 [b]
Table 3: Comparisons with related self-supervised methods on KITTI depth completion validation set. “M+L” and “S+L” denote “Mono+LiDAR” and “Stereo+LiDAR” respectively. The best and second best are ITALIC and BOLD. Modality | Method | RMSE | MAE | iRMSE |  iMAE
---|---|---|---|---|---
M+L | Sparsetodense [24] | 1301.05 | 352.22 | 4.08 | 1.61 [t]
S+L | LiStereo [9] | 1278.87 | 326.10 | 3.83 |  1.32 [t]
S+L | Ours | 1293.56 | 342.66 | 4.48 |  1.34 [b]
Figure 4: Visualization of evaluation results on KITTI depth completion [20]
selected validation set. The top two images are the left image and sparse
LiDAR depth. And the below four images visualizes the depth of Sparse-to-dense
[24], RGB_guide [25], PENet [29] and our method.

###  5.5 Comparison to supervised methods

Table 2 shows the comparisons with supervised methods. It can be seen that the
proposed method achieves close to the best errors in the S+L methods, ranking
first in MAE and second in RMSE and iMAE. However, we can see that some M+L
methods have lower errors than S+L methods. This is due to the S+L methods
rely heavily on accurate LiDAR points while images are primarily used for
guidance, whereas M+L methods rely mainly on stereo matching of binocular
images and sparse LiDAR points are only used for guidance or optimization. One
disadvantage of S+L methods is that they cannot infer depth in areas with no
LiDAR points, as shown in Fig. 4.

###  5.6 Comparison to self-supervised methods

The comparisons with self-supervised methods are shown in Table 3. We can see
that the proposed method ranks second in terms of RMSE, MAE, and iMAE, and is
very close to LiStereo [9], proving its effectiveness. More importantly, the
proposed method is scalable: the network can be configured with or without
sparse LiDAR input, as well as with a different number of GRU iterations for
different computational budgets (Fig. 2). And it is also possible to feed the
network stereo images with different exposures to generate high dynamic range
3D images [32].

##  6 Conclusions

We present a self-supervised stereo disparity estimation network with sparse
disparity input in this paper. To incorporate the sparse disparity input to
the stereo matching pipeline, the affinity of disparity is estimated. In each
GRU iteration, convolutional spatial propagation with affinity and sparse
disparity is used to optimize the disparity. The appearance difference and
disparity on sparse points are also used to train the network in a self-
supervised manner. We can train the network in any dataset using the self-
supervised learning method, as there are no ground-truth label requirements.
Our future work will include improving affinity propagation and introducing
stereo images with different exposures for the generation of high dynamic
range 3D images.

## References

  * [1] J.-R. Chang and Y.-S. Chen, Pyramid stereo matching network, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5410–5418, 2018
  * [2] X. Cheng, P. Wang, and R. Yang, Learning Depth with Convolutional Spatial Propagation Network, IEEE transactions on pattern analysis and machine intelligence, Vol. 42, No. 10, 2361–2397, 2019.
  * [3] X. Song, G. Yang, X. Zhu, H. Zhou, Z. Wang, and J. Shi, AdaStereo: a simple and efficient approach for adaptive stereo matching, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10328–10337, 2021
  * [4] F. Zhang, V. Prisacariu, R. Yang, and P. H. Torr, Ga-net: Guided aggregation net for end-to-end stereo matching, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 185–194, 2019
  * [5] K. Park, S. Kim, and K. Sohn, High-Precision Depth Estimation Using Uncalibrated LiDAR and Stereo Fusion, IEEE Trans. on Intelligent Transportation Systems, Vol. 21, No. 1, 321–335, 2020.
  * [6] X. Zhao, W. Chen, Z. Liu, X. Ma, L. Kong, X. Wu, H. Yue, and X. Yan, LiDAR-ToF-Binocular depth fusion using gradient priors, in 2020 Chinese Control And Decision Conference. Hefei, China, 2024–2029, 2020
  * [7] M. Poggi, D. Pallotti, F. Tosi, and S. Mattoccia, Guided Stereo Matching, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA, 979–988, 2019
  * [8] T.-H. Wang, H.-N. Hu, C. H. Lin, Y.-H. Tsai, W.-C. Chiu, and M. Sun, 3d lidar and stereo fusion using stereo matching network with conditional cost volume normalization, in IEEE/RSJ International Conference on Intelligent Robots and Systems, 5895–5902, 2019
  * [9] J. Zhang, M. S. Ramanagopal, R. Vasudevan, and M. Johnson-Roberson, Listereo: Generate dense depth maps from lidar and stereo imagery, in 2020 IEEE International Conference on Robotics and Automation, 7829–7836, 2020
  * [10] J. Choe, K. Joo, T. Imtiaz, and I. S. Kweon, Volumetric propagation network: Stereo-lidar fusion for long-range depth estimation, IEEE Robotics and Automation Letters, Vol. 6, No. 3, 4672–4679, 2021.
  * [11] H. Wang, R. Fan, P. Cai, and M. Liu, PVStereo: Pyramid Voting Module for End-to-End Self-Supervised Stereo Matching, IEEE Robotics and Automation Letters, Vol. 6, No. 3, 4353–4360, 2021.
  * [12] L. Lipson, Z. Teed, and J. Deng, RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching, arXiv preprint arXiv:2109.07547, 2021.
  * [13] Z. Teed and J. Deng, RAFT: Recurrent All-Pairs Field Transforms for Optical Flow, in European Conference on Computer Vision, 2020.
  * [14] Z. Liu, J. Liu, W. Chen, X. Wu, and Z. Li, Faminet: Learning real-time semi-supervised video object segmentation with steepest optimized optical flow, IEEE Trans. on Instrumentation and Measurement, Vol. 71, 1–16, 2022.
  * [15] Y. Wang, Y. Yang, Z. Yang, L. Zhao, P. Wang, and W. Xu, Occlusion aware unsupervised learning of optical flow, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4884–4893, 2018
  * [16] K. He, X. Zhang, S. Ren, and J. Sun, Deep Residual Learning for Image Recognition, in IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA, 770–778, 2016
  * [17] C. Godard, O. Mac Aodha, and G. J. Brostow, Unsupervised monocular depth estimation with left-right consistency, in Proceedings of the IEEE conference on computer vision and pattern recognition, 270–279, 2017
  * [18] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, Image quality assessment: from error visibility to structural similarity, IEEE transactions on image processing, Vol. 13, No. 4, 600–612, 2004.
  * [19] N. Mayer, E. Ilg, P. Hausser, P. Fischer, D. Cremers, A. Dosovitskiy, et.al, A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation, in Proceedings of the IEEE conference on computer vision and pattern recognition, 4040–4048, 2016
  * [20] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, Vision meets robotics: The kitti dataset, International Journal of Robotics Research, 2013.
  * [21] L. N. Smith and N. Topin, Super-convergence: Very fast training of neural networks using large learning rates, in Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, Vol. 11006, 1100612, 2019
  * [22] X. Zhao, X. Wu, J. Miao, W. Chen, P. C. Chen, and Z. Li, Alike: Accurate and lightweight keypoint detection and descriptor extraction, arXiv preprint arXiv:2112.02906, 2021.
  * [23] X. Zhao, J. Liu, X. Wu, W. Chen, F. Guo, and Z. Li, Probabilistic spatial distribution prior based attentional keypoints matching network, IEEE Trans. on Circuits and Systems for Video Technology, 2021.
  * [24] F. Ma, G. V. Cavalheiro, and S. Karaman, Self-supervised sparse-to-dense: Self-supervised depth completion from lidar and monocular camera, in International Conference on Robotics and Automation, 3288–3295, 2019
  * [25] W. Van Gansbeke, D. Neven, B. De Brabandere, and L. Van Gool, Sparse and noisy lidar completion with rgb guidance and uncertainty, in 16th international conference on machine vision applications, 1–6, 2019
  * [26] M. Jaritz, R. De Charette, E. Wirbel, X. Perrotton, and F. Nashashibi, Sparse and dense data with cnns: Depth completion and semantic segmentation, in International Conference on 3D Vision, 52–60, 2018
  * [27] J. Tang, F.-P. Tian, W. Feng, J. Li, and P. Tan, Learning guided convolutional network for depth completion, IEEE Trans. on Image Processing, Vol. 30, 1116–1129, 2020.
  * [28] J. Park, K. Joo, Z. Hu, C.-K. Liu, and I. So Kweon, Non-local spatial propagation network for depth completion, in European Conference on Computer Vision, Glasgow, UK, 120–136, 2020
  * [29] M. Hu, S. Wang, B. Li, S. Ning, L. Fan, and X. Gong, PENet: Towards Precise and Efficient Image Guided Depth Completion, in IEEE International Conference on Robotics and Automation, 2021.
  * [30] S. Zhao, M. Gong, H. Fu, and D. Tao, Adaptive context-aware multi-modal network for depth completion, IEEE Trans. on Image Processing, 2021\.
  * [31] T.-H. Wang, H.-N. Hu, C. H. Lin, Y.-H. Tsai, W.-C. Chiu, and M. Sun, 3d lidar and stereo fusion using stereo matching network with conditional cost volume normalization, in IEEE/RSJ International Conference on Intelligent Robots and Systems.x, 5895–5902, 2019
  * [32] F. Kou, Z. Wei, W. Chen, X. Wu, C. Wen, and Z. Li, Intelligent detail enhancement for exposure fusion, IEEE Trans. on Multimedia, Vol. 20, No. 2, 484–495, 2018.