# Light Field Neural Rendering

Mohammed Suhail1,2111Work done while interning at Google. Carlos Esteves4
Leonid Sigal1,2,3 Ameesh Makadia4<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
1University of British Columbia 2Vector Institute for AI 3Canada CIFAR AI
Chair 4Google

###### Abstract

Classical light field rendering for novel view synthesis can accurately
reproduce view-dependent effects such as reflection, refraction, and
translucency, but requires a dense view sampling of the scene. Methods based
on geometric reconstruction need only sparse views, but cannot accurately
model non-Lambertian effects. We introduce a model that combines the strengths
and mitigates the limitations of these two directions. By operating on a four-
dimensional representation of the light field, our model learns to represent
view-dependent effects accurately. By enforcing geometric constraints during
training and inference, the scene geometry is implicitly learned from a sparse
set of views. Concretely, we introduce a two-stage transformer-based model
that first aggregates features along epipolar lines, then aggregates features
along reference views to produce the color of a target ray. Our model
outperforms the state-of-the-art on multiple forward-facing and 360∘ datasets,
with larger margins on scenes with severe view-dependent variations. Code and
results can be found at light-field-neural-rendering.github.io. ††* Work done
while interning at Google.

##  1 Introduction

Synthesizing a novel view given a sparse set of images is a long-standing
challenge in computer vision and graphics [10, 42, 43]. Recent advances in 3D
neural rendering for view synthesis, in particular NeRF [32] and its
successors [37, 16, 59, 15, 28, 34], have brought us tantalizingly close to
the capability of creating photo-realistic images in complex environments. One
reason for NeRF’s success is its implicit 5D scene representation which maps a
3D scene point and 2D viewing direction to opacity and color. In principle,
such a representation could be perfectly suited to modeling view-dependent
effects such as the non-Lambertian reflectance of specular and translucent
surfaces. However, without regularization, this formulation permits degenerate
solutions due to the inherent ambiguity between 3D surface and radiance, where
an incorrect shape (opacity) can be coupled with a high-frequency radiance
function to minimize the optimization objective [62]. In practice, NeRF avoids
such degenerate solutions through its neural architecture design, where the
viewing direction is introduced only in the last layers of the MLP, thereby
limiting the expressivity of the radiance function, which effectively
translates to a smooth BRDF prior [62]. Thus, NeRF manages to avoid degenerate
solutions at the expense of fidelity in non-Lambertian effects (Fig. 1
highlights this particular limitation of the NeRF model). Photo-realistic
synthesis of non-Lambertian effects is one of the few remaining hurdles for
neural rendering techniques.

Figure 1: Novel view synthesis. On top is the target image to be rendered,
from the _Lab_ scene in the Shiny dataset [56]. Bottom row shows crops of
novel views generated by our proposed model, NeX [56], and NeRF [32]. Unlike
NeX and NeRF that fail to synthesize refractions on the test tube, our model
almost perfectly reconstructs these complex view-dependent effects. We
indicate the PSNR of the rendered images within parenthesis (higher is
better). Images can be zoomed for detail.

In this paper, we formulate view synthesis as rendering a sparsely observed
light field. The 4D light field [27], which measures the radiance along rays
in empty space, is often used for view synthesis [27, 6, 25]. Rendering a
novel view from a densely sampled light field can be achieved with signal
processing techniques (_e.g_., interpolation) and without any model of the 3D
geometry, but no such straightforward method exists with sparse light fields.
From sparse images, rendering often utilizes additional 3D geometric
constraints, such as predicted depth maps [47, 25], but performance is
sensitive to accurate depth estimates which are difficult to obtain for non-
Lambertian surfaces.

Motivated by these limitations, we introduce a novel method for rendering a
sparse light field. Our neural rendering function operates in the style of
image based rendering, where a target ray is synthesized using only observed
rays from nearby views. In lieu of explicit 3D information, our transformer
based rendering function is trained to fuse rays from nearby views exploiting
an additional inductive bias in the form of a multi-view geometric constraint,
namely the epipolar geometry. As shown in Fig. 1, our model is able to
faithfully reconstruct the sharp details and lighting effect in the most
challenging scene in the Shiny dataset [56].

Contributions. Our main contribution is the novel light field based neural
view synthesis model, capable of photorealistic modeling of non-Lambertian
effects (_e.g_., specularities and translucency). To address the core
challenge of sparsity of initial views, we leverage an inductive bias in the
form of a multi-view geometric constraint, namely the epipolar geometry, and a
transformer-based ray fusion. The resulting model produces higher fidelity
renderings for forward-facing as well as $None$ captures, compared to state-
of-the-art, achieving up to $5$ dB improvement in the most challenging scenes.
Further, as a byproduct of our design, we can easily obtain dense
correspondences and depth without further modifications, as well as
transparent visualization of the rendering process itself. Through ablations
we illustrate the importance of our individual design choices.

##  2 Related work

Light field rendering. Levoy and Hanrahan [27] defined the 4D light field as a
function that specifies the radiance of any given ray in free space. They
forwent geometric reasoning to directly synthesize novel views from input
samples. Lumigraph rendering [18] exploits proxy geometry to counter aliasing
effects that stem from irregularity or view under-sampling. Recent works [25,
47, 46, 6, 57] have explored learning based methods to light field rendering.
These methods, however, either require dense input sampling [25], have limited
range of motion [47] or are limited to simple scenes [46]. In this work, we
focus on novel view synthesis for complex scenes with challenging non-
Lambertian effects from a sparse set of viewpoints.

Neural scene representation. Representing shape and appearance of scenes using
neural networks has recently gained immense popularity. Explicit
representation-based methods use differentiable rendering to learn 3D
representation such as point clouds [58, 1, 41], meshes [51] or voxels [44,
29] for the scene. Implicit representation-based methods represent scenes
using continuous coordinate-based functions such as signed distance fields [2,
24, 17, 60, 8, 61] or occupancy fields [30, 35]. Scene Representation Networks
[45] use a differentiable ray marching algorithm along with a continuous
function that maps coordinates to features. NeRF [32] achieves photo-realistic
rendering by learning a function that maps points along a ray to color and
opacity followed by volumetric rendering. NeX [56] is a multiplane image-based
scene representation that addresses NeRF’s difficulty to model large view
dependent effects. However, NeX is still challenged in scenarios such as
interference patterns caused by reflection or refraction through liquid. In
this work, we introduce a model that can faithfully render novel views in the
presence of complex view-based effects in scenarios where other methods fail
(_e.g_., Fig. 1). For a comprehensive survey of recent advances in neural
rendering please refer to Tewari et al. [50].

Image-based rendering. Image-based rendering (IBR) methods [12, 14, 55, 9] are
built on the notions that novel views can be render by “borrowing” pixel
values from a given set of input images. Global geometry-based methods such as
Riegler and Koltun [40], Hedman and Kopf [19], Riegler and Koltun [39], Hedman
et al. [20] rely on dense reconstruction from input views to obtain a global
mesh for the scene. These meshes are used for projecting target rays onto
nearby view for feature and color extraction. Other methods such as Penner and
Zhang [36], Chaurasia et al. [9] infer depth maps using multi-view stereo
methods to compute warping transforms from the given input to target
viewpoints. Thies et al. [52] combine IBR with GAN-based image synthesis to
learn view-dependent effects. To overcome difficulties caused by error in
depth estimation, Choi et al. [11] estimate a depth uncertainty distribution
to refine images.

Recently, Wang et al. [55] introduced IBRNet, a NeRF-based model that
incorporates features from nearby views for rendering. Their architecture
predicts colors for each point on the ray as weighted average of colors from
neighboring views. The densities for each point are predicted by aggregating
information from all other points using a single attention layer. Similarly
for category-specific reconstruction, NerFormer [38] proposed to replace the
MLP in NeRF-WCE [23] with a transformer model to allow for spatial reasoning.
Our work crucially differs from these methods at their core: the rendering
framework. Our method employs a light field representation, forgoing the need
for volumetric rendering. Furthermore, we introduce a transformer-based model
that first reasons about correspondences to aggregate features along each
epipolar line, then reasons about occlusion and lighting effects to aggregate
features from multiple views to produce the final color.

##  3 Approach

Figure 2: Model Overview. Given a target ray to render, we identify reference
views and sample points along the epipolar lines corresponding to the target
ray. Features of these epipolar points along with the light field coordinates
of the target ray are inputs to the epipolar aggregation. This stage (blue),
_independently_ aggregates features along the epipolar lines for each
reference view, producing reference view features. The reference view features
along with the target ray are passed to the view aggregation stage (green),
which combines the reference view features to predict the target ray color.

Our goal is to synthesize novel views of a scene given a collection of input
images available during both training and inference. Our design is guided by
two key ideas, 1) using the four-dimensional parametrization of the light-
field as input enables capturing view-dependent effects with high fidelity,
and 2) enforcing constraints from multiple view geometry allows for view
synthesis with sparse input views.

These ideas enable faithfully recovering illumination effects as in classical
lightfield methods [27, 18], but require only a sparse view-sampling of the
scene, as in geometry-based methods [12, 33] which traditionally struggle
reproducing non-Lambertian effects. To implement them, we introduce an
epipolar-geometric inductive bias in conjunction with a transformer-based
architecture. Our model can render novel views from forward-facing photos as
well as $None$ scenes captured with cameras on a hemisphere.

In the following sections, we first introduce the light field representation,
then an overview of the model, followed by a detailed description of the
network architecture.

###  3.1 Light field parametrization

Light fields are functions on the space of oriented lines that associate a
radiance value to a given ray. In free space, as the radiance along the ray
remains constant, the space of rays has four degrees of freedom and can be
parametrized by 4D vectors. We consider two distinct parametrizations of light
field, the light slab [27] and the two-sphere [7].

##### Light slab.

We adopt the light slab parametrization for forward-facing captures. A light
slab consists of two parallel planes with their respective 2D coordinate
systems $(s,t)$ and $(u,v)$. Rays are then represented as a 4D tuple
$r=(s,t,u,v)$ containing the coordinates of intersections with the two planes
in their respective coordinate frames.

##### Two-sphere.

For $None$ scenes, we use the two-sphere parametrization [7] of the light
field. Given a sphere bounding a scene, rays from the camera are represented
using the colatitudes and longitudes at the two intersections with the sphere,
$r=(\theta_{1},\phi_{1},\theta_{2},\phi_{2})$.

Given a four-dimensional light field ray parametrization $r$, we learn a
neural rendering model $f$ that maps the rays to radiance values.

To obtain the ray coordinates for a given pixel in homogeneous coordinates
$x\in\textbf{RP}^{2}$, from an image taken using a camera with intrinsics $C$
and pose (extrinsics) $[R\;t]$, we first obtain the ray as a line $\ell$ in
world coordinates parametrized by $\delta$ as $\ell(\delta)=-R^{\top}t+\delta
R^{\top}C^{-1}x$, then solve for $\delta$ to obtain the intersections $r$ with
either the two planes or the sphere. To render an image, we evaluate the model
$f(r)$ for the rays associated to each target pixel.

###  3.2 Model overview

Optimizing a neural rendering model $f$ that directly maps 4D light field
coordinates to color fails to generalize to novel views when trained with a
sparse set of input views (see Sec. 4.3 for quantitative evaluation).

To address this challenge, we introduce a model that incorporates a geometric
inductive bias in the form of the epipolar constraints.

Given a target camera, we identify a set of neighboring views to be used to
enforce multiple-view consistency. During training, this set is constructed by
randomly choosing $K$ views from a subset of $N$ closest views. During
inference, the closest $K$ are chosen deterministically. We refer to the set
of $K$ chosen views as _reference views_.

Now given a target pixel $x$ to be rendered, we obtain its ray
parametrizations $\ell$ and $r$ as described in Sec. 3.1, sample a sequence of
$P$ points $p_{i}=\ell(\delta_{i})$ along the ray, and project each point to
each reference view as $x_{i}^{j}=C_{j}[R_{j}\;t_{j}]p_{i}$, where $C_{j}$,
$[R_{j}\;t_{j}]$ are the reference view camera intrinsics and extrinsics,
respectively, and $1\leq j\leq K$.

The collection $x^{j}=\\{x_{i}^{j}\\}_{1\leq i\leq P}$ consists of points
along the epipolar line of the target ray in the $j^{th}$ reference view. and
we refer to $x_{i}^{j}$ as _epipolar points_. To each epipolar point, we
associate its ray parametrization as described in Sec. 3.1, yielding the
collections $r^{j}=\\{r_{i}^{j}\\}_{1\leq i\leq P}$.

##### Epipolar feature aggregation.

The first stage of our model, represented by the function $f_{1}$, computes a
feature representation per reference view by aggregating features associated
to the epipolar points and target ray. We detail what those features are in
the following sections. Conceptually, the first stage computes the set of
features $\\{z^{j}\\}_{1\leq j\leq K}$ where $z^{j}=f_{1}(r,r^{j})$. This is
loosely related to classical multiple view geometry, where we look for a
correspondence to the target ray along the epipolar line. In our case,
however, there is no visual representation of the target ray, so the model
must learn to match the target ray coordinates with the available reference
features, and the output is a feature vector representing the view $j$.

##### View feature aggregation.

The second stage, represented by the function $f_{2}$, predicts the target ray
color by aggregating features associated to each reference view, given the
target ray representation. Conceptually, the color for pixel $x$ with
associated ray $r$ is predicted as $f(r)=f_{2}(r,\\{f_{1}(r,r^{j})\\})$. This
stage learns to reason about occlusion and illumination effects to combine
information from all views and produce the target ray color.

###  3.3 Network architectures

One possible approach would be to model $f_{1}$ and $f_{2}$ as multi-layer
perceptrons (MLP). However, this impedes the model from exploiting readily
available relational information that can be extracted from the epipolar
points, leading to sub-optimal performance (Sec. 4.3 quantifies this). Since
inputs to $f_{1}$ are a _sequence_ of epipolar points, and inputs to $f_{2}$
are a _set_ of reference view features, we propose to use transformers, which
excel in sequence and set modeling, to model both epipolar and view feature
aggregation.

Figure 3: Qualitative Comparison. Top: results on the _CD_ scene from Shiny
dataset [56]. Our method is able to retrieve sharper details in the
reflections on the bottle (_e.g_., the top-left of the insets) as well as the
interference patterns on the compact disk (_e.g_., rainbow and reflection on
the top-right). Bottom: results on the _Orchids_ scene from the real-forward
facing (RFF) dataset [31]. Our method recovers more accurately the shape of
the leaves. We also observe sharper texture on the leaves as well as the
petals.

####  3.3.1 Epipolar feature transformer ($f_{1}$)

This transformer, highlighted in blue in Fig. 2, combines the features of
points along the epipolar line based on the target rays.

The input is a sequence of $P+1$ features, with $P$ features from epipolar
points and one from the target ray. The feature vector for the target ray is
its own coordinates $r$. The feature for an epipolar point $x_{i}^{j}$ is a
concatenation of 1) ray coordinates $r_{i}^{j}$, 2) coordinates of $p_{i}$,
the 3D point along $r$ projected to $x_{i}^{j}$, 3) a learnable camera
embedding $k_{j}$, 4) visual features $v_{i}^{j}$ at $x_{i}^{j}$, obtained
from a lightweight CNN, and 5) the color $c_{i}^{j}$ at $x_{i}^{j}$.

Assuming the target pixel $x$ matches to an epipolar point $x_{i}^{j}$, the
corresponding point in the scene can be solved for and will have coordinates
of $p_{i}$. Including it as an epipolar point feature also plays the role of
positional encoding, since each point in the epipolar line correspond to some
depth value along the query ray. This type of positional encoding is richer
than the typical 1D encoding used in sequence modeling [53], and more
appropriate for modeling a 3D scene, as demonstrated in Sec. 4.3.

We further apply Fourier features [32, 49] positional encoding to facilitate
learning of high-frequency functions. This operation is performed by
$\gamma_{r}$ for ray coordinates and $\gamma_{p}$ for point coordinates, see
Sec. 4.1 for details. To summarize, each epipolar point $x_{i}^{j}$ is
represented by a feature

$\displaystyle
y_{i}^{j}=[\gamma_{r}(r_{i}^{j})\parallel\gamma_{p}(p_{i})\parallel
k_{j}\parallel v_{i}^{j}\parallel c_{i}^{j}],$ (1)

where $\parallel$ denotes concatenation. The epipolar transformer for view $j$
will take as inputs $[\gamma_{r}(r),\\{y_{i}^{j}\\}_{1\leq i\leq P}]$. A
linear layer first projects the features to the same dimension, then a self-
attention transformer is applied to the whole sequence.

We aggregate the $P$ outputs corresponding to the epipolar points
($\tilde{y}_{i}^{j}$) to obtain the _reference view features_. The aggregation
is a weighted average, with the weights computed using an attention mechanism
similar to the Graph Attention Networks (GAT) [54] as follows,

$\alpha_{i}^{j}=\frac{\exp\left(W_{1}\left[\tilde{r}\parallel{\tilde{y}_{i}^{j}}\right]\right)}{\sum_{k}\exp\left(W_{1}\left[\tilde{r}\parallel{\tilde{y}_{k}^{j}}\right]\right)},$
(2)

where $\tilde{r}$ are the output features of the target ray, and $W_{1}$ are
learned weights. The first stage is completed by repeating
$z^{j}=f_{1}(r,r^{j})=\sum_{i=1}^{P}\alpha_{i}^{j}\tilde{y}_{i}^{j}$ for all
views $1\leq j\leq K$.

####  3.3.2 View feature transformer ($f_{2}$)

This transformer, highlighted in green in Fig. 2, takes the target ray and the
set of features for each reference view. The input sequence is now
$[\gamma_{r}(r),\\{z^{j}\\}_{1\leq j\leq K}]$, where $z^{j}$ are the reference
view features computed by the first stage, and the output is a single feature
vector for the target ray. We use the same self-attention transformer
architecture as the epipolar feature aggregator. The transformer output
sequence $[\hat{r},\\{\tilde{z}^{j}\\}_{1\leq j\leq K}]$ is aggregated with a
weighted average using the same idea as the previous section. We compute the
weights $\beta^{j}$ with learnable weights $W_{2}$,

$\beta^{j}=\frac{\exp\left(W_{2}\left[{\hat{r}}\parallel{\tilde{z}^{j}}\right]\right)}{\sum_{k}\exp\left(W_{2}\left[\hat{r}\parallel{\tilde{z}^{k}}\right]\right)},$
(3)

then the output of this stage is the target ray feature
$\sum_{k=1}^{K}\beta^{k}\tilde{z}^{k}$, which is linearly projected and passed
by a sigmoid to produce the pixel color prediction $c$.

###  3.4 Loss

During training, we minimize the $\mathcal{L}_{2}$ loss between the observed
and predicted colors. We additionally include an auxiliary loss to encourage
the attention weights for the epipolar points ($\alpha_{i}^{j}$) and reference
views ($\beta^{j}$) to be interpretable, in the sense that high values of
$\alpha_{i}^{j}$ suggest a valid match to the target ray, while low values of
$\beta^{j}$ might indicate occlusion. This auxiliary loss also leads to more
accurate renderings (see Sec. 4.3). To compute it, we use the attention
weights to combine reference pixel colors and make a second color prediction
as

$c_{\text{aux}}=\sum_{j}\beta^{j}\left(\sum_{i}\alpha_{i}^{j}c_{i}^{j}\right),$
(4)

where $c_{i}^{j}$ is the color of the epipolar point $x_{i}^{j}$. The
auxiliary loss is then defined as the $\mathcal{L}_{2}$ loss between
$c_{\text{aux}}$ and the ground truth. The effect of this loss is two-fold: 1)
it incentivizes weights $\alpha_{i}^{j}$ to have lower entropy to avoid blurry
predictions in the auxiliary branch, and 2) it encourages weights $\beta^{j}$
to be high for unoccluded views.

##  4 Experiments

We show quantitative and qualitative comparisons against state-of-the-art
methods for novel view synthesis. We also perform an ablation study to analyze
the effectiveness of the components introduced in our method.

Model |  PSNR [dB] $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  Avg. $\downarrow$
---|---|---|---|---
LLFF [31] | 24.41 | 0.863 | 0.211 | 0.0656
NeRF [32] | 26.76 | 0.883 | 0.246 | 0.0562
IBRNet [55] | 26.73 | 0.851 | 0.175 | 0.0523
NeX [56] | 27.26 | 0.904 | 0.178 | 0.0473
Ours | 28.26 | 0.920 | 0.062 | 0.0297
Table 1: Results for the real forward-facing (RFF) dataset [31].

###  4.1 Implementation details

##### Network architecture.

We use similar transformer architectures as the ones recently introduced for
vision related tasks [13]. Each block consists of a single-headed self-
attention layer and an MLP with Gaussian error linear unit (GELU) activation
[22]. A residual connection is applied at every block, followed by a LayerNorm
(LN) [3]. Each transformer has 8 blocks and the internal feature size is 256.
The visual features $v_{i}^{j}$ are produced by a single convolutional layer
with 5 x 5 filters and 32 channels.

##### Positional encoding.

Following prior work [32, 49], we use Fourier features to encode input
coordinates to facilitate learning the high-frequency components required for
accurate rendering. For the light slab parametrization and the 3D points
$p_{i}$, we positionally-encode each ray coordinate [32] as
$\gamma_{r}(w)=\gamma_{p}(w)=\\{\sin(2^{k}w)\\}\cup\\{\cos(2^{k}w)\\}$ for
$0\leq k\leq 4$. For the two-sphere parametrization, we found it beneficial to
use a positional encoding based on evaluating the spherical harmonics at the
points $(\theta_{1},\phi_{1})$ and $(\theta_{2},\phi_{2})$, see the appendix
for details. The learnable camera embeddings $k_{j}$ are 256-dimensional.

##### Training/inference details.

In each training step, we randomly choose a target image and sample a batch of
random rays from it. The batch sizes are 4096 for the forward-facing datasets
and 8192 for Blender. We train for $250\,000$ iterations with the Adam
optimizer [26] and a linear learning rate decay schedule with $5000$ warm-up
steps. For inference, we sample contiguous blocks of rays to make a batch.
Training on a Blender scene takes around 23 hours on a 32-core TPUv3 slice.
Rendering an 800 x 800 image then takes around $9.2$ seconds.

###  4.2 Results

We compare our method with LLFF [31], NeRF [32], IBRNet [55], NeX [56] and
Mip-NeRF [4]. We compare against Mip-NeRF only on the Blender dataset because
for forward-facing captures, as noted by Barron et al. [4, Appx D], Mip-NeRF
performs on par with NeRF.

##### Metrics.

To measure the performance of our model we use three widely adopted metrics:
peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM),
and the learned perceptual image patch similarity (LPIPS) [63]. Following [4],
we additionally report the geometric mean of $10^{-\text{PSNR}/10}$,
$\sqrt{1-\text{SSIM}}$ and LPIPS, which provides a summary of three metrics
for easier comparison. We report the averages of each metric over all the
scenes in each dataset. Please refer to the appendix for a scene-wise
breakdown of the results.

Model |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  Avg. $\downarrow$
---|---|---|---|---
NeRF [32] | 25.60 | 0.851 | 0.259 | 0.0651
NeX [56] | 26.45 | 0.890 | 0.165 | 0.0499
IBRNet222We fine tune the pretrained model available at https://github.com/googleinterns/IBRNet on each scene in Shiny. [55] | 26.50 | 0.863 | 0.122 | 0.0468
Ours | 27.34 | 0.907 | 0.045 | 0.0294
Table 2: Results for the Shiny dataset from NeX [56]. Model |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  Avg. $\downarrow$
---|---|---|---|---
NeRF [32] | 31.01 | 0.953 | 0.050 | 0.0194
IBRNet [55] | 28.14 | 0.942 | 0.072 | 0.0299
Mip-NeRF [4] | 33.09 | 0.961 | 0.043 | 0.0161
Ours | 33.85 | 0.981 | 0.024 | 0.0110
Table 3: Results for the Blender dataset from NeRF [32].

####  4.2.1 Real-forward-facing (RFF) dataset

The RFF dataset introduced by Mildenhall et al. [31] consists of $8$ forward
facing captures of real-world scenes using a smartphone. For our experiments,
we use the same resolution and train/test splits as NeRF [32].

Table 1 reports the average metrics across all $8$ scenes in the RFF dataset.
We show qualitative comparisons on the _Orchids_ scene in Fig. 3. Compared to
the baselines, our method retains sharper detailed textures and produces
consistent shape boundaries on the leaves and petals.

####  4.2.2 Shiny dataset

The RFF dataset mostly consists of diffuse scenes with little view-dependent
effects. The Shiny dataset introduced in NeX [56] presents 8 scenes with
challenging view-dependent effects, captured by forward-facing cameras. We use
the same image resolution and splits as NeX.

We compare our model against NeX, IBRNet and NeRF on the Shiny dataset in Tab.
2. We report the average scores across all scenes in the dataset. Our model
consistently improves over the state-of-the-art in all metrics. We show
qualitative analysis of rendering on a test view from the _CD_ scene in Fig.
3. Our model is able to reconstruct the interference patterns on the disk and
reflections on the bottle with higher level of detail as compared to
baselines.

Model |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  Avg. $\downarrow$
---|---|---|---|---
Vanilla-NLF | 17.39 | 0.614 | 0.516 | 0.1802
$1$-MLP | 21.33 | 0.774 | 0.208 | 0.0900
$2$-MLP | 26.16 | 0.896 | 0.076 | 0.0390
No CNN ($v_{i}^{j}$) | 27.43 | 0.910 | 0.057 | 0.0314
No 3D Coordinates | 28.17 | 0.920 | 0.047 | 0.0273
No LCE ($k_{j}$) | 28.23 | 0.926 | 0.045 | 0.0264
Mean Pooling | 28.39 | 0.929 | 0.043 | 0.0255
No Auxiliary Loss | 28.43 | 0.931 | 0.043 | 0.0253
Ours | 28.78 | 0.934 | 0.038 | 0.0235
Table 4: Ablation study on the RFF dataset [31], with 25% the original
resolution (504 x 378). Refer to Sec. 4.3 for details.

####  4.2.3 Blender dataset

Our model is capable of rendering novel views of $None$ scenes. To evaluate
this case, we use the synthetic dataset introduced by Mildenhall et al. [32].
Each scene consists of 800 x 800 resolution images rendered from viewpoints
randomly sampled on a hemisphere around the object.

Table 3 reports the average performance across all scenes in the Blender
dataset. Our model improves over NeRF, IBRNet and Mip-NeRF on all metric and
achieves new state-of-the-art results. On the materials scene, which contains
reflections on metallic balls, we observe an improvement of around 4 on the
PSNR metric when compared to Mip-NeRF. We present the full table along with
qualitative comparisons in the appendix.

###  4.3 Ablation studies

To validate the effectiveness of different design decisions, we run the
following ablation experiments.

##### Geometric inductive bias.

We train a model, called ‘Vanilla-NLF’, that uses an MLP to predict the color
of a ray given only its light field representation, without consideration of
the scene geometry in form of epipolar constraints.

Figure 4: Correspondence Distribution. The per-point attention weights learned
by our model indicate potential correspondences to the target ray. We
visualize four target rays and one reference view. The weights are in log-
scale, from blue to red. The green circles highlight the point with highest
correspondence probability.

Transformers vs MLPs. We train variations of our model replacing the
transformers with MLPs. In the first variant, we replace each of the epipolar
and view transformers with MLPs (‘2-MLP’). The second variant replaces both
transformers by a single MLP that takes as input all epipolar point features
from all reference views along with the target ray and directly predicts the
color (‘1-MLP’). We detail the architectures in the appendix. We run a sweep
over the number of layers for these MLPs and report performance of the best
model.

Model Components. To probe the efficacy of the different components, we train
ablated models 1) without visual features $v_{i}^{j}$ (‘No CNN’), 2) without
3D coordinates $p_{i}$, 3) without the learnable camera embedding $k_{j}$ (‘No
LCE’), 4) replacing the attention based aggregation with mean pooling, and 5)
removing the auxiliary loss term.

Table 4 reports the ablation results. All models are trained on images from
the RFF datasets downsampled to 25% of the original resolution (504x 378). We
use the average metrics across all the scenes for comparison.

###  4.4 Interpreting the model

The use of transformers and epipolar geometry in our model permits
interpretation of the results via the attention weights. We demonstrate this
by extracting correspondences and depth maps. Also, our use of a four-
dimensional light field representation enables the construction of epipolar-
plane images (EPI) [5], which are interpretable reconstructions of the scene
geometry.

Dense correspondence. We can extract potential correspondences between a
target ray and a reference view $j$ by finding the largest attention weights
in $\\{\alpha_{i}^{j}\\}_{1\leq i\leq P}$. Figure 4 shows the weight
distribution of putative correspondences over the epipolar line for four
points of interest. For points (1) and (4) we observe unimodal distributions
with peaks at the point of correspondence. For point (2) we notice some
uncertainty, while for point (3), the distribution is multi-modal with peaks
around each blade with the highest peak near the correct correspondence.

---
Figure 5: Disparity Map. The per-point and per-view attention weights learned
by our model can be applied to estimate a disparity map by aggregating the
putative depths of each epipolar point on each reference view, for each target
ray.

Disparity map. Since each epipolar point corresponds to the projection of the
target ray at a certain depth (the putative depth), we can use the
correspondence distributions to estimate a depth map for a target ray. We
first extract all the epipolar point ($\alpha_{i}^{j}$) and reference view
($\beta^{j}$) attention weights as described in Sec. 3.3, then compute a
weighted average of putative depths, equivalent to applying Eq. 4 with colors
replaced by depths. Figure 5 shows an example of a the disparity map obtained
for a test view of the _Crest_ scene from the Shiny dataset [56].

##### Epipolar-plane images (EPI).

For a 4D light slab representation, we construct the EPI by querying our model
with two fixed and two variable coordinates, resulting in a 2D color image.
Physically, this corresponds to moving the camera along a 1D trajectory,
stacking the images of a line segment parallel to the trajectory. EPIs encode
information about specularities and scene geometry, where diffuse points
appear as lines and specular points appear as curves. We show the epipolar
slices for the _CD_ and _Flower_ scene in Fig. 6. The _Flower_ scene is
predominantly diffuse so we observe lines of varying slopes in the EPI, with
slopes inversely proportional to the depth. For the _CD_ scene, in addition to
lines, we observe curves at regions corresponding to the interference pattern
on the disk. This is due to the change in virtual apparent depth of the
specular points with change in view point [48, 31].

Figure 6: Epipolar-plane images (EPI). Our model represents the 4D light field, so constructing EPIs is natural. Each EPI vertically stacks images along the blue line, while the camera moves parallel to the blue line. Different depths show as lines of different slopes in the EPI, while view-dependent effects show as curves.  Ground Truth |  Ours |  Ground Truth |  Ours
---|---|---|---
Figure 7: Failure Cases. Our model is challenged by texture-less thin
repeating structures. Left: It produces distorted circles on the grill-like
structure in the _Tools_ scene from Shiny [56]. Right: Similar distortions
appear for the _Mic_ scene from Blender [32].

##  5 Limitations

Since our method relies on implicitly finding the correspondences for a target
pixel in nearby views, it is challenged by texture-less thin repeating
structures. As shown in Fig. 7, our model produces fuzzy details on the grill-
like structure in the _Tools_ scene from Shiny and on the wire-mesh on the
microphone from Blender.

Transformers are computationally expensive, resulting in slow training and
inference times. Our method is around 8 times slower than Mip-NeRF [4] on same
hardware. Our model does, however, compare favorably in terms of speed against
other transformer-based models. For example, NerFormer [38] takes around 180s
to render an 800 x 800 image on a single V100 GPU, while our method takes 6070
on the same hardware. We notice that our model suffers from overhead of 1)
random memory access on device and 2) data transfer between host and device.
We believe that it can be made more efficient with some engineering effort.

##  6 Conclusion

We present a light field based neural rendering method for novel-view
synthesis. Unlike prior volumetric rendering methods, our proposed model can
naturally handle real-world illumination effects by learning the light field
over a four-dimensional space. To address the dense sampling dependency of
light field rendering, we introduced a two-stage framework that incorporates
geometric inductive bias in the form of epipolar constraints. Our model leads
to significant improvement over previous state-of-the-art model for view
synthesis especially for scenes with challenging view-dependent effects.
Finally, the design of our model allows extracting dense correspondences,
disparity maps and epipolar-plane images without any additional training.

## References

  * Aliev et al. [2020] K.-A. Aliev, A. Sevastopolsky, M. Kolos, D. Ulyanov, and V. Lempitsky.  Neural point-based graphics.  In _European Conference on Computer Vision (ECCV)_ , pages 696–712, 2020.
  * Atzmon et al. [2019] M. Atzmon, N. Haim, L. Yariv, O. Israelov, H. Maron, and Y. Lipman.  Controlling neural level sets.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019\.
  * Ba et al. [2016] L. J. Ba, J. R. Kiros, and G. E. Hinton.  Layer normalization.  _CoRR_ , abs/1607.06450, 2016.
  * Barron et al. [2021] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5855–5864, 2021.
  * Bolles et al. [1987] R. C. Bolles, H. H. Baker, and D. H. Marimont.  Epipolar-plane image analysis: An approach to determining structure from motion.  _International Journal of Computer Vision (IJCV)_ , 1(1):7–55, 1987.
  * Buehler et al. [2001] C. Buehler, M. Bosse, L. McMillan, S. Gortler, and M. Cohen.  Unstructured lumigraph rendering.  In _Proceedings of the 28th annual conference on Computer Graphics and Interactive Techniques_ , pages 425–432, 2001.
  * Camahort et al. [1998] E. Camahort, A. Lerios, and D. Fussell.  Uniformly sampled light fields.  In _Eurographics Workshop on Rendering Techniques_ , pages 117–130, 1998.
  * Chabra et al. [2020] R. Chabra, J. E. Lenssen, E. Ilg, T. Schmidt, J. Straub, S. Lovegrove, and R. Newcombe.  Deep local shapes: Learning local sdf priors for detailed 3d reconstruction.  In _European Conference on Computer Vision (ECCV)_ , pages 608–625, 2020.
  * Chaurasia et al. [2013] G. Chaurasia, S. Duchene, O. Sorkine-Hornung, and G. Drettakis.  Depth synthesis and local warps for plausible image-based navigation.  _ACM Transactions on Graphics (TOG)_ , 32(3):1–12, 2013.
  * Chen and Williams [1993] S. E. Chen and L. Williams.  View interpolation for image synthesis.  In _Proceedings of the 20th annual conference on Computer Graphics and Interactive Techniques_ , pages 279–288, 1993.
  * Choi et al. [2019] I. Choi, O. Gallo, A. Troccoli, M. H. Kim, and J. Kautz.  Extreme view synthesis.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 7781–7790, 2019.
  * Debevec et al. [1996] P. E. Debevec, C. J. Taylor, and J. Malik.  Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach.  In _Proceedings of the 23rd annual conference on Computer Graphics and Interactive Techniques_ , pages 11–20, 1996.
  * Dosovitskiy et al. [2021] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  _International Conference on Learning Representations (ICLR)_ , 2021\.
  * Du et al. [2018] R. Du, M. Chuang, W. Chang, H. Hoppe, and A. Varshney.  Montage4d: Interactive seamless fusion of multiview video textures.  _ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games_ , 2018\.
  * Du et al. [2021] Y. Du, Y. Zhang, H.-X. Yu, J. B. Tenenbaum, and J. Wu.  Neural radiance flow for 4d view synthesis and video processing.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 14324–14334, 2021.
  * Gafni et al. [2021] G. Gafni, J. Thies, M. Zollhofer, and M. Nießner.  Dynamic neural radiance fields for monocular 4d facial avatar reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8649–8658, 2021.
  * Genova et al. [2020] K. Genova, F. Cole, A. Sud, A. Sarna, and T. Funkhouser.  Local deep implicit functions for 3d shape.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4857–4866, 2020.
  * Gortler et al. [1996] S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen.  The lumigraph.  In _Proceedings of the 23rd annual conference on Computer graphics and interactive techniques_ , pages 43–54, 1996.
  * Hedman and Kopf [2018] P. Hedman and J. Kopf.  Instant 3d photography.  _ACM Transactions on Graphics (TOG)_ , 37(4):1–12, 2018.
  * Hedman et al. [2017] P. Hedman, S. Alsisan, R. Szeliski, and J. Kopf.  Casual 3d photography.  _ACM Transactions on Graphics (TOG)_ , 36(6):1–15, 2017.
  * Heek et al. [2020] J. Heek, A. Levskaya, A. Oliver, M. Ritter, B. Rondepierre, A. Steiner, and M. van Zee.  Flax: A neural network library and ecosystem for JAX, 2020.  URL http://github.com/google/flax.
  * Hendrycks and Gimpel [2016] D. Hendrycks and K. Gimpel.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_ , 2016.
  * Henzler et al. [2021] P. Henzler, J. Reizenstein, P. Labatut, R. Shapovalov, T. Ritschel, A. Vedaldi, and D. Novotny.  Unsupervised learning of 3d object categories from videos in the wild.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4700–4709, 2021.
  * Jiang et al. [2020] C. Jiang, A. Sud, A. Makadia, J. Huang, M. Nießner, T. Funkhouser, et al.  Local implicit grid representations for 3d scenes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6001–6010, 2020.
  * Kalantari et al. [2016] N. K. Kalantari, T.-C. Wang, and R. Ramamoorthi.  Learning-based view synthesis for light field cameras.  _ACM Transactions on Graphics (TOG)_ , 35(6):1–10, 2016.
  * Kingma and Ba [2015] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  _International Conference on Learning Representations (ICLR)_ , 2015\.
  * Levoy and Hanrahan [1996] M. Levoy and P. Hanrahan.  Light field rendering.  In _Proceedings of the 23rd annual conference on Computer Graphics and Interactive Techniques_ , pages 31–42, 1996.
  * Lindell et al. [2021] D. B. Lindell, J. N. Martel, and G. Wetzstein.  Autoint: Automatic integration for fast neural volume rendering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 14556–14565, 2021.
  * Lombardi et al. [2019] S. Lombardi, T. Simon, J. Saragih, G. Schwartz, A. Lehrmann, and Y. Sheikh.  Neural volumes: Learning dynamic renderable volumes from images.  _ACM Transactions on Graphics (TOG)_ , 38(4):65:1–65:14, 2019.
  * Mescheder et al. [2019] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4460–4470, 2019.
  * Mildenhall et al. [2019] B. Mildenhall, P. P. Srinivasan, R. Ortiz-Cayon, N. K. Kalantari, R. Ramamoorthi, R. Ng, and A. Kar.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  _ACM Transactions on Graphics (TOG)_ , 38(4):1–14, 2019.
  * Mildenhall et al. [2020] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _European Conference on Computer Vision (ECCV)_ , pages 405–421, 2020.
  * Narayanan et al. [1998] P. Narayanan, P. W. Rander, and T. Kanade.  Constructing virtual worlds using dense stereo.  In _Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271)_ , pages 3–10. IEEE, 1998.
  * Oechsle et al. [2021] M. Oechsle, S. Peng, and A. Geiger.  Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5589–5599, 2021.
  * Peng et al. [2020] S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger.  Convolutional occupancy networks.  In _European Conference on Computer Vision (ECCV)_ , pages 523–540, 2020.
  * Penner and Zhang [2017] E. Penner and L. Zhang.  Soft 3d reconstruction for view synthesis.  _ACM Transactions on Graphics (TOG)_ , 36(6):1–11, 2017.
  * Pumarola et al. [2021] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10318–10327, 2021.
  * Reizenstein et al. [2021] J. Reizenstein, R. Shapovalov, P. grid, L. Sbordone, P. Labatut, and D. Novotny.  Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR)_ , pages 10901–10911, 2021.
  * Riegler and Koltun [2020] G. Riegler and V. Koltun.  Free view synthesis.  In _European Conference on Computer Vision (ECCV)_ , pages 623–640, 2020.
  * Riegler and Koltun [2021] G. Riegler and V. Koltun.  Stable view synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12216–12225, 2021.
  * Rückert et al. [2021] D. Rückert, L. Franke, and M. Stamminger.  Adop: Approximate differentiable one-pixel point rendering.  _arXiv preprint arXiv:2110.06635_ , 2021.
  * Shade et al. [1998] J. Shade, S. Gortler, L.-w. He, and R. Szeliski.  Layered depth images.  In _Proceedings of the 25th annual conference on Computer Graphics and Interactive Techniques_ , pages 231–242, 1998.
  * Shum and Kang [2000] H. Shum and S. B. Kang.  Review of image-based rendering techniques.  In _Visual Communications and Image Processing_ , volume 4067, pages 2–13, 2000.
  * Sitzmann et al. [2019a] V. Sitzmann, J. Thies, F. Heide, M. Nießner, G. Wetzstein, and M. Zollhofer.  Deepvoxels: Learning persistent 3d feature embeddings.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2437–2446, 2019a.
  * Sitzmann et al. [2019b] V. Sitzmann, M. Zollhöfer, and G. Wetzstein.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  In _Advances in Neural Information Processing Systems_ , 2019b.
  * Sitzmann et al. [2021] V. Sitzmann, S. Rezchikov, W. T. Freeman, J. B. Tenenbaum, and F. Durand.  Light field networks: Neural scene representations with single-evaluation rendering.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021\.
  * Srinivasan et al. [2017] P. P. Srinivasan, T. Wang, A. Sreelal, R. Ramamoorthi, and R. Ng.  Learning to synthesize a 4d rgbd light field from a single image.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , pages 2243–2251, 2017.
  * Swaminathan et al. [2002] R. Swaminathan, S. B. Kang, R. Szeliski, A. Criminisi, and S. K. Nayar.  On the motion and appearance of specularities in image sequences.  In _European Conference on Computer Vision (ECCV)_ , pages 508–523, 2002.
  * Tancik et al. [2020] M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. T. Barron, and R. Ng.  Fourier features let networks learn high frequency functions in low dimensional domains.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020\.
  * Tewari et al. [2021] A. Tewari, J. Thies, B. Mildenhall, P. Srinivasan, E. Tretschk, Y. Wang, C. Lassner, V. Sitzmann, R. Martin-Brualla, S. Lombardi, et al.  Advances in neural rendering.  _arXiv preprint arXiv:2111.05849_ , 2021.
  * Thies et al. [2019] J. Thies, M. Zollhöfer, and M. Nießner.  Deferred neural rendering: Image synthesis using neural textures.  _ACM Transactions on Graphics (TOG)_ , 38(4):1–12, 2019.
  * Thies et al. [2020] J. Thies, M. Zollhöfer, C. Theobalt, M. Stamminger, and M. Nießner.  IGNOR: Image-guided neural object rendering.  _International Conference on Learning Representations (ICLR)_ , 2020\.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.  Attention is all you need.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017\.
  * Veličković et al. [2018] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio.  Graph Attention Networks.  _International Conference on Learning Representations (ICLR)_ , 2018\.
  * Wang et al. [2021] Q. Wang, Z. Wang, K. Genova, P. P. Srinivasan, H. Zhou, J. T. Barron, R. Martin-Brualla, N. Snavely, and T. Funkhouser.  Ibrnet: Learning multi-view image-based rendering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4690–4699, 2021.
  * Wizadwongsa et al. [2021] S. Wizadwongsa, P. Phongthawee, J. Yenphraphai, and S. Suwajanakorn.  Nex: Real-time view synthesis with neural basis expansion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8534–8543, 2021.
  * Wu et al. [2021] G. Wu, Y. Liu, L. Fang, and T. Chai.  Revisiting light field rendering with deep anti-aliasing neural network.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * Wu et al. [2020] M. Wu, Y. Wang, Q. Hu, and J. Yu.  Multi-view neural human rendering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1682–1691, 2020.
  * Xian et al. [2021] W. Xian, J.-B. Huang, J. Kopf, and C. Kim.  Space-time neural irradiance fields for free-viewpoint video.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9421–9431, 2021.
  * Xu et al. [2019] Q. Xu, W. Wang, D. Ceylan, R. Mech, and U. Neumann.  DISN: Deep implicit surface network for high-quality single-view 3d reconstruction.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019\.
  * Yariv et al. [2021] L. Yariv, J. Gu, Y. Kasten, and Y. Lipman.  Volume rendering of neural implicit surfaces.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Zhang et al. [2020] K. Zhang, G. Riegler, N. Snavely, and V. Koltun.  NeRF++: Analyzing and improving neural radiance fields.  _arXiv preprint arXiv:2010.07492_ , 2020.
  * Zhang et al. [2018] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.

##  Appendix A Additional implementation details

###  A.1 MLP architecture in ablation

We detail the architecture of the ‘1-MLP‘ model introduced in Sec. 4.3. We
present the detailed architecture in Tab. A.1. We use a series of
DenseGeneral(DG) layers and a final Dense available in Flax [21]. The
architecture was determined by running a sweep over various depths. We found
that further increase in model capacity leads to poor generalization.

Layer | Input Dimension | Output Dimension
---|---|---
DG($F$, $256$) |  $B$ x $N$ x $P$ x $F$ |  $B$ x $N$ x $P$ x $256$
DG12($256$, $256$) |  $B$ x $N$ x $P$ x $256$ |  $B$ x $N$ x $P$ x $256$
DG($P$, $1$) |  $B$ x $N$ x $P$ x $256$ |  $B$ x $N$ x $256$
DG($N$, $1$) |  $B$ x $N$ x $256$ |  $B$ x $256$
Dense($256$, $1$) |  $B$ x $256$ |  $B$ x $3$
Table A.1: 1-MLP Architecture. We use notations $B$ for batch size, $N$ for
number of reference views, $P$ for number of epipolar projection and $F$ for
feature dimension. DG12 represents 12 layers of DenseGeneral. We also add skip
connections at every fourth layer in DG12. The final output corresponds to the
predicted color.

###  A.2 Spherical light field encoding

For $None$ scenes, we use the two-sphere light field parametrization [7]. Each
ray is represented by two points on the sphere, by the 4D tuple
$(\theta_{1},\phi_{1},\theta_{2},\phi_{2})$. To encode this representation we
found advantageous to use the spherical harmonics basis instead of the
sinusoidals. For a given ray, we evaluate a number of spherical harmonics at
each intersection and concatenate them to obtain its encoding,

$\tilde{Y}_{m}^{\ell}(\theta_{1},\phi_{1},\theta_{2},\phi_{2})=\left[Y_{m}^{\ell}(\theta_{1},\phi_{1})\parallel
Y_{m}^{\ell}(\theta_{2},\phi_{2})\right],$ (5)

where $Y_{m}^{\ell}(\theta,\phi)$ denotes the spherical harmonics of degree
$\ell$ and order $m$ evaluated at $(\theta,\phi)$. In our experiments, we
concatenate all the zonal and sectoral harmonics ($m=0$ and $m=\ell$) upto a
maximum degree of $4$.

To demonstrate the efficacy of the spherical harmonics encoding we conduct an
ablation on the blender dataset where we replace the spherical encoding with
the regular positional encoding in NeRF [32]. We refer to this model as
OursP.E.. We report the average metric on the blender dataset for this
ablation in Tab. A.2.

Model |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  Avg. $\downarrow$
---|---|---|---|---
OursP.E. | 33.18 | 0.979 | 0.027 | 0.0123
Ours | 33.85 | 0.981 | 0.024 | 0.0110
Table A.2: Light field encoding ablation on Blender dataset.

###  A.3 Metric computation

To compute the SSIM metric we use the function available in scikit-image
package. To compute the LPIPS on forward facing scene (RFF and Shiny), similar
to NeX, we use the VGG model222We use the library provided by
https://github.com/richzhang/PerceptualSimilarity. from [63]. On the blender
scenes, similar to Mip-NeRF, we use the VGG model available in tensorflow hub
to compute LPIPS. We use two different implementation on LPIPS to ensure
fairness of comparison.

##  Appendix B Additional results

###  B.1 Real-forward-facing dataset (RFF)

The RFF dataset introduced by Mildenhall et al. [31] consists of 8 forward
facing captures with each scene consisting of around 20 to 62 images. We
present the scene-wise breakdown of the results in Table B.3. The metrics for
NeRF and NeX are the ones reported in NeX [56].

|  | PSNR |  | SSIM |  | LPIPS
---|---|---|---|---|---|---
Model |  | NeRF | NeX | Ours |  | NeRF | NeX | Ours |  | NeRF | NeX | Ours
Fern |  | 25.49 | 25.63 | 24.86 |  | 0.866 | 0.887 | 0.886 |  | 0.278 | 0.205 | 0.135
Flower |  | 27.54 | 28.90 | 29.82 |  | 0.906 | 0.933 | 0.939 |  | 0.212 | 0.150 | 0.107
Fortress |  | 31.34 | 31.67 | 33.22 |  | 0.941 | 0.952 | 0.964 |  | 0.166 | 0.131 | 0.119
Horns |  | 28.02 | 28.46 | 29.78 |  | 0.915 | 0.934 | 0.957 |  | 0.258 | 0.173 | 0.121
Leaves |  | 21.34 | 21.96 | 22.47 |  | 0.782 | 0.832 | 0.856 |  | 0.308 | 0.173 | 0.110
Orchids |  | 20.67 | 20.42 | 21.05 |  | 0.755 | 0.765 | 0.807 |  | 0.312 | 0.242 | 0.173
Room |  | 32.25 | 32.32 | 34.54 |  | 0.972 | 0.975 | 0.987 |  | 0.196 | 0.161 | 0.104
Trex |  | 27.36 | 28.73 | 30.34 |  | 0.929 | 0.953 | 0.968 |  | 0.234 | 0.192 | 0.143
Table B.3: Scene-wise breakdown of quantitative results on the Real Forward-Facing dataset. |  | PSNR |  | SSIM |  | LPIPS
---|---|---|---|---|---|---
Model |  | NeRF | NeX | Ours |  | NeRF | NeX | Ours |  | NeRF | NeX | Ours
CD |  | 30.14 | 31.43 | 35.25 |  | 0.093 | 0.958 | 0.989 |  | 0.206 | 0.129 | 0.041
Tools |  | 27.45 | 28.16 | 26.55 |  | 0.938 | 0.953 | 0.945 |  | 0.204 | 0.151 | 0.130
Crest |  | 20.30 | 21.23 | 21.73 |  | 0.670 | 0.757 | 0.797 |  | 0.315 | 0.162 | 0.079
Seasoning |  | 27.79 | 28.60 | 28.34 |  | 0.898 | 0.928 | 0.936 |  | 0.276 | 0.168 | 0.102
Food |  | 23.32 | 23.68 | 22.88 |  | 0.796 | 0.832 | 0.821 |  | 0.308 | 0.203 | 0.151
Giants |  | 24.86 | 26.00 | 27.06 |  | 0.844 | 0.898 | 0.928 |  | 0.270 | 0.147 | 0.065
Lab |  | 29.60 | 30.43 | 35.28 |  | 0.936 | 0.949 | 0.989 |  | 0.182 | 0.146 | 0.066
Pasta |  | 21.23 | 22.07 | 21.63 |  | 0.789 | 0.844 | 0.855 |  | 0.311 | 0.211 | 0.096
Table B.4: Scene-wise breakdown of quantitative results on the Shiny dataset.

###  B.2 Shiny dataset

The Shiny dataset introduced in NeX [56] presents $8$ scenes with challenging
view dependent effects, captured by forward-facing cameras. We present the
scene-wise breakdown of the results in Tab. B.4. The metrics for NeRF and NeX
are the ones reported in NeX [56].

###  B.3 Blender dataset

The Blender dataset introduced by Mildenhall et al. [32] consists of 8 scenes
each containing $800\times 800$ resolution images rendered from viewpoints
randomly sampled on a hemisphere around the object. We present the scene-wise
breakdown of the results in Tab. B.5. The metrics for NeRF and Mip-NeRF were
obtained from Mip-NeRF [4].

|  | PSNR |  | SSIM |  | LPIPS
---|---|---|---|---|---|---
Model |  | NeRF | Mip-NeRF | Ours |  | NeRF | Mip-NeRF | Ours |  | NeRF | Mip-NeRF | Ours
Chair |  | 34.08 | 35.14 | 35.30 |  | 0.975 | 0.981 | 0.989 |  | 0.026 | 0.021 | 0.012
Drums |  | 25.03 | 25.48 | 25.83 |  | 0.925 | 0.932 | 0.955 |  | 0.071 | 0.065 | 0.045
Ficus |  | 30.43 | 33.29 | 33.38 |  | 0.967 | 0.980 | 0.987 |  | 0.032 | 0.020 | 0.010
Hotdog |  | 36.92 | 37.48 | 38.66 |  | 0.979 | 0.982 | 0.993 |  | 0.030 | 0.027 | 0.009
Lego |  | 33.28 | 35.7 | 35.76 |  | 0.968 | 0.978 | 0.989 |  | 0.031 | 0.021 | 0.010
Materials |  | 29.91 | 30.71 | 35.10 |  | 0.953 | 0.959 | 0.990 |  | 0.047 | 0.040 | 0.011
Mic |  | 34.53 | 36.51 | 35.32 |  | 0.987 | 0.991 | 0.992 |  | 0.012 | 0.009 | 0.008
Ship |  | 29.36 | 30.41 | 30.94 |  | 0.869 | 0.882 | 0.952 |  | 0.150 | 0.138 | 0.084
Table B.5: Scene-wise breakdown of quantitative results on the Blender
dataset.

##  Appendix C Additional experiments and visualizations

###  C.1 Plücker coordinates

Our experiments use ray parametrizations specific to the camera configuration
of each type of scene. For forward facing scenes, we employ the light slab
light field representation, while for $None$ scenes we use the two-sphere. In
this section, we explore the alternative of using Plücker coordinates, which
are generic and can represent any kind of camera configuration. Since our
architecture is agnostic to the light field parametrization, we simply replace
the input ray representation with the 6D Plücker coordinates to perform this
experiment. When using Plücker coordinates, we observe a drop of 0.18 PSNR on
the RFF dataset as compared to the light slab representation. Similarly, we
observe a drop of 0.25 PSNR on the Blender dataset when replacing the two-
sphere parametrization with Plücker coordinates. This suggests that for
particular configurations, the specific (and lower dimensional) ray
parametrizations have a slight advantage over a generic parametrization such
as Plücker coordinates.

Figure C.8: Crops from two different regions of the _Lab_ scene in the Shiny
dataset. The Vanilla-NLF model (described in Section 4.3) is able to retrieve
a majority of the refraction details but fails to reproduce high frequencies.
NeX reproduces sharp details but not the refractions. Our model does well in
both regions.

###  C.2 Handling view-dependent effects

While our model is built around geometric constraints (such as epipolar
geometry), the attention-based modeling provides the capability to downweigh
such constrains when not useful, in order to more directly associate a color
to ray coordinates (as in the Vanilla-NLF model described in Section 4.3). We
speculate that this, together with the convolutional features (which bring
some context) explains our superior performance on view-dependent effects.

We run a mini-ablation to investigate this hypothesis. Figure C.8 shows, for
the Lab scene from the Shiny dataset, one crop with transparency/refraction
and another that is diffuse and contains sharp details. Our model works well
on both regions which indicates its flexibility, in contrast with the
baselines.

###  C.3 Visualizing view attention

The attention weights $\beta^{j}$ (in Eq. 3) correspond to “importance” of
each reference view when rendering a target pixel. We visualize these
attention weight for a test image in the chair scene from Blender dataset in
Fig. C.9. We explain the visualization process in the figure caption.

Figure C.9: View attention weight visualization. We visualize the attention
weights $\beta^{j}$ for each rendered pixel for a test image from the chair
scene. For each target pixel, we consider three reference views. Thus we have
three attention weights $\beta^{1},\beta^{2}$ and $\beta^{3}$ corresponding to
reference views 1, 2 and 3 respectively. We treat these attention weights as
RGB value and visualize them as an image as shown above. Intuitively, this
image shows the contribution of each reference view when rendering a pixel.
For example, the cushion is predominantly green as it is most visible in
second reference view. Similarly the back of the chair contains almost equal
mix of red and blue as it is equally visible in reference views 1 and 3. We do
not show the attention weights for the background pixels for clarity of
visualization.


# Laboratory tests for the cosmic neutrino background using beta-decaying
nuclei

Bob McElrath  CERN, Geneva 23, CH-1211 Switzerland

###### Abstract

We point out that the Pauli blocking of neutrinos by cosmological relic
neutrinos can be a significant effect. For zero-energy neutrinos, the standard
parameters for the neutrino background temperature and density give a
suppression of approximately $1/2$. We show the effect this has on three-body
beta decays. The size of the effect is of the same order as the recently
suggested neutrino capture on beta-decaying nuclei.

###### keywords:

##  1 Introduction

The last remaining remnant of the big bang, which is composed of a known
particle is the Cosmic Neutrino Background (CNB). It decouples from thermal
equilibrium at $T\sim 2$ MeV. It gives information about the universe at a
time significantly before the decoupling of photons at $T\sim 1$ eV. The
processes responsible for their creation and decoupling are well understood
nuclear physics.

Today these neutrinos are expected to be extremely cold ($1.952{\rm
K}=1.68\times 10^{-4}{\rm eV}$). As such, they are extremely difficult to
detect due to the fact that weak interaction cross sections scale as
$(G_{F}E)^{2}$. They have a density of $\rho_{\nu}=3/22n_{\gamma}=56/{\rm
cm}^{3}$ per species of neutrino and anti-neutrino, corresponding to a
luminosity $\mathcal{L}=1.7\times 10^{13}/{\rm cm^{2}s}$ if the neutrinos were
massless. [1, 2] These numbers rely on a specific cosmological model which
could be substantially modified, if neutrinos cluster gravitationally, or if
they have nontrivial dynamics after freeze-out. [3] It has also recently been
shown that these neutrinos are a quantum liquid, and their fluctuations have
the quantum numbers of a graviton, opening the prospect that measurements of
relic neutrinos could then be compared with gravitational constants. [4]

Nuclei that undergo $\beta$-decay are a precise and specific laboratory to
look for the CNB. There exists a vast array of nuclei that can emit or absorb
neutrinos at a wide range of energies. A signal seen using $\beta$-decaying
nuclei constitutes a specific test because they are already known to emit or
absorb lepton number in specific ways. All other proposals could not in
principle tell if the effect was due to an object with lepton number. [1, 2]
High-energy neutrinos (e.g. Z-burst) can be absorbed by things which do not
carry lepton number, and anomalous forces could have a variety of sources that
have nothing to do with lepton number (for instance, a Dark Matter wind).
Finally the effects of neutrinos on the CMB cannot be disentangled from other
relativistic species that are not be fermionic or do not carry lepton number.

Rates using decaying nuclei are in principle much higher than other proposals
such as coherent scattering, because the energy for the observation is coming
from nuclear mass differences, and not from the neutrinos themselves. The
energy $Q$ in nuclear transitions is $\mathcal{O}({\rm MeV})$. Giving the CNB
neutrinos this energy in a coherent experiment would require moving with a
velocity corresponding to a boost factor $\gamma=Q/T_{\nu}\simeq 10^{10}$. For
comparison, $\gamma$ at the LHC with protons is about 15000, or 5500 with
Lead.

There are two ways to see an effect of the neutrino background using a beta-
decaying nucleus: add a neutrino to it or remove a neutrino from it. Both were
suggested by Weinberg in 1962. [5]

Adding a neutrino to the background is suppressed for momenta which are
already occupied by the CNB thermal distribution, due to the fact that
neutrinos are fermions and their chemical potential and average energy are
similar. This is an $\mathcal{O}(1)$ effect, if one can create neutrinos
having the correct energy.

Removing a neutrino from the CNB using nuclei is known as neutrino capture
($\nu$C). Capture of reactor neutrinos is the original mode used to discover
the neutrino. Recently there has been a surge of interest in this mode for
detecting the CNB using $\beta$-decaying nuclei, which can have zero
threshold.[6, 7, 8]

##  2 Pauli Blocking by The Cosmic Neutrino Background

The CNB is a thermal distribution in a particular frame $u^{\alpha}$ which we
assume to be coincident with the dipole from the Cosmic Microwave Background,
which points in the direction $(264.85\pm 0.10)^{\circ},(48.25\pm
0.04)^{\circ}$ in galactic coordinates, with velocity $368\pm 2$ km/s. Its
thermal distribution is then

$\displaystyle
F_{i}(\vec{p})=\left[e^{(p^{\alpha}u_{\alpha}-\mu_{i})/kT}+1\right]^{-1}$ (1)

for each species of neutrino and anti-neutrino $i$, having mass $m_{i}$ and
chemical potential $\mu_{i}$ and four-momentum $p^{\alpha}$ in the cosmic rest
frame $u^{\alpha}=(1,\vec{0})$ and this reduces to the usual non-relativistic
Fermi-Dirac distribution. The relativistic chemical potential is the Fermi
energy at zero temperature, $\mu_{i}=E_{F}=\sqrt{m^{2}+p_{F}^{2}}$, and the
nominal Fermi momentum predicted by the standard cosmological model is
$p_{F}=\sqrt[3]{3\pi^{2}\rho_{\nu}}=\sqrt[3]{3\pi^{2}\rho_{\overline{\nu}}}=3.6\times
10^{-5}$ eV, where $\rho_{\nu}$ is the number density per flavor. We will
refer to this as the “standard” chemical potential.

Figure 1: Sum of suppression factor $1-F_{i}(p)$ for three mass eigenstates
vs. neutrino momentum for several values of the neutrino mass, assuming a
standard chemical potential.

Figure 2: The differential event rate as a function of neutrino momentum for
several choices of neutrino mass and normal/inverted hierarchy, assuming a
standard chemical potential and the kinematics of Tritium.

A process which emits neutrinos has a suppression $[1-F_{i}(\vec{p})]$ due to
Pauli blocking from this thermal distribution. This is independent of whether
the neutrinos are described as a localized classical gas having small
uncertainty $\Delta x\ll n^{-1/3}$ or a quantum liquid $\Delta x\gg n^{-1/3}$
for number density $n$. For a beta decay this is

$d\Gamma=2\pi\sum_{i}\int|\mathcal{M}_{i}|^{2}\xi_{i}^{2}[1-F_{i}(\vec{p})]dPS$
(2)

where $dPS$ is the differential phase space, $\xi_{i}$ is the eigenvector
component of neutrino mass eigenstate $i$ in the electron neutrino direction,
$\mathcal{M}$ is the matrix element of the emission of an electron-type
neutrino with mass $m_{i}$, and one sums over the mass eigenstates since final
state emitted particles must be in a mass eigenstate. This suppression factor
is experimentally indistinguishable from $1$ except in a region in which the
emitted neutrino has the same energy as the CNB. We plot the suppression
factor, summed over flavors in Fig.2

The Matrix Element for the beta decay is

$\displaystyle|\mathcal{M}|^{2}=\frac{G_{F}^{2}}{128\pi^{3}M_{I}^{2}}\big{[}$
$\displaystyle(g_{V}+g_{A})^{2}(p_{J}\cdot p_{e})(p_{I}\cdot p_{\nu})$
$\displaystyle+$ $\displaystyle(g_{V}-g_{A})^{2}(p_{I}\cdot p_{e})(p_{J}\cdot
p_{\nu})$ $\displaystyle+$ $\displaystyle(g_{V}^{2}-g_{A}^{2})(p_{I}\cdot
p_{J})(p_{e}\cdot p_{\nu})\big{]}$

where $g_{V}$ and $g_{A}$ are the vector and axial vector weak charges of the
atom. For $I$=neutron, $g_{V}=g_{A}=-1/2$. The matrix element also reaches a
minimum $|\mathcal{M}|^{2}=0$ at $p_{\nu}$=0, so the event rate at $p_{\nu}=0$
is zero. Because of this, the minimum in Fig.2 is deceptive. The differential
event rate including the matrix element is plotted in Fig.2 for tritium,
assuming a standard chemical potential.

To place a neutrino into the background with significant suppression, we need
that the invariant $p^{\alpha}u_{\alpha}/kT_{\nu}\mathrel{\raise
1.29167pt\hbox{$<$\kern-7.5pt\lower 4.30554pt\hbox{$\sim$}}}E_{F}/kT_{\nu}$.
Since our velocity with respect to the cosmic rest frame is small
($\beta=\frac{v}{c}=1.23\times 10^{-3}$), one can ignore our velocity and
pretend we can do the experiment in the cosmic rest frame. 111One must use
Special relativity and not Galilean relativity here. The atoms and electron
may be non-relativistic, but the neutrino is relativistic.

In a normal $\beta^{\pm}$ decay an atom $I$ decays to atom $J$ by emitting an
electron or positron and an anti-neutrino or neutrino. If one can precisely
measure the momenta of $I$, $J$, and $e^{\pm}$, one can solve for the neutrino
momenta. This requires momentum resolution on each of order $\delta
p\mathrel{\raise 1.29167pt\hbox{$<$\kern-7.5pt\lower
4.30554pt\hbox{$\sim$}}}\sqrt{2mkT_{\nu}}$. If the initial state $I$ is at
rest, this corresponds to a temperature

$T=\frac{2}{3}\frac{m_{\nu}}{M_{I}}T_{\nu}\simeq 1.40\times
10^{-9}\left(\frac{m_{\nu}}{\rm eV}\right)\left(\frac{\rm
amu}{M_{I}}\right)\rm K.$

Stated another way, the de Broglie wavelength of the neutrinos is 1.2 mm.
Since the uncertainty on momentum scales with momentum, the initial state must
have a similar de Broglie wavelength. Modern atomic Bose-Einstein Condensate
and Degenerate Fermi Gas experiments using laser and evaporative cooling
routinely reach $10^{-9}$ K today. Another promising technology to get to
these precisions in the initial state is “Crystallized Beams”. [9]

The final state of the decay is the atom $J$ almost exactly back-to-back with
the electron or positron. Similarly these final state particles must be
measured with a precision

$\frac{\delta p}{p}\simeq\sqrt{\frac{2mkT_{\nu}}{Q(Q+2m_{e})}}\simeq
1.33\times 10^{-7}\sqrt{\frac{m_{\nu}}{eV}}\sqrt{\frac{18{\rm keV}}{Q}}.$

where in the last term we assume $Q<2m_{e}$.

One might wonder if the effect shown here can impact experiments such as
KATRIN which attempt to measure the neutrino mass using the highest energy
electrons in a beta decay. KATRIN attempts to measure mass due to the change
in slope and rate suppression near the endpoint, and they do not have the
resolution to see the actual endpoint itself. Their resolution is
approximately $\Delta E_{e}\sim 1$ eV. The effects here only effect the
highest (electron) energy bin, and reduce the number of events there by
$\mathcal{O}(10^{-18}N)$ where $N$ is the total decays they see. Existing
experiments simply do not have the rate for this effect to be a concern.

##  3 Acknowledgements

We thank Patrick Huber and Mats Landroos for fruitful discussions.

## References

  * [1] A. Ringwald, arXiv:hep-ph/0505024.
  * [2] G. B. Gelmini, Phys. Scripta T121 (2005) 131 [arXiv:hep-ph/0412305].
  * [3] A. Ringwald and Y. Y. Y. Wong, JCAP 0412 (2004) 005 [arXiv:hep-ph/0408241].
  * [4] B. McElrath, arXiv:0812.2696 [gr-qc].
  * [5] S. Weinberg, Phys. Rev. 128 (1962) 1457.
  * [6] A. G. Cocco, G. Mangano and M. Messina, JCAP 0706 (2007) 015 [arXiv:hep-ph/0703075].
  * [7] R. Lazauskas, P. Vogel and C. Volpe, J. Phys. G 35 (2008) 025001 [arXiv:0710.5312 [astro-ph]].
  * [8] M. Blennow, arXiv:0803.3762 [astro-ph].
  * [9] H. Danared, A. Källberg, K.-G. Rensfelt, and A. Simonsson, Phys. Rev. Lett. 88, 174801 (2002).