# FlipNeRF: Flipped Reflection Rays for Few-shot Novel View Synthesis

Seunghyeon Seo Yeonjin Chang Nojun Kwak
Seoul National Univeristy
{zzzlssh, yjean8315<EMAIL_ADDRESS>
###### Abstract

Neural Radiance Field (NeRF) has been a mainstream in novel view synthesis
with its remarkable quality of rendered images and simple architecture.
Although NeRF has been developed in various directions improving continuously
its performance, the necessity of a dense set of multi-view images still
exists as a stumbling block to progress for practical application. In this
work, we propose FlipNeRF, a novel regularization method for few-shot novel
view synthesis by utilizing our proposed flipped reflection rays. The flipped
reflection rays are explicitly derived from the input ray directions and
estimated normal vectors, and play a role of effective additional training
rays while enabling to estimate more accurate surface normals and learn the 3D
geometry effectively. Since the surface normal and the scene depth are both
derived from the estimated densities along a ray, the accurate surface normal
leads to more exact depth estimation, which is a key factor for few-shot novel
view synthesis. Furthermore, with our proposed Uncertainty-aware Emptiness
Loss and Bottleneck Feature Consistency Loss, FlipNeRF is able to estimate
more reliable outputs with reducing floating artifacts effectively across the
different scene structures, and enhance the feature-level consistency between
the pair of the rays cast toward the photo-consistent pixels without any
additional feature extractor, respectively. Our FlipNeRF achieves the SOTA
performance on the multiple benchmarks across all the scenarios. The codes and
more qualitative results are available in our project page:
https://shawn615.github.io/flipnerf/.

##  1 Introduction

(a) Sparse training views

mip-NeRF [1] |  Ref-NeRF [36] |  MixNeRF [31] |  FlipNeRF (Ours)
---|---|---|---

(b) Synthesized unseen views

Figure 1:  Synthesis results from sparse inputs. Our FlipNeRF significantly
improves rendering quality compared to other baselines. Compared to the
vanilla mip-NeRF [1] and MixNeRF [31], which is the state-of-the-art
regularization method, ours reduces the noises and floating artifacts
noticeably with superior surface normal estimation. Although Ref-NeRF [36]
estimates smooth normal vectors, it shows much inferior rendering results to
ours with a large chunk of noise under the few-shot setting.

Neural Radiance Field (NeRF) [24] has achieved great success in rendering
photo-realistic images from novel viewpoints. However, the necessity of a
dense set of training images remains as a practical bottleneck since it
suffers from significant performance degradation when trained with sparse
views.

There are two mainstreams for few-shot novel view synthesis: pre-training and
regularization methods, both of which focus on learning the 3D geometry
efficiently from sparse inputs. The pre-training methods [43, 5, 6, 39, 20,
12, 19, 29, 35, 14] require large-scale datasets consisting of different
scenes with multi-view images for injecting prior knowledge during the pre-
training, while the regularization methods [25, 31, 15, 11, 30, 8, 17] are
optimized per scene, exploiting additional training resources, _e.g_. unseen
viewpoints [25, 15, 17], depth map generation [8, 30], off-the-shelf models
[11, 25], and so on, for an effective regularization to alleviate overfitting.
Although the prior arts achieved promising results in novel view synthesis
from sparse inputs, there still exist hurdles to overcome. The large-scale
datasets, which are used for pre-training methods, are expensive to collect
and the NeRF model is prone to performance degradation for the out-of-
distribution dataset. On the other hand, the regularization methods heavily
rely on additional training resources which might not always be available and
require many heuristic factors, _e.g_. the choice of off-the-shelf models, the
hyperparameters for sampling unseen viewpoints, and so on.

In this paper, we propose FlipNeRF, which is an effective regularization
method exploiting the flipped reflection rays111The term ‘flipped’ is used
because the reflected ray has an opposite direction (from an object to a
camera). as additional training resources with filtering the ineffective newly
generated rays. We derive a batch of flipped reflection rays from the original
ray directions and estimated surface normals so that they are cast toward the
same target pixels of the original input ray. Compared to the existing
regularization methods which have mainly focused on the accurate depth
estimation from limited input views [31, 25, 15, 30, 8], our FlipNeRF is
trained to reconstruct surface normals accurately by learning to generate
effective reflection rays to be used in training. Since both estimated surface
normals and depths are derived from the volume densities representing
underlying 3D geometry, accurately estimating the surface normals of an object
naturally leads to more accurate depth maps.

Furthermore, we propose an effective regularization loss, Uncertainty-aware
Emptiness Loss (UE Loss), to reduce the floating artifacts effectively while
considering the uncertainty of the model’s outputs by using the estimated
scale parameters for mixture models. Since our FlipNeRF is built upon MixNeRF
[31], which is a regularization method achieving promising results by modeling
input rays with mixture density models [2], we are able to apply our proposed
loss without any modification of the architecture by using the estimated scale
parameters of each sample along a ray, which stand for the uncertainty of the
samples’ estimated probability density distributions.

Additionally, inspired by [6, 11, 15] which address the feature-level
consistency of targets under the sparse input setting, we encourage the
consistency for the pairs of bottleneck features between the original input
rays and flipped reflection rays. We leverage a Jensen-Shannon Divergence,
which is based on the similarity between the probability distributions, to
make the pairs of bottleneck feature distributions of original and flipped
reflection rays more similar to each other improving feature consistency.

We demonstrate the effectiveness of our proposed FlipNeRF through the
experiments on the multiple benchmarks, _e.g_. Realistic Synthetic 360∘ [24],
DTU [13], and LLFF [23]. Our method achieves state-of-the-art (SOTA)
performances compared to other baselines. Especially, ours outperforms other
baselines by a large margin with more accurate surface normals under the
extremely sparse settings such as 3/4-view setting which are the most
challenging ones. Our contributions are summarized as follows:

  * •

We propose an effective training framework for NeRF with sparse training
views, called FlipNeRF. It leverages flipped reflection rays to provide
additional training resources, resulting in more precise surface normals with
our proposed masking strategy to filter the ineffective rays.

  * •

We also propose an effective regularization loss, Uncertainty-aware Emptiness
Loss (UE Loss), which reduces floating artifacts with considering the
uncertainty of outputs, leading to more reliable estimation.

  * •

We enhance the consistency of bottleneck features between the original input
rays and flipped reflection rays by Jensen-Shannon Divergence, coined as
Bottleneck Feature Consistency Loss (BFC Loss), improving the robustness for
rendering from unseen viewpoints.

  * •

Our FlipNeRF achieves SOTA performance over the multiple benchmarks.
Especially, ours outperforms other baselines by a large margin in more
challenging scenarios, _e.g_. 3/4-view.

Figure 2: Overall framework of FlipNeRF. Our FlipNeRF utilize the newly
generated flipped reflection rays with our proposed UE Loss and BFC Loss as
well as existing MSE, NLL and Orientation losses. See Sec. 3 and Fig. 3 for
more details about generation process of flipped reflection rays and the loss
terms.

##  2 Related Works

###  2.1 Neural Radiance Field

Recently, Neural Radiance Field (NeRF) [24] has shown impressive performance
and potential in the novel view synthesis task. NeRF represents a scene with
an MLP, mapping coordinates and viewing directions to its colors and volume
density, and then creates a novel view through volume rendering. Subsequent
studies have developed NeRF in several directions, _e.g_. using conical
frustums instead of rays [1], reparameterizing an input viewing direction as
its reflection direction [36], and so on. These works have made significant
progress by addressing the various issues in novel view synthesis, but there
still exists a limitation in that NeRF requires a dense set of training images
and a lengthy training time. Many studies have addressed these issues [10, 28,
5], including the utilization of various data structures for faster training
and inference [9, 42] and attempts to train NeRF with only a few training
images. Our work focuses on enhancing the performance of NeRF when a sparse
set of views are provided as training images.

###  2.2 Few-Shot Novel View Synthesis

There are two main approaches for a few-shot novel view synthesis: the pre-
training and the regularization method. The pre-training methods require a
large dataset of multi-view scenes to provide prior knowledges of 3D geometry
to a NeRF model and then optionally finetune on the target scene [5, 6, 12,
19, 29, 35, 39, 43]. On the contrary, the regularization methods [31, 15, 11,
25, 8, 30, 17] are optimized per scene without pre-training process by
exploiting additional training resources, _e.g_. depth maps [8, 30] and
semantic consistency [11], as an extra supervision. Among them, [25, 15, 17]
adopt an unseen viewpoint sampling strategy to make up for insufficient
training views. However, these sampling processes require many hand-designed
factors such as the ranges of rotation, translation, jittering, and so on,
which can introduce artificial biases. Our work proposes a novel
regularization approach to derive a set of flipped reflection rays from
estimated surface normals and utilizes these for regularization, which does
not require a heuristic factor to be finetuned, resulting in more effective
training strategy with limited inputs.

###  2.3 Surface Normal Reconstruction

There is a line of research to recover accurate textures and lighting
conditions of objects with NeRF [26, 41, 38, 7, 34, 44, 3]. Although these
earlier studies successfully reconstruct the high-quality isosurfaces derived
from the scene representations, their rendering quality for novel views is
still inferior to the NeRF-like models. Meanwhile, Ref-NeRF [36] achieved
superior performance with remarkable quality of surface normals compared to
the existing NeRF models. Since the normal vectors utilized in NeRF framework
are derived from the negative normalized density gradients [34, 3, 36], which
represent the underlying geometry of 3D scenes, learning an accurate density
distribution along a ray is a key factor for surface normal reconstruction.
However, for the few-shot novel view synthesis, the prior works mostly focus
on the accurate depth estimation without attention to the surface normals,
both of which are derived from the estimated volume densities. In this work,
we approach the few-shot novel view synthesis problem with focusing on the
surface normals, which is another critical factor for an effective learning of
3D scene geometry. To the best of our knowledge, our work is the first attempt
to focus on the surface normal estimation for few-shot novel view synthesis.

##  3 Method

In this work, we propose an effective regularization method for few-shot novel
view synthesis with flipped reflection rays. Our FlipNeRF is built upon
MixNeRF [31] which leverages a mixture model framework (Sec. 3.1). We derive a
batch of flipped reflection rays and cast them toward the identical target
pixels as additional training rays (Sec. 3.2). Furthermore, we propose the
Uncertainty-aware Emptiness Loss and Bottleneck Feature Consistency Loss to
alleviate the floating artifacts adaptively based on the uncertainty and
enhance the consistency between the bottleneck feature distributions of the
original and flipped reflection rays, respectively (Sec. 3.3 and Sec. 3.4).
Finally, our FlipNeRF is trained to minimize the MSE and NLL losses as well as
the proposed regularization loss terms with their corresponding balancing
weights (Sec. 3.5). Fig. 2 shows an overview of our FlipNeRF.

Figure 3: Flipped reflection ray generation. Our FlipNeRF generates the
flipped reflection ray $\mathbf{r}^{\prime}$ from the estimated normal vector
$\hat{\mathbf{n}}$ and original input ray direction $\mathbf{d}$. With our
masking strategy, $\mathbf{r}_{2}^{\prime}$ is filtered out since it does not
satisfy the photo-consistency condition, _i.e_. $\theta_{2}$ is bigger than
90∘. The smaller $\theta$ is, the more photo-consistent the target pixel is,
where the pair of $\mathbf{r}$ and $\mathbf{r}^{\prime}$ are cast.

###  3.1 Preliminaries

#### NeRF.

The NeRF [24], which is an MLP-based neural network, represents a 3D scene as
a continuous radiance field of RGB color and volume density. For every point
sampled along a ray, the 3D coordinates $\mathbf{x}=(x,y,z)$ and viewing
directions $(\theta,\phi)$ are mapped to the colors $\mathbf{c}=(r,g,b)$ and
densities $\sigma$:

$F(\gamma(\mathbf{x}),\gamma(\hat{\mathbf{d}}))\to(\mathbf{c},\sigma),$ (1)

where $F(\cdot)$, $\gamma(\cdot)$, and $\hat{\mathbf{d}}$ indicate an MLP, the
positional encoding for the inputs, and the 3D Cartesian unit vector used as
an input viewing direction in practice, respectively.

The volumetric radiance field is rendered by alpha compositing the RGB values
along an input ray $\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$ [22], where
$\mathbf{o}$ and $\mathbf{d}$ denote the camera origin and unnormalized
direction vector, _i.e_.
$\mathbf{d}=\lVert\mathbf{d}\rVert_{2}\cdot\hat{\mathbf{d}}$, respectively.
The volume rendering integrals are denoted as follows:

$\mathbf{\hat{c}(r)}=\int_{t_{n}}^{t_{f}}T(t)\sigma(\mathbf{r}(t))\mathbf{c}(\mathbf{r}(t),\hat{\mathbf{d}})\,dt,$
(2)

where $T(t)=\exp(-\int_{t_{n}}^{t}\sigma(s)\,ds)$ indicates the degree of
transparency. In practice, it is approximated with numerical quadrature [24]
by sampling points along a ray.

The radiance field is trained to minimize the mean squared error (MSE) between
rendered and GT pixels:

$\mathcal{L}_{\text{MSE}}=\sum_{\mathbf{r}\in\mathcal{R}}||\mathbf{\hat{c}(r)}-\mathbf{c^{\text{GT}}(r)}||^{2}_{2}\,,$
(3)

where $\mathcal{R}$ is denoted as a set of rays.

#### MixNeRF.

Built upon mip-NeRF [1], which leveraged a cone tracing method and proposed an
integrated positional encoding to address an aliasing problem, MixNeRF [31]
estimates the joint probability distribution of color values and models a ray
with a mixture of densities:

$p(\mathbf{c}|\mathbf{r})=\sum_{i=1}^{M}\pi_{i}\mathcal{F}(\mathbf{c};\mu_{i}^{\mathbf{c}},\beta_{i}),$
(4)

where $M$ is the number of sampled points,
$\mathcal{F}(\mathbf{c};\mu_{i}^{\mathbf{c}},\beta_{i})$ denotes the Laplacian
distribution of RGB $\mathbf{c}$ with location parameter
$\mu_{i}^{\mathbf{c}}\in\\{\mu_{i}^{r},\mu_{i}^{g},\mu_{i}^{b}\\}$, _i.e_.
estimated RGB values of sample, and scale parameter
$\beta_{i}\in\\{\beta_{i}^{r},\beta_{i}^{g},\beta_{i}^{b}\\}$. The mixture
coefficient $\pi_{i}$ is derived from the estimated volume density
$\sigma_{i}$ as follows:

$\pi_{i}=\frac{w_{i}}{\sum_{m=1}^{M}w_{m}}=\frac{T_{i}(1-\exp(-\sigma_{i}\delta_{i}))}{\sum_{m=1}^{M}T_{m}(1-\exp(-\sigma_{m}\delta_{m}))},$
(5)

where $w_{i}$ and $\delta_{i}$ indicate the alpha blending weight and sample
interval, respectively. Thanks to the mixture model’s capacity of representing
complex distributions, MixNeRF learns the density distribution effectively
with sparse inputs by minimizing the negative log-likelihood (NLL) in Eq. 4.

Our FlipNeRF is built upon MixNeRF leveraging the mixture modeling framework
while achieving superior rendering quality with noticeably fewer artifacts and
more accurate surface normals to MixNeRF.

###  3.2 Auxiliary Flipped Reflection Ray

As shown in Fig. 3, we exploit a batch of flipped reflection rays
$\mathbf{r}^{\prime}\in\mathcal{R}^{\prime}$ as extra training resources,
which are derived from the original input ray directions $\mathbf{d}$ and
estimated surface normals $\hat{\mathbf{n}}$. First, we derive a flipped
reflection direction $\mathbf{d}^{\prime}$ from $\mathbf{d}$ and
$\hat{\mathbf{n}}$:

$\mathbf{d}^{\prime}=2(\mathbf{d}\cdot\hat{\mathbf{n}})\hat{\mathbf{n}}-\mathbf{d},$
(6)

where $\hat{\mathbf{n}}$ denotes the weighted sum of blending weights and
estimated normal vectors along a ray, _i.e_.
$\hat{\mathbf{n}}=\sum_{i=1}^{M}w_{i}\mathbf{n}_{i}$. 222Technically,
$\hat{\mathbf{n}}$ is not guaranteed to be a unit vector without an explicit
normalization process. However, we empirically found that the normalization
rather destabilizes the training and leads to the performance degradation.
Kindly refer to our supplementary material for related experiments. Note that
we use the gradient of volume density as estimated surface normals following
[3, 34, 36].

To generate the additional training rays based on $\mathbf{d}^{\prime}$, we
need a set of imaginary ray origins $\mathbf{o}^{\prime}$ located in a
suitable space considering the hitting point and the original input ray
origins $\mathbf{o}$. Since the vanilla NeRF models, which are trained with a
dense set of images, tend to have the blending weight distribution whose peak
is located on the point around the object surface
$\mathbf{p}_{s}=\mathbf{o}+t_{s}\mathbf{d}$ [8, 31], _i.e_. the $s$-th sample
whose blending weight is the highest along a ray. Therefore, we place
$\mathbf{o}^{\prime}$ so that the $s$-th sample of $\mathbf{r}^{\prime}$ is
$\mathbf{p}_{s}$:

$\mathbf{o}^{\prime}=\mathbf{p}_{s}-t_{s}\mathbf{d}^{\prime},$ (7)

resulting in our proposed flipped reflection ray,
$\mathbf{r}^{\prime}(t)=\mathbf{o}^{\prime}+t\mathbf{d}^{\prime}$. Compared to
the previous unseen viewpoint sampling strategies [15, 25, 17], our proposed
strategy does not rely on the randomness of unseen viewpoint sampling and
reduces the heuristic factors for sampling schemes, _e.g_. the range of
rotation, translation, and so on. Furthermore, since our newly generated
$\mathbf{r}^{\prime}$ are cast on the identical object surfaces where the
original input rays $\mathbf{r}$ are cast, _i.e_. the target pixels are photo-
consistent for the pair of $\mathbf{r}$ and $\mathbf{r}^{\prime}$ without any
sophisticated viewpoint sampling process, we are able to train
$\mathbf{r}^{\prime}$ effectively with the same GT pixels of $\mathbf{r}$.

However, since $\hat{\mathbf{n}}$, which are used to derive
$\mathbf{d}^{\prime}$, are not the ground truth but the estimation, there
exists a concern that even miscreated $\mathbf{r}^{\prime}$, which do not
satisfy photo-consistency, can be used for training. As a result, it might
lead to performance degradation while providing misleading training cues. To
address this problem, we mask the ineffective $\mathbf{r}^{\prime}$ by
considering the angle $\theta$ between $\hat{\mathbf{n}}$ and
$-\hat{\mathbf{d}}$ as follows:

$M(\mathbf{r}^{\prime})=\begin{cases}1&\text{if~{}}\arccos{(-(\hat{\mathbf{d}}\cdot\hat{\mathbf{n}}))}<\tau\\\
0&\text{otherwise}\end{cases},$ (8)

where $-(\hat{\mathbf{d}}\cdot\hat{\mathbf{n}})$ amounts to $\cos{\theta}$ of
original input rays and normal vectors, and $\tau$ indicates the threshold for
filtering the invalid rays, which we set as $90^{\circ}$ unless specified.
Through this masking process, only $\mathbf{r}^{\prime}$ which are cast toward
the photo-consistent point can be remained as we intend. Finally, our proposed
flipped reflection rays are modeled by mixture density like the original input
rays:

$p(\mathbf{c}|\mathbf{r}^{\prime})=\sum_{i=1}^{M}\pi_{i}^{\prime}\mathcal{F}(\mathbf{c};\mu_{i}^{\mathbf{c}}{{}^{\prime}},\beta_{i}^{\prime}).$
(9)

Additionally, we leverage the Orientation Loss proposed in Ref-NeRF [36] to
penalize the backward-facing normal vectors for learning accurate surface
normals:

$l_{\text{Ori.}}(\mathbf{r})=\sum_{i=1}^{M}w_{i}\max(0,\mathbf{n}_{i}\cdot\hat{\mathbf{d}})^{2}.$
(10)

Unlike Ref-NeRF, we penalize underlying density gradient normal
$\mathbf{n}_{i}$ instead of predicted normals.

Note that our FlipNeRF is fundamentally different compared to Ref-NeRF since
ours generates additional training rays through the derivation of reflection
direction without modification to original representations while Ref-NeRF
replaced the input viewing direction with its reflection direction,
reparameterizing the outgoing radiance.

###  3.3 Uncertainty-aware Regularization

Several regularization techniques have been proposed to reduce the floating
artifacts present in synthesized images, which is one of the major problems of
NeRF. Among them, we leverage the Emptiness Loss [37] which penalizes the
small blending weights along a ray as follows:

$l_{\text{Emp.}}(\mathbf{r})=\frac{1}{M}\sum_{i=1}^{M}\log(1+\eta\cdot
w_{i}),$ (11)

where the bigger $\eta$ is, the steeper the loss function becomes around $0$.

However, the naive application of existing regularization techniques with
limited training views might not be consistently helpful across the different
scenes due to the scene-by-scene different structure, resulting in overall
performance degradation. To address this problem, we propose Uncertainty-aware
Emptiness Loss (UE Loss) developed upon the Emptiness Loss, which reduces the
floating artifacts consistently over the different scenes by considering the
output uncertainty:

$\begin{split}l_{\text{UE}}(\mathbf{r})=\frac{1}{M}\sum_{i=1}^{M}\log(1+\rho\cdot\eta\cdot
w_{i}),\\\
\text{where}\quad\rho=\frac{1}{3}\sum_{c}^{\\{r,g,b\\}}\sum_{i=1}^{M}\beta_{i}^{c}.\end{split}$
(12)

$\rho$ amounts to the average of the summation of estimated scale parameters
of RGB color distributions from all samples along a ray, which we use as the
uncertainty of a ray. By our proposed UE Loss, we are able to regularize the
blending weights adaptively, _i.e_. the more uncertain a ray is, the more
penalized the blending weights along the ray are. It is able to reduce
floating artifacts consistently across the scenes with different structures
and enables to synthesize more reliable outputs by considering uncertainty.

###  3.4 Bottleneck Feature Consistency

Motivated by previous works addressing the feature-level consistency of
multiple views for few-shot novel view synthesis [6, 11, 15], we encourage the
consistency of bottleneck feature distributions between $\mathbf{r}$ and
$\mathbf{r}^{\prime}$, which are intermediate feature vectors, _i.e_. outputs
of the spatial MLP of NeRF, by Jensen-Shannon Divergence (JSD):

$l_{\text{BFC}}(\mathbf{r},\mathbf{r}^{\prime})=JSD(\psi(\mathbf{b}),\psi(\mathbf{b}^{\prime})),$
(13)

where $\psi(\cdot)$, $\mathbf{b}$ and $\mathbf{b}^{\prime}$ denote the softmax
function, the bottleneck features of $\mathbf{r}$ and $\mathbf{r}^{\prime}$,
respectively. While the existing methods [6, 11] rely on off-the-shelf feature
extractors like 2D CNN or CLIP [27] to address high-level feature consistency,
we regulate the pair of features effectively by enhancing consistency between
bottleneck features without depending on additional feature extractors.

###  3.5 Total Loss

Our FlipNeRF is not only trained to maximize the log-likelihood of the target
pixel $\mathbf{c}_{\text{GT}}$ for a set of original input rays $\mathcal{R}$,
but also for flipped reflection rays $\mathcal{R}_{M}^{\prime}$, where the
ineffective rays are excluded from the total flipped reflection rays
$\mathcal{R}^{\prime}$ by our masking strategy in Eq. 8. Likewise, the UE
Losses are applied for both $\mathcal{R}$ and $\mathcal{R}_{M}^{\prime}$.

Aggregating all, our total loss over a batch is as follows:

$\begin{split}\mathcal{L}_{\text{Total}}&=\mathcal{L}_{\text{MSE}}+\lambda_{1}\mathcal{L}_{\text{NLL}}+\lambda_{2}\mathcal{L}_{\text{NLL}}^{\prime}\\\
&+\lambda_{3}\mathcal{L}_{\text{UE}}+\lambda_{4}\mathcal{L}_{\text{UE}}^{\prime}+\lambda_{5}\mathcal{L}_{\text{BFC}}+\lambda_{6}\mathcal{L}_{\text{Ori.}},\end{split}$
(14)

where a set of $\lambda$’s are balancing weight terms for the losses.

##  4 Experiments

###  4.1 Experimental Settings

#### Implementation details.

Our FlipNeRF is built upon MixNeRF [31] based on the JAX codebase [4], which
is developed upon mip-NeRF [1]. The overall training scheme follows [25, 31].
We adopt a scene space annealing strategy for the early training phase. Also,
we apply the exponential decay and warm up for the learning rate. The Adam
optimizer [16] and the gradient clippings by value at 0.1 and norm at 0.1 are
used. We train our FlipNeRF for 500 pixel epochs with a batch size of 4,096 on
4 NVIDIA RTX 3090. Additionally, since the LLFF dataset [23] consists of
scenes with much more static movement of viewpoints compared to other
datasets, we set the threshold ($\tau$) of our masking strategy as 30∘ for the
experiments on LLFF. More detailed hyperparameters and our loss balancing
terms by the datasets and the number of training views are provided in the
supplementary material.

$\tau$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Average Err. $\downarrow$
---|---|---|---|---
30∘ | 18.62 | 0.747 | 0.206 | 0.121
60∘ | 18.12 | 0.723 | 0.237 | 0.126
90∘ |  red!5019.55 |  red!500.767 |  red!500.180 |  red!500.101
180∘ (No masking) |  red!2518.76 |  red!250.755 |  red!250.190 |  red!250.111

Table 1:  Comparison of masking conditions. Our masking strategy with $\tau$
of 90∘ achieves the best results, filtering out the ineffective flipped
reflection rays successfully.

(a) Effective $\mathbf{r}^{\prime}$ ratio by $\tau$

(b) Distribution of $\theta$

Figure 4: Analysis of the masking strategy.

#### Datasets and metrics.

We evaluate the performance of our FlipNeRF and baselines on the
representative benchmarks: Realistic Synthetic 360∘ [24], DTU [13], and LLFF
[23]. Realistic Synthetic 360∘ contains 8 synthetic scenes, each consisting of
400 multi-view rendered images with white background. To compare against other
representative baselines [15, 11, 31, 25, 1], we evaluate our FlipNeRF under
the scenarios of 4 and 8 views. For a fair comparison, we sample the first
$n$-image of the training set for the $n$-view scenario so that the identical
images are provided for training different methods following [31]. We use 200
images of the test set for evaluation. For DTU, which provide various scenes
including objects put on a white table with a black background, we conduct
experiments on the 15 specific scenes under the scenarios of 3, 6, and 9-view,
following the experimental protocol of [43]. We also conduct a series of
experiments for the analysis of FlipNeRF under 3-view setting as well as
comparison against other baselines. Additionally, we compare our FlipNeRF
against other baselines on LLFF consisting of real forward-facing scenes,
which is often tested as an out-of-distribution dataset for pre-training
methods. Following [24], every 8th image of each scene is used for a held-out
test set and the training views are evenly selected from the remaining images.
Like DTU, we report our results of the 3, 6, and 9-view scenarios, following
[43].

For the quantitative evaluation for rendered images, we adopt the mean of
PSNR, SSIM [40], LPIPS [45], and the geometric average [1]. Furthermore, we
also adopt the mean angular error (MAE∘) [36] and NLL [33, 32, 21, 18] for
evaluating the surface normals and uncertainty, respectively. Specifically,
following [33], we compute the NLL by deriving the probability of the GT pixel
values given a Gaussian distribution with the estimated RGB values as mean and
the uncertainty as variance, which is derived from the weighted sum of the
blending weights and estimated scale parameters along a ray in our FlipNeRF.
For DTU, we report the results evaluated by masked metrics to prevent
background bias, following [25, 31].

#### Baselines.

We compare our FlipNeRF against the SOTA regularization methods [11, 15, 25,
31] on Realistic Synthetic 360∘ as well as the vanilla mip-NeRF [1] and Ref-
NeRF [36], which is known for achieving promising results with accurate
surface normals. Furthermore, we compare ours against the representative pre-
training methods [5, 6, 43] as well as regularization methods on DTU and LLFF.
The pre-training baselines exploit the DTU and LLFF as pre-training dataset
and out-of-distribution test set, respectively, while the regularization
methods, mip-NeRF and Ref-NeRF are optimized per scene. Note that we report
the quantitative results of other baselines on DTU and LLFF from [25], which
achieved better results than its original papers by the modified training
scheme, and those on Realistic Synthetic 360∘ from [31], which trained the
baselines with the identical training views for a fair comparison.

###  4.2 Analysis of FlipNeRF

#### Analysis of flipped reflection rays.

As shown in Tab. 1, our masking strategy of filtering out the ineffective
flipped reflection rays with the threshold ($\tau$) of 90∘ achieves the best
performance among different options. With $\tau$ of 30∘ and 60∘, the rendering
quality is rather degraded since the newly generated rays are overly-filtered
and do not provide enough additional supervision as demonstrated in Fig. 4(a).
On the other hand, when we exploit all the flipped reflection rays without
masking, there exists a little improvement of performance compared to
$30^{\circ}$ and $60^{\circ}$ masking, but it is still much inferior to
$90^{\circ}$ due to the negative impact from the ineffective rays.
Additionally, Fig. 4(b) shows the distribution of $\theta$, _i.e_. angles
between the input viewing directions and normal vectors. As the smaller
$\theta$ is, the more photo-consistent the target pixel is, _i.e_. the more
effective the newly generated flipped reflection rays are for training. Since
our FlipNeRF is trained to estimate the accurate normal vectors, we are able
to exploit a set of more effective flipped reflection rays through the
training, which are cast on the more photo-consistent target pixels.
Furthermore, at the initial training phase, the invalid additional rays are
filtered effectively by our masking strategy, leading to high-quality
supervision and stabilizing the training.

| PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Average $\downarrow$ | NLL $\downarrow$
---|---|---|---|---|---
MixNeRF [31] | 18.95 | 0.744 | 0.203 | 0.113 | 9.99
FlipNeRF (Ours) |  |  |  |  |
w/o $\mathcal{L}_{\text{Emp.}}$ or $\mathcal{L}_{\text{UE}}$ |  red!2519.30 |  red!250.758 |  red!250.196 |  red!250.108 |  red!254.88
w/ $\mathcal{L}_{\text{Emptiness}}$ | 18.62 | 0.749 | 0.204 | 0.118 | 4.92
w/ $\mathcal{L}_{\text{UE}}$ |  red!5019.55 |  red!500.767 |  red!500.180 |  red!500.101 |  red!502.56

Table 2:  Effectiveness of $\mathcal{L}_{\text{UE}}$. Our proposed
$\mathcal{L}_{\text{UE}}$ improves the rendering quality consistently across
the scenes with considering the uncertainty.

|  RGB Image |  RGB Std. |  Depth Map |  Depth Std.
---|---|---|---|---

Figure 5:  Comparison of FlipNeRF and MixNeRF. Our FlipNeRF renders the images
from novel views with much fewer artifacts and more accurate depth maps than
MixNeRF. Considering the standard deviation of RGB and depths, our FlipNeRF is
able to estimate more reliable outputs than MixNeRF. For the std. map, the
darker the pixel is, the more certain the output is.

#### Uncertainty-aware regularization.

Tab. 2 shows the effectiveness of our proposed UE Loss. Compared to MixNeRF
[31] which models a ray with mixture of distributions as our FlipNeRF, ours
consistently achieves more reliable rendering results with much lower NLL.
Without $\mathcal{L}_{\text{Emp.}}$ [37] or our proposed
$\mathcal{L}_{\text{UE}}$, ours already outperforms MixNeRF by a large margin.
However, ours with naively leveraged $\mathcal{L}_{\text{Emp.}}$ rather shows
inferior results to MixNeRF. It shows that naive application of the existing
regularization technique for reducing artifacts under the few-shot setting can
lead to overall performance degradation due to the scene-by-scene various
structures. As illustrated in Fig. 5, with our proposed
$\mathcal{L}_{\text{UE}}$, ours improves both of the rendering quality and the
reliability of the model outputs by a large margin compared to MixNeRF.

| PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Average Err. $\downarrow$
---|---|---|---|---
w/o $\mathcal{L}_{\text{BFC}}$ | 18.79 |  red!250.755 | 0.200 | 0.115
w/ $\mathcal{L}_{\text{BFC}}$ |  |  |  |
MSE on $\mathbf{b}$ |  red!2518.83 |  red!250.755 | 0.197 |  red!250.113
Cos. Sim. on $\mathbf{b}$ | 18.77 |  red!250.755 |  red!250.193 | 0.115
JSD on $\mathbf{b}_{\mathbf{\hat{d}}}$ | 18.43 | 0.749 | 0.204 | 0.120
JSD on $\mathbf{b}$ |  red!5019.55 |  red!500.767 |  red!500.180 |  red!500.101

Table 3:  Comparison of different strategies for $\mathcal{L}_{\text{BFC}}$.
Our $\mathcal{L}_{\text{BFC}}$ achieves a significant performance gain
compared to other regularization schemes for feature consistency.

| $\mathcal{L}_{\text{NLL}}$ | $\mathcal{L}_{\text{NLL}}^{\prime}$ | ${\mathcal{L}_{\text{UE}}}^{\dagger}$ | $\mathcal{L}_{\text{BFC}}$ | $\mathcal{L}_{\text{Ori.}}$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Average $\downarrow$
---|---|---|---|---|---|---|---|---|---
(1) | ✓ |  |  |  |  | 17.44 | 0.729 | 0.187 | 0.130
(2) | ✓ | ✓ |  |  |  | 16.94 | 0.727 | 0.217 | 0.143
(3) | ✓ | ✓ | ✓ |  |  | 17.89 | 0.736 | 0.206 | 0.130
(4) | ✓ | ✓ | ✓ | ✓ |  | 19.01 | 0.755 |  red!250.181 |  red!250.107
(5) | ✓ | ✓ | ✓ |  | ✓ | 18.79 | 0.755 | 0.200 | 0.115
(6) | ✓ | ✓ |  | ✓ | ✓ |  red!2519.30 |  red!250.758 | 0.196 | 0.108
(7) | ✓ | ✓ | ✓ | ✓ | ✓ |  red!5019.55 |  red!500.767 |  red!500.180 |  red!500.101

Table 4:  Ablation study. $\dagger$ indicate that the losses are applied to
both $\mathbf{r}$ and $\mathbf{r}^{\prime}$.

#### Bottleneck feature consistency.

As shown in Tab. 3, there is no significant impact on the performance with
regularizing the bottleneck feature $\mathbf{b}$ using MSE or cosine
similarity. By our proposed $\mathcal{L}_{\text{BFC}}$ with JSD, we achieve a
considerable performance improvement. Interestingly, the performance rather
degrades when we apply $\mathcal{L}_{\text{BFC}}$ with JSD to
$\mathbf{b}_{\hat{\mathbf{d}}}$, _i.e_. the bottleneck feature conditioned
with input viewing direction $\hat{\mathbf{d}}$. We conjecture that the
reduced feature dimension of $\mathbf{b}_{\hat{\mathbf{d}}}$ reduces the
capacity of feature representations and prevents the model from improving
robustness.

###  4.3 Ablation Study

The quantitative results of our ablation study are reported in Tab. 4. With
only additionally exploiting our proposed flipped reflection rays, our
FlipNeRF achieves more degenerate results compared to the baseline ((1)
$\rightarrow$ (2)). However, we are able to achieve performance improvement by
a large margin with our proposed $\mathcal{L}_{\text{UE}}$ and
$\mathcal{L}_{\text{BFC}}$ ((2) $\rightarrow$ (3) $\rightarrow$ (4)). (5)
$\rightarrow$ (7) shows that enhancing the consistency of bottleneck features
between the pair of original and flipped reflection rays is considerably
effective. Additionally, by leveraging $\mathcal{L}_{\text{Ori.}}$ from Ref-
NeRF, we are able to estimate more accurate normal vectors, leading to more
effective flipped reflection rays and performance gain ((4) $\rightarrow$
(7)).

| PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Average Err. $\downarrow$ | MAE∘ $\downarrow$
---|---|---|---|---|---
4-view | 8-view | 4-view | 8-view | 4-view | 8-view | 4-view | 8-view | 4-view | 8-view
mip-NeRF [1] | 14.12 | 18.74 | 0.722 | 0.828 | 0.382 | 0.238 | 0.221 | 0.121 | 96.05 | 101.21
Ref-NeRF [36] | 18.09 |  red!2524.00 | 0.764 |  red!250.879 | 0.269 | 0.106 | 0.150 |  red!250.058 | 65.62 |  red!2557.93
DietNeRF [11] | 15.42 | 21.31 | 0.730 | 0.847 | 0.314 | 0.153 | 0.201 | 0.086 | - | -
InfoNeRF [15] | 18.44 | 22.01 | 0.792 | 0.852 | 0.223 | 0.133 | 0.119 | 0.073 | - | -
RegNeRF [25] | 13.71 | 19.11 | 0.786 | 0.841 | 0.346 | 0.200 | 0.210 | 0.122 |  red!2562.78 | 60.37
MixNeRF [31] |  red!2518.99 | 23.84 |  red!250.807 | 0.878 |  red!250.199 |  red!250.103 |  red!250.113 | 0.060 | 70.90 | 62.04
FlipNeRF (Ours) |  red!5020.60 |  red!5024.38 |  red!500.822 |  red!500.883 |  red!500.159 |  red!500.095 |  red!500.091 |  red!500.055 |  red!5058.72 |  red!5057.17

Table 5:  Quantitative results on Realistic Synthetic 360∘. Our FlipNeRF
achieves the SOTA performance among other baselines across all the scenarios
and metrics.

| PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Average Error $\downarrow$
---|---|---|---|---
| 3-view | 6-view | 9-view | 3-view | 6-view | 9-view | 3-view | 6-view | 9-view | 3-view | 6-view | 9-view
mip-NeRF [1] | 8.68 | 16.54 | 23.58 | 0.571 | 0.741 | 0.879 | 0.353 | 0.198 | 0.092 | 0.323 | 0.148 | 0.056
Pre-training.
PixelNeRF [43] | 16.82 | 19.11 | 20.40 | 0.695 | 0.745 | 0.768 | 0.270 | 0.232 | 0.220 | 0.147 | 0.115 | 0.100
PixelNeRF† [43] |  red!2518.95 | 20.56 | 21.83 | 0.710 | 0.753 | 0.781 | 0.269 | 0.223 | 0.203 | 0.125 | 0.104 | 0.090
SRF [6] | 15.32 | 17.54 | 18.35 | 0.671 | 0.730 | 0.752 | 0.304 | 0.250 | 0.232 | 0.171 | 0.132 | 0.120
SRF† [6] | 15.68 | 18.87 | 20.75 | 0.698 | 0.757 | 0.785 | 0.281 | 0.225 | 0.205 | 0.162 | 0.114 | 0.093
MVSNeRF [5] | 18.63 | 20.70 | 22.40 |  red!500.769 | 0.823 | 0.853 | 0.197 | 0.156 | 0.135 | 0.113 | 0.088 | 0.068
MVSNeRF† [5] | 18.54 | 20.49 | 22.22 |  red!500.769 | 0.822 | 0.853 | 0.197 | 0.155 | 0.135 | 0.113 | 0.089 | 0.069
Regularization.
DietNeRF [11] | 11.85 | 20.63 | 23.83 | 0.633 | 0.778 | 0.823 | 0.314 | 0.201 | 0.173 | 0.243 | 0.101 | 0.068
RegNeRF [25] | 18.89 | 22.20 | 24.93 | 0.745 |  red!500.841 |  red!500.884 |  red!250.190 | 0.117 | 0.089 |  red!250.112 | 0.071 | 0.047
MixNeRF [31] |  red!2518.95 |  red!2522.30 |  red!2525.03 | 0.744 | 0.835 | 0.879 | 0.203 |  red!250.102 |  red!250.065 | 0.113 |  red!250.066 |  red!250.042
FlipNeRF (Ours) |  red!5019.55 |  red!5022.45 |  red!5025.12 |  red!250.767 |  red!250.839 |  red!250.882 |  red!500.180 |  red!500.098 |  red!500.062 |  red!500.101 |  red!500.064 |  red!500.041

Table 6:  Comparison with baselines on DTU. Our FlipNeRF outperforms all the
pre-training and regularization methods in every scenario, especially by a
large margin under the 3-view setting. $\dagger$ indicates fine-tuning.

| PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Average Err. $\downarrow$
---|---|---|---|---
mip-NeRF [1] | 14.62 | 0.351 | 0.495 | 0.246
Pre-training.
PixelNeRF [43] | 7.93 | 0.272 | 0.682 | 0.461
PixelNeRF† [43] | 16.17 | 0.438 | 0.512 | 0.217
SRF [6] | 12.34 | 0.250 | 0.591 | 0.313
SRF† [6] | 17.07 | 0.436 | 0.529 | 0.203
MVSNeRF [5] | 17.25 | 0.557 | 0.356 | 0.171
MVSNeRF† [5] | 17.88 | 0.584 | 0.327 | 0.157
Regularization.
DietNeRF [11] | 14.94 | 0.370 | 0.496 | 0.240
RegNeRF [25] | 19.08 | 0.587 | 0.336 | 0.146
MixNeRF [31] |  red!2519.27 |  red!250.629 |  red!250.236 |  red!250.124
FlipNeRF (Ours) |  red!5019.34 |  red!500.631 |  red!500.235 |  red!500.123

Table 7:  Comparison with baselines on LLFF 3-view. $\dagger$ indicates fine-
tuning.

###  4.4 Comparison with other SOTA Methods

#### Realistic Synthetic 360∘.

As demonstrated in Tab. 5, our FlipNeRF achieves the SOTA performance across
all the evaluation metrics. Compared to MixNeRF which leverages a mixture
model framework as ours, our FlipNeRF improves the performance by a large
margin. Noticeably, our FlipNeRF estimates more accurate surface normals than
other baselines, leading to the performance gain with better reconstructed
fine details from limited input views as shown in Fig. 6(a). Additionally, the
vanilla Ref-NeRF, which shows great performance with accurate normal vectors,
achieves comparable or even better performance than other regularization
methods except ours. From this result, we are able to expect that estimating
the accurate surface normals is one of the key factors for learning 3D
geometry with sparse inputs. Note that the comparable MAE∘ of RegNeRF results
from the overly-smoothed depth estimation, not indicating the high-quality of
rendering results, as shown in Fig. 6(a).

|  mip-NeRF [1] |  Ref-NeRF [36] |  RegNeRF [25] |  MixNeRF [31] |  FlipNeRF (Ours) |  Ground Truth |
---|---|---|---|---|---|---|---

(a) Realistic Synthetic 360∘ 4-view

|  mip-NeRF [1] |  PixelNeRF [43] |  RegNeRF [25] |  MixNeRF [31] |  FlipNeRF (Ours) |  Ground Truth |
---|---|---|---|---|---|---|---

(b) DTU 3-view

Figure 6: Qualitative results on Realistic Synthetic 360∘ and DTU. More
results are provided in the supp. material.

#### DTU.

Our FlipNeRF achieves the best results across all the scenarios and most of
the evaluation metrics on DTU as shown in Tab. 6. Remarkably, ours trained
with 6-view outperforms all the pre-training methods trained with 9-view in
every metric except SSIM. Furthermore, ours trained with 3-view still
outperforms PixelNeRF [43] and SRF [6] trained with 6-view. Similar to the
results on Realistic Synthetic 360∘, ours outperforms other baselines by a
large margin especially under the 3-view, which is the most challenging
scenario, with reducing the floating artifacts successfully as shown in Fig.
6(b). Since the flipped reflection rays are effective training resources for
unseen views, the fewer the training views are provided, the more performance
gain is expected.

#### LLFF.

Table 7 compares our FlipNeRF against other baselines on LLFF, which is a real
forward-facing dataset. Although ours achieves the SOTA performance among
other baselines, it is much more marginal than those on Realistic Synthetic
360∘ and DTU. We conjecture the reason for the marginal improvement of our
proposed method can be the fact that a set of flipped reflection rays, which
are able to widely cover the unseen views, are not very useful for the scenes
in LLFF, where a set of camera poses are much less dynamic than other
datasets. In other words, our FlipNeRF is able to not only achieve a
competitive performance for the scenes consisting of a set of simple camera
poses, but also render the novel views in much higher quality for more
dynamically captured scenes with only a few shots. The rendered images are
provided in the supp. material.

##  5 Conclusion

In this work, we have focused on accurate surface normals, which is another
key factor for the few-shot novel view synthesis. Our proposed FlipNeRF
utilizes a set of flipped reflection rays as additional training resources,
which are simply derived from the estimated normal vectors and the input ray
directions. Since it does not require any heuristic factor for unseen view
generation, we are able to exploit these additional training resources with
much less burden. Furthermore, with our proposed UE Loss, FlipNeRF reduces the
floating artifacts consistently across the different scene structures while
considering the output uncertainty, leading to more reliable outputs. Also,
our proposed BFC Loss enhances the bottleneck feature consistency between the
rays cast on the photo-consistent pixels without leveraging the off-the-shelf
feature extractor, leading to performance improvement under the few-shot
setting. Our FlipNeRF achieves the SOTA performance with limited input views
among the other few-shot baselines and vanilla NeRF-like models. We expect
that our work is able to open another meaningful direction for the research of
few-shot novel view synthesis.

#### Limitations and future work.

Our FlipNeRF exploits the flipped reflection rays as a set of additional
training rays, leading to more accurate surface normal estimation. Although
Ref-NeRF achieved promising results with the high-quality surface
reconstruction, we use mip-NeRF representation instead of Ref-NeRF for a fair
comparison with other methods which are based on mip-NeRF. However, as shown
in Tab. 6 and Fig. 6(a), Ref-NeRF shows promising results without any
additional consideration for few-shot setting compared to the vanilla mip-
NeRF. Like our FlipNeRF, the accurate surface normal estimation leads to the
competitive performance of Ref-NeRF even in the few-shot scenarios. Therefore,
a combination of our FlipNeRF training framework with the Ref-NeRF
representation or further exploration with regard to the view-dependent
appearance for the few-shot novel view synthesis can be interesting directions
for future research.

#### Acknowledgements.

This work was supported by NRF (2021R1A2C3006659) and IITP (2021-0-01343) both
funded by Korean Government. It was also supported by Samsung Electronics
(IO201223-08260-01).

## References

  * [1] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855–5864, 2021.
  * [2] Christopher M Bishop.  Mixture density networks.  1994\.
  * [3] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch.  Nerd: Neural reflectance decomposition from image collections.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12684–12694, 2021.
  * [4] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, et al.  Jax: composable transformations of python+ numpy programs.  Version 0.2, 5:14–24, 2018.
  * [5] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14124–14133, 2021.
  * [6] Julian Chibane, Aayush Bansal, Verica Lazova, and Gerard Pons-Moll.  Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7911–7920, 2021.
  * [7] François Darmon, Bénédicte Bascle, Jean-Clément Devaux, Pascal Monasse, and Mathieu Aubry.  Improving neural implicit surfaces geometry with patch warping.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6260–6269, 2022.
  * [8] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan.  Depth-supervised nerf: Fewer views and faster training for free.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12882–12891, 2022.
  * [9] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5501–5510, 2022.
  * [10] Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin.  Fastnerf: High-fidelity neural rendering at 200fps.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14346–14355, 2021.
  * [11] Ajay Jain, Matthew Tancik, and Pieter Abbeel.  Putting nerf on a diet: Semantically consistent few-shot view synthesis.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5885–5894, 2021.
  * [12] Wonbong Jang and Lourdes Agapito.  Codenerf: Disentangled neural radiance fields for object categories.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12949–12958, 2021.
  * [13] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs.  Large scale multi-view stereopsis evaluation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 406–413, 2014.
  * [14] Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret.  Geonerf: Generalizing nerf with geometry priors.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18365–18375, 2022.
  * [15] Mijeong Kim, Seonguk Seo, and Bohyung Han.  Infonerf: Ray entropy minimization for few-shot neural volume rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12912–12921, 2022.
  * [16] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [17] Minseop Kwak, Jiuhn Song, and Seungryong Kim.  Geconerf: Few-shot neural radiance fields via geometric consistency.  arXiv preprint arXiv:2301.10941, 2023.
  * [18] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  Advances in neural information processing systems, 30, 2017.
  * [19] Jiaxin Li, Zijian Feng, Qi She, Henghui Ding, Changhu Wang, and Gim Hee Lee.  Mine: Towards continuous depth mpi with nerf for novel view synthesis.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12578–12588, 2021.
  * [20] Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang.  Neural rays for occlusion-aware image-based rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7824–7833, 2022.
  * [21] Antonio Loquercio, Mattia Segu, and Davide Scaramuzza.  A general framework for uncertainty estimation in deep learning.  IEEE Robotics and Automation Letters, 5(2):3153–3160, 2020.
  * [22] Nelson Max.  Optical models for direct volume rendering.  IEEE Transactions on Visualization and Computer Graphics, 1(2):99–108, 1995.
  * [23] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  ACM Transactions on Graphics (TOG), 38(4):1–14, 2019.
  * [24] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  Communications of the ACM, 65(1):99–106, 2021.
  * [25] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan.  Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5480–5490, 2022.
  * [26] Michael Oechsle, Songyou Peng, and Andreas Geiger.  Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5589–5599, 2021.
  * [27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [28] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger.  Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14335–14345, 2021.
  * [29] Konstantinos Rematas, Ricardo Martin-Brualla, and Vittorio Ferrari.  Sharf: Shape-conditioned radiance fields from a single view.  In International Conference on Machine Learning, pages 8948–8958. PMLR, 2021.
  * [30] Barbara Roessle, Jonathan T Barron, Ben Mildenhall, Pratul P Srinivasan, and Matthias Nießner.  Dense depth priors for neural radiance fields from sparse input views.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12892–12901, 2022.
  * [31] Seunghyeon Seo, Donghoon Han, Yeonjin Chang, and Nojun Kwak.  Mixnerf: Modeling a ray with mixture density for novel view synthesis from sparse inputs.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20659–20668, 2023.
  * [32] Jianxiong Shen, Antonio Agudo, Francesc Moreno-Noguer, and Adria Ruiz.  Conditional-flow nerf: Accurate 3d modelling with reliable uncertainty quantification.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part III, pages 540–557. Springer, 2022.
  * [33] Jianxiong Shen, Adria Ruiz, Antonio Agudo, and Francesc Moreno-Noguer.  Stochastic neural radiance fields: Quantifying uncertainty in implicit 3d representations.  In 2021 International Conference on 3D Vision (3DV), pages 972–981. IEEE, 2021.
  * [34] Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron.  Nerv: Neural reflectance and visibility fields for relighting and view synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7495–7504, 2021.
  * [35] Alex Trevithick and Bo Yang.  Grf: Learning a general radiance field for 3d representation and rendering.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15182–15192, 2021.
  * [36] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan.  Ref-nerf: Structured view-dependent appearance for neural radiance fields.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5481–5490. IEEE, 2022.
  * [37] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich.  Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation.  arXiv preprint arXiv:2212.00774, 2022.
  * [38] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  arXiv preprint arXiv:2106.10689, 2021.
  * [39] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser.  Ibrnet: Learning multi-view image-based rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690–4699, 2021.
  * [40] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [41] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman.  Volume rendering of neural implicit surfaces.  Advances in Neural Information Processing Systems, 34:4805–4815, 2021.
  * [42] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.  Plenoctrees for real-time rendering of neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5752–5761, 2021.
  * [43] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578–4587, 2021.
  * [44] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely.  Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5453–5462, 2021.
  * [45] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.

| Real. Syn. 360∘ [24] | DTU [13] | LLFF [23]
---|---|---|---
| 4-view | 8-view | 3-view | 6-view | 9-view | 3-view
Learning Rate | $[1\mathrm{e}{-3},1\mathrm{e}{-5}]$ | $[2\mathrm{e}{-3},2\mathrm{e}{-5}]$
Warm-up Step | 512 | 1024 | 512 | 1024 | 512
Delay Multiplier | $1\mathrm{e}{-2}$
$\lambda_{1}(\text{for }\mathcal{L}_{\text{NLL}})$ | $[4.0,1\mathrm{e}{-3}]$
$\lambda_{2}(\text{for }\mathcal{L}_{\text{NLL}}^{\prime})$ | $[4\mathrm{e}{-1},1\mathrm{e}{-4}]$ | $[4\mathrm{e}{-2},1\mathrm{e}{-5}]$ | $[4\mathrm{e}{-1},1\mathrm{e}{-4}]$ | $[4\mathrm{e}{-2},1\mathrm{e}{-5}]$ | $[4\mathrm{e}{-3},1\mathrm{e}{-6}]$ | $[4\mathrm{e}{-3},1\mathrm{e}{-6}]$
$\lambda_{3}(\text{for }\mathcal{L}_{\text{UE}})$ | $[1\mathrm{e}{-4},1\mathrm{e}{-1}]$ | $[1\mathrm{e}{-5},1\mathrm{e}{-2}]$ | $[1\mathrm{e}{-4},1\mathrm{e}{-1}]$ | $[1\mathrm{e}{-5},1\mathrm{e}{-2}]$ | $[1\mathrm{e}{-6},1\mathrm{e}{-3}]$ | $[1\mathrm{e}{-6},1\mathrm{e}{-3}]$
$\lambda_{4}(\text{for }\mathcal{L}_{\text{UE}}^{\prime})$ | $1\mathrm{e}{-2}$ | $1\mathrm{e}{-3}$ | $1\mathrm{e}{-3}$ | $1\mathrm{e}{-4}$ | $1\mathrm{e}{-5}$ | $1\mathrm{e}{-5}$
$\lambda_{5}(\text{for }\mathcal{L}_{\text{BFC}})$ | $1\mathrm{e}{-1}$ | $1\mathrm{e}{-2}$ | $1\mathrm{e}{-1}$ | $1\mathrm{e}{-2}$ | $1\mathrm{e}{-3}$ | $1\mathrm{e}{-3}$
$\lambda_{6}(\text{for }\mathcal{L}_{\text{Ori.}})$ | $1\mathrm{e}{-1}$ | $1\mathrm{e}{-2}$ | $1\mathrm{e}{-1}$ | $1\mathrm{e}{-2}$ | $1\mathrm{e}{-3}$ | $1\mathrm{e}{-3}$

Table A:  Details of hyperparameters and loss balancing terms. For each
dataset, the more training views are provided, the smaller $\lambda$’s we set
to prevent over-regularization, except $\lambda_{1}$ for
$\mathcal{L}_{\text{NLL}}$. $[a,b]$ indicates the annealing from $a$ to $b$.

##  A Implementation Details

The detailed hyperparameters and our loss balancing terms by the datasets and
the number of training views are provided in Tab. A.

w/ explicit normalization |  w/o explicit normalization
---|---

Figure A: Comparison between FlipNeRF using $\hat{\mathbf{n}}$ with and
without an explicit normalization. With explicitly normalized
$\hat{\mathbf{n}}$ (left), our FlipNeRF suffers from the training instability
and achieves degenerate results. We are able to achieve much superior
rendering quality with our proposed surface normals (right), which are trained
to approximate unit vectors, thanks to our regularization techniques and
masking strategy.

##  B Explicit Normalization for the Estimated Surface Normals

As mentioned in Sec. 3.2, we use the weighted sum of blending weights and
estimated normal vectors along a ray, _i.e_.
$\hat{\mathbf{n}}=\sum_{i=1}^{M}w_{i}\mathbf{n}_{i}$, as the surface normals
$\hat{\mathbf{n}}$ to derive a flipped reflection direction
$\mathbf{d}^{\prime}$. In this formulation, $\hat{\mathbf{n}}$ is not
guaranteed to be a unit vector without an explicit normalization process.
However, we empirically found that the normalization rather destabilizes the
training and leads to the performance degradation as shown in Fig. A. We
conjecture that the inaccurately generated $\mathbf{r}^{\prime}$ with an
explicit normalization provides wrong supervisory signals, especially during
the initial training stage when the surface normals are not accurately
estimated. Furthermore, as illustrated in Fig. B, $\hat{\mathbf{n}}$ is
trained naturally to approximate the unit vector through the training without
an explicit normalization process. With our proposed loss terms and
regularization techniques for accurate normal estimation, we are able to use
$\hat{\mathbf{n}}$ as the surface normal vector without any additional
normalization process.

Figure B: Distribution of $\lVert\hat{\mathbf{n}}\rVert_{2}$ without an
explicit normalization process. At an initial training stage, the estimated
surface normals $\hat{\mathbf{n}}$ are not unit vectors as most of
$\lVert\hat{\mathbf{n}}\rVert_{2}$ are far from 1. However, our proposed
$\hat{\mathbf{n}}$ is trained to be a unit vector naturally through the
training without an explicit normalization process, and most of
$\lVert\hat{\mathbf{n}}\rVert_{2}$ are concentrated close to 1 after the
training, which indicates that $\hat{\mathbf{n}}$ is successfully approximated
to a unit vector.

##  C Additional Qualitative Results

The additional qualitative comparisons are provided in Fig. C, Fig. D, and
Fig. E. Furthermore, we provide more qualitative results of our FlipNeRF in
Fig. F, Fig. G, and Fig. H. Our FlipNeRF shows superior rendering quality
compared to other baselines.

|  mip-NeRF [1] |  Ref-NeRF [36] |  RegNeRF [25] |  MixNeRF [31] |  FlipNeRF (Ours) |  Ground Truth |
---|---|---|---|---|---|---|---

(a) 4-view

(b) 8-view

Figure C: Additional qualitative comparisons on Realistic Synthetic 360∘.

|  mip-NeRF [1] |  PixelNeRF [43] |  RegNeRF [25] |  MixNeRF [31] |  FlipNeRF (Ours) |  Ground Truth |
---|---|---|---|---|---|---|---

(a) 3-view

(b) 6-view

(c) 9-view

Figure D: Additional qualitative comparisons on DTU. |  mip-NeRF [1] |  DietNeRF [11] |  RegNeRF [25] |  MixNeRF [31] |  FlipNeRF (Ours) |  Ground Truth |
---|---|---|---|---|---|---|---

Figure E: Qualitative comparisons on LLFF 3-view.

(a) 4-view

(b) 8-view

Figure F: Additional qualitative results of our FlipNeRF on Realistic
Synthetic 360∘.

(a) 3-view

(b) 6-view

(c) 9-view

Figure G: Additional qualitative results of our FlipNeRF on DTU.

Figure H: Additional qualitative results of our FlipNeRF on LLFF 3-view.


EPJ Web of Conferences Lattice2017  11institutetext: Institut fÃr Theoretische
Physik, WestfÃlische Wilhelms-UniversitÃt, MÃnster, Germany 22institutetext:
Institut fÃr Theoretische Physik, Justus-Liebig-UniversitÃt, GieÃen, Germany

# Spectrum of QCD at Finite Isospin Density

Philipp Scior Speaker, 11<EMAIL_ADDRESS>Lorenz von Smekal 22 Dominik
Smith 22

###### Abstract

We study the phase diagram of QCD at finite isospin density using two flavors
of staggered quarks. We investigate the low temperature region of the phase
diagram where we find a pion condensation phase at high chemical potential. We
started a basic analysis of the spectrum at finite isospin density. In
particular, we measured pion, rho and nucleon masses inside and outside of the
pion condensation phase. In agreement with previous studies in two-color QCD
at finite baryon density we find that the Polyakov loop does not depend on the
density in the staggered formulation.

##  1 Introduction

Knowledge about the phase diagram of QCD is of great importance for studying
various phenomena in nature. Unfortunately, lattice QCD is unable to provide
us with results for the bigger part of the phase diagram as lattice QCD
suffers from the fermion sign problem at finite baryon chemical potential
Dzhunushaliev2016 . Yet, there remains another interesting region of the phase
diagram that can be explored with standard lattice techniques, as there is no
sign problem in this region: the part of the phase diagram with finite isospin
density. By studying the finite isospin density region of the QCD phase
diagram we can learn about general features of relativistic field theories at
finite density that also occur in QCD with finite baryon density, e.g. the
Silverblaze property Cohen:2003kd . We may also use lattice studies in this
region to learn about lattice cutoff effects at finite density, like
saturation effects, and how to disentangle those effects from the relevant
continuum physics.
On the other hand, the finite isospin density region of the QCD phase diagram
is interesting by itself, as there are many systems in nature where we find an
imbalance of neutrons and protons, leading to a finite isospin density. Such
an imbalance occurs in nuclei, where there are more neutrons than protons,
neutron stars, heavy ion collisions and possibly the early universe
Schwarz2009 . Another intriguing feature that might not be of relevance for
the aforementioned systems but has close analogies in ultracold fermionic
quantum gases Boettcher2015  is the condensation of charged pions in the
ground state of finite isospin density QCD. It is known from earlier studies
PhysRevLett.86.592  that at low temperatures and high isospin chemical
potential we find a second order phase transition to a phase with a Bose-
Einstein condensate of charged pions.
In this contribution we concentrate particularly on the spectrum of QCD and
how the masses of bound states change with varying isospin chemical potential.
We investigate the masses of light mesons and the nucleons outside and inside
the pion condensation phase at low temperature.

##  2 Setup

We use the tree-level Symanzik improved gauge action and staggered quarks. Our
fermion matrix is

$M=\begin{pmatrix}D(\mu)&\lambda\eta_{5}\\\
-\lambda\eta_{5}&D(-\mu)\end{pmatrix}\,,$ (1)

where $D$ is the staggered Dirac operator coupled to the up- and respectively
the down-quark chemical potential, $\lambda$ is the the real coefficient of a
pion source term that we introduce to be able to compute the charged pion
condensate. Further, the pion source term also acts as a infrared cutoff -
similar to a twisted mass term - that enables us to invert the Dirac operator
in the pion condensation phase. Though its introduction is necessary for
computational reasons we have to extrapolate $\lambda\to 0$ to extract
physical results Brandt2016 . Finally we employ rooting to have $N_{f}=2$
degenerate quark flavors.

##  3 Phase Diagram

Figure 1: Bulk thermodynamic observables for the parameter set $N_{s}=16$,
$N_{t}=24$, $\beta=4.0$, $m=0.05$, $\lambda=0.005$ and several values of
$\mu_{I}$. The solid lines are predictions by chiral perturbation theory.

A good starting point to evaluate, if the algorithms work is the calculation
of bulk thermodynamic observables in the $(T,\mu_{I})$-plane of the phase
diagram. Previous studies that explored the phase diagram of QCD with finite
isospin density can be found in PhysRevLett.86.592 ; Detmold2012 ; Kogut2002-1
; Kamikado:2012bt ; Carignano2017 ; Brandt2016 . To test our setup we have
calculated the Polyakov loop $\langle|L|\rangle$, chiral condensate
$\langle\bar{\psi}\psi\rangle$, charged pion condensate $\langle\pi\rangle$
and the net isospin density $n$ for a fixed temperature with the parameter set
$N_{s}=16$, $N_{t}=24$, $\beta=4.0$, $m=0.05$, $\lambda=0.005$ for several
values of isospin chemical potential. Figure 1 shows the results for the
previously mentioned observables together with predictions from chiral
perturbation theory Kogut2000477 . We find that all thermodynamic observables
stay constant at their vacuum values until at some value of $\mu_{I}$ the pion
condensate and the net isospin density start to rise and the chiral condensate
starts to drop. This indicates that we entered the phase where the ground
state is a Bose-Einstein condensate of charged pions. The agreement of our
lattice data with the lines from chiral perturbation theory is good up to high
values of $\mu_{I}$ deep inside the pion condensation phase. Therefore, we
conclude that our algorithms work fine. Another point worth mentioning is that
the Polyakov loop stays at its vacuum value independent of $\mu_{I}$. This is
in contrast to earlier works on QCD-like theories at finite density with
Wilson fermions Cotter2012 . However it seems to be the generic case when
using staggered fermions Holicki2016 ; Braguta2016 . The difference between
both fermion discretizations at finite density should be explored by further
studies in the future.

##  4 Spectroscopy

| Operator | $J^{PC}$ | Particle
---|---|---|---
I | $\eta_{4}\bar{\chi}\chi$ | $0^{++}$ | -
|  | $0^{-+}$ | $\pi$
II | $\eta_{i}\eta_{5}\zeta_{i}\bar{\chi}\chi$ | $1^{++}$ | $a_{1}$
|  | $1^{--}$ | $\rho$
III | $\eta_{i}\eta_{5}\eta_{4}\zeta_{i}\zeta_{4}\bar{\chi}\chi$ | $1^{+-}$ | $b_{1}$
|  | $1^{--}$ | $\rho$
IV | $\epsilon_{abc}\chi_{a}\chi_{b}\chi_{c}$ | $\frac{1}{2}^{-}$ | N(1535)
|  | $\frac{1}{2}^{+}$ | N
Table 1: Operators, quantum number and particle identification for local
irreducible particle representations. Space-time arguments are omitted.

As we are using staggered fermions, we have to be careful in choosing our
lattice operators corresponding to the physical states in the continuum limit
Altmeyer1993 . For our analysis of the spectrum at finite isospin density we
use the four operators state in table 1. In particular we are interested in
measuring the masses of the pions, the rho mesons and the nucleons. The pion
masses can be easily extracted by measuring time-like correlators of operator
I. Extracting the rho and nucleon masses, we have to be a little more careful,
as operators II-IV will always contain a mixture of continuum states with
different parities. Thus, we choose two lattice operators containing the rho
meson to get some sense of our systematic uncertainties.

###  4.1 Spectroscopy with finite pion sources

Introducing a finite pion source term
$\frac{\lambda}{2}(\chi^{T}\tau_{2}\chi+\bar{\chi}\tau_{2}\bar{\chi}^{T})$ in
the Lagrangian, where $\tau_{2}$ is the second Pauli matrix in isospin space,
changes the particle correlators significantly. Not only does it act as an
infrared regulator for the eigenvalues of the Dirac operator, the introduction
of the source term also leads to the appearance of non-diagonal parts of the
fermion matrix, see equation (1). This is of course also true for the fermion
propagator

$G=M^{-1}=\begin{pmatrix}(D^{\dagger}_{\mu}D_{\mu}+\lambda^{2})^{-1}D^{\dagger}_{\mu}&-\eta_{5}(D^{\dagger}_{\mu}D_{\mu}+\lambda^{2})^{-1}\lambda\\\
\eta_{5}(D^{\dagger}_{-\mu}D_{-\mu}+\lambda^{2})^{-1}\lambda&(D^{\dagger}_{-\mu}D_{-\mu}+\lambda^{2})^{-1}D^{\dagger}_{-\mu}\end{pmatrix}\,.$
(2)

Thus, an up quark can propagate into a down quark and vice versa and we find
disconnected pieces contributing to the pion and rho correlators, even though
they are iso-triplet states. In the correlator for the neutral mesons we
further find additional terms also in the connected contributions. The pion
correlators are given by

$\displaystyle\langle\pi^{+}(t)\bar{\pi}^{+}(0)\rangle=$
$\displaystyle-\eta_{4}(t)\text{Tr\,}_{c}[G_{11}(t,0,\mu)G_{11}(0,t,-\mu)]+\mathcal{O}(\lambda^{2})\,,$
$\displaystyle\langle\pi^{-}(t)\bar{\pi}^{-}(0)\rangle=$
$\displaystyle-\eta_{4}(t)\eta_{5}(t)\text{Tr\,}_{c}[G_{22}(t,0,\mu)G_{22}^{\dagger}(t,0,\mu)]+\mathcal{O}(\lambda^{2})\,,$
(3) $\displaystyle\langle\pi^{0}(t)\bar{\pi}^{0}(0)\rangle=$
$\displaystyle-\eta_{4}(t)\eta_{5}(t)\text{Tr\,}_{c}[G_{22}^{\dagger}(t,0,\mu)G_{11}(t,0,\mu)-G_{21}^{\dagger}(t,0,\mu)G_{21}(t,0,\mu)$
$\displaystyle-
G_{12}^{\dagger}(t,0,\mu)G_{12}(t,0,\mu)+G_{11}^{\dagger}(t,0,\mu)G_{22}(t,0,\mu)]+\mathcal{O}(\lambda^{2})\,,$

where we made use of the fact that $G_{22}(-\mu)=G_{11}(\mu)$ and the
disconnected pieces are of order $\mathcal{O}(\lambda^{2})$. The nucleon
correlators receive additional connected contributions but, as in the standard
case, there are no disconnected pieces. The proton propagator reads

$\displaystyle\langle p(t)p(0)\rangle=$
$\displaystyle\epsilon_{abc}\epsilon_{ijk}\left\\{G_{22}^{ci}(t,0,\mu)\left(G_{11}^{bj}(t,0,\mu)G_{11}^{ak}(t,0,\mu)-G_{11}^{bk}(t,0,\mu)G_{11}^{aj}(t,0,\mu)\right)\right.$
$\displaystyle-
G_{21}^{cj}(t,0,\mu)\left(G_{12}^{bi}(t,0,\mu)G_{11}^{ak}(t,0,\mu)-G_{12}^{ai}(t,0,\mu)G_{11}^{bk}(t,0,\mu)\right)$
(4)
$\displaystyle+\left.G_{21}^{ck}(t,0,\mu)\left(G_{12}^{bi}(t,0,\mu)G_{11}^{aj}(t,0,\mu)-G_{12}^{ai}(t,0,\mu)G_{11}^{bj}(t,0,\mu)\right)\right\\}\,.$

We performed simulations and measurements of the spectrum for two parameter
sets: $\beta=\leavevmode\nobreak\ 4.0,$ $m=\leavevmode\nobreak\ 0.05,$
$\lambda=\leavevmode\nobreak\ 0.005$ and $\beta=3.5,$ $m=0.01,$
$\lambda=0.001$ both on $16^{3}\times 24$ lattices. We choose the ration
$\frac{\lambda}{m}=0.1$ to keep the correlator contributions from the pion
source terms small, as we do not include disconnected pieces in our
measurements. We are not able to dismiss the fact that the disconnected pieces
could have significant influence on the mesons masses however we checked the
significance of the additional connected contributions. Also the additional
connected pieces are of order $\mathcal{O}(\lambda^{2})$ and their omission in
the measurements only lead to neglect able changes in the nucleon and
$\pi^{0}$ correlators. Therefore, we conclude that also the disconnected
pieces of the the correlators should only lead to minor corrections.

###  4.2 Spectroscopy at vanishing isospin chemical potential

So far we have used two different parameter sets in our simulations. In both
cases simulations were performed on a $16^{3}\times 24$ lattice. Lattice
parameters were

  * (1)

$\quad\beta=4.0,$ $m=0.05,$ $\lambda=0.005$

  * (2)

$\quad\beta=3.5,$ $m=0.01,$ $\lambda=0.001$.

We started our analysis by extracting the vacuum masses of the pion, the rho
meson and the nucleons using the operators stated in table 1.

| $(1)$ | (2)
---|---|---
$am_{\pi}$ | 0.66(1) | 0.2597(1)
$am_{\rho}$ | 1.04(3) | -
$am_{N}$ | 1.64(5) | -
Table 2: Vacuum masses of the pion, rho meson and the nucleons for the two
parameter sets used.

The results for the masses are given in table 2. The masses for the rho meson
from channel II and III were compatible within errors. Unfortunately,for
parameter set 1 the ration of $m_{\pi}/m_{\rho}=0.635$ is much larger than the
physical value of $m_{\pi}/m_{\rho}=0.18$, indicating our pion is very heavy.
Therefore, we chose to investigate a second parameter set. Even though we
managed to get a lower pion mass in lattice units for parameter set 2 the
signal to noise ratio for all other correlation functions were too bad to
extract $\rho$ or nucleon masses. Thus, we are not able to make any statements
about the behavior of the $\rho$ meson, the nucleons or the physical pion
mass.

###  4.3 Spectroscopy at finite isospin chemical potential

In the vacuum masses of particles and their anti-particles are equal.
Therefore, particle correlators on a finite lattice with (anti-)periodic
boundary conditions in the time direction always show some symmetry. Depending
on the quantum numbers of the particular particle the correlators are either
symmetric or anti-symmetric. When working with staggered fermions, it is a
little more complicated as staggered lattice operators mix continuum states
with different quantum numbers. This may lead to the fact that one finds
different symmetries for the correlator parts defined for lattice points with
even and odd time coordinates Altmeyer1993 . By introducing a chemical
potential for the quarks we prefer particle over anti-particles or in other
words particles propagating forward in time over particles propagating
backwards in time. This results in the loss of symmetry in the correlation
functions. Introduction of an isospin chemical potential has the same effect
as we prefer particles with positive isopin over particles with negative
isospin, i.e. up-quark over down-quark but also up-quark over anti-up-quark.
For example the correlation function of the $\pi^{+}$ particle in the vacuum
is given by

$C(t)=(-1)^{t}A\left(e^{-m_{\pi}t}+e^{-m_{\pi}(N_{t}-t)}\right)=(-1)^{t}2A\cosh\left(m_{\pi}\left(t-\frac{N_{t}}{2}\right)\right)\,,$
(5)

where the masses of the forward and backward propagating masses are the same:
the vacuum pion mass. By introducing the isospin chemical potential we prefer
the $\pi^{+}$ over the $\pi^{-}$ and the correlation function now reads

$C(t,\mu_{I})=(-1)^{t}A\left(e^{-m_{\pi^{+}}(\mu_{I})t}+e^{-m_{\pi}^{-}(\mu_{I})(N_{t}-t)}\right)\,,$
(6)

(a) $a\mu_{I}=0$ (b) $a\mu_{I}=0.1$

Figure 2: $\pi^{+}$ correlation functions for $\beta=4.0,$ $m=0.05$,
$\lambda=0.005$. Left: correlator at $a\mu_{I}=0$ and a fit to eq. (5). The
correlator is symmetric around $N_{t}/2=12$. Right: correlator at
$a\mu_{I}=0.1$ and a fit to eq. (6). The symmetry around $N_{t}/2$ is lost.

with a splitting between the masses for $\pi^{+}$ and $\pi^{-}$ depending on
$\mu_{I}$. The correlator for the $\pi^{0}$ will always be symmetric as the
$\pi^{0}$ has isospin zero. However, depending on $\mu_{I}$ its mass my also
differ from the vacuum pion mass. Figure 2 shows the $\pi^{+}$ correlation
function for vanishing isospin chemical potential (a) and for an isospin
chemical potential of $a\mu_{I}=0.1$ (b). The mass splitting of the pions at
finite $\mu_{I}$ is clearly visible in the asymmetry of the correlation
function in (b).
Predictions for the pion mass splitting can be computed with chiral
perturbation theory Kogut2000477 . The prediction for the dispersion relation
of the pions away from the phase with a pion BEC is

$\displaystyle\pi^{+}:$ $\displaystyle\quad
E=\sqrt{\vec{p}^{2}+m_{\pi}^{2}}-\mu_{I}\,,$ $\displaystyle\pi^{-}:$
$\displaystyle\quad E=\sqrt{\vec{p}^{2}+m_{\pi}^{2}}+\mu_{I}\,,$ (7)
$\displaystyle\pi^{0}:$ $\displaystyle\quad
E=\sqrt{\vec{p}^{2}+m_{\pi}^{2}}\,.$

The dispersion relation explicitly show the splitting of the different pion
modes according to their isospin charge. When the isospin chemical potential
is $\mu_{I}^{c}=m_{\pi}$ we can excite $\pi^{+}$ particles with $\vec{p}=0$
essentially for free. This is exactly the critical point where the
condensation of $\pi^{+}$ to a BEC sets in. In our simulations we will never
find exactly massless pions as the pion source term acts an infrared cutoff
also for the pion mass. We can nevertheless include the pion source term into
the chiral perturbation theory calculations to make predictions for the pion
spectrum at finite isospin density for finite values of $\lambda$. Again, this
is in agreement with the phase diagram of isospin QCD: the introduction of the
pion source changes the second order phase transition to the pion BEC phase to
an analytic crossover.
Figure 3 show the pion masses and mass splittings in dependence of the isospin
chemical potential for both parameter sets used in the simulations.

(a) parameter set 1: $\beta=4.0,$ $m=0.05,$ $\lambda=0.005$ (b) parameter set
2: $\beta=3.5,$ $m=0.01,$ $\lambda=0.001$

Figure 3: Pion masses in dependence of the isospin chemical potential for both
parameter sets. The solid lines are predictions from leading order chiral
perturbation theory taking into account the finite value of $\lambda$.

The solid lines are predictions from leading order chiral perturbation theory,
taking into account the finite values of $\lambda$. For the parameter set 1 we
find a rather good agreement of our results with the predictions from chiral
perturbation theory up to $a\mu_{I}\approx 0.4$. For higher isospin chemical
potential we find that the masses for the $\pi+$ and $\pi^{-}$ deviate
slightly from the predictions. Still, the curves for the charged pion masses
level of in the pion condensation phase as predicted. The $\pi^{0}$ mass
starts to deviate from its predicted value also around $a\mu_{I}\approx 0.4$.
Even though chiral perturbation theory predicts an increase of the pions mass,
proportional to $\mu_{I}$, the masses from our data and the slope of the
resulting curve are significantly too large. For parameter set 2 the situation
is different. We find an excellent agreement of the predicted and measured
values of the $\pi^{0}$ mass for all values of $\mu_{I}$, even deep inside the
pion condensation phase. The agreement for the charged pion masses is also
very good, however we only have very few data points for the charged pion
masses, as we unfortunately loose signal for the charged pion correlators
already for quite small values of the isospin chemical potential. We are not
sure, why the $\pi^{0}$ mass for parameter set 1 starts too deviate
significantly from the predictions. Comparison of the situation with parameter
set 2 could indicate that is due to the lager pion mass. However, it is still
unclear why the deviations are much lager for the $\pi^{0}$ than for the
charged pions.
We extracted masses from all correlation functions of operators in table 1. As
already mentioned above we were not able to extract any masses, except for the
pions, for parameter set 2. For parameter set 1 we were able to measure
correlation functions for all channels, giving use access to the masses of the
$\rho$ mesons and the nucleons. Figure 4 shows the measured meson and baryon
masses in dependence of the isospin chemical potential. The masses of the
$\rho$ meson seem to run parallel to the pion masses for all values of
$\mu_{I}$ investigated. This is expected outside the pion condensation phase,
as we expect the excitation energy of a particle to be

(a) comparison of $\rho$ and pion masses (b) baryon masses

Figure 4: Meson and baryon masses in dependence of the isospin chemical
potential for parameter set 1. The solid line is a prediction from leading
order baryon chiral perturbation theory.

$E=\sqrt{\vec{p}^{2}+m^{2}}+I\mu_{I}\,,$ (8)

where $I$ is the isospin charge of the particle. The almost constant value of
the $\rho^{+}$ inside the phase with non-vanishing $\langle\pi\rangle$ could
indicate that the possible formation of a $\rho$ condensate is delayed to
higher $\mu_{I}$. On the other hand, we could have simply reached a lower
boundary for $m_{\rho}(\mu_{I})$ that comes with introduction of the pion
source term. This has to be investigated in the future by performing a
$\lambda\to 0$ extrapolation. In case of the nucleons, we find a near perfect
degeneracy of neutron and anti-proton and respectively the anti-neutron and
the proton, which is expected as the only difference in our setup between the
neutron and the anti-proton is the baryon number of the particles and all our
calculations are at vanishing baryon chemical potential. Outside the BEC phase
we find a near linear behavior of the nucleon masses. Inside the pion
condensation phase, the mass of the neutron and anti-proton seem to be almost
constant while the mass of the proton and anti-neutron still seem to decrease
further. In contrast a prediction from baryon chiral perturbation theory
PhysRevLett.86.592  is that the proton mass should not decrease further but
increase again, as the protons are repelled by the pion condensate. We have
not enough data yet to make a definite statement about the fate of the proton
inside the pion condensate and clarifying this issue will be an interesting
task for further research.

##  5 Summary

We have presented our first results for QCD at finite isopsin density and in
particular for the meson and baryon spectrum at finite $\mu_{I}$. We found
that the introduction of a finite pion source term leads to additional terms
in the correlation functions for all particles containing not only one single
quark species. This makes spectroscopy more challenging, especially in the
case of mesons where the pion source term leads to the appearance of
disconnected contributions to the correlators of all mesons. So far we only
investigated the connected pieces of the correlators. Yet, we do believe that
masses extracted from the connected pieces are already a good approximation to
the full answer, as the disconnected pieces should only contribute very little
in the parameter range we have chosen. Thus, we were able to extract pion,
$\rho$ and nucleon masses over a large range of $\mu_{I}$ until deep inside
the pion condensation phase. Unfortunately, the parameter sets we used for our
investigations so far are not ideal. The pion in the first parameter set is
much heavier as in nature and the second parameter set shows a very bad
signal-to-noise ratio for most correlators. Therefore, we will need to find
better parameter sets for our future simulations. Further, we will need to
extrapolate our results to vanishing pion source to get physically relevant
masses.

## References

  * (1) C. Gattringer, K. Langfeld, International Journal of Modern Physics A 31, 1643007 (2016), 1603.09517
  * (2) T.D. Cohen, Phys. Rev. Lett. 91, 222001 (2003), hep-ph/0307089
  * (3) D.J. Schwarz, M. Stuke, Journal of Cosmology and Astroparticle Physics 2009, 025 (2009), 0906.3434
  * (4) I. Boettcher, T.K. Herbst, J.M. Pawlowski, N. Strodthoff, L. von Smekal, C. Wetterich, Physics Letters, Section B: Nuclear, Elementary Particle and High-Energy Physics 742, 86 (2015), 1409.5232v1
  * (5) D.T. Son, M.A. Stephanov, Phys. Rev. Lett. 86, 592 (2001)
  * (6) B.B. Brandt, G. Endrodi (2016), 1611.06758
  * (7) W. Detmold, K. Orginos, Z. Shi, Physical Review D 86, 054507 (2012), 1205.4224
  * (8) J.B. Kogut, D.K. Sinclair, Physical Review D 66, 034505 (2002), 0202028
  * (9) K. Kamikado, N. Strodthoff, L. von Smekal, J. Wambach, Phys. Lett. B718, 1044 (2013), 1207.0400
  * (10) S. Carignano, L. Lepori, A. Mammarella, M. Mannarelli, G. Pagliaroli, European Physical Journal A 53, 1 (2017), 1610.06097
  * (11) J.B. Kogut, M.A. Stephanov, D. Toublan, J.J.M. Verbaarschot, A. Zhitnitsky, Nucl. Phys. B 582, 477 (2000)
  * (12) S. Cotter, P. Giudice, S. Hands, J.I. Skullerud, Physical Review D 87, 034507 (2013), 1210.4496
  * (13) L. Holicki, J. Wilhelm, D. Smith, B. Wellegehausen, L. Von Smekal, Proceedings of Science pp. 0–6 (2016), 1701.04664
  * (14) V. Braguta, E.M. Ilgenfritz, A. Kotov, A. Molochkov, A. Nikolaev, Physical Review D 94, 114510 (2016), 1605.04090
  * (15) R. Altmeyer, K. Born, M. Göckeler, E. Laermann, G. Schierholz, Nuclear Physics B 389, 445 (1993)