images and their registration, this method demonstrates the potential of
image-based representations for 3D scene understanding and view synthesis
tasks without the need for constructing detailed 3D models. The “Light Field
Rendering” [122] introduced the concept of light field rendering, a technique
for synthesizing novel views of a scene using a dense set of input images.
These light fields represent the radiance of a scene as a function of position
and direction and can be thought of as a 4D function. The paper described how
to efficiently sample, reconstruct, and render the light field using a set of
2D images, allowing for the synthesis of new views without the need for
explicit 3D geometry. Further, research such as “Physically-Valid View
Synthesis by Image Interpolation” [189] presented a view synthesis technique
that uses image interpolation to create novel views of a scene. It was based
on the idea of “view morphing,” which combined two or more input images to
synthesize a new view using epipolar constraints. The paper discussed a method
that took into account the geometric relationships between the input images,
the camera positions, and the scene structure, ensuring that the synthesized
views are physically valid. This research on Morphing has later also been used
in different research constituting different surfaces and objects [68].

Proceedings such as “Image-based Modeling and Rendering” [45] have hence been
influential contributors for theshift of focus from model-free novel view
synthesis approaches to model-based approaches. The talk surveyed different
image-based rendering techniques, which included both model-based and image-
based methods for synthesizing novel views of a scene. The authors discussed
the advantages and limitations of both approaches, highlighting the need for
more accurate and efficient algorithms for handling 3D data in novel view
synthesis. The traction gained from such keynote talks gained traction and led
to the development in the field of model-based novel view synthesis. Further,
it was also driven by the following advantages which accumulated over time.

  1. 1.

Availability of 3D data: With advances in 3D scanning technologies and the
increasing availability of 3D models, researchers have been able to leverage
this data to create more accurate and realistic novel view synthesis
algorithms.

  2. 2.

Advances in computer graphics and computational power: The development of
sophisticated rendering techniques and the increase in computational power
have made it possible to work with complex 3D models and generate realistic
novel views in real-time.

  3. 3.

Improved algorithms for 3D reconstruction: Researchers have developed more
accurate and efficient algorithms for recovering the 3D structure of a scene
from images, making it possible to use model-based approaches for novel view
synthesis.

  4. 4.

Handling occlusions: Model-based approaches can better handle occlusions and
disocclusions in the scene, as they have explicit 3D information about the
scene geometry.

Later attempts in Neural Rendering usually involved lengthy pipeline
operations for creating virtual models of the real scenes. These operations
start with a scanning process (using camera images) to capture the photometric
properties, and raw scene geometry is captured using depth scanners or dense
stereo matching. The Raw Scene Geometry capture is often noisy and provides
incomplete point cloud data. This incomplete data can further be processed by
applying surface reconstruction and meshing approaches. Particularly, with the
context of 3D reconstruction with commodity RGB-D sensors, researchers have
made significant progress enabling robust tracking [93, 156, 35, 228, 43], and
large scale 3D scene capture [27, 161, 253]. Then, the material and texture
estimation operations are leveraged to determine the photometric properties
from the obtained mesh of surface fragments and store them in different data
structures like texture maps [16], bump maps [15], view-dependent textures
[46], and surface light fields [232]. Basically, textures are alternative that
mitigate the missing spatial resolution problem where finding the consistent
UV-mapping becomes vital as geometry and color are decoupled. To compensate
for wrong geometry and camera drift, some approaches are based on finding non-
rigid warps [268, 87].

Lastly, the computationally-heavy rendering operations (ray-tracing/radiance
transfer estimation) generate photo-realistic views of the modeled scene. An
advantage of non-rigid warps discussed earlier is that 3D reconstructed
content can be visualized by standard rendering techniques and approach
directly generalizes to 4D capture as long as it can be reliably tracked [155,
52, 90]. The reconstructions discussed earlier can be used to synthesize
images from novel viewpoints. This pipeline of operations has been constructed
and polished by research in computer graphics society for several years.
Despite that this pipeline generates highly realistic results under controlled
settings, several operational stages of this pipeline and consequently the
entire pipeline remains brittle until the addition of the domain of Machine
Learning. This augmentation has hence encouraged people from Machine Learning,
Computer Vision, and Computer Graphic society to come together and make this
pipeline more robust. But, the imperfections in reconstruction directly appear
when translating the embeddings to visual artifacts in renderings, that became
the major hurdle in 2020s to make content creation from real-world accessible.

With the growth of Neural Networks and the inception of CNN-based
architectures like AlexNet [115], VGG-Net [193], and ResNets [79], the focus
shifted for Machine Learning techniques to permeate the domain of Computer
Vision and Computer Graphics. This is evident by the following research
literature which leverages CNNs exhaustively. “View Synthesis by Appearance
Flow” [270] introduced an end-to-end trainable CNN model that learned to
synthesize target views by predicting appearance flows, which were then used
to warp input images, allowing for view synthesis without explicitly
estimating 3D geometry. “Deep View Morphing” [98] proposed a deep learning-
based approach to view morphing, leveraging CNNs to learn and predict dense
correspondence maps between source and target images, enabling the generation
of high-quality novel views. Habtegebrial et al., in “Fast View Synthesis with
Deep Stereo Vision” [76], developed a deep learning-based method for view
synthesis that combined stereo vision techniques with a CNN, aiming to produce
accurate novel views with reduced computational complexity compared to other
learning-based approaches. Lastly, “InLoc: Indoor Visual Localization with
Dense Matching and View Synthesis” [208] by Taira et al. presented a large-
scale indoor localization system that matched dense local features between a
query image and a database of 3D point clouds [229], utilizing view synthesis
to improve the matching performance and achieve accurate and robust
localization in challenging indoor environments. This was an application of
stereo vision models in indoor 3D scene reconstruction.

An alternative direction is to fill in the missing content by leveraging
coarse geometry proxy based on high-resolution 2D textures [87]. PBG (Point-
based Graphics) approaches for rendering and re-rendering pipelines use a
collection of points or unconnected disks (surfels) for geometry modeling
instead of Surface Mesh Estimation [123, 70, 69, 111]. Whereas, the IBR
(Image-Based Rendering) techniques [142, 190, 66, 122] warp the Camera
Coordinate System using coarse approximations (low-frequency embeddings in
case of functional representation) of scene geometry to obtain the
photorealistic views, where the 3D geometry proxy is only used to select
suitable views for cross-projection and view-dependant blending [83, 19, 21,
264, 81]. Finally, the deep neural rendering methods [92, 153, 24, 20, 80]
replace the physics-based rendering with generative Neural Network to rectify
some of the mistakes of rendering network. One such method that has gained
traction from Computer Vision society is what is the primary focus of this
survey–NeRF.

LLFF (Local Light Field Fusion) (example depicted in fig. 4(a)) is one prior
work done by Mildenhall et al [147], some of whose authors were also
contributors for the work under focus in this survey. LLFFs leverage MPI
(Multiplane Image) scene representations [269] for view synthesis from an
irregular grid of sampled views for expanding each sampled view into a local
light fields [122] and then renders novel views by fusing these fields. LLFFs
achieve perceptual quality similar to Nyquist Rate Sampling (with $4000\times$
less views) by creating a bound for real world scenes creation/rendering which
is specifies the density of sample views for a given scene by extending the
traditional plenoptic sampling [22] theory. Here, the LLFFs result in a
quadratic decrease in the number of input views based on the number of
predicted planes for each layered scene representation and still maintain the
Nyquist level performance by specializing to the subset of natural scenes.
Further, this work was corroborated with a smartphone app and web-cum-desktop
application for capturing and viewing respectively.

This architecture is based on the idea that prescriptive sampling is vital for
useful IBR-algorithms. Hence, it extends the prior plenoptic sampling theory
to decrease the dense sampling requirements for traditional light field
rendering by decomposing a scene into $D$ depth ranges and individual-range-
based light field samples. This allows the camera sampling interval to be
increased by a factor of $D$ and the light field spectrum emitted by a scene
within each depth range to lie within a densely packed shape that is $D$ times
tighter that the full scene’s double-wedge spectrum as depicted in fig. 3.
This synthesis pipeline also leaves a possibility for it to be used in the
future light field hardwares to reduce required cameras. As discussed by Chai
et al [22], the Light Field’s Fourier support (without any occlusions and
Lambertian effects [254]) lies inside double-wedge shape within scene depths
limit $[z_{min},z_{max}]$ as given by the figure 3, whereas occlusion expands
the limits of this shape in the light field as the occluder acts as convolver
due to the kernel (lying on the line corresponding to the occluder’s depths)
with farther scene content. Further, Fourier support of the light field gets
limited due to the closest occluder which convolves the furthest scene
content’s line resulting in a parallelogram-like-shape as depicted in fig 3.
This can only be half packed when compared to double-wedge shape and hence
required maximum camera sampling interval $\Delta_{u}$ for an occluded light
field and is given be eqn. 57

$\Delta_{u}\leq\frac{1}{2K_{x}f\left(\frac{1}{z_{min}}-\frac{1}{z_{max}}\right)},$
(57)

In the continuous light field $B_{x}$ and camera spatial resolution
$\Delta_{x}$, $K_{x}$ is maximum spatial frequency in sampled light field. It
is given by the eqn. 58

$K_{x}=min\left(B_{x},\frac{1}{2\Delta_{x}}\right)$ (58)

LLFFs render novel views from an MPI scene representation (that consist of a
set of evenly disparity sampled fronto-parallel RGB$\alpha$ planes from view
frustrum of reference camera) from local-neighborhood’s continuous valued
camera poses by alpha compositing the color rays into the novel views with the
“over” operation [172]. This is similar to the early work by Lacroute et al
[118] and can be considered an encoding of a local light field much like light
field displays [226, 227]. Further, it alpha composite the depth range fields
or shield fields [119] from back to front for computing the full scene light
field to decrease the Sampling Rate for occlusions. These fields plays a vital
role in handling occlusions as the Fourier Spectra Union have less degree of
confidence when compared to the original occluded light fields as depicted in
fig 3. Each layer-by-layer alpha compositing step increases the Fourier
support by convolving previously accumulated light-field’s spectrum with
occluded depth layer spectrum. The width of Fourier Spectrum Support
Parallelogram for the reconstructed light field for each occluded depth range
light field will enjoy full Fourier Support width and is given and illustrate
by eqn. 59 and fig 3, respectively.

Figure 3: Traditional Plenoptic Sampling without occlusions [22] in (a) where
the double-wedge (in blue) encloses the fourier support of a light field
without occlusions. Here, the double-wedge width is used to calculate Nyquist
Rate Sampling, that determines by minimum and maximum scene depths
$[z_{min},z_{max}]$ and $K_{x}$ (max spatial frequency). In (b), the light
field is split into $D$ non-overlapping layers and Nyquist Sampling Rate
decreases by a factor of $D$ and (c) represents that without occlussions, the
full light field spectrum is sum of the spectra from each layer. In LLFF [147]
extends this to mitigate occlusions as continuous light fields are
reconstructed from MPIs. In (d), occlusions expand the Fourier supports to
——gm (where blue represents support without occlusions and purple regions
depict the occlusion expanding the fourier support) doubling the Nyquist
Sampling Rate. Further, (e) depicts that separate reconstruction of the light
field for $D$ layers decrease the Nyquist Sampling Rate by a factor of $D$
whereas (f) depicts the case that full light spectrum cannot be reconstructed
by summation over individual layer spectra because their support union in
smaller than support union in (a). Hence, the need for alpha compositing
individual light layers from back to front in the primal domain to compute the
full light field. Figures used from LLFF original paper [147]

$\frac{2K_{x}f\left(\frac{1}{z_{min}}-\frac{1}{z_{max}}\right)}{D},$ (59)

Hence, LLFFs are able to extend the layered plenoptic sampling framework and
handle occlusions uisng depth maps and increasing the required sampling
interval by a factor of $D$ and is given by the eqn. 60.

$\Delta_{u}\leq\frac{D}{2K_{x}f\left(\frac{1}{z_{min}}-\frac{1}{z_{max}}\right)},$
(60)

In addition to this, LLFFs also introduced an additional caveat for the
scene’s bouding volume to be within a frustums of atleast two neighboring
sampled views by introducing the concept of finite field of view instead of
infinite field of view [22, 254]. This bound is given by the camera sampling
interval 61 and the constrain given in the eqn 62.

$\Delta_{u}\leq\frac{W\Delta_{x}z_{min}}{2f},$ (61)

$\Delta_{u}\leq
min\left(\frac{D}{2K_{x}f\left(\frac{1}{z_{min}}-\frac{1}{z_{max}}\right)},\frac{W\Delta_{x}z_{min}}{2f}\right)$
(62)

Now, if we set $z_{max}=\infty$ for the traditional assumption and set
$K_{x}=\frac{1}{2\Delta_{x}}$ for allowing maximum representable frequency,
then we have eqn 63.

$\frac{\Delta_{u}}{\Delta_{x}z_{min}}=d_{max}\leq
min\left(D,\frac{W}{2}\right)$ (63)

where $d_{max}$ is the maximum pixel density of any scene between adjacent
scenes. Hence, the maximum density i.e. $d_{max}$ must be less than
$min(D,W/2)$ and it reduces to the Nyquist bound when $D=1$. Hence, this
promotion to $D$ depth layers decreases the required sampling rate by a factor
of $D$ until the required field of view overlaps for estimation of stereo
geometry. Now, further as we know that any scene can be a function of two
viewing directions hence this sampling rate compounds to $D^{2}$ sampling
reduction.

LLFFs practically synthesize new views from set of input images and camera
poses by first using a CNN to promote each captured input imahe to an MPI and
then reconstructing novel views by blending the rendered light field from the
nearby MPIs as depicted in fig 4(a) and fig 4(b). Firstly, 5 images of a scene
are taken (1 reference image and 4 nearest neighbors in 3D space) and
reprojected to $D$ depth planes and sampled linearly within reference view
frustum to get 5 $H\times W\times 3$ PSVs (plane sweep volumes). LLFFs employ
a 3D CNN for taking 5 PSVs as inputs to concatenate them along channel
dimension which results in opacity $\alpha$ for each MPI pixel $(x,y,d)$ and 5
color selection weights sum to unity at each MPI. This parametrization is
leveraged instead of fg-bg parameterization [269] as it prevents an MPI from
incorporating content occluded from the reference view but visible in other
input views. 3D convolutions enables the architecture to predict MPIs with $D$
number of variable planes by involving convolutions across the height, width,
and depth instead of fully 2D convolutions. This enables to jointly choose
disparity and view sampling densities to satisfy the eqn. 63

(a) A step-by-step representation of Multi-Plane Image Compositing in Local
Light Field Fusion by Mildenhall et al [147]

(b) LLFF (Local Light Field Fusion) [147]: Promotion to MPI Scene
Representation for each input image with $D$ RGB$\alpha$ planes at regularly
sampled depths in the input view’s camera frustrum. Here, each MPI can be used
to render continuously-valued novel views using alpha compositing of color
value along rays within a local neighborhood within a novel view’s (imaginary)
camera.

LLFFs reconstruct interpolated views as a weighted combination of renderings
from multiple nearby MPIs. This process combines local light field
approximations to a near-plane spanning light field and a far-planed
determines by the input views’ field-of-view. This allows to render new biew
path without constrains for 3D translation and rotation as in standard light
field rendering. LLFFs consider the accumulated alpha values as vital for each
MPI rendering when blending to allow MPI rendering to “fill in” content
occluded from camera views. LLFFs prediction network leverage RGB image sets
$C_{k}$ with camera poses $p_{k}$ and produce MPI sets $M_{k}$ for each image.
Each RGB$\alpha$ MPI plane was homographically warped into target pose
$p_{t}$’s reference frame using a predicted MPI $M_{k}$ and then alpha
composited together from back to front. All these steps lead to an RGB image
($C_{t,k}$) and an alpha image ($\alpha_{t,k}$) where the ouput is rendered at
pose $p_{t}$ using MPI pose $p_{k}$. Further, the final RGB output $C_{t}$ is
obtained by blending rendered images $C_{t,k}$ from multiple MPIs as one MPI
does not contain all content visible from the new camera pose as illustrated
in fig. 4(b) and given by eqn 64.

$C_{t}=\frac{\sum_{k}w_{t,k}\alpha_{t,k}C_{t,k}}{\sum_{k}w_{t,k}\alpha_{t,k}},$
(64)

where the method leverages scalar blending weights $w_{t,k}$ (can be smooth
filter) that are modulated by the corresponding accumulated alpha images
($\alpha_{t,k}$) and normalized so that $\alpha=1$. For instance, modulating
the blending weights by referring to eqn. 64 accumulated alpha values prevents
artifacts in $C_{t}$. Bilinear Interpolation from the 4 nearest MPIs is
leveraged for regular grid of data rather than the ideal sinc function
interpolation for extremely limited number of sampled views and efficiency.
All five MPIs are used for irregularly sampled data and $w_{t,k}\propto
exp(-\gamma\ell(p_{t},p_{k}))$. Here, $\ell$ is $L^{2}$ distance between
translation vectors of poses $p_{t}$ and $p_{k}$, and $\gamma$ is given by eqn
65.

$\gamma=\frac{f}{Dz_{min}},$ (65)

where $f$ is the focal length and $z_{min}$ is the minimum distance to the
scene. This strategy for MPI neighborhood blending is particularly efficient
for non-Lambertian renderings. Virtual apparent depth of a specularity changes
with viewpoint [207] for curved surfaces and hence, specularities can appear
as curves in light fields’ epipolar slices and lines can appear for diffuse
points. Here, each of the MPI represents a specularity for a local range of
views by placing the specularity at a single virtual depth. This approach
leverages real world and synthetic dataset of natural scenes [200, 175, 186,
187]. These results that quantitatively validate and demonstrate that LLFFs
match the perceptual quality of Nyquist rate Sampling while leveraging
$\approx 4000\times$ fewer input images has been demonstrated for over 60
diverse real world scenes and this work paved way for future IBR algorithms.

Further, there have been many countless works from 2015–2020 on representing
scenes as unstructured or weakly structured feature representations [210, 233,
55] but the first work to implicit learn the function for scene as continuous
representation was presented in SRNs (Scene Representation Networks).

Figure 5: A model schematic representation of SRNs (Scene Representation
Networks) by Sitzmann et al [197]

SRNs (proposed by Sitzmann et al; depicted in fig 5) enapsulated both the
geometry and the appearance of a scene by leveraging camera poses [197]. This
training was performed end-to-end without explicit suprervision in 3D
geometry, purely from a set of posed 2D images. These networks played an
important role in constructing the basis for NeRFs and gaining better
evaluations for scene reconstructions. All this was done using Differential
Ray Marching using LSTM (Long-Short Term Memory networks). Here, the objective
was to combine multiple different disciplines such as Geometric Deep Learning,
Neural Scene Representation, and Neural Image Synthesis, and Differentiable
Ray Marching to represent the scene as a continuous function $\phi$
responsible for matching a feature representation v of learned scene
properties for a particular spatial location:

$\phi=\mathbb{R}^{3}\rightarrow\mathbb{R}^{n},\textrm{{x}}\mapsto\phi(\textrm{{x}})=\textrm{{v}}$
(66)

where v may be a feature vector that encodes visual information related to the
scene or higher order information such as signed distance of x to closest
surface. Hence, this work became one of the first works to formulate Scenes as
continuous formulation. Earlier works used to store information in a
discretized manner such as Voxel Grids’ discretization in $\mathbb{R}^{3}$
[141, 105, 157, 271, 215, 196] and Point Clouds [174, 91, 128, 145] may have
sparsely discretized points anywhere in the space $\mathbb{R}^{3}$. Here, the
concepts of SRNs was the first one to encode all the information of a scene in
a function which was represented by an MLP (Multi-layer Perceptron) which
limited the scene information to the capacity of the MLP. Further, SRNs
perform Neural Rendering $\theta$ by mapping scene representation $\phi$,
intrinsic K, and extrinsic E to an image I as given by the following equation.

$\theta:\chi\times\mathbb{R}^{3\times 4}\times\mathbb{R}^{3\times
3}\rightarrow\mathbb{R}^{H\times W\times
3},(\phi,\textrm{{E}},\textrm{{K}})\mapsto\theta(\phi,\textrm{{E}},\textrm{{K}})=\textit{I}$
(67)

where $\chi$ represents universal space for functions. Here, these networks
faced a disadvantage at implicitly storing the geometry in the MLPs. Hence, a
learned neural ray marching algorithms was used in this approach and a pixel
generator network was proposed to map geometry/appearance features to colors.
The differentiable Ray Marching Algorithm takes inspiration from Classic
Sphere Tracing algorithm [77] where it solves the eqn 68 by starting at a
distance close to the camera and learning the step distance and stepping such
that ray intersects the geometry (where each step has signed distance length
to the closest surface point of the scene) [164].

$\begin{multlined}\textrm{arg min }d\\\ \textrm{s.t.
{r}}_{u,v}(d)\in\Omega,d>0\end{multlined}\textrm{arg min }d\\\ \textrm{s.t.
{r}}_{u,v}(d)\in\Omega,d>0$ (68)

where $\Omega$ refers to all the points on the surface of the scene. The step
length is learned using a RM-LSTM (Ray Marching Long-Short-Term Memory) [85].
Further, the authors behind SRN also leverage a pixel generator architecture
which is based on $1\times 1$ convolutions instead of 2D convolutions. This is
because the 2D convolutions may hinder multi-view consistency in case if they
are capturing the context of local area which is disrupted when bringing the
camera closer to another adjacent local area of a scene but it also discounts
the ability of these networks to work in a smaller memory budget. SRNs further
generalize across different scenes by jointly optimizing the following
objectives by leveraging stochastic gradient descent.

$\begin{multlined}\operatorname*{arg\,min}_{\\{\theta,\Psi,\\{\textrm{{z}}_{j}\\}^{m}_{j=1}\\}}\sum_{j=1}^{M}\sum_{i=1}^{N}\underbrace{||\Theta_{\theta}(\Phi_{\psi(\textrm{{z}}_{j})},\textbf{E}_{i}^{j},\textbf{K}_{i}^{j})||_{2}^{2}}_{\mathcal{L}_{\textrm{img}}}\\\
\begin{aligned}
&+\underbrace{\lambda_{depth}||\textrm{min({d}}_{i,final}^{j},0)||_{2}^{2}}_{\mathcal{L}_{\textrm{depth}}}\\\
&+\underbrace{\lambda_{latent}||\textrm{{z}}_{j}||_{2}^{2}}_{\mathcal{L}_{\textrm{latent}}}\end{aligned}\end{multlined}\operatorname*{arg\,min}_{\\{\theta,\Psi,\\{\textrm{{z}}_{j}\\}^{m}_{j=1}\\}}\sum_{j=1}^{M}\sum_{i=1}^{N}\underbrace{||\Theta_{\theta}(\Phi_{\psi(\textrm{{z}}_{j})},\textbf{E}_{i}^{j},\textbf{K}_{i}^{j})||_{2}^{2}}_{\mathcal{L}_{\textrm{img}}}\\\
\begin{aligned}
&+\underbrace{\lambda_{depth}||\textrm{min({d}}_{i,final}^{j},0)||_{2}^{2}}_{\mathcal{L}_{\textrm{depth}}}\\\
&+\underbrace{\lambda_{latent}||\textrm{{z}}_{j}||_{2}^{2}}_{\mathcal{L}_{\textrm{latent}}}\end{aligned}$
(69)

Here, the first loss was an $l_{2}$ loss which listed the similarity of the
constructed or rendered image based on ground-truth of that particular path of
area in a scene, whereas $\mathcal{L}_{\textrm{depth}}$ is the constraint in
eq. 68, and finally $\textbf{z}_{j}$ on Gaussian Distribution priors. This
equation can also further be modified for the purpose of Few-Shot
Reconstruction. The scope of experimentation is beyond the scope of this
paper, so, the readers are further advised to study the paper in detail to get
a better idea of the implementation details.

Lombardi et al [135] presented NV (Neural Volumes) as a way to learn both the
geometry and appearance variations simultaneously that led to better
generalization across viewpoints. Their work has been depicted in figure 6. To
address the issue of terminating in poor local minima when optimizing over a
mesh-based representation using gradient-based optimization, the authors used
a volumetric representation as a function of color and opacity at each
position in 3D space. Hence, helping in mitigating problems such as
accountability for reflectance variability or tracking topological evolution
in dynamic media in motion capture systems [78, 184, 4, 241]. Further, during
the optimization of this encoder-decoder model, semi-transparent
representation of geometry disperses gradient information along the ray of
integration, effectively widening the basin of convergence, enabling the
discovery of good solutions. To overcome the limitation of using intensive
memory for the sparse voxel grids, the authors worked with a warping technique
that indirectly escapes the restrictions imposed by a regular voxel grid
structure, allowing the learning algorithm to make the best use of available
memory. Further, since the voxel grid structure isn’t used directly there were
less grid-based artifacts in this approach.

Figure 6: A block representation of NVs (Neural Volumes) by Lombardi et al
[135]

Much like SRNs, the NV method is based on two main parts of the architecture
which consist of an encoder-decoder network that converges after mapping image
to a 3D volume V(x), and differentiable ray marching for rendering image from
V with a given camera parameters. This is analogous to an Auto-Encoder with
fixed function volume rendering operation for the final layer. Here, the model
maps from 3D location space to the 3D positions, $\textrm{x}\in\mathbb{R}$ as
stated in eqn. 70 given below.

$\textrm{{V}}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{4},\textrm{{V(x)}}=(\textrm{{V}}_{\textrm{rgb}}(\textbf{{x}}),\textrm{{V}}_{\alpha}(\textbf{{x}})),$
(70)

where $\textrm{{V}}_{\textrm{rgb}}(\textbf{{x}})\in\mathbb{R}^{3}$ and
$\textrm{{V}}_{\alpha}(\textbf{{x}})$ are the color and opacity (range
$[0,\infty]$) at the 3D coordinate x. This enables the model to learn soft
discrete volume representation that allowed gradients to flow backward easily
and model fine and transparent structures (like hair and smoke).
NV also discusses the potential of leveraging only 3 images from 3
orthogonally placed cameras to achieve maximize convergence while downsampling
an image by a factor of 8. Here, the Encoder is a Convolutional Variational
encoder whereas the authors leverage an MLP for Decoder (which demands
prohibitive memory requirement to produce reconstructions with higher
quality). Hence, Lombardi et al [135] model the volume function as a discrete
3D grid of voxels hence, gaining the output tensor in the form of discretized
voxels. The authors here optimized the decoder as a function $S(\textrm{{x};
{Y}}):\mathbb{R}^{3}\rightarrow\mathbb{R}^{C}$ to samples from the grid Y by
scaling continuous values in the range $[-1,1]$ to $[1,D]$ in each dimension
followed by trilinear interpolation. Hence, for general case of 3D cube with
center at $\textrm{{x}}_{0}$ with $W$ sides is given by the eqn. 71.

$\textrm{{V(x; z)}}=S\cdot\left(\frac{\textrm{x -
x}_{0}}{W/2};\textrm{{g(z)}}\right),$ (71)

Note that here the g(z) represents the MLP used for decoding of the Voxel
Grid. The authors also discussed how they can translate a similar architecture
with $1\times 1\times 1$ cube with 1024 channel for duplicating the function
generated by the MLP. Later, a softplus activation to restrain the voxels to
positive values rather than negative values that might occur after decoing.
Further, a warping function is applied to the voxel decoder to mitigate the
wasteful behavior of the Voxel Grids (also solved by Octree-based approaches
[181] but they are complex and require object distribution to be known a
priori). Hence, the authors used an inverse warping formulation as given in
the equation 72 below.

$\textrm{W}^{-1}(\textrm{x})\rightarrow\textrm{y, where {x,
y}}\in\mathbb{R}^{3},$ (72)

This helps to map points from a warp volume $\textrm{{W}}^{-1}(\textrm{{x}})$
to a template volume T(x) where both volumes are decoded from the dynamic
latent code z. Hence, in the eqn. 72, x is a 3D coordinate in the output space
and y is the corresponding 3D coordinate in RGB$\alpha$ template volume.
Hence, the mapping can be depicted by using the eqn. 73.

$\textrm{{V}}_{\textrm{RGB$\alpha$}}(\textbf{x})=\textrm{{T}}_{\textrm{RGB$\alpha$}}(\textrm{{W}}^{-1}(\textrm{{x}})).$
(73)

These warp fields were developed as a mixture of Affine Warps to produce an
affine warp field without encountering any overfitting issues or any non-
linear bending. This formulation has been depicted by the eqn. 74–75 given
below.

$\textrm{{W}}^{-1}(\textrm{{x}})=\sum_{i}\textrm{{A}}_{i}(\textrm{{x}})\textrm{{
}}a_{i}(\textrm{{x}}),$ (74)

$\textrm{with
}\textrm{{A}}_{i}(\textrm{{x}})=\textrm{{R}}_{i}(\textrm{{s}}_{i}\circ(\textrm{{x
- t}}_{i})),\textrm{
}a_{i}(\textrm{{x}})=\frac{w_{i}(\textrm{{A}}_{i}(\textrm{{x}}))}{\sum_{j}w_{i}(\textrm{{A}}_{i}(\textrm{{x}}))},$
(75)

where $\textrm{{A}}_{i}(\textrm{{x}})$ is the $i^{th}$ affine transformation,
$w_{i}$ is the weight of each warp, and $\circ$ is the element-wise
multiplication. Further, the authors also made the RGB decoder constrained by
viewpoint conditions by given the normalized direction of the camera to the
decoder along the encodings as an input. Further, the convolutions for the RGB
and $\alpha$ component were disentangled as RGB values were conditioned on the
viewpoint. Further, for the ray-tracing and End-to-End training this approach
leverages Semi-Transparent Volume Rendering and a combination of
Reconstruction Priors and Hyperparameter Tuning.
Here, the ray marching accumulates not only color but also opacity to model
the occlusions and when opacity hits the value of 1, then no color is
accumulated on the ray. Hence, the rendering process for this approach is as
given by the eqn. 76–77.

$\textrm{{I}}_{\textrm{rgb}}(\textrm{{p}})=\int_{t_{min}}^{t_{max}}\textrm{{V}}_{\textrm{rgb}}(\textrm{{r}}_{o}+t\textrm{{r}}_{d})\frac{\textrm{
d}\alpha(t)}{\textrm{d}t}\textrm{ d}t,$ (76)

$\textrm{where
}\alpha(t)=min\left(\int_{t_{min}}^{t}\textrm{{V}}_{\alpha}(\textrm{{r}}_{o}+s\textrm{{r}}_{d})\textrm{
d}s,1\right),$ (77)

$\textrm{and }\textrm{{r}}_{d}=\frac{\textrm{{P}}^{-1}\textrm{{p--
r}}_{o}}{||\textrm{{P}}^{-1}\textrm{{p--r}}_{o}||}\in\mathbb{R}^{3},$ (78)

where $\textrm{{I}}_{\textrm{rgb}}(\textrm{{p}})$ is the color at a pixel
location in the camera coordinate system with center
$\textrm{{r}}_{o}\in\mathbb{R}^{3}$, ray direction is given by the eqn. 78 for
the Image Coordination System-to-World Coordinate System.

Color calibration based on per-camera and per-channel bias $b$ and gain $g$
ensures that the reconstructed images is accounted for the slight differences
in overall intensity in the image. Further, a background estimation based on
the eqn. 79

$\widehat{\textrm{{I}}}_{\textrm{rgb}}(\textrm{{p}})=\left(1-\textrm{{I}}_{\alpha}(\textrm{{p}})\right)\textrm{{I}}_{\textrm{rgb}}^{\textrm{(bg)}}(\textrm{{p}})+\left(g\textrm{{I}}_{\textrm{rgb}}(\textrm{{p}})+b\right),$
(79)

where $\textrm{{I}}_{\textrm{rgb}}^{\textrm{(bg)}}$ and
$\widehat{\textrm{{I}}}_{\textrm{rgb}}$ are the background image extracted
based on static or non-changing background, and final image, respectively.
Further, NV introduces two priors in its architecture–total variation of the
log voxel opacities, and beta distribution (Beta(0.5, 0.5)) on the final image
opacity. These priors are demonstrated by the eqn. 80–81.

$\mathcal{P}_{\textrm{var\\_lvoxop}}(\textrm{{V}}_{\alpha})=\frac{1}{N}\sum_{x}\lambda_{\textrm{var\\_lvoxop}}\left|\left|\frac{\partial}{\partial\textrm{{x}}}\textrm{log
{V}}_{\alpha}\textrm{{(x)}}\right|\right|$ (80)

$\textrm{and
}\mathcal{P}_{\beta}(\textrm{{I}}_{\alpha})=\frac{1}{P}\sum_{\textrm{{p}}}\lambda_{\beta}\left[\textrm{log({I}${}_{\alpha}$({p}))
+ log(1 -- {I}${}_{\alpha}$({p}))}\right]$ (81)

where p, $P$, and $\mathcal{P}$ represent a pixel in image, number of pixels,
and the priors used. Further, the training objective of NV is based on the
equation 82 where $\textrm{{I}}_{\textrm{rgb}}^{*}$ is the ground truth image
and $D_{KL}(\textrm{{z}}||N(0,1))$ is the KL divergence between the latent
encoding z and Bell Distribution used in a VAE (Variational Autoencoder).

$\begin{multlined}l(\theta)=\frac{1}{P}\sum_{\textrm{{p}}}\left|\left|\widehat{\textrm{{I}}}_{\textrm{rgb}}(\textrm{{p}})\textbf{--}\textrm{{I}}_{\textrm{rgb}}^{*}(\textrm{{p}})\right|\right|^{2}\\\
\begin{aligned}
&+\lambda_{\textrm{KL}}D_{\textrm{KL}}\left(\textrm{{z}}||\mathcal{N}(0,1)\right)\\\
&+\mathcal{P}_{\textrm{var\\_lvoxop}}(\textrm{{V}}_{\alpha})\\\
&+\mathcal{P}_{\beta}(\textrm{{I}}_{\alpha}).\end{aligned}\end{multlined}l(\theta)=\frac{1}{P}\sum_{\textrm{{p}}}\left|\left|\widehat{\textrm{{I}}}_{\textrm{rgb}}(\textrm{{p}})\textbf{--}\textrm{{I}}_{\textrm{rgb}}^{*}(\textrm{{p}})\right|\right|^{2}\\\
\begin{aligned}
&+\lambda_{\textrm{KL}}D_{\textrm{KL}}\left(\textrm{{z}}||\mathcal{N}(0,1)\right)\\\
&+\mathcal{P}_{\textrm{var\\_lvoxop}}(\textrm{{V}}_{\alpha})\\\
&+\mathcal{P}_{\beta}(\textrm{{I}}_{\alpha}).\end{aligned}$ (82)

Various usecases and results for the same have been discussed in the paper.
Further, we refrain from discussing the implementation, training, and
hyperparameter details for this network architecture hence the reader is
advised to read the paper for more comprehensive intuition about the author’s
approach.

Thies et al proposed DNR (Deferred Neural Rendering) [212] to leverage
traditional graphics components and make those learnable using Neural
Networks. The main idea behind this work is to use imperfect 3D content from
photometric reconstructions with incomplete or otherwise noisy surface
geometry to produce photo-realistic (re-renderings). The contribution here was
the introduction of Neural Textures that were learned as feature maps from the
scene capture process. These are stored similar to texture maps on top of 3D
mesh proxies. High Dimensional Feature Maps contain more information (when
compared to traditional texture maps) that were encoded in high dimensional
feature maps and used deferred neural pipeline for interpretation. In this
approach, both the Deferred Neural Renderer and Neural Textures were trained
end-to-end which enabled synthesizing photo-realistic images with
imperfections in the original 3D content. This approach gave significant
explicit control over generated output such as synthesizing temporally-
consistent video re-renderings of recorded 3D scenes in 3D space. This allowed
Neural Textures to coherently re-render and manipulate existing video content
in real-time static and dynamic environments. Further, this approach has been
leveraged for several applications such as Novel View Synthesis, Scene
Editing, Facial Reenactments. Moreover, this rendering that relied on geometry
proxy (needs to be reconstructed in pre-processing step) was much faster than
traditional reenactments for high-resolution outputs. A drawback of this
approach was the retraining of neural textures for every object fed to the
DNR. This pipeline allowed generalization to happen on 10 objects used, which
suggested applications in the domain of transfer learning, segmentation, scene
illumination, surface reflectance, novel shading, and lighting capabilities
with usage of this pipeline. Further, this work had the capability to be
extended beyond the traditional 2D to volumetric grids. The Average Precision
for the Ground Truth proposal has been given in the eqn 83.

$AP@n=\frac{1}{GTP}\sum_{k}^{n}{P@k\times rel@k}$ (83)

Another Neural Rendering approach called NPBG (Neural Point-Based Graphics)
(proposed by Aliev et al [2]) explores the idea of augmenting each point with
a learnable neural descriptor that encodes local geometry and appearance.
Authors claim that this approach results in compelling results for scenes
scanned with standard RGB cameras and hand-held commodity RGB-D sensors even
with the objects that are challenging for standard mesh-based rendering. The
idea of NPBG combines the ideas behind IBR, NPG, and Neural Rendering
discussed briefly earlier. It uses a Deep CNN (Convolutional Neural Network)
to generate photorealistic scenes from novel viewpoints by leveraging Scene
Geometry Raw Point-Clouds Representation. Latent Vector (here, Neural
Descriptors) learning/estimation facilitates realistic rendering by describing
both geometric and photometric properties of scenes in conjunction with
learning of the rendering network. The fig. 7 gives an intuition into how NPBG
works and how the rendering algorithm is jointly optimized while learning the
Neural Descriptors.
This approach formulates the problem of Neural Rendering with the point clouds
($P=\\{p_{1},p_{2},...,p_{N}\\}$) and M-dimensional Neural Descriptors
($\mathbb{D}={d_{1},d_{2},...,d_{N}}$) given and the requirement for the
renderer to obtain a new view with a camera $\mathcal{C}$ (with extrinsic
parameter and intrinsic parameters).

Figure 7: A block diagram representation of NPBGs (Neural Point Based
Graphics) by Aliev et al [2]

The authors suggest to outline the rendering process such that descriptors are
used as pseudo-colors to project the point into the target view. Further, the
rendering network is used to map the pseudo-colors to photorealistic RGB
space. Formally, NPBG creates an $M-$channel raw-image of size $W\times H$,
and for each point $p_{i}$ projecting to $(x,y)$ given by the eqn. 84

$S(\textrm{{P, D,}}C)[[x],[y]]=d_{\textrm{i}},$ (84)

where $[a]$ is nearest integer for $a\in\mathbb{R}$. Further, this approach
proposes to leverage a z-buffer to remove occluded points as several points
can project onto the same pixel but the lack of topological information in
point clouds can result in the bleeding problem (where occluded surfaces and
background points could be seen through from front surface). This approach
proposes to leverage multi-scale progressive rendering to avoid relying on
choice of disk radius (important for splatting which uses a 3D elliptic disk
with an estimated radius to replace each point). Here, a sequence of images
$S[1],S[2],...,S[T]$, where the i-th image has the size of
$\frac{W}{2^{t}}\times\frac{H}{2^{t}}$ is obtained by rendering a point cloud
on a canvas pyramid of different resolution $T$ times by performing simple
point-cloud projection. Hence, $S[1]$ has the maximum details and is prone to
the most surface bleeding whereas $S[T]$ suffers from least surface bleeding
but has the minimum geometric details. Every other image in sequence
($S[2],S[3],...,S[T-1]$) has different detail-bleeding tradeoffs. Now, all the
raw sequence of images are mapped to three channel RGB image $I$ using a U-Net
[183] (with Gated Convolutions [252] to handle sparse inputs) based rendering
network ($\mathcal{R}_{\theta}$ as given in the eqn, 85 below.

$I(\textrm{{P, D,}}C,\theta)=\mathcal{R}_{\theta}\left(S[1](\textrm{{P,
D,}}C),....,S[T](\textrm{{P, D,}}C)\right)$ (85)

The, the raw images $S[i]$ are concatenated to the U-Net first block with
corresponding resolutions (this coarse-to-fine mechanism and mipmapping are
similar). Hence, rendering network provides the mechanism for implicit detail
selection in this case. Further, OpenGL is used for rasterization of raw image
sequences $S[1],S[2],...,S[T]$. The dimensionality of the descriptors is set
to eight ($M=8$).
The fitting process in this system assumes that during the fitting operation,
there are $K$ different scenes are available. Given point cloud
$\textrm{{P}}^{k}$ and training ground truth set with $L_{k}$ RGB images
$\textrm{{I}}^{k}-\\{I^{k,1},I^{k,2},...,I^{k,L_{k}}\\}$ and known camera
parameters $\\{C^{k,1},C^{k,2},...,C^{k,L_{k}}\\}$ for $k-$th scene, helps
formulate objective function $\mathcal{L}$ (mismatch between the rendered and
ground truth RGB image) as given in the eqn. 86.

$\mathcal{L}(\theta,\textrm{{D}}^{1},\textrm{{D}}^{2},...,\textrm{{D}}^{K})=\sum_{k=1}^{K}\sum_{l=1}^{L_{k}}\left(\Delta\left(\mathcal{R_{\theta}}(\\{S[i](\textrm{{P}}^{k},\textrm{{D}}^{k},C^{k,l})\\}_{i=1}^{T})\right),I^{k,l}\right).$
(86)

Here, $\Delta$ represents the mismatch between ground truth and rendered
images, $\textrm{{D}}^{k}$ are neural descriptors for point clouds of the
$k$-th scene and a perceptual loss [51, 102] helps compute the mismatch for
activations of a pretrained VGG16 network [193]. Both the neural descriptors
and Rendering network parameters in the training set of the scenes are used
for the optimizing the loss in eqn. 86 by the ADAM optimizer [109]. This
system can be used for transfer/incremental learning which suggests that the
results for new viewpoints tend to be better when rendering network is fitted
to multiple scenes of a similar kind when compared to a single scene. Hence,
this process is two-stage training process where the network is first pretrain
for the rendering network on a scene family, followed by fine-tuning the
rendering network to a new scene. In the later stage, the learning process 86
starts with no neural scenes descriptors and pretrained weights of the
rendering network.

The inspiration of NeRF architecture can be said to be derived from
Mildenahall and Tancik’s own work [147, 209], Sitzmann et al’s SIREN (Implicit
Neural Representations with Periodic Activation Functions) [194] networks
introduced in the same year. SIRENs [194] introduced a novel approach for
constructing implicit representations using periodic activation functions.
This technique has shown to provide a more efficient and stable way to
represent complex functions with deep neural networks. Whereas, “Fourier
features let networks learn high frequency functions in low dimensional
domains” [209] proposed a way to augment neural networks with Fourier features
to allow them to learn high-frequency functions (real-life image
representations) in low-dimensions. This approach enables neural networks to
represent complex functions with fewer parameters and to learn more
efficiently. Finally, the NeRFs build on the previous ideas to propose a
method to represent 3D scenes using a continuous function for the plenoptic
function that can be implicitly stored as Neural Representation. This approach
allowed for high-quality view synthesis by rendering images from any viewpoint
in a scene using much less storage when compared to polygonal meshes or
pointclouds. (Work in Progress for Surveys related to NeRFs…)

##  4 Important Architectures, Models, and Extensions

This section briefly discusses the Radiance Fields architecture and discusses
the improvements, extensions, and models for different applications. These
research works have been briefly described according to the year they were
published on Arxiv. The author wants to keep the source of the research
related to the Radiance Field Literature so as to stay consistent in the
source of information

  * •

Research in 2020: This subsection briefly discusses the research articles in
2020. NeRFs were introduced in the March of this year as a preprint rolled out
by Mildenhall et al [148].

    * –

Original NeRF [148] is the model that uses fully connected deep neural network
to represent a continuous 3D scene function, which takes a 3D point and a
viewing direction as inputs and outputs a color and density function values.

    * –

Fourier Features Let Neural Networks Learn High Frequency Functions in Low
Dimensional Domains [209] introduces Fourier features, which improve the
ability of neural networks to learn high-frequency functions in low-
dimensional domains. The authors propose mapping input coordinates to a
higher-dimensional space using a random Fourier feature mapping, which
significantly improves the network’s ability to model high-frequency
functions. This technique proves to be particularly useful for tasks such as
geometric modeling and texture synthesis.

    * –

GRAFs (Generative Radiance Fields) [188] extend NeRFs to the domain of
generative modelling. While NeRF focuses on learning a continuous 3D
representation of a scene from a set of 2D images with known camera
parameters, GRAFs aim to generate novel 3D scenes and synthesize views of
those scenes without relying on explicit 3D geometry. GRAFs combine the NeRFs
representation with a generative model like VAE (Variational Auto-Encoder) or
a GAN (Generative Adversarial Networks). The generative model learns a latent
space that encodes the underlying structure of the 3D scene with the
conditional NeRF mapping the points in this latent space to 3D radiance
fields. The training objective typically involves a combination of
reconstruction and regularization loss. After training, GRAFs can sample
latent vectors from the learned latent spaces and synthesize views of the
scenes using conditional NeRF model. This makes GRAFs particularly useful for
tasks such as view synthesis, 3D scene interpolation, and data augmentation,
as they can generate diverse and realistic 3D scenes without relying on
explicit 3D geometry.

    * –

NSVF (Neural Sparse Voxel Fields) [131] extends NeRF by introducing a sparse
voxel field representation, which significantly reduces memory requirements
and accelerates rendering. This method combines a sparse voxel octree with a
neural network to model scene appearance, enabling efficient reconstruction
and rendering of large scale scenes.

    * –

NeRF-W (NeRF in the Wild) [140] is an extension of the original NeRF model to
handle more complex, real-world scenes with uncontrolled illumination. NeRF-W
introduces additional input features to represent the spatially-varying
lighting conditions, improving the model’s ability to generalize across
diverse scenes.

    * –

Neural Reflectance Fields [14] presented a method to capture spatially-varying
appearance and fine-geometric details of real-world objects using NeRF. It
introduces a multi-scale reflectance field representation and a multi-view
photometric stereo method to estimate per-point reflectance and normals,
enabling high-fidelity reconstruction of complex objects.

    * –

Neural Face Reflectance Fields [211] proposed a method to reconstruct neural
face reflectance fields from a single monocular input image. This method
estimates per-pixel surface normals, albedo, lighting parameters and combines
them with NeRF-based representation to synthesize high-quality novel views of
the face.

    * –

GRF (Learning a General Radiance Fields for 3D Representation and Rendering)
[214] presents a method for learning a 3D representation of a scene and
rendering it using neural radiance fields. The authors introduce a multi-scale
representation that allows the network to model both global structures and
fine details of a scene. The resulting model can be used for various tasks,
such as novel view synthesis, relighting, and geometric reconstruction.

    * –

NeRF++ [257] model builds upon the original NeRF by incorporating additional
features such as spatially-varying reflectance and incorporating auxiliary
tasks to improve training. The main goal of NeRF++ is to enhance the quality
of the generated images while maintaining robustness and efficiency.

    * –

Neural Scene Graphs for Dynamic Scenes [165] focuses on representing and
rendering dynamic 3D scenes using neural scene graphs. The authors propose a
method that combines scene graph representations with neural radiance fields
to enable efficient rendering of complex and dynamic scenes. The method can
generate high-quality images of dynamic scenes with realistic lighting and
reflections, as well as handle occlusions and disocclusions.

    * –

GIRAFFE (Compositional Generative Neural Feature Fields) [160] presents a
compositional generative model that represents scenes as neural feature
fields. The method combines a scene graph representation with a conditional
NeRF model to synthesize novel views of complex scenes with varying objects,
poses, and appearances, enabling high-quality image synthesis and scene
editing.

    * –

DeRF (Decomposed Radiance Fields) [177] introduced a method to separate the
global illumination, local shading, and surface color components of a scene,
enabling efficient view synthesis and relighting. The method decomposes a
neural radiance field into these components and learns a hierarchical
representation that captures the spatial and directional dependencies in the
scene.

    * –

NerFies (Deformable Neural Radiance Fields) [166] extended NeRF to represent
non-rigidly deforming objects, such as human faces and bodies. The method
combines a neural radiance field with a parametric deformation model, enabling
reconstruction and view synthesis of dynamic scenes from monocular video.

    * –

Space-Time Neural Irradiance Fields [235] introduced a space-time
representation that models the temporal dynamics of free-viewpoint video. The
method extends NeRF to represent time-varying appearance and geometry,
enabling high-quality view synthesis and interpolation of dynamic scenes.

    * –

Neural Scene Flow Fields [127] presented a method to represent the scene flow
of dynamic objects using NeRF. The method learns a neural scene flow field
that models the temporal evolution of object appearance and geometry. By
encoding both spatial and temporal information, it enables high-quality view
synthesis of dynamic scenes from a limited number of input views.

    * –

D-NeRF (Neural Radiance Fields for Dynamic Scenes) [173] extended NeRF to
handle dynamic scenes by learning a temporally coherent neural radiance field
from a set of multi-view images with known camera parameters. D-NeRF models
the time-varying appearance and geometry by conditioning the neural radiance
field on a latent code that varies with time. The method introduces an
additional loss term to enforce temporal smoothness in the learned radiance
field, allowing for high-quality view synthesis of dynamic scenes with
temporal consistency. This approach enables the reconstruction of dynamic
scenes from multi-view videos and synthesizes novel views while maintaining
the appearance and motion continuity.

    * –

pixelNeRF [250] extends NeRF to generate high-quality novel views from one or
few input images, instead of requiring multiple images with known camera
parameters. It leverages 2D convolutions to efficiently aggregate information
from input images and incorporates this information into the neural radiance
field, allowing for improved view synthesis from limited input data.

    * –

Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction
[59] proposed a method to reconstruct dynamic neural radiance fields for human
faces from a single monocular video. The method captures the 4D facial
geometry and appearance, allowing for high-quality facial avatar
reconstruction and novel view synthesis with temporal consistency.

    * –

NeRD (Neural Reflectance Decomposition) [17] presented a method for learning
scene-specific neural reflectance models from image collections. The method
involves decomposing input images into albedo, normal, and lighting
components, allowing for more accurate and flexible representations of complex
scenes. This approach can be used to synthesize novel images, re-light scenes,
and perform geometry-aware image editing.

    * –

NeRV (Neural Reflectance and Visibility Fields) [202] introduced a method to
represent both reflectance and visibility fields for relighting and view
synthesis tasks. The method leverages a novel hierarchical representation that
enables efficient and high-quality rendering of complex scenes under novel
lighting conditions and viewpoints.

    * –

iNeRF (Inverting Neural Radiance Fields) [248] presented a method to invert
neural radiance fields to estimate the pose of an object or scene given a
single image. By learning a mapping from 2D images to the latent space of the
radiance field, iNeRF enables accurate pose estimation without explicit 3D
geometry.

    * –

Portrait Neural Radiance Fields [62] focused on generating portrait neural
radiance fields from a single input image. The method leverages a combination
of geometric priors, facial landmarks (such as 3DMM features), and a neural
radiance field representation to reconstruct 3D facial geometry and
appearance, enabling high-quality novel view synthesis. The priors are
incorporated from a dataset of 3D facial scans. The resulting method can
generate novel views and relight the face, even with limited input data.

    * –

Object-Centric Neural Scene Rendering [73] worked on an object-centric
representation for neural scene rendering. The method represents scenes as a
collection of object-centric neural radiance fields and learns to compose them
into a global scene representation, enabling high-quality view synthesis and
intuitive scene editing.

    * –

NeRFlow (Neural Radiance Flow) [54] extends NeRF to model the temporal
dynamics of video sequences. The method learns a continuous 4D radiance field
that represents both the spatial and temporal information in the video,
enabling high-quality view synthesis and video processing tasks, such as
interpolation and extrapolation.

    * –

Compositional Radiance Fields [224] presents a method to learn compositional
radiance fields for dynamic human heads. The method combines a parametric
human head model with a neural radiance field representation, allowing for
high-quality view synthesis and editing of dynamic human head models.

    * –

Non-Rigid Neural Radiance Fields [213] proposes a method to reconstruct and
synthesize novel views of non-rigidly deforming scenes from monocular video.
The method combines a neural radiance field with a non-rigid deformation
model, enabling high-quality reconstruction and view synthesis of dynamic
scenes.

    * –

Neural Body [170] paper proposes a method for representing and synthesizing
novel views of dynamic humans using structured latent codes and implicit
neural representations. The authors introduce a method that decomposes the
input data into body shape, pose, and appearance components, enabling the
network to learn a more accurate and flexible representation of the human
body. The method can generate realistic and temporally coherent novel views of
dynamic humans, even with sparse input data. Hence, the authors build upon the
NeRF framework by introducing structured latent codes to represent dynamic
humans. Specifically, they decompose the input data into body shape, pose, and
appearance components. This decomposition allows the model to capture the
complex and dynamic nature of human bodies more effectively than standard
NeRF. (Work in Progress…)

  * •

Research in 2021: This subsection briefly discusses the research articles in
the following year 2021. There were 110 research papers related to the field
of Neural Radiance Fields (not including survey papers) in 2021.

    * –

Non-line-of-sight Imaging via Neural Transient Fields [192] presented a method
for non-line-of-sight (NLOS) imaging using neural transient fields. The
authors proposed a learning-based framework that leverages transient light
measurements to reconstruct hidden objects. The key idea was to represent the
volumetric distribution of light in a scene using a neural network, which
learns the mapping from transient measurements to the 3D geometry and
reflectance of the hidden objects. While not a direct extension of NeRF, this
work shares similarities with NeRF in representing volumetric information
using neural networks.

    * –

Pixel-Aligned Volumetric Avatars [176] introduced a method for creating pixel-
aligned volumetric avatars from multi-view video data. It proposed a learning-
based approach that combines the benefits of traditional multi-view stereo and
neural rendering methods. By using a fully convolutional neural network to
predict a volumetric representation, the method generates high-quality avatars
with consistent appearance across multiple views. Although this research does
not directly extend NeRF, it shares the goal of creating high-quality 3D
representations that can be efficiently rendered into 2D images.

    * –

A-NeRF (Articulated Neural Radiance Fields) [204] was an extension of NeRF
that focused on learning human shape, appearance, and pose. It propose a
method that represents humans as an articulated structure, where each body
part has its own local neural radiance field. This decomposition allows A-NeRF
to capture the complex geometry and appearance of humans more effectively than
the original NeRF. The method can reconstruct detailed 3D models of humans and
render novel views with realistic appearance and accurate poses.

    * –

NeRF– (Neural Radiance Fields Without Known Camera Parameters) [223] was an
extension of NeRF that relaxes the requirement of having known camera
parameters. The authors propose a method that jointly learns the neural
radiance field and camera parameters from a collection of uncalibrated input
images. By incorporating differentiable camera optimization, the method can
estimate both the 3D scene representation and the camera parameters
simultaneously. NeRF– enables the use of NeRF for scenarios where camera
parameters are not known or only partially available.

    * –

ShaRF (Shape-Conditioned Radiance Fields) [180] is an extension of NeRF that
focuses on learning 3D scene representations from a single input view. It
combines a shape-conditioned radiance field with a shape prior learned from a
large dataset of 3D models. The method leverages this shape prior to guide the
learning of the radiance field, which enables accurate 3D reconstructions and
novel view synthesis even with limited input data. ShaRF demonstrates the
potential of using NeRF in single-view scenarios by incorporating additional
prior knowledge about the underlying 3D structure.

    * –

NeuTex [236] is an extension of NeRF that focuses on improving the efficiency
of volumetric neural rendering by introducing neural texture mapping. The
authors propose a two-stage approach that decouples geometry and appearance:
first, a coarse geometry is reconstructed using a neural radiance field, and
then, a neural texture map is learned to refine the appearance. The training
paradigm involves a combination of a differentiable ray-marching algorithm and
texture loss. The method allows for faster rendering and lower memory
requirements compared to the original NeRF, while maintaining high-quality
results for tasks like novel view synthesis and relighting.

    * –

Mixture of Volumetric Primitives for Efficient Neural Rendering [136]
introduces an improvement over NeRF by representing scenes as a mixture of
volumetric primitives, such as spheres and cylinders, which reduces the
complexity of the scene representation. The training paradigm involves
optimizing the parameters of these primitives, their positions, and a neural
radiance field for each primitive. The loss function is a combination of the
rendering loss and a regularization term. The method enables more efficient
rendering and storage while maintaining the quality of the generated images,
making it suitable for real-time applications and rendering large scenes.

    * –

DONeRF (Depth Oracle Networks-based Compact Radiance Fields) [154] is an
extension of NeRF that aims to achieve real-time rendering of compact radiance
fields. The authors propose using a depth oracle network to predict the depth
of the scene, which reduces the number of samples required for rendering. The
training paradigm involves a two-stage approach: first, the depth oracle
network is trained using supervised depth data, and then, the neural radiance
field is trained using the predicted depth. The loss function combines a
rendering loss and a depth loss. DONeRF significantly accelerates the
rendering process while maintaining high-quality results, making it suitable
for real-time applications.

    * –

NeX [230] is an improvement over NeRF that focuses on real-time view synthesis
by using a neural basis expansion approach. The authors propose representing
the scene as a linear combination of basis functions, which are learned using
a neural network. The training paradigm involves optimizing the neural network
to predict the basis coefficients and the radiance field. The loss function is
a combination of the rendering loss and a regularization term. NeX enables
real-time view synthesis with high-quality results and lower memory
requirements compared to NeRF, making it suitable for real-time applications
and virtual reality.

    * –

FastNeRF [64] is an improvement over NeRF that aims to achieve high-fidelity
neural rendering at high frame rates. The authors propose a multi-scale
hierarchical approach that accelerates rendering by leveraging a coarse-to-
fine strategy. The training paradigm involves training multiple neural
radiance fields at different levels of detail, and the loss function combines
the rendering loss and a multi-scale consistency loss. FastNeRF significantly
accelerates the rendering process while maintaining high-quality results,
making it suitable for applications that require high frame rates, such as
video games and virtual reality.

    * –

Neural Lumigraph Rendering [107] is an extension of NeRF that combines the
benefits of traditional Lumigraph rendering and neural radiance fields. The
authors propose a method that leverages a sparse set of input views to learn a
multi-plane representation of the scene. This multi-plane representation is
then used as input to a neural rendering network, which generates high-quality
novel views. The training paradigm involves optimizing the neural rendering
network using a rendering loss, which measures the difference between the
ground truth and the predicted images. This method offers a more efficient and
flexible rendering solution than NeRF and can be used for applications like
view synthesis and relighting.

    * –

iMAP (Implicit Mapping and Positioning) [205] is an application of NeRF that
focuses on real-time simultaneous mapping and positioning in 3D environments.
The authors propose a method that learns an implicit 3D map of the environment
using a neural radiance field and estimates the camera pose simultaneously.
The training paradigm involves optimizing the neural network to minimize the
re-projection error and a depth consistency loss. iMAP demonstrates the
potential of using NeRF for real-time 3D mapping and positioning tasks, such
as robot navigation and augmented reality.

    * –

Mip-NeRF [10] is an improvement over NeRF that introduces a multiscale
representation to reduce aliasing artifacts in the rendered images. The
authors propose a mipmapping approach that learns a hierarchy of neural
radiance fields at different levels of detail. The training paradigm involves
optimizing the neural networks at each level of detail using a rendering loss
and a consistency loss between adjacent levels. Mip-NeRF achieves higher-
quality rendering with fewer aliasing artifacts, making it suitable for
applications that require high-quality image synthesis, such as virtual
reality and film production.

    * –

Kilo-NeRF [178] is an improvement over NeRF that aims to speed up rendering by
using thousands of tiny multilayer perceptrons (MLPs) instead of a single
large MLP. The authors propose dividing the scene into small regions and
training a separate MLP for each region, which allows for more efficient
rendering and parallelization. The training paradigm involves optimizing the
MLPs using a rendering loss that measures the difference between the ground
truth and the predicted images. Kilo-NeRF significantly accelerates the
rendering process while maintaining high-quality results, making it suitable
for applications that require real-time rendering, such as video games and
virtual reality.

    * –

PlenOctrees [249] is an improvement over NeRF that focuses on real-time
rendering by introducing a hierarchical data structure called PlenOctrees. The
authors propose representing the scene using an octree, where each node stores
a neural radiance field that models the local geometry and appearance. The
training paradigm involves optimizing the neural radiance fields using a
rendering loss and a multi-scale consistency loss. PlenOctrees enables real-
time rendering of neural radiance fields with high-quality results and lower
memory requirements compared to NeRF, making it suitable for real-time
applications and virtual reality.

    * –

Baking Neural Radiance Fields [82] is an improvement over NeRF that focuses on
real-time view synthesis by converting learned neural radiance fields into a
more efficient representation for rendering. The authors propose a method that
bakes the neural radiance field into a set of layered depth images (LDIs) and
then uses these LDIs for real-time rendering. The training paradigm involves
optimizing the neural radiance field using a rendering loss that measures the
difference between the ground truth and the predicted images. This method
significantly accelerates the rendering process while maintaining high-quality
results, making it suitable for real-time applications such as video games and
virtual reality.

    * –

MINE (Multiview Image-based Neural Extensions) [124] is an extension of NeRF
that combines the benefits of multiplane images (MPIs) and NeRF for novel view
synthesis. The authors propose a method that learns a continuous MPI
representation using a neural network, which can be used for efficient
rendering of novel views. The training paradigm involves optimizing the neural
network using a rendering loss and a depth consistency loss. MINE demonstrates
the potential of combining the strengths of MPIs and NeRF to achieve high-
quality view synthesis with lower computational requirements, making it
suitable for real-time applications and virtual reality.

    * –

MVSNeRF (Multi-view Stereo Neural Radiance Fields) [25] is designed to
leverage the strengths of both multi-view stereo (MVS) and NeRF techniques.
MVSNeRF combines a coarse MVS reconstruction with the NeRF framework to
generate high-quality novel views. This hybrid approach takes advantage of the
geometric information provided by MVS to improve the efficiency and quality.

    * –

GNeRF (Generative Adversarial Network-based Neural Radiance Field) [144] is an
extension of NeRF that introduces a GAN-based framework for learning neural
radiance fields without known camera poses. The authors propose a method that
jointly learns the neural radiance field and camera poses using a combination
of a rendering loss, an adversarial loss, and a cycle consistency loss. The
training paradigm involves optimizing the neural radiance field, camera poses,
and the discriminator network. GNeRF demonstrates the potential of using GANs
to learn high-quality neural radiance fields in scenarios where camera poses
are unknown, making it suitable for applications like novel view synthesis and
3D reconstruction.

    * –

In-Place Scene Labelling and Understanding with Implicit Scene Representation
[265] presents an application of NeRF that focuses on in-place scene labeling
and understanding using implicit scene representations. The authors propose a
method that learns a joint representation of geometry, appearance, and
semantic information using a neural radiance field. The training paradigm
involves optimizing the neural network using a combination of rendering loss,
depth consistency loss, and a semantic loss. This method demonstrates the
potential of using NeRF for scene understanding tasks, such as semantic
segmentation and object recognition, in addition to view synthesis.

    * –

CAMPARI (Camera-Aware Multiplane Radiance Fields) [159] is an improvement over
NeRF that introduces a camera-aware decomposed generative framework for
learning neural radiance fields. The authors propose a method that decomposes
the input images into depth layers and learns a separate neural radiance field
for each layer. The training paradigm involves optimizing the neural networks
using a combination of rendering loss, depth consistency loss, and a cycle
consistency loss. CAMPARI enables more efficient rendering and better handling
of occlusions, making it suitable for applications like novel view synthesis
and relighting.

    * –

NeRF-VAE (Neural Radiance Fields - Variational Autoencoder) [113] is an
extension of NeRF that combines the geometry-aware representation of NeRF with
the generative capabilities of VAEs. The authors propose a method that learns
a latent space for 3D scenes, which can be used for generating novel scenes
and interpolating between existing ones. The training paradigm involves
optimizing the neural network using a combination of rendering loss, KL-
divergence loss, and a reconstruction loss. NeRF-VAE demonstrates the
potential of using NeRF for generative modeling tasks, such as 3D scene
generation and interpolation.

    * –

Unconstrained Scene Generation with Locally Conditioned Radiance Fields [50]
presents an extension of NeRF that focuses on unconstrained scene generation
using locally conditioned radiance fields. The authors propose a method that
conditions the neural radiance field on local spatial information, allowing it
to generate complex and diverse scenes. The training paradigm involves
optimizing the neural network using a combination of rendering loss and a
perceptual loss. This method demonstrates the potential of using NeRF for
generative modeling tasks, such as scene synthesis and object generation, in
an unconstrained setting.

    * –

Putting NeRF on a Diet [95] presents an improvement over NeRF that focuses on
semantically consistent few-shot view synthesis. The authors propose a method
that leverages semantic information to guide the neural radiance field
learning process, enabling it to synthesize novel views with as few as one
input image. The training paradigm involves optimizing the neural network
using a combination of rendering loss, semantic consistency loss, and a cycle
consistency loss. This method demonstrates the potential of using NeRF for
few-shot view synthesis tasks while maintaining semantic consistency, making
it suitable for applications like virtual reality and film production.

    * –

Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation [203]
presents an application of NeRF that focuses on decomposing 3D scenes into
objects using unsupervised volume segmentation. The authors propose a method
that learns to segment the scene into different objects by leveraging the
geometric information provided by the neural radiance field. The training
paradigm involves optimizing the neural network using a combination of
rendering loss and a segmentation loss. This method demonstrates the potential
of using NeRF for unsupervised object segmentation tasks, which can be useful
for applications like 3D object recognition and scene understanding.

    * –

Convolutional Neural Opacity Radiance Fields [138] is an improvement over NeRF
that introduces a convolutional architecture for learning neural radiance
fields. The authors propose a method that replaces the fully connected layers
in NeRF with convolutional layers, allowing the model to leverage local
spatial information and achieve better performance. The training paradigm
involves optimizing the neural network using a rendering loss that measures
the difference between the ground truth and the predicted images. This method
demonstrates the potential of using convolutional architectures in NeRF,
leading to more efficient and accurate rendering, making it suitable for
applications like novel view synthesis and relighting.

    * –

NARF (Neural Articulated Radiance Fields) [162] is an extension of NeRF that
focuses on modeling articulated objects, such as humans or animals, by
leveraging the articulated structure of the object. The authors propose a
method that combines NeRF with a kinematic model to represent the geometry and
appearance of articulated objects in a coherent and continuous manner. The
training paradigm involves optimizing the neural network using a combination
of rendering loss and a rigidity loss that encourages a consistent geometry
across poses. NARF demonstrates the potential of using NeRF for modeling
articulated objects, making it suitable for applications like character
animation and motion capture.

    * –

Neural RGB-D Scene Reconstruction [8] is an application of NeRF that focuses
on 3D scene reconstruction from RGB-D (color and depth) images. The authors
propose a method that leverages the depth information available in RGB-D
images to learn a more accurate and efficient neural radiance field. The
training paradigm involves optimizing the neural network using a rendering
loss that measures the difference between the ground truth and the predicted
images. This method demonstrates the potential of using NeRF for 3D scene
reconstruction tasks with RGB-D images, making it suitable for applications
like robotics and augmented reality.

    * –

BARF (Bundle-Adjusting Neural Radiance Fields) [129] is an improvement over
NeRF that introduces bundle adjustment, a technique commonly used in
structure-from-motion, for refining camera poses and optimizing the neural
radiance field simultaneously. The authors propose a method that jointly
optimizes the camera poses and the neural radiance field, leading to more
accurate and consistent scene reconstruction. The training paradigm involves
optimizing the neural network and camera poses using a combination of
rendering loss and a bundle adjustment loss. BARF demonstrates the potential
of incorporating bundle adjustment in NeRF, making it suitable for
applications like 3D reconstruction and novel view synthesis.

    * –

SRF (Stereo Radiance Fields) [33] is an extension of NeRF that focuses on
learning view synthesis for sparse views of novel scenes. The authors propose
a method that leverages stereo pairs of input images to learn a more accurate
and efficient neural radiance field. The training paradigm involves optimizing
the neural network using a combination of rendering loss, depth consistency
loss, and a stereo consistency loss that encourages consistency between the
stereo pairs. SRF demonstrates the potential of using NeRF for view synthesis
tasks with sparse input views, making it suitable for applications like
virtual reality and film production.

    * –

FiG-NeRF (Figure-Grounded Neural Radiance Fields) [238] is an extension of
NeRF that focuses on 3D object category modeling by learning a figure-ground
decomposition of the scene. The authors propose a method that learns separate
neural radiance fields for the object and the background, enabling the model
to focus on the object’s geometry and appearance. The training paradigm
involves optimizing the neural network using a combination of rendering loss
and a figure-ground consistency loss that encourages a consistent
decomposition of the scene. FiG-NeRF demonstrates the potential of using NeRF
for 3D object category modeling tasks, making it suitable for applications
like object recognition and scene understanding.

    * –

Shadow Neural Radiance Fields [49] is an application of NeRF that focuses on
multi-view satellite photogrammetry, specifically on reconstructing 3D
geometry and appearance of Earth’s surface from satellite images. The authors
propose a method that extends NeRF to account for the complex illumination
effects caused by shadows and atmospheric scattering in satellite images. The
training paradigm involves optimizing the neural network using a combination
of rendering loss, depth consistency loss, and a shadow consistency loss. This
method demonstrates the potential of using NeRF for satellite photogrammetry
tasks, making it suitable for applications like Earth observation and remote
sensing.

    * –

UNISURF (Unifying Neural Implicit Surfaces and Radiance Fields) [163] is an
improvement over NeRF that aims to unify neural implicit surface
representations and neural radiance fields for multi-view reconstruction. The
authors propose a method that combines the advantages of both representations,
enabling the model to learn geometry and appearance more accurately and
efficiently. The training paradigm involves optimizing the neural network
using a combination of rendering loss, depth consistency loss, and a surface
consistency loss. UNISURF demonstrates the potential of combining neural
implicit surfaces and radiance fields for multi-view reconstruction tasks,
making it suitable for applications like 3D reconstruction and novel view
synthesis.

    * –

Editable Free-Viewpoint Video using a Layered Neural Representation [256]
presents an application of NeRF that focuses on editable free-viewpoint video
using a layered neural representation. The authors propose a method that
decomposes the scene into a set of layered neural radiance fields, enabling
users to edit the content of the video by modifying individual layers. The
training paradigm involves optimizing the neural network using a combination
of rendering loss, depth consistency loss, and a layer consistency loss. This
method demonstrates the potential of using NeRF for editable free-viewpoint
video tasks, making it suitable for applications like video editing and
content creation.

    * –

Animatable Neural Radiance Fields [169] is an extension of NeRF that focuses
on modeling dynamic human bodies by incorporating a parametric human body
model into the neural radiance field framework. The authors propose a method
that learns a neural radiance field conditioned on the pose and shape
parameters of the human body model, enabling the generation of realistic
animations. The training paradigm involves optimizing the neural network using
a combination of rendering loss, pose consistency loss, and a shape
consistency loss. This method demonstrates the potential of using NeRF for
animating human bodies, making it suitable for applications like character
animation and motion capture.

    * –

Editing Conditional Radiance Fields [133] presents an extension of NeRF that
focuses on editing conditional radiance fields, allowing users to edit the
appearance and geometry of a scene. The authors propose a method that
conditions the neural radiance field on an editing vector, enabling the model
to generate edited versions of the scene by modifying the vector. The training
paradigm involves optimizing the neural network using a combination of
rendering loss, depth consistency loss, and an editing consistency loss. This
method demonstrates the potential of using NeRF for editing tasks, making it
suitable for applications like content creation, virtual reality, and film
production.

    * –

Dynamic View Synthesis from Dynamic Monocular Videos [61] is an application of
NeRF that focuses on dynamic view synthesis from dynamic monocular videos. The
authors propose a method that extends NeRF to handle dynamic scenes by
conditioning the neural radiance field on time, allowing the model to learn a
time-varying geometry and appearance. The training paradigm involves
optimizing the neural network using a combination of rendering loss, temporal
consistency loss, and a geometric consistency loss. This method demonstrates
the potential of using NeRF for dynamic view synthesis tasks, making it
suitable for applications like video editing, virtual reality, and film
production.

    * –

Recursive-NeRF [244] is an improvement over NeRF that aims to make the model
more efficient by using a dynamically growing neural radiance field. The
authors propose a method that recursively refines the radiance field by
training smaller MLPs (Multilayer Perceptrons) on top of the existing ones,
allowing the model to adapt to new views with minimal additional computation.
The training paradigm involves optimizing the neural network using a rendering
loss that measures the difference between the ground truth and the predicted
images. Recursive-NeRF demonstrates the potential of using a dynamically
growing NeRF for efficient view synthesis, making it suitable for real-time
applications.

    * –

Stylizing 3D Scene via Implicit Representation and HyperNetwork [32] presents
an extension of NeRF that focuses on stylizing 3D scenes using implicit
representation and hypernetworks. The authors propose a method that combines
NeRF with a hypernetwork, enabling the model to generate stylized versions of
the scene by learning a style-specific radiance field. The training paradigm
involves optimizing the neural network using a combination of rendering loss,
style consistency loss, and a content consistency loss. This method
demonstrates the potential of using NeRF for stylizing 3D scenes, making it
suitable for applications like content creation, virtual reality, and film
production.

    * –

NeRFactor [259] is an improvement over NeRF that focuses on factorizing shape
and reflectance under an unknown illumination. The authors propose a method
that extends NeRF to learn a separate neural representation for geometry,
reflectance, and illumination, allowing the model to disentangle these factors
and reconstruct the scene more accurately. The training paradigm involves
optimizing the neural network using a combination of rendering loss,
reflectance consistency loss, and a geometry consistency loss. NeRFactor
demonstrates the potential of using NeRF for disentangling shape, reflectance,
and illumination, making it suitable for applications like 3D reconstruction
and relighting.

    * –

Neural Actor [132] is an extension of NeRF that focuses on neural free-view
synthesis of human actors with pose control. The authors propose a method that
conditions the neural radiance field on the pose of the actor, allowing the
model to generate novel views of the actor in different poses. The training
paradigm involves optimizing the neural network using a combination of
rendering loss, pose consistency loss, and a geometric consistency loss.
Neural Actor demonstrates the potential of using NeRF for synthesizing human
actors with pose control, making it suitable for applications like character
animation, virtual reality, and film production.

    * –

Light Field Networks [195] is an improvement over NeRF that aims to speed up
rendering by using a single network evaluation. The authors propose a method
that learns a global light field representation, which can be sampled to
generate novel views without querying the network for each ray. The training
paradigm involves optimizing the neural network using a rendering loss that
measures the difference between the ground truth and the predicted images.
Light Field Networks demonstrates the potential of using a global light field
representation for efficient rendering, making it suitable for real-time
applications.

    * –

NeRF in detail (Learning to Sample for View Synthesis) [3] is an improvement
over NeRF that focuses on learning to sample rays more effectively for view
synthesis. The authors propose a method that combines NeRF with a sampling
network, which learns to predict importance sampling weights along each ray to
improve the efficiency of rendering. The training paradigm involves optimizing
the neural network using a rendering loss that measures the difference between
the ground truth and the predicted images. NeRF in detail demonstrates the
potential of using importance sampling to enhance the efficiency of NeRF,
making it suitable for real-time view synthesis tasks.

    * –

NeuS (Neural Surfaces) [222] is an improvement over NeRF that aims to learn
neural implicit surfaces by volume rendering for multi-view reconstruction.
The authors propose a method that combines volume rendering with implicit
surface representations, enabling the model to learn geometry and appearance
more accurately and efficiently. The training paradigm involves optimizing the
neural network using a combination of rendering loss, depth consistency loss,
and surface consistency loss. NeuS demonstrates the potential of using volume
rendering with implicit surface representations for multi-view reconstruction
tasks, making it suitable for applications like 3D reconstruction and novel
view synthesis.

    * –

Volume Rendering of Neural Implicit Surfaces [246] presents an extension of
NeRF that focuses on volume rendering of neural implicit surfaces. The authors
propose a method that combines NeRF with a voxel-based representation,
enabling the model to learn a more efficient and accurate representation of
the scene. The training paradigm involves optimizing the neural network using
a rendering loss that measures the difference between the ground truth and the
predicted images. Volume Rendering of Neural Implicit Surfaces demonstrates
the potential of using voxel-based representations with NeRF, making it
suitable for applications like 3D reconstruction and novel view synthesis.

    * –

HyperNeRF [167] is an improvement over NeRF that focuses on learning higher-
dimensional representations for topologically varying neural radiance fields.
The authors propose a method that extends NeRF to handle higher-dimensional
input spaces, allowing the model to represent more complex and topologically
varying scenes. The training paradigm involves optimizing the neural network
using a rendering loss that measures the difference between the ground truth
and the predicted images. HyperNeRF demonstrates the potential of using
higher-dimensional representations to enhance the expressiveness of NeRF,
making it suitable for modeling complex scenes and objects.

    * –

Animatable NeRFs from Monocular RGB Videos [26] presents an application of
NeRF that focuses on animatable neural radiance fields from monocular RGB
videos. The authors propose a method that extends NeRF to learn a time-varying
radiance field from monocular RGB videos, allowing the model to generate novel
views of dynamic scenes. The training paradigm involves optimizing the neural
network using a combination of rendering loss, temporal consistency loss, and
a geometric consistency loss. This method demonstrates the potential of using
NeRF for animating dynamic scenes, making it suitable for applications like
video editing, virtual reality, and film production.

    * –

Fast Training of Neural Lumigraph Representation using Meta Learning [13]
presents an improvement over NeRF that focuses on fast training of neural
lumigraph representation using meta-learning. The authors propose a method
that combines NeRF with meta-learning techniques, enabling the model to learn
a neural lumigraph representation more quickly. The training paradigm involves
optimizing the neural network using a rendering loss that measures the
difference between the ground truth and the predicted images, along with a
meta-learning loss that measures the generalization performance of the model.
This method demonstrates the potential of using meta-learning to speed up the
training of NeRF, making it suitable for real-time applications.

    * –

Depth-Supervised NeRF [47] is an improvement over NeRF that focuses on using
fewer views and faster training without sacrificing quality. The authors
propose a method that incorporates depth supervision into the NeRF training
process, allowing the model to learn more accurate and efficient
representations with fewer input views. The training paradigm involves
optimizing the neural network using a combination of rendering loss and depth
consistency loss. Depth-Supervised NeRF demonstrates the potential of using
depth supervision to enhance the efficiency of NeRF, making it suitable for
applications with limited input views.

    * –

3D Neural Scene Representation for VisuoMotor Control [126] presents an
application of NeRF that focuses on 3D neural scene representation for
visuomotor control. The authors propose a method that extends NeRF to learn a
compact and efficient 3D scene representation, which can be used for
visuomotor control tasks like robotic manipulation and navigation. The
training paradigm involves optimizing the neural network using a combination
of rendering loss and a task-specific loss that measures the performance of
the visuomotor control policy. This method demonstrates the potential of using
NeRF for visuomotor control applications, making it suitable for robotics and
autonomous systems.

    * –

Unsupervised Discovery of Object Radiance Fields [251] presents an extension
of NeRF that focuses on unsupervised discovery of object radiance fields. The
authors propose a method that learns object-centric radiance fields without
any supervision, allowing the model to discover and represent objects in a
scene in an unsupervised manner. The training paradigm involves optimizing the
neural network using a combination of rendering loss, object consistency loss,
and a scene consistency loss. Unsupervised Discovery of Object Radiance Fields
demonstrates the potential of using NeRF for unsupervised object discovery and
representation, making it suitable for applications like 3D reconstruction and
scene understanding.

    * –

Neural Rays for Occlusion-aware Image-based Rendering [134] is an improvement
over NeRF that focuses on occlusion-aware image-based rendering. The authors
propose a method called Neural Rays, which extends NeRF by explicitly modeling
occlusion in the radiance field. This enables the model to generate more
accurate and realistic novel views, especially in scenes with complex
occlusions. The training paradigm involves optimizing the neural network using
a combination of rendering loss and an occlusion-aware loss. Neural Rays
demonstrate the potential of incorporating occlusion information into NeRF,
making it suitable for applications like image-based rendering and 3D
reconstruction.

    * –

Differentiable Surface Rendering via Non-Differentiable Sampling [37] presents
an improvement over NeRF that focuses on differentiable surface rendering
using non-differentiable sampling. The authors propose a method that combines
NeRF with a differentiable rendering framework that can handle non-
differentiable sampling operations, enabling the model to learn more accurate
and efficient representations of surfaces. The training paradigm involves
optimizing the neural network using a combination of rendering loss and a
surface consistency loss. Differentiable Surface Rendering via Non-
Differentiable Sampling demonstrates the potential of using differentiable
rendering with NeRF, making it suitable for applications like 3D
reconstruction and novel view synthesis.

    * –

FLAME-in-NeRF [5] is an application of NeRF that focuses on free-view face
animation. The authors propose a method that combines NeRF with the FLAME
model, a parametric face model that encodes facial shape, expression, and pose
information. This enables the generation of novel views of animated faces with
high-quality and realistic appearance. The training paradigm involves
optimizing the neural network using a rendering loss that measures the
difference between the ground truth and the predicted images. FLAME-in-NeRF
demonstrates the potential of using NeRF for face animation, making it
suitable for applications like virtual reality, video games, and film
production.

    * –

iButter [221] is an application of NeRF that focuses on generating interactive
bullet time effects for human free-viewpoint rendering. The authors propose a
method that extends NeRF to handle dynamic human scenes, enabling the model to
generate novel views of humans in motion with a bullet time effect. The
training paradigm involves optimizing the neural network using a combination
of rendering loss and a temporal consistency loss. iButter demonstrates the
potential of using NeRF for interactive bullet time generation, making it
suitable for applications like video editing, virtual reality, and film
production.

    * –

Self-Calibrating Neural Radiance Fields [97] presents an improvement over NeRF
that focuses on self-calibrating neural radiance fields. The authors propose a
method that extends NeRF by learning to self-calibrate camera parameters,
enabling the model to handle input data with noisy or incomplete camera
parameters. The training paradigm involves optimizing the neural network using
a combination of rendering loss and a calibration consistency loss. Self-
Calibrating Neural Radiance Fields demonstrate the potential of incorporating
self-calibration into NeRF, making it suitable for applications like 3D
reconstruction and novel view synthesis, especially when working with
imperfect input data.

    * –

NeRFingMVS [225] presents an improvement over NeRF that focuses on indoor
multi-view stereo. The authors propose a method called NeRFing MVS, which
extends NeRF by incorporating guidance from a multi-view stereo reconstruction
to optimize the neural radiance fields for indoor scenes. The training
paradigm involves optimizing the neural network using a combination of
rendering loss and a guidance loss from the multi-view stereo reconstruction.
NeRFing MVS demonstrates the potential of incorporating multi-view stereo
guidance into NeRF, making it suitable for applications like indoor scene
reconstruction and novel view synthesis.

    * –

CodeNeRF [96] is an extension of NeRF that focuses on disentangling neural
radiance fields for object categories. The authors propose a method that
learns disentangled representations of shape and appearance in neural radiance
fields, enabling the model to generate novel views of objects from different
categories with consistent appearance properties. The training paradigm
involves optimizing the neural network using a rendering loss that measures
the difference between the ground truth and the predicted images, along with a
disentanglement loss that encourages separation of shape and appearance
features. CodeNeRF demonstrates the potential of using disentangled
representations in NeRF, making it suitable for applications like object
synthesis and 3D modeling.

    * –

Learning Object-Compositional Neural Radiance Fields for Editable Scene
Rendering [242] presents an extension of NeRF that focuses on learning object-
compositional neural radiance fields for editable scene rendering. The authors
propose a method that learns neural radiance fields for individual objects and
composes them into a scene, enabling the model to generate novel views of
editable scenes. The training paradigm involves optimizing the neural network
using a combination of rendering loss, object consistency loss, and a scene
consistency loss. This method demonstrates the potential of using object-
compositional radiance fields in NeRF, making it suitable for applications
like scene editing and virtual reality.

    * –

Stochastic Neural Radiance Fields [191] is an extension of NeRF that focuses
on quantifying uncertainty in implicit 3D representations. The authors propose
a method that incorporates stochasticity into the neural radiance field,
enabling the model to represent and quantify uncertainty in its predictions.
The training paradigm involves optimizing the neural network using a
combination of rendering loss and a stochastic consistency loss that
encourages the model to capture uncertainty. Stochastic Neural Radiance Fields
demonstrate the potential of incorporating uncertainty quantification into
NeRF, making it suitable for applications like robust 3D reconstruction and
probabilistic rendering.

    * –

Neural Human Performer [117] is an application of NeRF that focuses on
learning generalizable radiance fields for human performance rendering. The
authors propose a method that extends NeRF to handle dynamic human
performances, enabling the model to generate novel views of humans in motion
with high-quality and realistic appearance. The training paradigm involves
optimizing the neural network using a combination of rendering loss and a
temporal consistency loss. Neural Human Performer demonstrates the potential
of using NeRF for human performance rendering, making it suitable for
applications like virtual reality, video games, and film production.

    * –

TöRF (Time-of-Flight Radiance Fields) [7] is an extension of NeRF that
incorporates Time-of-Flight (ToF) information to enable view synthesis for
dynamic scenes. The authors propose a method that combines neural radiance
fields with ToF depth measurements, making it possible to generate novel views
of dynamic scenes with moving objects. The training paradigm involves
optimizing the neural network using a combination of rendering loss and a ToF
depth consistency loss. TöRF demonstrates the potential of using ToF
information in NeRF, making it suitable for applications like robotics,
autonomous vehicles, and augmented reality.

    * –

Vision-Only Robot Navigation in a Neural Radiance World [1] presents an
application of NeRF for vision-only robot navigation. The authors propose a
method that leverages neural radiance fields to build a 3D representation of
the environment, allowing a robot to navigate using only visual information.
The training paradigm involves optimizing the neural network using a
combination of rendering loss and a navigation loss that encourages the model
to learn a useful representation for navigation. This approach demonstrates
the potential of using NeRF for vision-only robot navigation, making it
suitable for applications like autonomous robots and drone navigation.

    * –

Neural Radiance Fields Approach to Deep Multi-View Photometric Stereo [106]
presents an extension of NeRF that focuses on deep multi-view photometric
stereo. The authors propose a method that combines neural radiance fields with
multi-view photometric stereo to reconstruct the shape and appearance of
objects in a scene. The training paradigm involves optimizing the neural
network using a combination of rendering loss and a photometric consistency
loss. This method demonstrates the potential of using NeRF for photometric
stereo, making it suitable for applications like 3D reconstruction and virtual
reality.

    * –

LENS (Localization Enhanced by NeRF Synthesis) [150] is an application of NeRF
that focuses on improving localization tasks using synthesized views. The
authors propose a method that leverages NeRF to generate synthetic views that
help in refining the localization estimates. The training paradigm involves
optimizing the neural network using a combination of rendering loss and a
localization loss that encourages the model to generate useful views for
localization. LENS demonstrates the potential of using NeRF for enhancing
localization tasks, making it suitable for applications like robotics,
autonomous vehicles, and augmented reality.

    * –

NeRS (Neural Reflectance Surfaces) [255] is an extension of NeRF that focuses
on sparse-view 3D reconstruction in the wild. The authors propose a method
that combines neural radiance fields with an additional reflectance component,
enabling the model to handle challenging real-world scenes with sparse views.
The training paradigm involves optimizing the neural network using a
combination of rendering loss and a reflectance consistency loss. NeRS
demonstrates the potential of using NeRF for sparse-view 3D reconstruction,
making it suitable for applications like 3D modeling and virtual reality.

    * –

StyleNeRF [71] is an extension of NeRF that incorporates style-based
generators for high-resolution image synthesis. The authors propose a method
that combines neural radiance fields with a style-based generator, enabling
high-quality 3D-aware image synthesis. The training paradigm involves
optimizing the neural network using a combination of rendering loss and a
style consistency loss. StyleNeRF demonstrates the potential of using NeRF for
high-resolution image synthesis, making it suitable for applications like
virtual reality, 3D modeling, and artistic image generation.

    * –

CIPS-3D [267] is an extension of NeRF that focuses on generating images using
conditionally-independent pixel synthesis. The authors propose a method that
combines neural radiance fields with a GAN-based generator, enabling 3D-aware
image synthesis. The training paradigm involves optimizing the neural network
using a combination of rendering loss and a GAN loss. CIPS-3D demonstrates the
potential of using NeRF for 3D-aware image generation, making it suitable for
applications like 3D modeling, virtual reality, and artistic image generation.

    * –

H-NeRF [239] is an extension of NeRF that focuses on rendering and temporal
reconstruction of humans in motion. The authors propose a method that combines
neural radiance fields with temporal information, enabling the model to
reconstruct and render dynamic human motions. The training paradigm involves
optimizing the neural network using a combination of rendering loss and a
temporal consistency loss. H-NeRF demonstrates the potential of using NeRF for
human motion reconstruction and rendering, making it suitable for applications
like animation, virtual reality, and video game development.

    * –

Dex-NeRF [89] is an application of NeRF that focuses on grasping transparent
objects. The authors propose a method that leverages NeRF to generate a 3D
representation of transparent objects, allowing a robot to grasp them using
visual information. The training paradigm involves optimizing the neural
network using a combination of rendering loss and a grasping loss that
encourages the model to learn a useful representation for grasping. Dex-NeRF
demonstrates the potential of using NeRF for grasping transparent objects,
making it suitable for applications like robotics, automation, and
manufacturing.

    * –

Neural-PIL [18] is an extension of NeRF that focuses on reflectance
decomposition using pre-integrated lighting. The authors propose a method that
combines neural radiance fields with a pre-integrated lighting model, enabling
the decomposition of reflectance properties of the scene. The training
paradigm involves optimizing the neural network using a combination of
rendering loss and a reflectance decomposition loss. Neural-PIL demonstrates
the potential of using NeRF for reflectance decomposition, making it suitable
for applications like 3D modeling, virtual reality, and computer graphics.

    * –

Template NeRF [72] is an extension of NeRF that models dense shape
correspondences from category-specific object images. The authors propose a
method that leverages a template neural radiance field to establish dense
shape correspondences across different instances of an object category. The
training paradigm involves optimizing the neural network using a combination
of rendering loss and a correspondence loss. Template NeRF is suitable for
applications like 3D modeling, computer graphics, and object recognition.

    * –

DIVeR (Deterministic Integration for Volumetric Rendering) [234] is an
improvement over NeRF, offering real-time and accurate neural radiance fields
with deterministic integration for volume rendering. The authors propose a
deterministic integration method that accelerates the rendering process while
maintaining accuracy. The training paradigm involves optimizing the neural
network using a rendering loss. DIVeR demonstrates the potential for real-time
volume rendering applications, such as virtual reality and 3D modeling.

    * –

DVGO (Direct Voxel Grid Optimization) [206] is an improvement over NeRF,
focusing on super-fast convergence for radiance field reconstruction. The
authors propose a direct voxel grid optimization method that accelerates the
convergence of the NeRF training process. The training paradigm involves
optimizing the neural network using a rendering loss. DVGO is suitable for
applications like 3D modeling, computer graphics, and real-time rendering.

    * –

Mip-NeRF 360 [11] is an extension of NeRF, focusing on unbounded anti-aliased
neural radiance fields. The authors propose a method that leverages mipmapping
to reduce aliasing artifacts in NeRF-based renderings. The training paradigm
involves optimizing the neural network using a rendering loss. Mip-NeRF 360 is
suitable for applications like virtual reality, computer graphics, and 3D
modeling, where high-quality renderings are essential.

    * –

VaxNeRF [112] is an improvement over NeRF that introduces voxel acceleration
for faster rendering. The authors propose a method that combines voxel-based
acceleration with neural radiance fields, resulting in faster rendering times
without sacrificing quality. The training paradigm involves optimizing the
neural network using a rendering loss. VaxNeRF is suitable for real-time
rendering applications, such as virtual reality and 3D modeling.

    * –

GeoNeRF [101] is an extension of NeRF that incorporates geometric priors for
better scene representation. The authors propose a method that integrates
geometric information into the NeRF framework, improving the quality and
accuracy of the rendered images. The training paradigm involves optimizing the
neural network using a combination of rendering loss and a geometric prior
loss. GeoNeRF is suitable for applications like 3D modeling, computer
graphics, and virtual reality.

    * –

NeRF in the Dark [146] is an extension of NeRF that focuses on high dynamic
range (HDR) view synthesis from noisy raw images. The authors propose a method
that leverages NeRF for rendering HDR images from raw, noisy input data. The
training paradigm involves optimizing the neural network using a combination
of rendering loss and a denoising loss. NeRF in the Dark is suitable for
applications like photography, computer graphics, and virtual reality, where
HDR rendering from noisy raw images is essential.

    * –

Deblur-NeRF [139] is an extension of NeRF that focuses on reconstructing
neural radiance fields from blurry images. The authors propose a method that
leverages NeRF to render sharp images from input data that is blurred due to
camera motion or other factors. The training paradigm involves optimizing the
neural network using a combination of rendering loss and a deblurring loss.
Deblur-NeRF is suitable for applications like photography, computer graphics,
and virtual reality, where rendering sharp images from blurry inputs is
important.

    * –

HDR-NeRF (High Dynamic Range Neural Radiance Fields) [88] is an extension of
NeRF, focusing on high dynamic range (HDR) neural radiance fields. The authors
propose a method that enables NeRF to handle HDR images by modeling both
radiance and color. The training paradigm involves optimizing the neural
network using a rendering loss. HDR-NeRF is suitable for applications like
photography, computer graphics, and virtual reality, where HDR rendering is
essential.

    * –

iLabel (Interactive Neural Scene Labelling) [266] is an application of NeRF
that focuses on interactive neural scene labeling. The authors propose an
interactive framework for labeling scenes with semantic information using a
neural radiance field. The training paradigm involves optimizing the neural
network using a combination of rendering loss and a semantic loss. iLabel is
suitable for applications like 3D scene understanding, computer graphics, and
virtual reality.

    * –

Urban Radiance Fields [179] is an application of NeRF that focuses on modeling
and rendering complex urban environments. The authors propose a method that
leverages NeRF to capture and render detailed urban scenes. The training
paradigm involves optimizing the neural network using a rendering loss. Urban
Radiance Fields is suitable for applications like urban planning, virtual
reality, and computer graphics.

    * –

NeRFReN (Neural Radiance Fields with ReflectioNs) [74] is an extension of NeRF
that incorporates reflections into the rendering process. The authors propose
a method that models reflections in neural radiance fields by incorporating a
reflection term in the NeRF formulation. The training paradigm involves
optimizing the neural network using a rendering loss. NeRFReN is suitable for
applications like computer graphics, virtual reality, and 3D modeling, where
realistic reflections are important.

    * –

Hallucinated Neural Radiance Fields in the Wild [31] is an improvement over
NeRF, focusing on rendering hallucinated neural radiance fields in the wild.
The authors propose a method that leverages NeRF to generate plausible
renderings of scenes with limited or no input data. The training paradigm
involves optimizing the neural network using a rendering loss. Hallucinated-
NeRF is suitable for applications like computer graphics, virtual reality, and
3D modeling, where input data may be scarce or unavailable.

    * –

NeuSample (Neural Sample Field for Efficient View Synthesis) [56] is an
improvement over NeRF that focuses on efficient view synthesis. The authors
propose a method that learns a neural sample field to guide the sampling
process, reducing the number of required samples for high-quality renderings.
The training paradigm involves optimizing the neural network using a rendering
loss. NeuSample is suitable for applications like virtual reality, computer
graphics, and real-time rendering.

    * –

RegNeRF (Regularized Neural Radiance Fields) [158] is an improvement over NeRF
that focuses on view synthesis from sparse inputs. The authors propose a
method that introduces regularization techniques to the NeRF framework to
improve its performance when synthesizing views from a limited number of input
images. The training paradigm involves optimizing the neural network using a
combination of rendering loss and regularization losses (e.g., smoothness and
sparsity losses). RegNeRF is suitable for applications like 3D modeling,
computer graphics, and virtual reality, where view synthesis from sparse input
data is a common challenge.

    * –

Zero-Shot Text-Guided Object Generation with Dream Fields (also known as Dream
Fields) [94] is an application of NeRF for text-guided object generation. The
authors propose a method that generates 3D objects conditioned on a given
textual description. The training paradigm involves optimizing a neural
network with a combination of rendering loss and textual conditioning loss.
Applications include 3D object generation, virtual world creation, and content
generation based on textual input. It extends NeRF by integrating textual
guidance into the framework.

    * –

Efficient NeRF with learned depth-guided sampling [130] is an improvement over
NeRF that focuses on speeding up the rendering process by incorporating
learned depth-guided sampling. The training paradigm involves optimizing the
neural network using a combination of rendering loss and a depth-guided
sampling loss. Applications include real-time rendering, computer graphics,
and virtual reality. This method improves NeRF’s efficiency by learning depth-
guided sampling strategies.

    * –

Learning Neural Light Fields with Ray-Space Embedding Networks [6] is an
extension of NeRF for learning neural light fields. The authors propose a
method that learns a compact representation of light fields using a ray-space
embedding network. The training paradigm involves optimizing the neural
network using a rendering loss. Applications include light field rendering,
computer graphics, and virtual reality. It extends NeRF by using a compact
representation for light fields.

    * –

Neural Head Avatars [67] is an application of NeRF that focuses on generating
3D head avatars from monocular RGB videos. The authors propose a method that
reconstructs neural radiance fields for human heads from single-view RGB video
input. The training paradigm involves optimizing the neural network using a
combination of rendering loss and additional losses for facial details.
Applications include virtual avatars, video conferencing, and virtual reality.
It extends NeRF by applying it to human head reconstruction from single-view
video input.

    * –

NeRF-SR [220] is an improvement over NeRF that focuses on generating high-
quality neural radiance fields using super-sampling. The authors propose a
method that employs super-sampling during the training and rendering processes
to improve the quality of the generated views. The training paradigm involves
optimizing the neural network using a rendering loss. Applications include
high-quality rendering, computer graphics, and virtual reality. It improves
NeRF’s rendering quality by incorporating super-sampling.

    * –

MoFaNeRF (Morphable Neural Radiance Fields) [272] is an extension of NeRF that
focuses on creating morphable neural radiance fields. The authors propose a
method that learns a single, unified neural radiance field capable of
representing multiple object instances within a category. The training
paradigm involves optimizing the neural network using a combination of
rendering loss and morphing loss. Applications include 3D object generation,
animation, and computer graphics. It extends NeRF by learning a single
representation capable of representing multiple object instances.

    * –

HumanNeRF (Generalizable Neural Human Radiance Field from Sparse inputs) [263]
is an extension of NeRF that focuses on reconstructing human shapes and
appearance from sparse input views. The authors propose a method that
leverages skeletal and semantic constraints to improve generalization and
efficiency. The training paradigm involves optimizing the neural network using
a combination of rendering loss and additional losses for skeletal and
semantic constraints. Applications include human shape reconstruction,
animation, and computer graphics. It extends NeRF by incorporating human-
specific constraints.

    * –

Dense Depth Priors for Neural Radiance Fields from Sparse Input Views [182] is
an improvement over NeRF that incorporates dense depth priors to handle sparse
input views. The authors propose a method that leverages dense depth
information as a prior to improve the quality and stability of the
reconstructed scene. The training paradigm involves optimizing the neural
network using a combination of rendering loss and depth prior loss.
Applications include scene reconstruction from sparse views and computer
graphics. It improves NeRF by incorporating dense depth priors.

    * –

CG-NeRF (Conditional Generative Neural Radiance Fields) [100] is an extension
of NeRF that introduces conditional generative neural radiance fields. The
authors propose a method that conditions the neural radiance field on external
information, such as object attributes or scene context. The training paradigm
involves optimizing the neural network using a combination of rendering loss
and conditioning loss. Applications include conditional scene generation and
computer graphics. It extends NeRF by introducing conditional generation
capabilities.

    * –

Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields
[217] is an extension of NeRF that focuses on modeling structured view-
dependent appearance. The authors propose a method that explicitly models
view-dependent reflections and refractions in the scene. The training paradigm
involves optimizing the neural network using a rendering loss and additional
losses for view-dependent appearance. Applications include rendering scenes
with complex reflections and refractions, and computer graphics. It extends
NeRF by modeling structured view-dependent appearance.

    * –

Geometry-Guided Progressive NeRF [28] is an improvement over NeRF that focuses
on efficient neural human rendering. The authors propose a method that
leverages geometric priors and a progressive training strategy to improve
rendering efficiency and generalization. The training paradigm involves
optimizing the neural network using a rendering loss and additional losses for
geometric constraints. Applications include human rendering, animation, and
computer graphics. It improves NeRF by incorporating geometric priors and a
progressive training strategy.

    * –

Deep Visual Constraints (Neural Implicit Models for Manipulation Planning from
Visual Input) [75] is an application of NeRF that focuses on manipulation
planning using visual input. The authors propose a method that learns an
implicit model of the scene and integrates it into a planning framework. The
training paradigm involves optimizing the neural network using a combination
of rendering loss and planning loss. Applications include robotic manipulation
planning, computer vision, and robotics. It extends NeRF by applying it to
manipulation planning from visual input.

    * –

CLIP-NeRF [219] is an extension of NeRF that combines textual and visual
information for scene manipulation. It leverages the CLIP (Contrastive
Language-Image Pretraining) model to condition the neural radiance fields on
both text and image inputs. The training paradigm involves optimizing the
neural network using a combination of rendering loss and CLIP loss.
Applications include text-and-image-based scene editing and computer graphics.
It extends NeRF by introducing text-and-image-driven manipulation
capabilities.

    * –

Neural Radiance Fields for Outdoor Scene Relighting [185] is an application of
NeRF that focuses on relighting outdoor scenes. The authors propose a method
that models outdoor lighting conditions and captures view-dependent
appearance, allowing for relighting under various conditions. The training
paradigm involves optimizing the neural network using a rendering loss and
additional losses for lighting and appearance. Applications include outdoor
scene rendering, virtual reality, and computer graphics. It applies NeRF to
outdoor scene relighting.

    * –

CityNeRF [237] is an extension of NeRF that scales the representation to
large-scale city scenes. The authors propose a method that leverages a
hierarchical structure and efficient rendering techniques to handle city-scale
data. The training paradigm involves optimizing the neural network using a
rendering loss and additional losses for spatial coherence and efficiency.
Applications include city-scale scene rendering, mapping, and urban planning.
It extends NeRF by scaling the representation to city-scale scenes.

    * –

HeadNeRF (A Real-Time NeRF-based Parametric Head Model) [86] is an application
of NeRF that focuses on real-time head modeling. The authors propose a method
that combines a parametric head model with a neural radiance field, enabling
real-time rendering and animation of head models. The training paradigm
involves optimizing the neural network using a rendering loss and additional
losses for parametric constraints. Applications include real-time head
rendering, animation, and computer graphics. It applies NeRF to real-time head
modeling.

    * –

GRAM (Generative Radiance Manifolds) [48] is an extension of NeRF that
introduces generative radiance manifolds for 3D-aware image generation. The
authors propose a method that leverages a manifold-based representation to
model scene geometry and appearance, enabling high-quality image synthesis.
The training paradigm involves optimizing the neural network using a
combination of rendering loss and manifold loss. Applications include 3D-aware
image generation and computer graphics. It extends NeRF by introducing
generative radiance manifolds.

    * –

Solving Inverse Problems with NeRFGANs [44] is an extension of NeRF that
combines the power of NeRF with GANs to solve inverse problems. The authors
propose a method that uses NeRF as a generator and a discriminator to optimize
the solution for various inverse problems. The training paradigm involves
optimizing the neural network using a combination of rendering loss and
adversarial loss. Applications include solving inverse problems in computer
graphics and computer vision. It extends NeRF by integrating it with GANs to
solve inverse problems.

    * –

HVTR (Hybrid Volumetric-Textural Rendering) is an extension of NeRF that
combines volumetric and textural rendering for human avatars. The authors
propose a method that jointly learns geometric, appearance, and textural
information, enabling more realistic rendering of human avatars. The training
paradigm involves optimizing the neural network using a combination of
rendering loss, texture loss, and geometric loss. Applications include human
avatar rendering, animation, and virtual reality. It extends NeRF by
introducing hybrid volumetric-textural rendering.

    * –

Mega-NeRF [216] is an extension of NeRF that scales the representation to
large-scale scenes, suitable for virtual fly-throughs. The authors propose a
method that leverages a hierarchical structure and efficient rendering
techniques to handle large-scale data. The training paradigm involves
optimizing the neural network using a rendering loss and additional losses for
spatial coherence and efficiency. Applications include large-scale scene
rendering, virtual reality, and computer graphics. It extends NeRF by scaling
the representation to large-scale scenes.

    * –

3D-Aware Image Synthesis via Learning Structural and Textural Representations
[240] is an application of NeRF that focuses on 3D-aware image synthesis. The
authors propose a method that learns structural and textural representations
to synthesize high-quality images while considering 3D geometry. The training
paradigm involves optimizing the neural network using a combination of
rendering loss, structure loss, and texture loss. Applications include
3D-aware image synthesis, computer graphics, and computer vision. It applies
NeRF to 3D-aware image synthesis.

    * –

Learning Implicit Body Representations from Double Diffusion Based Neural
Radiance Fields [245] is an extension of NeRF that focuses on learning
implicit body representations from double diffusion-based neural radiance
fields. The authors propose a method that leverages a double diffusion process
to capture fine-scale details and appearance information. The training
paradigm involves optimizing the neural network using a combination of
rendering loss, diffusion loss, and additional losses for geometry and
appearance. Applications include human body modeling, animation, and virtual
reality. It extends NeRF by introducing double diffusion-based neural radiance
fields.

    * –

BANMo (Building Animatable 3D Neural Models) [243] is an application of NeRF
that focuses on building animatable 3D neural models from casual videos. The
authors propose a method that leverages spatiotemporal information to learn
animatable neural models that can be applied to a variety of video inputs. The
training paradigm involves optimizing the neural network using a combination
of rendering loss, temporal coherence loss, and additional losses for
animation and appearance. Applications include video-based animation, computer
graphics, and virtual reality. It applies NeRF to animatable 3D neural models.

    * –

InfoNeRF (Ray Entropy Minimization for Few-Shot Neural Volume Rendering) [108]
is an improvement over NeRF that focuses on few-shot neural volume rendering.
The authors propose a method that minimizes ray entropy to improve the quality
of rendered images when trained with limited input views. The training
paradigm involves optimizing the neural network using a combination of
rendering loss and ray entropy loss. Applications include few-shot neural
volume rendering, computer graphics, and computer vision. It improves NeRF by
introducing ray entropy minimization for few-shot rendering. (Work in
Progress…)

  * •

Research in 2022: This subsection briefly discusses the research articles in
the last year. (Work in Progress…)

  * •

Research in 2023: This subsection briefly discusses the research articles in
this year till the month of April. (Work in Progress…)

(Work in Progress…)

##  5 Supporting Datasets

This section discusses the datasets that have been generated or directly used
for different types of tasks and evaluation purposes for a comprehensive
review of Radiance Fields and similar research ideas that are discussed later
in this survey. These comprise of the following.

  * •

LLFF (Light Field for the Language of Light and Form) dataset provides the
real-world scenes at 4K ultra-high resolution of training views [147]. This
dataset comprises of 8 forward-facing scenes with each scene having views
between 20 views and 60 views. These scenes were ($4032\times 3024$) while
NeRF-based methods use scenes with resolution ($1008\times 756$) for training
the MLPs and rendering (inference) from them. Further, NeRF-based methods also
use COLMAP [186, 187] for estimating the corresponding camera poses.

This dataset has been used for the following research articles:

    * –

Original NeRF (Neural Radiance Fields) [148]

    * –

NeRF-W (Neural Radiance Fields in the Wild) [140]

  * •

Tanks & Temples dataset was introduced by Knapitsch et al [110] in 2017. It is
widely used benchmark for evaluating 3D reconstruction algorithms. The dataset
consists of a diverse set of High-Resolution RGB-D (color and depth) scans of
light-scale indoor and outdoor scenes, captured using a high-quality
structured light scanner. The scenes in the dataset are categorized based on
“Tanks” (for indoor scenes), and “Temples” (for outdoor scenes). The dataset
provides ground truth 3D models of the scenes, which are reconstructed from
the RGB-D scans using a state-of-the-art 3D reconstruction pipeline. These
ground truth models can be used to quantitatively evaluate the performance of