11institutetext: University of Southern California, Los Angeles CA 90007, USA
https://www.usc.edu/ 22institutetext: 22email<EMAIL_ADDRESS>
# Neural Radiance Fields: Past, Present, and Future

Ansh Mittal 11 0000-0001-6648-6064

###### Abstract

The various aspects like modeling and interpreting 3D environments and
surroundings have enticed humans to progress their research in 3D Computer
Vision, Computer Graphics, and Machine Learning. An attempt made by Mildenhall
et al in their paper about NeRFs (Neural Radiance Fields) led to a boom in
Computer Graphics, Robotics, Computer Vision, and the possible scope of High-
Resolution Low-Storage Augmented Reality and Virtual Reality-based 3D models
have gained traction from res with more than 500 preprints related to NeRFs
published. This paper serves as a bridge for people starting to study these
fields by building on the basics of Mathematics, Geometry, Computer Vision,
and Computer Graphics to the difficulties encountered in Implicit
Representations at the intersection of all these disciplines. This survey
provides the history of rendering, Implicit Learning, and NeRFs, the
progression of research on NeRFs, and the potential applications and
implications of NeRFs in today’s world. In doing so, this survey categorizes
all the NeRF-related research in terms of the datasets used, objective
functions, applications solved, and evaluation criteria for these
applications.

###### Keywords:

NeRFs (Neural Radiance Fields) Rendering Computer Vision Deep Learning
Computer Graphics Robotics Augmented Reality Virtual Reality Implicit Machine
Learning

##  1 Introduction

Recently, implicit representation functions and machine learning have pervaded
through different applications and problem statements unsolvable in
traditional machine learning. And applications related to View Synthesis, 3D
Rendering, and Modeling have also started to leverage Radiance Fields (also
known as NeRFs). The preprints papers in this domain have been rising
exponentially with applications in different disciplines like 3D Computer
Vision, Astronomy, Climate change, and many other areas of study that require
images/3D geometry of locations. Although the work in Radiance Fields started
as a means to offer better View Synthesis, after the development in the
literature and GPU technology, it has permeated other fields such as Image
Reconstruction, Super Resolution, Pose Estimation, 3D Aware Image Synthesis,
Depth Estimation, Image Generation, 3D Reconstruction, Neural Rendering, and
other computer vision related problems. So, this survey aims to
comprehensively categorize the models, extensions, and applications of NeRFs
in terms of datasets, objective functions used, and problem statements
tackled, evaluation measures adopted while discussing the techniques and key
concepts associated with the field of NeRFs (Neural Radiance Fields) in the
research literature. Despite the comprehensiveness of surveys related to
Differentiable Rendering, Neural Radiance Fields, and Differential Geometry,
they fail to cater to someone starting research in this field. So, this survey
is able to bridge that gap while also discussing the history of how NeRF came
to be.

This research paper make the following important contributions to the already
existing research literature (such as Gao et al [63]).

  1. 1.

This paper can be used by a layman to understand basic concepts of
mathematics, computer graphics, computer vision, geometry (differential
geometry), etc.

  2. 2.

It makes a distinct classification based on the Loss Functions, Evaluation
metrics, Applications, and by the year it became online (on arxiv or other
sources on the Internet)

  3. 3.

It comprehensively tries to discuss the development of Neural Radiance Fields
and how Fourier Features can be used to model high-frequency function for
rendering [209, 147, 194]

  4. 4.

Further, based on the past and the papers published in 2020–2023, this paper
tries to trace out a path of future development of Neural Radiance Fields and
Neural Rendering

The §1 presents a brief introduction that this paper aims to resolve i.e. to
delineate a classification for NeRF models and discuss about its history and
future. The remainder of this paper has been organized as follow. §2 discusses
the fundamental and basic concepts and principles of interest in relation to
the research related to Neural Rendering and Novel View Synthesis. §3
discusses the various neural architectures before the conceptions of NeRFs. It
further discusses various surveys that have been put forward in this domain
and the need for this survey. §4 discusses the chronological description of
the models, extensions, and improvements over Neural Radiance Fields. §5
briefly discusses the datasets that have been leveraged till now in the field
of NeRFs. Further, these are also then taxonomized according to different
objectives (discussed in §6), different evaluation metrics discussed in §7),
and different applications (discussed in §8). All sections from §5–8 discuss
the research in the past couple of years in the field of NeRFs (limited till
December 2022). Finally, the future prospects of this family of models is
explored in §9. §10 concludes the paper and discusses some future prospects
for this research.

##  2 Basic Principles and Concepts

This section defines the concepts that should be helpful to the reader when
considering the possible objective functions and applications that Radiance
Fields tries to solve. This is by no means a comprehensive list of concepts
and the reader is advised to search on his own. For instance, Voxel Grids,
Octree, Triangular meshes, and Radiance Fields (from 2020 onwards) are some of
the ways that are used to store and represent 3D geometries.
These concepts can be divided into the basics of Computer Graphics and
Computer Vision. Since, NeRFs combine the fields of Computer Graphics,
Computer Vision, and Signal Processing, it is important to understand all
these concepts for better understanding the rest of the literature.

###  2.1 Computer Graphics

This section briefly describes some terms in Computer Graphics and that are
going to be helpful in reading the remainder of this article.

  * •

Pixel: In computer graphics, a pixel (short for ”picture element”) is the
smallest unit of a digital image. It represents a single point in a 2D grid of
square or rectangular cells called a raster image. Each pixel contains
information about its color and brightness, typically represented using binary
values. The combination of millions of pixels arranged in a specific pattern
creates a digital image that can be displayed on a screen or printed on paper.
In computer vision, a pixel is also the smallest unit of a digital image, but
its meaning is slightly different. In this context, a pixel is a mathematical
representation of an image that enables algorithms to analyze and process
visual information. Each pixel is assigned a numerical value based on its
intensity, which can range from 0 (black) to 255 (white) in grayscale images.
In color images, each pixel is represented by three or four values that define
its red, green, and blue color components, and optionally its alpha channel
for transparency.
In both computer graphics and computer vision, pixels are fundamental to the
creation and processing of digital images.

  * •

Resel: In computer graphics research, a resel (short for ”resolution element”)
is a unit of measurement used to quantify the resolution of an image. It is a
mathematical construct that represents the smallest distinguishable element in
an image, based on the size of the image’s point spread function (PSF).
The PSF describes how a point of light in the scene is blurred by the imaging
system before being recorded by a camera or other sensor. The resel is defined
as the area of the PSF that contains 50% of the total energy of the point of
light. In other words, it represents the smallest area in the image that can
be resolved as a distinct feature.
The resel is a useful metric in computer graphics research because it can be
used to compare the resolution of different imaging systems, or to evaluate
the effectiveness of image processing algorithms in enhancing or preserving
image detail. It is also used in fields such as microscopy and astronomy to
quantify the resolution of optical systems.

  * •

Voxel is a commonplace term used in 3D Computer Graphics and 3D Computer
Vision. It represents a discrete value on a grid in 3D space. It is analogous
to Pixel (in 2D bitmap space). Voxels don’t have their position information
encoded but a voxel is rendered based on relative position to other voxels. A
voxel has heavy use-case in GIS systems [34], Healthcare systems based on
Computer Vision [65, 60, 199, 40, 149], and Autonomous Vehicles [125, 41, 39,
30]. These can be used to form another data structure which is discussed
further.

  * •

Tixel or a tactile pixel is smallest measuring/transmitting element of a
tactile element of a tactile matrix. This is one of the fundamental units for
Haptic Technologies [104]. Here, the tactile matrix is a machine-readable
system, which transmits information from surface to the receiver that is the
exoskeleton interface for tactile capturing.

  * •

Texel or Texture element or Texture equivalent of a Pixel, is the most
fundamental measuring unit of Texture. Textures have an array of texels (much
like our computer images which have arrays of pixels). They are image regions
regions obtained through simple procedures like Thresholding and the midpoint
of centroids of two surrounding texels can be used to define the spatial
relationship between two texels in a texture.

  * •

Image Sharpening refers to techniques that highlight the edges and finer
details in an image. It is widely used as a preprocessing step for object
classification. This is done to increase local contrast and sharpening image.
In essence, it works by adding a high-pass filtered version of original image
to the image again. It is one of the most commonly used process for sharpening
one-dimensional signals and is called unsharp masking. Since, an image can be
considered a 2D-signal, hence, this operation can also be used for images.
This operation can also be represented by the eq 1

$S_{i,j}=\hat{x_{i,j}}+\lambda\cdot\textrm{F}(\hat{x_{i,j}})$ (1)

where $\hat{x_{i,j}}$ is the original pixel values for the image at the
coordinates $(i,j)$, $\textrm{F}(\cdot)$ is the high-pass filter for the
image, $\lambda$ is the tuning parameter greater than or equal to zero,
whereas the RHS represents the sharpened pixel for the image at the
coordinates $(i,j)$. $\lambda$ decides the desirable sharpness of the image.
In case of RGB images, $x_{i,j}$, $\lambda$, and $S_{i,j}$ become vectors with
3D.

  * •

Data Storage techniques These are some of the data storage techniques in 3D
Computer Vision and Computer Graphics. This list might not be an exhaustive
list but includes most of the formats referred during the future discussions
in this paper.

    * –

Voxel Grids represent the grid created by voxels, which are the extensions of
“Pixels” into the third dimension. Each voxel represents a small cubic volume
in space and can store various properties such as occupancy, color, or
texture. Voxel grids are commonly used for tasks such as 3D reconstruction,
scene understanding, and object recognition. They offer several advantages
over other data structures, such as point clouds or meshes, including
Regularty, Completeness, and Efficiency. Voxel grids can be constructed from
various sources of 3D data, such as point clouds, depth maps, or mesh models.
The size and resolution of the voxel grid can be adjusted to balance the
trade-off between accuracy and computational efficiency. Once constructed,
voxel grids can be used for a variety of tasks such as object segmentation,
feature extraction, and shape analysis.

    * –

Polygon (Triangular) Meshes is a collection of faces, vertices, and edges
leveraged to define the shape of the polyhedra object under consideration in
3D Computer Vision, Computer Graphics. These meshes might contain only
triangular (Triangular Meshes), Quadrilaterals (Quads), simple Polygons
(N-gons) as this can help in simplifying rendering. Occassionally, they might
be composed of Polygon with Holes or Concave Polygons.

    * –

QuadTree is a tree data structure to partition a two-dimensional space by
recursively dividing it into four quadrants or regions. Data at leaf node
varies but the leaf generally represent a “unit of interesting spatial
information”. The subdivided regions may be square or rectangular, or can even
have other arbitrary shapes. It was first coined in 1974 by Finkel et al [58].
Quadtrees and its variants generally tend to decompose the space into
adaptable cells. Each of these cells have a maximum capacity and the cells can
split when the maximum capacity is reached. A Tree-Pyramid is the spatial
analog of the complete trees. Hence, each of its nodes has four child nodes
except the leaf nodes. Further, all leave are on the same level. The data in
the Tree-Pyramid can be stored compactly in a Binary heap [201].

    * –

Octree is the 3D equivalent of a Quadtree where each node has exactly eight
children nodes. This data structure can be used to partition a 3D space by
dividing it into eight octants and is often used in 3D graphics and game
engines such as Unity and Unreal Engine. The term was pioneered by Donald
Meagher at Rensselaer Polytechnic Institute [143]. Octree, much like quadtrees
may be used as Point-Region Octree (PR-Octree; node stores an explicit 3D
point, the center of subdivision of that node; point defines one corner for
all eight children), Matrix-based Octree (MX-Octree; )

    * –

Point Cloud is a discrete representation of points in space. These points may
represent a 3D object or a shape. 3D scanners or photogrammetry softwares are
used for generating a Point Cloud. These types of representations are used for
several purposes such as 3D computer-aided design (CAD) models for
manufacturing parts, for metrology, quality inspections, animations,
rendering, etc.

  * •

Surface Normals describe the orientation of a surface at a particular point. A
surface normal is a vector that is perpendicular to the surface at the given
point. Surface normals are often used to determine the shading or lighting of
a 3D object, and are a key component in many rendering algorithms. The surface
normal vector can be computed using the gradient of the surface at the given
point. Eqn 2 defines the gradient for a smooth, continuous surface.

$\Delta f(x,y,z)=(\frac{\partial f}{\partial x},\frac{\partial f}{\partial
y},\frac{\partial f}{\partial z})$ (2)

where $f(x,y,z)$ is a scalar function that describes the surface. The surface
normal vector can then be computed by normalizing the gradient vector as given
by the eqn 3.

$\mathcal{N}=\frac{\Delta f(x,y,z)}{||\Delta f(x,y,z)||}$ (3)

where $||v||$ denotes the magnitude of vector $v$. The resulting $\mathcal{N}$
vector is a unit vector that points in the direction of the surface normal.
Surface normals can be computed for various types of 3D data, such as point
clouds, meshes, or voxel grids. In practice, surface normals are often
computed using a local neighborhood of points or vertices, such as by
averaging the normals of adjacent triangles in a mesh or by fitting a plane to
a set of nearby points in a point cloud.

  * •

Transmittance refers to the fraction of light that passes through a medium as
it travels from one point to another. It is often used in computer graphics to
model the effects of atmospheric scattering on the appearance of objects in a
scene. Transmittance is also used in machine learning to describe the ability
of a material or substance to transmit light through it, which is important in
fields such as optics and photonics.

  * •

Reflectance is the fraction of incident light that is reflected by a surface.
It is often used in computer graphics and machine learning to model the
appearance of materials and to infer properties of objects from images.
Reflectance can be affected by factors such as surface roughness, surface
orientation, and the wavelength of the incident light.

  * •

Radiance refers to the amount of light that is emitted, reflected, or
transmitted by a surface in a given direction. It is typically measured in
units of power per unit area per unit solid angle (e.g., watts per steradian
per square meter). Radiance is important in computer graphics because it
determines the color and brightness of objects in a scene as they are rendered
from different viewpoints and under different lighting conditions. In machine
learning, radiance is also used to describe the behavior of electromagnetic
radiation in the context of remote sensing and other applications.

  * •

Lighting

    * –

Specural Lighting is a type of lighting effect in computer graphics that
simulates the way light reflects from the shiny surface (such as glass or
metal). Its primarily used for creating highlights and glints on such surfaces
that helps to make them appear more realistic. It is based on the Phong
reflection model (discussed later when discussing Bidirectional Reflectance
Distribution Function) that describes how light interacts with surface in
terms of three components: Ambient, Specular, Diffuse. Specular component is
responsible for simulating the light off shiny surfaces.

The amount of specular lighting at a given point on a surface is determined by
the angle between the incoming light ray and the normal vector of the surface
at that point, as well as the angle between the outgoing reflection ray and
the viewing direction. These angles are defined by the incident vector
$\mathcal{L}$, the surface normal or the normal $\mathcal{N}$, and the viewing
direction vector $\mathcal{V}$. Hence, the angle between incident light ray
and surface normal is given by their dot product as depicted in the eqn 4

$\cos{\theta}=\mathcal{L}\cdot\mathcal{N}$ (4)

Reflection of the light ray from the surface is given by vector $\mathcal{R}$
that is the result of incident ray reflecting about the surface normal. This
has been given in the eqn. 5.

$\mathcal{R}=2(\mathcal{L}\cdot\mathcal{N})\mathcal{N}-\mathcal{L}$ (5)

and the angle between the reflected ray $\mathcal{R}$ and the viewing
direction $\mathcal{V}$ is given by the eqn. 6.

$\cos{\alpha}=\mathcal{R}\cdot\mathcal{V}$ (6)

The amount of specular lighting at a given point on the surface is then given
by the eqn. 7

$\mathcal{I}_{spec}=\textrm{K}_{s}\times\mathcal{I}_{light}\times\cos^{n}{\alpha}$
(7)

where $\mathcal{I}_{l}ight$ is the intensity of the incoming light,
$\textrm{K}_{s}$ is a material property that controls the amount of specular
reflection, and $n$ is the shininess or roughness of the surface. The shinier
the surface, the higher the value of $n$ and the more concentrated the
specular reflection will be.
In summary, specular lighting is a key component of the Phong reflection model
in computer graphics, and is used to simulate the reflection of light off of
shiny, reflective surfaces. The amount of specular lighting is determined by
the angle between the incoming light ray and the surface normal, as well as
the angle between the reflection ray and the viewing direction, and is
controlled by material properties such as $\textrm{K}_{s}$ and the shininess
of the surface.

    * –

Diffuse Lighting simulates the way light scatters off of matte, non-reflective
surfaces, such as wood or paper. It is used to create subtle shading and
shadows on such surfaces, which can help to make them appear more realistic.

The amount of diffuse lighting, much like Specular Lighting, at a given point
on a surface is determined by the angle between the incoming light ray and the
normal vector of the surface at that point. This angle is defined by the
incident vector $\mathcal{L}$ and the normal vector $\mathcal{N}$. The angle
between the incident light ray and the surface normal is given by the dot
product of the above mentioned values as given in the eqn. 8:

$\cos{\theta}=\mathcal{L}\cdot\mathcal{N}$ (8)

The amount of diffuse lighting at a given point on the surface is then given
by the eqn 9 given below.

$\mathcal{I}_{diff}=\textrm{K}_{d}\times\mathcal{I}_{light}\times\cos{\theta}$
(9)

where $\mathcal{I}_{l}ight$ is the intensity of the incoming light,
$\textrm{K}_{d}$ is a material property that controls the amount of diffuse
reflection, and $\cos{\theta}$ is the cosine of the angle between the incident
light ray and the surface normal. The higher the value of $\cos{\theta}$, the
more parallel the incoming light is to the surface, and the brighter the
resulting diffuse reflection will be.

So, diffuse lighting is used to simulate the scattering of light off of non-
reflective surfaces. The amount of diffuse lighting is determined by the angle
between the incoming light ray and the surface normal, and is controlled by
material properties such as $\textrm{K}_{d}$.

    * –

Ambient Lighting is a type of lighting effect in computer graphics that
simulated the way that light is scattered and reflected by the objects in the
environment. This is used to simulate the subtle, overall illumination of a
scene and can help to create a more realistic and natural-looking image.

In the Phong reflection model, ambient lighting is the simplest component and
represents the light that is scattered uniformly in all directions. It is not
affected by the direction of the light source or the surface normal, but is
instead a constant value that is added to the overall color of the object. The
amount of ambient lighting is determined by a material property known as the
ambient reflection coefficient, denoted as $\textrm{K}_{a}$.

The amount of ambient lighting at a given point on a surface is given by the
eqn. 10:

$\mathcal{I}_{amb}=\textrm{K}_{a}\mathcal{I}_{light}$ (10)

where $\mathcal{I}_{light}$ is the intensity of the ambient light in the
scene. The value of $\textrm{K}_{a}$ determines how much of this ambient light
is reflected by the surface.

Hence, ambient lighting is a simple and uniform type of lighting effect that
simulates the overall illumination of a scene. It is not affected by the
direction of the light source or the surface normal, but is instead a constant
value that is added to the overall color of the object. The amount of ambient
lighting is controlled by a material property known as the ambient reflection
coefficient.

  * •

Albedo is the property of a surface to be able to reflect sunlight from the
sun (/heat from the sun). Hence, light colored surface have a high albedo as
they reflect quite a lot of sun rays whereas dark surfaces absorb the rays
from the sun. In more technical terms, it can be stated as the ratio of a
measure of Diffuse Reflection of Solar radiation to the total amount of Solar
radiation.
In computer graphics research, albedo refers to the proportion of incoming
light that is reflected by a surface. It is an important parameter used in
material models to determine the color and reflectance properties of objects
in a scene. The albedo of a surface is typically represented as a value
between 0 and 1, where a value of 0 indicates that the surface absorbs all of
the incident light, and a value of 1 indicates that the surface reflects all
of the incident light.
The albedo of a surface can be calculated using the eqn 11 where $\mathcal{A}$
is albedo, $\mathcal{L}$ is the total amount of light that falls on the
surface (incident light), and reflected light $\mathcal{R}$ is amount of light
that is reflected by the surface.

$\mathcal{A}=\mathcal{R}/\mathcal{L}$ (11)

Albedo is often used along with other material properties (like specular
reflectivity, roughness) in Computer Graphics for modeling appearance of
surfaces in a scene. This dependence of albedo can be mathematically
represented as given in the eqn 12.

$\rho(\theta)=\frac{\rho_{0}+\rho_{1}\cos{\theta}}{1+\rho_{0}\cos{\theta}}$
(12)

where $\rho(\theta)$ represents directional hemisphere reflectance factor
representing ratio of reflected radiation to incident radiation at $\theta$
incidence angle, $\rho_{0},\rho_{1}$ are specular and diffuse reflectance
coefficient for ratios of reflected to incidence rays for perfectly smooth
(reflective) surface and perfectly rough (diffuse) surface. This means the
surface albedo depends on both specular and diffuse reflectance factors in
turn depending on the roughness, texture, reflectivity. As the angle of
incidence $\theta$ increases, the specular reflectance factor $\rho_{0}$
becomes more important, and the albedo of the surface increases. However, if
the surface is surrounded by objects that absorb light, the amount of
reflected light will be reduced, which will decrease the albedo of the
surface.
In addition to the surrounding environment, the albedo of a surface can also
be affected by the presence of shadows or other sources of occlusion. If a
surface is partially shaded, the amount of incident light will be reduced,
which will decrease the albedo of the surface. Overall, the albedo of a
surface is a complex phenomenon that depends on several factors. Albedo is an
important concept in computer graphics research because it plays a crucial
role in determining the overall appearance of objects in a scene, and it can
be used to create realistic lighting and shading effects.

  * •

Lambertian of a surface is the property that defines an ideal “matte” or
diffusely reflecting surface. In computer graphics, it is a type of surface
that scatters incident light equally in all directions. It is also known as a
diffuse surface, and it is an idealized model of many real-world surfaces such
as paper, cloth, and matte painted surfaces.
The Lambertian model assumes that the amount of light scattered by a surface
is proportional to the cosine of the angle between the incoming light
direction and the surface normal. This can be expressed using an eqn. 13 where
$\rho$ is reflectance factor constant value between 0 and 1.

$\mathcal{LR}=\frac{\rho}{\pi}$ (13)

The Lambertian model is characterized by a constant reflectance factor, which
is independent of the angle of incidence of the light. This means that a
Lambertian surface appears equally bright from all directions, and it does not
produce any specular highlights or reflections.
In computer graphics, the Lambertian model is often used as a simplifying
assumption for simulating the lighting of diffuse surfaces, as it is a simple
and efficient model that can produce realistic results in many cases. However,
it is important to note that not all surfaces behave like Lambertian surfaces,
and more complex models may be necessary for accurately simulating the
lighting of some surfaces, such as metals or plastics.

  * •

BRDF (Bidirectional Reflectance Distribution Function) is a mthematical
function to describe how light is reflected by surface in different
directions. It is a fundamental concept in Computer Graphics and COmputer
Vision. In Virtual Environment, it is used to model the reflection of light
from surfaces. It is represented by 4 variables which are incoming light
direction ($\omega_{i}$), outgoing light direction ($\omega_{o}$), surface
normal ($\mathcal{N}$), and light wavelength ($\lambda$). It has been
represented by the eqn. 14 where
$\textrm{dL}(\omega_{o}),\textrm{dE}(\omega_{i})$ are the differential amount
of light reflected in the direction of $\omega_{o}$ and differential amount of
incident energy in direction of $\omega_{i}$.

$f(\omega_{i},\omega_{o},\mathcal{N},\lambda)=\frac{\textrm{dL}(\omega_{o})}{\textrm{dE}(\omega_{i})},$
(14)

This function describes the amount of light reflected from surface in
particular direction for certain incident light direction and surface normal.
Many models have been proposed for the BRDF with different parameters and
assumptions. One such common model is the Lambertian BRDF which integrates the
eqns. 13–14 to get a model that assumes that the surface is perfectly diffuse
and reflects light equally in all directions and given by eqn. 15

$f(\omega_{i},\omega_{o},\mathcal{N},\lambda)=\frac{\rho}{\pi}$ (15)

Another model is called Phong BRDF which models specular reflections from
shiny surfaces and can be given by the eqn. 16

$f(\omega_{i},\omega_{o},\mathcal{N},\lambda)=\frac{\textrm{K}_{d}}{\pi}+\textrm{K}_{s}\frac{n+2}{2\pi}\cos^{n}{\theta},$
(16)

where $n$ is the specular exponent, higher values for which result in sharper
specular component, $\theta$ is the angle between the perfect specular
reflective direction and outgoing direction with angles clamped to $\pi/2$ to
avoid negative value, $\textrm{K}_{d},\textrm{K}_{s}$ are the diffuse
reflectivity (fraction of the incoming energy that is reflected diffusely) and
specular reflectivity (fraction of the incoming energy that is reflected
specularly). This equation can be written in the form of eqn. 17

$f(\omega_{i},\omega_{o},\mathcal{N},\lambda)=\frac{\textrm{K}_{d}}{\pi}+\frac{\textrm{K}_{s}}{\pi}(\mathcal{N}\cdot
h)^{\alpha},$ (17)

where $h$ is halfway vector between $\omega_{o}$ and $\omega_{i}$ and $\alpha$
is the specular component to control the specular highlight. Overall, the BRDF
is an important concept in computer graphics and computer vision, as it allows
us to model how light interacts with surfaces in virtual environments. By
understanding the properties of different BRDF models, we can create more
realistic and compelling virtual environments, and develop better algorithms
for computer vision applications such as object recognition and tracking.

  * •

Color Spaces are mathematical models that define a range of possible colors
and provide a mechanism for mapping those colors to numerical values. In
computer vision and computer graphics, color spaces are used to represent and
process colors in digital images and video. Here are some of the most commonly
used color spaces:

    * –

RGB (Red-Green-Blue) color space represents colors as combinations of red,
green, and blue values. It is an additive color model, meaning that colors are
created by adding together different amounts of red, green, and blue light.
RGB color space is widely used in digital displays and image capture devices.

    * –

CMYK (Cyan-Magenta-Yellow-Black) color space is used primarily in printing. It
represents colors as combinations of cyan, magenta, yellow, and black values.
CMYK is a subtractive color model, meaning that colors are created by
subtracting different amounts of ink from a white substrate.

    * –

HSV (Hue-Saturation-Value) color space represents colors using three
parameters. Hue represents the color itself, saturation represents the
intensity of the color, and value represents the brightness of the color. HSV
color space is often used in image processing applications. This can be
represented by the following eqns 18–23.

$\textrm{R}^{\prime}=\textrm{R}/255;\textrm{G}^{\prime}=\textrm{G}/255;\textrm{B}^{\prime}=\textrm{B}/255;$
(18)
$\textrm{C}_{max}=\max{(\textrm{R}^{\prime},\textrm{G}^{\prime},\textrm{B}^{\prime})},\textrm{
}\textrm{C}_{min}=\min{(\textrm{R}^{\prime},\textrm{G}^{\prime},\textrm{B}^{\prime})}$
(19)

$\Delta=\textrm{C}_{max}-\textrm{C}_{min}$ (20)

$\textrm{H}=\begin{cases}0\degree,\Delta=0,\\\
60\degree\times\left(\frac{\textrm{G}^{\prime}-\textrm{B}^{\prime}}{\Delta}mod6\right),\textrm{C}_{max}=\textrm{R}^{\prime}\\\
60\degree\times\left(\frac{\textrm{B}^{\prime}-\textrm{R}^{\prime}}{\Delta}\textrm{
}+2\right),\textrm{ }\textrm{C}_{max}=\textrm{G}^{\prime}\\\
60\degree\times\left(\frac{\textrm{R}^{\prime}-\textrm{G}^{\prime}}{\Delta}\textrm{
}+4\right),\textrm{ }\textrm{C}_{max}=\textrm{B}^{\prime}\\\ \end{cases}$ (21)
$\textrm{S}=\begin{cases}0\degree,\textrm{ }\textrm{C}_{max}=0,\\\
\frac{\Delta}{\textrm{C}_{max}},\textrm{C}_{max}\not=0\\\ \end{cases}$ (22)

$\textrm{V}=\textrm{C}_{max}$ (23)

    * –

LAB color space is a color space designed to separate color information from
brightness. It consists of three components: L, which represents brightness,
and a and b, which represent the color in the green-red and blue-yellow axes.
LAB color space is commonly used in image processing applications such as
color correction and color-based segmentation.

    * –

YUV color space separates color information from brightness, similar to LAB.
The Y component represents brightness, while the U and V components represent
the color in the blue-yellow and green-red axes. YUV color space is commonly
used in video processing and compression applications.

    * –

XYZ color space is a standard color space used by the International Commission
on Illumination (CIE). It is based on the human visual system and represents
all colors visible to the human eye. XYZ color space is often used as an
intermediate color space in color transformations.

  * •

Artifacts In computer graphics, artifacts refer to any unwanted or unintended
visual distortions or anomalies that occur in an image or animation. Artifacts
can be caused by various factors, such as limitations of hardware or software,
errors in algorithms or data processing, or limitations in the human visual
system.

There are many different types of artifacts that can occur in computer
graphics, including:

    1. 1.

Aliasing: This occurs when an image or object appears jagged or pixelated,
especially along edges or curves. Aliasing is caused by undersampling, where
the resolution of the image or object is not high enough to accurately
represent the details. In general, it refers to the unwanted effects that
arise when high-frequency signals appear in computer graphics. An instance of
this can be when downscaling any image from its original size so its quite
common in NeRFs as they work with a downsampled image dataset.
Aliasing is a common problem in computer graphics that occurs when the
resolution of an image is not high enough to accurately represent a high-
frequency signal, such as a diagonal line or a curve. This results in the
appearance of jagged edges or stair-step patterns, which can make the image
look pixelated or low-quality.
Aliasing is caused by the sampling rate, or the frequency at which the image
is sampled. The Nyquist-Shannon sampling theorem states that the sampling rate
must be at least twice the highest frequency component of the signal in order
to accurately represent it. If the sampling rate is too low, high-frequency
components of the signal will be lost or undersampled, leading to aliasing.
In computer graphics, aliasing can occur in several contexts. One common
example is when rendering lines or curves using rasterization techniques. In
this case, the line or curve is represented as a series of pixels, and if the
sampling rate is too low, the line or curve will appear jagged or stair-
stepped.
To reduce aliasing in computer graphics, a variety of techniques have been
developed, including anti-aliasing. Anti-aliasing works by oversampling the
image, and then using filtering techniques to smooth out the jagged edges or
stair-step patterns. This can be done by averaging the colors of neighboring
pixels, or by using more advanced filtering techniques such as Gaussian
filtering.
The amount of aliasing in an image can be quantified using a measure known as
the Nyquist frequency, which is defined as half the sampling rate. The Nyquist
frequency represents the highest frequency component that can be accurately
represented in the image. If a signal has a frequency component that is higher
than the Nyquist frequency, it will be undersampled and will result in
aliasing.
In summary, aliasing is a common problem in computer graphics that occurs when
the resolution of an image is not high enough to accurately represent a high-
frequency signal. This results in jagged edges or stair-step patterns, which
can make the image look pixelated or low-quality. The Nyquist-Shannon sampling
theorem states that the sampling rate must be at least twice the highest
frequency component of the signal in order to accurately represent it.
Techniques such as anti-aliasing can be used to reduce aliasing in computer
graphics.

    2. 2.

Moiré patterns: These are interference patterns that occur when two or more
repetitive patterns overlap, causing a new pattern to emerge. Moiré patterns
can be seen in printed images or screens, and they can also occur in computer
graphics when rendering textures or patterns.

    3. 3.

Shadow acne: This occurs when shadows appear jagged or pixelated, especially
on curved surfaces. Shadow acne is caused by numerical errors in the
algorithms used to calculate shadows.

    4. 4.

Banding: This occurs when gradients appear as distinct bands of color, rather
than a smooth transition. Banding can be caused by limitations in color depth
or by compression algorithms that reduce the number of colors used in an
image.

    5. 5.

Blooming: This occurs when bright areas of an image bleed into darker areas,
causing a halo effect. Blooming is caused by limitations in the dynamic range
of the display or by overexposure in the original image.

    6. 6.

Ghosting: This occurs when fast-moving objects or camera movements leave a
trail or afterimage. Ghosting is caused by limitations in the refresh rate of
the display or by motion blur.

    7. 7.

Artifacts in 3D rendering: These can include glitches or errors in geometry or
texture mapping, flickering or popping of objects, or distortions caused by
limitations in the rendering engine.

To avoid or minimize artifacts in computer graphics, various techniques and
technologies are used, such as anti-aliasing, high dynamic range rendering,
advanced shadowing algorithms, and color correction. Additionally, careful
calibration and testing can help identify and address potential sources of
artifacts.
So it can be said that artifacts in computer graphics are any unwanted or
unintended visual distortions or anomalies that occur in an image or
animation. Artifacts can be caused by various factors, such as limitations of
hardware or software, errors in algorithms or data processing, or limitations
in the human visual system. There are many different types of artifacts, and
various techniques and technologies are used to avoid or minimize them.

  * •

Alpha Compositing is a technique used in computer graphics to combine two or
more images together based on their transparency values. This technique is
commonly used in compositing and image editing applications.
In alpha compositing, each pixel of an image is represented by four values:
RGB$\alpha$. The $\alpha$ value represents the transparency of the pixel, with
a value of 0 indicating complete transparency and a value of 1 indicating
complete opacity. When two images are combined using alpha compositing, the
alpha values of each pixel are used to determine how much of each image should
be visible in the final result. It is described by the following eqn 24 given
below.

$\mathcal{C}=((1-\alpha)\mathcal{C}_{1})+(\alpha\mathcal{C}_{2})$ (24)

where $\mathcal{C},\mathcal{C}_{1},\mathcal{C}_{2}$ represent the color of
final pixel, pixel in the first image, and pixel in the second image. This
equation essentially blends the colors of the two images together based on
their alpha values, with the second image contributing more to the final
result if its alpha value is higher.
One common use of alpha compositing is to add an image with a transparent
background to another image. For example, if an image of a person with a
transparent background is overlaid onto a background image using alpha
compositing, the person will appear to be seamlessly integrated into the
background.
Another use of alpha compositing is to create effects such as fades, glows,
and shadows. By manipulating the alpha values of an image, it is possible to
create the illusion of transparency and depth, which can be used to add visual
interest and realism to a scene.

  * •

Baking is the process of performing offline calculations extensively and
caching results in a Vertex Attributes/Texture Maps. When coming across this
term in future, it might be used in context of generating low-level models,
lightmaps, or normal maps.
Baking in computer graphics refers to the process of precomputing and storing
lighting and other surface properties of 3D models or scenes into textures or
other forms of data that can be used to speed up rendering and improve
performance. This process is commonly used in real-time applications such as
video games, where real-time rendering performance is crucial.
Baking is done by simulating and rendering the lighting and other surface
properties of a 3D model or scene at a high level of detail, and then storing
the results in a texture or other form of data. This precomputed data can then
be used during real-time rendering to quickly and efficiently simulate the
lighting and surface properties of the 3D model or scene.
There are several types of data that can be baked, including lighting,
shadows, ambient occlusion, and reflection maps. Each of these types of data
can be computed and stored separately, and then combined during real-time
rendering to create a more realistic and detailed image.
The process of baking can be broken down into several steps. First, the 3D
model or scene is prepared for baking by optimizing the geometry, setting up
the lighting and material properties, and defining the areas that need to be
baked. Next, the baking process itself is performed, which involves rendering
the scene from multiple angles and at different levels of detail to capture
the necessary data. Finally, the baked data is stored in a texture or other
form of data that can be used during real-time rendering.
The amount of data that needs to be stored for baking can be quite large,
especially for complex scenes or models. To optimize performance, various
compression and optimization techniques can be used to reduce the size of the
data without compromising quality.
In summary, baking in computer graphics is the process of precomputing and
storing lighting and other surface properties of 3D models or scenes into
textures or other forms of data that can be used to speed up rendering and
improve performance. This process involves simulating and rendering the
lighting and other surface properties of a 3D model or scene at a high level
of detail, and then storing the results in a texture or other form of data.
Baking can be used to compute and store data for lighting, shadows, ambient
occlusion, and reflection maps, among other things.

  * •

Dynamic Range refers to the range of brightness or luminance values that can
be represented in an image or scene. It is a measure of the difference between
the brightest and darkest parts of the image, and is typically expressed in
terms of the ratio between the maximum and minimum luminance values. It can be
categorized as follows.

    * –

HDR (High-Dynamic Range) refers to the ability to capture, store, and display
a wider range of brightness and color values than traditional displays. HDR
technology is becoming increasingly popular in computer graphics and is
commonly used in applications such as video games, virtual reality, and video
production.
Traditional displays typically have a limited range of brightness and color
values that they can display, typically around 8 bits per channel (or 24 bits
total for RGB color), resulting in a maximum of 256 levels of brightness and
color. In contrast, HDR displays can support a much wider range of brightness
and color values, with some displays supporting up to 10 or 12 bits per
channel, resulting in up to thousands of levels of brightness and color.
HDR is achieved through a combination of hardware and software technologies.
In order to capture and store HDR images, cameras and sensors with higher
dynamic range capabilities are used, along with file formats such as EXR or
HDR that can store a wider range of brightness and color values. HDR images
can be processed and manipulated using specialized software that can take
advantage of the increased dynamic range, allowing for more realistic and
visually stunning results.
In addition to capturing and storing HDR images, HDR displays are also
required to properly display HDR content. This requires the use of specialized
hardware and software that can handle the increased dynamic range and color
gamut, and can map the wider range of brightness and color values to the
limited range of values that can be displayed on the screen.
One common technique used in HDR is tone mapping, which is the process of
mapping the wide range of brightness and color values in an HDR image to the
limited range of values that can be displayed on a traditional display. This
can be done using a variety of techniques, including logarithmic or linear
mapping, or through more advanced algorithms such as tone mapping operators.
In summary, High Dynamic Range (HDR) in computer graphics refers to the
ability to capture, store, and display a wider range of brightness and color
values than traditional displays. HDR is achieved through the use of
specialized hardware and software that can handle the increased dynamic range
and color gamut, and can map the wider range of brightness and color values to
the limited range of values that can be displayed on the screen. HDR
technology is becoming increasingly popular in computer graphics and is used
in applications such as video games, virtual reality, and video production.

    * –

LDR (Low-Dynamic Range) refers to images or video that have a limited range of
brightness and color values. LDR images are typically created by capturing or
rendering images with a fixed range of brightness and color values, and
displaying them on traditional displays with a limited dynamic range.
Traditional displays typically have a limited range of brightness and color
values that they can display, typically around 8 bits per channel (or 24 bits
total for RGB color), resulting in a maximum of 256 levels of brightness and
color. LDR images are captured, processed, and displayed within this limited
range, resulting in images that may appear less vibrant or less realistic than
HDR images.
In LDR images, the range of brightness and color values is limited by the
range of the display device. For example, if an image is captured with a range
of brightness values that exceeds the range of the display device, the
brightness values will be compressed or clipped to fit within the display
range. This can result in loss of detail in the highlights or shadows of the
image.
LDR images are still commonly used in many applications, such as traditional
video production and broadcast television, where the limited dynamic range of
displays is sufficient for the intended use. LDR images can also be created
from HDR images through a process called tone mapping, which maps the wider
range of brightness and color values in an HDR image to the limited range of
values that can be displayed on a traditional display.
In summary, Low Dynamic Range (LDR) in computer graphics refers to images or
video that have a limited range of brightness and color values. LDR images are
typically created by capturing or rendering images with a fixed range of
brightness and color values, and displaying them on traditional displays with
a limited dynamic range. LDR images are still commonly used in many
applications, such as traditional video production and broadcast television.
LDR images can also be created from HDR images through a process called tone
mapping.

  * •

Yaw, Pitch, Roll refer to the three degrees of freedom that are associated
with the movement of objects on their axes. There are six degrees of freedom
for a rigid body. An object can move up-down, left-right, forward-backward,
pitch, yaw, roll. This is better depicted by the fig. 1 given below. It would
become clear by looking at the following figure.

Figure 1: A schematic representation of six degrees of freedom

  * •

Gimbal Lock is a phenomenon in 3D computer graphics and animation that makes
the rotations of an object limited or ambiguous, that causes unexpected or
incorrect behavior. It occurs when using Euler angles to represent the 3D
space orientation of an object.
One of the common measures to represent 3D space orientation of an object is
the usage of Euler angles. And these Euler angles are used to define three
rotations around all three different axes. However, when the object is rotated
to certain positions, the rotations can become aligned or parallel, resulting
in a loss of a degree of freedom in the rotation. This leads to the condition
called Gimbal Lock.
When visualizing Gimbal Lock, assume a set of three nested rings, each
representing a rotation around a different axis (e.g., $x$, $y$, and $z$).
When the rings are aligned, the rotations around the $y$ and $z$ axes become
the same, resulting in a loss of 2-degrees of freedom in the rotation.
Assume having an object with three Euler angles representing its orientation:
yaw/heading (rotation around the y-axis), pitch (rotation around the x-axis),
and roll (rotation around the z-axis). Then this object orientation can be
represented the as a rotation matrix R, which can be calculated as given in
the eqn. 25 below.

$\textrm{{R}}=\textrm{{R}}_{x}\times\textrm{{R}}_{y}\times\textrm{{R}}_{z},$
(25)

where $\textrm{{R}}_{x}$, $\textrm{{R}}_{y}$, and $\textrm{{R}}_{z}$ are given
by the following eqns. 26.

$\textrm{{R}}_{z}=\begin{pmatrix}\cos{\theta_{r}}&-\sin{\theta_{r}}&0\\\
\sin{\theta_{r}}&\cos{\theta_{r}}&0\\\
0&0&1\end{pmatrix},\textrm{{R}}_{x}=\begin{pmatrix}1&0&0\\\
0&\cos{\theta_{p}}&-\sin{\theta_{p}}\\\ 0&\sin{\theta_{p}}&\cos{\theta_{p}}\\\
\end{pmatrix},\textrm{{R}}_{y}=\begin{pmatrix}\cos{\theta_{y}}&0&\sin{\theta_{y}}\\\
0&1&0\\\ -\sin{\theta_{y}}&0&\cos{\theta_{y}}\end{pmatrix},$ (26)

However, when the pitch angle approaches $\pm 90\degree$, the rotation around
the y-axis (yaw) and the rotation around the z-axis (roll) become aligned,
resulting in a loss of a degree of freedom. This is gimbal lock, and it can
cause unexpected or incorrect behavior in animations and simulations.

One solution to gimbal lock is to use quaternion rotations instead of Euler
angles, as quaternions do not suffer from gimbal lock. Alternatively, other
rotation representations, such as axis-angle or matrix representations, can
also be used to avoid gimbal lock.

###  2.2 Mathematics and Geometry

This section deals with the basic terminology in the discipline of Mathematics
and Geometry which is useful for the rest of the survey.

  * •

Hadamard Product is an operation that takes two matrices of the same size and
multiplies their corresponding entries together to produce a new matrix of the
same size. The Hadamard product is also known as the element-wise product,
point-wise product, or Schur product. The Hadamard product of two matrices A
and B of the same size is denoted by the symbol $\odot$ in the eqn 27 given
below.

$(\mathcal{A}\odot\mathcal{B})_{ij}=\mathcal{A}_{ij}\odot\mathcal{B}_{ij}$
(27)

where $i$ and $j$ represent the row and column indices of the matrices. Hence,
the Hadamard product produces a new matrix where each element is the product
of the corresponding elements in the original matrices. For instance, the
Hadamard product of the following eqn 28 given below.

$\mathcal{A}=\begin{pmatrix}1&2\\\ 3&4\end{pmatrix},\textrm{and
}\mathcal{B}=\begin{pmatrix}5&6\\\ 7&8\end{pmatrix},\textrm{then
}\mathcal{A}\odot\mathcal{B}=\begin{pmatrix}5&12\\\ 21&32\end{pmatrix}$ (28)

This is a useful operation in various applications of linear algebra, such as
image processing, computer vision, and machine learning. For example, in image
processing, the Hadamard product can be used to perform point-wise operations
on images, such as contrast enhancement or color manipulation. In machine
learning, the Hadamard product can be used to perform element-wise
multiplication of feature vectors, which can be useful for regularization or
feature selection.

  * •

Kronecker Product is a mathematical operation that takes two matrices and
produces a new matrix whose entries are all possible products of the entries
of the original matrices. It is denoted by the symbol $\otimes$. Now, when
given 2 matrices $\mathcal{A}_{(m\times n)}$ and $\mathcal{B}_{(p\times q)}$,
the Kronecker product $\mathcal{A}\otimes\mathcal{B}$ is given by the eqn 29

$(\mathcal{A}\otimes\mathcal{B})_{ij}=\mathcal{A}_{ij}\times\mathcal{B}_{kl}$
(29)

where $i,j,k,l$ represent row, column indices of the matrices. So, Kronecker
product produces a new matrix where each entry is product of the corresponding
entries in original matrices, arranged in a matrix block structure. For
instance, the Hadamard product of the following eqn 30 given below.

$\begin{array}[]{r}\mathcal{A}=\begin{pmatrix}1&2\\\
3&4\end{pmatrix},\textrm{and }\mathcal{B}=\begin{pmatrix}5&6\\\
7&8\end{pmatrix},\\\ \textrm{then
}\mathcal{A}\otimes\mathcal{B}=\begin{pmatrix}1\times\mathcal{B}&2\times\mathcal{B}\\\
3\times\mathcal{B}&4\times\mathcal{B}\end{pmatrix}\Rightarrow\begin{pmatrix}5&6&10&12\\\
7&8&14&16\\\ 15&18&20&24\\\ 21&24&28&32\end{pmatrix}\end{array}$ (30)

The Kronecker product is a useful operation in various applications of linear
algebra, such as signal processing, computer vision, and quantum mechanics.
For example, in signal processing, the Kronecker product can be used to model
the behavior of complex systems by combining simpler models. In computer
vision, the Kronecker product can be used to construct feature descriptors for
image analysis. In quantum mechanics, the Kronecker product is used to
describe the composite state of multiple quantum systems.

  * •

Signed Distance Function describes the distance between a point in space and a
geometrical object. It is often used in computer graphics, particularly in the
field of 3D rendering, to create realistic images of 3D objects. This function
returns a signed value, which indicates whether the point is inside or outside
the object. If the point is inside the object, the value returned by the SDF
is negative; if the point is outside the object, the value is positive. The
general equation for an SDF function is given in eqn. 31

$f(p)=d(p,O)$ (31)

where $p$ is point in space, $O$ is a geometrical object, and $d$ represents
the euclidean distance between $p$ and $O$. The different types of SDFs are
given below.

    * –

Sphere Signed Distance Function is used to describe a sphere and its equation
is as given by eqn 32

$f(p)=\textrm{{length}}(p-o_{c})-r$ (32)

where $p$ is point in space, $o_{c}$ is sphere center, $r$ is sphere radius,
length represents the Euclidean distance.

    * –

Box Signed Distance Function is used to describe a box and its equation is as
given by eqn 33

$\begin{multlined}f(p)=\max(|(p_{x}-center_{x})|-size_{x},0.0)\\\
\begin{aligned} &+\max(|(p_{y}-center_{y})|-size_{y},0.0)\\\
&+\max(|(p_{z}-center_{z})|-size_{z},0.0)\end{aligned}\end{multlined}f(p)=\max(|(p_{x}-center_{x})|-size_{x},0.0)\\\
\begin{aligned} &+\max(|(p_{y}-center_{y})|-size_{y},0.0)\\\
&+\max(|(p_{z}-center_{z})|-size_{z},0.0)\end{aligned}$ (33)

where $p$ is point in space, $o_{c}$ is box center, size is box size.

    * –

Cylinder Signed Distance Function is used to describe a cylinder and its
equation is as given by eqn 34

$f(p)=\textrm{{length}}(p.xz-o_{c}.xz)-r$ (34)

where $p$ is point in space, $o_{c}$ is cylinder center, $r$ is cylinder
radius, length represents the Euclidean distance.

    * –

Truncated Signed Distance Function is a mathematical function that is used to
represent a 3D shape in a volumetric way. It is commonly used in 3D computer
vision and robotics applications, such as 3D reconstruction, SLAM, and object
recognition. The TSDF function is an extension of the Signed Distance Function
(SDF) that allows the representation of both the exterior and interior of an
object. It does this by truncating the distance function at a certain
threshold value, which creates a smooth transition between the interior and
exterior regions of the object. TSDF can be represented by the eqn 35

$\mathcal{TSDF}(p)=\min{\left(\max{\left(\frac{f(p)}{\textrm{{dis}}_{trunc}},-1\right)},1\right)}$
(35)

where $p$ is 3D point in space, $f(p)$ is SDF at $p$ and
$\textrm{{dis}}_{trunc}$ is truncated threshold to determine distance beyond
where function is truncated. Output of TSDF is scalar that indicates distance
between $p$ and nearest surface of the object. The output sign indicates
whether the point is inside or outside of object. There are differeent types
of TSDF used in Computer Vision as given below.

      * *

Volumetric TSDF is used to represent a 3D object as a voxel grid. The distance
values are stored in the voxels of the grid, and the grid is updated as new
data is acquired.

      * *

Point-based TSDF is used to represent a 3D object as a set of 3D points. The
distance values are computed for each point, and the points are updated as new
data is acquired.

      * *

Mesh-based TSDF is used to represent a 3D object as a mesh. The distance
values are computed for each vertex of the mesh, and the mesh is updated as
new data is acquired.

    * –

NSDF (Neural Signed Distance Function) is a type of neural network
architecture that is used to learn a continuous representation of a 3D object
from a set of 3D points. It is commonly used in 3D shape reconstruction,
object recognition, and robotics applications. The NSDF function takes as
input a 3D point and outputs a signed distance value, which indicates the
distance between the point and the nearest surface of the object. The sign of
the distance value indicates whether the point is inside or outside the
object. NSDF can be represented by eqn 36–37 given below.

$f(p)=\textrm{{MLP}}(p),\textrm{so }$ (36)

$\mathcal{NSDF}(p)=\frac{f(p)}{||f(p)||_{2}}$ (37)

where $p$ is 3D point, $\textrm{{MLP}}(p)$ is MLP output applied to $p$, and
$||f(p)||_{2}$ is the L2 norm of the MLP output. The NSDF output is a unit
vector that points towards the nearest surface of the object. There are
different types of NSDF used in computer vision and robotics applications.
Some of them are:

      * *

PointNet++-based NSDF is based on the PointNet++ architecture, which is a
neural network architecture that is designed to work directly on point clouds.
The NSDF is learned by training the PointNet++ network to predict the signed
distance value for each point in the input point cloud.

      * *

Implicit Function-based NSDF is based on an implicit function that defines the
3D object surface as the zero level set of the function. The NSDF is learned
by training a neural network to approximate the implicit function using a set
of 3D points.

      * *

Atlas-based NSDF is based on an atlas of 3D shapes, where each shape is
represented as an NSDF. The NSDF is learned by training a neural network to
predict the correct NSDF for a given shape in the atlas.

      * *

Occupancy-based NSDF is based on an occupancy function that defines the
occupancy of a 3D space. The NSDF is learned by training a neural network to
predict the occupancy value for each point in the input point cloud.

    * –

Dual Contouring SDF is a method for generating a 3D mesh from the SDF
representation of an object. It uses a technique called dual contouring to
extract the surface mesh from the SDF.

    * –

Distance Field SDF is a method for generating a voxel grid that represents the
SDF of an object. It is useful for collision detection and path planning in
robotics applications.

    * –

Occupancy SDF is similar to the Distance Field SDF, but instead of
representing the SDF as a voxel grid, it represents it as an occupancy grid,
where each grid cell is either occupied or unoccupied. It is useful for object
recognition and scene understanding in computer vision applications.

    * –

Fast Marching SDF is a method for computing the SDF of an object using the
Fast Marching Method, which is an algorithm for solving partial differential
equations. It is useful for real-time applications that require fast
computation of the SDF.

    * –

Level Set SDF is a method for representing the SDF as the zero level set of a
higher-dimensional function. It is useful for modeling the motion of
deformable objects and for tracking the evolution of interfaces in fluids and
other physical systems.

  * •

Plenoptic Function is a mathematical representation of complete information of
light rays travelling in the 3D space. It describes how light is emitted from
the scene, travels through space, interacts with objects, and is finally
captured by a sensor or observer. The plenoptic function is a powerful tool
for modeling and simulating various optical effects, such as depth of field,
motion blur, and light field imaging. It is a seven-dimensional function that
takes into account the position of the light source, its wavelength, the
direction of the light rays, and time. This function can be depicted by the
eqn 38 given below.

$\mathcal{PF}(x,y,z,\theta,\phi,t,\lambda)=\mathcal{I}$ (38)

where $x,y,z$ is 3D space position of the point, $\theta,\phi$ represent the
direction of light rays (or sometimes the viewing direction, since light
direction is dependant on the viewer direction), $t$ represents time, and
$\mathcal{I}$ is the radiance of light intensity at the queried position,
directions, and times. In computer graphics, the plenoptic function is often
used to simulate the behavior of light in virtual scenes. One extension of the
plenoptic function is the bidirectional reflectance distribution function
(BRDF) (discussed). There are several extensions of the Plenoptic Function
(not related to Computer Graphic), which include:

    * –

Plenoptic Sampling Function represents the plenoptic function in a discrete
form, by sampling the function at discrete positions and directions. The
Plenoptic Sampling Function is used in the design of plenoptic cameras.

    * –

Compressed Plenoptic Function compresses the Plenoptic Function by exploiting
the sparsity of the function. It is useful for efficient storage and
transmission of plenoptic data.

    * –

Multi-View Plenoptic Function represents the plenoptic function from multiple
viewpoints. It takes into account the position and orientation of multiple
cameras or sensors and combines the plenoptic data from each viewpoint to
create a single multi-view plenoptic function.

    * –

Time-Varying Plenoptic Function represent the plenoptic function over time. It
takes into account the time-varying nature of light and captures the changes
in the plenoptic function over time.

    * –

Polarization Plenoptic Function represents the polarization state of light in
addition to the position, direction, and time. It captures the polarization
properties of light, which are important for a range of applications including
remote sensing and biomedical imaging.

    * –

SPF (Surface Plenoptic Function) is the 5-D functional representation of the
intensity of light reflected from the surface at every point on the surface,
in all directions, and at all times. It is a generalization of the Plenoptic
Function that takes into account the surface geometry and the orientation of
the surface normal. It is often depicted by the eqn 39 given below.

$\mathcal{SPF}(x,y,z,\theta,\phi)=\mathcal{L}$ (39)

The SPF is useful for a range of applications in computer vision, graphics,
and robotics, including surface reconstruction, object recognition, and image-
based rendering. One of the main advantages of the SPF is that it provides a
compact representation of the surface appearance that can be used to
synthesize novel views of the surface from any viewpoint and under any
lighting conditions. To estimate the SPF from images, several methods have
been proposed in the past, including Shape-from-Polarization, Shape-from-
Shading, and Shape-from-Gradient. These methods use different cues to estimate
the surface geometry and the surface normal, which are then used to estimate
the SPF.
The SPF is a powerful representation of surface appearance that can capture
complex lighting effects and surface properties such as reflectance and
transparency. However, it requires accurate estimation of the surface geometry
and orientation, which can be challenging in some cases, particularly for
complex or textured surfaces.

    * –

BTF (Bidirectional Texture Function) describes the variation of a surface’s
appearance with respect to viewing and illumination angles. BTF is a
mathematical model that describes the appearance of a surface as it changes
with respect to different viewing and illumination directions. It is a higher-
dimensional extension of the BRDF, which describes the reflection properties
of a surface. The BTF is typically defined by the eqn. 40

$\mathcal{BTF}(x,y,\omega_{i},\omega_{o},\omega_{v})$ (40)

where $x$ and $y$ are the spatial coordinates of the surface, $\omega_{i}$ and
$\omega_{o}$ are the incoming and outgoing light directions, and $\omega_{v}$
is the viewing direction. The BTF describes how the surface appearance changes
as the illumination and viewing directions change. The BTF can be measured
using specialized imaging techniques that capture the surface appearance under
a variety of viewing and illumination conditions. These measurements can be
used to construct a BTF model that can be used for rendering and simulation of
the surface appearance under arbitrary lighting and viewing conditions.
The BTF has many applications in computer graphics, including virtual reality,
video games, and visual effects. It allows for realistic rendering of surface
appearance under complex lighting conditions, such as indirect illumination
and multiple light sources. The BTF is often used in conjunction with other
reflectance models, such as the BRDF and the BSDF (bidirectional scattering
distribution function), to create more accurate and realistic renderings of
surfaces.

These extensions of the Plenoptic Function are important for a range of
applications in computer vision, robotics, remote sensing, and other fields
that involve capturing, analyzing, and synthesizing light in the 3D space.

  * •

Spherical Gaussians are a type of RBF (Radial Basis Function) that are
commonly used in computer graphics and computer vision applications to model
and represent 3D data. They are particularly useful for tasks such as surface
reconstruction, shape analysis, and point cloud processing. The basic form of
the spherical Gaussian function is given by the eqn 41.

$\mathcal{G}(x)=e^{-\frac{||x||^{2}}{\sigma^{2}}}$ (41)

where $x$ is 3D vector for point in space, $||x||$ is its Euclidean distance,
and $\sigma$ scales and controls the size of Gaussiann function.

Spherical Gaaussians are often combined to give a weighted sum of multiple
Gaussians, resulting in a more expressive and flexible function as described
in eqn 42

$f(x)=\sum_{i=1}^{N}w_{i}\times\mathcal{G}(x-c_{i})$ (42)

where $w_{i},c_{i}$ are the weight factor and center position, and its taken
over a total of $N$ centers.

The parameters of the spherical Gaussians, including the center positions,
weight factors, and scaling factor, can be learned from data using various
techniques such as least squares or maximum likelihood estimation. The
resulting function can then be used to interpolate or extrapolate 3D data,
such as point clouds or surface meshes.
Spherical Gaussians have several desirable properties, such as smoothness,
isotropy, and rotational symmetry. These properties make them well-suited for
modeling and analyzing 3D data, particularly when the data is noisy or
incomplete. Spherical Gaussians are also efficient to compute, making them
practical for real-time applications.

  * •

Eikonal Equation is a partial differential equation that describes the
propagation of waves. It is used in fields such as computer vision and
graphics, to model various phenomena, including surface deformation and motion
of fluids. It can have several forms but one of the general form of the
eikonal equation is given by the eqn. 43

$|\nabla u(x)|=f(x)$ (43)

In computer graphics and computer vision, the Eikonal equation is used to
compute the distance function to a surface or object. Given an input point
cloud or surface, the Eikonal equation is solved to compute the distance
function to the surface at every point in space. This distance function can be
used to create level sets, which are surfaces that represent a fixed distance
away from the input surface. Applications of the Eikonal Equation in computer
graphics and computer vision include Ray Tracing, Shape from shading, Image
segmentation, Medical imaging, level set computation, distance map
computation, texture synthesis, path planning, collision avoidance.

  * •

Fourier Series is a periodic representation of a function $f(x)$. It is
defined as a sum of sine and cosine functions of different frequencies and is
mathematically given by the following eqn 44 with a period of $2\pi$, and
$a_{0},a_{n},b_{n}$ being coefficients depending on function f(x).

$f(x)=\frac{a_{0}}{2}+\sum_{n=1}^{\infty}\left[a_{n}cos(nx)+b_{n}sin(nx)\right]$
(44)

Now, these coefficients can be given based on the following formulae given in
eqn 45

$\begin{array}[]{l}a_{0}=\frac{1}{\pi}\int_{-\pi}^{\pi}f(x)\,dx\\\
a_{n}=\frac{1}{\pi}\int_{-\pi}^{\pi}f(x)cos(nx)\,dx\\\
b_{b}=\frac{1}{\pi}\int_{-\pi}^{\pi}f(x)sin(nx)\,dx\\\ \end{array}$ (45)

  * •

Fourier Transform helps to transform function in the time domain to function
in the frequency domain. It and it’s inverse are given by the eqn 45 where $k$
is frequency variable, $i$ is the imaginary unit, $F(x)$ is the Fourier
transform, and $f(x)$ is the inverse fourier transform.

$\begin{array}[]{l}F(x)=\int_{-\pi}^{\pi}f(x)\exp^{-2\pi\iota kx}\,dx\\\
f(x)=\int_{-\pi}^{\pi}F(k)\exp^{2\pi\iota kx}\,dk\end{array}$ (46)

The Fourier transform allows us to analyze the frequency content of a function
and is widely used in signal processing, image processing, and many other
fields of engineering and science.

  * •

Discrete Fourier Transform is a mathematical technique that is a discrete
variant of the Fourier Transform discussed above. It transforms a discrete-
time sequence of N complex numbers $x[n]$ into a sequence of N complex numbers
$X[k]$ representing the frequency content of the signal. It is defined by the
eqn 47 where $k$ is frequency index ranging from $0$ to $N-1$.

$X[k]=\sum_{n=0}^{N-1}\left(x[n]\exp^{\frac{-2\pi\iota kn}{N}}\right)$ (47)

It can be calculated using matrix multiplication of the $N\times N$ DFT matrix
$F$ and the $N-1$ column vectors.

  * •

FFT (Fast-Fourier Transform) is used for computing the DFT of a sequence of N
complex numbers in $O(n\log{}n)$ rather than $O(n^{2})$ required by the naive
DFT. The FFT algorithm exploits the symmetry and periodicity properties of the
DFT to compute the transform more efficiently.

The FFT algorithm divides the sequence $x[n]$ into smaller sub-sequences,
computes the DFT of each sub-sequence recursively, and combines the results to
obtain the final DFT. The algorithm can be implemented using either a radix-2
or a radix-4 butterfly structure, which exploits the symmetry and periodicity
properties of the DFT to reduce the number of multiplications and additions
required. In computer vision, the FFT is widely used in image processing for
operations such as image filtering, convolution, and correlation. These
operations involve the computation of the Fourier transform of the image and
the filter kernel, multiplication in the frequency domain, and the inverse
Fourier transform to obtain the filtered image. The FFT algorithm allows these
operations to be performed efficiently on large images in real-time
applications such as video processing and computer vision systems.

  * •

Positional Encodings is used in Natural Language Processing and sequence
modeling to inject the position information of each token in a sequence into
the input representation. The idea is to provide the model with information
about the position of each token, which is crucial for tasks such as language
understanding and translation.

In Transformer-based models, the positional encoding is added to the input
embeddings before being fed to the model. The positional encoding is
calculated based on the position of each token in the sequence and a set of
learned parameters, which encode the position information into a continuous
vector space.

The equation for positional encoding is given in eqn 48 where
$\text{PE}{(pos,2i)}$ and $\text{PE}{(pos,2i+1)}$ are the $2i$-th and
$(2i+1)$-th elements of the positional encoding vector for the $pos$-th
position in the sequence, $d_{\text{model}}$ is the dimensionality of the
model, and $i$ is the index of the embedding dimension.

$\displaystyle
PE_{(pos,2i)}=\sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)$ (48)
$\displaystyle
PE_{(pos,2i+1)}=\cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$

The application of positional encoding is not limited to natural language
processing. It can also be used in computer vision and other domains where
sequential information is important. For example, in image processing,
positional encoding can be used to encode the position information of each
pixel in an image, which can be useful for tasks such as object detection,
segmentation, and classification. In machine learning, positional encoding can
be used to improve the performance of models that work with sequences, such as
time series forecasting and speech recognition.

In NeRF, the input to the neural network is a sequence of 3D points along each
camera ray, which are used to sample the radiance and volume density at each
point. To encode the positional information of each point, NeRF uses a
positional encoding scheme similar to the one used in Transformer-based
models.

The positional encoding in NeRF is calculated by the eqn. 49 given below where
$\text{PE}_{i}$ is the $i$-th element of the positional encoding vector, $d$
is the dimensionality of the positional encoding, and $x$, $y$, and $z$ are
the 3D coordinates of the point. The $\sin$ and $\cos$ terms allow the
encoding to capture both periodic and non-periodic features in the spatial
structure of the scene.

$\displaystyle\text{PE}_{\text{ray}}(i,2k)$
$\displaystyle=\sin\left(\frac{i}{10000^{2k/D}}\right)$ (49)
$\displaystyle\text{PE}_{\text{ray}}(i,2k+1)$
$\displaystyle=\cos\left(\frac{i}{10000^{2k/D}}\right)$

The use of positional encoding in NeRF allows the model to learn a continuous
function that can map any point in the 3D space to its corresponding color and
opacity. This enables the generation of photo-realistic 3D scenes from a set
of 2D images, which can have various applications in fields such as computer
graphics, virtual reality, and augmented reality.

  * •

MHRE (Multi-Resolution Hash Encoding) is used in Computer Science to represent
images using a compact binary code. It is based on the idea of using multiple
image resolutions to get local and global frequency information. Here, the
image is divided into multiple sub-regions and computing a hash code for each
sub-region at different resolutions. Hash code for each subregion is obtained
by comparing the pixel values in the sub-region to a predefined threshold. If
a pixel value is greater than the threshold, the corresponding bit in the hash
code is set to 1, otherwise it is set to 0. The hash codes for all sub-regions
at each resolution level are concatenated to form the final binary code for
the image.

The idea behind MRHE encoding of an image $I$, $i^{\textrm{th}}$ sub-region
hash code $h_{i}$, resolution level $r_{i}$ (with total $m$ resolutions) is
given by the eqn. 50

$\mathcal{MHRE}(I)=[h_{1}^{r_{1}},h_{2}^{r_{1}},...,h_{n}^{r_{1}},h_{1}^{r_{2}},h_{2}^{r_{2}},...,h_{n}^{r_{2}},...,h_{n}^{r_{m}}]$
(50)

The MRHE technique is particularly useful in applications such as image
retrieval, object recognition, and image matching, where large-scale image
databases need to be searched efficiently. The compact binary code generated
by MRHE can be used to quickly compare images and retrieve similar images from
a database. Additionally, the multiple resolutions used in MRHE capture both
local and global information, making it robust to variations in lighting,
texture, and scale.

  * •

Spectral Bias (/F-principle) is a phenomenon that can occur in computer vision
tasks when the data used to train a model has a specific spectral structure
that the model can exploit, leading to better performance than expected based
on its ability to capture the underlying statistical structure of the data.
In computer vision, the Fourier Transform is commonly used to analyze the
frequency domain of an image. The F-Principle states that the performance of a
machine learning model can be improved if the Fourier Transform of the
training data has a specific spectral structure, such as a concentrated or
sparse distribution of frequencies. This spectral structure can be exploited
by the model to learn more efficiently or accurately than if the distribution
of frequencies were more uniformly distributed.
For example, consider a dataset of images of faces. If the images in the
dataset have a similar lighting condition, such as all being taken in bright,
natural light, then the spectral structure of the dataset will be biased
towards certain frequencies related to that lighting condition. If a model is
trained on this dataset, it may learn to exploit this spectral bias and
perform well on images with similar lighting conditions, but may perform
poorly on images with different lighting conditions.
To mitigate the effects of spectral bias, it is important to use diverse and
representative datasets during training, and to evaluate the performance of
the model on out-of-distribution data to ensure it is generalizing well to new
situations. Additionally, techniques such as data augmentation, which
introduces variations in the training data, can help to reduce the impact of
spectral bias and improve the generalization performance of the model.

  * •

NTKs (Neural Tangent Kernels) are a mathematical tool used to analyze the
training dynamics of neural networks. It allows us to understand how the
weights of a neural network change during training and how this affects the
network’s ability to generalize to new data.

In the context of computer vision, NTK has been used to analyze the
performance of neural networks on image classification tasks. Additionally,
NTK has been applied to novel applications such as Neural Radiance Fields
(NeRF) which is a method for synthesizing photo-realistic 3D scenes from 2D
images.

The eqn 51 represents an NTK where $K_{NTK}(x,x^{\prime})$ is the NTK between
input vectors $x$ and $x^{\prime}$, $f(x)$ is the output of the neural network
for input $x$, and $w_{i}$ represents the weights of the neural network.

$\textrm{K}_{NTK}(x,x^{\prime})=\lim_{n\rightarrow\infty}\frac{1}{n}\sum_{i=1}^{n}\frac{\partial
f(x)}{\partial w_{i}}\frac{\partial f(x^{\prime})}{\partial w_{i}}$ (51)

The NTK is computed as the limit of the average product of the partial
derivatives of the network output with respect to the weights over an infinite
number of weights.

In the context of NeRF, NTK is used to analyze the performance of a neural
network that maps 2D images to 3D scenes. By analyzing the NTK, researchers
can gain insight into the relationship between the 2D images and the resulting
3D scene and design more effective neural network architectures. Specifically,
NTK allows for the efficient computation of the Jacobian matrix, which is used
to estimate the gradient of the loss function during training.

  * •

Tracing and Casting are two of the important techniques used in Computer
Vision and Computer Graphics. In Vision, tracing refers to the process of
tracking the movement of objects/points in sequence of images/frames. This can
be done by using various algorithms and techniques, such as optical flow,
feature tracking, and point matching. The goal of tracing in computer vision
is to understand the motion and dynamics of objects in a scene and extract
useful information, such as the velocity, acceleration, and trajectory of the
objects. Whereas in Graphics, tracing refers to the process of generating
photorealistic images by tracing the paths of light rays as they interact with
virtual objects in a scene. This is typically done using ray tracing
algorithms, which simulate the behavior of light as it interacts with objects
in the scene. The goal of tracing in computer graphics is to generate
realistic images that accurately simulate the lighting and materials of the
virtual objects in the scene.
Casting, on the other hand, refers to the process of projecting an object or
point onto a surface or plane in both computer vision and computer graphics.
In computer vision, casting is used to estimate the 3D position of an object
or point in the world from a 2D image or frame. This is typically done using
methods such as perspective projection or homography, which involve modeling
the relationship between the 3D world and the 2D image. The goal of casting in
computer vision is to estimate the position and orientation of objects in the
world from a 2D image or frame.
In computer graphics, casting is used to simulate the behavior of light as it
interacts with objects in a scene. This is typically done using methods such
as shadow casting and light mapping, which involve modeling the way that light
interacts with objects in the scene to create realistic shadows and lighting
effects. The goal of casting in computer graphics is to generate realistic
images that accurately simulate the behavior of light in the scene.

    * –

Path Tracing is a widely used algorithm in computer graphics for rendering
realistic images of 3D scenes. It is a Monte Carlo method that simulates the
behavior of light as it interacts with objects in the scene. The algorithm
works by tracing a path of light rays from the camera through the scene,
randomly sampling the possible paths of light rays as they reflect, refract,
and scatter off of objects in the scene. Path tracing has been used in a wide
range of applications, from film and animation to video game development. One
of the seminal papers on path tracing is work done by Kajiya et al [103],
which introduced the concept of a physically-based model for rendering 3D
scenes. Another influential work by Matt Pharr and Greg Humphreys [171]
provides a comprehensive introduction to the theory and practice of
physically-based rendering, including path tracing.

    * –

Ray Marching is used to sample multiple points from the 3D space rather than
just one point where the ray intersects with the 3D surface. This method is
used when analytical methods fail to provide reliable results. Its used in
computer graphics to render 3D scenes using volumetric databy tracing a ray
through a volume and sampling the density and color of the volume along the
ray’s path. Ray marching is commonly used to generate realistic images of
clouds, smoke, and other volumetric phenomena. Seminal works on ray marching
[121, 53] introduced the concept of volume rendering and the ray marching
algorithm. Another work by William E. Lorensen and Harvey E. Cline [168]
proposed a variant of ray marching for generating isosurfaces of volumetric
data.

    * –

Ray Tracing is a technique used for 3D offline rendering. It is achieved by
recursively tracing the path of rays through a scene. It tracing is a
technique used in graphics and vision to generate realistic images by tracing
the path of light as it interacts with objects in a scene. It works by tracing
rays of light from the camera through each pixel and calculating how the light
interacts with objects in the scene, including reflection, refraction, and
shadows. The first work to introduce ray tracing concept was by Turner Whitted
[218] as it presented a recursive algorithm for computing reflections and
refractions.

    * –

Distributed Ray Tracing is used in computer graphics and computer vision to
generate realistic images by simulating the propagation of light in a scene
using a distributed network of processors. It involves dividing the rendering
process into smaller tasks, distributing them across multiple processors, and
then combining the results to produce a final image. It is achieved by some
modifications to the process of Ray Tracing where multiple rays are traced
through each pixel to model soft phenomenon such as soft shadows, depth of
fields. One of the first works to introduce this concept was presented by Cook
et al [38] and it initiated the concept of parallelizing the ray tracing
algorithm across multiple processors.

    * –

Beam Tracing uses pyramid-shaped beams to address some of the shortcomings of
the ray-tracing. These shortcomings include Aliasing effects and the creation
of artifacts while rendering.

    * –

Ray Casting is generally a 2.5 D rendering method and is achieved by casting
non-recursive rays from the camera into the scene.

    * –

Sphere Tracing, also known as raymarching or distance field raymarching, is an
algorithm for rendering 3D objects using signed distance functions (SDFs). An
SDF is a function that, for any point in space, returns the shortest distance
to the surface of the object.

    * –

Marching Cube is a method for implicit surface triangulation. So, it is an
algorithm for generating polygonal meshes from 3D scalar fields, such as voxel
data or isosurfaces. The algorithm works by dividing the volume into small
cubes and determining the surface intersection points within each cube.

The following steps constitute the Marching Cube algorithm.

      1. 1.

Divide the volume into small cubes.

      2. 2.

For each cube, evaluate the scalar field at its 8 vertices.

      3. 3.

Determine the surface intersections by comparing the scalar values at the
vertices to a predefined isovalue.

      4. 4.

Create triangles to connect the intersection points within each cube.

      5. 5.

Combine the triangles to form the final polygonal mesh.

The mathematical formulation for each step is as follows.

      1. 1.

Let V(x, y, z) be a scalar field representing the 3D volume.

      2. 2.

For each cube vertex, evaluate the scalar field: v = V(x, y, z)

      3. 3.

Create a bitmask by comparing v to the isovalue.

      4. 4.

Use the bitmask to look up the corresponding edge intersections.

      5. 5.

Calculate the intersection points: I = (p1 + p2) / 2, where p1 and p2 are the
endpoints of the intersecting edge.

      6. 6.

Generate triangles connecting the intersection points within the cube.

Applications for Marching Cube include

      * *

Medical imaging and visualization (CT, MRI, etc.)

      * *

Geological data visualization

      * *

Fluid dynamics simulations

      * *

Terrain generation in video games and virtual environments

  * •

SLAM It is a technique often used at the intersection of Robotics and Computer
Vision. It further helps with several simple or complex Robotic operations. It
generally requires the leverage of C++ to work with robots for manipulation,
localization, path finding, etc.

  * •

RGB-D Registration is the use of Depth along with RGB features to register for
different application such as Nuclear Magnetic Resonance Imaging (NMRI), CT
scans, Geospatial imaging (when non-azimuthal orbit), etc.

  * •

DeepV2D (Deep Video to Depth) is a deep learning-based method for estimating
depth maps from a video sequence, leveraging differentiable Structure-from-
Motion (SfM) to improve the overall performance. The key idea of DeepV2D is to
integrate the geometric principles of SfM with a deep learning framework to
jointly optimize for depth, camera motion, and 3D structure. This approach
allows the model to learn depth estimation while enforcing geometric
consistency across multiple frames in a video sequence.

The main components of the DeepV2D framework are:

    1. 1.

CNN-based Depth Estimation: A convolutional neural network (CNN) is used to
estimate the initial depth maps for each frame in the video sequence. This
network is trained to predict depth maps from single images, but its
predictions may not be geometrically consistent across frames.

    2. 2.

Differentiable SfM: The differentiable SfM module enforces geometric
consistency across frames by optimizing the camera motion (poses) and refining
the depth estimates. The module consists of differentiable components for
camera pose estimation, warping, and a photometric loss function. The
differentiability of the SfM module allows the gradients to flow back through
the entire pipeline during training, improving both depth estimation and
camera motion estimation.

    3. 3.

Joint Optimization: DeepV2D jointly optimizes depth maps, camera poses, and 3D
structure by minimizing a combination of photometric and geometric loss
functions. The photometric loss measures the difference between the input
images and the images reconstructed by warping neighboring frames using the
estimated depth and camera poses. The geometric loss enforces smoothness in
the depth maps and penalizes large deviations from the initial CNN-based depth
predictions.

DeepV2D is trained in an end-to-end manner using backpropagation, allowing the
model to learn depth estimation and camera motion jointly. The integration of
differentiable SfM in the deep learning framework helps the model to enforce
geometric consistency across video frames and improve the overall performance
of depth estimation.

  * •

COLMAP is a library written in C++ for two of the vital operations in Computer
Vision and Computer Graphics namely, MVS and SfM which have been discussed
below.

    * –

MVS (Multi-View Stereo) is a computer vision technique used to reconstruct 3D
geometry from multiple 2D images taken from different viewpoints. It builds on
the principles of stereo vision, which involves estimating the depth of a
scene by comparing the slight differences in appearance between two or more
images, known as parallax.

The main steps involved in Multi-View Stereo are:

      1. 1.

Image Acquisition: Capture a set of images of a scene or object from multiple
viewpoints, ensuring sufficient overlap and coverage. Ideally, the camera’s
intrinsic (focal length, principal point, etc.) and extrinsic (position and
orientation) parameters should be known or estimated.

      2. 2.

Feature Extraction and Matching: For each image, identify distinctive points
or features and match them across the different images. This can be achieved
using feature detectors and descriptors, such as SIFT, SURF, ORB, etc.

      3. 3.

Pairwise Stereo Matching: Compute the depth or disparity maps for each pair of
images, based on the feature matches. This can be done using block-matching,
dynamic programming, or graph-cut algorithms, for example.

      4. 4.

3D Reconstruction: Merge the depth or disparity maps from all image pairs into
a single, consistent 3D model. This can involve techniques like depth map
fusion, volumetric reconstruction, or point cloud merging.

      5. 5.

Post-processing (optional): Refine the 3D model by applying methods like
meshing, texturing, and hole-filling to improve the visual quality and
completeness of the result.

Multi-View Stereo has various applications, including:

      * *

3D scanning and modeling of objects and environments

      * *

Photogrammetry

      * *

Augmented and virtual reality

      * *

SLAM (Robotics and autonomous navigation)

      * *

Cultural heritage preservation

In summary, Multi-View Stereo is a technique used to reconstruct 3D geometry
from multiple 2D images taken from different viewpoints. It involves feature
extraction and matching, pairwise stereo matching, and 3D reconstruction, and
can be applied in diverse fields such as photogrammetry, augmented reality,
and robotics.

    * –

SfM (Structure from Motion) is a computer vision technique used to estimate
the 3D structure of a scene and the motion (camera positions and orientations)
from a set of 2D images taken from different viewpoints. It combines elements
of both feature-based image matching and 3D reconstruction to create a sparse
3D point cloud or a more detailed 3D model.

The main steps involved in Structure-from-Motion are:

      * *

Image Acquisition: Capture a set of images of a scene or object from multiple
viewpoints, ensuring sufficient overlap and coverage.

      * *

Feature Extraction and Matching: For each image, identify distinctive points
or features and match them across the different images. This can be achieved
using feature detectors and descriptors, such as SIFT, SURF, ORB, etc.

      * *

Estimating Camera Poses and 3D Structure: Use the feature matches to
simultaneously estimate the camera’s extrinsic parameters (position and
orientation) and the 3D coordinates of the features in the scene. This can be
done using algorithms like bundle adjustment, which minimizes the reprojection
error between the observed 2D feature positions and the reprojected 3D points.

      * *

Incremental Reconstruction: Add images one at a time, updating the camera
poses and 3D structure iteratively. This process is known as incremental SfM
and helps maintain accuracy and consistency in the reconstruction.

      * *

Dense Reconstruction (optional): Perform a dense Multi-View Stereo (MVS) on
the sparse 3D point cloud obtained from SfM to generate a more detailed 3D
model.

      * *

Post-processing (optional): Refine the 3D model by applying methods like
meshing, texturing, and hole-filling to improve the visual quality and
completeness of the result.

In summary, Structure-from-Motion is a technique used to estimate the 3D
structure of a scene and the motion (camera positions and orientations) from a
set of 2D images taken from different viewpoints. It involves feature
extraction and matching, camera pose and 3D structure estimation, and optional
dense reconstruction and post-processing, and can be applied in diverse fields
such as photogrammetry, augmented reality, and robotics.

  * •

Structured Light Scanning and Rendering is a 3D operation for getting RGB-D
data using LASERs to scan the surface of the object which needs to be sampled
for the data. The LiDAR technology works on these principles. LiDAR is often
adjusted on cars for autonomous driving capabilities.

  * •

Time-of-Flight Scanning refers to the scanning carried out using planes or
quadcopters for the purpose of understanding the terrain under the position
where the plane/drone is flying.

  * •

Jitter refers to the noise while streaming a sequence of images using
different medium. This can restrain the company from getting optimal sampled
inp

  * •

Image and Facial Features

    * –

SIFT (Scale-Invariant Feature Transform) [137] is a feature extraction method
proposed by David Lowe in 1999. It is designed to be invariant to scale,
rotation, and illumination changes. The main steps of the SIFT algorithm are:

      1. 1.

Scale-space extrema detection: Identify potential interest points by searching
for local maxima and minima across multiple scales.

      2. 2.

Keypoint localization: Refine the candidate keypoints by eliminating low-
contrast and poorly localized points.

      3. 3.

Orientation assignment: Assign a dominant orientation to each keypoint based
on local gradient histograms.

      4. 4.

Keypoint descriptor computation: Create a feature descriptor for each keypoint
using histograms of gradient orientations in the keypoint’s local
neighborhood.

    * –

SURF (Speeded-Up Robust Features), proposed by Herbert Bay et al. in 2006, is
an image feature extraction method inspired by SIFT but designed to be faster
and more efficient. The main steps of the SURF algorithm are:

      1. 1.

Interest point detection: Use the Hessian matrix to detect blob-like
structures in the image at multiple scales.

      2. 2.

Keypoint localization: Identify the local maxima and minima of the determinant
of the Hessian matrix as potential keypoints.

      3. 3.

Orientation assignment: Assign a dominant orientation to each keypoint based
on the Haar wavelet responses in its local neighborhood.

      4. 4.

Keypoint descriptor computation: Create a feature descriptor for each keypoint
using Haar wavelet responses in the keypoint’s local neighborhood.

    * –

ORB (Oriented FAST and Rotated BRIEF) proposed by Ethan Rublee et al. in 2011,
is a fast binary descriptor that combines the strengths of the FAST keypoint
detector and the BRIEF descriptor. The main steps of the ORB algorithm are:

      1. 1.

Interest point detection: Use the Hessian matrix to detect blob-like
structures in the image at multiple scales.

      2. 2.

Keypoint detection: Use the FAST (Features from Accelerated Segment Test)
corner detector to identify interest points in the image.

      3. 3.

Orientation assignment: Compute an orientation for each keypoint based on the
intensity centroid of its local neighborhood.

      4. 4.

Keypoint descriptor computation: Create a feature descriptor for each keypoint
using the BRIEF (Binary Robust Independent Elementary Features) descriptor,
which is modified to be rotation-invariant.

    * –

3DMM feature refer to 62D features for the face alignment (yaw, pitch, roll)
that helps to understand the face of the person being considered for the
sampled input. 3DMM, introduced by Volker Blanz and Thomas Vetter in 1999, is
a statistical model that represents the 3D shape and texture of human faces.
The 3DMM is generated by analyzing a dataset of 3D face scans, capturing the
variations in shape and texture across individuals. The main components of a
3DMM are:

      1. 1.

Shape model: A principal component analysis (PCA) is applied to the 3D facial
geometry data to create a linear shape model, which can represent different
face shapes as a combination of the mean face shape and a set of principal
components.

      2. 2.

Texture model: Similarly, PCA is applied to the texture data (e.g., color or
reflectance) of the 3D face scans to create a linear texture model.

      3. 3.

Model fitting: Given a 2D image or a set of images, the 3DMM can be fit to the
input data by estimating the shape and texture coefficients, as well as the
camera parameters and illumination conditions, that minimize the discrepancy
between the input

  * •

RANSAC (RANdom SAmple Consensus) is an iterative algorithm used in the field
of computer vision and other areas for robustly estimating model parameters
from a dataset that contains outliers. The main idea behind RANSAC is to
repeatedly select a random subset of the data points, fit the model to this
subset, and evaluate the quality of the fit. The model with the highest
consensus from the data points is chosen as the final result. The RANSAC
algorithm consists of the following steps:

    * –

Randomly select a minimal subset of data points: Choose the minimum number of
data points required to fit the model (e.g., 2 points for a line, 3 points for
a plane, etc.).

    * –

Fit the model: Estimate the model parameters using the selected data points.

    * –

Evaluate the model: Calculate the error between the model’s predictions and
the remaining data points. If the error is below a predefined threshold,
consider the data point as an inlier, otherwise, as an outlier.

    * –

Count the inliers: Determine the number of inliers that support the fitted
model.

    * –

Iterate: Repeat steps 1-4 a predefined number of times or until a stopping
criterion is met (e.g., the proportion of inliers exceeds a threshold or the
maximum number of iterations is reached).

    * –

Refine the model: Fit the model again using all inliers from the best
iteration to obtain the final model parameters.

RANSAC is particularly useful when dealing with data that contains a
significant number of outliers, as it is resistant to their influence. The
algorithm is widely used in computer vision tasks, such as feature matching,
homography estimation, fundamental matrix estimation, and camera pose
estimation.

Hence, RANSAC is an iterative algorithm for robustly estimating model
parameters from a dataset with outliers. It works by repeatedly selecting a
random subset of data points, fitting the model, and evaluating the consensus
of the data points. The model with the highest consensus is chosen as the
final result. RANSAC is widely used in computer vision and other fields where
robust parameter estimation is required.

  * •

Selective Search is a region proposal algorithm used in the context of
computer vision, specifically for object detection tasks. The main idea behind
Selective Search is to generate potential bounding boxes or regions of
interest that are likely to contain objects in an image. These regions are
then used as input for a classifier or a more complex detection pipeline to
identify and classify objects.

Selective Search combines the strengths of both exhaustive search and
segmentation to create a diverse set of region proposals. The algorithm is
based on the assumption that objects can be identified at different scales and
are often characterized by distinct colors, textures, or intensity patterns.

The main steps of Selective Search are:

    1. 1.

Generate initial regions: Perform an initial over-segmentation of the image
using a simple method like the Felzenszwalb and Huttenlocher’s graph-based
segmentation algorithm. This step creates many small, highly detailed
segments.

    2. 2.

Extract features: For each segment, compute features like color, texture, and
shape.

    3. 3.

Hierarchical grouping: Recursively merge the most similar segments based on a
similarity measure computed from the extracted features. This step creates a
hierarchical structure of regions, where each level represents a different
granularity or scale.

    4. 4.

Generate region proposals: Collect regions from different levels of the
hierarchy and use non-maximum suppression to remove redundant or overlapping
proposals. This step produces a diverse set of potential object locations.

Selective Search is particularly useful as a preprocessing step for object
detection algorithms, as it reduces the number of candidate regions that need
to be processed by the actual object detector. This can significantly speed up
the detection process and improve the overall performance. Selective Search
was widely used in combination with traditional object detectors, like those
based on the Histogram of Oriented Gradients (HOG) features, and early deep
learning-based detectors, like R-CNN.

  * •

Bundle-Adjustment is an optimization technique used in computer vision and
photogrammetry for refining estimates of the 3D structure of a scene and the
camera parameters (poses and intrinsics) from a set of overlapping images. It
is often employed as a final step in Structure-from-Motion (SfM) and Multi-
View Stereo (MVS) pipelines to improve the accuracy and consistency of the 3D
reconstruction.

The main idea behind Bundle Adjustment is to minimize the reprojection error,
which is the difference between the observed 2D positions of features in the
images and the reprojected 3D points onto the images using the estimated
camera parameters and 3D structure. Bundle Adjustment is a non-linear least
squares optimization problem, as the relationship between the camera
parameters, 3D structure, and image measurements is non-linear.

The main components of Bundle Adjustment are:

    * –

Camera parameters: The intrinsic (e.g., focal length, principal point, lens
distortion) and extrinsic (e.g., rotation and translation) parameters of each
camera in the scene.

    * –

3D structure: The 3D coordinates of the scene points that are visible in the
images.

    * –

Image measurements: The 2D coordinates of the features in each image, usually
obtained through feature matching.

    * –

Objective function: The sum of the squared reprojection errors between the
observed image measurements and the reprojected 3D points, given the current
estimates of the camera parameters and 3D structure.

    * –

Optimization algorithm: A non-linear least squares solver, such as the
Levenberg-Marquardt algorithm, is used to iteratively minimize the objective
function.

So, Bundle Adjustment is an optimization technique used to refine the
estimates of the 3D structure of a scene and the camera parameters by
minimizing the reprojection error. It is widely employed in computer vision
and photogrammetry pipelines, such as Structure-from-Motion and Multi-View
Stereo, to improve the accuracy and consistency of the 3D reconstruction.

  * •

Shepard Metzler objects are complex 3D shapes, originally introduced by
psychologists Roger Shepard and Jacqueline Metzler in a 1971 study on mental
rotation. In the context of computer vision, these objects have been adopted
as a tool to assess and evaluate the performance of models in tasks related to
3D understanding, such as 3D object recognition, pose estimation, and
unsupervised feature learning.

Shepard-Metzler objects are typically constructed by combining several simple
3D shapes, such as cubes or cylinders, into a single complex structure. The
resulting objects exhibit a high degree of variability in terms of their
geometry, making them well-suited for testing the ability of computer vision
algorithms to recognize, analyze, and understand 3D structures.

Some common use cases of Shepard-Metzler objects in computer vision research
include:

    * –

Benchmarking: Researchers use Shepard-Metzler objects as a benchmark dataset
to evaluate the performance of their algorithms in tasks like 3D object
recognition and pose estimation. By using these complex objects, they can test
the robustness and generalization capabilities of their models.

    * –

Unsupervised feature learning: Shepard-Metzler objects are used to study
unsupervised feature learning in the context of 3D understanding. Researchers
may train models to learn meaningful representations of these objects, which
can then be used for downstream tasks such as classification, segmentation, or
reconstruction.

    * –

Mental rotation tasks: Inspired by the original psychological study, computer
vision researchers can use Shepard-Metzler objects to explore how models
perform mental rotation tasks, which involve determining whether two 3D
objects are the same when they are presented at different orientations.

  * •

Super-Resolution Tasks aim to enhance the quality of an image or recover
missing or degraded information. While super-resolution often refers to
increasing the spatial resolution of an image, it also encompasses other image
enhancement tasks like denoising, dehazing, deblurring, and inpainting. These
tasks address different types of image degradation and improve the overall
quality and visual appearance of an image.

    * –

Image Denoising is the process of removing noise from an image while
preserving its original details. Noise can be introduced during image
acquisition (e.g., due to sensor limitations) or transmission (e.g.,
compression artifacts). Denoising techniques range from traditional filtering
methods, such as Gaussian filters or non-local means, to more advanced deep
learning-based approaches, like denoising autoencoders and convolutional
neural networks.

    * –

Image Dehazing refers to the removal of haze or fog from an image to improve
its visibility and contrast. Haze is caused by the scattering of light due to
the presence of atmospheric particles, which can result in low contrast and
color shift. Dehazing methods usually estimate the transmission map and
atmospheric light to recover the haze-free image. Classical methods include
Dark Channel Prior and Guided Filtering, while deep learning-based methods
leverage convolutional neural networks for end-to-end dehazing.

    * –

Image Deblurring aims to recover a sharp image from a blurred one, typically
caused by camera shake, object motion, or out-of-focus effects. Deblurring
techniques estimate the blur kernel (also known as point spread function) and
use it to restore the sharp image. Traditional methods include blind
deconvolution and Wiener filtering, while deep learning-based methods utilize
convolutional neural networks to either directly estimate the sharp image or
the blur kernel.

    * –

Image Inpainting is the process of filling in missing or corrupted parts of an
image in a visually plausible manner. This task requires not only filling in
the missing regions but also preserving the structure and texture of the
surrounding areas. Classical inpainting methods include diffusion-based
techniques and patch-based approaches like exemplar-based inpainting. More
recent deep learning-based methods leverage generative models, such as
autoencoders and generative adversarial networks (GANs), to learn the
underlying structure and texture patterns and generate plausible content for
the missing regions.

    * –

SISR (Single Image Super-Resolution) aims to recover a high-resolution (HR)
image from a single low-resolution (LR) input image. Traditional methods
include interpolation-based techniques (e.g., bicubic interpolation, Lanczos
resampling) and learning-based approaches (e.g., sparse coding). More
recently, deep learning-based methods, such as super-resolution convolutional
neural networks (SRCNN) and generative adversarial networks (GANs), have
demonstrated significant improvements in reconstructing HR images with more
accurate textures and details.

    * –

Video super-resolution aims to improve the spatial resolution of video
sequences while maintaining temporal consistency. This task presents
additional challenges due to the need to consider motion between frames and
avoid flickering artifacts. Techniques for video super-resolution include
multi-frame methods, which exploit the temporal information present in
multiple frames, and deep learning-based methods, which use convolutional
neural networks (CNNs) or recurrent neural networks (RNNs) to model the
spatial and temporal relationships between frames.

    * –

Image colorization is the process of adding color to grayscale images while
preserving their original structure and texture. Traditional methods often
require user input in the form of color scribbles or reference images to guide
the colorization process. Deep learning-based approaches, such as
convolutional neural networks (CNNs) and generative adversarial networks
(GANs), have shown the ability to automatically learn color patterns and
generate plausible colorizations without user intervention.

    * –

Image-to-Image translation is a more general task that involves converting an
input image from one domain or representation to another, such as converting a
sketch to a photorealistic image or transforming a day-time scene to a night-
time scene. Deep learning-based methods, such as conditional GANs (e.g.,
Pix2Pix) and cycle-consistent GANs (e.g., CycleGAN), have been successful in
learning the mapping between different image domains without requiring paired
training data.

    * –

Depth estimation is the task of recovering depth information from 2D images,
which can be used for various applications like 3D reconstruction, autonomous
navigation, or augmented reality. Traditional methods typically rely on stereo
vision or structured light, while more recent deep learning-based approaches
use convolutional neural networks (CNNs) to predict depth maps directly from
single images or monocular video sequences.

  * •

Image Warping, also known as image transformation or image registration, is a
process used in computer vision and graphics to manipulate an image’s shape,
size, and orientation by applying a transformation to its pixel coordinates.
The goal of image warping is to change the appearance of an image while
preserving its structure and content. It is commonly used in tasks such as
image stitching, rectification, stabilization, and view synthesis.

Image warping can be performed using different types of transformations,
including:

    1. 1.

Rigid Transformations: These transformations preserve the distances between
points and include rotation, translation, and their combination. Rigid
transformations are often used in image registration to align images taken
from different viewpoints or at different times.

    2. 2.

Affine Transformations: Affine transformations extend rigid transformations by
adding scaling and shearing effects. They preserve parallelism but not
necessarily angles or distances. Affine transformations are commonly used in
tasks like image rectification, where the goal is to remove perspective
distortion and align parallel lines in the image.

    3. 3.

Homographies: Homographies are projective transformations that map points in
one image plane to another while preserving straight lines. They can represent
a wide range of transformations, including rotation, translation, scaling,
shearing, and perspective distortion. Homographies are frequently used in
tasks like image stitching, where multiple images of a scene are combined into
a single panoramic image, and image rectification.

    4. 4.

Non-linear or Deformable Transformations: These transformations can model
complex, non-linear changes in the image geometry, such as bending,
stretching, or twisting. Non-linear transformations are often employed in
tasks like image morphing, where one image is smoothly transformed into
another, or in deformable registration, where images with non-rigid structures
need to be aligned.

To apply a transformation to an image, a mapping function is used to determine
the correspondence between the source image’s pixel coordinates and the
destination image’s pixel coordinates. Then, a resampling or interpolation
method, such as nearest-neighbor, bilinear, or bicubic interpolation, is
applied to estimate the pixel values in the destination image.

  * •

Decal sheet or Texture Atlas is a “sticker” picture that is applied onto a 3D
mesh to represent a texture.

  * •

UV-mapping is the process of 3D surface projection to a 2D image for texture
mapping. Here, $U$ and $V$ represents the axes of 2D texture because in
Computer Graphics, $X,Y,Z$ are used to denote axes of 3D surfaces, points,
objects and $W$ is used to denote calculation of Quarternions rotations. In UV
texture map [151], an ordinary image can be used to paint a 3D surface with
color (much like what is done in Unity). The UV mapping technique involves
”programmatically” copying a triangle-shaped section of the image map and
pasting it onto a triangle on the object to assign pixels in the picture to
surface mappings on the polygon [152]. UV texturing is an alternative to
projection mapping; it simply maps into a texture space rather than the
physical space of the object (e.g., utilizing any pair of the model’s X, Y,
and Z coordinates or any modification of the position). The UV texture
coordinates are used in the rendering computation to specify how to paint the
three-dimensional surface.

Figure 2: A schematic representation of an application of a texture in the UV
space related to effect in 3D space

UV coordinates can be generated for each vertex of the mesh for a model
created as a polygon mesh. A way to achieve this is by unfolding the
triangular meshes at the seams, hence, laying out triangles on flat surface
automatically. A UV sphere might be transformed into an equirectangular
projection. The textures can be painted on the triangle individually after the
model is unwrapped and when rendering the 3D object, it can take up the
appropriate texture from the texture atlas.
UV coordinates are applied to each face optionally [152] so a shared spatial
vertex position can have different UV coordinates for each of its triangles so
adjacent triangles may be cut apart and positioned on texture maps’ different
areas. For finding the UV coordinates of a point $\hat{A}$ on a sphere with
$\hat{d}$ (distance of the point from the sphere’s origin) when sphere’s poles
are aligned with the $Y$, then the eq. 52

$u=0.5+\frac{\textrm{arctan2}(d_{z},d_{x})}{2\pi},v=0.5+\frac{\textrm{arcsin}(d_{y})}{\pi}$
(52)

Hence, the UV-mapping process can be described in simple three stage approach
where the mesh is first uwrapped, a texture is created and applied to the
respective face of the polyon.

  * •

Snell’s Law, also known as the Law of Refraction, describes the relationship
between the angles of incidence and refraction for a light wave passing
through the interface between two media with different refractive indices. It
is named after Dutch mathematician Willebrord Snellius, who formulated the law
in 1621.

Snell’s Law states that the ratio of the sine of the angle of incidence
($\theta_{1}$) to the sine of the angle of refraction ($\theta_{2}$) is equal
to the ratio of the refractive indices of the two media:

$\eta_{1}*\sin{\theta_{1}}=\eta_{2}*\sin{\theta_{2}}$ (53)

where $\eta_{1},\eta_{2}$ are the refractive indices of the first and second
media, respectively. The refractive index of a medium is a measure of how much
the speed of light is reduced inside the medium compared to its speed in a
vacuum. $\theta_{1}$ is the angle of incidence, which is the angle between the
incident light ray and the normal (a line perpendicular to the interface) at
the point where the light ray enters the second medium. $\theta_{2}$ is the
angle of refraction, which is the angle between the refracted light ray and
the normal at the point where the light ray enters the second medium. Snell’s
Law is a fundamental principle in optics and is used to explain various
phenomena such as refraction, total internal reflection, and the formation of
images by lenses. It is also employed in the design and analysis of optical
systems, including cameras, microscopes, and fiber optics.

  * •

Fresnel’s Law, also known as Fresnel’s Equations, describes the behavior of
light when it encounters the interface between two different media, such as
air and glass. It is named after French physicist Augustin-Jean Fresnel, who
developed the equations in the early 19th century. Fresnel’s Law helps explain
how light is reflected and transmitted (refracted) at the interface between
two media with different refractive indices. These equations are fundamental
to understanding various optical phenomena and are used in the design and
analysis of optical systems.

Fresnel’s Law consists of two equations that describe the reflection (eqn 54)
and transmission (eqn 55) coefficients for the parallel ($p$) and
perpendicular ($s$) components of the electric field of an electromagnetic
wave as described by the eqn 56.

    * –

Reflection coefficients:

$\displaystyle
R_{s}=\left(\frac{\eta_{1}*\cos{\theta_{1}}-\eta_{2}*\cos{\theta_{2}}}{\eta_{1}*\cos{\theta_{1}}+\eta_{2}*\cos{\theta_{2}}}\right)^{2}$
(54) $\displaystyle
R_{p}=\left(\frac{\eta_{1}*\cos{\theta_{2}}-\eta_{2}*\cos{\theta_{1}}}{\eta_{1}*\cos{\theta_{2}}+\eta_{2}*\cos{\theta_{1}}}\right)^{2}$

    * –

Transmission coefficients:

$\displaystyle T_{s}=1-R_{s}$ (55) $\displaystyle T_{p}=1-R_{p}$

The total reflection and transmission coefficients can be computed as the
average of the parallel ($p$) and perpendicular ($s$) components:

$\displaystyle R=(R_{s}+R_{p})/2$ (56) $\displaystyle T=(T_{s}+T_{p})/2$

These coefficients represent the fraction of the incident light that is
reflected and transmitted at the interface between the two media.

Fresnel’s Law is essential for understanding various optical phenomena such as
reflection, refraction, total internal reflection, and the behavior of
polarized light. It is also used in the design and analysis of optical
systems, including lenses, mirrors, coatings, and filters.

(Work in Progress…)

##  3 History of Neural Rendering and NeRFs

The inception of works in Novel View Synthesis can be traced back to early
research in computer graphics and computer vision, where researchers started
to explore the use of 3D models and rendering which were later adapted for
synthesizing novel views of a scene. One such work for rendering “Hierarchical
geometric models for visible surface algorithms” [36] allowed for more
efficient visibility determination and rendering of complex scenes. Although
this paper focused on rendering and not on novel view synthesis, the concepts
and techniques presented later had a significant impact on the development of
model-based novel view synthesis. A work [66] that was influence by this early
work introduced an IBR (image-based rendering) technique that used a
hierarchical data structure to represent the scene radiance as a function of
position and direction efficiently. It leveraged the idea of hierarchical
geometric models to render efficiently and determine visibility. It allowed
for the synthesis of novel views from a set of input images without requiring
explicit 3D geometry, making it an early powerful method for view synthesis
and image-based rendering.

Some attempts of Model-free Novel View Synthesis from Images Rendering date
back to 1990s during the time when the foundational book “Digital Image
Warping” [231] came out, providing a comprehensive overview of image warping
and morphing techniques. This book presented various image transformation
techniques, including geometric transformations, spatial transformations, and
image resampling. It covered the mathematical foundations and algorithms
behind all these techniques, such as coordinate transformations, interpolation
methods, and sampling theory. The concepts presented in this book have laid
the groundwork for various novel view synthesis methods, as they require image
warping to generate new views from input images. By offering both theoretical
background and practical examples, ”Digital Image Warping” has served as a
valuable resource for researchers and practitioners in computer graphics,
image processing, and computer vision, influencing the development of numerous
techniques and algorithms.

The book “Three-dimensional computer vision: a geometric viewpoint” [57]
introduced the geometric aspects of 3D computer vision comprehensively by
building on mathematical foundations of projective geometry, camera models,
and estimation techniques essential for many computer vision tasks. It further
presented various methods for recovering the 3D structure of a scene from
images, such as stereo vision, SfM, and SfS (shape from shading). The
geometric concepts and techniques it covered became the basis of many view
synthesis algorithms. By providing a thorough understanding of the geometric
principles underlying 3D computer vision, it enabled researchers to develop
more advanced and accurate methods for tasks such as novel view synthesis,
scene reconstruction, and object recognition. Both these books contributed
vitally to the field of computer vision and novel view synthesis by providing
foundational knowledge on image warping and 3D geometry. They influenced and
progressed the development of numerous techniques and algorithms in computer
vision, laying the groundwork for researchers to explore more advanced and
sophisticated methods for novel view synthesis and related tasks.

The “Digital Image Warping” [231] delved into practical aspects and
applications of image warping, such as image registration, image mosaicking,
and image morphing. Later works like Feature-based Image Metamorphosis [12],
View Interpolation for Image Synthesis [29], Image metamorphosis using snakes
and free-form deformations [120], Representation of scenes from collections of
images [116], and View Morphing [190], were heavily inspired by this work. At
that time, Feature-based Image Metamorphosis had been used to transform one
image into another by smoothly interpolating the features of the source and
target images. The metamorphosis process involves establishing correspondences
between key features in the source and target images, interpolating the
geometric and color information, and blending the two images over time to
create a smooth transition. Although it wasn’t a direct technique for novel
view synthesis, it was applied to this task with minor modifications such as
capturing multiple images of the scene, estimation of camera parameters (SfM
or Bundle Adjustment can be used), Identifying feature correspondences (SIFT,
SURF, or ORB), estimating Novel views (interpolating the camera parameters of
the input images), warping of input images (warping process leverage
techniques like piecewise affine warping or thin-plate splines), and finally
blending warped images (weighted blending).

An image-based rendering technique called view interpolation was introduced in
“View Interpolation for Image Synthesis” [29] which allowed for the generation
of novel views of a scene from a sparse set of input images. The method worked
by estimating the depth information between the input images, followed by
warping the images to the desired viewpoint, and finally combining the warped
images using a blending technique. This approach reduced the need for explicit
3D models and enables the synthesis of novel views with relatively simple
computations. Later, “Representation of scenes from collections of images”
[116] proposed a technique to construct a 3D representation of a scene from a
collection of images taken from different viewpoints. It involved estimating
the geometry of the scene using stereo vision techniques, constructing a 3D
mesh based on the depth information, and using image warping to project the
textures onto the mesh. The scene representation would then be used for
various applications, such as view synthesis or object recognition. “Image
metamorphosis using snakes and free-form deformations” [120] shared some
similarities with “Digital Image Warping” and presented an alternative method
for image morphing to combines snakes (active contours) and free-form
deformations to produce smooth and visually appealing transitions between
images. It involved using snakes (instead of features) to establish
correspondences between the source and target images and using free-form
deformations to interpolate the features and warp the images. It addressed
some of the limitations of traditional feature-based morphing techniques and
improves the quality of the morphing results. “View Morphing” [190] is an
image-based rendering technique that generates novel views of a scene by
interpolating between two or more input images. The approach involves
estimating the scene geometry, warping the input images to align them with the
desired viewpoint, and blending the images to create a smooth transition
between the viewpoints. The technique is able to handle occlusions and
disocclusions in the scene and can be used for various applications, such as
view synthesis, image mosaicking, and video summarization.

“3-D scene representation as a collection of images” proposed to use a
collection of images taken from different viewpoints, along with their camera
parameters, to represent a 3D scene. They develop a methodology for
registering these images in a common coordinate system and use this
registration to synthesize novel views of the scene by interpolating between
the input images. The approach is image-based, as it does not rely on explicit
3D geometry but rather uses the input images and their associated camera
parameters to synthesize new views of the scene. By focusing on the input