# NeuVV: Neural Volumetric Videos with Immersive Rendering and Editing

Jiakai Zhang  0000-0001-9477-3159 ShanghaiTech UniversityShanghaiChina Stereye
Intelligent Technology Co.,Ltd.China<EMAIL_ADDRESS>,  Liao Wang
ShanghaiTech UniversityShanghaiChina<EMAIL_ADDRESS>,  Xinhang Liu
ShanghaiTech UniversityShanghaiChina<EMAIL_ADDRESS>,  Fuqiang
Zhao  ShanghaiTech UniversityShanghaiChina<EMAIL_ADDRESS>,
Minzhang Li  ShanghaiTech UniversityShanghaiChina<EMAIL_ADDRESS>,
Haizhao Dai  ShanghaiTech UniversityShanghaiChina<EMAIL_ADDRESS>,
Boyuan Zhang  ShanghaiTech UniversityShanghaiChina<EMAIL_ADDRESS>,  Wei Yang  Huazhong University of Science and TechnologyWuhanChina
<EMAIL_ADDRESS>,  Lan Xu  ShanghaiTech UniversityShanghaiChina
<EMAIL_ADDRESS>and  Jingyi Yu  ShanghaiTech
UniversityShanghaiChina<EMAIL_ADDRESS>
###### Abstract.

Some of the most exciting experiences that Metaverse promises to offer, for
instance, live interactions with virtual characters in virtual environments,
require real-time photo-realistic rendering. 3D reconstruction approaches to
rendering, active or passive, still require extensive cleanup work to fix the
meshes or point clouds. In this paper, we present a neural volumography
technique called neural volumetric video or NeuVV to support immersive,
interactive, and spatial-temporal rendering of volumetric video contents with
photo-realism and in real-time. The core of NeuVV is to efficiently encode a
dynamic neural radiance field (NeRF) (Mildenhall et al., 2020) into renderable
and editable primitives. We introduce two types of factorization schemes: a
hyper-spherical harmonics (HH) decomposition for modeling smooth color
variations over space and time and a learnable basis representation for
modeling abrupt density and color changes caused by motion. NeuVV
factorization can be integrated into a Video Octree (VOctree) analogous to
PlenOctree (Yu et al., 2021b) to significantly accelerate training while
reducing memory overhead. Real-time NeuVV rendering further enables a class of
immersive content editing tools. Specifically, NeuVV treats each VOctree as a
primitive and implements volume-based depth ordering and alpha blending to
realize spatial-temporal compositions for content re-purposing. For example,
we demonstrate positioning varied manifestations of the same performance at
different 3D locations with different timing, adjusting color/texture of the
performer’s clothing, casting spotlight shadows and synthesizing distance
falloff lighting, etc, all at an interactive speed. We further develop a
hybrid neural-rasterization rendering framework to support consumer-level VR
headsets so that the aforementioned volumetric video viewing and editing, for
the first time, can be conducted immersively in virtual 3D space.

immersive rendering, novel view synthesis, neural rendering, visual editing,
neural representation, dynamic scene modeling

††ccs: Computing methodologies Computational photography††ccs: Computing
methodologies Image-based rendering Figure 1. Our neural volumetric video
technique, NeuVV, supports immersive, interactive, and spatial-temporal
rendering of volumetric performances with photo-realism and in real-time.
Using a hybrid neural-volumetric representation, NeuVV enables a user to move
freely in 3D space to watch a single or multiple performances (Left) with a VR
headset. She can also re-arrange and re-purpose the contents by adjusting the
position, size, timing, and appearance of individual performers as well as
adding shadow and certain lighting effects, all in real-time (right). Please
refer to the supplementary video for a live recording of the experience.

##  1\. Introduction

Volumetric Videos (VVs), as an emerging type of visual media, are quickly
expanding (not advancing) the horizons of the entertainment and movie
industries with unprecedented immersive experiences. Often seen in such
science-fiction films as Star Wars, VVs of human performances allow a user to
move about and interact with the 3D contents with six degrees of freedom. Over
the past decade, various versions of capture stages have been made available
worldwide to acquire synchronized multi-view 3D videos, ranging from pure RGB
camera-based systems (e.g., the CMU Panoptic Studio using hundreds of cameras
(Joo et al., 2018)) to RGBD based 3D scans (e.g., the Microsoft MR Capture
Studio (Collet et al., 2015)). Yet, to provide convincing VV experiences,
volumography techniques require using a wide range of tools beyond 3D capture;
they include compression, streaming, playback, and editing.

By far, the most widely adopted workflow to produce volumography is to create
a dynamic mesh of the performance where each frame corresponds to a mesh with
a texture map and all meshes maintain the same topology. For real
performances, producing high-quality meshes imposes significant challenges:
both photogrammetry and 3D scanning based reconstructions are sensitive to
occlusions, lack of textures, dark textures of clothing, etc, and their
results can contain holes and noises. Fixing the initial capture to meet
minimal immersive viewing requirements demands excessive fixing and cleanup
works by artists. A compromise is to start with a cleaned static mesh as a
base and augment it with performance capture. This, however, yields to
infidelity as the rigged performance appears artificial and fails to convey
the nuance of the movements. Colored point cloud sequences have emerged as an
alternative to meshes with a higher spatial resolution. However, they also
incur much higher data rates and require specialized rendering hardware to
mitigate visual artifacts.

Recent advances in neural rendering (Lombardi et al., 2019; Wu et al., 2020;
Mildenhall et al., 2020) can synthesize photo-realistic novel views without
heavy reliance on geometry proxy or tedious manual labor, showing unique
potentials to replace 3D capture. Most notably, the Neural Radiance Field
(NeRF) (Mildenhall et al., 2020) replaces the traditional notion of geometry
and appearance with a single neural network where any new camera views can be
realistically rendered by querying respective rays from the camera via neural
inference. Despite its effectiveness, NeRF and its extensions have been
largely focused on the static object, with a few exceptions (Zhang et al.,
2021a; Lu et al., 2020; Kasten et al., 2021) to directly tackle dynamic
scenes. Further, existing solutions are still a few orders of magnitudes
slower than real-time to support immersive volumography. Let alone
interactive, immersive content editing.

In this paper, we present a new neural volumography technique, NeuVV, to push
the envelope of neural rendering to tackle volumetric videos. In a nutshell,
NeuVV supports real-time volumographic rendering for immersive experiences,
i.e., users can view the contents in virtual 3D space and freely change
viewpoints by moving around. NeuVV further provides tools for flexibly
composing multiple performances in 3D space, enabling interactive editing in
both spatial and temporal dimensions, and rendering a new class of volumetric
special effects with high photo-realism (see Fig. 1). The core of NeuVV is to
efficiently encode a dynamic NeRF to account for appearance, geometry, and
motion from all viewpoints. Analogous to 5D NeRF, dynamic NeRF maps a 6D
vector (3D position + 2D view direction + 1D time) to color and density. To
account for angular and temporal variations at each position, i.e., view-
dependent appearance, we adopt factorization schemes by hyperspherical
harmonics (HH) (Avery, 2012). Further, we treat the position-specific density
separately as it only exhibits temporal variations while being invariant to
view directions. Hence, we further develop a learnable basis representation
for temporal compaction of densities. The factorized color and density can be
easily integrated into existing acceleration data structures such as the
PlenOctree (Yu et al., 2021b; Yu et al., 2021a). The resulting 104-dimensional
vector can effectively model variations in density and view-dependent color at
respective voxels. Compared to the brute-force approach of constructing per-
frame PlenOctree, NeuVV tackles each volumetric video sequence as a whole, in
both training and rendering, and therefore reduces the memory overhead and
computational time by two orders of magnitudes.

By treating each dynamic NeRF as a separate entity, NeuVV supports easy
spatial-temporal compositions for re-purposing the contents, and thereby
immersive and interactive real-time content editing. These include real-time
adjustments of the 3D locates and scales of multiple performers, re-timing and
thus coordinating the performers, and even duplicating the same performer to
produce varied manifestations in space and time. In addition, the employment
of HHs enables temporally coherent appearance and shading editing at the voxel
level. For example, we demonstrate adjusting the color/texture of the
clothing, casting spotlight shadows, synthesizing distance lighting falloffs,
etc, all with temporal coherence and in real-time. We further develop a hybrid
neural-rasterization rendering framework that supports consumer-level head-
mounted displays so that viewing and editing NeuVVs can be conducted
immersively in virtual space. As a byproduct, NeuVV directly supports free-
viewpoint video production at interactive speeds, enabling expert
videographers to deploy their skill set on 2D video footage to volumetric
videos, in a 3D virtual environment.

To summarize, our main contributions include:

  * •

We present a novel neural volumetric video (NeuVV) production pipeline for
enabling immersive viewing and real-time interacting and editing volumetric
human performances with high photo-realism.

  * •

NeuVV employs dynamic NeRF to represent volumetric videos and adopts the
hyper-spherical harmonics (HH) based Video Octree (VOctree) data structure for
efficient training and rendering.

  * •

NeuVV further provides a broad range of composition and editing tools to
support content re-arrangement and re-purposing in both space and time.

  * •

NeuVV supports hybrid neural-rasterization rendering on consumer-level HMDs,
enabling not only immersive viewing but also immersive content editing in 3D
virtual environments.

##  2\. RELATED WORK

#### Volumetric Videos.

Volumetric videos refer to the technique of capturing the 3D space and
subsequentially viewing it on a screen. A volumetric video appears like a
video and can be played back and viewed from a continuous range of viewpoints
chosen at any time. A number of techniques have been proposed to synthesize
point- and surface-based free-viewpoint video (FVV), including shape from
silhouettes (Wu et al., 2011; Ahmed et al., 2008), freeform 3D reconstruction
(Liu et al., 2009; Vlasic et al., 2009) and deformable models (Carranza et
al., 2003).

To get rid of template priors and achieve convenient deployment, one or more
depth sensors can be employed to help the reconstruction. (Newcombe et al.,
2015) proposes a template-free real-time dynamic 3D reconstruction system.
Other approaches enforces the deformation field to be approximately a Killing
vector field (Slavcheva et al., 2017) or a gradient flow in Sobolev
space(Slavcheva et al., 2018). Pirors like skeleton (Yu et al., 2017),
parametric body shape (Yu et al., 2018) or inertial measurement units (Zheng
et al., 2018) are used to facilitate the fusion. (Bozic et al., 2020) applies
data-driven approaches for non-rigid 3D reconstruction. Rather than using a
strict photometric consistency criterion, (Lombardi et al., 2019) learn a
generative model that tries to best match the input images without assuming
that objects in the scene are compositions of flat surfaces. (Seitz and Dyer,
1999; Kutulakos and Seitz, 2000) recovers the occupancy and color in a voxel
grid from multi-view images by evaluating the photo-consistency of each voxel
in a particular order. These approaches generally are difficult to tackle self
occluded and textureless regions, while other approaches rely on parametric
human models, which is limited to human body with tight clothes . In addition,
they struggle with thin structures and dense semi-transparent materials (e.g.,
hair and smoke).

#### Neural rendering.

Synthesizing photo-realistic images and videos is one of the fundamental tasks
in computer vision with many applications. Traditional methods rely on
explicit geometric representations, such as depth maps, point-cloud, meshes,
or multi-plane images. Recently, neural rendering techniques have been showing
great success in view synthesis of static or dynamic scenes with neural
representations, (Tewari et al., 2021) gives a great summary of recent work.
Notably, NeRF (Mildenhall et al., 2020) optimizes neural radiance fields which
represent each point in space with view-dependent color and density, then
traditional volume rendering is applied to render images. NeRF produces
unprecedented photo-realistic results for novel views and quickly becomes a
research focus. Similar to NeRF, recent work uses a variety of neural
representations like implicit surfaces (Wang et al., 2021a; Park et al., 2019)
for a more precise geometry, but they cannot handle dynamic scenes. To address
the dynamic scene reconstruction problem, (Park et al., 2020; Pumarola et al.,
2020; Li et al., 2020; Xian et al., 2020; Tretschk et al., 2020) learn
deformation fields from monocular video and then train a NeRF at canonical
space. They rely on heuristic regularizations, 2D optical flow prediction, or
depth images as priors, but these works suffer from large deformations and
limited viewing range. (Park et al., 2021) further learns deformation field
and radiance field in a higher-dimensional space to tackle the topological
changing problem.

(Zhang et al., 2021b; Li et al., 2021; Pumarola et al., 2020) learn
deformation field from multi-view videos and optimize a radiance field in
canonical space, their approach supports a larger viewing range and better
rendering quality compared to previous approaches. (Zhang et al., 2021b)
further supports certain spatial and temporal editing functions based on
dynamic layered neural representations. (Peng et al., 2021; Zhao et al., 2021)
use the parametric human model as prior to learn a dynamic radiance field for
human body using sparse views as inputs. However, such works are slow to
render free-viewpoint video for dynamic scenes, it takes about 30s to 1min to
render a single image at $1920\times 1080$ on a high-end GPU for NeRF, while
our approach uses a hybrid representation which can more efficiently rendering
for dynamic scenes in real-time.

#### Accelerating NeRF.

There are many existing work to accelerate NeRF (Liu et al., 2020; Reiser et
al., 2021; Yu et al., 2021b; Lindell et al., 2020; Lombardi et al., 2021; Yu
et al., 2021a; Müller et al., 2022). (Liu et al., 2020) uses a sparse octree
representation with a set of voxel-bounded implicit fields and achieves 10
times faster inference speed compared with the canonical NeRF. (Reiser et al.,
2021) uses thousands of tiny MLPs to speed up NeRF by more than 2000 times.
(Yu et al., 2021b) represents the view dependent colors with spherical
harmonics coefficients, and extract them from a radiance field into a sparse
octree-based representation, i.e., namely PlenOctree. Such representation runs
3000 times faster during rendering. Recently, (Yu et al., 2021a) directly
optimizes a sparse 3D grid without any neural networks and achieves more than
100 times faster training speed up and also support real-time rendering.
(Müller et al., 2022) achieves near-instant training time (around 5s to 1min)
of neural graphics primitives with a multi-resolution hash encoding. Though
these works are very effective at speeding up NeRF, they only support static
scenes. Directly extending them to dynamic scenes suffers from expensive
requirements of storage and GPU memory. Our approach uses hyperspherical
harmonics and low dimensional coefficients to reduce hardware requirement, and
achieves real-time inference speed.

Figure 2.  The pipeline of our approach for neural volumetric video (NeuVV)
generation. Given a dense set of synchronized RGB videos as inputs, our
approach first samples a 4D points $(x,y,z,t)$ in the volumetric video, and
then uses an MLP-based neural module $\Psi$ to predict density $\sigma$ and
view-dependent color $c$. Instead of directly inferring color, $\Psi$ predicts
coefficients $\mathbf{w}^{HH}$ of Hyperspherical Harmonics (HH) bases. $\Psi$
also predicts a hyper angle $\gamma$ which slices 4D HH into 3D Spherical
Harmonics (SH), which models the view-dependent color at a specific time
frame. We can finally obtain color $\mathbf{c}$ and density $\sigma$ from 3D
SH given the query point and ray’s direction. Our NeuVV presents a novel
neural representation of volumetric videos, which supports real-time rendering
and editing of the dynamic scene when converted to a Video Octree (VOctree)
representation.

#### Immersive Experience.

With the rapid advancement in VR/AR industry, especially with the emergence of
many commercial headsets, such as Oculus Quest 2 (Facebook Technologies, 2020)
and HTC Vive Pro 2 (Hongda International Electronics Co., 2020), immersive
experience is now immediately available general users. However, compared to
the advance in hardware, immersive content is relative limited. Many
researchers/institutes creates various devices to capture AR/VR content from
the real-time, examples include the Google Jump (Anderson et al., 2016),
Insta360 One X2 (Insta360, 2020)), and etc for high quality 360-degree
capturing. (Bertel et al., 2020) proposes an approach to quickly capture high
quality panorama for watching in VR headsets. But such panorama videos cannot
support the changing of viewing location. (Broxton et al., 2020a) presents a
system to capture, reconstruct, and finally render high quality immersive
videos using a semi-sphere camera array. (Orts-Escolano et al., 2016) proposes
a system that can achieve a real-time 3D reconstruction of the whole space
using multi-view RGB-D camera arrays. Such capture systems rely heavily on
explicit scene reconstruction algorithms, such as multi-view stereo (Zitnick
et al., 2004; Li et al., 2019; Yao et al., 2018), light field (Broxton et al.,
2020b; Levoy and Hanrahan, 1996; Gortler et al., 1996; Buehler et al., 2001;
Sitzmann et al., 2021), multi-plane images (MPIs) representations(Mildenhall
et al., 2019; Broxton et al., 2020a; Srinivasan et al., 2019) and image based
rendering techniques (Suo et al., 2021; Debevec et al., 1996; Carranza et al.,
2003; Snavely et al., 2006). (Zitnick et al., 2004) uses multi-view stereo
technique to estimate depth maps, then interpolate color images guided by the
estimated depth images. (Li et al., 2019) learn human depth priors from
thousands of Internet videos. (Mildenhall et al., 2019) uses multi-plane
images which can represent complicated scenes by interpolating RGBA values on
the planes. But it cannot support large changing of viewpoint. These
approaches either reconstruct the scene geometry explicitly, or rely on image
based rendering techniques. Reconstructing the scene geometry is always a
difficult task, especially for occluded and textureless regions. On the other
side, image based rendering technique produces images based on either pre-
captured depth image or estimated depth, and suffers from flicking artifacts.
Yet, our NeuVV does not rely on an geometry explicitly and hence avoid the
difficult geometry reconstruction problem.

##  3\. OVERVIEW

Fig. 2 shows the overall processing pipeline of NeuVV. The input to each NeuVV
is multi-view video sequences towards the performer. In our setting, we have
constructed a multi-view dome with a set of 66 synchronized RGB videos. We use
structure-from-motion to pre-calibrate the cameras so that all views have
known camera poses. For validations, we select a specific frame and conduct
static NeRF reconstruction. A number of options are available, from the
original NeRF reconstruction (Mildenhall et al., 2020), to accelerated
Plenoxel (Yu et al., 2021a), and to the latest, extremely fast NGP (Müller et
al., 2022). The reason to test on a static frame is to validate calibration as
well as to support conduct better foreground/background segmentation for
subsequent frames, to better produce NeuVV. Specifically, both Plenoxel and
NGP provide interfaces to limit the reconstruction volume and we use the
estimation for processing subsequent frames for NeuVV.

Recall NeuVV aims to approximate a dynamic radiance field using an implicit
but continuous spatial-temporal scene representation, by separately
factorizing the appearance, i.e., time-varying and view-dependent color, and
density, i.e., changes due to motion. For the former, we apply Hyperspherical
Harmonics (HH), originally designed for solving the Schrödinger equation as
basis functions. HH can be viewed as an elevation of Spherical Harmonic (SH)
by considering an additional time dimension. For the latter, notice volume
densities exhibit different temporal profiles than color: they are not view-
dependent but can vary sharply over time. We hence use a learnable basis
instead of HH for factorization.

To process NeuVV, the brute-force approach would be to directly train a NeRF
using the factorization, as in previous video-based NeRF (Zhang et al., 2021a;
Pumarola et al., 2020). Its downside is that NeRF is not readily supportive
for real-time rendering, critical for video viewing. We hence exploit the
PlenOctree designed for real-time rendering of static objects. Specifically,
we extend PlenOctree to Video Octree (VOctree) to conduct network training and
subsequent rendering based on HH and learnable factorizations. Finally, we
integrate VOctree into OpenVR via a hybrid neural-rasterization renderer, for
interaction and editing in immersive environments. NeuVV supports multiple
VOctree instances as well as duplicated instances for special volumetric video
effects.

##  4\. Neural Volumetric Video Factorization

Given a dense set of synchronized videos of dynamic performers with known
camera poses, we represent the captured scene as a dynamic radiance field that
can be modeled as a 6D function $\Phi$, which produces a volume density value
$\sigma$ and color $\mathbf{c}$ for each space location $(x,y,z)$, time $t$
and view direction $(\theta,\phi)$, i.e.:

(1) $\Phi(x,y,z,\theta,\phi,t)=\sigma,\mathbf{c}$

A brute-force implementation is to recover one NeRF for each timestamp $t$ and
then load individual frames. The approach suffers from several artifacts: it
inherently incurs high memory consumption, slow training, and cross-frame
inconsistency/flicking. Alternative approaches such as ST-NeRF (Zhang et al.,
2021b), D-NeRF (Pumarola et al., 2020), NeuralBody (Peng et al., 2021) and
HumanNeRF (Zhao et al., 2021) conduct spatial-temporal warping to map
individual frames to a common canonical space so that they only need to train
a single NeRF. The quality relies heavily on the accuracy of the estimated
warping field; when deformation is large or the performer contains too few or
too many textures, they tend to produce strong visual artifacts.

Figure 3. Recovering color from hyperspherical harmonics. By mapping a fixed
timestamp $t$ to a hyper angle $\gamma(t)$, the 4D hyperspherical harmonics
degenerates to 3D spherical harmonics. Given a spatial point $\mathbf{p}$ and
a viewing direction $(\theta,\phi)$ along the query ray, we can recover color
from spherical harmonics.

###  4.1. Hyperspherical Harmonics Factorization

NeuVV instead seeks to avoid the warping process: inspired by PlenOctree which
factorizes the view-dependent appearance via spherical harmonics functions,
NeuVV uses hyperspherical harmonic (HH) basis functions to further support
time-variant color. Specifically, we obtain the time-varying and view-
dependent color at each point $(x,y,z)$ as $\mathbf{c}(\theta,\phi,t)$ by
fixing $(x,y,z)$ in Eqn. 1.

The HHs are functions of hyper angles that describe the points on a
hypersphere. In the NeuVV setting, we use 4D HHs in which 3 dimensions are for
describing spherical harmonics parameterized by $\theta$, $\phi$ as in
PlenOctree, and 1 more dimension for the temporal dimension $t$. Consequently,
we can rewrite the HH basis functions as:

(2)
$\mathcal{H}^{m}_{nl}(\theta,\phi,\gamma)=A_{n,l}\sin^{l}(\gamma)C^{l+1}_{n-l}\big{(}\cos(\gamma)\big{)}\mathcal{S}^{m}_{l}(\theta,\phi)$

where

(3) $A_{n,l}=(2l)!!\sqrt{\frac{2(n+1)(n-l+1)!}{\pi(n+l+1)!}}$

$\gamma\in[0,\pi]$ is the hyperangle corresponding to the time dimension,
$C^{l+1}_{n-1}$ are Gengenbauer polynomials, and $\mathcal{S}^{m}_{l}$ are the
3D spherical harmonics. $l,m,n$ are integers, where $l$ denotes the degree of
the HH, $m$ is the order, and $n=0,1,2,...$, following $0\leq l\leq n$ and
$-l\leq m\leq l$. Notice that when we fix $t$, HH forms an SH with a time
dependent scaling factor. The complete derivations of HHs can be found in the
supplementary materials.

It is critical to note that all HH bases are smoothly varying function and
therefore their compositions will be highly continuous and smooth in 4D space.
This is preferred for view-dependent appearance, but problematic for
appearance change caused by relatively fast motions at a space point. To
resolve this issue, we introduce an additional a non-linear mapping function
$\gamma(\cdot)$ that maps linear timestamps to hyper viewing angles, and the
color then can be formulated summation of HH basis as:

(4)
$\mathbf{c}(\theta,\phi,t)=\sum_{m,n,l}w^{m}_{nl}\mathcal{H}^{m}_{nl}\big{(}\theta,\phi,\gamma(t)\big{)}$

where $w^{m}_{nl}$ is the coefficient of corresponding HH basis function and
$\mathbf{w}^{HH}$ represents the vectorized coefficients.

Figure 4. Recovering density from learned bases. For each 3D point $(x,y,z)$,
time varying density can be recovered from the weighted sum of learned bases
$\mathbf{w}^{\sigma}$ predicted by an MLP. As the changing of density may be
rapid, we use ReLU$(\cdot)$ function for non-linear mapping and that density
should not be negative.

###  4.2. Learnable Temporal Basis Factorization

Once we factorize the time-varying and view-dependent color using HHs, we
store volume density $\sigma(t)$ and $\gamma(t)$ for each timestamp $t$ given
a spatial location. Notice that temporal change of volume density is caused by
the occupancy/release of corresponding space point incurred by object motion.
Hence temporal variations of volume density generally follow certain patterns,
e.g., a moving hand passing through a space point indicates rapidly increasing
from 0, staying constant, and then decreasing to 0 of the density at the point
(see Fig. 4 for illustration). This indicates we can map the time-varying
volume density onto a shared set of high dimensional bases and then use
tailored low dimensional coefficients to refine the function. Such a strategy
reduces memory consumption and also accelerates training.

Specifically, consider the time varying density at point $\mathbf{p}$ as
$\Sigma=[\sigma_{1},\sigma_{2},\cdots,\sigma_{N}]\in\mathbb{R}^{N}$, where $N$
is the number of time frames. We first project it onto high dimensional
density bases
$A=[\mathbf{a_{1}},\mathbf{a_{2}},\cdots,\mathbf{a_{C}}]\in\mathbb{R}^{N\times
C}$, where $C$ is the number of bases and $C\leq N$ for time varying density.
We adopt a mapping function as:

(5) $\hat{\Sigma}=\text{ReLU}(A\mathbf{w}^{\sigma})$

Same as the NeRF setting, we can use an MLP $\mathcal{P}_{\sigma}$ to learn
the mapping weights $\mathbf{w}^{\sigma}\in\mathbb{R}^{C}$ from the spatial
location $[x,y,z]$ inputs. We then optimize $A$ by minimizing the summation of
differences between $\hat{\Sigma}$ and $\Sigma$ over the complete volume.

Similarly, we can map the hyper angles
$\Gamma=[\gamma(t_{1}),\gamma(t_{2}),\cdots,\gamma(t_{N})]\in[0,\pi]^{N}$ into
a set of bases $B\in\mathbb{R}^{N\times C}$ and use another MLP
$\mathcal{P}_{\gamma}$ to estimate the mapping weights $\mathbf{w^{\gamma}}$.

(6) $\hat{\Gamma}=\pi\cdot\text{Sigmoid}(B\mathbf{w}^{\gamma})$

#### Neural Mapping Module.

We integrate above discussed three networks for predicting
$\mathbf{w}^{\sigma},\mathbf{w}^{\gamma},\mathbf{w}^{HH}$ into a single MLP
network $\Psi$ as illustrated in Fig. 5.

(7) $\Psi(x,y,z)=\mathbf{w^{\sigma}},\mathbf{w^{\gamma}},\mathbf{w}^{HH}$

Given a location, view direction and time tuple $(x,y,z,\theta,\phi,t)$ as
input, we use $\Psi$ to predict the coefficients
$\mathbf{w^{\sigma}},\mathbf{w^{\gamma}},\mathbf{w}^{HH}$. And we can recover
the result color $\mathbf{c}$ using Eqn. 6 and 4 and volume density $\sigma$
by Eqn. 5.

Figure 5. Details of our network structure, which basically is a multi-layer
perceptron (MLP). For a 3D point $x$, we first apply positional encoding
$\text{PE}(\cdot)$ and send the result to the network. The network outputs
density coefficients $\mathbf{w}^{\sigma}\in\mathbb{R}^{C}$, hyper angle
coefficients $\mathbf{w}^{\gamma}\in\mathbb{R}^{L}$, and HH coefficients
$\mathbf{w}^{HH}\in\mathbb{R}^{L\times 3}$.

#### Training.

Similar to training the canonical PlenOctree for the original NeRF, we
synthesize spatial-time views of NeuVV via volume rendering. Specifically, we
set out to predict the color of ray $\mathbf{r}$ by sampling points along the
ray and accumulate their density $\sigma_{i}$ and color $c_{i}$ as:

(8)
$\displaystyle\hat{C}(\mathbf{r})=\sum_{i=1}^{|\mathcal{P}|}T_{i}(1-\text{exp}(-\sigma_{i}\delta_{i}))c_{i}$
$\displaystyle\text{where
}T_{i}=\text{exp}\left(-\sum_{j=0}^{i-1}\sigma_{j}\delta_{j}\right)$

where $\mathcal{P}=\\{p_{i}\\}_{i=1}^{|P|}$ is the set of sampled points
ordered from near to far, $\delta_{i}$ is the distance between the sampled
points, $\exp(\cdot)$ is the exponential function.

Similar to PlenOctree optimization where it is ideal to first conduct
foreground/background segmentation to minimize the volume, we conduct the same
foreground segmentation on NeuVV. In fact, using a video instead of an image
makes automatic segmentation even easier. In our implementation, we first use
the latest automatic video matting technique [VideoMatte240k] to first
separate moving foreground and the static background, and then randomly select
$20\%$ rays towards the background and mix them with all rays hitting the
foreground to train NeuVV. We observe such a strategy is advantageous than
discarding all background rays: using a small percentage of random background
rays imposes additional priors to the foreground and avoids overfitting the
dynamic foreground dynamic performer, especially when input views are unevenly
sampled.

To further prevent the network from learning static background, we blend the
predicted color $\hat{C}(\mathbf{r})$ with the captured background
$C_{bg}(\mathbf{r})$, using weights from the predicted alpha value
$\hat{\alpha}(\mathbf{r})$:

(9)
$\hat{C^{\prime}}(\mathbf{r})=\hat{\alpha}(\mathbf{r})\cdot\hat{C}(\mathbf{r})+(1-\hat{\alpha}(\mathbf{r}))\cdot
C_{bg}(\mathbf{r})$

where
$\hat{\alpha}(\mathbf{r})=\sum_{i=1}^{|\mathcal{P}|}T_{i}\big{(}1-\exp(-\sigma_{i}\delta_{i})\big{)}$.
This modified rendering scheme forces the network to learn an empty space
($\sigma$ = 0) for the background part.

Finally, we use the differences between observed colors in multi-view videos
and the rendered colors from NeuVV as loss to train our model via self-
supervised training:

(10)
$\mathcal{L}_{rgb}=\sum_{r\in\mathcal{R}}\|C(\mathbf{r})-\hat{C^{\prime}}(\mathbf{r})\|_{2}^{2}$

where $\mathcal{R}$ corresponds to the set of spatial temporal rays in each
training batch and $C(\mathbf{r})$ corresponds to the captured pixel color of
the input videos. We further use the same positional encoding and importance
sampling scheme as in original NeRF to enhance convergence.

###  4.3. Video Octree (VOctree) Representation

Same as NeRF, the brute-force approach of rendering NeuVV using MLP is slow as
it requires a neural network inference for many sampling points on each query
ray. For example, it takes around one minute to render a $1920\times 1080$
image on NVIDIA RTX-3090 GPU, prohibitively long for deployment to real-time
playback, let alone immersive rendering. We follow the PlenOctree technique
[PlenOctrees] that uses a video octree (VOctree) representation with pre-
tabulated density and SH coefficients for view dependent color. In our
implementation, we store coefficients
$\mathbf{w^{\sigma}},\mathbf{w^{\gamma}},\mathbf{w}^{HH}$ of each spatial
location into an octree-based representation. Instead of optimizing the MLP
and then tabulating the coefficients, we directly optimize the octree from the
multi-view video inputs.

#### Initialization.

For octree based representation, its efficiency is achieved by using larger
voxels for empty space while smaller voxels for occupied space with fine
details. Further, ray sampling points inside the same voxel may show
disturbances according to its relative position. Recall that [PlenOctree]
first evaluates the density in a dense voxel grid and filter out the voxels
with density lower than a threshold ($\sigma$ less than $1.0\times 10^{-5}$),
we sum up density for each voxel along time axis then filter it out use the
same threshold $1.0\times 10^{-5}$. Then inside each remaining voxel, We
sample random 256 points and take the average of
$\mathbf{w}^{HH},\mathbf{w}^{\sigma},\mathbf{w}^{\gamma}$ as the stored
coefficients for the voxel.

#### Rendering.

After initialization, our VOctree based NeuVV supports rendering of dynamic
entities with novel viewpoints in real-time. Specifically, given a ray we
determine the voxels on its path way along with lengths of line segments
$\\{\delta_{i}\\}_{i=1}^{D}$ inside voxels, where $i$ is the voxel index and
$D$ is the total number of voxels on the ray. We fetch coefficients
$\\{\mathbf{w}^{\sigma}_{i},\mathbf{w}^{\gamma}_{i},\mathbf{w}^{HH}_{i}\\}_{i=1}^{D}$
stored in the voxels. From Eqn. 5, 6, 4, we obtain
$\\{\sigma_{i},c_{i}\\}_{i=1}^{D}$ which are recovered density and color from
coefficients, then we obtain the resultant color by volume rendering technique
(Eqn. 8)

#### Optimization.

Recall the volume rendering process is differentiable. We can therefore
optimize weights stored in VOctree by gradient decent using classic
optimizers, such as SGD or Adam, using the RGB loss in Eqn. 10. For
implementation, we deduct the derivatives and write custom CUDA kernels and
achieve higher convergence speed, which is approximately 1,000 times faster
than the original PlenOctree implementation.

Directly optimizing the VOctree, however, leads to overfitting and
subsequently incurs noisy pixels on input/training video frames. We hence
impose an additional regularization term to mitigate the problem.
Specifically, we enforce the gradient of the difference between rendered image
$\hat{I}$ and ground truth image $I$ to be small as:

(11) $\mathcal{L}_{grad}=\sum_{i=1}^{N\times
M}\|\nabla|I_{i}-\hat{I_{i}}|\|_{2}^{2}$

where $N\times M$ is the total number of training views and $\nabla$
calculates the gradient. The final loss becomes:

(12) $\mathcal{L}_{total}=\mathcal{L}_{rgb}+\lambda_{grad}\mathcal{L}_{grad}$

where $\lambda_{grad}$ is a hyper-parameter to balance the RGB loss and
gradient loss. In all our experiments, we set $\lambda_{grad}$ to 0.1,
although it can be fine-tuned to achieve even better performances for
individual datasets.

##  5\. Immersive Rendering and Editing

Existing volumetric videos have been largely used to create 2D free-viewpoint
videos (FVV) (Wu et al., 2011; Ahmed et al., 2008) where expert videographers
apply their 2D footage editing skill sets. The capabilities of directly
editing volumetric videos in 3D are long time dreams for content producers.
The experience should be fun and compelling, with sample applications ranging
from 3D visual art creations, to virtual fitness training, and to cultural
heritage. Recent neural network based techniques(Zhang et al., 2021b) can
potentially support multi-view content editing but the process is still
conducted on 2D screens rather than in 3D environment. The challenges are two-
fold: 1) there lacks an immersive composition and editing tools to pair with
existing VR rendering engines and headsets and 2) it is essential to achieve
real-time rendering to make the 3D editing processing plausible. Since NeuVV
already addresses the second challenge, we set out to design truly immersive
composition and editing functionalities.

By using the VOctree to store space-time coefficients of NeuVVs, we develop a
toolkit to support a variety of editing functions including spatial and
temporal compositions for content re-pursing, content re-timing, and
duplication and varies manifestations. Further, the Octree-based NeuVV
representation enables volumetric appearance editing, e.g., we can change the
color/texture of the 3D clothing worn by the performer, producing spotlight
cast shadows and other relighting effects, all on the implicit representation
without the need of converting the Octree to meshes. In addition, a viewer
wearing the VR headset can perform along with the NeuVV where commodity motion
capture solutions can be used to compare/match the move of the viewer with the
virtual performer, enabling exciting new applications such as virtual fitness
trainer.

Figure 6. Spatial and temporal composition results. We can composite multiple
NeuVVs together by applying spatial editing function $\mathcal{A}$ and
temporal editing function $\mathcal{T}$. Furthermore, we use a depth-aware
alpha blending strategy to generate the correct occlusion effects. Figure 7.
Varied manifestations effect. NeuVV achieves varied manifestations effect,
Avalokiteshvara in Buddhist mythology, using constant memory and in real-time.

###  5.1. Spatial and Temporal Composition

NeuVV supports a variety of immersive spatial temporal editing operations. For
spatial editing, we use the 3D bounding of NeuVV as an anchor. A user can
adjust the bounding box in virtual space to scale, rotate, and re-position and
re-time the performance. Since NeuVV provides a continuous representation in
both space and time, the adjustment preserves photorealism in both appearance
and motion. Specifically, we model the spatial editing operator in terms of an
affine transformation $\mathcal{A}$. The transform from the original bounding
box $\mathbf{B}$ to the adjusted one $\mathbf{B^{\prime}}$ is
$\mathbf{B^{\prime}}=\mathcal{A}\circ\mathbf{B}$. We can subsequently apply
the same transform to all nodes in the VOctree.

Recall that in our NeuVV representation, the spatial and temporal dimensions
are disentangled. Hence we can also manipulate the time axis to create novel
temporal effects, while leaving the spatial contents unaffected. Given a
sequence of timestamps $T$, we apply a general mapping function $\mathcal{T}$
to obtain a new timestamp sequence $T^{\prime}=\mathcal{T}\circ T$. Typical
operators of $\mathcal{T}$ include all to one mapping (pausing), clipping
(partial playing), reversing (playing backward), jumping (fast forwarding),
looping, etc. We can hence uniformly model spatial and temporal editing as:

(13) $\Phi(\mathcal{A}\circ(x,y,z,\theta,\phi),\mathcal{T}\circ t)=\sigma,c$

#### Varied Manifestations.

One of the most unique visual effects in NeuVV is to create varied
manifestations of the same performer using only a single VOctree. The effect
was popularized largely by feature film The Matrix where many copies of Agent
Smith were created. Traditionally the process requires constructing a dynamic
3D model of the performer, replicate the model multiple times and position
individual model at different 3D locates, and use offline rendering engines to
produce the final footage. The more the duplicates, the more computational and
memory resources required and the slower the rendering process. By using
VOctree as the primitive, we show we can achieve real-time performance with
fixed memory consumption, disregarding the number of replicates.

The brute-force approach would be to load the same NeuVV multiple times for
rendering. However, since a NeuVV captures the complete plenoptic function in
both space and time, one can simply use a single NeuVV where its duplicates
can be treated as viewing it from different viewpoints and at different time
scales. Specifically, we can reuse the composition and re-timing operators in
Eqn. 13 to produce duplicated performers positioned at different 3D locates
with strategically designed, asynchronized movements. In Fig. 7, we show an
exemplary varied manifestation effect of the Thousand Armed Avalokiteshvara,
known in Buddhism, representing boundless great compassion. We discuss its
real-time implementation in Section 5.1.

#### Depth-aware Alpha Blending

When we compose multiple NeuVVs as primitives (even the duplicated ones) for
rendering, it is critical to conduct correct depth ordering. This is
particularly important as the user is expected to move around in 3D space to
view the contents at different viewpoints. Incorrect occlusions will greatly
affect visual realism. To tackle such a challenging problem, we propose a
simple yet effective depth blending algorithm that uses rendered depth maps
$\\{\hat{D}\\}_{i=1}^{L}$ and alpha mattes $\\{\hat{A}\\}_{i=1}^{L}$ to guided
the blending of RGB images $\\{\hat{I}\\}_{i=1}^{L}$ rendered from all NeuVVs.

Our key insight is inspired by the traditional rendering process, i.e., the
z-buffer technique more specifically. We first apply transformations to each
VOctree and render the corresponding depth, alpha map and color image by
tracing rays from the virtual camera. Since we adopt the octree structure, the
ray tracing process can be executed very efficiently and we can do the
rendering in a layer-wise manner.

For varied manifestations effects, there will be multiple iterations for
generating the queried time frame one at each time and then compose all time
frame results together. Since we are tracing rays from the same camera for all
VOctrees, we can naturally compare the depth values of each pixel to figure
out the occlusion relations and then conduct canonical alpha blending without
difficulty. This process is illustrated in (Algorithm. 1).

Input : $\\{I_{i}\\}_{i=1}^{L},\\{D_{i}\\}_{i=1}^{L},\\{A_{i}\\}_{i=1}^{L}$

Initialization: $I=I_{1},D=D_{1},A=A_{1}$

for  _$i=2,\ldots,L$_ do

$fg=D_{i}<=D,bg=D_{i}>D$

$I[fg]=A_{i}[fg]I_{i}[fg]+(1-A_{i}[fg])A[fg]I[fg]$

$I[bg]=A[bg]I[bg]+(1-A[bg])A_{i}[bg]I_{i}[bg]$

$D[fg]=D_{i}[fg]$

$A=A+A_{i}\cdot(1-A)$

end for

Output : Blended RGB image $I$, depth $D$, and alpha image $A$

ALGORITHM 1 Depth-aware Alpha Blending

###  5.2. Editing and Rendering

Our VOctree-based NVV representation further supports certain levels of
appearance editing. Adding lighting effects or changing appearance coherently
in both space and time have been particularly challenging on volumetric
videos. In the 2D videos, rotoscoping is widely adopted for tracking objects
over frames and subsequently consistent recoloring and retexuring. For
volumetric videos, it is simply infeasible to consistent rotoscope over all
frames and at all viewpoints. For NeuVV, the more challenging task is adding
lighting effects: as an implicit representation, NeuVV does not produce
explicit geometry such as a mesh that can be used for adding lighting effects.
We demonstrate how to use the VOctree structure to achieve certain classes of
appearance editing and relighting effects.

#### Appearance editing.

To edit appearance, we can first select the set of voxels of interests. If the
edits are conducted on 2D screen (e.g., for FVV generation), one can use
images/frames to map highlighted pixels to their corresponding voxels. Under
the VR setting, they can be directly conducted in 3D space by defining a 3D
region and selecting the voxels within using the controller. Recall, NeuVVs
adopts an implicit representation with coefficients $w^{\sigma}$ as latent
variables, direct editing of these coefficients, although possible, does not
readily produce meaningful results. Our editing function therefore aims to
modify the appearance of the corresponding VOctree rather than the content
itself. Nonetheless, this is sufficient for the user to modify the texture and
color of clothing. Specifically, we append 5 additional channels to each voxel
that represent the target RGB values $\mathbf{c}_{d}$, the target density
value $\sigma_{d}$, and the timestamps $t_{d}$ indicating which frames on this
voxel should be modified.

The challenge here is to determine which voxels to be edited and how to blend
with the original NeuVV VOctree. Consider painting a 2D pattern over the
NeuVV. Given the camera pose, we trace each pixel/ray towards the VOctree and
we locate the terminating voxel along the ray when the accumulated alpha
rendered using NeuVV is beyond a threshold (0.99 in our implementation). We
then assign the target color to the voxel. At render time, the target color
can be further blended with the NeuVV rendering results to further improve
view-consistency. In Fig. 11, we show free-viewpoint rendering of a ballerina
sequence after we paint Van Gogh’s starry night onto the original black tight
shirt. Note the complexity of appearance editing, via either region selection
or ray tracing, is significantly lower than the volume rendering process with
HH, as it does not require volume integration. So the appearance editing is
still in real-time and can be done interactively during the dynamic rendering
process.

#### Spotlight Lighting.

In a theatrical setting, spotlight produces artistic effects for enhancing
realism. They also help convey the nuance of human motion: when motion is
minute, its shadow variations can be still be highly apparent attributed to
perspective magnifications. Such changes of light and shadow can increase the
viewing experience of the viewer.

Producing spotlight shadows of NeuVVs is nearly identical to rendering shadow
map of meshes: we can position a virtual camera in the position of the point
light source and render the VOctree at the respective viewpoint. In
traditional shadow maps, shadows are created by conduct a visibility test
using the z-buffer. Since NeuVV builds on top volume rendering, we further use
the accumulated alpha values along rays. Specifically, we first render an
alpha map from the point light virtual camera, reserve the alpha map (as the
denser the alpha map, the higher the probability the ray been blocked and
hence induces shadow), and finally project it onto the ground. For faster
rendering, we choose to render shadows at lower resolution and then use low
pass Gaussian filters to remove Moire patterns. Fig. 11 shows sample cast
shadows of a dynamic performer. The figure and the supplementary video
demonstrate that under the VR setting, NeuVV produces visually consistent
shadows for better conveying subtle motions.

Another lighting effect is distance falloff: the closer the part of the
performer to the light source, the brighter it appears. Specifically, instead
of using the density accumulation as in shadow maps, we directly render the
depth map and compute the falloff in terms of the distance between the voxel
to the light. If we position the spotlight on top of the performers, their
faces will appear brighter than feet, creating special theatrical atmosphere.
Under the VR setting, we observe they produce more realistic encounters for
viewing volumetric performances.

###  5.3. 2D vs. 3D Rendering

Figure 8. NeuVV rendering under VR setting. Our VR rendering pipeline, which
renders multiple NVVs’ alpha, depth and RGB images simoutanouly in real-time
at given camera pose. Then we blend the images together in a depth-aware
manner.

While most volumetric videos are processed or viewed on desktops including the
latest neural representations, the best viewing and editing experiences should
be immersive and hence carried out under the VR setting when headsets are
available. We have implemented NeuVV renderers under both settings.

#### Free-Viewpoint Video Renderer

We first develop a Free-viewpoint Video (FVV) renderer based on NeuVV. Most
existing FVV players are based on 3D meshes or points, popularized by
Microsoft Capture Studio. The use of explicit representations have its
advantages and limitations: mesh rendering is directly supported by the
graphic hardware and can be integrated into existing rendering engines; yet
producing high quality meshes without extensive cleanup of the initial capture
is still extremely difficult. NeuVV’s implicit representation addresses the
visual quality issue but additional efforts are needed to fit it to existing
rendering pipelines.

In our implementation, VOctree builds upon the open source PlenOctree
originally designed for real-time rendering of NeRF-based static objects. We
modify the spherical harmonics (SH) bases in PlenOctree and replace them with
our HH bases for appearance rendering and learnable bases for density and
hyper angle. It is worth nothing VOctree supports the rendering a single
performer and multiple performers. For the former, ultra-fast rendering at a
lower resolution helps to check the quality of the trained neural
representations, e.g., to determine if the spatial-temporal videos can be
sufficiently replicated by the network with acceptable visual quality. For the
latter, it is particularly useful for re-purposing the contents by obtaining
real-time feedback on the final layout and visual effects of the FVV. This is
particularly important as many previous FVV generators, including the neural
ones, require long processing time instead of being interactively editable. In
our implementation, we have rewritten custom CUDA kernels as well as added
rendering capabilities of shadows and light falloff effects via the alpha and
depth maps. Once validated, the contents can be transferred to the VR renderer
to create immersive experiences.

Figure 9. Immersive fitness training demo. Our NeuVV renders views of the
coach in real-time and highlights body parts (red) corresponding to the
incorrect joints based on the differences between the reference skeleton and
the skeleton generated by mo-cap, which helps trainee to correct poses.

#### VR Renderer.

Most unique to our NeuVV renderer is its support for head-mounted displays
(HMDs). We have developed a NeuVV API based on OpenVR for supporting different
types of HMDs (Oculus, Mixed Reality, Vive, etc). In several examples shown in
the paper and the video, we demonstrate NeuVV VR rendering using Oculus Quest
2 on a single NVIDIA RTX-3090 GPU. We render stereo views at a resolution of
$1920\times 1080$. The NeuVV API takes camera pose of the headset from OpenVR
and renders individual VOctrees representing different performers with
algorithms discussed in Section XXX to tackle correct depth ordering. Shadows
and falloff lighting can be turned on and off using the controller. Fig. 8
shows the complete NeuVV VR rendering pipeline. A key advantage of the VR
renderer is it allows a user to compose and edit volumetric videos in 3D
space. We provide a group of interaction functionalities. For selection, we
use the position and the orientation of the controller to emit a line (ray)
towards the scene for selecting the target NeuVV in terms of its bounding box.
Once selected, the content can be re-positioned freely in 3D space, as if a
user is controlling a 3D object, largely thanks to real-time VOctree
rendering. We also provide a self-rotation function where the performer self-
rotates smoothly along the y-axis while the video plays along.

Recall that the original PlenOctree only supports free-viewpoint viewing, i.e,
the camera pose can change but the object cannot rotate otherwise the its
corresponding tree structure needs to be reconstructed. Therefore, we emulate
rotation of an NeuVV by transforming the viewpoint with respective to each
individual entity, i.e., we compute the corresponding viewpoint for each NeuVV
within the scene. To be more specific, we make the camera rotate around the
performer and keep it look at the performer.

To realize duplicated manifestation, our system provides a duplicated button.
Instead of making multiple copies of the VOctree which will significantly
increase memory consumption, we only create a new pointer to the same VOctree,
along with the transformed viewpoint and the desired re-timing map, as if it
were a different NeuVV. Rendering can then be carried as usual with depth
ordering support. In this way, we can create as many duplicates as possible
without incurring additional memory overhead. Finally, as NeuVV can also be
viewed as a video, we provide pause/play/forward/backward controls on the
controller, each implemented by adjusting respective timestamp controls as
shown in Section 5.1. The supplementary video provides many examples
demonstrating the NeuVV VR experience.

Figure 10. Capture system. Our capture system consists of 66 industry Z-CAM
cameras which are uniformly arranged around the captured performer to cover a
view range up to 1440 degrees (4 circles). Each of camera circles is focused
on lower body, full body, upper body or top views of performers. All the
cameras are calibrated and synchronized in advance, producing 66 RGB streams
at 3840 $\times$ 2160 resolution and 25 frames per-second.

#### Live User Motion Feedback.

In addition to composition and editing, we allow the user to perform along
with the virtual performers in NeuVV. A potentially useful function is hence
to highlight live user motions on the top of the NeuVV footage. This is
particularly useful for fitness training and dancing games in VR setting,
i.e., a home personal training who will remind the user about incorrect
postures that can also adverse effects.

There are many real-time motion capture solutions available and we adopt the
recent single camera technique(He et al., 2021) for convenience. It is able to
detect 21 key points of skeletons. We have developed an interface to our VR
NeuVV to allow the estimated mo-cap results feed directly back to the
renderer. As a reference, we preprocess the NeuVV of the trainer by conducting
multi-view skeleton detection. Notice that many of the volumetric videos in
this paper were captured using a dome system where each camera only captures a
partial view of the performer and skeleton detection is less robust. Therefore
we first render a multi-view full body sequence using NeuVV and then conduct
skeleton extraction. This produces very high quality skeletal movements. We
then compare the user movements with the performer’s and highlight their
differences in live viewing experiences.

Figure 11. Editing results. For When Van Gogh Meets Ballet (top), we edit the
clothes appearance by mapping Van Gogh’s famous painting Starry night, and
show some representative views. For Light and shadow (middle), we add virtual
light and cast the shadow of performers as virtual motion magnifier, we show
representative frames of edited VOctree of performer. For Waving (bottom), to
create waving effect of the same performer, we duplicate and shift her
location and timing. Figure 12. Qualitative comparison with Neural Volumes,
Neural Body, iButter and ST-NeRF. Note that our approach generalizes more
photo-realistic and finer details.

Fig.9 shows a typical example of fitness training where the user conducts deep
squad, one of the most important movements in leg training, along with the
virtual trainer represented using NeuVV. The details of squat movements are
very important for the effectiveness and safety of training, which is
difficult for beginners to grasp. Once motion discrepencies are detected, our
renderer not only highlights their differences but also suggests the user
moving about the trainer to observe the correct movements from the most
suitable view angle, a special treat provided by volumetric videos. Any time,
the user can use the VR controllers to pause, remind, re-position, and scale
the video content at will.

##  6\. RESULTS

We have validated NeuVV factorization on challenging volumetric videos
captured under real-world settings as well as implemented an array of
composition and editing tools suitable for 2D screens and 3D immersive
environments. We provide implementation details as well as the utilized
datasets captured by our multi-view system. We further compare NeuVV vs. other
alternatives, most of which are offline algorithms though. Nonetheless, we
show NeuVV outperforms them in visual quality and is much faster. We also
discuss different components of NeuVV and how they affect the results
qualitatively and quantitatively. Finally, we illustrate spatial-temporal
composition and editing functionalities of NeuVV as well as discuss its
limitations.

#### Implementation Details.

We have implemented the core NeuVV component, i.e., VOctree (Section 4.3) in
PyTorch with customized CUDA kernels for inference and back propagation. All
experiments are trained and optimized using a single NVIDIA Tesla A100 GPU or
a NVIDIA GeForce RTX3090 GPU. Real-time rendering either on s 2D screen or VR
headset is conducted on a single RTX3090. The most time consuming component of
NeuVV is training and generation. Depending on the number of video frames in
the captured scene (75 to 150 frames) and the complexity of the performer’s
motion, the training time ranges from 12 to 24 hours with an input resolution
of $960\times 540$, followed by a conversion from NeuVV to VOctree which takes
around 15 minutes per sequence. Finally, we optimize VOctree-based NeuVV with
an input image resolution $1920\times 1080$ where the processing time ranges
from 8 to 12 hours.

#### Datasets.

We have captured 20 multi-view video sequences, all with a single performer
acting inside the capture dome. Motions range from relatively static movements
such as hand waving to moderate ones as fitness training and dramatic ones as
dancing. We also have the performers wearing various types of clothing, from
high tight outfits as in the Ballerina sequence to high loose dresses and
robes in the Dunhuang dance sequence, to test the robustness of our approach.
Fig. 10 shows our capture system that consists of 66 industry Z-CAM cameras
which are uniformly arranged around the performer covering a view range up to
1440 degrees (4 circles at different latitudes). All the cameras are
calibrated and synchronized in advance, producing 66 RGB streams at 3840
$\times$ 2160 resolution and 25 frames per-second.

In order to obtain a high quality dataset, we have specially designed our
capture system. First, to obtain more detailed acquisition images, we orient
the cameras along the equator and on the second circle from top down to face
the lower and upper body of the performer, respectively. Cameras on the rest
two circles (the highest and the second lowest) are used to capture the
complete (full body) performer within their field-of-view. This strategy helps
to balance the resolution and reconstruction quality: if all views capture
individual fragments of the body, the calibration process will lead to large
errors and subsequently affect NeRF/NeuVV reconstruction; If all views capture
full body, the final resolution on faces and clothing will be low. Our
compromise ensures both high quality calibration and preservation of fine
details. The numbers of frames used in NeuVV range from 75 to 150 (3s to 6s),
depending on motion range and speed, in line with previous approaches.

Table 1. Quantitative comparison against several methods in terms of rendering
accuracy. Compared with ST-NeRF, NeuS, NeuralBody and iButter , our approach
achieves the best performance in PSNR,SSIM and MAE metrics. Note that NeuS is
per-frame training.

best second-best |
---|---
Method |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  MAE$\downarrow$ |  LPIPS$\downarrow$ | Realtime
Neural Body | 29.20 | 0.9777 | 0.0068 | 0.0728 | ✗
NueS | 27.07 | 0.9828 | 0.0053 | 0.0410 | ✗
iButter | 32.76 | 0.9859 | 0.0609 | 0.0032 | ✗
ST-NeRF | 32.57 | 0.9687 | 0.0043 | 0.0570 | ✗
Ours | 34.27 | 0.9875 | 0.0034 | 0.0529 | ✓

###  6.1. Rendering Comparisons

#### Comparisons to SOTA

Our approach is the first neural representation which enables real-time
dynamic rendering and editing and to the best of our knowledge. To demonstrate
the overall performance of our approach, we compare to the existing free-
viewpoint video methods based on neural rendering, including the implicit
methods NeuS (Wang et al., 2021a), iButter (Wang et al., 2021b), ST-NeRF
(Zhang et al., 2021b) and Neural Body (Peng et al., 2021) based on neural
radiance field. Note that NeuS only supports static scenes, so we only compare
single frame performance with it, the rest of methods support dynamic scenes,
we compare the whole sequence with them. For a fair comparison, all the
methods share the same training dataset as our approach. We choose 90 percent
of our captured views as training datasets, and the other 10 percent as novel
views for evaluation. As shown in Fig. 12, our approach achieves photo-
realistic free-viewpoint rendering with the most vivid rendering results in
terms of photo-realism and sharpness, which, in addition, can be done in real-
time.

For quantitative comparison, we adopt the peak signal-to-noise ratio (PSNR),
structural similarity index (SSIM), mean absolute error (MAE), and Learned
Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018) as metrics to
evaluate our rendering accuracy.

As shown in Tab. 1, our approach outperforms other methods in terms of all the
metrics for appearance. Such a qualitative comparison illustrates the
effectiveness of our approach to encode the spatial and temporal information
from our multi-view setting.

Figure 13. Qualitative evaluation on the number of bases and HH dimensions.
The setting with $C=31$ and $N=14$ achieves the satisfactory rendering quality
while higher number of bases and HH dimensions does not result in a
significant improvement.

#### Ablation Study.

We first evaluate our two main components in our method, including HH
dimensions in hyperspherical harmonic basis function and the number of
learnable bases of density and hyper angle. We perform various experiments for
different HH dimensions and latent space dimensions and decide the appro
choice of the hyperparameters in our algorithm based on the image quality
metrics, including PSNR, SSIM and MAE and the memory usage overhead.

#### Hyperspherical Harmonic Basis Function.

We first conduct an experiment to search for a compromising HH dimension $N$
in hyperspherical harmonic basis function to balance the realistic rendering
performance and memory usage.

As shown in Fig. 13 and Tab. 3, the results with $HH=11$ have a better
appearance than those using smaller hyperspherical harmonic dimensions and
have similar rendering quality and less storage cost than using even higher
dimensions. Therefore, $HH=11$ is a balanced choice on the hyperspherical
harmonic basis function.

#### Number of learnable bases.

We also carry out another experiment to explore the reasonable number of bases
$C$ for the time-varying density and hyper angles in Sec. 4.1.

As shown in Fig. 13 and Tab. 2, the results with the number of bases $C=31$
have a large improvement compared smaller number of bases, and then continue
to increase the bases number has no significant effect on the appearance
improvement but increases the memory usage. Our model keeps an outstanding
balance.

Table 2. Quantitative evaluation on the number of learnable base. Compared
with other choices, the setting with $C=31$ achieves the best balance among
rendering accuracy, time and storage.

best second-best
---
Latent dimensions | PSNR$\uparrow$ | SSIM $\uparrow$ | MAE $\downarrow$ | Storage (GB)$\downarrow$
$C=11$ | 28.99 | 0.9802 | 0.0067 | 0.716
$C=31$ (ours) | 31.01 | 0.9856 | 0.0051 | 1.427
$C=51$ | 31.04 | 0.9854 | 0.0052 | 1.534

Table 3. Quantitative evaluation on Hyperspherical Harmonic Basis Function.
Compared with other choices, the setting with $N=14$ achieves the best balance
among rendering accuracy, time and storage.

best second-best |
---|---
Basis | PSNR$\uparrow$ | SSIM $\uparrow$ | MAE $\downarrow$ | Storage (GB)$\downarrow$
$N=5$ | 28.89 | 0.9823 | 0.0066 | 0.957
$N=14$ (ours) | 31.01 | 0.9856 | 0.0051 | 1.427
$N=30$ | 31.60 | 0.9867 | 0.0048 | 2.131

###  6.2. Composition, Editing, and Lighting Effects

#### NeuVV vs. 3D Mesh.

Compared with 3D reconstruction methods, NeuVV as a hybrid implicit-explicit
representation is particularly useful to handle small, deformable, and semi-
transparent geometry. In Dunhuang flying apsaras sequence (Fig. 14), the
performer wears the traditional dancing dress with many long, narrow, thin,
and soft ribbons that exhibit complex mutual occlusions. Their geometry and
movements are difficult to recover or even manually model using 3D
representations. For example, active or passive scanning produces various
visual artifacts such as adhesiveness, holes, and noises whereas NeuVV
presents a unique advantage by faithfully reproducing plausible rendering at
any viewpoint without explicitly revealing the underlying geometry.

#### Duplication.

Fig. 7 demonstrates how to realize duplicated manifestations of the same
Dunhuang dancer. The supplementary video demonstrates how a user creates such
effects in virtual space: they first select the VOctree primitive using the
controller, then duplicate her multiple times and position individual
duplicates at different locations. Finally, they adjust the timing of the
movement of each duplicate and hit the play button on the controller to
synthesize visual effects similar to the Matrix which used to require
professional production. More excitingly, for the first time, a user can view
this effect in virtual environments. For example, by positioning the
duplications along a line, the front view produces an astounding visual effect
of a Thousand Armed Avalokiteshvara for conveying the goddess’ greatest
compassion whereas a side reveals the movements from different perspectives,
we show the similar effects in Fig. 11 Waving which to create waving effect of
the same performer. As aforementioned, duplications do not incur additional
memory cost as they share the same VOctree data and therefore it is indeed
possible to produce a multiple duplications and still render at an interactive
speed.

Figure 14. Reconstruction Result. We reconstruct one frame of our captured
dataset by RealityCapture(CapturingReality, 2021), it cannot handle small,
deformable, and semi-transparent geometry.

#### Composition.

Composition is a powerful tool in 2D videography. Composition of 3D videos
immersive environment is even more exciting. For example, to produce an
immersive musical or concert, it is essential to position pre-recorded
volumetric performances from different places around the world to the same
virtual space. Immersive viewing is achieved via our NeuVV + OpenVR framework
that support simultaneously rendering multiple VOctrees of different
performers at the same time. The current limit is on GPU memory: each VOctree
is about 1-2 GBs and on Nvidia RTX 3090 we can support at most 12 entities.
Fig. 1 shows an example that we put a ballet performance, a Dunhuang flying
apsaras, and modern dance on the same floor. Our spatial-temporal adjustment
tools can efficiently synchronize their movements where our depth sorted
rendering manages to produce correct occlusions as the viewer changes position
in virtual space. Since VOctree presents a neural volume representation with
opacity, translucency can achieve partial see-through effects.

#### Free-viewpoint Video.

A byproduct of our real-time, multi-VOctree rendering is the acceleration of
free-viewpoint video (FVV) production. Existing FVVs, especially the neural
ones (Zhang et al., 2021b), are produced offline. By providing real-time
rendering capability, NeuVV enables live feedback to the videographer, who can
adjust the position, size, and timing of the contents on the fly, greatly
improving production efficiency. With the support the latest near real-time
neural technique such as NGP (Müller et al., 2022), live performance
composition and editing in the form of NeuVVs may be practical in foreseeable
future. As illustrated in Fig. 11 When Van Gogh Meets Ballet, we show
representative frames in rendered FVV using edited VOtree, the edited results
achieve more artistic effects.

#### Lighting.

Traditionally lighting effects are achieved on explicit geometry such as
meshes. As a hybrid neural-volume representation, VOctree-based shadowing and
falloff estimation (Section 5) can produce certain lighting effects. Fig. 11
shows the lighting effect by positioning a point light source on side of
performer were the cast shadow serves as virtual motion magnifier. Nuances in
small movements such as hands and arms as well as clothes deformation are
better illustrated through time-varying shadows in real-time, adding another
layer of realism as if in real theaters. Falloff lighting further helps guide
the viewer’s focus on different parts and produce smooth transitions to real
or synthetic background. Both shadows and fallout lighting can be conducted in
one pass via the estimation of the alpha/depth map of VOctree, and by using a
low resolution shadow/depth map, they reduce the overall rendering speed (from
the viewer’s perspective) by about 20%. More advanced shading that requires
using surface normal, however, is not readily available in the current
representation, although latest extensions such as NeuS(Wang et al., 2021a)
may be integrated into VOctree as a potential remedy.

#### Interaction.

As the final example, we combine the motion of the viewer with the performer
in the experience of VR fitness training. One of the most exciting experiences
Metaverse promises is to offer live interactions with virtual characters in
virtual environments. In this specific case, a user should not only be able to
omnidirectionally watch the virtual trainer’s moves but also compare their own
moves with the trainer. In our implementation, we use a single camera motion
capture solution (He et al., 2021) that estimates 3D skeleton structures of
users as they move. We also precompute the ”ground truth” skeleton moves of
the trainer, by first rendering a multi-view video of whole body movements
also using NeuVV and then conducting multi-view skeleton estimation. Finally
we highlight skeleton discrepancies between the two on top of NeuVV rendering,
to remind the user about incorrect postures. The user can then pause and move
about the trainer with the right perspective for a replay.

###  6.3. Possible Extensions

NeuVV is designed to produce high quality multi-view video rendering instead
of 3D reconstruction, and therefore it cannot yet produce satisfactory
geometry from VOctree. Brute-force approaches such as converting per-frame
density field to meshes via thresholding and marching cubes lead to pure
reconstruction, especially under fast motions. This should be viewed as a
limitation as the results cannot be readily integrated into existing
production and rendering pipelines such as Unity, Unreal, Blender, etc., that
still rely on mesh inputs. Because the support for neural rendering is
provided on these engines, a possible extension is to resort to traditional or
neural geometric modeling tools.

For example, one can render foreground maps at an ultra dense set of views and
use the masks to conduct space carving. Alternatively, recent approaches based
on signed distance functions (SDF) such as NeuS (Wang et al., 2021a) may be
integrated into the NeuVV pipeline.

Same as existing neural approaches for handling dynamic objects, we use
relatively short footage (around 3$\sim$6 seconds). The challenges are multi-
fold. Longer clips correspond to longer training time and higher storage. In
particular, as NeuVV optimizes over all frames from all viewpoints, the memory
limit on the GPU restricts the length of the footage. Speed and memory aside,
long sequences may produce very large motions that cannot be fully capture by
HH and our learnable scheme.

One potential solution is to borrow the idea of keyframe based video
compression where the video can be truncated into smaller pieces, each
individually compressed or trained in our case. In video compression, only
changes that occur from one frame to the next are stored in the data stream.
It is possible that we can apply NeuVV training only on the residues, e.g., by
pre-processing videos at individual viewpoint and set out to optimize the
changes rather than the complete frames. Such a scheme may also provide a
viable streaming strategy of NeuVV and is our immediate future work.

Though our NeuVV exhibits capacity in photo-realistic rendering and editing of
volumetric video content in real-time, there are several limitations and
consequently possible extensions to our approach. Firstly, our NeuVV is a NeRF
based representation, compared to NeRF’s compelling novel view synthesis
ability, the geometry recovered is general lower quality.

Similarly, our NeuVV suffers the same geometry recovery problem given a static
time frame. Moreover, the recovered geometries exhibits ghosting effect when
the performer’s motion is too fast. This is because the change of volume
density is constraint by learnable bases, which can well handle smooth motion
but reluctant to fast density changes.

The lack of high quality geometry greatly limits the application of NeuVV as
current industrial graphics rendering engines, such as OpenGL and Unity3D,
only support a mesh based geometry representation. Before a natural
integration of neural rendering into traditional rendering engines, an
possible extension is resorting to cooperate with stronger geometry recovery
approaches, such as the signed distance function (SDF) and neural graphic
primitives.

Moreover, all videos demonstrations in our paper are relatively short (around
3$\sim$6 seconds) as NeuVV is more difficult to converge when the input video
is long. Also we may have to sacrifice some storage for high quality rendering
as motions in longer videos are likely to be complicated and we have to use
higher dimensions of the latent space to account for the complex motion.

We can borrow the concept of key frames in video compression to potentially
solve this problem. Particularly, we can separate a long video into small
segments, and each segment is defined by a key frame. Within each segment,
motion of the performer is relatively small. And hence we can optimize one
NeuVV for each segment effectively.

Finally, transferring NeuVV over internet is not efficient as we have to send
the whole volume representation at once, no matter which frame is of the
viewer’s interest. One possible solution is to directly slice the VOctree at a
given time frame to obtain the SH coefficient, and transform the the time
frame into a PlenOctree representation and then compress and transmit over
internet.

##  7\. CONCLUSION

We present a new neural volumography technique, NeuVV, which leverages the
neural rendering technique to tackle volumetric videos. We model the scene
captured by a volumetric video as a dynamic radiance field function, which
maps a 6D vector (3D position + 2D view direction + 1D time) to color and
density. Our NeuVV encodes a dynamic radiance field effectively, as the core
at our NeuVV is a factorization schemes by hyperspherical harmonics, to
account for the angular and temporal variations at each position. Density at a
specific position only exhibits temporal variations while being invariant to
view directions. Hence we further develop a learnable basis representation for
temporal compaction of densities. Similar to the PlenOctree (Yu et al.,
2021b), our NeuVV can be easily converted into an octree based representation,
which we call VOctree, for real-time rendering and editing. NeuVV tackles a
volumetric video sequence as a whole therefore reduces the memory overhead and
computational time by two orders of magnitudes.

For demonstration, we further provides tools based on NeuVV for flexibly
composing multiple performances in 3D space, enabling interactive editing in
both spatial and temporal dimensions, and rendering a new class of volumetric
special effects with high photo-realism. More specifically, we demonstrate
that NeuVV, or VOctree more precisely, allows for real-time adjustments of the
3D locates, scales of multiple performers, re-timing and thus coordinating the
performers, and even duplicating the same performer to produce varied
manifestations in space and time. To the best of our knowledge, NeuVV is the
first neural based volumography technique that supports real-time rendering
and interactive editing of volumetric videos.

## References

  * (1)
  * Ahmed et al. (2008) Naveed Ahmed, Christian Theobalt, Christian Rossl, Sebastian Thrun, and Hans-Peter Seidel. 2008\.  Dense correspondence finding for parametrization-free animation reconstruction from video. In _2008 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 1–8.
  * Anderson et al. (2016) Robert Anderson, David Gallup, Jonathan T Barron, Janne Kontkanen, Noah Snavely, Carlos Hernández, Sameer Agarwal, and Steven M Seitz. 2016.  Jump: virtual reality video.  _ACM Transactions on Graphics (TOG)_ 35, 6 (2016), 1–13.
  * Avery (2012) John S Avery. 2012\.  _Hyperspherical harmonics: applications in quantum theory_. Vol. 5.  Springer Science & Business Media.
  * Bertel et al. (2020) Tobias Bertel, Mingze Yuan, Reuben Lindroos, and Christian Richardt. 2020.  OmniPhotos: Casual 360° VR Photography with Motion Parallax. In _SIGGRAPH Asia 2020 Emerging Technologies_ (Virtual Event, Republic of Korea) _(SA ’20)_. Association for Computing Machinery, New York, NY, USA, Article 19, 2 pages.  https://doi.org/10.1145/3415255.3422884
  * Blanco et al. (1997) Miguel A. Blanco, M. Flórez, and M. Bermejo. 1997\.  Evaluation of the rotation matrices in the basis of real spherical harmonics.  _Journal of Molecular Structure: THEOCHEM_ 419, 1 (1997), 19–27.  https://doi.org/10.1016/S0166-1280(97)00185-1
  * Bonvallet et al. (2007) Bryan Bonvallet, Nikolla Griffin, and Jia Li. 2007.  A 3D Shape Descriptor: 4D Hyperspherical Harmonics ”an Exploration into the Fourth Dimension”. In _Proceedings of the IASTED International Conference on Graphics and Visualization in Engineering_ (Clearwater, Florida) _(GVE ’07)_. ACTA Press, USA, 113–116.
  * Bozic et al. (2020) Aljaz Bozic, Michael Zollhofer, Christian Theobalt, and Matthias Nießner. 2020.  Deepdeform: Learning non-rigid rgb-d reconstruction with semi-supervised data. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7002–7012.
  * Broxton et al. (2020a) Michael Broxton, John Flynn, Ryan Overbeck, Daniel Erickson, Peter Hedman, Matthew Duvall, Jason Dourgarian, Jay Busch, Matt Whalen, and Paul Debevec. 2020a.  Immersive light field video with a layered mesh representation.  _ACM Transactions on Graphics (TOG)_ 39, 4 (2020), 86–1.
  * Broxton et al. (2020b) Michael Broxton, John Flynn, Ryan Overbeck, Daniel Erickson, Peter Hedman, Matthew Duvall, Jason Dourgarian, Jay Busch, Matt Whalen, and Paul Debevec. 2020b.  Immersive Light Field Video with a Layered Mesh Representation.  _ACM Trans. Graph._ 39, 4, Article 86 (jul 2020), 15 pages.  https://doi.org/10.1145/3386569.3392485
  * Buehler et al. (2001) Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen. 2001.  Unstructured lumigraph rendering. In _Proceedings of the 28th annual conference on Computer graphics and interactive techniques_. 425–432.
  * CapturingReality (2021) CapturingReality. 2021\.  Reality Capture.  https://www.capturingreality.com/
  * Carranza et al. (2003) Joel Carranza, Christian Theobalt, Marcus A Magnor, and Hans-Peter Seidel. 2003.  Free-viewpoint video of human actors.  _ACM transactions on graphics (TOG)_ 22, 3 (2003), 569–577.
  * Collet et al. (2015) Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan. 2015.  High-quality streamable free-viewpoint video.  _ACM Transactions on Graphics (TOG)_ 34, 4 (2015), 69\.
  * Debevec et al. (1996) Paul E Debevec, Camillo J Taylor, and Jitendra Malik. 1996\.  Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In _Proceedings of the 23rd annual conference on Computer graphics and interactive techniques_. 11–20.
  * Facebook Technologies (2020) LLC Facebook Technologies. 2020\.  Oculus Quest 2.  https://www.oculus.com/quest-2/
  * Gortler et al. (1996) Steven J Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F Cohen. 1996.  The lumigraph. In _Proceedings of the 23rd annual conference on Computer graphics and interactive techniques_. 43–54.
  * He et al. (2021) Yannan He, Anqi Pang, Xin Chen, Han Liang, Minye Wu, Yuexin Ma, and Lan Xu. 2021.  Challencap: Monocular 3d capture of challenging human performances using multi-modal references. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 11400–11411.
  * Hongda International Electronics Co. (2020) Ltd. Hongda International Electronics Co. 2020\.  Oculus Quest 2.  https://www.vive.com/sea/product/vive-pro2/overview/
  * Insta360 (2020) Insta360. 2020.  Insta360 One X2.  https://www.insta360.com/product/insta360-onex2
  * Joo et al. (2018) Hanbyul Joo, Tomas Simon, and Yaser Sheikh. 2018.  Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies. In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Kasten et al. (2021) Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel. 2021\.  Layered Neural Atlases for Consistent Video Editing.  40, 6, Article 210 (dec 2021), 12 pages.  https://doi.org/10.1145/3478513.3480546
  * Kutulakos and Seitz (2000) Kiriakos N Kutulakos and Steven M Seitz. 2000.  A theory of shape by space carving.  _International journal of computer vision_ 38, 3 (2000), 199–218.
  * Levoy and Hanrahan (1996) Marc Levoy and Pat Hanrahan. 1996.  Light field rendering. In _Proceedings of the 23rd annual conference on Computer graphics and interactive techniques_. 31–42.
  * Li et al. (2021) Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, and Zhaoyang Lv. 2021\.  Neural 3d video synthesis.  _arXiv preprint arXiv:2103.02597_ (2021).
  * Li et al. (2019) Zhengqi Li, Tali Dekel, Forrester Cole, Richard Tucker, Noah Snavely, Ce Liu, and William T Freeman. 2019.  Learning the depths of moving people by watching frozen people. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 4521–4530.
  * Li et al. (2020) Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. 2020.  Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes.  _arXiv preprint arXiv:2011.13084_ (2020).
  * Lindell et al. (2020) David Lindell, Julien Martel, and Gordon Wetzstein. 2020\.  AutoInt: Automatic Integration for Fast Neural Volume Rendering.  _https://arxiv.org/abs/2012.01714_ (2020).
  * Liu et al. (2020) Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. 2020.  Neural sparse voxel fields.  _arXiv preprint arXiv:2007.11571_ (2020).
  * Liu et al. (2009) Yebin Liu, Qionghai Dai, and Wenli Xu. 2009.  A point-cloud-based multiview stereo algorithm for free-viewpoint video.  _IEEE transactions on visualization and computer graphics_ 16, 3 (2009), 407–418.
  * Lombardi et al. (2016) Andrea Lombardi, Federico Palazzetti, Vincenzo Aquilanti, Gaia Grossi, Alessandra Albernaz, Patricia Barreto, and Ana Cruz. 2016.  Spherical and hyperspherical harmonics representation of van der Waals aggregates, Vol. 1790. 020005\.  https://doi.org/10.1063/1.4968631
  * Lombardi et al. (2019) Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. 2019\.  Neural Volumes: Learning Dynamic Renderable Volumes from Images.  _ACM Trans. Graph._ 38, 4, Article 65 (July 2019), 14 pages.  https://doi.org/10.1145/3306346.3323020
  * Lombardi et al. (2021) Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason Saragih. 2021.  Mixture of Volumetric Primitives for Efficient Neural Rendering.  _ACM Trans. Graph._ 40, 4, Article 59 (jul 2021), 13 pages.  https://doi.org/10.1145/3450626.3459863
  * Lu et al. (2020) Erika Lu, Forrester Cole, Tali Dekel, Weidi Xie, Andrew Zisserman, David Salesin, William T. Freeman, and Michael Rubinstein. 2020.  Layered Neural Rendering for Retiming People in Video.  arXiv:2009.07833 [cs.CV]
  * Mildenhall et al. (2019) Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. 2019\.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  _ACM Transactions on Graphics (TOG)_ 38, 4 (2019), 1–14.
  * Mildenhall et al. (2020) Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2020\.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _arXiv preprint arXiv:2003.08934_ (2020).
  * Müller et al. (2022) Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022.  Instant Neural Graphics Primitives with a Multiresolution Hash Encoding.  _arXiv preprint arXiv:2201.05989_ (2022).
  * Newcombe et al. (2015) Richard A Newcombe, Dieter Fox, and Steven M Seitz. 2015\.  Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 343–352.
  * Orts-Escolano et al. (2016) Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L. Davidson, Sameh Khamis, Mingsong Dou, Vladimir Tankovich, Charles Loop, Qin Cai, Philip A. Chou, Sarah Mennicken, Julien Valentin, Vivek Pradeep, Shenlong Wang, Sing Bing Kang, Pushmeet Kohli, Yuliya Lutchyn, Cem Keskin, and Shahram Izadi. 2016\.  Holoportation: Virtual 3D Teleportation in Real-Time. In _Proceedings of the 29th Annual Symposium on User Interface Software and Technology_ (Tokyo, Japan) _(UIST ’16)_. Association for Computing Machinery, New York, NY, USA, 741–754.  https://doi.org/10.1145/2984511.2984517
  * Park et al. (2019) Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019\.  Deepsdf: Learning continuous signed distance functions for shape representation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 165–174.
  * Park et al. (2020) Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo-Martin Brualla. 2020\.  Deformable Neural Radiance Fields.  _arXiv preprint arXiv:2011.12948_ (2020).
  * Park et al. (2021) Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz. 2021.  HyperNeRF: a higher-dimensional representation for topologically varying neural radiance fields.  _ACM Transactions on Graphics (TOG)_ 40, 6 (2021), 1–12.
  * Pasha Hosseinbor et al. (2015) A. Pasha Hosseinbor, Moo K. Chung, Cheng Guan Koay, Stacey M. Schaefer, Carien M. van Reekum, Lara Peschke Schmitz, Matt Sutterer, Andrew L. Alexander, and Richard J. Davidson. 2015.  4D hyperspherical harmonic (HyperSPHARM) representation of surface anatomy: A holistic treatment of multiple disconnected anatomical structures.  _Medical Image Analysis_ 22, 1 (2015), 89–101.  https://doi.org/10.1016/j.media.2015.02.004
  * Peng et al. (2021) Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. 2021.  Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 9054–9063.
  * Pumarola et al. (2020) Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. 2020.  D-NeRF: Neural Radiance Fields for Dynamic Scenes.  _arXiv preprint arXiv:2011.13961_ (2020).
  * Reiser et al. (2021) Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. 2021\.  KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs.  arXiv:2103.13744 [cs.CV]
  * Seitz and Dyer (1999) Steven M Seitz and Charles R Dyer. 1999.  Photorealistic scene reconstruction by voxel coloring.  _International Journal of Computer Vision_ 35, 2 (1999), 151–173.
  * Sitzmann et al. (2021) Vincent Sitzmann, Semon Rezchikov, William T Freeman, Joshua B Tenenbaum, and Fredo Durand. 2021\.  Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering.  _arXiv preprint arXiv:2106.02634_ (2021).
  * Slavcheva et al. (2017) Miroslava Slavcheva, Maximilian Baust, Daniel Cremers, and Slobodan Ilic. 2017.  Killingfusion: Non-rigid 3d reconstruction without correspondences. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1386–1395.
  * Slavcheva et al. (2018) Miroslava Slavcheva, Maximilian Baust, and Slobodan Ilic. 2018\.  Sobolevfusion: 3d reconstruction of scenes undergoing free non-rigid motion. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 2646–2655.
  * Snavely et al. (2006) Noah Snavely, Steven M Seitz, and Richard Szeliski. 2006\.  Photo tourism: exploring photo collections in 3D.  In _ACM siggraph 2006 papers_. 835–846.
  * Srinivasan et al. (2019) Pratul P Srinivasan, Richard Tucker, Jonathan T Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely. 2019.  Pushing the boundaries of view extrapolation with multiplane images. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 175–184.
  * Suo et al. (2021) Xin Suo, Yuheng Jiang, Pei Lin, Yingliang Zhang, Minye Wu, Kaiwen Guo, and Lan Xu. 2021.  NeuralHumanFVV: Real-Time Neural Volumetric Human Performance Rendering using RGB Cameras. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 6226–6237.
  * Tewari et al. (2021) A. Tewari, O. Fried, J. Thies, V. Sitzmann, S. Lombardi, Z. Xu, T. Simon, M. Nießner, E. Tretschk, L. Liu, B. Mildenhall, P. Srinivasan, R. Pandey, S. Orts-Escolano, S. Fanello, M. Guo, G. Wetzstein, J.-Y. Zhu, C. Theobalt, M. Agrawala, D. B Goldman, and M. Zollhöfer. 2021.  Advances in Neural Rendering. In _ACM SIGGRAPH 2021 Courses_ (Virtual Event, USA) _(SIGGRAPH ’21)_. Association for Computing Machinery, New York, NY, USA, Article 1, 320 pages.  https://doi.org/10.1145/3450508.3464573
  * Tretschk et al. (2020) Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. 2020.  Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video.  _arXiv preprint arXiv:2012.12247_ (2020).
  * Vlasic et al. (2009) Daniel Vlasic, Pieter Peers, Ilya Baran, Paul Debevec, Jovan Popović, Szymon Rusinkiewicz, and Wojciech Matusik. 2009.  Dynamic shape capture using multi-view photometric stereo.  In _ACM SIGGRAPH Asia 2009 papers_. 1–11.
  * Wang et al. (2021b) Liao Wang, Ziyu Wang, Pei Lin, Yuheng Jiang, Xin Suo, Minye Wu, Lan Xu, and Jingyi Yu. 2021b.  iButter: Neural Interactive Bullet Time Generator for Human Free-viewpoint Rendering. In _Proceedings of the 29th ACM International Conference on Multimedia_. 4641–4650.
  * Wang et al. (2021a) Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. 2021a.  NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction.  _arXiv preprint arXiv:2106.10689_ (2021).
  * Wu et al. (2011) Chenglei Wu, Kiran Varanasi, Yebin Liu, Hans-Peter Seidel, and Christian Theobalt. 2011.  Shading-based dynamic shape refinement from multi-view video under general illumination. In _2011 International Conference on Computer Vision_. IEEE, 1108–1115.
  * Wu et al. (2020) Minye Wu, Yuehao Wang, Qiang Hu, and Jingyi Yu. 2020\.  Multi-View Neural Human Rendering. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Xian et al. (2020) Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. 2020.  Space-time Neural Irradiance Fields for Free-Viewpoint Video.  _arXiv preprint arXiv:2011.12950_ (2020).
  * Yao et al. (2018) Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. 2018.  Mvsnet: Depth inference for unstructured multi-view stereo. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 767–783.
  * Yu et al. (2021a) Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. 2021a.  Plenoxels: Radiance Fields without Neural Networks.  _arXiv preprint arXiv:2112.05131_ (2021).
  * Yu et al. (2021b) Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. 2021b.  Plenoctrees for real-time rendering of neural radiance fields.  _arXiv preprint arXiv:2103.14024_ (2021).
  * Yu et al. (2017) Tao Yu, Kaiwen Guo, Feng Xu, Yuan Dong, Zhaoqi Su, Jianhui Zhao, Jianguo Li, Qionghai Dai, and Yebin Liu. 2017.  Bodyfusion: Real-time capture of human motion and surface geometry using a single depth camera. In _Proceedings of the IEEE International Conference on Computer Vision_. 910–919.
  * Yu et al. (2018) Tao Yu, Zerong Zheng, Kaiwen Guo, Jianhui Zhao, Qionghai Dai, Hao Li, Gerard Pons-Moll, and Yebin Liu. 2018\.  Doublefusion: Real-time capture of human performances with inner body shapes from a single depth sensor. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 7287–7296.
  * Zhang et al. (2021a) Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao, Yanshun Zhang, Minye Wu, Yingliang Zhang, Lan Xu, and Jingyi Yu. 2021a.  Editable Free-Viewpoint Video Using a Layered Neural Representation.  _ACM Trans. Graph._ 40, 4, Article 149 (jul 2021), 18 pages.  https://doi.org/10.1145/3450626.3459756
  * Zhang et al. (2021b) Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao, Yanshun Zhang, Minye Wu, Yingliang Zhang, Lan Xu, and Jingyi Yu. 2021b.  Editable free-viewpoint video using a layered neural representation.  _ACM Transactions on Graphics (TOG)_ 40, 4 (2021), 1–18.
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In _CVPR_.
  * Zhao et al. (2021) Fuqiang Zhao, Wei Yang, Jiakai Zhang, Pei Lin, Yingliang Zhang, Jingyi Yu, and Lan Xu. 2021.  HumanNeRF: Generalizable Neural Human Radiance Field from Sparse Inputs.  _arXiv preprint arXiv:2112.02789_ (2021).
  * Zheng et al. (2018) Zerong Zheng, Tao Yu, Hao Li, Kaiwen Guo, Qionghai Dai, Lu Fang, and Yebin Liu. 2018.  Hybridfusion: Real-time performance capture using a single depth sensor and sparse imus. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 384–400.
  * Zitnick et al. (2004) C Lawrence Zitnick, Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski. 2004\.  High-quality video view interpolation using a layered representation.  _ACM transactions on graphics (TOG)_ 23, 3 (2004), 600–608.

## APPENDIX

###  A.1. Complex HH in 4D Hyperspherical Coordinates

Hyperspherical Harmonics are widely used in quantum mechanical and chemistry
field to solve few-body systems. It also has been used in computer graphics
visualization (Bonvallet et al., 2007) and the representation of complicated
brain subcortical structures (Pasha Hosseinbor et al., 2015)

4D complex Hyperspherical Harmonics can be derived from as complex 3D
Shpherical Harmonics (Lombardi et al., 2016)

(14)
$\mathcal{H}^{m}_{nl}(\theta,\phi,\gamma)=A_{n,l}\sin^{l}(\gamma)C^{l+1}_{n-l}\big{(}\cos(\gamma)\big{)}\mathcal{S}^{m}_{l}(\theta,\phi)$

where

(15) $A_{n,l}=(2l)!!\sqrt{\frac{2(n+1)(n-l+1)!}{\pi(n+l+1)!}}$

$\gamma,\theta\in[0,\pi]$, $\phi\in[0,2\pi]$ , $C^{l+1}_{n-1}$ are Gengenbauer
polynomials, and $\mathcal{S}^{m}_{l}$ are the 3D spherical harmonics. $l,m,n$
are integers, where $l$ denotes the degree of the HH, $m$ is the order, and
$n=0,1,2,...$, following $0\leq l\leq n$ and $-l\leq m\leq l$.

3D spherical harmonics $Y^{m}_{l}(\theta,\phi)$ are defined as below:

(16) $Y^{m}_{l}(\theta,\phi)=K^{m}_{l}P^{m}_{l}(\cos\theta)e^{im\phi}$

where

(17) $K^{m}_{l}=(-1)^{m}\sqrt{\dfrac{2l+1}{4\pi}\dfrac{(l-m)!}{(l+m)!}}$

and $P^{m}_{l}$ is associated Legendre polynomials.

###  A.2. Real-valued HH in 4D Cartesian Coordinates

It is hard to directly use HH in complex space in our approach as it has a
heavy burden for calculating its imaginary part and optimizing our network
weights by traditional grad descent methods. Thus, we derived how to transform
a 4D complex HH to be in real space. We have implemented a program to
iteratively solve and verify $N-dimensional$ HH basis function. And we will
release the code in the future.

real-valued HH in 4D Cartesian space input a 4D unit vector
$\mathbf{x}=[x_{1},x_{2},x_{3},x_{4}]^{T}$, the relationship between
$\mathbf{x}$ and $(\gamma,\theta,\phi)$ is as below:

$\displaystyle x_{1}$ $\displaystyle=$ $\displaystyle\cos(\gamma)$
$\displaystyle x_{2}$ $\displaystyle=$ $\displaystyle\sin(\gamma)\cos(\theta)$
$\displaystyle x_{3}$ $\displaystyle=$
$\displaystyle\sin(\gamma)\sin(\theta)\cos(\phi)$ $\displaystyle x_{4}$
$\displaystyle=$ $\displaystyle\sin(\gamma)\sin(\theta)\sin(\phi)$

where $\sum_{i=1}^{n}x_{i}^{2}=1$.

The real-valued SH $Y_{lm}$ has been given as (Blanco et al., 1997)

(19) $Y_{lm}=\left\\{\begin{aligned}
\frac{1}{\sqrt{2}}\left(Y^{m}_{l}+(-1)^{m}Y^{-m}_{l}\right)\quad\text{if
}m>0\\\ Y^{m}_{l}\quad\text{if }m=0\\\
\frac{1}{\sqrt{2}}\left(Y^{-m}_{l}-(-1)^{m}Y^{m}_{l}\right)\quad\text{if
}m<0\\\ \end{aligned}\right.$

We observe that the similar idea can be used to obtain real-valued HSH
$\mathcal{H}_{nlm}(\theta,\phi,\gamma)$ as the complex number is only from
$Y^{m}_{l}$, combine Eqn. 14 with Eqn. 19:

(20)
$\mathcal{H}_{nlm}(\theta,\phi,\gamma)=A_{n,l}sin^{l}(\gamma)C^{l+1}_{n-l}(\cos(\gamma))Y_{lm}(\theta,\phi)$

Finally, to transform 4D hypersphere coordinates to 4D Cartesian coordinates,
we then substitute $(\gamma,\theta,\phi)$ with $\mathbf{x}$. Using the same
definition of Eqn. A.2. We further introduce a separated Cartesian form of
$Y_{lm}(x_{2},x_{3},x_{4})$ in 3D Cartesian coordinates.

(21) $\begin{bmatrix}Y_{lm}\\\
Y_{l-m}\end{bmatrix}=\sqrt{\dfrac{2l+1}{4\pi}}\bar{\prod}^{m}_{l}(x_{2})\begin{bmatrix}A_{m}\\\
B_{m}\end{bmatrix},m>0$ (22)
$Y_{l0}=\sqrt{\dfrac{2l+1}{4\pi}}\bar{\prod}^{m}_{0}(x_{2})$

where

(23) $\displaystyle A_{m}(x_{3},x_{4})$ $\displaystyle=$
$\displaystyle\sum^{m}_{p=0}\tbinom{m}{p}x_{3}^{p}x_{4}^{m-p}\cos((m-p)\frac{\pi}{2})$
(24) $\displaystyle B_{m}(x_{3},x_{4})$ $\displaystyle=$
$\displaystyle\sum^{m}_{p=0}\tbinom{m}{p}x_{3}^{p}x_{4}^{m-p}\sin((m-p)\frac{\pi}{2})$

and

$\displaystyle\bar{\prod}^{m}_{l}(x_{2})$ $\displaystyle=$
$\displaystyle\sqrt{\dfrac{(l-m)!}{(l+m)!}}\sum\limits^{\lfloor(l-m)/2\rfloor}_{k=0}B_{k,lm}x_{2}^{l-2k-m}$
(25) $\displaystyle B_{k,lm}$ $\displaystyle=$
$\displaystyle(-1)^{k}2^{-l}\tbinom{l}{k}\tbinom{2l-2k}{l}\dfrac{(l-2k)!}{(l-2k-m)!}$

Finally, we have:

(27) $\begin{bmatrix}\mathcal{H}_{nlm}\\\
\mathcal{H}_{nl-m}\end{bmatrix}=A_{n,l}(1-x_{1}^{2})^{l/2}C^{n+l}_{n-l}(x_{1})\begin{bmatrix}Y_{lm}\\\
Y_{l-m}\end{bmatrix},m>0$

When $m=0$,

(28)
$\mathcal{H}_{nl0}=A_{n,l}(1-x_{1}^{2})^{l/2}C^{n+l}_{n-l}(x_{1})\cdot\sqrt{\dfrac{2l+1}{4\pi}}\bar{\prod}^{0}_{l}(x_{2})$

Using Eqn. 27 and Eqn. 28, we can derive the simplest forms of HH basis. The
similar idea can be used to derive more higher dimensional HH basis functions.