Finally, let us suppose that $1\leq{\prod_{{1\leq j<k\leq
d-1}}}\frac{w_{j}-w_{k}+k-j}{k-j}=\ell\leq d-2$. First, if $\ell=1$, the only
possible choice of the weight $w$ is $w_{1}=\cdots=w_{d-1}$. We must then
ensure that

$\prod_{1\leq j\leq d-1}\frac{w_{j}+d-j}{d-j}=d,$

which is possible only if $w_{1}=\cdots=w_{d-1}=1$. This, together with the
choice $w_{d}=0$ that we have made earlier, proves that $w=(1,\ldots,1,0)$
providing the second choice in the induction step. In particular, the
dimension of the representation determined by the weight $(1,1,\ldots,1,0)$ is
$d$. Now, we must establish that there is no other choice of $w$ satisfying
(5.2). This follows from Lemma 5.6 proved below. It is also easy to verify
directly: If $d=2$ or $3$, there is nothing more to be done. If $d>3$, then
fix $\ell:2\leq\ell\leq d-2$, and pick $w$ such that ${\prod_{{1\leq j<k\leq
d-1}}}\frac{w_{j}-w_{k}+k-j}{k-j}=d-\ell$. Having picked $w$, we also need

$\frac{d}{d-\ell}=\prod_{1\leq j\leq d-1}\frac{w_{j}+d-j}{d-j},$

that is,

$d!=(w_{1}+d-1)\cdots(w_{\ell}+d-\ell)(d-\ell)(w_{\ell+1}+d-\ell-1)\cdots(w_{d-1}+1),$

which is valid only if $w$ is of the form
$(1,\ldots,1,w_{\ell}=1,0,\ldots,0)$. For this choice of $w$, we see that

${\prod_{{1\leq j<k\leq d-1}}}\frac{w_{j}-w_{k}+k-j}{k-j}=\binom{d-1}{\ell},$

which can’t be equal to $\ell$ for any $d>3$. So, there are no more admissible
weights in this case. This completes the verification of the induction step
and therefore the proof of the claim. Now, the result follows directly from
Proposition 5.4. ∎

###### Proof of Lemma 5.6.

The proof is by induction on the dimension $d$. The base case of $d=3$ is
easily verified. Now, we assume by the induction hypothesis, that there are no
irreducible unitary representation such that

$2\leq t:={\prod_{{1\leq j<k\leq d-1}}}\frac{w_{j}-w_{k}+k-j}{k-j}\leq d-2.$

Thus the only choice for $t$ is either $t=1$, or $t\geq d-1$. To complete the
induction step, we have to show that there is no weight
$w=(w_{1},\ldots,w_{d-1},0)$ such that

$2\leq\ell:=\displaystyle{\prod_{\stackrel{{\scriptstyle 1\leq j<k\leq
d}}{{w_{d}=0}}}}\frac{w_{j}-w_{k}+k-j}{k-j}\leq d-1.$

If $t=1$, then the only possible choice of the weight $w$ is
$w_{1}=\cdots=w_{d-1}$, say $u$. From Equation (5.3), it follows that

$\prod_{1\leq j\leq d-1}\frac{u+d-j}{d-j}=\ell.$

However since the product on the left hand side of the equation above is an
increasing function of $u$ and its smallest value is $1$, the next possible
value is $d$, it follows that the value $\ell:2\leq\ell\leq d-1$ is not taken.
Now, let $t\geq d-1$ for some $w$. Then from Equation (5.3), we see that

$\frac{\ell}{t}=\prod_{1\leq j\leq d-1}\frac{w_{j}+d-j}{d-j}$

to ensure the existence of a $\ell$-dimensional representation. Since
$\tfrac{\ell}{t}\leq 1$ while the product on the right hand side of the
equation above is greater or equal to $1$, it follows that the two sides can
be equal only if $w_{1}=\cdots=w_{d-1}=0$. But then $t$ must be equal to $1$
contrary to our hypothesis. ∎

A. Koranyi has pointed out that $SU(d)$ is a simple Lie group with discrete
center and its Lie algebra $su(d)$ is simple. Therefore any non-trivial
homomorphism of it can have at most a discrete null space, i.e., has to be a
local isomorphism. So the image of a representation is a closed subgroup of
$\mathcal{U}(n)$, therefore must have the same dimension (as a Lie group) as
$SU(d)$. If $d>n$, then this is not possible proving Lemma 5.6.

E. K. Narayanan observed that a proof of Lemma 5.5 follows from the
description of the Lie algebra homomorphisms from $su(d)$ to $u(d)$, the Lie
algebra of $\mathcal{U}(d)$. A. Khare and C. Varughese independently of each
other have provided the following argument proving Lemma 5.5: Since $su(d)$ is
simple and $u(d)=su(d)\oplus\mathbb{R}$, it follows that any Lie algebra
homomorphism must map $su(d)$ to itself isomorphically. Also, the inequivalent
representastions of $su(d)$ are characterized by the outer automorphisms.
These are in one to one correspondence with automorphisms of the corresponding
Dynkin diagram. The Dynkin diagram of $su(d)$ is $A_{(d-1)}$ consisting of
$d-1$ dots connected by single lines. For $d>2$, the (graph) automorphism
group of $A_{(d-1)}$ is of order $2$ (identity and a reflection). It follows
that there are at most two inequivalent irreducible unitary representations of
$SU(d)$, $d\geq 2$.

We believe, it will be interesting to find an answer to the two questions: (a)
What possible values $\dim\pi$ can take if $d$ is fixed. (b) If $d$ and
$n=\dim\pi$ are fixed, how many $n$-dimensional inequivalent irreducible
unitary representations are there of the group $SU(d)$.

#### Acknowledgment

The authors are grateful to E. K. Narayanan for several lectures explaining
the parametrization and realization of the irreducible unitary representations
of $\mathcal{U}(n)$ and for going through some of the proofs carefully. We
also thank Sameer Chavan for several comments on a preliminary draft of this
paper.

## References

  * [1] J. Arazy, _A survey of invariant Hilbert spaces of analytic functions on bounded symmetric domains_ , in: Contemporary Mathematics, vol. 185 (1995), 7-65.
  * [2] O. P. Agrawal and N. Salinas, _Sharp kernels and canonical subspaces_ , Amer. J. Math. 110 (1988), 23-47.
  * [3] B. Bagchi and G. Misra, _Homogeneous tuples of multiplication operators on twisted Bergman spaces_ , J. Funct. Anal. 136 (1996), 171-213.
  * [4] B. Bagchi, S. Hazra and G. Misra, _A product formula for homogeneous characteristic functions_ , Integral Equations and Operator Theory, 95 (2023), Paper No. 8.
  * [5] S. Chavan and D. Yakubovich, _Spherical tuples of Hilbert space operators_ , Indiana Univ. Math. J. 64 (2015), 577-612.
  * [6] R. E. Curto and N. Salinas, _Generalized Bergman kernels and the Cowen-Douglas theory_ , Amer. J. Math. 106 (1984), 447-488.
  * [7] R. G. Douglas and G. Misra, _Equivalence of quotient Hilbert modules_ , Proc. Indian Acad. Sci. Math. Sci. 113 (2003), 281-291.
  * [8] R. G. Douglas and G. Misra, _Quasi-free resolutions of Hilbert modules_ , Integr. equ. oper. theory, 47 (2003), 435-456.
  * [9] J. Faraut and A. Korányi, _Function spaces and reproducing kernels on bounded symmetric domains_ , J. Funct. Anal. 88 (1990), 64-89.
  * [10] S. Ghara, S. Kumar, and P. Pramanick, _K-homogeneous tuple of operators on bounded symmetric domains_ , Isr. J. Math. 247 (2022), 331-360.
  * [11] S. Ghara and G. Misra, _Decomposition of the tensor product of two Hilbert modules_ , Operator theory, operator algebras and their interactions with geometry and topology – Ronald G. Douglas memorial volume, Birkhäuser/Springer, Cham (2020), 221-265.
  * [12] A. W. Knapp, _Representation theory of Semisimple Groups: An overview Based on examples_ , Princeton Landmarks in Mathematics, Princeton University Press, Princeton (2001).
  * [13] A. Korányi and G. Misra, _Homogeneous operators on Hilbert spaces of holomorphic functions_ , J. Funct. Anal. 254 (2008), 2419–2436.
  * [14] R. Narasimhan, _Several complex variables_ , Chicago Lect. Math., The University of Chicago Press, 1971.
  * [15] A. Sengupta, _Representing Finite Groups. A Semisimple Introduction_ , Springer, New York, NY, 2012.
  * [16] M. Taylor, _Lectures on Lie Groups_ , AMS Open Math Notes, 2017.
  * [17] S. Trivedi, _Bounded point evaluation for a finitely multicyclic commuting tuple of operators_ , Bull. Sci. Math. 162 (2020), 102875.
  * [18] H. Upmeier, _Toeplitz operators and index theory in several complex variables_ , Operator Theory: Advances and Applications, vol. 81, Birkhäuser Verlag, Basel, 1996.
  * [19] H. Upmeier, _Eigenvalues of $K$-invariant Toeplitz operators on bounded symmetric domains_, Integr. Equ. Oper. Theory, 93, (2021), Article no. 27.


*.jpg,.png,.PNG

# Space-time Neural Irradiance Fields for Free-Viewpoint Video

Wenqi Xian∗
Cornell Tech  Jia-Bin Huang
Virginia Tech  Johannes Kopf
Facebook  Changil Kim
Facebook

###### Abstract

††∗ This work was done while Wenqi was an intern at Facebook.

We present a method that learns a spatiotemporal neural irradiance field for
dynamic scenes from a single video. Our learned representation enables free-
viewpoint rendering of the input video. Our method builds upon recent advances
in implicit representations. Learning a spatiotemporal irradiance field from a
single video poses significant challenges because the video contains only one
observation of the scene at any point in time. The 3D geometry of a scene can
be legitimately represented in numerous ways since varying geometry (motion)
can be explained with varying appearance and vice versa. We address this
ambiguity by constraining the time-varying geometry of our dynamic scene
representation using the scene depth estimated from video depth estimation
methods, aggregating contents from individual frames into a single global
representation. We provide an extensive quantitative evaluation and
demonstrate compelling free-viewpoint rendering results.

https://video-nerf.github.io

Figure 1:  Our method takes a _single_ casually captured video as input and
learns a space-time neural irradiance field. (_Top_) Sample frames from the
input video. (_Middle_) Novel view images rendered from textured meshes
constructed from depth maps. (_Bottom_) Our results rendered from the proposed
space-time neural irradiance field.

##  1 Introduction

This paper addresses the problem of rendering a video from novel viewpoints.
Specifically, we learn a _globally consistent, dynamic_ scene representation
that can later be rendered from a novel viewpoint. We learn such a
representation from a casually captured _single_ video from everyday devices
such as smartphones, without the assistance of multi-camera rigs or other
dedicated hardware (which are typically not accessible to casual users).

Free-viewpoint video rendering typically requires a complicated hardware setup
consisting of multiple cameras to capture the scene of interest from different
viewpoints [82, 11, 14, 6]. The multi-camera setup in existing methods is
required because conventional 3D reconstruction algorithms (multi-view
stereopsis) assume a fully _static_ scene and thus can perform reconstruction
using the multiple captured viewpoints of a dynamic scene at any time. Most
methods represent the geometric reconstructions as some form of _per-frame_
representation (e.g., depth maps [82] or meshes[6]). Rendering from a novel
viewpoint can then be achieved, e.g., by warping available views using their
depth maps to the new viewpoint.

Following the success of single-image depth estimation, recent monocular video
depth estimation methods allow for the acquisition of consistent per-frame
depth estimates from only a single video [42, 78]. While still at an early
stage, this line of work opens up new possibilities, where monocular scene
depth estimates can be directly used for view synthesis. However, naïve
approaches such as per-frame depth-based warping would lead to unnatural
stretches and reveal holes in disoccluded regions (even with perfect depth
estimates). One can alleviate this by post-processing the incomplete rendering
[78]. However, such per-frame processing methods often lead to temporal
flickers. The core problem lies in the use of a _frame-wise representation_
(e.g., depth maps associated with the input images), and therefore suffer from
issues ranging from temporal inconsistency to high redundancy and thus
excessive storage requirements and data transfer bandwidth.

In this work, we build on recent monocular video depth estimation methods and
aggregate the entire spatiotemporal aspects of a dynamic scene in a single
global representation. While fusing multiple depth maps into a single, global
representation has a long tradition, most work on volumetric depth integration
has focused on static scenes [12, 50] or geometry alone without textures [49].
These methods typically use _discrete_ representations such as voxel grids,
meshes, or point clouds. Consequently, these methods often suffer from
premature hard decisions on geometry estimation and limited resolution due to
high storage requirements.

In this paper, we, instead, turn to the recent advances in neural implicit
representations, which allow for _continuous_ representations of a scene
without resolution loss. Recent work has shown that these representations
achieve high-quality view interpolation of complex _static_ scenes while
retaining their advantages over discrete representations [48, 79, 38]. The
current approaches to learn them, however, require either multiple posed
images of a _fully static_ scene [48, 79, 38] or ground truth 3D
representations [62, 63]. While videos often contain appearances of a scene
seen from multiple viewpoints, they only contain _exactly one_ viewpoint at
_any given time_. Combined with the dynamic nature of video, this renders it
nontrivial to extend current approaches to learn spatiotemporal
representations from a single video.

Specifically, we learn neural irradiance fields as a function of both space
and time for each video. We do not model view dependency, hence we use the
term irradiance. Using supervision from only color frames of the input video
as in [48] is futile, since the variations between frames can be explained
with _either_ a change of appearance _or_ geometry, or a _combination_ of
both. We resolve this ambiguity using the per-frame scene depth estimated from
monocular video depth estimation. Our depth supervision constrains the scene’s
geometry at any moment and disambiguates it from appearance variations. While
this enables us to encode physically correct appearance and geometry in a
global representation, it fails to fill the holes that could be seen at other
time steps in the video. We address this by encouraging the color and volume
density to propagate across time whenever spatial locations are not supervised
otherwise. The resulting representation allows us to render the video from
novel viewpoints and time: our implicit model can be queried at any
spatiotemporal location and rendered using standard volume rendering†††Note
that our method can render arbitrary viewpoints at all the _observed_ time
steps. We do not extrapolate or interpolate the time steps..

Our technical contributions include the following:

  * •

We aggregate frame-wise 2.5D representations into a globally consistent
spatiotemporal representation from a single monocular video.

  * •

We address the inherent motion–appearance ambiguity using video depth
supervision and constrain the disoccluded contents by propagating the color
and volume density across time.

  * •

We demonstrate a compelling free-viewpoint video rendering experience on
various casual videos shot from smartphones, preserving motion and texture
details while conveying a vivid sense of 3D.

##  2 Related Work

View synthesis for images. Creating novel views from multiple images is a
long-standing problem in computer vision and computer graphics. Existing
image-based rendering techniques first extract approximated geometric proxy
and create novel target views by warping and blending the corresponding
contents from multiple source frames [30, 59, 23, 61]. Multi-plane images
(MPIs) have been popular in the past few years as a geometric representation
[80, 47, 17, 35, 24], allowing for compelling novel view synthesis quality.
Many recent works further push the requirement of the number of input images
to a narrow-baseline stereo pair [80, 69, 10] or a single image [81, 73, 54,
74, 32, 65]. Recently, neural implicit representation has shown high-quality
view synthesis results by implicitly modeling the volume density and color of
the scene using the weights of a multi-layer perceptron [48, 79, 38]. Our work
uses NeRF [48] as our base scene representation for view synthesis. Unlike
NeRF that only models _static_ scenes, our focus is on creating new views from
arbitrary viewpoint and time for _dynamic_ scenes.

View synthesis for videos. Compared to images, view synthesis for video poses
significant challenges due to the need to handle _time-varying_ scene geometry
and appearances. Consequently, most of the existing methods typical require
laborious multi-camera setup [82, 11, 56, 13, 6], special hardware [1], or
synchronous video captures from multiple viewpoints [3, 4]. Several methods
can reduce the required number of input views by focusing on specific domains
such as performance capture [8, 14, 22] and video re-animation [31, 38, 39,
66, 9]. In contrast, our work aims to enable view synthesis of a _complex
dynamic scene_ at any given viewpoints and time from a _single_ video. Very
recently, Yoon et al. [78] also explore the same problem setup. In contrast to
[78] that processes each novel view _independently_ (by warping blending
multiple images), we can render an interpolation video with temporally smooth
transitions across viewpoints. Our method does not assume a simple two-layer
(foreground-background) model of the scene and can handle more generic scenes.

Neural implicit representation. Implicit representation has emerged as a
powerful tool to overcome conventional limitations of _discrete_ 3D
representations such as voxel grids or meshes. The core idea is to use a
multilayer perceptron (MLPs) to implicitly model the occupancy [45, 46],
signed distance functions [57, 2], object appearance [55], volumetric density
[16, 40, 67, 71, 68, 48] in the 3D space. Differentiable rendering techniques
enable training these models without accessing ground truth data for direct 3D
supervision [43, 52, 48, 37]. However, extending the above methods to handle
scene dynamics is not trivial due to the motion–appearance ambiguity.
Occupancy flow [51] achieves 4D reconstruction (3D shape + 1D time) by
learning continuous motion fields. Our work builds upon the recent advances in
neural implicit representation but focuses on representing a _dynamic video_.
Compared to 4D reconstruction in [51], our method differs in the following two
aspects. First, our method does not require direct 3D ground truth training
data. Second, in addition to model the time-varying 3D geometry, we also model
the appearance of complex scenes.

Video depth estimation. Estimating dense depth from a dynamic video is a
challenging task. Existing multi-view stereo (MVS) methods (either geometric-
based [64, 18] or learning-based [26, 76, 21, 41]) assume static scene and
thus not suitable for dynamic videos as it often produces erroneous depth for
moving objects or untextured regions. Several monocular video depth estimation
methods predict depth using the cost volume computed from nearby frames [70,
36]. Similar to MVS algorithms, these video-to-depth methods have difficulties
in handling dynamic scenes well. Very recently, hybrid methods that combine
MVS and single-image depth estimation models have been proposed [42, 78, 33].
Our work leverages the estimated depth from [42] to help resolve the ambiguity
when learning the spatiotemporal neural radiance fields using a single video.
Our approach renders photorealistic views with correctly filled dis-occluded
contents compared to view synthesis with per-frame depth-based warping.

Video completion. State-of-the-art video completion methods achieve temporally
consistent completion by propagating known contents to missing regions along
flow trajectories [27, 25, 75, 19]. One may first render a free-viewpoint
video according to each frame’s estimated depth, followed by filling the
missing pixels (disoccluded regions due to view changes) using video
completion algorithms. Our method also produces _completed_ novel views (i.e.,
with no missing pixels from disocclusion). However, unlike video completion
algorithms that inpaint the dis-occluded pixels in the _screen space_ , our
approach fills in the dis-occluded content implicitly in the _3D space_. Our
experiments validate that our approach produces significantly fewer artifacts
than the baseline method using video completion.

Depth map fusion. A line of research work focuses on fusing a sequence of RGB-
Depth images in a video into a _global_ with voxel-, point-based, signed
distance field-based representation [84]. Examples include 3D reconstruction
for static scenes [53, 29] or dynamic objects from a single [49, 28] or RGB-D
cameras [14, 5, 28, 49, 77, 83]. Our work differs from prior 3D reconstruction
methods in two aspects. First, we do not assume a fixed, canonical 3D model as
in existing dynamic 3D reconstruction methods and, therefore, can naturally
handle an entire dynamic scene (as opposed to only individual objects).
Second, our approach with neural implicit representations jointly models time-
varying geometry _and_ appearance.

Concurrent works. Several concurrent works on extending NeRF [48] to dynamic
scenes from monocular video have been proposed [58, 60, 34, 72, 15, 20]. These
methods either learn a static canonical radiance field with deformation [58,
60, 72] or a dynamic radiance field directly conditioned on time [34, 15, 20].
Our work belongs to the latter; while the other two works [34, 15] regularize
the training primarily with flow information, ours does so with dynamic scene
depth. We refer the readers to these papers for a complete picture.

##  3 Background

Our representation builds on the neural radiance field, or NeRF [48], which we
recap in this section. NeRF represents the radiance $\mathbf{c}=(r,g,b)$ and
differential volume density $\sigma$ at a 3D location $\mathbf{x}=(x,y,z)$ of
a scene observed from a viewing direction $\mathbf{d}=(\theta,\phi)$ as a
continuous multi-variate function using a multi-layer perceptron (MLP):
$F_{\mathrm{NeRF}}:(\mathbf{x},\mathbf{d})\rightarrow(\mathbf{c},\sigma)$.

The color of a pixel can be rendered by integrating the radiance modulated by
the volume density along the camera ray
$\mathbf{r}(s)=\mathbf{o}+s\mathbf{d}$, shot from the camera center through
the center of the pixel:

$C(\mathbf{r})=\int_{s}{}_{\mathrm{n}}^{s}{}_{\mathrm{f}}T(s)\sigma\Big{(}\mathbf{r}(s)\Big{)}\mathbf{c}\Big{(}\mathbf{r}(s),\,\mathbf{d}\Big{)}\mathrm{d}s,$
(1)

where

$T(s)=\exp\\!\left(-\\!\int_{s}{}_{\mathrm{n}}^{s}\sigma(\mathbf{r}(p))\mathrm{d}p\right)$
(2)

is the accumulated transmittance along the ray $\mathbf{r}$ up to $s$.

One can train the MLP using multiple posed images, capturing a _static_ scene
from different viewpoints. Specifically, we minimize the photometric loss that
compares the rendering through a ray $\mathbf{r}$ with the corresponding
ground truth color from an input image:

$\bm{\altmathcal{L}}_{\mathrm{NeRF}}=\sum_{\mathbf{r}\in\altmathcal{R}}\left\|\hat{C}(\mathbf{r})-C(\mathbf{r})\right\|_{2}^{2},$
(3)

where $\altmathcal{R}$ denotes a set of rays, and $C(\mathbf{r})$ and
$\hat{C}(\mathbf{r})$ the ground truth and the estimated color, respectively.

In the implementation, the continuous volume rendering of (1) is approximated
by numerical quadrature, i.e., computing the color using a finite number of
sampled 3D points along a ray and calculate the summation of the radiances,
weighted by the discrete transmittance. As this weighted summation process is
differentiable, the gradient can propagate backward for optimizing the MLP. We
perform the sampling in two steps. First, a ray is sampled uniformly in $s$,
and then, it is sampled with respect to the approximate transmittance so that
more samples are used around surfaces in the scene. The two groups of samples
are evaluated in separate _coarse_ and _fine_ networks, and both are used to
measure the loss (3).

##  4 Space-time Neural Irradiance Fields

We represent a 4D space-time irradiance field as a function that maps a
spatiotemporal location $(\mathbf{x},t)$ to the emitted color and volume
density, $F:(\mathbf{x},t)\rightarrow(\mathbf{c},\sigma)$. Our input video is
represented as a stream of RGB-D images,
$I_{t}:\mathbf{u}\rightarrow(\mathbf{c},d)$ at discrete time steps
$t\in\altmathcal{T}=\\{1,2,...,N_{\mathrm{f}}\\}$, where $\mathbf{u}=(u,v)$ is
2D pixel coordinates, and their associated camera calibrations
$\altmathcal{P}_{t}$.

A ray $\mathbf{r}$ at time $t$ can be determined by a pixel location
$\mathbf{u}$ and the camera calibration $\altmathcal{P}_{t}$: it marches from
the camera center through the center of pixel denoted by $\mathbf{u}$.
Additionally, we parameterize a ray such that the parameter $s$ denotes the
scene depth. This is achieved by setting the directional vector $\mathbf{d}$
such that its projection onto the principal ray has a unit norm in the camera
space.

Color reconstruction loss. To learn the implicit function $F$ from the input
video $I$, first and foremost, we constrain our representation $F$ such that
it reproduces the original video $I$ when rendered from the original viewpoint
for each frame. Specifically, we penalize the difference between the volume-
rendered image at each time $t$ and the corresponding input image $I_{t}$.
This amounts to the reconstruction loss of the original NeRF [48]:

$\bm{\altmathcal{L}}_{\mathrm{color}}=\sum_{(\mathbf{r}\\!,\,t)\in\altmathcal{R}}\left\|\hat{C}(\mathbf{r},t)-C(\mathbf{r},t)\right\|^{2}_{2},$
(4)

where $\altmathcal{R}$ is a batch of rays, each associated with a time $t$.

Unlike NeRF, for dynamic scenes, we have to reconstruct the _time-varying_
scene geometry at _every_ time $t$. However, a single video contains only one
observation of the scene at any point in time, rendering the estimation of
scene geometry severely under-constrained. That is, the 3D geometry of a scene
can be legitimately represented in numerous (infinitely possible) ways since
varying geometry can be explained with the varying appearance and vice versa.
For example, any input video can be reconstructed with a “a flat TV” solution
(with a planar geometry with each frame texture-mapped).

Thus, the color reconstruction loss provides the ground for accurate
reconstruction _only_ when the learned representation is rendered from the
same camera trajectory of the input, lacking any machinery that drives
learning correct geometry. Incorrect geometry would lead to artifacts as soon
as we start deviating from the original video’s camera trajectory, as shown in
Figure 2a.

(a) w/o depth loss

(b) w/ depth loss

Figure 2:  Depth loss. (a) A model trained _without_ our depth loss can
reconstruct the image from the original viewpoint well (_first_). However,
with a slight viewpoint change, the synthesized image (_second_) suffers from
strong visual artifacts due to incorrect geometry (_third_). (b) A trained
_with_ the proposed depth loss render the novel viewpoint without clearly
visible artifacts.

Depth reconstruction loss. We resolve this motion–appearance ambiguity by
constraining the time-varying geometry of our dynamic scene representation
using the per-frame scene depth of the input video (estimated from video depth
estimation methods). We estimate the scene depth from the learned volume
density of the scene, and measure its difference from the input depth $d_{t}$.
It is a non-trivial question on how to define the scene depth of a ray. One
possibility is to measure the distance where the accumulated transmittance $T$
becomes less than a certain threshold. Such an approach, however, involves
heuristics and hard decisions. Instead, we accumulate depth values along the
ray modulated both with the transmittance and volume density, similarly to the
depth composition in layered scene representations [73].

Our depth reconstruction loss is of the form:

$\bm{\altmathcal{L}}_{\mathrm{depth}}=\sum_{(\mathbf{r}\\!,\,t)\in\altmathcal{R}}\left\|\frac{1}{\hat{D}(\mathbf{r},t)}-\frac{1}{D(\mathbf{r},t)}\right\|^{2}_{2},$
(5)

where

$\hat{D}(\mathbf{r},t)=\int_{s}{}_{\mathrm{n}}^{s}{}_{\mathrm{f}}T(s,t)\,\sigma(\mathbf{r}(s),t)\,s\,\mathrm{d}s,$
(6)

is the integrated sample depth values along the ray and $D(\mathbf{r})$.

Empty-space loss. Constraining the depth predicted by our model using the
estimated scene depth is not sufficient to capture accurate scene geometry.
This is because the predicted depth is, in essence, a _weighted sum_ of depth
values along the ray. Consequently, we sometimes see haze-like visual
artifacts when rendering at novel views. We propose to encourage _empty space_
between the camera and the first visible scene surface to address this issue.
A similar idea has been used in volumetric depth integration [12]. To this
end, we penalize non-zero volume densities measures along each ray up to the
point no closer than a small margin
$\epsilon=0.05\cdot(s_{\mathrm{f}}-s_{\mathrm{n}})$ to the scene depth for
each ray:

$\bm{\altmathcal{L}}_{\mathrm{empty}}=\sum_{(\mathbf{r}\\!,\,t)\in\altmathcal{R}}\int_{s_{\mathrm{n}}}^{d_{t}(\mathbf{u})-\epsilon}\sigma(\mathbf{r}(s),t)\,\mathrm{d}s,$
(7)

where $\mathbf{u}$ denotes the pixel coordinates where $\mathbf{r}$ intersects
with the image plane at $t$, $d_{t}(\mathbf{u})$ denote the scene depth for
the pixel $\mathbf{u}$ at time $t$.

The empty-space loss combined with the depth reconstruction loss provides
geometric constraints for our representation up to and around _visible_ scene
surfaces at each frame. The learned representations can thus produce
geometrically correct novel view synthesis, as shown in Figure 2b.

Static scene loss. A large portion of spaces that is _hidden_ from the input
frame’s viewpoint at any given time is still _not constrained_ , i.e., the MLP
has not seen the 3D positions and time as input queries during training. As a
result, when these unconstrained spaces are disoccluded due to viewpoint
changes, they are prone to artifacts (see Figure 4 for an example). However,
there is a high chance that a portion of disoccluded spaces is observed from a
_different viewpoint_ at _another time_. Our idea is to constrain the MLP by
propagating these partially observed contents across time. However, instead of
explicitly correlating surfaces over time, e.g, using the scene flow, we
choose to constrain the spaces surrounding the surface regions. This allows us
to avoid misalignment of scene surfaces due to unreliable geometry estimates
or other image aberrations commonly seen in captured videos such as exposure
or color variations.

We make a simple assumption on _unobserved_ spaces: every part of the world
should stay static unless observed not as such. Enforcing this assumption
prevents the part of spaces that are not observed from going entirely
unconstrained. Our static scene constraint encourages the shared color and
volume density at the same spatial location $\mathbf{x}$ between two distinct
times $t$ and $t^{\prime}$:

$\bm{\altmathcal{L}}_{\mathrm{static}}=\sum_{(\mathbf{x},\,t)\in\altmathcal{X}}\left\|F(\mathbf{x},t)-F(\mathbf{x},t^{\prime})\right\|^{2}_{2},$
(8)

where both $(\mathbf{x},t)$ and $(\mathbf{x},t^{\prime})$ are _not_ close to
any visible surfaces, and $\altmathcal{X}$ denotes a set of sampling locations
where the loss is measured.

Figure 3:  Scene sampling. We measure the _Empty-space loss_ on 3D positions
until hitting the estimated scene depth (green). We use all the samples along
the ray (green and yellow) to compute the _Depth_ and _Color reconstruction
loss_. For _Static loss_ , we sample the scene from the union of the space
spanned by all camera rays (within the range of
$s\in[z_{\mathrm{n}},z_{\mathrm{f}}]$) of all input frames. We exclude any
samples that are close to any surface than $\epsilon$.

Scene sampling. While we have locations for the color, depth, and free-space
supervisions explicitly dictated by quadrature used by volume rendering [48],
we are free to choose _where_ we apply the static constraints. A
straightforward approach would be to use the same sampling locations that are
used for other losses. We can then randomly draw another time $t^{\prime}$
that is distinct from the current time $t$ and enforce the MLP to produce
similar appearances and volume densities at these two spatiotemporal
locations.

However, this still leaves a large part of the scene unconstrained when the
camera motion is large. Uniformly sampling in the scene bounding volume would
also not be ideal since sampling would be highly inefficient because of
perspective projection (except for special cases like a camera circling some
bounded volume).

As a simple solution to meet both the sampling efficiency and the sample
coverage, we propose to take the _union_ of all sampling locations along _all_
rays of _all_ frames to form our sample pool $\altmathcal{X}$. We exclude all
points that are closer to any observed surfaces than a threshold $\epsilon$
(see Figure 3). We randomly draw a fixed number of sampling locations from
this pool at each training iteration and add small random jitters to each
sampling location. At time $t^{\prime}$ the static scene loss is measured
against is also randomly chosen for each sample location $\mathbf{x}$, while
ensuring the resulting location $(\mathbf{x},t^{\prime})$ is not close to any
scene surfaces.

Figure 4: Static scene loss. Our static scene loss addresses the regions that
are _not_ constrained by the color and depth reconstruction loss, which
handles only _visible_ surfaces in the input frames (a). We render two frames
at $t_{1}=9$ and $t_{2}=90$ from the viewpoint of $t=40$. The camera has
panned in the top row and zoomed out in the bottom row. When these frames are
rendered from this novel viewpoint, previously hidden regions become
_disoccluded_. They are, without the static scene loss, completely
unconstrained and prone to ghosting or haze artifacts, as shown in (b). Our
static loss alleviate these artifacts.

Total loss. Our total loss for training the space-time irradiance fields is a
linear combination of all losses presented above:

$\bm{\altmathcal{L}}=\bm{\altmathcal{L}}_{\mathrm{color}}+\alpha\bm{\altmathcal{L}}_{\mathrm{depth}}+\beta\bm{\altmathcal{L}}_{\mathrm{empty}}+\gamma\bm{\altmathcal{L}}_{\mathrm{static}}.$
(9)

We validate the effectiveness of these losses in Section 5.

(a) Input

(b) Mesh

(c) Inpainted

(d) NeRF-T

(e) Ours

Figure 5:  Comparisons on novel view synthesis. We compare our results with
three baselines. (a) shows an input image and (b–d) show view synthesis
results. (b) shows rendered mesh representation; (c) inpainted version of (b);
(d) NeRF with an additional temporal parameter; (e) our method. The green
areas represent disocclusion. Notice how the baselines are unable to recover
the realistic appearance that respects an accurate geometry of the background
scene. Results are best appreciated in the supplementary videos.

Implementation details. We used the same MLP architecture as in NeRF [48],
except that we use 1024 activations for the first 8 layers instead of 256. We
calculate all the losses except the static scene loss on a batch of
$N_{\mathrm{r}}=1024$ rays that are randomly drawn from an input frame $I_{t}$
without replacement. We normalize the time $t$ such that
$\altmathcal{T}=[-1,1]$ and apply the positional encoding with 4 frequency
bands. We use the hierarchical volume sampling as in the original NeRF and
simultaneously train both the coarse and fine networks. We train the MLP for
800k iterations. It takes about 48 hours to train a network with about 100
video frames at the 960$\times$540 resolution with 4 NVIDIA V100 GPUs. Please
refer to the supplementary document for more details.

Table 1:  Ablation study. We trained our model on the synthetic “Sintel”
dataset with varying combinations of our losses and measure the view synthesis
quality using three metrics: (1) “PSNR (All)” the PSNR on all pixels, (2)
“PSNR (Occ.)” the PSNR on disoccluded pixels only, and (3) “SSIM” the
structural similarity. With all metrics, the higher the better. Bold faces
mean the best score, and the underlined mean the second best.

(a) Model |  (a) Time |  (a) $\bm{\altmathcal{L}}_{\mathrm{depth}}$ (5) |  (a) $\bm{\altmathcal{L}}_{\mathrm{empty}}$ (7) |  (a) $\bm{\altmathcal{L}}_{\mathrm{static}}$ (8) |  (a) $\bm{\altmathcal{L}}_{\mathrm{flow}}$ (a) |  (a) View dir. | “Bandage 1” | “Bandage 2” | “Sleeping 1” | “Sleeping 2” | “Alley 1” | “Bamboo 1”
---|---|---|---|---|---|---|---|---|---|---|---|---
PSNR (All) |  PSNR (Occ.) |  SSIM (All) |  PSNR (All) |  PSNR (Occ.) |  SSIM (All) |  PSNR (All) |  PSNR (Occ.) |  SSIM (All) |  PSNR (All) |  PSNR (Occ.) |  SSIM (All) |  PSNR (All) |  PSNR (Occ.) |  SSIM (All) |  PSNR (All) |  PSNR (Occ.) |  SSIM (All)
NeRF | - | - | - | - | - | - | 11.65 | 11.32 | 0.638 | 13.41 | 12.19 | 0.499 | 14.67 | 16.03 | 0.641 | 20.49 | 20.32 | 0.818 | 10.93 | 10.93 | 0.641 | 17.28 | 17.52 | 0.792
NeRF-T | ✓ | - | - | - | - | - | 11.81 | 11.27 | 0.658 | 13.84 | 12.26 | 0.462 | 15.45 | 15.81 | 0.663 | 13.83 | 22.08 | 0.480 | 11.50 | 12.50 | 0.665 | 15.59 | 15.32 | 0.732
1 | ✓ | ✓ | - | - | - | - | 14.82 | 12.87 | 0.796 | 18.61 | 18.81 | 0.807 | 21.85 | 21.43 | 0.830 | 31.01 | 32.93 | 0.965 | 24.67 | 25.28 | 0.930 | 28.35 | 26.34 | 0.966
2 | ✓ | ✓ | ✓ | - | - | - | 14.62 | 13.06 | 0.791 | 22.26 | 23.64 | 0.877 | 21.70 | 21.19 | 0.841 | 35.06 | 32.61 | 0.965 | 26.75 | 26.40 | 0.941 | 25.69 | 23.83 | 0.965
3 | ✓ | ✓ | - | ✓ | - | - | 15.48 | 13.15 | 0.812 | 20.41 | 22.11 | 0.860 | 21.85 | 21.59 | 0.841 | 35.79 | 34.79 | 0.982 | 25.34 | 21.42 | 0.937 | 27.20 | 23.59 | 0.957
4 (“Ours”) | ✓ | ✓ | ✓ | ✓ | - | - | 16.34 | 15.54 | 0.844 | 20.25 | 21.99 | 0.855 | 22.53 | 21.65 | 0.844 | 35.01 | 33.91 | 0.981 | 26.56 | 26.75 | 0.931 | 27.87 | 25.28 | 0.964
5 | ✓ | ✓ | - | ✓ | ✓ | - | 17.24 | 16.84 | 0.826 | 18.67 | 16.75 | 0.820 | 21.96 | 21.19 | 0.837 | 28.43 | 27.21 | 0.957 | 25.37 | 25.24 | 0.940 | 25.45 | 22.97 | 0.924
6 | ✓ | ✓ | ✓ | - | ✓ | - | 17.24 | 14.50 | 0.832 | 17.07 | 16.59 | 0.772 | 22.37 | 21.57 | 0.841 | 28.60 | 26.20 | 0.957 | 21.65 | 22.60 | 0.920 | 21.30 | 18.62 | 0.895
7 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 18.14 | 14.02 | 0.855 | 18.77 | 18.32 | 0.810 | 18.36 | 18.95 | 0.808 | 28.26 | 26.71 | 0.944 | 20.05 | 24.05 | 0.899 | 20.70 | 18.08 | 0.885

##  5 Experimental Results

We first compare our method with baseline approaches using the videos of
dynamic scenes. Note that we always render new views at one of the observed
times. That is, we do not evaluate our method’s capability of temporal
interpolation/extrapolation since our method is not designed to address it. We
then provide extensive quantitative ablation studies with the variants of our
model where each loss is added one at a time. We urge the readers to watch our
supplementary videos in _our project webpage_ (https://video-nerf.github.io),
where we provide free-viewpoint rendering of our learned representations.

Datasets. We use the videos of dynamic scenes from the recent consistent video
estimation method of Xuan et al. [42] along with the camera calibration and
the per-frame depth maps provided together. ‡‡‡We also intended to use the
dataset and video depth of Yoon et al. [78], but were unable to obtain the
depth maps used in their results. Their dataset (denoted by “CVD”) consists of
short videos of moving subjects captured by a smartphone.

For quantitative evaluation, we use the synthetic stereo videos from the MPI
Sintel dataset [7]. We select 6 videos that show a variety of characteristics
in terms of scene motion, camera motion, and the size of moving subjects. We
use the left video to train our models and render them from the right video’s
viewpoints for ground truth comparisons.

Baselines. We compare our method against several baseline methods: (“Mesh”)
textured mesh representations directly reconstructed from the input depth maps
as demonstrated by Xuan et al. [42]; (“Inpainted”) its inpainted version,
where disoccluded empty pixels in the 2D rendered images are inpainted using a
recent video inpainting method [19]; and (“NeRF-T”) a version of NeRF with an
extra time parameter, which is our model trained with only the color
reconstruction loss (4). Note that we do _not_ use the viewing directions for
the NeRF baseline.

Qualitative comparisons. Figure 5 presents the comparisons of our model
against the baselines using the “CVD” dataset, where we show view synthesis
results from novel viewpoints. We used our full method with all losses
presented in Section 4 to create these results. Please refer to _our project
webpage_ for the full video results.

Ablation studies. We trained our model with the different combinations of
losses and measured the view synthesis quality in three metrics, PSNR on the
entire image, PSNR on the disoccluded regions only, and SSIM on the entire
image. Table 1 summarizes the results. In addition to the four losses
presented in Section 4, we test an addition loss,
$\bm{\altmathcal{L}}_{\mathrm{flow}}$, which measures the consistency of the
color and the volume density between two corresponding spatiotemporal
locations via _scene flow_. We obtain the scene flow from the 2D optical flow
raised to 3D using the per-frame scene depth. Our motivation is to further
encourage temporal smoothness and disocclusion handling. Since we are
effectively gather view-dependent appearances with the scene flow loss, we
tested both models with and without the viewing directions used.

For quantitative evaluation, we train our models using the Sintel dataset. The
left videos are used to train the models rendered from the right videos’
viewpoints and then compared to the ground truth right videos. All metrics
aggregate the scores over all frames. Our model always works better when
trained with the depth loss. While varying depending on scene types, the
static loss and empty space loss help improve the results as well. However, we
find that the scene flow loss does not help improve quality. We suspect that
this is because casual videos often include strong image-space aberration such
as exposure or color changes and monocular depth estimates does not provide as
accurate scene depth as stereo-based methods do. For example, this could be
addressed by a latent code factoring out such variations, similarly done in
NeRF-W [44]. While our “Model-2” works slightly better than our full model
(“Model-4”) quantitatively, we have found that our full model works usually
the best for real data.

##  6 Conclusions

We have presented a simple yet effective algorithm for learning space-time
irradiance fields from single casually captured videos. Our core technical
contributions are (1) leveraging monocular video depth estimation to constrain
the time-varying geometry of our learned neural implicit functions and (2)
designing a static scene loss and a sampling strategy to propagate scene
contents across time. We extensively validate and justify our design choices
both visually and quantitatively on the Sintel dataset. We showcase free-
viewpoint video rendering of several challenging dynamic scenes captured with
hand-held cellphone cameras.

#### Acknowledgments.

We thank Ayush Saraf for his help with distributed training. All photos of
individuals in this paper are used with permission.

## References

  * [1] Benjamin Attal, Selena Ling, Aaron Gokaslan, Christian Richardt, and James Tompkin.  Matryodshka: Real-time 6dof video view synthesis using multi-sphere images.  In ECCV, 2020.
  * [2] Matan Atzmon and Yaron Lipman.  SAL: Sign agnostic learning of shapes from raw data.  In CVPR, 2020.
  * [3] Luca Ballan, Gabriel J Brostow, Jens Puwein, and Marc Pollefeys.  Unstructured video-based rendering: Interactive exploration of casually captured videos.  ACM TOG (Proc. SIGGRAPH), 2010.
  * [4] Aayush Bansal, Minh Vo, Yaser Sheikh, Deva Ramanan, and Srinivasa Narasimhan.  4d visualization of dynamic events from unconstrained multi-view videos.  In CVPR, 2020.
  * [5] Aljaz Bozic, Michael Zollhofer, Christian Theobalt, and Matthias Nießner.  Deepdeform: Learning non-rigid rgb-d reconstruction with semi-supervised data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7002–7012, 2020.
  * [6] Michael Broxton, John Flynn, Ryan Overbeck, Daniel Erickson, Peter Hedman, Matthew Duvall, Jason Dourgarian, Jay Busch, Matt Whalen, and Paul Debevec.  Immersive light field video with a layered mesh representation.  ACM TOG (Proc. SIGGRAPH), 39(4):86–1, 2020.
  * [7] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black.  A naturalistic open source movie for optical flow evaluation.  In ECCV, 2012.
  * [8] Joel Carranza, Christian Theobalt, Marcus A Magnor, and Hans-Peter Seidel.  Free-viewpoint video of human actors.  ACM TOG (Proc. SIGGRAPH), 22(3):569–577, 2003.
  * [9] Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros.  Everybody dance now.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5933–5942, 2019.
  * [10] Inchang Choi, Orazio Gallo, Alejandro Troccoli, Min H Kim, and Jan Kautz.  Extreme view synthesis.  In ICCV, 2019.
  * [11] Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan.  High-quality streamable free-viewpoint video.  ACM TOG (Proc. SIGGRAPH), 34(4):1–13, 2015.
  * [12] Brian Curless and Marc Levoy.  A volumetric method for building complex models from range images.  In SIGGRAPH, 1996.
  * [13] Łukasz Dąbała, Matthias Ziegler, Piotr Didyk, Frederik Zilly, Joachim Keinert, Karol Myszkowski, H-P Seidel, Przemyslaw Rokita, and Tobias Ritschel.  Efficient multi-image correspondences for on-line light field video processing.  In Computer Graphics Forum, 2016.
  * [14] Mingsong Dou, Sameh Khamis, Yury Degtyarev, Philip Davidson, Sean Ryan Fanello, Adarsh Kowdle, Sergio Orts Escolano, Christoph Rhemann, David Kim, Jonathan Taylor, et al.  Fusion4d: Real-time performance capture of challenging scenes.  ACM Transactions on Graphics (TOG), 35(4):1–13, 2016.
  * [15] Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu.  Neural radiance flow for 4d view synthesis and video processing.  arXiv e-prints, pages arXiv–2012, 2020.
  * [16] SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al.  Neural scene representation and rendering.  Science, 360(6394):1204–1210, 2018.
  * [17] John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker.  DeepView: View synthesis with learned gradient descent.  In CVPR, 2019.
  * [18] Yasutaka Furukawa and Carlos Hernández.  Multi-view stereo: A tutorial.  Foundations and Trends® in Computer Graphics and Vision, 9(1-2):1–148, 2015.
  * [19] Chen Gao, Ayush Saraf, Jia-Bin Huang, and Johannes Kopf.  Flow-edge guided video completion.  In ECCV, 2020.
  * [20] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang.  Dynamic view synthesis from dynamic monocular video.  arXiv preprint arXiv:2105.06468, 2021.
  * [21] Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan.  Cascade cost volume for high-resolution multi-view stereo and stereo matching.  In CVPR, 2020.
  * [22] Marc Habermann, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, and Christian Theobalt.  LiveCap: Real-time human performance capture from monocular video.  ACM TOG (Proc. SIGGRAPH), 38(2):1–17, 2019.
  * [23] Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Brostow.  Deep blending for free-viewpoint image-based rendering.  ACM TOG (Proc. SIGGRAPH), 37(6):1–15, 2018.
  * [24] Hsin-Ping Huang, Hung-Yu Tseng, Hsin-Ying Lee, and Jia-Bin Huang.  Semantic view synthesis.  In ECCV, 2020.
  * [25] Jia-Bin Huang, Sing Bing Kang, Narendra Ahuja, and Johannes Kopf.  Temporally coherent completion of dynamic video.  ACM TOG (Proc. SIGGRAPH Asia), 35(6):1–11, 2016.
  * [26] Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Jia-Bin Huang.  DeepMVS: Learning multi-view stereopsis.  In CVPR, 2018.
  * [27] Shachar Ilan and Ariel Shamir.  A survey on data-driven video completion.  Computer Graphics Forum, 34(6):60–85, 2015.
  * [28] Matthias Innmann, Michael Zollhöfer, Matthias Nießner, Christian Theobalt, and Marc Stamminger.  VolumeDeform: Real-time volumetric non-rigid reconstruction.  In ECCV, pages 362–379, 2016.
  * [29] Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al.  KinectFusion: real-time 3d reconstruction and interaction using a moving depth camera.  In Proceedings of the 24th annual ACM symposium on User interface software and technology, 2011.
  * [30] Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi.  Learning-based view synthesis for light field cameras.  ACM TOG (Proc. SIGGRAPH), 35(6):1–10, 2016.
  * [31] Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, and Christian Theobalt.  Deep video portraits.  ACM Transactions on Graphics (TOG), 37(4):1–14, 2018.
  * [32] Johannes Kopf, Kevin Matzen, Suhib Alsisan, Ocean Quigley, Francis Ge, Yangming Chong, Josh Patterson, Jan-Michael Frahm, Shu Wu, Matthew Yu, et al.  One shot 3d photography.  ACM TOG (Proc. SIGGRAPH), 39(4):76–1, 2020.
  * [33] Johannes Kopf, Xuejian Rong, and Jia-Bin Huang.  Robust consistent video depth estimation.  In CVPR, 2021.
  * [34] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang.  Neural scene flow fields for space-time view synthesis of dynamic scenes.  arXiv preprint arXiv:2011.13084, 2020.
  * [35] Zhengqi Li, Wenqi Xian, Abe Davis, and Noah Snavely.  Crowdsampling the plenoptic function.  In European Conference on Computer Vision, pages 178–196. Springer, 2020.
  * [36] Chao Liu, Jinwei Gu, Kihwan Kim, Srinivasa G Narasimhan, and Jan Kautz.  Neural rgb (r) d sensing: Depth and uncertainty from a video camera.  In CVPR, 2019.
  * [37] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt.  Neural sparse voxel fields.  In NeurIPS, 2020.
  * [38] Lingjie Liu, Weipeng Xu, Marc Habermann, Michael Zollhoefer, Florian Bernard, Hyeongwoo Kim, Wenping Wang, and Christian Theobalt.  Neural human video rendering by learning dynamic textures and rendering-to-video translation.  IEEE Transactions on Visualization and Computer Graphics, 2020.
  * [39] Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Hyeongwoo Kim, Florian Bernard, Marc Habermann, Wenping Wang, and Christian Theobalt.  Neural animation and reenactment of human actor videos.  arXiv preprint arXiv:1809.03658, 2018.
  * [40] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh.  Neural volumes: Learning dynamic renderable volumes from images.  arXiv preprint arXiv:1906.07751, 2019.
  * [41] Xiaoxiao Long, Lingjie Liu, Christian Theobalt, and Wenping Wang.  Occlusion-aware depth estimation with adaptive normal constraints.  In European Conference on Computer Vision, pages 640–657. Springer, 2020.
  * [42] Xuan Luo, Jia-Bin Huang, Richard Szeliski, Kevin Matzen, and Johannes Kopf.  Consistent video depth estimation.  ACM TOG (Proc. SIGGRAPH), 39(4), 2020.
  * [43] R Mantiuk and V Sundstedt.  State of the art on neural rendering.  STAR, 39(2), 2020.
  * [44] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth.  NeRF in the wild: Neural radiance fields for unconstrained photo collections.  In arXiv, 2020.
  * [45] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In CVPR, 2019.
  * [46] Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson.  Implicit surface representations as layers in neural networks.  In ICCV, 2019.
  * [47] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  ACM TOG (Proc. SIGGRAPH), 38(4):1–14, 2019.
  * [48] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  NeRF: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [49] Richard A Newcombe, Dieter Fox, and Steven M Seitz.  DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time.  In CVPR, 2015.
  * [50] Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohli, Jamie Shotton, Steve Hodges, and Andrew W. Fitzgibbon.  KinectFusion: Real-time dense surface mapping and tracking.  In ISMAR, 2011.
  * [51] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.  Occupancy flow: 4d reconstruction by learning particle dynamics.  In ICCV, 2019.
  * [52] M. Niemeyer, Lars M. Mescheder, Michael Oechsle, and A. Geiger.  Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision.  CVPR, 2020.
  * [53] Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger.  Real-time 3d reconstruction at scale using voxel hashing.  ACM TOG (Proc. SIGGRAPH), 32(6):1–11, 2013.
  * [54] Simon Niklaus, Long Mai, Jimei Yang, and Feng Liu.  3d ken burns effect from a single image.  ACM TOG (Proc. SIGGRAPH Asia), 38(6):1–15, 2019.
  * [55] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger.  Texture fields: Learning texture representations in function space.  In ICCV, 2019.
  * [56] Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L Davidson, Sameh Khamis, Mingsong Dou, et al.  Holoportation: Virtual 3d teleportation in real-time.  In Proceedings Annual Symposium on User Interface Software and Technology, 2016.
  * [57] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In CVPR, 2019.
  * [58] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo-Martin Brualla.  Deformable neural radiance fields.  arXiv preprint arXiv:2011.12948, 2020.
  * [59] Eric Penner and Li Zhang.  Soft 3d reconstruction for view synthesis.  ACM Transactions on Graphics (TOG), 36(6):1–11, 2017.
  * [60] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  arXiv preprint arXiv:2011.13961, 2020.
  * [61] Gernot Riegler and Vladlen Koltun.  Free view synthesis.  In ECCV, 2020.
  * [62] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li.  PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In ICCV, 2019.
  * [63] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo.  PIFuHD: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization.  In CVPR, 2020.
  * [64] Johannes L Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In CVPR, 2016.
  * [65] Meng-Li Shih, Shih-Yang Su, Johannes Kopf, and Jia-Bin Huang.  3d photography using context-aware layered depth inpainting.  In CVPR, 2020.
  * [66] Aliaksandra Shysheya, Egor Zakharov, Kara-Ali Aliev, Renat Bashirov, Egor Burkov, Karim Iskakov, Aleksei Ivakhnenko, Yury Malkov, Igor Pasechnik, Dmitry Ulyanov, et al.  Textured neural avatars.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2387–2397, 2019.
  * [67] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer.  Deepvoxels: Learning persistent 3d feature embeddings.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2437–2446, 2019.
  * [68] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  In NeurIPS, 2019.
  * [69] Pratul P Srinivasan, Richard Tucker, Jonathan T Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely.  Pushing the boundaries of view extrapolation with multiplane images.  In CVPR, 2019.
  * [70] Zachary Teed and Jia Deng.  DeepV2D: Video to depth with differentiable structure from motion.  In ICLR, 2020.
  * [71] Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Deferred neural rendering: Image synthesis using neural textures.  ACM Transactions on Graphics (TOG), 38(4):1–12, 2019.
  * [72] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt.  Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a deforming scene from monocular video.  arXiv preprint arXiv:2012.12247, 2020.
  * [73] Richard Tucker and Noah Snavely.  Single-view view synthesis with multiplane images.  In CVPR, 2020.
  * [74] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson.  Synsin: End-to-end view synthesis from a single image.  In CVPR, 2020.
  * [75] Rui Xu, Xiaoxiao Li, Bolei Zhou, and Chen Change Loy.  Deep flow-guided video inpainting.  In CVPR, 2019.
  * [76] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan.  MVSNet: Depth inference for unstructured multi-view stereo.  In ECCV, 2018.
  * [77] Mao Ye and Ruigang Yang.  Real-time simultaneous pose and shape estimation for articulated objects using a single depth camera.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2345–2352, 2014.
  * [78] Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz.  Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera.  In CVPR, 2020.
  * [79] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.  Nerf++: Analyzing and improving neural radiance fields.  arXiv preprint arXiv:2010.07492, 2020.
  * [80] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely.  Stereo magnification: Learning view synthesis using multiplane images.  ACM TOG (Proc. SIGGRAPH), 2018.
  * [81] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros.  View synthesis by appearance flow.  In ECCV, 2016.
  * [82] C Lawrence Zitnick, Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski.  High-quality video view interpolation using a layered representation.  ACM TOG (Proc. SIGGRAPH), 23(3):600–608, 2004.
  * [83] Michael Zollhöfer, Matthias Nießner, Shahram Izadi, Christoph Rehmann, Christopher Zach, Matthew Fisher, Chenglei Wu, Andrew Fitzgibbon, Charles Loop, Christian Theobalt, et al.  Real-time non-rigid reconstruction using an rgb-d camera.  ACM Transactions on Graphics (ToG), 33(4):1–12, 2014.
  * [84] Michael Zollhöfer, Patrick Stotko, Andreas Görlitz, Christian Theobalt, Matthias Nießner, Reinhard Klein, and Andreas Kolb.  State of the art on 3d reconstruction with rgb-d cameras.  Computer graphics forum, 37(2):625–652, 2018.

##  Appendix A Additional Details

In this appendix, we provide implementation details (in Section A.1), training
details (in Section A.2) and additional quantitative comparison in Section
A.3. We include additional qualitative results in our project website
https://video-nerf.github.io. In particular, we test on 10 different videos
and provide comparison with baselines and different loss configurations.

###  A.1 Implementation Details

We implement our framework using PyTorch. In all our experiments, we
empirically set the hyper-parameters as $\alpha=1$, $\beta=100$, and
$\gamma=10$.

We calculate all the losses except the static scene loss on a batch of
$N_{\mathrm{r}}=1024$ rays that are randomly drawn from an input frame $I_{t}$
without replacement. We randomly choose $N_{\mathrm{s}}=1024$ from
$\altmathcal{X}$ at each step (also without replacement) for the static scene
loss. We normalize the time $t$ such that $\altmathcal{T}=[-1,1]$ and apply
the positional encoding with 4 frequency bands. Following NeRF [48], we apply
positional encoding to spatial positions $\mathbf{x}$. While we do not use the
normalized device coordinates, we sample each ray uniformly in _inverse_
depth. We set the depth range $z_{\mathrm{n}}$ and $z_{\mathrm{f}}$ as the
global minimum and maximum of all frames’ depth values.

###  A.2 Training Details

We used the same MLP architecture as in [48], except that we use 1024
activations for the first 8 MLP layers instead of 256. Our models are trained
with various combinations of the losses presented in the main paper but
otherwise with the same hyperparameters. We used the Adam optimizer with
momentum parameters $\beta_{1}=0.9$ and $\beta_{2}=0.999$ and a learning rate
of 0.0005. We train the MLP for 800k iterations. It takes about 48 hours to
train a network with about 100 video frames at the 960$\times$540 resolution
with 4 NVIDIA V100 GPUs.

###  A.3 Additional Comparisons to Baseline

Table 2: Reported PSNR on disoccluded pixels only.

Methods | Bandage1 | Bandage2 | Sleeping1 | Sleeping2 | Alley1 | Bamboo1
---|---|---|---|---|---|---
Inpainted | 16.52 | 13.32 | 20.08 | 17.52 | 14.82 | 12.23
Ours | 15.54 | 21.99 | 21.65 | 33.91 | 26.75 | 25.28

In this section, we provide a quantitative comparison to the inpainted mesh
method. We rendered the ground-truth mesh provided by the Sintel dataset in 2D
and inpaint the missing pixels in disoccluded areas using state-of-the-arts
video completion algorithms. Then, we evaluate how well our method handles
disoccluded areas, compared with the inpainted mesh method, for which we used
the same Sintel GT depth. Table 2 shows the quantitative evaluation measured
in PSNR metric. It demonstrates that inpainting is not sufficient to get good
disocclusion contents, and validates that our approach produces significantly
fewer artifacts than the baseline method using video completion.