# Neural Articulated Radiance Field

Atsuhiro Noguchi${}^{\text{1}}\;$ Xiao Sun${}^{\text{2}}\;$ Stephen
Lin${}^{\text{2}}\;$ Tatsuya Harada${}^{\text{1,3}}$

${}^{\text{1}}$The University of Tokyo $\;{}^{\text{2}}$Microsoft Research
Asia $\;{}^{\text{3}}$RIKEN

###### Abstract

We present Neural Articulated Radiance Field (NARF), a novel deformable 3D
representation for articulated objects learned from images. While recent
advances in 3D implicit representation have made it possible to learn models
of complex objects, learning pose-controllable representations of articulated
objects remains a challenge, as current methods require 3D shape supervision
and are unable to render appearance. In formulating an implicit representation
of 3D articulated objects, our method considers only the rigid transformation
of the most relevant object part in solving for the radiance field at each 3D
location. In this way, the proposed method represents pose-dependent changes
without significantly increasing the computational complexity. NARF is fully
differentiable and can be trained from images with pose annotations. Moreover,
through the use of an autoencoder, it can learn appearance variations over
multiple instances of an object class. Experiments show that the proposed
method is efficient and can generalize well to novel poses. The code is
available for research purposes at https://github.com/nogu-atsu/NARF.

Figure 1: Training pipeline of Disentangled NARF ($\text{NARF}_{D}$). NARF is
an efficient pose-aware 3D representation trained from only pose-annotated
images. The learned representation is part-based and able to render novel
poses of articulated 3D objects by changing the input object pose
configurations.

##  1 Introduction

In this work, we aim to learn a representation for rendering novel views and
poses of 3D articulated objects, such as human bodies, from images. Our
approach follows the inverse graphics paradigm [29, 28, 39, 30] of analyzing
an image by attempting to synthesize it with compact graphics codes. These
codes are typically disentangled to allow for rendering of scenes/objects with
fine-grained control over individual appearance properties such as object
location, pose, lighting, texture, and shape. For the case of humans,
synthesis of novel views and poses can be useful for applications such as
movie making, photo editing, virtual clothing [68, 31] and motion transfer
[35, 4].

Various inverse graphics based approaches have been specifically designed for
static scenes [64, 21, 57], rigid objects [9, 72, 66, 59, 3], blend shapes for
keypoints [65, 4] and dense meshes [68, 35]. However, efficient deformation
modeling of articulated 3D objects using neural networks remains a challenging
task due to the large variance of joint locations (especially for endpoints
such as hands), severe self-occlusions, and high non-linearity in forward
kinematic transformations [73]. Though work has been done to enable explicit
control over the underlying human pose [35] and key point locations [4], their
neural rendering methods are either limited to 2D [4], which prevents modeling
of view-dependent appearance [7], or based on mesh representations [35], where
rendering quality can be affected by the resolution of the discrete template
mesh.

Recent progress on the implicit representation of 3D objects and scenes, such
as signed distance functions [20, 45] and occupancy fields [12, 40], has
greatly promoted the development of the inverse graphics paradigm. Such
representations are lightweight in model size, continuous, and differentiable,
making them highly practical in comparison with the previously-dominant
volumetric representations [11, 16, 24, 41, 49, 58, 60, 71]. Particularly,
Mildenhall et al. [42] propose the neural radiance field (NeRF) that takes a
single continuous 5D coordinate (3D spatial location and 2D viewing direction)
as input and outputs the volume density and view-dependent emitted radiance at
each spatial location. Combined with a classical differentiable volume
rendering technique [23], it is able to synthesize novel views by learning
from a sparse set of input views of static scenes. NeRF completely discards
the mesh-based representation and replaces it with a radiance-based model
which can effectively and efficiently encode view-dependent appearance,
enabling it to reproduce scenes of complex geometry with high fidelity.

In this paper, we extend NeRF to an articulated NeRF, called a Neural
Articulated Radiance Field (NARF), to represent articulated 3D objects.
Accounting for 3D articulation within the NeRF framework is a challenging
problem because a complex, non-linear relationship exists between a kinematic
representation of 3D articulations and the resulting radiance field, making it
hard to model implicitly in a neural network [73]. In addition, the radiance
field at a given 3D location is influenced by at most a single articulated
part and its parents along the kinematic tree, while the full kinematic model
is provided as input. As a result, dependencies of the output to irrelevant
parts may inadvertently be learned, which is known to hurt model
generalization to poses unseen in training [69].

To address these issues, we propose a method that predicts the radiance field
at a 3D location based on only the most relevant articulated part. This part
is identified using a set of sub-networks that output the probability for each
part given the 3D location and the 3D geometric configuration of the parts.
The spatial configurations of parts are computed explicitly with a kinematic
model, rather than modeled implicitly in the network. A NARF then predicts the
density and view-dependent radiance of the 3D location conditioned on the
properties of only the selected part. An overview of the method is shown in
Fig. 1.

The presented NARF has the following properties:

  * •

It learns a disentangled representation of camera viewpoint, bone parameters,
and bone pose, allowing these properties to be individually controlled in
rendering.

  * •

A dense 3D representation is learned from a sparse set of 2D images with pose
annotations of the articulated object, which could potentially be obtained
through external pose estimation techniques on multi-view images with known
camera parameters [22].

  * •

Part segmentation is learned from images with pose annotation. Additional
supervision is not needed.

  * •

NARF can be trained for articulated objects of various shape and appearance,
through the use of an autoencoder that extracts latent shape and appearance
vectors which are additionally disentangled.

With this approach, it becomes possible to render both novel views and poses
of articulated 3D objects from pose annotated 2D images with little increase
in computational complexity.

##  2 Related Work

#### Articulated 3D Shape Representations

The deformation of articulated objects is traditionally modeled by skinning
techniques [18, 19, 32, 33] in which the location of surface mesh vertices is
determined from bone transformations controlled by the kinematics [2].
Effective skinning models with subtle pose-dependent and identity-dependent
deformation modeling have been developed for the human body [36, 17, 44] and
animals [74]. However, the representation capacity of skinning based models is
limited to the resolution of the discrete template mesh, and sophisticated
shading techniques [51] are usually required for high quality image rendering.
In addition, a large amount of 3D scan data and expert supervision are
required to prepare a template mesh.

Recently, Deng et al. [8] proposed a neural network based articulated shape
representation (NASA). NASA learns the neural indicator/occupancy function [6,
40, 46] of every point in space, conditioned on a latent pose vector that
encodes a piece-wise decomposition. NASA provides a continuous and
differentiable representation for 3D articulated shapes. However, ground truth
occupancy is required to train the network, and NASA does not learn
appearance, a critical element for rendering.

#### Articulated Pose Conditioned Image Generation

The recent advance of image generation models such as variational autoencoders
(VAE) [27] and generative adversarial networks (GANs) [15] provides powerful
tools for generating realistic-looking images. Image generation for
articulated objects (typically persons) conditioned on target poses is an
important direction with various applications like movie making, photo
editing, virtual clothing [68, 31] and motion transfer [35, 4]. A majority of
these works [4, 37, 38, 55, 62, 10, 1] generate the image of a person in a
target pose by learning a GAN model from 2D keypoint maps of the target pose.
The appearance information of this person is provided by explicit
concatenation with an image of this person in the target pose [37, 55],
automatically encoded for a single person [4] or using auto-encoders [38].
These works are limited to 2D, which prevents modeling of view-dependent
appearance [7]. Some works [68] leverage the underlying 3D mesh representation
and transfer appearance from one mesh to another using aligned mesh triangles.
The quality of a mesh based representation is bounded by the resolution of its
discrete template mesh, and a 3D template mesh is required.

#### Implicit 3D representation

Our work builds on the recent success of the implicit 3D representation. This
representation is memory efficient, continuous, and topology-free, and has
been used for learning 3d shape [40, 46, 56], 3d texture [43], static scenes
[42, 57], parts decomposition [14, 13], articulated objects [8], deformation
[50, 34, 67, 47], 3d reconstruction from sparse images [52, 53], and image
synthesis [54, 5].

Early methods required ground truth 3D geometry [40, 46], but in combination
with differentiable rendering, they evolved to learn from 2D images. In
particular, neural radiance fields (NeRF) [42] is capable of learning a 3D
representation of complex scenes using only multi-view posed images. However,
NeRF addresses static scenes and cannot handle deformable objects. Very
recently, methods have been proposed to extend NeRF to learn deformations and
dynamics [50, 34, 67, 47]. These models have been successful in learning
deformable implicit representations using posed video frames [50, 34, 67, 47].
However, these models do not take into account the structure of the object, so
they cannot generate images with explicit pose control.

##  3 Method

In this section, we present neural articulated radiance field, a novel
implicit representation for articulated 3D objects based on NeRF. We start by
briefly reviewing the basic NeRF formulation for static scenes in Sec. 3.1. In
Sec. 3.2, NeRF is extended to be conditioned on pose via a kinematic model,
and a straightforward baseline is derived from this. We reformulate the pose-
conditioned NeRF to allow for rigid object transformations as well as global
shape variations in Sec. 3.3. In Sec. 3.4, we represent articulated 3D objects
as a composition of movable rigid object parts controlled by forward kinematic
rules. To achieve constant model complexity with respect to the number of
object parts, we propose an efficient Disentangled NARF architecture. The
training strategy is then presented in Sec. 3.5.

###  3.1 Neural Radiance Field Revisited

A neural network is used to represent a Radiance Field such that 3D location
${\bf x}=(x,y,z)$ and 2D viewing direction ${\bf d}$ is converted to density
$\sigma$ and RGB color value $c$. The density $\sigma$ acts like a
differential opacity controlling how much radiance is accumulated by a ray
passing through ${\bf x}$ [42].

$F_{\Theta}:(\gamma({\bf x}),\gamma({\bf d}))\rightarrow(c,\sigma),$ (1)

where $\gamma(p)=[(\text{sin}(2^{l}\pi p),\text{cos}(2^{l}\pi p)]_{0}^{L}$ is
a positional encoding (PE) layer that maps an input scalar into a higher
dimensional space to represent high-frequency detail of the scene.
$F_{\Theta}$ consists of two ReLU MLP networks. Specifically, the volume
density $\sigma$ is a function of the location $\bf x$ only, while the RGB
color $c$ is a function of both location ${\bf x}$ and viewing direction ${\bf
d}$.

$F_{\Theta_{\sigma}}:(\gamma({\bf x}))\rightarrow(\sigma,{\bf
h}),\,\,F_{\Theta_{c}}:({\bf h},\gamma({\bf d}))\rightarrow(c),$ (2)

where ${\bf h}$ is a hidden feature vector.

Classical volume rendering [23] is used to render the color ${\bf C}({\bf r})$
of a camera ray ${\bf r}(t)={\bf o}+t{\bf d}$ with near and far bounds $t_{n}$
and $t_{f}$, and where ${\bf o}$ denotes the camera position.

$\displaystyle T(t)$ $\displaystyle=\exp(-\int_{t_{n}}^{t}\sigma({\bf
r}(s))ds),$ (3) $\displaystyle{\bf C}({\bf r})$
$\displaystyle=\int_{t_{n}}^{t_{f}}T(t)\sigma({\bf r}(t))c({\bf r}(t),{\bf
d})dt$ (4)

$T(t)$ denotes the accumulated transmittance along the ray. The integrals are
computed by a discrete approximation over sampled points along the ray ${\bf
r}$.

$\displaystyle
T_{j}=\exp\left(-\sum_{k=1}^{j-1}\sigma_{k}\delta_{k}\right),{\bf C}({\bf
r})=\sum_{j=1}^{N}T_{j}(1-\exp(-\sigma_{j}\delta_{j}))c_{j}$ (5)

Here, $\sigma_{j}$ and $c_{j}$ are the density and color at the $j^{th}$ point
on the ray ${\bf r}$, and $\delta_{j}$ is the distance between the $j^{th}$
and $(j+1)^{th}$ sample points.

NeRF is trained on images of a single static scene taken from multiple views
with known camera parameters. The density and color of each location are
trained so that the rendered image for each of the views becomes close to its
ground truth. After training, high resolution images can be synthesized from
any viewpoint.

###  3.2 Pose-Conditioned NeRF: A Baseline

Our goal is to extend the representation capacity of NeRF from static scenes
to deformable articulated objects whose configurations can be described by a
kinematic model [2]. The radiance field of a 3D location is thus conditioned
on the pose configuration. Once this “pose-conditioned NeRF” is learned, novel
poses can be rendered in addition to novel views, by changing the input pose
configurations.

In this work, we focus on modeling articulated objects without considering the
background. Therefore, for compactness, we assume that the backgrounds are
pre-cleaned.

#### Kinematic Model

Formally, the kinematic model [2] represents an articulated object of $P+1$
joints, including endpoints, and $P$ bones in a tree structure where one of
the joints is selected as the root joint and each remaining joint is linked to
its single parent joint by a bone of fixed length.

Specifically, the root joint $J_{0}$ is defined by a global transformation
matrix ${\bf T}^{0}$. Let $\boldsymbol{\zeta}_{i}$ be the bone length from the
$i^{th}$ joint $J_{i}$ to its parent, $i\in\\{1,...P\\}$, and
$\boldsymbol{\theta}_{i}$ denotes the rotation angles of the joint with
respect to its parent joint. A bone, considered as a rigid object, defines a
local rigid transformation between a joint and its parent. The transformation
matrix ${\bf T}^{i}_{local}$ is computed as

${\bf
T}^{i}_{local}=\text{Rot}(\boldsymbol{\theta}_{i})\text{Trans}(\boldsymbol{\zeta}_{i}),$
(6)

where Rot and Trans are the rotation and translation matrix, respectively. The
global transformation from the root joint to joint $J_{i}$ can thus be
obtained by multiplying the transformation matrices along the bones from the
root joint to the $i^{th}$ joint:

${\bf T}^{i}=(\Pi_{k\in\text{Pa}(i)}{\bf T}^{k}_{local}){\bf T}^{0}$ (7)

where $\text{Pa}(i)$ includes the $i^{th}$ joint and all of its parent joints
along the kinematic tree. The corresponding global rigid transformation
$l^{i}=\\{R^{i},{\bf t}^{i}\\}$ for the $i^{th}$ joint can then be obtained
from the transformation matrix ${\bf T}^{i}$.

#### Baseline

The most straightforward way to condition the radiance field at a 3D location
${\bf x}$ on a kinematic pose configuration $\mathcal{P}=\\{{\bf
T}^{0},\boldsymbol{\zeta},\boldsymbol{\theta}\\}$ is to directly concatenate a
vector representing $\mathcal{P}$ as the model input. Since the forward
kinematic computation is a complex non-linear function [73] that is hard to
simulate in neural networks, we use the transformations $l^{i}=\\{R^{i},{\bf
t}^{i}\\}$ obtained by the forward kinematics as network inputs.

$\displaystyle F_{\Theta}^{\mathcal{P}}:(\gamma({\bf
x}),\gamma(\\{l^{i}|i=1,...,P\\}),\gamma({\bf d}))\rightarrow(\sigma,c)$ (8)

We refer to this naive approach as Pose-conditioned NeRF (P-NeRF). The
implementation details can be found in the supplemental material. Though
P-NeRF establishes dependency between the radiance field and pose,
generalization with this model is difficult because of the following two
reasons.

  * •

_Implicit Transformations._ An articulated object consists of several rigid
bodies, and the surface points on the object should move with the rigid
transformations of the parts when the pose changes. Therefore, the movement of
points can be explicitly described using rigid body transformations of each
part, but such transformations may be difficult for a neural network to learn
implicitly.

  * •

_Part Dependency._ The density at a 3D location depends only on the parameters
of the bone it lies on and its parents along the kinematic tree. However, all
the parameters are used to estimate the radiance field of a single location in
Eq. 8. As the training on such 3D locations is backpropagated to all the
parameters, the network may learn erroneous dependencies that do not
physically exist. Correct pose predictions may still be obtained for test
poses seen in the training data, but model generalization to novel poses may
be degraded [69].

Towards addressing the above issues, we decompose the articulated object into
$P$ rigid object parts. Each part has its own local coordinate system defined
by the rigid transformation $l^{i}=\\{R^{i},{\bf t}^{i}\\}$, which is
explicitly estimated using forward kinematics, rather than modeled implicitly
by a neural network. Then, we show how a rigidly transformed object part can
be effectively modeled in a rigidly transformed neural radiance field (RT-
NeRF) in Section 3.3. Based on RT-NeRF, we describe in Section 3.4 how to
train a single unified NeRF that encodes multiple parts in a manner that
avoids the part dependency issue.

###  3.3 Rigidly Transformed Neural Radiance Field

Given a rigid transformation $l=\\{R,{\bf t}\\}$ of an object, we now estimate
the radiance field in the object coordinate system where the density is
constant with respect to a _local_ 3D location. Formally,

$F_{\Theta_{\sigma}}^{l}:(\gamma({\bf x}^{l}))\rightarrow(\sigma,{\bf h})$ (9)

where ${\bf x}^{l}=R^{-1}({\bf x}-{\bf t})$ represents the 3D location in the
local object coordinate system.

We expect the model to handle certain shape variations. For example, the limb
length and thickness of a child should differ significantly from those of an
adult. To account for shape variation, we further condition the model on bone
parameter $\boldsymbol{\zeta}$.

$F_{\Theta_{\sigma}}^{l,\boldsymbol{\zeta}}:(\gamma({\bf
x}^{l}),\gamma(\boldsymbol{\zeta}))\rightarrow(\sigma,{\bf h})$ (10)

Meanwhile, the color $c$ at a local 3D location may change with a
transformation of the object coordinate system, as this may lead to changes in
the local lighting condition. Since the RGB color $c$ at a local 3D location
should further depend on rigid transformation $l$, we use a 6D vector
$\mathfrak{se}(3)$ representation $\boldsymbol{\xi}$ of transformation $l$ as
a network input.

$F_{\Theta_{c}}^{l,\boldsymbol{\zeta}}:({\bf h},\gamma({\bf
d}^{l}),\gamma(\boldsymbol{\xi}))\rightarrow(c)$ (11)

where ${\bf d}^{l}=R^{-1}{\bf d}$ is the 2D view direction in the object
coordinate system.

Combining Eqs. 10-11, the rigidly transformed neural radiance field (RT-NeRF)
defined in the $l=\\{R,{\bf t}\\}$ space is expressed as

$F_{\Theta}^{l,\boldsymbol{\zeta}}:(\gamma({\bf x}^{l}),\gamma({\bf
d}^{l}),\gamma(\boldsymbol{\xi}),\gamma(\boldsymbol{\zeta}))\rightarrow(c,\sigma)$
(12)

RT-NeRF serves as the basic building block in the neural articulated radiance
field, and we will next show how it is utilized to overcome the ’Implicit
Transformations’ and ’Part Dependency’ issues.

###  3.4 Neural Articulated Radiance Field

Figure 2: Three types of neural articulated radiance field. Inputs other than
position ${\bf x}$ are omitted for greater clarity.

The proposed neural articulated radiance field (NARF) is built upon RT-NeRF.
We first introduce two basic solutions, _Part-Wise NARF_ and _Holistic NARF_ ,
and analyze the pros and cons of each. Then, we propose our final solution
named _Disentangled NARF_ that shares the merits of both _Part-Wise_ and
_Holistic_ NARF. Conceptual figures are visualized in Fig. 2.

#### Part-Wise NARF ($\text{NARF}_{P}$)

Given a kinematic 3D pose configuration of an articulated object $\\{{\bf
T}^{0},\boldsymbol{\zeta},\theta\\}$, we first compute the global rigid
transformation $\\{l^{i}|i=1,...,P\\}$ for each rigid part using the forward
kinematics in Eqs. 6-7. To estimate the density and color $(\sigma,c)$ of a
global 3D location ${\bf x}$ from a 2D viewing direction ${\bf d}$, we train a
separate RT-NeRF, $F_{\Theta^{i}}^{l^{i},\boldsymbol{\zeta}}$,

$\displaystyle{\bf x}^{l^{i}}={R^{i}}^{-1}({\bf x}-{\bf t}^{i}),{\bf
d}^{l^{i}}={R^{i}}^{-1}{\bf d}$ (13) $\displaystyle
F_{\Theta^{i}}^{l^{i},\boldsymbol{\zeta}}:(\gamma(x^{l^{i}}),\gamma({\bf
d}^{l^{i}}),\gamma(\boldsymbol{\xi}^{i}),\gamma(\boldsymbol{\zeta}))\rightarrow(c^{i},\sigma^{i}),$
(14)

for each part using Eq. 12 and combine the densities and colors
$\\{\sigma^{i},c^{i}|i=1,...,P\\}$ estimated by different RT-NeRFs into one.
We denote this approach as _Part-Wise NARF_ ($\text{NARF}_{P}$).

Since a surface point of an object can belong to only one of the object parts,
only one of the estimates in $\\{\sigma^{i},c^{i}|i=1,...,P\\}$ should be
nonzero. The density and color $(\sigma,c)$ of a global 3D location ${\bf x}$
can be determined by taking the estimate with the highest density. However,
the max operation is not differentiable, so we instead use the softmax
function, which is a differentiable weighted sum over all the estimates:

$\displaystyle\sigma=\frac{\sum_{i=1}^{P}\exp(\sigma^{i}/\tau)\sigma^{i}}{\sum_{i=1}^{P}\exp(\sigma^{i}/\tau)},c=\frac{\sum_{i=1}^{P}\exp(\sigma^{i}/\tau)c^{i}}{\sum_{i=1}^{P}\exp(\sigma^{i}/\tau)},$
(15)

where $\tau$ is the temperature parameter of the softmax function. Volume
rendering is then applied using the combined density $\sigma$ and color $c$ to
generate the rendered color ${\bf C}({\bf r})$ by Eq. 5. Since the rendering
and softmax operations are both differentiable, the image reconstruction loss
can pass gradients to all the RT-NeRF models for effective training.

We note that, in addition to color ${\bf C}({\bf r})$, the foreground mask
${\bf M}({\bf r})$ can also be estimated as an integral of the opacity along
the camera ray:

${\bf M}({\bf r})=\sum_{j=1}^{N}T_{j}(1-\exp(-\sigma_{j}\delta_{j}))$ (16)

We can further render a segmentation image that indicates which RT-NeRF
(object part) is used for rendering each pixel:

$\displaystyle s_{j}({\bf
r})=\text{arg}\max_{i}\\{\sigma_{j}^{i}|i\in[1,P]\\}$ (17) $\displaystyle{\bf
S}^{i}({\bf r})=\sum_{j=1}^{N}T_{j}(1-\exp(-\sigma_{j}\delta_{j}))(s_{j}({\bf
r})==i)$ (18)

where $s_{j}$ is the index of the part with the greatest density, and ${\bf
S}^{i}$ denotes a segmentation mask for the $i^{th}$ part.

Discussion. The $\text{NARF}_{P}$ approach models the rigidly transformed
parts of an articulated object in separate RT-NeRFs, where each part has a
consistent radiance field under different 3D pose configurations. As the rigid
transformation of each RT-NeRF is computed explicitly via forward kinematics,
rather than implicitly within the network, the issue of implicit
transformations and part dependency are avoided. The part dependency issue is
also addressed by taking the estimate with the highest density while
suppressing the contribution of other parts for a global 3D location in Eq.
15. However, its computation is inefficient for the following reasons.

  * •

The computational cost is proportional to the number of object parts, limiting
the representation capacity for complex articulated objects.

  * •

Training is dominated by the large number of zero density point samples. As a
surface point on an object can belong to only one of the object parts, it will
be trained as a zero density sample for the remaining parts. Since parts with
small densities do not affect the value of the equation very much, it is not
really necessary to calculate the density of those parts.

#### Holistic NARF ($\text{NARF}_{H}$)

To address the above issues, we present another approach that combines the
inputs of the RT-NeRF models in $\text{NARF}_{P}$ then feeds them as a whole
into a single NeRF model for direct regression of the final density and color
$(\sigma,c)$. We call this approach _Holistic NARF_ ($\text{NARF}_{H}$).
Formally,

$\displaystyle
F_{\Theta_{\sigma}}^{l,\boldsymbol{\zeta}}:\text{Cat}(\\{\gamma({\bf
x}^{l^{i}})|i\in[1,P]\\},\gamma(\boldsymbol{\zeta}))\rightarrow(\sigma,{\bf
h}),$ (19) $\displaystyle
F_{\Theta_{c}}^{l,\boldsymbol{\zeta}}:\text{Cat}({\bf h},\\{(\gamma({\bf
d}^{l^{i}}),\gamma(\boldsymbol{\xi}^{i}))|i\in[1,P]\\})\rightarrow(c),$ (20)

where Cat denotes the concatenation operator.

Discussion. There is only a single NeRF model trained in $\text{NARF}_{H}$.
The computational cost is almost constant to the number of object parts and
the zero density problem is naturally avoided. However, unlike _Part-Wise
NARF_ , $\text{NARF}_{H}$ does not satisfy _Part Dependency_ , because all
parameters are considered for each 3D location. Moreover, object part
segmentation masks cannot be generated from Eq. 18 without part dependencies.

#### Disentangled NARF ($\text{NARF}_{D}$)

We propose _Disentangled NARF_ ($\text{NARF}_{D}$) which shares the merits of
both $\text{NARF}_{P}$ and $\text{NARF}_{H}$ while avoiding their weaknesses
by introducing a selector $\mathcal{S}$.

The selector $\mathcal{S}$ identifies which object part a global 3D location
${\bf x}$ belongs to. $\mathcal{S}$ consists of $P$ lightweight sub-networks
for each part. For the $i^{th}$ part, a sub-network $O_{\Gamma}^{i}$ takes the
local 3D position of ${\bf x}$ in $l^{i}=\\{R^{i},{\bf t}^{i}\\}$ and the bone
parameter $\boldsymbol{\zeta}$ as input and outputs the probability $p^{i}$ of
${\bf x}$ belonging to the $i^{th}$ part. Since ${\bf x}$ should be assigned
to only one of the object parts, the softmax activation is used to normalize
the selector’s outputs:

$\displaystyle O_{\Gamma}^{i}:(\gamma({\bf
x}^{l^{i}}),\gamma(\boldsymbol{\zeta}))\rightarrow(o^{i}),\,$ $\displaystyle
p^{i}=\frac{\exp(o^{i})}{\sum_{k=1}^{P}\exp(o^{k})}$ (21)

It can be seen that $O_{\Gamma}^{i}$ is actually an occupancy network [40, 8]
defined _in the local object coordinate system_. Comparing this with NASA [8],
NASA’s occupancy networks learn absolute occupancy values to estimate an
explicit surface, but our networks learn relative occupancy values to other
parts for part selection. For implementation, we use a two-layer MLP with ten
hidden nodes for each occupancy net, which is lightweight yet effective.

_Disentangled NARF_ is defined by (softly) masking out the irrelevant parts in
the concatenated input using the outputs $p^{i}$ of the selector.

$\displaystyle
F_{\Theta_{\sigma}}^{l,\boldsymbol{\zeta}}:\text{Cat}(\\{\gamma({\bf
x}^{l^{i}})*p^{i}|i\in[1,P]\\},\gamma(\boldsymbol{\zeta}))\rightarrow(\sigma,{\bf
h}),$ (22) $\displaystyle
F_{\Theta_{c}}^{l,\boldsymbol{\zeta}}:\text{Cat}({\bf h},\\{(\gamma({\bf
d}^{l^{i}})*p^{i},\gamma(\boldsymbol{\xi}^{i})*p^{i})|i\in[1,P]\\})\rightarrow(c)$
(23)

Note that though we have removed the dependency on irrelevant parts by masking
their inputs, the resulting input is still in the form of a concatenation.
This is done purposely because all the bones share a single NeRF, which needs
to distinguish the different bones in order to generate the corresponding
density and color. Different bones are distinguished by dimensions of the
concatenated vector that represent them, similar to part identity encoding.
The detailed network architecture can be found in Fig. 7 of the supplemental
material.

Since the selector outputs the probabilities of a global 3D location belonging
to each part, we can generate the segmentation mask by selecting the locations
occupied by a specific part followed by Eq. 18:

$\displaystyle s_{j}({\bf r})=\text{arg}\max_{i}\\{p_{j}^{i}|i\in[1,P]\\}$
(24)

###  3.5 Training Details

The positional encoding dimensions we set for the 3D location ${\bf x}$ and
other parameters are 10 and 4 respectively. During training, at each
optimization iteration, we randomly sample a batch of camera rays from the set
of all pixels, and then follow the hierarchical volume sampling strategy of
the original NeRF [42] to query $N$ samples for each ray. With known kinematic
3D pose configuration $\\{{\bf T}^{0},\boldsymbol{\zeta},\theta\\}$, the
samples’ densities and colors are estimated by the NARF model. Volume
rendering is then used to render the color ${\bf C}({\bf r})$ and mask ${\bf
M}({\bf r})$ of this ray using Eq. 5 and 16, respectively. The loss is the
total squared error between the rendered and true pixel colors and masks.

$\displaystyle\mathcal{L}=\sum_{{\bf r}\in\mathcal{R}}[||\hat{\bf C}({\bf
r})-{\bf C}({\bf r})||_{2}^{2}+||\hat{\bf M}({\bf r})-{\bf M}({\bf
r})||_{2}^{2}]$ (25)

where $\mathcal{R}$ is the set of rays in each batch, $\hat{\bf C}$ and
$\hat{\bf M}$ are the ground truth color and foreground mask. In the
supplemental material, we empirically show that the extra mask loss helps to
learn a cleaner background. Other training details on the learning rate, batch
size and optimizer can be found in the supplemental material.

##  4 Results of Training on a Single Object

In this section, we evaluate our model in the case of a single articulated 3D
object.

#### Dataset and Settings

We create our own synthetic dataset of human bodies for experimentation. It
consists of two persons, one male and one female, selected from the human 3D
textured mesh (THUman) dataset [70]. Each person has 56 and 48 different
poses, 26 of which are used for training and the others for testing. We render
100 images with various orientations and scaling of each mesh for training and
20 for testing, ending up with 2600 training images for each person. Note that
the viewpoint distribution of training and testing sets are the same under
this setting. We denote this test data setting as the novel pose/same view
setting. All rendered images have a resolution of 128 $\times$ 128\.
Additionally, we introduce three other test settings for a more comprehensive
comparison. The same pose/same view setting uses testing images rendered from
the same poses and same viewpoint distribution as in training. The novel
pose/novel view setting uses novel poses and a different viewpoint
distribution than in training. Finally, the same pose/novel view setting uses
the same poses but the viewpoint distribution is different from training. The
kinematic 3D pose configurations are inferred from the SMPL model parameters
provided by the THUman dataset and we use bone length as the bone parameter
$\zeta$ in Eq. 10.

| Cost | Same pose, same view | Novel pose, same view | Same pose, novel view | Novel pose, novel view
---|---|---|---|---|---
Method | #Params | #FLOPS | #Memory | Mask$\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | Mask $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | Mask $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | Mask $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$
CNN | 15.6M | - | - | 76.9 | 29.12 | 0.9429 | 134.8 | 27.30 | 0.9211 | 365.9 | 25.19 | 0.8532 | 392.2 | 24.53 | 0.8470
P-NeRF | 0.85M | 156M | 356K | 778.7 | 21.42 | 0.8006 | 1077.0 | 20.42 | 0.7696 | 844.9 | 21.19 | 0.7897 | 1110.1 | 20.27 | 0.7648
D-NARF | 0.66M | 121M | 382K | 2182.6 | 18.90 | 0.1143 | 2308.2 | 18.81 | 0.1140 | 2137.3 | 19.09 | 0.1144 | 2241.3 | 18.88 | 0.1133
$\text{NARF}_{P}$ | 11.8M | 2140M | 6544K | 92.0 | 28.56 | 0.9258 | 116.2 | 26.83 | 0.9052 | 101.5 | 27.54 | 0.9144 | 125.8 | 26.50 | 0.9104
$\text{NARF}_{H}$ | 1.06M | 197M | 344K | 55.6 | 29.91 | 0.9470 | 376.8 | 24.09 | 0.8665 | 70.5 | 28.81 | 0.9370 | 374.6 | 23.98 | 0.8646
$\text{NARF}_{D}$ | 1.10M | 205M | 382K | 50.5 | 30.86 | 0.9586 | 114.4 | 27.93 | 0.9317 | 64.1 | 29.44 | 0.9466 | 123.8 | 27.24 | 0.9230

Table 1: Quantitative comparison for a single object. Best results in bold.
Figure 3: Generative results comparison for a single object with a novel pose
and novel view. One model per person identity. Triangles point to areas that
should be noted. $\text{NARF}_{D}$ best generalizes to novel view/novel poses
among the six methods. Figure 4: Disentangled representations learned by
$\text{NARF}_{D}$. For bone parameter interpolation, the right leg length is
interpolated.

#### Metrics

Three metrics are used to evaluate performance. The peak signal to noise ratio
(PSNR) and structural similarity index (SSIM) [63] are two commonly used
evaluation metrics for image reconstruction (higher is better). In addition,
we introduce the L2 distance error of mask images (Mask), which better
describes how close the 3D shape of the rendered object is to the ground truth
(lower is better).

#### Baselines

In addition to the three variants of NARF ($\text{NARF}_{P}$,
$\text{NARF}_{H}$ and $\text{NARF}_{D}$), three other baselines are included
for comparison. The first is a 2D _CNN-based_ method similar to [4] that
generates the target subject image from “pose stick figures”. The pose stick
figures in our case are generated by projecting the 3D joints into the 2D
image (with given camera parameters) then adding lines to connect these 2D
keypoints. The second is the _P-NeRF_ method described in Sec. 3.2. The third
one is _D-N ARF_, a simple extension of D-NERF [47] to articulated objects.
D-NARF aims to learn the mapping $\Psi:{\bf x}\rightarrow{\bf x}^{\prime}$
that transforms a given point to its position in a canonical shape space. In
our implementation, a static NeRF model for a canonical pose $\mathcal{P}^{c}$
is learned, then a mapping network [47] $\Psi$ estimates the deformation field
between the scene of a specific pose instance $\mathcal{P}$ and the scene of
the canonical pose $\mathcal{P}^{c}$. The details of the three baselines can
be found in the supplemental material.

#### Results

Quantitative comparison results are given in Table 1. #Params, #FLOPS, and
#Memory denote the number of parameters, floating point operations per ray,
and number of elements to preserve during forward propagation per ray, which
is proportional to the memory cost.

It can be seen that our method, $\text{NARF}_{D}$, outperforms the others
under all the evaluation metrics and test data settings (best results shown in
bold). Particularly, it exhibits high performance under novel pose and/or
novel view settings (slight performance drop on SSIM within 4%) with low
computational cost (close to a single NeRF model, P-NeRF). Hence, we can
conclude that $\text{NARF}_{D}$ effectively and efficiently learns the
radiance field of an articulated 3D object and the model generalizes to novel
poses and views with high fidelity.

In contrast, all the other methods are deficient in one way or another. The
CNN-based method fails when tested under novel views (10% performance drop on
SSIM) since it is difficult to learn an effective 3D representation from 2D
inputs. P-NeRF and D-NARF fail in almost all cases mainly due to both the
Implicit Transformations and Part Dependency issues. $\text{NARF}_{P}$
exhibits good performance and generalization ability but requires much more
computation ($10\times$ #FLOPS and $17\times$ #Memory of $\text{NARF}_{D}$).
$\text{NARF}_{H}$ is less effective when tested on novel poses (8% performance
drop on SSIM) due to the Part Dependency issue.

Qualitative results under the novel pose/novel view setting are shown in Fig.
3. Rendered RGB images (first row), depth maps (second row), and part
segmentation maps (third row), as well as ground truth RGB images (bottom left
corner) are displayed. It can be seen that the NeRF based methods (except for
the CNN-based one) can obtain depth images and the “part dependent” methods
($\text{NARF}_{P}$ and $\text{NARF}_{D}$) can obtain segmentation maps. Our
final solution $\text{NARF}_{D}$ generates higher quality RGB, depth and
segmentation maps for novel views and poses than the others. Moreover, as
shown in Fig. 4, $\text{NARF}_{D}$ learns a disentangled representation of
camera viewpoint, bone parameters and pose, allowing these appearance
properties to be individually controlled in rendering.

##  5 Appearance Variation with Autoencoder

In this section, we train an autoencoder based on NARF to model shape and
appearance variation among multiple articulated objects. The autoencoder
consists of an encoder and decoder. First, a 2D CNN-based encoder is used to
generate a latent vector ${\bf z}$ from an input image. The obtained latent
vector together with the given camera viewpoint and human pose are fed into
our NARF based decoder to reconstruct the input image.

Following the implementation for the NeRF-based generator [54], we first
decompose ${\bf z}$ into a shape latent vector ${\bf z}_{s}$ and an appearance
latent vector ${\bf z}_{a}$. Then, ${\bf z}_{s}$ is concatenated to the
density-dependent inputs, namely, the positionally encoded location ${\bf x}$
and bone parameters $\boldsymbol{\zeta}$. Meanwhile, ${\bf z}_{a}$ is
concatenated to the color-dependent inputs, namely, the positionally encoded
view direction ${\bf d}$ and the local transformation $\boldsymbol{\xi}$.
Specifically, when combining the autoencoder with the $\text{NARF}_{D}$ model,
we have

$F_{\Theta_{\sigma}}^{l,\boldsymbol{\zeta},{\bf z}}:\text{Cat}(\\{\gamma({\bf
x}^{l^{i}})*p^{i}|i\in[1,P]\\},\gamma(\boldsymbol{\zeta}),{\bf
z}_{s})\rightarrow(\sigma,{\bf h}),$ (26)
$F_{\Theta_{c}}^{l,\boldsymbol{\zeta},{\bf z}}:\text{Cat}({\bf
h},\\{(\gamma({\bf
d}^{l^{i}})*p^{i},\gamma(\boldsymbol{\xi}^{i})*p^{i})|i\in[1,P]\\},{\bf
z}_{a})\rightarrow(c)$ (27)

The encoder and decoder are trained jointly using the same loss in Eq. 25. For
the experiment, we use the best performing $\text{NARF}_{D}$ by default.
Comparison results for other models can be found in the supplemental material.

#### Dataset

We create another synthetic human body dataset from THUman [70] for
experimentation. All of the males (112 in total) and poses (35 on average for
each person) in the THUman dataset are used. We render 10 images for each pose
with randomly sampled viewpoints to generate 35450 images for training and
3940 images for evaluation. All rendered images have a resolution of 128
$\times$ 128.

#### Results

Fig. 5 shows the reconstructed RGB images as well as the additional depth
images and segmentation maps generated from the input RGB images. A single
autoencoder is trained for all objects, indicating that appearance variation
is effectively modeled. Fig. 6 shows that the NARF-based autoencoder learns a
disentangled representation of camera viewpoint, bone parameters, human pose,
and color appearance, allowing these properties to be individually controlled
in rendering. For color appearance, it is controlled by replacing the
appearance latent vector ${\bf z}_{a}$ with that of another person. Additional
results can be found in the supplemental material.

Figure 5: Single-view reconstruction results from $\text{NARF}_{D}$ with an
autoencoder. All outputs are generated from a single model. Figure 6:
Disentangled representations learned with an autoencoder. For bone parameter
interpolation, the head position is interpolated.

##  6 Conclusion and Future work

In this paper, we propose a method for learning implicit representations for
articulated objects. We show that it is possible to learn explicitly
controllable representations of viewpoint, pose, bone parameters, and
appearance from 3D pose annotated images. Although pose annotation is required
to train the model, the model is differentiable and thus may be extended to
reduce the required supervisory information, for example, by simultaneously
training 3D pose estimation and segmentation with the model. In addition,
since the proposed representation provides explicit 3D shape and part
segmentation, it may be applied to unsupervised depth estimation and
segmentation learning.

##  7 Acknowledgement

This work was supported by D-CORE Grant from Microsoft Research Asia and
partially supported by JST AIP Acceleration Research Grant Number JPMJCR20U3,
and JSPS KAKENHI Grant Number JP19H01115. We would like to thank Sho Maeoki,
Thomas Westfechtel, and Yang Li for the helpful discussions. We are also
grateful for the GPU resources provided by Microsoft Azure Machine Learning.

## References

  * [1] Guha Balakrishnan, Amy Zhao, Adrian V Dalca, Fredo Durand, and John Guttag.  Synthesizing images of humans in unseen poses.  In CVPR, 2018.
  * [2] Oene Bottema and Bernard Roth.  Theoretical kinematics, volume 24.  1990\.
  * [3] Arunkumar Byravan and Dieter Fox.  Se3-nets: Learning rigid body motion using deep neural networks.  In ICRA, 2017.
  * [4] Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros.  Everybody dance now.  In ICCV, 2019.
  * [5] Eric Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In CVPR, 2021.
  * [6] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In CVPR, 2019.
  * [7] Paul E Debevec, Camillo J Taylor, and Jitendra Malik.  Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach.  In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, 1996.
  * [8] Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, and Andrea Tagliasacchi.  Nasa: Neural articulated shape approximation.  In ECCV, 2020.
  * [9] Alexey Dosovitskiy, Jost Tobias Springenberg, and Thomas Brox.  Learning to generate chairs with convolutional neural networks.  In CVPR, 2015.
  * [10] Patrick Esser, Ekaterina Sutter, and Björn Ommer.  A variational u-net for conditional appearance and shape generation.  In CVPR, 2018.
  * [11] John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker.  Deepview: View synthesis with learned gradient descent.  In CVPR, 2019.
  * [12] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser.  Local deep implicit functions for 3d shape.  In CVPR, 2020.
  * [13] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser.  Local deep implicit functions for 3d shape.  In CVPR, 2020.
  * [14] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser.  Learning shape templates with structured implicit functions.  In CVPR, 2019.
  * [15] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In NIPS, 2014.
  * [16] Phlipp Henzler, Volker Rasche, Timo Ropinski, and Tobias Ritschel.  Single-image tomography: 3d volumes from 2d cranial x-rays.  In Computer Graphics Forum, volume 37, pages 377–388, 2018.
  * [17] Nikolas Hesse, Sergi Pujades, Michael J Black, Michael Arens, Ulrich G Hofmann, and A Sebastian Schroeder.  Learning and tracking the 3d body shape of freely moving infants from rgb-d sequences.  TPAMI, 42(10):2540–2551, 2019.
  * [18] Alec Jacobson, Zhigang Deng, Ladislav Kavan, and JP Lewis.  Skinning: Real-time shape deformation.  In ACM SIGGRAPH 2014 Courses, 2014.
  * [19] Doug L James and Christopher D Twigg.  Skinning mesh animations.  ACM Transactions on Graphics (TOG), 24(3):399–407, 2005.
  * [20] Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser, et al.  Local implicit grid representations for 3d scenes.  In CVPR, 2020.
  * [21] Justin Johnson, Agrim Gupta, and Li Fei-Fei.  Image generation from scene graphs.  In CVPR, 2018.
  * [22] Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timothy Scott Godisart, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh.  Panoptic studio: A massively multiview system for social interaction capture.  TPAMI, 2017.
  * [23] James T Kajiya and Brian P Von Herzen.  Ray tracing volume densities.  ACM SIGGRAPH computer graphics, 18(3):165–174, 1984.
  * [24] Abhishek Kar, Christian Häne, and Jitendra Malik.  Learning a multi-view stereo machine.  In NIPS, 2017.
  * [25] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of GANs for improved quality, stability, and variation.  In ICLR, 2018.
  * [26] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [27] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  In ICLR, 2014.
  * [28] Tejas D Kulkarni, Pushmeet Kohli, Joshua B Tenenbaum, and Vikash Mansinghka.  Picture: A probabilistic programming language for scene perception.  In CVPR, 2015.
  * [29] Tejas D Kulkarni, Vikash K Mansinghka, Pushmeet Kohli, and Joshua B Tenenbaum.  Inverse graphics with probabilistic cad models.  arXiv preprint arXiv:1407.1339, 2014.
  * [30] Tejas D Kulkarni, Will Whitney, Pushmeet Kohli, and Joshua B Tenenbaum.  Deep convolutional inverse graphics network.  In NIPS, 2015.
  * [31] Christoph Lassner, Gerard Pons-Moll, and Peter V Gehler.  A generative model of people in clothing.  In ICCV, 2017.
  * [32] Binh Huy Le and Zhigang Deng.  Smooth skinning decomposition with rigid bones.  ACM Transactions on Graphics (TOG), 31(6):1–10, 2012.
  * [33] John P Lewis, Matt Cordner, and Nickson Fong.  Pose space deformation: a unified approach to shape interpolation and skeleton-driven deformation.  In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, 2000.
  * [34] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang.  Neural scene flow fields for space-time view synthesis of dynamic scenes.  In CVPR, 2021.
  * [35] Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Hyeongwoo Kim, Florian Bernard, Marc Habermann, Wenping Wang, and Christian Theobalt.  Neural rendering and reenactment of human actor videos.  ACM Transactions on Graphics (TOG), 38(5):1–14, 2019.
  * [36] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  Smpl: A skinned multi-person linear model.  ACM transactions on graphics (TOG), 34(6):1–16, 2015.
  * [37] Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool.  Pose guided person image generation.  In NIPS, 2017.
  * [38] Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool, Bernt Schiele, and Mario Fritz.  Disentangled person image generation.  In CVPR, 2018.
  * [39] Vikash K Mansinghka, Tejas D Kulkarni, Yura N Perov, and Joshua B Tenenbaum.  Approximate bayesian image interpretation using generative probabilistic graphics programs.  In NIPS, 2013.
  * [40] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In CVPR, 2019.
  * [41] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  ACM Transactions on Graphics (TOG), 38(4):1–14, 2019.
  * [42] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [43] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger.  Texture fields: Learning texture representations in function space.  In ICCV, 2019.
  * [44] Ahmed A A Osman, Timo Bolkart, and Michael J. Black.  STAR: A sparse trained articulated human body regressor.  In ECCV, 2020.
  * [45] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In CVPR, 2019.
  * [46] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In CVPR, 2019.
  * [47] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo-Martin Brualla.  Nerfies: Deformable neural radiance fields.  In ICCV, 2021.
  * [48] Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou.  Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans.  In CVPR, 2021.
  * [49] Eric Penner and Li Zhang.  Soft 3d reconstruction for view synthesis.  ACM Transactions on Graphics (TOG), 36(6):1–11, 2017.
  * [50] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In CVPR, 2021.
  * [51] Randi J Rost, Bill Licea-Kane, Dan Ginsburg, John Kessenich, Barthold Lichtenbelt, Hugh Malan, and Mike Weiblen.  OpenGL shading language.  2009\.
  * [52] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In ICCV, 2019.
  * [53] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo.  Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization.  In CVPR, 2020.
  * [54] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  Graf: Generative radiance fields for 3d-aware image synthesis.  NeurIPS, 2020.
  * [55] Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuiliere, and Nicu Sebe.  Deformable gans for pose-based human image generation.  In CVPR, 2018.
  * [56] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  In NeurIPS, 2020.
  * [57] Vincent Sitzmann, Michael Zollh ö fer, and Gordon Wetzstein.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  In NeurIPS, 2019.
  * [58] Pratul P Srinivasan, Richard Tucker, Jonathan T Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely.  Pushing the boundaries of view extrapolation with multiplane images.  In CVPR, 2019.
  * [59] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox.  Multi-view 3d models from single images with a convolutional network.  In ECCV, 2016.
  * [60] Shubham Tulsiani, Tinghui Zhou, Alexei A Efros, and Jitendra Malik.  Multi-view supervision for single-view reconstruction via differentiable ray consistency.  In CVPR, 2017.
  * [61] Gül Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, and Cordelia Schmid.  Learning from synthetic humans.  In CVPR, 2017.
  * [62] Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee.  Learning to generate long-term future via hierarchical prediction.  In ICML, 2017.
  * [63] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4), 2004.
  * [64] Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli.  Neural scene de-rendering.  In CVPR, 2017.
  * [65] Jiajun Wu, Tianfan Xue, Joseph J Lim, Yuandong Tian, Joshua B Tenenbaum, Antonio Torralba, and William T Freeman.  Single image 3d interpreter network.  In ECCV, 2016.
  * [66] Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee.  Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision.  In NIPS, 2016.
  * [67] Wentao Yuan, Zhaoyang Lv, Tanner Schmidt, and Steven Lovegrove.  Star: Self-supervised tracking and reconstruction of rigid objects in motion with neural rendering.  In CVPR, 2021.
  * [68] Mihai Zanfir, Alin-Ionut Popa, Andrei Zanfir, and Cristian Sminchisescu.  Human appearance transfer.  In CVPR, 2018.
  * [69] Ailing Zeng, Xiao Sun, Fuyang Huang, Minhao Liu, Qiang Xu, and Stephen Lin.  Srnet: Improving generalization in 3d human pose estimation with a split-and-recombine approach.  In ECCV, 2020.
  * [70] Zerong Zheng, Tao Yu, Yixuan Wei, Qionghai Dai, and Yebin Liu.  Deephuman: 3d human reconstruction from a single image.  In ICCV, 2019.
  * [71] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely.  Stereo magnification: Learning view synthesis using multiplane images.  In SIGGRAPH, 2018.
  * [72] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros.  View synthesis by appearance flow.  In ECCV, 2016.
  * [73] Xingyi Zhou, Xiao Sun, Wei Zhang, Shuang Liang, and Yichen Wei.  Deep kinematic pose regression.  In ECCV, 2016.
  * [74] Silvia Zuffi, Angjoo Kanazawa, David Jacobs, and Michael J. Black.  3D menagerie: Modeling the 3D shape and pose of animals.  In CVPR, 2017.

Supplemental Materials: Neural Articulated Radiance Field

Figure 7: Network architecture of Disentangled NARF in our experiments. (a)
Network architecture of $F_{\Theta_{\sigma}}^{l,\zeta}$ and
$F_{\Theta_{c}}^{l,\zeta}$ in Eqs. 22 and 23 in the main paper, modified from
the original NeRF architecture diagram. ‘$+$’ represents concatenation
operation. (b) Network architecture of the selector $\mathcal{S}$.

##  Appendix A Ablation Studies for AutoEncoder

Figure 8: Generative results comparison for AutoEncoder

| Cost | Same pose, same view | Novel pose, same view | Same pose, novel view | Novel pose, novel view
---|---|---|---|---|---
Method | #Params | #FLPOS | #Memory | Mask$\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | Mask $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | Mask $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | Mask $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$
CNN | 15.6M | - | - | 89.0 | 25.59 | 0.8966 | 157.2 | 24.70 | 0.8757 | 385.0 | 22.70 | 0.8213 | 381.5 | 22.98 | 0.8243
P-NeRF | 0.85M | 156M | 356K | 1526.6 | 19.24 | 0.5362 | 1572.1 | 19.60 | 0.5346 | 1911.3 | 18.13 | 0.4666 | 1963.2 | 19.08 | 0.4746
D-NARF | 0.66M | 121M | 382K | 2067.2 | 18.27 | 0.6733 | 2060.2 | 18.81 | 0.6706 | 2026.7 | 18.00 | 0.5983 | 2049.5 | 18.82 | 0.6121
$\text{NARF}_{P}$ | 11.8M | 2140M | 6544K | 159.5 | 22.57 | 0.8250 | 182.3 | 22.58 | 0.8211 | 196.4 | 21.32 | 0.8056 | 205.0 | 21.80 | 0.8088
$\text{NARF}_{H}$ | 1.06M | 197M | 344K | 201.1 | 22.89 | 0.8244 | 225.1 | 23.14 | 0.8205 | 265.1 | 21.48 | 0.7891 | 275.1 | 22.36 | 0.7961
$\text{NARF}_{D}$ | 1.10M | 205M | 382K | 123.4 | 23.84 | 0.8568 | 163.5 | 23.55 | 0.8435 | 166.2 | 22.25 | 0.8313 | 186.3 | 22.81 | 0.8294

Table 2: Quantitative comparison for autoencoders. Best results in bold.

In Table 1 of the main paper, we quantitatively evaluate our model in the case
of a single articulated 3D object, while in Section 5 in the main paper, we
only show qualitative results in the case of Autoencoder. Here, we supplement
those results with the quantitative ablation study for Autoencoder in Table 2.
The same test data settings (namely “same pose/same view”, “novel pose/same
view”, “same pose/novel view”, and “novel pose/novel view”), metrics (namely
PSNR, SSIM, and Mask), and baseline methods (namely $\text{NARF}_{P}$,
$\text{NARF}_{H}$ and $\text{NARF}_{D}$, _CNN-based_ , _P-NeRF_ and _D-N ARF_)
are used for comparison. The same dataset as in Section 5 of the main paper is
used for experimentation.

At testing phase, when extracting the latent shape and appearance vectors
(${\bf z}_{s}$ and ${\bf z}_{a}$) using the encoder, we use images under the
same viewpoint distribution as in the training images as input. Then, images
from novel views and poses are rendered by combining ${\bf z}_{s}$ and ${\bf
z}_{a}$ with unseen views and poses in the training data.

#### Results

Quantitative comparison results are given in Table 2. Qualitative results
under the novel pose/novel view setting are shown in Fig. 8. Consistent with
the case of a single object, our method $\text{NARF}_{D}$ outperforms the
others under all the evaluation metrics and test data settings (best results
shown in bold). High quality depth and segmentation images are jointly
generated as shown in Fig. 8 (rightmost column). CNN based models cannot
represent 3D structure effectively, so the performance drops significantly in
the “novel view” settings. Fig. 8 (left-top) shows that in the novel view
testing, the CNN-based method produces fuzzy images. Meanwhile, it cannot
generate depth and segmentation images. P-NeRF and D-NARF fail in almost all
settings due to implicit transformation and part dependency problems.
$\text{NARF}_{P}$ generally performs well, but the computational cost is too
high. $\text{NARF}_{H}$ has poor performance on “novel pose” due to part
dependency issues. The performance drop on novel pose in the Autoencoder case
is not as significant as in the single object case (shown in Table 1 of the
main paper) since the pose diversity in the training data is much larger in
the Autoencoder case.

##  Appendix B Ablation Studies in RT-NeRF

In Section 3.3 of the main paper, we introduced the rigidly transformed neural
radiance field (RT-NeRF) to effectively model a rigidly transformed object
part. Here, we evaluate the effectiveness of the two most critical design
elements in RT-NeRF. The first is the _explicit transformation_ that converts
a global 3D location into the _local_ coordinate system and the local 3D
location is then used to estimate the density using Eqs. 9 and 10 of the main
paper. The second is the _pose-dependent color_ estimation defined in Eq. 11
of the main paper. It takes the 6D vector $\mathfrak{se}(3)$ representation
$\boldsymbol{\xi}$ of transformation $l$ as a network input to estimate the
RGB color $c$. To this end, two more baseline methods are introduced
accordingly to compare to _RT-NeRF_. The first is the rigid pose conditioned
NeRF (_RP-NeRF_) that takes the global 3D location and the rigid
transformation $\boldsymbol{\xi}$ as network inputs, similar to the P-NeRF
defined in Eq. 8 of the main paper. The second is _RT-NeRF w/o
$\boldsymbol{\xi}$_ that estimates the RGB color $c$ without using the
transformation $\boldsymbol{\xi}$ as input in Eq. 11 of the main paper.

#### Dataset

We create a synthetic rigid object dataset of a rendered bulldozer using
Blender (a software for rendering) for experimentation. In the dataset, the
object (a bulldozer) can rigidly transform in the world coordinate system. For
each rendered image, both rigid transformation and the camera viewpoint are
randomly set. The camera will be translated to point to the center of the
object so that the object will appear in the center of the rendered image. The
resolution for all rendered images is set to 200 $\times$ 200\. In total, 480
images are used for training and another 20 images are used for testing. The
loss function is the same as in Eq. 25 of the main paper.

#### Results

The quantitative results are shown in Table 3 and the qualitative results are
shown in Fig. 9. The experimental results show that RP-NeRF is unable to learn
a good 3D representation and fails to generalize to novel poses and views. In
contrast, RT-NeRF effectively models the rigidly transformed object by
_explicitly_ transforming the global 3D location into the local coordinate
system. In addition, the color estimation without the transformation input is
less effective. This is concluded by comparing the results of “RT-NeRF w/o
$\xi$” with the results of “RT-NeRF”. Quantitatively, the performance of “RT-
NeRF w/o $\xi$” drops significantly under the Mask metric in Table 3.
Qualitatively, the rendered images from “RT-NeRF w/o $\xi$” look blurry
compared to “RT-NeRF” in Fig. 9.

Figure 9: Qualitative results for RT-NeRF comparison

Method | Mask$\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$
---|---|---|---
RP-NeRF | 4511.0 | 11.25 | 0.3103
RT-NeRF w/o $\xi$ | 29.2 | 19.83 | 0.8255
RT-NeRF | 16.9 | 20.05 | 0.8388

Table 3: RT-NeRF quantitative comparison. Best results in bold.

##  Appendix C Additional Ablation Studies in NARF

| Cost | Same pose, same view | Novel pose, same view | Same pose, novel view | Novel pose, novel view
---|---|---|---|---|---
Method | #Params | #FLPOS | #Memory | Mask L2 $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | Mask L2 $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | Mask L2 $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | Mask L2 $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$
CNN | 15.6M | - | - | 76.9 | 29.12 | 0.9429 | 134.8 | 27.30 | 0.9211 | 365.9 | 25.19 | 0.8532 | 392.2 | 24.53 | 0.8470
P-NeRF | 0.85M | 156M | 356K | 778.7 | 21.42 | 0.8006 | 1077.0 | 20.42 | 0.7696 | 844.9 | 21.19 | 0.7897 | 1110.1 | 20.27 | 0.7648
D-NARF | 0.66M | 121M | 382K | 2182.6 | 18.90 | 0.1143 | 2308.2 | 18.81 | 0.1140 | 2137.3 | 19.09 | 0.1144 | 2241.3 | 18.88 | 0.1133
$\text{NARF}_{P}$ | 11.8M | 2149M | 6544K | 92.0 | 28.56 | 0.9258 | 116.2 | 26.83 | 0.9052 | 101.5 | 27.54 | 0.9144 | 125.8 | 26.50 | 0.9104
$\text{NARF}_{H}$ | 1.06M | 197M | 344K | 55.6 | 29.91 | 0.9470 | 376.8 | 24.09 | 0.8665 | 70.5 | 28.81 | 0.9370 | 374.6 | 23.98 | 0.8646
$\text{NARF}_{D}$ | 1.10M | 205M | 382K | 50.5 | 30.86 | 0.9586 | 114.4 | 27.93 | 0.9317 | 64.1 | 29.44 | 0.9466 | 123.8 | 27.24 | 0.9230
$\text{NARF}_{D}sigmoid$ | 1.10M | 205M | 382K | 50.8 | 30.74 | 0.9578 | 117.7 | 27.83 | 0.9304 | 64.6 | 29.35 | 0.9459 | 125.4 | 26.17 | 0.9129
$\text{NARF}_{P}32$ | 0.32M | 59M | 824K | 114.8 | 27.85 | 0.9191 | 139.9 | 26.86 | 0.9068 | 122.9 | 27.14 | 0.9105 | 147.3 | 25.74 | 0.8933
$\text{NARF}_{P}64$ | 0.98M | 178M | 1641K | 105.5 | 27.86 | 0.9212 | 126.9 | 26.78 | 0.9094 | 113.8 | 27.21 | 0.9127 | 133.1 | 26.28 | 0.9041
$\text{NARF}_{P}128$ | 3.28M | 596M | 3275K | 93.8 | 28.46 | 0.9284 | 116.1 | 27.15 | 0.9135 | 102.9 | 27.56 | 0.9180 | 124.2 | 26.23 | 0.9068
$\text{NARF}_{P}256$ | 11.8M | 2149M | 6544K | 92.0 | 28.56 | 0.9258 | 116.2 | 26.83 | 0.9052 | 101.5 | 27.54 | 0.9144 | 125.8 | 26.50 | 0.9104
$\text{NARF}_{P},\tau=0.01$ | 0.32M | 59M | 824K | 240.1 | 25.81 | 0.8730 | 282.0 | 24.92 | 0.8541 | 256.0 | 25.32 | 0.8624 | 287.9 | 23.72 | 0.8333
$\text{NARF}_{P},\tau=0.1$ | 0.32M | 59M | 824K | 168.1 | 26.79 | 0.8995 | 196.7 | 25.92 | 0.8863 | 176.1 | 26.26 | 0.8911 | 204.8 | 25.10 | 0.8756
$\text{NARF}_{P},\tau=1$ | 0.32M | 59M | 824K | 141.5 | 27.24 | 0.9077 | 169.9 | 26.29 | 0.8949 | 150.9 | 26.56 | 0.8989 | 178.2 | 25.16 | 0.8809
$\text{NARF}_{P},\tau=10$ | 0.32M | 59M | 824K | 114.8 | 27.85 | 0.9191 | 139.9 | 26.86 | 0.9068 | 122.9 | 27.14 | 0.9105 | 147.3 | 25.74 | 0.8933
$\text{NARF}_{P},\tau=100$ | 0.32M | 59M | 824K | 107.9 | 27.83 | 0.9190 | 130.2 | 26.96 | 0.9059 | 115.9 | 27.23 | 0.9117 | 137.5 | 26.24 | 0.8983
$\text{NARF}_{P},\tau=1000$ | 0.32M | 59M | 824K | 109.0 | 27.44 | 0.9137 | 130.8 | 26.68 | 0.9014 | 117.0 | 26.91 | 0.9072 | 138.3 | 26.08 | 0.8953
$\text{NARF}_{P}$ w/o $L_{mask}$ | 0.32M | 59M | 824K | 193.9 | 28.01 | 0.9131 | 220.2 | 26.72 | 0.8994 | 208.7 | 27.33 | 0.9028 | 229.8 | 25.54 | 0.8792
$\text{NARF}_{H}$ w/o $L_{mask}$ | 1.06M | 197M | 344K | 6816.2 | 24.25 | 0.7004 | 6893.1 | 21.10 | 0.6636 | 6876.3 | 23.40 | 0.7029 | 6957.2 | 20.53 | 0.5244
$\text{NARF}_{D}$ w/o $L_{mask}$ | 1.10M | 205M | 382K | 113.4 | 31.39 | 0.9630 | 165.7 | 28.09 | 0.9346 | 132.8 | 29.90 | 0.9508 | 263.7 | 25.71 | 0.8889

Table 4: Quantitative results of ablation studies.

In this section, we provide more ablation studies on the effects of the mask
loss (the second item in Eq. 25 of the main paper), temperature parameter for
$\text{NARF}_{P}$ (in Eq. 15 of the main paper), and softmax activation
function for $\text{NARF}_{D}$ (in Eq. 21 of the main paper).

#### W/ and w/o mask loss

We conduct experiments to evaluate the effect of the mask loss (the second
term of Eq. 25 of the main paper) added to the rendered mask image. While the
color loss optimizes the final RGB color of the rendered pixels, the gradient
from the mask loss directly optimizes the densities of the 3D locations on the
camera rays. Therefore, the additional mask loss is helpful in learning 3D
shapes efficiently. The quantitative and qualitative results of w/ and w/o
mask loss are shown in Table 4 and Fig. 10 respectively. In Table 4, it can be
seen that performances of all the three variants of NARF drop significantly
without the mask loss, especially under the novel pose/novel view setting.
Particularly, in Fig. 10, the $\text{NARF}_{H}$ model is not able to converge
at all without the mask loss. The $\text{NARF}_{P}$ and $\text{NARF}_{D}$
models still work without the mask loss but the rendered images get very
blurry, especially on the background regions around the object.

Figure 10: Comparison of mask loss.

#### Temperature parameter $\tau$ in $\text{NARF}_{P}$

We study the effect of the temperature parameter $\tau\in(0,\infty)$ in
$\text{NARF}_{P}$ (in Eq. 15 of the main paper). The temperature parameter
determines how soft the selection is among the multiple RT-NeRFs. When $\tau$
is close to $0$, hard selection is performed. Though the _Part Dependency_
prior is strictly satisfied in this case, convergence in the training is
difficult since the highest current estimate will completely block the
gradient from back-propagating to the others. It is especially worse in the
early stage of the training when the highest estimate is almost random. In
turn, when $\tau$ is close to $\infty$, averaging is performed. In this case,
the gradient is back-propagated to all RT-NeRFs, but the _Part Dependency_
issue arises again, which will harm the generalization ability to novel poses.
The quantitative and qualitative results are shown in Table 4 and Fig. 11,
respectively. We empirically use the best-performing $\tau=100$ setting as
shown in Table 4.

Figure 11: Comparison of temperature $\tau$ of $\text{NARF}_{P}$.

#### Softmax vs. sigmoid activation in $\text{NARF}_{D}$

In Eq. 21 of the main paper, we use softmax activation, which is also
motivated by the _Part Dependency_ prior. Here, we provide the results of
using the sigmoid activation as an alternative. Formally, Eq. 21 of the main
paper is replaced with Eq. 28:

$\displaystyle O_{\Gamma}^{i}:(\gamma({\bf
x}^{l^{i}}),\gamma(\boldsymbol{\zeta}))\rightarrow(o^{i}),\,$ $\displaystyle
p^{i}=\frac{1}{1+\exp(-o^{i})}$ (28)

The quantitative results are shown in Table 4. In conclusion, softmax
outperforms sigmoid, especially in the “novel pose/novel view” setting.

##  Appendix D Implementation Details in P-NeRF

In Eq. 8 of the main paper, P-NeRF takes a global 3D location ${\bf x}$ and
the part transformations $\\{l^{i}|i=1,...,P\\}$ as input. For implementation,
we use the 6D vector $\mathfrak{se}(3)$ representation
${\boldsymbol{\xi}}^{i}$ of transformation $l^{i}$, and concatenate them
together with ${\bf x}$ and the bone length $\zeta$ as the network input.
Positional encoding is performed before the concatenation. Formally,

$\displaystyle F_{\Theta}^{\mathcal{P}}:(\gamma({\bf
x}),\\{\gamma(\xi^{i})|i=1,...,P\\},\gamma(\zeta),\gamma({\bf
d}))\rightarrow(\sigma,c).$ (29)

The density and color sub-networks are defined as

$\displaystyle F_{\Theta_{\sigma}}^{\mathcal{P}}:(\gamma({\bf
x}),\\{\gamma(\xi^{i})|i=1,...,P\\},\gamma(\zeta))\rightarrow(\sigma,{\bf
h}),$ (30) $\displaystyle F_{\Theta_{c}}^{\mathcal{P}}:({\bf
h},\\{\gamma(\xi^{i})|i=1,...,P\\},\gamma({\bf d}))\rightarrow(c).$ (31)

##  Appendix E Implementation Details in D-NARF

D-NERF [47] uses a canonical template and learns the observation-to-canonical
deformation

$\Psi:({\bf x},\omega)\rightarrow{\bf x}^{\prime}$ (32)

where $\omega$ is a deformation latent code.

D-NERF is defined on the deformed position ${\bf x}^{\prime}$ in the canonical
template,

$G:(\Psi({\bf x},\omega),{\bf d},\psi)\rightarrow(\sigma,c)$ (33)

where $\psi$ is a latent appearance code.

In our case, $\omega$ and $\psi$ correspond to the pose configuration
$\mathcal{P}$ and the appearance latent vector ${\bf z}_{a}$ respectively.
$\Psi$ is implemented using MLP, which seems to suffer from the problem of
implicit transformation. In our setup, the pose of each part is given, so it
can be implemented more directly. In our implementation, we first use the
occupancy network similar to the one defined in Eq. 21 of the main paper to
decide which part the input point belongs to.

$\displaystyle O_{\Gamma}^{i}:(\gamma({\bf
x}^{l^{i}}),\gamma(\boldsymbol{\zeta}))\rightarrow(o^{i}),$ (34)
$\displaystyle p^{i}=\frac{\exp(o^{i})}{\sum_{k=1}^{P}\exp(o^{k})},$ (35)

Then, we calculate the coordinates ${\bf x}^{\prime}$ on the canonical shape
as

${\bf x}^{\prime}=\sum_{i}({\bf x}^{l^{i}}+{\bf
t}_{\text{canonical}}^{i})*p_{i}$ (36)

where ${\bf t}_{\text{canonical}}^{i}$ is the origin of a canonical pose’s
$i^{\text{th}}$ part in the global coordinate system. View direction in the
canonical space and the transformation vector are defined as

$\displaystyle{\bf d}=\sum_{i}{\bf
d}^{l^{i}}*p_{i},\,\boldsymbol{\xi}=\sum_{i}\boldsymbol{\xi}^{i}*p_{i}$ (37)

Then, the D-NARF we have implemented in the experiment is defined as

$F_{\Theta_{\sigma}}^{l,\boldsymbol{\zeta}}:(\gamma({\bf
x}^{\prime}),\gamma(\boldsymbol{\zeta}))\rightarrow(\sigma,{\bf h}),$ (38)
$F_{\Theta_{c}}^{l,\boldsymbol{\zeta}}:({\bf h},\gamma({\bf
d}),\gamma(\boldsymbol{\xi}))\rightarrow(c).$ (39)

This implementation is similar to the implementation of $\text{NARF}_{D}$,
differing only in how the coordinates are input to the model. The results of
the experiments show that a concatenation-based $\text{NARF}_{D}$ model that
retains the coordinates for all parts is more effective than a transformation
on the input coordinates in D-NARF.

##  Appendix F Training Details

We used the Adam [26] optimizer with an initial equalized learning rate [25]
of $0.01$. The learning rate is decayed to $0.99995\times$ of the previous
iteration. Particularly, P-NeRF and $\text{NARF}_{H}$ based autoencoders are
trained with an initial learning rate of 0.001 since the training will explode
if a learning rate of 0.01 is used. The batch size is set to 16 for all
experiments. We sample as many camera rays as can be fit in the GPU memory.
The training converges at about 100,000 iterations. The training of our method
$\text{NARF}_{D}$ takes 24 hours on 4 V100 GPUs. The code for creating our
synthetic datasets is available at https://github.com/nogu-atsu/NARF.

##  Appendix G Cross Dataset Evaluation on SURREAL Dataset

In order to verify the generalization ability of NARF autoencoder across
different datasets, we use a cross dataset evaluation protocol that trains the
model on THUman dataset [70], then tests it on SURREAL dataset [61]. There are
several differences between the THUman and SURREAL datasets. First, even
though the human samples in SURREAL contain rendered SMPL meshes and textures
as in THUman, but unlike in THUman, the meshes do not contain clothing.
Second, the camera and shape parameters in SURREAL, including the distance
between camera and human, human pose distribution and the range of body size
are quite different from that in THUman. Even so, we test images from the test
set of SURREAL using the NARF autoencoder trained on THUman. The qualitative
reconstruction results are shown in Fig. 12. From left to right, we show the
input SURREAL images, their reconstructed images, and the images re-rendered
under different pose configurations. Since the THUman dataset is not diverse
enough in terms of clothing and body size, the reconstructions of short pants
and fat people are less effective. But the novel view and pose rendering
results look quite reasonable.

Figure 12: Qualitative results on SURREAL dataset. The cross dataset protocol
is used for evaluation. Figure 13: Results of $\text{NARF}_{D}$ on real human
images

##  Appendix H Experiment on Real Human Images

In this section, we test our approach on a real human dataset ZJU-MOCAP [48].
ZJU-MOCAP is a multi-view person video dataset. For each frame, SMPL
parameters [36] are given. We use the first 1969 frames (90%, 2185 frames in
total) of the Taichi class video for training and the remaining 216 frames
(10%) for testing (novel pose). The resolution of the image is 512 $\times$
512.

The qualitative results of $\text{NARF}_{D}$ on this dataset are shown in Fig.
13. The left part of Fig. 13 shows the pose used in the training, but rendered
from novel viewpoints, and the right part of Fig. 13 shows the novel
pose/novel view testing results. The quality of the rendered images is not as
good as testing on our synthetic datasets. This might be caused by the
assumption that the parts are rigid objects, which may not be perfectly
satisfied for real images. For example, loose clothes may move when a person
makes a movement. This issue can be considered in future work, for example, by
learning latent variables to account for both pose-dependent and pose-
independent deformations similar to Neural Body [48].

Although the quality of the rendered images for a real person from our method
still has room for improvement, we believe that the proposed explicitly
controllable representation of viewpoint, pose, bone parameters, and
appearance for the articulated object is an important contribution.