captionUnsupported captionThe option

# Efficient Meshy Neural Fields for Animatable Human Avatars

Xiaoke Huang1 Yiji Cheng1 Yansong Tang1∗ Xiu Li1 Jie Zhou2 Jiwen Lu2
{1Shenzhen International Graduate School, 2Department of Automation}, Tsinghua
University

###### Abstract

Efficiently digitizing high-fidelity animatable human avatars from videos is a
challenging and active research topic. Recent volume rendering-based neural
representations open a new way for human digitization with their friendly
usability and photo-realistic reconstruction quality. However, they are
inefficient for long optimization times and slow inference speed; their
implicit nature results in entangled geometry, materials, and dynamics of
humans, which are hard to edit afterward. Such drawbacks prevent their direct
applicability to downstream applications, especially the prominent
rasterization-based graphic ones. We present EMA, a method that Efficiently
learns Meshy neural fields to reconstruct animatable human Avatars. It jointly
optimizes explicit triangular canonical mesh, spatial-varying material, and
motion dynamics, via inverse rendering in an end-to-end fashion. Each above
component is derived from separate neural fields, relaxing the requirement of
a template, or rigging. The mesh representation is highly compatible with the
efficient rasterization-based renderer, thus our method only takes about an
hour of training and can render in real-time. Moreover, only minutes of
optimization are enough for plausible reconstruction results. The
disentanglement of meshes enables direct downstream applications. Extensive
experiments illustrate the very competitive performance and significant speed
boost against previous methods. We also showcase applications including novel
pose synthesis, material editing, and relighting. The project page:
https://xk-huang.github.io/ema/.

Figure 1: EMA efficiently and jointly learns canonical shapes, materials, and
motions via differentiable inverse rendering in an end-to-end manner. The
method does not require any predefined templates or riggings. The derived
avatars are animatable and can be directly applied to the graphics renderer
and downstream tasks. All figures are best viewed in color.

††footnotetext: ∗Corresponding author.

##  1 Introduction

Recent years have witnessed the rise of human digitization [25, 2, 70, 3, 72].
This technology greatly impacts the entertainment, education, design, and
engineering industry. There is a well-developed industry solution for this
task. High-fidelity reconstruction of humans can be achieved either with full-
body laser scans [77], dense synchronized multi-view cameras [91, 90], or
light stages [2]. However, these settings are expensive and tedious to deploy
and consist of a complex processing pipeline, preventing the technology’s
democratization.

Another solution is to view the problem as inverse rendering and learn digital
humans directly from custom-collected data. Traditional approaches directly
optimize explicit mesh representation [53, 17, 68] which suffers from the
problems of smooth geometry and coarse textures [71, 4]. Besides, they require
professional artists to design human templates, rigging, and unwrapped UV
coordinates. Recently, with the help of volumetric-based implicit
representations [58, 65, 57] and neural rendering [43, 50, 83], one can easily
digitize a quality-plausible human avatar from video footage [31, 89].
Particularly, volumetric-based implicit representations [58, 70] can
reconstruct scenes or objects with much higher fidelity against previous
neural renderer [83, 71], and is more user-friendly as it does not need any
human templates, pre-set rigging, or UV coordinates. Captured visual footage
and corresponding skeleton tracking are enough for training. However, better
reconstructions and more friendly usability are at the expense of the
following factors. 1) Inefficiency: They require longer optimization times
(typically tens of hours or days) and inference slowly. Volume rendering [58,
52] formulates images by querying the densities and colors of millions of
spatial coordinates. In the training stage, due to memory constraints, only a
small fraction of points are sampled which leads to slow convergence speed. 2)
Entangled representations: The geometry, materials, and motion dynamics are
entangled in the neural networks. Due to the implicit nature of neural nets,
one can hardly edit one property without touching the others [93, 51]. 3)
Graphics incompatibility: Volume rendering is incompatible with the current
popular graphic pipeline, which renders triangular/quadrilateral meshes
efficiently with the rasterization technique. Many downstream applications
require mesh rasterization in their workflow (_e.g_., editing [18], simulation
[7], real-time rendering [59], ray-tracing [84]). Although there are
approaches [54, 42] can convert volumetric fields into meshes, the gaps from
discrete sampling degrade the output quality in terms of both meshes and
textures.

To address these issues, we present EMA, a method based on Efficient Meshy
neural fields to reconstruct animatable human Avatars. Our method enjoys
flexibility from implicit representations and efficiency from explicit meshes,
yet still maintains high-fidelity reconstruction quality. Given video
sequences and the corresponding pose tracking, our method digitizes humans in
terms of canonical triangular meshes, physically-based rendering (PBR)
materials, and skinning weights w.r.t. skeletons. We jointly learn the above
components via inverse rendering [43, 14, 13] in an end-to-end manner. Each of
them is derived from a separate neural field, which relaxes the requirements
of a preset human template, rigging, or UV coordinates. Specifically, we
predict a canonical mesh out of a signed distance field (SDF) by
differentiable marching tetrahedra [79, 20, 19, 61], then we extend the
marching tetrahedra [79] for spatial-varying materials by utilizing a neural
field to predict PBR materials on the mesh surfaces after rasterization [61,
26, 43]. To make the canonical mesh animatable, we take another neural field
to model the forward linear blend skinning for the meshes. Given a posed
skeleton, the canonical mesh is then transformed into the corresponding poses.
Finally, we shade the mesh with a rasterization-based differentiable renderer
[43] and train our models with a photo-metric loss. After training, we export
the mesh with materials and discard the neural fields.

There are several merits of our method design. 1) Efficiency: Powered by
efficient mesh rendering, our method can render in real-time. Besides, the
training speed is boosted as well, since we compute loss holistically on the
whole image and the gradients only flow on the mesh surface. In contrast,
volume rendering takes limited pixels for loss computation and back-propagates
the gradients in the whole space. Our method only needs about an hour of
training and minutes of optimization are enough for plausible avatar
reconstruction. 2) Disentangled representations: Our shape, materials, and
motion modules are disentangled naturally by design, which facilitates
editing. Besides, Canonical meshes with forward skinning modeling handle the
out-of-distribution poses better. 3) Graphics compatibility: Our derived mesh
representation is compatible with the prominent graphic pipeline, which leads
to instant downstream applications (_e.g_., the shape and materials can be
edited directly in design software [18]). To further improve reconstruction
quality, we additionally optimize image-based environment lights and non-rigid
motions.

We conduct extensive experiments on standards benchmarks H36M [30] and ZJU-
MoCap [70]. Our method achieves very competitive performance for novel view
synthesis, generalizes better for novel poses, and significantly improves both
training time and inference speed against previous arts. Our research-oriented
code reaches real-time inference speed (100+ FPS for rendering $512\times 512$
images). We in addition showcase applications including novel pose synthesis,
material editing, and relighting.

##  2 Related Works

Figure 2: The pipeline of EMA. EMA jointly optimizes canonical shapes,
materials, lights, and motions via efficient differentiable inverse rendering.
The canonical shapes are attained firstly through the differentiable marching
tetrahedra [19, 79, 61], which converts SDF fields into meshes. Next, it
queries PBR materials, including diffuse colors, roughness, and specularity on
the mesh surface. Meanwhile, the skinning weights and per-vertices offsets are
predicted on the surface as well, which are then applied to the canonical
meshes with the guide of input skeletons. Finally, a rasterization-based
differentiable renderer takes in the posed meshes, materials, and environment
lights, and renders the final avatars efficiently.

Explicit Representations for Human Modeling: It is intuitive to model the
surfaces of humans with mesh. However, humans are highly varied in both shape
and appearance and have a large pose space, which all contribute to a high
dimensional space. Researchers first model humans with limited clothes. One of
the prevalent methods is parametric models [6, 53, 68, 74, 81]. Fitting humans
from scans is inapplicable in real-world applications. Thus, [34, 9, 41, 96,
95, 40, 82] estimate the human surface from images or videos. To model the
clothed human, [71, 5, 4] deform the template human vertices in canonical
T-pose. However, these methods are prone to capturing coarse geometry due to
the limited capacity of the deformation layer. Besides, the textures are
modeled with sphere harmonics which are far from photo-realistic. Our method
takes the mesh as our core representation to enable efficient training and
rendering and realize the topological change of shape and photo-realistic
texture via neural fields.

Implicit Representations for Human Modeling: Implicit representations [65, 57,
58] model the objects in a continuous manner, whose explicit entity cannot be
attained directly. Specifically, Signed Distance Function [65], Occupancy
Field [57] and Radiance Field [58] are all parametrized by neural networks.
Given full-body scans as 3D supervision, [75, 76, 28, 29, 5] learned the SDFs
or occupancy fields directly from images, which could predict photo-realistic
human avatars in inference phrase. [70, 81, 49, 69, 45, 31, 12, 88, 99, 63,
102] leveraged the radiance field for more photo-realistic human avatars from
multi-view images or single-view videos without any 3D supervision. Although
implicit representations improve reconstruction quality against explicit ones,
they still have drawbacks, _e.g_., large computation burden or poor geometry.s
Besides, volume rendering is incompatible with graphics hardware, thus the
outputs are inapplicable in downstream applications without further post-
processing. Our method absorbs the merits of implicit representations by using
neural networks to predict photo-realistic textures and shape fields,
leveraging [79] to convert SDFs to explicit meshes, which are fully compatible
with the graphic pipeline.

Hybrid Representations for Human Modeling: There are two tracks of literature
modeling humans with explicit geometry representations and implicit texture
representations. One track of literature [38, 101] leveraged neural rendering
techniques [83]. Meshes [71, 101, 4, 3] or point clouds [86] are commonly
chosen explicit representations. Moreover, fine-grained geometry and textures
are learned by neural networks. However, these methods are either only
applicable for novel view synthesis [86] or restricted to self-rotation video
captures [4, 3]. Besides, the neural renderers have limitations, _e.g_.,
stitching texture [36, 37], and baked textures into the renderer. In contrast,
the human avatars learned by our method are compatible with graphics pipeline,
which means they are applicable in downstream tasks, _e.g_., re-posing,
editing in design software. The other track of literature leveraged neural
networks to learn both geometry and textures based on differentiable rendering
[43, 14, 13]. It [50, 8, 43, 73, 44, 72] equips traditional graphics pipeline
with the ability of error backpropagation, which make scene properties (i.e,
assets, lights, cameras poses, etc.) optimizable through gradient descent
w.r.t photo-metric loss. Thus, we can learn both geometry and textures that
are compatible with existing graphics hardware. However, the geometry
optimization process is non-convex and highly unstable [23], so it is hard to
give fine-grained geometry details. Besides, the topology of the mesh is fixed
leading to limited shape modeling. We leverage [61, 79] to convert SDFs to
meshes with differentiable marching tets, and model the motion dynamics of
humans with additional neural fields. Our method enjoys flexibility from
implicit representations and efficiency brought by explicit meshes, yet still
maintains high-fidelity reconstructions.

##  3 Method

We formulate the problem as inverse rendering and extend [61] to model dynamic
actors that are driven solely by skeletons. The canonical shapes, materials,
lights, and actor motions are learned jointly in an end-to-end manner. The
rendering happens with an efficient rasterization-based differentiable
renderer [43].

Optimization Task: Let $\Phi$ denote all the trainable parameters (i.e., SDF
values and corresponding vertices offset parameters for canonical geometry,
spatial-varying and pose-dependent materials and light probe parameters for
shading, and forward skinning weights and non-rigid vertices offset parameters
for motion). For a given camera pose $\mathbf{c}$ and a tracked skeleton pose
$\mathbf{P}$, we render the image $I_{\Phi}(\mathbf{c},\mathbf{P})$ with a
differentiable renderer, and compute loss with a loss function $L$, against
the reference image $I_{\mathit{ref}}(\mathbf{c},\mathbf{P})$. The
optimization goal is to minimize the empirical risk:

$\underset{\phi}{\mathrm{argmin}}\
\mathbb{E}_{\mathbf{c},\mathbf{P}}\big{[}L\big{(}I_{\Phi}(\mathbf{c},\mathbf{P}),I_{\mathit{ref}}(\mathbf{c},\mathbf{P})\big{)}\big{]}.$
(1)

The parameters $\Phi$ are optimized with Adam [39] optimizer. Following [61],
our loss function $L=L_{\mathrm{img}}+L_{\mathrm{mask}}+L_{\mathrm{reg}}$
consists of three parts: an image loss $L_{\mathit{img}}$ using $\ell_{1}$
norm on tone mapped color, and mask loss $L_{\mathrm{mask}}$ using squared
$\ell_{2}$, and regularization losses $L_{\mathrm{reg}}$ to improve the
quality of canonical geometry, materials, lights, and motion. At each
optimization step, our method holistically learns both shape and materials
from the whole image, while the volume rendering-based implicit counterparts
only learn from limited pixels. Powered by an efficient rasterization-based
renderer, our method enjoys both faster convergence and real-time rendering
speed. For optimization losses and implementation details, please refer to our
supplementary.

###  3.1 Canonical Geometry

Rasterization-based differentiable renderers take triangular meshes as input,
which means the whole optimization process happens over the mesh
representation. Previous works [4, 3] require a mesh template to assist
optimization as either a good initialization or a shape regularization. The
templates have fixed topology under limited resolutions which harm the
geometry quality. Besides, to make the learned geometry generalize to novel
poses, the underlying geometry representations should lie in a canonical
space.

We utilize the differentiable marching tetrahedra [79, 19] algorithm, which
converts SDF fields into triangular meshes to model the actors in canonical
space. This method enjoys the merit of both template and topology-free from
implicit SDF representations and outputs triangle meshes that are directly
applicable to rasterization-based renderers.

Let $\mathbf{V}_{\mathrm{tet}}$, $\mathbf{F}_{\mathrm{tet}}$,
$\mathbf{T}_{\mathrm{tet}}$ and be the pre-defined vertices, faces, and UV
coordinates of the tetrahedra grid. We parameterize both per-tet vertice SDF
value $\mathbf{S}$ and vertices offsets $\Delta\mathbf{V}_{\mathrm{tet}}$ with
a coordiante-based nerual net:

$F_{\Phi_{\mathit{geom}}}:(\mathbf{V}_{\mathrm{tet}})\to(\mathbf{S},\Delta\mathbf{V}_{\mathrm{tet}}),$
(2)

The canonical mesh
$\mathbf{M}_{\mathrm{c}}=(\mathbf{V}_{\mathrm{c}},\mathbf{F}_{\mathrm{c}},\mathbf{T}_{\mathrm{c}})$
(i.e, canonical mesh vertices, faces, and UV map coordinates) is derived by
marching tetrahedra operator $\Pi$:

$\Pi:(\mathbf{V}_{\mathrm{tet}},\mathbf{F}_{\mathrm{tet}},\mathbf{T}_{\mathrm{tet}},\mathbf{S},\Delta\mathbf{V}_{\mathrm{tet}})\to\\\
(\mathbf{V}_{\mathrm{c}},\mathbf{F}_{\mathrm{c}},\mathbf{T}_{\mathrm{c}}).$
(3)

Specifically, the vertices of the canonical mesh are computed by
$\mathbf{v}_{c}^{ij}=\frac{\mathbf{v}_{\mathit{tet}}^{\prime
i}s_{j}-\mathbf{v}_{\mathit{tet}}^{\prime j}s_{i}}{s_{j}-s_{i}}$, where
$\mathbf{v}_{\mathit{tet}}^{\prime i}=\mathbf{v}^{i}+\Delta\mathbf{v}^{i}$ and
$\mathrm{sign}(s_{i})\neq\mathrm{sign}(s_{j})$. In other words, only the edges
that cross the surface of canonical mesh participate the marching tetrahedra
operator, which makes the mesh extraction both computation and memory-
efficient.

After training, we can discard the SDF and deformation neural nets
$F_{\Phi_{\mathit{geom}}}$ and store the derived meshes. That leads to zero
computation overhead in inference time.

Figure 3: Qualitative results of novel view synthesis on the H36M and ZJU-
MoCap datasets. [70, 69] generates blurry textures compared with our method.
The mesh representations and forward skinning modeling help to improve
generalization. Left: Hn36M dataset. Right: ZJU-MoCap dataset. Up: Training
pose. Down: Novel pose. Zoom in for a better view.

###  3.2 Shading Model

Materials: we use a physically-based material model [56], which is directly
applicable to our differentiable renderer. It consists of a diffuse term with
an isotopic GGX lobe representing specularity. Concretely, it consists of
three parts: 1) diffuse lobe $\mathbf{k}_{d}$ has four components, i.e. RGB
color channels and an additional alpha channel; 2) specular lobe comprises a
roughness value $r$ for GGX normal distribution function and a metalness
factor $m$ which interpolates the sense of reality from plastic to pure
metallic appearance. The specular highlight color is given by
$\mathbf{k}_{s}=(1-m)\cdot 0.04+m\cdot\mathbf{k}_{d}$. We store the specular
lobe into texture $\mathbf{k}_{\mathrm{orm}}=(o,r,m)$ following the
convention, where the channel $o$ is unused. To compensate for the global
illumination, we alternatively store the ambient occlusion value into $o$. 3)
normal maps $\mathbf{n}$ represents the fine-grained geometry details. The
diffues color $\mathbf{k}_{d}$, texture $\mathbf{k}_{\mathrm{orm}}$, and
normal maps $\mathbf{n}$ are parametrized by an neural net:

$F_{\Phi_{\mathrm{mat}}}:(\mathbf{v}_{c},\mathbf{P})\to(\mathbf{k}_{d},\mathbf{k}_{\mathrm{orm}},\mathbf{n}).$
(4)

We query the vertices after rasterization and barycentric interpolation. The
PBR material is further conditioned on poses to model the pose-dependent
shading effect.

Lights: Our method learns a fixed environment light directly from the
reference images. The lights are represented as a cube light. Given direction
$\omega_{o}$, We follow the render equation [32] to compute the outgoing
radiance $L\left(\omega_{o}\right)$:

$L\left(\omega_{o}\right)=\int_{\Omega}L_{i}\left(\omega_{i}\right)f\left(\omega_{i},\omega_{o}\right)\left(\omega_{i}\cdot\mathbf{n}\right)d\omega_{i},$
(5)

The outgoing radiance is the integral of the incident radiance
$L_{i}\left(\omega_{i}\right)$ and the BRDF
$f\left(\omega_{i},\omega_{o}\right)$. We do not use spherical Gaussians [13]
or spherical harmonics [10, 97] to approximate the image-based lighting.
Instead, we follow [61] using the split sum approximation that capable of
modeling all-frequency image-based lighting:

$\begin{split}L\left(\omega_{o}\right)&\approx\int_{\Omega}f\left(\omega_{i},\omega_{o}\right)\left(\omega_{i}\cdot\mathbf{n}\right)d\omega_{i}\\\
&\quad\int_{\Omega}L_{i}\left(\omega_{i}\right)D\left(\omega_{i},\omega_{o}\right)\left(\omega_{i}\cdot\mathbf{n}\right)d\omega_{i}.\end{split}$
(6)

The materials and lights are optimized jointly with geometry and motion
modules in an end-to-end manner. The decomposed design of geometry and
shading, along with compatibility with the triangle renderer enables editing
and content creation instantly after training. For details about light
modeling, please refer to the supplementary.

###  3.3 Motion Model

Since we derived mesh-based actors in canonical space with materials and
lights, it is intuitive and natural to choose forward linear skinning as our
motion model. Given a skeleton with $B$ bones, the skeleton poses
$\mathbf{P}=\\{\mathbf{T}_{1},\mathbf{T}_{2},\ldots,\mathbf{T}_{B}\\}$, where
each $\mathbf{T}_{i}$ represents the transformation on bone $i$, and the blend
skinning weights $\mathbf{W}=\\{w_{1},w_{2},\ldots,w_{B}\\}$, we deform each
mesh vertice $\mathbf{v}_{c}$ in canonical space to the posed vertice
$\mathbf{v}_{w}$ in world space by:

$\mathbf{v}_{w}=\mathrm{LBS}(\mathbf{v}_{c},\mathbf{P},\mathbf{W})=(\sum\limits_{i=1}^{B}w_{i}\mathbf{T}_{i})\mathbf{v}_{c},$
(7)

To compensate for non-rigid cloth dynamics, we further add a layer of pose-
dependent non-rigid offsets $\Delta\mathbf{v}_{c}$ for canonical meshes:

$\mathbf{v}_{w}=\mathrm{LBS}(\mathbf{v}_{c}+\Delta\mathbf{v}_{c},\mathbf{P},\mathbf{W}),$
(8)

Where the blend skinning weights and the pose-dependent non-rigid offsets are,
respectively, parameterized by neural nets whose inputs are canonical mesh
vertices:

$F_{\Phi_{\mathrm{LBS}}}:(\mathbf{v}_{c})\to\mathbf{W},$ (9)
$\quad\;F_{\Phi_{\mathrm{nr}}}:(\mathbf{v}_{c},\mathbf{P})\to\Delta\mathbf{v}_{c}.$
(10)

Modeling forward skinning is efficient for training as it only forward once in
each optimization step, while the volume-based methods [45, 88, 15] solve the
root-finding problem for canonical points in every iteration. After training,
we can export the skinning weight from neural nets which removes the extra
computation burden for inference.

Table 1: Quantitative results. On the marker-based H36M, our method achieves
SOTA performance in all optimization durations. While on the markerless ZJU-
MoCap, our method is comparable with previous arts. “T.F.” means template-
free; “Rep.” means representation; “T.T” means the training time; $\ast$
denotes the evaluation on a subset of validation splits.

| H36M | ZJUMOCAP
---|---|---
|  |  |  | Training pose | Novel pose | Training pose | Novel pose
| T.F. | Rep. | T.T. | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
NB [70] |  | NV |  $\sim$10 h | 23.31 | 0.902 | 22.59 | 0.882 | 28.10 | 0.944 | 23.49 | 0.885
SA-NeRF [92] |  | NV |  $\sim$30 h | 24.28 | 0.909 | 23.25 | 0.892 |  [HTML]FFFC9E28.27 |  [HTML]FFFC9E0.945 |  [HTML]FFFC9E24.42 |  [HTML]FFFC9E0.902
Ani-NeRF [69] | $\checkmark$ | NV |  $\sim$10 h | 23.00 | 0.890 | 22.55 | 0.880 | 26.19 | 0.921 | 23.38 | 0.892
ARAH [88] | $\checkmark$ | NV |  $\sim$48 h |  [HTML]FFCCC9 24.79 |  [HTML]FFCCC90.918 |  [HTML]FFFC9E 23.42 |  [HTML]FFFC9E 0.896 |  [HTML]FFCCC928.51 |  [HTML]FFCCC90.948 |  [HTML]FFCCC924.63 |  [HTML]FFCCC90.911
Ours | $\checkmark$ | Hybr |  $\sim$1 h |  [HTML]FFFC9E24.72 |  [HTML]FFFC9E0.916 |  [HTML]FFCCC923.64 |  [HTML]FFCCC90.899 | 26.57 | 0.901 | 24.38 | 0.875
NB |  | NV |  | 20.58 | 0.879 | 20.27 | 0.867 |  [HTML]FFCCC926.87 |  [HTML]FFFC9E0.922 | 23.67 |  [HTML]FFFC9E0.885
SA-NeRF |  | NV |  | 21.03 | 0.878 | 20.71 | 0.869 | 24.92 | 0.882 | 23.38 | 0.869
Ani-NeRF | $\checkmark$ | NV |  | 22.54 | 0.872 | 21.79 | 0.856 | 21.23 | 0.659 | 20.65 | 0.652
ARAH | $\checkmark$ | NV |  |  [HTML]FFFC9E24.25 |  [HTML]FFFC9E0.904 |  [HTML]FFFC9E23.61 |  [HTML]FFFC9E0.892 | 26.33 |  [HTML]FFCCC90.924 |  [HTML]FFCCC924.67 |  [HTML]FFCCC90.911
Ours | $\checkmark$ | Hybr | $\sim$1 h∗ |  [HTML]FFCCC924.83 |  [HTML]FFCCC90.917 |  [HTML]FFCCC923.64 |  [HTML]FFCCC90.899 |  [HTML]FFFC9E26.66 | 0.901 |  [HTML]FFFC9E24.64 | 0.880
NB |  | NV |  | 20.54 | 0.863 | 20.15 | 0.853 |  [HTML]FFFC9E25.37 |  [HTML]FFFC9E0.894 | 23.54 | 0.873
SA-NeRF |  | NV |  | 20.81 | 0.848 | 20.49 | 0.841 | 24.48 | 0.878 | 23.75 | 0.872
Ani-NeRF | $\checkmark$ | NV |  | 20.57 | 0.822 | 20.22 | 0.806 | 21.17 | 0.652 | 21.16 | 0.656
ARAH | $\checkmark$ | NV |  |  [HTML]FFFC9E23.83 |  [HTML]FFFC9E0.895 |  [HTML]FFFC9E23.13 |  [HTML]FFFC9E0.884 | 25.09 |  [HTML]FFCCC90.906 |  [HTML]FFFC9E24.21 |  [HTML]FFCCC90.898
Ours | $\checkmark$ | Hybr | $\sim$10 m∗ |  [HTML]FFCCC924.27 |  [HTML]FFCCC90.909 |  [HTML]FFCCC923.37 |  [HTML]FFCCC90.897 |  [HTML]FFCCC925.51 | 0.888 |  [HTML]FFCCC924.42 |  [HTML]FFFC9E0.878

##  4 Experiments

###  4.1 Dataset and Metrics

H36M consists of 4 multi-view cameras and uses marker-based motion capture to
collect human poses. Each video contains a single subject performing a complex
action. We follow [69] data protocol which includes subject S1, S5-S9, and
S11. The videos are split into two parts: “training poses” for novel view
synthesis and “Unseen poses” for novel pose synthesis. Among the video frames,
3 views are used for training, and the rest views are for evaluation. The
novel view and novel pose metrics are computed on rest views. We use the same
data preprocessing as [69].

ZJU-MoCap records 9 subjects performing complex actions with 23 cameras. The
human poses are obtained with a markerless motion capture system. Thus the
pose tracking is rather noisier compared with H36M. Likewise, there are two
sets of video frames, “training poses” for novel view synthesis and “Unseen
poses” for novel pose synthesis. 4 evenly distributed camera views are chosen
for training, and the rest 19 views are for evaluation. Again, the evaluation
metrics are computed on rest views. The same data protocol and processing
approaches are adopted following [70, 69].

Metrics. We follow the typical protocol in [70, 69] using two metrics to
measure image quality: peak signal-to-noise ratio (PSNR) and structural
similarity index (SSIM).

###  4.2 Evaluation and Comparison

Baselines. We compare our method with template-based methods [70, 92] and
template-free methods [69, 88]. Here we list the average metric values with
different training times to illustrate our very competitive performance and
significant speed boost. 1) Tempelate-based methods. Neural Body (NB) [70]
learns a set of latent codes anchored to a deformable template mesh to provide
geometry guidance. Surface-Aligned NeRF (SA-NeRF) [92] proposes projecting a
point onto a mesh surface to align surface points and signed height to the
surface. 2) Template-free methods. Animatable NeRF (Ani-NeRF) [69] introduces
neural blend weight fields to produce the deformation fields instead of
explicit template control. ARAH [88] combines an articulated implicit surface
representation with volume rendering and proposes a novel joint root-finding
algorithm.

Comparisons with state-of-the-arts. Table 1 illustrates the quantitative
comparisons with previous arts. Notably, our method achieves very competitive
performance within much less training time. The previous volume rendering-
based counterparts spend tens of hours of optimization time, while our method
only takes an hour of training (for previous SOTA method ARAH [88], it takes
about 2 days of training). On the marker-based H36M dataset, our method
reaches the SOTA performance in terms of novel view synthesis on training
poses and outperforms previous SOTA (ARAH [88]) for novel view synthesis on
novel poses, which indicates that our method can generalize better on novel
poses. The significant boost in training speeds lies in, on the one hand, the
core mesh representation which can be rendered efficiently with the current
graphic pipeline [43]. On the other hand, the triangular renderer uses less
memory. Thus we can compute losses over the whole image to learn the
representations holistically. In contrast, previous methods are limited to
much fewer sampled pixels in each optimization step.

On the markerless ZJU-Mocap dataset, our method falls behind for training
poses novel view synthesis and ranks 3rd place in terms of unseen poses novel
view synthesis among the competitors. We argue that the quality of pose
tracking results in the performance gaps between the two datasets. The
markerless pose tracking data are much noisier than the marker-based ones
(_e.g_., the tracked skeleton sequence is jittering, and the naked human [53]
rendering is misaligned with human parsings), which makes our performance
saturated by harming the multi-view consistency. The problem is even amplified
with the holistic loss computation over the whole pixels. We conduct
additional ablation on pose tracking quality on H36M in Sec. 3. Besides, our
non-rigid modeling is only over the surface (no topology change), which is
less powerful than the volume rendering-based ones (with topology change).

We further each method under the same optimization duration in Table 1. For
the extremely low inference speed of our competitor, we only evaluate at most
10 frames in each subject, and for ZJU MoCap we only choose another 4 evenly
spaced cameras as the evaluation views. For both 1 hour and 10 minutes
optimization time, our method outperforms other methods for both training
poses and unseen poses novel view synthesis on the marker-based H36M dataset.
On the markerless ZJU-Mocap dataset, our method is comparable with previous
SOTA in terms of PSNR and SSIM for both evaluation splits.

Figure 3 illustrates the qualitative comparisons between our method and
previous arts under the same optimization duration. It is worth noting that on
both H36M and ZJU-MoCap datasets, our method can synthesize clearer and more
fine-grained images against competitors, which raises the misalignment of the
quantitative metrics for measuring image similarity.

Figure 4: Rendering Efficiency

Rendering Efficiency: We provide the rendering speed of our method against
previous methods in Figure 4. Our method reaches real-time inference speed
(100+ FPS for rendering 512×512 images), which is hundreds of times faster
than the previous ones. Our method takes considerably less memory than the
previous ones.

###  4.3 Ablation Studies

Table 2:  The ablation on each module from our method. The mesh tends to be noisy and poor for rendering novel poses without MLP parametrization for the geometry module; Removing the non-rigid module harms the convergence of our model due to the disability to solve multi-view inconsistency; PBR materials improve the overall shading quality by joint modeling both decomposed materials and lighting.  | Training Pose | Novel Pose
---|---|---
| PSNR | SSIM | PSNR | SSIM
w/o SDF MLP | 25.17 | 0.913 | 23.37 | 0.874
w/o Non-rigid | 25.03 | 0.909 | 23.45 | 0.877
w/o PBR | 25.10 | 0.914 | 23.44 | 0.878
w/o Specular | 25.24 | 0.915 | 23.58 | 0.879
Full | 25.26 | 0.916 | 23.52 | 0.879
Figure 5: Qualitative ablation on each module. The SDF MLP improves the mesh
smoothness; non-rigid modeling proves the texture quality by solving the
multi-view consistency of cloth dynamics; The PBR materials have a larger
capacity for modeling complex materials and lighting against the only-RGB and
the no-specular counterparts, which further facilitates both mesh and material
learning.

We conduct ablation studies on the H36M S9 subject.

The parametrization type for SDF field. The SDF fields can either be
parameterized as either MLPs or value fields. Table 2 and Figure 5 show that
using MLP to predict SDF values results in a smoother mesh surface that is
watertight. MLP offers extrapolation ability to predict invisible parts and
keep the mesh watertight. While directly optimizing SDF value fields leads to
a jiggling mesh surface and holes in invisible parts during training (_e.g_.,
underarm).

The shading model type in geometry module. We compare PBR shading models with
directly predicting RGB colors and PBR without shading specular. Table 2 and
Figure 5 show that PBR shading models lead to higher metrics against RGB
predications, which indicates that PBR materials can better model complex
textures and lights for dynamic humans. Removing the specular term in PBR does
not affect the performance much. We conjecture that there is less specularity
in human skin and clothes materials.

The impact of the non-rigid net in motion module. As shown in Table 2 and
Figures 5, modeling pose-dependent non-rigid dynamics of clothes improves the
overall reconstruction quality. It facilitates the aggregation of shading
information for multi-view inputs during training.

Figure 6: Qualitative results of models trained on poses from marker-less and marker-based systems. Figure 7: Comparison of models trained with different numbers of camera views on the subject “S9”. Table 3: The ablations results on data quality and quantity on H36M S9 subject, in terms of PSNR and SSIM (higher is better). The better the data quality, the better the reconstruction results. | Training pose | Novel pose
---|---|---
| PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
(a) type of pose tracking
w/o marker | 24.73 | 0.893 | 22.60 | 0.853
w/ marker | 25.53 | 0.911 | 23.80 | 0.879
(b) number of training views
1 view | 25.09 | 0.906 | 22.97 | 0.866
2 views | 25.56 | 0.911 | 23.76 | 0.878
3 views | 25.57 | 0.911 | 23.67 | 0.876
(c) number of training frames
1 frame | 20.93 | 0.817 | 19.58 | 0.785
100 frames | 23.99 | 0.882 | 22.49 | 0.856
200 frames | 25.27 | 0.905 | 23.32 | 0.873
800 frames | 24.89 | 0.900 | 23.16 | 0.873

The impact of human tracking quality. Table 3 (a) and Figure 6 show that using
marker-based pose-tracking data can give better results. The same phenomenon
has been stated in [69]. Noisy marker-less pose-tracking harms the
optimization process by damaging the multi-view consistency and the exact pose
for shading optimization, which leads to blurry textures.

The impact of training view amount. Table 3 (b) and Figure 7 reveal that
giving one camera of view degrades the overall reconstruction quality, and
multi-view consistency improves the final results. The model can aggregate
multi-view information for better shading optimization, thus leading to
clearer surface materials.

The impact of training frame amount. As the number of training frames
increases, the rendering quality on novel view and novel pose increases as
well (Table 3 (c) and Figure 8). Notice that the reconstruction quality
saturated after using a certain amount of training frames, the same results
can be observed in [69] as well.

Figure 8: Comparison of models trained with different numbers of video frames
on the subject “S9”.

###  4.4 Applications

After training, we can export mesh representations, which enables instant
downstream applications. We showcase two examples of novel pose synthesis,
material editing, and human relighting in Figure 1. For more examples, please
refer to our supplementary.

##  5 Discussions and Conclusions

Discussions: Our method leverage mesh as our core representation, which
enables us efficiency for both training and rendering. However, the resolution
of mesh is fixed in our pipeline, preventing fine-grained geometry and texture
reconstruction. One possible solution could be tetrahedra grids sub-division
[78, 21, 33]. But it may break the SDF values around the derived meshes since
there is no regularization over the whole SDF field. Our non-rigid modeling
has less capacity, since we assume there is no topology change of mesh wrt.
the non-rigid motion. Otherwise, we cannot query materials and motions in the
canonical shape. One can solve it via the dense correspondence between the
meshes before and after applying non-rigid motions [1, 94], yet such an
operation may increase computation drastically.

Conclusions: we present EMA, which learns human avatars through hybrid meshy
neural fields efficiently. EMA jointly learns hybrid canonical geometry,
materials, lights, and motions via a rasterization-based differentiable
renderer. It only requires one hour of training and can render in real-time
with a triangle renderer. Minutes of training can produce plausible results.
Our method enjoys flexibility from implicit representations and efficiency
from explicit meshes. Experiments on the standard benchmark indicate the
competitive performance and generalization results of our method. The
digitized avatars can be directly used in downstream tasks. We showcase
examples including novel pose synthesis, material editing, and human
relighting.

## References

  * [1] Naveed Ahmed, Christian Theobalt, Christian Rössl, Sebastian Thrun, and Hans-Peter Seidel.  Dense correspondence finding for parametrization-free animation reconstruction from video.  In CVPR, pages 1–8, 2008.
  * [2] Oleg Alexander, Mike Rogers, William Lambeth, Jen-Yuan Chiang, Wan-Chun Ma, Chuan-Chang Wang, and Paul E. Debevec.  The digital emily project: Achieving a photorealistic digital actor.  IEEE Comput. Graph. Appl., 30(4):20–31, 2010.
  * [3] Thiemo Alldieck, Marcus A. Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll.  Detailed human avatars from monocular video.  In 3DV, pages 98–109, 2018.
  * [4] Thiemo Alldieck, Marcus A. Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll.  Video based reconstruction of 3d people models.  In CVPR, pages 8387–8397, 2018.
  * [5] Thiemo Alldieck, Mihai Zanfir, and Cristian Sminchisescu.  Photorealistic monocular 3d reconstruction of humans wearing clothing.  In CVPR, pages 1496–1505, 2022.
  * [6] Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis.  SCAPE: shape completion and animation of people.  ACM Trans. Graph., 24(3):408–416, 2005.
  * [7] Jan Bender, Matthias Müller, Miguel A. Otaduy, Matthias Teschner, and Miles Macklin.  A survey on position-based simulation methods in computer graphics.  Comput. Graph. Forum, 33(6):228–251, 2014.
  * [8] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3d faces.  In SIGGRAPH, pages 187–194, 1999.
  * [9] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter V. Gehler, Javier Romero, and Michael J. Black.  Keep it SMPL: automatic estimation of 3d human pose and shape from a single image.  In ECCV, pages 561–578, 2016.
  * [10] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P. A. Lensch.  Nerd: Neural reflectance decomposition from image collections.  In ICCV, pages 12664–12674, 2021.
  * [11] Lu Chen, Jiao Sun, and Wei Xu.  FAWA: fast adversarial watermark attack on optical character recognition (OCR) systems.  In ECML, pages 547–563, 2020.
  * [12] Mingfei Chen, Jianfeng Zhang, Xiangyu Xu, Lijuan Liu, Yujun Cai, Jiashi Feng, and Shuicheng Yan.  Geometry-guided progressive nerf for generalizable and efficient neural human rendering.  In ECCV, pages 222–239, 2022.
  * [13] Wenzheng Chen, Huan Ling, Jun Gao, Edward J. Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler.  Learning to predict 3d objects with an interpolation-based differentiable renderer.  In NeurIPS, pages 9605–9616, 2019.
  * [14] Wenzheng Chen, Joey Litalien, Jun Gao, Zian Wang, Clement Fuji Tsang, Sameh Khamis, Or Litany, and Sanja Fidler.  DIB-R++: learning to predict lighting and material with a hybrid differentiable renderer.  In NeurIPS, pages 22834–22848, 2021.
  * [15] Xu Chen, Yufeng Zheng, Michael J. Black, Otmar Hilliges, and Andreas Geiger.  SNARF: differentiable forward skinning for animating non-rigid neural implicit shapes.  In ICCV, pages 11574–11584, 2021.
  * [16] Robert L. Cook and Kenneth E. Torrance.  A reflectance model for computer graphics.  ACM Trans. Graph., 1(1):7–24, 1982.
  * [17] Haoshu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu.  RMPE: regional multi-person pose estimation.  In ICCV, pages 2353–2362, 2017.
  * [18] Blender Foundation.  Blender.org - Home of the Blender project - Free and Open 3D Creation Software.
  * [19] Jun Gao, Wenzheng Chen, Tommy Xiang, Alec Jacobson, Morgan McGuire, and Sanja Fidler.  Learning deformable tetrahedral meshes for 3d reconstruction.  In NeurIPS, 2020.
  * [20] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler.  GET3D: A generative model of high quality 3d textured shapes learned from images.  arXiv:2209.11163, 2022.
  * [21] William Gao, April Wang, Gal Metzer, Raymond A. Yeh, and Rana Hanocka.  Tetgan: A convolutional neural network for tetrahedral mesh generation.  In BMVC, page 365, 2022.
  * [22] Ke Gong, Xiaodan Liang, Yicheng Li, Yimin Chen, Ming Yang, and Liang Lin.  Instance-Level Human Parsing via Part Grouping Network.  In ECCV, pages 805–822, 2018.
  * [23] Philip-William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, and Justus Thies.  Neural head avatars from monocular RGB videos.  In CVPR, pages 18632–18643, 2022.
  * [24] Chen Guo, Tianjian Jiang, Xu Chen, Jie Song, and Otmar Hilliges.  Vid2avatar: 3d avatar reconstruction from videos in the wild via self-supervised scene decomposition.  arXiv:2302.11566, 2023.
  * [25] Marc Habermann, Weipeng Xu, Michael Zollhöfer, Gerard Pons-Moll, and Christian Theobalt.  Deepcap: Monocular human performance capture using weak supervision.  In CVPR, pages 5051–5062, 2020.
  * [26] Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg.  Shape, light & material decomposition from images using monte carlo rendering and denoising.  arXiv:2206.03380, 2022.
  * [27] Jon Hasselgren, Jacob Munkberg, Jaakko Lehtinen, Miika Aittala, and Samuli Laine.  Appearance-Driven Automatic 3D Model Simplification.  arXiv:2104.03989, 2021.
  * [28] Tong He, Yuanlu Xu, Shunsuke Saito, Stefano Soatto, and Tony Tung.  ARCH++: animation-ready clothed human reconstruction revisited.  In ICCV, pages 11026–11036, 2021.
  * [29] Zeng Huang, Yuanlu Xu, Christoph Lassner, Hao Li, and Tony Tung.  ARCH: animatable reconstruction of clothed humans.  In CVPR, pages 3090–3099, 2020.
  * [30] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.  Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.  IEEE Trans. Pattern Anal. Mach. Intell., 36(7):1325–1339, 2014\.
  * [31] Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and Anurag Ranjan.  Neuman: Neural human radiance field from a single video.  In ECCV, pages 402–418, 2022.
  * [32] James T. Kajiya.  The rendering equation.  In SIGGRAPH, pages 143–150, 1986.
  * [33] Nikolai Kalischek, Torben Peters, Jan D. Wegner, and Konrad Schindler.  Tetrahedral diffusion models for 3d shape generation.  arXiv:2211.13220, 2022.
  * [34] Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose.  In CVPR, pages 7122–7131, 2018.
  * [35] Brian Karis.  Real shading in unreal engine 4.  SIGGRAPH 2013 Course: Physically Based Shading in Theory and Practice, 2013.
  * [36] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  IEEE Trans. Pattern Anal. Mach. Intell., 43(12):4217–4228, 2021\.
  * [37] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In CVPR, pages 8107–8116, 2020.
  * [38] Taras Khakhulin, Vanessa Sklyarova, Victor Lempitsky, and Egor Zakharov.  Realistic one-shot mesh-based head avatars.  In ECCV, pages 345–362, 2022.
  * [39] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [40] Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black.  VIBE: video inference for human body pose and shape estimation.  In CVPR, pages 5252–5262, 2020.
  * [41] Muhammed Kocabas, Chun-Hao P. Huang, Otmar Hilliges, and Michael J. Black.  PARE: part attention regressor for 3d human body estimation.  In ICCV, pages 11107–11117, 2021.
  * [42] François Labelle and Jonathan Richard Shewchuk.  Isosurface stuffing: fast tetrahedral meshes with good dihedral angles.  ACM Trans. Graph., 26(3):57, 2007.
  * [43] Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila.  Modular primitives for high-performance differentiable rendering.  ACM Trans. Graph., 39(6):194:1–194:14, 2020.
  * [44] Christoph Lassner and Michael Zollhöfer.  Pulsar: Efficient sphere-based neural rendering.  In CVPR, pages 1440–1449, 2021.
  * [45] Ruilong Li, Julian Tanke, Minh Vo, Michael Zollhöfer, Jürgen Gall, Angjoo Kanazawa, and Christoph Lassner.  TAVA: template-free animatable volumetric actors.  In ECCV, pages 419–436, 2022.
  * [46] Ruilong Li, Sha Yang, David A. Ross, and Angjoo Kanazawa.  Ai choreographer: Music conditioned 3d dance generation with aist++.  In ICCV, pages 13381–13392, 2021.
  * [47] Xiaoting Li, Lingwei Chen, Jinquan Zhang, James R. Larus, and Dinghao Wu.  Watermarking-based defense against adversarial attacks on deep neural networks.  In International Joint Conference on Neural Networks, IJCNN 2021, Shenzhen, China, July 18-22, 2021, pages 1–8, 2021.
  * [48] Hsueh-Ti Derek Liu, Francis Williams, Alec Jacobson, Sanja Fidler, and Or Litany.  Learning smooth neural functions via lipschitz regularization.  In SIGGRAPH, pages 31:1–31:13, 2022.
  * [49] Lingjie Liu, Marc Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, and Christian Theobalt.  Neural actor: neural free-view synthesis of human actors with pose control.  ACM Trans. Graph., 40(6):219:1–219:16, 2021.
  * [50] Shichen Liu, Weikai Chen, Tianye Li, and Hao Li.  Soft rasterizer: A differentiable renderer for image-based 3d reasoning.  In ICCV, pages 7707–7716, 2019.
  * [51] Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, and Bryan Russell.  Editing conditional radiance fields.  In ICCV, pages 5753–5763, 2021.
  * [52] Stephen Lombardi, Tomas Simon, Jason M. Saragih, Gabriel Schwartz, Andreas M. Lehrmann, and Yaser Sheikh.  Neural volumes: learning dynamic renderable volumes from images.  ACM Trans. Graph., 38(4):65:1–65:14, 2019.
  * [53] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  SMPL: a skinned multi-person linear model.  ACM Trans. Graph., 34(6):248:1–248:16, 2015.
  * [54] William E. Lorensen and Harvey E. Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  In SIGGRAPH, pages 163–169, 1987.
  * [55] Yi Ma, Stefano Soatto, Jana Kosecka, and S. Shankar Sastry.  An Invitation to 3-D Vision.  Springer, 2004.
  * [56] Stephen McAuley, Stephen Hill, Naty Hoffman, Yoshiharu Gotanda, Brian E. Smits, Brent Burley, and Adam Martinez.  Practical physically-based shading in film and game production.  In SIGGRAPH, pages 10:1–10:7, 2012.
  * [57] Lars M. Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In CVPR, pages 4460–4470, 2019.
  * [58] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: representing scenes as neural radiance fields for view synthesis.  Commun. ACM, 65(1):99–106, 2022.
  * [59] Tomas Möller, Eric Haines, and Nathaniel Hoffman.  Real-time rendering, 3rd Edition.  Peters, 2008.
  * [60] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  ACM Trans. Graph., 41(4):102:1–102:15, 2022.
  * [61] Jacob Munkberg, Wenzheng Chen, Jon Hasselgren, Alex Evans, Tianchang Shen, Thomas Müller, Jun Gao, and Sanja Fidler.  Extracting triangular 3d models, materials, and lighting from images.  In CVPR, pages 8270–8280, 2022.
  * [62] Thanh Nguyen, Quoc Viet Hung Nguyen, Dung Tien Nguyen, Duc Thanh Nguyen, Thien Huynh-The, Saeid Nahavandi, Thanh Tam Nguyen, Quoc-Viet Pham, and Cuong M. Nguyen.  Deep learning for deepfakes creation and detection: A survey.  Comput. Vis. Image Underst., 223:103525, 2022.
  * [63] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada.  Neural articulated radiance field.  In ICCV, pages 5742–5752, 2021.
  * [64] Deng Pan, Lixian Sun, Rui Wang, Xingjian Zhang, and Richard O. Sinnott.  Deepfake detection through deep learning.  In BDCAT, pages 134–143, 2020.
  * [65] Jeong Joon Park, Peter Florence, Julian Straub, Richard A. Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In CVPR, pages 165–174, 2019.
  * [66] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B. Goldman, Steven M. Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In ICCV, pages 5845–5854, 2021.
  * [67] David A. Patterson, Joseph Gonzalez, Quoc V. Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David R. So, Maud Texier, and Jeff Dean.  Carbon emissions and large neural network training.  arXiv:2104.10350, 2021.
  * [68] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black.  Expressive body capture: 3d hands, face, and body from a single image.  In CVPR, pages 10975–10985, 2019.
  * [69] Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao.  Animatable neural radiance fields for modeling dynamic human bodies.  In ICCV, pages 14294–14303, 2021.
  * [70] Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou.  Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans.  In CVPR, pages 9054–9063, 2021.
  * [71] Sergey Prokudin, Michael J. Black, and Javier Romero.  Smplpix: Neural avatars from 3d human models.  In WACV, pages 1809–1818, 2021.
  * [72] Amit Raj, Julian Tanke, James Hays, Minh Vo, Carsten Stoll, and Christoph Lassner.  ANR: articulated neural rendering for virtual avatars.  In CVPR, pages 3722–3731, 2021.
  * [73] Nikhila Ravi, Jeremy Reizenstein, David Novotný, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari.  Accelerating 3d deep learning with pytorch3d.  arXiv:2007.08501, 2020.
  * [74] Javier Romero, Dimitrios Tzionas, and Michael J. Black.  Embodied hands: modeling and capturing hands and bodies together.  ACM Trans. Graph., 36(6):245:1–245:17, 2017.
  * [75] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Hao Li, and Angjoo Kanazawa.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In ICCV, pages 2304–2314, 2019.
  * [76] Shunsuke Saito, Tomas Simon, Jason M. Saragih, and Hanbyul Joo.  Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization.  In CVPR, pages 81–90, 2020.
  * [77] Shunsuke Saito, Jinlong Yang, Qianli Ma, and Michael J. Black.  Scanimate: Weakly supervised learning of skinned clothed avatar networks.  In CVPR, pages 2886–2897, 2021.
  * [78] Scott Schaefer, Jan Hakenberg, and Joe D. Warren.  Smooth subdivision of tetrahedral meshes.  In SGP, pages 147–154, 2004.
  * [79] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler.  Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis.  In NeurIPS, pages 6087–6101, 2021.
  * [80] Michael Stokes, Matthew Anderson, Srinivasan Chandrasekar, and Ricardo Motta.  A Standard Default Color Space for the Internet - sRGB, 1996.
  * [81] Shih-Yang Su, Frank Yu, Michael Zollhöfer, and Helge Rhodin.  A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose.  In NeurIPS, pages 12278–12291, 2021.
  * [82] Yu Sun, Qian Bao, Wu Liu, Yili Fu, Michael J. Black, and Tao Mei.  Monocular, one-stage, regression of multiple 3d people.  In ICCV, pages 11159–11168, 2021.
  * [83] Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Deferred neural rendering: image synthesis using neural textures.  ACM Trans. Graph., 38(4):66:1–66:12, 2019.
  * [84] Ingo Wald, Will Usher, Nathan Morrical, Laura Lediaev, and Valerio Pascucci.  RTX beyond ray tracing: Exploring the use of hardware ray tracing cores for tet-mesh point location.  In HPG, pages 7–13, 2019.
  * [85] Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance.  Microfacet models for refraction through rough surfaces.  In Proceedings of the Eurographics Symposium on Rendering Techniques, Grenoble, France, 2007, pages 195–206, 2007.
  * [86] Liao Wang, Ziyu Wang, Pei Lin, Yuheng Jiang, Xin Suo, Minye Wu, Lan Xu, and Jingyi Yu.  ibutter: Neural interactive bullet time generator for human free-viewpoint rendering.  In ACMMM, pages 4641–4650, 2021.
  * [87] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  In NeurIPS, pages 27171–27183, 2021.
  * [88] Shaofei Wang, Katja Schwarz, Andreas Geiger, and Siyu Tang.  ARAH: animatable volume rendering of articulated human sdfs.  In ECCV, pages 1–19, 2022.
  * [89] Chung-Yi Weng, Brian Curless, Pratul P. Srinivasan, Jonathan T. Barron, and Ira Kemelmacher-Shlizerman.  Humannerf: Free-viewpoint rendering of moving people from monocular video.  In CVPR, pages 16189–16199, 2022.
  * [90] Donglai Xiang, Timur M. Bagautdinov, Tuur Stuyck, Fabian Prada, Javier Romero, Weipeng Xu, Shunsuke Saito, Jingfan Guo, Breannan Smith, Takaaki Shiratori, Yaser Sheikh, Jessica K. Hodgins, and Chenglei Wu.  Dressing avatars: Deep photorealistic appearance for physically simulated clothing.  ACM Trans. Graph., 41(6):222:1–222:15, 2022.
  * [91] Donglai Xiang, Fabian Prada, Timur M. Bagautdinov, Weipeng Xu, Yuan Dong, He Wen, Jessica K. Hodgins, and Chenglei Wu.  Modeling clothing as a separate layer for an animatable human avatar.  ACM Trans. Graph., 40(6):199:1–199:15, 2021.
  * [92] Tianhan Xu, Yasuhiro Fujita, and Eiichi Matsumoto.  Surface-aligned neural radiance fields for controllable 3d human synthesis.  In CVPR, pages 15862–15871, 2022.
  * [93] Yu-Jie Yuan, Yang-Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao.  Nerf-editing: Geometry editing of neural radiance fields.  In CVPR, pages 18332–18343, 2022.
  * [94] Wang Zeng, Wanli Ouyang, Ping Luo, Wentao Liu, and Xiaogang Wang.  3d human mesh regression with dense correspondence.  In CVPR, pages 7052–7061, 2020.
  * [95] Hongwen Zhang, Yating Tian, Yuxiang Zhang, Mengcheng Li, Liang An, Zhenan Sun, and Yebin Liu.  Pymaf-x: Towards well-aligned full-body model regression from monocular images.  arXiv:2207.06400, 2022.
  * [96] Hongwen Zhang, Yating Tian, Xinchi Zhou, Wanli Ouyang, Yebin Liu, Limin Wang, and Zhenan Sun.  Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop.  In ICCV, pages 11426–11436, 2021.
  * [97] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely.  Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting.  In CVPR, pages 5453–5462, 2021.
  * [98] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.  Nerf++: Analyzing and improving neural radiance fields.  arXiv:2010.07492, 2020.
  * [99] Ruiqi Zhang and Jie Chen.  NDF: neural deformable fields for dynamic human modelling.  In ECCV, pages 37–52, 2022.
  * [100] Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul E. Debevec, William T. Freeman, and Jonathan T. Barron.  Nerfactor: Neural factorization of shape and reflectance under an unknown illumination.  ACM Trans. Graph., 40:237:1–237:18, 2021.
  * [101] Hao Zhao, Jinsong Zhang, Yu-Kun Lai, Zerong Zheng, Yingdi Xie, Yebin Liu, and Kun Li.  High-fidelity human avatars from a single RGB camera.  In CVPR, pages 15883–15892, 2022.
  * [102] Zerong Zheng, Han Huang, Tao Yu, Hongwen Zhang, Yandong Guo, and Yebin Liu.  Structured local radiance fields for human avatar modeling.  In CVPR, pages 15872–15882, 2022.

## Appendix

Thank you for reading our supplementary materials! Here we provide depth
descriptions of our method, including details about loss functions (Sec. 6),
image-based lighting (Sec. 7), and image-based lighting (Sec. 8). Then we
present additional ablations about training views (Sec. 9) and skinning module
(Sec. 9). Additional experimental results are illustrated in Sec. 10 and Sec.
12. We showcase application examples in Sec. 11. In the end, we discuss
limitations and social impact in Sec. 13. We strongly encourage our readers to
view the supplemental video for a more comprehensive visual perception.

##  6 Loss Functions

Our loss function $L=L_{\mathrm{img}}+L_{\mathrm{mask}}+L_{\mathrm{reg}}$ is
composed of three parts: an image loss $L_{\mathit{img}}$ using $\ell_{1}$
norm on tone mapped colors, and mask loss $L_{\mathrm{mask}}$ using squared
$\ell_{2}$, and regularization losses $L_{\mathrm{reg}}$ to improve the
quality of canonical geometry, materials, lights, and motion.

Image loss: our renderer utilizes physically-based shading to produce high-
dynamic range (HDR) images. Then the complex materials and environmental
lights are elaborately optimized. Thus our loss function requires a full range
of floating point values. We follow [27, 61, 26] to compute $\ell_{1}$ norm on
tone mapped colors. Specifically, we first transform linear radiance values
$i$ according to a tone-mapping operator $T(i)=\Gamma(\log(i+1))$, in which
$\Gamma(i)$ is a linear RGB to sRGB transformation function [80]:

$\displaystyle\Gamma(i)$ $\displaystyle=\begin{cases}12.92i&i\leq 0.0031308\\\
(1+a)i^{1/2.4}-a&i>0.0031308\end{cases}$ (11) $\displaystyle a$
$\displaystyle=0.055,$

Mask loss: The renderer [43] renders both the shaded images and the
corresponding rasterization masks in a differentiable manner. Therefore, we
compute the $\ell_{2}$ norm between the masks and the preprocessed mattings
(in both ZJU-MoCap and H36M benchmarks, we use the provided preprocessed
subject masks from [70, 22]), akin to the traditional shape-from-silhouette
[55] technique. The mask loss is parallel with the image loss, yet facilitates
the course of shading optimization by making shape convergence super fast in
about a hundred training steps.

Regularizers: We need various priors to encourage the optimization to converge
at a place where the geometry, materials, and lighting are well separated and
smooth enough [61, 26]. Therefore, we choose to minimize regularization during
training.

We introduce smoothness to PBR materials in terms of albedo $\mathbf{k}_{d}$,
specular parameters $\mathbf{k}_{\mathrm{orm}}$, and surface geometry nomral
$\mathbf{n}$ as following:

$L_{\mathbf{k}}=\frac{1}{\left|\mathbf{x}_{\text{surf
}}\right|}\sum_{\mathbf{x}_{\text{surf
}}}\left|\mathbf{k}\left(\mathbf{x}_{\text{surf
}}\right)-\mathbf{k}\left(\mathbf{x}_{\text{surf
}}+\mathbf{\epsilon}\right)\right|,$ (12)

where ${\left|\mathbf{x}_{\text{surf }}\right|}$ is a surface point on the
surface in canonical space and
$\mathbf{\epsilon}\sim\mathcal{N}(0,\sigma\\!\\!=\\!\\!0.01)$ is a small
random offset. We regularize the geometry normal on the surface of the
canonical mesh derived from the SDF field for a seek of a smoother surface and
avoidance of holes in the surface.

We regularize light by assuming the neutral spectrum in the real world.
Specifically, given the per-channel average radiance densities $\bar{c_{i}}$,
we penalize the color shifts as:

$L_{\text{light
}}=\frac{1}{3}\sum_{i=0}^{3}\left|{c_{i}}-\frac{1}{3}\sum_{i=0}^{3}{c_{i}}\right|,$
(13)

To encourage a watertight surface and reduce floating meshes both inside and
outside the subject, we impose regularizations on the SDF field as:

$\displaystyle L_{\mathrm{sdf}}=\sum_{i,j\in\mathrm{S}_{e}}$ $\displaystyle
H\left(\sigma\left(s_{i}\right),\operatorname{sign}\left(s_{j}\right)\right)$
(14)
$\displaystyle+H\left(\sigma\left(s_{j}\right),\operatorname{sign}\left(s_{i}\right)\right),$

where $\mathrm{S}_{e}$ is the set of all vertex along their edges in which the
signs of the SDF values are different (_i.e_.,
$\mathrm{sign}(s_{i})\neq\mathrm{sign}(s_{j})$). To remove the floating meshes
outside the surface, we impose an additional loss. For a triangle surface $f$
extracted by marching tetrahedra, if $f$ is invisible, we encourage its SDF
values to be positive as:

$L_{\mathrm{invis}}=\sum_{i\in\mathrm{S}_{\mathrm{invis}}}H(\sigma\left(s_{i}\right),1).$
(15)

We weigh the above terms and use the loss for all our experiments:

$\displaystyle L$ $\displaystyle=L_{\text{image}}+L_{\text{mask}}$ (16)
$\displaystyle+\underbrace{\lambda_{\mathbf{k}_{\mathrm{d}}}}_{=0.03}L_{\mathbf{k}_{\mathrm{d}}}+\underbrace{\lambda_{\mathbf{k}_{\mathrm{orm}}}}_{=0.05}L_{\mathbf{k}_{\mathrm{orm}}}+\underbrace{\lambda_{\mathbf{n}}}_{=0.025}L_{\mathbf{n}}$
$\displaystyle+\underbrace{\lambda_{\text{light}}}_{=0.005}L_{\text{light}}+\underbrace{\lambda_{\text{sdf}}}_{=0.02}L_{\text{sdf}}+\underbrace{\lambda_{\text{invis}}}_{=0.01}L_{\text{invis}}.$

##  7 Image-based Lighting

The split sum shading model is widely used in real-time rendering [59], giving
both realism and efficiency against spherical Gaussians (SG) and spherical
harmonics (SH) [10, 13, 97]. We use a differentiable split sum [35] shading
model to approximate rendering equation [32] for image-based environment light
learning as [61]:

$\begin{split}L\left(\omega_{o}\right)&\approx\int_{\Omega}f\left(\omega_{i},\omega_{o}\right)\left(\omega_{i}\cdot\mathbf{n}\right)d\omega_{i}\\\
&\quad\int_{\Omega}L_{i}\left(\omega_{i}\right)D\left(\omega_{i},\omega_{o}\right)\left(\omega_{i}\cdot\mathbf{n}\right)d\omega_{i}.\end{split}$
(17)

where $D$ is the GGX normal distribution function (NDF) [85] in a Cook-
Torrance microfacet specular shading model [16]. The first term contributes to
the specular BSDF wrt. a solid white environment light, which depends solely
on the roughness $r$ of the BSDF and the light-surface angles
$\cos\theta=\omega_{i}\cdot\mathbf{n}$. The second term contributes to the
integral of the incoming radiance with the GGX normal distribution function,
$D$. Both terms can be pre-computed and filtered to reduce computation [35].

The training parameters are texels of a cube light map whose resolution is
$6\times 512\times 512$. The pre-integrated lighting for the least roughness
values is derived from the base level, and multiple smaller mip levels are
constructed from it [35]. Each mip-map is filtered by average-pooling the base
level of the current resolution and is convolved with the GGX normal
distribution function. The per mip-level filter bounds are pre-computed as
well. We leverage a PyTorch implementation with CUDA extensions from [61].
Moreover, a cube map is created to represent the diffuse lighting in a low
resolution, akin to the filtered specular probe. It shares the same
optimizable parameters and is average-pooled to the mip level with $r=1$
roughness. The pre-filtering only involves the first term in Eq. 17.

##  8 Implementation Details

SDF network. We parametrize the SDF field with an MLP to increase surface
water-tight and smoothness. We choose the MLP architecture from [58], which
consists of 6 frequency bands for positional encoding, and 8 linear layers,
each having 256 neurons, followed by ReLU activations. We implicitly
regularize the smoothness by increasing the Lipschitz property in the SDF
field[48].

Material network. The material model is a small MLP with hash-encoding [60] as
the materials query is computationally extensive. The MLP consists of two
linear layers, each having 32 neurons, followed by ReLU activations. The hash-
encoding has a spatial resolution of 4096 and the rest configures are the same
as [61]. To reduce computation, we predict all material channels at once with
one backbone network. Besides, we introduce inductive bias of materials of
clothed humans in the real world, by providing minimum and maximum values for
each materials channel. We follow [100] to limit the albedo
$\mathbf{k}_{d}\in[0.03,0.8]$, and the roughness $\mathbf{k}_{r}\in[0.08,1]$.
The texels in the environment light are randomly initialized between
$[0.25,0.75]$.

Motion networks. For the motion module, we use the same MLP architecture as
[15, 88], which is similar to our SDF MLP. To resolve the problem where the
training pose variation is too limited for skinning field learning (_e.g_.,
self-rotation video without any limbs movements), we initialize the MLP with
the pre-trained skinning model provided by [88], and impose $\ell_{2}$ norm
for the skinning weights logits between our predictions and the ground truth
from SMPL [53]. We ablate the design choices in Sec. 9. For the non-rigid
modeling, we use another 4-layer ReLU MLP with a 4-frequency-band positional
encoding. We also progressively anneal its encoding for 5k iterations as [66].
The weights of the last layer are initialized with a uniform distribution
$\mathcal{U}(-10^{-5},10^{-5})$, _i.e_. initializing the non-rigid offsets to
be close to zero and not interfering with the major optimizations of geometry
and materials.

Optimization. We use Adam [39] as our default optimizer. We optimize the
subject for 5k steps for 1024$\times$1024 images or 10k steps 512$\times$512
images. We disable the perturbed normal map during optimization as it leads to
SDF collapsing abruptly at a certain step (_i.e_., all SDF values are positive
or negative where marching tetrahedra fails). The optimization process takes
about an hour on a single NVIDIA GTX3090 GPU. The indicative results with
plausible quality appear after a few minutes, which is quite faster than our
counterparts [70, 69, 88, 92]. Such superior efficiency could largely
accelerate downstream applications. The training visualization is presented in
the supplemental video.

Tetrahedra grids. We start with a tetrahedra grid with $128\times 128$
resolution, including 192k tetrahedra and 37k vertices. Each tetrahedron can
produce at most 2 triangles by marching tetrahedra algorithm [61, 79, 19]. To
increase the resolution of the tetrahedra grid, we subdivide the grid at the
500th step. To avoid the out-of-memory problem caused by the vast amount of
floating meshes in the void space at the beginning of training, we pre-train
the SDF network to match a visual hull of humans in canonical space. The hull
could be derived from either the skeleton capsules or the SMPL [53] mesh. Note
that we only pre-train for 500 iterations, which leads to a very coarse shape
akin to the visual hull rather than the given ground truth mesh. The
initialized mesh is presented in the training visualization part of the
supplemental video.

##  9 Additional Ablations

Number of Training Views. Table 4 and Figure 9 show that giving one camera of
view degrades the overall reconstruction quality, and multi-view consistency
improves the final results. The model can aggregate multi-view information for
better shading optimization, thus leading to clearer surface materials.

Table 4: Ablation results of training views on the ZJU-MoCap 313 subject. | Training pose | Novel pose
---|---|---
ZJU-MoCap 313 | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
1 view | 24.39 | 0.913 | 21.45 | 0.869
2 views | 28.06 | 0.945 | 22.81 | 0.888
3 views | 28.50 | 0.956 | 23.17 | 0.894
4 views | 29.04 | 0.961 | 23.20 | 0.896
Figure 9: Ablation study of training views on the ZJU-MoCap 313 subject.

The effect of Skinning Module Design Table 5-6 and Figure 10-12 reveal that
the initialization with pre-trained skinning net and the regularization on
surface skinning improve the overall reconstruction quality. The
initialization provides skinning prior which helps to speed up geometry
convergence. From Figure 10-11, the geometry details improve with the
initialization under the same training time.

The regularization on surface skinning prevents geometry degradation. Figure
12 indicates that our model can not learn correct canonical geometry without
the initialization and the regularization. The mesh distortion is reduced with
the regularization.

Table 5:  The ablation on skinning module of H36M S9 dataset.

| Training Pose | Novel Pose
---|---|---
H36M S9 | PSNR | SSIM | PSNR | SSIM
w/o skinning init. & reg. | 24.88 | 0.905 | 21.97 | 0.851
w/ skinning intialization | 26.28 | 0.926 | 24.47 | 0.897
w/ skinning regularization | 26.24 | 0.925 | 24.34 | 0.896
Full | 26.29 | 0.926 | 24.53 | 0.899

Table 6:  The ablation on skinning module of ZJU-MoCap 313 dataset.

| Training Pose | Novel Pose
---|---|---
ZJU-MoCap 313 | PSNR | SSIM | PSNR | SSIM
w/o skinning init. & reg. | 27.46 | 0.949 | 20.31 | 0.831
w/ skinning intialization | 28.82 | 0.958 | 23.08 | 0.893
w/ skinning regularization | 28.80 | 0.959 | 23.14 | 0.895
Full | 29.05 | 0.961 | 23.27 | 0.897

Figure 10: Ablation study of the skinning module on the H36M S9 subject.
Figure 11: Ablation study of the skinning module on the ZJU-MoCap 313 subject.
Figure 12: Ablation study of the skinning module on the ZJU-MoCap 313 subject.

Effect of SDF Network The MLP parametrization of the SDF field keeps our
surface both water-tight and smooth, as shown in Figure 13.

Figure 13: Ablation study of SDF field parametrization.

##  10 More Comparisons

We present full quantitative comparisons in Table 9, Table 7, Table 10, and
Table 8. Meanwhile, more qualitative comparisons are illustrated in Figure 14,
Figure 15, and Figure 16.

Figure 14: Qualitative results of novel pose synthesis on H36M dataset. Zoom
in for a better view. Figure 15: Qualitative results of novel pose synthesis
on ZJU-MoCap dataset. “N/A” denotes nothing to render due to no convergence.
Zoom in for a better view. Figure 16: Qualitative results of novel pose
synthesis on H36M and ZJU-MoCap datasets with the full models. Zoom in for a
better view. Table 7: Quantitative results of training pose novel view
synthesis of H36M dataset.

| Training pose
---|---
PSNR | SSIM
NB | SA-NeRF | Ani-NeRF | ARAH | Ours | NB | SA-NeRF | Ani-NeRF | ARAH | Ours
S1 | 22.87 | 23.71 | 22.05 | 24.45 | 24.56 | 0.897 | 0.915 | 0.888 | 0.919 | 0.919
S5 | 24.60 | 24.78 | 23.27 | 24.54 | 24.51 | 0.917 | 0.909 | 0.892 | 0.918 | 0.920
S6 | 22.82 | 23.22 | 21.13 | 24.61 | 24.55 | 0.888 | 0.881 | 0.854 | 0.903 | 0.902
S7 | 23.17 | 22.59 | 22.50 | 24.31 | 24.05 | 0.914 | 0.905 | 0.890 | 0.919 | 0.916
S8 | 21.72 | 24.55 | 22.75 | 24.02 | 23.94 | 0.894 | 0.922 | 0.898 | 0.921 | 0.920
S9 | 24.28 | 25.31 | 24.72 | 26.20 | 25.99 | 0.910 | 0.913 | 0.908 | 0.924 | 0.919
S11 | 23.70 | 25.83 | 24.55 | 25.43 | 25.48 | 0.896 | 0.917 | 0.902 | 0.921 | 0.915
Average | 23.31 | 24.28 | 23.00 | 24.79 | 24.72 | 0.902 | 0.909 | 0.890 | 0.918 | 0.916

Table 8: Quantitative results of unseen pose novel view synthesis of H36M
dataset.

| Unseen pose
---|---
PSNR | SSIM
NB | SA-NeRF | Ani-NeRF | ARAH | Ours | NB | SA-NeRF | Ani-NeRF | ARAH | Ours
S1 | 21.93 | 22.67 | 19.96 | 23.08 | 23.72 | 0.873 | 0.890 | 0.855 | 0.899 | 0.904
S5 | 23.33 | 23.27 | 20.02 | 22.79 | 23.13 | 0.893 | 0.881 | 0.840 | 0.890 | 0.898
S6 | 23.26 | 23.23 | 23.64 | 24.04 | 24.17 | 0.888 | 0.888 | 0.882 | 0.900 | 0.903
S7 | 22.40 | 22.51 | 21.76 | 22.58 | 22.72 | 0.888 | 0.898 | 0.869 | 0.891 | 0.889
S8 | 20.78 | 23.06 | 21.63 | 22.34 | 22.71 | 0.872 | 0.904 | 0.877 | 0.896 | 0.902
S9 | 22.87 | 23.84 | 21.95 | 24.36 | 24.54 | 0.880 | 0.889 | 0.871 | 0.894 | 0.895
S11 | 23.54 | 24.19 | 22.55 | 24.78 | 24.47 | 0.879 | 0.891 | 0.875 | 0.902 | 0.900
Average | 22.59 | 23.25 | 21.64 | 23.42 | 23.64 | 0.882 | 0.892 | 0.867 | 0.896 | 0.899

Table 9: Quantitative results of training pose novel view synthesis of ZJU-
MoCap dataset.

| Training pose
---|---
PSNR | SSIM
NB | SA-NeRF | Ani-NeRF | ARAH | Ours | NB | SA-NeRF | Ani-NeRF | ARAH | Ours
Twirl(313) | 30.56 | 31.32 | 29.80 | 31.60 | 29.67 | 0.971 | 0.974 | 0.963 | 0.973 | 0.947
Taichi(315) | 27.24 | 27.25 | 23.10 | 27.00 | 24.21 | 0.962 | 0.962 | 0.917 | 0.965 | 0.919
Swing1(392) | 29.44 | 29.29 | 28.00 | 29.50 | 27.58 | 0.946 | 0.946 | 0.931 | 0.948 | 0.899
Swing2(393) | 28.44 | 28.76 | 26.10 | 27.70 | 25.91 | 0.940 | 0.941 | 0.916 | 0.940 | 0.890
Swing3(394) | 27.58 | 27.50 | 27.50 | 28.90 | 27.67 | 0.939 | 0.938 | 0.924 | 0.945 | 0.902
Warmup(377) | 27.64 | 27.67 | 24.20 | 27.80 | 26.69 | 0.951 | 0.954 | 0.925 | 0.956 | 0.926
Punch1(386) | 28.60 | 28.81 | 25.60 | 29.20 | 27.65 | 0.931 | 0.931 | 0.878 | 0.934 | 0.881
Punch2(387) | 25.79 | 26.08 | 25.40 | 27.00 | 25.68 | 0.928 | 0.929 | 0.926 | 0.945 | 0.908
Kick(390) | 27.59 | 27.77 | 26.00 | 27.90 | 24.08 | 0.926 | 0.927 | 0.912 | 0.929 | 0.840
Average | 28.10 | 26.19 | 28.27 | 28.51 | 26.57 | 0.944 | 0.945 | 0.921 | 0.948 | 0.901

Table 10: Quantitative results of unseen pose novel view synthesis of ZJU-
MoCap dataset.

| Unseen pose
---|---
PSNR | SSIM
NB | SA-NeRF | Ani-NeRF | ARAH | Ours | NB | SA-NeRF | Ani-NeRF | ARAH | Ours
Twirl(313) | 23.95 | 24.33 | 22.80 | 24.40 | 23.63 | 0.905 | 0.908 | 0.863 | 0.914 | 0.878
Taichi(315) | 19.56 | 19.87 | 18.47 | 20.00 | 20.42 | 0.852 | 0.863 | 0.795 | 0.881 | 0.850
Swing1(392) | 25.76 | 26.27 | 18.44 | 26.20 | 25.49 | 0.909 | 0.927 | 0.670 | 0.927 | 0.883
Swing2(393) | 23.80 | 24.96 | 21.87 | 24.40 | 24.31 | 0.878 | 0.900 | 0.836 | 0.915 | 0.883
Swing3(394) | 23.25 | 24.24 | 17.69 | 25.20 | 24.72 | 0.893 | 0.908 | 0.792 | 0.908 | 0.870
Warmup(377) | 23.91 | 25.34 | 23.28 | 25.50 | 24.80 | 0.909 | 0.928 | 0.901 | 0.933 | 0.894
Punch1(386) | 25.68 | 27.30 | 25.55 | 27.00 | 26.24 | 0.881 | 0.905 | 0.872 | 0.910 | 0.853
Punch2(387) | 21.60 | 23.08 | 21.92 | 24.20 | 24.06 | 0.870 | 0.890 | 0.838 | 0.917 | 0.889
Kick(390) | 23.90 | 24.43 | 23.90 | 24.80 | 25.79 | 0.870 | 0.889 | 0.887 | 0.896 | 0.873
Average | 23.49 | 24.42 | 21.55 | 24.63 | 24.38 | 0.885 | 0.902 | 0.828 | 0.911 | 0.875

##  11 Applications

We showcase relighting, texture editing, and novel poses synthesis on AIST
dataset [46] in Figure 17, Figure 18, and Figure 19 separately. All the above
applications are presented in the supplemental video.

Figure 17: Relighting visualization. Zoom in for a better view. We strongly
encourage our readers to view the supplemental video for a more comprehensive
visual perception. Figure 18: Texture editing visualization. Zoom in for a
better view. We strongly encourage our readers to view the supplemental video
for a more comprehensive visual perception. Figure 19: Extreme pose
visualization. Zoom in for a better view. We strongly encourage our readers to
view the supplemental video for a more comprehensive visual perception.

##  12 Mesh Visualizations

We visualize the canonical mesh and present the number of faces of each mesh
in Figure 20 and Figure 12. Note that the number of faces for each mesh is
quite small. Though increasing the resolution of tetrahedra grids may improve
the details of both geometry and materials, we do not conduct this experiment
for it is orthogonal to our technical contributions.

Figure 20: Mesh visualization on the H36M dataset. Zoom in for a better view.
Figure 21: Mesh visualization on the ZJU-MoCap dataset. Zoom in for a better
view.

##  13 Limitations and Further Discussions

Our method is biased for shape-material ambiguity [88, 45, 81, 63, 69, 87,
98]. Taking subject 315 from ZJU-MoCap as an example, the strips in the
T-shirt are modeled as ravines on the surface. The high contrast color in the
cloth surface makes our model biased for shape modeling. That might be
resolved by introducing additional surface regularizers or pre-defined
parameters for the materials.

Need foreground mask to enable the mesh optimization, akin to shape-from-
silhouette. One future direction might be equipping our method with the
ability to separate foreground and background automatically [31, 24]. It is
also promising to model the background simultaneously during foreground
subject optimization [31, 24], which eliminates the requirement of foreground
mask processing.

Our method can digitize humans from visual footage, which may involve avatar
misuse without the permission of the owners. Methods like implicit adversarial
watermarks [11, 47] that disable the neural nets inference could assist the
video creation to protect their portrait rights. Another concern is the deep
fake misuse [62], which corrupts the identity in the visual footage rendered
by our model. Methods like deep fake detection [64] could help to discover and
prevent deep fake creations. Besides, our method involves training with GPUs,
which leads to carbon emissions and increasing global warming [67].


* Khadkikar  _et al._ (2021) S. Khadkikar, A. R. Raduta, M. Oertel, and A. Sedrakian, Phys. Rev. C 103, 055811 (2021), arXiv:2102.00988 [astro-ph.HE] .
  * Alford  _et al._ (2018) M. G. Alford, L. Bovard, M. Hanauske, L. Rezzolla, and K. Schwenzer, Phys. Rev. Lett. 120, 041101 (2018), arXiv:1707.09475 [gr-qc] .
  * Alford  _et al._ (2019) M. Alford, A. Harutyunyan, and A. Sedrakian, Phys. Rev. D 100, 103021 (2019), arXiv:1907.04192 [astro-ph.HE] .
  * Routray  _et al._ (2020) T. R. Routray, S. P. Pattnaik, C. Gonzalez-Boquera, X. Viñas, M. Centelles, and B. Behera, (2020), arXiv:2006.15430 [nucl-th] .
  * Parotto  _et al._ (2020) P. Parotto, M. Bluhm, D. Mroczek, M. Nahrgang, J. Noronha-Hostler, K. Rajagopal, C. Ratti, T. Schäfer, and M. Stephanov, Phys. Rev. C 101, 034901 (2020), arXiv:1805.05249 [hep-ph] .
  * Grefa  _et al._ (2021) J. Grefa, J. Noronha, J. Noronha-Hostler, I. Portillo, C. Ratti, and R. Rougemont, (2021), arXiv:2102.12042 [nucl-th] .
  * Stafford  _et al._ (2021) J. M. Stafford, D. Mroczek, A. R. Nava Acuna, J. Noronha-Hostler, P. Parotto, D. R. P. Price, and C. Ratti, (2021), arXiv:2103.08146 [hep-ph] .
  * Aoki  _et al._ (2006) Y. Aoki, G. Endrodi, Z. Fodor, S. D. Katz, and K. K. Szabo, Nature 443, 675 (2006), arXiv:hep-lat/0611014 .
  * Borsányi  _et al._ (2021) S. Borsányi, Z. Fodor, J. N. Guenther, R. Kara, S. D. Katz, P. Parotto, A. Pásztor, C. Ratti, and K. K. Szabó, (2021), arXiv:2102.06660 [hep-lat] .
  * Asakawa and Yazaki (1989) M. Asakawa and K. Yazaki, Nucl. Phys. A 504, 668 (1989).
  * Berges and Rajagopal (1999) J. Berges and K. Rajagopal, Nucl. Phys. B 538, 215 (1999), arXiv:hep-ph/9804233 .
  * Halasz  _et al._ (1998) A. M. Halasz, A. D. Jackson, R. E. Shrock, M. A. Stephanov, and J. J. M. Verbaarschot, Phys. Rev. D 58, 096007 (1998), arXiv:hep-ph/9804290 .
  * Adam  _et al._ (2021) J. Adam _et al._ (STAR), Phys. Rev. Lett. 126, 092301 (2021), arXiv:2001.02852 [nucl-ex] .
  * Dexheimer  _et al._ (2021b) V. Dexheimer, J. Noronha, J. Noronha-Hostler, C. Ratti, and N. Yunes, J. Phys. G 48, 073001 (2021b), arXiv:2010.08834 [nucl-th] .
  * Ratti (2018) C. Ratti, Rept. Prog. Phys. 81, 084301 (2018), arXiv:1804.07810 [hep-lat] .
  * Bzdak  _et al._ (2020) A. Bzdak, S. Esumi, V. Koch, J. Liao, M. Stephanov, and N. Xu, Phys. Rept. 853, 1 (2020), arXiv:1906.00936 [nucl-th] .
  * Monnai  _et al._ (2021) A. Monnai, B. Schenke, and C. Shen, Int. J. Mod. Phys. A 36, 2130007 (2021), arXiv:2101.11591 [nucl-th] .
  * Gerlach (1968) U. H. Gerlach, Phys. Rev. 172, 1325 (1968).
  * Kampfer (1981) B. Kampfer, J. Phys. A 14, L471 (1981).
  * Han and Prakash (2020) S. Han and M. Prakash, Astrophys. J. 899, 164 (2020), arXiv:2006.02207 [astro-ph.HE] .
  * Pang  _et al._ (2020) P. T. H. Pang, T. Dietrich, I. Tews, and C. Van Den Broeck, Phys. Rev. Res. 2, 033514 (2020), arXiv:2006.14936 [astro-ph.HE] .
  * Jakobus  _et al._ (2021) P. Jakobus, A. Motornenko, R. O. Gomes, J. Steinheimer, and H. Stoecker, Eur. Phys. J. C 81, 41 (2021), arXiv:2004.07026 [nucl-th] .
  * Glendenning (1992) N. K. Glendenning, Phys. Rev. D 46, 1274 (1992).
  * Baym (1979) G. Baym, Physica A 96, 131 (1979).
  * Dexheimer  _et al._ (2015) V. Dexheimer, R. Negreiros, and S. Schramm, Phys. Rev. C 91, 055808 (2015), arXiv:1411.4623 [astro-ph.HE] .
  * Dutra  _et al._ (2016) M. Dutra, O. Lourenço, and D. P. Menezes, Phys. Rev. C 93, 025806 (2016), [Erratum: Phys.Rev.C 94, 049901 (2016)], arXiv:1510.02060 [astro-ph.HE] .
  * McLerran and Reddy (2019) L. McLerran and S. Reddy, Phys. Rev. Lett. 122, 122701 (2019), arXiv:1811.12503 [nucl-th] .
  * Alford and Sedrakian (2017) M. G. Alford and A. Sedrakian, Phys. Rev. Lett. 119, 161104 (2017), arXiv:1706.01592 [astro-ph.HE] .
  * Zacchi  _et al._ (2016) A. Zacchi, M. Hanauske, and J. Schaffner-Bielich, Phys. Rev. D 93, 065011 (2016), arXiv:1510.00180 [nucl-th] .
  * Alvarez-Castillo  _et al._ (2019) D. E. Alvarez-Castillo, D. B. Blaschke, A. G. Grunfeld, and V. P. Pagura, Phys. Rev. D 99, 063010 (2019), arXiv:1805.04105 [hep-ph] .
  * Li  _et al._ (2020b) J. J. Li, A. Sedrakian, and M. Alford, Phys. Rev. D 101, 063022 (2020b), arXiv:1911.00276 [astro-ph.HE] .
  * Wang  _et al._ (2019) Q.-w. Wang, C. Shi, Y. Yan, and H.-S. Zong, (2019), arXiv:1912.02312 [hep-ph] .
  * Bitaghsir Fadafan  _et al._ (2020) K. Bitaghsir Fadafan, J. Cruz Rojas, and N. Evans, Phys. Rev. D 101, 126005 (2020), arXiv:1911.12705 [hep-ph] .
  * Xia  _et al._ (2021) C. Xia, Z. Zhu, X. Zhou, and A. Li, Chin. Phys. C 45, 055104 (2021), arXiv:1906.00826 [nucl-th] .
  * Yazdizadeh and Bordbar (2019) T. Yazdizadeh and G. H. Bordbar, Iran. J. Sci. Technol. A 43, 2691 (2019), arXiv:1906.00175 [nucl-th] .
  * Shahrbaf  _et al._ (2020) M. Shahrbaf, D. Blaschke, A. G. Grunfeld, and H. R. Moshfegh, Phys. Rev. C 101, 025807 (2020), arXiv:1908.04740 [nucl-th] .
  * Zacchi and Schaffner-Bielich (2019) A. Zacchi and J. Schaffner-Bielich, Phys. Rev. D 100, 123024 (2019), arXiv:1909.12071 [nucl-th] .
  * Zhao and Lattimer (2020) T. Zhao and J. M. Lattimer, Phys. Rev. D 102, 023021 (2020), arXiv:2004.08293 [astro-ph.HE] .
  * Lopes and Menezes (2021) L. L. Lopes and D. P. Menezes, Nucl. Phys. A 1009, 122171 (2021), arXiv:2004.07909 [astro-ph.HE] .
  * Blaschke  _et al._ (2020) D. Blaschke, H. Grigorian, and G. Röpke, Particles 3, 477 (2020), arXiv:2005.10218 [nucl-th] .
  * Duarte  _et al._ (2020) D. C. Duarte, S. Hernandez-Ortiz, and K. S. Jeong, Phys. Rev. C 102, 025203 (2020), arXiv:2003.02362 [nucl-th] .
  * Rho (2020) M. Rho, (2020), arXiv:2004.09082 [nucl-th] .
  * Marczenko (2020) M. Marczenko, Eur. Phys. J. ST 229, 3651 (2020), arXiv:2005.14535 [nucl-th] .
  * Minamikawa  _et al._ (2021) T. Minamikawa, T. Kojo, and M. Harada, Phys. Rev. C 103, 045205 (2021), arXiv:2011.13684 [nucl-th] .
  * Hippert  _et al._ (2021) M. Hippert, E. S. Fraga, and J. Noronha, (2021), arXiv:2105.04535 [nucl-th] .
  * Pisarski (2021) R. D. Pisarski, Phys. Rev. D 103, L071504 (2021), arXiv:2101.05813 [nucl-th] .
  * Sen and Sivertsen (2020) S. Sen and L. Sivertsen, (2020), arXiv:2011.04681 [astro-ph.HE] .
  * Stone  _et al._ (2021a) J. R. Stone, V. Dexheimer, P. A. M. Guichon, A. W. Thomas, and S. Typel, Mon. Not. Roy. Astron. Soc. 502, 3476 (2021a), arXiv:1906.11100 [nucl-th] .
  * Kapusta and Welle (2021) J. I. Kapusta and T. Welle, (2021), arXiv:2103.16633 [nucl-th] .
  * Somasundaram and Margueron (2021) R. Somasundaram and J. Margueron, (2021), arXiv:2104.13612 [astro-ph.HE] .
  * Motornenko  _et al._ (2020) A. Motornenko, J. Steinheimer, V. Vovchenko, S. Schramm, and H. Stoecker, Phys. Rev. C 101, 034904 (2020), arXiv:1905.00866 [hep-ph] .
  * Baym  _et al._ (2019) G. Baym, S. Furusawa, T. Hatsuda, T. Kojo, and H. Togashi, Astrophys. J. 885, 42 (2019), arXiv:1903.08963 [astro-ph.HE] .
  * Sen and Warrington (2021) S. Sen and N. C. Warrington, Nucl. Phys. A 1006, 122059 (2021), arXiv:2002.11133 [nucl-th] .
  * Abbott  _et al._ (2017) B. P. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. Lett. 119, 161101 (2017), arXiv:1710.05832 [gr-qc] .
  * Riley  _et al._ (2019) T. E. Riley _et al._ , Astrophys. J. Lett. 887, L21 (2019), arXiv:1912.05702 [astro-ph.HE] .
  * Miller  _et al._ (2019) M. C. Miller _et al._ , Astrophys. J. Lett. 887, L24 (2019), arXiv:1912.05705 [astro-ph.HE] .
  * Jayasinghe  _et al._ (2021) T. Jayasinghe _et al._ ,  (2021), 10.1093/mnras/stab907, arXiv:2101.02212 [astro-ph.SR] .
  * Baym  _et al._ (2018) G. Baym, T. Hatsuda, T. Kojo, P. D. Powell, Y. Song, and T. Takatsuka, Rept. Prog. Phys. 81, 056902 (2018), arXiv:1707.04966 [astro-ph.HE] .
  * Rather  _et al._ (2021a) I. A. Rather, A. A. Usmani, and S. K. Patra, Nucl. Phys. A 1010, 122189 (2021a), arXiv:2009.12613 [nucl-th] .
  * Ferreira and Providência (2020) M. Ferreira and C. Providência, Phys. Rev. D 102, 103003 (2020), arXiv:2010.05588 [nucl-th] .
  * Riley  _et al._ (2021) T. E. Riley _et al._ , (2021), arXiv:2105.06980 [astro-ph.HE] .
  * Miller  _et al._ (2021) M. C. Miller _et al._ , (2021), arXiv:2105.06979 [astro-ph.HE] .
  * Yagi and Yunes (2017a) K. Yagi and N. Yunes, Phys. Rept. 681, 1 (2017a), arXiv:1608.02582 [gr-qc] .
  * Martinon  _et al._ (2014) G. Martinon, A. Maselli, L. Gualtieri, and V. Ferrari, Phys. Rev. D 90, 064026 (2014), arXiv:1406.7661 [gr-qc] .
  * Alford  _et al._ (2017) M. G. Alford, S. P. Harris, and P. S. Sachdeva, Astrophys. J. 847, 109 (2017), arXiv:1705.09880 [astro-ph.HE] .
  * Pereira  _et al._ (2018) J. P. Pereira, C. V. Flores, and G. Lugones, Astrophys. J. 860, 12 (2018), arXiv:1706.09371 [gr-qc] .
  * Alford  _et al._ (2015a) M. G. Alford, S. Han, and K. Schwenzer, Phys. Rev. C 91, 055804 (2015a), arXiv:1404.5279 [astro-ph.SR] .
  * Abbott  _et al._ (2018) B. P. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. Lett. 121, 161101 (2018), arXiv:1805.11581 [gr-qc] .
  * Landry and Essick (2019) P. Landry and R. Essick, Phys. Rev. D 99, 084049 (2019), arXiv:1811.12529 [gr-qc] .
  * Essick  _et al._ (2020) R. Essick, P. Landry, and D. E. Holz, Phys. Rev. D 101, 063007 (2020), arXiv:1910.09740 [astro-ph.HE] .
  * Landry  _et al._ (2020) P. Landry, R. Essick, and K. Chatziioannou, Phys. Rev. D 101, 123007 (2020), arXiv:2003.04880 [astro-ph.HE] .
  * Yagi and Yunes (2016) K. Yagi and N. Yunes, Class. Quant. Grav. 33, 13LT01 (2016), arXiv:1512.02639 [gr-qc] .
  * Yagi and Yunes (2017b) K. Yagi and N. Yunes, Class. Quant. Grav. 34, 015006 (2017b), arXiv:1608.06187 [gr-qc] .
  * Kurkela  _et al._ (2010) A. Kurkela, P. Romatschke, and A. Vuorinen, Phys. Rev. D 81, 105021 (2010), arXiv:0912.1856 [hep-ph] .
  * Kurkela  _et al._ (2014) A. Kurkela, E. S. Fraga, J. Schaffner-Bielich, and A. Vuorinen, Astrophys. J. 789, 127 (2014), arXiv:1402.6618 [astro-ph.HE] .
  * Hebeler  _et al._ (2013) K. Hebeler, J. M. Lattimer, C. J. Pethick, and A. Schwenk, Astrophys. J. 773, 11 (2013), arXiv:1303.4662 [astro-ph.SR] .
  * Dutra  _et al._ (2012) M. Dutra, O. Lourenco, J. S. Sa Martins, A. Delfino, J. R. Stone, and P. D. Stevenson, Phys. Rev. C 85, 035201 (2012), arXiv:1202.3902 [nucl-th] .
  * Tsang  _et al._ (2012) M. B. Tsang _et al._ , Phys. Rev. C 86, 015803 (2012), arXiv:1204.0466 [nucl-ex] .
  * Fattoyev  _et al._ (2013) F. J. Fattoyev, J. Carvajal, W. G. Newton, and B.-A. Li, Phys. Rev. C 87, 015806 (2013), arXiv:1210.3402 [nucl-th] .
  * Weissenborn  _et al._ (2012) S. Weissenborn, D. Chatterjee, and J. Schaffner-Bielich, Nucl. Phys. A 881, 62 (2012), arXiv:1111.6049 [astro-ph.HE] .
  * Li  _et al._ (2018) J. J. Li, A. Sedrakian, and F. Weber, Phys. Lett. B 783, 234 (2018), arXiv:1803.03661 [nucl-th] .
  * Dexheimer  _et al._ (2008a) V. Dexheimer, S. Schramm, and D. Zschiesche, Phys. Rev. C 77, 025803 (2008a), arXiv:0710.4192 [nucl-th] .
  * Moustakidis  _et al._ (2017) C. C. Moustakidis, T. Gaitanos, C. Margaritis, and G. A. Lalazissis, Phys. Rev. C 95, 045801 (2017), [Erratum: Phys.Rev.C 95, 059904 (2017)], arXiv:1608.00344 [nucl-th] .
  * Bedaque and Steiner (2015) P. Bedaque and A. W. Steiner, Phys. Rev. Lett. 114, 031103 (2015), arXiv:1408.5116 [nucl-th] .
  * Malfatti  _et al._ (2020) G. Malfatti, M. G. Orsaria, I. F. Ranea-Sandoval, G. A. Contrera, and F. Weber, Phys. Rev. D 102, 063008 (2020), arXiv:2008.06459 [astro-ph.HE] .
  * Guichon  _et al._ (1996) P. A. M. Guichon, K. Saito, E. N. Rodionov, and A. W. Thomas, Nucl. Phys. A 601, 349 (1996), arXiv:nucl-th/9509034 .
  * Dexheimer and Schramm (2008) V. Dexheimer and S. Schramm, Astrophys. J. 683, 943 (2008), arXiv:0802.1999 [astro-ph] .
  * Dexheimer and Schramm (2010) V. A. Dexheimer and S. Schramm, Phys. Rev. C 81, 045201 (2010), arXiv:0901.1748 [astro-ph.SR] .
  * Fortin  _et al._ (2016) M. Fortin, C. Providencia, A. R. Raduta, F. Gulminelli, J. L. Zdunik, P. Haensel, and M. Bejger, Phys. Rev. C 94, 035804 (2016), arXiv:1604.01944 [astro-ph.SR] .
  * Abbott  _et al._ (2019) B. P. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. X 9, 011001 (2019), arXiv:1805.11579 [gr-qc] .
  * (115) https://github.com/jnoronhahostler/Neutron_Star_EOS.
  * Chabanat  _et al._ (1998) E. Chabanat, P. Bonche, P. Haensel, J. Meyer, and R. Schaeffer, Nucl. Phys. A 635, 231 (1998), [Erratum: Nucl.Phys.A 643, 441–441 (1998)].
  * Douchin  _et al._ (2000) F. Douchin, P. Haensel, and J. Meyer, Nucl. Phys. A 665, 419 (2000).
  * Douchin and Haensel (2000) F. Douchin and P. Haensel, Phys. Lett. B 485, 107 (2000), arXiv:astro-ph/0006135 .
  * Douchin and Haensel (2001) F. Douchin and P. Haensel, Astron. Astrophys. 380, 151 (2001), arXiv:astro-ph/0111092 .
  * Togashi  _et al._ (2017) H. Togashi, K. Nakazato, Y. Takehara, S. Yamamuro, H. Suzuki, and M. Takano, Nucl. Phys. A 961, 78 (2017), arXiv:1702.05324 [nucl-th] .
  * Gulminelli and Raduta (2015) F. Gulminelli and A. R. Raduta, Phys. Rev. C 92, 055803 (2015), arXiv:1504.04493 [nucl-th] .
  * Tews  _et al._ (2018a) I. Tews, J. Carlson, S. Gandolfi, and S. Reddy, Astrophys. J. 860, 149 (2018a), arXiv:1801.01923 [nucl-th] .
  * Yagi  _et al._ (2014) K. Yagi, K. Kyutoku, G. Pappas, N. Yunes, and T. A. Apostolatos, Phys. Rev. D 89, 124013 (2014), arXiv:1403.6243 [gr-qc] .
  * Jiménez and Fraga (2021) J. C. Jiménez and E. S. Fraga, (2021), arXiv:2104.13480 [hep-ph] .
  * Zhang and Li (2020) N.-B. Zhang and B.-A. Li, Astrophys. J. 902, 38 (2020), arXiv:2007.02513 [astro-ph.HE] .
  * Rather  _et al._ (2021b) I. A. Rather, U. Rahaman, M. Imran, H. C. Das, A. A. Usmani, and S. K. Patra, Phys. Rev. C 103, 055814 (2021b), arXiv:2102.04067 [nucl-th] .
  * Hernandez-Vivanco  _et al._ (2021) F. Hernandez-Vivanco, P. D. Lasky, E. Thrane, R. Smith, D. Chatterjee, S. Banik, T. Motta, and A. Thomas, (2021), arXiv:2101.04782 [astro-ph.HE] .
  * Sen (2021) D. Sen, Phys. Rev. C 103, 045804 (2021), arXiv:2103.14136 [nucl-th] .
  * Rather  _et al._ (2021c) I. A. Rather, U. Rahaman, V. Dexheimer, A. A. Usmani, and S. K. Patra, (2021c), arXiv:2104.05950 [nucl-th] .
  * Tews  _et al._ (2018b) I. Tews, J. Margueron, and S. Reddy, Phys. Rev. C 98, 045804 (2018b), arXiv:1804.02783 [nucl-th] .
  * Han  _et al._ (2021) M.-Z. Han, J.-L. Jiang, S.-P. Tang, and Y.-Z. Fan, (2021), arXiv:2103.05408 [hep-ph] .
  * Ducoin  _et al._ (2011) C. Ducoin, J. Margueron, C. Providencia, and I. Vidana, Phys. Rev. C 83, 045810 (2011), arXiv:1102.1283 [nucl-th] .
  * Gamba  _et al._ (2020) R. Gamba, J. S. Read, and L. E. Wade, Class. Quant. Grav. 37, 025008 (2020), arXiv:1902.04616 [gr-qc] .
  * Li  _et al._ (2021) A. Li, Z. Miao, S. Han, and B. Zhang, Astrophys. J. 913, 27 (2021), arXiv:2103.15119 [astro-ph.HE] .
  * Zhu  _et al._ (2018) Z.-Y. Zhu, E.-P. Zhou, and A. Li, Astrophys. J. 862, 98 (2018), arXiv:1802.05510 [nucl-th] .
  * Hu  _et al._ (2020) J. Hu, S. Bao, Y. Zhang, K. Nakazato, K. Sumiyoshi, and H. Shen, PTEP 2020, 043D01 (2020), arXiv:2002.00562 [nucl-th] .
  * Balberg  _et al._ (1999) S. Balberg, I. Lichtenstadt, and G. B. Cook, Astrophys. J. Suppl. 121, 515 (1999), arXiv:astro-ph/9810361 .
  * Haensel  _et al._ (2002) P. Haensel, K. P. Levenfish, and D. G. Yakovlev, Astron. Astrophys. 394, 213 (2002), arXiv:astro-ph/0208078 .
  * Dexheimer  _et al._ (2008b) V. A. Dexheimer, C. A. Z. Vasconcellos, and B. E. J. Bodmann, Phys. Rev. C 77, 065803 (2008b), arXiv:0708.0131 [astro-ph] .
  * Casali and Menezes (2010) R. H. Casali and D. P. Menezes, Brazilian Journal of Physics 40, 166 (2010).
  * Miyatsu  _et al._ (2017) T. Miyatsu, T. Kambe, and K. Saito, PoS INPC2016, 135 (2017), arXiv:1702.05871 [nucl-th] .
  * Stone  _et al._ (2021b) J. R. Stone, V. Dexheimer, P. A. M. Guichon, A. W. Thomas, and S. Typel, Mon. Not. Roy. Astron. Soc. 502, 3476 (2021b), arXiv:1906.11100 [nucl-th] .
  * Motta  _et al._ (2021) T. F. Motta, P. A. M. Guichon, and A. W. Thomas, Nucl. Phys. A 1009, 122157 (2021), arXiv:2009.10908 [nucl-th] .
  * Schneider  _et al._ (2019) A. S. Schneider, C. Constantinou, B. Muccioli, and M. Prakash, Phys. Rev. C 100, 025803 (2019), arXiv:1901.09652 [nucl-th] .
  * Alford  _et al._ (2013) M. G. Alford, S. Han, and M. Prakash, Phys. Rev. D 88, 083013 (2013), arXiv:1302.4732 [astro-ph.SR] .
  * Alford  _et al._ (2015b) M. G. Alford, G. F. Burgio, S. Han, G. Taranto, and D. Zappalà, Phys. Rev. D 92, 083002 (2015b), arXiv:1501.07902 [nucl-th] .
  * Alford and Han (2016) M. G. Alford and S. Han, Eur. Phys. J. A 52, 62 (2016), arXiv:1508.01261 [nucl-th] .
  * Ranea-Sandoval  _et al._ (2016) I. F. Ranea-Sandoval, S. Han, M. G. Orsaria, G. A. Contrera, F. Weber, and M. G. Alford, Phys. Rev. C 93, 045812 (2016), arXiv:1512.09183 [nucl-th] .
  * Han and Steiner (2019) S. Han and A. W. Steiner, Phys. Rev. D 99, 083014 (2019), arXiv:1810.10967 [nucl-th] .
  * Chatziioannou and Han (2020) K. Chatziioannou and S. Han, Phys. Rev. D 101, 044019 (2020), arXiv:1911.07091 [gr-qc] .
  * Mishustin  _et al._ (2003) I. N. Mishustin, M. Hanauske, A. Bhattacharyya, L. M. Satarov, H. Stoecker, and W. Greiner, Phys. Lett. B 552, 1 (2003), arXiv:hep-ph/0210422 .
  * Jokela  _et al._ (2021) N. Jokela, M. Järvinen, G. Nijs, and J. Remes, Phys. Rev. D 103, 086004 (2021), arXiv:2006.01141 [hep-ph] .
  * Kojo  _et al._ (2020) T. Kojo, D. Hou, J. Okafor, and H. Togashi, (2020), arXiv:2012.01650 [astro-ph.HE] .
  * Alford  _et al._ (2005) M. Alford, M. Braby, M. W. Paris, and S. Reddy, Astrophys. J. 629, 969 (2005), arXiv:nucl-th/0411016 .
  * Christian  _et al._ (2019) J.-E. Christian, A. Zacchi, and J. Schaffner-Bielich, Phys. Rev. D 99, 023009 (2019), arXiv:1809.03333 [astro-ph.HE] .
  * Miao  _et al._ (2020) Z. Miao, A. Li, Z. Zhu, and S. Han, Astrophys. J. 904, 103 (2020), arXiv:2006.00839 [nucl-th] .
  * Benitez  _et al._ (2021) E. Benitez, J. Weller, V. Guedes, C. Chirenti, and M. C. Miller, Phys. Rev. D 103, 023007 (2021), arXiv:2010.02619 [astro-ph.HE] .
  * Bauswein  _et al._ (2021) A. Bauswein, S. Blacker, G. Lioutas, T. Soultanis, V. Vijayan, and N. Stergioulas, Phys. Rev. D 103, 123004 (2021), arXiv:2010.04461 [astro-ph.HE] .
  * Ayriyan  _et al._ (2018) A. Ayriyan, N. U. Bastian, D. Blaschke, H. Grigorian, K. Maslov, and D. N. Voskresensky, Phys. Rev. C 97, 045802 (2018), arXiv:1711.03926 [nucl-th] .
  * Baacke (1977) J. Baacke, Acta Phys. Polon. B 8, 625 (1977).
  * Steinheimer  _et al._ (2011) J. Steinheimer, S. Schramm, and H. Stocker, J. Phys. G 38, 035001 (2011), arXiv:1009.5239 [hep-ph] .
  * Satarov  _et al._ (2015) L. M. Satarov, K. A. Bugaev, and I. N. Mishustin, Phys. Rev. C 91, 055203 (2015), arXiv:1411.0959 [nucl-th] .
  * Dexheimer  _et al._ (2013) V. Dexheimer, J. Steinheimer, R. Negreiros, and S. Schramm, Phys. Rev. C 87, 015804 (2013), arXiv:1206.3086 [astro-ph.HE] .
  * Mukherjee  _et al._ (2017) A. Mukherjee, S. Schramm, J. Steinheimer, and V. Dexheimer, Astron. Astrophys. 608, A110 (2017), arXiv:1706.09191 [nucl-th] .
  * Yagi and Yunes (2013a) K. Yagi and N. Yunes, Phys. Rev. D 88, 023009 (2013a), arXiv:1303.1528 [gr-qc] .
  * Yagi and Yunes (2013b) K. Yagi and N. Yunes, Science 341, 365 (2013b), arXiv:1302.4499 [gr-qc] .