# Drivable Avatar Clothing: Faithful Full-Body Telepresence with Dynamic
Clothing Driven by Sparse RGB-D Input

Donglai Xiang<EMAIL_ADDRESS>Carnegie Mellon UniversityUSA ,  Fabian
Prada<EMAIL_ADDRESS>Meta Reality Labs ResearchUSA ,  Zhe Cao
<EMAIL_ADDRESS>Meta Reality Labs ResearchUSA ,  Kaiwen Guo
<EMAIL_ADDRESS>Meta Reality Labs ResearchUSA ,  Chenglei Wu
<EMAIL_ADDRESS>Meta Reality Labs ResearchUSA ,  Jessica Hodgins
<EMAIL_ADDRESS>Carnegie Mellon UniversityUSA and  Timur Bagautdinov
<EMAIL_ADDRESS>Meta Reality Labs ResearchUSA

(2023)

###### Abstract.

Clothing is an important part of human appearance but challenging to model in
photorealistic avatars. In this work we present avatars with dynamically
moving loose clothing that can be faithfully driven by sparse RGB-D inputs as
well as body and face motion. We propose a Neural Iterative Closest Point
(N-ICP) algorithm that can efficiently track the coarse garment shape given
sparse depth input. Given the coarse tracking results, the input RGB-D images
are then remapped to texel-aligned features, which are fed into the drivable
avatar models to faithfully reconstruct appearance details. We evaluate our
method against recent image-driven synthesis baselines, and conduct a
comprehensive analysis of the N-ICP algorithm. We demonstrate that our method
can generalize to a novel testing environment, while preserving the ability to
produce high-fidelity and faithful clothing dynamics and appearance.

Telepresence, photorealistic avatars, clothing capture

Work done when DX was a visiting researcher at Meta. ZC and CW are now at
Google.

††submissionid: 155††journalyear: 2023††copyright: rightsretained††conference:
SIGGRAPH Asia 2023 Conference Papers; December 12–15, 2023; Sydney, NSW,
Australia††booktitle: SIGGRAPH Asia 2023 Conference Papers (SA Conference
Papers ’23), December 12–15, 2023, Sydney, NSW, Australia††doi:
10.1145/3610548.3618136††isbn: 979-8-4007-0315-7/23/12††ccs: Computing
methodologies Computer graphics††ccs: Computing methodologies Computer vision
Figure 1. We present photorealistic full-body avatars that can be driven by
sparse RGB-D views (along with body pose and facial keypoints) and faithfully
reproduce the appearance and dynamics of challenging loose clothing from the
input views. We show the input views, our output and the ground truth
reference images in each group of results.

##  1\. Introduction

Photorealistic avatars are important for enabling truly immersive and
believable telepresence experiences. An ideal telepresence application should
not only produce plausible-looking results, but also be complete and accurate:
all salient aspects of human appearance should be captured and resynthesized
to fully match the real-world states of the subject. However, these properties
are particularly challenging to achieve with clothing, an integral part of
human appearance, due to its complex movements on human body.

To deal with this challenge, some previous methods go beyond treating clothing
effectively as a part of the human body (Bagautdinov et al., 2021; Liu et al.,
2021) and perform explicit modeling of clothing as a separate layer on top of
the underlying body (Xiang et al., 2021, 2022). These methods can work well
for pose-driven animation, i.e., synthesizing plausible clothing deformation
and photorealistic appearance that are perceptually compatible with the input
pose signal. However, there is no guarantee that the animation output will
faithfully reproduce the actual states of clothing (Fig. 6), and potentially
distorting the conveyed social signals. In fact, the dynamics of clothing
cannot be fully explained by the body pose of the current frame or a few
previous frames. Given two distinct initial states of clothing, the same body
motion can result in completely different trajectories of clothing
deformations, especially for loose garments like skirts or dresses. Therefore,
it is impossible to infer accurate clothing states given such incomplete input
signals.

An alternate approach for telepresence relies heavily on the availability of
sensory inputs without a strong human prior. For example, volumetric fusion
methods (Newcombe et al., 2015; Dou et al., 2016; Dou et al., 2017) produce a
complete geometrical representation of a scene by tracking and fusing
observations from sparse RGB-D cameras. Neural implicit functions can also be
utilized to reconstruct a dynamically moving human surface from sparse camera
inputs (Yu et al., 2021b; Shao et al., 2022a), or even to directly model the
radiance field of clothed human appearance (Lin et al., 2022; Shao et al.,
2022b). In theory, these methods are flexible enough to be able to reconstruct
arbitrary shape from the given input streams. However, due to a lack of model
constraints, it is generally more challenging for these methods to achieve
high-fidelity temporal coherency especially with noisy or incomplete input,
and the output quality is heavily tied with the sensory input. For example, it
is hard to produce the sharp and subtle detail in hands (Shao et al., 2022b)
with the limited resolution and noise, and the observed and unobserved regions
from the input can have obvious difference in the output quality (Shao et al.,
2022a). Human priors have been introduced to regularize the predictions (Kwon
et al., 2021, 2023), but the ability to reliably handle loose clothing has not
been clearly demonstrated.

To leverage the benefits of both families of approaches, we can rely on
explicit avatar models as a prior, but expand the driving signal to include
the denser input in addition to the body pose. We build avatars with dynamic
clothing that can be driven from a sparse set of RGB-D cameras (usually three
unless otherwise stated). This formulation allows for more faithful
resynthesis of the human appearance, including clothing details. We build on
top of DVA (Remelli et al., 2022), which proposes the texel-conditioned
avatar, an encoder-decoder model that takes in UV-aligned driving features and
predicts geometry and appearance for rendering. However, DVA only works well
for tight clothing that closely follows underlying body, due to the limitation
in relying on a generic body shape prior.

To better handle loose clothing, our insight is to introduce a tracking stage
that coarsely aligns the loose clothing surface with the input depth. More
specifically, we propose a simple-yet-effective Neural Iterative Closest
Points (N-ICP) algorithm to iteratively update a clothing deformation model
given the feedback from surface error in a data-driven manner. N-ICP combines
the flexibility of the classical ICP methods which allows us to handle large
clothing deformations, while relying on learning for more efficient inference
and reliable geometry estimates. In contrast to DVA, which uses coarse body
geometry to extract features, the N-ICP tracking allows us to extract more
accurate and meaningful texel-aligned features. It also eases the burden on
the encoder-decoder model, since large deformations and misalignments are
handled by the coarse tracking, and ultimately leads to better quality and
generalization. In addition, several technical components have been leveraged
to further improve the texel-conditioned avatars. To aid geometry prediction,
we augment texel-aligned features with geometry features computed from depth
and coarsely tracked geometry. To improve appearance, we adopt a specific
perceptual loss to encourage high-frequency texture detail on the predicted
clothing.

Our contributions can be summarized as follows:

  * •

We develop photorealistic full-body avatars with dynamic clothing that
faithfully reproduces the original states of subject’s appearance and
geometry. These avatars are driven by sparse (up to 3) RGB-D inputs and enable
free-viewpoint rendering.

  * •

As an important component of our system, we introduce a Neural Iterative
Closest Point algorithm that learns to iteratively find the most effective
parameter update to track the input point cloud efficiently with a deformation
model.

  * •

Our avatars can directly generalize to a novel testing environment with a
different background and illumination, while capturing the complex clothing
dynamics and preserving the original high-quality appearance from the training
data. We also provide an option to finetune the model to adapt to the new
appearance in the novel environment.

Figure 2. An overview of our method. The Neural Iterative Closest Point module
efficiently tracks the clothing surface from the input point cloud
$\mathbf{P}$ with a clothing deformation model
$\mathcal{D}(\boldsymbol{\theta})$; the initial tracking result $\mathbf{D}$
is then used to unwrap the driving images $I_{c}$ and depth maps $D_{c}$ into
texel-aligned features $\mathbf{F}_{I},\mathbf{F}_{D}$, which are then fed
into the texel-conditioned avatar, together with body pose
$\boldsymbol{\rho}$, facial keypoints and viewpoint $\mathbf{v}$, to predict
the output image.

##  2\. Related Work

We focus on related work in three areas: photorealistic clothed avatars,
sensing-based telepresence, and image-conditioned novel-view synthesis and
learning to optimize for non-rigid tracking.

#### Photorealistic Clothed Avatars.

Here we discuss approaches pose-driven avatars that model both the shape and
photorealistic appearance of clothed humans leanrned directly from captured
images in a data-driven manner. Depending on the appearance model, these
methods can be roughly divided into three categories: methods that combine
coarse human geometry with Deferred Neural Rendering (Thies et al., 2019;
Grigorev et al., 2021; Raj et al., 2021), methods that incorporate a human
prior into Neural Radiance Fields (Mildenhall et al., 2020; Peng et al.,
2021a; Liu et al., 2021; Li et al., 2022; Su et al., 2021; Zheng et al.,
2022), and methods based on mesh and dynamic texture (Shysheya et al., 2019;
Habermann et al., 2021). Our method is more closely related to the Full-Body
Codec Avatars (Bagautdinov et al., 2021) among the third category, as well as
the followup work that can handle dynamic clothing (Xiang et al., 2021, 2022).
Most of these methods learn a mapping from body pose or motion to clothed body
appearance, without resolving the ambiguity in clothing states for similar
body poses. Drivable Volumetric Avatars (DVA) (Remelli et al., 2022) is an
exception, which is additionally conditioned on a dynamic texture unwrapped
from sparse driving views to incorporate the clothing information. However, it
is limited to tight clothing due to a constraint in body representation. Our
method can further track and render dynamically moving loose clothing
realistically.

#### Sensing-Based Telepresence Approaches.

Another category of approaches for telepresence rely more heavily on the
sensing input for surface reconstruction, usually from one or several RGB(-D)
cameras. Volumetric fusion (Newcombe et al., 2015; Dou et al., 2016; Dou et
al., 2017) and multiview-conditioned implicit functions (Suo et al., 2021; Yu
et al., 2021b; Shao et al., 2022a) have been utilized to reconstruct scene
geometry from the sensor input. Early work only reconstructs the geometry
(Newcombe et al., 2015), but later work also predicts color by warping and
blending RGB information from input views (Dou et al., 2016; Lawrence et al.,
2021), possibly assisted by deep neural networks (Suo et al., 2021; Yu et al.,
2021b; Shao et al., 2022a). Neural rendering has also been introduced (Martin-
Brualla et al., 2018; Nguyen-Ha et al., 2022) to compensate for artifacts in
the reconstructed geometry. These sensing-based methods enjoy the flexibility
in handling varying topology, but are generally more sensitive to noisy or
missing input than the model-based approaches described in the previous
section.

#### Image-Conditioned Novel-View Synthesis.

Our task can also be regarded as Novel-view Synthesis (NVS) based on sparse
input images. Generalizable NeRF is a group of methods that extend the Neural
Radiance Field (NeRF) (Mildenhall et al., 2020) and allow the reconstruction
of a scene with sparse images as input without per-scene optimization (Yu et
al., 2021a; Wang et al., 2021; Chen et al., 2021; Lin et al., 2022; Shao et
al., 2022b). This formulation is thus more suitable for telepresence than the
original NeRF, but tends to perform less well when the target view is far away
from the sparse input views due to the inherent 3D ambiguity. To alleviate
this problem, some methods incorporate prior knowledge of the human body to
achieve better quality. Neural Body (Peng et al., 2021b) utilizes the SMPL
model to aggregate the temporal information over the multiview videos. It
still does not allow direct use of novel images as input, but this limitation
is addressed in Neural Human Performer (Kwon et al., 2021) and GP-NeRF (Chen
et al., 2022). Several other methods (Zhao et al., 2022; Gao et al., 2022,
2023; Kwon et al., 2023) predict radiance fields in a canonical space with the
help of forward and inverse skinning transformations. These methods
additionally allow generalization across identities, which is not the focus of
our work. KeypointNeRF (Mihajlovic et al., 2022) utilizes 3D body keypoints to
encode the spatial information in the rendered volume. These methods have
shown limited capability to handle dynamic loose clothing due to the body
representation.

#### Learning to Optimize for Non-Rigid Tracking.

Many non-rigid tracking and reconstruction problems are traditionally
formulated as optimization. Examples include SMPL model fitting for 3D human
pose estimation (Bogo et al., 2016), and non-rigid ICP for deformable surface
tracking (Guo et al., 2015). In recent years, researchers attempted to
incorporate deep neural networks into these problems (Bhatnagar et al., 2020;
Bozic et al., 2021; Li et al., 2020; Bozic et al., 2020). Our N-ICP
formulation essentially treats the neural network as an optimization solver
that iteratively generates a parameter update. Some work (Song et al., 2020;
Corona et al., 2022) explores a similar idea for monocular human pose
reconstruction in a supervised setting. RMA-Net (Feng et al., 2021) also
addresses the non-rigid registration problem with recurrent update. In
addition to the difference in deformation model and loss function, our method
integrates learning into optimization more deeply by explicit feeding the
error and gradient into the network for update prediction, which is shown to
be the key to effectiveness in the ablation study.

##  3\. Method Overview

Our method takes as input RGB-D images from sparse (up to 3) views, as well as
3D body pose in the form of joint angles (and facial keypoints if available),
and generates photorealistic rendering of the subject from an arbitrary
viewpoint. The model is trained on images of the subject captured in a dense
camera system. We adopt the two-layer representation that has proven effective
in previous work on loose clothing (Xiang et al., 2021, 2022). Our method
consists of two major modules. First, in the Neural Iterative Closest Point
module, we coarsely track the loose clothing surface given the fused point
cloud from the input depth maps using a deformation graph model. Second, we
convert the sparse driving RGB-D images into texel-aligned features and feed
them into texel-conditioned clothed avatars to infer detailed geometry and
view-dependent texture, which are then rasterized to form the output image.
The overall pipeline is illustrated in Fig. 2.

##  4\. Neural Iterative Closest Point

As the first step of our approach, we introduce a Neural Iterative Closest
Point (N-ICP) algorithm to coarsely track the dynamic clothing surface using a
deformation graph representation of the clothing geometry. Such a module is
needed for two reasons. First, coarse tracking provides rough correspondences
on the clothing surface across different frames. Previous work (Xiang et al.,
2021, 2022) shows that such canonicalization reduces the variance in
appearance that needs to be modelled by the downstream module and leads to
improved quality. Second, compared with skeleton-level tracking (Remelli et
al., 2022), the deformation graph enjoys the flexibility to track the surface
at a higher accuracy from the input depth, so that the following stage (Sec.
5) only needs to predict a small geometry correction. This concept of coarse-
to-fine modeling has also proven useful in previous work, e.g. (Habermann et
al., 2021).

#### Non-Rigid ICP

The classical approach to track a deforming surface is the non-rigid Iterative
Closest Point (ICP) algorithm (Li et al., 2008). Given a deformation model
$\mathcal{D}$ controlled by deformation parameters $\boldsymbol{\theta}$, the
non-rigid ICP algorithm tracks an input point cloud111At test time, the point
cloud comes from fused driving depth images. We only use the points from the
regions of dynamic clothing identified by image segmentation. $\mathbf{P}$ by
solving the following minimization problem

(1)
$\displaystyle\min_{\boldsymbol{\theta}}L_{\text{ICP}}(\boldsymbol{\theta};\mathbf{P})=\min_{\boldsymbol{\theta}}\sum_{\mathbf{p}\in\mathbf{P}}\|\mathcal{C}(\mathbf{p},\mathcal{D}(\boldsymbol{\theta}))-\mathbf{p}\|^{2},$

where $\mathcal{C}$ queries the closest point on the deformed mesh
$\mathcal{D}(\boldsymbol{\theta})$ from a point $\mathbf{p}$ in the point
cloud based on Euclidean or projective distance. To keep the deformed mesh in
a smooth shape, some regularization terms are often used together with Eq. (1)
to penalize extreme distortion. This non-linear optimization problem is
usually solved by the Gauss-Newton method, especially its Levenberg–Marquardt
variant. Concretely, in each iteration of optimization, the algorithm
evaluates the residual vector $\mathbf{r}$, which consists of offset
$\mathbf{r}(\mathbf{p})=\mathcal{C}(\mathbf{p},\mathcal{D}(\boldsymbol{\theta}))-\mathbf{p}$
for each point $\mathbf{p}$ in the point cloud, and the Jacobian $\mathbf{J}$
of each residual term with respect to the parameters. Then the algorithm
searches for the update $\Delta\boldsymbol{\theta}$ by solving the linear
system
$(\mathbf{J}^{T}\mathbf{J}+\lambda\mathbf{I})\Delta\boldsymbol{\theta}=-\mathbf{J}^{T}\mathbf{r}$,
where $\lambda>0$ is a damping coefficient. The parameters are updated by
$\boldsymbol{\theta}^{(i+1)}=\boldsymbol{\theta}^{(i)}+\Delta\boldsymbol{\theta}$,
where $i$ denotes the index of iteration.

Figure 3. The Neural Iterative Closest Point algorithm. In each iteration, we
perform the closest point query between the input point cloud $\mathbf{P}$ and
deformed clothing model $\mathcal{D}(\boldsymbol{\theta}^{(i)})$. The residual
$\mathbf{r}$ and gradient $\mathbf{g}$ are then passed to a neural network
$\mathcal{M}$, which finds the best update $\Delta\boldsymbol{\theta}$ to the
parameters. This process is repeated for $N$ iterations.

#### N-ICP

The problem of driving avatars for online telepresence poses a challenge in
terms of both robustness and speed for the non-rigid tracking algorithm. The
nonlinear minimization problem in Eq. (1) relies heavily on good
initialization, so classical method usually requires sequential tracking that
is hard to recover from failure and technically demanding GPU implementation
to meet the runtime constraint (Zollhöfer et al., 2014; Newcombe et al.,
2015).

This challenge motivates us to introduce a Neural Iterative Closest Point
(N-ICP) algorithm, where the goal is to leverage the prior learned by a deep
neural network to make an efficient and robust prediction of the update
direction $\Delta\boldsymbol{\theta}$. We utilize a PointNet (Qi et al., 2017)
architecture to operate on the input point cloud $\mathbf{P}$. In addition to
the coordinates of each point $\mathbf{p}$, we also use the offset
$\mathbf{r}(\mathbf{p})=\mathcal{C}(\mathbf{p},\mathcal{D}(\boldsymbol{\theta}))-\mathbf{p}$
as a feature, which includes the essential closest point information for
solving the non-rigid ICP problem. Inspired by the classical optimization
paradigm, we also a provide first-order derivative to the network. For the
ease of computation, we use the gradient $\mathbf{g}=\mathbf{J}^{T}\mathbf{r}$
of total loss with respect to the parameters, which can be automatically
derived in most modern neural network libraries (Paszke et al., 2019), or
manually derived for better computation efficiency. The formulation of the
network $\mathcal{M}$ can be written as
$\Delta\boldsymbol{\theta}=\mathcal{M}(\mathbf{P},\mathbf{r},\mathbf{g})$. The
update
$\boldsymbol{\theta}^{(i+1)}=\boldsymbol{\theta}^{(i)}+\Delta\boldsymbol{\theta}$
is then performed for $N=3$ times.

#### Weakly-Supervised Learning

We train the network in a weakly-supervised manner without requiring the
ground truth deformation parameters $\boldsymbol{\theta}$. Instead, we only
use the clothing geometry $\overline{\mathbf{P}}$ reconstructed by Multi-View
Stereo (MVS). Compared with $\mathbf{P}$ which has missing parts due to
occlusion from the sparse depth input, $\overline{\mathbf{P}}$ includes the
complete clothing geometry reconstructed from all cameras in the capture
studio, and is only used at training time for supervision. Our loss function
for training the network is written as
$L_{\text{N-ICP}}=\frac{1}{N}\sum_{i=1}^{N}L_{\text{ICP}}(\boldsymbol{\theta}^{(i)},\overline{\mathbf{P}})$
plus regularization (see supplementary document). In other words, the network
is trained to find an update to the parameters so that the deformed mesh can
most efficiently track the clothing surface for every ICP iteration.

#### Clothing Deformation Model

ICP-style methods require good initialization to converge to the right local
minimum. We observe that the underlying body pose can provide coarse
information about body orientation and articulation as a good starting point.
Thus we adopt a hierarchical deformation model for dynamic loose clothing: an
outer layer of Linear Blend Skinning (LBS) $\mathcal{W}$ with respect to body
pose, and an inner layer of deformation graphs $\mathcal{E}$ (Sumner et al.,
2007):
$\mathcal{D}(\boldsymbol{\theta})=\mathcal{W}(\mathcal{E}(\mathbf{M},\boldsymbol{\theta}),\boldsymbol{\rho})$,
where $\mathbf{M}$ is the template shape of the clothing defined in the
canonical space of LBS. The underlying body pose $\boldsymbol{\rho}$ can be
obtained from sparse RGB-D input by vision-based keypoint detection followed
by inverse kinematics (Mehta et al., 2017), and it remains fixed during the
N-ICP process. In this formulation, $\boldsymbol{\theta}$ is defined as the
rotation and translation of deformation graph nodes, and we set its
initialization to be the rest pose $\boldsymbol{\theta}^{(0)}=\mathbf{0}$.
Thanks to the global transformation and body articulation encoded in
$\boldsymbol{\rho}$,
$\mathcal{D}(\boldsymbol{\theta}^{(0)})=\mathcal{W}(\mathbf{M},\boldsymbol{\rho})$
is close enough to the target $\mathbf{P}$ for N-ICP to converge nicely. This
formulation also makes it efficient to perform per-frame tracking, preventing
the failure caused by error accumulation in sequential tracking. The complete
N-ICP process is illustrated in Fig. 3.

##  5\. Texel-Conditioned Clothed Avatars

The N-ICP algorithm in Sec. 4 can track large clothing dynamics, providing a
good starting point for rendering the clothed body appearance. However, the
underlying deformation model is designed to only capture large geometry
deformations, and does not model fine geometrical detail or appearance.
Therefore, in the next step, we develop a clothed avatar that can produce
high-fidelity geometry and appearance when conditioned on both sparse RGB-D
views as well as the output from the previous N-ICP stage. The critical
question here is how to faithfully reconstruct the appearance detail from the
sparse driving views for dynamic clothing.

We build upon the texel-conditioned avatars from Drivable Volumetric Avatars
(DVA) (Remelli et al., 2022). DVA takes in driving signals of several RGB
images mapped to texel-aligned features, as opposed to conditioned primarily
on pose (Bagautdinov et al., 2021), and predicts geometry and view-dependent
appearance that can reproduce the full-body appearance. Formally, in DVA, the
input feature $\mathbf{F}^{b}_{I}$ is the mean unwrapped image from multiple
RGB driving views based on the skinned mesh
$\mathbf{W}^{b}_{\boldsymbol{\rho}}=\mathcal{W}(\mathbf{M}^{b},\boldsymbol{\rho})$
of the body template $\mathbf{M}^{b}$. This unwrapping process can be written
as

(2)
$\displaystyle\mathbf{F}^{b}_{I}=\mathcal{U}(\\{I_{c}\\},\mathbf{W}^{b}_{\boldsymbol{\rho}}),$

where $\mathcal{U}$ denotes the unwrapping operation and $\\{I_{c}\\}$ denotes
the set of driving images. The neural avatar $\mathcal{A}^{b}$ is a
convolutional encoder-decoder that takes in $\mathbf{F}^{b}_{I}$, viewpoint
$\mathbf{v}$ and body pose $\boldsymbol{\rho}$, and predicts the geometry
corrective $\delta\mathbf{G}^{b}$ and appearance $\mathbf{T}^{b}$ with
$[\delta\mathbf{G}^{b},\mathbf{T}^{b}]=\mathcal{A}^{b}(\mathbf{F}^{b}_{I},\mathbf{v},\boldsymbol{\rho})$.
The final geometry $\mathbf{G}^{b}$ is obtained by a function $\mathcal{G}$
that applies $\delta\mathbf{G}^{b}$ on top of the LBS mesh
$\mathbf{W}^{b}_{\boldsymbol{\rho}}$ with a pre-defined coordinate
transformation

(3)
$\displaystyle\mathbf{G}^{b}=\mathcal{G}(\mathbf{W}^{b}_{\boldsymbol{\rho}},\delta\mathbf{G}^{b}),$

which is then used to render222The geometry $\mathbf{G}$ and appearance
$\mathbf{T}$ may take different forms depending on whether the Mixture of
Volumetric Primitive (MVP) (Lombardi et al., 2021) or mesh-texture formulation
(Lombardi et al., 2018) is adopted as the rendering model, but the high-level
concepts are similar and described here in a unified manner. The original DVA
method (Remelli et al., 2022) uses MVP, while we use mesh and texture. the
output image together with the view-conditioned appearance $\mathbf{T}^{b}$.

#### Texel-Conditioned Clothing Avatars.

The DVA baseline, however, struggles to handle the large deformation of
dynamic clothing. The root of the problem is that the LBS mesh
$\mathbf{W}^{b}_{\boldsymbol{\rho}}$, which encodes only the skeleton-level
motion, is too coarse to serve as the base geometry for dynamic clothing. The
large deviation of the LBS mesh $\mathbf{W}^{b}_{\boldsymbol{\rho}}$ from the
true clothing surface has two consequences: first, the unwrapping operation in
Eq. (2) cannot effectively capture the appearance detail; second, it places a
heavy burden for the network $\mathcal{A}^{b}$ to predict a large offset
$\delta\mathbf{G}^{b}$ to update the geometry in Eq. (3).

One of the key ideas of our clothed avatar model is to use the non-rigid
tracking result $\mathbf{D}=\mathcal{D}(\boldsymbol{\theta}^{(N)})$ from the
final N-ICP iteration as the starting point. Because $\mathbf{D}$ is already
well-aligned with the clothing surface, we can obtain better texel-aligned
features with more appearance detail from the unwrapping operation
$\mathbf{F}_{I}=\mathcal{U}(\\{I_{c}\\},\mathbf{D})$. In order to further
guide the estimation of the geometry corrective using the driving depth
images, we also provide a “depth offset” feature $\mathbf{F}_{D}$ as input.
For each pixel $[x,y]$ in camera $c$ with depth value $D_{c}$, we associate it
with the rendered depth $d_{c}$ from the tracked geometry $\mathbf{D}$ at the
same pixel location and compute the offset as

(4) $\displaystyle
O_{c}[x,y]=\mathbf{R}_{c}(d_{c}[x,y]-D_{c}[x,y])\mathbf{K}_{c}^{-1}\begin{bmatrix}x&y&1\end{bmatrix}^{T}+\mathbf{t}_{c},$

where $\mathbf{K}_{c}$ is the camera projection matrix, and
$\\{\mathbf{R}_{c},\mathbf{t}_{c}\\}$ are the rigid transformation from each
camera frame to a unified body root coordinate frame. The depth offset feature
is then a 3-channel average tensor obtained by unwrapping $\\{O_{c}\\}$ for
each driving camera $c$, $\mathbf{F}_{D}=\mathcal{U}(\\{O_{c}\\},\mathbf{D})$.
Thus, $\mathbf{F}_{D}$ can be regarded as the offset to be corrected on top of
$\mathbf{D}$ in order to match the sensor depth for each UV location. Our
clothing model takes in the texel-aligned features $\mathbf{F}_{I}$ and
$\mathbf{F}_{D}$ as well as the viewpoint $\mathbf{v}$, and predicts the
geometry corrective $\delta\mathbf{G}$ and texture $\mathbf{T}$ with
$[\delta\mathbf{G},\mathbf{T}]=\mathcal{A}(\mathbf{F}_{I},\mathbf{F}_{D},\mathbf{v})$.
Finally, the geometry $\mathbf{G}$ and texture $\mathbf{T}$ are used to render
the output image. Following (Xiang et al., 2021), the clothing is modelled as
a separate layer from the underlying body avatar. The body avatar can be
conditioned on body pose $\boldsymbol{\rho}$ or additionally on texel-aligned
features (Remelli et al., 2022). To train the model, besides the standard
image reconstruction loss and mesh regularization, we use a part segmentation
loss and an ID-MRF perceptual loss (Wang et al., 2018), which are detailed in
the supplementary document.

##  6\. Results

###  6.1. Capture Setup and Detail

We capture a total of three sets of garments: (1) a red dress with a short
full skirt; (2) a flared, short skirt333The paired upper-body garments are
tight and modelled in the body layer. in floral pattern with a bottom ruffle;
(3) a loose T-shirt and a long skirt. The training data are captured in a
multi-view capture studio equipped with roughly 150 RGB cameras. This dense
capture setup enables us to use Multi-View Stereo (MVS) to reconstruct
geometry, which can be rasterized to depth and used as input to drive our
avatars. We split out a small segment of each sequence for testing purpose and
use the multi-view captured images as ground truth for evaluation.

To demonstrate the application of our method for telepresence, we additionally
capture the subjects with the same garments in a novel environment with
different background and illumination. It is designed to be a simpler capture
setup, where nine Kinect RGB-D cameras are deployed, and we select three
cameras as driving input to our model. We also split the data in the novel
environment for fine-tuning and testing.

###  6.2. Evaluation of Neural Iterative Closest Point

We conduct an evaluation on the N-ICP algorithm using the dress sequence. We
use the ground truth from offline registration by non-rigid ICP from (Xiang et
al., 2022). For this evaluation, we adopt the same input as our full method: a
point cloud fusing the depth maps from three driving views. We report the
evaluation metric of the mean squared point-to-triangle distance for both
directions from prediction to ground-truth and from ground truth to
prediction.

Figure 4. Comparison between our method and classical non-rigid ICP with
different types of optimization solvers. For each method, we plot the runtime
in seconds vs. the Mean Squared Surface Error (MSE) in ${\mathrm{mm}}^{2}$ for
the surface tracking results. The square markers on the curves denote
individual steps in the optimization.

We compare N-ICP with classical optimization solvers, including L-BFGS and
nonlinear Conjugate Gradient with strong Wolfe linear search implemented in
PyTorch-Minimize (Feinman, 2021), and Levenberg-Marquart implemented in
Theseus (Pineda et al., 2022). All the methods are implemented in CUDA PyTorch
with analytically computed derivatives. The results are shown in Fig. 4. Our
method converges at a faster rate than the baseline methods, including both
the gradient-based methods (L-BFGS, CG and ours) and the more complicated
Jacobian-based Levenberg-Marquardt method. For ablation studies on the
formulation of our N-ICP network, please refer to our supplementary document.

###  6.3. Full Method Evaluation

Table 1. Quantitative comparison with various baselines and ablation studies using the loose T-shirt and long skirt sequence captured in the multi-view studio. The metrics are computed on the whole rendered images with a plain background from two different views. We compare with DVA (Remelli et al., 2022) and its two-layer variant, ENeRF (Lin et al., 2022), KeypointNeRF (Mihajlovic et al., 2022), and sensing-based baselines. Ablation studies are also included. Methods |  $L_{1}\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$
---|---|---|---|---
DVA |  2.89 |  27.96 |  0.9218 |  0.0932
DVA (Two-layer) |  2.90 |  28.03 |  0.9219 |  0.0912
ENeRF (Masked) |  3.07 |  27.21 |  0.9130 |  0.0772
KeypointNeRF |  3.14 |  27.42 |  0.9111 |  0.1123
Depth-based warping |  3.22 |  26.94 |  0.9171 |  0.1111
\+ LookinGood UNet |  2.27 |  29.91 |  0.9382 |  0.0865
Ours w/o N-ICP |  2.29 |  29.04 |  0.9299 |  0.0755
Ours w/o depth offset |  2.19 |  29.35 |  0.9327 |  0.0728
Ours w/o part loss |  2.11 |  29.63 |  0.9354 |  0.0724
Ours full |  1.93 |  30.25 |  0.9404 |  0.0686

In this section, we present evaluation and comparison using high-quality data
from the dense multi-view capture studio. For comparison with most methods, we
report quantitative results on the the challenging loose T-shirt and long
skirt sequence in in Tab. 1, except for pose-driven clothed avatars because
the tucked T-shirt is extremely challenging to simulate in (Xiang et al.,
2022). Therefore for this category we report results on the skirt with floral
pattern shown in Tab. 2.

#### Comparison w/ DVA (Remelli et al., 2022).

Although DVA adopts the Mixture of Volumetric Primitive (MVP) (Lombardi et
al., 2021) formulation that can render any structure in theory, the spatial
arrangement of the primitives relies on the guidance of base body mesh as
initialization. Both the original method and its variant with body and
clothing layers are guided by LBS transformation, which is too coarse to
capture the motion of loose clothing. Qualitative comparison is shown on the
left of Fig. 7.

#### Comparison w/ NeRF-based methods.

NeRF has been used to model dynamically moving human appearance, but many of
them requires per-scene/frame fitting (Işık et al., 2023; Shao et al., 2023)
and are not directly applicable to our telepresence setting. We only compare
our method with representative approaches of generalizable NeRF that directly
take in sparse image input. ENeRF (Lin et al., 2022) focuses on the setting of
interpolating a target view from several _most nearby_ source views without
human prior, but does not perform well in our setting due to the difficulty in
associating views given wide camera baselines. KeypointNeRF (Mihajlovic et
al., 2022) provides relative encoding of body keypoints as an additional input
feature for radiance prediction but cannot handle clothing motion that does
not closely follow the body motion. Qualitative comparison is shown on the
right side of Fig. 7.

Figure 5. Comparison with basic sensing-based baselines. Given the depth input
from 3 views, we run TSDF fusion (Curless and Levoy, 1996; Dong et al., 2022)
to obtain a proxy geometry (first column), and then warp the driving RGB
images to the target view (second column). We train a UNet following
LookinGood (Martin-Brualla et al., 2018) to inpaint the missing regions (third
column).

#### Comparison w/ sensing-based baselines.

Most sensing-based approaches (Dou et al., 2017; Yu et al., 2021b; Martin-
Brualla et al., 2018; Shao et al., 2022a) are proprietary and tightly
integrated with their capture system without available open-source
implementations. Therefore, we rather compare our method with baselines that
can be implemented with moderate effort, listed in the third group of results
in Tab. 1, including image warping based on TSDF fusion and further applying a
U-Net follwing the idea of LookinGood (Martin-Brualla et al., 2018). For
detail please refer to our supplementary document.

Figure 6. Qualitative comparison with Clothing Codec Avatars (Xiang et al., 2021) and Dressing Avatars (Xiang et al., 2022). The results are shown on the top row; the input views and ground truth are shown on the bottom. The $L_{1}$ error in the skirt region is reported beside the names of the methods. Table 2. Quantitative comparison with Clothing Codec Avatars (Xiang et al., 2021) and Dressing Avatars (Xiang et al., 2022) on the skirt sequence. The metrics are computed in the skirt region. Methods |  $L_{1}$$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$
---|---|---|---|---
Clothing CAs |  25.46 |  17.81 |  0.2431 |  0.507
Dressing Avatars |  27.95 |  17.01 |  0.1818 |  0.486
Ours |  15.67 |  21.99 |  0.6527 |  0.203

#### Ablation studies.

We show ablation studies on several components of our framework and the
results are shown in Fig. 8 and the fourth group of Tab. 1. First, the initial
tracking by N-ICP provides a basis for the whole framework to estimate the
correct clothing geometry and appearance. Without this component, the
initialization by only body pose is too coarse and leads to obvious artifacts.
Second, we demonstrate that the additional depth offset input to the encoder-
decoder allows our method to predict more accurate overall clothing shape.
Third, we verify that the part segmentation loss helps to produce correct
body-clothing boundary. Last, we compare the results with and without ID-MRF
loss, which preserves the high-frequency texture detail in our output when the
predicted geometry is not completely photometrically aligned with the driving
images.

#### Comparison w/ pose-driven avatars.

A major motivation of our framework is that the underlying body motion does
not contain enough information to fully determine the loose clothing states.
We validate this intuition by comparison with Clothing Codec Avatars (Xiang et
al., 2021) and Dressing Avatars (Xiang et al., 2022) shown in Fig. 6 and Tab.
2. Clothing Codec Avatars struggle to learn the mapping from a sequence of
body pose to large clothing dynamics. Dressing Avatars produce clothing motion
that looks _realistic_ with the help of physics-based simulation, but its
output is not _faithful_ to the actual motion because of the lack of an
efficient approach for estimating underlying physical parameters for
simulation. Our method utilizes the driving signal from sparse RGB-D input to
achieve faithful clothing telepresence, which is verified by the low error on
the evaluation metrics in Tab. 2. Please refer to the supplementary document
for a more detailed discussion on the difference in their formulation (e.g.
supervised vs. self-supervised).

###  6.4. Results in the Novel Capture Environment

For the novel capture environment, we test our model in two different
scenarios: without and with fine-tuning, both shown in Fig. 9. The first
scenario refers to a direct application of the model trained from the dense
capture studio to the same subject but in the novel environment. For this
scenario, our model needs to handle the difference in the input RGB-D images
between the training and testing environments, such as illumination and sensor
setup. Note that our model is trained to be robust to these variations in the
input and preserve the appearance style from the original training data in our
output (see supplementary document), but with the unseen body and clothing
motion captured in the novel environment. This experiment demonstrates the
ability of our method to directly generalize to a novel environment.

In the second scenario, we test our model after fine-tuning it with the
training data captured in the new environment. Note that we use the same three
Kinects for both driving input and ground truth supervision. During fine-
tuning, the output of our model adapts to the new appearance caused by the
illumination and sensor specification in the environment, as well as the
change in body over time such as hair style. We test our model with a even
more aggressive change in the body-layer appearance from the tank-top to the
green capture suit in the second row of Fig. 9. The fine-tuning step can be
viewed as an option to further boost the model output quality if time and
computation budgets permit.

##  7\. Conclusion

We presented a framework for building photorealistic full-body avatars that
can be driven by sparse RGB-D inputs and faithfully reproduce the motion of
loose clothing. Our method accurately reconstructs challenging clothing
appearance of the subjects, thus tackling a major drawback of existing pose-
driven avatars. As per limitations, our model is still person- and garment-
specific and cannot handle clothing motion that falls far outside of the
deformation space of the deformation graph model, such as topology change.
Interesting future directions would be to extend our method to multi-identity
setting, and develop a formulation that can handle more generic garment
categories, e.g. with implicit representations.

## References

  * (1)
  * Bagautdinov et al. (2021) Timur Bagautdinov, Chenglei Wu, Tomas Simon, Fabian Prada, Takaaki Shiratori, Shih-En Wei, Weipeng Xu, Yaser Sheikh, and Jason Saragih. 2021\.  Driving-signal aware full-body avatars.  _ACM Transactions on Graphics (TOG)_ 40, 4 (2021).
  * Bhatnagar et al. (2020) Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. 2020\.  Loopreg: Self-supervised learning of implicit surface correspondences, pose and shape for 3d human mesh registration.  _Advances in Neural Information Processing Systems_ 33 (2020).
  * Bogo et al. (2016) Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. 2016.  Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image. In _European Conference on Computer Vision_.
  * Bozic et al. (2020) Aljaz Bozic, Pablo Palafox, Michael Zollhöfer, Angela Dai, Justus Thies, and Matthias Nießner. 2020.  Neural non-rigid tracking.  _Advances in Neural Information Processing Systems_ 33 (2020).
  * Bozic et al. (2021) Aljaz Bozic, Pablo Palafox, Michael Zollhofer, Justus Thies, Angela Dai, and Matthias Nießner. 2021.  Neural deformation graphs for globally-consistent non-rigid reconstruction. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Chen et al. (2021) Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. 2021.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_.
  * Chen et al. (2022) Mingfei Chen, Jianfeng Zhang, Xiangyu Xu, Lijuan Liu, Yujun Cai, Jiashi Feng, and Shuicheng Yan. 2022.  Geometry-guided progressive nerf for generalizable and efficient neural human rendering. In _European Conference on Computer Vision_.
  * Corona et al. (2022) Enric Corona, Gerard Pons-Moll, Guillem Alenyà, and Francesc Moreno-Noguer. 2022.  Learned vertex descent: a new direction for 3D human model fitting. In _European Conference on Computer Vision_.
  * Curless and Levoy (1996) Brian Curless and Marc Levoy. 1996.  A volumetric method for building complex models from range images. In _SIGGRAPH 1996 Conference Papers_.
  * Dong et al. (2022) Wei Dong, Yixing Lao, Michael Kaess, and Vladlen Koltun. 2022\.  ASH: A modern framework for parallel spatial hashing in 3D perception.  _IEEE transactions on pattern analysis and machine intelligence_ (2022).
  * Dou et al. (2017) Mingsong Dou, Philip Davidson, Sean Ryan Fanello, Sameh Khamis, Adarsh Kowdle, Christoph Rhemann, Vladimir Tankovich, and Shahram Izadi. 2017.  Motion2fusion: Real-time volumetric performance capture.  _ACM Transactions on Graphics (TOG)_ 36, 6 (2017).
  * Dou et al. (2016) Mingsong Dou, Sameh Khamis, Yury Degtyarev, Philip Davidson, Sean Ryan Fanello, Adarsh Kowdle, Sergio Orts Escolano, Christoph Rhemann, David Kim, Jonathan Taylor, et al. 2016\.  Fusion4d: Real-time performance capture of challenging scenes.  _ACM Transactions on Graphics (TOG)_ 35, 4 (2016).
  * Feinman (2021) Reuben Feinman. 2021\.  Pytorch-minimize: a library for numerical optimization with autograd.  https://github.com/rfeinman/pytorch-minimize
  * Feng et al. (2021) Wanquan Feng, Juyong Zhang, Hongrui Cai, Haofei Xu, Junhui Hou, and Hujun Bao. 2021\.  Recurrent multi-view alignment network for unsupervised surface registration. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Gao et al. (2023) Qingzhe Gao, Yiming Wang, Libin Liu, Lingjie Liu, Christian Theobalt, and Baoquan Chen. 2023\.  Neural novel actor: Learning a generalized animatable neural representation for human actors.  _IEEE Transactions on Visualization and Computer Graphics_ (2023).
  * Gao et al. (2022) Xiangjun Gao, Jiaolong Yang, Jongyoo Kim, Sida Peng, Zicheng Liu, and Xin Tong. 2022\.  MPS-NeRF: Generalizable 3D Human Rendering from Multiview Images.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2022).
  * Grigorev et al. (2021) Artur Grigorev, Karim Iskakov, Anastasia Ianina, Renat Bashirov, Ilya Zakharkin, Alexander Vakhitov, and Victor Lempitsky. 2021\.  Stylepeople: A generative model of fullbody human avatars. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Guo et al. (2015) Kaiwen Guo, Feng Xu, Yangang Wang, Yebin Liu, and Qionghai Dai. 2015.  Robust non-rigid motion tracking and surface reconstruction using l0 regularization. In _Proceedings of the IEEE International Conference on Computer Vision_.
  * Habermann et al. (2021) Marc Habermann, Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, and Christian Theobalt. 2021.  Real-time deep dynamic characters.  _ACM Transactions on Graphics (TOG)_ 40, 4 (2021).
  * Işık et al. (2023) Mustafa Işık, Martin Rünz, Markos Georgopoulos, Taras Khakhulin, Jonathan Starck, Lourdes Agapito, and Matthias Nießner. 2023.  HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion.  _ACM Transactions on Graphics (TOG)_ 42, 4 (2023).
  * Kwon et al. (2021) Youngjoong Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs. 2021\.  Neural human performer: Learning generalizable radiance fields for human performance rendering.  _Advances in Neural Information Processing Systems_ 34 (2021).
  * Kwon et al. (2023) YoungJoong Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs. 2023\.  Neural Image-based Avatars: Generalizable Radiance Fields for Human Avatar Modeling. In _International Conference on Learning Representations_.
  * Lawrence et al. (2021) Jason Lawrence, Danb Goldman, Supreeth Achar, Gregory Major Blascovich, Joseph G Desloge, Tommy Fortes, Eric M Gomez, Sascha Häberling, Hugues Hoppe, Andy Huibers, et al. 2021\.  Project starline: a high-fidelity telepresence system.  _ACM Transactions on Graphics (TOG)_ 40, 6 (2021).
  * Li et al. (2008) Hao Li, Robert W Sumner, and Mark Pauly. 2008.  Global correspondence optimization for non-rigid registration of depth scans. In _Computer graphics forum_ , Vol. 27.
  * Li et al. (2022) Ruilong Li, Julian Tanke, Minh Vo, Michael Zollhöfer, Jürgen Gall, Angjoo Kanazawa, and Christoph Lassner. 2022.  Tava: Template-free animatable volumetric actors. In _European Conference on Computer Vision_.
  * Li et al. (2020) Yang Li, Aljaz Bozic, Tianwei Zhang, Yanli Ji, Tatsuya Harada, and Matthias Nießner. 2020.  Learning to optimize non-rigid tracking. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Lin et al. (2022) Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou. 2022.  Efficient Neural Radiance Fields for Interactive Free-viewpoint Video. In _SIGGRAPH Asia 2022 Conference Papers_.
  * Liu et al. (2021) Lingjie Liu, Marc Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, and Christian Theobalt. 2021.  Neural actor: Neural free-view synthesis of human actors with pose control.  _ACM Transactions on Graphics (TOG)_ 40, 6 (2021).
  * Lombardi et al. (2018) Stephen Lombardi, Jason Saragih, Tomas Simon, and Yaser Sheikh. 2018.  Deep appearance models for face rendering.  _ACM Transactions on Graphics (TOG)_ 37, 4 (2018).
  * Lombardi et al. (2021) Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason Saragih. 2021.  Mixture of volumetric primitives for efficient neural rendering.  _ACM Transactions on Graphics (TOG)_ 40, 4 (2021).
  * Martin-Brualla et al. (2018) Ricardo Martin-Brualla, Rohit Pandey, Shuoran Yang, Pavel Pidlypenskyi, Jonathan Taylor, Julien Valentin, Sameh Khamis, Philip Davidson, Anastasia Tkach, Peter Lincoln, et al. 2018\.  LookinGood: enhancing performance capture with real-time neural re-rendering.  _ACM Transactions on Graphics (TOG)_ 37, 6 (2018).
  * Mehta et al. (2017) Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, and Christian Theobalt. 2017.  Vnect: Real-time 3d human pose estimation with a single rgb camera.  _ACM Transactions on Graphics (TOG)_ 36, 4 (2017).
  * Mihajlovic et al. (2022) Marko Mihajlovic, Aayush Bansal, Michael Zollhoefer, Siyu Tang, and Shunsuke Saito. 2022.  KeypointNeRF: Generalizing image-based volumetric avatars using relative spatial encoding of keypoints. In _European Conference on Computer Vision_.
  * Mildenhall et al. (2020) Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2020\.  Nerf: Representing scenes as neural radiance fields for view synthesis. In _European Conference on Computer Vision_.
  * Newcombe et al. (2015) Richard A Newcombe, Dieter Fox, and Steven M Seitz. 2015\.  Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
  * Nguyen-Ha et al. (2022) Phong Nguyen-Ha, Nikolaos Sarafianos, Christoph Lassner, Janne Heikkilä, and Tony Tung. 2022\.  Free-Viewpoint RGB-D Human Performance Capture and Rendering. In _European Conference on Computer Vision_.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Peng et al. (2021a) Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. 2021a.  Animatable neural radiance fields for modeling dynamic human bodies. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_.
  * Peng et al. (2021b) Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. 2021b.  Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Pineda et al. (2022) Luis Pineda, Taosha Fan, Maurizio Monge, Shobha Venkataraman, Paloma Sodhi, Ricky TQ Chen, Joseph Ortiz, Daniel DeTone, Austin Wang, Stuart Anderson, et al. 2022\.  Theseus: A library for differentiable nonlinear optimization.  _Advances in Neural Information Processing Systems_ 35 (2022).
  * Qi et al. (2017) Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. 2017\.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  _Advances in Neural Information Processing Systems_ 30 (2017).
  * Raj et al. (2021) Amit Raj, Julian Tanke, James Hays, Minh Vo, Carsten Stoll, and Christoph Lassner. 2021\.  ANR: Articulated Neural Rendering for Virtual Avatars. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Remelli et al. (2022) Edoardo Remelli, Timur Bagautdinov, Shunsuke Saito, Chenglei Wu, Tomas Simon, Shih-En Wei, Kaiwen Guo, Zhe Cao, Fabian Prada, Jason Saragih, et al. 2022\.  Drivable Volumetric Avatars using Texel-Aligned Features. In _SIGGRAPH 2022 Conference Papers_.
  * Shao et al. (2022a) Ruizhi Shao, Liliang Chen, Zerong Zheng, Hongwen Zhang, Yuxiang Zhang, Han Huang, Yandong Guo, and Yebin Liu. 2022a.  FloRen: Real-time High-quality Human Performance Rendering via Appearance Flow Using Sparse RGB Cameras. In _SIGGRAPH Asia 2022 Conference Papers_.
  * Shao et al. (2022b) Ruizhi Shao, Hongwen Zhang, He Zhang, Mingjia Chen, Yan-Pei Cao, Tao Yu, and Yebin Liu. 2022b.  Doublefield: Bridging the neural surface and radiance fields for high-fidelity human reconstruction and rendering. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Shao et al. (2023) Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu. 2023\.  Tensor4D: Efficient Neural 4D Decomposition for High-fidelity Dynamic Reconstruction and Rendering. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Shysheya et al. (2019) Aliaksandra Shysheya, Egor Zakharov, Kara-Ali Aliev, Renat Bashirov, Egor Burkov, Karim Iskakov, Aleksei Ivakhnenko, Yury Malkov, Igor Pasechnik, Dmitry Ulyanov, et al. 2019\.  Textured neural avatars. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Song et al. (2020) Jie Song, Xu Chen, and Otmar Hilliges. 2020.  Human body model fitting by learned gradient descent. In _European Conference on Computer Vision_.
  * Su et al. (2021) Shih-Yang Su, Frank Yu, Michael Zollhöfer, and Helge Rhodin. 2021.  A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose.  _Advances in Neural Information Processing Systems_ 34 (2021).
  * Sumner et al. (2007) Robert W Sumner, Johannes Schmid, and Mark Pauly. 2007\.  Embedded deformation for shape manipulation.  _ACM Transactions on Graphics (TOG)_ 26, 3 (2007).
  * Suo et al. (2021) Xin Suo, Yuheng Jiang, Pei Lin, Yingliang Zhang, Minye Wu, Kaiwen Guo, and Lan Xu. 2021.  Neuralhumanfvv: Real-time neural volumetric human performance rendering using rgb cameras. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_.
  * Thies et al. (2019) Justus Thies, Michael Zollhöfer, and Matthias Nießner. 2019\.  Deferred neural rendering: Image synthesis using neural textures.  _ACM Transactions on Graphics (TOG)_ 38, 4 (2019).
  * Wang et al. (2021) Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. 2021.  Ibrnet: Learning multi-view image-based rendering. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Wang et al. (2018) Yi Wang, Xin Tao, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia. 2018.  Image inpainting via generative multi-column convolutional neural networks.  _Advances in neural information processing systems_ 31 (2018).
  * Xiang et al. (2022) Donglai Xiang, Timur Bagautdinov, Tuur Stuyck, Fabian Prada, Javier Romero, Weipeng Xu, Shunsuke Saito, Jingfan Guo, Breannan Smith, Takaaki Shiratori, et al. 2022\.  Dressing Avatars: Deep Photorealistic Appearance for Physically Simulated Clothing.  _ACM Transactions on Graphics (TOG)_ 41, 6 (2022).
  * Xiang et al. (2021) Donglai Xiang, Fabian Prada, Timur Bagautdinov, Weipeng Xu, Yuan Dong, He Wen, Jessica Hodgins, and Chenglei Wu. 2021\.  Modeling clothing as a separate layer for an animatable human avatar.  _ACM Transactions on Graphics (TOG)_ 40, 6 (2021).
  * Yu et al. (2021a) Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. 2021a.  pixelnerf: Neural radiance fields from one or few images. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Yu et al. (2021b) Tao Yu, Zerong Zheng, Kaiwen Guo, Pengpeng Liu, Qionghai Dai, and Yebin Liu. 2021b.  Function4d: Real-time human volumetric capture from very sparse consumer rgbd sensors. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Zhao et al. (2022) Fuqiang Zhao, Wei Yang, Jiakai Zhang, Pei Lin, Yingliang Zhang, Jingyi Yu, and Lan Xu. 2022.  Humannerf: Efficiently generated human radiance field from sparse inputs. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Zheng et al. (2022) Zerong Zheng, Han Huang, Tao Yu, Hongwen Zhang, Yandong Guo, and Yebin Liu. 2022\.  Structured local radiance fields for human avatar modeling. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Zollhöfer et al. (2014) Michael Zollhöfer, Matthias Nießner, Shahram Izadi, Christoph Rehmann, Christopher Zach, Matthew Fisher, Chenglei Wu, Andrew Fitzgibbon, Charles Loop, Christian Theobalt, et al. 2014\.  Real-time non-rigid reconstruction using an RGB-D camera.  _ACM Transactions on Graphics (TOG)_ 33, 4 (2014).

Figure 7. Left: comparison with Drivable Volumetric Avatars (DVA) (Remelli et
al., 2022) and its two-layer variant. Right: comparison with NeRF-based
methods (Lin et al., 2022; Mihajlovic et al., 2022). We mask out the black
background of results in ENeRF (Lin et al., 2022) with ground truth
segmentation. Figure 8. Ablation studies on components in our framework,
including N-ICP, depth offset as input to the encoder-decoder, part
segmentation loss, and ID-MRF loss. Each result is shown in comparison to our
full output and the ground truth. Figure 9. Testing results in the novel
capture environment. On the left we show in the input RGB-images. We show our
output from 2 different viewpoints both before and after fine-tuning. These
frames are not seen during fine-tuning. The subject on the second row wears
different upper-body garments between the captures in the original and new
environment. The tank top is preserved in the body layer before fine-tuning
and adapted after that.