# Semantic-Preserved Point-based Human Avatar

Lixiang Lin Jianke Zhu
ZheJiang University
<EMAIL_ADDRESS>
###### Abstract

To enable realistic experience in AR/VR and digital entertainment, we present
the first point-based human avatar model that embodies the entirety expressive
range of digital humans. We employ two MLPs to model pose-dependent
deformation and linear skinning (LBS) weights. The representation of
appearance relies on a decoder and the features that attached to each point.
In contrast to alternative implicit approaches, the oriented points
representation not only provides a more intuitive way to model human avatar
animation but also significantly reduces both training and inference time.
Moreover, we propose a novel method to transfer semantic information from the
SMPL-X model to the points, which enables to better understand human body
movements. By leveraging the semantic information of points, we can facilitate
virtual try-on and human avatar composition through exchanging the points of
same category across different subjects. Experimental results demonstrate the
efficacy of our presented method. Code will be made publicly available at
https://github.com/l1346792580123/spa

##  1 Introduction

Nonverbal human elements such like body posture, facial expressions, and
appearance are essential to human communication. The presence of these
nonverbal cues highlights the need of capturing the complete essence of human
expressiveness so that we can achieve immersive and lifelike experiences in
AR/VR. Currently, the available solutions can be broadly categorized into two
groups, including model-based methods [20, 26, 5, 13] and model-free
approaches [33, 34, 6, 35, 36]. Unfortunately, both of them have their own
limitations and drawbacks.

Figure 1: The conventional implicit approach involves traversing each position
in the space to obtain the SDF or occupancy value, and then passing through
the marching cube (MC) [21] to get the final mesh. While our method
intuitively animate the oriented points. Our points representation can produce
the final mesh through Poisson reconstruction (PSR) [14] in no time while
preserving the semantic information of the human body. Points with semantics
allow us to exchange points of different individuals to achieve human avatar
composition.

The development of parametric body models, such like SMPL family [20, 26], has
played a crucial role in digital human modeling. These models utilize PCA
shape parameters and Euler angle of joints to represent human in variety
poses. While the fixed topology and limited resolution of the 3D mesh restrict
their flexibility and prevent modeling of clothing or hair. Consequently,
capturing the complete appearance of humans becomes challenging with these
models, especially when considering clothed bodies.

Implicit representations offer a promising solution to overcome these
limitations. Chen et al [6] combined the continuous implicit functions with
learned forward skinning to build articulate human avatars, which can produce
reasonable results for arbitrary poses. Saito et al [35] proposed an end-to-
end framework that aims to learn geometric cycle-consistency while considering
both posed and unposed shapes in a weakly supervised manner. Shen et al [36]
incorporated part-aware initialization strategy on the top of SNARF to reduce
the ambiguity in iterative root finding. Moreover, hand pose, facial
expressions and appearance are included to model expressive human avatar in a
holistic fashion. Although having achieved promising results, implicit method
still suffers some problems. The implicit representation is not
straightforward, as it requires forward inference to derive geometric
information, such as occupancy, SDF, etc. It takes long time to obtain final
mesh since all grid points in space are passed to MLP to query geometric
information. Moreover, implicit representations cannot maintain semantic
information. To depict the same individual in various outfits, it requires to
train distinct models for each attire variation.

To tackle these problems, we propose an oriented points-based method to model
human avatar. As depicted in Fig. 1, our oriented points representation
proficiently models human avatars with minimal inference time. We introduce a
novel approach to transfer the semantic information from SMPL-X model [26] to
our oriented points representation. Enriching points with semantics enhances
our ability to analyze the meaning of human movements. Additionally, we
combine points from various subjects based on their semantic attributes to
generate new human avatars, a valuable application for virtual try-on
scenarios. Oriented points representation offers a more intuitive alternative
comparing to implicit methods. By leveraging the Poisson Surface
Reconstruction technique [14], we can generate the final mesh instantly,
whereas implicit methods may require several seconds. We integrate two MLPs to
capture pose-dependent deformations and linear blend skinning (LBS) weights.
These oriented points are then skinned using SMPL-X pose parameters, enabling
us to represent a wide range of human poses. We affix a trainable neural
texture to each point as appearance representation. This neural texture is
decoded through an MLP to determine the final color. The texture of the
vertices on the reconstructed mesh is derived as a weighted average of their
k-nearest points.

In summary, the main contributions of our paper are in the following.

  * •

We present the first point based human avatar that can capture body pose, hand
pose, facial expressions and appearance.

  * •

We propose a novel method to transfer the semantic information of SMPL-X model
to the point-based avatar, which enables some real-word application such like
virtual try-on.

  * •

Experimental evaluations against state-of-the-art methods demonstrate the
accuracy and efficiency of our proposed method.

Figure 2: Method Overview. We select a scan from training set as template
scan. During training, the sampled points are transformed to pose space
through DeltaNet and LBSNet, and then the loss is computed by comparing it
with points sampled from the posed scan. During inference, we transfer the
semantic information of SMPL-X mesh to sampled points. The sampled points can
be reposed according to SMPL-X parameters, and can also be composited with
other points to generate a new avatar. For clearness, we omit the Poisson
reconstruction from points to mesh.

##  2 Related Works

###  2.1 Parametric Human Models

Parametric human models [3, 20, 12, 26] try to represent the shape and pose of
animatable people with a few parameters. Some models focus on a part of human
like hands [32] or faces [4, 18]. Parametric models are favored due to their
seamless integration with existing computer graphics pipelines and the
advantage of a compact parameter space that facilitates effective learning.
Traditional approaches [5, 39, 26] rely on nonlinear optimization solver to
deduce the reasonable parameters from 2D keypoints, which are usually
computationally inefficient. Many methods [13, 17, 8, 10, 16, 40] adopt deep
neural network to directly predict parameters from a single RGB image.
Furthermore, the widespread adoption of graph convolutional networks [7, 31]
has made it possible to efficiently reconstruct human meshes from 2D
keypoints. Because of the constrained expressive capacity of parametric human
models, these methods typically generate a representation of the human body
with minimal clothing, neglecting the finer details of garments, hair, and
other accessories. Some efforts have been devoted to representing clothing as
an offset layer from the underlying body [2, 1, 44, 30]. While the
introduction of an offset layer has enhanced their representation
capabilities, the gap between these SMPL+D methods and real-world human body
is difficult to bridge.

###  2.2 Implicit Human Models

Implicit human models utilize voxels or implicit functions to describe
geometry, circumventing the need for explicit models and offering a much
larger solution space for capturing intricate details. Voxels representations
[38, 43] demand cubic memory, which hinders these methods from obtaining the
high resolution reconstruction results. Rather than relying on voxels,
implicit functions parameterized by MLPs present a memory efficient way for
representing geometry. Given a spatial point, signed distance fields (SDF)
[24], occupancy [22, 33, 34], or density [23] can be estimated by MLPs, and
triangulate mesh can be extracted by marching cube algorithm [21]. Although
implicit functions approaches offer memory efficient geometry representation,
more training and inference time are required as a trade-off. Training
implicit function-based methods can be time-consuming, as the entire MLP is
updated during each iteration. In the inference phase, high-resolution grid
points are fed into the implicit functions to retrieve geometric information.
A recent work [28] called SAP introduces a differentiable Poisson solver to
bridge oriented point clouds, implicit indicator functions, and meshes
altogether. SAP employs light-weight oriented point clouds to represent
shapes, which greatly reduces training time and inference time.

###  2.3 Human Avatar

Numerous endeavors have been undertaken to create an animatable human avatar
by amalgamating parametric human models and implicit human models [9, 6, 35,
36]. NASA [9] exploits per body-part occupancy networks [22] to represent
articulated human bodies, which can potentially introduce artifacts for unseen
poses. SNARF [6] proposes a forward warping field and incorporates SMPL [20]
skeleton to learn pose-independent skinning. SCANimate [35] utilizes a weakly
supervised learning method that takes raw 3D scans and turns them into an
animatable avatar without surface registration. X-Avatar [36] proposes a part-
aware initialization and sampling strategies to improve the correspondence
search in iterative root finding. SMPL-X [26] skeleton is adopted to model
body pose, hand pose, facial expressions together. Despite achieving promising
results, these methods still face several challenges. The process of acquiring
the final mesh is time-consuming, as it involves passing all grid points in
space through the MLP to retrieve geometric information. Additionally,
implicit representations struggle to preserve semantic information. Also,
there are some works that model dynamic humans from multiview videos [29, 27].
These methods utilize image pixels as their optimization target, which can
effectively generate images of humans in various poses. Nevertheless, they
struggle to capture accurate geometry and exhibit weak generalization when
faced with unseen poses due to the limitations of volume rendering and density
representation.

##  3 Methods

In this section, we introduce our oriented point-based method for human
avatars modeling, which preserves semantic information. Fig. 2 shows the
overview of our proposed method. We firstly introduce the oriented points
shape representation in Section 3.1. Then, we delve the representations of
pose-dependent deformation and skinning weights in Section 3.2. Moreover, we
elaborate on how we transfer semantic information from SMPL-X model to
oriented points in Section 3.3. Additionally, we explore the appearance
representation in Section 3.4. The specific details of our implementation can
be found in Section 3.5.

###  3.1 Shape Representation

Differently from those implicit methods [6, 35, 36], we employ oriented point
clouds $S=\\{\mathbf{x}\in\mathbb{R}^{3},\mathbf{n}\in\mathbb{R}^{3}\\}$ as
shape representation. Oriented point clouds representation is interpretable,
lightweight and fast comparing to implicit representation. Implicit methods
store the mesh within the MLP and necessitate forward inference to query
geometric information at individual positions. We represent the mesh
explicitly through the positions and normals of points. In implicit methods,
it is time consuming to query all grid positions for each pose in order to
acquire occupancy or SDF value and subsequently employ marching cube [21] to
generate final mesh. In contrast, by taking advantage of oriented point clouds
representation, we can intuitively observe pose-dependent deformation and
obtain final mesh through Poisson Surface Reconstruction [14] very
efficiently.

###  3.2 Oriented Points Deformation

As shown in Fig. 3, the oriented points undergo an initial deformation step
using DeltaNet, followed by the estimation of skinning weights through LBSNet.
Subsequently, the points are linear skinned to the pose space according to the
skeleton of SMPL-X model. We start by transforming the points from the
template space to the canonical space and subsequently into the pose space.

Similar to [42], we utilize a coordinate-based MLP to map each sampled point
$\mathbf{x}$ to both an offset and an Euler angle, enabling the modeling of
pose-dependent deformation.

$f_{d}:\mathbb{R}^{3}\times\mathbb{R}^{|\theta_{b}|+|\psi|}\rightarrow\Delta\in\mathbb{R}^{3},\Theta\in\mathbb{R}^{3}$
(1) $\mathbf{x}_{c}=\mathbf{x}+\Delta$ (2)
$\mathbf{n}_{c}=R(\Theta)\mathbf{n}$ (3)

where the offset $\Delta$ and the Euler angle $\Theta$ are conditioned by body
pose $\theta_{b}$, facial expression $\psi$ and points positions $\mathbf{x}$.
$R(\Theta)$ represents the rotation matrix computed from Euler angle. We
incorporate position encoding [23] to model high-frequency details. The
sampled points and normals are firstly displaced by the pose-dependent offset
and rotation matrix, which are further deformed to pose space according to
linear blend skinning.

Figure 3: Points Deformation. The sampled points are employed pose-dependent
deformation through DeltaNet. Then we estimate LBS weights through LBSNet.
Points are firstly skinned to canonical space and then to pose space. Mesh can
be extracted through Poisson reconstruction.

Similar to previous work [6, 41, 36], we employ another MLP to represent the
skinning weight field in the template space

$f_{s}:\mathbb{R}^{3}\rightarrow\mathbb{R}^{N_{b}},$ (4)

where $N_{b}$ represents the number of bones, including body, finger and face
bones. We incorporate linear blend skinning to model the pose transformation.
Specifically, each bone $\text{b}_{i}$ corresponds to a weight $w_{i}$ and a
bone transformations $\text{B}_{i}$. $\text{B}_{i}$ is computed by pose
$\theta$ and kinematic tree of SMPL-X model. For each point $\mathbf{x}_{c}$
in template space, the corresponding point $\mathbf{x}_{d}$ in pose space can
be determined by

$\displaystyle
f_{s}(\mathbf{x}_{c})=\\{w_{1}(\mathbf{x}_{c}),...,w_{N_{b}}(\mathbf{x}_{c})\\}$
(5) $\displaystyle\text{w.r.t.}\quad w_{i}\geq 0\ \text{and}\
\sum\nolimits_{i}w_{i}=1$
$\mathbf{x}_{d}=\sum_{i=1}^{N_{b}}w_{i}(\mathbf{x}_{c})\text{B}_{i}{\text{B}^{c}_{i}}^{-1}\mathbf{x}_{c}$
(6)
$\mathbf{n}_{d}=\sum_{i=1}^{N_{b}}w_{i}(\mathbf{x}_{c})\text{B}_{i}{\text{B}^{c}_{i}}^{-1}\mathbf{n}_{c}$
(7)

where $\text{B}_{i}$ and $\text{B}^{c}_{i}$ represent the bone transformations
corresponding to pose $\theta$ and template pose $\theta_{t}$, respectively.
Since the linear skinning of SMPL-X model is performed in canonical space, we
first transform points to canonical space and then to pose space. The deformed
points $\mathbf{x}_{d}$ are employed to calculate loss with the points sampled
from pose scan $\mathbf{x}_{p}$

$\mathcal{L}=\mathcal{L}_{chamfer}+\mathcal{L}_{EMD}+\mathcal{L}_{normal}+\mathcal{L}_{reg}.$
(8)

We employ regular Chamfer distance as preliminary loss function as follows

$\begin{split}\mathcal{L}_{chamfer}&=\frac{1}{|\mathbf{x}_{d}|}\sum\limits_{i\in|\mathbf{x}_{d}|}\min\limits_{j\in|\mathbf{x}_{p}|}(||\mathbf{x}_{d}^{i}-\mathbf{x}_{p}^{j}||_{2}^{2})\\\
&+\frac{1}{|\mathbf{x}_{p}|}\sum\limits_{j\in|\mathbf{x}_{p}|}\min\limits_{i\in\mathbf{x}_{d}}(||\mathbf{x}_{p}^{j}-\mathbf{x}_{d}^{i}||_{2}^{2}).\end{split}$
(9)

However, only chamfer distance may cause uneven density distribution and
blurred details, which affects the quality of the mesh generated by Poisson
reconstruction. We additionally employ Earth Mover’s Distance (EMD) [19] to
avoid this situation

$\mathcal{L}_{EMD}=\min\limits_{\phi:\mathbf{x}_{d}\rightarrow\mathbf{x}_{p}}\frac{1}{|\mathbf{x}_{d}|}\sum\limits_{i\in\mathbf{x}_{d}}||x_{d}^{i}-\phi(x_{d}^{i})||,$
(10)

where $\phi$ is a bijection. EMD loss minimizes the dissimilarity between two
point sets and force $\mathbf{x}_{d}$ to be evenly distributed since
$\mathbf{x}_{p}$ are sampled evenly. In order to capture clothing details,
normal consistency loss is employed

$\mathcal{L}_{normal}=\frac{1}{|\mathbf{x}_{d}|}\sum\limits_{i\in\mathbf{x}_{d}}1-\cos(\mathbf{n}_{d}^{i},\phi(\mathbf{n}_{d}^{i}))$
(11)

where $n_{d}^{i}$ is the normal vector of point $\mathbf{x}_{d}^{i}$. Cosine
similarity enforces normal consistency. Moreover, we regularize the LBS
weights to be close to those of SMPL-X and constrain the offset to tend to
zero.

$\begin{split}\mathcal{L}_{reg}&=\frac{1}{|\mathbf{x}_{x|}}\sum\limits_{i=1}^{N_{b}}||{w_{i}(\mathbf{x}_{x})}-w_{i}(\mathbf{x}_{x})^{*}||_{2}^{2}\\\
&+\frac{1}{|\mathbf{x}|}||f_{d}(\mathbf{x};\theta_{b},\psi)||_{2}^{2}\end{split}$
(12)

where $\mathbf{x}_{x}$ represents the vertices of registered SMPL-X mesh in
template space. $w_{i}(\mathbf{x}_{x})^{*}$ is the LBS weights defined by
SMPL-X model.

Figure 4: Semantic Transfer. We divide SMPL-X models into 8 parts. We optimize
the point cloud with semantic information sampled from SMPL-X mesh according
to template scan. Then the sampled point cloud can be reposed like SMPL-X
model.

###  3.3 Semantic Transfer

As shown in Fig. 4, once the geometry training is done, we transfer the
semantic information of SMPL-X mesh to point clouds. SMPL-X model [26] is a
parametric human model that is parameterized by shape $\beta$, pose $\theta$
and facial expressions $\psi$.

The semantic information of SMPL-X model is predefined, since the topology of
SMPL-X is fixed. We categorize the SMPL-X points into 8 parts: {head, body,
left arm, right arm, left hand, right hand, left leg, right leg}. The label of
SMPL-X faces are derived from the corresponding points. We fix the relative
position of sampled points to the faces so that the sampled points can
maintain correspondence and semantic information. To align the template scan,
sampled point clouds $\mathbf{x}_{s}$ are employed in the same Chamfer loss
and EMD loss described in Section 3.2. After transferring the points, the
semantic points can be deformed according to different pose parameters. By
leveraging the correspondences, we are able to swap point clouds belonging to
the same label to composite avatar. Once the semantic information of each
point is determined, we freeze the DeltaNet and LBSNet and optimize the
appearance feature.

###  3.4 Appearance Representation

Once the geometry training and semantic transfer are completed, we attach a
learnable feature to each point, serving as its appearance representation. We
first train an autoencoder for color representation as follows

$c=D(E(c)).$ (13)

Subsequently, we freeze the decoder and optimize the feature. We transform the
semantic points to pose space and perform loss calculation with sampled points
at point level. For each sampled point $\mathbf{x}_{p}^{i}$, we aggregate
features from k-nearest semantic points. We employ inverse distance weighting
(IDW) to weight features

$\mathcal{F}_{p}^{i}=\frac{\sum_{j\in\mathcal{N}(i)}d_{i}^{j}\mathcal{F}_{s}^{j}}{\sum_{j\in\mathcal{N}(i)}d_{i}^{j}},$
(14)

where $\mathcal{F}_{s}^{j}$ indicates the learnable feature of point
$\mathbf{x}_{p}^{j}$. $\mathcal{N}(i)$ represents the k-nearest points of
$\mathbf{x}_{p}^{i}$. $d_{i}^{j}$ is the distance between the sampled point
$\mathbf{x}_{p}^{i}$ and semantic point $\mathbf{x}_{s}^{j}$. The aggregated
feature is then passed to the decoder $D$ to obtain the final color and
compute the loss as below

$\mathcal{L}_{color}=\frac{1}{|\mathbf{x}_{p}|}\sum\limits_{i\in|\mathbf{x}_{p}|}||D(\mathcal{F}_{p}^{i})-c_{p}^{i}||_{2}^{2},$
(15)

where $c_{p}^{j}$ is the color of point $\mathbf{x}_{p}^{j}$.

###  3.5 Implementation Details

We implement the proposed method in PyTorch [25]. We train the points
deformation with 100 epochs and train the appearance feature for another 20
epochs. At each iteration, we sample $51,200$ points from the template scan
and pose scan, respectively. We employ Adam optimizer [15] for optimization.
The learning rate for DeltaNet, LBSNet, and feature are $5\times 10^{-4}$,
$1\times 10^{-4}$, $1\times 10^{-3}$, respectively. Pose $\theta_{b}$ and
facial expressions $\psi$ are firstly encoded to $16$ dimensional features
through network, and then concatenated with points‘ positions as input.
DeltaNet consists of 8 layers with hidden layers of width 512, and a skip
connection from the input to the middle layer. LBSNet consists of 5 layers,
with hidden layers of width 128. We also employ position encoding in LBSNet to
capture complex hand and face poses. The dimension of position encoding used
by DeltaNet and LBSNet is 4. We incorporate hierarchical softmax introduced in
SNARF [6] to better estimate LBS weights. The weight of soft blend is $20$.
Color autoencoder consists of 8 layers with hidden layers 256. The dimension
of feature is 16. We employ the Chamfer loss and EMD loss from Kaolin [11] and
[19], respectively. We set the weights of the losses to
$\lambda_{chamfer}=5000,\lambda_{EMD}=5000,\lambda_{normal}=1,\lambda_{reg}=100,\lambda_{color}=10$,
respectively.

##  4 Experiments

In this section, we present the experimental results on GRAB [37] and X-Humans
[36]. We compare our proposed method with the state-of-the-art implicit
methods. Moreover, we conduct ablation studies to evaluate the effectiveness
of proposed losses. Furthermore, we demonstrate the effectiveness of point
representation with semantics in avatar composition.

Table 1: Quantitative results on GRAB dataset

Method | CD$\downarrow$ | CD-MAX $\downarrow$ | NC $\uparrow$ | IoU $\uparrow$
---|---|---|---|---
All | Hands | All | Hands | All | Hands | All | Hands
SCANimate [35] | 2.60 | 8.39 | 54.75 | 54.22 | 0.967 | 0.760 | 0.941 | 0.569
SNARF [6] | 1.37 | 5.13 | 33.86 | 33.51 | 0.977 | 0.818 | 0.967 | 0.739
X-Avatar [36] | 0.94 | 0.79 | 21.43 | 4.79 | 0.985 | 0.957 | 0.991 | 0.895
Ours | 0.85 | 0.77 | 19.76 | 4.71 | 0.985 | 0.962 | 0.993 | 0.901

Figure 5: Qualitative Comparison on GRAB dataset Our method outperforms the
implicit methods [35, 6, 36].

###  4.1 Results on GRAB dataset

GRAB dataset [37] consists of minimally clothed SMPL-X meshes with diverse
hand poses and facial expressions. We follow the same partition strategy as
X-Avatar for evaluation. We evaluate the geometric accuracy via volumetric
IoU, Chamfer distance (CD) and normal consistency (NC) metrics. Volumetric IoU
provides a measure of the overlap between the reconstructed mesh and the
ground-truth mesh. Chamfer distance quantifies the reconstruction accuracy
comparing to the ground-truth mesh. Normal consistency evaluates the fineness
of reconstructed local details. We compare our proposed method against recent
implicit human avatar methods including SCANimate [35], SNARF [6] and X-avatar
[36].

Table 1 presents the quantitative results on GRAB dataset. Our method
demonstrates comparable performance with X-Avatar and surpasses SCANimate and
SNARF. SCANimate and SNARF exhibit subpar performance when it comes to face
and hand reconstruction. Fig. 5 shows the qualitatively results. Both Xavatar
and our method excel in accurately reconstructing both facial expressions and
hand gestures. SCANimate recovers a mean hand and SNARF cannot handle complex
hand poses, resulting in the large errors.

Table 2: Quantitative results on X-avatar dataset. Method | CD$\downarrow$ | CD-MAX $\downarrow$ | NC $\uparrow$ | IoU $\uparrow$ | Time $\downarrow$
---|---|---|---|---|---
All | Hands | All | Hands | All | Hands | All | Hands | Training | Inference
SCANimate [35] | 6.54 | 9.78 | 59.71 | 48.32 | 0.925 | 0.726 | 0.919 | 0.55 | 30 h | 3.0 s
SNARF [6] | 5.05 | 7.23 | 55.06 | 37.15 | 0.934 | 0.788 | 0.937 | 0.608 | 18 h | 3.5 s
X-avatar [36] | 4.43 | 5.14 | 47.56 | 22.15 | 0.939 | 0.793 | 0.965 | 0.776 | 25 h | 4.1 s
Ours | 4.30 | 4.96 | 44.36 | 20.22 | 0.945 | 0.832 | 0.968 | 0.787 |  10 h |  0.1 s
Figure 6: Qualitative Comparison on X-Humans dataset Our method outperforms
these implicit methods.

###  4.2 Results on X-Humans

X-Humans [36] dataset comprises textured 3D clothed scans of humans,
encompassing a wide range of body poses, hand gestures and facial expressions.
These scans are captured using a multi-view volumetric capture stage. A custom
registration pipeline effectively extracts the SMPL-X parameters for each
scan. We adopt the identical evaluation metrics used in GRAB dataset. We also
conduct a comparison of our approach against SCANimate, SNARF and X-avatar.

The quantitative results obtained from the X-Humans dataset are summarized in
Table 2. Our proposed approach successfully attains the lowest Chamfer
distance, highest normal consistency and volumetric IoU, demonstrating
exceptional performance in all three metrics. Fig. 6 shows the qualitatively
results. Similar to the results on GRAB dataset, both SCANimate and SNARF fall
short in accurately capturing hand poses and facial expressions, resulting in
erroneous face and hand results. Although X-Avatar is able to recover correct
facial expressions and hand poses, implicit representations may lead to
artifacts in some extreme poses. Furthermore, it has limited ability to
recover clothing details.

Efficiency Comparison Our proposed method has advantages in terms of both
training time and inference time. SCANimate acquires the consistency between
canonical space and pose space conversions in a weakly supervised manner,
which requires additional optimization of an inverse LBS network. To enable
weak supervision, SCANimate needs to pre-train the LBS network and perform
linear blend skinning twice during training. While it takes long time for
SNARF and X-Avatar to find canonical correspondences which satisfy the forward
skinning equation via iterative root finding. To enable part-aware
initialization, X-Avatar dedicates extra time during the training phase to
compute the category of each point. During inference, these implicit methods
need to query all grid positions to obtain occupancy value and employ marching
cube [21] to extract the final mesh. While multi-scale methods can help
alleviate the computational load, there remains a considerable number of SDF
values that must be calculated at various locations. By taking advantage of
oriented points representation, we can obtain the final mesh efficiently
through Poisson reconstruction. It takes 0.8 seconds when we adopt traditional
CPU-based Poisson reconstruction. By employing the GPU-based Poisson solver
[28] and torchmucbes111https://github.com/tatsy/torchmcubes, the processing
time decreases to 0.1 seconds. All the experiments about computational time
are conducted on the same machine with a single NVIDIA 3090Ti GPU. The
dimension of marching cube used by implicit methods and the depth of Poisson
reconstruction are $512$ and $9$ respectively.

Table 3: Quantitative Ablation Study on the Proposed Loss

Method | CD$\downarrow$ | CD-MAX $\downarrow$ | NC $\uparrow$ | IoU $\uparrow$
---|---|---|---|---
All | Hands | All | Hands | All | Hands | All | Hands
w/o emd | 4.38 | 5.01 | 45.93 | 20.64 | 0.940 | 0.818 | 0.962 | 0.779
w/o chamfer | 4.36 | 5.00 | 44.50 | 20.28 | 0.943 | 0.820 | 0.964 | 0.782
w/o normal | 4.39 | 5.18 | 45.83 | 21.24 | 0.935 | 0.805 | 0.965 | 0.784
w/o reg | 4.43 | 5.27 | 45.53 | 23.78 | 0.940 | 0.823 | 0.960 | 0.770

###  4.3 Ablation Studies

We have performed comprehensive ablation studies to analyze the impact of our
proposed loss functions. The quantitative results are presented in Table 3.
The EMD loss mitigates issues related to uneven point density distribution and
preserves fine details, thereby enhancing the overall quality of the
reconstruction. The uniform distribution facilitated by the EMD loss also
enhance better appearance recovery because we utilize KNN to interpolate the
neural textures. The Chamfer distance loss effectively minimizes discrepancies
between the sampled points and posed points, resulting in improved alignment.
Additionally, the normal loss contributes to enhancing normal consistency and
facilitating the recovery of intricate clothing details. The LBS
regularization loss aids in the precise recovery of hand shapes and reduces
the Chamfer distance within the hand region.

|  |  |  |  |  |  |
---|---|---|---|---|---|---|---
|  |  |  |  |  |  |
Ori Mesh | Texture transfer | Points transfer | Points transfer | Ori Mesh | Texture transfer | Points transfer | Points transfer
Figure 7: Composition results of different avatars on X-Humans Dataset. We
show qualitative results of composited avatars. We can transfer only the
texture or both the texture and the point clouds.

###  4.4 Avatar Composition

Another advantage of oriented points representation is its ability to preserve
semantic information for avatar composition. Despite utilizing part-aware
initialization and sampling strategies, implicit methods fall short in
obtaining the semantic information of the reconstructed human meshes during
inference. Every vertex is handled without discrimination. As a consequence,
implicit methods need to train a distinct model for each subject.

Our proposed novel method enriches points with semantic information from
SMPL-X model, enabling us to identify the category associated with each vertex
in the reconstructed meshes. Utilizing these categorizations, we can conduct a
more thorough analysis of human movements. The inherent benefits of Poisson
reconstruction enable us to seamlessly merge disparate points. Through
exchanging textures and points belonging to the same category across different
individuals, our approach enables virtual try-on and the composition of human
avatars. To preserve the clothing geometry, we can either exclusively
interchange the neural texture of the points or simultaneously swap both the
points and texture to create a composite human avatar. Fig. 7 shows the
results of composited avatars. The promising results indicate that our method
is capable of generating realistic human bodies in both texture transfer and
points transfer scenarios.

##  5 Limitations and Conclusions

By transferring the semantic information of the SMPL-X model to the clothed
avatar, our proposed method encounters challenges in handling loose clothing,
such as long skirts. Moreover, Poisson reconstruction [14] aims to create a
surface that minimizes the differences in normals at neighboring vertices,
which tends to produce smoothing meshes.

In this paper, we introduce an efficient point based human avatar that
comprehensively captures poses, expressions and appearances. We utilize two
MLPs to model pose-dependent deformation and estimate LBS weights. Poisson
reconstruction provides an efficient way to transform the oriented points into
meshes in a timely manner. Comparing to implicit representation, our oriented
point clouds representation provides interpretability to the avatar model.
Furthermore, we propose a novel method for transferring the semantic
information from SMPL-X model to the point clouds, which enables to create
avatars by compositing different subjects. Experimental results demonstrate
that our proposed approach performs comparable to the state-of-the-art
implicit approaches while requiring less training and inference time.

## References

  * [1] Thiemo Alldieck, Marcus A. Magnor, Bharat Lal Bhatnagar, Christian Theobalt, and Gerard Pons-Moll.  Learning to reconstruct people in clothing from a single RGB camera.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 1175–1186, 2019.
  * [2] Thiemo Alldieck, Marcus A. Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll.  Detailed human avatars from monocular video.  In International Conference on 3D Vision, 3DV, pages 98–109, 2018\.
  * [3] Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis.  SCAPE: shape completion and animation of people.  ACM Trans. Graph., 24(3):408–416, 2005.
  * [4] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3d faces.  In SIGGRAPH, pages 187–194, 1999.
  * [5] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter V. Gehler, Javier Romero, and Michael J. Black.  Keep it SMPL: automatic estimation of 3d human pose and shape from a single image.  In Proc. of the European Conf. on Computer Vision (ECCV), volume 9909, pages 561–578, 2016.
  * [6] Xu Chen, Yufeng Zheng, Michael J. Black, Otmar Hilliges, and Andreas Geiger.  SNARF: differentiable forward skinning for animating non-rigid neural implicit shapes.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2021.
  * [7] Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee.  Pose2mesh: Graph convolutional network for 3d human pose and mesh recovery from a 2d human pose.  In Proc. of the European Conf. on Computer Vision (ECCV), volume 12352, pages 769–787, 2020.
  * [8] Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black.  Monocular expressive body regression through body-driven attention.  In Proc. of the European Conf. on Computer Vision (ECCV), volume 12355, pages 20–40, 2020.
  * [9] Boyang Deng, John P. Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey E. Hinton, Mohammad Norouzi, and Andrea Tagliasacchi.  NASA neural articulated shape approximation.  In Proc. of the European Conf. on Computer Vision (ECCV), volume 12352, pages 612–628, 2020.
  * [10] Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black.  Collaborative regression of expressive bodies using moderation.  In International Conference on 3D Vision, 3DV, pages 792–804, 2021\.
  * [11] Clement Fuji Tsang, Maria Shugrina, Jean Francois Lafleche, Towaki Takikawa, Jiehan Wang, Charles Loop, Wenzheng Chen, Krishna Murthy Jatavallabhula, Edward Smith, Artem Rozantsev, Or Perel, Tianchang Shen, Jun Gao, Sanja Fidler, Gavriel State, Jason Gorski, Tommy Xiang, Jianing Li, Michael Li, and Rev Lebaredian.  Kaolin: A pytorch library for accelerating 3d deep learning research.  https://github.com/NVIDIAGameWorks/kaolin, 2022.
  * [12] Hanbyul Joo, Tomas Simon, and Yaser Sheikh.  Total capture: A 3d deformation model for tracking faces, hands, and bodies.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 8320–8329, 2018.
  * [13] Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 7122–7131, 2018.
  * [14] Michael M. Kazhdan, Matthew Bolitho, and Hugues Hoppe.  Poisson surface reconstruction.  In Proceedings of the Fourth Eurographics Symposium on Geometry Processing (SGP), 2006.
  * [15] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Int. Conf. Learn. Represent., 2015.
  * [16] Muhammed Kocabas, Chun-Hao P. Huang, Otmar Hilliges, and Michael J. Black.  PARE: part attention regressor for 3d human body estimation.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), pages 11107–11117, 2021.
  * [17] Nikos Kolotouros, Georgios Pavlakos, Michael J. Black, and Kostas Daniilidis.  Learning to reconstruct 3d human pose and shape via model-fitting in the loop.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), pages 2252–2261, 2019.
  * [18] Tianye Li, Timo Bolkart, Michael J. Black, Hao Li, and Javier Romero.  Learning a model of facial shape and expression from 4d scans.  ACM Trans. Graph., 36(6):194:1–194:17, 2017.
  * [19] Minghua Liu, Lu Sheng, Sheng Yang, Jing Shao, and Shi-Min Hu.  Morphing and sampling network for dense point cloud completion.  In AAAI, pages 11596–11603, 2020.
  * [20] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  SMPL: a skinned multi-person linear model.  ACM Trans. Graph., 34(6):248:1–248:16, 2015.
  * [21] William E. Lorensen and Harvey E. Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  In SIGGRAPH, pages 163–169. ACM, 1987.
  * [22] Lars M. Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 4460–4470, 2019.
  * [23] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In Proc. of the European Conf. on Computer Vision (ECCV), pages 405–421, 2020.
  * [24] Jeong Joon Park, Peter R. Florence, Julian Straub, Richard A. Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 165–174, 2019.
  * [25] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
  * [26] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black.  Expressive body capture: 3d hands, face, and body from a single image.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 10975–10985, 2019.
  * [27] Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao.  Animatable neural radiance fields for modeling dynamic human bodies.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), pages 14294–14303, 2021.
  * [28] Songyou Peng, Chiyu ”Max” Jiang, Yiyi Liao, Michael Niemeyer, Marc Pollefeys, and Andreas Geiger.  Shape as points: A differentiable poisson solver.  In Advances in Neural Information Processing Systems (NeurIPS), 2021\.
  * [29] Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou.  Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [30] Gerard Pons-Moll, Sergi Pujades, Sonny Hu, and Michael J. Black.  Clothcap: seamless 4d clothing capture and retargeting.  ACM Trans. Graph., 36(4):73:1–73:15, 2017.
  * [31] Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J. Black.  Generating 3d faces using convolutional mesh autoencoders.  In Proc. of the European Conf. on Computer Vision (ECCV), volume 11207, pages 725–741, 2018.
  * [32] Javier Romero, Dimitrios Tzionas, and Michael J. Black.  Embodied hands: modeling and capturing hands and bodies together.  ACM Trans. Graph., 36(6):245:1–245:17, 2017.
  * [33] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Hao Li, and Angjoo Kanazawa.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), pages 2304–2314, 2019.
  * [34] Shunsuke Saito, Tomas Simon, Jason M. Saragih, and Hanbyul Joo.  Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 81–90, 2020.
  * [35] Shunsuke Saito, Jinlong Yang, Qianli Ma, and Michael J. Black.  Scanimate: Weakly supervised learning of skinned clothed avatar networks.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 2886–2897, 2021.
  * [36] Kaiyue Shen, Chen Guo, Manuel Kaufmann, Juan Zarate, Julien Valentin, Jie Song, and Otmar Hilliges.  X-avatar: Expressive human avatars.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [37] Omid Taheri, Nima Ghorbani, Michael J. Black, and Dimitrios Tzionas.  GRAB: A dataset of whole-body human grasping of objects.  In Proc. of the European Conf. on Computer Vision (ECCV), volume 12349, pages 581–600, 2020.
  * [38] Gül Varol, Duygu Ceylan, Bryan C. Russell, Jimei Yang, Ersin Yumer, Ivan Laptev, and Cordelia Schmid.  Bodynet: Volumetric inference of 3d human body shapes.  In Proc. of the European Conf. on Computer Vision (ECCV), volume 11211, pages 20–38, 2018.
  * [39] Donglai Xiang, Hanbyul Joo, and Yaser Sheikh.  Monocular total capture: Posing face, body, and hands in the wild.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 10965–10974, 2019.
  * [40] Hongwen Zhang, Yating Tian, Xinchi Zhou, Wanli Ouyang, Yebin Liu, Limin Wang, and Zhenan Sun.  Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), pages 11426–11436, 2021.
  * [41] Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C. Bühler, Xu Chen, Michael J. Black, and Otmar Hilliges.  I M avatar: Implicit morphable head avatars from videos.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 13535–13545, 2022.
  * [42] Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael J. Black, and Otmar Hilliges.  Pointavatar: Deformable point-based head avatars from videos.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 21057–21067, 2023.
  * [43] Zerong Zheng, Tao Yu, Yixuan Wei, Qionghai Dai, and Yebin Liu.  Deephuman: 3d human reconstruction from a single image.  In Proc. of the IEEE International Conf. on Computer Vision (ICCV), pages 7738–7748, 2019.
  * [44] Hao Zhu, Xinxin Zuo, Sen Wang, Xun Cao, and Ruigang Yang.  Detailed human shape estimation from a single image by hierarchical mesh deformation.  In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 4491–4500, 2019.


* Zhang  _et al._ [2022] J. Zhang, C. Zhang, J. Yang, and B. Capogrosso-Sansone, Supersolid phases of lattice dipoles tilted in three dimensions, Phys. Rev. A 105, 063302 (2022).
  * Bandyopadhyay  _et al._ [2022] S. Bandyopadhyay, H. Sable, D. Gaur, R. Bai, S. Mukerjee, and D. Angom, Quantum phases of dipolar bosons in a multilayer optical lattice, Phys. Rev. A 106, 043301 (2022).
  * Hebib  _et al._ [2023] Y. Hebib, C. Zhang, J. Yang, and B. Capogrosso-Sansone, Quantum phases of lattice dipolar bosons coupled to a high-finesse cavity, Phys. Rev. A 107, 043318 (2023).
  * Nishino and Okunishi [1996] T. Nishino and K. Okunishi, Corner Transfer Matrix Renormalization Group Method, J. Phys. Soc. Jpn. 65, 891 (1996).
  * Orús and Vidal [2009] R. Orús and G. Vidal, Simulation of two-dimensional quantum systems on an infinite lattice revisited: Corner transfer matrix for tensor contraction, Phys. Rev. B 80, 094403 (2009).
  * Corboz  _et al._ [2014] P. Corboz, T. M. Rice, and M. Troyer, Competing States in the $t$-${J}$ Model: Uniform $d$-Wave State versus Stripe State, Phys. Rev. Lett. 113, 046402 (2014).
  * Liao  _et al._ [2019] H.-J. Liao, J.-G. Liu, L. Wang, and T. Xiang, Differentiable Programming Tensor Networks, Phys. Rev. X 9, 031041 (2019).
  * Hasik  _et al._ [2020] J. Hasik, G. B. Mbeng, W.-L. Tu, and S.-S. Diop, A tensor network library for two-dimensional lattice models, https://github.com/jurajHasik/peps-torch (2020).
  * Tu  _et al._ [2022] W.-L. Tu, E.-G. Moon, K.-W. Lee, W. E. Pickett, and H.-Y. Lee, Field-induced Bose-Einstein condensation and supersolid in the two-dimensional Kondo necklace, Commun. Phys. 5, 130 (2022).
  * Tu  _et al._ [2023] W.-L. Tu, X. Lyu, S. R. Ghazanfari, H.-K. Wu, H.-Y. Lee, and N. Kawashima, Cubic ferromagnet and emergent ${U}(1)$ symmetry on its phase boundary, Phys. Rev. B 107, 224406 (2023).