# XAGen: 3D Expressive Human Avatars Generation

Zhongcong Xu
Show Lab
National University of Singapore
<EMAIL_ADDRESS>
&Jianfeng Zhang
ByteDance
<EMAIL_ADDRESS>
&Jun Hao Liew
ByteDance
<EMAIL_ADDRESS>
&Jiashi Feng
ByteDance
<EMAIL_ADDRESS>
&Mike Zheng Shou
Show Lab
National University of Singapore
<EMAIL_ADDRESS>Corresponding author

###### Abstract

Recent advances in 3D-aware GAN models have enabled the generation of
realistic and controllable human body images. However, existing methods focus
on the control of major body joints, neglecting the manipulation of expressive
attributes, such as facial expressions, jaw poses, hand poses, and so on. In
this work, we present XAGen, the first 3D generative model for human avatars
capable of expressive control over body, face, and hands. To enhance the
fidelity of small-scale regions like face and hands, we devise a multi-scale
and multi-part 3D representation that models fine details. Based on this
representation, we propose a multi-part rendering technique that disentangles
the synthesis of body, face, and hands to ease model training and enhance
geometric quality. Furthermore, we design multi-part discriminators that
evaluate the quality of the generated avatars with respect to their appearance
and fine-grained control capabilities. Experiments show that XAGen surpasses
state-of-the-art methods in terms of realism, diversity, and expressive
control abilities. Code and data will be made available at
https://showlab.github.io/xagen.

##  1 Introduction

3D avatars present an opportunity to create experiences that are exceptionally
authentic and immersive in telepresence [10], augmented reality (AR) [22], and
virtual reality (VR) [50]. These applications [1, 52, 3, 35] require the
capture of human expressiveness, including poses, gestures, expressions, and
others, to enable photo-realistic generation [65, 70], animation [56], and
interaction [33] in virtual environments.

Traditional methods [11, 60, 4, 20, 23] typically create virtual avatars based
on template registration or expensive multi-camera light stages in well-
controlled environments. Recent efforts [69, 43, 5, 26, 16] have explored the
use of generative models to produce 3D human bodies and clothing based on
input parameters, such as SMPL [38], without the need of 3D supervision.
Despite these advancements, current approaches are limited in their ability to
handle expressive attributes of the human body, such as facial expressions and
hand poses, as they primarily focus on body pose and shape conditions. Yet,
there exist scenarios where fine-grained control ability is strongly desired,
_e.g._ , performing social interactions with non-verbal body languages in
Metaverse, or driving digital characters to talk with various expressions and
gestures, etc. Due to the lack of comprehensive modeling of the full human
body, existing approaches [43, 5, 26] fail to provide control ability beyond
the sparse joints of major body skeleton, leading to simple and unnatural
animation.

Figure 1: XAGen can synthesize realistic 3D avatars with detailed geometry,
while providing disentangled control over expressive attributes, _i.e._ ,
facial expressions, jaw, body, and hand poses.

In this work, our objective is to enhance the fine-grained control
capabilities of GAN-based human avatar generation model. To achieve this, we
introduce the first eXpressive 3D human Avatar Generation model (XAGen) that
can (1) synthesize high-quality 3D human avatars with diverse realistic
appearances and detailed geometries; (2) provide independent control
capabilities for fine-grained attributes, including body poses, hand poses,
jaw poses, shapes, and facial expressions.

XAGen is built upon recent unconditional 3D-aware generation models for static
images [7, 44]. One straightforward approach to implement fully animatable
avatar generation is extending 3D GAN models to condition on expressive
control signals, such as SMPL-X [47]. Though conceptually simple, such a
direct modification of conditioning signal cannot guarantee promising
appearance quality and control ability, particularly for two crucial yet
challenging regions, _i.e._ , the face and hands. This is because (1) Compared
with body, face and hands contain similar or even more articulations. In
addition, their scales are much smaller than arms, torso, and legs in a human
body image, which hinders the gradient propagation from supervision. (2) Face
and hands are entangled with the articulated human body and thus will be
severely affected by large body pose deformation, leading to optimization
difficulty when training solely on full-body image collections.

To address the above challenges, we decompose the learning process of body,
face, and hands by adopting a multi-scale and multi-part 3D representation and
rendering multiple parts independently using their respective observation
viewpoints and control parameters. The rendered images are passed to multi-
part discriminators, which provide multi-scale supervision during the training
process. With these careful designs, XAGen can synthesize photo-realistic 3D
human avatars that can be animated effectively by manipulating the
corresponding control parameters for expressions and poses, as depicted in
Figure 1. We conduct extensive experiments on a variety of benchmarks [18, 68,
14, 36], demonstrating the superiority of XAGen over state-of-the-arts in
terms of appearance, geometry, and controllability. Moreover, XAGen supports
various downstream applications such as text-guided avatar creation and audio-
driven animation, expanding its potential for practical scenarios.

Our contributions are three-fold: (1) To the best of our knowledge, XAGen is
the first 3D GAN model for fully animatable human avatar generation. (2) We
propose a novel framework that incorporates multi-scale and multi-part 3D
representation together with multi-part rendering technique to enhance the
quality and control ability, particularly for the face and hands. (3)
Experiments demonstrate XAGen surpasses state-of-the-art methods in terms of
both quality and controllability, which enables various downstream
applications, including text-guided avatar synthesis and audio-driven
animation.

##  2 Related work

Generative models for avatar creation. Generative models [27, 28, 51] have
demonstrated unprecedented capability for synthesizing high-resolution photo-
realistic images. Building upon these generative models, follow-up works [7,
44, 55, 59, 63] have focused on extending 2D image generation to the 3D domain
by incorporating neural radiance field [42] or differentiable rasterization
[29]. Although enabling 3D-aware generation, these works fail to provide
control ability to manipulate the synthesized portrait images. To address this
limitation, recent research efforts [64, 26, 43, 69, 57, 16, 71] have explored
animatable 3D avatar generation leveraging parametric models for face [32] and
body [38]. These works employ inverse [31] or forward [9] skinning techniques
to control the facial attributes or body poses of the generated canonical
avatars [69, 71]. For human body avatars, additional challenges arise due to
their articulation properties. Consequently, generative models for human
avatars have explored effective 3D representation designs. Among them, ENARF
[43] divides an efficient 3D representation [7] into multiple parts, with each
part representing one bone. EVA3D [26] employs a similar multi-part design by
developing a compositional neural radiance field. Despite enabling body
control, such representation fails to generate the details of human faces or
hands since these parts only occupy small regions in the human body images.

Our method differs in two aspects. First, existing works can either control
face or body, whereas ours is the first 3D avatar generation model with
simultaneous fine-grained control over the face, body, and hands. Second, we
devise a multi-scale and multi-part 3D representation, allowing for generating
human body with high fidelity even for small regions like face and hands.

Expressive 3D human modeling. Existing 3D human reconstruction approaches can
be categorized into two main categories depending on whether explicit or
implicit representations are used. Explicit representations mainly utilize the
pre-defined mesh topology, such as statistical parametric models [38, 62, 45,
2] or personalized mesh registrations [19, 12], to model naked human bodies
with various poses and shapes. To enhance the expressiveness, recent works
have developed expressive statistical models capable of representing details
beyond major human body [47, 46, 17] or introduced the surface deformation to
capture fine-grained features [30, 58]. On the other hand, leveraging the
remarkable advances in implicit neural representations [41, 42], another line
of research has proposed to either rely purely on implicit representations
[53] or combine it with statistical models [61, 48, 8] to reconstruct
expressive 3D human bodies. The most recent work [15, 54] proposed to learn a
single full-body avatar from multi-part portrait videos or 3D scans. In
contrast, our approach focuses on developing 3D generative model for fully
animatable human avatars, which is trainable on only unstructured 2D image
collections.

##  3 Method

In this section, we introduce XAGen, a 3D generative model for synthesizing
photo-realistic human avatars with expressive and disentangled controllability
over facial expression, shape, jaw pose, body pose, and hand pose. Figure 2
depicts the pipeline of our method.

Figure 2:  Pipeline of XAGen. Given a random noise $\mathbf{z}$, the canonical
generator synthesizes the avatar in the format of canonical multi-part and
multi-scale Tri-planes given the corresponding camera pose $c_{\text{b}}$. We
then deform the canonical avatar under the guidance of control parameters
$p_{\text{*}}$ to render multi-part images using respective camera poses
$c_{\text{*}}$ and upsample the images using a super-resolution module.
Discriminators encode the output images, camera poses, and control parameters
into real or fake probabilities to critique the rendered images. IS represents
inverse skinning.

Given a random noise $\mathbf{z}$ sampled from Gaussian distribution, XAGen
first synthesizes a human avatar with canonical body, face, and hand
configurations. In this work, we use X-pose [34] and neutral shape, face, and
hand as canonical configurations. We leverage Tri-plane [7] as the fundamental
building block of 3D representation in our canonical generator. To increase
the capability of 3D representation for the smaller-scale face and hands, we
introduce multi-part and multi-scale designs into the canonical Tri-plane
(Sec. 3.1). A mapping network first encodes $\mathbf{z}$ and the camera
viewpoint of body $c_{\text{b}}$ into latent code $\mathbf{w}$. The canonical
generator then synthesizes three Tri-planes $\mathcal{F}_{k}$ conditioned on
$\mathbf{w}$, where $k\in\\{\text{b},\text{f},\text{h}\\}$ which stands for
$\\{\text{body},\text{face},\text{hand}\\}$.

Based on the generated canonical avatar, we deform it from canonical space to
observation space under the guidance of control signal $p_{\text{b}}$
parameterized by an expressive statistical full body model, _i.e._ , SMPL-X
[47]. We adopt volumetric rendering [39] to synthesize the full body image.
However, due to the scale imbalance between the face/hands and body, rendering
only the full body image cannot guarantee quality for these detailed regions.
To address this issue, we propose a multi-part rendering technique (Sec. 3.2).
Specifically, we employ part-aware deformation and rendering based on the
control parameters ($p_{\text{f}}$ and $p_{\text{h}}$) and cameras
($c_{\text{f}}$ and $c_{\text{h}}$). Accordingly, to ensure the plausibility
and controllability of the generated avatars, we develop multi-part
discriminators to critique the rendered images (Sec. 3.3).

###  3.1 Multi-scale and Multi-part Representation

XAGen is designed for expressive human avatars with an emphasis on the high-
quality face and hands. However, the scale imbalance between face/hands and
body may hamper the fidelity of the corresponding regions. To address this
issue, we propose a simple yet effective multi-scale and multi-part
representation for expressive human avatar generation. Our multi-scale
representation builds upon the efficient 3D representation, _i.e._ , Tri-plane
[7], which stores the generated features on three orthogonal planes.
Specifically, we design three Tri-planes for body, face, and hands, denoted as
$\mathcal{F}_{\text{b}}\in\mathbb{R}^{W_{\text{b}}\times W_{\text{b}}\times
3C}$, $\mathcal{F}_{\text{f}}\in\mathbb{R}^{W_{\text{f}}\times
W_{\text{f}}\times 3C}$, and
$\mathcal{F}_{\text{h}}\in\mathbb{R}^{W_{\text{h}}\times W_{\text{h}}\times
3C}$, respectively. The size of the face and hand Tri-planes is set to half of
the body Tri-plane, with $W_{\text{f}}=W_{\text{h}}=W_{\text{b}}/2$.

As depicted in Figure 2, our canonical generator first synthesizes a compact
feature map $\mathcal{F}\in\mathbb{R}^{W_{\text{b}}\times W_{\text{b}}\times
9C/2}$, where $C$ represents the number of channels. We then separate and
reshape $\mathcal{F}$ into $\mathcal{F}_{k}$, where
$k\in\\{\text{b},\text{f},\text{h}\\}$, representing the canonical space of
the generated human avatar. Furthermore, to save computation cost, we exploit
the symmetry property of hands to represent both left and right hands using
one single $\mathcal{F}_{\text{h}}$ through a horizontal flip operation (refer
to Appendix for details).

###  3.2 Multi-part Rendering

Our method is trainable on unstructured 2D human images. Although this largely
reduces the difficulty and cost to obtain data, the training is highly under-
constrained due to the presence of diverse poses, faces, and clothes. To
facilitate the training process and improve the appearance quality, we propose
a multi-part rendering strategy. This strategy allows XAGen to learn each part
based on the independent camera poses, which further enhances the geometry
quality of the face and hands.

Specifically, for each training image, we utilize a pretrained model [17] to
estimate SMPL-X parameters $\\{p_{\text{b}},p_{\text{f}},p_{\text{h}}\\}$ and
camera poses $\\{c_{\text{b}},c_{\text{f}},c_{\text{h}}\\}$ for body, face,
and hands, respectively. In the rendering stage, we shoot rays using
$\\{c_{\text{b}},c_{\text{f}},c_{\text{h}}\\}$ and sample points
$\\{\mathbf{x}_{\mathbf{o}}^{\text{b}},\mathbf{x}_{\mathbf{o}}^{\text{f}},\mathbf{x}_{\mathbf{o}}^{\text{h}}\\}$
along the rays in the observation space. To compute the feature for each
point, we employ inverse linear-blend skinning [31], which finds the
transformation of each point from observation space to canonical space
produced by the canonical generator. Based on the parameter $p_{k}$, where
$k\in\\{\text{b},\text{f},\text{h}\\}$, SMPL-X yields an expressive human body
model $(\mathbf{v},\mathbf{w})$, where $\mathbf{v}\in\mathbb{R}^{N\times 3}$
represents $N$ vertices, and $\mathbf{w}\in\mathbb{R}^{N\times J}$ represents
the skinning weights of each vertex with respect to joint $J$. For each point
$\mathbf{x}_{\mathbf{o}}^{k,i}$, where $i=1\cdots M_{k}$ and $M_{k}$ is the
number of sampled points, we find its nearest neighbour $\mathbf{n}$ from
vertices $\mathbf{v}$. We then compute the corresponding transformation from
observation space to canonical space

$T^{k,i}=(\sum_{j}\mathbf{w}_{j}^{\mathbf{n}}\begin{bmatrix}R_{j}&t_{j}\\\
\boldsymbol{0}&\boldsymbol{1}\end{bmatrix}\begin{bmatrix}\boldsymbol{I}&\Delta^{\mathbf{n}}\\\
\boldsymbol{0}&\boldsymbol{1}\end{bmatrix})^{-1},$ (1)

where $j=1\cdots J$, $R_{j}$ and $t_{j}$ are derived from $p_{k}$ with
Rodrigues formula [6], and $\Delta^{\mathbf{n}}$ represents the offset caused
by pose and shape for vertex $\mathbf{n}$, which is calculated by SMPL-X.
Based on this inverse transformation, we can calculate the coordinates for
each point in canonical space $\mathbf{x}_{\mathbf{c}}^{k,i}$ as

$\mathbf{x}_{\mathbf{c}}^{k,i}=T^{k,i}\mathbf{x}_{\mathbf{o}}^{k,i},$ (2)

where we apply homogeneous coordinates for the calculation.

For the face and hands rendering, _i.e._ , $k\in\\{\text{f},\text{h}\\}$, we
directly interpolate their corresponding Tri-plane $\mathcal{F}_{\text{f}}$
and $\mathcal{F}_{\text{h}}$ to compute the feature
$\mathbf{f}_{\mathbf{c}}^{\text{f},i}$ and
$\mathbf{f}_{\mathbf{c}}^{\text{h},i}$. Regarding the body rendering, we first
define three bounding boxes
$\mathbb{B}_{\text{f}},\mathbb{B}_{\text{lh}},\mathbb{B}_{\text{rh}}$ for
face, left and right hands in canonical body space. Then, we query canonical
body points that are outside these bounding boxes from body Tri-plane
$\mathcal{F}_{\text{b}}$, while the canonical points inside these boxes from
$\mathcal{F}_{\text{f}}$ and $\mathcal{F}_{\text{h}}$. The query process for
body point $\mathbf{x}_{\mathbf{c}}^{\text{b},i}$ is mathematically formulated
as

$\mathbf{f}_{\mathbf{c}}^{\text{b},i}=\begin{cases}Q(\mathbf{x}_{\mathbf{c}}^{\text{b},i},\mathcal{F}_{\text{f}}),&\text{if}~{}\mathbf{x}_{\mathbf{c}}^{\text{b},i}\in\mathbb{B}_{\text{f}},\\\
Q(\mathbf{x}_{\mathbf{c}}^{\text{b},i},\mathcal{F}_{\text{h}}),&\text{if}~{}\mathbf{x}_{\mathbf{c}}^{\text{b},i}\in\\{\mathbb{B}_{\text{rh}},\mathbb{B}_{\text{lh}}\\},\\\
Q(\mathbf{x}_{\mathbf{c}}^{\text{b},i},\mathcal{F}_{\text{b}}),&\text{if}~{}\mathbf{x}_{\mathbf{c}}^{\text{b},i}\notin\\{\mathbb{B}_{\text{f}},\mathbb{B}_{\text{lh}},\mathbb{B}_{\text{rh}}\\},\end{cases}$
(3)

where $Q$ denotes querying the feature for the given point from the
corresponding Tri-planes.

Once the features $\mathbf{f}_{\mathbf{c}}^{k,i}$ are obtained, they are
encoded into color $\mathbf{c}$ and geometry $d$ via two lightweight multi-
layer perceptrons (MLP), where
$\mathbf{c}=\text{MLP}_{\text{c}}(\mathbf{f}_{\mathbf{c}}^{k,i})$. Inspired by
prior works [44, 26, 69], we employ signed distance field (SDF) as a proxy to
model geometry. Additionally, following [26, 69], we also query a base SDF
$d_{\mathbf{c}}$ in the canonical space, and predict delta SDF, such that
$d=d_{\mathbf{c}}+\text{MLP}_{d}(\mathbf{f}_{\mathbf{c}}^{k,i},d_{\mathbf{c}})$.
We then convert the SDF value into density
$\sigma=\frac{1}{\alpha}\text{Sigmoid}(\frac{-d}{\alpha})$ for volume
rendering, where $\alpha$ is a learnable parameter.

To handle the body features queried from multiple Tri-planes, we apply feature
composition on RGB and density using a window function [37] for smoothness
transition. Specifically, if point $\mathbf{x}_{\mathbf{c},\text{b}}^{k,i}$ is
located in the overlapping region between the body and other parts (face,
right hand, and left hand), their features are sampled from both Tri-planes
and linearly blended together. More details on the feature composition can be
found in the Appendix. Finally, volume rendering is applied to synthesize raw
images for body, face, and hands, denoted as
$\\{I_{\text{b}}^{\text{raw}},I_{\text{f}}^{\text{raw}},I_{\text{h}}^{\text{raw}}\\}$.
These raw images are then upsampled into high-resolution images
$\\{I_{\text{b}},I_{\text{f}},I_{\text{h}}\\}$ by a super-resolution module.

###  3.3 Multi-part Discriminators

Based on the images synthesized by XAGen generator, we design a discriminator
module to critique the generation results. To ensure both the fine-grained
fidelity of appearance and geometry as well as disentangled control over the
full body, including face and hands, we introduce multi-part discriminators to
encode images $\\{I_{\text{b}},I_{\text{f}},I_{\text{h}}\\}$ into real-fake
scores for adversarial training. As depicted in Figure 2, these discriminators
are conditioned on the respective camera poses to encode 3D priors, resulting
in improved geometries as demonstrated in our experiments. To enhance the
control ability of the face and hands, we further condition face discriminator
on expression and shape parameters
$[p_{\text{f}}^{\psi},p_{\text{f}}^{\beta}]$, and condition hand discriminator
on hand pose $p_{\text{h}}^{\theta}$. We encode the camera pose and condition
parameters into intermediate embeddings by two separate MLPs and pass them to
the discriminators. The multi-part discriminator is formulated as

$s_{k}=\mathcal{D}_{k}(I_{k},\text{MLP}_{k}^{c}(c_{k})+\text{MLP}_{k}^{p}(p^{\prime}_{k})),\text{where}~{}p^{\prime}_{k}=\begin{cases}\varnothing,&\text{if}~{}k=\text{b}\\\
[p_{\text{f}}^{\psi},p_{\text{f}}^{\beta}],&\text{if}~{}k=\text{f}\\\
p_{\text{h}}^{\theta},&\text{if}~{}k=\text{h}\end{cases}.$ (4)

Here $s_{k}$ denotes the probability of each image $I_{k}$ being sampled from
real data, and $\mathcal{D}_{k}$ refers to the discriminator corresponding to
the specific body part $k$. For body part, no conditioning parameters are used
because we empirically find that the condition for body hinders the learning
of appearance.

###  3.4 Training Losses

The non-saturating GAN loss [21] is computed for each discriminator, resulting
in $L_{\text{b}}$, $L_{\text{f}}$, and $L_{\text{h}}$. We also regularize
these discriminators using R1 regularization loss [40] $L_{\text{R1}}$. To
improve the plausibility and smoothness of geometry, we compute minimal
surface loss $L_{\text{Minsurf}}$, Eikonal loss $L_{\text{Eik}}$, and human
prior regularization loss $L_{\text{Prior}}$ as suggested in previous works
[44, 69].

Due to the occlusion in the full body images, some training samples may not
contain visible faces or hands. Thus, we balance the loss terms for both
generator and discriminator based on the visibility of face
$\mathcal{M}_{\text{f}}$ and hands $\mathcal{M}_{\text{h}}$, which denote
whether face and hands are detected or not. The overall loss term of XAGen is
formulated as

$\displaystyle
L_{\mathcal{G}}=~{}L_{\text{b}}^{\mathcal{G}}+\lambda_{\text{f}}$
$\displaystyle\mathcal{M}_{\text{f}}\odot
L_{\text{f}}^{\mathcal{G}}+\lambda_{\text{h}}\mathcal{M}_{\text{h}}^{\mathcal{G}}\odot
L_{\text{h}}+\lambda_{\text{Minsurf}}L_{\text{Minsurf}}+\lambda_{\text{Eik}}L_{\text{Eik}}+\lambda_{\text{Prior}}L_{\text{Prior}},$
(5) $\displaystyle L_{\mathcal{D}}={}$ $\displaystyle
L_{\text{b}}^{\mathcal{D}}+L_{\text{R1}}^{\text{b}}+\lambda_{\text{f}}\mathcal{M}_{\text{f}}\odot(L_{\text{f}}^{\mathcal{D}}+L_{\text{R1}}^{\text{f}})+\lambda_{\text{h}}\mathcal{M}_{\text{h}}\odot(L_{\text{h}}^{\mathcal{D}}+L_{\text{R1}}^{\text{h}}),$

where $\odot$ means instance-wise multiplication, and $\lambda_{*}$ are the
weighting factors for each term.

##  4 Experiments

We evaluate the performance of XAGen on four datasets, _i.e._ , DeepFashion
[36], MPV [68], UBC [14], and SHHQ [18]. These datasets contain diverse full
body images of clothed individuals. For each image in the dataset, we process
it to obtain aligned body, face and hand crops, and their corresponding camera
poses and SMPL-X parameters. Please refer to Appendix for more details.

###  4.1 Comparisons

Baselines. We compare XAGen with four state-of-the-art 3D GAN models for
animatable human image generation: ENARF [43], EVA3D [26], AvatarGen [69], and
AG3D [16]. All these methods utilize 3D human priors to enable the
controllability of body pose. ENARF conditions on sparse skeletons, while
others condition on SMPL [38] model. Additionally, AvatarGen and AG3D
incorporate an extra face discriminator to enhance face quality. We adopt the
official implementations of ENARF and EVA3D, and cite results from AG3D
directly. As for AvatarGen, it is reproduced and conditioned on SMPL-X to
align with the setup of our model.

Quantitative comparisons. The fidelity of synthesized image is measured by
Frechet Inception Distance (FID) [24] computed between $50K$ generated images
and all the available real images in each dataset. To study the appearance
quality for face and hands, we further crop face (resolution $64^{2}$) and
hands (resolution $48^{2}$) regions from the generated and real images to
compute $\text{FID}_{\text{f}}$ and $\text{FID}_{\text{h}}$. To evaluate pose
control ability, we compute Percentage of Correct Keypoints (PCK) between $5K$
real images and images generated using the same pose condition parameters of
real images under a distance threshold of 0.1. To evaluate this ability in
face and hand regions, we also report $\text{PCK}_{\text{f}}$ and
$\text{PCK}_{\text{h}}$. Another critical evaluation for a fully controllable
generative model is the disentangled control of fine-grained attributes.
Inspired by previous works [13, 64], we select one attribute from
$\\{\text{expression},\text{shape},\text{jaw pose},\text{body pose},\text{hand
pose}\\}$, and modify the selected attribute while keeping others fixed for
each synthesis. We then estimate the SMPL-X parameters for $1K$ generated
images using a pre-trained 3D human reconstruction model [17] and compute the
Mean Square Error (MSE) for the selected attribute between the input and
estimated parameters.

Table 1: Quantitative comparisons with baselines in terms of appearance and overall control ability, with best results in bold. F.Ctl. indicates whether the approach generates fully controllable human body or not. ∗We implement AvatarGen by conditioning it on SMPL-X. |  |  DeepFashion [36] |  MPV [14]
---|---|---|---
| F.Ctl. |  FID$\downarrow$ |  FID${}_{\text{f}}$$\downarrow$ |  FID${}_{\text{h}}$$\downarrow$ |  PCK$\uparrow$ |  PCK${}_{\text{f}}$$\uparrow$ |  PCK${}_{\text{h}}$$\uparrow$ |  FID$\downarrow$ |  FID${}_{\text{f}}$$\downarrow$ |  FID${}_{\text{h}}$$\downarrow$ |  PCK$\uparrow$ |  PCK${}_{\text{f}}$$\uparrow$ |  PCK${}_{\text{h}}$$\uparrow$
ENARF [43] | ✗ | 68.62 | 52.17 | 46.86 | 3.54 | 3.79 | 1.34 | 65.97 | 47.71 | 37.08 | 3.06 | 3.55 | 0.67
EVA3D [26] | ✗ | 15.91 | 14.63 | 48.10 | 56.36 | 75.43 | 23.14 | 14.98 | 27.48 | 32.54 | 33.00 | 42.47 | 19.24
AG3D [16] | ✗ | 10.93 | 14.79 | - | - | - | - | - | - | - | - | - | -
AvatarGen [69]∗ | ✓ | 9.53 | 13.96 | 27.68 | 60.12 | 73.38 | 46.50 | 10.06 | 13.08 | 19.75 | 38.32 | 45.26 | 30.75
XAGen (Ours) | ✓ | 8.55 | 10.69 | 24.26 | 66.04 | 87.06 | 47.56 | 7.94 | 12.07 | 17.35 | 48.84 | 63.77 | 32.01
|  |  UBC [68] |  SHHQ [18]
| F.Ctl. |  FID$\downarrow$ |  FID${}_{\text{f}}$$\downarrow$ |  FID${}_{\text{h}}$$\downarrow$ |  PCK$\uparrow$ |  PCK${}_{\text{f}}$$\uparrow$ |  PCK${}_{\text{h}}$$\uparrow$ |  FID$\downarrow$ |  FID${}_{\text{f}}$$\downarrow$ |  FID${}_{\text{h}}$$\downarrow$ |  PCK$\uparrow$ |  PCK${}_{\text{f}}$$\uparrow$ |  PCK${}_{\text{h}}$$\uparrow$
ENARF [43] | ✗ | 36.39 | 34.27 | 32.72 | 6.90 | 7.44 | 6.37 | 79.29 | 50.19 | 46.97 | 4.43 | 4.62 | 2.71
EVA3D [26] | ✗ | 12.61 | 36.87 | 45.66 | 36.31 | 55.31 | 8.38 | 11.99 | 20.04 | 39.83 | 31.24 | 37.60 | 18.38
AG3D [16] | ✗ | 11.04 | 15.83 | - | - | - | - | - | - | - | - | - | -
AvatarGen [69]∗ | ✓ | 9.75 | 13.23 | 18.09 | 65.31 | 77.09 | 55.09 | 10.52 | 12.57 | 28.21 | 59.18 | 78.71 | 36.29
XAGen (Ours) | ✓ | 8.80 | 9.82 | 16.72 | 69.18 | 84.18 | 55.17 | 5.88 | 10.06 | 19.23 | 65.14 | 91.44 | 38.53

Table 1 summarizes the results for appearance quality and pose control ability
for body, face, and hands. It demonstrates that XAGen outperforms existing
methods w.r.t. all the evaluation metrics, indicating its superior performance
in generating controllable photo-realistic human images with high-quality face
and hands. Notably, XAGen shows significant improvements over the most recent
method AG3D, achieving more than 20% improvement in FID and
$\text{FID}_{\text{f}}$ on both DeepFashion and UBC datasets. Additionally,
XAGen achieves state-of-the-art pose control ability, with substantial
performance boost in $\text{PCK}_{\text{f}}$, _e.g._ , a relative improvement
of 40.90% on MPV dataset against baseline.

Table 2 presents the results for the disentangled control ability of XAGen
compared to the baseline methods. It is worth noting that ENARF and EVA3D are
not fully controllable, but we still report all the evaluation metrics for
these two methods to show the controllability lower bound. Notably, the
generated images of ENARF are blurry. Thus, our pose estimator cannot estimate
precise jaw poses, which leads to an outlier on UBC jaw pose. In general,
XAGen demonstrates state-of-the-art performance for fine-grained controls,
particularly in expression, jaw, and hand pose, improving upon baseline by
38.29%, 25.93%, and 33.87% respectively on SHHQ dataset which contains diverse
facial expressions and hand gestures. These results highlight the
effectiveness of XAGen in enabling disentangled control over specific
attributes of the generated human avatar images.

Table 2: Quantitative comparisons with baselines in terms of disentangled control ability measured by MSE. We report Jaw$\times 10^{\mathrm{-4}}$ and others $\times 10^{\mathrm{-2}}$ for simplicity, with best results in bold. ∗We implement AvatarGen by conditioning it on SMPL-X. |  DeepFashion [36] |  MPV [14]
---|---|---
|  Exp$\downarrow$ |  Shape$\downarrow$ |  Jaw$\downarrow$ |  Body$\downarrow$ |  Hand$\downarrow$ |  Exp$\downarrow$ |  Shape$\downarrow$ |  Jaw$\downarrow$ |  Body$\downarrow$ |  Hand$\downarrow$
ENARF [43] | 13.47 | 6.30 | 5.79 | 3.14 | 9.87 | 11.21 | 4.91 | 8.36 | 2.75 | 12.90
EVA3D [26] | 6.03 | 2.87 | 5.11 | 1.78 | 3.68 | 9.97 | 4.14 | 13.83 | 1.80 | 4.65
AvatarGen [69]∗ | 4.92 | 3.06 | 5.05 | 1.23 | 3.17 | 8.98 | 3.88 | 15.22 | 1.11 | 3.47
XAGen (Ours) | 4.46 | 2.77 | 3.67 | 1.26 | 2.95 | 6.31 | 3.88 | 7.43 | 0.94 | 2.23
|  UBC [68] |  SHHQ [18]
|  Exp$\downarrow$ |  Shape$\downarrow$ |  Jaw$\downarrow$ |  Body$\downarrow$ |  Hand$\downarrow$ |  Exp$\downarrow$ |  Shape$\downarrow$ |  Jaw$\downarrow$ |  Body$\downarrow$ |  Hand$\downarrow$
ENARF [43] | 10.70 | 6.11 | 3.62 | 1.07 | 8.19 | 14.51 | 6.43 | 8.16 | 3.27 | 9.83
EVA3D [26] | 7.00 | 2.98 | 5.36 | 1.00 | 2.78 | 7.43 | 4.15 | 9.26 | 1.93 | 5.15
AvatarGen [69]∗ | 9.59 | 4.50 | 9.34 | 1.22 | 3.01 | 9.01 | 3.99 | 8.87 | 1.52 | 4.99
XAGen (Ours) | 5.35 | 2.57 | 4.76 | 0.73 | 1.63 | 5.56 | 3.66 | 6.57 | 1.24 | 3.30

Qualitative comparisons.

Figure 3: Comparisons against baselines in terms of appearance and 3D
geometry. Our method produces photo-realistic human images with superior
detailed geometries.

Figure 3 provides qualitative comparisons between XAGen and baselines. From
the results, we observe that ENARF struggles to produce reasonable geometry or
realistic images due to the limitations of low training resolution. While
EVA3D and AvatarGen achieve higher quality, they still fail to synthesize
high-fidelity appearance and geometry for the face and hands. In contrast,
XAGen demonstrates superior performance with detailed geometries for face and
hands regions, resulting in more visually appealing human avatar images. The
improvement of XAGen against baseline models is also confirmed by the
perceptual user study, which is summarized in Table 3. Notably, XAGen achieves
the best perceptual preference scores for both image appearance ($\geq
57.2\%$) and geometry ($\geq 48.3\%$) on all the benchmark datasets.

Figure 4 showcases qualitative results for fine-grained control ability. We
first observe that ENARF fails to generate a correct arm for the given body
pose. Although EVA3D demonstrates a better pose condition ability, its shape
conditioning ability is limited and the generated face suffers from
unrealistic scaling. On the other hand, AvatarGen shows comparable results for
pose and shape control. However, when it comes to expression, jaw pose, and
hand pose controls, ours significantly outperforms AvatarGen, _e.g._ ,
AvatarGen produces distortion in mouth region and blurred fingers while XAGen
demonstrates natural faces and correct hand poses.

Table 3: We conduct a perceptual human study and report participants’ preferences on images and geometries generated by our method and baselines. It is measured by preference rate (%), with best results in bold. RGB represents image, and Geo represents geometry. | DeepFashion | MPV | UBC | SHHQ
---|---|---|---|---
| RGB$\uparrow$ | Geo$\uparrow$ | RGB$\uparrow$ | Geo$\uparrow$ | RGB$\uparrow$ | Geo | RGB$\uparrow$ | Geo
ENARF | 0.0 | 0.0 | 0.0 | 0.0 | 0.6 | 0.0 | 0.0 | 0.0
EVA3D | 17.3 | 35.6 | 15.0 | 17.2 | 7.8 | 34.4 | 11.3 | 15.5
AvatarGen | 15.4 | 16.1 | 17.2 | 18.9 | 34.4 | 3.9 | 28.2 | 28.6
XAGen (Ours) | 67.3 | 48.3 | 67.8 | 63.9 | 57.2 | 61.7 | 60.5 | 55.9
Figure 4: Qualitative comparisons in terms of disentangled control ability.
Our method exhibits state-of-the-art control abilities for body pose, shape,
expression, jaw pose, and hand pose. Table 4: Ablations of our method on SHHQ
dataset. We vary our representation, rendering method, and discriminators to
investigate their effectiveness.

Repr. |  FID$\downarrow$ |  FID${}_{\text{f}}$$\downarrow$ |  FID${}_{\text{h}}$$\downarrow$
---|---|---|---
w/o both | 11.50 | 12.57 | 20.97
w/ face | 11.27 | 11.95 | 20.10
w/ hand | 9.64 | 11.61 | 19.92
w/ both | 5.88 | 10.06 | 19.23
(a) The effect of multi-scale and multi-part representations.

Face |  FID$\downarrow$ |  FID${}_{\text{f}}$$\downarrow$ |  Exp$\downarrow$ |  Jaw$\downarrow$
---|---|---|---|---
w/o Rend | 14.53 | 20.63 | 6.58 | 7.26
w/o Disc | 7.40 | 9.20 | 6.27 | 6.58
w/ both | 5.88 | 10.06 | 5.56 | 6.57
(b) The effect of face rendering and face discriminator.

Hand |  FID$\downarrow$ |  FID${}_{\text{h}}$$\downarrow$ |  Hand$\downarrow$
---|---|---|---
w/o Rend | 14.28 | 26.66 | 4.51
w/o Disc | 7.78 | 16.74 | 4.46
w/ both | 5.88 | 19.23 | 3.33
(c) The effect of hand rendering and hand discriminator.

###  4.2 Ablation studies

To verify the design choices in our method, we conduct ablation studies on
SHHQ dataset, which contains diverse appearances, _i.e._ , various human body,
face, and hand poses as well as clothes.

Representation. XAGen adopts a multi-scale and multi-part representation to
improve the quality for face and hands regions. We study the necessity of this
design by removing Tri-planes for face and hands. Table 4(a) provides the
results, indicating that using only a single full-body Tri-plane (without any
specific Tri-planes for face or hands) results in a significant degradation in
appearance quality. Adding either face or hand Tri-plane can alleviate this
issue and all the FID metrics drop slightly. The best results are achieved
when both face and hand Tri-planes are enabled, demonstrating the importance
of our multi-scale and multi-part representation.

Multi-part rendering. In our model, we render multiple parts independently in
the forward process to disentangle the learning of body, face, and hands.
Table 4(b) demonstrates that independent rendering for face is crucial, as it
significantly improves both fidelity (FID${}_{\text{f}}$: 20.63 vs. 10.06) and
control ability (Exp: 6.58 vs. 5.56, Jaw: 7.26 vs. 6.57) for face. Similarly,
without rendering for hand, FID${}_{\text{h}}$ increases from 18.85 to 25.94,
and MSE increases from 3.28 to 4.55 (Table 4(c)). The effectiveness of multi-
part rendering is further supported by the qualitative results shown in Figure
5. Without independent rendering, the geometry quality degrades. For example,
the eyes and mouth are collapsed without face rendering, and the model also
fails to synthesize geometric details for hand when hand rendering is
disabled. These highlight the importance of multi-part rendering in
facilitating the learning of 3D geometries for different body parts.

Discriminators. To study the effect of multi-part discriminators, we disable
each of them during training. As shown in Table 4(b), without face
discriminator, the overall appearance quality deteriorates. Despite the slight
improvement in face appearance, there is a drop in the control ability, as
evidenced by the increase in the MSE values for expression and jaw pose. A
similar observation can be made for hand discriminator in Table 4(c).
Furthermore, the qualitative results shown in Figure 5 provide visual evidence
of the impact of the face and hand discriminators on the 3D geometries. When
they are removed, the geometries for face and hand collapse.

Figure 5: Qualitative results for the ablations on multi-part rendering and
discriminators.

###  4.3 Applications

(a) Text-guided avatar synthesis.

(b) Audio-driven avatar animation.

Figure 6: Downstream applications of our method.

Text-guided avatar synthesis. Inspired by recent works [25, 69, 67] on text-
guided avatar generation, we leverage a pretrained vision-language encoder
CLIP [49] to guide the generation process using the given text prompt. The
text-guided avatar generation process involves randomly sampling a latent code
$\mathbf{z}$ and a control parameter $p_{\text{b}}$ from the dataset, and
optimizing $\mathbf{z}$ by maximizing the CLIP similarities between the
synthesized image and text prompt. As shown in Figure 6(a), the generated
human avatars exhibit the text-specified attributes, _i.e._ , hair and clothes
adhere to the given text prompt (_e.g._ , brown hair and red T-shirt). The
generated avatar can be re-targeted by novel SMPL-X parameters, allowing for
additional control and customization of the synthesis.

Audio-driven animation. The ability of XAGen to generate fully animatable
human avatars with fine-grained control (Figure 1) opens up possibilities for
audio-driven animation. The 3D avatars can be driven by arbitrary SMPL-X
motion sequences generated by recent works such as [66] given audio inputs.
Specifically, we sample an audio stream and SMPL-X sequence from TalkSHOW [66]
and use it to animate the generated avatars. As shown in Figure 6(b), XAGen is
able to synthesize temporally consistent video animations where the jaw poses
of the avatars are synchronized with the audio stream (highlighted in red
box). Additionally, the generated avatars are generalizable given novel body
poses and hand gestures, allowing diverse and expressive animations.

##  5 Limitations

Although XAGen is able to synthesize photo-realistic and fully animatable
human avatars, there are still areas where improvements can be made: (1) XAGen
relies on pre-estimated SMPL-X parameters, the inaccurate SMPL-X may introduce
potential errors into our model, which can lead to artifacts and degraded body
images. Please refer to Sup. Mat. for the experimental analysis of this issue.
We believe our method can benefit from a more accurate SMPL-X estimation
method or corrective operations. (2) SMPL-X only represents naked body. Thus,
methods built upon SMPL-X could struggle with modeling loose clothing, which
is a long-standing challenge for 3D human modeling. We believe an advanced
human body prior or independent clothing modeling approach is helpful to
alleviate this issue. (3) Face and hand images in existing human body datasets
lack diversity and sharpness, which affects the fidelity of our generation
results, particularly for the novel hand poses that are out-of-distribution. A
more diverse dataset with high-quality face and hand images could help tackle
this problem. (4) XAGen utilizes inverse blend skinning to deform the points
from canonical space to the observation space. However, this process could
introduce errors, particularly when computing nearest neighbors for query
points located in the connection or interaction regions. Thus, exploring more
robust and accurate techniques, such as forward skinning [9], could open up
new directions for future work.

##  6 Conclusion

This work introduces XAGen, a novel 3D avatar generation framework that offers
expressive control over facial expression, shape, body pose, jaw pose, and
hand pose. Through the use of multi-scale and multi-part representation, XAGen
can model details for small-scale regions like faces and hands. By adopting
multi-part rendering, XAGen disentangles the learning process and produces
realistic details for appearance and geometry. With multi-part discriminators,
our model is capable of synthesizing high-quality human avatars with
disentangled fine-grained control ability. The capabilities of XAGen open up a
range of possibilities for downstream applications, such as text-guided avatar
synthesis and audio-driven animation.

## Acknowledgement

This project is supported by the National Research Foundation, Singapore under
its NRFF Award NRF-NRFF13-2021-0008, and the Ministry of Education, Singapore,
under the Academic Research Fund Tier 1 (FY2022).

## References

  * Alexander et al. [2010] O. Alexander, M. Rogers, W. Lambeth, J.-Y. Chiang, W.-C. Ma, C.-C. Wang, and P. Debevec.  The digital emily project: Achieving a photorealistic digital actor.  _IEEE Computer Graphics and Applications_ , 2010.
  * Anguelov et al. [2005] D. Anguelov, P. Srinivasan, D. Koller, S. Thrun, J. Rodgers, and J. Davis.  Scape: shape completion and animation of people.  In _ACM SIGGRAPH_ , 2005.
  * Bagautdinov et al. [2021] T. Bagautdinov, C. Wu, T. Simon, F. Prada, T. Shiratori, S.-E. Wei, W. Xu, Y. Sheikh, and J. Saragih.  Driving-signal aware full-body avatars.  _ACM Trans. on Graphics_ , 2021.
  * Beeler et al. [2011] T. Beeler, F. Hahn, D. Bradley, B. Bickel, P. Beardsley, C. Gotsman, R. W. Sumner, and M. Gross.  High-quality passive facial performance capture using anchor frames.  In _ACM SIGGRAPH_ , 2011.
  * Bergman et al. [2022] A. Bergman, P. Kellnhofer, W. Yifan, E. Chan, D. Lindell, and G. Wetzstein.  Generative neural articulated radiance fields.  In _NeurIPS_ , 2022.
  * Bregler et al. [2004] C. Bregler, J. Malik, and K. Pullen.  Twist based acquisition and tracking of animal and human kinematics.  _Int’l. J. Computer Vision_ , 2004.
  * Chan et al. [2022] E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, L. J. Guibas, J. Tremblay, S. Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In _CVPR_ , 2022.
  * Chen et al. [2022] M. Chen, J. Zhang, X. Xu, L. Liu, Y. Cai, J. Feng, and S. Yan.  Geometry-guided progressive nerf for generalizable and efficient neural human rendering.  In _ECCV_ , 2022.
  * Chen et al. [2021] X. Chen, Y. Zheng, M. J. Black, O. Hilliges, and A. Geiger.  Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes.  In _CVPR_ , 2021.
  * Chu et al. [2020] H. Chu, S. Ma, F. De la Torre, S. Fidler, and Y. Sheikh.  Expressive telepresence via modular codec avatars.  In _ECCV_ , 2020.
  * Collet et al. [2015] A. Collet, M. Chuang, P. Sweeney, D. Gillett, D. Evseev, D. Calabrese, H. Hoppe, A. Kirk, and S. Sullivan.  High-quality streamable free-viewpoint video.  _ACM Trans. on Graphics_ , 2015.
  * De Aguiar et al. [2008] E. De Aguiar, C. Stoll, C. Theobalt, N. Ahmed, H.-P. Seidel, and S. Thrun.  Performance capture from sparse multi-view video.  In _ACM SIGGRAPH_ , 2008.
  * Deng et al. [2020] Y. Deng, J. Yang, D. Chen, F. Wen, and X. Tong.  Disentangled and controllable face image generation via 3d imitative-contrastive learning.  In _CVPR_ , 2020.
  * Dong et al. [2019] H. Dong, X. Liang, X. Shen, B. Wang, H. Lai, J. Zhu, Z. Hu, and J. Yin.  Towards multi-pose guided virtual try-on network.  In _CVPR_ , 2019.
  * Dong et al. [2022] J. Dong, Q. Fang, Y. Guo, S. Peng, Q. Shuai, X. Zhou, and H. Bao.  Totalselfscan: Learning full-body avatars from self-portrait videos of faces, hands, and bodies.  In _NeurIPS_ , 2022.
  * Dong et al. [2023] Z. Dong, X. Chen, J. Yang, M. J. Black, O. Hilliges, and A. Geiger.  Ag3d: Learning to generate 3d avatars from 2d image collections.  In _ICCV_ , 2023.
  * Feng et al. [2021] Y. Feng, V. Choutas, T. Bolkart, D. Tzionas, and M. J. Black.  Collaborative regression of expressive bodies using moderation.  In _3DV_ , 2021.
  * Fu et al. [2022] J. Fu, S. Li, Y. Jiang, K.-Y. Lin, C. Qian, C. C. Loy, W. Wu, and Z. Liu.  Stylegan-human: A data-centric odyssey of human generation.  In _ECCV_ , 2022.
  * Gall et al. [2009] J. Gall, C. Stoll, E. De Aguiar, C. Theobalt, B. Rosenhahn, and H.-P. Seidel.  Motion capture using joint skeleton tracking and surface estimation.  In _CVPR_ , 2009.
  * Ghosh et al. [2011] A. Ghosh, G. Fyffe, B. Tunwattanapong, J. Busch, X. Yu, and P. Debevec.  Multiview face capture using polarized spherical gradient illumination.  _ACM Trans. on Graphics_ , 2011.
  * Goodfellow et al. [2020] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial networks.  _Communications of the ACM_ , 2020.
  * Grauman et al. [2022] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al.  Ego4d: Around the world in 3,000 hours of egocentric video.  In _CVPR_ , 2022.
  * Guo et al. [2019] K. Guo, P. Lincoln, P. Davidson, J. Busch, X. Yu, M. Whalen, G. Harvey, S. Orts-Escolano, R. Pandey, J. Dourgarian, et al.  The relightables: Volumetric performance capture of humans with realistic relighting.  _ACM Trans. on Graphics_ , 2019.
  * Heusel et al. [2017] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _NeurIPS_ , 2017.
  * Hong et al. [2022] F. Hong, M. Zhang, L. Pan, Z. Cai, L. Yang, and Z. Liu.  Avatarclip: Zero-shot text-driven generation and animation of 3d avatars.  _ACM Trans. on Graphics_ , 2022.
  * Hong et al. [2023] F. Hong, Z. Chen, Y. LAN, L. Pan, and Z. Liu.  EVA3d: Compositional 3d human generation from 2d image collections.  In _ICLR_ , 2023.
  * Karras et al. [2019] T. Karras, S. Laine, and T. Aila.  A style-based generator architecture for generative adversarial networks.  In _CVPR_ , 2019.
  * Karras et al. [2021] T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila.  Alias-free generative adversarial networks.  In _NeurIPS_ , 2021.
  * Kato et al. [2018] H. Kato, Y. Ushiku, and T. Harada.  Neural 3d mesh renderer.  In _CVPR_ , 2018.
  * Kolotouros et al. [2019] N. Kolotouros, G. Pavlakos, and K. Daniilidis.  Convolutional mesh regression for single-image human shape reconstruction.  In _CVPR_ , 2019.
  * Lewis et al. [2000] J. P. Lewis, M. Cordner, and N. Fong.  Pose space deformation: a unified approach to shape interpolation and skeleton-driven deformation.  In _CGIT_ , 2000.
  * Li et al. [2017] T. Li, T. Bolkart, M. J. Black, H. Li, and J. Romero.  Learning a model of facial shape and expression from 4D scans.  _ACM Trans. on Graphics_ , 2017.
  * Liu et al. [2023] J.-W. Liu, Y.-P. Cao, T. Yang, Z. Xu, J. Keppo, Y. Shan, X. Qie, and M. Z. Shou.  Hosnerf: Dynamic human-object-scene neural radiance fields from a single video.  In _ICCV_ , 2023.
  * Liu et al. [2021a] L. Liu, M. Habermann, V. Rudnev, K. Sarkar, J. Gu, and C. Theobalt.  Neural actor: Neural free-view synthesis of human actors with pose control.  _ACM Trans. on Graphics_ , 2021a.
  * Liu et al. [2021b] T. Liu, J. Zhang, X. Nie, Y. Wei, S. Wei, Y. Zhao, and J. Feng.  Spatial-aware texture transformer for high-fidelity garment transfer.  _IEEE Transactions on Image Processing_ , 2021b.
  * Liu et al. [2016] Z. Liu, P. Luo, S. Qiu, X. Wang, and X. Tang.  Deepfashion: Powering robust clothes recognition and retrieval with rich annotations.  In _CVPR_ , 2016.
  * Lombardi et al. [2021] S. Lombardi, T. Simon, G. Schwartz, M. Zollhoefer, Y. Sheikh, and J. Saragih.  Mixture of volumetric primitives for efficient neural rendering.  _ACM Trans. on Graphics_ , 2021.
  * Loper et al. [2015] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black.  Smpl: A skinned multi-person linear model.  _ACM Trans. on Graphics_ , 2015.
  * Max [1995] N. Max.  Optical models for direct volume rendering.  _IEEE Trans. on Visualization and Computer Graphics_ , 1995.
  * Mescheder et al. [2018] L. Mescheder, A. Geiger, and S. Nowozin.  Which training methods for gans do actually converge?  In _ICLR_ , 2018.
  * Mescheder et al. [2019] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In _CVPR_ , 2019.
  * Mildenhall et al. [2020] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _ECCV_ , 2020.
  * Noguchi et al. [2022] A. Noguchi, X. Sun, S. Lin, and T. Harada.  Unsupervised learning of efficient geometry-aware neural articulated representations.  In _ECCV_ , 2022.
  * Or-El et al. [2022] R. Or-El, X. Luo, M. Shan, E. Shechtman, J. J. Park, and I. Kemelmacher-Shlizerman.  Stylesdf: High-resolution 3d-consistent image and geometry generation.  In _CVPR_ , 2022.
  * Osman et al. [2020] A. A. Osman, T. Bolkart, and M. J. Black.  Star: Sparse trained articulated human body regressor.  In _ECCV_ , 2020.
  * Osman et al. [2022] A. A. Osman, T. Bolkart, D. Tzionas, and M. J. Black.  Supr: A sparse unified part-based human representation.  In _3DV_ , 2022.
  * Pavlakos et al. [2019] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black.  Expressive body capture: 3d hands, face, and body from a single image.  In _CVPR_ , 2019.
  * Peng et al. [2021] S. Peng, Y. Zhang, Y. Xu, Q. Wang, Q. Shuai, H. Bao, and X. Zhou.  Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans.  In _CVPR_ , 2021.
  * Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Remelli et al. [2022] E. Remelli, T. Bagautdinov, S. Saito, C. Wu, T. Simon, S.-E. Wei, K. Guo, Z. Cao, F. Prada, J. Saragih, et al.  Drivable volumetric avatars using texel-aligned features.  In _ACM SIGGRAPH_ , 2022.
  * Rombach et al. [2022] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , 2022.
  * Romero et al. [2017] J. Romero, D. Tzionas, and M. J. Black.  Embodied hands: modeling and capturing hands and bodies together.  _ACM Trans. on Graphics_ , 2017.
  * Saito et al. [2019] S. Saito, Z. Huang, R. Natsume, S. Morishima, A. Kanazawa, and H. Li.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In _ICCV_ , 2019.
  * Shen et al. [2023] K. Shen, C. Guo, M. Kaufmann, J. J. Zarate, J. Valentin, J. Song, and O. Hilliges.  X-avatar: Expressive human avatars.  In _CVPR_ , 2023.
  * Shi et al. [2021] Y. Shi, D. Aggarwal, and A. K. Jain.  Lifting 2d stylegan for 3d-aware face generation.  In _CVPR_ , 2021.
  * Siarohin et al. [2021] A. Siarohin, O. J. Woodford, J. Ren, M. Chai, and S. Tulyakov.  Motion representations for articulated animation.  In _CVPR_ , 2021.
  * Sun et al. [2023] J. Sun, X. Wang, L. Wang, X. Li, Y. Zhang, H. Zhang, and Y. Liu.  Next3d: Generative neural texture rasterization for 3d-aware head avatars.  In _CVPR_ , 2023.
  * Tang et al. [2019] S. Tang, F. Tan, K. Cheng, Z. Li, S. Zhu, and P. Tan.  A neural network for detailed human depth estimation from a single image.  In _ICCV_ , 2019.
  * Wang et al. [2023] T. Wang, B. Zhang, T. Zhang, S. Gu, J. Bao, T. Baltrusaitis, J. Shen, D. Chen, F. Wen, Q. Chen, et al.  Rodin: A generative model for sculpting 3d digital avatars using diffusion.  In _CVPR_ , 2023.
  * Xiang et al. [2021] D. Xiang, F. Prada, T. Bagautdinov, W. Xu, Y. Dong, H. Wen, J. Hodgins, and C. Wu.  Modeling clothing as a separate layer for an animatable human avatar.  _ACM Trans. on Graphics_ , 2021.
  * Xiu et al. [2022] Y. Xiu, J. Yang, D. Tzionas, and M. J. Black.  Icon: implicit clothed humans obtained from normals.  In _CVPR_ , 2022.
  * Xu et al. [2020] H. Xu, E. G. Bazavan, A. Zanfir, W. T. Freeman, R. Sukthankar, and C. Sminchisescu.  Ghum & ghuml: Generative 3d human shape and articulated pose models.  In _CVPR_ , 2020.
  * Xu et al. [2023a] H. Xu, G. Song, Z. Jiang, J. Zhang, Y. Shi, J. Liu, W. Ma, J. Feng, and L. Luo.  Omniavatar: Geometry-guided controllable 3d head synthesis.  In _CVPR_ , 2023a.
  * Xu et al. [2023b] H. Xu, G. Song, Z. Jiang, J. Zhang, Y. Shi, J. Liu, W. Ma, J. Feng, and L. Luo.  Omniavatar: Geometry-guided controllable 3d head synthesis.  In _CVPR_ , 2023b.
  * Xu et al. [2023c] Z. Xu, J. Zhang, J. Liew, W. Zhang, S. Bai, J. Feng, and M. Z. Shou.  Pv3d: A 3d generative model for portrait video generation.  In _ICLR_ , 2023c.
  * Yi et al. [2023] H. Yi, H. Liang, Y. Liu, Q. Cao, Y. Wen, T. Bolkart, D. Tao, and M. J. Black.  Generating holistic 3d human motion from speech.  In _CVPR_ , 2023.
  * Youwang et al. [2022] K. Youwang, K. Ji-Yeon, and T.-H. Oh.  Clip-actor: Text-driven recommendation and stylization for animating human meshes.  In _ECCV_ , 2022.
  * Zablotskaia et al. [2019] P. Zablotskaia, A. Siarohin, B. Zhao, and L. Sigal.  Dwnet: Dense warp-based network for pose-guided human video generation.  In _BMVC_ , 2019.
  * Zhang et al. [2023a] J. Zhang, Z. Jiang, D. Yang, H. Xu, Y. Shi, G. Song, Z. Xu, X. Wang, and J. Feng.  Avatargen: a 3d generative model for animatable human avatars.  In _ECCV Workshop_ , 2023a.
  * Zhang et al. [2023b] J. Zhang, H. Yan, Z. Xu, J. Feng, and J. H. Liew.  Magicavatar: Multimodal avatar generation and animation.  _arXiv_ , 2023b.
  * Zhang et al. [2023c] X. Zhang, J. Zhang, C. Rohan, H. Xu, G. Song, Y. Yang, and J. Feng.  Getavatar: Generative textured meshes for animatable human avatars.  In _ICCV_ , 2023c.


# Wav2Vec2.0 on the Edge: Performance Evaluation

Santosh Gondi
Meta Inc

###### Abstract

Wav2Vec2.0 [1] is a state-of-the-art model which learns speech representations
through unlabeled speech data, aka, self supervised learning. The pretrained
model is then fine tuned on small amounts of labeled data to use it for
speech-to-text and machine translation tasks. Wav2Vec 2.0 is a transformative
solution for low resource languages as it is mainly developed using unlabeled
audio data. Getting large amounts of labeled data is resource intensive and
especially challenging to do for low resource languages such as Swahilli,
Tatar, etc. Furthermore, Wav2Vec2.0 word-error-rate(WER) matches or surpasses
the very recent supervised learning algorithms while using 100x [1] less
labeled data. Given its importance and enormous potential in enabling speech
based tasks on world’s 7000 languages [12], it is key to evaluate the
accuracy, latency and efficiency of this model on low resource and low power
edge devices and investigate the feasibility of using it in such devices for
private, secure and reliable speech based tasks. On-device speech tasks
preclude sending audio data to the server hence inherently providing privacy,
reduced latency and enhanced reliability. In this paper, Wav2Vec2.0 model’s
accuracy and latency has been evaluated on Raspberry Pi along with the KenLM
[6] language model for speech recognition tasks. How to tune certain
parameters to achieve desired level of WER rate and latency while meeting the
CPU, memory and energy budgets of the product has been discussed.

##  1 Introduction

Deep learning based speech recognition has come a long way in the past decade
[7]. DL is a subfield of machine learning which imitates the neural network
structure of the human brain for pattern matching and classification. The
major reason for its popularity is that it does not need feature engineering.
It independently extracts the features based on the patterns in the labeled
training dataset. However, it needs large quantities of labeled data to
achieve higher prediction accuracy. On the contrary, Self-supervised learning
(SSL) [25] doesn’t need labeled data. It learns to understand the structures
and patterns of input just based on raw input data. The general technique of
self-supervised learning is to predict any unobserved or hidden part (or
property) of the input from any observed or unhidden part of the input. This
learning is very helpful in cases where labeled data is very difficult to get
or doesn’t exist such as for translation or recognition of low resource
languages. Wav2Vec2.0, just referred to as Wav2Vec in the rest of the paper,
is trained with a self-supervised algorithm using unlabeled LibriSpeech
dataset. It learns meaningful representations from raw speech data. Then the
model is fine tuned with a small amount of labeled data to use for downstream
tasks such as translation and speech recognition.

Automatic speech recognition has many practical applications on all the
digital devices people use everyday. Voice search for music or document, voice
to text for messaging and translation, voice assistant for navigating apps or
controlling home devices, to name a few. On-device speech recognition has
additional benefits of being more secure and reliable by not having to send
audio data to servers over a network connection. On-device based speech
recognition can enable military, healthcare, voice based accessibility apps
where the network connection may not exist. Given the transformative
capability, the Wav2Vec is bringing to world languages, the latency and
feasibility of it for on-device execution is explored using Raspberry Pi as a
proxy for low end edge devices.

The Wav2Vec model outputs the sequence of token probabilities in the alphabet
set. A simple arg-max followed by tokenizer yields sufficiently good accuracy.
However, by using a language on top of the Wav2Vec output significantly
improves the WER, though the decoding using language model increases the
memory footprint of the transcription task. In this paper, WER, latency and
memory are analyzed using a simple tokenizer as well as the Trie based KenLM
language model.

In this paper, the performance of base Wav2Vec models is evaluated on
Raspberry Pi and different metrics like accuracy, latency and efficiency
trade-offs to consider during deployment are discussed. This study is an
extension of the previous work [2, 3]. Here the focus is on Wav2Vec decoding
with language model and energy measurements along with accuracy and
performance. The previous ASR evaluations on edge devices have focused on RNN
[4]/LSTM [8, 22] based models[15]. The transformer [23] models are based on
different architecture and boost the accuracy quite significantly. There is no
other work which evaluated the latency and feasibility of using latest ASR
systems such as Wav2Vec, on the edge devices.

The rest of the paper is organized as follows: In the background section
Wav2Vec and language models are discussed. Experimental setup section covers
the steps for preparing the model and setting up for inferencing and energy
measurements. Accuracy, latency and efficiency metrics are covered in the
results section. Finally, paper ends with a summary and outlook.

##  2 Background

Wav2Vec takes raw 16K sampled audio as input. It encodes speech audio through
multi-layer CNN and then resulting speech representations are masked before
being fed to a Transformer network. The model is trained via a contrastive
task where the true latent representations are to be distinguished from
distractors. Transformer is a sequence-to-sequence architecture originally
proposed for machine translation [23]. It has since been adopted for ASR, with
the difference being that the input is audio frames instead of the text like
in translation tasks. The Wav2Vec feature encoder contains seven blocks and
the temporal convolutions in each block have 512 channels with strides
(5,2,2,2,2,2,2) and kernel widths (10,3,3,3,3,2,2) [1]. Base model contains 12
transformer blocks, model dimension 768, inner dimension (FFN) 3,072 and 8
attention heads.

Wav2Vec inference with simple arg-max tokenizer(greedy) as well as using beam
search decoding with KenLM language models is explored. Beam search explores
the search space in a greedy breadth-first fashion retaining only certain top
candidates in each step [24]. The number of candidates kept during search is
called beam width. Beam search with a language model is used to boost the
accuracy of speech-to-text output. The emitted logits with probabilities over
alphabet space are overlapped with language model probabilities for potential
sentences, keeping only few top candidates at each step. Using higher beam
sizes might yield better accuracy but is not practical to use very high beam
size on memory constrained devices. Experiments in this paper have explored
the beam size of 10 and 100 and analysis of the tradeoff between accuracy and
memory is provided.

KenLM [6] is a widely used library that implements data structures for
efficient language model queries. Trie based KenLM for decoding is used has
been used for experiments in this paper . It uses Trie data structure with
bit-level packing, sorted records, interpolation search for efficient memory
usage. A 4-gram KenLM language model for the LibriSpeech dataset is used in
the experiments.

Figure 2 shows the simplified flow of the ASR process with Wav2Vec model.

Figure 1: Wav2Vec inference.

##  3 Experimental Setup

###  3.1 Model preparation

PyTorch[17] based models are used for the evaluation. PyTorch is an open
source machine learning framework based on torch library. Figure 3 shows the
steps for preparing the models for inferencing on edge devices.

Figure 2: Model preparation steps [3]

This section goes over a few of the PyTorch tools and APIs used in this
evaluation.

####  3.1.1 TorchScript

TorchScript is the means by which PyTorch models are productionized. The
python based modules are optimized, serialized and saved in intermediate
representation(IR) format when converted to a torch script module for mobile
inference [18]. TorchScript modules are created by tracing or scripting
methods. Tracing works by executing the model with sample inputs and capturing
the computations. Whereas, scripting does static inspection by going through
the model recursively. The benefit of scripting is that it correctly handles
the loops and control statements in the module. A saved script module can then
be loaded either in a Python or C++ environment for inferencing purposes.
ScriptModules for Wav2Vec is generated by scripting method after applying
quantization and optimizations. The inferencing is done using Python.

####  3.1.2 PyTorch mobile

PyTorch mobile [19] provides a set of APIs for optimizing the models for
mobile platforms. It uses module fusing, operator fusing, quantization among
other things to optimize the models. By quantization, model weights and
computation are reduced to int8 precision from float precision. This reduces
memory, storage and computation cycles of the model. The device backend was
set to qnnpack during inference in the script as :
_torch.backends.quantized.engine=’qnnpack’_. QNNPACK[21] (Quantized Neural
Networks PACKage) is a mobile-optimized library for low-precision high-
performance tensor computations.

####  3.1.3 Wav2Vec model

A Google Colab instance was used for importing, optimizing and saving the
model as TorchScript module. Pretrained Wav2Vec models from _huggingface_
model hub[9] were used. For this evaluation, a base model with 100hr and full
960hr hour fine-tuning was used. 100hr fine tuned means the base Wav2Vec model
was trained on 960hr of LibriSpeech unlabeled audio data with self-supervised
learning. This model would have learned the basic speech structures. Then a
100hr labeled audio data is used to fine tune the model with CTC[5] loss
function for speech recognition task in english. The base model has 93 million
parameters. The unquantized original model is scripted format is 377 MB. where
as the quantized scripted model is 207 MB. The model dimentionality doesn’t
vary between 100hr and 960hr fine tuned models. The Wav2Vec model emits the
logits(probability of tokens). The decoding process which coverts these token
probabilities to text transcriptions could be simple tokenizer or a BeamSearch
decoder with a language model. In this experiment the accuracy and latency
with both tokenizer and language model based decoding was evaluated.

####  3.1.4 KenLM

BeamSearchDecoderCTC module from pyctcdecode[16] for beam search decoding was
used. A readily available 4-gram librespeech language model from [10] was
used. KenLM repository was built from source to generate binary executables,
such as _build_binary_ , required for converting 4-gram ARPA language models
to Trie based binary files. The same utility was also used for quantizing the
language model. ARPA language model is text based, large in size and takes
longer to load. It is good for interoperability between different packages.
However, in production usecases it is converted to binary format to reduce
loading latency and memory footprint.

###  3.2 Datasets

LibriSpeech [14], a standard ASR bench-marking dataset with $\sim$1,000 hours
of labeled english speech from various audiobooks was used for evaluation. It
has a combination of train, dev and test datasets. For evaluation, test and
dev data sets are used.

###  3.3 Raspberry Pi setup

Raspberry Pi 4 B is used for evaluation. It is a basic headless module( Figure
4) with no additional peripherals attached to it. Relevant device specs are
provided in Table 1. The default Raspberry Pi OS is 32 bit which is not
compatible with PyTorch. Therefore, a 64 bit OS version was installed on Pi.

Table 1: Raspberry Pi 4 B spec. Name | Spec
---|---
Chip | BCM2711
CPU | Quad core Cortex-A72 (ARM v8) 64-bit SoC
Clock speed | 1.5GHz
RAM | 4GB SDRAM
Caches | 32 KB data + 48 KB instruction L1 cache per core. 1MB L2 cache
Storage | 32GB micro SD card
OS | 64bit Raspberry Pi OS
pip packages | torch, transformers, pyctcdecode, numpy, jiwer, soundfile
Python version | 3.7
Power supply | 5V DC via USB-C connector

####  3.3.1 Core and Thread configurations

The ASR inference was studied by configuring the Python script to run on 1, 2,
3 and 4 CPU cores to analyze tradeoffs in terms of accuracy, latency and
efficiency. Linux _taskset_ [11] command is used to associate evaluation
script to certain CPU cores. Using different intra-op and inter-op thread
settings in PyTorch[20] was also explored to examine the similar tradeoffs.
Experiments showed that the inter-op thread settings do not have an effect on
latency and intra-op thread settings have similar effects as core allocations
reserved using linux based taskset command.

###  3.4 Energy measurements

A USB power meter[13] for measuring the power consumption on Raspberry Pi was
used. Figure 4 shows the connection setup and Figure 4 shows the flow for
monitoring energy during ASR inference. The monitoring device provides voltage
and current values every second. Watts and mWh(milli-watt-hours) during
different states of the experiment were computed from these continuous voltage
and current values. An app on the android phone connects to USB meter over
bluetooth and collects voltage and current values during experiment and the
readings are saved in a file. The execution script reads the data from file
and computes the energy/power usage during specific configuration of
experiment.

Figure 3: Connection setup

Figure 4: Measurement flow

##  4 Results

In this section accuracy(WER), latency(RTF) and efficiency(CPU, memory and
power consumption) metrics are presented.

LibriSpeech test and dev datasets together contain $\sim 21$ hours of labeled
audio data. It would have taken many days for the inference to execute on all
combination of models with all the audio in these data sets. To save time, 10%
of the audio files in each of the four data sets were sampled for inference
and evaluation. The same sampled datasets were used in all the test runs thus
ensuring that the results are comparable.

###  4.1 WER

Table 2 shows the WER of quantized and unquantized models for all the datasets
with different LM configurations. WER reported in [1] is also shown as Server
with LM for comparison.

Table 2: WER(%) Dataset | Model | No LM | full LM | quantized LM | Server with LM
---|---|---|---|---|---
|  |  | b_w 10 | b_w 100 | b_w 10 | b_w 100 |
test-clean | 100hr quantized | 6.8 | 4.8 | 4.6 | 5.1 | 4.8 |
100hr | 6.4 | 4.9 | 4.6 | 4.8 | 4.6 | 3.4
960hr quantized | 4.0 | 3.3 | 3.3 | 3.3 | 3.4 |
960hr | 3.7 | 3.1 | 3.1 | 3.1 | 3.1 | 2.6
test-other | 100hr quantized | 15.5 | 11.2 | 10.4 | 11.5 | 10.4 |
100hr | 14.9 | 11.2 | 11.0 | 11.2 | 11.0 | 8.0
960hr quantized | 10.0 | 7.5 | 7.2 | 7.5 | 7.2 |
960hr | 9.0 | 7.6 | 7.4 | 7.6 | 7.4 | 6.1
dev-clean | 100hr quantized | 7.8 | 4.7 | 4.3 | 4.7 | 4.3 |
100hr | 7.0 | 4.4 | 4.1 | 4.4 | 4.1 | 2.7
960hr quantized | 4.7 | 3.3 | 3.2 | 3.2 | 3.1 |
960hr | 4.2 | 3.2 | 3.2 | 3.2 | 3.2 | 2.0
dev-other | 100hr quantized | 15.8 | 11.0 | 9.9 | 11.0 | 9.8 |
100hr | 15.0 | 10.7 | 9.8 | 10.6 | 9.7 | 7.9
960hr quantized | 11.0 | 8.7 | 8.1 | 8.7 | 8.2 |
960hr | 10.3 | 7.8 | 7.5 | 7.8 | 7.5 | 5.9

Measurements are done with 100-hour and 960-hour pretrained models. With the
language model, the WER is at least $\sim$30 pcs better than the WER measured
for inference without language model. Furthermore, there is at least $\sim$3
pcs improvement with the beam size of 100 compared to beam size of 10. With
the quantized language model, the WER is $\sim$25 pcs better than inference
run without language model. Between quantized and unquantized models, the
unquantized language model yields the best error rate. However the quantized
model has lower memory footprint which may be desirable for low resource edge
devices such as smart speakers, displays, thermostats, etc. The other datasets
show higher WER than clean data sets. This is expected since the other data
set is noisier than clean datasets. Also, server side WER is lower in all the
cases because in the paper [1] unquantized Wav2Vec and language model with
1500 beam size was used. On low resource edge devices, it is not feasible to
use very high beam size because memory won’t be sufficient to support it.
Considering the memory constraints of embedded environments, experiments with
beam widths of 10 and 100 were considered. Beam width 50 was also tested which
yielded similar results as 100.

###  4.2 RTF

RTF(real-time-factor) is defined as inference time divided by utterance
duration. For example, a RTF of 0.5 means a 5 sec utterance can be transcribed
in 2.5 secs. Table 3 shows the RTF of quantized and unquantized models with
different core and thread configurations. Since RTF doesn’t vary across
datasets, just the dev-clean data set was used for RTF evaluations. Different
set of CPU cores were configured to simulate environments similar to embedded
devices like wearables and hand held devices. WER remained the same in
different core and thread configurations. However, RTF increased as the number
of cores allowed to be used by PyTorch reduced. The key finding from below
experiments was that the RTF didn’t degrade much between 3 and 4 cores.
Another experiment was done where the inference was allowed to run on all the
cores (4 cores) but the number of threads that PyTorch can use for intra-op
parallelization was controlled. The default intra-op threads used by PyTorch
is 4. An interesting finding is when 2 threads are configured and allowed to
run on 4 cores, the RTF was very similar to using the default of 4 threads.
Using language model for decoding didn’t noticeably increase the RTF.

Table 3: RTF

(a) Core configs Model | 1 core | 2 cores | 3 cores | 4 cores
---|---|---|---|---
W2V quant | 2.3 | 1.35 | 1.07 | 0.98
W2V | 2.48 | 1.66 | 1.45 | 1.36

(b) Intra-op thread configs Model | 1 thread | 2 threads
---|---|---
W2V quant | 1.51 | 1.13
W2V | 1.70 | 1.42

###  4.3 Energy

Figure 6, 6 , 8 and 8 show the power variation when quantized model is run
with 1, 2, 3 and 4 core environments. From the graph, it can be seen that the
energy consumption is higher as the number of cores increase. The steady state
wattage in normal mode is $\sim 3.1W$ (5.2 V x 0.6 A) on Raspberry Pi. This
increases by $\sim 1.1W$, $\sim 1.7W$, $\sim 2.3W$ and $\sim 2.9W$ for 1,2,3
and 4 cores, respectively during ASR inference.

Figure 5: Power plot with 1 core

Figure 6: Power plot with 2 core

Figure 7: Power plot with 3 core

Figure 8: Power plot with all cores

Figures 10 & 10 shows the energy consumption of quantized and unquantized
models with different core and thread configurations for converting 10 second
long utterance to text. The unquantized full model consumes $\sim 11mWh$ of
energy for 10 secs of utterance whereas the quantized model consumes $\sim
27\%$ less energy compared unquantized model. By selectively configuring fewer
cores or using 2 or fewer intra-op threads, $\sim 20\%$ energy is saved. Using
language model decoding doesn’t noticeably increase the energy consumption.

Figure 9: For quantized models

Figure 10: For unquantized models

###  4.4 Memory and CPU

Figures 12 & 12 shows the memory footprint and CPU utilization during
different evaluation setups. An unquantized language model takes $\sim 40$ pcs
more memory than quantized version. Quantized language model on the other hand
has a better memory footprint and takes 25 pcs more memory compared to an
inference run with no language model. The configuration of 3 cores vs 2 intra-
op threads takes similar amount of CPU.

Figure 11: Memory footprint

Figure 12: CPU load

##  5 Discussion

Different levers can be used to achieve desired accuracy and latency on
Wav2Vec while knowing or limiting the memory, CPU and energy costs of the ASR
inference with those choices. On a device with fewer CPU cores such as smart
thermostat or in multi-user scenarios such as playing the music while sending
a dictated message with smart glass, 2 or 3 cores can be allocated for
inference while leaving the rest of the cores for other tasks. If the device
is less on memory, such as smart watch a quantized model with no language
model can be chosen. Lesser memory comes with reduced ASR accuracy but that
may be tolerable on device with limited set of voice commands. If there is no
memory or CPU resource constraint on the device such as modern smartphone
quantized Wav2Vec model with LM support can be chosen to achieve accurate
speech recognition for accessibility needs or hands free messaging. On a
device with small battery such as a smart watch, the model config with least
amount of energy consumption can be chosen after modeling the typical usage of
the device in a day and how often the speech recognition is used in a day. For
example, if a user sends 10 messages per day using dictation, speech to text
conversion with Wav2Vec quantized model would consume $\sim$9% of
battery(considering Apple Watch battery capacity of 309 mWh) for average
utterance length of 5 secs.

##  6 Conclusion

Wav2Vec model inference on raspberry Pi with different configurations has been
evaluated. The pre-trained and fine tuned model achieves near realtime (RTF=1)
latency with state-of-the-art WER on Raspberry Pi. The memory accuracy of the
model improves $\sim 30\%$ by using language model for decoding albeit with
$\sim 200\%$ of additional memory cost. By using quantized language model,
memory footprint can be reduced substantially with small sacrifice to
accuracy. The energy footprint of quantized model is $\sim 27\%$ lower than
unquantized model. The energy footprint is strongly co-related with CPU usage.
Considering these points, it can be concluded that the model isn’t suitable
for executing on every resource constrained embedded devices because of high
memory footprint. But it can be used in an application on modern smartphone
for on-device speech recognition. With further quantization, pruning and
distillation, resource footprint of Wav2Vec inference can be reduced.

## Acknowledgments and Disclosure of Funding

I would like to thank Vineel Pratap, a Research Engineer and co-worker in
Meta, for discussing the idea of evaluating transformer performance on edge
devices.

## References

  * Baevski et al. [2020] A. Baevski, H. Zhou, A. Mohamed, and M. Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations, 2020.
  * Gondi and Pratap [2021a] S. Gondi and V. Pratap.  Performance evaluation of offline speech recognition on edge devices.  _Electronics_ , 10(21), 2021a.  ISSN 2079-9292.  doi: 10.3390/electronics10212697.  URL https://www.mdpi.com/2079-9292/10/21/2697.
  * Gondi and Pratap [2021b] S. Gondi and V. Pratap.  Performance and efficiency evaluation of asr inference on the edge.  _Sustainability_ , 13(22), 2021b.  ISSN 2071-1050.  doi: 10.3390/su132212392.  URL https://www.mdpi.com/2071-1050/13/22/12392.
  * Graves [2012] A. Graves.  Sequence transduction with recurrent neural networks, 2012.
  * Graves et al. [2006] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber.  Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks.  In _Proceedings of the 23rd International Conference on Machine Learning_ , ICML ’06, page 369–376, New York, NY, USA, 2006. Association for Computing Machinery.  ISBN 1595933832.  doi: 10.1145/1143844.1143891.  URL https://doi.org/10.1145/1143844.1143891.
  * Heafield [2011] K. Heafield.  Kenlm: Faster and smaller language model queries.  In _In Proc. of the Sixth Workshop on Statistical Machine Translation_ , 2011.
  * Hinton et al. [2012] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury.  Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups.  _IEEE Signal Processing Magazine_ , 29(6):82–97, 2012.  doi: 10.1109/MSP.2012.2205597.
  * Hochreiter and Schmidhuber [1997] S. Hochreiter and J. Schmidhuber.  Long Short-Term Memory.  _Neural Computation_ , 9(8):1735–1780, 11 1997\.  ISSN 0899-7667.  doi: 10.1162/neco.1997.9.8.1735.  URL https://doi.org/10.1162/neco.1997.9.8.1735.
  * [9] huggingface wav2vec2.  https://huggingface.co/facebook/wav2vec2-base-960h.
  * [10] LibriSpeech language models.  https://www.openslr.org/11/.
  * [11] Linux taskset.  https://man7.org/linux/man-pages/man1/taskset.1.html.
  * M. P. Lewis, G. F. Simon, and C. D. Fennig. [2016] M. P. Lewis, G. F. Simon, and C. D. Fennig.  ennig. Ethnologue: Languages of the world, nineteenth edition.  http://www.ethnologue.com, 2016.
  * [13] Makerhawk.  Um25c usb power meter.  https://www.makerhawk.com/.
  * Panayotov et al. [2015] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur.  Librispeech: An asr corpus based on public domain audio books.  In _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5206–5210, 2015.  doi: 10.1109/ICASSP.2015.7178964.
  * Peinl et al. [2020] R. Peinl, B. Rizk, and R. Szabad.  Open source speech recognition on edge devices.  In _2020 10th International Conference on Advanced Computer Information Technologies (ACIT)_ , pages 441–445, 2020.  doi: 10.1109/ACIT49673.2020.9208978.
  * [16] pyctcdecode.  https://github.com/kensho-technologies/pyctcdecode.
  * [17] PyTorch.  https://pytorch.org/.
  * [18] PyTorch Jit.  https://pytorch.org/docs/stable/jit.html.
  * [19] PyTorch Mobile.  https://pytorch.org/mobile/home/.
  * [20] PyTorch Threading.  https://pytorch.org/docs/stable/notes/cpu_threading_torchscript_inference.html.
  * [21] QNNPACK.  https://github.com/pytorch/QNNPACK.
  * Staudemeyer and Morris [2019] R. C. Staudemeyer and E. R. Morris.  Understanding lstm – a tutorial into long short-term memory recurrent neural networks, 2019.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.  Attention is all you need, 2017.
  * Vijayakumar et al. [2018] A. K. Vijayakumar, M. Cogswell, R. R. Selvaraju, Q. Sun, S. Lee, D. Crandall, and D. Batra.  Diverse beam search: Decoding diverse solutions from neural sequence models, 2018.
  * Yann LeCun and Ishan Misra [2021] Yann LeCun and Ishan Misra.  Self-supervised learning: The dark matter of intelligence.  https://ai.facebook.com/blog/self-supervised-learning-the-dark-matter-of-intelligence/, 2021\.