# Generating Holistic 3D Human Motion from Speech

Hongwei Yi1∗ Hualin Liang2∗ Yifei Liu2∗ Qiong Cao3†
Yandong Wen1 Timo Bolkart1 Dacheng Tao34 Michael J. Black1†
1Max Planck Institute for Intelligent Systems, Tübingen, Germany
2South China University of Technology 3JD Explore Academy 4The University of
Sydney
{hongwei.yi, yandong.wen, timo.bolkart<EMAIL_ADDRESS>
{hualinliang3, yifei9697, mathqiong2012<EMAIL_ADDRESS>
###### Abstract

This work addresses the problem of generating 3D holistic body motions from
human speech. Given a speech recording, we synthesize sequences of 3D body
poses, hand gestures, and facial expressions that are realistic and diverse.
To achieve this, we first build a high-quality dataset of 3D holistic body
meshes with synchronous speech. We then define a novel speech-to-motion
generation framework in which the face, body, and hands are modeled
separately. The separated modeling stems from the fact that face articulation
strongly correlates with human speech, while body poses and hand gestures are
less correlated. Specifically, we employ an autoencoder for face motions, and
a compositional vector-quantized variational autoencoder (VQ-VAE) for the body
and hand motions. The compositional VQ-VAE is key to generating diverse
results. Additionally, we propose a cross-conditional autoregressive model
that generates body poses and hand gestures, leading to coherent and realistic
motions. Extensive experiments and user studies demonstrate that our proposed
approach achieves state-of-the-art performance both qualitatively and
quantitatively. Our dataset and code are released for research purposes at
https://talkshow.is.tue.mpg.de/.

Figure 1:  Speech-to-motion translation example. Given a speech signal as
input, our approach generates realistic, coherent, and diverse holistic body
motions, that is, the body motion together with facial expressions and hand
gestures. From top to bottom: the input audio, the corresponding transcript,
video frames, and the generated motions. Note that the audio is the only input
to our approach, while the transcript and video frames are just shown as
reference. **footnotetext: Equal Contribution.$\dagger$$\dagger$footnotetext:
Joint Corresponding Authors.

##  1 Introduction

According to linguistics and psychology studies, humans may use body language
to convey the emotions and the attitudes for a more impact message when giving
talks [21, 28]. Motion cues such as facial expression, body posture and hand
movement all play a part. For instance, people may change their gestures when
shifting to a new topic [48], or wave hands when greeting to the audience.
Recent methods have shown rapid progress on studying the translation from
human speech to body motion, and can be roughly divided into rule-based [34]
and learning-based [30, 31, 19, 20, 22, 51] methods. Often, the body motion in
these methods is depicted by either voxel-based facial expression [26], or 3D
mesh-based head/upper-body movement [12, 42, 15, 41, 3], or 2D/3D joints of
face, hands, and body [20, 22, 51]. However, this is not sufficient to
understand human behavior. Humans communicate with their bodies, hands and
facial expression details together. Capturing such coordinated activities as
well as the full 3D surface in tune with speech is critical for virtual agents
to behave realistically and interact with listeners meaningfully.

In this work, we focus on generating the conversational body, hand gestures as
well as the facial expression of a talking person from speech. To do this, we
must learn a cross-modal mapping between audio and 3D holistic body motion,
which is very challenging in practice for several reasons. First, datasets of
3D holistic body meshes and synchronous speech recordings are scarce, and they
are difficult to acquire due to the complex motion capture systems. Second,
real humans often vary in shape, and their faces and hands are highly
deformable. It is not trivial to generate both realistic and stable results of
3D holistic body mesh efficiently. Lastly, as different body parts correlate
differently with speech audio, it is difficult to model cross-modal mapping
and generate realistic and diverse holistic body motions.

We address the above challenges and learn to model the conversational dynamics
in a data-driven way. Firstly, to overcome the issue of data scarcity, we
present a new set of 3D holistic body mesh annotations with synchronous audio
from in-the-wild videos. This dataset was previously used for learning 2D/3D
gesture modeling with 2D body keypoint annotations [20] and 3D keypoint
annotations of the holistic body [22] by applying existing models separately.
Apart from facilitating speech and motion modeling, our dataset can also
support broad research topics like realistic digital human rendering. Then, to
support our data-driven approach to modeling speech-to-motion translation, an
accurate holistic body mesh is needed. Existing models of 3D body have focused
on capturing either the body shape and pose isolated from hands and faces [6,
25], or the different parts together yet resulting in unrealistic or unstable
results especially when applied to video sequences [37, 17, 55]. To solve
this, we present SHOW, which stands for ``Synchronous Holistic Optimization in
the Wild''. Specifically, SHOW adapts SMPLify-X [37] to the videos of talking
persons, and further improves it with several good practices in terms of
stability, accuracy, and efficiency. Figure 9 shows our reconstruction
results.

Lastly, we investigate the translation from audio to 3D holistic body motion
represented as a 3D mesh (Figure Generating Holistic 3D Human Motion from
Speech). We propose TalkSHOW, the first approach to autoregressively
synthesize realistic and diverse body, hand gestures as well as the facial
expression of a talking person from speech. Motivated by the fact that the
face part (i.e. mouth region) is strongly correlated with the audio signal,
while the body and hands are less correlated or even uncorrelated, TalkSHOW
designs separate motion generators for different parts and gives each part
full play. For the face part, to model the highly correlated nature of
phoneme-to-lip motion, we design a simple encoder-decoder based face generator
that encodes rich phoneme information by incorporating the pretrained wav2vec
2.0 [5]. On the other hand, to predict the non-deterministic body and hands
motion, we devise a novel VQ-VAE [46] based framework to learn a compositional
quantized space of motion, which efficiently captures a diverse range of
motions. With the learned discrete representation, we further propose a novel
autoregressive model to predict a multinomial distribution of future motion,
cross-conditioned between existing motions. From this, a wide range of motion
modes representing coherent poses can be sampled, leading to more realistic
motion generation.

We quantitatively evaluate the realism and diversity of our synthesized motion
compared to ground truth and baseline methods and ablations. To further
corroborate our qualitative results, we evaluate our approach through an
extensive user study. Both quantitative and qualitative studies demonstrate
the state-of-the-art quality of our speech-synthesized full expressive 3D
character animations.

| Dataset
---
| Head
---
| Hand
---
| Body
---
| Holistic Body
---
Connection
| In-the-wild
---
| Length
---
| Annotations
---
Multiface [49] | 3D mesh | ✘ | ✘ | ✘ | ✘ | - | multi-camera
BIWI [16] | 3D mesh | ✘ | ✘ | ✘ | ✘ | - | 3D-scanner
VOCASET [12] | 3D mesh | ✘ | ✘ | ✘ | ✘ | - | 4D-scan
Takeuchi et.al [44] | ✘ | ✘ | 3D keypoint | ✘ | ✘ | 5h | MoCap
Trinity [19] | ✘ | ✘ | 3D keypoint | ✘ | ✘ | 4h | MoCap
Yoon et.al [53, 54] | ✘ | ✘ | 3D keypoint | ✘ | ✔ | 52h | p-GT
Speech2Gesture [20] | ✘ | 2D keypoint | 2D keypoint | ✘ | ✔ | 144h | p-GT
Habibie et.al [22] | 3D mesh | 3D keypoint | 3D keypoint | ✘ | ✔ | 33h | p-GT
Ours | 3D mesh | 3D mesh | 3D mesh | ✔ | ✔ | 27h | p-GT
Table 1: The comparisons of different speech-to-motion datasets.

##  2 Related work

###  2.1 Holistic body reconstruction

Recently, there have been many efforts on 3D holistic body mesh recovery [24,
50, 11, 37, 55]. SMPLify-X [37] fits a parametric and expressive model SMPL-X
[37], on 2D keypoints obtained by off-the-shelf detectors (e.g. OpenPose [7]).
PIXIE [17] introduces a novel moderator to estimate the confidence of part-
specific features. These features are fused and fed to independent regressors.
PyMAF-X [55] improves the body and hand estimation with spatial alignment
attention, and adapts integration strategy with the elbow-twist compensation
to avoid unnatural wrist poses. In this work, we adapt the optimization-based
SMPLify-X to videos of talking persons, and improve the stability and accuracy
with several good practices in terms of initialization, data term and
regularization.

###  2.2 Speech-to-motion Datasets

The existing speech-to-motion datasets can be roughly categorized as in-house
and in-the-wild. The annotations of in-house datasets [44, 19, 16, 12, 49] are
accurate but often on a small scale since the multi-camera system for data
capturing is expensive and laborious. Moreover, these datasets only provide
annotations of head [16, 49, 12] or body [44, 19], which can not be used in
whole-body generation. To learn richer and more diverse speaking styles and
emotions, [53, 54] propose to use in-the-wild videos. The annotations are
pseudo ground truth (p-GT) given by advanced reconstruction approaches, _e.g._
[7]. However, these released datasets use either 2D keypoints or 3D keypoints
with 3D head mesh for holistic body representation. This disconnected
representation limits the possible applications of the generated talking
motions. In contrast to the aforementioned work, our dataset, reconstructed by
SHOW, consists of holistic body meshes and synchronized speeches, covering a
wide range of body poses, hand gestures, and facial expressions. More details
can be found in Table 1.

###  2.3 Speech-driven motion generation.

Previous studies on speech-driven motion generation can be roughly divided
into rule-based and learning-based methods. Rule-based methods [8, 29, 31, 38]
generate body motions by mapping inputs to pre-collected motion units with
manually designed rules, which are explainable and controllable but expensive
for complex motion patterns. Thanks to more and more publicly released
datasets, motion generation from the speech is significantly advanced by data-
driven approaches. Ginosar et al. [20] translate speech into a sequence of
conversational 2D joints with temporal convolution. Habibie et al. [22]
propose to generate 3D facial meshes and 3D keypoints of the body and hands
from speech. However, these methods can not generate diverse motions from
given speech recordings. To improve the diversity of the generated motion, VAE
[40, 51, 32], VQ-VAE [3], and normalising-flow [1] are also employed in this
problem.

Different from most existing approaches, TalkSHOW adopts two VQ-VAEs to model
the body and hand motions separately. This novel design allows the learned
quantized space to be compositional and more expressive for conversational
gestures. Moreover, we generate the body and hand motions by a cross-
conditional autoregressive model, where the latest motion is conditioned on
the previous body poses and hand gestures.

##  3 Dataset

In this section, we introduce a high-quality audiovisual dataset, which
consists of expressive 3D body meshes at 30fps, and their synchronized audio
of 22K sample rate. The 3D body meshes are reconstructed from in-the-wild
monocular videos and are used as our pseudo ground truth (p-GT) in speech-to-
motion generation. We provide detailed descriptions of this dataset in Sec.
3.1 and highlight several good practices for obtaining more accurate p-GT from
videos in Sec. 3.2. This dataset is shown to be effective for training speech-
to-motion models in our experiments.

###  3.1 Dataset Description

The dataset is built from the in-the-wild talking videos of different persons
with various talking styles. We use the same video sources from [20] for
straightforward comparisons with the previous work. To facilitate the
subsequent 3D body reconstruction, we manually filter the videos if they are
in any following cases: (i) low resolution ($<$720p), (ii) occluded hand(s),
or (iii) invalid download link. The filtering leads to a high-quality dataset
of 26.9 hours from 4 persons. For the mini-batch processing, the raw videos
are cropped into short clips ($<$10 seconds). Direct comparisons to the
existing datasets can be found in Table 1.

Expressive 3D whole-body meshes are reconstructed from these videos and used
as the p-GT. Specifically, the 3D holistic body meshes consist of face, hands,
and bodies in a connected way, which is achieved by adopting a well-designed
3D topology from SMPL-X [37]. As a result, we represent the p-GT of the
dataset as SMPL-X parameters. Given a video clip of $T$ frames, the p-GT
comprises parameters of a shared body shape $\beta\in\mathbb{R}^{300}$, poses
$\left\\{\theta_{t}|\theta_{t}\in\mathbb{R}^{156}\right\\}_{t=1}^{T}$, a
shared camera pose $\theta^{c}\in\mathbb{R}^{3}$ and translation
$\epsilon\in\mathbb{R}^{3}$, and facial expressions
$\left\\{\psi_{t}|\psi_{t}\in\mathbb{R}^{100}\right\\}_{t=1}^{T}$. Here the
pose $\theta_{t}$ includes jaw pose $\theta^{jaw}_{t}\in\mathbb{R}^{3}$, body
pose $\theta^{b}_{t}\in\mathbb{R}^{63}$, and hand pose
$\theta^{h}_{t}\in\mathbb{R}^{90}$.

We note that this dataset can not only be used in speech and motion modeling,
but also supports broad research topics like realistic digital human rendering
and holistic body recovery from videos, etc.

Figure 2:  The 3D holistic body reconstruction results from SMPLify-X, PIXIE,
PyMAF-X, and ours. Compared to other methods, ours produces more accurate and
stable results with details.

###  3.2 Good Practices for Improving p-GT

Numerous methods can be used to reconstruct 3D whole-body meshes from a single
image [37, 17, 55]. In this work, we adopt an optimization based approach,
SMPLify-X [37], for its good performance in 3D whole-body reconstruction. To
further improve the stability, accuracy, and efficiency, we present SHOW which
adapts SMPLify-X to the videos of talking persons with several good practices.
In the following, we briefly summarize our efforts for improving the p-GT.
More details are given in the supplemental material.

Initialization. A good initialization can significantly accelerate and
stabilize the SMPLify-X optimization. For this reason, we employ several
advanced regression-based approaches to the videos, and use the resulting
predictions as the initial parameters of SMPLify-X. Specifically, PIXIE [17],
PyMAF-X [55], and DECA [18] are used to initialize $\theta^{b}$, $\theta^{h}$,
and $\theta^{f}$, respectively. The camera is assumed to be static, and its
parameters $\theta^{c}$ and $\epsilon$ are estimated by PIXIE [17] as well.

Data Term. The joint re-projection loss is the most important data objective
function in SMPLify-X, as it optimizes the difference between joints extracted
from the SMPL-X model, projected into the image, with joints predicted with
OpenPose [7]. Here we extend the data term by incorporating body silhouettes
from DeepLab V3, facial landmarks from MediaPipe [27], and facial shapes from
MICA [57]. Further, we use a photometric loss between the rendered faces and
the input image to better capture facial details.

Regularization. Different regularization terms in SMPLify-X prevent the
reconstruction of unrealistic bodies. To derive more reasonable
regularizations, we explicitly take the video prior into account and make the
following assumptions. First, the speaker in each video clip remains the same.
This is further verified by a face recognition pipeline using the ArcFace
model [13]. So we can use a consistent parameter $\beta$ to represent the
holistic body shape. Second, the holistic body pose, facial expression, and
environmental lighting in video clips change smoothly over time. This temporal
smoothness assumption has proven useful in many previous approaches [57, 52],
and we observe similar improvements in our experiments. Third, the person's
surface does not have self-penetration, which should be self-evident in the
real world.

Overall, the p-GT can be significantly improved by incorporating the
aforementioned practices. The results are demonstrated in Figure 9, and we
recommend watching the supplemental video for more intuitive comparisons.

##  4 Method

Figure 3: Overview of the proposed TalkSHOW. We employ a simple encoder-
decoder model for face motions, and a novel framework for body and hand
motions. Specifically, this novel framework first learns VQ-VAEs on each piece
separately to obtain a compositional quantized space which efficiently
captures a diverse range of motion. From the learned discrete codebooks, we
can sample a wide range of plausible motion modes. Then, we autoregressively
predict a multinomial distribution of the body/hand motion at future timestep.
Our predictor is designed to be cross-conditioned between the body and hand
motions to keep the synchronization of the holistic body. Lastly, we obtain
the future body/hand motions by decoding a sampled codebook index sampled from
the distribution. The blue, black, and green lines indicate processes involved
at training and inference, training, and inference stages, respectively. (Best
viewed in color.)

Given a speech recording, our goal is to generate conversational body poses,
hand gestures as well as facial expressions that match the speech in a
plausible way. Motivated by the fact that the face part is highly correlated
to the speech signal, while the body and hand parts are less correlated, we
propose TalkSHOW, a novel framework that can model speech and different human
parts separately. In the following, we present an encoder-decoder based face
generator in Sec. 4.2, and a body and hand generator in Sec. 4.3.

###  4.1 Preliminary

Let $M=\\{m_{t}\\}^{T}_{t=1}$ be a temporal sequence of the p-GT holistic
motions $m_{i}$ provided in Sec. 3. We use $M^{f}$, $M^{b}$ and $M^{h}$ to
denote the motion of the face, body and hands, respectively. For time interval
$[1,t]$, the segments of facial, body and hand motions are denoted as
$M^{f}_{1:t}=(m^{f}_{1},\ldots,m^{f}_{t})$,
$M^{b}_{1:t}=(m^{b}_{1},\ldots,m^{b}_{t})$ and
$M^{h}_{1:t}=(m^{h}_{1},\ldots,m^{h}_{t})$, respectively. Specifically, at
each time step $t$, the facial representation
$m^{f}_{t}=[\theta^{f}_{t},\psi_{t}]\in\mathbb{R}^{103}$ is a concatenation of
jaw orientation and expression, and the body and hand motions are represented
by their poses $m^{b}_{t}=\theta^{b}_{t}\in\mathbb{R}^{63}$ and
$m^{h}_{t}=\theta^{h}_{t}\in\mathbb{R}^{90}$, respectively.

###  4.2 Face Generator

Given a raw audio $A_{1:T}$, our face generator $G_{F}$ aims to generate
expressive facial motions
$\widehat{M}^{f}_{1:T}=(\widehat{m}^{f}_{1},\ldots,\widehat{m}^{f}_{T})\in\mathbb{R}^{103\times
T}$ close to $M^{f}_{1:T}\in\mathbb{R}^{103\times T}$.

Figure 3 (A) illustrates our idea. In order to produce synchronized mouth
motions [15], we leverage a pretrained speech model, wav2vec 2.0 [5].
Specifically, the encoder consists of an audio feature extractor and a
transformer encoder [47], leading to a 768-dimensional speech representation.
A linear projection layer is added on top of the encoder to reduce the
dimension to 256. Our decoder comprises six layers of temporal convolutional
networks (TCNs) followed by a full-connected layer. We train the encoder and
decoder with Mean Square Error (MSE) loss.

###  4.3 Body and Hand Generator

Given an audio input, we aim to generate a temporal sequence of realistic and
diverse motions for the body and hands, _i.e._
$\widehat{M}^{b}_{1:T}=(\widehat{m}^{b}_{1},\ldots,\widehat{m}^{b}_{T})\in\mathbb{R}^{63\times
T}$ and
$\widehat{M}^{h}_{1:T}=(\widehat{m}^{h}_{1},\ldots,\widehat{m}^{h}_{T})\in\mathbb{R}^{90\times
T}$, respectively. Figure 3 (B) illustrates our idea. Instead of learning a
direct mapping from audio to motion, we first leverage the recent advances of
VQ-VAE [46] to learn a multi-mode distribution space for body and hand
motions. Specifically, we encode and quantize the body and hand motions into
two finite codebooks, from which we can sample a wide range of plausible body
and hand combinations. Then, we introduce a novel cross-conditional
autoregressive model over the learned codebooks, which allows us to predict
diverse body and hand motions. Our predictor is designed to be cross-
conditioned between the body and hand to keep the synchronization of the
holistic body. Lastly, we obtain the future body/hand motions by decoding
codebook indices sampled from the distribution.

#### Representation.

We use 64-dimensional MFCC features [43] as audio representation for body and
hand generation, i.e., $A_{1:T}=(a_{1},\ldots,a_{T})\in\mathbb{R}^{64\times
T}$. Since body and hand gestures are more correlated to the rhythm and beat
instead of phonemes, low-dimensional MFCC features are sufficient to produce
plausible gestures from audio. Besides, considering that speakers often
present different motion styles, we also leverage the modality of speaker
identity $I$ to differentiate those styles. We represent it as a one-hot
vector $I\in\\{0,1\\}^{N_{I}}$, where $N_{I}$ is the number of speakers.

#### Compositional Quantized Motion Codebooks.

The vanilla VQ-VAE learns a discrete codebook
$\mathcal{Z}=\\{z_{i}\\}_{i=1}^{|\mathcal{Z}|}$ consisting of multiple vectors
$z_{i}\in\mathbb{R}^{d_{z}}$ to quantize the latent space of input. To further
expand the range that the learned codebook can represent, we divide the
motions into compositional pieces, i.e., body and hands, and learn VQ-VAEs on
each piece separately. By doing this, the body and hand movements are encoded
and quantized into two separate finite codebooks
$\mathcal{Z}^{b}=\\{z^{b}_{i}\\}_{i=1}^{|\mathcal{Z}_{b}|}$ and
$\mathcal{Z}^{h}=\\{z^{h}_{j}\\}_{j=1}^{|\mathcal{Z}_{h}|}$, where
$z^{b}_{i},z^{h}_{i}\in\mathbb{R}^{d_{z}}$ with lengths $|\mathcal{Z}^{b}|$
and $|\mathcal{Z}^{h}|$ respectively, from which we can combine
$|\mathcal{Z}^{b}|\times|\mathcal{Z}^{h}|$ different body-hands pose code
pairs $(z^{b}_{i},z^{h}_{j})$ to expand motion diversity. In this scheme,
given an input of the sequence of body and hand motions
$M^{b}_{1:T}\in\mathbb{R}^{63\times T}$ and
$M^{h}_{1:T}\in\mathbb{R}^{90\times T}$, we first encode them into the feature
sequence
$E^{b}_{1:\tau}=(e^{b}_{1},\ldots,e^{b}_{\tau})\in\mathbb{R}^{64\times\tau}$
and
$E^{h}_{1:\tau}=(e^{h}_{1},\ldots,e^{h}_{\tau})\in\mathbb{R}^{64\times\tau}$,
where $\tau=\frac{T}{w}$ and $w$ is the temporal window size. Then, we
quantize the embedding by mapping it into the nearest code in the
corresponding codebook:

$\displaystyle z^{b}_{t}$
$\displaystyle=\mathop{\arg\min_{z^{b}_{k}\in\mathcal{Z}^{b}}}\|e^{b}_{t}-z^{b}_{k}\|\in\mathbb{R}^{64},$
(1) $\displaystyle z^{h}_{t}$
$\displaystyle=\mathop{\arg\min_{z^{h}_{k}\in\mathcal{Z}^{h}}}\|e^{h}_{t}-z^{h}_{k}\|\in\mathbb{R}^{64}.$

Finally, the quantized features
$Z^{b}_{1:\tau}=(z^{b}_{1},\dots,z^{b}_{\tau})\in\mathbb{R}^{64\times\tau}$
and
$Z^{h}_{1:\tau}=(z^{h}_{1},\dots,z^{h}_{\tau})\in\mathbb{R}^{64\times\tau}$
are fed into the decoder for the synthesis.

We train the encoder and decoder simultaneously with the codebook with the
following loss function:

$\displaystyle\mathcal{L}_{VQ}$
$\displaystyle=\mathcal{L}_{rec}(M_{1:T},\widehat{M}_{1:T})+\left\|\operatorname{sg}[E_{1:T}]-Z_{1:T}\right\|$
(2)
$\displaystyle+\beta\left\|E_{1:T}-\operatorname{sg}\left[Z_{1:T}\right]\right\|,$

where $\mathcal{L}_{rec}$ is an MSE reconstruction loss, $\operatorname{sg}$
is a stop gradient operation [10] to calculate codebooks loss, and the third
part is a ``commitment'' loss with a trade-off $\beta$.

#### Cross-Conditional Autoregressive Modeling.

After we learn the compositional quantized codebooks, any body and hand
motions can be represented as a sequence of codebook vectors via the encoder
and quantization. Following the common paradigm, we use the representation of
a sequence of corresponding codebook indices, in the form of one-hot vectors,
of the nearest codebook entry per element, which is denoted as
$C^{b}_{1:\tau}=(c^{b}_{1},\dots,c^{b}_{\tau})\in\mathbb{R}^{|\mathcal{Z}^{b}|\times\tau}$
and
$C^{h}_{1:\tau}=(c^{h}_{1},\dots,c^{h}_{\tau})\in\mathbb{R}^{|\mathcal{Z}^{h}|\times\tau}$.

Now, with the quantized motion representation, we design a temporal
autoregressive model over it to predict the distribution of possible next
motions, given the input audio embedding $A$ and existing motions. Besides, we
enable the modality input of identity $I$ to distinguish different gesture
styles. Because we model the body and hands independently, to keep the
consistency of the holistic body and thus predict realistic gestures, we
exploit the mutual information and design our model to be cross-conditioned
between the body and hand motions. Specifically, following Bayes’ Rule, we
model the joint probability of $C^{b}_{1:\tau}$ and $C^{h}_{1:\tau}$ as
follows:

$\displaystyle p(C^{b}_{1:\tau},C^{h}_{1:\tau}\mid
A_{1:\tau},I)=\prod_{t=1}^{\tau}$ $\displaystyle p\left(c^{b}_{t}\mid
c^{b}_{<t},c^{h}_{<t},a_{\leq t},I\right)$ (3) $\displaystyle
p\left(c^{h}_{t}\mid c^{b}_{\leq t},c^{h}_{<t},a_{\leq t},I\right).$

Note that our cross-condition modeling between the body and hand motions makes
the most of mutual information in two ways: (1) the current body/hand motions
(i.e. $c^{b}_{t}$/$c^{h}_{t}$) depend on past hand/body motions information
(i.e. $c^{b}_{<t}$/$c^{h}_{<t}$); (2) we argue that the current body motion
$c^{b}_{t}$ is also responsible for predicting the distribution of current
hand motions. Such modeling guarantees the coherence of the body and hand
motions as a whole and thus achieves realistic gestures. Gated PixelCNN [45]
is adopted to model these quantities. During the training phase, the quantized
body/hand motions representation concatenated with the audio and identity
features is used for training. A teacher-forcing scheme and cross-entropy loss
are adopted for the optimization. At inference, the model predicts multinomial
distributions of the future body and hand motions, from which we can sample to
acquire codebook indices for each motion. A codebook lookup is then conducted
to retrieve the corresponding quantized element of motion, which we feed into
the decoder for the final synthesis. Figure 3 (B) illustrates the pipeline.
More training details are given in the supplemental material.

##  5 Experiments

Method | Face
---|---
|  L2 $\downarrow$
---
|  LVD $\downarrow$
---
Habibie et al. [22] | 0.139 | 0.257
TalkSHOW (Ours) | 0.130 | 0.248
Method | Body&Hands
|  RS $\uparrow$
---
|  Variation $\uparrow$
---
Habibie et al. [22] | 0.146 | 0
Audio Encoder-Decoder | 0.214 | 0
Audio VAE | 0.182 | 0.044
Audio+Motion VAE | 0.240 | 0.176
TalkSHOW (Ours) | 0.414 | 0.821
Table 2: Comparison to Habibie et al. [22] and several baselines. $\uparrow$
indicates higher is better and $\downarrow$ indicates lower is better.

We evaluate the ability of our method in generating body motions from the
speech on the created dataset both quantitatively and qualitatively.
Specifically, we choose video sequences longer than 3s and split them into
80%/10%/10% for the train/val/test set. Several metrics are used to measure
the realism and diversity of the generated motions including facial expression
and hand poses. Furthermore, we conduct user studies to assess the performance
of our method.

###  5.1 Experimental Setup

#### Evaluation Metrics.

Because we model face motion as a deterministic task and the body and hand
motions as a non-deterministic task, we assess the generated motion in terms
of the realism and the synchronization of face motion, the realism and the
diversity of body and hand motions. Specifically, the following metrics are
adopted:

  * •

L2: L2 distance between p-GT and generated facial landmarks, including jaw
joints and lip shape [56, 35].

  * •

LVD: Landmark Velocity Difference calculates the velocity difference between
p-GT and generated facial landmarks, which measures the synchronization
between speech and facial expression [56].

  * •

RS: Score on the realism of the generated body and hand motion. Following [51,
2], we trained a binary classifier to discriminate real samples from fake ones
and the prediction represents the realistic score.

  * •

Variation: As used in [35], diversity is measured by the variance across the
time series sequence of body and hand motions.

#### Compared methods.

We compare TalkSHOW to Habibie et al. [22], a SOTA speech-to-motion method.
Besides, we compare several baselines for modeling body and hand motions when
using the same face generator as ours:

  * •

Audio Encoder-Decoder. It encodes input audio and outputs motions, which is
used by [20, 22].

  * •

Audio VAE. Given the input audio, the VAE-like structure encodes audio into a
Gaussian distribution, and then the sampled audio is fed into the decoder,
which transforms the sample into motions.

  * •

Audio+Motion VAE. Given the input motion and audio, it adopts a VAE-like
structure with two encoders to encode motion and audio into Gaussian
distribution, respectively, and then the sampled motion and audio are
concatenated and fed into the decoder for the synthesis.

###  5.2 Quantitative Analysis

Table 2 shows the comparison results. We see that our method outperforms
Habibie et al. [22] across all metrics. Particularly, our method surpasses it
in terms of L2 and LVD, which demonstrates the effectiveness of our face
generator for generating realistic facial expressions. Besides, our method
significantly outperforms it in terms of variation, which demonstrates the
powerful capacity to generate diverse body and hand motions benefiting from
our proposed compositional quantized motion representation. Moreover,
regarding the realism (RS) for body and hand motion, we surpass Habibie et al.
[22] considerably, which confirms the effectiveness of our cross-conditioned
autoregressive model in generating realistic motion.

On the other hand, compared to VAE-based models, our method achieves
significant gains in both realism and diversity. In particular, we obtain a
much higher result in diversity. This indicates the advantage of the learned
compositional quantized motion codebooks, which effectively memorize multiple
motion modes of the body and hands and thus boost the diversity of the
generated body and hand gestures.

###  5.3 Qualitative Analysis

Figure 4:  Our method generates diverse motions consistent with the rhythm of
the input audio. For instance, we can generate different movements of hands
corresponding to the strengthening tone of ``But'' in the speech, e.g. using
left hand (top), right hand (middle), or both hands (bottom).  Figure 5:
Given speech audio as input, our method generates facial expressions with
accurate lip shapes.

Figure 4 shows examples of our generated 3D holistic body motion from speech.
We see that given the word ``But'' from the speech represents a strengthening
tone of voice, our method generates plausible holistic body motions with hands
up before saying ``But'' and hands down after saying ``But''. Notably, the
generated motions are diverse in many aspects, e.g. the range of motion and
which hands to use. For a better illustration of the differences between
generated motion samples, we overlap the motions in the same frame with
different levels of transparency.

Figure 5 illustrates the qualitative performance of our face generator. Our
approach generates realistic face motions including consistent lip motions
with the corresponding phonemes such as /f/, /t/, /b/, and /æ/. Further
examples can be found in the supplemental video.

###  5.4 Model Ablation

#### Effect of wav2vec feature in face generation.

We evaluate the effect of the wav2vec feature used in face generation compared
to the MFCC feature. We add an extra encoder to increase the dimension of the
MFCC feature from 64 to 256 for a fair comparison. The wav2vec-based model
outperforms the MFCC-based model in both metrics (0.130 vs. 0.165 in L2 and
0.251 vs. 0.277 in LVD) due to its more powerful capacity in modeling the
relationship between audio and phoneme. Moreover, we experimentally find that
the wav2vec-based model can generalize well to unseen identities, see
supplementary for more details.

#### Effect of compositional quantized motion codebooks.

We analyze the capability of the proposed compositional quantized motion
codebooks of VQ-VAEs in efficiently capturing the diverse motion modes
represented in motion data, which leads to accurate reconstruction. To this
end, we compare VQ-VAE with a single codebook. Reconstruction Error $RE$ is
adopted as the metric, in which a lower reconstruction loss indicates a higher
capacity. Figure 6 illustrates the results. We see that compared to VQ-VAE
with a single codebook, VQ-VAEs with compositional codebooks yield
consistently lower $RE$ across different codebook sizes. This demonstrates the
effectiveness of the proposed compositional codebooks in modeling the diverse
motion modes.

#### Effect of cross-conditional modeling.

We evaluate the effect of cross-conditional modeling. In contrast to cross-
conditional modeling (w/ c-c), the model without cross condition (w/o c-c)
generates body and hand motions independently. Our method w/ c-c yields a
higher realistic score than that w/o c-c ($0.414$ vs. $0.409$), benefiting
from the cross-conditional modeling between the body and hand motions which
leads to more coherent and realistic motions. Note that due to the higher
realism, our method w/ c-c attains a slight reduction in diversity ($0.821$
vs. $0.922$ in variance), which is reasonable.

Figure 6:  The comparison of VQ-VAE and VQ-VAEs with compositional codebooks.

###  5.5 User Study

We conduct user studies with Google Forms to evaluate the quality of our
reconstruction and generation methods, respectively. We randomly sample 40
videos in total with 10 videos from each speaker. Ten participants take part
in it.

#### Reconstruction.

We assess the quality of our holistic body reconstruct results against PyMAF-X
[55], compared with the ground truth. Participants are asked to answer the
following questions with Yes or No: Does the reconstructed
face/hands/body/full-body match the input video? Table 3 reports the average
percentage of answers that the reconstructed results match the input video. We
see that our method outperforms PyMAF-X by a large margin.

| Method
---
| face
---
| body
---
| hands
---
| holistic body
---
PyMAF-X [55] | 0.323 | 0.500 | 0.438 | 0.193
SHOW (ours) | 0.898 | 0.738 | 0.800 | 0.768
Table 3:  User study results on reconstruction. For each method, we report the
average percentage of answers that the reconstructed results match the input
video.

#### Motion Generation.

We use A/B testing to evaluate our generation results, compared to the p-GT
and Habibie et al. [22]. Specifically, participants are asked to answer the
following questions with A or B: For the face/body&hands/overall region, which
one is a better match with the given speech? Table 4 reports the average
preference percentage of answers. We see that participants favor our method
over Habibie et al. in terms of all the regions. Not surprisingly,
participants perceive the p-GT better over both methods, with our method
preferred by much more users.

| Method
---
| face
---
| body and hands
---
| holistic body
---
[22] vs. p-GT | 0.153 | 0.141 | 0.169
TalkSHOW (Ours) vs. p-GT | 0.478 | 0.464 | 0.458
TalkSHOW (Ours) vs. [22] | 0.888 | 0.910 | 0.913

Table 4:  User study results on motion generation. We use A vs. B testing and
report the average percentage of answers where A is preferred over B.

##  6 Conclusion

In this work, we propose the first approach TalkSHOW to generate 3D holistic
body meshes from speech. We devise a simple and effective encoder-decoder for
realistic face generation with accurate lip shape. For body and hands, we
enable diverse generation and coherent prediction with compositional VQ-VAE
and cross-conditional modeling, respectively. Moreover, we contribute a new
set of accurate 3D holistic body meshes with synchronous audios from in-the-
wild videos. The annotations are obtained by an empirical approach designed
for videos. Experimental results demonstrate that our proposed approach
achieves state-of-the-art performance both qualitatively and quantitatively.

Acknowledgments. We thank Wojciech Zielonka, Justus Thies for helping us
incorporate MICA into our reconstruction method, Changxing Ding, Huaiguang
Jiang, Yao Feng, Zhen Liu, and Weiyang Liu for the insightful discussions, and
Benjamin Pellkofer for IT support. This work was supported by the German
Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ:
01IS18039B.

Disclosure. MJB has received research gift funds from Adobe, Intel, Nvidia,
Meta/Facebook, and Amazon. MJB has financial interests in Amazon, Datagen
Technologies, and Meshcapade GmbH.

## References

  * Alexanderson et al. [2020] Simon Alexanderson, Gustav Eje Henter, Taras Kucherenko, and Jonas Beskow.  Style-controllable speech-driven gesture synthesis using normalising flows.  In _Computer Graphics Forum_ , volume 39, pages 487–496. Wiley Online Library, 2020.
  * Aliakbarian et al. [2020] Sadegh Aliakbarian, Fatemeh Sadat Saleh, Mathieu Salzmann, Lars Petersson, and Stephen Gould.  A stochastic conditioning scheme for diverse human motion prediction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5223–5232, 2020.
  * Ao et al. [2022] Tenglong Ao, Qingzhe Gao, Yuke Lou, Baoquan Chen, and Libin Liu.  Rhythmic gesticulator: Rhythm-aware co-speech gesture synthesis with hierarchical neural embeddings.  _arXiv preprint arXiv:2210.01448_ , 2022.
  * Ba et al. [2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Baevski et al. [2020] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in Neural Information Processing Systems_ , 33:12449–12460, 2020.
  * Bogo et al. [2016] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black.  Keep it smpl: Automatic estimation of 3d human pose and shape from a single image.  In _European conference on computer vision_ , pages 561–578. Springer, 2016.
  * Cao et al. [2017] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7291–7299, 2017.
  * Cassell et al. [2001] Justine Cassell, Hannes Högni Vilhjálmsson, and Timothy Bickmore.  Beat: the behavior expression animation toolkit.  In _Proceedings of the 28th annual conference on Computer graphics and interactive techniques_ , pages 477–486, 2001.
  * Chen et al. [2017] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  _CoRR_ , abs/1706.05587, 2017.  URL http://arxiv.org/abs/1706.05587.
  * Chen and He [2021] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15750–15758, 2021.
  * Choutas et al. [2020] Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black.  Monocular expressive body regression through body-driven attention.  In _European Conference on Computer Vision (ECCV)_ , pages 20–40, 2020.  URL https://expose.is.tue.mpg.de.
  * Cudeiro et al. [2019] Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael Black.  Capture, learning, and synthesis of 3D speaking styles.  In _Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 10101–10111, 2019.
  * Deng et al. [2019a] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4690–4699, 2019a.
  * Deng et al. [2019b] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong.  Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 0–0, 2019b.
  * Fan et al. [2022] Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura.  Faceformer: Speech-driven 3d facial animation with transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18770–18780, 2022.
  * Fanelli et al. [2010] Gabriele Fanelli, Juergen Gall, Harald Romsdorfer, Thibaut Weise, and Luc Van Gool.  A 3-D Audio-Visual Corpus of Affective Communication.  _IEEE Transactions on Multimedia_ , 12(6):591–598, October 2010.  ISSN 1941-0077.  doi: 10.1109/TMM.2010.2052239.  Conference Name: IEEE Transactions on Multimedia.
  * Feng et al. [2021a] Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, and Michael J Black.  Collaborative regression of expressive bodies using moderation.  In _2021 International Conference on 3D Vision (3DV)_ , pages 792–804. IEEE, 2021a.
  * Feng et al. [2021b] Yao Feng, Haiwen Feng, Michael J Black, and Timo Bolkart.  Learning an animatable detailed 3d face model from in-the-wild images.  _ACM Transactions on Graphics (TOG)_ , 40(4):1–13, 2021b.
  * Ferstl and McDonnell [2018] Ylva Ferstl and Rachel McDonnell.  Investigating the use of recurrent motion modelling for speech gesture generation.  In _Proceedings of the 18th International Conference on Intelligent Virtual Agents_ , pages 93–98, 2018.
  * Ginosar et al. [2019] S. Ginosar, A. Bar, G. Kohavi, C. Chan, A. Owens, and J. Malik.  Learning individual styles of conversational gesture.  In _Computer Vision and Pattern Recognition (CVPR)_. IEEE, June 2019\.
  * Goldin-Meadow [1999] Susan Goldin-Meadow.  The role of gesture in communication and thinking.  _Trends in cognitive sciences_ , 3(11):419–429, 1999.
  * Habibie et al. [2021] Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Lingjie Liu, Hans-Peter Seidel, Gerard Pons-Moll, Mohamed Elgharib, and Christian Theobalt.  Learning speech-driven 3d conversational gestures from video.  In _Proceedings of the 21st ACM International Conference on Intelligent Virtual Agents_ , pages 101–108, 2021.
  * Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _International conference on machine learning_ , pages 448–456. PMLR, 2015.
  * Joo et al. [2018] Hanbyul Joo, Tomas Simon, and Yaser Sheikh.  Total capture: A 3d deformation model for tracking faces, hands, and bodies.  In _Computer Vision and Pattern Recognition (CVPR)_ , pages 8320–8329, 2018.
  * Kanazawa et al. [2018] Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7122–7131, 2018.
  * Karras et al. [2017] Tero Karras, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen.  Audio-driven facial animation by joint end-to-end learning of pose and emotion.  _ACM Transactions on Graphics (TOG)_ , 36(4):1–12, 2017.
  * Kartynnik et al. [2019] Yury Kartynnik, Artsiom Ablavatski, Ivan Grishchenko, and Matthias Grundmann.  Real-time facial surface geometry from monocular video on mobile gpus.  _arXiv preprint arXiv:1907.06724_ , 2019.
  * Kendon [2004] Adam Kendon.  _Gesture: Visible action as utterance_.  Cambridge University Press, 2004.
  * Kopp and Wachsmuth [2004] Stefan Kopp and Ipke Wachsmuth.  Synthesizing multimodal utterances for conversational agents.  _Computer animation and virtual worlds_ , 15(1):39–52, 2004.
  * Levine et al. [2009] Sergey Levine, Christian Theobalt, and Vladlen Koltun.  Real-time prosody-driven synthesis of body language.  In _ACM SIGGRAPH Asia 2009 papers_ , pages 1–10. 2009.
  * Levine et al. [2010] Sergey Levine, Philipp Krähenbühl, Sebastian Thrun, and Vladlen Koltun.  Gesture controllers.  In _ACM SIGGRAPH 2010 papers_ , pages 1–11. 2010.
  * Li et al. [2021] Jing Li, Di Kang, Wenjie Pei, Xuefei Zhe, Ying Zhang, Zhenyu He, and Linchao Bao.  Audio2gestures: Generating diverse gestures from speech audio with conditional variational autoencoders.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 11293–11302, 2021.
  * Maas et al. [2013] Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al.  Rectifier nonlinearities improve neural network acoustic models.  In _Proc. icml_ , volume 30, page 3. Atlanta, Georgia, USA, 2013.
  * Marsella et al. [2013] Stacy Marsella, Yuyu Xu, Margaux Lhommet, Andrew Feng, Stefan Scherer, and Ari Shapiro.  Virtual character performance from speech.  In _Proceedings of the 12th ACM SIGGRAPH/Eurographics symposium on computer animation_ , pages 25–35, 2013.
  * Ng et al. [2022] Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, and Shiry Ginosar.  Learning to listen: Modeling non-deterministic dyadic facial motion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 20395–20405, 2022.
  * Nocedal and Wright [2006] Jorge Nocedal and Stephen J Wright.  Nonlinear equations.  _Numerical Optimization_ , pages 270–302, 2006.
  * Pavlakos et al. [2019] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black.  Expressive body capture: 3d hands, face, and body from a single image.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10975–10985, 2019.
  * Poggi et al. [2005] Isabella Poggi, Catherine Pelachaud, F de Rosis, Valeria Carofiglio, and B De Carolis.  Greta. a believable embodied conversational agent.  In _Multimodal intelligent information presentation_ , pages 3–25. Springer, 2005.
  * Prokudin et al. [2020] Sergey Prokudin, Michael J. Black, and Javier Romero.  SMPLpix: Neural Avatars from 3D Human Models.  Technical Report arXiv:2008.06872, arXiv, November 2020.  URL http://arxiv.org/abs/2008.06872.  arXiv:2008.06872 [cs] type: article.
  * Qian et al. [2021] Shenhan Qian, Zhi Tu, Yihao Zhi, Wen Liu, and Shenghua Gao.  Speech drives templates: Co-speech gesture synthesis with learned templates.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 11077–11086, 2021.
  * Richard et al. [2021] Alexander Richard, Michael Zollhöfer, Yandong Wen, Fernando De la Torre, and Yaser Sheikh.  Meshtalk: 3d face animation from speech using cross-modality disentanglement.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1173–1182, 2021.
  * Sadoughi et al. [2017] Najmeh Sadoughi, Yang Liu, and Carlos Busso.  Meaningful head movements driven by emotional synthetic speech.  _Speech Communication_ , 95:87–99, 2017.
  * Sahidullah and Saha [2012] Md Sahidullah and Goutam Saha.  Design, analysis and experimental evaluation of block based transformation in mfcc computation for speaker recognition.  _Speech communication_ , 54(4):543–565, 2012\.
  * Takeuchi et al. [2017] Kenta Takeuchi, Souichirou Kubota, Keisuke Suzuki, Dai Hasegawa, and Hiroshi Sakuta.  Creating a gesture-speech dataset for speech-based automatic gesture generation.  In _International Conference on Human-Computer Interaction_ , pages 198–202. Springer, 2017.
  * Van den Oord et al. [2016] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al.  Conditional image generation with pixelcnn decoders.  _Advances in neural information processing systems_ , 29, 2016.
  * Van Den Oord et al. [2017] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wagner et al. [2014] Petra Wagner, Zofia Malisz, and Stefan Kopp.  Gesture and speech in interaction: An overview, 2014.
  * Wuu et al. [2022] Cheng-hsin Wuu, Ningyuan Zheng, Scott Ardisson, Rohan Bali, Danielle Belko, Eric Brockmeyer, Lucas Evans, Timothy Godisart, Hyowon Ha, Alexander Hypes, Taylor Koska, Steven Krenn, Stephen Lombardi, Xiaomin Luo, Kevyn McPhail, Laura Millerschoen, Michal Perdoch, Mark Pitts, Alexander Richard, Jason Saragih, Junko Saragih, Takaaki Shiratori, Tomas Simon, Matt Stewart, Autumn Trimble, Xinshuo Weng, David Whitewolf, Chenglei Wu, Shoou-I Yu, and Yaser Sheikh.  Multiface: A dataset for neural face rendering.  In _arXiv_ , 2022.  doi: 10.48550/ARXIV.2207.11243.  URL https://arxiv.org/abs/2207.11243.
  * Xu et al. [2020] Hongyi Xu, Eduard Gabriel Bazavan, Andrei Zanfir, William T Freeman, Rahul Sukthankar, and Cristian Sminchisescu.  Ghum & ghuml: Generative 3d human shape and articulated pose models.  In _Computer Vision and Pattern Recognition (CVPR)_ , pages 6184–6193, 2020.
  * Xu et al. [2022] Jing Xu, Wei Zhang, Yalong Bai, Qibin Sun, and Tao Mei.  Freeform body motion generation from speech.  _arXiv preprint arXiv:2203.02291_ , 2022.
  * Yi et al. [2022] Hongwei Yi, Chun-Hao P. Huang, Dimitrios Tzionas, Muhammed Kocabas, Mohamed Hassan, Siyu Tang, Justus Thies, and Michael J. Black.  Human-aware object placement for visual environment reconstruction.  In _Computer Vision and Pattern Recognition (CVPR)_ , June 2022.
  * Yoon et al. [2019] Youngwoo Yoon, Woo-Ri Ko, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee.  Robots learn social skills: End-to-end learning of co-speech gesture generation for humanoid robots.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 4303–4309. IEEE, 2019.
  * Yoon et al. [2020] Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee.  Speech gesture generation from the trimodal context of text, audio, and speaker identity.  _ACM Transactions on Graphics (TOG)_ , 39(6):1–16, 2020.
  * Zhang et al. [2021] Hongwen Zhang, Yating Tian, Xinchi Zhou, Wanli Ouyang, Yebin Liu, Limin Wang, and Zhenan Sun.  Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 11446–11456, 2021.
  * Zhou et al. [2020] Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li.  Makelttalk: speaker-aware talking-head animation.  _ACM Transactions on Graphics (TOG)_ , 39(6):1–15, 2020.
  * Zielonka et al. [2022] Wojciech Zielonka, Timo Bolkart, and Justus Thies.  Towards metrical reconstruction of human faces.  In _European Conference on Computer Vision (ECCV)_ , 2022.

##  Appendix A Dataset

###  A.1 Dataset Description

Our dataset is built from the in-the-wild talking videos of four persons with
various poses. The dataset contains high-quality 3D holistic body mesh
annotations that are reconstructed from video clips of 26.9 hours in total.
Each clip is less than 10 seconds. Fig. 7 illustrates the distributions of
video durations from different characters.

Figure 7:  The distribution of the number of short clips for each character
(0-10 seconds) of different speakers.

###  A.2 Good Practices for Improving p-GT

Figure 8:  The architecture of SHOW. It consists of initialization and
optimization modules. Specifically, given an input the image sequence,
firstly, PIXIE [17], DECA [14] and PyMAF-X [55] are used to initialize the
parameters of SMPL-X. Secondly, the optimization routine incorporates body
silhouettes from DeepLab V3 [9], facial landmarks from MediaPipe [27], and
facial shapes from MICA [57]. Then, it uses a photometric loss between the
rendered faces and the input image to better capture facial details. Lastly,
SHOW outputs the final results.

#### Preliminary.

The 3D holistic body meshes consist of face, hands, and body, which is
achieved by adopting SMPL-X [37]. It uses standard vertex-based linear blend
skinning with learned corrective blend shapes and has $N=10475$ vertices and
$K=67$ joints. Let $W$ be the linear blend skinning function, the predicted
mesh vertices can be represented as
$v=W(\theta,\psi,\beta)\in\mathbb{R}^{N\times 3}$. Let
$V=\\{v_{t}|v_{t}\in\mathbb{R}^{N\times 3}\\}^{T}_{t=1}$ and
$J=\\{j_{t}|j_{t}\in\mathbb{R}^{67}\\}^{T}_{t=1}$ be the temporal sequence of
mesh vertices and its 3D joint locations regressed from a linear regressor. We
also denote $P^{b}=\\{p^{b}_{t}|p^{b}_{t}\in\mathbb{R}^{32}\\}^{T}_{t=1}$ and
$P^{h}=\\{p^{h}_{t}|p^{h}_{t}\in\mathbb{R}^{24}\\}^{T}_{t=1}$ as the temporal
sequence of the coefficients of the latent space of VPoser and low dimensional
pose space after principal component analysis (PCA) for the body and hands
respectively. For time interval $[1:t]$, $V_{1:t}=(v_{1},...,v_{t})$,
$J_{1:t}=(j_{1},...,j_{t})$, $P^{b}_{1:t}=(p^{b}_{1},...,p^{b}_{t})$ and
$P^{h}_{1:t}=(p^{h}_{1},...,p^{h}_{t})$ represent segments of mesh vertices,
3D joints, body pose, and hand pose, respectively.

#### Initialization.

Since optimization-based methods are often slow and sensitive to the
initialization. In contrast, regression-based methods tend to give a
reasonable, but not well pixel-aligned results. Therefore, we use the results
from PIXIE [17] and PyMAF-X [55] to initialize the parameters of body and hand
pose, respectively. Results from DECA [18] are used to initialize the
parameters of jaw pose and facial expression.

#### Data terms.

We extend the data term by incorporating body silhouettes, facial landmarks,
facial shapes, and facial details.

Firstly, to deal with the imperfect 2D landmarks by Openpose [7], we introduce
the silhouette constraint to encourage the rendered SMPL-X body to be inside
the human body mask. Ground-truth person segmentations are expensive to obtain
for in-the-wild datasets. Hence, we employ an off-the-shelf segmentation
model, Deeplab V3 [9] to generate p-GT person semantics mask
$M_{sil}\in\mathbb{R}^{T\times h\times w}$, where H and W are the height and
width of the input image. Pytorch3D is used as the differential renderer to
process the rendered pixels of all mesh triangles, leading to the predicted
semantics mask $\widehat{M}_{sil}\in\mathbb{R}^{T\times h\times w}$. The
silhouette loss term is given by:

$\mathcal{L}_{sil}=\sum||d(\widehat{M}_{sil})\odot d_{edt}(g(M_{sil}))||_{2},$
(4)

where $g(x)=MaxPool(x)-x$ is a function for detecting the edge of the binary
mask. $d_{edt}$ is a distance function to calculate the smallest Euclidean
distance from the background point to the silhouette boundary.

Secondly, to get a better facial geometry in SMPL-X, we minimize the
difference between the facial shape in SMPL-X and the reconstructed facial
shape from MICA [57]. We term this as a facial shape objective
$\mathcal{L}_{FS}$ given by:

$\mathcal{L}_{FS}=||{M}_{g1}({V_{SMPL-X}})-M_{g2}({V}_{MICA}+{t}_{FS})||_{2},$
(5)

where $V_{SMPL-X}\in\mathbb{R}^{N\times 3}$ is the SMPL-X vertices at neutral
pose (i.e. $\theta=0,\psi=0$). ${V}_{MICA}\in\mathbb{R}^{5023\times 3}$ is the
MICA shape, and ${t}_{FS}\in\mathbb{R}^{3}$ is the offset of ${V}_{MICA}$ from
$V_{SMPL-X}$. $M_{g1}$ and $M_{g2}$ are functions that maps the original mesh
vertices of $V_{SMPL-X}$ and ${V}_{MICA}$ to the corresponding $1787$ vertices
of frontal face part, respectively.

Thirdly, to get better facial expression, we use MediaPipe [27] to extract
$105$ of $468$ dense 2D facial landmarks for each image. The loss term
$\mathcal{L}_{FE}$ is calculated as:

$\mathcal{L}_{FE}=\sum_{t}||{U}_{1:t}-\widehat{U}_{1:t}||_{2},$ (6)

where ${U}_{1:t}$ and $\widehat{U}_{i}$ are temporal segments of landmarks
from MediaPipe [27] and the 2D projection of the corresponding 3D joints
$J_{1:t}$, respectively.

Lastly, to obtain high-frequency resolution facial details, we employ face
expression tracking to monocular RGB images in a self-supervised fashion.
Specially, we follow [18, 57] to reconstruct the face jointly with an
illumination model based on spherical harmonics and a Lambertian material
assumption:

$\mathcal{L}_{FR}=\sum_{t}||I_{r}(\mathcal{M}_{S2F}(V_{1:t}))-I_{1:t}^{head}||_{2},$
(7)

where $\mathcal{M}_{S2F}$ is a function that selects the head part of
${V_{1:t}}$. $I_{r}$ is the forward pass of differential rendering.
$I_{1:t}^{head}$ is the cropped head image from input image. Note that we
choose different scales (e.g. $256,512,1024$) for different stages in the
optimization procedure.

#### Regularization.

Different regularization terms in SMPLify-X prevent the reconstruction of
unrealistic bodies. To derive more reasonable regularization terms, we
explicitly take the video prior into account.

To reduce the jittery results caused by the noisy 2D detected keypoints, we
introduce a smooth term for body and motion poses ($P^{b}$ and $P^{h}$). They
are defined as:

$\mathcal{M}_{b}=\sum_{t}||P^{b}_{2:t}-P^{b}_{1:t-1}||_{2},$ (8)
$\mathcal{M}_{h}=\sum_{t}||P^{h}_{2:t}-P^{h}_{1:t-1}||_{2}.$ (9)

We also add constant-velocity smooth term $\mathcal{M}_{j}$ on ${J}$:

$\mathcal{M}_{j}=\sum_{t}||J_{3:t}+J_{1:t-3}-2\times J_{2:t-2}||_{2},$ (10)

Furthermore, to prevent the inter-penetration of two hands, we use Collision
Penalizer [37] and denote this loss term as $L_{pen}$.

#### Training Losses.

The final objective function is given by:

$E(\beta,\\{\theta\\}_{t=1}^{T},\\{\psi\\}_{t=1}^{T},\psi_{light},\psi_{lbs},t_{FS})=\\\
\sum_{t=1}^{T}(E_{SMPLify-X}(t))+\lambda_{FE}\mathcal{L}_{FE}+\lambda_{FS}\mathcal{L}_{FS}+\lambda_{FR}\mathcal{L}_{FR}+\\\
\lambda_{mb}\mathcal{M}_{b}+\lambda_{mh}\mathcal{M}_{h}+\lambda_{mj}\mathcal{M}_{j}+\lambda_{sil}\mathcal{L}_{sil}+\lambda_{pen}\mathcal{L}_{pen},$
(11)

where $\psi_{light}\in R^{3}$ is the spherical harmonic coefficients
representing the environmental illumination. $\psi_{lbs}\in R^{128}$ is the
linear blend skinning parameters of albedo model. $E_{SMPLify-X}(t)$ is the
basic prior on single image as describe in [37]. Weights $\lambda$ steer the
influence of each term.

#### Optimization.

Following [37], we adopt the Limited-memory BFGS [36] with strong wolfe line
search for optimization. An iterative fitting routine is used for better
fitting. With proper initialization, we minimize the objective function using
a five-stage fitting procedure to avoid the local minima trap and reduce the
optimization time. The learning rate is set to 1. As the required GPU memory
increases dramatically with the image batch size for neural rendering, we use
a mini-batch of $50$ on NVIDIA Tesla V100.

Figure 9:  The 3D holistic body reconstructions of four subjects from
SMPLify-X, PIXIE, PyMAF-X, and ours. Compared to other methods, ours produces
more accurate and stable results with details.

##  Appendix B Network Architecture Details

###  B.1 Face Generator

The raw audio input is normalized to zero mean and unit variance, and then is
fed to encoder, which consists of an audio feature extractor, a transformer
encoder, and a full-connected layer. The audio feature extractor is followed
by an interpolation operation, in which the audio feature is re-sampled into
target frames. For the decoder, it comprises six temporal convolution layers
(with a kernel size, stride and padding of 3, 1 and 1 respectively) and a
full-connected layer. Each temporal convolution layer is followed by layer
normalization [4] and a Leaky RELU activation function [33]. We adopt SGD with
momentum and a learning rate of 0.001 as the optimizer. The face generator is
trained with batchsize of 1 for 100 epochs, in which each batch contains a
full-length audio and corresponding facial motions.

###  B.2 Body and Hand Generator

#### VQ-VAEs Details.

The VQ-VAE takes body or hand motions as input. The encoder of each VQ-VAE is
composed of three residual layers, which includes three temporal convolution
layers (with a kernel size, stride and padding of 3, 1 and 1 respectively)
followed by batch normalization [23] and a Leaky RELU activation function
[33]. The encoder is interleaved with a temporal convolution layer with a
kernel size, stride and padding of 4, 2 and 1 respectively after every
residual layer except the last so that the temporal window size $w$ is equal
to 4. On the top of the encoder, a full-connected layer is added to reduce the
dimension before quantization. The decoder is symmetric with the encoder. We
adopt Adam with $\beta_{1}=0.9$, $\beta_{2}=0.999$ and a learning rate of
0.0001 as the optimizer. The commitment loss weight $\beta$ is set to 0.25.
The VQ-VAEs are trained with a batchsize of 128 and a sequence length of 88
frames for 100 epochs.

#### Autoregressive Model Details.

The autoregressive model consists of an audio encoder and a Gated PixelCNN
[45]. The audio encoder, which has the same structure as the VQ-VAE encoder,
takes MFCC feature as input. Then we concatenate the output of the audio
encoder and VQ-VAEs encoders and feed it to the Gated PixelCNN. The Gated
PixelCNN has 15 gated convolution layers conditioned on identity, in which the
convolution kernel is masked to make sure the model cannot read future
information. We adopt Adam with $\beta_{1}=0.9$, $\beta_{2}=0.999$ and a
learning rate of 0.0001 as the optimizer. The autoregressive model is trained
with a batchsize of 128 and a sequence length of 88 frames for 100 epochs.

##  Appendix C Application

One application of our speech-to-motion generation is to create the photo-
realistic neural avatars through neural renderers such as SMPLpix [39]. Given
the mesh vertices provided by TalkSHOW and their colors, we first project them
onto the image plane. Then, with the projected mesh vertices, SMPLpix allows
us to efficiently synthesise photo-realistic images of humans. As TalkSHOW can
produce continuous yet diverse motions, integrating SMPLpix with our motion
generation framework enables us generate human avatars under different poses
(see Fig. 10), leading to end-to-end photo-realistic video generation.

Figure 10:  The application with SMPLpix to create photo-realistic neural
avatars. Top row (input): the mesh vertices provided by TalkSHOW and their
colors projected onto the image plane, bottom row: rendered output.

##  Appendix D Discussions

#### Reconstruction.

SHOW is based on SMPLify-X whose supervision signal is obtained from 2D
keypoint reprojection. Thus, it is sensitive to severe hand shape deformation
and heavy occlusion. A future direction would be to leverage advanced hand
model with rich shape and pose space. Besides, SHOW can only handle static
camera cases currently. In the future, we plan to extend it to moving cameras.

#### Audio2motion.

While we have demonstrated that TalkSHOW can generate realistic, coherent, and
diverse holistic body motion with facial expression, body and hand motions, it
is subject to a limitation which can be addressed in future. For the face
generator, we mainly focus on the face motion (e.g. lip motion) and might not
handle the very complex facial motions such as eye blinks. In the future, we
plan to extend to model this sort of part.

##  Appendix E Risks and Potential Misuse

This work is intended for studying the translation from human speech to
holistic body motion, helping building virtual agents to behave realistically
and interact with listeners meaningfully. Since our techniques can generate a
realistic and diverse 3D talking humans from audio, there is a risk that such
technique could be potentially misused for fake video generation. For
instance, a fake speech could be used to construct highly realistic 3D
holistic body motion while it never happened. Thus, we should use such
technology responsibly and carefully. We hope to raise the public’s awareness
about a safe use of such technology.