# FaceXHuBERT: Text-less Speech-driven E(X)pressive 3D Facial Animation
Synthesis Using Self-Supervised Speech Representation Learning

Kazi Injamamul Haque
Utrecht University
<EMAIL_ADDRESS>Zerrin Yumak
Utrecht University
<EMAIL_ADDRESS>
###### Abstract

This paper presents FaceXHuBERT, a text-less speech-driven 3D facial animation
generation method that allows to capture personalized and subtle cues in
speech (e.g. identity, emotion and hesitation). It is also very robust to
background noise and can handle audio recorded in a variety of situations
(e.g. multiple people speaking). Recent approaches employ end-to-end deep
learning taking into account both audio and text as input to generate facial
animation for the whole face. However, scarcity of publicly available
expressive audio-3D facial animation datasets poses a major bottleneck. The
resulting animations still have issues regarding accurate lip-synching,
expressivity, person-specific information and generalizability. We effectively
employ self-supervised pretrained HuBERT model in the training process that
allows us to incorporate both lexical and non-lexical information in the audio
without using a large lexicon. Additionally, guiding the training with a
binary emotion condition and speaker identity distinguishes the tiniest subtle
facial motion. We carried out extensive objective and subjective evaluation in
comparison to ground-truth and state-of-the-art work. A perceptual user study
demonstrates that our approach produces superior results with respect to the
realism of the animation 78% of the time in comparison to the state-of-the-
art. In addition, our method is 4 times faster eliminating the use of complex
sequential models such as transformers. We strongly recommend watching the
supplementary video before reading the paper. We also provide the
implementation and evaluation codes with a GitHub repository link.
https://github.com/galib360/FaceXHuBERT

##  1 Introduction

Figure 1: FaceXHuBERT: An end-to-end encoder-decoder architecture that encodes
audios using self-supervised pretrained speech model HuBERT and decodes to
vertex displacements using GRU followed by a fully connected linear layer that
produces 3D facial animation as 3D mesh sequences.

Speech-driven 3D facial animation is a growing yet challenging research area
with applications to games, VR/AR and film production. Conversational virtual
humans with social and emotional interaction capabilities are used in a range
of applications such as chatbots for customer service and marketing,
simulations for education and healthcare and remote communication. Facial
expressions are the first point of attention in conversational communication
and humans are very receptive to subtle nuances in facial animation which is
explained by the uncanny valley theory [43].

Typically, facial animation workflows rely on professional technical artists
using blendshape facial animation [39] or performance capture aiming to
mitigate most of the labor intensive work [2, 3, 1]. However, as these
characters take place in more interactive applications, the demand to
automatically generate their behavior on-the-fly increases. Research on facial
animation focuses on 2D talking faces [62, 40, 35, 51], 3D facial animation
constructed from 2D images and videos [27, 41, 21, 65] and 3D speech-driven
facial animation [58, 37, 54, 67, 20, 25, 8]. In this paper, we propose a
novel approach for 3D speech-driven facial animation.

3D speech-driven facial animation is either based on phoneme-based approaches
using procedural algorithms [4, 17] or data-driven approaches using machine
learning [59], motion graphs [15] and deep learning [58, 67]. The former
requires explicit definition of co-articulation rules and requires manual
work. While the latter aims to eliminate that by learning speech-animation
parameters mapping from data, it still relies on intermediary representations
of speech units. Recent approaches on 3D speech animation synthesis
effectively employ end-to-end deep learning models [37, 54, 67, 20, 25]
eliminating the need for intermediary representations. Large speech and
language models [30, 13] open the way towards more realistic speech-driven
facial animation. However, the lack of 3D facial animation data matching audio
and text poses a major bottleneck and current models cannot generalize to
arbitrary speech input. Most recently, Fan et al. [25] suggested a self-
supervised speech representation learning method using transformers to
mitigate these issues. However, the method cannot handle expressive
animations, the results are still far from capturing personalized and subtle
cues in speech and the method is computationally very expensive to train.

Our work improves on this work and proposes FaceXHuBERT, a text-less speech-
driven expressive 3D facial animation generation method using self-supervised
speech representation learning. In our proposed encoder-decoder network (see
Fig. 1), we effectively employ self-supervised pretrained HuBERT model to
incorporate and encode both lexical and non-lexical information without using
a large lexicon and speech-3D data pairs allowing it to generalize to any
speech input. Our method is simple and efficient eliminating the use of
complex sequential models such as transformers and instead uses a decoder
based on Gated Recurrent Units. The main contributions of our work are:

  * •

An efficient text-less speech-driven expressive 3D facial animation method
using self-supervised speech representation learning. FaceXHuBERT produces
expressive and realistic animations in an efficient way using a HuBERT-based
encoder and GRU-based decoder without the use of a large lexicon and using
only audio input. Additionally, guiding the training with an emotion condition
and speaker identity distinguishes the tiniest subtle facial motions. The
results show that our method produces more realistic results in a more
efficient manner (i.e. 4 times faster in comparison to a vanilla transformer
and almost 3 times faster in comparison to state-of-the-art [25]).

  * •

Proof of self-supervised pretrained speech model HuBERT [30] for the
downstream task of expressive 3D facial animation synthesis. Our method
produces accurate lip-sync as well as allows to capture personalized and
subtle cues in speech (e.g. identity, emotion and hesitation). It is also very
robust to background noise and can handle audio recorded in a variety of
situations (e.g. multiple people speaking, background noise, laughter, lip-
smacking).

  * •

Extensive objective and subjective analyses. We compared our method to state-
of-the-art and ground-truth as well as made comparisons between various seq-
to-seq neural network architectures using 3D vertex error as an objective
metric. Subjective analysis includes qualitative generalizability analysis in
terms of different languages, text-to-speech, noise, low-quality audio input
and single subject training. We also conducted several perceptual user
studies. Our results demonstrate that our approach produces superior results
with respect to the realism of the animation 78% of the time in comparison to
the state-of-the-art [25].

##  2 Related Work

Extensive research has been conducted in the domain of automatic facial
expression analysis and synthesis in the 2D pixel domain for the purpose of
detecting expressions [60, 56, 34], for generating audio-driven talking faces
[29, 66, 62, 40, 35] or for video-based facial re-enactment/face swapping [57,
48, 51].

The approaches for 3D facial animation synthesis can be classified into video-
driven and audio-driven facial animation. While the former focuses on
transferring facial animation from 2D videos to 3D faces, the latter maps
speech (audio and text) to 3D facial animation parameters. Earlier works on
video-driven facial animation focused on optimization-based 3D facial
performance capture [11, 14, 32], while recent works use deep learning [55,
27, 41, 21, 44]. For an extensive survey on 3D face reconstruction, tracking
and morphable models, we refer to [42, 68, 23]. Some methods use retargeting
algorithms to convert facial expressions from one 3D mesh to the other [53,
16] or from 2D images to 3D faces [64, 65]. Finally, there is a group of
research focusing on physics-based animation of faces [33, 10]. In our work,
we focus on 3D speech-driven facial animation using deep learning.

#### 3D Speech-Driven Facial Animation

3D speech-driven facial animation typically uses phoneme-based procedural
approaches [4, 17]. Although these methods come with the advantage of
animation control and easy integration to artist-friendly pipelines, they are
not fully automatic and require defining explicit rules for co-articulation.
Another line of research uses machine learning [59] or graph-based approaches
[15] to learn speech-animation mappings from data. These methods rely on
blending between speech units and cannot capture the complexity of the
dynamics of visual speech [58]. They rather focus on the lower face and are
not robust to emotion and style variations. Recent approaches on 3D speech
animation synthesis effectively employ deep learning models [58, 37, 54, 67,
20, 25, 8]. Taylor _et al_. [58] proposes a sliding window approach instead of
an RNN focusing on capturing neighborhoods of context and coarticulation
effects. VisemeNet [67] builds upon the viseme-based JALI[4] model and
combines this with an LSTM-based neural network. However, these two methods
[58, 67] still rely on intermediary representations of phonemes and they focus
on the mouth movement. Most previous works do not include automatic tongue
animation except [8]. Collecting large-scale datasets using professional
performance capture workflows is expensive and time consuming but the
resulting faces are highly realistic. To elevate this disadvantage, some
methods use 3D automatic face reconstruction methods from in-the-wild videos,
which are especially useful in situations where professional performance
capture systems are not available, e.g. dyadic speech-driven facial animation
[36, 47]. However, these methods are prone to 3D reconstruction errors and
cannot generate results that are as realistic as the former.

Closest to our work are Karras _et al_. [37], Cudeiro _et al_. [20], Richard
_et al_. [54], Fan _et al_. [24, 25]. Karras _et al_. [37] proposes an end-to-
end convolutional neural network that learns a mapping from input waveforms to
the 3D vertex coordinates of a face model. They aim to resolve the ambiguity
in mapping between audio and face by introducing an additional emotion
component to the network, which is learned from data. However, the method is
not trained on multiple speakers and cannot handle identity variations and
requires a longer-term audio context to infer the emotional state. Instead,
Cudeiro _et al_. [20] presents the audio-driven facial animation method VOCA
that generalizes to new speakers using a training dataset with 12 subjects
eliminating the need for retargeting. However, VOCA fails to realistically
synthesize upper face motion and does not include emotional variations.
Similar to VOCA, Richard _et al_. [54] aims for audio-driven animation that
can capture variations in multiple speakers including a much larger dataset of
250 subjects. They address the problem of lack of upper face motions using a
categorical latent space that disentangles audio-correlated and audio-
uncorrelated information based on a cross-modality loss. Fan _et al_. [24]
proposes an audio and text-driven facial animation method that incorporates
the large language model GPT-2[52] to encode the textual information. The
authors found that combined audio and text input yielded better results than
audio-only or text-only model. However, the results still have problems
regarding accurate lip-sync. Most closely related to our work is
FaceFormer[25] which uses a self-supervised pretrained speech model that
addresses the scarcity of available data in existing audio-visual datasets.
The model produces superior results in comparison to Cudeiro _et al_. [20] and
Richard _et al_. [54] using a modified version of transformers to handle
longer sequences of data. However, none of these methods can handle arbitrary
variations in speech input while producing accurate lower and upper facial
animation for multiple identities and emotions.

##  3 Problem Formulation

We formalize the task of audio-driven 3D facial animation as a generic
sequence modeling (seq2seq) problem in which the input sequence is a raw audio
waveform whereas the output sequence is a 3D face mesh sequence (i.e. 4D
scan). Hence, the problem can be formalized as follows:

Given audio $A$ and ground-truth 3D mesh sequence
Y$=(y_{1},y_{2},y_{3},...,y_{T_{Y}})$, $T_{Y}$ is the total number of
available visual frames or 3D scanned frames in the sequence. Therefore, one
sequence of Y is a $(T_{Y},V)$ dimensional matrix where $V$ denotes the number
of 3D vertices in the mesh topology. On the input side, since audio stream $A$
is a continuous data stream, with the help of an encoder, we encode the
continuous audio into a discrete representation
$X=(x_{1},x_{2},x_{3},...,x_{T_{X}})$ where $X$ is a $(T_{X},B)$ dimensional
matrix and $T_{X}$ and $B$ are the discreet time-steps and the encoded
representation respectively.

The goal is to train an end-to-end architecture to learn the mapping between
$A$ (together with additional conditions) and Y to generate
$\hat{Y}=(\hat{y}_{1},\hat{y}_{2},\hat{y}_{3},...,\hat{y}_{T_{y}})$ so that
the $\hat{Y}$ best approximates Y.

##  4 Proposed Approach

We present FaceXHuBERT, an end-to-end encoder-decoder neural network
architecture. Our model uses the pretrained HuBERT speech model as the audio
encoder while for the decoder, we use Gated Recurrent Unit (GRU) [18]. Fig. 2
shows the overall architecture of our proposed approach. The encoder encodes
the continuous audio information into discreet time-step representations and
adjusts the representations so that the time-steps match with that of the face
scan data. The decoder incorporates the emotion and subject identity
information, consists of a 2-layered GRU with hidden size 256 followed by a
fully connected linear layer. The decoder regresses vertex displacements and
adds to the subject’s template mesh to generate the predicted mesh sequence.
Algorithm 1 in the supplementary material depicts the overall steps of the
proposed network. In the next two subsections, we describe the details of the
FaceXHuBERT Encoder and Decoder.

Figure 2: FaceXHuBERT: The encoder encodes the audio waveform $A$ and produces
discreet frame level embedding. The Input Representation Adjustment module in
the encoder adjusts the encoded information with the output 4D scan data and
produces $X$ such that $T_{X}=T_{Y}=T$. The Decoder takes in $X$ and with the
help of the 2-layered 256 hidden sized GRU, produces the hidden representation
$H$. Additional conditions such as the speaker identity and emotion label are
embedded and multiplied with $H$ before the hidden representation is decoded
into vertex displacement values and added to the corresponding subject’s
neutral face to produce the animation output $\hat{Y}$. The loss function is
computed based on $\hat{Y}$ and ground-truth (GT), Y.

###  4.1 FaceXHuBERT Encoder

Our proposed method effectively adopts the state-of-the-art self-supervised
pretrained speech model HuBERT in the encoder for the downstream task of 3D
facial animation generation. Since it is able to learn and produce high
quality discreet hidden representations of continuous audio streams combining
both acoustic and language information, the authors of HuBERT recommend to
consider using HuBERT pretrained representations for a variety of downstream
tasks [30]. HuBERT architecture introduces a BERT-like [22] masked language
modelling encoder for the transformer layers. It introduces a simple cross-
entropy loss for predicting masked units in contrast to its predecessor
Wave2Vec 2.0’s [9] complex contrastive loss. In addition, unlike Wave2Vec 2.0,
HuBERT is trained with multiple iterations. During the first iteration, HuBERT
uses unsupervised simple k-means clustering for acoustic unit discovery to
facilitate the self-supervised masked language modeling learning that takes
place in the second iteration. In the second iteration, the training is done
on the discovered discrete hidden units with a predictive loss on the masked
regions only, forcing the model to learn a combined acoustic and language
model using a BERT-like encoder, hence the name H(idden)-u(nit)-BERT. The
model is trained on 960 hours of unlabeled speech data [49] which contains
English recordings of copyright-free audiobooks by volunteers from the
internet. The authors claim that this approach is the first big step towards
text-less Natural Language Processing (NLP). For detailed explanation of how
different HuBERT models were trained and for comparison to previous work, we
refer to the original paper [30].

The FaceXHuBERT Encoder is composed of a CNN encoder that discretizes the
continuous audio data into 512 dimensional representations. Feature Projection
layer projects the 512 dimensional representation into 768 dimensional
representation, a positional convolution embedding layer and 12 transformer
layers to capture the contextual information in the sequence. In our approach,
we adopt the “base” HuBERT model with 95M parameters which produces 768
dimensional embedding at the last hidden state. We initialize the pretrained
weights and freeze the model parameters including the CNN feature encoder
layer, feature projection layer and the first two transformer layers. The last
ten transformer layers are kept unfrozen and remain trainable. HuBERT
generates a feature sequence in 20ms windows (i.e. 50 fps). Therefore, in our
architecture, we encode a one second of audio into 50 frames with 768
dimensional embeddings. For example, for a training data with 4 seconds of
audio stream, the output from the encoder will be $(4\times 50,768)=(200,768)$
dimensional matrix.

#### Input Representation Adjustment

This module adjusts the input representation and output representations and do
not contain any trainable parameters. This is devised to ensure the one-to-one
frame level relationship between decoder input $X$ and output $Y$ such that
$T_{X}=T_{Y}=T$. This function is generically devised in such a way that it
can handle any input-output frequency pair. More details on this can be found
in the supplementary material.

###  4.2 FaceXHuBERT Decoder

For the FaceXHuBERT Decoder, we use a Gated Recurrent Unit (GRU) instead of a
complex transformer model. Our extensive analysis shows that, combined with
the HuBERT encoder, GRU-based decoder produces realistic results in a more
efficient manner. Our GRU-based decoder consists of 2 layers with hidden unit
size of 256, followed by one fully connected linear layer that maps the last
hidden state to vertex displacement values of the 3D vertices of the face. It
represents the faces in terms of their displacement values with respect to the
neutral template vertices of a given subject. Between the GRU and the fully
connected layer, we add the additional conditions and fuse them with the
hidden state representation with element-wise multiplication. The additional
conditions are (i) subject identity and (ii) emotion (neutral or expressive in
our experiments). We defined the training subjects and the emotion label as
one-hot vectors and linearly embed them with two separate 256 dimensional
vectors to facilitate the element-wise multiplications.

Eq. 1, Eq. 2, Eq. 3 and Eq. 4 show the core computations of our decoder’s
forward propagation.

$\Gamma_{r}=\sigma(W_{r}^{l}[a_{t-1}^{l},a_{t}^{l}]+b_{r}^{l})$ (1)
$\tilde{a}_{t}^{l}=tanh(W_{a}^{l}[\Gamma_{r}\odot
a_{t-1}^{l},a_{t}^{l}]+b_{a}^{l})$ (2)
$\Gamma_{u}=\sigma(W_{u}^{l}[a_{t-1}^{l},a_{t}^{l}]+b_{u}^{l})$ (3)
$a_{t}^{l}=\Gamma_{u}\odot\tilde{a}_{t}^{l}+(1-\Gamma_{u})\odot a_{t-1}^{l}$
(4)

The subscript $t$ denotes the frame number or time-step in the sequence
whereas the superscript $l$ denotes the hidden layer where $l=[1,2]$ (i.e. two
hidden layers). When $l=1$, the activation values $a_{t}^{l}$ (not
$a_{t-1}^{l}$) in Eq. 1, Eq. 2 and Eq. 3 take the input value $x_{t}$, hence,
$a_{t}^{1}=x_{t}$. When $l=2$, the activation values $a_{t}^{2}=h_{t}$, each
being 256 dimensional hidden unit. The second GRU layer produces the hidden
units $H=[h_{1},h_{2},h_{3},...,h_{T}]$ for a sequence with $T$ frames.
Furthermore, we initialize $a_{0}=\vec{0}$ to start the training.

Eq. 5, Eq. 6, Eq. 7 and Eq. 8 show how the subject identity and emotion
conditions are incorporated into the network.

$S=W_{S}\cdot[SubjectOneHot]+b_{S}$ (5) $E=W_{E}\cdot[EmotionOneHot]+b_{E}$
(6) $\tilde{H}=H\odot S\odot E$ (7)
$\hat{Y}=(W_{\hat{Y}}\cdot\tilde{H}+b_{\hat{Y}})\oplus[NeutralFace]$ (8)

The subject label and the emotion label are represented as one-hot vectors
that we linearly embed to 256 dimensional $S$ and $E$ vectors using Eq. 5 and
Eq. 6 respectively. In Eq. 7, the hidden representation $H$ with dimensions
$(T,256)$ is multiplied in an element-wise manner with both the embedding
vectors of the given subject, $S$ (i.e. training subjects) and the style of
the given sequence, $E$ (i.e. neutral or expressive). Finally, in Eq. 8, the
fully connected linear layer decodes the hidden representation into $(T,V)$
dimensional vertex displacement values which is then added to the respective
subject’s neutral face vertex values, where $[NeutralFace]$ is a $(1,V)$
dimensional data.

(a) Neutrally generated animation

(b) Expressively generated animation

(c) Difference between (a) & (b)

(d) Neutrally generated animation

(e) Expressively generated animation

(f) Difference between (d) & (e)

Figure 3: Effect of the emotion label during inference: our approach can
generate facial animations that are style controllable by a binary emotion
label. Given two in-the-wild audio signal examples, Figs. 3(a), 3(b) and 3(c)
correspond to the male example whereas Figs. 3(f), 3(e) and 3(f) correspond to
the female example. Figs. 3(a) and 3(d) are generated neutrally whereas Figs.
3(b) and 3(e) are generated expressively. Figs. 3(c) and 3(f) show the
colorized differences based on per-vertex distances between neutrally and
expressively generated face meshes where extreme red depicts 100% of the
computed distance and extreme blue depicts 0% of the computed distance. It is
evident that the emotion signal effects the facial regions that are
uncorrelated with speech.

##  5 Experimental Setup

###  5.1 Dataset and Pre-processing

For our experiments, we used the BIWI[26] dataset which is the only dataset
that is available publicly upon request with both emotion and identity labels.
VOCASET[45] and Multiface[63] datasets allow incorporating identities but not
emotions. BIWI contains synchronized audio-4D scan pairs of 14 human subjects
uttering 40 phonetically balanced English sentences twice: first neutrally,
second with emotional expressions. Therefore, the dataset contains $(14\times
40\times 2)=1120$ sequences of audio-4D scan pairs. In reality, due to some
missing sequences from 4D scans, there are in total 1088 audio-4D pairs. The
sequences in the dataset are 4.39 seconds in duration on average including
both neutral and emotional sequences. Furthermore, on average, neutral
sentences are 4.28 seconds long whereas the emotional sentences are 8.90
seconds long. To ensure efficient training, we pre-process the data by scaling
the 3D vertices to have a uniform range of values for all three coordinates
across the dataset (e.g. [-0.5,0.5]). The data is captured at 25fps with 23370
3D vertices in the mesh topology. The pre-processed dataset is then split in
90% train, 5% validation and 5% test sets. More details on dataset pre-
processing can be found in the supplementary material.

###  5.2 Model Experiments

We experimented with different seq-to-seq neural network architectures
including a vanilla RNN, LSTM, GRU and Transformer as FaceXHuBERT Decoder. In
this section, we will briefly discuss about the model experiments and their
results.

HuBERT-RNN model is devised by using RNN with the 2 hidden layers and with
hidden size 256. The resulting 3D facial animations are acceptable in terms of
visual quality and lip-sync. However, it does not generalize well for longer
in-the-wild audios. The resultant animations are not smooth, noisy and show
visual artefacts.

HuBERT-LSTM model uses LSTM with 2 hidden layers and with hidden size 256. The
resulting animations start to move in the very beginning but immediately
becomes static and maintains the stagnant pose throughout the remaining frames
of the sequence. As mentioned in [28], LSTM dominates for short term motion
generation yet soon converges to a mean pose.

HuBERT-Transformer uses a vanilla transformer as decoder. We trained using
both teacher forcing and autoregressive approaches. The former yields static
facial pose throughout the animation sequence while the latter produces
meaningful animations. However, due to the network complexity, the training
time is 4 times slower when compared to our proposed approach. Additionally,
the transformer based approach is constrained by a pre-defined maximum length
of input sequence.

HuBERT-GRU network architecture of this model is our proposed model described
in Sec. 4. Using GRU in the vertex decoder yields audio coherent animations
for both test-set sequences as well as arbitrarily long in-the-wild audio
sequences recorded in variety of situations. This model produces the most
realistic and expressive results. Extensive evaluations can be found in the
following section.

Training Details and Tools All the models in our experiments were trained for
100 epochs on an HP ZBook Fury G7 laptop with an Intel Core i7-10850H 2.7 GHz
(12 cores) CPU, 32GB RAM and Nvidia Quadro RTX 3000 6GB VRAM. We optimized on
Huber Loss function [31] and used Adam optimizer [38] during the training. The
dropout value of the recurrent units was set to $0.3$. We used PyTorch [50]
for implementing the models. Meshlab [19] and PyMeshLab [46] were used to
compute mean face vertex error for quantitative evaluation. Trimesh [7]
together with OpenCV [12] and ffmpeg [61] were used for rendering and
visualizing. The entire codebase for training, evaluating and visualizing can
be found in the supplementary material.

##  6 Evaluation

Evaluation of the 3D facial animation generation is challenging and there is
no unanimously accepted single objective metric in the literature. Therefore,
in addition to extensive quantitative evaluations using mean face vertex error
as an objective metric, we also conducted several qualitative evaluations in
terms of visual assessments. Furthermore, since facial animation is
perceptual, we conducted user studies to compare our approach with ground-
truth as well as the state-of-the-art. Because FaceFormer proved to be
superior than the other state-of-the-art works, we compared our approach only
with FaceFormer[25].

Model | Mean Face | Training
---|---|---
Type | Vertex Error5 | Time (h)
HuBERT-RNN | $5.06$ | $\approx 5.33$
HuBERT-LSTM | $5.98$ | $\approx 5.55$
HuBERT-Transformer1 | $8.74$ | $\approx 5.27$
HuBERT-Transformer2 | $5.13$ | $\approx 25.00$
FaceFormer3 | $5.95$ | $\approx 6.38$
FaceXHuBERT3 | $\mathbf{5.45}$ | $\mathbf{\approx 1.52}$
FaceFormer4 | $6.36$ | $\approx 16.11$
FaceXHuBERT4 | $\mathbf{4.80}$ | $\mathbf{\approx 5.10}$
FaceXHuBERT- w/o emo | $4.93$ | $\approx 5.69$
HuBERT-FaceFormer | $4.96$ | $\approx 19.58$
1 Teacher-forcing scheme. Produces static animations.
2 Autoregressive scheme.
3 Model trained on only 6 subjects for fair comparison.
4 Model trained on the whole dataset (i.e. all 14 subjects). FaceFormer
does not produce good results when trained on the whole dataset.
5 In millimeters (mm) when the face resides in a 1$m^{3}$ bounding box.
Table 1: Objective evaluation results of the experiments and trained models.
Our approach not only produces the minimum Mean Face Vertex Error on test-set
sequences but also reduces the training time significantly. FaceXHuBERT is
more than 4 times faster than transformer based architecture and almost 3
times faster than the state-of-the-art.

###  6.1 Quantitative and Qualitative Evaluation

We measure and compare our proposed methodology quantitatively based on mean
face vertex error. For each test-set sequence we take the vertex distance
values of the predicted data with respect to the ground-truth reference for
all the frames. We take the arithmetic mean of the calculated differences
across the frames for all the test sequences and compute the average value to
get the “Mean Face Vertex Error”. Tab. 1 reports this error value together
with corresponding training time for each of the models we presented in Sec.
5.2. In addition, we also compared our approach with the state-of-the-art
FaceFormer model. It is to be noted that FaceFormer fails to generate
meaningful animations when trained on the whole BIWI dataset (i.e. all 14
subjects with all available sequences). Therefore, for a fair comparison, in
addition to training on the whole dataset, we trained FaceFormer and
FaceXHuBERT with the FaceFormer’s suggested training including only 6 training
subjects. The proposed FaceXHuBERT yields the minimum mean face vertex error
while reducing the network complexity and training time significantly.
Additionally, FaceFormer trains well on the whole dataset only when we replace
their Wav2Vec2.0-based audio encoder with FaceXHuBERT encoder (HuBERT-
FaceFormer in Tab. 1). However, the mean face vertex error is higher and the
training is almost 4 times slower in comparison to ours. Besides, we trained
our approach without incorporating the emotion label (FaceXHuBERT-w/o emo in
Tab. 1) similar to how FaceFormer is trained.

The quality of animations generated by FaceXHuBERT has been studied carefully
to understand the generalizability capabilities of the model in terms of
various aspects such as- smoothness of the animation, coherence with respect
to speech, different background noises in input audio, multiple speakers,
different languages, different subjects, use of TTS (text2speech) instead of
real audio and limited training data. The proposed approach is robust in terms
of all the above-mentioned aspects. Additionally, by training the network
without the binary emotion label (FaceXHuBERT- w/o emo) in the decoder, we
lose the expressive style control capability during inference. In this case,
facial expressiveness of the generated animations rely solely on the audio
signal and are qualitatively slightly less expressive than the proposed
model’s predictions. Yet it still captures and distinguishes between neutral
and emotional aspects in speech signals. Furthermore, we qualitatively
compared our approach to the FaceFormer (see Fig. 4) and found that
FaceXHuBERT generates more expressive animations that are closer to the
ground-truth. Unlike our approach, FaceFormer is not robust to variety of
noises overlapping with the audio signal and produces visual artefacts. We
recommend watching the supplementary video for visual quality judgement.

(a) GT

(b) FaceFormer

(c) Ours

(d) Difference

Figure 4: Qualitative comparison of expressiveness between FaceFormer and
FaceXHuBERT (Ours). Given an audio sequence from the test-set, Fig. 4(a) is
the ground-truth whereas Figs. 4(b) and 4(c) are the corresponding frames from
animations generated using FaceFormer and FaceXHuBERT respectively. Fig. 4(d)
shows the colorized per vertex distance computed between Figs. 4(b) and 4(c).
Animation generated by our approach is more expressive because the upper face
region is more responsive and expressive to the emotionally expressive audio
sequences than the one of FaceFormer.

###  6.2 Perceptual Evaluation

> “A work of art doesn’t exist outside the perception of the audience.”
>
> Abbas Kiarostami

In order to demonstrate the realism of facial animation produced by our
proposed approach and to compare to ground-truth and current state-of-the-art,
we conducted perceptual user studies. The user studies were hosted on
Qualtrics [6] and carried out using Prolific [5], ensuring that the
participants get compensated appropriately. We conducted three separate user
studies where the users selected their preferences based on the realism of the
rendered 3D facial animation videos. In Experiment I, the users indicated
their preference between Ground-truth data vs Ours. In Experiment II, the
users were shown side-by-side comparisons of FaceFormer and FaceXHuBERT
generated with the same audio input. Lastly, in Experiment III the users
depicted their preference between neutrally and expressively generated
animations both using our approach. Fig. 3 quantitatively and visually shows
the difference between neutrally and expressively generated animations. The
facial regions (i.e. upper face, eye region, cheeks) that are expected to be
effected by the emotion condition deform differently than in the corresponding
neutral animation. To solidify our argument, the third experiment was
conducted to prove that the expressiveness in facial animation is actually
perceived by users.

User Preference | Realism
---|---
I. Ours vs. Ground-Truth | $25.45\pm 13.20$
II. Ours vs. FaceFormer | $77.73\pm 17.59$
III. Ours-Emotional vs. Ours-Neutral | $66.35\pm 5.66$
Table 2: User study results: The Realism score depicts the average percentage
of participants that preferred the left item to the right item in the
corresponding row in terms of expressiveness.

For all three experiments, the participants where shown a series of A vs. B
video pairs and were asked to choose one between the two animations that looks
more realistic and more expressive. In order to ensure good quality data, we
included random attention check questions. The attention check questions
consist of a video pair in which one of the videos is totally out-of-sync with
the accompanied audio.

The results of the three experiments are reported in Tab. 2. In Experiment I,
on average, 25.45% of the participants preferred animation generated by our
model over the ground-truth animation. This is expected as the generated
animations do not model some of the facial motion such as the eye blinks that
are present in the ground-truth. Furthermore, the ground-truth has more
variations across the whole face. In Experiment II, on an average, 77.73% of
the participants preferred animations generated by our model over the
animations generated by FaceFormer. In Experiment III, on average, 66.35% of
the participants preferred animations generated by our model with expressive
signal over the animations generated without the expressive signal.

##  7 Ablation Study

#### Ablation on FaceXHuBERT Encoder

We conducted extensive experiments to understand and optimize the effect of
HuBERT in our encoder. This ablation study is devised by freezing the model’s
pretrained weights at various layers starting from no-freezing (i.e. all the
layer parameters are trainable) to all weights frozen (i.e. all the layers are
frozen to their pretrained weights). The results of the ablation study is
reported in Tab. 3. In light of the encoder structure as described in Sec.
4.1, the network configurations for this ablation study are as follows- (i) no
layers are frozen, (ii) CNN encoder is frozen, (iii) CNN encoder and feature
projection layers are frozen. For models (iv) to (viii), in addition to
freezing the CNN encoder and feature projection layer, we incrementally freeze
two transformer layers. For model (ix), only the last transformer layer is
kept trainable whereas for (x), the entire pretrained model is frozen during
training. Although all the above mentioned configurations yield qualitatively
coherent animations, we found that (iv) produces the least mean face vertex
error. Furthermore, as we move from model (iv) to (x), the animations become
more stiffer and less expressive.

#### Ablation on FaceXHuBERT Decoder

We experimented with different configurations of the GRU structure in terms of
number of hidden layers and hidden unit size to optimize for the mean vertex
error. Tab. 4 shows the results of the FaceXHuBERT decoder ablation study. The
first column represents the number of hidden layers (e.g. 2L) and hidden size
(e.g. 128) in the GRU structure. All the configurations mentioned in the table
produce qualitatively coherent animations but training the proposed
configuration results in producing the least mean face vertex error, ensuring
that the predictions are more closer to the ground-truth, hence more realistic
and expressive.

##  8 Discussion and Limitations

Using a self-supervised pretrained speech model such as HuBERT produces
significant improvements for the 3D speech-driven facial animation task. It
clearly shows the importance of the encoder model, while using a simple
decoder component. It does not only have the ability to disambiguate speech
uncorrelated factors for facial animation, but also addresses the scarcity of
synchronized audio-visual datasets by incorporating pretrained speech
representations based on a large speech model. We assume that it can be
adopted to solve similar downstream tasks such as audio-driven gesture
synthesis. Additionally, we showed that guiding the training with an emotion
label captures the facial deformations uncorrelated with and correlated with
emotion context.

Due to the limitation of the BIWI dataset, we could only guide the learning in
a binary manner (i.e. neutral and expressive). However, we assume that with a
balanced dataset containing specific emotion categories in the data, we will
be able to learn and generate audio-driven facial animations for respective
emotion categories. Furthermore, our approach is limited to offline animation
generation. In the future, we plan to extend our work to be real-time
friendly. Our proposed approach works on 3D meshes and still needs to be
mapped to a rigged character. Additionally, since the face scans of the
dataset do not contain eyes and tongue, our method could not take into account
animations of some face parts such as eye gaze and tongue.

Model | Mean Face | No. of | Training
---|---|---|---
variant | Vertex Error (mm) | parameters | Time (h)
(i) | $5.17$ | $114166622$ | $\approx 8.19$
(ii) |  $4.8$6 | $109966174$ | $\approx 5.75$
(iii) | $5.00$ | $109571166$ | $\approx 6.38$
(iv) | $4.80$ | $95395422$ | $\approx 5.10$
(v) | $4.84$ | $81219678$ | $\approx 5.83$
(vi) | $4.85$ | $67043934$ | $\approx 5.50$
(vii) | $4.89$ | $52868190$ | $\approx 5.27$
(viii) | $5.02$ | $38692446$ | $\approx 5.13$
(ix) | $4.95$ | $31604574$ | $\approx 4.86$
(x) | $5.24$ | $19795678$ | $\approx 3.88$
Table 3: FaceXHuBERT Encoder Ablation Study Results. Model (iv) depicts the proposed approach. Model | Mean Face | No. of | Training
---|---|---|---
variant | Vertex Error (mm) | parameters | Time (h)
2L-256 | $\mathbf{4.80}$ | $95395422$ | $\approx 5.10$
1L-256 | $4.93$ | $95000670$ | $\approx 5.55$
2L-128 | $4.92$ | $85385566$ | $\approx 5.33$
2L-64 | $5.00$ | $80491230$ | $\approx 5.27$
2L-32 | $5.19$ | $78071710$ | $\approx 4.30$
Table 4: FaceXHuBERT Decoder Ablation Study Results. Model 2L-256 depicts the
proposed approach.

##  9 Conclusion

In this paper, we presented FaceXHuBERT, an efficient text-less speech-driven
expressive 3D facial animation method using self-supervised speech
representation learning. At the core of our model is the pretrained HuBERT-
based encoder combined with an efficient GRU-based decoder instead of a
complex model based on transformers. Our method can produce accurate lip-sync
and expressive facial animations for arbitrary audio input without the need of
long training times and large dataset. Our method does not only produce
accurate lip-sync but also captures personalized and subtle cues in speech
(e.g. identity, emotion and hesitation). It is also very robust to background
noise and can handle audio recorded in a variety of situations (e.g. multiple
people speaking, background noise, laughter, lip-smacking). Our extensive
objective and subjective analyses show that FaceXHuBERT outperforms the state-
of-the-art. We hope that our approach will be a stepping stone towards text-
less speech-driven expressive 3D facial animation.

Ethical Consideration Models trained on face scans can easily be used for
generating synthetic content that can jeopardize humans and their privacy. We
must act responsibly by considering the aspects pertaining to privacy and
ethics.

Acknowledgement We would like to thank the authors of FaceFormer for making
their code available and to ETH Zurich CVL for providing us access to the Biwi
3D Audiovisual Corpus. We express our gratitude to our colleagues for
providing different language audios and for the feedback.

## References

  * [1] Di4d.  https://di4d.com/.
  * [2] Dynamixyz.  https://www.dynamixyz.com.
  * [3] Faceware.  https://facewaretech.com/.
  * [4] Jali.  https://jaliresearch.com/.
  * [5] Prolific.  https://www.prolific.co.
  * [6] Qualtrics.  https://www.qualtrics.com.
  * [7] Trimesh.  https://trimsh.org/.
  * [8] Mónica Villanueva Aylagas, Héctor Anadon Leon, Mattias Teye, and Konrad Tollmar.  Voice2face: Audio-driven facial and tongue rig animations with cvaes.  In EUROGRAPHICS SYMPOSIUM ON COMPUTER ANIMATION (SCA 2022, 2022\.
  * [9] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  CoRR, abs/2006.11477, 2020.
  * [10] V. Barrielle and N. Stoiber.  Realtime performance-driven physical simulation for facial animation.  Computer Graphics Forum, 38(1):151–166, June 2018.
  * [11] Sofien Bouaziz, Yangang Wang, and Mark Pauly.  Online modeling for realtime facial animation.  ACM Trans. Graph., 32(4), jul 2013.
  * [12] G. Bradski.  The OpenCV Library.  Dr. Dobb’s Journal of Software Tools, 2000.
  * [13] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  * [14] Chen Cao, Qiming Hou, and Kun Zhou.  Displaced dynamic expression regression for real-time facial tracking and animation.  ACM Trans. Graph., 33(4), jul 2014.
  * [15] Yong Cao, Wen C. Tien, Petros Faloutsos, and Frédéric Pighin.  Expressive speech-driven facial animation.  ACM Trans. Graph., 24(4):1283–1302, oct 2005.
  * [16] Prashanth Chandran, Loïc Ciccone, Markus Gross, and Derek Bradley.  Local anatomically-constrained facial performance retargeting.  ACM Trans. Graph., 41(4), jul 2022.
  * [17] Constantinos Charalambous, Zerrin Yumak, and A.F. van der Stappen.  Audio‐driven emotional speech animation for interactive virtual characters.  Computer Animation and Virtual Worlds, 30, 2019.
  * [18] Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio.  On the properties of neural machine translation: Encoder-decoder approaches, 2014.
  * [19] Paolo Cignoni, Marco Callieri, Massimiliano Corsini, Matteo Dellepiane, Fabio Ganovelli, and Guido Ranzuglia.  MeshLab: an Open-Source Mesh Processing Tool.  In Vittorio Scarano, Rosario De Chiara, and Ugo Erra, editors, Eurographics Italian Chapter Conference. The Eurographics Association, 2008.
  * [20] Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael Black.  Capture, learning, and synthesis of 3D speaking styles.  In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 10101–10111, 2019.
  * [21] Radek Danecek, Michael J. Black, and Timo Bolkart.  EMOCA: Emotion driven monocular face capture and animation.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 20311–20322, 2022.
  * [22] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2018.
  * [23] Bernhard Egger, William A. P. Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, Christian Theobalt, Volker Blanz, and Thomas Vetter.  3d morphable face models—past, present, and future.  ACM Trans. Graph., 39(5), jun 2020.
  * [24] Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura.  Joint audio-text model for expressive speech-driven 3d facial animation, 2021.
  * [25] Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura.  Faceformer: Speech-driven 3d facial animation with transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [26] G. Fanelli, J. Gall, H. Romsdorfer, T. Weise, and L. Van Gool.  A 3-d audio-visual corpus of affective communication.  IEEE Transactions on Multimedia, 12(6):591 – 598, October 2010\.
  * [27] Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart.  Learning an animatable detailed 3D face model from in-the-wild images.  volume 40, 2021.
  * [28] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik.  Recurrent network models for human dynamics.  In 2015 IEEE International Conference on Computer Vision (ICCV), pages 4346–4354, 2015.
  * [29] Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang.  Ad-nerf: Audio driven neural radiance fields for talking head synthesis.  CoRR, abs/2103.11078, 2021.
  * [30] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021.
  * [31] Peter J. Huber.  Robust Estimation of a Location Parameter.  The Annals of Mathematical Statistics, 35(1):73 – 101, 1964.
  * [32] Alexandru Eugen Ichim, Sofien Bouaziz, and Mark Pauly.  Dynamic 3d avatar creation from hand-held video input.  ACM Trans. Graph., 34(4), jul 2015.
  * [33] Alexandru-Eugen Ichim, Petr Kadleček, Ladislav Kavan, and Mark Pauly.  Phace: Physics-based face modeling and animation.  ACM Trans. Graph., 36(4), jul 2017.
  * [34] Geethu Miriam Jacob and Bjorn Stenger.  Facial action unit detection with transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7680–7689, 2021.
  * [35] Xinya Ji, Hang Zhou, Kaisiyuan Wang, Qianyi Wu, Wayne Wu, Feng Xu, and Xun Cao.  Eamm: One-shot emotional talking face via audio-based emotion-aware motion model.  In ACM SIGGRAPH 2022 Conference Proceedings, SIGGRAPH ’22, New York, NY, USA, 2022. Association for Computing Machinery.
  * [36] Patrik Jonell, Taras Kucherenko, Gustav Eje Henter, and Jonas Beskow.  Let’s face it: Probabilistic multi-modal interlocutor-aware generation of facial gestures in dyadic settings.  In International Conference on Intelligent Virtual Agents (IVA ’20). ACM, 2020.
  * [37] Tero Karras, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen.  Audio-driven facial animation by joint end-to-end learning of pose and emotion.  ACM Trans. Graph., 36(4), jul 2017.
  * [38] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2014.  cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
  * [39] J. P. Lewis, Ken Anjyo, Taehyun Rhee, Mengjie Zhang, Fred Pighin, and Zhigang Deng.  Practice and Theory of Blendshape Facial Models.  In Sylvain Lefebvre and Michela Spagnuolo, editors, Eurographics 2014 - State of the Art Reports. The Eurographics Association, 2014.
  * [40] Yuanxun Lu, Jinxiang Chai, and Xun Cao.  Live Speech Portraits: Real-time photorealistic talking-head animation.  ACM Transactions on Graphics, 40(6), 12 2021.
  * [41] Shugao Ma, Tomas Simon, Jason Saragih, Dawei Wang, Yuecheng Li, Fernando De la Torre, and Yaser Sheikh.  Pixel codec avatars.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 64–73, June 2021.
  * [42] Araceli Morales, Gemma Piella, and Federico M. Sukno.  Survey on 3d face reconstruction from uncalibrated images.  Computer Science Review, 40:100400, 2021.
  * [43] Masahiro Mori, Karl F. MacDorman, and Norri Kageki.  The uncanny valley [from the field].  IEEE Robotics & Automation Magazine, 19(2):98–100, 2012.
  * [44] Lucio Moser, Chinyu Chien, Mark Williams, Jose Serra, Darren Hendler, and Doug Roble.  Semi-supervised video-driven facial animation transfer for production.  ACM Trans. Graph., 40(6), dec 2021.
  * [45] MPG.  VOCASET, 2019.
  * [46] Alessandro Muntoni and Paolo Cignoni.  PyMeshLab, Jan. 2021.
  * [47] Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, , Trevor Darrell, Angjoo Kanazawa, and Shiry Ginosar.  Learning to listen: Modeling non-deterministic dyadic facial motion.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [48] Yuval Nirkin, Yosi Keller, and Tal Hassner.  Fsgan: Subject agnostic face swapping and reenactment.  2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7183–7192, 2019.
  * [49] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur.  Librispeech: An asr corpus based on public domain audio books.  In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210, 2015.
  * [50] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [51] Ivan Perov, Daiheng Gao, Nikolay Chervoniy, Kunlin Liu, Sugasa Marangonda, Chris Umé, Mr. Dpfks, Carl Shift Facenheim, Luis RP, Jian Jiang, Sheng Zhang, Pingyu Wu, Bo Zhou, and Weiming Zhang.  Deepfacelab: Integrated, flexible and extensible face-swapping framework, 2020.
  * [52] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * [53] Roger Blanco i Ribera, Eduard Zell, J. P. Lewis, Junyong Noh, and Mario Botsch.  Facial retargeting with automatic range of motion alignment.  ACM Trans. Graph., 36(4), jul 2017.
  * [54] Alexander Richard, Michael Zollhoefer, Yandong Wen, Fernando de la Torre, and Yaser Sheikh.  Meshtalk: 3d face animation from speech using cross-modality disentanglement, 2021.
  * [55] Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael Black.  Learning to regress 3D face shape and expression from an image without 3D supervision.  In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 7763–7772, June 2019.
  * [56] Zhiwen Shao, Zhilei Liu, Jianfei Cai, Yunsheng Wu, and Lizhuang Ma.  Facial action unit detection using attention and relation learning.  CoRR, abs/1808.03457, 2018.
  * [57] Supasorn Suwajanakorn, Steven M. Seitz, and Ira Kemelmacher-Shlizerman.  Synthesizing obama: Learning lip sync from audio.  ACM Trans. Graph., 36(4), jul 2017.
  * [58] Sarah Taylor, Taehwan Kim, Yisong Yue, Moshe Mahler, James Krahe, Anastasio Garcia Rodriguez, Jessica Hodgins, and Iain Matthews.  A deep learning approach for generalized speech animation.  ACM Transactions on Graphics, 36(4):1–11, July 2017.
  * [59] Sarah L. Taylor, Moshe Mahler, Barry-John Theobald, and Iain Matthews.  Dynamic units of visual speech.  In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, SCA ’12, page 275–284, Goslar, DEU, 2012. Eurographics Association.
  * [60] Y.-I. Tian, T. Kanade, and J.F. Cohn.  Recognizing action units for facial expression analysis.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(2):97–115, 2001.
  * [61] Suramya Tomar.  Converting video formats with ffmpeg.  Linux Journal, 2006(146):10, 2006.
  * [62] Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu.  One-shot free-view neural talking-head synthesis for video conferencing.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
  * [63] Cheng-hsin Wuu, Ningyuan Zheng, Scott Ardisson, Rohan Bali, Danielle Belko, Eric Brockmeyer, Lucas Evans, Timothy Godisart, Hyowon Ha, Alexander Hypes, Taylor Koska, Steven Krenn, Stephen Lombardi, Xiaomin Luo, Kevyn McPhail, Laura Millerschoen, Michal Perdoch, Mark Pitts, Alexander Richard, Jason Saragih, Junko Saragih, Takaaki Shiratori, Tomas Simon, Matt Stewart, Autumn Trimble, Xinshuo Weng, David Whitewolf, Chenglei Wu, Shoou-I Yu, and Yaser Sheikh.  Multiface: A dataset for neural face rendering.  In arXiv, 2022.
  * [64] Haotian Yang, Hao Zhu, Yanru Wang, Mingkai Huang, Qiu Shen, Ruigang Yang, and Xun Cao.  Facescape: A large-scale high quality 3d face dataset and detailed riggable 3d face prediction.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 598–607, 2020.
  * [65] Juyong Zhang, Keyu Chen, and Jianmin Zheng.  Facial expression retargeting from human to avatar made easy.  IEEE Transactions on Visualization and Computer Graphics, 28(2):1274–1287, 2022.
  * [66] Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu.  Pose-controllable talking face generation by implicitly modularized audio-visual representation.  CoRR, abs/2104.11116, 2021.
  * [67] Yang Zhou, Zhan Xu, Chris Landreth, Evangelos Kalogerakis, Subhransu Maji, and Karan Singh.  Visemenet: Audio-driven animator-centric speech animation.  ACM Trans. Graph., 37(4), jul 2018.
  * [68] M. Zollhöfer, J. Thies, P. Garrido, D. Bradley, T. Beeler, P. Pérez, M. Stamminger, M. Nießner, and C. Theobalt.  State of the art on monocular 3d face reconstruction, tracking, and applications.  Computer Graphics Forum, 37(2):523–550, 2018.

##  Appendix A Supplementary Material

Figure 5: Attention check warning message.

###  A.1 FaceXHuBERT Algorithm

Algorithm 1 depicts a high level pseudo code of the training procedure of
FaceXHuBERT described in section Sec. 4. Given an audio waveform $A$, Subject
Label, Emotion Label as inputs and corresponding 4D scan $Y$ as output, the
proposed network learns the mapping between the inputs and the output during
training epochs. The network optimizes on Huber Loss function and uses Adam
optimizer to update the weights and biases during backpropagation.

Algorithm 1 Network Training

Given input audio ($A$) and output 4D-Scan ($Y$):

$A=AudioWaveform$ $\triangleright$ Raw Audio Waveform

$Y=[T,V]$ -D Matrix $\triangleright$ Ground-Truth 4D Data

$S\leftarrow SubjectEmbedding(SubjectOneHot)$

$E\leftarrow EmotionEmbedding(EmotionOneHot)$

$B\leftarrow AudioEncoder(A)$

$X\leftarrow InputRepresentationAdjustment(B)$

$H\leftarrow GRU(X)$

$\tilde{H}\leftarrow H\odot S\odot E$

$\hat{Y}\leftarrow FullyConnected(\tilde{H})$

$\mathcal{L}\leftarrow HuberLoss(Y,\hat{Y})$

$AdamOptimizer()$

###  A.2 Input Representation Adjustment

This module adjusts the input representation and output representations and do
not contain any trainable parameters. This is devised to ensure the one-to-one
frame level relationship between input $X$ and output $Y$ such that
$T_{X}=T_{Y}=T$. This function is generically devised in such a way that it
can handle any input-output frequency pair given, $f_{o}\leq f_{i}$, where
$f_{i}$ is the frequency of encoded discreet representation of the input audio
and $f_{o}$ is the frequency of the face scan data. If
$\frac{f_{i}}{f_{o}}=k\in Z^{+}$ where $Z^{+}$ denotes the set of positive
integers, then the adjustment is a straightforward reshape function such that
input dimension $(T_{X},B)=(kT_{Y},B)$ becomes a $(T_{Y},kB)$ dimensional
data. If $k\notin Z^{+}$, we resample the input representation using linear
interpolation so that input dimension $(T_{X},B)$ becomes $(\lceil k\rceil
T_{Y},B)$ before reshaping the embedding into $(T_{Y},\lceil k\rceil B)$
dimensional data to ensure $T_{X}=T_{Y}=T$. Here $\lceil k\rceil$ denotes the
ceiling of the decimal representation, in other words we take the next
positive integer. In our implementation, we ensure that the value of $k=2$ for
it to be coherent with the implemented network architecture. In case of other
datasets, where $k\neq 2$, the model definition needs to be slightly modified
in terms of input dimension of the GRU in the decoder.

###  A.3 Data Pre-process

Algorithm 2 represents the dataset pre-processing procedure. This step is a
prerequisite to train the proposed model effectively. The vertex data in BIWI
dataset is not normalized. Without scaling the data to a certain uniform
range, the network does not train well. Although this is not the only way to
normalize the data, we highly recommend normalizing the dataset so that all
three coordinates (i.e. 3D coordinates- X, Y, Z) have the same range of values
(e.g. [-0.5,0.5], [-1,1] or [0,1]) before starting training. Appropriate step-
by-step guide together with code to prepare and pre-process the dataset are
available in the project’s GitHub repository provided with the supplementary
material to facilitate reproducibility of our work.

Algorithm 2 Data Pre-process

$Neutral=[ArrayOfSubjectsWithNeutralFaces]$

$Templates=[EmptyArray]$

while There is $Subject$ to process do

$S\leftarrow Neutral[Subject]$

$X\leftarrow S[:,0]$; $Y\leftarrow S[:,1]$; $Z\leftarrow S[:,2]$

$S[:,0]\leftarrow(X-mean(X))\div(max(X)-min(X))$

$S[:,1]\leftarrow(Y-mean(Y))\div(max(Y)-min(Y))$

$S[:,2]\leftarrow(Z-mean(Z))\div(max(Z)-min(Z))$

$Templates.append(S)$

while There is $sequence$ to process do

while There is $frame$ to process do

$\tilde{X}\leftarrow\frac{(sequence[frame,:,0]-mean(X))}{(max(X)-min(X))}$

$\tilde{Y}\leftarrow\frac{(sequence[frame,:,1]-mean(Y))}{(max(Y)-min(Y))}$

$\tilde{Z}\leftarrow\frac{(sequence[frame,:,2]-mean(Z))}{(max(Z)-min(Z))}$

$sequence[frame,:,0]\leftarrow\tilde{X}$

$sequence[frame,:,1]\leftarrow\tilde{Y}$

$sequence[frame,:,2]\leftarrow\tilde{Z}$

end while

$SaveProcessedSequence(sequence)$

end while

end while

###  A.4 User studies

For perceptual evaluation described in section Sec. 6.2, three similar user
study experiments were conducted. In all three experiments, the participants
were randomly shown 12 pairs (to ensure a good duration of the study) of
facial animation videos and asked to choose the one that is realistic and more
expressive than the other in accordance with the audio. Fig. 6 is the
introduction message that the participants were shown. Fig. 7 shows the user
interface of the survey related to the user study experiments. In total, we
were able to recruit 147 participants from different demographic backgrounds
for the experiments ensuring a good sample representation of the population.
Among this 147 participants, 51 participated in Experiment I, 31 in Experiment
II and 65 in Experiment III.

Figure 6: Introduction page of the user study surveys. Figure 7: User
interface of user study surveys.

Additionally, the participants were shown attention check question items,
where one of the videos was facial animation that was generated by a model
trained on MFCC (Mel-frequency cepstral coefficients) features instead of
FaceXHuBERT Encoder and the other is either ground-truth or generated by our
approach. The animations generated by MFCC based model do not produce coherent
animations where the lip-sync is incongruous to the accompanied audio. If the
participants had chosen the animation video generated by MFCC based model,
there were shown the warning message depicted in Fig. 5. Those participants’
responses are then manually reviewed for inconsistency in the user study data.