# Learn2Talk: 3D Talking Face Learns from 2D Talking Face

Yixiang Zhuang, Baoping Cheng, Yao Cheng, Yuntao Jin, Renshuai Liu, Chengyang
Li, Xuan Cheng, , Jing Liao, , Juncong Lin  This work was supported in part by
the Natural Science Foundation of Fujian Province of China under Grant
2023J05001, in part by the Natural Science Foundation of Xiamen, China under
Grant 3502Z20227012, in part by the NSFC under Grant 62077039, in part by the
Fundamental Research Funds for the Central Universities under Grant
20720230106, and in part by the GRF grant (CityU 11216122) from the Research
Grants Council (RGC) of Hong Kong. _Corresponding author: Xuan Cheng_. Yixiang
Zhuang, Yuntao Jin, Renshuai Liu, Chengyang Li, Juncong Lin and Xuan Cheng are
with the School of Informatics, Xiamen University, Xiamen 361005, China
(e-mail<EMAIL_ADDRESS>chengxuan@xmu.edu.cn). Baoping Cheng and Yao Cheng
are with the China Mobile (Hangzhou) Information Technology Co., Ltd.,
Hangzhou 311121, China. Jing Liao is with the Department of Computer Science,
City University of Hong Kong, Hong Kong 999077, China (e-mail:
jingliao@cityu.edu.hk).

###### Abstract

Speech-driven facial animation methods usually contain two main classes, 3D
and 2D talking face, both of which attract considerable research attention in
recent years. However, to the best of our knowledge, the research on 3D
talking face does not go deeper as 2D talking face, in the aspect of lip-
synchronization (lip-sync) and speech perception. To mind the gap between the
two sub-fields, we propose a learning framework named Learn2Talk, which can
construct a better 3D talking face network by exploiting two expertise points
from the field of 2D talking face. Firstly, inspired by the audio-video sync
network, a 3D sync-lip expert model is devised for the pursuit of lip-sync
between audio and 3D facial motion. Secondly, a teacher model selected from 2D
talking face methods is used to guide the training of the audio-to-3D motions
regression network to yield more 3D vertex accuracy. Extensive experiments
show the advantages of the proposed framework in terms of lip-sync, vertex
accuracy and speech perception, compared with state-of-the-arts. Finally, we
show two applications of the proposed framework: audio-visual speech
recognition and speech-driven 3D Gaussian Splatting based avatar animation.
The project page of this paper is: https://lkjkjoiuiu.github.io/Learn2Talk/.

###### Index Terms:

Speech-driven, 3D Facial Animation, 2D Talking face, Transformer, 3D Gaussian
Splatting.

Figure 1: Given a speech audio, our method can synthesize facial motion, head
motion and drive a 3D Gaussian Splatting [1] based head avatar.

##  I Introduction

Human speech is by nature bimodal [2], visual and audio. The technology of
generating realistic 2D/3D facial animations from audio can serve as a bridge
between digital realm and human emotions, enabling a more natural and
intuitive form of communication. In recent years, significant progress has
been witnessed in the field of speech-driven facial animation, due its widely
potential applications across immersive interactions, news reporting and live
e-commerce, and the recent boom of generative models, ranging from GANs [3, 4,
5] to Diffusion Models [6, 7, 8].

There are mainly two research lines in the filed of speech-driven facial
animation, 2D and 3D talking face. The 2D talking face methods [9, 10, 11, 12,
13, 14, 15, 16, 17, 18, 19, 20, 21] usually generate the lip motion or head
motion in the pixel space (e.g. image, video) to match the given input audio
stream, while the 3D talking face methods use the temporal 3D vertex data
(e.g. 3D face template [22, 23, 24, 25], blendshape parameters [26, 27, 28])
to represent facial motion. Compared with 2D talking face methods, 3D talking
face methods can synthesize more subtle lip movements, since the fine-grained
lip shape correction can be better performed in the 3D space. Additionally and
importantly, 3D facial animation is perhaps a more appropriate solution in the
standardized virtual human production workflow, where the 3D model of virtual
human is firstly made by the modeling artists and then be driven by different
modalities, e.g. speech-driven [29, 30, 31], text-driven [32, 33, 34] and
video-driven [35, 36, 37]. On the other hand, however, the research on 2D
talking face [20, 11] is going deeper in the aspect of lip-sync and speech
perception, both of which are important to speech-driven facial animation.

There are evidences in the research community to support our claims. 1) The
state-of-the-art 3D talking face methods [24, 25] usually use the 3D
reconstruction errors in all lip vertices (taking the maximum) to measure the
lip-sync, whereas the 2D talking face methods prefer to use the pre-trained
SyncNet [38] to estimate the time offset between the audio and the generated
video. SyncNet [38] is a direct measurement of the temporal relationship
between speech audio and facial motion, while the 3D reconstruction error well
represents per-frame 3D accuracy of lip vertices. 2) Through the visual
comparisons between the talking face videos rendered by 3D methods like
FaceFormer [24] and CodeTalker [25], and produced by 2D methods like SadTalker
[20], we find that the 2D talking face videos usually show more expressive
movements of the mouth corresponding to the speech. For example, a speech-
aware talking face should display obvious lip-roundness when speaking a
rounded-vowel, such as /o/ /u/.

To mind the gap between the two sub-fields in speech-driven facial animation,
we propose a novel framework named _Learn2Talk_ , which can construct a better
audio-to-3D motion regression network by learning the expertise from the field
of 2D talking face. To realize the framework, we propose two innovative
network designs in the training of the audio-to-3D motion regression network.
1) Inspired by SyncNet [38] which is widely used in 2D talking face methods,
we develop a 3D lip-sync expert model named _SyncNet3D_ , to impose the lip-
syncing constraint in training and evaluate the sync between audio and
regressed 3D motions in testing. 2) A pre-trained 2D talking face network is
employed as the teacher model to supervise the student model, namely the
regression network, through the _lipreading constraint_ implemented by the
pre-trained lipreading network [39] and differentiable renderer. As shown in
Fig. 1, our proposed framework achieves the synthesis of facial and head
motion, and the driving of head avatar. The concept diagram of the framework
is shown in Fig. 2.

Finally, the contributions of this paper are summarized as follows:

  * •

_Framework._ We propose a novel learning framework for 3D talking face, which
learns the lip-syncing and speech perception capabilities from 2D talking face
methods. To the best of our knowledge, this research line has never been
explored by previous methods.

  * •

_Lip-sync._ We successfully extend SyncNet from the pixels domain to the 3D
motions domain. The proposed SyncNet3D can be used as discriminator in
training to enhance lip-sync, and as metrics in testing to complement the
evaluation of synthesized 3D motions’ quality. This part of work addresses a
critical issue existed in almost all 3D motion synthesis methods: _how to
explicitly model the sync between 3D motions and input signals?_

  * •

_Speech Perception._ We propose to distill the knowledge from 2D talking face
methods to the audio-to-3D motion regression model by the lipreading
constraint. As such, the predicted 3D facial motions yield more accuracy in
lip vertices, thus eliciting similar perception with the corresponding audios.
Before the proposing of our method, the lipreading constraint is mainly used
in video-driven 3D animation, but _rarely in audio-driven 3D animation._

  * •

_SOTA._ Both the quantitative comparisons in public datasets, and the
extensive visual comparisons on nearly 20 audio clips in the supplementary
video, clearly show that our method advances the state of the art.

  * •

_Application._ As an application, our method can be used to drive a full head
avatar built by 3D Gaussian Splatting (3DGS) [1], thus enabling the first
speech-driven 3DGS-based avatar animation in the research community.

Figure 2: Concept diagram of Learn2Talk framework.

##  II Related Work

###  II-A Speech-driven 2D Facial Animation

The methods in this field can animate a portrait image or edit a portrait
video to match the input driving audio. From the methodological perspective,
the methods are roughly categorized into two classes [2]: one-stage framework
and two-stage framework.

The one-stage framework [9, 10, 11, 12, 13, 14] usually generates talking face
video from the driving source (e.g. audio-image or audio-video) in an end-to-
end fashion. Speech2Vid [9] is among the first to explore one-stage framework,
which uses two encoders to respectively extract identity features and spoken
features from the reference image and driving audio, and then fuses the two
features as input into an image decoder to output synthesized video. Besides
lip motion, PC-AVS [14] can also generate head motion by using the third
encoder to extract the identity-ignore head pose features from another pose
source video. Other works put the efforts mostly on developing powerful
discriminators to check generation quality in the generative adversarial
learning, such as using optical flow stream [10], synced/unsynced contrastive
loss [12], pre-trained lip-sync expert [11] and multiple discriminators
focused on lip-sync, frame quality and temporality [13].

Recent advances mostly adopt the two-stage framework [15, 16, 17, 18, 19, 20],
which contains two cascaded modules: firstly mapping the driving source to
intermediate facial parameters by deep neural networks, and then rendering the
output video based on the the learned facial parameters. Facial landmarks are
the commonly used intermediate representation, since they can capture the
facial deformations caused by talking, expressions and head movements. For
example, ATVGnet [15] and Wav2Mov [16] use facial landmark PCA components to
decrease the effect of none-audio-correlated factors, while MakeItTalk [17]
uses the relative landmark displacements to generalize to large variety of
faces. As facial landmarks are still in a coupled space, there are also
methods that explore the use of 3D facial landmarks [18] and 3D Morphable
Model [19, 20]. In the aspect of rendering output video, GAN is mostly used
[17, 18, 16, 15] and is separately trained with previous mapping network.
Instead of generating pixels by multiple up-sampling layers, DINet [21]
spatially deform the reference image to match the driving audio, thus
generating high-fidelity video. Similarly, SadTalker [20] produces the warping
fields from unsupervised 3D keypoints, and then warps the reference image
under the guidance of warping fields to generate final video.

###  II-B Speech-driven 3D Facial Animation

The methods in this field can animate a 3D face model by predicting vertex
positions from driving audio. To the best of our knowledge, the amount of
research works in this filed is relatively less than speech-driven 2D facial
animation. Karras et al. [40] take advantage of LSTM to learn the mapping from
audio to 3D vertex coordinates and utilize an extra emotion to control the
emotional state of the face puppet. VOCA [22] extents the model to multiple
subjects by using identity conditioning, addressing the issue of
specialization for a particular speaker [40]. In VOCA, the extracted audio
features are concatenated with the onehot vector of a speaker, and then the
fused features are decoded to output 3D vertex displacements instead of vertex
coordinates. Noting that VOCA exhibits uncanny or static upper-face animation,
MeshTalk [23] proposes a categorical latent space to disentangle audio-
correlated and audio-uncorrelated information, so that the plausible motion of
uncorrelated facial regions can be synthesized. GDPnet [41] improves on VOCA
with more geometry-related deformation, which is achieved by adding the non-
linear face reconstruction representation as the guidance of latent space. To
tackle the inaccurate lip movement associated with phoneme-level features from
short audio windows, FaceFormer [24] proposes a transformer-based
autoregressive model to encode long-term audio contexts, and gains important
performance promotion. Aiming at reducing the cross-modal ambiguity,
CodeTalker [25] uses VQ-VAE [42] to firstly learn a codebook by self-
reconstruction over real facial motions, and then deploys FaceFormer in the
learned discrete motion space. Inspired by the recent progress in Diffusion
Models, there are also methods [28, 43] that adopt the diffusion model to
capture the many-to-many relationship between speech and facial motion.

###  II-C Video-driven 3D Facial Animation

We review the video-driven 3D facial animation methods briefly. The single
image 3D face reconstruction methods, such as RingNet [44], Deep3DFace [45],
DECA [46], EMOCA [47] and DCT [48], can be used to generate 3D facial motions
through a frame-by-frame fashion on videos. There are also methods [49, 50,
51, 52, 53] that exploit the dynamic information of monocular face videos to
constrain the subject’s facial shape or impose temporal coherence on the face
reconstruction. For example, DDE [49] imposes the temporal regularization on
the 3DMM [54, 55] fitting of per-frame facial landmarks. Later on, Cao et al.
[50] improve on DDE by training a local detail regressor to synthesize the
medium frequency facial details in each frame. Built upon the facial landmarks
fitting framework, Wang et al. [51] and Wang et al. [56] add the tracking of
3D eye gaze by exploring the classification of iris and pupil pixels. Garrido
et al. [52] propose a parametric shape prior that encodes the plausible
subspace of facial identity and expression variations to reconstruct a fully
personalized 3D face rig from input video. The most recent method [53] use
DECA [46] to estimate per-frame 3D face model, and then use the lipreading
network [39] to enhance the visual speech-aware perceptual in the generated 3D
facial motions.

Figure 3: The pipeline of the proposed Learn2Talk framework. In training, all
modules are used. In inference, only the student model is used, including
embedding layer, transformed decoder and speech encoder.

##  III The Proposed Framework

###  III-A Overview

Problem Setting. The speech-driven 3D facial animation is formulated as the
seq2seq learning problem, which predicts the 3D facial motions from a speech
audio. Let $\mathbf{Y}_{1:T}=(\mathbf{y}_{1},...,\mathbf{y}_{T})$ be a
sequence of 3D facial motions, where $\mathbf{y}_{t}\in\mathbb{R}^{V\times 3}$
denotes the 3D mesh model of $V$ vertices in frame $t$. Let
$\mathbf{X}_{1:T}=(\mathbf{x}_{1},...,\mathbf{x}_{T})$ be a sequence of audio
snippets, where $\mathbf{x}_{t}\in\mathbb{R}^{d}$ denotes the snippet of $d$
samples aligned with the corresponding motion frame $\mathbf{y}_{t}$. The goal
is to train a regression network which can sequentially generate
$\mathbf{Y}_{1:T}$ from the inputted $\mathbf{X}_{1:T}$, so that the 3D face
mesh model can be animated as a lip-synchronized 3D talking face in the
inference stage.

Pipeline. Fig. 3 shows the pipeline of the proposed framework. FaceFormer [24]
is chosen as the student model, which predicts 3D facial motions from both
audio context and past motions. Meanwhile, a pre-trained 2D talking face
network is employed as the teacher model, e.g. Wav2Lip [11], SadTalker [20]
etc.. The speech audio $\mathbf{X}_{1:T}$ is firstly input to teacher model to
produce the image sequence of talking lips. In the other stream, the predicted
sequence $\hat{\mathbf{Y}}_{1:T}$ by the student model is input into the
differentiable renderer to produce the rendered talking lips. These two
streams are contrasted in the pre-trained lipreading network [39], thus
forming the lipread loss. Besides, the pre-trained SyncNet3D serves as the
lip-sync discriminator, where a predicted motion clip $\hat{\mathbf{Y}}_{1:W}$
and an audio clip $\mathbf{X}_{1:W}$ ($W$ denotes window size) are contrasted
there to form the 3D sync loss. Together with 3D vertex reconstruction loss,
the student model is trained in a supervised manner.

###  III-B Student Model

The student model contains two main modules, speech encoder and cross-modal
decoder. The speech encoder $E_{\text{speech}}$ adopts the architecture of the
state-of-the-art self-supervised pretrained speech model, wav2vec 2.0 [57],
which consists of an audio feature extractor and a multi-layer transformer
encoder [58]. The audio feature extractor is a temporal convolutions network
(TCN), which converts the speech of raw waveform into feature vectors. The
transformer encoder is a stack of multi-head self-attention and feed-forward
layers, which converts the audio feature vectors into contextualized speech
representations.

The cross-modal decoder $D_{\text{cross-modal}}$ contains an embedding block
and a multi-layer transformer decoder equippe with both self-attention and
cross-attention. The embedding block combines the past facial motions
$\hat{\mathbf{Y}}_{1:t-1}$ and the style embedding via:

$\mathbf{F}_{emb}^{1:t-1}=\mathcal{P}_{\theta}(\hat{\mathbf{Y}}_{1:t-1})+\mathbf{B}\cdot\frac{\mathbf{s}}{||\mathbf{s}||_{1}},$
(1)

where $\mathcal{P}_{\theta}$ denotes a linear projection layer, and
$\mathbf{B}$ denotes the learnable basis vector that span the style space
linearly. The transformer decoder is equipped with the causal self-attention
to learn the dependencies between each frame in the context of the past facial
motions, and the cross-modal attention to align the audio and motion
modalities. The newly predicted motion $\hat{\mathbf{y}}_{t}$ is used to
update the past motions as $\hat{\mathbf{Y}}_{1:t}$, in preparation for the
next prediction. The recursive process can be formally written as:

$\hat{\mathbf{y}}_{t}=D_{\text{cross-
modal}}(E_{\text{speech}}(\mathbf{X}_{1:T}),\mathbf{s},\hat{\mathbf{Y}}_{1:t-1}).$
(2)

In the training, the TCN and transformer encoder in $E_{\text{speech}}$ are
initialized with the pre-trained wav2vec 2.0 weights to benefit from the
speech representation learning from large-scale corpora. The parameters in TCN
are fixed, whereas transformer encoder in $E_{\text{speech}}$, the embedding
block and transformer decoder in $D_{\text{cross-modal}}$ are learnable. The
training loss is the Mean Squared Error (MSE) between the outputs
$\hat{\mathbf{Y}}_{1:T}=(\hat{\mathbf{y}}_{1},...,\hat{\mathbf{y}}_{T})$ and
the ground truth $\mathbf{Y}_{1:T}=(\mathbf{y}_{1},...,\mathbf{y}_{T})$:

$\mathcal{L}_{\text{vertex}}=\sum_{t=1}^{T}\sum_{v=1}^{V}||\hat{\mathbf{y}}_{t,v}-\mathbf{y}_{t,v}||_{2}.$
(3)

This loss is computed when all frames in the 3D facial motion sequence are
produced.

| Lip-sync | Video Quality
---|---|---
| LSE-D$\downarrow$ | LSE-C$\uparrow$ | FID$\downarrow$ | CPBD$\uparrow$ | CSIM$\uparrow$
Real Video | 6.982 | 8.211 | 0.0000 | 0.428 | 1.000
Wave2Lip${\dagger}$ | 5.535 | 10.211 | 21.725 | 0.368 | 0.849
PC-AVS | 6.355 | 9.053 | 69.127 | 0.206 | 0.683
MakeItTalk | 10.059 | 5.110 | 28.243 | 0.283 | 0.838
AVCT | 10.055 | 4.932 | 22.432 | 0.295 | 0.811
SadTalker | 7.772 | 7.290 | 22.057 | 0.335 | 0.843
Real Video* | 6.499 | 8.993 | N/A | N/A | N/A
ATCG* | 8.422 | 7.061 | N/A | N/A | N/A
DINet* | 6.842 | 8.377 | N/A | N/A | N/A
TABLE I: The quantitative evaluation results of seven 2D talking face methods
over the HDTF dataset [19]. The best score in each column is marked with bold.

###  III-C Teacher Models Selection

There are many alternative state-of-the-art 2D talking face methods, but we
only need the one-shot, head motion excluded methods to make them compatible
with our framework. Tab. I shows the evaluation results of seven 2D talking
face methods over the HDTF dataset [19], including Wav2Lip [11], PC-AVS [14],
MakeItTalk [17], AVCT [59], SadTalker [20], ATCG [15] and DINet [21]. HDTF
contains in-the-wild videos collected 720P or 1080P resolution. Two classes of
metrics are used to quantitatively evaluate the lip-sync and video quality of
the generated talking face videos, including distance score (LSE-D) [38],
confidence score (LSE-C) [38], Frechet Inception Distance (FID) [60],
cumulative probability blur detection (CPBD) [61] and cosine similarity of
identity (CSIM). The statistics come from the experiment conducted in [20] and
another similar experiment in [21] (denoted by $*$). Since the experiments in
[20, 21] selected different videos as testing data, there is a slight bias in
the metrics LSE-D and LSE-C. Although Wav2Lip${\dagger}$ acheives the best
quantitative results in video quality, it only animates the lip region while
other regions are the same as the original frame. Hence, actually, it’s
SadTalker that acheives the best video quality.

From the statistics in Tab. I, we can observe that Wave2Lip performs well in
the lip-sync, while SadTalker exhibits good performance in high quality video
generation. Besides lip-sync, video quality should be considered in selecting
teacher models, since the quality of generated video has much impact on the
transferring process that is based on the lip reading from image sequence.
Hence, we select five methods as the teacher model $G_{\text{teach}}$,
including Wav2Lip, SadTalker, DINet, PC-AVS and MakeItTalk, based on their
overall performance in lip-sync and video quality.

###  III-D SyncNet3D

Current state-of-the-art 3D talking face methods [22, 23, 41, 24, 25, 28, 43]
usually use the 3D vertex reconstruction loss defined in Eq. 3 as the main
training objective. In the reference stage, the 3D lip vertex error (LVE),
which calculates the maximal $L_{2}$ distance of all lip vertices to the
ground truth in each frame and then takes average over all frames, is used as
the main metric to measure lip-sync. The relationship in granular level
between audio and 3D facial motion should be explicitly modeled, e.g. the
audio lags the motion by 60ms or the audio leads the motion by 25ms, which can
further support supervising the network and measuring the lip-sync in results.

Based on the above observations, we propose the SyncNet3D which is accurate in
detecting sync error in 3D facial motions and its corresponding audio. The
inputs to SyncNet3D contain a mesh stream and a audio stream. In the mesh
stream, a window of $W$ consecutive frames of 3D mouth mesh model,
$\mathbf{M}_{1:W}=(\mathbf{m}_{1},...,\mathbf{m}_{W})$, is fetched from
$\mathbf{Y}_{1:T}$ by a sliding window, and then be encoded by a transformer
and three-layer MLP into the mesh embedding $\mathbf{M}_{emd}^{1:W}$. In the
audio stream, a window of $W$ frames of audio snippets,
$\mathbf{X}_{1:W}=(\mathbf{x}_{1},...,\mathbf{x}_{W})$, are fetched from
$\mathbf{X}_{1:T}$ by the sliding window, and then be converted to Mel
spectrogram. The Mel spectrogram is futher encoded by five convolutional
layers and two-layer MLP into the audio embedding $\mathbf{X}_{emd}^{1:W}$,
which has the same shape with $\mathbf{M}_{emd}^{1:W}$. The contrastive loss
[38] between $\mathbf{M}_{emd}^{1:W}$ and $\mathbf{X}_{emd}^{1:W}$ is used as
the training objective, aiming at making $\mathbf{M}_{emd}^{1:W}$ and
$\mathbf{X}_{emd}^{1:W}$ similar for genuine pairs, and different for false
pairs. The contrastive loss is defined as:

$\begin{split}\mathcal{L}_{\text{con1}}=\sum_{n=1}^{N}(y)d^{2}+(1-y)\text{max}(p-d,0)^{2},\\\
d=||\mathbf{M}_{emd}^{1:W}-\mathbf{X}_{emd}^{1:W}||_{2},\quad\quad\quad\quad\end{split}$
(4)

where $N$ denotes the number of training samples, $y\in[0,1]$ denotes the
binary similarity metric and $p$ denotes the margin parameter.

We pre-train a SyncNet3D using Eq. 4 to measure the lip-sync in the
quantitative evaluation (Sect. IV-A), and pre-train another SyncNet3D which
serves as the parameter-fixed discriminator in the training of Learn2Talk
(Sect. III-E), using a different contrastive loss [11]:

$\begin{split}\mathcal{L}_{\text{con2}}=-\frac{1}{N}\sum_{n=1}^{N}(y)\log{d}+(1-y)log{(1-d)},\\\
d=\text{CosSim}(\mathbf{M}_{emd}^{1:W},\mathbf{X}_{emd}^{1:W}),\quad\quad\quad\quad\end{split}$
(5)

where $\text{CosSim}(\cdot,\cdot)$ denotes the cosine similarity between two
vectors. The cosine similarity will be normalized to the range of $[0,1]$,
thus indicating the probability of being synchronous. The reason of pre-
training another SyncNet3D using Eq. 5 is that, the optimization is easier to
converge with the cosine similarity than the $L_{2}$ distance. In contrast,
it’s easier to separate the features of genuine and false pairs with the
$L_{2}$ distance and the maximum operation in Eq. 4.

###  III-E Training Losses

Lipread Loss. The teacher model supervises the training of Learn2Talk through
the lipread loss. We use the lipreading network [39] pre-trained on the Lip
Reading in the Wild 3 (LRS3) dataset [39] to compute the lipread loss. The
network takes a sequence of grayscale images cropped around the mouth as
input, and outputs a predicted sequence of characters. The model architecture
consists of a 3D convolutional kernel, a 2D ResNet-18, a 12-layer conformer
[62], and lastly a transformer decoder layer that outputs the predicted
sequence. The network is trained by using Connectionist Temporal
Classification equipped with attention. When used as the encoder
$E_{\text{lipread}}$ for talking face, the network discards the 12-layer
conformer and the transformer decoder layer, and output the embedding from the
2D ResNet-18 layer.

The speech audio $\mathbf{X}_{1:T}$ is input to the teacher model
$G_{\text{teach}}$ to produce the image sequence of 2D talking face. Then, the
grayscale images are cropped around mouth and input to $E_{\text{lipread}}$ to
get the embedding $E_{\text{lipread}}(G_{\text{teach}}(\mathbf{X}_{1:T}))$.
Meanwhile, the predicted 3D facial motions $\hat{\mathbf{Y}}_{1:T}$ together
with a facial texture shown in Fig. 4, are injected into the differentiable
renderer $DR$ to produce the rendered image sequence of 3D talking face.
Similarly, the grayscale images are cropped around mouth and input to
$E_{\text{lipread}}$ to get the embedding
$E_{\text{lipread}}(DR(\hat{\mathbf{Y}}_{1:T}))$. Finally, the lipread loss is
defined as the cosine similarity between two embeddings:

$\begin{split}\mathcal{L}_{\text{lipread}}=\text{CosSim}(E_{\text{lipread}}(G_{\text{teach}}(\mathbf{X}_{1:T})),\\\
E_{\text{lipread}}(DR(\hat{\mathbf{Y}}_{1:T}))).\end{split}$ (6)

This loss is computed when all frames $\hat{\mathbf{Y}}_{1:T}$ in the 3D
facial motion sequence have been produced. Thanks to the differentiable
renderer, the gradients in $\mathcal{L}_{\text{lipread}}$ can be back-
propagated in the optimization.

3D Sync Loss. The SyncNet3D pre-trained by Eq. 5 is used as the encoder to get
two embeddings $\hat{\mathbf{M}}_{emd}^{t+1:t+W},\mathbf{X}_{emd}^{t+1:t+W}$
respectively from $\hat{\mathbf{Y}}_{t+1,t+W}$, $\mathbf{X}_{t+1,t+W}$. Then,
the 3D sync loss is defined as:

$\begin{split}\mathcal{L}_{\text{sync}}=\sum_{t=0}^{T-W}\text{CosSim}(\hat{\mathbf{M}}_{emd}^{t+1:t+W},\mathbf{X}_{emd}^{t+1:t+W}).\end{split}$
(7)

Overall Loss. The overall loss is defined as:

$\begin{split}\mathcal{L}=\mathcal{L}_{\text{vertex}}+\lambda_{1}\mathcal{L}_{\text{sync}}+\lambda_{2}\mathcal{L}_{\text{lipread}},\end{split}$
(8)

where $\lambda_{1},\lambda_{2}$ are two hyper-parameters.

###  III-F Head Motion Synthesis

Apart from the facial motion synthesis, the head motion synthesis from audio
can results in more vivid 3D facial animation. To achieve this goal, we
directly adopt the PoseVAE network proposed in SadTalker [20], and incorporate
it in our framework by applying the predicted rotation and translation vectors
on the 3D mesh model $\mathbf{y}_{t}$ in each frame. PoseVAE uses the VAE
model [63] conditioned by audio clip to learn the distribution of the
6-dimensional head movement. As the head motion synthesis is performed
independently and parallelly with the facial motion synthesis, it’s a optional
process in our framework.

Figure 4: The static image (a) is used in one-shot 2D talking face methods to
generate animated face (d). The facial textures (b)(c) are used in rendering
3D facial motions to videos (e)(f).

##  IV Experiments

###  IV-A Datasets and Evaluation Metrics

Datasets. We use two widely used datasets in the field of 3D talking face,
BIWI [64] and VOCASET [22], to train and test different methods in the
experiments. Both datasets contain the sequences of 3D face scans together
with corresponding utterances spoken in English. 1) BIWI is a 3D audio-visual
corpus of affective speech and facial expression in the form of dense dynamic
3D face geometries, containing totally 40 sentences uttered by 14 subjects.
Each sentence was recorded twice, with and without emotion, and is 4.67
seconds long averagely. The 3D face dynamics are captured at 25fps, each with
23,370 vertices and registered topology. We only use the emotional subset and
follow the data splits in [24]. Specifically, the training set (BIWI-Train)
contains 192 sentences, the validation set (BIWI-Val) contains 24 sentences,
the testing set (BIWI-Test-A) contains 24 sentences spoken by 6 seen subjects.
2) VOCASET is comprised of 480 paired audio-visual sequences recorded from 12
subjects. 255 unique sentences partially shared among different speakers, are
included in VOCASET. Each sequence is captured at 60 fps and is between 3 and
4 seconds averagely. Each 3D face mesh is registered to the FLAME [65]
topology with 5,023 vertices. We adopt the same training (VOCA-Train),
validation (VOCA-Val), and testing (VOCA-Test) splits as VOCA [22] and
FaceFormer [24] for fair comparisons.

Evaluation Metrics. To quantitatively evaluate the different methods in terms
of 3D lip-sync and 3D vertex reconstruction quality, we adopt four metrics:

  * –

LSE-D: the lip-sync error distance calculated by the pre-trained SyncNet3D
with Eq. 4. A lower LSE-D denotes a higher match of audio and 3D motions.

  * –

LSE-C: the lip-sync error confidence calculated by the pre-trained SyncNet3D
with Eq. 4. Higher the confidence, the better correlation of audio and 3D
motions.

  * –

LVE: the lip vertex error proposed in [23]. It calculates the maximal distance
of all 3D lip vertices to the ground truth in each frame and then takes
average over all frames. A lower LVE denotes a better 3D lip vertex
reconstruction quality.

  * –

FDD: the upper-face dynamics deviation proposed in [25]. FDD calculates the
deviation of temporal variation between the predicted 3D upper faces and
ground truth. The lower FDD, the higher consistency with the trend of upper
facial dynamics.

LVE and FDD, calculated in the 3D space, can’t comprehensively measure the
lip-sync, since the speech audios are not involved. As the necessary
complements, LSE-D and LSE-C measure the lip-sync in another aspect, where the
relationship between speech audio and 3D motions is modeled. Summarily, both
types of metrics should be adopted in the quantitative evaluation.

###  IV-B Implementations

Learn2Talk. The student model in Learn2Talk is trained with Adam optimizer.
The learning rate is set to $1e-4$. We set $\lambda_{1}=5E-6,\lambda_{2}=1E-2$
in BIWI, and set $\lambda_{1}=5E-4,\lambda_{2}=5E-7$ in VOCASET, as the 3D
face models used are different. The training duration is 100 epochs, which
takes about $3\sim 5$ hours on a NVIDIA RTX 4090 GPU.

SyncNet3D. SyncNet3D is separately trained on BIWI-Train and VOCA-Train. The
sliding window size $W$ is set as 5 in BIWI and 6 in VOCASET, with the
consideration of 25fps and 30fps in two datasets. It takes about $12\sim 15$
hours to train SyncNet3D on a NVIDIA RTX 4090 GPU.

###  IV-C Study on SyncNet3D

We conduct an experiment to validate the proposed SyncNet3D, in the aspect of
determining lip-sync error in audio-3D sequences. Specially, we firstly train
a SyncNet3D with Eq. 4 using BIWI-Train data, in which the genuine audio-3D
pairs are generated by taking a $W$-frame audio clip and the corresponding 3D
motion clip, the false audio-3D pairs are generated by only shifting the audio
randomly by up to 2 seconds. Then, we construct the test dataset from BIWI-
Test-A, using the same strategy. Finally, we use the pre-trained SyncNet3D to
determine the lip-sync error in the test dataset. The $L_{2}$ distance (Eq. 4)
between audio embedding and 3D motion embedding is used as the similarity
metric. The histogram in Fig. 5 shows the distribution of the $L_{2}$
distances, which indicates that our SyncNet3D can distinguish being
synchronous or asynchronous.

Figure 5: The distribution of $L_{2}$ distances for genuine and false audio-3D
motions pairs.

###  IV-D Study on Teacher Models Selection

As five 2D talking face methods are selected as the teacher model, we conduct
an experiment on BIWI to show the teaching quality of the five models on
student model. We use the public pre-trained networks of Wav2Lip [11], PC-AVS
[14], MakeItTalk [17], DINet [21] and SadTalker [20] to produce five sets of
2D talking videos, which take the audios in BIWI-Train as input. Then, the
student model is trained on BIWI-Train, together supervised by the set of 2D
talking videos. Note that, in the training, only one set of 2D talking videos
is used as supervisor. In order to emphasize the impact of teacher model, we
discard the 3D sync loss. If none teacher model is used, our method degrades
to FaceFormer. The quantitative evaluation results are reported in Tab. II. It
could be observed that: 1) all teacher models can improve the 3D
reconstruction ability of the student model, especially SadTalker; 2) except
for PC-AVS, all teacher models have a negative impact on the metrics about
lip-sync. The lipread loss that we use to supervise the student model, is once
computed on a whole sequence (averagely 4.67s in BIWI) of the predicted 3D
motions. The lipreading network can be viewed as a global feature extractor
for 3D motions, but LSE-D and LSE-C measure the sync in a clip (0.2s in BIWI)
and then average all clips. Hence, the supervision in global level may hinder
the lip-sync measured in local level. More analysis about the relationship
between lip-sync and lipread is provided in the Ablation Study.

Through this study on teacher selection, we choose SadTalker as the only
teacher model for the rest of experiments, considering its overall teaching
performance in 3D vertex reconstruction and 3D lip-sync. The ability of lip-
sync can be further improved by adding 3D sync loss.

| 3D Lip-sync | 3D Vertex Quality
---|---|---
| LSE-D$\downarrow$ | LSE-C$\uparrow$ | LVE$\downarrow$ | FDD$\downarrow$
Learn Wav2Lip | 9.265 | 8.865 | 5.2399 | 5.2150
Learn PC-AVS | 9.027 | 9.026 | 5.1226 | 4.3717
Learn MakeItTalk | 9.169 | 9.009 | 4.9816 | 4.0992
Learn DINet | 9.370 | 8.745 | 4.9205 | 3.5140
Learn SadTalker | 9.434 | 8.933 | 4.6971 | 3.4083
Learn None (FaceFormer) | 9.101 | 9.182 | 5.3388 | 4.4103
TABLE II: Study on Teacher Models Selection. The quantitative evaluation results over BIWI-Test-A dataset are reported. The measuring unit is $10^{-4}$ mm in LVE, and $10^{-5}$ mm in FDD. The best score in each column is marked with bold, and the second best score is marked with blue. Datasets | Methods | 3D Lip-sync | 3D Vertex Quality
---|---|---|---
LSE-D$\downarrow$ | LSE-C$\uparrow$ | LVE$\downarrow$ | FDD$\downarrow$
BIWI | FaceFormer | 9.101 | 9.182 | 5.3388 | 4.4103
CodeTalker | 9.090 | 9.039 | 4.8133 | 4.1244
Learn2Talk | 8.897 | 9.449 | 5.0003 | 3.7756
VOCASET | FaceFormer | 11.720 | 6.916 | 2.5419 | N/A
CodeTalker | 12.455 | 6.704 | 2.8708 | N/A
Learn2Talk | 10.593 | 9.838 | 2.5148 | N/A
TABLE III: Methods comparison. The quantitative evaluation results over BIWI-
Test-A and VOCA-Test are reported. The best score in each metric is marked
with bold.

###  IV-E Methods Comparison

Competitors. We quantitatively and qualitatively compare Learn2Talk with two
state-of-the-art methods, FaceFormer [24] and CodeTalker [25]. We use their
pre-trained networks on BIWI-Train and VOCA-Train for evaluation. All the
three methods require conditioning on a training speaker identity during
inference. For unseen subjects, we obtain the predictions of the three methods
by conditioning on all training identities, and choose the best one. As
FaceFormer and CodeTalker can’t generate head motion, our method excludes the
synthesis of head motion when making comparisons.

Figure 6: Visual comparisons of sampled facial motions animated by different
methods on VOCA-Test (left) and BIWI-Test-A (right).

Quantitative Evaluation. According to the statistics in Tab. III, the proposed
Learn2Talk outperforms FaceFormer and CodeTalker in the term of 3D lip-sync by
a large margin, both in BIWI-Test-A and VOCA-Test, suggesting that Learn2Talk
produces more accurate lip-synchronized movements than the competitors. In the
metric of LVE in VOCA-Test, Learn2Talk also surpass the two competitors. When
it comes LVE in BIWI-Test-A, Learn2Talk outperforms FaceFormer but slightly
lags behind CodeTalker. Besides, Learn2Talk achieves better score in FDD than
the competitors, indicating that Learn2Talk can enable more precise dynamic
changes in upper-face. Overall, Learn2Talk has shown obvious advantages both
in 3D lip-sync and 3D vertex quality, compared with the state-of-the-arts.

Qualitative Evaluation. We visually compare our method with the competitors in
Fig. 6. We illustrate six typical frames of synthesized facial animations that
speak at specific syllables. Compared with the competitors, the lip movements
produced by Learn2Talk are more accurately articulated with the speech signals
and also more consistent with the reference. In the 1st row and the right part
of 2nd row, Learn2Talk produces better lip-sync with proper mouth open when
pronouncing diphthong /ei/, /ai/ and monophthong /$\ae$/. In the left part of
2nd and 3rd rows, Learn2Talk can generate proper pouting when pronouncing the
voiced consonant in “jui” and “jour”. In the right part of 3rd row, our mouth
shape is more similar to reference when pronouncing “ha”.

Fig. 7 shows the 3D reconstruction error heatmaps of the sampled frames,
produced by the three methods. The per-vertex errors are color-coded on the
reconstructed mesh for visualization. It could be observed that, our method
obtains more accurate vertices both in mouth region and upper-face region.

Fig. 8 shows the visual comparisons on other languages, including Chinese,
Japanese and Korean. Although trained on English datasets, our method can
generalize well to any other language. This is because that our method uses
the signal characteristics of the speech audio, instead of words or
linguistics. From Fig. 8, it could be observed that our method generates more
expressive animation, with obvious movements in opening mouth, pouting and
closing mouth. The comparisons on facial animations with all frames are
available in the supplementary video.

| VOCA-Test
---|---
| Lip-sync | Realism
Learn2Talk vs. VOCA | 73.57 (6.8e-7) | 77.86 (1.6e-6)
Learn2Talk vs. FaceFormer | 70.71 (2.4e-3) | 69.29 (1.0e-3)
Learn2Talk vs. CodeTalker | 66.43 (4.1e-4) | 67.14 (4.6e-5)
TABLE IV: User study results on VOCA-Test. The percentage of answers where Learn2Talk is preferred over the competitor is listed. The numbers in the brackets denote p-value. Datasets | Methods | 3D Lip-sync | 3D Vertex Quality
---|---|---|---
LSE-D$\downarrow$ | LSE-C$\uparrow$ | LVE$\downarrow$ | FDD$\downarrow$
BIWI | w/o sync | 9.434 | 8.933 | 4.6971 | 3.4083
w/o lipread | 8.563 | 9.652 | 5.0375 | 5.1844
full model | 8.897 | 9.449 | 5.0003 | 3.7756
VOCASET | w/o sync | 11.717 | 6.286 | 2.3581 | N/A
w/o lipread | 10.956 | 9.926 | 2.8233 | N/A
full model | 10.593 | 9.838 | 2.5148 | N/A
TABLE V: Ablation study. The quantitative evaluation results over BIWI-Test-A
and VOCA-Test are reported. The best score in each metric is marked with bold,
and the second best score is marked with blue.

User Study. We conduct a user study to evaluate the quality of the animated
faces by VOCA [22], FaceFormer [24], CodeTalker [25] and Learn2Talk, in the
terms of perceptual lip-sync and facial realism. We adopt A/B test for each
comparison, e.g. ours vs. FaceFormer, and take the random order. In each A/B
test, the user watches the facial animations, listens to the audio, refers to
the textual sentence, and answers the questions. About 30 A vs. B pairs are
created for VOCA-Test. Each pair is judged by 15 different participants,
finally yielding 450 entries. We calculate the radio of participants who
prefer Learn2Taker over the competitor. The percentage of A/B testing is
tabulated in Tab. IV, which shows that participants favor Learn2Talk over the
competitors. We attribute this to the facial animation synthesized by
Learn2Talk having more well-synchronized lip movements and vivid facial
motions. We also conduct ANOVA tests in the user study, in which the p-value
less than 0.05 indicates a statistically significant difference between our
method and the competitor. In the supplementary video, we show some visual
comparisons used in the user study.

Figure 7: 3D reconstruction error heatmaps of sampled facial motions predicted
by different methods on VOCA-Test (left) and BIWI-Test-A (right). Figure 8:
Visual comparisons of sampled facial motions predicted by different methods
across languages, including Chinese, Japanese and Korean.

###  IV-F Ablation Study

We conduct ablation study on both BIWI-Test-A and VOCA-Test to evaluate the
two key designs in Learn2Talk: the lipread loss and the 3D sync loss. To
remove the lipread loss in training, the modules related to the lipread loss
are all removed, including differentiable renderer, lipreading network and
teacher model. Similarly, to move the 3D sync loss, SyncNet3D is removed while
other modules are preserved. The quantitative evaluation results are reported
Tab. V.

Effect of the lipread loss. The model without lipread loss is denoted as “w/o
lipread” both in Tab. V. Compared with the full model, w/o lipread will
introduce more errors related to 3D vertex quality while improve the metrics
in lip-sync. This phenomenon indicates that lipread loss is more effective in
the traditional 3D face reconstruction from audio.

Effect of the 3D sync loss. The model without sync loss is denoted as “w/o
sync” in Tab. V. Compared with the full model, w/o sync will decrease the
errors related to 3D vertex quality while degenerate the metrics in lip-sync.
This phenomenon clearly indicates that 3D sync loss is expert in achieving
lip-sync.

Discussion. From the ablation study introduced above, we draw the conclusion
that there is a tradeoff between the lipread loss and the 3D sync loss. These
two losses are designed for different purposes, which are not harmonious in
one optimization framework. The lipread loss introduces the speech
articulation measured by 3D distance, while the 3D sync loss enhances the lip-
sync measured by milliseconds. For example, when speaking /a:/, the lipread
loss makes the mouth open as much as possible, while the 3D sync loss makes
the mouth open as quickly as possible. The lipread loss yields more spatial
responses, while sync loss yields only temporal responses, which makes them
adversarial in training. To balance the two losses, we tune the two weighting
parameters $\lambda_{1},\lambda_{2}$ through 4 or 5 parameter combinations.

It’s also worth noting that, together with the statistics reported in Tab.
III, the two variants of Learn2Talk, w/o sync and w/o lipread, have more
advantages than Learn2Talk itself over FaceFormer and CodeTalker respectively
in 3D lip-sync and 3D vertex quality.

Training Data | WER $\downarrow$
---|---
Real Audio + Real Video (Ground Truth) | 11.91
Real Audio + Synthesized Video by FaceFormer | 13.96
Real Audio + Synthesized Video by CodeTalker | 14.27
Real Audio + Synthesized Video by Learn2Talk | 13.31
Synthesized Video by FaceFormer | 56.99
Synthesized Video by CodeTalker | 56.45
Synthesized Video by Learn2Talk | 50.28
TABLE VI: Audio-visual speech recognition. The quantitative speech recognition
results on the subset of LRS3 are reported. The lower WER, the better
recognition performance. The best score in each metric is marked with bold.
Figure 9: Visualization of the sampled frames of the speech-driven 3DGS-based
head avatar animation. In each frame, we show the FLAME model generated by our
method (1st row-left), and the front view (1st row-right), two side views (2nd
row) of the driven avatar.

###  IV-G Applications

Audio-visual speech recognition. The first application is to synthesize the
audio-visual data for the task of the audio-visual speech recognition. We use
Learn2Talk to synthesize the video clips from the audios on the subset of LRS3
[39], and then construct the labeled audio-visual corpora which can be used to
train and test the audio-visual speech recognition network [66]. Such
synthesized audio-visual data is valuable in labeled data augmentation and
personal privacy protection. The performance of the trained speech recognition
networks with different type of training data is reported in Tab. VI. The Word
Error Rate (WER) of the recognition model trained on the synthesized data by
Learn2Talk (13.31%) is closer to that of the model trained on the ground truth
data (11.91%), compared with FaceFormer (13.96%) and CodeTalker (14.27%).
Moreover, if the ground truth audios are not involved in training and testing,
the speech recognition performance degrades dramatically, but Learn2Talk
(50.28%) still performs better than FaceFormer (56.99%) and CodeTalker
(56.45%).

To conclude, the synthesized videos from audios by our method elicit more
similar speech perception with the corresponding ground-truth audios, compared
with FaceFormer and CodeTalker.

Speech-driven 3DGS-based avatar animation. The recent 3DGS method [1, 67]
achieves high rendering quality for novel-view synthesis with real-time
performance. Instead of training the neutral network in 3D space such as NeRF
[68], it optimizes the discrete geometric primitives, namely 3D Gaussians.
3DGS has already been used to reconstruct the head avatar [69, 70] and the
human body avatar [71, 72] from multiple view images. The second application
of our method is that, it enables the speech-driven 3DGS-based head avatar, as
shown in Fig. 9. We briefly introduce how to implement this application.
Firstly, we construct the head avatar of a specific subject from multiple view
images by using [69]. Then, we produce the sequence of mesh models from the
driving audio by using Learn2Talk. The sequence of mesh models is further
converted to the sequence of FLAME parameters [65]. The FLAME parameters in
each frame are inputted to the built head avatar, finally resulting in the
3DGS animation. Our speech-driven 3DGS animation is well in-sync with the
driving audio, and supports multiple view synthesis in each frame. More
results of the speech-driven 3DGS animation are available in the supplementary
video.

##  V Conclusion and Limitations

Current researches on 3D and 2D talking face are almost conducted in two
separate lines. To mind the gap, we propose a novel framework which learns the
expertise from the field of 2D talking face to improve the capability of 3D
talking face methods. By comparing to the existing state-of-the-arts, our
proposed framework shows superiority in achieving lip-sync and speech
perception.

The main limitation of the proposed framework is the tradeoff between the 3D
lip-sync loss and the lipread loss, which makes capabilities of our two key
designs not maximized. One future work is to explore the use of multi-task
learning to balance the two losses. Other limitations of our method are that,
currently it doesn’t include the synthesis of eye blinking, 3D gaze and
emotion from driving audio. We plan to continue improving our method from the
above functional aspects, aiming at achieving the highly expressive and
lifelike 3D facial animation.

## References

  * [1] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering,” _ACM Trans. Graph._ , vol. 42, no. 4, pp. 139:1–139:14, 2023.
  * [2] C. Sheng, G. Kuang, L. Bai, C. Hou, Y. Guo, X. Xu, M. Pietikäinen, and L. Liu, “Deep learning for visual speech analysis: A survey,” _arXiv preprint arXiv:2205.10839_ , 2022.
  * [3] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio, “Generative adversarial nets,” in _Proc. of NIPS_ , Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, Eds., 2014, pp. 2672–2680.
  * [4] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in _Proc. of CVPR_ , 2019, pp. 4401–4410.
  * [5] R. Liu, C. Li, H. Cao, Y. Zheng, M. Zeng, and X. Cheng, “EMEF: ensemble multi-exposure image fusion,” in _Proc. of AAAI_ , 2023, pp. 1710–1718.
  * [6] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in _Proc. of NIPS_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., 2020.
  * [7] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proc. of CVPR_ , 2022, pp. 10 674–10 685.
  * [8] R. Liu, B. Ma, W. Zhang, Z. Hu, C. Fan, T. Lv, Y. Ding, and X. Cheng, “Towards a simultaneous and granular identity-expression control in personalized face generation,” _arXiv preprint arXiv:2401.01207_ , 2024.
  * [9] J. S. Chung, A. Jamaludin, and A. Zisserman, “You said that?” in _Proc. of BMVC_ , 2017.
  * [10] L. Chen, Z. Li, R. K. Maddox, Z. Duan, and C. Xu, “Lip movements generation at a glance,” in _Proc. of ECCV_ , V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Eds., vol. 11211, 2018, pp. 538–553.
  * [11] K. R. Prajwal, R. Mukhopadhyay, V. P. Namboodiri, and C. V. Jawahar, “A lip sync expert is all you need for speech to lip generation in the wild,” in _Proc. of ACM MM_ , C. W. Chen, R. Cucchiara, X. Hua, G. Qi, E. Ricci, Z. Zhang, and R. Zimmermann, Eds., 2020, pp. 484–492.
  * [12] K. R. Prajwal, R. Mukhopadhyay, J. Philip, A. Jha, V. P. Namboodiri, and C. V. Jawahar, “Towards automatic face-to-face translation,” in _Proc. of ACM MM_ , L. Amsaleg, B. Huet, M. A. Larson, G. Gravier, H. Hung, C. Ngo, and W. T. Ooi, Eds., 2019, pp. 1428–1436.
  * [13] K. Vougioukas, S. Petridis, and M. Pantic, “Realistic speech-driven facial animation with gans,” _Int. J. Comput. Vis._ , vol. 128, no. 5, pp. 1398–1413, 2020.
  * [14] H. Zhou, Y. Sun, W. Wu, C. C. Loy, X. Wang, and Z. Liu, “Pose-controllable talking face generation by implicitly modularized audio-visual representation,” in _Proc. of CVPR_ , 2021, pp. 4176–4186.
  * [15] L. Chen, R. K. Maddox, Z. Duan, and C. Xu, “Hierarchical cross-modal talking face generation with dynamic pixel-wise loss,” in _Proc. of CVPR_ , 2019, pp. 7832–7841.
  * [16] L. Chen, G. Cui, C. Liu, Z. Li, Z. Kou, Y. Xu, and C. Xu, “Talking-head generation with rhythmic head motion,” in _Proc. of ECCV_ , A. Vedaldi, H. Bischof, T. Brox, and J. Frahm, Eds., vol. 12354, 2020, pp. 35–51.
  * [17] Y. Zhou, X. Han, E. Shechtman, J. Echevarria, E. Kalogerakis, and D. Li, “Makelttalk: speaker-aware talking-head animation,” _ACM Trans. Graph._ , vol. 39, no. 6, pp. 221:1–221:15, 2020.
  * [18] Y. Lu, J. Chai, and X. Cao, “Live speech portraits: real-time photorealistic talking-head animation,” _ACM Trans. Graph._ , vol. 40, no. 6, pp. 220:1–220:17, 2021.
  * [19] Z. Zhang, L. Li, Y. Ding, and C. Fan, “Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset,” in _Proc. of CVPR_ , 2021, pp. 3661–3670.
  * [20] W. Zhang, X. Cun, X. Wang, Y. Zhang, X. Shen, Y. Guo, Y. Shan, and F. Wang, “Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation,” in _Proc. of CVPR_ , 2023, pp. 8652–8661.
  * [21] Z. Zhang, Z. Hu, W. Deng, C. Fan, T. Lv, and Y. Ding, “Dinet: Deformation inpainting network for realistic face visually dubbing on high resolution video,” in _Proc. of AAAI_ , B. Williams, Y. Chen, and J. Neville, Eds., 2023, pp. 3543–3551.
  * [22] D. Cudeiro, T. Bolkart, C. Laidlaw, A. Ranjan, and M. J. Black, “Capture, learning, and synthesis of 3d speaking styles,” in _Proc. of CVPR_ , 2019, pp. 10 101–10 111.
  * [23] A. Richard, M. Zollhöfer, Y. Wen, F. D. la Torre, and Y. Sheikh, “Meshtalk: 3d face animation from speech using cross-modality disentanglement,” in _Proc. of ICCV_ , 2021, pp. 1153–1162.
  * [24] Y. Fan, Z. Lin, J. Saito, W. Wang, and T. Komura, “Faceformer: Speech-driven 3d facial animation with transformers,” in _Proc. of CVPR_ , 2022, pp. 18 749–18 758.
  * [25] J. Xing, M. Xia, Y. Zhang, X. Cun, J. Wang, and T. Wong, “Codetalker: Speech-driven 3d facial animation with discrete motion prior,” in _Proc. of CVPR_ , 2023, pp. 12 780–12 790.
  * [26] Y. Zhou, Z. Xu, C. Landreth, E. Kalogerakis, S. Maji, and K. Singh, “Visemenet: audio-driven animator-centric speech animation,” _ACM Trans. Graph._ , vol. 37, no. 4, p. 161, 2018.
  * [27] M. V. Aylagas, H. A. Leon, M. Teye, and K. Tollmar, “Voice2face: Audio-driven facial and tongue rig animations with cvaes,” _Comput. Graph. Forum_ , vol. 41, no. 8, pp. 255–265, 2022.
  * [28] S. Stan, K. I. Haque, and Z. Yumak, “Facediffuser: Speech-driven 3d facial animation synthesis using diffusion,” in _ACM Conference on Motion, Interaction and Games_ , J. Pettré, B. Solenthaler, R. McDonnell, and C. Peters, Eds., 2023, pp. 13:1–13:11.
  * [29] H. Yi, H. Liang, Y. Liu, Q. Cao, Y. Wen, T. Bolkart, D. Tao, and M. J. Black, “Generating holistic 3d human motion from speech,” in _Proc. of CVPR_ , 2023, pp. 469–480.
  * [30] T. Ao, Q. Gao, Y. Lou, B. Chen, and L. Liu, “Rhythmic gesticulator: Rhythm-aware co-speech gesture synthesis with hierarchical neural embeddings,” _ACM Trans. Graph._ , vol. 41, no. 6, pp. 209:1–209:19, 2022.
  * [31] S. Alexanderson, R. Nagy, J. Beskow, and G. E. Henter, “Listen, denoise, action! audio-driven motion synthesis with diffusion models,” _ACM Trans. Graph._ , vol. 42, no. 4, pp. 44:1–44:20, 2023.
  * [32] N. Athanasiou, M. Petrovich, M. J. Black, and G. Varol, “TEACH: temporal action composition for 3d humans,” in _Proc. of 3DV_ , 2022, pp. 414–423.
  * [33] Z. Zhou and B. Wang, “UDE: A unified driving engine for human motion generation,” in _Proc. of CVPR_ , 2023, pp. 5632–5641.
  * [34] J. Lin, J. Chang, L. Liu, G. Li, L. Lin, Q. Tian, and C. W. Chen, “Being comes from not-being: Open-vocabulary text-to-motion generation with wordless training,” in _Proc. of CVPR_ , 2023, pp. 23 222–23 231.
  * [35] Z. Wan, Z. Li, M. Tian, J. Liu, S. Yi, and H. Li, “Encoder-decoder with multi-level attention for 3d human shape and pose estimation,” in _Proc. of ICCV_ , 2021, pp. 13 013–13 022.
  * [36] Y. Yuan, S. Wei, T. Simon, K. Kitani, and J. M. Saragih, “Simpoe: Simulated character control for 3d human pose estimation,” in _Proc. of CVPR_ , 2021, pp. 7159–7169.
  * [37] W. Wei, J. Lin, T. Liu, and H. M. Liao, “Capturing humans in motion: Temporal-attentive 3d human pose and shape estimation from monocular video,” in _Proc. of CVPR_ , 2022, pp. 13 201–13 210.
  * [38] J. S. Chung and A. Zisserman, “Out of time: Automated lip sync in the wild,” in _Proc. of ACCV Workshop_ , C. Chen, J. Lu, and K. Ma, Eds., vol. 10117, 2016, pp. 251–263.
  * [39] P. Ma, S. Petridis, and M. Pantic, “Visual speech recognition for multiple languages in the wild,” _Nat. Mac. Intell._ , vol. 4, no. 11, pp. 930–939, 2022.
  * [40] T. Karras, T. Aila, S. Laine, A. Herva, and J. Lehtinen, “Audio-driven facial animation by joint end-to-end learning of pose and emotion,” _ACM Trans. Graph._ , vol. 36, no. 4, pp. 94:1–94:12, 2017.
  * [41] J. Liu, B. Hui, K. Li, Y. Liu, Y. Lai, Y. Zhang, Y. Liu, and J. Yang, “Geometry-guided dense perspective network for speech-driven facial animation,” _IEEE Trans. Vis. Comput. Graph._ , vol. 28, no. 12, pp. 4873–4886, 2022.
  * [42] A. van den Oord, O. Vinyals, and K. Kavukcuoglu, “Neural discrete representation learning,” in _Proc. of NIPS_ , I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, Eds., 2017, pp. 6306–6315.
  * [43] Z. Sun, T. Lv, S. Ye, M. G. Lin, J. Sheng, Y.-H. Wen, M. Yu, and Y. jin Liu, “Diffposetalk: Speech-driven stylistic 3d facial animation and head pose generation via diffusion models,” _arXiv preprint arXiv:2310.00434_ , 2023.
  * [44] S. Sanyal, T. Bolkart, H. Feng, and M. J. Black, “Learning to regress 3d face shape and expression from an image without 3d supervision,” in _Proc. of CVPR_ , 2019, pp. 7763–7772.
  * [45] Y. Deng, J. Yang, S. Xu, D. Chen, Y. Jia, and X. Tong, “Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set,” in _Proc. of CVPR Workshops_ , 2019, pp. 285–295.
  * [46] Y. Feng, H. Feng, M. J. Black, and T. Bolkart, “Learning an animatable detailed 3d face model from in-the-wild images,” _ACM Trans. Graph._ , vol. 40, no. 4, pp. 88:1–88:13, 2021.
  * [47] R. Danecek, M. J. Black, and T. Bolkart, “EMOCA: emotion driven monocular face capture and animation,” in _Proc. of CVPR_ , 2022, pp. 20 279–20 290.
  * [48] R. Liu, Y. Cheng, S. Huang, C. Li, and X. Cheng, “Transformer-based high-fidelity facial displacement completion for detailed 3d face reconstruction,” _IEEE Trans. on Multim._ , pp. 1–13, 2023.
  * [49] C. Cao, Q. Hou, and K. Zhou, “Displaced dynamic expression regression for real-time facial tracking and animation,” _ACM Trans. Graph._ , vol. 33, no. 4, pp. 43:1–43:10, 2014.
  * [50] C. Cao, D. Bradley, K. Zhou, and T. Beeler, “Real-time high-fidelity facial performance capture,” _ACM Trans. Graph._ , vol. 34, no. 4, pp. 46:1–46:9, 2015.
  * [51] C. Wang, F. Shi, S. Xia, and J. Chai, “Realtime 3d eye gaze animation using a single RGB camera,” _ACM Trans. Graph._ , vol. 35, no. 4, pp. 118:1–118:14, 2016.
  * [52] P. Garrido, M. Zollhöfer, D. Casas, L. Valgaerts, K. Varanasi, P. Pérez, and C. Theobalt, “Reconstruction of personalized 3d face rigs from monocular video,” _ACM Trans. Graph._ , vol. 35, no. 3, pp. 28:1–28:15, 2016.
  * [53] P. P. Filntisis, G. Retsinas, F. P. Papantoniou, A. Katsamanis, A. Roussos, and P. Maragos, “Visual speech-aware perceptual 3d facial expression reconstruction from videos,” _arXiv preprint arXiv:2207.11094_ , 2022.
  * [54] V. Blanz and T. Vetter, “A morphable model for the synthesis of 3d faces,” in _Proc. of SIGGRAPH_ , 1999, pp. 187–194.
  * [55] D. Vlasic, M. Brand, H. Pfister, and J. Popovic, “Face transfer with multilinear models,” _ACM Trans. Graph._ , vol. 24, no. 3, pp. 426–433, 2005.
  * [56] Z. Wang, J. Chai, and S. Xia, “Realtime and accurate 3d eye gaze capture with dcnn-based iris and pupil segmentation,” _IEEE Trans. Vis. Comput. Graph._ , vol. 27, no. 1, pp. 190–203, 2021.
  * [57] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in _Proc. of NIPS_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., 2020.
  * [58] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Proc. of NIPS_ , I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, Eds., 2017, pp. 5998–6008.
  * [59] S. Wang, L. Li, Y. Ding, and X. Yu, “One-shot talking face generation from single-speaker audio-visual correlation learning,” in _Proc. of AAAI_ , 2022, pp. 2531–2539.
  * [60] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” in _Proc. of NIPS_ , I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, Eds., 2017, pp. 6626–6637.
  * [61] N. D. Narvekar and L. J. Karam, “A no-reference image blur metric based on the cumulative probability of blur detection (CPBD),” _IEEE Trans. Image Process._ , vol. 20, no. 9, pp. 2678–2683, 2011.
  * [62] A. Gulati, J. Qin, C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in _Proc. of Interspeech_ , H. Meng, B. Xu, and T. F. Zheng, Eds., 2020, pp. 5036–5040.
  * [63] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” in _Proc. of ICLR_ , Y. Bengio and Y. LeCun, Eds., 2014.
  * [64] G. Fanelli, J. Gall, H. Romsdorfer, T. Weise, and L. V. Gool, “A 3-d audio-visual corpus of affective communication,” _IEEE Trans. Multim._ , vol. 12, no. 6, pp. 591–598, 2010.
  * [65] T. Li, T. Bolkart, M. J. Black, H. Li, and J. Romero, “Learning a model of facial shape and expression from 4d scans,” _ACM Trans. Graph._ , vol. 36, no. 6, pp. 194:1–194:17, 2017.
  * [66] P. Ma, A. Haliassos, A. Fernandez-Lopez, H. Chen, S. Petridis, and M. Pantic, “Auto-avsr: Audio-visual speech recognition with automatic labels,” in _Proc. of ICASSP_ , 2023, pp. 1–5.
  * [67] G. Chen and W. Wang, “A survey on 3d gaussian splatting,” _arXiv preprint arXiv:2401.03890_ , 2024.
  * [68] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” in _Proc. of ECCV_ , A. Vedaldi, H. Bischof, T. Brox, and J. Frahm, Eds., vol. 12346, 2020, pp. 405–421.
  * [69] S. Qian, T. Kirschstein, L. Schoneveld, D. Davoli, S. Giebenhain, and M. Nießner, “Gaussianavatars: Photorealistic head avatars with rigged 3d gaussians,” _arXiv preprint arXiv:2312.02069_ , 2023.
  * [70] Y. Xu, B. Chen, Z. Li, H. Zhang, L. Wang, Z. Zheng, and Y. Liu, “Gaussian head avatar: Ultra high-fidelity head avatar via dynamic gaussians,” _arXiv preprint arXiv:2312.03029_ , 2023.
  * [71] Z. Li, Z. Zheng, L. Wang, and Y. Liu, “Animatable gaussians: Learning pose-dependent gaussian maps for high-fidelity human avatar modeling,” _arXiv preprint arXiv:2311.16096_ , 2023.
  * [72] Y. Jiang, Q. Liao, X. Li, L. Ma, Q. Zhang, C. Zhang, Z. Lu, and Y. Shan, “Uv gaussians: Joint learning of mesh deformation and gaussian textures for human avatar modeling,” _arXiv preprint arXiv:2403.11589_ , 2024.