# DiffusionTalker: Personalization and Acceleration for Speech-Driven 3D Face
Diffuser

Peng Chen1,2∗, Xiaobao Wei1,2∗, Ming Lu3, Yitong Zhu1,2,
Naiming Yao1, Xingyu Xiao4, Hui Chen1†
1Institute of Software, Chinese Academy of Sciences
2University of Chinese Academy of Sciences
3Intel Labs China 4Tsinghua University
<EMAIL_ADDRESS>
###### Abstract

Speech-driven 3D facial animation has been an attractive task in both academia
and industry. Traditional methods mostly focus on learning a deterministic
mapping from speech to animation. Recent approaches start to consider the non-
deterministic fact of speech-driven 3D face animation and employ the diffusion
model for the task. However, personalizing facial animation and accelerating
animation generation are still two major limitations of existing diffusion-
based methods. To address the above limitations, we propose DiffusionTalker, a
diffusion-based method that utilizes contrastive learning to personalize 3D
facial animation and knowledge distillation to accelerate 3D animation
generation. Specifically, to enable personalization, we introduce a learnable
talking identity to aggregate knowledge in audio sequences. The proposed
identity embeddings extract customized facial cues across different people in
a contrastive learning manner. During inference, users can obtain personalized
facial animation based on input audio, reflecting a specific talking style.
With a trained diffusion model with hundreds of steps, we distill it into a
lightweight model with 8 steps for acceleration. Extensive experiments are
conducted to demonstrate that our method outperforms state-of-the-art methods.
The code will be released. We have provided the supplementary video:
https://chenvoid.github.io/DiffusionTalker/

11footnotetext: Equal Contribution.22footnotetext: Corresponding Author.

##  1 Introduction

Figure 1: DiffusionTalker. We reduce the steps of the diffusion model for
faster inference by knowledge distillation. Based on the model with fewer
steps, given an audio sequence, we can find a matching identity embedding in
the identity library to personalize the speaker’s talking style.

Speech-driven 3D facial animation is a crucial task in virtual reality [46],
augmented reality [8] and computer gaming [32, 10, 49] applications. To
achieve high-fidelity facial animation, animators need to iteratively adjust
facial parameters to fit target performers. With the development of deep
learning, end-to-end speech-driven facial animation synthesis has been widely
explored. Recent works focus on deep learning-based 3D facial animation, which
can produce more vivid animation and facilitate cost reduction. Based on the
convolution neural network [9, 35] or transformer [45, 11, 30], these works
try to learn a mapping between audio input and mesh or blendshape [22] output.
Traditional methods mainly focus on learning a deterministic mapping from
speech to animation [9, 35, 11, 30]. Recently, the diffusion model has shown
promising potential in learning non-deterministic mapping [16, 41, 36, 42]. In
other words, when strictly controlling lip movements [40], the upper face,
which has weak correlation with speech, will exhibit a greater diversity of
animations. It is the inherent non-deterministic facial cues dispersed across
the face.

However, recent methods still have two major limitations. First, they suffer
from a prolonged inference time [35, 30, 42], which constitutes a critical
limitation in terms of engineering applications. Secondly, in real life,
people can recognize a familiar person simply by listening to his voice,
discerning his identity based on unique vocal characteristics without seeing
his face. Therefore, different individuals exhibit distinct speaking
characteristics [1, 33], indicating the importance of personalization for
speech-driven 3D facial animation.

To address these issues, we propose a novel diffusion-based approach named
DiffusionTalker, for producing high-quality 3D facial animations based on the
user’s speaking characteristics in a short time, as shown in Fig. 1. For the
personalization capacity, we introduce novel identity embeddings corresponding
to the speaking characteristics of the training people. We then train the
identity encoder to encode embeddings to identity features, which respond to
audio features in a contrastive learning manner [34, 44]. For the synthesis
acceleration, we utilize knowledge distillation [39] to distill knowledge from
teacher DiffusionTalker with enormous steps to student DiffusionTalker with
few steps. To leverage dense information from dense sampling steps, we obtain
student DiffusionTalker with a progressive training scheme. During inference,
with the voice from a new performer, DiffusionTalker first extracts features
from the voice and searches for the best corresponding identity embedding.
Then, all features and the noise are concatenated and sent to the GRU-based
[7] talker decoder for denoising. In such an end-to-end learning pipeline,
DiffusionTalker can generate high-fidelity 3D facial animation efficiently.
Our main contributions are summarized as follows:

  * •

We introduce DiffusionTalker, an end-to-end neural network for generating
personalized speech-driven 3D facial animation based on Denoising Diffusion
Probabilistic Model (DDPM) [16]. We first apply contrastive learning to get
personalized animation, by pulling encoded audio features and learnable
identity embeddings together while pushing the mismatched farther apart.

  * •

To reduce the time cost and maintain high-fidelity generation, we utilize
knowledge distillation for DiffusionTalker. After the distillation process,
DiffusionTalker generates animation efficiently and obtains 65.5 times
acceleration with only 8 time steps.

  * •

Experimental results show that our DiffusionTalker outperforms the state-of-
the-art methods with 0.0969 average lip error while maintaining adequate
diversities.

##  2 Related Work

Speech-driven 3D Facial Animation. Methods for speech-driven 3D facial
animation can be divided into two branches: phoneme-based methods [3] and
data-driven based methods [31, 43, 18]. Phoneme-based methods, such as JALI
[10, 49], come with the advantage of easy integration into other artists’
pipelines. However, they require intermediary representations of phonemes for
co-articulation. Different from phoneme-based methods, data-driven based
methods can learn the mapping between audio and facial animation
automatically. VOCA [9] is the first work that uses CNN to map voice to 3D
mesh. They also propose a one-hot-style embedding for a controllable output.
MeshTalk [35] tries to disentanglement the audio and 3D animation in a
categorical latent space, but consumes longer time with less than 100ms of
audio as input. FaceFormer [11] is the first work to utilize an autoregressive
transformer as the model backbone and applies the attention mechanism to
extract audio context features. EmoTalk [30] focuses on integrating emotion
into FaceFormer for better facial animation with a cross-attention module
[45]. Upon methods, all learn a deterministic mapping between audio and facial
animation, while neglecting the non-deterministic nature. FaceDiffuser [42]
first attempts to apply the diffusion model to the task, but lacks the ability
to personalized output. To achieve personalized facial animation,
DiffusionTalker utilizes contrastive learning-based identity embeddings to
store personal information.

Personalization with Diffusion Model. Personalization is a crucial technique
in the research field of diffusion models. One notable approach is DreamBooth
[38]. This method presents a novel pathway for the personalization of text-to-
image diffusion models. Given a few images of a particular subject, the pre-
trained text-to-image model can be fine-tuned to associate a unique identifier
with that subject. Kumari et al. [21] also find that optimizing a few
parameters in diffusion models can represent new concepts of the given images.
They further propose to combine multiple concepts via closed-form constrained
optimization. Liu et al.[25] find that a small portion of the neurons can
correspond to a particular subject. They use the statistics of network
gradients to identify the neurons. Concatenating multiple clusters of concept
neurons can vividly generate all related concepts in a single image. Han et
al.[14] propose to fine-tune the singular values of the weight matrices,
leading to a compact and efficient parameter space for personalization. In our
work, we integrated the contrastive learning [34] for matching of audio and
identity modalities into the diffusion model training to achieve
personalization of speaking styles based on input audio.

Figure 2: Pipeline of DiffusionTalker. DiffusionTalker continuously removes
Gaussian noise from noise-added facial animation during the Denoise Process
while updating the model’s parameters to generate facial animation based on
input speech. In each step, the model consists of two parts: the
personalization adapter, which uses contrastive learning to match speech and
identity features, and the forward procedure, which adds noise to the facial
parameters. Finally, all the information is fed into the talker decoder to
predict facial animation. DiffusionTalker employs a training approach which is
knowledge distillation and reduces the number of sampling steps by half.

Distillation for Diffusion Acceleration. Diffusion models have demonstrated
excellent potential for content generation [36, 42, 5, 19], and outperforming
GANs [12] and autoregressive models [2, 29]. However, they suffer from slow
generation due to iterative denoising. Previous studies have suggested the
application of knowledge distillation to improve inference speed [15].
Specifically, a faster student model can be trained to replicate the output of
a pretrained teacher diffusion model. Meng et al.[26] first train a student
model to mimic the outputs of a pre-trained model, followed by a progressive
distillation. Li et al.[23] introduce an optimized UNet architecture [37, 28]
that identifies and eliminates redundancies within the original model. It
employs data distillation to streamline computational tasks in the image
decoder and improves the efficiency of step distillation by incorporating
advanced training strategies and regularization informed by classifier-free
guidance. BOOT [13] circumvents the dependency on extensive offline
computation using a data-free distillation algorithm, significantly enhancing
the efficiency of synthetic data generation. Salimans et al. [39] propose a
progressive distillation approach, halving the number of required sampling
steps each time. We adopt this approach in our work, using a pre-trained
large-step DiffusionTalker as a teacher model to distill a small-step student
model. The aim is to accelerate the inference while also enhancing the
generalization ability of the student model through knowledge distillation.

##  3 Methodology

In this section, we first briefly review the diffusion model. Subsequently, we
proceed to elaborate on the pipeline of the proposed DiffusionTalker. Finally,
we further elucidate the novel designs introduced in DiffusionTalker.

###  3.1 Preliminary

Denoising Diffusion Probabilistic Models (DDPMs) [16] have become a pivotal
element in content generation [4, 27]. The fundamental function of DDPMs is to
learn the distribution of training data and generate images that closely match
this distribution. The DDPM process consists of two stages: the diffusion
process and the denoise process. In the diffusion process, noise is added
incrementally to the image, gradually transforming it into pure noise. The
diffusion process can be characterized by a series of variance variables
$\alpha_{1},\alpha_{2},\ldots,\alpha_{T}$, which scale the data distribution
at each step $t$ through the diffusion sequence. The latent variable at any
step $t$ can be expressed as a function of the initial data $\bm{x}_{0}$ and
noise $\bm{\epsilon}$, such that:

$\bm{x}_{t}=\sqrt{\bar{\alpha_{t}}}\bm{x}_{0}+\sqrt{1-\bar{\alpha_{t}}}\bm{\epsilon}$
(1)

where $\bm{x}_{0}$ denotes the initial data distribution and $\bm{x}_{t}$
represents the denoised image at step $t$. $\bm{\epsilon}$ is noise sampled
from a standard Gaussian distribution $\mathcal{N}(0,I)$.
$q(\bm{x}_{t}|\bm{x}_{t-1})$ is the conditional distribution of $\bm{x}_{t}$
given $\bm{x}_{t-1}$, which describes the process of adding Gaussian noise to
$\bm{x}_{t-1}$, resulting in a noisier latent variable $\bm{x}_{t}$ at each
step. The variables $\beta_{1},\beta_{2},\ldots,\beta_{T}$ correspond to the
noise levels added at each step of the diffusion process. These variables are
related to the variance variables through the relationships as follows.

$\alpha_{t}=1-\beta_{t}$ (2)
$\bar{\alpha_{t}}=\alpha_{1}\alpha_{2}\ldots\alpha_{t}$ (3)

The denoise process is designed to reverse the diffusion process by predicting
$\bm{x}_{t-1}$ from $\bm{x}_{t}$. The goal is to approximate the true data
distribution $p(\bm{x}_{t-1}|\bm{x}_{t})$ starting from the noise
distribution. By optimizing the parameters $\theta$, the denoise process uses
a parameterized model $p_{\theta}(\bm{x}_{t-1}|\bm{x}_{t})$ to closely
approximate the true conditional distribution $p(\bm{x}_{t-1}|\bm{x}_{t})$,
and ultimately, the data distribution $p(\bm{x})$.

During DDPM training, the objective at each step is to accurately predict the
noise sampled from a Gaussian distribution. Consequently, the training loss
can be formulated as follows:

$loss=||\bm{\epsilon}-\epsilon_{\theta}(\sqrt{\bar{\alpha_{t}}}\bm{x}_{0}+\sqrt{1-\bar{\alpha_{t}}}\bm{\epsilon},t)||^{2}$
(4)

where $\bm{\epsilon}$ is Gaussian noise, $\epsilon_{\theta}$ is the model that
predicts t-step Gaussian noise and $\theta$ is the parameters of it.

During inference, $\bm{x}_{t-1}$ can be represented as follows:

$\bm{x}_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(\bm{x}_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\alpha_{t}}}\epsilon_{\theta}(\bm{x}_{t},t)\right)+\sigma_{t}\mathbf{z}$
(5)

where $\sigma_{t}\mathbf{z}$ is a random noise term to generate diversity.

###  3.2 Overall Pipeline

DiffusionTalker is trained based on the DDPM, utilizing audio and identity as
conditions to guide the denoise process of facial animation synthesis, thereby
generating speech-driven 3D facial animation, as shown in Fig. 2. The specific
formula employed in this process is as follows:

$\bm{\hat{x}}=DiffusionTalker_{\theta}(\bm{a},\bm{i},\bm{x}_{t},t)$ (6)

where $\theta$ indicates model parameters in DiffsionTalker, $\bm{a}$ is the
input audio sequence, $\bm{i}$ is the identity embedding from the identity
embedding library, $\bm{x}_{t}$ is $\bm{x}_{0}$ after t steps of the DDPM
noise addition process and $\bm{\hat{x}}$ is the predicted facial animation
which will be fitted to the ground truth $\bm{x}_{0}$.

In the denoise process of DiffusionTalker, Gaussian noise is continuously
removed from noise-added facial animation, and model parameters are updated to
predict facial animation based on speech. Each step in this process includes
two key components: the personalization adapter and the forward procedure. The
former includes an identity embedding library, where each embedding
corresponds to an audio sequence. The identity embedding and the audio
sequence are encoded using an identity encoder and an audio encoder,
respectively. Contrastive learning [47, 6, 44] is then applied to match the
features of both, allowing unknown input audio to find a similar identity
embedding in the identity library, thus achieving personalization of the
talking style during inference. The identity embedding library can be
continuously enriched and expanded by fine-tuning DiffusionTalker. The forward
procedure involves the noise addition process, where Gaussian noise is added
to the ground truth for $t$ steps, with each step encoded by a step encoder.
Finally, all components are fed into the talker decoder to predict facial
animation parameters. To accelerate inference speed, we trained
DiffusionTalker using knowledge distillation. This approach allows distilling
a teacher model with $2n$ steps into a student model with $n$ steps,
accelerating the speed of speech-driven 3D face animation synthesis. We will
introduce the modules in the following sections.

###  3.3 Personalization Adapter

Figure 3: Personalization Adapter. During the training process, the
personalization adapter updates the identity embedding library, whereas the
inference process involves searching for the identity embedding within this
library.

To bridge the gap between audio and talking identity matching, we propose the
personalization adapter by contrastive learning, ultimately achieving the
personalization of identity based on speech during inference. In the
personalization adapter, we have constructed an identity embedding library
based on the speakers, with each speaker corresponding to an identity
embedding. We have configured these embeddings as trainable parameters during
the training process. Both the audio sequence and its corresponding identity
embedding are fed into the network as a training pair. The former is input
into a pre-trained HuBERT model [17] serving as the audio encoder to extract
audio features, while the latter is received by an identity encoder with an
MLP network to generate identity features. These identity embeddings and audio
features are further passed on to the next layer of the network. Identity
features and audio features are used to train the network with contrastive
learning.

Regarding contrastive learning, we draw inspiration from [34]. However, in our
approach, we have fixed the parameters of the audio encoder and only trained
the identity encoder and learnable identity embeddings. The objective is to
predict the correct relationship between identity embeddings and audio
sequences. In terms of the specific details of contrastive learning, as shown
in Fig. 3, we first apply L2 normalization to both the identity embeddings and
audio features to normalize vector lengths, preserving only directional
features. Secondly, we perform matrix multiplication on the normalized
representations of both, and the resulting output is used to compute the
cross-entropy loss with the labels. This loss is then used to update the
parameters of the personalization adapter.

During inference, a given audio sequence is input into the audio encoder,
generating an audio feature. This feature is then matrix-multiplied with the
features of all embeddings in the identity embedding library. The embedding
with the highest similarity is identified as the talking identity matching the
input audio sequence.

###  3.4 Knowledge Distillation

To accelerate the inference of DiffusionTalker, we utilize knowledge
distillation for fewer denoise steps. Inspired by progressive distillation
[39], we introduce a student model $\hat{s}_{\eta}(\bm{w}_{t})$ with $N$ steps
to match the pre-trained teacher model $\hat{t}_{\theta}(\bm{w}_{t})$ with
$2N$ steps. $\bm{w}_{t}$ is defined as follows:

$\bm{w}_{t}=(\bm{a},\bm{i},\bm{x}_{t},t)$ (7)

where $\bm{a}$ is the audio sequence, $\bm{i}$ is the identity embedding
corresponding to $\bm{a}$, and $t$ is time step. We first initialize the
student model with a copy of the teacher model. Then we sample data and add
noise to it according to the student DDPM $N$ steps. Before applying the
student model $\hat{s}_{\eta}(\bm{w}_{t})$ to denoise, we utilize the teacher
model $\hat{t}_{\theta}(\bm{w}_{t})$ to calculate the denoise output of $2N$
and $2N-1$ steps. To obtain a progressive distillation, we make one student
DDPM step $N$ to match two teacher DDPM steps $2N$ and $2N-1$.

Assume that the signal-to-noise ratio at time steps $\tau$ is
$\alpha^{2}_{\tau}/\sigma^{2}_{\tau}$, then the corresponding data
$\bm{x}_{0}$ added with noise $\bm{\epsilon}$ is
$\bm{x}_{\tau}=\alpha_{\tau}\bm{x}_{0}+\sigma_{\tau}\bm{\epsilon},\bm{\epsilon}\sim
N(0,I)$. For the teacher model, we can obtain noisy data
$\bm{x}_{\tau^{\prime}}$ at time step $\tau^{\prime}=2\tau$ as the following:

$\bm{x}_{\tau^{\prime}}=\alpha_{\tau^{\prime}}\hat{t}_{\theta}(\bm{w}_{\tau})+\frac{\sigma_{\tau^{\prime}}}{\sigma_{\tau}}(\bm{x}_{\tau}-\alpha_{\tau}\hat{t}_{\theta}(\bm{w}_{\tau}))$
(8)

For $\tau^{\prime\prime}=2\tau-1$, noisy data $\bm{x}_{\tau^{\prime\prime}}$
can also be calculated with the above equation. As illustration in [39], the
final target $\widetilde{\bm{x}}$ from the teacher model can be formulated as:

$\widetilde{\bm{x}}=\frac{\bm{x}_{\tau^{\prime\prime}}-(\sigma_{\tau^{\prime\prime}}/\sigma_{\tau})\bm{x}_{\tau}}{\alpha_{\tau^{\prime\prime}}-(\sigma_{\tau^{\prime\prime}}/\sigma_{\tau})\alpha_{\tau}}$
(9)

We apply $\widetilde{\bm{x}}$ to supervise the student model along with other
losses until convergence. Then we treat the student model with $N$ steps as
the new teacher model and repeat the same progressive knowledge distillation
process. Finally, DiffusionTalker can be distilled into several steps while
maintaining high-generation quality.

###  3.5 Training and Inference

####  3.5.1 Training

In our training process, we randomly select a time step $t$ from a uniform
distribution ranging from 1 to $T$. We then add noise to $\bm{x}_{0}$ for $t$
steps to obtain $\bm{x}_{t}$. Audio-identity training pairs are fed into the
audio encoder and the identity encoder respectively to extract features. We
conduct contrastive learning between the identity feature and the audio
feature. The step $t$ is input into the step encoder to obtain a step
embedding, which is then aligned with other features. Subsequently, the
identity embedding, audio feature, step embedding, and $\bm{x}_{t}$ are
concatenated and input into the talker decoder for denoising, producing
$\hat{\bm{x}}$. The model parameters are optimized based on the L2 loss
between $\hat{\bm{x}}$ and $\bm{x}_{0}$. The trainable components of this
process include the identity embeddings, identity encoder, step encoder, and
talker decoder, while the parameters of the audio encoder remain fixed. We
propose five types of loss functions, namely, expression loss, lip loss,
contrast loss, velocity loss, and distillation loss, each specifically
targeted toward different constraint objectives.

During the training process of the teacher model, the overall loss is
formulated as:

$L_{tea}=\lambda_{1}L_{exp}+\lambda_{2}L_{lip}+\lambda_{3}L_{con}+\lambda_{4}L_{vel}$
(10)

During the knowledge distillation process of the student model, the overall
loss is formulated as:

$L_{stu}=L_{tea}+\lambda_{5}L_{dis}$ (11)

where $\lambda_{1}$ = 1.0, $\lambda_{2}$ = 1.0, $\lambda_{3}$ = 0.007,
$\lambda_{4}$ = 0.5, $\lambda_{5}$ = 0.1 are fixed in all our experiments. The
formulation of these loss functions will be explained below:

Expression Loss. Expression loss is a constraint applied to all facial
parameters, and its definition is as follows:

$L_{exp}=\|\bm{\hat{x}}-\bm{x}_{0}\|^{2}$ (12)

where $\bm{\hat{x}}$ is the prediction of facial parameters, $\bm{x}_{0}$ is
the ground truth of animation.

Lip Loss. In order to improve the accuracy of lip movement, we selected
parameters related to the lips and applied lip loss to constrain these
parameters.

$L_{lip}=\|\bm{\hat{x}}^{\prime}-\bm{x}^{\prime}_{0}\|^{2}$ (13)

where $\bm{\hat{x}}^{\prime}$ is the predicted parameters of lip area,
$\bm{x}^{\prime}_{0}$ is the label of lip animation.

Contrast Loss. Contrast loss, as a target function for contrastive learning,
is utilized to achieve the personalization of the talking identity. Its
definition is as follows:

$L_{con}=-\frac{1}{M}\sum_{i=1}^{M}\log\frac{\sum_{t=0}^{m}e^{h_{i}^{\top}h_{t}/\tau}}{\sum_{j=0}^{M}e^{h_{i}^{\top}h_{j}/\tau}},$
(14)

where $h$ denotes the samples involved in contrastive learning, $M$ is the
number of samples, $m$ is the number of positive samples corresponding to
$i^{th}$ sample $h_{i}$, and $\tau$ is a temperature hyper-parameter.

Velocity Loss. We add the velocity loss from EmoTalk to enhance the smoothness
between predicted animation frames. It is defined as follows:

$L_{vel}=\sum_{i=2}^{n}\|(\bm{\hat{f}}^{i}-\bm{\hat{f}}^{i-1})-(\bm{f}_{0}^{i}-\bm{f}_{0}^{i-1})\|^{2}$
(15)

where $n$ is the number of total animation frames, and $\bm{f}$ is the
animation parameters for a single frame.

Distill Loss. We employ distill loss as the objective function for knowledge
distillation, and its definition is as follows:

$L_{dis}=max(\frac{\alpha^{2}_{\tau}}{\sigma^{2}_{\tau}},1)\|\widetilde{\bm{x}}-\hat{s}_{\eta}(\bm{w}_{\tau})\|^{2}$
(16)

####  3.5.2 Inference

During the inference process, an input audio sequence will find its matched
identity embedding in the personalization adapter, which serves as the talking
identity. This embedding, along with the audio feature and other intermediate
features, is concatenated and input into the talker decoder for denoising.
After $N$ steps of denoising, the final facial parameters are predicted. This
process, due to the inclusion of the talking identity, achieves user
personalization.

##  4 Experiments

###  4.1 Experimental Settings

Datasets. In the field of speech-driven 3D facial animation, the two most
common approaches are blendshape-based facial animation [22] and vertex-based
facial animation. Correspondingly, the datasets used are also categorized into
blendshape-based and vertex-based datasets. Motivated by the objective of
speed enhancement in DiffusionTalker, we trained our DiffusionTalker on the
blendshape-based BEAT dataset [24] for its fewer parameters. BEAT are
collected by Apple ARKit with 52 blendshape coefficients. It consists of voice
recordings and corresponding blendshape-based facial animations from 30
different individuals. Each person says the same content but with their unique
speaking style.We utilize a subset with about 32 hours of audio data from the
full dataset. We further segmented the audio data into 11,427 sequences, each
with a duration of 10 seconds. We follow the data preprocessing in
FaceDiffuser [42].

We also employed VOCASET [9] as a zero-shot test set to more effectively
evaluate the robustness of DiffusionTalker. VOCASET, a vertex-based dataset,
comprises 480 facial animation sequences and the corresponding speech data,
featuring 12 subjects. Each facial animation sequence consists of 5,023
vertices, multiplied by 3 to represent spatial coordinates in the $x$, $y$,
and $z$ dimensions. The duration of each speech data segment is approximately
3-4 seconds. For testing, we adopted the Blendshape conversion method from
EmoTalk [30], enabling the transformation of VOCASET into 52 blendshape
coefficients.

Implementation Details. We follow the implementation details specified in
FaceDiffuser [42]. For model architectures, we utilize the pretrained HuBERT
[17] as an audio encoder to extract audio features. The dimension of identity
embeddings is set to 32 and is sent to the identity encoder. We further
concatenate the encoded features and employ a two-layer GRU. The dimension of
the hidden layer is set as 256. To train the whole pipeline, we utilize the
Adam optimizer [20] with $(\beta_{1},\beta_{2})=(0.9,0.999)$ to update
parameters, and the learning rate is set to $1e^{-4}$. We set a linear $\beta$
schedule for 32 steps for teacher models. We first train the teacher model
with 50 epochs. Then we copy the weights of the teacher model into the student
model and apply the proposed knowledge distillation techniques to train the
student model with half-time steps. We iteratively set the student model as
the teacher model to further reduce sampling time steps. All experiments are
conducted on NVIDIA V100 GPUs. More implementation details are reported in the
supplementary material.

Evaluation Metrics. To quantitatively assess the performance and robustness of
DiffusionTalker, we employed four key evaluation metrics: MBE, LBE, FDD, and
ITF.

(MBE) Mean Blendshape Error measures the average Euclidean distance between
the predicted blendshape coefficients and the ground truth. Lower MBE values
indicate higher accuracy, as they represent smaller deviations from the ground
truth.

(LBE) Lip Blendshape Error measures the discrepancy between the predicted and
ground truth blendshape coefficients associated with the lip region.

(FDD) Facial Dynamics Deviation [48] measures the deviation in the dynamics of
the upper face region between the generated sequence and the ground truth.

(ITF) Inference Time per Frame is defined as the inference time per audio
frame, calculated by dividing the total inference time by the total number of
audio frames. This metric is employed to quantitatively measure and compare
the inference speed of models.

###  4.2 Experimental Results

####  4.2.1 Quantitative Evaluation

We trained six DiffusionTalker models from scratch on the BEAT dataset, with
steps set covering 256, 128, 64, 32, 16, and 8 respectively. Subsequently, we
employed the knowledge distillation approach where a 32-step model served as
the teacher model to train a 16-step student model. This 16-step student model
was then used as a new teacher to distill an 8-step student model.

In our quantitative experiments, we conduct three sets of tests. In the first
set, we assess and compare the performance metrics of all nine models, ranging
from 256 to 8 steps. In the second set, we retrain FaceDiffuser on the BEAT
dataset. EmoTalk, FaceDiffuser, and the 8-step distilled DiffusionTalker are
tested on BEAT, and metrics such as MBE, LBE, FDD, and ITF are reported. The
reason for selecting EmoTalk and FaceDiffuser as baselines is that other
methods do not release the training code on blendshape-based datasets. In the
third set, we retrain FaceDiffuser on the VOCASET and conduct an evaluation.
We use the test results from EmoTalk’s paper, including methods like
FaceFormer and EmoTalk on VOCASET. Our 8-step distilled DiffusionTalker
trained on BEAT is directly tested on VOCASET and compared with other methods.
In the third set, We focus on lip accuracy and inference time, hence the LVE
and ITF metrics are reported.

Table 1: The performance and inference speed of our model vary with different
step counts. Specifically, the inference speed of the 8-step model is enhanced
when using NVIDIA TensorRT.

Dataset | Steps | Distilled | LBE$\downarrow$ |  $\|$FDD $\|$$\downarrow$ | ITF($10^{-4}$s)$\downarrow$
---|---|---|---|---|---
BEAT | 256 | ✗ | 0.1050 | 0.1693 | 255.5
128 | ✗ | 0.1059 | 0.1445 | 132.4
64 | ✗ | 0.1063 | 0.1517 | 65.3
32 | ✗ | 0.1053 | 0.1647 | 32.2
16 | ✗ | 0.1097 | 0.1656 | 16.8
8 | ✗ | 0.1177 | 0.1709 | 7.0
16 | ✓ | 0.0957 | 0.1433 | 16.8
8 | ✓ | 0.0969 | 0.1558 | 7.1
8(TRT) | ✓ | 0.0975 | 0.1563 | 3.9

Table 2: The quantitative results evaluated on the 3D blendshape dataset.
Various models were tested on the BEAT dataset, with the best results
highlighted in bold. Among them, FaceDiffuser is a 1000-step model. Our model
demonstrates superior performance in MBE, LBE, and ITF, except for the FDD
metric.

Dataset | Method | MBE $\downarrow$ | LBE $\downarrow$ |  $\|$FDD $\|$$\downarrow$ | ITF($10^{-4}$s)$\downarrow$
---|---|---|---|---|---
| EmoTalk | 1.0537 | 0.2628 | 0.0751 | 9.3
BEAT | FaceDiffuser | 0.4332 | 0.1071 | 0.1786 | 1000.9
FaceDiffuser-8 | 0.8372 | 0.2115 | 0.0589 | 8.2
Ours-8 | 0.4094 | 0.0969 | 0.1558 | 7.1

Table 3: The quantitative results evaluated on the 3D vertex dataset. Various
models were tested on the VOCASET dataset, with the best results highlighted
in bold. Our model, without being trained on the VOCASET dataset, demonstrated
optimal performance in terms of the LVE and ITF metrics under the zero-shot
condition.

Dataset | Method | LVE $\downarrow$ | ITF($10^{-4}$s)$\downarrow$ | Zero-Shot
---|---|---|---|---
VOCASET | FaceFormer | 4.418 | 54.2 | ✗
EmoTalk | 4.134 | 9.3 | ✓
FaceDiffuser-8 | 4.713 | 8.2 | ✗
Ours-8 | 4.054 | 7.1 | ✓

Figure 4: ITF vs. Steps. The inference speeds of the model with and without
NVIDIA TensorRT vary under different step counts.

Inference Speed. As shown in Tab. 1, the ITF decreases as the number of steps
reduces, indicating that a decrease in steps leads to an increase in the
inference speed. This observation aligns with our general understanding.
Consequently, we plotted ITF against the number of steps, as depicted in Fig.
4, which demonstrates a direct proportional relationship between ITF and
steps, further indicating an inverse relationship between model inference
speed and the number of steps. The last row of Tab. 1 highlights that upon
integrating the 8-step distilled model with NVIDIA TensorRT, the ITF decreased
from $7.1\times 10^{-4}s$ to $3.9\times 10^{-4}s$, resulting in an 82.1%
increase in inference speed.

We compared the 8-step distilled model on the BEAT and VOCASET test sets with
other methods, as illustrated in Tab. 2 and 3. The model exhibited the best
ITF metric in both cases, indicating the fastest inference speed. Our method
achieves a similar speed compared with FaceDiffuser-8, however, our
quantitative results are much better thanks to the personalization ability.

Fidelity and Accuracy. Lip movement precision is a crucial factor for
evaluating fidelity and accuracy in the speech-driven 3D facial animation
field. As indicated in Tab. 1, without knowledge distillation, the LBE and FDD
metrics slightly increase as the number of steps decreases. This suggests that
reducing the steps in a diffusion model can lead to a decrease in accuracy.
During the transition from 16 to 8 steps with knowledge distillation, this
trend persists. However, models obtained through knowledge distillation at
identical step counts (16 or 8) significantly outperform their non-distilled
counterparts in terms of LBE and FDD. This implies that models derived via
knowledge distillation exhibit enhanced accuracy. This trend can be attributed
to (1) a reduction in the model’s steps, leading to fewer sampling steps and
potentially inadequate de-noising, affecting precision; (2) during knowledge
distillation, the student model learns soft labels from the teacher model,
encompassing the latter’s understanding of the data, which might aid the
student model in better generalizing to unseen data; (3) using the soft
outputs of the teacher model as targets may provide a regularizing effect,
helping to prevent overfitting in the student model.

Tab. 2 and 3 demonstrate that our approach achieves state-of-the-art
performance in MBE and LBE metrics, both on BEAT and VOCASET. However, our
model exhibits suboptimal performance in the FDD metric. This suggests a
weaker inference capability in terms of facial dynamics deviation,
particularly in the upper face region. This limitation could stem from the
model’s excessive focus on lip accuracy constraints, leading to a
comparatively lesser emphasis on dynamics of areas with weak correlation to
speech.

Identity Matching. Identity matching is a crucial component for achieving
personalization ability. We tested our 8-step distilled model on the whole
BEAT dataset. As shown in Tab. 4, the results indicate that our contrastive
learning approach effectively integrates the audio and identity modalities,
achieving high accuracy in identity matching.

Table 4: Identity matching results.

Metrics | Precision | Recall | F1 Score
---|---|---|---
Results(%) | 99.996 | 99.986 | 99.991

Table 5: Ablation study for our 8-step model. We conducted an evaluation of
key components within our model by assessing the impact on MBE and LBE metrics
in their absence.

Ablation Settings | MBE$\downarrow$ | LBE $\downarrow$ |
---|---|---|---
Ours | 0.4094 | 0.0969 |
w/o knowledge distillation | 0.4545 | 0.1177 |
w/o $L_{lip}$ loss | 0.4222 | 0.1037 |
w/o $L_{vel}$ loss | 0.4414 | 0.1129 |
w/o identity embedding | 0.4306 | 0.1034 |

Effectiveness of Each Component. We conducted a study to evaluate the impact
of each component. As presented in Tab. 5, both metrics increase without
knowledge distillation. The $L_{lip}$ loss specifically reduces the error rate
in the lip region, while the $L_{vel}$ loss focuses on the differences in
overall blendshapes between frames. The absence of identity embedding could
lead to a decrease in performance. This study shows the effectiveness of each
component.

####  4.2.2 Qualitative Evaluation

Figure 5: Qualitative comparisons on various talking identities with other
methods. We selected audio sequences from five different individuals as tests.
In each of the cases representing different talking identities, we emitted the
$/^{\prime}\varepsilon ks/$ sound in the word ”excellent.” We conducted
qualitative comparisons with ground truth respectively using VOCA, EmoTalk,
and DiffusionTalker.

Talking Identity Personalization. As shown in Fig. 5, we choose five audio
sequences from different people. For instance, Person 1 consistently speaks
with exaggerated facial expressions, displaying larger facial movement
amplitudes. Compared with other methods, DiffusionTalker generates facial
animations that are closer to the ground truth, effectively demonstrating the
personal talking identity. On the other hand, VOCA and EmoTalk are less
proficient in this regard, failing to capture and display the unique speaking
expression habits and styles of individuals.The qualitative comparisons tested
on VOCASET and user study are reported in the supplementary materials.

Figure 6: Facial Expression Diversity. The location of eyebrowns is non-
deterministic among various tests.

Facial Expression Diversity. As illustrated in Fig. 6. Under the same input
audio and identity, we conducted three inferences. It can be observed that, in
the same audio frame, the corresponding animation frames show noticeable
differences in the position of the eyebrows, demonstrating the non-
deterministic fact of speech-driven 3D face animation.

##  5 Conclusion

To generate personalized speech-driven 3D facial animation in a short time, we
propose a diffusion-based method named DiffusionTalker. The paper applies
contrastive learning between encoded audio features and learnable talking
identity to aggregate personal information. To further accelerate the
inference time, we distill dense knowledge from a teacher model with huge
sampling steps into a student model with much fewer sampling steps while
maintaining competitive accuracy. Finally, we conduct detailed experiments on
two datasets BEAT and VOCASET, demonstrating the effectiveness of our proposed
method.In the future, we will explore more natural 3D facial animation, and
personalized facial texture features based on speech show promising potential.

## References

  * Argente [1992] Joan A Argente.  From speech to speaking styles.  _Speech communication_ , 11(4-5):325–335, 1992.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Charalambous et al. [2019] Constantinos Charalambous, Zerrin Yumak, and A Frank van der Stappen.  Audio-driven emotional speech animation for interactive virtual characters.  _Computer Animation and Virtual Worlds_ , 30(3-4):e1892, 2019.
  * Chen et al. [2020a] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan.  Wavegrad: Estimating gradients for waveform generation.  _arXiv preprint arXiv:2009.00713_ , 2020a.
  * Chen et al. [2023] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo.  Diffusiondet: Diffusion model for object detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 19830–19843, 2023.
  * Chen et al. [2020b] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020b.
  * Cho et al. [2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_ , 2014.
  * Chollet et al. [2009] Gérard Chollet, Anna Esposito, Annie Gentes, Patrick Horain, Walid Karam, Zhenbo Li, Catherine Pelachaud, Patrick Perrot, Dijana Petrovska-Delacrétaz, Dianle Zhou, et al.  Multimodal human machine interactions in virtual and augmented reality.  _Multimodal Signals: Cognitive and Algorithmic Issues: COST Action 2102 and euCognition International School Vietri sul Mare, Italy, April 21-26, 2008 Revised Selected and Invited Papers_ , pages 1–23, 2009.
  * Cudeiro et al. [2019] Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael J Black.  Capture, learning, and synthesis of 3d speaking styles.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10101–10111, 2019.
  * Edwards et al. [2016] Pif Edwards, Chris Landreth, Eugene Fiume, and Karan Singh.  Jali: an animator-centric viseme model for expressive lip synchronization.  _ACM Transactions on graphics (TOG)_ , 35(4):1–11, 2016.
  * Fan et al. [2022] Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura.  Faceformer: Speech-driven 3d facial animation with transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18770–18780, 2022.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27, 2014.
  * Gu et al. [2023] Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, and Joshua M Susskind.  Boot: Data-free distillation of denoising diffusion models with bootstrapping.  In _ICML 2023 Workshop on Structured Probabilistic Inference $\\{$$\backslash$&$\\}$ Generative Modeling_, 2023.
  * Han et al. [2023] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang.  Svdiff: Compact parameter space for diffusion fine-tuning.  _arXiv preprint arXiv:2303.11305_ , 2023.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Hsu et al. [2021] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3451–3460, 2021.
  * Huang et al. [2018] Dong-Yan Huang, Ellensi Chandra, Xiangting Yang, Ying Zhou, Huaiping Ming, Weisi Lin, Minghui Dong, and Haizhou Li.  Visual speech emotion conversion using deep learning for 3d talking head.  In _Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and first Multi-Modal Affective Computing of Large-Scale Multimedia Data_ , pages 7–13, 2018.
  * Ji et al. [2023] Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo.  Ddp: Diffusion model for dense visual prediction.  _arXiv preprint arXiv:2303.17559_ , 2023.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kumari et al. [2023] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1931–1941, 2023.
  * Lewis et al. [2014] John P Lewis, Ken Anjyo, Taehyun Rhee, Mengjie Zhang, Frederic H Pighin, and Zhigang Deng.  Practice and theory of blendshape facial models.  _Eurographics (State of the Art Reports)_ , 1(8):2, 2014.
  * Li et al. [2023] Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren.  Snapfusion: Text-to-image diffusion model on mobile devices within two seconds.  _arXiv preprint arXiv:2306.00980_ , 2023.
  * Liu et al. [2022] Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng.  Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis.  In _Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VII_ , page 612–630, Berlin, Heidelberg, 2022. Springer-Verlag.
  * Liu et al. [2023] Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao.  Cones: Concept neurons in diffusion models for customized generation.  _arXiv preprint arXiv:2303.05125_ , 2023.
  * Meng et al. [2023] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans.  On distillation of guided diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14297–14306, 2023.
  * Nichol et al. [2021] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Oktay et al. [2018] Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al.  Attention u-net: Learning where to look for the pancreas.  _arXiv preprint arXiv:1804.03999_ , 2018.
  * Oord et al. [2016] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.  Wavenet: A generative model for raw audio.  _arXiv preprint arXiv:1609.03499_ , 2016.
  * Peng et al. [2023] Ziqiao Peng, Haoyu Wu, Zhenbo Song, Hao Xu, Xiangyu Zhu, Jun He, Hongyan Liu, and Zhaoxin Fan.  Emotalk: Speech-driven emotional disentanglement for 3d face animation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 20687–20697, 2023.
  * Pham et al. [2018] Hai Xuan Pham, Yuting Wang, and Vladimir Pavlovic.  End-to-end learning for 3d facial animation from speech.  In _Proceedings of the 20th ACM International Conference on Multimodal Interaction_ , pages 361–365, 2018.
  * Ping et al. [2013] Heng Yu Ping, Lili Nurliyana Abdullah, Puteri Suhaiza Sulaiman, and Alfian Abdul Halin.  Computer facial animation: A review.  _International Journal of Computer Theory and Engineering_ , 5(4):658, 2013.
  * Prajwal et al. [2020] KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar.  Learning individual speaking styles for accurate lip to speech synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13796–13805, 2020.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Richard et al. [2021] Alexander Richard, Michael Zollhöfer, Yandong Wen, Fernando De la Torre, and Yaser Sheikh.  Meshtalk: 3d face animation from speech using cross-modality disentanglement.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1173–1182, 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22500–22510, 2023.
  * Salimans and Ho [2022] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  _arXiv preprint arXiv:2202.00512_ , 2022.
  * Sheng et al. [2022] Changchong Sheng, Gangyao Kuang, Liang Bai, Chenping Hou, Yulan Guo, Xin Xu, Matti Pietikäinen, and Li Liu.  Deep learning for visual speech analysis: A survey.  _arXiv preprint arXiv:2205.10839_ , 2022.
  * Song et al. [2020] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020.
  * Stan et al. [2023] Stefan Stan, Kazi Injamamul Haque, and Zerrin Yumak.  Facediffuser: Speech-driven 3d facial animation synthesis using diffusion.  _arXiv preprint arXiv:2309.11306_ , 2023.
  * Taylor et al. [2017] Sarah Taylor, Taehwan Kim, Yisong Yue, Moshe Mahler, James Krahe, Anastasio Garcia Rodriguez, Jessica Hodgins, and Iain Matthews.  A deep learning approach for generalized speech animation.  _ACM Transactions On Graphics (TOG)_ , 36(4):1–11, 2017.
  * Tian et al. [2020] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16_ , pages 776–794. Springer, 2020.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wohlgenannt et al. [2020] Isabell Wohlgenannt, Alexander Simons, and Stefan Stieglitz.  Virtual reality.  _Business & Information Systems Engineering_, 62:455–461, 2020.
  * Wu et al. [2018] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3733–3742, 2018.
  * Xing et al. [2023] Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien-Tsin Wong.  Codetalker: Speech-driven 3d facial animation with discrete motion prior.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12780–12790, 2023.
  * Zhou et al. [2018] Yang Zhou, Zhan Xu, Chris Landreth, Evangelos Kalogerakis, Subhransu Maji, and Karan Singh.  Visemenet: Audio-driven animator-centric speech animation.  _ACM Transactions on Graphics (TOG)_ , 37(4):1–10, 2018.