# GSTalker: Real-time Audio-Driven Talking Face Generation via Deformable
Gaussian Splatting

Bo Chen<EMAIL_ADDRESS>Shanghai Jiaotong UniversityShanghaiChina ,
Shoukang Hu<EMAIL_ADDRESS>Nanyang Technological
UniversitySingaporeSingapore ,  Qi Chen<EMAIL_ADDRESS>Shanghai
Jiaotong UniversityShanghaiChina ,  Chenpeng Du<EMAIL_ADDRESS>Shanghai Jiaotong UniversityShanghaiChina ,  Ran Yi<EMAIL_ADDRESS>Shanghai Jiaotong UniversityShanghaiChina ,  Yanmin Qian
<EMAIL_ADDRESS>Shanghai Jiaotong UniversityShanghaiChina and  Xie Chen
<EMAIL_ADDRESS>Shanghai Jiaotong UniversityShanghaiChina

###### Abstract.

We present GSTalker, a 3D audio-driven talking face generation model with
Gaussian Splatting for both fast training (40 minutes) and real-time rendering
(125 FPS) with a 3$\sim$5 minute video for training material, in comparison
with previous 2D and 3D NeRF-based modeling frameworks which require hours of
training and seconds of rendering per frame. Specifically, GSTalker learns an
audio-driven Gaussian deformation field to translate and transform 3D
Gaussians to synchronize with audio information, in which multi-resolution
hashing grid-based tri-plane and temporal smooth module are incorporated to
learn accurate deformation for fine-grained facial details. In addition, a
pose-conditioned deformation field is designed to model the stabilized torso.
To enable efficient optimization of the condition Gaussian deformation field,
we initialize 3D Gaussians by learning a coarse static Gaussian
representation. Extensive experiments in person-specific videos with audio
tracks validate that GSTalker can generate high-fidelity and audio-lips
synchronized results with fast training and real-time rendering speed.

Audio-Driven Talking Face, Gaussian Splatting, Neural Radiance Field

††ccs: Computing methodologies Animation††ccs: Computing methodologies
Rendering

##  1\. INTRODUCTION

Audio-driven talking face generation has received increasing attention due to
its promising prospect in various applications, including digital humans,
virtual video conferencing, visual dubbing, etc. Recent methods (Prajwal et
al., 2020; Zhou et al., 2020; Lu et al., 2021; Gururani et al., 2023; Du et
al., 2023; Shen et al., 2023) either learn 2D audio-driven talking faces
through direct mapping from audio to face landmarks or 3D audio-driven talking
faces with neural radiance field (NeRF) implicit representation. However,
these methods typically require expensive computational resources for training
and rendering, posing potential challenges to their real-world applications.
For example, although NeRF-based 3D audio-driven talking face models achieve
consistent multi-view rendering results, they consume up to 36 hours of
training time on a 5-minute Obama video (Guo et al., 2021) to learn a 3D
audio-driven talking face model with a rendering speed of 0.08 frames per
second (FPS).

To improve the optimization speed of audio-driven talking face models, a
recent line of research (Tang et al., 2022; Li et al., 2023) focuses on
learning efficient 3D representation for 3D audio-driven talking face models.
For example, instead of learning neural radiance field from coordinate neural
networks, RAD-NeRF and ER-NeRF incorporate multi-resolution hashing grid
(Müller et al., 2022) representation with pruning techniques like occupancy
grid pruning to improve training and rendering speed. Although these methods
can accelerate the training and rendering by a large margin, they still
require hours of training time and their rendering quality can be improved
especially for detailed facial parts like teeth. A bottleneck hindering their
training and rendering efficiency is that the backward mapping-based ray-cast
rendering algorithm performs volume rendering on hundreds of sampled points
for each pixel.

Recently, point-based 3D Gaussian Splatting (3D-GS) (Kerbl et al., 2023) has
been proposed to fast render static (Kerbl et al., 2023) and dynamic scenes
(Luiten et al., 2023; Yang et al., 2023; Wu et al., 2023) through splatting a
restricted set of discrete 3D Gaussians. Motivated by these progresses, we
attempt to introduce 3D-GS to 3D audio-driven talking face generation.
Although this venue is promising, there exist two main challenges we need to
tackle: 1) How to utilize audio information to drive the 3D talking face and
torso in a 3D Gaussian Splatting framework? The variety and abstraction of
speech signals make speech-driven tasks different from previous static or
dynamic scene reconstruction tasks. A new framework tailored for 3D audio-
driven talking face generation should be designed by conditioning audio
information in the 3D Gaussians Splatting framework. 2) How to optimize the
efficiency the 3D audio-driven talking face generation model? Current 3D-GS
methods initialize 3D Gaussians either from a random point cloud or a sparse
point cloud produced from Structure-from-Motion (SfM) (Schonberger and Frahm,
2016; Snavely et al., 2006). These two initialization approaches are mainly
designed for static scenes with multi-view images, which overlooks the
internal structure of taking faces. For example, we observe that 3D Gaussians
inherited from the random point cloud are difficult to converge in the 3D
audio-driven talking face generation task.

To address the above two challenges, we propose a novel framework called
GSTalker: which enables rapid training and real-time rendering of audio-driven
talking face generation via deformable Gaussian Splatting. The detailed
strategies are shown as follows. 1) Deformable Gaussian Splatting for Audio-
Driven Talking Face Generation. Inspired by previous research in deformable
neural radiance fields (Park et al., 2021), we encode a motion-independent
static taking face in a canonical space with 3D Gaussians and model a
deformable field for position and covariance offsets with audio information or
head pose as a condition. To learn an accurate deformable mapping from audio
to offsets of static 3D Gaussians, we introduce a tri-plane multi-resolution
hash encoder to represent Gaussian spatial information and a temporal smooth
module to smooth the audio representation extracted by a pre-trained ASR model
(Hsu et al., 2021; Amodei et al., 2016; Baevski et al., 2020). For each video
frame, the tri-plane features and smoothed audio representation are
concatenated into a tiny MLP deformable field to predict the position and
covariance offsets of the static 3D Gaussians. With the audio-driven
deformable 3D Gaussians, we can fast render the talking face through a tile-
based differentiable rasterizer. 2) Efficient optimization of 3D audio-driven
taking face generation model. To achieve efficient optimization, we maintain a
static Gaussian initialization from talking face images instead of randomly
initiating the 3D Gaussians. Specifically, we first use the automatic parsing
method (Lee et al., 2020) to initialize the head and the torso part
respectively. For the head region, we optimize a set of static 3D Gaussians
with all talking face images and use it as the initialization for 3D Gaussians
in the head region. For the torso part, we optimize the 3D Gaussians with a
fixed camera stance as the torso part approximates the translation motion in
the 2D plane. Thanks to the rendering speed of the point-based Gaussian
Splatting, the above initializations can be finished in about one minute.
Since the above initialization contains information about talking face images,
the deformable field can be quickly optimized to learn audio-dependent offsets
of 3D Gaussians for eventual talking face generation.

Our main contributions are as follows:

  1. (1)

We present GSTalker, a 3D audio-driven talking face generation model with
Gaussian Splatting for both fast training (40 minutes) and real-time rendering
(125 FPS), in which an audio-driven deformable field together with a multi-
resolution hashing grid-based tri-plane and a temporal smooth module is
incorporated to learn fine-grained facial details.

  2. (2)

To enable efficient optimization of audio-driven talking face generation, we
learn static initializations of 3D Gaussians for head and torso regions from
talking face images.

  3. (3)

Extensive experiments in person-specific videos with audio tracks validate
that GSTalker can generate high-fidelity and audio-lips synchronized results
with fast training and real-time rendering speed.

##  2\. RELATED WORK

###  2.1. Audio-Driven Talking Face Generation

Audio-driven talking face generation aims to synthesize synchronized speaking
videos of a specified person given the input audio. Recently there have been
active and consistent research efforts on generating high-fidelity images and
achieving better lip-audio synchronization, the methods they have proposed can
be categorized into 2D-based method (Thies et al., 2020; Lu et al., 2021;
Gururani et al., 2023; Du et al., 2023; Stypułkowski et al., 2023) and 3D
NeRF-based method (Guo et al., 2021; Liu et al., 2022; Ye et al., 2023; Tang
et al., 2022). Wav2lip (Prajwal et al., 2020) proposes to use a pre-trained
lip-sync expert model as the discriminator for lip-syncing a talking face
video and performs adversarial training. Neural voice puppetry (NVP) (Thies et
al., 2020) leverages 3DMM coefficients and neural textures to synthesize mouth
movement. However, these two methods can only synthesize the mouth but fail to
control the head pose. SPACE (Gururani et al., 2023) proposes a latent
representation based on a pre-trained face-vid2vid model to achieve one-shot
video synthesis but still struggles for multi-view consistency. Recently, the
Diffusion Denoising Probabilistic Model (DDPM) (Ho et al., 2020) exhibits
unique advantages in talking face generation. In DAE-Talker (Du et al., 2023),
a diffusion autoencoder (DAE) is first pre-trained on face images and attains
a well-defined latent space, and then the model is trained to predict the
latent representations of the DAE from speech. Diffused Head (Stypułkowski et
al., 2023) proposes an auto-regressive frames prediction manner with diffusion
models. Diffusion-based methods have achieved impressive image quality but
slow inference.

NeRF (Mildenhall et al., 2021) is another promising direction and has been
widely explored in audio-driven talking face generation. AD-NeRF (Guo et al.,
2021) maps the audio features to dynamic neural radiance fields for portrait
rendering. Geneface (Ye et al., 2023) improves the generalizability of the
NeRF-based methods by introducing a pre-trained generative audio-to-motion
model. RAD-NeRF (Tang et al., 2022) firstly introduces multi-resolution hash
grids to encode spatial and phonetic information separately, which greatly
accelerates NeRF-based method training and inference. ER-NeRF (Li et al.,
2023) leverages tri-plane hash representation to reduce hash collision and
thus achieve fast convergence. and a cross-modal fusion mechanism has been
proposed to enhance lip-speech synchronization. Although NeRF-based methods
can achieve photorealistic image quality and multi-view consistency, their
training and inference are still computationally expensive because of the ray-
cast operation and volume rendering computation in NeRF.

###  2.2. Efficient Neural Representation

In some scenarios, the generation of talking-head videos needs to operate in
real-time. In light of the intensive training cost and slow inference lied in
vanilla NeRF (Mildenhall et al., 2021) approaches, various hybrid explicit-
implicit representations (Sun et al., 2022; Chen et al., 2022; Müller et al.,
2022) are proposed to accelerate neural rendering. Instant-NGP (Müller et al.,
2022) adopts a multi-resolution hash table to replace parts of the
computationally expensive fully connected layers, significantly improving
rendering efficiency without loss of computational precision. Recently 3D
Gaussian Splatting (3D-GS) (Kerbl et al., 2023) achieves the state-of-the-art
visual quality and real-time rendering for novel-view synthesis. Specifically,
3D-GS adopts 3D Gaussians as a discrete representation of the scene and allows
parameter optimization based on differentiable anisotropic splatting. Compared
to the methods based on volume rendering of implicit neural radiance fields,
3D-GS reduces unnecessary spatial computation and the fast visibility-aware
rendering algorithm achieves parallel optimization across pixels. Since 3D-GS
is only suitable for multi-view reconstruction of static scenes, many methods
(Luiten et al., 2023; Yang et al., 2023; Wu et al., 2023) have been proposed
to achieve dynamic scene reconstruction based on 3D-GS. Dynamic3DGS (Luiten et
al., 2023) creates positions and properties for 3D Gaussians at each timestamp
to achieve dynamic scene reconstruction. 4D-GS (Wu et al., 2023) proposes
time-conditioned deformation fields to predict the deformation and movement of
Gaussian. Recently, many approaches (Zielonka et al., 2023; Qian et al., 2023;
Hu and Liu, 2023; Xu et al., 2023; Hu et al., 2023; Lei et al., 2023) have
also tried to build dynamic 3D human models based on 3D-GS and parametric
human model SMPL (Loper et al., 2023). Our work first introduces 3D-GS to
audio-driven talking face generation, enabling faster convergence and high-
fidelity video generation.

##  3\. method

Figure 1. Overview of GSTalker. We model the whole talking face in 3D
Gaussians and deformation fields. Firstly, a static initialization stage is
used to optimize the coarse static Gaussian of the face from a random point
cloud. Then an audio-conditioned deformation field fuses spatial features from
a multi-resolution tri-plane hash grid with audio features from an audio
encoder to predict the position and shape changes of the 3D Gaussians. Given a
camera pose, the deformed Gaussians achieve real-time rendering using the
differentiable rasterizer. For the torso part, a similar pose-condition
deformation field is adopted to drive the stabilizing motion of the torso.

Similar to previous NeRF-based methods (Guo et al., 2021; Tang et al., 2022;
Li et al., 2023), Given a 3-5 minute video of a specified portrait speaking in
front of a fixed camera as training material, we aim to synthesize a talking
face video of the portrait with a new speech. We use the camera pose estimated
by the face-tracker (Thies et al., 2016) to simulate the head motion. For
audio signal processing, we use a pre-trained ASR model (Amodei et al., 2016;
Baevski et al., 2020) for feature extraction. Since the motion of the head and
torso parts is inconsistent, we use the Semantic Parsing Method (Lee et al.,
2020) to segment the head, torso, and background parts. In the following part,
we first review the preliminaries of 3D Gaussian Splatting (Kerbl et al.,
2023) in Sec. 3.1. Then we elaborate on the proposed real-time audio talking
face generation framework via deformable Gaussian Splatting in Sec. 3.2,
followed by the details of the efficient optimization as presented in Sec.
3.3.

###  3.1. Preliminaries

3D Gaussian Splatting. In 3D Gaussian Splatting (Kerbl et al., 2023), a static
scene can be represented by a 3D Gaussian set. Each 3D Gaussian consists of a
series of optimizable parameters, including a mean position $\mu$, positive
semi-definite covariance matrices $\Sigma$, e.t.,

(1) $G(x)=e^{-\frac{1}{2}(x-\mu)^{T}\Sigma^{-1}(x-\mu)}$

To facilitate optimization, $\Sigma$ is usually decomposed into a scaling
matrix $S$ and rotation matrix $R$:

(2) $\Sigma=RSS^{T}R^{T}$

and each matrix can be converted into two optimizable vectors: a 3D vector
$s\in R^{3}$ for scaling and a quaternion $q\in R^{4}$ for rotation.

Then we transform the 3D Gaussian from world space to image space. As shown in
(Zwicker et al., 2001), given a viewing transformation $W$ and a affine
approximation Jacobian $J$ of the projective transformation, the covariance
matrix $\Sigma^{\prime}$ in ray coordinates is computed by following:

(3) $\Sigma^{\prime}=JW\Sigma W^{T}J^{T}$

For each pixel on the image, the color $C$ is computed by blending N ordered
points overlapping the pixel:

(4) $C=\sum_{i\in N}c_{i}\alpha_{i}\prod_{j=1}^{i-1}(1-\alpha_{j})$

where $c_{i}$ is represented by spherical harmonics (SH) coefficients and
$\alpha_{i}$ is the corresponding learned opacity.

###  3.2. Deformable Gaussian Splatting for Talking Face

Previously, the NeRF-based method (Guo et al., 2021; Tang et al., 2022; Li et
al., 2023) directly condition implicit function to model the topological
changes of the talking face: $F:f,g,d\rightarrow c,\sigma$, where $f,g,d$ are
the spatial feature, audio feature, and view direction respectively.
Nevertheless, encoding 3D Gaussian directly for each audio feature is
inefficient in expressing dynamic scenes. Noting that only a small portion of
the facial regions in the talking face have significant topological
deformations, such as the mouth and eyes, it is intuitively possible to
maintain a static 3D Gaussian $G$ in a canonical space to characterize the
physical geometry of the head and use a deformation field $F$ to learn the
audio-driven deformation information about 3D Gaussian. Since each 3D Gaussian
has a different level of correlation with the audio features, for example, the
points in the mouth region are more closely affected by audio and have more
frequent movement, we take the spatial features of the different 3D Gaussian
and the audio features $A$ as inputs to the deformation field to predict the
deformed 3D Gaussian $G^{\prime}$. Then the deformed 3D Gaussian renders the
talking face image $I$ through a tile-based differentiable rasterizer. The
whole process can be formulated as follows:

$I=Splatting(G^{\prime}|G,F,A)$

Multi-resolution Hashing Grid-based Tri-plane. Multiresolution Tri-plane Hash
encoder utilizes a structured mesh to store latent features instead of a
partially computationally expensive fully connected layer, and the
multiresolution feature fusion well considers the motion similarity of
neighboring Gaussian points. The Tri-plane hash method simultaneously reduces
unnecessary computations in Gaussian sparse regions and hash conflicts in
Gaussian dense regions. Specifically, as shown in Figure 1, we first project
the positions $\textbf{x}=(x,y,z)\in R^{3}$ of each 3D Gaussian into a multi-
resolution hash tri-plane grid, and the projected coordinate will be
interpolated over different resolution levels of 2D hash grid to obtain high
dimensional optimizable spatial features $f_{H}$.

To reduce the noise between voice frames and enhance the temporal correlation
between generated video frames, the audio HuBERT encoder (Hsu et al., 2021)
successively utilizes 1D convolutional networks and self-attention modules to
temporally smooth the audio features extracted from the pre-trained ASR models
into feature $f_{a}$. To control the blinking action, as in RAD-NeRF (Tang et
al., 2022), we compute the ratio of the open eye area to the whole image area
as our eye feature $f_{e}$. These features will be concatenated and fed into a
tiny MLP decoder to predict the positional movement $\Delta\textbf{x}$,
rotation change $\Delta q$, and scaling change $\Delta s$ of 3D Gaussians:

(4) $\Delta\textbf{x},\Delta q,\Delta s=MLP(f_{H},f_{a},f_{e}).$

Then the deformed 3D Gaussians is formulated as $G(\textbf{x}+\Delta
x,q+\Delta q,s+\Delta s,SH,\alpha)$. In our framework, we assume the audio
information has a negligible effect on the deformation of spherical harmonic
coefficients (SH) and opacity $\alpha$. We keep the SH coefficients and
opacity $\alpha$ of deformable 3D Gaussians unchanged during the audio-driven
deformation. This design allows us to employ a lighter network for the
deformation network and accelerates the training process.

Since the torso usually orients the camera frontally and its movement is a
tiny translation motion, we adopt a similar setting as AD-NeRF (Guo et al.,
2021): using head pose $\boldsymbol{P}=(\boldsymbol{R},t)$ rather than audio
as the condition to drive the torso’s movement, where $\boldsymbol{R},t$ are
the rotation and translation matrix respectively. Specifically, we train a
Gaussian deformation field with head pose $\boldsymbol{P}$ as the condition.
To better inscribe the details, before we feed the head pose $\boldsymbol{P}$
into the deformation field, we use positional encoding (Mildenhall et al.,
2021) $\gamma(p)=(sin(2^{i}\pi p),cos(2^{i}\pi p))_{i=0}^{L-1}$ to map it to a
higher-dimensional feature $f_{p}$, where L=6 in our experiments. The pose
features $f_{p}$ and the spatial feature $f_{H}$ encoded from the hash grid
are then concatenated together to predict the motion and deformation of 3D
Gaussians:

(5) $\Delta\textbf{x},\Delta q,\Delta s=MLP(f_{H},f_{p}).$

Since the torso’s movement involves only one viewpoint, the deformed 3D
Gaussians $G(\textbf{x}+\Delta x,q+\Delta q,s+\Delta s,SH,\alpha)$ is
projected under a fixed view $\boldsymbol{\Pi_{0}}=(\boldsymbol{R_{0}},t_{0})$
for differentiable splatting.

###  3.3. Efficient Optimization of 3D Audio-Driven Talking Face Generation

Compared with the disordered and random Gaussian initialization, the Gaussian
initialization that can preliminarily characterize the geometry makes it
easier to obtain a more convenient deformation process with specific physical
meaning. Unlike the current 3DGS-based avatar which a 3D parametric model is
used geometrically before Gaussian initialization, we get a coarse static 3D
Gaussian representation in canonical space without adding a neural network.

Methods | PSNR$\uparrow$ | LPIPS$\downarrow$ | LMD$\downarrow$ | Sync$\uparrow$ | AUE$\downarrow$ | Trainging time | Inference FPS
---|---|---|---|---|---|---|---
Ground Truth | $\infty$ | 0 | 0 | 7.491 | 0 | - | -
MakeItTalk(Zhou et al., 2020) | - | - | 5.481 | 5.112 | 1.519 | - | 12
Wav2lip(Prajwal et al., 2020) | - | - | 3.718 | 6.892 | 1.218 | - | 15
AD-NeRF(Guo et al., 2021) | 31.39 | 0.0857 | 2.917 | 5.426 | 0.991 | 36h | 0.08
RAD-NeRF(Tang et al., 2022) | 34.00 | 0.0387 | 2.696 | 6.664 | 0.882 | 7h | 40
ER-NeRF(Li et al., 2023) | 33.10 | 0.0291 | 2.740 | 5.708 | 0.854 | 2h | 34
GSTalker | 34.65 | 0.0151 | 2.695 | 5.775 | 0.862 | 40min | 125

Table 1. The quantitative comparison under the self-driven setting. We show
the best results with bold. MakeItTalk is unable to specify head posture, and
Wav2lip can see the mouth shape of the ground truth, so we use other voices as
audio input. For a fair comparison, we do not compare the PSNR and LPIPS of
these two methods. We only compare the training time of person-specific
methods.

shop

back

the

Ground-Truth

GSTalker

ER-NeRF

RAD-NeRF

AD-NeRF

MaketItTalk

Wav2Lip

Figure 2. The qualitative comparison with 2D-bsed and 3D NeRF-based Mehods. We
represent the image results generated by our method and all the baselines on
the AD-NeRF(Guo et al., 2021) dataset.

Static Gaussian initialization. From the perspective of the camera space, the
movement of the whole head can be divided into rigid movements related to the
head posture, including the translation and rotation of the head and the
expression transformation related to facial expressions, while speech is only
related to facial expressions including lips and is not correlated with head
posture. Therefore, we explicitly utilize the camera transformation $\Pi$ to
simulate the head movement $P$ instead of the model transformation which
adopts the additional pose control parameters that need to be learned. On the
other hand, the rigid motion of the head does not change the geometry of the
head, so this transformation method does not affect the learning of
initializing the head geometry. After that, we regarded the whole process of
3D Gaussian initialization as a static scene reconstruction process, ignoring
the movement part of the entire facial expression related to speech, and
focusing on learning the static facial structure and skin texture.
Specifically, for each iteration i, the 3D Gaussians is formulated as
$G(i)=\\{x_{i},s_{i},q_{i},\alpha_{i},c_{i}\\}$, we randomly select an image
$I_{gt}(A_{t},P_{t})$ from the monocular video:
$\\{I(A_{1},P_{1}),...,I(A_{n},P_{n})\\}$ as the ground truth, where $A$
represents the speech feature and $P$ represents the head pose, then we render
the image $I(G(i),R_{t},T_{t},K)$, where $K$ is the intrinsic matrix and
$\\{R_{t},T_{t}\\}$ is the camera pose which is the inverse of head poses
$P_{t}$. By continuously calculating the loss between $I$ and $I_{gt}$, we
obtain an initialized 3D Gaussian representation
$G(n)=\\{x_{n},s_{n},q_{n},\alpha_{n},c_{n}\\}$of the head after n iterations.
Our experiments reveal that the initialization of 3D Gaussians has a huge
influence on the convergence of the 3D audio-driven talking face generation.
As mentioned in (Guo et al., 2021), the camera pose is computed from the head
pose and the movement of the head part is not consistent with the movement of
the torso part, so we need to divide the portrait image into the head region
and torso region at the beginning. Since the torso usually orients the camera
frontally and its movement is a tiny translation motion, we set a fixed camera
pose for rendering and the rest of the optimization process is similar to the
head region. For convenience, we randomly initialize 10,000 3D Gaussians in
the space of a cube, then we only need tens of seconds to complete the entire
static Gaussian initialization process because the whole process of Gaussian
splatting is completely differentiable. Adaptive Density Control Strategy. In
addition to optimizing the learnable parameters of 3D Gaussian itself, we also
follow the prune, clone, and split strategies in 3D-GS to adaptively control
the amount of 3D Gaussians. Since the clone and split operations are based on
the position gradients and magnitude of the scaling matrix, for some regions
of frequent movement such as the eyes and mouth, it will gather more and
smaller 3D Gaussian. For the less moving part, there will be no more redundant
Gaussian because the losses converge faster. The adaptive density control
strategy is more beneficial to more refined and accurate control of the moving
part in the later conditional Gaussian deformation field without introducing
redundant computations.

Training Objectives. We use the L1 regularization term $L_{color}$ for each
pixel’s color between the rendered image $I$ and the ground truth image
$I_{gt}$ which is also used in 3D-GS. To facilitate a better blend between the
rendered head region, torso region, and the background, we further use the
mask loss $L_{mask}$ as supervision for the transparency of each pixel. For
some rigid and sharp parts of the face, such as teeth, the pixel-based loss is
often difficult to optimize. Therefore we added the LPIPS loss (Zhang et al.,
2018) $L_{lpips}$ based on depth perception. Based on the landmark, we give
additional patch L1 loss $L_{lips}$to the lip region P to enhance the
reconstruction of strongly correlated regions of speech. Overall, the total
loss function is formulated as follows:

$L=L_{color}+\lambda_{1}L_{mask}+\lambda_{2}L_{lpips}+\lambda_{3}L_{lips}$

##  4\. experiments

###  4.1. Experimental Settings

####  4.1.1. Dataset

We utilize five person-specific videos with audio track as our training
material which are also used in (Guo et al., 2021; Ye et al., 2023) for a fair
comparison. Each video lasts for 3 $\sim$ 5 minutes and the frame rate is
adjusted to 25 fps. The raw video in AD-NeRF (Guo et al., 2021) is cropped to
450 $\times$ 450, while other videos in (Ye et al., 2023) are cropped to 512
$\times$ 512\. Besides, we also extracted two audio tracks from public demos
of SynObama (Suwajanakorn et al., 2017) and NVP (Thies et al., 2020), named
Testset A and Testset B respectively, as part of our evaluation of the
sources.

####  4.1.2. Implementation Details

For each video, we randomly initialize 10,000 Gaussian points within a cube of
side length 2. Then we optimize our Gaussian properties individually for
10,000 iterations in the static initialization phase and optimize our Gaussian
points and deformation field simultaneously for 100,000 iterations. For each
2D hash encoder, we set L = 14, and F = 1, and resolutions increased from 64
to 512. Adam optimizer(Kingma and Ba, 2014) with a learning rate of 5e-3 for
the hash encoder and 1.6e-5 for the deformation network. We perform all of our
experiments on a single RTX 3090 GPU.

####  4.1.3. Comparing baselines and metrics

Several typical and popular baseline models are adopted for comparison,
including Wav2-Lip (Prajwal et al., 2020), MakeItTalk (Zhou et al., 2020) and
3D-NeRF based person-specific methods: AD-NeRF (Guo et al., 2021), RAD-NeRF
(Tang et al., 2022) and ER-NeRF (Li et al., 2023). The metrics to evaluate
talking face models consist of image quality, lip synchronization, and
efficiency. More specifically, we employ the Peak Signal-to-Noise Ratio (PSNR)
and Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018) to
measure the quality of image generation. To measure lip synchronization, the
landmark distance (LMD) (Chen et al., 2018) is adopted to calculate the L2
distance of the mouth region landmarks between the synthesized image and the
real picture. We also use SyncNet confidence score (Sync) (Chung and
Zisserman, 2017) which uses a pre-trained SyncNet to measure the lip-sync
error between the generated video frames and the corresponding speech segment.
Action units error (AUE) (Chung and Zisserman, 2017) is used to evaluate the
degree of activity synchronization of the lip-related facial muscles. Besides,
we also compare the time overhead of training and inference between the
different methods.

###  4.2. Quantitative Evaluation

####  4.2.1. Comparison Settings

We take two settings for the quantitative comparison: 1) Self-driven setting,
the speaker of the audio tracks in the train, and test sets are the same. 2)
Cross-driven setting, we use the Testset A and Testset B, whose speakers are
unseen during the model training. We train all models on the AD-NeRF (Guo et
al., 2021) dataset for fairness. Since the ground truth video is not available
for the cross-driven set, Sync and AUE are adopted as evaluation metrics.

Ground-Truth

RAD-NeRF

ER-NeRF

GSTalker

Figure 3. The qualitative comparison with real-time NeRF-based methods. We
show more generated results compared to the real-time NeRF-based methods.
These results include the performance across genders and languages in the
self-driven setting.

| TestsetA | TestsetB
---|---|---
Methods | Sync$\uparrow$ | AUE$\downarrow$ | Sync$\uparrow$ | AUE$\downarrow$
Ground Truth | 6.855 | 0 | 7.486 | 0
MakeItTalk(Zhou et al., 2020) | 5.351 | 2.121 | 5.091 | 1.444
Wav2Lip(Prajwal et al., 2020) | 7.912 | 1.822 | 8.149 | 1.259
AD-NeRF(Guo et al., 2021) | 4.855 | 1.943 | 4.698 | 1.541
RAD-NeRF(Tang et al., 2022) | 5.747 | 1.788 | 6.372 | 1.237
ER-NeRF(Li et al., 2023) | 5.830 | 1.782 | 6.800 | 1.156
Ours | 6.299 | 1.771 | 6.718 | 1.251

Table 2. The quantitative results under the cross-driven setting. We compare
the audio-lips synchronization and facial motion similarity with the other
methods under the cross-driven setting. We show the best and second best
results with blod and underline, respectively.

####  4.2.2. Evaluation Results

We present the quantitative comparison results under the self-driven setting
in Table 1, Compared with the best competitive baseline model (ER-NeRF), our
proposed GSTalker achieves $2\sim 3$ times speedup in both training and
inference. Meanwhile, our method presents superior picture quality performance
not only at the pixel level but also at the depth perception level. In terms
of audio-lip synchronization, our methods achieve the lowest landmark distance
on the mouth and promising performance on Sync and AUE. In the cross-driven
setting experiment in Table 2, our method ranks among the best in both the
Sync and AUE metrics, which reveals the excellent generalization ability
across the speaker for our proposed method.

###  4.3. Qualitative Evaluation

####  4.3.1. Evaluation Results

We represent the visual results of all methods in the AD-NeRF (Guo et al.,
2021) dataset in Figure 2. Although Wav2lip and MakeItTalk work with arbitrary
portrait inputs and perform well on some metrics, it is still difficult to
produce realistic and high-fidelity videos. For the 3D NeRF-based methods, we
synthesized both the head and torso parts for Evaluation. AD-NeRF which uses a
vanilla NeRF for scene representation, tends to generate ambiguous torso and
audio-lip unsynchronized mouth. RAD-NeRF and ER-NeRF, while able to synthesize
high-quality torso and facial details, still struggle with fine teeth
structure. Further, we compare our method with RAD-NeRF and ER-NeRF on more
portraits as presented in Figure 3. It is found that our method achieves finer
hair and facial details and a clearer torso, as well as sharper teeth and
improved audio-lip synchronization.

####  4.3.2. User Study

A user study is employed to further evaluate user-specific real-world
experiences. Specifically, we selected 36 generated video results from
different methods and surveyed 26 users on the following three aspects: (1)
Lip-sync Accuracy; (2) Motion Naturalness; and (3) Image Fidelity. The
statistical average results are shown in Table 3. Our method yields the best
head naturalness performance, in addition to being among the best in terms of
lip-sync accuracy and image fidelity. Note that our approach only requires
less training time as well as having faster inference speeds

Methods | Lip-sync Accuracy | Motion Naturalness | Image Fidelity
---|---|---|---
MakeItTalk | 3.55 | 3.55 | 3.28
Wav2lip | 3.19 | 3.26 | 2.70
AD-NeRF | 3.93 | 3.51 | 3.91
RAD-NeRF | 4.10 | 3.95 | 4.16
ER-NeRF | 4.09 | 4.11 | 4.22
GSTalke | 4.04 | 4.14 | 4.18

Table 3. User study. The rating is on a scale of 1-5, the higher the better.
We show the best with bold and second best results with underline.

###  4.4. Ablation Study

An ablation study of network architecture and optimization solutions,
including the qualitative and quantitative results, are shown in Table 4. The
experiments are carried out on the AD-NeRF (Guo et al., 2021) dataset under
the self-driven setting.

Methods | PSNR$\uparrow$ | LPIPS$\downarrow$ | LMD$\downarrow$ | Sync$\uparrow$
---|---|---|---|---
Ground Truth | $\infty$ | 0 | 0 | 7.491
w/o static initialization | 34.25 | 0.0169 | 3.068 | 1.506
w/o hash grid | 34.39 | 0.0159 | 2.798 | 5.438
w/o LPIPS loss | 34.66 | 0.0259 | 2.773 | 5.305
Ours | 34.65 | 0.0151 | 2.695 | 5.775

Table 4. Ablation Study on network architecture and optimization solutions.
Comparing image quality and audio-lips synchronization on static Gaussian
initialization, multiresolution hash grid, and perceptual loss supervision

Methods | PSNR$\uparrow$ | LPIPS$\downarrow$ | LMD$\downarrow$ | Snyc$\uparrow$
---|---|---|---|---
Ground Truth | $\infty$ | 0 | 0 | 7.491
Deepspeech(Amodei et al., 2016) | 34.65 | 0.0151 | 2.695 | 5.775
Wav2Vec(Baevski et al., 2020) | 34.65 | 0.0149 | 2.661 | 5.903
Hubert(Hsu et al., 2021) | 34.66 | 0.0152 | 2.657 | 5.960

Table 5. Ablation study on the input audio feature. We compare the effect of
different types of speech features on our model and find that speech features
(Baevski et al., 2020; Hsu et al., 2021) trained in a self-supervised manner
have better performance.

Static Gaussian Initialization. We employ a static Gaussian initialization
stage before training our deformation field to accelerate the convergence of
our speech-driven speaker generation. The results in Table 4 show that
although the comparable image quality can still be achieved without static
initialization, it is difficult to synthesize audio-lip synchronized videos,
which suggests that a static Gaussian initialization stage can greatly
accelerate the convergence of our audio-driven deformation field and achieve
higher audio-lip synchronization accuracy.

Muti-resolution hash grid. The role of the multi-resolution hash tri-plane
encoder in our model is to obtain more compact spatial features. In the
ablation study experiment, we directly feed the 3D coordinates of Gaussians
into the deformation network and find that it degrades both the image quality
and Lip-sync. This indicates that the multi-resolution hash grid can better
learn the motion correlation between Gaussians.

Perceptual loss. We also conduct experiments in the case of not adding LPIPS
loss only using pixel-level loss as a supervisor. In the case of a tiny
difference in PSNR, we can visually find it difficult to synthesize sharp
tooth details, which shows that LPIPS loss can synthesize better visual
effects.

##  5\. CONCLUSION

In this paper, we present a novel real-time audio-driven talking face
generation framework that employs a Gaussian deformation field conditioned on
speech and poses to learn the motion of the face and torso, respectively. Our
method achieves high quality while significantly improving training and
inference efficiency compared with 2D-based and 3D NeRF-based methods.
Extensive experiments demonstrate the effectiveness of our proposed methods.

## References

  * (1)
  * Amodei et al. (2016) Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. 2016\.  Deep speech 2: End-to-end speech recognition in English and Mandarin. In _International conference on machine learning_. PMLR, 173–182.
  * Baevski et al. (2020) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in neural information processing systems_ 33 (2020), 12449–12460.
  * Chen et al. (2022) Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. 2022.  Tensorf: Tensorial radiance fields. In _European Conference on Computer Vision_. Springer, 333–350.
  * Chen et al. (2018) Lele Chen, Zhiheng Li, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. 2018.  Lip movements generation at a glance. In _Proceedings of the European conference on computer vision (ECCV)_. 520–535.
  * Chung and Zisserman (2017) Joon Son Chung and Andrew Zisserman. 2017.  Out of time: automated lip sync in the wild. In _Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13_. Springer, 251–263.
  * Du et al. (2023) Chenpeng Du, Qi Chen, Tianyu He, Xu Tan, Xie Chen, Kai Yu, Sheng Zhao, and Jiang Bian. 2023.  DAE-talker: High fidelity speech-driven talking face generation with diffusion autoencoder. In _Proceedings of the 31st ACM International Conference on Multimedia_. 4281–4289.
  * Guo et al. (2021) Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. 2021.  AD-NeRF: Audio driven neural radiance fields for talking head synthesis. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 5784–5794.
  * Gururani et al. (2023) Siddharth Gururani, Arun Mallya, Ting-Chun Wang, Rafael Valle, and Ming-Yu Liu. 2023.  SPACE: Speech-driven portrait animation with controllable expression. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 20914–20923.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ 33 (2020), 6840–6851.
  * Hsu et al. (2021) Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021.  HuBERT: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 29 (2021), 3451–3460.
  * Hu et al. (2023) Liangxiao Hu, Hongwen Zhang, Yuxiang Zhang, Boyao Zhou, Boning Liu, Shengping Zhang, and Liqiang Nie. 2023.  GaussianAvatar: Towards realistic human Avatar modeling from a single video via animatable 3D Gaussians.  _arXiv preprint arXiv:2312.02134_ (2023).
  * Hu and Liu (2023) Shoukang Hu and Ziwei Liu. 2023.  GauHuman: Articulated Gaussian splatting from monocular human videos.  _arXiv preprint arXiv:2312.02973_ (2023).
  * Kerbl et al. (2023) Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 2023.  3D Gaussian splatting for real-time radiance field rendering.  _ACM Transactions on Graphics (ToG)_ 42, 4 (2023), 1–14.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Lee et al. (2020) Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. 2020.  Maskgan: Towards diverse and interactive facial image manipulation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5549–5558.
  * Lei et al. (2023) Jiahui Lei, Yufu Wang, Georgios Pavlakos, Lingjie Liu, and Kostas Daniilidis. 2023.  Gart: Gaussian articulated template models.  _arXiv preprint arXiv:2311.16099_ (2023).
  * Li et al. (2023) Jiahe Li, Jiawei Zhang, Xiao Bai, Jun Zhou, and Lin Gu. 2023.  Efficient region-aware neural radiance fields for high-fidelity talking portrait synthesis. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 7568–7578.
  * Liu et al. (2022) Xian Liu, Yinghao Xu, Qianyi Wu, Hang Zhou, Wayne Wu, and Bolei Zhou. 2022.  Semantic-aware implicit neural audio-driven video portrait generation. In _European Conference on Computer Vision_. Springer, 106–125.
  * Loper et al. (2023) Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. 2023.  SMPL: A skinned multi-person linear model.  In _Seminal Graphics Papers: Pushing the Boundaries, Volume 2_. 851–866.
  * Lu et al. (2021) Yuanxun Lu, Jinxiang Chai, and Xun Cao. 2021.  Live speech portraits: real-time photorealistic talking-head animation.  _ACM Transactions on Graphics (TOG)_ 40, 6 (2021), 1–17.
  * Luiten et al. (2023) Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. 2023.  Dynamic 3D Gaussians: Tracking by persistent dynamic view synthesis.  _arXiv preprint arXiv:2308.09713_ (2023).
  * Mildenhall et al. (2021) Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2021.  NeRF: Representing scenes as neural radiance fields for view synthesis.  _Commun. ACM_ 65, 1 (2021), 99–106.
  * Müller et al. (2022) Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Transactions on Graphics (ToG)_ 41, 4 (2022), 1–15.
  * Park et al. (2021) Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. 2021.  Nerfies: Deformable neural radiance fields. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 5865–5874.
  * Prajwal et al. (2020) KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. 2020.  A lip sync expert is all you need for speech to lip generation in the wild. In _Proceedings of the 28th ACM international conference on multimedia_. 484–492.
  * Qian et al. (2023) Zhiyin Qian, Shaofei Wang, Marko Mihajlovic, Andreas Geiger, and Siyu Tang. 2023.  3DGS-Avatar: Animatable Avatars via deformable 3D Gaussian splatting.  _arXiv preprint arXiv:2312.09228_ (2023).
  * Schonberger and Frahm (2016) Johannes L Schonberger and Jan-Michael Frahm. 2016.  Structure-from-motion revisited. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4104–4113.
  * Shen et al. (2023) Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li, Zheng Zhu, Jie Zhou, and Jiwen Lu. 2023.  DiffTalk: Crafting diffusion models for generalized audio-driven portraits animation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 1982–1991.
  * Snavely et al. (2006) Noah Snavely, Steven M Seitz, and Richard Szeliski. 2006.  Photo tourism: exploring photo collections in 3D.  In _ACM siggraph 2006 papers_. 835–846.
  * Stypułkowski et al. (2023) Michał Stypułkowski, Konstantinos Vougioukas, Sen He, Maciej Zieba, Stavros Petridis, and Maja Pantic. 2023.  Diffused heads: Diffusion models beat GANs on talking-face generation.  _arXiv preprint arXiv:2301.03396_ (2023).
  * Sun et al. (2022) Cheng Sun, Min Sun, and Hwann-Tzong Chen. 2022.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5459–5469.
  * Suwajanakorn et al. (2017) Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. 2017.  Synthesizing Obama: learning lip sync from audio.  _ACM Transactions on Graphics (ToG)_ 36, 4 (2017), 1–13.
  * Tang et al. (2022) Jiaxiang Tang, Kaisiyuan Wang, Hang Zhou, Xiaokang Chen, Dongliang He, Tianshu Hu, Jingtuo Liu, Gang Zeng, and Jingdong Wang. 2022.  Real-time neural radiance talking portrait synthesis via audio-spatial decomposition.  _arXiv preprint arXiv:2211.12368_ (2022).
  * Thies et al. (2020) Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. 2020.  Neural voice puppetry: Audio-driven facial reenactment. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16_. Springer, 716–731.
  * Thies et al. (2016) Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. 2016.  Face2face: Real-time face capture and reenactment of RGB videos. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2387–2395.
  * Wu et al. (2023) Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 2023.  4D Gaussian splatting for real-time dynamic scene rendering.  _arXiv preprint arXiv:2310.08528_ (2023).
  * Xu et al. (2023) Yuelang Xu, Benwang Chen, Zhe Li, Hongwen Zhang, Lizhen Wang, Zerong Zheng, and Yebin Liu. 2023.  Gaussian Head Avatar: Ultra high-fidelity head Avatar via dynamic Gaussians.  _arXiv preprint arXiv:2312.03029_ (2023).
  * Yang et al. (2023) Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. 2023.  Deformable 3D Gaussians for high-fidelity monocular dynamic scene reconstruction.  _arXiv preprint arXiv:2309.13101_ (2023).
  * Ye et al. (2023) Zhenhui Ye, Ziyue Jiang, Yi Ren, Jinglin Liu, JinZheng He, and Zhou Zhao. 2023.  Geneface: Generalized and high-fidelity audio-driven 3D talking face synthesis.  _arXiv preprint arXiv:2301.13430_ (2023).
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018.  The unreasonable effectiveness of deep features as a perceptual metric. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 586–595.
  * Zhou et al. (2020) Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li. 2020.  Makelttalk: speaker-aware talking-head animation.  _ACM Transactions On Graphics (TOG)_ 39, 6 (2020), 1–15.
  * Zielonka et al. (2023) Wojciech Zielonka, Timur Bagautdinov, Shunsuke Saito, Michael Zollhöfer, Justus Thies, and Javier Romero. 2023.  Drivable 3D Gaussian Avatars.  _arXiv preprint arXiv:2311.08581_ (2023).
  * Zwicker et al. (2001) Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. 2001.  EWA volume splatting. In _Proceedings Visualization, 2001. VIS’01._ IEEE, 29–538.