#  DFA-NeRF: Personalized Talking Head Generation via Disentangled Face
Attributes Neural Rendering

Shunyu Yao RuiZhe Zhong Yichao Yan Guangtao Zhai Xiaokang Yang
Shanghai Jiao Tong University

###### Abstract

While recent advances in deep neural networks have made it possible to render
high-quality images, generating photo-realistic and personalized talking head
remains challenging. With given audio, the key to tackling this task is
synchronizing lip movement and simultaneously generating personalized
attributes like head movement and eye blink. In this work, we observe that the
input audio is highly correlated to lip motion while less correlated to other
personalized attributes (e.g., head movements). Inspired by this, we propose a
novel framework based on neural radiance field to pursue high-fidelity and
personalized talking head generation. Specifically, neural radiance field
takes lip movements features and personalized attributes as two disentangled
conditions, where lip movements are directly predicted from the audio inputs
to achieve lip-synchronized generation. In the meanwhile, personalized
attributes are sampled from a probabilistic model, where we design a
Transformer-based variational autoencoder sampled from Gaussian Process to
learn plausible and natural-looking head pose and eye blink. Experiments on
several benchmarks demonstrate that our method achieves significantly better
results than state-of-the-art methods.

##  1 Introduction

Synthesizing dynamic talking head driven by audio is of great importance to
various applications, such as film production and online meeting. However,
generating photo-realistic and expressive talking heads remains an open
challenge, which not only contain accurate lip motions, but also present
personalized eye blinks and head movements.

Traditional talking head generation models [49, 9, 47, 42, 60, 12] focus on
synthesizing audio-synchronized lip motion, but only generate lip motion with
fixed head poses. To address this issue, some recent works consider
personalized attributes [52, 55, 58, 50, 48, 7]. However, these methods [52,
55, 58] generate personalized information with a deterministic model and the
results are short of diversity, leading to a repetitive pattern. Most of works
utilize Generative Adversarial Networks (GAN) [7, 9] to generate the final
images. However, GAN is prone to fall into mode collapse and it can only
generate images with a fixed resolution. Recently, Neural Radiance Field
(NeRF) [31] achieves to render high-quality talking head images with unlimited
resolution [21, 19, 29]. Nevertheless, these works neglect the personalized
face attributes and fail to synchronize audio with lip motion accurately.

In this work, we propose a novel framework, which we name Disentangled Face
Attributes Neural Radiance Field (DFA-NeRF), to capture synchronized lip
motion and personalized attributes simultaneously. As shown in Fig.
LABEL:fig:intro, instead of directly feeding the audio into neural radiance
field [21], we propose first to predict the lip motion and personalized
attributes from audio and then optimize dynamic NeRF conditioning on these
features to synthesize talking head images. In this way, lip motion and
personalized attributes are regarded as two disentangled representations,
which guarantees the generated images to be lip-synchronized and have natural
movement.

The remaining question is how to predict these two types of features from
audios. Our critical insight is that lip motion is highly related to auditory
phonetics, while the personalized information, such as head poses and eye
blinks, is weakly associated with audio and varies from person to person.
Therefore, we argue that the key to tackling this challenge lies in two folds.
1) As audio is strongly correlated with lip motion, it is more appropriate to
approximate the lip motion with a deterministic model. Inspired by the recent
success of contrastive learning in audio-visual synchronization tasks [56,
32], we introduce a contrastive learning strategy to synchronize the audio
feature with the feature of lip motion. 2) The personalized attributes such as
head poses and eye blinks are random and probabilistic. In order to model the
distribution of the attributes and generate long time series, we propose a
probabilistic model named Transformer Variational Autoencoder (VAE). VAE [5]
can generate smooth output and map the data into Gaussian distribution, while
the attention mechanism in Transformer [44] helps VAE learn the long-time
dependence for time series. Furthermore, we model the temporal dynamics in the
Transformer-VAE with a Gaussian Process (GP). With the deterministic and
probabilistic model, our framework can generate disentangled motion features
that align well with the talking head, providing better conditions for neural
rendering.

In summary, our contributions include 1) We propose two disentangled
conditions (_i.e_., lip motion and personalized attributes) for neural
radiance field to generate high-fidelity and natural talking head. 2) We
propose a self-supervised learning method based on AutoEncoder to disentangle
lip motion and personalized attributes. 3) We design a deterministic model to
synchronize audio and lip motion and a probabilistic model to generate
personalized attributes. 4) We conduct extensive experiments on several
benchmarks, and results demonstrate our method outperforms the state-of-the-
art methods in generating high-quality talking head.

##  2 Related Work

Audio-Driven Talking Head Generation. Generating photo-realistic video
portraits in line with any input audio stream has long been a popular research
topic in computer graphics and vision [16, 17, 6]. Some methods aim at finding
out the exact correspondence between audio and frames [39, 11, 56, 27, 59, 15,
34, 38, 40].

However, these methods usually ignore head motion since it is hard to separate
head posture from facial movement. Some methods are based on the 3D face
reconstruction algorithm [20, 2, 46, 14, 12] and GAN [13, 35, 52, 28]. They
estimate intermediate representations such as 3D face shapes [24, 43, 52] or
2D landmarks [53, 45] to assist the generation process. Unfortunately, low-
dimensional intermediate representation is not capable of describing dynamic
face deformations. Unlike these methods, instead of directly using
intermediate face representations, our method only extracts the 3DMM face
expression parameters to disentangle the mouth features and eye blink features
in an implicit way. Moreover, the disentangled mouth features will synchronize
with audio features in a high-dimensional latent space. Recently, the neural
radiance field (NeRF) [31, 36, 41, 25] has been widely applied in 3D-related
tasks because it can accurately reproduce complex scenes with implicit neural
representation. Some recent works _et al_. [21, 19] leverage NeRF to represent
faces with audio features as conditions. Zhou _et al_. [57] modularizes audio-
visual representations by devising an implicit low-dimension pose code.

Our framework is also built on NeRF. Instead of directly sending the audio to
NeRF, we propose two disentangled representations to provide improved
conditions.

Figure 1: An overview of our proposed framework. Our method mainly consists of two parts: face attributes extraction and volume rendering. For the first part, pose and expression parameters are extracted from input videos. Then face attributes disentanglement module $F_{d}$ is introduced to disentangle eye blink embeddings $\mathbf{f}_{e}$ and lip motion embeddings $\mathbf{f}_{m}$. We use Transformer GP-VAE to generate personalized attributes such as head poses and eye blink features. We use a contrastive learning method for the lip motion to synchronize the audio feature $\mathbf{f}_{a}$ with the mouth movement feature $\mathbf{f}_{m}$. Afterward, in the volume rendering stage, we use generated pose $\mathbf{h}^{\prime}$ as the view direction. Meanwhile, the generated eye blink feature and synchronized audio feature are concatenated and serve as a condition $\mathbf{f}_{c}$ of the NeRF. Finally, we use volume rendering to render the image. Aspects |  ATVG[9] |  Wav2lip[34] |  MakeItTalk[58] |  LSP[30] |  Yi _et al_. [52] |  Chen _et al_. [7] |  FACIAL[54] |  AD-NeRF[21] | Ours
---|---|---|---|---|---|---|---|---|---
Target | arbitrary | arbitrary | arbitrary | specific | specific | arbitrary | arbitrary | specific | specific
Audio feature | not sync | sync | not sync | not sync | not sync | not sync | not sync | not sync | sync
Framework | GAN | GAN | GAN | GAN | GAN | GAN | GAN | NeRF | NeRF
Personalized | No | No | heads, eyes | heads,eyes | heads, eyes | heads | heads, eyes | No | heads, eyes
Face models | 2D landmarks | No | 2D landmarks | 2D landmarks | 3DMM | 3DMM | 3DMM | No | No
Table 1: We conclude five different aspects of several talking head generation
works compared to our method: the methods work for a specific target or
arbitrary images; The audio feature is synchronized with lip motions or not;
The network architecture for image synthesis; The ability to generate
personalized attributes and if they use any intermediate face models.

Personalized Face Attributes Generation. Personalized attributes are essential
for generating natural talking head and have been studied in several prior
methods. Yi _et al_. [52] use LSTM [22] to process the input audio to generate
expression and head pose. However, the generated head poses are predicted with
a deterministic model, lacking diversity. MakeItTalk [37] generates head pose
through LSTM and MLP. However, for the LSTM network, the generation process is
an auto-regressive task, making the head pose tend to _freeze_. In FACIAL
[54], GAN is applied to learn attributes of the given audio and video, whereas
the generation process is not explainable nor controllable. LSP [30] adopts a
multi-dimensional Gaussian distribution to model the probabilistic process of
head poses and body motions. The problem is that the distribution must be
learned from scratch for different targets.

We propose a Transformer GP-VAE to generate personalized attributes, such as
head poses and eye blinks. The Transformer architecture enables the model to
generate long time series, and the VAE makes the generation process
controllable and explainable. Furthermore, to model the temporal dynamics of
the face attributes sequences, we adopt Gaussian Process to sample in VAE’s
latent space. The main differences between other methods and our approach are
summarized in Table 1.

##  3 Approach

###  3.1 Overview

As shown in Fig. 1, given a source speech audio, our framework aims to
construct natural talking heads. First, we extract the pose and 3DMM face
expression parameters from video frames. We then design an AutoEncoder $F_{d}$
to disentangle the expression parameters into eye blink embeddings
$\mathbf{f}_{e}$ and mouth embeddings $\mathbf{f}_{m}$ in a self-supervised
manner. In the meantime, the input audio is sent to a CNN model to extract the
audio features $\mathbf{f}_{a}$. After that, we design a contrastive learning
method to get the synchronized audio feature $\mathbf{f}_{c}$. For head pose
$\mathbf{h}$ and eye blink embeddings $\mathbf{f}_{e}$, a Transformer VAE with
Gaussian Process is designed to model the probabilistic characteristic of
these attributes. Finally, a dynamic NeRF is introduced to generate the final
image with these conditions.

###  3.2 Face Attributes Disentanglement

We use 3D Morphable Face Models (3DMM) [33] to extract face expression
parameters. The 3D face landmark coordinates $\mathbf{S}$ can be represented
as the combination of 3DMM expression and geometry parameters:

$\mathbf{S}=\mathbf{\bar{S}}+\mathbf{B}_{id}\mathbf{F}_{id}+\mathbf{B}_{exp}\mathbf{F}_{exp},$
(1)

where $\bar{\mathbf{S}}\in\mathbb{R}^{3N}$ is the averaged facial mesh,
$\mathbf{B}_{id}$ and $\mathbf{B}_{exp}$ are the PCA basis of geometry and
expression. $\mathbf{F}_{id}$ and $\mathbf{F}_{exp}$ are the coefficients of
geometry and expression basis. Then we select 68 points in the facial mesh
$\mathbf{S}$ to get the face landmarks $\mathbf{l}$ as in FaceWarehouse [4].

Whereas the expression code of 3DMM is entangled and unexplainable because it
is based on PCA, which makes it difficult to control the movement of the mouth
and eyes separately. In order to generate disentangled face attributes, we
propose a self-supervised learning method strategy. Specifically, we design a
fully connected AutoEncoder, as shown in Fig. 2. $\mathbf{l}_{m_{A}}^{e_{A}}$
denotes the face landmarks of a target A. The subscript $m_{A}$ means the
mouth landmarks of target A and $e_{A}$ denotes the eye landmarks. The
AutoEncoder generates two intermediate embeddings $\mathbf{f}_{m}$,
$\mathbf{f}_{e}$ with an encoder, which can be represented as
$\mathbf{f}_{m_{A}},\mathbf{f}_{e_{A}}=E_{L}(\mathbf{l}_{m_{A}}^{e_{A}})$. In
the meantime, the decoder can be represented as
$\mathbf{l}_{m_{A}}^{{}^{\prime}e_{A}}=D_{L}(\mathbf{f}_{m_{A}},\mathbf{f}_{e_{A}})$,
which will be utilized to reconstruct the face landmarks with $L_{1}$ loss.
Given another face B and the corresponding face landmark
$\mathbf{l}_{m_{B}}^{e_{B}}$, we randomly switch the mouth embedding
$\mathbf{f}_{m}$ or eye blink embedding $\mathbf{f}_{e}$ with their
corresponding landmarks of face A and B. Afterwards, we can reconstruct
different landmarks with:

$\mathcal{L}_{rec}=\lVert\mathbf{l}_{m_{B}}^{{}^{\prime}e_{A}}-\mathbf{l}_{m_{B}}^{e_{A}}\rVert_{1}+\lVert\mathbf{l}_{m_{A}}^{{}^{\prime}e_{B}}-\mathbf{l}_{m_{A}}^{e_{B}}\rVert_{1}.\\\
$ (2)

In this way, the AutoEncoder can reconstruct the face landmarks with different
combinations of face attributes. Consequently, the embedding $\mathbf{f}_{e}$
and $\mathbf{f}_{m}$ can represent eyes and mouth movement in the learned
latent space.

Figure 2: Illustration of face attributes disentanglement: the mouth embedding
code is switched as an example. $\mathbf{f}_{e}$ represents the eye blink
embedding code, $\mathbf{f}_{m}$ denotes the mouth embedding code. The
subscripts $A$ and $B$ indicate different people.

###  3.3 Audio-Lip Synchronization

After obtaining the mouth embedding $\mathbf{f}_{m}$, we need to establish the
relationship between the lip motion and the input audio. As indicated by
previous works [34, 56], learning the natural synchronization between visual
mouth movements and auditory utterances is valuable for the talking face
generation. Instead of using visual clues, we choose to synchronize the mouth
movement embedding with the auditory utterances directly. Specifically, we
employ a CNN audio encoder $E_{a}$ to extract the phonemic feature
$\mathbf{f}_{a}$ from the input audio: $\mathbf{f}_{a}=E_{a}(\mathbf{a})$,
where $\mathbf{a}$ denotes the input audio data. We adopt a contrastive
learning strategy to align audio features with mouth features to seek their
synchronization. Specifically, we regard the timely aligned audio and mouth
features $(\mathbf{f}_{a},\mathbf{f}_{m})$ as a positive pair, while the non-
aligned pair $(\mathbf{f}_{a}^{-},\mathbf{f}_{m})$ is treated as negative.

We use the binary cross-entropy loss for contrastive learning, where the
distance between timely aligned audio-mouth pairs should be closer than non-
aligned pairs:

$\mathcal{L}_{con}=-\frac{1}{N}\sum^{N}_{i=1}y_{i}\log(d(\mathbf{f}_{m},\mathbf{f}_{a}))+(1-y_{i})\log(1-d(\mathbf{f}_{m},\mathbf{f}_{a}^{-})),$
(3)

where
${d}(\mathbf{f}_{a},\mathbf{f}_{m})=\frac{\mathbf{f}_{a}\cdot\mathbf{f}_{m}}{\lVert\mathbf{f_{a}}\rVert_{2}\cdot\lVert\mathbf{f_{m}}\rVert_{2}}$
denotes the cosine distance, $y_{i}=1$ for positive samples and $y_{i}=0$ for
negative samples and the total number of samples is $N$. During training, we
randomly sample timely aligned audio and mouth sequences as positive pairs,
while the negative pairs come from different videos or the same video with a
time-shift.

(a)

(b)

Figure 3: a) Structure of Transformer VAE. This module will restructure the
input face attribute sequences $\mathbf{h}$ and learn a Gaussian prior latent
space $Z$. The encoder $E_{h}$ and decoder $D$ are both Transformers. b)
Structure of Transformer-based cross-modal encoder. Audio and BOP sequences
share a common feature space, same as the trained latent space $Z$ in (a). We
use the reparameterization trick to resample new latent codes
$\mathbf{z}_{ah}$ through Gaussian Process. The trained decoder $D$ decodes
$\mathbf{z}_{ah}$ to predict future face attributes.

###  3.4 Personalized Attributes Generation

To generate personalized attributes such as eye blinks and head poses, we
propose a controllable probabilistic model named Transformer GP-VAE in Fig. 3,
aiming to generate smooth and long time series. Given a face attribute (head
poses or eye blinks) sequence $\mathbf{h}_{1:T}$ with length $T$ and a more
extended conditioning audio sequence $\mathbf{a}_{1:T^{\prime}}$ with length
$T^{\prime}$, we need to generate the face attribute embeddings
$\mathbf{h}_{T+1:T^{\prime}}$ of the future $T^{\prime}-T$ frames. Our method
consists of two major parts. $\mathbf{(1)}$ Latent space construction, where
we train a Transformer-VAE with Gaussian Process (GP) on a large dataset to
build a mapping between the input face attributes sequence and a latent space
$Z$. $\mathbf{(2)}$ Face attributes generation, where we fine-tune a cross-
modal encoder on a selected person to embed both Beginning of Pose (BOP) and
audio into the learned latent space $Z$. Thus, we can generate new face
attributes sequences in an autoregressive manner. Different from the original
GP-VAE [18] and MGP-VAE [3], our Transformer GP-VAE can generate various-
length time series with cross-modal input.

Latent Space Construction. Assume our dataset contains $N$ continuous time
series $\mathbf{h}^{(i)}\in\mathbb{R}^{T\times d}$, where $T$ is the length of
sequence and $d$ is the dimension of the vector. We construct VAE with a
Transformer illustrated in Fig. LABEL:subfig-1-1. We have also considered
LSTM, but its outputs tend to converge to a fixed value and have no
variations. The multi-head attention mechanism and fully-connected attention
in Transformer tackle this frame-freezing problem. The detailed comparison
experiments will be discussed in Section 4.

Specifically, the VAE aims to map the time series $\mathbf{h}$ to a
multivariate normal distribution (latent space $Z$) with mean vector
$\bm{\mu}$ and covariance matrix $\mathbf{K}$:

$\bm{\mu}=\bm{\mu}_{\phi}(\mathbf{h}),\mathbf{K}=\mathbf{K}_{\phi}(\mathbf{h}),$
(4)

where $\phi$ are parameters of the encoder $E_{h}$ in VAE and the posterior
distribution of $\mathbf{z}\in Z$ is presented as:

$q_{\phi}(\mathbf{z}\,|\,\mathbf{h})=\mathcal{N}\left(\bm{\mu},\mathbf{K}\right).$
(5)

Then we use the reparameterization trick to resample $\hat{\mathbf{z}}\sim
q_{\phi}(\mathbf{z}\,|\,\mathbf{h})$ and decode it into
$\hat{\mathbf{h}}=D(\hat{\mathbf{z}})$. The Mean Square Error (MSE) with batch
size $B$ is calculated to reconstruct $\hat{\mathbf{h}}$:

$\mathcal{L}_{rec}=\frac{1}{B}\sum_{i=1}^{B}\lVert\hat{\mathbf{h}}^{(i)}-\mathbf{h}^{(i)}\rVert_{2}.$
(6)

Considering the prior distribution of $\mathbf{z}\in Z$, the data points
should be $N$ i.i.d. samples in the original VAE [26]. A time sequence does
not satisfy such property due to the solid temporal correlation among frames.
To address this, we utilize Gaussian Process to model the temporal
correlation. Given the time indices $\mathbf{t}$ and the corresponding
$\mathbf{z}$ in $Z$ space, the prior distribution of $\mathbf{z}$ is:

$p(\mathbf{z})=\mathcal{N}(\bm{\mu}(\mathbf{t}),\mathbf{K(t,t)}),$ (7)

where $\bm{\mu}(\mathbf{t})$ and $\mathbf{K}(\mathbf{t},\mathbf{t})$ are
determined by the kernel function in GP, which we employ Cauchy kernel
function [18]. KL divergence is calculated between prior and posterior
distributions as another loss term:

$\mathcal{L}_{\mathrm{KL}}=\mathrm{KL}\left(p(\mathbf{z}),q_{\phi}(\mathbf{z}\,|\,\mathbf{h})\right).$
(8)

After the VAE is trained, we can get the distribution of $\mathbf{z}$ and the
corresponding latent space $Z$. As we train the VAE with a large dataset, the
model can generalize well because the latent space $Z$ contains different face
attributes of the training sequences.

Figure 4: Qualitative results. The top rows are the reference source videos.
ATVG [9] can only generate the cropped frontal talking head. LSP [30] can
generate natural head movements and eye blinks, but the lip motion is not
accurately synchronized with the input voice. Though MakeitTalk [58] can
achieve subtle head movements and eye blinks, the mouth shapes are not
accurate, which are marked with a box. The lip motion of Wav2lip [34] is
accurate, but it can only synthesize the mouth movement of the talking face.
AD-NeRF [21] use two NeRFs to generate the head, lower body separately, and we
observe a clear white gap between the head part and the body part(marked with
a box in the figure). Our method generates photo-realistic talking heads with
diverse head movements.

Face Attributes Generation. We fine-tune a cross-modal encoder on a selected
person to generate personalized face attributes illustrated in Fig.
LABEL:subfig-1-2. To generate novel face attributes, we need to encode both
audio and BOP of a selected person into the trained latent space $Z$ in VAE,
where audio serves as the conditional information and BOP is needed for
initialization. Besides, the trained encoder $E_{h}$ and decoder $D$ in VAE
will be used in this stage.

For the audio information, we employ the same audio encoder $E_{a}$ which has
been used in the audio-lip synchronization module to extract the input audio
feature $\mathbf{f}_{a}=E_{a}(\mathbf{a}_{1:T})$. In the meantime, we use the
trained encoder ($E_{h}$) in Transformer VAE to encode the BOP to the trained
latent space $Z$, _i.e_., $\mathbf{f}_{h}=E_{h}(\mathbf{h}_{1:\tau})$, where
$\tau$ is the length of BOP sequence. Afterwards, these two features are
concatenated in temporal dimension as
$\mathbf{f}_{ah}=\left[\mathbf{f}_{a}:\mathbf{f}_{h}\right]$ and fed into
$E_{ah}$ to get the mean vector $\bm{\mu}^{\prime}$ and covariance matrix
$\mathbf{K}^{\prime}$ of cross-modal latent codes $\mathbf{z}$’s distribution
like $\mathbf{z}\sim\mathcal{N}(\bm{\mu}^{\prime},\mathbf{K}^{\prime})$. We
use the reparameterization trick to resample $\mathbf{z}_{ah}$ from
$\mathcal{N}(\bm{\mu}^{\prime},\mathbf{K}^{\prime})$ with Gaussian Process.
Finally, the trained decoder ($D$) in Transformer VAE will decode
$\mathbf{z}_{ah}$ into the predicted time series
$\hat{\mathbf{h}}_{\tau+1:T}=D(\mathbf{z}_{ah})$. The loss function of
training cross-modal encoder can be calculated as:

$\mathcal{L}_{CME}=\frac{1}{B}\sum_{i=1}^{B}\left\|\,\hat{\mathbf{h}}_{\tau+1:T}^{(i)}-\mathbf{h}_{\tau+1:T}^{(i)}\,\right\|.$
(9)

In the test phase, we feed the given audio and BOP to the cross-modal encoder
to get the distribution of $\mathbf{z}_{ah}$. By resampling $\mathbf{z}_{ah}$
in this distribution and feeding $\mathbf{z}_{ah}$ to decoder $D$, we can get
the generated future face attributes. Note that we can also sample
$\mathbf{z}$ in the trained latent space $Z$ and feed it to the decoder to
generate various face attributes without the input audio, which makes our
Transformer-VAE more universal and controllable.

###  3.5 Neural Scene Representation for Talking Head

After obtaining the generated head poses, eye blink features $\mathbf{f}_{e}$
and synchronized audio features $\mathbf{f}_{a}$, we employ a neural radiance
field to generate the final image with these conditions. We first concatenate
audio embedding $\mathbf{f}_{a}$ and eye blink embedding $\mathbf{f}_{e}$ into
a new embedding $\mathbf{f}_{c}$. Then we present a conditional radiance field
with this new embedding serving as input. After we transform the head pose
from camera space into canonical space, we can directly use the head pose to
replace the view direction $\mathbf{d}$ of the radiance field. Finally, the
embedding $\mathbf{f}_{c}$, view direction $\mathbf{d}$ and 3D location
$\mathbf{x}$ in canonical space constitute the input of the implicit function
$F_{\theta}$. In practice, $F_{\theta}$ is realized by a multi-layer
perceptron. With all concatenated input vectors, the network $F_{\theta}$ will
estimate color values $\mathbf{c}$ accompanied with densities $\sigma$ along
with the dispatched rays. The entire implicit function can be formulated as:

$F_{\theta}:(\mathbf{f},\mathbf{d},\mathbf{x})\longrightarrow(\mathbf{c},\sigma).$
(10)

Then we can employ the volume rendering process by accumulating the sampled
density $\sigma$ and RGB values $\mathbf{c}$ along with the rays $r$ cast
through each pixel to compute the output color $C(\mathbf{r})$ for image
rendering results. Finally, we use the photo-metric reconstruction error
$\mathcal{L}_{photo}$ between the rendering outputs and the ground truth to
train our NeRF:

$\mathcal{L}_{photo}=\sum_{\mathbf{r}\in\mathcal{R}}{\lVert\hat{C}(\mathbf{r})-C(\mathbf{r})\rVert^{2}},$
(11)

Where $\mathcal{R}$ means the collections of the rays. Please refer to
supplementary material for more information on volume rendering and neural
radiance field.

| Testset A by [30] | Testset B by [30]
---|---|---
Method | $\text{PSNR}\uparrow$ | $\text{SSIM}\uparrow$ | $\text{LMD}\downarrow$ | $\text{Sync}\uparrow$ | blinks/s | $\text{PSNR}\uparrow$ | $\text{SSIM}\uparrow$ | $\text{LMD}\downarrow$ | $\text{Sync}\uparrow$ | blinks/s
ATVG [9] | 26.41 | 0.803 | 5.32 | 7.2 | N/A | 25.43 | 0.808 | 5.63 | 7.0 | N/A
MakeitTalk [58] | 26.33 | 0.814 | 4.0 | 7.32 | 0.37 | 27.14 | 0.822 | 7.62 | 5.2 | 0.37
Wav2Lip [34] | 27.63 | 0.854 | 5.63 | 9.8 | N/A | 27.32 | 0.851 | 5.45 | 9.5 | N/A
LSP [30] | 28.33 | 0.921 | 4.72 | 5.4 | 0.41 | 29.36 | 0.834 | 4.53 | 5.2 | 0.38
AD-NeRF [21] | 28.72 | 0.914 | 5.42 | 4.3 | 0.12 | 31.12 | 0.903 | 4.86 | 4.5 | 0.84
Ground Truth | N/A | 1.000 | 0.00 | 9.4 | 0.42 | N/A | 1.000 | 0.00 | 7.6 | 0.32
DFA-NeRF (ours) | 29.12 | 0.938 | 3.74 | 7.3 | 0.46 | 32.09 | 0.917 | 3.84 | 7.1 | 0.37
Table 2: Evaluation with different methods. We use the videos _May_ and _Obama2_ as the testset A and testset B proposed in [30]. We train a new model with the method proposed in AD-NeRF [21] and use the pre-trained models provided by all other methods to compare. We also report the eye blink frequency where N/A means the method fails to generate eye blink motions. Best results are in bold. MOS on / Approach | ATVG | MakeitTalk | Wav2lip | LSP | AD-NeRF | DFA-NeRF (ours)
---|---|---|---|---|---|---
Auido-Visual Sync | 3.42 | 2.18 | 3.76 | 3.66 | 3.65 | 3.84
Naturalness of Head Movement | 1.32 | 2.43 | 1.47 | 4.03 | 3.63 | 4.12
Naturalness of Eye Blink | 1.12 | 3.12 | 1.32 | 4.06 | 2.72 | 4.08
Realness of Image | 1.52 | 2.24 | 2.53 | 3.84 | 3.63 | 3.78
Table 3: User study analyses measured by Mean Opinion Scores. Higher is
better.

##  4 Experiments

###  4.1 Experimental Setup

Datasets. We leverage the LRS2 [10] and HDTF [55] datasets for the face
attributes disentanglement and audio to lip synchronization tasks. LRS2
consists of thousands of spoken sentences from BBC television. Each sentence
contains up to 100 characters. HDTF is a new dataset with high-resolution
videos. It is collected from youtube and consists of about 16 hours of 720P or
1080P videos. In order to compare with previous SOTA methods [21, 30], we also
adopt the training videos in AD-Nerf [21] and Live Speech Portraits [30]. For
the LRS2 and HDTF datasets, we first estimate the 3DMM face expression
parameters as in [14] and fix them to the front view. All the video data are
resampled to 25 FPS. As for the audio data, we extract the MFCC audio feature
with the windows size of 10 milliseconds and make sure that the audio
sequences and mouth movements sequences are strictly aligned.

Evaluation Metrics. We conduct quantitative evaluations on metrics previously
involved in the talking head generation field. We employ Peak Signal-to-Noise
Ratio (PSNR) [23] and Structural Similarity (SSIM) [23] to measure the image
quality. Besides, we utilize both Landmarks Distance (LMD) [8] around the
mouths and the conﬁdence score (Syncconf) proposed in SyncNet [5] to evaluate
the accuracy of mouth shapes and lip synchronization. Furthermore, we
calculate the eye blink frequency (blinks/s) to evaluate the eye blink
naturalness compared to the target person.

###  4.2 Comparison with State-of-the-arts

We compare our DFA-NeRF with the SOTA methods, including three identity-
independent approaches, _i.e_., ATVG [9], MakeItTalk [58], Wav2Lip [34] and
two identity-specific approaches, _i.e_. AD-NeRF [21] LSP [30].

Qualitative Results. As subject evaluation is crucial for talking head
generation, we show the generation results compared to the state-of-the-art
methods. As Fig. 4 shows, our method can generate more realistic results and
diverse head motions. Only our approach and AD-NeRF can generate unlimited
resolution videos among these methods. AD-NeRF[21] uses two NeRF to render the
head and torso separately and blend them. Therefore, it cannot avoid artifacts
appearing at the neck. In contrast, we adopt one NeRF to simultaneously render
the head and torso, consequently generating more appealing results.

Quantitative Results. The quantitative results are reported in Table 2. It can
be observed that our method achieves the best results under most of the
metrics on the test dataset. In Testset A, our model achieves 29.12 in PSNR
and 0.938 in SSIM, notably outperforming prior arts. Our model also achieves
the lowest LMD, which means it can predict the most accurate mouth shapes
compared to other methods. As for the experiments on eye blink frequency, we
observe that ATVG[9] and Wav2Lip[34] fail to generate the eye blink motions.
The eye blink frequency of AD-NeRF[21] is low and the eyes cannot completely
close. The eye blink frequency of MakeitTalk[58] is fixed for the two testset.
Only our method and LSP[30] can generate personalized eye blinks naturally.
The reason for the high Syncconf in Wav2Lip [45] is that it only generates the
lip movements of the input videos, and the Syncconf is even higher than the
ground truth video. The Syncconf of our method is closest to the ground truth,
which means the input audio and our generated lip motion are well
synchronized.

User Study. We conduct user studies with 20 attendees on 36 videos generated
by ours and the five other methods. The driving audio is selected from three
different languages: English, Chinese and German. We adopt the widely used
Mean Opinion Scores (MOS) [51] rating protocol. Each participant is asked to
rate from 1-5 for the talking-head generation results based on four major
aspects: audio-visual synchronization quality, the naturalness of head
movement, the naturalness of eye blink and image realness. We collect the
rating results and compute the average score of each method. The statistics
are shown in Table 3. Most of the method has comparable Audio-Visual
synchronized score except for MakeitTalk [58]. Wav2lip [34] and ATVG [9]
cannot change head poses and generate eye blink motions. Therefore, their
naturalness of head movement and eye blink scores are relatively lower.
Although LSP [30] achieves the highest score of the image realness(3.84), our
method achieves comparable results(3.78). Users prefer our results on audio-
visual synchronization, the naturalness of head movement and eye blink,
proving the effectiveness of the proposed method.

Figure 5: PCA of different people’s personalized face attributes in $Z$ space.
Left: with audio as the condition. Right: without audio as the condition.
Different kinds of colors represent different identities. We find that the
audio information helps the cross-modal encoder predict personalized face
attributes.

###  4.3 Further Analysis

Analysis of Personalized Attributes. In our framework, we employ audio to
predict the personalized attributes. To verify whether the audio information
helps identify personal attributes, we visualize the PCA results of different
people’s personalized attributes in the $Z$ space with or without audio. Fig.
5 demonstrates that the personalized attributes are more distinct conditioned
on audio.

Method | D-Rot$\,\,\downarrow$ | D-Pos$\,\,\downarrow$
---|---|---
MLP VAE | 17.81% | 20.95%
LSTM VAE | 19.06% | 22.81%
Trans VAE | 3.45% | 5.21%
Trans GP-VAE (Ours) | 3.19% | 4.47%
Table 4: Ablation studies with the structure and sample process of VAE. Trans
refers to Transformer.

Effectiveness of Personalized Attributes. Here, we analyze the importance of
the disentangled eye blink features and synchronized audio features. As shown
in Fig. 6, the eyes cannot even close without the eye blink features. Table 5
reveals the impact of different audio features. We use LMD and SyncNet
confidence scores to measure the accuracy and synchronization of the lip
motion. The results show that MFCC and DeepSpeech[1] features cannot generate
synchronized lip motions accurately. With the help of contrastive learning,
our feature can generate much better results.

Impact of VAE Structure and Gaussian Process. We conduct ablation studies on
the structure of VAE and the impact of the Gaussian Process. The results are
shown in Table 4. We use D-Rot/Pos in MakeItTalk[58] to measure head rotation
and translation prediction accuracy. It can be observed that both MLP and LSTM
cannot achieve satisfactory results, where MLP ignores the temporal relation
of the time series and LSTM tends to generate still outputs. Transformer-based
architecture improves performance significantly, proving the effectiveness of
Transformer on long time series prediction. Additionally, Gaussian Process can
capture the temporal dynamics among frames, consequently improving the final
performance. More analysis and experiments can be found in the supplementary
material.

Figure 6: Ablation study of the disentangled eye blink features. We present continuous sequences of synthesized videos. The upper row reveals that the eyes fail to blink without the feature $\mathbf{f}_{e}$. Our method can synthesize natural eye blink images with $\mathbf{f}_{e}$ Mehod | LMD$\,\,\downarrow$ | Sync${}_{conf}\,\,\uparrow$
---|---|---
MFCC | 5.71 | 4.6
Deep Speech [1] | 4.63 | 5.1
Ground Truth | 0.00 | 7.8
Ours | 3.86 | 7.3
Table 5: Ablation study for the input audio feature in NeRF. We use MFCC
features or Deep Speech features as conditions of NeRF. LMD and lip sync
scores are adopted to evaluate the lip motion accuracy. Ground Truth refers to
the ground truth scores of LMD and Syncconf. Our method achieves the best
performance.

##  5 Discussion and Conclusion

We have proposed Disentangled Face Attributes NeRF (DFA-NeRF) for talking head
generation. We argue that an excellent talking-head video requires three
properties, _i.e_., maintaining lip-syncing at a semantic level, keeping high
visual quality and containing personalized spontaneous motions. DFA-NeRF
achieves these properties w.r.t. both personalized attributes and lip motion
in talking heads with controllable head pose and eye blink. Moreover,
quantitative and qualitative experiments demonstrate that our DFA-NeRF can
synthesize high-fidelity talking head videos.

Limitation. Our method is not applicable for multiple input voices. Studies on
speaker diarization splitting up audio into homogeneous segments provide a
probable solution. Since we do not focus on fast inference, the rendering
process is time-consuming while this can be relieved by acceleration methods.

Broader Impacts. Our method can generate natural and realistic talking head
videos with few requirements. So it may bring potential ethical issues and
negative social impacts. Therefore, researchers must treat DFA-NeRF cautiously
and rationally. Meanwhile, the community should promote studies on face forge
detection to automatically keep these videos away from social media

## References

  * [1] Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al.  Deep speech 2: End-to-end speech recognition in english and mandarin.  In ICML, 2016.
  * [2] Robert Anderson, Björn Stenger, Vincent Wan, and Roberto Cipolla.  An expressive text-driven 3d talking head.  In SIGGRAPH, 2013.
  * [3] Sarthak Bhagat, Shagun Uppal, Zhuyun Yin, and Nengli Lim.  Disentangling multiple features in video sequences using gaussian processes in variational autoencoders.  In ECCV.
  * [4] Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou.  Facewarehouse: A 3d facial expression database for visual computing.  TVCG, 2014.
  * [5] Francesco Paolo Casale, Adrian V. Dalca, Luca Saglietti, Jennifer Listgarten, and Nicoló Fusi.  Gaussian process prior variational autoencoders.  2018\.
  * [6] Lele Chen, Guofeng Cui, Ziyi Kou, Haitian Zheng, and Chenliang Xu.  What comprises a good talking-head video generation?  2020\.
  * [7] Lele Chen, Guofeng Cui, Celong Liu, Zhong Li, Ziyi Kou, Yi Xu, and Chenliang Xu.  Talking-head generation with rhythmic head motion.  In ECCV, 2020.
  * [8] Lele Chen, Zhiheng Li, Ross K. Maddox, Zhiyao Duan, and Chenliang Xu.  Lip movements generation at a glance.  In ECCV, 2018.
  * [9] Lele Chen, Ross K. Maddox, Zhiyao Duan, and Chenliang Xu.  Hierarchical cross-modal talking face generation with dynamic pixel-wise loss.  In CVPR, 2019.
  * [10] Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman.  Lip reading sentences in the wild.  In CVPR, 2017.
  * [11] Joon Son Chung and Andrew Zisserman.  Out of time: Automated lip sync in the wild.  In ACCV, 2016.
  * [12] Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael J. Black.  Capture, learning, and synthesis of 3d speaking styles.  In CVPR, 2019.
  * [13] Filipe Antonio de Barros Reis, Paula Dornhofer Paro Costa, and José Mario De Martino.  Deeply emotional talking head: A generative adversarial network approach to expressive speech synthesis with emotion control.  In SIGGRAPH Posters, 2020.
  * [14] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong.  Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set.  In CVPRW, 2019.
  * [15] Pif Edwards, Chris Landreth, Eugene Fiume, and Karan Singh.  Jali: an animator-centric viseme model for expressive lip synchronization.  TOG, 2016.
  * [16] Bo Fan, Lijuan Wang, Frank K Soong, and Lei Xie.  Photo-real talking head with deep bidirectional lstm.  In ICASSP, 2015.
  * [17] Bo Fan, Lei Xie, Shan Yang, Lijuan Wang, and Frank K Soong.  A deep bidirectional lstm approach for video-realistic talking head.  Multimedia Tools and Applications, 2016.
  * [18] Vincent Fortuin, Dmitry Baranchuk, Gunnar Rätsch, and Stephan Mandt.  GP-VAE: deep probabilistic time series imputation.  In AISTATS, 2020.
  * [19] Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner.  Dynamic neural radiance fields for monocular 4d facial avatar reconstruction.  In CVPR, 2021.
  * [20] Yudong Guo, Jianfei Cai, Boyi Jiang, Jianmin Zheng, et al.  Cnn-based real-time dense face reconstruction with inverse-rendered photo-realistic face images.  PAMI, 2018.
  * [21] Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang.  Ad-nerf: Audio driven neural radiance fields for talking head synthesis.  In ICCV, 2021.
  * [22] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural Computation, 1997.
  * [23] Alain Horé and Djemel Ziou.  Image quality metrics: PSNR vs. SSIM.  In ICPR, 2010.
  * [24] Tero Karras, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen.  Audio-driven facial animation by joint end-to-end learning of pose and emotion.  TOG, 2017.
  * [25] Petr Kellnhofer, Lars C Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, and Gordon Wetzstein.  Neural lumigraph rendering.  In CVPR, 2021.
  * [26] D. P. Kingma and M. Welling.  Auto-encoding variational bayes.  In ICLR, 2014.
  * [27] Prajwal KR, Rudrabha Mukhopadhyay, Jerin Philip, Abhishek Jha, Vinay Namboodiri, and CV Jawahar.  Towards automatic face-to-face translation.  In ACMMM, 2019.
  * [28] Avisek Lahiri, Vivek Kwatra, Christian Frueh, John Lewis, and Chris Bregler.  Lipsync3d: Data-efficient learning of personalized 3d talking faces from video using pose and lighting normalization.  In CVPR, 2021.
  * [29] Stephen Lombardi, Tomas Simon, Jason M. Saragih, Gabriel Schwartz, Andreas M. Lehrmann, and Yaser Sheikh.  Neural volumes: learning dynamic renderable volumes from images.  TOG, 2019.
  * [30] Yuanxun Lu, Jinxiang Chai, and Xun Cao.  Live Speech Portraits: Real-time photorealistic talking-head animation.  TOG, 2021.
  * [31] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [32] Arsha Nagrani, Samuel Albanie, and Andrew Zisserman.  Seeing voices and hearing faces: Cross-modal biometric matching.  In CVPR, 2018.
  * [33] Stylianos Ploumpis, Haoyang Wang, Nick E. Pears, William A. P. Smith, and Stefanos Zafeiriou.  Combining 3d morphable models: A large scale face-and-head model.  In CVPR, 2019.
  * [34] KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar.  A lip sync expert is all you need for speech to lip generation in the wild.  In ACMMM, 2020.
  * [35] Albert Pumarola, Antonio Agudo, Aleix M. Martínez, Alberto Sanfeliu, and Francesc Moreno-Noguer.  Ganimation: Anatomically-aware facial animation from a single image.  In ECCV, 2018.
  * [36] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In CVPR, 2021.
  * [37] Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, and Mark Hasegawa-Johnson.  Autovc: Zero-shot voice style transfer with only autoencoder loss.  In ICML, 2019.
  * [38] Najmeh Sadoughi and Carlos Busso.  Speech-driven expressive talking lips with conditional sequential generative adversarial networks.  IEEE Transactions on Affective Computing, 2019.
  * [39] Oliver Schreer, Roman Englert, Peter Eisert, and Ralf Tanger.  Real-time vision and speech driven avatars for multimedia applications.  TMM, 2008.
  * [40] Taiki Shimba, Ryuhei Sakurai, Hirotake Yamazoe, and Joo-Ho Lee.  Talking heads synthesis from audio with deep neural networks.  In SII, 2015.
  * [41] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  2019\.
  * [42] Yang Song, Jingwen Zhu, Dawei Li, Xiaolong Wang, and Hairong Qi.  Talking face generation by conditional recurrent adversarial network.  IJCAI, 2019.
  * [43] Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner.  Neural voice puppetry: Audio-driven facial reenactment.  In ECCV, 2020.
  * [44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NIPS, 2017.
  * [45] Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy.  Mead: A large-scale audio-visual dataset for emotional talking-face generation.  In ECCV, 2020.
  * [46] Lijuan Wang, Wei Han, and Frank K Soong.  High quality lip-sync animation for 3d photo-realistic talking head.  In ICASSP, 2012.
  * [47] Lijuan Wang, Xiaojun Qian, Wei Han, and Frank K. Soong.  Synthesizing photo-real talking head via trajectory-guided sample selection.  In INTERSPEECH, 2010.
  * [48] Xinsheng Wang, Qicong Xie, Jihua Zhu, Lei Xie, et al.  Anyonenet: Synchronized speech and talking head generation for arbitrary person.  arXiv preprint arXiv:2108.04325, 2021.
  * [49] Xin Wen, Miao Wang, Christian Richardt, Ze-Yin Chen, and Shi-Min Hu.  Photorealistic audio-driven video portraits.  TVCG, 2020.
  * [50] Haozhe Wu, Jia Jia, Haoyu Wang, Yishun Dou, Chao Duan, and Qingshan Deng.  Imitating arbitrary talking style for realistic audio-driven talking face synthesis.  In ACMMM, 2021.
  * [51] Jie Xu, Liyuan Xing, Andrew Perkis, and Yuming Jiang.  On the properties of mean opinion scores for quality of experience management.  In ISM, 2011.
  * [52] Ran Yi, Zipeng Ye, Juyong Zhang, Hujun Bao, and Yong-Jin Liu.  Audio-driven talking face video generation with learning-based personalized head pose.  arXiv preprint arXiv:2002.10137, 2020.
  * [53] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky.  Few-shot adversarial learning of realistic neural talking head models.  In ICCV, 2019.
  * [54] Chenxu Zhang, Yifan Zhao, Yifei Huang, Ming Zeng, Saifeng Ni, Madhukar Budagavi, and Xiaohu Guo.  Facial: Synthesizing dynamic talking face with implicit attribute learning.  In ICCV, 2021.
  * [55] Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan.  Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset.  In CVPR, 2021.
  * [56] Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang.  Talking face generation by adversarially disentangled audio-visual representation.  In AAAI, 2019.
  * [57] Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu.  Pose-controllable talking face generation by implicitly modularized audio-visual representation.  In CVPR, 2021.
  * [58] Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li.  Makelttalk: speaker-aware talking-head animation.  TOG, 2020.
  * [59] Yang Zhou, Zhan Xu, Chris Landreth, Evangelos Kalogerakis, Subhransu Maji, and Karan Singh.  Visemenet: Audio-driven animator-centric speech animation.  TOG, 2018.
  * [60] Hao Zhu, Huaibo Huang, Yi Li, Aihua Zheng, and Ran He.  Arbitrary talking face generation via attentional audio-visual coherence learning.  IJCAI, 2020.