# Masked Lip-Sync Prediction by Audio-Visual Contextual Exploitation in
Transformers

Yasheng Sun  Tokyo Institute of TechnologyTokyoJapan<EMAIL_ADDRESS>,
Hang Zhou  Baidu Inc.ShanghaiChina<EMAIL_ADDRESS>,  Kaisiyuan Wang  The
University of SydneySydneyAustralia<EMAIL_ADDRESS>,  Qianyi Wu
Monash UniversityMelbourneAustralia<EMAIL_ADDRESS>,  Zhibin Hong  Baidu
Inc.ShenzhenChina<EMAIL_ADDRESS>,  Jingtuo Liu  Baidu Inc.BeijingChina
<EMAIL_ADDRESS>,  Errui Ding  Baidu Inc.BeijingChina<EMAIL_ADDRESS>,  Jingdong Wang  Baidu Inc.BeijingChina<EMAIL_ADDRESS>,  Ziwei Liu
Nanyang Technological UniversitySingaporeSingapore<EMAIL_ADDRESS>and
Hideki Koike  Tokyo Institute of TechnologyTokyoJapan<EMAIL_ADDRESS>
(2022)

###### Abstract.

Previous studies have explored generating accurately lip-synced talking faces
for arbitrary targets given audio conditions. However, most of them deform or
generate the whole facial area, leading to non-realistic results. In this
work, we delve into the formulation of altering only the mouth shapes of the
target person. This requires masking a large percentage of the original image
and seamlessly inpainting it with the aid of audio and reference frames. To
this end, we propose the Audio-Visual Context-Aware Transformer (AV-CAT)
framework, which produces accurate lip-sync with photo-realistic quality by
predicting the masked mouth shapes. Our key insight is _to exploit desired
contextual information provided in audio and visual modalities thoroughly with
delicately designed Transformers_. Specifically, we propose a convolution-
Transformer hybrid backbone and design an attention-based fusion strategy for
filling the masked parts. It uniformly attends to the textural information on
the unmasked regions and the reference frame. Then the semantic audio
information is involved in enhancing the self-attention computation.
Additionally, a refinement network with audio injection improves both image
and lip-sync quality. Extensive experiments validate that our model can
generate high-fidelity lip-synced results for arbitrary subjects.

Figure 1. Qualitative results of our method. We focus on the task of high-
fidelity person-agnostic lip-sync generation, which modifies the mouth shapes
of any target template video according to the audio source. Here our lip-sync
results should have the same mouth shape as the synced video to the audio
source. The figures are selected from VoxCeleb (Nagrani et al., 2017) and
VoxCeleb2 (Chung et al., 2018) ©Visual Geometry Group (CC BY).

Lip-Sync, Transformer, Audio-Visual Learning.

††submissionid: 277††journalyear: 2022††copyright: acmcopyright††conference:
SIGGRAPH Asia 2022 Conference Papers; December 6–9, 2022; Daegu, Republic of
Korea††booktitle: SIGGRAPH Asia 2022 Conference Papers (SA ’22 Conference
Papers), December 6–9, 2022, Daegu, Republic of Korea††price: 15.00††doi:
10.1145/3550469.3555393††isbn: 978-1-4503-9470-3/22/12††ccs: Computing
methodologies Animation††ccs: Computing methodologies Neural networks

##  1\. Introduction

Driving human mouth movements with speech audio is in great need in the field
of virtual human creation, entertainment, the film industry, and the detection
of deep fakes. The ability to drive arbitrary target video can vastly improve
the applicability of such models. However, previous studies focusing on
arbitrary targets mostly operate on the whole face (Jamaludin et al., 2019;
Zhou et al., 2019; Zhou et al., 2020; Sun et al., 2021; Zhou et al., 2021;
Chen et al., 2018, 2019; Vougioukas et al., 2019; Chen et al., 2020; Song et
al., 2018; Wu et al., 2021; Wang et al., 2021, 2022; Ji et al., 2022). They
either reconstruct (Zhou et al., 2019, 2021; Zhou et al., 2020) or warp (Wang
et al., 2021)] the whole head with generative models. As the generated moving
head cannot be attached to a body, results under such a protocol can hardly be
used in real-world applications.

Modifying only the mouth movements of a target video with audio input is an
intriguing property that can be directly applied to scenarios like audio
dubbing. Previous researchers investigate the mouth modification problem
mainly in case that relatively long videos are available for training (Thies
et al., 2020; Song et al., 2020; Ji et al., 2021; Suwajanakorn et al., 2017).
These methods suffer from two main drawbacks: 1) They cannot be directly
applied to any-length videos, which limits their applicability. 2) They rely
on intermediate structural priors such as 2D/3D landmarks or 3D models. The
inaccuracy in such predictions would impede the audio-to-mouth prediction
procedure. Only a few studies (Prajwal et al., 2020; Park et al., 2022) are
proposed to tackle the problem of audio-driven lip-sync for an arbitrary
subject. Wav2Lip (Prajwal et al., 2020) specifically recovers the lower half
of a face with speech and a reference frame. Though their mouth movements
match nicely with the audio, they can only produce results with low quality,
particularly around the mouth.

We identify that such a setting of modifying an arbitrary target image with
audio is a particular type of conditional masked image prediction (Chang et
al., 2022), which we refer to as _masked lip-sync prediction_. Specifically,
we mask most of the lower face. Two types of _contextual_ information,
including the textural appearance information and the semantic mouth
information, are needed for filling the masked areas. They can be extracted
from three perspectives: 1) The visible part of the masked image itself which
contains pose and environment information; 2) An appearance reference image;
and 3) the semantic speech content information from audio. Balancing such
_contextual information_ requires modeling non-local intrinsic, non-aligned
extrinsic, and cross-modal information. This is particularly challenging for
convolutional networks that rely on deeper layers for fusing distant contexts,
especially when the expected quality of the generated results is high. The
objective of preserving more details from the reference images naturally
contradicts the goal of mouth modifications with audio.

In this work, we achieve masked lip-sync prediction for an arbitrary person
through our Audio-Visual Context-Aware Transformer (AV-CAT) and successfully
produce high-quality results. Our key insight is that _the desired semantic
and appearance contextual information provided in both audio-visual modalities
can be thoroughly exploited with a delicately designed Transformer structure_.
We choose the backbone network inspired by recent advances on Vision
Transformers (Vaswani et al., 2017; Dosovitskiy et al., 2021; Liu et al.,
2021) and their applications in image modeling (Bao et al., 2021; Chen et al.,
2022a; Chang et al., 2022; Esser et al., 2021a). As the attention mechanisms
(Wang et al., 2018a) can naturally treat information at arbitrary positions
equally, Transformers have already been leveraged in generative tasks
including super resolution (Yang et al., 2020), inpainting (Zheng et al.,
2021; Li et al., 2022; Yu et al., 2021) and image synthesis (Esser et al.,
2021b; Zhang et al., 2021; Chang et al., 2022). They naturally suit our target
of fusing the contextual information across frames and modalities. .

In detail, we adopt a lightweight hybrid convolution-Transformer backbone
structure to cope with the fusion challenge. It takes advantage of both the
inductive bias in convolution layers and explores patch-wise cross-frame and
cross-modal information fusion in Transformers. We design a Cross-modal
Contextual Fusion Transformer block (CCF-Transformer) which consists of a
cross-frame mutual attention and a cross-modal audio information fusion
strategy based on basic Swin Transformer (Liu et al., 2021) designs. While it
accounts mostly for coarse-grained mouth movements, it fails to model delicate
details. To cope with this issue, we propose an audio-injected Refinement
Network. It is a UNet-like structure with modulated convolutions conditioned
on audio. With this refinement module, both the generation quality and the
mouth movement accuracy can be improved. Extensive experiments validate the
effectiveness of each model. Through our delicately designed framework, our
model is able to generate lip-synced results with high fidelity.

Our contributions can be summarized as follows: 1) We leverage cross-frame and
cross-modal context information for masked lip-sync prediction with delicately
designed attention blocks. 2) We propose the refinement network, which
enhances the generated image quality and lip-sync results with the modulated
injection of audio information. 3) Extensive experiments validate that our
Audio-Visual Context-Aware Transformer framework generates high-fidelity lip-
sync results for arbitrary subjects.

##  2\. Related Work

###  2.1. Audio-Driven Lip-Synced Talking Face

Driving a target portrait with speech audio has long been a popular research
topic in the computer graphics and computer vision communities. Generally,
previous approaches can be classified into person-specific and person-agnostic
settings.

####  2.1.1. Person-Specific Lip-Sync Synthesis.

Most person-specific modeling methods generate high video quality
(Suwajanakorn et al., 2017; Thies et al., 2020; Song et al., 2020; Ji et al.,
2021; Guo et al., 2021; Lahiri et al., 2021; Li et al., 2021; Liu et al.,
2022). Some of them focus on altering the mouth areas for photo-realistic
results (Suwajanakorn et al., 2017; Thies et al., 2020; Song et al., 2020).
Suwajanakorn et al. (2017) design an audio-to-mouth module to synthesize high-
quality talking face videos of Obama. Song et al. (2020) introduce 3DMM (Blanz
and Vetter, 1999) to translate the speech content into the landmarks of the
mouth area. Thies et al. (2020) leverage a similar idea for mouth rendering.
On the other hand, Guo et al. (2021) and Liu et al. (2022) use NeRF, Ji et al.
(2021) use 2d landmarks and Lahiri et al. (2021) use neural rendering to
synthesize the whole talking faces. All the above results enjoy high video
quality, however, their methods cannot be applied to a person with few
(seconds of) data, which limits their applicability.

Figure 2. Overview of our _Audio-Visual Context-Aware Transformer (AV-CAT)_
framework. It consists of a convolution-Transformer-hybrid backbone network
and a Refinement Network. The Reference image $I_{r}$ is processed in the
reference branch with Basic Transformer stages (Basic-TS). The audio
information is processed in the audio branch. Both this information is fused
to the CCF-Transformer stages (CCF-TS) in the main branch. The figures are
selected from VoxCeleb (Nagrani et al., 2017) ©Visual Geometry Group (CC BY).

####  2.1.2. Person-Agnostic Lip-Sync Synthesis

Other methods focus on the person-agnostic setting where only one image or a
short clip is provided as a reference. Speech2Vid (Jamaludin et al., 2019)
studies the problem for the first time in an end-to-end manner. Then Chen et
al. (2019) leverage 2D facial landmarks as guidance. Zhou et al. (2020)
propose to use 3D landmark displacements and synthesize videos including head
poses and blinks. Zhou et al. (2021) choose to modularize audio-visual
representations and manage to generate taking faces with different head poses.
Particularly, Yi et al. (2020) and Chen et al. (2020) intend to generate more
rhythmic realistic results with a short clip. However, methods modeling the
whole head tend to change the background and body deformation, making them
less realistic.

Specifically, Wav2Lip (Prajwal et al., 2020) focuses on the mouth areas and
inpaints the lower half of the face using speech audio and another set of
reference frames. However, they rely on CNN-based UNet with skip-connections
and concatenate audio information to the bottleneck of the network, which
leads to blurry results. In the pursuit of high-quality and person-agnostic
lip-sync, we adopt a similar setting as Wav2Lip and identify the problem as
masked lip-sync prediction. Our method can synthesize clearly more realistic
results than previous person-agnostic methods.

###  2.2. Image Synthesis with Transformers

The Transformer architecture (Vaswani et al., 2017) has received growing
interest from various tasks in computer vision (Dosovitskiy et al., 2021; Bao
et al., 2021; Liu et al., 2021; Esser et al., 2021a; Chang et al., 2022; Li et
al., 2022; He et al., 2021). Recently, Transformer has also extended its reach
to low-level vision tasks, such as image synthesis (Van Den Oord et al., 2017;
Esser et al., 2021a) and image restoration (Yang et al., 2020, 2021; Zheng et
al., 2021; Li et al., 2022; Yu et al., 2021).

For conditional image restoration tasks (e.g., super-resolution and
inpainting), Yang et al. (2020) propose a texture transformer network for
image super-resolution by leveraging different high/low-resolution references.
Yang et al. (2021) present a transformer-based framework for continuous image
super-resolution with implicit function. Li et al. (2022) introduce a mask-
aware transformer (MAT) for large-hole inpainting. Similar to the Transformer-
based approaches mentioned above, our AV-CAT also aims to fill in the masked
region and synthesize high-fidelity images with reasonable textural appearance
and correct semantic mouth movements.

##  3\. Methodology

In this section, we introduce our Audio-Visual Context-Aware Transformer (AV-
CAT) as illustrated in Fig. 2. The whole framework contains a convolution-
Transformer-hybrid backbone network that accounts for information fusion at
coarse level and a convolutional Refinement Network that fixes details. We
will first introduce our masked lip-sync prediction training setting in
Section 3.1. Then we detailedly depict the design of the backbone network and
information fusion strategies. Finally, we introduce the Refinement Network
and the training objectives.

###  3.1. Training Setting

Our masked lip-sync prediction setting is similar to Wav2Lip (Prajwal et al.,
2020). Given a training video $\textbf{V}=\\{I_{1},\dots,I_{T}\\}$ with
corresponding audio clip $\textbf{a}=\\{a_{1},\dots,a_{T}\\}$, our model
processes one target frame $I_{t}\in\textbf{V}$ at a time. We mask out most
areas on the lower half of $I_{t}$ to $I^{m}_{t}=M*I_{t}$ with a fixed mask
$M$. The training goal is to recover $I_{t}$ with the corresponding audio
$a_{t}$. To recover the mouth-shape irrelevant information such as textures on
the face as well as the masked hair and backgrounds, a reference frame
$I_{r}\in\textbf{V}$ is involved. We expect the reference frame to provide
textual information only and avoid exacting the mouth shapes directly from it.
Thus it is sampled from the same video but at a different timestamp. The
reference frame and the target can be selected as the same during testing.

###  3.2. Audio-Visual Context-Aware Transformer

The convolution-Transformer-hybrid backbone network consists of three
branches: the _main branch_ , _reference branch_ and the _audio branch_. They
target processing the target frame, reference frame, and audio information,
respectively. The main branch consists of a convolutional encoder head, 5
_Cross-Modal Contextual Fusion Transformer (CCF-Transformer)_ stages, and a
convolutional decoder. Specifically, the CCF-Transformer is built upon the
_Basic Transformer_ blocks modified from Swin Transformers (Liu et al., 2021).
We design our reference branch without involving extra parameters. It uses the
same convolutional encoder $E_{e}$ for feature map encoding. Moreover, it
consists of 5 stages of Basic-Transformer blocks which share all learnable
weights with CCF-Transformer’s sub-modules. The audio branch encodes an audio
feature $f_{a}$ with encoder $E_{a}$. Both $f_{a}$ and intermediate features
of the Basic-Transformer blocks are sent into the CCF-Transformer blocks for
information fusion.

####  3.2.1. Convolutional Encoder and Decoder.

We choose to preserve 4 layers of convolution operations for the low-level
representation downsampling in encoder $E_{e}$ and another 4 for upsampling in
decoder $Net_{d}$. It has been verified that the inductive bias at early
layers is effective for Transformer learning (Raghu et al., 2021; Xiao et al.,
2021; Li et al., 2022). Specifically, the convolutional encoder $E_{e}$
processes an image to a feature map F with $1/4$ the size of $I_{t}$. It
encodes the target and the reference frame to feature maps $\textbf{F}_{t}$
and $\textbf{F}_{r}$ respectively.

####  3.2.2. Basic Transformer Structure.

The reference feature map $\textbf{F}_{r}\in\mathbb{R}^{h\times w\times
C_{0}}$ is then sent into our Basic Transformer of 5 stages, each containing
several blocks. It is built based on Swin Transformer (Liu et al., 2021) with
modifications on the hierarchical representations inspired by (Li et al.,
2022), including the removal of the layer normalization. While Swin
Transformer downsamples the feature patches at each stage by fusing
neighboring $2\times 2$ patches, we consider a downsample-upsample strategy
given our generative task property. At the beginning of the first three
stages, we downsample the feature map with a stride-2 convolution with
$2\times 2$ kernel size. Differently, we upsample the feature maps at the last
two stages with transposed convolutions.

The self-attentions is computed within local windows of size $w_{w}\times
w_{w}$ as performed in (Liu et al., 2021), which saves computational cost. We
adopt the standard multi-head self-attention (MSA) in the Basic Transformer
blocks:

(1)
$\displaystyle\text{Attention}(\textbf{Q},\textbf{K},\textbf{V})=\text{Softmax}({\textbf{Q}\textbf{K}^{\text{T}}}/{\sqrt{d_{k}}})\textbf{V},$

where Q, K, V are all tokens within the same window. After each attention
operation, the window is shifted by $(\lfloor w_{w}/2\rfloor,\lfloor
w_{w}/2\rfloor)$ pixels.

We visualize the architecture of one Basic Transformer block in Fig. 3 (a).
Note that all the learnable parameters in the Basic Transformer are jointly
learned with CCF-Transformer. This is to ensure the same semantics among both
Transformer blocks and memory saving. The block is modified according to (Li
et al., 2022).

####  3.2.3. Cross-Modal Contextual Fusion Transformer Blocks.

The target’s feature map $\textbf{F}_{t}\in\mathbb{R}^{h\times w\times C_{0}}$
is sent into CCF-Transformer which shares similar setups as the Basic
Transformer, e.g., the resolution of each stage, the sliding window operation,
and the computation of self-attentions. Differently, the CCF-Transformer
receives extra contextual information from the reference branch and the audio
branch: 1) The cross-frame reference information provided from the Basic
Transformer blocks. It is fused to CCF-Transformer blocks. 2) The cross-modal
audio information is mapped to the last 3 stages through a fully connected
layer.

Detailedly, at each block of the CCF-Transformer, an additional multi-head
mutual attention (MMA) is developed in parallel with MSA between the target
feature map $F_{t\\{k,l\\}}$ and reference feature map $F_{r\\{k,l\\}}$, where
$l$ and $k$ denote the $l$th block of the $k$th stage. In the MMA, the tokens
from $F_{t\\{k,l\\}}$ are treated as queries and $F_{r\\{k,l\\}}$ serve as
keys and values. This is to retrieve desired appearance information for
inpainting missing areas. Then, the MMA and MSA results are concatenated
together and mapped to the original feature dimension by a fully connected
layer.

Then for the last three stages of CCF-Transformer, the audio feature $f_{a}$
is mapped to $f_{a\\{k,l\\}}$ and sent into the Transformer blocks to operate
on the fused feature mentioned above. The overall computation of one CCF-
Transformer block can be written as:

(2) $\displaystyle
z^{i}_{\\{k,l\\}}=\text{FC}_{1}(\text{Cat}[\textbf{MSA}(F_{t\\{k,l\\}}),\textbf{MMA}(F_{r\\{k,l\\}},F_{t\\{k,l\\}})]),$
(3) $\displaystyle
z^{i}_{\\{k,l\\}}=\text{FC}_{2}(f_{a\\{k,l\\}}+z^{i}_{\\{k,l\\}})~{}~{}(\text{if
k > 2}),$ (4) $\displaystyle
z^{ii}_{\\{k,l\\}}=z^{i}_{\\{k,l\\}}+F_{t\\{k,l\\}},$ (5) $\displaystyle
F_{t\\{k,l+1\\}}=\text{MLP}(z^{ii}_{\\{l,k\\}})+z^{ii}_{\\{l,k\\}},$

which renders the feature map $F_{t\\{k,l+1\\}}$ of the next block. In the
equations, Cat[] denotes feature concatenation and $\text{FC}_{i}$ are
different fully connected layers that are not shared with Basic Transformer
blocks. Notably, the intuition for only injecting three layers of audio
information is that audio usually interacts with high-level visual features
(Zhou et al., 2019; Prajwal et al., 2020). Experiments show that injecting
audio into all stages makes little difference.

The final output of the CCF-Transformer is sent into the convolutional decoder
$Net_{d}$ to predict an intermediate result $\hat{I}^{p}_{t}$.

Figure 3. The architecture of our modified transformer blocks. (a) shows one
block of the Basic Transformer in one stage, and (b) shows one block of the
CCF-Transformer.

###  3.3. Refinement Network

Note that though the convolution-Transformer-hybrid backbone (CTH backbone)
manages to fuse effective information, it fails at handling fine-grained
details. On the other hand, we identify that UNet-like (Ronneberger et al.,
2015) structures are suitable for improving roughly good results. Thus, we
propose the Refinement Network $G_{R}$. It takes the $\hat{I}^{p}_{t}$
predicted from the CTH backbone and the original masked target
$I^{M}_{t}=M*I_{t}$ for generating more harmonic results with higher quality.
Specifically, the input of the Refinement Network is
$\hat{I}^{p^{\prime}}_{t}=M*I_{t}+(\textbf{1}-M)*\hat{I}^{p}_{t}$ and it
predicts $I^{p}_{t}=G_{R}(\hat{I}^{p^{\prime}}_{t})$.

Specifically, motivated by previous research on talking head generation that
audio features can be effectively expressed with modulated convolution (Zhou
et al., 2021), we adopt a similar protocol for enhancing cross-modal learning.
We map audio feature $f_{a}$ to a style vector $s$ for each convolution
operation in the UNet. For each value $W_{mnq}$ in any convolution kernel
weight $W$, where $m$ channel-wise position of the input, $n$ is its position
on the output channel and $q$ stands for spatial location, we modulate it
according to $s$’s given its channel-wise position $m$:

(6) $\displaystyle W^{\prime}_{mnq}=\frac{s_{m}\cdot
W_{mnq}}{\sqrt{{\sum}_{m,q}(s_{m}\cdot W_{mnq})^{2}+\epsilon}},$

where $\epsilon$ is set as a small value to avoid numerical errors.

###  3.4. Learning Objectives

The Learning objectives are mainly the VGG and GAN losses that are leveraged
in various generation tasks (Isola et al., 2017; Wang et al., 2018b; Zhou et
al., 2021). Normally the VGG loss is used for perceptual similarity. Here we
extend the VGG loss by computing from the first layer of the convolution to
the $N_{vgg}$’s layer. This accounts for low-level information reconstruction.
The loss functions are written as:

(7)
$\displaystyle\begin{split}\mathcal{L}_{\text{GAN}}=&~{}\underset{\text{G}}{\text{min}}\underset{{D}}{\text{max}}(\mathbb{E}_{I_{t}}[\log{D}(I_{t})]\\\
&+\mathbb{E}_{I^{p}_{t}}[\log(1-{D}(I^{p}_{t}))]),\\\ \end{split}$ (8)
$\displaystyle\mathcal{L}_{\text{vgg}}=\sum_{n=1}^{N_{P}}{\|\text{VGG}_{n}({I}_{t})-\text{VGG}_{n}(I^{p}_{t})\|_{1}},$

where $D$ denotes a discriminator.

Besides, we also re-train the SyncNet (Chung and Zisserman, 2016b)
discriminator for constraining temporal consistency and audio-lip
synchronization as performed in Wav2Lip (Prajwal et al., 2020) . It takes 5
consecutive predicted frames $I^{p}_{t:t+5}$ as input. The loss is carried out
between $I^{p}_{t:t+5}$ and their corresponding audio representations
$a_{t:t+5}$.

(9)
$\displaystyle\mathcal{L}_{sync}=\text{SyncNet}(åI^{p}_{t:t+5},a_{t:t+5}).$

The overall learning objective can be summarized as:

(10)
$\displaystyle\mathcal{L}_{all}=\mathcal{L}_{\text{GAN}}+\lambda_{vgg}\mathcal{L}_{VGG}+\lambda_{sync}\mathcal{L}_{sync},$

Figure 4. Qualitative Results. The top row shows the corresponding videos of the input audio. PC-AVS (Zhou et al., 2021) produces accurate lips, it is constrained to the cropped area. MakeitTalk (Zhou et al., 2020) fails to generate accurate mouth shape and lacks head dynamics. Wav2Lip (Prajwal et al., 2020) syncs well with audios, however, the mouth areas are blurry with their method. Our AV-CAT generates realistic results with synced mouth movements. The figures are selected from VoxCeleb ©Visual Geometry Group (CC BY) and LRW ©BBC. Table 1. Quantitative results on LRW and VoxCeleb. For LMD the lower the better, and the higher the better for other metrics. | LRW (Chung and Zisserman, 2016a) | VoxCeleb (Nagrani et al., 2017)
---|---|---
Method | $\text{SSIM}\uparrow$ | $\text{PSNR}\uparrow$ | LMD $\downarrow$ | $\text{Sync}_{conf}\uparrow$ | $\text{SSIM}\uparrow$ | $\text{PSNR}\uparrow$ | LMD $\downarrow$ | $\text{Sync}_{conf}\uparrow$
Wav2Lip (Prajwal et al., 2020) | 0.937 | 34.25 | 2.54 | 6.7 | 0.885 | 32.34 | 8.44 | 5.2
MakeitTalk (Zhou et al., 2020) | 0.690 | 31.09 | 5.03 | 3.1 | 0.814 | 29.51 | 29.13 | 2.3
PC-AVS (Zhou et al., 2021) | 0.895 | 33.87 | 3.04 | 6.2 | 0.865 | 32.67 | 8.91 | 5.4
Ground Truth | 1.000 | 100.00 | 0.00 | 6.5 | 1.000 | 100.0 | 0.00 | 5.4
AV-CAT (Ours) | 0.938 | 36.34 | 2.83 | 6.0 | 0.889 | 33.41 | 8.64 | 5.1

##  4\. Experiments

##### Datasets.

All the experiments are conducted on two audio-visual datasets LRW (Chung and
Zisserman, 2016a) and VoxCeleb (Nagrani et al., 2017). Both of them are
composed of in-the-wild data.

  * •

LRW (Chung and Zisserman, 2016a). LRW is an audio-visual dataset collected
from BBC news with high visual quality. This dataset consists of 1,000
utterances in total covering 500 words, and each utterance lasts about 1
second.

  * •

VoxCeleb (Nagrani et al., 2017). VoxCeleb is a large-scale audio-visual
dataset including 1,251 celebrities. The videos in VoxCeleb are all longer
than 3 seconds but distributed in different visual qualities.

In our experiments, we follow their train/test split strategies to train and
test our AV-CAT framework. Our method is trained on LRW and about one-fifth of
VoxCeleb.

##### Implementation Details.

We process all videos at 25 fps at the size of $256\times 256$. We adopt a
similar setting as (Prajwal et al., 2020) but leave 16 pixels more on the left
and right sides unmasked. Thus 43.75% of the input image is masked for every
frame. All faces in all frames are aligned according to the eyes during
training and inference. We then follow the audio processing of (Zhou et al.,
2021), with 16kHz sampling rate and converted to mel-spectrograms with FFT
window size 1280, hop length 160, and 80 Mel filter-banks. The mel-spectrogram
corresponds to the time-length of 5 frames with the target frame in the middle
is sampled as the condition. At the 5 stages in both Basic Transformer and
CCF-Transformer, there are {2, 3, 4, 3, 2} blocks. The window sizes are {8,
16, 16, 16, 8}, respectively. The $\lambda_{vgg}$ is set as 1 and
$\lambda_{sync}$ empirically set as 0.0001. Note that the network processes
each frame independently during inference. The consecutive frames are only
used in SyncNet during training.

##### Comparing Methods.

We compare our AV-CAT framework with three person-agnostic state-of-the-art
methods, MakeitTalk (Zhou et al., 2020), PC-AVS (Zhou et al., 2021) and
Wav2Lip (Prajwal et al., 2020). MakeitTalk and PC-AVS all synthesize talking
heads with head pose movements. Wav2Lip (Prajwal et al., 2020) shares the same
setting as ours.

###  4.1. Quantitative Evaluation

The evaluation is constructed in a self-reconstruction manner. Given that it
is not fair to use the whole ground truth information directly as a reference
input, we uniformly select the first frame as the reference frame for all
methods to ensure that it cannot provide lip-sync information.

####  4.1.1. Evaluation Metrics.

We evaluate our AV-CAT on both generation quality and lip synchronization. For
the generation quality, we follow previous studies (Zhou et al., 2019; Chen et
al., 2019; Zhou et al., 2021) and adopt the commonly used SSIM (Wang et al.,
2004), PSNR as metrics. In terms of lip synchronization, we follow (Chen et
al., 2019; Zhou et al., 2021) to use the landmarks distances around the mouth
region (LMD) to account for the generated mouth shape. Then we employ the
confidence score of SyncNet (Chung and Zisserman, 2016b) to evaluate the
synchronization quality as performed in (Vougioukas et al., 2019; Zhou et al.,
2021). Note that we use the officially SyncNet as a metric instead of our
self-trained for the evaluation.

Figure 5. Ablation studies with visual results. Only the full model can
synthesize videos with both correct mouth movement and face shape. The figures
are selected from VoxCeleb ©Visual Geometry Group (CC BY) and LRW ©BBC.

####  4.1.2. Evaluation Results.

We perform quantitative comparisons on the test set of LRW (Chung and
Zisserman, 2016a) and VoxCeleb (Nagrani et al., 2017) datasets. The results
are summarized in Table 1. Compared to existing methods, our AV-CAT achieves
significant improvement in the generation quality. Meanwhile, our AV-CAT also
achieves comparable performance with the state-of-the-art lip-sync performance
on both datasets. Our LMD score is slightly worse than Wav2Lip, possibly
caused by the observation that we tend to generate more exaggerated lip
movements. This stands the same for the SyncNet scores. It is possible that
our self-trained SyncNet discriminator does not perform as well as the one
used in Wav2Lip.

Particularly, we argue there is generally a trade-off between perceptual
quality and synchronization and our model balances it better than traditional
CNN-based models. We also re-train a Wav2Lip model with the same setting and
dataset as ours (denoted as Wav2Lip-L) and find it achieves worse performance
than the original model. Both the limited receptive fields and the skip-
connections at higher resolutions restrict the network’s ability.

###  4.2. Qualitative Evaluation

We also conduct a qualitative comparison, including the results in Fig. 4 and
a subjective evaluation. For video samples, please refer to our supplementary
materials111Demo videos are available at https://hangz-nju-
cuhk.github.io/projects/AV-CAT. . The first line of Fig 4 shows the frames of
the original videos that correspond to input audio. As shown in Fig 4,
MakeitTalk fails to predict accurate mouth shapes due to the prediction error
from the 3D landmarks. Moreover, its generated head movements are relatively
subtle compared with original frames. On the other hand, PC-AVS synthesizes
basically correct mouth shapes, but their results are limited in the cropped
facial areas, making them non-realistic. Wav2Lip synthesizes satisfactory lip-
sync performance. However, the inpainted mouth region is always blurry, given
that they can only handle images of low qualities. Our AV-CAT generates high
fidelity results with much sharper mouth boundaries and clearer textures.

####  4.2.1. User Study.

We additionally perform a user study, where 15 participants are invited to
evaluate results generated by our AV-CAT and three comparing methods. 20
videos are sampled from VoxCeleb and 20 from LRW. We randomly select the
driving audios from the test set under the cross-video driving setting.
Following the commonly used Mean Opinion Scores (MOS) protocol, the
participants are required to rate (1-5) considering the following factors: (1)
Lip-sync quality; (2) the perceptual quality and (3) the realness of generated
videos. The results are reported in Table 2. It can be seen that our results
reach the best on all three perspectives, which proves the high quality of our
method.

###  4.3. Ablation Studies

We conduct ablation studies further to demonstrate the contributions of
different components in our AV-CAT. Specifically, we construct three variants
by removing the specified components: 1) First, to evaluate the effectiveness
of the transformer architecture, we remove the CTH backbone, and directly feed
the masked frames and references into the Refinement Network (denoted as “RN
only”). Since our Refinement Network is designed based on U-Net and audio
injection. We also try to replace the CTH backbone network to the larger
version of Wav2Lip in our model (Wav2Lip-L + RN). 2) Then in order to evaluate
the contribution of the Refinement Network, we remove the Refinement Network,
and evaluate the intermediate output of the transformer backbone (denoted as
“w/o RN”). 3) Finally, to evaluate the contribution that SyncNet makes to the
lip-synchronization quality, we train another model without using the
$\mathcal{L}_{sync}$ (denoted as “w/o Sync”).

The numerical results are shown in Table 3 and the visual results are shown in
Fig. 5, respectively. Compared to our Full Model, “RN only” achieves the worst
results on LMD and SyncNet scores due to the redundant mouth shape information
from the reference frame (the results are close to the target template as
shown in Fig. 5). “W/o RN” obtains degraded performance in all metrics, which
represents the inconsistent texture of the mouth region (see second row of
Fig. 5). The effectiveness of our CTH backbone network can further be shown as
the results combining Wav2Lip-L with RN perform worse than our full model.
“w/o Sync” fails to generate synchronized lip movements. Both the numerical
results and visual results demonstrate the effectiveness of our components in
AV-CAT.

Table 2. User study measured by Mean Opinion Scores. Larger is higher, with the maximum value to be 5. MOS on $\setminus$ Method | PC-AVS | Wav2Lip | MakeitTalk | AV-CAT
---|---|---|---|---
Lip-Synch Quality | 3.54 | 3.68 | 2.52 | 3.82
Generation Quality | 3.94 | 3.85 | 2.95 | 4.30
Video Realness | 3.56 | 3.48 | 3.37 | 4.18
Table 3. Ablation study with quantitative comparisons on VoxCeleb. The results are shown when we vary the accessibility of the Transformer network, refinement network, and the optimization objective. Method | $\text{SSIM}\uparrow$ |  PSNR $\uparrow$ |  LMD $\downarrow$ |  $\text{Sync}_{conf}$$\uparrow$
---|---|---|---|---
RN only | 0.812 | 30.61 | 10.68 | 3.9
w/o RN | 0.748 | 29.31 | 10.16 | 4.9
Wav2Lip-L + RN | 0.874 | 32.29 | 8.72 | 4.8
w/o Sync | 0.868 | 32.93 | 9.34 | 4.7
Full Model | 0.889 | 33.41 | 8.64 | 5.1

##  5\. Conclusion

In this work, we present the Audio-Visual Context-Aware Transformer (AV-CAT),
which produces high-quality lip-synced talking faces. We highlight several
important proprieties. 1) We formulate the problem in a masked lip-sync
prediction manner, and propose to learn cross-frame and cross-modal context
information with Transformers. 2) Our design shows that audios can modulate
UNet structures for quality improvement. 3) Our results are validated to be
more realistic than the previous state of the arts.

##### Limitations and Future Work.

1) The model is not sensitive to certain consonants. Possibly, the data are
not evenly distributed. The state of the reference image could also slightly
affect the results. 2) Moreover, our model cannot handle delicate details such
as mimicking the speaking style of a specific person or casting shadows from
the jaw. 3) More advanced audio representations (Baevski et al., 2020; Chen et
al., 2022b) can be involved in future studies.

##### Ethical Considerations.

Our method might be leveraged for malicious uses such as creating deepfakes.
We will restrict the license of our model to research use only and share it
with the deepfake detection community. We will also try actively adding
watermarks into the generation process for deepfake identification.

## References

  * (1)
  * Baevski et al. (2020) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in Neural Information Processing Systems_ (2020).
  * Bao et al. (2021) Hangbo Bao, Li Dong, and Furu Wei. 2021.  Beit: Bert pre-training of image transformers.  _arXiv preprint arXiv:2106.08254_ (2021).
  * Blanz and Vetter (1999) Volker Blanz and Thomas Vetter. 1999.  A morphable model for the synthesis of 3D faces. In _Proceedings of the 26th annual conference on Computer graphics and interactive techniques_. 187–194.
  * Chang et al. (2022) Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. 2022.  MaskGIT: Masked Generative Image Transformer.  _arXiv preprint arXiv:2202.04200_ (2022).
  * Chen et al. (2020) Lele Chen, Guofeng Cui, Celong Liu, Zhong Li, Ziyi Kou, Yi Xu, and Chenliang Xu. 2020.  Talking-head Generation with Rhythmic Head Motion. In _European Conference on Computer Vision_. Springer, 35–51.
  * Chen et al. (2018) Lele Chen, Zhiheng Li, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. 2018.  Lip movements generation at a glance. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 520–535.
  * Chen et al. (2019) Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. 2019\.  Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7832–7841.
  * Chen et al. (2022b) Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. 2022b.  Wavlm: Large-scale self-supervised pre-training for full stack speech processing.  _IEEE Journal of Selected Topics in Signal Processing_ (2022).
  * Chen et al. (2022a) Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. 2022a.  Context autoencoder for self-supervised representation learning.  _arXiv preprint arXiv:2202.03026_ (2022).
  * Chung et al. (2018) J. S. Chung, A. Nagrani, and A. Zisserman. 2018.  VoxCeleb2: Deep Speaker Recognition. In _INTERSPEECH_.
  * Chung and Zisserman (2016a) Joon Son Chung and Andrew Zisserman. 2016a.  Lip reading in the wild. In _Asian conference on computer vision_. Springer, 87–103.
  * Chung and Zisserman (2016b) Joon Son Chung and Andrew Zisserman. 2016b.  Out of time: automated lip sync in the wild. In _ACCV_.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2021\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _ICLR_ (2021).
  * Esser et al. (2021b) Patrick Esser, Robin Rombach, Andreas Blattmann, and Bjorn Ommer. 2021b.  Imagebart: Bidirectional context with multinomial diffusion for autoregressive image synthesis.  _Advances in Neural Information Processing Systems_ 34 (2021).
  * Esser et al. (2021a) Patrick Esser, Robin Rombach, and Bjorn Ommer. 2021a.  Taming transformers for high-resolution image synthesis. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 12873–12883.
  * Guo et al. (2021) Yudong Guo, Keyu Chen, Sen Liang, Yongjin Liu, Hujun Bao, and Juyong Zhang. 2021\.  AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis. In _IEEE/CVF International Conference on Computer Vision (ICCV)_.
  * He et al. (2021) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2021\.  Masked autoencoders are scalable vision learners.  _arXiv preprint arXiv:2111.06377_ (2021).
  * Isola et al. (2017) Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017.  Image-to-Image Translation with Conditional Adversarial Networks.  _CVPR_ (2017).
  * Jamaludin et al. (2019) Amir Jamaludin, Joon Son Chung, and Andrew Zisserman. 2019\.  You said that?: Synthesising talking faces from audio.  _International Journal of Computer Vision_ 127, 11 (2019), 1767–1779.
  * Ji et al. (2022) Xinya Ji, Hang Zhou, Kaisiyuan Wang, Qianyi Wu, Wayne Wu, Feng Xu, and Xun Cao. 2022.  EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model.  _SIGGRAPH_ (2022).
  * Ji et al. (2021) Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, and Feng Xu. 2021.  Audio-driven emotional video portraits. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 14080–14089.
  * Lahiri et al. (2021) Avisek Lahiri, Vivek Kwatra, Christian Frueh, John Lewis, and Chris Bregler. 2021.  LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalization. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2755–2764.
  * Li et al. (2021) Lincheng Li, Suzhen Wang, Zhimeng Zhang, Yu Ding, Yixing Zheng, Xin Yu, and Changjie Fan. 2021.  Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 1911–1920.
  * Li et al. (2022) Wenbo Li, Zhe Lin, Kun Zhou, Lu Qi, Yi Wang, and Jiaya Jia. 2022.  MAT: Mask-Aware Transformer for Large Hole Image Inpainting.  _CVPR_ (2022).
  * Liu et al. (2022) Xian Liu, Yinghao Xu, Qianyi Wu, Hang Zhou, Wayne Wu, and Bolei Zhou. 2022\.  Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation.  _ECCV_ (2022).
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021.  Swin transformer: Hierarchical vision transformer using shifted windows. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 10012–10022.
  * Nagrani et al. (2017) A. Nagrani, J. S. Chung, and A. Zisserman. 2017.  VoxCeleb: a large-scale speaker identification dataset. In _INTERSPEECH_.
  * Park et al. (2022) Se Jin Park, Minsu Kim, Joanna Hong, Jeongsoo Choi, and Yong Man Ro. 2022.  SyncTalkFace: Talking Face Generation with Precise Lip-syncing via Audio-Lip Memory. In _AAAI Conference on Artificial Intelligence_. Association for the Advancement of Artificial Intelligence.
  * Prajwal et al. (2020) KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. 2020.  A lip sync expert is all you need for speech to lip generation in the wild. In _Proceedings of the 28th ACM International Conference on Multimedia_. 484–492.
  * Raghu et al. (2021) Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. 2021\.  Do vision transformers see like convolutional neural networks?  _Advances in Neural Information Processing Systems_ 34 (2021).
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015\.  U-net: Convolutional networks for biomedical image segmentation. In _International Conference on Medical image computing and computer-assisted intervention_. Springer, 234–241.
  * Song et al. (2020) Linsen Song, Wayne Wu, Chen Qian, Ran He, and Chen Change Loy. 2020.  Everybody’s Talkin’: Let Me Talk as You Want.  _arXiv preprint arXiv:2001.05201_ (2020).
  * Song et al. (2018) Yang Song, Jingwen Zhu, Dawei Li, Xiaolong Wang, and Hairong Qi. 2018.  Talking face generation by conditional recurrent adversarial network.  _arXiv preprint arXiv:1804.04786_ (2018).
  * Sun et al. (2021) Yasheng Sun, Hang Zhou, Ziwei Liu, and Hideki Koike. 2021\.  Speech2Talking-Face: Inferring and Driving a Face with Synchronized Audio-Visual Representation.. In _IJCAI_.
  * Suwajanakorn et al. (2017) Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. 2017.  Synthesizing obama: learning lip sync from audio.  _ACM Transactions on Graphics (ToG)_ 36, 4 (2017), 1–13.
  * Thies et al. (2020) Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. 2020\.  Neural voice puppetry: Audio-driven facial reenactment. In _European Conference on Computer Vision_. Springer, 716–731.
  * Van Den Oord et al. (2017) Aaron Van Den Oord, Oriol Vinyals, et al. 2017\.  Neural discrete representation learning.  _Advances in neural information processing systems_ 30 (2017).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Vougioukas et al. (2019) Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. 2019\.  Realistic speech-driven facial animation with gans.  _International Journal of Computer Vision_ (2019).
  * Wang et al. (2021) Suzhen Wang, Lincheng Li, Yu Ding, Changjie Fan, and Xin Yu. 2021.  Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion.  _arXiv preprint arXiv:2107.09293_ (2021).
  * Wang et al. (2022) Suzhen Wang, Lincheng Li, Yu Ding, and Xin Yu. 2022\.  One-shot talking face generation from single-speaker audio-visual correlation learning. In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Wang et al. (2018b) Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. 2018b.  High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
  * Wang et al. (2018a) Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018a.  Non-local neural networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 7794–7803.
  * Wang et al. (2004) Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004\.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ 13, 4 (2004), 600–612.
  * Wu et al. (2021) Haozhe Wu, Jia Jia, Haoyu Wang, Yishun Dou, Chao Duan, and Qingshan Deng. 2021\.  Imitating Arbitrary Talking Style for Realistic Audio-Driven Talking Face Synthesis. In _Proceedings of the 29th ACM International Conference on Multimedia_. 1478–1486.
  * Xiao et al. (2021) Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. 2021\.  Early convolutions help transformers see better.  _Advances in Neural Information Processing Systems_ 34 (2021), 30392–30400.
  * Yang et al. (2020) Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. 2020.  Learning Texture Transformer Network for Image Super-Resolution. In _CVPR_.
  * Yang et al. (2021) Jingyu Yang, Sheng Shen, Huanjing Yue, and Kun Li. 2021\.  Implicit Transformer Network for Screen Content Image Continuous Super-Resolution.  _Advances in Neural Information Processing Systems_ 34 (2021).
  * Yi et al. (2020) Ran Yi, Zipeng Ye, Juyong Zhang, Hujun Bao, and Yong-Jin Liu. 2020.  Audio-driven talking face video generation with learning-based personalized head pose.  _arXiv e-prints_ (2020), arXiv–2002.
  * Yu et al. (2021) Yingchen Yu, Fangneng Zhan, Rongliang Wu, Jianxiong Pan, Kaiwen Cui, Shijian Lu, Feiying Ma, Xuansong Xie, and Chunyan Miao. 2021.  Diverse image inpainting with bidirectional and autoregressive transformers. In _Proceedings of the 29th ACM International Conference on Multimedia_. 69–78.
  * Zhang et al. (2021) Bowen Zhang, Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, and Baining Guo. 2021\.  StyleSwin: Transformer-based GAN for High-resolution Image Generation.  _arXiv preprint arXiv:2112.10762_ (2021).
  * Zheng et al. (2021) Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. 2021.  Tfill: Image completion via a transformer-based architecture.  _arXiv preprint arXiv:2104.00845_ (2021).
  * Zhou et al. (2019) Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. 2019.  Talking face generation by adversarially disentangled audio-visual representation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 9299–9306.
  * Zhou et al. (2021) Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. 2021\.  Pose-controllable talking face generation by implicitly modularized audio-visual representation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 4176–4186.
  * Zhou et al. (2020) Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li. 2020.  MakeltTalk: speaker-aware talking-head animation.  _ACM Transactions on Graphics (TOG)_ 39, 6 (2020), 1–15.