# Extreme-scale Talking-Face Video Upsampling with Audio-Visual Priors

Sindhu B Hegde<EMAIL_ADDRESS>International Institute of
Information TechnologyHyderabad, India ,  Rudrabha Mukhopadhyay
<EMAIL_ADDRESS>International Institute of Information
TechnologyHyderabad, India ,  Vinay P Namboodiri<EMAIL_ADDRESS>University
of BathUnited Kingdom and  C. V. Jawahar<EMAIL_ADDRESS>International
Institute of Information TechnologyHyderabad, India

(2022)

###### Abstract.

In this paper, we explore an interesting question of what can be obtained from
an $8\times 8$ pixel video sequence. Surprisingly, it turns out to be quite a
lot. We show that when we process this $8\times 8$ video with the right set of
audio and image priors, we can obtain a full-length, $256\times 256$ video. We
achieve this $32\times$ scaling of an extremely low-resolution input using our
novel audio-visual upsampling network. The audio prior helps to recover the
elemental facial details and precise lip shapes and a single high-resolution
target identity image prior provides us with rich appearance details. Our
approach is an end-to-end multi-stage framework. The first stage produces a
coarse intermediate output video that can be then used to animate single
target identity image and generate realistic, accurate and high-quality
outputs. Our approach is simple and performs exceedingly well (an $8\times$
improvement in FID score) compared to previous super-resolution methods. We
also extend our model to talking-face video compression, and show that we
obtain a $3.5\times$ improvement in terms of bits/pixel over the previous
state-of-the-art. The results from our network are thoroughly analyzed through
extensive ablation experiments (in the paper and supplementary material). We
also provide the demo video along with code and models on our
website111http://cvit.iiit.ac.in/research/projects/cvit-projects/talking-face-
video-upsampling.

video upsampling, video super-resolution, video compression, talking-face
videos, audio-visual learning

††journalyear: 2022††copyright: acmlicensed††conference: Proceedings of the
30th ACM International Conference on Multimedia; October 10–14, 2022; Lisboa,
Portugal††booktitle: Proceedings of the 30th ACM International Conference on
Multimedia (MM ’22), October 10–14, 2022, Lisboa, Portugal††price: 15.00††doi:
10.1145/3503161.3548080††isbn: 978-1-4503-9203-7/22/10††ccs: Computing
methodologies Reconstruction††ccs: Computing methodologies Neural networks
Figure 1. We solve the problem of upsampling extremely low-resolution (LR)
talking-face videos to generate high-resolution (HR) outputs. Our approach
exploits LR frames ($8\times 8$ pixels), corresponding audio signal and a
single HR target identity image to synthesize realistic, high-quality talking-
face videos ($256\times 256$ pixels). Please check our project page for video
results.

##  1\. Introduction

Over the years, we have always been fascinated with questions that can push
the limits of computer vision. For instance, can we recognize actions (Efros
et al., 2003) or objects (Torralba et al., 2007) in videos/images of about
$30\times 30$ pixels? Or how small a face can we detect in an image? It turns
out, the face can be as tiny as $3$ pixels tall (Hu and Ramanan, 2017)!
Recovering extremely feeble signals has also led to remarkable achievements,
such as the imaging of the black hole. In this vein, we explore whether we can
upsample talking-face videos with resolutions as low as $8\times 8$ pixels.
Clearly, this is an exceptionally challenging task. Surprisingly, we reveal
that one can obtain realistic high-resolution (HR) talking-faces ($256\times
256$ pixels) when provided with the right set of additional information. We
utilize a single target identity image and the accompanying audio to upsample
$8\times 8$ video to a full $256\times 256$ dimensional video (a fascinating
$32\times$ scale-factor) that far exceeds previous methods. In today’s
digitally connected world, where talking-face videos are among the most common
forms of video content, our multi-modal system can have numerous potential
applications. Some of them include: (i) video conferencing in low-bandwidth
situations, (ii) recovering low-quality archival footage of public talks and
speeches and (iii) enhancing videos captured from a distance with a high
camera zoom.

Challenges: Although our task has promising applications, synthesizing HR
videos from extremely low-resolution (LR) inputs (e.g., $8\times 8$ pixels) is
a very challenging task. Essential face attributes such as identity, age and
gender are almost entirely lost at such low resolutions and cannot be directly
recovered (see LR video in Figure 1). Apart from reconstructing these
elementary identity details, the network must also learn to predict the
original head poses and accurate lip shapes. Given the arduous nature of the
task, it is evident that the model will struggle to achieve the desired
quality if it relies solely on the LR input. Thus, in our work, we argue that
considering appropriate priors is quintessential to obtain high-quality
results. Specifically, we assess the importance of two kinds of priors: (i)
audio signal, which can help to recover elemental facial attributes and can
significantly improve lip shape generation; (ii) a single HR target identity
image which can aid in restoring fine-grained details such as skin texture,
color, hair, teeth and surrounding background. The target identity image can
be any sample frame, either from the same video or any other image with
similar characteristics in terms of the face identity, pose, clothing and
background. To understand our task better, we will now explore how our work is
connected to some of the existing problems in literature.

Super-Resolution (SR) Perspective: The SR literature till date has focused on
super-resolving inputs (either faces, generic images or videos) (Kim et al.,
2019; Haris et al., 2018, 2019; Chu et al., 2020) where sufficient information
is already available (e.g., $256\times 256$ pixels input). None of these
methods can handle extreme scale-factors like $32\times$. When the input
resolution is very low (like $8\times 8$), we observe that most of the current
SR works (Kim et al., 2019; Ma et al., 2020; Haris et al., 2018; Chen et al.,
2020; Chu et al., 2020; Sajjadi et al., 2018; Haris et al., 2019) generate
sub-optimal results. The essential visual attributes such as the identity,
face texture and lip shape do not accurately match the original face. This is
natural because the network is forced to speculate these details without
adequate priors. Thus, the existing methods: (i) do not aim to preserve the
specific identity details and (ii) do not explicitly deal with talking-face
videos where specialized temporal information like lip synchronisation must be
maintained throughout the video.

In our work, we propose to address these limitations by generating high-
quality talking-face videos with accurate lip-sync. It is important to note
that although we aim to generate HR talking-faces from LR inputs, the task we
are attempting is very different compared to the typical SR problem. The use
of a single HD target identity image and synthesizing from extremely LR
talking-faces ($8\times 8$ pixels) sets us considerably apart from the
traditional SR.

Compression Perspective: Our task enables applications such as low-bandwidth
video calling; thus making it closely related to the task of talking-face
video compression. However, unlike the existing works like “os-synth” (Wang et
al., 2021) where 3D face keypoints are transmitted, we take a unique path in
our work. We propose to transmit the LR frames to extract the face structure,
motion and pose information. Transmitting keypoints have several limitations:
(i) Keypoints can only be extracted if we have the HR video beforehand. While
the availability of actual video might be a possibility in video conferencing,
this poses severe constraints in various other applications where the actual
HD video is not present; (ii) Keypoints do not encode adequate head pose
information, thus requiring additional specialized head pose estimation
models; (iii) Keypoints do not cover other information like background,
lighting, accessories like glasses and beards that could be present in
talking-faces. Thus, in our work, we demonstrate the advantages of using LR
frames and achieve better compression ratio compared to the standard codecs,
while also not compromising on the desired quality.

Talking-Face Animation Perspective: Our task also shares similarities with
audio-driven talking-face generation (A2TF) (Jamaludin et al., 2019; K R et
al., 2019; Prajwal et al., 2020; Zhou et al., 2020; Wu et al., 2021) and face
re-enactment (FR) (Siarohin et al., 2019; Zhou et al., 2021) tasks. A2TF works
aim to generate videos of a target identity conditioned on the audio signal
(input: single target identity image + audio). FR methods ingest a single
target identity image and an HD video of a different identity with an aim to
animate the target image according to the motion of the driving video (input:
single target identity image + HD video as pose prior). Although we agree that
in terms of the problem space, our work resembles A2TF/FR, we want to point
out that our focus is very different - upsampling extremely low-resolution
videos. There are key differences as noted in Table 1. We leverage positive
aspects from these dimensions to solve an entirely new task - generating HD
videos from extremely LR inputs while preserving the exact same facial
features, e.g., pose and identity. Nevertheless, we also include a comparison
with A2TF and FR works in Section 4.3.

Table 1. Key differences between audio-driven talking-face generation (A2TF), face re-enactment (FR) and proposed task. | Pose prior | Lips are in-sync | Matches GT | Can be used
---|---|---|---|---
| used | with audio? | frames? | for SR?
A2TF | None | $\checkmark$ |  $\times$ (changes pose) | $\times$
FR | HD video | $\checkmark$ |  $\checkmark$ (same id-recons.) | $\times$
Ours | LR frames | $\checkmark$ | $\checkmark$ | $\checkmark$

Overview of this Work: In this work, we propose a talking-face video
upsampling framework, where the core idea is to utilize adequate priors to
generate high-quality ($256\times 256$) videos from extremely low-resolution
inputs. A gallery showing our synthesized frames is displayed in Figure 2. We
conduct extensive experiments and comparisons with state-of-the-art methods
for the tasks of super-resolution and compression. To the best of our
knowledge, we are the first to synthesize high-quality talking-faces at scale-
factors of $32\times$ from an input as small as $8\times 8$ pixels. We also
show how our network can be utilized for low-bandwidth video conferencing,
along with a demo video on our project page.

Figure 2. Input: LR frames (top); Output: HR frames (bottom). Our approach
effectively synthesizes a diverse set of HR talking-faces, handling different
poses, gender, age and race. Our method also works for “human-like” synthetic
faces even though it was trained only on real faces.

To summarize, the major contributions of our work are:

  * •

We present a novel audio-visual network that can upsample very low-resolution
talking-face videos at scale-factors previously unseen in video SR literature
($32\times$).

  * •

Our approach to make use of adequate priors: (i) audio signal and (ii) a
single target identity, achieves significant improvements over the existing
works for SR and compression tasks.

  * •

Our system has strong real-world use-cases, owing to its ability to preserve
true identity information. We specifically demonstrate the superior quality of
our results for low-bandwidth video conferencing and achieve a significant
reduction in bandwidth compared to H.264 standard.

##  2\. Related Work

Image & Video Super-Resolution: Single image super-resolution (SISR) has
progressed tremendously with the advent of deep learning and CNNs (Dong et
al., 2016). A flurry of works (Kim et al., 2015; Lim et al., 2017; Timofte et
al., 2018; Haris et al., 2018; Wu et al., 2021; Tang et al., 2021; Yang, 2019)
have followed this initial method, thereby improving the performance by many
folds. To account for specialized facial attributes which are not present in
generic images, face super-resolution (FSR) methods (Chen et al., 2020; Bulat
et al., 2018; Kim et al., 2021) with specific loss functions like facial
landmarks (Ma et al., 2020) and facial heatmaps (Kim et al., 2019) were
proposed. However, these FSR approaches focus on scale-factors up to
$8\times$, which is in stark contrast to our attempt of $32\times$ SR.

Recently, PULSE (Menon et al., 2020) proposed a StyleGAN-based approach to
super-resolve static faces at high scale-factors of $64\times$. However, the
model generates imaginary faces of people who do not exist, posing severe
constraints on applications where the person’s identity needs to be
matched/recovered correctly. Conversely, in our work, we aim to generate HR
faces of the specific (real) identity.

To capture the temporal aspect which is not present in the static faces, video
SR approaches came into picture. From early recurrent architectures (Héas et
al., 2015) to more recent advancements (Sajjadi et al., 2018; Haris et al.,
2019; Chu et al., 2020), impressive results have been achieved for scale-
factors up to $4\times$. However, these generic video SR methods produce
blurred results with many artifacts when the input resolution is very low
($8\times 8$) and the scale-factors are very high ($>8\times$). Moreover,
“talking-face videos” have their own set of additional challenges, which are
neither tackled in video SR nor in static FSR works. Thus, in our work, we
overcome these limitations and design a novel network that specifically deals
with talking-face videos.

Talking-Face Video Generation: Audio-driven talking-face generation (A2TF) is
an active research area. Various methods have been proposed (Fried et al.,
2019; Yao et al., 2020; Prajwal et al., 2020; de Barros Reis et al., 2020;
Zhou et al., 2020) to accurately morph the lip movements of input target
identity to be in-sync with the corresponding speech. Face re-enactment (FR)
works (Thies et al., 2018; Kim et al., 2018a; Siarohin et al., 2019) have also
shown impressive performance in transferring head motions and expressions
based on guiding videos. While our task shares some similarities with A2TF/FR
works, we differ in the fact that we are given very sparse information in the
form of very low-resolution input. This is not the case in talking-face
generation models where HR frames (or extracted landmarks) are used for
conditional generation.

Few works (Kim et al., 2018a; Zhang et al., 2021) use a 3D model as an
intermediate step to recover high-quality videos. The major benefit of
adapting a 3D model is the superior quality of the output generations. But, it
comes with an additional overhead - the computational complexity, making such
heavy models very impractical for mobile hardware deployments. In contrast,
our method is (i) simple, (ii) generalised, since it can be applied for any
in-the-wild identity (unlike some of the 3D models that require speaker-
specific training) and (iii) does not require specialized large-scale 3D
datasets to train the models.

Data Compression using Deep Learning: Deep learning has lead to profound
improvements in image and video compression techniques. Starting from initial
auto-encoder based methods (Dumas et al., 2018; Xu et al., 2014) to flow-based
approaches (Agustsson et al., 2020; Ho et al., 2019), there have been multiple
efforts to obtain a compact image/video representation. To enable video calls
at a reduced bandwidth, specific methods like (Narayanan and Kiong, 2013;
Oquab et al., 2021) have been designed. In “SRVC” (Khani et al., 2021) authors
used video SR as a tool for compression, but at a scale-factor of merely
$2\times$. Recently, “os-synth” (Wang et al., 2021) demonstrated impressive
results by transmitting a sequence of learned 3D facial keypoints. However, as
discussed previously, transmitting keypoints has its own set of limitations.
Thus, unlike the existing compression methods, we take a path of extreme-scale
($32\times$) SR for the first time and transmit very low-resolution ($8\times
8$) videos. We show that our proposed approach of utilizing the audio and the
visual modalities can enable video-conferencing in bandwidth-limited regions,
while also achieving a better compression ratio over the existing works.

##  3\. Learning to Upsample LR Videos

We start the discussion by highlighting the critical elements of our approach.
We then present our framework with a detailed description of the modules
involved.

###  3.1. Critical Elements of our Approach

Audio Prior: As discussed previously, when the input resolution is a meager
$8\times 8$ pixel video, the ambiguity and the loss of information is so
paramount that the person’s original identity characteristics are barely
discernible. In such situations, we show that audio can aid in the recovery of
dominant facial traits of the person because the audio and the face share
multiple common features (Oh et al., 2019; Nagrani et al., 2018; Kim et al.,
2018b) like gender, age and ethnicity. We exploit the audio signal not only to
disambiguate the LR input, but also to greatly improve the lip shape
generation. Although precise lip shape is not a crucial necessity for static
face SR, it is a very important aspect of video SR where the generated lip
movements should sync with the given speech. In our work, we explore the
natural correlation between speech and lips (Prajwal et al., 2020; Afouras et
al., 2018; Hegde et al., 2021; Chung et al., 2019) to generate accurate lip
movements.

Visual Prior: To be able to generate faces that replicate the actual identity,
it is important to preserve sharp details like face texture, lip colour, hair,
teeth and skin tone. Most of the current works hallucinate these details,
leading to significant variations in fine-grained information. Such models are
thus unusable for real-world use-cases, where videos of a specific identity
needs to be generated. We argue that considering the adequate prior
information is of utmost importance to: (i) generate a video of a specific
identity and (ii) reconstruct the high-quality facial details. To achieve
this, we provide our network with a single HR image of the target identity,
which helps to transfer the identity-specific sharp details to the synthesized
video. In most applications, a single HR identity image is easily accessible.
For example, during video conferencing, the first frame can be transmitted in
the original resolution.

###  3.2. Our Approach

Figure 3. We propose a novel audio-visual talking-face upsampling framework.
Our approach recovers the basic facial attributes like the identity structure,
pose and motion using a backbone network. We then consider a single HR target
identity image to capture the fine-grained details. Our end-to-end trainable
animation network ingests the intermediate outputs (from backbone network) and
the target identity and reconstructs realistic, high-quality videos that
accurately match the actual identity.

An overview of our proposed framework is depicted in Figure 3. The goal of our
work is to generate a sequence of HR frames,
$F_{hr}=(hr_{1},hr_{2},...,hr_{n})$ from the LR input,
$F_{lr}=(lr_{1},lr_{2},...,lr_{n})$ that accurately match the ground-truth
frames $F_{gt}=(gt_{1},gt_{2},...,gt_{n})$. We consider the corresponding
audio signal, $A$ and a single HR target identity, $F_{id}$ as prior
information. We detail the different components of our model and the network
architecture below.

Backbone Network: We pre-train a backbone network to generate a “driving
video” for our face animation network. It extracts the relevant features of
the identity, face structure, pose and motion information. We observe that
pre-training the network before training the face animation network helps to
improve the model’s performance because the basic face details and motion
information captured is essential to animate the HR face (details about the
animation network is explained below). We empirically demonstrate the
importance of this in the ablation study (in supplementary).

Visual Encoder: We extract the visual features from the LR frames $F_{lr}$
using a visual encoder, which comprises a series of $3$D convolution layers
with residual connections. Our visual encoder resembles multiple previous
models (Afouras et al., 2020; Hegde et al., 2021; Chung et al., 2019) designed
specifically for processing talking-face videos. The input to the visual
encoder is a contiguous window of $T$ LR frames $l:(N,T,3,8,8)$ where $N$
refers to the batch size and $T$ refers to the window frames (here $T$=5). The
encoder processes these input frames and generates the visual embeddings,
$f_{v}:(N,T,512,8,8)$.

Audio Encoder: We consider the corresponding audio segment $A$ and extract the
melspectrogram representation using a window length of $25$ms with a hop
length of $10$ms sampled at $16$kHz. The melspectrograms $(T^{\prime},80)$ are
given to the audio encoder, which is a stack of residual $1$D convolutions
with appropriate strides to match the visual time-steps $T$. The generated
features ($N,T,512$) are then upsampled using transpose convolution layers to
obtain the audio embeddings, $f_{a}:(N,T,512,8,8)$.

Identity Structure $\&$ Motion Predictor: We concatenate the learned visual
and audio embeddings in the latent space (along the channel dimension) to
obtain $f_{cat}=(N,T,1024,8,8)$. Inspired from Deep Back-Projection Network
(DBPN) (Haris et al., 2018), we use iterative upsampling and downsampling
layers in our module, where the primary idea is to effectively capture the
mutual relationship between the LR and HR frames. We consider the fused
representation $f_{cat}$ and stack the time steps along the batch dimension to
obtain $f_{cat}:(N*T,1024,8,8)$. As the visual and the audio encoders have
already captured the temporal information, the stacking strategy improves the
convergence speed and also gives us the desired performance. The output of
this block is a sequence of frames ($F_{int}$) of resolution $256\times 256$
pixels, which encapsulates the elemental facial details, face structure, pose
and motion information. The network is trained to minimize the $L_{1}$
reconstruction loss:

(1) $L_{\mathrm{rec}}=\frac{1}{N}\sum^{N}_{i=1}||F_{int}-F_{gt}||_{1}$

Face Animation Network: To synthesize the HR videos of the target identity by
preserving all the details, we consider a single HR image of the target
identity as our input. This is in-line with the audio-driven talking-face
generation works (Prajwal et al., 2020; Zhou et al., 2021; Siarohin et al.,
2019; Wang et al., 2021) where a single target identity is considered to
replicate the identity-specific details. The target identity image helps to
capture the fine-grained features like face texture, skin tone, hair and lip
color, which are otherwise not recovered in the existing FSR works (Ma et al.,
2020; Haris et al., 2018; Chen et al., 2020). These details are crucial,
especially when our input is a mere $8\times 8$ pixel video and make our model
applicable in cases where we must match the actual identity to the maximum
extent. We adopt one of the popular methods, FOMM (Siarohin et al., 2019) to
animate the target identity based on the driving video obtained from our
backbone network.

Overview of First-Order Motion Model (FOMM): FOMM ingests a target identity
image (to extract the appearance) and a driving video (to extract the pose and
motion). The learned latent representation of motion in the driving video is
combined with the target identity to synthesize the output video. During
training, the model observes the target-driving image pairs and predicts a
dense motion field which is later encoded using a keypoint detection network.
The target image is then rendered according to the learned trajectories in the
driving video. We refer the reader to (Siarohin et al., 2019) for more details
about FOMM. Note that FOMM is designed to work even when the target image and
the driving video are of different identities. However, this feature is not
necessary in our work where the aim is to preserve the target identity, so we
make appropriate modifications, as described below.

Adapting FOMM to our Task: For our task at hand, the goal is to animate the HR
target identity image $F_{id}$ in accordance with the motion of the driving
video $F_{int}$. As described previously, our backbone network reconstructs
the basic identity attributes like face structure, age and gender. We thus
generate a residual mask as the output of the animation network and add the
intermediate outputs $F_{int}$ (see Figure 3) to obtain a realistic HR
talking-face video $F_{hr}$ of the target identity as the final output. We
fine-tune the entire network (including the backbone network) end-to-end, by
optimizing the FOMM loss $L_{fomm}$ and our task-specific losses, $L_{rec}$,
$L_{region}$ and $L_{sync}$. The FOMM loss $L_{fomm}$ consists of: (i) a
VGG-19 based perceptual loss at multiple resolutions and (ii) an equivariance
constraint to enforce the model to predict consistent keypoints to known
geometric transformations. We describe other task-specific losses that we use
in out network below.

Enforcing Local Correspondence: In our experiments, we observed that the model
at times generates frames where the facial regions like eyes, eyebrows and
lips are slightly off-position. This occurs if the target identity ingested by
our animation network is significantly different from the driving video.
Hence, to further improve the generation quality, we add a face landmark-based
region loss, which penalizes the network for generating incorrect regions. We
compute the face landmarks (Bulat and Tzimiropoulos, 2017) for both GT
$F_{gt}$ and generated frames $F_{hr}$ and extract the following $R$ face
regions: lips, nose, eyes and eyebrows. We then add a patch-based local loss
by minimizing the $L_{2}$ distance for all these $R$ regions (here $R$=4) to
ensure that the predicted regions are as close as possible to the actual
regions.

(2)
$L_{\mathrm{region}}=\frac{1}{N}\sum^{N}_{i=1}\sum^{R}_{r=1}||F_{hr}-F_{gt}||_{2}$

Enforcing Accurate Lip-sync: Prior works (Prajwal et al., 2020) involving
speech and lip movements have shown that using a lip-sync discriminator can
greatly benefit in enforcing strong audio-visual correspondences. This can
also be observed in our task, as learning to synthesize HR frames from very
low-resolution input might lead to generating lips that are out-of-sync with
the audio segment. Thus, we pre-train a lip-sync discriminator, “SyncNet”
adapted from (Chung et al., 2019), trained to maximize the cosine similarity
between the lip-speech pair $(F_{gt},A)$ when they are in-sync (and minimise
the similarity if they are out-of-sync). Once trained, we use this network as
a frozen discriminator to penalize the generated frames $F_{hr}$ if they do
not match the corresponding audio segment. In our end-to-end network, we
minimize the sync loss:

(3) $L_{\mathrm{sync}}=-\frac{1}{N}\sum^{N}_{i=1}\log(\dfrac{f_{hr}\cdot
a}{max(\lVert f_{hr}\rVert_{2}\cdot\lVert a\rVert_{2},\epsilon)})$

###  3.3. Training Settings and Datasets

The final loss function is the combination of the above losses:

(4)

$L_{\mathrm{HR}}=\lambda_{rec}L_{\mathrm{rec}}+L_{fomm}+\lambda_{region}L_{\mathrm{region}}+\lambda_{sync}L_{\mathrm{sync}}$

In our experiments, we set $\lambda_{rec}=50$, $\lambda_{region}=100$ and
$\lambda_{sync}=0.05$. We provide the details regarding pre-processing and
training settings in supplementary file on our project page.

Datasets: We train our model using AVSpeech (Ephrat et al., 2018) and
VoxCeleb2 (Chung et al., 2018) datasets; both containing talking-face videos
spanning a wide variety of identities, languages and poses. For AVSpeech data,
we extract the face tracks using an off-the-shelf face detector (Zhang et al.,
2017). We curate a set of $50$ hours for training and $\sim 3$ hours from the
official test split for testing and verified it for accurate lip-sync using
SyncNet (Chung and Zisserman, 2016). We also benchmark our model on VoxCeleb2
data which comprises face tracks with a fair amount of background. Owing to
computational limitations, we randomly sample a subset of $100$ hours for
training and use the full official test split for testing. Note that there are
no overlaps between the identities used in training and testing sets in both
datasets. The code, models and file-lists are released on our website for
reproducibility and future research.

##  4\. Experiments

###  4.1. Extreme-scale Super-Resolution

Table 2. Quantitative comparison for $32\times$ SR on AVSpeech (Ephrat et al., 2018) and VoxCeleb2 (Chung et al., 2018) datasets. Our method outperforms the baselines by a significant margin across all metrics. Note that the baselines have also been trained with a single identity image. Dataset |  AVSpeech (Ephrat et al., 2018) |  VoxCeleb2 (Chung et al., 2018)
---|---|---
Method | PSNR$\uparrow$ | SSIM$\uparrow$ | FID$\downarrow$ | LMD$\downarrow$ | LSE-D$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | FID$\downarrow$ | LMD$\downarrow$ | LSE-D$\downarrow$
Bicubic | 22.33 | 0.60 | 102.41 | 0.246 | 14.18 | 22.16 | 0.60 | 105.14 | 0.255 | 17.83
SPARNet (Chen et al., 2020) | 23.17 | 0.68 | 92.14 | 0.201 | 12.87 | 22.98 | 0.67 | 83.01 | 0.228 | 14.07
TecoGAN (Chu et al., 2020) | 19.26 | 0.62 | 84.73 | 0.213 | 13.01 | 16.91 | 0.54 | 82.19 | 0.234 | 14.12
Ours | 25.06 | 0.73 | 11.54 | 0.162 | 12.43 | 24.95 | 0.71 | 14.10 | 0.196 | 13.91
Figure 4. Qualitative comparisons on AVSpeech dataset (Ephrat et al., 2018).
Our method captures the rich identity-specific attributes like eyeballs, hair
strands, face texture and lip shape, far better compared to the existing
approaches. Figure 5. Qualitative comparisons on the models trained on
VoxCeleb2 dataset (Chung et al., 2018). Our method surpasses the existing
baselines in generating the outputs that accurately match the ground-truth
identity.

Baselines: The state-of-the-art works in video SR literature super-resolve up
to a scale-factor of $4\times$. We thus re-train the existing state-of-the-art
video SR method, TecoGAN (Chu et al., 2020) at a scale-factor of $32\times$ on
the same training dataset as ours. We extend the existing face SR approach,
SPARNet (Chen et al., 2020) to work for video SR by appropriately modifying
the architecture (ingest a window of $5$ frames) and train using the same
settings as mentioned above. These methods originally do not consider a HR
target identity as input; thus, it would be unfair to compare them without the
identity information. Hence we provide a HR target identity image to these
models in a manner typically used in talking-face generation methods:
concatenating it channel-wise with the input.

Metrics: We evaluate our SR model on: (i) PSNR, (ii) SSIM, (iii) Fréchet
Inception Distance (FID) (Heusel et al., 2018), (iv) Landmark Distance (LMD)
(Bulat and Tzimiropoulos, 2017) and (v) Lip-Sync Error Distance (LSE-D)
(Prajwal et al., 2020). More details about the metrics can be found in the
supplementary file.

Results: We compare our results with existing SR approaches at extreme scale-
factor of $32\times$ in Table 2. As we can see from the table, our method
outperforms the existing works by a significant margin on both AVSpeech
(Ephrat et al., 2018) and VoxCeleb2 (Chung et al., 2018) datasets. None of the
current techniques match the ground-truth identity (measured using PSNR) and
perceptual quality (measured using FID) of our generations. The LSE-D metric
indicates that our method achieves accurate lip-synchronization with audio,
thus validating our claim that the audio signal enables us to generate far
more accurate lip shapes than the competing methods. Our method also surpasses
the existing approaches in preserving the overall face structure (measured
using SSIM and LMD).

Figures 4 and 5 show the qualitative comparisons. We can clearly observe that
our models generate results with far fewer artifacts and captures rich, fine-
grained details. Although all the comparison methods consider the HR target
identity as input, they do not match the quality of our generations. This
shows that our overall network design is highly effective in making use of the
available target identity image. In the examples, we can also see the diverse
range of our models’ generative capabilities: eyeballs with precise eye color
(Fig. 4: row $1$), microphone (Fig. 4: row $2$), hair strands (Fig. 4: row
$2$), lip shape (Fig. 4: rows $2$,$3$ and Fig. 5: rows $1$,$2$), face texture
such as wrinkles (Fig. 4: row $3$), beard (Fig. 5: rows $1$,$2$). More visual
examples can be found on our project page.

Ablation study: We validate the design choices of our network by analyzing the
importance of audio signal, landmark-based region loss, the use of different
target identity images and several other additional experiments, along with
human evaluations in our supplementary. We also compare the performance of
different models at multiple scale-factors like $4\times$, $8\times$,
$16\times$ and $32\times$ in supplementary.

###  4.2. Talking-Face Video Compression

One of the major applications of our system is in compressing talking-face
videos to reduce the bandwidth in video conferencing applications. We can
transmit the LR frames ($8\times 8$ pixels) along with the audio signal on the
sender’s side and the receiver can reconstruct the high-quality video
($256\times 256$ pixels) using a single HR target identity image. A sample
video calling demo is illustrated in Figure 6. We assume that a single target
identity image can be sent at the beginning (e.g., $1^{st}$ frame) and hence
does not consume additional bandwidth. Note that this is very different from
the standard codecs, where full resolution I-frames are transmitted at regular
intervals. Also, since the audio signal is always accompanied in a video call,
we do not consider this as an extra overhead.

Table 3. Quantitative comparison for talking-face video compression on VoxCeleb2 dataset (Chung et al., 2018). We achieve the best trade-off in terms of quality versus compression ratio. Our method achieves the lowest FID (indicates very high perceptual quality) and a very low/comparable BPP. Method | BPP$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | FID$\downarrow$
---|---|---|---|---
H.264 (CRF=23) (min. compression) | 0.109 | 32.96 | 0.79 | 9.75
H.264 (CRF=36) | 0.027 | 19.24 | 0.67 | 30.12
H.266 | 0.0076 | 23.27 | 0.70 | 58.32
fs-vid2vid (Wang et al., 2019) | n/a | 20.36 | 0.71 | 85.76
os-synth (Wang et al., 2021) | 0.016 | 24.37 | 0.80 | 69.13
Ours | 0.023 | 24.95 | 0.71 | 14.10
Ours (Frame-Interpolation) | 0.0046 | 23.72 | 0.68 | 14.51

Baselines: We benchmark our model’s capability using the existing talking-face
video compression methods: few-shot vid2vid (fs-vid2vid) (Wang et al., 2019),
one-shot free-view synthesis (os-synth) (Wang et al., 2021) and the standard
codecs: (i) H.264 (with CRF of $23$ and $36$) and (ii) H.266 (implemented
using vvenc: https://github.com/fraunhoferhhi/vvenc). Since we train and
evaluate on the same dataset (VoxCeleb2 (Chung et al., 2018)), we directly
take the scores reported in os-synth (Wang et al., 2021) for comparison.

Metrics: We compare the compression factor using the standard bits-per-pixels
(BPP) metric and measure the reconstruction quality using PSNR, SSIM and FID
metrics.

Figure 6. Illustration of low-bandwidth video calling enabled by our system.
Note that HR frames shown at both the sender’s end are taken from an actual
video call recording (credits: https://www.youtube.com/watch?v=lQJD8RAq3lY).
Figure 7. Failure cases: (a) Drastic color changes, (b) Extreme variations of
facial expressions and (c) Rapid head motions.

Results: Table 3 shows the comparison of our approach with the competing
methods. We calculate the average BPP across all test videos for H.264 and
H.266 codecs. The number of bits required by the current state-of-the-art “os-
synth” (Wang et al., 2021) to represent a $256\times 256$ image is $1056$ (20
keypoints: $(20\times 6+12)\times 8$). Our approach requires $1536$ bits
($8\times 8\times 3\times 8$), with a BPP of $0.023$. The BPP obtained using
our method beats H.264 and is comparable to os-synth (Wang et al., 2021). In
terms of perceptual quality, we can observe that there is a trade-off in
compression factor v/s quality for H.264, (i) CRF of $23$: Higher quality,
with very less compression and (ii) CRF of $36$: Better compression, but poor
reconstruction. While H.266 achieves comparable PSNR and SSIM measures with a
BPP of $0.0076$, it still lags behind perceptually as shown by the FID metric.
In contrast, our method is able to generate higher quality videos (remarkably
low FID), with a better/comparable compression factor.

Computation Comparison: Our network has $\sim 20\%$ fewer parameters ($143$M)
while being $2\times$ faster ($50$FPS) than state-of-the-art compression
method (os-synth) on an NVIDIA 2080Ti GPU. Previous talking-face generation
works focus on obtaining plausible faces, whereas our aim is to develop the
first architecture to reconstruct the specific-identity by preserving most of
the actual identity details; future works can build upon this to achieve
optimal design. Further, model optimization techniques can be applied to the
current design to make it suitable for mobile hardware deployment.

Frame-Interpolation Network: To further reduce the bandwidth consumption, we
develop a baseline frame-interpolation network with an aim to upsample $5$FPS
videos to $25$FPS. While such networks have been studied in literature (Jiang
et al., 2018), most of them restrict the amount of upsampling, mainly due to
the computational resources involved in training the bulky networks (often
consisting of $3$D CNNs). In contrast, we take the advantage of LR videos and
design a model to increase the temporal resolution of frames ($5$FPS to
$25$FPS) for the first time. Our encoder-decoder based model processes $5$FPS
LR frames ($5,3,8,8$) and upsamples it $5\times$ to generate $25$FPS LR frames
as output ($25,3,8,8$). We refer the reader to our supplementary for more
details. The lower resolution of both the input and output allows us to train
a very light $3$D CNN model with only $0.2M$ parameters. This strategy
effectively permits us to transmit $5$FPS LR videos. On receiving these
frames, frame-interpolation network initially upsamples them to $25$FPS LR
videos, which can subsequently be ingested by our spatial talking-face video
upsampling network to render the final HR video. As shown in Table 3, our
frame-interpolation network achieves a further reduction in BPP, since only
$1$ in $5$ LR frames needs to be transferred. We thus obtain $\sim 6\times$
and $\sim 25\times$ reduction in bandwidth compared to os-synth (Wang et al.,
2021) and H.264 codec (with CRF $23$) respectively, without a significant
compromise in the generated quality.

###  4.3. Audio-driven Talking-Face Generation (A2TF) and Face Re-enactment
(FR)

Baselines: For A2TF, we compare with Wav2Lip (Prajwal et al., 2020) and
MakeItTalk (Zhou et al., 2020). Talking-face videos are generated using the
audio segment and the first frame of the original HD video as inputs (same
strategy as our model). For FR, the original models take the actual HD frames
as the driving video input, however, we do not have access to the actual HD
video in our model. Thus, we upsample the LR input using the existing video SR
method, TecoGAN (Chu et al., 2020) (trained on VoxCeleb2 dataset) and use it
as our input driving video, along with the first frame of the original HD
video as target identity input.

Metrics: Along with the standard metrics, we specifically evaluate the ability
of models to match the original identity using head pose estimation metrics
(Dhingra, 2022; Hempel et al., 2022). We convert the rotation
matrix222https://learnopencv.com/head-pose-estimation-using-opencv-and-dlib/
to Euler angles and report Mean Absolute Error (MAE) of these angles.

Results: Table 4 shows the performance of different A2TF and FR methods in
comparison to our approach. A2TF models generate accurate lip-sync (LSE-D),
but they fail to match the pose of the original identity (as noted by Yaw,
Pitch, Roll and MAE metrics). FR methods also face similar issues and do not
match the identity, as they do not have the actual HD video as driving frames.
Our model is clearly the most applicable for this task, is consistent across
all metrics and produces noticeably better results. Most importantly, our
model preserves the identity to the maximum extent.

Table 4. Quantitative comparison with A2TF (Prajwal et al., 2020; Zhou et al., 2020) and FR (Siarohin et al., 2019; Zhou et al., 2021) methods on VoxCeleb2 test set. Method | PSNR$\uparrow$ | SSIM$\uparrow$ | FID$\downarrow$ | LMD$\downarrow$ | LSE-D$\downarrow$ | Yaw$\downarrow$ | Pitch$\downarrow$ | Roll$\downarrow$ | MAE$\downarrow$
---|---|---|---|---|---|---|---|---|---
Wav2Lip (Prajwal et al., 2020) | 14.18 | 0.32 | 8.15 | 4.320 | 9.19 | 24.68 | 38.31 | 28.94 | 30.64
MakeItTalk (Zhou et al., 2020) | 18.88 | 0.49 | 31.19 | 2.012 | 11.91 | 26.29 | 40.13 | 31.42 | 32.61
FOMM (Siarohin et al., 2019) | 20.14 | 0.56 | 21.18 | 0.864 | 14.03 | 19.14 | 30.57 | 22.76 | 24.35
PC-AVS (Zhou et al., 2021) | 15.68 | 0.37 | 33.38 | 1.063 | 8.42 | 22.27 | 31.89 | 25.80 | 26.65
Ours | 24.95 | 0.71 | 14.10 | 0.196 | 13.91 | 13.55 | 21.01 | 15.48 | 16.68

##  5\. Limitations and Future Directions

Although our method generates realistic results for a wide variety of inputs,
there are certain situations as shown in Figure 7 where our model results in
sub-optimal generations. For example, if the color contrast changes
drastically as the video progresses, the model fails to capture these details
(Figure 7 (a)). Significant variations in facial expressions is another case
where our model struggles to replicate the details (Figure 7 (b)). Explicitly
handling the expressions is an interesting direction that can be investigated
in the future, which we currently do not handle in our work. Finally, in the
case of sudden/rapid changes in view, camera angle, or head movements, our
model attempts to generate smooth transitions (Figure 7 (c)). However, we
found our method to be stable over a large variety of inputs and anticipate
that our idea of utilizing extremely LR frames will be a basis for other
domains and applications.

##  6\. Conclusion

In this work, we present a novel framework for extreme-scale talking-face
video super-resolution and compression. We show that by considering
appropriate priors (audio signal and a single target identity image), we are
able to generate realistic, high-quality talking-face videos from (very) low-
resolution frames. Our method handles various inputs, including but not
limited to people of different ages, gender and ethnicity. Most importantly,
our framework is the first of its kind to produce photo-realistic lip-synced
talking-face videos while also matching the actual identity. By dramatically
reducing the bandwidth requirements, our approach can be utilized as a tool
for a seamless video-conferencing experience. We believe our core idea of
exploiting very low-resolution videos along with adequate priors will be an
important step towards the future of super-resolution and low-bandwidth video-
conferencing.

## References

  * (1)
  * Afouras et al. (2018) T. Afouras, J. S. Chung, and A. Zisserman. 2018.  The Conversation: Deep Audio-Visual Speech Enhancement. In _INTERSPEECH_.
  * Afouras et al. (2020) Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. 2020.  Self-Supervised Learning of Audio-Visual Objects from Video. In _European Conference on Computer Vision_.
  * Agustsson et al. (2020) Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George Toderici. 2020\.  Scale-Space Flow for End-to-End Optimized Video Compression. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Bulat and Tzimiropoulos (2017) Adrian Bulat and Georgios Tzimiropoulos. 2017.  How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks). In _International Conference on Computer Vision_.
  * Bulat et al. (2018) Adrian Bulat, Jing Yang, and Georgios Tzimiropoulos. 2018\.  To learn image super-resolution, use a GAN to learn how to do image degradation first. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 185–200.
  * Chen et al. (2020) Chaofeng Chen, Dihong Gong, Hao Wang, Zhifeng Li, and Kwan-Yee K. Wong. 2020.  Learning Spatial Attention for Face Super-Resolution.  _IEEE Transactions on Image Processing (TIP)_.
  * Chu et al. (2020) Mengyu Chu, You Xie, Jonas Mayer, Laura Leal-Taixé, and Nils Thuerey. 2020.  Learning Temporal Coherence via Self-Supervision for GAN-Based Video Generation.  _ACM Trans. Graph._ 39 (July 2020), 13 pages.  https://doi.org/10.1145/3386569.3392457
  * Chung et al. (2018) J. S. Chung, A. Nagrani, and A. Zisserman. 2018.  VoxCeleb2: Deep Speaker Recognition. In _INTERSPEECH_.
  * Chung and Zisserman (2016) J. S. Chung and A. Zisserman. 2016.  Out of time: automated lip sync in the wild. In _Workshop on Multi-view Lip-reading, ACCV_.
  * Chung et al. (2019) S. Chung, J. S. Chung, and H. Kang. 2019\.  Perfect Match: Improved Cross-modal Embeddings for Audio-visual Synchronisation. In _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. 3965–3969.  https://doi.org/10.1109/ICASSP.2019.8682524
  * de Barros Reis et al. (2020) Filipe Antonio de Barros Reis, Paula Dornhofer Paro Costa, and José Mario de Martino. 2020\.  Deeply Emotional Talking Head: A Generative Adversarial Network Approach to Expressive Speech Synthesis with Emotion Control. In _ACM SIGGRAPH 2020 Posters_ (Virtual Event, USA) _(SIGGRAPH ’20)_. Association for Computing Machinery, New York, NY, USA, Article 33, 2 pages.  https://doi.org/10.1145/3388770.3407417
  * Dhingra (2022) Naina Dhingra. 2022\.  LwPosr: Lightweight Efficient Fine Grained Head Pose Estimation. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 1495–1505.
  * Dong et al. (2016) Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2016\.  Image Super-Resolution Using Deep Convolutional Networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 38, 2 (2016), 295–307.  https://doi.org/10.1109/TPAMI.2015.2439281
  * Dumas et al. (2018) Thierry Dumas, Aline Roumy, and Christine Guillemot. 2018\.  Autoencoder Based Image Compression: Can the Learning be Quantization Independent?  _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ (2018), 1188–1192.
  * Efros et al. (2003) Efros, Berg, Mori, and Malik. 2003\.  Recognizing action at a distance. In _Proceedings Ninth IEEE International Conference on Computer Vision_. 726–733 vol.2.  https://doi.org/10.1109/ICCV.2003.1238420
  * Ephrat et al. (2018) Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T. Freeman, and Michael Rubinstein. 2018.  Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation.  _ACM Trans. Graph._ 37, 4, Article 112 (2018), 11 pages.  https://doi.org/10.1145/3197517.3201357
  * Fried et al. (2019) Ohad Fried, Ayush Tewari, Michael Zollhöfer, Adam Finkelstein, Eli Shechtman, Dan B Goldman, Kyle Genova, Zeyu Jin, Christian Theobalt, and Maneesh Agrawala. 2019.  Text-Based Editing of Talking-Head Video.  _ACM Trans. Graph._ 38, Article 68 (2019), 14 pages.  https://doi.org/10.1145/3306346.3323028
  * Haris et al. (2018) Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita. 2018\.  Deep Back-Projection Networks for Super-Resolution. In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 1664–1673.  https://doi.org/10.1109/CVPR.2018.00179
  * Haris et al. (2019) Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita. 2019\.  Recurrent Back-Projection Network for Video Super-Resolution. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Héas et al. (2015) Patrick Héas, Angélique Drémeau, and Cédric Herzet. 2015.  Estimation of Super-Resolved Video Dynamics.  _CoRR_ abs/1506.00473 (2015).  arXiv:1506.00473
  * Hegde et al. (2021) Sindhu B. Hegde, K.R. Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, and C.V. Jawahar. 2021\.  Visual Speech Enhancement Without a Real Visual Stream. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_. 1926–1935.
  * Hempel et al. (2022) Thorsten Hempel, Ahmed A Abdelrahman, and Ayoub Al-Hamadi. 2022\.  6D Rotation Representation For Unconstrained Head Pose Estimation.  _arXiv preprint arXiv:2202.12555_ (2022).
  * Heusel et al. (2018) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2018\.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium.  arXiv:1706.08500 [cs.LG]
  * Ho et al. (2019) Jonathan Ho, Evan Lohn, and P. Abbeel. 2019.  Compression with Flows via Local Bits-Back Coding. In _NeurIPS_.
  * Hu and Ramanan (2017) Peiyun Hu and Deva Ramanan. 2017.  Finding Tiny Faces. In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Jamaludin et al. (2019) Amir Jamaludin, Joon Son Chung, and Andrew Zisserman. 2019\.  You said that? : Synthesising talking faces from audio.  _International Journal of Computer Vision_ (2019).
  * Jiang et al. (2018) Huaizu Jiang, Deqing Sun, V. Jampani, Ming-Hsuan Yang, Erik G. Learned-Miller, and Jan Kautz. 2018\.  Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation.  _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2018), 9000–9008.
  * K R et al. (2019) Prajwal K R, Rudrabha Mukhopadhyay, Jerin Philip, Abhishek Jha, Vinay Namboodiri, and C V Jawahar. 2019.  Towards Automatic Face-to-Face Translation. In _Proceedings of the 27th ACM International Conference on Multimedia_ (Nice, France) _(MM ’19)_. ACM, 9 pages.  https://doi.org/10.1145/3343031.3351066
  * Khani et al. (2021) Mehrdad Khani, Vibhaalakshmi Sivaraman, and Mohammad Alizadeh. 2021.  Efficient Video Compression via Content-Adaptive Super-Resolution. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 4521–4530.
  * Kim et al. (2018b) Changil Kim, Hijung Valentina Shin, Tae-Hyun Oh, Alexandre Kaspar, Mohamed Elgharib, and Wojciech Matusik. 2018b.  On Learning Associations of Faces and Voices. In _Proceedings of Asian Conference on Computer Vision (ACCV)_.
  * Kim et al. (2019) Deokyun Kim, Minseon Kim, Gihyun Kwon, and Dae-Shik Kim. 2019\.  Progressive Face Super-Resolution via Attention to Facial Landmark. In _Proceedings of the 30th British Machine Vision Conference (BMVC)_.
  * Kim et al. (2018a) H. Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Nießner, P. Pérez, Christian Richardt, M. Zollhöfer, and C. Theobalt. 2018a.  Deep video portraits.  _ACM Transactions on Graphics (TOG)_ 37 (2018), 1 – 14.
  * Kim et al. (2015) Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2015.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks.  _CoRR_ abs/1511.04587 (2015).  arXiv:1511.04587 http://arxiv.org/abs/1511.04587
  * Kim et al. (2021) Jonghyun Kim, Gen Li, Inyong Yun, Cheolkon Jung, and Joongkyu Kim. 2021.  Edge and identity preserving network for face super-resolution.  _Neurocomputing_ 446 (2021), 11–22.  https://doi.org/10.1016/j.neucom.2021.03.048
  * Lim et al. (2017) Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017.  Enhanced Deep Residual Networks for Single Image Super-Resolution. In _2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_. 1132–1140.  https://doi.org/10.1109/CVPRW.2017.151
  * Ma et al. (2020) Cheng Ma, Zhenyu Jiang, Yongming Rao, Jiwen Lu, and J. Zhou. 2020.  Deep Face Super-Resolution With Iterative Collaboration Between Attentive Recovery and Landmark Estimation.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ (2020), 5568–5577.
  * Menon et al. (2020) Sachit Menon, Alex Damian, McCourt Hu, Nikhil Ravi, and Cynthia Rudin. 2020.  PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models. In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Nagrani et al. (2018) Arsha Nagrani, Samuel Albanie, and Andrew Zisserman. 2018\.  Seeing Voices and Hearing Faces: Cross-Modal Biometric Matching.  _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ (2018), 8427–8436.
  * Narayanan and Kiong (2013) Arun Shankar Narayanan and Tan Kok Kiong. 2013.  Video conferencing solution for medical applications in low-bandwidth networks. In _2013 Australasian Telecommunication Networks and Applications Conference (ATNAC)_. 195–200.  https://doi.org/10.1109/ATNAC.2013.6705380
  * Oh et al. (2019) Tae-Hyun Oh, Tali Dekel, Changil Kim, Inbar Mosseri, William T. Freeman, Michael Rubinstein, and Wojciech Matusik. 2019.  Speech2Face: Learning the Face Behind a Voice. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 7531–7540.  https://doi.org/10.1109/CVPR.2019.00772
  * Oquab et al. (2021) Maxime Oquab, Pierre Stock, Oran Gafni, Daniel Haziza, Tao Xu, Peizhao Zhang, Onur Çelebi, Yana Hasson, Patrick Labatut, Bobo Bose-Kolanu, Thibault Peyronel, and Camille Couprie. 2021.  Low Bandwidth Video-Chat Compression using Deep Generative Models.  _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ (2021), 2388–2397.
  * Prajwal et al. (2020) K R Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, and C.V. Jawahar. 2020.  A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild. In _Proceedings of the 28th ACM International Conference on Multimedia_ _(MM ’20)_. 484–492.  https://doi.org/10.1145/3394171.3413532
  * Sajjadi et al. (2018) Mehdi S. M. Sajjadi, Raviteja Vemulapalli, and Matthew Brown. 2018.  Frame-Recurrent Video Super-Resolution. In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Siarohin et al. (2019) Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. 2019\.  First Order Motion Model for Image Animation. In _Conference on Neural Information Processing Systems (NeurIPS)_.
  * Tang et al. (2021) Jiaxiang Tang, Xiaokang Chen, and Gang Zeng. 2021.  _Joint Implicit Image Function for Guided Depth Super-Resolution_.  Association for Computing Machinery, New York, NY, USA, 4390–4399.  https://doi.org/10.1145/3474085.3475584
  * Thies et al. (2018) Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. 2018\.  Face2Face: Real-Time Face Capture and Reenactment of RGB Videos.  _Commun. ACM_ 62, 1 (Dec. 2018), 96–104.  https://doi.org/10.1145/3292039
  * Timofte et al. (2018) Radu Timofte, Shuhang Gu, Jiqing Wu, and Luc Van Gool. 2018\.  NTIRE 2018 Challenge on Single Image Super-Resolution: Methods and Results. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_.
  * Torralba et al. (2007) Antonio Torralba, Kevin P. Murphy, and William T. Freeman. 2007\.  Sharing Visual Features for Multiclass and Multiview Object Detection.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 29, 5 (2007), 854–869.  https://doi.org/10.1109/TPAMI.2007.1055
  * Wang et al. (2019) Ting-Chun Wang, Ming-Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro. 2019\.  Few-shot Video-to-Video Synthesis. In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Wang et al. (2021) Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. 2021\.  One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
  * Wu et al. (2021) Haozhe Wu, Jia Jia, Haoyu Wang, Yishun Dou, Chao Duan, and Qingshan Deng. 2021\.  _Imitating Arbitrary Talking Style for Realistic Audio-Driven Talking Face Synthesis_.  Association for Computing Machinery, New York, NY, USA, 1478–1486.  https://doi.org/10.1145/3474085.3475280
  * Xu et al. (2014) Li Xu, Jimmy SJ Ren, Ce Liu, and Jiaya Jia. 2014\.  Deep Convolutional Neural Network for Image Deconvolution. In _Advances in Neural Information Processing Systems_ , Vol. 27. Curran Associates, Inc.
  * Yang (2019) Bin-Cheng Yang. 2019\.  Super Resolution Using Dual Path Connections _(MM ’19)_. Association for Computing Machinery, New York, NY, USA, 1552–1560.  https://doi.org/10.1145/3343031.3350878
  * Yao et al. (2020) Xin-Wei Yao, Ohad Fried, K. Fatahalian, and Maneesh Agrawala. 2020\.  Iterative Text-based Editing of Talking-heads Using Neural Retargeting.  _ArXiv_ abs/2011.10688 (2020).
  * Zhang et al. (2021) Chenxu Zhang, Yifan Zhao, Yifei Huang, Ming Zeng, Saifeng Ni, Madhukar Budagavi, and Xiaohu Guo. 2021.  FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learning. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 3867–3876.
  * Zhang et al. (2017) Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo Wang, and S. Li. 2017\.  S3FD: Single Shot Scale-Invariant Face Detector.  _2017 IEEE International Conference on Computer Vision (ICCV)_ (2017), 192–201.
  * Zhou et al. (2021) Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. 2021\.  Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zhou et al. (2020) Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li. 2020.  MakeItTalk: Speaker-Aware Talking-Head Animation.  _ACM Transactions on Graphics_ 39, 6 (2020).

##  Appendix A SUPPLEMENTARY MATERIAL

###  A.1. Additional Network and Training Details

####  A.1.1. Frame-Interpolation Network

As discussed in the paper, we design a frame-interpolation network to further
reduce the bandwidth consumption. Our frame-interpolation network is a simple
encoder-decoder based architecture, majorly consisting of $3$D CNN layers. The
pipeline for transmitting $5$FPS low-resolution (LR) frames and applying our
frame-interpolation network to generate $25$FPS LR frames, and further
upsampling them to obtain the high-resolution (HR) talking-face video as
output is shown in Figure 8.

Figure 8. Our designed pipeline to transmit $5$FPS LR videos and generate the
final $25$FPS HR videos. We achieve a major reduction in the bits/pixel (BPP)
using our frame-interpolation network, since only $1$ in $5$ LR frames needs
to be transmitted.

####  A.1.2. Training Details

For training, we randomly sample a contiguous window of $T=5$ ($0.2$ seconds)
frames from the AVSpeech (Ephrat et al., 2018) or VoxCeleb2 (Chung et al.,
2018) datasets, resized to $256\times 256$ pixels. We extract the
corresponding $0.2$ seconds audio signal sampled at $16$kHz. Bicubic
degradation is used to downscale the ground-truth images $F_{gt}$ and generate
the corresponding LR images $F_{lr}$ at the given scale-factor. We train our
network using the Adam optimizer (adam) with an initial learning rate of
$1e^{-4}$, with a learning rate decay by a factor of $10$ after every half of
the total epochs. The training is stopped once the validation loss does not
improve for $10$ epochs.

####  A.1.3. Metrics

We use multiple metrics to evaluate the different aspects of the results. The
details of these metrics are given below.

PSNR: PSNR measures the mean squared error (MSE) between the generated and
ground-truth frames, thus correlating with the reconstruction quality.

SSIM: To measure the structural similarity between the patches of the frames,
we use the SSIM metric. SSIM is more robust to global illumination changes
than PSNR, which only considers pixel-wise absolute errors.

FID: To evaluate the perceptual quality of the generated frames, we use
Fréchet Inception Distance (FID), which measures the distance between the
generated and the ground-truth distributions.

LMD: To evaluate the structure of the generated faces, we report the Landmark-
distance (LMD) using FAN-based (Bulat and Tzimiropoulos, 2017) landmark
detector. We compute the landmark distance for face regions like eyes, lips,
eyebrows and nose for the generated and ground-truth frames.

LSE-D: We follow Wav2Lip (Prajwal et al., 2020) in measuring the lip-sync
accuracy by using the Lip Sync Error Distance (LSE-D) metric calculated using
the pre-trained SyncNet333https://github.com/joonson/syncnet_python (Chung and
Zisserman, 2016).

###  A.2. Additional Experiments and Evaluations

####  A.2.1. SR at multiple scale-factors

Figure 9. Qualitative results of our method at different scale-factors
($4\times$, $8\times$, $16\times$ and $32\times$). We can observe that our
approach generates consistent results across all scales. It is impressive to
note the high-quality results at high scale-factors like $32\times$. Figure
10. Performance comparison of different methods (metric: FID) at multiple
scale-factors. The gap in the performance of our method versus the existing
approaches, especially at higher scale-factors like $32\times$ indicates the
capability and robustness of our technique.

To analyze the performance of our model at various scale-factors, we plot the
variation of the FID metric across the scale-factors of $4\times$, $8\times$,
$16\times$, and $32\times$ in Figure 10. We can observe that the gap in
performance of our method versus the existing approaches increases as the
scale-factor increases. At smaller scale-factors like $4\times$, the existing
models SPARNet (Chen et al., 2020) and TecoGAN (Chu et al., 2020) performs
decently, however, at higher scale-factors, our model generates significantly
better results. This demonstrates that our model is able to recover the lost
details and produces plausible high-quality results even from very low-
resolution inputs.

Figure 9 shows the qualitative results generated by our upsampling models at
multiple scale- factors. We can observe consistent performance of our models
across the scale-factors. The level of facial details captured by our method
is impressive, even at higher scale-factors like $32\times$. The lip shape,
hair, and teeth (Figure 9 \- row $1$) and eyebrows, face texture and eyeballs
(Figure 9 \- row $2$) demonstrates that our model generates sharp, high-
frequency details, thus preserving most of the identity-specific information.

####  A.2.2. Analysis of the Target Identity Image

Our network takes a single HR target identity image as input. As discussed in
the paper, this image can be any sample frame, either from the same video or
any other image, with similar characteristics in terms of the face identity,
pose, clothing and background. We empirically verify this below using two
cases, one where we provide a target image of the same identity but from a
different video, and another where we provide a different identity altogether.

##### (i) Same Identity - Different Image:

As shown in Figure 11, we can see that our model is able to generate decent
results for an input target identity image sampled from a different video,
with slight variations in terms of clothing, illumination, background and
pose. Although there is room for improvement in quality, we can see that our
model captures the motion well and is also able to replicate the basic
identity details.

Figure 11. Our method works decently when the target identity image given as
input is from a different video of the same person.

##### (ii) Different Identity Image:

Figure 12 depicts our results when the input identity image is from a
different identity altogether. As Section $5$ of the paper explains, our model
does not work if the target identity and the LR frames do not match. This is
one of the advantages of our method since the creation of harmful “deepfakes”
is directly not possible from our model. We encourage the readers to see our
demo video for the video results.

Figure 12. Our method does not generate intelligible HR outputs if the target
identity image given as input is from a different person than the LR input
video, thus avoiding direct creation of “deepfakes”.

####  A.2.3. Human Evaluations

We conduct human evaluations as a part of subjective assessment, primarily to
evaluate the quality of the generated videos. We randomly select $20$ videos
from the curated AVSpeech test set (Ephrat et al., 2018) and in each video, we
select a random contiguous $5$ seconds video segment (a window of $125$
frames). The results for these segments generated from all the comparison
methods, our approach and the corresponding ground-truth videos are displayed
in random order. We ask $30$ participants to rate these videos on a scale of
$1-5$ based on the perceptual satisfaction. The participant group spans
members of $22$ \- $40$ years with an almost equal male-female ratio. The
users are asked to consider the overall quality based on the facial texture,
lip-sync and the sharpness of frames. The mean opinion scores of all the
participants are displayed in Figure 13. In-line with our qualitative and
quantitative results, our approach outperforms all the other methods by a very
large margin. Another interesting point to note is that our scores are quite
close to the scores of ground-truth videos, indicating that we are able to
recover the actual details to a large extent.

Figure 13. Mean opinion scores of various methods obtained from human
evaluations. Higher scores indicate better quality and perceptual
satisfaction.

###  A.3. Additional Qualitative Results

####  A.3.1. Talking-Face Video Upsampling

We show additional qualitative samples generated by our model in Figure 14.
In-line with the previous results, our model is able to accurately match the
ground-truth and replicate the identity-specific details very well.

Figure 14. Additional qualitative results: Our model captures the identity-
specific facial details and generates faithful results. Figure 15. Qualitative
comparison with Audio-driven Talking-Face Generation (A2TF) and Face Re-
enactment (FR) methods. A2TF methods fail to preserve the accurate pose of the
actual identity; FR methods also face similar issues and also degrades the
quality of generations. In contrast, our method produces the best results in
comparison, preserves the actual identity and also generates high-quality
outputs.

####  A.3.2. Audio-driven Talking-Face Generation (A2TF) and Face Re-enactment
(FR) comparison

In Section $4.3$ of our main paper, we demonstrated the quantitative
comparisons with A2TF and FR methods. To further understand and analyse these
comparisons, we include qualitative comparison in Figure 15. Similar to Table
$4$, we compare with two A2Tf works, (i) Wav2Lip (Prajwal et al., 2020) and
(ii) MakeItTalk (Zhou et al., 2020) and two FR works, (i) FOMM (Siarohin et
al., 2019) and (ii) PCAVS (pcavs). From the figure, we can infer that the A2TF
methods do not preserve the head movements of the original video. This is
because these methods ingest a single target identity image and animate only
the lip region (lower-half of the face), thus the upper half of the face,
i.e., the head pose remains static (as that of the target identity input). The
results generated by FR methods are also not very accurate, in terms of the
pose and also the generation quality. In comparison, we find that our method
produces the best results, thus validating the robustness of our approach.
Please refer to our demo video for full video comparison.

###  A.4. Ablation Experiments

In order to analyse the various design choices of our network, we perform
several ablation experiments. All the results are reported on the test set of
AVSpeech dataset (Ephrat et al., 2018).

####  A.4.1. Importance of the Audio Signal

We train a network by discarding the audio input in our proposed network to
highlight the importance of the audio signal. As shown in Table 5, it results
in poor reconstruction of faces and inaccurate lip shape generation (indicated
by LSE-D metric). This signifies the importance of the audio information in
generating plausible faces as well as precise lip and mouth shapes. Without
the audio signal, the model does not have adequate information in the LR input
to recover the fine-grained lip shape, an essential feature in talking-face
videos. The audio information also assists the other facial features that
define the identity and appearance of a person.

We also show a qualitative sample in Figure 16 to illustrate the benefits of
incorporating the audio signal in our pipeline. We replace the audio input
with an audio signal from a different speaker. From Figure 16, we can observe
that the audio signal is crucial for precise lip-shape generation and identity
recovery.

Table 5. The audio signal helps to recover basic identity details and also enforces accurate lip shape generation. Method |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  FID$\downarrow$ |  LMD$\downarrow$ |  LSE-D$\downarrow$
---|---|---|---|---|---
W/o audio | 24.09 | 0.70 | 15.14 | 0.178 | 13.98
With audio | 25.06 | 0.73 | 11.54 | 0.162 | 12.43
Figure 16. Audio signal helps in accurate lip-shape generation.

####  A.4.2. Significance of our Proposed Landmark-based Region Loss

We assess the importance of optimizing the landmark-based region loss in Table
6 and Figure 17. Without using the region loss, the model struggles to
estimate the accurate pose of different face regions, for example, eyes and
nose (see Figure 17 \- column $2$). A large difference, especially in the LMD
metric, can be observed in Table 6. This occurs due to the lack of supervision
regarding the human face structure. We provide this additional control using
the landmark-based region loss, which helps not only in enforcing the local
correspondence, but also assists in smooth transitions for pose variations
(can be seen in the demo video).

Table 6. The landmark-based region loss helps to preserve the overall face structure. Method |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  FID$\downarrow$ |  LMD$\downarrow$ |  LSE-D$\downarrow$
---|---|---|---|---|---
W/o region loss | 23.96 | 0.67 | 15.54 | 0.230 | 13.12
With region loss | 25.06 | 0.73 | 11.54 | 0.162 | 12.43
Figure 17. Optimizing the landmark-based region loss improves the overall
generation quality by handling the pose variations and predicting the accurate
face regions.

####  A.4.3. Importance of the Backbone Network

As discussed in Section $3.2$ of the paper, we pre-train our backbone network
using the input LR frames and audio signal, to extract the elementary facial
details and motion information. To analyze the importance of pre-training, we
directly train our animation model end-to-end without pre-training the
backbone. Table 7 shows the results. We can notice the effect of pre-training
in Table 7. The outputs from the backbone network are considered as “driving
video” in our end-to-end trainable animation model; thus, capturing the basic
face details is crucial for the model to work well.

Table 7. Pre-training the backbone network helps to capture the basic face details and motion information, thus improving the overall model’s performance. Method |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  FID$\downarrow$ |  LMD$\downarrow$ |  LSE-D$\downarrow$
---|---|---|---|---|---
W/o pre-training | 21.90 | 0.69 | 19.23 | 0.220 | 16.61
With pre-training | 25.06 | 0.73 | 11.54 | 0.162 | 12.43

####  A.4.4. Temporal Window

We justify the use of $T=5$ contiguous frames in our network in Table 8. The
performance of the network deteriorates when only $3$ frames are used. This is
mainly because the network does not learn the temporal aspect without adequate
context. The performance of using $5$ and $7$ frames are very similar, which
suggests that the context of $5$ frames is sufficient for the model,
especially to learn specialized temporal aspects such as lip-sync. This is
inline with the previous work (Prajwal et al., 2020) where it is shown that
the best performance is achieved using a window of $5$ frames.

Table 8. A context window of $T=5$ contiguous frames allows the model to effectively capture the temporal relationships such as lip-sync. Frames |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  FID$\downarrow$ |  LMD$\downarrow$ |  LSE-D$\downarrow$
---|---|---|---|---|---
$3$ | 23.88 | 0.69 | 15.78 | 0.187 | 12.78
$5$ | 25.06 | 0.73 | 11.54 | 0.162 | 12.43
$7$ | 25.13 | 0.74 | 11.10 | 0.157 | 12.28

###  A.5. A Baseline Model for Handling Emotions

Figure 18. Qualitative results from our model on the emotional dataset, MEAD
(MEAD_eccv_2020). Our model handles the emotions and generates consistent
results across all the emotions.

The datasets we considered so far, namely, AVSpeech (Ephrat et al., 2018) and
VoxCeleb2 (Chung et al., 2018) do not contain emotion annotations and are not
specifically designed to evaluate the expressions. Thus, we consider to use
the large-scale emotional dataset, “Multi-view Emotional Audio-visual Dataset”
(MEAD) (MEAD_eccv_2020). MEAD is a challenging corpus comprising eight
different emotions from $60$ actors at different intensity levels, captured at
multiple view angles. To validate the capability of our model in handling the
variations in expressions, we train our talking-face upsampling model on MEAD.
Note that we do not add or modify our architecture for this training, all the
training settings are the same as we used for other datasets.

Figure 18 shows the results generated by our model. We can see that our model
is able to capture emotions very well. Since MEAD contains large amount of
speaker-specific data, our model learns the fine-grained characteristics of
individual speakers. This is reflected in the results shown in Figure 18.
Details like wide opening of mouth, glabellar lines (wrinkles) on the forehead
and dimples near teeth are also synthesized by our model. Appropriate
extensions (like emotion classifier discriminator) can be added to our model
in the future to be able to handle extreme variations of expressions (like
shown in Figure $8(b)$ of the main paper).