# DATA STANDARDIZATION FOR ROBUST LIP SYNC

###### Abstract

Lip sync is a fundamental audio-visual task. However, existing lip sync
methods fall short of being robust to the incredible diversity of videos taken
in the wild, and the majority of the diversity is caused by compound
distracting factors that could degrade existing lip sync methods. To address
these issues, this paper proposes a data standardization pipeline that can
produce standardized expressive images while preserving lip motion information
from the input and reducing the effects of compound distracting factors. Based
on recent advances in 3D face reconstruction, we first create a model that can
consistently disentangle expressions, with lip motion information embedded.
Then, to reduce the effects of compound distracting factors on synthesized
images, we synthesize images with only expressions from the input,
intentionally setting all other attributes at predefined values independent of
the input. Using synthesized images, existing lip sync methods improve their
data efficiency and robustness, and they achieve competitive performance for
the active speaker detection task.

Index Terms—  Lip motions, disentanglement, synthesis.

##  1 INTRODUCTION

Lip sync is a fundamental audio-visual(AV) task. Its primary function is to
determine when the audio and visual streams are out of sync. For this reason,
it is frequently formulated as a cross-modal matching task and is resolved by
comparing the representations of two modalities [1]. Active speaker detection
(ASD), whose goal is to detect which or no subject in visual streams is the
speaker, can be solved using such a formulation. A solid baseline for
resolving ASD is established by performing lip sync on each subject who
presents [2].

Lip sync performance, however, degrades in real-world conditions, just like
some other AV tasks do. One major cause of this problem is the very diverse
nature of videos taken in the wild, with the majority of the diversity caused
by distracting factors that could degrade existing lip sync methods. For
instance, there is a considerable performance decrease when faces are in large
head poses [3][4].

There are primarily two types of strategies in the literature for improving
the robustness of audio-visual methods. One is a data-driven strategy.
Previous research, for example, has shown that training with a large number of
non-frontal faces progressively makes audio-visual systems invariant to big
head poses [3][4]. However, adopting an entirely data-driven strategy would
make lip sync data-hungry and hard to optimize, as a significant amount of
data covering a wide range of combinations would be needed to address compound
distracting factors.

As a complement, others take a more domain-knowledge-based strategy. For
instance, several face frontalization techniques, achieved via either 3D
guided image warping [5] or generative models [6], are proposed as one way to
explicitly reduce the effects of head poses. According to domain knowledge, in
addition to head poses, the effects of subject-related factors (primarily
identity and appearance) and scene-related factors (primarily illumination and
background) should also be considered as distracting factors, for the reason
that judgments of audio-visual synchronization should not be influenced by
their variations. Meanwhile, lip motion information should be preserved as
they are the primary visual cue for lip sync. Methods that can disentangle lip
motions from compound perturbing factors are thus desired.

Lip motions can be viewed as a form of facial expression in a broader sense.
ExpNet [7] and DeepExp3D [8] demonstrate that the 3D Morphable Model (3DMM)
[9] may be used to acquire disentangled expressions. The 3DMM defines a
parametric space where expression is unrelated to other attributes. In
particular, this expression subspace is demonstrated to capture speaking-
related facial motions well [8]. Moreover, the image formation model (see
Sec.3) built on top of 3DMM provides a larger parametric space with additional
control over attributes that are related to many factors.

This paper proposes a data standardization pipeline (DSP) that can produce
standardized expressive images with the effects of compound distracting
factors reduced by leveraging the image formation model. First, a network is
developed that can consistently disentangle expression from the input at the
video level. Then, using the image formation model, expressions disentangled
from the input are utilized to synthesize expressive images, with all other
attributes corresponding to distracting factors set to predefined values
independent of the input to reduce their effects on the synthesized images.
Experiments demonstrate that by utilizing the DSP, existing lip sync methods
increase their data efficiency and generalizability and achieve competitive
performance for the ASD task on the recent ASW dataset [2].

The rest of the paper is organized as follows: Sec.2 discusses related works.
Then, building on the preliminary information provided in Sec.3, the DSP is
then thoroughly described in Sec.4. The experimental settings and results are
presented in Sec.5. Finally, Sec.6 concludes the paper.

##  2 RELATED WORK

3DMM coefficients. Several ambiguities make estimating 3DMM coefficients
difficult [9]. Major expression-related ambiguity results from the fact that
certain changes in facial shape can be ascribed to either identity variations
or expressions, which can be inferred from Eq.1. One way of resolving the
ambiguity is to use multiple images of a subject, assuming that they are of
the same underlying subject-specific attributes. In a recent benchmark [10],
Deep3D [11] and MGCNet [12] are shown to fit the mouth area well. But they
exhibit inconsistent ascriptions across images as they attempt to estimate all
attributes using preimage features. Others [13], on the other hand, optimize
global subject-specific attributes for images of a subject and ascribe image-
varying facial morphings to the expression. However, optimization-based
methods are less robust in the wild [7]. In this paper, we integrate this
global constraint into Deep3D [11] to improve expression disentanglement
consistency.

Face synthesis/generation. There are various methods for face frontalization.
Many people utilize generative models to translate a profile face into a
frontal one. However, they may not preserve the expression of the input image
[5]. Recently, [5] proposes to synthesize expression-preserving frontal faces
via warping. However, adapting those methods to deal with compound factors
would be difficult. Otherwise, it has been shown that full parametric models
(see Sec.3) can be used to synthesize faces with diversified attribute
combinations [11][12]. This method involves both controllability and steady
synthesis quality. In this paper, we explore its potential in handling
compound distracting factors.

Lip sync and ASD. For lip sync, SyncNet [3] is a reliable baseline.
PerfectMatch [1] enhances performance by introducing a multi-way matching
approach. With regard to ASD, diverse viewpoints exist. The most widely used
AVA dataset [14] just demands the association of audio and visual data,
whereas the more recent ASW dataset [2] requires the synchronization of two
modalities, which excludes instances like dub-subbed movies. Since lip motions
and dubbings are unlikely to match, in this paper, we resolve to the latter.

##  3 PERLIMINARIES

The 3DMM. In 3DMM, the textured 3D face model
$\boldsymbol{F}(\boldsymbol{S},\boldsymbol{T})$ is defined by the face shape
$\boldsymbol{S}$ and the face texture $\boldsymbol{T}$, and both are
represented as parametric linear models:

$\displaystyle\boldsymbol{S}(\boldsymbol{\alpha},\boldsymbol{\beta})$
$\displaystyle=\boldsymbol{\bar{S}}+\boldsymbol{B}_{id}\boldsymbol{\alpha}+\boldsymbol{B}_{exp}\boldsymbol{\beta}$
(1) $\displaystyle\boldsymbol{T}(\boldsymbol{\delta})$
$\displaystyle=\boldsymbol{\bar{T}}+\boldsymbol{B}_{tex}\boldsymbol{\delta}$
(2)

where $\boldsymbol{\bar{S}}$ and $\boldsymbol{\bar{T}}$ are the mean face
shape and texture; $\boldsymbol{B}_{id}$, $\boldsymbol{B}_{exp}$, and
$\boldsymbol{B}_{tex}$ are PCA bases for identity, expression, and texture,
respectively. Specifically, Basel Face Model 2009 [15] and FaceWarehouse [16]
built bases are adopted, thus identity
$\boldsymbol{\alpha}\in\mathbb{R}^{80}$, expression
$\boldsymbol{\beta}\in\mathbb{R}^{64}$, and texture
$\boldsymbol{\delta}\in\mathbb{R}^{80}$.

The illumination model. Under the distant smooth illumination assumption and
the Lambertian skin surface assumption, the illumination is modeled via
Spherical Harmonics(SH) [17], and per-vertex radiosity is

$\boldsymbol{t}^{\prime}_{i}(\boldsymbol{n}_{i},\boldsymbol{t}_{i}|\boldsymbol{\gamma})=\boldsymbol{t}_{i}\sum_{b=1}^{B^{2}}\gamma_{b}\Phi_{b}(\boldsymbol{n}_{i})$
(3)

where $\boldsymbol{n}_{i}$ is the normal, $\boldsymbol{t}_{i}$ is the texture
color, $\Phi_{b}$ are SH basis functions. We choose $B$=3 for each of the red,
green, and blue channels, resulting in illumination coefficients
$\boldsymbol{\gamma}\in\mathbb{R}^{27}$.

The image formation model. The pose $\boldsymbol{p}\in\mathbb{R}^{6}$ is
represented by a rotation $\boldsymbol{R}\in SO(3)$, parameterized with three
Euler angles, and a translation $\boldsymbol{t}\in\mathbb{R}^{3}$. Thus, the
image formation process can be presented as

$\boldsymbol{I}(\boldsymbol{\alpha},\boldsymbol{\beta},\boldsymbol{\delta},\boldsymbol{\gamma},\boldsymbol{p})=\Pi(\boldsymbol{R}\boldsymbol{F}(\boldsymbol{S},\boldsymbol{T}^{\prime})+\boldsymbol{t})$
(4)

where $\Pi:\mathbb{R}^{3}\rightarrow\mathbb{R}^{2}$ is the perspective
projection matrix; and $\boldsymbol{T}^{\prime}$ is the illuminated 3D face
texture.

By setting proper coefficients, we can set faces and environments accordingly
and synthesize images of corresponding attributes through rendering. See [9]
for more details.

##  4 APPROACH

In this section, we introduce the design requirements and implementations of
the proposed DSP.

###  4.1 Overview of DSP

As depicted in Fig.1, the visual front-end, audio front-end, and task-related
back-end are the three parts that make up modern lip sync model architectures
[1]. The DSP should be placed between the raw images and the visual front-end.
It takes raw images as inputs and produces standardized images, which are then
fed into the visual front-end. Further, the DSP is built using the parametric
image formation model described in Sec.3 and thus has two components: a
network for estimating the required coefficients from the input, and a
renderer that uses the coefficients to synthesize images. Their
implementations should fulfill the design requirements.

  1. 1.

The DSP is robust on its own in the wild.

  2. 2.

Preserving lip motion information from the raw images.

  3. 3.

Reducing compound distracting factors from the raw images.

The DSP satisfies requirement 1 since the parametric image formation model is
physical-rule based. To satisfy requirement 2, a network that can accurately
estimate expression coefficients from the raw images is developed, see Sec.4.2
to 4.5. To satisfy requirement 3, synthesized images are produced with
modified coefficients, see Sec.4.6.

###  4.2 Training Dataset Construction

For training, several image collections are necessary. First, a large number
of single-speaker videos are gathered, and then each video is processed for
face detection/tracking and 3D landmark detection [11] in order to create face
tracks [2]. Each face track is a sequential sequence of a subject’s facial
images, and a subject may show up in more than one face track. Then, around
$M$ facial images are sampled from each of the high-quality face tracks [8].
Finally, the dataset contains $K$ collections, and each has $M_{k}$ pairs of
facial images and landmarks $\\{l_{n}\\}$ of a subject, i.e.
$\boldsymbol{C}_{k}=\left\\{\left(\boldsymbol{I}_{i},\\{l_{n}\\}_{i}\right)\right\\}_{i=1}^{M_{k}}$.

We use speaking videos for the following reasons: 1) Facial images from short-
duration videos are more likely to fulfill the assumption that they are of the
same underlying subject-specific attributes. 2) They enhance training by
introducing diversified lip motions and implicit temporal stability.

Fig. 1: Lip sync with DSP schematic.

(a) Model architecture

(b) Face synthesis

Fig. 2: Overview of the proposed approach.

###  4.3 Model Architecture

The model consists of a regression network and two trainable matrices, as
depicted in Fig.2(a). The estimation of preimage attributes is carried out
using the regression network, termed as E-Net. Similar to [11] [12], the E-Net
is implemented as a ResNet-50 [18] with the last fully-connected layer of 97
dimensions, for regressing a 97-dimensional vector of preimage attributes
($\boldsymbol{\beta}\in\mathbb{R}^{64}$,
$\boldsymbol{\gamma}\in\mathbb{R}^{27}$, $\boldsymbol{p}\in\mathbb{R}^{6}$).
One of those trainable matrices is the shared identity matrix of dimension
80-by-$K$, where 80 is the dimension of identity
($\boldsymbol{\alpha}\in\mathbb{R}^{80}$) and $K$ is the number of collections
in the dataset. It is denoted as shared because the identity shared by all of
the facial images in the $k$-th collection is represented in its $k$-th
column. Similarly, the other is the shared texture matrix of dimension
80-by-$K$ since $\boldsymbol{\delta}\in\mathbb{R}^{80}$.

###  4.4 Loss Functions

To measure the discrepancy for an estimated coefficient
$\boldsymbol{\hat{x}}=\\{\boldsymbol{\hat{\alpha}}_{k},\boldsymbol{\hat{\delta}}_{k},\boldsymbol{\hat{\beta}},\boldsymbol{\hat{p}},\boldsymbol{\hat{\gamma}}\\}$,
different losses are used, following [11].

The photometric loss. It measures the photometric consistency between the
input $\boldsymbol{I}$ and the synthesized one $\boldsymbol{I}^{\prime}$.

$L_{pho}\left(\boldsymbol{x}\right)=\dfrac{\sum_{i\in\boldsymbol{\mathcal{M}}}\boldsymbol{A}_{i}\cdot\left\|\boldsymbol{I}_{i}-\boldsymbol{I}_{i}^{\prime}\left(\boldsymbol{x}\right)\right\|_{2}}{\sum_{i\in\boldsymbol{\mathcal{M}}}\boldsymbol{A}_{i}}$
(5)

where $\boldsymbol{\mathcal{M}}$ and $\boldsymbol{A}$ are two masks indicating
the facial area and the per-pixel skin probability [11], respectively.

The landmarks distance(LMD) loss. It measures the geometric consistency
between detected landmarks $\\{l_{n}\\}$ and ones $\\{l_{n}^{\prime}\\}$
reprojected from predefined 3D shape vertices [11].

$L_{lan}\left(\boldsymbol{x}\right)=\dfrac{1}{N}\sum^{N}_{n=1}\omega_{n}\left\|l_{n}-l_{n}^{\prime}\left(\boldsymbol{x}\right)\right\|_{2}$
(6)

where $\omega_{n}$ is the weight of the $n$-th landmark. Lip landmarks are
weighted to 10, otherwise 1.

The regularization loss. It adds zero-mean Gaussian distribution prior
constraints on 3DMM coefficients.

$L_{reg}\left(\boldsymbol{x}\right)=\omega_{\boldsymbol{\alpha}}\left\|\boldsymbol{\alpha}\right\|^{2}+\omega_{\boldsymbol{\beta}}\left\|\boldsymbol{\beta}\right\|^{2}+\omega_{\boldsymbol{\delta}}\left\|\boldsymbol{\delta}\right\|^{2}$
(7)

where $\omega_{\boldsymbol{\alpha}}=0.5$, $\omega_{\boldsymbol{\beta}}=2.0$
and $\omega_{\boldsymbol{\delta}}=0.5$.

Note that a stronger regularization is applied to the expression term to
prevent identifying all variations in facial shapes as expressions and making
the identity attribute trivial.

In total, all terms are weighted combined as

$\displaystyle L\left(\boldsymbol{x}\right)=$
$\displaystyle\lambda_{pho}L_{pho}\left(\boldsymbol{x}\right)+\lambda_{lan}L_{lan}\left(\boldsymbol{x}\right)+\lambda_{reg}L_{reg}\left(\boldsymbol{x}\right)$
(8)

where $\lambda_{pho}=1.92$, $\lambda_{lan}=1.6e{-3}$, and
$\lambda_{reg}=3.0e{-4}$.

###  4.5 Training Strategy

The whole model, including the E-Net and two trainable matrices, is trained
jointly with built collections in a way similar to [11], with the following
modifications made. 1) As depicted in Fig.2(a), instead of being regressed,
subject-specific attributes are trainable parameters indexed from shared
matrices and shall be updated during training. 2) For a batch, we sample
multiple collections first, then several data from each collection. Data from
the same collection constrain one another and assist in resolving ambiguities
as they share the same subject-specific attributes but differing preimage
ones.

###  4.6 Face Synthesis

We propose to synthesize a standardized face with depth as an input for
subsequent AV tasks. As described in Sec.4.1, it is expected that the effects
of compound distracting factors from the raw inputs will be reduced in the
synthesized images while the expression information from the raw input will be
preserved. This can be done by setting appropriate coefficients to the image
formation model, as stated in Sec.3.

In specific, we utilize modified coefficients for rendering instead of all
coefficients estimated from the raw input. As depicted in Fig.2(b), the
trained E-Net estimates the expression singly from the input. Meanwhile,
attributes corresponding to distracting factors are explicitly set to
predefined default values, regardless of inputs. By rendering with modified
coefficients, the synthesized images are devoid of the effects of compound
distracting factors while preserving the expression from the raw images. For
the ease of subsequent AV tasks, we set uniform white illumination, mean
identity, mean texture, and frontal pose as defaults, so that
$\boldsymbol{\hat{x}}=\\{\boldsymbol{\alpha}=\boldsymbol{0},\boldsymbol{\delta}=\boldsymbol{0},\boldsymbol{\hat{\beta}},\boldsymbol{\gamma}=\boldsymbol{\gamma}_{\boldsymbol{0}},\boldsymbol{p}=\boldsymbol{p}_{\boldsymbol{0}}\\}$,
and render an expressive image
$\boldsymbol{I}^{\prime}(\boldsymbol{\hat{x}})$. Examples are given in Fig.3,
where raw images (column a) vary greatly whereas synthesized images (column e)
are standardized and free of distracting factors (pose, illumination, etc.).

We can also render a corresponding pseudo depth map $\boldsymbol{D}^{\prime}$
in order to more accurately describe 3D lip motions. This is done by rendering
with $\boldsymbol{Z}^{\prime}_{\boldsymbol{S}}$ instead of illuminated
$\boldsymbol{T}^{\prime}$ as the texture (see Eq.4), where
$\boldsymbol{Z}^{\prime}_{\boldsymbol{S}}$ is the normalized Z-coordinator of
$\boldsymbol{S}$,

$\boldsymbol{Z}^{\prime}_{\boldsymbol{S}}=\frac{\boldsymbol{Z}_{\boldsymbol{S}}-\min(\boldsymbol{Z}_{\boldsymbol{S}})}{\max(\boldsymbol{Z}_{\boldsymbol{S}})-\min(\boldsymbol{Z}_{\boldsymbol{S}})}$
(9)

Overall, we synthesize expressive RGB/RGBD images from raw facial images.
Notably, we also reduce the effect of the often poor image quality found in
raw data via synthesis.

##  5 EXPERIMENTS

###  5.1 Datasets

We develop the proposed model and the lip-sync model using VoxCeleb2 [19]. Its
dev-split has over 1 million utterances from 145569 videos of 5994 subjects
and its test-split has 36237 utterances from 4911 videos of 118 subjects.

To develop the proposed model, we collect 5000 subjects from the dev-split.
Ten utterances are sampled from each of the two videos for each subject,
resulting in $K=50k$ collections with each having about $M=50$ samples.
Separately, a minidev-split consisting of 5000 utterances from 500 subjects is
sampled from the dev-split for training the lip-sync model, while the entire
test split is used for evaluation. Note that the minidev-split is less than
${1}/{7}$ the size of the test-split.

Regarding the ASD task, we directly evaluate the trained lip-sync models on
the ASW dataset’s test split [2]. The test split consists of 53 videos (51
accessible), with 4.5 hours of active (i.e., sync) face tracks and 3.4 hours
of inactive ones.

Fig. 3: Snapshots of synthesized data. (a) raw images; (b)-(d) synthesized RGB, with MGCNet estimated id, Deep3D estimated id, and Deep3D estimated id and exp, respectively; (e)-(f) synthesized RGBD with E-Net estimated exp. Table 1: Lip-fit accuracies. Lower is better. Method |  Attributes | Group LMD
---|---|---
min | max | avg.
GT |  $\boldsymbol{\alpha}_{gt}$, $\boldsymbol{\beta}_{gt}$ | 1.085 | 1.093 | 1.091
Deep3D [11] |  $\boldsymbol{\alpha}_{gt}$, $\boldsymbol{\hat{\beta}}$ | 2.948 | 2.975 | 2.967
$\boldsymbol{\hat{\alpha}}$, $\boldsymbol{\hat{\beta}}$ | 2.339 | 2.369 | 2.358
MGCNet [12] |  $\boldsymbol{\alpha}_{gt}$, $\boldsymbol{\hat{\beta}}$ | 3.353 | 3.372 | 3.365
$\boldsymbol{\hat{\alpha}}$, $\boldsymbol{\hat{\beta}}$ | 2.902 | 2.920 | 2.912
Ours E-Net |  $\boldsymbol{\alpha}_{gt}$, $\boldsymbol{\hat{\beta}}$ | 1.235 | 1.241 | 1.240

###  5.2 Results on Expression Disentanglement

According to Sec.4.5, we first train the proposed model. More specifically,
two matrices are initialized with Gaussian noise
$\mathcal{N}(\mu=0,\,\sigma=0.01)$ and the E-Net is ImageNet-pretrained. Two
collections and eight samples from each are sampled for each batch. With a
constant learning rate of $1.0e{-4}$, the model is trained using the Adam
optimizer for 20 epochs.

Due to the lack of datasets with ground truth (GT) expressions of
$\boldsymbol{\beta}\in\mathbb{R}^{64}$, we use 5-fold cross-validation for the
evaluation. Particularly, $50k$ collections are split into 5 groups. Each
time, we train with four groups and evaluate the holdout group, whose GT
coefficients are obtained by overfitting it with the proposed method. We use
the LMD (see Eq.6) on lip landmarks as a metric. The group LMD is defined as
the mean of the collection LMDs, and each is defined as the mean of the data
LMDs. We form mixed coefficients in order to isolate the effects of certain
estimated attributes. For instance, the mixed coefficients are formed as
$\boldsymbol{\hat{x}}=\\{\boldsymbol{\alpha}_{gt},\boldsymbol{\delta}_{gt},\boldsymbol{\hat{\beta}},\boldsymbol{p}_{gt},\boldsymbol{\gamma}_{gt}\\}$
in order to evaluate the estimated expression.

We compare with Deep3D [11] and MGCNet [12]. The min, max, and average group
LMDs are reported in Tab.1, and results with GTs are also included as
references. For Deep3D and MGCNet, results with estimated expressions perform
worse than counterparts with GT expressions, and additionally performing
image-wise identity estimation can improve results. These results suggest that
they incorrectly ascribe some parts of face morphings to identity variations
(see Fig.3(b)(c)), and as a result, their estimated expressions when used
alone cannot adequately describe lip motions. Our results using the E-Net
estimated expressions, otherwise, are more in line with those using GTs,
showing that the E-Net can more consistently attribute facial morphings to
expressions.

Table 2: Lip-sync accuracies of different methods. Input Mode | Attributes | Train | Eval | Acc.
---|---|---|---|---
Raw data | none | dev | test | 94.1%
Raw data | none | minidev | test | 88.9%
RGB [11] |  $\boldsymbol{\hat{\alpha}}$, $\boldsymbol{\hat{\beta}}$ | minidev | test | 97.4%
RGBD [11] |  $\boldsymbol{\hat{\alpha}}$, $\boldsymbol{\hat{\beta}}$ | minidev | test | 98.1%
RGB(ours) | $\boldsymbol{\hat{\beta}}$ | minidev | test | 99.1%
RGBD(ours) | $\boldsymbol{\hat{\beta}}$ | minidev | test | 99.2%
Table 3: ASD performances of different methods. Method | AP | AUROC | EER
---|---|---|---
Self-supervised [2] | 0.924 | 0.962 | 0.083
RGBD [11] | 0.929 | 0.954 | 0.109
RGBD(ours) | 0.957 | 0.971 | 0.079

###  5.3 Results on Audio-Visual Tasks

We conduct experiments on lip sync and ASD tasks. We adopt the self-supervised
PerfectMatch [1] for lip sync, using synthesized images as the visual input
and Mel-spectrograms as the audio input. We first train a model on the dev-
split of VoxCeleb2 using the raw face tracks.Then, a variety of versions are
trained using the minidev split, including raw face tracks, synthesized images
with E-Net estimated expressions, and synthesized images with estimated
identities and expressions from Deep3D.

For evaluation, we follow [1] to extract audio and visual features from every
0.2s segment with a stride of 0.04s. Then, we calculate the cosine similarity
between each visual feature and each audio feature within a $\pm 15$ frame
window, and we determine the offset providing the min distance. If the
determined offset is within $\pm 1$ frame of the GT offset, it is correct.

The evaluation results are listed in Tab.2. With less data, models trained
with synthesized images outperform those trained with raw face tracks.
Further, models trained with RGBD images yield better results. This may be the
case because depth information makes 3D lip motions more distinct, see Fig.3.
Particularly, models trained with E-Net estimated expressions consistently
outperform others, suggesting that improved lip motion description may
directly benefit lip sync.

For the ASD task, we directly apply the most effective lip-sync models to the
ASW [2] test split. Following [2], an AV pair with a distance greater than a
threshold is identified as being active. In our case, the pipeline
preprocesses face tracks to produce synthesized images, and to increase
robustness, cosine similarities are averaged across 15 frames [1]. Results in
Tab.3 reveal that lip sync models exhibit great generalizability by matching
the strong baseline [2] in all metrics, including average precision (AP), the
area under the receiver operating characteristic (AUROC), and equal error rate
(EER).

##  6 CONCLUSION

This paper presents a DSP for lip sync. The pipeline provides considerably
more room for introducing domain knowledge and enables us to synthesize
standardized expressive images with consistently disentangled lip motion
information and reduced effects of compound distracting factors. Experimental
results suggest that existing lip sync methods improve their data efficiency
and generalizability when equipped with the DSP. In a broader sense, it may be
adopted to assist other tasks, e.g. lip reading, encountering similar
challenges, especially when datasets are limited.

## References

  * [1] Soo-Whan Chung, Joon Son Chung, and Hong-Goo Kang,  “Perfect match: Improved cross-modal embeddings for audio-visual synchronisation,”  in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 3965–3969.
  * [2] You Jin Kim, Hee-Soo Heo, Soyeon Choe, Soo-Whan Chung, Yoohwan Kwon, Bong-Jin Lee, Youngki Kwon, and Joon Son Chung,  “Look Who’s Talking: Active Speaker Detection in the Wild,”  in Proc. Interspeech 2021, 2021, pp. 3675–3679.
  * [3] Joon Son Son and Andrew Zisserman,  “Lip reading in profile,”  in Proceedings of the British Machine Vision Conference (BMVC), September 2017, pp. 155.1–155.11.
  * [4] Shiyang Cheng, Pingchuan Ma, Georgios Tzimiropoulos, Stavros Petridis, Adrian Bulat, Jie Shen, and Maja Pantic,  “Towards pose-invariant lip-reading,”  in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 4357–4361.
  * [5] Zhiqi Kang, Radu Horaud, and Mostafa Sadeghi,  “Robust face frontalization for visual speech recognition,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, October 2021, pp. 2485–2495.
  * [6] Alexandros Koumparoulis and Gerasimos Potamianos,  “Deep view2view mapping for view-invariant lipreading,”  in 2018 IEEE Spoken Language Technology Workshop (SLT), 2018, pp. 588–594.
  * [7] Feng-Ju Chang, Anh Tuan Tran, Tal Hassner, Iacopo Masi, Ram Nevatia, and Gerard Medioni,  “Expnet: Landmark-free, deep, 3d facial expressions,”  in 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), 2018, pp. 122–129.
  * [8] Mohammad Rami Koujan, Luma Alharbawee, Giorgos Giannakakis, Nicolas Pugeault, and Anastasios Roussos,  “Real-time facial expression recognition “in the wild” by disentangling 3d expression from identity,”  in 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), 2020, pp. 24–31.
  * [9] Bernhard Egger, William A. P. Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, Christian Theobalt, Volker Blanz, and Thomas Vetter,  “3d morphable face models—past, present, and future,”  ACM Trans. Graph., vol. 39, no. 5, jun 2020.
  * [10] Zenghao Chai, Haoxian Zhang, Jing Ren, Di Kang, Zhengzhuo Xu, Xuefei Zhe, Chun Yuan, and Linchao Bao,  “Realy: Rethinking the evaluation of 3d face reconstruction,”  in Proceedings of the European Conference on Computer Vision (ECCV), 2022.
  * [11] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong,  “Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
  * [12] Jiaxiang Shang, Tianwei Shen, Shiwei Li, Lei Zhou, Mingmin Zhen, Tian Fang, and Long Quan,  “Self-supervised monocular 3d face reconstruction by occlusion-aware multi-view geometry consistency,”  in Computer Vision – ECCV 2020, Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, Eds., Cham, 2020, pp. 53–70, Springer International Publishing.
  * [13] Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Niessner,  “Face2face: Real-time face capture and reenactment of rgb videos,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [14] Joseph Roth, Sourish Chaudhuri, Ondrej Klejch, Radhika Marvin, Andrew Gallagher, Liat Kaver, Sharadh Ramaswamy, Arkadiusz Stopczynski, Cordelia Schmid, Zhonghua Xi, and Caroline Pantofaru,  “Ava active speaker: An audio-visual dataset for active speaker detection,”  in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 4492–4496.
  * [15] Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter,  “A 3d face model for pose and illumination invariant face recognition,”  in 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance, 2009, pp. 296–301.
  * [16] Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou,  “Facewarehouse: A 3d facial expression database for visual computing,”  IEEE Transactions on Visualization and Computer Graphics, vol. 20, no. 3, pp. 413–425, 2014.
  * [17] Ravi Ramamoorthi and Pat Hanrahan,  “An efficient representation for irradiance environment maps,”  in Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, New York, NY, USA, 2001, SIGGRAPH ’01, p. 497–500, Association for Computing Machinery.
  * [18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [19] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman,  “Voxceleb2: Deep speaker recognition,”  Proc. Interspeech 2018, pp. 1086–1090, 2018.