# Universal construction of genuinely entangled subspaces of any size

Maciej Demianowicz  Institute of Physics and Applied Computer Science, Faculty
of Applied Physics and Mathematics, Gdańsk University of Technology,
Narutowicza 11/12, 80-233 Gdańsk, Poland<EMAIL_ADDRESS>
###### Abstract

We put forward a construction of genuinely entangled subspaces of any
permissible dimensionality for any number of parties and local dimensions. The
construction is based on the Chebotarev theorem on roots of unity.

##  I Introduction

Genuinely entangled subspaces (GESs) [1], i.e., subspaces composed only of
genuinely multiparty entangled (GME) quantum states, have recently attracted
considerable attention due to their theoretical and practical importance [2,
3, 4, 5]. One of the most important problem in the field is their efficient
constructions. In the recent years, there have been few proposal regarding
this issue [1, 6, 7, 8, 9]. Importantly, however, none of the constructions is
universal in a sense that it gives a practical recipe to construct GESs of
any, including the maximal, dimensionality in the general multipartite
scenario. We fill this gap in the present paper. With this aim we exploit the
Chebotarev theorem on roots of unity, or, equivalently, total nonsingularity
of the discrete Fourier transform (DFT) matrices of prime order. The presented
GESs are given as orthocomplements of subspaces spanned by nonorthogonal
unextendible product bases (UPBs), which are carefully constructed from DFT
matrices. The construction we put forward here is inspired by one from Ref.
[10], where the properties of these matrices were used to build minimal UPBs
in certain bipartite systems.

##  II Preliminaries

We consider Hilbert spaces ${\cal
H}=\mathbb{C}^{d}\otimes\cdots\otimes\mathbb{C}^{d}$. Subsystems are denoted
as $A_{0}A_{1}\dots A_{n-1}=:\bf{A}$ and pure states as
$|\psi\rangle_{X},|\varphi\rangle_{X},\cdots$, where the subscript corresponds
to parties holding them. Column vectors are written as $(x,y,\dots)$, i.e., we
omit the transposition.

We call an $n$–partite pure state $|\psi\rangle_{\bf{A}}$ fully product if it
can be written as
$|\psi\rangle_{\bf{A}}=|\varphi\rangle_{A_{0}}\otimes|\phi\rangle_{A_{1}}\otimes\cdots\otimes|\xi\rangle_{A_{n-1}}$.
Otherwise, it is said to be entangled. An entangled multipartite state
$|\psi\rangle_{\bf{A}}$, which cannot be written in a biproduct form, i.e.,
$|\psi\rangle_{\bf{A}}\neq|\varphi\rangle_{S}\otimes|\phi\rangle_{\bar{S}}$
for any bipartite cut (bipartition) $S|\bar{S}$, $S\cup\bar{S}=\bf{A}$, is
said to be genuinely multiparty entangled (GME).

The notions defined for single states transfer naturally to subspaces. A
subspace ${\cal S}\subset{\cal H}$ is then called a completely entangled
subspace (CES) iff all states belonging to it are entangled [11, 12, 13].
Further, if all states from ${\cal S}\subset{\cal H}$ are GME, then ${\cal S}$
is called a genuinely entangled subspace (GES) [1]. The maximal dimension of a
GES of ${\cal H}_{d^{n}}$ equals $(d^{n-1}-1)(d-1)$.

A common approach to the construction of entangled subspaces is to use
unextendible product bases (UPBs). A UPB is a set of fully product vectors
such that there is no other fully product vector orthogonal to all of them
[14, 10, 15, 16, 17, 18]. Depending on whether the vectors of a UPB are
mutually orthogonal or not, we speak of orthogonal or nonorthogonal UPBs,
respectively. Clearly, in both cases, by definition, the orthocomplement of a
subspace spanned be a UPB is a CES. A multipartite UPB with the additional
property that it is a UPB for any bipartition of the parties leads to a GES.
This innocent observation allowed us to build (large but not maximal) GESs as
orthcomplements of subspaces spanned by nonorthogonal UPBs (nUPBs) [1]. The
core of the construction was the following beautiful result about local
spanning [14, 10]:

######  Fact 1.

Given is a set of product vectors from $\mathbb{C}^{m}\otimes\mathbb{C}^{n}$:
$B=\\{|\varphi_{i}\rangle\otimes|\psi_{i}\rangle\\}_{i=1}^{k}$ with $k\geq
m+n-1$. If any $m$ vectors $|\varphi_{i}\rangle$ span $\mathbb{C}^{m}$ and any
$n$ vectors $|\psi_{i}\rangle$ span $\mathbb{C}^{n}$, then there does not
exist a product vector orthogonal to all the vectors from $B$.

Vectors $|\varphi_{i}\rangle$ and $|\psi_{i}\rangle$ with the property as in
the fact above are said to possess the spanning property. In the context of
GESs, Fact 1 must be used for any bipartition of $\bf{A}$. Such consideration
reveals that there must be at least $d^{n-1}+d-1$ states in the set.

Another crucial fact for the remainder of the present paper is the following
result, which is known as the Chebotarev theorem on roots of unity [19] (see,
e.g., [20] for a recent proof):

######  Fact 2.

Let $p$ be a prime and $\omega$ be a primitive $p$th root of unity,
$\omega=\mathrm{e}^{2\pi\mathrm{i}/p}$. Then, the minors of the matrix
$[\omega^{ij}]_{i,j=0}^{p-1}$ are non–vanishing. In other words, all square
submatrices, including the matrix itself, are non–singular.

Matrices from the fact for general $p$ (not necessarily a prime), up to a
constant factor, are known as the discrete Fourier transform (DFT) matrices
and we will refer to them as such in what follows. Phrasing Fact 2
differently, it holds that DFT matrices of prime order are totally
nonsingular.

##  III Construction

We now present our construction, which is based on nUPBs and Fact 1. In
general, the most difficult part in such approach is ensuring the spanning
property to hold for any bipartition. Here, we achieve this by constructing
bases for which this property will be satisfied automatically without the need
to check it.

###  III.1 Basic construction

Consider the following set of (unnormalized) fully product vectors from
$(\mathbb{C}^{d})^{\otimes n}$:

$\displaystyle|\Psi^{(i)}\rangle_{\bf{A}}$ $\displaystyle=$
$\displaystyle\bigotimes_{m=0}^{n-1}\left(\omega^{k_{m,0}^{(i)}},\omega^{k_{m,1}^{(i)}},\dots,\omega^{k_{m,d-1}^{(i)}}\right)_{A_{m}}$
$\displaystyle=$
$\displaystyle\bigotimes_{m=0}^{n-1}\left(\sum_{s_{m}=0}^{d-1}\omega^{k^{(i)}_{m,s_{m}}}|s_{m}\rangle\right)_{A_{m}},$
$\displaystyle\hskip 14.22636pti=0,1,2,\dots,K-1,\quad K\geq d^{n-1}+d-1,$

where $\omega=\mathrm{e}^{2\pi\mathrm{i}/p}$, and the numbers
$k^{(i)}_{m,s_{m}}$ and $p$ are to be determined.

Construct a matrix:

$M=\sum_{i=0}^{k-1}\left(|\Psi^{(i)}\rangle_{\bf{A}}\langle i|\right)^{T},$
(2)

that is a matrix with rows being the product vectors (III.1). Consider any
bipartition $S|\bar{S}$ of $\bf{A}$ and write
$|\Psi^{(i)}\rangle_{\bf{A}}=|\varphi^{(i)}\rangle_{S}\otimes|\phi^{(i)}\rangle_{\bar{S}}$.
Let

$M_{S}=\sum_{i=0}^{k-1}\left(|\varphi^{(i)}\rangle_{S}\langle
i|\right)^{T},\quad
M_{\bar{S}}=\sum_{i=0}^{k-1}\left(|\phi^{(i)}\rangle_{\bar{S}}\langle
i|\right)^{T}.$ (3)

It is easy to see that both $M_{S}$ and $M_{\bar{S}}$ are submatrices of $M$.

We will now show that the numbers $k_{m,s_{m}}^{(i)}$ and $p$ can be chosen in
such a way that $M$ is a submatrix of a DFT matrix with prime $p$. With such a
choice, due to Fact 2, we have that (i) $M$ is full rank and, in consequence,
vectors $|\Psi^{(i)}\rangle_{\bf{A}}$ are linearly independent, and (ii) any
square submatrices of $M_{S}$ and $M_{\bar{S}}$, for any $S|\bar{S}$, are also
full rank guaranteeing the spanning property to hold for any bipartition. In
turn, by Fact 1, the orthocomplement of the span of
$|\Psi^{(i)}\rangle_{\bf{A}}$ is a GES.

By direct inspection, we find that the elements of the matrix $M$ are given as

$M_{ij}=\displaystyle\omega^{\sum_{m=0}^{n-1}k_{m,s_{m}}^{(i)}},\quad
j=\sum_{m=0}^{n-1}s_{m}d^{n-1-m},$ (4)

$i=0,1,\dots,K-1$, $s_{m}=0,1,\dots,d-1$. This means that with the choice

$k_{m,s_{m}}^{(i)}=is_{m}d^{n-1-m},\quad\textrm{prime}\;\;p\geq d^{n}$ (5)

we achieve the goal we have set for ourselves. The condition on $p$ follows
from the requirement that $M$ must fit in a DFT matrix.

We have thus proved the main result of the present communication:

Main result. The orthocomplement of the span of vectors (III.1) with $k$’s and
$p$ given by (5) is a GES of dimension $d^{n}-K$.

In the above we have concentrated on, the most natural from the practical
point of view, case of equal local dimensions, however, it is easy to realize
that the result generalizes to arbitrary local dimensions.

###  III.2 Extension

Due to the properties of the determinant, one realizes that a DFT matrix of
prime order $p$ remains totally nonsingular if its columns are multiplied by
non–zero constants. We can use this fact and consider more general nUPBs than
those given by (III.1). More precisely, one can use the following vectors in
the construction:

$\displaystyle|\tilde{\Psi}^{(i)}\rangle_{\bf{A}}$ $\displaystyle=$
$\displaystyle\bigotimes_{m=0}^{n-1}\left(\sum_{s_{m}=0}^{d-1}h_{m,s_{m}}\omega^{k^{(i)}_{m,s_{m}}}|s_{m}\rangle\right)_{A_{m}}.$
(6)

with $h_{m,s_{m}}\neq 0$; one of the $h$’s on each party can be set to one due
to normalization.

Moreover, the choice of $k$’s given by (5) is clearly not the only one
possible to have the desired property of $M$ and we have a lot of further
freedom in choosing these parameters along with the value of $p$.

We thus see that the construction aside from its universality offers quite a
lot of generality.

###  III.3 Example: three–qubit case

In the three qubit case, for the basic construction we obtain

$\displaystyle|\Psi^{(0)}\rangle_{A_{0}A_{1}A_{2}}=(1,1)_{A_{0}}\otimes(1,1)_{A_{1}}\otimes(1,1)_{A_{2}},$
$\displaystyle|\Psi^{(1)}\rangle_{A_{0}A_{1}A_{2}}=(1,\omega^{4})_{A_{0}}\otimes(1,\omega^{2})_{A_{1}}\otimes(1,\omega^{1})_{A_{2}},$
$\displaystyle|\Psi^{(2)}\rangle_{A_{0}A_{1}A_{2}}=(1,\omega^{8})_{A_{0}}\otimes(1,\omega^{4})_{A_{1}}\otimes(1,\omega^{2})_{A_{2}},$
$\displaystyle|\Psi^{(3)}\rangle_{A_{0}A_{1}A_{2}}=(1,\omega^{12})_{A_{0}}\otimes(1,\omega^{6})_{A_{1}}\otimes(1,\omega^{3})_{A_{2}},$
$\displaystyle|\Psi^{(4)}\rangle_{A_{0}A_{1}A_{2}}=(1,\omega^{16})_{A_{0}}\otimes(1,\omega^{8})_{A_{1}}\otimes(1,\omega^{4})_{A_{2}},$

where we can choose $\omega^{2\pi\mathrm{i}/11}$.

##  IV Conclusions and outlook

We have shown how to construct genuinely entangled subspaces (GESs) of any
permissible size in any multipatite setting. The construction is based on
nonorthogonal unextendible product bases (nUPBs), which are built upon the
Chebotarev theorem on roots of unity giving a certain characterization of the
discrete Fourier transform (DFT) matrices.

The use of nUPBs instead of orthogonal UPBs is necessary to have a
construction working universally in any multiparty scenario. If we dropped
this assumption, we would not be able to take into account the case of qubit
subsystems since, as it is well known, orthogonal UPBs do not exist in
$2\otimes m$ systems. Moreover, UPBs with orthogonal members are known not to
exist in some cases of $d$ and $n$, which again excludes their use in a
universal approach. However, it is still a big open problem whether in some
dimensions one could use orthogonal UPBs to construct GESs. Their existence
would have important consequences. It is not clear to us whether one could
attack this problem using DFT matrices, or more generally, Butson–type
Hadamard matrices, in the spirit of the quadratic residues construction of
Ref. [10]. Anyhow, it seems that the use of number theory might be a promising
path to resolve the question of the existence of GESs orthogonal to UPBs. On a
less fundamental level, there also remains open the problem of constructing
GESs of arbitrary sizes with explicit orthogonal basis.

## References

  * [1] M. Demianowicz and R. Augusiak, Phys. Rev. A 98, 012312 (2018).
  * [2] S. Agrawal, S. Halder, M. Banik, Phys. Rev. A 99, 032335 (2019).
  * [3] K. Wang, L. Chen, L. Zhao, Y. Guo, Quantum Inf. Process. 18, 202 (2019).
  * [4] F. Baccari, R. Augusiak, I. Šupić, and A. Acín, Phys. Rev. Lett. 125, 260507 (2020).
  * [5] M. Demianowicz, G. Rajchel–Mieldzioć, and R. Augusiak, New J. Phys. 23, 103016 (2021).
  * [6] M. Demianowicz and R. Augusiak, Phys. Rev. A 100, 062318 (2019).
  * [7] M. Demianowicz and R. Augusiak, Quant. Inf. Proc. 19, 199 (2020).
  * [8] O. Makuta and R. Augusiak, New J. Phys. 23, 043042 (2020).
  * [9] K. Antipin, J. Phys. A: Math. Theor. in press, doi.org/10.1088/1751-8121/ac37e5
  * [10] D. P. DiVincenzo, T. Mor, P. W. Shor, J. A. Smolin, B. M. Terhal, Comm. Math. Phys. 238, 379 (2003).
  * [11] K. Parthasarathy, Proceedings Mathematical Sciences 114, 365 (2004).
  * [12] B. V. R. Bhat, Int. J. Quantum Inf. 4, 325 (2006).
  * [13] J. Walgate and A. J. Scott, J. Phys. A 41, 375305 (2008).
  * [14] C. H. Bennett, D. P. DiVincenzo, T. Mor, P. W. Shor, J. A. Smolin, and B. M. Terhal, Phys. Rev. Lett. 82, 5385 (1999).
  * [15] N. Alon and L. Lovasz, Journal of Combinatorial Theory A, Series A 95, 169 (2001).
  * [16] N. Johnston, J. Phys. A: Math. Theor. 47, 424034 (2014).
  * [17] A. O. Pittenger, Linear Alg. Appl. 359, 235 (2003).
  * [18] Ł. Skowronek, J. Math. Phys. 52, 122202 (2011).
  * [19] N. G. Chebotarev, Uspekhi Mat. Nauk, 3(4), 3 (1948).
  * [20] T. Tao, Math. Res. Lett. 12, 121 (2005).


# Towards Accurate Lip-to-Speech Synthesis in-the-Wild

Sindhu Hegde<EMAIL_ADDRESS>VGG, University of OxfordUnited Kingdom ,
Rudrabha Mukhopadhyay<EMAIL_ADDRESS>CVIT, IIIT
HyderabadIndia ,  C.V Jawahar<EMAIL_ADDRESS>CVIT, IIIT HyderabadIndia
and  Vinay Namboodiri<EMAIL_ADDRESS>University of BathUnited Kingdom

(2018; 2023)

###### Abstract.

In this paper, we introduce a novel approach to address the task of
synthesizing speech from silent videos of any in-the-wild speaker solely based
on lip movements. The traditional approach of directly generating speech from
lip videos faces the challenge of not being able to learn a robust language
model from speech alone, resulting in unsatisfactory outcomes. To overcome
this issue, we propose incorporating noisy text supervision using a state-of-
the-art lip-to-text network that instills language information into our model.
The noisy text is generated using a pre-trained lip-to-text model, enabling
our approach to work without text annotations during inference. We design a
visual text-to-speech network that utilizes the visual stream to generate
accurate speech, which is in-sync with the silent input video. We perform
extensive experiments and ablation studies, demonstrating our approach’s
superiority over the current state-of-the-art methods on various benchmark
datasets. Further, we demonstrate an essential practical application of our
method in assistive technology by generating speech for an ALS patient who has
lost the voice but can make mouth movements. Our demo video, code, and
additional details can be found at
http://cvit.iiit.ac.in/research/projects/cvit-projects/ms-l2s-itw.

Lip-reading, Lip-to-Speech, Assistive Technology, Speech Generation

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: ACM Multimedia; Oct 28–Nov 05, 2023; Ottawa,
Canada††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††submissionid:
409††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
31st ACM International Conference on Multimedia; October 29-November 3, 2023;
Ottawa, ON, Canada††booktitle: Proceedings of the 31st ACM International
Conference on Multimedia (MM ’23), October 29-November 3, 2023, Ottawa, ON,
Canada††price: 15.00††doi: 10.1145/3581783.3611787††isbn:
979-8-4007-0108-5/23/10††ccs: Computing methodologies Neural networks††ccs:
Computing methodologies Computer vision tasks

Figure 1. We propose a novel approach for multi-speaker lip-to-speech
synthesis in the wild. Prior works try to learn a language model directly from
raw speech, which only provides weak supervision due to the presence of other
acoustic variations such as voice, accents, and prosody. We solve this problem
by relying on recent advancements in lip-to-text generation. We condition on
the noisy text outputs and lip video to generate natural speech with clearly
pronounced words.

##  1\. Introduction

As multi-sensory beings, humans have the ability to experience the world
through various senses, adapting to make the most of what we perceive. When
one sense is diminished, like hearing, the brain compensates by relying more
on other senses, like vision, to maintain independence and navigate the
environment. One of the most prevalent ways for deaf individuals to comprehend
speech is through lip reading - the process of understanding spoken content
purely from silent lip movements. Therefore, lip reading has numerous
applications, ranging from searching through old silent films to being used as
an assistive technology for people who are unable to speak. For example,
people with Amyotrophic Lateral Sclerosis (ALS) often lose their ability to
speak due to issues with their vocal cords but can mouth words. Therefore,
developing automated lip reading technologies can improve the communication
abilities of people with ALS. In the past few years, previous works have shown
that it is far easier for deep learning models to perform lip reading than it
is for humans (Chung et al., 2017). There has been a growing interest in the
research community (Afouras et al., 2018a, b; Chung and Zisserman, 2016a;
Chung et al., 2017) to solve the task of transcribing silent lip movements
into text. While giant strides are being made to solve lip-to-text generation,
only a handful of works go a step further to generate speech from silent lip
movements. The widening gap between these two closely related tasks is
becoming increasingly evident. In other words, while lip-to-text models are
reaching word error rates as low as $17\%$ WER (Serdyuk et al., 2022), there
are no practically usable lip-to-speech systems that generate natural,
meaningful speech for in-the-wild identities.

##### Applications for lip-to-speech synthesis

Lip-to-Speech generation is not only more exciting and challenging but also
has more far-reaching applications. For instance, having a live video call
with a person who has lost their voice is more engaging than reading the text
transcription of their silent lip movements. Speech can convey more: it is
possible to express emotions through silent lip movements if it is converted
to speech rather than text. Speech is also more instantaneous for the
listener: it is easier and faster to hear than to read text. The immersive
user experience is a significant impetus for developing algorithms that
translates lip movements directly to speech. Lip-to-Speech can also be applied
in forensic investigations, where silent footage can be analyzed to determine
what was being said or generate speech from archival film footage. Our work
showcases the practical impact of lip-to-speech generation by generating
speech for the silent lip movements of an ALS patient, a feat not previously
demonstrated by other models in this field. We also show results on reading
mouth movements of people suffering from hearing loss to demonstrate the
effectiveness of accurate lip-to-speech synthesis further. More details about
the applications can be found in Section 4.3.

##### The challenges in lip-to-speech synthesis

However, synthesizing speech from silent talking-face videos is far more
challenging than lip-to-text generation. Not only does the model first needs
to distill the content from the lip movements, but it also needs to generate
the final speech by accurately modeling the speaker attributes like voice,
accent, and style. Owing to these difficulties, until recently, most of the
existing works (Ephrat and Peleg, 2017; Vougioukas et al., 2019; Prajwal et
al., 2020a) focused on building either single-speaker models or constrained
multi-speaker models with limited corpus size as well as limited vocabulary.
While more recent efforts like (Prajwal et al., 2020a) have extended lip-to-
speech to “in-the-wild” environments, most of these models are speaker-
specific in nature, i.e., they only work on speakers they are trained on. The
speaker-independent models (Hegde et al., 2022; Kim et al., 2022; Mira et al.,
2022; Kim et al., 2023) suffer from numerous weaknesses and fail to accurately
learn language and speech attributes like voice, prosody, etc. Due to the
sheer difficulty of the task, they turn out to be well below expectations for
an end-user application.

###  1.1. Our Contributions

First, we would like to clearly state that our goal is to be able to generate
speech for a silent lip video of any speaker in the wild and in any desired
target voice. The fundamental premise of this paper is that current models
trying to solve this task struggle to learn language and speech attributes
because they try to learn both of these solely from speech supervision. They
inadvertently disregard the advancements that have been made in the sibling
task of the lip-to-text generation. We argue that learning lip-to-speech
without any understanding/supervision of text is very challenging to achieve,
which can be observed in all the current models, where they fail to produce
clear and intelligible outputs in unconstrained settings. Our key idea is to
use the fact that lip-to-text models already have learned the task of
extracting content from lips, as they are trained with text supervision. We
show that we can achieve accurate, natural, and high-quality speech outputs by
using the silent video input and the noisy text transcriptions from a pre-
trained lip-to-text model. We train a visual text-to-speech model to
synthesize speech that syncs with the input video. Our approach outperforms
the existing multi-speaker lip-to-speech models by a significant margin in
terms of both qualitative and quantitative evaluations. We highlight our key
contributions below.

  1. (1)

Lip-to-Speech generation is challenging with only speech supervision, and
current models struggle to learn language and speech attributes. To address
this, we propose using a pre-trained lip-to-text model’s output to aid in lip-
to-speech generation.

  2. (2)

Our approach involves training a visual text-to-speech model to synthesize
speech that syncs with the input video. Using this approach, we outperform
existing multi-speaker lip-to-speech models in qualitative and quantitative
evaluations.

  3. (3)

Our proposed approach enables accurate and natural speech output for silent
lip videos of any speaker in any desired target voice.

##  2\. Related Works

##### Lip-to-Speech

Generating speech directly from lip movements has always been a challenge for
researchers due to the ill-posed nature of the task. While recognizing content
from lip movements itself is challenging due to the one-to-many relation
present between visemes and phonemes, lip-to-speech synthesis also needs to
model speech-related variations like voice, accents, and prosody. Therefore,
it was initially attempted in laboratory setups (Cooke et al., 2006) using
datasets with a minimal vocabulary. The initial works to (Ephrat et al., 2017;
Ephrat and Peleg, 2017; Vougioukas et al., 2019; Akbari et al., 2017) used
fully convolutional neural networks to learn a mapping between lip movements
and speech. These models were also trained on speaker-specific data and did
not work for unseen speakers. Moreover, such models were incapable of handling
“in-the-wild” settings. To mitigate this issue, a sequence-to-sequence model,
Lip2Wav (Prajwal et al., 2020a), was proposed that was trained on large
amounts of speaker-specific data taken from in-the-wild YouTube videos.
Lip2Wav worked reasonably well on speakers seen during training but did not
extend to handling unseen speakers. Several more recent studies (Hegde et al.,
2022; Kim et al., 2022, 2023; Mira et al., 2022) have attempted to develop
end-to-end lip-to-speech synthesis models on large datasets containing data
from hundreds of speakers. For instance, in (Hegde et al., 2022), authors
proposed a variational approach that matches the distributions of lip
movements and speech segments to project them into a shared space, which
allows for handling the high variations of in-the-wild speakers to some
extent. Meanwhile, both (Kim et al., 2022; Mira et al., 2022) utilized a
transformer-based approach to convert lip-to-speech synthesis into a sequence-
to-sequence problem, where a sequence of lip movements is translated into a
sequence of speech tokens. Additionally, (Kim et al., 2023) treated lip-to-
speech synthesis as a multi-tasking problem, where the model predicts text
transcripts in addition to speech forcing the model to learn better content
information. However, all of these models produce sub-par outputs with words
that are barely uttered and often unintelligible with unnatural voice and
prosody. We argue that these works are limited because they do not use the
recent advancements in the sibling task, lip-to-text, which we discuss below.

##### Lip-to-Text

Lip reading has become synonymous with lip-to-text, and several advancements
have been made to achieve highly accurate text transcriptions of silent lip
videos. The very first works (Assael et al., 2016) used datasets (Cooke et
al., 2006; Harte and Gillen, 2015) collected in a laboratory setup and trained
simple RNN-based architectures. One of the initial approaches (Chung and
Zisserman, 2016a) handling the “in-the-wild” setup posed the problem in a
word-level classification setting and continued using RNN-based models.
Subsequent approaches (Chung et al., 2017) have attempted to tackle this task
by posing it as a sequence-to-sequence problem. The goal here is to translate
a sequence of lip movements into a sequence of characters. Approaches like
(Afouras et al., 2018a) compared multiple sequence-to-sequence architectures,
including transformers, to achieve favorable word error rates (WER). A more
recent work (Prajwal et al., 2022) uses a variant of transformer and specific
attention to the mouth region through visual transformer pooling and achieves
state-of-the-art results in lip reading. Other works such as AV-HuBERT (Shi et
al., 2022) have also improved the state-of-the-art for in-the-wild lip
reading, paving the way for practical applications. Our work demonstrates the
importance of pre-trained lip-to-text models in achieving precise lip-to-
speech synthesis in uncontrolled environments.

##### Text-to-speech

Our proposed approach uses text as an intermediate output and converts it into
speech. Text-to-Speech has long interested the speech community and has
historically seen a lot of progress. Modern deep learning architectures (Shen
et al., 2018; Ping et al., 2018; Ren et al., 2020) enable several industrial
and multimedia applications. They are accurate in terms of uttered content and
have natural speaking styles, prosody, and voice. The efforts slowly shifted
towards multi-speaker scenario, which is far more challenging due to the sheer
variation in prosody, voice, and other speaker-related attributes like pitch
and tone. Most of these methods are provided with a separate voice token (Jia
et al., 2018) containing identity information about the target speaker. The
voice tokens are generated from a short speech segment of a target speaker and
often only capture the voice information while failing to capture the prosody.
Thus, there has also been a push for using additional information like lip
movements (Hassid et al., 2022; Hu et al., 2021) to help TTS models generate
higher-quality outputs. Since lip-to-speech synthesis requires us to generate
accurate prosody and speaking style, our approach is inspired by this line of
work. We design a visual text-to-speech model that works uses both text and
visual features from state-of-the-art lip reading network (Prajwal et al.,
2020a) to generate natural, accurate speech. Our Visual TTS is integral to
generating speech that perfectly syncs with the input silent lip video.

##  3\. Lip-to-Speech Synthesis in the Wild

Figure 2. Overview of our approach. We first extract the visual features and
the text predictions from a pre-trained lip-to-text network. Using a visual
text-to-speech (TTS) model, we can generate speech outputs that sync with the
silent video input. The visual TTS encodes the visual and textual (in the form
of phonemes) inputs and aligns them in time using the scaled dot-product
attention. For each query video time-step, we retrieve the phoneme to utter at
that time using this attention mechanism. After adding the speaker identity
embedding, these are then upsampled and decoded into melspectrograms. The
melspectrograms are converted into natural waveforms using a pre-trained
vocoder.

We start the discussion by highlighting the key issues and challenges in the
existing lip-to-speech works, leading to their poor performance in
unconstrained settings. We then present our framework with a detailed
description of the modules involved.

###  3.1. Issues and challenges in existing works

####  3.1.1. Learning language from speech

As discussed previously, all the current lip-to-speech works directly generate
speech from the lips. It is known that learning a language model is crucial
for reading the lips accurately. However, the current multi-speaker lip-to-
speech models are sub-par because they try to learn a language model in the
speech modality, which contains a large diversity of speaker identities,
styles, accents, and prosody. Thus, we argue that we need some other way of
incorporating language knowledge.

####  3.1.2. The missing block of lip-to-speech: lip-to-text

Lip-to-Speech synthesis models have two tasks: (i) inferring the content from
lips and (ii) inferring the style in which that content is spoken. If we have
the content that is being spoken, then our task is now reduced to just
generating the speech that matches the silent lip video. This is the premise
of our paper. But how do we get this text information, especially when we only
have a silent lip video as input? We show that we can get this text
information from pre-trained lip-to-text models: a class of models closely
related to our task at hand but have been mainly ignored in previous works on
lip-to-speech synthesis. We design an approach that can build upon the current
works (Afouras et al., 2018a; Prajwal et al., 2022) in lip reading, i.e., pre-
trained lip-to-text models, and generate far more accurate speech outputs.

####  3.1.3. Achieving accurate lip-sync

Now that we have the text, the next step is to generate the “right” kind of
speech output. That is, the generated speech must match the input lip
sequence. There are many ways to utter the same sentence, but only one will
match the input lip video. Thus, it is worth noting that a trivial text-to-
speech will not serve our task. Instead, we need a text-to-speech model that
is also conditioned on video input.

###  3.2. Our Approach

The task of lip-to-speech synthesis is formulated as follows: given a silent
video frame sequence of a person talking $S_{v}=\\{I_{1},I_{2},...,I_{p}\\}$,
the goal is to generate the corresponding speech melspectrogram sequence,
$S_{m}=\\{M_{1},M_{2},...,M_{q}\\}$. An overview of our proposed two-stage
framework is depicted in Figure 2. We use a state-of-the-art lip-to-text model
(Prajwal et al., 2022) to obtain the lip features and noisy text
transcriptions for the given silent lip video $S_{v}$. We design a visual
text-to-speech model conditioned on (i) the noisy text and (ii) the lip
features to produce high-quality speech outputs $S_{m}$ that are in sync with
the input silent lip video. This solves the task of lip-to-speech synthesis.

####  3.2.1. Pre-trained Lip-to-Text

Lip-to-Text networks ingest the silent talking face videos as input and
transcribe what is being uttered. The lip-to-text models (Afouras et al.,
2018c; Chung and Zisserman, 2016a; Chung et al., 2017; Afouras et al., 2018b,
a; Prajwal et al., 2022) are now capable of generating text transcripts with
word error rates as low as $20-30\%$. One such recent work trained on public
datasets is VTP (Prajwal et al., 2022). Specifically, this work stands out due
to its two unique aspects: (i) it has been shown to be data efficient, (ii) it
contains a strong visual backbone that can attend to and extract accurate lip
features. These visual features have been shown to work well in several other
tasks such as visual keyword spotting (R et al., 2021) and even spotting mouth
movements in sign language (Momeni et al., 2022). The sub-word units
considered to learn the text representations help to model the ambiguities of
the task significantly better than using characters. Also, the sub-word tokens
are semantically meaningful and provide a language prior, thereby resulting in
a significant performance boost. In addition, the visual representations
learned using the backbone network track and aggregate the spatio-temporal lip
movement features, mainly due to the strong attention-based pooling mechanism.
We thus propose to adopt this model as our first module to generate text and
visual features from silent lip videos.

##### Overview of VTP (Prajwal et al., 2022):

The model inputs a sequence of $5$ consecutive frames,
$S_{v}\in\mathbb{R}^{T\times H\times W\times 3}$ ($T=5$) and extracts low-
level features using a spatio-temporal residual CNN block. These individual
frame-wise features are then processed using the visual transformer pooling
block, which consists of a series of transformer layers. The output from this
block, $z_{t}$ is a self-attended feature map, which is used along with a
learnable query vector, $Q_{att}$ to obtain the spatially weighted average of
$z_{t}$. These compact per-frame visual representations are stacked along the
temporal dimension to obtain a temporal embedding sequence,
$g\in\mathbb{R}^{T\times fd}$, where $fd$ is the transformer feature
dimension. Finally, the learned visual representations are passed to the
transformer encoder-decoder network to predict the text outputs in the form of
sub-word tokens in an auto-regressive fashion. The model uses beam search
decoding (Freitag and Al-Onaizan, 2017) and language model rescoring (Chan et
al., 2016) strategies to obtain the final sentence outputs $S_{t}$. We refer
the reader to (Prajwal et al., 2022) for more details on the architecture and
training strategies.

##### Adopting VTP for our task:

For our task at hand, we use the pre-trained VTP network to obtain: (i) text
predictions - the final decoded output of the model $S_{t}$, and (ii) per-
frame visual representations - $g$. The text predictions directly act as input
to our speech generation module. The visual representations serve as a
condition for speech generation, which is crucial to obtain speech that is in-
sync with the silent input video.

####  3.2.2. Visual Text-to-Speech

Once we have the accurate text predictions, the next step is to generate the
corresponding speech sequence $S_{m}$, which is in-sync with the input video
clip $S_{v}$. As explained in Section 3.1.3, directly using state-of-the-art
TTS models to synthesize speech from text inputs will generate out-of-sync
speech that does not match the input video. We thus design a TTS network by
conditioning the model on the input video features. Our visual TTS network
majorly comprises five components: (i) Text Encoder, (ii) Visual Encoder,
(iii) Visual-Text Attention, (iv) Speaker Embedding, and (v) Spectrogram
Decoder. We delve into each of these components below.

##### Text Encoder:

As followed in most of the TTS networks (Ren et al., 2020), we extract phoneme
representations from text input, which is then given as input to the
Transformer encoder layers. Our text encoder block is similar to the one used
in FastSpeech2 (Ren et al., 2020), which consists of a positional encoding
layer and Feed-Forward Transformer (FFT) layers. The phonemes are transformed
to encode the semantic representation and output the text embedding vectors,
$E_{text}$ of dimension: $N\times d$, where $d$ is the transformer feature
dimension.

##### Visual Encoder:

The input to our visual encoder is the visual feature sequence $g$ obtained
from the lip-to-text network. We highlight that these visual features, which
capture the lip shape and motion, play a crucial role in generating speech
that syncs with the input video. Since these representations were also learned
using text supervision, they are likely to reflect accurate content
information. This starkly contrasts previous works that directly learn visual
representations from speech supervision only, which can lead to sub-par visual
representations that might contain other unnecessary information, such as the
input face identity. The superiority of these representations is one of the
critical reasons for our overall network’s performance. The extracted $N\times
T\times fd$ dimensional representations are given as input to the Transformer
encoder layers as shown in Figure 2. Similar to the text encoder network, the
visual encoder consists of a positional encoding followed by a series of FFT
blocks. The encoder network outputs the learned visual embeddings, $E_{vis}$
of dimension: $N\times T\times d$.

##### Visual-Text Attention:

Once we obtain the text and the visual embeddings, the next and the most
important step is to find the alignment between these embeddings in time:
which phoneme must be uttered when? The generated speech must take the content
from the text embeddings and simultaneously, it should also temporally align
(sync) it with the video frames. In order to achieve this, we employ a scaled-
dot product attention (Vaswani et al., 2017) mechanism to learn the
correspondence between text and video frames. Specifically, the visual
embeddings $E_{vis}$ act as query, and the text embeddings $E_{text}$ act as
keys and values.

(1) $Attention(Q,K,V)=Scaled-Dot\>Product\\\
Attention(E_{vis},E_{text},E_{text})\in\mathbb{R}^{T\times d}$

Through this attention module, the network learns the video-text temporal
alignment, which synchronizes the generated speech with the input video
frames. Now between the video sequence and melspectrograms, it is known that
there exists a natural temporal alignment. The length of the melspectrograms
is a constant $n$ times the length of the video. Thus, the attention output
$A$ is up-sampled $n$ times to directly obtain the melspectrogram duration.
This eliminates the need to train a separate duration predictor as done in
FastSpeech2 (Ren et al., 2020). In other words, the text-to-video alignment
network already determines the duration of each phoneme in the speech output.

##### Speaker Embedding:

Unlike single-speaker models, we want our model to generate speech for any
arbitrary speaker in the wild. Thus, our model also needs the voice input of
the target speaker to generate the speech in his/her voice. We consider a
random one-second audio segment of the target speaker and extract the speaker
embedding vector $E_{voice}$ using the pre-trained identity
network111github.com/CorentinJ/Real-Time-Voice-Cloning.

##### Spectrogram Decoder:

The speaker embedding vector $E_{voice}$ is added to the upsampled attention
output $A$ to obtain voice-aware content representation. The spectrogram
decoder, consisting of transformer decoder layers, ingests this representation
and generates the melspectrogram sequence $S_{m}$. To further improve speech
quality, as done in most of the TTS networks, we adopt a pre-trained neural
vocoder model BigVGAN (gil Lee et al., 2022), to synthesize the speech from
the melspectrogram output. Note that this step is only used during inference
to obtain high-quality speech outputs.

###  3.3. Datasets and Training Settings

####  3.3.1. Datasets

We evaluate both constrained and unconstrained datasets to analyze the model’s
performance. The first corpus we experiment with is the TCD-TIMIT (Harte and
Gillen, 2015) lip speaker dataset, which comprises lab-recorded videos of $3$
speakers. Next, we consider the word-level LRW (Chung and Zisserman, 2016a)
dataset, consisting of around $150$ hours of single-word utterances from
hundreds of speakers. We then move on to the more challenging large-scale
datasets: LRS2 (Chung et al., 2017) and LRS3 (Afouras et al., 2018b). The LRS2
data comprises thousands of speakers from BBC programs with a vocabulary of
$59k$ and around $230$ hours of video clips (both “train” and “pre-train” sets
together). The LRS3 dataset, on the other hand, is also a large-scale dataset
with a total of approximately $430$ hours (“train” and “pre-train” sets) of
video data with $150k$ utterances. It consists of thousands of spoken
sentences from TED and TEDx talks in English. We train and test the
performance of our network using the official splits of LRW, LRS2 and LRS3
datasets and use the train-test split proposed in Lip2Wav (Prajwal et al.,
2020a) for TIMIT dataset (Harte and Gillen, 2015).

####  3.3.2. Data pre-processing

We sample the video frames at $25$ FPS and follow the pre-processing procedure
of VTP (Prajwal et al., 2022) to obtain the face crops. For the speech
segments, we compute STFT and then melspectrograms of $80$ mel-bands, with a
hop length of $10ms$ and a window length of $25$ms, sampled at $16$kHz. We use
an open-source grapheme-to-phoneme tool for text processing to obtain the
phoneme inputs for our Visual TTS model.

####  3.3.3. Model configuration and training

Our Visual TTS model comprises $4$ FFT blocks in the text and visual encoders
and $6$ FFT blocks in the spectrogram decoder network. The visual embeddings
obtained from VTP are $512$-dimensional embeddings for each frame. In the
video-text attention sub-network, the upsample factor, $n$ is set to $4$. For
the speaker embedding, the identity network outputs a $256$ dimensional vector
for each speech sample. For the lip-to-text network, we use the publicly
released pre-trained model222https://github.com/prajwalkr/vtp (trained on LRS2
(Chung et al., 2017) and LRS3 (Afouras et al., 2018b)). Our visual TTS model
is trained on a single NVIDIA $2080$ Ti GPU. We use the Adam optimizer (Kingma
and Ba, 2015) with $\beta_{1}=0.9$, $\beta_{2}=0.98$, and $\epsilon=10^{-9}$
and follow the same learning rate schedule as done in (Prajwal et al., 2022).
We set the batch size to be $16$ for all the datasets and train the model
using $L_{1}$ reconstruction loss for approximately $900k$ steps (until
convergence). We also train BigVGAN vocoder (gil Lee et al., 2022) and use it
during inference to generate the speech from the output melspectrograms.

##  4\. Experiments

We present our approach’s quantitative results and comparisons with the
existing methods. As automatic speech metrics are imperfect, we also show MOS
scores using human evaluation. Finally, we demonstrate a real-world
application of lip-to-speech for the first time by voicing the silent lip
movements of an ALS patient.

###  4.1. Quantitative Evaluations

Metrics: We measure the quality of the generated speech using the standard
speech metrics: Perceptual Evaluation of Speech Quality (PESQ), Short-Time
Objective Intelligibility measure (STOI), and its extended version (ESTOI).
PESQ measures the clarity and overall perceptual quality of speech and STOI
and ESTOI measure the intelligibility of speech. Further, as discussed
previously, it is very crucial to generate speech that is in sync with the
input video. We use the lip-sync metrics, Lip-Sync Error - Confidence (LSE-C)
and Lip-Sync Error - Distance (LSE-D) (Chung and Zisserman, 2016b) to evaluate
whether the output speech matches the input lip movements. We use the public
implementations of all the above metrics for a fair comparison.

####  4.1.1. Speech Synthesis in Constrained Settings

Comparisons: To evaluate lip-to-speech methods on the constrained single-
speaker TCD-TIMIT dataset, we compare four existing approaches: (i) GAN-based
(Vougioukas et al., 2019), (ii) Lip2Wav (Prajwal et al., 2020a), (iii) VAE-GAN
(Hegde et al., 2022), and (iv) VCA-GAN (Kim et al., 2022). We adopt the same
settings as Lip2Wav (Prajwal et al., 2020a) and report the scores (Prajwal et
al., 2020a) and VCA-GAN (Kim et al., 2022). We use the Wav2Lip (Prajwal et
al., 2020b) repository to compute the LSE-C and LSE-D scores for each method.
We have excluded the metrics for a particular model that were not mentioned in
the original papers or for which no publicly available pre-trained checkpoint
exists.

Results: Table 1 contains the results on the TCD-TIMIT dataset. We observe
that our approach achieves comparable results to previous methods in
constrained settings with minimal data of only $3$ speakers. However, the
significant benefits of our approach can be seen in unconstrained settings,
which we describe below.

Table 1. We compare against state-of-the-art methods on several standard
multi-speaker benchmarks using standard metrics. The generated outputs from
our model are most natural (PESQ), most accurate (STOI, ESTOI) and in perfect
sync with the video input (LSE-C, LSE-D) in the in-the-wild videos of LRW
(Chung and Zisserman, 2016a), LRS2 (Chung et al., 2017) and LRS3 (Afouras et
al., 2018b).

Dataset | Method | PESQ$\uparrow$ | STOI$\uparrow$ | ESTOI$\uparrow$ | LSE-C$\uparrow$ | LSE-D$\downarrow$
---|---|---|---|---|---|---
TCD-TIMIT (Harte and Gillen, 2015) |  GAN-based (Vougioukas et al., 2019) | 1.22 | 0.51 | 0.32 | - | -
Lip2Wav (Prajwal et al., 2020a) | 1.35 | 0.56 | 0.36 | 6.610 | 7.815
VCA-GAN (Kim et al., 2022) | 1.43 | 0.58 | 0.40 | - | -
VAE-GAN (Hegde et al., 2022) | 1.35 | 0.55 | 0.35 | - | -
Ours | 1.34 | 0.61 | 0.42 | 6.623 | 6.901
LRW (Chung and Zisserman, 2016a) |  GAN-based (Vougioukas et al., 2019) | 0.72 | 0.10 | 0.02 | 1.983 | 9.426
Lip2Wav (Prajwal et al., 2020a) | 1.19 | 0.54 | 0.34 | 2.526 | 8.286
VAE-GAN (Hegde et al., 2022) | 0.78 | 0.15 | 0.03 | 2.538 | 8.173
VCA-GAN (Kim et al., 2022) | 1.33 | 0.56 | 0.36 | - | -
SVTS (Mira et al., 2022) | 1.49 | 0.64 | 0.48 | - | -
Multi-task L2S (Kim et al., 2023) | 1.56 | 0.64 | 0.47 | 4.876 | 8.102
Lip-to-Text + TTS baseline | 0.69 | 0.10 | 0.01 | 1.993 | 12.872
Ours | 1.61 | 0.71 | 0.56 | 6.812 | 6.974
LRS2 (Chung et al., 2017) |  Lip2Wav (Prajwal et al., 2020a) | 0.58 | 0.28 | 0.11 | 1.874 | 11.48
VAE-GAN (Vougioukas et al., 2019) | 0.60 | 0.34 | 0.17 | 2.507 | 8.155
VCA-GAN (Kim et al., 2022) | 1.24 | 0.40 | 0.13 | 4.016 | 7.914
SVTS (Mira et al., 2022) | 1.34 | 0.49 | 0.29 | - | -
Multi-task L2S (Kim et al., 2023) | 1.36 | 0.52 | 0.34 | 4.001 | 8.192
Lip-to-Text + TTS baseline | 0.53 | 0.19 | 0.02 | 2.013 | 15.891
Ours | 1.47 | 0.65 | 0.47 | 8.083 | 6.586
LRS3 (Afouras et al., 2018b) |  VAE-GAN (Vougioukas et al., 2019) | 0.51 | 0.30 | 0.15 | 2.063 | 8.256
VCA-GAN (Kim et al., 2022) | 1.23 | 0.47 | 0.20 | 3.905 | 8.392
SVTS (Mira et al., 2022) | 1.25 | 0.50 | 0.27 | - | -
Multi-task L2S (Kim et al., 2023) | 1.31 | 0.48 | 0.26 | 3.876 | 8.677
Lip-to-Text + TTS baseline | 0.42 | 0.16 | 0.01 | 1.771 | 17.882
Ours | 1.39 | 0.58 | 0.37 | 7.886 | 6.850

####  4.1.2. Speech Synthesis in Unconstrained Settings

Comparisons: In order to assess the performance of lip-to-speech methods in
unconstrained scenarios, we employ three datasets: word-level LRW (Chung and
Zisserman, 2016a), sentence-level LRS2 (Chung et al., 2017), and LRS3 (Afouras
et al., 2018b). While the authors of VAE-GAN (Hegde et al., 2022) have re-
trained the GAN-based (Vougioukas et al., 2019) and Lip2Wav (Prajwal et al.,
2020a) models in a multi-speaker context, we present the scores from their
original study for comparison. For VCA-GAN (Kim et al., 2022), SVTS (Mira et
al., 2022), and Multi-task Lip-to-Speech synthesis (Kim et al., 2023), we
adopt the speech metric (PESQ, STOI and ESTOI) scores from (Kim et al., 2023).
Further, we use publicly accessible pre-trained checkpoints for VCA-
GAN333https://github.com/ms-dot-k/Visual-Context-Attentional-GAN and
Multitask-L2S444https://github.com/ms-dot-k/Lip-to-Speech-Synthesis-in-the-
Wild to generate speech on LRS2 and LRS3 test sets for the former, and LRS2,
LRS3, and LRW test sets for the latter. We utilize these generations to
compute the LSE metrics for both techniques, wherever applicable. Lastly, we
include results for a baseline approach that leverages lip-to-text conversion
followed by multi-speaker TTS without a visual stream in the TTS model. Our
evaluation does not include all metrics that were not originally reported in
the papers or for which no pre-trained model is publicly available.

Results: We present the results on the challenging LRW, LRS2, and LRS3
datasets in Table 1. Our model consistently outperforms the existing methods
by a significant margin on all these datasets. Since GAN-based (Vougioukas et
al., 2019) model was proposed to work for constrained laboratory recorded
datasets, we can observe that extending this model in unconstrained settings
does not yield satisfactory results. Lip2Wav (Prajwal et al., 2020a) performs
decently on the word-level LRW dataset; however, it fails to learn the audio-
visual alignment on the LRS2 dataset, thus leading to very poor performance.
We discard this model for further comparison on the LRS3 dataset. VAE-GAN
(Hegde et al., 2022), VCA-GAN (Kim et al., 2022), SVTS (Mira et al., 2022) and
Multitask-L2S (Kim et al., 2023) generate speech that is in-sync with the
input video; however, they fail to synthesize accurate content. The quality of
the generated speech is often non-intelligible and leads to lower scores in
speech quality metrics. The Lip-to-Text $+$ TTS baseline model is on the
opposite spectrum, where the model generates the content well but fails to
capture the lip-sync, mainly because the model cannot infer the speed,
prosody, and accents of speakers just from the text input. Our model, on the
other hand, is capable of generating both the actual spoken content as well as
maintaining precise lip synchronization. As we can see from the table, we
outperform the previous methods in all the speech quality metrics, indicating
the robustness and superiority of our approach. In Figure 3, we depict how our
model temporally aligns video and text sequences in the process of generating
speech. We encourage the reader to view our demo video comprising multiple
qualitative samples and comparisons.

Figure 3. We visualize the video-text alignment from the scaled dot product
attention step of our model. We observe that the model learns a strong
monotonic near-diagonal attention, as expected.

###  4.2. Human Evaluations

To evaluate the applicability of our method in real-world scenarios, we
conduct subjective human evaluations. We ask $25$ volunteers to assess the
quality of speech generations. The participant group has an almost equal male-
female ratio, spanning an age group of $20-45$ years. We randomly select $10$
long sentences ($10$ seconds or longer) from the test set of LRS3 (Afouras et
al., 2018b) and present the results from different methods to the
participants. We ask them to rate the samples on a scale of $1-5$ based on the
following criteria: (A) Intelligibility (is the speech meaningful?), (B)
Content clarity (are the words clear?), (C) Sync Accuracy, (D) Overall
perceptual quality of the talking head video $+$ audio. We report the mean
opinion scores in Table 2. On par with the quantitative evaluations, our
method is highly rated over other approaches in all the criteria listed above.
As expected, the speech intelligibility is slightly rated higher for the Lip-
to-Text $+$ TTS baseline. However, the overall perceptual quality for this
baseline sharply falls due to the lack of sync between the spoken content and
the lip movements. Overall, Table 2 clearly signifies that our network is able
to generate speech with more clarity, which sounds more natural and is of
considerably higher quality.

Table 2. (A) Intelligibility, (B) Content clarity, (C) Sync Accuracy, (D) Overall perceptual quality. Our model produces natural and realistic speech outputs that is largely preferred by the users in comparison to other approaches. Method | (A) | (B) | (C) | (D)
---|---|---|---|---
GAN-based (Vougioukas et al., 2019) | 2.05 | 1.87 | 1.99 | 2.12
Lip2Wav (Prajwal et al., 2020a) | 1.01 | 1.03 | 1.34 | 1.01
VAE-GAN (Hegde et al., 2022) | 1.07 | 1.33 | 2.18 | 2.57
VCA-GAN (Kim et al., 2022) | 2.18 | 1.88 | 2.97 | 2.54
Multi-task L2S (Kim et al., 2023) | 2.19 | 1.85 | 3.01 | 2.64
Lip-to-Text + TTS baseline | 3.61 | 2.87 | 1.01 | 2.96
Ours | 3.49 | 3.52 | 3.82 | 3.31

###  4.3. Applications in Assistive Technology

Lip-to-Speech synthesis has a host of applications in a world that is becoming
increasingly digital. Simple applications such as performing video calls in
quiet environments, filling in the audio interruptions due to technical
issues, eliminating unwanted background chatter, etc., can be made possible
with accurate lip-to-speech. We believe the most significant application of
lip-to-speech can be in assistive technologies. It can revolutionize the
current assistive systems used to improve the communication ability of people
suffering from various disorders affecting their speech. Patients suffering
from vocal cord disabilities can mouth words to communicate naturally with the
world around them. The synthesized speech can be personalized and also be in
sync with the speaker’s lip movements.

##### Generating Speech for a Patient suffering from ALS:

A recent work (Sen et al., 2021) proposed a lip reading technique for a
patient suffering from ALS. The patient has feeble vocal cord movements but
can mouth words silently. The authors of the paper collected limited amounts
of data from the patient and trained a model to recognize words and sentences
from his lip movements. As a result of our significant improvement in this
task, we can now demonstrate the benefit of using lip-to-speech as a future
assistive technology. Through the help of authors of (Sen et al., 2021), we
evaluate our lip-to-speech system on the patients’ data. Please note that the
data was anonymized and was used only for research purposes. We find that the
lip-to-text module generates fairly accurate text (WER of $\approx 37\%$), and
the visual TTS model generates clear speech in sync with the patient’s lip
movements. This is the first demonstration of automatic lip-to-speech
synthesis for an unseen speaker in an entirely out-of-domain real-world
application. Further, we also test our model on the other deaf speakers
studied in (Sen et al., 2021) and observe accurate performance.

Figure 4. We demonstrate our model on an ALS patient who cannot voice words
but can mouth them. We can generate the speech corresponding to the silent lip
movements. Lip-to-Speech can thus be a cheap and non-invasive method to assist
someone who has lost their voice.

##### Ethical Considerations

We acknowledge that our work has the potential to generate synthetic speech
for videos, given that we only require a $1-$second voice sample from any
target speaker. However, since the video provides context and constrains the
output speech, the generated speech will likely follow the original content
closely. We recognize the importance of ethical considerations regarding using
such models and ensure that our models will only be shared with users who
consent to limit their usage to research-oriented and ethically valid tasks.

##  5\. Ablation Studies

In this section, we perform several ablation studies to understand the effect
of different components of our model. All the ablation experiments are
performed on the LRS2 (Chung et al., 2017) test set.

##### Effect of different pre-trained lip-to-text models

Additional experiments were conducted using other pre-trained lip-to-text
models, specifically DeepLR (Afouras et al., 2018a) and AV-HuBERT (Shi et al.,
2022). The former had a WER of $51.3$ on the LRS2 test set, while the latter
had a WER of $46.6$. The input text to our pre-trained Visual TTS module was
taken from different Lip-to-Text models. Additionally, we also directly
provide the ground-truth text from the LRS2 test set. As shown in Table 3, our
approach recovered speech that was somewhat accurate, despite the presence of
noisy text transcripts from both models. This can be attributed to two
factors: (i) the correction of errors made by the lip-to-text network by our
pipeline to some extent (demonstrated in the demo video); and (ii) the reduced
difference in scores between homonyms such as ”ship and sheep” or ”berth and
birth” in the audio domain.

Table 3. Comparison of using generated text from different lip-to-text network in our pipeline. We also report the WER of the lip reading model (L2T-WER) on the LRS2 test set as a reference. Method | L2T-WER | PESQ$\uparrow$ | STOI$\uparrow$ | ESTOI$\uparrow$ | LSE-C$\uparrow$ | LSE-D$\downarrow$
---|---|---|---|---|---|---
Deep Lip Reading (Afouras et al., 2018a) | 51.3 | 1.17 | 0.40 | 0.22 | 7.847 | 6.904
AV-HuBERT (Shi et al., 2022) | 46.1 | 1.27 | 0.53 | 0.40 | 7.960 | 7.003
VTP (Ours) | 22.6 | 1.47 | 0.65 | 0.47 | 8.083 | 6.586
GT text | - | 1.51 | 0.69 | 0.50 | 8.781 | 6.106

##### Effect of different visual representations

We train our proposed Visual Text-to-Speech module with RGB face crops instead
of the VTP embeddings to generate speech conditioned on text and lip
movements. Based on our observations from Table 4, VTP embeddings are the most
suitable for this task because they excel in localizing and representing the
shape of the speaker’s lips.

Table 4. We present the effect of using different visual representations for training the Visual TTS module. Method | PESQ$\uparrow$ | STOI$\uparrow$ | ESTOI$\uparrow$ | LSE-C$\uparrow$ | LSE-D$\downarrow$
---|---|---|---|---|---
Face crops | 1.17 | 0.40 | 0.22 | 7.847 | 6.904
VTP (Ours) | 1.47 | 0.65 | 0.47 | 8.083 | 6.586

##  6\. Limitations

In our work, we address the problem of lip-to-speech networks not being able
to learn a language model directly from speech supervision. We do so by using
a pre-trained lip-to-text network. While our model does not require ground-
truth text annotations, the lip-to-text model which we build upon has been
trained with text supervision. However, recent efforts in self-supervised pre-
training have led to a sharp decrease in the number of text annotations
required for training accurate lip-to-text (Shi et al., 2022), making it
easier to extend such models to lip-to-speech using our approach. Currently,
we have only tested our model in English, and it remains to be validated in
other languages.

##  7\. Conclusion

Our research presents an innovative approach to unconstrained multi-speaker
lip-to-speech synthesis that outperforms previous methods by incorporating
language and visual information from a highly accurate lip-to-text model. We
demonstrate significant improvements in lip-to-speech synthesis, generating
high-quality outputs that seamlessly synchronize with silent lip video. Our
study can potentially open up exciting avenues for future research. We are
particularly encouraged by the success of our approach in assistive
technology, where we have shown that our method can generate accurate speech
from silent lip movements of individuals with speech impairments. Overall, we
are optimistic about the possibilities of our approach to improve
communication and enhance the quality of life for people with speech
impairments, and we look forward to seeing our work drive further progress in
this field.

Acknowledgement: This work is supported by MeitY, Government of India

## References

  * (1)
  * Afouras et al. (2018c) Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. 2018c.  Deep audio-visual speech recognition.  _IEEE transactions on pattern analysis and machine intelligence_ (2018).
  * Afouras et al. (2018a) Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. 2018a.  Deep Lip Reading: a comparison of models and an online application. In _INTERSPEECH_.
  * Afouras et al. (2018b) Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. 2018b.  LRS3-TED: a large-scale dataset for visual speech recognition.  _arXiv preprint arXiv:1809.00496_ (2018).
  * Akbari et al. (2017) Hassan Akbari, Himani Arora, Liangliang Cao, and Nima Mesgarani. 2017.  Lip2Audspec: Speech Reconstruction from Silent Lip Movements Video.  _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ (2017), 2516–2520.
  * Assael et al. (2016) Yannis M Assael, Brendan Shillingford, Shimon Whiteson, and Nando De Freitas. 2016.  Lipnet: End-to-end sentence-level lipreading.  _arXiv preprint arXiv:1611.01599_ (2016).
  * Chan et al. (2016) William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. 2016\.  Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. 4960–4964.  https://doi.org/10.1109/ICASSP.2016.7472621
  * Chung et al. (2017) Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. 2017.  Lip reading sentences in the wild. In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 3444–3453.
  * Chung and Zisserman (2016a) Joon Son Chung and Andrew Zisserman. 2016a.  Lip reading in the wild. In _Asian Conference on Computer Vision_. Springer, 87–103.
  * Chung and Zisserman (2016b) J. S. Chung and A. Zisserman. 2016b.  Out of time: automated lip sync in the wild. In _Workshop on Multi-view Lip-reading, ACCV_.
  * Cooke et al. (2006) Martin Cooke, Jon Barker, Stuart Cunningham, and Xu Shao. 2006\.  An audio-visual corpus for speech perception and automatic speech recognition.  _The Journal of the Acoustical Society of America_ 120, 5 (2006), 2421–2424.
  * Ephrat et al. (2017) Ariel Ephrat, Tavi Halperin, and Shmuel Peleg. 2017\.  Improved Speech Reconstruction from Silent Video.  _2017 IEEE International Conference on Computer Vision Workshops (ICCVW)_ (2017), 455–462.
  * Ephrat and Peleg (2017) Ariel Ephrat and Shmuel Peleg. 2017.  Vid2Speech: speech reconstruction from silent video. In _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_.
  * Freitag and Al-Onaizan (2017) Markus Freitag and Yaser Al-Onaizan. 2017.  Beam Search Strategies for Neural Machine Translation. In _NMT@ACL_.
  * gil Lee et al. (2022) Sang gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sung-Hoon Yoon. 2022.  BigVGAN: A Universal Neural Vocoder with Large-Scale Training.  _ArXiv_ abs/2206.04658 (2022).
  * Harte and Gillen (2015) Naomi Harte and Eoin Gillen. 2015.  TCD-TIMIT: An audio-visual corpus of continuous speech.  _IEEE Transactions on Multimedia_ 17, 5 (2015), 603–615.
  * Hassid et al. (2022) Michael Hassid, Michelle Tadmor Ramanovich, Brendan Shillingford, Miaosen Wang, Ye Jia, and Tal Remez. 2022.  More than Words: In-the-Wild Visually-Driven Prosody for Text-to-Speech.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ (2022), 10577–10587.
  * Hegde et al. (2022) Sindhu B. Hegde, K R Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, and C.V. Jawahar. 2022\.  Lip-to-Speech Synthesis for Arbitrary Speakers in the Wild. In _Proceedings of the 30th ACM International Conference on Multimedia_ (Lisboa, Portugal) _(MM ’22)_. Association for Computing Machinery, New York, NY, USA, 6250–6258.  https://doi.org/10.1145/3503161.3548081
  * Hu et al. (2021) Chenxu Hu, Qiao Tian, Tingle Li, Yuping Wang, Yuxuan Wang, and Hang Zhao. 2021\.  Neural Dubber: Dubbing for Videos According to Scripts. In _NeurIPS_.
  * Jia et al. (2018) Ye Jia, Yu Zhang, Ron J. Weiss, Quan Wang, Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, and Yonghui Wu. 2018.  Transfer Learning from Speaker Verification to Multispeaker Text-to-Speech Synthesis. In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ _(NIPS’18)_. Curran Associates Inc., 4485–4495.
  * Kim et al. (2022) Minsu Kim, Joanna Hong, and Yong Man Ro. 2022.  Lip to Speech Synthesis with Visual Context Attentional GAN. In _Neural Information Processing Systems_.
  * Kim et al. (2023) Minsu Kim, Joanna Hong, and Yong Man Ro. 2023.  Lip-to-Speech Synthesis in the Wild with Multi-task Learning.  _ArXiv_ abs/2302.08841 (2023).
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization.  _CoRR_ abs/1412.6980 (2015).
  * Mira et al. (2022) Rodrigo Mira, Alexandros Haliassos, Stavros Petridis, Björn Schuller, and Maja Pantic. 2022\.  SVTS: Scalable Video-to-Speech Synthesis. In _Interspeech_.
  * Momeni et al. (2022) Liliane Momeni, Hannah Bull, Prajwal K R, Samuel Albanie, Gül Varol, and Andrew Zisserman. 2022\.  Automatic dense annotation of large-vocabulary sign language videos. In _ECCV_.
  * Ping et al. (2018) Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O. Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. 2018\.  Deep Voice 3: 2000-Speaker Neural Text-to-Speech. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=HJtEm4p6Z
  * Prajwal et al. (2022) K R Prajwal, Triantafyllos Afouras, and Andrew Zisserman. 2022\.  Sub-Word Level Lip Reading With Visual Attention. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 5162–5172.
  * Prajwal et al. (2020a) K R Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, and C.V. Jawahar. 2020a.  Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis. In _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Prajwal et al. (2020b) K R Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, and C.V. Jawahar. 2020b.  A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild. In _Proceedings of the 28th ACM International Conference on Multimedia_ (Seattle, WA, USA) _(MM ’20)_. Association for Computing Machinery, New York, NY, USA, 484–492.  https://doi.org/10.1145/3394171.3413532
  * R et al. (2021) Prajwal K R, Liliane Momeni, Triantafyllos Afouras, and Andrew Zisserman. 2021.  Visual Keyword Spotting with Attention. In _BMVC_.
  * Ren et al. (2020) Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2020\.  Fastspeech 2: Fast and high-quality end-to-end text to speech.  _arXiv preprint arXiv:2006.04558_ (2020).
  * Sen et al. (2021) Bipasha Sen, Aditya Agarwal, Rudrabha Mukhopadhyay, Vinay Namboodiri, and CV Jawahar. 2021\.  Personalized One-Shot Lipreading for an ALS Patient.  _arXiv preprint arXiv:2111.01740_ (2021).
  * Serdyuk et al. (2022) Dmitriy Serdyuk, Otavio Braga, and Olivier Siohan. 2022\.  Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition for Single and Muti-Person Video. In _Proc. Interspeech 2022_. 2833–2837.  https://doi.org/10.21437/Interspeech.2022-10920
  * Shen et al. (2018) Jonathan Shen, R. Pang, Ron J. Weiss, M. Schuster, Navdeep Jaitly, Z. Yang, Z. Chen, Yu Zhang, Yuxuan Wang, R. Skerry-Ryan, R. A. Saurous, Yannis Agiomyrgiannakis, and Y. Wu. 2018.  Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions.  _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ (2018), 4779–4783.
  * Shi et al. (2022) Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdel rahman Mohamed. 2022.  Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction.  _ArXiv_ abs/2201.02184 (2022).
  * Vaswani et al. (2017) Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is All you Need.  _ArXiv_ abs/1706.03762 (2017).
  * Vougioukas et al. (2019) Konstantinos Vougioukas, Pingchuan Ma, Stavros Petridis, and Maja Pantic. 2019.  Video-Driven Speech Reconstruction using Generative Adversarial Networks.  _arXiv preprint arXiv:1906.06301_ (2019).