* de Blok et al. (2016) de Blok W. J. G., et al., 2016, in MeerKAT Science: On the Pathway to the SKA. p. 7 (arXiv:1709.08458)
  * de Blok et al. (2018) de Blok W. J. G., et al., 2018, darkblueApJ, 865, 26
  * de Vaucouleurs et al. (1991) de Vaucouleurs G., de Vaucouleurs A., Corwin Jr. H. G., Buta R. J., Paturel G., Fouque P., 1991, Third Reference Catalogue of Bright Galaxies
  * van den Bosch et al. (2008) van den Bosch F. C., Aquino D., Yang X., Mo H. J., Pasquali A., McIntosh D. H., Weinmann S. M., Kang X., 2008, darkblueMNRAS, 387, 79
  * von der Linden et al. (2010) von der Linden A., Wild V., Kauffmann G., White S. D. M., Weinmann S., 2010, darkblueMNRAS, 404, 1231


A Rigorous Holographic Bound on
AdS Scale Separation

Eric Perlmutter

Université Paris-Saclay, CNRS, CEA, Institut de Physique Théorique, 91191,
Gif-sur-Yvette, France
Institut des Hautes Études Scientifiques, 91440, Bures-sur-Yvette, France

<EMAIL_ADDRESS>
We give an elementary proof of the following property of unitary, interacting
four-dimensional $\mathcal{N}=2$ superconformal field theories: at large
central charge $c$, there exist at least $\sqrt{c}$ single-trace, scalar
superconformal primary operators with dimensions $\Delta\lesssim\sqrt{c}$
(suppressing multiplicative logarithmic corrections). This follows from a
stronger, more refined bound on the spectral density in terms of the
asymptotic growth rate of the central charge. The proof employs known results
on the structure of Coulomb branch operators. Interpreted holographically,
this bounds the possible degree of scale separation in semiclassical AdS5
half-maximal supergravity. In particular, the bulk must contain an infinite
tower of charged scalar states of energies parametrically below the large
black hole threshold, $E_{\rm BH}\sim c$. We address the extreme case of AdS5
pure supergravity, ruling it out as the asymptotic limit of certain sequences
in theory space, though the general question remains open.

###### Contents

  1. 1 The Problem
  2. 2 The Ingredients
  3. 3 The Proof
  4. 4 The Bulk
  5. A The Appendix

##  The Problem

The question of AdS scale separation is whether a consistent theory of quantum
gravity can, in the semiclassical limit, admit an AdS vacuum without
parametrically large extra dimensions.

Recent years have seen significant efforts to scrutinize and extend earlier
proposals for scale-separated solutions of string theory [1, 2, 3]. This
uptick was spurred in part by the AdS distance conjecture [4] and other
Swampland considerations, and the pressing cosmological implications of this
question vis-à-vis possible de Sitter uplift.

A direct bulk approach to the problem is inherently limited: to tuck the extra
dimensions away at a parametrically small scale, one must grapple with the
full machinery of string/M-theory, not just its 10- or 11-dimensional
supergravity limit. For this reason, it seems fair to say that gravity
constructions are unlikely to definitively answer this question beyond a
reasonable doubt, unless and until non-perturbative string/M-theory are well-
understood.

Boundary CFT approaches, however, offer hope for unambiguous resolution. There
is a growing appetite for the conformal bootstrap to solve this problem, using
axiomatic constraints to rigorous ends. In particular, one might realistically
hope to exclude, or bound the degree of, AdS scale separation holographically
using the large $N$ bootstrap (as opposed to explicitly constructing a
candidate CFT). Some older [5, 6] and newer [7, 8, 9, 10, 11] works explore
the holographic dictionary for putative scale-separated string backgrounds,
but fundamental constraints are missing. A collage of constructions, claims,
counter-claims and conjectures is well-reviewed in [12].

For better or worse, the conformal bootstrap is not magic: the CFT avatar of
excluding AdS scale separation is still a hard problem. As emphasized in [13,
14], this question differs fundamentally from the canonical bootstrap endeavor
of maximizing the spectral gap to the first primary operator: large (i.e. AdS-
sized) extra dimensions in gravity are manifest as infinite towers of
operators in CFT. Their characteristic energy $\Delta$ and asymptotic density
$\rho(\Delta)$ encode the size and number of bulk dimensions, respectively,
quantities which can be read off from a positive sum rule for CFT correlators
of light operators [14].

Even supersymmetric versions of this problem, as articulated in [15], remain
open, providing compelling targets for the superconformal bootstrap. It is not
known whether continuous R-symmetry of a superconformal field theory (SCFT) is
always geometrized in AdS, despite familiar examples under the lamppost. Does
quantum gravity admit, say, a maximally supersymmetric “pure AdS5” background
without a large $S^{5}$? There are arguments both for [16] and against [17]
this possibility in the literature. The analogous questions on the CFT side
concern the existence of “exotic” SCFTs without infinite towers of BPS
operators, which are a boundary hallmark of a large compact manifold in the
bulk.

This paper takes a step forward on the supersymmetric version of this problem,
proving a universal result for 4d $\mathcal{N}=2$ SCFTs at large central
charges. Our proof uses nothing more than established facts about Coulomb
branch geometry. The main result is in (3.8)–(3.10), a constraint on the
spectrum of Coulomb branch primaries. Extremizing over theory space gives the
absolute bound (3.10), valid for all $\mathcal{N}=2$ SCFTs (subject to widely-
held assumptions detailed below), as quoted in the Abstract in slightly
simplified form.

The result is modest, and its proof, elementary; but its consequences for
gravity are significant. By AdS/CFT, these theories are dual to asymptotically
AdS5 vacua of semiclassical gravity with (at least) half-maximal
supersymmetry. The result establishes the necessity of an infinite (Planckian)
number of $U(1)$-charged, scalar bulk states which are parametrically below
the large AdS5 black hole threshold, $E_{\rm BH}\sim c$. For $a\approx c$,
this is a quantitative bound on the degree of scale separation in AdS5
Einstein supergravity. We address the extreme sub-case of a putative theory of
semiclassical AdS5 pure supergravity, which we define (after some discussion
in Appendix A) as a theory with only supergravitons parametrically below the
five-dimensional Planck scale: though we cannot unequivocally rule this
spectrum out, we do exclude it as the asymptotic limit of certain paths
through superconformal theory space.

##  The Ingredients

We consider unitary, local $\mathcal{N}=2$ SCFTs in four dimensions; this is a
vast subject (see e.g. [18, 19]), but we will keep our presentation to the
point. We are interested in the BPS spectra of these theories at large central
charge. An $\mathcal{N}=2$ SCFT is partially characterized by its Coulomb
branch, of rank $r\in\mathbb{N}$. The Coulomb branch may be parameterized by
the vacuum expectation values of $r$ superconformal primaries, the generators
of the chiral ring, which are the bottom components of protected
$\mathcal{E}_{r(0,0)}$-type superconformal multiplets in the notation of [20]
($L\overline{B_{1}}[0;0]^{(0;r)}$ multiplets in the notation of [21]). These
operators, call them $\\{{\cal O}_{i}\\}$, have vanishing $SU(2)_{R}$ charge,
and $U(1)_{r}$ charge equal to their conformal dimensions, $r_{i}=\Delta_{i}$.
In all known examples, the $\\{{\cal O}_{i}\\}$ are Lorentz scalars, which we
have assumed here.

Let us state up front that we are making the oft-used assumption that the
Coulomb branch is freely generated, i.e. that the generators satisfy no non-
trivial relations.111We thank Mario Martone and Leonardo Rastelli for
discussions about the status of this assumption. This allows for an
unambiguous assignment of dimensions $\Delta_{i}$ to operators ${\cal O}_{i}$.
There is a whole story here, with possible violations of this property if
discrete gaugings are allowed, leading to complex singularities of the Coulomb
branch geometry [22, 23, 24, 25]. Having emphasized this, we henceforth take
the Coulomb branch to be freely generated, while keeping in mind the
limitations of this assumption at the most abstract level of $\mathcal{N}=2$
SCFT.

We will need two ingredients for our proof.

#### i) Central charge sum rule

Recent work on the intricacies of Coulomb branch geometry has established the
following central charge formulae in $\mathcal{N}=2$ SCFTs [26]:

$\begin{split}2a&=-{r\over 12}+{\mathbb{S}(r)\over 2}+f(\text{\tt other
Coulomb branch data})\\\ c&={r\over 6}+f(\text{\tt other Coulomb branch
data})\end{split}$ (2.1)

where

$\mathbb{S}(r):=\sum_{i=1}^{r}\Delta_{i}$ (2.2)

Happily, we won’t need the unspecified function of other Coulomb branch data
in order to prove the desired result at large central charge. Instead we take
a simpler approach. First, these results imply the following formula,
previously established in a more limited context by Shapere and Tachikawa
[27]:

$2a-c={\mathbb{S}(r)\over 2}-{r\over
4}\,,\quad\text{where}\quad\mathbb{S}(r):=\sum_{i=1}^{r}\Delta_{i}$ (2.3)

Noting the unitarity bound $\Delta_{i}\geq 1$, saturated by a free vector
multiplet, an interacting unitary $\mathcal{N}=2$ SCFT obeys $\mathbb{S}(r)>r$
and hence $2a-c>{r\over 4}$. Besides finding the independent formulae (2.1),
one achievement of the work since [27] is that it establishes the Shapere-
Tachikawa formula (2.3) in fuller generality, without the assumptions of [27]
(e.g. that there is a Lagrangian point on the SCFT moduli space).222The work
[26] and its predecessors make their own assumptions, of course, one of which
(free generation) was discussed earlier. Nevertheless their results are
expected to be very robust, sufficient to perform a quite general
classification of $\mathcal{N}=2$ SCFTs. In indirect support of this, the
rationality of central charges was confirmed by [28] using different, Higgs
branch-based methods and employing independent assumptions. We will use this
formula in conjunction with the Hofman-Maldacena bound [29] for local
$\mathcal{N}=2$ SCFTs,

${1\over 2}\leq{a\over c}\leq{5\over 4}\,.$ (2.4)

The lower and upper bounds are realized by free hypermultiplets and free
vector multiplets, repectively.

#### ii) Coulomb branch operator dimensions

Our second ingredient is a fascinating feature of $\mathcal{N}=2$ SCFT
operator spectra: at a given rank $r$, the Coulomb branch dimensions
$\\{\Delta_{i}\\}$ are drawn from a finite, $r$-dependent set of rationals. As
proven in [30, 31], this set admits a remarkably concise characterization:

$\Delta_{i}\in\left\\{{n\over m}\,\Big{|}\,\varphi(n)\leq
2r\,,\leavevmode\nobreak\ \leavevmode\nobreak\ 0<m\leq
n\,,\leavevmode\nobreak\ \leavevmode\nobreak\ (m,n)=1\right\\}$ (2.5)

where $\varphi(n)$ is Euler’s totient function. Most of the number-theoretic
structure of this result will not be used here. All we need is the existence
of $\Delta_{\rm max}$, the largest possible dimension at rank $r$:

$\Delta_{\rm max}:=\text{max}\,\big{\\{}n\,|\,\varphi(n)\leq 2r\big{\\}}$
(2.6)

For finite $r$, $\Delta_{\rm max}$ is finite. In what follows, we order the
dimensions in a given $r$-tuple as

$1\leq\Delta_{1}\leq\Delta_{2}\leq\cdots\leq\Delta_{r}\leq\Delta_{\rm max}\,.$
(2.7)

###  Large $r,a,c$ limits

Consider the large rank limit, $r\rightarrow\infty$. We first establish the
following warmup result:333Throughout the paper we use $\sim$ to indicate
asymptotic scaling up to numerical prefactors, and $\approx$ to indicate
asymptotic equality including numerical prefactors, with additive corrections
left implicit in all cases.

$\mathbb{S}(r)\sim r$ is inconsistent with the existence of the SCFT at large
rank.

More precisely, $\mathbb{S}(r)\sim r$ is incompatible with a finite partition
function on the spatial sphere. The proof is simply that a unitary theory can
only achieve linear scaling of $\mathbb{S}(r)$ if ${\cal O}(r)$ of the $r$
Coulomb branch primaries have dimensions bounded above by $\Delta_{*}\sim{\cal
O}(1)$; but this violates finiteness of $Z_{S^{1}_{\beta}\times S^{3}}={\rm
tr}_{\mathcal{H}}(e^{-\beta H})$, which requires that

$\int_{0}^{{\rm finite}}d\Delta\,\rho(\Delta)<\infty$ (2.8)

where $\rho(\Delta)$ is the spectral density of local operators on $S^{3}$.
Therefore, the large $r$ limit does not exist.444Let us register a
complementary (but more restricted) argument that $a\approx c\sim r$ theories
cannot be dual to semiclassical Einstein gravity specifically: with this
scaling, the species scale $\Lambda_{s}=(c/n_{L})^{1/3}$, where $n_{L}$ is the
number of bulk fields with energies below $\Lambda_{s}$, would become ${\cal
O}(1)$, invalidating Einstein gravity as a good semiclassical EFT. One can
rephrase this conclusion as the absence of “large $c$ $\mathcal{N}=2$ vector
models”; we suggest that this may also be true of $\mathcal{N}=1$ SCFTs, for a
suitable replacement of the Coulomb branch rank.555The works [32, 33] study
putative 4d $\mathcal{N}=1$ SCFTs, labeled “dense” theories, which have $a\sim
c\sim N$, where $N$ is the rank of a UV gauge theory from which the IR SCFT
flows. These theories have bands of operators within finite ${\cal O}(1)$
windows with $1/N$-suppressed splittings, which therefore become continuous in
the large $N$ limit, violating the finiteness criterion (2.8).

So, at $r\rightarrow\infty$ we have $\mathbb{S}(r)\gg r$, and hence

$2a-c\approx{\mathbb{S}(r)\over 2}$ (2.9)

Therefore, $r\rightarrow\infty$ implies $2a-c\rightarrow\infty$. Combining
this with (2.4) implies that $a$ and $c$ both grow large independently:

$r\rightarrow\infty\qquad\Rightarrow\qquad a,c\rightarrow\infty$ (2.10)

That is, large rank implies large central charges.

The implication runs in reverse as well, modulo two small caveats. The first
is if $2a-c$ is finite to leading order: in particular, $r$ can remain finite
at large central charge if and only if

${a\over c}\approx{1\over 2}+{\cal O}\left({1\over c}\right)$ (2.11)

as $c\rightarrow\infty$ (with a strictly positive correction term). The case
${a\over c}={1\over 2}$ is only known to be realized by free hypers; adopting
this perspective, the possibility (2.11) can be phrased as the theory
“becoming free” at large central charge. In addition, one must note the
logical possibility that the SCFT at large central charge could in principle
have no Coulomb branch, i.e. $r=0$; but it is widely believed that there are
no interacting $\mathcal{N}=2$ SCFTs with $r=0$.

All told, we can summarize as follows: assuming the existence of a Coulomb
branch,

$a,c\rightarrow\infty\qquad\Leftrightarrow\qquad r\rightarrow\infty\,,$ (2.12)

with the only possible exception to the $\Rightarrow$ direction being the edge
case ${a\over c}\approx{1\over 2}$. This case is, at any rate, maximally far
from the Einstein regime in which $a\approx c$, to which we soon turn.

As for the Coulomb branch dimensions at $r\rightarrow\infty$, the upper bound
$\Delta_{\rm max}$ scales asymptotically as [30]

$\Delta_{\rm max}\approx 2e^{\gamma_{E}}\,r\log\log r$ (2.13)

where $\gamma_{E}\approx 0.577$ is the Euler gamma function.

##  The Proof

The key observation is that the upper-boundedness of Coulomb branch
dimensions, which in turn determine the central charges, implies a spectral
bound of the former when expressed in terms of the latter.

In particular, bounding spectra at large central charge becomes an
extremization problem to be solved in the asymptotic limit
$r\rightarrow\infty$: given an ordered $r$-tuple
$\\{\Delta_{1},\ldots,\Delta_{r}\\}$, maximize this set as a function of
central charges, where the latter are determined by $\mathbb{S}(r)$ via (2.3)
and (2.4). We are using “maximize this set” as a placeholder for different
choices of extremization, depending on the problem of interest. For instance,
to extremize the spectral gap à la bootstrap, one solves a maximin problem,
choosing to maximize $\Delta_{1}$ among allowed $r$-tuples.

The main result now follows quickly. We established earlier that
$\mathbb{S}(r)\gg r$, but we now note the obvious upper bound as well,

$\mathbb{S}(r)\leq r\Delta_{\rm max}\approx 2e^{\gamma_{E}}r^{2}\log\log r\,.$
(3.1)

In fact, strictly maximizing all $r$ operator dimensions is disallowed by more
subtle number-theoretic aspects of Coulomb branch geometry – when
$\Delta_{i}=\overline{\Delta}$ for some $\overline{\Delta}$, the latter must
actually be drawn from the allowed set at $r=1$ [30] – so in the
$r\rightarrow\infty$ limit, one ought to interpret the upper bound as

$\mathbb{S}(r)\approx r\Delta_{\rm max}$ (3.2)

with negative splittings that are subleading in $r$. For simplicity, and our
ultimate interest in the Einstein gravity regime, we specialize to $a\approx
c$. Then plugging (2.13) and (3.2) into (2.9) gives

$c\approx e^{\gamma_{E}}r^{2}\log\log r\,.$ (3.3)

In terms of central charge,

$\Delta_{\rm max}\approx 2e^{\gamma_{E}/2}\sqrt{c}\sqrt{\log\log c}\,.$ (3.4)

Therefore, there are $r\approx e^{-\gamma_{E}/2}{\sqrt{c}/\sqrt{\log\log c}}$
operators with dimensions $\Delta_{i}\leq\Delta_{\rm max}$. Simplifying this
exact statement by dropping the logs and prefactors gives the result quoted in
the Abstract. This clearly generalizes away from $a\approx c$ to any
asymptotic ratio of $a/c$ at large rank: thanks to Hofman-Maldacena, $a$ and
$c$ will have identical $r$-scaling, but different numerical prefactors
determined by their ${\cal O}(1)$ ratio.

More generally, we can classify theories by the growth of central charge with
the rank:

$\mathbb{S}(r)\sim r^{z}\,,\quad 1\leq z\leq 2\,.$ (3.5)

Multiplicative logarithmic enhancements – required at $z=1$ (cf. (2.8)) and
possible at $z>1$ (e.g. cf. (3.3)) – are left implicit to avoid clutter. Then
for any finite $2a-c>0$, whereupon $a\sim c\sim r^{z}$, the extremal spectrum
consistent with this scaling, i.e. the sparsest possible low-lying spectrum,
is obtained by taking the lowest $\approx r$ operator dimensions to all
approach

$\Delta_{*}(z)\sim r^{z-1}\,.$ (3.6)

This corresponds to parametric solution of a constrained optimization problem:
maximize $\Delta_{r}$, then maximize $\Delta_{r-1}$, and so on until
maximization of $\Delta_{1}$, all subject to the scaling (3.5). In other
words, defining a spectral function $\mathbb{N}(\Delta_{-},\Delta_{+})$
counting primaries in an interval,

$\mathbb{N}(\Delta_{-},\Delta_{+}):=\int_{\Delta_{-}}^{\Delta_{+}}d\Delta\,\rho(\Delta)\,,\quad\mathbb{N}_{*}(z):=\mathbb{N}(1,\Delta_{*}(z))$
(3.7)

where $\rho(\Delta)$ is the spectral density of Coulomb branch generators, the
scaling (3.5) requires that $\mathbb{N}_{*}(z)\approx r$, and the extremal
spectrum has these states “piled up” near $\Delta_{*}(z)$.666Equivalently,
$\mathbb{N}(\Delta_{-},\Delta_{\rm max})\sim o(r)$ for any
$\Delta_{-}\gg\Delta_{*}(z)$: there can be states with
$\Delta\gg\Delta_{*}(z)$, but not too many of them, lest the scaling (3.5) be
violated. Therefore, and restating in terms of central charge
$c\rightarrow\infty$,  $c\sim
r^{z}\quad\Rightarrow\quad\exists\leavevmode\nobreak\ \leavevmode\nobreak\
\mathbb{N}_{*}(z)\leavevmode\nobreak\ \text{operators}\leavevmode\nobreak\
\\{{\cal O}_{i}\\}\leavevmode\nobreak\ \text{with}\leavevmode\nobreak\
\Delta_{i}\lesssim\Delta_{*}(z)$ (3.8) where $\mathbb{N}_{*}(z)\sim
c^{1/z}\,,\quad\Delta_{*}(z)\sim c^{1-{1/z}}$ (3.9) This is the general
result. Scanning over possible SCFTs by varying $z$, the spectral gap as a
function of $c$ is maximized at $z=2$ dressed with doubly logarithmic
corrections (3.3), giving the absolute bound derived earlier, valid for all
$\mathcal{N}=2$ SCFTs:  $\exists\leavevmode\nobreak\ \leavevmode\nobreak\
\mathbb{N}_{*}\approx e^{-\gamma_{E}/2}{\sqrt{c}\over\sqrt{\log\log
c}}\leavevmode\nobreak\ \text{operators}\leavevmode\nobreak\ \\{{\cal
O}_{i}\\}\leavevmode\nobreak\ \text{with}\leavevmode\nobreak\
\Delta_{i}\lesssim\Delta_{\rm max}\approx
2e^{\gamma_{E}/2}\sqrt{c}\sqrt{\log\log c}$ (3.10)

##  The Bulk

To summarize, unitary interacting $\mathcal{N}=2$ SCFTs at large central
charge contain an infinite tower of $U(1)_{r}$-charged, scalar, single-trace,
superconformal primary local operators $\\{{\cal O}_{i}\\}$ with dimensions
$\Delta_{i}<\Delta_{*}$, where

$\Delta_{*}\sim c^{1-{1/z}}\ll c\,,\qquad 1\leq z\leq 2$ (4.1)

with $z$ defined by the asymptotic scaling $c\sim r^{z}$ at large rank $r$ of
the Coulomb branch. The number of these operators scales as $\sim c^{1/z}$.
The absolute bound for all $\mathcal{N}=2$ SCFTs (obeying the assumptions of
[30, 31]) is in (3.10). These are special (protected) operators, namely, the
generators of the Coulomb branch chiral ring, of vanishing $SU(2)_{R}$ charge
and $U(1)_{r}$ charge $r_{i}=\Delta_{i}$. We have emphasized in (4.1) the sub-
linear growth of $\Delta_{*}$ in the central charge.

Translating to the semiclassical bulk and taking $a\approx c\sim 1/G_{N}$, the
inverse five-dimensional Newton’s constant, (4.1) is a bound on the degree of
scale separation possible in AdS5 supergravity with at least half-maximal
supersymmetry: there must exist an infinite tower of $U(1)$-charged scalar
states $\\{\phi_{i}\\}$ with energies parametrically below the large black
hole threshold. The latter is set by the mass of large AdS5-Schwarzchild black
holes,

$E_{\rm BH}\sim{\cal O}(1/G_{N})\,.$ (4.2)

In particular, all $\phi_{i}$ obey $E_{i}<E_{*}$ where

$E_{*}\sim(E_{\rm BH})^{1-{1/z}}\ll E_{\rm BH}\,.$ (4.3)

It is perhaps clearer to discuss the absolute bound (3.10) for all such
theories, which is (up to a doubly-logarithmic prefactor) the $z=2$
specialization of the above: there must exist $\sim\sqrt{E_{\rm BH}}$
$U(1)$-charged scalar states $\\{\phi_{i}\\}$ with energies
$E_{i}<E_{*}\sim\sqrt{E_{\rm BH}}$.

We emphasize once again that this furnishes a bound on AdS scale separation
because it implies the existence of an infinity of such excitations, a total
number that scales with the Planck scale, in the semiclassical limit of weak
gravitational coupling. Note that a Planckian number of states in an ${\cal
O}(1)$ window around Planckian energies is compatible with the existence of
the large $c$ limit because the states are not fixed-energy states.777For
example, at higher energies $\Delta\gg E_{\rm BH}$, black hole states obey
$\mathbb{N}(\Delta,\Delta+\delta)\sim e^{S(\Delta)}$ for $\delta\sim{\cal
O}(1)$.

#### Comments

What are these $U(1)_{r}$-charged bulk states? In canonical AdS/CFT dual
pairs, of course, Coulomb branch generators are dual to KK modes on a large
internal space, with energies of order one in AdS units; should the states
$\\{\phi_{i}\\}$ be realized geometrically, the characteristic scale of extra
dimensions is bounded below as

$L_{\rm KK}\gtrsim\ell_{p}^{3/2}$ (4.4)

where $\ell_{p}$ is the five-dimensional Planck scale. But remaining
steadfastly agnostic in the bootstrap spirit, there are many other
possibilities, particularly if the states $\\{\phi_{i}\\}$ have Planckian
energies. In known AdS5 compactifications of string/M-theory, the spectrum
also contains small black holes, with horizon radii obeying $\ell_{p}\lesssim
r_{h}\ll L_{\rm AdS}$; but non-singular, non-hairy BPS black holes in AdS5
[34] must carry large angular momenta, in contrast to the scalar states we are
considering. Small black holes are far from the only possibility. In
string/M-theory, there is a rich spectrum of semiclassical bulk configurations
with energies $M_{p}\lesssim E\lesssim E_{\rm BH}$, including D-branes, giant
gravitons [35], small black holes (possibly with hair [36, 37, 38, 39]),
microstate geometries [40], topological stars [41], grey galaxies [42], and
perhaps yet-undiscovered configurations of matter and black holes in thermal
equlibrium. Depending on the energies $\\{E_{i}\\}$, some of these may be
candidate bulk descriptions of $\\{\phi_{i}\\}$.

We wish to emphasize the conceptual point that BPS states are effective
tracers of extra dimensions in AdS/CFT. For one, the HPPS higher-spin gap
condition [43] for emergence of AdSd+1 Einstein gravity from CFT is blind to
the number of large dimensions $D\geq d+1$ of the bulk; but BPS sectors can be
sensitive to $D$, because towers of R-charged light states may (and, perhaps,
must) be geometrized. More generally, in a $D$-dimensional bulk gravitational
effective field theory, the Planckian states are dual to CFTd operators with
conformal dimensions

$\Delta\sim M_{p,D}\sim c^{\,{1/(D-2)}}\,.$ (4.5)

These operators are generically unprotected. For example, we recall the short
string states in AdS${}_{5}\times S^{5}$ at fixed $g_{s}$, dual to Konishi-
type operators in $\mathcal{N}=4$ SYM with $\Delta\sim\lambda^{1/4}\sim
c^{1/8}$. So although the $c$-scaling of these unprotected operator dimensions
encodes the macroscopic dimensionality of the bulk, it is generally easier to
access protected states than unprotected states in CFT.

Note that (4.3) is broadly applicable to semiclassical gravity: it is not
specific to Einstein gravity, instead relying only on asymptotic scalings
valid for any SCFT with ${a\over c}>{1\over 2}$. In particular, semiclassical
string effects cannot push the compactification scale below (4.4). In
addition, the result applies to SCFTs irrespective of whether they have an
independent higher-spin gap scale, $\Delta_{\rm gap}$ (i.e. it applies to both
string and M-theory duals). This generality follows from a mutual
compatibility of the known moduli-independence of both the central charges
[44] and the Coulomb branch operator dimensions, built into the
$\mathcal{N}=2$ central charge formulae.

It is interesting from the gravity point of view that the Coulomb branch
dimensions completely fix the strength of the gravitational interaction and
the $R^{2}$ correction (and hence the species scale). This suggests a deeper
relation between the Coulomb branch data and BPS black hole sectors that would
be nice to understand.

###  Pure supergravity

Our result also addresses the question of whether AdS5 pure supergravity
exists. It is important to examine what “pure (super)gravity” in more than
three bulk dimensions ought to even mean. We give an extended discussion of
this topic in Appendix A.

Low-energy effective field theory provides a natural and agnostic parametric
definition of semiclassical AdSd+1 pure gravity: namely, the absence of non-
graviton states below Planckian energies $E_{*}\sim M_{p}$, where $M_{p}$ is
the $(d+1)$-dimensional Planck scale. Although Planckian degrees of freedom
lack a universal abstract characterization, the validity of gravitational
effective field theory suggests this threshold. Unlike AdS3 gravity, this
threshold cannot be made sharp with an ${\cal O}(1)$ coefficient.

Taking the effective field theory view and applying the holographic relation
$c\sim M_{p}^{d-1}$ to the $d=4$ case, a putative CFT dual to AdS5 pure
(super)gravity has a (super)conformal primary spectrum comprised solely of
(super-)stress tensor composites up to

$\Delta_{*}\sim c^{1/3}\,.$ (4.6)

Ruling out AdS5 (super)gravity thus amounts to proving that a dual CFT must
have a non-stress tensor single-trace primary parametrically below this
threshold. We return now to our result (4.3). For the range $z<{3\over 2}$, we
have ruled out AdS5 pure supergravity as the asymptotic limit, violating the
gap condition by an infinite tower of sub-Planckian states. Stated conversely,
a putative CFT dual to AdS5 pure supergravity must have central charge scaling
faster than $c\sim r^{3/2}$ as $r\rightarrow\infty$.888The threshold $c\sim
r^{3/2}$ is also notable from a different perspective: it is the scaling for
which $\delta c_{r}:=c_{r+1}-c_{r}\sim M_{p}$. The possible significance of
this comment will be motivated elsewhere [45].

There is a sensible analogy to be drawn here with the modular bootstrap in 2d
CFTs at large central charge [46]. In unitary 2d CFTs, the state-of-the-art
upper bound on the first non-vacuum Virasoro primary dimension at
$c\rightarrow\infty$ is [47]

$\Delta_{*}\lesssim{c\over 9.1}\,.$ (4.7)

This sits between the classical threshold at $\Delta\approx{c\over 12}$ for
small BTZ black holes and the onset of universal Cardy thermodynamics at
$\Delta\approx{c\over 6}$ [48]. Similarly, our bound (3.10) for 4d
$\mathcal{N}=2$ SCFTs sits between the Planck scale threshold at $\Delta\sim
c^{1/3}$ for small black holes and the onset of universal AdS5-Schwarzchild
thermodynamics at $\Delta\sim c$. This comparison (meant only to guide the
mind) cannot be made apples-to-apples because of the aforementioned
differences in dimensional analysis. Relatedly, we have chosen to phrase the
CFT2 result more optimistically, in terms of thermodynamics instead of horizon
size, because the continuum of geometrically large BTZ black holes, which have
horizon radius $r_{h}\gtrsim{\cal O}(1)$ in AdS units, have
$\Delta\approx{c\over 12}(1+r_{h}^{2})$, which is well below (4.7). But the
bounds are similarly situated with respect to bulk AdS black hole spectra. We
expect both bounds (3.10) and (4.7) to be sub-optimal.

Indeed, there are early indications of much stronger bounds on Coulomb branch
data of $\mathcal{N}=2$ SCFTs at $c\rightarrow\infty$ that could rule out pure
supergravity unconditionally [49]. As noted earlier, $r$-tuples are subject to
a host of intricate number-theoretic constraints implied by Coulomb branch
geometry. Their systematic exploration at $r\rightarrow\infty$ is an
intriguing and well-defined open problem in $\mathcal{N}=2$ SCFT. As a first
step in that direction, one can use these constraints to rule out pure
supergravity for a specific sequence of large rank SCFTs. In particular,
consider a sequence of SCFTs of rank

$r={1\over 2}\prod_{i=1}^{N}(p_{i}-1)$ (4.8)

with $N\in\mathbb{N}$, where $p_{i}$ is the $i$’th prime number. These are the
ranks for which $\Delta_{\rm max}$ is an available Coulomb branch generator
dimension. Then one can show that if such an $r$-tuple contains
$\Delta_{r}=\Delta_{\rm max}$, it must also contain a dimension which grows
slower than any power of $c$ at $c\rightarrow\infty$:

$\Delta_{r}=\Delta_{\rm max}\quad\Rightarrow\quad\Delta_{1}\ll
c^{\varepsilon}\leavevmode\nobreak\ \leavevmode\nobreak\
\forall\leavevmode\nobreak\ \leavevmode\nobreak\ \varepsilon>0\,.$ (4.9)

This excludes a pure supergravity limit for this sequence of SCFTs by a wide
margin. How representative this is of generic paths out to
$c\rightarrow\infty$ in the space of $\mathcal{N}=2$ SCFTs touches on a subtle
question in general abstract CFT. Regardless, (4.9) indicates the potential
for a host of hidden Coulomb branch constraints in the generic case, perhaps
strong enough to rule out pure supergravity or even any degree of Planckian
scale separation.999We expect even more potential for progress if we combine
Coulomb and Higgs branch physics. See [50] for an approach to classification
of $\mathcal{N}=2$ SCFTs that does so to impressive effect. As noted there, as
$r\rightarrow\infty$, the degree $d$ of the modular differential equation
governing the Higgs branch VOA seems to scale at least linearly with the rank.
Finding an emergent “effective” approach at large $r$ and large $d$ poses a
difficult but interesting challenge.

Ultimately, we would like to understand more deeply the physical mechanism
underlying our result, in hopes of extending beyond 4d $\mathcal{N}=2$ SCFTs.
Analytic bootstrap bounds with a bearing on the AdS scale separation question
may be more accessible than previously thought.

## Acknowledgments

We thank Mario Martone, Leonardo Rastelli and Ben Smith for helpful
discussions and related collaboration, as well as the participants of the KITP
conference “Gravity from algebra: modern field theory methods for holography”
where this work was first presented. This research was supported by ERC
Starting Grant 853507, and in part by the National Science Foundation under
Grant No. NSF PHY-1748958.

##  Appendix A The Appendix

We give a short discussion of some issues in defining “AdSd+1 pure
(super)gravity” in $d>2$.

In the case of AdS3 gravity, BTZ black holes exist universally above a
threshold $\Delta_{*}\approx{c-1\over 12}$. In a putative theory of pure
gravity, this result is believed to receive corrections non-perturbatively in
$1/c$ [51], but is valid to all orders in the semiclassical $1/c$
expansion.101010One way to see this is to note that gravitational higher-
derivative corrections to the three-dimensional Einstein-Hilbert action can be
removed by field redefinitions, to any order in perturbation theory. This
implies that the black hole threshold is invariant in a derivative expansion
upon using the suitably renormalized relation between $G_{N}$ and $c$ [52].
The “-1” accounts for graviton loops. In contrast, there is no universal sharp
transition in higher dimensional gravity. The essential difference is seen by
dimensional analysis: since $E_{\rm BH}\sim{\cal O}(M_{p}^{d-1})$ in AdSd+1,
there is a parametric gap between the Planck scale and the large black hole
threshold when $d>2$.

In the main text we recalled some of the myriad objects living at energy
scales $M_{p}\lesssim E\lesssim E_{\rm BH}$ in string/M-theory. The physics at
these scales is non-universal for both IR and UV reasons:

  * •

Some configurations (e.g. small and/or hairy black holes, other admixtures in
thermal equilibrium) depend on the low-energy matter content and/or
compactification details.

  * •

Others (e.g. D-branes, giant gravitons, extra-dimensional constructions of
microstate geometries) encode details of the UV completion. But so far, as we
lack a proof of string universality, none of these is known to be required in
a UV completion of Einstein (super)gravity.

  * •

Finally, there are states that we believe to exist, about which little is
known microscopically (e.g. the microscopic description of Planckian states in
AdS compactifications of M-theory, dual to maximally supersymmetric CFTs).

For these reasons, $E_{\rm BH}$ is the threshold for universal high-energy
states in semiclassical Einstein gravity. This is a sense in which the bound
(4.3) is notable. Still, effective field theory suggests $M_{p}$ as a natural
threshold for non-graviton states in “AdSd+1 pure gravity,” despite leaving
some room for interpretation.

Another feature of the higher-dimensional case relative to AdS3 is that it is
even less clear how to define “pure gravity” in the quantum regime, of finite
$G_{N}$. The topological aspect of AdS3 gravity allows for exact computations
in certain cases (e.g. chiral gravity partition functions [53], or on-shell
partition functions of fixed topology saddles in Einstein gravity [54]) which
one may attempt to leverage into a (partial) definition of, or toy model for,
a quantum theory. Such a theory would be holgraphically dual to a CFT at
finite central charge $c$ which extremizes bootstrap constraints – a statement
which can in principle be made precise. In higher dimensions, gravity is non-
topological, and we know of no analogous simplifications. Moreover, the
higher-dimensional theories have more parameters than just the central charge
$c$. In $d=4$, say, what values should $a/c$ be “allowed” to take in a CFT
dual to pure quantum gravity? This point is better phrased with reference to a
geometric limit: there is apparent freedom in how $a/c$ is allowed to vary
along a sequence of CFTs which approaches pure gravity, and hence $a\approx
c$, asymptotically as $c\rightarrow\infty$. Similar remarks apply to $t_{4}$,
the coefficient of the third independent tensor structure in the stress tensor
three-point function $\langle TTT\rangle$. (In classical Einstein gravity,
$t_{4}=0$.) We note that imposing maximal supersymmetry in $d=4$ fixes $a=c$
and $t_{4}=0$, partially fixing these apparent freedoms, though this
simplification fails in $d=6$, in which $a/c$ is not fixed even with maximal
supersymmetry (e.g. for the known $A_{N-1}$ (2,0) SCFT, $a=4N^{3}-3N-1$ and
$c={16\over 7}N^{3}-{9\over 7}N-1$).

## References

  * [1] S. Kachru, R. Kallosh, A. D. Linde, and S. P. Trivedi, De Sitter vacua in string theory, Phys. Rev. D 68 (2003) 046005, [hep-th/0301240].
  * [2] O. DeWolfe, A. Giryavets, S. Kachru, and W. Taylor, Type IIA moduli stabilization, JHEP 07 (2005) 066, [hep-th/0505160].
  * [3] V. Balasubramanian, P. Berglund, J. P. Conlon, and F. Quevedo, Systematics of moduli stabilisation in Calabi-Yau flux compactifications, JHEP 03 (2005) 007, [hep-th/0502058].
  * [4] D. Lüst, E. Palti, and C. Vafa, AdS and the Swampland, Phys. Lett. B 797 (2019) 134867, [arXiv:1906.05225].
  * [5] O. Aharony, Y. E. Antebi, and M. Berkooz, On the Conformal Field Theory Duals of type IIA AdS(4) Flux Compactifications, JHEP 02 (2008) 093, [arXiv:0801.3326].
  * [6] S. de Alwis, R. K. Gupta, F. Quevedo, and R. Valandro, On KKLT/CFT and LVS/CFT Dualities, JHEP 07 (2015) 036, [arXiv:1412.6999].
  * [7] J. P. Conlon and F. Quevedo, Putting the Boot into the Swampland, JHEP 03 (2019) 005, [arXiv:1811.06276].
  * [8] J. P. Conlon, S. Ning, and F. Revello, Exploring The Holographic Swampland, arXiv:2110.06245.
  * [9] F. Apers, M. Montero, T. Van Riet, and T. Wrase, Comments on classical AdS flux vacua with scale separation, arXiv:2202.00682.
  * [10] F. Apers, J. P. Conlon, S. Ning, and F. Revello, Integer conformal dimensions for type IIa flux vacua, Phys. Rev. D 105 (2022), no. 10 106029, [arXiv:2202.09330].
  * [11] F. Apers, Aspects of AdS flux vacua with integer conformal dimensions, JHEP 05 (2023) 040, [arXiv:2211.04187].
  * [12] T. Coudarchet, Hiding the extra dimensions: A review on scale separation in string theory, Phys. Rept. 1064 (2024) 1–28, [arXiv:2311.12105].
  * [13] J. Polchinski and E. Silverstein, Dual Purpose Landscaping Tools: Small Extra Dimensions in AdS/CFT, pp. 365–390.  8, 2009.  arXiv:0908.0756.
  * [14] L. F. Alday and E. Perlmutter, Growing Extra Dimensions in AdS/CFT, JHEP 08 (2019) 084, [arXiv:1906.01477].
  * [15] R. Gopakumar, E. Perlmutter, S. S. Pufu, and X. Yin, Snowmass White Paper: Bootstrapping String Theory, arXiv:2202.07163.
  * [16] L. F. Alday and S. M. Chester, Pure Anti–de Sitter Supergravity and the Conformal Bootstrap, Phys. Rev. Lett. 129 (2022), no. 21 211601, [arXiv:2207.05085].
  * [17] M. Montero, M. Rocek, and C. Vafa, Pure supersymmetric AdS and the Swampland, JHEP 01 (2023) 094, [arXiv:2212.01697].
  * [18] C. Beem, M. Lemos, P. Liendo, L. Rastelli, and B. C. van Rees, The $\mathcal{N}=2$ superconformal bootstrap, JHEP 03 (2016) 183, [arXiv:1412.7541].
  * [19] M. Akhond, G. Arias-Tamargo, A. Mininno, H.-Y. Sun, Z. Sun, Y. Wang, and F. Xu, The hitchhiker’s guide to 4d $\mathcal{N}=2$ superconformal field theories, SciPost Phys. Lect. Notes 64 (2022) 1, [arXiv:2112.14764].
  * [20] F. A. Dolan and H. Osborn, On short and semi-short representations for four-dimensional superconformal symmetry, Annals Phys. 307 (2003) 41–89, [hep-th/0209056].
  * [21] C. Cordova, T. T. Dumitrescu, and K. Intriligator, Multiplets of Superconformal Symmetry in Diverse Dimensions, JHEP 03 (2019) 163, [arXiv:1612.00809].
  * [22] P. C. Argyres, Y. Lü, and M. Martone, Seiberg-Witten geometries for Coulomb branch chiral rings which are not freely generated, JHEP 06 (2017) 144, [arXiv:1704.05110].
  * [23] P. C. Argyres and M. Martone, Coulomb branches with complex singularities, JHEP 06 (2018) 045, [arXiv:1804.03152].
  * [24] A. Bourget, A. Pini, and D. Rodr∞́guez-Gómez, Gauge theories from principally extended disconnected gauge groups, Nucl. Phys. B 940 (2019) 351–376, [arXiv:1804.01108].
  * [25] P. Argyres and M. Martone, Construction and classification of Coulomb branch geometries, arXiv:2003.04954.
  * [26] M. Martone, Towards the classification of rank-r $\mathcal{N}$ = 2 SCFTs. Part I. Twisted partition function and central charge formulae, JHEP 12 (2020) 021, [arXiv:2006.16255].
  * [27] A. D. Shapere and Y. Tachikawa, Central charges of N=2 superconformal field theories in four dimensions, JHEP 09 (2008) 109, [arXiv:0804.1957].
  * [28] L. Rastelli and B. C. Rayhaun, Rationality in Four Dimensions, arXiv:2308.06312.
  * [29] D. M. Hofman and J. Maldacena, Conformal collider physics: Energy and charge correlations, JHEP 05 (2008) 012, [arXiv:0803.1467].
  * [30] P. C. Argyres and M. Martone, Scaling dimensions of Coulomb branch operators of 4d N=2 superconformal field theories, arXiv:1801.06554.
  * [31] M. Caorsi and S. Cecotti, Geometric classification of 4d $\mathcal{N}=2$ SCFTs, JHEP 07 (2018) 138, [arXiv:1801.04542].
  * [32] P. Agarwal and J. Song, Large N Gauge Theories with a Dense Spectrum and the Weak Gravity Conjecture, JHEP 05 (2021) 124, [arXiv:1912.12881].
  * [33] P. Agarwal, K.-H. Lee, and J. Song, Classification of large N superconformal gauge theories with a dense spectrum, JHEP 10 (2021) 049, [arXiv:2007.16165].
  * [34] J. B. Gutowski and H. S. Reall, Supersymmetric AdS(5) black holes, JHEP 02 (2004) 006, [hep-th/0401042].
  * [35] V. Balasubramanian, M. Berkooz, A. Naqvi, and M. J. Strassler, Giant gravitons in conformal field theory, JHEP 04 (2002) 034, [hep-th/0107119].
  * [36] P. Basu, J. Bhattacharya, S. Bhattacharyya, R. Loganayagam, S. Minwalla, and V. Umesh, Small Hairy Black Holes in Global AdS Spacetime, JHEP 10 (2010) 045, [arXiv:1003.3232].
  * [37] S. Bhattacharyya, S. Minwalla, and K. Papadodimas, Small Hairy Black Holes in $AdS_{5}xS^{5}$, JHEP 11 (2011) 035, [arXiv:1005.1287].
  * [38] J. Markeviciute and J. E. Santos, Hairy black holes in AdS5 × S5, JHEP 06 (2016) 096, [arXiv:1602.03893].
  * [39] S. Choi, S. Kim, E. Lee, S. Lee, and J. Park, Towards quantum black hole microstates, JHEP 11 (2023) 175, [arXiv:2304.10155].
  * [40] I. Bena and N. P. Warner, Resolving the Structure of Black Holes: Philosophizing with a Hammer, arXiv:1311.4538.
  * [41] I. Bah and P. Heidmann, Topological Stars and Black Holes, Phys. Rev. Lett. 126 (2021), no. 15 151101, [arXiv:2011.08851].
  * [42] S. Kim, S. Kundu, E. Lee, J. Lee, S. Minwalla, and C. Patel, Grey Galaxies’ as an endpoint of the Kerr-AdS superradiant instability, JHEP 11 (2023) 024, [arXiv:2305.08922].
  * [43] I. Heemskerk, J. Penedones, J. Polchinski, and J. Sully, Holography from Conformal Field Theory, JHEP 10 (2009) 079, [arXiv:0907.0151].
  * [44] D. Anselmi, D. Z. Freedman, M. T. Grisaru, and A. A. Johansen, Nonperturbative formulas for central functions of supersymmetric gauge theories, Nucl. Phys. B 526 (1998) 543–571, [hep-th/9708042].
  * [45] E. Perlmutter.  KITP seminar and work to appear: “What is N? On the Large Scale Structure of CFT Space”.
  * [46] S. Hellerman, A Universal Inequality for CFT and Quantum Gravity, JHEP 08 (2011) 130, [arXiv:0902.2790].
  * [47] N. Afkhami-Jeddi, T. Hartman, and A. Tajdini, Fast Conformal Bootstrap and Constraints on 3d Gravity, JHEP 05 (2019) 087, [arXiv:1903.06272].
  * [48] T. Hartman, C. A. Keller, and B. Stoica, Universal Spectrum of 2d Conformal Field Theory in the Large c Limit, JHEP 09 (2014) 118, [arXiv:1405.5137].
  * [49] E. Perlmutter and B. Smith.  Work in progress.
  * [50] J. Kaidi, M. Martone, L. Rastelli, and M. Weaver, Needles in a haystack. An algorithmic approach to the classification of 4d $\mathcal{N}$ = 2 SCFTs, JHEP 03 (2022) 210, [arXiv:2202.06959].
  * [51] H. Maxfield and G. J. Turiaci, The path integral of 3D gravity near extremality; or, JT gravity with defects as a matrix integral, JHEP 01 (2021) 118, [arXiv:2006.11317].
  * [52] P. Kraus and F. Larsen, Microscopic black hole entropy in theories with higher derivatives, JHEP 09 (2005) 034, [hep-th/0506176].
  * [53] L. Eberhardt, Off-shell Partition Functions in 3d Gravity, arXiv:2204.09789.
  * [54] S. Collier, L. Eberhardt, and M. Zhang, Solving 3d gravity with Virasoro TQFT, SciPost Phys. 15 (2023), no. 4 151, [arXiv:2304.13650].


# Lipper: Synthesizing Thy Speech using Multi-View Lipreading

Yaman Kumar
Adobe
<EMAIL_ADDRESS>&Rohit Jain
MIDAS Lab, NSIT-Delhi
<EMAIL_ADDRESS>&Khwaja Mohd. Salik
MIDAS Lab, NSIT-Delhi
<EMAIL_ADDRESS>Rajiv Ratn Shah
MIDAS Lab, IIIT-Delhi
<EMAIL_ADDRESS>&Yifang Yin
NUS, Singapore
<EMAIL_ADDRESS>&Roger Zimmermann
NUS, Singapore
<EMAIL_ADDRESS>
###### Abstract

Lipreading has a lot of potential applications such as in the domain of
surveillance and video conferencing. Despite this, most of the work in
building lipreading systems has been limited to classifying silent videos into
classes representing text phrases. However, there are multiple problems
associated with making lipreading a text-based classification task like its
dependence on a particular language and vocabulary mapping. Thus, in this
paper we propose a multi-view lipreading to audio system, namely Lipper, which
models it as a regression task. The model takes silent videos as input and
produces speech as the output. With multi-view silent videos, we observe an
improvement over single-view speech reconstruction results. We show this by
presenting an exhaustive set of experiments for speaker-dependent, out-of-
vocabulary and speaker-independent settings. Further, we compare the delay
values of Lipper with other speechreading systems in order to show the real-
time nature of audio produced. We also perform a user study for the audios
produced in order to understand the level of comprehensibility of audios
produced using Lipper.

## Introduction

Human speech is bimodal in nature, with the two modalities coming from the
auditory and optical senses. An example in this regard comes from experiments
by McGurk and MacDonald in their appropriately titled paper, “Hearing lips and
seeing voices” (?), where they show that subjects when shown mouth images
speaking _/ga/_ but with the sound of _/ba/_ , perceived it as _/da/_.
However, the recent work in lipreading domain has decoupled the visual signals
from the auditory ones. Instead, most of the lipreading projects treat this
problem as a classification task where they consider speech videos from a
restricted vocabulary of a particular language. Then, models are trained to
classify those videos into a fixed number of classes made up of that limited
vocabulary. However, scaling that approach to multiple languages and a
complete vocabulary is a difficult task. In addition, humans do not always
speak valid statements. Meaningless, gibberish or non-language and vocabulary-
conformant speech (for instance, a human speaker making animal sounds) cannot
be modeled using a restrictive approach like a classification model. Thus,
with this in mind, we present a model, namely Lipper, which given a silent
video consisting of lip-movements reconstructs the speech of the speaker. It
does this by modeling lipreading as a regression rather than a classification
task.

### Distinction between Speech-reconstruction, Speech-reading, and Speech-
recognition systems

Lipper is a _speech reconstruction_ model. People can easily get confused
between speech reconstruction, reading and recognition systems. The only
commonality amongst all of these systems, pertaining to our work, is that
lipreading has been shown to be effective in all the three tasks.

Taking the case of speech recognition systems, they perform the task of
identifying the speaker of a speech, and therefore, are not related to this
work. However, lipreading based speech-reading and reconstruction systems do
share some common features. Thus, with this in mind, in explaining this work,
we focus on speech-reading and speech-reconstruction systems _only_.

While on one hand, speech-reading systems involve identifying what a person
says (using text as the identification metric), on the other hand, the
objective of speech-reconstruction systems is to generate the _speech_ of a
person (using audio as the output generated). However, generation of the
speech using reconstruction systems may _not_ involve identifying _what_ a
person says. For instance, speech can be generated even for illegible tokens
(such as any random permutation of characters), but due to the vocabulary
dependence of speech-reading models, it becomes difficult to identify these
illegible tokens. Consider another example where this might be useful,
multilingual people often speak in code-switched languages (for e.g., in
India, a code-switched version of “I was in London the last month”, can be, “
_Pichle mahine_ , I was in London”). For identifying that kind of speech, a
speechreading system has to consider all the possible combinations of words in
both the languages and then identify them. This quickly becomes un-scalable as
the number of languages and the size of each vocabulary increase.

The reason for the language and vocabulary independence of speech
reconstruction models like Lipper is that for the generation of a sound, lip,
nose, throat and other movements are required, but not the vocabulary (or
language) per se. Thus, following this reasoning, one may directly translate
and map lip movements to speech without referencing word or sentence mappings.
This is what Lipper does. Additionally, unlike other models which have to wait
for the complete sentence to get over before they can start speechreading,
Lipper can begin to produce audio as soon as it detects some lip-movements,
thus becoming a near real-time system. Since it is not contingent on any
particular language or vocabulary mapping, thus it is a _language and
vocabulary independent_ model.

In this paper, we present an exhaustive set of experiments on Lipper for
speech-reconstruction. Lipper can take multiple views into account and then
lipread them to produce sound. First, in Section Speaker-Dependent Results, we
present thorough experiments for exploring the quality of speech-
reconstruction on all possible combinations of all the views. Second, in
Section Speaker-Independent Results, using the best-view combination, we
explore the results of Lipper for speaker-independent settings. Third, in
Section Speaker Dependent OOV Results, we note the results of Lipper for out-
of-vocabulary (OOV) phrases. Fourth, in Section Comparison of Delays, we
compare the delays for speech-reconstruction and speech-reading systems thus
demonstrating the difference of time taken between them. Fifth, in Section
User-Study, we do a user study for the speech-comprehensibility of the sound
generated. Sixth, in Section Demonstration of Reconstructed Audios, we provide
links to some of the videos which show speaker-dependent and speaker-
independent results produced using Lipper. Seventh, though the audios produced
by Lipper might be noisy in some cases, but they do capture and contain the
content of the speech of a speaker. Thus, in order to show that, we present
text-classification results in Section Text-Prediction Model on the encoded
audios produced. In the end, before concluding the paper, we present the
future research directions in lipreading domain.

Figure 1: End-to-end diagram for Lipper

## Related Work

Despite much research in lipreading domain, it is still seen as a
classification task in which, given some silent videos, a model has to
classify those videos into a limited and fixed size of lexicon (?; ?; ?; ?; ?;
?; ?; ?; ?). There have also been a few works on speech-reconstruction as well
(?; ?; ?; ?). However, the problem of view and pose-variation has been dealt
by a very few lipreading systems (?).

Most of the authors have worked on frontal-view lipreading only. Lipreading on
just frontal view is a major problem since a speaker cannot be expected to
always face the camera while speaking. In the speechreading domain, there have
been a few works which have worked on some views other than the frontal view
(?; ?; ?; ?; ?) but dealing with pose-variation is still a challenge. In
addition, the problem is compounded by the absence of multi-view datasets. A
very limited number of datasets exist for fostering research in multi-view
lipreading. One of such datasets is Oulu-VS2 (?) which provides five different
views of speakers shot concurrently. On this dataset, combination of multiple
poses was tried for speechreading by (?) and for speech-reconstruction by (?;
?). Given visual feeds from multiple cameras, the authors showed that
combining multiple views would result in better accuracy in speechreading (?;
?; ?; ?) and better speech quality (?; ?) for speech-reconstruction.

Another task in lipreading domain is dealing with pose-variation. Usually,
different models are made for dealing with different poses (?; ?; ?; ?; ?).
The other approach for dealing with pose-variation is to extract pose-
invariant features and then use them in speech-reading (?; ?; ?; ?). However,
the chief limitation of these systems is their low accuracies which prevents
their usage. There have been very few works on speech reconstruction using
single view visual feed (?; ?). However, as noted earlier, neither were the
systems tested on pose variation, nor speaker-independent settings or on
multiple views. We show Lipper’s performance on pose-invariant multi-view
speech-reconstruction.

## Lipper: Design and Development

In this section, we describe the architecture of Lipper (as shown in Figure
1). Primarily, Lipper is composed of a view classifier followed by a
STCNN+BiGRU (a combination of Spatio-Temporal Convolutional Neural Network and
Bidirectional Gated Recurrent Units) network (as shown in Figure 2). As shown
in the diagram, the view-classifier takes input from multiple cameras, and
then, maps the speaker view (can be from frontal view, i.e., 0∘ to profile
view, i.e., 90∘) to the nearest pose from the pose-set {0∘, 30∘, 45∘, 60∘ and
90∘}. Once this has been mapped, based on the view mapping provided by the
classifier, the decision logic decides on two issues:

  1. 1.

Which view combinations to utilize. This is based on the experiments shown in
Section Speaker-Dependent Results. The decision logic may or may not decide to
utilize all the available data.

  2. 2.

Based on the above decision, it chooses the appropriate speech-reconstruction
model which takes the multi-view visual input feed and generates speech.

The system generates two types of outputs: audio and associated text. The
audio is generated after decoding the encoded audio produced using the neural
network. The text is generated by taking encoded audio as input in another
neural network which performs classification of the encoded audio into
predefined categories.

### Classifier Model

True ∘ / Pred. ∘ | 0∘ | 30∘ | 45∘ | 60∘ | 90∘
---|---|---|---|---|---
0∘ | 1546 | 13 | 1 | 0 | 0
30∘ | 76 | 1399 | 78 | 7 | 0
45∘ | 3 | 19 | 1500 | 38 | 0
60∘ | 0 | 0 | 20 | 1538 | 2
90∘ | 0 | 0 | 0 | 2 | 1558
Table 1: Confusion matrix for the view classifier

The classifier model uses transfer learning to classify lip-poses. It consists
of a VGG-16 model (?) pretrained on ImageNet images followed by one dense
layer with 1024 units and then by one softmax layer with five units. The
VGG-16 model helps in extracting the visual features from the lip region
images. Since this is a multi-class classification problem, we use the cross-
entropy loss to train the system. A visual representation of the
classification model is given in Figure 3. For training the model, we used
lip-region images of size 224x224. While training, we use the batch size as
100 and then we train the system for 30 epochs with Adam optimization. The
confusion matrix of the classifier is given in the Table 1. For training and
testing, we used a uniform class distribution with equal number of samples
from each of the classes. The overall accuracy as calculated from Table 1 is
96.7%.

Figure 2: STCNN and BiGRU based architecture used for speech reading and
reconstruction  Figure 3: View-classifier model for Lipper. It classifies lip
region images into five categories from the set {0∘, 30∘, 45∘, 60∘ and 90∘}

### Speech-Reconstruction Model

STCNN+BiGRU network helps Lipper to deal with video based data. STCNN layers
extract the visual features while BiGRU layers help it to take care of
temporal nature of the data. For the audio features, we use Linear Predictive
Coding (LPC) (?) for representing audio. LPC order was found out to be optimal
at 24. Line Spectrum Pairs (LSPs) (?) can represent LPC coefficients in a
quantization robust manner. The network takes input images of lip-region of
size 128x128 and produces the output as LPC+LSP encoded audio. The network
consists of 7 layers of STCNN (of size 32, 32, 64, 64, 128 and 128
respectively) followed by 2 layers of BiGRU (of size 64 and 32 respectively).
This is finally followed by the output layer of size 50 which produces the
encoded audio.

We use 60 epochs for training and 20 epochs for fine-tuning the network. We
first sample audio at a sampling rate of 20,000 and then encode it using LPC
order of 24. This is then fed to the network along with the image sequence in
timesteps of 5 during the training time.

For different experiments Lipper’s training happens in different formats:

  1. 1.

For speaker dependent experiments, Lipper was trained on all possible
combination of views (5 views available in total, thus
$\binom{5}{1}$+$\binom{5}{2}$+$\binom{5}{3}$+$\binom{5}{4}$+$\binom{5}{5}$
possible combinations) for each individual speaker such that out of three
phrases for a given speaker in the database (?), it was trained on two and one
was kept for testing the system. The results corresponding to that are given
in Section Speaker-Dependent Results.

  2. 2.

For speaker independent settings, the best view combination as obtained from
the experiments above was taken. The model then was trained on all but two
speakers for all their phrases. Then, it was tested on the two speakers left
out for the speech generated. The results corresponding to that are given in
Section Speaker-Independent Results.

  3. 3.

For out-of-vocabulary settings, Lipper was trained individually for all the
speakers. The training strategy followed was such that it was trained ten
times and in each iteration, one of the ten phrases was left out of the
training data, and was included in the test data. Thus, in each iteration,
Lipper was tested on a phrase which it had never seen. The results
corresponding to that are given in Section Speaker-Dependent OOV Results.

### Text-Prediction Model

The text-prediction model takes encoded audio of the reconstructed speech as
input and classifies those encoded audios into text classes. For collecting
the input for this model, we used a pre-trained speech-reconstruction
STCNN+Bi-GRU network and then obtained the output of all the silent videos
present in the database.

The network has four fully connected layers (with sizes as 1000, 500, 100 and
10) with dropout (0.5) after the layers with sizes 500 and 100. The loss
function used was cross-entropy loss and the optimizer as Adam. The network
was trained with batch size as ten and number of epochs as twenty.

For making a text predicting model, we tried two different train-test data
configurations:

  1. 1.

In the first configuration, we randomly divided all the encoded audios of all
the speakers into train, test and validation data with the ratio as (70, 10
and 20) respectively.

  2. 2.

In the second configuration, we took all the encoded audios of 70% of the
speakers as the training data, and divided the rest 30% speakers’ audios into
10% and 20% to be used as validation data and test data respectively.

The results for both these configuration compared with those of (?) (which is
the state-of-the-art speachreading model on OuluVS2) are presented in Table 2.
We present the best accuracy reported in the paper by (?) for comparison. As
shown in the table, data configuration-1 performs much better than even the
best-performing model of (?). We believe this is due to the train-test
configuration of Lipper itself. Randomly classifying the video into one of the
ten classes present in the database would have led to 10% accuracy. Thus, even
considering the second configuration, the accuracy of the text-prediction
model is non-trivial. This implies that the audios produced by Lipper,
although might be noisy in some cases, capture the content of the speech of a
speaker.

| Config-1 | Config-2 | (?)
---|---|---|---
Accuracy | 97.0 | 78.5 | 95.6

Table 2: Accuracy of the two configurations of the text-prediction model
trained compared with the best results as reported in (?)

## Evaluation

### Database

For training and testing Lipper, we use all the speakers of OuluVS2 database
(?) for speech-reconstruction purposes. OuluVS2 is a multi-view audio-visual
dataset with 53 speakers of various ethnicities like European, Indian, Chinese
and American. These 53 speakers speak 10 phrases with five cameras recording
them simultaneously from five different angles. The angles considered are {0∘,
30∘, 45∘, 60∘ and 90∘}. The speakers speak at different pace and also stop in
between while speaking. A listing of all the phrases is given in Table 3.
Thus, this dataset serves well for the task at hand. This database has been
used in other similar studies as well (?; ?; ?; ?; ?; ?; ?; ?).

Table 3: List of phrases uttered by speakers in OuluVS2

S.No. | Phrases
---|---
1. | Excuse me
2. | Goodbye
3. | Hello
4. | How are you
5. | Nice to meet you

S.No. | Phrases
---|---
6. | See you
7. | I am sorry
8. | Thank you
9. | Have a good time
10. | You are welcome

### Evaluation Metric

We choose Perceptual Evaluation of Speech Quality (PESQ) (?) as the evaluation
metric to judge the quality of the sound. This metric has been used by other
speech-reconstruction works as well (?; ?; ?; ?). PESQ is a ITU-T recommended
standard for evaluating speech quality of 3.2 kHz codecs (?). For comparing
two audios, PESQ first level aligns them, then after passing them through
filters time aligns them. It then passes the audios through auditory transform
and finally extracts two distortion parameters which denote the difference
between the transform of the two signals. Finally, these signals are
aggregated in frequency and time, and are mapped to a MOS (mean opinion
score). The range of PESQ varies from -0.5 to 4.5, where speech quality
increases with increasing score.

## Results

This section presents the results obtained for speaker-dependent, speaker-
independent, out-of-vocabulary settings for Lipper. Additionally, it also
reports the results for delay measurements on Lipper.

### Speaker-Dependent Results

#### Speaker Dependent Single-View Results

Table 4: Mean readings for single-view PESQ scores for Lipper and (?)  Views | 0 degree | 30 degree | 45 degree | 60 degree | 90 degree
---|---|---|---|---|---
Lipper | 2.002 | 1.750 | 1.642 | 1.744 | 1.804
(?) | 1.72 | 1.57 | 1.48 | 1.46 | 1.52

The results for Lipper when trained and tested on single-view visual feeds are
shown in Table 4. We have compared our results with (?) which train a similar
speech-reconstruction network and tested it on single-view visual feed. As can
be seen from the table, our results are better for all the views than (?) but
the general trend of PESQ scores is similar in both the systems. In both the
models, frontal view outperforms all other views and obtains a PESQ score of
2.002 and 1.72 respectively.

#### Speaker Dependent Multi-View Results

Table 5 presents the PESQ scores for all the possible two-view combinations.
As can be seen, the combination of 0∘ and 45∘ outperforms all the other
possible combinations. Closely following it is the combination of 0∘ and 30∘.
It should also be noted that the PESQ scores, in general, for all the possible
views have been benefited after a combination with some other view. For
instance, 30∘ in combination with 0∘, experiences a gain in PESQ by over 6%.
Thus with respect to placement of two cameras, in regards to obtaining best
quality of audios, one should place the cameras at an angle of 45∘ between
them. This, as shown by the table, would produce an audio which would carry
the maximum quality.

Table 5: Mean readings for double-view PESQ scores

View Union | PESQ
---|---
0∘+30∘ | 2.125
0∘+45∘ | 2.130
0∘+60∘ | 1.952
0∘+90∘ | 1.982
30∘+45∘ | 1.991

View Union | PESQ
---|---
30∘+60∘ | 1.842
30∘+90∘ | 2.021
45∘+60∘ | 1.960
45∘+90∘ | 1.930
60∘+90∘ | 1.920

Table 6 presents the mean PESQ scores for all possible three view
combinations. The views combination 0∘, 45∘ and 60∘ outperforms all the
combinations and presents the best results obtained till now. This is a
stupendous increase of over 32% for both 45∘ and 60∘ when considering their
single view PESQ scores only. Moreover, even while considering the combination
of 45∘ and 60∘, their association with 0∘ leads to a non-trivial increase of
more than 18%. Although, it can be noted that not all possible 3 view
combinations experience a gain over their 2 view counterparts. This might be
because of less training data available due to which a larger network could
not be trained appropriately.

Table 6: Mean readings for triple-view PESQ scores

View Union | PESQ
---|---
0∘+30∘+45∘ | 1.975
0∘+30∘+60∘ | 2.112
0∘+30∘+90∘ | 2.005
0∘+45∘+60∘ | 2.315
0∘+45∘+90∘ | 1.814

View Union | PESQ
---|---
0∘+60∘+90∘ | 1.987
30∘+45∘+60∘ | 1.931
30∘+45∘+90∘ | 1.903
30∘+60∘+90∘ | 1.965
45∘+60∘+90∘ | 1.838

Results for all possible 4-view combinations are shown in Table 7. In most
cases, there is not a major increase in PESQ scores from the three-view
combinations or in some cases, even a decrease in the scores is observed. As
has been stated above, the reason for this decline in performance can be due
to non-availability of adequate data for training the larger network. However,
some view combinations have better scores than their individual view
counterparts.

Table 8 shows the result obtained on all-view combinations. As can be seen,
although the network at this stage outperforms all of its single view
counterparts but does not perform as good as the best possible three view
combination of 0∘, 45∘ and 60∘. We have not compared 2-views, 3-views, 4-views
and 5-views combinations with models by other authors since as mentioned in
the Section Related Work, there were no models previously who have worked on
combinations of multiple views.

Table 7: Mean readings for quadruple-view PESQ scores View Union | PESQ
---|---
0∘+30∘+45∘+60∘ | 2.11
0∘+30∘+45∘+90∘ | 1.916
0∘+45∘+60∘+90∘ | 2.147
0∘+30∘+60∘+90∘ | 2.071
30∘+45∘+60∘+90∘ | 1.948
Table 8: Mean reading for all-view PESQ scores Combination of views | Mean PESQ scores
---|---
0∘+30∘+45∘+60∘+90∘ | 2.086

### Speaker Dependent OOV Results

Table 9: Mean readings for single view PESQ on the OuluVS2 database for out of vocabulary (OOV) phrases  Phrases | Mean PESQ scores
---|---
Excuse Me | 1.79
Goodbye | 1.66
Hello | 1.82
How are you | 1.84
Nice to meet you | 1.57
See you | 1.64
I am sorry | 1.68
Thank you | 1.55
Have a good time | 1.46
You are welcome | 1.60

One of the major strong points for any speech reconstruction system is their
ability to reconstruct speech on phrases which were not present in the
training set. For a system which treats lipreading as a classification task,
this is not possible since essentially, these systems have to mark any video
with the limited classes that they consider. However, human language (or, in
general, sounds made by humans) presents a very wide vocabulary and cannot be
modeled easily with speechreading models.

In the Table 9, we present the PESQ scores using the overall best model (0∘,
45∘ and 60∘ combination), for each of the phrases from the Table 3 considered
as out-of-vocabulary in different iterations. The OOV PESQ scores though
lesser than their speaker-dependent counterparts are not inconsequential.

### Speaker Independent Results

This section presents the speaker independent results obtained using Lipper.
We do not evaluate the speaker independent results on every combination of
multiple views. We choose those combinations which prove to be the best in
speaker dependent settings. The results for the male and female speakers
(Speakers 38 and 39, respectively), are presented in the Table 10. It can be
observed that the results for speaker dependent models are significantly
better than the speaker independent one. We believe this is so since Lipper
does not only depend on lip movements of individual speakers but also their
voices. The lip-movements of the speakers although are different, but carry
the commonality of movement while speaking the same words. However, since
Lipper depends not only on lip-movements but also voices of the speakers and
since the voice of each speaker is different, thus the model does not perform
well on the PESQ score evaluation for unknown speakers. Thus, in speaker
independent settings, Lipper is not able to learn the person-specific voice
features which are crucial for getting high scores using PESQ as the
evaluation metric. This can explain the noticeable difference between the
speaker dependent and independent results obtained using Lipper. Additionally,
it can also be noted that results for male speaker are better than the female
one, we believe this is so since the number of male speakers in the dataset
are in a majority thus forming a bias for Lipper in favour of male voice
generation.

### Comparison of Delays

One of the major advantages of Lipper is it being a near real-time system. In
this section, we compare the end-to-end delay in getting speech from Lipper
and a similar work of that of speechreading by (?) thus confirming the
validity of Lipper being a small delay speech-reconstruction system. The
comparison of delay values is reported in Table 11. It is to be noted that the
delay values for speechreading work of (?) depend on the phrase spoken, longer
the phrase higher is the delay.

Table 10: Readings on the best-view combination PESQ scores for speaker-independent settings View Union | Male | Female
---|---|---
0∘ | 1.90 | 1.76
0∘+45∘ | 2.03 | 1.85
0∘+45∘ \+ 60∘ | 1.94 | 1.86
0∘+45∘ \+ 60∘ \+ 90∘ | 1.91 | 1.82
0∘+30∘\+ 45∘ \+ 60∘ \+ 90∘ | 1.91 | 1.83
Table 11: Mean readings for Delay values (in secs) on Lipper and (?)  Video | Lipper(s) | Petridis et al.(s)
---|---|---
Excuse Me | 0.169 | 1.26
Goodbye | 0.169 | 0.94
Hello | 0.169 | 0.98
How are you | 0.169 | 1.24
Nice to meet you | 0.169 | 1.4
See you | 0.169 | 1.34
I am sorry | 0.169 | 1.44
Thank you | 0.169 | 1.09
Have a good time | 0.169 | 1.61
You are welcome | 0.169 | 1.95

### User-Study

Table 12: User studies for the reconstructed audios Study | Accuracy | Variance
---|---|---
Audio-only | 80.25 | 2.72
Audio-visual | 81.25 | 1.97

Although PESQ is a standard numeric measure which can give an idea of the
virtue of a speech and can give a reference metric for comparison but from our
experiments, it was observed that the measurements produced by PESQ were not
always perfect. Even for some noisy audios, the PESQ scores were high and the
vice-versa was also found out to be true. Due to these observations, a user
study was done to understand the intelligibilty of the audios. Two types of
user studies were done for doing this analysis:

  1. 1.

In the first study, we gave reconstructed speech of 25 speakers to 10
different annotators who were given 4 options consisting of 1 true and 3
random options (amongst the 10 classes). Each annotator was asked to listen to
the audios produced by the model as many times as he likes and then choose the
best option amongst the 4 classes given.

  2. 2.

In the second study, in accordance with a real video-conference environment,
we showed the speaker’s videos to the annotators with the reconstructed audio
playing along with the video. Then, the annotators were asked to choose among
10 phrase classes for the audio-video sequence that they just listened to.

The results for both the user studies are given in Table 12. In the audio-only
study, a dice throw could get 25% of the annotations right but the annotation
accuracy turned out to be 80.25 with annotator accuracy variance value as
2.72%. In the audio-visual study, although a dice-throw would have led 10%
annotations correct, Lipper achieved an annotation accuracy of 81.25 with
inter-annotator accuracy variance as 1.97%.

### Demonstration of Reconstructed Audios

Just numeric results cannot do justice to reconstructed _speech_ output. Thus,
we have made a video listing consisting of all the reconstructed speech
outputs as part of a _Youtube_ channel. The readers are encouraged to view the
video playlist at
https://www.youtube.com/playlist?list=PL9rvax0EIUA4LNaXSeVX5Kt6gu2IBBnsg. This
playlist contains videos consisting of speech reconstructed using speaker
dependent model, speaker independent model, OOV phrases and videos of some
non-dataset speakers who speak multiple languages (Hindi, English and
Chinese). The non-dataset speakers and the languages they speak show the
language and vocabulary independence of the model. For them, we train speaker-
dependent models, and first carefully get their lip-region videos and then
reconstruct the speech using the model generated. Please use headphones to be
able to listen to the reconstructed speech better. It is worth noting that in
the demonstration111We obtain the reconstructed videos with the best 3-view
(0∘, 45∘ and 60∘ combination). The audios are played three times so that
readers can easily understand them., the audio is in complete _sync_ with the
video. In addition, the speaker’s voice is comprehensible and can be
understood.

## Conclusion and Future Work

### Future Research Directions

As explained in this paper, not much research has happened in speech
reconstruction domain. Thus, there are a lot of areas where speech-
reconstruction system can be improved.

As shown in the Section Demonstration of Reconstructed Audios, the audios are
robotic in nature. One of the main reasons for this is that voice is generated
not just using mouth, but also using nose, throat and tongue. Since Lipper
takes only lip-region into account, thus the voice generated cannot have
emotion, prosody or modulation. Therefore, speech reconstruction systems have
to work to make the audios more real-life.

Currently, the system only works in controlled environment where speakers are
not moving much and are looking into the camera at a stable angle. However, in
the real world scenario, this cannot be the case. The speakers will turn and
twist and their poses would vary dramatically, thus, going forward speech-
reconstruction has to take that into account.

In this paper, although speaker-independent settings were explored, but as was
seen, the system does not work very well on them. This is a major problem for
the deployment of speech-reconstruction systems in their use-case scenarios.

### Conclusion

In this paper, the authors proposed a real-time, language and vocabulary
independent, multi-view accounting and speaker-independent speech
reconstruction system, namely Lipper, which utilizes multi-view visual feeds
to generate the speech of a speaker. Lipper extracts features directly from
the pixels of the multi-view videos. It then learns those spatial features
jointly along with temporal features to finally reconstruct speech of a user.
The proposed system showed significant intelligibility for the audios
constructed. The best combination of views was found to be the combination of
0∘, 45∘ and 60∘. This combination produced a significant gain over other
possible views and their combinations. We also showed the experiments of out-
of-vocabulary phrases for speech reconstruction and the delay between getting
speech for speachreading and speech-reconstruction systems.

## Acknowledgement

This research was supported in part by the National Natural Science Foundation
of China under Grant no. 61472266 and by the National University of Singapore
(Suzhou) Research Institute, 377 Lin Quan Street, Suzhou Industrial Park,
Jiang Su, People’s Republic of China, 215123.

MIDAS lab gratefully acknowledges the support of NVIDIA Corporation with the
donation of a Titan Xp GPU used for this research.

## References

  * [Anina et al. 2015] Anina, I.; Zhou, Z.; Zhao, G.; and Pietikäinen, M.  2015\.  Ouluvs2: A multi-view audiovisual database for non-rigid mouth motion analysis.  In 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), 2015,, volume 1, 1–5.  IEEE.
  * [Assael et al. 2016] Assael, Y. M.; Shillingford, B.; Whiteson, S.; and de Freitas, N.  2016\.  Lipnet: end-to-end sentence-level lipreading.
  * [Chung and Zisserman 2017] Chung, J. S., and Zisserman, A.  2017\.  Lip reading in profile.  BMVC.
  * [Chung et al. 2016] Chung, J. S.; Senior, A.; Vinyals, O.; and Zisserman, A.  2016\.  Lip reading sentences in the wild.  arXiv preprint arXiv:1611.05358 2\.
  * [Cornu and Milner 2015] Cornu, T. L., and Milner, B.  2015\.  Reconstructing intelligible audio speech from visual speech features.  In Sixteenth Annual Conference of the International Speech Communication Association.
  * [Ephrat and Peleg 2017] Ephrat, A., and Peleg, S.  2017\.  Vid2speech: speech reconstruction from silent video.  In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017, 5095–5099.  IEEE.
  * [Ephrat, Halperin, and Peleg 2017] Ephrat, A.; Halperin, T.; and Peleg, S.  2017\.  Improved speech reconstruction from silent video.  In ICCV 2017 Workshop on Computer Vision for Audio-Visual Media.
  * [Estellers and Thiran 2011] Estellers, V., and Thiran, J.-P.  2011\.  Multipose audio-visual speech recognition.  In Signal Processing Conference, 2011 19th European, 1065–1069.  IEEE.
  * [Fant 2012] Fant, G.  2012\.  Acoustic theory of speech production: with calculations based on X-ray studies of Russian articulations, volume 2.  Walter de Gruyter.
  * [Itakura 1975] Itakura, F.  1975\.  Line spectrum representation of linear predictor coefficients of speech signals.  The Journal of the Acoustical Society of America 57(S1):S35–S35.
  * [Kumar et al. 2018a] Kumar, Y.; Aggarwal, M.; Nawal, P.; Satoh, S.; Shah, R. R.; and Zimmermann, R.  2018a.  Harnessing ai for speech reconstruction using multi-view silent video feed.  In 2018 ACM Multimedia Conference on Multimedia Conference, 1976–1983.  ACM.
  * [Kumar et al. 2018b] Kumar, Y.; Jain, R.; Salik, M.; ratn Shah, R.; Zimmermann, R.; and Yin, Y.  2018b.  Mylipper: A personalized system for speech reconstruction using multi-view visual feeds.  In 2018 IEEE International Symposium on Multimedia (ISM), 159–166.  IEEE.
  * [Kumar, Chen, and Stern 2007] Kumar, K.; Chen, T.; and Stern, R. M.  2007\.  Profile view lip reading.  In IEEE International Conference on Acoustics, Speech and Signal Processing, 2007. ICASSP 2007., volume 4, IV–429.  IEEE.
  * [Lan, Theobald, and Harvey 2012] Lan, Y.; Theobald, B.-J.; and Harvey, R.  2012\.  View independent computer lip-reading.  In IEEE International Conference on Multimedia and Expo (ICME), 2012, 432–437.  IEEE.
  * [Lee, Lee, and Kim 2016] Lee, D.; Lee, J.; and Kim, K.-E.  2016\.  Multi-view automatic lip-reading using neural network.  In Asian Conference on Computer Vision, 290–302.  Springer.
  * [Lucey and Potamianos 2006] Lucey, P., and Potamianos, G.  2006\.  Lipreading using profile versus frontal views.  In IEEE Workshop on Multimedia Signal Processing, 24–28.  IEEE.
  * [Lucey, Sridharan, and Dean 2008] Lucey, P. J.; Sridharan, S.; and Dean, D. B.  2008\.  Continuous pose-invariant lipreading.
  * [McGurk and MacDonald 1976] McGurk, H., and MacDonald, J.  1976\.  Hearing lips and seeing voices.  Nature 264(5588):746.
  * [Ngiam et al. 2011] Ngiam, J.; Khosla, A.; Kim, M.; Nam, J.; Lee, H.; and Ng, A. Y.  2011\.  Multimodal deep learning.  In Proceedings of the 28th international conference on machine learning (ICML-11), 689–696.
  * [Ong and Bowden 2011] Ong, E.-J., and Bowden, R.  2011\.  Learning sequential patterns for lipreading.  In BMVC 2011-Proceedings of the British Machine Vision Conference 2011.  The British Machine Vision Association and Society for Pattern Recognition.
  * [Pei, Kim, and Zha 2013] Pei, Y.; Kim, T.-K.; and Zha, H.  2013\.  Unsupervised random forest manifold alignment for lipreading.  In IEEE International Conference on Computer Vision, 129–136.  IEEE.
  * [Petridis et al. 2017] Petridis, S.; Wang, Y.; Li, Z.; and Pantic, M.  2017\.  End-to-end multi-view lipreading.  arXiv preprint arXiv:1709.00443.
  * [Recommendation 2001] Recommendation, I.-T.  2001\.  Perceptual evaluation of speech quality (pesq): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs.  Rec. ITU-T P. 862.
  * [Rekik, Ben-Hamadou, and Mahdi 2014] Rekik, A.; Ben-Hamadou, A.; and Mahdi, W.  2014\.  A new visual speech recognition approach for rgb-d cameras.  In International Conference Image Analysis and Recognition, 21–28.  Springer.
  * [Rekik, Ben-Hamadou, and Mahdi 2015] Rekik, A.; Ben-Hamadou, A.; and Mahdi, W.  2015\.  Unified system for visual speech recognition and speaker identification.  In International Conference on Advanced Concepts for Intelligent Vision Systems, 381–390.  Springer.
  * [Rix et al. 2001] Rix, A. W.; Beerends, J. G.; Hollier, M. P.; and Hekstra, A. P.  2001\.  Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs.  In IEEE Transactions on Acoustics, Speech, and Signal Processing, volume 2, 749–752.  IEEE.
  * [Saitoh et al. 2016] Saitoh, T.; Zhou, Z.; Zhao, G.; and Pietikäinen, M.  2016\.  Concatenated frame image based cnn for visual speech recognition.  In Asian Conference on Computer Vision, 277–289.  Springer.
  * [Shah and Zimmermann 2017] Shah, R., and Zimmermann, R.  2017\.  Multimodal analysis of user-generated multimedia content.  Springer.
  * [Simonyan and Zisserman 2014] Simonyan, K., and Zisserman, A.  2014\.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556.
  * [Zhou et al. 2014] Zhou, Z.; Zhao, G.; Hong, X.; and Pietikäinen, M.  2014\.  A review of recent advances in visual speech decoding.  Image and vision computing 32(9):590–605.
  * [Zhou, Zhao, and Pietikäinen 2011] Zhou, Z.; Zhao, G.; and Pietikäinen, M.  2011\.  Towards a practical lipreading system.  In IEEE International Conference on Computer Vision, 137–144.  IEEE.
  * [Zimmermann et al. 2016] Zimmermann, M.; Ghazi, M. M.; Ekenel, H. K.; and Thiran, J.-P.  2016\.  Visual speech recognition using pca networks and lstms in a tandem gmm-hmm system.  In Asian Conference on Computer Vision, 264–276.  Springer.