# Unveiling Neutrino Masses: Insights from Robust (e)BOSS Data Analysis and
Prospects for DESI and Beyond

Hernán E. Noriega  0000-0002-3397-3998<EMAIL_ADDRESS>Instituto de
Ciencias Físicas, Universidad Nacional Autónoma de México, 62210, Cuernavaca,
Morelos.  Instituto de Física, Universidad Nacional Autónoma de México, Apdo.
Postal 20-364, 01000, D.F, México.  Alejandro Aviles  0000-0001-5998-3986
<EMAIL_ADDRESS>Instituto de Ciencias Físicas, Universidad Nacional
Autónoma de México, 62210, Cuernavaca, Morelos.

###### Abstract

Recent findings from DESI BAO combined with Planck CMB data have set an upper
limit on the total neutrino mass of $\sum m_{\nu}<0.072\,\text{eV}$ (95%
confidence level), ruling out the inverted hierarchy. Indeed, methods that
rely on the background expansion of the Universe tend to suggest negative
neutrino masses. In this work, we contribute to the quest for accurately
constraining neutrino mass using cosmological probes. By conducting a full-
shape analysis on data from BOSS, eBOSS, and synthetic power spectra, we
showed that projection effects can significantly influence constraints on
neutrino mass, rendering these measurements largely unreliable. Our results
highlight the need for better techniques to measure the neutrino mass
accurately. Based on the large-scale structure suppression, we identified a
critical blind spot in the full-shape analysis. By splitting the galaxy power
spectrum into broadband and wiggles, we noticed that information on neutrino
mass is primarily extracted from the suppressed wiggles rather than broadband
suppression. This opens the possibility of developing alternative methods
based only on the wiggles of the power spectrum that can be more robust than
those heavily reliant on background evolution.

##  I Introduction

Neutrino oscillation experiments have accurately measured the squared-mass
differences between their definite mass states [1], implying that there are at
least two states of massive neutrinos. These states can be ordered either in a
normal hierarchy (NH), where the sum of their masses satisfies
$M_{\nu}\equiv\sum_{i}m_{\nu,i}>0.058\,\text{eV}$, or in an inverted hierarchy
(IH), with $M_{\nu}>0.10\,\text{eV}$. On the other hand, the most recent
results from the Karlsruhe Tritium Neutrino (KATRIN) experiment have put an
upper bound of $0.45\,\text{eV}$ at the $90\,\%$ confidence level (c.l.) on
the lightest state [2].

However, the most constraining upper bounds on the neutrino masses so far have
come from cosmology [3, 4], although indirectly, by measuring their
gravitational influence over the expansion history of the Universe and the
large-scale structure (LSS); see [5, 6] for reviews on neutrino physics in
cosmology. Recently, the Dark Energy Spectroscopic Instrument (DESI) put a
very tight constraint on the sum of the neutrino masses,
$M_{\nu}<0.072\,\text{eV}\,\,(95\,\%\,\text{c.l.})$, by combining data from
the cosmic microwave background radiation (CMB) and DESI BAO [4], assuming the
flat $\Lambda$CDM model. However, when considering evolving dark energy with
the CPL (or $w_{0}$–$w_{a}$) parametrization, the mass constraint is severely
relaxed to $M_{\nu}<0.195\,\text{eV}$ [4]. Despite the success of such
analyses, the neutrino effect is very mild, and even a small discrepancy in
the measurements of the other cosmological parameters can yield misleading
results due to degeneracies.

An aspect that has attracted attention is that the posterior profile obtained
from CMB+DESI BAO data peaks at $M_{\nu}=0$, exhibiting a behavior akin to the
tail of a distribution that finds its best-fit at negative neutrino masses.
This is not new, but was observed using CMB data alone [3], which further
constrains $M_{\nu}<0.24\,\text{eV}\,\,(95\,\%\,\text{c.l.})$, that is
improved to $M_{\nu}<0.12\,\text{eV}\,\,(95\,\%\,\text{c.l.})$ by adding BAO
data from the Baryon Oscillation Spectroscopic Survey (BOSS) DR12 galaxies
[7]. The peak profile behavior has also been observed using measurements of
clustering involving galaxies, quasars, and the Lyman-$\alpha$ forest from the
completed Sloan Digital Sky Survey (SDSS) together with CMB data [8]. The
reason behind this behavior in the posterior profile is unclear. A recent
study argues that the peak at $M_{\nu}=0$ and the remarkably tight constraints
imposed by DESI, nearly crossing the threshold $M_{\nu}<0.06\,\text{eV}$ at
$2\,\sigma$, could be a signal of new physics in the neutrino sector [9].
Similarly, this behavior could also point to modified gravity, which often
exhibits degeneracy with massive neutrinos [10, 11].

In this work, we utilize full-shape modeling of the galaxy power spectrum to
revisit the constraints on the total neutrino mass using LSS data from BOSS
DR12 galaxies [12, 7] and extended BOSS (eBOSS) quasars [13, 8]. Our goal is
to understand the sources that could lead neutrino mass constraints to favor
negative values. Specifically, this work addresses the following questions: Is
the peak at $M_{\nu}=0$ also seen in full-shape analyses of (e)BOSS data? Are
the constraints on neutrino mass affected by projection effects? If so, can
the inclusion of more data help alleviate these effects? To this end, we
analyze individual redshift bins of BOSS and eBOSS, as well as the combined
datasets. The analysis shows that the well-marked peak at $M_{\nu}=0$ is
driven by eBOSS, which surprisingly places tighter constraints on the neutrino
mass than the two non-overlapping redshift bins of BOSS, $z_{1}$ and $z_{3}$,
together. This behavior persists even with the inclusion of more information
through external priors. To gain more insight, we construct noiseless
synthetic data generated with the flat $\Lambda$CDM model using Planck 2018
best-fit values for cosmological parameters, including
$M_{\nu}=0.06\,\text{eV}$. In both cases (using real or synthetic data), we
obtain very similar profiles for the neutrino mass posteriors. This suggests
that projection effects (also known as prior volume effects) significantly
impact neutrino mass constraints, we verify their presence by comparing
frequentist and Bayesian approaches.

Given the reasons stated above, it is worthwhile to explore alternative
approaches for robustly extracting information on neutrino mass. In
particular, we are interested in approaches that primarily rely on the
suppression of the galaxy power spectrum, and not on the background evolution.
By splitting the power spectrum into BAO wiggles and broadband contributions,
we show that the information on the sum of the neutrino masses is primarily
extracted from the former, contrary to common belief. This discovery opens the
possibility of constructing methods that rely solely on extracting the
neutrino mass from the relative amplitude of the BAO wiggles. Although one
expects that such methodologies would provide looser constraints, they could
be more robust against changes in the background expansion of the Universe.

The rest of the paper is organized as follows. In sect. II we present the
full-shape modeling we use in this paper; in sect. III we show the datasets,
likelihood and priors used throughout; in sect. IV we present our numerical
results; in sect. V we delve into the wiggle-broadband decomposition. Finally,
we present our conclusions and final remarks in sect. VI.

##  II Theory

The Perturbation Theory (PT) and Effective Field Theory (EFT) for LSS has been
developed over the past decades by several authors, e.g. [14, 15, 16, 17], and
has recently proven useful for directly fitting the galaxy power spectra [18,
19] (see also [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30]). Since then, a few
codes has been released to compute the galaxy power spectrum in redshift
space: including class-pt [21], velocileptors [31, 26], PyBird [32], CLASS-
OneLoop [33], all of them utilizing Einstein-de Sitter (EdS) kernels. In this
work we compute the galaxy power spectrum using the code FOLPS111Fast 1-loop
power spectrum (FOLPS). Publicly available at
https://github.com/henoriega/FOLPS-nu. JAX implementation:
https://github.com/cosmodesi/folpsax. [34], which is based on the PT/EFT in
the presence of massive neutrinos developed in [35, 10, 34], and employs the
perturbative fk-kernels, as named in [10, 36]. These kernels maintain the
scale-dependence of the growth factors $f(k,t)=d\log D_{+}(k,t)/d\log a(t)$,
which arise from the introduction of the neutrino mass as a new scale into the
theory. The use of these kernels has been shown to provide more stringent
constraints on the sum of neutrino masses than the EdS ones under certain
settings [37]. Moreover, the computation time of loop corrections is very
fast, ensuring both efficiency and improved accuracy [34], making fk-kernels a
highly attractive alternative.

We use the galaxy power spectrum [38, 39, 40, 41, 10, 42, 26, 43]

$\displaystyle P^{\text{EFT}}_{s}(k,\mu)$
$\displaystyle=P_{\delta\delta}(k)+2f_{0}\mu^{2}P_{\delta\theta}(k)+f_{0}^{2}\mu^{4}P_{\theta\theta}(k)$
$\displaystyle\quad+A(k,\mu)+D(k,\mu)$
$\displaystyle\quad+(\alpha_{0}+\alpha_{2}\mu^{2}+\alpha_{4}\mu^{4}+\alpha_{6}\mu^{6})k^{2}P_{L}(k)$
$\displaystyle\quad+\alpha^{\rm shot}_{0}+\alpha^{\rm shot}_{2}(k\mu)^{2}.$
(1)

The first line in this equation is the non-linear Kaiser power spectrum [38,
39]. The function $A(k,\mu)$ was introduced in [40] and is obtained from
correlators of three fields, either density or velocity fields, while function
$D(k,\mu)$ correspond to correlators of four fields. Expressions for these
functions can be found in [41, 34]. The last two lines include the EFT
countertems $\alpha_{0,2,4,6}$ and stochastic noise parameters $\alpha^{\rm
shot}_{0,2}$. In this work we use the third order bias expansion of [15],
which consider bias parameters $b_{1}$, $b_{2}$, $b_{s^{2}}$ and
$b_{\text{3nl}}$.222Alternatively, FOLPS can use the bias expansion with
$b_{\mathcal{G}_{2}}$ and $b_{\Gamma_{3}}$ [44] instead of $b_{s^{2}}$ and
$b_{\text{3nl}}$. We refer the reader to [34, 36] for details in all the
expressions of the theory for the beyond EdS kernels.

Before moving to fitting the neutrino mass, we want to compare the performance
of FOLPS with previous BOSS analyses, within the framework of the flat
$\Lambda$CDM model, assuming a fixed neutrino mass, $M_{\nu}=0.06\,\text{eV}$.
In Fig. 1 we show such comparisons to PyBird [19] (D’Amico et al. 2020),
class-pt [18] (Ivanov et al. 2020) and velocileptors [45] (Chen et al. 2022).
Discrepancies in the results mainly stem from variations in nuisance
parameters and their priors.333In [46] it is shown that most of the
differences between PyBird and class-pt results previously observed in [25]
arise from weight effects due to the choice of EFT counterterms priors.

Recently, the debate on EFT model comparisons has concluded, demonstrating
that different EFT codes yield nearly identical constraints under equivalent
settings (nuisance parameters, priors, etc). This was shown in [47] using
FOLPS, velocileptors, and PyBird on LRG, ELG, and QSO mocks, as well as
noiseless synthetic data generated by EFT models, even for mocks with
extremely large volumes.

Finally, it is interesting to note from Fig. 1 that all BOSS analyses
presented with colored dots and error bars reach lower values on $A_{s}$
compared to Planck. This behavior is well-known, see e.g. [48, 49, 50, 51].
However, constraints on $A_{s}$ can be improved by sampling bias parameters
scaled by the amplitude of matter fluctuations $\sigma_{8}$, rather than
directly sampling bias parameters [52]. The results of using scaled biases
with $\sigma_{8}$, indicated by red stars and error bars, are consistent with
Planck within $1\,\sigma$. We will revisit this point in Sect.IV.

Figure 1:  Comparison of 1-dimensional credible intervals (means and 68 %
c.l.) from full-shape analyses of the BOSS power spectrum using different
pipelines [18, 19, 45], assuming a flat $\Lambda$CDM model with
$M_{\nu}=0.06\,\text{eV}$. Points (stars) with error bars represent results
from sampling bias parameters directly (scaled with powers of $\sigma_{8}$).
The gray regions (dashed lines) correspond to the 68% c.l. (best-fit) of
Planck 2018 [3].

##  III Data and likelihood

In this section we briefly introduce the datasets [53] and the analysis setup
[37] used throughout this work.

###  III.1 Datasets

We analyze the clustering of galaxies drawn from the publicity available BOSS
DR12 galaxy sample [12, 7] and eBOSS DR16 quasar sample [13, 8], which are
part of the SDSS-III and -IV projects, respectively. For the BOSS galaxy
sample, we concentrate on the two non-overlapping redshift bins
$0.2<z_{1}<0.5$ and $0.5<z_{3}<0.75$ with effective redshifts $z_{\rm
eff}=0.38$ and $z_{\rm eff}=0.61$, respectively. At high redshifts, we limited
our analysis to the eBOSS quasars dataset, covering the range $0.8<z<2.2$ with
an effective redshift of $z_{\rm eff}=1.52$.

We use the products (data power spectra multipoles, window functions, and
covariance matrices) provided in
[53].444https://fbeutler.github.io/hub/deconv_paper.html The measured power
spectra multipoles were computed assuming a fiducial flat $\Lambda$CDM
cosmology with $\Omega^{\rm fid}_{m}=0.31$, introducing artificial anisotropy
in the galaxy clustering. This anisotropy is accounted for through the Alcock-
Paczynski (AP) effect (see e.g., Section 5.1 from [37]). Besides the AP
effect, the survey geometry also impacts the measured power spectra. We
account for this by convolving the clustering signal with the corresponding
window function [53]. We estimate the numerical covariance matrices from a
suite of galaxy mocks, which match the clustering properties and geometry of
BOSS and eBOSS samples. Covariances for the BOSS DR12 galaxy sample are
extracted from $2\times 2048$ realizations of the MultiDark-Patchy mock
catalogs for NGC and SGC [54]. Meanwhile, eBOSS DR16 quasar covariances are
computed from $2\times 1000$ realizations of the EZmocks [55], also for NGC
and SGC patches.

In addition to analyzing real data from BOSS galaxies and eBOSS quasars, we
examine noiseless synthetic data generated from our theoretical model at the
same redshifts, using the Planck 2018 cosmology with $M_{\nu}=0.06\,\text{eV}$
as input. The analysis of synthetic data maintains consistency with the real
data analysis by employing the same covariance, window function, and settings.

###  III.2 Settings

Throughout this work, we fit the monopole and quadrupole adopting the wave-
number range $0.01\leq k\,[h\,{\rm Mpc^{-1}}]\leq 0.20$. Our model has been
shown to provide unbiased results within this range for mocks with BOSS-like
volume [34] and even for extremely large volumes [37, 47]. Our base setup
allows the following parameters to vary:

$\displaystyle{\rm Cosmological:}\,\\{M_{\nu},\,h,\,\omega_{\rm
b},\,\omega_{\rm c},\,\ln(10^{10}A_{s})\\},$ (2) $\displaystyle{\rm
Nuisance:}\,\\{b_{1},\,b_{2},\,\alpha_{0},\,\alpha_{2},\,\alpha^{\rm
shot}_{0},\,\alpha^{\rm shot}_{2}\\}.$ (3)

The cosmological parameters are varied over sufficiently wide ranges to be
considered uninformative, except for the baryons, for which we impose the BBN
prior $\omega_{b}=0.02237\pm 0.00037$ [56, 57]. We also keep the spectral
index fixed $n_{s}=0.9649$ at the best-fit value from Planck 2018.
Additionally, we fix the tidal bias and non-local bias according to
coevolution theory [58, 59, 60], using $b_{s^{2}}=-\frac{4}{7}(b_{1}-1)$ and
$b_{3nl}=\frac{32}{315}(b_{1}-1)$. The use of coevolution slightly narrows the
posterior distributions, but does not alter our conclusions [37].

Figure 2: Constraints from BOSS $z_{1}$ ($z_{\text{eff}}=0.31$) and $z_{3}$
($z_{\text{eff}}=0.61$) galaxies, and eBOSS quasars at redshift
$z_{\text{eff}}=1.52$. The upper panel shows the results from fitting the real
data, while the bottom panel shows the results from fitting synthetic
noiseless (synth.) data. Vertical dashed lines represent the best-fit values
from Planck 2018.

In the Bayesian framework, we analytically marginalize over the EFT and
stochastic parameters, as they enter at linear order in Eq. (II). The
marginalization process sets priors on these parameters; we choose the
agnostic option of employing uninformative flat priors on the EFT and
stochastic parameters, thereby avoiding potential prior weighting effects from
an incorrect choice of these priors [37, 46]. The marginalized Likelihood
$\mathcal{L}$ is shown in Eq. (A.9) of [37]. The complete (unmarginalized)
likelihood is employed in the frequentist profile likelihood approach. That
is, $\ln\mathcal{L}=-\tfrac{1}{2}\chi^{2}$, with $\chi^{2}\equiv\Delta
D^{T}\text{Cov}^{-1}\Delta D$ and $\Delta D$ representing the residual between
the model and the data vectors. In this method, we handle the nuisance
parameters directly, as opposed to marginalization, which is commonly used in
the Bayesian approach.

##  IV Data analysis

In this section we present our results, obtained using a flat $\Lambda$CDM
model with the total neutrino mass $M_{\nu}$ as an additional free parameter.
Our base model adopts one massive and two massless neutrino states.
Additionally, we incorporate a BBN prior on the baryon density, otherwise
stated, and set the spectral index to Planck’s best-fit value, as discussed in
Sec. III. The linear $cb$ (cold dark matter + baryon) power spectrum is
obtained from the code
CLASS555https://lesgourg.github.io/class_public/class.html [61], serving as
input for FOLPS, which calculates the 1-loop corrections and, subsequently,
the galaxy power spectrum multipoles that we compare against the data. To
explore the parameter space, we conduct Markov Chain Monte Carlo (MCMC)
samplings using the emcee code666https://emcee.readthedocs.io/ [62]. Contour
plots, 1-dimensional posterior plots, and confidence intervals are computed
using the GetDist Python package [63].

Firstly, we independently fit the BOSS $z_{1}$ and $z_{3}$ galaxies, as well
as eBOSS quasars. The corresponding constraints are presented in the upper
panel of Fig. 2. As expected BOSS, $z_{1}$ and $z_{3}$ alone have small
constraining power on the neutrino mass,

$\displaystyle M_{\nu}<1.67\,\text{eV}\quad(95\,\%,\text{BOSS $z_{1}$}+{\rm
BBN}),$ (4) $\displaystyle M_{\nu}<1.94\,\text{eV}\quad(95\,\%,\text{BOSS
$z_{3}$}+{\rm BBN}).$ (5)

However, as noticed in [64], the upper bounds imposed by eBOSS are
unexpectedly stringent,

$\displaystyle M_{\nu}<1.13\,\text{eV}\quad(95\,\%,\text{eBOSS}+{\rm BBN}),$
(6)

despite this dataset being the noisiest and having the least amount of data
among those utilized in this work. As expected, the rest of cosmological
parameters show more relaxed constraints for this dataset. Moreover, the
posterior distribution of neutrino mass peaks at $M_{\nu}=0$, resembling the
tail of a truncated distribution that would otherwise peak at negative mass
values. This peculiar pattern could be attributed to various factors, ranging
from statistical flukes to potential indications of new physics. To ascertain
whether this phenomenon persists under controlled conditions, we conduct fits
to noiseless synthetic datasets generated with our theoretical model employing
the Planck 2018 best-fit parameters, with $M_{\nu}=0.06\,\text{eV}$, while
keeping the same covariance matrices and window functions as those used with
the real data. Surprisingly, the results demonstrate striking similarities for
all cosmological parameters, with the posterior for the neutrino mass
exhibiting an almost identical profile and

$\displaystyle M_{\nu}<1.24\,\text{eV}\quad(95\,\%,\text{synthetic eBOSS +
BBN}).$ (7)

This behavior may indicate that we are observing projection effects that drive
the estimated neutrino mass toward negative values. This is plausible as it is
only observed in eBOSS data, which are inherently less constraining, and thus,
more susceptible to such prior volume effects. Indeed, in the bottom panel of
Fig. 2, we can observe small shifts between the peak of the 1-dimensional
distributions and the cosmological parameters used to construct the noiseless
synthetic data. Despite these shifts falling within $1\,\sigma$ intervals, one
might expect better performance with noiseless data, as recently observed in
[47, 37], where the distributions match the cosmological parameters, even when
using exceedingly large volumes.

Figure 3: 2-dimensional constraints in the $M_{\nu}$-$\Omega_{m}$ space (top
panel) and $M_{\nu}$-$h$ space (bottom panel). LSS refers to BOSS + eBOSS.
Planck (+BAO) refers to the results obtained from CMB data using TT, TE,
EE+lowE+lensing (+BAO from BOSS DR12).  Figure 4: 1-dimensional constraints on
$M_{\nu}$. Labels for LSS and Planck (+BAO) are described in the caption of
Fig. 3.

Projection effects are typically alleviated with the addition of more data.
Thus, we incorporate the information from the acoustic scale at the last
scattering through the parameter $100\times\theta_{*}=1.04110\pm 0.00031$ from
Planck 2018, as a Gaussian prior to our dataset. The outcomes are shown in
Fig. 3, where we can notice that $\theta_{*}$ contributes to tighter
constraints on $\Omega_{m}$ and $h$, subsequently leading to a more stringent
inference of the neutrino mass given by

$\displaystyle M_{\nu}<0.21\,\text{eV}\quad(95\,\%,\text{LSS + BBN +
$\theta_{*}$}),$ (8)

where LSS refers to the result of jointly fitting the BOSS and eBOSS data.
This result achieves slightly better constraints on the neutrino mass compared
to those obtained from CMB data by Planck (TT, TE, EE+lowE+lensing) [3].
Intermediate results without a $\theta_{*}$ or BBN prior, as well as those
using three degenerate neutrino mass states with physically motivated priors
are presented in Table 1. These results show that the upper bound on $M_{\nu}$
significantly depends on the choice of its prior, whereas the rest of the
cosmological parameters are largely unaffected by this prior choice.

In Fig. 3, several important points are illustrated. In the top panel, it is
observed that the additional information provided by BBN and $\theta_{*}$ have
complementary degeneracy directions. Therefore, the combined analysis LSS +
BBN + $\theta_{*}$ achieves significantly greater constraining power on
$\Omega_{m}$ and $M_{\nu}$. Specifically, the constraint on $\Omega_{m}$ is
predominantly influenced by the inclusion of the BBN prior (dashed purple
line). In contrast, $\theta_{*}$ does not significantly aid in constraining
$\Omega_{m}$ (blue line) as much as BBN does, but it substantially lowers the
upper bound on the neutrino mass. The combination of BBN and $\theta_{*}$
results in the $M_{\nu}$–$\Omega_{m}$ posterior following the profile of
$\theta_{*}$ alone for small $\Omega_{m}$. It would continue this trend, but
BBN helps to tighten the upper bound of $\Omega_{m}$.

Following with the bottom panel of Fig. 3, it is observed that the inclusion
of $\theta_{*}$ to LSS+BBN drastically tightens the constraints on $M_{\nu}$
and recovers the degeneracy direction seen in the Planck results. Despite the
high constraining power, the peak at $M_{\nu}=0$ and potential projection
effects remain, as illustrated in Fig. 4. In those plots, we also show the
posteriors for Planck and Planck + BAO (BOSS DR12) for comparisons.

At this point, one might speculate that the peculiar behavior of the peak at
$M_{\nu}=0$ could be linked to the tendency of $A_{s}$ towards lower values
compared to Planck, as previously observed in Fig. 1 for the $\Lambda$CDM
model with fixed neutrino mass. Since, in the case of interest with free
neutrino mass, these two parameters are degenerate. However, Fig. 8 shows that
the posterior distribution of $M_{\nu}$ is unaffected by the lower $A_{s}$
values compared to Planck.

To gain more insight on $M_{\nu}$ posteriors for eBOSS-like data, we reanalyze
the noiseless synthetic data with the covariance matrix rescaled by a factor
of $1/25$ (labeled as $\text{cov}_{25}$), corresponding to a volume 25 times
larger than that of eBOSS. In this case, we are able to recover all
cosmological parameters with good precision, as shown in Fig. 5. However, the
neutrino mass profile still appears to have its maximum at negative mass
values, though it shows a notably improvement compared to the non-rescaled
covariance case (labeled as $\text{cov}_{1}$). Furthermore, the posterior
profile of the neutrino mass shows a slight increase in probability density
around the expected $M_{\nu}=0.06\,\text{eV}$.

Figure 5: Constraints for noiseless synthetic eBOSS data using the non-
rescaled covariance (cov1) and the $\times 25$ larger volume covariance
(cov25). Vertical dashed lines represent the best-fit values from Planck 2018.
Figure 6: Left panel: $\Delta\chi^{2}_{\rm min}$ for a set of $M_{\nu}$ values
obtained by profiling the likelihood using the iminuit code on noiseless
synthetic eBOSS data with the rescaled covariance matrix $\text{cov}_{25}$.
The intersection between the black horizontal line at $\Delta\chi^{2}_{\rm
min}=3.84$ and the vertical dash-dotted blue line can be used to approximately
estimate the upper bound at the 95% c.l. [65]. Right panel: Comparison of
marginalized posterior distributions obtained from the profile likelihood (PL)
and MCMC fits. The dashed blue and dot-dashed green lines correspond to the
results of sampling the neutrino mass $M_{\nu}$ and nuisance parameters
(Nuis.) while keeping the other cosmological parameters at their true values.
The orange curves (labeled all) represent the results obtained from MCMC when
fitting $\Lambda$CDM parameters, including the neutrino mass and nuisances.
The dotted posterior, already presented in Fig. 5, is obtained using the
baseline priors introduced in Sect. III, while the solid one corresponds to
the results using the Jeffreys prior. In both panels, the vertical dashed gray
lines correspond to the maximum a posteriori probability (MAP), located by
construction at $M_{\nu}=0.06\,\text{eV}$.

A method to identify projection effects consists on comparing results between
Bayesian MCMC and the frequentist profile likelihood (PL) [66], since by
construction the latter is unaffected by prior choices and marginalization
[67, 68]. In cosmology this method has been used for Planck CMB data in [69,
70], and more recently it has been used to analyze the galaxy power spectrum
full-shape fits [71, 11, 72, 73, 74]. Computing the PL requires the evaluation
of the parameters of interest —in our case, $M_{\nu}$— across a range of
values, while maximizing the likelihood $\mathcal{L}$ (or minimizing
$\chi^{2}=-2\ln\mathcal{L}$) with respect to all other parameters for each
value of $M_{\nu}$. In this work we obtain the PL using the MINOS algorithm
from the iminuit package777https://scikit-hep.org/iminuit/index.html [75, 76],
by profiling only the neutrino mass parameter. The resulting PL is then
expressed as a $\Delta\chi^{2}_{\rm min}$ distribution as a function of
$M_{\nu}$.

Figure 6 shows the comparison between Bayesian and frequentist approaches
using eBOSS synthetic data and rescaled covariance matrix $\text{cov}_{25}$.
Several key points can be identified. Firstly, the PL analysis confirms that
the theoretical model prefers $M_{\nu}=0.06\,\text{eV}$, as expected because
the data is generated by the same model.

Secondly, the right panel shows similar posterior distributions between the
MCMC (dot-dashed green line) and PL (dashed blue line) approaches when
profiling the neutrino mass and the nuisance parameters while maintaining the
other cosmological parameters at their true values. However, there are still
slight differences in the posterior distributions. Notably, the peak of the
MCMC analysis is subtly shifted to lower values compared to the expected
value. Hence this shift should be the consequence of projection effects
arising from the marginalization over nuisance parameters in the Bayesian
inference. These effects become more pronounced when extending the sampling
space to the rest of the $\Lambda$CDM parameters (labeled all). This is
evidenced by the orange dotted line, where the peak shifts to $M_{\nu}=0$.
This suggests that the latter is influenced by projection effects arising not
only from the marginalization over nuisance parameters, as observed in [46,
71] for fixed neutrino mass, but also from the rest of cosmological
parameters. By incorporating the Jeffreys prior, represented by the solid
orange line, the profile distribution peaks closer to the expected value
$M_{\nu}=0.06\,\text{eV}$. The significant improvement observed when using the
Jeffreys prior can be attributed to their ability to partially cancel the
contributions from the Laplace term (i.e., $\ln[\det(\mathcal{A}_{ij})]$ in
Eq. (A.9) of [37]), which are directly associated with projection effects
[68].

##  V Wiggle vs Broadband suppression

Figure 7: Frankenstein Experiment. The left panel displays noiseless synthetic
data for all possible combinations of broadband (BB) and wiggle (W) features.
The right panel presents the 1-dimensional marginalized posterior
distributions of $M_{\nu}$ for fits on each synthetic data set, showing that
the information on neutrino masses is primarily extracted from the wiggles
rather than the broadband suppression.

The previous section demonstrates that the current full-shape analysis is not
reliable for extracting the neutrino mass from SDSS or even DESI data.
Therefore, in this section, we revisit the suppression of the power spectrum
to shift the focus away from the effects of neutrinos on background evolution,
which can lead to high degeneracies between the neutrino mass and other
cosmological parameters.

We first note, that regarding the broadband information of the power spectrum
a degeneracy between $\Omega_{m}$ and $M_{\nu}$ could be expected, as massive
neutrinos suppress the power spectrum, suggesting a compensatory increase in
matter density. A similar effect is observed with $A_{s}$ and $h$, though
$A_{s}$ shows a milder correlation while $h$ is highly sensitive to
redshift.888A similar behavior is observed for $f(R)$ scale-dependent modified
gravity (MG) in [36]. In that case, however, MG produces an enhancement of the
power spectrum instead of a suppression, and $\Omega_{m}$ and $M_{\nu}$ become
anticorrelated. Furthermore, in viables $f(R)$ scenarios the background
evolution is essentially the same as in $\Lambda$CDM. This leads us to
question: where is the information about neutrino mass coming from? Aside from
the background evolution, is the neutrino mass inferred from the wiggles
rather than broadband information? To explore this, we conduct an unorthodox
experiment, in the style of Frankenstein’s monster [77]. We consider two
linear power spectra, one with neutrino mass $M_{\nu}=0.0\,\text{eV}$ and a
second with a large neutrino mass, $M_{\nu}=0.5\,\text{eV}$. We fix the
densities $\omega_{c}$ and $\omega_{b}$, hence the models have different total
amount of matter $\omega_{m}$ and the effects of massive neutrinos in the
wiggles is mainly a modulation in their relative amplitude, with only a very
small shift in their frequency. We decompose the two linear power spectra into
broadband + wiggles and mix the pieces to get four models. Afterwards, the
linear spectra are evolved non-linearly using FOLPS, employing nuisance
parameters previously obtained from the best fit to the synthetic data with
massless neutrino cosmology, and produce the following four noiseless
synthetic data:

  * •

B00W00 - Broadband of the model with mass $M_{\nu}=0.0\,\text{eV}$, wiggles of
the model with mass $M_{\nu}=0.0\,\text{eV}$,

  * •

B00W05 - Broadband of the model with mass $M_{\nu}=0.0\,\text{eV}$, wiggles of
the model with mass $M_{\nu}=0.5\,\text{eV}$,

  * •

B05W00 - Broadband of the model with mass $M_{\nu}=0.5\,\text{eV}$, wiggles of
the model with mass $M_{\nu}=0.0\,\text{eV}$,

  * •

B05W05 - Broadband of the model with mass $M_{\nu}=0.5\,\text{eV}$, wiggles of
the model with mass $M_{\nu}=0.5\,\text{eV}$,

where B00W00 and B05W05 are used as the original control samples, while the
mixed broadband and wiggles correspond to the Frankensteins. For the fits, we
utilize the rescaled covariance matrix $\text{cov}_{25}$ and the window
function employed for the eBOSS synthetic data.

In our analysis we fit the monopole and quadrupole multipoles using FOLPS. The
results are shown in Fig. 7. As can be seen, the models with wiggles for a sum
neutrino mass $M_{\nu}=0.5\,\text{eV}$ produce nearly identical results and
both peak at 0.5 eV. This is interesting because the BB00W05 model (solid red
line) was obtained from the non-suppressed broadband for massless neutrinos.
On the other hand, the two models with wiggles obtained from linear power
spectrum neutrinos with $M_{\nu}=0.0\,\text{eV}$ are consistent with massless
neutrinos, even though the BB05W00 model has a highly suppressed broadband.

From these results we conclude that the relevant information regarding the
damping of the power spectrum due to the neutrino mass resides in the wiggles,
and the full-shape analysis is blind to the suppression of the broadband.

##  VI Conclusions

In this work, we revisited the constraints on neutrino mass extracted from the
full-shape analysis of BOSS galaxies and eBOSS quasars. Surprisingly, we found
that eBOSS places tighter constraints on the neutrino mass compared to the two
non-overlapping redshift bins of BOSS, $z_{1}$ and $z_{3}$. Moreover, it is
responsible for the well-marked peak at $M_{\nu}=0$ in the joint analysis of
BOSS + eBOSS, a signature also observed in other works using CMB and BAO data
[3, 7, 8, 9, 69]. To further understand the observed behavior, we generated
noiseless synthetic data for the flat $\Lambda$CDM model using the Planck 2018
best-fit values with $M_{\nu}=0.06\,\text{eV}$ as input cosmology. Ideally,
one would expect to accurately recover the input cosmological parameters, as
the noiseless synthetic data was generated using the same theoretical model
and settings. However, we found very similar patterns in the neutrino mass
posteriors as those seen in real data, indicating the presence of strong
projection effects on the neutrino mass constraints.

We investigated whether incorporating additional constraining data alleviates
the impact of projection effects on neutrino constraints. To explore this, we
performed a combined analysis of BOSS and eBOSS data, integrating informative
priors on baryons from BBN and the acoustic scale at last scattering from
Planck 2018. Although this analysis achieved high constraining power, putting
slightly tighter bounds on neutrino mass $M_{\nu}<0.21\,\text{eV}$ (95% c.l.)
compared to the limit of $M_{\nu}<0.24\,\text{eV}$ (95% c.l.) obtained by
Planck data alone, the peak at $M_{\nu}=0$ and prior volume effects still
persisted. Furthermore, we compared outcomes from Bayesian MCMC and
frequentist profile likelihood approaches. Importantly, the latter is
inherently immune to projection effects. Our analysis was conducted in a
controlled scenario using noiseless synthetic data that mimicked eBOSS
spectra, along with a rescaled covariance matrix reflecting a volume 25 times
larger than the employed in eBOSS. Our preliminary test evaluated how nuisance
parameters alone affect the constraints on neutrino mass. Subsequently, we
analyzed how these constraints are influenced by both nuisance and
cosmological parameters. Our findings confirm that neutrino mass constraints
derived from Bayesian analysis are influenced by projection effects stemming
from nuisance parameters. These results are consistent with previous studies
based on the $\Lambda$CDM model with fixed neutrino masses, which found that
cosmological parameters are similarly affected [46, 71]. However, we also
discovered that cosmological parameters significantly introduce projection
effects on neutrino mass constraints. Furthermore, we observed that utilizing
the Jeffreys prior helps to partially mitigate them.

Our findings highlight the importance of carefully handling systematics in
full-shape analyses, especially when aiming to obtain reliable constraints on
neutrino mass. This is a crucial consideration for current surveys like DESI
and beyond, where poor control of projection effects could lead to incorrect
constraints. We claim that these unpleasent outcomes are due to large
degeneracies with parameters affected by the background evolution of the
Universe. For this reason, we revisited the use of power spectrum suppression
to infer the neutrino mass and argue that it should be considered as an
alternative approach to extracting information about neutrino mass, as
initially proposed in [78], rather than relying solely on the cumulative
effect of the expansion history of the Universe over the structure formation.
We analyze this suppression and, contrary to common belief, found that most
clustering information on neutrino mass is derived from the relative amplitude
of the BAO wiggles rather than from the broadband suppression. This discovery
opens the possibility of developing new analysis methods that rely solely on
the wiggles information. Despite providing weaker constraints on neutrino
mass, such methods could be more robust than those heavily reliant on
background expansion.

###### Acknowledgements.

This work is partially supported by CONAHCyT grant CBF2023-2024-162 and PAPIIT
IG102123.

##  Appendix A COMPLEMENTARY RESULTS

Model / dataset |  $M_{\nu}\,[\text{eV}]$ |  $h$ |  $\Omega_{m}$
---|---|---|---
|  |  |
|  95% c.l. |  68% c.l. |  68% c.l.
LSS + BBN |  $<0.668$ |  $0.686\pm 0.011$ |  $0.311^{+0.012}_{-0.020}$
LSS + $\theta_{*}$ |  $<0.348$ |  $0.640^{+0.020}_{-0.027}$ |  $0.321^{+0.015}_{-0.021}$
LSS + BBN + $\theta_{*}$ |  $<0.214$ |  $0.6793\pm 0.0073$ |  $0.2992\pm 0.0094$
LSS + BBN + $\theta_{*}$ (Deg.; $M_{\nu}>0$) |  $<0.245$ |  $0.6791\pm 0.0074$ |  $0.2993\pm 0.0095$
LSS + BBN + $\theta_{*}$ (Deg.; $M_{\nu}>0.058\,\text{eV}$) |  $<0.274$ |  $0.6772\pm 0.0072$ |  $0.3006\pm 0.0093$
LSS + BBN + $\theta_{*}$ (Deg.; $M_{\nu}>0.10\,\text{eV})$ |  $<0.300$ |  $0.6760\pm 0.0072$ |  $0.3014\pm 0.0094$
Table 1: Constraints on cosmological parameters obtained by jointly fitting
BOSS + eBOSS data in combination with external priors. Results with two-sided
error bars refer to the marginalized means and 68% c.l., while the upper
bounds on $M_{\nu}$ correspond to 95% limits. The constraints based on a model
with three degenerate mass states are labeled as ‘Deg.’ and include physically
motivated priors on neutrino mass: $M_{\nu}>0$, NH
$(M_{\nu}>0.058\,\text{eV})$, and IH ($M_{\nu}>0.10\,\text{eV}$).

In this appendix, we present the constraints on cosmological parameters
obtained by jointly fitting data from BOSS galaxies and eBOSS quasars,
incorporating informative priors from BBN and the acoustic scale $\theta_{*}$
at last scattering from Planck 2018 (labeled as LSS+BBN+$\theta_{*}$), as well
as adding only BBN or $\theta_{*}$ separately. 1-dimensional confidence
intervals are shown in Table 1. We further put constraints on the neutrino
mass for IH and NH, which are obtained by simply putting priors of
$M_{\nu}>0.10\,\text{eV}$ and $M_{\nu}>0.058\,\text{eV}$, respectively.

Furthermore, Fig. 8 compares the results of directly sampling the bias
parameters $b_{n}=\\{b_{1},b_{2},b_{\text{s}^{2}},b_{\text{3nl}}\\}$ (dash-
dotted black line) against sampling the scaled bias parameters at each step of
the MCMC,
$b_{n}\sigma^{m}_{8}=\\{b_{1}\sigma_{8},b_{2}\sigma^{2}_{8},b_{\text{s}^{2}}\sigma^{2}_{8},b_{\text{3nl}}\sigma^{3}_{8}\\}$
(solid purple line), following [52], and converting back to the original bias
parameters $b_{n}$ to feed FOLPS and generate the multipoles. This approach
helps prevent the downward shift in $A_{s}$ (and $\sigma_{8}$). Specifically,
we find that the constraints improve from $\ln(10^{10}A_{s})=2.84\pm 0.12$
($\sigma_{8}=0.710\pm 0.043$) to $\ln(10^{10}A_{s})=2.96\pm 0.12$
($\sigma_{8}=0.755\pm 0.043$) at 68% confidence level. However, it does not
eliminate the projection effects on $M_{\nu}$. Indeed, the rest of the
cosmological parameters remain unchanged.

Figure 8: Upper panel: Comparison of 2-dimensional constraints in the
$M_{\nu}$-$\ln(10^{10}A_{s})$ and $M_{\nu}$-$\sigma_{8}$ spaces for the base
setup (dash-dotted black line) and when scaling bias parameters with
$\sigma_{8}$ (solid purple line). The vertical dashed gray lines represent the
best-fit values from Planck 2018. Bottom panel: Comparison of 1-dimensional
marginalized posterior distributions of cosmological parameters.

## References

  * [1] I. Esteban, M. C. Gonzalez-Garcia, M. Maltoni, T. Schwetz and A. Zhou, _The fate of hints: updated global analysis of three-flavor neutrino oscillations_ , _JHEP_ 09 (2020) 178, [2007.14792].
  * [2] M. Aker et al., _Direct neutrino-mass measurement based on 259 days of KATRIN data_ , 2406.13516.
  * [3] Planck collaboration, N. Aghanim et al., _Planck 2018 results. VI. Cosmological parameters_ , _Astron. Astrophys._ 641 (2020) A6, [1807.06209].
  * [4] DESI collaboration, A. G. Adame et al., _DESI 2024 VI: Cosmological Constraints from the Measurements of Baryon Acoustic Oscillations_ , 2404.03002.
  * [5] J. Lesgourgues and S. Pastor, _Massive neutrinos and cosmology_ , _Phys. Rept._ 429 (2006) 307–379, [astro-ph/0603494].
  * [6] Y. Y. Y. Wong, _Neutrino mass in cosmology: status and prospects_ , _Ann. Rev. Nucl. Part. Sci._ 61 (2011) 69–98, [1111.1436].
  * [7] BOSS collaboration, S. Alam et al., _The clustering of galaxies in the completed SDSS-III Baryon Oscillation Spectroscopic Survey: cosmological analysis of the DR12 galaxy sample_ , _Mon. Not. Roy. Astron. Soc._ 470 (2017) 2617–2652, [1607.03155].
  * [8] eBOSS collaboration, S. Alam et al., _Completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: Cosmological implications from two decades of spectroscopic surveys at the Apache Point Observatory_ , _Phys. Rev. D_ 103 (2021) 083533, [2007.08991].
  * [9] N. Craig, D. Green, J. Meyers and S. Rajendran, _No $\nu$s is Good News_, 2405.00836.
  * [10] A. Aviles, A. Banerjee, G. Niz and Z. Slepian, _Clustering in massive neutrino cosmologies via Eulerian Perturbation Theory_ , _JCAP_ 11 (2021) 028, [2106.13771].
  * [11] C. Moretti, M. Tsedrik, P. Carrilho and A. Pourtsidou, _Modified gravity and massive neutrinos: constraints from the full shape analysis of BOSS galaxies and forecasts for Stage IV surveys_ , _JCAP_ 12 (2023) 025, [2306.09275].
  * [12] K. S. Dawson, D. J. Schlegel, C. P. Ahn, S. F. Anderson, É. Aubourg, S. Bailey et al., _The Baryon Oscillation Spectroscopic Survey of SDSS-III_ , _AJ_ 145 (Jan., 2013) 10, [1208.0022].
  * [13] K. S. Dawson et al., _The SDSS-IV extended Baryon Oscillation Spectroscopic Survey: Overview and Early Data_ , _Astron. J._ 151 (2016) 44, [1508.04473].
  * [14] F. Bernardeau, S. Colombi, E. Gaztanaga and R. Scoccimarro, _Large scale structure of the universe and cosmological perturbation theory_ , _Phys. Rept._ 367 (2002) 1–248, [astro-ph/0112551].
  * [15] P. McDonald and A. Roy, _Clustering of dark matter tracers: generalizing bias for the coming era of precision LSS_ , _JCAP_ 0908 (2009) 020, [0902.0991].
  * [16] D. Baumann, A. Nicolis, L. Senatore and M. Zaldarriaga, _Cosmological Non-Linearities as an Effective Fluid_ , _JCAP_ 07 (2012) 051, [1004.2488].
  * [17] Z. Vlah, M. White and A. Aviles, _A Lagrangian effective field theory_ , _JCAP_ 09 (2015) 014, [1506.05264].
  * [18] M. M. Ivanov, M. Simonović and M. Zaldarriaga, _Cosmological Parameters from the BOSS Galaxy Power Spectrum_ , _JCAP_ 05 (2020) 042, [1909.05277].
  * [19] G. D’Amico, J. Gleyzes, N. Kokron, K. Markovic, L. Senatore, P. Zhang et al., _The Cosmological Analysis of the SDSS/BOSS data from the Effective Field Theory of Large-Scale Structure_ , _JCAP_ 05 (2020) 005, [1909.05271].
  * [20] D. Wadekar, M. M. Ivanov and R. Scoccimarro, _Cosmological constraints from BOSS with analytic covariance matrices_ , _Phys. Rev. D_ 102 (2020) 123521, [2009.00622].
  * [21] A. Chudaykin, M. M. Ivanov, O. H. Philcox and M. Simonović, _Nonlinear perturbation theory extension of the Boltzmann code CLASS_ , _Phys. Rev. D_ 102 (2020) 063533, [2004.10607].
  * [22] O. H. E. Philcox and M. M. Ivanov, _BOSS DR12 full-shape cosmology: $\Lambda$CDM constraints from the large-scale galaxy power spectrum and bispectrum monopole_, _Phys. Rev. D_ 105 (2022) 043517, [2112.04515].
  * [23] O. H. E. Philcox, M. M. Ivanov, G. Cabass, M. Simonović, M. Zaldarriaga and T. Nishimichi, _Cosmology with the Redshift-Space Galaxy Bispectrum Monopole at One-Loop Order_ , 2206.02800.
  * [24] I. Tanseri, S. Hagstotz, S. Vagnozzi, E. Giusarma and K. Freese, _Updated neutrino mass constraints from galaxy clustering and CMB lensing-galaxy cross-correlation measurements_ , 2207.01913.
  * [25] T. Nishimichi, G. D’Amico, M. M. Ivanov, L. Senatore, M. Simonović, M. Takada et al., _Blinded challenge for precision cosmology with large-scale structure: results from effective field theory for the redshift-space galaxy power spectrum_ , _Phys. Rev. D_ 102 (2020) 123541, [2003.08277].
  * [26] S.-F. Chen, Z. Vlah, E. Castorina and M. White, _Redshift-Space Distortions in Lagrangian Perturbation Theory_ , _JCAP_ 03 (2021) 100, [2012.04636].
  * [27] M. Tsedrik, C. Moretti, P. Carrilho, F. Rizzo and A. Pourtsidou, _Interacting dark energy from the joint analysis of the power spectrum and bispectrum multipoles with the EFTofLSS_ , 2207.13011.
  * [28] P. Carrilho, C. Moretti and A. Pourtsidou, _Cosmology with the EFTofLSS and BOSS: dark energy constraints and a note on priors_ , 2207.14784.
  * [29] R. C. Nunes, S. Vagnozzi, S. Kumar, E. Di Valentino and O. Mena, _New tests of dark sector interactions from the full-shape galaxy power spectrum_ , _Phys. Rev. D_ 105 (2022) 123506, [2203.08093].
  * [30] S. Ramirez, M. Icaza-Lizaola, S. Fromenteau, M. Vargas-Magaña and A. Aviles, _Full Shape Cosmology Analysis from BOSS in configuration space using Neural Network Acceleration_ , 2310.17834.
  * [31] S.-F. Chen, Z. Vlah and M. White, _Consistent Modeling of Velocity Statistics and Redshift-Space Distortions in One-Loop Perturbation Theory_ , _JCAP_ 07 (2020) 062, [2005.00523].
  * [32] G. D’Amico, L. Senatore and P. Zhang, _Limits on $w$CDM from the EFTofLSS with the PyBird code_, _JCAP_ 01 (2021) 006, [2003.07956].
  * [33] D. Linde, A. Moradinezhad Dizgah, C. Radermacher, S. Casas and J. Lesgourgues, _CLASS-OneLoop: Accurate and Unbiased Inference from Spectroscopic Galaxy Surveys_ , 2402.09778.
  * [34] H. E. Noriega, A. Aviles, S. Fromenteau and M. Vargas-Magaña, _Fast computation of non-linear power spectrum in cosmologies with massive neutrinos_ , _JCAP_ 11 (2022) 038, [2208.02791].
  * [35] A. Aviles and A. Banerjee, _A Lagrangian Perturbation Theory in the presence of massive neutrinos_ , _JCAP_ 10 (2020) 034, [2007.06508].
  * [36] M. A. Rodriguez-Meza, A. Aviles, H. E. Noriega, C.-Z. Ruan, B. Li, M. Vargas-Magaña et al., _fkPT: constraining scale-dependent modified gravity with the full-shape galaxy power spectrum_ , _JCAP_ 03 (2024) 049, [2312.10510].
  * [37] H. E. Noriega et al., _Comparing Compressed and Full-modeling Analyses with FOLPS: Implications for DESI 2024 and beyond_ , 2404.07269.
  * [38] N. Kaiser, _On the Spatial correlations of Abell clusters_ , _Astrophys. J. Lett._ 284 (1984) L9–L12.
  * [39] R. Scoccimarro, _Redshift-space distortions, pairwise velocities and nonlinearities_ , _Phys. Rev._ D70 (2004) 083007, [astro-ph/0407214].
  * [40] A. Taruya, T. Nishimichi and S. Saito, _Baryon Acoustic Oscillations in 2D: Modeling Redshift-space Power Spectrum from Perturbation Theory_ , _Phys. Rev._ D82 (2010) 063522, [1006.0699].
  * [41] A. Aviles, G. Valogiannis, M. A. Rodriguez-Meza, J. L. Cervantes-Cota, B. Li and R. Bean, _Redshift space power spectrum beyond Einstein-de Sitter kernels_ , _JCAP_ 04 (2021) 039, [2012.05077].
  * [42] A. Perko, L. Senatore, E. Jennings and R. H. Wechsler, _Biased Tracers in Redshift Space in the EFT of Large-Scale Structure_ , 1610.09321.
  * [43] Z. Vlah and M. White, _Exploring redshift-space distortions in large-scale structure_ , _JCAP_ 1903 (2019) 007, [1812.02775].
  * [44] V. Assassi, D. Baumann, D. Green and M. Zaldarriaga, _Renormalized Halo Bias_ , _JCAP_ 08 (2014) 056, [1402.5916].
  * [45] S.-F. Chen, Z. Vlah and M. White, _A new analysis of galaxy 2-point functions in the BOSS survey, including full-shape information and post-reconstruction BAO_ , _JCAP_ 02 (2022) 008, [2110.05530].
  * [46] T. Simon, P. Zhang, V. Poulin and T. L. Smith, _Consistency of effective field theory analyses of the BOSS power spectrum_ , _Phys. Rev. D_ 107 (2023) 123530, [2208.05929].
  * [47] M. Maus et al., _A comparison of effective field theory models of redshift space galaxy power spectra for DESI 2024 and future surveys_ , 2404.07272.
  * [48] T. Tröster et al., _Cosmology from large-scale structure: Constraining $\Lambda$CDM with BOSS_, _Astron. Astrophys._ 633 (2020) L10, [1909.11006].
  * [49] E. Di Valentino et al., _Cosmology Intertwined III: $f\sigma_{8}$ and $S_{8}$_, _Astropart. Phys._ 131 (2021) 102604, [2008.11285].
  * [50] E. Abdalla et al., _Cosmology intertwined: A review of the particle physics, astrophysics, and cosmology associated with the cosmological tensions and anomalies_ , _JHEAp_ 34 (2022) 49–211, [2203.06142].
  * [51] S.-F. Chen, M. M. Ivanov, O. H. E. Philcox and L. Wenzl, _Suppression without Thawing: Constraining Structure Formation and Dark Energy with Galaxy Clustering_ , 2406.13388.
  * [52] M. Maus et al., _An analysis of parameter compression and full-modeling techniques with Velocileptors for DESI 2024 and beyond_ , 2404.07312.
  * [53] F. Beutler and P. McDonald, _Unified galaxy power spectrum measurements from 6dFGS, BOSS, and eBOSS_ , _JCAP_ 11 (2021) 031, [2106.06324].
  * [54] F.-S. Kitaura et al., _The clustering of galaxies in the SDSS-III Baryon Oscillation Spectroscopic Survey: mock galaxy catalogues for the BOSS Final Data Release_ , _Mon. Not. Roy. Astron. Soc._ 456 (2016) 4156–4173, [1509.06400].
  * [55] eBOSS collaboration, C. Zhao et al., _The completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: 1000 multi-tracer mock catalogues with redshift evolution and systematics for galaxies and quasars of the final data release_ , _Mon. Not. Roy. Astron. Soc._ 503 (2021) 1149–1173, [2007.08997].
  * [56] E. Aver, K. A. Olive and E. D. Skillman, _The effects of He I $\lambda$10830 on helium abundance determinations_, _JCAP_ 07 (2015) 011, [1503.08146].
  * [57] R. J. Cooke, M. Pettini and C. C. Steidel, _One Percent Determination of the Primordial Deuterium Abundance_ , _Astrophys. J._ 855 (2018) 102, [1710.11129].
  * [58] K. C. Chan, R. Scoccimarro and R. K. Sheth, _Gravity and Large-Scale Non-local Bias_ , _Phys. Rev. D_ 85 (2012) 083509, [1201.3614].
  * [59] T. Baldauf, U. Seljak, V. Desjacques and P. McDonald, _Evidence for Quadratic Tidal Tensor Bias from the Halo Bispectrum_ , _Phys. Rev. D_ 86 (2012) 083540, [1201.4827].
  * [60] S. Saito, T. Baldauf, Z. Vlah, U. Seljak, T. Okumura and P. McDonald, _Understanding higher-order nonlocal halo bias at large scales by combining the power spectrum with the bispectrum_ , _Phys. Rev._ D90 (2014) 123522, [1405.1447].
  * [61] D. Blas, J. Lesgourgues and T. Tram, _The Cosmic Linear Anisotropy Solving System (CLASS) II: Approximation schemes_ , _JCAP_ 07 (2011) 034, [1104.2933].
  * [62] D. Foreman-Mackey, D. W. Hogg, D. Lang and J. Goodman, _emcee: The MCMC Hammer_ , _Publ. Astron. Soc. Pac._ 125 (2013) 306–312, [1202.3665].
  * [63] A. Lewis, _GetDist: a Python package for analysing Monte Carlo samples_ , 1910.13970.
  * [64] T. Simon, P. Zhang and V. Poulin, _Cosmological inference from the EFTofLSS: the eBOSS QSO full-shape analysis_ , _JCAP_ 07 (2023) 041, [2210.14931].
  * [65] J. Neyman, _Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability_ , _Phil. Trans. Roy. Soc. Lond. A_ 236 (1937) 333–380.
  * [66] R. D. Cousins, _Why isn’t every physicist a Bayesian?_ , _Am. J. Phys._ 63 (1995) 398.
  * [67] J. Hamann, S. Hannestad, G. G. Raffelt and Y. Y. Y. Wong, _Observational bounds on the cosmic radiation density_ , _JCAP_ 08 (2007) 021, [0705.0440].
  * [68] B. Hadzhiyska, K. Wolz, S. Azzoni, D. Alonso, C. García-García, J. Ruiz-Zapatero et al., _Cosmology with 6 parameters in the Stage-IV era: efficient marginalisation over nuisance parameters_ , 2301.11895.
  * [69] Planck collaboration, P. A. R. Ade et al., _Planck intermediate results. XVI. Profile likelihoods for cosmological parameters_ , _Astron. Astrophys._ 566 (2014) A54, [1311.1657].
  * [70] Planck collaboration, P. A. R. Ade et al., _Planck 2013 results. XVI. Cosmological parameters_ , _Astron. Astrophys._ 571 (2014) A16, [1303.5076].
  * [71] E. B. Holm, L. Herold, T. Simon, E. G. M. Ferreira, S. Hannestad, V. Poulin et al., _Bayesian and frequentist investigation of prior effects in EFT of LSS analyses of full-shape BOSS and eBOSS data_ , _Phys. Rev. D_ 108 (2023) 123514, [2309.04468].
  * [72] L. Herold, E. G. M. Ferreira and E. Komatsu, _New Constraint on Early Dark Energy from Planck and BOSS Data Using the Profile Likelihood_ , _Astrophys. J. Lett._ 929 (2022) L16, [2112.12140].
  * [73] M. Maus, S.-F. Chen and M. White, _A comparison of template vs. direct model fitting for redshift-space distortions in BOSS_ , _JCAP_ 06 (2023) 005, [2302.07430].
  * [74] J. S. Cruz, S. Hannestad, E. B. Holm, F. Niedermann, M. S. Sloth and T. Tram, _Profiling cold new early dark energy_ , _Phys. Rev. D_ 108 (2023) 023518, [2302.07934].
  * [75] F. James and M. Roos, _Minuit: A System for Function Minimization and Analysis of the Parameter Errors and Correlations_ , _Comput. Phys. Commun._ 10 (1975) 343–367.
  * [76] H. Dembinski and P. O. et al., _scikit-hep/iminuit_ , .
  * [77] M. W. Shelley, _Frankenstein, or, The Modern Prometheus: the 1818 Text_.  Oxford University Press, Oxford; New York, 1998.
  * [78] W. Hu, D. J. Eisenstein and M. Tegmark, _Weighing neutrinos with galaxy surveys_ , _Phys. Rev. Lett._ 80 (1998) 5255–5258, [astro-ph/9712057].


# LipFormer: Learning to Lipread Unseen Speakers based on Visual-Landmark
Transformers

Feng Xue, Yu Li, Deyin Liu, Yincen Xie, Lin Wu, Richang Hong F. Xue, Y. Li, Y.
Xie, L. Wu, R. Hong are Hefei University of Technology, China. D. Liu is with
Anhui University, ChinaCorresponding author: Richang Hong

###### Abstract

Lipreading refers to understanding and further translating the speech of a
speaker in the video into natural language. State-of-the-art lipreading
methods excel in interpreting overlap speakers, i.e., speakers appear in both
training and inference sets. However, generalizing these methods to unseen
speakers incurs catastrophic performance degradation due to the limited number
of speakers in training bank and the evident visual variations caused by the
shape/color of lips for different speakers. Therefore, merely depending on the
visible changes of lips tends to cause model overfitting. To address this
problem, we propose to use multi-modal features across visual and landmarks,
which can describe the lip motion irrespective to the speaker identities.
Then, we develop a sentence-level lipreading framework based on visual-
landmark transformers, namely LipFormer. Specifically, LipFormer consists of a
lip motion stream, a facial landmark stream, and a cross-modal fusion. The
embeddings from the two streams are produced by self-attention, which are fed
to the cross-attention module to achieve the alignment between visuals and
landmarks. Finally, the resulting fused features can be decoded to output
texts by a cascade seq2seq model. Experiments demonstrate that our method can
effectively enhance the model generalization to unseen speakers.

###### Index Terms:

Lipreading, Landmarks, Transformer, Lip motion.

##  I Introduction

Lipreading is the inference on the speech of a speaker from a video clip,
which could be presented with/without audial signals [1, 2, 3]. Lipreading
offers an effective way to infer text, alternative to speech recognition,
which renders implausible in disturbing circumstances, e.g, unknown speakers
in the wild. Besides, lipreading shows enormous values to real-world
applications, such as silent-movie processing and silent conversations [4, 5,
6].

Benefited by deep learning, lipreading has also witnessed its remarkable
progression, which has demonstrated its trend to even surpass experienced
subject experts. Early efforts are made to perform lipreading only at word-
level [7, 8]. However, such lipreading method only corresponds to one word at
each time. Compared to word-level lipreading, sentence-level lipreading [9,
10, 11, 12, 13] is more accurate in sentence prediction by predicting the
texts depending on the contextual priors. For example, Assael et.al[14]
proposed LipNet, which combines VGG[15], LSTM[16], and CTC[17], and thus
achieved an accuracy of 95.2 on the GRID dataset[18]. In[19], the authors
developed an approach based on attribute learning and contrast learning, which
greatly improved the performance of lipreading. However, the majority of
current lipreading models are only trained and tested on publicly available
datasets, which are limited in their training sample size and number of
speakers. Moreover, the performance improvement of these methods are
incremental to unseen speakers as they are mainly developed in the case of
overlap speakers. (i.e., the test speaker has ever appeared in the training
set). We hypothesize that these methods describe the lip motion using visual
clues, however, if the model is only trained with visual lip motion, it will
cause the overfitting due to the visual variations caused by the shape/color
of lips and pronunciation habit of a particular speaker. As a consequence,
this hampers the generalization ability of the model. For example, as shown in
Figure 1, being overfitted to the visual variations e.g., the lip shape, the
model translates into different texts even when two speakers say the same
word. Therefore, developing a lipreading method simply using lip motion may
slump the translation accuracy especially in unseen speakers. In real-life
applications, a lipreading system is often required to make lip-to-text
predictions for new faces, which may not be observed in the training bank.
Also, learning a model with good generalization ability to unseen speaks is
paramount to downstream applications.

Figure 1:  The visual variations of lip motion can easily cause the
overfitting of a lipreading model by associating spurious correlation between
motion and texts. In this paper, we propose to use landmarks to calibrate the
cross-modal association. Figure 2:  The proposed LipFormer is an end-to-end
two-stream architecture built upon a visual branch and a landmark branch. The
input to the visual branch is a sequence of lip images. The input to the
landmark branch is a 340-dimensional vector extracted from speaker frame. The
embeddings from the two streams are forwarded to a cross-attention module to
achieve the alignment, which allows for the cross-modal fusion. The resulting
fused features can be decoded to output texts by a cascaded seq2seq model.
Definitions of notations can be seen in Table I.

Given limited access to the number of samples and speaker identities, we aim
to enhance the model in lipreading by calibrating the association between
motion and texts via corrective alignment, such that the model can generalize
to unseen speakers. In this paper, we explore to use landmarks as a corrective
offset to achieve the true underlying association between lip motion and
sentences for lipreading. In [20], the author proposed to learn object
structural representations by using landmarks as a complementary feature to
the pretrained deep representation in recognizing visual attributes. In [21],
the authors proposed to extract facial geometric features by using landmarks,
and both geometric and texture-based features can be used to improve the
accuracy of facial expression recognition. Such a method can also help the
model generalize to new faces, which are out of the training set. Inspired by
those approaches, in this paper, we propose to introduce landmark features
independent to the visual appearance of lips, and thus can eliminate the
visual variations between speakers. In fact, landmarks encode positional
priors of the speaker’s face and lips (i.e., lip and facial landmarks). The
motion trajectory encoded by those landmarks effectively describes lip motion
irrespective to speakers. And the learned landmark embedding is less
influenced by the visual variations. It shows potential to better the
generalization of a lipreading model to the unseen speakers in inference.

In this paper, we aim to improve the generalization of a lipreading model in
recognizing unseen speakers. To achieve this goal, we propose a sentence-level
lipreading framework based on visual-landmark transformers, dubbed LipFormer.
Specifically, we describe the lip motion with features from two modalities:
visual features and landmarks. The visual features are extracted from lip
regions for each speaker at frame level, and then self-attended to be
discriminative. However, encoding the lip motions only using visual
information can easily lead to over-fitting, owing to the strong bias towards
visual variations caused by the shape/color of a speaker’s lips. To this end,
we propose to use landmarks extracted from face and lips as motion trajectory
to eliminate such variations. Then, we employ the cross-attention to align and
fuse such cross-modal features. The cross-attention of transformer can
effectively learn the correspondence between the visual and landmark
embeddings, so as to improve the joint representations for cross-modal fusion.
Finally, we employ a cascaded sequence-to-sequence (seq2seq) to decode the
fused features and generate the texts.

The main contributions of this paper are summarised below:

1) We propose a sentence-level lipreading framework based on visual-landmark
transformers, which introduces corrective landmarks to minimize the biased
visual variations, making the model generalize to unseen speakers.

2) The proposed model uses cross-modal features to describe lip motion, and a
cross-attention is adopted to achieve the alignment between visuals and
landmarks, which promotes the fusion of heterogeneous features and further
improves the generalization of the model.

3) Extensive experiments are conducted on benchmark datasets to demonstrate
the effectiveness of the proposed method in interpreting unseen speakers and a
SOTA performance is achieved.

TABLE I: Notations and Definitions. Symbol | Definition
---|---
$f^{v}_{T}$, $f^{m}_{T}$ | visual feature, landmark feature
$f^{vm}$ | visual-landmark embedding sequence
$f^{p}$ | pinyin embedding sequence
$GRU_{e}$ | The subscript e indicates the encoder
$GRU_{d}$ | The subscript d indicates the decoder
$GRU^{vm}_{e}$ | GRU unit in visual-landmark encoder
$GRU^{p}_{e}$, $GRU^{p}_{d}$ | GRU unit in pinyin encoder and pinyin decoder
$GRU^{c}_{d}$ | GRU unit in character decoder
$h^{vm}_{e}$, $h^{p}_{e}$ | visual-landmark encoder output, pinyin encoder output
$c^{vm}_{p}$, $c^{p}_{c}$, $c^{vm}_{c}$ | context vector calculated by the attention

##  II Related work

In this section, we briefly review the lipreading methods, which can be
categorized into two lines: conventional methods and deep learning-based
lipreading methods.

###  II-A Traditional lipreading methods

1) Pixel-based methods. They assumes that all pixels in the lip region contain
vision-related information, and uses the pixel value of the lip region as the
original features. The features are reduced in different ways to obtain
expressive features. For example, Potamianos et al.[22] proposed HiLDA, which
is widely used as a visual feature extractor in speech recognition tasks.
Lucey et al.[23] further considered local features based on this, the author
extracted local features of image patch, fusing global features with local
features to further improve recognition accuracy. Tim et al.[24] normalize and
concatenate the AAM features of consecutive frames to extract spatio-temporal
features by linear transformation. 2) Shape-based methods. They extract
features based on the shape of the lip region (lips, chin, etc.). For example,
Papcun et al.[25] used articulatory features (AFs) for lipreading, but since
this kind of features is too simple to distinguish similar word, it is
generally applied to small-scale recognition tasks. Chan[26] combined
geometric features with PCA features of the lip as visual features. Luettin et
al. [27] applied the ASM model to lipreading, generating features from the
coordinates of several key points. However, the shape-based model assumes that
most of the information related to visual is on the contour represented by
feature points, which inevitably leads to information loss.

Benefited by deep learning, lipreading has also witnessed its remarkable
progression. Compared with traditional methods, deep learning-based methods
has powerful feature learning capability. The deep learning method avoids the
complex hand-crafted feature extraction process, and the performance of its
model can be further enhanced with large-scale data.

###  II-B Deep learning-based lipreading methods

Lipreading can be carried out at word-level and sentence-level lipreading.
Early efforts are made to perform lipreading only at word-level, such
lipreading video only corresponds to one word and had a small range of
applications. For example, Chung [28] designed two CNN structures, i.e., early
fusion and multiple towers, which can be combined for an all-once word-level
translation from a sequence of lip motion. Petridis et al [7] proposed an end-
to-end audio-visual model based on residual networks and BiGRU, which
simultaneously learns to extract features directly from the image and audio.
Stafylakis et al[8] combined 3D-CNN and 2D-CNN to extract visual features and
obtained higher accuracy on the LRW dataset.

In this paper, we focus on sentence-level lipreading. Compared to word-level
lipreading, sentence-level lipreading is more accurate, since it predicts the
texts depending on the contextual priors. The LipNet [14] is the first end-to-
end sentence-level lipreading, which consists of 3DCNN, BiGRU, and CTC. LipNet
achieves 95.2$\%$ accuracy on the GRID dataset. In [29, 30, 31], the model
structure is similar to LipNet. However, CTC loss has conditional
independence, i.e., each output unit is individually predicting the
probability of a label. Therefore, CTC loss will focus on local information of
adjacent frames, which is not suitable for predicting labels that require
contextual information to discriminate. Considering the problems incurred by
CTC loss, Xu et al.[12] proposed LCANet, which stacks two layers of Highway
networks in 3DCNN. This can highly improve the quality of the extracted
features. They essentially used attention mechanism to overcome the
shortcomings of conditional independence assumption in CTC. The following work
[19] improves the performance of the lipreading model by introducing attribute
learning and contrast learning into the sentence-level lipreading pipeline.

Other lipreading methods are based on the seq2seq model. The most
representative model is WAS[32], which uses a 5-layer 2DCNN and LSTM to
extract visual features and auditory features. These features are fed to the
seq2seq module to generate texts. Zhao [11] proposed CSSMCM, which combines
factors such as pinyin and tones to help predict Chinese characters based on
visual information. In [1], the authors proposed a knowledge distillation
method that uses a speech recognition pre-trained model as a teacher model to
optimize a lipreading model as a student model, improving the accuracy of
lipreading.

Due to the excellent performance of Transformer, Zhou et al.[33] apply
transformer to speech recognition. Ma et al.[34] apply the transformer to
lipreading, and thus proposed the CTCH-LipNet, which first used 3DCNN to
extract visual features, and then a cascaded architecture that consists of two
transformers to predict pinyin and Chinese characters. Ma et al.[35] use both
video and audio as input, and the transformer decodes the features into texts.

However, most existing lipreading methods use overlap speakers by default in
experimental evaluation. This is prone to be overfitting to overlap speakers
in the training set, while perform poorly to the unseen speakers. In this
paper, we propose a sentence-level lipreading framework based on visual-
landmark transformers that generalize the model to unseen speakers, thus
solving the lipreading problem of unseen speakers.

##  III The Method

The architecture of our proposed LipFormer is shown in Fig 2, which consists
of four modules: 1) a visual stream that extracts visual features of lip
regions; 2) a landmark stream that encodes the trajectory of lip/facial
movement of a speaker across the sequence; 3) a cross-modal fusion that learns
the alignment between visuals and landmarks; and 4) a cascades seq2seq model
for mapping fused features to texts.

###  III-A Visual Embedding

For each video clip, we first extracted the face image in each frame by using
the DLib face detector, and then apply an affine transformation to each face
image to obtain the mouth-centered cropping with 160 × 80 pixel as the lip
region. For a video clip with $T$ frames, we can have a lip region sequence
$\\{T_{i}\\}$, where $I_{i}$ is the frame of the $i$th step ($i=1,\ldots,T$).
We first learn the per-frame feature embedding by applying the 3D convolution
[36], followed by a ReLu layer and a max-pooling layer. During training,
dropout regularization is used along with the 3DCNN to alleviate the
saturation problem.

Figure 3:  The two-stream architecture with a visual branch (a) and a landmark
branch (b). Given the input as a sequence of lip regions, we use 3DCNN
combined with channel attention to extract the features, followed by Bi-GRU to
encode temporal orders. The input to the landmark branch is the landmark
embedding: the difference of the angle matrix between adjacent frames. The
angle matrix is calculated between 20 landmarks of the lip and 17 landmarks of
the face contour for each frame and encoded as a 340-dimensional vector.

The visual features extracted by 3DCNN contain a lot of irrelevant
information, such as the shape of lip, pronunciation habits of different
speakers, etc. That information largely affects the accuracy of text generated
by the decoder. In this paper, we combine 3DCNN with channel attention. The
channel attention mechanism can learns the weights of each channel and
improves the performance of useful visual features by suppressing irrelevant
features. The obtained feature vector for each frame is denoted by
$f^{v}_{i}$.

To aggregate the spatial grids, two different spatial context descriptors are
generated from the input feature map by using average pooling and maximum
pooling, and these two outputs are fed into a shared network MLP to generate
the channel attention map. Multiply the attention map and the input features
to focus on important features. The structure of the visual branch is shown in
Fig.3 (a).

Limited by the size of convolution kernel, CNN can only extract short-term
spatio-temporal visual features. The bi-directional GRU [37, 38] is applied to
extract the long-term features:

$[s^{v}_{i},o^{v}_{i}]=GRU(f^{v}_{i},s^{v}_{i-1}),$ (1)

where $s^{v}_{i}$, $o^{v}_{i}$ are the hidden vector and output vector of the
GRU, respectively.

###  III-B Landmark Embeddings

Merely depending on the visual appearance will lead to inferior performance
for lipreading. A primary reason is that lip motion has a significant visual
variation caused by the very different shape and color of the lip or a
particular pronunciation habit of a speaker. To eliminate the visual
variations, we propose taking the landmarks as another feature embedding. The
landmark embedding will be less influenced by the lip appearance and has
better generalization ability to the unseen speakers during the training. The
distribution of the 68 facial landmarks is shown in Fig. 3(b), among which
there are 20 landmarks for lip, 17 landmarks for facial contour, and 31
landmarks for eyes, eyebrows, and noses. The structure of the landmark branch
is shown in Fig. 3 (b).

More specifically, we utilize both lip landmarks and facial contour landmarks
to construct the landmark embedding for lipreading. The change of facial
contour position is found to be the most obvious with the lip motion. The
motion trajectory between facial landmarks effectively describes lip motion.
The features are constructed as follows: 1) We first calculate the angle
between 20 landmarks of the lip and 17 landmarks of the face contour for each
frame, i.e. the cosine, to obtain the angle matrix of (1,340); 2) We then
compute the difference angle matrix for two adjacent frames to represent the
motion change of landmarks. 3) For the T-frame, the (B,T,340) angle matrix is
obtained as the input to the landmark branch. The matrix difference is the
per-frame landmark embedding, denoted by $f^{m}_{i}$ for the $i$th frame.
Given $\left\\{f^{m}_{1},f^{m}_{i},...f^{m}_{T}\right\\}$ of all the $T$
frames, we apply a bi-directional GRU to extract the long-term features so as
to obtain the output feature sequence
$\left\\{s^{m}_{1},s^{m}_{i},...s^{m}_{T}\right\\}$.

###  III-C Cross-Modal Fusion via Transformer

The embeddings from the two streams are fed to the transformer. We use the
transformer-encoder to achieve the cross-modal fusion. The encoder consists of
an encoder layer, which is composed of three parts: a self-attention module, a
cross-attention module, and a feed-forward network. The embeddings from the
two streams are produced by self-attention, which are fed to the cross-
attention module to achieve the alignment between visuals and landmarks. Three
matrices of query $Q$, key $K$, and value $V$ as input to the self-attention,
which are generated from the input sequence $z$. Self-attention module
extracting global information to establish global long-term dependency of lip
motion. Cross-attention takes the Q of the current modality (e.g., video) and
the K/V obtained in the opposite modality (e.g., landmark) as input, for each
visual feature embedding, different weights are assigned to each landmark
feature embedding by cross-attention, and the matching of weights can achieve
visual-landmark embedding alignment to achieve cross-modal feature fusion. The
output of the attention module is:

$Attention=\textbf{softmax}\left(\frac{Q^{i}{(K^{i})^{T}}}{\sqrt{d}}\right)\cdot
V^{i}.$ (2)

where $d$ is the length of the embedding vector.

The encoder layer is implemented as a feed-forward network, which contains two
fully-connected layers with a ReLU non-linearity:

$FFN(x)=FC(ReLU(FC(x))).$ (3)

###  III-D Text Generation

With the fused features as input, we feed them to a cascade sequence-to-
sequence model. The seq2seq model is with an encoder-decoder structure, where
both the encoder and decoder are LSTM models (sometimes GRU models). Encoder-
Decoder model can predict arbitrary sequence correspondence. In the Chinese
dataset, there are fewer pinyin categories, making it easier to predict the
pinyin. For this reason, we choose pinyin as a middle layer when predicting
Chinese characters and use a cascaded seq2seq model to decode text.

###  III-E Pinyin Prediction

With the fused features $f^{vm}$ as input, we feed them to a Pinyin seq2seq
model to decode the $f^{vm}$ into pinyin. We refer to the encoder and decoder
as visual-landmark encoder and pinyin decoder, in which the encoder processes
the visual-landmark feature sequence, and the decoder predicts the pinyin
sequence. The GRU of encoder calculating the hidden layer vector by inputting
$f^{vm}$ as:

$(h^{vm}_{e})_{i}=GRU^{vm}_{e}((h^{vm}_{e})_{i-1},e^{vm}_{i}).$ (4)

The decoder progressively computes the hidden layer vector
$\left(h^{p}_{d}\right)_{i}$ based on the predicted result $p_{i}$ of the
previous time step and the hidden state $\left(h^{p}_{d}\right)_{i-1}$of the
previous time step:

$(h^{p}_{d})_{i}=GRU^{p}_{d}((h^{p}_{d})_{i-1},E(p_{i})),$ (5)

where the embedding function $E(\cdot)$ maps $p_{i}$ to its vector space. To
further utilize the information contained in the input vector, we apply the
attention module to compute the weights of all hidden layer vectors in the
encoder to generate a context vector, that assists the decoder predicting the
pinyin. The weights are calculated as:

$att=\textbf{softmax}(FC(tanh(FC(Concat((h^{p}_{d})_{i}),h^{p}_{e})))).$ (6)

Finally, the probability distribution of the current pinyin is calculated by
splicing the GRU output and the context vector as:

$P(p_{i})=\textbf{softmax}(FC((h^{p}_{d})_{i},h^{p}_{e}\cdot att.$ (7)

###  III-F Character Prediction

Similarly, With the pinyin sequence as input, we feed them to a character
seq2seq model to translating pinyin to character, the computation process is
similar to that of predicting pinyin sequences. First, the pinyin sequence is
input to the pinyin encoder, and the GRU of the pinyin encoder calculating the
hidden layer vector as:

$(h^{p}_{e})_{i}=GRU^{p}_{e}((h^{p}_{e})_{i-1},e^{p}_{i}).$ (8)

In addition, a dual attention mechanism is employed in the process of
predicting characters in order to take both visual-landmark and pinyin
information. The character decoder calculates the context vector based on the
visual-landmark encoder output and the pinyin encoder output, and then
predicts the character:

$(h^{c}_{d})_{i}=GRU^{c}_{d}((h^{c}_{d})_{i-1},E(c_{i})).$ (9)
$c^{vm}_{i}=h^{vm}_{e}\cdot att((h^{c}_{d})_{i},h^{vm}_{e})).$ (10)
$c^{p}_{i}=h^{p}_{e}\cdot att((h^{c}_{d})_{i},h^{p}_{e})).$ (11)
$P(c_{i})=\textbf{softmax}(FC((h^{c}_{d})_{i},c^{vm}_{i},c^{p}_{i}).$ (12)

###  III-G Loss Function

To improve the prediction accuracy, the model first predicts pinyin and then
translating pinyin into Chinese characters. We jointly optimize the loss
function of these two processes. The loss function defined as:

$L=L_{p}+L_{c},$ (13)

where $L_{p}=-{\textstyle\sum_{n=1}^{N}}logP(p_{n}|x,p_{1},p_{2},...p_{n-1})$,
$L_{c}=-{\textstyle\sum_{n=1}^{N}}logP(c_{n}|x,c_{1},c_{2},...c_{n-1})$

##  IV Experiments

In this section, we validate the proposed method by conducting extensive
experiments on two benchmark datasets: CMLR [11] and GRID [18].

Figure 4:  Static images from the CMLR dataset. Figure 5:  Static images from
the GRID dataset.

###  IV-A Dataset

The Chinese Mandarin Lip Reading (CMLR) dataset [11] is the largest Chinese
Mandarin sentence lipreading dataset (example images are shown in Fig 4). The
whole dataset contains sequences recorded by 11 speakers. We split the dataset
to form the training and test set with 9 speakers and 2 speakers,
respectively. Note the speakers are non-overlap in training and test. For
experiments on overlap speakers, we divided the training, valid and test sets
by following [11]. The CMLR division protocol is shown in Table II.

TABLE II: Statistics for the CMLR dataset. | Set | Speakers | Sentences
---|---|---|---
Unseen | Train | 9 | 81094
Test | 2 | 20978
Overlap | Train | 11 | 71452
Test | 11 | 20418

The GRID dataset [18] has 33 speakers recorded, and is a widely used dataset
in lipreading (example images are shown in Fig 5). Each sentences consists of
a sequence of verb + color + preposition + letter + number + adverb. e.g. ”bin
blue at f five again”. To split the datasets into unseen and overlap speakers,
we follow the setting as suggested in [14]. The division protocol are provided
in Table III.

TABLE III: Statistics for the GRID dataset. | Set | Speakers | Sentences
---|---|---|---
Unseen | Train | 29 | 28837
Test | 4 | 3986
Overlap | Train | 33 | 24408
Test | 33 | 8415

###  IV-B Evaluation Metrics

To measure the performance of the proposed method and the baselines, we adopt
the widely used evaluation metrics in Automatic Speech Recognition: Word Error
Rate (WER) and Character Error Rate (CER). WER/CER is defined as the minimum
number of word/character operations (including substitution, deletion, and
insertion operations), which are required to convert the predicted label into
the ground truth, and then divided by the number of words/characters in the
ground truth. The calculation is defined as follows:

$WER/CER=100\cdot\frac{S+D+I}{N},$ (14)

where $S$ denotes the substitution, $D$ represents the deletion, $I$ denotes
the insertion, and $N$ is the number of words in the ground truth. Note that a
smaller WER/CER indicates a higher prediction accuracy. In addition, the
evaluation metric we use on the Chinese dataset CMLR is CER only, where CER
means the Character Error Rate. The CER is calculated in the same way as WER,
that is, each Chinese character is regarded as an English word.

###  IV-C Implementation Details

For each video clip, we first extracted the face image in each frame using the
DLib face detector [39]. The coordinates of the landmarks generated by the
face detector are used as the input of the landmark branch. The affine
transformation is applied to each face image to obtain 160 × 80-pixel mouth-
centered crop as the input to the visual branch. (see Fig 6). This experiment
uses the Adam optimizer to optimize the parameters with an initial learning
rate of 0.0003. When each training error did not improve within 4 epochs, the
learning rate decreases by 50$\%$.

Figure 6:  Examples of the mouth-centered crop.

###  IV-D Competitors

We compared our method with several state-of-the-art methods: LipNet[14],
CSSMCM[11], CALLip[19], LCSNet[40] and WAS[32].

LipNet: LipNet is the first end-to-end sentence-level lipreading model that
achieves an accuracy of 95.2 on the GRID dataset.

CSSMCM: This model is specifically designed for Chinese lipreading, combining
factors such as pinyin and tones to help predict Chinese characters.

CALLip: CALLip improves the performance of the model by introducing attribute
learning and contrast learning into the sentence-level lipreading pipeline,
which using an attribute learning module to extract speaker identity features
and eliminate cross-speaker variations.

LCSNet: This model extract features that are more relevant to the lip motion
by the channel attention module and the selective feature fusion module to
improve recognition accuracy.

WAS: This baseline uses video information to predict sentences by seq2seq
model, and achieves advanced performance on the LRS dataset.

TABLE IV: Performance comparison with state-of-the-arts on CMLR. -: results not available. Best results are in boldface. Methods | Unseen | Overlap
---|---|---
LipNet | 52.18 | 33.41
WAS | - | 38.93
CSSMCM | 50.08 | 32.48
CALLip | - | 31.18
LCSNet | 46.98 | 30.03
LipFormer | 43.18 | 27.79
TABLE V: Performance comparison with state-of-the arts on GRID. Best results are in boldface. Methods | Unseen | Overlap
---|---|---
LipNet | 17.5 | 4.8
WAS | 14.6 | 3.0
CALLip | - | 2.48
LCSNet | 11.6 | 2.3
LipFormer | 9.64 | 1.45

###  IV-E Comparison with Competitors

To prove the effectiveness of the proposed method, we first compare it to SOTA
competitors on the CMLR and GRID datasets. We empirically observed that the
CTC loss could result in non-convergence during the training on the CMLR
dataset. Hence, we replaced the CTC with the cascaded Seq2Seq module. We
report the results for both the unseen and overlap speakers of two datasets.
Experimental results are shown in Tables IV and V, respectively.

Table IV shows that the LipFormer has a notable improvement over the state-of-
the-art methods for unseen speakers. Comparing to LipNet, LipFormer uses
landmarks to jointly describe the lip motion. The comparison results show that
the adoption of multi-modal features is beneficial to performance improvement.
Compared with CSSMCM, the WER of LipFormer is further reduced by 6.9$\%$. One
reason is that LipFormer learned more consistent features of lip motion with
multi-modal features, so that the model is well-generalized to the unseen
speakers. For the overlap speakers, LipFormer outperforms the SOTA methods on
the CMLR dataset and achieves a character error rate of 27.79$\%$. It shows
that LipFormer is well-suited to both unseen and overlap speakers.

Table V shows the comparison results of different methods on the GRID dataset.
Compared with LipNet, LipFormer can reduced the WER by 7.86$\%$ for unseen
speakers and WER by 3.35$\%$ for overlap speakers. This affirms that landmarks
can help solving the generalisation problem. It can also be observed that
LipFormer outperforms other methods for both unseen and overlap speakers, even
thought these speakers are from different ethnics.

TABLE VI: Performance of LipFormer and its variants on CMLR. The front-end structure of LipFormer is Visual-only+Landmark+Transformer. Best results are in boldface. # | Methods | Unseen | Overlap
---|---|---|---
1 | Visual-only | 48.14 | 29.15
2 | Visual-only+Landmark | 43.48 | 28.0
3 | LipFormer | 43.18 | 27.79
TABLE VII: Performance of LipFormer and its variants on GRID. Best results are in boldface. # | Methods | Unseen | Overlap
---|---|---|---
1 | Visual-only | 12.35 | 2.82
2 | Visual-only+Landmark | 10.24 | 2.2
3 | LipFormer | 9.64 | 1.45

(a)

(b)

Figure 7: Performance of model variants on CMLR dataset. (a) Unseen. (b)
Overlap.

(a)

(b)

Figure 8: Performance of model variants on GRID dataset. (a) Unseen. (b)
Overlap.

###  IV-F Ablation study

To verify that the landmark branch and transformer module can effectively
improve the lipreading accuracy for unseen speakers, we decouple the LipFormer
framework and tested them on the CMLR and GRID datasets. The experimental
results are shown in Tables VI and VII, respectively. Fig 7a and Fig 8b shows
the WER curves of model variants on two datasets.

TABLE VIII: Performance of models with different embedding sizes on CMLR dataset. # | Methods | Unseen | Overlap
---|---|---|---
1 | 256 | 43.61 | 28.34
2 | 512 | 43.18 | 27.79
3 | 1024 | 43.47 | 28.12
TABLE IX: Prediction of different models on CMLR and GRID. Methods | CMLR | GRID
---|---|---
Unseen | Overlap | Unseen | Overlap
Ground truth | 人口较少名族发展规划 | 一些新做法取代了老传统 | bin blue at f three please | bin blue at d two soon
Visual-only | 中外较少名族发展模范 | 议程改进取代了措施 | bin blue at s three please | bin blue by d five soon
Visual-only+Landmark | 人口较少名族检查规划 | 一些新做法取代了老烧 | bin blue at l three please | bin blue at d two again
LipFormer | 人口较少名族发展规划 | 一些新做法取代了老传统 | bin blue at f three please | bin blue at d two soon

The variants of our method, i.e., #1 and #2, which fuse the visual branch and
landmark branch outperform that of the model with visual-only branch.
Specifically, compared with visual-only model, for the unseen and overlap
speakers, the WER of method #2 reduced by 4.66$\%$ and 1.15$\%$ on the CMLR
dataset and reduced by 2.11$\%$ and 0.62$\%$ on the GRID dataset,
respectively. This demonstrates that landmark can normalizes the lip shapes of
different speakers, eliminate the irrelevant visual variations, enhance the
generalization ability of the model to unseen speakers. Comparing LipFormer
with Method #2, experimental results show that the transformer module learns
the correspondence between visual-landmark embedding to achieve fusion of
cross-modal heterogeneous features, further improve the performance of the
model.

###  IV-G Sensitivity to Hyper-parameter

In this section, we conduct experiments on CMLR to investigate the effect of
hyper-parameter in LipFormer on the model performance. The feature extractor
encodes each time step as a feature embedding. To investigate the size of the
feature embedding in the landmark branch affects the learning performance of
the model, to this end, the size of embedding is controlled by controlling the
number of output channels of the GRU in the landmark branch. Three different
sizes are designed:256, 512, 1024.

Table VIII summarize the performance of each model for different embedding
sizes. In general, as the number of embedding increases, the performance of
the model will be better. However, a too-large embedding size may cause
overfitting of the model and the performance of the model may decrease instead
of increasing. The optimal number of channels for the first layer of GRU
output is 512 on the CMLR dataset both for unseen speakers and overlap
speakers.

###  IV-H Case Study

To qualitatively analyze the performance of the proposed model, this section
evaluates a part of the predicted results. Table IX show some sentences
generated by each variant model on the CMLR and GRID datasets, where the
characters highlighted in red are incorrect.

It can be seen from Table IX that there are some differences between the
sentences generated by Visual-only model and the ground truth on the CMLR
dataset, such as predicting”人口” (which means population) as ”中外”(which means
Chinese-foreign), predicting ”一些” (which means some) as ”议程” (which means
agenda) ,etc. This shows that the model translates different texts when people
say the same word due to the visual variations such as lip shape. Visual-
only+Landmark model generates sentences that are closer to the ground truth,
which indicates that landmark can increase the accuracy of different speakers
when pronouncing the same sentence. The sentences generated by LipFormer are
correct, This shows that Transformer can promote the fusion of cross modal
information. On the GRID dataset, some letters are easy to predict errors, for
example, ”f” is predicted to be ”s”. These errors are mainly caused by letters
with similar pronunciations.

###  IV-I Alignment Visualization

Figure 9a and Figure 9b visualize the alignment between visuals and landmarks
at 75 and 100 frames on the CMLR dataset by the cross-attention module,
respectively. Each row in the figure represents a visual modality, and each
column represents a landmark modality. The highlighted area in the figure
indicates the degree of alignment between the visual-landmark feature
embedding during feature fusion.

Figure 9 shows that in the process of feature fusion, cross-modal attention
can learn the corresponding relationship between two modal feature embeddings,
achieve the alignment between visual-landmark embedding, the diagonal trend is
obvious. When pronouncing the same word, attention is concentrated on the
corresponding different modal frames.

(a)

(b)

Figure 9: Illustration of the alignment of visual-landmark embedding on the
CMLR dataset. Where the vertical axis represents the visual embedding and the
horizontal axis represents the landmark embedding.

##  V Conclusion

In this paper, we propose a cross-modal Transformer framework for sentence-
level lipreading, which can generalize to unseen speakers by using landmarks
as motion trajectories to calibre the visual variations. The model can level
up the alignment of heterogeneous features by cross-modal fusion suggested by
the cross-attention. Extensive experimental results show that our framework
can effectively generalize to unseen speakers. In future work, we put forward
the relevance of this research in more challenging datasets, e.g., side-view
speakers.

## References

  * [1] Y. Zhao, R. Xu, X. Wang, P. Hou, H. Tang, and M. Song, “Hearing lips: Improving lip reading by distilling speech recognizers,” 2019.
  * [2] X. Zhang, H. Gong, X. Dai, F. Yang, N. Liu, and M. Liu, “Understanding pictograph with facial features: end-to-end sentence-level lip reading of chinese,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, no. 01, 2019, pp. 9211–9218.
  * [3] S. Hilder, R. W. Harvey, and B.-J. Theobald, “Comparison of human and machine-based lip-reading.” in _AVSP_ , 2009, pp. 86–89.
  * [4] J. O. Kim, W. Lee, J. Hwang, K. S. Baik, and C. H. Chung, “Lip print recognition for security systems by multi-resolution architecture,” _Future Generation Computer Systems_ , vol. 20, no. 2, pp. 295–301, 2004\.
  * [5] O. K. Jin, W. Lee, J. Hwang, K. S. Baik, and C. H. Chung, “Lip print recognition for security systems by multi-resolution architecture,” _Future Generation Computer Systems_ , vol. 20, no. 2, pp. 295–301, 2004\.
  * [6] B. Xu, C. Lu, Y. Guo, and J. Wang, “Discriminative multi-modality speech recognition,” 2020.
  * [7] S. Petridis, T. Stafylakis, P. Ma, F. Cai, G. Tzimiropoulos, and M. Pantic, “End-to-end audiovisual speech recognition,” in _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2018, pp. 6548–6552.
  * [8] T. Stafylakis and G. Tzimiropoulos, “Combining residual networks with lstms for lipreading,” _arXiv preprint arXiv:1703.04105_ , 2017.
  * [9] T. Zhang, L. He, X. Li, and G. Feng, “Efficient end-to-end sentence-level lipreading with temporal convolutional networks,” _Applied Sciences_ , vol. 11, no. 15, p. 6975, 2021.
  * [10] X. Zhao, S. Yang, S. Shan, and X. Chen, “Mutual information maximization for effective lip reading,” in _2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)_. IEEE, 2020, pp. 420–427.
  * [11] Y. Zhao, R. Xu, and M. Song, “A cascade sequence-to-sequence model for chinese mandarin lip reading,” in _Proceedings of the ACM Multimedia Asia_ , 2019, pp. 1–6.
  * [12] K. Xu, D. Li, N. Cassimatis, and X. Wang, “Lcanet: End-to-end lipreading with cascaded attention-ctc,” in _2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)_. IEEE, 2018, pp. 548–555.
  * [13] J. Liu, Y. Ren, Z. Zhao, C. Zhang, B. Huai, and J. Yuan, “Fastlr: Non-autoregressive lipreading model with integrate-and-fire,” in _Proceedings of the 28th ACM International Conference on Multimedia_ , 2020, pp. 4328–4336.
  * [14] Y. M. Assael, B. Shillingford, S. Whiteson, and N. De Freitas, “Lipnet: End-to-end sentence-level lipreading,” _arXiv preprint arXiv:1611.01599_ , 2016.
  * [15] K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman, “Return of the devil in the details: Delving deep into convolutional nets,” _arXiv e-prints_ , 2014.
  * [16] J. Chung, C. Gulcehre, K. H. Cho, and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” _Eprint Arxiv_ , 2014\.
  * [17] A. Auvolat and T. Mesnard, “Connectionist temporal classification: Labelling unsegmented sequences with recurrent neural networks,” 2006.
  * [18] M. Cooke, J. Barker, S. Cunningham, and X. Shao, “An audio-visual corpus for speech perception and automatic speech recognition,” _The Journal of the Acoustical Society of America_ , vol. 120, no. 5, pp. 2421–2424, 2006.
  * [19] Y. Huang, X. Liang, and C. Fang, “Callip: Lipreading using contrastive and attribute learning,” in _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021, pp. 2492–2500.
  * [20] Y. Zhang, Y. Guo, Y. Jin, Y. Luo, Z. He, and H. Lee, “Unsupervised discovery of object landmarks as structural representations,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 2694–2703.
  * [21] M. A. Haghpanah, E. Saeedizade, M. T. Masouleh, and A. Kalhor, “Real-time facial expression recognition using facial landmarks and neural networks,” in _2022 International Conference on Machine Vision and Image Processing (MVIP)_. IEEE, 2022, pp. 1–7.
  * [22] V. Estellers, M. Gurban, and J. P. Thiran, “On dynamic stream weighting for audio-visual speech recognition,” _IEEE Transactions on Audio Speech and Language Processing_ , vol. 20, no. 4, pp. 1145–1157, 2012.
  * [23] P. J. Lucey, G. Potamianos, and S. Sridharan, “Patch-based analysis of visual speech from multiple views,” in _Proceedings of the International Conference on Auditory-Visual Speech Processing 2008_ , 2008.
  * [24] T. Sheerman-Chase, E.-J. Ong, and R. Bowden, “Cultural factors in the regression of non-verbal communication perception,” in _2011 IEEE international conference on computer vision workshops (ICCV workshops)_. IEEE, 2011, pp. 1242–1249.
  * [25] G. Papcun, J. Hochberg, T. R. Thomas, F. Laroche, and S. Levy, “Inferring articulation and recognizing gestures from acoustics with a neural network trained on x‐ray microbeam data,” _Journal of the Acoustical Society of America_ , vol. 92, no. 2 Pt 1, pp. 688–700, 1992.
  * [26] M. T. Chan, “Hmm-based audio-visual speech recognition integrating geometric- and appearance-based visual features,” in _IEEE Fourth Workshop on Multimedia Signal Processing_ , 2001.
  * [27] J. Luettin and N. A. Thacker, “Speechreading using probabilistic models,” _Computer Vision and Image Understanding_ , vol. 65, no. 2, pp. 163–178, 1997\.
  * [28] J. S. Chung and A. Zisserman, “Lip reading in the wild,” in _Computer Vision – ACCV 2016_ , 2017, pp. 87–103.
  * [29] I. Fung and B. Mak, “End-to-end low-resource lip-reading with maxout cnn and lstm,” 2018, pp. 2511–2515.
  * [30] M. Wand and J. Schmidhuber, “Improving speaker-independent lipreading with domain-adversarial training,” 2017.
  * [31] M. Wand, J. Schmidhuber, and N. T. Vu, “Investigations on end- to-end audiovisual fusion,” 2018.
  * [32] J. Son Chung, A. Senior, O. Vinyals, and A. Zisserman, “Lip reading sentences in the wild,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 6447–6456.
  * [33] S. Zhou, L. Dong, S. Xu, and B. Xu, “Syllable-based sequence-to-sequence speech recognition with the transformer in mandarin chinese,” _arXiv preprint arXiv:1804.10752_ , 2018.
  * [34] S. Ma, S. Wang, and X. Lin, “A transformer-based model for sentence-level chinese mandarin lipreading,” in _2020 IEEE Fifth International Conference on Data Science in Cyberspace (DSC)_ , 2020.
  * [35] P. Ma, S. Petridis, and M. Pantic, “Visual speech recognition for multiple languages in the wild,” 2022.
  * [36] L. Wu, Y. Wang, L. Shao, and M. Wang, “3d personvlad: Learning deep global representations for video-based person re-identification,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 30, no. 11, pp. 3347–3359, 2019.
  * [37] L. Wu, Y. Wang, J. Gao, M. Wang, Z.-J. Zha, and D. Tao, “Deep co-attention based comparators for relative representation learning in person re-identification,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 32, no. 2, pp. 722–735, 2021.
  * [38] D. Chen, M. Wang, H. Chen, L. Wu, J. Qin, and W. Peng, “Cross-modal retrieval with heterogeneous graph embedding,” in _Proceedings of the 30th ACM International Conference on Multimedia_ , 2022, pp. 3291–3300.
  * [39] A. Amodio, M. Ermidoro, D. Maggi, S. Formentin, and S. M. Savaresi, “Automatic detection of driver impairment based on pupillary light reflex,” _IEEE transactions on intelligent transportation systems_ , vol. 20, no. 8, pp. 3038–3048, 2018.
  * [40] F. xue, T. Yang, K. Liu, Z. Hong, M. Cao, D. Guo, and R. Hong, “Lcsnet: End-to-end lipreading with channel-aware feature selection,” _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ , 2022.