# Heavy resonances and the oblique parameters $S$ and $T$ ∗

Ignasi Rosell<EMAIL_ADDRESS>Departamento de Matemáticas, Física y Ciencias
Tecnológicas, Universidad Cardenal Herrera-CEU, CEU Universities, 46115 Alfara
del Patriarca, València, Spain  Antonio Pich<EMAIL_ADDRESS>IFIC,
Universitat de València – CSIC, Apt. Correus 22085, 46071 València, Spain
Juan José Sanz-Cillero<EMAIL_ADDRESS>Departamento de Física Teórica
and IPARCOS, Universidad Complutense de Madrid, E-28040 Madrid, Spain

###### Abstract

It has been confirmed experimentally the existence of a mass gap between
Standard Model (SM) and eventual Beyond Standard Model (BSM) fields.
Therefore, the use of effective approaches to search for fingerprints of New
Physics is very appealing. A non-linear realizations of the Electroweak
Symmetry Breaking is considered here, where the Higgs is a singlet with free
couplings and the SM fields are also coupled to bosonic heavy resonances. A
one-loop-level calculation of the oblique $S$ and $T$ parameters is presented
here. This analysis allows us to constrain resonance masses to be above the
TeV scale, $M_{R}\\!\mathrel{\hbox to0.0pt{\lower 4.0pt\hbox{\hskip
1.0pt$\sim$}\hss}\raise 1.0pt\hbox{$>$}}\\!3\,$TeV, in good agreement with our
previous determinations, where these observables were computed with a more
simplified Lagrangian.

###### keywords:

Beyond Standard Model , Effective Field Theories , Higgs Physics , Chiral
Lagrangians.

##  1 Introduction

The LHC results have confirmed the Standard Model (SM) as the correct
framework for explaining electroweak interactions within the energy ranges
examined up to now. The discovery of a Higgs-like particle [1], with couplings
very aligned with SM predictions, has effectively filled in the complete set
of fundamental fields according to the SM, and there have been no new states
discovered to date. Consequently, the available data indicate the presence of
a mass gap between the SM and any hypothetical degrees of freedom associated
with New Physics (NP). This gap serves as a rationale for employing effective
field theories to systematically investigate low-energy physics for
indications of NP scales.

Effective field theories are built upon several key components, being the
particle content, the symmetries and the power counting the most important
ones. In the context of electroweak theory, the choice of power counting
method hinges on how the Higgs field $h$ is introduced [2, 3]. There are two
different approaches: the commonly used linear realization of Electroweak
Symmetry Breaking (EWSB), where the Higgs is considered part of a doublet
together with the three electroweak (EW) Goldstone bosons $\vec{\varphi}$, as
in the SM; or the more encompassing non-linear realization, which does not
presume any specific relationship between the Higgs and the three Goldstone
fields. Here we opt for the latter approach [4], employing an expansion based
on generalized momenta. It is worth noting that the linear realization can be
seen as a special case within the broader framework of the non-linear
approach.

We examine a strongly-coupled scenario involving heavy spin-1 resonances
interacting with the SM particles. Our objective is to evaluate the bounds
placed on the masses of these heavy resonances based on the phenomenology
derived from the electroweak oblique parameters [5].

In Ref. [6] we presented a one-loop calculation of the $S$ and $T$ parameters
within strongly-coupled scenarios of this kind. Our analysis yielded robust
and model-independent constraints on both the Higgs couplings and the heavy
scales involved. By making only modest assumptions about the high-energy
behavior of the underlying fundamental theory, we demonstrated that precision
electroweak data necessitate the Higgs-like scalar to possess a $hWW$ coupling
very close to the SM one. Simultaneously, we found that the masses of vector
and axial-vector resonances should exhibit considerable degeneracy and be
above $4\,$TeV. The considerably larger dataset collected in recent years has
provided a much more precise experimental measurement of the $hWW$ coupling,
$\kappa_{W}$. Consequently, we no longer treat $\kappa_{W}$ as an independent
free parameter, opting instead to use its experimentally determined value as
an input. This adjustment enables us to broaden the scope of our analysis to
include a wider range of interactions: while our prior work solely focused on
the bosonic P-even sector [6], we incorporate now both P-even and P-odd
operators into our approach. Here we show the initial phase in the effort to
enhance and update the results reported in Ref. [6]; we plan to complete this
update by incorporating fermionic contributions as well [7].

In Section 2, we introduce our Lagrangian including the resonances, while
Section 3 outlines the calculation of the $S$ and $T$ parameters, up to next-
to-leading order (NLO), through dispersive representations. Section 4 is
dedicated to exploring the practical implications of these results from a
phenomenological perspective. Ultimately, we provide a summary in Section 5.

##  2 The theoretical framework

###  2.1 The effective resonance Lagrangian

In the non-linear realization of EWSB, operators are not ordered following
their canonical dimensions and they must be organized taking into account
their behavior at low momenta, the chiral dimensions [8]. Here we focus on the
NLO resonance contributions to the $S$ and $T$ parameters derived exclusively
from the lightest bosonic absorptive cuts ($\varphi\varphi$ and $h\varphi$).
Consequently, we only need to consider bosonic operators involving at most one
spin-1 resonance field. Following the notation of Ref. [4], the relevant CP-
conserving Lagrangian for our calculations reads

$\displaystyle\Delta\mathcal{L}_{\mathrm{RT}}=\frac{v^{2}}{4}\,\left(1\\!+\\!\frac{\displaystyle
2\kappa_{W}}{\displaystyle
v}h\right)\langle\,u_{\mu}u^{\mu}\,\rangle_{2}\qquad\qquad\quad\quad$
$\displaystyle+\,\langle\,V^{1}_{3\,\mu\nu}\left(\frac{\displaystyle
F_{V}}{\displaystyle 2\sqrt{2}}f_{+}^{\mu\nu}\\!+\\!\frac{\displaystyle
iG_{V}}{\displaystyle
2\sqrt{2}}[u^{\mu},u^{\nu}]\\!+\\!\frac{\displaystyle\widetilde{F}_{V}}{\displaystyle
2\sqrt{2}}f_{-}^{\mu\nu}\right.$
$\displaystyle\left.+\frac{\displaystyle\widetilde{\lambda}_{1}^{hV}}{\displaystyle\sqrt{2}}\left[(\partial^{\mu}h)u^{\nu}\\!-\\!(\partial^{\nu}h)u^{\mu}\right]\right)\,\rangle_{2}$
$\displaystyle+\,\langle\,A^{1}_{3\,\mu\nu}\left(\frac{\displaystyle
F_{A}}{\displaystyle
2\sqrt{2}}f_{-}^{\mu\nu}\\!+\\!\frac{\displaystyle\lambda_{1}^{hA}}{\displaystyle\sqrt{2}}\left[(\partial^{\mu}h)u^{\nu}\\!-\\!(\partial^{\nu}h)u^{\mu}\right]\right.$
$\displaystyle \left.+\frac{\displaystyle\widetilde{F}_{A}}{\displaystyle
2\sqrt{2}}f_{+}^{\mu\nu}\\!+\\!\frac{\displaystyle
i\widetilde{G}_{A}}{\displaystyle
2\sqrt{2}}[u^{\mu},u^{\nu}]\right)\,\rangle_{2}\,,$ (1)

where $V^{1}_{3\,\mu\nu}$ and $A^{1}_{3\,\mu\nu}$ introduce color-singlet
custodial-triplet resonances with $J^{PC}$ quantum numbers $1^{--}$ (V) and
$1^{++}$ (A) by using an antisymmetric formalism. The Goldstone fields are
parametrized through the SU(2) matrix
$U=u^{2}=\exp{(i\vec{\sigma}\vec{\varphi}}/v)$, where
$v=(\sqrt{2}G_{F})^{-1/2}=246\;\mathrm{GeV}$ is the EWSB scale,
$u_{\mu}=-iu^{\dagger}D_{\mu}Uu^{\dagger}$ with $D_{\mu}$ the appropriate
covariant derivative, and $f_{\pm}^{\mu\nu}$ contain the gauge-boson field
strengths. Note that couplings with tilde are related to odd-parity operators.

###  2.2 Short-distance constraints

From (1) one observes the presence of ten resonance parameters (including
masses), so that the use of asymptotic constraints is fundamental to reduce
the number of unknown parameters and, consequently, to be able to obtain
phenomenological bounds. Moreover, this resonance Lagrangian is assumed to be
an interpolation between the low- and the high-energy regimes and, as a
result, considering a reasonable high-energy behavior is an important
ingredient of this effective approach:

  1. 1.

Requiring that two-Goldstone ($\varphi\varphi$) and Higgs-Goldstone
($h\varphi$) vector and axial form factors vanish at high energies allow us to
determine the couplings $G_{V}$, $\widetilde{G}_{A}$, $\lambda_{1}^{hA}$ and
$\widetilde{\lambda}_{1}^{hV}$ in terms of the remaining parameters [9],

$\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\frac{G_{V}}{F_{A}}\\!=\\!-\frac{\widetilde{G}_{A}}{\widetilde{F}_{V}}\\!=\\!\frac{\lambda_{1}^{hA}v}{\kappa_{W}F_{V}}\\!=\\!-\frac{\widetilde{\lambda}_{1}^{hV}v}{\kappa_{W}\widetilde{F}_{A}}\\!=\\!\frac{v^{2}}{F_{V}F_{A}\\!-\\!\widetilde{F}_{V}\widetilde{F}_{A}}.$
(2)

  2. 2.

The assumed chiral symmetry of the underlying electroweak theory suggests that
the $W^{3}B$ correlator is an order parameter of the EWSB and it vanishes at
high energies in asymptotically-free gauge theories as $1/s^{3}$ [10], giving
rise to the 1st and 2nd Weinberg Sum Rules (WSRs) [11]:

    1. (a)

1st WSR (vanishing of the $1/s$ term). At leading-order (LO) it implies [9]:

$\\!\\!\\!\\!\\!\left(F_{V}^{2}-\widetilde{F}_{V}^{2}\right)-\left(F_{A}^{2}-\widetilde{F}_{A}^{2}\right)\,=\,v^{2}\,,$
(3)

whereas at NLO it implies the more involved [7]

$\\!\\!\\!\\!\\!\left(F_{V}^{2}\\!-\\!\widetilde{F}_{V}^{2}\right)\\!-\\!\left(F_{A}^{2}\\!-\\!\widetilde{F}_{A}^{2}\right)=v^{2}\left(1\\!+\\!\delta_{{}_{\rm
NLO}}^{(1)}\right)\,,$ (4)

plus an additional conditions $\widetilde{\delta}_{{}_{\rm NLO}}^{(1)}=0$.
Note that $\delta_{{}_{\rm NLO}}^{(1)}$ and $\widetilde{\delta}_{{}_{\rm
NLO}}^{(1)}$ are obtained from the LO high-energy expansion of the one-loop
contribution of the related correlator [6].

    2. (b)

2nd WSR (vanishing of the $1/s^{2}$ term). At LO it implies [9]:

$\\!\\!\\!\\!\\!\left(F_{V}^{2}-\widetilde{F}_{V}^{2}\right)M_{V}^{2}-\left(F_{A}^{2}-\widetilde{F}_{A}^{2}\right)M_{A}^{2}\,=\,0\,.$
(5)

At NLO (5) convers into [7]

$\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\left(F_{V}^{2}\\!-\\!\widetilde{F}_{V}^{2}\right)M_{V}^{2}-\left(F_{A}^{2}\\!-\\!\widetilde{F}_{A}^{2}\right)M_{A}^{2}=v^{2}M_{V}^{2}\delta_{{}_{\rm
NLO}}^{(2)},$ (6)

plus an additional conditions $\widetilde{\delta}_{{}_{\rm NLO}}^{(2)}=0$.
Note again that $\delta_{{}_{\rm NLO}}^{(2)}$ and $\widetilde{\delta}_{{}_{\rm
NLO}}^{(2)}$ are obtained from the NLO high-energy expansion of the one-loop
contribution of the related correlator [6].

Whereas the 1st WSR is expected to be satisfied in gauge theories with
nontrivial ultraviolet (UV) fixed points, the applicability of the 2nd WSR
hinges on the nature of the UV theory under consideration.

It is interesting to stress that in the absence of P-odd couplings (3) and (5)
together require $M_{A}\\!>\\!M_{V}$ and this mass hierarchy continues to hold
if odd-parity couplings are assumed to be smaller than even-parity ones,
$\widetilde{F}_{V,A}\ll F_{V,A}$, which is a reasonable assumption we make.

##  3 Oblique Electroweak Observables: $S$ and $T$ at NLO

###  3.1 The observables

We adopt here the notation used in Ref. [6]. The calculations are carried out
in the Landau gauge, which ensures that the gauge boson propagators are
transverse and their self-energies,

$\displaystyle\mathcal{L}_{\mathrm{v.p.}}=$
$\displaystyle-\frac{1}{2}W^{3}_{\mu}\,\Pi^{\mu\nu}_{33}(s)W^{3}_{\nu}-\frac{1}{2}B_{\mu}\,\Pi^{\mu\nu}_{00}(s)B_{\nu}$
$\displaystyle\quad-W^{3}_{\mu}\,\Pi^{\mu\nu}_{30}(s)B_{\nu}-W^{+}_{\mu}\,\Pi^{\mu\nu}_{WW}(s)W^{-}_{\nu}\,,\phantom{\frac{1}{2}}$
(7)

can be written as

$\Pi^{\mu\nu}_{ij}(q^{2})\,=\,\left(-g^{\mu\nu}+\frac{q^{\mu}q^{\nu}}{q^{2}}\right)\;\Pi_{ij}(q^{2}).$
(8)

The definitions of $S$ and $T$ involve $e_{3}$ and $e_{1}$,

$e_{3}\,=\,\frac{\displaystyle g}{\displaystyle
g^{\prime}}\;\widetilde{\Pi}_{30}(0)\,,\qquad
e_{1}\,=\,\frac{\Pi_{33}(0)-\Pi_{WW}(0)}{M_{W}^{2}}\,,$ (9)

where one has removed the tree-level Goldstone contribution from $\Pi_{30}(s)$
[5]:

$\Pi_{30}(s)\,=\,s\,\widetilde{\Pi}_{30}(s)\,+\,\frac{g^{2}\tan{\theta_{W}}}{4}\,v^{2}\,.$
(10)

The $S$ and $T$ parameters are determined by the discrepancies between $e_{3}$
and $e_{1}$ and their respective SM contributions, denoted as $e_{3}^{\rm SM}$
and $e_{1}^{\rm SM}$, respectively:

$\\!\\!\\!\\!\\!S=\frac{\displaystyle 16\pi}{\displaystyle
g^{2}}\big{(}e_{3}-e_{3}^{\rm SM}\big{)},\quad T=\frac{\displaystyle
4\pi}{\displaystyle g^{2}\sin^{2}{\theta_{W}}}\big{(}e_{1}-e_{1}^{\rm
SM}\big{)}.$ (11)

###  3.2 Dispersive relations

For the calculations of $S$ and $T$ we use dispersive representations [5, 6]:

$\displaystyle\\!\\!\\!\\!\\!S$ $\displaystyle=$
$\displaystyle\frac{\displaystyle 16\pi}{\displaystyle
g^{2}\tan\theta_{W}}\;\displaystyle{\int}_{0}^{\infty}\frac{\displaystyle{\rm
ds}}{\displaystyle s}\;[\rho_{S}(s)\,-\,\rho_{S}(s)^{\rm SM}]\,,$
$\displaystyle\\!\\!\\!\\!\\!T$ $\displaystyle=$
$\displaystyle\frac{\displaystyle 4\pi}{\displaystyle
g^{\prime\,2}\cos^{2}\theta_{W}}\;\displaystyle{\int}_{0}^{\infty}\frac{\displaystyle{\rm
ds}}{\displaystyle s^{2}}\;[\rho_{T}(s)\,-\,\rho_{T}(s)^{\rm SM}]\,,$ (12)

with the spectral functions

$\displaystyle\\!\\!\\!\\!\\!\rho_{S}(s)$ $\displaystyle=$
$\displaystyle\frac{\displaystyle
1}{\displaystyle\pi}\,\mbox{Im}\widetilde{\Pi}_{30}(s)\,,$
$\displaystyle\\!\\!\\!\\!\\!\rho_{T}(s)$ $\displaystyle=$
$\displaystyle\frac{1}{\pi}\mbox{Im}[\Sigma^{(0)}(s)-\Sigma^{(+)}(s)]\,,$ (13)

being $\Sigma$ the corresponding Goldstone self-energy. At LO in $g$ and
$g^{\prime}$ the SM one-loop spectral function read

$\displaystyle\\!\\!\\!\\!\\!\rho_{S}(s)^{\rm SM}\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!\frac{g^{2}\tan\theta_{W}}{192\pi^{2}}\left[\theta(s)\\!-\\!\left(1\\!-\\!\frac{m_{h}^{2}}{s}\right)^{3}\theta\left(s\\!-\\!m_{h}^{2}\right)\right]\\!,$
$\displaystyle\\!\\!\\!\\!\\!\rho_{T}(s)^{\rm SM}\\!\\!$ $\displaystyle=$
$\displaystyle\\!\\!\frac{\displaystyle 3g^{\prime\,2}s}{\displaystyle
64\pi^{2}}\;\bigg{[}-\theta(s)+\left(1-\frac{m_{h}^{4}}{s^{2}}\right)\theta(s-m_{h}^{2})\bigg{]}\,.$

###  3.3 Leading-order calculation

At LO $T$ vanishes ($T_{\mathrm{LO}}=0$), while there is a LO contribution to
$S$,

$\\!\\!\\!\\!\\!\\!\\!\left.\Pi_{30}(s)\right|_{\mathrm{LO}}=\frac{g^{2}\tan{\theta_{W}}}{4}s\left(\\!\frac{v^{2}}{s}\\!+\\!\frac{F_{V}^{2}\\!-\\!\widetilde{F}_{V}^{2}}{M_{V}^{2}\\!-\\!s}\\!-\\!\frac{F_{A}^{2}\\!-\\!\widetilde{F}_{A}^{2}}{M_{A}^{2}\\!-\\!s}\right).$
(15)

so that, and by using (9-11),

$S_{\mathrm{LO}}\,=\,4\pi\left(\frac{F_{V}^{2}-\widetilde{F}_{V}^{2}}{M_{V}^{2}}-\frac{F_{A}^{2}-\widetilde{F}_{A}^{2}}{M_{A}^{2}}\right)\,.$
(16)

If one assumes the 1st and the 2nd WSRs of (3) and (5), $S_{\mathrm{LO}}$ is
determined in terms of only resonance masses,

$S_{\mathrm{LO}}=4\pi
v^{2}\left(\frac{1}{M_{V}^{2}}+\frac{1}{M_{A}^{2}}\right).$ (17)

If only the 1st WSR is considered, and assuming $M_{A}>M_{V}$ and
$\widetilde{F}_{A}^{2}<F_{A}^{2}$, (3) allows us to obtain a bound:

$\\!\\!\\!\\!\\!S_{\mathrm{LO}}=4\pi\left\\{\\!\frac{v^{2}}{M_{V}^{2}}\\!+\\!\left(\\!F_{A}^{2}\\!-\\!\widetilde{F}_{A}^{2}\\!\right)\left(\\!\frac{1}{M_{V}^{2}}\\!-\\!\frac{1}{M_{A}^{2}}\right)\right\\}>\frac{4\pi
v^{2}}{M_{V}^{2}}.$ (18)

###  3.4 Next-to-leading-order calculation

At NLO the number of resonance parameters increases and we have opted for an
expansion in small $\widetilde{F}_{V,A}/F_{V,A}$, taking into account that
odd-parity couplings are expected in our study to be smaller than even-parity
couplings, that is, $\widetilde{F}_{V,A}/F_{V,A}\ll 1$. If one considers the
1st and the 2nd WSRs of (4) and (6), the bosonic contributions studied here
determine $S$ in terms of only $\kappa_{W}$, $M_{V,A}$ and the expansion
parameters $\widetilde{F}_{V,A}/F_{V,A}$:

$\displaystyle\\!\\!\\!\\!\\!\\!\\!S_{\mathrm{NLO}}\\!\\!\\!$
$\displaystyle\\!\\!\\!=\\!\\!\\!$ $\displaystyle\\!\\!\\!4\pi
v^{2}\bigg{(}\frac{1}{M_{V}^{2}}+\frac{1}{M_{A}^{2}}\bigg{)}+S_{\mathrm{NLO}}^{\mathrm{P-even}}+S_{\mathrm{NLO}}^{\mathrm{P-odd}}\,,$
$\displaystyle\\!\\!\\!\\!\\!\\!\\!S_{\mathrm{NLO}}^{\mathrm{P-even}}\\!\\!\\!\\!$
$\displaystyle\\!\\!\\!=\\!\\!\\!$
$\displaystyle\\!\\!\\!\frac{1}{12\pi}\left[\left(1-\kappa_{W}^{2}\right)\left(\log\frac{M_{V}^{2}}{m_{h}^{2}}-\frac{11}{6}\right)\right.$
$\displaystyle\qquad\left.+\kappa_{W}^{2}\left(\frac{M_{A}^{2}}{M_{V}^{2}}-1\right)\log\frac{M_{A}^{2}}{M_{V}^{2}}\right]\,,$
$\displaystyle\\!\\!\\!\\!\\!\\!\\!S_{\mathrm{NLO}}^{\mathrm{P-odd}}\\!\\!\\!$
$\displaystyle\\!\\!\\!=\\!\\!\\!$
$\displaystyle\\!\\!\\!\frac{1}{12\pi}\left(\frac{\widetilde{F}_{V}^{2}}{F_{V}^{2}}+2\kappa_{W}^{2}\frac{\widetilde{F}_{V}\widetilde{F}_{A}}{F_{V}F_{A}}-\kappa_{W}^{2}\frac{\widetilde{F}_{A}^{2}}{F_{A}^{2}}\right)\times$
(19)
$\displaystyle\qquad\left(\frac{M_{A}^{2}}{M_{V}^{2}}\\!-\\!1\right)\log\frac{M_{A}^{2}}{M_{V}^{2}}\\!+\\!\mathcal{O}\left(\\!\frac{\widetilde{F}^{4}_{V,A}}{F^{4}_{V,A}}\\!\right)\,.$

As at LO, assuming only the 1st WSR of (4) and $M_{A}\\!>\\!M_{V}$ allow us to
get a lower bound for $S_{\mathrm{NLO}}$,

$\displaystyle\\!\\!\\!\\!\\!\\!\\!S_{\mathrm{NLO}}\\!\\!\\!$
$\displaystyle\\!\\!\\!>\\!\\!\\!$ $\displaystyle\\!\\!\\!\frac{4\pi
v^{2}}{M_{V}^{2}}+\Delta S_{\mathrm{NLO}}^{\mathrm{P-even}}+\Delta
S_{\mathrm{NLO}}^{\mathrm{P-odd}}\,,$
$\displaystyle\\!\\!\\!\\!\\!\\!\\!\Delta
S_{\mathrm{NLO}}^{\mathrm{P-even}}\\!\\!\\!\\!$
$\displaystyle\\!\\!\\!=\\!\\!\\!$
$\displaystyle\\!\\!\\!\frac{1}{12\pi}\left[\left(1-\kappa_{W}^{2}\right)\left(\log\frac{M_{V}^{2}}{m_{h}^{2}}-\frac{11}{6}\right)\right.$
$\displaystyle\qquad\left.-\,\kappa_{W}^{2}\,\Bigg{(}\log\frac{M_{A}^{2}}{M_{V}^{2}}-1+\frac{M_{A}^{2}}{M_{V}^{2}}\Bigg{)}\right]\,,$
$\displaystyle\\!\\!\\!\\!\\!\\!\\!\Delta
S_{\mathrm{NLO}}^{\mathrm{P-odd}}\\!\\!\\!$ $\displaystyle\\!\\!\\!=\\!\\!\\!$
$\displaystyle\\!\\!\\!\frac{1}{12\pi}\left\\{\left[\frac{\widetilde{F}_{V}^{2}}{F_{V}^{2}}+\kappa_{W}^{2}\frac{\widetilde{F}_{A}}{F_{A}}\left(2\frac{\widetilde{F}_{V}}{F_{V}}-\frac{\widetilde{F}_{A}}{F_{A}}\right)\right]\times\right.$
$\displaystyle\qquad\left.\left.\left(1-\frac{M_{A}^{2}}{M_{V}^{2}}\right)\\!+\\!\log\frac{M_{A}^{2}}{M_{V}^{2}}\times\right.\right.$
$\displaystyle\qquad\left.\left(\frac{\widetilde{F}_{V}^{2}}{F_{V}^{2}}\\!-\\!\kappa_{W}^{2}\frac{\widetilde{F}_{A}^{2}}{F_{A}^{2}}\\!-\\!2\frac{\widetilde{F}_{V}\widetilde{F}_{A}}{F_{V}F_{A}}\right)\right\\}\\!+\\!\mathcal{O}\left(\\!\frac{\widetilde{F}^{4}_{V,A}}{F^{4}_{V,A}}\\!\right)\\!.$
(20)

Independently of the assumption about the WSRs, $T$ is determined again in
terms of only $\kappa_{W}$, $M_{V,A}$ and $\widetilde{F}_{V,A}/F_{V,A}$:

$\displaystyle\\!\\!\\!\\!\\!\\!\\!T_{\mathrm{NLO}}\\!\\!\\!$
$\displaystyle\\!\\!\\!=\\!\\!\\!$
$\displaystyle\\!\\!\\!T_{\mathrm{NLO}}^{\mathrm{P-even}}+T_{\mathrm{NLO}}^{\mathrm{P-odd}}\,,$
$\displaystyle\\!\\!\\!\\!\\!\\!\\!T_{\mathrm{NLO}}^{\mathrm{P-even}}\\!\\!\\!\\!$
$\displaystyle\\!\\!\\!=\\!\\!\\!$
$\displaystyle\\!\\!\\!\\!\frac{3}{16\pi\\!\cos^{2}\\!\theta_{W}}\\!\left[\\!\left(1\\!-\\!\kappa_{W}^{2}\right)\\!\left(\\!1\\!-\\!\log\\!{\frac{M_{V}^{2}}{m_{h}^{2}}}\\!\right)\\!+\\!\kappa_{W}^{2}\log\\!{\frac{M_{A}^{2}}{M_{V}^{2}}}\right]\\!,$
$\displaystyle\\!\\!\\!\\!\\!\\!\\!T_{\mathrm{NLO}}^{\mathrm{P-odd}}\\!\\!\\!$
$\displaystyle\\!\\!\\!=\\!\\!\\!$
$\displaystyle\\!\\!\\!\frac{3}{16\pi\\!\cos^{2}\\!\theta_{W}}\Bigg{\\{}2\kappa_{W}^{2}\\!\frac{\widetilde{F}_{A}}{F_{A}}\\!-\\!2\frac{\widetilde{F}_{V}}{F_{V}}\\!+\\!\frac{M_{V}^{2}}{M_{A}^{2}\\!-\\!M_{V}^{2}}\times$
$\displaystyle\qquad\log\frac{M_{A}^{2}}{M_{V}^{2}}\left(2\frac{\widetilde{F}_{V}}{F_{V}}\\!-\\!2\kappa_{W}^{2}\frac{M_{A}^{2}}{M_{V}^{2}}\frac{\widetilde{F}_{A}}{F_{A}}\right)$
$\displaystyle\qquad\\!+\\!\frac{M_{V}^{2}}{M_{A}^{2}\\!-\\!M_{V}^{2}}\log\frac{M_{A}^{2}}{M_{V}^{2}}\left[\,\left(\kappa_{W}^{2}\frac{\widetilde{F}_{A}^{2}}{F_{A}^{2}}\\!-\\!\frac{\widetilde{F}_{V}^{2}}{F_{V}^{2}}\right)\times\right.$
$\displaystyle\qquad\left.\left(1\\!+\\!\frac{M_{A}^{2}}{M_{V}^{2}}\right)+2\,\frac{\widetilde{F}_{V}\widetilde{F}_{A}}{F_{V}F_{A}}\left(\kappa_{W}^{2}\frac{M_{A}^{2}}{M_{V}^{2}}\\!-\\!1\right)\right]$
$\displaystyle\qquad+2\left(\frac{\widetilde{F}_{V}^{2}}{F_{V}^{2}}\\!-\\!\\!\kappa_{W}^{2}\frac{\widetilde{F}_{A}^{2}}{F_{A}^{2}}\\!+\\!\left(1\\!-\\!\kappa_{W}^{2}\right)\\!\frac{\widetilde{F}_{V}\widetilde{F}_{A}}{F_{V}F_{A}}\\!\right)\\!\Bigg{\\}}$
$\displaystyle $
$\displaystyle\qquad+\mathcal{O}\left(\\!\frac{\widetilde{F}^{3}_{V,A}}{F^{3}_{V,A}}\\!\right).$
(21)

##  4 Phenomenology

Now we would like to analyze the bounds on the resonance masses by comparing
the theoretical determinations given in (19-21) with the experimental values
of the oblique parameters. For $S$ and $T$ we consider the experimental values
reported by the Particle Data Group [12], $S=-0.01\pm 0.07$ and $T=0.04\pm
0.06$ (with a correlation of $0.92$), while for the $hWW$ coupling we use the
value given in Ref. [13], $\kappa_{W}=1.01\pm 0.06$. Let us remind the
assumptions performed in the NLO calculation: only the lightest bosonic
absorptive cuts are taken into account ($\varphi\varphi$ and $h\varphi$); odd-
parity couplings are supposed to be subleading and, consequently, an expansion
in $\widetilde{F}_{V,A}/F_{V,A}$ is followed; and one assumes that
$M_{A}>M_{V}$ (when both WSRs are considered, this is not an assumption, but a
requirement).

###  4.1 Phenomenology at LO

At LO $T$ vanishes and $S$ is given in (17), if one assumes both WSRs, and
(18), if one considers only the 1st WSR. These results are identical to the
ones we obtained in Ref. [6], since the inclusion of P-odd operators has not
changed the LO predictions. In Figure 1 we show these predictions, together
with the experimentally allowed region at 68% and 95% CL [12]. The gray area
assumes both WSRs and $M_{A}\\!>\\!M_{V}$. The colored curves indicate
explicitly the predicted results for $M_{A}=M_{V}$ (orange),
$M_{A}=1.1\,M_{V}$ (blue), $M_{A}=1.2\,M_{V}$ (red) and $M_{A}\to\infty$ (dark
gray). When only the 1st WSR is considered, the allowed range gets enlarged to
the brown region. Note that the experimental data imply $M_{V}\\!>\\!2\,$TeV.

Figure 1: LO predictions for $S$. The green area covers the experimentally
allowed region, at 68% and 95% CL [12]. The gray region assumes the two WSRs
and we indicate explicitly the corresponding lines for $M_{A}=M_{V}$ (orange),
$M_{A}=1.1\,M_{V}$ (blue), $M_{A}=1.2\,M_{V}$ (red) and $M_{A}\to\infty$ (dark
gray). If only the 1st WSR is considered, the allowed region is given by both,
the gray and the brown areas.

###  4.2 Phenomenology at NLO considering both WSRs

In this case the expressions of $S$ and $T$ are reported in (19) and (21),
respectively. Note that they are given in terms of only four free parameters:
$M_{V}$, $M_{A}$, $\widetilde{F}_{V}/F_{V}$ and $\widetilde{F}_{A}/F_{A}$, but
the last two ones are expected to be small in our expansion and a normal
distribution with $\widetilde{F}_{V,A}/F_{V,A}=0.00\pm 0.33$ is assumed.
Moreover, the additional constraint $\widetilde{\delta}_{{}_{\rm
NLO}}^{(2)}=0$ allows us to fix $M_{A}$ too (implying very close values of
$M_{A}$ to $M_{V}$). In Figure 2 we show the results. The experimental data
imply much higher values for $M_{V}$.

Figure 2: NLO determinations of $S$ and $T$ assuming both WSRs. The ellipses
give the experimentally allowed regions of $S$ and $T$ at $68$%, $95$% and
$99$% CL [12]. The different colors of the points correspond to different
values of $M_{V}$: $M_{V}=2$ (red), $3$ (orange), $4$ (yellow), $5$ (green),
$6$ (blue) and $7$ (purple) TeV. The values of $\kappa_{W}$ and
$\widetilde{F}_{V,A}/F_{V,A}$ have been generated considering normal
distributions given by $\kappa_{W}=1.01\pm 0.06$ [13] and
$\widetilde{F}_{V,A}/F_{V,A}=0.00\pm 0.33$.

###  4.3 Phenomenology at NLO considering only the 1st WSR

Now the lower bound of $S$ in given in (20) and the expresssion of $T$ is
shown in (21). Please take note that again they are expressed using just four
resonance parameters: $M_{V}$, $M_{A}$, $\widetilde{F}_{V}/F_{V}$ and
$\widetilde{F}_{A}/F_{A}$. As in the previous case for
$\widetilde{F}_{V}/F_{V}$ and $\widetilde{F}_{A}/F_{A}$ we consider a normal
distribution with $\widetilde{F}_{V,A}/F_{V,A}=0.00\pm 0.33$. The comparison
between our determinations and the experimental values is shown in Figure 3,
leading to $M_{V}\\!\mathrel{\hbox to0.0pt{\lower 4.0pt\hbox{\hskip
1.0pt$\sim$}\hss}\raise 1.0pt\hbox{$>$}}\\!3\,$TeV.

Figure 3: NLO determinations of $S$ assuming only the 1st WSR, so only lower
bounds of $S$ are shown. The ellipses give the experimentally allowed regions
of $S$ and $T$ at $68$%, $95$% and $99$% CL [12]. The different colors of the
points correspond to different values of $M_{V}$: $M_{V}=2$ (red), $3$
(orange), $4$ (yellow), $5$ (green), $6$ (blue) and $7$ (purple) TeV. We have
considered different values of $M_{A}$ in terms of $M_{V}$: $M_{A}=M_{V}$
(left), $M_{A}=1.2M_{V}$ (centered) and $M_{A}=1.5M_{V}$ (right). The values
of $\kappa_{W}$ and $\widetilde{F}_{V,A}/F_{V,A}$ have been generated
considering normal distributions given by $\kappa_{W}=1.01\pm 0.06$ [13] and
$\widetilde{F}_{V,A}/F_{V,A}=0.00\pm 0.33$.

##  5 Conclusions

We conduct a calculation of the $S$ and $T$ oblique parameters at the next-to-
leading-order level, employing an effective methodology that incorporates
spin-1 heavy resonances. Our approach adopts a broad non-linear realization of
Electroweak Symmetry Breaking (EWSB), devoid of any presumptions regarding the
precise connection between the Higgs and the Goldstones. This results
represents a first step in the update of the results reported in Ref. [6],
encompassing a more general Lagrangian [4] and taking into account the latest
experimental constraints of Ref. [12]. In the near future we will conclude
this update by considering fermionic contributions too [7].

It is worth highlighting that employing dispersive relations has eliminated
the reliance on arbitrary cut-off values, which can introduce unphysical
aspects. Additionally, we incorporate essential high-energy constraints by
presuming the presence of well-behaved form factors [9] and invoking the
Weinberg Sum Rules (WSRs) [11]. This strategic approach enables us to express
$S$ and $T$ using only a small set of resonance parameters: $M_{V}$, $M_{A}$,
$\widetilde{F}_{V}/F_{V}$ and $\widetilde{F}_{A}/F_{A}$.

We assume that odd-parity couplings are subleading, so that we can employ an
expansion in $\widetilde{F}_{V,A}/F_{V,A}$. This assumption allows us to
consider a normal distribution for $\widetilde{F}_{A}/F_{A}$, with
$\widetilde{F}_{V,A}/F_{V,A}=0.00\pm 0.33$. Within the context of both
Weinberg Sum Rules (WSRs), $M_{A}$ must be a little larger than $M_{V}$ and
our findings are graphically illustrated in Figure 2. In the case where we
disregard the 2nd WSR, we assume that $M_{A}>M_{V}$ and we examine the
alignment between our predictions and experimental constraints in Figure 3.

The primary inference drawn from our analysis is that the present electroweak
precision data permits the existence of massive resonances at the natural
electroweak scale, $M_{R}\\!\mathrel{\hbox to0.0pt{\lower 4.0pt\hbox{\hskip
1.0pt$\sim$}\hss}\raise 1.0pt\hbox{$>$}}\\!3\,$TeV. Our findings align with
the conclusions drawn in our earlier studies of Refs. [9, 6].

## Acknowledtments

We wish to thank the organizers for the pleasant conference. This work has
been supported in part by the Spanish Government (PID2019-108655GB-I00,
PID2020-114473GB-I00, PID2022-137003NB-I00); by the Generalitat Valenciana
(PROMETEU/2021/071); by the Universidad Cardenal Herrera-CEU (INDI22/15); and
by the ESI International Chair@CEU-UCH.

## Declaration of Generative AI and AI-assisted technologies in the writing
process

During the preparation of this work the authors used ChatGPT in order to
improve readability and language. After using this tool, the authors reviewed
and edited the content as needed and take full responsibility for the content
of the publication.

## References

  * [1] G. Aad et al. [ATLAS Collaboration], “Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC,” Phys. Lett. B 716 (2012) 1 [arXiv:1207.7214 [hep-ex]]; S. Chatrchyan et al. [CMS Collaboration], “Observation of a New Boson at a Mass of 125 GeV with the CMS Experiment at the LHC,” Phys. Lett. B 716 (2012) 30 [arXiv:1207.7235 [hep-ex]].
  * [2] G. Buchalla, O. Catà, A. Celis and C. Krause, “Standard Model Extended by a Heavy Singlet: Linear vs. Nonlinear EFT,” Nucl. Phys. B 917 (2017) 209 [arXiv:1608.03564 [hep-ph]].
  * [3] A. Pich, “Effective Field Theory with Nambu-Goldstone Modes,” Lecture Notes of the Les Houches Summer School, Vol. 108, Session CVIII (Oxford University Press, UK, 2020), 137-219 [arXiv:1804.05664 [hep-ph]].
  * [4] A. Pich, I. Rosell, J. Santos and J. J. Sanz-Cillero, “Fingerprints of heavy scales in electroweak effective Lagrangians,” JHEP 1704 (2017) 012 [arXiv:1609.06659 [hep-ph]]; C. Krause, A. Pich, I. Rosell, J. Santos and J. J. Sanz-Cillero, “Colorful Imprints of Heavy States in the Electroweak Effective Theory,” JHEP 1905 (2019) 092 [arXiv:1810.10544 [hep-ph]].
  * [5] M. E. Peskin and T. Takeuchi, “A New constraint on a strongly interacting Higgs sector,” Phys. Rev. Lett. 65 (1990) 964; “Estimation of oblique electroweak corrections,” Phys. Rev. D 46 (1992) 381.
  * [6] A. Pich, I. Rosell and J. J. Sanz-Cillero, “Viability of strongly-coupled scenarios with a light Higgs-like boson,” Phys. Rev. Lett. 110 (2013) 181801 [arXiv:1212.6769 [hep-ph]]; “Oblique S and T Constraints on Electroweak Strongly-Coupled Models with a Light Higgs,” JHEP 1401 (2014) 157 [arXiv:1310.3121 [hep-ph]].
  * [7] A. Pich, I. Rosell and J. J. Sanz-Cillero, work in preparation.
  * [8] S. Weinberg, “Phenomenological Lagrangians,” Physica A 96 (1979) no.1-2, 327.
  * [9] A. Pich, I. Rosell, J. Santos and J. J. Sanz-Cillero, “Low-energy signals of strongly-coupled electroweak symmetry-breaking scenarios,” Phys. Rev. D 93 (2016) no.5, 055041 [arXiv:1510.03114 [hep-ph]]; A. Pich, I. Rosell and J. J. Sanz-Cillero, “Bottom-up approach within the electroweak effective theory: Constraining heavy resonances,” Phys. Rev. D 102 (2020) no.3, 035012 [arXiv:2004.02827 [hep-ph]].
  * [10] C. W. Bernard, A. Duncan, J. LoSecco and S. Weinberg, “Exact Spectral Function Sum Rules,” Phys. Rev. D 12 (1975) 792.
  * [11] S. Weinberg, “Precise relations between the spectra of vector and axial vector mesons,” Phys. Rev. Lett. 18 (1967) 507.
  * [12] R. L. Workman et al. [Particle Data Group], “Review of Particle Physics,” PTEP 2022 (2022), 083C01
  * [13] J. de Blas, O. Eberhardt and C. Krause, “Current and Future Constraints on Higgs Couplings in the Nonlinear Effective Theory,” JHEP 1807 (2018) 048 [arXiv:1803.00939 [hep-ph]].


# EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-
to-Speech Models

Maureen de Seyssel1,2 Antony D’Avirro1 Adina Williams1 Emmanuel Dupoux1,2
1Meta AI Research
2ENS, EHESS, CNRS, PSL University, France
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We introduce EmphAssess, a prosodic benchmark designed to evaluate the
capability of speech-to-speech models to encode and reproduce prosodic
emphasis. We apply this to two tasks: speech resynthesis and speech-to-speech
translation. In both cases, the benchmark evaluates the ability of the model
to encode emphasis in the speech input and accurately reproduce it in the
output, potentially across a change of speaker and language. As part of the
evaluation pipeline, we introduce EmphaClass, a new model that classifies
emphasis at the frame or word level.

EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-
Speech Models

Maureen de Seyssel1,2 Antony D’Avirro1 Adina Williams1 Emmanuel Dupoux1,2
1Meta AI Research 2ENS, EHESS, CNRS, PSL University, France
<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

In recent years, significant advancements have been made in the development of
Self-Supervised Learning (SSL) models for speech, extending beyond the
traditional text-only methods prevalent in the field Mohamed et al. (2022).
Such speech-based models find successful application across various domains
from generative language modelling Lakhotia et al. (2021); Borsos et al.
(2023); Nguyen et al. (2023b) to speech-to-speech translation (S2ST) Jia et
al. (2019, 2022); Lee et al. (2021); Rubenstein et al. (2023); Barrault et al.
(2023). Unlike text-only models, they exploit additional cues present in the
speech signal which are absent in textual input.

One crucial speech-only cue is prosody. Also termed the “music of speech”
(Wennerstrom, 2001), prosody is marked by the perceived loudness, rhythm, and
pitch of speech. Prosody not only adds naturalness to an utterance but also
has the capacity to modify the meaning of the conveyed message, both at a
global level, such as in the expression of different emotions, and at a local
level, by influencing the interpretation of individual phrases or words
(Cutler et al., 1997; Dahan, 2015). For instance, slower speech may suggest
hesitation, while altering something like pause placement can actually change
the segmentation into words or syntactic constituents, with downstream
consequences for the meaning. Hence, accurately capturing these prosodic
elements is essential in SSL speech models for any application Avila and Ward
(2023).

To address this, Kharitonov et al. (2021) proposed explicitly adding
prosodically-relevant information such as fundamental frequency and duration
to the speech representations models learn, while others aimed at explicitly
modelling emotions in such representations (Gan et al., 2022; Duret et al.,
2023). Although some progress has been made, robust evaluation metrics for
prosody remain scarce, and human evaluation, while insightful, is subjective -
which can limit reproducibility; as well as being expensive and time intensive
- which can hinder its utility in large-scale applications.

Objective evaluations of prosody fall into two main categories: one focuses on
utterance-level features like emotion and speech rate to assess global
prosody, and the other examines local prosody, which is concerned with
prosodic effects at the level of a word or a phrase, such as breaks, turn ends
and emphasis. In addition, one may address prosody for two classes of models:
generative decoder-only models (the speech equivalent of GPT Radford et al.
(2018) (e.g. GSLM, Lakhotia et al., 2021; AudioLM, Borsos et al., 2023; dGSLM,
Nguyen et al. (2023b)), and speech-to-speech (encoder-decoder) approaches,
which take speech as input and produce output in a different voice (speech
resynthesis) or a different language (S2ST). In this paper, we address the
second class of models.

In the context of speech-to-speech (S2S) models, evaluating global prosody can
be relatively straightforward, as the features are not directly related to the
lexical content. The assessment of local prosody, however, presents more of a
challenge, as it necessitates mapping at the lexical level. This can be
relatively feasible in the context of speech resynthesis, where the model
directly reconstructs the input signal and, therefore, preserves lexical
content (e.g., by correlating prosodic attributes such as duration and
fundamental frequency (F0) between input and output utterances; Suni et al.,
2020). However, this becomes more complicated when evaluating S2ST models, as
one needs to ensure the correct prosodic feature is applied to the correct
word(s) Duret et al. (2023) (alignment problem).

Although scarce, there have been recent efforts made to establish benchmarks
in the prosodic evaluation of speech models allowing models comparison,
including evaluation corpora and pipelines, both at the global prosodic level
(pragramtic information : Lin et al. (2023) and at the local prosodic level
(prosodic pauses: de Seyssel et al. (2023). Yet, there is a need for more
benchmarks to cover other aspects of prosody, and all types of speech models.

In this work, we introduce the EmphAssess benchmark, which is focused on local
prosody for speech-to-speech models and includes: (i) an new, automatic
pipeline for emphasis evaluation that is modular, handles multiple languages
and kinds of outputs (including paraphrases and translations, (ii) a novel
dataset, the EmphAssess test set, for evaluating model emphasis preservation
in English and Spanish according to our pipeline, and (iii) EmphaClass, an
emphasis classifier that we finetuned with English data over an existing
multilingual SSL model to support our pipeline.

##  2 Background

#### Emphasis as a prosodic feature.

Emphasis, the phonetically-realized importance given to particular words or
phrases, is critical for interpreting language. Some of the most important
correlates of emphasis are fundamental frequency (f0), duration, and amplitude
Terken and Hermes (2000); Mo (2008), although the weight and behaviour of each
can vary across languages Ladd and Arvaniti (2023). These acoustic attributes
collectively shape the prosodic contours that signal emphasis in speech.
Altering the emphasis in a sentence such as “I never said he stole my bag"
from “he" to “stole" can drastically change its meaning. Such nuances are
essential for models to process, if they are to have an accurate
representation of speech, be they generative language models or S2ST systems.
In fact, the issue of accurate emphasis transfer in S2ST models has attracted
some research attention over the years. Studies by Tsiartas et al. (2013); Do
et al. (2016, 2018) approach this topic using cascaded models (with separate
Automatic Speech Recognition, Machine Translation, and Text-to-Speech models).
A more recent approach by Huang et al. (2023) integrates the two first
components into a single encoder module capable of multilingual embeddings.
Similar to other prosodic features, emphasis in S2S models is primarily
evaluated through human evaluation Tsiartas et al. (2013); Huang et al.
(2023), although Do et al. (2016, 2018) proposed leveraging an emphasis
classification algorithm to calculate F1 scores by matching emphasised words
in the input and output utterances. Yet, this method is limited to a single
language pair and cannot handle variations in translation outputs, only
recognising one “gold” translation per dataset utterance. Consequently, this
metric is ill-suited for comprehensive automatic benchmarking across various
models.

#### Word-level emphasis classification.

As suggested by Do et al. (2016, 2018), a robust word-level emphasis
classification system is critical in automatic evaluation of emphasis transfer
in S2ST models. Existing algorithms, predominantly designed for text-to-speech
applications, often rely on traditionally engineered features (e.g. MFCCs or
Fbanks), sometimes augmented with other prosodic-related information (e.g. F0,
duration) Do et al. (2016); Heba et al. (2017); Ning et al. (2017); Zhang et
al. (2018). Some also incorporate lexical information from textual transcripts
Brenier et al. (2005); Zhou et al. (2020). However, these models frequently
suffer from limited generalisability across different datasets, voice types,
and languages. There is a compelling argument for using the speech waveform
directly as input to enhance generalisability. To our knowledge, the only
study to have adopted this approach is that of Vaidya et al. (2022), which
employed a CRNN framework for classifying emphasis in children’s speech; their
work, however, was limited to a single language (and is not open-sourced). We
propose that leveraging pretrained models trained on multilingual datasets
could result in significant advancements in this field.

Figure 1: Overview of the EmphAssess evaluation pipeline. Left panel : Output
generation. Right panel : Input-output emphasis comparison.

##  3 Introducing EmphAssess

In this study, we introduce EmphAssess, a versatile automatic benchmark for
evaluating emphasis preservation in S2S models, including S2ST ones.
Essentially, this benchmark comprises a carefully curated dataset of English
utterances with emphasised words, accompanied by an automatic evaluation
pipeline, and results on some of the most recent S2S SSL models. Our
evaluation framework, inspired by the methodology of Do et al. (2016, 2018),
assesses emphasis alignment between the source and the model’s output
utterances. Our benchmark’s novelty lies in its capacity to handle various
output types, including paraphrases and translations.

Guided by the data we have for setting optimal baselines, the EmphAssess
benchmark is specifically designed for English-to-English and English-to-
Spanish S2S models. However, our work goes further, laying the groundwork for
extending this benchmark to other language pairs. Moreover, the evaluation
pipeline itself is already capable of being applied to a broad spectrum of
language pairs. Also, while we focus here on unsupervised speech language
models, EmphAssess is versatile enough to be applied to any S2S framework.

The EmphAssess evaluation pipeline’s modular structure is a key feature, with
each module designed to function independently and allow for straightforward
modifications. We leverage a suite of distinct open-source models, each
finetuned for particular tasks. The pipeline can therefore be upgraded to
incorporate improvements in each module seamlessly. Although such enhancements
may necessitate a re-evaluation of the models within our benchmark, this
inherent adaptability is a considerable benefit, ensuring EmphAsses can remain
current with the latest research for years to come.

Finally, we introduce and open-source, as part of this automatic evaluation
pipeline, a novel emphasis classifier at the word level: EmphaClass. This
classifier is finetuned over an existing multilingual SSL model with the hope
of enhancing its robustness across multiple languages and variability.

The evaluation code, emphasis classifier and dataset introduced in this paper
are available in our related
repository111https://github.com/facebookresearch/emphassess.

##  4 The EmphAssess Dataset

The EmphAssess dataset comprises synthetically generated speech utterances,
each containing at least one emphasised word. Accompanying these utterances
are metadata detailing the transcription, the positional index of the
emphasised word(s), and information about the synthetic voice employed for
synthesis. In total, the dataset boasts 3652 speech samples derived from 913
unique transcripts (with each transcript being rendered in 4 distinct voices).

The dataset generation started with a selection of transcripts from a list of
handwritten transcripts with emphasis annotations previously created for
company-internal Text-to-Speech purposes. Transcripts containing characters
beyond letters or specific punctuation marks222Retained punctuation characters
include: [,:;.?!()] or those featuring proper nouns (identified using the NLTK
toolkit; Bird 2006) were excluded. Moreover, we ensured a minimum of two
distinct versions with different emphases for string identical sentences
(those with matching word tokens but possibly differing emphasis position
indices). This approach was adopted to mitigate any bias should a model
exhibit a preference for emphasising a particular word over others. Finally,
we filtered out transcripts that could face alignment challenges with
emphasised words during translation. We set up an algorithm to assess the
difficulty of aligning emphasised words in an English sentence with their
counterparts in multiple target languages, using the SimAlign word-alignment
tool Sabet et al. (2020). Simply put, if an emphasised word in the source
matched consistently to a corresponding word across a list of other languages
(German, French, Spanish, and Chinese), the sentence was labelled “easy”;
otherwise, it was deemed “difficult.” Only “easy” transcripts were retained
for our dataset. We were left with 913 distinct transcriptions (with varying
emphases) derived from a pool of 299 unique transcriptions. We ensured that
the distribution of transcripts was well balanced, in terms of where the
emphasis was located.

Next, we employed an internal Text-to-Speech (TTS) tool with a 16 kHz sample
rate to synthesise all 913 transcripts, each in the four distinct open-source
English Expresso voices Nguyen et al. (2023a), namely ex01, ex02, ex03 and
ex04, resulting in a comprehensive set of 3,652 speech samples.

Finally, we compiled a dataset that is available as part of the benchmark.
This dataset comprised four columns: an ‘id’ that denotes the unique
identifier for each speech segment, a ‘src_sentence’ that contains the
corresponding tokenised text transcript presented in list format, a
‘gold_emphasis’ that highlights the index of the emphasised word(s) also in
list format, and a ‘voice’ that specifies the particular Expresso voice
employed for the synthesis.

##  5 The EmphaAssess Evaluation Pipeline

The evaluation pipeline, as illustrated in Figure 1, is divided into two main
stages. The first one (left panel) corresponds to the generation of utterances
from the evaluated S2S model. That is, for each utterance from; the EmphAssess
dataset, we need to generate the corresponding utterance output from the
evaluated model. Hence, this inference stage is dependent on the model tested,
and we will not expand on it here.

In the second stage (right panel), we perform the automatic evaluation by
comparing the input and output utterances. The objective is twofold: firstly,
to ascertain whether the emphasis is retained in the generated utterance, and
secondly, to determine whether the emphasis is correctly positioned on the
corresponding word. At this stage, available resources include the input
(original) utterance, the corresponding output utterance, and the tokenised
transcript of the input with the location of the emphasised word(s)
identified. A schematic overview of the evaluation pipeline is shown in the
right panel of Figure 1. Initially, we obtain a transcription of the generated
utterance (1) and the time-aligned word boundaries (2). This information can
be used in addition to the raw waveform to detect emphasis at the word level
in the output utterance using a classifier (3). At this stage, we must
determine which word(s) in the generated utterance should be emphasised to
obtain evaluation scores (4). We use word-to-word alignment at the text level
to address this, a technique borrowed from the machine translation field.
Finally, we can use this information to compute precision, recall and F1 score
(5). We will now detail our methodology for each of these steps.

###  5.1 Automatic speech Recognition and word-level forced time-alignment

To achieve accurate transcription of the generated utterance and its
associated word-level time-alignments, we utilise the WhisperX system Bain et
al. (2023). This system, which relies on the weakly supervised speech
recognition model Whisper Radford et al. (2023) for speech transcription,
allows retrieval of accurate word-level timestamps, in a variety of languages.

###  5.2 Word Emphasis Classification

As the next step requires detecting emphasis at the word level from the
waveform and its corresponding transcription, we propose EmphaClass, a new
model for emphasis classification. Our approach was centred around finetuning
a pretrained SSL speech model through a frame-classification task to classify
a frame as either emphasised or not. We can then aggregate frame-level scores
to derive word-level emphasis classifications.

Figure 2: Illustrative example of emphasis classification with the trained
classifier. Top: gold annotations. Bottom: Emphasis classifier predictions.

Data. We utilised speech sourced from the English Expressive Expresso dataset
Nguyen et al. (2023a). Indeed, this dataset comprises utterances that contain
emphasised words, accompanied by their annotations, presented in a diverse
range of speaking styles. We retained only those utterances that had at least
one word emphasised. We divided the four speakers into two for validation
(ex03 and ex04) and two for the test set (ex01 and ex02). Additionally, we had
utterances from six other speakers recorded under identical conditions and
with similar emphasis annotations. These were utilised to create an internal
training set, amounting to 2.06 hours of speech. We then used the Montreal
Forced Aligner to align the transcription with the audio and obtain reliable
word boundaries McAuliffe et al. (2017). We subsequently processed the data to
provide annotations at the frame level regarding emphasis. We deem a frame as
‘emphasised’ if it falls within a word annotated as such, with each frame
corresponding to 20ms of speech.

Emphasis classifier architecture. We finetuned the multilingual SSL speech
model, XLS-R Babu et al. (2021), grounded in the Wav2Vec 2.0 architecture
Baevski et al. (2020). This finetuning encompassed a binary frame
classification task using cross-entropy loss, and was carried out using the
Wav2Vec2ForAudioFrameClassification method from HuggingFace (Wolf et al.,
2019). Our choice of the XLS-R model for extended training and evaluations
stemmed from its exceptional performance metrics and promising potential for
cross-language generalisation.

Evaluation. We use F1 score as the primary metric for evaluating our emphasis
classifier, both at the frame and word level. For word-level classification,
we compute the average accuracy of the frames within the boundaries of each
word. A word was deemed emphasised if more than 50% of its frames were
classified as such. A representative example of this classification is
illustrated in Figure 2.

We evaluate the classifier on our test set split of the Expresso dataset, but
also on the utterances used in our EmphAssess dataset. Results are presented
in Table 1. The scores suggest that the model performs well at classifying
emphasis in both the Expresso dataset 78.4% and the Emphasses dataset 93.48%.
The lower scores from the Expresso dataset, compared to the EmphAssess
dataset, can be attributed to two factors. Firstly, the Expresso dataset
incorporates utterances with speaking styles where the emphasis is notably
challenging to discern, such as whispering and laughing. Secondly, using
synthetic voices in EmphAssess might offer more consistent and clearer
patterns of emphasis than the natural utterances from Expresso, making it
easier for the classifier to discern, and thus leading to higher accuracy
scores.

Test data | Frame-level (%) | Word-level(%)
---|---|---
F1 | Prec. | Rec. | F1 | Prec. | Rec.
EmphAssess | 89.77 | 89.71 | 91.72 | 93.48 | 93.81 | 94.04
Expresso EN | 75.52 | 60.82 | 76.90 | 78.40 | 56.93 | 76.90

Table 1: Results of EmphaClasson The EmphAssess dataset and a subset of the
Expresso dataset. F1 score, precision and recall

We also ran cross-languages analyses, testing the model on other languages,
which results showed that the model can, to some extent, classify other
languages. This suggests our research may have utility beyond just the English
and Spanish languages we explicitly support. More information is presented in
Appendix A.

###  5.3 Word-to-word alignment

Returning to the automatic emphasis evaluation pipeline, we can detect which
word(s) is emphasised in an output utterance with the classifier described
above, given a waveform, its transcriptions and word boundaries. At this
point, we need to identify which word(s) should be emphasised in the output
utterance to compute a score for the quality of emphasis transfer. This step
is vital because it lets us evaluate any output utterance, including
paraphrases and translations, without being limited to a “gold” output. To do
this, we use a word-to-word alignment algorithm, often seen in machine
translation, especially the SimAlign one Sabet et al. (2020). This tool can
align words between two text sentences. Although typically used in machine
translation, it’s also effective for paraphrases in the same language. A key
benefit of SimAlign is that it works across many languages without requiring
finetuning. For our needs, we compare the original text input with the output
utterance transcription from the ASR to see which word(s) match the emphasised
word in the original sentence.

Figure 3: Precision, recall and F1 scores on the EmphAssess benchmark. Left :
English-to-English models and English Emphasis classifier. Right : English-to-
Spanish models and Spanish Emphasis classifier.

###  5.4 Metrics

In the final step, we compare the words that were meant to be emphasised (from
the previous step) with the words that were actually emphasised (from the
emphasis classification phase). By doing this comparison, we can determine
precision, recall, and F1 scores for the whole dataset.

##  6 Results

We benchmarked a series of models on the EmphAssess evaluation, both within
language (English to English) and using translations (English to Spanish).

###  6.1 English models

We first present results on models that generate speech with the target and
source language being identical, here English (left panel of Figure 3). This
encompasses models that undergo an encoding-decoding method, simply
resynthesising the learnt units and those which can learn paraphrases.

For a topline evaluation, we matched the input utterances from EmphAssess with
themselves (that is, we pretended the output utterances were the same as the
input ones). This gave us an insight into the best achievable scores, with any
potential loss in performance due to problems in the dataset or the various
comparison stages. This topline produced an F1 score of 89%, indicating that
our cascaded pipeline performs well. It should also be noted that we consider
a baseline to yield scores of 0, corresponding to a model which does not
encode emphasis and thus should not produce any emphasis.

We first assessed the generative GSLM model Nguyen et al. (2023b),
specifically the HuBert, 100 units version. This model initially encodes
speech into continuous forms using HuBert Hsu et al. (2021), which are then
quantised into units for language modelling. Subsequently, a synthesiser
converts these units back to speech. In our study, we extracted the quantised
representations from our EmphAssess dataset’s speech samples and directly
resynthesised them, bypassing the generative language modelling phase. Despite
scoring notably lower than the topline with an F1 of 42%, the model
successfully transferred some emphasis to the output utterances. This
indicates the presence of prosodic information within these units learned from
SSL speech model, a finding supported by de Seyssel et al. (2022, 2023).

We also assessed the pGSLM variant, which incorporates extra prosodic features
during training to enhance prosody modelling Kharitonov et al. (2021)333We
opted for the variant with continuous input and shift, as it was the top
performer in de Seyssel et al. (2023).. Notably, the pGSLM models achieved
scores close to the topline, with an F1 of 88%, highlighting their excellent
proficiency in encoding emphasis accurately.

Finally, we assessed the Seamless M4T model Barrault et al. (2023), forcing it
to generate outputs in English. Contrary to the previous models, which
generate output constrained in their lexical input, this one is primarily a
S2ST model and can output paraphrases. We did not expect these models to
encode any prosodic information given to their architecture, an expectation
which was actually supported by a very low score on EmphAssess (18%).

###  6.2 Generalising the pipeline to S2S translation

We now want to discuss how we can adapt our pipeline to S2ST capabilities.
While most target languages can be evaluated directly using the existing
pipeline, there are several considerations to remember. Firstly, it is
essential to establish a validated topline. In other words, when introducing a
new target language, we require validated translated utterances of the input
English dataset in the desired language to have a topline in this target
language. This process necessitates human validation, not only for the text
translation, but also to either synthesise or record this translation with the
correct emphasis, depending on the available resources. This new set of
utterances can additionally serve as an input test set when we want to modify
the source language to one other than English.

Furthermore, we might want to modify or adapt some of the stages of the
automatic evaluation pipeline in order to be better suited to the new
language. For example, we have gathered evidence indicating that the emphasis
classifier performs better when trained in the specific language it will be
evaluated in. Thus, retraining it with emphasis data in the target language
can prove advantageous, albeit demanding the corresponding larger dataset.

We undertook a two-step process to modify our evaluation for English-to-
Spanish translation. Firstly, external annotators translated the input
sentences into Spanish, ensuring the inclusion of emphasis annotations.
Subsequently, these translated sentences were synthesised into Spanish using
our in-house TTS (Text-to-Speech) voices designed for Spanish, with a focus on
retaining emphasis. Additionally, we adjusted the emphasis classifier to one
specifically trained for Spanish as it yielded better results on Spanish data
(see Appendix A).

As depicted in the right panel of Figure 3, the ‘topline,’ which aligns the
English input with the synthesised Spanish voices as the output, achieved a
score of 58%. While this result is reasonable, it notably lags behind the
English topline. This decline may be attributed to various factors, including
challenges in the synthesised voices, as we observed that our Spanish TTS
voices do not emphasise as effectively as desired. Furthermore, issues in
different stages of our automatic evaluation pipeline might contribute (for
instance, the emphasis classifier’s performance is suboptimal in English,
hinting that models trained in other languages might be more suitable).
Additionally, linguistic differences could play a role, with Spanish emphasis
potentially being less prominent than in English or conveyed through
alternative means, possibly paraphrastically in the text itself. Nonetheless,
having this topline facilitates the comparison of other models and the
assessment of their relative performance. Subsequently, we evaluated the
Seamless M4T model Barrault et al. (2023) in its English-to-Spanish
translation capability, which yielded an F1 score of 14%. This result, akin to
its English-to-English counterpart, suggests that the M4T model does not
effectively capture emphasis.

###  6.3 Human Evaluation

To gauge human performance on the task, we conducted an evaluation with expert
annotators. These annotators were presented with an utterance and its word-
tokenised transcription, and were tasked with marking words they considered to
be emphasised. Importantly, they were not obliged to mark any word as
emphasised if they didn’t perceive any. This evaluation was carried out on a
subset of the data, incorporating both English and Spanish utterances, with
native annotators for each language. Figure 3 shows precision, recall, and F1
scores for English-to-English and English-to-Spanish, respectively. For
English-to-Spanish, the human topline is set using the same Spanish utterances
from the Spanish topline. These metrics were calculated by comparing the
annotators’ identification of emphasis against the ‘gold standard’ annotation
with which we synthesised the utterances.

Focusing first on the English dataset, the annotators achieved a commendable
precision score of 86%, although this was offset by a lower recall score
(50%). The lower recall could be attributed to annotators not perceiving
emphasis in numerous sentences (Note: it is often harder to perceive emphasis
in utterances taken out of context); nonetheless, the high precision score is
encouraging. Turning our attention to the Spanish dataset, both recall and
precision scores were lower. This aligns with our hypothesis that the quality
of voice synthesis in Spanish was not up to par. It may also suggest that the
nuances of emphasis might be linguistically specific, thereby differing
between English and Spanish.

##  7 Discussion and Conclusion

We have introduced an evaluation framework for emphasis in speech-to-speech
(S2S) models. This framework comprises an English dataset, an automated
evaluation pipeline, and a results benchmark focusing on English-to-English
and English-to-Spanish models. Crucially, our framework offers a generalisable
approach applicable to other language pairs, the only major requirement being
the acquisition of a relevant dataset to establish a reliable gold standard.

Additionally, we have open-sourced an emphasis-classification model that has
been finetuned on English data. The model builds on a multilingual SSL
architecture and has shown impressive accuracy in classifying emphasised
speech in English on our dataset, along with reasonable performance in other
languages (for further details, refer to the Appendix). The model’s robustness
in English makes it a plausible starting point for finetuning classifiers in
other languages, potentially minimising the volume of data needed for
training. Interestingly, the fact that the successful results were achieved
without retraining the encoder, suggests that the inherent features in the
original XLS-R model were adequate for emphasis classification. Yet, it is
important to underscore that we recommend using this classifier specifically
for our defined use case (that is the presented benchmark and evaluation
pipeline), as further testing is required to enhance its robustness and ensure
its efficacy in detecting more nuanced forms of emphasis across other
datasets.

Furthermore, the robustness of our evaluation process relies on the quality of
multiple pipeline components, including Automatic Speech Recognition, forced
alignment, and word-to-word alignment. Therefore, it is crucial to be mindful
that errors could arise at various stages. Yet, the modular nature of the
pipeline allows for continual improvements and assures that inter-model
comparisons remain valid.

Lastly, our study is limited to binary categorisation of emphasis. Future
endeavours could explore varying degrees of emphasis, although this would
require more advanced models. For instance, capturing subtle differences in
emphasis between the input and output of an S2S system could be a valuable
addition to this line of research.

Beyond the limitations we have identified, there is an existing agenda for
future research centring around the evaluation of prosody within SSL models.
Firstly, on the subject of emphasis, we aim to scrutinise its functional role
more closely—specifically, its ability to convey importance. We intend to
investigate whether such a function is intrinsically represented within these
models. Beyond emphasis, other aspects of prosody, such as turn-taking and
speech grouping, merit attention. We are interested in determining whether
these elements, too, are encoded within SSL models. Improved benchmarks and
evaluations for these prosodic features could pave the way for the development
of more expressive and nuanced models.

To conclude, the EmphAssess benchmark sets a new standard for the evaluation
of prosodic features in S2S models, offering both methodological contributions
and actionable insights that could pave the way for more natural and effective
machine-generated speech across various applications.

## Acknowledgements

ED in his EHESS capacity has been funded by the Agence Nationale pour la
Recherche (ANR-17-EURE-0017 Frontcog, ANR-10-IDEX-0001-02 PSL*,
ANR-19-P3IA-0001 PRAIRIE 3IA Institute) and a grant from CIFAR (Learning in
Machines and Brains).

## References

  * Avila and Ward (2023) Jonathan E Avila and Nigel G Ward. 2023.  Towards cross-language prosody transfer for dialog.  _arXiv preprint arXiv:2307.04123_.
  * Babu et al. (2021) Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al. 2021.  Xls-r: Self-supervised cross-lingual speech representation learning at scale.  _arXiv preprint arXiv:2111.09296_.
  * Baevski et al. (2020) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in neural information processing systems_ , 33:12449–12460.
  * Bain et al. (2023) Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. 2023.  Whisperx: Time-accurate speech transcription of long-form audio.  _arXiv preprint arXiv:2303.00747_.
  * Barrault et al. (2023) Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, et al. 2023.  Seamlessm4t-massively multilingual & multimodal machine translation.  _arXiv preprint arXiv:2308.11596_.
  * Bird (2006) Steven Bird. 2006.  Nltk: the natural language toolkit.  In _Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions_ , pages 69–72.
  * Borsos et al. (2023) Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. 2023.  Audiolm: a language modeling approach to audio generation.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_.
  * Brenier et al. (2005) Jason M Brenier, Daniel M Cer, and Daniel Jurafsky. 2005.  The detection of emphatic words using acoustic and lexical features.  In _Ninth European Conference on Speech Communication and Technology_.
  * Cutler et al. (1997) Anne Cutler, Delphine Dahan, and Wilma Van Donselaar. 1997.  Prosody in the comprehension of spoken language: A literature review.  _Language and speech_ , 40(2):141–201.
  * Dahan (2015) Delphine Dahan. 2015.  Prosody and language comprehension.  _Wiley Interdisciplinary Reviews: Cognitive Science_ , 6(5):441–452.
  * de Seyssel et al. (2022) Maureen de Seyssel, Marvin Lavechin, Yossi Adi, Emmanuel Dupoux, and Guillaume Wisniewski. 2022.  Probing phoneme, language and speaker information in unsupervised speech representations.  In _Interspeech 2022_.
  * de Seyssel et al. (2023) Maureen de Seyssel, Marvin Lavechin, Hadrien Titeux, Arthur Thomas, Gwendal Virlet, Andrea Santos Revilla, Guillaume Wisniewski, Bogdan Ludusan, and Emmanuel Dupoux. 2023.  Prosaudit, a prosodic benchmark for self-supervised speech models.  In _Interspeech 2023_.
  * Do et al. (2018) Quoc Truong Do, Sakriani Sakti, and Satoshi Nakamura. 2018.  Sequence-to-sequence models for emphasis speech translation.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 26(10):1873–1883.
  * Do et al. (2016) Quoc Truong Do, Tomoki Toda, Graham Neubig, Sakriani Sakti, and Satoshi Nakamura. 2016.  Preserving word-level emphasis in speech-to-speech translation.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 25(3):544–556.
  * Duret et al. (2023) Jarod Duret, Benjamin O’Brien, Yannick Estève, and Titouan Parcollet. 2023.  Enhancing expressivity transfer in textless speech-to-speech translation.  _arXiv preprint arXiv:2310.07279_.
  * Gan et al. (2022) Wendong Gan, Bolong Wen, Ying Yan, Haitao Chen, Zhichao Wang, Hongqiang Du, Lei Xie, Kaixuan Guo, and Hai Li. 2022.  Iqdubbing: Prosody modeling based on discrete self-supervised speech representation for expressive voice conversion.  _arXiv preprint arXiv:2201.00269_.
  * Heba et al. (2017) Abdelwahab Heba, Thomas Pellegrini, Tom Jorquera, Régine André-Obrecht, and Jean-Pierre Lorré. 2017.  Lexical emphasis detection in spoken french using f-banks and neural networks.  In _International Conference on Statistical Language and Speech Processing_ , pages 241–249. Springer.
  * Hsu et al. (2021) Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3451–3460.
  * Huang et al. (2023) Wen-Chin Huang, Benjamin Peloquin, Justine Kao, Changhan Wang, Hongyu Gong, Elizabeth Salesky, Yossi Adi, Ann Lee, and Peng-Jen Chen. 2023.  A holistic cascade system, benchmark, and human evaluation protocol for expressive speech-to-speech translation.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5. IEEE.
  * Jia et al. (2022) Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, and Roi Pomerantz. 2022.  Translatotron 2: High-quality direct speech-to-speech translation with voice preservation.  In _International Conference on Machine Learning_ , pages 10120–10134. PMLR.
  * Jia et al. (2019) Ye Jia, Ron J Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. 2019.  Direct speech-to-speech translation with a sequence-to-sequence model.  _arXiv preprint arXiv:1904.06037_.
  * Kharitonov et al. (2021) Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Morgane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, et al. 2021.  Text-free prosody-aware generative spoken language modeling.  _arXiv preprint arXiv:2109.03264_.
  * Ladd and Arvaniti (2023) D Robert Ladd and Amalia Arvaniti. 2023.  Prosodic prominence across languages.  _Annual Review of Linguistics_ , 9:171–193.
  * Lakhotia et al. (2021) Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, et al. 2021.  On generative spoken language modeling from raw audio.  _Transactions of the Association for Computational Linguistics_ , 9:1336–1354.
  * Lee et al. (2021) Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, et al. 2021.  Textless speech-to-speech translation on real data.  _arXiv preprint arXiv:2112.08352_.
  * Lin et al. (2023) Guan-Ting Lin, Chi-Luen Feng, Wei-Ping Huang, Yuan Tseng, Tzu-Han Lin, Chen-An Li, Hung-yi Lee, and Nigel G Ward. 2023.  On the utility of self-supervised models for prosody-related tasks.  In _2022 IEEE Spoken Language Technology Workshop (SLT)_ , pages 1104–1111. IEEE.
  * McAuliffe et al. (2017) Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017.  Montreal forced aligner: Trainable text-speech alignment using kaldi.  In _Interspeech_ , volume 2017, pages 498–502.
  * Mo (2008) Yoonsook Mo. 2008.  Acoustic correlates of prosodic prominence for naiïve listeners of american english.  In _Annual Meeting of the Berkeley Linguistics Society_ , volume 34, pages 257–267.
  * Mohamed et al. (2022) Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maaløe, et al. 2022.  Self-supervised speech representation learning: A review.  _IEEE Journal of Selected Topics in Signal Processing_.
  * Nguyen et al. (2023a) Tu Anh Nguyen, Wei-Ning Hsu, Antony d’Avirro, Bowen Shi, Itai Gat, Maryam Fazel-Zarani, Tal Remez, Jade Copet, Gabriel Synnaeve, Michael Hassid, et al. 2023a.  Expresso: A benchmark and analysis of discrete expressive speech resynthesis.  _arXiv preprint arXiv:2308.05725_.
  * Nguyen et al. (2023b) Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, et al. 2023b.  Generative spoken dialogue language modeling.  _Transactions of the Association for Computational Linguistics_ , 11:250–266.
  * Ning et al. (2017) Yishuang Ning, Zhiyong Wu, Runnan Li, Jia Jia, Mingxing Xu, Helen Meng, and Lianhong Cai. 2017.  Learning cross-lingual knowledge with multilingual blstm for emphasis detection with limited training data.  In _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5615–5619. IEEE.
  * Radford et al. (2023) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023.  Robust speech recognition via large-scale weak supervision.  In _International Conference on Machine Learning_ , pages 28492–28518. PMLR.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018.  Improving language understanding by generative pre-training.
  * Rubenstein et al. (2023) Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. 2023.  Audiopalm: A large language model that can speak and listen.  _arXiv preprint arXiv:2306.12925_.
  * Sabet et al. (2020) Masoud Jalili Sabet, Philipp Dufter, François Yvon, and Hinrich Schütze. 2020.  Simalign: High quality word alignments without parallel training data using static and contextualized embeddings.  _arXiv preprint arXiv:2004.08728_.
  * Suni et al. (2020) Antti Suni, Sofoklis Kakouros, Martti Vainio, and Juraj Šimko. 2020.  Prosodic prominence and boundaries in sequence-to-sequence speech synthesis.  _arXiv preprint arXiv:2006.15967_.
  * Terken and Hermes (2000) Jacques Terken and Dik Hermes. 2000.  The perception of prosodic prominence.  In _Prosody: Theory and experiment: Studies presented to Gösta Bruce_ , pages 89–127. Springer.
  * Tsiartas et al. (2013) Andreas Tsiartas, Panayiotis G Georgiou, and Shrikanth S Narayanan. 2013.  A study on the effect of prosodic emphasis transfer on overall speech translation quality.  In _2013 IEEE International Conference on Acoustics, Speech and Signal Processing_ , pages 8396–8400. IEEE.
  * Vaidya et al. (2022) Mithilesh Vaidya, Kamini Sabu, and Preeti Rao. 2022.  Deep learning for prominence detection in children’s read speech.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 8157–8161. IEEE.
  * Wennerstrom (2001) Ann Wennerstrom. 2001.  _The music of everyday speech: Prosody and discourse analysis_.  Oxford University Press.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Zhang et al. (2018) Long Zhang, Jia Jia, Fanbo Meng, Suping Zhou, Wei Chen, Cunjun Zhang, and Runnan Li. 2018.  Emphasis detection for voice dialogue applications using multi-channel convolutional bidirectional long short-term memory network.  In _2018 11th International Symposium on Chinese Spoken Language Processing (ISCSLP)_ , pages 210–214. IEEE.
  * Zhou et al. (2020) Suping Zhou, Jia Jia, Long Zhang, Yanfeng Wang, Wei Chen, Fanbo Meng, Fei Yu, and Jialie Shen. 2020.  Inferring emphasis for real voice data: an attentive multimodal neural network approach.  In _MultiMedia Modeling: 26th International Conference, MMM 2020, Daejeon, South Korea, January 5–8, 2020, Proceedings, Part II 26_ , pages 52–62. Springer.

##  Appendix A Cross-language generalisation in the classifier

Using a Spanish company-internal variant of the Expresso dataset, we trained
and tested the classifier on Spanish data in an identical manner to our
approach with English. We should however note that the version of the data we
had was of lesser recording quality than the English one.

The classifier’s outcomes when evaluated on both the English and Spanish train
sets are presented in Table 2. The most important observation from the results
is the classifier’s superior performance when trained and tested on the same
language. Cross-language assessments, especially from English-trained models
tested on Spanish data, manifested a decline in performance. Nevertheless,
despite the noted challenges, the results demonstrate that the classifier is
able to detect emphasis, even across languages. It is also worth that the
Spanish dataset was of considerably lower quality than the English one and is
just used here for demonstration purposes. It is plausible that this quality
might have affected the model’s performance. Therefore, a more definitive
assessment of its cross-language generalisation potential would necessitate
testing on datasets of other languages, ideally of comparable quality to the
English version.

We also extended the evaluation of the English and Spanish emphasis
classifiers to additional languages, using internal datasets to compile test
sets mirroring the structure of the English ones, each featuring 2 to 3
speakers. These are summarised in Table 2. Intriguingly, the Spanish
classifier outperformed across all tested languages, a finding readily
attributable to linguistic similarities in the case of Italian, French, and
Portuguese, but less so for Vietnamese. Furthermore, in some instances,
performance on non-native test sets was on par with, or even surpassed, native
datasets; for example, a word-level F1 score of 84.4% was achieved on the
Portuguese test set. These observations imply the feasibility of applying
classifiers to languages they were not specifically trained on, particularly
when sufficient training data is lacking, and suggest the merit in
experimenting with classifiers based on different languages. Additional
results could potentially advocate for the benefits of multi-language training
approaches. An additional point of interest arises from the performance of the
Vietnamese test sets. Vietnam’s tonal nature, which distinctly shapes its
emphasis patterns, ostensibly diverges from the prosodic systems used in
Romance and Germanic languages. Despite these fundamental differences, the
fact that the Spanish-trained classifier achieved commendable results with
Vietnamese indicates that it may be recognising universal features of emphasis
that transcend language-specific prosodic systems.

|  | Frame-level metrics (%) | Word-level metrics (%)
---|---|---|---
Test data | Train data | F1 score | Precision | Recall | F1 score | Precision | Recall
English | English | 75.52 | 77.48 | 76.9 | 78.4 | 78.96 | 79.46
English | Spanish | 67.36 | 68.74 | 71.95 | 68.66 | 66.73 | 75.21
Spanish | English | 55.75 | 60.82 | 55.16 | 56.14 | 56.93 | 57.92
Spanish | Spanish | 72.52 | 73.26 | 75.12 | 73.92 | 74.21 | 76.32
Vietnamese | English | 61.65 | 68.98 | 61.51 | 64.59 | 70.63 | 63.7
Vietnamese | Spanish | 71.21 | 71.82 | 76.32 | 75.48 | 77.69 | 78.2
Italian | English | 56.79 | 70.61 | 52.86 | 56.12 | 57.18 | 57.61
Italian | Spanish | 64.72 | 72.64 | 63.46 | 67.81 | 68.42 | 70.41
French | English | 60.18 | 62.81 | 63.31 | 65.08 | 65.85 | 67.07
French | Spanish | 62.50 | 63.09 | 68.05 | 68.17 | 67.64 | 72.41
Portuguese | English | 71.84 | 83.56 | 68.41 | 72.86 | 73.17 | 74.69
Portuguese | Spanish | 79.84 | 82.93 | 80.08 | 84.4 | 84.15 | 87.1
Table 2: Performance metrics of the emphasis classifier across multiple
languages, benchmarked using F1 score, precision, and recall. The classifier
is trained either on English or Spanish data sets. Rows highlighted in grey
represent instances where the training and test data languages are identical.


# CLEAN algorithm implementation comparisons between popular software packages

Daniel Wright  Karel Adámek  Wesley Armour E-mail address:
<EMAIL_ADDRESS>
###### Abstract

The CLEAN algorithm, first published by Högbom, [1974] and its later variants
such as Multiscale CLEAN (msCLEAN) by Cornwell, [2008], has been the most
popular tool for deconvolution in radio astronomy. Interferometric imaging
used in aperture synthesis radio telescopes requires deconvolution for removal
of the telescopes point spread function from the observed images. We have
compared source fluxes produced by different implementations of Högbom and
msCLEAN (WSCLEAN ([Offringa et al.,, 2014], [Offringa and Smirnov,, 2017]),
CASA [The CASA Team et al,, 2022]) with a prototype implementation of Högbom
and msCLEAN for the Square Kilometer Array (SKA) on two datasets. First is a
simulation of multiple point sources of known intensity using Högbom, where
none of the software packages detected all the simulated point sources to
within 1.0% of the simulated values. The second is of supernova remnant
G055.7+3.4 taken by the Karl G. Jansky Very Large Array (VLA) using msCLEAN,
where each of the software packages produced different images for the same
settings.

##  1 Introduction

Synthesis imaging is a fundamental technique used in radio astronomy to
compute the sky brightness distribution (an image) from frequency elements
(visibilities) measured by the radio interferometer. As the visibilities are
being sampled in the frequency domain by a sampling function, the resulting
image produced by the inverse Fourier transform is convolved with the point
spread function (PSF). In the case of radio interferometry, the sampling
function is very sparse as a radio interferometer has a limited number of
antennas that observe the sky, making the PSF present in the image
particularly obstructive to science. The actual image $I_{\mathrm{Actual}}$ is
therefore convolved with PSF giving us so-called dirty image
$I_{\mathrm{Dirty}}$:

$I_{\mathrm{Dirty}}=I_{\mathrm{Actual}}*\mathrm{PSF}\,.$

In order to recover the actual image, the PSF must be deconvolved from the
dirty image. Due to the presence of noise in the image, it is not possible to
use deconvolution directly, so a number of algorithms have been developed to
overcome this. One of these is the CLEAN algorithm.

###  1.1 CLEAN Algorithms

The CLEAN algorithm, as first published by Högbom, [1974] and in its further
versions, is a matching pursuit non-linear iterative deconvolution algorithm.
In this work, we demonstrate an implementation of two variants of CLEAN:

Högbom CLEAN [Högbom,, 1974]: Iteratively scans the image for a maxima, each
time one is found, the PSF is centred on the point and a fraction of it is
subtracted from the image. A fraction of the maximum value is added to a model
image at the same position as the maxima was detected. This continues until an
iteration limit or the maxima found is below a set threshold. Finally, the
model image is convolved with an idealised telescope beam and any residual
flux left in the original image is added to the model image to produce the
deconvolved image.

msCLEAN [Cornwell,, 2008]: Uses the same principles as Högbom, but to better
fit the extended emission structures it uses scaling kernels to produce scaled
versions of the image. The maxima is searched for across all the scaled
versions, with a bias value used to enable comparisons of maximum values
across different scales. PSFs are also scaled with the same kernels, to allow
PSF subtraction from each scaled image.

Högbom CLEAN is more suited to point source images, and msCLEAN is more suited
to extended emissions. That is how we deploy and test these algorithms in this
work.

Image reconstruction in radio interferometry is often performed in a
major/minor cycle loop, as is shown in the Figure 1. This work is focused on
the performance of the minor cycle deconvolution only, in order to assess the
previously described CLEAN algorithms, no major cycles have been performed on
the test data.

Figure 1:  Diagram of the Major/Minor Cycle of Image Reconstruction used in
radio iterferometry

In this work we use existing packages for deconvolution of radio images (CASA,
WSCLEAN) along with a prototype Square Kilometer Array (SKA) software package
to produce the output of deconvolution.

###  1.2 Datasets

The simulated point source dataset is a dirty image which contains 30 sources
of various locations and intensities. The simulated instrument is the VLA in
configuration D. The simulation was performed with the OSKAR software
[Dulwich,, 2020]. This dirty image is shown in Figure 2 on the left. The
extended emission dataset is a dirty image from an observation of the
supernova remnant G055.7+3.4 taken by the VLA in configuration D on 23rd
August 2010 [NRAO,, 2023]. The dirty image is shown in the Figure 2 on the
right.

Figure 2:  Dirty images of point (left) and extended emission (right) sources.

##  2 Results

The parameters controlling threshold, gain per cycle and cycle limit were kept
consistent across the different packages (CASA, WSCLEAN, SKA prototype) used
for all tests. Högbom CLEAN was run in each package with a maximum iteration
limit of 3,000 and a minimum flux threshold of 1mJy. These results are shown
in Figure 3.

Figure 3:  Högbom CLEAN deconvolved images.

The msCLEAN implementations were run with a limit of 1,000 iterations and a
threshold of 0.12mJy. Large differences can be seen in the flux and structure
of the images produced. These results are shown in the Figure 4.

Figure 4: msCLEAN deconvolved images.

##  3 Conclusion

For the Högbom results, the flux of each source deconvolved is measured and
its flux compared to the known flux input to the starting simulation, any
deviation is noted.

  * •

WSCLEAN: mean = -3.45%, stdev = 11.31%, min = -29.37%, max = 19.20%

  * •

CASA: mean = -5.17%, stdev = 12.23%, min = -36.16%, max = 11.18%

  * •

SKA prototype: mean = -1.01%, stdev = 6.81%, min = -7.21%, max = 16.91%

It can be concluded that none of the Högbom implementations are particularly
accurate without the aid of a major cycle. It can be concluded that the SKA
prototype is well-aligned with other packages.

The msCLEAN results are compared back to the dirty image to find the
difference and therefore the change in flux in the CLEANed image, this is
shown in the Figure 5.

Figure 5: Difference Between Input Dirty Image and CLEANed Output Results

Figure 5 shows that different amounts of flux have been removed by each
implementation. The differences in the results of the algorithms show that the
various optimisations and design choices used in each of the packages have a
large effect on the final image, they do not produce matching images, even
though they are using the same underlying algorithm. The SKA prototype of
msCLEAN has the correct shape but performs deeper CLEAN than other packages.

## References

  * Cornwell, [2008] Cornwell, T. J. (2008).  Multiscale clean deconvolution of radio synthesis images.  IEEE Journal of Selected Topics in Signal Processing, 2(5):793–801.
  * Dulwich, [2020] Dulwich, F. (2020).  Oskar 2.7.6.
  * Högbom, [1974] Högbom, J. A. (1974).  Aperture Synthesis with a Non-Regular Distribution of Interferometer Baselines.  Astronomy and Astrophysics Supplement, 15:417.
  * NRAO, [2023] NRAO (2023).  Vla casa imaging-casa6.2.0.
  * Offringa et al., [2014] Offringa, A. R., McKinley, B., Hurley-Walker, et al. (2014).  WSClean: an implementation of a fast, generic wide-field imager for radio astronomy.  MNRAS, 444(1):606–619.
  * Offringa and Smirnov, [2017] Offringa, A. R. and Smirnov, O. (2017).  An optimized algorithm for multiscale wideband deconvolution of radio astronomical images.  MNRAS, 471(1):301–316.
  * The CASA Team et al, [2022] The CASA Team et al (2022).  Casa, the common astronomy software applications for radio astronomy.  Publications of the Astronomical Society of the Pacific, 134(1041):114501.

## Acknowledgements

This work was supported by the Science and Technology Facilities Council
[grant number ST/W001969/1].

The National Radio Astronomy Observatory is a facility of the National Science
Foundation operated under cooperative agreement by Associated Universities,
Inc.