# Determining cosmological-model-independent $H_{0}$ and post-Newtonian
parameter with time-delay lenses and supernovae

Tonghua Liu1 and Kai Liao2
1School of Physics and Optoelectronic Engineering, Yangtze University,
Jingzhou, 434023, China
2Department of Astronomy, School of Physics and Technology, Wuhan University,
Wuhan 430072, China
<EMAIL_ADDRESS>
(Accepted XXX. Received YYY; in original form ZZZ)

###### Abstract

Strong gravitational lensing provides a natural opportunity to test General
Relativity (GR). We propose a model-independent method for simultaneous
constraining on Hubble constant ($H_{0}$) and post-Newtonian parameter
($\gamma_{\rm{PPN}}$) using strong lensing systems and observational SNe Ia.
The time-delay measurements from strong lesning can directly determine the
Hubble constant, and the lens distance inferred from the spectroscopic
measurement of the stellar kinematics of the deflector galaxy can help us to
constrain the post-Newtonian parameter. We seek the Pantheon dataset and
reconstruct unanchored distances using Gaussian process regression to achieve
the cosmological model-independent GR testing instead of assuming a specific
model, which can reduce possible bias on GR testing and measurement of Hubble
constant. Combining the reconstructed unanchored distances and the four
H0LiCOW lenses datasets, our results are
$H_{0}=72.9^{+2.0}_{-2.3}\mathrm{~{}km~{}s^{-1}~{}Mpc^{-1}}$ and
$\gamma_{\rm{PPN}}=0.89^{+0.17}_{-0.15}$. All the lenses show that there is no
obvious evidence to support GR deviation within observational uncertainties.
In the subsequent analysis, we consider a ratio of distance ${D_{\Delta
t}}/{D^{{}^{\prime}}_{d}}$ method to further avoid the influence of $H_{0}$ on
GR testing. The results show that, except J1206 within the $\sim 1.2\sigma$
observational uncertainty, the remaining 3 lenses support GR holds within the
$1\sigma$ observational uncertainties.

###### keywords:

cosmology: cosmological parameters - distance scale - gravitational lensing:
strong

††pubyear: 2023

##  1 Introduction

The modern theory of cosmology is based on two pillars, Einstein’s theory of
General Relativity (GR) and the cosmological principle. The former was the
first to equate the gravitational field with the curvature of space-time and
was extremely successful in describing the gravitational interaction between
matter. The latter describes that our Universe is homogeneous and isotropic on
large scales. The Lambda cold dark matter ($\rm{\Lambda}CDM$) model is
consistent with most popular observational evidences, such as the observations
of type Ia supernovae (SNe Ia) [44], Cosmic Microwave Background Radiation
(CMB) [40], and regarded as the standard cosmological model. However, there is
a recent tension of 4.4$\sigma$ or more between the Hubble constant inferred
by CMB observations under the assumption of a flat $\rm{\Lambda}CDM$ model
[41] and its value measured through Cepheid-calibrated distance ladder by the
Supernova $H_{0}$ for the Equation of State collaboration (SH0ES) [45]. Such
inconsistency could be caused by unknown systematic errors in astrophysical
observations or reveal new physics significantly different from $\Lambda$CDM
model. In a recent work, Brout et al. [7] reanalyzed and re-calibrated the
photometric systems in the Pantheon+sample of SN Ia, including those in the
SH0ES distance-ladder measurement of $H_{0}$, and suggested that supernova
calibration is not currently capable of resolving Hubble tension.

As an important prediction of GR, gravitational lensing is a powerful tool to
study the velocity dispersion function of early-type galaxies [33, 18, 12],
the distribution of dark matter [11, 46, 34, 37], and cosmological parameters
[55, 6, 30, 14]. More importantly, the time-delay measurements between
multiple images from strong gravitational lensin provide a valuable
opportunity for determination of $H_{0}$ [43]. In the milestone work of Wong
et al. [60], the six gravitationally lensed quasars with well-measured time
delays were jointly used to constrain the $H_{0}$ by the $H_{0}$ Lenses in
COSMOGRAIL’s Wellspring (H0LiCOW) collaboration. Assuming a flat $\Lambda$CDM
model, the H0LiCOW collaboration reported the results on $H_{0}$ with these
six lensed quasars, $H_{0}=73.3^{+1.7}_{-1.8}$
$\mathrm{~{}km~{}s^{-1}~{}Mpc^{-1}}$, which is consistent with local
measurements from SN Ia, but in $3.1\sigma$ tension with CMB observations.
However, it needs to be stressed that both the Planck and H0LiCOW inferred
$H_{0}$ values are based on GR plus $\rm{\Lambda}CDM$ model, that inspires us
to investigate the validity of GR with cosmological model-independent way. The
validity of GR can be verified by constraining the parametrized post-Newtonian
(PPN) parameter $\gamma_{\rm{PPN}}$ (since GR predicts exactly
$\gamma_{\rm{PPN}}\equiv 1$), which describes the spatial curvature generated
by an unit rest mass. In recent years, especially on the solar system scale,
many tests on GR have made great achievements in extremely high precision (see
review [61] for more works about testing GR). However, testing GR at extra-
galactic scale is still not very precise. For instance, there is only $\sim
20\%$ precision on $\gamma_{\rm{PPN}}$ at $10-100$ Mpc scales by using the
joint measurements of weak gravitational lensing and redshift-space
distortions [51, 4]. At cosmological scales, strong gravitational lensing
systems provide an effective way to probe deviation of GR [5, 10, 29, 58].
Recently, the work by Collett et al. [15] used a nearby strong gravitational
lens ESO 325-G004 to test GR and reported the constrained results on
$\gamma_{\rm{PPN}}=0.97\pm 0.09$ at $1\sigma$ confidence level (C.L). In
further research, Yang et al. [62] proposed a new methodology through time-
delay measurements combined with the stellar kinematics of the deflector lens
from strong lensing to simultaneously constrain $H_{0}$ and
$\gamma_{\rm{PPN}}$, and showed that there is no obvious deviation from the GR
with the result $\gamma_{\rm{PPN}}=0.87^{+0.19}_{-0.17}$. However, it should
be emphasized here that these work are cosmological model-dependent (in
$\rm{\Lambda}CDM$ model). The testing GR should be done without invoking any
particular background cosmology model in order to reduce potential bias from
the forms of the parametric or model assumptions.

Inspired by above, we propose a cosmological model-independent method to
constrain the Hubble constant and PPN parameter simultaneously at cosmological
scales using four strongly lensed quasars published by H0LiCOW with both time
delay distance and lens distance. For the distance information required to
constrain the PPN parameter, we seek for unanchored (or relative) distances
from SN Ia observations using a Gaussian process (GP) regression. Intuitively,
the Time-Delay Strong Lensing (TDSL) measurements can directly determine the
Hubble constant, and the lens distance inferred from the spectroscopic
measurement of the stellar kinematics of the deflector galaxy can help us to
constrain the post-Newtonian parameter. This paper is organized as follows: In
Section 2 we introduce the methodology and H0LiCOW lensing data including
reconstructed $H_{0}D^{L}$ using GP regression. The constrained results on
$H_{0}$ and $\gamma_{\rm{PPN}}$ and discussions are given in Section 3. We
conclusion our result in Section 4. The natural units of $c=G=1$ are adopted
throughout this work.

##  2 Methodology and H0LiCOW lensing data

###  2.1 Distances inferred from H0LiCOW program

In the limit of a weak gravitational field, the Schwarzschild line element of
space-time can be expressed as

$ds^{2}=-\big{(}1+2\Psi\big{)}dt^{2}+\big{(}1-2\Phi\big{)}dr^{2}+r^{2}d\Omega^{2},$
(1)

where $\Psi$ is the Newtonian potential and $\Phi$ represents the spatial
curvature potential, and $\Omega$ is the angle in the invariant orbital plane.
The ratio $\gamma_{\rm{PPN}}=\Phi/\Psi$ denotes as the PPN parameter, which
describes the spatial curvature generated per unit mass. It should be
emphasized that the PPN parameter $\gamma_{\rm{PPN}}$ is predicted to be one
or $\Psi=\Phi$. In this work, we assume that $\gamma_{\rm{PPN}}$ is a constant
at the lens galaxy scales.

The crucial idea of using strong lensing systems to test gravity is through
two different mass measurements, i.e., gravitational mass inferred from the
lensing image, and dynamical mass obtained from the spectroscopic measurement
of stellar kinematics of the deflector galaxy. The motion of non-relativistic
matters (usually made up of baryonic matter and dark matter) is governed by
the Newtonian gravitational potential $\Psi$, which obeys Poisson equation.
However, the motion of relativistic particles is sensitive to both potentials.
Testing $\gamma_{\rm{PPN}}$ requires observing the motion of relativistic and
non-relativistic particles around the same massive object. Thus, strong
lensing systems provide a natural laboratory to test gravity and further
measure the PPN parameter $\gamma_{\rm{PPN}}$. The difference between
dynamical mass and lensing mass can be directly used for difference between
$\Psi$ and $\Psi_{+}=\frac{\Psi+\Phi}{2}=(\frac{1+\gamma_{\rm{PPN}}}{2})\Psi$
(namely Weyl potential). In the framework of PPN, the deflection angle
contains the lensing mass information, $\alpha_{\rm
PPN}(\theta)=(\frac{1+\gamma_{\rm{PPN}}}{2})\alpha_{\rm{GR}}(\theta)$, and
effective lensing potential (the integral of the Weyl potential along the
line-of-sight) rescales as $\psi_{+}=(\frac{1+\gamma_{\rm{PPN}}}{2})\psi$, as
well as convergence field
$\kappa^{\prime}=(\frac{1+\gamma_{\rm{PPN}}}{2})\kappa$. It is worth noting
that the deflector angle is directly related to the cosmological distance, so
the PPN parameter and the cosmological distance are highly degenerated. This
is one of the main limitations of using strong lensing systems to test
gravity. To break this degeneracy, additional data needs to be taken into
account, either the cosmological or the gravitational one. The time delay
measurements are able to break the degeneracy alone.

For a given strong lensing system, quasar acting as background source, time
delays between multiple images can be measured from variable AGN light curves,
and determined by both the geometry of the Universe as well as the
gravitational potential of the lensing galaxy through [48]

$\Delta t=D_{\Delta t}\left[\phi(\theta_{\rm A},\beta)-\phi(\theta_{\rm
B},\beta)\right]=D_{\Delta t}\Delta\phi_{\rm AB}(\xi_{\rm lens}),$ (2)

where $\phi(\theta,\beta)=\left[{(\theta-\beta)^{2}}/{2}-\psi(\theta)\right]$
is the Fermat potential at images, $\beta$ is the source position, $\xi_{\rm
lens}$ is the lens model parameter. The cosmological background is reflected
in so-called "time delay distance" $D_{\Delta t}=(1+z_{\rm d}){D_{\rm d}D_{\rm
s}}/{D_{\rm ds}}$, which is inverse proportional to $H_{0}$. The key here is
the Fermat potential difference $\Delta\phi_{\rm AB}(\xi_{\rm lens})$, which
can be reconstructed by high-resolution lensing imaging from space telescopes.
As we mentioned above, in the PPN framework, the inferred mass parameters are
rescaled by a factor of $(1+\gamma_{\rm{PPN}})/2$. Therefore, we denote the
actually inferred lens model parameters in the Fermat potential as
$\xi^{{}^{\prime}}_{\rm lens}$. We rewrite the time-delay distance as

$D_{\Delta t}=(1+z_{\rm d})\frac{D_{\rm d}D_{\rm s}}{D_{\rm ds}}=\frac{\Delta
t_{\rm AB}}{\Delta\phi_{\rm AB}(\xi^{\prime}_{\rm lens})}\,.$ (3)

This is the first distance we need. It can be obtained from both the
measurements of time delay and the Fermat potential reconstructed with
parameter $\xi^{{}^{\prime}}_{\rm lens}$.

On the other hand, the stellar kinematics of lensing galaxies are only
sensitive to the Newtonian potential $\Psi$, which are independent of PPN
parameters. It can also be obtained from the modeling of the kinematic
observable in lensing galaxies. The modeling of the stellar kinematic in
lensing galaxies by the H0LiCOW collaboration is quite mature, and here we
give a brief introduction, thus provide the reader with a detailed background.
The dynamics of stars with the luminosity density distribution of lenses
$\rho_{*}(r)$ in the gravitational potential $\Psi$ follows the Jeans
equation. In the limit of a relaxed (vanishing time derivatives) and
spherically symmetric system, with the only distinction between radial
($\sigma_{r}$) and tangential ($\sigma_{t}$) dispersions, the anisotropic
Jeans equation is

$\frac{\partial(\rho_{*}\sigma_{r}^{2})}{\partial
r}+\frac{2\beta_{\rm{ani}}(r)\rho_{*}\sigma_{r}^{2}}{r}=-\rho_{*}\frac{\partial\Psi}{\partial
r}\,,$ (4)

where $\beta_{\rm{ani}}(r)\equiv 1-\frac{\sigma_{t}^{2}}{\sigma_{r}^{2}}$ is
the stellar distribution anisotropy. The luminosity-weighted projected
velocity dispersion $\sigma_{s}$ is
$I(R)\sigma_{s}^{2}=2\int_{R}^{\infty}\left(1-\beta_{\rm{ani}}(r)\frac{R^{2}}{r^{2}}\right)\frac{\rho_{*}\sigma_{r}^{2}rdr}{\sqrt{r^{2}-R^{2}}}$
[53], where $I(R)$ is the projected light distribution and $R$ is the
projected radius. Considering observational conditions, the luminosity-
weighted line-of-sight velocity dispersion $\sigma_{v}$ within an aperture
$\mathcal{A}$ is real observations, which is given by
$\sigma_{v}^{2}=\frac{\int_{\mathcal{A}}[I(R)\sigma_{s}^{2}*\mathcal{P}]dA}{\int_{\mathcal{A}}[I(R)*\mathcal{P}]dA}$,
where $\mathcal{P}$ is point spread function (PSF) convolution of the seeing.
The prediction of the stellar kinematics requires a three-dimensional stellar
density $\rho_{*}$ and mass $M(r)$ profile. In terms of imaging data, the
information about the parameters of the lens mass surface density with
parameters mass $\xi_{\rm{lens}}$ and the surface brightness of the lens with
parameters $\xi_{\rm{light}}$ can be extract. Finally, the prediction of any
$\sigma_{v}$ from any model can be decomposed into cosmological part
$D_{s}/D_{ds}$ and stellar kinematics part
$J(\xi_{\rm{lens}},\xi_{\rm{light}},\beta_{\rm{ani}})$ [9]. The function $J$
captures all of the model components calculated from the sky angle (from the
imaging data) and the anisotropy distribution of the stellar orbit (from the
spectroscopy). This allows us to obtain the distance ratio $D_{s}/D_{ds}$ from
the well-measured velocity dispersion, independent of the cosmological model
and time delays, but still relies on the lens model $\xi_{\rm{lens}}$ (not the
$\xi^{{}^{\prime}}_{\rm{lens}}$ under PPN) [8, 9]

$\frac{D_{\rm s}}{D_{\rm
ds}}=\frac{\sigma_{v}^{2}}{c^{2}J(\xi_{\rm{lens}},\xi_{\rm{light}},\beta_{\rm{ani}})}\;.$
(5)

The lens model parameter in $J$ is the “unrescaled” $\xi_{\rm{lens}}$. Here,
we use $\xi^{\prime}_{\rm{lens}}$ to replace $\xi_{\rm{lens}}$, the
corresponding distance ratio shall also be rescaled, as

$\frac{2}{1+\gamma_{\rm{PPN}}}\frac{D_{\rm s}}{D_{\rm
ds}}=\frac{\sigma_{v}^{2}}{c^{2}J(\xi^{\prime}_{\rm{lens}},\xi_{\rm{light}},\beta_{\rm{ani}})}\,.$
(6)

Furthermore, we can define rescaled deflector galaxy distance $D_{\rm
d}^{\prime}=\frac{1+\gamma_{\rm{PPN}}}{2}D_{\rm d}$. By combining Eqs. (3) and
(6), we obtain [8, 9]

$\displaystyle D_{\rm d}^{\prime}=\frac{1}{1+z_{\rm d}}\frac{c\Delta t_{\rm
AB}}{\Delta\phi_{\rm
AB}(\xi^{\prime}_{\rm{lens}})}\frac{c^{2}J(\xi^{\prime}_{\rm{lens}},\xi_{\rm{light}},\beta_{\rm{ani}})}{\sigma_{v}^{2}}\,.$
(7)

This is the second distance we need. More details for these two distance
obtained from strong lensing systems refer to work [60] and references
therein.

Thanks to the H0LiCOW collaboration, four lenses (namely RXJ1131-1231 [54,
55], PG1115+080 [13], B1608+656 111This len was given in the form of skewed
log-normal distribution, due to the absence of blindly analysis with respect
to the cosmological parameters [53, 21], J1206+4332 [9]) posterior
distributions with both angular diameter distances of lens $D_{d}$ and time
delay distances $D_{\Delta t}$ have been published, making it easier to use
them. There are available on the website of H0LiCOW 222http://www.h0licow.org.
The redshifts of both lens and source, the time delay distances, and the
angular diameter distance to the lenses for these lensed quasar systems are
summarized in Table 2 of Wong et al. [60]. For more relevant work by using
these lensing systems, we refer the reader to see the literature [17, 32, 2,
52, 25, 42, 27, 31].

The next problem is the acquisition of these two distance information.
However, since $D_{d}$ depend on the cosmological model, $D_{\Delta t}$ also
change in different cosmological models. The traditional method is to assume a
particular cosmological model, such as a flat $\rm{\Lambda}CDM$ model, but it
is cosmological model-dependent. In this work, we follow the reconstructed
method used in work [26], and seek for current SN Ia observations to determine
distances, even though the observations of SN Ia are anchored relative
distances.

###  2.2 Unanchored distance from observations of SNe Ia using GP regression

As the most explosive variable source in Universe, due to the nature of SNe Ia
(as standard candles), they were regarded as powerful cosmological probes. It
was the observations of SNe Ia that led to the discovery of the accelerating
expansion of Universe. We use recent Pantheon dataset, consisting of 1048 SNe
Ia spanning the redshift range $0.01<z<2.3$ [47]. To combine the Pantheon SNe
and the H0LiCOW strong lenses datasets, we generate samples of the unanchored
luminosity distance $H_{0}D^{L}$ from the posterior of the Pantheon dataset.
In order to perform the posterior sampling in a cosmological model-independent
way, the GP regression [19, 20, 22, 49, 50, 23] is considered here by using
the code GPHist333https://github.com/dkirkby/gphist [24]. GP regression is
powerful tool for reconstruction of function since the regression occurs in an
infinite-dimensional function space without overfitting problem [22]. GP
regression works by generating a large sample of functions $\gamma(z)$
determined by the covariance function. The covariance between these functions
can be described by a kernel function. We adopt a squared-exponential kernel
to parameterize the covariance

$\langle\gamma(z_{1})\gamma(z_{2})\rangle=\sigma_{f}^{2}\,\exp\\{-[s(z_{1})-s(z_{2})]^{2}/(2\ell^{2})\\},$
(8)

with hyperparameters $\sigma_{f}$ and $\ell$ that are marginalized over. The
$\gamma(z)$ is a random function inferred from the distribution defined by the
covariance, and we adopt $\gamma(z)=\ln([H^{\rm fid}(z)/H_{0}]/[H(z))/H_{0}])$
to generate expansion histories $H(z)/H_{0}$ by using Pantheon dataset. Here
$H^{\rm fid}(z)/H_{0}$ is chosen to be the best fit $\Lambda$CDM model for the
Pantheon data and serves the role of the mean function for GP regression.
Since the final reconstruction result is not completely independent of the
mean function, it has some influence on the final reconstruction result
because the value of the hyperparameter helps to track the deviation from the
mean function. And the true model should be very close to flat
$\rm{\Lambda}CDM$, hence, our choice for the mean function is very reasonable
[49, 50, 1].

To highlight the purpose of our work, i.e., the Hubble constant and
$\gamma_{\rm{PPN}}$, we assume a flat universe here. With the reconstructed
expansion history $H(z)/H_{0}$, the unanchored SN luminosity distances can be
calculated

$H_{0}D^{L}(z)=(1+z)\int^{z}_{0}dz^{\prime}/[H(z^{\prime})/H_{0}]\ .$ (9)

The 1000 unanchored luminosity distance curves $H_{0}D^{L}(z)$ reconstructed
from the SN data are shown in Fig. 1. It shows the shape of the cosmic
distance-redshift relation of Pantheon data very well. It should be noted that
the redshift of SN dataset well covers range of four strong lensing system
redshifts, so the redshift range needn’t extrapolation.

Figure 1:  The unanchored luminosity distance curves $H_{0}D^{L}(z)$
reconstructed from the Pantheon SN Ia data for a representative sample of the
1000 GP realizations.

###  2.3 Simultaneous measurements on Hubble constant and PPN parameters

Figure 2: The simultaneous constraints of $H_{0}$ (left panel) and
$\gamma_{\rm{PPN}}$ (right panel) from four of the H0LiCOW lenses. The dashed
line is $\gamma_{\rm{PPN}}=1$ predicted by GR.

In summary, combining the three measurements, namely optical lensing image,
deflector spectroscopies as well as time delays, two distances ($D_{\Delta
t},D_{\rm d}^{\prime}$) can be inferred, simultaneously, based on Eqs. (3) and
(7). It should be stressed that there is only deflector galaxy distance
($D_{\rm d}^{\prime}$) carried the information of the PPN parameter, while
time delay distance $(D_{\Delta t})$ is sensitive to $H_{0}$. Thus, the
combination of $D_{\Delta t}$ and $D_{\rm d}^{\prime}$ directly provides a new
way for simultaneous measurement of $H_{0}$ and $\gamma_{\rm{PPN}}$.

The steps for simultaneous constraining $H_{0}$ and $\gamma_{\rm{PPN}}$ are
summarized as follows:

  1. 1.

Draw 1000 unanchored luminosity distances curves $H_{0}D^{L}$ from SN data,
and convert to unanchored angular diameter distances $H_{0}D^{A}$ for serving
the strong lensing systems.

  2. 2.

Calculate 1000 values of $H_{0}D_{d}$ at the lens from the 1000 $H_{0}D^{A}$
curves for serving four posterior distributions of lens from H0LiCOW program;
Adopt same procedure for source redshifts of the four strong lens systems to
calculate 1000 values of $H_{0}D_{s}$; Combine these $H_{0}D_{d}$ and
$H_{0}D_{s}$ to calculate $H_{0}D_{\Delta t}$ for each system using
$H_{0}D_{\Delta t}=(1+z_{d})(H_{0}D_{d})(H_{0}D_{s})/(H_{0}D_{ds})$444We use
the standard distance relation to obtain the angular diameter distance between
the lens and the source [59] for a spatially flat universe
$D_{ds}=D_{s}-[(1+z_{d})/(1+z_{s})]D_{d}$.;

  3. 3.

Compute the likelihood, for each of the 1000 realizations, from the H0LiCOW’s
$D_{\Delta t}$ and $D_{d}$ data for each lens system for many values of
$H_{0}$ and $\gamma_{\rm{PPN}}$;

  4. 4.

Multiply the four likelihoods to form the full likelihood for each
realization, for each values of $H_{0}$ and $\gamma_{\rm{PPN}}$;

  5. 5.

Marginalize over the realizations to form the posterior distributions of
$H_{0}$ and $\gamma_{\rm{PPN}}$.

##  3 Results and discussions

Working on four posteriors of $D_{\Delta t}$ and $D_{d}$ published by H0LiCOW
program and combining reconstructed unanchored SN dataset, we obtain the final
distributions for Hubble constant $H_{0}$ and PPN parameter
$\gamma_{\rm{PPN}}$. Our model-independent constraints are
$H_{0}=72.9^{+2.0}_{-2.3}\mathrm{~{}km~{}s^{-1}~{}Mpc^{-1}}$ and
$\gamma_{\rm{PPN}}=0.89^{+0.17}_{-0.15}$ (median value plus the $16^{th}$ and
$84^{th}$ percentiles around this) for combination of all lenses. The one-
dimensional posterior distributions are shown in Fig. 2. The numerical
constraint results for four individual lenses can be found in Table 1. One can
see that constraint results of all the lenses show that GR is supported within
observational uncertainties, and there is no obvious evidence of GR deviation.
This can be compared to the results of Yang et al. [62] of
$H_{0}=73.65^{1.95}_{-2.26}\mathrm{~{}km~{}s^{-1}~{}Mpc^{-1}}$ and
$\gamma_{\rm{PPN}}=0.87^{+0.19}_{-0.17}$ within a flat $\rm{\Lambda}CDM$ for
combining all lenses, our results are agreement with their results, which
support the robustness of our method. Our results on $H_{0}$ are consistent
with the time delay and time delay plus SN results [16, 60, 56] made within
specific cosmological models or with polynomial fitting of the distance
relation. However, it should be stressed that, comparing with assuming a
specific model, combinations of strong lensing and current astronomical probe
such as SN Ia can reduce possible bias in our work. More importantly, there is
no significant increase in uncertainty.

Table 1:  Summary of the constraints on the Hubble constant $H_{0}$ and $\gamma_{\rm{PPN}}$ from four of the H0LiCOW lenses. Lenses | $H_{0}(\mathrm{~{}km~{}s^{-1}~{}Mpc^{-1}})$ | $\gamma_{\rm{PPN}}$
---|---|---
All | $72.9^{+2.0}_{-2.3}$ | $0.89^{+0.17}_{-0.15}$
RXJ1131 | $77.6^{+3.3}_{-3.4}$ | $0.94^{+0.33}_{-0.27}$
B1608 | $69.5^{+3.2}_{-3.3}$ | $0.73^{+0.27}_{-0.23}$
PG1115 | $80.4^{+7.8}_{-7.1}$ | $0.67^{+0.41}_{-0.33}$
J1206 | $71.8^{+7.2}_{-6.8}$ | $1.44^{+0.73}_{-0.61}$
Figure 3: The one-dimensional posterior distributions of $\gamma_{\rm{PPN}}$ using distance ratio method with four H0LiCOW lenses. The dashed line is $\gamma_{\rm{PPN}}=1$ predicted by GR. Table 2: Summary of the constraints on $\gamma_{\rm{PPN}}$ using the distance ratio method from four H0LiCOW lenses. Lenses | All | RXJ1131 | B1608 | PG1115 | J1206
---|---|---|---|---|---
$\gamma_{\rm{PPN}}$ | $1.07^{+0.20}_{-0.17}$ | $0.85^{+0.34}_{-0.28}$ | $0.73^{+0.27}_{-0.23}$ | $0.84^{+0.30}_{-0.36}$ | $1.78^{+0.87}_{-0.67}$

On the other hand, Hubble constant and PPN parameter are in fact completely
degenerate together, this can also be seen in work of Yang et al. [62]. From a
theoretical point of view, PPN parameter is encoded in the inferred angular
diameter distance to the lens, $D^{\prime}_{\rm d}=D_{\rm
d}(1+\gamma_{\rm{PPN}})/2$, which leads that PPN parameter and cosmological
distance (related to Hubble constant) are degenerated. For instance, if
$\gamma_{\rm{PPN}}$ increase, it means the enhancement of gravitational force.
One also can keep gravity unmodified, but change the corresponding distances.
To try to break this degeneracy and do not assume any value for $H_{0}$, we
can also consider ratios of distances ${D_{\Delta
t}}/{D^{{}^{\prime}}_{d}}=\frac{2}{1+\gamma_{\rm{PPN}}}{D_{\rm s}}/{D_{\rm
ds}}$, which are independent of $H_{0}$. The final results are displayed in
Fig 3. We see that the mean values of $\gamma_{\rm{PPN}}$ for four individual
lenses have a little changes (though not significant), the numerical results
are shown in Table 2. For combination of all lenses, the $\gamma_{\rm{PPN}}$
parameter is even closer to one. However, for the lens J1206, the
corresponding constraint result is $\gamma_{\rm{PPN}}=1.78^{+0.87}_{-0.67}$.
Although GR is somewhat deviated within the $1\sigma$ confidence level, GR is
still supported within the $\sim 1.2\sigma$ confidence level. As pointed out
in work of Millon et al. [35], the dispersion measurements do not play a
significant role in the $H_{0}$ estimation of the H0LiCOW analysis, except for
the lens J1206. In other word, in the analysis of the distance ratios, since
we have not given any value for $H_{0}$, the degeneracy between $H_{0}$ and
$\gamma_{\rm{PPN}}$ causes a shift in $\gamma_{\rm{PPN}}$. In addition, for
the lens B1608, the constraint of $\gamma_{\rm{PPN}}$ using the distance ratio
does not change at all, the $D_{\Delta t}$ and $D_{d}$ of this lens are
completely independent, because of the lack of blind analysis of this lens.
Since no new data has been added, we do not expect any improvement in
precision, and our results show this point.

##  4 Conclusion

In this work, we propose a model-independent method for simultaneous
constraining on Hubble constant and post-Newtonian parameter using time-delay
strong lensing systems and observational SNe Ia. To match the lensing and
source redshifts of the four strong lensing systems analyzed by H0LiCOW
program, we use GP regression to reconstruct distance from observations of SNe
Ia instead of assuming a specific model. Although the observations of SNe Ia
provide the unanchored or relative distance, strong lensing systems encode
absolute distance. Thus, such dataset combinations can anchor cosmological
distances.

Firstly, we directly use four posteriors of $D_{\Delta t}$ (inferred from
lensing mass) and $D_{d}$ (inferred from dynamic mass) published by H0LiCOW
program, combining with reconstructed unanchored SN dataset. For combination
of all lenses, we find that the constraint result on PPN parameter is
$\gamma_{\rm{PPN}}=0.89^{+0.17}_{-0.15}$ which demonstrates that GR is
supported within observational uncertainties. The result on Hubble constant is
$H_{0}=72.9^{+2.0}_{-2.3}\mathrm{~{}km~{}s^{-1}~{}Mpc^{-1}}$, which is
consistent with the time delay and time delay plus SN results [16, 60, 56]
made within specific cosmological models or with polynomial fitting of the
distance relation.

Secondly, in order to avoid the influence of $H_{0}$ on GR testing, we do not
assume any value for $H_{0}$, and consider a ratio of distance ${D_{\Delta
t}}/{D^{{}^{\prime}}_{d}}$ method to test PPN parameter. This method can
independently give a constraint on $\gamma_{\rm{PPN}}$, because the distance
ratio contains only velocity dispersion measurements and the dispersion
measurements do not play a significant role in the $H_{0}$. The mean values of
$\gamma_{\rm{PPN}}$ for four individual lenses have a little changes (though
not significant) using ratio of distance method. However, the constraint
result using the lens J1206 is $\gamma_{\rm{PPN}}=1.78^{+0.87}_{-0.67}$, which
no longer supports that GR holds within observational uncertainty.

As a final remark, we point out that time-delay lenses plus observations of SN
provide a quite promising and model independent method to test General
Relativity. The uncertainty of model-independent analyses with such dataset
combination can be comparable to the uncertainty of assuming specific models,
while reducing possible biases. We also look forward to a large amount of
future data, not only from strong lensing system, but also from the SN Ia,
allowing us to further improve the measurements of $H_{0}$ and
$\gamma_{\rm{PPN}}$. In the future, current surveys such as Dark Energy Survey
(DES) [57] and Hyper SuprimeCam Survey (HSC) [36], and the upcoming wide-area
and deep surveys like the Large Synoptic Survey Telescope (LSST) [38] and
Euclid and WFIRST satellites [3, 39] with much wider field of view and higher
sensitivity will be able to discover and precisely localize a large number of
lensed quasars and even other lensed sources, which will have well-measured
time delays. The High-resolution imaging from space telescopes such as HST or
ground-based adaptive optics will help to better model the stellar kinematic
in lensing galaxies. In addition, SN Ia data will continue to improve as well,
playing an important role as a dense sampler of cosmic expansion history over
a wide range of redshifts. All these help us to carry out more accurate
analysis of General Relativity, Hubble constant, and cosmology in our
subsequent work.

##  5 Acknowledgments

This work was supported by National Natural Science Foundation of China under
Grant Nos. 12203009, 12222302, 11973034. Liu. T.-H was supported by Chutian
Scholars Program in Hubei Province (X2023007). Liao. K was supported by Funds
for the Central Universities (Wuhan University 1302/600460081).

##  6 Data availability statements

The data underlying this article will be shared on reasonable request to the
corresponding author.

## References

  * Aghamousa et al. [2017] Aghamousa, A., Hamann, J., & Shafieloo, A. 2017, JCAP., 2017, 031. doi:10.1088/1475-7516/2017/09/031
  * Bag et al. [2022] Bag, S., Shafieloo, A., Liao, K., et al. 2022, ApJ., 927, 191. doi:10.3847/1538-4357/ac51cb
  * Barnacka [2018] Barnacka, A. 2018, Phys. Rep., 778, 1. doi:10.1016/j.physrep.2018.10.001
  * Blake et al. [2016] Blake, C., Joudaki, S., Heymans, C., et al. 2016, MNRAS., 456, 2806. doi:10.1093/mnras/stv2875
  * Bolton et al. [2006] Bolton, A. S., Rappaport, S., & Burles, S. 2006, PRD, 74, 061501. doi:10.1103/PhysRevD.74.061501
  * Bonvin et al. [2017] Bonvin, V., Courbin, F., Suyu, S. H., et al. 2017, MNRAS., 465, 4914. doi:10.1093/mnras/stw3006
  * Brout et al. [2022] Brout, D., Taylor, G., Scolnic, D., et al. 2022, ApJ., 938, 111. doi:10.3847/1538-4357/ac8bcc
  * Birrer et al. [2016] Birrer, S., Amara, A., & Refregier, A. 2016, JCAP., 2016, 020. doi:10.1088/1475-7516/2016/08/020
  * Birrer et al. [2019] Birrer, S., Treu, T., Rusu, C. E., et al. 2019, MNRAS., 484, 4726. doi:10.1093/mnras/stz200
  * Cao et al. [2017] Cao, S., Li, X., Biesiada, M., et al. 2017, ApJ., 835, 92. doi:10.3847/1538-4357/835/1/92
  * Cao et al. [2022] Cao, S., Qi, J., Cao, Z., et al. 2022, A&A., 659, L5. doi:10.1051/0004-6361/202142694
  * Chae [2007] Chae, K.-H. 2007, ApJL., 658, L71. doi:10.1086/516569
  * Chen et al. [2019a] Chen, G. C.-F., Fassnacht, C. D., Suyu, S. H., et al. 2019a, MNRAS., 490, 1743. doi:10.1093/mnras/stz2547
  * Chen et al. [2019b] Chen, Y., Li, R., Shu, Y., et al. 2019b, MNRAS., 488, 3745. doi:10.1093/mnras/stz1902
  * Collett et al. [2018] Collett, T. E., Oldham, L. J., Smith, R. J., et al. 2018, Science, 360, 1342. doi:10.1126/science.aao2469
  * Collett et al. [2019] Collett, T., Montanari, F., & Räsänen, S. 2019, Phys. Rev. Lett., 123, 231101. doi:10.1103/PhysRevLett.123.231101
  * Ding et al. [2021] Ding, X., Treu, T., Birrer, S., et al. 2021, MNRAS., 503, 1096. doi:10.1093/mnras/stab484
  * Geng et al. [2021] Geng, S., Cao, S., Liu, Y., et al. 2021, MNRAS., 503, 1319. doi:10.1093/mnras/stab519
  * Holsclaw et al. [2010] Holsclaw, T., Alam, U., Sansó, B., et al. 2010, PRD, 82, 103502. doi:10.1103/PhysRevD.82.103502
  * Holsclaw et al. [2010b] Holsclaw, T., Alam, U., Sansó, B., et al. 2010b, Phys. Rev. Lett., 105, 241302. doi:10.1103/PhysRevLett.105.241302
  * Jee et al. [2019] Jee, I., Suyu, S. H., Komatsu, E., et al. 2019, Science, 365, 1134. doi:10.1126/science.aat7371
  * Joudaki et al. [2018] Joudaki, S., Kaplinghat, M., Keeley, R., et al. 2018, PRD, 97, 123501. doi:10.1103/PhysRevD.97.123501
  * Keeley et al. [2019] Keeley, R. E., Joudaki, S., Kaplinghat, M., et al. 2019, JCAP., 2019, 035. doi:10.1088/1475-7516/2019/12/035
  * Kirkby & Keeley [2017] Kirkby, D., Keeley, R. 2017, doi:10.5281/zenodo.999564
  * Liao et al. [2015] Liao, K., Treu, T., Marshall, P., et al. 2015, ApJ., 800, 11. doi:10.1088/0004-637X/800/1/11
  * Liao et al. [2019] Liao, K., Shafieloo, A., Keeley, R. E., et al. 2019, ApJL., 886, L23. doi:10.3847/2041-8213/ab5308
  * Liao et al. [2020] Liao, K., Shafieloo, A., Keeley, R. E., et al. 2020, ApJL., 895, L29. doi:10.3847/2041-8213/ab8dbb
  * Liao [2021] Liao, K. 2021, ApJ., 906, 26. doi:10.3847/1538-4357/abc876
  * Liu et al. [2022] Liu, X.-H., Li, Z.-H., Qi, J.-Z., et al. 2022, ApJ., 927, 28. doi:10.3847/1538-4357/ac4c3b
  * Liu et al. [2019] Liu, T., Cao, S., Zhang, J., et al. 2019, ApJ., 886, 94. doi:10.3847/1538-4357/ab4bc3
  * Liu et al. [2022] Liu, T., Cao, S., Biesiada, M., et al. 2022, ApJ., 939, 37. doi:10.3847/1538-4357/ac93f3
  * Liu et al. [2022] Liu, T., Cao, S., Li, X., et al. 2022, A&A., 668, A51. doi:10.1051/0004-6361/202243375
  * Matsumoto & Futamase [2008] Matsumoto, A. & Futamase, T. 2008, MNRAS., 384, 843. doi:10.1111/j.1365-2966.2007.12769.x
  * Mellier et al. [1993] Mellier, Y., Fort, B., & Kneib, J.-P. 1993, ApJ., 407, 33. doi:10.1086/172490
  * Millon et al. [2020] Millon, M., Galan, A., Courbin, F., et al. 2020, A&A., 639, A101. doi:10.1051/0004-6361/201937351
  * More et al. [2017] More, A., Lee, C.-H., Oguri, M., et al. 2017, MNRAS., 465, 2411. doi:10.1093/mnras/stw2924
  * Newman et al. [2009] Newman, A. B., Treu, T., Ellis, R. S., et al. 2009, ApJ., 706, 1078. doi:10.1088/0004-637X/706/2/1078
  * Oguri & Marshall [2010] Oguri, M. & Marshall, P. J. 2010, MNRAS., 405, 2579. doi:10.1111/j.1365-2966.2010.16639.x
  * Petrushevska et al. [2018] Petrushevska, T., Goobar, A., Lagattuta, D. J., et al. 2018, A&A., 614, A103. doi:10.1051/0004-6361/201731552
  * Planck Collaboration et al. [2016] Planck Collaboration, Ade, P. A. R., Aghanim, N., et al. 2016, A&A., 594, A13. doi:10.1051/0004-6361/201525830
  * Planck Collaboration et al. [2020] Planck Collaboration, Aghanim, N., Akrami, Y., et al. 2020, A&A., 641, A6
  * Rathna Kumar et al. [2015] Rathna Kumar, S., Stalin, C. S., & Prabhu, T. P. 2015, A&A., 580, A38. doi:10.1051/0004-6361/201423977
  * Refsdal [1964] Refsdal, S. 1964, MNRAS., 128, 307. doi:10.1093/mnras/128.4.307
  * Riess et al. [2007] Riess, A. G., Strolger, L.-G., Casertano, S., et al. 2007, ApJ., 659, 98. doi:10.1086/510378
  * Riess et al. [2019] Riess, A. G., Casertano, S., Yuan, W., et al. 2019, ApJ., 876, 85
  * Ruff et al. [2011] Ruff, A. J., Gavazzi, R., Marshall, P. J., et al. 2011, ApJ., 727, 96. doi:10.1088/0004-637X/727/2/96
  * Scolnic et al. [2018] Scolnic, D. M., Jones, D. O., Rest, A., et al. 2018, ApJ., 859, 101. doi:10.3847/1538-4357/aab9bb
  * Shapiro [1964] Shapiro, I. I. 1964, Phys. Rev. Lett., 13, 789. doi:10.1103/PhysRevLett.13.789
  * Shafieloo et al. [2012] Shafieloo, A., Kim, A. G., & Linder, E. V. 2012, PRD, 85, 123530. doi:10.1103/PhysRevD.85.123530
  * Shafieloo et al. [2013] Shafieloo, A., Kim, A. G., & Linder, E. V. 2013, PRD, 87, 023520. doi:10.1103/PhysRevD.87.023520
  * Simpson et al. [2013] Simpson, F., Heymans, C., Parkinson, D., et al. 2013, MNRAS., 429, 2249. doi:10.1093/mnras/sts493
  * Sonnenfeld [2021] Sonnenfeld, A. 2021, A&A., 656, A153. doi:10.1051/0004-6361/202142062
  * Suyu et al. [2010] Suyu, S. H., Marshall, P. J., Auger, M. W., et al. 2010, ApJ., 711, 201. doi:10.1088/0004-637X/711/1/201
  * Suyu et al. [2013] Suyu, S. H., Auger, M. W., Hilbert, S., et al. 2013, ApJ., 766, 70. doi:10.1088/0004-637X/766/2/70
  * Suyu et al. [2014] Suyu, S. H., Treu, T., Hilbert, S., et al. 2014, ApJL., 788, L35. doi:10.1088/2041-8205/788/2/L35
  * Taubenberger et al. [2019] Taubenberger, S., Suyu, S. H., Komatsu, E., et al. 2019, A&A., 628, L7. doi:10.1051/0004-6361/201935980
  * Treu et al. [2018] Treu, T., Agnello, A., Baumer, M. A., et al. 2018, MNRAS., 481, 1041. doi:10.1093/mnras/sty2329
  * Wei et al. [2022] Wei, J.-J., Chen, Y., Cao, S., et al. 2022, ApJL., 927, L1. doi:10.3847/2041-8213/ac551e
  * Weinberg [1972] Weinberg, S. 1972, Gravitation and Cosmology: Principles and Applications of the General Theory of Relativity
  * Wong et al. [2020] Wong, K. C., Suyu, S. H., Chen, G. C.-F., et al. 2020, MNRAS., 498, 1420. doi:10.1093/mnras/stz3094
  * Will [2014] Will, C. M. 2014, Living Reviews in Relativity, 17, 4. doi:10.12942/lrr-2014-4
  * Yang et al. [2020] Yang, T., Birrer, S., & Hu, B. 2020, MNRAS., 497, L56. doi:10.1093/mnrasl/slaa107


# MSNER: A Multilingual Speech Dataset for Named Entity Recognition

###### Abstract

While extensively explored in text-based tasks, Named Entity Recognition (NER)
remains largely neglected in spoken language understanding. Existing resources
are limited to a single, English-only dataset. This paper addresses this gap
by introducing MSNER, a freely available, multilingual speech corpus annotated
with named entities. It provides annotations to the VoxPopuli dataset in four
languages (Dutch, French, German, and Spanish). We have also releasing an
efficient annotation tool that leverages automatic pre-annotations for faster
manual refinement. This results in 590 and 15 hours of silver-annotated speech
for training and validation, alongside a 17-hour, manually-annotated
evaluation set. We further provide an analysis comparing silver and gold
annotations. Finally, we present baseline NER models to stimulate further
research on this newly available dataset.

Keywords: Spoken Named Entity Recognition, Spoken Language Understanding,
Speech Dataset

MSNER: A Multilingual Speech Dataset for Named Entity Recognition

Quentin Meeus1,2, Marie-Francine Moens1, Hugo Van hamme2
---
20th Joint ACL-ISO Workshop on Interoperable Semantic Annotation
1 LIIR Lab, Computer Science Dpt., KU Leuven 2 PSI, Electrical Engineering
Dpt., KU Leuven
<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

In an increasingly interconnected world where language knows no boundaries,
the field of Speech Processing is undergoing a transformative shift towards
multilingual applications. One such pivotal area is Spoken Named Entity
Recognition (Spoken NER). Named Entity Recognition (NER) is a natural language
processing (NLP) task that involves the identification and categorization of
named entities within a text, typically into predefined categories such as
names of persons, organizations, locations, dates, numerical values, and more.
The primary objective of NER is to automatically recognize and extract
specific pieces of information from unstructured text, making it easier to
analyze and understand the content. NER plays a crucial role in various NLP
applications, including information retrieval, question answering, sentiment
analysis, and language understanding. In contrast, Spoken NER extracts named
entities from audio documents, a task that is considerably more challenging.
Indeed, aside from the inherent difficulties associated with speech
processing, Spoken NER requires not only to identify and classify the
entities, but also to transcribe them correctly. Variability in pronunciation,
accents, and dialects can make the detection and especially the spelling of
named entities very challenging. On the other hand, prosody, intonation and
emphasis are cues that may be crucial for NER but are not readily available in
written text. Recognizing the pressing need to facilitate cross-lingual
research and to provide comprehensive evaluation resources for Spoken NER
models, we have undertaken the task of manually annotating the popular speech
dataset VoxPopuli’s test sets in four languages: Dutch, French, German, and
Spanish. Additionally, we also provide machine-made annotations on the
training and validation sets.

In the following sections, we provide a detailed overview of our efforts in
the domain of Spoken NER. First, we give an overview of related works and
datasets. Then, we introduce the newly annotated dataset and provide
information about its size, multilingual coverage, and its potential
significance in advancing Spoken NER technology. Additionally, we describe the
methodology employed in the dataset’s creation, breaking down the annotation
process and data preparation. We also introduce the user-friendly annotation
interface we’ve developed for this purpose. Furthermore, we present the
results of various experiments and benchmarks conducted using this dataset.
These experiments demonstrate its utility in evaluating Spoken NER models
across the chosen languages, highlighting its role in advancing research and
development in this field.

In summary, this article describes our contributions to the field of
multilingual Spoken NER, including the dataset’s creation, annotation
methodology, and its role in advancing research in this domain.

## 2\. Literature Review

In the field of NLP, there is not one unified label set. Both generic and
specialized datasets exist with their own label sets defined. Specialized
datasets might cover large amounts of topics with specific vocabulary and
entities. For example, a NER system for doctors would include medications,
dosages, medical reasons, etc. Uzuner et al. (2010), and biomedical entities
include names of proteins, chemical, disease, or species Crichton et al.
(2017). Other datasets provide more generic entities that cover broader
landscapes. One of the most widely used is CoNLL-2003 Tjong Kim Sang and De
Meulder (2003), although it comes with only four entity types (LOC, ORG, PER
and MISC). OntoNotes v5 enriches this set with 14 more classes (Table 2), to
include things such as numbers, dates, and laws. Its high quality makes it one
of the most widely used NER datasets, although it only covers three languages:
English, Arabic and Chinese. Another notable mention is Tedeschi et al.
(2021), which adds a few more generic classes to OntoNotes definitions to
cover things such as animal names, diseases, food, and plants, and released a
dataset derived from Wikipedia where named entities were annotated
automatically with an annotation pipeline that effectively combined pretrained
language models and knowledge-based approaches. A follow-up dataset was
published covering more languages Tedeschi and Navigli (2022).

Currently, we know of only one Spoken NER dataset that is openly distributed
as SLUE Shon et al. (2021). This is an annotated subset of the larger
VoxPopuli dataset Wang et al. (2021), which comprises audio recordings and
corresponding transcripts of sessions held in the European Parliament. The
annotated portion of the dataset include approximately 25 hours of speech,
divided into three subsets: 3/5 for training, 1/5 for validation, and 1/5 for
testing purposes. While this initiative is a significant step forward, SLUE
exclusively covers the English language. They used the same entities as
OntoNotes (Weischedel et al., 2013) although in practice, they combine some
types and remove rare ones to produce a new label set (Table 2, Column 2).

Another task in spoken language understanding is similar to Spoken NER: slot
filling. This is the identification of information relevant to specific
applications, such as flight booking Hemphill et al. (1990). Although they
share many grounds, there is a major difference: slot filling relates to a
specific application, and in this regard, covers a much narrower domain than
NER, often consisting of short commands for a computer interface (Lugosch et
al., 2019; Saade et al., 2018; Bastianelli et al., 2020; Lugosch et al., 2021;
Renkens and Van hamme, 2018) or a booking system (Hemphill et al., 1990).

Since the vast majority of entity recognition datasets are text-based, the
same goes for the applications. Consequently, NER is often framed as a token
classification task, where each word or word piece must be assigned an entity
type. Since an entity can cover many tokens, the entity classes are redefined
in the BIO format, a widely used tagging scheme in NER tasks Ramshaw and
Marcus (1995). This format provides a structured way to label and distinguish
the boundaries of named entities within the text. Each word or token is tagged
with one of three labels: “B” marks the beginning, or first word of an entity,
“I” indicates the continuation of the named entity and always follows the “B”
tag, and “O” is used for words that are not part of an entity. This marker,
together with the entity type, makes the target for the classification task.
Other annotation schemes are extensions of this (e.g. IO, IOBES, IOE, etc.).
The major drawback of the BIO format is its inability to represent nested
entities.

The modern approach to NER is to add linear layers to a pretrained language
model and fine-tune it on the chosen NER dataset. Sometimes, a conditional
random field (CRF) Lafferty et al. (2001) is added to learn the transition
probabilities between the label classes Ushio and Camacho-Collados (2021). In
Spoken NER, the two main approaches are pipeline and end-to-end models. As the
name suggests, pipeline models first use automatic speech recognition to
transcribe an audio recording, then use NER to predict the entities. In
contrast, end-to-end models do not force the model to make hard decisions by
choosing one token over another. Instead, it predicts entities directly from
the hidden states. Finally, hybrid models or multitask models predict both the
entities and the transcriptions simultaneously Meeus et al. (2023).

subset | language | duration | size | entities
---|---|---|---|---
train | DE | 224.5 h | 86,410 | 97,492
ES | 141.5 h | 47,611 | 66,482
FR | 186h | 65,952 | 80,255
NL | 38.5 h | 16,533 | 19,566
dev | DE | 4h | 1,610 | 1,880
ES | 4h48 | 1,529 | 2,094
FR | 4h22 | 1,527 | 1,884
NL | 2h16 | 963 | 1,074
test | DE | 5h | 1,966 | 2,061
ES | 5h | 1,512 | 2,198
FR | 4h30 | 1,656 | 2,004
NL | 2h30 | 1,120 | 1,272
Table 1: MSNER Dataset statistics OntoNotes5 | SLUE | DE | ES | FR | NL | Examples
---|---|---|---|---|---|---
date | WHEN | 307 | 276 | 243 | 113 | 125 years ago, 15 maart, 1815—1830, 1997
time | 12 | 21 | 10 | 8 | 24 hours, acht uur, de hele dag, mañana
cardinal number | QUANT | 136 | 167 | 123 | 91 | 1, 10, 10 miljoen, 11, 11 billion
ordinal number | 82 | 100 | 79 | 45 | First, Ten derde, dritten
quantity | 6 | 2 | 5 | 1 | one and a half meter, two inches
money | 26 | 16 | 18 | 8 | 200 million EUR, Dertig miljoen euro
percent | 21 | 28 | 13 | 22 | 1 procent, 100%, 15 Prozent
geopolitical area | PLACE | 259 | 285 | 283 | 176 | Amsterdam, Australië, Barcelona, Belgium
location | 128 | 139 | 214 | 110 | Afrika, Balkanlanden, Europe
group | NORP | 229 | 244 | 285 | 213 | African, American, Christian
organization | ORG | 621 | 638 | 527 | 362 | Amnesty International, Charlie Hebdo
law | LAW | 64 | 108 | 33 | 22 | Paris Accords, US Constitution
person | PERSON | 123 | 131 | 100 | 67 | Angela Merkel, Barroso, Beyoncé
facility | - | 6 | 2 | 8 | 12 | Guantánamo, White House
event | - | 23 | 25 | 21 | 8 | Europees Semester, Rio conferentie
work of art | - | 6 | 3 | 4 | 4 | Green Book, Koran
product | - | 4 | 1 | 2 | 8 | 2G, 4G, 5G, iPhone
language | - | 3 | 12 | 6 | 2 | Latin, Nederlands, Español
Table 2: Number of annotated entities per entity type in the test sets. Column
SLUE correspond to the ‘combined’ entity set proposed by Shon et al. (2021).

## 3\. Dataset description

The MSNER dataset is an annotated version of the VoxPopuli dataset Wang et al.
(2021) in four languages – Dutch, French, German, and Spanish. VoxPopuli is a
collection of recorded sessions from the European Parliament, segmented to
contain one or more sentence by one speaker. For each language in scope, we
provide three annotated subsets (Table 1): a training and development set with
machine-generated “silver” annotations, and a test set with manual “gold”
annotations. The subsets of the four languages in scope were annotated
according to OntoNotes’ 18 classes. The test sets were manually annotated by
the authors following the methodology outlined in Section 4. Each example in
the annotated dataset contains the VoxPopuli ID to identify the relevant audio
recording in the original dataset, the transcribed sentence and the annotated
named entities, that is, the list of entities, each composed of a text and a
label component (Figure 1). For the silver label datasets, we also provide a
probability score of each predicted entity. We discuss in Section 6 how this
number is related to the uncertainty of the model.

We use the 18-classes OntoNotes label set (Weischedel et al., 2013). However,
following the example from Shon et al. (2021), we provide annotations by using
an alternative label set that combines entity types like places or numbers and
discard the rarest classes like languages, events, and work of art (Table 2
Column 2).

Figure 1: Annotated example

## 4\. Methodology

Figure 2: Evaluation of text-based pretrained NER model against our
annotations. Bright colors correspond to the F1-score and faded colors
correspond to the label-F1 score, a metric that ignores spelling mistakes and
segmentation errors.

We provide two kinds of label quality: machine-generated “silver” labels and
human-annotated “gold” labels. For obvious reasons, the silver labels are much
cheaper and easier to produce. Therefore, we only provide human-made
annotations for the test sets, and the training and validation sets
annotations are entirely machine-generated. The methodology follows these four
broad steps: (1) filtering out recordings without or with misaligned
transcripts, (2) generate silver labels for all subsets, (3) manually annotate
the test sets and (4) verify the human-made annotations to identify and
rectify potential labelling errors. We detail each step in the following
paragraphs.

### 4.1. Filtering

The VoxPopuli dataset contains a few alignment errors between the spoken
content and its corresponding transcript. To address this issue, we employed
an automatic speech recognition (ASR) system, initially transcribing the
spoken utterances and subsequently calculating the word error rate by
comparing the ASR-generated sentence to the provided transcript. For this
task, we opted for the Whisper large v2 ASR model (Radford et al., 2022),
because it showed near state-of-the-art performance across the selected
languages. Notably, this model has been meticulously trained on extensive,
well-curated data to perform both audio translation and transcription tasks.
For the training and development sets, we filter out examples with a WER
larger than 20%, without verifying that the excluded examples were indeed
problematic. This discards about 20% of the German and Dutch utterances, 10%
of the French examples and 6% of the Spanish utterances.
For the test sets, instances where the word error rate (WER) between the
machine-generated transcription and the original transcript exceeded 20%, we
conducted a meticulous review process. This involved listening to the audio
recording and cross-referencing it with the existing transcript. When
feasible, we made necessary corrections to the transcript. However, in cases
where multiple speakers were heard in the recording or no speech is present,
we removed the problematic utterance from the dataset.

### 4.2. Pseudo-annotations

We employed an established text-based Named Entity Recognition (NER) model to
predict entities within the gold transcript. We chose to use the XLM-RoBERTa
large pretrained model (Conneau et al., 2019), fine-tuned specifically on the
OntoNotes v5 dataset (Weischedel et al., 2013). This model is readily
accessible through the HuggingFace
repository111https://huggingface.co/asahi417/tner-xlm-roberta-base-ontonotes5.
While it’s important to note that this particular model’s fine-tuning was
conducted solely on English data, its robustness and efficacy across multiple
languages were remarkable. In our evaluation, we observed impressive
performance, with most sentences annotated correctly.

### 4.3. Annotation Tool

For each of the 6,254 pre-annotated sentences in the test sets, we corrected
the annotations predicted by the model. For this purpose, we have developed a
command line tool to quickly add, edit, merge or remove annotations in a
sentence. This utility displays the pre-annotated sentence with a summary of
the annotations below. Annotations appear as colored XML tags both in the text
and in the summary. An annotated English translation can be displayed. The
annotator then has access to both the original sentence and the translation to
make sure that the annotations are as accurate as possible. When presented
with a sentence, the annotator has the choice to add a new annotation, delete
an existing one, merge two annotations together or modify an annotation,
either by changing the type or by adding or removing words. Once a sentence
has been annotated, it is saved to a file in JSON format. Following this
methodology and with the help of this tool, we were able to save a lot of time
and effort without sacrificing accuracy. For this reason, we make the tool
available online so that others will have the opportunity to contribute to
this field of research by easily annotating more data in many more languages.

As mentioned in Section 3, we not only provide annotations according to
OntoNotes 18 classes, but also the 7-classes combined set proposed in Shon et
al. (2021). However, we chose to completely reannotate the examples where
entities are removed, instead of simply removing all the annotations of the
same type from the dataset. To illustrate this, consider the following
example:

> `<event>` 15th conference on speech of Toronto `</event>`

According to the combined set conversion rules (Table 2), all the entities of
type `<event>` are to be discarded. Doing that would lead to two unannotated
entities, ‘15th’ as a number and ‘Toronto’ as a place. Instead, we re-annotate
the examples containing removed entities to make sure that we are not
penalizing the models for correct assumptions.

Figure 3: Distribution of predicted probability score per class given the
target class for the text-based model’s predictions Figure 4: Confusion
matrix, normalized to show the probability distribution of the tags predicted
with the text-based model.

### 4.4. Verification

Finally, we verify the integrity of the test annotations by deriving a number
of heuristics and rules that the annotations must abide. This involved
grouping the annotations by category and verify each list one by one,
comparing them to one another, searching in the text for frequent annotated
terms to identify missing annotations, etc. In this last step, we also fix
some remaining transcription issues. For example, we realized that VoxPopuli
transcripts omitted the symbol “%”, and sometimes the word “thousands” (in all
languages). Consequently, for all entities marked as cardinal number, we added
the missing tokens when necessary, following the rules specific to the
language222In French and in Spanish, the symbol “%” is generally used, but in
German and in Dutch, the word is more commonly spelled as Prozent or procent,
respectively.. Another error often made by the text-based NER model is to
predict the article as being part of the entity. As multiple sources advocate
against doing so, we abided by the main guidelines Maekawa (2018); Benikova et
al. (2014).

### 4.5. Distribution

The annotated datasets are distributed in two formats: As JSON Lines files
available on GitHub333https://github.com/qmeeus/MSNER, and on the HuggingFace
repository Wolf et al. (2020). There is one file per subset and per language,
where each line is an annotated example. The audio files can be obtained by
downloading VoxPopuli and matching the audio ID. The dataset version hosted on
HuggingFace contains the audio recordings and the preprocessed annotations in
BIO format, so that a researcher can already use the dataset after only two
lines of code.

## 5\. Evaluation Metrics

Following Shon et al. (2021), we recommend evaluating model predictions with
the micro-averaged F1-score. The F1-score is the harmonic mean of precision
and recall, calculated from an unordered list of named entities predicted for
each utterance. Precision is the proportion of correctly predicted entities
among all predicted entities, and recall is the proportion of ground truth
entities that were correctly detected. An entity is considered to be predicted
correctly if both the type and spelling are identical to the ground truth. To
allow multiple entities with the same spelling and type in a sentence, we add
a unique identifier to each entity/type pair. We recommend using the micro-
averaged F1-score because the dataset is unbalanced. The label F1-score only
considers the predicted type of the entity for correctness, leaving the
transcribed entity out of the computations. This metric ignores spelling
mistakes and segmentation errors. We provide an evaluation
script444https://raw.githubusercontent.com/qmeeus/MSNER/main/src/evaluate.py
to compute these metrics and generate a breakdown of the prediction results
per entity type.

## 6\. Experiments

### 6.1. Setup

The first analysis compares the annotated test sets to the pseudo-annotations
generated by the text-based NER model. Since the silver-label training and
validation sets were generated with this model, this analysis is valuable for
anyone intending to use these datasets for training. Indeed, it gives insights
into the entities that are often confused with one another or remain
undetected. It also gives some insights on the reliability of the model’s
confidence score in assessing whether a prediction is correct.

We also consider two methods to predict named entities from speech, with a
pipeline and an end-to-end model. The end-to-end model is a transformer
encoder-decoder trained to perform both ASR and NER with a multitask objective
Meeus et al. (2023). This model is initialized from Whisper Large V2 Radford
et al. (2022), with an additional SLU module connected to the layers of the
decoder with an adaptor. The end-to-end model was fine-tuned on English SLUE-
VoxPopuli Shon et al. (2021). The pipeline model transcribes the audio files
and subsequently annotates the transcriptions. For the ASR model, we use
Whisper Large V2 Radford et al. (2022). For the pipeline model, we provide two
options to allow for a better comparison. In Table 3, we use XML-RoBERTa fine-
tuned on OntoNotes v5 (Weischedel et al., 2013) and compare it to the
predictions generated by the text-based NER model from the gold transcripts.
In Table 4, we fine-tuned the same XML-RoBERTa on SLUE-VoxPopuli Shon et al.
(2021), which provides a fair comparison to the end-to-end model.

Although both models rely on multilingual pretrained models, the fine-tuning
dataset is entirely in English. Therefore, we evaluate the ability of these
models to generalize from one language (English) to other languages (Dutch,
French, German, and Spanish). Before computing the F1-scores, we normalize the
text by putting it in lower case and removing symbols. It should be noted that
the evaluation script does normalize the text further, which could have its
importance depending on the model to be evaluated.

All results are presented on the human-annotated test sets proposed in this
article.

### 6.2. Results

Figure 3 shows the distribution of calculated probabilities for predicted ‘B’
and ‘O’ tags conditional to whether they were predicted correctly or not. For
each token position $k$, the probability of the most likely tag $i^{*}$ is
computed as follows:

$P(y^{k}=i^{*})=\max_{i}{\frac{e^{z^{k}_{i}}}{\sum_{j}e^{z^{k}_{j}}}}$

where $z^{k}_{1..N}$ are the logits predicted by the model for the token at
position $k$. We observe that, on average, annotations for which there was no
agreement between the annotator and the NER model were predicted with a lower
probability than annotations that were correctly annotated from the start.
However, we observe major differences between the class distributions. For the
most frequent classes, like ‘O’, ‘organization’ or ‘date’, the probability
distributions overlap considerably, and one should be careful if using this
score as a proxy for the model’s uncertainty. This is not surprising, as
transformers are known to be overconfident Ye et al. (2023). For rare
quantitative classes like ‘percent’ and ‘quantity’, the model shows confidence
when predictions are correct, and uncertain otherwise. This indicates that for
those particular classes, the given probability could be relied upon when
estimating the model’s uncertainty. The score breakdown by entity and language
(Figure 2) indicates that in general, there are no major differences across
languages, except for rare classes, where the variability increases
significantly.

Figure 4 shows the confusion matrix of the NER model predictions against the
manual annotations. Most errors are undetected entities (bottom row in Figure
4) and segmentation errors (I-tags predicted instead of B-tags and inversely,
are visible on the lighter diagonals above and below the main diagonal). Some
entities remain undetected more often than not, e.g. “work of art” and
“event”, which is a sign that predictions are less reliable for these rare
classes. Some other types are often confused with one another, like “money”
and “cardinal number”. However, all types seem to have at most two confused
types. We notice that “geopolitical area” is most often confused with
“location” and “law”. In the latter case, this is because many laws are named
after cities (e.g. the Paris Agreement, the Warsaw Treaty).

Table 3 compares the text-based NER predictions with the NER predictions
obtained from the ASR transcript and generated by the same text-based NER
model. The OntoNotes dataset, although in English, provides many well-curated
annotations and the NER model trained on this dataset seem to generalize well
to the other languages. However, this model was not trained to handle
automatic transcripts and we observe a considerable drop in performance when
it is asked to process ASR outputs. To make a fair comparison with the end-to-
end model, we fine-tune XML-RoBERTa on SLUE-VoxPopuli and report the results
in Table 4. The fine-tuning dataset being of much modest size (14.5 hours of
training data), the models do not have many examples to learn from. The end-
to-end model has a slight advantage because it learns simultaneously the ASR
and NER tasks, and it is able to share part of its architecture between both
tasks. For example, it seems well able to identify the presence of entities
despite a lot of transcription and segmentation errors, as evidenced by the
large label F1-score. In contrast, the pipeline suffers much more from the
transcription errors because it was pretrained on curated texts and is not
expecting noisy ASR transcriptions.

The text-based NER model performs best for Dutch, then German, French and
finally Spanish. As the model was trained on English annotations, this ranking
is not a surprise, although the ability of the model to transfer to other
languages is impressive. However, for the speech processing models, the same
conclusion cannot be drawn. The entity F1-score seem to be correlated with the
word error rate, which is influenced by the availability of the different
languages in the pretraining set. In other words, for speech models, this is
the model’s ability to transcribe foreign languages that will drive the
quality of the predictions, rather than how similar the evaluation and the
pretraining language are. The label-F1 indicates how accurate a model is at
detecting the presence of entity types, disregarding of its ability to
transcribe it correctly. Looking at those numbers, we observe again the same
behavior as with the text-based entity predictions, namely that entities are
more likely to be accurately detected when the evaluation language is more
similar to the finetuning language.

Model | Metric | DE | ES | FR | NL
---|---|---|---|---|---
Gold |  F1 ($\uparrow$) | 77.4 | 70.1 | 71.1 | 79.9
Label-F1 ($\uparrow$) | 89.7 | 90.3 | 89.1 | 94.4
ASR |  F1 ($\uparrow$) | 52.4 | 50.6 | 44.7 | 52.7
Label-F1 ($\uparrow$) | 66.2 | 63.6 | 59.4 | 66.1
WER ($\downarrow$) | 12.0 | 8.6 | 11.1 | 13.1
Table 3: Performance of text-based NER model trained on OntoNotes. Gold corresponds to the model’s predictions from the gold transcripts and ASR corresponds to the model’s predictions on the ASR transcripts. Model | Metric | DE | ES | FR | NL
---|---|---|---|---|---
Pipeline |  F1 ($\uparrow$) | 30.8 | 36.3 | 37.2 | 36.3
Label-F1 ($\uparrow$) | 42.7 | 51.6 | 49.5 | 45.9
WER ($\downarrow$) | 12.0 | 8.6 | 11.1 | 13.1
End2End |  F1 ($\uparrow$) | 38.3 | 41.3 | 39.6 | 31.2
Label-F1 ($\uparrow$) | 76.8 | 77.1 | 78.3 | 78.4
WER ($\downarrow$) | 13.3 | 10.5 | 14.5 | 18.2
Table 4: Provided baselines on the annotated test sets for a pipeline ASR/NER
model and an end-to-end multitask model. Both models were fine-tuned on SLUE-
VoxPopuli Shon et al. (2021)

## 7\. Conclusion

In this manuscript, we have presented MSNER, a new dataset for evaluating
multilingual Spoken NER systems. Although NER is a popular topic in NLP, this
task has remained mostly unexplored in speech processing and spoken language
understanding. To address this issue, we have used a pretrained model to
annotate the VoxPopuli training and validation subsets in Dutch, French,
German, and Spanish. Additionally, to provide researcher with a gold standard
dataset for evaluating their Spoken NER models, the authors have manually
annotated the test sets for these subsets. By analyzing the predictions of a
text-based NER model, and comparing them with our annotations, we were able to
identify points of attentions for researchers who intend to train a model on
silver annotations. For example, in some cases, the model confidence on the
predictions can serve as a basis to estimate the correctness of the
prediction, but this must be done carefully, since we have seen that
transformers can be overconfident. Counter-intuitively, we have shown that
most frequent classes are not always the ones where the model’s uncertainty is
most reliable. We also looked at the classes that were often confused with one
another, which gave us some ideas about which errors might be present in the
training and validation sets.

We also provide baselines on the newly annotated evaluation subsets. We
selected a pipeline and an end-to-end SLU model, both fine-tuned on English
SLUE VoxPopuli Shon et al. (2021), and we evaluate them on the manually
annotated test sets. We saw that in a low resource scenario, the end-to-end
model seems to benefit from learning simultaneously to transcribe and to
annotate, which allows a better generalization across languages than the
pipeline model fine-tuned on the same dataset. Finally, we found that the
performance of text-based models on unseen languages is correlated with the
similarity of the evaluation language with English. However, for speech
models, this is the multilingual transcription accuracy that is the main
driver for NER performance. Interestingly, we have seen that the end-to-end
model was able to identify the presence of entities much better than the
pipeline model, despite a similar overall performance, which illustrate the
advantage of sharing parameters across tasks.

## 8\. Acknowledgements

This research received funding from the Flemish Government under the
“Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen” programme.

## 9\. Bibliographical References

##

  * Bastianelli et al. (2020) Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. 2020\.  SLURP: A spoken language understanding resource package.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics.
  * Benikova et al. (2014) Darina Benikova, Chris Biemann, and Marc Reznicek. 2014.  NoSta-D named entity annotation for German: Guidelines and dataset.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 2524–2531, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Unsupervised cross-lingual representation learning at scale.  _CoRR_ , abs/1911.02116.
  * Crichton et al. (2017) Gamal Crichton, Sampo Pyysalo, Billy Chiu, and Anna Korhonen. 2017.  A neural network multi-task learning approach to biomedical named entity recognition.  _BMC Bioinformatics_ , 18.
  * Hemphill et al. (1990) Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990.  The ATIS spoken language systems pilot corpus.  In _Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania_.
  * Lafferty et al. (2001) John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.  In _Proceedings of the Eighteenth International Conference on Machine Learning_ , ICML. Morgan Kaufmann Publishers Inc.
  * Lugosch et al. (2021) Loren Lugosch, Piyush Papreja, Mirco Ravanelli, Abdelwahab Heba, and Titouan Parcollet. 2021.  Timers and such: A practical benchmark for spoken language understanding with numbers.  _CoRR_ , abs/2104.01604.
  * Lugosch et al. (2019) Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, and Yoshua Bengio. 2019.  Speech Model Pre-Training for End-to-End Spoken Language Understanding.  In _Interspeech_.
  * Maekawa (2018) Emi Maekawa. 2018.  Annotation guidelines for named entities.  online.
  * Meeus et al. (2023) Quentin Meeus, Marie-Francine Moens, and Hugo Van Hamme. 2023.  Whisper-slu: Extending a pretrained speech-to-text transformer for low resource spoken language understanding.  In _2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_.
  * Radford et al. (2022) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022.  Robust speech recognition via large-scale weak supervision.  _CoRR_.
  * Ramshaw and Marcus (1995) Lance Ramshaw and Mitch Marcus. 1995.  Text chunking using transformation-based learning.  In _Third Workshop on Very Large Corpora_.
  * Renkens and Van hamme (2018) Vincent Renkens and Hugo Van hamme. 2018.  Capsule networks for low resource spoken language understanding.  In _Proc. Interspeech_. International Speech Communication Association.
  * Saade et al. (2018) Alaa Saade, Alice Coucke, Alexandre Caulier, Joseph Dureau, Adrien Ball, Théodore Bluche, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, and Mael Primet. 2018.  Spoken language understanding on the edge.  _CoRR_.
  * Shon et al. (2021) Suwon Shon, Ankita Pasad, Felix Wu, Pablo Brusco, Yoav Artzi, Karen Livescu, and Kyu J. Han. 2021.  SLUE: new benchmark tasks for spoken language understanding evaluation on natural speech.  _CoRR_.
  * Tedeschi et al. (2021) Simone Tedeschi, Simone Conia, Francesco Cecconi, and Roberto Navigli. 2021.  Named entity recognition for entity linking: What works and what’s next.
  * Tedeschi and Navigli (2022) Simone Tedeschi and Roberto Navigli. 2022.  MultiNERD: A multilingual, multi-genre and fine-grained dataset for named entity recognition (and disambiguation).  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , Seattle, United States. Association for Computational Linguistics.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_.
  * Ushio and Camacho-Collados (2021) Asahi Ushio and Jose Camacho-Collados. 2021.  T-NER: An all-round python library for transformer-based named entity recognition.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_. Association for Computational Linguistics.
  * Uzuner et al. (2010) Özlem Uzuner, Imre Solti, Fei Xia, and Eithon Cadag. 2010.  Community annotation experiment for ground truth generation for the i2b2 medication challenge.  _Journal of the American Medical Informatics Association_ , 17(5).
  * Wang et al. (2021) Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021.  VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation.  Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _EMNLP_. ACL.
  * Ye et al. (2023) Wenqian Ye, Yunsheng Ma, Xu Cao, and Kun Tang. 2023.  Mitigating transformer overconfidence via Lipschitz regularization.  In _Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence_ , volume 216 of _Proceedings of Machine Learning Research_ , pages 2422–2432. PMLR.

## 10\. Language Resource References

##

  * Weischedel et al. (2013) Ralph Weischedel and Martha Palmer and Mitchell Marcus and Eduard Hovy and Sameer Pradhan and Lance Ramshaw and Nianwen Xue and Ann Taylor and Jeff Kaufman and Michelle Franchini and Mohammed El-Bachouti and Robert Belvin and Ann Houston. 2013.  _OntoNotes Release 5.0_.  Linguistic Data Consortium LDC2013T19, ISLRN 151-738-649-048-2.