# Lagrangian Perturbation Theory for Biased Tracers: Halo numbers are
conserved?

Peter Espenshade Center for Theoretical Astrophysics and Cosmology, Department
of Astrophysics, University of Zürich, Winterthurerstrasse 190, CH-8057,
Zürich, Switzerland  Jaiyul Yoo Center for Theoretical Astrophysics and
Cosmology, Department of Astrophysics, University of Zürich,
Winterthurerstrasse 190, CH-8057, Zürich, Switzerland  Department of Physics,
University of Zürich, Winterthurerstrasse 190, CH-8057, Zürich, Switzerland
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The Lagrangian perturbation theory (LPT) provides a simple yet powerful way of
computing the nonlinear matter power spectrum, and it has been applied to
biased tracers such as halos and galaxies. The number conservation of matter
particles allows a simple relation between the fluctuations at the initial and
the late times, which is essential in deriving the exact expression for the
nonlinear matter power spectrum. Here we explore the significance of the
number conservation in the LPT. For example, biased tracers in contrast evolve
through mergers and accretion, violating the assumption of number
conservation. We use $N$-body simulations to demonstrate that if applied to
halos at early time the LPT predictions for halos overestimates the power
spectrum at $z=0$ by a factor of three for the mass bin sample $\Delta\log
M_{h}~{}(h^{-1}M_{\odot})=0.5$ at $z\simeq 3$, while the predictions can be
made consistent if we impose by hand the number conservation of halos in the
simulations throughout the entire evolution. This numerical experiment proves
that the number conservation is crucial in the LPT formalism. In reality, LPT
modeling of biased tracers involves marginalization of unknown bias
parameters, and the bias functional can represent a weight on the particles at
initial time that make up today’s halos, satisfying the number conservation.
We discuss the implications for field-level models based on the LPT
applications.

##  1 Introduction

The Lagrangian perturbation theory offers a framework to trace the matter
distribution from early time to present days (Zel’dovich 1970; see also
Buchert 1992; Bouchet et al. 1992; Buchert & Ehlers 1993; Buchert 1994;
Bouchet et al. 1995; Catelan 1995; Bernardeau et al. 2002). Following the
pioneering work by Crocce & Scoccimarro (2006); Matsubara (2008a), it has
become an indispensable tool in modeling the bulk motion and the subsequent
resummation of the perturbative contributions. More importantly, the
Lagrangian perturbation theory provides a simple, but at the same time very
powerful way of computing the nonlinear matter power spectrum (Schneider &
Bartelmann, 1995; Taylor & Hamilton, 1996; Matsubara, 2008a):

$\displaystyle P_{m}(k,t)=\int d^{3}q~{}e^{-i\bm{k}\cdot\bm{q}}~{}\langle
e^{-i\bm{k}\cdot\Delta\bm{\Psi}}\rangle~{},$ (1)

where $\Delta\bm{\Psi}\equiv\bm{\Psi}(\bm{q}_{1},t)-\bm{\Psi}(\bm{q}_{2},t)$,
$\bm{q}\equiv\bm{q}_{1}-\bm{q}_{2}$, and $\bm{\Psi}$ is the displacement field
from the Lagrangian position $\bm{q}$ at initial time $t_{i}$ to the Eulerian
position at late time $t$

$\displaystyle\bm{x}(\bm{q},t)=\bm{q}+\bm{\Psi}(\bm{q},t)~{}.$ (2)

These are comoving coordinates (not physical coordinates). The expression for
the nonlinear matter power spectrum in Equation (1) is fully exact, until the
one-to-one mapping in Equation (2) between $\bm{q}$ and $\bm{x}$ breaks down
when shell crossing of particles takes place at late time on nonlinear scales
(see, however, McDonald & Vlah 2018). Hence the task of computing the
nonlinear matter power spectrum in practice boils down to computing the multi-
point correlation functions for the displacement field in Fourier space up to
a desired order in perturbations (Taylor & Hamilton, 1996; Matsubara, 2008a,
b; Carlson et al., 2013; Vlah et al., 2015a).

The critical element in the derivation is the conservation of matter particles

$\displaystyle n_{m}(\bm{x},t)~{}dV=n_{m}(\bm{q},t_{i})~{}dV_{i}~{},$ (3)

where $dV=a^{3}(t)~{}d^{3}x$ and $dV_{i}=a^{3}(t_{i})~{}d^{3}q$ are the
physical volumes at times $t$ and $t_{i}$ with the corresponding scale factors
$a(t)$ and $a(t_{i})$ occupied by matter particles, subject to the relation in
Equation (2). Since the background number density decreases
$\bar{n}_{m}\propto 1/a^{3},$ the conservation equation (3) becomes

$\displaystyle\left[1+\delta_{m}(\bm{x},t)\right]~{}d^{3}x=\left[1+\delta_{m}(\bm{q},t_{i})\right]~{}d^{3}q~{},$
(4)

and the change in the comoving volume factor (or the Jacobian viewed as a
coordinate transformation) is

$\displaystyle\left|\frac{d^{3}x}{d^{3}q}\right|^{-1}=1+\delta_{m}(\bm{x},t)~{},$
(5)

where the fluctuation $\delta_{m}(\bm{q},t_{i})$ at $t_{i}$ is assumed zero in
literature by setting the initial time $t_{i}$ to be sufficiently early. Given
the Jacobian, the matter density fluctuation can be written in Fourier space
as

$\displaystyle\delta_{m}(\bm{k},t)=\int
d^{3}q~{}e^{-i\bm{k}\cdot(\bm{q}+\bm{\Psi})}~{},$ (6)

which then allows one to write the exact expression for the matter power
spectrum in Equation (1).

The Lagrangian perturbation theory has been applied to biased tracers (or
halos) (Matsubara, 2008b; Padmanabhan & White, 2009; Matsubara, 2011; Carlson
et al., 2013; White, 2014), and the only change in the above equations is that
we cannot set the fluctuation $\delta_{h}(\bm{q},t_{i})$ zero at $t_{i}$:

$\displaystyle\left[1+\delta_{h}(\bm{x},t)\right]~{}d^{3}x=\left[1+\delta_{h}(\bm{q},t_{i})\right]~{}d^{3}q~{},$
(7)

as the biased tracers form in more biased density peaks at early time, while
the matter fluctuations are more homogeneous. However, the key and often
unstated assumption about the application of the Lagrangian perturbation
theory to biased tracers (or Eqs. [7] and [13]) is the same number
conservation, such that the background number density of the biased tracers
evolves as the matter particles,

$\displaystyle\text{unstated assumption:}\quad\bar{n}_{h}\propto
1/a^{3}\propto\bar{n}_{m}~{},$ (8)

which we already know is incorrect in general, invalidating Equation (7) for
biased tracers. Dark matter halos as the simplest biased tracer evolve through
regular mergers with other dark matter halos and continuous accretion of
smooth matter components. In fact, halo mergers account for $\approx 60\%$ of
the total halo mass growth for halo masses spanning the range
$10^{9}M_{\odot}$ to $10^{14}M_{\odot}$ and over the redshift range $1<z<3$
(Mo & White, 1996; Sheth & Tormen, 1999a; Tinker et al., 2008, 2010). Large-
scale galaxy surveys also showed a complicated redshift evolution of the mean
number density of various galaxy samples, and none of them evolves like the
matter component (Eisenstein et al., 2001; White et al., 2011; Brammer et al.,
2011; Conselice et al., 2016). Without the number conservation, and hence
Equation (7), one cannot obtain the powerful Equation (13) for biased tracers,
like Equations (1) and (6) for matter. Therefore, it is natural to expect
systematic errors in the Lagrangian perturbation theory predictions, when
applied to biased tracers without the number conservation.

In general, the Lagrangian perturbation theory for biased tracers is in
practice applied to the halo samples in the Eulerian frame with
marginalization of unknown bias parameters, rather than applied to the halo
samples in the Lagrangian frame. However, it can also be directly applied to
the halo samples in the Lagrangian frame as in the Press-Schechter formalism
(Press & Schechter, 1974; Bardeen et al., 1986; Bond et al., 1991; Mo & White,
1996; Catelan et al., 1998; Sheth & Tormen, 1999b; Bernardeau et al., 2002;
Desjacques, 2008; Matsubara, 2008b, 2011; Carlson et al., 2013; White, 2014;
Matsubara & Desjacques, 2016; Desjacques et al., 2018). In this work, we use
dark matter halos in numerical simulations to quantify how significant the
number conservation is in the Lagrangian perturbation theory predictions for
biased tracers in terms of describing the evolution of dark matter halos and
its clustering. In Section 4, we discuss the implication of our findings for
the applications of the Lagrangian perturbation theory in practice.

##  2 Biased Tracers

###  2.1 Lagrangian Perturbation Theory for Biased Tracers

Given the concise and powerful expression for the exact nonlinear matter power
spectrum in the Lagrangian perturbation theory, the formalism has been applied
to biased tracers (Matsubara, 2008b; Padmanabhan & White, 2009; Matsubara,
2011; Carlson et al., 2013; White, 2014). Under the assumption that the number
conservation for biased tracers is valid, Equations (4) and (7) can be used in
the Lagrangian perturbation theory to model the fluctuation $\delta_{h}$ for
biased tracers (or halos) at late time

$\displaystyle
1+\delta_{h}(\bm{x},t)=\left[1+\delta_{h}(\bm{q},t_{i})\right]\left[1+\delta_{m}(\bm{x},t)\right]~{},$
(9)

where we used Equation (5) for the Jacobian.

Expanding the relation in Equation (9) to the linear order in perturbations,
we obtain the well-known relation of the Eulerian bias to the Lagrangian bias
factor

$\displaystyle b_{1}=1+b_{1}^{\text{L}}~{},$ (10)

where we defined the bias parameters as

$\displaystyle\delta_{h}(\bm{x},t)=b_{1}~{}\delta_{m}(\bm{x},t)~{},\quad\delta_{h}(\bm{q},t_{i})=b_{1}^{\text{L}}~{}\delta_{m}^{(1)}(\bm{q},t)~{},$
(11)

in terms of the nonlinear matter fluctuation $\delta_{m}$ for the Eulerian
bias and the linear matter fluctuation $\delta_{m}^{(1)}$ extrapolated to late
times for the Lagrangian bias. These bias parameters are the key predictions
in the Lagrangian perturbation theory that will be tested against numerical
simulations. The bias parameters can be further expanded to include higher
order corrections, the tidal tensor bias (Szalay, 1988; Fry & Gaztanaga, 1993;
Catelan et al., 2000; Ma & Fry, 2000; Baldauf et al., 2012), as in the
effective field theory approach (Mirbabayi et al., 2015; Senatore, 2015; Vlah
et al., 2015b; Ivanov et al., 2020). Here we restrict our investigation to the
linear order in perturbations and defer further in-depth investigations of
higher order corrections to future work.

###  2.2 Numerical Modeling of Biased Tracers

Here we present our numerical methods to track the evolution of biased tracers
from early time to late time. We ran ten dark matter-only PKDGRAV3 (Potter et
al., 2017) simulations with $2048^{3}$ particles in an $800$ $h^{-1}$Mpc
periodic box and $51$ $h^{-1}$kpc softening length yielding a $5.2\cdot
10^{9}~{}h^{-1}M_{\odot}$ particle mass. We supplied PKDGRAV3 with the
transfer function computed using the Boltzmann solver CLASS (Blas et al.,
2011), and we used the second-order Lagrangian perturbation theory initial
conditions at starting redshift $z=49.$ We adopt the $\Lambda$CDM parameters:
dark matter density $\Omega_{\text{DM}}=0.27$, baryon density
$\Omega_{b}=0.049$, Hubble constant $h=0.67$, scalar spectral index
$n_{s}=0.97$, and primordial power spectrum amplitude $\ln(10^{10}A_{s})=3.0$,
consistent with the Planck results (PLANCK Collaboration, 2020).

We use our simulation data to identify dark matter halos as biased tracers by
using the friends-of-friends algorithm (Davis et al., 1985) implemented in
nbodykit (Hand et al., 2018) with linking length $b=0.2$. We require that each
halo be composed of at least $100$ particles (or the minimum mass
$M_{\text{min}}=5.2\cdot 10^{11}~{}h^{-1}M_{\odot}$). The halo catalogs are
created at each redshift, starting from $z=2.9$ until $z=0$, in units of
$\Delta z\sim 0.5$, and each dark matter particle is tagged with a unique
identification number, such that we can trace which halo individual particles
belong to and thereby how halos at one redshift evolve to the next redshift.

In particular, we focus on halo samples at early time $z_{\star}=2.9$ and
study how these halo samples evolve forward in time to $z=0$. Note, however,
that $z_{\star}=2.9$ is not the initial time $t_{i}$ in the Lagrangian space,
but an intermediate time to infer the bias evolution at $z=0$. We do not
consider higher redshift because we need a large number of halos to ensure
that their power spectra are not dominated by shot noise. At $z_{\star}$, we
partition the halo mass range into three mass bins that, in order of
increasing halo mass, contain 30%, 60%, and 10% of the halos, according to the
halo mass function (Tinker et al., 2010).

To quantify the systematic errors in the Lagrangian perturbation theory
predictions for biased tracers, we compute the power spectra $P_{h}$ of the
halo samples at $z=0$ from simulations and compare to the predictions based on
the bias parameter obtained from the simulations at $z_{\star}$. We compute
the Eulerian bias $b_{1}(z=0)$ at $z=0$ using $b_{1}(z_{\star})$ as

$\displaystyle
b_{1}(z=0)=1+\left[b_{1}(z_{\star})-1\right]\frac{D(z_{\star})}{D(z=0)}~{},$
(12)

where $z_{\star}=2.9$, and $D$ is the linear growth function. This equation is
derived from Equation (9) at linear order in the perturbations, bypassing the
need to compute $b_{1}^{L}$ at the initial time $t_{i}$. We compute the bias
at $z_{\star}$ using
$b_{1}(z_{\star})=\sqrt{P_{h}(z_{\star})/P_{m}(z_{\star})}$ over $k\leq
0.03~{}h~{}\rm{Mpc}^{-1}$.

###  2.3 How Do Halos Evolve through Mergers?

Dark matter halos evolve via numerous mergers with other halos and continuous
accretion of mass, such that unlike dark matter particles the identities of
halos are rather difficult to track through their evolution. For example, the
standard method (or the standard merger tree) finds the most massive
progenitor of a given halo from the previous output time in a numerical
simulation and assigns its identity to the given halo (Benson et al., 2000;
Springel et al., 2001; Croton et al., 2006; Kauffmann et al., 1999).
Consequently, when two halos merge into one, a lower-mass halo loses its
identity in the standard merger tree. The reality is much more complicated,
but this simple scheme makes physical sense when the mass ratio is large.

The assumption in the Lagrangian perturbation theory for biased tracers is,
however, that the number of halos is conserved. To impose this assumption in
numerical simulations, we construct a number-conserving merger tree, in which
we start with a sample of halos at $z_{\star}$ and keep their identities
throughout the evolution, even if they merge into a halo of larger mass, no
matter how large it is. In case of a merger of two halos with identities we
keep, we consider the merged halo as two individual halos at the same position
(double counting). In addition, we consider a slight variation of the number-
conserving merger tree and construct a nearly number-conserving merger tree by
not double counting in the number-conserving merger tree.

In summary, given a halo sample at $z_{\star}$, we construct three different
halo samples at $z=0$ by following the halos according to three different
identification schemes. Certainly, a halo sample according to the number-
conserving merger tree is unnatural, but the number of halos at $z_{\star}$
remains the same at $z=0$, which satisfies the basic assumption in the
Lagrangian perturbation theory for biased tracers. In contrast, a halo sample
at $z=0$ following the standard merger tree is significantly reduced in
number, as the halos in the sample are incorporated into other larger halos by
$z=0$. The crucial difference in three different halo samples is the validity
of the number conservation.

Figure 1: Evolution of halos from mass-bin samples. Halos in one of three
mass-bins samples (dark gray band) at $z_{\star}=2.9$ are followed forward in
time. The curves are normalized to unity to facilitate the comparison. The
standard merger tree (dotted) at $z=0$ shows that halos at $z_{\star}$ grow in
mass, but most (89%) of them disappeared by becoming part of larger mass halos
via merger. In contrast, if we keep tracking halos at $z_{\star}$ even when
they merge with larger halos, or they merge with each other, the number-
conserving merger tree (solid; see text) at $z=0$ shows that the halos in the
narrow mass bin (dark gray band) at $z_{\star}$ are spread over a large range
of mass in halos. The nearly-number-conserving merger tree (dashed; see text)
shows the change in the halo distribution from the solid curve if we prevent
double counting. In the same way, the standard merger tree can be used to
trace back the progenitors at $z_{\star}$ (dot-dashed) from the sample at
$z=0$ (light gray band).

##  3 Comparison to Numerical Simulations

Figure 2: Halo power spectra. Dark gray curve (solid) shows the power spectrum
of the halo sample at $z_{\star}$ shown in the dark gray band in Figure 1.
Various other curves show the power spectra of the halo samples at $z=0$
evolved via the standard (dotted), the number-conserving (solid), and the
nearly number-conserving (dashed) merger trees from the halo sample at
$z_{\star}$. The light gray curve (solid) shows the prediction at $z=0$ of the
Lagrangian perturbation theory based on the bias parameter of the halos at
$z_{\star}$. It matches the power spectrum of the halo sample following the
number-conserving merger tree. Horizontal curves show the shot noises.

We are now ready to quantify how much systematic errors in the Lagrangian
perturbation theory for biased tracers arise due to the violation of the
assumption that the number of halos is conserved throughout the evolution.
This issue will be addressed with two separate questions: how badly is the
number conservation of halos violated? and how well do the Lagrangian
perturbation theory predictions for halos match the halo power spectrum in
simulations? How this issue affects its applications in practice will be
addressed in Section 4.

First, we investigate how badly the number conservation of halos is violated
through mergers, and Figure 1 shows how halos evolve in time. Consider a halo
sample in a narrow mass bin $11.82\leq\log M(h^{-1}M_{\odot})<12.32$ at
$z_{\star}=2.9$ shown in the dark gray band, and the halos in the sample grow
in mass via accretion and merger in time. According to the standard merger
tree, the distribution of this halo sample at $z=0$ is shown as the dotted
curve. The total number of halos in the sample at $z_{\star}$ is reduced by
89% at $z=0$, as the halos in the sample merge with larger halos and they lose
their identity. Furthermore, their mass distribution today is broader than the
initial mass range at $z_{\star}$. The same exercise can be performed backward
in time. Consider a halo sample in a narrow mass bin $12.77\leq\log
M(h^{-1}M_{\odot})<13.07$ at $z=0$ shown in the light gray band, which can be
representative of an observed galaxy sample today. According to the standard
merger tree, the distribution of this halo sample at early time $z_{\star}$ is
shown as the dot-dashed curve. The progenitor halos that make up the light
gray band today originate from a very broad distribution in mass at early time
(39% of the progenitors at $z_{\star}$ are below the mass resolution). Similar
trends are observed across halo samples of different mass. This simple
exercise confirms the well-established fact that halos are not static,
identical objects throughout the evolution like dark matter particles, but
evolving, number-conservation-violating objects that change their properties
like mass and shape via mergers and accretions. Of course, galaxies with
various observable properties follow much more complicated evolutionary
tracks.

The Lagrangian perturbation theory for biased tracers assumes that the number
of biased tracers is conserved. To compare to this null hypothesis, we
construct a halo sample by artificially imposing the number conservation of
halos and tracking their evolution from $z_{\star}$ to $z=0$. Solid and dashed
curves in Figure 1 illustrate the halo distributions at $z=0$ from the halo
sample at $z_{\star}$ in the dark gray band, according to the number-
conserving (solid) and the nearly number-conserving (dashed) schemes, in which
we always keep the identities of the progenitor halos, even when they merge
into a larger halo. Consequently, two or more halos separately identified at
$z_{\star}$ could end up in the same halo at $z=0$. In the number-conserving
scheme, those in the same halo are all counted as individual halos at the same
position, while in the nearly number-conserving scheme this double counting is
lifted (hence the number conservation is slightly violated).

For both cases (solid and dashed) the halos at $z_{\star}$ in the dark gray
band are spread over a larger range of mass than in the standard merger tree
(dotted). Certainly, many of those halos at $z_{\star}$ that disappear in the
standard merger tree are part of massive halos at $z=0$. For example, those
halos at $z_{\star}$ comprise only 6% of the total mass in the histogram with
$M\geq 10^{15}~{}h^{-1}M_{\odot}$, but we keep their identities in these two
artificial merger trees. In particular, 52% of the halos at $z_{\star}$ are
double counted (or more) at $z=0$, shown as the difference between solid and
dashed curves. The halo sample in the dark gray band at $z_{\star}$ maintains
its identities in the solid curve at $z=0$ with the number of halos conserved,
and hence its evolution can be correctly modeled by the Lagrangian
perturbation theory.

To answer the second question, we measure in Figure 2 the power spectra of
three halo samples at $z=0$ evolved from the same halo sample in the dark gray
band at $z_{\star}$, according to the standard (dotted), the number-conserving
(solid), and the nearly number-conserving (dashed) merger trees. Compared to
the standard merger tree, two other number-conserving trees have more halos in
number, or lower shot noises. The distribution over halo mass in Figure 1
determines the bias of each halo sample, and the presence of more massive
halos in the number-conserving merger tree boosts the halo power spectrum.
Finally, the light gray curve is the prediction for the halo power spectrum at
$z=0$ from the halo sample in the dark gray band at $z_{\star}$, where we used
Equation (12) to predict $b_{1}(z=0)$ by measuring $b_{1}(z_{\star})$ from
simulations. The agreement with the halo power spectrum from the number-
conserving merger tree underpins that the assumption of the number
conservation is crucial in the Lagrangian perturbation theory for biased
tracers. Both Figures 1 and 2 are computed by averaging the halo samples from
our ten simulations. The halo sample in the dark gray band contains 830,000
halos at $z_{\star}$ on average.

##  4 Conclusion and Discussion

The Lagrangian perturbation theory (Zel’dovich 1970; see, e.g., Buchert 1992;
Bouchet et al. 1992; Buchert & Ehlers 1993; Buchert 1994; Bouchet et al. 1995;
Catelan 1995; Bernardeau et al. 2002) is a very powerful tool for modeling the
nonlinear matter power spectrum due to its simple analytical expression in
Equation (1), valid until shell crossing takes place on small scales. Its
applications to biased tracers such as dark matter halos and galaxies are
widely popular for the same reason (see, e.g., Schneider & Bartelmann 1995;
Taylor & Hamilton 1996; Matsubara 2008a), and the nonlinear power spectrum for
biased tracers can be obtained from

$\displaystyle\delta_{h}(\bm{k},t)=\int
d^{3}q~{}e^{-i\bm{k}\cdot(\bm{q}+\bm{\Psi})}\left[1+\delta_{h}(\bm{q},t_{i})\right]~{}.$
(13)

In this Letter, we have shown that this simple analytic expression for biased
tracers is possible only when the number of biased tracers is strictly
conserved just as dark matter particles and hence it cannot be literally
applied to dark matter halos or galaxies that evolve with mergers and
accretion ($\bar{n}_{h}\neq 1/a^{3}$).

By using numerical simulations and dark matter halos as our biased tracers, we
have demonstrated that the Lagrangian perturbation theory prediction
overestimates the halo power spectrum at $z=0$ by a factor of 2.8 for a halo
sample of mass $11.82\leq\log M(h^{-1}M_{\odot})<12.32$ defined at
$z_{\star}=2.9$. We have rectified the halo evolution by continuously tracing
the individual halos from initial time until present, even if they merge into
a larger halo and make up a tiny fraction of the merged halo. For the halo
sample constructed this way, the number of halos stays constant throughout the
evolution, and not surprisingly the Lagrangian perturbation theory prediction
is in good agreement with the numerical simulation output.

In the Press-Schechter formalism (Press & Schechter, 1974; Bardeen et al.,
1986; Bond et al., 1991), the halo mass function at high redshift can be used
to compute the bias parameters in the Lagrangian space (Mo & White, 1996;
Catelan et al., 1998; Sheth & Tormen, 1999b; Bernardeau et al., 2002;
Desjacques, 2008; Matsubara, 2008b, 2011; Carlson et al., 2013; White, 2014;
Matsubara & Desjacques, 2016; Desjacques et al., 2018). While we have directly
applied the Lagrangian perturbation theory to halos at high redshift as in the
Press-Schechter formalism, its applications in general involve marginalization
of unknown bias parameters, rather than taking the bias parameters in the
Lagrangian frame as input parameters. If $b_{1}$ is allowed to be free, there
is no problem in matching the halo power spectrum at the linear order. What
about the higher-order bias parameters such as $b_{2}$, the tidal bias
$b_{s^{2}}$, and so on? These bias parameters are associated with the
corresponding higher-order operators ${\cal
O}({\bm{q}})\ni\delta^{2},~{}s^{2}$, which are often described in the recent
field-level models (Vlah et al., 2016; Schmittfull et al., 2019; Chen et al.,
2020; Schmidt, 2021) based on the Lagrangian perturbation theory. The field-
level models assume that the halo number density fluctuation
$\delta_{h}({\bm{k}},t)$ can be written as in Equation (13) but with a general
function ${\cal F}_{h}({\bm{q}})$ in place of
$[1+\delta_{h}({\bm{q}},t_{i})]$, which can then be perturbatively expanded in
terms of operators ${\cal O}_{i}({\bm{q}})$ with bias parameters $b_{i}$. The
relation in Equation (13) with ${\cal F}_{h}$ is still subject to the validity
of the number conservation.

However, one can interpret ${\cal F}_{h}$ as the field describing today’s
halos in the Lagrangian space (or the collection of particles that make up
today’s halos), rather than some halo fluctuation field $\delta_{h}$ that can
be identified in simulations at $t_{i}$. In this way, the number conservation
is automatically satisfied, and it can represent a weight on the particle
distribution in the Lagrangian space. As shown in our number-conserving merger
tree, this general function ${\cal F}_{h}$ in the Lagrangian space would then
be similar to a reverse number-conserving merger tree of halos today. With its
free bias parameters, it should be flexible enough to be non-local or even
time-dependent to accommodate the complicated evolution history of biased
tracers.

Real computations in the field-level models for biased tracers start with
operators ${\cal O}_{i}({\bm{q}})$ in the Lagrangian frame by shifting them to
the Eulerian frame $\tilde{\cal O}_{i}({\bm{x}},t)$ according to the
displacement field (a.k.a. shifted operators):

$\displaystyle\tilde{\cal O}_{i}(\bm{k},t):=\int d^{3}q~{}{\cal
O}_{i}(\bm{q})~{}e^{-i\bm{k}\cdot(\bm{q}+\bm{\Psi})}~{},$ (14)

for each arbitrary bias parameter $b_{i}$. Given a series of shifted
operators, halos are then modeled with arbitrary bias parameters, which in the
end amounts to the effective field theory approach in the Eulerian frame
(Mirbabayi et al., 2015; Senatore, 2015; Vlah et al., 2015b; Ivanov et al.,
2020) (but of course with different operators). However, without the number
conservation of any field associated with each operator ${\cal O}_{i}$, the
shifted operator in Equation (14) is not the Fourier transform of the field
evolved from ${\cal O}_{i}({\bm{q}})$ in the Lagrangian frame. For example,
one can compute the field of $\delta^{2}({\bm{x}},t)$ from simulations at $t$
and Fourier transform it to compare to one obtained from
$\delta^{2}({\bm{q}})$ by Equation (14). Of course, this procedure might also
be fine, because in the end $\tilde{\cal O}_{i}(\bm{k},t)$ defined in Equation
(14) is some operator in the Eulerian space with large-scale bulk motion
accounted for. But it is clear that two fields ${\cal O}_{i}(\bm{q})$ and
${\cal O}_{i}(\bm{x},t)$ are not related.

In practice, however, it was recently shown (Vlah et al., 2016; Schmittfull et
al., 2019; Chen et al., 2020; Schmidt, 2021) that the field-level modeling of
biased tracers in fact works extremely well. With large degree of freedom in
the bias parameters, cosmological parameters are successfully extracted
(Nishimichi et al., 2020; d’Amico et al., 2020; Ivanov et al., 2020; Chen et
al., 2021) after marginalizing over bias parameters. Given that the utility of
the effective field theory modeling depends on the usefulness of the (shifted)
operators in the Eulerian frame, it is puzzling that the shifted operators
work so well in modeling biased tracers at late time. We plan to perform
follow-up investigations in the future work.

## Acknowledgments

We acknowledge useful discussions with Julian Adamek, Andrej Obuljen, and
Zvonimir Vlah. We also thank the anonymous referee for the comments. This work
is supported by the Swiss National Science Foundation Grant CRSII5_173716. We
acknowledge access to Piz Daint and Alps at the Swiss National Supercomputing
Centre under the project ID UZH-24.

## References

  * Baldauf et al. (2012) Baldauf, T., Seljak, U., Desjacques, V., & McDonald, P. 2012, Phys. Rev. D, 86, 083540,
  * Bardeen et al. (1986) Bardeen, J. M., Bond, J. R., Kaiser, N., & Szalay, A. S. 1986, ApJ, 304, 15,
  * Benson et al. (2000) Benson, A. J., Cole, S., Frenk, C. S., Baugh, C. M., & Lacey, C. G. 2000, MNRAS, 311, 793,
  * Bernardeau et al. (2002) Bernardeau, F., Colombi, S., Gaztañaga, E., & Scoccimarro, R. 2002, Phys. Rep., 367, 1,
  * Bernardeau et al. (2002) Bernardeau, F., Colombi, S., Gaztañaga, E., & Scoccimarro, R. 2002, Phys. Rep., 367, 1,
  * Blas et al. (2011) Blas, D., Lesgourgues, J., & Tram, T. 2011, JCAP, 2011, 034,
  * Bond et al. (1991) Bond, J. R., Cole, S., Efstathiou, G., & Kaiser, N. 1991, ApJ, 379, 440
  * Bouchet et al. (1995) Bouchet, F. R., Colombi, S., Hivon, E., & Juszkiewicz, R. 1995, A&A, 296, 575,
  * Bouchet et al. (1992) Bouchet, F. R., Juszkiewicz, R., Colombi, S., & Pellat, R. 1992, ApJ, 394, L5,
  * Brammer et al. (2011) Brammer, G. B., Whitaker, K. E., van Dokkum, P. G., et al. 2011, ApJ, 739, 24,
  * Buchert (1992) Buchert, T. 1992, MNRAS, 254, 729,
  * Buchert (1994) —. 1994, MNRAS, 267, 811,
  * Buchert & Ehlers (1993) Buchert, T., & Ehlers, J. 1993, MNRAS, 264, 375,
  * Carlson et al. (2013) Carlson, J., Reid, B., & White, M. 2013, MNRAS, 429, 1674,
  * Catelan (1995) Catelan, P. 1995, MNRAS, 276, 115,
  * Catelan et al. (1998) Catelan, P., Lucchin, F., Matarrese, S., & Porciani, C. 1998, MNRAS, 297, 692
  * Catelan et al. (2000) Catelan, P., Porciani, C., & Kamionkowski, M. 2000, MNRAS, 318, L39,
  * Chen et al. (2021) Chen, S.-F., Vlah, Z., Castorina, E., & White, M. 2021, J. Cosmology Astropart. Phys, 2021, 100,
  * Chen et al. (2020) Chen, S.-F., Vlah, Z., & White, M. 2020, J. Cosmology Astropart. Phys, 2020, 062,
  * Conselice et al. (2016) Conselice, C. J., Wilkinson, A., Duncan, K., & Mortlock, A. 2016, ApJ, 830, 83,
  * Crocce & Scoccimarro (2006) Crocce, M., & Scoccimarro, R. 2006, Phys. Rev. D, 73, 063519,
  * Croton et al. (2006) Croton, D. J., Springel, V., White, S. D. M., et al. 2006, MNRAS, 365, 11,
  * d’Amico et al. (2020) d’Amico, G., Gleyzes, J., Kokron, N., et al. 2020, J. Cosmology Astropart. Phys, 2020, 005,
  * Davis et al. (1985) Davis, M., Efstathiou, G., Frenk, C. S., & White, S. D. M. 1985, ApJ, 292, 371,
  * Desjacques (2008) Desjacques, V. 2008, Phys. Rev. D, 78, 103503,
  * Desjacques et al. (2018) Desjacques, V., Jeong, D., & Schmidt, F. 2018, Phys. Rep., 733, 1,
  * Eisenstein et al. (2001) Eisenstein, D. J., Annis, J., Gunn, J. E., et al. 2001, AJ, 122, 2267,
  * Fry & Gaztanaga (1993) Fry, J. N., & Gaztanaga, E. 1993, ApJ, 413, 447,
  * Hand et al. (2018) Hand, N., Feng, Y., Beutler, F., et al. 2018, AJ, 156, 160,
  * Ivanov et al. (2020) Ivanov, M. M., Simonović, M., & Zaldarriaga, M. 2020, J. Cosmology Astropart. Phys, 2020, 042,
  * Ivanov et al. (2020) Ivanov, M. M., Simonović, M., & Zaldarriaga, M. 2020, J. Cosmology Astropart. Phys, 2020, 042,
  * Kauffmann et al. (1999) Kauffmann, G., Colberg, J. M., Diaferio, A., & White, S. D. M. 1999, MNRAS, 303, 188,
  * Ma & Fry (2000) Ma, C.-P., & Fry, J. N. 2000, ApJ, 543, 503,
  * Matsubara (2008a) Matsubara, T. 2008a, Phys. Rev. D, 77, 063530,
  * Matsubara (2008b) —. 2008b, Phys. Rev. D, 78, 083519,
  * Matsubara (2011) —. 2011, Phys. Rev. D, 83, 083518,
  * Matsubara & Desjacques (2016) Matsubara, T., & Desjacques, V. 2016, Phys. Rev. D, 93, 123522,
  * McDonald & Vlah (2018) McDonald, P., & Vlah, Z. 2018, Phys. Rev. D, 97, 023508,
  * Mirbabayi et al. (2015) Mirbabayi, M., Schmidt, F., & Zaldarriaga, M. 2015, J. Cosmology Astropart. Phys, 2015, 030,
  * Mo & White (1996) Mo, H. J., & White, S. D. M. 1996, MNRAS, 282, 347,
  * Mo & White (1996) Mo, H. J., & White, S. D. M. 1996, MNRAS, 282, 347
  * Nishimichi et al. (2020) Nishimichi, T., D’Amico, G., Ivanov, M. M., et al. 2020, Phys. Rev. D, 102, 123541,
  * Padmanabhan & White (2009) Padmanabhan, N., & White, M. 2009, Phys. Rev. D, 80, 063508,
  * PLANCK Collaboration (2020) PLANCK Collaboration. 2020, A&A, 641, A1,
  * Potter et al. (2017) Potter, D., Stadel, J., & Teyssier, R. 2017, CompAC, 4, 2,
  * Press & Schechter (1974) Press, W. H., & Schechter, P. 1974, ApJ, 187, 425
  * Schmidt (2021) Schmidt, F. 2021, J. Cosmology Astropart. Phys, 2021, 033,
  * Schmittfull et al. (2019) Schmittfull, M., Simonović, M., Assassi, V., & Zaldarriaga, M. 2019, Phys. Rev. D, 100, 043514,
  * Schneider & Bartelmann (1995) Schneider, P., & Bartelmann, M. 1995, MNRAS, 273, 475,
  * Senatore (2015) Senatore, L. 2015, J. Cosmology Astropart. Phys, 2015, 007,
  * Sheth & Tormen (1999a) Sheth, R. K., & Tormen, G. 1999a, MNRAS, 308, 119,
  * Sheth & Tormen (1999b) —. 1999b, MNRAS, 308, 119
  * Springel et al. (2001) Springel, V., White, S. D. M., Tormen, G., & Kauffmann, G. 2001, MNRAS, 328, 726,
  * Szalay (1988) Szalay, A. S. 1988, ApJ, 333, 21,
  * Taylor & Hamilton (1996) Taylor, A. N., & Hamilton, A. J. S. 1996, MNRAS, 282, 767,
  * Tinker et al. (2008) Tinker, J., Kravtsov, A. V., Klypin, A., et al. 2008, ApJ, 688, 709,
  * Tinker et al. (2010) Tinker, J. L., Robertson, B. E., Kravtsov, A. V., et al. 2010, ApJ, 724, 878,
  * Vlah et al. (2016) Vlah, Z., Castorina, E., & White, M. 2016, J. Cosmology Astropart. Phys, 2016, 007,
  * Vlah et al. (2015a) Vlah, Z., Seljak, U., & Baldauf, T. 2015a, Phys. Rev. D, 91, 023508,
  * Vlah et al. (2015b) Vlah, Z., White, M., & Aviles, A. 2015b, J. Cosmology Astropart. Phys, 2015, 014,
  * White (2014) White, M. 2014, MNRAS, 439, 3630,
  * White (2014) White, M. 2014, MNRAS, 439, 3630,
  * White et al. (2011) White, M., Blanton, M., Bolton, A., et al. 2011, ApJ, 728, 126,
  * Zel’dovich (1970) Zel’dovich, Y. B. 1970, A&A, 5, 84


* [GS87] Étienne Ghys and Vlad Sergiescu.  Sur un groupe remarquable de difféomorphismes du cercle.  Commentarii Mathematici Helvetici, 62:185–239, 1987.
  * [HN13] Ale Jan Homburg and Meysam Nassiri.  Robust minimality of iterated function systems with two generators.  Ergodic Theory and Dynamical Systems, 34(6):1914–1929, June 2013\.
  * [HPS77] M Hirsch, C Pugh, and Michael Shub.  Invariant manifolds (lecture notes in mathematics, 583), 1977.
  * [Hut81] John E. Hutchinson.  Fractals and self similarity.  Indiana University Mathematics Journal, 30(5):713–747, 1981.
  * [KKO18] Victor Kleptsyn, Yury Kudryashov, and Alexey Okunev.  Classification of generic semigroup actions of circle diffeomorphisms.  arXiv preprint arXiv:1804.00951, 2018.
  * [KN09] Andres Koropecki and Meysam Nassiri.  Transitivity of generic semigroups of area-preserving surface diffeomorphisms.  arXiv preprint arXiv:0905.0305, 2009.
  * [LSSV22] Krzysztof Leśniak, Nina Snigireva, Filip Strobin, and Andrew Vince.  Transition phenomena for the attractor of an iterated function system*.  Nonlinearity, 35(10):5396, 09 2022.
  * [Moi77] Edwin E. Moise.  Homeomorphisms between Cantor sets, pages 83–90.  Springer New York, New York, NY, 1977.
  * [Nav11] Andrés Navas.  Groups of circle diffeomorphisms.  University of Chicago Press, 2011.
  * [Nob15] Felipe Nobili.  Minimality of invariant laminations for partially hyperbolic attractors.  Nonlinearity, 28(6):1897, 2015.
  * [NP12] Meysam Nassiri and Enrique R Pujals.  Robust transitivity in hamiltonian dynamics.  In Annales scientifiques de l’École normale supérieure, volume 45, pages 191–239, 2012.
  * [RHRHU07] Federico Rodriguez Hertz, Maria Alejandra Rodriguez Hertz, and Raul Ures.  Some results on the integrability of the center bundle for partially hyperbolic diffeomorphisms.  In Partially hyperbolic dynamics, laminations, and Teichmüller flow, volume 51 of Fields Inst. Commun., pages 103–109. Amer. Math. Soc., Providence, RI, 2007.
  * [Ryp19] Miroslav Rypka.  Stability of multivalued attractor.  Topological Methods in Nonlinear Analysis, 53(1):97 – 109, 2019\.
  * [Sar23] Aliasghar Sarizadeh.  Attractor for minimal iterated function systems, 2023.
  * [Sri08] Sashi Mohan Srivastava.  A course on Borel sets, volume 180.  Springer Science & Business Media, 2008.


# Evaluating Byte and Wordpiece Level Models
for Massively Multilingual Semantic Parsing

Massimo Nicosia and Francesco Piccinno
Google Research, Zürich
<EMAIL_ADDRESS>
###### Abstract

Token free approaches have been successfully applied to a series of word and
span level tasks. In this work, we compare a byte-level (ByT5) and a wordpiece
based (mT5) sequence to sequence model on the 51 languages of the MASSIVE
multilingual semantic parsing dataset. We examine multiple experimental
settings: (i) zero-shot, (ii) full gold data and (iii) zero-shot with
synthetic data. By leveraging a state-of-the-art label projection method for
machine translated examples, we are able to reduce the gap in exact match
accuracy to only $5$ points with respect to a model trained on gold data from
all the languages. We additionally provide insights on the cross-lingual
transfer of ByT5 and show how the model compares with respect to mT5 across
all parameter sizes.

##  1 Introduction

Semantic parsers map natural languages utterances into logical forms (LFs). In
the context of conversational agents Artzi and Zettlemoyer (2011), robotics
Dukes (2014) or question answering systems Berant et al. (2013), task-oriented
semantic parsers map user queries (e.g. _“set an 8 am alarm”_) to machine
readable LFs (e.g. [IN:CREATE_ALARM [SL:TIME 8 am ]]), in the form of
structured interpretations that can be understood and executed by downstream
components. Learning parsers requires training data in the form of <utterance,
LF> pairs. Such data is costly to obtain especially at large scale Berant et
al. (2013), since expert annotators have to derive the correct LFs given an
input utterance. This problem is exacerbated in a multilingual setting, where
the availability of annotators, especially for non top-tier languages, is
scarce and therefore even more expensive.

With the release of MASSIVE FitzGerald et al. (2022), the research community
has now access to a massively multilingual semantic parsing dataset that can
be used to evaluate large language models fine-tuned on the task and to study
cross-lingual transfer for numerous languages.

On the multilinguality front, token-free models with byte or character based
vocabularies have gained strength given their competitiveness with respect to
traditional subword-based pretrained language models. Models such as ByT5 Xu
et al. (2020), Canine Clark et al. (2022) and the Charformer Tay et al. (2022)
have been applied to popular multilingual benchmarks obtaining state-of-the-
art results.

In this paper, we perform the first in-depth evaluation of a token-free model
in the context of multilingual semantic parsing. We compare the ByT5 and mT5
Xue et al. (2021) models across different parameter sizes and data regime
settings. In addition to that, we build a map of the cross-lingual transfer
for all the languages in MASSIVE. Lastly, we show that with the use of machine
translated synthetic data the accuracy of a state-of-the-art multilingual
parser can be just $5$ points lower than the same parser trained with all the
available multilingual supervision. To incentivize research on synthetic data
augmentation approaches, we release the MASSIVE English training utterances
translated to 50 languages.111We release the translations in 50 languages of
the MASSIVE English training examples obtained with an in-house translation
system at https://goo.gle/massive-translations

##  2 The MASSIVE Dataset

MASSIVE FitzGerald et al. (2022) is a semantic parsing dataset covering $51$
languages, $18$ domains, $60$ intents and $55$ slots. The dataset was created
by professional translators starting from the English SLURP dataset
Bastianelli et al. (2020). A significant portion of the translations have been
localized too, following the recent trend in multilingual benchmarks of
replacing western-centric entities with entities that are more relevant for
the target languages Lin et al. (2021); Ding et al. (2022); Majewska et al.
(2022).

###  2.1 Pre and Post Processing

The annotated instances in the MASSIVE dataset come in the following format:

⬇

intent: alarm_set

annot_utt: despiértame a las [time : nueve de la mañana] el [date : viernes]

To shorten the target output and save the model from generating and
potentially hallucinating unnecessary words, we map the former to the
following format taken from MTOP Li et al. (2021):

⬇

[IN:ALARM_SET [SL:TIME nueve de la mañana ] [SL:DATE viernes ] ]

For evaluation, we use a simple inverse post-processing step based on string
matching to convert the model outputs back to MASSIVE format.

###  2.2 Synthetic Data with Translate-and-Fill

A common approach to create multilingual synthetic data from available
examples is to use machine translation Moradshahi et al. (2020); Sherborne et
al. (2020). Utterances are translated and LF annotations are projected using
word aligners and noise reduction heuristics. We instead adopt the approach
from Nicosia et al. (2021), Translate-and-Fill (TaF), a label projection
method in which a filler model reconstructs the full LF starting from an
utterance and its LF signature.

We train an mT5-xxl filler model on English instances and then directly
generate the LFs of translated examples in a zero-shot fashion. Since the slot
order between English and translated utterances may differ, we canonicalize
the generated synthetic interpretations reordering the slots as they would
occur in the translations. We have also noticed in the filler output that for
some languages the slot boundaries may fall inside words. For languages with
white space tokenization, we move slot boundaries to word boundaries if
needed.

As an example, given an input utterance _“despiértame a las nueve el viernes”_
and [IN:ALARM_SET [SL:DATE el vier ] [SL:TIME nueve ] ] as LF, the process
looks as follows. First the arguments are reordered according to the order of
appearance in the original sentence: [IN:ALARM_SET [SL:TIME nueve ] [SL:DATE
vier ] ]. Then slot boundaries that fall within words are extended, correcting
the prediction for the second argument from [SL:DATE vier ] to [SL:DATE
viernes ].

##  3 Experiments

We use MASSIVE as a test bed for two model families, ByT5 and mT5, evaluating
them at all sizes in three different data settings. We report _Intent
Accuracy_ (IA) and _Exact Match_ (EM) accuracy. We do not perform any hyper-
parameter tuning: we train for $\mathrm{3}\mathrm{0}\mathrm{K}$ steps with a
fixed learning rate of 0.0001 and a batch size of $128$ for all models but
xxl, for which batch size was reduced to $32$. We run fine tuning on Cloud TPU
v3 with an input/target length of $1024$/$512$ for ByT5 and $512$/$512$ for
mT5. To minimize compute, all the reported results are from single runs. We
experiment with three different settings, summarized below:

  1. 1.

Zero-shot setting. Training is performed on English data only, and the model
selection is done on the English development set. Results are reported in
Table 1.

  2. 2.

Gold-data setting. Training is performed on all the MASSIVE data, that
includes 51 languages. Model selection is performed averaging the accuracy on
the multilingual development sets. Results are reported Table 2.

  3. 3.

Synthetic data setting (TaF). Training is performed on English and
multilingual data that is synthetically generated via TaF. Results are
reported in Table 3. Our entry based on this approach ranked 1st in the Zero-
Shot Task of the MMNLU-22 Multilingual Semantic Parsing competition organized
by Amazon and co-located with EMNLP 2022.222https://mmnlu-22.github.io

Model | IA | EM |
---|---|---|---
ByT5-small | $49.26$ | $20.36$ |
ByT5-base | $64.3$ | $33.47$ |
ByT5-large | $66.53$ | $28.43$ |
ByT5-xl | $80.96$ | $41.7$ |
ByT5-xxl | $81.73$ | $38.28$ |
mT5-small | $51.75$ | $17.59$ |
mT5-base | $55.91$ | $17.73$ |
mT5-large | $67.23$ | $25.14$ |
mT5-xl | $79.97$ | $45.60$ |
mT5-xxl | 82.44 | 50.21 |
Table 1: Zero-shot *T5 parsers performance when training on English only. Model | IA | EM |
---|---|---|---
ByT5-small | $85.59$ | $66.60$ |
ByT5-base | $85.93$ | $67.54$ |
ByT5-large | $84.02$ | $62.92$ |
ByT5-xl | $87.01$ | $68.29$ |
ByT5-xxl | 87.27 | 68.66 |
mT5-small | $73.29$ | $46.65$ |
mT5-base | $82.03$ | $58.24$ |
mT5-large | $85.58$ | $64.13$ |
mT5-xl | $87.24$ | $68.47$ |
mT5-xxl | $86.79$ | $63.33$ |
Table 2: *T5 parsers performance when training on all the available gold data.

We can see a pattern that is common to all the experiments: at smaller sizes,
ByT5 has much better EM accuracy then the corresponding mT5 models. As stated
in Xu et al. (2020), this may be explained by the fact that at these sizes
less than 0.3% of ByT5 parameters are locked in embedding tables and a larger
amount of dense parameters is updated during training. mT5 parameters are
instead dominated by the embedding tables, which are updated less often than
the dense layers. In addition to that, ByT5-large is worse than ByT5-base at
span labeling, which is a word level task. Both our observations confirm the
findings in Xu et al. (2020).

Model | IA | EM |
---|---|---|---
ByT5-small | $83.32$ | $59.32$ |
ByT5-base | $84.59$ | $61.24$ |
ByT5-large | $82.82$ | $58.09$ |
ByT5-xl | $85.90$ | $62.98$ |
ByT5-xxl | $86.48$ | 64.18 |
mT5-small | $73.64$ | $43.19$ |
mT5-base | $80.79$ | $51.76$ |
mT5-large | $83.99$ | $57.43$ |
mT5-xl | $86.07$ | $62.33$ |
mT5-xxl | 86.69 | $62.49$ |
Table 3: *T5 parsers performance when training on English and synthetic TaF
data.

In the synthetic data setting (Table 3), IA almost matches the IA of models
from the gold data setting. If we consider EM accuracy, we are only 5% points
behind the upper bound performance of the multilingually supervised -xxl
models (see Table 2). This indicates that synthetic data augmentation is a
viable approach for the i18n of semantic parsers. Please refer to Table 9 in
the appendix for results on individual languages.

##  4 Additional Experiments and Results

In zero-shot evaluations, English is the most studied language given the
availability of labeled data. Recent work has shown that this language may not
be the best at cross-lingual transfer Turc et al. (2021). Since MASSIVE
provides training and test data for all its languages, we can evaluate the
zero-shot performance of each language. We train $51$ ByT5-base model for a
fixed number of steps ($\mathrm{1}\mathrm{k}$ steps, $128$ batch size) and
collect the results on the development sets in Figure 2. By summing the EMs on
rows we can understand how much a fine-tuning language (donor) improves the
others. If we sum over columns, we can see how much transfer a target language
(receiver) gets from the others. We report some statistics about best/worst
donor/receiver languages in Table 4. Interestingly, English is not among the
top donors, while it is the one that is being improved the most by other
languages. We speculate that the better English LM representations may already
have an intrinsic notion of semantic concepts that are then quickly
individuated if supervision for such concepts is provided in other languages.
From Figure 2, we see that some languages (am, sw, km, cy) clearly need
annotated data. We hope that this map could help prioritize data collection
efforts.

Figure 1: Differences in EM for an mT5-xxl model evaluated on queries of the
test set that have been both translated and localized, vs only translated.

MASSIVE examples contain an interesting piece of metadata that indicates if an
utterance has been translated and localized (i.e. original entities have been
substituted with entities more culturally relevant for the target language),
or translated only. We split the test sets in two parts according to this
information and report in Figure 1 the EM accuracies of the same mT5-xxl
model. We examine the three data settings studied in this paper. Accuracies on
localized utterances are consistently lower. The performance difference in the
synthetic data setting is relatively small but it still suggests that creating
synthetic examples with entities that are local to the target language may
improve the robustness of the parser.

Figure 2: Zero-shot EM accuracies of individual ByT5-base models fine-tuned on a single language (y-axis) and evaluated on dev sets from all languages (x-axis). | Best to worst
---|---
Donor | fr, de, es, nl, pl, $\cdots$, mn, am, sw, km, cy
Receiver | en, de, pt, fr, sv, $\cdots$, zh, am, mn, sw, cy
Table 4: Top-5 Best/worst donor/receiver.

In the appendix, we report the accuracy for each individual intent on the
union of the test set examples from all languages (Table 8). In Table 5, we
report the 6 intents with the lowest accuracy. Most examples belong to the
GENERAL_QUIRKY intent. The latter is likely a bucket intent covering all the
utterances that are generic or out-of-domain (we could not find an exhaustive
description of this intent in the SLURP datasetBastianelli et al. (2020)). The
common parser mistake is to classify such queries as belonging to a more
specific intent that can plausibly be associated with that query.

Intent | IA | Support
---|---|---
GENERAL_GREET | $19.6$ | $51$
MUSIC_SETTINGS | $27.1$ | $306$
AUDIO_VOLUME_OTHER | $54.9$ | $306$
GENERAL_QUIRKY | $55.6$ | $8619$
IOT_HUE_LIGHTON | $61.4$ | $153$
MUSIC_DISLIKENESS | $74.5$ | $204$
Table 5: IA of the ByT5-xxl+TaF model for the lowest scoring intents
(considering all languages).

Finally, we compare our NMT translations of the training set with the
corresponding gold translations produced by professional translators. We
summarize the most interesting information in Table 6 (full comparison in
Table 8 included in the appendix). Indic languages (*_IN and bn_BD) have an
higher average match than other languages. This may suggest that translations
in these languages are more unambiguous or that translators may have relied on
a MT during the translation task.

Language sets |  Avg Match (%)
---|---
All languages | $21.3$
All but Indic languages | $17.3$
Indic languages | $50.8$
Table 6: Percentages of NMT translations matching human translations in
MASSIVE training set.

##  5 Related Work

Multilingual models are architecturally similar to monolingual transformer-
based models but they are pretrained on multilingual corpora. These models
include XLM Lample and Conneau (2019), XLM-R Conneau et al. (2020) and mT5 Xue
et al. (2021), the multilingual version of T5 Raffel et al. (2020). They all
use a subword vocabulary, a choice that may result in poor performance for
languages with limited amount of data Wang et al. (2021). Token-free models
such as ByT5 Xu et al. (2020), Canine Clark et al. (2022) and Charformer Tay
et al. (2022) were designed to avoid this issue and have been applied to
popular multilingual benchmarks obtaining state-of-the-art results. In this
work, we compare the multilinguality and the generative capabilities of mT5
and ByT5 in a massively multilingual semantic parsing task.

Data augmentation is the process of creating synthetic labeled data from
available annotated examples. One approach in the multilinguality space is to
translate annotated data in one language, e.g. English, to other languages.
Neural machine translation is a strong baseline as it has been shown in recent
cross-lingual evaluation benchmarks Hu et al. (2020); Ladhak et al. (2020).
While translation works quite well for classification tasks where the label is
at instance level, sequence tagging or parsing tasks require an annotation
projection step because labels are at token level. Translate-and-align methods
use bilingual word aligners, statistical Brown et al. (1993); Vogel et al.
(1996); Och and Ney (2000, 2003), and neural Schuster et al. (2019); Chen et
al. (2020); Zenkel et al. (2020). More recent works removes this explicit
alignment requirement Dong and Lapata (2018); Zhang et al. (2019); Wiseman et
al. (2018). In our work, we use a label projection method based on pretrained
language models Nicosia et al. (2021) that reconstructs a full semantic parse
from an utterance and a signature of the same parse.

##  6 Conclusions

In this paper, we evaluated ByT5 and mT5 Xue et al. (2021) models in a
massively multilingual semantic parsing task, showing that ByT5 is
particularly competitive at smaller sizes. We have provided a map of the
cross-lingual transfer for all the languages in MASSIVE and demonstrated that
synthetic examples created with NMT are effective for building accurate
semantic parsers.

## Limitations

This work uses seq2seq models as parsers. Different output formats can yield
better or worse results as shown in Paolini et al. (2021). We do not focus on
tweaking formats or on modeling improvements such as constrained decoding for
a more faithful generation. We adopt a compact output representation that
reduces the text the model has to generate (and hallucinations) and gives us
competitive results. In the cross-lingual transfer experiments, we train each
model for a small fixed number of steps. If we train for longer, the
representations start to change significantly and cross-lingual performances
vary quite unpredictably. We leave for the future an investigation of the
learning dynamics in this setting and the design of possible remedies.

## References

  * Artzi and Zettlemoyer (2011) Yoav Artzi and Luke Zettlemoyer. 2011.  Bootstrapping semantic parsers from conversations.  In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_ , pages 421–432.
  * Bastianelli et al. (2020) Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. 2020\.  SLURP: A spoken language understanding resource package.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7252–7262, Online. Association for Computational Linguistics.
  * Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013.  Semantic parsing on freebase from question-answer pairs.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pages 1533–1544.
  * Brown et al. (1993) Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993.  The mathematics of statistical machine translation: Parameter estimation.  _Computational Linguistics_ , 19(2):263–311.
  * Chen et al. (2020) Yun Chen, Yang Liu, Guanhua Chen, Xin Jiang, and Qun Liu. 2020.  Accurate word alignment induction from neural machine translation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 566–576, Online. Association for Computational Linguistics.
  * Clark et al. (2022) Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022.  Canine: Pre-training an efficient tokenization-free encoder for language representation.  _Transactions of the Association for Computational Linguistics_ , 10:73–91.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Ding et al. (2022) Bosheng Ding, Junjie Hu, Lidong Bing, Mahani Aljunied, Shafiq Joty, Luo Si, and Chunyan Miao. 2022.  GlobalWoZ: Globalizing MultiWoZ to develop multilingual task-oriented dialogue systems.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1639–1657, Dublin, Ireland. Association for Computational Linguistics.
  * Dong and Lapata (2018) Li Dong and Mirella Lapata. 2018.  Coarse-to-fine decoding for neural semantic parsing.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 731–742, Melbourne, Australia. Association for Computational Linguistics.
  * Dukes (2014) Kais Dukes. 2014.  Semeval-2014 task 6: Supervised semantic parsing of robotic spatial commands.  In _SemEval@ COLING_ , pages 45–53.
  * FitzGerald et al. (2022) Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natarajan. 2022.  Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 4411–4421. PMLR.
  * Ladhak et al. (2020) Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020.  WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4034–4048, Online. Association for Computational Linguistics.
  * Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019.  Cross-lingual language model pretraining.  _CoRR_ , abs/1901.07291.
  * Li et al. (2021) Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021.  MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2950–2962, Online. Association for Computational Linguistics.
  * Lin et al. (2021) Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, Peng Xu, Feijun Jiang, Yuxiang Hu, Chen Shi, and Pascale Fung. 2021.  Bitod: A bilingual multi-domain dataset for task-oriented dialogue modeling.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_.
  * Majewska et al. (2022) Olga Majewska, Evgeniia Razumovskaia, Edoardo Maria Ponti, Ivan Vulić, and Anna Korhonen. 2022.  Cross-lingual dialogue dataset creation via outline-based generation.
  * Moradshahi et al. (2020) Mehrad Moradshahi, Giovanni Campagna, Sina Semnani, Silei Xu, and Monica Lam. 2020\.  Localizing open-ontology QA semantic parsers in a day using machine translation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5970–5983, Online. Association for Computational Linguistics.
  * Nicosia et al. (2021) Massimo Nicosia, Zhongdi Qu, and Yasemin Altun. 2021.  Translate & Fill: Improving zero-shot multilingual semantic parsing with synthetic data.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 3272–3284, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Och and Ney (2000) Franz Josef Och and Hermann Ney. 2000.  Improved statistical alignment models.  In _Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics_ , pages 440–447, Hong Kong. Association for Computational Linguistics.
  * Och and Ney (2003) Franz Josef Och and Hermann Ney. 2003.  A systematic comparison of various statistical alignment models.  _Computational Linguistics_ , 29(1):19–51.
  * Paolini et al. (2021) Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, RISHITA ANUBHAI, Cicero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021\.  Structured prediction as translation between augmented natural languages.  In _International Conference on Learning Representations_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Schuster et al. (2019) Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019.  Cross-lingual transfer learning for multilingual task oriented dialog.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3795–3805. Association for Computational Linguistics.
  * Sherborne et al. (2020) Tom Sherborne, Yumo Xu, and Mirella Lapata. 2020.  Bootstrapping a crosslingual semantic parser.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 499–517, Online. Association for Computational Linguistics.
  * Tay et al. (2022) Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2022.  Charformer: Fast character transformers via gradient-based subword tokenization.  In _International Conference on Learning Representations_.
  * Turc et al. (2021) Iulia Turc, Kenton Lee, Jacob Eisenstein, Ming-Wei Chang, and Kristina Toutanova. 2021.  Revisiting the primacy of english in zero-shot cross-lingual transfer.  _ArXiv_ , abs/2106.16171.
  * Vogel et al. (1996) Stephan Vogel, Hermann Ney, and Christoph Tillmann. 1996.  HMM-based word alignment in statistical translation.  In _COLING 1996 Volume 2: The 16th International Conference on Computational Linguistics_.
  * Wang et al. (2021) Xinyi Wang, Sebastian Ruder, and Graham Neubig. 2021.  Multi-view subword regularization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 473–482, Online. Association for Computational Linguistics.
  * Wiseman et al. (2018) Sam Wiseman, Stuart Shieber, and Alexander Rush. 2018.  Learning neural templates for text generation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3174–3187, Brussels, Belgium. Association for Computational Linguistics.
  * Xu et al. (2020) Weijia Xu, Batool Haider, and Saab Mansour. 2020.  End-to-end slot alignment and recognition for cross-lingual NLU.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5052–5063, Online. Association for Computational Linguistics.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mt5: A massively multilingual pre-trained text-to-text transformer.
  * Zenkel et al. (2020) Thomas Zenkel, Joern Wuebker, and John DeNero. 2020.  End-to-end neural word alignment outperforms GIZA++.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1605–1617, Online. Association for Computational Linguistics.
  * Zhang et al. (2019) Xiang Zhang, Shizhu He, Kang Liu, and Jun Zhao. 2019.  AdaNSP: Uncertainty-driven adaptive decoding in neural semantic parsing.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4265–4270, Florence, Italy. Association for Computational Linguistics.

##  Appendix A Comparing NMT with Gold Translations

In Table 8, we compare how many times the NMT translated utterances match the
gold translations produced by professional translators. We restrict the match
to utterances that have been translated and not localized in the target
language, since NMT cannot perform the localization step. In addition, we
preprocess all compared utterances with unicode normalization, we strip
whitespaces and punctuation. In general, indic locales have higher match rates
compared to other locales. Please also note that we translate English to pt_BR
(Brazilian Portuguese) and this explains the low match for pt_PT.

##  Appendix B Intent Accuracy Performance

In Table 8, we report the accuracy for each individual intent on the union of
the test set examples from all languages using ByT5-xxl + TaF.

| NMT vs Gold | Non-localized
---|---|---
| Translations Matches | sentences
Language | (%) | (#) | (#)
kn_IN | $68.7$ | $6524$ | $9497$
te_IN | $54.1$ | $4841$ | $8941$
bn_BD | $52.6$ | $4458$ | $8471$
ta_IN | $48.3$ | $4301$ | $8898$
hi_IN | $46.5$ | $4101$ | $8827$
nl_NL | $38.5$ | $3878$ | $10\,070$
fr_FR | $36.0$ | $3736$ | $10\,385$
ml_IN | $34.7$ | $2985$ | $8607$
tl_PH | $34.0$ | $3397$ | $10\,000$
af_ZA | $32.8$ | $3160$ | $9640$
tr_TR | $32.1$ | $2998$ | $9330$
sw_KE | $26.1$ | $2336$ | $8965$
sv_SE | $25.9$ | $2465$ | $9504$
nb_NO | $23.8$ | $2402$ | $10\,083$
vi_VN | $21.6$ | $2000$ | $9255$
ms_MY | $21.6$ | $1880$ | $8702$
jv_ID | $21.1$ | $1947$ | $9208$
pl_PL | $21.0$ | $2017$ | $9618$
da_DK | $20.4$ | $1933$ | $9470$
id_ID | $20.4$ | $1882$ | $9227$
es_ES | $19.5$ | $1876$ | $9596$
zh_CN | $19.0$ | $1661$ | $8727$
zh_TW | $18.2$ | $1638$ | $8976$
it_IT | $17.9$ | $1596$ | $8916$
fi_FI | $17.5$ | $1669$ | $9558$
ru_RU | $17.4$ | $1550$ | $8912$
hy_AM | $16.9$ | $1809$ | $10\,707$
is_IS | $16.1$ | $1491$ | $9270$
km_KH | $16.1$ | $1491$ | $9276$
cy_GB | $15.9$ | $1578$ | $9936$
sl_SL | $14.7$ | $1313$ | $8913$
am_ET | $14.6$ | $1267$ | $8658$
hu_HU | $14.5$ | $1331$ | $9198$
ur_PK | $14.4$ | $1260$ | $8761$
de_DE | $14.2$ | $1422$ | $9992$
lv_LV | $12.4$ | $1071$ | $8650$
he_IL | $12.3$ | $1123$ | $9159$
sq_AL | $12.2$ | $1035$ | $8460$
az_AZ | $12.1$ | $1102$ | $9081$
th_TH | $11.7$ | $1041$ | $8894$
ro_RO | $10.9$ | $1001$ | $9197$
el_GR | $10.5$ | $934$ | $8879$
pt_PT | $9.9$ | $934$ | $9392$
ar_SA | $9.9$ | $871$ | $8814$
mn_MN | $8.9$ | $785$ | $8826$
fa_IR | $8.3$ | $718$ | $8686$
ja_JP | $7.4$ | $704$ | $9487$
ka_GE | $7.4$ | $701$ | $9528$
ko_KR | $3.9$ | $341$ | $8804$
my_MM | $2.0$ | $171$ | $8765$
Table 7: Number of verbatim matches between Gold translation and NMT
translations.

Intent | IA | Support
---|---|---
GENERAL_GREET | $19.6$ | $51$
MUSIC_SETTINGS | $27.1$ | $306$
AUDIO_VOLUME_OTHER | $54.9$ | $306$
GENERAL_QUIRKY | $55.6$ | $8619$
IOT_HUE_LIGHTON | $61.4$ | $153$
MUSIC_DISLIKENESS | $74.5$ | $204$
DATETIME_CONVERT | $75.6$ | $765$
IOT_WEMO_ON | $76.3$ | $510$
PLAY_AUDIOBOOK | $78.0$ | $2091$
TRANSPORT_QUERY | $78.1$ | $2601$
RECOMMENDATION_EVENTS | $78.3$ | $2193$
RECOMMENDATION_MOVIES | $79.2$ | $1020$
CALENDAR_QUERY | $80.6$ | $6426$
QA_FACTOID | $82.4$ | $7191$
IOT_HUE_LIGHTUP | $82.5$ | $1377$
LISTS_QUERY | $82.6$ | $2601$
AUDIO_VOLUME_UP | $83.0$ | $663$
SOCIAL_QUERY | $83.9$ | $1275$
MUSIC_QUERY | $84.0$ | $1785$
EMAIL_ADDCONTACT | $84.5$ | $612$
MUSIC_LIKENESS | $84.7$ | $1836$
EMAIL_QUERYCONTACT | $84.8$ | $1326$
TAKEAWAY_QUERY | $85.0$ | $1785$
LISTS_CREATEORADD | $85.6$ | $1989$
QA_DEFINITION | $86.3$ | $2907$
LISTS_REMOVE | $86.3$ | $2652$
COOKING_RECIPE | $86.6$ | $3672$
NEWS_QUERY | $86.9$ | $6324$
PLAY_MUSIC | $87.1$ | $8976$
TAKEAWAY_ORDER | $87.3$ | $1122$
IOT_HUE_LIGHTDIM | $87.4$ | $1071$
PLAY_PODCASTS | $87.6$ | $3213$
PLAY_GAME | $87.7$ | $1785$
ALARM_SET | $89.5$ | $2091$
PLAY_RADIO | $90.0$ | $3672$
CALENDAR_SET | $90.2$ | $10\,659$
RECOMMENDATION_LOCATIONS | $90.4$ | $1581$
QA_MATHS | $90.7$ | $1275$
AUDIO_VOLUME_DOWN | $90.7$ | $561$
SOCIAL_POST | $91.1$ | $4131$
IOT_WEMO_OFF | $91.3$ | $918$
AUDIO_VOLUME_MUTE | $91.7$ | $1632$
ALARM_QUERY | $91.8$ | $1734$
GENERAL_JOKE | $92.0$ | $969$
EMAIL_QUERY | $93.0$ | $6069$
TRANSPORT_TICKET | $93.1$ | $1785$
CALENDAR_REMOVE | $93.4$ | $3417$
EMAIL_SENDEMAIL | $94.0$ | $5814$
IOT_CLEANING | $94.2$ | $1326$
WEATHER_QUERY | $94.6$ | $7956$
IOT_HUE_LIGHTOFF | $94.8$ | $2193$
TRANSPORT_TAXI | $95.3$ | $1173$
IOT_HUE_LIGHTCHANGE | $95.4$ | $1836$
ALARM_REMOVE | $95.5$ | $1071$
QA_STOCK | $95.6$ | $1326$
DATETIME_QUERY | $95.8$ | $4488$
TRANSPORT_TRAFFIC | $96.3$ | $765$
QA_CURRENCY | $96.6$ | $1989$
IOT_COFFEE | $97.9$ | $1836$
Table 8: IA of the ByT5-xxl+TaF model for all intents (all languages).

##  Appendix C Performance on all Languages

In Table 9, we report Exact Match on all the 51 languages, for the three
different experimental setups described in Section 3, across two models (mT5
and ByT5) and two model sizes (base and xxl).

| Zero Shot | Synthetic (TaF) | Gold
---|---|---|---
| base | xxl | base | xxl | base | xxl
Language | mT5 | ByT5 | mT5 | ByT5 | mT5 | ByT5 | mT5 | ByT5 | mT5 | ByT5 | mT5 | ByT5
af_ZA | $21.6$ | 51.1 | $58.0$ | 59.7 | $53.7$ | 64.7 | $65.6$ | 66.8 | $59.4$ | 68.5 | $65.9$ | 69.3
am_ET | $4.7$ | 15.9 | 40.7 | $22.0$ | $40.8$ | 54.4 | 61.2 | $61.0$ | $48.7$ | 61.3 | $62.0$ | 65.8
ar_SA | $14.6$ | 27.8 | 43.6 | $23.3$ | $45.9$ | 56.1 | $60.1$ | 60.5 | $52.3$ | 64.7 | $61.1$ | 66.0
az_AZ | $8.9$ | 31.2 | 41.8 | $34.0$ | $46.4$ | 61.6 | $61.9$ | 63.6 | $57.0$ | 69.0 | $62.6$ | 69.6
bn_BD | $10.8$ | 19.5 | 45.9 | $25.3$ | $51.0$ | 62.1 | $64.3$ | 65.6 | $57.6$ | 67.6 | $64.6$ | 69.5
cy_GB | $5.9$ | 16.4 | 42.8 | $40.2$ | $35.7$ | 56.1 | $61.5$ | 64.2 | $42.1$ | 65.3 | $61.4$ | 69.2
da_DK | $30.2$ | 53.1 | 60.9 | $54.2$ | $57.8$ | 67.5 | $67.3$ | 68.7 | $64.4$ | 71.7 | $67.9$ | 71.3
de_DE | $28.3$ | 55.3 | 59.8 | $59.5$ | $60.2$ | 67.8 | $67.5$ | 68.8 | $64.1$ | 70.4 | $68.0$ | 70.2
el_GR | $17.4$ | 31.5 | 57.2 | $27.9$ | $55.5$ | 64.2 | $65.5$ | 66.6 | $62.0$ | 68.3 | $66.6$ | 68.7
en_US | $65.5$ | 72.2 | 74.0 | $73.3$ | $68.5$ | 72.6 | 73.7 | $73.0$ | $68.9$ | 72.7 | 73.3 | $72.6$
es_ES | $26.1$ | 50.8 | 55.6 | $52.2$ | $58.7$ | 65.1 | $65.0$ | 65.9 | $61.1$ | 67.2 | $65.9$ | 66.2
fa_IR | $17.6$ | 32.8 | 54.4 | $24.0$ | $54.9$ | 62.2 | $63.2$ | 64.4 | $59.9$ | 69.1 | $63.4$ | 69.7
fi_FI | $16.3$ | 36.9 | 52.5 | $47.4$ | $51.2$ | 65.9 | $65.6$ | 68.2 | $59.4$ | 71.1 | $66.8$ | 71.5
fr_FR | $29.9$ | 53.5 | 58.5 | $54.3$ | $59.3$ | 64.4 | $65.1$ | 65.6 | $62.3$ | 66.5 | $65.8$ | 67.2
he_IL | $9.7$ | 21.0 | 40.4 | $24.0$ | $50.1$ | 59.4 | $61.0$ | 63.2 | $57.5$ | 67.3 | $62.3$ | 68.4
hi_IN | $14.1$ | 26.3 | 52.9 | $26.2$ | $54.4$ | 62.6 | $64.2$ | 64.4 | $59.3$ | 66.5 | $64.5$ | 67.2
hu_HU | $17.5$ | 33.5 | 45.3 | $32.9$ | $51.8$ | 62.2 | $64.2$ | $64.2$ | $58.2$ | 68.5 | $65.2$ | 69.5
hy_AM | $11.7$ | 20.5 | 44.6 | $24.7$ | $49.8$ | 58.4 | $60.3$ | 62.2 | $57.8$ | 67.7 | $61.7$ | 68.9
id_ID | $24.1$ | 48.3 | $58.6$ | 61.5 | $59.0$ | 64.6 | $65.5$ | 67.1 | $63.4$ | 68.8 | $66.2$ | 69.0
is_IS | $11.6$ | 32.1 | 47.2 | $31.7$ | $47.6$ | 60.9 | $63.4$ | 65.9 | $54.6$ | 68.5 | $63.4$ | 69.6
it_IT | $25.3$ | 52.5 | $59.5$ | $59.5$ | $57.2$ | 63.0 | $64.6$ | 65.5 | $60.2$ | 67.6 | $65.7$ | 67.3
ja_JP | 26.8 | $23.3$ | 46.6 | $29.3$ | $51.0$ | 55.6 | $57.3$ | 58.8 | $60.5$ | 65.8 | $58.7$ | 67.0
jv_ID | $10.7$ | 22.9 | $45.8$ | 46.2 | $42.5$ | 58.9 | $62.1$ | 63.9 | $48.5$ | 66.5 | $62.6$ | 68.5
ka_GE | $9.7$ | 17.9 | 39.9 | $22.1$ | $45.4$ | 52.9 | $54.8$ | 57.1 | $54.5$ | 63.8 | $56.2$ | 66.8
km_KH | $11.4$ | 18.0 | 44.8 | $23.6$ | $39.2$ | 51.8 | $51.7$ | 55.7 | $54.7$ | 63.8 | $54.3$ | 67.0
kn_IN | $8.8$ | 20.2 | 41.9 | $25.4$ | $47.4$ | 58.6 | $55.8$ | 61.7 | $52.1$ | 63.8 | $56.6$ | 65.8
ko_KR | $11.0$ | 16.3 | 49.8 | $24.8$ | $54.1$ | 61.5 | $65.6$ | 65.8 | $60.2$ | 68.7 | $66.4$ | 70.3
lv_LV | $11.6$ | 40.3 | 51.9 | $33.7$ | $52.4$ | 61.2 | $63.0$ | 64.6 | $59.0$ | 69.6 | $64.1$ | 70.4
ml_IN | $10.1$ | 19.4 | 41.2 | $25.8$ | $47.9$ | 55.3 | $55.0$ | 58.5 | $59.4$ | 68.2 | $55.6$ | 69.2
mn_MN | $7.4$ | 13.4 | 38.9 | $22.2$ | $46.9$ | 57.0 | $60.2$ | 62.7 | $53.8$ | 66.1 | $61.5$ | 68.7
ms_MY | $21.7$ | 45.0 | $54.8$ | 59.9 | $57.1$ | 65.7 | $67.7$ | 68.0 | $60.6$ | 69.3 | $68.4$ | 68.9
my_MM | $10.7$ | 13.8 | 48.7 | $23.1$ | $51.5$ | 59.8 | $61.9$ | 66.1 | $59.3$ | 68.8 | $64.3$ | 72.6
nb_NO | $26.9$ | 50.6 | 60.7 | $56.3$ | $60.7$ | 68.0 | $68.8$ | 70.2 | $65.0$ | 70.5 | $69.9$ | 70.7
nl_NL | $28.3$ | 55.2 | $60.1$ | 63.3 | $60.2$ | 66.5 | $67.4$ | 67.5 | $64.7$ | 68.4 | $68.3$ | 70.0
pl_PL | $19.0$ | 47.1 | 50.7 | $46.0$ | $56.2$ | 61.8 | $62.0$ | 63.3 | $59.7$ | 65.9 | $62.5$ | 66.5
pt_PT | $28.1$ | 52.0 | 60.8 | $50.6$ | $61.5$ | 65.9 | $66.8$ | 67.6 | $63.6$ | 68.7 | $67.5$ | 68.2
ro_RO | $22.8$ | 45.7 | 57.4 | $52.7$ | $55.8$ | 64.5 | $65.7$ | 67.1 | $60.2$ | 68.5 | $65.9$ | 69.6
ru_RU | $19.0$ | 26.1 | 49.0 | $26.1$ | $56.9$ | 61.6 | $63.5$ | 63.8 | $63.5$ | 68.8 | $64.0$ | 69.5
sl_SL | $15.8$ | 43.7 | 52.8 | $47.8$ | $53.2$ | 63.5 | $64.5$ | 64.8 | $57.7$ | 68.0 | $64.5$ | 68.8
sq_AL | $15.3$ | 42.1 | 48.0 | $39.9$ | $48.8$ | 61.1 | $61.2$ | 63.5 | $54.2$ | 68.9 | $61.3$ | 68.5
sv_SE | $26.0$ | 54.4 | 61.8 | $53.0$ | $62.6$ | 70.1 | $70.6$ | 71.1 | $65.9$ | 72.0 | $71.2$ | 71.5
sw_KE | $9.6$ | 15.6 | 44.0 | $41.9$ | $44.2$ | 58.7 | $58.2$ | 59.6 | $48.0$ | 66.3 | $58.6$ | 66.8
ta_IN | $10.9$ | 19.9 | 41.1 | $24.3$ | $48.2$ | 55.5 | $56.4$ | 58.3 | $56.6$ | 64.9 | $58.0$ | 66.0
te_IN | $7.8$ | 21.6 | 46.4 | $25.1$ | $43.6$ | 60.0 | $55.4$ | 62.7 | $51.4$ | 65.0 | $55.1$ | 67.5
th_TH | $21.8$ | 31.3 | 55.0 | $26.8$ | $47.4$ | 62.1 | $62.2$ | 66.9 | $63.2$ | 72.0 | $64.6$ | 74.2
tl_PH | $18.9$ | 42.0 | $56.9$ | 58.7 | $53.2$ | 62.4 | $65.7$ | 66.1 | $56.7$ | 66.5 | $66.5$ | 68.5
tr_TR | $14.4$ | 35.2 | 48.4 | $38.5$ | $51.6$ | 64.9 | $65.5$ | 66.2 | $58.5$ | 69.4 | $65.5$ | 69.4
ur_PK | $9.7$ | 22.7 | 49.2 | $22.8$ | $50.5$ | 59.5 | $61.5$ | 61.9 | $54.1$ | 63.3 | $62.6$ | 65.7
vi_VN | $15.1$ | 35.1 | 55.9 | $36.4$ | $49.8$ | 57.5 | $61.0$ | 62.3 | $55.5$ | 67.0 | $62.1$ | 68.2
zh_CN | 22.1 | $17.3$ | 31.7 | $24.1$ | $45.6$ | 54.1 | $53.0$ | 57.9 | $60.8$ | 65.9 | $54.9$ | 66.6
zh_TW | 21.2 | $16.5$ | 32.4 | $24.2$ | $45.2$ | 51.8 | $52.0$ | 54.5 | $58.2$ | 62.2 | $53.8$ | 63.9
Average | $17.7$ | 33.5 | 50.2 | $38.3$ | $51.8$ | 61.2 | $62.5$ | 64.2 | $58.2$ | 67.5 | $63.3$ | 68.7
Table 9: *T5 parsers Exact Match on individual languages in the Zero-Shot, TaF
and Gold settings.