[acronym]long-short

# A unified machine learning approach for reconstructing hadronically decaying
tau leptons

Tani, Laurits1
<EMAIL_ADDRESS>Seeba, Nalong-Norman1
<EMAIL_ADDRESS>Vanaveski, Hardi1
<EMAIL_ADDRESS>Pata, Joosep1
<EMAIL_ADDRESS>Lange, Torben1
<EMAIL_ADDRESS>
(1 _National Institute Of Chemical Physics And Biophysics (NICPB), Rävala pst.
10, 10143 Tallinn, Estonia_
)

###### Abstract

Tau leptons serve as an important tool for studying the production of Higgs
and electroweak bosons, both within and beyond the Standard Model of particle
physics. Accurate reconstruction and identification of hadronically decaying
tau leptons is a crucial task for current and future high energy physics
experiments. Given the advances in jet tagging, we demonstrate how tau lepton
reconstruction can be decomposed into tau identification, kinematic
reconstruction, and decay mode classification in a multi-task machine learning
setup. Based on an electron-positron collision dataset with full detector
simulation and reconstruction, we show that common jet tagging architectures
can be effectively used for these subtasks. We achieve comparable momentum
resolutions of 2-3% with all the tested models, while the precision of
reconstructing individual decay modes is between 80—95%. This paper also
serves as an introduction to a new publicly available
$\mathtt{Fu}\tau\mathtt{ure}$ dataset and provides recipes for the development
and training of tau reconstruction algorithms, while allowing to study
resilience to domain shifts and the use of foundation models for such tasks.

##  1 Introduction

Tau leptons ($\tau$) serve as an important tool for tests of the Standard
Model of particle physics (SM) in the electroweak sector as well as for
searches physics beyond the SM (BSM) both at current and future high energy
physics experiments.

Due to its relatively high mass, the $\tau$ lepton couples strongly to the
Higgs boson ($H$), allowing to test the Higgs boson coupling to third
generation fermions [1]. As $H\rightarrow\tau\tau$ decays have a rather clean
signature, they allow to investigate also other Higgs boson properties and
rare Higgs processes. These include, for example, Higgs boson pair production,
directly probing the Higgs boson potential [2, 3], and the Higgs bosons CP
properties [4, 5].

Furthermore, studying $\tau$ lepton properties — such as its mass, lifetime
and branching fractions — allows us to test lepton universality of charged
current between different fermion generations, while spin polarization
measurements of the $\tau$ leptons allows us to probe neutral-current
interactions [6, 7].

Next, the $\tau$ can also be used to search for lepton flavor violating
processes in $\tau$ lepton decays [6, 7] as well as in decays of $Z$ bosons
[8, 9] and Higgs bosons [10, 11] into a $\tau$ lepton and an electron or muon,
forbidden in the SM.

Measuring $\tau$ lepton properties enables one to test a variety of BSM
theories predicting new particles decaying into $\tau$ leptons. These theories
include models with additional heavy charged and neutral gauge bosons [12, 13,
14], models of third generation leptoquarks [1], supersymmetric models [15,
16, 17, 18, 19, 20, 21, 22, 23], and models with an extended Higgs sector [24,
25, 26, 27, 28].

$\tau$ leptons have a very short lifetime of only $2.9\times 10^{-13}$ seconds
[29], short enough to decay before interacting with the detector material or
before undergoing any radiative processes. In about a third of the cases, the
$\tau$ decays into another, lighter lepton (electron or muon) and a neutrino.
As the neutrino does not interact with the detectors used in high energy
physics experiments, it can not be reconstructed and thus, leptonically
decaying taus ($\tau_{\ell}$) leave a signature in the detector that is mostly
indistinguishable from an electron or muon, and therefore don’t need dedicated
reconstruction algorithms.

More interesting for the context of this paper are hadronically decaying taus
($\tau_{h}$). In two thirds of the cases the $\tau$ lepton decays into a
neutrino, an odd number of charged (typically pions $\pi^{\pm}$ or kaons
$K^{\pm}$) hadrons ($h^{\pm}$) and a number of neutral pions ($\pi^{0}$). The
latter particles decay almost instantly into a pair of photons. Most dominant
$\tau_{h}$ decay modes feature up to three $h^{\pm}$ and two or fewer
$\pi^{0}$. An overview of the relative branching fractions for the different
$\tau_{h}$ decays is given in Fig. 1.

Figure 1: Overview of the branching fraction for various $\tau$ decay modes.
The numbers are taken from the particle data group [29].

Correctly reconstructing the $\tau_{h}$ and identifying it from other particle
signatures represents a significant combinatorial problem, that eluded a fully
machine learning based solution for many years. Instead, it relied on a
combination of more classical algorithms, such as the hadron-plus-strips (HPS)
[30, 31] for reconstruction, and machine learning-based methods for the
$\tau_{h}$ identification. As a system of multiple final state particles,
$\tau_{h}$ decays are on first glance especially difficult to differentiate
from jets produced by other high energy processes such as the hadronization of
a gluon or a quark. Thus, the problem of identifying a $\tau_{h}$ [32, 30, 33,
34, 35, 36, 37, 38, 31, 39, 40] lies in the realm of ”jet-tagging” which in
recent years has been thoroughly explored with machine learning techniques,
usually focusing on the differentiation of heavy quarks, such as b– and
c–quarks, from lighter (u, d, s) quarks and gluons with examples given in
Refs. [41, 42, 43].

The substantial advances in jet-tagging have been made by exploiting deep
learning techniques from other fields, such as the use of transformers,
originally developed for language modeling. As seen previously in [44], these
techniques can also be used to effectively identify $\tau_{h}$, with the two
tested architectures, LorentzNet [45] and ParticleTransformer [46],
outperforming more typical, heavily optimized approaches such as HPS \+
DeepTau [38] without any fine-tuning done by domain experts. Here, we show
that such models can also be used for reconstructing the properties of the
$\tau_{h}$.

In addition to the correct identification of the $\tau_{h}$ candidates from
jets, existing $\tau_{h}$ reconstruction chains found in experiments such as
Compact Muon Solenoid (CMS) and A Toroidal LHC Apparatus (ATLAS) aim to
determine the exact decay mode of the $\tau_{h}$ decay and to precisely
reconstruct the $\tau_{h}$ momentum.

Only the $\tau_{h}$ decay modes with the largest branching ratio (BR) are
targeted by the state-of-the-art (SOTA) algorithms. These decay modes include
$h^{\pm}$; $h^{\pm}+\pi^{0}$; $h^{\pm}+\geq 2\pi^{0}$; $h^{\pm}h^{\mp}h^{\pm}$
and $h^{\pm}h^{\mp}h^{\pm}+\geq\pi^{0}$, with the rest of the $\tau_{h}$
decays being usually classified into the “Rare” or “Other” category. Similarly
to electron-positron ($e^{+}e^{-}$) experiments, where the decay modes are
usually classified by boosted decision trees (BDTs) [47] or a neural network
(NN) [48], the classification in proton-proton (pp) experiments is done either
with a combinatorial [49, 31] approach or using BDTs on top of reconstructed
$\tau_{h}$ candidates [50]. The classification precision for $h^{\pm}$ and
$h^{\pm}h^{\mp}h^{\pm}$ is $\geq 90\%$ for both pp as well as $e^{+}e^{-}$
collider experiments. However, as reconstructing multiple $\pi^{0}$s in pp
collider experiments is a more challenging task, with the precision for such
decay modes featuring $\pi^{0}$s being $<60\%$ [31, 49, 50, 51] The
corresponding precision for $e^{+}e^{-}$ collider experiments is $\geq 85\%$
[48, 47, 52].

In pp experiments, the $\tau_{h}$ kinematic reconstruction is done both in a
combinatorial approach [31, 34] and a mixture of combinatorics and boosted
regression trees (BRTs) [50, 34, 53, 35]. The performance of such approaches
is comparable, achieving an average energy resolution of $\sim 14\%$ [31, 34]
for the energies ranging from 30 GeV to 300 GeV, with the resolution degrading
at higher energies for up to 20% for higher energies. The energy resolution is
better in the core region of the jet, being in the order of 5—7%, while the
tail resolution is 18—30% [50, 34, 53, 35]. Despite the $\tau_{h}$ kinematic
reconstruction being done for energy, then, similarly to jet reconstruction,
we are regressing the $p_{T}$ of the $\tau_{h}$.

In addition to the above mentioned tasks of $\tau_{h}$ identification,
kinematic regression and its decay mode reconstruction, ML methods have also
been studied for various other scenarios in $\tau$ physics. These include, for
example, mass reconstruction of heavy gauge boson decaying into $\tau$ leptons
[54] and boosted di-$\tau$ system identification [55].

In this work, we aim to expand the applications of deep learning based jet
tagging algorithms to the problems of $\tau_{h}$ energy regression and decay
mode classification, with the ultimate goal of providing a recipe for a
complete $\tau_{h}$ reconstruction in an end-to-end approach with the detector
of choice. To further facilitate the development and studies of new $\tau_{h}$
reconstruction algorithms, we also provide a well documented version of our
$\mathtt{Fu}\tau\mathtt{ure}$ dataset.

The dataset and an overview of its features are given in Sec. 2 with further
details along with the dataset itself in Ref. [56]. We formulate $\tau_{h}$
reconstruction as a set of machine learning tasks in Sec. 3 and demonstrate
how three different types of models with a varying degree of expressiveness
and priors can be employed for the tasks. We study the performance of the
models on the tasks in Sec. 4 and give a short summary and outlook to future
work in Sec. 5.

##  2 The $\mathtt{Fu}\tau\mathtt{ure}$ dataset

With this paper, we provide the first version of the
$\mathtt{Fu}\tau\mathtt{ure}$ dataset. This dataset includes $\sqrt{s}=380$
GeV $e^{+}/e^{-}$ samples with
$Z/\gamma^{*}\rightarrow\tau\tau,\,ZH\rightarrow Z\tau\tau$ and
$Z/\gamma^{*}\rightarrow qq$ processes with approximately 2 million events in
each category. For the simulation and reconstruction we used the Compact
Linear Collider (CLIC) detector [57] setup. The dataset is updated with
respect to [44] with higher statistics, and is now released with documentation
for the first time. The generation of the events is described in Sec. 2.1 with
an overview and description of the features available in Sec. 2.2. The public
dataset along with more technical information can be found in Ref. [56]. For
the studies in this paper, only $\tau_{h}$ jets in the
$Z/\gamma^{*}\rightarrow\tau\tau$ and $ZH\rightarrow Z\tau\tau$ datasets were
used.

###  2.1 Monte Carlo samples and event reconstruction

The $\mathtt{Fu}\tau\mathtt{ure}$ dataset is generated using Pythia8 [58] with
the same generator settings as in Ref. [44, 59]. After generation the events
undergo a full detector simulation using Geant4 [60] with the CLIC like
detector (CLICdet) (CLIC_o3_v14) [61] setup before being reconstructed using
the Marlin reconstruction code [62] and interfaced using the Key4HEP package
[63]. The features in our dataset are then extracted using the particle
candidates found by processing the reconstructed events with the PandoraPF
algorithm [64, 65, 66].

The CLICdet design has been thoroughly studied over the recent years, and its
design is similar to the CLIC Like Detector (CLD), foreseen for the upcoming
FCC-ee detector providing a relevant benchmark scenario for physics at future
$e^{+}/e^{-}$ experiments. Similarly to other contemporary detectors at hadron
colliders, the CLICdet detector features a layered design with a high
precision tracking system featuring a silicon pixel and tracking detector, a
Si-W electromagnetic sampling calorimeter and and a steel hadronic sampling
calorimeter encased in a 4T solenoid and a dedicated muon system. The expected
physics performance with some preliminary studies is discussed in Ref. [67].

A potentially important background for the $\tau_{h}$ reconstruction from the
overlay of $\gamma\gamma\rightarrow\,$hadrons is currently not included in our
simulation. The study and inclusion of this background is left for future
iterations of this dataset.

###  2.2 Input features and validation

The basis of our dataset are particle flow candidates from PandoraPF with four
momenta, charge, and candidate labels for electrons ($e$), muons ($\mu$),
photons ($\gamma$), charged hadrons ($h^{\pm}$), and neutral hadrons
($h^{0}$). These candidates are clustered into jets using the generalized
$k_{T}$ algorithm for $e^{+}/e^{-}$ collisions (ee_genkt) [68] with parameters
of $p=-1$, $R=0.4$, and a minimum $p_{\mathrm{T}}\geq 5~{}\mathrm{GeV}$ to
serve as the seeds for tau identification. The dataset contains the four
momenta of these reconstructed jets, and the four momenta, charge and the
particle label of the PandoraPF candidates within them.

As $\tau$-leptons have a small but finite lifetime corresponding to a travel
distance of a few mm in the detector, variables sensitive to this special
topology such as transverse ($d_{xy}$) and longitudinal ($d_{z}$) impact
parameters of the tracks of charged particles have long been used for
$\tau_{h}$ identification. Such impact parameters are currently not part of
the Key4HEP format and are thus calculated by us. In the context of this work,
we use the same linear approximation in this calculation as in Ref. [44], with
more details to be found in Ref. [69].

The ground truth is based on stable particles at the generator level, before
detector simulation. These particles are clustered into generator-level jets
with the same algorithm as used for the clustering of the the reconstructed
jets, but without a cut on $p_{\mathrm{T}}$. The generator-level jets are
matched to generator-level $\tau$ leptons as well as reconstructed jets using
$\Delta R<0.3$. For each reconstructed jet, we can then define up to three
target values related to $\tau$ lepton reconstruction:

  * •

a binary flag isTau if it was matched to a generator-level hadronically
decaying $\tau$ lepton ($\tau_{h}$),

  * •

if matched, the categorical decay mode of the $\tau_{h}$ in terms of the
number of charged and neutral hadrons
$\mathrm{DM}^{\mathrm{true}}\in\\{0,1,\dots,15\\}$,

  * •

if matched, the visible (neutrinoless) and reconstructable transverse momentum
$p_{\mathrm{T}}^{\mathrm{vis,true}}$ of the $\tau_{h}$ lepton.

Thus, the dataset consists of reconstructed jets, with a variable number of
reconstructed particles per jet, and with up to three target labels for each
reconstructed jet. While the models we subsequently study are invariant to
particle ordering within the jet, we sort the particles in each jet in
$p_{\mathrm{T}}$ descending order in the interest of reusability.

The dataset has been extensively investigated and tested for consistency and
can be validated with the provided software found in Ref. [70] and Ref. [56].
As an example, Fig. 2 and Fig. 3 show illustrate the two research problems at
hand: the first figure shows the jet substructure for two different $\tau_{h}$
decay modes in the $ZH$ sample — decays into a single charged hadron and three
charged hadrons. The second figure shows the truth-level energy of the visible
$\tau_{h}$ decay products we regress from the jets.

Figure 2: Average $p_{\mathrm{T}}$ of particle flow candidates per bin in the
$\Delta\eta-\Delta\phi$ plane around the jet axis, aggregated for jets in the
ZH dataset, matched to a single prong and three-prong, more than one neutral
pion $\tau_{h}$ decay modes. Figure 3: Distribution of visible $\tau$
component of the $\tau$ transverse momentum of the generator/truth-level for
jets in the $Z/\gamma^{*}\rightarrow\tau\tau$ and $ZH\rightarrow Z\tau\tau$
datasets.

##  3 Tau reconstruction with ML

Reconstructing hadronically decaying tau leptons using ML can be defined as a
multi-task machine learning problem:

$\Phi(\mathrm{jet\ features},\mathrm{particle\
features})\rightarrow\\{\texttt{isTau},\mathrm{DM}^{\mathrm{true}},p_{\mathrm{T}}^{\mathrm{vis,true}}\\}\,\,,$

where $\Phi$ is a trainable model. Note that $\Phi$ may consist of a single
model, separate models, or even a single backbone model with fine-tuned output
layers for each task.

We addressed isTau classification in Ref. [44], finding that in this dataset,
a transformer-based approach performs well compared to the alternatives for
$\tau_{h}$ identification.

In this paper, we address reconstructing the hadronic $\tau$ decay modes
(DMs), as given in Fig. 1, from a $\tau_{h}$ candidate jet and regressing the
visible momentum of the $\tau_{h}$ lepton
$p_{\mathrm{T}}^{\mathrm{vis,true}}$.

We use the ParticleTransformer [71] and LorentzNet [72] architectures as the
main models due to their expressiveness on jet tasks. As a cross-check, we
also test a simpler algorithm based on deep sets [73]. While expressive,
transformer-based models can be resource-intensive and challenging to run in
real time on constrained hardware such as field programmable gate arrays
(FPGAs), or to introduce physics-informed priors. The LorentzNet architecture
allows to establish how much of the performance is based purely on kinematic
information, while using a strong inductive bias based on Lorentz symmetry.
The DeepSet-based architecture serves as a simple cross-check model, which
tests how much the previous, more expressive models add on top of a very
simple baseline. Moreover, DeepSet-type models are straightforward to deploy
on constrained hardware such as FPGAs, and may represent a practical trade-off
where accuracy needs to be balanced with inference throughput and latency [74,
75].

We thus use the $\mathtt{Fu}\tau\mathtt{ure}$ dataset to compare three model
architectures on two different tasks. We use the same set of input features
for all models and tasks, consisting of the particle kinematics
$p_{x},p_{y},p_{z},E$ and the additional features:

  * •

particle charge $q\in\\{-1,0,+1\\}$,

  * •

particle labels isChargedHadron, isNeutralHadron, isPhoton, isElectron, isMuon
based on particle flow reconstruction,

  * •

$\log{p_{\mathrm{T}}}$, $\log{E}$ of the particles,

  * •

$\Delta\eta$, $\Delta\phi$ of the particle with respect to the jet,

  * •

relative $\log{p_{\mathrm{T}}/p_{\mathrm{T}}^{\mathrm{jet}}}$,
$\log{E/E_{jet}}$ with respect to the jet.

Currently no $\tau$ lifetime information was used for the momentum and decay
mode reconstruction as their impact on the performance in the current setting
will be rather modest. However, the inclusion of these variables will more
important in future studies with the inclusion of the
$\gamma\gamma\rightarrow$ hadrons overlay background, that similarly to
“pileup” from simultaneously occurring collisions at hadron colliders, could
contaminate the jets with additional low energy particles spoiling the
momentum reconstruction.

We pick the first 16 particles in $p_{\mathrm{T}}$-descending order from each
jet for the subsequent studies for efficient transformer training, which
introduces a negligible fraction of data loss. The models we study are
invariant to particle ordering in the jet.

For the training we use the $Z/\gamma^{*}\rightarrow\tau\tau$ sample, while
the final result is evaluated on $ZH\rightarrow Z\tau\tau$, never used in
training, to ensure the models are able to generalize across datasets.

Similarly to Ref. [76] we use
$\log{[p_{\mathrm{T}}^{\mathrm{vis,true}}/p_{\mathrm{T}}^{\mathrm{jet}}]}$ as
the target for momentum reconstruction, since the logarithm of the ratio of
the total visible transverse momentum of the $\tau$ components compared to the
jet $p_{\mathrm{T}}$ is distributed approximately normally. We use the Huber
loss [77] for the regression task, as it is less sensitive to outliers than
the mean squared error. For the decay mode classification task we one-hot
encode $\mathrm{DM}^{\mathrm{true}}$ and use the standard cross-entropy loss
for the multi-classification task.

The trainings are performed over a maximum of 100 epochs with a batch size of
1024 using the AdamW [78] optimizer. We cross-validate the training by redoing
the it 3 times on different subsets of the training dataset and using
different neural network initializations. Each training runs for approximately
8 hours on a single 8GB Nvidia RTX 2070S GPU. No hypertuning is performed at
this stage, as our goal is not to find the most optimal configuration for this
specific dataset, but rather to demonstrate generically that such ML
algorithms can be used for end-to-end $\tau_{h}$ reconstruction.

The top part of Fig. 4 shows the loss curves for all three algorithms for both
tasks. As can be seen, more expressive ParticleTransformer and LorentzNet
architectures converge both faster than the simpler DeepSet algorithm and also
achieve overall lower ultimate validation losses as shown in the bottom part
of the same figure.

Figure 4:  Training and validation loss curves for the DeepSet, LorentzNet and
ParticleTransformer algorithms in the $\tau_{h}$ momentum regression task
(Top-Left) and the $\tau_{h}$ decay mode classification task (Top-Right).
Bottom: Comparison of the best achieved validation losses for the three
different algorithms in both the $\tau_{h}$ momentum regression and $\tau_{h}$
decay mode classification task.

##  4 Results

To quantify the performance of the three algorithms in the $\tau_{h}$ momentum
regression it is useful to look both at the scale and resolution of the
resulting $\tau_{h}$-momentum distribution. The former is given by the median
of the ratio of the predicted and the true visible $\tau_{h}$-momentum
$p_{\mathrm{T}}^{\mathrm{vis,pred}}/p_{\mathrm{T}}^{\mathrm{vis,true}}$. A
median of one means that on average momentum is predicted correctly, while
values above (below) one mean that the momentum on average is
over(under)-predicted. The resolution is given by the width of the
$p_{\mathrm{T}}^{\mathrm{vis,pred}}/p_{\mathrm{T}}^{\mathrm{vis,true}}$
distribution. For our results we use the interquartile range (IQR) instead of
the standard deviation as a measure for the width as it is less sensitive
towards outliers. The IQR is given by the difference in the position of the
25% and 75% quantile of the distribution, normalized by the 50% quantile,
giving the width of the central part of the distribution relative to its
median. Fig. 5 shows both the scale as well as the resolution of the regressed
$p_{\mathrm{T}}$ distribution as a function of the truth level $\tau_{h}$
$p_{\mathrm{T}}$ for all three algorithms.

Figure 5: Scale (Left) and resolution (Right) of the $\tau_{h}$ momentum
response distribution given by the mean and the IQR of the
$p_{\mathrm{T}}^{\mathrm{vis,pred}}/p_{\mathrm{T}}^{\mathrm{vis,true}}$
distribution. Both scale and response are displayed as a function of the truth
level visible $\tau$ $p_{\mathrm{T}}$ for the $ZH\rightarrow Z\tau\tau$ sample
for all three algorithms. As a comparison, the scale and resolution are also
shown when using the input jet $p_{\mathrm{T}}$ directly as the tau momentum
without applying any regression.

It can be seen that in the bulk of the $\tau_{h}$ $p_{\mathrm{T}}$ between 15
and 100 GeV all three algorithms predict the scale of the $\tau_{h}$ momentum
very well, within 0.5%. This compares to an over-prediction of about 1.5% when
using the input jet directly as a $\tau_{h}$ without any further regression.
Similarly, all three algorithms achieve a resolution of 2.5% to 3% in the same
$\tau_{h}$ $p_{\mathrm{T}}$ range with the ParticleTransformer giving the best
results. This compares to a resolution of about 5% when using the input jet
directly as a $\tau_{h}$. At high $\tau$ $p_{\mathrm{T}}$ where the number of
jets in the $Z\rightarrow\tau\tau$ sample used for training drops, the
performance of all three algorithms, but in particular the ParticleTransformer
and LorentzNet, is reduced, under-predicting the $\tau_{h}$ momentum by up to
a few percent while the resolution worsens.

Quantifying the quality of the $\tau_{h}$ decay mode classification is a more
difficult task as the fraction of taus for the different decay modes varies,
with the most likely decay into one charged hadron and one neutral pion at
about 26% of all tau decays, and decays into three charged hadrons and one
neutral pion at only about 5%. On the other hand, depending on the underlying
physics analysis, the identification of the right number of charged hadrons,
and subsequently the correct charge of the $\tau_{h}$, might be physically
more interesting than the differentiation of decay modes with zero, one or two
neutral hadrons. Hence the overall precision of the decay mode classification
might not present the most accurate picture. Instead, in Fig. 6 we show the
precision broken down by decay mode for all three algorithms and the confusion
matrix of true/predicted decay modes for the ParticleTransformer algorithm in
Fig. 7. The latter does not only break down the precision of the algorithm as
a function of the decay mode, but also helps us judge the migration of
misidentified decay modes to other classes. We have chosen the
ParticleTransformer algorithm as it shows the best overall loss as can be seen
in Fig. 4.

Figure 6: The precision of $\tau_{h}$ decay mode reconstruction in channels
with a varying number of charged and neutral particles in the $ZH\rightarrow
Z\tau\tau$ dataset. The “Overall” category represents the general performance
of a model across all decay modes weighted by the branching ratio of the given
decay mode. Figure 7: Tau decay mode reconstruction confusion matrix for
ParticleTransformer on the $ZH\rightarrow Z\tau\tau$ dataset.

Overall a precision of 80% to 95% is achieved for decay mode classification,
with the most difficult decay modes being the ones with a multitude of neutral
pions as well as rare decays. In general, the ParticleTransformer algorithm
performs the best for these difficult cases with a high number of final state
particles. Notably, the performance of the ParticleTransformer algorithm is
significantly worse for the overflow category (“Rare”). We hypothesize that
the ParticleTransformer approach is affected more significantly than the other
networks by the limited training statistics. Nevertheless, given the broadly
comparable overall precision of the models on the decay mode reconstruction
task, the choice in a particular experiment should be driven by the available
computational budget.

##  5 Summary and Outlook

In this paper we define $\tau_{h}$ reconstruction as a multi-task machine
learning problem consisting of binary classification for $\tau_{h}$
identification, momentum regression, and decay mode multi-classification,
create a realistic benchmark dataset for this task and compare the performance
of several architectures on the reconstruction subtasks. The
$\mathtt{Fu}\tau\mathtt{ure}$ dataset is available in Ref. [56] and contains
samples to test machine learning algorithms for $\tau_{h}$ reconstruction.
This paper builds on previous work [44] showing how transformer based
architectures can be used to reliably reconstruct and identify hadronic $\tau$
decays. Here, we demonstrate that these architectures also perform well in
regressing the $\tau_{h}$ momentum as well as identifying the $\tau_{h}$ decay
mode, thus allowing to treat $\tau_{h}$ lepton identification and
reconstruction as a single machine learning problem. For the momentum
regression a resolution of about 3% with a momentum scale about 0.5%-1% within
the true tau momentum could be achieved for the bulk of the $\tau$ momentum
distribution. Depending on the specific decay mode a precision between 80% and
95% could be achieved for the the classification task, with
ParticleTransformer outperforming other architectures for the more difficult
decay mode with a higher number of final state particles.

Thus far, we have trained separate models for each subtask from scratch, while
recent work in the direction of foundation models has shown promise that using
pretrained backbone models with task-specific fine-tuning can reduce the
amount of required training samples, as well as the required inference budget
[79, 80]. In future work, it may be useful to investigate the dependence of
the architectures on available training statistics and sample composition to
ensure robustness under domain shift scenarios. This dataset also allows to
study the trade-offs between bias and variance in terms of using physics-
inspired networks such as LorentzNet on a limited set of input features, vs.
using a wide variety of input features in more generic architectures such as
ParticleTransformer, to identify the relative feature importances for the
subtasks. The dataset and training and validation setup can be generalized in
a straightforward way to FCC-ee and other future collider scenarios.

## Acknowledgments

This work has been supported by the Estonian Research Council grants PSG864,
RVTT3-KBFI and TK202.

## Data availability

The dataset used in this paper is made available in Ref. [56]. The software
used to produce the results can be found in Ref. [70].

## Author contributions (CRediT)

Laurits Tani: Conceptualization, Methodology, Software, Validation, Formal
analysis, Investigation, Data Curation, Writing - Original Draft, Writing -
Review & Editing, Visualization. Nalong-Norman Seeba: Software, Validation,
Formal analysis, Investigation, Visualization. Joosep Pata: Conceptualization,
Methodology, Software, Validation, Formal analysis, Investigation, Data
Curation, Writing - Original Draft, Writing - Review & Editing, Visualization,
Supervision, Project administration, Funding administration. Torben Lange:
Conceptualization, Methodology, Formal analysis, Writing - Original Draft.
Hardi Vanaveski: Software, Validation, Formal analysis, Investigation,
Visualization.

## References

  * [1] ATLAS Collaboration  “Searches for third-generation scalar lepto-quarks in $\sqrt{s}=13$ TeV pp collisions with the ATLAS detector”  In _JHEP_ 06, 2019, pp. 144  DOI: 10.1007/JHEP06(2019)144
  * [2] ATLAS Collaboration  “Search for resonant and non-resonant Higgs boson pair production in the $bb\tau^{+}\tau^{-}$ decay channel using $13$ TeV pp collision data from the ATLAS detector”  In _JHEP_ 07, 2023, pp. 040  DOI: 10.1007/JHEP07(2023)040
  * [3] CMS Collaboration  “Search for non-resonant Higgs boson pair production in final state with two bottom quarks and two tau leptons in proton-proton collisions at $\sqrt{s}=13$ TeV”  In _Phys. Lett. B_ 842, 2023, pp. 137531  DOI: 10.1016/j.physletb.2022.137531
  * [4] ATLAS Collaboration  “Measurement of the CP properties of Higgs boson interactions with $\tau$-leptons with the ATLAS detector”  In _Eur. Phys. J. C_ 83.7, 2023, pp. 563  DOI: 10.1140/epjc/s10052-023-11583-y
  * [5] Armen Tumasyan  “Analysis of the $CP$ structure of the Yukawa coupling between the Higgs boson and $\tau$ leptons in proton-proton collisions at $\sqrt{s}$ = 13 TeV”  In _JHEP_ 06, 2022, pp. 012  DOI: 10.1007/JHEP06(2022)012
  * [6] Mogens Dam  “The $\tau$ challenges at FCC-ee”  In _Eur. Phys. J. Plus_ 136.9, 2021, pp. 963  DOI: 10.1140/epjp/s13360-021-01894-y
  * [7] Antonio Pich  “Challenges for tau physics at the TeraZ”  In _Eur. Phys. J. Plus_ 136.11, 2021, pp. 1117  DOI: 10.1140/epjp/s13360-021-02077-5
  * [8] Mogens Dam  “Tau-lepton physics at the FCC-ee circular $e^{+}e^{-}$ collider”  In _SciPost Phys. Proc._ 1, 2019, pp. 041  DOI: 10.21468/SciPostPhysProc.1.041
  * [9] ATLAS Collaboration  “A search for lepton-flavor-violating decays of the Z boson into a $\tau$ lepton and a light lepton with the ATLAS detector”  In _Phys. Rev. D_ 98, 2018, pp. 092010  DOI: 10.1103/PhysRevD.98.092010
  * [10] Roni Harnik, Joachim Kopp and Jure Zupan  “Flavor violating Higgs decays”  In _JHEP_ 03, 2013, pp. 026  DOI: 10.1007/JHEP03(2013)026
  * [11] CMS Collaboration  “Search for lepton flavour violating decays of the Higgs boson to $\mu\tau$ and $e\tau$ in proton-proton collisions at $\sqrt{s}=13$ TeV”  In _JHEP_ 06, 2018, pp. 001  DOI: 10.1007/JHEP06(2018)001
  * [12] ATLAS Collaboration  “Search for additional heavy neutral Higgs and gauge bosons in the ditau final state produced in 36 fb-1 of pp collisions at $\sqrt{s}=13$ TeV with the ATLAS detector”  In _JHEP_ 01, 2018, pp. 055  DOI: 10.1007/JHEP01(2018)055
  * [13] ATLAS Collaboration  “Search for high-mass resonances decaying to $e^{+}\nu$ in pp collisions at $\sqrt{s}=13$ TeV with the ATLAS detector”  In _Phys. Rev. Lett._ 120, 2018, pp. 161802  DOI: 10.1103/PhysRevLett.120.161802
  * [14] CMS Collaboration  “Search for new physics in the $\tau$ lepton plus missing transverse momentum final state in proton-proton collisions at $\sqrt{s}=13$ TeV”  In _JHEP_ 09, 2023, pp. 051  DOI: 10.1007/JHEP09(2023)051
  * [15] ATLAS Collaboration  “Search for the direct production of charginos and neutralinos in final states with tau leptons in $\sqrt{s}=13$ TeV pp collisions with the ATLAS detector”  In _Eur. Phys. J. C_ 78, 2018, pp. 154  DOI: 10.1140/epjc/s10052-018-5583-9
  * [16] ATLAS Collaboration  “Search for top squarks decaying to tau sleptons in pp collisions at $\sqrt{s}=13$ TeV with the ATLAS detector”  In _Phys. Rev. D_ 98, 2018, pp. 032008  DOI: 10.1103/PhysRevD.98.032008
  * [17] ATLAS Collaboration  “Search for squarks and gluinos in final states with hadronically decaying $\tau$ leptons, jets, and missing transverse momentum using pp collisions at $\sqrt{s}=13$ TeV with the ATLAS detector”  In _Phys. Rev. D_ 99, 2019, pp. 012009  DOI: 10.1103/PhysRevD.99.012009
  * [18] CMS Collaboration  “Search for direct pair production of supersymmetric partners to the $\tau$ lepton in proton-proton collisions at $\sqrt{s}=13$ TeV”  In _Eur. Phys. J. C_ 80, 2020, pp. 189  DOI: 10.1140/epjc/s10052-020-7739-7
  * [19] CMS Collaboration  “Search for Supersymmetry with a compressed mass spectrum in events with a soft $\tau$ lepton, a highly energetic jet, and large missing transverse momentum in proton-proton collisions at $\sqrt{s}=13$ TeV”  In _Phys. Rev. Lett._ 124, 2020, pp. 041803  DOI: 10.1103/PhysRevLett.124.041803
  * [20] ATLAS Collaboration  “Search for direct stau production in events with two hadronic $\tau$ leptons in $\sqrt{s}=13$ TeV pp collisions with the ATLAS detector”  In _Phys. Rev. D_ 101, 2020, pp. 032009  DOI: 10.1103/PhysRevD.101.032009
  * [21] ATLAS Collaboration  “Search for bottom squark pair production in pp collision events at $\sqrt{s}=13$ TeV with hadronically decaying $\tau$ leptons, b jets and missing transverse momentum using the ATLAS detector”  In _Phys. Rev. D_ 104, 2021, pp. 032014  DOI: 10.1103/PhysRevD.104.032014
  * [22] CMS Collaboration  “Search for electroweak production of charginos and neutralinos in proton-proton collisions at $\sqrt{s}=13$ TeV”  In _JHEP_ 04, 2022, pp. 147  DOI: 10.1007/JHEP04(2022)147
  * [23] CMS Collaboration  “Search for top squark pair production in a final state with at least one hadronically decaying tau lepton in proton-proton collisions at $\sqrt{s}=13$ TeV”  In _JHEP_ 07, 2023, pp. 110  DOI: 10.1007/JHEP07(2023)110
  * [24] ATLAS Collaboration  “Search for charged Higgs bosons decaying via $H^{\pm}\to\tau^{-}\nu_{\tau}$ in the $\tau$+jets and $\tau$+lepton final states with $36$ fb-1 of pp collision data recorded at $\sqrt{s}=13$ TeV with the ATLAS experiment”  In _JHEP_ 09, 2018, pp. 139  DOI: 10.1007/JHEP09(2018)139
  * [25] CMS Collaboration  “Search for charged Higgs bosons in the $H^{\pm}\to\tau^{-}\nu_{\tau}$ decay channel in proton-proton collisions at $\sqrt{s}=13$ TeV”  In _JHEP_ 07, 2019, pp. 142  DOI: 10.1007/JHEP07(2019)142
  * [26] ATLAS Collaboration  “Search for heavy Higgs bosons decaying into two tau leptons with the ATLAS detector using pp collisions at $\sqrt{s}=13$ TeV”  In _Phys. Rev. Lett._ 125, 2020, pp. 051801  DOI: 10.1103/PhysRevLett.125.051801
  * [27] CMS Collaboration  “Searches for additional Higgs bosons and for vector lepto-quarks in $\tau\tau$ final states in proton-proton collisions at $\sqrt{s}=13$ TeV”  In _JHEP_ 07, 2023, pp. 073  DOI: 10.1007/JHEP07(2023)073
  * [28] CMS Collaboration  “Search for a heavy Higgs boson decaying into two lighter Higgs bosons in the $\tau\tau bb$ final state at $13$ TeV”  In _JHEP_ 11, 2021, pp. 057  DOI: 10.1007/JHEP11(2021)057
  * [29] R.. Workman  “Review of Particle Physics”  In _PTEP_ 2022, 2022, pp. 083C01  DOI: 10.1093/ptep/ptac097
  * [30] CMS collaboration  “Performance of $\tau$-lepton reconstruction and identification in CMS”  In _Journal of Instrumentation_ 7.01 IOP Publishing, 2012, pp. P01001
  * [31] Vardan Khachatryan  “Reconstruction and identification of $\tau$ lepton decays to hadrons and $\nu$τ at CMS”  In _JINST_ 11.01, 2016, pp. P01019  DOI: 10.1088/1748-0221/11/01/P01019
  * [32] Lucas Russell  “Identification of Hadronic Tau Lepton Decays with Domain Adaptation using Adversarial Machine Learning at CMS” Presented 28 Jun 2022, 2022  URL: https://cds.cern.ch/record/2827366
  * [33] ATLAS Collaboration  “Reconstruction, Identification, and Calibration of hadronically decaying tau leptons with the ATLAS detector for the LHC Run 3 and reprocessed Run 2 data”  In _CERN Document Server_ , 2022
  * [34] Camilla Galloni  “Hadronic tau reconstruction and identification performance in ATLAS and CMS”  In _PoS_ LHCP2018, 2018, pp. 228  DOI: 10.22323/1.321.0228
  * [35] ATLAS Collaboration  “Measurement of the tau lepton reconstruction and identification performance in the ATLAS experiment using $pp$ collisions at $\sqrt{s}=13$ TeV”  In _CERN Document Server_ , 2017
  * [36] ATLAS Collaboration  “Identification of hadronic tau lepton decays using neural networks in the ATLAS experiment”  In _CERN Document Server_ , 2019
  * [37] Izaak Neutelings  “Hadronic tau reconstruction and identification performance in ATLAS and CMS”  In _PoS_ LHCP2020, 2021, pp. 045  DOI: 10.22323/1.382.0045
  * [38] CMS Collaboration  “Identification of hadronic tau lepton decays using a deep neural network”  In _JINST_ 17, 2022, pp. P07023  DOI: 10.1088/1748-0221/17/07/P07023
  * [39] Andris Huang et al.  “Heterogeneous Graph Neural Network for identifying hadronically decayed tau leptons at the High Luminosity LHC”  In _JINST_ 18.07, 2023, pp. P07001  DOI: 10.1088/1748-0221/18/07/P07001
  * [40] Nico Madysa  “Identification of Hadronic Tau Lepton Decays at the ATLAS Detector Using Artificial Neural Networks”, 2015
  * [41] Spandan Mondal and Luca Mastrolorenzo  “Machine Learning in High Energy Physics: A review of heavy-flavor jet tagging at the LHC”  In _arXiv_ , 2024  arXiv:2404.01071 [hep-ex]
  * [42] Frédéric A. Dreyer and Huilin Qu  “Jet tagging in the Lund plane with graph networks”  In _JHEP_ 03, 2021, pp. 052  DOI: 10.1007/JHEP03(2021)052
  * [43] Taoli Cheng  “Recursive Neural Networks in Quark/Gluon Tagging”  In _Comput. Softw. Big Sci._ 2.1, 2018, pp. 3  DOI: 10.1007/s41781-018-0007-y
  * [44] Torben Lange et al.  “Tau lepton identification and reconstruction: A new frontier for jet-tagging ML algorithms”  In _Comput. Phys. Commun._ 298, 2024, pp. 109095  DOI: 10.1016/j.cpc.2024.109095
  * [45] Shiqi Gong et al.  “An efficient Lorentz equivariant graph neural network for jet tagging”  In _JHEP_ 07, 2022, pp. 030  DOI: 10.1007/JHEP07(2022)030
  * [46] Huilin Qu, Congqiao Li and Sitian Qian  “Particle Transformer for jet tagging”  In _International conference on machine learning_ , 2022, pp. 18281  PMLR  arXiv:2202.03772 [hep-ph]
  * [47] Boruo Xu  “Detectors and physics at a future linear collider”, 2017
  * [48] Stefano Giagu, Luca Torresi and Matteo Di Filippo  “Tau lepton identification with graph neural networks at future electron–positron colliders”  In _Frontiers in Physics_ 10 Frontiers, 2022, pp. 909205
  * [49] Ze Chen  “Tau reconstruction in CMS exploiting machine learning techniques”, 2022
  * [50] ATLAS Collaboration  “Reconstruction of hadronic decay products of tau leptons with the ATLAS experiment”  In _Eur. Phys. J. C_ 76.5, 2016, pp. 295  DOI: 10.1140/epjc/s10052-016-4110-0
  * [51] Torrey Adam Saxton  “Computer Vision and Application to the Classification of the Hadronic Decay Modes of the Tau Lepton with the ATLAS detector”, 2018  URL: https://cds.cern.ch/record/2634314
  * [52] Trong Hieu Tran et al.  “Reconstruction and classification of tau lepton decays with ILD”  In _Eur. Phys. J. C_ 76.8, 2016, pp. 468  DOI: 10.1140/epjc/s10052-016-4315-2
  * [53] Michael Hübner  “Measurement of the tau lepton reconstruction and identification performance in the ATLAS experiment using pp collisions at $\sqrt{s}=13$ TeV”, 2018
  * [54] M.. Krishnan, Aruna Kumar Nayak and Asrith Krishna Radhakrishnan  “Invariant mass reconstruction of heavy gauge bosons decaying to $\tau$ leptons using machine learning techniques”  In _Eur. Phys. J. C_ 84.3, 2024, pp. 219  DOI: 10.1140/epjc/s10052-024-12527-w
  * [55] Nadav Tamir et al.  “Neural networks for boosted di-$\tau$ identification”  In _JINST_ 19.07, 2024, pp. P07004  DOI: 10.1088/1748-0221/19/07/P07004
  * [56] Laurits Tani et al.  “Fu$\tau$ure - dataset for studies, development, and training of algorithms for reconstructing and identifying hadronically decaying tau leptons”  URL: https://doi.org/10.5281/zenodo.12664634
  * [57] Lucie Linssen, Akiya Miyamoto, Marcel Stanitzki and Harry Weerts  “Physics and detectors at CLIC: CLIC conceptual design report”, 2012  DOI: 10.5170/CERN-2012-003
  * [58] Christian Bierlich et al.  “A comprehensive guide to the physics and usage of PYTHIA 8.3”  In _SciPost Phys. Codebases_ SciPost, 2022, pp. 8  DOI: 10.21468/SciPostPhysCodeb.8
  * [59] Yasmine Amhis et al.  “Prospects for $\textrm{B}_{\textrm{c}}^{+}\to\tau^{+}\nu_{\tau}$ at FCC-ee”  In _JHEP_ 12, 2021, pp. 133  DOI: 10.1007/JHEP12(2021)133
  * [60] GEANT4 Collaboration  “Geant4: A simulation toolkit”  In _Nucl. Instrum. Meth. A_ 506, 2003, pp. 250  DOI: 10.1016/S0168-9002(03)01368-8
  * [61] CLIC Collaboration  “CLICdet: The post-CDR CLIC detector model”, 2017  URL: https://cds.cern.ch/record/2254048
  * [62] F. Gaede  “Marlin and LCCD: Software tools for the ILC”  In _Nucl. Instrum. Meth. A_ 559, 2006, pp. 177  DOI: 10.1016/j.nima.2005.11.138
  * [63] Gerardo Ganis, Clément Helsens and Valentin Völkl  “Key4hep, a framework for future HEP experiments and its use in FCC”  In _Eur. Phys. J. Plus_ 137, 2022, pp. 149  DOI: 10.1140/epjp/s13360-021-02213-1
  * [64] J.. Marshall and M.. Thomson  “The Pandora software development kit for particle flow calorimetry”  In _J. Phys. Conf. Ser._ 396, 2012, pp. 022034  DOI: 10.1088/1742-6596/396/2/022034
  * [65] J.. Marshall, A. Münnich and M.. Thomson  “Performance of particle flow calorimetry at CLIC”  In _Nucl. Instrum. Meth. A_ 700, 2013, pp. 153  DOI: 10.1016/j.nima.2012.10.038
  * [66] J.. Marshall and M.. Thomson  “The Pandora software development kit for pattern recognition”  In _Eur. Phys. J. C_ 75, 2015, pp. 439  DOI: 10.1140/epjc/s10052-015-3659-3
  * [67] H. Abramowicz  “Higgs physics at the CLIC electron–positron linear collider”  In _Eur. Phys. J. C_ 77, 2017, pp. 475  DOI: 10.1140/epjc/s10052-017-4968-5
  * [68] M. Boronat et al.  “Jet reconstruction at high-energy electron–positron colliders”  In _Eur. Phys. J. C_ 78, 2018, pp. 144  DOI: 10.1140/epjc/s10052-018-5594-6
  * [69] J.. Jackson  “Classical electrodynamics”  John Wiley & Sons, 1999
  * [70] Laurits Tani et al.  “HEP-KBFI/ml-tau-en-reg: arXiv submission (v1.1)”  URL: https://doi.org/10.5281/zenodo.12684862
  * [71] Huilin Qu, Congqiao Li and Sitian Qian  “Particle transformer for jet tagging”  In _International Conference on Machine Learning_ , 2022, pp. 18281–18292  PMLR
  * [72] Shiqi Gong et al.  “An efficient Lorentz equivariant graph neural network for jet tagging”  In _Journal of High Energy Physics_ 2022.7 Springer, 2022, pp. 1–22
  * [73] Manzil Zaheer et al.  “Deep sets”  In _Advances in neural information processing systems_ 30, 2017
  * [74] Maayan Yaary et al.  “Trees versus Neural Networks for enhancing tau lepton real-time selection in proton-proton collisions”, 2023  arXiv:2306.06743 [hep-ex]
  * [75] Jona Motta  “Development and firmware implementation of a Machine Learning based hadronic Tau lepton Level-1 Trigger algorithm in CMS for the HL-LHC”  In _PoS_ EPS-HEP2023, 2024, pp. 590  DOI: 10.22323/1.449.0590
  * [76] Daniel Holmberg  “Jet Energy Corrections with Graph Neural Network Regression”, 2022
  * [77] Peter J. Huber  “Robust Estimation of a Location Parameter”  In _The Annals of Mathematical Statistics_ 35.1, 1964, pp. 73–101  DOI: 10.1214/aoms/1177703732
  * [78] Ilya Loshchilov and Frank Hutter  “Decoupled weight decay regularization”, 2017  arXiv:1711.05101 [cs.LG]
  * [79] Vinicius Mikuni and Benjamin Nachman  “OmniLearn: A Method to Simultaneously Facilitate All Jet Physics Tasks”, 2024  arXiv:2404.16091 [hep-ph]
  * [80] Joschka Birk, Anna Hallin and Gregor Kasieczka  “OmniJet-$\alpha$: The first cross-task foundation model for particle physics”, 2024  arXiv:2403.05618 [hep-ph]


# The Obscure Limitation of Modular
Multilingual Language Models

Muhammad Farid Adilazuarda$\star^{1}$, Samuel Cahyawijaya$\star^{2}$, Ayu
Purwarianti1
1Institut Teknologi Bandung 2 HKUST
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We expose the limitation of modular multilingual language models (MLMs) in
multilingual inference scenarios with unknown languages. Existing evaluations
of modular MLMs exclude the involvement of language identification (LID)
modules, which obscures the performance of real-case multilingual scenarios of
modular MLMs. In this work, we showcase the effect of adding LID on the
multilingual evaluation of modular MLMs and provide discussions for closing
the performance gap of caused by the pipelined approach of LID and modular
MLMs.

Figure 1: Modular MLMs incorporate language-specific adapters to learn new
languages. This renders them language-dependent and reliant on external LID
for inference.

##  1 Introduction

Multilingual language models (MLMs) suffer from the capacity limitation
problem known as the curse of multilinguality, which penalizes the efficiency
of MLMs, both in terms of training and inference, for acquiring new languages.
Prior works (Pfeiffer et al., 2020; Ansell et al., 2021; Pfeiffer et al.,
2022) alleviate the inference inefficiency bottleneck of the curse of
multilinguality by introducing modularity in MLMs through language adapters.
This modularity allows MLMs to scale the number of parameters with minimal
cost on the training and inference speed. One limitation of modular MLMs is
that, as shown in Figure 1, the language of the input needs to be known prior
to the inference step for selecting the language adapter. Nevertheless,
multilingual evaluations of these modular MLMs make an assumption that an
ideal language identification is given and use the language metadata provided
on the evaluation data to select the correct language adapter. This produces a
gap between modular MLMs in the simulated setting and in the real multilingual
scenario. In this work, we address the evaluation gap and further discuss how
to mitigate the limitation of modular MLMs.

##  2 Related Works

#### Multilingual Language Model

MLMs (Conneau et al., 2020; Liu et al., 2020; Xue et al., 2021; Workshop et
al., 2022) are effective for solving various language understanding and
generation in various languages (Hu et al., 2020; Wilie et al., 2020;
Cahyawijaya et al., 2021; Adelani et al., 2022; Kumar et al., 2022). To solve
the curse of multilinguality of MLMs, the modular MLM approach is introduced.
MAD-X (Pfeiffer et al., 2020) and MAD-G Ansell et al. (2021) use adapt MLMs to
new languages by using language adapters. X-MOD (Pfeiffer et al., 2022)
introduces modularity during pre-training which better aligns modular MLMs
across languages.

#### Language Identification (LID)

The LID task is introduced over five decades ago (Gold, 1967). Since then,
various methods for LID have been introduced, such as n-gram similarity
(Cavnar & Trenkle, 1994), naive bayes (Baldwin & Lui, 2010; Lui & Baldwin,
2012; Sites, 2013), and gaussian mixture (Lui et al., 2014). More recently,
embedding-based methods using character (Salcianu et al., 2020) and subwords
(Joulin et al., 2017) have also been introduced. In this work, we explore the
effect of utilizing these LID modules on the performance of modular MLMs.

##  3 Experimental Setting

For our experiments, we utilize MASSIVE (FitzGerald et al., 2022), a
multilingual intent classification dataset covering 52 typologically-diverse
languages. We select 24 languages from MASSIVE and group them into 3 different
resource groups based on the language size in CommonCrawl
111https://commoncrawl.github.io/cc-crawl-statistics/plots/languages , i.e.,
high-resource languages (HRL), medium-resource languages (MRL), and low-
resource languages (LRL). A detailed list of languages under study and the
resource grouping is described in Appendix A. For the LID, we incorporate 5
off-the-shelf LID models, i.e., LangDetect (Nakatani, 2011), langid.py (Lui &
Baldwin, 2012), FastText LID (Joulin et al., 2017), CLD2 (Sites, 2013), and
CLD3 (Salcianu et al., 2020). We evaluate these LIDs and take the best two
LIDs for the multilingual evaluation with unknown languages. For the modular
MLM, we utilize MAD-X Pfeiffer et al. (2020) with mBERT backbone. We compare
the MAD-X with LID against two direct fine-tuned MLMs and MAD-X without LID.
We use accuracy score as the evaluation metric in our experiment.

LID Model | HRL | MRL | LRL | AVG
---|---|---|---|---
Fully support languages under study
FastText | 97.22 | 96.26 | 88.96 | 93.89
CLD3 | 87.84 | 89.30 | 91.47 | 89.57
CLD2 | 76.07 | 90.85 | 85.14 | 83.17
Partially support languages under study222We zero out the performance for all
the unsupported languages.
langid.py | 92.00 | 93.04 | 76.12 | 86.31
LangDetect | 69.26 | 96.45 | 42.97 | 66.20

Table 1: Accuracy score of LIDs on MASSIVE. Most LIDs perform well on HRL and
MRL, but the score falls short on LRL. Bold and underline denote first and
second best, respectively.

NLU Model | HRL | MRL | LRL | AVG
---|---|---|---|---
Direct fine-tuning
XLMR | 86.03 | 84.76 | 83.20 | 84.65
mBERT | 84.76 | 82.50 | 80.62 | 82.64
Language adapter tuning
MAD-X (No LID) | 83.30 | 80.96 | 79.46 | 81.27
MAD-X (FastText) | 75.21 | 78.08 | 72.46 | 74.90
MAD-X (CLD3) | 72.90 | 75.20 | 72.89 | 73.47

Table 2: Accuracy score of MLMs on MASSIVE. Incorporating LID decays the
performance of the language-adapter model. Bold denotes the best performance.

##  4 Result & Discussion

Based on the result of the LID experiment in Table 2, we select FastText and
CLD3 for evaluating modular MLMs with unknown languages. The modular MLMs
result is shown in Table 2. For the modular MLM without LID, our result aligns
with prior works Pfeiffer et al. (2020); Ansell et al. (2021) yielding a
slightly lower score compared to the direct fine-tuned models. Both modular
MLMs with LID produce an even lower performance in all language resource
groups compared to the modular MLM without LID, resulting in a gap of
$\sim$7-8% accuracy score over all language groups. The detailed result of our
experiment is shown in Appendix B.

We clearly observe that existing off-the-shelf LID is far from the ideal case
which widens the gap to the direct fine-tuning approach and raises an open
question for closing the performance gap. To address the question, it is
important to understand the limitations of using modular MLMs with off-the-
shelf LIDs. Several potential limitations that might occur include: 1)
distribution shift of LIDs caused by domain and time differences, 2) label
mismatch between LID and the language adapter, and 3) other linguistic
problems that affect LIDs such as code-mixing and creole language. We leave
the exploration of the solution to these potential limitations for future
works.

##  5 Conclusion

In this work, we show the limitation of modular multilingual language models
(MLMs) in inferencing with unknown languages. We evaluate the effect of using
off-the-shelf LID modules on the evaluation of modular MLMs. Our result
suggests that using off-the-shelf LID modules significantly decreases the
performance of modular MLMs by $\sim$7-8% accuracy which widens the gap
between modular MLMs and non-modular MLMs. In addition, we discuss several
potential limitations that might contribute to the performance gap of using
off-the-shelf LID with modular MLMs.

### URM Statement

All authors of this paper qualify as an underrepresented minority (URM) for
the “Tiny Papers” track at ICLR 2023.

## References

  * Adelani et al. (2022) David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba Alabi, Shamsuddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye, Amelia Taylor, Fatoumata Kabore, Chris Chinenye Emezue, Anuoluwapo Aremu, Perez Ogayo, Catherine Gitau, Edwin Munkoh-Buabeng, Victoire Memdjokam Koagne, Allahsera Auguste Tapo, Tebogo Macucwa, Vukosi Marivate, Mboning Tchiaze Elvis, Tajuddeen Gwadabe, Tosin Adewumi, Orevaoghene Ahia, Joyce Nakatumba-Nabende, Neo Lerato Mokono, Ignatius Ezeani, Chiamaka Chukwuneke, Mofetoluwa Oluwaseun Adeyemi, Gilles Quentin Hacheme, Idris Abdulmumin, Odunayo Ogundepo, Oreen Yousuf, Tatiana Moteu, and Dietrich Klakow.  MasakhaNER 2.0: Africa-centric transfer learning for named entity recognition.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 4488–4508, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  URL https://aclanthology.org/2022.emnlp-main.298.
  * Ansell et al. (2021) Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Sebastian Ruder, Goran Glavaš, Ivan Vulić, and Anna Korhonen.  MAD-G: Multilingual adapter generation for efficient cross-lingual transfer.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pp. 4762–4781, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.findings-emnlp.410.  URL https://aclanthology.org/2021.findings-emnlp.410.
  * Baldwin & Lui (2010) Timothy Baldwin and Marco Lui.  Language identification: The long and the short of the matter.  In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , pp. 229–237, Los Angeles, California, June 2010. Association for Computational Linguistics.  URL https://aclanthology.org/N10-1027.
  * Cahyawijaya et al. (2021) Samuel Cahyawijaya, Genta Indra Winata, Bryan Wilie, Karissa Vincentio, Xiaohong Li, Adhiguna Kuncoro, Sebastian Ruder, Zhi Yuan Lim, Syafri Bahar, Masayu Khodra, Ayu Purwarianti, and Pascale Fung.  IndoNLG: Benchmark and resources for evaluating Indonesian natural language generation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 8875–8898, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.699.  URL https://aclanthology.org/2021.emnlp-main.699.
  * Cavnar & Trenkle (1994) William B. Cavnar and John M. Trenkle.  N-gram-based text categorization.  In _Proceedings of SDAIR-94, 3rd Annual Symposium on Document Analysis and Information Retrieval_ , pp. 161–175, Las Vegas, US, 1994.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 8440–8451, 2020.
  * FitzGerald et al. (2022) Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natarajan.  Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages, 2022.
  * Gold (1967) E Mark Gold.  Language identification in the limit.  _Information and Control_ , 10(5):447–474, 1967\.  ISSN 0019-9958.  doi: https://doi.org/10.1016/S0019-9958(67)91165-5.  URL https://www.sciencedirect.com/science/article/pii/S0019995867911655.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson.  XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In Hal Daumé III and Aarti Singh (eds.), _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pp. 4411–4421. PMLR, 13–18 Jul 2020.  URL https://proceedings.mlr.press/v119/hu20b.html.
  * Joulin et al. (2017) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov.  Bag of tricks for efficient text classification.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pp. 427–431, Valencia, Spain, April 2017. Association for Computational Linguistics.  URL https://aclanthology.org/E17-2068.
  * Kumar et al. (2022) Aman Kumar, Himani Shrotriya, Prachi Sahu, Amogh Mishra, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, Mitesh M. Khapra, and Pratyush Kumar.  IndicNLG benchmark: Multilingual datasets for diverse NLG tasks in Indic languages.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 5363–5394, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  URL https://aclanthology.org/2022.emnlp-main.360.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742, 2020.  doi: 10.1162/tacl˙a˙00343.  URL https://aclanthology.org/2020.tacl-1.47.
  * Lui & Baldwin (2012) Marco Lui and Timothy Baldwin.  langid.py: An off-the-shelf language identification tool.  In _Proceedings of the ACL 2012 System Demonstrations_ , pp. 25–30, Jeju Island, Korea, July 2012. Association for Computational Linguistics.  URL https://aclanthology.org/P12-3005.
  * Lui et al. (2014) Marco Lui, Jey Han Lau, and Timothy Baldwin.  Automatic detection and language identification of multilingual documents.  _Transactions of the Association for Computational Linguistics_ , 2:27–40, 2014.  doi: 10.1162/tacl˙a˙00163.  URL https://aclanthology.org/Q14-1003.
  * Nakatani (2011) Shuyo Nakatani.  Language detection library for java, 2011.  URL https://github.com/shuyo/language-detection.
  * Pfeiffer et al. (2020) Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder.  MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 7654–7673, Online, November 2020\. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.617.  URL https://aclanthology.org/2020.emnlp-main.617.
  * Pfeiffer et al. (2022) Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe.  Lifting the curse of multilinguality by pre-training modular transformers.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 3479–3495, Seattle, United States, July 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.naacl-main.255.  URL https://aclanthology.org/2022.naacl-main.255.
  * Salcianu et al. (2020) Alex Salcianu, Andy Golding, Anton Bakalov, Chris Alberti, Daniel Andor, David Weiss, Emily Pitler, Greg Coppola, Jason Riesa, Kuzman Ganchev, Michael Ringgaard, Nan Hua, Ryan McDonald, Slav Petrov, Stefan Istrate, and Terry Koo.  Compact language detector v3 (cld3), 2020.  URL https://github.com/google/cld3.
  * Sites (2013) Richard Sites.  Compact language detector v2 (cld2), 2013.  URL https://github.com/CLD2Owners/cld2.
  * Wilie et al. (2020) Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti.  IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pp. 843–857, 2020.
  * Workshop et al. (2022) BigScience Workshop, :, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, Dragomir Radev, Eduardo González Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, María Grandury, Mario Šaško, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaieh Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Lepercq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davut Emre Taşar, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szczechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Matteo Manica, Nihal Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry, Trishala Neeraj, Urmish Thakker, Vikas Raunak, Xiangru Tang, Zheng-Xin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jaesung Tae, Jason Phang, Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, Sanchit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurélie Névéol, Charles Lovering, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bogdanov, Genta Indra Winata, Hailey Schoelkopf, Jan-Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdeněk Kasner, Alice Rueda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh HajiHosseini, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Muñoz Ferrandis, Danish Contractor, David Lansky, Davis David, Douwe Kiela, Duong A. Nguyen, Edward Tan, Emi Baylor, Ezinwanne Ozoani, Fatima Mirza, Frankline Ononiwu, Habib Rezanejad, Hessie Jones, Indrani Bhattacharya, Irene Solaiman, Irina Sedenko, Isar Nejadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim Elbadri, Margot Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Rajani, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas Wang, Sourav Roy, Sylvain Viguier, Thanh Le, Tobi Oyebade, Trieu Le, Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, Antonio Miranda-Escalada, Ayush Singh, Benjamin Beilharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñán, Daniel Molano, Dian Yu, Enrique Manjavacas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Rangasai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Marc Pàmies, Maria A Castillo, Marianna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cullan, Michael Weinberg, Michiel De Wolf, Mina Mihaljcic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Sinee Sang-aroonsiri, Srishti Kumar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gigant, Tomoya Kainuma, Wojciech Kusa, Yanis Labrak, Yash Shailesh Bajaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf.  Bloom: A 176b-parameter open-access multilingual language model, 2022\.  URL https://arxiv.org/abs/2211.05100.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 483–498, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.41.  URL https://aclanthology.org/2021.naacl-main.41.

##  Appendix A Language Under Study

We provide the list of all languages under study along with the language
resource group in Table 3. Language resource is grouped by the size of
language data in CommonCrawl, i.e., high-resource languages ($\geq$1%, medium-
resource languages (($\geq$0.1%), and low-resource languages ($<$1%).

Language | #Speaker | CC Size | Resource Group
---|---|---|---
ar-SA | 360M | 0.665% | MRL
bn-BD | 300M | 0.093% | LRL
de-DE | 95M | 5.662% | HRL
el-GR | 13.5M | 0.597% | MRL
en-US | 373M | 46.320% | HRL
es-ES | 493M | 4.435% | HRL
fi-FI | 5.4M | 0.398% | LRL
fr-FR | 300M | 4.604% | HRL
hi-IN | 528M | 0.155% | LRL
hu-HU | 13M | 0.599% | MRL
hy-AM | 5.4M | 0.032% | LRL
id-ID | 300M | 0.781% | MRL
is-IS | 0.3M | 0.038% | LRL
ja-JP | 128M | 4.532% | HRL
jv-ID | 82M | 0.002% | LRL
ka-GE | 3.7M | 0.037% | LRL
ko-KR | 79.3M | 0.679% | MRL
lv-LV | 1.2M | 0.082% | LRL
my-MM | 33M | 0.012% | LRL
pt-PT | 250M | 1.482% | HRL
ru-RU | 258M | 5.717% | HRL
vi-VN | 70M | 0.962% | MRL
zh-CN | 920M | 4.837% | HRL
zh-TW | 4.6M | 4.837%333We use the same number of zh-CN and zh-TW, since there is no Chinese (zh) language variation in CommonCrawl. | HRL

Table 3: List of languages under study in our experiments. The number of
speaker information is retrieved from Wikipedia.

##  Appendix B Detailed Experiment Result

We provide the complete per language result for the language identification
and the modular MLMs experiments in Table 4 and Table 5.

Language | LID-Fasttext | CLD3 | CLD2 | langid | LangDetect
---|---|---|---|---|---
ar-SA | 94.25 | 86.45 | 81.58 | 91.78 | 94.13
bn-BD | 99.72 | 97.52 | 89.57 | 96.93 | 99.76
de-DE | 97.70 | 88.59 | 89.73 | 92.83 | 82.54
el-GR | 99.68 | 96.91 | 99.77 | 99.84 | 99.64
en-US | 98.61 | 79.44 | 93.43 | 93.96 | 87.82
es-ES | 96.20 | 78.24 | 73.14 | 86.87 | 86.55
fi-FI | 97.70 | 92.91 | 92.90 | 92.08 | 96.09
fr-FR | 98.35 | 87.53 | 85.23 | 94.77 | 94.80
hi-IN | 98.44 | 88.21 | 97.83 | 87.94 | 93.54
hu-HU | 98.54 | 92.24 | 93.89 | 95.34 | 96.71
hy-AM | 99.90 | 98.37 | 99.92 | 99.17 | 0.00
id-ID | 87.20 | 65.86 | 73.54 | 72.68 | 89.32
is-IS | 89.93 | 92.64 | 90.88 | 92.97 | 0.00
ja-JP | 99.41 | 96.63 | 99.04 | 99.11 | 96.23
jv-ID | 24.75 | 68.10 | 0.00 | 22.04 | 0.00
ka-GE | 99.56 | 98.49 | 99.95 | 99.65 | 0.00
ko-KR | 99.50 | 98.47 | 99.03 | 99.96 | 99.36
lv-LV | 90.73 | 90.06 | 95.25 | 94.33 | 97.32
my-MM | 99.93 | 96.90 | 99.97 | 0.00 | 0.00
pt-PT | 92.17 | 83.42 | 77.39 | 77.74 | 84.05
ru-RU | 99.27 | 84.48 | 82.35 | 83.79 | 91.32
vi-VN | 98.41 | 95.85 | 97.26 | 98.62 | 99.53
zh-CN | 97.55 | 98.07 | 84.33 | 99.64 | 0.00
zh-TW | 95.76 | 94.19 | 0.03 | 99.31 | 0.00
Average | 93.89 | 89.57 | 83.17 | 86.31 | 66.20

Table 4: Per language results of language identification evaluation in
MASSIVE.

Language | XLMR | mBERT | MAD-X |  | MAD-X
---
w/ FastText
| MAD-X
---
w/ CLD3
ar-SA | 79.32 | 78.35 | 75.72 | 71.92 | 67.79
bn-BD | 83.25 | 80.23 | 78.61 | 76.36 | 74.95
de-DE | 85.54 | 83.59 | 81.81 | 79.49 | 76.90
el-GR | 85.07 | 81.74 | 80.93 | 79.56 | 78.51
en-US | 88.16 | 86.45 | 85.78 | 83.89 | 83.15
es-ES | 86.18 | 84.97 | 82.58 | 80.97 | 76.43
fi-FI | 85.24 | 82.55 | 82.55 | 79.86 | 77.07
fr-FR | 86.48 | 86.11 | 83.69 | 82.35 | 80.03
hi-IN | 84.63 | 82.38 | 80.73 | 78.14 | 72.73
hu-HU | 85.68 | 82.65 | 81.57 | 80.13 | 76.40
hy-AM | 84.23 | 81.20 | 80.43 | 78.78 | 77.91
id-ID | 86.52 | 84.67 | 82.01 | 76.03 | 69.30
is-IS | 84.16 | 82.21 | 80.40 | 71.49 | 73.57
ja-JP | 85.78 | 84.70 | 83.22 | 82.04 | 81.27
jv-ID | 81.20 | 81.57 | 78.58 | 45.70 | 59.68
ka-GE | 79.19 | 75.25 | 73.23 | 70.85 | 70.17
ko-KR | 85.51 | 84.30 | 82.99 | 81.14 | 80.56
lv-LV | 84.73 | 82.18 | 82.08 | 74.58 | 74.95
my-MM | 82.18 | 78.01 | 78.48 | 76.36 | 74.98
pt-PT | 86.35 | 85.27 | 83.59 | 80.56 | 77.77
ru-RU | 86.65 | 83.96 | 83.52 | 81.74 | 75.45
vi-VN | 86.48 | 83.32 | 82.52 | 79.72 | 78.61
zh-CN | 85.41 | 85.24 | 84.23 | 53.09 | 52.69
zh-TW | 83.73 | 82.55 | 81.27 | 52.79 | 52.45
Average | 84.65 | 82.64 | 81.27 | 74.90 | 73.47

Table 5: Per language accuracy score of multilingual language models in
MASSIVE.