# Measuring arousal and stress physiology on Esports, a League of Legends case
study

David Berga  Eurecat Centre Tecnològic  Alexande Pereda  Eurecat Centre
Tecnològic  Eleonora De Filippi  Eurecat Centre Tecnològic  Arijit Nandi
Eurecat Centre Tecnològic  Eulàlia Febrer Coll  eDojo - Electronic Dojo S.L.
Marta Reverte  eDojo - Electronic Dojo S.L.  Lautaro J. Russo  Mental Gaming
S.L.

###### Abstract

Esports gaming is an area in which videogame players need to cooperate and
compete with each other, influencing their cognitive load, processing, stress,
and social skills. Here it is unknown to which extent competitive videogame
play using a desktop setting can affect the physiological responses of
players’ autonomic nervous system. For such, we propose a study where we have
measured distinct electrodermal and cardiac activity metrics over competitive
players during several League of Legends gameplay sessions in a Esports
stadium. We mainly found that game performance (whether winning or losing the
game) significantly affects both electrodermal and cardiac activity, where
players who lost the game showed higher stress-related physiological
responses, as compared to winning players. We also found that important
specific in-game events such as ”Killing”, ”Dying” or ”Destroying Turret”
significantly increased both electrodermal and cardiac activity over players
more than other less-relevant events such as ”Placing Wards” or ”Destroying
Turret Plates”. Finally, by analyzing activity over player roles we found
different trends of activity on these measurements, this could foster the
exploration on human physiology with a higher set of participants in future
Esports studies.

##  1 Introduction

Research in the field of psychology has traditionally focused on three main
forms of both emotional and attentional responses: subjective perceptions of
the individual about their own state, effects on behavior, and changes in
their physiological patterns, such as acceleration or deceleration of heart
rate and increase in skin conductivity [7, 14]. Each one of these approaches
shows both advantages and disadvantages. First, auto-informed methods, such as
questionnaires or interviews, for which participants are directly asked to
report their status are the only way to access the individual’s subjective
perception are limited by the individual’s own ability to introspect, since
many psychological processes can occur unconsciously or with low levels of
consciousness [15]. Moreover, in certain cases it is possible that cognitive
biases (such as social desirability bias) interfere with the reports, making
the information not entirely reliable. For this reason, in the field of
experimental psychology research, the analysis of physiological responses (for
example, variations in heart rate, skin conductivity, activation of facial
muscles, etc.) has been introduced as a way of obtaining information about the
cognitive and emotional processes of individuals in an indirect way. On the
other hand, the main disadvantage of the physiological methods with respect to
the self-reported ones is that physiological experimentation requires a higher
amount of resources and thus data gathering is taken in a much slower pace, so
it is recommended to combine both methodologies with qualitative studies for
which data can be obtained from a large number of participants, and laboratory
studies with smaller samples [8]. An experimental suggestion is to adapt the
gathering of game events according to the type of physiological sub-measure
that is desired to capture. Here are the main physiological measures that we
used in our study:

#### Electrodermal activity (or EDA, also known as galvanic skin response or
GSR)

is a correlate of the activation of the sympathetic branch of the autonomic
nervous system, which provides reliable information with a high temporal
resolution about participants’ physiological arousal, related to the intensity
of experienced cognitive-emotional responses.

####  Cardiac activity (ECG or electrocardiogram), or optical pulse (PPG)

is a measure of Heart Rate. This signal is mediated by both the sympathetic
and parasympathetic systems, thus responding to both physiological arousal and
emotional regulation processes. This is dependent on both the intensity of
emotions and their hedonic load, also with the appearance of cognitive
resources for stimulus processing. The variability of heart rate can be
analyzed by looking at the ratio of sympathetic vs parasympathetic activity
within the PPG signal.

### Related Work

Physiological studies were previously carried out in the game literature [12,
3, 2] analyzing signals such as the ECG/PPG and the GSR/EDA, showing relevant
differences in the participants given their affective state, mostly validated
with questionnaires. Various other studies have also compared muscle signals
(Electomyography/EMG) [1], brain signals (Electroencephalography/EEG) [10],
and facial gestures [18]. The main focus of some of these latter studies is to
assess the affective state of participants from these measures, classifying
each one of the 7 affective categories of basic emotions initially defined by
[9]: anger, sadness, fear, disgust, joy, surprise, contempt, and neutral. Each
of the affective categories is standardized and validated with different
psychophysiological tests such as the Russell test [17] or the Self Assessment
Mannekin [6], serving as a guideline for the validation of physiological
measures that could affect the affective responses of each individual. In this
case, the EDA and the ECG are signals widely validated by the literature that
can give insight into the changes in the affective states of each individual
in a real-time manner.

In a recent review by Leis $\&$ Lautenbach [13], 17 studies were meta-analyzed
in Esports contexts for psychological and physiological stress, and it was
concluded that simply playing in an Esports non-competitive environment
produced no stress reactions, whereas in competitive environments several
studies reported increases in anxiety levels, cortisol levels, and
physiological sympathetic activation, all three indicators of stress [11, 25].
However, stress is not the only interesting indicator to consider in Esports
environments, whether competitive or not, peripheral physiology can also
provide insight into various aspects of cognitive/emotional information
processing, such as emotion, valence (polarity), engagement, boredom,
frustration, etc. The benefits of extracting this information have been
evident for the case of stress [5] and fatigue [20, 21]. They performed a very
comprehensive sensory analysis that results in the submission of a dataset
collected from 10 players on professional and amateur teams in 22 League of
Legends video matches, including the simultaneous recording of the
physiological (i.e. movements, pulse, saccades) and psychological (self-
reported post-match survey) activity of five players, with interesting results
such as the lack of correlation between stress and concentration levels for
professional players. We take a similar approach, focusing on a simultaneous
exploration of electrodermal activity and cardiac activity in all five players
of a team in different events.

#### Contributions

In our study we have done the physiological analysis in an Esports stadium
environment where users engage in a widely played desktop videogame ”League of
Legends”. These are our main objectives:

  * •

Evaluate the affective responses by analyzing distinct EDA and ECG metrics
depending on game performance (winning or losing).

  * •

Analyse differences in physiological responses over game events (e.g.
”Killing”, ”Dying”, ”Kill Assist”, ”Destroy Turret”, ”Destroy Turret Plate”,
”Placing Ward”, etc.)

  * •

Investigate statistics in individual physiological responses based on events
and player roles (Jungle, Middle, Utility, Bottom, and Top)

  * •

Provide the software tools necessary for the analysis and synchronization of
League of Legends game events with physiological data of EDA and ECG.

##  2 Methods

###  2.1 Experimental Design

#### Sensors and Software

For our experiments we used a Shimmer3111https://shimmersensing.com/ pack with
5 simultaneous GSR+ Units providing Galvanic Skin Response for acquisition of
Electrodermal Activity (EDA), as well as Optical Pulse (PPG) estimating heart
rate variations. We developed our own Python tools (available in
github222https://github.com/dberga/riotwatcher-shimmer-pynput) for capturing
data and sending it through Bluetooth (using pyserial and pylsl) and
synchronizing that data with events of the game (using riotwatcher) for later
statistical analysis for EDA (Ledalab333http://www.ledalab.de/ V3.4.9) and PPG
(HeartPy444https://github.com/smritisridhar41/Heartrate_Analysis/).

#### League of Legends Skills and Mechanics

League of Legends555https://www.leagueoflegends.com/ is a Multiplayer Online
Battle Arena (MOBA) videogame developed by Riot Games
666https://www.riotgames.com/en in 2009. It has become one (if not) the most
popular game of the current generation of online
videogames777https://medium.com/codex/the-growing-trend-of-esports-
why-2023-is-going-to-be-a-massive-year-for-esports-e17fe39ac58a and has a
significant impact on the economy of many other industrial and technology
sectors (i.e. branding, consumables, social networks and media content). The
original League of Legends game type mechanics (Summoner Rift) consists on a
competition between 2 teams of 5 players in which each team has to destroy the
enemy base or ”Nexus”. The battle zone is composed by 3 lanes (top, mid and
bot) as well as a jungle (in between these lanes). Each team lane has 4
turrets and an inhibitor, and two more turrets in the Nexus. However, before
reaching the enemy base, each team needs to destroy the turrets and the
inhibitor of one of the enemy lanes. The usual teamwork consists on two
players (Carry/Bottom and Utility/Support) controlling the bot lane, the
Top/Tank controlling the top lane, the Middle controlling the mid lane and the
Jungle that moves around all the lanes. As in other MOBAs, players will need
to cooperate and sometimes play aggressively to kill the opponent’s champions
and overcome the enemy positions. Riot users (summoners) have a specific
experience
level888https://leagueoflegends.fandom.com/wiki/Experience_(summoner) (given
its in-game time) and
rank999https://leagueoflegends.fandom.com/wiki/Rank_(League_of_Legends) (given
its actual performance in ranked games).

#### Game Sessions and Subjects

A total of 4 sessions have been performed in an Asobu eSports
Experience101010https://asobuesports.com/ with 16 participants (contacted and
selected by United Gamers Academy111111https://unitedgamers.pro/) in the
gameplay experimentation. The participants were recruited with questionnaires
reporting age, gender, skill level, rank level and game preferences. Only
players with higher rank levels above silver III were selected for the study.
Subjects’ age was between 18 and 25 y.o. and 4 reported to be female whilst 12
male. Average player level was 216 (with lowest 82 and biggest 402) and
corresponded to silver-gold S12 competitive rank gamers. From the 16 subjects,
we captured data from 4 sessions of a total of 12 participants playing a
specific team during Summoner’s Rift gameplay (avg time 30-45 min), later
filtered on 7 with enough valid events for statistical comparison. We cut the
recording of these participants from the start to the end of the game and we
set specific window times for each event (i.e. 5 sec). This data is
synchronized with events downloaded from riotwatcher api121212Riot API tokens
available in https://developer.riotgames.com/. Some events available for
capture are ”killing”, ”dying”, ”kill assist”, ”special killing”, ”item
purchased”, ”level up”, ”ward placed”, ”building kill”, ”champion transform”,
”turret plate destroyed” and ”elite monster kill”. After gameplay we annotated
riot’s metadata for each participant such as game session data (total
kills/deaths, damage done, etc.), win or loss condition and player roles (top,
mid, bot, utility and jungle).

###  2.2 Physiological data processing

#### GSR preprocessing

The MatLab-based toolbox Ledalab [4] was used for the GSR signal preprocessing
and analysis. First, we carried out a preliminary visual examination to look
for periodic drift in the signal, which reflects artifacts, and we resampled
the raw signal to 50Hz using
Neurokit2131313https://neuropsychology.github.io/NeuroKit/. The following
preprocessing operations were then carried out using Ledalab toolbox: low-pass
Butterworth filtering with a cutoff frequency of 5 Hz, and smoothing to
eliminate any remaining artifacts. Finally, we performed an event-related
analysis utilizing the Continuous Decomposition Analysis to extract the
features indicating Skin Conductance Responses (SCRs) (CDA). By extracting the
phasic (driver) information underlying EDA, this approach attempts to obtain
the signal features of the underlying sudomotor nerve activity. Skin
conductance data is deconvoluted by the overall response shape, considerably
enhancing temporal accuracy. This method enables the extraction of continuous
phasic and tonic activity based on traditional deconvolution within a
predetermined time window, which for us corresponded to a window comprising
the three seconds before an event marker to the five following seconds. The
number of SCRs within the response window, response latency for the first SCR,
mean SCR amplitudes, maximum phasic, and average tonic activity within the
specified window were therefore collected for each event described in the
previous section.

#### PPG data processing

Processing and analysis of raw PPG data were conducted using the Python-based
toolkit ”Heartpy” [24, 23], specialized for the analysis of PPG signal as
compared to ECG. At every heartbeat, blood perfuses via the capillaries and
arteries, causing the skin to become discolored. The PPG detects this
discoloration. The systolic peak, diastolic notch, and diastolic peak make up
the signal. First, as we did with the GSR signal, we resampled the raw PPG
signal to 50Hz using Neurokit2. Then, we run the processing algorithm that
comes with the Heartpy toolkit and which allows for the peak detection to
extract reliable time-domain measures, such as beats per minute (BPM), and
Interbeat Intervals (IBI). Furthermore, for each event, we extracted measures
that reflect Heart Rate Variability (HRV) such as the RMSSD (root mean square
of successive differences) and the SDSD (standard deviation of successive
differences).

##  3 Results

We performed data curation for our statistical analysis using data from 7
participants (a total of 2 game sessions with recorded measures in which 3
participants played twice) with enough event samples for later analysis and
processing. Some of the data not mentioned in participants results was
discarded for the final evaluation due to the incorrect samples from the
sensor data and/or given the lack of relevant events in the gameplay to sync
with the sensor data (e.g. given too much time being dead without interacting
with game objects nor players). Here we select the players’ sensor data
collected with enough samples to make valid statistical evaluations.

###  3.1 Physiological results: Skin Conductance

We have processed raw GSR data with Ledalab to extract the following measures:
nrSCRs (total skin conductance number ”#” of responses above threshold),
Latency (delay/surpassed time ”s” to elicit EDA with respect to the event),
Amplitude (mean activity ”mV” inside the event window), PhasicMax (max phasic
value ”mV” from the gap with respect the response and the event window) and
Tonic (max tonic activity ”mV” with respect window). Previous literature in
electrodermal physiology has shown EDA can be a reliable quantifier of
sympathetic dynamics [16], meaning higher EDA correlated with higher
sympathetic (stress/alert) levels.

wincon | nrSCR | Latency | Amplitude | PhasicMax | Tonic
---|---|---|---|---|---
WIN | 1.853±1.730 | *-.151±1.926 | *.274±.631 | .545±1.165 | *12.577±7.047
LOSS | 2.599±2.210 | -.758±1.698 | .190±.414 | .370±.545 | 6.881±4.743
TOTAL | 2.372±2.101 | -.574±1.788 | .215±.490 | .423±.788 | 8.611±6.121

Table 1: Win and Loss mean GSR statistics (2 sessions) by stacking all events
in one statistic. *$p<0.05$

EVENT | nrSCR | Latency | Amplitude | PhasicMax | Tonic | N
---|---|---|---|---|---|---
KILL | *1.225±1.170 | 0.0485±2.114 | *0.031±0.0416 | *0.101±0.066 | 7.764±3.233 | 9
DIE | 2.423±2.102 | -0.191±0.739 | 0.323±0.627 | 0.504±0.862 | *12.198±5.818 | 35
PLACE WARD | 2.32±2.160 | -0.295±2.145 | 0.225±0.448 | 0.152±0.138 | *12.064±5.410 | 75
DES.TURRET | *2.182±2.214 | -0.611±1.537 | *0.156±0.233 | 0.250±0.439 | 9.745±6.270 | 79
DES.PLATE | *2.175±1.885 | -0.457±1.837 | 0.294±0.587 | 0.457±0.680 | 3.087±1.768 | 49

Table 2: Event mean GSR statistics (2 sessions) from events ”Killing”,
”Dying”, ”Placing Ward”, ”Destroying Turret” and ”Destroying Turret Plate”.
*$p<0.05$

Event | Role | nrSCR | Latency | Amplitude | PhasicMax | Tonic | N
---|---|---|---|---|---|---|---
KILL | Jungle | 1.200±2.168 | .644±1.578 | .039±.056 | .114±.085 | 7.974±3.427 | 5
| Middle | 1.333±.577 | -.713±3.271 | .024±.017 | .100±.014 | 5.993±.479 | 3
| Utility | .000±.000 | .000±.000 | .000±.000 | .007±.000 | 11.78±.000 | 1
| Bottom | .000±.000 | .000±.000 | .000±.000 | .000±.000 | .000±.000 | 0
| Top | .000±.000 | .000±.000 | .000±.000 | .000±.000 | .000±.000 | 0
DIE | Jungle | 3.000±2.793 | -.436±2.147 | .522±1.037 | .694±1.490 | 11.97±4.11 | 11
| Middle | .800±1.789 | -.276±.617 | .034±.076 | .163±.148 | 6.157±.780 | 5
| Utility | 2.333±.577 | -2.060±1.45 | .083±.039 | .041±.014 | 11.79±1.45 | 3
| Bottom | 2.909±2.071 | -.969±1.125 | .158±.170 | .279±.223 | 6.329±.415 | 11
| Top | 1.737±1.968 | -.645±1.377 | .127±.267 | .292±.333 | 10.84±9.42 | 19
PLACE | Jungle | 3.000±2.512 | .201±2.827 | .151±.189 | .241±.250 | 9.458±4.04 | 14
WARD | Middle | .750±1.035 | -.598±1.109 | .029±.054 | .064±.105 | 9.40±2.91 | 8
| Utility | .250±.500 | .560±1.120 | .005±.010 | .238±.203 | 9.537±1.67 | 4
| Bottom | .000±.000 | .000±.000 | .000±.000 | .000±.000 | .000±.000 | 0
| Top | 2.778±.833 | -.544±1.462 | .675±1.048 | 1.335±2.00 | 21.59±2.16 | 9
DES. | Jungle | 3.353±2.396 | -.813±1.615 | .186±.198 | .311±.342 | 14.05±3.70 | 17
TURRET | Middle | 1.800±2.168 | -.592±.849 | .068±.088 | .278±.214 | 5.802±.444 | 5
| Utility | 2.458±1.911 | -.419±2.134 | .177±.287 | .623±1.294 | 3.355±4.192 | 24
| Bottom | 3.077±1.935 | -.592±2.155 | .350±.562 | .546±.718 | 6.248±.617 | 13
| Top | 1.750±1.915 | -.488±1.661 | .082±.127 | .454±.494 | 6.731±7.723 | 16
DES. | Jungle | 3.647±1.967 | -.673±1.915 | .678±1.02 | .844±1.136 | 11.036±2.89 | 17
PLATE | Middle | 1.400±1.174 | -.794±1.194 | .044±.053 | .218±.270 | 5.962±.420 | 10
| Utility | 2.571±2.181 | -1.298±1.51 | .148±.185 | .354±.303 | 5.897±3.913 | 21
| Bottom | 3.615±1.758 | -.740±2.099 | .152±.185 | .233±.198 | 6.215±.366 | 13
| Top | 1.278±1.965 | -.243±1.888 | .195±.489 | .368±.439 | 8.588±9.302 | 18
ALL | Jungle | 3.125±2.380 | -.375±2.089 | .355±.716 | .488±.893 | 11.41±4.02 | 64
EVENTS | Middle | 1.194±1.376 | -.619±1.267 | .040±.059 | .167±.199 | 6.858±2.119 | 31
| Utility | 2.283±1.984 | -.778±1.861 | .144±.229 | .443±.902 | 5.465±4.515 | 53
| Bottom | 3.216±1.888 | -.756±1.842 | .223±.366 | .357±.468 | 6.260±.471 | 37
| Top | 1.758±1.853 | -.473±1.593 | .215±.523 | .507±.891 | 10.69±9.41 | 62

Table 3: Role-dependent mean nrSCR (#), Latency (sec), Amplitude (mV),
PhasicMax (mV) and Tonic (mV) statistics (2 sessions) from events ”Killing”,
”Dying”, ”Placing Ward”, ”Destroying Turret” and ”Destroying Turret Plate”. N
are event occurrences. *$p<0.05$

In Table 1 we show mean statistics of nrSCR, Latency, Amplitude, PhasicMax,
and Tonic values of players that win the gameplay and lose the gameplay.
Similarly, in TABLE 2 we show statistics for events ”Killing”, ”Dying”, ”Place
Ward”, ”Destroy Turret” and ”Destroy Turret Plate”. We expand these statistics
in Table 3 filtering player roles in the game.

Given the Chi-squared measured distributions (non-parametric) we performed
Friedman’s tests over win-loss and event data for each GSR metric. On
analyzing winning or losing the match (Tables 1-2), we found significant
differences in player’s activity during ”Killing” in nrSCR, Amplitude, and
PhasicMax activity ($p$=.046, $\chi^{2}$=4.000). We also observed significant
differences when winning/losing the game during ”Destroying Turret” in nrSCR
and Amplitude ($p$=.020, $\chi^{2}$=5.444) as well as ”Destroying Plate” for
nrSCR ($p$=.008, $\chi^{2}$=7.143) with a trend in Amplitude ($p$=.071,
$\chi^{2}$=3.266). The tonic activity was only significantly distinct
depending on winning/losing for the events of ”Dying” ($p$=.035,
$\chi^{2}$=4.455) and ”Placing Ward” ($p$=.002, $\chi^{2}$=10.000).

We also analyzed in distributions of GSR activity between all events for the
same winners and found significant differences in Latency ($p$=.024,
$\chi^{2}$=11.265), Amplitude ($p$=.041, $\chi^{2}$=9.959), Tonic activity
($p$=.010,$\chi^{2}$=13.28) and a trend for PhasicMax ($p$=.092,
$\chi^{2}$=8.000). In the same analysis, we did not find differences in GSR
activity between the events in the case of losing the game.

###  3.2 Physiological results: Heart Rate

We have processed raw PPG data with Heartpy to obtain the BPM (”#” beats per
minute), IBI (time ”ms” of interbeat interval or R-R), SDNN (standard
deviation of intervals ”ms” between adjacent beats of the IBI of normal sinus
beats), SDSD (standard deviation of successive differences between adjacent
R-R intervals ”ms”) and RMSSD (root mean square of successive differences
between adjacent R-R intervals ”ms”). The latter metrics (SDSD and RMSSD) are
related to the measurement of HRV (heart rate variability). Indeed, higher HRV
(higher values of SDSD or RMSSD) can represent parasympathetic/vagal activity
( associated with a state of relaxation), while a lower HRV (lower values for
SDSD or RMSSD) represents sympathetic/flight-or-fight activity (being stressed
or alert; [22]). Here we have to point out that studies on HRV are commonly
analyzed over large timeline streams of heart rate data (about 5 min or more;
[19]) for more simple and large tasks. However, our measurements of HRV are
considering 5 to 10-second windows of activity according to the League of
Legends fast-paced events. In TABLES 4-6 we show mean statistics of pulse
metrics according to win condition, event, and role.

wincon | BPM | IBI | SDNN | SDSD | RMSSD
---|---|---|---|---|---
WIN | 172.145±112.795 | 531.156±332.714 | 128.673±48.392 | *100.174±46.513 | 201.980±87.488
LOSS | *93.540±50.271 | *743.045±212.297 | 76.840±52.382 | 57.060±44.479 | 117.470±92.097
TOTAL | 114.062±79.606 | 687.724±265.416 | 90.373±56.104 | 68.316±48.750 | 139.534±98.038

Table 4: Win and Loss mean PPG statistics (2 sessions) by stacking all events
in one statistic. *$p<0.05$

EVENT | BPM | IBI | SDNN | SDSD | RMSSD | N
---|---|---|---|---|---|---
KILL | 68.620±10.396 | 892.429±139.157 | 73.404±46.177 | 56.974±31.484 | 99.246±53.674 | 7
DIE | 103.381±65.949 | 725.839±249.362 | 86.549±53.718 | *67.178±46.320 | *132.395±85.887 | 37
PLACE WARD | 121.905±75.396 | 647.536±278.835 | 92.073±53.922 | *75.722±54.092 | 143.491±93.544 | 31
DES.TURRET | 118.132±94.834 | 678.146±260.378 | 94.580±55.782 | 72.386±51.682 | 145.934±96.149 | 57
DES.PLATE | 117.418±78.046 | 672.916±275.895 | *89.918±60.236 | *63.527±46.951 | 140.361±110.991 | 71

Table 5: Event mean PPG statistics (2 sessions) from events ”Killing”,
”Dying”, ”Placing Ward”, ”Destroying Turret” and ”Destroying Turret Plate”.
*$p<0.05$

After performing Friedman tests over PPG metrics for all events and we found
significant differences in SDSD ($p=$.016, $\chi$=10.371) when winning the
game, while when losing the game we found differences in BPM and IBI
($p=$.041, $\chi$=8.28). We also tested if there were differences when winning
or losing the game for each specific event and we saw significant differences
for ”Destroying Turret Plate” between win and loss for SDSD ($p=$5.32$\times
10^{-4}$, $\chi$=12.0) and RMSSD ($p=$.004, $\chi$=8.333), and ”Dying” (SDSD;
$p=$.011, $\chi$=6.4)(RMSSD; $p=$.002, $\chi$=10) as well as for SDSD when
”Placing a Ward” ($p=$.011, $\chi$=6.4).

Event | Role | BPM | IBI | SDNN | SDSD | RMSSD | N
---|---|---|---|---|---|---|---
KILL | Jungle | 63.195±10.832 | 967.750±142.479 | 84.5987±57.767 | 66.8081±40.275 | 113.310±67.023 | 4
| Middle | 75.852±3.301 | 792.000±34.176 | 58.4780±28.399 | 43.8612±9.449 | 80.492±31.328 | 3
| Utility | .000±.000 | .000±.000 | .000±.000 | .000±.000 | .000±.000 | 0
| Bottom | .000±.000 | .000±.000 | .000±.000 | .000±.000 | .000±.000 | 0
| Top | .000±.000 | .000±.000 | .000±.000 | .000±.000 | .000±.000 | 0
DIE | Jungle | 75.215±8.200 | 806.800±93.180 | 69.730±48.760 | 49.185±41.834 | 99.139±63.691 | 10
| Middle | 80.846±8.515 | 748.667±77.577 | 67.150±31.274 | 55.873±25.619 | 95.511±31.233 | 5
| Utility | 176.58±60.409 | 375.714±157.912 | 158.366±16.102 | 104.249±26.433 | 254.481±58.683 | 3
| Bottom | 71.112±5.453 | 848.111±64.272 | 58.362±25.412 | 48.458±29.071 | 97.162±44.958 | 9
| Top | 149.90±100.33 | 628.457±392.726 | 116.888±63.553 | 96.551±58.618 | 179.177±108.76 | 37
PLACE | Jungle | 72.025±8.956 | 845.821±111.874 | 68.544±40.886 | 54.092±32.402 | 104.573±69.816 | 13
WARD | Middle | 75.660±5.498 | 796.250±59.320 | 60.883±25.517 | 41.489±21.775 | 80.912±39.805 | 4
| Utility | 151.11±70.545 | 513.417±302.925 | 111.647±53.012 | 96.855±71.343 | 175.363±99.071 | 8
| Bottom | .000±.000 | .000±.000 | .000±.000 | .000±.000 | .000±.000 | 0
| Top | 221.87±73.442 | 297.601±101.160 | 137.749±61.433 | 117.229±51.861 | 227.036±95.086 | 6
DES. | Jungle | 81.985±10.532 | 742.917±92.725 | 66.538±35.728 | 46.652±31.067 | 85.828±44.360 | 15
TURRET | Middle | 96.998±49.985 | 696.306±174.643 | 83.246±45.988 | 53.699±26.111 | 128.027±86.284 | 12
| Utility | 176.16±91.996 | 455.294±261.896 | 144.335±49.317 | 121.310±43.578 | 241.965±84.882 | 14
| Bottom | 86.635±19.452 | 721.889±157.714 | 127.253±59.682 | 110.750±68.850 | 184.948±96.020 | 6
| Top | 135.37±176.90 | 844.944±376.067 | 60.987±44.726 | 41.902±40.029 | 99.731±75.180 | 10
DES. | Jungle | 76.362±8.247 | 795.708±100.562 | 60.503±46.094 | 46.599±35.956 | 87.578±64.595 | 16
PLATE | Middle | 76.874±6.404 | 785.385±63.742 | 61.381±29.526 | 47.089±34.529 | 87.474±37.099 | 13
| Utility | 202.95±88.446 | 355.705±182.813 | 129.698±42.153 | 111.779±44.273 | 205.033±81.351 | 16
| Bottom | 96.659±35.939 | 691.758±214.864 | 113.291±80.127 | 51.086±25.804 | 193.149±167.07 | 11
| Top | 120.34±94.250 | 769.005±378.607 | 86.455±68.067 | 53.487±50.697 | 134.803±127.02 | 15
ALL | Jungle | 75.739±10.170 | 807.065±114.546 | 67.119±42.557 | 50.132±34.495 | 94.703±59.744 | 58
EVENTS | Middle | 83.723±29.622 | 753.243±116.137 | 68.963±35.358 | 49.553±27.323 | 100.437±58.497 | 37
| Utility | 181.76±84.367 | 421.948±236.871 | 133.272±46.371 | 111.570±48.563 | 215.473±86.246 | 41
| Bottom | 85.503±27.045 | 752.833±172.923 | 97.499±66.037 | 63.945±46.580 | 158.030±125.32 | 26
| Top | 146.07±119.30 | 684.261±387.718 | 95.172±64.519 | 70.493±56.388 | 150.569±111.94 | 41

Table 6: Role-dependent mean BPM (#), IBI (ms), SDNN (ms), SDSD (ms) and RMSSD
(ms) statistics (2 sessions) from events ”Killing”, ”Dying”, ”Placing Ward”,
”Destroying Turret” and ”Destroying Turret Plate”. N are event occurrences.
*$p<0.05$

##  4 Conclusions

This study shows the potential of using physiological measurements (EDA and
ECG) over Esports and desktop gameplay environments. In this study, we
characterized physiological responses depending on performance, events as well
as participants’ roles in the game. In most cases, we found significant
differences in EDA (for nrSCR, Amplitude, and PhasicMax activity) during
”Killing”, ”Destroying Turret” or ”Destroying Turret Plate” between players
that are winning the game and players that are losing the game. When players
were winning the game, they showed distinct patterns of physiological activity
depending on the events in the game (e.g., ”Killing”, ”Destroying Turret”,
”Destroying Plate”). In contrast, we did not find any significant difference
between these events for players that were losing the game. This can hinder
the possibility that players that perform badly show similar physiological
states across the game, while players that perform well have distinct
physiological behavior during the game. For the case of ECG, SDSD was
significantly distinct for players that were winning the game between
different events. On the other side, we found that only IBI and BPM measures
showed significant differences for players that were losing the game. Overall,
players that performed better (winning) showed significantly higher
parasympathetic activity (i.e., relaxation) than the ones that were losing.

These results suggest that poor game performance induces higher stress or to
be in a state of alert, while players that perform better tend to remain in
more relaxed states. Furthermore, the analysis for specific events, like
”Dying”, ”Destroying Turret Plate” or ”Placing Ward”, has shown that players
have distinct values of SDSD and RMSSD, with Killing” or ”Dying” events
inducing higher sympathetic activity (lower HRV).

Despite the lack of physiological samples for participants and game sessions
we obtained enough measurements to pinpoint differences in-game performance
and events. By having a higher number of participants and game sessions we
would suggest undergoing similar studies, not only for analyzing physiology
over game performance and events but also for conducting an in-depth analysis
of game roles, champions, player level, and type of match (beyond League of
Legends’ summoner’s rift) in relation with EDA and ECG measurements.

### Author Contributions

David Berga contributed to the development, data analysis, experimentation,
writing and reviewing of the paper. Alexandre Pereda contributed to the
experimental design with subjects and reviewing the paper. Eleonora de Filippi
contributed to the data analysis and statistics as well as writing and
reviewing of the paper. Arijit Nandi contributed to the development tools of
shimmer. Eulàlia Febrer, Marta Reverté and Lautaro Russo contributed to the
session management and contact with parters for the experimental setting.

### Funding

This study has been possible through the Grant IRC 2020 (Reference
ACE033/21/000046) funded by ACCIÓ (Catalan Agency for Business
Competitiveness), from the project ”eSports-LAB” lead by INDESCAT (Associació
Catalana Clúster de la Indústria de l’Esport), partners with Generalitat de
Catalunya, EsportCat and Fundació Catalana per l’Esport.

### Conflicts of Interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper. Experiment participants signed an authorization form prior to
the study to authorize the usage of the captured physiological data as well as
performance from their Riot Gamertag and remained anonymous according to the
Spanish national law LOPD (Ley Orgánica de Protección de Datos de Carácter
Personal).

## References

  * [1] M.. Ahsan, Muhammad Ibn Ibrahimy and Othman Omran Khalifa  “EMG signal classification for human computer interaction: a review”  In _European Journal of Scientific Research]_ 33, 2009, pp. 480–501
  * [2] Ashwaq Alhargan, Neil Cooke and Tareq Binjammaz  “Multimodal affect recognition in an interactive gaming environment using eye tracking and speech signals”  In _Proceedings of the 19th ACM International Conference on Multimodal Interaction_ ACM, 2017  DOI: 10.1145/3136755.3137016
  * [3] Jan K. Argasiński and Iwona Grabska-Gradzińska  “Patterns in Serious Game Design and Evaluation Application of Eye-Tracker and Biosensors”  In _Artificial Intelligence and Soft Computing_ Springer International Publishing, 2017, pp. 367–377  DOI: 10.1007/978-3-319-59060-8˙33
  * [4] Mathias Benedek and Christian Kaernbach  “A continuous measure of phasic electrodermal activity”  In _Journal of neuroscience methods_ 190.1 Elsevier, 2010, pp. 80–91
  * [5] Paris Mavromoustakos Blom, Sander Bakkes and Pieter Spronck  “Towards Multi-modal Stress Response Modelling in Competitive League of Legends”  In _2019 IEEE Conference on Games (CoG)_ IEEE, 2019  DOI: 10.1109/cig.2019.8848004
  * [6] Margaret M. Bradley and Peter J. Lang  “Measuring emotion: The self-assessment manikin and the semantic differential”  In _Journal of Behavior Therapy and Experimental Psychiatry_ 25.1 Elsevier BV, 1994, pp. 49–59  DOI: 10.1016/0005-7916(94)90063-9
  * [7] Margaret M. Bradley and Peter J. Lang  “Measuring Emotion: Behavior, Feeling, and Physiology”  In _Cognitive Neuroscience of Emotion_ Oxford University Press, 2000, pp. 25–49
  * [8] John T. Cacioppo, Hsi Yuan Chen and Stephanie Cacioppo  “Reciprocal Influences Between Loneliness and Self-Centeredness: A Cross-Lagged Panel Analysis in a Population-Based Sample of African American, Hispanic, and Caucasian Adults”  In _Personality and Social Psychology Bulletin_ 43.8 SAGE Publications, 2017, pp. 1125–1135  DOI: 10.1177/0146167217705120
  * [9] Paul Ekman  “Basic Emotions”  In _Handbook of Cognition and Emotion_ John Wiley & Sons, Ltd, 2005, pp. 45–60  DOI: 10.1002/0470013494.ch3
  * [10] Tehmina Hafeez et al.  “EEG in game user analysis: A framework for expertise classification during gameplay”  In _PLOS ONE_ 16.6 Public Library of Science (PLoS), 2021, pp. e0246913  DOI: 10.1371/journal.pone.0246913
  * [11] B.. Jones, T. Tan and S.. Bloom  “Minireview: Glucagon in Stress and Energy Homeostasis”  In _Endocrinology_ 153.3 The Endocrine Society, 2012, pp. 1049–1054  DOI: 10.1210/en.2011-1979
  * [12] J. Kivikangas et al.  “A review of the use of psychophysiological methods in game research”  In _Journal of Gaming & Virtual Worlds_ 3.3 Intellect, 2011, pp. 181–199  DOI: 10.1386/jgvw.3.3.181˙1
  * [13] Oliver Leis and Franziska Lautenbach  “Psychological and physiological stress in non-competitive and competitive esports settings: A systematic review”  In _Psychology of Sport and Exercise_ 51 Elsevier BV, 2020, pp. 101738  DOI: 10.1016/j.psychsport.2020.101738
  * [14] Iris B. Mauss and Michael D. Robinson  “Measures of emotion: A review”  In _Cognition & Emotion_ 23.2 Informa UK Limited, 2009, pp. 209–237  DOI: 10.1080/02699930802204677
  * [15] Richard E. Nisbett and Timothy D. Wilson  “The halo effect: Evidence for unconscious alteration of judgments.”  In _Journal of Personality and Social Psychology_ 35.4 American Psychological Association (APA), 1977, pp. 250–256  DOI: 10.1037/0022-3514.35.4.250
  * [16] Hugo F. Posada-Quintero, John P. Florian, Álvaro D. Orjuela-Cañón and Ki H. Chon  “Highly sensitive index of sympathetic activity based on time-frequency spectral analysis of electrodermal activity”  In _American Journal of Physiology-Regulatory, Integrative and Comparative Physiology_ 311.3 American Physiological Society, 2016, pp. R582–R591  DOI: 10.1152/ajpregu.00180.2016
  * [17] James A. Russell  “A circumplex model of affect.”  In _Journal of Personality and Social Psychology_ 39.6 American Psychological Association (APA), 1980, pp. 1161–1178  DOI: 10.1037/h0077714
  * [18] Anas Samara, Leo Galway, Raymond Bond and Hui Wang  “Affective state detection via facial expression analysis within a human computer interaction context”  In _Journal of Ambient Intelligence and Humanized Computing_ 10.6 Springer ScienceBusiness Media LLC, 2017, pp. 2175–2184  DOI: 10.1007/s12652-017-0636-8
  * [19] Fred Shaffer and J.. Ginsberg  “An Overview of Heart Rate Variability Metrics and Norms”  In _Frontiers in Public Health_ 5 Frontiers Media SA, 2017  DOI: 10.3389/fpubh.2017.00258
  * [20] Anton Smerdov et al.  “Detecting Video Game Player Burnout with the Use of Sensor Data and Machine Learning”, IEEE Internet of Things Journal, 21 April 2021, 2020  DOI: 10.1109/JIOT.2021.3074740
  * [21] Anton Smerdov, Bo Zhou, Paul Lukowicz and Andrey Somov  “Collection and Validation of Psychophysiological Data from Professional and Amateur Players: a Multimodal eSports Dataset”  In _arXiv preprint arXiv:2011.00958_ , 2020
  * [22] Gaetano Valenza, Luca Citi, J. Saul and Riccardo Barbieri  “Measures of sympathetic and parasympathetic autonomic outflow from heartbeat dynamics”  In _Journal of Applied Physiology_ 125.1 American Physiological Society, 2018, pp. 19–39  DOI: 10.1152/japplphysiol.00842.2017
  * [23] Paul Van Gent, Haneen Farah, Nicole Van Nes and Bart Van Arem  “HeartPy: A novel heart rate algorithm for the analysis of noisy signals”  In _Transportation research part F: traffic psychology and behaviour_ 66 Elsevier, 2019, pp. 368–378
  * [24] Paul Gent, Haneen Farah, Nicole Nes and Bart Arem  “Analysing noisy driver physiology real-time using off-the-shelf sensors: Heart rate analysis software from the taking the fast lane project”  In _Journal of Open Research Software_ 7.1 Ubiquity Press, 2019
  * [25] Habib Yaribeygi et al.  “The impact of stress on body function: a review”  In _EXCLI Journal; 16:Doc1057; ISSN 1611-2156_ IfADo - Leibniz Research Centre for Working EnvironmentHuman Factors, Dortmund, 2017  DOI: 10.17179/EXCLI2017-480


FPR

    false positive rate
LID

    language identification
NLP

    natural language processing
MSA

    Modern Standard Arabic
UDHR

    Universal Declaration of Human Rights

# An Open Dataset and Model for Language Identification

Laurie Burchell  Alexandra Birch  Nikolay Bogoychev  Kenneth Heafield
Institute for Language, Cognition, and Computation
School of Informatics, University of Edinburgh
10 Crichton Street, Edinburgh, EH8 9AB, UK
<EMAIL_ADDRESS>
###### Abstract

lid is a fundamental step in many natural language processing pipelines.
However, current language identification (LID) systems are far from perfect,
particularly on lower-resource languages. We present a LID model which
achieves a macro-average F1 score of 0.93 and a false positive rate of 0.033
across 201 languages, outperforming previous work. We achieve this by training
on a curated dataset of monolingual data, the reliability of which we ensure
by auditing a sample from each source and each language manually. We make both
the model and the dataset available to the research community. Finally, we
carry out detailed analysis into our model’s performance, both in comparison
to existing open models and by language class.

##  1 Introduction

lid is a foundational step in many natural language processing (NLP)
pipelines. It is used not only to select data in the relevant language but
also to exclude ‘noise’. For this reason, effective LID systems are key for
building useful and representative NLP applications.

Despite their importance, recent work has found that existing LID algorithms
perform poorly in practice compared to test performance (Caswell et al.,
2020). The problem is particularly acute for low-resource languages: Kreutzer
et al. (2022) found a positive Spearman rank correlation between quality of
data and size of language for all of the LID-filtered multilingual datasets
they studied. In addition, for a significant fraction of the language corpora
they studied, less than half of the sentences were in the correct language.
They point out that such low-quality data not only leads to poor performance
in downstream tasks, but that it also contributes to ‘representation washing’,
where the community is given a false view of the actual progress of low-
resource NLP.

For applications such as corpus filtering, LID systems need to be fast,
reliable, and cover as many languages as possible. There are several open LID
models offering quick classification and high language coverage, such as CLD3
or the work of Costa-jussà et al. (2022). However, to the best of our
knowledge, none of the commonly-used scalable LID systems make their training
data public. This paper addresses this gap through the following
contributions:

  * •

We provide a curated and open dataset covering 201 languages. We audit a
sample from each source and each language making up this dataset manually to
ensure quality.

  * •

We train a LID model on this dataset which outperforms previous open models.
We make this model publicly available.111github.com/laurieburchell/open-lid-
dataset

  * •

We analyse our model and use our findings to highlight open problems in LID
research.

##  2 Background

There is a long history of research into LID using a plethora of methods
(Jauhiainen et al., 2019). For high-coverage LID, Dunn (2020) presents a model
covering 464 languages, whilst Brown (2014) includes as many as 1366 language
varieties. Unlike our work, the training data in both cases has not been
manually checked for quality. Recent work by Adebara et al. (2022) presents a
LID system covering 517 African languages and varieties where the training
data has been curated manually. However, as far as we are aware this data is
not easily available.

Costa-jussà et al. (2022) released a substantial piece of research aiming to
improve machine translation coverage for over 200 languages. As part of this,
they provided several professionally-translated datasets for use as test and
development sets. For this reason, we use their system as our benchmark.
However, whilst they did release scripts to recreate their parallel
data,222github.com/facebookresearch/fairseq/tree/nllb they did not provide—or
even document—the monolingual data used to train their LID system, saying only
that they use “publicly available datasets” supplemented with their own
dataset NLLB-Seed. By providing an open dataset, we aim to facilitate futher
research.

##  3 Dataset

###  3.1 Data sources

We wanted to be as confident as possible that our dataset had reliable
language labels, so as to avoid the problems noted in existing corpora
(Kreutzer et al., 2022). We therefore avoided web-crawled datasets and instead
chose sources where we felt the collection methodology made it very likely
that the language labels were correct.

The majority of our source datasets were derived from news sites, Wikipedia,
or religious text, though some come from other domains (e.g. transcribed
conversations, literature, or social media). A drawback of this approach is
that most of the text is in a formal style. Further work could collect data
from a wider range of domains whilst maintaining trust in the labels. We
checked that each dataset was either under an open license for research
purposes or described as free to use. A full list of sources is given in
Appendix A, and further information including licenses is available in the
code repository accompanying this paper.

####  3.1.1 Language selection

Our initial aim was to cover the same languages present in the FLORES-200
Evaluation Benchmark333github.com/facebookresearch/flores/blob/main/flores200
so that we could use this dataset for evaluation and compare our results
directly with Costa-jussà et al. (2022). However, during the curation process,
we decided to exclude three languages.

Firstly, though Akan and Twi are both included as separate languages in
FLORES-200, Akan is actually a macrolanguage covering a language continuum
which includes Twi. Given the other languages in FLORES-200 are individual
languages, we decided to exclude Akan.

Secondly, FLORES-200 includes Modern Standard Arabic (MSA) written in Latin
script. It is true that Arabic dialects are often written in Latin characters
in informal situations (e.g. social media). However, MSA is a form of
standardised Arabic which is not usually used in informal situations. Since we
could not any find naturally-occurring training data, we excluded MSA from the
dataset.

Finally, we excluded Minangkabau in Arabic script because it is now rarely
written this way, making it difficult to find useful training
data.444omniglot.com/writing/minangkabau.htm, ethnologue.com/language/min

###  3.2 Manual audit process

The first step in our manual audit was to check and standardise language
labels, as these are often inconsistent or idiosyncratic (Kreutzer et al.,
2022). We chose to copy the language codes in Costa-jussà et al. (2022), and
reassign macrolanguage or ambiguous language codes in the data sources we
found to the dominant individual language. Whilst this resulted in more useful
data for some languages, for other languages we had to be more conservative.
For example, we originally reassigned text labelled as the macrolanguage Malay
(msa_Latn) to Standard Malay, but this led to a large drop in performance as
the former covers a very diverse set of languages.

Two of the authors then carried out a manual audit of a random sample of all data sources and languages:555Specifically, we used the following command on each file to select 500 lines to audit: ‘shuf <file> | head -n 500 | less one a native Bulgarian speaker (able to read Cyrillic and Latin scripts and Chinese characters), and the other a native English speaker (able to read Latin, Arabic and Hebrew scripts). For languages we knew, we checked the language was what we expected. For unfamiliar languages in a script we could read, we compared the sample to the Universal Declaration of Human Rights (UDHR) or failing that, to a sample of text on Wikipedia. We compared features of the text which are common in previous LID algorithms and could be identified easily by humans: similar diacritics, word lengths, common words, loan words matching the right cultural background, similar suffixes and prefixes, and vowel/consonant patterns (Jauhiainen et al., 2019, Section 5). For scripts we could not read, we checked that all lines of the sample matched the script in the UDHR.

###  3.3 Preprocessing

We kept preprocessing minimal so that the process was as language agnostic as
possible. We used the scripts provided with Moses (Koehn et al., 2007) to
remove non-printing characters and detokenise the data where necessary. We
then filtered the data so that each line contained at least one character in
the expected script (as defined by Perl) to allow for borrowings. Finally, we
followed Arivazhagan et al. (2019) and Costa-jussà et al. (2022) and sampled
proportionally to $p_{l}^{0.3}$, where $p_{l}$ is the fraction of lines in the
dataset which are in language $l$. This aims to ameliorate class skew issues.

###  3.4 Dataset description

The final dataset contains 121 million lines of data in 201 language classes.
Before sampling, the mean number of lines per language is 602,812. The
smallest class contains 532 lines of data (South Azerbaijani) and the largest
contains 7.5 million lines of data (English). There is a full breakdown of
lines of training data by language in Appendix C.

##  4 Model and hardware

We used our open dataset to train a fasttext LID model using the command-line
tool (Joulin et al., 2017). It embeds character-level n-grams from the input
text, and then uses these as input to a multiclass linear classifier. We used
the same hyperparameters as Costa-jussà et al. (2022) (NLLB), which we list in
Appendix B. We trained our model on one Ice Lake node of the CSD3 HPC service.
Each node has 76 CPUs and 256GiB of RAM. Our model takes c. 1hr 45mins to
train and contains 60.5 million parameters. Inference over the 206,448 lines
of the test set takes 22.4 secs (9216.4 lines/sec).

|  | FLORES-200∗ | FLORES200${}^{*}\cap$ NLLB | FLORES-200${}^{*}\cap$ CLD3
---|---|---|---|---
|  | 201 languages | 193 languages | 95 languages
System | Supported languages. | F1 $\uparrow$ | FPR $\downarrow$ | F1 $\uparrow$ | FPR $\downarrow$ | F1 $\uparrow$ | FPR $\downarrow$
CLD3 | 107 | - | - | - | - | 0.968 | 0.030
NLLB | 218 | - | - | 0.950 | 0.023 | 0.985 | 0.019
Our model | 201 | 0.927 | 0.033 | 0.959 | 0.020 | 0.989 | 0.011
Table 1: A comparison of open-source LID systems. Supported languages gives
the number of languages the classifier claims to support. Each column gives
the classifier’s performance on a test set containing the intersection of
languages each classifier claims to support. We report macro-averages of F1
scores and FPRs.

##  5 Evaluation

###  5.1 Test sets

We use the FLORES-200 benchmark provided by Costa-jussà et al. (2022) for
evaluation. It consists of 842 distinct web articles sourced from English-
language Wikimedia projects, with each sentence professionally translated into
204 languages. The target side is human-verified as in the right language,
making it suitable for use as a LID evaluation set. For each language, 997
sentences are available for development and 1012 for dev-test (our test
set).666992 sentences are withheld by Costa-jussà et al. (2022) as a hidden
test set. We remove the three languages discussed in Section 3.1.1 from
FLORES-200, leaving 201 languages in the test set: FLORES-200∗.

###  5.2 Other LID systems

We compare our model’s performance to two other open-source LID systems:
nllb218e (NLLB)777tinyurl.com/nllblid218e and pycld3 0.22
(CLD3).888pypi.org/project/pycld3 We discuss how we ensured a fair comparison
below.

NLLB is a fasttext model. We were surprised to discover that whilst it does
cover 218 languages, it only includes 193 of the 201 languages in FLORES-200∗.
This is despite the fact that the NLLB LID model and the original FLORES-200
evaluation set were created as part of the same work (Costa-jussà et al.,
2022). Referring to the analysis in the original paper, the authors note that
“Arabic languoids and Akan/Twi have been merged after linguistic analysis”
(Costa-jussà et al., 2022, Table 5, p. 32). We discuss the reason to merge
Akan and Twi in Section 3.1.1, but we judge Arabic dialects to be close but
distinct languages. Our model performs poorly on Arabic dialects with the
highest F1 score only 0.4894 (Moroccan Arabic). This is likely due to the
general difficulty of distinguishing close languages combined with
particularly sparse training data. We assume these poor results led to Arabic
dialects (save MSA) being excluded from the NLLB LID classifier. We remove
eight Arabic dialects from the test set when comparing our model and NLLB,
leaving 193 languages.

CLD3 is an n-gram based neural network model for LID. It uses different
language codes to the other two models, so we normalise all predictions to
BCP-47 macrolanguage codes to allow fair comparison. We test on the 95
languages that all models have in common after normalisation.

##  6 Results

Our results are given in Table 1. We evaluate all models using F1 scores and
false positive rate (FPR). We report macro-averages to avoid down-weighting
low-resource languages (Kreutzer et al., 2022). Following Caswell et al.
(2020), we report FPR to give a better indication of real-world performance
when there is significant class skew.

We achieve an F1 score of 0.927 and a FPR of 0.033 on FLORES-200∗. We also
outperform both NLLB and CLD3 on the mutual subsets of FLORES-200∗. Since NLLB
and our model share the same architecture and the same parameters, we
attribute our success to our training data selection and manual audit process.

Notably, our F1 score jumps to 0.959 and FPR falls to 0.020 when we exclude
the eight Arabic dialects from the test set to compare with NLLB. The 95
languages covered by CLD3, NLLB, and our model are mostly high resource, and
so it is unsurprising that we achieve the highest F1 score (0.989) and lowest
FPR (0.011) on this subset.

We notice that the Pearson correlation between the number of lines of training
data and F1 score for each language is only 0.0242. This is not unexpected:
some of the least resourced languages achieve perfect scores on the test set
due to high domain overlap, whereas the higher-resourced languages might get
lower scores on the test set but have better robustness across domains. Full
results by language are available in Appendix C.

###  6.1 Performance by language category

Using the taxonomy and list of languages in Joshi et al. (2020), we label each
of the languages in our dataset according to its level of data availability (0
= least resourced, 5 = best resourced). We leave out 5 languages missing from
the taxonomy, plus the 8 Arabic dialects not covered by NLLB. Table 2 compares
the mean F1 score and FPR of our model and for that of Costa-jussà et al.
(2022) (NLLB). Our model has a higher or equal F1 score in every category and
a lower or equal FPR in every category but one, showing our model’s improved
performance across languages with different amounts of available data.

We note that class zero (the least-resourced languages) shows the smallest
change in performance. We speculate that this is an artifact of the curation
of our training dataset. For the best-resourced languages with more sources to
choose from, it is likely that there is a significant difference between our
training data and that used to train the model in Costa-jussà et al. (2022).
However, for the least-resourced languages, the sheer lack of resources means
that overlap between our data and that used by Costa-jussà et al. (2022) is
more likely. We suspect this is the reason we see little difference in
performance for class zero in Table 2. Unfortunately, without access to the
training data used to train NLLB, we cannot verify this assumption.

|  | F1 $\uparrow$ | FPR $\downarrow$
---|---|---|---
Class | Count | Ours | NLLB | Ours | NLLB
0 | 28 | 0.900 | 0.897 | 0.014 | 0.013
1 | 94 | 0.981 | 0.968 | 0.013 | 0.013
2 | 16 | 0.990 | 0.963 | 0.009 | 0.043
3 | 25 | 0.983 | 0.974 | 0.007 | 0.013
4 | 18 | 0.951 | 0.951 | 0.051 | 0.055
5 | 7 | 0.897 | 0.855 | 0.163 | 0.620
Table 2: For each language class in the taxonomy of Joshi et al. (2020), we
give the count of the languages covered by the classifier in that class, mean
F1 score, and mean FPR for our model and for that of Costa-jussà et al. (2022)
(NLLB). 0–5 = least to best resourced.

###  6.2 Case study: Chinese languages

Despite our model outperforming NLLB overall, NLLB achieved a noticeably
higher F1 score on Yue Chinese (0.488 vs. 0.006). Figure 1 shows the confusion
matrices for our model and NLLB between the three Chinese languages. Our model
performs well on Simplified and Traditional Chinese, but almost never predicts
Yue Chinese, instead classifying it as Chinese (Traditional). The NLLB model
is also unable to distinguish between Yue and Chinese (Traditional), but mixes
the two classes instead.

We asked four native speakers to inspect our training data and the FLORES-200
test set. They noted that there was a mismatch in domain for Yue Chinese, as
much of our training data was written colloquial Yue Chinese whereas the test
set consisted of formal writing. Furthermore, they were unable to distinguish
with high confidence between Yue and Chinese (Traditional) as the two
languages are very similar when written formally. This is an example of a
wider problem with LID: the language covered by a particular label may vary
widely, making single-label classification difficult.

Figure 1: Confusion matrices for our model (L) and NLLB (R), showing the
confusion in classification by each model on the FLORES-200 test set between
Chinese (Simplified) (zho_Hans), Chinese (Traditional) (zho_Hant), and Yue
Chinese (yue_Hant) classes.

##  7 Conclusion

We present an open dataset covering 201 languages, which we curate and audit
manually to ensure high confidence in its data and language labels. We
demonstrate the quality of our dataset by using it to train a high-performing
and scalable LID model. Finally, we provide detailed analysis into its
performance by class. We make both our model and our dataset available to the
research community.

## Limitations

Our dataset and model only covers 201 languages: the ones we were able to test
with the FLORES-200 Evaluation Benchmark. In addition, because our test set
consists of sentences from a single domain (wiki articles), performance on
this test set may not reflect how well our classifier works in other domains.
Future work could create a LID test set representative of web data where these
classifiers are often applied. Finally, most of the data was not audited by
native speakers as would be ideal. Future versions of this dataset should have
more languages verified by native speakers, with a focus on the least
resourced languages.

## Ethics Statement

Our work aims to broaden NLP coverage by allowing practitioners to identify
relevant data in more languages. However, we note that LID is inherently a
normative activity that risks excluding minority dialects, scripts, or entire
microlanguages from a macrolanguage. Choosing which languages to cover may
reinforce power imbalances, as only some groups gain access to NLP
technologies.

In addition, errors in LID can have a significant impact on downstream
performance, particularly (as is often the case) when a system is used as a
‘black box’. The performance of our classifier is not equal across languages
which could lead to worse downstream performance for particular groups. We
mitigate this by providing metrics by class.

## Acknowledgements

This work was supported in part by the UKRI Centre for Doctoral Training in
Natural Language Processing, funded by the UKRI (grant EP/S022481/1) and the
University of Edinburgh, School of Informatics and School of Philosophy,
Psychology & Language Sciences.

The experiments in this paper were performed using resources provided by the
Cambridge Service for Data Driven Discovery (CSD3) operated by the University
of Cambridge Research Computing Service (www.csd3.cam.ac.uk), provided by Dell
EMC and Intel using Tier-2 funding from the Engineering and Physical Sciences
Research Council (capital grant EP/P020259/1), and DiRAC funding from the
Science and Technology Facilities Council (www.dirac.ac.uk).

Special thanks to Pinzhen Chen, Steven Chien, Bryan Li, Lushi Chen and
Victoria Lee for their help with Chinese languages.

## References

  * Abu Kwaik et al. (2018) Kathrein Abu Kwaik, Motaz Saad, Stergios Chatzikyriakidis, and Simon Dobnik. 2018\.  Shami: A corpus of Levantine Arabic dialects.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Adebara et al. (2022) Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Alcides Alcoba Inciarte. 2022.  Afrolid: A neural language identification tool for african languages.  _arXiv preprint arXiv:2210.11744_.
  * Agić and Vulić (2019) Željko Agić and Ivan Vulić. 2019.  JW300: A wide-coverage parallel corpus for low-resource languages.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3204–3210, Florence, Italy. Association for Computational Linguistics.
  * Alsarsour et al. (2018) Israa Alsarsour, Esraa Mohamed, Reem Suwaileh, and Tamer Elsayed. 2018.  DART: A large dataset of dialectal Arabic tweets.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Arivazhagan et al. (2019) Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019\.  Massively multilingual neural machine translation in the wild: Findings and challenges.  _arXiv preprint arXiv:1907.05019_.
  * Barrault et al. (2020) Loïc Barrault, Magdalena Biesialska, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. 2020.  Findings of the 2020 conference on machine translation (WMT20).  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 1–55, Online. Association for Computational Linguistics.
  * Barrault et al. (2019) Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019.  Findings of the 2019 conference on machine translation (WMT19).  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 1–61, Florence, Italy. Association for Computational Linguistics.
  * Bojar et al. (2013) Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013.  Findings of the 2013 Workshop on Statistical Machine Translation.  In _Proceedings of the Eighth Workshop on Statistical Machine Translation_ , pages 1–44, Sofia, Bulgaria. Association for Computational Linguistics.
  * Bojar et al. (2014) Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014.  Findings of the 2014 workshop on statistical machine translation.  In _Proceedings of the Ninth Workshop on Statistical Machine Translation_ , pages 12–58, Baltimore, Maryland, USA. Association for Computational Linguistics.
  * Bojar et al. (2017) Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017.  Findings of the 2017 conference on machine translation (WMT17).  In _Proceedings of the Second Conference on Machine Translation_ , pages 169–214, Copenhagen, Denmark. Association for Computational Linguistics.
  * Bojar et al. (2016) Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016.  Findings of the 2016 conference on machine translation.  In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_ , pages 131–198, Berlin, Germany. Association for Computational Linguistics.
  * Bojar et al. (2015) Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015\.  Findings of the 2015 workshop on statistical machine translation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 1–46, Lisbon, Portugal. Association for Computational Linguistics.
  * Bojar et al. (2018) Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018.  Findings of the 2018 conference on machine translation (WMT18).  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 272–303, Belgium, Brussels. Association for Computational Linguistics.
  * Bouamor et al. (2019) Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019.  The MADAR shared task on Arabic fine-grained dialect identification.  In _Proceedings of the Fourth Arabic Natural Language Processing Workshop_ , pages 199–207, Florence, Italy. Association for Computational Linguistics.
  * Brown (2014) Ralf Brown. 2014.  Non-linear mapping for improved identification of 1300+ languages.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 627–632, Doha, Qatar. Association for Computational Linguistics.
  * Brown (2012) Ralf D Brown. 2012.  Finding and identifying text in 900+ languages.  _Digital Investigation_ , 9:S34–S43.
  * Caswell et al. (2020) Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna. 2020.  Language ID in the wild: Unexpected challenges on the path to a thousand-language web text corpus.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6588–6608, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Costa-jussà et al. (2022) Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022.  No Language Left Behind: Scaling Human-Centered Machine Translation.  _arXiv preprint arXiv:2207.04672_.
  * Dunn (2020) Jonathan Dunn. 2020.  Mapping languages: The corpus of global language use.  _Language Resources and Evaluation_ , 54(4):999–1018.
  * El-Haj et al. (2018) Mahmoud El-Haj, Paul Rayson, and Mariam Aboelezz. 2018.  Arabic dialect identification in the context of bivalency and code-switching.  In _Proceedings of the 11th International Conference on Language Resources and Evaluation, Miyazaki, Japan._ , pages 3622–3627. European Language Resources Association.
  * Esplà et al. (2019) Miquel Esplà, Mikel Forcada, Gema Ramírez-Sánchez, and Hieu Hoang. 2019\.  ParaCrawl: Web-scale parallel corpora for the languages of the EU.  In _Proceedings of Machine Translation Summit XVII: Translator, Project and User Tracks_ , pages 118–119, Dublin, Ireland. European Association for Machine Translation.
  * Goldhahn et al. (2012) Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff. 2012.  Building large monolingual dictionaries at the Leipzig corpora collection: From 100 to 200 languages.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 759–765, Istanbul, Turkey. European Language Resources Association (ELRA).
  * Góngora et al. (2022) Santiago Góngora, Nicolás Giossa, and Luis Chiruzzo. 2022.  Can we use word embeddings for enhancing Guarani-Spanish machine translation? In _Proceedings of the Fifth Workshop on the Use of Computational Methods in the Study of Endangered Languages_ , pages 127–132, Dublin, Ireland. Association for Computational Linguistics.
  * Gowda et al. (2021) Thamme Gowda, Zhao Zhang, Chris Mattmann, and Jonathan May. 2021.  Many-to-English machine translation tools, data, and pretrained models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_ , pages 306–316, Online. Association for Computational Linguistics.
  * Hasan et al. (2021) Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021.  XL-sum: Large-scale multilingual abstractive summarization for 44 languages.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4693–4703, Online. Association for Computational Linguistics.
  * Huidrom et al. (2021) Rudali Huidrom, Yves Lepage, and Khogendra Khomdram. 2021.  EM corpus: a comparable corpus for a less-resourced language pair Manipuri-English.  In _Proceedings of the 14th Workshop on Building and Using Comparable Corpora (BUCC 2021)_ , pages 60–67, Online (Virtual Mode). INCOMA Ltd.
  * Jauhiainen et al. (2019) Tommi Jauhiainen, Marco Lui, Marcos Zampieri, Timothy Baldwin, and Krister Lindén. 2019.  Automatic language identification in texts: A survey.  _Journal of Artificial Intelligence Research_ , 65:675–782.
  * Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020.  The state and fate of linguistic diversity and inclusion in the NLP world.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6282–6293, Online. Association for Computational Linguistics.
  * Joulin et al. (2017) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017.  Bag of tricks for efficient text classification.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 427–431, Valencia, Spain. Association for Computational Linguistics.
  * Kashefi (2018) Omid Kashefi. 2018.  Mizan: A large persian-english parallel corpus.  _arXiv preprint arXiv:1801.02107_.
  * Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007\.  Moses: Open source toolkit for statistical machine translation.  In _Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions_ , pages 177–180, Prague, Czech Republic. Association for Computational Linguistics.
  * Kreutzer et al. (2022) Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. 2022.  Quality at a glance: An audit of web-crawled multilingual datasets.  _Transactions of the Association for Computational Linguistics_ , 10:50–72.
  * Kunchukuttan et al. (2018) Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhattacharyya. 2018.  The IIT Bombay English-Hindi parallel corpus.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Luke and Wong (2015) Kang Kwong Luke and May LY Wong. 2015.  The hong kong cantonese corpus: design and uses.  _Journal of Chinese Linguistics Monograph Series_ , 1(25):312–333.
  * Medhaffar et al. (2017) Salima Medhaffar, Fethi Bougares, Yannick Estève, and Lamia Hadrich-Belguith. 2017.  Sentiment analysis of Tunisian dialects: Linguistic ressources and experiments.  In _Proceedings of the Third Arabic Natural Language Processing Workshop_ , pages 55–61, Valencia, Spain. Association for Computational Linguistics.
  * Meftouh et al. (2015) Karima Meftouh, Salima Harrat, Salma Jamoussi, Mourad Abbas, and Kamel Smaili. 2015\.  Machine translation experiments on PADIC: A parallel Arabic DIalect corpus.  In _Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation_ , pages 26–34, Shanghai, China.
  * Mirzakhalov et al. (2021) Jamshidbek Mirzakhalov, Anoop Babu, Duygu Ataman, Sherzod Kariev, Francis Tyers, Otabek Abduraufov, Mammad Hajili, Sardana Ivanova, Abror Khaytbaev, Antonio Laverghetta Jr., Bekhzodbek Moydinboyev, Esra Onal, Shaxnoza Pulatova, Ahsan Wahab, Orhan Firat, and Sriram Chellappan. 2021.  A large-scale study of machine translation in Turkic languages.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5876–5890, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Ojha (2019) Atul Kr Ojha. 2019.  English-bhojpuri smt system: Insights from the karaka model.  _arXiv preprint arXiv:1905.02239_.
  * Pilevar et al. (2011) Mohammad Taher Pilevar, Heshaam Faili, and Abdol Hamid Pilevar. 2011.  Tep: Tehran english-persian parallel corpus.  In _International Conference on Intelligent Text Processing and Computational Linguistics_ , pages 68–79. Springer.
  * Post et al. (2012) Matt Post, Chris Callison-Burch, and Miles Osborne. 2012.  Constructing parallel corpora for six Indian languages via crowdsourcing.  In _Proceedings of the Seventh Workshop on Statistical Machine Translation_ , pages 401–409, Montréal, Canada. Association for Computational Linguistics.
  * Qi et al. (2018) Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018\.  When and why are pre-trained word embeddings useful for neural machine translation? In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 529–535, New Orleans, Louisiana. Association for Computational Linguistics.
  * Rozis and Skadiņš (2017) Roberts Rozis and Raivis Skadiņš. 2017.  Tilde MODEL - multilingual open data for EU languages.  In _Proceedings of the 21st Nordic Conference on Computational Linguistics_ , pages 263–265, Gothenburg, Sweden. Association for Computational Linguistics.
  * Thoma (2018) Martin Thoma. 2018.  The wili benchmark dataset for written language identification.  _arXiv preprint arXiv:1801.07779_.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in OPUS.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA).
  * Zahir (2022) Jihad Zahir. 2022.  Iadd: An integrated arabic dialect identification dataset.  _Data in Brief_ , 40:107777.
  * Zaidan and Callison-Burch (2011) Omar F. Zaidan and Chris Callison-Burch. 2011.  The Arabic online commentary dataset: an annotated dataset of informal Arabic with high dialectal content.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 37–41, Portland, Oregon, USA. Association for Computational Linguistics.
  * Zhang et al. (2020) Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020.  Improving massively multilingual neural machine translation and zero-shot translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1628–1639, Online. Association for Computational Linguistics.
  * Ziemski et al. (2016) Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016.  The United Nations parallel corpus v1.0.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 3530–3534, Portorož, Slovenia. European Language Resources Association (ELRA).

##  Appendix A Data sources

We use the following data sources to build our open dataset. We chose sources
as those which were likely to have trustworthy language labels and which did
not rely on other LID systems for labelling.

  * •

Arabic Dialects Dataset (El-Haj et al., 2018)

  * •

Bhojpuri Language Technological Resources Project (BLTR) (Ojha, 2019)

  * •

Global Voices (Tiedemann, 2012)

  * •

Guaraní Parallel Set (Góngora et al., 2022)

  * •

The Hong Kong Cantonese corpus (HKCanCor) (Luke and Wong, 2015)

  * •

Integrated dataset for Arabic Dialect Identification (IADD) (Zahir, 2022;
Alsarsour et al., 2018; Abu Kwaik et al., 2018; Medhaffar et al., 2017;
Meftouh et al., 2015; Zaidan and Callison-Burch, 2011)

  * •

Leipzig Corpora Collection (Goldhahn et al., 2012)

  * •

LTI LangID Corpus (Brown, 2012)

  * •

MADAR 2019 Shared Task on Arabic Fine-grained Dialect Identification (Bouamor
et al., 2019)

  * •

EM corpus (Huidrom et al., 2021)

  * •

MIZAN (Kashefi, 2018)

  * •

MT-560 (Gowda et al., 2021; Tiedemann, 2012; Post et al., 2012; Ziemski et
al., 2016; Rozis and Skadiņš, 2017; Kunchukuttan et al., 2018; Agić and Vulić,
2019; Esplà et al., 2019; Qi et al., 2018; Zhang et al., 2020; Bojar et al.,
2013, 2014, 2015, 2016, 2017, 2018; Barrault et al., 2019, 2020)

  * •

NLLB Seed (Costa-jussà et al., 2022)

  * •

SETIMES news corpus (Tiedemann, 2012)

  * •

Tatoeba collection (Tiedemann, 2012)

  * •

Tehran English-Persian Parallel (TEP) Corpus (Pilevar et al., 2011)

  * •

Turkish Interlingua (TIL) corpus (Mirzakhalov et al., 2021)

  * •

WiLI benchmark dataset (Thoma, 2018)

  * •

XL-Sum summarisation dataset (Hasan et al., 2021)

##  Appendix B LID model hyperparameters

  * •

Loss: softmax

  * •

Epochs: 2

  * •

Learning rate: 0.8

  * •

Embedding dimension: 256

  * •

Minimum number of word occurences: 1000

  * •

Character n-grams: 2–5

  * •

Word n-grams: 1

  * •

Bucket size: 1,000,000

  * •

Threads: 68

All other hyperparameters are set to fasttext defaults.

##  Appendix C Performance of our LID model by language

|  |  | Our model | NLLB
---|---|---|---|---
Language code | Language | Training data |  F1 score $\uparrow$ |  FPR $\downarrow$ |  F1 score $\uparrow$ |  FPR $\downarrow$
ace_Arab | Acehnese | 6191 | 0.9679 | 0.0079 | 0.9704 | 0.0074
ace_Latn | Acehnese | 18032 | 0.9980 | 0.0005 | 0.9936 | 0.0035
acm_Arab | Mesopotamian Arabic | 4862 | 0.0328 | 0.0040 | - | -
acq_Arab | Ta’izzi-Adeni Arabic | 1598 | 0.0020 | 0.0000 | - | -
aeb_Arab | Tunisian Arabic | 18758 | 0.3398 | 0.0479 | - | -
afr_Latn | Afrikaans | 1045638 | 0.9995 | 0.0000 | 0.9985 | 0.0010
ajp_Arab | South Levantine Arabic | 28190 | 0.1906 | 0.0158 | - | -
als_Latn | Tosk Albanian | 506379 | 1.0000 | 0.0000 | 0.9980 | 0.0020
amh_Ethi | Amharic | 606866 | 0.9995 | 0.0005 | 0.9990 | 0.0010
apc_Arab | North Levantine Arabic | 67952 | 0.2334 | 0.0983 | - | -
arb_Arab | Modern Standard Arabic | 7000000 | 0.3077 | 1.1280 | 0.1903 | 4.2579
ars_Arab | Najdi Arabic | 23194 | 0.0184 | 0.1374 | - | -
ary_Arab | Moroccan Arabic | 25411 | 0.4894 | 0.7643 | - | -
arz_Arab | Egyptian Arabic | 52327 | 0.4235 | 1.0875 | - | -
asm_Beng | Assamese | 161726 | 1.0000 | 0.0000 | 1.0000 | 0.0000
ast_Latn | Asturian | 35815 | 0.9901 | 0.0045 | 0.9902 | 0.0069
awa_Deva | Awadhi | 4957 | 0.6770 | 0.0040 | 0.9611 | 0.0084
ayr_Latn | Central Aymara | 142628 | 1.0000 | 0.0000 | 0.9980 | 0.0005
azb_Arab | South Azerbaijani | 532 | 0.7514 | 0.0000 | 0.8805 | 0.0069
azj_Latn | North Azerbaijani | 462672 | 0.9990 | 0.0005 | 0.9970 | 0.0030
bak_Cyrl | Bashkir | 65942 | 1.0000 | 0.0000 | 0.9990 | 0.0005
bam_Latn | Bambara | 9538 | 0.6107 | 0.4926 | 0.6194 | 0.4826
ban_Latn | Balinese | 15404 | 0.9789 | 0.0015 | 0.9712 | 0.0030
bel_Cyrl | Belarusian | 84846 | 1.0000 | 0.0000 | 1.0000 | 0.0000
bem_Latn | Bemba | 383559 | 0.9796 | 0.0193 | 0.9739 | 0.0252
ben_Beng | Bengali | 490226 | 0.9925 | 0.0000 | 0.9995 | 0.0005
bho_Deva | Bhojpuri | 69367 | 0.8921 | 0.1136 | 0.9335 | 0.0153
bjn_Arab | Banjar | 6192 | 0.9604 | 0.0257 | 0.9524 | 0.0163
bjn_Latn | Banjar | 21475 | 0.9857 | 0.0064 | 0.8336 | 0.1721
bod_Tibt | Standard Tibetan | 2514 | 0.8045 | 0.0000 | 0.9637 | 0.0366
bos_Latn | Bosnian | 330473 | 0.6928 | 0.0939 | 0.5954 | 0.0584
bug_Latn | Buginese | 7527 | 0.9970 | 0.0005 | 0.9765 | 0.0054
bul_Cyrl | Bulgarian | 610545 | 1.0000 | 0.0000 | 0.9995 | 0.0000
cat_Latn | Catalan | 115963 | 1.0000 | 0.0000 | 0.9873 | 0.0129
ceb_Latn | Cebuano | 1002342 | 0.9995 | 0.0005 | 0.9995 | 0.0000
ces_Latn | Czech | 424828 | 0.9975 | 0.0015 | 0.9990 | 0.0010
cjk_Latn | Chokwe | 36244 | 0.9023 | 0.0025 | 0.8688 | 0.0089
ckb_Arab | Central Kurdish | 17792 | 1.0000 | 0.0000 | 1.0000 | 0.0000
crh_Latn | Crimean Tatar | 19148 | 0.9920 | 0.0005 | 0.9829 | 0.0000
cym_Latn | Welsh | 98719 | 1.0000 | 0.0000 | 1.0000 | 0.0000
dan_Latn | Danish | 2789406 | 0.9881 | 0.0035 | 0.9946 | 0.0020
deu_Latn | German | 653914 | 1.0000 | 0.0000 | 0.9907 | 0.0094
dik_Latn | Southwestern Dinka | 25911 | 0.9995 | 0.0000 | 0.9925 | 0.0000
dyu_Latn | Dyula | 17351 | 0.0421 | 0.0282 | 0.0480 | 0.0228
dzo_Tibt | Dzongkha | 6899 | 0.8585 | 0.1635 | 0.9679 | 0.0005
ell_Grek | Greek | 3312774 | 1.0000 | 0.0000 | 1.0000 | 0.0000
eng_Latn | English | 7544560 | 0.9941 | 0.0049 | 0.9792 | 0.0213
epo_Latn | Esperanto | 339280 | 1.0000 | 0.0000 | 0.9970 | 0.0030
est_Latn | Estonian | 3331470 | 0.9990 | 0.0005 | 0.9985 | 0.0015
eus_Latn | Basque | 622029 | 0.9990 | 0.0005 | 0.9985 | 0.0015
ewe_Latn | Ewe | 585267 | 0.9980 | 0.0020 | 0.9970 | 0.0030
fao_Latn | Faroese | 40022 | 1.0000 | 0.0000 | 0.5052 | 0.0000
fij_Latn | Fijian | 360981 | 0.9985 | 0.0005 | 1.0000 | 0.0000
fin_Latn | Finnish | 2613970 | 0.9995 | 0.0005 | 0.9995 | 0.0005
fon_Latn | Fon | 31875 | 0.9980 | 0.0000 | 0.9970 | 0.0000
fra_Latn | French | 586938 | 0.9950 | 0.0000 | 0.9961 | 0.0035
fur_Latn | Friulian | 55622 | 0.9985 | 0.0015 | 0.9980 | 0.0000
fuv_Latn | Nigerian Fulfulde | 14419 | 0.9865 | 0.0005 | 0.9810 | 0.0040
gaz_Latn | West Central Oromo | 335769 | 0.9990 | 0.0010 | 0.9995 | 0.0005
gla_Latn | Scottish Gaelic | 52665 | 0.9975 | 0.0025 | 0.9985 | 0.0010
gle_Latn | Irish | 211460 | 1.0000 | 0.0000 | 0.9980 | 0.0020
glg_Latn | Galician | 42017 | 0.9970 | 0.0025 | 0.9931 | 0.0049
grn_Latn | Guarani | 57458 | 0.9975 | 0.0025 | 0.9965 | 0.0015
guj_Gujr | Gujarati | 836618 | 1.0000 | 0.0000 | 1.0000 | 0.0000
hat_Latn | Haitian Creole | 299853 | 0.9970 | 0.0030 | 0.9985 | 0.0005
hau_Latn | Hausa | 347741 | 0.9893 | 0.0109 | 0.9970 | 0.0025
heb_Hebr | Hebrew | 944918 | 0.9990 | 0.0010 | 1.0000 | 0.0000
hin_Deva | Hindi | 1089471 | 0.8477 | 0.1749 | 0.8722 | 0.1454
hne_Deva | Chhattisgarhi | 52819 | 0.9362 | 0.0311 | 0.9300 | 0.0134
hrv_Latn | Croatian | 832967 | 0.7441 | 0.1863 | 0.7335 | 0.2645
hun_Latn | Hungarian | 2870535 | 1.0000 | 0.0000 | 0.9926 | 0.0074
hye_Armn | Armenian | 368832 | 1.0000 | 0.0000 | 1.0000 | 0.0000
ibo_Latn | Igbo | 491594 | 0.9995 | 0.0005 | 0.9995 | 0.0005
ilo_Latn | Ilocano | 976648 | 0.9990 | 0.0010 | 0.9985 | 0.0015
ind_Latn | Indonesian | 1694230 | 0.9279 | 0.0435 | 0.8198 | 0.2087
isl_Latn | Icelandic | 43554 | 1.0000 | 0.0000 | 0.7621 | 0.3125
ita_Latn | Italian | 479663 | 0.9940 | 0.0000 | 0.9721 | 0.0282
jav_Latn | Javanese | 65595 | 0.9917 | 0.0079 | 0.9767 | 0.0218
jpn_Jpan | Japanese | 876783 | 1.0000 | 0.0000 | 0.9808 | 0.0104
kab_Latn | Kabyle | 52634 | 0.8551 | 0.1695 | 0.8579 | 0.1652
kac_Latn | Jingpho | 11365 | 1.0000 | 0.0000 | 1.0000 | 0.0000
kam_Latn | Kamba | 52674 | 0.9001 | 0.0005 | 0.7581 | 0.0010
kan_Knda | Kannada | 357780 | 1.0000 | 0.0000 | 1.0000 | 0.0000
kas_Arab | Kashmiri | 6203 | 0.9839 | 0.0000 | 0.9710 | 0.0000
kas_Deva | Kashmiri | 6694 | 0.9860 | 0.0010 | 0.9840 | 0.0005
kat_Geor | Georgian | 417604 | 1.0000 | 0.0000 | 1.0000 | 0.0000
kaz_Cyrl | Kazakh | 51577 | 0.9995 | 0.0000 | 0.9995 | 0.0000
kbp_Latn | Kabiye | 53275 | 1.0000 | 0.0000 | 1.0000 | 0.0000
kea_Latn | Kabuverdianu | 5665 | 0.9652 | 0.0000 | 0.9610 | 0.0000
khk_Cyrl | Halh Mongolian | 168540 | 1.0000 | 0.0000 | 1.0000 | 0.0000
khm_Khmr | Khmer | 60513 | 0.9995 | 0.0000 | 0.9990 | 0.0000
kik_Latn | Kikuyu | 96402 | 0.9628 | 0.0376 | 0.9636 | 0.0341
kin_Latn | Kinyarwanda | 447057 | 0.8872 | 0.0069 | 0.9788 | 0.0119
kir_Cyrl | Kyrgyz | 372399 | 1.0000 | 0.0000 | 1.0000 | 0.0000
kmb_Latn | Kimbundu | 92635 | 0.9394 | 0.0534 | 0.9361 | 0.0514
kmr_Latn | Northern Kurdish | 15490 | 0.9985 | 0.0010 | 0.9956 | 0.0045
knc_Arab | Central Kanuri | 6196 | 0.7017 | 0.0000 | 0.7026 | 0.0000
knc_Latn | Central Kanuri | 6256 | 0.9990 | 0.0005 | 0.9965 | 0.0015
kon_Latn | Kikongo | 209801 | 0.9946 | 0.0045 | 0.9936 | 0.0049
kor_Hang | Korean | 1772136 | 1.0000 | 0.0000 | 0.9961 | 0.0040
lao_Laoo | Lao | 23529 | 1.0000 | 0.0000 | 0.9995 | 0.0000
lij_Latn | Ligurian | 28641 | 0.9980 | 0.0015 | 0.9774 | 0.0025
lim_Latn | Limburgish | 48151 | 0.9965 | 0.0015 | 0.9870 | 0.0010
lin_Latn | Lingala | 546344 | 0.9990 | 0.0010 | 0.9956 | 0.0030
lit_Latn | Lithuanian | 2663659 | 0.9985 | 0.0010 | 0.9990 | 0.0010
lmo_Latn | Lombard | 35402 | 0.9975 | 0.0020 | 0.9696 | 0.0109
ltg_Latn | Latgalian | 15585 | 0.9985 | 0.0000 | 0.9920 | 0.0000
ltz_Latn | Luxembourgish | 37674 | 0.9995 | 0.0000 | 0.9995 | 0.0000
lua_Latn | Luba-Kasai | 292972 | 0.9960 | 0.0005 | 0.9936 | 0.0035
lug_Latn | Ganda | 251105 | 0.9941 | 0.0045 | 0.9921 | 0.0069
luo_Latn | Luo | 138159 | 0.9985 | 0.0015 | 0.9975 | 0.0005
lus_Latn | Mizo | 195262 | 0.9985 | 0.0000 | 0.9945 | 0.0005
lvs_Latn | Standard Latvian | 2872096 | 0.9990 | 0.0005 | 0.9936 | 0.0064
mag_Deva | Magahi | 6208 | 0.9620 | 0.0133 | 0.9311 | 0.0213
mai_Deva | Maithili | 15385 | 0.9880 | 0.0010 | 0.9871 | 0.0040
mal_Mlym | Malayalam | 379786 | 1.0000 | 0.0000 | 1.0000 | 0.0000
mar_Deva | Marathi | 1017951 | 0.9990 | 0.0010 | 0.9951 | 0.0049
min_Latn | Minangkabau | 31469 | 0.9931 | 0.0030 | 0.5143 | 0.0010
mkd_Cyrl | Macedonian | 561725 | 0.9995 | 0.0005 | 1.0000 | 0.0000
mlt_Latn | Maltese | 2219213 | 0.9985 | 0.0015 | 0.9995 | 0.0005
mni_Beng | Meitei | 47146 | 0.9941 | 0.0059 | 0.9995 | 0.0000
mos_Latn | Mossi | 197187 | 0.9814 | 0.0005 | 0.9684 | 0.0000
mri_Latn | Maori | 48792 | 0.9995 | 0.0005 | 0.9985 | 0.0005
mya_Mymr | Burmese | 452194 | 1.0000 | 0.0000 | 1.0000 | 0.0000
nld_Latn | Dutch | 2929602 | 0.9970 | 0.0015 | 0.9830 | 0.0173
nno_Latn | Norwegian Nynorsk | 101140 | 0.9828 | 0.0104 | 0.9697 | 0.0208
nob_Latn | Norwegian Bokmal | 1783598 | 0.9719 | 0.0148 | 0.9829 | 0.0139
npi_Deva | Nepali | 60345 | 0.9980 | 0.0020 | 0.9980 | 0.0020
nso_Latn | Northern Sotho | 560068 | 0.9868 | 0.0119 | 0.9839 | 0.0134
nus_Latn | Nuer | 6295 | 0.9995 | 0.0000 | 0.9980 | 0.0015
nya_Latn | Nyanja | 789078 | 0.9966 | 0.0035 | 0.9460 | 0.0163
oci_Latn | Occitan | 32683 | 0.9941 | 0.0054 | 0.9835 | 0.0163
ory_Orya | Odia | 92355 | 1.0000 | 0.0000 | 1.0000 | 0.0000
pag_Latn | Pangasinan | 294618 | 0.9990 | 0.0005 | 0.9970 | 0.0010
pan_Guru | Eastern Panjabi | 357487 | 1.0000 | 0.0000 | 1.0000 | 0.0000
pap_Latn | Papiamento | 403991 | 0.9768 | 0.0232 | 0.9839 | 0.0158
pbt_Arab | Southern Pasto | 63256 | 0.9980 | 0.0015 | 0.9970 | 0.0010
pes_Arab | Western Persian | 1758215 | 0.5570 | 0.5356 | 0.6385 | 0.4381
plt_Latn | Plateau Malgasy | 47284 | 1.0000 | 0.0000 | 1.0000 | 0.0000
pol_Latn | Polish | 3403455 | 0.9956 | 0.0045 | 0.9849 | 0.0153
por_Latn | Portuguese | 3800360 | 0.9941 | 0.0040 | 0.9854 | 0.0143
prs_Arab | Dari | 6662 | 0.5144 | 0.1122 | 0.4589 | 0.0608
quy_Latn | Ayacucho Quechua | 154448 | 1.0000 | 0.0000 | 1.0000 | 0.0000
ron_Latn | Romanian | 443200 | 0.9985 | 0.0015 | 0.9985 | 0.0015
run_Latn | Rundi | 459617 | 0.9044 | 0.0973 | 0.9782 | 0.0104
rus_Cyrl | Russian | 7000000 | 0.9990 | 0.0005 | 0.9990 | 0.0010
sag_Latn | Sango | 255491 | 0.9990 | 0.0000 | 0.9970 | 0.0005
san_Deva | Sanskrit | 39988 | 0.9900 | 0.0000 | 0.9885 | 0.0010
sat_Olck | Santali | 8875 | 1.0000 | 0.0000 | 1.0000 | 0.0000
scn_Latn | Sicilian | 40023 | 0.9956 | 0.0035 | 0.9936 | 0.0054
shn_Mymr | Shan | 21051 | 1.0000 | 0.0000 | 0.9985 | 0.0000
sin_Sinh | Sinhala | 361636 | 1.0000 | 0.0000 | 1.0000 | 0.0000
slk_Latn | Slovak | 3153492 | 0.9970 | 0.0010 | 0.9995 | 0.0005
slv_Latn | Slovenian | 3023266 | 0.9966 | 0.0030 | 0.9985 | 0.0015
smo_Latn | Samoan | 367828 | 0.9985 | 0.0010 | 0.9985 | 0.0010
sna_Latn | Shona | 764419 | 0.9941 | 0.0059 | 0.9941 | 0.0059
snd_Arab | Sindhi | 26107 | 0.9990 | 0.0000 | 0.9980 | 0.0020
som_Latn | Somali | 217413 | 0.9995 | 0.0005 | 1.0000 | 0.0000
sot_Latn | Southern Sotho | 2030 | 0.9567 | 0.0000 | 0.7552 | 0.0000
spa_Latn | Spanish | 677548 | 0.9921 | 0.0049 | 0.9922 | 0.0074
srd_Latn | Sardinian | 47480 | 0.9961 | 0.0030 | 0.9773 | 0.0000
srp_Cyrl | Serbian | 310259 | 0.9995 | 0.0000 | 1.0000 | 0.0000
ssw_Latn | Swati | 114900 | 0.9911 | 0.0020 | 0.9916 | 0.0015
sun_Latn | Sundanese | 47458 | 0.9926 | 0.0035 | 0.9599 | 0.0252
swe_Latn | Swedish | 2747052 | 1.0000 | 0.0000 | 0.9990 | 0.0005
swh_Latn | Swahili | 228559 | 0.9284 | 0.0771 | 0.8815 | 0.1345
szl_Latn | Silesian | 34065 | 0.9960 | 0.0000 | 0.9875 | 0.0015
tam_Taml | Tamil | 552180 | 1.0000 | 0.0000 | 1.0000 | 0.0000
taq_Latn | Tamasheq | 10266 | 0.7907 | 0.0010 | 0.7916 | 0.0000
taq_Tfng | Tamasheq | 6203 | 0.9505 | 0.0084 | 0.8513 | 0.0000
tat_Cyrl | Tatar | 257828 | 1.0000 | 0.0000 | 0.9995 | 0.0000
tel_Telu | Telugu | 276504 | 0.9990 | 0.0000 | 1.0000 | 0.0000
tgk_Cyrl | Tajik | 135652 | 1.0000 | 0.0000 | 1.0000 | 0.0000
tgl_Latn | Tagalog | 1189616 | 1.0000 | 0.0000 | 0.9970 | 0.0025
tha_Thai | Thai | 734727 | 1.0000 | 0.0000 | 1.0000 | 0.0000
tir_Ethi | Tigrinya | 333639 | 0.9995 | 0.0000 | 0.9995 | 0.0000
tpi_Latn | Tok Pisin | 471651 | 1.0000 | 0.0000 | 0.9980 | 0.0000
tsn_Latn | Tswana | 784851 | 0.9693 | 0.0311 | 0.8424 | 0.1859
tso_Latn | Tsonga | 756533 | 0.9961 | 0.0035 | 0.9907 | 0.0089
tuk_Latn | Turkmen | 160757 | 1.0000 | 0.0000 | 1.0000 | 0.0000
tum_Latn | Tumbuka | 237138 | 0.9956 | 0.0035 | 0.9816 | 0.0183
tur_Latn | Turkish | 823575 | 0.9936 | 0.0064 | 0.9840 | 0.0163
twi_Latn | Twi | 545217 | 0.9990 | 0.0000 | 0.9420 | 0.0005
tzm_Tfng | Central Atlas Tamazight | 8142 | 0.9535 | 0.0395 | 0.8854 | 0.1296
uig_Arab | Uyghur | 57231 | 1.0000 | 0.0000 | 0.9995 | 0.0005
ukr_Cyrl | Ukrainian | 1140463 | 0.9995 | 0.0005 | 1.0000 | 0.0000
umb_Latn | Umbundu | 220396 | 0.9776 | 0.0079 | 0.9687 | 0.0208
urd_Arab | Urdu | 412736 | 0.9849 | 0.0153 | 0.9735 | 0.0272
uzn_Latn | Northern Uzbek | 1519230 | 0.9990 | 0.0010 | 0.9995 | 0.0005
vec_Latn | Venetian | 43478 | 0.9961 | 0.0020 | 0.9916 | 0.0035
vie_Latn | Vietnamese | 881145 | 0.9995 | 0.0005 | 0.9873 | 0.0129
war_Latn | Waray | 282772 | 1.0000 | 0.0000 | 0.9990 | 0.0010
wol_Latn | Wolof | 28784 | 0.9970 | 0.0020 | 0.9950 | 0.0010
xho_Latn | Xhosa | 921590 | 0.9858 | 0.0119 | 0.9779 | 0.0148
ydd_Hebr | Eastern Yiddish | 911 | 0.9990 | 0.0000 | 1.0000 | 0.0000
yor_Latn | Yoruba | 531904 | 0.9990 | 0.0010 | 0.9956 | 0.0030
yue_Hant | Yue Chinese | 63254 | 0.0059 | 0.0025 | 0.4877 | 0.3229
zho_Hans | Chinese (Simplified) | 1046823 | 0.9891 | 0.0054 | 0.8559 | 0.0277
zho_Hant | Chinese (Traditional) | 2018541 | 0.6605 | 0.5020 | 0.4651 | 0.2176
zsm_Latn | Standard Malay | 404380 | 0.9495 | 0.0346 | 0.9351 | 0.0307
zul_Latn | Zulu | 951688 | 0.9828 | 0.0104 | 0.9696 | 0.0267
Table 3: For each language covered by our model, we give the number of lines
of deduplicated training data in our dataset, as well as the class F1 score
and class false positive rate (FPR) for our model and for the model described
in Costa-jussà et al. (2022) (NLLB).

  *[LID]: language identification
  *[NLP]: natural language processing
  *[MSA]: Modern Standard Arabic
  *[UDHR]: Universal Declaration of Human Rights
  *[FPRs]: false positive rate
  *[FPR]: false positive rate