# mBBC: Exploring the Multilingual Maze

Sina Bagheri Nezhad
Portland State University
<EMAIL_ADDRESS>
Ameeta Agrawal
Portland State University
<EMAIL_ADDRESS>

###### Abstract

Multilingual language models have gained significant attention in recent
years, enabling the development of applications that cater to diverse
linguistic contexts. In this paper, we present a comprehensive evaluation of
three prominent multilingual language models: mBERT, XLM-R, and GPT-3. Using
the self-supervised task of next token prediction, we assess their performance
across a diverse set of languages, with a focus on understanding the impact of
resource availability, word order, language family, and script type on model
accuracy. Our findings reveal that resource availability plays a crucial role
in model performance, with higher resource levels leading to improved
accuracy. We also identify the complex relationship between resource
availability, language families, and script types, highlighting the need for
further investigation into language-specific characteristics and structural
variations. Additionally, our statistical inference analysis identifies
significant features contributing to model performance, providing insights for
model selection and deployment. Our study contributes to a deeper
understanding of multilingual language models and informs future research and
development to enhance their performance and generalizability across languages
and linguistic contexts.

##  1 Introduction

Multilingual language models have transformed natural language processing
(NLP) by enabling applications like machine translation and sentiment analysis
in multiple languages Yang et al. (2023); Barbieri et al. (2022). However,
most research has focused on a limited set of widely spoken languages Anil et
al. (2023); Scao et al. (2022); Brown et al. (2020); OpenAI (2023); Touvron et
al. (2023), leaving a gap in broader understanding of multilingual models’
performance across languages with distinct linguistic properties.

To address this gap, we comprehensively evaluate multilingual language models
on the same task. In order to facilitate our analysis, we create a novel
multilingual dataset of BBC news articles in 43 languages, called mBBC, which
allows us to assess the models’ performance across different language
families, resource availability, script types, and more. By focusing on
recently published real-world news articles, we capture the challenges and
opportunities posed by different languages. We evaluate three state-of-the-art
models: mBERT Devlin et al. (2019), XLM-R Conneau et al. (2020) and GPT-3
Brown et al. (2020), representing different architectures and training
methods.

Our goal is to rigorously assess the performance and robustness of these
models on the multilingual BBC news dataset. We aim to gain insights into
their effectiveness in handling languages with varying properties. Our
evaluation approach captures language nuances without relying on extensive
preprocessing or language identification techniques.

We first present our methodology for constructing mBBC dataset, including data
collection and source selection. Then we explain the evaluation framework and
experimental setup used to evaluate mBERT, XLM-R and GPT-3. Additionally, we
analyze and discuss our empirical results, highlighting variations or trends
across languages.

The key findings of this study highlight the significance of resource
availability, language families, script type, and word order in the
performance of multilingual language models. Firstly, resource availability
strongly influences model performance, with languages having higher resource
levels consistently achieving better results. This underscores the importance
of linguistic resources for training multilingual models. Secondly, the
relationship between resource availability, language families, and model
performance is complex, indicating that language-specific characteristics and
structural variations play a role in model performance. While the impact of
script type varied among models, the GPT-3 model consistently performed better
with the Latin script. Word order had relatively less impact on mBERT and
XLM-R, but GPT-3 demonstrated sensitivity to word order.

We hope that our findings will help researchers and practitioners develop more
inclusive and effective multilingual NLP systems. The key contributions of
this work are as follows:

Dataset Construction. We created a unique multilingual dataset with BBC news
articles in 43 languages. This unbiased dataset ensures reliable evaluations
on unseen data. Our code for generating the dataset is accessible.
111https://github.com/PortNLP/mBBC

Methodology. We propose a methodology for evaluating multilingual models using
our dataset and a token prediction task.

Analysis and Insights. Through extensive experiments, we analyze the
performance of mBERT, XLM-R and GPT-3 across different languages, families,
and scripts. This analysis provides valuable insights into the models’
strengths and limitations.

##  2 Related Work

Multilingual natural language processing (NLP) research has made significant
strides, witnessing the development and evaluation of a myriad of advanced
multilingual language models. In recent years, a notable trend in this field
has been the emergence of multilingual models trained using combined language
datasets. Examples of such models include mBERT Devlin et al. (2019), XLM-R
Conneau et al. (2020), mBART Liu et al. (2020), mT5 Xue et al. (2021), BLOOM
Scao et al. (2022), GPT-3 Brown et al. (2020), GPT-4 OpenAI (2023), LLaMA
Touvron et al. (2023), PaLM Chowdhery et al. (2022), and PaLM 2 Anil et al.
(2023).

These sophisticated models, trained on diverse and combined language datasets,
mark a paradigm shift in multilingual NLP research. The convergence of
linguistic diversity and advanced machine learning techniques exemplified by
these models underscores the cutting-edge developments in the multilingual NLP
landscape.

Understanding multilingual language modeling challenges has been explored in
various studies. For instance, inflectional morphology’s impact on language
modeling was observed in a Bible translations corpus Park et al. (2021).
Another study proposed evaluating multilingual NLP models by predicting
performance based on existing scores Srinivasan et al. (2021), while RelateLM
leveraged language relatedness and transliteration for low-resource language
model adaptation Khemchandani et al. (2021). Additionally, the complexity of
language modeling is influenced more by simpler statistics than morphological
complexity Mielke et al. (2019). In a recent study by Lai et al. (2023), the
performance of the ChatGPT language model and similar LLMs was evaluated
comprehensively. Although the study covered multiple tasks and a total of 37
languages, it was limited to comparing only a handful of languages under the
same task, finding that ChatGPT performed worse on different NLP tasks and
languages compared to previous models.

In contrast, our research aims to conduct a comprehensive analysis of
multilingual language models across diverse languages, language families,
scripts, and resource availability alongside each other by evaluating several
multilingual models on the same task.

Previous studies mainly compared a limited set of languages on the same task
primarily because of limited availability of annotated datasets. Existing
datasets for multilingual evaluation have predominantly focused on specific
tasks and smaller groups of languages, including Universal Dependencies Nivre
et al. (2016), CoNLL Tjong Kim Sang (2002); Tjong Kim Sang and De Meulder
(2003), XNLI Conneau et al. (2018), DARPA LORELEI Strassel and Tracey (2016),
TyDi Zhang et al. (2021), XWikis Perez-Beltrachini and Lapata (2021), XQuAD
Artetxe et al. (2020), MEE Pouran Ben Veyseh et al. (2022), MECI Lai et al.
(2022), XGLUE Liang et al. (2020), and XTREME Hu et al. (2020), among others.
While these datasets enable researchers to evaluate multilingual learning
models and track progress in the field, the availability of annotated datasets
for comprehensive multilingual evaluation remains limited due to the numerous
challenges of creating such labeled datasets.

With the aim of studying multilingual models on the same task, we were
motivated to create a novel unannotated dataset comprising BBC news articles
in 43 languages.

Region | Language | ISO 639 | Word Order | Language Family | Script | Resources
---|---|---|---|---|---|---
Africa | Afaan Oromoo | gaz | SOV | Afro-Asiatic | Latin | 1
Amharic | amh | SOV | Afro-Asiatic | Ethiopic | 2
French | fra | SVO | Indo-European | Latin | 5
Hausa | hau | SVO | Afro-Asiatic | Latin | 2
Igbo | ibo | SVO | Niger-Congo | Latin | 1
Kirundi (Rundi) | run | SVO | Niger-Congo | Latin | 1
Nigerian Pidgin | pcm | SVO | Creole | Latin | 0
Somali | som | SOV | Afro-Asiatic | Latin | 1
Swahili | swh | SVO | Niger-Congo | Latin | 2
Tigrinya | tir | SOV | Afro-Asiatic | Ethiopic | 2
Yoruba | yor | SVO | Niger-Congo | Latin | 2
Asia (Central) | Kyrgyz | kir | SOV | Turkic | Cyrillic | 1
Uzbek | uzb | SOV | Turkic | Cyrillic | 3
Asia (Pacific) | Burmese | mya | SOV | Sino-Tibetan | Mon–Burmese | 1
Chinese | zho | SVO | Sino-Tibetan | Han | 5
Indonesian | ind | SVO | Austronesian | Latin | 3
Japanese | jpn | SOV | Japonic | Japanese | 5
Korean | kor | SOV | Koreanic | Korean | 4
Thai | tha | SVO | Kra-Dai | Thai | 3
Vietnamese | vie | SVO | Austro-Asiatic | Latin | 4
Asia (South) | Bengali | ben | SOV | Indo-European | Bengali | 3
Gujarati | guj | SOV | Indo-European | Gujarati | 1
Hindi | hin | SOV | Indo-European | Devanagari | 4
Marathi | mar | SOV | Indo-European | Devanagari | 2
Nepali | npi | SOV | Indo-European | Devanagari | 1
Pashto | pst | SOV | Indo-European | Arabic | 1
Punjabi | pan | SOV | Indo-European | Gurmukhi | 2
Sinhala | sin | SOV | Indo-European | Sinhala | 0
Tamil | tam | SOV | Dravidian | Tamil | 3
Telugu | tel | SOV | Dravidian | Telugu | 1
Urdu | urd | SOV | Indo-European | Arabic | 3
Europe | Azerbaijani | azj | SOV | Turkic | Latin | 0
English | eng | SVO | Indo-European | Latin | 5
Scottish Gaelic | gla | VSO | Indo-European | Latin | 1
Russian | rus | SVO | Indo-European | Cyrillic | 4
Serbian | srp | SVO | Indo-European | Latin | 4
Turkish | tur | SOV | Turkic | Latin | 4
Ukrainian | ukr | SVO | Indo-European | Cyrillic | 3
Welsh | cym | VSO | Indo-European | Latin | 1
Latin America | Portuguese | por | SVO | Indo-European | Latin | 4
Spanish | spa | SVO | Indo-European | Latin | 5
Middle East | Arabic | arb | SVO | Afro-Asiatic | Arabic | 5
Persian | pes | SOV | Indo-European | Arabic | 4
Table 1: Linguistic Diversity in mBBC Dataset. The "Region" column specifies
the target audience for BBC News in specific languages. The "Resources" column
denotes resource level ranging from 0 to 5, with 0 indicating extremely low-
resource and 5 indicating high-resource.

##  3 mBBC: Multilingual BBC News Dataset

To construct our multilingual dataset – mBBC – we gathered news articles from
various BBC news websites in 43 different languages. This selection was based
on the fact that BBC broadcasts news in these 43 languages, providing a global
coverage across continents, and spanning a diverse range of language families,
scripts, resource-levels, and word order ensuring a comprehensive
representation of linguistic diversity. Table 1 describes the mBBC dataset’s
languages, including language family, script, and other relevant linguistic
characteristics. The information sources include the BBC’s official
website222https://www.bbc.co.uk/ws/languages,
Ethnologue333https://www.ethnologue.com,
ScriptSource444https://www.scriptsource.org, and Joshi et al. (2020).

We collected data from various language families such as Indo-European, Sino-
Tibetan, Niger-Congo, Austronesian, Dravidian, and more, encompassing several
scripts like Latin, Cyrillic, Arabic, Devanagari, Chinese characters, and
others. This extensive representation facilitates a comprehensive evaluation
of multilingual language models across different linguistic contexts.
Moreover, the dataset includes both high-resource languages like English,
Spanish, and French, benefiting from extensive linguistic resources, as well
as low-resource languages such as Somali, Burmese, and Nepali, with limited
resources or smaller speaker populations Joshi et al. (2020). Including
languages with varying resource levels enables us to assess the adaptability
and effectiveness of multilingual language models across diverse linguistic
settings. To ensure an unbiased and robust analysis, our dataset consists of
news articles of minimum text length of 500 characters, sourced from reputable
sources in 2023, ensuring the models studied have not seen the data during
training.

The data in Table 1 reveals that Indo-European languages dominate our dataset,
reflecting their status as the most widely spoken language family in the world
Ethnologue (2022). Conversely, other language families are represented by a
comparable number of languages. In terms of writing systems, the Latin script
prevails, being used by nearly 70% of the global population Vaughan (2020). On
the other hand, other scripts exhibit a relatively uniform distribution.

Word order refers to the arrangement of syntactic constituents within a
language. Within the mBBC dataset, we observe languages with diverse word
orders, including Subject-Object-Verb (SOV), Subject-Verb-Object (SVO), and
Verb-Subject-Object (VSO) orders. Notably, the most frequent word order in our
dataset is SOV, followed by SVO, aligning with the broader distribution
observed in languages worldwide. Across the globe, approximately 43.3% of
languages exhibit an SOV word order, while around 40.3% follow an SVO word
order, and approximately 9.5% adopt a VSO word order Hammarström (2016).

Furthermore, our dataset contains a significant number of low resource
languages, with the majority falling under resource level 1. This indicates
that while linguistic resources may be limited for many languages, they are
still utilized by communities and services such as BBC News, which broadcasts
in these languages. Additional charts are included in Appendix A.

##  4 Exploring the Multilingual Maze

In this section, we describe the methodology for our analysis as well as the
experimental setup.

Figure 1: Average accuracy of GPT-3, XLM-R, and mBERT across different
resource levels. Figure 2: Average accuracy of GPT-3, XLM-R, and mBERT across
language families and resource levels. The results within each language family
are averaged for all languages of the same resource levels. For example,
within the Afro-Asiatic language family, languages at resource level 1 (Afaan
Oromoo and Somali) achieved an average accuracy of 0.68 on GPT-3. Figure 3:
Average accuracy of GPT-3, XLM-R, and mBERT for each script

###  4.1 Task: Next Token Prediction

To assess the performance of the multilingual language models along the
various language-related dimensions, we use the self-supervised task of next
token prediction which does not require any labeled data. Given a sequence of
$n$ tokens, the task is to predict the next token $n+1$. We formulate this as
a binary classification task: if the ground truth token matches any token in
the top $k$ predicted tokens generated by the models, then the output is
considered to be correct555In our experiments, $n=30$ and $k=5$.. This form of
evaluation provides insights into the models’ ability to generate accurate
predictions for the next token, indicating their understanding of the context
and language semantics. For each model, we utilize their respective tokenizers
to preprocess the input sequences and tokenize them appropriately. Here is an
example illustrating the task:

  * Input Sequence: “Ministry of Health in Sudan has reported that 36% of health facilities in Khartoum city are non-functional and another 25% are unrespon … "

  * Ground Truth Next Token: “##sive"

  * Top Predicted Tokens by mBERT: {‘##sible’, ‘##dent’, ‘##sive’, ‘##ding’, ‘##ded’}

In this example, since the ground truth token is present in the set of
predicted tokens, the model’s output is considered correct according to the
evaluation criteria. By considering the top $k$ predictions, we account for
possible variations in the ranking of candidate tokens while still capturing
successful predictions. For each language in our dataset, we experiment with
2000 samples. This sample size allowed us to obtain statistically significant
results while ensuring computational feasibility. The experimental procedure
and implementation details are described in Appendix B.

###  4.2 Experimental Setup

We use three multilingual language models in our analysis: mBERT (bert-base-
multilingual-cased) Devlin et al. (2019), XLM-R (xlm-roberta-base) Conneau et
al. (2020), and GPT-3 (text-davinci-003) Brown et al. (2020). These widely-
used models were chosen for their extensive language support, maximizing the
number of languages covered in our analysis.

Among the languages available in mBBC, mBERT was able to support 32, while
XLM-R supported 38, with 31 of them overlapping with mBERT’s supported
languages. We executed GPT-3 on all 43 languages in our dataset.

##  5 Results and Analysis

In this section, we present the results of our evaluation of multilingual
language models and analyze their performance based on various factors
including resources, language family, script, and their interactions. The
language-specific results for each model are summarized in Figure 12 in
Appendix C, providing a comprehensive overview of the performance across
different languages.

###  5.1 Impact of Resource Level

To understand the impact of resource availability on model performance, we
examined the relationship between resource levels and the models’ performance.
Figure 1 illustrates the performance of GPT-3, XLM-R and mBERT across varying
resource levels. Despite the notable difference in performance, with GPT-3
showing the highest performance and XLM-R displaying the weakest performance,
the three models maintain consistency in terms of resource availability
trends. The results clearly indicate that models perform the best on languages
with a resource level of 5, suggesting a strong correlation between the
availability of linguistic resources and the models’ performance. However, the
differences between the other resource levels are not as distinct, with the
results of languages in level 2 often lower than those of 0 or 1, implying
that the influence of resource availability on model performance is less
pronounced in those cases. In other words, except for resource 5 level
languages, increased resources do not necessarily guarantee improved
performance.

In addition to evaluating the impact of general resource availability on model
performance, we conducted further experiments to delve into the relationship
between model-specific language distribution statistics and performance across
different resource levels. For model-specific resources, we calculated the
correlation in two ways. First, we computed the correlation based on the row
number of the training set. Second, we divided the data into six bins (0 to 5)
based on the size of the training set, and calculated correlations within
these bins. The results of these experiments are summarized in Table 2.
Pearson correlation scores were computed to measure the relationship between
model performance and (a) general resource levels, as defined by Joshi et al.
(2020), and (b) model-specific resource availability.

| XLM-R | mBERT | GPT-3
---|---|---|---
Resource Levels Joshi et al. (2020) | 0.13 | 0.23 | 0.26
Model-specific Resources (Raw) | -0.07 | 0.23 | 0.20
Model-specific Resources (Binned) | -0.08 | 0.20 | 0.19
Table 2: Pearson Correlation Scores between Model Performance and Resource
Levels

The correlation scores presented in Table 2 highlight an intriguing pattern.
When examining the relationship between general resource levels (as defined by
Joshi et al. (2020)) and model performance, we observe moderate positive
correlations for all three models. However, when considering model-specific
resource availability, as measured by raw statistics and interval bins, the
correlations vary. While mBERT and GPT-3 exhibit consistent positive
correlations, XLM-R shows a negative weaker correlation. These findings shows
that general resource levels play more significant role in determining model
performance.

###  5.2 Impact of Resource Level and Language Families

Figure 2 presents the results of our next analysis, where we study the impact
of resource availability within each language family on multilingual language
model performance. We make several observations: (i) There is a considerable
difference in the performance between the same resource levels but different
language families, e.g., level 3 of Austronesian as compared to level 3 of
Dravidian or Indo-European. While some of these differences may be in part due
to the number of different languages present in each group, the results of
this fine-grained analysis suggest that resource levels alone may not be
sufficient indicator of performance. (ii) For those language families where
several resource levels exist, except a few outliers, the GPT-3 model mostly
works better for higher-resource languages. However, this trend is not as
consistent in the other two models. (iii) Turkic languages performed best with
GPT-3 at resource level 4, while Japonic languages performed best with XLM-R
at resource level 5, and mBERT excelled with Kra-Dai at resource level 3.
These results suggest notable differences among these multilingual models,
potentially due to the amount of language-specific training data available for
these models. However, the details of training data are becoming increasingly
difficult to obtain (e.g., GPT-3).

These findings demonstrate the complex relationship between language families,
resource availability, and model performance. While resource availability is
important, other factors also influence performance within specific language
families.

Figure 4: Average accuracy of GPT-3, XLM-R, and mBERT by word order, broken
down by resources.

###  5.3 Impact of Resource Level and Script

Next, we analyze the impact of resource levels and script types on
multilingual language model performance (Figure 3). Surprisingly, the mBERT
and XLM-R models showed no clear patterns between script types and
performance. Performance across script types appeared evenly distributed,
indicating that script type alone may not strongly predict model performance.

In contrast, the performance analysis of the GPT-3 model reveals a consistent
superiority of the Latin script over other scripts across all resource levels,
as well as the greater variance in performance across different scripts as
compared to other models. This underscores the influence of script selection
on the accuracy of the GPT-3 model, but not on the other two models.

###  5.4 Impact of Word order

Word order, also known as linear order, refers to the arrangement of syntactic
constituents within a language. The study of word order typology examines how
different languages employ distinct orders. In Figure 4, we present the
accuracy of different models across three word orders. Notably, we observe
that word order has a relatively minor impact on the accuracy of mBERT and
XLM-R, whereas it significantly affects the performance of GPT-3.

GPT-3 demonstrates better accuracy on subject-verb-object (SVO) languages
compared to subject-object-verb (SOV) languages across all resource levels.
Additionally, GPT-3 achieves its best performance on verb-subject-object (VSO)
languages, even though they are classified as low resource languages. These
findings indicate that word order plays a significant role in shaping the
performance of GPT-3, suggesting a heightened sensitivity of the model to the
structural characteristics of different word orders.

###  5.5 Statistical Inference

To collectively analyze and understand the intricate interplay of multiple
factors, we use decision tree models for statistical inference to identify
influential features, which were then subjected to Fisher’s exact test Fisher
(1922) to determine significant differences. Decision trees are trained to
predict the accuracy of models based on language features and thus analyzing
them allows us to gain insights into the significance of features on the
accuracy of next token prediction by the models.

The decision tree analysis of the GPT-3 model presented in Figure 5 identified
the importance of script type (Latin or not), word order (SVO or not), and
model specific resource level, which were supported by Fisher’s exact tests (p
< 0.001).

The decision tree analysis of XLM-R (Figure 6) and mBERT (Figure 7) models, on
the other hand, consistently demonstrated that general and model specific
resource level and script type play important rule in language models
performance, and statistical tests confirming significant performance
differences (p-value < 0.001). Also language family is critical for mBERT
(p-value <0.001). The full set of decisions trees have been included in
Appendix C.

Figure 5: GPT-3 performance analysis: Decision tree visualization. _Value_
refers to the expected accuracy of the model in predicting the next token. The
root value is 0.408, representing the expected accuracy across all languages.
It has two children: the right child represents languages in Latin script with
an expected accuracy of 0.635, and the left child represents languages in
other scripts with a value of 0.229. The Latin script is crucial as it divides
languages into two distinct sets with a significant difference in expected
values.  Figure 6: XLM-R performance analysis: Decision tree visualization
Figure 7: mBERT performance analysis: Decision tree visualization

##  6 Discussion

Our study evaluates the performance of multilingual language models,
specifically mBERT, XLM-R, and GPT-3, using the self-supervised task of next
token prediction. We conducted experiments to compare 43 distinct languages on
the same task to gain insights into the predictive capabilities of widely-used
multilingual models and understand the influence of resource availability,
word order, language family, and script type on their performance. Our key
observations can be summarized as follows:

  * •

Resource availability strongly correlates with model performance, especially
for languages with the highest resource level (5) which consistently achieved
better performance across the evaluated models. However, this distinction was
less clear for lower resource levels. Nonetheless, this highlights the
significance of linguistic resources for training (and possibly fine-tuning)
multilingual language models.

  * •

The relationship between resource availability, language families, and model
performance, however, proved complex. While some language families exhibited
consistent patterns across models, others showed varying results. This
suggests that factors beyond resource availability, such as language-specific
characteristics and structural variations, influence model performance.
Moreover, among the three models studied, there were notable differences,
potentially due to their different training corpora.

  * •

The impact of script type on model performance varied among the evaluated
models. While mBERT and XLM-R showed no clear patterns between script types,
the GPT-3 model consistently performed better with the Latin script. This
indicates that script type alone may not be a strong predictor of model
performance, requiring further investigation into language-specific
tokenization and training data.

  * •

Our analysis explored the influence of word order on model performance. We
found that word order has varying effects on the accuracy of different models.
Specifically, mBERT and XLM-R exhibited relatively consistent performance
across different word orders, suggesting that word order has less impact on
their predictions. However, GPT-3 demonstrated significant sensitivity to word
order, with notable variations in accuracy.

  * •

Statistical inference analysis identified significant features contributing to
model performance. mBERT exhibited less performance for Turkic language family
and languages written in Latin, while XLM-R showed influence from factors such
as Arabic and Latin scripts, and low-resource status. The GPT-3 model
performed well with the Latin script and larger resources.

##  7 Conclusion

Our extensive evaluation of multilingual language models across 43 diverse
languages highlighted several key findings. We demonstrated the significant
impact of resource availability on model performance, underscoring the need
for expanded linguistic resources to enhance the capabilities of these models.
Our results also revealed the complex relationship between resource
availability, word order, language families, and script types, emphasizing the
influence of language-specific characteristics on model performance.
Additionally, our findings indicated that script type and word order have
varying effects on the performance of multilingual language models. One
potential avenue for future research is to explore the impact of additional
factors, such as language-specific morphological features or syntactic
structures, on model performance.

## Limitations

Our study has several limitations that warrant acknowledgement. Firstly, the
evaluation relied on a single dataset constructed from BBC news articles,
which may not fully encompass the diversity of languages and language usages.
To obtain a more comprehensive understanding of multilingual language model
performance, future work should incorporate additional datasets from diverse
domains and genres.

We do not consider the impact of the amount of language-specific data
available within the training corpora of these multilingual models because
that information is not available for all the models considered in our
analysis.

Moreover, the dataset used in our study focused on a specific time period, and
its representativeness may evolve over time. Regular updates to the dataset
would be necessary to ensure its ongoing relevance and alignment with the
dynamic nature of language use.

Another limitation is the absence of fine-grained language identification and
preprocessing steps in our data collection process. While this enabled direct
retrieval of articles from specific news sources in each language, it may have
introduced noise and inconsistencies into the dataset. Future research should
consider integrating robust language identification and preprocessing
techniques to enhance the quality and consistency of the dataset.

Addressing these limitations will contribute to a more robust and
comprehensive evaluation of multilingual language models, enabling a deeper
understanding of their performance across diverse languages and linguistic
contexts.

## Ethics Statement

The experimental setup and code implementation ensured adherence to ethical
guidelines, data usage agreements, and compliance with the terms of service of
the respective language models and data sources. The research team also
recognized the importance of inclusivity and fairness by considering a diverse
set of languages and language families in the evaluation, thereby avoiding
biases and promoting balanced representation.

## References

  * Anil et al. (2023) Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. 2023.  Palm 2 technical report.
  * Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020.  On the cross-lingual transferability of monolingual representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4623–4637, Online. Association for Computational Linguistics.
  * Barbieri et al. (2022) Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collados. 2022.  XLM-T: Multilingual language models in Twitter for sentiment analysis and beyond.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 258–266, Marseille, France. European Language Resources Association.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022\.  Palm: Scaling language modeling with pathways.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ethnologue (2022) Ethnologue. 2022.  What are the largest language families?
  * Fisher (1922) Ronald A Fisher. 1922.  On the interpretation of $\chi$ 2 from contingency tables, and the calculation of p.  _Journal of the royal statistical society_ , 85(1):87–94.
  * Hammarström (2016) Harald Hammarström. 2016.  Linguistic diversity and language evolution.  _Journal of Language Evolution_ , 1(1):19–29.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In _International Conference on Machine Learning_ , pages 4411–4421. PMLR.
  * Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020.  The state and fate of linguistic diversity and inclusion in the NLP world.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6282–6293, Online. Association for Computational Linguistics.
  * Khemchandani et al. (2021) Yash Khemchandani, Sarvesh Mehtani, Vaidehi Patil, Abhijeet Awasthi, Partha Talukdar, and Sunita Sarawagi. 2021.  Exploiting language relatedness for low web-resource language model adaptation: An Indic languages study.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1312–1323, Online. Association for Computational Linguistics.
  * Lai et al. (2023) Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023.  Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning.  _arXiv preprint arXiv:2304.05613_.
  * Lai et al. (2022) Viet Dac Lai, Amir Pouran Ben Veyseh, Minh Van Nguyen, Franck Dernoncourt, and Thien Huu Nguyen. 2022.  MECI: A multilingual dataset for event causality identification.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 2346–2356, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Liang et al. (2020) Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020.  XGLUE: A new benchmark dataset for cross-lingual pre-training, understanding and generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6008–6018, Online. Association for Computational Linguistics.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual Denoising Pre-training for Neural Machine Translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Mielke et al. (2019) Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. 2019\.  What kind of language is hard to language-model? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4975–4989, Florence, Italy. Association for Computational Linguistics.
  * Nivre et al. (2016) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016.  Universal Dependencies v1: A multilingual treebank collection.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 1659–1666, Portorož, Slovenia. European Language Resources Association (ELRA).
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Park et al. (2021) Hyunji Hayley Park, Katherine J. Zhang, Coleman Haley, Kenneth Steimel, Han Liu, and Lane Schwartz. 2021.  Morphology matters: A multilingual language modeling analysis.  _Transactions of the Association for Computational Linguistics_ , 9:261–276.
  * Perez-Beltrachini and Lapata (2021) Laura Perez-Beltrachini and Mirella Lapata. 2021.  Models and datasets for cross-lingual summarisation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9408–9423, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Pouran Ben Veyseh et al. (2022) Amir Pouran Ben Veyseh, Javid Ebrahimi, Franck Dernoncourt, and Thien Nguyen. 2022\.  MEE: A novel multilingual event extraction dataset.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9603–9613, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Srinivasan et al. (2021) Anirudh Srinivasan, Sunayana Sitaram, Tanuja Ganu, Sandipan Dandapat, Kalika Bali, and Monojit Choudhury. 2021.  Predicting the performance of multilingual nlp models.  _arXiv preprint arXiv:2110.08875_.
  * Strassel and Tracey (2016) Stephanie Strassel and Jennifer Tracey. 2016.  LORELEI language packs: Data, tools, and resources for technology development in low resource languages.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 3273–3280, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Tjong Kim Sang (2002) Erik F. Tjong Kim Sang. 2002.  Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition.  In _COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002)_.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pages 142–147.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.
  * Vaughan (2020) Don Vaughan. 2020.  The world’s 5 most commonly used writing systems.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Yang et al. (2023) Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. 2023.  Bigtrans: Augmenting large language models with multilingual translation capability over 100 languages.  _arXiv preprint arXiv:2305.18098_.
  * Zhang et al. (2021) Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021.  Mr. TyDi: A multi-lingual benchmark for dense retrieval.  In _Proceedings of the 1st Workshop on Multilingual Representation Learning_ , pages 127–137, Punta Cana, Dominican Republic. Association for Computational Linguistics.

## Appendix

##  Appendix A mBBC overview

Figures 8, 9, 10, and 11 present the distribution of languages in the mBBC
dataset according to resource levels, language family, script, and word order,
respectively.

##  Appendix B Experimental Procedure and Implementation

To evaluate the models, we followed a consistent experimental procedure across
all languages:

  1. 1.

Select a language from the dataset and load the corresponding language model
(e.g., bert-base-multilingual-cased for supported languages).

  2. 2.

Randomly sample 2000 instances from the dataset for the chosen language.

  3. 3.

For each instance, provide the model with an input sequence of 30 tokens and
prompt it to predict the next token.

  4. 4.

Rank the predicted tokens based on their probability scores.

  5. 5.

Check if the ground truth token appears in the top 5 predicted tokens.

  6. 6.

Repeat steps 1-5 for each language and model combination.

For our experiments, we leveraged the HuggingFace transformers library, a
popular and flexible NLP library, to evaluate mBERT and XLM-R. This library
offered a convenient and efficient framework for conducting experiments with
different language models. To execute the experiments, we utilized Google
Colab with a T4 GPU accelerator, enabling us to efficiently process a large
number of samples across multiple languages and reduce overall processing
time. For the GPT-3 model, we employed the OpenAI API to implement and assess
its performance in our tests.

Figure 8: Count of languages for each resources. Figure 9: Count of languages
for each language family. Figure 10: Count of languages for each script.
Figure 11: Count of language for word order.

##  Appendix C Additional Results

The performance of each model in different languages, along with their
respective language families and resource sizes, is depicted in Figure 12.
This comprehensive visualization provides a clear overview of how each model
performs across various languages and highlights the relationship between
language characteristics and model performance.

To gain deeper insights into the decision-making process of each model,
Figures 14, 15, and 13 present the complete decision trees for mBERT, XLM-R,
and GPT-3, respectively. These decision trees provide a detailed
representation of the factors influencing the models’ predictions, offering a
comprehensive view of the underlying mechanisms employed by each model.

Figure 12: Accuracy of GPT-3, XLM-R, and mBERT for each language, with
language family indicated by color and resources indicated by size Figure 13:
Decision tree visualization for GPT-3 model Figure 14: Decision tree
visualization for mBERT model Figure 15: Decision tree visualization for XLM-R
model