#  Two Stacks Are Better Than One: A Comparison of Language Modeling and
Translation as Multilingual Pretraining Objectives

Zihao Li 1, Shaoxiong Ji ∗1, Timothee Mickus ∗1, Vincent Segonne 2 Jörg
Tiedemann 1
1University of Helsinki 2 Université Bretagne Sud
<EMAIL_ADDRESS>2univ-ubs.fr}

###### Abstract

Pretrained language models (PLMs) display impressive performances and have
captured the attention of the NLP community. Establishing the best practices
in pretraining has therefore become a major point of focus for much of NLP
research—especially since the insights developed for monolingual English
models need not carry to more complex multilingual. One significant caveat of
the current state of the art is that different works are rarely comparable:
they often discuss different parameter counts, training data, and evaluation
methodology.

This paper proposes a comparison of multilingual pretraining objectives in a
controlled methodological environment. We ensure that training data and model
architectures are comparable, and discuss the downstream performances across 6
languages that we observe in probing and fine-tuning scenarios. We make two
key observations: (1) the architecture dictates which pretraining objective is
optimal; (2) multilingual translation is a very effective pre-training
objective under the right conditions. We make our code, data, and model
weights available at https://github.com/Helsinki-NLP/lm-vs-mt.

**footnotetext: Equal contribution and corresponding authors.

##  1 Introduction

The introduction of BERT (Devlin et al., 2019) has introduced a change of
paradigm in the NLP landscape and has ushered in a thorough investment of the
NLP research community in developing large language model that can be adapted
to novel situations, i.e., foundational models. The design, training and
evaluation of these foundational models has become a significant enterprise of
its own.

In recent years, that sustained interest has shifted also to encompass
multilingual models (e.g., Muennighoff et al., 2022; Alves et al., 2024).
There is considerable variation as to how such models are trained: For
instance, some rely on datasets comprising multiple languages without explicit
cross-lingual supervision (e.g., Liu et al., 2020), and some use explicit
supervision (Xue et al., 2021). One complication that arises from this
blossoming field of study is that much of the work being carried out is not
directly comparable beyond the raw performances on some well-established
benchmark, a procedure which may well be flawed (Gorman and Bedrick, 2019).
Avoiding apples-to-oranges comparison requires a methodical approach in
strictly comparable circumstances, which is the stance we adopt in this paper.

In short, we focus on two variables—model architecture and pretraining
objectives—and set out to train five models in strictly comparable conditions
and compare their monolingual performances in three downstream applications:
sentiment analysis, named entity recognition, and POS-tagging. The scope of
our study spans from encoder-decoder machine translation models to encoder-
only BERT-like masked language models, categorizing them into double-stacks
(encoder-decoder) and single-stacks (encoder-only or decoder-only) models. We
intend to answer two research questions:

  1. (i)

Does the explicit cross-lingual training signal of translation objectives
foster better downstream performances in monolingual tasks?

  2. (ii)

Is the optimal choice of architecture independent of the training objective?

There are a prima facie reasons to favor either answers to both of these
questions. For instance, the successes of multilingual pretrained language
models (LM) on cross-lingual tasks have been underscored time and time again
(Wu and Dredze, 2019, e.g.,), yet explicit alignments such as linear mapping
(Wang et al., 2019) and L2 alignment (Cao et al., 2020) between source and
target languages do not necessarily improve the quality of cross-lingual
representations (Wu and Dredze, 2020).

Our findings suggest that insofar as a BART denoising autoencoder architecture
is concerned, a translation pretraining objective consistently outperforms a
denoising pretraining objective. However, for single-stack transformers, we
find causal language models to perform well in probing and masked language
models to generally outperform translation and causal objectives in fine-
tuning. This leads us to conclude that the optimal pre-training objective
depends on the architecture. Furthermore, the best downstream results we
observe appear to stem from a machine-translation system, highlighting that MT
encoder-decoder systems might constitute an understudied but potentially very
impactful pretrained model.

##  2 Methods and Settings

We start our inquiry by adopting a principled stance: We train strictly
comparable models with MT and LM objectives before contrasting their
performances on monolingual tasks.

### Models and objectives.

To allow a systematic evaluation, we train models with various neural network
architectures and learning objectives. All models are based on the transformer
architecture (Vaswani et al., 2017) and implemented in fairseq (Ott et al.,
2019). We consider both double-stacks (encoder-decoder) and single-stacks
(encoder-only or decoder-only) models.

The two double-stacks models are variants of the BART architecture of (Lewis
et al., 2020); they are trained either on a straightforward machine
translation (MT) objective, using language tokens to distinguish the source,
or on the original denoising auto-encoder objective of Lewis et al.. We refer
to these two models as 2-LM and 2-MT respectively.

We also consider three single-stack models: (i) an encoder-only model trained
on the masked language modeling objective (MLM) of Devlin et al. (2019); (ii)
an autoregressive causal language model (CLM), similar to Radford et al.
(2019); and (iii) an autoregressive model trained to generate a sentence,
followed by its translation in the language specified by a given control
token, known as a translation language model (TLM) as proposed by Conneau and
Lample (2019).111 In this work, we only focus on the causal variant of TLM
proposed by Conneau and Lample.  We provide an example datapoint for each pre-
training objective in Table 3, Appendix A.

### Pretraining conditions.

Our core focus is on guaranteeing comparable conditions across the different
pre-training objectives we consider. This entails that our datasets need to be
doubly structured: both in documents for CLM pretraining; and as aligned
bitexts for MT pretraining. Two datasets broadly match these criteria: the
UNPC (Ziemski et al., 2016) and OpenSubtitles (OpSub; Tiedemann, 2012)
corpora. The choice also narrows down the languages considered in this study:
we take the set of languages present in both resources, namely the six
languages in UNPC: Arabic (AR), Chinese (ZH), English (EN), French (FR),
Russian (RU), and Spanish (ES).

To guarantee that models are trained on the same data, whenever a document is
available in multiple languages, we greedily assign it to the least
represented language pair thus far and discard all other possible language
pairs where it could have contributed; we then discard documents which cannot
be used as bitexts. This ensures that all documents are used exactly once for
both document-level and bitext-level pretraining objectives. Dataset
statistics are shown in Table 4, Appendix B.

To ensure a fair comparison, we control key variables, including tokenization
(100k BPE pieces; Sennrich et al., 2016), number of transformer layers (12),
hidden dimensions (512), attention heads (8), and feedforward layer dimensions
(2048). We perform 600k steps of updates,222 Improvements in cross-entropy
over the validation set were always marginal after this stage.  using the
largest batch size that fits the GPU memory, deploy distributed training to
make a global batch size of 4096, and apply the Adam optimizer (Kingma and Ba,
2017). Owing to the computational requirements, we only train one seed for
each of the five types of models considered.

### Downstream evaluation.

The evaluations encompassed both sequence-level and token-level classification
tasks using datasets tailored for sentiment analysis (SA), named entity
recognition (NER), and part-of-speech (POS) tagging.

For SA, we utilized the Amazon review dataset (Hou et al., 2024) in English,
Spanish, French, and Chinese. RuReviews (Smetanin and Komarov, 2019) for
Russian, and ar_res_reviews (ElSahar and El-Beltagy, 2015) for Arabic. While
the datasets for most languages were pre-split, ar_res_reviews required manual
division into training, validation, and testing sets, using an 8:1:1 ratio.

For NER, we model the problem as an entity span extraction using a BIO scheme.
In practice, we classify tokens into three basic categories: Beginning of an
entity (B), Inside an entity (I), or Outside any entity (O). We use the
MultiCoNER v2 dataset (Fetahu et al., 2023) for English, Spanish, French, and
Chinese, MultiCoNER v1 (Malmasi et al., 2022) for Russian and the AQMAR
Wikipedia NER corpus (Mohit et al., 2012a) for Arabic. Simplifying the NER
task to these fundamental categories allows us to focus more on assessing the
basic entity recognition capabilities of the models without the additional
complexity of differentiating numerous entity types, which can vary
significantly between languages and datasets.

For POS tagging, we utilized the Universal Dependencies (UD) 2.0 datasets
(Nivre et al., 2020), selecting specific corpora tailored to each language to
ensure both linguistic diversity and relevance. We select multiple UD corpus
per language, such that each language dataset comprises approximately 160,000
tokens, which are then split into training, validation, and testing segments
with an 8:1:1 ratio. See also Appendix B.

We evaluate the performances of the encoder output representations for the
2-MT and 2-LM models and of the last hidden representation before the
vocabulary projection for the single-stack models.

The evaluation of the models involves two distinct experimental approaches to
optimize performance: probing and fine-tuning. In the probing experiments,
only the parameters of the classification heads are adjusted. This method
primarily tests the raw capability of the pre-trained models’ embeddings to
adapt to specific tasks with minimal parameter changes, preserving the
underlying pre-trained network structure. Conversely, in the fine-tuning
experiments, all parameters of the models are adjusted. This approach allows
the entire model to adapt to the specifics of the task, potentially leading to
higher performance at the cost of altering the pre-trained weights
significantly.

For both experimental approaches, each model is trained for 10 epochs to
ensure sufficient learning without overfitting. We optimize parameters with
AdamW (Loshchilov and Hutter, 2017), with a constant learning rate of 0.0001
across all tasks and models. This setup was chosen to standardize the training
process, providing a fair basis for comparing the performance outcomes across
different models and tasks. We reproduce probing and finetuning for 5 seeds to
ensure stability.

##  3 Results

Setup | Languages
---|---
EN | ES | FR | ZH | RU | AR
SA | 2-LM | 42.86 $\pm$ | 0.86 | 42.80 $\pm$ | 0.69 | 43.00 $\pm$ | 0.60 | 40.41 $\pm$ | 1.02 | 65.83 $\pm$ | 0.70 | 70.88 $\pm$ | 1.62
2-MT |  46.71 $\pm$ | 0.88 |  46.64 $\pm$ | 0.55 |  46.10 $\pm$ | 0.43 |  43.74 $\pm$ | 0.65 |  68.79 $\pm$ | 0.42 |  73.77 $\pm$ | 0.97
NER | 2-LM | 82.69 $\pm$ | 0.09 | 84.74 $\pm$ | 0.07 | 82.80 $\pm$ | 0.06 | 78.88 $\pm$ | 0.25 | 77.93 $\pm$ | 0.15 | 85.28 $\pm$ | 0.22
2-MT |  89.47 $\pm$ | 0.06 |  90.54 $\pm$ | 0.04 |  89.41 $\pm$ | 0.10 |  88.78 $\pm$ | 0.09 |  83.39 $\pm$ | 0.22 |  89.70 $\pm$ | 0.18
POS | 2-LM | 78.85 $\pm$ | 0.29 | 78.12 $\pm$ | 0.25 | 81.57 $\pm$ | 0.32 | 66.09 $\pm$ | 0.25 | 77.93 $\pm$ | 0.12 | 47.68 $\pm$ | 0.10
2-MT |  92.22 $\pm$ | 0.14 |  90.59 $\pm$ | 0.20 |  95.39 $\pm$ | 0.10 |  75.87 $\pm$ | 0.17 |  93.20 $\pm$ | 0.08 |  61.84 $\pm$ | 0.24

(a)  Probing

Setup | Languages
---|---
EN | ES | FR | ZH | RU | AR
SA | 2-LM | 52.26 $\pm$ | 0.55 | 52.89 $\pm$ | 0.69 | 52.99 $\pm$ | 0.59 | 48.64 $\pm$ | 0.36 | 73.89 $\pm$ | 0.43 | 79.74 $\pm$ | 1.36
2-MT |  54.76 $\pm$ | 0.58 |  55.56 $\pm$ | 0.49 |  54.75 $\pm$ | 0.42 |  50.55 $\pm$ | 0.68 |  74.77 $\pm$ | 0.50 |  81.49 $\pm$ | 1.49
NER | 2-LM | 91.13 $\pm$ | 0.12 | 91.82 $\pm$ | 0.21 | 91.58 $\pm$ | 0.10 | 92.30 $\pm$ | 0.10 | 85.34 $\pm$ | 0.39 | 89.05 $\pm$ | 0.13
2-MT |  93.46 $\pm$ | 0.09 |  94.22 $\pm$ | 0.09 |  93.84 $\pm$ | 0.04 |  93.75 $\pm$ | 0.32 |  89.07 $\pm$ | 0.11 |  93.26 $\pm$ | 0.15
POS | 2-LM | 92.42 $\pm$ | 0.28 | 90.41 $\pm$ | 0.16 | 95.21 $\pm$ | 0.13 | 82.30 $\pm$ | 0.48 | 95.36 $\pm$ | 0.20 | 69.57 $\pm$ | 0.24
2-MT |  95.98 $\pm$ | 0.08 |  94.29 $\pm$ | 0.05 |  98.05 $\pm$ | 0.17 |  90.18 $\pm$ | 0.15 |  97.00 $\pm$ | 0.07 |  74.47 $\pm$ | 0.08

(b) Fine-tuning

Table 1: Accuracy ($\times$ 100) of double-stack models ($\pm$ s.d. over 5
runs).

### Double-stack models.

We first compare the performance of 2-LM and 2-MT across several key language
processing tasks including SA, NER, and POS tagging. Results are shown in
Tables 1a and 1b. The pretraining objectives play a significant role in
shaping the models’ effectiveness. Specifically, 2-MT, which is pretrained
with a machine translation objective, consistently outperforms 2-LM, which
utilizes a denoising objective. This pattern is consistent across all
languages tested after fine-tuning as well as probing.

Setup | Languages
---|---
EN | ES | FR | ZH | RU | AR
SA | CLM |  35.14 $\pm$ | 0.92 |  35.66 $\pm$ | 1.10 |  34.14 $\pm$ | 1.63 |  33.62 $\pm$ | 0.83 |  57.57 $\pm$ | 1.11 |  67.71 $\pm$ | 2.24
MLM | 34.26 $\pm$ | 1.34 | 34.82 $\pm$ | 1.58 | 33.90 $\pm$ | 1.12 | 32.52 $\pm$ | 1.65 | 54.55 $\pm$ | 1.86 | 65.94 $\pm$ | 3.30
TLM | 29.68 $\pm$ | 2.22 | 32.20 $\pm$ | 3.07 | 32.26 $\pm$ | 2.34 | 29.88 $\pm$ | 4.17 | 56.45 $\pm$ | 1.81 | 64.45 $\pm$ | 1.81
NER | CLM |  80.27 $\pm$ | 0.12 |  82.59 $\pm$ | 0.06 |  80.38 $\pm$ | 0.12 |  77.92 $\pm$ | 0.28 |  76.39 $\pm$ | 0.03 | 84.17 $\pm$ | 0.08
MLM | 78.77 $\pm$ | 0.02 | 81.61 $\pm$ | 0.00 | 79.11 $\pm$ | 0.01 | 70.67 $\pm$ | 0.10 | 76.34 $\pm$ | 0.01 |  84.29 $\pm$ | 0.00
TLM | 79.10 $\pm$ | 0.06 | 81.94 $\pm$ | 0.13 | 79.56 $\pm$ | 0.14 | 77.26 $\pm$ | 0.24 |  76.39 $\pm$ | 0.02 | 84.26 $\pm$ | 0.02
POS | CLM |  69.06 $\pm$ | 0.38 |  70.32 $\pm$ | 0.50 |  76.67 $\pm$ | 0.46 |  51.40 $\pm$ | 0.47 |  59.64 $\pm$ | 0.62 |  43.49 $\pm$ | 0.40
MLM | 37.92 $\pm$ | 0.61 | 44.26 $\pm$ | 0.11 | 46.89 $\pm$ | 0.32 | 31.16 $\pm$ | 0.21 | 34.62 $\pm$ | 0.16 | 34.71 $\pm$ | 0.94
TLM | 62.96 $\pm$ | 1.02 | 62.08 $\pm$ | 1.99 | 63.89 $\pm$ | 1.06 | 50.46 $\pm$ | 0.53 | 54.27 $\pm$ | 0.87 | 40.94 $\pm$ | 1.16

(a) Probing

Setup | Languages
---|---
EN | ES | FR | ZH | RU | AR
SA | CLM |  55.23 $\pm$ | 0.72 | 47.81 $\pm$ | 15.55 | 54.84 $\pm$ | 0.62 | 51.18 $\pm$ | 0.94 | 75.07 $\pm$ | 0.21 | 66.18 $\pm$ | 21.74
MLM | 55.22 $\pm$ | 0.92 | 55.67 $\pm$ | 1.77 | 54.08 $\pm$ | 2.43 | 51.00 $\pm$ | 1.07 | 74.53 $\pm$ | 1.36 |  75.00 $\pm$ | 3.48
TLM | 55.14 $\pm$ | 0.92 |  55.84 $\pm$ | 0.59 |  55.22 $\pm$ | 0.98 |  51.46 $\pm$ | 0.53 |  75.31 $\pm$ | 0.57 | 72.75 $\pm$ | 2.25
NER | CLM | 89.91 $\pm$ | 0.33 | 91.42 $\pm$ | 0.15 | 90.65 $\pm$ | 0.17 | 89.97 $\pm$ | 0.14 | 83.20 $\pm$ | 0.31 |  87.50 $\pm$ | 2.22
MLM |  93.31 $\pm$ | 0.57 |  93.93 $\pm$ | 0.60 |  93.67 $\pm$ | 0.30 |  92.99 $\pm$ | 0.99 |  87.49 $\pm$ | 0.78 | 85.78 $\pm$ | 3.30
TLM | 89.88 $\pm$ | 0.06 | 91.45 $\pm$ | 0.25 | 90.49 $\pm$ | 0.23 | 90.10 $\pm$ | 0.14 | 83.76 $\pm$ | 0.63 | 84.29 $\pm$ | 0.00
POS | CLM | 91.72 $\pm$ | 0.14 | 90.51 $\pm$ | 0.13 | 95.75 $\pm$ | 0.10 | 78.61 $\pm$ | 0.31 | 85.50 $\pm$ | 0.15 | 57.43 $\pm$ | 1.63
MLM |  96.00 $\pm$ | 0.15 |  94.45 $\pm$ | 0.13 |  97.94 $\pm$ | 0.20 |  89.96 $\pm$ | 0.71 |  96.69 $\pm$ | 0.13 |  74.35 $\pm$ | 0.53
TLM | 91.68 $\pm$ | 0.19 | 90.38 $\pm$ | 0.20 | 86.99 $\pm$ | 19.40 | 78.50 $\pm$ | 0.52 | 85.71 $\pm$ | 0.18 | 59.11 $\pm$ | 0.50

(b) Fine-tuning

Table 2: Accuracy ($\times$ 100) of single-stack models ($\pm$ s.d. over 5
runs).

### Single-stack models.

Turning to the single-stack models (CLM, MLM, TLM), we find a somewhat more
complex picture. In a probing context (cf. Table 2a), we find the CLM to be
almost always the most effective—with the exception of NER in Arabic. As for
finetuning (Table 2b), while the MLM generally ranks first on all POS and NER
datasets, the TLM is usually effective for SA.333 However, remark that unlike
with the BART-based models, SA results are not stable when we shift metrics
from accuracy to F1 (see Tables 6 and 7 in Appendix C). The difference in F1
between the top two models is often $\leq 0.01$, making it difficult to
ascertain that one model strictly dominates.

### Discussion.

A first global observation that we can make for these results is that single-
stack and double-stack models appear to behave differently. While the MT
objective yields the highest performances for BART-type models, the downstream
performances of the TLM do not really stand out compared to the CLM in probing
and the MLM in fine-tuning scenarios. It is important to note that the
performances stem at least in part from the architecture itself: 2-MT and 2-LM
both consistently outperform all single-stack models in probing.

Fine-tuning also tends to minimize the difference between single-stack and
double-stack models—which suggests that the higher quality of BART-based
representations could be an artifact of training limitations. Moreover, the
relative ranks of the three single-stack models fluctuate much more than what
we see for the double-stack models, owing to no little extent to the
oftentimes momentous variation across seeds for single-stack models. We
therefore conjecture that while a translation objective can yield a clear
training signal towards semantically, this comes with two caveats: first, the
signal can only be leveraged with dedicated and separate modeling of source
and target (viz. double-stack models); second, this advantage is much less
consequential in fine-tuning scenarios.

##  4 Related works

Multilingual foundation models have flourished in recent years (a.o., Conneau
and Lample, 2019; Liu et al., 2020; Xue et al., 2021; Muennighoff et al.,
2022; Alves et al., 2024; üstün2024aya), and with them so have studies of
their representations (Conneau et al., 2020; Kassner et al., 2021; Choudhury
and Deshpande, 2021; Fierro and Søgaard, 2022; Hämmerl et al., 2023, a.o.).

This work is specifically related to Conneau and Lample (2019), which also
compares MLM, CLM, and TLM but does not normalize the training data. Another
point of comparison is Ji et al. (2024), which studies the impact of MT
continued pretraining in BART on cross-lingual downstream tasks. Monolingual
evaluation of multilingual systems has also been broached a.o. by Rust et al.
(2021).

##  5 Conclusion

This paper conducts an empirical study of how pretraining conditions of
multilingual models impact downstream performances in probing and fine-tuning
scenarios. Overall, we find that double-stack BART-based models fare much
better than single-stack models in probing scenarios, but the difference is
overall less clear when it comes to fine-tuning. We also find some tentative
evidence that translation objectives can be highly effective for model
pretraining in precise circumstances: Namely, the most effective model overall
on downstream tasks is an MT-pretrained BART-like model, which outperforms
both a more traditional denoising objective for BART as well as decoder-only
CLM and encoder-only MLM models. This suggests that translation can serve as a
powerful pretraining objective, although it is currently under-explored in
this regard.444 There are reasonable objections against using MT models as
pretrained multilingual foundation models—namely, unlike auto-regressive
causal language models, their generation capabilities are strictly tied to
translation, thereby requiring some degree of multilingualism from end-users.

Another crucial aspect of our study is that we present strictly comparable
models, trained on comparable data, with comparable parameter counts and
unified implementations. While this entails some limitations, especially with
regard to the scale of models and data used, we nonetheless believe that a
strict comparison can help discriminate between the various factors at play in
other works. Here, we find clear evidence that CLM pretraining objectives,
such as those used in GPT, outperform MLM-based models, such as BERT, in
probing scenarios; we are also able to isolate and highlight how the optimal
choice of pretraining objective is contingent on the architecture being
employed.

## Limitations

This study employs models that are not large in terms of model parameters in
the era of large language models. Such constraint potentially hinders the
generalizability of our results to much larger architectures that are capable
of handling a broader array of linguistic nuances. Furthermore, our study
focuses on a select group of languages and specific NLP tasks. This focus
might limit the applicability of our findings to other linguistic contexts or
more complex real-world applications where diverse language phenomena or
different task demands play a crucial role.

Another limitation is our reliance on specific corpora. The datasets utilized,
while valuable, represent a potential source of selection bias. They may not
fully encompass the vast diversity of global language use, thus skewing the
model training and evaluation. Such a bias could affect the robustness and
effectiveness of the pretrained models when applied to languages that are not
well-represented in the training data.

## References

  * AllSet Learning (2023) AllSet Learning. 2023.  Chinese grammar wiki.
  * Alves et al. (2024) Duarte M. Alves, José Pombal, Nuno M. Guerreiro, Pedro H. Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and André F. T. Martins. 2024\.  Tower: An open multilingual large language model for translation-related tasks.
  * Cao et al. (2020) Steven Cao, Nikita Kitaev, and Dan Klein. 2020.  Multilingual alignment of contextual word representations.  In _International Conference on Learning Representations_.
  * Choudhury and Deshpande (2021) Monojit Choudhury and Amit Deshpande. 2021.  How linguistically fair are multilingual pre-trained language models? _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(14):12710–12718.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  Cross-lingual language model pretraining.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Conneau et al. (2020) Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020\.  Emerging cross-lingual structure in pretrained language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6022–6034, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * ElSahar and El-Beltagy (2015) Hady ElSahar and Samhaa R El-Beltagy. 2015.  Building large arabic multi-domain resources for sentiment analysis.  In _International conference on intelligent text processing and computational linguistics_ , pages 23–34. Springer.
  * Fetahu et al. (2023) Besnik Fetahu, Zhiyu Chen, Sudipta Kar, Oleg Rokhlenko, and Shervin Malmasi. 2023\.  MultiCoNER v2: a large multilingual dataset for fine-grained and noisy named entity recognition.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 2027–2051, Singapore. Association for Computational Linguistics.
  * Fierro and Søgaard (2022) Constanza Fierro and Anders Søgaard. 2022.  Factual consistency of multilingual pretrained language models.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3046–3052, Dublin, Ireland. Association for Computational Linguistics.
  * Gorman and Bedrick (2019) Kyle Gorman and Steven Bedrick. 2019.  We need to talk about standard splits.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2786–2791, Florence, Italy. Association for Computational Linguistics.
  * Guillaume et al. (2019) Bruno Guillaume, Marie-Catherine de Marneffe, and Guy Perrier. 2019.  Conversion et améliorations de corpus du français annotés en Universal Dependencies [conversion and improvement of Universal Dependencies French corpora].  _Traitement Automatique des Langues_ , 60(2):71–95.
  * Hämmerl et al. (2023) Katharina Hämmerl, Alina Fastowski, Jindřich Libovický, and Alexander Fraser. 2023.  Exploring anisotropy and outliers in multilingual language models for cross-lingual semantic sentence similarity.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 7023–7037, Toronto, Canada. Association for Computational Linguistics.
  * Hou et al. (2024) Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley. 2024\.  Bridging language and items for retrieval and recommendation.  _arXiv preprint arXiv:2403.03952_.
  * Ji et al. (2024) Shaoxiong Ji, Timothee Mickus, Vincent Segonne, and Jörg Tiedemann. 2024.  Can machine translation bridge multilingual pretraining and cross-lingual transfer learning? In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_ , pages 2809–2818, Torino, Italia. ELRA and ICCL.
  * Kassner et al. (2021) Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021.  Multilingual LAMA: Investigating knowledge in multilingual pretrained language models.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 3250–3258, Online. Association for Computational Linguistics.
  * Kingma and Ba (2017) Diederik P. Kingma and Jimmy Ba. 2017.  Adam: A method for stochastic optimization.
  * Lee et al. (2017) John Lee, Herman Leung, and Keying Li. 2017.  Towards Universal Dependencies for learner Chinese.  In _Proceedings of the NoDaLiDa 2017 Workshop on Universal Dependencies (UDW 2017)_ , pages 67–71, Gothenburg, Sweden. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2022) Yixuan Li, Gerdes Kim, Guillaume Bruno, and Dan Zeman. 2022.  Ud chinese patentchar.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Lyashevskaya et al. (2018) Olga Lyashevskaya, Olga Rudina, Natalia Vlasova, and Anna Zhuravleva. 2018.  Ud russian taiga.
  * Malmasi et al. (2022) Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rokhlenko. 2022\.  MultiCoNER: A large-scale multilingual dataset for complex named entity recognition.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 3798–3809, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * McDonald et al. (2013) Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, Claudia Bedini, Núria Bertomeu Castelló, and Jungmee Lee. 2013.  Universal Dependency annotation for multilingual parsing.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 92–97, Sofia, Bulgaria. Association for Computational Linguistics.
  * Mohit et al. (2012a) Behrang Mohit, Nathan Schneider, Rishav Bhowmick, Kemal Oflazer, and Noah A. Smith. 2012a.  Recall-oriented learning of named entities in Arabic Wikipedia.  In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 162–173, Avignon, France. Association for Computational Linguistics.
  * Mohit et al. (2012b) Behrang Mohit, Nathan Schneider, Rishav Bhowmick, Kemal Oflazer, and Noah A Smith. 2012b.  Recall-oriented learning of named entities in arabic wikipedia.  In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 162–173.
  * Muennighoff et al. (2022) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022.  Crosslingual generalization through multitask finetuning.  _arXiv preprint arXiv:2211.01786_.
  * Nivre et al. (2017) Joakim Nivre, Željko Agić, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe, Masayuki Asahara, Luma Ateyah, Mohammed Attia, Aitziber Atutxa, Elena Badmaeva, Miguel Ballesteros, Esha Banerjee, Sebastian Bank, John Bauer, Kepa Bengoetxea, Riyaz Ahmad Bhat, Eckhard Bick, Cristina Bosco, Gosse Bouma, Sam Bowman, Aljoscha Burchardt, Marie Candito, Gauthier Caron, Gülşen Cebiroğlu Eryiğit, Giuseppe G. A. Celano, Savas Cetin, Fabricio Chalub, Jinho Choi, Yongseok Cho, Silvie Cinková, Çağrı Çöltekin, Miriam Connor, Marie-Catherine de Marneffe, Valeria de Paiva, Arantza Diaz de Ilarraza, Kaja Dobrovoljc, Timothy Dozat, Kira Droganova, Marhaba Eli, Ali Elkahky, Tomaž Erjavec, Richárd Farkas, Hector Fernandez Alcalde, Jennifer Foster, Cláudia Freitas, Katarína Gajdošová, Daniel Galbraith, Marcos Garcia, Filip Ginter, Iakes Goenaga, Koldo Gojenola, Memduh Gökırmak, Yoav Goldberg, Xavier Gómez Guinovart, Berta Gonzáles Saavedra, Matias Grioni, Normunds Grūzītis, Bruno Guillaume, Nizar Habash, Jan Hajič, Jan Hajič jr., Linh Hà Mỹ, Kim Harris, Dag Haug, Barbora Hladká, Jaroslava Hlaváčová, Petter Hohle, Radu Ion, Elena Irimia, Anders Johannsen, Fredrik Jørgensen, Hüner Kaşıkara, Hiroshi Kanayama, Jenna Kanerva, Tolga Kayadelen, Václava Kettnerová, Jesse Kirchner, Natalia Kotsyba, Simon Krek, Sookyoung Kwak, Veronika Laippala, Lorenzo Lambertino, Tatiana Lando, Phôêng Lê Hồng, Alessandro Lenci, Saran Lertpradit, Herman Leung, Cheuk Ying Li, Josie Li, Nikola Ljubešić, Olga Loginova, Olga Lyashevskaya, Teresa Lynn, Vivien Macketanz, Aibek Makazhanov, Michael Mandl, Christopher Manning, Ruli Manurung, Cătălina Mărănduc, David Mareček, Katrin Marheinecke, Héctor Martínez Alonso, André Martins, Jan Mašek, Yuji Matsumoto, Ryan McDonald, Gustavo Mendonça, Anna Missilä, Verginica Mititelu, Yusuke Miyao, Simonetta Montemagni, Amir More, Laura Moreno Romero, Shunsuke Mori, Bohdan Moskalevskyi, Kadri Muischnek, Nina Mustafina, Kaili Müürisep, Pinkey Nainwani, Anna Nedoluzhko, Lôêng Nguyễn Thị, Huyền Nguyễn Thị Minh, Vitaly Nikolaev, Rattima Nitisaroj, Hanna Nurmi, Stina Ojala, Petya Osenova, Lilja Øvrelid, Elena Pascual, Marco Passarotti, Cenel-Augusto Perez, Guy Perrier, Slav Petrov, Jussi Piitulainen, Emily Pitler, Barbara Plank, Martin Popel, Lauma Pretkalniņa, Prokopis Prokopidis, Tiina Puolakainen, Sampo Pyysalo, Alexandre Rademaker, Livy Real, Siva Reddy, Georg Rehm, Larissa Rinaldi, Laura Rituma, Rudolf Rosa, Davide Rovati, Shadi Saleh, Manuela Sanguinetti, Baiba Saulīte, Yanin Sawanakunanon, Sebastian Schuster, Djamé Seddah, Wolfgang Seeker, Mojgan Seraji, Lena Shakurova, Mo Shen, Atsuko Shimada, Muh Shohibussirri, Natalia Silveira, Maria Simi, Radu Simionescu, Katalin Simkó, Mária Šimková, Kiril Simov, Aaron Smith, Antonio Stella, Jana Strnadová, Alane Suhr, Umut Sulubacak, Zsolt Szántó, Dima Taji, Takaaki Tanaka, Trond Trosterud, Anna Trukhina, Reut Tsarfaty, Francis Tyers, Sumire Uematsu, Zdeňka Urešová, Larraitz Uria, Hans Uszkoreit, Gertjan van Noord, Viktor Varga, Veronika Vincze, Jonathan North Washington, Zhuoran Yu, Zdeněk Žabokrtský, Daniel Zeman, and Hanzhi Zhu. 2017.  Universal dependencies 2.0 – CoNLL 2017 shared task development and test data.  LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University.
  * Nivre et al. (2020) Joakim Nivre, Marie-Catherine De Marneffe, Filip Ginter, Jan Hajič, Christopher D Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020.  Universal dependencies v2: An evergrowing multilingual treebank collection.  _arXiv preprint arXiv:2004.10643_.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Qi et al. (2019) Peng Qi, Koichi Yasuoka, and Dan Zeman. 2019.  Ud chinese gsdsimp.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Rust et al. (2021) Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021.  How good is your tokenizer? on the monolingual performance of multilingual language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3118–3135, Online. Association for Computational Linguistics.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Smetanin and Komarov (2019) Sergey Smetanin and Michail Komarov. 2019.  Sentiment analysis of product reviews in russian using convolutional neural networks.  In _2019 IEEE 21st Conference on Business Informatics (CBI)_ , volume 01, pages 482–486.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in opus.  In _Proceedings of LREC_ , volume 2012, pages 2214–2218.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Wang et al. (2019) Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. 2019.  Cross-lingual bert transformation for zero-shot dependency parsing.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5721–5727.
  * Wong et al. (2017) Tak-sum Wong, Kim Gerdes, Herman Leung, and John Lee. 2017.  Quantitative comparative syntax on the Cantonese-Mandarin parallel dependency treebank.  In _Proceedings of the Fourth International Conference on Dependency Linguistics (Depling 2017)_ , pages 266–275, Pisa, Italy. Linköping University Electronic Press.
  * Wu and Dredze (2019) Shijie Wu and Mark Dredze. 2019.  Beto, bentz, becas: The surprising cross-lingual effectiveness of bert.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 833–844.
  * Wu and Dredze (2020) Shijie Wu and Mark Dredze. 2020.  Do explicit alignments robustly improve multilingual encoders?  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4471–4482.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Zeldes (2017) Amir Zeldes. 2017.  The GUM corpus: Creating multilayer resources in the classroom.  _Language Resources and Evaluation_ , 51(3):581–612.
  * Zeman et al. (2023) Dan Zeman, Kirian Guiller, and Bruno Guillaume. 2023.  Ud chinese beginner.
  * Zemánek (2008) Otakar Smrž Viktor Bielický Iveta Kouřilová Jakub Kráčmar Zemánek. 2008.  Dependency treebank : A word on the million words.
  * Ziemski et al. (2016) Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016.  The United Nations parallel corpus v1.0.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 3530–3534, Portorož, Slovenia. European Language Resources Association (ELRA).

##  Appendix A Overview of pretraining objectives

Objective |  Source input |  Target output
---|---|---
2-LM |  _D’autres _mesures _de _ce _type _vont _être [MASK] [MASK], _en _coopération _avec _d’autres _associations _de _Rom s, _de _Sin tis _et _de [MASK] _du _voyage _(<< C am min anti >> ). </s> |  <s> _D’autres _mesures _de _ce _type _vont _être _appliqu ées, _en _coopération _avec _d’autres _associations _de _Rom s, _de _Sin tis _et _de _gens _du _voyage _(<< C am min anti >> ). </s>
2-MT |  <fr> _D’autres _mesures _de _ce _type _vont _être _appliqu ées, _en _coopération _avec _d’autres _associations _de _Rom s, _de _Sin tis _et _de _gens _du _voyage _(<< C am min anti >> ). |  <s> _Other _similar _measures _are _going _to _be _taken _in _cooperation _with _other _Rom a, _Sin ti _and _Travel lers _(" C am min anti ") _associ ations. </s>
CLM |  ... _Divers _accords _ad _hoc _ont _été _conclus _à _cet _effet _par _le _Ministère _de _l’éducation _et _l’as sociation _Op era _Nom ad i. _D’autres _mesures _de _ce _type _vont _être _appliqu ées, _en _coopération _avec _d’autres _associations _de _Rom s, _de _Sin tis _et _de _gens _du _voyage _(<< C am min anti >> ). ... |  ... _accords _ad _hoc _ont _été _conclus _à _cet _effet _par _le _Ministère _de _l’éducation _et _l’as sociation _Op era _Nom ad i. _D’autres _mesures _de _ce _type _vont _être _appliqu ées, _en _coopération _avec _d’autres _associations _de _Rom s, _de _Sin tis _et _de _gens _du _voyage _(<< C am min anti >> ). ...
TLM |  _D’autres _mesures _de _ce _type _vont _être _appliqu ées, _en _coopération _avec _d’autres _associations _de _Rom s, _de _Sin tis _et _de _gens _du _voyage _(<< C am min anti >> ). <fr2en> _Other _similar _measures _are _going _to _be _taken _in _cooperation _with _other _Rom a, _Sin ti _and _Travel lers _(" C am min anti ") _associ ations. |  _mesures _de _ce _type _vont _être _appliqu ées, _en _coopération _avec _d’autres _associations _de _Rom s, _de _Sin tis _et _de _gens _du _voyage _(<< C am min anti >> ). <fr2en> _Other _similar _measures _are _going _to _be _taken _in _cooperation _with _other _Rom a, _Sin ti _and _Travel lers _(" C am min anti ") _associ ations. </s>
MLM |  <s> _D’autres _mesures _de _ce _type _vont _être [MASK] [MASK], _en _coopération _avec _d’autres _associations _de _Rom s, _de _Sin tis _et _de [MASK] _du _voyage _(<< C am min anti >> ). </s> |  <s> _D’autres _mesures _de _ce _type _vont _être _appliqu ées, _en _coopération _avec _d’autres _associations _de _Rom s, _de _Sin tis _et _de _gens _du _voyage _(<< C am min anti >> ). </s>
Table 3: Overview of the different objectives considered in this study. Top
two rows: two-stacks (encoder-decoder) models; bottom three rows: single-stack
(encoder-only or decoder-only) models.

Table 3 displays an example data point for all pre-training objectives we
consider. In principle, the CLM is a document-level objective, i.e., the full
document would be used as an input rather than the two sentences we show here.

##  Appendix B Datasets statistics

| Train | Validation | Test | Total
---|---|---|---|---
UNPC | $114\,376\,177$ | $76\,303$ | $40\,712$ | $114\,493\,192$
OpSub | $81\,622\,353$ | $359\,035$ | $77\,342$ | $82\,058\,730$
Total | $195\,998\,530$ | $435\,338$ | $118\,054$ | $196\,551\,922$

Table 4: Number of sentences in pretraining corpora.

An overview of the volume of data available for pretraining is displayed in
Table 4. The majority of the data were used for training.

Task | Language | Dataset | Class Count | Train | Validation | Test | Total
---|---|---|---|---|---|---|---
SA | EN | Amazon Review (Hou et al., 2024) | 5 | 200000 | 5000 | 5000 | 210000
ES | Amazon Review | 5 | 200000 | 5000 | 5000 | 210000
FR | Amazon Review | 5 | 200000 | 5000 | 5000 | 210000
ZH | Amazon Review | 5 | 200000 | 5000 | 5000 | 210000
RU | RuReviews (Smetanin and Komarov, 2019) | 3 | 85601 | 2143 | 2137 | 89881
AR |  ar_res_reviews (ElSahar and El-Beltagy, 2015) | 2 | 6680 | 835 | 835 | 8350
NER | EN | MultiCoNER v2 (Fetahu et al., 2023) | 3 | 253011 | 13323 | 3773671 | 4040005
ES | MultiCoNER v2 | 3 | 262814 | 13462 | 3925900 | 4202176
FR | MultiCoNER v2 | 3 | 247743 | 13062 | 3742924 | 4003729
ZH | MultiCoNER v2 | 3 | 245606 | 12816 | 489605 | 748027
RU | MultiCoNER v1 (Malmasi et al., 2022) | 3 | 242384 | 12787 | 2061318 | 2316489
AR | AQMAR Wikipedia NER corpus (Mohit et al., 2012b) | 3 | 57053 | 8615 | 8185 | 73853
POS | EN | UD_English-GUM (Zeldes, 2017) | 16 | 128391 | 16070 | 15554 | 160015
ES | UD_Spanish-GSD (McDonald et al., 2013) | 16 | 127459 | 16916 | 15645 | 160020
FR | UD_French-GSD (Guillaume et al., 2019) | 15 | 127638 | 16207 | 16167 | 160012
ZH | UD_Chinese-Beginner (Zeman et al., 2023; AllSet Learning, 2023)+ | 16 | 128935 | 15680 | 15758 | 160373
UD_Chinese-PUD (Nivre et al., 2017)+ |  |  |  |  |
UD_Chinese-HK (Wong et al., 2017)+ |  |  |  |  |
UD_Chinese-CFL (Lee et al., 2017)+ |  |  |  |  |
UD_Chinese-PatentChar (Li et al., 2022)+ |  |  |  |  |
UD_Chinese-GSDSmp (Qi et al., 2019) |  |  |  |  |
RU | UD_Russian-Taiga (Lyashevskaya et al., 2018) | 16 | 127647 | 16175 | 16184 | 160006
AR | UD_Arabic-PADT (Zemánek, 2008) | 16 | 127552 | 16608 | 15848 | 160008
Table 5: Statistics of datasets used for downstream evaluation tasks.

In Table 5, we present an overview of the datasets used for downstream
evaluation.

##  Appendix C Detailed results

Task | Model | Languages
---|---|---
EN | ES | FR | ZH | RU | AR
SA | 2-LM | 0.4130 $\pm$ | 0.0118 | 0.4120 $\pm$ | 0.0160 | 0.4166 $\pm$ | 0.0076 | 0.3859 $\pm$ | 0.0156 | 0.6599 $\pm$ | 0.0101 | 0.6343 $\pm$ | 0.0232
2-MT | 0.4588 $\pm$ | 0.0092 | 0.4554 $\pm$ | 0.0053 | 0.4448 $\pm$ | 0.0158 | 0.4260 $\pm$ | 0.0070 | 0.6935 $\pm$ | 0.0052 | 0.6864 $\pm$ | 0.0105
CLM | 0.3183 $\pm$ | 0.0099 | 0.3351 $\pm$ | 0.0198 | 0.3066 $\pm$ | 0.0192 | 0.3104 $\pm$ | 0.0135 | 0.5693 $\pm$ | 0.0107 | 0.5886 $\pm$ | 0.0106
MLM | 0.3236 $\pm$ | 0.0270 | 0.3188 $\pm$ | 0.0188 | 0.3153 $\pm$ | 0.0088 | 0.2936 $\pm$ | 0.0107 | 0.5434 $\pm$ | 0.0236 | 0.5804 $\pm$ | 0.0104
TLM | 0.2593 $\pm$ | 0.0298 | 0.2768 $\pm$ | 0.0589 | 0.2528 $\pm$ | 0.0487 | 0.2344 $\pm$ | 0.0539 | 0.5537 $\pm$ | 0.0307 | 0.5487 $\pm$ | 0.0190
NER | 2-LM | 0.5830 $\pm$ | 0.0057 | 0.5616 $\pm$ | 0.0070 | 0.5627 $\pm$ | 0.0039 | 0.5653 $\pm$ | 0.0164 | 0.4178 $\pm$ | 0.0100 | 0.4310 $\pm$ | 0.0179
2-MT | 0.7778 $\pm$ | 0.0014 | 0.7660 $\pm$ | 0.0014 | 0.7716 $\pm$ | 0.0031 | 0.7871 $\pm$ | 0.0043 | 0.6551 $\pm$ | 0.0088 | 0.7311 $\pm$ | 0.0099
CLM | 0.4516 $\pm$ | 0.0110 | 0.4213 $\pm$ | 0.0075 | 0.4306 $\pm$ | 0.0131 | 0.5086 $\pm$ | 0.0053 | 0.3004 $\pm$ | 0.0034 | 0.3223 $\pm$ | 0.0054
MLM | 0.3003 $\pm$ | 0.0017 | 0.2997 $\pm$ | 0.0001 | 0.3021 $\pm$ | 0.0019 | 0.3341 $\pm$ | 0.0108 | 0.2891 $\pm$ | 0.0001 | 0.3094 $\pm$ | 0.0000
TLM | 0.3485 $\pm$ | 0.0074 | 0.3471 $\pm$ | 0.0152 | 0.3499 $\pm$ | 0.0173 | 0.4876 $\pm$ | 0.0230 | 0.2941 $\pm$ | 0.0015 | 0.3094 $\pm$ | 0.0001
POS | 2-LM | 0.7241 $\pm$ | 0.0040 | 0.6607 $\pm$ | 0.0042 | 0.6848 $\pm$ | 0.0074 | 0.5964 $\pm$ | 0.0072 | 0.7427 $\pm$ | 0.0030 | 0.4678 $\pm$ | 0.0016
2-MT | 0.8520 $\pm$ | 0.0065 | 0.7685 $\pm$ | 0.0203 | 0.8300 $\pm$ | 0.0017 | 0.7002 $\pm$ | 0.0029 | 0.8587 $\pm$ | 0.0055 | 0.6575 $\pm$ | 0.0032
CLM | 0.5621 $\pm$ | 0.0069 | 0.5422 $\pm$ | 0.0066 | 0.5568 $\pm$ | 0.0064 | 0.3761 $\pm$ | 0.0148 | 0.4975 $\pm$ | 0.0140 | 0.3040 $\pm$ | 0.0106
MLM | 0.2157 $\pm$ | 0.0063 | 0.1499 $\pm$ | 0.0055 | 0.1722 $\pm$ | 0.0084 | 0.0717 $\pm$ | 0.0040 | 0.1275 $\pm$ | 0.0080 | 0.1511 $\pm$ | 0.0127
TLM | 0.4741 $\pm$ | 0.0147 | 0.3759 $\pm$ | 0.0378 | 0.3744 $\pm$ | 0.0153 | 0.3314 $\pm$ | 0.0112 | 0.3798 $\pm$ | 0.0097 | 0.2299 $\pm$ | 0.0215
Table 6: Macro F1 score using probing technique. Task | Model | Languages
---|---|---
EN | ES | FR | ZH | RU | AR
SA | 2-LM | 0.5213 $\pm$ | 0.0068 | 0.5254 $\pm$ | 0.0083 | 0.5244 $\pm$ | 0.0135 | 0.4739 $\pm$ | 0.0096 | 0.7421 $\pm$ | 0.0059 | 0.7522 $\pm$ | 0.0151
2-MT | 0.5407 $\pm$ | 0.0086 | 0.5510 $\pm$ | 0.0084 | 0.5398 $\pm$ | 0.0054 | 0.4956 $\pm$ | 0.0093 | 0.7522 $\pm$ | 0.0056 | 0.7767 $\pm$ | 0.0156
CLM | 0.5443 $\pm$ | 0.0072 | 0.4446 $\pm$ | 0.2115 | 0.5421 $\pm$ | 0.0089 | 0.5015 $\pm$ | 0.0187 | 0.7553 $\pm$ | 0.0015 | 0.5283 $\pm$ | 0.2328
MLM | 0.5441 $\pm$ | 0.0107 | 0.5466 $\pm$ | 0.0314 | 0.5348 $\pm$ | 0.0237 | 0.4972 $\pm$ | 0.0142 | 0.7509 $\pm$ | 0.0135 | 0.5695 $\pm$ | 0.1427
TLM | 0.5358 $\pm$ | 0.0186 | 0.5501 $\pm$ | 0.0128 | 0.5474 $\pm$ | 0.0137 | 0.5069 $\pm$ | 0.0119 | 0.7586 $\pm$ | 0.0057 | 0.4599 $\pm$ | 0.0943
NER | 2-LM | 0.8200 $\pm$ | 0.0042 | 0.8092 $\pm$ | 0.0053 | 0.8259 $\pm$ | 0.0035 | 0.8626 $\pm$ | 0.0022 | 0.7215 $\pm$ | 0.0122 | 0.7274 $\pm$ | 0.0093
2-MT | 0.8670 $\pm$ | 0.0017 | 0.8651 $\pm$ | 0.0022 | 0.8727 $\pm$ | 0.0018 | 0.8897 $\pm$ | 0.0042 | 0.7934 $\pm$ | 0.0039 | 0.8685 $\pm$ | 0.0046
CLM | 0.7950 $\pm$ | 0.0064 | 0.8053 $\pm$ | 0.0028 | 0.8099 $\pm$ | 0.0044 | 0.8129 $\pm$ | 0.0021 | 0.6622 $\pm$ | 0.0182 | 0.5994 $\pm$ | 0.1880
MLM | 0.8635 $\pm$ | 0.0123 | 0.8580 $\pm$ | 0.0142 | 0.8706 $\pm$ | 0.0055 | 0.8739 $\pm$ | 0.0199 | 0.7629 $\pm$ | 0.0172 | 0.4113 $\pm$ | 0.2254
TLM | 0.7908 $\pm$ | 0.0028 | 0.8024 $\pm$ | 0.0081 | 0.8067 $\pm$ | 0.0047 | 0.8120 $\pm$ | 0.0032 | 0.6758 $\pm$ | 0.0312 | 0.3094 $\pm$ | 0.0000
POS | 2-LM | 0.8925 $\pm$ | 0.0039 | 0.7365 $\pm$ | 0.0025 | 0.8496 $\pm$ | 0.0034 | 0.8088 $\pm$ | 0.0059 | 0.8984 $\pm$ | 0.0055 | 0.7769 $\pm$ | 0.0102
2-MT | 0.9314 $\pm$ | 0.0024 | 0.7826 $\pm$ | 0.0235 | 0.8866 $\pm$ | 0.0074 | 0.8842 $\pm$ | 0.0059 | 0.9285 $\pm$ | 0.0029 | 0.8660 $\pm$ | 0.0088
CLM | 0.8752 $\pm$ | 0.0042 | 0.7854 $\pm$ | 0.0024 | 0.8573 $\pm$ | 0.0041 | 0.7906 $\pm$ | 0.0195 | 0.8264 $\pm$ | 0.0104 | 0.5932 $\pm$ | 0.0194
MLM | 0.9177 $\pm$ | 0.0068 | 0.8079 $\pm$ | 0.0259 | 0.8851 $\pm$ | 0.0019 | 0.8313 $\pm$ | 0.0079 | 0.9226 $\pm$ | 0.0048 | 0.8602 $\pm$ | 0.0132
TLM | 0.8782 $\pm$ | 0.0045 | 0.7830 $\pm$ | 0.0067 | 0.7421 $\pm$ | 0.2503 | 0.7876 $\pm$ | 0.0271 | 0.8247 $\pm$ | 0.0088 | 0.6201 $\pm$ | 0.0071
Table 7: Macro F1 score after model fine-tuning.

In Table 6 and Table 7, we present the macro-f1 score of models in the
downstream evaluation.