# mLUKE: The Power of Entity Representations
in Multilingual Pretrained Language Models

Ryokan Ri1,2
<EMAIL_ADDRESS>Ikuya Yamada1
<EMAIL_ADDRESS>Yoshimasa Tsuruoka2
<EMAIL_ADDRESS>1Studio Ousia, Tokyo, Japan 2The University of
Tokyo, Tokyo, Japan Work done as an intern at Studio Ousia.

###### Abstract

Recent studies have shown that multilingual pretrained language models can be
effectively improved with cross-lingual alignment information from Wikipedia
entities. However, existing methods only exploit entity information in
pretraining and do not explicitly use entities in downstream tasks. In this
study, we explore the effectiveness of leveraging entity representations for
downstream cross-lingual tasks. We train a multilingual language model with 24
languages with entity representations and show the model consistently
outperforms word-based pretrained models in various cross-lingual transfer
tasks. We also analyze the model and the key insight is that incorporating
entity representations into the input allows us to extract more language-
agnostic features. We also evaluate the model with a multilingual cloze prompt
task with the mLAMA dataset. We show that entity-based prompt elicits correct
factual knowledge more likely than using only word representations.

##  1 Introduction

Pretrained language models have become crucial for achieving state-of-the-art
performance in modern natural language processing. In particular, multilingual
language models (Conneau and Lample, 2019; Conneau et al., 2020a; Doddapaneni
et al., 2021) have attracted considerable attention particularly due to their
utility in cross-lingual transfer.

In zero-shot cross-lingual transfer, a pretrained encoder is fine-tuned in a
single resource-rich language (typically English), and then evaluated on other
languages never seen during fine-tuning. A key to solving cross-lingual
transfer tasks is to obtain representations that generalize well across
languages. Several studies aim to improve multilingual models with cross-
lingual supervision such as bilingual word dictionaries (Conneau et al.,
2020b) or parallel sentences (Conneau and Lample, 2019).

Another source of such information is the cross-lingual mappings of Wikipedia
entities (articles). Wikipedia entities are aligned across languages via
inter-language links and the text contains numerous entity annotations
(hyperlinks). With these data, models can learn cross-lingual correspondence
such as the words Tokyo and 東京 refers to the same entity. Wikipedia entity
annotations have been shown to provide rich cross-lingual alignment
information to improve multilingual language models (Iacer Calixto and Pasini,
2021; Xiaoze Jian and Duan, 2021). However, previous studies only incorporate
entity information through an auxiliary loss function during pretraining, and
the models do not explicitly have entity representations used for downstream
tasks.

In this study, we investigate the effectiveness of entity representations in
multilingual language models. Entity representations are known to enhance
language models in mono-lingual settings (Zhang et al., 2019; Peters et al.,
2019; Wang et al., 2021; Xiong et al., 2020; Yamada et al., 2020) presumably
by introducing real-world knowledge. We argue that using entity
representations facilitates cross-lingual transfer by providing language-
independent features. To this end, we present a multilingual extension of LUKE
(Yamada et al., 2020). The model is trained with the multilingual masked
language modeling (MLM) task as well as the masked entity prediction (MEP)
task with Wikipedia entity embeddings.

We investigate two ways of using the entity representations in cross-lingual
transfer tasks: (1) perform entity linking for the input text, and append the
detected entity tokens to the input sequence. The entity tokens are expected
to provide language-independent features to the model. We evaluate this
approach with cross-lingual question answering (QA) datasets: XQuAD (Artetxe
et al., 2020) and MLQA (Lewis et al., 2020); (2) use the entity [MASK] token
from the MEP task as a language-independent feature extractor. In the MEP
task, word tokens in a mention span are associated with an entity [MASK]
token, the contextualized representation of which is used to train the model
to predict its original identity. Here, we apply similar input formulations to
tasks involving mention-span classification, relation extraction (RE) and
named entity recognition (NER): the attribute of a mention or a pair of
mentions is predicted using their contextualized entity [MASK] feature. We
evaluate this approach with the RELX (Köksal and Özgür, 2020) and CoNLL NER
(Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003) datasets.

The experimental results show that these entity-based approaches consistently
outperform word-based baselines. Our analysis reveals that entity
representations provide more language-agnostic features to solve the
downstream tasks.

We also explore solving a multilingual zero-shot cloze prompt task (Liu et
al., 2021) with the entity [MASK] token. Recent studies have shown that we can
address various downstream tasks by querying a language model for blanks in
prompts (Petroni et al., 2019; Cui et al., 2021). Typically, the answer tokens
are predicted from the model’s word-piece vocabulary but here we incorporate
the prediction from the entity vocabulary queried by the entity [MASK] token.
We evaluate our approach with the mLAMA dataset (Kassner et al., 2021) in
various languages and show that using the entity [MASK] token reduces language
bias and elicits correct factual knowledge more likely than using only the
word [MASK] token.

Figure 1: How to use entity representations in downstream tasks. The input
entity embeddings are associated with their mentions (indicated by dotted
lines) via positional embeddings.

##  2 Multilingual Language Models with Entity Representations

###  2.1 Model: mulitlingual LUKE

To evaluate the effectiveness of entity representations for cross-lingual
downstream tasks, we introduce a new multilingual language model based on a
bidirectional transformer encoder: Multilingual LUKE (mLUKE), a multilingual
extension of LUKE (Yamada et al., 2020). The model is trained with the masked
language modeling (MLM) task (Vaswani et al., 2017) as well as the masked
entity prediction (MEP) task. In MEP, some of the input entity tokens are
randomly masked with the special entity [MASK] token, and the model is trained
to predict the original entities. Note that the entity [MASK] token is
different from the word [MASK] token for MLM.

The model takes as input a tokenized text ($w_{1},w_{2},...,w_{m}$) and the
entities appearing in the text ($e_{1},e_{2},...,e_{n}$), and compute the
contextualized representation for each token
($\mathbf{h}_{w_{1}},\mathbf{h}_{w_{2}},...,\mathbf{h}_{w_{m}}$ and
$\mathbf{h}_{e_{1}},\mathbf{h}_{e_{2}},...,\mathbf{h}_{e_{n}}$). The word and
entity tokens equally undergo self-attention computation (i.e., no entity-
aware self-attention in the original work) after embedding layers.

The word and entity embeddings are computed as the summation of the following
three embeddings: token embeddings, type embeddings, and position embeddings
(Devlin et al., 2019). The entity tokens are associated with the word tokens
through position embeddings: the position of an entity token is defined as the
positions of its corresponding word tokens, and the entity position embeddings
are summed over the positions.

Model Configuration. Our model configuration follows XLM-RoBERTa-base (Conneau
et al., 2020a), a variant of BERT (Devlin et al., 2019) trained with
CommonCrawl data from 100 languages, and the parameters in common (e.g., the
weights of the transformer encoder and the word embeddings) are initialized
using the checkpoint from the Transformers
library.111https://huggingface.co/transformers/

The size of the entity embeddings is set to 256 and they are projected to the
size of the word embeddings before fed into the encoder.

###  2.2 Training Corpus: Wikipedia

We use Wikipedia dumps in 24 languages (Appendix A) as the training data. The
languages are selected to cover reasonable numbers of languages that appear in
downstream cross-lingual datasets. We generate input sequences by splitting
the content of each page into sequences of sentences comprising $\leq$ 512
words with their entity annotations (i.e., hyperlinks). During training, data
are sampled from each language with $n_{i}$ items with the following
multinomial distribution:

$\displaystyle p_{i}=\frac{n_{i}^{\alpha}}{\sum_{k=1}^{N}n_{k}^{\alpha}},$ (1)

where $\alpha$ is a smoothing parameter and set to 0.7 following multilingual
BERT.222https://github.com/google-research/bert/blob/master/multilingual.md

Entity Vocabulary. Entities in the model are defined as Wikipedia articles.
The articles from different languages are aligned through inter-language
links333https://en.wikipedia.org/wiki/Help:Interlanguage_links. We build an
inter-language database from the wikidatawiki dump from November 30, 2020. and
the aligned articles are treated as a single entity. We include in the
vocabulary the most frequent 1.2M entities in terms of the number of
hyperlinks that appear across at least three languages to facilitate cross-
lingual learning.

Optimization. We optimize the model with a batch size of 2048 for 1M steps in
total using AdamW (Loshchilov and Hutter, 2019) with warmup and linear decay
of the learning rate. To stabilize training, we perform pretraining in two
stages: (1) in the first 500K steps, we update only those parameters that are
randomly initialized (e.g., entity embeddings); (2) we update all parameters
in the remaining 500K steps. The learning rate scheduler is reset at each
training stage. For further details on hyperparameters, please see Appendix A.

###  2.3 Baseline Models

We compare the primary model that we investigate, multilingual LUKE used with
entity representations (mLUKE-E), against several baseline pretrained models
and an ablation model based on word representations:

mBERT (Devlin et al., 2019) is one of the earliest multilingual language
models. We provide this results as a reference.

XLM-R (Conneau et al., 2020a) is the model that mLUKE-E is built on. This
result indicates how our additional pretraining step and entity representation
impacts the performance. Since earlier studies (Liu et al., 2019; Lan et al.,
2020) indicated longer pretraining would simply improve performance, we also
train another model based on XLM-R with extra pretraining following the same
configuration of mLUKE.

mLUKE-W is an ablation model of mLUKE-E. This model discards the entity
embeddings learned during pretraining and only takes word tokens as input as
with the other baseline models. The results from this model indicate the
effect of cross-lingual alignment from pretraining with entity information
(Iacer Calixto and Pasini, 2021; Xiaoze Jian and Duan, 2021), and the
comparison with this model will highlight the effect of using entity
representations for downstream tasks in mLUKE-E.

XQuAD | en | es | de | el | ru | tr | ar | vi | th | zh | hi | avg.
---|---|---|---|---|---|---|---|---|---|---|---|---
mBERT | 84.5 | 76.1 | 73.1 | 59.0 | 70.2 | 53.2 | 62.1 | 68.5 | 40.7 | 58.3 | 57.0 | 63.9
XLM-R | 84.0 | 76.5 | 76.4 | 73.9 | 74.4 | 67.8 | 68.1 | 74.2 | 66.8 | 61.5 | 68.7 | 72.0
\+ extra training | 86.1 | 76.9 | 76.5 | 73.7 | 74.7 | 66.3 | 68.2 | 74.5 | 67.7 | 64.7 | 66.6 | 72.4
mLUKE-W | 85.7 | 78.0 | 77.4 | 74.7 | 75.7 | 68.3 | 71.7 | 75.9 | 67.1 | 65.1 | 69.9 | 73.6
mLUKE-E | 86.3 | 78.9 | 78.9 | 73.9 | 76.0 | 68.8 | 71.4 | 76.4 | 67.5 | 65.9 | 72.2 | 74.2

MLQA | en | es | de | ar | hi | vi | zh | avg. | G-XLT avg.
---|---|---|---|---|---|---|---|---|---
mBERT | 79.1 | 65.9 | 58.6 | 48.6 | 44.8 | 58.5 | 58.1 | 59.1 | 40.9
XLM-R | 79.7 | 67.7 | 62.2 | 55.8 | 59.9 | 65.3 | 62.5 | 64.7 | 33.4
\+ extra training | 81.3 | 69.8 | 65.0 | 54.8 | 59.3 | 65.6 | 64.2 | 65.7 | 50.2
mLUKE-W | 81.3 | 69.7 | 65.4 | 60.4 | 63.2 | 68.3 | 66.1 | 67.8 | 54.0
mLUKE-E | 80.8 | 70.0 | 65.5 | 60.8 | 63.7 | 68.4 | 66.2 | 67.9 | 55.6

Table 1: Results on the XQuAD and MLQA dataset in the cross-lingual transfer
settings. The scores are from the best model tuned with the English
development data.

##  3 Adding Entities as Language-Agnostic Features in QA

We evaluate the approach of adding entity embeddings to the input of mLUKE-E
with cross-lingual extractive question answering (QA) tasks. The task is,
given a question and a context passage, to extract the answer span from the
context. Here the entity embeddings provide language-agnostic features and
thus are expected to facilitate cross-lingual transfer learning.

###  3.1 Main Experiments

Datasets. We finetune the pretrained models with the SQuAD 1.1 dataset
(Rajpurkar et al., 2016), and evaluate them with the two multilingual
datasets: XQuAD (Artetxe et al., 2020) and MLQA (Lewis et al., 2020). XQuAD is
created by translating a subset of the SQuAD development set while the source
of MLQA is natural text in Wikipedia. Besides multiple monolingual evaluation
data splits, MLQA also offers data to evaluate generalized cross-lingual
transfer (G-XLT), where the question and context texts are in different
languages.

Models. All QA models used in this experiment follow Devlin et al. (2019). The
model takes the question and context word tokens as input and predicts a score
for each span of the context word tokens. The span with the highest score is
predicted as the answer to the question. The details of finetuning are in
Appendix B.

The mLUKE-E model takes entity tokens as additional features in the input
(Figure 1) to enrich word representations. The entities are automatically
detected using a heuristic string matching based on the original Wikipedia
article from which the dataset instance is created. See Appendix C for more
details.

Results. Table 1 summarizes the model’s F1 scores for each language. On the
effectiveness of entity representations, mLUKE-E performs better than its
word-based counterpart mLUKE-W (0.6 average points improvements in the XQuAD
average score, 0.1 points improvements in MLQA), which indicates the input
entity tokens provide useful features to facilitate cross-lingual transfer.
The usefulness of entities is demonstrated especially in the MLQA’s
generalized cross-lingual transfer setting (full results available in Appendix
D). mLUKE-E exhibits a substantial 1.6 point improvement in the G-XLT average
score over mLUKE-W. This suggests that entity representations are beneficial
in a challenging situation where the model needs to capture language-agnostic
semantics from text segments in different languages.

We also observe that on both XQuAD and MLQA, XLM-R benefits from extra
training (an improvement of 72.0 to 72.4 in the average score from XQuAD and
65 .7 to 67.8 in MLQA). The mLUKE-W model further improves the average score
from XLM-R with extra training (1.2 points improvement in XQuAD and 2.1 points
in MLQA), showing the effectiveness of the MEP task to improve multilingual
representations.

###  3.2 Analysis

How do the entity representations help the model in cross-lingual transfer? In
the mLUKE-E model, the input entity tokens annotate mention spans on which the
model performs prediction. We hypothesize that this allows the encoder to
inject language-agnostic entity knowledge into span representations, which
help better align representations across languages. To support this
hypothesis, we compare the degree of alignment between span representations
before and after adding entity embeddings in the input, i.e., mLUKE-W and
mLUKE-E.

Task. We quantify the degree of alignment as performance on the contextualized
word retrieval (CWR) task (Cao et al., 2020). The task is, given a word within
a sentence in the query language, to find the word with the same meaning in
the context from a candidate pool in the target language.

Dataset. We use the MLQA dev set (Lewis et al., 2020). As MLQA is constructed
from parallel sentences mined from Wikipedia, some sentences and answer spans
are aligned and thus the dataset can be easily adapted for the CWR task. As
the query and target word, we use the answer span444Answer spans are not
necessarily a word, but here we generalize the task as span retrieval for our
purpose. annotated in the dataset, which is also parallel across the
languages. We use the English dataset as the query language and other
languages as the target. We discard query instances that do not have their
parallel data in the target language. The candidate pool is all answer spans
in the target language data.

Models. We evaluate mLUKE-W and mLUKE-E without fine-tuning. The retrieval is
performed by ranking the cosine similarity of contextualized span
representations, which is computed by mean-pooling the output word vectors in
the span.

Results. Table 2 shows the performance in terms of the mean reciprocal rank
score. We observe that the scores of mLUKE-E are higher than mLUKE-W across
all the languages. This demonstrates that adding entities improves the degree
of alignment of span representations, which explains the improvement of
mLUKE-E in the cross-lingual QA task.

| ar | de | es | hi | vi | zh | avg.
---|---|---|---|---|---|---|---
mLUKE-W | 55.6 | 66.1 | 68.4 | 60.4 | 69.7 | 56.1 | 62.7
mLUKE-E | 56.9 | 68.1 | 70.4 | 61.5 | 71.2 | 60.0 | 64.7
Table 2: The mean reciprocal rank score of the CWR task with the MLQA dev set.

##  4 The Entity MASK Token as Feature Extractor in Relation Extraction and
NER

In this section, we evaluate the approach of using the entity [MASK] token to
extract features from mLUKE-E for two entity-related tasks: relation
extraction and named entity recognition.

We formulate both tasks as the classification of mention spans. The baseline
models extract the feature of spans as the contextualized representations of
word tokens, while mLUKE-E extracts the feature as the contextualized
representations of the special language-independent entity tokens associated
with the mentions (Figure 1). We demonstrate that this approach consistently
improves the performance in cross-lingual transfer.

###  4.1 Relation Extraction

Relation Extraction (RE) is a task to determine the correct relation between
the two (head and tail) entities in a sentence.

Datasets. We fine-tune the models with the English KBP-37 dataset (Zhang and
Wang, 2015) and evaluate the models with the RELX dataset (Köksal and Özgür,
2020), which is created by translating a subset of 502 sentences from KBP-37’s
test set into four different languages. Following Köksal and Özgür (2020), we
report the macro average of F1 scores of the 18 relations.

Models. In the input text, the head and tail entities are surrounded with
special markers (<ent>, <ent2>). The baseline models extract the feature
vectors for the entities as the contextualized vector of the first marker
followed by their mentions. The two entity features are concatenated and fed
into a linear classifier to predict their relation.

For mLUKE-E, we introduce two special entities, [HEAD] and [TAIL], to
represent the head and tail entities (Yamada et al., 2020). Their embeddings
are initialized with the entity [MASK] embedding. They are added to the input
sequence being associated with the entity mentions in the input, and their
contextualized representations are extracted as the feature vectors. As with
the word-based models, the features are concatenated and input to a linear
classifier.

| RE | NER
---|---|---
| en | de | es | fr | tr | avg. | en | de | du | es | avg.
mBERT | 65.0 | 57.3 | 61.6 | 58.9 | 56.2 | 59.8 | 89.7 | 70.0 | 75.2 | 77.1 | 78.0
XLM-R | 66.5 | 60.8 | 62.9 | 60.9 | 57.7 | 61.7 | 91.5 | 74.3 | 80.7 | 79.8 | 81.6
\+ extra training | 67.0 | 61.3 | 62.9 | 64.3 | 61.9 | 63.5 | 91.8 | 75.7 | 80.3 | 79.8 | 81.9
mLUKE-W | 68.7 | 64.3 | 65.8 | 62.1 | 65.0 | 65.2 | 91.6 | 75.1 | 80.2 | 79.2 | 81.5
mLUKE-E | 69.3 | 64.5 | 65.2 | 64.7 | 68.7 | 66.5 | 93.6 | 77.2 | 81.8 | 77.7 | 82.6
Table 3: Results on relation extraction (RE) and named entity recognition
(NER) with the RELX and CoNLL datasets.

###  4.2 Named Entity Recognition

Named Entity Recognition (NER) is the task to detect entities in a sentence
and classify their type. We use the CoNLL-2003 English dataset (Tjong Kim Sang
and De Meulder, 2003) as the training data, and evaluate the models with the
CoNLL-2003 German dataset and the CoNLL-2002 Spanish and Dutch dataset (Tjong
Kim Sang, 2002).

Models. We adopt the model of Sohrab and Miwa (2018) as the baseline model,
which enumerates all possible spans in a sentence and classifies them into the
target entity types or non-entity type. In this experiment, we enumerate spans
with at most 16 tokens. For the baseline models, the span features are
computed as the concatenation of the word representations of the first and
last tokens. The span features are fed into a linear classifier to predict
their entity type.

The input of mLUKE-E contains the entity [MASK] tokens associated with all
possible spans. The span features are computed as the contextualized
representations of the entity [MASK] tokens. The features are input to a
linear classifier as with the word-based models.

###  4.3 Main Results

The results are shown in Table 3. In cross-lingual transfer settings, mLUKE-E
performs the best overall except for Spanish (es). The mLUKE-E model
outperforms its word-based counterpart mLUKE-W in average score by 1.3 points
in RE and 1.1 points in NER, which shows entity-based feature extraction is
useful in cross-lingual tasks.

We also observe that XLM-R benefits from extra training (1.8 average points
improvement in RE and 0.3 points in NER).

###  4.4 Analysis

The performance gain of mLUKE-E over mLUKE-W can be partly explained as the
entity [MASK] token extracts better features for predicting entity attributes
because it resembles how mLUKE is pretrained with the MEP task. We hypothesize
that there exists another factor for the improvement in cross-lingual
performance: language neutrality of representations.

Unlike some word tokens, entity tokens including the entity [MASK] token are
shared across languages and their contextualized representations may be less
affected by the difference of input languages, resulting in features that
generalize well for cross-lingual transfer. To find out if the entity-based
features are actually more language-independent than word-based features, we
evaluate the modularity (Fujinuma et al., 2019) of the features extracted for
the RELX dataset.

Modularity is computed for the $k$-nearest neighbor graph of embeddings and
measures the degree to which embeddings tend to form clusters within the same
language. We refer readers to Fujinuma et al. (2019) for how to compute the
metric. Note that the maximum value of modularity is 1 and 0 means the
embeddings are completely randomly distributed regardless of language.

| de | es | fr | tr
---|---|---|---|---
mLUKE-W | 0.71 | 0.74 | 0.74 | 0.84
mLUKE-E | 0.25 | 0.28 | 0.24 | 0.36
Table 4: The modularity is computed with two different features from the same
mLUKE model. The data are from a pair of English and another language in the
RELX dataset.

We compute the modularity of the concatenated vectors of head and tail entity
features from the English and another language’s dataset. The results from
mLUKE-W and mLUKE-E before fine-tuning are shown in Table 4. The modularity of
mLUKE-E is much lower than mLUKE-W demonstrating that entity-based features
are more language-neutral. However, with entity-based features, the
modularities are still greater than zero. In particular, the modularity
computed with Turkish, which is the most distant language from English here,
is significantly higher than the others, indicating that the contextualized
entity-based features are still somewhat language-dependent.

##  5 Cloze Prompt Task with Entity Representations

In this section, we show that using the entity representations is effective in
a cloze prompt task (Liu et al., 2021) with the mLAMA dataset (Kassner et al.,
2021). The task is, given a cloze template such as “[X] was born in [Y]” with
[X] filled with an entity (e.g., Mozart), to predict a correct entity in [Y]
(e.g., Austria). We adopt the typed querying setting (Kassner et al., 2021),
where a template has a set of candidate answer entities and the prediction
becomes the one with the highest score assigned by the language model.

Model. As in Kassner et al. (2021), the word-based baseline models compute the
candidate score as the log-probability from the MLM classifier. When a
candidate entity in [Y] is tokenized into multiple tokens, the same number of
the word [MASK] tokens are placed in the input sequence, and the score is
computed by taking the average of the log-probabilities for its individual
tokens.

On the other hand, mLUKE-E computes the log-probability of the candidate
entity in [Y] with the entity [MASK] token. Each candidate entity is
associated with an entity in mLUKE’s entity vocabulary via string matching.
The input sequence has the entity [MASK] token associated with the word [MASK]
tokens in [Y], and the candidate score is computed as the log-probability from
the MEP classifier. When the candidate entity is not found in the entity
vocabulary, the log-probability from the word [MASK] tokens are used instead.
We also try additionally appending the entity token of [X] to the input
sequence if the entity is found in the vocabulary.

| ar | en | fi | fr | id | ja | ru | vi | zh | avg.
---|---|---|---|---|---|---|---|---|---|---
mBERT | 15.1 | 34.8 | 19.2 | 27.1 | 37.4 | 14.2 | 17.4 | 36.5 | 24.2 | 25.1
XLM-R | 14.9 | 24.2 | 14.5 | 16.1 | 24.6 | 11.4 | 12.6 | 23.0 | 15.5 | 17.4
\+ extra training | 20.7 | 31.6 | 19.2 | 25.0 | 38.2 | 19.1 | 20.6 | 33.8 | 28.1 | 26.3
mLUKE-W | 21.3 | 27.1 | 15.9 | 23.0 | 36.6 | 18.0 | 19.6 | 31.0 | 26.9 | 24.4
mLUKE-E ([Y]) | 21.1 | 33.4 | 22.9 | 25.3 | 38.8 | 24.2 | 20.0 | 38.3 | 28.4 | 28.0
mLUKE-E ([X] & [Y]) | 26.3 | 40.7 | 30.7 | 30.9 | 48.4 | 28.8 | 24.6 | 46.5 | 35.1 | 34.7
Table 5: The top-1 accuracies from 9 languages from the mLAMA dataset.

Results. We experiment in total with 16 languages which are available both in
the mLAMA dataset and the mLUKE’s entity vocabulary. Here we only present the
results from 9 languages (the full results in Appendix E), as we can make
similar observations with the other languages. Table 5 summarizes the top-1
accuracy results.

We observe that XLM-R performs notably worse than mBERT as mentioned in
Kassner et al. (2021). However, with extra training with the Wikipedia corpus,
XLM-R shows a significant 8.9 points improvement in the average score and
outperforms mBERT (26.3 vs. 25.1). We conjecture that this shows the
importance of the training corpus for this task. The original XLM-R is only
trained with the CommonCrawl corpus (Conneau et al., 2020a), text scraped from
a wide variety of web pages, while mBERT and XLM-R + training are trained on
Wikipedia. The performance gaps indicate that Wikipedia is particularly useful
for the model to learn factual knowledge.

The mLUKE-W model lags behind XLM-R + training by 1.9 average points but we
can see 1.7 points improvement from XLM-R + training to mLUKE-E ([Y])
indicating entity representations are more suitable to elicit correct factual
knowledge from mLUKE than word representations. Adding the entity
corresponding to [X] to the input when available in the entity vocabulary
(mLUKE-E ([X] & [Y])) further pushes the performance by 6.7 points to 34.7 %,
which further demonstrates the effectiveness of entity representations.

Analysis of Language Bias. Kassner et al. (2021) notes that the prediction of
mBERT is biased by the input language. For example, when queried in Italian
(e.g., “[X] e stato creato in [MASK].”), the model tends to predict entities
that often appear in Italian text (e.g., Italy) for any question to answer
location. We expect that using entity representations would reduce language
bias since entities are more language-independent than words.

We qualitatively assess the degree of language bias in the models looking at
their false-positive predictions. We show the top false-positive prediction
for the template “[X] was founded in [Y].” for each model in Table 6, together
with the top-$1$ false-positive ratio, that is, the ratio of the number of the
most common false-positive prediction to the total false predictions, which
indicates how much the false predictions are dominated by few frequent
entities.

| en | ja | fr
---|---|---|---
mBERT | The Bahamas, 41% (355/870) | Japan, 82% (361/439) | Pays-Bas, 71% (632/895)
XLM-R | London, 78% (664/850) | Japan, 99% (437/440) | Allemagne, 96% (877/916)
\+ extra training | Australia, 27% (247/899) | Japan, 99% (437/442) | Allemagne, 93% (854/917)
mLUKE-W | Germany, 22% (198/895) | Japan, 97% (428/442) | Allemagne, 99% (906/918)
mLUKE-E ([Y]) | London, 37% (310/846) | Japan, 56% (241/430) | Suède, 40% (362/908)
mLUKE-E ([X] & [Y]) | London, 27% (213/797) | Japan, 44% (176/401) | Suède, 30% (266/895)
Table 6: The top false-positive predictions in three languages for the
template “[X] was founded in [Y].” for each model. The predictions in the
original language are translated into English.

The examples show that the different models exhibit bias towards different
entities as in English and French, although in Japanese the model consistently
tends to predict Japan. Looking at the degree of language bias, mLUKE-E ([X] &
[Y]) exhibits lower top-$1$ false-positive ratios overall (27% in fr, 44% in
ja, and 30% in fr), which indicates using entity representations reduces
language bias. However, lower language bias does not necessarily mean better
performance: in French (fr), mLUKE gives a lower top-$1$ false-positive ratio
than mBERT (30% vs. 71%) but their numbers of total false predictions are the
same (895). Language bias is only one of several factors in the performance
bottleneck.

##  6 Related Work

###  6.1 Multilingual Pretrained Language Models

Multilingual pretrained language models have recently seen a surge of interest
due to their effectiveness in cross-lingual transfer learning (Conneau and
Lample, 2019; Liu et al., 2020). A straightforward way to train such models is
multilingual masked language modeling (mMLM) (Devlin et al., 2019; Conneau et
al., 2020a), i.e., training a single model with a collection of monolingual
corpora in multiple languages. Although models trained with mMLM exhibit a
strong cross-lingual ability without any cross-lingual supervision
(Karthikeyan K and Roth, 2020; Conneau et al., 2020b), several studies aim to
develop better multilingual models with explicit cross-lingual supervision
such as bilingual word dictionaries (Conneau et al., 2020b) or parallel
sentences (Conneau and Lample, 2019). In this study, we build a multilingual
pretrained language model on the basis of XLM-RoBERTa (Conneau et al., 2020a),
trained with mMLM as well as the masked entity prediction (MEP) (Yamada et
al., 2020) with entity representations.

###  6.2 Pretrained Language Models with Entity Knowledge

Language models trained with a large corpus contain knowledge about real-world
entities, which is useful for entity-related downstream tasks such as relation
classification, named entity recognition, and question answering. Previous
studies have shown that we can improve language models for such tasks by
incorporating entity information into the model (Zhang et al., 2019; Peters et
al., 2019; Wang et al., 2021; Xiong et al., 2020; Févry et al., 2020; Yamada
et al., 2020).

When incorporated into multilingual language models, entity information can
bring another benefit: entities may serve as anchors for the model to align
representations across languages. Multilingual knowledge bases often offer
mappings between different surface forms across languages for the same entity.
A typical data source is Wikipedia, which covers more than 100 languages and
the text in the Wikipedia articles contains numerous entity annotations
(hyperlinks). Iacer Calixto and Pasini (2021) fine-tuned the top two layers of
multilingual BERT by predicting language-agnostic entity ID from hyperlinks in
Wikipedia articles. As our concurrent work, Xiaoze Jian and Duan (2021)
trained a model based on XLM-RoBERTa with an entity prediction task along with
an object entailment prediction task. While the previous studies focus on
improving cross-lingual language representations by pretraining with entity
information, our work investigates a multilingual model not only pretrained
with entities but also explicitly having entity representations and how to
extract better features from such model.

##  7 Conclusion

In this study, we investigated the effectiveness of entity representations in
multilingual language models. Our pretrained model, mLUKE, not only exhibits
strong empirical results only with the word inputs (mLUKE-W) but also shows
even better performance with the entity representations (mLUKE-E) in cross-
lingual transfer tasks. We also show that a cloze-prompt-style fact completion
task can effectively be solved with the query and answer space in the entity
vocabulary. Our results suggest a promising direction to pursue further on how
to leverage entity representations in multilingual tasks.

## References

  * Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020.  On the Cross-lingual Transferability of Monolingual Representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Cao et al. (2020) Steven Cao, Nikita Kitaev, and Dan Klein. 2020.  Multilingual Alignment of Contextual Word Representations.
  * Conneau et al. (2020a) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020a.  Unsupervised Cross-lingual Representation Learning at Scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  Cross-lingual Language Model Pretraining.  In _Advances in Neural Information Processing Systems_ , volume 32.
  * Conneau et al. (2020b) Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020b.  Emerging Cross-lingual Structure in Pretrained Language Models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Cui et al. (2021) Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. 2021.  Template-Based Named Entity Recognition Using BART.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1_.
  * Doddapaneni et al. (2021) Sumanth Doddapaneni, Gowtham Ramesh, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh M. Khapra. 2021.  A Primer on Pretrained Multilingual Language Models.  _ArXiv_ , abs/2107.00676.
  * Févry et al. (2020) Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. 2020.  Entities as Experts: Sparse Memory Access with Entity Supervision.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_.
  * Fujinuma et al. (2019) Yoshinari Fujinuma, Jordan Boyd-Graber, and Michael J. Paul. 2019.  A Resource-Free Evaluation Metric for Cross-Lingual Word Embeddings Based on Graph Modularity.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
  * Iacer Calixto and Pasini (2021) Alessandro Raganato Iacer Calixto and Tommaso Pasini. 2021.  Wikipedia Entities as Rendezvous across Languages: Grounding Multilingual Language Models by Predicting Wikipedia Hyperlinks.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics_.
  * Karthikeyan K and Roth (2020) Stephen Mayhew Karthikeyan K, Zihan Wang and Dan Roth. 2020.  Cross-Lingual Ability of Multilingual BERT: An Empirical Study.  In _International Conference on Learning Representations_.
  * Kassner et al. (2021) Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021.  Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_.
  * Köksal and Özgür (2020) Abdullatif Köksal and Arzucan Özgür. 2020.  The RELX Dataset and Matching the Multilingual Blanks for Cross-Lingual Relation Classification.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations.  In _International Conference on Learning Representations_.
  * Lewis et al. (2020) Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020\.  MLQA: Evaluating Cross-lingual Extractive Question Answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Liu et al. (2021) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021.  Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing.  _ArXiv_ , abs/2107.13586.
  * Liu et al. (2019) Y. Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _ArXiv_ , abs/1907.11692.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual Denoising Pre-training for Neural Machine Translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled Weight Decay Regularization.  In _International Conference on Learning Representations_.
  * Peters et al. (2019) Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019.  Knowledge Enhanced Contextual Word Representations.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing_.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language Models as Knowledge Bases?  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing_.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ Questions for Machine Comprehension of Text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_.
  * Sohrab and Miwa (2018) Mohammad Golam Sohrab and Makoto Miwa. 2018.  Deep Exhaustive Model for Nested Named Entity Recognition.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_.
  * Tjong Kim Sang (2002) Erik F. Tjong Kim Sang. 2002.  Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition.  In _COLING-02: The 6th Conference on Natural Language Learning 2002_.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention Is All You Need.  In _Advances in Neural Information Processing Systems_ , volume 30.
  * Wang et al. (2021) Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juan-Zi Li, and J. Tang. 2021\.  KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation.  _Transactions of the Association for Computational Linguistics_ , 9:176–194.
  * Xiaoze Jian and Duan (2021) Weizhu Chen Xiaoze Jian, Yaobo Liang and Nan Duan. 2021.  XLM-K: Improving Cross-Lingual Language Model Pre-Training with Multilingual Knowledge.  _ArXiv_ , abs/2109.12573.
  * Xiong et al. (2020) Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2020.  Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model.  In _International Conference on Learning Representations_.
  * Yamada et al. (2020) Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020\.  LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_.
  * Zhang and Wang (2015) Dongxu Zhang and Dong Wang. 2015.  Relation Classification via Recurrent Neural Network.  _ArXiv_ , abs/1508.01006.
  * Zhang et al. (2019) Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019.  ERNIE: Enhanced Language Representation with Informative Entities.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.

## Appendix for “mLUKE: The Power of Entity Representations in Multilingual
Pretrained Language Models”

##  Appendix A Details of pretraining

Dataset. We download the Wikipedia dumps from December 1st, 2020. We show the
24 languages included in the dataset on Table 7, along with the data size and
the number of entities in the vocabulary.

Language Code | size | # entities in vocab | Language Code | size | # entities in vocab
---|---|---|---|---|---
ar | 851M | 427,460 | ko | 537M | 378,399
bn | 117M | 62,595 | nl | 1.1G | 483,277
de | 3.5G | 540,347 | pl | 1.3G | 489,109
el | 315M | 135,277 | pt | 1.0G | 537,028
en | 6.9G | 613,718 | ru | 2.5G | 529,171
es | 2.1G | 587,525 | sv | 1.1G | 390,313
fi | 480M | 300,333 | sw | 27M | 30,129
fr | 3.1G | 630,355 | te | 66M | 14,368
hi | 90M | 54,038 | th | 153M | 100,231
id | 327M | 217,758 | tr | 326M | 297,280
it | 1.9G | 590,147 | vi | 516M | 263,424
ja | 2.3G | 369,470 | zh | 955M | 332,970
|  |  | Total | 31.4G | 8,374,722
Table 7: Training Data Statistics: the size of training data, and the number
of entities found in the 1.2M entity vocabulary.

Optimization. We optimize the mLUKE model for 1M steps in total using AdamW
(Loshchilov and Hutter, 2019) with learning rate warmup and linear decay of
the learning rate. The pretraining consists of two stages: (1) in the first
500K steps, we update only those parameters that are randomly initialized
(e.g., entity embeddings); (2) we update all parameters in the remaining 500K
steps. The learning rate scheduler is reset at each training stage. The
detailed hyper-parameters are shown in Table 8.

Name | Value Name | Value |
---|---|---|---
Maximum word length | 512 | Mask probability for entities | 15%
Batch size | 2048 | The size of word embeddings | 768
Peak learning rate | 1e-4 | The size of entity embeddings | 256
Peak learning rate (first 500K steps) | 5e-4 | Dropout | 0.1
Learning rate decay | linear | Weight decay | 0.01
Warmup steps | 2500 | Adam $\beta_{1}$ | 0.9
Mask probability for words | 15% | Adam $\beta_{2}$ | 0.999
Random-word probability for words | 10% | Adam $\epsilon$ | 1e-6
Unmasked probability for words | 10% | Gradient clipping | none
Table 8: Hyper-parameters used to pretrain mLUKE.

##  Appendix B Details of Downstream Experiments

For each downstream task, we perform hyperparameter tuning for all the models
with the same computational budget to ensure fair comparison. The models are
optimized with the AdamW optimizer (Loshchilov and Hutter, 2019) with the
weight decay term set to 0.01 and a linear warmup scheduler. The learning rate
is linearly increased to a specified value in the first 6 % of training steps,
and then gradually decreased to zero towards the end. Table 9 summarizes the
task-specific hyperparameter search spaces.

|  | QA
---
(SQuAD)
| Relation Classification
---
(KBP37)
| NER
---
(CoNLL 2003)
Learning rate | 2e-5 | 2e-5 | 2e-5
Batch size | {16, 32} | {4, 8, 16} | {4, 8, 16}
Epochs | 2 | 5 | 5
# of random seeds | 3 | 3 | 3
Table 9: The hyperparameter search spaces of downstream experiments.

##  Appendix C Detecting Entities in the QA datasets

For each question–passage pair in the QA datasets, we first create a mapping
from the entity mention strings (e.g., “U.S.”) to their referent Wikipedia
entities (e.g., United States) using the entity hyperlinks on the source
Wikipedia page of the passage. We then perform simple string matching to
extract all entity names in the question and the passage and treat all matched
entity names as entity annotations for their referent entities. We ignore an
entity name if the name refers to multiple entities on the page. Further, to
reduce noise, we also exclude an entity name if its link probability, the
probability that the name appears as a hyperlink in Wikipedia, is lower than
1%.

The XQuAD datasets are created by translating English Wikipedia articles into
target languages. For those translated data, we translate the mention-entity
mapping created from the source English articles by the following procedure:
for all the entities found in the source article, we map the entity into the
corresponding article in the target language through inter-language links, and
then collect its possible mention strings (i.e.,  hyperlinks to the article)
from a Wikipedia dump of the target language; the entities in the source
article and the collected mention strings form the mention-entity mapping for
the translated article.

##  Appendix D Full Results of MLQA

c/q | en | es | de | ar | hi | vi | zh
---|---|---|---|---|---|---|---
en | 79.1 | 65.4 | 63.4 | 37.9 | 29.7 | 47.1 | 43.2
es | 67.7 | 65.9 | 58.2 | 38.2 | 24.4 | 43.6 | 39.5
de | 61.7 | 55.9 | 58.6 | 32.3 | 29.7 | 38.4 | 36.8
ar | 49.9 | 43.2 | 44.6 | 48.6 | 23.4 | 29.4 | 27.1
hi | 47.0 | 37.8 | 39.1 | 26.2 | 44.8 | 28.0 | 23.0
vi | 59.9 | 49.4 | 48.6 | 26.7 | 25.6 | 58.5 | 40.7
zh | 55.3 | 44.2 | 45.3 | 28.3 | 22.7 | 38.7 | 58.1

Table 10: MLQA full results of mBERT

.

c/q | en | es | de | ar | hi | vi | zh
---|---|---|---|---|---|---|---
en | 79.6 | 52.3 | 59.6 | 30.8 | 43.2 | 40.0 | 36.0
es | 67.0 | 67.7 | 52.0 | 25.2 | 31.8 | 32.9 | 31.5
de | 59.5 | 41.7 | 62.1 | 22.2 | 27.8 | 29.2 | 29.5
ar | 49.6 | 23.2 | 30.9 | 55.8 | 10.6 | 11.6 | 10.3
hi | 58.5 | 34.6 | 42.3 | 17.8 | 59.8 | 22.4 | 23.0
vi | 61.1 | 28.1 | 39.5 | 17.0 | 27.5 | 65.2 | 26.5
zh | 55.2 | 22.7 | 28.1 | 9.26 | 21.1 | 17.5 | 62.4
Table 11: MLQA full results of XLM-R

c/q | en | es | de | ar | hi | vi | zh
---|---|---|---|---|---|---|---
en | 81.3 | 71.2 | 70.1 | 40.6 | 52.3 | 54.8 | 48.2
es | 70.6 | 69.8 | 66.2 | 43.3 | 47.9 | 52.8 | 49.0
de | 64.4 | 60.4 | 64.9 | 36.8 | 42.3 | 44.3 | 42.9
ar | 59.3 | 52.3 | 52.2 | 54.8 | 30.3 | 37.1 | 31.5
hi | 65.0 | 56.5 | 56.8 | 33.8 | 59.3 | 43.0 | 39.9
vi | 67.0 | 57.1 | 58.2 | 31.7 | 43.8 | 65.5 | 44.0
zh | 62.4 | 53.7 | 54.2 | 33.3 | 40.2 | 44.8 | 64.2
Table 12: MLQA full results of XLM-R + training

c/q | en | es | de | ar | hi | vi | zh
---|---|---|---|---|---|---|---
en | 81.2 | 69.5 | 69.1 | 53.6 | 60.8 | 60.4 | 58.4
es | 70.3 | 69.6 | 65.5 | 52.1 | 52.9 | 56.1 | 56.4
de | 64.7 | 59.8 | 65.3 | 45.4 | 48.9 | 49.9 | 49.3
ar | 60.4 | 52.3 | 54.3 | 60.3 | 34.0 | 43.4 | 41.3
hi | 65.5 | 56.9 | 58.3 | 35.4 | 63.1 | 49.0 | 44.6
vi | 66.8 | 54.4 | 57.1 | 39.7 | 49.3 | 68.3 | 52.4
zh | 63.2 | 55.1 | 56.6 | 39.8 | 43.3 | 49.6 | 66.1
Table 13: MLQA full results of mLUKE-W

c/q | en | es | de | ar | hi | vi | zh
---|---|---|---|---|---|---|---
en | 80.8 | 71.3 | 69.9 | 55.9 | 61.9 | 62.8 | 62.1
es | 70.6 | 69.9 | 66.4 | 52.6 | 53.7 | 57.6 | 58.0
de | 65.2 | 61.2 | 65.4 | 47.2 | 49.3 | 51.8 | 51.7
ar | 61.1 | 54.6 | 56.9 | 60.7 | 39.5 | 47.0 | 44.8
hi | 65.1 | 58.4 | 59.2 | 38.3 | 63.7 | 50.5 | 46.2
vi | 66.7 | 56.5 | 59.5 | 44.3 | 51.1 | 68.4 | 54.2
zh | 62.7 | 56.3 | 56.2 | 41.1 | 44.3 | 51.7 | 66.2
Table 14: MLQA full results of mLUKE-E

##  Appendix E Full Results of mLAMA

| ar | bn | de | el | en | es | fi | fr
---|---|---|---|---|---|---|---|---
mBERT | 15.1 | 12.7 | 28.6 | 19.4 | 34.8 | 30.2 | 19.2 | 27.1
XLM-R | 14.9 | 7.5 | 18.4 | 12.7 | 24.2 | 18.5 | 14.5 | 16.1
\+ training | 20.7 | 14.0 | 29.3 | 18.2 | 31.6 | 26.4 | 19.2 | 25.0
mLUKE-W | 21.3 | 12.9 | 25.7 | 17.5 | 27.1 | 23.3 | 15.9 | 23.0
mLUKE-E ([Y]) | 25.6 | 21.6 | 32.9 | 25.2 | 34.9 | 28.5 | 24.7 | 27.7
mLUKE-E ([X] & [Y]) | 37.3 | 32.3 | 43.7 | 34.4 | 43.2 | 36.4 | 35.3 | 34.2

| id | ja | ko | pl | pt | ru | vi | zh | avg.
---|---|---|---|---|---|---|---|---|---
mBERT | 37.4 | 14.2 | 17.8 | 21.9 | 32.0 | 17.4 | 36.5 | 24.2 | 24.3
XLM-R | 24.6 | 11.4 | 10.9 | 16.6 | 22.2 | 12.6 | 23.0 | 15.5 | 16.5
\+ training | 38.2 | 19.1 | 21.4 | 20.5 | 29.6 | 20.6 | 33.8 | 28.1 | 24.7
mLUKE-W | 36.6 | 18.0 | 17.9 | 20.2 | 29.4 | 19.6 | 31.0 | 26.9 | 22.9
mLUKE-E ([Y]) | 35.3 | 27.2 | 26.3 | 25.7 | 34.7 | 23.8 | 39.1 | 29.5 | 28.9
mLUKE-E ([X] & [Y]) | 47.6 | 37.7 | 41.6 | 37.7 | 44.8 | 31.4 | 50.1 | 41.6 | 39.3

Table 15: The average of Top-1 accuracies from 16 languages from the mLAMA
dataset.