# Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge Retrieval
from Foundation Language Models

Tim Schott, Daniel Furman, and Shreshta Bhat
University of California, Berkeley
{timschott, daniel_furman<EMAIL_ADDRESS>Details of author
contributions listed in the appendix.

###### Abstract

In this work, we evaluate the capacity for foundation models to retrieve
encyclopedic knowledge across a wide range of languages, topics, and contexts.
To support this effort, we 1) produce a new dataset containing 303k factual
associations in 20 different languages, 2) formulate a new counterfactual
knowledge assessment, Polyglot or Not, and 3) benchmark 5 foundation models in
a multilingual setting and a diverse set of 20 models in an English-only
setting. We observed significant accuracy differences in models of interest,
with Meta’s LLaMA topping both the multilingual and English-only assessments.
Error analysis reveals a significant deficiency in LLaMA’s ability to retrieve
facts in languages written in the Cyrillic script and gaps in its
understanding of facts based on the location and gender of entailed subjects.
Ultimately, we argue that the promise of utilizing foundation language models
as bonafide polyglots is greatly diminished when they are tasked with
retrieving information in languages other than English. Supporting code and
data are openly released.

##  1 Introduction

Can foundation language models be used as multilingual knowledge bases? Here,
we investigate the extent to which today’s foundation language models perform
as polyglots. Large Language Models (LLMs) of this class are typically trained
in a self-supervised fashion on massive corpora of texts Radford et al.
(2018); Lewis et al. (2020); Raffel et al. (2020); Brown et al. (2020); Black
et al. (2022); OpenAI (2023); Touvron et al. (2023). Foundation models warrant
further study because they typify a paradigm shift in the field: all-purpose
models performing new tasks robustly in a zero or few-shot setting — without
additional fine-tuning Brown et al. (2020). This concentration of usage and
ostensible credibility, however, engenders some negative consequences, like
foundation models inheriting the biases they come across during training
Bolukbasi et al. (2016); Argyle et al. (2022). What’s more, foundation models
spend the majority of their training phase absorbing information in English;
Touvron et al. (2023)’s LLaMA devotes two-thirds of its language modeling
dataset to a English-only subset of CommonCrawl Wenzek et al. (2020). As a
result, foundation language models are potentially deficient when performing
tasks in languages other than English Devlin et al. (2019); Kassner et al.
(2021).

Here, we perform tests to determine whether foundation models can be utilized
as multilingual knowledge bases. To do so, we create a new assessment for
probing factual knowledge in LLMs, Polyglot or Not. Our approach finds roots
in the Contrastive Knowledge Assessment set forth by Dong et al. (2022), as
well as Petroni et al. (2019)’s usage of “fill-in-the-blank” cloze statements
to inspect the suitability of language models for use as knowledge bases.

We break new ground by scaling a counterfactual knowledge assessment to a
broad group of foundation language models across 20 different languages. All
told, we evaluate 5 open-source foundation models of interest using a novel
dataset of 303k multilingual fact-completions. We demonstrate that LLaMA
greatly outperforms comparable models on multilingual fact retrieval. We also
run a monolingual version of the test for 20 models like GPT-NeoX Black et al.
(2022) on the 26k English fact-completions included in our dataset. LLaMA,
again, leads the pack.

From there, we also analyze the cases in which LLaMA fails, conjecturing that
an analysis of its errors in particular ought to shed the most light about the
state of today’s best foundation models and their capacity to adequately
retrieve (or not) factual associations across different languages. We find a
significant drop-off in the model’s ability to retrieve facts in languages
written in the Cyrillic script, plus significant differences in the richness
of relevant training information from each language’s Wikipedia, with “denser”
pages strongly correlated with high performance on our assessment. We also
investigate the level of accuracy across the natural groupings in our dataset,
entities (subjects and objects) and relations (categories). To complement our
quantitative analyses, we qualitatively assess LLaMA’s mistakes with a
particular focus on data quality and semantic understanding.

##  2 Related Work

### Language Models as Knowledge Bases

LLMs are capable of encoding an impressive amount of facts about the world
Roberts et al. (2020). Regardless of their underlying architecture, LLMs
encode facts as key-value pairs within their memory Geva et al. (2022); Meng
et al. (2022b). This capability enables practical use; expose an LLM to enough
associations during self-supervised training, and it will excel at a variety
of downstream tasks Kaplan et al. (2020). However, alongside useful facts,
models can memorize dubious information during the self-supervised training
process Bender et al. (2021), especially if the text corpora utilized are
unrepresentative of the world Dodge et al. (2021).

### Assessing LLM Knowledge

We formalize factual associations with the triplet $\langle s,r,o\rangle$
where $s$ and $o$ denote the subject and object entity and $r$ is a linking
relation Elsahar et al. (2018). Thus, we can transform the fact "Paris is the
capital of France" into $\langle Paris,capital\>of,France\rangle$ where Paris
corresponds to $s$, "capital of" corresponds to $r$, and France corresponds to
$o$. Following previous work from Petroni et al. (2019), we prompt a model $M$
with a cloze style fill-in-the-blank statement, masking out $o$. To assess if
$M$ knows a piece of knowledge, we can calculate $P_{M}(o\mid s,r)$. Typical
evaluation of this quantity involves a rank-based reward Petroni et al. (2019)
to the model for correctly placing $o$ higher-up in the schedule of potential
next tokens. However, this practice encodes a number of pitfalls, including a
bias towards predicting tokens that appear frequently in the training data and
unsatisfactory outcomes for "inexhaustible" questions that have numerous
correct answers Dong et al. (2022).

### Contrastive Knowledge Assessment

For our benchmark, we utilize a version of the Contrastive Knowledge
Assessment (CKA) technique proposed in previous work Dong et al. (2022).
Erroneous "counterfactuals" $\langle s,r,o^{\prime}\rangle$ like $\langle
Paris,capital\>of,Italy^{\prime}\rangle$ are used to assess a model $M$’s
understanding of $\langle s,r,o\rangle$. If $M$ truly knows the fact,
$P_{M}(o\mid s,r)$ should be larger than $P_{M}(o^{\prime}\mid s,r)$ Dong et
al. (2022). Formally, CKA measures whether $M$ correctly understands a fact
$\langle s,r,o\rangle$ for a model by:

$\displaystyle\text{CKA}_{\text{M}}(s,r,o)$ $\displaystyle=\frac{P_{M}(o\mid
s,r)}{\mathbb{E}_{o^{\prime}}[\,{P_{M}(o^{\prime}\mid s,r)}]\,}$

When the likelihood of the correct association outweighs its belief in the
counterfactuals ($\text{CKA}_{\text{M}}(s,r,o)>1$), we conclude the model
understands the factual association. This approach alleviates the issues that
arise from simply ranking a model’s vocabulary-wide token probabilities at
inference because incorporating counterfactuals encourages the model to
demonstrate generalized knowledge base understanding. Our work builds off Dong
et al. (2022) by applying CKA to a multilingual dataset for the first time.

### Multilingual Knowledge Assessments

Previous research has indeed made strides towards investigating multilingual
knowledge in LLMs. For example, Jiang et al. (2020) and Kassner et al. (2021)
apply translated cloze statements to multilingual BERT Devlin et al. (2019)
and RoBERTa Liu et al. (2019); Conneau et al. (2020) models. Our work builds
off of their research in a few different ways. To start, we opt for a
contrastive approach that utilizes counterfactual objects to elicit a
knowledge prediction from the model. We believe this provides a more coherent
picture of how well a model grasps a concept, which is especially important as
we make connections across languages with limited training data. Additionally,
we inspect a diverse set of LLMs rather than restricting our experiments to
one particular architecture. Tables 1 and 3 include 14 “decoder-only” models
(e.g., LLaMA), 3 “encoder-decoder” models (e.g., T5), and 3 “encoder-only”
models (e.g., BERT). It’s worth noting that the LLM nomenclature obfuscates
the similarity between architectures: "decoder-only" actually means "auto-
regressive encoder-decoder", "encoder-decoder" refers to an "encoder with an
auto-regressive decoder", and "encoder-only" actually means "encoder with a
non-auto-regressive decoder." Due to these similarities, it is sensible and
informative to include LLM variants from each of these architectures in
evaluation tests such as ours.

##  3 Task

We formulate our task using a cloze test: given some context, predict the next
tokens. We perform the multilingual, Polyglot or Not evaluation on 5
foundation models in 20 different languages. We also scored 20 foundation
models on English-only data. All evaluated models are open-source with weights
available directly from the Hugging Face model hub, with the exception of
LLaMA111Weights for LLaMA were accessed via the approval of Meta AI and used
in accordance with the License (see link for more details.). While we would
have liked to test closed-source LLMs such as OpenAI’s GPT-4 OpenAI (2023),
these models don’t provide vocabulary-wide token probabilities at inference
and are thus incompatible with our test at the time of this writing; see
Assessing Knowledge in Open vs. Closed-Source Models in the Limitations
section for more details.

To carry out the Polyglot or Not assessment for each language, we use an LLM
$M$ to solicit cloze sentence completions for the factual statements in our
dataset. The percentage of fact-completions that $M$ retrieves correctly is
calculated by tallying up the number of factual associations where
$\text{CKA}_{\text{M}}(s,r,o)>1$ and dividing by the total number of factual
associations per language. Our process accommodates the differing tokenization
schemes of the models used by removing special tokens from text generation and
ensuring that the probability scores considered correspond to the first target
token appearing to the right of the unmasked stem. In addition, we nimbly
conduct all this work in a decidedly low-resource setting; see GPU Costs in
the Limitations section for more details on this matter.

##  4 Dataset

We first merged two Wikidata datasets from prior research Meng et al. (2022a);
Dong et al. (2022) that share common roots in the T-Rex Elsahar et al. (2018)
repository and thus are highly suitable for assessing fact recall in English.
After amalgamation, we substantially improved this data post by filtering for
grammatical errors, inaccuracies, and relations that posit questions about
biased or sensitive topics. These improvements are detailed in the Dataset
Processing section of the Appendix. We also enhance the data’s visibility by
making it available in a wide variety of formats across different channels,
such as Hugging Face222https://huggingface.co/datasets/Polyglot-or-Not. After
our combination and filtering steps, the data consisted of 26254 English
language factual associations.

We then translated the dataset from English into 19 target languages suitable
for use with a multilingual fact-completion test. We used the Google Translate
API333https://cloud.google.com/translate (and some custom wrapper functions)
to generate the non-English cuts. These efforts resulted in a wide variety of
non-English counterfactuals to test on, covering 20 languages with Latin and
Cyrilic scripts: bg, ca, cs, da, de, en, es, fr, hr, hu, it, nl, pl, pt, ro,
ru, sl, sr, sv, and uk. Our translation protocol mirrors prior multilingual
benchmarks such as the programmatic translation of MMLU Hendrycks et al.
(2021) prompts when analyzing GPT-4 OpenAI (2023). In addition, work from
Kassner et al. (2021) shows that evaluation differences for Wikidata are
minimal when using machine translated versus manually translated cloze
statements.

Our full dataset includes 303k cumulative testing statements, with each
including the following 7 features: dataset_id (primary key), stem, true,
false, relation, subject, and object. All told, there are 31 relation
categories, 76036 unique subjects, 18837 unique objects, 18503 unique true
completions, and 88224 unique false completions. The masked true/false objects
always appear on the right-hand side of the statement. Thus, different
languages contain varying numbers of statements due to differing capacities of
each translated statement to satisfy this stipulation while adhering to a
target language’s grammar. On average, a given factual association appears in
12 of the 20 languages tested.

model | accuracy (%) | params | n tokens
---|---|---|---
llama-33b |  79.31 (+/- 0.74) | 32.5B | 1.4T
m-bert |  62.00 (+/- 0.87) | 110M | -
bloom-7b1 |  57.70 (+/- 0.88) | 7.1B | 341B
xlm-roberta |  56.03 (+/- 0.90) | 355M | 295B
mt5-xl |  52.51 (+/- 0.91) | 3.7B | -
Random Baseline | 50 | - | -
Table 1: Multilingual test leaderboard. Here, accuracy refers to the average
performance of each model across 20 distinct languages. The uncertainty
estimates represent averaged 95% confidence intervals computed from 10000
bootstrap iterations per language. Params and n tokens record each model’s
number of parameters and number of dataset tokens, respectively (when such
data is available). These results suggest that the tested models struggle to
recall facts in a multilingual setting, as compared to their English-only
performance (Table 3). For instance, on average, LLaMA-33B’s accuracy
decreased by approximately 11% from English to non-English languages.

##  5 Results

Below, we compare model performance across multilingual and English-only fact-
completion experiments. The multilingual results represent mean performance
across each of the 20 languages included in our Polyglot or Not test. Table 1
presents these multilingual outputs for 5 foundational models. Notably, LLaMA
significantly outperformed the other four models. To delve deeper into
multilingual performance, we display LLaMA-33B’s accuracy on each of the 20
languages individually in Table 2 and Figure 1. The data indicates that this
model scores higher on languages written in Latin script than those written in
Cyrillic script (Ukrainian, Bulgarian, Russian and Serbian). A chi-squared
test confirms that LLaMA-33B’s test performance is dependent on language
script ($\chi^{2}=3570.576,p<0.001$). Lastly, Table 3 summarizes the
evaluation results of 20 foundation models in a monolingual setting, using
only English data. Consistent with the multilingual findings, LLaMA models of
different sizes are the most performant.

language | accuracy (%) | pairs
---|---|---
English |  89.40 (+/- 0.38) | 26254
German |  85.74 (+/- 0.53) | 16287
Dutch |  85.35 (+/- 0.46) | 25590
Italian |  84.39 (+/- 0.49) | 20448
French |  84.18 (+/- 0.52) | 18395
Swedish |  84.06 (+/- 0.48) | 21576
Catalan |  84.01 (+/- 0.52) | 18898
Portuguese |  83.81 (+/- 0.48) | 22974
Romanian |  82.72 (+/- 0.57) | 17568
Danish |  81.79 (+/- 0.50) | 23365
Spanish |  81.74 (+/- 0.57) | 18786
Czech |  77.94 (+/- 0.84) | 9427
Polish |  77.50 (+/- 0.84) | 9484
Croatian |  76.69 (+/- 0.97) | 7358
Slovenian |  75.99 (+/- 0.95) | 7873
Hungarian |  75.74 (+/- 1.24) | 4650
Ukrainian |  73.00 (+/- 0.98) | 7918
Bulgarian |  72.50 (+/- 0.61) | 20577
Russian |  69.72 (+/- 1.57) | 3289
Serbian |  60.01 (+/- 1.30) | 5426
Random Baseline | 50 | -
Table 2: LLaMA-33B’s test performance across languages. Here, accuracy denotes the LLaMA-33B model’s performance assessed individually for each language, while pairs refers to the number of stem/fact items evaluated per language. LLaMA-33B demonstrates higher proficiency with languages utilizing the Latin script as compared to those using the Cyrillic script (Ukrainian, Bulgarian, Russian, and Serbian). A chi-squared test substantiates a significant dependency of the model’s test performance on the language script ($\chi^{2}=3570.576,p<0.001$). For a graphical representation of these results, refer to Figure 1 in the Appendix. model | accuracy (%) | params | n tokens
---|---|---|---
llama-33b |  89.40 (+/- 0.38) | 32.5B | 1.4T
llama-13b |  86.66 (+/- 0.42) | 12.5B | 1T
llama-7b |  85.53 (+/- 0.43) | 6.7B | 1T
redpajama-7b |  85.07 (+/- 0.44) | 7B | 800B
mpt-7b |  83.39 (+/- 0.46) | 7B | 1T
opt-30b |  82.09 (+/- 0.47) | 30B | 180B
redpajama-3b |  82.09 (+/- 0.47) | 3B | 800B
opt-13b |  81.94 (+/- 0.46) | 13B | 180B
gpt-neox-20b |  81.50 (+/- 0.47) | 20B | 420B
gpt-j-6b |  81.14 (+/- 0.47) | 6B | 420B
pythia-12b |  80.53 (+/- 0.48) | 12B | 420B
t5-v1-xxl |  76.55 (+/- 0.52) | 13B | 34B
bloom-7b1 |  76.16 (+/- 0.51) | 7.1B | 341B
gpt2-xl |  73.76 (+/- 0.54) | 1.5B | -
bert |  72.60 (+/- 0.54) | 110M | -
m-bert |  71.80 (+/- 0.55) | 110M | -
stablelm-7b |  68.85 (+/- 0.55) | 7B | 1.5T
mt5-xxl |  61.58 (+/- 0.59) | 13B | -
xlm-roberta |  61.55 (+/- 0.59) | 355M | 295B
mt5-xl |  59.96 (+/- 0.59) | 3.7B | -
Random Baseline | 50 | - | -
Table 3: Monolingual test leaderboard. Here, accuracy refers to model
performance on English-only data. The uncertainty estimates are 95% confidence
intervals computed from 10000 bootstrap iterations. Params and n tokens record
each model’s number of parameters and number of dataset tokens, respectively
(when such data is available). Consistent with the trends in Table 1, LLaMAs
of varying sizes emerge as the front-runners.

##  6 Analysis

### Sizing Considerations: Training Data and Model Parameters

LLaMA excels in both the English-only and multilingual experiments, which
challenges some previous paradigms related to model size and training corpus
size Kaplan et al. (2020). Our results suggest that the amount and quality of
data with which a foundation model is trained on is likely more important than
other features, such as sheer parameter count, as Hoffmann et al. (2022) and
Touvron et al. (2023) also find.

For one, the LLaMA model family’s performance on the English-only experiment
suggests that dataset size plays a critical role in LLM proficiency. As per
Touvron et al. (2023), the 33B LLaMA variant undergoes self-supervised
training with a 1.4T token corpus while the 13B and 7B models only see 1T
tokens, and, because each of the LLaMA models are from the same family, their
English-only performances offer an interesting side-by-side comparison into
the impact of sizing up a training data corpus. The greatest performance jump
occurs between 13B and 33B (2.8%), which is more than twice (2.5x) as much as
the difference between 7B and 13B (1.13%). These findings suggest that dataset
size is perhaps the most influential factor differentiating the fact recall
performance of the LLaMA variants. Intriguingly, similar patterns are observed
elsewhere in the Natural Language Understanding literature. For example, the
greatest difference in LLaMA performance on four popular benchmarks emerges
between the 13B and 33B models (4.2% for 7B to 13B, 5.1% from 13B to 33B, 1.4%
from 33B to 65B), as is on display on Hugging Face’s "Open LLM
Leaderboard"444https://huggingface.co/spaces/open_llm_leaderboard.

Similar patterns emerge across the gamut of models included in our
experiments. Firstly, we continue to see that sheer parameter count is not
very predictive of test performance. For instance, the relatively small 110M
parameter m-Bert model Devlin et al. (2019) performs second best in our
multilingual test. Secondly, RedPajama-7b’s placing a close second in the
English-only experiment behind LLaMA indicates that specific characteristics
found in LLaMA-like language modeling datasets are perhaps important in
boosting model performance. As per its
documentation555https://huggingface.co/datasets/RedPajama-Data-1T,
RedPajama-7b was trained on a clean-room, fully open-source implementation of
the LLaMA language modeling dataset. Therefore, our results evince the
efficacy of using a LLaMA-like dataset for language modeling over others, such
as T5’s c4, GPT-NeoX-20B’s The Pile Gao et al. (2020), and m-Bert’s Wikipedia.

### Subject Entity Error Analysis

We analyzed LLaMA-33B’s errors to get a better sense of systemic gaps in its
fact recall. Helpfully, the metadata in our Polyglot or Not dataset provides
natural analytical groupings to guide this exploration. To start, we surveyed
the subject entities in our dataset.

One direction we pursued was determining whether LLaMA-33B systematically
erred when prompted with associations entailing geographic locations as the
subject entities. The 3213 geographic entities we worked with appear in 48606
prompts in our 20 language assessment. The full process for extracting these
is detailed in the Geographic Labeling section of the Appendix. Overall,
LLaMA-33B answered these types of questions correctly at an 89.94% clip. On a
per-continent basis, the highest-performing continent was Asia with 93.31%
accuracy over 10729 questions and the lowest was Antarctica with 80.65%
accuracy over 5167 questions. Additionally, a chi-squared test for
independence comparing LLaMA-33B’s performance on geographic questions related
to Asian locations compared to those in Europe confirms the model’s superior
performance on Asian locations is significant ($\chi^{2}=66.408,p<0.001$).

In a similar vein, we explored whether or not LLaMA-33B’s errors were
systematically related to whether the gender of the fact’s subject was a man
or a woman. The 951 entities we worked with appear in 16003 prompts in our 20
language assessment. The full process for extracting these is detailed in the
Gender Labeling section of the Appendix. LLaMA answered 75.87% of these
questions correctly in total, which is a noticeable 14.07% performance dropoff
compared to geographic entities. Male subjects are nearly 5 times as common as
female subjects in the sample we worked with, yet the model performs slightly
worse on facts about male subjects. A chi-squared test for independence
comparing LLaMA’s performance on questions about male subjects compared to
female subjects confirms the model’s superior performance on questions about
female subjects is significant ($\chi^{2}=69.096,p<0.001$).

### Wikipedia’s Relationship to LLaMA’s Performance

LLaMA learns information in different languages from training on Wikipedia
articles. To better understand the connection between this important data
source and LLaMA-33B’s performance on Polyglot or Not, we investigated the
quality of available data on target language Wikipedia pages. A simple first
step is to gauge the amount of available content the model could have seen at
training time via tabulating the number of articles posted in that language
during LLaMA training period in 2022. Table 4 reflects these counts for all
the languages included in our assessment and demonstrates, for example, how
Catalan Wikipedia is almost 3 times as large as Danish Wikipedia. However, the
sheer volume of articles available in a given language is not necessarily the
clearest indicator of the quality and diversity of information in that
language. For example, the Cebuano language, which was not viewed by LLaMA,
boasts the second-largest Wikipedia catalog by article count, yet almost all
of these articles were added by an automatic bot narrowly focused on
documenting living organisms and biological
species666https://www.wikidata.org/wiki/Wikidata:Statistics/Wikipedia.

To go a level deeper, we randomly sampled articles from our languages of
interest and analyzed their "density" judged on a few factors. The process for
producing these metrics is detailed in the Wikipedia Entity Analysis Process
section of the Appendix. As shown in Table 5, we recorded word count, the
number of named entities that appear in the article (both total and unique),
and the number of named subject entities in our dataset that appear in the
article which we refer to as "target" entities (both total and unique). We
tally these results for the 14 of the 20 languages of interest that possess an
open-source SpaCy777https://spacy.io named entity recognition tagger. Our
methodology spotlights articles that frequently include what we view as
crucial components of information in our dataset: subjects. Indeed, an article
that mentions 8 different target entities is likely to be more robust and
informative than an article that narrowly focuses on 1 target entity. This
also tracks with LLMs tendency to exhibit frequency bias and more accurately
report on subjects they frequently see during training Dong et al. (2022).

Analysis of the sampled articles reveals major coverage gaps when comparing
Wikipedias of various languages. Spanish articles have double the words of
those written in Polish. German articles contain twice as many named entities
as Danish articles. Beyond these and other fascinating summary findings, we
observe a strong and significant correlation (Pearson’s $r=0.78,p<0.001$)
between the average unique target entities on the page and LLaMA’s performance
on Polyglot or Not. Thus, the more subjects on a Wikipedia page, the better
LLaMA recalled facts in that language. This further demonstrates that dataset
quality is strongly connected to performance on our assessment. The imbalance
also complicates future researchers’ use of Wikipedia as a robust multilingual
knowledge base.

### Qualitative Insights

Through a comprehensive analysis of errors, we examined the predominant errors
in different languages and across language families. Our findings reveal that
a significant contributing factor to these errors, particularly in under-
resourced languages, is the influence of frequency bias. For instance, we
observed numerous instances where simple geographical facts concerning
Antarctica were inaccurately represented in many languages. In the English
language prompt "Cape Monaco is a part of the continent of", LLaMA erred in
completing the cloze statement with the incorrect "Europe" rather than the
correct "Antarctica." The Wikipedia entry for Cape Monaco makes numerous
references to European people and places (including, of course, its
appellation)888https://en.wikipedia.org/wiki/Cape_Monaco. In this situation,
then, LLaMA prioritizes the mere presence of a European entity rather than
comprehensively ingesting the knowledge connecting the geographic location to
its correct continent.

We also observed a notable phenomenon in many languages where proper nouns
such as names of countries and languages are, in grammatically correct
writing, not to be capitalized. In these cases, our model displayed a tendency
to rank irrelevant, uncapitalized words higher than the correct capitalized
answers. For example, in the Spanish language prompt "En Kerala entienden" the
model predicted "finlandés" (lowercase) instead of "Inglés" (uppercase). This
behavior can possibly be attributed to the model being trained to recognize
"ingles" (lowercase) as a valid answer, while lacking exposure to "Ingles"
(uppercase) in similar contexts. Despite the contextual irrelevance, the model
erroneously prioritized the lowercase word over the correct uppercase word.
This observation suggests that the frequency and exposure of specific word
forms during training have disproportionately influenced the predictions,
overshadowing the importance of semantic context.

When conducting pairwise comparisons between English and other languages,
exclusively considering common facts, the relative ranking remains largely
consistent with overall performance. Cross-lingual transfer of knowledge
exhibits a lack of reliability, as languages other than English demonstrate
varying degrees of inaccuracies in their interpretation of prompts that are
related to English speaking countries, with Slavic languages exhibiting more
significant deviations than others.

##  7 Future Work

To expand this work, we envision a few next steps. As our data finds its roots
in two projects Dong et al. (2022); Meng et al. (2022a) centered around
calibrating LLMs, carrying out model editing in a multilingual setting
presents a practical next step for our project. To pursue this, we’ve
constructed a proof-of-concept999https://github.com/daniel-furman/Polyglot-or-
Not/blob/main/notebooks/fact_calibration_notebooks in line with Meng et al.
(2022a)’s model editing algorithm that repairs erroneous factual association
in LLMs. We hope to expand this into a robust toolkit for correcting all
detected errors in an inputted open-source model, across all 20 of the
languages we tested in. This effort would enhance the practical use of LLMs,
because organizations could identify and then remedy important factual
associations they need models to sufficiently understand. This would also
support the ongoing fight against LLMs in misinformation campaigns Goldstein
et al. (2023).

Additionally, we plan to test new LLMs as they are released and more massive
LLMs than we were able to test here. Firstly, applying our test continuously
to the newest open-source models is one way that we can maintain this work’s
downstream impact. Secondly, we would like to expand our test to larger
models, such as LLaMA-65B. Testing larger LLMs would provide more evidence for
our training data vs. model size scaling arguments, as detailed in the Sizing
Considerations: Training Data and Model Parameters section of the Analysis.
For example, with the LLaMA-65B results in hand, we would have more
information to base our claim that LLaMA’s language modeling dataset size is
more important than its parameter count for fact recall performance. Beyond
incorporating more models, a logical next step is also expanding the variety
of languages we use in our test, in particular those that use scripts besides
Latin and Cyrillic. To support this work, we are consulting with native Hindi
and Japanese speakers to gauge how we might modify our cloze statements to be
compatible in those languages.

We would also like to expand our error analysis by pulling in additional
fields of metadata to gain more insights about the training data used by
different LLMs. There are challenging questions to be grappled with in this
domain, such as whether it’s possible to causally trace a model’s erring on a
particular fact to artifacts in its training data (rather than the associate
work we report in this piece). Helpfully, contemporary work Elazar et al.
(2023) on marrying causal inference with LLMs provides a fertile basis for
this exploration.

Lastly, capturing human performance on each of the languages tested, in the
vein of prior human evaluations for NLP tasks like Nangia and Bowman (2019)’s
research on the GLUE benchmark, would enhance Polyglot or Not.

##  8 Conclusion

Polyglot or Not is the first multilingual and counterfactual knowledge
assessment for open-source Large Language Models. 303k factual associations in
20 different languages comprise our novel dataset, which is designed to assess
the capability of foundation models in retrieving encyclopedic facts across
diverse languages, topics, and contexts. We conducted evaluations using 5
foundation models in a multilingual setting, along with a varied set of 20
models in an English-only context.

Meta’s LLaMA model demonstrated superior performance on both our multilingual
and English-only assessments. These results confirm the importance of massive
and high-quality language modeling datasets and also ratify the viability of
the use of counterfactuals in LLM benchmarking. However, following thorough
error analysis, LLaMA’s limitations in retrieving facts in languages other
than English — particularly those using the Cyrillic script — poses
significant challenges. Further error analysis revealed additional
shortcomings, including limitations associated with using Wikipedia as a
multilingual knowledge base and limited cross-lingual knowledge transfer.
These findings emphasize the need for future research to improve training
methodologies and data collection for foundation language models. By doing so,
multilingual models can be better equipped for broader application in diverse
linguistic contexts.

## Limitations

### Assessing Knowledge in Open vs. Closed-Source Models

One prerequisite for carrying out Polyglot or Not is access to the full
schedule of vocabulary-level token score probabilities generated when an LLM
synthesizes text. For this reason, researchers in related inquiries typically
work with fully open-source LLMs available from NLP hubs such as Hugging Face
Jiang et al. (2020); Meng et al. (2022a); Dong et al. (2022). Closed-source
LLMs, meanwhile, lack this transparency rendering their generated texts
resistant to analysis. Notably, OpenAI’s GPT-3
API101010https://platform.openai.com/docs/api-reference/completions only
surfaces the probabilities of the 5 most likely next tokens, a functionality
which Hendrycks et al. (2021) leveraged to apply GPT-3 to their evaluation
task. We submitted a request for this limit to be raised through the official
channel OpenAI provides — a fully automated, chat-bot customer service agent —
and we have yet to receive a response. What’s more, the GPT-4 API
nixed111111https://github.com/MicrosoftDocs/azure-
docs/blob/main/articles/cognitive-services/openai/includes/chat-markup-
language.md this functionality entirely, thwarting an important avenue for
research into their newest foundation model and adding an additional layer of
opacity into how their systems produce results OpenAI (2023). Unfortunately,
as it stands today, the largest foundation models from other research
consortiums such as DeepMind’s Gopher Rae et al. (2022), Google’s LaMDA
Thoppilan et al. (2022), and Huawei’s PanGu-Sigma Ren et al. (2023) are all
proprietary.

### GPU Costs

We performed experiments on a variety of model families and sizes. This
required many hundreds of hours of GPU usage. In total, we batched over 100
model runs that required approximately 200 hours of GPU usage. For instance,
running LLaMA-7B on the 22974 Portuguese factual associations in our dataset
required 2.5 hours of GPU usage. In addition to having to schedule long-
periods of compute uptime, we were also constrained by fixed resource
requirements, using workstations with a single NVIDIA GPU. Thus, we could not
evaluate the gamut of truly massive models in our experiments. Going forward,
more accommodations need to be made for groups to effectively experiment with
LLMs, in particular as industry players release models that require extremely
demanding compute requirements to comprehensively evaluate. We hope that
techniques such as Dao et al. (2022) into more efficient language modeling
techniques for both training and inference become widely implemented,
researched, and democratized.

## Ethics Statement

Although we test a language model’s ability to serve as multilingual knowledge
bases, we do not find these models to be particularly reliable sources of
knowledge as none of the models scored above 90% for any of the languages that
we tested. We thus caution readers that a LLM should not be used as an
authoritative source of facts. Our Polyglot or Not test sheds light on the
types of languages, topics, and contexts where LLMs are more likely to produce
factual errors, but the same methods might also enable a malicious actor to
check whether a particular set of facts is committed to model memory and
subsequently insert damaging information into a model that was not originally
present in the training data with other methods, such as the MEMIT algorithm
proposed by Meng et al. (2022b). Lastly, while our work points to the need for
testing low-resource languages, our evaluation test is restricted to a
relatively small number of languages (20), most of which are high-resource. We
intentionally use the 20 languages included in the LLaMA language modeling
dataset in this work. However, future work must delve deeper into fact-
completion testing for low-resource languages and devote attention to a larger
number of languages.

## Acknowledgments

The research reported in this article was supported by funding from the School
of Information at the University of California, Berkeley. We also extend
special thanks to our faculty advisor, David Bamman.

## References

  * Argyle et al. (2022) Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua Gubler, Christopher Rytting, and David Wingate. 2022.  Out of one, many: Using language models to simulate human samples.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , page 819–862.
  * Bender et al. (2021) Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big? In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, page 610–623, New York, NY, USA. Association for Computing Machinery.
  * Black et al. (2022) Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022.  Gpt-neox-20b: An open-source autoregressive language model.  In _Proceedings of BigScience Episode 5 – Workshop on Challenges and Perspectives in Creating Large Language Models_ , page 95–136, virtual+Dublin. Association for Computational Linguistics.
  * Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  In _Advances in Neural Information Processing Systems_ , volume 29. Curran Associates, Inc.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, page 1877–1901. Curran Associates, Inc.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , page 8440–8451, Online. Association for Computational Linguistics.
  * Dao et al. (2022) Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022.  Flashattention: Fast and memory-efficient exact attention with io-awareness.  (arXiv:2205.14135).
  * Dettmers et al. (2022) Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022.  Llm.int8(): 8-bit matrix multiplication for transformers at scale.  (arXiv:2208.07339).
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , page 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dodge et al. (2021) Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021.  Documenting large webtext corpora: A case study on the colossal clean crawled corpus.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , page 1286–1305, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Dong et al. (2022) Qingxiu Dong, Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui, and Lei Li. 2022\.  Calibrating factual knowledge in pretrained language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , page 5937–5947, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Elazar et al. (2023) Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schütze, and Yoav Goldberg. 2023.  Measuring causal effects of data statistics on language model’s ‘factual’ predictions.  (arXiv:2207.14251).
  * Elsahar et al. (2018) Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018.  T-rex: A large scale alignment of natural language with knowledge base triples.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020.  The pile: An 800gb dataset of diverse text for language modeling.  (arXiv:2101.00027).
  * Geva et al. (2022) Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. 2022.  Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space.  (arXiv:2203.14680).
  * Goldstein et al. (2023) Josh A. Goldstein, Girish Sastry, Micah Musser, Renee DiResta, Matthew Gentzel, and Katerina Sedova. 2023.  Generative language models and automated influence operations: Emerging threats and potential mitigations.  (arXiv:2301.04246).
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring massive multitask language understanding.  (arXiv:2009.03300).
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022.  Training compute-optimal large language models.  (arXiv:2203.15556).
  * Jiang et al. (2020) Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020.  X-factr: Multilingual factual knowledge retrieval from pretrained language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , page 5943–5959, Online. Association for Computational Linguistics.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.  (arXiv:2001.08361).
  * Kassner et al. (2021) Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021.  Multilingual lama: Investigating knowledge in multilingual pretrained language models.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , page 3250–3258, Online. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , page 7871–7880, Online. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Meng et al. (2022a) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022a.  Locating and editing factual associations in gpt.  (arXiv:2202.05262).
  * Meng et al. (2022b) Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2022b.  Mass-editing memory in a transformer.  (arXiv:2210.07229).
  * Nangia and Bowman (2019) Nikita Nangia and Samuel R. Bowman. 2019.  Human vs. muppet: A conservative estimate of human performance on the GLUE benchmark.  _CoRR_ , abs/1905.10425.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  (arXiv:2303.08774).
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , page 2463–2473, Hong Kong, China. Association for Computational Linguistics.
  * Radford et al. (2018) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018.  Language models are unsupervised multitask learners.
  * Rae et al. (2022) Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. 2022.  Scaling language models: Methods, analysis & insights from training gopher.  (arXiv:2112.11446).
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  (arXiv:1910.10683).
  * Ren et al. (2023) Xiaozhe Ren, Pingyi Zhou, Xinfan Meng, Xinjing Huang, Yadao Wang, Weichao Wang, Pengfei Li, Xiaoda Zhang, Alexander Podolskiy, Grigory Arshinov, Andrey Bout, Irina Piontkovskaya, Jiansheng Wei, Xin Jiang, Teng Su, Qun Liu, and Jun Yao. 2023\.  Pangu-$\Sigma$: Towards trillion parameter language model with sparse heterogeneous computing.  (arXiv:2303.10845).
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , page 5418–5426, Online. Association for Computational Linguistics.
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022.  Lamda: Language models for dialog applications.  (arXiv:2201.08239).
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.  (arXiv:2302.13971).
  * Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020.  Ccnet: Extracting high quality monolingual datasets from web crawl data.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , page 4003–4012, Marseille, France. European Language Resources Association.

## Appendix

### Author contributions

  * •

Tim Schott: writing and coding

  * •

Daniel Furman: writing and coding

  * •

Shreshta Bhat: writing and coding

Equal contributions were contributed by each of the authors.

### Reproducibility

Supporting code and data are openly released on github.com and huggingface.co,
respectively.

Models were loaded and run using
Transformers121212https://github.com/huggingface/transformers and
BitsandBytes131313https://github.com/TimDettmers/bitsandbytes. See the
Experiments on Model Precision section of the Appendix for more details.
Subsequent steps of the Polyglot or Not test were executed with
PyTorch141414https://github.com/pytorch/pytorch.

In regards to compute resources, the experiments were run on workstations with
a single NVIDIA GPU. Specifically, an A100 40GB was used for larger models
(those possessing >15B parameters, approximately) while a Tesla T4 15GB was
used for smaller ones.

### Dataset Processing

The Calinet Dong et al. (2022) dataset originally contained 50451 stem/fact
items which we consider "valid" as the masked object appears on the right-hand
side of the statement. Many of these stem/fact pairings were paraphrased,
though, to support their model rewrite process, which this paper does not
explore. After removing paraphrases, we were left with 11960 statements from
this data pool. Meanwhile, the ROME Meng et al. (2022a) dataset contributed
21919 valid stem/fact pairs, all of which were unique. We merged the data and
were left with 33870 stem/fact pairs. From there, we performed the following
enhancements:

  * •

Removed 227 stem/fact pairs that were manually flagged as errors

  * •

Removed 371 stem/fact pairs with "a/an + _" due to consistent grammatical
errors

  * •

Removed 3088 stem/fact pairs where the correct fact is explicitly stated in
the stem, rendering the completion trivial

  * •

Removed 610 stem/fact pairs that were relation P190 (sister city) due to
consistent inaccuracies

  * •

Removed 418 stem/fact pairs that were relation P140 (religion) to filter
sensitive topics

  * •

Removed 490 stem/fact pairs that were relation P530 (diplomatic ties) due to
consistent inaccuracies

  * •

Removed 1427 stem/fact pairs that were relation P27 (citizen of) due to
consistent inaccuracies

  * •

Removed 576 stem/fact pairs that were relation P463 (affiliated with) due to
consistent inaccuracies

  * •

Removed 39 stem/fact pairs that compared football with soccer due to cultural
differences in these word meanings

  * •

Removed 131 stem/fact pairs with "expired at" wording due to awkward phrasing

  * •

Removed 50 stem/fact pairs with "-language" wording due to awkward phrasing

  * •

Removed 73 stem/fact pairs with facts/counterfacts starting with "the" due to
the frequency of the work "the" in language modeling datasets

  * •

Removed 125 stem/fact pair duplicates to retain a dataset of entirely unique
items

Our straightforward improvements provide more validity to our pool of data and
its ultimate use in Polyglot of Not, such as removing the over 3000 statements
whose correct answer can be found in the unmasked portion. After this and
other filtering, we are left with 26254 unique rows in the final English
subset of our dataset.

### Geographic Labeling

We sought a labeled dataset of geographic entities connected to the continents
they’re located on. To do so, we filtered our original dataset down to the
Wikidata relation IDs that most clearly signal that a geographic entity, such
as Paris or France, occupies the subject of the stem: capital (relation P17 \+
P1376), continent (P30), country (P36), shares border with (P47), and is in
the territory of (P131). Then, we extracted the unique, English translations
of the subjects from this data, leaving us with 3427 "geographic" entities in
our dataset. To more quickly move into substantive analysis, we utilized a
Generative AI assistant, ChatGPT (GPT-3.5)151515https://chat.openai.com/, to
label these entities by geographic continent. Our prompt (see below) offered
an option for an "unsure" label if the assistant did not know the correct
answer, the location stretched across multiple continents, etc. Of the 3427 we
requested prompts for, the assistant labeled 3213 with a tag for one of the
world’s continents. To verify the veracity of the labels we randomly sampled
10% of the labeled data and found that the affixed continent labels were
correctly applied to every entity in the validation sample. The resulting
labeled data provided interesting terrain for mining insights, as detailed in
the Subject Entity Error Analysis subsection. Prompt used:

I have a list of locations. Can you return the continent on which they are
located in the following format:

Iran|AS
Bavaria|EU
Pennsylvania|NA

If there are items in the list that don’t seem like locations or perhaps are
very difficult to classify you can write "unsure" beside those, e.g.

WTJU-FM|unsure
Ottoman Empire|unsure

### Gender Labeling

We also desired a labeled dataset of person entities connected to their
assigned birth gender, as understood in the popular consciousness. To do so,
we filtered our original dataset down to the Wikidata relation IDs that most
clearly signal that a person entity, such as Steve Jobs or Marie Curie,
occupies the subject of the stem: place of death (relation P20), position held
(P39), field of work (P101), native language (P103), occupation (P106,
employer (P108), position played on team (P413), sport (P641), work location
(P937), and instrument (P1303). We followed a near-identical procedure for
Gender Labeling as we did for Geographic Labeling, using ChatGPT to label
these identities by gender. However, because there are far more people
entities in our dataset after filtering for these relation IDs (7905 in total)
we randomly sampled a portion of them, extracting 1200 unique entities to hand
off to ChatGPT. Our prompt (see below) for gender also offered an option for
an "unsure" label if the assistant did not know the correct answer, the entity
wasn’t a name, etc. Of the 1200 we requested prompts for, the assistant
labeled 1057 with a gender tag. To verify the veracity of the labels we
randomly sampled 10% of the labeled data and found that the affixed gender
labels were correctly applied to every entity in the validation sample. The
resulting labeled data is also explored in the Subject Entity Error Analysis
subsection. Prompt used:

I have a list of names. Can you return the gender (male, female, or other) in
the following format:

Sundar Pichai|Male
Brigitte Fontaine|Female

If there are items in the list that don’t seem like names or perhaps are very
difficult to classify, you can write "unsure" beside those, e.g.

WTJU-FM|unsure
Wagnerian|unsure

### Wikipedia Entity Analysis Process

To produce Table 5, we began by randomly sampling 10k pages from every
language of interest’s Wikipedia via its REST
API161616https://www.mediawiki.org/wiki/API:Main_page. We did this because we
wanted to gather a sample of the text in the article body for each language.
From there, we extracted the body content of these articles and performed
minimal preprocessing such as removing citations and navigation headers. With
the clean page content in hand, we then used a named entity recognition from
SpaCy171717https://spacy.io/api/entityrecognizer. SpaCy provides models for 14
of the 20 languages LLaMA was tested on. For each of these languages, the
core_news_lg tagger was used save for English where we used the core_web_lg
tagger. We then tallied counts for the entities found in each page. We tracked
the overall and distinct number of entities found in each article. We also
stored the overall and distinct subset of entities that are found in each
article and who appear in our Polyglot or Not dataset. All source code for
above can be found in our repository181818https://github.com/daniel-
furman/Polyglot-or-
Not/blob/main/src/wikipedia_entity_analysis/wiki_analysis.py.

### Experiments on Model Precision

We explore the impact of matrix multiplication precision on both model
performance and memory usage here. Specifically, we ran LLaMA-7B and LLaMA-13B
on the English-only portion of our dataset under both 16-bit and 8-bit
configurations. In the case of 16-bit precision, all values were simply
assigned the float16 dtype. For 8-bit precision, we adopted the mixed-
precision algorithm presented by Dettmers et al. (2022), which uses the int8
dtype for the majority of the values and the float16 dtype for all remaining
outliers. As shown in Table 6, there were noticeable differences in model
performance between the two precision levels, with slight but measurable drop
off between the 16-bit and 8-bit variants (0.35-0.47%). The savings in memory
consumption, however, were much more significant by comparison. By opting for
8-bit over 16-bit precision, we reduce the memory footprint of the two models
by 1.8x. Based on these findings, we determined that the trade-offs between
performance and memory footprint were acceptable for our Polyglot or Not test.
We thus elected to employ 8-bit precision throughout the experiments above,
and, as a result, each of the models fit on workstations equipped with a
single NVIDIA GPU.

language | article count
---|---
English | 6,513,291
German | 2,698,267
Swedish | 2,551,218
French | 2,430,636
Dutch | 2,092,862
Russian | 1,828,011
Spanish | 1,782,912
Italian | 1,758,843
Polish | 1,525,414
Ukrainian | 1,160,183
Portuguese | 1,093,217
Catalan | 702,281
Serbian | 659,580
Hungarian | 505,754
Czech | 505,105
Romanian | 431,067
Bulgarian | 282,130
Danish | 280,923
Croatian | 212,088
Slovenian | 176,565
Table 4: Wikipedia page counts. The number of articles available on Wikipedia during LLaMA’s training time period of June 2022, as reflected by the article count for each language surfaced on archive.org (arranged descending by article count). Even a high-resource language like Romanian possesses a rather small Wikipedia in comparison to other languages like French. (The corresponding archive.org URLs, which link to the initial archived copy of the language’s homepage on or as close as possible to June 15th, 2022 can be found in our codebase.)  language | words | entities | unique entities | targets | unique targets |  accuracy (%)
---|---|---|---|---|---|---
Catalan | 384.43 | 26.51 | 19.54 | 3.82 | 2.31 | 84.01
Croatian | 273.29 | 24.98 | 19.68 | 0.87 | 0.60 | 76.69
Danish | 255.77 | 22.71 | 16.90 | 3.30 | 2.07 | 81.79
Dutch | 156.54 | 25.25 | 19.78 | 1.85 | 1.28 | 85.35
English | 463.15 | 70.12 | 50.19 | 6.86 | 3.68 | 89.40
French | 491.93 | 36.82 | 26.27 | 4.49 | 2.58 | 84.18
German | 418.59 | 40.95 | 30.46 | 3.90 | 2.40 | 85.74
Italian | 391.67 | 31.09 | 21.52 | 3.96 | 2.24 | 84.39
Polish | 214.29 | 32.70 | 25.74 | 0.90 | 0.60 | 77.50
Portuguese | 320.44 | 24.95 | 17.86 | 3.60 | 2.18 | 83.81
Romanian | 231.10 | 42.72 | 33.68 | 2.39 | 1.62 | 82.72
Russian | 382.84 | 35.70 | 26.35 | 0.86 | 0.58 | 69.72
Spanish | 470.56 | 33.04 | 23.80 | 4.25 | 2.41 | 81.74
Swedish | 95.03 | 9.37 | 7.42 | 1.30 | 0.95 | 84.06
Ukrainian | 283.64 | 25.32 | 19.90 | 1.21 | 0.80 | 73.00
Table 5: Wikipedia content analysis. Results of performing named-entity recognition on a random sample of 10k Wikipedia articles across 15 languages (arranged alphabetically by language name). Reported metrics correspond to per-page averages: words is the article word count as reported by SpaCy’s language specific tokenizer. Entities and unique entities represent the total and distinct entity counts, respectively, from SpaCy’s named-entity recognition tagger on the page text while the targets and unique targets columns correspond to the counts of entities that occupy the subject position of stems in our dataset. LLaMA’s test accuracy for each language occupies the right-most column, as is also displayed in Table 1 and Figure 1. We find that LLaMA’s performance is significantly correlated with the number of unique target entities found in our sampled pages (Pearson’s $r=0.78,p<0.001$). Other takeaways include the rather low average word count of articles on Swedish language Wikipedia due to its high proportion of machine generated pages. model | precision | accuracy (%) | memory footprint (GB)
---|---|---|---
llama-13b | 16-bit |  87.01 (+/- 0.40) | 26.2
llama-13b | 8-bit |  86.66 (+/- 0.42) | 14.5
llama-7b | 16-bit |  86.00 (+/- 0.42) | 14.7
llama-7b | 8-bit |  85.53 (+/- 0.43) | 8.3
Table 6: Precision experiments for LLaMA-7B and LLaMA-13B. Here, accuracy
denotes the model’s performance on English-only data. A small dip in accuracy
(0.35-0.47%) is observed between 16-bit and 8-bit precisions. Figure 1:
LLaMA-33B’s test performance across languages. The model scores higher on
languages written in Latin script than those written in Cyrillic script
(Ukrainian, Bulgarian, Russian and Serbian). A chi-squared test confirms that
the model’s test performance is dependent on language script
($\chi^{2}=3570.576,p<0.001$).