#  Turning English-centric LLMs Into Polyglots:
How Much Multilinguality Is Needed?

Tannon Kew1 Florian Schottmann2,3 Rico Sennrich1,4
1University of Zurich, 2Textshuttle, 3ETH Zurich, 4University of Edinburgh
<EMAIL_ADDRESS><EMAIL_ADDRESS>Work partially conducted
during an internship at Textshuttle.

###### Abstract

The vast majority of today’s large language models are English-centric, having
been pretrained predominantly on English text. Yet, in order to meet user
expectations, models need to be able to respond appropriately in multiple
languages once deployed in downstream applications. Given limited exposure to
other languages during pretraining, cross-lingual transfer is important for
achieving decent performance in non-English settings. In this work, we
investigate just how much multilinguality is required during finetuning to
elicit strong cross-lingual generalisation across a range of tasks and target
languages. We find that, compared to English-only finetuning, multilingual
instruction tuning with as few as three languages significantly improves a
model’s cross-lingual transfer abilities on generative tasks that assume
input/output language agreement, while being of less importance for highly
structured tasks. Our code and data is available at
https://github.com/ZurichNLP/multilingual-instruction-tuning.

Turning English-centric LLMs Into Polyglots:
How Much Multilinguality Is Needed?

Tannon Kew1††thanks:  Work partially conducted during an internship at
Textshuttle. Florian Schottmann2,3 Rico Sennrich1,4 1University of Zurich,
2Textshuttle, 3ETH Zurich, 4University of Edinburgh<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Conversational instruction tuning is a popular method for aligning large
pretrained language models (LLMs) with user expectations such that they can
effectively respond to a user’s input query and follow natural language
instructions Ouyang et al. (2022); Wang et al. (2023); Chiang et al. (2023).
An implicit expectation of conversational chatbots is that the language of a
model’s response should match that of the user’s input query. For instance,
unless otherwise specified, a German-language input should result in a German-
language output. However, since the vast majority of pretraining and tuning
data used to develop these models is in English, many instruction-tuned LLMs
struggle to respond consistently in other languages Touvron et al. (2023b);
Chen et al. (2023); Ye et al. (2023); Zhang et al. (2023b).

Figure 1:  Input/output (IO) language agreement for English (en), German (de),
Bulgarian (bg) and Icelandic (is) when instruction tuning on monolingual
English (Mono) or on multilingual data (Multi-Guanaco). Striped bars indicate
that the target language is included in the finetuning data. Error bars show a
confidence interval of 95%.

Despite this limited exposure, English-centric LLMs such as Llama 2 can
achieve near-perfect input/output (IO) language agreement when tuned with
relatively few multilingual conversational instructions. Figure 1 depicts IO
language agreement, as measured by
OpenLID111https://github.com/laurieburchell/open-lid-dataset Burchell et al.
(2023) and compares multilingual tuning on the language-diverse Guanaco
dataset Dettmers et al. (2023); Köpf et al. (2023) with monolingual tuning on
an English-only subset of instructions. As can be seen, multilingual tuning
elicits strong IO language agreement, well within the bounds of language
identification error rates, for non-English languages seen to differing
degrees during pretraining and finetuning without degrading performance on
English.

This observation raises two major questions, which we aim to address in this
paper: Q1: How much multilinguality is required during finetuning to elicit
few- and even zero-shot cross-lingual generalisation in English-centric LLMs?
Q2: Which languages and tasks benefit most from multilingual instruction
tuning of English-centric LLMs?

To investigate these questions, we instruction-tune English-centric LLMs,
varying the number of languages seen during finetuning and evaluate
performance in multiple target languages and on different tasks. We consider
both high- and low-resource language settings with different scripts and
investigate performance on generative tasks, such as open-ended chat and
extractive question answering, as well as structured tasks aimed at assessing
commonsense reasoning and language understanding.

Our results indicate that multilingual instruction tuning is crucial for
eliciting cross-lingual transfer on generative tasks that assume IO language
agreement while being less important in structured tasks that are commonly
used to benchmark LLM performance. Furthermore, we empirically show that only
a small number of finetuning languages is required to promote cross-lingual
transfer. This highlights that tuning data for all potential target languages
is not necessary to derive a capable polyglot chat model from an English-
centric LLM.

##  2 Related Work

###  2.1 Instruction Tuning LLMs

Instruction tuning describes a supervised finetuning (SFT) strategy that aims
to provide a model with a diverse set of demonstrations of user input queries
paired with desirable model outputs. Unlike task-specific SFT, instruction
tuning aims to promote cross-task generalisation, allowing for a ‘generalist’
model that is capable of completing any text-based task on the basis of
natural language instructions provided at inference time Mishra et al. (2022);
Wei et al. (2022); Wang et al. (2022); Sanh et al. (2022); Longpre et al.
(2023). Meanwhile, framing instructions in a conversational manner and over
multiple dialogue turns has been shown to be effective at deriving performant
chat models Taori et al. (2023); Conover et al. (2023); Chiang et al. (2023);
Dettmers et al. (2023); Ding et al. (2023). Furthermore, LLM instruction
tuning remains effective given relatively limited labelled data (Ouyang et
al., 2022; Touvron et al., 2023b; Zhou et al., 2023), parameter efficient
methods Hu et al. (2021); Zhang et al. (2023a) and model quantisation Dettmers
et al. (2023); Li et al. (2023).

###  2.2 Cross-lingual Transfer in English-centric LLMs

Despite numerous pushes towards improving NLP support for non-English and low-
resource languages Costa-jussà et al. (2022); Le Scao et al. (2023), the vast
majority of today’s publicly available LLMs are English-centric. For instance,
GPT-3’s training data consisted of approximately 93% English documents with
the remaining 7% pertaining to other languages Brown et al.
(2020).222https://github.com/openai/gpt-3/blob/master/dataset_statistics This
trend is further reflected in popular open-weight LLMs (see Table 1). One
potential reason for this could be the “the curse of multilinguality” Conneau
et al. (2020) which describes how as multilinguality increases, evaluation
performance decreases. For example, multilingual LLMs have been shown to
sacrifice performance on English tasks due to having to share a finite amount
of pre-training tokens and model parameters across more languages Lin et al.
(2022); Le Scao et al. (2022). Consequently, the cross-lingual transfer
abilities of more performant, English-centric models is important for many NLP
practitioners looking to deploy these models in real-world settings.

A number of works have reported on the multilingual abilities of proprietary
models such as GPT-3 and its derivatives across a range of NLU and NLG
benchmarks Lai et al. (2023a); Holmström et al. (2023); Armengol-Estapé et al.
(2022). On tasks such as machine translation (MT), GPT-3.5-Turbo has been
shown to outperform dedicated systems for some high-resource languages Hendy
et al. (2023); Lu et al. (2023); Jiao et al. (2023); Bang et al. (2023),
particularly when translating between two non-English languages Laskar et al.
(2023). However, performance on low-resource languages generally lags behind.

Meanwhile, other studies have focused on open-weight LLMs. Ye et al. (2023)
compare multilingual reasoning capabilities of pretrained BLOOM Le Scao et al.
(2023) and LLaMA Touvron et al. (2023a) and find that despite minimal amounts
of non-English pretraining data, LLaMA has strong cross-lingual transfer
abilities. Muennighoff et al. (2023) find that multilingual multitask
finetuning improves BLOOM’s zero-shot abilities across all target languages
compared with English-only finetuning. Finally, Chen et al. (2023) show that
large-scale multilingual instruction tuning can improve performance on open-
ended chat in multiple target languages.

This last work is most closely related to our current work but differs in a
few key factors. Firstly, we investigate multilingual chat capabilities of
more recent English-centric models, pretrained with significantly more data.
Secondly, inspired by Zhou et al. (2023) we adopt a less-is-more finetuning
approach, controlling the amount and distribution of multilingual data. In
doing so, we focus our analysis on the minimal amount of multilinguality
needed to elicit cross-lingual transfer. Finally, our evaluations consider
more target languages and distinct downstream tasks.

##  3 Experimental Setup

To explore the multilingual capabilities of English-centric LLMs, we
instruction-tune a series of models on a fixed-size set of examples, varying
the number of languages available. Following this, we evaluate the resulting
models in multiple target languages on four distinct tasks that are
representative of how LLMs may be used in downstream applications.

###  3.1 English-centric LLMs

A prevailing trend in the development of recent LLMs is a clear focus on
scaling up the size of the pretraining corpus Hoffmann et al. (2022). For
instance, open-weight LLMs such as Falcon Almazrouei et al. (2023) and Llama 2
Touvron et al. (2023b) were trained on 1.5 and 2 trillion tokens respectively.
Yet, although these numbers far surpass the 300 billion tokens used to train
GPT-3 Brown et al. (2020), the distribution of language coverage remains
similar across models with more than 90% pertaining to English (see Table 1).
For our main experiments, we focus on Llama 2 7b, but also consider Llama 2
70b (§5.2) and Falcon 7b (Appendix E) to study the effect of model scaling and
different training approaches respectively.

###  3.2 Instruction-tuning Data

For instruction tuning, we take inspiration from Dettmers et al. (2023) and
finetune on high-quality conversations from the OpenAssistant dataset Köpf et
al. (2023). These conversations comprise multiple dialogue turns between
crowdworkers who were asked to either interact with or assume the role of a
helpful AI assistant.

In contrast to Dettmers et al. (2023), who use all 9,846 top-rated
conversations to train their ‘Guanaco’ models, we subsample training instances
from the Guanaco dataset in order to control the amount of multilinguality.
Specifically, we sample 3,200 unique English instances as an initial
monolingual dataset, which we refer to as ‘Mono’. To construct datasets for
multilingual finetuning we sample 200 unique training examples from each of
five most frequent non-English languages in Guanaco (Spanish, Russian, German,
Chinese, French). Given these subsets, we incrementally substitute English
examples in Mono for non-English ones, one language at a time, following the
order of how frequently each language appears in Guanaco. The resulting
multilingual datasets are denoted as Multi-$i$, where $i$ equals the number of
distinct languages included. For comparison, we also train models on the full
Guanaco dataset, which includes more than 30 distinct languages.

###  3.3 Instruction-tuning and Inference Settings

For efficient instruction tuning, we use LoRA adapters ($R=64$, $\alpha=16$),
leveraging Hugging Face’s TRL library333https://github.com/huggingface/trl. We
train all models for 2,000 update steps, using an effective batch size of 64
and a learning rate of $1^{e-5}$. Sequences longer than 1024 tokens are
truncated. For 7-billion parameter models, we use two NVIDIA GeForce RTX 3090
with 24GB of memory. The time required for each training run is approximately
8 hours. For the larger 70-billion parameter model, discussed in §5.2, we use
the same hyperparameters and train on four NVIDIA A100 GPUs with 80GB of
memory each. Here, a single training run takes approximately 20 hours.

At inference time, we load the instruction-tuned models with vLLM Kwon et al.
(2023). For open-ended generation tasks, we use top-p sampling (p=0.9) with a
temperature of 0.8 and frequency penalty of 0.2. For the more constrained
generation tasks we reduce the temperature to 0.001. To account for randomness
under these sampling parameters, we report aggregated results for all tasks by
repeating inference experiments with three random seeds.

###  3.4 Evaluation Tasks and Languages

As evaluation tasks, we consider open-ended chat, extractive question
answering, commonsense reasoning and natural language inference. As all of
these tasks differ in terms of the availability and representation of ground-
truth labels, we describe the specific evaluation strategies in the following
section.

Finally, we investigate performance on different target languages, which we
select based on the makeup of Llama 2’s pretraining data (see Table 1). In
doing so, we aim to investigate  i) the supervised setting, where a target
language has been seen during pretraining and finetuning;  ii) the zero-shot
cross-lingual setting, where a target language may have been seen during
pretraining but not during finetuning Wu and Dredze (2019); and  iii) the
extreme low-resource setting, where the target language has not intentionally
been seen at any stage of the training procedure.  According to Touvron et al.
(2023b), German (de), French (fr), Swedish (sv), Chinese (zh), Spanish (es)
and Russian (ru) represent the most frequently seen non-English pretraining
languages with an estimated 3.4 to 2.6 billion tokens. Catalan (ca), Norwegian
(no) and Bulgarian (bg) represent languages less frequently seen during
pretraining with an estimated 800 to 400 million tokens. And finally, we
select Icelandic (is), Hindi (hi) and Greek (el) to represent extremely low-
resource languages, whose frequency in Llama 2’s pretraining data is not
available but may still appear in small amounts due to intentional or
unintentional contamination Blevins and Zettlemoyer (2022). Unfortunately,
existing benchmark evaluation datasets do not cover all of these languages and
thus we limit the target languages in those tasks to the available subset of
our target languages.

##  4 Experiments

###  4.1 Chat

General-purpose chatbots are a popular application of instruction-tuned LLMs.
To evaluate chat performance, we make use of the AlpacaEval prompt dataset
Dubois et al. (2023), which includes a diverse set of prompts for open-ended
questions, creative writing, brainstorming, and other tasks. We randomly
sample 300 prompts and translate these into each target language. As a
translation engine, we follow Lai et al. (2023b) and use GPT-3.5-Turbo444To
translate chat prompts from AlpacaEval, we use gpt-3.5-turbo-0613.. In
contrast to using dedicated translation systems, employing GPT-3.5-Turbo for
this purpose has the advantage of being able to explicitly specify
instructions that allow for preserving code blocks, tables and terminology,
which we include as part of our translation prompt (see Figure B).
Furthermore, since GPT-3.5-Turbo is trained on instruction- and
conversational-style data, we expect it to perform well at translating in this
domain.

To automatically evaluate open-ended chat responses, we leverage LLM-as-a-
judge (Zheng et al., 2023). Following Zhou et al. (2023), given an input
prompt and model’s response, we ask GPT-3.5-Turbo555For chat evaluation, we
use gpt-3.5-turbo-1106 due to its longer context window and cheaper inference
costs. to grade the helpfulness of the response on a 6-point Likert scale (see
Figure 16 for the prompt used). For each evaluation instance, we provide the
prompt and model-generated response directly in the target language, which we
found to be on par with evaluating via first translating responses into
English (cf. Hada et al., 2023) (see Appendix C for more details). As noted by
Chen et al. (2023), GPT-3.5-Turbo sometimes ignores the fact that the output
language differs from the input language. Therefore, we force a score of 1
(indicating least helpful) if the language of the response does not match the
intended target language.

#### Results

Figure 2:  Average helpfulness of chat responses from Llama 2 7b with
incremental multilingual instruction tuning. Striped bars indicate that the
target language is included in the finetuning data. Error bars show a
confidence interval of 95%.  Figure 3:  X-QuAD results for Llama 2 7b with
incremental multilingual instruction tuning. Results are shown for both
multilingual prompting (en:xx) and monolingual prompting (xx:xx). Striped bars
indicate that the target language is included in the finetuning data. Error
bars show a confidence interval of 95%.

Figure 2 shows the helpfulness scores assigned by GPT-3.5-Turbo for all models
for each of the target languages.

For English, we observe a reasonably strong and stable performance across all
incremental multilingual instruction tuning settings, indicating that chat
performance does not diminish with the increase in non-English training
examples. In contrast, for most non-English target languages, performance
increases significantly when moving from monolingual to bilingual instruction
tuning. Most notably though, performance tends to plateau when instruction
tuning with as few as three different languages. This observation holds for
languages in both the supervised setting and the zero-shot setting, indicating
that three finetuning languages is sufficient to elicit zero-shot cross-
lingual transfer on this task. For languages in the extremely low-resource
category, performance remains low despite multilingual finetuning. Manual
inspection of these outputs reveal that, while these responses generally look
convincing at first glance and are sufficient for language identification (see
Figure 1), they are mostly nonsensical. This suggests that useful zero-shot
cross-lingual transfer abilities of Llama 2 7b are limited to languages seen
with higher frequency during pretraining.

###  4.2 Extractive Question Answering

Figure 4:  X-CSQA results for Llama 2 7b with incremental multilingual
instruction tuning. Striped bars indicate that the target language is included
in the finetuning data. Error bars show a confidence interval of 95%.  Figure
5:  XNLI results for Llama 2 7b with incremental multilingual instruction
tuning. Striped bars indicate that the target language is included in the
finetuning data. Error bars show a confidence interval of 95%.

In contrast to open-ended questions commonly used to query LLMs in chat
settings, extractive question answering requires the model to identify
relevant answer spans within longer context passages provided as part of the
prompt. This kind of task closely resembles a retrieval augmented generation
(RAG) setting, which is a popular method for extending an LLM’s knowledge with
additional data not available during training Lewis et al. (2020); Izacard and
Grave (2021). To evaluate our models on this task in multiple target
languages, we use XQuAD Artetxe et al. (2020).666We report results measured on
the validation split of XQuAD since labels for the test split are not publicly
available. This provides 1,190 QA pairs that were professionally translated
into different languages.

For this task, we consider both a multilingual prompt setting and a
monolingual prompt setting. The former presents the task instruction in
English and provides the context passage and question in the relevant target
language (en:xx), while the latter is entirely in the target language (xx:xx).
As a starting point, we borrow the English prompt from Lai et al. (2023a) and
manually translate it into each of the target languages considered.
Additionally, we include a standardised response prefix as part of the prompt,
effectively force-decoding the response “Based on the passage, the answer to
the question is”. This allows us to better isolate the relevant answer string
in the generative model’s output. A response is considered correct if the
model’s answer string matches the ground truth answer after minimal
postprocessing.777We find that some postprocessing of model outputs is
necessary for certain languages. Specifically, when queried with German and
Russian prompts, all models consistently repeated the question before
providing the extracted answer. To handle such edge cases, we strip away the
question and truncate the system output to a maximum of 50 characters or the
first line break, whichever comes first. An example of our prompting strategy
for this task and model outputs is shown in Table 2.

#### Results

Figure 3 shows the performance on each of our target languages in XQuAD given
incremental multilingual instruction tuning.

Similar to the results on the chat task (§4.1), we observe no performance
degradation on English as the ratio of multilingual instructions increases.
For most non-English target languages in the supervised setting, we see that
when using the multilingual prompting strategy (en:xx), performance remains
relatively uniform, with the exception of Chinese. In contrast, given the
monolingual prompting strategy (xx:xx), multilingual instruction tuning can
lead to small but noticeable gains in performance over monolingual tuning.
Specifically, for German and Chinese, performance improves when moving from
monolingual to bilingual instruction tuning and again plateaus rather quickly
with as few as three languages. For languages in the extremely low-resource
setting (Hindi and Greek), multilingual instruction tuning fails to improve
the model’s ability to handle these languages. While performance gains on this
task are generally less pronounced than on the open-ended chat task, we note
that zero-shot extractive QA is inherently challenging for LLMs tuned on
conversational instructions as they tend to generate verbose responses rather
than the single word or entity that correctly matches the ground truth.

###  4.3 Commonsense Reasoning

Effectively understanding natural language requires some representation of the
natural world and how concepts can relate to one another. Commonsense
knowledge describes the set of general facts that reflects this. To evaluate
how well English-centric models can reason across languages, we use the X-CSQA
dataset.888We report results for X-CSQA measured on the validation set of
1,000 questions. This dataset consists of questions paired with multiple
choice answers which aim at assessing general, language-agnostic world
knowledge involving different types of commonsense reasoning Talmor et al.
(2019); Lin et al. (2021). Given a question and a set of five possible answers
from A-E, we prompt the model to output the letter corresponding to the answer
that best answers the question. Again, we borrow the English prompt template
for this task from Lai et al. (2023a) as a starting point and translate it
into each target language. An example of the prompt used is given in Table 3.
Finally we assess performance with the monolingual prompting strategy (xx:xx).

#### Results

Figure 4 shows the accuracy on X-CSQA given incremental multilingual
instruction tuning. Again, for English, we observe that multilingual
instruction tuning does not significantly degrade performance. However, for
non-English target languages, incremental multilingual instruction tuning
fails to deliver improvements over monolingual tuning. This contrasts with the
results on the previous tasks considered.

Figure 6:  Chat performance of Llama 2 7b given multilingual instruction
tuning using native non-English (Native NE) examples compared to using
translated non-English (MT NE) examples. Error bars show a confidence interval
of 95%.

###  4.4 XNLI

Natural language inference (NLI) is an important skill for LLMs, especially as
input and output sequences grow to consist of multiple sentences. Given two
sentences, this task aims to recognise a relationship between them either as
entailment, contradiction or neutral. To assess an LLM’s ability to solve this
task given multilingual instruction tuning, we use XNLI (Conneau et al., 2018)
and evaluate performance on the official test split.

For this task, we use the implementation in the lm-evaluation harness Gao et
al. (2023)999https://github.com/EleutherAI/lm-evaluation-harness, which uses
language-specific prompts for each target language (i.e. monolingual prompting
(xx:xx)). Instead of querying the model to generate the desired label,
multiple queries are constructed for each test instance (one for each possible
label) and scored by the model. The sequence with the highest likelihood is
chosen as the model’s answer.

#### Results

Figure 5 shows the accuracy measured on our target languages from XNLI given
incremental multilingual instruction tuning. Strikingly, performance for all
target languages remains uniform regardless of the amount of multilinguality
used in finetuning. Furthermore, the equal performance between the Guanaco
baseline and our Mono and Multi-$i$ finetuned models suggests that scaling up
instruction diversity and the number of languages seen during finetuning
beyond just six languages is still insufficient for delivering noticeable
gains on this task.

##  5 Further Analysis

The results on four distinct evaluation tasks show that multilingual
instruction tuning benefits open-ended chat in non-English languages most
strongly (§4.1). In this section, we focus on this task to investigate the
impact of instruction diversity and model scaling.

###  5.1 Instruction Diversity vs. Language Diversity

Figure 7:  Chat performance of Llama 2 70b (denoted by triangular points) and
Llama 2 7b (semi-transparent bars) with incremental multilingual instruction
tuning. Striped bars indicate that the target language is included in the
finetuning data. Error bars show a confidence interval of 95%.

Diversity is a key factor for sample efficient instruction tuning Zhou et al.
(2023). Since our experiments in §4 make use of native non-English training
instances from the OpenAssistant dataset, a potential confounding factor could
be that adding more languages during finetuning also introduces more diverse
training instructions. To investigate this, we retrain Multi-$i$ models using
translated instruction-tuning data from Mono in place of native non-English
examples, following the same incremental recipe as described in §3.2. This
ensures that the data distribution remains constant as multilinguality
increases. As a translation engine, we again use GPT-3.5-Turbo101010Since
conversational training instances can be quite long, sometimes exceeding the
default 4k token context window, we use gpt-3.5-tubo-16k for this translation
task. and the prompt template provided in Figure 10.

Figure 6 compares the resulting performance on the open-ended chat task on a
subset of our target languages (results for the remaining target languages are
provided in Figure 15). Notably, we observe no significant differences between
tuning with distinctly native non-English examples compared to those derived
via automatic translation. This indicates that the gains attributed to
increased multilinguality are not conflated with an increase in the diversity
of instructions.

###  5.2 Scaling up Model Size

In order to investigate the effect of model scaling, we also repeat our chat
experiments using Llama 2 70b as the underlying base model. Figure 7 shows the
resulting helpfulness scores assigned by the LLM judge. Most notably,
performance on non-English languages seen relatively frequently during
pretraining is dramatically improved, often matching that of English.
Secondly, we observe that the larger model’s performance on most non-English
target languages tends to plateau with just two finetuning languages, unlike
the smaller model that required three. Finally, while the performance on
languages in the extremely low-resource scenario remains low, it exhibits a
substantial relative improvement compared to the 7-billion parameter model.
These results indicate that model scaling is extremely beneficial for
exploiting the multilingual capabilities in English-centric models. This
aligns with the assertion of Shaham et al. (2023) that larger models are more
adept at handling multilinguality.

##  6 Discussion

Our findings show that multilingual instruction tuning can elicit zero-shot
cross-lingual transfer in English-centric models, though its effectiveness on
downstream performance varies across tasks. Notably, we observe significant
performance gains for open-ended chat and noticeable gains on extractive QA
with monolingual prompting. In comparison, for highly structured tasks such as
X-CSQA and XNLI that impose a strict constraint on the output space regardless
of the input language (e.g., choosing from options ‘A’, ‘B’, ‘C’, etc.),
multilingual instruction tuning does not seem to provide substantial benefits.
This distinction highlights that multilingual instruction tuning is most
beneficial for open-ended generative tasks where there is an implicit
agreement between the input query language and the language output by the
model.

We posit that our findings align with the superficial alignment hypothesis
Zhou et al. (2023). While instruction tuning guides the model towards a
desirable ‘subdistribution of formats’ to use, a small amount of multilingual
instruction data allows the model to learn a simple mapping between input and
output languages. In essence, it helps steer the model towards language-
specific ‘subdistributions of formats’ that agree with the language of the
input. Therefore, we conclude that multilingual instruction tuning is of most
benefit in scenarios where IO language agreement is required.

Most surprisingly, we find that multilingual instruction tuning with just
three languages is sufficient to encourage IO language agreement in smaller
LLMs, resulting in improved cross-lingual transfer on generative tasks. As
model size increases, however, the number languages required decreases from
three to two, indicating that larger models can learn this mapping easier. In
both model sizes, adding more instruction tuning languages – including the
target language itself – provides no significant gains.

Despite benefits from cross-lingual transfer, our results highlight a clear
gap in performance between English and non-English target languages on all
tasks. This indicates that small-scale multilingual instruction tuning is
insufficient to improve the intrinsic multilingual capabilities of English-
centric LLMs and that more advanced interventions may be required (e.g.
Pfeiffer et al., 2020; ImaniGooghari et al., 2023).

##  7 Conclusion

In this paper, we investigated the use of multilingual instruction tuning to
elicit multilingual capabilities of English-centric LLMs. Our results show
that finetuning with as few as three languages can promote cross-lingual
transfer and allow models to better exploit the relatively small amounts of
non-English data seen during pretraining. Experiments conducted on four
distinct tasks revealed that this can lead to significant performance
improvements on open-ended generative tasks that assume input and output
language agreement. Furthermore, we observed that these effects are amplified
in larger models, requiring only two finetuning languages for successful
multilingual instruction tuning. The effectiveness of cross-lingual transfer,
which reduces the need for extensive multilingual instruction tuning, is good
news for LLM developers. However, future work could explore other methods to
encourage smaller models to better leverage the limited non-English data seen
during pretraining and whether there are tasks for which language-specific
instruction tuning is of higher importance.

##  8 Limitations

This work focused on detecting and evaluating non-English responses from
English-centric LLMs. Our experiments demonstrated that cross-lingual
transfer, which can be elicited given small amount of multilingual instruction
tuning, can be effective for improvinging the non-English performance of
English-centric models. That said, our analysis focused primarily on Indo-
European languages, except for Chinese. As a consequence, it remains unclear
how well our results generalise to more distant or topologically diverse
languages, however, performance gains observed on Chinese suggest that
generalisation is more dependent on exposure during pretraining than language
similarity.

For our experiments and analysis on model training data, we relied on
automatic language identification. To this end, we employed the OpenLID model
from Burchell et al. (2023). Despite low error rates achieved by this model,
language identification is not perfect and can lead to some texts being
misidentified. To mitigate the risk of language contamination Blevins and
Zettlemoyer (2022) in our controlled experiments, we only included training
examples whose language is identified with a confidence threshold $\geq$ 0.8.

In §5.1 we investigated the impact of multilingual diversity versus training
example diversity. While our findings revealed that there is no significant
difference between these two settings, we note that even when finetuning with
the original native non-English examples, task diversity may be inherently
limited by design of the data collection. For instance, regardless of the
language used, crowdworkers were asked to follow the same set of
guidelines111111https://projects.laion.ai/Open-
Assistant/docs/guides/guidelines when creating the data.

## Acknowledgements

We sincerely thank our friends and colleagues, Fabian Aiolfi, Thea Bächler,
Anastassia Shaitarova, Finnur Ágúst Ingimundarson, Cui Ding, Andrianos
Michail, Janine Aeberhard, Arnisa Fazla, Farhad Nooralahzadeh, Omnia Ibrahim,
Xuan Lam and Manasi Muglikar, for helping us with language-specific questions
and validating translations used in our experiments, as well as Lena Bolliger
and Patrick Haller for providing valuable feedback. Rico Sennrich acknowledges
funding by the Swiss National Science Foundation (project MUTAMUR; no.
213976).

## References

  * Almazrouei et al. (2023) Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023.  The falcon series of open language models.
  * Armengol-Estapé et al. (2022) Jordi Armengol-Estapé, Ona de Gibert Bonet, and Maite Melero. 2022.  On the multilingual capabilities of very large-scale English language models.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 3056–3068, Marseille, France. European Language Resources Association.
  * Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020.  On the cross-lingual transferability of monolingual representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4623–4637, Online. Association for Computational Linguistics.
  * Bang et al. (2023) Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023.  A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.
  * Blevins and Zettlemoyer (2022) Terra Blevins and Luke Zettlemoyer. 2022.  Language contamination helps explains the cross-lingual capabilities of English pretrained models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 3563–3574, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Burchell et al. (2023) Laurie Burchell, Alexandra Birch, Nikolay Bogoychev, and Kenneth Heafield. 2023.  An open dataset and model for language identification.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 865–879, Toronto, Canada. Association for Computational Linguistics.
  * Chen et al. (2023) Pinzhen Chen, Shaoxiong Ji, Nikolay Bogoychev, Barry Haddow, and Kenneth Heafield. 2023.  Monolingual or multilingual instruction tuning: Which makes a better alpaca.
  * Chiang and Lee (2023) Cheng-Han Chiang and Hung-yi Lee. 2023.  Can large language models be an alternative to human evaluations? In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15607–15631, Toronto, Canada. Association for Computational Linguistics.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
  * Conover et al. (2023) Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023.  Free dolly: Introducing the world’s first truly open instruction-tuned llm.
  * Costa-jussà et al. (2022) Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022.  No language left behind: Scaling human-centered machine translation.
  * Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023.  Qlora: Efficient finetuning of quantized llms.  _arXiv preprint arXiv:2305.14314_.
  * Ding et al. (2023) Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023.  Enhancing chat language models by scaling high-quality instructional conversations.
  * Dubois et al. (2023) Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Alpacafarm: A simulation framework for methods that learn from human feedback.
  * Gao et al. (2023) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2023.  A framework for few-shot language model evaluation.
  * Hada et al. (2023) Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, and Sunayana Sitaram. 2023.  Are large language model-based evaluators the solution to scaling up multilingual evaluation?
  * Hendy et al. (2023) Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023.  How good are gpt models at machine translation? a comprehensive evaluation.
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022.  Training compute-optimal large language models.
  * Holmström et al. (2023) Oskar Holmström, Jenny Kunz, and Marco Kuhlmann. 2023.  Bridging the resource gap: Exploring the efficacy of English and multilingual LLMs for Swedish.  In _Proceedings of the Second Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2023)_ , pages 92–110, Tórshavn, the Faroe Islands. Association for Computational Linguistics.
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.
  * ImaniGooghari et al. (2023) Ayyoob ImaniGooghari, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André Martins, François Yvon, and Hinrich Schütze. 2023.  Glot500: Scaling multilingual corpora and language models to 500 languages.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1082–1117, Toronto, Canada. Association for Computational Linguistics.
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Leveraging passage retrieval with generative models for open domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 874–880, Online. Association for Computational Linguistics.
  * Jiao et al. (2023) Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023.  Is chatgpt a good translator? yes with gpt-4 as the engine.
  * Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023.  Efficient memory management for large language model serving with pagedattention.
  * Köpf et al. (2023) Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. 2023.  Openassistant conversations–democratizing large language model alignment.  _arXiv preprint arXiv:2304.07327_.
  * Lai et al. (2023a) Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023a.  Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning.
  * Lai et al. (2023b) Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2023b.  Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback.
  * Laskar et al. (2023) Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Huang. 2023.  A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 431–469, Toronto, Canada. Association for Computational Linguistics.
  * Le Scao et al. (2023) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, Dragomir Radev, Eduardo González Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, María Grandury, Mario Šaško, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaieh Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Lepercq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szczechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Matteo Manica, Nihal Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry, Trishala Neeraj, Urmish Thakker, Vikas Raunak, Xiangru Tang, Zheng-Xin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jaesung Tae, Jason Phang, Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, Sanchit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurélie Névéol, Charles Lovering, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bogdanov, Genta Indra Winata, Hailey Schoelkopf, Jan-Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shani Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdeněk Kasner, Alice Rueda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh HajiHosseini, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Muñoz Ferrandis, Danish Contractor, David Lansky, Davis David, Douwe Kiela, Duong A. Nguyen, Edward Tan, Emi Baylor, Ezinwanne Ozoani, Fatima Mirza, Frankline Ononiwu, Habib Rezanejad, Hessie Jones, Indrani Bhattacharya, Irene Solaiman, Irina Sedenko, Isar Nejadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Karen Fort, Livia Dutra, Mairon Samagaio, Maraim Elbadri, Margot Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Rajani, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas Wang, Sourav Roy, Sylvain Viguier, Thanh Le, Tobi Oyebade, Trieu Le, Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, Antonio Miranda-Escalada, Ayush Singh, Benjamin Beilharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñán, Daniel Molano, Dian Yu, Enrique Manjavacas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Rangasai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Marc Pàmies, Maria A Castillo, Marianna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cullan, Michael Weinberg, Michiel De Wolf, Mina Mihaljcic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Sinee Sang-aroonsiri, Srishti Kumar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gigant, Tomoya Kainuma, Wojciech Kusa, Yanis Labrak, Yash Shailesh Bajaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. 2023.  Bloom: A 176b-parameter open-access multilingual language model.
  * Le Scao et al. (2022) Teven Le Scao, Thomas Wang, Daniel Hesslow, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang, Ofir Press, Colin Raffel, Victor Sanh, Sheng Shen, Lintang Sutawika, Jaesung Tae, Zheng Xin Yong, Julien Launay, and Iz Beltagy. 2022.  What language model to train if you have one million GPU hours? In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 765–782, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474.
  * Li et al. (2023) Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. 2023.  Loftq: Lora-fine-tuning-aware quantization for large language models.
  * Lin et al. (2021) Bill Yuchen Lin, Seyeon Lee, Xiaoyang Qiao, and Xiang Ren. 2021.  Common sense beyond English: Evaluating and improving multilingual language models for commonsense reasoning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1274–1287, Online. Association for Computational Linguistics.
  * Lin et al. (2022) Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2022.  Few-shot learning with multilingual generative language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Longpre et al. (2023) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023.  The flan collection: Designing data and methods for effective instruction tuning.
  * Lu et al. (2023) Hongyuan Lu, Haoyang Huang, Dongdong Zhang, Haoran Yang, Wai Lam, and Furu Wei. 2023.  Chain-of-dictionary prompting elicits translation in large language models.
  * Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022.  Cross-task generalization via natural language crowdsourcing instructions.
  * Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023.  Crosslingual generalization through multitask finetuning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15991–16111, Toronto, Canada. Association for Computational Linguistics.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 27730–27744. Curran Associates, Inc.
  * Penedo et al. (2023a) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023a.  The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only.  _arXiv preprint arXiv:2306.01116_.
  * Penedo et al. (2023b) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023b.  The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only.
  * Pfeiffer et al. (2020) Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020.  MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7654–7673, Online. Association for Computational Linguistics.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022.  Multitask prompted training enables zero-shot task generalization.
  * Shaham et al. (2023) Uri Shaham, Maha Elbayad, Vedanuj Goswami, Omer Levy, and Shruti Bhosale. 2023.  Causes and cures for interference in multilingual translation.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15849–15863, Toronto, Canada. Association for Computational Linguistics.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  CommonsenseQA: A question answering challenge targeting commonsense knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a.  Llama: Open and efficient foundation language models.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b.  Llama 2: Open foundation and fine-tuned chat models.
  * Wang et al. (2023) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023.  Self-instruct: Aligning language models with self-generated instructions.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13484–13508, Toronto, Canada. Association for Computational Linguistics.
  * Wang et al. (2022) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022.  Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5085–5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022.  Finetuned language models are zero-shot learners.
  * Wu and Dredze (2019) Shijie Wu and Mark Dredze. 2019.  Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 833–844, Hong Kong, China. Association for Computational Linguistics.
  * Ye et al. (2023) Jiacheng Ye, Xijia Tao, and Lingpeng Kong. 2023.  Language versatilists vs. specialists: An empirical revisiting on multilingual transfer ability.
  * Zhang et al. (2023a) Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. 2023a.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.
  * Zhang et al. (2023b) Zhihan Zhang, Dong-Ho Lee, Yuwei Fang, Wenhao Yu, Mengzhao Jia, Meng Jiang, and Francesco Barbieri. 2023b.  Plug: Leveraging pivot language in cross-lingual instruction tuning.  _arXiv preprint arXiv:2311.08711_.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _CoRR_ , abs/2306.05685.
  * Zhou et al. (2023) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023.  Lima: Less is more for alignment.

##  Appendix A Pretraining Data for English-centric LLMs

LLMs are pretrained on enormous amounts of unlabelled text gathered from
existing corpora and the web. With the notable exception of BLOOM (Le Scao et
al., 2023), very limited information is offered about the distribution of
languages represented in datasets used to train LLMs and the subsequent
performance on non-English languages. Table 1 provides an overview of the
document-level language distributions of the LLMs used in this paper. For
Llama 2, the information is taken from the original paper Touvron et al.
(2023b), in which the authors analyse the training data using a fastText
language classifier on corpus documents with a threshold of 0.5. For GPT-3, we
use the official dataset statistics made available on
Github121212https://github.com/openai/gpt-3/blob/master/dataset_statistics/languages_by_document_count.csv
which provide document-level language identification information. To gather
statistics for Falcon, we inspected a sample of the RefinedWeb dataset Penedo
et al. (2023a) that was constructed to train this family of
models.131313https://huggingface.co/datasets/tiiuae/falcon-refinedweb Using
the OpenLID fastText model from Burchell et al. (2023), we identify the most
frequent document-level languages based on approximately 320 million examples
from this corpus. Languages are identified with a confidence threshold of
$\geq$ 0.6 and predictions below this threshold are aggregated under
‘unknown’.

Llama 2 | Falcon | GPT-3
---|---|---
# | Language | Percent | # | Language | Percent | # | Language | Percent
1 | English | 89.70% | 1 | English | 94.790% | 1 | English | 93.69%
2 | unknown | 8.38% | 2 | unknown | 4.944% | 2 | German | 1.20%
3 | German | 0.17% | 3 | Standard Malay | 0.031% | 3 | French | 1.02%
4 | French | 0.16% | 4 | Swahili | 0.027% | 4 | Portuguese | 0.64%
5 | Swedish | 0.15% | 5 | French | 0.016% | 5 | Italian | 0.58%
6 | Chinese | 0.13% | 6 | Portuguese | 0.012% | 6 | Spanish | 0.51%
7 | Spanish | 0.13% | 7 | Chinese (Traditional) | 0.011% | 7 | Dutch | 0.37%
8 | Russian | 0.13% | 8 | Sardinian | 0.011% | 8 | Polish | 0.25%
9 | Dutch | 0.12% | 9 | Dutch | 0.009% | 9 | Japanese | 0.25%
10 | Italian | 0.11% | 10 | Spanish | 0.009% | 10 | Danish | 0.16%
11 | Japanese | 0.10% | 11 | Italian | 0.008% | 11 | Norwegian | 0.15%
12 | Polish | 0.09% | 12 | German | 0.008% | 12 | Romanian | 0.13%
13 | Portuguese | 0.09% | 13 | Danish | 0.005% | 13 | Finnish | 0.13%
14 | Vietnamese | 0.08% | 14 | Russian | 0.005% | 14 | Chinese (Simplified) | 0.12%
15 | Ukranian | 0.07% | 15 | Indonesian | 0.005% | 15 | Russian | 0.11%
16 | Korean | 0.06% | 16 | Swedish | 0.005% | 16 | Czech | 0.10%
17 | Catalan | 0.04% | 17 | Somali | 0.005% | 17 | Swedish | 0.06%
18 | Serbian | 0.04% | 18 | Vietnamese | 0.004% | 18 | Hungarian | 0.06%
19 | Bahasa Indonesian | 0.03% | 19 | Venetian | 0.004% | 19 | Chinese (Traditional) | 0.04%
20 | Czech | 0.03% | 20 | Chinese (Simplified) | 0.004% | 20 | Bahasa Indonesian | 0.04%
21 | Finnish | 0.03% | 21 | Tagalog | 0.004% | 21 | Croation | 0.04%
22 | Hungarian | 0.03% | 22 | Northern Uzbek | 0.003% | 22 | Turkish | 0.04%
23 | Norwegian | 0.03% | 23 | Limburgish | 0.003% | 23 | Catalan | 0.03%
24 | Romanian | 0.03% | 24 | Polish | 0.003% | 24 | Vietnamese | 0.03%
25 | Bulgarian | 0.02% | 25 | Turkish | 0.003% | 25 | Slovenian | 0.03%
26 | Danish | 0.02% | 26 | Norwegian Nynorsk | 0.003% | 26 | Estonian | 0.02%
27 | Slovenian | 0.01% | 27 | Norwegian Bokmal | 0.003% | 27 | Slovak | 0.02%
28 | Croatian | 0.01% | 28 | Romanian | 0.002% | 28 | Korean | 0.02%
⋮ | ⋮ | ⋮
n/a | Icelandic | n/a | 30 | Greek | 0.002% | 31 | Greek | 0.017%
⋮ | ⋮ | ⋮
n/a | Hindi | n/a | 60 | Bulgarian | 0.001% | 45 | Hindi | 0.004%
⋮ | ⋮ | ⋮
n/a | Greek | n/a | 72 | Icelandic | 0.001% | 47 | Icelandic | 0.004%
⋮ | ⋮ | 48 | Bulgarian | 0.003%
⋮ | 102 | Hindi | 0.0001% | ⋮
⋮ | ⋮ | ⋮

Table 1:  Distribution of document languages in pretraining datasets for
popular English-centric LLMs. For Llama 2 and GPT-3, statistics are taken from
the original works. For Falcon, statistics are estimated based on a sample of
the RefinedWeb corpus (roughly 320 million documents). Note that for Llama 2
and Falcon, ‘unknown’ indicates texts identified below a predefined confidence
threshold (0.5 for Llama 2, 0.65 for our analysis with RefinedWeb). Touvron et
al. (2023b) state that this data partially pertain to programming code. Target
languages for our experiments are highlighted in bold.

##  Appendix B Translating AlpacaEval Prompts

You are a helpful assistant. Translate the following text into
{{target_language}}. Keep the structure of the original text and preserve
things like code and names. Please ensure that your response contains only the
translated text. The translation must convey the same meaning as the original
and be natural for native speakers with correct grammar and proper word
choices. Your translation must also use exact terminology to provide accurate
information even for the experts in the related fields. Original:
{{source_text}} Translation into {{target_language}}: Figure 8:  Prompt
template used to translate AlpacaEval with GPT-3.5-Turbo. Values in curly
braces represent placeholders. The value specified for ‘target language’ is
English name of the language (e.g., ‘German’) we are translating into, except
for Greek and Chinese, where we specify standard modern Greek and Mandarin
Chinese respectively.

To translate AlpacaEval prompts from English into each target language, we
query GPT-3.5-Turbo using the template in Figure 8. To validate the translated
prompts, we manually inspected a sample of the outputs in various languages.
This inspection revealed that the translations were typically decent, although
often included literal translations for metaphorical expressions rather than
how a native speaker might express themselves. For instance, English ‘bullet
points’ was translated literally into Hindi, rather than an arguably more
appropriate phrasing such as ‘important points’. Additionally, in languages
that distinguish between formal and informal or gendered pronouns (e.g.,
German, French, Hindi), the formal and male forms are dominant, which may be
less representative of how native speakers actually interact with LLM
chatbots.

##  Appendix C Direct vs. Translated Evaluation for Non-English Chat Responses

Figure 9: Distribution of helpfulness scores assigned by our LLM judge,
GPT-3.5-Turbo, using direct evaluation in the specified target language and
after translating target-language responses to English.

While using a powerful LLM to evaluate the outputs of other models has been
shown to achieve reasonable agreement with human judgements in English (Zheng
et al., 2023; Chiang and Lee, 2023), it is unclear whether this agreement
transfers to all languages under investigation. Recent work by Hada et al.
(2023) has shown that agreement between human and LLM judges tends to be lower
for non-English languages, especially in the case of low-resource and non-
Latin scripted languages, where the LLM judge tends to be overly optimistic in
its assessment. However, for certain assessment criteria, such as linguistic
acceptability and general content quality, they also confirm that inter-
annotator agreement between LLM-based evaluators and humans is inline with
that of multiple human annotators.

To investigate this potential bias, we compared scores assigned by the LLM
judge on model-generated responses directly in each non-English target
language and on their English translations. For each non-English prompt-
response pair, we use GPT-3.5-Turbo (gpt-3.5-turbo-1106) and the prompt shown
in Figure 8, specifying English as the target language. The resulting
translated response is then paired with its corresponding AlpacaEval prompt in
its original English form and presented to the LLM judge using the prompt in
Figure 16.

Figure 9 shows that the distribution of assigned scores in the direct and
translated evaluation settings is very similar for most languages. For
languages that use non-Latin scripts (e.g., Chinese, Russian, Bulgarian), we
observe that GPT-3.5-Turbo tends to assign slightly higher scores more
frequently when evaluating directly on the non-English prompt/response pairs.
This finding agrees with those from Hada et al. (2023) and indicates that LLM-
based evaluations in non-English languages tend to be slightly overly
optimistic and should be considered with caution. Nevertheless, we observe
that the discrepancy between direct and translated evaluation is relatively
minor for the languages considered and leads to negligible differences in the
overall average helpfulness score. Because of this, and to keep evaluation
costs to a minimum, we opt to use the direct evaluation strategy for our
experiments.

##  Appendix D Translating Non-English Guanaco Training Examples into English

You are a helpful assistant. Translate the following conversation between a
human and an AI assistant into {{target_language}}. Keep the structure of the
original text and preserve things like code, names and role labels (e.g. <S1>,
<S2>). Please ensure that your response contains only the translated text. The
translation must convey the same meaning as the original and be natural for
native speakers with correct grammar and proper word choices. Your translation
must also use exact terminology to provide accurate information even for the
experts in the related fields. Original: {{training_instance}} Translation
into {{target_language}} Figure 10:  Prompt template used to translate Guanaco
training instances from English into various target languages.

In order to investigate the effect of language diversity compared to
instruction diversity, we translate a subset of Guanaco’s English training
examples into non-English target languages and use these to create MT-based
Multi-$i$ instruction-tuning datasets. By default, speaker roles in Guanaco
are denoted with ‘### Human:’ and ‘### Assistant’. To ensure that these are
never translated and the dialogue structure is maintained, we substitute them
with special tokens ‘<S1>’ and ‘<S2>’ and explicitly tell the model to leave
these tokens intact (see Figure 10). Before training, the special tokens back
to their original form.

##  Appendix E Results with Falcon 7b

In order to assess whether our findings generalise to other LLMs, we repeat
our experiments using Falcon 7b Almazrouei et al. (2023).

###  E.1 Chat

Figure 11 shows the helpfulness scores assigned by our LLM judge for Falcon 7b
given incremental multilingual instruction tuning across all target languages.
Similarly to our results with Llama 2 7b (see Figure 2), cross-lingual
transfer is elicited after finetuning with relatively few languages, and no
additional gains observed when including more than three languages. That said,
Falcon 7b appears to show strong performance on French, even without
multilingual finetuning, indicating that, despite being an English-centric
model, it has strong capabilities in French out of the box. For Spanish,
German and Chinese, performance is comparable to that of Llama 2 7b. However,
for all other languages, responses are often ranked least helpful, indicating
that Falcon 7b’s multilingual capabilities are limited strictly to major
European languages and Chinese.

###  E.2 Extractive Question Answering

Figure 12 shows the results of Falcon 7b on XQuAD. While performance is
generally lower than that of Llama 2 7b on this task (see Figure 3), we
observe a similar, albeit weaker, effect of multilingual finetuning within the
supported languages (Spanish, German, Chinese).

###  E.3 Commonsense Reasoning

Figure 13 shows the results of Falcon 7b on X-CSQA. Strikingly, in contrast to
the results achieved with Llama 2 7b (Figure 4), Falcon 7b fails to score
above random performance across all target languages. Regarding the effect of
multilingual instruction tuning, we again see that it fails to deliver any
performance improvements on this highly structured task.

###  E.4 XNLI

Figure 14 shows the results of Falcon 7b on XNLI. Similar to the results
attained with Llama 2 7b (see Figure 5), we observe no significant differences
in performance given different degrees of multilingual instruction tuning.

###  E.5 Discussion

Overall, these experiments indicate that Falcon’s multilingual capabilities
are considerably narrower than Llama 2’s. We hypothesise that this may be a
result of more stringent filtering of web-scraped pretraining data Penedo et
al. (2023b), possibly resulting in less accidental contamination.
Interestingly though, according to our estimated statistics based on the
sample of the RefinedWeb corpus (Table 1), which makes up a large portion of
Falcon’s pretraining data, Russian appears as frequently as Chinese and
Swedish. However, we find that Russian texts are, on average, three times
longer when tokenized with Falcon’s tokenizer compared to Llama 2’s. This
indicates that, despite having a much larger vocabulary (65k for Falcon vs.
32k for Llama 2), Cyrillic-scripted languages such as Russian and Bulgarian
are outside of the model’s intended domain and performance on these languages
is inherently limited.

Figure 11:  Average helpfulness of chat responses from Falcon 7b with
incremental multilingual instruction tuning. Striped bars indicate that the
target language is included in the finetuning data. Error bars show a
confidence interval of 95%.  Figure 12:  X-QuAD results for Falcon 7b with
incremental multilingual instruction tuning. Results are shown for both
multilingual prompting (en:xx) and monolingual prompting (xx:xx). Striped bars
indicate that the target language is included in the finetuning data. Error
bars show a confidence interval of 95%.  Figure 13:  X-CSQA results for Falcon
7b with incremental multilingual instruction tuning. Striped bars indicate
that the target language is included in the finetuning data. Error bars show a
confidence interval of 95%.  Figure 14: XNLI results for Falcon 7b with
incremental multilingual instruction tuning. Striped bars indicate that the
target language is included in the finetuning data. Error bars show a
confidence interval of 95%.

Figure 15:  Comparison of Llama 2 7b with multilingual instruction tuning
using native (Native NE) and translated non-English (MT NE) examples for
additional target languages not shown in Figure 6. Error bars show a
confidence interval of 95%.  You are an expert language evaluator.

You are evaluating a response that has been submitted for a particular task,
using a specific set of standards. Below is the data:

[BEGIN DATA]
***
[Task]: {{task}}
***
[Submission]: {{submission}}
***
[Criterion]: helpfulness:
‘‘1’’: ‘‘Not helpful - The generated text is completely irrelevant, unclear,
or incomplete. It does not provide any useful information to the user.’’
‘‘2’’: ‘‘Somewhat helpful - The generated text has some relevance to the
user’s question, but it may be unclear or incomplete. It provides only partial
information, or the information provided may not be useful for the user’s
needs.’’
‘‘3’’: ‘‘Moderately helpful - The generated text is relevant to the user’s
question, and it provides a clear and complete answer. However, it may lack
detail or explanation that would be helpful for the user.’’
‘‘4’’: ‘‘Helpful - The generated text is quite relevant to the user’s
question, and it provides a clear, complete, and detailed answer. It offers
additional information or explanations that are useful for the user. However,
some of the points of the response are somewhat repetitive or could be
combined for greater clarity and concision.’’
‘‘5’’: ‘‘Very helpful - The generated text is highly relevant to the user’s
question, and it provides a clear, complete, and detailed answer. It offers
additional information, explanations, or analogies that are not only useful
but also insightful and valuable to the user. However, the structured of the
response is not well-organized and there is no clear progression or logical
sequence of different points in the response.’’
‘‘6’’: ‘‘Highly helpful - The generated text provides a clear, complete, and
detailed answer. It offers additional information or explanations that are not
only useful but also insightful and valuable to the user. The response is also
in a logical and easy-to-follow manner by explicitly using headings, bullet
points, or numbered lists to break up the information and make it easier to
read.’’
***
[END DATA]

Does the submission meet the criterion? First, write out in a step by step
manner your reasoning about the criterion to be sure that your conclusion is
correct. Avoid simply stating the correct answers at the outset. Provide your
response as a RFC8259 compliant JSON following this schema:

{‘‘reasoning’’: str, ‘‘score’’: int} Figure 16: Prompt template used to query the LLM judge in order to assess the helpfulness of an open-ended chat response. This prompt closely follows the one used by Zhou et al. (2023) but extends it by specifying the return value as a valid JSON-like object. Prompt: |  ### Human: Answer the question from the given passage. Your answer should be directly extracted from the passage, and it should be a single entity, name, or number, not a sentence.

Passage:

Die Verteidigung der Panthers gab nur 308 Punkte ab und belegte den sechsten
Platz in der Liga, während sie die NFL mit 24 Interceptions in dieser
Kategorie anführte und sich mit vier Pro Bowl-Selektionen rühmen konnte. Pro
Bowl Defensive Tackle Kawann Short führte das Team mit 11 Sacks an, erzwang
zudem drei Fumbles und erzielte zwei Fumble Recoverys. Mario Addison,
ebenfalls Lineman, addierte $6\frac{1}{2}$ Sacks hinzu. Die Panthers-Line
präsentierte auch den erfahrenen Defensive End Jared Allen, einen 5-fachen
Pro-Bowler, der mit 136 Sacks der aktive Anführer in der NFL-Kategorie
Karriere-Sacks war, sowie den Defensive End Kony Ealy, der 5 Sacks in nur 9
Starts erzielte. Nach ihnen wurden zwei der drei Linebacker der Panthers
ausgewählt, um im Pro Bowl zu spielen: Thomas Davis und Luke Kuechly. Davis
erzielte $5\frac{1}{2}$ Sacks, vier erzwungene Fumbles und vier Interceptions,
während Kuechly das Team bei den Tackles anführte (118), zwei Fumbles erzwang
und vier Pässe abfing. Carolinas Secondarys bestanden aus dem Pro Bowl-Safety
Kurt Coleman, der das Team mit einem Karrierehoch von sieben Interceptions
anführte und gleichzeitig 88 Tackles erzielen konnte, und Pro Bowl-Cornerback
Josh Norman, der sich während der Saison zur Shutdown Corner entwickelte und
vier Interceptions erzielte, von denen zwei zu Touchdowns für sein Team
wurden.

Question:

Mit wie vielen Interceptions erzielte Josh Norman im Jahr 2015 Touchdowns?

Note: Your answer should be directly extracted from the passage and be a
single entity, name, or number. Provide the answer in quotations.

### Assistant: Based on the passage, the answer to the question is ‘‘
---|---
Reference: |  zwei
System: |  two".

In the passage, it is stated that Josh Norman “erzielte vier Interceptions,
von denen zwei zu Touchdowns für sein Team wurden.” This means that Norman
intercepted four passes during the season, and two of those interceptions were
returned for touchdowns.
Table 2:  Example input prompt, reference and model response for the XQuAD task. Monospaced font represents the prompt template used for each test instance. Here, we show the multilingual prompting strategy (en:xx), where the task instruction and response prefix is provided English, while the context passage and question are in the target language (in this case, German). A correct answer in this setting requires the model to continue the response by switching into the target language, which it fails to do in this particular example.  Prompt: |  ### Human: Dir wird eine Frage vorgelegt, die mehrere Antwortmöglichkeiten hat. Wähle die am besten geeignete Option aus ‘‘A’’, ‘‘B’’, ‘‘C’’, ‘‘D’’ oder ‘‘E’’ basierend auf deinem gesunden Menschenverstand. Bitte gebe deine Antwort in Form eines einzelnen Buchstabens in Anführungszeichen an.

Frage:

Sarah stellte ihr Essen in einen Mülleimer neben ihrem Schreibtisch. Wo mag
sie gewesen sein?

Antwortmöglichkeiten:

A: Bushaltestelle
B: Ecke
C: Motel
D: Park
E: Büro

### Assistant: ‘‘
---|---
Reference: |  E
System: |  E”
Table 3:  Example of the X-CSQA task. Monospaced font represents the prompt
template used for each test instance. Here, the prompt strategy is monolingual
(xx:xx) with the task instruction provided in the same language as the test
instance (in this case, German). A correct answer requires the model to output
the letter corresponding to the ground-truth answer.