# Understanding and Mitigating Language Confusion in LLMs

Kelly Marchisio, Wei-Yin Ko††footnotemark: , Alexandre Bérard, Théo Dehaze,
Sebastian Ruder††footnotemark:
Cohere Equal contribution. Correspondence<EMAIL_ADDRESS>
###### Abstract

We investigate a surprising limitation of LLMs: their inability to
consistently generate text in a user’s desired language. We create the
Language Confusion Benchmark (LCB) to evaluate such failures, covering 15
typologically diverse languages with existing and newly-created English and
multilingual prompts. We evaluate a range of LLMs on monolingual and cross-
lingual generation reflecting practical use cases, finding that Llama Instruct
and Mistral models exhibit high degrees of language confusion and even the
strongest models fail to consistently respond in the correct language. We
observe that base and English-centric instruct models are more prone to
language confusion, which is aggravated by complex prompts and high sampling
temperatures. We find that language confusion can be partially mitigated via
few-shot prompting, multilingual SFT and preference tuning. We release our
language confusion benchmark, which serves as a first layer of efficient,
scalable multilingual evaluation.111The data is available at
https://github.com/for-ai/language-confusion.

Understanding and Mitigating Language Confusion in LLMs

Kelly Marchisio††thanks: Equal contribution. Correspondence:
<EMAIL_ADDRESS>Wei-Yin Ko††footnotemark: , Alexandre Bérard, Théo
Dehaze, Sebastian Ruder††footnotemark:  Cohere

##  1 Introduction

Large language models (LLMs) are increasingly used in a variety of
applications across the globe Kaddour et al. (2023). While early LLMs focused
on English Joshi et al. (2020); Hu et al. (2020), recent models are more
multilingual. Nevertheless, LLMs do not provide equal utility to speakers of
non-English languages due to higher latency, increased costs, and reduced
performance Ahia et al. (2023); Asai et al. (2023); Held et al. (2023).

Figure 1: Language Confusion can occur at the word level, line level, or over
the entire output response.

To be useful, an LLM must understand intent and provide a response that is
appropriate in both form, e.g., correct grammar, style, tone, register, and
content, e.g., truthful, coherent, concise Grice (1975); Wilson and Sperber
(2012). While content-related issues such as hallucinations have attracted
substantial attention Ji et al. (2023); Bang et al. (2023), they are often
subtle and difficult to evaluate Gudibande et al. (2024); Hosking et al.
(2024), particularly in multilingual settings Guerreiro et al. (2023). Form-
related errors indicate a more obvious failure to fulfill a request and—in
extreme cases—may cause confusing or unintelligible responses.

We identify a surprising form limitation that drastically reduces LLMs’
utility for non-English languages: LLMs are often unable to consistently
generate text in the user’s desired language, or the appropriate language
given the context. We call this category of error “language confusion”.222We
use this term to indicate that this is an _erroneous_ behaviour rather than
_natural_ alternations between languages, i.e., code-switching Doğruöz et al.
(2021).

Take an Arabic prompt as an example: an LLM may inappropriately respond fully
in English (full-response confusion), produce some lines in the desired
language and some in another language (line-level confusion), or sporadically
insert single words or phrases in another language (word-level confusion).
Figure 1 shows example errors.

We investigate language confusion on the line and word level333For word-level
confusion, we focus on English words given the English-centricity of most LLMs
and the difficulty of reliably reproducing and measuring non-English word
switches. Albeit rarely, occurrences of non-English word-level confusions do
happen, as shown in Figure A1. in two practical settings: a) Monolingual
generation, where a user queries the LLM in a given language, _implicitly_
requesting an answer in the same language; and b) cross-lingual generation,
where a user _explicitly_ instructs a model to generate text in a different
language.

We create and release a language confusion benchmark covering 15 typologically
diverse languages, sourcing prompts from public English and multilingual
instruction datasets, and additionally creating new data with more complex
prompts. We evaluate a range of state-of-the-art LLMs including Llama, Command
R, Mistral, and OpenAI family models. We find that Llama Instruct and Mistral
LLMs exhibit severe language confusion in many languages. While Command R and
OpenAI models fare much better on monolingual generation, even the strongest
cannot consistently generate text in the correct language cross-lingually.

Our contributions are the following:

  1. 1.

We identify and describe the issue of language confusion in LLMs.

  2. 2.

We introduce a new benchmark and metrics to measure language confusion in
LLMs.

  3. 3.

We perform a systematic evaluation of various LLMs, investigating when
language confusion occurs in practice.

  4. 4.

We propose methods to mitigate language confusion in LLMs.

| Dataset name | Reference | Nature of data | $|L|$ | $|D|$ | Languages | $W$
---|---|---|---|---|---|---|---
Mono-lingual | Aya | Singh et al. (2024) | Human-generated | 100 | 500 | en, tr, ar, zh, pt | 9
Dolly | Singh et al. (2024) | MT post-edited | 100 | 500 | hi, ru, fr, ar, es | 10
Okapi | Lai et al. (2023) | Synthetic + MT | 100 | 1.2k | en, fr, it, de, zh, vi, | 13
ru, es, id, pt, ar, hi
Native prompts | Ours | Human-generated | 100 | 400 | es, fr, ja, ko | 19
Cross-lingual | Okapi | Lai et al. (2023) | Synthetic | 100 | 1.5k | $\mathcal{L}$ | 15
ShareGPT | https://sharegpt.com/ | Human-generated | 100 | 1.5k | $\mathcal{L}$ | 18
| Complex prompts | Ours | Human-generated | 99 | 1.5k | $\mathcal{L}$ | 159

Table 1: Data sources in the LCB for monolingual and cross-lingual generation.
$|D|$ is the total number of examples per data source and $|L|$ is the number
of examples per language. For the cross-lingual setting, the model is
instructed in English to generate in the target language $l\in\mathcal{L}$
where $\mathcal{L}=\\{\text{fr, de, es, pt, it, ja, ko, zh, ar, tr, hi, ru,
id, vi}\\}$. $W$ is the median length in words of the prompts in each dataset.

##  2 Language Confusion Benchmark

While some datasets to evaluate LLMs’ performance on _natural_ code-switched
data exist Khanuja et al. (2020); Winata et al. (2023), there are none
designed to assess language confusion in LLMs. We create the Language
Confusion Benchmark (LCB) by collecting a diverse set of prompts reflecting
realistic use cases across a typologically diverse set of languages. The
benchmark is easily extensible, cheap, and efficient to evaluate.

###  2.1 Generation settings

We measure language confusion in two settings: monolingual and cross-lingual
generation.

#### Monolingual generation

A speaker queries a model in language $l$ and expects a response in $l$. This
is the most common usage scenario as users often prefer to interact with
technology in their native language Kantar and IAMAI (2023).

#### Cross-lingual generation

A user instructs a model in language $l$ to fulfill a request in another
language $l^{\prime}$. In this challenging setting, the requested language
$l^{\prime}$ is different from the instruction language $l$. This setting is
relevant in applications where multilingual outputs are required, but
optimizing a prompt for each input language is inefficient in practice or when
a user requires a generation in a language they do not speak. We set the
instruction language to English.

###  2.2 Language Confusion Metrics

To detect language confusion, we rely on off-the-shelf language identification
(LID) tools. We employ fastText Joulin et al. (2016) as a strong alternative
to expensive LLM-based evaluation.

#### Line-level detection

We split a response into lines (by newline character) and check each line
against the user’s desired language with fastText.444We only apply fastText to
sequences of more than 4 words as its LID predictions are less precise for
shorter sequences.

#### Word-level detection

Off-the-shelf tools do not support word-level LID and LLMs only achieve 79–86
F1 detecting word-level code-switching Zhang et al. (2023), too low for use as
automatic evaluators. Observing that word-level confusion in top LLMs mainly
occurs with English, we opt for a heuristic evaluation restricted to a subset
of languages where it achieves high precision. We thus focus on non-Latin
script languages, where Latin script words are trivially detected. To avoid
natural code-switching false positives,555E.g., Japanese users may use English
acronyms: AI に関する記事を書きます。(“Write an article about AI.”) we check for English
words that do not typically occur in target language text.666We source words
from https://gist.github.com/WChargin/8927565, which is based on the Linux
dictionary usually stored in /usr/share/dict/words. To reduce false positives,
we exclude capitalized words from the dictionary (which are often proper nouns
or acronyms). We evaluate word-level confusion in Arabic (ar), Hindi (hi),
Japanese (ja), Korean (ko), Russian (ru), and Simplified Chinese (zh).

#### Binary evaluation

A response is only correct when entirely in the correct language, as even one
instance of language confusion can damage intelligibility and cause a jarring
user experience. We calculate binary metrics to indicate whether a response
contains any instance of a) a line in an incorrect language and b) an isolated
English word/phrase for languages using non-Latin scripts. These main metrics
in our evaluation are defined below.

Line-level pass rate (LPR): percentage of model responses that pass our line-
level language confusion detector without error. A response is “correct” if
all lines match the user’s desired language.

$\text{LPR}=\frac{|R\setminus E_{L}|}{|R|}$

where $R$ is the set of all responses and $E_{L}$ the set of responses that
contain line-level errors.777For completeness, we show results for “line-level
language accuracy” (the fraction of lines in the correct language across all
responses of an LLM) in Tables A1 and A2 at §A.1

Word-level pass rate (WPR): percentage of responses where all words are in the
desired language. We exclude responses with line-level errors as most line-
level errors would also be counted as word-level errors, making it difficult
to disentangle the two error types. Detecting word-level language confusion is
highly challenging so we only evaluate on target languages that use a non-
Latin script and look for erroneous English words—the most common type of
word-level language confusion.

$\text{WPR}=\frac{|(R\setminus E_{L})\setminus E_{W}|}{|R\setminus E_{L}|}$

where $R$ is the set of all responses, $E_{L}$ the set of responses with line-
level errors and $E_{W}$ the set of responses with word-level errors.

|  Prompt |  Translation |  Data source
---|---|---|---
Monolingual |  问：如何清洗和保养筷子？ |  Q: How to clean and maintain chopsticks? |  Aya
¿Cómo escapar de un helicóptero atrapado en el agua? |  How to escape from a helicopter stuck in water? |  Dolly
Erklären Sie, wie der Gini-Index berechnet wird. |  Explain how the Gini index is calculated. |  Okapi
日本で救急隊員を目指す人は、どのような教育やトレーニングを受ける必要がありますか。 |  What kind of education and training do people who want to become emergency workers in Japan need to undergo? |  Native prompts (Ours)
Cross-lingual |  Generate an essay in Korean of at least 500 words that argues in favor of regulating artificial intelligence. |  Okapi
Respond in French. You are a medical communications expert. Please provide a summary on how pharma companies are approaching diversity and inclusion, and health inequalities globally. Focus on the general approach and include information on clinical trials. |  Share- GPT
Based solely on the text below: 1. Extract the statistical techniques and machine learning algorithms analysts employ to uncover relationships and patterns within the data. 2. Generate 5 fill-in-the-blanks style questions 3. Summarize the text in 100 words […] Reply in Turkish. |  Complex prompts (Ours)
Table 2: An example prompt from each dataset used for monolingual and cross-
lingual generation. English translations are shown for convenience. For cross-
lingual generation, prompts are in English and have been amended with an
instruction to generate the output in another language. The complex prompt
example is truncated.

Language confusion pass rate (LCPR): harmonic mean888Similar to F1, this gives
higher importance to low values. of LPR and WPR:

$\text{LCPR}=2\times\frac{\text{LPR}\times\text{WPR}}{\text{LPR}+\text{WPR}}$

LCPR is elucidating in cases of severe issues producing output in the correct
language.999Imagine a model’s LPR for Arabic is a dismal 1% because only 1/100
responses are fully Arabic, the rest English. WPR is a deceptively high 100%:
as outputs are fully English, no English word appears in an Arabic line. The
model scores 2% LCPR, however, reflective of severe confusion.

###  2.3 Data sources

The monolingual and cross-lingual tasks respectively comprise 2600 and 4500
prompts in total, across 15 typologically diverse languages: English, French,
German, Spanish, Portuguese, Italian, Japanese, Korean, Chinese, Arabic,
Turkish, Hindi, Russian, Indonesian, and Vietnamese. Details are shown in
Table 1.

The prompts are sourced from the datasets below, focusing on human-annotated
or human-edited prompts. We filter each dataset to make it most useful for
evaluating language confusion.

#### Aya

250 original human-written prompts in 7 languages each from the aya-human-
annotated subset of the Aya Evaluation Suite Singh et al. (2024).

#### Dolly

200 machine-translated Dolly Conover et al. (2023) prompts post-edited by
fluent speakers for 6 languages from the Aya Evaluation Suite’s dolly-human-
edited subset Singh et al. (2024).

#### Okapi

We use the multilingual-alpaca-52k subset of the Okapi data, which contains
ChatGPT-generated translations of 52k English instructions from Alpaca Taori
et al. (2023) into 26 languages.

#### ShareGPT

We use prompts from the first turn of 90,000 mostly English user conversations
with ChatGPT, scraped via the ShareGPT API101010https://sharegpt.com/ before
it was shut down.111111https://huggingface.co/datasets/RyokoAI/ShareGPT52K

#### Native prompts (Ours)

For Japanese and Korean, under-represented in the above datasets, as well as
Spanish and French, we commission native annotators to collect our own prompts
(see §A.2).

#### Complex prompts (Ours)

As prompts from the above sources are relatively simple, we collect complex
English prompts written by human annotators.

###  2.4 Data Filtering and Processing

#### Suitability for LID

As LID tools underperform on short sequences and non-standard text, we
manually filter out: a) examples answerable with a single word/phrase; b)
multiple-choice questions and prompts asking for lists; c) prompts requiring
code generation, math equations, or data formats such as HTML. For datasets
where completions are available, we prioritize the prompts with the longest
completions (i.e. more than 4 words).

#### Western-centric responses

Many datasets created via translation contain questions about Western-centric
concepts (e.g., US National Parks, presidents or US-based brands) which can
cause false positives with our word-level detector. We manually filter out
such questions.

#### Prompt format

For cross-lingual generation, we semi-automatically amend prompts with an
instruction to generate in a target language (see §A.3 for details). Prompts
are used as-is for monolingual generation. Some examples are shown in Table 2.

| Monolingual
---|---
| avg | ar | de | en | es | fr | hi | id | it | ja | ko | pt | ru | tr | vi | zh
Llama 2 70B-I | 48.3 | 0.3 | 59.0 | 99.0 | 95.7 | 87.7 | 1.0 | 62.0 | 72.0 | 7.0 | 0.0 | 91.0 | 88.9 | 33.0 | 17.0 | 10.5
Llama 3 70B-I | 46.0 | 21.7 | 31.0 | 100.0 | 98.3 | 88.7 | 23.0 | 21.0 | 88.0 | 10.0 | 0.0 | 95.5 | 77.0 | 18.0 | 10.0 | 8.0
Mixtral 8x7B | 73.0 | 48.3 | 90.9 | 99.5 | 89.3 | 95.3 | 71.0 | 58.0 | 72.0 | 66.7 | 61.2 | 85.0 | 65.0 | 90.0 | 57.0 | 45.5
Mistral Large | 69.9 | 48.0 | 98.0 | 99.0 | 99.0 | 100.0 | 19.0 | 31.0 | 99.0 | 48.0 | 64.0 | 79.5 | 98.0 | 71.0 | 29.0 | 66.0
Command R | 98.6 | 100.0 | 98.0 | 99.5 | 95.7 | 99.3 | 100.0 | 92.0 | 99.0 | 100.0 | 100.0 | 98.5 | 100.0 | 99.0 | 99.0 | 98.5
Command R+ | 99.2 | 99.7 | 100.0 | 100.0 | 99.3 | 99.7 | 100.0 | 97.0 | 100.0 | 99.0 | 100.0 | 97.5 | 100.0 | 100.0 | 99.0 | 97.5
GPT-3.5 Turbo | 99.1 | 100.0 | 100.0 | 99.5 | 99.7 | 100.0 | 99.0 | 96.0 | 100.0 | 98.0 | 100.0 | 98.0 | 100.0 | 100.0 | 99.0 | 97.0
GPT-4 Turbo | 99.3 | 99.0 | 100.0 | 100.0 | 99.3 | 99.3 | 100.0 | 96.0 | 99.0 | 100.0 | 100.0 | 98.0 | 100.0 | 100.0 | 100.0 | 99.0
| Cross-lingual
| avg | ar | de | - | es | fr | hi | id | it | ja | ko | pt | ru | tr | vi | zh
Llama 2 70B-I | 38.4 | 12.4 | 52.3 | - | 77.3 | 71.1 | 21.2 | 46.0 | 66.5 | 16.2 | 4.8 | 75.9 | 38.3 | 24.0 | 20.4 | 11.1
Llama 3 70B-I | 30.3 | 31.1 | 34.7 | - | 61.1 | 53.1 | 46.4 | 25.4 | 36.4 | 1.4 | 0.8 | 54.4 | 38.4 | 17.4 | 18.7 | 4.3
Mixtral 8x7B | 69.0 | 59.1 | 76.4 | - | 79.1 | 79.0 | 39.2 | 72.8 | 85.0 | 57.9 | 56.9 | 79.4 | 72.4 | 76.0 | 75.8 | 57.5
Mistral Large | 58.2 | 36.1 | 74.5 | - | 68.5 | 71.9 | 58.5 | 59.2 | 65.8 | 44.5 | 41.1 | 64.5 | 63.3 | 65.9 | 54.8 | 46.5
Command R | 68.1 | 61.6 | 63.2 | - | 72.5 | 74.4 | 65.5 | 70.8 | 65.7 | 65.3 | 69.2 | 67.2 | 69.4 | 67.7 | 65.7 | 75.0
Command R+ | 91.2 | 93.4 | 91.6 | - | 91.7 | 91.5 | 90.2 | 85.9 | 93.8 | 93.8 | 91.1 | 88.5 | 93.0 | 92.0 | 91.1 | 89.5
GPT-3.5 Turbo | 89.8 | 90.8 | 90.2 | - | 93.3 | 87.8 | 92.0 | 84.5 | 91.3 | 88.3 | 90.3 | 89.9 | 91.8 | 89.2 | 91.8 | 86.4
GPT-4 Turbo | 90.3 | 88.9 | 93.0 | - | 93.1 | 90.7 | 91.0 | 87.3 | 91.8 | 87.7 | 89.7 | 91.0 | 90.0 | 91.4 | 90.0 | 87.9

Table 3: Line-level pass rate (LPR) on monolingual and cross-lingual generation, by language. | Monolingual | Cross-lingual
---|---|---
Llama 2 70B-I | 97.9 | 84.2
Llama 3 70B-I | 93.0 | 94.4
Mixtral 8x7B | 73.7 | 68.2
Mistral Large | 98.4 | 93.8
Command R | 96.3 | 94.0
Command R+ | 99.4 | 95.1
GPT-3.5 Turbo | 99.8 | 98.7
GPT-4 Turbo | 99.7 | 96.6
Table 4: Average word-level pass rate (WPR). For a detailed by-language
breakdown, see Table A3.

##  3 Experiments

#### Models

We evaluate the following LLMs covering various scales and model families:
Llama 2 70B Instruct Touvron et al. (2023), Llama 3 70B
Instruct,121212https://ai.meta.com/blog/meta-llama-3/ Command R (35B
parameters),131313https://cohere.com/blog/command-r Command R+ (104B
parameters),141414https://cohere.com/blog/command-r-plus-microsoft-azure
Mixtral 8x7B Jiang et al. (2024), Mistral
Large,151515https://mistral.ai/news/mistral-large/ GPT-3.5 Turbo
(gpt-3.5-turbo-012524; Brown et al., 2020) and GPT-4 Turbo
(gpt-4-turbo-040924; Achiam et al., 2023). For Llama and Command models, we
also evaluate base versions (see §4.5). We generate at most 100 tokens per
prompt using nucleus sampling with $p=0.75$ and $T=0.3$.

LPR, WPR, and LCPR results are in Tables 3, 4, and A4, respectively.161616We
find results show little variance across runs (see §A.4). We show language
confusion examples for different models in Table A5.

#### Monolingual generation

Command and GPT models perform well on average on the line level (LPR in
$[98.6,99.3]$), but Llama and Mistral models struggle to consistently generate
text in the correct language (LPR in $[48.3,73.0]$). Mistral Large is better
on some European languages while Llama models exhibit language confusion even
for high-resource languages such as German. Most models have WPR within the
same range, though Mixtral 8x7B is considerably worse. GPT-4 Turbo is
strongest on LCPR, with GPT-3.5 Turbo and Command R+ tied for second. Llama
models’ preference for English responses leads to very low LCPR.

#### Cross-lingual generation

In the challenging cross-lingual setting, the best models have LPRs in the low
90s. Command and OpenAI models are best, with Command R+ best overall. Llama
models perform poorly: both scoring in the 30s due to a tendency to respond in
English. Mistral Large is worse than Mixtral, even in European languages. For
non-Latin script languages, Command R+ and OpenAI models are again best. LCPR
shows a similar trend as in the monolingual task, albeit with lower scores:
GPT-3.5 Turbo performs best, with Command R+ and GPT-4 Turbo following.
Command R is substantially worse than R+ cross-lingually, unlike
monolingually.

##  4 Analyses

###  4.1 Impact of dataset

In creating the language confusion test sets, we aimed to include prompts
covering various use cases and domains. We show differences of LPR by dataset
in Tables 5 and A7 for cross-lingual and monolingual generation, with WPR and
LCPR in Tables A8 and A9 in §A.5. Differences between datasets are small for
monolingual generation. Cross-lingually, the difference is more pronounced:
models perform fairly well on Okapi and ShareGPT, but are much worse on our
Complex prompts, indicating their more challenging nature.

| avg | Okapi | ShareGPT | Complex (Ours)
---|---|---|---|---
Llama 2 70B-I | 38.4 | 43.4 | 46.0 | 25.7
Llama 3 70B-I | 30.3 | 35.0 | 39.5 | 16.3
Mixtral 8x7B | 69.0 | 77.4 | 79.8 | 49.9
Mistral Large | 58.2 | 68.0 | 56.2 | 50.5
Command R | 68.1 | 75.6 | 89.7 | 39.0
Command R+ | 91.2 | 96.1 | 98.7 | 78.9
GPT-3.5 Turbo | 89.8 | 97.7 | 96.8 | 75.0
GPT-4 Turbo | 90.3 | 96.6 | 96.4 | 77.7

Table 5: Line-level pass rate (LPR) by dataset on cross-lingual generation.
Figure 2: A model is vulnerable to world-level language confusion when the
number of tokens in the sampling nucleus is high, and the distribution is
flat. Metrics: Shannon entropy; in brackets: # of tokens in nucleus.

###  4.2 Impact of prompt length

We analyze whether the difficulty of our Complex prompts is caused by their
much higher length (see Table 1) by grouping the prompts into three length
buckets: short, medium and long, each with one third of the prompts. Table A10
shows the LPR of several models over the different lengths. We find no clear
pattern, suggesting that higher confusion is caused by prompt complexity
rather than length.

###  4.3 Impact of instruction position

Cross-lingual prompts include an instruction with the desired output language
at the beginning, at the end, or integrated in the prompt (e.g., “Write an
essay in Korean […]”). Table A11 shows that across all models, line-level
confusion is low for isolated instructions, with similar performance whether
they are at the start or the end. Integrated instructions cause more
confusion: Command R has only 69% LPR on this type of prompt, versus $\sim$85%
for isolated types. This difficulty can be greatly reduced with one-shot
prompting (80.6% LPR; see §6.3).171717Even though our demonstrations in few-
shot prompting do not have integrated instructions.

###  4.4 Impact of quantization

Quantization maps higher-precision LLM weights and activations to lower
precision, reducing storage and inference costs, potentially at the cost of
performance. We compare FP16 with W8, W8A8, and W4 variants of Command R+ on
monolingual generation. Negative effects appear at W4, shown in Table A12.
Details of quantization are in §A.7.

###  4.5 Impact of instruction tuning

We compare the base with instruction-tuned variants of Llama and Command
models on monolingual generation in Table 6. While instruction-tuned Command R
models exhibit less language confusion than their base versions, instruction-
tuned Llama models are much more confused, indicating English-centric
instruction tuning, which is confirmed by our mitigation experiments (see
§6.3).

| avg | ar | hi | ja | ko | vi | zh
---|---|---|---|---|---|---|---
Llama 2 70B | 98.5 | 99.6 | 100.0 | 100.0 | 100.0 | 98.0 | 93.2
Llama 2 70B-I | 6.0 | 0.3 | 1.0 | 7.0 | 0.0 | 17.0 | 10.5
Llama 3 70B | 94.7 | 96.7 | 97.9 | 87.9 | 98.8 | 97.0 | 90.0
Llama 3 70B-I | 12.1 | 21.7 | 23.0 | 10.0 | 0.0 | 10.0 | 8.0
Command R base | 85.9 | 94.9 | 81.0 | 93.9 | 94.2 | 83.0 | 68.1
Command R | 99.6 | 100.0 | 100.0 | 100.0 | 100.0 | 99.0 | 98.5
Command R+ base | 78.4 | 92.8 | 67.0 | 90.5 | 93.5 | 65.7 | 60.9
Command R+ | 99.2 | 99.7 | 100.0 | 99.0 | 100.0 | 99.0 | 97.5

Table 6: Line-level pass rate of base vs instruction-tuned LLMs on monolingual
generation for a subset of languages. Full results in Table A15, §A.8.

##  5 When does language confusion occur?

We study a sample of prompts to better understand where language confusion
occurs. Intuitively, if a token in an undesired language is assigned
sufficient probability, it may be sampled. We observe that language confusion
typically occurs when the distribution over next tokens is flat and nucleus is
large (see §A.9 for background).

We generate responses to 15 Chinese prompts from Okapi with Command
R.181818$k=0,p=0.75,t=0.7$ We examine outputs to identify instances of English
language confusion, finding it in 5 of 15 outputs. For each, we find the first
position where an English token was elicited, which we call the confusion
point (CP).191919When intentional, such points are referred to in the code-
switching literature as “switch points”. We coin a new term here to indicate
that this switching is erroneous. There are 9 such CPs.202020There are 3
examples of line-level confusion. We label the initial switching point from
Chinese to English as the CP.

We calculate Shannon entropy (Shannon, 1948) and nucleus size at each sampling
point. We show an example output of Command R in Figure 2, indicating Shannon
entropy and the number of tokens in the nucleus at select sampling points, and
the next possible tokens with normalized probabilities at the confusion point.
In the example, “called” was third most likely but occurred with sufficient
probability to be sampled (0.221). We generate 100 tokens per prompt, so there
are 1500 points: 9 of which are CPs. We refer to the others as $\neg$CPs.

Figure 3: Effect of Temperature ($T$) in Nucleus Sampling. Tokens in the
nucleus at $p=0.75$ are bold. Middle: Effect of $T$ on the softmax
probabilities (Equation 1). Right: Effect of $T$ on the probabilities of
tokens in the nucleus right before sampling (Equation A.9). As $T$ increases,
the token 狐狸 has less chance to be sampled.

We average nucleus size and entropy over examples containing no instances of
language confusion (10 samples, 1000 $\neg$CPs), at least one instance (5
examples, 9 CPs, 491 $\neg$CPs), and overall (15 samples, 9 CPs, 1491
$\neg$CPs). Results are in Table 7.212121E.g., For outputs exhibiting language
confusion, the average nucleus size @CP is: $\frac{\text{sum of nucleus sizes
at all CPs}}{9}=\frac{32}{9}=3.56$. Outputs with and without language
confusion show similar average nucleus size and entropy (1.64 vs. 1.61, and
0.353 vs. 0.365). At CPs, however, average nucleus size and entropy is
considerably higher: 3.56 nucleus size and 1.228 entropy vs 1.61/0.356 at
$\neg$CPs, indicating that confusion tends to occur when nucleus size and
entropy are high.

| Avg. Nucleus Size | Avg. Entropy
---|---|---
| Overall | @CP |  $\neg$@CP | Overall | @CP |  $\neg$@CP
Has CP | 1.64 | 3.56 | 1.61 | 0.353 | 1.228 | 0.337
No CP | 1.61 | - | 1.61 | 0.365 | - | 0.365
All | 1.62 | 3.56 | 1.61 | 0.361 | 1.228 | 0.356

Table 7: Avg. nucleus size, entropy at confusion points (sampling points where
language switch did [@CP] or did not [$\neg$@CP] occur) for 15 Chinese
responses. Responses are split into those which had at least one CP (“Has CP”)
or zero CPs (“No CP”).

##  6 Mitigating Language Confusion

Based on the insights from our analyses in §4.5 and §5, we propose inference-
time and training-time mitigations for language confusion.

###  6.1 Reducing temperature and nucleus size

Modifying sampling hyper-parameters affects which tokens are chosen at
inference time. Section A.9 explains nucleus sampling with temperature, and
Figure 3 has a toy example of manipulating the hyperparameter $T$. Framing
language confusion as an undesired side-effect of sampling, it is intuitive
that we might control it by sharpening the distribution over the next tokens
at each timestep.

We try to reduce the chance of having a wrong-language token sampled by
manipulating temperature and nucleus size. Table 8 shows the results on
monolingual WPR for Command R. Higher $T$ encourages language confusion: $T=1$
shows an average WPR of only 83.5%, and as low as 72.0% and 69.5% for Japanese
and Chinese. Increasing $p$, resulting in a smaller nucleus, has a smaller
effect.

| avg | ar | hi | ja | ko | ru | zh
---|---|---|---|---|---|---|---
T=0.0 | 97.2 | 97.6 | 100.0 | 96.9 | 97.0 | 96.0 | 95.9
T=0.3 | 96.3 | 99.3 | 99.0 | 93.9 | 97.0 | 96.0 | 92.3
T=0.5 | 96.4 | 97.9 | 99.0 | 94.9 | 99.0 | 95.0 | 92.3
T=0.7 | 94.2 | 98.0 | 98.0 | 91.8 | 93.9 | 93.0 | 90.3
T=1.0 | 86.5 | 95.9 | 93.8 | 74.5 | 92.8 | 87.5 | 74.7
p=0.1 | 97.4 | 98.3 | 100.0 | 98.0 | 97.0 | 95.0 | 95.9
p=0.3 | 97.3 | 98.0 | 100.0 | 99.0 | 97.0 | 94.0 | 95.9
p=0.5 | 97.6 | 98.0 | 100.0 | 96.9 | 96.0 | 98.0 | 96.9
p=0.75 | 96.3 | 99.3 | 99.0 | 93.9 | 97.0 | 96.0 | 92.3
Table 8: Effect of varying temperature ($T$) or nucleus size ($p$) on
monolingual word-level language confusion (WPR) of Command R. Default values
are $p=0.75$ and $T=0.3$. Best score. Worst score.

###  6.2 Few-shot prompting

Non-instruction-tuned LLMs often do not answer instructions as expected. When
prompted with an instruction in a non-English language, for instance, Command
R Base often translates it instead of answering. We thus employ few-shot
prompting to guide Command R Base towards the correct behavior. We cherry-pick
5 prompt/answer pairs in English and translate them with Google
Translate.222222Few-shot demonstrations are in different languages than the
current task’s target language. The goal is to help a model understand the
task, not find the language to generate into.

For cross-lingual generation, we use a similar format to the test sets. Figure
A2 shows the template used to prompt the base model. For comparison, we also
apply one-shot prompting to Command R where the example is presented as turns
in a conversation. Table 9 shows the results.

Few-shot prompting greatly reduces Command R Base’s language confusion and
almost completely eliminates the problem in the monolingual setting. While
one-shot prompting with Command R is detrimental monolingually—indicating a
difficulty of dealing with demonstrations in other languages—it enables the
model to better follow the cross-lingual instructions.

| Monolingual | Cross-lingual
---|---|---
| LPR | WPR | LPR | WPR
Command R Base | 86.2 | 98.7 | 1.1 | 100.0
\+ Q/A template (0-shot) | 85.3 | 99.7 | 20.9 | 97.0
\+ 1-shot | 94.1 | 100.0 | 90.7 | 98.6
\+ 5-shot | 99.0 | 100.0 | 95.0 | 99.7
\+ English SFT | 77.8 | 96.2 | 78.3 | 91.7
\+ English pref. tuning | 74.3 | 90.9 | 85.7 | 87.4
\+ Multilingual SFT | 98.3 | 95.5 | 78.2 | 90.0
\+ Multi. pref. tuning | 98.9 | 93.4 | 89.4 | 86.9
_Command R_ | 98.6 | 96.3 | 68.1 | 94.0
\+ 1-shot | 68.3 | 92.7 | 82.9 | 92.3

Table 9: Effect of few-shot prompting and instruction tuning on language
confusion.

###  6.3 Multilingual instruction tuning

We additionally study the impact of English-centric and multilingual
instruction tuning by fine-tuning Command R Base with several techniques:

#### English-only tuning

We fine-tune the base model with English-only publicly available instruction
data (SFT; Touvron et al., 2023), then apply English-only preference tuning to
this model.

#### Multilingual tuning

We extend the English data with multilingual data Aryabumi et al. (2024). Due
to the scarcity of multilingual data, our SFT data mixture is 90% English. For
preference tuning, we use 50% multilingual data. Results are in Table 9.

English-only tuning (both SFT and pref. tuning) exacerbates language confusion
monolingually: likely the reason for high language confusion of Llama-Instruct
models (see §4.5). English-only preference tuning has a strong negative impact
on word-level confusion. However, SFT with just 10% multilingual data is
enough to almost eliminate the problem of line-level confusion monolingually.
Cross-lingually, multilingual tuning does not give better line-level
performance than English-only tuning. This may be because cross-lingual
datasets only have English prompts and it is more important for the model to
learn to follow English instructions (e.g., “Reply in French”).

##  7 Discussion

We have identified the issue of language confusion and demonstrated which
model training and hyper-parameter choices impact it. However, there are
several aspects that require further discussion.

#### Behavior of base models

As observed in §4.5, language confusion in base models is not correlated with
their downstream performance. Stronger base models such as Command R+ and
Llama 3 70B are more confused than Command R and Llama 2 70B respectively.
Given the occurrence of translations and token in other languages in pre-
training Blevins and Zettlemoyer (2022), we expect base models to exhibit some
degree of language switching while instruction tuning then reinforces the
desired behavior.

#### English-centricity

Despite LLMs in general exhibiting impressive multilingual generative
capabilities, the fact that they are most likely to switch to English—both on
the sentence and word level—is another example of their English-centric nature
Hu et al. (2020); Zhao et al. (2024). Our experiments in §6.3 further
highlight the negative impact of overly English-centric instruction tuning,
which is also illustrated by Llama Instruct models’ high language confusion.

#### Preference Tuning

In Table 9, we observe that WPR decreases after preference tuning. Citing
similar observations by Yuan et al. (2024),232323Also,
https://wandb.ai/eric_anthony_mitchell/dpo-
demos/runs/og8q3euz?nw=nwusereric_anthony_mitchell Yan et al. (2024) observe a
decrease in the likelihood of both preferred and unpreferred data points as
DPO (Rafailov et al., 2023) training progresses, and propose a theoretical
explanation. Xu et al. (2024) remark that “DPO is prone to generating a biased
policy that favors out-of-distribution responses, leading to unpredictable
behaviors.” It is plausible that if preference learning decreases token
likelihoods for examples seen during its training (e.g. common English words),
then the relative likelihoods of unseen/rare tokens increases, explaining the
large decrease in WPR we see for +English SFT +English pref. tuning. The
hypothesis that preference learning encourages unfavorable behaviors such as
language confusion is worthy of further exploration.

#### Other factors

There are other factors that may affect language confusion. We suspect that
word-level confusion is related to under-training. Certain non-English tokens,
particularly in low-density regions that are rarely encountered during pre-
training, may be under-trained and lack calibration. This can then lead to
English tokens being sampled due to the absence of in-language tokens with
higher likelihood as seen in §5. Future work might investigate this
hypothesis.

##  8 Related Work

#### Code-switching

There has been much research on _natural_ alternations between languages,
i.e., code-switching Doğruöz et al. (2021) in natural language processing
(NLP). Prior work focused on evaluating capabilities on standard NLP tasks
using code-switching data Khanuja et al. (2020); Winata et al. (2023)
including sentiment analysis, machine translation, summarization and word-
level language identification. These tasks typically employ data created by
humans where word-level code-switching occurs between English and another
language such as Hindi, Spanish, or Arabic. Current models still struggle to
generate and understand code-switched text in some languages Yong et al.
(2023); Zhang et al. (2023). In contrast, we investigate _unnatural_ and
_erroneous_ code-switching—or language confusion—in the LLM’s generations.

#### Language confusion

Prior work has observed ‘source language hallucinations’ in zero-shot cross-
lingual transfer Vu et al. (2022); Li and Murray (2023); Pfeiffer et al.
(2023); Chirkova and Nikoulina (2024) when models are fine-tuned on English
data and applied to generate text in another language. The problem of line-
level language confusion is known in the machine translation field as ‘off-
target translation’ Chen et al. (2023); Sennrich et al. (2024). It typically
occurs on English-centric multilingual models, when used in a zero-shot manner
(to translate between two languages unseen at training). For LLMs, there is no
source language per se. A few studies Kew et al. (2023); Chen et al. (2024)
have provided evidence of LLMs generating in an incorrect language on the
response level. To the best of our knowledge, we are the first to show results
on the sentence and word level and the first to systematically study language
confusion in LLMs.

##  9 Conclusion

We have introduced the Language Confusion Benchmark to evaluate language
confusion in LLMs. We have shown that some LLMs exhibit severe language
confusion in many languages and even the strongest LLMs do not achieve perfect
performance cross-lingually. We observed that base and English-centric
instruct models are particularly susceptible to language confusion, which is
exacerbated by complex prompts. Finally, we proposed measures to mitigate
language confusion at inference (reducing temperature and nucleus size, few-
shot prompting) and at training (multilingual instruction tuning). Our
benchmark is efficient to evaluate and easy to extend and can help ensure that
models achieve equal utility across languages.

## Acknowledgements

We thank Sander Land, Florian Strub, Mo Gheshlaghi Azar, and Olivier Pietquin
for early discussions on language confusion. We are grateful to Sander Land
for help with an initial version of a script to analyze nucleus sampling
generations. We thank our internal annotators and the team at Cohere.

## Limitations

As this is to our knowledge, the first study on language confusion in LLMs,
there are several natural research directions that we were not able to cover:

#### Conversations

We focus on single-turn inputs, and do not consider the impact of multiple
turns of a conversation or users employing different languages across
different turns.

#### Code-switched input

We focus on the setting where a user’s prompt is in a single language (the
target language in the monolingual setting and English in the cross-lingual
setting). We do not consider inputs that are naturally code-switched.

#### Inputs with cross-lingual context

Content may not always be available in a user’s language, which is relevant
for applications such as cross-lingual summarization or cross-lingual QA.

#### Language varieties

We evaluate generation into standardized languages. Future work may expand to
language varieties and dialects, styles, and registers.

Our metrics are applied to model outputs of at most 100 tokens. But what is a
token depends on the model’s tokenizer. Models with more aggressive
tokenization could be advantaged with regard to the binary metrics (LPR and
WPR). Likewise, because we allow models to stop their generation early (by
producing an end-of-sequence token), models which are less verbose could have
an advantage.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Ahia et al. (2023) Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023.  Do all languages cost the same? tokenization in the era of commercial language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 9904–9923, Singapore. Association for Computational Linguistics.
  * Aryabumi et al. (2024) Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Kelly Marchisio, Sebastian Ruder, et al. 2024.  Aya 23: Open weight releases to further multilingual progress.  _arXiv preprint arXiv:2405.15032_.
  * Asai et al. (2023) Akari Asai, Sneha Kudugunta, Xinyan Velocity Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannaneh Hajishirzi. 2023.  Buffet: Benchmarking large language models for few-shot cross-lingual transfer.  _arXiv preprint arXiv:2305.14857_.
  * Bang et al. (2023) Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023.  A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity.  In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 675–718, Nusa Dua, Bali. Association for Computational Linguistics.
  * Blevins and Zettlemoyer (2022) Terra Blevins and Luke Zettlemoyer. 2022.  Language contamination helps explains the cross-lingual capabilities of English pretrained models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 3563–3574, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chen et al. (2023) Liang Chen, Shuming Ma, Dongdong Zhang, Furu Wei, and Baobao Chang. 2023.  On the off-target problem of zero-shot multilingual neural machine translation.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 9542–9558, Toronto, Canada. Association for Computational Linguistics.
  * Chen et al. (2024) Pinzhen Chen, Shaoxiong Ji, Nikolay Bogoychev, Andrey Kutuzov, Barry Haddow, and Kenneth Heafield. 2024.  Monolingual or multilingual instruction tuning: Which makes a better alpaca.  In _Findings of the Association for Computational Linguistics: EACL 2024_ , pages 1347–1356, St. Julian’s, Malta. Association for Computational Linguistics.
  * Chirkova and Nikoulina (2024) Nadezhda Chirkova and Vassilina Nikoulina. 2024.  Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks.  In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_ , pages 7222–7238, Mexico City, Mexico. Association for Computational Linguistics.
  * Conover et al. (2023) Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023.  Free dolly: Introducing the world’s first truly open instruction-tuned llm.  _Company Blog of Databricks_.
  * Doğruöz et al. (2021) A. Seza Doğruöz, Sunayana Sitaram, Barbara E. Bullock, and Almeida Jacqueline Toribio. 2021.  A survey of code-switching: Linguistic and social perspectives for language technologies.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1654–1666, Online. Association for Computational Linguistics.
  * Frantar et al. (2022) Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022.  GPTQ: Accurate post-training compression for generative pretrained transformers.  _arXiv preprint arXiv:2210.17323_.
  * Grice (1975) Herbert P Grice. 1975.  Logic and conversation.  In _Speech acts_ , pages 41–58. Brill.
  * Gudibande et al. (2024) Arnav Gudibande, Eric Wallace, Charlie Victor Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2024.  The false promise of imitating proprietary language models.  In _Proceedings of ICLR 2024_.
  * Guerreiro et al. (2023) Nuno M Guerreiro, Duarte M Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André FT Martins. 2023.  Hallucinations in large multilingual translation models.  _Transactions of the Association for Computational Linguistics_ , 11:1500–1517.
  * Held et al. (2023) William Held, Camille Harris, Michael Best, and Diyi Yang. 2023.  A material lens on coloniality in nlp.  _arXiv preprint arXiv:2311.08391_.
  * Holtzman et al. (2019) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019.  The curious case of neural text degeneration.  In _International Conference on Learning Representations_.
  * Hosking et al. (2024) Tom Hosking, Phil Blunsom, and Max Bartolo. 2024.  Human feedback is not gold standard.  In _Proceedings of ICLR 2024_.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In _International Conference on Machine Learning_ , pages 4411–4421. PMLR.
  * Ji et al. (2023) Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. 2023.  Towards mitigating llm hallucination via self reflection.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 1827–1843.
  * Jiang et al. (2024) Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024.  Mixtral of experts.  _arXiv preprint arXiv:2401.04088_.
  * Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020.  The state and fate of linguistic diversity and inclusion in the NLP world.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6282–6293, Online. Association for Computational Linguistics.
  * Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016.  Bag of tricks for efficient text classification.  _arXiv preprint arXiv:1607.01759_.
  * Kaddour et al. (2023) Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023.  Challenges and applications of large language models.  _arXiv preprint arXiv:2307.10169_.
  * Kantar and IAMAI (2023) Kantar and IAMAI. 2023.  Internet in India 2023.  Technical report, Internet And Mobile Association of India.
  * Kew et al. (2023) Tannon Kew, Florian Schottmann, and Rico Sennrich. 2023.  Turning english-centric llms into polyglots: How much multilinguality is needed?  _arXiv preprint arXiv:2312.12683_.
  * Khanuja et al. (2020) Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, and Monojit Choudhury. 2020.  GLUECoS: An evaluation benchmark for code-switched NLP.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3575–3585, Online. Association for Computational Linguistics.
  * Lai et al. (2023) Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. 2023.  Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 318–327, Singapore. Association for Computational Linguistics.
  * Li and Murray (2023) Tianjian Li and Kenton Murray. 2023.  Why does zero-shot cross-lingual generation fail? an explanation and a solution.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 12461–12476, Toronto, Canada. Association for Computational Linguistics.
  * Pfeiffer et al. (2023) Jonas Pfeiffer, Francesco Piccinno, Massimo Nicosia, Xinyi Wang, Machel Reid, and Sebastian Ruder. 2023.  mmT5: Modular multilingual pre-training solves source language hallucinations.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 1978–2008, Singapore. Association for Computational Linguistics.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023.  Direct preference optimization: Your language model is secretly a reward model.  In _Advances in Neural Information Processing Systems_ , volume 36, pages 53728–53741. Curran Associates, Inc.
  * Sennrich et al. (2024) Rico Sennrich, Jannis Vamvas, and Alireza Mohammadshahi. 2024.  Mitigating hallucinations and off-target machine translation with source-contrastive and language-contrastive decoding.  In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 21–33, St. Julian’s, Malta. Association for Computational Linguistics.
  * Shannon (1948) Claude Elwood Shannon. 1948.  A mathematical theory of communication.  _The Bell system technical journal_ , 27(3):379–423.
  * Singh et al. (2024) Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, et al. 2024.  Aya dataset: An open-access collection for multilingual instruction tuning.  _arXiv preprint arXiv:2402.06619_.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023.  Alpaca: A strong, replicable instruction-following model.  _Stanford Center for Research on Foundation Models_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Vu et al. (2022) Tu Vu, Aditya Barua, Brian Lester, Daniel Cer, Mohit Iyyer, and Noah Constant. 2022.  Overcoming catastrophic forgetting in zero-shot cross-lingual generation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9279–9300, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Wilson and Sperber (2012) Deirdre Wilson and Dan Sperber. 2012.  Linguistic form and relevance.  _Wilson & Sperber (Eds.), Meaning and Relevance_, pages 149–168.
  * Winata et al. (2023) Genta Winata, Alham Fikri Aji, Zheng Xin Yong, and Thamar Solorio. 2023.  The decades progress on code-switching research in NLP: A systematic survey on trends and challenges.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 2936–2978, Toronto, Canada. Association for Computational Linguistics.
  * Xu et al. (2024) Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. 2024.  Is dpo superior to ppo for llm alignment? a comprehensive study.  _Preprint_ , arXiv:2404.10719.
  * Yan et al. (2024) Yuzi Yan, Yibo Miao, Jialian Li, Yipin Zhang, Jian Xie, Zhijie Deng, and Dong Yan. 2024.  3d-properties: Identifying challenges in dpo and charting a path forward.  _Preprint_ , arXiv:2406.07327.
  * Yong et al. (2023) Zheng Xin Yong, Ruochen Zhang, Jessica Forde, Skyler Wang, Arjun Subramonian, Holy Lovenia, Samuel Cahyawijaya, Genta Winata, Lintang Sutawika, Jan Christian Blaise Cruz, Yin Lin Tan, Long Phan, Long Phan, Rowena Garcia, Thamar Solorio, and Alham Aji. 2023.  Prompting multilingual large language models to generate code-mixed texts: The case of south East Asian languages.  In _Proceedings of the 6th Workshop on Computational Approaches to Linguistic Code-Switching_ , pages 43–63, Singapore. Association for Computational Linguistics.
  * Yuan et al. (2024) Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. 2024.  Advancing llm reasoning generalists with preference trees.  _Preprint_ , arXiv:2404.02078.
  * Zhang et al. (2023) Ruochen Zhang, Samuel Cahyawijaya, Jan Christian Blaise Cruz, Genta Winata, and Alham Aji. 2023.  Multilingual large language models are not (yet) code-switchers.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 12567–12582, Singapore. Association for Computational Linguistics.
  * Zhao et al. (2024) Jun Zhao, Zhihao Zhang, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024.  Llama beyond english: An empirical study on language capability transfer.  _arXiv preprint arXiv:2401.01055_.

##  Appendix A Appendix

###  A.1 Extended Results

Prompt: que partes tiene una noticia? Completion: Una noticia típica suele
tener las siguientes partes: … Prompt: y los epigrafes? Completion: … Los
epígraf瓦解 también se pueden utilizar para resaltar citas importantes,
proporcionar transiciones entre temas o simplemente dividir el texto en
secciones más manejables y digeribles. Figure A1: Example of non-English word-
level language confusion produced by an LLM.

| Monolingual
---|---
| avg | ar | de | en | es | fr | hi | id | it | ja | ko | pt | ru | tr | vi | zh
Llama 2 70B-I | 49.8 | 3.2 | 60.3 | 99.7 | 96.7 | 89.8 | 2.3 | 65.6 | 73.0 | 7.3 | 0.7 | 92.5 | 91.2 | 35.5 | 17.7 | 11.7
Llama 3 70B-I | 47.3 | 27.7 | 31.0 | 100.0 | 98.8 | 89.1 | 25.4 | 21.5 | 88.9 | 11.2 | 2.3 | 97.8 | 77.0 | 19.6 | 10.4 | 9.4
Mixtral 8x7B | 74.2 | 49.5 | 90.9 | 99.8 | 90.0 | 95.3 | 72.2 | 60.2 | 72.0 | 70.6 | 62.1 | 86.2 | 65.7 | 90.0 | 59.4 | 49.2
Mistral Large | 70.3 | 48.7 | 98.0 | 99.6 | 99.5 | 100.0 | 19.5 | 31.0 | 99.0 | 48.0 | 64.3 | 80.9 | 98.7 | 71.5 | 29.0 | 67.0
Command R | 99.5 | 100.0 | 99.3 | 99.9 | 98.7 | 99.9 | 100.0 | 96.8 | 99.7 | 100.0 | 100.0 | 99.7 | 100.0 | 99.0 | 99.9 | 99.6
Command R+ | 99.8 | 99.9 | 100.0 | 100.0 | 99.8 | 99.9 | 100.0 | 99.0 | 100.0 | 99.8 | 100.0 | 99.4 | 100.0 | 100.0 | 99.5 | 99.1
GPT-3.5 Turbo | 99.5 | 100.0 | 100.0 | 99.8 | 99.8 | 100.0 | 99.5 | 98.3 | 100.0 | 98.7 | 100.0 | 99.5 | 100.0 | 100.0 | 99.0 | 98.3
GPT-4 Turbo | 99.7 | 99.2 | 100.0 | 100.0 | 99.8 | 99.7 | 100.0 | 98.0 | 99.7 | 100.0 | 100.0 | 99.4 | 100.0 | 100.0 | 100.0 | 99.7

Table A1: Line-level language ID accuracy of different LLMs on monolingual
generation. Line accuracy is calculated by $\frac{\text{\\# lines responded in
correct language}}{\text{Total \\# of lines generated}}$

| Cross-lingual
---|---
| avg | ar | de | es | fr | hi | id | it | ja | ko | pt | ru | tr | vi | zh
Llama 2 70B-I | 46.0 | 23.5 | 58.3 | 80.5 | 74.6 | 31.4 | 51.1 | 73.4 | 26.9 | 15.9 | 79.9 | 47.0 | 28.1 | 30.0 | 23.3
Llama 3 70B-I | 53.4 | 60.3 | 65.8 | 77.4 | 73.8 | 69.7 | 58.2 | 65.5 | 6.2 | 2.8 | 75.8 | 65.2 | 55.2 | 61.4 | 10.4
Mixtral 8x7B | 72.7 | 61.5 | 79.7 | 82.5 | 82.5 | 41.7 | 78.9 | 85.9 | 64.1 | 61.3 | 83.0 | 75.2 | 78.1 | 79.4 | 64.6
Mistral Large | 65.7 | 45.4 | 79.8 | 74.9 | 76.7 | 63.3 | 68.6 | 73.5 | 58.2 | 51.9 | 72.4 | 68.7 | 69.5 | 61.4 | 56.1
Command R | 75.7 | 67.5 | 75.4 | 80.0 | 81.7 | 69.7 | 78.9 | 74.1 | 75.4 | 76.3 | 73.2 | 76.0 | 76.3 | 72.3 | 83.1
Command R+ | 95.3 | 95.9 | 95.5 | 95.8 | 95.3 | 94.0 | 93.2 | 95.9 | 96.4 | 95.4 | 94.5 | 96.1 | 94.9 | 95.8 | 95.0
GPT-3.5 Turbo | 91.5 | 91.5 | 91.5 | 94.9 | 90.9 | 92.2 | 88.0 | 92.7 | 89.7 | 91.3 | 93.9 | 92.6 | 90.5 | 92.5 | 89.4
GPT-4 Turbo | 92.4 | 91.0 | 94.0 | 94.7 | 93.1 | 92.6 | 92.0 | 93.2 | 90.5 | 91.5 | 94.4 | 92.0 | 92.4 | 91.5 | 90.3

Table A2: Line-level language ID accuracy of different LLMs on cross-lingual
generation. Line accuracy is calculated by $\frac{\text{\\# lines responded in
correct language}}{\text{Total \\# of lines generated}}$

| Monolingual | Cross-lingual
---|---|---
| Avg | ar | hi | ja | ko | ru | zh | Avg | ar | hi | ja | ko | ru | zh
Llama 2 70B-I | 97.9 | 100.0 | 100.0 | 100.0 | 100.0 | 93.2 | 94.4 | 84.2 | 91.4 | 85.3 | 84.4 | 86.7 | 81.5 | 75.9
Llama 3 70B-I | 93.0 | 95.6 | 95.7 | 80.0 | 100.0 | 93.5 | 93.3 | 94.4 | 95.7 | 97.7 | 91.7 | 100.0 | 89.1 | 92.3
Mixtral 8x7B | 73.7 | 86.0 | 78.9 | 68.2 | 61.7 | 83.1 | 64.7 | 68.2 | 76.3 | 71.5 | 51.5 | 67.3 | 80.5 | 62.1
Mistral Large | 98.4 | 100.0 | 94.7 | 97.9 | 100.0 | 99.0 | 98.9 | 93.8 | 93.5 | 95.4 | 91.5 | 92.0 | 92.8 | 97.7
Command R | 96.3 | 99.3 | 99.0 | 93.9 | 97.0 | 96.0 | 92.3 | 94.0 | 94.3 | 98.6 | 88.5 | 97.2 | 94.0 | 91.1
Command R+ | 99.4 | 99.7 | 100.0 | 99.0 | 100.0 | 98.0 | 100.0 | 95.1 | 97.9 | 96.0 | 95.5 | 96.1 | 89.7 | 95.6
GPT-3.5 Turbo | 99.8 | 100.0 | 100.0 | 99.0 | 100.0 | 100.0 | 100.0 | 98.7 | 99.0 | 99.1 | 98.6 | 98.5 | 98.3 | 99.0
GPT-4 Turbo | 99.7 | 100.0 | 100.0 | 100.0 | 99.0 | 100.0 | 99.0 | 96.6 | 97.4 | 97.3 | 95.6 | 96.8 | 95.7 | 97.1

Table A3: Word-level pass rate (WPR) on monolingual and cross-lingual
generation (% of responses containing no English words). Llama models are
instruction-tuned variants.

| Monolingual | Cross-lingual
---|---|---
| Avg | ar | hi | ja | ko | ru | zh | Avg | ar | hi | ja | ko | ru | zh
Llama 2 70B-I | 20.9 | 0.7 | 2.0 | 13.1 | 0.0 | 91.0 | 18.9 | 27.2 | 21.8 | 33.9 | 27.2 | 9.0 | 52.1 | 19.3
Llama 3 70B-I | 31.6 | 35.3 | 37.1 | 17.8 | 0.0 | 84.5 | 14.7 | 29.3 | 46.9 | 62.9 | 2.7 | 1.5 | 53.7 | 8.3
Mixtral 8x7B | 65.3 | 61.9 | 74.7 | 67.4 | 61.4 | 72.9 | 53.4 | 61.6 | 66.6 | 50.6 | 54.5 | 61.7 | 76.2 | 59.7
Mistral Large | 69.4 | 64.9 | 31.7 | 64.4 | 78.0 | 98.5 | 79.2 | 63.3 | 52.1 | 72.5 | 59.9 | 56.9 | 75.3 | 63.0
Command R | 98.0 | 99.7 | 99.5 | 96.9 | 98.5 | 98.0 | 95.3 | 78.6 | 74.5 | 78.7 | 75.1 | 80.9 | 79.9 | 82.3
Command R+ | 99.4 | 99.7 | 100.0 | 99.0 | 100.0 | 99.0 | 98.7 | 93.4 | 95.6 | 93.0 | 94.6 | 93.5 | 91.3 | 92.5
GPT-3.5 Turbo | 99.4 | 100.0 | 99.5 | 98.5 | 100.0 | 100.0 | 98.5 | 94.1 | 94.7 | 95.4 | 93.2 | 94.2 | 94.9 | 92.3
GPT-4 Turbo | 99.7 | 99.5 | 100.0 | 100.0 | 99.5 | 100.0 | 99.0 | 92.8 | 93.0 | 94.1 | 91.5 | 93.1 | 92.8 | 92.3

Table A4: Aggregate language confusion pass rate (LCPR) of LLMs on monolingual
and cross-lingual tasks.

models | prompt | generation | confusion type
---|---|---|---
Llama 3 70B-I |  ローイングエクササイズの効果について論じる記事を1,000文字以内で書いてください。 |  **The Effects of Rowing Exercise: A Comprehensive Review**\n\nRowing exercise has gained popularity in recent years due to its numerous benefits for physical and mental health. As a low-impact, full-body workout, rowing has been shown to improve cardiovascular fitness, increase muscle strength and endurance, and enhance overall well-being. In this article, we will discuss the effects of rowing exercise and explore its advantages for individuals of all fitness levels.\n\n**Cardiovascular Benefits**\n\nRowing is an excellent cardiovascular | Full response
Mixtral 8x7B |  问题：油在生活中有哪些用途？ |  ’油在我们的日常生活中有许多用途，主要包括：\n\n1. 烹饪：油是烹饪中不可或缺的一种原料，它可以用来油炸、煎炸、烧烤等 verschiedenen Kochtechniken. Es kann auch als Salatöl oder Dressing verwendet werden.\n’ | Line level
Command R |  교실에서 시작하는 디지털 시민성 교육에 관한 교육 자료를 초등학교 수준으로 쉬운 말투를 사용하여 200단어 이내로 작성해줘. |  디지털 시민이란 인터넷과 디지털 기술로 연결된 세상에서 다른 사람들과 상호작용하고, 소통하는 방법을 아는 사람을 말해. 디지털 세상에서 우리는 좋은 친구들이나 가족들처럼 행동해야 해. \n\n교실에서 디지털 시민이 되는 법을 배워보아요! 우리는 먼저 인터넷에서 주의해야 할 점을 알아볼 거야. 디지털 세상에서 우리 would 안전한 웹사이트만 방문하고, 우리 개인정보를 소중히 지켜야 해. | Word level
Table A5: Examples of different types of language confusion across models.

###  A.2 Annotator statistics

Prompts were written by native-level speakers of the respective languages,
each of whom is also fluent in English. Annotators were paid by the hour, with
compensation above the federal minimum wage of the country of employment.

###  A.3 Cross-lingual prompt generation

In order to generate prompts for the cross-lingual setting, we semi-
automatically amend English prompts with an English instruction to generate
text in the target language. For each English prompt and target language, we
randomly choose an English instruction (e.g., “Respond in French”, “Reply in
Turkish”). To control for the position of the instruction in the prompt, for
each prompt separate examples are created where an instruction is inserted at
the beginning and at the end respectively. For each prompt, we additionally
generate another example where an instruction is manually integrated into the
prompt, e.g., “Generate an essay in Korean”. The same process is applied to
Okapi, ShareGPT, and our complex prompts. Table 2 shows representative
examples where instructions are integrated in the middle, inserted at the
beginning and at the end of the original prompts.

###  A.4 Metric Variability

We find there is little variability LPR and WPR between runs. We run
monolingual and crosslingual language confusion tasks on the FP16 and W4g
variants of Command R/R+ 5 times each, and report the minimum, maximum, and
mean of LPR and WPR in Table A6.

|  | Monolingual LPR | Monolingual WPR
---|---|---|---
|  | avg | ar | de | es | fr | hi | id | it | ja | ko | pt | ru | tr | vi | zh | avg | ar | hi | ja | ko | ru | zh
Command R+ fp16 | min | 99.1 | 98.6 | 100.0 | 99.0 | 98.7 | 100.0 | 95.0 | 100.0 | 98.0 | 100.0 | 97.5 | 100.0 | 100.0 | 99.0 | 97.0 | 99.1 | 99.3 | 100.0 | 99.0 | 98.0 | 97.0 | 100.0
| mean | 99.3 | 99.3 | 100.0 | 99.3 | 99.6 | 100.0 | 96.6 | 100.0 | 98.6 | 100.0 | 98.3 | 100.0 | 100.0 | 99.8 | 97.9 | 99.4 | 99.7 | 100.0 | 99.8 | 99.4 | 97.6 | 100.0
| max | 99.5 | 100.0 | 100.0 | 99.7 | 100.0 | 100.0 | 99.0 | 100.0 | 99.0 | 100.0 | 99.0 | 100.0 | 100.0 | 100.0 | 99.0 | 99.6 | 100.0 | 100.0 | 100.0 | 100.0 | 99.0 | 100.0
Command R+ w4g | min | 99.3 | 98.6 | 100.0 | 99.3 | 99.7 | 100.0 | 97.0 | 100.0 | 97.0 | 100.0 | 97.5 | 100.0 | 100.0 | 100.0 | 98.5 | 99.5 | 100.0 | 100.0 | 99.0 | 99.0 | 99.0 | 98.5
| mean | 99.4 | 99.1 | 100.0 | 99.6 | 99.9 | 100.0 | 97.8 | 100.0 | 97.4 | 100.0 | 98.1 | 100.0 | 100.0 | 100.0 | 98.9 | 99.7 | 100.0 | 100.0 | 99.8 | 99.8 | 99.4 | 99.2
| max | 99.5 | 99.6 | 100.0 | 99.7 | 100.0 | 100.0 | 99.0 | 100.0 | 98.0 | 100.0 | 98.5 | 100.0 | 100.0 | 100.0 | 99.5 | 99.8 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0
Command R fp16 | min | 98.3 | 98.6 | 97.0 | 98.0 | 98.7 | 100.0 | 92.0 | 99.0 | 98.0 | 99.0 | 98.5 | 99.0 | 98.0 | 100.0 | 96.5 | 95.1 | 97.3 | 95.0 | 90.7 | 97.0 | 93.9 | 92.3
| mean | 98.6 | 99.2 | 97.0 | 98.1 | 99.2 | 100.0 | 93.6 | 99.6 | 99.6 | 99.0 | 99.0 | 99.8 | 98.6 | 100.0 | 97.7 | 95.6 | 97.8 | 96.0 | 92.5 | 97.6 | 96.2 | 93.4
| max | 98.8 | 99.3 | 97.0 | 98.3 | 99.3 | 100.0 | 95.0 | 100.0 | 100.0 | 99.0 | 99.5 | 100.0 | 99.0 | 100.0 | 98.5 | 96.5 | 98.7 | 97.0 | 96.0 | 98.0 | 97.0 | 94.4
Command R w4g | min | 98.5 | 99.0 | 95.0 | 96.3 | 99.7 | 99.0 | 96.0 | 100.0 | 99.0 | 97.0 | 98.0 | 100.0 | 98.0 | 99.0 | 98.0 | 95.7 | 98.3 | 97.0 | 86.9 | 95.9 | 96.0 | 93.9
| mean | 98.6 | 99.4 | 95.0 | 96.5 | 99.9 | 99.8 | 97.2 | 100.0 | 99.8 | 97.0 | 98.3 | 100.0 | 98.8 | 99.8 | 98.6 | 95.9 | 98.4 | 97.6 | 89.1 | 96.7 | 98.4 | 95.2
| max | 98.7 | 100.0 | 95.0 | 97.0 | 100.0 | 100.0 | 98.0 | 100.0 | 100.0 | 97.0 | 98.5 | 100.0 | 99.0 | 100.0 | 99.5 | 96.2 | 98.6 | 98.0 | 91.9 | 97.9 | 100.0 | 96.0
|  | Crosslingual LPR | Crosslingual WPR
|  | avg | ar | de | es | fr | hi | id | it | ja | ko | pt | ru | tr | vi | zh | avg | ar | hi | ja | ko | ru | zh
Command R+ fp16 | min | 90.8 | 92.5 | 90.3 | 90.0 | 91.1 | 88.1 | 87.8 | 92.4 | 92.4 | 90.6 | 87.5 | 89.8 | 91.5 | 90.1 | 90.2 | 94.8 | 97.5 | 94.4 | 94.7 | 94.6 | 90.2 | 94.4
| mean | 91.0 | 93.0 | 90.6 | 91.2 | 91.6 | 88.8 | 88.2 | 93.0 | 93.1 | 91.1 | 88.3 | 91.3 | 92.0 | 90.8 | 91.3 | 95.1 | 97.9 | 95.2 | 95.7 | 95.1 | 91.1 | 95.9
| max | 91.2 | 93.6 | 91.0 | 92.4 | 91.9 | 89.2 | 88.6 | 93.4 | 94.1 | 91.3 | 89.6 | 92.4 | 92.9 | 91.5 | 92.5 | 95.4 | 98.3 | 95.7 | 96.6 | 95.6 | 92.0 | 96.5
Command R+ w4g | min | 90.0 | 89.9 | 89.2 | 90.4 | 91.6 | 87.5 | 85.6 | 92.5 | 92.4 | 89.8 | 84.1 | 90.6 | 89.8 | 90.4 | 89.2 | 94.8 | 97.2 | 94.2 | 96.0 | 94.9 | 88.7 | 95.7
| mean | 90.1 | 90.6 | 89.9 | 90.7 | 91.7 | 88.4 | 86.5 | 93.1 | 92.8 | 90.6 | 85.4 | 91.0 | 90.7 | 90.7 | 89.6 | 95.3 | 97.7 | 94.8 | 96.5 | 95.8 | 90.3 | 96.5
| max | 90.2 | 91.1 | 90.7 | 91.0 | 91.7 | 89.1 | 87.4 | 93.8 | 93.4 | 91.1 | 85.9 | 91.4 | 91.5 | 91.0 | 90.5 | 95.6 | 98.2 | 95.9 | 97.3 | 96.4 | 91.1 | 97.6
Command R fp16 | min | 66.3 | 57.9 | 58.6 | 68.8 | 72.1 | 63.9 | 68.7 | 62.9 | 65.1 | 68.6 | 63.8 | 67.4 | 63.6 | 64.2 | 74.2 | 92.9 | 97.5 | 97.6 | 88.6 | 90.5 | 86.6 | 90.2
| mean | 66.4 | 58.8 | 59.6 | 69.0 | 73.0 | 64.0 | 69.3 | 63.6 | 66.3 | 69.2 | 64.2 | 68.8 | 64.2 | 65.1 | 74.6 | 93.6 | 98.5 | 98.4 | 90.5 | 93.3 | 88.7 | 92.1
| max | 66.6 | 59.5 | 60.0 | 69.5 | 73.8 | 64.3 | 69.8 | 63.9 | 67.4 | 69.6 | 64.4 | 69.7 | 64.8 | 66.3 | 74.8 | 94.4 | 99.2 | 99.6 | 92.1 | 95.1 | 91.0 | 93.4
Command R w4g | min | 67.0 | 60.3 | 58.9 | 72.4 | 73.0 | 67.3 | 66.6 | 64.2 | 65.0 | 69.5 | 64.1 | 68.9 | 64.2 | 65.7 | 72.5 | 92.5 | 96.3 | 94.8 | 89.2 | 92.8 | 86.5 | 90.8
| mean | 67.2 | 60.4 | 59.3 | 72.8 | 73.3 | 67.7 | 67.9 | 64.8 | 65.4 | 70.4 | 64.6 | 69.7 | 64.6 | 66.1 | 73.0 | 93.0 | 97.5 | 95.6 | 90.3 | 94.8 | 88.0 | 92.3
| max | 67.3 | 60.6 | 59.6 | 73.0 | 73.8 | 68.0 | 68.6 | 65.2 | 66.0 | 71.1 | 65.1 | 70.6 | 65.2 | 66.4 | 73.8 | 93.6 | 98.4 | 96.2 | 91.6 | 95.9 | 89.3 | 93.6

Table A6: Variability in LPR & WPR metrics for Command R/R+ models over 5
evaluation runs.

###  A.5 Dataset Language Confusion Metrics

| Avg | Okapi | Aya | Dolly | Native (Ours)
---|---|---|---|---|---
Llama 2 70B-I | 54.0 | 57.2 | 49.6 | 55.2 | 49.0
Llama 3 70B-I | 56.2 | 56.0 | 50.8 | 61.8 | 47.0
Mixtral 8x7B | 74.9 | 70.7 | 78.3 | 75.8 | 78.2
Mistral Large | 75.6 | 68.4 | 82.0 | 76.4 | 77.8
Command R | 98.7 | 97.8 | 99.4 | 99.0 | 98.8
Command R+ | 99.3 | 99.0 | 99.0 | 99.8 | 99.2
GPT-3.5 Turbo | 99.1 | 99.0 | 98.6 | 99.8 | 99.5
GPT-4 Turbo | 99.4 | 98.9 | 99.4 | 99.8 | 99.2

Table A7: LPR by dataset on monolingual generation.

| Monolingual | Cross-lingual
---|---|---
| Avg | Okapi | Aya | Dolly | Native (Ours) | Avg | Okapi | ShareGPT | Complex (Ours)
Llama 2 70B-I | 97.4 | 100.0 | 94.4 | 97.7 | 100.0 | 84.2 | 90.0 | 78.9 | 83.7
Llama 3 70B-I | 95.1 | 100.0 | 93.3 | 91.9 | 90.0 | 94.4 | 98.9 | 87.9 | 96.4
Mixtral 8x7B | 78.0 | 77.9 | 74.8 | 81.3 | 64.9 | 68.2 | 75.8 | 68.0 | 60.8
Mistral Large | 98.9 | 100.0 | 98.9 | 97.9 | 99.0 | 93.8 | 98.7 | 95.4 | 87.3
Command R | 96.5 | 97.4 | 93.8 | 98.3 | 95.5 | 94.0 | 95.3 | 93.7 | 92.9
Command R+ | 99.7 | 100.0 | 100.0 | 99.0 | 99.5 | 95.1 | 97.4 | 97.6 | 90.3
GPT-3.5 Turbo | 100.0 | 100.0 | 100.0 | 100.0 | 99.5 | 98.7 | 100.0 | 99.8 | 96.4
GPT-4 Turbo | 99.7 | 100.0 | 99.0 | 100.0 | 99.5 | 96.6 | 99.8 | 99.5 | 90.6

Table A8: WPR by dataset for monolingual and cross-lingual generation.

| Monolingual | Cross-lingual
---|---|---
| Avg | Okapi | Aya | Dolly | Native (Ours) | Avg | Okapi | ShareGPT | Complex (Ours)
Llama 2 70B-I | 69.4 | 72.8 | 65.0 | 70.5 | 65.8 | 52.0 | 58.6 | 58.1 | 39.3
Llama 3 70B-I | 70.5 | 71.8 | 65.8 | 73.9 | 61.8 | 44.7 | 51.7 | 54.5 | 27.9
Mixtral 8x7B | 76.4 | 74.1 | 76.5 | 78.5 | 71.0 | 68.3 | 76.6 | 73.4 | 54.8
Mistral Large | 85.6 | 81.2 | 89.7 | 85.8 | 87.1 | 71.7 | 80.5 | 70.8 | 64.0
Command R | 97.6 | 97.6 | 96.5 | 98.7 | 97.1 | 76.9 | 84.3 | 91.6 | 54.9
Command R+ | 99.5 | 99.5 | 99.5 | 99.4 | 99.4 | 93.0 | 96.7 | 98.2 | 84.2
GPT-3.5 Turbo | 99.6 | 99.5 | 99.3 | 99.9 | 99.5 | 93.8 | 98.8 | 98.3 | 84.4
GPT-4 Turbo | 99.5 | 99.4 | 99.2 | 99.9 | 99.4 | 93.3 | 98.2 | 97.9 | 83.7

Table A9: LCPR by dataset for monolingual and cross-lingual generation.

###  A.6 Extended Impacts

| Short | Medium | Long
---|---|---|---
| [21, 65] | [68, 224] | [227, 2971]
Command R | 42.7 | 28.1 | 38.0
\+ one-shot | 59.5 | 49.1 | 67.7
Command R+ | 82.1 | 74.8 | 79.8
GPT-4 | 89.3 | 72.9 | 71.8

Table A10: Line-level pass rate (LPR) on our “Complex prompts” dataset for crosslingual generation, depending on length of the prompts. We sort the prompts by length and split them into 3 length buckets of the same size (each containing one third of the prompts). [a, b]: min and max length in words of each bucket’s prompts. | Start | Integrated | End
---|---|---|---
Command R | 86.7 | 69.0 | 85.1
\+ one-shot | 88.7 | 80.6 | 90.6
Command R+ | 94.4 | 90.3 | 95.2
GPT-4 | 93.0 | 91.7 | 95.0
Table A11: Line-level pass rate (LPR) on crosslingual generation depending on
the position of the language control instruction. “Integrated” corresponds to
instructions of the form “Write an essay of 100 words in Korean about
artificial intelligence.” “Start” and “End” are isolated instructions of the
form “Reply in French.” placed either at the start or at the end of the
prompt.

| Line-level Pass Rate (LPR)
---|---
| avg | ar | hi | ja | ko | ru | zh | id
FP16 | 99.3 | 99.0 | 100.0 | 99.0 | 100.0 | 100.0 | 97.5 | 97.0
W8 | 99.3 | 99.7 | 100.0 | 98.0 | 100.0 | 100.0 | 99.0 | 97.0
W8A8 | 99.5 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 98.5 | 98.0
W4-g | 98.8 | 99.3 | 99.0 | 98.0 | 100.0 | 100.0 | 98.5 | 91.0
| Word-level Pass Rate (WPR)
FP16 | 98.2 | 99.3 | 100.0 | 100.0 | 99.0 | 95.0 | 96.0 | -
W8 | 98.3 | 99.3 | 99.0 | 99.0 | 100.0 | 96.0 | 96.5 | -
W8A8 | 98.7 | 99.3 | 99.0 | 100.0 | 100.0 | 97.0 | 97.0 | -
W4-g | 98.1 | 99.3 | 100.0 | 98.0 | 99.0 | 93.9 | 98.5 | -

Table A12: Effect of quantization on LPR and WPR on monolingual generation.

| Monolingual Datasets
---|---
| avg | Okapi | Aya | Dolly | Native (Ours)
Llama 2 70B-I | 54.0 | 57.2 | 49.6 | 55.2 | 49.0
Llama 3 70B-I | 56.2 | 56.0 | 50.8 | 61.8 | 47.0
Mixtral 8x7B | 74.9 | 70.7 | 78.3 | 75.8 | 78.2
Mistral Large | 75.6 | 68.4 | 82.0 | 76.4 | 77.8
Command R | 98.7 | 97.8 | 99.4 | 99.0 | 98.8
Command R+ | 99.3 | 99.0 | 99.0 | 99.8 | 99.2
GPT-3.5 Turbo | 99.1 | 99.0 | 98.6 | 99.8 | 99.5
GPT-4 Turbo | 99.4 | 98.9 | 99.4 | 99.8 | 99.2

Table A13: Line-level pass rate (LPR) by dataset on monolingual generation. | avg | ar | hi | ja | ko | ru | zh
---|---|---|---|---|---|---|---
T=0.0 | 93.3 | 98.9 | 94.6 | 86.4 | 96.7 | 93.0 | 90.0
T=0.3 | 94.0 | 94.3 | 98.6 | 88.5 | 97.2 | 94.0 | 91.1
T=0.5 | 94.2 | 98.9 | 96.1 | 87.1 | 95.2 | 95.5 | 92.3
T=0.7 | 94.6 | 97.9 | 99.1 | 90.8 | 95.7 | 94.6 | 89.6
T=1.0 | 80.1 | 93.5 | 88.6 | 70.5 | 85.5 | 71.0 | 71.2
p=0.1 | 93.6 | 97.4 | 96.6 | 89.0 | 97.5 | 93.3 | 87.6
p=0.3 | 94.0 | 98.0 | 97.6 | 85.7 | 98.4 | 92.0 | 92.3
p=0.5 | 94.0 | 95.0 | 97.3 | 89.0 | 98.8 | 93.8 | 90.1
p=0.75 | 94.0 | 94.3 | 98.6 | 88.5 | 97.2 | 94.0 | 91.1
Table A14: Effect of varying temperature ($T$) or nucleus size ($p$) on cross-
lingual word-level language confusion (WPR) of Command R. Default values are
$p=0.75$ and $T=0.3$. Best score.

###  A.7 Quantization

It is common to train at half-precision floating-point (FP16), where weights
and activations of a network use 16 bits (2 bytes) to represent a floating-
point value. It is common to quantize weights to INT8 (8-bit integers), often
referred to as W8. More extreme quantization of weights to INT4 (4-bit) is
called W4. Quantizing both weights and activations to INT8 is commonly called
W8A8.

In Section 4.4, we compare FP16 with W8, W8A8, and W4242424W4 with group-wise
scaling using GPTQ (Frantar et al., 2022). variants of Command R+ on
monolingual generation.

###  A.8 Base vs Instruction Tuning comparison

Q: Write your answer in French. How should I choose what cheese to buy? A: Il
existe de nombreux types de fromages différents, donc le choix du fromage à
acheter dépend des préférences personnelles, de la disponibilité et de
l’utilisation prévue. […] Q: What is the difference between pets and cattle?
Reply in Arabic. A: Figure A2: Template used for few-shot prompting the base
models. The model’s answers are truncated to prevent the generation of new
questions. For the instruct variants, we use similar prompting, except that
the Q/A examples are formatted as User/Chatbot turns using the model’s chat
template.

| Avg | ar | de | en | es | fr | hi | id | it | ja | ko | pt | ru | tr | vi | zh
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Llama 2 70B | 97.5 | 100.0 | 92.9 | 99.5 | 98.0 | 96.5 | 98.9 | 91.9 | 98.0 | 100.0 | 100.0 | 95.8 | 100.0 | 98.7 | 98.0 | 93.6
Llama 2 70B-I | 48.3 | 0.3 | 59.0 | 99.0 | 95.7 | 87.7 | 1.0 | 62.0 | 72.0 | 7.0 | 0.0 | 91.0 | 88.9 | 33.0 | 17.0 | 10.5
Llama 3 70B | 94.8 | 95.6 | 96.9 | 99.0 | 96.6 | 97.6 | 97.9 | 82.8 | 93.9 | 88.9 | 100.0 | 96.2 | 100.0 | 94.4 | 93.9 | 89.1
Llama 3 70B-I | 46.0 | 21.7 | 31.0 | 100.0 | 98.3 | 88.7 | 23.0 | 21.0 | 88.0 | 10.0 | 0.0 | 95.5 | 77.0 | 18.0 | 10.0 | 8.0
Command R base | 86.3 | 94.5 | 83.8 | 98.5 | 89.6 | 86.6 | 81.0 | 69.4 | 79.0 | 98.0 | 94.3 | 83.9 | 93.6 | 91.0 | 79.6 | 71.1
Command R | 98.6 | 100.0 | 98.0 | 99.5 | 95.7 | 99.3 | 100.0 | 92.0 | 99.0 | 100.0 | 100.0 | 98.5 | 100.0 | 99.0 | 99.0 | 98.5
Command R+ base | 82.1 | 93.2 | 71.4 | 98.0 | 81.9 | 86.7 | 68.7 | 80.8 | 65.0 | 92.7 | 92.1 | 79.8 | 93.5 | 95.0 | 70.7 | 62.4
Command R+ | 99.2 | 99.7 | 100.0 | 100.0 | 99.3 | 99.7 | 100.0 | 97.0 | 100.0 | 99.0 | 100.0 | 97.5 | 100.0 | 100.0 | 99.0 | 97.5

Table A15: LPR of base vs instruction-tuned LLMs for monolingual generation.

###  A.9 Nucleus Sampling with Temperature

Previous work shows that greedy decoding empirically leads to repetition and
generally low-quality generations. One solution is adding stochasticity to
decoding, such as by sampling the next token from the top-K next most likely
tokens or from the top-P of the probability distribution over next tokens. The
latter is called “nucleus sampling” Holtzman et al. (2019), and can be
combined with top-K and temperature sampling.

Let $\boldsymbol{x}=[x_{0},x_{1},x_{2},...,x_{n-1}]$ be a sequence of tokens.
We describe how to sample the next token in the sequence via nucleus sampling
with temperature given a language model with vocabulary $V$.

The nucleus of a probability distribution for $0~{}<~{}p~{}\leq~{}1$ is the
smallest subset $V^{\prime}~{}\subset~{}V$ such that the summed probabilities
of $V^{\prime}$ are greater than or equal to $p$.252525Some implementations
will define $V^{\prime}$ as the largest subset $V^{\prime}~{}\subset~{}V$ s.t.
the summed probabilities of $V^{\prime}$ is less-than-or-equal-to $p$. We use
the definition from the original work. Nucleus sampling samples the next token
$x_{n}$ at random from this set. The function top_p returns $V^{\prime}$:

$\text{top\\_p}(\boldsymbol{x},V,p)=v\in V\text{ s.t.
}\sum_{i}P(v_{i}|\boldsymbol{x})\geq p$

Let $z\in\mathbb{R}^{|V|}$ be an output vector of logits. We apply the softmax
function over element $z_{i}$ given temperature $T$ to transform its logit
$z_{i}$ to a probability over vocabulary tokens, as below. Let $v_{i}$ be the
vocabulary token corresponding to logit $z_{i}$.

$P(v_{i}|\boldsymbol{x})=\textit{softmax}(z_{i},T)=\frac{e^{\frac{z_{i}}{T}}}{\sum_{j}e^{\frac{z_{j}}{T}}}$
(1)

The next token $x_{n}$ is chosen at random from normalized probabilities in
the nucleus:

$P_{normalized}(v^{\prime}_{i}|\boldsymbol{x})=\frac{P(v^{\prime}_{i}|\boldsymbol{x})}{\sum_{v^{\prime}_{j}\in
V^{\prime}}P(v^{\prime}_{j}|\boldsymbol{x})}$

$T$ controls the peakiness of the distribution at the sampling point. As $T$
increases, the distribution becomes more uniform. Figure A3 shows the effect
on Equation 1 for a toy example. As $T$ increases, previously unlikely words
become more likely, and highly likely words have lower probabilities.

We walk through Figure 3 as an example.

#### Lowering Temperature

Imagine we run inference with $p=0.75,T=1$. Given previous tokens ‘the’, ‘
quick’, and ‘ brown’, the next token options are ‘ fox’, ‘ dog’, ‘ cube’, ‘
狐狸’, and ‘ after’ with logits $[0.75,0.20,-0.10,-0.20,-0.30]$. Applying the
softmax, ‘ fox’, ‘ dog’, ‘ cube’, and ‘ 狐狸’ remain in the nucleus with
respective likelihoods $[0.418,0.241,0.179,0.162]$. ‘ 狐狸’ has an over 16%
chance of being sampled at the next step—a high risk for language confusion.
Increasing $T$ to $2.0$, ‘ 狐狸’ has an over 20% chance of being elicited.
Reducing $T$ to $0.5$, however, we sharpen the distribution by shifting
probability mass to ‘ fox’, ‘ dog’, and ‘ cube’ such that ‘ 狐狸’ falls out of
the nucleus and cannot be sampled, thus eliminating the risk of language
confusion at this timestep.

#### Reducing Nucleus Size

The distribution may also be sharpened by decreasing nucleus size. Keeping
$T=1$ but reducing nucleus size to $p=0.7$, ‘ 狐狸’ is not in the nucleus. The
probabilities over next tokens in the nucleus are then: ${\textit{``
fox''}:0.499,\textit{`` dog''}:0.287,\textit{`` cube''}:0.213}$, and the risk
of language confusion is eliminated.

Figure A3: Effect of increasing temperature $T$ on output distribution. As $T$
$\uparrow$, the distribution flattens.