# Boosting Cross-lingual Transferability in Multilingual Models
via In-Context Learning

Sunkyoung Kim1,* Dayeon Ki2,*,$\dagger$ Yireun Kim1 Jinsik Lee1
1LG AI Research 2Korea University
{sunkyoung.kim, yireun.kim<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Existing cross-lingual transfer (CLT) prompting methods are only concerned
with monolingual demonstration examples in the source language. In this paper,
we propose In-CLT, a novel cross-lingual transfer prompting method that
leverages both source and target languages to construct the demonstration
examples. We conduct comprehensive evaluations on multilingual benchmarks,
focusing on question answering tasks. Experiment results show that In-CLT
prompt not only improves multilingual models’ cross-lingual transferability,
but also demonstrates remarkable unseen language generalization ability. In-
CLT prompting, in particular, improves model performance by 10 to 20% points
on average when compared to prior cross-lingual transfer approaches. We also
observe the surprising performance gain on the other multilingual benchmarks,
especially in reasoning tasks. Furthermore, we investigate the relationship
between lexical similarity and pre-training corpora in terms of the cross-
lingual transfer gap.

00footnotetext: Equal Contribution.00footnotetext: Work done during internship
at LG AI Research.

##  1 Introduction

Cross-lingual transferability is an ability to transfer and utilize knowledge
learned from a rich language source, typically English, to low-resource
languages. Expecting to have such capabilities, multi-lingual pre-trained
models have been studied following English pre-trained models; including mBERT
based on BERT (Devlin et al., 2019) , XLM-R (Conneau et al., 2019) based on
RoBERTa (Liu et al., 2019), and mT5 (Xue et al., 2020) based on T5 (Raffel et
al., 2019). With no exception to decoder-only generative architectures, the
corresponding billion-scale multilingual models have been studied following
the successful emergence of GPT-3 (Brown et al., 2020), which gained
prominence from its in-context learning ability. BLOOM (Scao et al., 2022) and
XGLM (Lin et al., 2021) have shown high in-context learning performances in
various downstream multilingual tasks.

(a) XQuAD

(b) MLQA

Figure 1: Model performance on multilingual QA tasks for BLOOM 7.1B and XGLM
7.5B when $k=5$ shots. Each bar represents the average F1 score for Out-CLT
and In-CLT prompt respectively.

However, there still lacks in-depth studies about prompting for few-shot tasks
those require cross-lingual knowledge transfer. In particular, we tackle two
multilingual QA tasks, XQuAD (Artetxe et al., 2020) and MLQA (Lewis et al.,
2020), which requires the comprehension ability to answer with given context.
Furthermore, a model should be able to transfer such ability learned in
English to low resource languages to achieve high performance in the cross-
lingual few-shot setting.

Figure 2: Out-CLT and In-CLT prompting examples when the source language is
English and the target is German for in-context learning. German questions and
answer in the In-CLT are parallel to Out-CLT. Gray colored text is the answer
and the text in the bracket is the translated English dataset.

In this paper, we propose a novel cross-lingual transfer prompting method, In-
CLT. This prompting composes the few-shot examples crossing the source and
target language inside the demonstration in order to better transfer knowledge
between both languages. We compare two different prompting methods for cross-
lingual knowledge transfer: In-CLT (Inside cross-lingual transfer) and Out-CLT
(Outside cross-lingual transfer). Out-CLT, which is equivalent to the same-
language-prompting introduced by Lin et al. (2021), composes demonstration
examples with English and a query example with a target language. We depict a
detailed example of the demonstration composed by each In-CLT and Out-CLT
method in Figure 2.

Experimental results on XQuAD and MLQA show the effectiveness of the In-CLT
method, outperforming the prior cross-lingual transfer method with monolingual
demonstration examples. Especially when increasing the model size and the
number of few-shot examples, multilingual models’ cross-lingual
transferability with In-CLT increases to a greater extent. We also observe
that our prompting method shows remarkable unseen language generalization
ability, indicating its potential in unseen settings. It stills shows
significant boosting performance gain on the other multilingual tasks (PAWS-X
(Yang et al., 2019), XNLI Conneau et al. (2018), XCOPA Ponti et al. (2020)).
Furthermore, we conclude that In-CLT stimulates cross-lingual transferability
in multilingual models, which aligns with the scaling law.

We further extend our analysis to cross-lingual transfer gaps in relation to
lexical similarity and the pre-training corpus. Cross-lingual transfer via In-
CLT prompting is effective for lexically similar languages to English for the
majority of the target languages. On the other hand, models struggle to
transfer knowledge from English to a target language that is not learned
during pre-training.

To summarize, our contributions are as follows.

  * •

We propose the In-CLT prompting method, which remarkably enhances and
stimulates cross-lingual transferability of multilingual models.

  * •

In-CLT prompting shows superior unseen language generalization ability and
scaling law.

  * •

We conduct extensive experiments on the autoregressive multilingual models’
cross-lingual capabilities.

##  2 Cross-lingual Transfer Prompting

We focus on two prompting approaches to measure the cross-lingual capability
in few-shot settings: Out-CLT (Outside Cross-lingual Transfer) and In-CLT
(Inside Cross-lingual Transfer). The major difference between the two
prompting methods lies in how they compose demonstration examples. As shown in
Figure 2, Out-CLT resembles the existing cross-lingual transfer prompting
method where demonstrations and query examples are in the source and target
language, respectively. Recently, Ahuja et al. (2023) adopt Out-CLT to
evaluate multilingual models, called as zero-shot cross-lingual transfer.

Moreover, Lin et al. (2021) introduce two variants of the Out-CLT prompt;
same-language-prompting and source-language-prompting. While same-language-
prompting is equivalent to the Out-CLT prompt, source-language-prompting only
uses the answer template in the source language for classification tasks. It
is difficult to make a fair comparison that we generate the answer instead of
calculating the log-likelihood to the label space.

We suggest a novel prompting method, In-CLT, where the source and target
languages are crossed within the demonstration examples. We alternate two
languages at the attribute level (context, question, and answer for
multilingual QA.

###  2.1 Inside Cross-lingual Transfer (In-CLT)

In-CLT is a novel prompting method that reflects both the characteristics of
cross-lingual transfer and inter-sentential code switching. 111Linguistically,
code-switching can be largely divided into intra-sentential (switching
languages within a single sentence), inter-sentential (switching languages by
sentences), and extra-sentential (tag switching), which involves inserting
tags from different languages into sentences. We aim to catalyze the cross-
lingual capability of the model by crossing languages in the demonstration
examples.

The motivation of In-CLT is rooted from understanding the mechanism of human-
level cross-lingual ability. Humans are potentially multilingual by nature
Hammarberg (2010) and can not only transfer across languages, but also
understand text mixed in multiple languages Neuser (2017). However, previous
cross-lingual prompting methods have mainly focused on the transfer ability
from one language to another.

Therefore, we design the In-CLT prompt to reflect the ability of the model to
simultaneously understand and transfer multilingual text. In-CLT is an
advanced language mixing method which aids the model to understand both the
source and target languages through seamless implicit translation. We expect
our prompt to act as a trigger to activate the cross-lingual thinking ability
of multilingual models.

In-CLT prompting method composes a few-shot example by mixing context as the
source language and the question-answer pair as the target language (src-tgt-
tgt structure). This is a setting for investigating the effect of the same-
language bias in multilingual models. Roy et al. (2020) observed that models
have a tendency to answer in the same language as the question and refers to
this as ‘same-language bias’. Likewise, we examine whether models can take
advantages of such bias.

##  3 Experiments

Task MLQA XQuAD Model / Prompt MONO Out-CLT In-CLT MONO Out-CLT In-CLT BLOOM
1.7B 28.98 24.85 25.10 22.15 18.05 20.60 BLOOM 3B 36.61 33.12 34.97 26.45
24.33 26.97 BLOOM 7.1B 46.59 44.82 45.35 36.39 34.39 37.06 XGLM 1.7B 31.40
28.76 28.91 30.87 28.23 31.18 XGLM 2.9B 37.73 31.99 34.61 37.82 33.70 36.76
XGLM 7.5B 44.82 36.29 39.31 38.21 37.90 41.53

Table 1: Average F1 performance on k-shot (k=5). Note that for all prompting
methods, the demonstration examples are composed differently but every query
example is same in target language.

We conduct extensive experiments on typologically diverse languages using
billion-scale autoregressive multilingual models. We focus on multilngual
question answering tasks to verify cross-lingual transferability from
knowledge source to the answer the question in different languages through in-
context learning. We limit to explore unidirectional transfer and fix the
source language as English because of its resource-rich nature. We measure the
performance on generated model output. All languages are annotated following
the ISO 639-1 Code in experiments.

###  3.1 Setups

#### Models

We consider two publicly available decoder-only billion-scale multilingual
models for evaluation.

  * •

XGLM (Lin et al., 2021) is pre-trained with the CC100 (Common Crawl) corpus,
including 30 different languages. XGLM provides 4 model size variations: 564M,
1.7B, 2.9B, and 7.5B. In order to focus only on billion-scale models, 1.7B,
2.9B, and 7.5B models as are used as baselines.

  * •

BLOOM (Scao et al., 2022) uses the ROOTS (Laurençon et al., 2022) corpus that
consists of 46 languages and 13 programming languages for pre-training. BLOOM
presents 6 model size variations: 560M, 1.1B, 1.7B, 3B, 7B, and 175B. Similar
to XGLM, we focus on the billion-scale variants, but do not include the 175B
model for evaluation due to resource limits.

#### Prompt Methods

We conduct experiment for the three promptings for each shot and language
variation. Three methods require same inference query examples in target
language. Detailed prompt template.

  * •

MONO (Ahuja et al., 2023) is monolingual demonstrations and a query example
when both are in the target language.

  * •

Out-CLT is the existing cross-lingual transfer prompting method where
demonstration examples are in the source language (English) and the query
example is in the target language. It is also called as zero-shot cross-
lingual tranfer setup (Ahuja et al., 2023).

  * •

In-CLT is the aforementioned our cross-lingual transfer prompting method
described in Section 2. The language composition of the demonstration examples
crosses the source and target languages by making the passage in the source
and question-answer (QA) pairs in the target.

###  3.2 Mulitilingual QA Benchmarks

#### Tasks

We mainly conduct extensive experiments on two multilingual QA tasks which
have knowledge source to answer and parallel data instance to all language
subsets.

  * •

MLQA (Lewis et al., 2020) has been created by automatically aligning Wikipedia
paragraphs in multiple languages and annotating paragraphs with question-
answer pairs. It is similar to SQuAD v1 (Rajpurkar et al., 2016), which is an
English reading comprehension QA task. MLQA consists of 6 different languages
(ar, de, es, hi, vi, zh).

  * •

XQuAD (Artetxe et al., 2020) has the same scheme as the MLQA dataset but only
with a test set. We construct quality filtered machine-translated validation
set using Google Translator based on the SQuAD v1 dataset. Detailed data
construction process is described in Appendix A. XQuAD dataset ranges from 11
different languages (ar, de, es, hi, ro, ru, th, tr, vi, zh).

#### Overall results

We measure the average F1 score of the generated answer given the passage and
question for each of the target language. As shown in Table 1, In-CLT prompt
is better than Out-CLT in terms of larger models. Surprisingly, In-CLT prompt
in XQuAD tasks on 7B scale models surpass the upper bound, the MONO prompt,
which only requires transferability within a single language. These results
validate that building an alignment between the source and target language via
in-context learning is helpful for transferring knowledge to tasks in the
target language.

Task PAWS-X XNLI XCOPA Model / Prompt MONO Out-CLT In-CLT MONO Out-CLT In-CLT
MONO Out-CLT In-CLT BLOOM 7.1B 49.50 37.20 49.16 33.45 26.29 33.79 42.47 35.67
43.56 XGLM 7.5B 48.72 37.65 48.34 32.45 29.63 32.53 46.31 39.98 48.47

Table 2: Average F1 performance on k-shot (k=5). Note that for all prompting
methods, the demonstration examples are composed differently but every query
example is same in target language.

#### Model performance at scale

In-CLT prompt shows significant increase when the model size becomes larger
following the scaling law (Kaplan et al., 2020), as shown in Table 1. From 1B
to 7B, In-CLT prompt shows the following performance gains: +20.25%p (MLQA),
+16.46%p (XQuAD) in BLOOM, +10.4%p (MLQA), +10.35%p (XQuAD) in XGLM. Larger
model size is also effective at narrowing the cross-lingual transfer gap
between In-CLT and MONO prompting methods. In-CLT prompt becomes more
effective with at least 3B scale model. Therefore, a more powerful performance
improvement can be expected on over 100B scale models.

###  3.3 Other Multilingual Understanding Benchmarks

We also evaluate on diverse multilingual tasks in addition to the question
answering task. For each task, we define the prompt templates as a question
answering task as done in exisiting prompting methods (Muennighoff et al.,
2022; Shi et al., 2022). We demonstrate the details of each prompts in
Appendix C. In this experiment, we mainly evaluate on the 7B-scale
multilingual models, which showed the most prominent cross-lingual
transferability in previous experiments. Instead of the loglikelihood
comparison, we measure more strictly using the exact match score of the
generated answer.

#### Tasks

We choose three multilingual understanding tasks which have parallel data
instances for all target languages. For each task, we use the validation set
for the few-shot examples and the test set to evaluate.

  * •

PAWS-X (Yang et al., 2019) is to identify whether two sentences are paraphrase
or not in 6 different languages (fr, es, de, zh, jp, ko).

  * •

XNLI (Conneau et al., 2018) is a classification task that detect the
relationship of two statements; the premise and the hypothesis. It requires
reasoning skills to solve the task. MultiXNLI Williams et al. (2018) is an
multilingual extension of XNLI in 15 languages (fr, es, de, el, bg, ru, tr,
ar, vi, th, zh, sw, ur).

  * •

XCOPA (Ponti et al., 2020) is a commonsense reasoning task that chooses the
proper causal action to the premise between two options. It is also an
multilingual version of COPA (Roemmele et al., 2011) with 11 languages (et,
ht, id, it, qu, sw, ta, th, tr, vi, zh).

#### Overall results

In-CLT prompt consistently outperforms Out-CLT prompt on all of the
multilingual understanding benchmarks as shown in Table 2. In-CLT prompting
also seems to be powerful in multilingual understanding tasks. Remarkably, In-
CLT prompt even surpasses the MONO prompt on reasoning tasks (XNLI, XCOPA). We
observe that Out-CLT prompt shows bad performance when the target language is
unseen.

##  4 Analysis

We further analyze the effect of In-CLT prompt on XQuAD. To verify our prompt
design, we compare prompts with different combinations of the language of the
question and the answer. Moreover, we examine our experimental results in
terms of the seen languages and lexical similarity between the source and
target language.

###  4.1 Ablation of Prompt

In-CLT prompt adopt a question-answer pair in the target language and passage
in the source language ($Q_{tgt}$, $A_{tgt}$). We compare two prompting
variants of In-CLT; only question in the source language and only answer in
the source language. In Table 3, our prompt design achieves the best
performance compared to its variants. Other prompts also show cross-lingual
transferability that sustains less performance loss to some extent. However,
In-CLT is the most effective option to promote cross-lingual transferability
by taking advantage of the same language bias.

$(Q_{tgt}$-$A_{tgt})$ $(Q_{tgt}$-$Q_{src})$ $(Q_{src}$-$Q_{tgt})$ BLOOM 7.1B
37.06 36.42 35.49 XGLM 7.5B 41.53 40.07 37.37

Table 3: Comparison on the prompt design that differentiate the target
language used in the question and the answer. ($Q_{tgt}$, $A_{tgt}$)
represents our In-CLT prompt.

###  4.2 Unseen Languages

We divide the 11 target languages of XQuAD into seen and unseen language
groups. Here, ‘seen’ refers to languages that are observed during pre-training
and ‘unseen’ are languages that are not. This analysis is only limited to
BLOOM because all 11 target languages are seen in XGLM. Seen language group
for BLOOM includes Arabic, Spanish, Hindi, Chinese, Vietnamese, while unseen
language group contains German, Romanian, Russian, Greek, Thai, Turkish.
Appendix B for each model shows the type and size (in GiB) of languages used
during the pre-training step.

We calculate the cross-lingual transfer gap between the performance on English
with monolingual prompt and average performance on all other languages for
each group with cross-lingual prompts. As shown in Table 4, In-CLT prompt
mitigate transfer gap 2-3%p compared to Out-CLT. However, the model still
struggles to transfer knowledge from English to a target language that is
unseen during pre-training.

Lang | Prompt | F1 ($\Delta$)
---|---|---
En | MONO | 68.17
Seen | Out-CLT | 51.52 (16.65)
In-CLT | 54.91 (13.26)
Unseen | Out-CLT | 20.12 (48.05)
In-CLT | 22.19 (45.98)
Table 4: $\Delta$ represents cross-lingual transfer gap between English
performance with monolingual prompt and the average performance for
seen/unseen language groups with cross-lingual transfer prompt.

###  4.3 Lexical Similarity

We observe the cross-lingual transfer gap of In-CLT prompt with lexical
similarity against English. Cross-lingual transfer gap is a difference between
MONO results for English and In-CLT results for all other target languages.
Lower cross-lingual transfer gap indicates better knowledge transfer from
English to the target languages. For the lexical similarity measure between
two languages, we use the genetic proximity value provided by
Elinguistics222http://www.elinguistics.net/Compare_Languages.aspx. The genetic
proximity score is measured on a scale of 0 to 100.

(a) BLOOM 7.1B

(b) XGLM 7.5B

Figure 3: Positive correlation between lexical similarity and cross-lingual
transfer gap. The transfer gap with value 0 indicates cross-lingual knowledge
“perfectly transferred” from English to the target language. Outliers,
Vietnamese and Romanian, are excluded from this visualization.

We observe a positive correlation between the cross-lingual transfer gap and a
lexical similarity against English, as shown in Figure 3. XGLM shows a
stronger Pearson’s correlation ($r$ = 0.62) than BLOOM ($r$ = 0.43). XGLM
observed all of the target languages during the pre-training stage, therefore,
the model learned the syntactic feature of languages. It results in a stronger
correlation than BLOOM.

However, BLOOM observed only a part of the target languages (es, hi, zh, ar,
vi) unlike XGLM. Instead, seen languages that are used for pre-training show a
very strong correlation ($r$ = 0.95). Unseen languages themselves show a
strong correlation as well ($r$ = 0.88). For most of the target languages,
cross-lingual transfer through In-CLT prompting was more effective for
lexically similar languages to English.

##  5 Related Work

###  5.1 Zero-/Few-shot Cross-lingual Transfer

Prior work on cross-lingual transfer have mostly focused on fine-tuning a
model on a source language (usually a high-resource language such as English)
and inferring on a target language. This method aims to evaluate how well
multilingual models transfer knowledge from one language to another. Existed
researches mainly focus on encoder based models. Several studies Pires et al.
(2019); Hsu et al. (2019) show that encoder based multilingual pre-trained
models have zero-shot cross-lingual transfer ability in various downstream
tasks on multi-task multilingual benchmarks such as XTREME Hu et al. (2020)
and XGLUE Liang et al. (2020). Few-shot cross-lingual transfer studies
Lauscher et al. (2020); Winata et al. (2021) shows performance gains using
few-shot examples. As an concurrent work, decoder-based large language models
scaling up to 100B with prompts discover multilingual capabilites (Ahuja et
al., 2023; Huang et al., 2023). Likewise, we focus on the few-shot cross-
lingual transfer and suggest a novel prompting method where the source and
target languages are mixed within a single demonstration example.

###  5.2 In-context Learning

Brown et al. (2020) first suggest in-context learning(ICL) as one branch of
meta-training 333In the context of language models, meta-training means that
models first develop a range of skills at training time and utilize those
abilities at inference time to adapt to a target task. Brown et al. (2020).
ICL is to concatenate a query example and a piece of description together to
form a prompt, which is fed into the model for prediction (Radford et al.,
2019). ICL is broadly recognized as an effective route for large-scale models
as it does not require parameter updates. In the context of cross-lingual
transfer, demonstration examples for ICL are composed only with the source
language and the query example is evaluated in a different target language, as
shown in Figure 2 (a). ICL with cross-lingual transfer (Winata et al., 2021;
Ahuja et al., 2023) or Chain of Thoughts (CoT) style cross-lingual prompting
(Shi et al., 2022; Huang et al., 2023) has concurrently been studied. However,
most of the previous methods limit their few-shot example composition in a
signle language without examining the cross-lingual capability when languages
are mixed. To the best of our knowledge, we are the first to explore the
cross-lingual capabilities of multilingual models when the source and target
languages are mixed in demonstration examples to learn stronger cross-lingual
alignment.

##  6 Conclusion

In this work, we probed the cross-lingual capabilities of the existing
autoregressive multilingual models via in-context learning. We proposed a
novel few-shot prompting method, In-CLT, which compose the demonstration
examples across source and target languages. As highlighted in our extensive
experiments, In-CLT is superior to Out-CLT, the previous cross-lingual
transfer setting. We discovered, in particular, that matching the QA pairs in
the target language in demonstration examples effectively serves as a stimulus
for the models to elicit more “cross-lingually”. Therefore, it is helpful for
unseen language generalization that not used for training the model. In the
future, we hope to apply In-CLT prompting to empower english-dominant large
language model such as GPT-3, GPT-3.5 in diverse languages.

## Acknowledgements

We thank Hyeongu Yun, Hyunjik Jo for helpful discussions and feedback on our
paper.

## References

  * Ahuja et al. (2023) Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, et al. 2023\.  Mega: Multilingual evaluation of generative ai.  _arXiv preprint arXiv:2303.12528_.
  * Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020.  On the cross-lingual transferability of monolingual representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4623–4637, Online. Association for Computational Linguistics.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and et al. Askell. 2020.  Language models are few-shot learners.
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Unsupervised cross-lingual representation learning at scale.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  Xnli: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Hammarberg (2010) Björn Hammarberg. 2010.  The languages of the multilingual: Some conceptual and terminological issues (vol. 48, no. 2-3, 2010, pp. 91-104.).
  * Hsu et al. (2019) Tsung-Yuan Hsu, Chi-Liang Liu, and Hung-yi Lee. 2019.  Zero-shot reading comprehension by cross-lingual transfer learning with multi-lingual language representation model.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5933–5940, Hong Kong, China. Association for Computational Linguistics.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization.  _CoRR_ , abs/2003.11080.
  * Huang et al. (2023) Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023.  Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting.  _arXiv preprint arXiv:2305.07004_.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.
  * Laurençon et al. (2022) Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gérard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Romero Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Vu Minh Chien, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Ifeoluwa Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Margaret Mitchell, Sasha Luccioni, and Yacine Jernite. 2022.  The bigscience ROOTS corpus: A 1.6TB composite multilingual dataset.  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
  * Lauscher et al. (2020) Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020.  From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4483–4499, Online. Association for Computational Linguistics.
  * Lewis et al. (2020) Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020\.  MLQA: Evaluating cross-lingual extractive question answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7315–7330, Online. Association for Computational Linguistics.
  * Liang et al. (2020) Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020.  Xglue: A new benchmark dataset for cross-lingual pre-training, understanding and generation.
  * Lin et al. (2021) Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona T. Diab, Veselin Stoyanov, and Xian Li. 2021.  Few-shot learning with multilingual language models.  _CoRR_ , abs/2112.10668.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
  * Muennighoff et al. (2022) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2022\.  Crosslingual generalization through multitask finetuning.
  * Neuser (2017) Hannah Neuser. 2017.  Source language of lexical transfer in multilingual learners.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019.  How multilingual is multilingual BERT? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4996–5001, Florence, Italy. Association for Computational Linguistics.
  * Ponti et al. (2020) Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020.  Xcopa: A multilingual dataset for causal commonsense reasoning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2362–2376.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Roemmele et al. (2011) Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011.  Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In _AAAI spring symposium: logical formalizations of commonsense reasoning_ , pages 90–95.
  * Roy et al. (2020) Uma Roy, Noah Constant, Rami Al-Rfou, Aditya Barua, Aaron Phillips, and Yinfei Yang. 2020.  LAReQA: Language-agnostic answer retrieval from a multilingual pool.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5919–5930, Online. Association for Computational Linguistics.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, and et al. Rush, Alexander M. 2022.  Bloom: A 176b-parameter open-access multilingual language model.
  * Shi et al. (2022) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2022.  Language models are multilingual chain-of-thought reasoners.
  * Shliazhko et al. (2022) Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2022.  mgpt: Few-shot learners go multilingual.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122.
  * Winata et al. (2021) Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, and Pascale Fung. 2021.  Language models are few-shot multilingual learners.  In _Proceedings of the 1st Workshop on Multilingual Representation Learning_ , pages 1–15, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Xue et al. (2020) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020.  mt5: A massively multilingual pre-trained text-to-text transformer.
  * Yang et al. (2019) Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019.  Paws-x: A cross-lingual adversarial dataset for paraphrase identification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3687–3692.
  * Zhu et al. (2019) Junnan Zhu, Qian Wang, Yining Wang, Yu Zhou, Jiajun Zhang, Shaonan Wang, and Chengqing Zong. 2019.  NCLS: Neural cross-lingual summarization.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3054–3064, Hong Kong, China. Association for Computational Linguistics.

##  Appendix A Data Preparation

In this section, we introduce details of data preparation of XQuAD (Artetxe et
al., 2020), where parallel few-shot examples in 11 target language variations
are added. We also illustrate a dataset pipeline to efficiently build parallel
few-shot examples.

###  A.1 Extension of XQuAD

XQuAD is a parallel multilingual extractive question answering dataset, human-
translated in 11 languages from a subset of SQuAD v1.1 (Rajpurkar et al.,
2016). SQuAD v1.1 is a crowd-sourced extractive question answering dataset to
find an answer for a question within a given passage from Wikipedia.

XQuAD consists of 1,190 queries in 11 language variations (en, ar, de, el, es,
hi, ro, ru, th, tr, vi, zh). XQuAD doesn’t have a distinct validation set,
therefore, we translate a portion of XQuAD to use as demonstration examples.
We evaluate on the XQuAD test set using our machine-translated demonstration
examples as few-shots. Instead of finding the answer span within the passage,
we evaluate the “generated” answer of the model given a passage and question
following the previous work (Shliazhko et al., 2022).

###  A.2 Dataset Pipeline

Since XQuAD (Artetxe et al., 2020) only provides test sets, we construct
parallel demonstration examples for 12 target language variations by random
selection of the SQuAD v1.1 training set. To translate target languages, we
build a dataset pipleline to obtain a high-quality machine-translated dataset.

Our dataset construction procedure can be divided into five steps, as follows.

  * •

(Step 1) Randomly select the subset from SQuAD v1.1 training dataset and
translate it using a machine translation system.

  * •

(Step 2) Extract quality estimation candidates, context (passages) and
questions.

  * •

(Step 3) Select high-quality candidates applying round-trip translation
strategy and place examples into the bucket candidate pool.

  * •

(Step 4) Filter out duplicate questions and answers not included in context.

First, we translate the data instances from random sampled SQuAD v1.1 training
dataset to 11 target languages using the machine translation system, Google
Translate API. To handle the data quality issue from using automatic
translation system, we adopt a quality estimation process using the round-trip
translation (RTT) strategy. RTT, also known as bi-directional translation, is
the process of first translating into the target language (Trans1) and then
translating into the source language (Trans2).

Next, we extract context and questions for quality control candidates. We
calculate the BLEU scores per candidate between Trans1 and Trans2 in a similar
way that the cross-lingual summarization dataset is constructed (Zhu et al.,
2019). Subsequently, we construct a candidate pool by dropping data instances
that do not reach our threshold (BLEU Papineni et al. (2002) score $\geq$ 50).

From the candidate pool, we apply three filtering rules to refine a parallel
extractive QA dataset; (1) matching unique id values for all language
variations to be parallel; (2) checking if the answer is in the context; and
(3) removing examples when it contains duplicate questions. Finally, we
randomly select a number of demonstration examples per target language during
in-context learning.

##  Appendix B Pre-training Corpus

In Section 4, we analyzed that the cross-lingual transfer gap differs whether
the language is seen or unseen during pre-training stage. Accordingly, we
report the composition of the pre-training corpora of multilingual models in
detail. BLOOM (Scao et al., 2022) is trained on the ROOTS (Laurençon et al.,
2022) corpus, a composite collection of 1.61 TB of text consisting of 46
languages and 13 programming languages. XGLM (Lin et al., 2021) is trained on
the CC100 (Common Crawl) corpus, including 30 different languages. The
languages and their size (in GiB) in the pre-training corpus is shown in Table
5 and Table 6.

Language | ISO code | Size (GiB)
---|---|---
Akan | ak | 0.00070
Arabic | ar | 74.85
Assamese | as | 0.29
Bambara | bm | 0.00039
Basque | eu | 2.36
Bengali | bn | 18.61
Catalan | ca | 17.79
Chichewa | ny | 0.012
chiShona | sn | 0.066
Chitumbuka | tum | 0.00017
English | en | 484.95
Fon | fon | 0.025
French | fr | 208.24
Gujarati | gu | 1.20
Hindi | hi | 24.62
Igbo | ig | 0.014
Indonesian | id | 19.97
isiXhosa | xh | 0.14
isiZulu | zu | 0.085
Kannada | kn | 2.10
Kikuyu | ki | 0.00036
Kinyarwanda | rw | 0.40
Kirundi | rn | 0.033
Lingala | ln | 0.017
Luganda | lg | 0.047
Malayalam | ml | 3.66
Marathi | mr | 1.78
Nepali | ne | 2.44
Northern Sotho | nso | 0.018
Odia | or | 1.16
Portuguese | pt | 79.28
Punjabi | pa | 1.57
Sesotho | st | 0.00075
Setswana | tn | 0.0015
Simplified Chinese | zhs | 261.02
Spanish | es | 175.10
Swahili | sw | 0.24
Tamil | ta | 7.99
Telugu | te | 2.99
Traditional Chinese | zht | 0.76
Twi | tw | 0.0013
Urdu | ur | 2.78
Vietnamese | vi | 43.71
Wolof | wo | 0.0036
Xitsonga | ts | 0.00071
Yoruba | yo | 0.90
Programming Languages | - | 174.70
Table 5: Languages in the pre-training corpus of BLOOM. Language | ISO code | Size (GiB) |  | Language | ISO code | Size (GiB)
---|---|---|---|---|---|---
Afrikaans | af | 3.04 |  | Lao | lo | 0.59
Albanian | sq | 8.76 |  | Latin | la | 2.20
Amharic | am | 0.85 |  | Latvian | lv | 10.67
Arabic | ar | 64.34 |  | Lingala | ln | 0.02
Arabic Romanized | - | 1.65 |  | Lithuanian | lt | 18.38
Armenian | hy | 7.16 |  | Macedonian | mk | 6.05
Assamese | as | 0.10 |  | Malagasy | mg | 0.38
Azerbaijani | az | 8.41 |  | Malay | ms | 38.57
Bambara | bm | 0.02 |  | Malayalam | ml | 8.08
Basque | eu | 0.35 |  | Marathi | mr | 3.59
Belarusian | be | 5.81 |  | Mongolian | mn | 4.26
Bengali | bn | 11.19 |  | Nepali | ne | 5.32
Bengali Romanized | - | 0.48 |  | Norwegian | no | 70.89
Bosnian | bs | 4.00 |  | Oriya | or | 0.62
Breton | br | 0.16 |  | Oromo | om | 0.09
Bulgarian | bg | 61.10 |  | Pashto | ps | 1.97
Burmese | my | 0.74 |  | Persian | fa | 57.46
Catalan | ca | 26.90 |  | Polish | pl | 73.59
Chinese | zh | 485.32 |  | Portuguese | pt | 147.12
Croatian | hr | 54.27 |  | Punjabi | pa | 0.93
Czech | xs | 32.46 |  | Quechua | qu | 0.01
Danish | da | 64.74 |  | Romanian | ro | 93.63
Dutch | nl | 68.36 |  | Russian | ru | 1007.38
English | en | 3324.45 |  | Sanskrit | sa | 0.42
Esperanto | eo | 0.14 |  | Scottish Gaelic | gd | 0.05
Estonian | et | 12.18 |  | Serbian | sr | 12.68
Finnish | fi | 67.28 |  | Sindhi | sd | 0.04
French | fr | 303.76 |  | Sinhalese | si | 3.96
Fula | ff | 0.03 |  | Slovak | sk | 30.70
Galician | gl | 0.50 |  | Slovene | sl | 14.97
Ganda | lg | 0.05 |  | Somali | so | 1.56
Georgian | ka | 10.48 |  | Spanish | es | 363.83
German | de | 369.60 |  | Sundanese | su | 0.30
Greek | el | 180.37 |  | Swahili | sw | 3.19
Guaraní | gn | 0.03 |  | Swati | ss | 0.01
Gujarati | gu | 2.10 |  | Swedish | sv | 36.54
Haitian | ht | 0.28 |  | Tagalog | tl | 8.13
Hausa | ha | 0.42 |  | Tamil | ta | 12.36
Hebrew | he | 42.13 |  | Tamil Romanized | - | 0.13
Hindi | hi | 26.63 |  | Telugu | te | 5.28
Hindi Romanized | - | 6.60 |  | Telugu Romanized | - | 0.11
Hungarian | hu | 89.87 |  | Thai | th | 99.86
Icelandic | is | 4.21 |  | Tigrinya | ti | 0.19
Igbo | ig | 0.06 |  | Tswana | tn | 0.03
Indonesian | id | 67.51 |  | Turkish | tr | 51.51
Inuktitut | iu | 0.03 |  | Ukrainian | uk | 156.68
irish | ga | 0.15 |  | Unassigned | - | 2.30
Italian | it | 170.76 |  | Urdu | ur | 7.77
Japanese | ja | 293.39 |  | Urdu Romanized | - | 0.73
Javanese | jv | 1.23 |  | Uzbek | uz | 1.64
Kannada | kn | 3.41 |  | Vietnamese | vi | 50.45
Kazakh | kk | 8.40 |  | Welsh | cy | 1.27
Khmer | km | 2.14 |  | Western Frisian | fy | 0.04
Kongo | kg | 0.01 |  | Wolof | wo | 0.03
Korean | ko | 79.08 |  | Xhosa | xh | 0.09
Kurdish | ku | 0.10 |  | Yoruba | yo | 0.14
Kyregyz | ky | 0.34 |  | Zulu | zu | 0.14
Table 6: Languages in the pre-training corpus of XGLM.

##  Appendix C Demonstration Examples Templates

We describe the QA formatted templates used for demonstration example
construction in Table 7. We adopt prompt template from existed works
(Muennighoff et al., 2022; Shi et al., 2022). For In-CLT prompt, the question-
answer pair are written the target language including attributes such as
"Question:" and the passage part is written in the source language.

Tasks Prompt Template MLQA, XQuAD Passage: {context} Question: {question}
Answer: PAWS-X Sentence 1: {sentence1} Sentence 2: sentence2 Question: Can we
rewrite Sentence 1 to Sentence 2? Yes or No? Answer: XNLI Premise: {premise}
Question: {hypothesis} True, False, or Neither? Answer: XCOPA Premise:
{premise} Question: What was the question? {% if question == "cause" %} What
happened as a result? Options -{choice1} -{choice2} Answer:

Table 7: Prompt template to construct demonstration examples for each
multilingual tasks. Boldface represents the component which is constantly
written in the source language. Other components are written in the target
language.