# Breaking Language Barriers with a LEAP:
Learning Strategies for Polyglot LLMs

Akshay Nambi&Vaibhav Balloli &Mercy Ranjit&Tanuja GanuKabir Ahuja&Sunayana
Sitaram&Kalika BaliMicrosoft Research India
<EMAIL_ADDRESS>
###### Abstract

Large language models (LLMs) are at the forefront of transforming numerous
domains globally. However, their inclusivity and effectiveness remain limited
for non-Latin scripts and low-resource languages. This paper tackles the
imperative challenge of enhancing the multilingual performance of LLMs,
specifically focusing on Generative models. Through systematic investigation
and evaluation of diverse languages using popular question-answering (QA)
datasets, we present novel techniques that unlock the true potential of LLMs
in a polyglot landscape. Our approach encompasses three key strategies that
yield remarkable improvements in multilingual proficiency. First, by
meticulously optimizing prompts tailored for polyglot LLMs, we unlock their
latent capabilities, resulting in substantial performance boosts across
languages. Second, we introduce a new hybrid approach that synergizes GPT
generation with multilingual embeddings and achieves significant multilingual
performance improvement on critical tasks like QA and retrieval. Finally, to
further propel the performance of polyglot LLMs, we introduce a novel learning
algorithm that dynamically selects the optimal prompt strategy, LLM model, and
embeddings per query. This dynamic adaptation maximizes the efficacy of LLMs
across languages, outperforming best static and random strategies. Our results
show substantial advancements in multilingual understanding and generation
across a diverse range of languages.

##  1 Introduction

Large Language Models (LLMs) such as ChatGPT and GPT-4 have revolutionized AI
research with their groundbreaking advancements in recent years
DBLP:journals/corr/abs-2005-14165 ; ouyang2022training ; openai2023gpt4 .
These LLMs, despite being language-focused models, exhibit extraordinary
capabilities across diverse tasks, ranging from abstraction, comprehension to
vision and coding DBLP:journals/corr/abs-2107-03374 ; ramesh2022hierarchical ;
DBLP:journals/corr/abs-2005-14165 .

The transformative potential of LLMs is revolutionizing the user experience in
diverse domains, including education and agriculture, as they act as
intelligent co-pilots, reshaping teaching methods, farming practices, and
several such professional interactions with AI-powered systems liu2023summary
. However, the current landscape predominantly favors LLMs optimized for
English and Latin script languages mega , limiting their effectiveness and
widespread applicability to non-English languages. This discrepancy inhibits
the realization of their full potential, hampering inclusivity in the global
adoption of these models huang2023languages . Recent studies have attempted
comprehensive evaluations of LLMs on multilingual datasets covering a wide
range of tasks mega , revealing a notable performance gap between LLMs such as
GPT-3 and state-of-the-art multilingual models like TULRv6 patra2022beyond .
For instance, Figure 1 illustrates the performance comparison of GPT-3 (1) and
GPT3.5 Turbo (2) against SOTA models (3 and 4) on popular multilingual
Question Answering (QA) datasets, namely IndicQA ai4bharat2022indicqa and
TyDiQA tydiqa . It is evident that while GPT3.5 Turbo demonstrates improved
performance compared to its predecessor GPT-3, there still exists a
substantial gap when compared to SOTA models.

Figure 1: Performance comparison of GPT-x models against SOTA Multilingual
models. 1:GPT-3, 2:GPT3.5 Turbo, 3:Muril(IndicQA), XLMR(TyDiQA),
4:IndicBert(IndicQA), TuLR(TyDiQA).

Due to this, the question at hand is how to improve the performance of
generative models on non-English languages? This poses significant challenges,
including: 1\. Scarcity of Training Data: The availability of high-quality
labeled data is limited for languages other than English and a few high-
resource languages, impeding optimal pre-training of multilingual LLMs
hämmerl2022multilingual ; DBLP:journals/corr/abs-2004-06748 . 2\. Prompt
Tuning Sensitivity: Tuning prompts for different languages and tasks is
laborious and hard to scale tu2022prompttuning ; huang2022zeroshot . 3\. Lack
of access and resources to pre-train and fine-tune models: The lack of open-
source availability for some existing models and the computational cost of
training and fine-tuning these models hinders customization for specific
languages.

Recently, there has been considerable interest in exploring prompting
strategies for multilingual LLMs. lin2021few explore various prompting
strategies such as handcrafted native language prompts, translated prompts and
cross-lingual prompting across 7 multilingual datasets. fu2022polyglot explore
native language and cross-lingual prompting across 6 tasks covering 24
datasets and find that cross-lingual prompting, in which the prompt
instruction is provided in English leads to highest mT5 performance.
chen2022multilingual investigates various prompting choices including hard-vs-
soft prompting, in-language vs. cross-lingual and the word order in prompts
for relation classification. zhao2021discrete also explore discrete, soft and
mixed prompting for multilingual Natural Language Inference datasets and find
that prompting outperforms fine-tuning the XLMR model
DBLP:journals/corr/abs-1911-02116 for these tasks. Recently, there has been
interest in exploring Chain-of-Thought weichain ; shi2022language ;
huang2023not prompting for multilingual scenarios, which leads to superior
performance across reasoning tasks even for low-resource languages. Despite
all this recent interest in multilingual prompting, there is no definitive
prompting strategy that seems to perform consistently and significantly better
than others across tasks and languages, which serves as the motivation for our
work.

Our work, LEAP (Learning Strategies for Polyglot LLMs), focuses on enhancing
the multilingual performance of Black-box LLMs through three key techniques:

  1. 1.

Optimizing Prompts for Polyglot LLMs: We focus on tailoring prompts for
polyglot LLMs. By carefully crafting prompts that leverage the unique
characteristics and capabilities of LLMs, we tap into their latent potential
and unlock substantial performance improvements across languages.

  2. 2.

Hybrid Approach with GPT Generation and Multilingual Embeddings: Synergizing
GPT generation with multilingual embeddings excels in generating coherent and
contextually relevant text and enables a deeper understanding of cross-
linguistic relationships and context, enhancing multilingual performance in
tasks like question answering and retrieval.

  3. 3.

Dynamic Learning Algorithm for Performance Optimization: Our learning
algorithm dynamically selects the optimal prompt strategy, LLM model, and
multilingual embeddings based on real-time human feedback and system
evaluation metrics, maximizing efficacy across languages and outperforming
static and random strategies. This dynamic adaptation ensures continuous
optimization for each language, leading to significant advancements in
multilingual performance.

We evaluated the effectiveness of the aforementioned techniques on two widely
used Question Answering (QA) datasets: IndicQA and TyDiQA, spanning across 18
languages. Furthermore, we show that current system evaluation metrics are not
representative of task performance and present challenges and insights towards
addressing this. Through extensive experimentation and analysis, our results
demonstrate over 15-20% improvement in multilingual performance across these
diverse languages. This showcases the effectiveness and versatility of our
approach, which is extensible to any of the other prompting techniques
described in the literature.

##  2 Experimental Setup

In this work, we focus on QA tasks as they showcase the model’s ability to
provide accurate and informative responses to user queries. By excelling at QA
tasks, LLMs can be more valuable in real-world applications such as
information retrieval, virtual assistants, and so on.

###  2.1 Dataset

Table 1: Datasets

IndicQA | TyDiQA
---|---
Lang | # Q | Lang | # Q
as | 1789 | bn | 180
bn | 1763 | te | 874
gu | 2017 | fi | 1031
hi | 1547 | ko | 414
kn | 1517 | ru | 1079
ml | 1589 | ar | 1314
mr | 1604 | en | 654
or | 1680 | id | 773
pa | 1542 | sw | 596
ta | 1804 |  |
te | 1734 |  |

We employ two popular multilingual QA datasets (see Table 1), with the task
being: Given a passage that is guaranteed to contain the answer, predict the
single contiguous span of characters that answers the question.

1.IndicQA ai4bharat2022indicqa : This is a manually curated QA dataset in 11
Indic languages. The context paragraphs are chosen from Wikipedia articles
whose topics are closely related to Indic culture, history, etc. The dataset
consists of 18K+ questions across all 11 languages.

2.TyDiQA tydiqa : This QA dataset covers 9 typologically diverse languages.
The task consists of two sub-tasks - passage selection and minimum answer span
(Gold-P). For our experiments, we consider the Gold-P task, where only the
gold answer passage is provided rather than the entire Wikipedia article.

###  2.2 Evaluation Metrics for Multilingual QA Task

F1 score is the commonly used metric in QA tasks
DBLP:journals/corr/RajpurkarZLL16 . It is computed over the individual words
in the prediction against those in the True Answer (Ground Truth). SQuAD-F1
evaluation DBLP:journals/corr/RajpurkarZLL16 is a widely used approach to
calculate F1 score that performs answer-preprocessing such as lowercasing,
stripping punctuation, stripping English articles, and white space
tokenization. However, for fairer multilingual evaluation there is additional
preprocessing required as described by MLQA-F1
DBLP:journals/corr/abs-1910-07475 such as stripping all unicode punctuations,
stand-alone articles, etc. Our detailed evaluation shows that MLQA-F1 is much
closer to human feedback than SQuAD-F1 for multilingual evaluation (see
Section 5). Hence we use MLQA-F1 as our system evaluation metric.

###  2.3 GPT-x models

We conduct experiments on the following OpenAI GPT models: GPT-3 (text-
davinci-003) azureopenaigptthree , GPT3.5 Turbo azureopenaiturbo (same as
ChatGPT), which are available via API access through the Azure OpenAI service.
As the access to GPT-4 azurepoenaigptfour was limited and responses were
subject to significant delays, our evaluation and results are focused on the
above two models. However, our limited experiments with GPT-4 show that the
strategies proposed in our work can be applied to newer GPT-x models and other
LLMs to achieve similar improvements in multilingual performance. We use
temperature=0, and max_tokens=2048 across all our experiments.

##  3 Prompt Strategies for Polyglot LLMs

Performance of generative models can be very sensitive to prompt tuning and
from prior works mega , we find that this is the case even with non-English
languages. Through extensive experimentation and analysis, we carefully design
few prompt templates optimized for polyglot LLMs.

Our notation is as follows: Each prompt consists of an Instruction and
Context, where contexts are passages based on which questions need to be
answered in case of QA. We use two approaches - zero-shot, where we do not
pass any examples and few-shot, where we pass few examples based on the
strategy being used. We refer to the language that the query is issued in as
the source language, which is also the language that we want to provide the
final response in. We use the following prompting strategies in our work:

1\. Monolingual (Mono): In this technique, the Instruction and Context are in
the source language. Few-shot examples are also from the source language.

2\. Translate-Test (Trans): This approach translates (via an automatic Machine
Translation system azuretranslate ) the Instruction and Context into English,
queries the model in English and back-translates the result into the source
language (Roundtripping through English).

3\. Similar high-resourced language (Sim): This is the same translate-test
except that roundtripping is through another pivot language. This is a viable
approach because newer models perform well on many high-resource languages. We
select the pivot language such that it is closer to the source language with
respect to language feature similaritiesmalaviya17emnlp captured in lang2vec
littell2017uriel . We constrain the pivot language to be medium-high resource
language belonging to the language class joshi-etal-2020-state of 3 to 5.
Refer to supplementary material for additional details.

4\. Aggregation Source (Agg_Src): We introduce a new prompting strategy that
aggregates the responses of different prompting strategies (listed above).
Although this results in more calls to the LLM, it can be useful to exploit
the strengths of different prompting strategies and information contained in
different languages into a single response. In Agg_Src, we aggregate the
responses from Mono and Trans and Sim into the source native language and
return the best aggregated response.

5\. Aggregation Translate (Agg_Trans) This is similar to Agg_Src, except that
we aggregate the responses from Mono, Trans, and Sim into English and then
translate it into the source native language using an Machine Translation
system.

Figure 2: Prompt Strategies: IndicQA.

Models | GPT3-DV003 | GPT3.5-Turbo
---|---|---
Lang. | Mono | Trans | Sim |  | Agg
---
Trans
| Agg
---
Src
Mono | Trans | Sim |  | Agg
---
Trans
| Agg
---
Src
as | 0.30 | 0.33 | 0.26 | 0.35 | 0.31 | 0.43 | 0.35 | 0.32 | 0.34 | 0.40
bn | 0.32 | 0.34 | 0.22 | 0.35 | 0.34 | 0.45 | 0.36 | 0.29 | 0.35 | 0.46
gu | 0.24 | 0.31 | 0.22 | 0.32 | 0.24 | 0.49 | 0.34 | 0.30 | 0.34 | 0.47
hi | 0.35 | 0.51 | 0.23 | 0.49 | 0.43 | 0.54 | 0.53 | 0.38 | 0.52 | 0.58
kn | 0.23 | 0.30 | 0.11 | 0.29 | 0.27 | 0.46 | 0.29 | 0.20 | 0.28 | 0.43
ml | 0.11 | 0.31 | 0.05 | 0.30 | 0.10 | 0.18 | 0.30 | 0.10 | 0.27 | 0.22
mr | 0.28 | 0.32 | 0.21 | 0.31 | 0.24 | 0.49 | 0.34 | 0.28 | 0.35 | 0.44
or | 0.24 | 0.28 | 0.24 | 0.28 | 0.23 | 0.52 | 0.56 | 0.30 | 0.34 | 0.42
pa | 0.32 | 0.40 | 0.26 | 0.37 | 0.38 | 0.37 | 0.29 | 0.24 | 0.39 | 0.33
ta | 0.12 | 0.41 | NA | 0.40 | 0.16 | 0.17 | 0.40 | NA | 0.37 | 0.22
te | 0.09 | 0.36 | 0.06 | 0.35 | 0.11 | 0.18 | 0.36 | 0.08 | 0.33 | 0.15
freq | 5.9% | 27.8% | 41.0% | 24.7% | 0.9% | 30.1% | 23.6% | 12.9% | 18.7% | 14.5%

Figure 3: Prompt Strategies: TyDiQA.

Models | GPT3.5-Turbo
---|---
Lang. | Mono | Trans | Sim |  | Agg
---
Trans
| Agg
---
Src
bn | 0.61 | 0.40 | 0.37 | 0.39 | 0.62
te | 0.57 | 0.42 | 0.20 | 0.40 | 0.56
fi | 0.66 | 0.51 | 0.51 | 0.50 | 0.58
ko | 0.63 | 0.46 | 0.43 | 0.49 | 0.64
ru | 0.52 | 0.45 | 0.43 | 0.43 | 0.48
ar | 0.56 | 0.45 | NA | 0.40 | 0.50
en | 0.66 | NA | 0.62 | 0.62 | 0.62
id | 0.67 | 0.60 | 0.52 | 0.54 | 0.63
sw | 0.69 | 0.59 | NA | 0.58 | 0.72
freq | 39.6% | 19.4% | 17.9% | 10.9% | 12.3%

To summarize, we use the following prompting strategies in our work: Mono,
Trans, Sim, Agg_Src, Agg_Trans. While we utilize these 5 strategies, our
learning framework (Section 6) can incorporate any other prompting strategies
described previously.

Prompting Strategies Results. We found that few-shot examples improve the
performance of GPT-x models compared to zero-shot. Hence, we present only few-
shot results going forward. Refer to supplementary material for additional
details on zero-shot results. Table 3 shows the results of the individual
prompting strategies for each language for the IndicQA dataset for the GPT-3
and GPT3.5 Turbo models using few-shot examples. We can see that for GPT-3
model, the Agg_Trans strategy outperforms other strategies for some languages,
with the Trans strategy being the best for other languages. For GPT3.5 Turbo,
we see that the Mono strategy is preferred for many languages, with the Trans
strategy being best for the remaining languages. For Punjabi (pa), we find
that the Agg_Trans strategy works best for GPT3.5 Turbo. Given the significant
performance improvement from GPT-3 to GPT3.5 Turbo, we restrict our results to
GPT3.5 Turbo in the reminder of the paper. For TyDiQA, we find that the Mono
strategy is preferred for most languages except Bengali (bn) and Korean (ko),
for which the Agg_Src is best. Finally, the table’s final row presents the
frequency distributions indicating which strategy yielded the highest F1 score
across all languages. In general, we find that for the GPT3.5 Turbo model,
monolingual performance is better than for GPT-3.

Summary: There is no single best strategy that works best across datasets,
models and languages, thus necessitating a learning algorithm that can
determine the best strategy to maximize performance.

##  4 Hybrid approach

Presently, the majority of LLMs are predominantly trained using English and a
limited selection of other high-resource languages. Consequently, these models
exhibit a deficiency in comprehending the nuances of medium and low-resource
languages. For many real-world tasks, such as question answering and
summarization, LLMs need to include non-parametric knowledge from a private
knowledge-base or contextual information alongside the input query for their
synthesis. Retrieval Augmented Generation (RAG)
DBLP:journals/corr/abs-2005-11401 is employed to first retrieve the documents
or chunks of information which are most relevant for the input query, followed
by synthesis using the retrieved information. This process involves the
following steps- 1) encoding the knowledge-base documents/context using a text
embedding model and indexing the embeddings for fast retrieval, 2) encoding
the query using the same text embedding model, 3) conducting a similarity
search between the query embedding and the document embeddings indexed in step
1, with the top $k$ similar documents retrieved for further synthesis, and 4)
forwarding the retrieved $k$ documents/chunks to the LLM along with the
original query for response synthesis.

(a) Max gain across strategies.

(b) Freq. distribution for all strategies.

(c) Overall Freq. distribution.

Figure 4: Hybrid approach Performance analysis on IndicQA and TyDiQA datasets.
Strategies- 1: Mono, 2: Trans, 3: Sim, 4: Agg_trans, 5: Agg_Src. Embeddings-
GPT-Emd: Ada embedding and ML-Emd: SOTA Multilingual embeddings.

OpenAI offers a general-purpose text embedding model, namely text-
embedding-002 azureopenaiadaembedding , (referred as GPT-emb) that serves as
the default embedding model for GPT text generation. Nevertheless, there
exists a substantial collection of multilingual embedding models, such as
MuRIL: Multilingual Representations for Indian Languages
DBLP:journals/corr/abs-2103-10730 , Multilingual Universal Sentence Encoders-
distiluse-base-multilingual-cased-v2/v1 DBLP:journals/corr/abs-1803-11175 ;
DBLP:journals/corr/abs-2004-09813 , and Cohere multilingual embedding model -
multilingual-22-12 coheremultilingual . These multilingual models facilitate
the mapping of text to a semantic vector space, or "embeddings", while
positioning semantically similar texts across various languages in close
proximity. In order to address multilingual queries effectively, polyglot LLMs
must be capable of responding using either their intrinsic knowledge or
multilingual context information supplied in a retrieval-augmented setting.
When reasoning within a multilingual embedding space, polyglot LLMs could
benefit from utilizing an embedding model proficient in the source language or
languages akin to it, rather than relying solely on default GPT-based
embeddings, particularly for languages that are inadequately represented in
the pre-trained model.

Consequently, our study aims to tackle the challenge of limited representation
of multiple languages in LLMs by combining the best of both worlds. This
hybrid approach integrates the cross-lingual semantic understanding of
multilingual embeddings with the text generation and world knowledge
understanding capabilities of LLMs. To achieve this, we employ the LlamaIndex
(GPTIndex) llama framework, which provides inter-operability and incorporates
various multilingual embedding models for encoding and indexing the context
within the given question-answering task setup. We utilize the MuRIL model,
specifically designed for Indic languages for the evaluation on the IndicQA
dataset, alongside XLMR DBLP:journals/corr/abs-1911-02116 embeddings for the
TyDiQA dataset comprising other global languages. These are referred as
multilingual embeddings (ML-emb). Our hybrid approach also proves beneficial
when used in conjunction with other prompt strategies previously discussed,
such as translating to similar languages or aggregating answers across
multiple languages, owing to the cross-lingual semantic understanding
facilitated by multilingual embeddings.

Figure 4 demonstrates the effectiveness of our hybrid approach. Figure 4 shows
the maximum gains in MLQA-F1 scores across languages by using multilingual
embeddings- ML-Emb (MuRIL for IndicQA and XLM-R for TyDiQA) over the MLQA-F1
scores of default GPT embeddings - GPT-emb. This shows that multilingual
embeddings provide positive gains in MLQA-F1 scores for both the datasets and
across all five strategies. Figure 4 shows the frequency distribution of how
many times a combined strategy $(x,y)$ of using an embedding $x$ and a
prompting strategy $y$ (discussed in Section 3) has given the best MLQA-F1
score for IndicQA and TyDiQA datasets. We can observe that incorporating
multilingual embeddings provide better results compared to GPT embeddings and
is generally preferred more (high frequency distribution compared to GPT-emd).
Finally, Figure 4 shows the frequency distribution of how many times the
multilingual embedding (combined with one of the best prompt strategies)
across all languages outperforms GPT embedding approach evaluated on MLQA-F1
score metric for both the datasets. The Hybrid approach of incorporating
multilingual embeddings clearly wins on both the datasets with 75% frequency.
The average MLQA-F1 score improvement using multilingual embeddings across all
languages in IndicQA dataset is 13% and in TyDiQA dataset is 8% . Further,
note that yet there is no single best strategy that works best across
datasets, models and languages.

Summary: Our novel hybrid approach, combining GPT generation and multilingual
embeddings with Retrieval Augmented Generation setting, results in an average
improvement of 10% in multilingual MLQA-F1 performance, with a maximum gain of
25%.

##  5 Are current evaluation metrics representative of the task performance?

In Section 2.2, we described the usage the MLQA-F1 metric (System evaluation
metric) instead of the commonly used SQuAD-F1 metric for fairer and more
accurate evaluation of multilingual QA tasks. Our evaluation revealed an
approximate 10% error in SQuAD-F1 scores when compared to MLQA-F1 scores
across the IndicQA and TyDiQA datasets. As we rely on these metrics to select
the optimal prompt strategy and embeddings model for a given language, it
becomes crucial to employ representative metrics that aid in the selection of
appropriate strategies.

Comparison with human annotations (HA). To assess the accuracy and quality of
current system evaluation metrics, i.e., MLQA-F1, we conducted a comparison
with human annotations score (HA), as the ultimate aim of GPT-x model
deployments is to cater to users worldwide. We randomly selected 100 questions
from the IndicQA dataset across six languages (bn, hi, ta, kn, mr, gu) and
presented them to native speakers, who are part of our research group. Each
annotator received the context paragraph, along with the question and the
ground truth (GT) answer. For each question, the annotator was then presented
with 10 predicted answers generated using different strategies (5 strategies
for each of GPT and Multilingual embeddings). The annotator had the options to
select YES if the predicted answer completely matched the GT answer (i.e.,
Score = 1), NO if there was no match (i.e., Score = 0), and PARTIAL if the
predicted answer partially matched the GT answer (i.e., Score = 0.5 or F1
against GT). This metric we call as Human Annotator (HA) score. Refer to
supplementary material for additional details on the user interface used to
collect the annotations.

Figure 5: Error is metrics when HA score is compared with MLQA-F1 and
GPTAnnotator.

| F1 computation
---|---
Lg. |  MLQA F1 |  GPTAnno |  HA
kn |  Mono-ML |  Mono-GPT |  Mono-GPT
gu |  Mono-GPT |  Trans-GPT |  Trans-GPT
mr |  Mono-ML |  Mono-ML |  Mono-ML
bn |  Mono-ML |  AggTrans-GPT |  AggTrans-GPT
hi |  AggTrans-GPT |  Trans-GPT |  Trans-GPT
ta |  Mono-ML |  Trans-GPT |  Trans-GPT

Figure 6: Best Strategy selected based on different metrics

Figure 7: Examples showing system evaluation metrics are not representative of
task performance.

Figure 6 shows the error between human annotations score and MLQA-F1 scores
for the six languages across all strategies (represented by pink-colored box
plots). The MLQA-F1 scores on these 100 samples (6 languages) exhibited an
average error of 25% and a maximum error of 51%, which is notably significant.
This error is mainly due to the variations in the generated answer by the
GPT-x models. Figure 7 presents a snapshot of the performance of GPT3.5 Turbo
on the IndicQA dataset for Kannada (kn) and Marathi (mr) languages. Each
question is accompanied by the ground truth (GT) answer and the generated
results from different strategies. In the left example for Kannada, although
the generated answers from the models are not identical, they are still
factually accurate, providing an alternative way to refer to "British".
Similarly, in the right example, the answer is an abbreviation for an
organization, while the generated answers consist of a combination of
abbreviated and fully expanded versions. However, due to the word-level
matching nature of the system evaluation metric (MLQA-F1), the MLQA-F1 scores
for these cases would be exceptionally low, and even zero, whereas HA score
would be close to 1 as all of them are correct answers. These examples
demonstrate the limitations of relying solely on system evaluation metrics for
capturing the quality and correctness of generated answers especially using
Generative models. Hence, the question arises: Are the current system
evaluation metrics truly indicative of task performance? should we define a
new evaluation metric for different tasks using generative models?

In order to obtain an accurate assessment of model performance, it is
essential to evaluate task performance against human annotations (HA), which
provide the closest approximation to ground truth. However, obtaining HA
scores for all tasks and languages is impractical. Moreover, relying solely on
system metrics like MLQA-F1 is problematic, as there exists a notable
disparity between MLQA-F1 and HA scores.

Can GPT-x model be utilized as a human annotator? In this study, we
investigate the potential of employing a GPTAnnotator, where a GPT3.5 Turbo
model is utilized to validate the answers generated by another GPT3.5 Turbo
model. Specifically, we use a second LLM model to verify and annotate the
output of the first model as either YES/NO/PARTIAL (similar to what human
annotators did as described previously). Refer to supplementary material for
additional details on the prompts used by the second LLM model for
annotations. Figure 6 shows the error between human annotations score and
GPTAnnotator score for the six languages across all strategies (represented by
blue-colored box plots). Notably, the GPTAnnotator errors are significantly
lower compared to MLQA-F1 errors when compared to HA scores. Additionally,
across all languages, GPTAnnotator scores reduces the error by 30% compared to
MLQA-F1 scores and is much closer to HA scores.

Furthermore, due to the disparity in these scores, the optimal strategy for
each language varies depending on the metric used for computation, namely
MLQA-F1, GPTAnnotator score, or HA score. Table 6 displays the best strategy
chosen per language based on the MLQA-F1, GPTAnnotator, and HA scores. It is
evident that the best strategies identified using MLQA-F1 scores differ
significantly from those determined by HA. Conversely, the best strategies
derived from GPTAnnotator align with the best strategies determined by HA.
Thus suggesting that GPTAnnotator F1 score can effectively guide the selection
of optimal strategies.

Summary: Our analysis reveals a fundamental limitation in the current system
evaluation metrics for generative models, highlighting the urgent need for new
evaluation metrics. We propose GPTAnnotator, a powerful solution to address
this challenge and validate the quality of answers.

##  6 Learning Algorithms

Thus far, we have demonstrated prompting strategies and novel Hybrid Approach
have significantly improved multilingual performance. The next key
contribution of LEAP is determining the optimal approach and strategy for
different queries in each language.

As demonstrated in Section 3 and Section 4, it is evident that a one-size-
fits-all approach does not exist when it comes to selecting the optimal
strategy for diverse multilingual datasets, models, and languages. This leads
us to the crucial question: How can we dynamically determine the most
appropriate strategy for each query, ensuring maximum multilingual
performance?

Formulation. We frame our strategy selection problem as a learning problem,
where a learning agent is tasked with choosing an optimal strategy. In this
scenario, the agent’s objective is to select the most suitable approach and
prompt strategy, and subsequently receive rewards based on the quality of the
chosen combination. These rewards serve as valuable feedback for the agent to
learn and improve its decision-making capabilities.

The action space available to the agent encompasses all possible combinations
of prompting strategies and embedding approaches. The available choices
consist of five prompting strategies (Mono, Trans, Sim, Agg_Trans, Agg_Src)
and two embedding approaches (GPT-emd, ML-emd), resulting in a total of ten
choices. It is worth noting that while additional dimensions, such as the type
of LLM model (GPT-3, GPT3.5 Turbo, GPT-4) or temperature setting (ranging from
0 to 1), can be introduced, we have focused on the GPT3.5 Turbo model with a
fixed temperature for the sake of simplicity. However, it should be emphasized
that the learning algorithms presented in this work are generic and can be
easily extended to accommodate additional dimensions. Furthermore, our
experimentation involves exploring various metrics, including MLQA-F1,
GPTAnnotator score, and HA score (discussed in Section 5), which serve as
rewards in our learning algorithm.

###  6.1 Multi-armed Bandits (MAB)

We first employ MAB as a learning agent to dynamically select the most
effective strategy per query. The different strategies and embeddings can be
considered as the "arms" of the bandit, and the goal is to identify the arm
that yields the highest reward or performance for each query. The algorithm
starts with an initial exploration phase and then shifts towards exploiting
the best-performing strategies while still exploring other options. This
approach allows us to continuously improve the performance.

We employ two popular algorithms used in MAB setting, (i) $\epsilon$-Greedy:
It is simple yet effective algorithm, that works by selecting the best-known
strategy with a probability of 1-$\epsilon$, where $\epsilon$ is a small
constant. $\epsilon$-Greedy is simpler to implement and has a more
deterministic exploration strategy. (ii) Thompson sampling graepel2010web : It
takes a Bayesian approach to decision-making and maintains a belief
distribution over the performance of each strategy. Thompson sampling samples
a strategy from these distributions and selects the one with the highest
sampled value. The rewards received after choosing a particular arm are the
metrics as defined in Section 5, i.e., MLQA-F1 and GPTAnnotator scores.

Train-Test Data. In all the following approaches, we maintain a consistent
experiment pipeline by splitting the available data ($\textstyle\chi$) into
${\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}_{train}$
for training and
${\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}_{test}$
for evaluation, using a 75:25 split unless stated otherwise. The reported
numbers represent averages from three independent evaluations. In addition to
the performance of the learning agents, we include baselines for the best
static choice based on the training data and a random agent that makes random
selections.

Figure 8: MAB performance.

Results. The results in Figure 8 illustrate the performance of our MAB
learning agent using two rewards (MLQA-F1 and GPTAnnotator scores) across
various strategies: best static, random, $\epsilon$-Greedy, and Thompson
sampling, applied to the IndicQA dataset across all languages. It is evident
that our learning approaches are comparable to best static strategies,
indicating limited improvement in performance. This limitation arises from our
current MAB approaches, which focuses on identifying a single best arm without
incorporating any information about the query. Ideally, a good learning agent
can perform better when given some context about the query. This design allows
us to incorporate various feedbacks that are easily available in end-user
applications like YES/PARTIAL/NO annotations from the target users to better
select our strategies. Recognizing the importance of context, we believe that
providing additional information can enable bandits to better select the right
strategies, leading to improved performance.

###  6.2 Contextual Bandits (CBs)

Figure 9: CB performance.

Our learning algorithm aims to effectively utilize feedback, including
automated metrics such as MLQA-F1 or GPTAnnotator scores, as well as human-
generated annotations (HA). Unlike traditional multi-armed bandits (MABs), we
utilize contextual bandits (CBs) agrawal2013thompson to select strategies
based on contextual information or features. CBs offer several advantages in
strategy selection: (i) they improve decision-making by continuously learning
from outcomes and refining the selection process, and (ii) they enable
adaptation to different contexts and optimize performance on a per-query
basis. Contextual information, such as the query, language, dataset, and other
relevant factors, provides valuable insights that inform the bandit
algorithm’s decision-making process.

We start by assigning a "default arm" that is always selected initially, and
the reward obtained from this arm serves as the context for the contextual
bandit algorithm to choose among the other available arms. This approach
allows us to incorporate feedback from various sources based on the specific
application. The observed reward is determined by taking the maximum of the
two rewards received (the one from the default arm and the one based on the
selection from the CB). We explore different combinations of feedback and
observed reward based on the metrics in Section 5:

  1. 1.

[CB-1] MLQA-F1: Both feedback and observed reward are based on MLQA-F1 scores.

  2. 2.

[CB-2] GPTAnnotator score: Both feedback and observed reward are based on
GPTAnnotator scores. This is because GPTAnnotator scores are closer to human
annotations (HA) than MLQA-F1 (see Figure 6) and hence should result in better
performance.

We conduct extensive experiments on both context-reward strategies mentioned
above with three evaluation metrics, i.e., MLQA-F1, GPTAnnotator, and HA
scores.

Results on IndicQA dataset. Figure 9 shows the multilingual performance
improvement for both CB-1 and CB-2 approaches against the best static (based
on training data) and random strategy. For CB-1 we use the MLQA-F1 as the
evaluation metric and for CB-2 we use GPTAnnotator score as the metric. When
we use MLQA-F1 as both feedback and reward (for the default arm and CB
selected arm), CB-1 outperforms best static and random policies by 13% and
20%, respectively. Similarly, when we use GPTAnnotator scores as feedback and
reward, we improve the absolute performance from 65% using MLQA-F1 to 82%
using GPTAnnotator score and also outperform static and random by 10% and 12%,
respectively. Table 11 shows the performance improvements of CB-1 and CB-2
across the best static and random strategies for all languages in the IndicQA
dataset.

As mentioned in Section 5, it is imperative to evaluate the learning
algorithms performance against HA scores, as they are closest to the ground
truth. Table 11 shows the performance improvement of CB-1 and CB-2 on static
and random policies for the 100 samples with HA score. We can see that both
CB-1 and CB-2 outperform the best static strategy and random using HA score as
evaluation metric by 15% and 20%, respectively. Thus, showcasing the efficacy
of our learning algorithms.

Figure 10: Performance improvement using CBs on IndicQA

| CB-1 | Random | Best Static | CB-2 | Random | Best Static
---|---|---|---|---|---|---
Lang | F1 | F1 | F1 | GPTA. | GPTA. | GPTA.
ml | 0.51 | 0.33 | 0.34 | 0.73 | 0.63 | 0.64
mr | 0.74 | 0.46 | 0.64 | 0.86 | 0.71 | 0.77
kn | 0.62 | 0.42 | 0.49 | 0.81 | 0.70 | 0.71
hi | 0.82 | 0.60 | 0.75 | 0.85 | 0.72 | 0.74
gu | 0.64 | 0.46 | 0.53 | 0.80 | 0.70 | 0.72
bn | 0.65 | 0.49 | 0.57 | 0.85 | 0.77 | 0.74
ta | 0.60 | 0.46 | 0.46 | 0.85 | 0.72 | 0.74
te | 0.61 | 0.42 | 0.44 | 0.83 | 0.71 | 0.76
avg. | 0.65 | 0.46 | 0.53 | 0.82 | 0.71 | 0.73

Figure 11: Performance improvement using CB on HA score on 100 questions

| CB-1 | Random | Best | CB-2 | Random | Best
---|---|---|---|---|---|---
|  |  | Static |  |  | Static
Lang | HA | HA | HA | HA | HA | HA
kn | 0.74 | 0.51 | 0.59 | 0.75 | 0.56 | 0.59
gu | 0.80 | 0.59 | 0.70 | 0.81 | 0.63 | 0.70
hi | 0.84 | 0.69 | 0.74 | 0.83 | 0.67 | 0.66
mr | 0.85 | 0.63 | 0.71 | 0.82 | 0.66 | 0.65
ta | 0.76 | 0.53 | 0.57 | 0.75 | 0.56 | 0.57
bn | 0.76 | 0.61 | 0.63 | 0.78 | 0.61 | 0.62
avg | 0.79 | 0.59 | 0.66 | 0.79 | 0.62 | 0.63

Results on TyDiQA dataset. Since we do not have HA scores for TyDiQA, we show
CB-1 and CB-2 performance on MLQA-F1 and GPTAnnotator scores. Figure 13 shows
the multilingual performance improvement for both CB-1 and CB-2 approaches
against the best static choice and random strategy for TyDiQA. We see that
CB-1 outperforms best static and random policies by 12% and 20%, respectively.
Similarly, CB-2 outperforms static and random by 7% and 12%, respectively.
Table 13 shows the performance improvements of CB-1 and CB-2 across the best
static and random strategy for all languages in TyDiQA dataset.

Figure 12: Performance improvement using CBs for TyDiQA.

| CB-1 | Random | Best Static | CB-2 | Random | Best Static
---|---|---|---|---|---|---
Lang | F1 | F1 | F1 | GPTA. | GPTA. | GPTA.
ar | 0.70 | 0.58 | 0.60 | 0.98 | 0.88 | 0.93
bn | 0.60 | 0.40 | 0.49 | 0.85 | 0.67 | 0.78
fi | 0.90 | 0.72 | 0.80 | 0.95 | 0.83 | 0.90
ru | 0.85 | 0.68 | 0.82 | 0.99 | 0.89 | 0.98
sw | 0.86 | 0.67 | 0.68 | 0.95 | 0.82 | 0.86
ko | 0.93 | 0.59 | 0.81 | 0.98 | 0.90 | 0.94
en | 0.64 | 0.49 | 0.46 | 0.89 | 0.76 | 0.77
te | 0.77 | 0.59 | 0.60 | 0.90 | 0.77 | 0.80
id | 0.91 | 0.73 | 0.82 | 0.97 | 0.85 | 0.93
avg | 0.80 | 0.61 | 0.68 | 0.94 | 0.82 | 0.87

Figure 13: CB performance on TyDiQA.

Summary: Our learning algorithms demonstrate the ability to identify the
optimal strategy for maximum performance on a per-query basis, spanning
various prompts and embeddings. Our agents outperforms best static and random
strategy against all metrics including HA scores. Our results shows an
impressive average multilingual performance improvement of 15-20% across
languages.

##  7 Conclusions

This work represents a significant leap forward in enhancing multilingual
performance on black-box LLMs. Our multilingual prompting strategies and
hybrid approach combining GPT generation and multilingual embeddings have
yielded substantial improvements in system metrics. The utilization of a
contextual bandit learning algorithm for strategy selection has resulted in an
average improvement of 15-20% across all languages. Additionally, the
introduction of the GPTAnnotator showcases the potential for more reliable
evaluation metrics in the future.
Future Research: While this study marks an important milestone in improving
multilingual performance, there are exciting research directions to explore.
Validating the effectiveness of the learning algorithm outputs at scale
through human feedback is a crucial step in demonstrating their real-world
impact. Incorporating human feedback to enhance the learning algorithms and
addressing sample efficiency challenges presents intriguing possibilities.
Furthermore, leveraging the insights and data obtained from this work to fine-
tune open LLMs models can further elevate multilingual performance.
Limitations and Broader Research: It is important to note that while our work
significantly improves multilingual performance, the system is still not fully
inclusive, and as a community, we must continue to think collectively about
ways to ensure LLMs are accessible to all. Finally, while our key
contributions including learning algorithms are generic, the optimal
strategies and embeddings may differ from one dataset to another. As the
demand for multilingual language models continues to rise, our findings lay
the foundation for future advancements for enhancing Polyglot LLMs performance
in diverse linguistic contexts.

## References

  * [1] Shipra Agrawal and Navin Goyal.  Thompson sampling for contextual bandits with linear payoffs.  In International conference on machine learning, pages 127–135. PMLR, 2013.
  * [2] Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, et al.  Mega: Multilingual evaluation of generative ai.  arXiv preprint arXiv:2303.12528, 2023.
  * [3] AI4Bharat.  Indicqa: A multilingual question answering dataset for 12 indic languages.  https://huggingface.co/datasets/ai4bharat/IndicQA, 2022.
  * [4] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  CoRR, abs/2005.14165, 2020.
  * [5] Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil.  Universal sentence encoder.  CoRR, abs/1803.11175, 2018.
  * [6] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating large language models trained on code.  CoRR, abs/2107.03374, 2021.
  * [7] Yuxuan Chen, David Harbecke, and Leonhard Hennig.  Multilingual relation classification via efficient and effective prompting.  arXiv preprint arXiv:2210.13838, 2022.
  * [8] Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki.  Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages.  Transactions of the Association for Computational Linguistics, 2020\.
  * [9] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  Unsupervised cross-lingual representation learning at scale.  CoRR, abs/1911.02116, 2019.
  * [10] Cohere corporation.  Multilingual embedding models.  https://docs.cohere.com/docs/multilingual-language-models, 2023\.
  * [11] Microsoft corporation.  Azure cognitive services translator services.  https://learn.microsoft.com/en-us/azure/cognitive-services/translator/, 2023\.
  * [12] Microsoft corporation.  Azure openai service.  https://azure.microsoft.com/en-us/products/cognitive-services/openai-service, 2023\.
  * [13] Microsoft corporation.  Embedding models: Azure openai service.  https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/models#embeddings-models-1, 2023\.
  * [14] Microsoft corporation.  Gpt-3: Azure openai service.  https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/models#gpt-3-models, 2023\.
  * [15] Microsoft corporation.  Gpt 3.5 turbo: Azure openai service.  https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/models#chatgpt-gpt-35-turbo, 2023\.
  * [16] Microsoft corporation.  Gpt-4: Azure openai service.  https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/models#gpt-4-models, 2023\.
  * [17] Jinlan Fu, See-Kiong Ng, and Pengfei Liu.  Polyglot prompt: Multilingual multitask promptraining.  arXiv preprint arXiv:2204.14264, 2022.
  * [18] Thore Graepel, Joaquin Quinonero Candela, Thomas Borchert, and Ralf Herbrich.  Web-scale bayesian click-through rate prediction for sponsored search advertising in microsoft’s bing search engine.  Omnipress, 2010.
  * [19] Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei.  Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting, 2023.
  * [20] Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei.  Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting.  arXiv preprint arXiv:2305.07004, 2023.
  * [21] Lianzhe Huang, Shuming Ma, Dongdong Zhang, Furu Wei, and Houfeng Wang.  Zero-shot cross-lingual transfer of prompt-based tuning with a unified multilingual prompt, 2022.
  * [22] Katharina Hämmerl, Björn Deiseroth, Patrick Schramowski, Jindřich Libovický, Alexander Fraser, and Kristian Kersting.  Do multilingual language models capture differing moral norms?, 2022.
  * [23] Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury.  The state and fate of linguistic diversity and inclusion in the NLP world.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online, July 2020. Association for Computational Linguistics.
  * [24] Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, Shruti Gupta, Subhash Chandra Bose Gali, Vish Subramanian, and Partha P. Talukdar.  Muril: Multilingual representations for indian languages.  CoRR, abs/2103.10730, 2021.
  * [25] Patrick S. H. Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk.  MLQA: evaluating cross-lingual extractive question answering.  CoRR, abs/1910.07475, 2019.
  * [26] Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela.  Retrieval-augmented generation for knowledge-intensive NLP tasks.  CoRR, abs/2005.11401, 2020.
  * [27] Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al.  Few-shot learning with multilingual language models.  arXiv preprint arXiv:2112.10668, 2021.
  * [28] Patrick Littell, David R Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin.  Uriel and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors.  In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, volume 2, pages 8–14, 2017.
  * [29] Jerry Liu.  LlamaIndex, November 2022.
  * [30] Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, Zihao Wu, Dajiang Zhu, Xiang Li, Ning Qiang, Dingang Shen, Tianming Liu, and Bao Ge.  Summary of chatgpt/gpt-4 research and perspective towards the future of large language models, 2023.
  * [31] Chaitanya Malaviya, Graham Neubig, and Patrick Littell.  Learning language representations for typology prediction.  In Conference on Empirical Methods in Natural Language Processing (EMNLP), Copenhagen, Denmark, September 2017.
  * [32] OpenAI.  Gpt-4 technical report, 2023.
  * [33] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback, 2022\.
  * [34] Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, and Xia Song.  Beyond english-centric bitexts for better multilingual language representation learning.  arXiv preprint arXiv:2210.14867, 2022.
  * [35] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  Squad: 100, 000+ questions for machine comprehension of text.  CoRR, abs/1606.05250, 2016.
  * [36] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents, 2022\.
  * [37] Nils Reimers and Iryna Gurevych.  Making monolingual sentence embeddings multilingual using knowledge distillation.  CoRR, abs/2004.09813, 2020.
  * [38] Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al.  Language models are multilingual chain-of-thought reasoners.  arXiv preprint arXiv:2210.03057, 2022.
  * [39] Lifu Tu, Caiming Xiong, and Yingbo Zhou.  Prompt-tuning can be much better than fine-tuning on cross-lingual understanding with multilingual language models, 2022.
  * [40] Xinyi Wang, Yulia Tsvetkov, and Graham Neubig.  Balancing training for multilingual neural machine translation.  CoRR, abs/2004.06748, 2020.
  * [41] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  In Advances in Neural Information Processing Systems, 2023.
  * [42] Mengjie Zhao and Hinrich Schütze.  Discrete and soft prompting for multilingual models.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8547–8555, 2021.

##  8 Supplementary Material

###  8.1 Similar Language Algorithm

Section 3 introduced various prompt strategies and prompt templates that we
have optimized for polyglot LLMs. One of the prompt strategies defined is
round-tripping the input in source language through "Similar high-resourced
language (Sim)". In this section, we present the algorithm for identifying the
right set of similar high-resourced languages for a given source language. For
every language, we associate its class attribute between 0-5 based on the
classes defined in [23]. Here, class 5 represents very high-resourced
languages like English, whereas 0 represents very low-resourced languages like
Gondi, Mundari, etc. We use the language similarity metrics based on language
feature similarities[31] captured in lang2vec [28]. We give higher preference
to the languages with Latin script since the languages with Latin script have
shown better performance on GPTx models[2].

$w_{Latin}\leftarrow 0.9$;

Function  _GetRelevanceScore(_$d$ , $l_{cls}$, $isLatin$_)_:

$w\leftarrow 1$;

if  _$isLatin$_ then

$w\leftarrow w_{Latin}$

$score\leftarrow w\times d/l_{cls}$;

return $score$

Algorithm 1 Get language relevance score based on language similarity
distance, the class of related language and whether the related language has
Latin script.

Data: Source language $l_{s}$

Result: A set of similar high-resourced languages $L_{similar}$

$L_{similar}\leftarrow\emptyset$

Language class threshold $cls_{threshold}\leftarrow 3$

Languages similarity distance threshold $dist_{threshold}\leftarrow 0.5$

for  _$l\in L$_ do

if  _$class(l)\geq cls_{threshold}$_ then

Similarity distance between $l$ and $l_{S}$: $d\leftarrow
lang2vec\\_distance([^{\prime}syntactic^{\prime},^{\prime}genetic^{\prime},^{\prime}geographic^{\prime}],l,l_{s})$;
$RelevanceScore\leftarrow GetRelevanceScore(average(d),class(l),isLatin(l)$;

if  _$RelevanceScore\leq dist_{threshold}$_ then

$L_{similar}.add(l)$

Algorithm 2 Identifying similar high-resourced languages for a given language

###  8.2 Evaluations with Context only (Zero-shot)

In this section, we present MLQA-F1 score on IndicQA and TyDiQA when only the
context from the dataset is provided as input with no examples (zero-shot).
Table 17 and 17 shows the performance of this approach against different
prompt strategies. The last row shows the frequency distribution across all
strategies. Note that, from Table 3 and 3 we can see that performance with
few-shot is significantly better than zero-shot.

Figure 14: Prompt Strategies: IndicQA.

Models | GPT-DV003 | GPT3.5-Turbo
---|---|---
Lang | Mono | Trans | Sim |  | Agg
---
Src
| Agg
---
Trans
Mono | Trans | Sim |  | Agg
---
Src
| Agg
---
Trans
as | 0.28 | 0.31 | 0.26 | 0.34 | 0.34 | 0.41 | 0.31 | 0.32 | 0.35 | 0.40
bn | 0.30 | 0.33 | 0.23 | 0.33 | 0.38 | 0.54 | 0.34 | 0.30 | 0.35 | 0.49
gu | 0.27 | 0.30 | 0.23 | 0.31 | 0.30 | 0.50 | 0.33 | 0.30 | 0.33 | 0.48
hi | 0.40 | 0.50 | 0.24 | 0.49 | 0.47 | 0.57 | 0.53 | 0.38 | 0.51 | 0.54
kn | 0.24 | 0.29 | 0.12 | 0.29 | 0.26 | 0.44 | 0.29 | 0.20 | 0.29 | 0.41
ml | 0.12 | 0.28 | 0.08 | 0.29 | 0.17 | 0.28 | 0.30 | 0.15 | 0.28 | 0.21
mr | 0.29 | 0.29 | 0.22 | 0.30 | 0.27 | 0.47 | 0.33 | 0.28 | 0.34 | 0.45
or | 0.23 | 0.27 | 0.23 | 0.27 | 0.30 | 0.37 | 0.30 | 0.27 | 0.31 | 0.39
pa | 0.33 | 0.38 | 0.27 | 0.37 | 0.38 | 0.51 | 0.44 | 0.39 | 0.45 | 0.50
ta | 0.14 | 0.38 | NA | 0.38 | 0.21 | 0.29 | 0.39 | NA | 0.37 | 0.23
te | 0.08 | 0.34 | 0.06 | 0.13 | null | 0.34 | 0.35 | 0.14 | 0.33 | 0.15
freq | 5.86% | 27.54% | 40.94% | 24.68% | 0.98% | 66.00% | 15.41% | 14.13% | 3.84% | 0.62%

Figure 15: Prompt Strategies: TyDiQA.

Models | GPT3.5-Turbo
---|---
Lang | Mono | Trans | Sim |  | Agg
---
Src
| Agg
---
Trans
ar | 0.50 | 0.41 | null | 0.40 | 0.49
bn | 0.62 | 0.40 | 0.39 | 0.41 | 0.64
en | 0.60 | NA | 0.59 | 0.60 | 0.61
fi | 0.60 | 0.48 | 0.48 | 0.51 | 0.56
id | 0.64 | 0.54 | 0.45 | 0.54 | 0.62
ko | 0.60 | 0.46 | 0.37 | 0.47 | 0.59
ru | 0.46 | 0.42 | 0.42 | 0.42 | 0.45
sw | 0.71 | 0.59 | NA | 0.58 | 0.73
te | 0.61 | 0.42 | 0.23 | 0.37 | 0.60
freq | 44.25% | 2.00% |  46.27% | 1.72% | 5.76%

###  8.3 Human Annotation Task Details

We build a simple human annotation interface using
Streamlit111https://streamlit.io/ where the context, the question related to
the context, and the ground truth answer for each record are fetched from the
IndicQA dataset[3]. In this evaluation task, the annotators are first
presented with a passage that acts as the context required to answer the
question which is shown along with the ground truth answer. The annotators are
then asked to evaluate the answers generated by the LLM using different
strategies based on the ground truth answer provided, by answering one of the
following options: "Yes", "No" or "Partial". Here is the instruction provided
to the annotators.

First, select your language and go through the context under the title
"Context GT" once. Then, look at the question and try to answer this question
and compare it with the ground truth answer. Next, for all the available
answers, choose: 1. "Yes" if the answer is absolutely correct(minor
punctuation errors are allowed) 2. "Partial" if the answer captures some part
of the core answer, but has grammatical mistakes or minor errors(spelling,
etc.) that make the answer partially correct. 3. "No" if the answer is
completely wrong

Based on the human annotations for each question, we then recompute the F1
score. The updated F1 scores are calculated using Algorithm 3, where $evals$
contains evaluations for all the strategies annotated by the human annotator.

Data: $ground\\_truth$, $gpt\\_answers$, $evals$

Result: $eval\\_scores$

$eval\\_scores\leftarrow[]$; $valid\\_answers\leftarrow[]$;

$evals=get\\_eval(gpt\\_answers)$; $valid\\_answers.append(ground\\_truth)$;
for  _$i\leftarrow 0$ to $len(gpt\\_answers)$_ do

if  _$evals[i]="Yes"$_ then

$valid\\_answers.append(gpt\\_answers[i])$;

for  _$i\leftarrow 0$ to $len(gpt\\_answers)$_ do

$eval\\_f1.append(compute\\_score(gpt\\_answers[i],valid\\_answers))$;

Algorithm 3 Evaluation Algorithm when using Human Annotator or GPTAnnotator

Figure 16: Arguments and hyperparameters used for OpenAI calls

| Hyperparameter/
---
Argument
Value
Temperature | 0.0
OPENAI API Version | 2023-03-15 preview

Figure 17: Hyperparameters used for Multi-Arm Bandit(MAB) and Contextual
Bandit

Bandit | MAB(Thompson Sampling) | MAB($\epsilon$-Greedy) | Contextual Bandit
---|---|---|---
Iterations | 5000 | 5000 | 5000
Sampling | Beta(1,1) |  $\epsilon$=0.2 |  $\epsilon$=0.2

###  8.4 GPT Eval process

In Section 5, we introduced GPTAnnotator, where GPT models perform the
evaluation of the answer generated when compared to the ground truth. Similar
to the human evaluation task described in the previous subsection 8.3, the
GPTAnnotator is tasked to evaluate the LLM responses based on the available
ground truth for the given record. The prompt below is used for GPT3.5 Turbo
in order to evaluate the answers.

You are a multilingual evaluation assistant. Users will send in a query,
context text, the correct answer for the query based on the context text, and
also an answer that needs to be evaluated. You will evaluate the answer based
on the context text and the correct answer that the user has sent and respond
with Yes, No, or Partial based on the below evaluation instructions.
Instructions: 1. Yes if the answer is absolutely correct. 2. Partial if the
answer captures some part of the correct answer, but has minor errors like
grammatical or spelling mistakes, etc. 3.No if the answer is completely wrong.

The updated F1 Scores for each of the strategy is calculated using Algorithm 3
where $evals$ contains "Yes", "No" or "Partial" evaluations as judged by the
GPTAnnotator.

###  8.5 Inference Cost and compute

We utilize the Azure OpenAI[12] subscription to access all the GPT-x models.
Additionally, our experiments are run on a
Standard_NC24ads_A100_v4222https://learn.microsoft.com/en-us/azure/virtual-
machines/nc-a100-v4-series Virtual Machine on Azure cloud where a single A100
is utilized to generate the embeddings for the Hybrid Approach (see Section
4). For evaluating various prompt strategies such as Mono, Translate-test, and
Similar, the system relies on just one LLM call, which would be the typical
case in any system. However, for the other two aggregate prompt strategies,
our system has to perform multiple LLM calls. Further for the hybrid approach
presented in Section 4, it doesn’t require any additional inference calls as
we are just replacing one embedding model with another. Finally, all our
evaluations for bandits rely on data collected offline and hence require very
minimal computation for training the bandits (and our training is done within
just a few minutes). Thus enabling learning the right strategy dynamically and
in real time.

###  8.6 Hyperparameters

Here, we present all the hyperparameters (and other parameters/arguments) used
for generating responses from the GPT-x models and training the bandits
presented in Section 6. Table 17 provides the arguments and the
hyperparameters used for evaluating all the strategies on IndicQA and TyDiQA
datasets. Additionally, Table 17 provides the hyperparameters for the
bandits(MAB and Contextual Bandit).