# MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities,
Models and Tasks

Sanchit Ahuja Divyanshu Aggarwal Varun Gumma Ishaan Watts Ashutosh Sathe
Millicent Ochieng Rishav Hada Prachi Jain Maxamed Axmed Kalika Bali Sunayana
Sitaram Microsoft Corporation Contact<EMAIL_ADDRESS>
###### Abstract

Recently, there has been a rapid advancement in research on Large Language
Models (LLMs), resulting in significant progress in several Natural Language
Processing (NLP) tasks. Consequently, there has been a surge in LLM evaluation
research to comprehend the models’ capabilities and limitations. However, much
of this research has been confined to the English language, leaving LLM
building and evaluation for non-English languages relatively unexplored. There
has been an introduction of several new LLMs, necessitating their evaluation
on non-English languages. This study aims to expand our MEGA benchmarking
suite by including six new datasets to form the MEGAVERSE benchmark. The
benchmark comprises 22 datasets covering 81 languages, including low-resource
African languages. We evaluate several state-of-the-art LLMs like
GPT-3.5-Turbo, GPT4, PaLM2, and Llama2 on the MEGAVERSE datasets.
Additionally, we include two multimodal datasets in the benchmark and assess
the performance of the LLaVa-v1.5 model. Our experiments suggest that GPT4 and
PaLM2 outperform the Llama models on various tasks, notably on low-resource
languages, with GPT4 outperforming PaLM2 on more datasets than vice versa.
However, issues such as data contamination must be addressed to obtain an
accurate assessment of LLM performance on non-English languages.

##  1 Introduction

Large Language Models (LLMs) have surpassed the performance of older language
models on several tasks and benchmarks, sometimes even approaching or
exceeding human performance. However, it is not always clear whether this is
due to the increased capabilities of these models, or other effects, such as
artifacts in datasets, test dataset contamination, and the lack of datasets
that measure the true capabilities of these models. Thus, evaluation of Large
Language Models has become an important field of study.

Most of the work on evaluating LLMs via benchmarking Liang et al. (2022),
qualitative tests for specific capabilities Bubeck et al. (2023) or human
evaluation have focused on English. However, studies have shown that there is
a large gap between the capabilities of LLMs in English and other languages
Choudhury et al. (2023). Evaluation of LLMs in languages other than English is
challenging due to a variety of factors, including the lack of benchmarks
covering a large number of languages from diverse language families and the
lack of multilingual benchmarks covering tasks such as reasoning, chat, and
dialogue. Due to the small number of datasets available, test data
contamination becomes even more of a hurdle. Therefore, it is crucial to
prioritize multilingual evaluation to enhance the development of more
effective multilingual models. Neglecting this critical aspect may result in a
significant population being left behind and widen the digital divide.

Our prior work on performing a comprehensive benchmarking of LLMs across 16
datasets and 71 languages MEGA - Multilingual Evaluation of Generative AI
Ahuja et al. (2023) yielded the following observations: the largest model that
we evaluated, GPT4 OpenAI (2023) comes close to but in most cases does not
surpass the performance of SOTA fine-tuned language models such as TULRv6
Patra et al. (2023). GPT models also perform worse in languages that are
written in scripts other than the Latin script, and on low-resource languages.
Other LLMs that we tested, such as BLOOMZ Muennighoff et al. (2023) performed
worse than GPT4 except on tasks that they had been fine-tuned on.

Several new models have been introduced since our previous work on MEGA. There
is also growing interest in multimodal LLMs, and the intersection of
multimodal and multilingual LLMs has not been well studied. In this work, we
present an extension of our MEGA benchmarking study, which we refer to as
MEGAVERSE.

Our contributions are as follows:

  * •

We introduce six new benchmarks into our MEGA benchmarking suite, thus
extending coverage to 22 datasets and 81 languages including many low-resource
African languages.

  * •

We benchmark five new SOTA LLMs - PaLM2 Google (2023), Llama2 (3 variants)
Touvron et al. (2023) and LLaVA-v1.5 Liu et al. (2023a), in addition to GPT4
and GPT-3.5-Turbo.

  * •

We benchmark the multimodal LLaVA-v1.5 model Liu et al. (2023a) on two new
multilingual multimodal datasets.

  * •

We present results and an analysis of trends across these models with
directions for future research.

##  2 Related work

###  2.1 Evaluation of LLMs

Recently, there has been an increasing interest in evaluating LLMs on a wide
range of capabilities, given the surge in their popularity and effectiveness.
BIG-bench by Srivastava et al. (2023) consists of 204 tasks to evaluate LLMs
consisting of a diverse range of problems to evaluate the capabilities of
LLMs. While BIG-bench includes tasks in non-English languages as well, they
are largely related to translation. Liang et al. (2022) proposed Holistic
Evaluation of Language Models (HELM) defining a taxonomy of scenarios (tasks,
domains, and languages) and metrics (eg. accuracy, calibration, toxicity) that
define the space of LLM evaluation, and evaluate 30 language models on 42
scenarios and 7 metrics. However, all the scenarios are focused on datasets in
standard English or dialects, and they highlight coverage of languages as an
important area for improvement. Recent work Bubeck et al. (2023), has pointed
out the limitations of using standard NLP benchmarks to evaluate generative
models, due to the pace at which these benchmarks become saturated. There are
also concerns about benchmark contamination in LLM evaluation. Zhou et al.
(2023) show that test dataset contamination in training and fine-tuning data
lead to a significant impact in LLM performance.

###  2.2 Multilingual Benchmarks and Evaluation

Bang et al. (2023) evaluates the multilingual capabilities of ChatGPT and
shows that it fails to generalize to low-resource languages with non-latin
scripts. However, multilingual evaluation is performed only on a few tasks,
and a subset of 50-100 examples are used for testing the model. Hendy et al.
(2023) evaluate the translation abilities of GPT-3.5 models and find that
these models, while performing well in translating high-resource languages,
their capabilities for low-resource languages are limited. BUFFET Asai et al.
(2023) covering 54 languages across 15 datasets and Lai et al. (2023) covering
37 languages across 7 datasets also perform multilingual benchmarking of large
language models such as ChatGPT and BLOOMZ.

Our benchmarking suite MEGA is extended in MEGAVERSE to encompass 6 new
datasets, among which are two multimodal datasets. Furthermore, we conduct
benchmarking on state-of-the-art language models such as PaLM2 and Llama2, and
compare the latest LLMs in terms of their multilingual performance.

##  3 Experimental Setup

Dataset | GPT4 | GPT-3.5-Turbo | PaLM2 | Llama2-7b | Llama2-13b | Llama2-70b
---|---|---|---|---|---|---
XNLI | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
IndicXNLI | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$
GLUECoS NLI | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$
PAWS-X | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
XCOPA | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
XStoryCloze | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
En-Es-CS | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$
TyDiQA-GoldP | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
MLQA | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
XQUAD | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
IndicQA | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$
UDPOS | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$
PANX | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$
WinoMT | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$
Jigsaw | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$
XLSUM | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
Belebele | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
AfriQA | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$
IN22 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$
X-RiSAWOZ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$
Table 1: Experimental details per dataset and model. Some languages are not
supported by some of the models, due to which experiments on those datasets
are not performed, and some models are skipped due to resource constraints. We
also evaluate LLaVA-v1.5 on MaRVL and XM-3600 datasets to extend our
evaluation and analysis across modalities.

###  3.1 Datasets

We perform experiments on the 16 datasets that are part of the MEGA suite -
XNLI Conneau et al. (2018), IndicXNLI Aggarwal et al. (2022), GLUECoS NLI
Khanuja et al. (2020), PAWS-X Yang et al. (2019), XCOPA Ponti et al. (2020),
XStoryCloze Lin et al. (2022), GLUECoS Sentiment Analysis (En-Es-CS) Vilares
et al. (2016), TyDiQA-GoldP Clark et al. (2020), MLQA Lewis et al. (2020),
XQUAD Artetxe et al. (2020), IndicQA Doddapaneni et al. (2023), PAN-X Pan et
al. (2017), UDPOS Nivre et al. (2018), Jigsaw Kivlichan et al. (2020), WinoMT
Stanovsky et al. (2019) and XLSum Hasan et al. (2021). These datasets include
a mix of classification, Question Answering, Sequence Labeling, and Natural
Language Generation datasets, along with two datasets covering the Responsible
AI tasks of toxicity detection and bias. The datasets we include also contain
a mix of translated datasets verified by native speakers, as well as datasets
created independently for each language. For a more detailed description of
the datasets included in the original MEGA paper, we refer the readers to
Ahuja et al. (2023). We describe the 6 datasets added to MEGAVERSE below.

####  3.1.1 Belebele

Belebele Bandarkar et al. (2023) is a multiple choice machine reading
comprehension (MRC) dataset is parallel across 122 languages. Each question is
linked to a short passage from the FLORES-200 dataset Team et al. (2022). The
questions were created by human annotators and the human annotation procedure
was carefully curated to create questions that discriminate between different
levels of language comprehension. This process was reinforced by extensive
quality checks. In this paper, we evaluated Arabic, Czech, Danish, German,
English, Spanish, Finnish, French, Hebrew, Hungarian, Italian, Japanese,
Korean, Dutch, Norwegian, Polish, Portuguese, Russian, Swedish, Thai, Turkish,
Chinese Simplified and Chinese Traditional. We evaluated GPT4 and PaLM2 and
results for Llama2 and GPT-3.5-Turbo are reported from the dataset paper.

##### Prompt:

We evaluated our models on zero-shot prompting using instructions proposed by
Bandarkar et al. (2023) 111https://github.com/EleutherAI/lm-evaluation-
harness/pull/885.

Task Instruction $\mathcal{I}$:You are an AI assistant whose purpose is to
perform reading comprehension task. Given the following passage, query, and
answer choices, output the letter corresponding to the correct answer.

Template $f_{temp}$:
{instruction}
###
Passage:
{passage}
###
Query:
{query}
###
Choices:
(A) {A}
(B) {B}
(C) {C}
(D) {D}
###
Answer:

Figure 1: Belebele MRC Prompt

We perform zero-shot monolingual prompting for our experiments, as this
dataset does not have a dev set.

####  3.1.2 AfriQA

AfriQA Ogundepo et al. (2023) is a Question Answering dataset that does not
have a context passage. It covers 10 African languages - Bemba, Fon, Hausa,
Igbo, Kinyarwanda, Swahili, Twi, Wolof, and Yorùbá. We use the few-shot size
of $k$ = 4 and the monolingual prompting strategy for this dataset and perform
experiments on the GPT and Llama models, as the PaLM2 model only supports
Swahili.

Task Instruction $\mathcal{I}$: You are an NLP assistant trained to answer
questions directly. For each question provided, respond with the most accurate
and concise answer. The answer should be in the same language as the question.
Template $f_{temp}$:
Q: {question}
A: {answer}

Figure 2: AfriQA Prompt

####  3.1.3 XRiSAWOZ

XRiSAWOZ (Moradshahi et al., 2023) is a (domain specific) task oriented
dialogue modeling dataset. The dataset is a multilingual (English, Hindi,
French, Korean) translation of RiSAWOZ dataset (Quan et al., 2020) which was
Chinese. XRiSAWOZ also includes an English-Hindi code mixed setting.

Each dialogue in XRiSAWOZ is confined to a narrow domain and the conversation
agent must make use of structured knowledge available in the database to
answer user queries. We refer the reader to Moradshahi et al. (2023) for more
details and only present the summary of 4 subtasks below:

  * •

Dialogue State Tracking (DST): Generate “belief state” (represented with a
semi-structured, SQL-like language) of the agent for the current turn based on
(agent’s) previous belief state, last 2 (agent’s) dialogue acts and current
user utterance.

  * •

API Call Detection (API): Detect whether an API call to query the domain-
specific database is necessary for the current dialogue state.

  * •

Dialogue Act Generation (DA): Generate “dialogue act” (also represented with a
semi-structured language) based on the current belief state, last 2 agent
dialogue acts, user utterance as well as the result of an API call.

  * •

Response Generation (RG): Generate natural language response from the current
dialogue act.

There are about 4000 dialogue turns per setting in the dataset, each of which
needs to be evaluated for the 4 above subtasks requiring 16000 model
evaluations per language. Due to limited compute, we currently present the
results only on 10% of data i.e. about 400 dialogue turns across 3 domains.
The prompts used for this dataset are presented in Section 3.3.

####  3.1.4 IN22

IN22 Gala et al. (2023) is a translation benchmark for all 22 scheduled Indic
languages which is offered in two flavors, IN22-Gen and IN22-Conv. IN22-Gen is
a general-purpose multi-domain evaluation subset of IN22 which has been
curated from two sources: Wikipedia and Web Sources offering diverse content
spanning news, entertainment, culture, legal, and India-centric topics.
IN22-Conv is the conversation domain subset of IN22 and is designed to assess
translation quality of day-to-day conversations. The General subset consists
of 1024 sentences translated across 22 Indic languages while the
Conversational benchmark has 1503 sentences translated in the same manner,
enabling evaluation in 506 directions. Since it is not feasible to evaluate
all 22 languages due to resource constraints, we select the following 14
languages: Assamese, Bengali, English, Gujarati, Hindi, Kannada, Kashmiri,
Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu, Urdu. The following
prompt was employed for the translation task:

You are an AI assistant whose purpose is to perform translation. Given the
following sentence in {source}, translate it to {target}. If you cannot
translate the given sentence to {target}, just return “**********" (10
asterisks)

Figure 3: Translation Prompt

####  3.1.5 MaRVL

MaRVL (Multicultural Reasoning over Vision and Language) Liu et al. (2021) is
a dataset of images and associated captions. The concepts and images collected
were entirely driven by native speakers and are representative of various
cultures across the globe and span 5 languages, i.e., Indonesian, Chinese,
Swahili, Tamil, Turkish. Each instance in the dataset consists of a pair of
images (left image and right image) and a statement, and the task is to
determine whether the statement is consistent with respect to the given pair
of images. The following prompt was employed for MaRVL:

Presented below are two distinct images placed side by side and a below
caption in {language}. Analyze the images carefully and then read the below
provided caption. Your task is to determine whether the caption accurately and
truthfully describes the content, details, and scenario depicted in both
images. Consider all visible elements, expressions, interactions, and any
potential nuances that may influence the accuracy of the caption. Return
‘TRUE‘ if the caption is true, and ‘FALSE‘ otherwise.

CAPTION: {caption}

Figure 4: MaRVL Prompt

####  3.1.6 XM-3600

Crossmodal-3600 Thapliyal et al. (2022) is a multilingual image captioning
dataset consisting of 3600 geographically diverse images directly captioned in
36 different languages, avoiding any inconsistencies due to translations. We
once again select 20 out of the 36 languages due to resource constraints:
Arabic, Chinese, Czech, Danish, Dutch, English, Finnish, French, German,
Italian, Japanese, Korean, Norwegian, Polish, Portuguese, Russian, Spanish,
Swedish, Thai, Turkish. The following prompt was employed for XM-3600:

Analyze the image provided and generate a **concise caption** in {language}
that accurately describes the main elements and actions taking place in the
scene. Focus on highlighting the key components and the context in which they
are found, ensuring that the description is not overly detailed but still
provides enough information to understand the scenario. Additionally, include
a brief mention of the background or setting to give a fuller picture of the
scene. Aim for a balance between brevity and informativeness, capturing the
essence of the image in a few well-chosen words. If you cannot caption the
image in {language}, simply return “**********" (10 asterisks).

Figure 5: XM-3600 Prompt

###  3.2 Models

  * •

GPT-3.5-Turbo Ouyang et al. (2022) is an LLM developed by OpenAI by fine-
tuning GPT-3 on a dataset of labeler demonstrations of the desired model
behavior in a supervised learning setup. This was further fine-tuned on a
dataset of ranked outputs using reinforcement learning with human feedback
(RLHF). For our experiments, we accessed GPT-3.5-Turbo via Azure API.

  * •

GPT4 OpenAI (2023) is a transformer-based model pre-trained to predict the
next token in a document. Similar to GPT-3.5-Turbo, GPT4 was also fine-tuned
with RLHF. For our experiments, we accessed GPT4 via Azure API.

  * •

Llama2 Touvron et al. (2023) series, developed by Meta, is a collection of
pretrained and fine-tuned large language models (LLMs) ranging in scale from 7
billion to 70 billion parameters. It builds upon its predecessor, LLaMa, with
several enhancements. The models are available in three parameter sizes: 7B,
13B, and 70B, and come in both pretrained and chat model flavors. In this
work, we evaluate the Llama2 chat models of sizes 7B, 13B, and 70B.

  * •

PaLM2 Anil et al. (2023) PaLM2 is the successor to PaLM, the previous language
model developed by Google. It is much more parameter-efficient compared to its
predecessor. Although the technical report talks about a family of PaLM models
(Small, Medium, and Large) and support for multiple languages, the Vertex API
(which we use) does not disclose which model they are exposing and only
exposes a definite number of languages222https://cloud.google.com/vertex-
ai/docs/generative-ai/learn/models?_ga=2.106729799.-571187399.1699178589. We
use two variants of the model, one with a context window size of 8192 (text-
bison@001)and another with a context window size of 32k (text-bison-32k).
Since the API supports a limited number of languages, we only run experiments
on supported languages and skip the other languages in the MEGAVERSE datasets.

  * •

LLaVa-v1.5 Liu et al. (2023a) is a 13B multimodal opensource LLM with Vicuna
Chiang et al. (2023) as its backbone and CLIP ViT-L/14 Radford et al. (2021)
as its image processor. It is an improvement over its predecessor LLaVa Liu et
al. (2023b) achieved by better visual instruction tuning, response formatting
prompts, adding a dedicated NLP vision-language connector, including academic
task-oriented data, and additional scaling.

Although there may be various filters, classifiers, and other components
running behind the APIs, which makes it unfair to compare them to locally run
models, we have chosen to do so in order to investigate and compare the
performance of the available options for multilingual LLMs today. Refer to
Table 1 for a detailed model evaluation checklist to see an overview of our
experiments.

###  3.3 Prompting strategies

In our prior work Ahuja et al. (2023), we experiment with multiple prompting
strategies: translate-test, zero-shot cross-lingual prompting, and monolingual
prompting.

We define five main components to define the prompts: i) a test example
$x_{\mathrm{test}}$ for which the predictions are to be made; ii) $k$ few-shot
exemplars $\\{(x_{i},y_{i})\\}_{i=1}^{k}$, that are used to provide in-context
supervision to the model; iii) a task instruction $\mathcal{I}$ which
describes the instruction in text for the task to LLM; iv) a prompt template
$f_{\mathrm{temp}}(x)$ which turns a dataset input example into a text format
that can be used for prompting; and v) an answer verbalizer
$f_{\mathrm{verb}}(y)$ that maps the label $y$ to a textual representation. In
the MEGA framework, we consider the instruction, template, and verbalizer
together as a single template. For multilingual setups as highlighted in Lin
et al. (2022) and Shi et al. (2022), some additional variations to consider
include, the choice of the language of the few-shot examples, the language of
the prompt template, and the language of the test examples.

In our previous work, we show that the monolingual prompting variation
outperforms the zero-shot cross-lingual prompting variation for most datasets,
with the translate-test variation performing better than monolingual for a few
low-resource languages. We find that the gap between translate-test and
monolingual prompting is minimal for models such as GPT4, and so for this work
default to monolingual prompting except when specified otherwise. In cases
where dev datasets are not available in the target language, we use zero-shot
cross-lingual (zs-cl) prompting, which is the same as monolingual (mono)
prompting except for the language of the few-shot examples, which are in
English is zs-cl vs. the target language in mono. Note that the instruction of
the prompt remains in English, as it is shown to outperform writing
instructions in the target language in Ahuja et al. (2023).

####  3.3.1 XRisaWoz

We explain the prompting strategy used for XRisaWoz in detail in this section,
as it is a complex dataset compared to the other datasets we consider. We use
monolingual prompting with few-shot examples in the target language. Note that
DST and DA states still use English syntax (e.g. equal_to) with constants
(strings, names, values, etc.) from the target language. This is in line with
how Moradshahi et al. (2023) have preprocessed the data. The task prompt is
also in English for all the languages. We next detail the evaluation strategy
and metrics:

End-to-end evaluation. In this setting, the conversation agent starts with an
empty / NULL dialogue state and with a gold user utterance for turn 1. Based
on that, it generates a new dialogue state (DST), makes an API call if
necessary (API), generates a new dialogue act (DA) and finally verbalizes (RG)
it to the user. Based on the response, the agent receives a gold user
utterance for turn 2 and the conversation progresses till gold user utterances
are exhausted.

Turn-by-turn evaluation. Here, we construct “gold” inputs for each turn
assuming the conversation progresses perfectly for the previous turns. We
infer each subtask based on these gold inputs.

We do NOT perform end-to-end evaluation since models do not always generate
precise states, API calls, or dialogue acts. This is likely because the models
are not explicitly finetuned for that (sub)task. Nearly all the models in our
study overgenerate the state/API calls/dialogue acts with extra text. It is
somewhat unreasonable to expect models to generate syntactically correct semi-
structured language with only in-context learning. Furthermore, the RG task
also expects a somewhat shorter response but the models almost always
overgenerate the response despite explicitly asking them to not overgenerate.

We use a preprocessed version of the dataset from Moradshahi et al. (2023) to
perform turn-by-turn evaluation. For inferring a subtask on a dialogue turn,
we provide in-context examples corresponding to the same turn from other
domains. If for a particular turn, sufficient in-context examples are not
available, we look for the latest previous turn for which sufficient in-
context examples are available. E.g. Assume the following turn to count
distribution and $k$ = 4 (number of in-context examples). Turns 1 through 3
have 50 samples, Turn 4 has 10 examples, Turn 5 has 3 examples, and Turn 6 has
1 example. Now at Turn 5, we do not have sufficient examples from Turn 5.
Therefore, we sample in-context examples from turn 4. Similarly, at Turn 6 we
do not have sufficient examples from Turn 6 or Turn 5 so we use Turn 4 in-
context examples for Turn 6 as well.

Metrics. ($\uparrow$) indicates higher values are better. ($\downarrow$)
indicates lower is better.

  1. 1.

BLEU ($\uparrow$) – Fluency of natural language response (RG) with respect to
gold outcomes.

  2. 2.

Slot Error Rate ($\downarrow$) – Factual correctness of generated response
(Wen et al., 2015). This is 1 if the response contains all entities present in
the gold response and 0 otherwise.

  3. 3.

(Averaged/Task) Success Rate ($\uparrow$) – Was the agent able to complete the
user’s requests by providing all the requested information for a particular
domain? (Lin et al., 2021)

  4. 4.

API Accuracy ($\uparrow$) – 1 if the model correctly predicted an API call
along with all its constraints and 0 otherwise.

  5. 5.

Dialogue Act Accuracy ($\uparrow$) – 1 if the model correctly predicted a
dialogue act along with all correct entities and 0 otherwise.

  6. 6.

Joint Goal Accuracy ($\uparrow$) – 1 if the model correctly predicted a
dialogue state along with all correct slot-relation-value triplets and 0
otherwise (Budzianowski et al., 2018).

Prompts. Each subtask uses a different “task prompt” the general structure of
the overall prompt is as follows:

$\langle$ TASK PROMPT. Refer to each task below. $\rangle$
{
Learning example #i:
Turn ID: turn_id
Database: db_id
Context: gold_context
Answer: gold_answer
} for i in range(k) # (in-context examples)
Target example #i:
Turn ID: turn_id
Database: db_id
Context: gold_context
Answer: $\langle$model-completion-here$\rangle$

Figure 6: General prompt structure for X-RiSAWOZ

For chat-based (e.g. Llama2 chat) models, we drop “Learning example$\dots$”
and “Target example$\dots$” and use the ChatGPT-like prompt format with task
prompt in the “system” prompt, {Turn ID, Database, Context} in the “user”
prompt and “Answer” in the “assistant” prompt. We use the the dataset provided
by Moradshahi et al. (2023) in which the context is preprocessed to include
all the relevant information (e.g. previous dialogue acts or states) for a
task .

You are a helpful NLP assistant solving the “Task Oriented Dialogue" problem.
In particular, you are solving the "Dialogue State Prediction" subtask. In
Dialogue State Prediction, you must describe what is the state of the dialogue
given the history using SQL-like structure. The syntax can be understood from
the examples below. Based on the learning examples given below, complete the
“Answer" part of the target example. Do not print any additional information.

Figure 7: Task prompt for “DST” subtask in X-RiSAWOZ

You are a helpful NLP assistant solving the “Task Oriented Dialogue" problem.
In particular, you are solving the "API Call Detection" subtask. In API call
detection, your task is to identify whether the dialogue can be continued with
whatever context we already have. "yes" here means that additional data must
be queried using an API for continuing the dialog while "no" means that API
call is not required. Based on the learning examples given below, complete the
“Answer" part of the target example. Do not print any additional information.

Figure 8: Task prompt for “API” subtask in X-RiSAWOZ

You are a helpful NLP assistant solving the “Task Oriented Dialogue" problem.
In particular, you are solving the "Dialogue Act Prediction" subtask. In
Dialogue Act Generation, you must generate the next dialogue action based on
the given context. This will be an SQL-like structure. The syntax can be
understood from the examples below. Based on the learning examples given
below, complete the “Answer" part of the target example. Do not print any
additional information.

Figure 9: Task prompt for “DA” subtask in X-RiSAWOZ

You are a helpful NLP assistant solving the “Task Oriented Dialogue" problem.
In particular, you are solving the "Response Generation" subtask. In Response
Generation, your task is to produce a natural language response from the
chatbot given the context of the conversation. Based on the learning examples
given below, complete the “Answer" part of the target example. Do not print
any additional information.

Figure 10: Task prompt for “RG” subtask in X-RiSAWOZ

##  4 Results

We now look at results across all models and languages for each dataset.

####  4.0.1 XNLI

We performed experiments on all models: Both versions of GPT, the three Llama
variants, and PaLM2, as shown in Figure 11. We see that all models perform
best on English, with slightly lower performance on Greek and German, with
lower performance on languages like Hindi, Thai, Urdu, and Swahili. Overall
PaLM2 performs best, closely followed by GPT4. GPT-3.5-Turbo is worse on all
languages, however, we find that all three Llama models perform substantially
worse. Since XNLI is a popular dataset, dataset contamination cannot be ruled
out. We perform contamination analysis for GPT4 in MEGA Ahuja et al. (2023)
and plan to check for contamination in the other models in future work.

Figure 11: Results for XNLI across all languages and models for monolingual
prompting

####  4.0.2 IndicXNLI

We performed experiments on IndicXNLI on the GPT models as well as Llama
models, however, the Llama models gave scores of 0 for all languages, which is
why we do not plot them in Figure 12. We find that GPT4 outperforms
GPT-3.5-Turbo on all languages with the highest scores on Hindi, Punjabi, and
Bengali. However, the overall accuracy is not very high on any language
compared to the XNLI results seen earlier.

Figure 12: Results for Indic-XNLI across all languages for monolingual
prompting

####  4.0.3 GLUECos NLI

We run experiments for the GLUECoS NLI dataset on GPT-3.5-Turbo and PaLM2.
GPT-3.5-Turbo obtains an accuracy of 0.78, GPT4’s accuracy is 0.90 while the
accuracy of PaLM2 is 0.82, showing that all models can do well on this NLI
task.

####  4.0.4 PAWS-X

We performed experiments on PAWS-X on all models, as shown in Figure 13. We
find that both GPT models have almost equal performance across all languages,
and PaLM2 outperforms the GPT models on all languages. Note that this dataset
contains high-resource languages, which may explain the high accuracies that
all models achieve. However, dataset contamination cannot be ruled out, as
shown in Ahuja et al. (2023). We also see that the Llama models perform worse
than the GPT models and PaLM2, although the difference in performance is not
as large as in some of the other datasets.

Figure 13: Results for PAWSX across all languages and models for monolingual
prompting

####  4.0.5 XCOPA

We performed experiments on XCOPA on all models, as shown in Figure 14. Once
again, we find that the performance of GPT4 and PaLM2 are comparable, with
GPT4 outperforming PaLM2 on Estonian, Italian, Swahili, and Turkish, while
PaLM2 outperforms GPT4 on Indonesian and Thai. Notably, they are both better
than GPT3.5Turbo, which performs substantially better than the Llama2 models
except in Quechua, for which no model performs well. However, the results on
all other languages for GPT4 and PaLM2 are extremely high, which may be due to
dataset contamination as shown in Ahuja et al. (2023).

Figure 14: Results for XCOPA across all languages and models for monolingual
prompting

####  4.0.6 XStoryCloze

We performed experiments for XStoryCloze with the GPT, PaLM, and Llama models,
however, the Llama models gave scores of 0 for all languages, hence we omit
them from Figure 15. For this dataset, we find that the gap between the GPT
models and PaLM2 is very high, with both GPT models performing extremely well.
The contamination experiments from Ahuja et al. (2023) show a low possibility
of dataset contamination for GPT4, which indicates that the GPT models are
able to perform this task well.

Figure 15: Results for XStoryCloze across all languages and models for
monolingual prompting

####  4.0.7 GLUECoS Sentiment Analysis (En-Es-CS)

We run experiments for the En-Es-CS dataset on code-mixed Sentiment Analysis
on GPT-3.5-Turbo and PaLM2. GPT3.5Turbo obtains an accuracy of 0.67, GPT4’s
accuracy is 0.45 while the accuracy of PaLM2 is 0.51. Surprisingly,
GPT-3.5-Turbo outperforms both GPT4 and PaLM2 on this task.

####  4.0.8 TyDiQA GoldP

We performed experiments on TyDiQA on all models, as shown in Figure 16. We
find that GPT4 outperforms all other models, with PaLM2 and GPT3.5Turbo having
comparable performance and the Llama models having very poor performance.
Surprisingly, the best-performing languages are Swahili and Indonesian, which
are low-resource languages. However, dataset contamination in GPT4 cannot be
ruled out, as shown in Ahuja et al. (2023).

Figure 16: Results for TyDiQA across all languages and models for monolingual
prompting

####  4.0.9 MLQA

We performed experiments on MLQA on all models, as shown in Figure 17. GPT4
outperforms all other models for this dataset except for German. The
Llama2-13B model performs well for some languages, such as Arabic, German, and
Spanish but performs poorly on Chinese Hindi, and Vietnamese. This is one of
the few datasets where PaLM2 struggles, particularly for Arabic and Chinese.
Dataset contamination in GPT4 cannot be ruled out, as shown in Ahuja et al.
(2023).

Figure 17: Results for MLQA across all languages and models for monolingual
prompting

####  4.0.10 XQUAD

We performed experiments on MLQA on all models, as shown in Figure 18. GPT4 is
the best-performing model across languages, followed closely by GPT-3.5-Turbo.
PaLM2 outperforms GPT-3.5-Turbo on Greek and Thai, while all three Llama
models perform poorly on this dataset. Dataset contamination in GPT4 cannot be
ruled out, as shown in Ahuja et al. (2023).

Figure 18: Results for XQUAD across all languages and models for zero-shot
cross-lingual prompting

####  4.0.11 IndicQA

We performed experiments on IndicXNLI on the GPT models as well as Llama
models, however, the Llama models gave scores of 0 for all languages, which is
why we omit them from Figure 19. We use the zero-shot cross-lingual prompting
strategy due to the absence of a dev set. GPT4 performs better than
GPT-3.5-Turbo, with the best performance seen for Hindi, Marathi, and Bengali.

Figure 19: Results for IndicQA across all languages with zero-shot cross-
lingual prompting

####  4.0.12 PAN-X

We performed experiments on PAN-X on the GPT models and PaLM2, as shown in
Figure 20. GPT4 outperforms the other two models, with GPT-3.5-Turbo
outperforming PaLM2 on several languages. However, all models perform poorly
on Thai, Japanese, and Chinese on this sequence labeling task. Since this is
an older dataset, GPT4 data contamination cannot be ruled out as shown in
Ahuja et al. (2023).

Figure 20: Results for PAN-X across all languages with monolingual prompting

####  4.0.13 UDPOS

Similar to PAN-X, we performed experiments on UDPOS on the GPT models and
PaLM2, as shown in Figure 21. All three models show similar high performance
across languages, except for Arabic, Greek, Hebrew, Hindi, and Vietnamese,
where PaLM2 performs best. GPT4 data contamination cannot be ruled out as
shown in Ahuja et al. (2023).

Figure 21: Results for UDPOS across all languages with monolingual prompting

####  4.0.14 Jigsaw

We perform experiments on the Jigsaw dataset for GPT-3.5-Turbo and PaLM2 using
the monolingual prompting strategy and find that both models perform very well
on all languages. Since the dataset cannot be accessed without download,
models are less likely to be contaminated with this dataset Ahuja et al.
(2023).

Figure 22: Results for Jigsaw across models and languages

####  4.0.15 WinoMT

We perform experiments on the WinoMT dataset only for GPT-3.5-Turbo using the
monolingual prompting strategy and report the results for completeness. We
find that the model does not perform well on any of the languages.

Figure 23: Results for WinoMT across languages

####  4.0.16 XLSum

We performed experiments on XLSum on all models, as shown in Figure 24 using
the monolingual prompting strategy. Overall, we find that GPT4 outperforms all
other models, with some exceptions. GPT-3.5-Turbo performs best for African
languages like Swahili, Somali, and Yoruba, while the Llama models perform
best for Arabic, Kyrgyz, Vietnamese, and Welsh. According to the contamination
analysis in Ahuja et al. (2023), it is possible, though less likely that GPT4
is contaminated with this dataset.

Figure 24: Results for XLSum across all languages and models for monolingual
prompting

####  4.0.17 BeleBele

We perform experiments for GPT4 and PaLM2 and report Llama2-chat-70B and
GPT-3.5-Turbo results from Bandarkar et al. (2023), shown in Figure 25. We use
the monolingual prompting strategy, however, we do not use few-shot examples
in the target language or English. We perform experiments on Arabic, Czech,
Danish, German, English, Spanish, Finnish, French, Hebrew, Hungarian, Italian,
Japanese, Korean, Dutch, Norwegian, Polish, Portuguese, Russian, Swedish,
Thai, Turkish, Chinese Simplified and Chinese Traditional.

Figure 25: Results for Belebele across all languages with monolingual
prompting without few-shot examples

GPT4 and PaLM2 outperform GPT-3.5-Turbo and Llama2, in all languages, with the
performance of GPT4 and PaLM2 being comparable. Most models do well due to the
multiple-choice question-answering nature of the task, which makes parsing
outputs and evaluation simpler and increases the probability of success even
for weaker models.

####  4.0.18 AfriQA

We perform experiments for GPT-3.5-Turbo, GPT4, and Llama2. We do not run
experiments with PaLM2 for this dataset as most of the languages in the
dataset are not supported. GPT4 outperforms the GPT-3.5-Turbo model, while the
Llama2 models perform very poorly on all languages.

Figure 26: Results for AfriQA across all languages with monolingual prompting

####  4.0.19 IN22

We report our results on the IN22-Gen subset and randomly select $k$ = 8
translation pairs from the development set of FLORES-200 Team et al. (2022) as
in-context examples. Due to GPT4 resource constraints, we select 50 pairs from
the test set for evaluation. We report GPT-3.5-Turbo 0-shot scores from Gala
et al. (2023) for comparison. For consistency, we use the
indic_nlp_library333https://github.com/anoopkunchukuttan/indic_nlp_library and
the evaluation scripts444https://github.com/AI4Bharat/IndicTrans2 from Gala et
al. (2023) to tokenize the predictions and references while computing chrF++
Popović (2017) for Indic languages. We do not evaluate PaLM2 on this dataset,
as most languages in this dataset are not supported by it.

Figure 27 demonstrates the results of GPT4, GPT-3.5-Turbo (0 and 8-shot), and
Llama2 on IN22 Gen. We find that Llama-2 performs poorly on all Indic
languages with the scores barely crossing 2 or 3 chrF++ points in the En-Indic
direction, whereas the performance is better on the Indic-En direction. GPT4
performs the best among all models considered, and the effect of in-context
examples seems negligible across languages for GPT-3.5-Turbo. A plausible
explanation for this could be that these languages have enough abundance in
the GPT-3.5-Turbo, that in-context examples do not matter, or that the domains
of FLORES-200 and IN22 are disjoint in this case.

We also find that GPT4 performs consistently well across languages in Indic-En
direction, while significant fluctuations can be seen in the En-Indic
direction. Further, we notice that prompting the model in the En-Indic
direction is much slower compared to the Indic-En direction as more tokens
need to be generated in the case of the former due to the high fertility of
Indic languages in the tokenization process.

For GPT-3.5-Turbo, Kashmiri, (despite being a low-resource language) and Urdu
are the highest performing language on the En-Indic side and second best on
Indic-En. Similarly, the scores of Nepali and Hindi are close. Assamese is the
lowest-resource language and shows the worst performance in the En-Indic
direction, while Tamil performs worst on Indic-En. Surprisingly, Kannada and
Telugu which share a common script perform better in a 0-shot setting than
with few-shot examples.

((a)) Indic-En

((b)) En-Indic

Figure 27: chrF++ scores of IN22-Gen. Note that, Kashmiri 0-shot scores were
not covered in Gala et al. (2023)

####  4.0.20 XRiSAWOZ

Figure 28 shows the performance of the two GPT models, PaLM2 and the
Llama2-70B variant on XRiSAWOZ with the monolingual prompting strategy in
terms of Average Success Rate defined in 3. We find that both GPT models
outperform PaLM2, with Llama2-70B also outperforming PaLM2 slightly in English
and French.

Figure 28: Average Success Rate for XRiSAWOZ across models and languages

Despite the turn-by-turn evaluations, models still overgenerate the DST and DA
answers. We find the structured answer within the response with the help of a
regex. In particular, GPT-3.5-Turbo and Llama2 are poor in terms of
overgeneration, leading to lower BLEU scores. We suspect dataset contamination
on the Chinese split i.e. the original RiSAWOZ dataset since the performance
of PaLM2, GPT-3.5-Turbo, and GPT4 models on Chinese is remarkably better than
other languages. Specifically, we observe less than 10% SER on all 3 of these
models. The average success rate, API accuracy, DA accuracy as well as JGA are
significantly better on Chinese as compared to other languages (even English)
for these 3 models. On average, GPT-4 is the best-performing model followed
closely by GPT-3.5-Turbo. PaLM performs worse than both GPT models but is
significantly better than Llama2 whose performance was the worst among all
models across all tasks.

For all models, the performance on English is significantly better than the
performance on Hindi. The stronger performance on English seems to be helping
all the models in the code-mixed English-Hindi setting. While the performance
in English-Hindi setting is not comparable to English, all models show a
significant improvement in English-Hindi performance as compared to Hindi-only
performance. Looking at individual sub-tasks, we find that models struggle to
predict the correct dialogue state or dialogue act despite scoring high API
accuracy.

####  4.0.21 MaRVL

Since LLaVA-v1.5-13B is the only multimodal model under consideration, we
evaluate the multimodal datasets only on LLaVA. However, we use an additional
prompting strategy, translate-test and compared its performance to monolingual
prompting. Figure LABEL:fig:marvl_llava shows the F1 scores of the model
across 5 languages. We see that F1 scores are low overall, with the lowest
score on Tamil. The translate-test strategy outperforms monolingual prompting
for all languages except Indonesian, however, the performance is still poor
compared to a random classification.

Figure 29: F1 scores for the Llava model on MaRVL. We used two prompting
strategies, monolingual and translate-test.

####  4.0.22 XM-3600

Next, we test the same model Llava on the XM-3600 dataset, which is an image
captioning dataset, and use the chrF++ metric to report scores. chrF++ Popović
(2017) is a metric used to evaluate the quality of machine-generated
translations which considers both precision and recall of character n-grams in
the reference and candidate translations. By focusing on character-level
information, this metric can capture subtle nuances and improvements in
translation quality, especially in languages with complex morphology or
syntax. For this task of image captioning, we employ chrF++ to compare how
close the machine-generated caption is to the gold captions provided in the
dataset. Any scores greater than or equal to 40 can be considered moderate,
and scores below 20 are poor. We see poor performance for most languages that
are not written in Latin script, especially Japanese, Korean, Russian, Thai,
and Chinese. Most Latin script high-resource languages such as Finnish,
French, German, Dutch, Spanish, and Italian outperform or come close to
English performance, with lower-resource languages such as Danish, Czech
Polish, and Norwegian performing slightly worse.

Figure 30: chrF++ scores for the Llava model on XM-3600. We use monolingual
prompting as the prompting strategy.

##  5 Discussion

In this work, we benchmark 22 datasets covering 81 languages across 6 models –
GPT-3.5-Turbo, GPT4, PaLM2, Llama2 (3 versions) and LLaVa-v1.5. We find
similar trends across most datasets we study - GPT4 and PaLM2 outperform the
Llama models, particularly on low-resource languages. Llama performs
particularly poorly on Indian languages and African languages in MEGAVERSE,
indicating that it is not ready for deployment without fine-tuning on these
languages.

We find that GPT4 and PaLM2 perform best on different datasets – GPT4
outperforms PaLM2 on TyDiQA, XCOPA, XStoryCloze, XQuaD, PANX, XLSum and
XRisaWoz, while PaLM2 outperforms GPT4 on XNLI, PAWS-X and Belebele. Thus,
GPT4 fares better overall on multilingual datasets, compared to PaLM2. GPT4
can also run on more languages, as there is no restriction on the API, and
hence it can be evaluated on more datasets than PaLM2.

Results on translation datasets show that high-resource languages are robust
to few-shot examples. Providing in-context examples of such languages is not
always useful, but is usually helpful for low-resource languages. We find that
standard metrics such as ROUGE-L are not sufficient for generative models. For
example, in the XLSum dataset, we observed a lot of languages being assigned
very low ROUGE scores even when the summarization was reasonable. This
necessitates further research on building better metrics for evaluating
generative models.

We introduce two multimodal datasets in MEGAVERSE and find that the Llava
model performs well on the image captioning task for some high-resource
languages, but performs poorly on the reasoning dataset. In future work, we
also plan to benchmark GPT4 on multi-modality after access to it becomes
public555https://openai.com/blog/new-models-and-developer-products-announced-
at-devday. Benchmarking and improving multilingual multimodal models is an
important open area of research.

In this work, we compare black-box API access models such as GPT4 and PaLM2
with models that can be run locally, such as Llama2. This comparison may seem
unfair because there may be many other models or systems behind APIs that may
affect the performance of the foundation model on a dataset. However, we
believe that this comparison is necessary to evaluate the choices available
for deploying LLM-based applications in non-English languages.

Dataset contamination is a critical issue that affects English and non-English
language benchmarking studies. Given that new multilingual evaluation datasets
are difficult to create due to resource and funding constraints, it is
imperative that they are not included in the training data of LLMs. To achieve
this objective, we need to enhance our ability to identify instances of
contamination, as well as implement measures to avoid future contamination.

Our future research will also focus on studying the factors that affect
performance, such as the quantity of pre-training and fine-tuning data,
tokenizer fertility, and other relevant factors on the new models and datasets
we have included. We plan to release all the code necessary for running the
MEGAVERSE benchmark to facilitate research in the critical field of
multilingual LLM evaluation.

##  6 Limitations

Our work is subject to the following limitations:

###  6.1 Dataset contamination

We present a contamination analysis of the datasets included in MEGA for GPT4
in Ahuja et al. (2023), however, we do not perform contamination analysis on
the new datasets we include or on other models such as PaLM2 and Llama2. We
leave this as an important future research direction to pursue.

###  6.2 Prompt tuning

LLMs are sensitive to prompting, and we do not perform extensive prompt tuning
for the new datasets, as they do not have many prompts to choose from, unlike
the original MEGA datasets where we perform prompt tuning on English dev sets.
We also do not experiment with prompting variations, such as translate-test
and zero-shot cross-lingual prompting, or more complex strategies such as
Chain of Thought prompting due to resource constraints.

###  6.3 Experiments on limited data and datasets

Due to resource constraints, we perform experiments on partial datasets when
indicated, and do not evaluate all models on all datasets. We plan to do so in
future work.

###  6.4 Focus on task accuracy

We perform limited experiments on RAI datasets and do not perform experiments
on other important dimensions such as fairness, bias, robustness, efficiency
etc., mainly due to the lack of such datasets for non-English languages. This
is an important future research direction.

## References

  * Aggarwal et al. (2022) Divyanshu Aggarwal, Vivek Gupta, and Anoop Kunchukuttan. 2022.  IndicXNLI: Evaluating multilingual inference for Indian languages.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 10994–11006, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Ahuja et al. (2023) Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, et al. 2023.  Mega: Multilingual evaluation of generative ai.  _arXiv preprint arXiv:2303.12528_.
  * Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023.  Palm 2 technical report.  _arXiv preprint arXiv:2305.10403_.
  * Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020.  On the cross-lingual transferability of monolingual representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4623–4637.
  * Asai et al. (2023) Akari Asai, Sneha Kudugunta, Xinyan Velocity Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannaneh Hajishirzi. 2023.  Buffet: Benchmarking large language models for few-shot cross-lingual transfer.  _arXiv cs.CL 2305.14857_.
  * Bandarkar et al. (2023) Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2023.  The belebele benchmark: a parallel reading comprehension dataset in 122 language variants.  _arXiv preprint arXiv:2308.16884_.
  * Bang et al. (2023) Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023.  A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.  _arXiv preprint arXiv:2302.04023_.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023.  Sparks of artificial general intelligence: Early experiments with gpt-4.
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5016–5026, Brussels, Belgium. Association for Computational Linguistics.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Choudhury et al. (2023) De Choudhury et al. 2023.  Ask me in english instead: Cross-lingual evaluation of large language models for healthcare queries.  _arXiv preprint arXiv:2310.13132_.
  * Clark et al. (2020) Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.  Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages.  _Transactions of the Association for Computational Linguistics_ , 8:454–470.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating cross-lingual sentence representations.  In _Proceedings of EMNLP 2018_ , pages 2475–2485.
  * Doddapaneni et al. (2023) Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2023.  Towards leaving no Indic language behind: Building monolingual corpora, benchmark and models for Indic languages.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 12402–12426, Toronto, Canada. Association for Computational Linguistics.
  * Gala et al. (2023) Jay Gala, Pranjal A Chitale, Raghavan AK, Sumanth Doddapaneni, Varun Gumma, Aswanth Kumar, Janki Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, et al. 2023.  Indictrans2: Towards high-quality and accessible machine translation models for all 22 scheduled indian languages.  _arXiv preprint arXiv:2305.16307_.
  * Google (2023) Google. 2023.  Palm-2 technical report.
  * Hasan et al. (2021) Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M Sohel Rahman, and Rifat Shahriyar. 2021.  Xl-sum: Large-scale multilingual abstractive summarization for 44 languages.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4693–4703.
  * Hendy et al. (2023) Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023.  How good are gpt models at machine translation? a comprehensive evaluation.  _arXiv preprint arXiv:2302.09210_.
  * Khanuja et al. (2020) Simran Khanuja, Sandipan Dandapat, Sunayana Sitaram, and Monojit Choudhury. 2020.  A new dataset for natural language inference from code-mixed conversations.  In _Proceedings of the The 4th Workshop on Computational Approaches to Code Switching_ , pages 9–16.
  * Kivlichan et al. (2020) Ian Kivlichan, Jeffrey Sorensen, Julia Elliott, Lucy Vasserman, Martin Görner, and Phil Culliton. 2020.  Jigsaw multilingual toxic comment classification.
  * Lai et al. (2023) Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023.  Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning.
  * Lewis et al. (2020) Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020.  Mlqa: Evaluating cross-lingual extractive question answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7315–7330.
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022.  Holistic evaluation of language models.  _arXiv preprint arXiv:2211.09110_.
  * Lin et al. (2022) Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2022.  Few-shot learning with multilingual generative language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Lin et al. (2021) Zhaojiang Lin, Andrea Madotto, Genta Winata, Peng Xu, Feijun Jiang, Yuxiang Hu, Chen Shi, and Pascale N Fung. 2021.  Bitod: A bilingual multi-domain dataset for task-oriented dialogue modeling.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks_ , volume 1. Curran.
  * Liu et al. (2021) Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021.  Visually grounded reasoning across languages and cultures.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10467–10485.
  * Liu et al. (2023a) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a.  Improved baselines with visual instruction tuning.
  * Liu et al. (2023b) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_.
  * Moradshahi et al. (2023) Mehrad Moradshahi, Tianhao Shen, Kalika Bali, Monojit Choudhury, Gael de Chalendar, Anmol Goel, Sungkyun Kim, Prashant Kodali, Ponnurangam Kumaraguru, Nasredine Semmar, Sina Semnani, Jiwon Seo, Vivek Seshadri, Manish Shrivastava, Michael Sun, Aditya Yadavalli, Chaobin You, Deyi Xiong, and Monica Lam. 2023.  X-RiSAWOZ: High-quality end-to-end multilingual dialogue datasets and few-shot agents.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 2773–2794, Toronto, Canada. Association for Computational Linguistics.
  * Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023.  Crosslingual generalization through multitask finetuning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15991–16111, Toronto, Canada. Association for Computational Linguistics.
  * Nivre et al. (2018) Joakim Nivre, Mitchell Abrams, Željko Agić, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, et al. 2018.  Universal dependencies 2.2.
  * Ogundepo et al. (2023) Odunayo Ogundepo, Tajuddeen R Gwadabe, Clara E Rivera, Jonathan H Clark, Sebastian Ruder, David Ifeoluwa Adelani, Bonaventure FP Dossou, Abdou Aziz DIOP, Claytone Sikasote, Gilles Hacheme, et al. 2023.  Afriqa: Cross-lingual open-retrieval question answering for african languages.  _arXiv preprint arXiv:2305.06897_.
  * OpenAI (2023) OpenAI. 2023.  Gpt4 technical report.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Pan et al. (2017) Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017.  Cross-lingual name tagging and linking for 282 languages.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1946–1958.
  * Patra et al. (2023) Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, and Xia Song. 2023.  Beyond English-centric bitexts for better multilingual language representation learning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15354–15373, Toronto, Canada. Association for Computational Linguistics.
  * Ponti et al. (2020) Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020.  Xcopa: A multilingual dataset for causal commonsense reasoning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2362–2376.
  * Popović (2017) Maja Popović. 2017.  chrF++: words helping character n-grams.  In _Proceedings of the Second Conference on Machine Translation_ , pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics.
  * Quan et al. (2020) Jun Quan, Shian Zhang, Qian Cao, Zizhong Li, and Deyi Xiong. 2020.  RiSAWOZ: A large-scale multi-domain Wizard-of-Oz dataset with rich semantic annotations for task-oriented dialogue modeling.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 930–940, Online. Association for Computational Linguistics.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8748–8763. PMLR.
  * Shi et al. (2022) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2022.  Language models are multilingual chain-of-thought reasoners.  _CoRR_ , abs/2210.03057.
  * Srivastava et al. (2023) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-López, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoń, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michał Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan A. Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. 2023.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
  * Stanovsky et al. (2019) Gabriel Stanovsky, Noah A Smith, and Luke Zettlemoyer. 2019.  Evaluating gender bias in machine translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1679–1684.
  * Team et al. (2022) NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022.  No language left behind: Scaling human-centered machine translation.
  * Thapliyal et al. (2022) Ashish V Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut. 2022.  Crossmodal-3600: A massively multilingual multimodal evaluation dataset.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 715–729.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Vilares et al. (2016) David Vilares, Miguel A Alonso, and Carlos Gómez-Rodríguez. 2016.  En-es-cs: An english-spanish code-switching twitter corpus for multilingual sentiment analysis.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 4149–4153.
  * Wen et al. (2015) Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015.  Semantically conditioned LSTM-based natural language generation for spoken dialogue systems.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1711–1721, Lisbon, Portugal. Association for Computational Linguistics.
  * Yang et al. (2019) Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019.  PAWS-X: A cross-lingual adversarial dataset for paraphrase identification.  In _Proceedings of EMNLP 2019_ , pages 3685–3690.
  * Zhou et al. (2023) Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023.  Don’t make your llm an evaluation benchmark cheater.  _arXiv preprint arXiv:2311.01964_.