# IrokoBench: A New Benchmark for African Languages in the Age of Large
Language Models

David Ifeoluwa Adelani1∗, Jessica Ojo2∗, Israel Abebe Azime3∗, Jian Yun
Zhuang4,
Jesujoba O. Alabi3, Xuanli He1, Millicent Ochieng5, Sara Hooker6, Andiswa
Bukula7,
En-Shiun Annie Lee8, Chiamaka Chukwuneke9, Happy Buzaaba10, Blessing Sibanda∗,
Godson Kalipe∗, Jonathan Mukiibi11∗, Salomon Kabongo12∗, Foutse Yuehgoh13∗,
Mmasibidi Setaka7, Lolwethu Ndolela∗, Nkiruka Odu∗, Rooweither Mabuya7,
Shamsuddeen Hassan Muhammad14, Salomey Osei15∗, Sokhar Samb16∗,
Tadesse Kebede Guge17∗, Pontus Stenetorp1

∗Masakhane NLP 1University College London, 2Lelapa AI, 3Saarland University,
4University of Toronto,
5Microsoft Research Africa, 6Cohere For AI, 7SADiLaR, 8Ontario Tech
University, 9Lancaster University,
10Princeton university, 11Makerere University, 12Leibniz Universität Hannover,
13Le CNAM,
14Imperial College London, 15Universidad de Deusto, 16DAUST, 17Haramaya
University

###### Abstract

Despite the widespread adoption of Large language models (LLMs), their
remarkable capabilities remain limited to a few high-resource languages.
Additionally, many low-resource languages (e.g., African languages) are often
evaluated only on basic text classification tasks due to the lack of
appropriate or comprehensive benchmarks outside of high-resource languages. In
this paper, we introduce IrokoBench—a human-translated benchmark dataset for
16 typologically-diverse low-resource African languages covering three tasks:
natural language inference (AfriXNLI), mathematical reasoning (AfriMGSM), and
multi-choice knowledge-based QA (AfriMMLU). We use IrokoBench to evaluate
zero-shot, few-shot, and translate-test settings (where test sets are
translated into English) across 10 open and four proprietary LLMs. Our
evaluation reveals a significant performance gap between high-resource
languages (such as English and French) and low-resource African languages. We
observe a significant performance gap between open and proprietary models,
with the highest performing open model, Aya-101 only at 58% of the best-
performing proprietary model GPT-4o performance. Machine translating the test
set to English before evaluation helped to close the gap for larger models
that are English-centric, like LLaMa 3 70B. These findings suggest that more
efforts are needed to develop and adapt LLMs for African languages.

##  1 Introduction

In recent years, the capabilities of large language models (LLMs) have greatly
improved, from coherent chat experiences to solving complex and knowledge-
intensive tasks like mathematical reasoning, coding, and question answering
[47, 33, 26]. These models have also demonstrated the ability to quickly learn
new and challenging tasks with few in-context learning examples and through
chain-of-thought reasoning [17, 53, 64]. However, most state-of-the-art LLMs
are primarily trained on high-resource languages, resulting in sub-optimal
performance for languages unseen during pre-training [58, 45]. Furthermore,
this language coverage bias is reflected in the evaluation stage,
predominantly conducted in English and a few other high-resource languages.

There has been considerable effort to create benchmarks for African languages,
but they typically cover simpler tasks, or are specific to narrow domains such
as machine translation, and, more recently, reading comprehension [13, 11].
Some diverse reasoning benchmarks have included Swahili—the most spoken native
African language—in benchmarks like commonsense reasoning [48] and natural
language inference [21]. However, performance in Swahili does not reflect the
performance on the over 2,000+ languages in Africa, which often lack
substantial digital textual resources comparable to that available for
Swahili. Consequently, current multilingual evaluations of LLMs do not
accurately reflect capabilities in reasoning and knowledge-intensive tasks
across the majority of African languages. Furthermore, the few comprehensive
evaluations that exist across languages often rely on machine translation of
English benchmarks [55]. While automatic translation is a popular approach
given the cost and time investment required for human translation, it often
suffers from noise and biases. For example, translated datasets often
introduce artefacts from the translation model [61, 28, 51] or fail to reflect
cultural context appropriately [63, 32, 49]. Automatic curation may also
amplify any of the ubiquitous issues with the quality of broad pretraining
sets [39, 35, 24, 18].

In this paper, we seek to address both the _diversity and breadth_ of
evaluation coverage. We introduce IrokoBench, a human curated benchmark
dataset for 16 typologically diverse African languages which encompasses three
complex tasks: natural language inference (NLI), mathematical reasoning, and
multi-choice knowledge-based QA. The datasets were created by human
translating a subset of English cross-lingual NLI (XNLI) [21], English
Multilingual Grade School Math (MGSM) [54], and Massive Multitask Language
Understanding (MMLU) [29], evaluation datasets into each of the 16 languages
using professional translators.

We conduct a large-scale evalution of IrokoBench to evaluate zero-shot, few-
shot, and translate-test settings (where test sets are translated into
English) across 10 open and four proprietary LLMs. Our main contributions can
be enumerated as follows:

  1. 1.

We introduce and release IrokoBench, a human-translated benchmark that
includes languages from various geographical regions: six from West Africa,
five from East Africa, four from Southern Africa, and one from Central Africa,
all with varying degrees of “low-resourcedness.” [34].

  2. 2.

Sharp cliff in performance across all models on low-resource languages Our
evaluation shows a large gap (  $\sim$ $45$% on average) between the
performance of high-resource languages (e.g., English) and African languages
on all LLMs evaluated. Notably, Swahili performs better than other African
languages, likely due to its frequent use as a benchmark.

  3. 3.

Models generally perform poorly in in-language evaluation When evaluating
models with prompts in native languages, state-of-the-art open models such as
Gemma, Llama 3 are outperformed by massively multilingual models like Aya-101.
This can be attributed to a lack of trained models to respond in the native
languages of the users. Machine translating the test set to English before
evaluation helped to close the gap for English-centric models; however,
requiring users to always translate their prompts from their native language
to English may not be desirable behavior.

  4. 4.

IrokoBench highlights the performance divide between open and proprietary
models on low-resource languages. We find that proprietary closed models
generally outperform open models for African languages. However, even these
proprietary models exhibit substantial performance drops, due to the limited
monolingual web data for African languages. The lowest performance is observed
in languages such as Ewe, Lingala, Luganda, Twi and Wolof, which each have
less than 50 million characters of available data [36]. Among the tasks
evaluated, AfriMGSM proves most challenging for LLMs, followed by AfriMMLU and
AfriXNLI.

These results underline the need for focused development and adaptation of
LLMs to better support African languages, especially those with limited data
resources. We release IrokoBench on HuggingFace under the CC BY-SA 4.0 licence
https://huggingface.co/collections/masakhane/irokobench-665a21b6d4714ed3f81af3b1
to further multilingual evaluation and research.

##  2 Related Work

#### Multilingual Evaluation of LLMs:

The evaluation of multilingual capabilities of LLMs has garnered significant
attention. This has led to an increase in research that explores their
performance across diverse linguistic landscapes [7, 8, 37, 31, 14, 60, 55].
Despite this growing interest, there remains a notable lack of representation
of African low-resource languages in these studies. [45, 12] address a broader
spectrum of African languages, aligning more closely with our research.
However, their study focuses on conventional NLP tasks, such as text
classification, named entity recognition, question answering, and text
generation. To address the lack of difficult benchmarks, a few works
automatically translated MMLU benchmarks [47, 38, 60], but they do propagate
errors of machine translation (MT) engines. Moreover, this is not applicable
to low-resource languages with low-quality MT systems [2, 57]. Our research
advances this by evaluating LLMs on more complex tasks using newly developed,
human-annotated benchmarks specifically for African languages.

#### African Benchmark Datasets:

Due to the limited representation of African languages in the field of NLP,
there has been a growing effort to create benchmark datasets for African
languages to enable research on these languages, especially evaluating
existing NLP tools on these languages. Initiatives such as Masakhane have been
instrumental in the creation of standard benchmark for tasks such as machine
translation [2], named entity recognition [5, 3], part of speech tagging [23],
news topic classification [6], and sentiment analysis [43]. There are also
several multilingual benchmark datasets that cover a few African languages,
such as SIB-200 [4], Flores [27, 57], Aya dataset and Collection [55] and
Taxi1500 [40]. However, despite all these efforts, African languages still
lack quality and more difficult datasets to evaluate the reasoning
capabilities of LLMs, which have become ubiquitous. Hence, this work
contributes to this space by creating datasets to evaluate general use LLMs in
African languages in a variety of downstream tasks.

##  3 IrokoBench

Figure 1: Task Description for IrokoBench datasets. Both AfriMGSM and AfriMMLU
focus on QA, while AfriXNLI focuses on natural language inference between two
pairs of sentences. For clarity, this figure provides examples in English.
Table 1: The IrokoBench datasets: dataset name, number of African languages
covered, data split, and the subjects or domains covered. We included English,
French, and Swahili in all benchmarks.

| No. of | No. instances |
---|---|---|---
Dataset | languages | Train/Dev/Test |  Subjects / Domains
AfriMGSM | 15 (excl. Swahili) | 8 / - / 250 |  grade school mathematics
AfriMMLU | 17 (incl. French) | 25 / 83 / 500 |  elementary mathematics, high-school geography, International law, global facts, high school microeconomics
AfriXNLI | 15 (excl. Swahili) | \- / 450 / 600 |  face-to-face, telephone, oxford university press (oup), fiction, travel, government, nineeleven, letters, slate, verbatim

###  3.1 Languages covered by IrokoBench

We cover 18 languages in our evaluation, 16 native African languages, and two
European languages (English and French) which are widely spoken and official
in many African countries. English and French are also the source languages we
translated from. We cover 16 diverse and widely spoken African languages from
four regions of Africa: six from West Africa (Ewe, Hausa, Igbo, Twi, Wolof,
Yoruba), five from East Africa (Amharic, Kinyarwanda, Luganda, Swahili, and
Oromo), four from Southern Africa (chiShona, isiXhosa, isiZulu, and Sesotho),
and Central Africa (Lingala). The African languages are from two biggest
language families in Africa: three from Afro-Asiatic and 13 from the Niger-
Congo family—where we cover eight Bantu languages. Table 7 provides an
overview of the languages covered, including their family, regions, and the
number of native speakers.

###  3.2 Tasks covered by IrokoBench

The selection of these tasks is primarily driven by their coverage across
various domains and downstream tasks for diverse use cases. Additionally, they
enable the evaluation of logical, abstract, and reasoning capabilities in
LLMs, which is the hallmark of human intelligence [16, 30]. Figure 1 provides
examples of the different tasks covered in our datasets. We provide their
descriptions below:

#### AfriXNLI

The task of NLI involves the classification of a pair of sentences—a premise
and a hypothesis as to entailment, neutral, or contradiction semantic
relation. For example, the sentence “so I’m not really sure why” contradicts
“I am certain as to the reason why” but has a neutral relation to “I don’t
know why he transferred schools”. Here, we human translate the English portion
of XNLI (a multilingual dataset comprising 15 languages, including Swahili)
into the 15 African languages (excluding Swahili). While the original XNLI
dataset has over 2,500/5,000 as DEV/TEST split, Each language in AfriXNLI has
only 450 DEV instances and 600 test instances. We selected an equal number of
instances from the 10 domains of XNLI. The task is evaluated using the
accuracy metric since the dataset has balanced classes.

#### AfriMMLU

This is a multi-choice knowledge QA curated from freely available online
sources by undergraduate and graduate students in the USA. The subjects cover
simple general knowledge questions like “global fact” to highly-technical
questions like “professional law” and “professional medicine”. MMLU are often
grouped into STEM, Humanities, Social Sciences, and Others category. We focus
on five subjects that we believe present less tendency to be culturally
unbiased and that are not too difficult to translate since many of the
subjects covered are only taught in African countries using English or French,
making it extremely difficult to translate highly technical subjects,
especially the STEM subjects. Table 1 shows the five subjects covered: two
social science subjects (high-school geography and high-school
microeconomics), one STEM subject (elementary mathematics), one humanities
subject (international law), and one OTHER category (global facts). In total,
we translated 608 question-answer pairs, with 500 instances in the test split,
100 questions per subject. The task is evaluated using the option prediction
accuracy.

#### AfriMGSM

This is a QA task with questions obtained from grade school mathematical word
problems created by human problem writers. AfriMGSM expands the original MGSM
dataset [54], which contains 250 QA pairs and 11 languages (including
Swahili), to 15 more languages. The dataset consists of 8 training examples
for few-shot and chain-of-thought prompting and 250 as a test set. We
evaluated this task using the Exact Match metric, which is popularly used for
QA tasks.

###  3.3 Data collection process

#### Translation

We recruited language coordinators for each of the 16 African languages and
French, and asked them to recruit professional translators to translate the
sentences. The translation process took about two months, they started with
XNLI, then MGSM and MMLU. Each translator received an appropriate remuneration
for their work. 111We recruited a logistic company in Kenya that managed all
recruitment and payments—each country has different rates. Most of the
translators translated from English except for Ewe, Lingala and Wolof
translators that translated from French since they are from the Francophone
region of Africa. Additionally, we translated the MMLU dataset to French by
professional translators and from French to these three languages. Many of the
Francophone translators understand French and English but are more fluent in
French, so they could cross-check from English if the French sentences were
not clear enough.

#### Quality control

Regarding quality control, language coordinators reviewed and corrected any
poorly translated sentences. Translators received payment only after this
phase to ensure the quality of translations. For additional checks, we
computed COMET [50] quality estimation (QE) scores between the human
translation and the original sentences based on AfriCOMET QE metric [62]. In
general, the distribution of the scores (range between 0 and 1) reflect that
most translated sentences are between $0.7$ and $1.0$ for about 13 language
pairs except for Lingala, Twi, and Wolof where the average is around $0.5$.
Further analysis shows that we cannot rely on these scores for those three
languages since they are not covered in the pre-training of the original
AfroXLMR encoder [9] used to build the AfriCOMET QE metric. Similar findings
were reported in the original AfriCOMET QE paper that Twi had worse
correlation with human judgement (i.e., 0.279 for Pearson, and 0.060 for
Spearman) [62]. We provide further analysis of the COMET scores in Appendix
A.3.

###  3.4 LLMs used for evaluation

#### Open LLMs

We evaluate on three _encoder-decoder open LLM_ : Flan-T5-XXL [19], mT0-XXL-MT
[42], and Aya-101 [59] that have been instruction fine-tuned on T5 (FlanT5)
and multilingual T5 pre-trained on 101 languages (mT0 and Aya-101) models.
Furthermore, these models are also all designed to be _massively multilingual_
and explicitly optimized to work outside of English. The instruction data were
constructed from several supervised NLP tasks with diverse prompts in
different languages. Aya-101 additionally includes some synthetic data
generated from Cohere Command model [20], and a large amount of translated
data from English to cover several languages [55]. The languages covered
during instruction tuning differ for different models, mT0, FlanT5, and Aya
covered 46, 60, and 101 languages respectively.

Additionally we evaluate of seven _decoder-only open LLM_ models: BLOOMZ 7B
[65], Gemma 7B [56], LLaMa 2 7B [58], LLaMa 3 (8B & 70B) [41], Command-(R &
R+) [20]. These models’ weights are openly available under various licenses,
ranging from fully permissive to non-commercial, research-only licenses. We
evaluate the instruction-tuned variant of these models. Generally, with the
exception of BLOOMZ, these models did not report including any African
languages in their pre-training. BLOOMZ included 22 African languages, mainly
from Niger-Congo language family.

#### Closed LLM

We limit our evaluation to only OpenAI GPT (3.5-0125, 4-Turbo-0125, 4o) [46],
and Claude OPUS [10] models. Recent work has shown that proprietary models
tend to exhibit better multilingual capabilities [8], although specifics
regarding their pre-training and instruction fine-tuning processes are not
disclosed.

###  3.5 Evaluation settings

#### Evaluation Set-up

We conduct two types of evaluations: in-language evaluation and translate-test
evaluation, where test instances are automatically translated into English
using a machine translation engine. For automatic translation, we use NLLB-200
(3.3B) [57]. In both in-language and translate-test setups, we perform cross-
lingual transfer experiments from English and zero-shot evaluations by
prompting LLMs. Few-shot evaluations are performed only for the two best
models (one open and one closed model) in the in-language setting.

We use the EleutherAI LM Evaluation Harness (lm-eval) tool [15]—a popular
evaluation tool that is helping to standardize LLM evaluation, especially for
open models on HuggingFace Model Hub. For closed models, except for the
AfriMGSM task where we still use lm-eval, we employ a verbalizer [25, 52] for
prediction and evaluation. We provide more details in Appendix A.2.

#### Cross-lingual transfer experiments

We first conduct a study on cross-lingual transfer in a supervised learning
setting by fine-tuning the English training data ($400K$ instances) from
Conneau et al. [21] and evaluating on the remaining languages. This experiment
focuses solely on the NLI task due to the availability of training data for
supervised learning. The evaluation employs several masked language models,
including XLM-R [22], Serengeti [1], AfroXLMR-{base, large} [9], AfroXLMR-76L
[4]. We report the result of the best model in the paper and others in
Appendix A.5.

#### Zero- and few-shot evaluation

In a zero-shot setting, we use the prompts detailed in Table 8. For few-shot
evaluations, we conduct a 5-shot assessment for both AfriMMLU and AfriXNLI,
and an 8-shot assessment for AfriMGSM.

Table 2: Main results: Average performance of various LLMs on AfriMMLU,
AfriXNLI and AfriMGSM. Except for AfriMGSM, which uses the Exact Match metric,
others use the Accuracy.

|  | AfriXNLI | AfriMMLU | AfriMGSM | Ave. | Ave. | Ave. | Ave.
---|---|---|---|---|---|---|---|---
|  | in- | translate | in- | translate | in- | translate | in- | translate | English | French
Model | size | lang. | test | lang. | test | lang. | test | lang. | test | lang. | lang.
AfroXLMR-76L | 559M | 65.7 | 63.6 | – | – | – | – | 65.7 | 63.6 | 88.2 | 83.3
FlanT5-XXL | 11B | 36.7 | 53.2 | 25.4 | 35.1 | 1.1 | 8.2 | 21.1 | 32.2 | 44.7 | 35.4
mT0-XXL-MT | 13B | 51.0 | 49.9 | 28.2 | 28.9 | 2.6 | 2.6 | 27.3 | 27.1 | 34.4 | 33.3
Aya-101 | 13B | 49.9 | 49.4 | 30.3 | 31.1 | 3.3 | 6.6 | 27.9 | 29.0 | 36.2 | 35.0
BLOOMZ 7B | 7B | 39.4 | 47.5 | 23.5 | 26.4 | 4.1 | 2.1 | 22.4 | 25.3 | 32.0 | 29.9
Gemma 7B | 7B | 36.4 | 42.0 | 29.5 | 38.3 | 4.6 | 8.2 | 23.5 | 29.5 | 41.2 | 37.6
LLaMa 2 7B | 7B | 33.4 | 42.3 | 24.7 | 29.9 | 1.7 | 5.9 | 19.9 | 26.0 | 34.8 | 32.3
LLaMa 3 8B | 8B | 35.8 | 44.1 | 27.5 | 34.4 | 2.9 | 27.0 | 22.0 | 35.2 | 47.0 | 39.5
LLaMa 3 70B | 70B | 34.2 | 40.9 | 34.1 | 47.4 | 8.2 | 45.3 | 25.5 | 44.5 | 70.7 | 57.7
Command R | 35B | 34.0 | 46.1 | 27.0 | 36.5 | 1.6 | 5.8 | 20.9 | 29.5 | 42.1 | 38.4
Command R+ | 104B | 37.9 | 45.4 | 26.1 | 35.7 | 2.3 | 9.2 | 22.1 | 30.1 | 47.8 | 39.3
Claude OPUS | UNK | 58.1 | 56.4 | 43.0 | 47.6 | 25.9 | 32.7 | 42.3 | 45.6 | 73.3 | 64.0
GPT-3.5 Turbo | UNK | 41.5 | 52.2 | 31.8 | 39.6 | 5.3 | 19.3 | 26.2 | 37.0 | 64.7 | 54.1
GPT-4-Turbo | UNK | 60.0 | 52.8 | 50.9 | 47.0 | 26.8 | 25.1 | 45.9 | 41.6 | 81.7 | 74.1
GPT-4o | UNK | 62.7 | 59.9 | 52.3 | 52.3 | 29.3 | 32.4 | 48.1 | 48.2 | 72.5 | 65.1
Average |  |  |  |  |  |  |  | 28.2 | 34.3 | 51.6 | 45.4

##  4 Results

#### Large performance gaps between high-resource languages and African
languages

Table 2 shows the result of zero-shot evaluation for various LLMs. On average,
there is a significant performance gap between African languages and English
(up to $45\%$) and French (up to $36\%$). The best model for African languages
is GPT-4o, with an average performance of 48.1 across the evaluated tasks.
However, GPT-4-Turbo outperforms GPT-4o for English and French, showing
improvements of 9.2 and 9.0, respectively, across all tasks. It appears that
GPT-4o’s enhancements for low-resource languages may negatively impact its
performance on high-resource languages. Finally, as shown in Table 7, the
languages with the lowest performance have the least monolingual data on the
web.

#### Large performance gaps between closed and open weights models

Our results, as presented in Table 2, indicate that the closed models Claude
OPUS, GPT-4-Turbo, and GPT-4o consistently outperform the open models on our
IrokoBench benchmarks. The closed models achieve average performance scores
ranging from $42.3$ to $48.1$ across all tasks. The performance gap between
the best closed model (GPT-4o) and the best open model (Aya-101) is $20.2$.
Notably, the largest performance differences are observed in the AfriMMLU and
AfriMGSM tasks, where GPT-4o outperforms LLaMa 3 70B by $18.2$ and $21.1$,
respectively. For the AfriXNLI task, mT0-XXL-MT and Aya-101 perform better
than LLaMa 3 70B and also achieve the highest scores among all open-weight
models.

Majority models perform worse for in-language prompting Most users would
prefer to prompt in their native language; however, we find that almost all
models we benchmark perform better with prompts translated into English. Only
a few exceptions, such as GPT-4-Turbo, mT0-XXL-MT, and Aya-101, perform better
in the in-language evaluation. Among open-source models, Aya-101 performs the
best in the in-language evaluation, surpassing much larger open-weight models
like LLaMa 3 70B. Specifically, LLaMa 3 70B and LLaMa 3 8B benefit the most
from the translate-test approach, showing average improvements of $19.0$ and
$13.1$, respectively. Notably, LLaMa 3 70B achieves the best overall results
for the AfriMGSM task with the translate-test, outperforming GPT-4o by more
than $12.9$. Similarly, we observe significant performance gains for the
FlanT5-XXL and GPT-3.5 Turbo models on the AfriXNLI and AfriMGSM tasks when
using the translate-test. We attribute this boost to the fact that these
models are heavily English centric.

Table 3: AfriMMLU results in in-language and translate-test scenarios: Option
prediction accuracy per language. Average computed on only African languages.

Model | eng | fra | amh | ewe | hau | ibo | kin | lin | lug | orm | sna | sot | swa | twi | wol | xho | yor | zul | avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Prompt LLMs in African Language |  |  |  |  |  |  |  |  |  |  |
Aya | 41.8 | 36.4 | 32.8 | 26.8 | 32.2 | 35.8 | 30.0 | 26.0 | 30.8 | 30.6 | 27.8 | 34.2 | 35.0 | 25.8 | 24.6 | 31.2 | 29.8 | 31.8 | 30.3
LLaMa 3 8B | 49.0 | 45.6 | 26.8 | 25.8 | 29.6 | 29.4 | 28.4 | 28.8 | 22.4 | 29.2 | 29.4 | 27.0 | 30.4 | 24.8 | 27.6 | 24.6 | 28 | 28.0 | 27.5
LLaMa 3 70B | 74.4 | 68.6 | 33.0 | 31.8 | 33.6 | 37.8 | 32.4 | 35.0 | 30.8 | 33.0 | 33.4 | 35.2 | 44.8 | 33.7 | 32.0 | 30.0 | 33.8 | 35.0 | 34.1
GPT-4 Turbo | 80.7 | 76.5 | 37.7 | 23.9 | 55.2 | 54.5 | 54.9 | 46.8 | 40.9 | 39.8 | 54.1 | 55.4 | 61.0 | 30.1 | 31.0 | 56.2 | 48.9 | 58.4 | 50.9
GPT-4o | 78.6 | 71.4 | 55.2 | 31.6 | 57.4 | 55.8 | 56.8 | 55.4 | 45.0 | 48.8 | 58.4 | 62.0 | 67.4 | 37.6 | 32.6 | 60.8 | 50.8 | 60.6 | 52.3
Translate-Test (Eval. in English) |  |  |  |  |  |  |  |  |  |  |
Aya |  | 37.2 | 32.0 | 30.4 | 30.8 | 31.6 | 29.2 | 32.6 | 29.2 | 29.0 | 33.4 | 33.8 | 31.6 | 30.8 | 25.4 | 32.8 | 32.6 | 32.4 | 31.1
LLaMa 3 8B |  | 43.4 | 38.4 | 35.2 | 34.4 | 33.2 | 36.0 | 36.0 | 27.8 | 34.2 | 31.4 | 38.8 | 43.6 | 29.0 | 29.6 | 36.6 | 33.2 | 33.6 | 34.4
LLaMa 3 70B |  | 64.4 | 54.4 | 42.2 | 44.4 | 48.6 | 47.2 | 49.8 | 41.2 | 49.6 | 48.4 | 54.0 | 60.0 | 41.4 | 29.8 | 49.6 | 49.2 | 49.2 | 47.4
GPT-4 Turbo |  | 61.0 | 50.0 | 37.6 | 43.2 | 47.6 | 48.2 | 46.8 | 40.4 | 44.6 | 46.2 | 53.0 | 72.7 | 36.4 | 28.0 | 48.4 | 46.6 | 49.0 | 47.0
GPT-4o |  | 67.2 | 57.6 | 40.6 | 49.4 | 56.4 | 55.0 | 50.2 | 41.8 | 51.6 | 49.8 | 59.0 | 80.0 | 37.2 | 33.0 | 53.0 | 52.4 | 54.2 | 52.3

Table 4: AfriMGSM results in in-language and translate-test scenarios: Exact
Match score per language. Average computed on only African languages

Model | eng | fra | amh | ewe | hau | ibo | kin | lin | lug | orm | sna | sot | swa | twi | wol | xho | yor | zul | avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Prompt LLMs in African Language |  |  |  |  |  |  |  |  |  |  |
Aya | 24.8 | 22.0 | 7.6 | 2.8 | 5.6 | 4.0 | 6.8 | 2.0 | 4.4 | 2.4 | 3.2 | 3.2 | 11.2 | 1.2 | 2.4 | 2.0 | 3.6 | 4.0 | 4.2
LLaMa 3 8B | 24.8 | 22.0 | 7.6 | 2.8 | 4.8 | 4.4 | 5.6 | 2.0 | 4.8 | 2.4 | 2.4 | 3.2 | 11.2 | 1.2 | 2.4 | 1.6 | 3.2 | 3.6 | 4.0
LLaMa 3 70B | 86.4 | 54.4 | 5.2 | 2.8 | 15.6 | 9.2 | 8.4 | 5.6 | 8.0 | 4.0 | 4.8 | 6.0 | 34.8 | 3.2 | 4.0 | 4.4 | 7.2 | 7.6 | 8.2
GPT-4 Turbo | 79.6 | 64.0 | 8.4 | 6.8 | 40.8 | 21.6 | 41.6 | 30.8 | 24.4 | 31.2 | 38.4 | 38.0 | 54.4 | 6.8 | 9.2 | 31.2 | 29.2 | 31.2 | 27.8
GPT-4o | 58.4 | 43.6 | 4.4 | 5.6 | 42.0 | 32.0 | 36.4 | 30.8 | 22.8 | 39.2 | 33.2 | 28.8 | 44.8 | 10.4 | 12.4 | 29.2 | 30.8 | 28.4 | 27.0
Translate-Test (Eval. in English) |  |  |  |  |  |  |  |  |  |  |
Aya |  | 8.0 | 8.0 | 6.0 | 7.6 | 4.0 | 7.2 | 5.2 | 4.4 | 5.2 | 8.4 | 8.0 | 10.8 | 4.4 | 5.6 | 6.0 | 8.4 | 6.4 | 6.6
LLaMa 3 8B |  | 44.0 | 31.6 | 18.0 | 27.2 | 24.0 | 34.0 | 27.2 | 23.2 | 28.8 | 26.8 | 33.6 | 38.0 | 19.6 | 12.8 | 28.8 | 29.6 | 28.8 | 27.0
LLaMa 3 70B |  | 69.6 | 53.6 | 27.6 | 48.4 | 40.4 | 54.0 | 47.6 | 34.0 | 47.6 | 46.8 | 54.0 | 69.6 | 29.2 | 20.8 | 50.0 | 49.2 | 51.2 | 45.3
GPT-4 Turbo |  | 63.6 | 42.0 | 20.4 | 39.2 | 35.2 | 47.6 | 41.6 | 27.2 | 38.8 | 39.2 | 46.0 | 56.8 | 23.2 | 12.4 | 41.2 | 42.0 | 40.4 | 37.1
GPT-4o |  | 52.0 | 18.4 | 37.6 | 36.4 | 28.0 | 34.8 | 38.4 | 28.4 | 32.8 | 32.4 | 36.8 | 45.2 | 17.6 | 12.4 | 31.6 | 36.0 | 38.0 | 31.6

Table 5: AfriXNLI results in in-language and translate-test scenarios: Option
prediction accuracy per language. Average computed on only African languages.

Model | eng | fra | amh | ewe | hau | ibo | kin | lin | lug | orm | sna | sot | swa | twi | wol | xho | yor | zul | avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Prompt LLMs in African Language |  |  |  |  |  |  |  |  |  |  |
AfroXLMR-76L | 88.2 | 83.3 | 78.5 | 58.3 | 73.3 | 70.0 | 65.8 | 33.3 | 68.0 | 69.3 | 70.8 | 70.8 | 73.3 | 59.5 | 51.8 | 73.0 | 63.2 | 72.5 | 65.7
Aya | 61.5 | 60.5 | 58.2 | 42.2 | 56.8 | 54.7 | 48.7 | 34.5 | 44.5 | 51.7 | 55.8 | 54.7 | 55.2 | 48.2 | 37.3 | 53.7 | 48.3 | 54.3 | 49.9
LLaMa 3 8B | 54.0 | 51.8 | 34.5 | 33.8 | 36.0 | 39.5 | 33.8 | 32.5 | 35.5 | 34.7 | 37.8 | 35.0 | 39.5 | 33.0 | 33.5 | 38.1 | 38.1 | 36.8 | 35.8
LLaMa 3 70B | 51.3 | 50.2 | 36.8 | 33.3 | 35.3 | 36.5 | 34.5 | 33.0 | 33.5 | 33.3 | 33.8 | 34.3 | 36.5 | 33.0 | 33.5 | 34.0 | 33.0 | 33.0 | 34.2
GPT-4 Turbo | 86.0 | 82.2 | 65.2 | 38.7 | 69.3 | 69.5 | 67.7 | 32.2 | 65.0 | 58.3 | 67.5 | 65.7 | 67.2 | 47.5 | 39.7 | 72.0 | 65.3 | 69.2 | 60.0
GPT-4o | 86.2 | 78.7 | 66.7 | 48.3 | 69.2 | 68.2 | 66.8 | 31.2 | 67.2 | 66.2 | 69.8 | 68.3 | 72.5 | 53.2 | 49.5 | 72.5 | 63.5 | 70.0 | 62.7
Translate-Test (Eval. in English) |  |  |  |  |  |  |  |  |  |  |
AfroXLMR-76L |  | 83.0 | 73.7 | 54.3 | 67.2 | 66.0 | 63.0 | 32.8 | 65.7 | 65.8 | 71.2 | 70.2 | 73.0 | 56.8 | 47.5 | 74.2 | 63.7 | 72.0 | 63.6
Aya |  | 60.3 | 55.2 | 47.0 | 51.8 | 52.2 | 50.3 | 33.0 | 49.0 | 49.0 | 52.0 | 53.3 | 54.0 | 45.5 | 41.5 | 54.5 | 48.8 | 54.0 | 49.4
LLaMa 3 8B |  | 52.2 | 48.3 | 43.3 | 43.2 | 44.8 | 45.3 | 33.7 | 46.0 | 45.2 | 45.5 | 47.5 | 47.5 | 40.5 | 38.7 | 44.5 | 44.2 | 47.0 | 44.1
LLaMa 3 70B |  | 48.5 | 42.8 | 41.3 | 43.7 | 39.8 | 40.3 | 33.3 | 43.3 | 39.7 | 41.1 | 43.0 | 44.3 | 39.7 | 38.7 | 42.3 | 38.2 | 42.7 | 40.9
GPT-4 Turbo |  | 75.2 | 64.0 | 43.0 | 52.8 | 59.0 | 55.8 | 31.7 | 51.0 | 52.5 | 55.8 | 58.5 | 60.2 | 45.0 | 39.7 | 63.2 | 52.5 | 59.5 | 52.8
GPT-4o |  | 80.0 | 69.0 | 50.7 | 62.2 | 63.3 | 61.0 | 33.0 | 63.2 | 58.7 | 65.8 | 67.0 | 69.5 | 51.3 | 46.2 | 69.7 | 60.5 | 67.3 | 59.9

Table 6: Ablation results: Effect of using different prompts for AfriXNLI.
Best results per prompt are in bold. Average computed on only African
languages.

Model | eng | fra | amh | ewe | hau | ibo | kin | lin | lug | orm | sna | sot | swa | twi | wol | xho | yor | zul | avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
ANLI prompt [44]—simple prompt |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
Aya | 61.5 | 60.5 | 58.2 | 42.2 | 56.8 | 54.7 | 48.7 | 34.5 | 44.5 | 51.7 | 55.8 | 54.7 | 55.2 | 48.2 | 37.3 | 53.7 | 48.3 | 54.3 | 49.9
Gemma 7B | 51.5 | 47.8 | 34.3 | 35.3 | 38.8 | 39 | 35.8 | 33.2 | 34.8 | 35.7 | 38.8 | 35.7 | 39.3 | 37.2 | 35.8 | 37.2 | 36.5 | 35.7 | 36.4
LLaMA 3 70B | 51.3 | 50.2 | 36.8 | 33.3 | 35.3 | 36.5 | 34.5 | 33 | 33.5 | 33.3 | 33.8 | 34.3 | 36.5 | 33 | 33.5 | 34 | 33 | 33 | 34.2
Opus | 69.3 | 67.5 | 54.8 | 41.3 | 55.5 | 52.8 | 46.8 | 27.8 | 51.3 | 52.7 | 50.2 | 56.8 | 57.8 | 46.7 | 41.0 | 57.7 | 53.3 | 54.5 | 50.1
GPT-4o | 79.8 | 74.5 | 61.8 | 42.0 | 61.0 | 58.2 | 55.8 | 33.2 | 58.0 | 58.3 | 62.2 | 62.0 | 65.5 | 49.2 | 46.5 | 62.5 | 55.2 | 59.8 | 55.7
Lai’s prompt [37]—detailed prompt |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
Aya | 49.4 | 45.1 | 47.1 | 32.3 | 44.2 | 43.8 | 39.1 | 21.5 | 40.3 | 38.5 | 41.7 | 43.7 | 43.7 | 38 | 27.3 | 46.2 | 39.8 | 43.2 | 39.4
Gemma 7B | 64 | 58.2 | 19.5 | 19.1 | 24.2 | 25.8 | 22.2 | 24.2 | 20.2 | 20.6 | 25.8 | 20.3 | 46.7 | 19.6 | 19 | 31.1 | 20.3 | 25.8 | 24.0
LLaMA 3 70B | 72.8 | 51.4 | 41.9 | 27.5 | 47.1 | 43 | 39.8 | 23.9 | 38 | 38.2 | 37.9 | 37.4 | 52.1 | 22.1 | 29.1 | 39 | 36.7 | 36.3 | 36.9
Opus | 85.7 | 74.7 | 61.3 | 54.5 | 61.5 | 45.0 | 64.5 | 30.5 | 63.7 | 50.2 | 57.0 | 68.3 | 70.5 | 56.0 | 50.0 | 68.8 | 63.7 | 63.7 | 58.1
GPT-4o | 86.2 | 78.7 | 66.7 | 48.3 | 69.2 | 68.2 | 66.8 | 31.2 | 67.2 | 66.2 | 69.8 | 68.3 | 72.5 | 53.2 | 49.5 | 72.5 | 63.5 | 70.0 | 62.7

###  4.1 Task-specific results

Here, we examine the individual language performance per task, comparing which
task prefers in-language v.s. translate-test evaluation. We compare the
performance on a subset of LLMs: Aya-101, LLaMa 3 (8B & 70B), GPT-4-Turbo, and
GPT-4o. Other LLMs are in Appendix A.4.

#### AfriMMLU evaluation is better when prompting in-language

Table 3 shows the result of different LLMs on AfriMMLU using in-language vs.
translate-test. We found in-language prompting to be better on 10 out of the
16 African languages we evaluated; ibo, orm, wol, and yor have comparable
performance with two approaches. However, we find a large improvement in the
performance of the translate-test for swa and ewe with a boost of $12.4$
(GPT-4o comparison) and $10.4$ (LLaMa 3 70B comparison), respectively. For
Swahili (swa), this improvement may be due to the strong machine translation
performance on NLLB-200 , as reported in Adelani et al. [2] and Wang et al.
[62].

#### AfriMGSM performance receives significant boost with translate-test
evaluation

AfriMGSM is our most difficult benchmark in terms of overall performance on
in-language evaluation. Table 4 shows that we can achieve a significant boost
in performance with the translate-test on all languages we evaluated. We
hypothesize that the current LLMs are better at reasoning in English than any
other language. Interestingly, LLaMa 3 70B has the overall best performance
boost of approximately $82\%$, and the performance boost confirms our
hypothesis since it is more English-centric.

###  4.2 Few-shot results, Cross-Lingual Transfer, Sensitivity to Prompt
Templates

#### Cross-lingual transfer in-language achieves better results

When there is large enough labeled data in English, we could leverage this
cross-lingual signal for zero-shot evaluation. We trained on 400k English NLI
examples, and we performed zero-shot transfer in in-language and translate-
test setting. Table 5 shows that cross-lingual transfer using an Africa-
centric smaller language model (AfroXLMR-76L) gave better results than
prompting LLMs on average. AfroXLMR-76L has been pre-trained on all languages
in IrokoBench, which explains the impressive performance. However, for
multilingual encoders that have not seen some of the languages, prompting
GPT-4o seems to be better, as shown in Appendix A.5.

#### Impact of few-shot vs zero-shot

Figure 2 shows the few-shot results for the IrokoBench datasets leveraging
Aya-101, LLaMa 3 70B and GPT-4o when we provide few examples in in-language
setting. We found out that Aya-101 LLM consistently benefited the least from
additional few shots examples in all cases with only $+1.4$, $+2.0$ and $+0.1$
respectively. On the other hand, we see a large boost LLaMa 3 70B on both
AfriXNLI and AfriMMLU tasks with $12.4$ and $6.5$ points improvement
respectively. We hypothesize that additional examples in the language LLaMa 3
70B was not trained on were particularly helpful for the classification tasks.
However, for mathematical reasoning tasks, this was unhelpful since LLaMa 3
70B is only able to reason properly in English as shown in Table 4. In
general, GPT-4o benefits from adddtional few-shot examples, there is large
improvement for AfriMGSM ($+10.5$) but smaller points improvement on AfriXNLI
and AfriMMLU are smaller with $+2.0$ and $+0.4$ respectively.

One possibility for the difference in added performance from few-shot in part
is due to the difference in context length. Aya-101 was by necessity built
upon the mT5 [66] pre-trained base model given it was one of the few pre-
trained models that had been trained on 101 languages. mT5 is relatively
outdated given the rapid advances in LLM technology since its release in 2019.
In particular, the context length of Aya-101 is only 1024 versus 8000 for
LLaMa 3 70B and 32000 for GPT-4o. Hence, the truncation of some few-shot
examples may disproportionately impact Aya-101.

#### Sensitivity to prompt templates

To understand whether sensitivity to prompts impacts results we perform an
ablation for NLI where we evaluate results given two different prompts. Table
6 shows the results of two prompts we tested for: the first one is simpler
[44], where the model is required to output true, false or neither, while the
second prompt provides a detailed task description [37], and the output we are
expecting is the label entailment. contradiction or neutral. We found out that
open models like Aya-101 and Gemma 7B prefer the simpler prompt with Aya-101
achieving close to $50.0$ accuracy points on the task on average. However,
closed models like Claude OPUS and GPT-4o prefer the Lai et al. [37] detailed
prompt with $8.0$ and $7.0$ improvement in performance over the simpler
prompt. In general, regardless of the prompt selection, GPT-4o is better than
open models, but we can achieve better results for open models by providing a
more detailed task description. While this evaluation is only for two prompts,
an extensive prompt search would be important to explore all the tasks we
evaluated.

(a) AfriXNLI

(b) AfriMMLU

(c) AfriMGSM

Figure 2: Few-shot evaluation on IrokoBench datasets, we performed 8-shots for
AfriMGSM and 5-shots for the others.

##  5 Conclusion

In this paper, we introduced IrokoBench, a new benchmark for evaluating large
language models (LLMs) on African languages. IrokoBench comprises three
datasets focused on different tasks: natural language inference (AfriXNLI),
multi-choice knowledge QA (AfriMMLU), and mathematical reasoning (AfriMGSM).
Unlike previous benchmarks, which primarily involve simple text classification
tasks, these datasets assessed the LLMs’ abilities in complex and knowledge-
intensive areas. Our evaluation revealed a significant performance gap between
high-resource languages (e.g., English and French) and African languages.
Additionally, we observed a substantial disparity in performance between open
models and proprietary models, with the latter generally outperforming the
former, particularly in mathematical reasoning tasks. We hope that IrokoBench
will serve as a valuable benchmark for evaluating future LLMs developed or
adapted for African languages.

#### Limitations

Our benchmark has a few limitations: (1) The benchmark is human-translated
which may include some translationese effects, it would have been better if
they are all generated in the native African languages. However, this parallel
translation allows us to evaluate and compare the same sentences in all these
languages. (2) We only cover three language families in Africa, Nilo-Saharan,
Austronesian, and Khoisan language groups are missing, one of the reason we
excluded them is either lack of contact with professional translators or
limited translation budget, we hope to extend to more languages in the future.
(3) Prompts used for evaluation are not exhaustive some of the results may
change with appropriate a more careful prompt search. The best prompt for
different LLMs might be different.

## Acknowledgment

This work was carried out with support from Lacuna Fund, an initiative co-
founded by The Rockefeller Foundation, Google.org, and Canada’s International
Development Research Centre. David Adelani acknowledges the support of
DeepMind Academic Fellowship programme. This work was supported in part by
compute credits from Oracle and Cohere For AI Research Grant. Also, we are
grateful to OpenAI for providing API credits through their Researcher Access
API programme to Masakhane for the evaluation of GPT-3.5 and GPT-4 LLMs.
Finally, we are grateful to Hailey Schoelkopf and Lintang Sutawika for their
assistance with the EleutherAI lm-eval tool.

## References

  * Adebara et al. [2023] Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Alcides Alcoba Inciarte.  SERENGETI: Massively multilingual language models for Africa.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 1498–1537, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-acl.97.  URL https://aclanthology.org/2023.findings-acl.97.
  * Adelani et al. [2022a] David Adelani, Jesujoba Alabi, Angela Fan, Julia Kreutzer, Xiaoyu Shen, Machel Reid, Dana Ruiter, Dietrich Klakow, Peter Nabende, Ernie Chang, Tajuddeen Gwadabe, Freshia Sackey, Bonaventure F. P. Dossou, Chris Emezue, Colin Leong, Michael Beukman, Shamsuddeen Muhammad, Guyo Jarso, Oreen Yousuf, Andre Niyongabo Rubungo, Gilles Hacheme, Eric Peter Wairagala, Muhammad Umair Nasir, Benjamin Ajibade, Tunde Ajayi, Yvonne Gitau, Jade Abbott, Mohamed Ahmed, Millicent Ochieng, Anuoluwapo Aremu, Perez Ogayo, Jonathan Mukiibi, Fatoumata Ouoba Kabore, Godson Kalipe, Derguene Mbaye, Allahsera Auguste Tapo, Victoire Memdjokam Koagne, Edwin Munkoh-Buabeng, Valencia Wagner, Idris Abdulmumin, Ayodele Awokoya, Happy Buzaaba, Blessing Sibanda, Andiswa Bukula, and Sam Manthalu.  A few thousand translations go a long way! leveraging pre-trained models for African news translation.  In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors, _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3053–3070, Seattle, United States, July 2022a. Association for Computational Linguistics.  doi: 10.18653/v1/2022.naacl-main.223.  URL https://aclanthology.org/2022.naacl-main.223.
  * Adelani et al. [2022b] David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba Alabi, Shamsuddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye, Amelia Taylor, Fatoumata Kabore, Chris Chinenye Emezue, Anuoluwapo Aremu, Perez Ogayo, Catherine Gitau, Edwin Munkoh-Buabeng, Victoire Memdjokam Koagne, Allahsera Auguste Tapo, Tebogo Macucwa, Vukosi Marivate, Mboning Tchiaze Elvis, Tajuddeen Gwadabe, Tosin Adewumi, Orevaoghene Ahia, and Joyce Nakatumba-Nabende.  MasakhaNER 2.0: Africa-centric transfer learning for named entity recognition.  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 4488–4508, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguistics.  doi: 10.18653/v1/2022.emnlp-main.298.  URL https://aclanthology.org/2022.emnlp-main.298.
  * Adelani et al. [2024] David Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba Alabi, Yanke Mao, Haonan Gao, and En-Shiun Lee.  SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects.  In Yvette Graham and Matthew Purver, editors, _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 226–245, St. Julian’s, Malta, March 2024. Association for Computational Linguistics.  URL https://aclanthology.org/2024.eacl-long.14.
  * Adelani et al. [2021] David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Jesujoba Alabi, Seid Muhie Yimam, Tajuddeen Rabiu Gwadabe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, Verrah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chukwuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akinode, Deborah Nabagereka, Maurice Katusiime, Ayodele Awokoya, Mouhamadane MBOUP, Dibora Gebreyohannes, Henok Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulaye Faye, Blessing Sibanda, Orevaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thierno Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Salomey Osei.  MasakhaNER: Named entity recognition for African languages.  _Transactions of the Association for Computational Linguistics_ , 9:1116–1131, 2021.  doi: 10.1162/tacl_a_00416.  URL https://aclanthology.org/2021.tacl-1.66.
  * Adelani et al. [2023] David Ifeoluwa Adelani, Marek Masiak, Israel Abebe Azime, Jesujoba Alabi, Atnafu Lambebo Tonja, Christine Mwase, Odunayo Ogundepo, Bonaventure F. P. Dossou, Akintunde Oladipo, Doreen Nixdorf, Chris Chinenye Emezue, Sana Al-azzawi, Blessing Sibanda, Davis David, Lolwethu Ndolela, Jonathan Mukiibi, Tunde Ajayi, Tatiana Moteu, Brian Odhiambo, Abraham Owodunni, Nnaemeka Obiefuna, Muhidin Mohamed, Shamsuddeen Hassan Muhammad, Teshome Mulugeta Ababu, Saheed Abdullahi Salahudeen, Mesay Gemeda Yigezu, Tajuddeen Gwadabe, Idris Abdulmumin, Mahlet Taye, Oluwabusayo Awoyomi, Iyanuoluwa Shode, Tolulope Adelani, Habiba Abdulganiyu, Abdul-Hakeem Omotayo, Adetola Adeeko, Abeeb Afolabi, Anuoluwapo Aremu, Olanrewaju Samuel, Clemencia Siro, Wangari Kimotho, Onyekachi Ogbu, Chinedu Mbonu, Chiamaka Chukwuneke, Samuel Fanijo, Jessica Ojo, Oyinkansola Awosan, Tadesse Kebede, Toadoum Sari Sakayo, Pamela Nyatsine, Freedmore Sidume, Oreen Yousuf, Mardiyyah Oduwole, Kanda Tshinu, Ussen Kimanuka, Thina Diko, Siyanda Nxakama, Sinodos Nigusse, Abdulmejid Johar, Shafie Mohamed, Fuad Mire Hassan, Moges Ahmed Mehamed, Evrard Ngabire, Jules Jules, Ivan Ssenkungu, and Pontus Stenetorp.  MasakhaNEWS: News topic classification for African languages.  In Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, and Adila Alfa Krisnadhi, editors, _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 144–159, Nusa Dua, Bali, November 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.ijcnlp-main.10.  URL https://aclanthology.org/2023.ijcnlp-main.10.
  * Ahuja et al. [2023a] Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram.  MEGA: Multilingual evaluation of generative AI.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 4232–4267, Singapore, December 2023a. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.258.  URL https://aclanthology.org/2023.emnlp-main.258.
  * Ahuja et al. [2023b] Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram.  Megaverse: Benchmarking large language models across languages, modalities, models and tasks.  _ArXiv_ , abs/2311.07463, 2023b.  URL https://api.semanticscholar.org/CorpusID:265150423.
  * Alabi et al. [2022] Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow.  Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning.  In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na, editors, _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 4336–4349, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics.  URL https://aclanthology.org/2022.coling-1.382.
  * Anthropic [2024] Anthropic.  Claude — anthropic.com.  https://www.anthropic.com/claude, 2024.  [Accessed 01-06-2024].
  * Aremu et al. [2023] Anuoluwapo Aremu, Jesujoba O. Alabi, and David Ifeoluwa Adelani.  Yorc: Yoruba reading comprehension dataset, 2023.
  * Azime et al. [2024] Israel Abebe Azime, Mitiku Yohannes Fuge, Atnafu Lambebo Tonja, Tadesse Destaw Belay, Aman Kassahun Wassie, Eyasu Shiferaw Jada, Yonas Chanie, Walelign Tewabe Sewunetie, and Seid Muhie Yimam.  Enhancing amharic-llama: Integrating task specific and generative datasets.  _arXiv preprint arXiv:2402.08015_ , 2024.
  * Bandarkar et al. [2023] Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa.  The belebele benchmark: a parallel reading comprehension dataset in 122 language variants, 2023.
  * Bang et al. [2023] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung.  A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity.  In Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, and Adila Alfa Krisnadhi, editors, _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 675–718, Nusa Dua, Bali, November 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.ijcnlp-main.45.  URL https://aclanthology.org/2023.ijcnlp-main.45.
  * Biderman et al. [2024] Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A. Wang, Genta Indra Winata, François Yvon, and Andy Zou.  Lessons from the trenches on reproducible evaluation of language models, 2024.
  * Bowman et al. [2015] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning.  A large annotated corpus for learning natural language inference.  In Lluís Màrquez, Chris Callison-Burch, and Jian Su, editors, _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal, September 2015. Association for Computational Linguistics.  doi: 10.18653/v1/D15-1075.  URL https://aclanthology.org/D15-1075.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * Caswell et al. [2020] Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna.  Language ID in the wild: Unexpected challenges on the path to a thousand-language web text corpus.  In Donia Scott, Nuria Bel, and Chengqing Zong, editors, _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6588–6608, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.  doi: 10.18653/v1/2020.coling-main.579.  URL https://aclanthology.org/2020.coling-main.579.
  * Chung et al. [2022] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei.  Scaling instruction-finetuned language models, 2022.
  * Cohere [2024] Cohere.  Command R — cohere.com.  https://cohere.com/command, 2024.  [Accessed 01-06-2024].
  * Conneau et al. [2018] Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov.  XNLI: Evaluating cross-lingual sentence representations.  In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-1269.  URL https://aclanthology.org/D18-1269.
  * Conneau et al. [2020] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  Unsupervised cross-lingual representation learning at scale.  In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.747.  URL https://aclanthology.org/2020.acl-main.747.
  * Dione et al. [2023] Cheikh M. Bamba Dione, David Ifeoluwa Adelani, Peter Nabende, Jesujoba Alabi, Thapelo Sindane, Happy Buzaaba, Shamsuddeen Hassan Muhammad, Chris Chinenye Emezue, Perez Ogayo, Anuoluwapo Aremu, Catherine Gitau, Derguene Mbaye, Jonathan Mukiibi, Blessing Sibanda, Bonaventure F. P. Dossou, Andiswa Bukula, Rooweither Mabuya, Allahsera Auguste Tapo, Edwin Munkoh-Buabeng, Victoire Memdjokam Koagne, Fatoumata Ouoba Kabore, Amelia Taylor, Godson Kalipe, Tebogo Macucwa, Vukosi Marivate, Tajuddeen Gwadabe, Mboning Tchiaze Elvis, Ikechukwu Onyenwe, Gratien Atindogbe, Tolulope Adelani, Idris Akinade, Olanrewaju Samuel, Marien Nahimana, Théogène Musabeyezu, Emile Niyomutabazi, Ester Chimhenga, Kudzai Gotosa, Patrick Mizha, Apelete Agbolo, Seydou Traore, Chinedu Uchechukwu, Aliyu Yusuf, Muhammad Abdullahi, and Dietrich Klakow.  MasakhaPOS: Part-of-speech tagging for typologically diverse African languages.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 10883–10900, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.609.  URL https://aclanthology.org/2023.acl-long.609.
  * Ferrara [2023] Emilio Ferrara.  Should chatgpt be biased? challenges and risks of bias in large language models.  _arXiv preprint arXiv:2304.03738_ , 2023.
  * Gao et al. [2021] Tianyu Gao, Adam Fisch, and Danqi Chen.  Making pre-trained language models better few-shot learners.  In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3816–3830, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.295.  URL https://aclanthology.org/2021.acl-long.295.
  * Gemini-Team et al. [2024] Gemini-Team, Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry, Lepikhin, Timothy Lillicrap, Jean baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, and et al.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
  * Goyal et al. [2021] Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan.  The flores-101 evaluation benchmark for low-resource and multilingual machine translation.  2021\.
  * Hartung et al. [2023] Kai Hartung, Aaricia Herygers, Shubham Kurlekar, Khabbab Zakaria, Taylan Volkan, Sören Gröttrup, and Munir Georges.  Measuring sentiment bias in machine translation, 2023.
  * Hendrycks et al. [2021a] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2021a.
  * Hendrycks et al. [2021b] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.  Measuring mathematical problem solving with the MATH dataset.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021b.  URL https://openreview.net/forum?id=7Bywt2mQsCe.
  * Hendy et al. [2023] Amr Hendy, Mohamed Gomaa Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla.  How good are gpt models at machine translation? a comprehensive evaluation.  _ArXiv_ , abs/2302.09210, 2023.  URL https://api.semanticscholar.org/CorpusID:257038384.
  * Ji et al. [2023] Meng Ji, Meng Ji, Pierrette Bouillon, and Mark Seligman.  _Cultural and Linguistic Bias of Neural Machine Translation Technology_ , page 100–128.  Studies in Natural Language Processing. Cambridge University Press, 2023.  doi: 10.1017/9781108938976.005.
  * Jiang et al. [2024] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mixtral of experts, 2024.
  * Joshi et al. [2020] Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury.  The state and fate of linguistic diversity and inclusion in the NLP world.  In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6282–6293, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.560.  URL https://aclanthology.org/2020.acl-main.560.
  * Kreutzer et al. [2022] Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi.  Quality at a glance: An audit of web-crawled multilingual datasets.  _Transactions of the Association for Computational Linguistics_ , 10:50–72, 2022.  doi: 10.1162/tacl_a_00447.  URL https://aclanthology.org/2022.tacl-1.4.
  * Kudugunta et al. [2023] Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A. Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat.  Madlad-400: A multilingual and document-level large audited dataset, 2023.
  * Lai et al. [2023a] Viet Lai, Nghia Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Nguyen.  ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 13171–13189, Singapore, December 2023a. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-emnlp.878.  URL https://aclanthology.org/2023.findings-emnlp.878.
  * Lai et al. [2023b] Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen.  Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback.  In Yansong Feng and Els Lefever, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 318–327, Singapore, December 2023b. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-demo.28.  URL https://aclanthology.org/2023.emnlp-demo.28.
  * Luccioni and Viviano [2021] Alexandra Luccioni and Joseph Viviano.  What’s in the box? an analysis of undesirable content in the Common Crawl corpus.  In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 182–189, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-short.24.  URL https://aclanthology.org/2021.acl-short.24.
  * Ma et al. [2023] Chunlan Ma, Ayyoob ImaniGooghari, Haotian Ye, Ehsaneddin Asgari, and Hinrich Schütze.  Taxi1500: A multilingual dataset for text classification in 1500 languages, 2023.
  * Meta [2024] Meta.  Introducing Meta Llama 3: The most capable openly available LLM to date — ai.meta.com.  https://ai.meta.com/blog/meta-llama-3/, 2024.  [Accessed 01-06-2024].
  * Muennighoff et al. [2023] Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.  Crosslingual generalization through multitask finetuning.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15991–16111, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.891.  URL https://aclanthology.org/2023.acl-long.891.
  * Muhammad et al. [2023] Shamsuddeen Muhammad, Idris Abdulmumin, Abinew Ayele, Nedjma Ousidhoum, David Adelani, Seid Yimam, Ibrahim Ahmad, Meriem Beloucif, Saif Mohammad, Sebastian Ruder, Oumaima Hourrane, Alipio Jorge, Pavel Brazdil, Felermino Ali, Davis David, Salomey Osei, Bello Shehu-Bello, Falalu Lawan, Tajuddeen Gwadabe, Samuel Rutunda, Tadesse Belay, Wendimu Messelle, Hailu Balcha, Sisay Chala, Hagos Gebremichael, Bernard Opoku, and Stephen Arthur.  AfriSenti: A Twitter sentiment analysis benchmark for African languages.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 13968–13981, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.862.  URL https://aclanthology.org/2023.emnlp-main.862.
  * Nie et al. [2020] Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.  Adversarial NLI: A new benchmark for natural language understanding.  In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4885–4901, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.441.  URL https://aclanthology.org/2020.acl-main.441.
  * Ojo et al. [2023] Jessica Ojo, Kelechi Ogueji, Pontus Stenetorp, and David I Adelani.  How good are large language models on african languages?  _arXiv preprint arXiv:2311.07978_ , 2023.
  * OpenAI [2024] OpenAI.  Introducing ChatGPT.  https://openai.com/index/chatgpt/, 2024.  [Accessed 01-06-2024].
  * OpenAI et al. [2024] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, and et al.  Gpt-4 technical report, 2024.
  * Ponti et al. [2020] Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen.  XCOPA: A multilingual dataset for causal commonsense reasoning.  In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2362–2376, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.185.  URL https://aclanthology.org/2020.emnlp-main.185.
  * Pudjiati et al. [2022] Danti Pudjiati, Ninuk Lustyantie, Ifan Iskandar, and Tira Nur Fitria.  Post-editing of machine translation: Creating a better translation of cultural specific terms.  _Language Circle: Journal of Language and Literature_ , 17(1):61–73, 2022.
  * Rei et al. [2020] Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie.  COMET: A neural framework for MT evaluation.  In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.213.  URL https://aclanthology.org/2020.emnlp-main.213.
  * Savoldi et al. [2021] Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi.  Gender bias in machine translation, 2021.
  * Schick and Schütze [2021] Timo Schick and Hinrich Schütze.  It’s not just size that matters: Small language models are also few-shot learners.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2339–2352, 2021.
  * Shi et al. [2022] Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al.  Language models are multilingual chain-of-thought reasoners.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Shi et al. [2023] Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei.  Language models are multilingual chain-of-thought reasoners.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=fR3wGCk-IXp.
  * Singh et al. [2024] Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergün, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Minh Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, and Sara Hooker.  Aya dataset: An open-access collection for multilingual instruction tuning, 2024.
  * Team et al. [2024] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy.  Gemma: Open models based on gemini research and technology, 2024.
  * Team et al. [2022] NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang.  No language left behind: Scaling human-centered machine translation, 2022.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, and et al.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * Ustun et al. [2024] A. Ustun, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker.  Aya model: An instruction finetuned open-access multilingual language model.  _ArXiv_ , abs/2402.07827, 2024.  URL https://api.semanticscholar.org/CorpusID:267627803.
  * Üstün et al. [2024] Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker.  Aya model: An instruction finetuned open-access multilingual language model, 2024.
  * Vanmassenhove et al. [2021] Eva Vanmassenhove, Dimitar Shterionov, and Matthew Gwilliam.  Machine translationese: Effects of algorithmic bias on linguistic complexity in machine translation.  In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty, editors, _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2203–2213, Online, April 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.eacl-main.188.  URL https://aclanthology.org/2021.eacl-main.188.
  * Wang et al. [2023] Jiayi Wang, David Ifeoluwa Adelani, Sweta Agrawal, Marek Masiak, Ricardo Rei, Eleftheria Briakou, Marine Carpuat, Xuanli He, Sofia Bourhim, Andiswa Bukula, Muhidin A. Mohamed, Temitayo Olatoye, Tosin P. Adewumi, Hamam Mokayede, Christine Mwase, Wangui Kimotho, Foutse Yuehgoh, Anuoluwapo Aremu, Jessica Ojo, Shamsuddeen Hassan Muhammad, Salomey Osei, Abdul-Hakeem Omotayo, Chiamaka Chukwuneke, Perez Ogayo, Oumaima Hourrane, Salma El Anigri, Lolwethu Ndolela, Thabiso Mangwana, Shafie Abdi Mohamed, Ayinde Hassan, Oluwabusayo Olufunke Awoyomi, Lama Alkhaled, Sana Al-Azzawi, Naome A. Etori, Millicent Ochieng, Clemencia Siro, Samuel Njoroge, Eric Muchiri, Wangari Kimotho, Lyse Naomi Wamba Momo, Daud Abolade, Simbiat Ajao, Iyanuoluwa Shode, Ricky Macharm, Ruqayya Nasir Iro, Saheed Salahudeen Abdullahi, Stephen E. Moore, Bernard Opoku, Zainab Akinjobi, Abeeb Afolabi, Nnaemeka Obiefuna, Onyekachi Raphael Ogbu, Sam Brian, Verrah Otiende, Chinedu E. Mbonu, Sakayo Toadoum Sari, Yao Lu, and Pontus Stenetorp.  Afrimte and africomet: Enhancing comet to embrace under-resourced african languages.  2023\.  URL https://api.semanticscholar.org/CorpusID:269042892.
  * Wang et al. [2022] Jun Wang, Benjamin Rubinstein, and Trevor Cohn.  Measuring and mitigating name biases in neural machine translation.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2576–2590, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.184.  URL https://aclanthology.org/2022.acl-long.184.
  * Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837, 2022.
  * Workshop et al. [2023] BigScience Workshop, :, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, and et al.  Bloom: A 176b-parameter open-access multilingual language model, 2023.
  * Xue et al. [2021] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.  mT5: A massively multilingual pre-trained text-to-text transformer.  In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.41.  URL https://aclanthology.org/2021.naacl-main.41.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes] See Section 5

    3. (c)

Did you discuss any potential negative societal impacts of your work? [N/A]

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments (e.g., for benchmarks)…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
See Section 3.5, more details in A.2

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes] See Section 3.5

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [N/A]

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes] See
Section 3.2

    2. (b)

Did you mention the license of the assets? [Yes] See Section 3.2

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [N/A]

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [N/A]

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [N/A]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [Yes] See Section 3.3 for data and 3.4 for models

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [Yes] See Section 3.3

##  Appendix A Technical Appendix

Table 7: Languages covered in IrokoBench: including language family, region,
number of L1 & L2 speakers, size of monolingual data on the web (in MADLAD
corpus)

Language | Family/branch | Region | # speakers | # chars in MADLAD (MB) | In Aya | In BLOOMZ
---|---|---|---|---|---|---
English (eng) | Indo-European / Germanic | Across Africa | 1457M | 9,000,000MB | ✓ | ✓
French (fra) | Indo-European /Romance | Across Africa | 310M | 1,000,000MB | ✓ | ✓
Kiswahili (swa) | Niger-Congo / Bantu | East & Central Africa | 71M-106M | 2,400MB | ✓ | ✓
Kinyarwanda (kin) | Niger-Congo / Bantu | East Africa | 10M | 749MB | ✓ | ✓
Hausa (hau) | Afro-Asiatic / Chadic | West Africa | 77M | 630MB | ✓ | ✘
Amharic (amh) | Afro-Asiatic / Ethio-Semitic | East Africa | 57M | 509MB | ✓ | ✘
isiXhosa (xho) | Niger-Congo / Bantu | Southern Africa | 19M | 287MB | ✓ | ✓
chiShona (sna) | Niger-Congo / Bantu | Southern Africa | 11M | 266MB | ✓ | ✓
isiZulu (xho) | Niger-Congo / Bantu | Southern Africa | 27M | 257MB | ✓ | ✓
Igbo (ibo) | Niger-Congo / Volta-Niger | West Africa | 31M | 251MB | ✓ | ✓
Yorùbá (yor) | Niger-Congo / Volta-Niger | West Africa | 46M | 239MB | ✓ | ✓
Sesotho (sot) | Niger-Congo / Bantu | Southern Africa | 13M | 227MB | ✓ | ✓
Oromo (orm) | Afro-Asiatic / Cushitic | East Africa | 37M | 88MB | ✘ | ✘
Luganda (lug) | Niger-Congo / Bantu | Central Africa | 11M | 48MB | ✘ | ✓
Ewe (ewe) | Niger-Congo / Kwa | West Africa | 7M | 33MB | ✘ | ✓
Twi (twi) | Niger-Congo / Kwa | West Africa | 9M | 25MB | ✓ | ✓
Lingala (lin) | Niger-Congo / Bantu | Central Africa | 40M | 22MB | ✘ | ✓
Wolof (wol) | Niger-Congo / Senegambia | West Africa | 5M | 5MB | ✘ | ✓

###  A.1 Language covered

Table 7 provide the languages covered in the IrokoBench, their language
family, regions located in Africa, number of speakers, and size of monolingual
data available on the web based on the MADLAD cleaned corpus [36]—we only
report number of characters in mega bytes. Additionally, we added an
indication whether this language is covered in the pre-training of Aya-101 and
BLOOMZ 7B LLMs.

###  A.2 Evaluation Tool

We use the EleutherAI LM Evaluation Harness (lm-eval) tool [15]—a popular
evaluation tool that is helping to standardize LLM evaluation. The tool allows
for three types of evaluation: log-likelihood, perplexity, and generation. The
log-likelihood is more suitable for multiple-choice tasks since it helps to
restrict the model’s option to fewer choices—more appropriate for weaker
models. However, the log-likelihood approach cannot evaluate the generative
capabilities of LLMs to generate coherent and relevant answers. Moreover,
closed models are only accessible via API and do not provide access to the log
probabilities, making it impossible to use the log-likelihood approach. To
extract the correct answers for the task, we employed a verbalizer [25, 52].
For AfriMGSM, we used the default verbalizer provided by the tool. However,
for AfriXNLI and AfriMMLU, we manually created a verbalizer for the closed
models and used the log-likelihood request type for the open models.Table 8
provides the prompt templates used for evaluation present in the tool. For
XNLI, we employed two types of prompts to understand the sensitivity of
evaluation to type of prompt. We report on these results in Table 6.

Table 8: Prompt templates used for different tasks and datasets

.  Task/Dataset Prompt AfriMGSM Question: {question} Answer: AfriMMLU You are
a highly knowledgeable and intelligent artificial intelligence model answers
multiple-choice questions about {subject} Question: {question} Choices: A:
{choice1} B: {choice2} C: {choice3} D: {choice4} Answer: AfriXNLI (ANLI prompt
[44]) {premise} Question: {hypothesis} True, False, or Neither? Answer:
AfriXNLI (Lai’s prompt [37]) Please identify whether the premise entails or
contradicts the hypothesis in the following premise and hypothesis. The answer
should be exact entailment, contradiction, or neutral. Premise: {premise}
Hypothesis: {hypothesis} Is it entailment, contradiction, or neutral?

(a) Amharic

(b) Yorùbá

(c) isiZulu

(d) Lingala

(e) Twi

(f) Wolof

Figure 3: Evaluation of AfriXNLI translations using AfriCOMET metric scores.

###  A.3 AfriCOMET metric scores for XNLI translation

We employ AfriCOMET evaluation metrics, as developed by Wang et al. [62], to
automatically assess the quality of translations for our newly created
benchmarks. Figure 3 depicts the histogram of scores obtained from AfriCOMET
for AfriXNLI, illustrating promising results and offering compelling evidence
for the effectiveness of our translations (Amharic, Yorùbá, isiZulu). However,
the performance of this metric depends on if the language we are evaluating is
covered in the pre-training of the base model of the metric i.e. AfroXLMR-
large. In the case of Lingala, Twi and Wolof, the performance of the metric
does not correlate with the human translation since they are not covered in
AfroXLMR. Similar findings were reported in the original AfriCOMET QE paper
that Twi had worse correlation with human judgement (i.e., 0.279 for Pearson,
and 0.060 for Spearman) [62].

###  A.4 Task-specific results for all models

We provide the entire results of all LLMs on AfriMMLU, AfriXNLI and AfriMGSM
tasks as shown in Table 9, Table 10 and Table 11

Table 9: AfriMMLU results in in-language and translate-test scenarios: Option
prediction accuracy per language. Average computed on only African languages.
Best result in bold

Model | eng | fra | amh | ewe | hau | ibo | kin | lin | lug | orm | sna | sot | swa | twi | wol | xho | yor | zul | avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Prompt LLMs in African Language |  |  |  |  |  |  |  |  |  |  |
FlanT5-XXL | 48.4 | 37.6 | 22.4 | 22.6 | 24.2 | 27.0 | 24.8 | 26.6 | 22.6 | 27.4 | 25.5 | 25.2 | 28.4 | 27.2 | 26.2 | 24.8 | 25.6 | 25.8 | 25.4
mT0-XXL-MT | 36.2 | 33.6 | 29.4 | 24.6 | 28.4 | 28.4 | 28.2 | 28.8 | 26.4 | 27.6 | 26.6 | 31.8 | 31.8 | 29.0 | 26.0 | 25.6 | 28.8 | 29.6 | 28.2
Aya-101 | 41.8 | 36.4 | 32.8 | 26.8 | 32.2 | 35.8 | 30.0 | 26.0 | 30.8 | 30.6 | 27.8 | 34.2 | 35.0 | 25.8 | 24.6 | 31.2 | 29.8 | 31.8 | 30.3
BLOOMZ 7B | 33.0 | 29.2 | 20.6 | 25.0 | 22.2 | 23.6 | 23.2 | 24.2 | 23.0 | 25.4 | 24.8 | 23.0 | 26.0 | 19.5 | 23.4 | 23.2 | 25.6 | 24.0 | 23.5
Gemma 7B | 51.2 | 47.0 | 26.6 | 27.2 | 29.0 | 31.0 | 28.2 | 31.0 | 30.4 | 30.8 | 29.4 | 29.2 | 34.0 | 28.4 | 28.8 | 27.0 | 29.0 | 32.4 | 29.5
LLaMa 2 7B | 34.8 | 34.8 | 23.4 | 22.6 | 22.4 | 26.0 | 24.2 | 27.6 | 26.0 | 24.6 | 25.8 | 25.4 | 25.2 | 24.6 | 25.4 | 24.0 | 25.0 | 23.6 | 24.7
LLaMa 3 8B | 49.0 | 45.6 | 26.8 | 25.8 | 29.6 | 29.4 | 28.4 | 28.8 | 22.4 | 29.2 | 29.4 | 27.0 | 30.4 | 24.8 | 27.6 | 24.6 | 28.0 | 28.0 | 27.5
LLaMa 3 70B | 74.4 | 68.6 | 33.0 | 31.8 | 33.6 | 37.8 | 32.4 | 35.0 | 30.8 | 33.0 | 33.4 | 35.2 | 44.8 | 33.7 | 32.0 | 30.0 | 33.8 | 35.0 | 34.1
Command R | 49.8 | 50.6 | 25.2 | 24.4 | 27.6 | 27.6 | 23.8 | 29.8 | 27.6 | 29.8 | 27.6 | 24.6 | 28.2 | 27.0 | 23.8 | 28.4 | 27.0 | 30.2 | 27.0
Command R+ | 50.4 | 44.4 | 25.2 | 25.0 | 25.0 | 25.4 | 23.4 | 27.6 | 26.6 | 24.4 | 29.6 | 29.6 | 26.8 | 23.2 | 25.4 | 26.0 | 28.8 | 25.8 | 26.1
Claude OPUS | 74.6 | 64.4 | 57.6 | 33.6 | 39.4 | 43.6 | 42.2 | 43.6 | 41.0 | 40.4 | 43.4 | 47.0 | 55.0 | 38.2 | 33.2 | 42.6 | 43.6 | 43.8 | 43.0
GPT-3.5 Turbo | 61.8 | 56.4 | 30.6 | 29.0 | 33.4 | 32.6 | 32.0 | 31.8 | 29.2 | 31.0 | 34.4 | 32.4 | 40.6 | 29.4 | 29.2 | 29.4 | 30.8 | 33.0 | 31.8
GPT-4 Turbo | 80.7 | 76.5 | 37.7 | 23.9 | 55.2 | 54.5 | 54.9 | 46.8 | 40.9 | 39.8 | 54.1 | 55.4 | 61.0 | 30.1 | 31.0 | 56.2 | 48.9 | 58.4 | 50.9
GPT-4o | 78.6 | 71.4 | 55.2 | 31.6 | 57.4 | 55.8 | 56.8 | 55.4 | 45.0 | 48.8 | 58.4 | 62.0 | 67.4 | 37.6 | 32.6 | 60.8 | 50.8 | 60.6 | 52.3
Translate-Test (Eval. in English) |  |  |  |  |  |  |  |  |  |  |
FlanT5-XXL |  | 43.2 | 39.8 | 33.6 | 35.6 | 34.8 | 37.8 | 33.8 | 31.6 | 35.6 | 39.0 | 37.8 | 38.6 | 30.4 | 25.0 | 35.2 | 36.2 | 37.2 | 35.1
mT0-XXL-MT |  | 34.0 | 31.0 | 27.2 | 30.6 | 28.0 | 27.4 | 27.6 | 25.8 | 28.2 | 31.2 | 31.2 | 29.0 | 30.4 | 26.6 | 29.4 | 30.2 | 28.0 | 28.9
Aya -101 |  | 37.2 | 32.0 | 30.4 | 30.8 | 31.6 | 29.2 | 32.6 | 29.2 | 29.0 | 33.4 | 33.8 | 31.6 | 30.8 | 25.4 | 32.8 | 32.6 | 32.4 | 31.1
BLOOMZ 7B |  | 29.4 | 26.4 | 26.6 | 27.6 | 21.8 | 24.2 | 25.6 | 26.4 | 27.0 | 26.6 | 28.2 | 31.0 | 27.8 | 24.8 | 26.6 | 24.4 | 27.6 | 26.4
Gemma 7B |  | 44.8 | 39.2 | 37.8 | 36.2 | 38.2 | 41.8 | 38.0 | 35.8 | 37.6 | 40.2 | 41.2 | 42.8 | 34.8 | 33.0 | 38.0 | 37.6 | 40.2 | 38.3
LLaMa 2 7B |  | 33.0 | 31.0 | 29.8 | 29.0 | 32.6 | 29.4 | 29.6 | 27.4 | 30.0 | 27.4 | 30.2 | 31.4 | 28.6 | 28.6 | 34.2 | 28.4 | 30.6 | 29.9
LLaMa 3 8B |  | 43.4 | 38.4 | 35.2 | 34.4 | 33.2 | 36.0 | 36.0 | 27.8 | 34.2 | 31.4 | 38.8 | 43.6 | 29.0 | 29.6 | 36.6 | 33.2 | 33.6 | 34.4
LLaMa 3 70B |  | 64.4 | 54.4 | 42.2 | 44.4 | 48.6 | 47.2 | 49.8 | 41.2 | 49.6 | 48.4 | 54.0 | 60.0 | 41.4 | 29.8 | 49.6 | 49.2 | 49.2 | 47.4
Command R |  | 47.4 | 41.6 | 31.8 | 36.2 | 36.6 | 36.8 | 38.2 | 29.0 | 37.8 | 34.0 | 43.0 | 44.8 | 31.4 | 28.8 | 42.0 | 35.0 | 36.8 | 36.5
Command R+ |  | 44.0 | 36.4 | 31.8 | 35.4 | 37.4 | 36.6 | 35.8 | 34.2 | 35.6 | 37.6 | 37.0 | 44.0 | 31.6 | 26.6 | 37.4 | 37.2 | 36.6 | 35.7
Claude OPUS |  | 63.2 | 52.4 | 39.8 | 46.2 | 48.8 | 49.2 | 46.2 | 40.2 | 45.6 | 48.0 | 52.4 | 75.3 | 37.6 | 31.0 | 51.2 | 47.4 | 50.2 | 47.6
GPT-3.5 Turbo |  | 51.0 | 41.4 | 35.0 | 36.8 | 41.6 | 38.4 | 39.6 | 34.8 | 40.2 | 40.6 | 46.0 | 54.9 | 31.2 | 27.6 | 38.4 | 38.6 | 37.4 | 39.6
GPT-4 Turbo |  | 61.0 | 50.0 | 37.6 | 43.2 | 47.6 | 48.2 | 46.8 | 40.4 | 44.6 | 46.2 | 53.0 | 72.7 | 36.4 | 28.0 | 48.4 | 46.6 | 49.0 | 47.0
GPT-4o |  | 67.2 | 57.6 | 40.6 | 49.4 | 56.4 | 55.0 | 50.2 | 41.8 | 51.6 | 49.8 | 59.0 | 80.0 | 37.2 | 33.0 | 53.0 | 52.4 | 54.2 | 52.3

Table 10: AfriXNLI results in in-language and translate-test scenarios: Option
prediction accuracy per language. Average computed on only African languages.
Best result per language in bold

Model | eng | fra | amh | ewe | hau | ibo | kin | lin | lug | orm | sna | sot | swa | twi | wol | xho | yor | zul | avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Prompt LLMs in African Language |  |  |  |  |  |  |  |  |  |  |
AfroXLMR-76L | 88.2 | 83.3 | 78.5 | 58.3 | 73.3 | 70.0 | 65.8 | 33.3 | 68.0 | 69.3 | 70.8 | 70.8 | 73.3 | 59.5 | 51.8 | 73.0 | 63.2 | 72.5 | 65.7
FlanT5-XXL | 72.2 | 63.7 | 34.8 | 37.0 | 37.0 | 40.2 | 38.5 | 32.8 | 35.3 | 35.1 | 36.6 | 36.1 | 38.3 | 37.3 | 37.0 | 37.3 | 38.2 | 35.3 | 36.7
mT0-XXL-MT | 63.5 | 61.5 | 58.3 | 38.5 | 57.5 | 56.5 | 51.3 | 33.5 | 54.2 | 47.2 | 54.8 | 56.0 | 55.5 | 48.8 | 39.7 | 56.3 | 52.3 | 55.0 | 51.0
Aya-101 | 61.5 | 60.5 | 58.2 | 42.2 | 56.8 | 54.7 | 48.7 | 34.5 | 44.5 | 51.7 | 55.8 | 54.7 | 55.2 | 48.2 | 37.3 | 53.7 | 48.3 | 54.3 | 49.9
BLOOMZ 7B | 60.3 | 56.0 | 36.8 | 35.7 | 36.5 | 44.7 | 38.5 | 33.8 | 41.5 | 35.2 | 43.2 | 40.5 | 45.8 | 37.5 | 36.0 | 39.8 | 45.2 | 40.3 | 39.4
Gemma 7B | 51.5 | 47.8 | 34.3 | 35.3 | 38.8 | 39.0 | 35.8 | 33.2 | 34.8 | 35.7 | 38.8 | 35.7 | 39.3 | 37.2 | 35.8 | 37.2 | 36.5 | 35.7 | 36.4
LLaMa 2 7B | 50.0 | 48.5 | 33.7 | 33.3 | 33.2 | 33.5 | 33.5 | 33.8 | 33.5 | 33.3 | 32.8 | 33.5 | 34.0 | 34.2 | 31.2 | 34.0 | 33.3 | 33.3 | 33.4
LLaMa 3 8B | 54.0 | 51.8 | 34.5 | 33.8 | 36.0 | 39.5 | 33.8 | 32.5 | 35.5 | 34.7 | 37.8 | 35.0 | 39.5 | 33.0 | 33.5 | 38.1 | 38.1 | 36.8 | 35.8
LLaMa 3 70B | 51.3 | 50.2 | 36.8 | 33.3 | 35.3 | 36.5 | 34.5 | 33.0 | 33.5 | 33.3 | 33.8 | 34.3 | 36.5 | 33.0 | 33.5 | 34.0 | 33.0 | 33.0 | 34.2
CommandR | 57.3 | 48.3 | 33.2 | 33.5 | 33.8 | 33.2 | 34.2 | 33.5 | 33.7 | 33.7 | 33.8 | 33.7 | 36.8 | 34.7 | 36.2 | 33.5 | 33.8 | 33.3 | 34.0
CommandR+ | 56.5 | 51.5 | 41.0 | 33.5 | 38.8 | 40.5 | 38.2 | 33.0 | 37.8 | 34.0 | 37.0 | 37.7 | 43.0 | 36.2 | 36.5 | 37.2 | 43.2 | 39.5 | 37.9
Claude OPUS | 85.7 | 74.7 | 61.3 | 54.5 | 61.5 | 45.0 | 64.5 | 30.5 | 63.7 | 50.2 | 57.0 | 68.3 | 70.5 | 56.0 | 50.0 | 68.8 | 63.7 | 63.7 | 58.1
GPT-3.5 Turbo | 72.0 | 64.7 | 36.7 | 36.2 | 40.5 | 40.3 | 44.7 | 34.8 | 42.7 | 39.8 | 47.5 | 41.3 | 58.0 | 38.8 | 37.0 | 43.0 | 39.7 | 42.7 | 41.5
GPT-4 Turbo | 86.0 | 82.2 | 65.2 | 38.7 | 69.3 | 69.5 | 67.7 | 32.2 | 65.0 | 58.3 | 67.5 | 65.7 | 67.2 | 47.5 | 39.7 | 72.0 | 65.3 | 69.2 | 60.0
GPT-4o | 86.2 | 78.7 | 66.7 | 48.3 | 69.2 | 68.2 | 66.8 | 31.2 | 67.2 | 66.2 | 69.8 | 68.3 | 72.5 | 53.2 | 49.5 | 72.5 | 63.5 | 70.0 | 62.7
Translate-Test (Eval. in English) |  |  |  |  |  |  |  |  |  |  |
AfroXLMR-76L |  | 83.0 | 73.7 | 54.3 | 67.2 | 66.0 | 63.0 | 32.8 | 65.7 | 65.8 | 71.2 | 70.2 | 73.0 | 56.8 | 47.5 | 74.2 | 63.7 | 72.0 | 63.6
FlanT5-XXL |  | 68.3 | 61.3 | 47.7 | 56.7 | 52.7 | 53.3 | 33.7 | 52.8 | 55.0 | 57.3 | 57.2 | 59.7 | 49.7 | 43.0 | 60.5 | 52.1 | 59.0 | 53.2
mT0-XXL-MT |  | 59.3 | 54.5 | 45.3 | 52.7 | 50.0 | 49.8 | 34.5 | 48.2 | 50.2 | 55.0 | 53.0 | 56.8 | 46.2 | 42.7 | 54.8 | 50.7 | 54.3 | 49.9
Aya-101 |  | 60.3 | 55.2 | 47.0 | 51.8 | 52.2 | 50.3 | 33.0 | 49.0 | 49.0 | 52.0 | 53.3 | 54.0 | 45.5 | 41.5 | 54.5 | 48.8 | 54.0 | 49.4
BLOOMZ 7B |  | 56.8 | 52.8 | 42.7 | 51.3 | 48.5 | 47.3 | 34.2 | 44.5 | 48.0 | 52.0 | 51.3 | 51.2 | 44.7 | 40.2 | 52.8 | 47.3 | 51.3 | 47.5
Gemma 7B |  | 49.2 | 43.6 | 41.3 | 43.3 | 42.5 | 41.2 | 33.3 | 43.7 | 40.8 | 44.3 | 44.0 | 44.3 | 40.0 | 37.5 | 45.8 | 41.0 | 45.0 | 42.0
LLaMa 2 7B |  | 48.0 | 46.7 | 39.3 | 43.7 | 43.0 | 42.2 | 34.0 | 42.0 | 41.8 | 42.3 | 43.7 | 45.7 | 42.0 | 39.2 | 46.3 | 40.5 | 44.3 | 42.3
LLaMa 3 8B |  | 52.2 | 48.3 | 43.3 | 43.2 | 44.8 | 45.3 | 33.7 | 46.0 | 45.2 | 45.5 | 47.5 | 47.5 | 40.5 | 38.7 | 44.5 | 44.2 | 47.0 | 44.1
LLaMa 3 70B |  | 48.5 | 42.8 | 41.3 | 43.7 | 39.8 | 40.3 | 33.3 | 43.3 | 39.7 | 41.1 | 43.0 | 44.3 | 39.7 | 38.7 | 42.3 | 38.2 | 42.7 | 40.9
CommandR |  | 54.3 | 50.5 | 45.2 | 49.5 | 47.2 | 46.5 | 34.5 | 47.2 | 39.4 | 48.3 | 48.2 | 49.8 | 43.7 | 40.5 | 50.7 | 46.3 | 50.0 | 46.1
CommandR+ |  | 55.0 | 51.2 | 42.5 | 47.7 | 45.3 | 45.3 | 35.2 | 46.2 | 43.8 | 48.0 | 47.8 | 49.5 | 41.3 | 39.5 | 47.3 | 46.2 | 49.7 | 45.4
Claude OPUS |  | 56.0 | 65.7 | 45.7 | 76.3 | 59.0 | 55.0 | 32.0 | 57.7 | 55.8 | 61.5 | 62.5 | 63.8 | 46.3 | 41.8 | 63.5 | 55.3 | 59.8 | 56.4
GPT-3.5 Turbo |  | 65.8 | 59.2 | 47.3 | 51.0 | 56.0 | 53.8 | 33.0 | 54.0 | 52.0 | 53.5 | 58.0 | 56.3 | 47.2 | 43.5 | 59.8 | 52.8 | 57.7 | 52.2
GPT-4 Turbo |  | 75.2 | 64.0 | 43.0 | 52.8 | 59.0 | 55.8 | 31.7 | 51.0 | 52.5 | 55.8 | 58.5 | 60.2 | 45.0 | 39.7 | 63.2 | 52.5 | 59.5 | 52.8
GPT-4o |  | 80.0 | 69.0 | 50.7 | 62.2 | 63.3 | 61.0 | 33.0 | 63.2 | 58.7 | 65.8 | 67.0 | 69.5 | 51.3 | 46.2 | 69.7 | 60.5 | 67.3 | 59.9

Table 11: AfriMGSM results in in-language and translate-test scenarios: Exact
Match score per language. Average computed on only African languages. Best
result per language in bold

Model | eng | fra | amh | ewe | hau | ibo | kin | lin | lug | orm | sna | sot | swa | twi | wol | xho | yor | zul | avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Prompt LLMs in African Language |  |  |  |  |  |  |  |  |  |  |
FlanT5-XXL | 24.0 | 22.0 | 7.6 | 2.8 | 4.4 | 4.0 | 5.6 | 2.0 | 4.4 | 2.4 | 2.4 | 2.8 | 10.4 | 1.2 | 2.4 | 1.6 | 3.2 | 3.6 | 3.8
mT0-XXL-MT | 24.8 | 22.0 | 7.6 | 2.8 | 4.8 | 4.0 | 6.0 | 2.0 | 4.4 | 2.4 | 2.8 | 3.2 | 11.2 | 1.2 | 2.4 | 1.6 | 3.6 | 3.6 | 4.0
Aya-101 | 24.8 | 22.0 | 7.6 | 2.8 | 5.6 | 4.0 | 6.8 | 2.0 | 4.4 | 2.4 | 3.2 | 3.2 | 11.2 | 1.2 | 2.4 | 2.0 | 3.6 | 4.0 | 4.2
BLOOMZ 7B | 24.8 | 22.0 | 7.6 | 2.8 | 5.6 | 4.0 | 6.4 | 2.0 | 4.4 | 2.4 | 2.8 | 2.8 | 11.2 | 1.2 | 2.4 | 2.0 | 3.2 | 4.4 | 4.1
Gemma 7B | 24.8 | 22.0 | 7.6 | 3.2 | 4.4 | 4.4 | 5.6 | 2.0 | 4.4 | 2.4 | 2.4 | 3.6 | 11.2 | 1.2 | 2.4 | 1.6 | 3.2 | 4.0 | 4.0
LLaMa 2 7B | 24.8 | 22.0 | 7.6 | 2.8 | 4.8 | 4.4 | 6.4 | 2.0 | 4.8 | 2.4 | 2.8 | 2.8 | 11.2 | 1.2 | 2.4 | 1.6 | 3.2 | 4.4 | 4.1
LLaMa 3 8B | 24.8 | 22.0 | 7.6 | 2.8 | 4.8 | 4.4 | 5.6 | 2.0 | 4.8 | 2.4 | 2.4 | 3.2 | 11.2 | 1.2 | 2.4 | 1.6 | 3.2 | 3.6 | 4.0
LLaMa 3 70B | 86.4 | 54.4 | 5.2 | 2.8 | 15.6 | 9.2 | 8.4 | 5.6 | 8.0 | 4.0 | 4.8 | 6.0 | 34.8 | 3.2 | 4.0 | 4.4 | 7.2 | 7.6 | 8.2
Command R | 8.0 | 6.0 | 3.2 | 2.0 | 2.8 | 2.0 | 3.2 | 2.8 | 2.0 | 2.8 | 2.0 | 2.4 | 4.4 | 1.6 | 2.0 | 3.6 | 1.6 | 1.6 | 2.5
Command R+ | 12.8 | 12.0 | 2.8 | 1.6 | 3.2 | 1.6 | 4.0 | 3.2 | 2.8 | 2.4 | 3.6 | 2.8 | 7.6 | 3.6 | 2.8 | 4.0 | 4.8 | 3.2 | 3.4
Claude OPUS | 59.6 | 52.8 | 35.6 | 22.0 | 28.0 | 17.2 | 28.4 | 24.0 | 28.0 | 20.4 | 25.2 | 31.6 | 42.0 | 19.6 | 12.4 | 25.2 | 28.4 | 26.8 | 25.9
GPT-3.5 Turbo | 60.0 | 43.2 | 1.2 | 0.8 | 5.2 | 1.2 | 5.6 | 2.8 | 7.2 | 3.2 | 2.8 | 3.6 | 35.2 | 3.6 | 2.8 | 3.2 | 2.0 | 2.4 | 5.2
GPT-4 Turbo | 79.6 | 64.0 | 8.4 | 6.8 | 40.8 | 21.6 | 41.6 | 30.8 | 24.4 | 31.2 | 38.4 | 38.0 | 54.4 | 6.8 | 9.2 | 31.2 | 29.2 | 31.2 | 27.8
GPT-4o | 58.4 | 43.6 | 4.4 | 5.6 | 42.0 | 32.0 | 36.4 | 30.8 | 22.8 | 39.2 | 33.2 | 28.8 | 44.8 | 10.4 | 12.4 | 29.2 | 30.8 | 28.4 | 27.0
Translate-Test (Eval. in English) |  |  |  |  |  |  |  |  |  |  |
FlanT5-XXL |  | 0.8 | 2.4 | 3.6 | 2.0 | 1.6 | 2.8 | 4.4 | 3.2 | 3.2 | 2.8 | 2.0 | 5.6 | 0.8 | 2.4 | 2.0 | 2.4 | 1.2 | 2.7
mT0-XXL-MT |  | 3.6 | 2.4 | 1.6 | 1.6 | 1.6 | 3.2 | 2.4 | 2.8 | 2.4 | 2.4 | 2.4 | 3.6 | 1.6 | 2.8 | 4.0 | 2.8 | 4.0 | 2.6
Aya-101 |  | 8.0 | 8.0 | 6.0 | 7.6 | 4.0 | 7.2 | 5.2 | 4.4 | 5.2 | 8.4 | 8.0 | 10.8 | 4.4 | 5.6 | 6.0 | 8.4 | 6.4 | 6.6
BLOOMZ 7B |  | 2.0 | 2.4 | 1.2 | 0.8 | 1.6 | 2.0 | 2.4 | 3.2 | 2.4 | 2.0 | 2.0 | 4.8 | 0.8 | 0.8 | 2.4 | 2.0 | 2.0 | 2.1
Gemma 7B |  | 10.4 | 8.0 | 5.2 | 7.6 | 7.2 | 12.0 | 8.8 | 9.2 | 7.6 | 10.0 | 7.6 | 11.6 | 4.8 | 5.2 | 9.2 | 8.8 | 8.4 | 8.2
LLaMa 2 7B |  | 9.6 | 4.0 | 5.2 | 3.6 | 7.2 | 7.2 | 6.0 | 6.4 | 6.0 | 7.2 | 6.0 | 9.2 | 5.2 | 3.2 | 6.8 | 6.4 | 5.2 | 5.9
LLaMa 3 8B |  | 44.0 | 31.6 | 18.0 | 27.2 | 24.0 | 34.0 | 27.2 | 23.2 | 28.8 | 26.8 | 33.6 | 38.0 | 19.6 | 12.8 | 28.8 | 29.6 | 28.8 | 27.0
LLaMa 3 70B |  | 69.6 | 53.6 | 27.6 | 48.4 | 40.4 | 54.0 | 47.6 | 34.0 | 47.6 | 46.8 | 54.0 | 69.6 | 29.2 | 20.8 | 50.0 | 49.2 | 51.2 | 45.3
Command R |  | 9.6 | 8.0 | 2.4 | 4.8 | 4.8 | 7.6 | 4.8 | 5.2 | 6.8 | 5.6 | 8.4 | 8.8 | 5.2 | 2.8 | 4.0 | 6.0 | 7.2 | 5.8
Command R+ |  | 11.2 | 7.2 | 5.6 | 9.6 | 8.8 | 10.4 | 10.8 | 8.0 | 9.6 | 7.2 | 10.4 | 13.2 | 7.2 | 5.6 | 10.4 | 10.0 | 12.4 | 9.2
Claude OPUS |  | 47.2 | 38.8 | 23.2 | 33.2 | 22.0 | 41.2 | 36.0 | 26.4 | 32.4 | 36.0 | 40.8 | 48.4 | 19.6 | 16.8 | 35.6 | 36.4 | 36.4 | 32.7
GPT-3.5 Turbo |  | 50.0 | 36.0 | 20.0 | 33.2 | 29.2 | 41.2 | 33.2 | 26.0 | 31.2 | 36.0 | 40.0 | 51.2 | 20.8 | 11.6 | 35.2 | 36.8 | 36.0 | 32.4
GPT-4 Turbo |  | 63.6 | 42.0 | 20.4 | 39.2 | 35.2 | 47.6 | 41.6 | 27.2 | 38.8 | 39.2 | 46.0 | 56.8 | 23.2 | 12.4 | 41.2 | 42.0 | 40.4 | 37.1
GPT-4o |  | 52.0 | 18.4 | 37.6 | 36.4 | 28.0 | 34.8 | 38.4 | 28.4 | 32.8 | 32.4 | 36.8 | 45.2 | 17.6 | 12.4 | 31.6 | 36.0 | 38.0 | 31.6

###  A.5 Cross-lingual transfer results for XNLI

In Table 12, we compare different multilingual masked language model (MLM)
performance on African languages. XLM-R-large has 559M parameters and is
trained on 100 languages, but only a few African languages are covered (amh,
hau, orm, swa, and xho). Serengeti, on the other hand, has been pre-trained on
all languages in IrokoBench, but it only has 240M parameters. AfroXLMR was
adapted from XLM-R through continual pre-training on 17 African languages
including 11 in IrokoBench (amh, hau, ibo, kin, orm, sna, sot, swa, xho, yor,
and zul). AfroXLMR-76L follows the same technique by performing continual pre-
training on XLM-R-large on 76 languages (72 African), all languages covered in
IrokoBench are part of its pre-training.

We found Africa-centric MLM to perform better on average than massively
multilingual models like XLM-R-large. Serengeti and AfroXLMR-base improved
over larger-sized XLM-R-large by $+3.5$ and $+5.3$ points, respectively.
Similarly, fine-tuning AfroXLMR-large, a larger version of AfroXLMR-base,
results in an improved boost in performance with $11.8$ points. The best
overall results were achieved by AfroXLMR-76L with a $16.1$ boost in
performance over XLM-R-large. This is probably because all the languages are
used in pre-training. We make use of AfroXLMR-76L has the baseline for all
LLMs. Interestingly, we find GPT-4o to be competitive or better than other
MLMs except the AfroXLMR-76L on average.

Table 12: Cross-lingual transfer results on AfriXNLI: We fine-tuned various
multilingual encoders on English training data, and evaluated on other
languages. Best result per language in bold

Model | eng | fra | amh | ewe | hau | ibo | kin | lin | lug | orm | sna | sot | swa | twi | wol | xho | yor | zul | avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
XLM-R-large | 90.5 | 84.3 | 75.8 | 37.2 | 68.2 | 39.3 | 40.5 | 32.8 | 37.7 | 59.2 | 41.2 | 39.7 | 74.2 | 39.0 | 43.8 | 63.5 | 38.5 | 62.2 | 49.6
Serengeti | 77.3 | 60.7 | 54.8 | 51.3 | 61.0 | 56.0 | 55.2 | 36.2 | 55.3 | 46.3 | 58.3 | 55.0 | 66.2 | 42.7 | 43.8 | 56.3 | 53.5 | 57.8 | 53.1
Afro-XLMR-base | 81.5 | 78.5 | 71.3 | 36.3 | 68.8 | 59.8 | 57.7 | 37.8 | 44.2 | 56.7 | 59.8 | 61.5 | 67.0 | 40.8 | 41.0 | 61.0 | 52.2 | 62.7 | 54.9
Afro-XLMR-large | 86.5 | 82.3 | 77.2 | 39.7 | 75.2 | 69.8 | 64.5 | 35.3 | 57.8 | 69.7 | 68.0 | 69.2 | 74.3 | 39.5 | 39.8 | 69.0 | 61.0 | 72.2 | 61.4
Afro-XLMR-76L-large | 88.2 | 83.3 | 78.5 | 58.3 | 73.3 | 70.0 | 65.8 | 33.3 | 68.0 | 69.3 | 70.8 | 70.8 | 73.3 | 59.5 | 51.8 | 73.0 | 63.2 | 72.5 | 65.7
GPT-4o | 86.2 | 78.7 | 66.7 | 48.3 | 69.2 | 68.2 | 66.8 | 31.2 | 67.2 | 66.2 | 69.8 | 68.3 | 72.5 | 53.2 | 49.5 | 72.5 | 63.5 | 70.0 | 62.7