# How good are Large Language Models on African Languages?

Jessica Ojo∗† Kelechi Ogueji∗ Pontus Stenetorp‡ David Ifeoluwa Adelani‡,∗
∗Masakhane, †Lelapa AI, South Africa, and ‡University College London, United
Kingdom
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Recent advancements in natural language processing have led to the
proliferation of large language models (LLMs). These models have been shown to
yield good performance, using in-context learning, even on unseen tasks and
languages. Additionally, they have been widely adopted as language-model-as-a-
service commercial APIs like GPT-4 API. However, their performance on African
languages is largely unknown. We present an analysis of three popular large
language models (mT0, LLaMa 2, and GPT-4) on five tasks (news topic
classification, sentiment classification, machine translation, question
answering, and named entity recognition) across 30 African languages, spanning
different language families and geographical regions. Our results suggest that
all LLMs produce below-par performance on African languages, and there is a
large gap in performance compared to high-resource languages like English most
tasks. We find that GPT-4 has an average or impressive performance on
classification tasks but very poor results on generative tasks like machine
translation. Surprisingly, we find that mT0 had the best overall on cross-
lingual QA, better than the state-of-the-art supervised model (i.e. fine-tuned
mT5) and GPT-4 on African languages. Overall, LLaMa 2 records the worst
performance due to its limited multilingual capabilities and English-centric
pre-training corpus. In general, our findings present a call-to-action to
ensure African languages are well represented in large language models, given
their growing popularity.

##  1 Introduction

Large language models have risen to the fore of natural language processing.
These models have been shown to achieve state-of-the-art performances on
several tasks. More recently, focus has shifted from the pretrain-finetune
paradigm (Howard and Ruder, 2018; Devlin et al., 2019; Liu et al., 2019;
Raffel et al., 2020) to in-context learning or pre-train-prompt paradigm
(Brown et al., 2020; Lin et al., 2021; Wei et al., 2022; Chowdhery et al.,
2022; Liu et al., 2023; Chung et al., 2022a; Sanh et al., 2022; Dong et al.,
2023). However, the evaluation of these prompting techniques are often limited
to high-resource languages.

In-context learning proves that prompting large language models (LLMs) with
some task-specific examples allows them to perform well on test examples of
that task, all without updating the model’s parameters. This has led to
reduced computation costs and has made it possible to create language-models-
as-a-service (Sun et al., 2022), in the form of commercial APIs. Commercial
language models have become very prevalent. For context, the recently released
ChatGPT amassed 100 million users in two months, 111https://chat.openai.com/
making it the fastest growing consumer app in recent history. Given their
dominance and inevitable continual rise, it is important to understand how
these models perform on low-resource African languages that are often under-
represented in NLP.

In this paper, we present an extensive analysis to close this gap by
evaluating a popular large language model using in-context learning on African
languages. We focus our evaluation on three popular LLMs: mT0 (Muennighoff et
al., 2023)—obtained by multitask prompted fine-tuning of mT5 (Xue et al.,
2021), LLaMa 2 (Touvron et al., 2023) and GPT-4 (OpenAI, 2023a). . Evaluation
is performed on five tasks—news topic classification, sentiment
classification, machine translation, named entity recognition, and question
answering. Our experiments, spanning 30 African languages from different
language families and geographical locations, suggests that commercial
language models do not perform well on African languages. In particular, we
note a large disparity in performance, depending on the evaluation task—models
perform better on text classification than question answering and machine
translation that are generative tasks. In general, the performance is worse
than high-resource languages.

Our evaluation shows that GPT-4 achieve more than 80% of the performance of
full-supervised fine-tuning on the news topic classification and sentiment
classification, but a bit lower performance—62% of full-supervised fine-tuning
on named entity recognition respectively. On the other hand, performance of
generative tasks like machine translation (MT) was poor. In comparison to MT
evaluation on high-resource languages (e.g. English-German and French-German),
our evaluation shows the gap in performance between LLM and full-supervised
fine-tuning is wider for African languages.

In general, other LLMs have worse results than GPT-4 on most tasks. However,
for cross-lingual QA, mT0 had the best overall performance, even exceeding the
state-of-the-art supervised model (i.e. fine-tuned mT5). Overall, LLaMa 2
records the worst performance due to its limited multilingual. Our work sheds
light on the need to ensure the inclusion of African languages in the
development of large language models, given their inevitable adoption in our
daily lives.

##  2 Languages and evaluation tasks

We cover 30 African languages from four language families (Afro-Asiatic,
Niger-Congo, Nilo-Saharan and English-Creole). We evaluated these languages on
five NLP tasks covering news classification, sentiment classification, named
entity recognition (NER) , question answering (QA) and machine translation
(MT). Table 6 shows the languages and tasks we evaluated on. Appendix A shows
the languages and tasks we evaluated on.

###  2.1 Evaluation tasks and datasets

#### News Classification:

MasakhaNEWS (Adelani et al., 2023) is a multilingual news classification
dataset covering 16 typologically-diverse languages spoken in Africa including
English and French. Evaluation is based on the accuracy.

#### Sentiment Classification

AfriSenti (Muhammad et al., 2023) is a multilingual sentiment classification
dataset for 14 languages spoken in Africa including Mozambican Portuguese. The
goal of the task is to classify tweets into positive, negative and neutral
classes. Evaluation is based on F1-score.

#### Named Entity Recognition:

MasakhaNER (Adelani et al., 2021, 2022b) is a named entity recognition dataset
covering 21 African languages. In this evaluation, we make of the MasakhaNER-X
(Ruder et al., 2023) dataset where the problem of entity recognition has been
reformulated as text-to-text generation i.e. the entire text of “Jens is an
employee of Amazon” as an input will predict “PERSON: Jens && ORG: Amazon”.

#### Question Answering

We focus on cross-lingual open-retrieval question answering (XOR QA) where the
answer and context are provided in a high resource language while the question
is provided in an African language. We use the AfriQA dataset (Ogundepo et
al., 2023) which consists of 12000+ examples across 10 African languages.
Evaluation is based on Exact Match and F1-score.

#### Machine Translation

We use the MAFAND-MT machine translation dataset from Adelani et al. (2022a)
which covers 16 African languages, professionally translated. The parallel
data is based on the news domain. Here, we compare the performance of fine-
tuning M2M-100 on few thousand parallel sentences e.g. 5K sentences, to the
performance of GPT-4 translation.

##  3 Experimental Setup

|  | High-resource | African languages |
---|---|---|---|---
Model | Size | eng | fra | amh | hau | ibo | lin | lug | pcm | orm | run | sna | som | swa | tir | xho | yor | avg
Fine-tune: SOTA |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
AfroXLMR-large | 550M | 93.1 | 91.1 | 94.4 | 92.2 | 93.4 | 93.7 | 89.9 | 92.1 | 98.8 | 92.7 | 95.4 | 86.9 | 87.7 | 89.5 | 97.3 | 94.0 | 92.7
Prompting of LLMs |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
GPT-4 | - | 84.7 | 82.6 | 91.1 | 74.4 | 82.2 | 82.4 | 84.1 | 94.7 | 78.8 | 88.5 | 78.1 | 79.7 | 79.2 | 75.7 | 87.5 | 93.7 | 85.6
mT0 | 13B | 64.7 | 58.3 | 64.8 | 65.6 | 63.6 | 62.3 | 56.7 | 74.4 | 57.4 | 58.8 | 82.6 | 52.3 | 57.8 | 52.0 | 69.7 | 61.7 | 62.8
mT0-MT | 13B | 68.7 | 58.0 | 63.5 | 72.1 | 70.5 | 63.4 | 74.1 | 81.8 | 56.3 | 61.4 | 72.1 | 56.0 | 58.1 | 55.2 | 84.6 | 74.0 | 67.4
LLaMa 2 | 13B | 61.0 | 45.1 | 7.1 | 37.2 | 60.7 | 66.1 | 63.2 | 70.4 | 22.6 | 63.4 | 69.6 | 48.8 | 50.5 | 3.9 | 61.3 | 41.1 | 47.6

Table 1: News Classification Results: We compare the F1 score of various LLMs’
results with that of the current state of the art result obtained from Adelani
et al. (2023). Best results per language are in bold.

|  | High-resource | African languages |
---|---|---|---|---
Model | Size | eng | por-mz | amh | arq | ary | hau | ibo | kin | pcm | swa | tso | twi | yor | avg
Fine-tune: SOTA |  |  |  |  |  |  |  |  |  |  |  |  |  |
AfroXLMR-large | 550M | 68.1 | 71.6 | 61.6 | 68.3 | 56.6 | 80.7 | 79.5 | 70.6 | 68.7 | 63.4 | 47.3 | 64.3 | 74.1 | 66.8
Prompting of LLMs |  |  |  |  |  |  |  |  |  |  |  |  |  |
GPT-4 | - | 66.1 | 60.4 | 69.9 | 61.3 | 56.6 | 41.8 | 66.7 | 58.6 | 57.7 | 65.0 | 26.4 | 48.5 | 55.6 | 55.3
mT0 | 13B | 41.2 | 16.0 | 67.2 | 50.4 | 37.0 | 40.5 | 26.7 | 36.3 | 63.6 | 20.9 | 47.5 | 43.5 | 35.6 | 42.6
mT0-MT | 13B | 37.2 | 16.5 | 70.2 | 58.5 | 34.6 | 36.1 | 27.2 | 39.5 | 50.7 | 18.7 | 42.1 | 35.9 | 23.7 | 39.7
LLaMa 2 | 13B | 52.8 | 32.3 | 10.5 | 26.2 | 37.4 | 25.5 | 35.1 | 34.2 | 24.3 | 49.7 | 30.5 | 23.9 | 24.0 | 29.2

Table 2: Sentiment Analysis Results: We compare the Accuracy score of various
LLMs’ results with that of the current state of the art result obtained from
Muhammad et al. (2023). Best results per language are in bold.

|  | Question human-translated to EN/FR | Question in an African languages
---|---|---|---
Model | Size | bem | fon | hau | ibo | kin | swa | twi | yor | zul | avg | bem | fon | hau | ibo | kin | swa | twi | yor | zul | avg
Fine-tune: SOTA |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
mT5-base | 580M | 48.8 | 41.4 | 58.5 | 66.6 | 60.8 | 52.3 | 55.4 | 44.6 | 54.9 | 60.2 | 2.9 | 5.1 | 25.8 | 41.7 | 25.5 | 29.4 | 5.2 | 11.9 | 24.7 | 19.1
Prompting of LLMs |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
GPT-4 | - | 60.2 | 56.4 | 65.9 | 78.1 | 41.6 | 66.5 | 62.7 | 74.1 | 68.7 | 66.1 | 18.4 | 22.8 | 17.0 | 25.0 | 23.7 | 22.2 | 21.2 | 19.0 | 19.2 | 20.9
mT0 | 13B | 74.4 | 70.7 | 78.8 | 84.4 | 72.3 | 72.1 | 75.6 | 79.3 | 79.4 | 76.1 | 45.8 | 44.0 | 70.7 | 79.5 | 70.2 | 71.8 | 52.7 | 72.6 | 74.3 | 64.6
mT0-MT | 13B | 76.1 | 73.9 | 80.3 | 83.7 | 74.8 | 70.7 | 73.8 | 77.9 | 80.3 | 76.8 | 46.9 | 46.4 | 68.3 | 81.7 | 71.3 | 69.9 | 47.6 | 69.5 | 74.4 | 64.0
LLaMa 2 | 13B | 63.5 | 55.6 | 70.5 | 75.5 | 63.5 | 65.4 | 62.6 | 74.6 | 63.1 | 66.0 | 27.7 | 35.6 | 25.5 | 37.2 | 22.6 | 42.9 | 23.7 | 24.9 | 24.1 | 29.6

Table 3: Question Answering Results: We compare the F1-score of various LLMs’
results (both target and high resource) with that of the current state of the
art result obtained from Ogundepo et al. (2023).

|  | French Centric | English Centric |
---|---|---|---|---
Model | Size | deu | bam | bbj | ewe | fon | mos | wol | deu | hau | ibo | lug | pcm | swa | tsn | twi | yor | zul | avg
xx-fr/en |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
M2M-100 | 418M | 51.9 | 45.6 | 26.5 | 30.9 | 27.5 | 17.0 | 33.8 | 57.6 | 35.1 | 46.1 | 46.4 | 36.7 | 68.6 | 55.8 | 45.2 | 35.1 | 35.2 | 39.0
GPT-4 | - | 66.7 | 10.8 | 7.3 | 15.5 | 6.1 | 11.0 | 14.7 | 66.3 | 14.7 | 21.8 | 23.2 | 58.8 | 19.8 | 21.7 | 21.1 | 13.6 | 20.7 | 18.7
mT0 | 13B | 27.2 | 27.2 | 16.2 | 26.3 | 24.7 | 16.1 | 23.1 | 28.9 | 32.0 | 31.2 | 36.9 | 44.9 | 25.4 | 28.4 | 26.1 | 35.7 | 34.8 | 28.6
mT0-MT | 13B | 63.1 | 32.9 | 13.9 | 33.1 | 27.9 | 16.3 | 27.7 | 68.2 | 38.1 | 46.8 | 48.7 | 56.9 | 57.1 | 53.5 | 38.2 | 40.8 | 54.4 | 39.1
LLaMa 2 | 13B | 45.0 | 17.8 | 15.3 | 21.2 | 18.2 | 17.1 | 18.0 | 53.2 | 17.4 | 23.1 | 29.2 | 54.8 | 32.9 | 24.0 | 24.4 | 20.8 | 22.6 | 23.8
fr/en-xx |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
M2M-100 | 418M | 59.0 | 48.2 | 23.1 | 30.9 | 27.6 | 16.7 | 35.7 | 53.36 | 43.3 | 50.0 | 39.0 | 64.0 | 56.4 | 52.0 | 38.2 | 35.9 | 51.2 | 40.8
GPT-4 | - | 57.4 | 4.9 | 5.2 | 5.9 | 3.3 | 5.7 | 5.3 | 60.3 | 36.1 | 35.7 | 38.64 | 53.4 | 59.0 | 43.6 | 32.0 | 18.1 | 45.1 | 18.7
mT0 | 13B | 15.4 | 8.6 | 8.7 | 11.8 | 6.9 | 12.3 | 11.0 | 16.1 | 15.4 | 23.5 | 21.5 | 34.2 | 23.1 | 17.3 | 12.1 | 6.3 | 19.6 | 15.5
mT0-MT | 13B | 24.9 | 17.7 | 11.5 | 20.1 | 9.1 | 14.6 | 16.5 | 25.0 | 23.11 | 38.5 | 28.6 | 48.3 | 48.3 | 34.3 | 29.9 | 15.2 | 38.1 | 26.2
LLaMa 2 | 13B | 22.4 | 13.2 | 6.5 | 16.8 | 11.0 | 10.9 | 15.1 | 22.4 | 14.7 | 16.3 | 14.1 | 21.4 | 41.3 | 24.4 | 19.5 | 10.4 | 20.3 | 17.1

Table 4: Machine Translation Results: We compare the ChrF scores of the GPT-4
results with that of the current state of the art result obtained from Adelani
et al. (2022a). Best results per language are in bold.

We focus our evaluations on the following LLMs: mT0-13B (-MT), LLaMa 2 13B,
and GPT-4. 222Experiments make use of gpt-4-0613. mT0-13B (Muennighoff et al.,
2023) is an LLM obtained by fine-tuning mT5-XXL (a 13B parameter size text-to-
text model and also the largest) on a collection of multitask prompted
datasets known as xP3 (Crosslingual Public Pool of Prompts) while mT0-13B-MT
was fine-tuned on xP3mt where prompts are provided in 20 languages (machine-
translated from English). 333https://huggingface.co/datasets/bigscience/xP3mt
On the other hand, LLaMa 2 (Touvron et al., 2023) is a popular open-sourced
LLM with chat functionality, the number of parameters ranges from 7B to 70B,
we make use of 13B chat model since it is the largest model that can fit a
single A100 GPU. GPT-4 is a Transformer-style model pre-trained to predict the
next token followed by a set of instructions in a prompt based on human
feedback (OpenAI, 2023b).

###  3.1 Prompt Templates

We designed our prompts in a zero-shot cross lingual manner, that is the
prompt context and query is designed in English while the text to be analyzed
is provided in the target African language. For each task, we designed simple
prompts that tend to reasonable results on few examples. We prompt the LLMs
using only English language since it has been shown that English prompts
perform better, on average, than in-language prompts (Lin et al., 2021; Shi et
al., 2022), so we do not explore prompting in the target language for both
tasks. Appendix B provides details on the prompt used for each task.

###  3.2 State-of-the-art models

Here, we compare the performance GPT-4 on African languages with:

  1. 1.

State-of-the-art (SOTA)—full-supervised setting results i.e. pre-trained
language models fine-tuned on labelled training data.

  2. 2.

High-resource languages (HRL) (e.g. English or French) evaluation results
available for the task, whenever it is available.

By comparing with high-resource languages, we can compare the gap in
performance with SOTA for low-resource African languages. We provide details
of the SOTA models in Appendix C.

##  4 Results

Model | Size | avg
---|---|---
Fine-tune: SOTA |
AfroXLMR-large | 550M | 84.6
Prompting of LLMs |
GPT-4 | - | 55.6
mT0 | 13B | 0.0
mT0-MT | 13B | 0.0
LLaMa 2 | 13B | 17.8

Table 5: Named Entity Recognition Results: We compare the F1-score of various
LLMs with that of the current state of the art result.

Here, we discuss the key findings in comparing LLMs performance on African
languages with SOTA models across the five different tasks. We further report
the gap in performance when compared to HRLs. 444We provide some examples of
several LLM outputs in

#### Large gap persists between the performance of HRLs and African languages

Table 3 shows the result of QA, which clearly demonstrates that providing
questions in English/French which is also the language of the context passage
achieves significantly better performance than providing questions in an
African language both for the fully-supervised setting and prompting of LLMs
setup. The performance gap is as wide as $-45.2$ and $-36.4$ for GPT-4 and
LLaMa 2 but smaller ($-11.5$) for mT0-13B and mT0-13B-MT because of their
impressive performance. Similarly, for machine translation (Table 4), in the
directions of fr-deu and en-deu, GPT-4 surprisingly gave better performance
than the baseline M2M-100, while the other LLMs seem to struggle in this
direction with ChrF score of $22.4-25.0$, their performance is better than the
average performance on African languages ($17.1$). The drop in performance is
even wider for the direction of fr-deu ($45.0$) and en-deu ($53.2$) when
compared to the average performance on African languages ($23.8$). For the
classification tasks, we also observe this trend, however, some African
languages also have similar impressive performance.

#### GPT-4 achieves more than 80% of SOTA’s performance on classification
tasks

On the news topic classification (Table 1), the performance on English
($84.7$) and French ($82.6$) is very similar to the average performance on
African languages ($85.6$) due to the simplicity of the task. Although for
sentiment classification (Table 2), there is some gap in performance for
English ($-10.8$) and Mozambique Portuguese ($-5.1$). Other LLMs generally
performs subpar compared to GPT-4.

#### mT0 achieves better performance than SOTA on cross-lingual QA

Surprisingly, we find mT0 achieved the best performance (see Table 3) even
when the questions are provided in an African language. We hypothesize that
this performance is probably due to the large number of QA datasets in
xP3—used for creating the mT0 model.

#### Fine-tuning with multilingual prompts helps mT0-13B-MT to be competitive
on MT

Our evaluation shows that mT0-13B-MT significantly perform better than
mT0-13B, the performance gap is wider for MT ($\sim+10$) than any other tasks
we evaluated on ($<5.0$). The effective performance is due to the multilingual
prompts used in developing the mT0-13B-MT instead of the English-only prompt
as shown in Muennighoff et al. (2022). mT0 generally outperforms other LLMs on
MT because the multitask prompted datasets used in creating mT0 includes
several MT datasets for African languages like WMT African dataset
555https://huggingface.co/datasets/allenai/wmt22_african and Flores-101 (Goyal
et al., 2022).

#### LLaMa 2 often struggles due to limited multilingual abilities

In general LLaMa 2 achieves the worse performance compared to other models of
similar sizes like mT0-13B, this is because of the pre-training corpus of
LLaMa 2 that is mostly English centric.

#### All models struggles with token classification

On average, all LLMs gave a poor result for NER (see Table 5), mT0 do not seem
to follow the template of result we provide with the “$$” separator of
entities. LLaMa 2 also often repeat the one-shot example we provided as the
output. Only GPT-4 has an average performance on the task ($55.6)$. Appendix D
provides the full breakdown of results by languages.

##  5 Related Works

#### Instruction Fine-tuning

Growing the size of language models does not inherently make them better at
following a user’s intent (Ouyang et al., 2022), fine-tuning language models
on a collection of tasks described via instructions substantially improves
zero-shot performance on unseen tasks (Wei et al., 2022). This method involves
further training the large language model on a dataset consisting of
instruction-output pairs in a supervised fashion enabling the LLMs to adhere
to human instructions. Several recent work agrees that instruction fine-tuning
generally improves the usability of pretrained large language models (Chung et
al., 2022b; Ouyang et al., 2022; Zhang et al., 2023b; Jiang et al., 2023).
This has also been adopted in multilingual settings (Holmström and
Doostmohammadi, 2023)

#### Multitask prompted fine-tuning (MTF)

While instruction fine-tuning is useful for understanding the user intents and
teaching the models to respond to instructions, multitask prompted fine-tuning
aims at making the model generalize to new tasks in a zero-shot setting (Sanh
et al., 2022). Muennighoff et al. (2023) which adapts MTF in multilingual
settings find that fine-tuning large multilingual language models on English
tasks with English prompts allows for task generalization to non-English
languages

#### Multilingual in-context Learning

LLMs have proven successful in multilingual settings. Lin et al. (2021)
trained a multilingual generative language models on datasets covering diverse
set of languages studying their few and zero shot learning capabilities on a
wide range of tasks. Their model outperforms GPT3 on several multilingual
tasks. Muennighoff et al. (2022) perform multitask prompted fine-tuning on
multilingual pretrained language models and observe impressive zero-shot
generalization to tasks in unseen languages. Following findings from Blevins
and Zettlemoyer (2022) that non-English dataset present in the pretraining
corpora of English language models explains their surprising cross-lingual
ability, Chowdhery et al. (2022) deliberately introduce non-English corpora
($\approx 22\%$) into the pretraining corpora of their PaLM model and achieve
impressive few-shot multilingual performance. Shi et al. (2022) evaluate GPT3
and PaLM on a newly introduced grade school mathematics multilingual
benchmark. They find that using prompts with intermediate reasoning steps in
English consistently led to competitive or better results than those written
in the native language of the question.

Recently, (Jiao et al., 2023) evaluate ChatGPT on machine translation and find
that, while it is competitive with other commercial translation APIs such as
Google translate666https://translate.google.com/, it is less robust on other
domains such as biomedical. Another concurrent work (Zhang et al., 2023a)
conducts a study on the performance of GLM (Zeng et al., 2022) on machine
translation. They note several interesting findings on the effect of prompt
template, examples and language. Despite the plethora of works on multilingual
prompting, little to no African languages are usually contained in the
evaluation sets of nearly all of these works. When present, they are often
obtained by translating the existing datasets of other languages (Yu et al.,
2022) This method has been shown to contain artifacts that can inflate the
performance of models evaluated on such datasets (Artetxe et al., 2020). Ahuja
et al. (2023) performs a comprehensive evaluation of the GPT models on
standard NLP benchmarks, covering 16 NLP datasets across 70 typologically
diverse languages. Zhu et al. (2023) finds that performing crosslingual
instruction fine-tuning and multilingual instruction fine-tuning on LLaMA
outperforms the non mulitilingual LLaMA base models on translation dataset
Flores-101.

The closest to our work is Abott et al. (2023) and Adelani et al. (2023), the
former evaluated GPT-3.5 on named entity recognition and machine translation
on only isiZulu while the latter only evaluated GPT-3.5 on the news-topic
classification task. However, our work is different from this as we compare
GPT-4 LLM in the evaluation of five different tasks across 30 languages.

##  6 Conclusion

We have presented an analysis of the performance of different language models
on African languages. Our results shows that there is a large gap in
performance between HRLs and African languages. Future work should focus on
methods to best adapt LLMs to unseen low-resource languages.

## References

  * Abott et al. (2023) Jade Abott, Bonaventure Dossou, and Rooweither Mbuya. 2023.  Comparing africa-centric models to openai’s gpt3.5.
  * Adelani et al. (2022a) David Adelani, Jesujoba Alabi, Angela Fan, Julia Kreutzer, Xiaoyu Shen, Machel Reid, Dana Ruiter, Dietrich Klakow, Peter Nabende, Ernie Chang, Tajuddeen Gwadabe, Freshia Sackey, Bonaventure F. P. Dossou, Chris Emezue, Colin Leong, Michael Beukman, Shamsuddeen Muhammad, Guyo Jarso, Oreen Yousuf, Andre Niyongabo Rubungo, Gilles Hacheme, Eric Peter Wairagala, Muhammad Umair Nasir, Benjamin Ajibade, Tunde Ajayi, Yvonne Gitau, Jade Abbott, Mohamed Ahmed, Millicent Ochieng, Anuoluwapo Aremu, Perez Ogayo, Jonathan Mukiibi, Fatoumata Ouoba Kabore, Godson Kalipe, Derguene Mbaye, Allahsera Auguste Tapo, Victoire Memdjokam Koagne, Edwin Munkoh-Buabeng, Valencia Wagner, Idris Abdulmumin, Ayodele Awokoya, Happy Buzaaba, Blessing Sibanda, Andiswa Bukula, and Sam Manthalu. 2022a.  A few thousand translations go a long way! leveraging pre-trained models for African news translation.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3053–3070, Seattle, United States. Association for Computational Linguistics.
  * Adelani et al. (2022b) David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba Alabi, Shamsuddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye, Amelia Taylor, Fatoumata Kabore, Chris Chinenye Emezue, Anuoluwapo Aremu, Perez Ogayo, Catherine Gitau, Edwin Munkoh-Buabeng, Victoire Memdjokam Koagne, Allahsera Auguste Tapo, Tebogo Macucwa, Vukosi Marivate, Mboning Tchiaze Elvis, Tajuddeen Gwadabe, Tosin Adewumi, Orevaoghene Ahia, Joyce Nakatumba-Nabende, Neo Lerato Mokono, Ignatius Ezeani, Chiamaka Chukwuneke, Mofetoluwa Oluwaseun Adeyemi, Gilles Quentin Hacheme, Idris Abdulmumin, Odunayo Ogundepo, Oreen Yousuf, Tatiana Moteu, and Dietrich Klakow. 2022b.  MasakhaNER 2.0: Africa-centric transfer learning for named entity recognition.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 4488–4508, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Adelani et al. (2021) David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Jesujoba Alabi, Seid Muhie Yimam, Tajuddeen Rabiu Gwadabe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, Verrah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chukwuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akinode, Deborah Nabagereka, Maurice Katusiime, Ayodele Awokoya, Mouhamadane MBOUP, Dibora Gebreyohannes, Henok Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulaye Faye, Blessing Sibanda, Orevaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thierno Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Salomey Osei. 2021.  MasakhaNER: Named entity recognition for African languages.  _Transactions of the Association for Computational Linguistics_ , 9:1116–1131.
  * Adelani et al. (2023) David Ifeoluwa Adelani, Marek Masiak, Israel Abebe Azime, Jesujoba Alabi, Atnafu Lambebo Tonja, Christine Mwase, Odunayo Ogundepo, Bonaventure F. P. Dossou, Akintunde Oladipo, Doreen Nixdorf, Chris Chinenye Emezue, sana al azzawi, Blessing Sibanda, Davis David, Lolwethu Ndolela, Jonathan Mukiibi, Tunde Ajayi, Tatiana Moteu, Brian Odhiambo, Abraham Owodunni, Nnaemeka Obiefuna, Muhidin Mohamed, Shamsuddeen Hassan Muhammad, Teshome Mulugeta Ababu, Saheed Abdullahi Salahudeen, Mesay Gemeda Yigezu, Tajuddeen Gwadabe, Idris Abdulmumin, Mahlet Taye, Oluwabusayo Awoyomi, Iyanuoluwa Shode, Tolulope Adelani, Habiba Abdulganiyu, Abdul-Hakeem Omotayo, Adetola Adeeko, Abeeb Afolabi, Anuoluwapo Aremu, Olanrewaju Samuel, Clemencia Siro, Wangari Kimotho, Onyekachi Ogbu, Chinedu Mbonu, Chiamaka Chukwuneke, Samuel Fanijo, Jessica Ojo, Oyinkansola Awosan, Tadesse Kebede, Toadoum Sari Sakayo, Pamela Nyatsine, Freedmore Sidume, Oreen Yousuf, Mardiyyah Oduwole, Tshinu Tshinu, Ussen Kimanuka, Thina Diko, Siyanda Nxakama, Sinodos Nigusse, Abdulmejid Johar, Shafie Mohamed, Fuad Mire Hassan, Moges Ahmed Mehamed, Evrard Ngabire, Jules Jules, Ivan Ssenkungu, and Pontus Stenetorp. 2023.  Masakhanews: News topic classification for african languages.
  * Ahuja et al. (2023) Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. 2023.  Mega: Multilingual evaluation of generative ai.
  * Alabi et al. (2022) Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. 2022\.  Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 4336–4349, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Artetxe et al. (2020) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020.  Translation artifacts in cross-lingual transfer learning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7674–7684, Online. Association for Computational Linguistics.
  * Blevins and Zettlemoyer (2022) Terra Blevins and Luke Zettlemoyer. 2022.  Language contamination helps explain the cross-lingual capabilities of english pretrained models.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Chung et al. (2022a) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022a.  Scaling instruction-finetuned language models.
  * Chung et al. (2022b) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022b.  Scaling instruction-finetuned language models.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dong et al. (2023) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023.  A survey for in-context learning.
  * Fan et al. (2021) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2021.  Beyond english-centric multilingual machine translation.  _J. Mach. Learn. Res._ , 22(1).
  * Gowda et al. (2021) Thamme Gowda, Zhao Zhang, Chris Mattmann, and Jonathan May. 2021.  Many-to-English machine translation tools, data, and pretrained models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_ , pages 306–316, Online. Association for Computational Linguistics.
  * Goyal et al. (2022) Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022.  The Flores-101 evaluation benchmark for low-resource and multilingual machine translation.  _Transactions of the Association for Computational Linguistics_ , 10:522–538.
  * Holmström and Doostmohammadi (2023) Oskar Holmström and Ehsan Doostmohammadi. 2023.  Making instruction finetuning accessible to non-English languages: A case study on Swedish models.  In _Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)_ , pages 634–642, Tórshavn, Faroe Islands. University of Tartu Library.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal language model fine-tuning for text classification.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 328–339, Melbourne, Australia. Association for Computational Linguistics.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7b.
  * Jiao et al. (2023) Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023.  Is chatgpt a good translator? a preliminary study.
  * Lin et al. (2021) Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona T. Diab, Veselin Stoyanov, and Xian Li. 2021.  Few-shot learning with multilingual language models.  _CoRR_ , abs/2112.10668.
  * Liu et al. (2023) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _ACM Comput. Surv._ , 55(9).
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _arXiv preprint_ , abs/1907.11692.
  * Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023\.  Crosslingual generalization through multitask finetuning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15991–16111, Toronto, Canada. Association for Computational Linguistics.
  * Muennighoff et al. (2022) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2022\.  Crosslingual generalization through multitask finetuning.
  * Muhammad et al. (2023) Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Abinew Ali Ayele, Nedjma Ousidhoum, David Ifeoluwa Adelani, Seid Muhie Yimam, Ibrahim Sa’id Ahmad, Meriem Beloucif, Saif Mohammad, Sebastian Ruder, Oumaima Hourrane, Pavel Brazdil, Felermino Dário Mário António Ali, Davis Davis, Salomey Osei, Bello Shehu Bello, Falalu Ibrahim, Tajuddeen Gwadabe, Samuel Rutunda, Tadesse Belay, Wendimu Baye Messelle, Hailu Beshada Balcha, Sisay Adugna Chala, Hagos Tesfahun Gebremichael, Bernard Opoku, and Steven Arthur. 2023.  AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages.
  * Nekoto et al. (2020) Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddeen Muhammad, Salomon Kabongo Kabenamualu, Salomey Osei, Freshia Sackey, Rubungo Andre Niyongabo, Ricky Macharm, Perez Ogayo, Orevaoghene Ahia, Musie Meressa Berhe, Mofetoluwa Adeyemi, Masabata Mokgesi-Selinga, Lawrence Okegbemi, Laura Martinus, Kolawole Tajudeen, Kevin Degila, Kelechi Ogueji, Kathleen Siminyu, Julia Kreutzer, Jason Webster, Jamiil Toure Ali, Jade Abbott, Iroro Orife, Ignatius Ezeani, Idris Abdulkadir Dangana, Herman Kamper, Hady Elsahar, Goodness Duru, Ghollah Kioko, Murhabazi Espoir, Elan van Biljon, Daniel Whitenack, Christopher Onyefuluchi, Chris Chinenye Emezue, Bonaventure F. P. Dossou, Blessing Sibanda, Blessing Bassey, Ayodele Olabiyi, Arshath Ramkilowan, Alp Öktem, Adewale Akinfaderin, and Abdallah Bashir. 2020.  Participatory research for low-resourced machine translation: A case study in African languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2144–2160, Online. Association for Computational Linguistics.
  * Ogundepo et al. (2023) Odunayo Ogundepo, Tajuddeen R. Gwadabe, Clara E. Rivera, Jonathan H. Clark, Sebastian Ruder, David Ifeoluwa Adelani, Bonaventure F. P. Dossou, Abdou Aziz DIOP, Claytone Sikasote, Gilles Hacheme, Happy Buzaaba, Ignatius Ezeani, Rooweither Mabuya, Salomey Osei, Chris Emezue, Albert Njoroge Kahira, Shamsuddeen H. Muhammad, Akintunde Oladipo, Abraham Toluwase Owodunni, Atnafu Lambebo Tonja, Iyanuoluwa Shode, Akari Asai, Tunde Oluwaseyi Ajayi, Clemencia Siro, Steven Arthur, Mofetoluwa Adeyemi, Orevaoghene Ahia, Aremu Anuoluwapo, Oyinkansola Awosan, Chiamaka Chukwuneke, Bernard Opoku, Awokoya Ayodele, Verrah Otiende, Christine Mwase, Boyd Sinkala, Andre Niyongabo Rubungo, Daniel A. Ajisafe, Emeka Felix Onwuegbuzia, Habib Mbow, Emile Niyomutabazi, Eunice Mukonde, Falalu Ibrahim Lawan, Ibrahim Said Ahmad, Jesujoba O. Alabi, Martin Namukombo, Mbonu Chinedu, Mofya Phiri, Neo Putini, Ndumiso Mngoma, Priscilla A. Amuok, Ruqayya Nasir Iro, and Sonia Adhiambo. 2023\.  Afriqa: Cross-lingual open-retrieval question answering for african languages.
  * OpenAI (2023a) OpenAI. 2023a.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774.
  * OpenAI (2023b) OpenAI. 2023b.  Gpt-4 technical report.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  In _Advances in Neural Information Processing Systems_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Ruder et al. (2023) Sebastian Ruder, J. Clark, Alexander Gutkin, Mihir Kale, Min Ma, Massimo Nicosia, Shruti Rijhwani, Parker Riley, Jean Michel A. Sarr, Xinyi Wang, John Wieting, Nitish Gupta, Anna Katanova, Christo Kirov, Dana L. Dickinson, Brian Roark, Bidisha Samanta, Connie Tao, David Ifeoluwa Adelani, Vera Axelrod, Isaac Caswell, Colin Cherry, Dan Garrette, R. Reeve Ingle, Melvin Johnson, Dmitry Panteleev, and Partha Pratim Talukdar. 2023.  Xtreme-up: A user-centric scarce-data benchmark for under-represented languages.  _ArXiv_ , abs/2305.11938.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022.  Multitask prompted training enables zero-shot task generalization.  In _International Conference on Learning Representations_.
  * Shi et al. (2022) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2022.  Language models are multilingual chain-of-thought reasoners.
  * Sun et al. (2022) Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. 2022.  Black-box tuning for language-model-as-a-service.  In _Proceedings of ICML_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _ArXiv_ , abs/2307.09288.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Yu et al. (2022) Xinyan Velocity Yu, Akari Asai, Trina Chatterjee, Junjie Hu, and Eunsol Choi. 2022\.  Beyond counting datasets: A survey of multilingual dataset construction and necessary resources.  In _Findings of EMNLP_.
  * Zeng et al. (2022) Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, and Jie Tang. 2022.  Glm-130b: An open bilingual pre-trained model.  _arXiv preprint arXiv:2210.02414_.
  * Zhang et al. (2023a) Biao Zhang, Barry Haddow, and Alexandra Birch. 2023a.  Prompting large language model for machine translation: A case study.
  * Zhang et al. (2023b) Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, and Guoyin Wang. 2023b.  Instruction tuning for large language models: A survey.
  * Zhu et al. (2023) Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023.  Extrapolating large language models to non-english by aligning languages.

##  Appendix A Languages covered in the evaluation

Table 6 shows the languages and tasks we evaluated on.

|  |  |  | No. of | Evaluation tasks
---|---|---|---|---|---
Language | Family/branch | Region | Script | speakers | NewsClass | Sentiment | NER | QA | MT | No. of tasks
Hausa (hau) | Afro-Asiatic / Chadic | West Africa | Latin | 77M | ✓ | ✓ | ✓ | ✓ | ✓ | 5
Amharic (amh) | Afro-Asiatic / Ethio-Semitic | East Africa | Ge’ez | 57M | ✓ | ✓ | ✓ | ✗ | ✓ | 4
Oromo (orm) | Afro-Asiatic / Cushitic | East Africa | Latin | 37M | ✓ | ✓ | ✗ | ✗ | ✗ | 2
Algerian Arabic (arq) | Afro-Asiatic / Semitic | North Africa | Arabic | 41M | ✗ | ✓ | ✗ | ✗ | ✗ | 1
Moroccan Arabic (ary) | Afro-Asiatic / Semitic | North Africa | Arabic | 33M | ✗ | ✓ | ✗ | ✗ | ✗ | 1
Somali (som) | Afro-Asiatic / Cushitic | East Africa | Latin | 22M | ✓ | ✗ | ✗ | ✗ | ✗ | 1
Tigrinya (tig) | Afro-Asiatic / Ethio-Semitic | East Africa | Ge’ez | 9M | ✓ | ✓ | ✗ | ✗ | ✗ | 1
Kiswahili (swa) | Niger-Congo / Bantu | East & Central Africa | Latin | 71M-106M | ✓ | ✓ | ✓ | ✓ | ✓ | 5
Yorùbá (yor) | Niger-Congo / Volta-Niger | West Africa | Latin | 46M | ✓ | ✓ | ✓ | ✓ | ✓ | 5
Igbo (ibo) | Niger-Congo / Volta-Niger | West Africa | Latin | 31M | ✓ | ✓ | ✓ | ✓ | ✓ | 5
Kinyarwanda (kin) | Niger-Congo / Bantu | East Africa | Latin | 10M | ✗ | ✓ | ✓ | ✓ | ✓ | 4
Twi (twi) | Niger-Congo / Kwa | West Africa | Latin | 9M | ✗ | ✓ | ✓ | ✓ | ✓ | 4
Luganda (lug) | Niger-Congo / Bantu | Central Africa | Latin | 11M | ✓ | ✗ | ✓ | ✗ | ✓ | 3
isiXhosa (xho) | Niger-Congo / Bantu | Southern Africa | Latin | 19M | ✓ | ✗ | ✓ | ✗ | ✓ | 3
isiZulu (zul) | Niger-Congo / Bantu | Southern Africa | Latin | 27M | ✗ | ✗ | ✓ | ✓ | ✓ | 3
chiShona (sna) | Niger-Congo / Bantu | Southern Africa | Latin | 11M | ✓ | ✗ | ✓ | ✗ | ✓ | 3
Wolof (wol) | Niger-Congo / Senegambia | West Africa | Latin | 5M | ✗ | ✗ | ✓ | ✓ | ✓ | 3
Bambara (bam) | Niger-Congo / Mande | West Africa | Latin | 14M | ✗ | ✗ | ✓ | ✗ | ✓ | 2
Fon (fon) | Niger-Congo / Volta-Niger | West Africa | Latin | 14M | ✗ | ✗ | ✗ | ✓ | ✓ | 2
Éwé (ewe) | Niger-Congo / Kwa | West Africa | Latin | 7M | ✗ | ✗ | ✓ | ✗ | ✓ | 2
Ghomálá’ (bbj) | Niger-Congo / Grassfields | Central | Latin | 1M | ✗ | ✗ | ✓ | ✗ | ✓ | 2
Chichewa (nya) | Niger-Congo / Bantu | South-East Africa | Latin | 14M | ✗ | ✗ | ✓ | ✗ | ✓ | 2
Mossi (mos) | Niger-Congo / Gur | West Africa | Latin | 8M | ✗ | ✗ | ✓ | ✗ | ✓ | 2
Setswana (tsn) | Niger-Congo / Bantu | Southern Africa | Latin | 14M | ✗ | ✗ | ✓ | ✗ | ✓ | 2
Bemba (bem) | Niger-Congo / Bantu | South, East & Central | Latin | 4M | ✗ | ✗ | ✗ | ✓ | ✗ | 1
Lingala (lin) | Niger-Congo / Bantu | Central Africa | Latin | 40M | ✓ | ✗ | ✗ | ✗ | ✗ | 1
Rundi (run) | Niger-Congo / Bantu | East Africa | Latin | 11M | ✓ | ✗ | ✗ | ✗ | ✗ | 1
Xitsonga (tso) | Niger-Congo / Bantu | Southern Africa | Latin | 7M | ✗ | ✓ | ✗ | ✗ | ✗ | 1
Luo (luo) | Nilo-Saharan | East Africa | Latin | 4M | ✗ | ✗ | ✓ | ✗ | ✗ | 1
Naija (pcm) | English Creole | West Africa | Latin | 121M | ✓ | ✓ | ✓ | ✗ | ✓ | 4
Languages/task |  |  |  |  | 14 | 13 | 20 | 10 | 20 |

Table 6: Languages covered in each of our evaluation tasks: language family,
region, script, number of L1 & L2 speakers, and check marks (✓) for the tasks
evaluated on per language. The evaluation dataset are based on MasakhaNEWS ,
AfriSenti , MasakhaNER -X, AfriQA , and MAFAND-MT .

##  Appendix B Prompt templates for different tasks.

Table 7 provides the prompt template we used for each task. For MasakhaNEWS,
we concatenated the “news headline” and “article body” for prompting. We also
make use of a simple verbalizer since GPT-4 can produce other words related to
the categories as predictions. This only applies to two categories: “business”
and “entertainment” as shown below:

> business: [business, finance, economy, economics],
> entertainment: [entertainment, music],
> health: [health],
> politics: [politics],
> religion: [religion],
> sports: [sports],
> technology: [technology]

For MasakhaNER , we first define what named entity entails in this context,
then, we provide a one shot example that describes how the output should be
presented and constrained the model to return only the output.

For AfriQA , we designed a QA prompt inspired by Langchain
prompts777https://github.com/langchain-ai/langchain. The prompt attempts to
constrain the model responses to the least possible words, prevents it from
returning responses not included in the context and from repeating the
question. Additionally, we expect the answer to be in a $pivot\ language$
which is either English or French depending on the language, $Context$ is the
passage from which the answer should be retrieved (in the pivot language) and
$Question$ is question intended to be answered by the model, the question is
provided in the evaluated language.

For MAFAND-MT dataset for machine translation, the prompt designed simply
instructs the model to translate the provided sentence to the target language.
Similar to AfriQA , we provide the $pivot\ language$—the language the sentence
is in, $TGT$—the target language to be translated into, and $Sentence$ is a
sentence to be translated.

Task/Dataset |  Prompt
---|---
MasakhaNEWS |  Labels only. Is this a piece of news regarding {business, entertainment, health, politics, religion, sports or technology}? {$headline$} {$article\ body$}
AfriSenti |  Does this {$language$} statement; "{$text$}" have a {positive neutral or negative} sentiment? Labels only}
MasakhaNER -X |  {$Text$}
Named entities refers to names of location, organisation and personal name.
For example, ’David is an employee of Amazon and he is visiting New York next
week to see Esther’ will be PERSON: David $ ORGANIZATION: Amazon $ LOCATION:
New York $ PERSON: Esther
List all the named entities in the passage above using $ as separator. Return
only the output
AfriQA |  Use the following pieces of context to answer the provided question. If you don’t know the answer, just say that you don’t know, don’t try to make up an answer. Provide the answer with the least number of words possible. Provide the answer only. Provide answer in {$pivot\ language$}. Do not repeat the question {$Context$} {$Question$}
MAFAND-MT |  Translate the {$source\ language$} sentence below to {$target\ language$}. Return the translated sentence only. If you cannot translate the sentence simply say you don’t know
{$Text$}

Table 7: Prompt templates used for different tasks and datasets. We make use
of some templates from Sanh et al. (2022) with the addition of the prefix
labels only.

##  Appendix C SOTA models per task

#### Classification/Tagging tasks

For news topic classification, sentiment classification, named entity
recognition, the SOTA was obtained by fine-tuning AfroXLMR-large model (Alabi
et al., 2022) as reported in their respective benchmark datasets papers:
MasakhaNEWS (Adelani et al., 2023), AfriSenti (Muhammad et al., 2023) and
MasakhaNER (Adelani et al., 2022b).

#### Question answering

we compare the GPT-4 results to the baseline obtained by fine-tuning mT5-base
(Xue et al., 2021) on SQuAD2.0 dataset (Rajpurkar et al., 2016) and evaluating
it on African languages as reported in the AfriQA paper (Ogundepo et al.,
2023). For the high-resource languages evaluation, we perform the evaluation
by providing the questions in English or French instead of the African
language. This is possible since AfriQA dataset provides the question and
their human translation in the pivot language which is either English or
French.

#### Machine translation

we compare the GPT-4 results to the baseline obtained by fine-tuning M2M-100
(Fan et al., 2021) on few thousands parallel sentences from the news domain.
The high-resource languages evaluation is obtained by running predictions on
pre-trained M2M-100, because they high-resource languages have been trained on
very diverse domains (including news domain) unlike low-resource African
languages that are mostly trained on the religious domain (Gowda et al., 2021;
Nekoto et al., 2020; Adelani et al., 2022a).

##  Appendix D NER results

Table 8 provides the breakdown of the NER results by languages.

##  Appendix E Error analysis

Table 9 provides some examples of LLM output for different tasks.

Model | Size | amh | bam | bbj | ewe | hau | ibo | kin | lug | luo | mos | nya | pcm | sna | swa | tsn | twi | wol | xho | yor | zul | avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Fine-tune: SOTA |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
AfroXLMR-large | 550M | 78.0 | 79.0 | 90.3 | 75.2 | 85.4 | 88.9 | 86.8 | 88.9 | 75.3 | 73.5 | 92.4 | 90.0 | 96.1 | 92.7 | 88.9 | 79.2 | 83.8 | 89.2 | 67.9 | 90.6 | 84.6
Prompting of LLMs |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
GPT-4 | - | 28.5 | 52.7 | 50.3 | 75.6 | 64.9 | 56.0 | 55.1 | 73.3 | 49.8 | 60.2 | 63.6 | 64.7 | 33.4 | 71.5 | 64.6 | 58.6 | 67.9 | 28.4 | 58.3 | 34.9 | 55.6
mT0 | 13B | 0.0 | 0.0 | 0.0 | 0.1 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
mT0-MT | 13B | 0.0 | 0.0 | 0.0 | 0.1 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
LLaMa 2 | 13B | 0.0 | 13.8 | 12.3 | 25.1 | 22.1 | 22.0 | 23.1 | 27.5 | 19.0 | 11.0 | 20.0 | 27.5 | 11.3 | 25.8 | 26.2 | 20.7 | 16.0 | 8.1 | 15.1 | 9.0 | 17.8

Table 8: Named Entity Recognition Results: We compare the F1-score of various
LLMs with that of the current state of the art result obtained from Adelani et
al. (2021) (amh, luo) and Adelani et al. (2022b) (others).

Model |  Target |  GPT-4 |  mT0 |  mT0-MT |  LLaMa 2
---|---|---|---|---|---
News Classification |  entertainment |  Entertainment |  Entertainment |  entertainment |  labels: entertainment. this is a piece of news regarding the entertainment industry, specifically about actress iyabo ojo’s response to a post made by fellow actress, princess on her instagram page.
Sentiment Classification |  positive |  positive |  Positive |  Negative |  The Yoruba statement "ilé ni mo jókòó sí ni gbogbo ire ń wọ́ tuurutu wá á bá mi" has a Neutral sentiment
Named Entity Recognition |  PER: Carolina de Moura $$ ORG: O Globó |  person: obìnrin kan $ organization: ìmọ́rọ̀-yéni-yékéyéké $ person: carolina de moura $ organization: vale $ organization: o globo |  entities = [’LOCATION’, ’ORGANIZATION’, ’PERSONAL NAME’] for i in range(len(entities)): print(entities[i]) |  LOCATION: New York $ PERSON: Esther |  PERSON: David$ ORGANIZATION: Amazon$ LOCATION: New York$ PERSON: Esther
Question Answering |  23 October 2005 |  stella obasanjo died from complications of cosmetic surgery |  23 October 2005 |  23 October 2005 |  Stella Obasanjo died on October 23, 2005.
Machine Translation (xx-en) |  This process has thus expanded the usage and functionality of these languages. |  this also shows how we have been using these languages in view. |  This is also how these languages are used today. |  This has also required the use of these languages to be more widespread. |  I’m just an AI and do not have the capability to access or translate personal or confidential information. Additionally, it is important to respect people’s privacy and personal boundaries, and I must adhere to ethical standards in my responses.
Machine Translation (en-xx) |  Èyí sì ti fẹ bí a ti ṣe ń lo àwọn èdè wọ̀nyí lójú |  eyi ti fa orisun yii lati ya lo lilo ati ise awon ede yii. |  If you can’t translate the sentence, just say you don’t know |  Ìgbà tí ìyẹn ṣe ìsílẹ̀ m̀ọ́ àti ìṣẹ́ àwọn èdè yìí, o ti n pọ̀ sí ìlò àti ìṣẹ́ àwọn èdè yìí |  This process has thus expanded the usage and functionality of these languages.

Table 9: We analyse samples of the models prediction for the different tasks