# AfriWOZ: Corpus for Exploiting Cross-Lingual Transferability for Generation
of Dialogues in Low-Resource, African Languages

Tosin Adewumi⋄1†, Mofetoluwa Adeyemi†, Aremu Anuoluwapo†, Bukola Peters2,
Happy Buzaaba†,
Oyerinde Samuel†, Amina Mardiyyah Rufai†, Benjamin Ajibade†, Tajudeen
Gwadabe†,
Mory Moussou Koulibaly Traore†, Tunde Ajayi†, Shamsuddeen Muhammad†, Ahmed
Baruwa†,
Paul Owoicho†, Tolulope Ogunremi†, Phylis Ngigi†, Orevaoghene Ahia†, Ruqayya
Nasir†,
Foteini Liwicki1 corresponding author Marcus Liwicki1
1ML Group, Luleå University of Technology/ †Masakhane/ 2CIS

###### Abstract

Dialogue generation is an important NLP task fraught with many challenges. The
challenges become more daunting for low-resource African languages. To enable
the creation of dialogue agents for African languages, we contribute the first
high-quality dialogue datasets for 6 African languages: Swahili, Wolof, Hausa,
Nigerian Pidgin English, Kinyarwanda & Yorùbá. These datasets consist of 1,500
turns each, which we translate from a portion of the English multi-domain
MultiWOZ dataset. Subsequently, we investigate & analyze the effectiveness of
modelling through transfer learning by utilziing state-of-the-art (SoTA)deep
monolingual models: DialoGPT and BlenderBot. We compare the models with a
simple seq2seq baseline using perplexity. Besides this, we conduct human
evaluation of single-turn conversations by using majority votes and measure
inter-annotator agreement (IAA). We find that the hypothesis that deep
monolingual models learn some abstractions that generalize across languages
holds. We observe human-like conversations, to different degrees, in 5 out of
the 6 languages. The language with the most transferable properties is the
Nigerian Pidgin English, with a human-likeness score of 78.1%, of which 34.4%
are unanimous. We freely provide the datasets and host the model
checkpoints/demos on the HuggingFace hub for public access.

##  1 Introduction

The ability to understand and converse fluently in natural language is
considered a major component of intelligence. Over the years, open-domain
conversational (or dialogue) systems have evolved Weizenbaum (1969); Zhang et
al. (2020); Roller et al. (2021); Adiwardana et al. (2020); Adewumi et al.
(2019). Advances in deep neural networks, such as the Transformer-based
architectures, have brought improvements to the field Vaswani et al. (2017);
Devlin et al. (2018a); Radford et al. (2019); He et al. (2020). These models
have demonstrated SoTA performances in Natural Language Understanding (NLU)and
Natural Language Generation (NLG)tasks Wang et al. (2019); Gehrmann et al.
(2021).

While significant advancements have been made in the field, the majority of
focus has been on the English language. For example, many models were
originally pretrained on English data, though researchers have recently been
producing some multilingual versions Devlin et al. (2018b); Conneau and Lample
(2019); Xue et al. (2021). Some of these multilingual models, however, have
been shown to have poor performance compared to models trained completely on
the target language data Virtanen et al. (2019); Rönnqvist et al. (2019). NLP
challenges get more daunting for languages that do not have sufficient data to
train with, usually called low-resource languages Nekoto et al. (2020);
Adewumi et al. (2020); Adelani et al. (2021). Thus, the multilingual versions
of the deep models do not cover many of these languages. For example, Table 1
reveals languages not covered by some of the multilingual models and Google
Machine Translate111As of April 15, 2022. This shows many languages are still
under-represented. Besides the challenge of availability of data or high-
quality data, there are also technical Roller et al. (2021) and ethical
challenges Dinan et al. (2020); Javed et al. (2021).

Language | Multilingual model
---|---
| mBERT | mBART | mT5 | XLM-R | AfriBERTa | Google MT
Pidgin English | X | X | X | X | $\sqrt{}$ | X
Yorùbá | $\sqrt{}$ | X | $\sqrt{}$ | X | $\sqrt{}$ | $\sqrt{}$
Hausa | X | X | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$
Wolof | X | X | X | X | X | X
Swahili | $\sqrt{}$ | X | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$
Kinyarwanda | X | X | X | X | X | $\sqrt{}$

Table 1:  The languages in some models: $\sqrt{}$: yes, X: no

The motivation and contributions of this study are to (1) create the first
high-quality dialogue datasets for the target languages from the benchmark
MulitWOZ dataset Budzianowski et al. (2018) and (2) investigate by transfer
learning, for open-domain dialogue systems, the hypothesis that deep
monolingual models learn some abstractions that generalise across languages
Artetxe et al. (2020). The contribution of the data provides other side-
contributions because they may be adapted for other NLP tasks, such as Machine
Translation (MT), task-based dialogue systems and automatic speech recognition
(ASR), among others. We obtain promising results that apparently validate the
stated hypothesis and obtain better human evaluation results for 2 of the
languages than what was shown for Swedish in a similar setup by Adewumi et al.
(2022). We freely provide the codes, datasets222github.com/masakhane-
io/chatbots-african-languages and model checkpoints/demos for public use on
the HuggingFace hub333huggingface.co/tosin. The findings of this study seem to
be the first for the languages concerned under open-domain dialogue setting,
to the best of our knowledge.

The rest of this paper is organised as follows. The ‘languages of the study’
section (2) presents brief details of the languages; the methodology section
(4) describes the experimental setup, data models and modes of evaluation,
including the newly-introduced credibility unanimous score (CUS)for IAA; the
results & discussion section (5) presents the tables of results and evaluation
for all the models, including the error analysis; the conclusion section (7)
then follows after the related work section (6) and limitation section (8).

##  2 Languages of the Study

The vast majority of work in dialogue focuses on high-resource languages like
English Zhang et al. (2020); Adiwardana et al. (2020), Chinese, with limited
work in other high-resource languages. We focus on 6 African languages spoken
by millions of people. The languages were selected for this work based on
their diversity and the availability of contributors. They cover countries in
West, East, Central and Southern Africa Heine et al. (2000) and over 239
million speakers combined. Examples of translated sentences for each language
are given in Table 2. The examples are from the training set of the English
MultiWOZ dataset.

Language |  Example of 3 Conversation turns
---|---
English |  I have several options for you;
|  I want to book it for 2 people and 2 nights starting from Saturday.
|  That is all I need to know. Thanks, good bye.
Nigerian Pidgin English |  I get plenty options for you;
|  I wan book am for 2 people for 2 night for Saturday
|  Na everything wey i need to know. thank you. good bye
Yorùbá |  Mo ní awọ́n àṣàyàn púpọ̀ fún ọ;
|  Mo fé ṣe ìwé fún ènìyàn méjì àti fún alẹ́ méjì tí ó bẹ́rẹ̀ láti ọjọ́
Sátìdeé.
|  Ìyẹn ni gbogbo ohun tí mo nílò láti mọ̀. O ṣeun, Ó dàbò.
Hausa |  Ina da zabubbuka da yawa a gare ku;
|  Ina so in yi wa mutane 2 da dare 2 farawa daga ranar Asabar.
|  Wannan shine kawai abin da nake bukatar sani. Godiya, bye bye.
Wolof |  Amna ay tanneef yu bari ngir yaw;
|  Soxla jënd ngir ñaari niit ak ñaari guddi mu tambelee Gawu
|  Dedet li rek la soxla. jerejef. ba benen yoon
Swahili |  Nina chaguzi kadhaa kwako;
|  Nataka kuihifadhi kwa watu 2 na usiku 2 kuanzia Jumamosi.
|  Hiyo ndiyo yote ninahitaji kujua. Asante, kwaheri.
Kinyarwanda |  Mfite henshi naguhitiramo hari;
|  Ndashaka kubika imyanya ku bantu 2 n’amajoro 2 guhera ku wa Gatandatu.
|  Ibyo ni byo nari nkeneye kumenya. Urakoze, murabeho.
Table 2:  Translation examples from the English MultiWOZ data for the six
languages.

##### Swahili

Swahili is a Bantu language. It is spoken by the Bantu people in the southern
half of Africa Polomé (1967). It is an official language of the East African
Community (EAC) countries. These include: Uganda, Burundi, Kenya, Tanzania,
Rwanda, South Sudan and the Democratic Republic of the Congo (DRC). It is a
lingua franca of other areas like Malawi, Mozambique, the southern tip of
Somalia, and Zambia Polomé (1967). There are more than 50 million speakers of
the language. 444swahililanguage.stanford.edu It is also one of the working
languages of the African Union.

##### Wolof

Wolof is spoken in Senegal, Mauritania and the Gambia. More than 7 million
people are believed to speak the
language555worlddata.info/languages/wolof.php. It is of the Senegambian branch
of the Niger–Congo language phylum, which is the largest language phylum in
the world Heine et al. (2000). Unlike most other languages of the Niger-Congo
phylum, Wolof is not a tonal language.

##### Hausa

Hausa is a Chadic language spoken by the Hausa people. It is mainly within the
northern part of Nigeria and the southern part of Niger. It has significant
minorities in Cameroon, Chad, and Benin. It is the most widely spoken language
within the Chadic branch of the Afroasiatic phylum Heine et al. (2000). It has
more than 40 million speakers666britannica.com/topic/Hausa-language.

##### Nigerian Pidgin English

Nigerian Pidgin English is a grammatically simplified means of communication
among the ethnic groups in Nigeria. Its vocabulary and grammar are limited and
often drawn from the English language. It is popular among young people
Ihemere (2006). About 75 million are estimated to speak the language but the
exact number is difficult to estimate since it is not an official
language777bbc.com/news/world-africa-38000387.

##### Kinyarwanda

Kinyarwanda is an official language of Rwanda and a dialect of the Rwanda-
Rundi language spoken in Rwanda Heine et al. (2000). It is one of the four
official languages of Rwanda. Over 22 million people are estimated to speak
the language888worlddata.info/languages/kinyarwanda.php.

##### Yorùbá

Yorùbá is predominantly spoken in Southwestern Nigeria by the ethnic Yorùbá
people Heine et al. (2000). It is primarily spoken in a dialectal area
spanning Nigeria and Benin with smaller migrated communities in Cote d’Ivoire,
Sierra Leone and The Gambia. The number of Yorùbá speakers is more than 45
million999worlddata.info/languages/yoruba.php.

##  3 An African Dialogue Dataset: AfriWOZ

The Yorùbá language has small dialogue data online101010YorubaYeMi-
textbook.pdf & theyorubablog.com, unlike the other languages. We chose to use
these sources for Yorùbá because of the local entities represented in the data
and then augment the data if necessary. As a result of the scarcity or non-
existent dialogue data for most of the languages, the authors decided to
translate an English dialogue dataset. The poll was between
Reddit111111reddit.com/ and MultiWOZ Budzianowski et al. (2018). Most
contributors voted in favour of MultiWOZ, though it is from task-oriented
dialogues, instead of Reddit because of the high probability of toxic content
Roller et al. (2021). Indeed, in order to address the challenge of toxic
comments in dialogues Dinan et al. (2019), Solaiman and Dennison (2021)
advocated for the approach of carefully curating dataset as a safe approach.
They observed that the adjustment of a model’s behavior is possible with a
small, hand-curated dataset. This approach takes ethical considerations into
account Jurafsky and Martin (2020); Javed et al. (2021). We follow this
approach.

###  3.1 MultiWOZ

MultiWOZ is a collection of human-human written conversations that span
multiple domains and topics. It has gone through improvements and extensions
over the years and currently has about 10,000 dialogues Eric et al. (2020).
Its multiple domain/topic coverage, though limited, makes it ideal for open-
domain modeling. Indeed, Budzianowski et al. (2018) experimented with it for
neural response generation, showing its usefulness across a range of dialogue
tasks. It has over 113,000 turns in the training set and over 14,700 turns
each in both the validation and test sets. Some of the domains covered are
hospital, police, attraction, hotel, restaurant, taxi, train and booking.

In our work, we extracted and translated the first 1,000 turns from the
training set and the first 250 turns each from the validation and test sets
for the languages. Only 200 turns from the MultiWOZ training set were added to
make up the 1,000 turns for the Yorùbá data. The two Yorùbá sources are a mix
of short dialogues in different scenarios including the market, home and
school. We call the collection of these corpora AfriWOZ. It is interesting to
note that though the data sizes are small, they are still larger than the COPA
benchmark dataset available on the SuperGLUE Wang et al. (2019). In line with
data acquisition standards Bender and Friedman (2018), we provide the short
data statement below and Table 3 gives characteristics of the dataset.

> Short data statement for the AfriWOZ dataset.
> This is the AfriWOZ dataset for training and evaluating open-domain dialogue
> models.
> The licence for using this dataset comes under CC-BY 4.0.
> Total natural languages: 6 (Swahili, Wolof, Hausa, Nigerian Pidgin English,
> Kinyarwanda & Yorùbá)
> Total turns in the training set per language: 1,000
> Total turns in the validation set per language: 250
> Total turns in the test set per language: 250
> Domains covered in the data include hotel, restaurant, taxi and booking.
> The long version of this data statement is in the appendix.

Language | Characteristics
---|---
| Source | Translation method
Pidgin English | M | HT
Yorùbá | B+M | HT
Hausa | M | MT+HR
Wolof | M | HT
Swahili | M | HT
Kinyarwanda | M | HT

Table 3:  AfriWOZ dataset characteristics. Each contains 1,500 turns. (M:
MultiWOZ; B: Blog; HT: human translation; MT: machine translation; HR: human
review)

###  3.2 Translation Quality

The translators, recruited online on Slack121212slack.com/, are native/L1
speakers of the target languages and second/L2 (but dominant) speakers of
English. They were to use either of the two possibilities for translation:
human translation or MT through Google MT plus human review of all
translations, for quality control (QC). Each corpus is reviewed by the
coordinator of each language. Particularly, the Yorùbá language had a linguist
review the data. The risk of translating English conversations into unnatural
conversations in the target languages was mitigated by using native speakers
instead of just MT.

###  3.3 Translation Challenges

The two main human translation challenges encountered include handling English
entities and reframing English conversations for cultural relevance in the
target languages. Generally, entities in the data were retained, especially as
this may facilitate MT task. In the future, this may be changed or two
versions of the data maintained: one version with all the English entities and
a second version with each language’s common entities. The experience and
cultural background of the native speakers made it relatively simple to frame
the English conversations into what seem natural in the target languages.

##  4 Experiments

We compare 3 models: dialogue generative pre-trained transformer
(DialoGPT)Zhang et al. (2020), BlenderBot 90M Roller et al. (2021) and a
simple Seq2Seq with attention mechanism, as a baseline, based on the ParlAI
platform by Miller et al. (2017). Experiments were conducted using a
participatory approach Nekoto et al. (2020) on Google Colaboratory with free
GPUs. Some experiments were on a shared DGX-1 machine with 8 × 32GB Nvidia
V100 GPUs. The server runs on Ubuntu 18 and has 80 CPU cores. Each experiment
was conducted 3 times and the average perplexity (including standard
deviation) was obtained.

###  4.1 Models

The finetuning/training process for BlenderBot 90M and the seq2seq models was
for about 20 minutes each. Finetuning DialoGPT on each of the datasets for 3
epochs takes less than 20 minutes. We did not do extensive hyperparameter
search due to the constraints of time and resources. The decoding algorithm
across the models was set as top-k (k=100) and top-p (p=0.7). We do not
finetune/update the default tokenizer with new tokens or words from the target
languages. Instead, we leverage the default/generic tokenizers of the selected
models. We also attempted to compare the AfriBERTa tokenizer, which is trained
for several African languages, by swapping it in for DialoGPT, but this was
incompatible. In addition, we recognize that the 3 models do not have exactly
the same parameters or configuration and are, therefore, not expected to have
the same performance.

####  4.1.1 DialoGPT

Zhang et al. (2020) introduced 3 sizes of the DialoGPT: the large, medium and
small. It is an English pretrained model for open-domain chatbots based on
GPT-2. It was trained on 147M turns of Reddit comments. It uses byte-pair
encoding (BPE)tokenizer. The medium model is reputed to have the best
performance compared to its large and small versions. In this work, however,
we use the small version to minimize the problem of overfitting over small
datasets. We utilize the pretrained model from the HuggingFace hub and the
generic autotokenizer Wolf et al. (2020). The small model has 117M parameters,
12 layers and uses a vocabulary of 50,257 entries. We use a batch size of 2
during finetuning because of memory constraints and perform ablation studies
over the conversation context with values of 7 and 14, noting though that
larger context sizes will bring memory challenges Adiwardana et al. (2020).

####  4.1.2 BlenderBot 90M

The model is a pretrained transformer model loaded from the ParlAI hub Miller
et al. (2017). It has 8 layers, 16 heads, uses Adam optimizer and byte-level
BPE for tokenization. It has 87.5M trainable parameters, a batch size of 6 for
finetuning and starts with the learning rate of 1e-5. A variant of English
Reddit discussions covering a vast range of topics and totaling 1.5B comments
was used to train the model. However, the data consists of group discussions
instead of direct two-way conversational data.

####  4.1.3 Seq2Seq

The seq2seq is an encoder-decoder model that is based on the LSTM architecture
Hochreiter and Schmidhuber (1997) and uses the attention mechanism Bahdanau et
al. (2015). It was trained from scratch (random initialization) on the
datasets in order to compare as a baseline. The model has 805,994 trainable
parameters and uses a batch size of 64.

###  4.2 Evaluation

For automatic evaluation, we follow Adiwardana et al. (2020) and report only
perplexity. This is because automatic metrics typically used in MT, such as
BLEU, are poor for open-domain dialogue systems Jurafsky and Martin (2020);
Lundell Vinkler and Yu (2020). Having multiple valid responses to prompts as
reference is important for meaningful automated evaluations Gangal et al.
(2021). These multiple valid responses are usually difficult to construct.
Probably the best evaluation is done by humans, though this may be subjective.
For human evaluation, we follow a similar method as in the original work by
Zhang et al. (2020).

####  4.2.1 Perplexity

Perplexity shows how well a model predicts a sample. It minimizes the
uncertainty of predicting the next token. Ideally, the lower the perplexity,
the better the model performs and the higher the perplexity, the more unsure
the model is at predicting the next token Adiwardana et al. (2020). This is
used often to evaluate the language models built with n-grams of text dataset
Sennrich (2012). Perplexity has been shown to correlate with the human
evaluation metric called Sensibleness and Specificity Average (SSA) by
Adiwardana et al. (2020). However, correlation of perplexity with human
judgment is not always straightforward, as observed by Roller et al. (2021)
and Hashimoto et al. (2019).

####  4.2.2 Human Evaluation

We use the observer evaluation method, where evaluators (or annotators) read
transcripts of conversation Jurafsky and Martin (2020). We ask human
evaluators to rate single-turn conversations for human-likeness on a Likert
scale with 3 entries (human-like (H), non-human-like (N) or uncertain (U)).
The reason is that lack of long-term contextual information is still an
existing problem in conversational systems Zhang et al. (2020). A copy of each
transcript is given to 3 native speakers per language to evaluate. A total of
32 single-turn conversations are generated per language and 3 credibility test
conversations spread out within the transcript (at positions 11, 21 and 26) to
make up 35. Putting more test conversations would have been desirable but we
chose to balance this with the attention-span of the annotators, as lengthy
transcripts demand more time. A random list was generated and used to select
the same 32 prompts for all the languages from each test set. Only one model,
which had the best perplexity across languages, was used to generate the
conversations: DialoGPT c7 x 1,000 (having context size 7 and 1,000 training
turns), though small scale human evaluation is carried out to verify sample
conversations from the other models: BlenderBot 90M and the seq2seq. The
transcripts are available online2.

Out of the total (24) transcripts returned, 6 were not credible. Three
credible evaluations per language were processed for result computation.
Simple majority vote decided the annotation of each single-turn conversation.
The credibility test conversations fulfil 2 goals: 1) they help us check if
annotators are qualified or paying attention and 2) they introduce a new way
to determine IAA in a simple way, especially since the tests are homogeneous
to the rest of the conversations. We call this credibility unanimous score
(CUS)and discuss it further in the next section. Discredited evaluations are
the ones that failed 2 or more out of the 3 credibility test conversations by
marking them as anything but H. The 3 credibility conversations are prompts
and responses directly from the test set instead of generated responses from
the model. A simple instruction for every evaluator at the top of the
transcript of conversations is given below.

> Below are 35 different conversations by 2 speakers. Please mark each one as
> Human-like (H) or Non human-like (N) or Uncertain (U) based on your own
> understanding of what is human-like.

Selection of evaluators
The evaluators/annotators were recruited online on Slack. They are also
native/L1 speakers of the target languages and second/L2 (but dominant)
speakers of English. These are unbiased respondents who are not connected to
the translation of the datasets nor did they take part in the training of the
models.

####  4.2.3 CUS

The basic assumption behind CUS is that if homogeneous samples that are
introduced into the transcript can be used for establishing the credibility of
the annotators, then they may be used for establishing their agreement. It may
be seen as a proxy over the entire transcript. CUS is more intuitive, easier
to calculate (as it’s based on percentages) and seemingly less sensitive to
changes in the number of categories being evaluated, compared to Fleiss Kappa
(k). It is based on unanimous votes across the homogeneous samples. The
probability of obtaining high CUS rises when the benchmark score for annotator
credibility is raised. For example, if the benchmark scores for accepting
annotators’ work in two different jobs are 51% and 71%, then the probability
of getting a higher CUS is higher in the latter. This gives CUS an advantage
over using raw percentages over the actual samples, due to the possibility of
agreements by chance, which is likely in raw percentages.

##  5 Results & Discussion

###  5.1 Performance on African Languages

Table 4 shows the perplexity results across the three models for the African
languages. DialoGPT with a context size of 14 achieves the best (lowest
perplexity) result for each language, in the table. This is inspite of using
half the training size that is used for the BlenderBot 90M and Seq2Seq models.
Generally, DialoGPT performs best across the languages but there are languages
that do not perform so well and the Hausa language Seq2Seq overfits. In the
relevant tables, sd, c7, and c14 stand for standard deviation, context size 7,
and context size 14, respectively. Also, bold figures are the better values
per language.

Language | Model | Training turns | Perplexity
---|---|---|---
|  |  | Dev (sd) | Test (sd)
Nigerian Pidgin English | DialoGPT c14 | 500 | 67.57 (2.53) | 90.18 (3.24)
| BlenderBot 90M | 1,000 | 81.23 (0) | 81.23 (0)
| Seq2Seq | 1,000 | 277.2 (15) | 277.2 (15)
Yorùbá | DialoGPT c14 | 500 | 12.63 (0.47) | 10.66 (0.40)
| BlenderBot 90M | 1,000 | 154.43 (0.06) | 154.43 (0.06)
| Seq2Seq | 1,000 | 45.85 (1.41) | 45.85 (1.41)
Hausa | DialoGPT c14 | 500 | 26.40 (0.75) | 35.95 (0.73)
| BlenderBot 90M | 1,000 | 39.39 (1.61) | 39.39 (1.61)
| Seq2Seq | 1,000 | 1.92 (0.12) | 1.92 (0.12)
Wolof | DialoGPT c14 | 500 | 15.2 (0.09) | 26.41 (0.10)
| BlenderBot 90M | 1,000 | 108.7 (0) | 108.7 (0)
| Seq2Seq | 1,000 | 401.6 (10.39) | 401.6 (10.39)
Swahili | DialoGPT c14 | 500 | 20.03 (0.29) | 17.02 (0.22)
| BlenderBot 90M | 1,000 | 128.8 (0.10) | 128.8 (0.10)
| Seq2Seq | 1,000 | 134.5 (2.75) | 134.5 (2.75)
Kinyarwanda | DialoGPT c14 | 500 | 24.47 (0.17) | 26.45 (0.17)
| BlenderBot 90M | 1,000 | 177.87 (0.06) | 177.87 (0.06)
| Seq2Seq | 1,000 | 195.07 (7.66) | 195.07 (7.66)
Table 4:  Results across the 3 main models Language | Training turns | Perplexity
---|---|---
|  | Dev (sd) | Test (sd)
Nigerian Pidgin English | 500 | 42.55 (0) | 52.81 (0)
| 1,000 |  37.95 (0.66) |  46.56 (1.13)
Yorùbá | 500 | 10.52 (0.04) | 9.65 (0.01)
| 1,000 |  7.22 (0.06) |  8.76 (0.08)
Hausa | 500 | 18.53 (0.23) | 25.7 (0.4)
| 1,000 |  9.92 (0.05) |  12.89 (0.04)
Wolof | 500 | 15.2 (0.09) | 26.41 (0.10)
| 1,000 |  14.91 (0.3) |  25.85 (0.04)
Swahili | 500 | 15.55 (0.17) | 14.22 (0.14)
| 1,000 |  9.63 (0) |  9.36 (0.03)
Kinyarwanda | 500 | 19.28 (0.19) | 21.62 (0.22)
| 1,000 |  10.85 (0) |  14.18 (0.08)
Table 5:  Ablation study of DialoGPT-c7 over training turns. Bold figures are
the better values per language.

###  5.2 Performance vs. Amount of Data or Context size

Taking the best model from Table 4, which is DialoGPT, and doing ablation
studies over the training set size and the context size, we obtain results in
Tables 5 and 6, respectively. We observe that increasing the training set size
by doubling the number of turns brings improvement by lowering the perplexity
for the model of each language. Doubling the context size, however, does not
have a similar effect. Performance, in terms of perplexity, only improves when
we half the context size from 14 to 7. The better values are given in bold in
each table. The results are statistically significant, as all p-values (p <
0.0001) for the difference of two means of the two-sample t-test (between the
two lowest results) for all the languages are smaller than the alpha (0.05).
Given that these results are obtained with small data, we believe increasing
the data size will improve the results.

Language | Context size | Perplexity
---|---|---
|  | Dev (sd) | Test (sd)
Nigerian Pidgin English | c7 |  37.95 (0.66) |  46.56 (1.13)
| c14 | 70.21 (2.17) | 92.23 (2.33)
Yorùbá | c7 |  7.22 (0.06) |  8.76 (0.08)
| c14 | 7.63 (0.13) | 9.11 (0.14)
Hausa | c7 |  9.92 (0.05) |  12.89 (0.04)
| c14 | 11.30 (0.04) | 15.16 (0.05)
Wolof | c7 |  14.91 (0.3) |  25.85 (0.04)
| c14 | 16.61 (0.2) | 30.37 (0.08)
Swahili | c7 |  9.63 (0) |  9.36 (0.03)
| c14 | 11.07 (0.04) | 10.71 (0.05)
Kinyarwanda | c7 |  10.85 (0) |  14.18 (0.08)
| c14 | 12.84 (0.1) | 17.43 (0.14)
Table 6:  Ablation study of DialoGPT over context sizes for training set with 1,000 turns. Bold figures are the better values per language. Model language | Scale (majority votes - 2/3) | CUS | Fliess k
---|---|---|---
| H (%) | U (%) | N (%) | 3-way (%) | % |
Nigerian Pidgin English | 78.1 | 0 | 6.3 | 15.6 | 66.7 | -0.079
Yorùbá | 0 | 3.1 | 75 | 21.9 | 33.3 | -0.154
Hausa | 31.3 | 6.3 | 53.1 | 9.4 | 66.7 | 0.228
Wolof | 65.6 | 0 | 31.3 | 3.1 | 100 | 0.070
Swahili | 28.1 | 15.6 | 34.4 | 21.9 | 66.7 | 0.067
Kinyarwanda | 28.1 | 25 | 34.4 | 12.5 | 66.7 | 0.091
| unanimous votes - 3/3 |  |
Nigerian Pidgin English | 34.4 | 0 | 0 | - | 66.7 |
Yorùbá | 0 | 0 | 25 | - | 33.3 |
Hausa | 12.5 | 0 | 21.9 | - | 66.7 |
Wolof | 15.6 | 0 | 9.4 | - | 100 |
Swahili | 9.4 | 0 | 9.4 | - | 66.7 |
Kinyarwanda | 9.4 | 0 | 6.3 | - | 66.7 |
Table 7: Human evaluation results of 3 annotators on 3 classes using single-
turn conversations. A recent human-human upperbound is 92.1%, according to
Adewumi et al. (2022). The subjective Kappa example of 2 annotators on 2
classes does not apply here since Kappa is lower when classes are more Sim and
Wright (2005). - implies not applicable.

###  5.3 Human evaluation

We observe from Table 7 that the single-turn conversations of the Nigerian
Pidgin English are judged as human-like 78.1% of the time by majority votes.
34.4% of them are unanimously judged as human-like, which is higher than both
the 3-way split (when each annotator voted for each different category) of
15.6% or non-human-like of 6.3%. This is intuitive, since Pidgin English is
closely related to the English language, which is the language of pretraining.
Meanwhile, the Yorùbá transcript has 0% human-like single-turn conversation.
This may be because of a combination of reasons, including the language’s
morphology and written accent, among others. It has the most peculiarities in
written form, as shown in Table 2, making it challenging for the model. Wolof,
Hausa, Kinyarwanda and Swahili follow after Nigerian Pidgin English with
65.6%, 31.3%, 28.1% and 28.1% of conversations judged as human-like,
respectively.

Figure 1 depicts the human-likeness scores and the credibility unanimous
scores for the languages, as given in Table 7. When we compare the best-
performing (Nigerian Pidgin English) with a recent human-human upper-bound
(92.1%) for conversations, given by Adewumi et al. (2022), we observe that
this best-performing model is still behind in terms of performance.

Figure 1: Bar chart of human-likeness scores and credibility unanimous scores
(cus) for the languages. Generally, CUS is high for the languages, except for
Yorùbá.

The Fleiss k scores are not interpretable using the Kappa 2 annotators on 2
classes guide Landis and Koch (1977), as our study uses 3 annotators on 3
classes and the Kappa is lower when the classes are more Sim and Wright
(2005). Indeed, our study confirms the observation made by Gwet (2014) that
the interpretation guide may be more harmful than helpful. Perez Almendros et
al. (2020) also report how the Kappa score rose from 41% to 61% when the
classes were reduced from 3 to 2. We believe that CUS is a more intuitive and
reliable method, at least, in this case because of the homogeneous test
conversations.

###  5.4 Error Analysis

We highlight a few of the challenges of the models. DialoGPT, though the best
model in our investigation, shows weaknesses, particularly with the Yorùbá
language by failing to generate the correct word morphology, sometimes. Below
are 2 examples where the conversations are not coherent.

  * •

prompt: Ẹsẹ̀ ń ro mí

  * •

response: Eèlóò ni orúkọ ọdún?

  * •

prompt: Ìyẹn náà da, wọ́n ṣè ń darúgbó náà ni

  * •

response: Kì ni orúkọ́ orúnju rẸ?

On the other hand also, BlenderBot 90M mixes English responses with its
generation sometimes, even though the prompts are of the foreign target
language. Two examples of prompts in Hausa and responses are given below.
There, we also see that it echoes the prompt given to it in the first pair of
example.

  * •

prompt: Barka dai, me za ku iya gaya mani game da gidan abincin bangkok na
burni?

  * •

response: barka dai , za za ku iya gaya mani game da gidan abincin bangkok na
burni ? i ’ m not sure if you ’ re being sarcastic or not .

  * •

prompt: Ee na mutum 2 da 19:30 na ranar Lahadi don Allah. Ina bukatan lambar
nuni kuma.

  * •

response: haha , i ’ m not sure if i ’ m the only one who thinks that .

A major drawback of the seq2seq model is the overly repetitive nature of some
of its responses across different prompts. Two examples of the prompts and
generated responses for the Nigerian Pidgin English are given below. We see
exactly the same response for two different prompts and the response is not
coherent.

  * •

prompt: I need one place to chop for the center wey dey expensive. Good bye

  * •

response: I dey go like .

  * •

prompt: e dey okay. i fit get that number. you fit also recommend me hotel wey
dey expensive

  * •

response: I dey go like .

###  5.5 Post-Evaluation Feedback

Some of the evaluators, in post-evaluation feedback, explained that coherence
of the conversation mattered as a deciding factor in their judgment. So did
the grammar. For example, considering Yorùbá, responses that referenced
inanimate objects as if in the context of animate objects or humans were voted
as non-human-like. Wrong placement of an accent mark or character was also
motivation for voting conversations as non-human-like. For Wolof, many of the
conversations are human-like except for cases where the responses were
inconsistent with the prompt or question given. For example, there were
conversations that were hard to judge because the responses are questions to
the prompts, which happen to be questions themselves. Such conversations were
awarded the uncertain votes by the particular annotator.

##  6 Related Work

There are a number of pretrained models for open-domain conversational
systems. Some of them include Texar Hu et al. (2018), DLGnet Olabiyi and
Mueller (2019), Meena Adiwardana et al. (2020) and BlenderBot Roller et al.
(2021). These are pretrained on dialogue datasets. In BlenderBot 2 Komeili et
al. (2021); Xu et al. (2021), the same BlendedSkillTalk (BST) Smith et al.
(2020) collection of datasets used for BlenderBot 1 Roller et al. (2021) is
used to train the model, in addition to 3 others. There exist, also, models
pretrained on large text and adapted for conversational systems. Such models
include T5 Raffel et al. (2020) and BART Lewis et al. (2020). Another
pretrained model on conversational data, DialoGPT, was trained on Reddit
conversations of 147M exchanges Zhang et al. (2020). In single-turn
conversations, it achieved performance close to that of humans in open-domain
dialogues. DialoGPT is based on GPT-2 Radford et al. (2019). It is an
autoregressive model, which achieved SoTA results in different NLP tasks
Radford et al. (2019).

Solaiman and Dennison (2021) observed different harmful outputs in GPT-3, the
successor of the GPT-2 model. They discovered that a mitigating factor is
carefully curating a small dataset, which determines the behaviour of the
model outputs. They made a good case for fine-tuning non-toxic text compared
to reducing toxicity through controllable methods using filters or control
tokens. Topics such as history, science and government were covered in the
dataset (Solaiman and Dennison, 2021). The 80 texts in the values-targeted
dataset utilized by Solaiman and Dennison (2021) range in length from 40 to
340 words.

Recently, Artetxe et al. (2020) hypothesised that deep monolingual models
learn some abstractions that generalise across languages, while working on
cross-lingual transferability. This is in contrast to the past hypothesis that
attributes the generalization ability of multilingual models to the shared
subword vocabulary used across the languages and joint training, as
demonstrated for mBERT Pires et al. (2019). Besides mBERT, there other
multilingual deep models Ogueji et al. (2021); Devlin et al. (2018b); Conneau
and Lample (2019); Reid et al. (2021); Xue et al. (2021). The performance of
such multilingual models on low-resource languages and unseen languages are
known to be relatively poor Pfeiffer et al. (2020); Wang et al. (2021).

In evaluating the performance of open-domain chatbots, it has been shown that
automatic metrics, like the BLEU score, can be very poor but they are still
used in some cases Lundell Vinkler and Yu (2020). Conversation turns per
session is another metric of interest Zhou et al. (2020). Perplexity is also
widely used for intrinsic evaluation of language models and its theoretical
minimum, which is its best value, is 1 Adiwardana et al. (2020). Probably the
best evaluation is done by human evaluators (or annotators) but this can be
subjective. The judgment of human evaluators is seen as very important,
especially since humans are usually the end-users of such systems Zhang et al.
(2020).

##  7 Conclusion

In this study, we presented the new high-quality AfriWOZ dataset for dialogue
modelling for 6 African languages. We also demonstrated the cross-lingual
transferability hypothesis for the 6 African languages and observe that it is
possible to different degrees of success. The English pretrained DialoGPT
model resulted in the best perplexity scores across the languages. AfriWOZ may
be extended to the total 143,000 dialogue turns in the MultiWOZ to achieve
better performance in modelling. Better performance may also be achieved if
the tokenizers are optimized on the target languages by training from scratch
or finetuning, as this will allow more native tokens to be represented. It may
be worthwhile to construct a transferability index/matrix for various
languages. This will indicate the amount of benefit that may be harnessed from
utilising such properties in different downstream tasks.

##  8 Limitations

The data used to finetune the models are relatively small and cover only a few
domains, hence, the generation capabilities of the models are limited.
Furthermore, though we made effort to use carefully curated dialogue data and
avoid personally identifiable information (PII), the potential to generate
offensive output is still present, as the pretrained models retain biases in
the pretraining data.

## Acknowledgment

Our profound appreciation goes to Angela Fan, Clemencia Siro and Wilhelmina
Onyothi Nekoto for their contributions to this work. Also, we wish to thank
the many evaluators who judged the conversation transcripts for the various
languages.

## References

  * Adelani et al. (2021) David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Chester Lignos, Constantine Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Jesujoba Alabi, Seid Muhie Yimam, Tajuddeen Rabiu Gwadabe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, Verrah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chukwuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akinode, Deborah Nabagereka, Maurice Katusiime, Ayodele Awokoya, Mouhamadane MBOUP, Dibora Gebreyohannes, Henok Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulaye Faye, Blessing Sibanda, Orevaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thierno Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Salomey Osei. 2021.  MasakhaNER: Named Entity Recognition for African Languages.  _Transactions of the Association for Computational Linguistics_ , 9:1116–1131.
  * Adewumi et al. (2022) Tosin Adewumi, Rickard Brännvall, Nosheen Abid, Maryam Pahlavan, Sana Sabah Sabry, Foteini Liwicki, and Marcus Liwicki. 2022.  Småprat: Dialogpt for natural language generation of swedish dialogue by transfer learning.  In _5th Northern Lights Deep Learning Workshop, Tromsø, Norway_ , volume 3. Septentrio Academic Publishing.
  * Adewumi et al. (2019) Tosin P Adewumi, Foteini Liwicki, and Marcus Liwicki. 2019.  Conversational systems in machine learning from the point of view of the philosophy of science—using alime chat and related studies.  _Philosophies_ , 4(3):41.
  * Adewumi et al. (2020) Tosin P Adewumi, Foteini Liwicki, and Marcus Liwicki. 2020.  The challenge of diacritics in yoruba embeddings.  _arXiv preprint arXiv:2011.07605_.
  * Adiwardana et al. (2020) Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020\.  Towards a human-like open-domain chatbot.  _arXiv preprint arXiv:2001.09977_.
  * Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020.  On the cross-lingual transferability of monolingual representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4623–4637, Online. Association for Computational Linguistics.
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  In _International Conference on Learning Representations, ICLR 2015_.
  * Bender and Friedman (2018) Emily M Bender and Batya Friedman. 2018.  Data statements for natural language processing: Toward mitigating system bias and enabling better science.  _Transactions of the Association for Computational Linguistics_ , 6:587–604.
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5016–5026, Brussels, Belgium. Association for Computational Linguistics.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  Cross-lingual language model pretraining.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Devlin et al. (2018a) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018a.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Devlin et al. (2018b) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018b.  Multilingual bert.
  * Dinan et al. (2020) Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston. 2020.  Queens are powerful too: Mitigating gender bias in dialogue generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8173–8188, Online. Association for Computational Linguistics.
  * Dinan et al. (2019) Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019.  Build it break it fix it for dialogue safety: Robustness from adversarial human attack.  _arXiv preprint arXiv:1908.06083_.
  * Eric et al. (2020) Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, and Dilek Hakkani-Tur. 2020.  Multiwoz 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 422–428, Marseille, France. European Language Resources Association.
  * Gangal et al. (2021) Varun Gangal, Harsh Jhamtani, Eduard Hovy, and Taylor Berg-Kirkpatrick. 2021.  Improving automated evaluation of open domain dialog via diverse reference augmentation.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4079–4090, Online. Association for Computational Linguistics.
  * Gehrmann et al. (2021) Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D Dhole, et al. 2021.  The gem benchmark: Natural language generation, its evaluation and metrics.  _arXiv preprint arXiv:2102.01672_.
  * Gwet (2014) Kilem L Gwet. 2014.  _Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters_.  Advanced Analytics, LLC.
  * Hashimoto et al. (2019) Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. 2019.  Unifying human and statistical evaluation for natural language generation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1689–1701, Minneapolis, Minnesota. Association for Computational Linguistics.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020.  Deberta: Decoding-enhanced bert with disentangled attention.  _arXiv preprint arXiv:2006.03654_.
  * Heine et al. (2000) Bernd Heine, Derek Nurse, et al. 2000.  _African languages: An introduction_.  Cambridge University Press.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Hu et al. (2018) Zhiting Hu, Haoran Shi, Bowen Tan, Wentao Wang, Zichao Yang, Tiancheng Zhao, Junxian He, Lianhui Qin, Di Wang, Xuezhe Ma, et al. 2018.  Texar: A modularized, versatile, and extensible toolkit for text generation.  _arXiv preprint arXiv:1809.00794_.
  * Ihemere (2006) Kelechukwu Uchechukwu Ihemere. 2006.  A basic description and analytic treatment of noun clauses in nigerian pidgin.  _Nordic journal of African studies_ , 15(3):296–313.
  * Javed et al. (2021) Saleha Javed, Tosin P Adewumi, Foteini Simistira Liwicki, and Marcus Liwicki. 2021\.  Understanding the role of objectivity in machine learning and research evaluation.  _Philosophies_ , 6(1):22.
  * Jurafsky and Martin (2020) D. Jurafsky and J.H. Martin. 2020.  _Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition_.  Dorling Kindersley Pvt, Limited.
  * Komeili et al. (2021) Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2021.  Internet-augmented dialogue generation.  _arXiv preprint arXiv:2107.07566_.
  * Landis and Koch (1977) J Richard Landis and Gary G Koch. 1977.  The measurement of observer agreement for categorical data.  _biometrics_ , pages 159–174.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lundell Vinkler and Yu (2020) Mikael Lundell Vinkler and Peilin Yu. 2020.  Conversational chatbots with memory-based question and answer generation.
  * Miller et al. (2017) Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017.  ParlAI: A dialog research software platform.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 79–84, Copenhagen, Denmark. Association for Computational Linguistics.
  * Nekoto et al. (2020) Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddeen Muhammad, Salomon Kabongo Kabenamualu, Salomey Osei, Freshia Sackey, Rubungo Andre Niyongabo, Ricky Macharm, Perez Ogayo, Orevaoghene Ahia, Musie Meressa Berhe, Mofetoluwa Adeyemi, Masabata Mokgesi-Selinga, Lawrence Okegbemi, Laura Martinus, Kolawole Tajudeen, Kevin Degila, Kelechi Ogueji, Kathleen Siminyu, Julia Kreutzer, Jason Webster, Jamiil Toure Ali, Jade Abbott, Iroro Orife, Ignatius Ezeani, Idris Abdulkadir Dangana, Herman Kamper, Hady Elsahar, Goodness Duru, Ghollah Kioko, Murhabazi Espoir, Elan van Biljon, Daniel Whitenack, Christopher Onyefuluchi, Chris Chinenye Emezue, Bonaventure F. P. Dossou, Blessing Sibanda, Blessing Bassey, Ayodele Olabiyi, Arshath Ramkilowan, Alp Öktem, Adewale Akinfaderin, and Abdallah Bashir. 2020.  Participatory research for low-resourced machine translation: A case study in African languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2144–2160, Online. Association for Computational Linguistics.
  * Ogueji et al. (2021) Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin. 2021.  Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages.  In _Proceedings of the 1st Workshop on Multilingual Representation Learning_ , pages 116–126, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Olabiyi and Mueller (2019) Oluwatobi Olabiyi and Erik T Mueller. 2019.  Multiturn dialogue response generation with autoregressive transformer models.  _arXiv preprint arXiv:1908.01841_.
  * Perez Almendros et al. (2020) Carla Perez Almendros, Luis Espinosa Anke, and Steven Schockaert. 2020.  Don’t patronize me! an annotated dataset with patronizing and condescending language towards vulnerable communities.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 5891–5902, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Pfeiffer et al. (2020) Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020.  Adapterhub: A framework for adapting transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020): Systems Demonstrations_ , pages 46–54, Online. Association for Computational Linguistics.
  * Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019.  How multilingual is multilingual BERT? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4996–5001, Florence, Italy. Association for Computational Linguistics.
  * Polomé (1967) Edgar C Polomé. 1967.  Swahili language handbook.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Reid et al. (2021) Machel Reid, Junjie Hu, Graham Neubig, and Yutaka Matsuo. 2021.  AfroMT: Pretraining strategies and reproducible benchmarks for translation of 8 African languages.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1306–1320, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Roller et al. (2021) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021.  Recipes for building an open-domain chatbot.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 300–325, Online. Association for Computational Linguistics.
  * Rönnqvist et al. (2019) Samuel Rönnqvist, Jenna Kanerva, Tapio Salakoski, and Filip Ginter. 2019.  Is multilingual bert fluent in language generation?  _arXiv preprint arXiv:1910.03806_.
  * Sennrich (2012) Rico Sennrich. 2012.  Perplexity minimization for translation model domain adaptation in statistical machine translation.  Association For Computational Linguistics.
  * Sim and Wright (2005) Julius Sim and Chris C Wright. 2005.  The kappa statistic in reliability studies: use, interpretation, and sample size requirements.  _Physical therapy_ , 85(3):257–268.
  * Smith et al. (2020) Eric Michael Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. 2020.  Can you put it all together: Evaluating conversational agents’ ability to blend skills.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2021–2030, Online. Association for Computational Linguistics.
  * Solaiman and Dennison (2021) Irene Solaiman and Christy Dennison. 2021.  Process for adapting language models to society (palms) with values-targeted datasets.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Virtanen et al. (2019) Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019.  Multilingual is not enough: Bert for finnish.  _arXiv preprint arXiv:1912.07076_.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _arXiv preprint arXiv:1905.00537_.
  * Wang et al. (2021) Zirui Wang, Adams Wei Yu, Orhan Firat, and Yuan Cao. 2021.  Towards zero-label language learning.  _arXiv preprint arXiv:2109.09193_.
  * Weizenbaum (1969) J Weizenbaum. 1969.  A computer program for the study of natural language.  _Fonte: Stanford: http://web. stanford. edu/class/linguist238/p36_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xu et al. (2021) Jing Xu, Arthur Szlam, and Jason Weston. 2021.  Beyond goldfish memory: Long-term open-domain conversation.  _arXiv preprint arXiv:2107.07567_.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Zhang et al. (2020) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020.  Dialogpt: Large-scale generative pre-training for conversational response generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 270–278.
  * Zhou et al. (2020) Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum. 2020.  The design and implementation of xiaoice, an empathetic social chatbot.  _Computational Linguistics_ , 46(1):53–93.

Data statement for the AfriWOZ dataset for open-domain dialogue & other NLP
models.
---
|  Details
Curation rationale |  Due to the unavailability of dialogue data for low-resource African languages, this dataset was created.
Dataset language |  Swahili, Wolof, Hausa, Nigerian Pidgin English, Kinyarwanda & Yorùbá
|  Demographics of contributors
No of contributors |  19
Age |  -
Gender |  Male & Female
Language |  L1
|  Demographics of annotators
No of annotators |  Not applicable
|  Data characteristics
Total samples |  1,500 turns per language
Total natural languages |  6 (Swahili, Wolof, Hausa, Nigerian Pidgin English, Kinyarwanda & Yorùbá)
Training set turns per language |  1,000
Validation set turns per language |  250
Test set turns per language |  250
Domains covered |  hotel, restaurant, taxi and booking.
Base data |  MultiWOZ and 2 blogs for Yorùbá only.
|  Others
IAA |  CUS 33.3% - 100%
Licence |  CC-BY 4.0.
Table 8:

## Acronyms

ASR

    automatic speech recognition
BART

    Bidirectional and Auto-Regressive Transformers
BERT

    Bidirectional Encoder Representations from Transformers
BLEU

    bilingual evaluation understudy
BPE

    byte-pair encoding
CUS

    credibility unanimous score
DialoGPT

    dialogue generative pre-trained transformer
IAA

    inter-annotator agreement
LSTM

    Long Short Term Memory Network
MT

    Machine Translation
MultiWOZ

    Multi-Domain Wizard-of-Oz
NLG

    Natural Language Generation
NLP

    Natural Language Processing
NLU

    Natural Language Understanding
PII

    personally identifiable information
SoTA

    state-of-the-art
T5

    Text-to-Text Transfer Transformer
XLM-R

    Cross-Lingual Model-RoBERTa