#  Aya Dataset: An Open-Access Collection
for Multilingual Instruction Tuning

name=Shivalika Singh  affiliation=1  name=Freddie Vargus  affiliation=1
name=Daniel D’souza  affiliation=1  name=Börje F. Karlsson  affiliation=2
name=Abinaya Mahendiran  affiliation=1  name=Wei-Yin Ko  affiliation=3
name=Herumb Shandilya  affiliation=1  name=Jay Patel  affiliation=4
name=Deividas Mataciunas  affiliation=1  name=Laura O’Mahony  affiliation=5
name=Mike Zhang  affiliation=6  name=Ramith Hettiarachchi  affiliation=7
name=Joseph Wilson  affiliation=8  name=Marina Machado  affiliation=3
name=Luisa Souza Moura  affiliation=3  name=Dominik Krzemiński  affiliation=1
name=Hakimeh Fadaei  affiliation=1  name=Irem Ergün  affiliation=3
name=Ifeoma Okoh  affiliation=1  name=Aisha Alaagib  affiliation=1  name=Oshan
Mudannayake  affiliation=1  name=Zaid Alyafeai  affiliation=9  name=Vu Minh
Chien  affiliation=1  name=Sebastian Ruder  affiliation=3  name=Surya
Guthikonda  affiliation=1  name=Emad A. Alghamdi  affiliation=10
name=Sebastian Gehrmann  affiliation=11  name=Niklas Muennighoff
affiliation=1  name=Max Bartolo  affiliation=3  name=Julia Kreutzer
affiliation=12  name=Ahmet Üstün  affiliation=12  name=Marzieh Fadaee
affiliation=12  name=Sara Hooker  affiliation=12

###### Abstract

Datasets are foundational to many breakthroughs in modern artificial
intelligence. Many recent achievements in the space of natural language
processing (NLP) can be attributed to the fine-tuning of pre-trained models on
a diverse set of tasks that enables a large language model (LLM) to respond to
instructions. Instruction fine-tuning (IFT) requires specifically constructed
and annotated datasets. However, existing datasets are almost all in the
English language. In this work, our primary goal is to bridge the language gap
by building a human-curated instruction-following dataset spanning 65
languages. We worked with fluent speakers of languages from around the world
to collect natural instances of instructions and completions. Furthermore, we
create the most extensive multilingual collection to date, comprising 513
million instances through templating and translating existing datasets across
114 languages. In total, we contribute four key resources: we develop and
open-source the Annotation Platform, the Dataset, the Collection, and the
Evaluation Suite. The initiative also serves as a valuable case study in
participatory research, involving collaborators from 119 countries. We see
this as a valuable framework for future research collaborations that aim to
bridge gaps in resources.

Cohere For AI Community Beijing Academy of Artificial Intelligence Cohere
Binghamton University University of Limerick IT University of Copenhagen MIT
University of Toronto King Fahd University of Petroleum and Minerals King
Abdulaziz University, ASAS.AI Bloomberg LP Cohere For AI [*]Shivalika Singh
<EMAIL_ADDRESS>Marzieh Fadaee<EMAIL_ADDRESS>Sara Hooker
<EMAIL_ADDRESS>
##  1 Introduction

Datasets are static representations of the world, far from the rich, ever-
evolving environment we navigate as humans. Yet, these frozen snapshots in
time are the foundation upon which progress in AI has been built. Many recent
breakthroughs in language modeling can be attributed to fine-tuning pre-
trained models on a diverse set of tasks that enable a Large Language Model
(LLM) to follow instructions [McCann et al., 2018; Sanh et al., 2022; Wei et
al., 2022a; Muennighoff et al., 2023c; Longpre et al., 2023a]. Instruction
fine-tuning (IFT) leverages the precept that Natural Language Processing (NLP)
tasks can be described via natural language instructions, such as “What were
the reviews like for the Barbie movie?" or “Write a recipe from the following
list of ingredients." This process requires _prompts_ to be paired with
expected _completions_ [Ziegler et al., 2020; Ouyang et al., 2022] aiming to
capture the variety of ways an LLM can be used in downstream tasks. Yet, the
very act of curating data imparts a viewpoint about what distributions we want
our model to represent and what is forgotten. So, what do these widely used
datasets tell us about the assumptions underlying these breakthroughs?

Figure 1: Aya Dataset, Aya Collection & Aya Evaluation Suite. On the left, we
show examples of contributions in the Aya Dataset. These are original human-
curated prompt-completion pairs written by fluent speakers of 65 languages. On
the right, we have the Collection, an aggregation of 44 monolingual and
multilingual templated instruction datasets and 19 translated datasets ranging
over 114 languages and three main tasks: Text Classification, Natural Language
Generation, and Question Answering. The bottom block showcases the Evaluation
Suite for multilingual open-ended generation. This collection consists of
original annotation and post-edits of translations covering several languages,
and translation of high-quality and universal prompts into 101 languages. We
indicate the number of languages in a dataset with the value in the blue ovals
in the figure. (Translated datasets have been visually merged due to space
constraints).

More than 7,000 languages111https://www.ethnologue.com/ are spoken around the
world today, with a considerable number facing the challenges of being low-
resourced, under-represented, or disappearing [Maxwell & Hughes, 2006; Simons,
2019; Moran & Chiarcos, 2020; Secretariat, 2022; Gao & Liu, 2023; Ilhomovna &
Yuldasheva, 2023; Marivate et al., 2020]. In contrast, the most widely used
datasets and breakthroughs in NLP have coalesced around a few data-rich
languages [Longpre et al., 2023b; Taori et al., 2023; Chung et al., 2022; Fan
et al., 2021; Dodge et al., 2021; Lucy et al., 2024]. IFT datasets are no
exception; the creation of these datasets has almost entirely focused on
English. Furthermore, the vast majority of the creators of these works
originate from a few countries [Longpre et al., 2023b; Zhang et al., 2022].

The factors underlying the construction of the datasets impact how models
perform for users around the world. Models perform better on the distribution
they are trained to mimic [Kunchukuttan et al., 2021]. This often introduces
known biases towards languages [Schwartz et al., 2022; Kotek et al., 2023;
Khandelwal et al., 2023; Vashishtha et al., 2023; Khondaker et al., 2023] and
dialects [Jørgensen et al., 2015; Blodgett et al., 2016; Zampieri et al.,
2017; Sun et al., 2023] not included during training and introduces critical
security flaws [Yong et al., 2023a; Nasr et al., 2023; Li et al., 2023b; Lukas
et al., 2023; Deng et al., 2023].

Datasets aren’t simply raw materials that fuel breakthroughs but also make the
poor poorer and the rich richer [Held et al., 2023; Durmus et al., 2023;
Robinson et al., 2023]. Disparities in the access to technological resources
predates the advent of LLMs [Garrette et al., 2013]. However, as LLMs become
more sophisticated and widely available, non-English languages will remain
under-represented and will likely become more so. The imbalance between
languages has created a growing divide in the cost of using this technology as
marginalized languages require more tokens and incur higher latency for
generations [Ji et al., 2023b; Cui et al., 2023], consigning speakers of low-
performing languages to lower quality technology [Held et al., 2023; Durmus et
al., 2023; Nicholas & Bhatia, 2023; Ojo et al., 2023]. Often, speakers of low-
resource languages do not have the resources to improve NLP technology for
their language, facing a low-resource double bind with limited access to both
compute and data [Ahia et al., 2021].

Dataset |  #Instances |  #Langs |  % English |  Generation method |  Permissive license
---|---|---|---|---|---
Llama2 IFT data [Touvron et al., 2023] |  NA |  27 |  90% |  Human-annotations SFT datasets |  ✗
Alpaca [Taori et al., 2023] |  52K |  1 |  100% |  Synthetic data generation IFT datasets |  $\approx$
P3 [Sanh et al., 2022] |  12M |  1 |  100% |  Template generation given applied to English datasets |  ✓
Flan 2022 [Longpre et al., 2023a] |  15M |  60 |  100% |  Template generation applied to English datasets |  ✓
xP3 [Muennighoff et al., 2023c] |  81M |  46 |  39% |  Template generation applied to English datasets |  ✓
Sweinstruct [Holmström & Doostmohammadi, 2023] |  68K |  1 |  0% |  Machine translation English IFT datasets |  $\approx$
Okapi [Dac Lai et al., 2023] |  158K |  26 |  45% |  Machine translation English IFT datasets |  ✓
Bactrian-X [Li et al., 2023a] |  3.4M |  52 |  2% |  Machine translation + synthetic data generation |  $\approx$
Dataset |  204K |  65 |  2% |  Original IFT Human-annotations |  ✓
Collection |  513M |  114 |  3.5% |  Template Generation and translating existing datasets |  ✓
Table 1: Comparison of different instruction-tuning datasets. ✓represents
permissive licenses that allow commercial use while $\approx$ represents
restrictive licenses that do not allow commercial use. ✗represents non
availability of license.

In this work, our goal is to reduce this linguistic inequality. Efforts that
aim to improve multilingual performance have often focused on improving data
coverage [Chen et al., 2023b]. However, most of the limited effort to date has
focused on multilingual pre-training [Scao et al., 2022a; Wei et al., 2023;
Lample & Conneau, 2019] with even less work centered on imparting instruction
following abilities. Approaches that have tried to translate English instruct-
style datasets into other languages often suffer from translation biases
[Vanmassenhove et al., 2021; Hartung et al., 2023; Savoldi et al., 2021;
Muennighoff et al., 2023c] or fail to reflect cultural context appropriately
[Wang et al., 2022a; Ji et al., 2023a; Pudjiati et al., 2022]. Automatic
curation of multilingual datasets is a logical —and sometimes necessary—
approach but often suffers from noise and biases. This makes it difficult to
validate the quality of the created datasets [Kreutzer et al., 2022; Luccioni
& Viviano, 2021; Ferrara, 2023; Caswell et al., 2020] or requires the curation
of manual templates which often result in low instruction and completion
diversity [Muennighoff et al., 2023c] critical for model performance [Naik et
al., 2023; Chung et al., 2023; Li et al., 2023e; Lahoti et al., 2023].

In contrast, a key aspect of our work focused on harder-to-obtain human-
curated data from fluent speakers of a language. This curation process has
received far less attention due to lack of access to fluent speakers,
especially in low-resource languages [Joshi et al., 2019]. We chose to close
this gap by conducting a year-long participatory research initiative that
involved working with fluent speakers of languages from around the world to
collect human-curated instances of instructions and completions. By leveraging
best practices from open-source and crowd-sourced science projects [Franzoni &
Sauermann, 2014; Beck et al., 2022; Lenart-Gansiniec et al., 2023], we built a
simple and intuitive user interface, the Annotation Platform222This platform
is accessible at: https://aya.for.ai (UI) which served as the central platform
for contributors to join the 333The word has its origins in the Akan (Twi)
language and is translated as “fern” in English [Willis, 1998]. 444represents
endurance, resourcefulness, and defiance – like a fern growing in barren
conditions. project. In total, we had 2,997 collaborators spread across 119
countries around the world. Their collective efforts resulted in the dataset
which is the largest human-curated multilingual instruction-finetuned dataset
to date, containing 204,114 high-quality annotations in 65 languages.

Additionally, we release and transform 44 pre-existing datasets into sets of
instruction-completion pairs by crafting diverse templates manually, relying
on fluent speakers for each language. We further expand this collection by
translating datasets from English into 101 languages. We refer to this
expanded collection of 513 million instances covering 114 languages in total
as the collection, which to date, is the most extensive collection of
multilingual instruction-finetuning (IFT) data.

Overall, contributes four key resources: Annotation Platform (UI); Dataset;
Collection, and Evaluation Suite. Figure 1 shows a visual representation of
the Dataset and Collection. Below, we briefly describe these core
contributions:

  1. 1.

Annotation Platform(UI): We built a robust annotation tool to facilitate the
collection of high-quality multilingual data in an instruction-style format
supporting 182 languages, including dialects. Over eight months, we had a
total of 2,997 registered users spanning 119 countries and 134 languages,
including dialects.

  2. 2.

Dataset: We created the largest human-annotated multilingual instruction
finetuning dataset to date, consisting of over 204K instances that cover 65
languages. We include a data card [Pushkarna et al., 2022] for the Dataset in
Appendix J.

  3. 3.

Collection: We collected instruction-style templates from fluent speakers and
applied them to a curated list of 44 datasets, including tasks such as Text
Classification, Text Generation, Machine Translation, Paraphrasing, and Open-
domain Question Answering. Some of these datasets also include equivalent
multilingual versions produced through translation. We release 513M instances
that cover 114 languages. These contributions are made available as an open-
source collection. We include a data card for the Collection in Appendix J.

  4. 4.

Evaluation Suite: We curate and release a diverse evaluation suite for
multilingual open-ended generation quality. It consists of 250 human-written
prompts for each of 7 languages, 200 automatically translated but human-
selected prompts for 101 languages (114 dialects), and human-edited prompts of
the latter for 6 languages, and the English originals. The first set
represents culturally-grounded and original prompts, while the translated and
post-edited prompts are sourced from English Dolly [Conover et al., 2023] and
selected for their cross-cultural relevance. We include a data card for the
Collection in Appendix J.

By fully open sourcing the Aya Dataset, Aya Collection and Evaluation Suite
with a permissive Apache 2.0
License555https://www.apache.org/licenses/LICENSE-2.0 as well as the code for
our annotation platform, we hope to empower researchers and practitioners to
further advance multilingual models and applications. All datasets are
accessible for
download.666https://hf.co/datasets/CohereForAI/aya_dataset777https://hf.co/datasets/CohereForAI/aya_collection888https://hf.co/datasets/CohereForAI/aya_evaluation_suite

Paper Organization  Section 2 discusses the design and development of the
Annotation Platform, as well as the preparation of the Dataset, and Section 3
presents a detailed analysis of the Dataset. Section 4 and Section 5 contain
discussion and analysis of the Collection. Section 6 describes the details of
the evaluation suite curated in this project. In Section 7, we describe our
approach to participatory research. In Section 8, we review the existing
literature, and in Section 9 we discuss the limitations of our work. Section
10 concludes the paper.

##  2 Annotation Platform & Dataset

###  2.1 Annotation Platform

The goal of the project is to facilitate annotations to a crowd-sourced
dataset by individuals fluent in different languages. Inputs from fluent
speakers of each language ensure that the dataset is more likely to be
organic, fluent, and representative of the speakers’ cultures. Including
fluent and native speakers from various regions poses significant logistical
challenges involving meticulous data selection, quality control measures, and
custom annotation tools. We developed the Annotation Platform to streamline
the data collection process worldwide, accommodating a large number of
decentralized contributors across multiple languages.

User Interfaces (UIs) play a pivotal role in the context of NLP data
collection, serving as the primary point of interaction between human
annotators and the data collection process. The Annotation
Platform999https://aya.for.ai/ had to accommodate users in 119 countries
collecting data across 134 languages. We designed the platform with a few key
principles in mind, such as accessibility and ease of use for users who were
unfamiliar with AI and machine learning. As part of our contribution, we fully
open-source the code for our UI101010https://github.com/for-ai/aya-
annotations-ui.

Accessibility  As users worldwide use different devices and operating systems,
we decided to support both mobile and desktop interfaces [Muhammad et al.,
2023]. Approximately 54% of users accessed UI via desktop browsers while 46%
utilized mobile browsers. We attribute the high fraction of mobile users to
the skew towards mobile users in the Global South [Avle et al., 2018]. We
supported Single Sign-On (SSO) capabilities to enable seamless tracking of
user profiles and reward users with points for contributing data across
multiple sessions. We initially only supported Discord sign-on but discovered
that Discord is inaccessible or not widely used in certain countries. Also,
the necessity of a platform-specific account created an obstacle to user
engagement with . This prompted us to add Google sign-on as an alternative
option.

Figure 2: Geographical distribution of the users registered on the platform.

Languages Supported  project contributors could select the languages they are
proficient in when signing up using the UI. They could then make annotations
in the language(s) they selected. Given the sheer number of languages we could
collect annotations for, we chose to prioritize annotation support for the 101
languages available in the mT5 model [Xue et al., 2021]. We note that
ultimately, some of these languages didn’t receive enough contributions to
include them in the final dataset. Conversely, we received substantial
contributions from languages not initially part of the original list, like
Wolof, leading to their inclusion; the final Dataset covers 65 languages.
Table LABEL:tab:language_codes provides details of these languages.

Contributors  We aimed to include individuals from diverse backgrounds—not
limited to AI experts—enabling anyone proficient in a language to contribute.
Our pool of contributors ultimately reflects this inclusive approach. During
the registration process, we request specific demographic details from each UI
user such as country of residence, languages of fluent communication, gender,
age range, and familiar dialects. We display the onboarding form in Figure 17
in the Appendix. The community of contributors includes 2,997 registered users
across 134 languages.

Figure 3: Left: Distribution of registered users on the UI by age using
specified values. Right: Distribution of registered users on the UI by gender
using specified values

Demographics Figure 3 illustrates the demographics of registered UI users by
age and gender. Regarding the age profiles of users, more than two-thirds were
aged between 18 and 35. Approximately 68.1% of users identified themselves as
male and 28.5% as female. Overall, 6.6% of users self-reported dialects.
Within this group, 75% specified one dialect, 20% specific two dialects, and
the remaining 5% specified three or more dialects, with a maximum of six.

Figure 4: Distribution of total contributions across different regions.

During the development of , registered users were geographically distributed
across 119 countries based on their residence. Certain countries like
Afghanistan, Bulgaria, Kuwait, and Tajikistan had just one registered user.
Figure 2 displays this global distribution, highlighting India with the
highest number of registered users (346 out of 2,997).

Geographic-Based Contribution Assessment  We grouped the languages by the
regions in which they either originate or are widely spoken. The language
statistics by region for the original 101 languages we wanted to cover are as
follows: 14 languages in Africa, 41 languages in Asia, 42 languages in Europe,
and 4 languages in Latin America (See Appendix C.1 for more details and some
exceptions of the distribution). As seen in Figure 4, more than half of all
contributions for the project came from Asia with 58.8%, followed by the
African region with 27.4%. Europe, Latin America, and other regions account
for the remaining 13.8% of the contributions.

We observe a large skew in terms of regional contributions, which deserves
further research to understand why certain networks of contributors remained
motivated for the entire project. These disparities in participation may be
due to opportunity cost in time [Gerosa et al., 2021; Wu et al., 2007],
cultural beliefs around sharing data [Huang et al., 2023], or the belief that
the language in question is not well served by the current technology
[Nicholas & Bhatia, 2023].

Acknowledgement of contributions  Recognition and transparency were maintained
throughout the project through the use of a leaderboard111111The Leaderboard
is accessible at: https://aya.for.ai/leaderboard/ to acknowledge
contributions. We implemented a scoring system where contributors earned a
maximum of three points for each re-annotation, with one point awarded for
rating the prompt and completion, one point for editing the prompt, and one
point for editing the completion. Each original annotation was awarded with
two points. We describe the different annotation tasks in the UI in detail in
Section 2.2.

The Leaderboard is organized to display daily, weekly, and cumulative scores,
providing a comprehensive overview of user contributions. The users have the
flexibility to filter scores based on specific languages, allowing for a sense
of community amongst contributors of a particular language. This design aimed
to boost contributors’ motivation to provide high-quality inputs for their
chosen languages. Figure 19 shows an example of the leaderboard. We discuss
further details on collaborating with the community in Section 7.

###  2.2 Annotation tasks

On the Annotation Platform, contributors were able to contribute to three
different tasks, following the find-fix-verify paradigm [Bernstein et al.,
2015]: Writing new examples from scratch (original annotations), editing
existing examples to improve the quality and comprehensiveness (re-
annotations), and giving feedback on the quality of existing contributions
(annotation feedback). We describe each briefly below:

####  2.2.1 Original Annotations

This task facilitates the inclusion of human-generated organic content by
allowing annotators to submit original prompt-completion pairs in their
language. Existing multilingual models have been shown to produce generations
influenced by Western culture [Yuan et al., 2021; Naous et al., 2023; Lee et
al., 2023] reflecting the underlying representation bias [Mehrabi et al.,
2021] of their training datasets. This task aims to encourage annotators to
submit fresh samples that are representative of their language, culture,
literature, history, and region. The guidelines for contributors is available
in Appendix B.3.

(a) Example of an original annotation contribution.

(b) Example of a re-annotation contribution.

Figure 5: Demonstration of a sample original annotation and re-annotation
contribution, in English. (a) exemplifies an original contribution input by an
annotator from scratch. (b) shows a sample of re-annotation. Here, the
annotator may have improved upon either a prompt and completion pair pulled
from the database or a prompt and completion originally created by another
contributor.

####  2.2.2 Re-Annotations

The purpose of this task is to facilitate the re-annotation or editing of
prompt and completion pairs. The decision to add a re-annotation task partly
stems from the need to help annotators understand the expected format of
instruction-style datasets and to convey the variety of tasks in existing
datasets, including question answering [Saad-Falcon et al., 2023; Arefeen et
al., 2023], summarization [Stiennon et al., 2020; Wu et al., 2021],
paraphrasing [Witteveen & Andrews, 2019; Reimers & Gurevych, 2019], and
translation [NLLB-Team et al., 2022; Barrault et al., 2023]. Editing examples
from existing datasets not only helped familiarize annotators with the
expected format but also allowed for human evaluation and rating of existing
widely used instruction-style datasets.

In total, we collected datasets from 19 public data sources and translated
them into 114 available languages, including dialects using the NLLB 3.3B
parameter machine translation model [NLLB-Team et al., 2022]. From each
collection, we randomly chose 100 examples (per dataset and per language),
creating our dataset for annotation, after which we had 1M translated prompt-
completion pairs initially populated in the UI as re-annotation tasks. These
translated pairs served as a starting point for prompts and completions which
annotators could improve. We release the raw translations as part of the
Collection, provide more details about the provenance of the translated
datasets, and how they were selected in Section 4.2.

In addition to translated examples, there are other available data sources
suitable for re-annotation: original pairs, pre-existing instruction-style
datasets (e.g., xP3), and the transformation of datasets into an instruction-
style format, i.e., templated datasets. By re-annotating examples from
different sources, we simultaneously enhance the quality of individual
examples while obtaining a signal on the overall quality of the dataset in a
specific language. A demonstration of a re-annotation, where an annotator
strengthens a given prompt/completion, is shown in Figure 5(b).

####  2.2.3 Annotation Feedback

Data quality is critical to ensure that a model can represent a language well.
Learning from noisy, low-quality datasets harms the overall model performance
and the relatively high cost of encoding these noisy examples is a misuse of
capacity [Hsueh et al., 2009; Dodge et al., 2021; Luccioni & Viviano, 2021;
Kreutzer et al., 2022]. Prior work has shown that improvements to quality
through data pruning or selection can have an significant impact on the
downstream performance of a model [Longpre et al., 2023c; Marion et al., 2023;
Boubdir et al., 2023; Yang et al., 2023]. In particular, for instruction-
tuning datasets, a small subset of higher-quality instructions can greatly
outperform a larger volume of lower-quality instructions [AlShikh et al.,
2023; Zhou et al., 2023; Chen et al., 2023a]. Given these findings, ensuring
high quality contributions is of paramount importance. Ensuring consistent
quality is particularly challenging in an open science initiative with a large
number of contributors. We face two key challenges:

Changes in the Annotator Pool. During the year-long project, annotators joined
and left the project at different points depending on their interests and
availability. As a result, the window of contribution for each annotator was
different. Only a small fraction of annotators participated for the entire
duration of the year-long project. Annotators were active for an average of
1.3 sessions. Figure 6 presents a histogram depicting the distribution of user
engagement based on the number of days they actively contributed. On average,
annotators spent five days contributing to the project. Annotators tended to
be highly active shortly after joining, but their activity declined over time.
There was a subgroup of annotators who maintained consistent activity over
extended periods.

Figure 6: The distribution of annotators’ engagement based on the number of
days they actively contributed in UI

Varying level of experience with AI. An important goal of this project was to
have a diverse pool of annotators and we thus did not limit the selection
criteria to working knowledge of language models or AI in general. As a
result, there were different levels of understanding amongst the annotators
what was meant by a prompt and completion. For example, we found at least one
contributor with 3,684 contributions to three languages (English, Somali,
Standard Arabic) who failed to structure their submissions as a prompt with a
question. Instead, the contributor used an extract of text as the prompt and
its continuation in the completion. Prefacing such prompts with an instruction
such as “Complete the following partial extract of text:" would have been a
more suitable format. While we routinely provided examples to contributors,
there was a clear need for a systematic way to review and measure the quality
of submissions.

Validating the quality of contributions We follow a peer-review approach where
each annotator acts as a reviewer for the other annotators working on the same
language. These reviews form the basis for a quality score which is displayed
on the leaderboard in the UI. The quality score for an annotator is calculated
by averaging the combined average ratings of their examples provided by other
annotators who serve as reviewers. We provide more details about how
annotations are reviewed in the Appendix Section B.1. All three tasks in the
UI are connected in a sequential pipeline where submissions from “Original
Annotations” are reviewed in the “Re-Annotations” task, and the re-annotations
are further reviewed as part of the “Annotation Feedback” task. This
systematic approach allows for a robust evaluation and enhancement of the
collected data.

###  2.3 Criteria for Inclusion in Dataset

The Dataset includes all original annotations and a subset of all re-
annotations. We only release re-annotations if there is a difference between
the original and the edited version. To determine this subset, we compute the
sum of edit distances d (Levenshtein distance [Levenshtein et al., 1966])
between the original and re-annotated prompts and completions on the character
level and use an acceptance threshold of ($d\geq 5$). This ensures that we do
not release duplicates of existing data.

Only languages with at least 50 contributions were included in the final
release of Dataset. This threshold was picked as it represents a balance
between achieving a reasonable level of data quality and considering the
practical limitations of human resources for some languages. The goal is to
include as many languages as possible without lowering the overall quality of
the dataset. Table LABEL:tab:language_codes lists details of the languages
included in the Dataset.

|  | Count
---|---|---
Original Annotations |  | 138,844
Re-Annotations | xP3 datasets | 2859
Translated datasets | 7757
Templated datasets | 11013
| Original Annotations | 43641
Dataset Total |  | 204,114
Table 2: Dataset Statistics (number of pairs of prompts and completions
obtained through various annotation tasks).

##  3 Analysis of Dataset

###  3.1 Statistics

The Dataset contains a total of 204,114 instances collected via the Annotation
Platform. Table 2 provides the breakdown of original annotations and re-
annotations in the final dataset. The dataset covers 65 languages: 22 high-
resource, 12 mid-resource, and 31 low-resource languages (See Appendix E for
more details on our language mappings).

###  3.2 Length of Dataset

One objective of this project was to collect fluid original human prompts and
completions. Table 3 provides examples of prompts and completions from the
Dataset. During the data collection process, annotators were provided with
examples and guidelines but were also trusted to explore their own creativity
and cultural background to come up with new examples. As a result, it is
meaningful to understand differences in aggregate statistics like length
across datasets, language type and relationship with perceived quality.

Figure 7: Average Completion Length before and after re-annotation. Here (*)
indicates the subset of all dataset categories (xP3, translated, templated,
and original annotations) that were included in the Dataset after re-
annotation. Re-annotation improves average completion length across all
datasets.  Figure 8: Average prompt and completion length of instances in the
Dataset across different language categories (high (HR), mid (MR) and low (LR)
resource languages, see Table LABEL:tab:language_codes).

Impact of Re-Annotation  When editing existing instances, we instructed the
annotators to prioritize enhancing both the quality and richness of the
prompts and completions. The average length of completions before and after
edits are shown in Figure 7. We observe that across all data sources, the
average length of completion increased after editing. On average, the length
of completions after edits is 25% longer than before edits. We observed the
largest increase for original annotations surfaced in the UI – which were 40%
longer on average than the original length.

Length difference across language groups  The average prompt and completion
length (number of characters) observed across these different language groups
is shown in Figure 8. A distinct contrast exists in completion lengths between
mid and low-resource languages when compared to high-resource languages. Long
completions and complete sentences are valuable in instruction-tuning
datasets, particularly when training multilingual models to generate content
in those languages.

Length vs. Perceived Data Quality  Although longer completions can be valuable
for training models to generate long and natural text, it does not necessarily
imply higher quality. Using annotators’ feedback in the UI, we further
investigate the impact of length on the perceived quality of the samples.
Figure 9 showcases this analysis. We observe a positive correlation between
how long the prompts and completions are and their resulting average approval
ratio. Specifically, when we plot combined prompt and completion length
against quality, we observe a correlation coefficient of 0.27. This finding
emphasizes the importance of using longer prompts and completions and
incorporating complete sentences to ensure a positive human experience when
engaging with such a model.

Length difference by language  Figure 25 in the Appendix illustrates the
statistics per language. We observe an array of patterns that differed across
languages. For instance in Japanese, completions are on average 31% shorter
than prompts. On the other end, for Urdu and Yoruba, completions are notably
long relative to prompts. On average, completions are 1258% and 2516% longer
than the corresponding prompts for Urdu and Yoruba, respectively. The average
completion length in Yoruba is 1591% longer than the average prompt length in
Japanese. Figure 26 provides the average length of the combination of prompts
and completions per language.

Comparison in Completion length Across Datasets The Dataset has considerably
longer completions on average when compared with other data collections as
shown in Figure 12. This is particularly noteworthy given that the Dataset is
human-curated. Given the presence of longer completions in the training data
for many low-resource languages, we expect that models trained on the Dataset
will generate longer and more natural responses.

---
Table 3: Examples of prompt and completions in the Dataset. Figure 9:
Relationship between Average Prompt and Completion Length in characters and
the Average Approval Rate of the example.

###  3.3 Annotator Skew

A feature of participatory research projects is the challenge of establishing
and maintaining a balanced number of annotations across groups of annotators.
In the project, the number of annotators per language varied due to numerous
factors. As a result, the distribution of annotators is not uniform across
languages. Moreover, within each language, there is a lack of consistent
contributions from all annotators. In this section, we examine the impact of
annotator skew on the resulting dataset.

Annotator Skew Across Languages. Annotators were encouraged to contribute to
any language in which they could comfortably read and write and were asked to
focus most of their efforts on languages other than English. Although a
significant number of participants registered for many languages, the
engagement level of annotators was not equal, which resulted in considerable
differences in the number of contributions across languages. Figure 10 (top)
provides an overview of the percentage of each language present in the final
compilation. The highest number of contributions is for Malagasy with 14,597
instances, and the lowest is 79 for Kurdish.

Figure 10: Top: Ratio of all annotations per language with respect to the
whole dataset. Bottom: Ratio of annotations done by the top-k most active
contributors ($k=1,...,5$). Languages annotations follow their respective ISO
codes from Table LABEL:tab:language_codes.

Annotator Skew Within a Language. The final contributions for each language in
the Dataset are not evenly distributed among annotators. The median number of
annotators per language is 15 (mean is 24.75) with one language having only a
single active annotator (Sindhi) and some having over 80 annotators (English
and Portuguese). Note that annotators made contributions at varying rates, and
there is no direct correlation between the number of annotators and the
ultimate count of language contributions. A limited pool of annotators for
some languages implies that most instances in that language originate from a
smaller group of individuals. Figure 10 (bottom) illustrates the proportion of
instances in a language originating from the most active annotators. We
observe a skewed pattern where for 12 languages, the 5 most active annotators
contributed all examples. There is an uneven distribution of contributions for
many languages because those languages had a smaller number of voluntary
annotators throughout the entire project despite rigorous outreach.
Additionally, we did not establish a specific quota for annotators to meet;
everyone contributed as they desired, resulting in varying levels of activity
among annotators.

The most extreme cases are Zulu and Sindhi, where one annotator in each
language volunteered for all contributions in Annotation and Re-annotation
tasks. Thus, in Figure 10 their top-1 contributor ratio is 1.0 and does not
change when moving to top-2 or further. The languages with the least skewed
distributions are Malagasy, Tamil, Nepali, Hindi, English and Portuguese. The
language English also had the highest number of unique annotators with 130
individuals out of which 95 annotators contributed to English as their second
language for annotation purposes. Given the uneven distribution of annotators
per language, it is important to acknowledge that individual annotator quality
has a disproportionate influence on some languages.

###  3.4 The impact of introducing the Score

As part of our collaborative annotation effort in , we emphasized the
importance of quality as well as long completions that contain clear responses
to the instructions specified in the prompt during the project. To encourage
high-quality examples from the annotators, we introduced the Score (Section
B.2) halfway through the project to focus on the quality, in addition to the
quantity, of contributions.

The Score encouraged participants to incorporate more edits during annotation,
with one specific guideline urging them to transform short answers into full
sentences or paragraphs. Figure 11(right) shows the change in the completion
lengths over time. We observe that after introduction of the score, there is a
marked uptick in the completion length of all submitted annotations.

(a) Volume of submissions over time

(b) Average completion length over time

Figure 11: The volume of original annotations and re-annotations increases
after the introduction of Score. We also observe a marked uptick in the
completion length of all submitted annotations with the introduction of the
Score.

##  4 Aya Collection

We introduce the Collection, a comprehensive, large corpus of datasets that
can be used by researchers around the world to train multilingual models. Our
goal is only to include datasets with permissive licensing for manipulation
and
redistribution.121212https://en.wikipedia.org/wiki/Permissive_software_license
Where possible, we report the license associated with each dataset within the
Collection.

The Collection consists of three different sources of data:

  1. 1.

Templated data: We collaborated with fluent speakers to create templates that
allowed for the automatic expansion of existing datasets into various
languages.

  2. 2.

Translated data: We translated a hand-selected subset of 19 datasets into 101
languages (114 dialects) using the NLLB 3.3B parameter machine translation
model [NLLB-Team et al., 2022]. The full list of datasets translated is listed
in Appendix Table LABEL:tab:aya_collection_translated.

  3. 3.

Dataset: We release the Dataset described in Section 3 as a subset of the
overall collection. This is the only dataset in the collection that is human-
annotated in its entirety.

Figure 12: Comparison of completion lengths between Dataset, Collection, and
xP3 (excluding the "code" split).

Dataset Selection Criteria  The templated and translated datasets in the
Collection were selectively hand picked to achieve a mix of different task
types. Our criteria prioritized datasets with high-quality natural and
complete sentences, suitable for creating pairs of prompts and completions.
Datasets that could potentially yield single-word answers were deliberately
excluded. Finally, to create a high-quality collection, we examined all
datasets and excluded those identified as unclean or noisy, primarily
attributable to their automatic creation processes.

###  4.1 Templating Existing Datasets

We explored the automatic expansion of existing datasets in various languages
with human-written _prompt templates_ , following previous works [Mishra et
al., 2022; Bach et al., 2022; Wei et al., 2022a; Wang et al., 2022e]. Unlike
prior works that still either use English prompts in a multilingual dataset or
rely on automatic translation to generate multilingual prompts, to our
knowledge, Collection is the first broad effort to involve fluent speakers in
creating prompts unique to their language to expand existing datasets for
instruction tuning.

We used the PromptSource framework [Bach et al., 2022] to template these
datasets. We asked community members to submit instructions and create
templates for datasets in the languages they were proficient in. Our process
includes: 1) Templating datasets with instructions in the same language as the
original dataset; 2) If the dataset is not in English, annotating instructions
in English. Our input prompts can be monolingual or code-mixed, depending on
whether we apply templates in the same language or in English to the dataset
of a particular language. Note that code-mixed input prompts here refer to a
structured mixing of English instructions with non-English monolingual data
[Lin et al., 2022], which is different from the typical sociolinguistic
definition of code-mixing (or code-switching) of languages in natural
conversational utterances [Winata et al., 2023a; Yong et al., 2023c; Doğruöz
et al., 2023; Srivastava & Singh, 2021].

We examined the suggested templates and subsequently converted each dataset
into an instruction-style format. We release these datasets under the
Collection. We list the details of all datasets we apply templates to in
Appendix Table LABEL:tab:list_of_templated_datasets_aya_collection.

###  4.2 Automatic Translation

Research has demonstrated that training models with translated data can yield
significant benefits [Aharoni et al., 2019; Zhang et al., 2018b; Tang et al.,
2021]. We experiment with improving coverage of low-resource languages by
selectively translating high-quality datasets from various existing
collections.

Setup We selectively pick 19 high-quality IFT datasets from xP3 [Muennighoff
et al., 2023c], the Flan Collection [Longpre et al., 2023a], Dolly [Conover et
al., 2023], along with additional sources such as SODA [Kim et al., 2022] and
Mintaka [Sen et al., 2022]. Datasets were prioritized for translation based on
the richness of task diversity and length of completions. The complete list of
these datasets is given in Appendix LABEL:tab:aya_collection_translated. These
translations are available and open-source as part of the Collection. We
process datasets for translation using the No Language Left Behind (NLLB)
[NLLB-Team et al., 2022] machine translation model, which is capable of
single-sentence translations between 200 different languages and dialects in
various scripts. For best performance, we use the largest NLLB model with 3.3B
parameters.

Figure 13: ChrF++ scores for the NLLB translation model, averaged across
resourcefulness buckets.

Translation Quality  Appendix Section G.1 lists NLLB translation quality for
each of the languages of interest, as reported in [NLLB-Team et al., 2022].
Figure 13 shows the translation quality across languages grouped by their
resourcefulness. The mean ChrF++ score on FLORES is 48.17 (min: 10.9, max:
69.6) for translations out of English, with a few outliers for HR and LR. We
interpret this optimistically as strong enough to sufficiently serve our
translation needs. However, upon inspection of translation outputs for fine-
tuning data, we encounter significant translation errors with Standard Arabic
in Latin script and Minangkabau in Arabic script, so we exclude them from our
translated dataset. In total, 19 public datasets were translated into 101
languages (114 dialects). Details of these datasets can be found in Appendix
Table LABEL:tab:aya_collection_translated.

In addition to releasing the translated datasets used as a basis for re-
annotation, we also translated Dolly [Conover et al., 2023]. Dolly is a 15k
instruction dataset Databricks collected by relying on its employees as
annotators [Conover et al., 2023]. Annotators were instructed to curate prompt
and completion pairs in each of eight different instruction categories. In
contrast to the mentioned NLP datasets, Dolly was purposefully designed to
align language models with human expectations. It stands out as a high-
quality, manually curated dataset covering a range of topics including
brainstorming, classification, closed question answering, generation,
information extraction, open question answering, and summarization. The
addition of the translated Dolly datasets is a valuable resource for languages
that experience a scarcity of conversational instruction fine-tuning datasets.

The list of datasets, along with the number of languages, templates, and other
statistics, can be found in Appendix Table
LABEL:tab:aya_collection_translated.

##  5 Analysis of Collection

Main Task Type | Fine-grained Task Type
---|---
Question Answering | —
Natural Language Generation | Summarization
| Translation
| Paraphrasing
| Dialogue
| Text Simplification
Text Classification | Sentiment Analysis
| Information Extraction
| Named Entity Recognition
| Event Linking
| Natural Language Inference
| Document Representation
Table 4: Task Taxonomy of NLP tasks in the Collection.

###  5.1 Statistics

Overview  The Collection consists of existing NLP datasets that are templated
to include instructions as well as datasets already in instruction format
submitted by the community. Table
LABEL:tab:list_of_templated_datasets_aya_collection shows the detailed list of
datasets. The full list of templates is available in Section J. The final
Collection consists of 44 multilingual and non-English templated datasets and
19 translated datasets, with 513M individual instances. Overall, the
collection covers 114 languages131313We release the Dataset as part of the
Collection, bringing the total number of languages in the collection to 115.
However, for the sake of clarity, when referencing the Collection statistics
in this paper, we exclude the Dataset..

Tasks Covered Across Templated and Translated Datasets  We aim to include
datasets from various tasks in the collection while ensuring that they follow
our selection criteria. Table 4 illustrates our task coverage in the
Collection, drawing inspiration from xP3 and the Flan Collection. We have a
total of three main task types: Question Answering (QA), Natural Language
Generation (NLG), and Text Classification (TC). Within these larger umbrella
tasks, we define several finer-grained task types based on the datasets,
resulting in a total of 11 finer-grained task types. These finer-grained task
types are determined by the frequency of datasets in the Collection
encapsulating that task.

Figure 14: Number of prompt/completion pairs in each language in the
Collection (templated). Many languages with limited digital presence, as
indicated by a low number of Wikipedia pages, are well-represented in the
templated portion of the Collection. Note that absolute Both axes are in log-
scale.

For QA, we decided to keep only the main task type, as the intended goal of
question-answering tasks is clear: Answer a proposed question. The type of the
question can be different: open-ended, close-ended, multiple-choice, single
response. For NLG, finer-grained task types include Summarization,
Translation, Paraphrasing, Dialogue (Generation), and Text Simplification. For
TC, we include the following finer-grained task types: Sentiment Analysis,
Information Extraction, Named Entity Recognition, Event Linking, Natural
Language Inference, and Scientific Document Representation. Finally, we label
the task categories of each dataset in the Collection in Table
LABEL:tab:templated_taxonomy and Table LABEL:tab:translated_taxonomy. If we
are not able to find a fine-grained task type for the dataset, we keep the
main task type.

Language Balance  One of the objectives of templating (and translating)
existing datasets is to broaden the available resources for languages that
have limited digital data. To examine if our final collection adheres to a
similar distribution pattern, we use the number of Wikipedia pages in each
language as a proxy for the online presence of its fluent speakers. Figure 14
showcases that although the number of instances for languages varies in the
Collection (templated subset), it does not disadvantage languages with fewer
Wikipedia pages. The distribution still ensures a reasonable coverage across
all languages. It is imperative to emphasize that our analysis does not
involve a direct comparison of absolute values, given the disparate units of
measurement involved. Instead, we examine the patterns of data scarcity for
various languages in our collection versus Wikipedia. Including the translated
datasets in the Collection further reduces disparities between languages and
contributes to creating a more balanced collection.

Prompt and Completion Lengths  Figure 15 shows the distribution of length
across languages. No discernible pattern is observed when examining lengths
for high-resource languages compared to low-resource languages. Low-resource
languages appear at both ends of the distribution, occupying both the head and
tail. In the Collection some low-resource languages (e.g., Somali and Amharic)
have longer average completions length than medium or even high-resource
languages. The dedication of individual participants in identifying datasets
in their own language and templating them has made a significant difference
for many languages.

Figure 15: The average length of prompts and completions for high (HR), medium
(MR) and low-resource (LR) languages in Collection.

###  5.2 Quality Assessment of All Different Data Sources

As previously stated, binary feedback on the quality of the prompt-completion
pairings was collected from the annotators. We define the average approval
ratio per dataset which serves as a valuable metric for assessing the quality
of datasets across various languages and diverse data sources. We compute the
average approval ratio as $\mathcal{T_{+}}/\mathcal{T}$, where
$\mathcal{T_{+}}$ represents the total number of thumbs up, and $\mathcal{T}$
represents the total number of votes per dataset. An average approval ratio of
1.0 would indicate that every annotation was perceived to be of good quality
and all prompts and completions had received a thumbs up. An average approval
ratio of 0.0 would indicate that every annotation was perceived to be of poor
quality, and all prompts and completions had received a thumbs down. We
constrained our quality analysis to the 40 datasets in our pool for which we
had at least 20 instances of feedback.

Figure 16: Average approval ratio per dataset group, constrained to datasets
receiving at least 20 votes.

Overall, we observe that the majority of datasets were of above average (0.5)
quality based on their approval ratio, with all translated data as well as
Original Annotations being above average. However, across all the datasets
within each group —xP3, Templated, Translated, and original annotations—
original annotations were perceived to be of the highest quality, with an
average approval ratio of approximately 0.81, compared to the lowest quality
dataset, xP3, which had an average approval ratio of approximately 0.50. This
aligns with our intuition that carefully curated datasets lead to high-quality
annotations as perceived by human annotators. Figure 16 provides a summary of
the results for each group. Figure 23 in the Appendix provides approval ratios
per datasets in each group.

##  6 Evaluation Suite

Lastly, as part of the project we curate and release an evaluation suite
tailored for multilingual models. This set is a valuable contribution in
tackling the scarcity of multilingual data, a challenge that becomes even more
apparent when considering evaluation sets. While there are several test sets
available for evaluating multilingual models [Conneau et al., 2018; Ponti et
al., 2020; Lin et al., 2022], they focus primarily on discriminative tasks. To
evaluate multilingual models’ generations, the literature includes task-
specific evaluation sets such as Translation [Goyal et al., 2021b],
Summarization [Hasan et al., 2021] and Question Answering [Clark et al.,
2020]. However there is currently a gap in evaluating open-ended generation
capabilities of LLMs within a multilingual context. We aim to address this gap
by curating a multilingual evaluation set tailored for assessing the open-
ended generation capabilities of LLMs, such as brainstorming, planning, and
other unstructured, long-form responses.

To strike a balance between language coverage and the quality that comes with
human attention, we create an evaluation suite that includes (1) human-curated
examples in a limited set of languages, (2) automatically translations of
handpicked examples into a more extensive number of languages, and (3) human-
post-edited translations into a small number of languages. We consider two
primary sources of data: original annotations from dataset (comprising new
examples culturally curated for different languages) and Dolly prompts (high-
quality, human-written examples carefully selected to have a universal reach).
The subsets comprising the evaluation suite are:

aya-human-annotated test set  For ease of future adoption, we have partitioned
the dataset into training and testing splits. The test set of the Dataset
contains 1,750 of the total instances (250 instances from 7 languages),
selected at random from original annotations. Our goal is to achieve a
balanced representation of languages in the test set and ensure a sufficient
number of examples per language. To guarantee enough data remains for
training, we focused on languages with at least 2000 original annotations. In
order to ensure linguistic diversity, we included languages that were varied
in terms of high, mid, or low-resourcedness, as well as script and language
families. For those reasons, the test set consists of English (high-resource,
Latin script, Indo-European), Portuguese (mid-resource, Latin script, Indo-
European), Simplified Chinese (high-resource, Han, Sino-Tibetan), Standard
Arabic (high-resource, Arabic script, Afro-Asiatic), Telugu (low-resource,
Telugu script, Dravidian), Turkish (mid-resource, Latin script, Turkic), and
Yoruba (low-resource, Latin script, Atlantic-Congo). See Table
LABEL:tab:language_codes for more details.

dolly-machine-translated test set  We separate a curated subset of 200 Dolly
prompts [Conover et al., 2023] to serve as an additional translated evaluation
set. Our aim with this selection was to exclude any culturally or
geographically specific prompts and completions. Hence, two reviewers
inspected a set of initially 500 English prompts that were uniformly sampled
based on the task categories in Dolly. The reviewers excluded prompts that
rely on geographic knowledge such as “Looking at cities in Australia that are
on the east coast and the west coast of the country, which coast are the
cities of Fremantle, Sydney, Brisbane, Perth, Cairns, Townsville, Newcastle
located on?”, or prompts such as “Why is NFL football called football when
players use their hands mainly?” that rely on overly specific cultural
references. When two reviewers disagreed, a third reviewer was asked to break
the tie. We kept prompts such as “Is art useless?” or “Write a short paragraph
about why you should not have both a pet cat and a pet bird.” and questions
that refer to geographic specific knowledge where the supporting evidence was
provided in the prompt itself e.g., “Given a reference text about Minister for
Food, Agriculture and Fisheries of Denmark, when was the position created and
was was it named?”. Although not perfect, the intention behind this selection
was to gather a test set that allows us to evaluate the fluency and quality of
responses in various languages while avoiding model assessment on prompts tied
to specific cultural or geographic references that might have language-
dependent validity. We automatically translate the prompts with NLLB into 101
languages and their dialects that are captured by NLLB. Including the original
English prompts this dataset covers 115 dialects.

dolly-human-edited test set  The automatic translation process may introduce
errors in the prompts that render them nonsensical. For example, the prompt
“Which is a species of fish? Bleak or Weary” requires domain expertise to
choose the right translation of the fish names rather than literal
translations of the adjectives (as e.g. in the NLLB Translation into Spanish:
“Desanimado o cansado.” (=“discouraged or tired”)). If the prompt does not
make any sense, there is no clear expectation and measurement of what a good
and correct completion should look like. To confidently interpret evaluation
results, it is imperative to establish a reliable set of prompts for
evaluation. To enhance the reliability of testing on these prompts, we
therefore enlist professional human annotators to post-edit the examples (e.g.
for the example above “Alburno o Cansado” (=“[Fish name] or Tired”). We post-
edit the prompts for a subset of six languages: Arabic, Hindi, Spanish,
French, Serbian and Russian. Appendix F describes the post-editing process and
effort in more detail. The example above illustrates that some prompts, even
when translated correctly, might still not transfer well into other
languages—which is the main difference between a translated English-centric
set like this and an evaluation set originally written in each target language
like aya-human-annotated.

We open-source the dolly-machine-translated test set to be an additional
resource for researchers, although warn that the expressiveness of a
translated evaluation set is limited by the quality of the translation model
(and human post-edit) and may adversely impact an estimate of ability in
languages where translations are not adequate [Nogara et al., 2023].
Ultimately, this is a compromise between having evaluation coverage in a more
complete set of languages (101 languages and 114 dialects in total) versus
having human-annotated evaluation sets. If using the automatically translated
test set, we recommend it be paired and reported with the professionally post-
edited dolly-human-edited for 6 languages, or the aya-human-annotated set
which also only covers 7 languages but is entirely created by proficient
target language speakers.

##  7 A Participatory Approach to Research

Recent breakthroughs in NLP have predominantly come from narrow collaborations
that involve researchers from a handful of institutions and regions of the
world [Nakamura et al., 2023]. This reliance on small, specialized
collaboration networks has been shown to hinder innovation [Park et al.,
2023]. Dataset creation as a process has often been undervalued, with
minimization of the value of creators’ contributions [Andress et al., 2020;
Peng et al., 2021; Hanley et al., 2020]. Under such conditions, the richness
and diversity of the data are often compromised, as it reflects a limited
perspective that aligns with the interests of those who wield greater power in
these transactions. Data is not, as metaphors such as ‘data mining’ [Puschmann
& Burgess, 2014], or ‘data is the new oil’ [Stark & Hoffmann, 2019; Awati &
Shum, 2015], might suggest, a natural resource waiting to be exploited.
Whenever we engage with data, we are also engaging with the connections that
data has to the people who produce, prepare, and distribute it [Seaver, 2021;
Pinel C, 2020; Crawford, 2021]. Participatory approaches in AI design and
research are one way to address gaps in access to resources needed for
research: through collaborative partnerships with language speakers and local
communities.

is an example of a participatory research project [Birhane et al., 2022;
Corbett et al., 2023; Delgado et al., 2023]. Here, the research is the result
of a broad cross-institutional, global collaboration. This type of cross-
sectional work facilitates the collection of vital linguistic data and
community engagement, which is crucial for developing effective language
technologies [Joshi et al., 2019; $\forall$ et al., 2020]. We describe below
some of the guiding principles we followed throughout the year-long project.

Fluid Ownership and Growth  Our open science framework allowed us to challenge
the norms of how computer science usually proceeds [Wittenburg, 2021; Sabou et
al., 2012]. Traditional research approaches often involve rigid hierarchies;
typically, research is conducted within academic institutions or corporate
labs where roles are clearly defined, and collaboration is mostly synchronous,
relying on in-person meetings or real-time communication. In contrast, took a
decentralized and democratic approach to collaboration, supporting fluid
leadership and flexible role adoption. This empowered members to take
initiative and lead in areas where they had passion or expertise, regardless
of their position in academia, or when they became involved in the project.
For example, members became Language Ambassadors at many different points
during the year-long project, and mentorship roles evolved naturally with more
experienced researchers providing guidance to those more junior (see Appendix
C for more details of different roles in the project).

Organizational Structure  The communication channels and organizational
structure of were designed to facilitate rich collaboration that could evolve
with the interests of participating researchers over the year-long project.
For example, most communication between independent researchers involved
within was asynchronous over Discord, which allowed researchers in different
time zones to participate in discussions. Monthly meetings were open for
anyone to attend and were recorded for asynchronous viewing. We describe the
structure of meetings and communication more thoroughly in Appendix D.1 and
D.2.

Inclusion and Access  The open nature of the UI allowed us to bypass the gate-
keeping mechanisms of academic science that often marginalize non-English
speakers and people without formal academic credentials [West et al., 2020].
Expertise in the command of a spoken or written language is clearly distinct
from expertise in machine learning. The inclusion of such a wide range of
volunteers gave us more representative data in a wide variety of languages and
also gave volunteers a glimpse into the often obscure world of machine
learning.

Who Participated in  The motivations of contributors were not based on
financial remuneration but on ideals of community, identity, and social
justice. Participants saw their roles as Language Ambassadors and contributors
as crucial to ensuring the inclusion of their languages in the ongoing
transition to a digital, information-driven economy. The Language Ambassador
for Malagasy, a language-driven to the risk of extinction by colonial French
rule in Madagascar [Spolsky, 2018], is planning hackathons in 2024 to use the
Dataset to create voice-to-text apps that will help non-literate speakers of
Malagasy participate in the modern economy. In Telugu, a traditional genre of
poetry known as Sathakam is an integral part of the educational system.
However, chatbots that can translate text into Telugu have little to no
understanding of the Sathakam form. The Telugu Language Ambassador told a
newspaper in Toronto that “in , we made sure to include as many Sathakams as
we could find” [Castaldo, 2023].

These motivations are not peripheral to the strength of the final Dataset but
are key factors in the data’s provenance [Loukissas, 2019]. These qualitative
dimensions remind us that language is, for the people who use it every day, an
intimately social phenomenon. Beyond the symbolic notation that connects
tokens to referents in the real world, we find a robust network of social
relations that are necessary for languages to flourish [Sidnell & Enfield,
2012; Goodwin, 2017; Agha, 2006]. The social interactions between
contributors, ML researchers, and social scientists in the project were
crucial to its success. Contributors shared playlists of their favorite songs
from their home country, recipes from their childhood, and snapshots of the
views from their home offices. They debated subtle nuances of how they wanted
their language represented in the dataset and pushed back on some of the
assumptions made by project coordinators on what constituted a distinct
language as opposed to a regional dialect (see Section 9). More than one
contributor sat down with their grandparents to contribute to a language that
spanned three generations of use.

The realities of the conditions under which many people work and live were
present every day. For example, Zoom meetings were cut short for some
volunteers due to power outages in their countries or lack of access to a
stable internet connection. Burmese, a language spoken in Myanmar, started out
strong in the project with a group of 35 motivated volunteers but saw a sudden
pause in contributions as civil war broke out in the country resulting in the
withdrawal of the volunteers from the project [Petty, 2023]. The Language
Ambassador for Armenian also had to drop out of the project because of a
conflict in that country [Reuters, 2023]. In some countries, postal services
only functioned a few days per month because of ongoing warfare, creating
challenges for organizers when mailing out gifts to thank committed
volunteers. Ultimately, organizers were not able to send gifts to thank
volunteers who participated from Somalia, Yemen and Palestine. For Somalia and
Yemen, both Canada Post, DHL and Fedex where not able to support shipments.
For Palestine, the cost of shipment proved to be prohibitively expensive –
with an estimated shipping cost of $294$ US dollars per t-shirt. These geo-
political realities shaped both our contributors’ experience as well as the
progress of the project.

Including these factors in our post-mortem analysis of the project is crucial
to understanding both the motivation of people willing to volunteer for open-
science projects, and also to understanding the data itself: its breadth, its
provenance, its shortcomings, and its living history.

Table 5: 65 languages in the Dataset and 114 languages in the Collection, each language’s corresponding script, family, subgrouping, and if it is classified as “lower-”, “mid-” or “higher”-resourced according to the taxonomy classes by [Joshi et al., 2020] (low: [0, 1, 2], mid: [3], high: [4, 5]). The language is either included in the Dataset (  $\diamondsuit$ ), Collection (  $\spadesuit$ ), or both. Note that Ngaju (nij) and Toba Batak (bbc) are not listed in [Joshi et al., 2020]. ISO Code | Language | Script | Family | Subgrouping | Resources | Included
---|---|---|---|---|---|---
ace | Achinese | Arabic/Latin | Austronesian | Malayo-Polynesian | Low |  |  $\spadesuit$
---
afr | Afrikaans | Latin | Indo-European | Germanic | Mid |  |  $\spadesuit$
---
amh | Amharic | Ge’ez | Afro-Asiatic | Semitic | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
ara | Arabic | Arabic | Afro-Asiatic | Semitic | High |  |
---
$\diamondsuit$
$\spadesuit$
---
aze | Azerbaijani | Arabic/Latin | Turkic | Common Turkic | Low |  |  $\spadesuit$
---
ban | Balinese | Latin | Austronesian | Malayo-Polynesian | Low |  |  $\spadesuit$
---
bbc | Toba Batak | Latin | Austronesian | Malayo-Polynesian | Low |  |  $\spadesuit$
---
bel | Belarusian | Cyrillic | Indo-European | Balto-Slavic | Mid |  |  $\spadesuit$
---
bem | Bemba | Latin | Niger-Congo | Atlantic-Congo | Low |  |  $\spadesuit$
---
ben | Bengali | Bengali | Indo-European | Indo-Aryan | Mid |  |
---
$\diamondsuit$
$\spadesuit$
---
bjn | Banjar | Arabic/Latin | Austronesian | Malayo-Polynesian | Low |  |  $\spadesuit$
---
bul | Bulgarian | Cyrillic | Indo-European | Balto-Slavic | Mid |  |  $\spadesuit$
---
cat | Catalan | Latin | Indo-European | Italic | High |  |  $\spadesuit$
---
ceb | Cebuano | Latin | Austronesian | Malayo-Polynesian | Mid |  |
---
$\diamondsuit$
$\spadesuit$
---
ces | Czech | Latin | Indo-European | Balto-Slavic | High |  |  $\spadesuit$
---
cym | Welsh | Latin | Indo-European | Celtic | Low |  |  $\spadesuit$
---
dan | Danish | Latin | Indo-European | Germanic | Mid |  |
---
$\diamondsuit$
$\spadesuit$
---
deu | German | Latin | Indo-European | Germanic | High |  |
---
$\diamondsuit$
$\spadesuit$
---
ell | Greek | Greek | Indo-European | Graeco-Phrygian | Mid |  |
---
$\diamondsuit$
$\spadesuit$
---
eng | English | Latin | Indo-European | Germanic | High |  |
---
$\diamondsuit$
$\spadesuit$
---
epo | Esperanto | Latin | Constructed | Esperantic | Low |  |  $\spadesuit$
---
est | Estonian | Latin | Uralic | Finnic | Med |  |  $\spadesuit$
---
eus | Basque | Latin | Basque | - | High |  |
---
$\diamondsuit$
$\spadesuit$
---
fil | Filipino | Latin | Austronesian | Malayo-Polynesian | Mid |  |
---
$\diamondsuit$
$\spadesuit$
---
fin | Finnish | Latin | Uralic | Finnic | Mid |  |
---
$\diamondsuit$
$\spadesuit$
---
fon | Fon | Latin | Niger-Congo | Atlantic-Congo | Low |  |  $\spadesuit$
---
fra | French | Latin | Indo-European | Italic | High |  |
---
$\diamondsuit$
$\spadesuit$
---
gla | Scottish Gaelic | Latin | Indo-European | Celtic | Low |  |  $\spadesuit$
---
gle | Irish | Latin | Indo-European | Celtic | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
glg | Galician | Latin | Indo-European | Italic | Med |  |  $\spadesuit$
---
guj | Gujarati | Gujarati | Indo-European | Indo-Aryan | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
hat | Haitian Creole | Latin | Indo-European | Italic | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
hau | Hausa | Latin | Afro-Asiatic | Chadic | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
heb | Hebrew | Hebrew | Afro-Asiatic | Semitic | Mid |  |  $\spadesuit$
---
hin | Hindi | Devanagari | Indo-European | Indo-Aryan | High |  |
---
$\diamondsuit$
$\spadesuit$
---
hrv | Croatian | Latin | Indo-European | Balto-Slavic. | High |  |  $\spadesuit$
---
hun | Hungarian | Latin | Uralic | - | High |  |
---
$\diamondsuit$
$\spadesuit$
---
hye | Armenian | Armenian | Indo-European | Armenic | Low |  |  $\spadesuit$
---
ibo | Igbo | Latin | Atlantic-Congo | Benue-Congo | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
ind | Indonesian | Latin | Austronesian | Malayo-Polynesian | Mid |  |
---
$\diamondsuit$
$\spadesuit$
---
isl | Icelandic | Latin | Indo-European | Germanic | Low |  |  $\spadesuit$
---
ita | Italian | Latin | Indo-European | Italic | High |  |
---
$\diamondsuit$
$\spadesuit$
---
jav | Javanese | Latin | Austronesian | Malayo-Polynesian | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
jpn | Japanese | Japanese | Japonic | Japanesic | High |  |
---
$\diamondsuit$
$\spadesuit$
---
kan | Kannada | Kannada | Dravidian | South Dravidian | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
kas | Kashmiri | Arabic | Indo-European | Indo-Aryan | Low |  |  $\spadesuit$
---
kat | Georgian | Georgian | Kartvelian | Georgian-Zan | Mid |  |  $\spadesuit$
---
kau | Kanuri | Arabic/Latin | Saharan | Western Saharan | Low |  |  $\spadesuit$
---
kaz | Kazakh | Cyrillic | Turkic | Common Turkic | Mid |  |  $\spadesuit$
---
khm | Khmer | Khmer | Austroasiatic | Khmeric | Low |  |  $\spadesuit$
---
kin | Kinyarwanda | Latin | Niger-Congo | Atlantic-Congo | Low |  |  $\spadesuit$
---
kir | Kyrgyz | Cyrillic | Turkic | Common Turkic | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
kor | Korean | Hangul | Koreanic | Korean | Mid |  |
---
$\diamondsuit$
$\spadesuit$
---
kur | Kurdish | Latin | Indo-European | Iranian | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
lao | Lao | Lao | Tai-Kadai | Kam-Tai | Low |  |  $\spadesuit$
---
lav | Latvian | Latin | Indo-European | Balto-Slavic | Mid |  |  $\spadesuit$
---
lij | Ligurian | Latin | Indo-European | Italic | Low |  |  $\spadesuit$
---
lit | Lithuanian | Latin | Indo-European | Balto-Slavic | Mid |  |
---
$\diamondsuit$
$\spadesuit$
---
ltz | Luxembourgish | Latin | Indo-European | Germanic | Low |  |  $\spadesuit$
---
mad | Madurese | Latin | Austronesian | Malayo-Polynesian | Low |  |  $\spadesuit$
---
mal | Malayalam | Malayalam | Dravidian | South Dravidian | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
man | Manipuri | Bengali | Sino-Tibetan | Kuki-Chin-Naga | Low |  |  $\spadesuit$
---
mar | Marathi | Devanagari | Indo-European | Indo-Aryan | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
min | Minangkabau | Latin | Austronesian | Malayo-Polynesian | Low |  |  $\spadesuit$
---
mkd | Macedonian | Cyrillic | Indo-European | Balto-Slavic | Low |  |  $\spadesuit$
---
mlg | Malagasy | Latin | Austronesian | Malayo-Polynesian | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
mlt | Maltese | Latin | Afro-Asiatic | Semitic | Low |  |  $\spadesuit$
---
mon | Mongolian | Cyrillic | Mongolic-Khitan | Mongolic | Low |  |  $\spadesuit$
---
mri | Maori | Latin | Austronesian | Malayo-Polynesian | Low |  |  $\spadesuit$
---
msa | Malay | Latin | Austronesian | Malayo-Polynesian | Mid |  |
---
$\diamondsuit$
$\spadesuit$
---
mya | Burmese | Myanmar | Sino-Tibetan | Burmo-Qiangic | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
nep | Nepali | Devanagari | Indo-European | Indo-Aryan | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
nij | Ngaju | Latin | Austronesian | Malayo-Polynesian | Low |  |  $\spadesuit$
---
nld | Dutch | Latin | Indo-European | Germanic | High |  |
---
$\diamondsuit$
$\spadesuit$
---
nor | Norwegian | Latin | Indo-European | Germanic | Low |  |  $\spadesuit$
---
nso | Northern Sotho | Latin | Atlantic-Congo | Benue-Congo | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
nya | Chichewa | Latin | Atlantic-Congo | Benue-Congo | Low |  |
---
$\diamondsuit$
pan | Punjabi | Gurmukhi | Indo-European | Indo-Aryan | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
pes | Persian | Arabic | Indo-European | Iranian | High |  |
---
$\diamondsuit$
$\spadesuit$
---
pol | Polish | Latin | Indo-European | Balto-Slavic | High |  |
---
$\diamondsuit$
$\spadesuit$
---
por | Portuguese | Latin | Indo-European | Italic | High |  |
---
$\diamondsuit$
$\spadesuit$
---
pus | Pashto | Arabic | Indo-European | Iranian | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
ron | Romanian | Latin | Indo-European | Italic | Mid |  |  $\spadesuit$
---
rus | Russian | Cyrillic | Indo-European | Balto-Slavic | High |  |
---
$\diamondsuit$
$\spadesuit$
---
sin | Sinhala | Sinhala | Indo-European | Indo-Aryan | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
slk | Slovak | Latin | Indo-European | Balto-Slavic | Mid |  |  $\spadesuit$
---
slv | Slovenian | Latin | Indo-European | Balto-Slavic | Mid |  |  $\spadesuit$
---
smo | Samoan | Latin | Austronesian | Malayo-Polynesian | Low |  |  $\spadesuit$
---
sna | Shona | Latin | Indo-European | Indo-Aryan | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
snd | Sindhi | Arabic | Indo-European | Indo-Aryan | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
som | Somali | Latin | Afro-Asiatic | Cushitic | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
sot | Southern Sotho | Latin | Atlantic-Congo | Benue-Congo | Low |  |  $\spadesuit$
---
spa | Spanish | Latin | Indo-European | Italic | High |  |
---
$\diamondsuit$
$\spadesuit$
---
sqi | Albanian | Latin | Indo-European | Albanian | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
srp | Serbian | Cyrillic | Indo-European | Balto-Slavic | High |  |
---
$\diamondsuit$
$\spadesuit$
---
sun | Sundanese | Latin | Austronesian | Malayo-Polynesian | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
swa | Swahili | Latin | Atlantic-Congo | Benue-Congo | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
swe | Swedish | Latin | Indo-European | Germanic | High |  |
---
$\diamondsuit$
$\spadesuit$
---
tam | Tamil | Tamil | Dravidian | South Dravidian | Mid |  |
---
$\diamondsuit$
$\spadesuit$
---
taq | Tamasheq | Latin/Tifinagh | Afro-Asiatic | Berber | Low |  |  $\spadesuit$
---
tel | Telugu | Telugu | Dravidian | South Dravidian | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
tgk | Tajik | Cyrillic | Indo-European | Iranian | Low |  |  $\spadesuit$
---
tha | Thai | Thai | Tai-Kadai | Kam-Tai | Mid |  |
---
$\diamondsuit$
$\spadesuit$
---
tur | Turkish | Latin | Turkic | Common Turkic | High |  |
---
$\diamondsuit$
$\spadesuit$
---
twi | Twi | Latin | Niger-Congo | Atlantic-Congo | Low |  |  $\spadesuit$
---
ukr | Ukrainian | Cyrillic | Indo-European | Balto-Slavic | Mid |  |
---
$\diamondsuit$
$\spadesuit$
---
urd | Urdu | Arabic | Indo-European | Indo-Aryan | Mid |  |
---
$\diamondsuit$
$\spadesuit$
---
uzb | Uzbek | Latin | Turkic | Common Turkic | Med |  |  $\spadesuit$
---
vie | Vietnamese | Latin | Austroasiatic | Vietic | High |  |
---
$\diamondsuit$
$\spadesuit$
---
wol | Wolof | Latin | Atlantic-Congo | North-Central Atlantic | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
xho | Xhosa | Latin | Atlantic-Congo | Benue-Congo | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
yid | Yiddish | Hebrew | Indo-European | Germanic | Low |  |  $\spadesuit$
---
yor | Yorùbá | Latin | Atlantic-Congo | Benue-Congo | Low |  |
---
$\diamondsuit$
$\spadesuit$
---
zho | Chinese | Han | Sino-Tibetan | Sinitic | High |  |
---
$\diamondsuit$
$\spadesuit$
---
zul | Zulu | Latin | Atlantic-Congo | Benue-Congo | Low |  |
---
$\diamondsuit$
$\spadesuit$
---

##  8 Related Work

###  8.1 Multilingual datasets

Low-resource languages have long been a challenge in NLP, with limited data
impacting task performance [Kunchukuttan et al., 2021]. To address this,
researchers have explored techniques like data augmentation [Sennrich et al.,
2016; Dhole et al., 2021], transfer learning [Zoph et al., 2016], repeating
[Luukkonen et al., 2023; Muennighoff et al., 2023b], and multilingual models
[Dabre et al., 2020; Muennighoff et al., 2023c; Yong et al., 2023b], achieving
promising results in areas like machine translation. Here, we focus on efforts
that are centered on multilingual dataset creation.

Several works have created large-scale multilingual corpora. These are often
unstructured texts, ideal for large-scale unsupervised pre-training [Abadji et
al., 2021; Ortiz Su’arez et al., 2019; Scao et al., 2022a, b; Laurençon et
al., 2022; Kudugunta et al., 2023; Whitehouse et al., 2023]. Another group of
multilingual datasets is focused on machine translation [Lucia Specia et al.,
2010; Fan et al., 2021]. They consist of parallel texts in two or more
languages, enabling models to learn the mappings between them. Ideally,
machine translation datasets encompass diverse domains and language pairs,
from commonly spoken languages to resource-scarce ones, promoting inclusivity
and linguistic diversity. One of the most extensive collections of parallel
corpora is available at the OPUS project website141414https://opus.nlpl.eu
[Tiedemann, 2012]. Large capacity models for language understanding may obtain
strong performance on high-resource languages while greatly improving low-
resource languages [Goyal et al., 2021a]. In Whitehouse et al. [2023], the
effectiveness of LLM-powered data augmentation in cross-lingual commonsense
reasoning was demonstrated. An improved performance was shown when smaller
cross-lingual models were finetuned with data generated by LLMs. Some recently
released datasets focus on specialized language domains such as law [Niklaus
et al., 2023], education [Zhang et al., 2023c], or healthcare [Wang et al.,
2023].

These corpora often suffer from inadequate data quality and require extensive
cleaning [Abadji et al., 2022; Kreutzer et al., 2022]. Task-specific datasets,
such as XCOPA [Ponti et al., 2020] or XNLI [Conneau et al., 2018], are smaller
in scale but offer higher quality data targeted at a specific model capability
such as cross-lingual understanding and transfer learning. This type of data
is crucial for evaluating and enhancing the performance of models in diverse
linguistic contexts.

No Language Left Behind [NLLB-Team et al., 2022] open-sourced bitext, mined
bitext, and data generated using back-translation in 200+ languages
specifically for text-to-text translation. While Seamless4MT [Barrault et al.,
2023] released the metadata of SeamlessAlign, an open multimodal translation
dataset, there are relatively fewer works for data creation/curation in low-
resource languages. Cahyawijaya et al. [2023] introduced NusaCrowd, a
standardized collection of 137 datasets covering 19 Indonesian local languages
in text, speech, and image modalities. Our work differs from previous datasets
as we create a large-scale instruction-tuning dataset spanning hundreds of
different tasks, yet retain high-quality by involving human annotation and
rigorous quality control across the entire data creation process.

###  8.2 Instruction-tuning datasets

Instruction-tuning datasets are collections of human-curated instructions and
response pairs, templatized NLP tasks, or synthetic instructions generated by
a language model. There are a growing number of NLP meta-datasets such as
Natural instructions [Mishra et al., 2022], SuperNatural Instructions[Wang et
al., 2022d], Flan 2021 [Wei et al., 2022a], Flan 2022 [Longpre et al., 2023a],
Public Pool of Prompts (P3) [Sanh et al., 2022], Unnatural Instructions
[Honovich et al., 2023], OPT-IML [Iyer et al., 2022], inter alia [Khashabi et
al., 2020; Ye et al., 2021; Min et al., 2021] that collate numerous
instruction finetuned datasets together. Some work focuses on specific
applications such as dialogue [Köpf et al., 2023], structured knowledge
grounding [Xie et al., 2022], or chain-of-thought reasoning [Wei et al.,
2022b; Kim et al., 2023]. Manual efforts include Open Assistant [Köpf et al.,
2023] crowd-sourcing volunteers who wrote both instructions and responses,
Databricks employees creating 15k examples in Dolly [Conover et al., 2023],
and LIMA [Zhou et al., 2023] which is a collection of 1,000 author-curated IFT
examples.

Synthetic instruction-tuning datasets comprise instructions sampled from a
language model, such as the Self-Instruct dataset [Wang et al., 2022b]
generated by GPT-3 [Brown et al., 2020], the Alpaca dataset [Taori et al.,
2023] generated by GPT-3.5, and the Guanaco dataset [Joseph Cheung, 2023].
Increasingly, the synthetic generation of instruction-finetuned datasets is
more sophisticated. [Xu et al., 2023a] propose a novel Evol-Instruct framework
to obtain complex and difficult instructions gradually. [Luo et al., 2023] and
[Gunasekar et al., 2023] further expand this idea to promote reasoning, code
generation, and algorithmic skills. InstructionWild [Ni et al., 2023] and
ShareGPT151515https://sharegpt.com/ are collections of user-shared
conversations with ChatGPT.

###  8.3 Multilingual Instruction-Tuning Datasets

Despite ever-larger collections of IFT datasets, prior work has been largely
English-centric. Most approaches to extend instruction finetuned datasets
outside of English have relied on 1) translating English datasets into other
languages [Holmström & Doostmohammadi, 2023; Li et al., 2023a; Winata et al.,
2023b], 2) template based dataset creation [Yu et al., 2023; Gupta et al.,
2023] or 3) human curating instruction datasets in languages outside of
English [Muennighoff et al., 2023c; Li et al., 2023c; Wang et al., 2022c].
There have been some notable exceptions with large proportions of non-English
data [Joseph Cheung, 2023; Köpf et al., 2023; Lai et al., 2023; Li et al.,
2023a; Longpre et al., 2023a; Muennighoff et al., 2023a, c; Zhuo et al., 2024;
Nguyen et al., 2023].

Template-Based Datasets. The most relevant effort is recent work by
[Muennighoff et al., 2023c] releasing Crosslingual Public Pool of Prompts
(xP3). xP3 expands the P3 taxonomy and adds 28 new multilingual datasets.
However, their datasets usually use the same template in different languages,
thus limiting task diversity. For example, a random batch from their dataset
may include the same sample in different languages multiple times. Their xP3
corpus has task instructions exclusively in English. In [Muennighoff et al.,
2023c], the experiments with matching the task instruction to the respective
language of the sample via machine translation (xP3mt) showed slightly
improved performance for non-English task instructions at inference. Our work
is distinct in that our human-curated constructed dataset is unique for each
of the 65 languages. Such diversity has been emphasized as a key ingredient
for instruction tuning [Longpre et al., 2023a]. Further, we create non-English
task instructions via human annotators, ensuring these are of high-quality,
which is another pillar of a good performance [Zhou et al., 2023].

Machine Translated Datasets. Machine-translated prompts often lack variability
and the cultural nuance inherent in natively written text. However, they are
still useful for expanding the language coverage of the training data and can
help bridge the resource gap for languages with limited training data [Urbizu
et al., 2023; Lin et al., 2022]. They can also adapt already-trained
instruction-tuned language models to follow instructions in new languages
[Yong et al., 2023b]. Furthermore, LLMs trained on designed prompts have also
been shown to be successful at tasks like EAE (Event Argument Extraction) from
multilingual data in a zero-shot setup [Huang et al., 2022]. [Zhang et al.,
2023a] constructed high-quality Chinese instructions from existing English
instruction datasets. They first translated the English instructions into
Chinese and then used a human verification process to determine whether these
translations were usable; the verified dataset set consists of around 200k
Chinese instruction-tuning samples. [Li et al., 2023a] constructed instruction
data for 52 popular languages using Google Translate to translate English
prompts and completions from Alpaca [Taori et al., 2023] (52K) and Dolly
[Conover et al., 2023] (15K) dataset, then used this data to finetune LLaMA
[Touvron et al., 2023] using the LoRA [Hu et al., 2021] technology. [Zhang et
al., 2023b] prompted LLMs to translate a task request, which was overlaid with
the more granular user-based corrects. This process naturally connects
different languages as well as human preferences with LLMs, leveraging LLaMA
[Touvron et al., 2023] for foundational support and employing automatic
construction of interactive translation instructions for instructional tuning,
thereby enhancing the model’s multilingual capability and alignment with
diverse linguistic needs.

Human-Curated Multilingual Examples. Most relevant to our work on the dataset
are other datasets that have been curated by humans, often in English.
Databricks collected a 15k instruction dataset databricks-dolly-15k by relying
on its employees as annotators [Conover et al., 2023]. Annotators were
instructed to curate prompt / response pairs in each of eight different
instruction categories. [Köpf et al., 2023] released the OpenAssistant corpus
with over 10,000 dialogues from more than 13,500 international annotators.
While this dataset contains multilingual annotations, this was not an explicit
goal of the initiative. In contrast to our corpus which only has 2.05%
contributions in English, 42.8% of the OpenAssistant project remains in
English [Köpf et al., 2023].

###  8.4 Participatory Research in Machine Learning

> If you want to go fast go alone; if you want to go far, go together. —
> African Proverb

Prior participatory research initiatives have centered around regions or
specific tasks like translation or character recognition. For example,
[Clanuwat et al., 2018] tackles the problem of reading and understanding
Kuzushiji, a cursive style of Japanese writing no longer in common use.
Another example of culturally diverse data collection is [Liu et al., 2021],
which recruited native speakers from five languages (Indonesian, Swahili,
Tamil, Turkish, and Mandarin Chinese) that are typologically, genealogically,
and geographically diverse, to provide images of concepts that are
representative of their cultures. Then, they recruited native-speaking
professional linguists to write captions for these images. However, this
dataset is small (less than 8,000 data points) and thus limited to evaluation
only. It is worth noting that these works are solely focused on the image
domain, unlike our work, which concentrates on text.

More relevant to our work are participatory data creation initiatives focused
on NLP. [Guevara-Rukoz et al., 2020] presents a study focusing on the creation
of a crowd-sourced corpus for Latin American Spanish dialects to address the
scarcity of resources for these languages. [$\forall$ et al., 2020] focuses on
the task of Machine Translation (MT), and curates a dataset in 30 under-
represented African languages according to a participatory research framework.
Our work is very much in the spirit of these prior efforts, with differences
in terms of global rather than regional focus. In contrast to these works,
which have a specific regional focus, collaborators came from multiple
continents covering a diverse range of languages.

Several works have explored the organizational structures required to
facilitate the development of research communities around under-represented
languages. [Siminyu et al., 2021] details work on the AI4D - African Language
Program, which aimed to enhance language resources for African languages. The
outcome included creating over nine open-source African language datasets and
establishing baseline models, demonstrating the program’s significant impact
on language technology for African languages. [Azunre et al., 2021] describes
the establishment of NLP Ghana, with its collaborative open-source community.
[Strassel & Tracey, 2016] discusses the challenges of developing resources for
low-resource languages under the LORELEI (Low Resource Languages for Emergent
Incidents) program. They focus on the pressing need for digital resources in
these languages, particularly in critical situations such as mitigating the
effects of natural disasters.

Open science community initiatives like yield significant advancements in
language modeling. Related efforts (in terms of compute and resources
required) can be found in the BigScience Workshop [Akiki et al., 2022], which
began in 2021. The BigScience project was initiated to address the limitations
in LLM development, emphasizing open science and inclusive collaboration.
Leveraging open science principles, it united a global network of researchers
working to collaboratively and ethically enhance machine learning. Their work
culminated in key developments like the BLOOM model [Scao et al., 2022a] and
ROOTS corpus [Laurençon et al., 2022]. These achievements underscore the value
of community-driven, ethical, and diverse research programs for large-scale
language technologies. Following Big Science, there have been other recent
efforts on open science in language modeling [Groeneveld et al., 2024;
Soldaini et al., 2024].

##  9 Limitations of our work

  1. 1.

Language and dialect coverage: The Dataset and Collection cover 65 and 114
languages respectively—significantly more than existing multilingual datasets.
However, this is still only a tiny fraction of the world’s linguistic
diversity. Of the world’s approximately 7,000 languages, only half of them are
captured in any sort of written form [Adda et al., 2016]. Of this half, only a
few hundred are included on the internet in machine readable corpora [Adda et
al., 2016]. This means that 93% of the world’s languages are still not being
used to train LLMs. It is also notoriously difficult to determine the dividing
line between different languages and different dialects of the same language
[Rooy, 2021]. Geo-cultural variation within a language often gives rise to new
dialects or creoles over time [Zampieri et al., 2020; Wolfram, 1997; Brown et
al., 2020; Lent et al., 2022; Blaschke et al., 2023] and, as such, dialects
can serve an important function in establishing and maintaining cultural
identity [Falck et al., 2012]. Many different dialects that are generally
recognized as belonging to a single parent language are not represented in the
dataset. For example, in the case of Malay, one of the largest Southeast Asian
languages in the dataset, there are no contributions for regional dialects
that are widely spoken in certain states of Malaysia. Contributions by
volunteers who wished to self-identify as speaking a particular dialect were
tagged as such in the data to allow for limited analysis of the use of
regional dialects in annotations. Lastly, socio-linguistic data show that
multilingual speakers often ‘code-switch’ between languages or dialects
depending on context [Myers-Scotton, 2017], but in this project, we kept the
languages isolated to make them easier to classify and to be used downstream
for language-specific applications. also does not cover programming languages.
There has been prior work on covering diverse programming languages [Li et
al., 2023d; Allal et al., 2023] and we leave further explorations in this
direction to future work.

  2. 2.

Uneven distribution of contributions: As explored in Section 3.3, despite the
large number of participants, the activity of annotators was skewed, with a
‘long tail’ of annotators only contributing one or two annotations. Relatively
few contributors accounted for the most annotations (see Figure 10 \- bottom).
Similarly, there is a huge gap between languages with the highest number of
contributions and ones with the lowest number of contributions. Consequently,
this suggests potential unevenness in dataset distributions across different
languages and a lack of annotator diversity within some languages dominated by
one or two frequent contributors.

  3. 3.

Cultural or personal bias: Another limitation is the presence of annotations
with particular cultural biases. Some languages in our dataset have limited
representation, with only a few annotators responsible for annotating the bulk
of their dataset. This might mean that data for a particular language is
dominated by annotations that represent the opinions or priorities of a
particular contributor or could represent a narrow selection of cultural
viewpoints. For example, annotations in French might contain many examples
about the history of France, its food, songs, and other cultural practices,
but not contain much information about the cultural heritage of French-
speaking communities in Québec, Togo, or Senegal [Vigouroux, 2013]. This bias
is particularly problematic given the skewed distribution of the most active
annotators. There is also a potential bias in the availability of particular
kinds of content. For example, it is easier to find online text from news
sites for many African languages than it is to find text from other domains.
Accordingly, these datasets will be skewed towards the grammar and lexicon
used in news reports instead of the kind of natural language people use in
everyday life [Hovy & Prabhumoye, 2021].

  4. 4.

Gendered pronouns: Many of the languages in the Dataset only contain pronouns
that are explicitly gendered (e.g., Arabic) or that lack gender-neutral third-
person pronouns for gender-neutral reference (e.g. Estonian). This means that
in responding to prompts that might not specify a gender, care needs to be
taken to ensure that responses remain neutral as to the gender of any assumed
participants [Ghosh & Caliskan, 2023]. For example, if a response requires
reference to “a teacher” in French, the annotator would need to include
references to both “un/e enseignant/e”. While care was taken to ensure neutral
responses for new annotations, gendered annotations in existing datasets might
not have been flagged, as they are not, strictly speaking, incorrect. Instead,
they merely presuppose a gendered reading where one might not be implied
[Hardmeier & Guillou, 2018].

  5. 5.

Formality distinctions: Many of the languages in the Dataset also require the
speaker or annotator to make situational choices as to the formality of the
pronoun used in response to a particular prompt. Languages such as Japanese,
Persian, Indonesian, Javanese, Yoruba, French, Spanish, and German include
different levels of honorifics that are used in formal or informal settings,
or used between community members who differ in status (determined by a
variety of factors such as age, profession, seniority, or ethnicity) [Brown &
Gilman, 1968]. In Yoruba, for example, the pronoun that roughly translates as
"they" can either be used as a singular honorific or as a third-person plural
pronoun [Yusuf, 2022]. We deferred to the individual annotators in crafting
their responses, allowing them to rely on the norms of their particular speech
community to determine how to respond. Often, these decisions hinged on the
content being discussed, or on how formally the prompt was crafted in the
original data set. When in doubt, annotators were asked to imagine what kind
of ‘voice’ they would expect an LLM to have when answering a given prompt
[Wilson, 2023].

This means our released dataset contains many languages that have varying
levels of standardization and differing style guidelines. Standardization is
often deeply intertwined with power and identity, and the manner of speech may
be connected to aspects of identity like age, education level, tribal
affiliation, and religion. The lack of standardization is also largely due to
regional and cultural differences across the same language, exemplified by
Portuguese in the dataset: European Portuguese diverges from Brazilian
Portuguese not only in formality but also in grammar, spelling, and
vocabulary. Often, standards are projected by others to ensure adherence to
cultural values [Bourdieu, 1987; De Mauro et al., 2015; Haugen, 1959; Rickford
et al., 2012].

  6. 6.

Toxic or offensive speech: The Annotation Platform did not contain specific
flags for toxic, harmful, or offensive speech, so it is possible that
malicious users could submit unsafe data. We believe this is of relatively low
risk because of the high rate of human-verified annotations and peer-review,
making it unlikely that toxic prompts or completions made it into the final
dataset. However, there is no guarantee that every entry was audited. While
data poisoning has rarely been observed as a viable threat in practice, it has
been demonstrated to be of concern for instruction-tuning with very few
examples [Xu et al., 2023b; Wan et al., 2023] and for pre-training under
realistic conditions [Carlini et al., 2023]. During the eight months of crowd-
sourced annotating, there were no reported cases of hateful or toxic speech in
the existing datasets nor were there any instances of offensive speech
reported in the peer-reviewing phase of new annotations.

We also note that data that might be offensive to one annotator might not be
offensive to another, for instance, the completion of a prompt that asks for a
definition of the word “woke” [Castaldo, September 16, 2023]. Prompts written
on partisan political topics, or the inclusion of political advertisements or
campaign messages could cause offense depending on the political proclivities
of the annotator. In short, we tried to mitigate offensive speech by relying
heavily on human annotation and peer review, but there is no guarantee that
all such data points were removed from the corpus.

  7. 7.

Accounting for mislabeled data: The Annotation Platform did not contain any
components that enabled re-labeling the assigned language of annotations. This
may result in prompts and completions that appear under a particular language,
but were submitted incorrectly and would need to be re-categorized into a
different language. Additionally, while we trusted annotators were able to
follow directions and had a high rate of manual auditing, some examples likely
made it into the Dataset that were not in instruction-style format or were
free-form texts.

##  10 Conclusion

Open participatory research continues to be under-resourced and undervalued,
particularly when that work focuses on data creation [Sambasivan et al.,
2021]. involved participants from many different countries, different ages,
and different levels of familiarity with the field of natural language
processing. We see continued opportunity for computational linguists and
machine-learning engineers to collaborate with social scientists such as
sociolinguists, anthropologists, sociologists, and media studies scholars. As
new norms in open science emerge [Krishna, 2020; Bowser et al., 2020],
collaborations like these can help ensure that projects in NLP are motivated
by an understanding of what language means to the people who use it every day.

With , we hope to change the way data is created for multilingual NLP
research. In line with this view, we release the Dataset which is the first
human-curated open-source, multilingual instruction-style dataset consisting
of 204,114 prompt-completion pairs covering 65 languages. This dataset was
built with the help of our open-science community of 2,997 collaborators from
119 countries over a period of eight months.

We also release the Collection, which consists of 44 instruction-style
datasets. These were prepared by transforming existing NLP datasets into
prompt-completion pairs that can be leveraged for instruction tuning.
Furthermore, we translate several high-quality datasets into 101 languages,
thereby expanding coverage, particularly for many low-resource languages. This
collection consists of 513M prompt and completion pairs covering 114 languages
in total and is the largest multilingual instruction-finetuning collection
today. Additionally, we release Evaluation Suite, consisting of human-curated
examples in 13 languages and translation of carefully selected prompts in 101
languages. Finally, we are also open-sourcing the Annotation Platform so that
communities can continue to use the platform to support the process of
multilingual data collection. We hope these communities continue to grow and
develop, and to connect speakers of low-resource languages around the world.

## References

  * Abadji et al. [2021] Julien Abadji, Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot.  Ungoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus.  In Harald Lüngen, Marc Kupietz, Piotr Bański, Adrien Barbaresi, Simon Clematide, and Ines Pisetta (eds.), _CMLC 2021-9th Workshop on Challenges in the Management of Large Corpora_ , Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-9) 2021. Limerick, 12 July 2021 (Online-Event), pp. 1 – 9, Mannheim, 2021. Leibniz-Institut für Deutsche Sprache.  10.14618/ids-pub-10468.  URL https://nbn-resolving.org/urn:nbn:de:bsz:mh39-104688.
  * Abadji et al. [2022] Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot.  Towards a cleaner document-oriented multilingual crawled corpus.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pp. 4344–4355, Marseille, France, June 2022. European Language Resources Association.  URL https://aclanthology.org/2022.lrec-1.463.
  * Abedissa et al. [2023] Tilahun Abedissa, Ricardo Usbeck, and Yaregal Assabie.  AmQA: Amharic Question Answering Dataset.  _arXiv preprint arXiv:2303.03290_ , 2023.
  * Adda et al. [2016] Gilles Adda, Sebastian Stüker, Martine Adda-Decker, Odette Ambouroue, Laurent Besacier, David Blachon, Hélène Bonneau-Maynard, Pierre Godard, Fatima Hamlaoui, Dmitry Idiatov, Guy-Noël Kouarata, Lori Lamel, Emmanuel-Moselly Makasso, Annie Rialland, Mark Van de Velde, François Yvon, and Sabine Zerbian.  Breaking the unwritten language barrier: The bulb project.  _Procedia Computer Science_ , 81:8–14, 2016.  ISSN 1877-0509.  https://doi.org/10.1016/j.procs.2016.04.023.  URL https://www.sciencedirect.com/science/article/pii/S1877050916300370.  SLTU-2016 5th Workshop on Spoken Language Technologies for Under-resourced languages 09-12 May 2016 Yogyakarta, Indonesia.
  * Adelani et al. [2023] David Ifeoluwa Adelani, Marek Masiak, Israel Abebe Azime, Jesujoba Alabi, Atnafu Lambebo Tonja, Christine Mwase, Odunayo Ogundepo, Bonaventure F. P. Dossou, Akintunde Oladipo, Doreen Nixdorf, Chris Chinenye Emezue, sana al azzawi, Blessing Sibanda, Davis David, Lolwethu Ndolela, Jonathan Mukiibi, Tunde Ajayi, Tatiana Moteu, Brian Odhiambo, Abraham Owodunni, Nnaemeka Obiefuna, Muhidin Mohamed, Shamsuddeen Hassan Muhammad, Teshome Mulugeta Ababu, Saheed Abdullahi Salahudeen, Mesay Gemeda Yigezu, Tajuddeen Gwadabe, Idris Abdulmumin, Mahlet Taye, Oluwabusayo Awoyomi, Iyanuoluwa Shode, Tolulope Adelani, Habiba Abdulganiyu, Abdul-Hakeem Omotayo, Adetola Adeeko, Abeeb Afolabi, Anuoluwapo Aremu, Olanrewaju Samuel, Clemencia Siro, Wangari Kimotho, Onyekachi Ogbu, Chinedu Mbonu, Chiamaka Chukwuneke, Samuel Fanijo, Jessica Ojo, Oyinkansola Awosan, Tadesse Kebede, Toadoum Sari Sakayo, Pamela Nyatsine, Freedmore Sidume, Oreen Yousuf, Mardiyyah Oduwole, Tshinu Tshinu, Ussen Kimanuka, Thina Diko, Siyanda Nxakama, Sinodos Nigusse, Abdulmejid Johar, Shafie Mohamed, Fuad Mire Hassan, Moges Ahmed Mehamed, Evrard Ngabire, Jules Jules, Ivan Ssenkungu, and Pontus Stenetorp.  MasakhaNEWS: News Topic Classification for African Languages, 2023.
  * Agha [2006] Asif Agha.  _Language and Social Relations_.  Studies in the Social and Cultural Foundations of Language. Cambridge University Press, 2006.
  * Aharoni et al. [2019] Roee Aharoni, Melvin Johnson, and Orhan Firat.  Massively multilingual neural machine translation, 2019.
  * Ahia et al. [2021] Orevaoghene Ahia, Julia Kreutzer, and Sara Hooker.  The low-resource double bind: An empirical study of pruning for low-resource machine translation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pp. 3316–3333, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  10.18653/v1/2021.findings-emnlp.282.  URL https://aclanthology.org/2021.findings-emnlp.282.
  * AhmadMustafa [2023a] AhmadMustafa.  Urdu-Instruct-News-Article-Generation.  https://huggingface.co/datasets/AhmadMustafa/Urdu-Instruct-News-Article-Generation, 2023a.  Accessed: 2023-11-28.
  * AhmadMustafa [2023b] AhmadMustafa.  Urdu-Instruct-News-Category-Classification.  https://huggingface.co/datasets/AhmadMustafa/Urdu-Instruct-News-Category-Classification, 2023b.  Accessed: 2023-11-28.
  * AhmadMustafa [2023c] AhmadMustafa.  Urdu-Instruct-News-Headline-Generation.  https://huggingface.co/datasets/AhmadMustafa/Urdu-Instruct-News-Headline-Generation, 2023c.  Accessed: 2023-11-28.
  * AI Tamil Nadu [2023a] AI Tamil Nadu.  Tamil stories.  https://huggingface.co/datasets/aitamilnadu/tamil_stories, 2023a.  Accessed: 2023-12-15.
  * AI Tamil Nadu [2023b] AI Tamil Nadu.  Thirukkural Instruct.  https://huggingface.co/datasets/aitamilnadu/thirukkural_instruct, 2023b.  Accessed: 2023-11-30.
  * Akiki et al. [2022] Christopher Akiki, Giada Pistilli, Margot Mieskes, Matthias Gallé, Thomas Wolf, Suzana Ilić, and Yacine Jernite.  Bigscience: A case study in the social construction of a multilingual large language model, 2022.
  * Allal et al. [2023] Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al.  Santacoder: don’t reach for the stars!  _arXiv preprint arXiv:2301.03988_ , 2023.
  * AlShikh et al. [2023] Waseem AlShikh, Manhal Daaboul, Kirk Goddard, Brock Imel, Kiran Kamble, Parikshith Kulkarni, and Melisa Russak.  Becoming self-instruct: introducing early stopping criteria for minimal instruct tuning.  _arXiv preprint arXiv:2307.03692_ , 2023.
  * Amirkhani et al. [2023] Hossein Amirkhani, Mohammad AzariJafari, Soroush Faridan-Jahromi, Zeinab Kouhkan, Zohreh Pourjafari, and Azadeh Amirak.  Farstail: A Persian natural language inference dataset.  _Soft Computing_ , 2023.  10.1007/s00500-023-08959-3.
  * Andress et al. [2020] Lauri Andress, Tristen Hall, Sheila Davis, Judith Levine, Kimberly Cripps, and Dominique Guinn.  Addressing power dynamics in community-engaged research partnerships.  _Journal of Patient-Reported Outcomes 4: 24_ , 2020.
  * Arefeen et al. [2023] Md Adnan Arefeen, Biplob Debnath, and Srimat Chakradhar.  Leancontext: Cost-efficient domain-specific question answering using llms.  _arXiv preprint arXiv:2309.00841_ , 2023.
  * Artetxe et al. [2019] Mikel Artetxe, Sebastian Ruder, and Dani Yogatama.  On the cross-lingual transferability of monolingual representations.  _CoRR_ , abs/1910.11856, 2019.
  * Avle et al. [2018] Seyram Avle, Emmanuel Quartey, and David Hutchful.  Research on mobile phone data in the global south.  _The Oxford Handbook of Networked Communication_ , 2018.  URL https://api.semanticscholar.org/CorpusID:168167342.
  * Awati & Shum [2015] Kailash Awati and Simon Buckingham Shum.  Big data metaphors we live by.  _Towards Data Science_ , 2015.  URL https://towardsdatascience.com/big-data-metaphors-we-live-by-98d3fa44ebf8.
  * Azunre et al. [2021] Paul Azunre, Salomey Osei, Salomey Afua Addo, Lawrence Asamoah Adu-Gyamfi, Stephen Moore, Bernard Adabankah, Bernard Opoku, Clara Asare-Nyarko, Samuel Nyarko, Cynthia Amoaba, Esther Dansoa Appiah, Felix Akwerh, Richard Nii Lante Lawson, Joel Budu, Emmanuel Debrah, Nana Adowaa Boateng, Wisdom Ofori, Edwin Buabeng-Munkoh, Franklin Adjei, Isaac. K. E. Ampomah, Joseph Otoo., Reindorf Nartey Borkor, Standylove Birago Mensah, Lucien Mensah, Mark Amoako Marcel, Anokye Acheampong Amponsah, and James Ben Hayfron-Acquah.  Nlp for ghanaian languages.  _ArXiv_ , abs/2103.15475, 2021.  URL https://api.semanticscholar.org/CorpusID:232404908.
  * Bach et al. [2022] Stephen H Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, et al.  Promptsource: An integrated development environment and repository for natural language prompts.  _arXiv preprint arXiv:2202.01279_ , 2022.
  * Barrault et al. [2023] Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, et al.  Seamlessm4t-massively multilingual & multimodal machine translation.  _arXiv preprint arXiv:2308.11596_ , 2023.
  * Bartolo et al. [2020] Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp.  Beat the ai: Investigating adversarial human annotation for reading comprehension.  _Transactions of the Association for Computational Linguistics_ , 8:662–678, 2020.  10.1162/tacl_a_00338.  URL https://doi.org/10.1162/tacl_a_00338.
  * Bastanfard et al. [2023] Azam Bastanfard, Mohammad Shahabipour, and Dariush Amirkhani.  Crowdsourcing of labeling image objects: an online gamification application for data collection.  _Multimedia Tools and Applications_ , Aug 2023.  ISSN 1573-7721.  10.1007/s11042-023-16325-6.  URL https://doi.org/10.1007/s11042-023-16325-6.
  * Beck et al. [2022] Susanne Beck, Carsten Bergenholtz, Marcel Bogers, Tiare-Maria Brasseur, Marie Louise Conradsen, Diletta Di Marco, Andreas P. Distel, Leonhard Dobusch, Daniel Dörler, Agnes Effert, Benedikt Fecher, Despoina Filiou, Lars Frederiksen, Thomas Gillier, Christoph Grimpe, Marc Gruber, Carolin Haeussler, Florian Heigl, Karin Hoisl, Katie Hyslop, Olga Kokshagina, Marcel LaFlamme, Cornelia Lawson, Hila Lifshitz-Assaf, Wolfgang Lukas, Markus Nordberg, Maria Theresa Norn, Marion Poetz, Marisa Ponti, Gernot Pruschak, Laia Pujol Priego, Agnieszka Radziwon, Janet Rafner, Gergana Romanova, Alexander Ruser, Henry Sauermann, Sonali K. Shah, Jacob F. Sherson, Julia Suess-Reyes, Christopher L. Tucci, Philipp Tuertscher, Jane Bjørn Vedel, Theresa Velden, Roberto Verganti, Jonathan Wareham, Andrea Wiggins, and Sunny Mosangzi Xu.  The open innovation in science research field: a collaborative conceptualisation approach.  _Industry and Innovation_ , 29(2):136–185, 2022.  10.1080/13662716.2020.1792274.  URL https://doi.org/10.1080/13662716.2020.1792274.
  * Berant et al. [2013] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang.  Semantic parsing on Freebase from question-answer pairs.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pp. 1533–1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.  URL https://aclanthology.org/D13-1160.
  * Bernstein et al. [2015] Michael S. Bernstein, Greg Little, Robert C. Miller, Björn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell, and Katrina Panovich.  Soylent: a word processor with a crowd inside.  _Commun. ACM_ , 58(8):85–94, 2015.  10.1145/2791285.  URL https://doi.org/10.1145/2791285.
  * Bird [2022] Steven Bird.  Local languages, third spaces, and other high-resource scenarios.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 7817–7829, Dublin, Ireland, May 2022. Association for Computational Linguistics.  10.18653/v1/2022.acl-long.539.  URL https://aclanthology.org/2022.acl-long.539.
  * Birhane et al. [2022] Abeba Birhane, William Isaac, Vinodkumar Prabhakaran, Mark Diaz, Madeleine Clare Elish, Iason Gabriel, and Shakir Mohamed.  Power to the people? opportunities and challenges for participatory ai.  In _Equity and Access in Algorithms, Mechanisms, and Optimization_ , EAAMO ’22. ACM, October 2022.  10.1145/3551624.3555290.  URL http://dx.doi.org/10.1145/3551624.3555290.
  * Bisk et al. [2020] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi.  Piqa: Reasoning about physical commonsense in natural language.  In _Thirty-Fourth AAAI Conference on Artificial Intelligence_ , 2020.
  * Blaschke et al. [2023] Verena Blaschke, Hinrich Schuetze, and Barbara Plank.  A survey of corpora for Germanic low-resource languages and dialects.  In _Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)_ , pp. 392–414, Tórshavn, Faroe Islands, May 2023. University of Tartu Library.  URL https://aclanthology.org/2023.nodalida-1.41.
  * Blodgett et al. [2016] Su Lin Blodgett, Lisa Green, and Brendan O’Connor.  Demographic dialectal variation in social media: A case study of African-American English.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pp. 1119–1130, Austin, Texas, November 2016. Association for Computational Linguistics.  10.18653/v1/D16-1120.  URL https://aclanthology.org/D16-1120.
  * Botha et al. [2018] Jan A. Botha, Manaal Faruqui, John Alex, Jason Baldridge, and Dipanjan Das.  Learning to split and rephrase from Wikipedia edit history.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pp. 732–737, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.  10.18653/v1/D18-1080.  URL https://aclanthology.org/D18-1080.
  * Boubdir et al. [2023] Meriem Boubdir, Edward Kim, Beyza Ermis, Marzieh Fadaee, and Sara Hooker.  Which prompts make the difference? data prioritization for efficient human llm evaluation, 2023.
  * Bourdieu [1987] Pierre Bourdieu.  What makes a social class? on the theoretical and practical existence of groups.  _Berkeley journal of sociology_ , 32:1–17, 1987.
  * Bowser et al. [2020] Anne Bowser, Caren Cooper, Alex De Sherbinin, Andrea Wiggins, Peter Brenton, Tyng-Ruey Chuang, Elaine Faustman, Mordechai Haklay, and Metis Meloche.  Still in need of norms: the state of the data in citizen science.  _Citizen Science: Theory and Practice_ , 5(1), 2020.
  * Brown & Gilman [1968] Roger Brown and Albert Gilman.  _THE PRONOUNS OF POWER AND SOLIDARITY_ , pp. 252–275.  De Gruyter Mouton, Berlin, Boston, 1968.  ISBN 9783110805376.  doi:10.1515/9783110805376.252.  URL https://doi.org/10.1515/9783110805376.252.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * Cahyawijaya et al. [2023] Samuel Cahyawijaya, Holy Lovenia, Alham Fikri Aji, Genta Winata, Bryan Wilie, Fajri Koto, Rahmad Mahendra, Christian Wibisono, Ade Romadhony, Karissa Vincentio, et al.  Nusacrowd: Open source initiative for indonesian nlp resources.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pp. 13745–13818, 2023.
  * Carlini et al. [2023] Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr.  Poisoning web-scale training datasets is practical, 2023.
  * Castaldo [2023] Joe Castaldo.  “AI chatbots fall short in dozens of languages. A non-profit project aims to fix that".  _Globe & Mail_, 2023.
  * Castaldo [September 16, 2023] Joe Castaldo.  Meet the gig workers making ai machines more accurate, capable and powerful.  _The Globe and Mail_ , September 16, 2023.
  * Caswell et al. [2020] Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna.  Language ID in the wild: Unexpected challenges on the path to a thousand-language web text corpus.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pp. 6588–6608, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.  10.18653/v1/2020.coling-main.579.  URL https://aclanthology.org/2020.coling-main.579.
  * Chen et al. [2023a] Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al.  Alpagasus: Training a better alpaca with fewer data.  _arXiv preprint arXiv:2307.08701_ , 2023a.
  * Chen et al. [2023b] Pinzhen Chen, Shaoxiong Ji, Nikolay Bogoychev, Barry Haddow, and Kenneth Heafield.  Monolingual or multilingual instruction tuning: Which makes a better alpaca, 2023b.
  * Chen et al. [2023c] Sanxing Chen, Yongqiang Chen, and Börje F. Karlsson.  Dataset and baseline system for multi-lingual extraction and normalization of temporal and numerical expressions.  _arXiv preprint arXiv:2303.18103_ , 2023c.
  * Chung et al. [2022] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ , 2022.
  * Chung et al. [2023] John Chung, Ece Kamar, and Saleema Amershi.  Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, 2023.  10.18653/v1/2023.acl-long.34.  URL http://dx.doi.org/10.18653/v1/2023.acl-long.34.
  * Clanuwat et al. [2018] Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha.  Deep learning for classical japanese literature.  _ArXiv_ , abs/1812.01718, 2018.  URL https://api.semanticscholar.org/CorpusID:54458639.
  * Clark et al. [2019] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  In _NAACL_ , 2019.
  * Clark et al. [2020] Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki.  Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages.  _Transactions of the Association for Computational Linguistics_ , 2020.
  * Clark et al. [2018] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv:1803.05457v1_ , 2018.
  * Conneau et al. [2018] Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov.  Xnli: Evaluating cross-lingual sentence representations.  _arXiv preprint arXiv:1809.05053_ , 2018.
  * Conover et al. [2023] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin.  Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM, 2023.  URL https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm.
  * ConseggioLigure [2023a] ConseggioLigure.  Lij news instruct ita-lij.  https://huggingface.co/datasets/ConseggioLigure/lijnews-instruct-ita-lij, 2023a.  Accessed: 2023-11-28.
  * ConseggioLigure [2023b] ConseggioLigure.  Lij news instruct lij-ita.  https://huggingface.co/datasets/ConseggioLigure/lijnews-instruct-lij-ita, 2023b.  Accessed: 2023-11-28.
  * ConseggioLigure [2023c] ConseggioLigure.  Seed instruct eng-lij.  https://huggingface.co/datasets/ConseggioLigure/seed-instruct-eng-lij, 2023c.  Accessed: 2023-11-28.
  * ConseggioLigure [2023d] ConseggioLigure.  Seed instruct lij-eng.  https://huggingface.co/datasets/ConseggioLigure/seed-instruct-lij-eng, 2023d.  Accessed: 2023-11-28.
  * Corbett et al. [2023] Eric Corbett, Emily Denton, and Sheena Erete.  Power and public participation in ai.  In _Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization_ , EAAMO ’23, New York, NY, USA, 2023. Association for Computing Machinery.  ISBN 9798400703812.  10.1145/3617694.3623228.  URL https://doi.org/10.1145/3617694.3623228.
  * Crawford [2021] Kate Crawford.  _Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence_.  Yale University Press, New Haven, Connecticut, 2021.
  * Cui et al. [2019] Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu.  A span-extraction dataset for Chinese machine reading comprehension.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 5883–5889, Hong Kong, China, November 2019. Association for Computational Linguistics.  10.18653/v1/D19-1600.  URL https://aclanthology.org/D19-1600.
  * Cui et al. [2023] Yiming Cui, Ziqing Yang, and Xin Yao.  Efficient and effective text encoding for chinese llama and alpaca, 2023.
  * Dabre et al. [2020] Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan.  A survey of multilingual neural machine translation.  _ACM Comput. Surv._ , 53(5), sep 2020.  ISSN 0360-0300.  10.1145/3406095.  URL https://doi.org/10.1145/3406095.
  * Dac Lai et al. [2023] Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen.  Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback.  _arXiv e-prints_ , pp. arXiv–2307, 2023.
  * Dasigi et al. [2019] Pradeep Dasigi, Nelson F. Liu, Ana Marasovic, Noah A. Smith, and Matt Gardner.  Quoref: A reading comprehension dataset with questions requiring coreferential reasoning.  _arXiv:1908.05803v2_ , 2019.
  * de Franga et al. [2015] Flavio A. de Franga, Adriana S. Vivacqua, and Maria Luiza M. Campos.  Designing a gamification mechanism to encourage contributions in a crowdsourcing system.  In _2015 IEEE 19th International Conference on Computer Supported Cooperative Work in Design (CSCWD)_ , pp. 462–466, 2015.  10.1109/CSCWD.2015.7231003.
  * De Mauro et al. [2015] Andrea De Mauro, Marco Greco, and Michele Grimaldi.  What is big data? a consensual definition and a review of key research topics.  In _AIP conference proceedings_ , volume 1644, pp. 97–104. American Institute of Physics, 2015.
  * Delgado et al. [2023] Fernando Delgado, Stephen Yang, Michael Madaio, and Qian Yang.  The participatory turn in ai design: Theoretical foundations and the current state of practice.  _Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization_ , 2023.  URL https://api.semanticscholar.org/CorpusID:263605822.