# SEACrowd: A Multilingual Multimodal Data Hub
and Benchmark Suite for Southeast Asian Languages

Holy Lovenia★,1,2 Rahmad Mahendra★,3,2 Salsabil Maulana Akbar★,2 Lester James
V. Miranda★,4
Jennifer Santoso★,5 Elyanah Aco★,6 Akhdan Fadhilah★,7 Jonibek Mansurov★,8
Joseph Marvin Imperial★,9,10
Onno P. Kampman★,11 Joel Ruben Antony Moniz★,6 Muhammad Ravi Shulthan
Habibi★,3,2 Frederikus Hudi★,12,13
Railey Montalan★,1 Ryan Ignatius6 Joanito Agili Lopo14 William Nixon15 Börje
F. Karlsson16 James Jaya6
Ryandito Diandaru6 Yuze Gao6 Patrick Amadeus15 Bin Wang6 Jan Christian Blaise
Cruz17 Chenxi Whitehouse18
Ivan Halim Parmonangan19 Maria Khelli15 Wenyu Zhang6 Lucky Susanto20 Reynard
Adha Ryanda21
Sonny Lazuardi Hermawan22 Dan John Velasco17 Muhammad Dehan Al Kautsar15 Willy
Fitra Hendria6
Yasmin Moslem23 Noah Flynn24 Muhammad Farid Adilazuarda8 Haochen Li6 Johanes
Lee15 R. Damanhuri25
Shuo Sun6 Muhammad Reza Qorib26 Amirbek Djanibekov8 Wei Qi Leong1 Quyet V.
Do27 Niklas Muennighoff28
Tanrada Pansuwan18 Ilham Firdausi Putra6 Yan Xu29,27 Ngee Chia Tai1 Ayu
Purwarianti6,30
Sebastian Ruder31 William Tjhi1 Peerat Limkonchotiwat★,32 Alham Fikri Aji★,8
Sedrick Keh★,33
Genta Indra Winata★,2 Ruochen Zhang★,34 Fajri Koto★,8,2 Zheng-Xin Yong★,34
Samuel Cahyawijaya★,27,2
1AI Singapore 2IndoNLP 3Universitas Indonesia 4Allen Institute for Artificial
Intelligence 5RevComm, Inc.
6Independent Researcher 7Tohoku University 8MBZUAI 9University of Bath
10National University Philippines
11MOH Office for Healthcare Transformation (MOHT) 12NAIST 13Works Applications
Lab 14Universitas Gadjah Mada
15Institut Teknologi Bandung 16Beijing Academy of Artificial Intelligence
(BAAI) 17Samsung Research Philippines
18University of Cambridge 19Queensland University of Technology 20Monash
University Indonesia 21Imperial College London
22Independent Design Engineer 23Bering Lab 24Amazon 25Universitas Diponegoro
26NUS 27HKUST 28Contextual AI
29Huawei Noah’s Ark Lab 30Prosa.ai 31Cohere 32VISTEC 33Toyota Research
Institute 34Brown University
★Major contributors

###### Abstract

Southeast Asia (SEA) is a region rich in linguistic diversity and cultural
variety, with over 1,300 indigenous languages and a population of 671 million
people. However, prevailing AI models suffer from a significant lack of
representation of texts, images, and audio datasets from SEA, compromising the
quality of AI models for SEA languages. Evaluating models for SEA languages is
challenging due to the scarcity of high-quality datasets, compounded by the
dominance of English training data, raising concerns about potential cultural
misrepresentation. To address these challenges, we introduce SEACrowd, a
collaborative initiative that consolidates a comprehensive resource
hub111https://seacrowd.github.io/seacrowd-catalogue/ that fills the resource
gap by providing standardized corpora222https://github.com/SEACrowd/seacrowd-
datahub/ in nearly 1,000 SEA languages across three modalities. Through our
SEACrowd benchmarks, we assess the quality of AI models on 36 indigenous
languages across 13 tasks, offering valuable insights into the current AI
landscape in SEA. Furthermore, we propose strategies to facilitate greater AI
advancements, maximizing potential utility and resource equity for the future
of AI in SEA.

SEACrowd: A Multilingual Multimodal Data Hub
and Benchmark Suite for Southeast Asian Languages

Holy Lovenia★,1,2 Rahmad Mahendra★,3,2 Salsabil Maulana Akbar★,2 Lester James
V. Miranda★,4 Jennifer Santoso★,5 Elyanah Aco★,6 Akhdan Fadhilah★,7 Jonibek
Mansurov★,8 Joseph Marvin Imperial★,9,10 Onno P. Kampman★,11 Joel Ruben Antony
Moniz★,6 Muhammad Ravi Shulthan Habibi★,3,2 Frederikus Hudi★,12,13 Railey
Montalan★,1 Ryan Ignatius6 Joanito Agili Lopo14 William Nixon15 Börje F.
Karlsson16 James Jaya6 Ryandito Diandaru6 Yuze Gao6 Patrick Amadeus15 Bin
Wang6 Jan Christian Blaise Cruz17 Chenxi Whitehouse18 Ivan Halim Parmonangan19
Maria Khelli15 Wenyu Zhang6 Lucky Susanto20 Reynard Adha Ryanda21 Sonny
Lazuardi Hermawan22 Dan John Velasco17 Muhammad Dehan Al Kautsar15 Willy Fitra
Hendria6 Yasmin Moslem23 Noah Flynn24 Muhammad Farid Adilazuarda8 Haochen Li6
Johanes Lee15 R. Damanhuri25 Shuo Sun6 Muhammad Reza Qorib26 Amirbek
Djanibekov8 Wei Qi Leong1 Quyet V. Do27 Niklas Muennighoff28 Tanrada
Pansuwan18 Ilham Firdausi Putra6 Yan Xu29,27 Ngee Chia Tai1 Ayu
Purwarianti6,30 Sebastian Ruder31 William Tjhi1 Peerat Limkonchotiwat★,32
Alham Fikri Aji★,8 Sedrick Keh★,33 Genta Indra Winata★,2 Ruochen Zhang★,34
Fajri Koto★,8,2 Zheng-Xin Yong★,34 Samuel Cahyawijaya★,27,2 1AI Singapore
2IndoNLP 3Universitas Indonesia 4Allen Institute for Artificial Intelligence
5RevComm, Inc. 6Independent Researcher 7Tohoku University 8MBZUAI 9University
of Bath 10National University Philippines 11MOH Office for Healthcare
Transformation (MOHT) 12NAIST 13Works Applications Lab 14Universitas Gadjah
Mada 15Institut Teknologi Bandung 16Beijing Academy of Artificial Intelligence
(BAAI) 17Samsung Research Philippines 18University of Cambridge 19Queensland
University of Technology 20Monash University Indonesia 21Imperial College
London 22Independent Design Engineer 23Bering Lab 24Amazon 25Universitas
Diponegoro 26NUS 27HKUST 28Contextual AI 29Huawei Noah’s Ark Lab 30Prosa.ai
31Cohere 32VISTEC 33Toyota Research Institute 34Brown University ★Major
contributors

##  1 Introduction

Despite the Southeast Asia (SEA) region being home to 1,300 indigenous
languages (18% of the world’s languages) and 671 million people (8.75% of the
world’s population), the representation of texts, images, and audio datasets
from this region is significantly lacking in machine learning pre-training
models. This deficiency negatively impacts the model quality for SEA
languages. The language coverage of SEA languages in two common pre-training
resources, Common Crawl333https://commoncrawl.github.io/cc-crawl-
statistics/plots/languages and C4 Xue et al. (2021), is extremely limited,
with only 2.36% (in 11 languages) and 10.62% (in 11 languages), respectively.
In modalities beyond text, the representation is even more limited. For
instance, Common Voice, one of the largest multilingual speech corpora,
includes only 6 SEA indigenous languages Conneau et al. (2021); Ardila et al.
(2020), and LAION-5B, one of the largest multilingual vision-language (VL)
corpora, includes only 12 SEA indigenous languages Schuhmann et al. (2022).
While datasets for other SEA indigenous languages may exist, they are often
scattered, insufficiently documented, or varied in quality and formatting,
thereby making access and usage challenging Cahyawijaya et al. (2023a); Joshi
et al. (2020); Aji et al. (2023).

In terms of evaluation, the sparse availability of high-quality test tasks for
these languages also complicates evaluating models for SEA languages. Despite
there being 1,300+ languages in the SEA region, prior works Winata et al.
(2023); Cahyawijaya et al. (2021); Koto and Koto (2020); Zhang et al. (2024);
Wang et al. (2024); Nguyen et al. (2023); Leong et al. (2023); Yong et al.
(2023) have only evaluated fewer than 10 SEA languages collectively. The
actual performance of current models on most SEA languages remains largely
unknown.

Moreover, the dominance of Anglocentric training data potentially results in
cultural bias when generating texts, images, or audio in underrepresented SEA
languages Søgaard (2022); Talat et al. (2022). Further, Durmus et al. (2023);
AlKhamissi et al. (2024); Cahyawijaya et al. (2024a) have shown that the
learned representations in LMs often fail to reflect local cultural values in
SEA Koto et al. (2024); Liu et al. (2024); Adilazuarda et al. (2024). This
raises concerns about the ability of current LLMs to generate natural, high-
quality texts for this region. Furthermore, the discrepancy in language
support creates language barriers in technological access and risks
marginalizing minority groups who do not speak the dominant language.

Figure 1: Mapping between tasks, schemas, modalities, and language regions
across 498 datasheets in SEACrowd.

In this work, we investigate the current AI progress for SEA languages by
addressing the challenges of resources, evaluation, and generation quality.
Our contributions are three-fold:

  * •

We bridge the resource gap by centralizing and standardizing $\sim$500 corpora
in nearly 1,000 SEA languages in SEACrowd, a comprehensive and standardized
resource center, across 3 modalities: text, image, and audio.

  * •

We close the evaluation gap in SEA languages with the SEACrowd Benchmarks,
which cover 38 SEA indigenous languages on 13 tasks across 3 modalities,
providing insights into the performance of a diverse spectrum of AI models.
Further, our study reveals that the generative outputs of existing LLMs
exhibit a closer resemblance to translationese rather than natural data in 9
SEA languages.

  * •

We offer insights and strategies for the future development of AI in SEA,
aiming to promote a sustainable and prosperous future through continuous AI
advancements.

##  2 SEACrowd

SEACrowd represents the first comprehensive AI dataset collection initiative
for SEA, developed through a collaborative effort among researchers and
engineers primarily based in the SEA region. As addressed in §1, resource
scarcity and the scattered nature of the data are crucial challenges in SEA.
SEACrowd addresses these issues through two primary contributions: 1)
consolidating datasheets to enhance data discoverability; and 2) standardizing
dataloaders for easier use, especially in multiple dataset loading. We also
follow data provenance practices (Longpre et al., 2023) to preserve the
proprietary rights of dataset owners.

#### Consolidating datasheets

We invited contributors to submit datasheet forms for publicly available
datasets across all modalities including text, audio, and image in SEA
languages and/or cultures. These datasheets include detailed information about
each dataset, such as data subset(s), description, task, language, license,
URL access, annotation method(s), annotation validation, relevant
publications, publication venue, and data splits. For each submission, we
manually verify and correct it as necessary to ensure datasheet accuracy.

#### Standardizing dataloaders

For each approved datasheet, we created a standardized dataloader wrapper to
facilitate ready-to-use data access since only 38.4% of the consolidated data
sources were originally hosted on Hugging Face444https://huggingface.co/. To
support diverse task types, we carefully designed the standardized seacrowd
schema to support different data structures and modalities (see Appendix F).
We also adhere to data provenance practices (Longpre et al., 2023) and
document the relevant metadata (e.g., license) in the dataloaders.
Furthermore, we engaged with data owners and successfully converted three
private datasets into public ones.

These efforts have culminated in 498 datasheets in SEACrowd Catalogue and 399
dataloaders in SEACrowd Data Hub (§2.1). Notably, our centralized data
repository covers $\sim$1,000 SEA languages, underscoring the extensive
linguistic diversity captured by SEACrowd. We elaborate on the SEACrowd
dataset statistics in §2.2. SEACrowd’s contribution guidelines, progression
details, and reviewing procedure are in Appendix C, D, and E.

###  2.1 SEACrowd Catalogue & Data Hub

SEACrowd comprises two interconnected platforms: SEACrowd Catalogue555SEACrowd
Catalogue is also present in csv format. and SEACrowd Data Hub. These
platforms work in tandem to consolidate the datasheet submissions and provide
a standardized pipeline for SEACrowd. Specifically, Catalogue houses the
datasheets (metadata), while Data Hub stores the standardized dataloaders and
the seacrowd library666All codes are available under Apache License 2.0. for
the schemas and configurations (Appendix F). These systems share information
on the datasheets and dataloaders, allowing users to seamlessly explore and
utilize them.

(a) NLU evaluation

(b) NLG evaluation

Figure 2: Zero-shot model performance across NLU and NLG tasks in SEA
languages.

Model |  Gini $\downarrow$
---|---
Commercial |
GPT-4 |  0.201
Command-R |  0.184
English |
Mistral |  0.159
Llama3 |  0.131
Falcon |  0.238
Multilingual |
mT0 |  0.131
BLOOMZ |  0.228
BactrianX-Llama |  0.163
AYA-23 |  0.183
AYA-101 |  0.095
SEA regional |
SEA-LION |  0.204
SeaLLM v2.5 |  0.116
Sailor |  0.145
SEA country |
Cendol-mT5 |  0.378
Cendol-Llama2 |  0.267
Merak v4 |  0.199
WangchanX-Llama3 |  0.153
Malaysian Llama3 |  0.179

Table 1: Language equity across baselines based on Gini coefficient weighted
by population ($\tau=0.5$).

###  2.2 Datasets in SEACrowd

SEACrowd consolidates 498 datasheets with diverse tasks in SEA languages and
provides standardized access through dataloaders to 399 of them. As shown in
Figure 1, approximately 81% of the datasets in SEACrowd are textual data, with
the remaining $\sim$8% and $\sim$11% being VL and speech, respectively. The
complete list of SEA indigenous languages covered by SEACrowd and their
mapping to the relevant SEA regions are provided in Appendix K. Around
$\sim$53% of the datasets have a commercially permissive license.

A total of 83 tasks are provided in SEACrowd with a breakdown of 66 in NLP
(e.g., abusive language detection, intent classification, instruction tuning,
named entity recognition, etc.), 10 in VL (image-to-text generation, sign
language recognition, video captioning, etc.), and 7 in speech (e.g.,
automatic speech recognition, text-to-speech, speech emotion recognition, and
others). These tasks are then standardized into 20 dataloader schemas
described in Appendix F. Further discussion regarding resources in SEACrowd is
in §5.1.

##  3 SEACrowd Benchmarks

To understand the capability of state-of-the-art models, we conduct
comprehensive evaluations of existing LLMs, VLMs, and speech models from
various architectures and training approaches. To construct a benchmark
suite777https://github.com/SEACrowd/seacrowd-experiments, we select a subset
of the dataset that has been manually annotated and/or validated from the data
presented in §2.2. More details regarding the data subsets, baselines, and
prompts used for the evaluations are given in Appendix G.1, G.2, and G.3.

###  3.1 Datasets

#### NLP

Our natural language understanding (NLU) benchmark consists of 131 data
subsets and 7 tasks: sentiment analysis, topic classification, natural
language inference (NLI), commonsense reasoning, exam-style multiple-choice
question answering (QA), culture understanding, and reading comprehension. It
covers English (eng) and 33 SEA indigenous languages.

We utilize 100 data subsets for the natural language generation (NLG)
benchmark, which covers machine translation (MT) between English and SEA
languages from both directions, summarization, as well as extractive or
abstractive question answering, covering 27 SEA indigenous languages.

#### Speech

We employ 19 automatic speech recognition (ASR) data subsets to evaluate the
capability of speech models in 15 SEA indigenous languages.

#### VL

We assess the models on image captioning using four data subsets in 4 SEA
indigenous languages, i.e., Filipino (fil), Indonesian (ind), Thai (tha), and
Vietnamese (vie). This disparity in the evaluation scale is due to the fact
that only a few datasets in SEACrowd are VL datasets, and even fewer are
annotated by humans.

###  3.2 Baselines

Complete details regarding the model architectures, model sizes, seen
languages, corresponding publications, and other aspects are in Appendix G.2.

#### NLP

To evaluate the zero-shot performance of instruction-tuned LLMs on SEA
languages, we benchmark two commercial, i.e., GPT-4 OpenAI et al. (2024) and
Command-R888https://docs.cohere.com/docs/command-r, and 17 open-source
baselines, the majority of which are $\sim$7B-13B parameters. We categorize
the open-source baselines according to the language(s) coverage in pre-
training and/or instruction tuning, i.e., 1) English: Llama3 Touvron et al.
(2023), Mistral Jiang et al. (2023), and Falcon Almazrouei et al. (2023); 2)
Multilingual: AYA-101, AYA-23 Üstün et al. (2024), mT0, BLOOMZ Muennighoff et
al. (2022), and BactrianX-Llama Li et al. (2023a); 3) SEA regional: SEA-LION
Singapore (2023), Sailor Dou et al. (2024), and SeaLLM Nguyen et al. (2023);
and 4) SEA country-specific: Cendol-mT5, Cendol-Llama2 Cahyawijaya et al.
(2024b), Merak Ichsan (2023) from Indonesia, WangchanX-Llama3 Phatthiyaphaibun
et al. (2024) from Thailand, and Malaysian-
Llama3999https://huggingface.co/mesolitica/malaysian-llama-3-8b-instruct-16k
from Malaysia.

#### Speech

We evaluate the zero-shot performances of state-of-the-art multilingual pre-
trained speech models in transcribing speech in SEA languages. Specifically,
we consider Whisper v3 Radford et al. (2023), MMS 1B Pratap et al. (2024), and
Seamless M4T v2 Communication et al. (2023), which have shown proficiency in
accurately transcribing multiple languages without fine-tuning. Additionally,
we include models that are fine-tuned on specific language(s), SEA or English,
based on 1) Wav2Vec2 XLSR Conneau et al. (2021) and 2) XLS-R Babu et al.
(2021), known for their cross-lingual speech representation learning by pre-
training on raw speech waveforms across diverse languages, with XLS-R offering
broader language coverage, and 3) Whisper, which leverages weakly supervised
pre-training on spectrograms of speech in diverse languages. The specific
fine-tuned models are evaluated: XLSR on ind, jav, sun; XLSR and Whisper on
Indonesian (ind); XLSR and Whisper on Thai (tha); XLS-R on Tagalog (tgl);
XLS-R on Burmese (mya); XLS-R and Whisper on Khmer (khm); and XLSR on English
(eng). See Appendix G.2 for details.

#### VL

We consider state-of-the-art VLMs primarily trained on English pre-training
and instruction-following data: LLaVA Liu et al. (2023b, a), InstructBLIP Dai
et al. (2024), and Idefics2 Laurençon et al. (2024), and VLMs trained in a
multilingual manner: mBLIP Geigle et al. (2023) and PaliGemma Gemma Team et
al. (2024), to assess their image captioning ability in SEA languages.

###  3.3 Experimental Settings

We conduct all evaluations in a zero-shot fashion. We employ 3 prompt
templates in English for each NLU task and 1 for each NLG task. We utilize the
weighted F1 score to measure the model performance on NLU tasks and n-gram
reference-based metrics, i.e., chrF++ Popović (2015, 2017) and ROUGE-L Lin
(2004), on NLG tasks. As for VL, aside from a prompt template in English, we
also use a prompt template in the respective SEA indigenous language per data
subset. We report CIDEr Vedantam et al. (2015) for the image captioning task.
For ASR, we use word error rate (WER) for languages with Latin script and
character error rate (CER) for those with non-Latin script.

##  4 Result & Analysis

Figure 3: Speech model error rate (%$\downarrow$) across existing ASR tasks in
SEA languages.

###  4.1 State-of-the-Art Models on SEA languages

#### LLMs

Figure 2(a) and 2(b) illustrate the overall model performance of the LLM
baselines in SEA languages for both NLU tasks and NLG tasks. In our NLU
evaluation, AYA-101, a large multilingual instruction-tuned language model
covering 101 languages, demonstrates the best zero-shot performance. It is
followed by the commercial baselines, which achieve a median of $\sim$0.6
weighted F1-score. Sailor and SeaLLM, models specifically trained with SEA
languages, also display competitive performance. Similarly, mT0 exhibits
strong generalization abilities due to its exposure to $\sim$100 languages in
pre-training, including those from the SEA region Muennighoff et al. (2022).
In contrast, most English and SEA country-specific baselines perform less
effectively, likely due to their narrow focus on English or a limited set of
SEA languages, such as Indonesian languages for Cendol and Thai for WangchanX-
Llama3. Similar and consistent trends are observed for the NLG tasks, except
for GPT-4’s poorer performance across summarization, abstractive or extractive
QA, and MT, indicating its ineffectiveness in producing acceptable outputs in
SEA languages for these tasks. Appendix G.4 describes the performance of LLMs
per language.

To analyze the equality in model performance across SEA languages, following
Khanuja et al. (2023), we utilize the Gini coefficient—originally used to
observe income equality Dorfman (1979)—weighted by demand and parameterized by
$\tau$. Here, $\tau=1$ corresponds to a demographic notion of demand,
considering language population size, while $\tau=0$ does not take population
size into account Blasi et al. (2022). Table 1 shows that models trained on
more SEA languages, such as multilingual and SEA regional baselines, generally
exhibit greater language equity. For instance, although Command-R and GPT-4
are competitive performance-wise against AYA-101 and mT0, AYA-101 and mT0
demonstrate higher equality across all SEA languages under study. This trend
is consistent across different $\tau$ (see Appendix G.5).

Figure 4: Existing VLMs produce subpar image captions in SEA languages.

#### Speech models

Figure 3 presents the off-the-shelf speech model performance on ASR across
languages in SEA, measured by the error rate percentage. 9 of the 15 SEA
languages in our speech evaluation belong to the Austronesian language family.
The other 6 are khm and vie, which belong to Austro-Asiatic, cnh and mya
belong to Sino-Tibetan, as well as tha and vie belong Kra-Dai language family.
Generally, the multilingual pre-trained baselines have a competitive
generalization capability across the languages, although it varies by
language. For instance, Whisper v3 demonstrates significantly higher
effectiveness for national languages such as ind, zlm, fil, tha, and vie,
while performing less optimally for other indigenous languages. Conversely,
Seamless M4T v2 shows a more balanced performance across the languages.
Regarding fine-tuned baselines, error rates decrease for their seen languages.
The fine-tuned Whisper models, however, manage to better optimize for the
target language while retaining their original capabilities in other SEA
languages compared to their Wav2Vec2 XLSR and XLS-R counterparts, despite both
having been pre-trained in a multilingual manner. This observation aligns with
the findings of Rouditchenko et al. (2023), who find that the number of hours
seen per language and language family during pre-training is predictive of how
the models compare, in which Whisper’s pre-training data duration for these
four language families exceeds that of XLSR.

#### VLMs

Figure 4 depicts the zero-shot performance of off-the-shelf VLMs on image
captioning in SEA indigenous languages. Despite the capability of LLMs for
zero-shot cross-lingual generalization Huang et al. (2021); Täckström et al.
(2012); Neubig and Hu (2018); Artetxe et al. (2020), VLMs trained only in
English (i.e., InstructBLIP, LLaVA, and Idefics2) fail to exhibit this
capability, struggling to generate adequate image captions in SEA languages.
Multilingual VL pre-training is crucial to achieving aligned multilingual
representations Burns et al. (2020); Li et al. (2023b); Huang et al. (2021).
For instance, PaliGemma and mBLIP generate better image captions in tha and
fil when prompted in the relevant SEA languages.

Model |  Natural outputs
---|---
SEA-LION |  58.57%
AYA-23 |  43.57%
Sailor |  37.86%
Cendol-Llama2 |  37.37%
Malaysian Llama3 |  36.90%
WangchanX-Llama3 |  30.24%
Falcon |  29.52%
BactrianX-Llama |  28.10%
SeaLLM |  27.38%
Merak |  26.19%
BLOOMZ |  25.00%
Cendol-MT5 |  24.05%
Command-R |  20.95%
mT0-XL |  19.76%
Mistral |  19.52%
GPT-4 |  16.67%
Llama3 |  14.05%
AYA-101 |  8.33%

(a) Avg. by models

Language |  Natural outputs
---|---
Indonesian (ind) |  41.58%
Vietnamese (vie) |  37.31%
Thai (tha) |  34.21%
Khmer (khm) |  29.21%
Lao (lao) |  28.42%
Malay (zlm) |  22.24%
Burmese (mya) |  19.47%
Filipino (fil) |  12.22%
English (eng)† |  8.95%

(b) Avg. by languages

Table 2: Current LLMs are still incapable of generating natural texts in SEA
languages. †As spoken in SEA regions, not worldwide.

(a) Language coverage

(b) Annotation quality

(c) Cultural relevance

Figure 5: The resource gap in SEA in terms of language coverage, annotation
quality, and cultural relevance.

However, when prompted in eng, the performance of these multilingual baselines
varies notably. PaliGemma’s performance diminishes completely, while mBLIP’s
performance shows both increases and decreases across different SEA languages.
This raises the question of whether the multilingual VLMs can maintain
consistent performance across different languages used in the instructions and
the tasks. It highlights the need for further research into the mechanisms
that drive these variations and how to achieve robust multilingual performance
in VLMs across diverse linguistic contexts. Understanding these dynamics is
crucial for improving VLMs’ generalization capabilities and ensuring equitable
performance across all languages, despite most related works focusing on
monolingual visual instruction tuning Liu et al. (2023b); Gong et al. (2023);
Zhu et al. (2024).

###  4.2 Generation Quality in SEA Languages: Translationese vs. Natural

#### Classifying Translationese in SEA Languages

To analyze the generation quality of LLMs in SEA languages, we build a text
classifier to discriminate between translationese and natural texts Riley et
al. (2020). We construct a translationese classification training and testing
dataset using 49 and 62 data subsets, respectively, covering approximately
39.9k and 51.5k sentences across 9 SEA languages: English (eng), Indonesian
(ind), Khmer (khm), Lao (lao), Burmese (mya), Filipino (fil), Thai (tha),
Vietnamese (vie), and Malay (zlm). The training and test data are detailed in
Appendix H.1.

We fine-tune a classifier from mDeBERTaV3 He et al. (2020,
2022)101010https://huggingface.co/microsoft/mdeberta-v3-base using these data
and achieve 79.08% accuracy on the test set in predicting translationese
across these 9 languages. The detailed results and ablation studies of our
translationese classifier experiments are provided in Appendix H.2. This
classifier enables us to assess the generation quality of LLMs by
distinguishing between translationese and naturally occurring text, providing
insights into the models’ performance in producing authentic language output.

#### Generation Quality of LLMs

We evaluate the generation quality of LLMs in 9 SEA languages by generating
answers to natural, general, and safety questions from Sea-Bench Nguyen et al.
(2023). As shown in Table 2(a), LLMs with extensive language coverage but less
focus on SEA languages, e.g., AYA-101 Üstün et al. (2024), GPT-4 OpenAI et al.
(2024), mT0 Muennighoff et al. (2023); Xue et al. (2021), and Llama3 AI@Meta
(2024), tend to produce natural sentences less than 20% of the time. In
contrast, models with narrower language coverage but a greater focus on SEA
languages, such as Cendol-Llama2 Cahyawijaya et al. (2024b), Sailor Dou et al.
(2024), AYA-23 Aryabumi et al. (2024), and SEA-LION Singapore (2023), generate
more natural sentences over 35% of the time.

However, even the LLM with the least translationese generation, SEA-LION, only
produces natural SEA sentences 57.71% of the time, highlighting a significant
quality gap in generating natural sentences in SEA languages. As displayed in
Table 2(b), the translationese issue varies across SEA languages. Languages
such as Tagalog (tgl), Burmese (mya), and Malay (zlm) have more severe
translationese problems, with existing LLMs producing natural sentences only
11.58%, 19.47%, and 22.24% of the time, respectively. This underscores the
need for further improvements in LLMs to more effectively address the
linguistic diversity and complexity of SEA languages.

##  5 Discussions

###  5.1 Resource Gaps in SEA

#### Coverage

SEACrowd covers 980 out of the 1,308 languages spoken in SEA (74.9%). Despite
this high coverage, language representation in SEACrowd exhibits a very long-
tail distribution, with over 700 languages having only 1 or 2 datasets, and
only 23 languages having 20 datasets or more. These less represented languages
typically exist only in the form of lexicons Asgari et al. (2020); List et al.
(2022) or unlabeled data Leong et al. (2022); Kudugunta et al. (2024); Nguyen
et al. (2024). Existing tasks in SEACrowd still cover only a small portion of
languages. For instance, sentiment analysis data is available for only 22
languages, and named entity recognition (NER) data is available for just 17
languages. Furthermore, for modalities beyond text, SEA resources are
extremely underrepresented. Approximately 90% of SEA indigenous languages lack
both speech and VL datasets.

Figure 6: SEA languages prioritization based on (top) current utility and
(bottom) resource availability. The languages are ranked based on the
descending order of the area size of their missing potential  .

#### Quality

78.7% of the datasets in SEACrowd are published in peer-reviewed venues, and
most of the data has undergone external validation. The overall quality of the
datasets in SEACrowd is depicted in Figure 5(b). We compile the reported data
construction methods by the authors, considering both the data collection
method (i.e., data source) and label annotation validation (i.e., quality
control). Nearly 19% of the datasets in SEACrowd have machine-generated and
machine-translated annotations, while more than 80% were obtained from online
texts (e.g., web crawling) and expert generation. In terms of label annotation
validation, 62.4% of the datasets have been fully manually checked, while the
remaining portion is partially validated and automatically checked. Note that
these statistics only provide an initial indication of dataset collection
quality on the surface and do not necessarily reflect the exact quality. Only
a few datasets (6%) in SEACrowd report their detailed quality metrics (e.g.,
inter-annotator agreement scores). A deeper investigation is required for
future work.

#### Cultural Relevance

The resource gap in SEA extends to the cultural aspect, where
misrepresentation can lead to offensive behaviors, e.g., cultural
appropriation and stereotyping Evans et al. (2020); Glotov (2023). As a proxy
of the cultural relevance of SEA datasets, we manually curated 259 data
subsets used in SEACrowd evaluation based on their data source. Specifically,
we categorize them whether they are 1) translated from another language, 2)
crawled from local sources, or 3) hand-crafted to capture cultural relevance.
In Figure 5(c), approximately 70% lack cultural relevance, as many are
machine-translated from English sources. About 20% are taken from local news,
social media, or other local outlets, which potentially contain some
culturally relevant data. Only the remaining 10% are designed to consider
cultural relevance, derived from studies highlighting serious deficiencies in
cultural understanding by LLMs for underrepresented languages Kabra et al.
(2023); Koto et al. (2023a); Wibowo et al. (2023); Liu et al. (2024); Koto et
al. (2024).

###  5.2 Conclusion & Future Work

Southeast Asia is home to highly diverse languages and cultures; the majority
of its people do not use English as their primary language(s). The utility of
English-first AI is limited for the majority of Southeast Asian users,
especially in critical sectors like healthcare and education. Through
SEACrowd, we have explored the AI landscape in SEA and bridged the gaps in
resources, evaluation, and naturalness analysis of AI models in SEA languages.
Further, our initiative has nurtured an open-source research community, which
will actively continue to add and maintain datasheets and dataloaders, as well
as drive AI research and developments in SEA.

Nonetheless, AI development in SEA requires concentrated efforts by a range of
stakeholders, who may prioritize differently when it comes to incorporating
the region’s 1,300+ languages into AI models. Moving forward, our work
suggests AI development in SEA should prioritize two key metrics: 1) potential
utility and 2) resource equity.111111https://github.com/SEACrowd/globalutility

#### Potential utility

Potential utility is defined as the gap between current utility and ideal
utility, in which model capability acts as a proxy for utility. Based on
potential utility, unsurprisingly the development of the national languages
(except for English and Chinese used in Singapore), i.e., Indonesian (ind),
Burmese (mya), Vietnamese (vie), Thai (tha), Filipino (fil), Khmer (khm),
Malay (zlm), and Lao (lao) in Figure 6, will bring the biggest benefit. Among
them, we identify notable gaps in the naturalness of Malay, Burmese, and
Filipino AI-generated outputs (§4.2). Focused efforts in resource building for
these languages may move the needle the most for utility. Beyond the national
languages, growing local languages or dialects with large speaker bases, e.g.,
Javanese (jav), Sundanese (sun), and Hmong (hmn), is key.

#### Resource equity

Resource equity is defined as the gap between existing and ideal resource
availability (Figure 6). We found that many local languages or dialects still
fall short of the expected level of resources. These include Northeastern Thai
(tts), Northern Thai (nod), Hmong Do (hmv), Southern Thai (sou), Cebuano
(ceb), Ilocano (ilo), and others. Efforts to narrow these gaps would not only
help preserve these languages but also ensure the continuation of the cultural
heritage of the speakers of these languages. More details on SEA language
prioritization for different weightings of demand can be found in Appendix I.

To improve these metrics, governments, and industry leaders in the region
should invest in R&D activities to improve regional language capability for
both the national languages and local dialects. This could include funding for
open data collection and collaborations with local communities to address the
resource gap in local languages. This also requires long-term sustainable
strategies, such as catalyzing profitable use cases based on inclusive AI
models, promoting fair and responsible compensation schemes for data workers,
and orchestrating win-win exemplar collaborations between data owners, AI, and
application developers.

## Acknowledgments

We would like to thank our amazing contributors: Joshua Spergel, Tiezheng Yu,
Parinthapat Pengpun, Bin Wang, Ishan Jindal, Muhammad Satrio, Jipeng Zhang,
Bhavish Pahwa, Haryo Akbarianto Wibowo, Hiroki Nomoto, Yohanes Sigit Purnomo
W.P., Ahmad Fathan Hidayatullah, Bryan Wilie, Ruhiyah Faradishi Widiaputri,
Rafif Rabbani, Fawwaz Mayda, Manoj Khatri, Supryadi Supryadi, Virach
Sornlertlamvanich, Pavaris Ruangchutiphophan, Erland Hilman Fuadi, Mega
Fransiska, Richardy Sapan, and Camilla Johnine Cosme for their hard work in
submitting datasheets and implementing dataloaders for SEACrowd.

This work is supported by the National Research Foundation, Singapore under
its AI Singapore Programme; PhD Fellowship Award, the Hong Kong University of
Science and Technology; and PF20-43679 Hong Kong PhD Fellowship Scheme,
Research Grant Council, Hong Kong. JMI is funded by National University
Philippines and the UKRI Centre for Doctoral Training in Accountable,
Responsible and Transparent AI [EP/S023437/1] of the University of Bath. In
addition, we would like to express our gratitude to Cohere For AI sfor
providing research grants that enabled us to perform our experiments using a
commercial baseline, specifically Command-R.

## Limitations

While our work covers nearly 1,000 SEA languages, many dialects, which are
considered as belonging to a parent language, are missing from our evaluation
benchmark. For instance, for the Malay language, only Standard Malay (zsm) is
evaluated, but not other dialects such as Sarawak Malay (zlm-sar).
Furthermore, the majority of our datasets also do not contain code-switched
texts, which is a common linguistic phenomenon of SEA language usage (Aji et
al., 2023). Moreover, the language coverage of different evaluation tasks
varies significantly. For instance, text-based tasks cover 34 languages in
total, whereas vision-language tasks only cover four languages.

## Ethics Statement

In developing an evaluation benchmark for SEA languages, we have taken several
steps to ensure ethical considerations are addressed comprehensively. First,
the data used for this benchmark is sourced from publicly available resources,
ensuring compliance with legal and ethical standards regarding data privacy.
Where applicable, explicit consent was obtained from data contributors.
Furthermore, all the datasets and resources utilized in this benchmark are
used in accordance with their respective licenses. Second, our benchmark aims
to be inclusive, representing a wide range of SEA languages, including those
that are underrepresented in current linguistic resources. Lastly, our
research process, including data collection, benchmark development, and
evaluation methodologies, is entirely open-sourced and is documented
transparently to enable reproducibility and accountability.

## References

  * Adelani et al. (2022a) David Adelani, Jesujoba Alabi, Angela Fan, Julia Kreutzer, Xiaoyu Shen, Machel Reid, Dana Ruiter, Dietrich Klakow, Peter Nabende, Ernie Chang, Tajuddeen Gwadabe, Freshia Sackey, Bonaventure F. P. Dossou, Chris Emezue, Colin Leong, Michael Beukman, Shamsuddeen Muhammad, Guyo Jarso, Oreen Yousuf, Andre Niyongabo Rubungo, Gilles Hacheme, Eric Peter Wairagala, Muhammad Umair Nasir, Benjamin Ajibade, Tunde Ajayi, Yvonne Gitau, Jade Abbott, Mohamed Ahmed, Millicent Ochieng, Anuoluwapo Aremu, Perez Ogayo, Jonathan Mukiibi, Fatoumata Ouoba Kabore, Godson Kalipe, Derguene Mbaye, Allahsera Auguste Tapo, Victoire Memdjokam Koagne, Edwin Munkoh-Buabeng, Valencia Wagner, Idris Abdulmumin, Ayodele Awokoya, Happy Buzaaba, Blessing Sibanda, Andiswa Bukula, and Sam Manthalu. 2022a.  A few thousand translations go a long way! leveraging pre-trained models for African news translation.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3053–3070, Seattle, United States. Association for Computational Linguistics.
  * Adelani et al. (2024) David Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba Alabi, Yanke Mao, Haonan Gao, and En-Shiun Lee. 2024.  SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects.  In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 226–245, St. Julian’s, Malta. Association for Computational Linguistics.
  * Adelani et al. (2022b) David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba Alabi, Shamsuddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye, Amelia Taylor, Fatoumata Kabore, Chris Chinenye Emezue, Anuoluwapo Aremu, Perez Ogayo, Catherine Gitau, Edwin Munkoh-Buabeng, Victoire Memdjokam Koagne, Allahsera Auguste Tapo, Tebogo Macucwa, Vukosi Marivate, Mboning Tchiaze Elvis, Tajuddeen Gwadabe, Tosin Adewumi, Orevaoghene Ahia, and Joyce Nakatumba-Nabende. 2022b.  MasakhaNER 2.0: Africa-centric transfer learning for named entity recognition.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 4488–4508, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Adelani et al. (2021) David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Jesujoba Alabi, Seid Muhie Yimam, Tajuddeen Rabiu Gwadabe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, Verrah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chukwuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akinode, Deborah Nabagereka, Maurice Katusiime, Ayodele Awokoya, Mouhamadane MBOUP, Dibora Gebreyohannes, Henok Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulaye Faye, Blessing Sibanda, Orevaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thierno Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Salomey Osei. 2021.  MasakhaNER: Named entity recognition for African languages.  _Transactions of the Association for Computational Linguistics_ , 9:1116–1131.
  * Adelani et al. (2023) David Ifeoluwa Adelani, Marek Masiak, Israel Abebe Azime, Jesujoba Alabi, Atnafu Lambebo Tonja, Christine Mwase, Odunayo Ogundepo, Bonaventure F. P. Dossou, Akintunde Oladipo, Doreen Nixdorf, Chris Chinenye Emezue, Sana Al-azzawi, Blessing Sibanda, Davis David, Lolwethu Ndolela, Jonathan Mukiibi, Tunde Ajayi, Tatiana Moteu, Brian Odhiambo, Abraham Owodunni, Nnaemeka Obiefuna, Muhidin Mohamed, Shamsuddeen Hassan Muhammad, Teshome Mulugeta Ababu, Saheed Abdullahi Salahudeen, Mesay Gemeda Yigezu, Tajuddeen Gwadabe, Idris Abdulmumin, Mahlet Taye, Oluwabusayo Awoyomi, Iyanuoluwa Shode, Tolulope Adelani, Habiba Abdulganiyu, Abdul-Hakeem Omotayo, Adetola Adeeko, Abeeb Afolabi, Anuoluwapo Aremu, Olanrewaju Samuel, Clemencia Siro, Wangari Kimotho, Onyekachi Ogbu, Chinedu Mbonu, Chiamaka Chukwuneke, Samuel Fanijo, Jessica Ojo, Oyinkansola Awosan, Tadesse Kebede, Toadoum Sari Sakayo, Pamela Nyatsine, Freedmore Sidume, Oreen Yousuf, Mardiyyah Oduwole, Kanda Tshinu, Ussen Kimanuka, Thina Diko, Siyanda Nxakama, Sinodos Nigusse, Abdulmejid Johar, Shafie Mohamed, Fuad Mire Hassan, Moges Ahmed Mehamed, Evrard Ngabire, Jules Jules, Ivan Ssenkungu, and Pontus Stenetorp. 2023.  MasakhaNEWS: News topic classification for African languages.  In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 144–159, Nusa Dua, Bali. Association for Computational Linguistics.
  * Adilazuarda et al. (2023) Muhammad Farid Adilazuarda, Samuel Cahyawijaya, and Ayu Purwarianti. 2023.  The obscure limitation of modular multilingual language models.  _ICLR Tiny Papers 2023_.
  * Adilazuarda et al. (2024) Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh, Ashutosh Dwivedi, Alham Fikri Aji, Jacki O’Neill, Ashutosh Modi, and Monojit Choudhury. 2024.  Towards measuring and modeling "culture" in llms: A survey.  _Preprint_ , arXiv:2403.15412.
  * AI@Meta (2024) AI@Meta. 2024.  Llama 3 model card.
  * Aji et al. (2023) Alham Fikri Aji, Jessica Zosa Forde, Alyssa Marie Loo, Lintang Sutawika, Skyler Wang, Genta Indra Winata, Zheng-Xin Yong, Ruochen Zhang, A. Seza Doğruöz, Yin Lin Tan, and Jan Christian Blaise Cruz. 2023.  Current status of NLP in south East Asia with insights from multilingualism and language diversity.  In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: Tutorial Abstract_ , pages 8–13, Nusa Dua, Bali. Association for Computational Linguistics.
  * Aji et al. (2022) Alham Fikri Aji, Genta Indra Winata, Fajri Koto, Samuel Cahyawijaya, Ade Romadhony, Rahmad Mahendra, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Timothy Baldwin, Jey Han Lau, and Sebastian Ruder. 2022.  One country, 700+ languages: NLP challenges for underrepresented languages and dialects in Indonesia.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7226–7249, Dublin, Ireland. Association for Computational Linguistics.
  * AlKhamissi et al. (2024) Badr AlKhamissi, Muhammad ElNokrashy, Mai AlKhamissi, and Mona Diab. 2024.  Investigating cultural alignment of large language models.  _Preprint_ , arXiv:2402.13231.
  * Almazrouei et al. (2023) Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023.  Falcon-40B: an open large language model with state-of-the-art performance.
  * Ardila et al. (2020) Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020.  Common voice: A massively-multilingual speech corpus.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 4218–4222, Marseille, France. European Language Resources Association.
  * Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020.  On the cross-lingual transferability of monolingual representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4623–4637, Online. Association for Computational Linguistics.
  * Aryabumi et al. (2024) Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Kelly Marchisio, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. 2024.  Aya 23: Open weight releases to further multilingual progress.  _Preprint_ , arXiv:2405.15032.
  * Asai et al. (2023) Akari Asai, Sneha Kudugunta, Xinyan Velocity Yu, Terra Blevins, Hila B Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannaneh Hajishirzi. 2023.  BUFFET: Benchmarking large language models for cross-lingual few-shot transfer.  _Preprint_ , arXiv:2305.14857.
  * Asgari et al. (2020) Ehsaneddin Asgari, Fabienne Braune, Benjamin Roth, Christoph Ringlstetter, and Mohammad Mofrad. 2020.  UniSent: Universal adaptable sentiment lexica for 1000+ languages.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 4113–4120, Marseille, France. European Language Resources Association.
  * Astuti et al. (2023) Laksmita Widya Astuti, Yunita Sari, and Suprapto. 2023.  Code-mixed sentiment analysis using transformer for twitter social media data.  _International Journal of Advanced Computer Science and Applications_ , 14(10).
  * Babu et al. (2021) Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. 2021.  Xls-r: Self-supervised cross-lingual speech representation learning at scale.  _Preprint_ , arXiv:2111.09296.
  * Bandarkar et al. (2023) Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2023.  The belebele benchmark: a parallel reading comprehension dataset in 122 language variants.  _arXiv preprint arXiv:2308.16884_.
  * Blasi et al. (2022) Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2022.  Systematic inequalities in language technology performance across the world’s languages.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5486–5505, Dublin, Ireland. Association for Computational Linguistics.
  * Burns et al. (2020) Andrea Burns, Donghyun Kim, Derry Wijaya, Kate Saenko, and Bryan A Plummer. 2020.  Learning to scale multilingual representations for vision-language tasks.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16_ , pages 197–213. Springer.
  * Cahyawijaya et al. (2022) Samuel Cahyawijaya, Alham Fikri Aji, Holy Lovenia, Genta Indra Winata, Bryan Wilie, Rahmad Mahendra, Fajri Koto, David Moeljadi, Karissa Vincentio, Ade Romadhony, and Ayu Purwarianti. 2022.  Nusacrowd: A call for open and reproducible nlp research in indonesian languages.  _Preprint_ , arXiv:2207.10524.
  * Cahyawijaya et al. (2024a) Samuel Cahyawijaya, Delong Chen, Yejin Bang, Leila Khalatbari, Bryan Wilie, Ziwei Ji, Etsuko Ishii, and Pascale Fung. 2024a.  High-dimension human value representation in large language models.  _arXiv preprint arXiv:2404.07900_.
  * Cahyawijaya et al. (2023a) Samuel Cahyawijaya, Holy Lovenia, Alham Fikri Aji, Genta Winata, Bryan Wilie, Fajri Koto, Rahmad Mahendra, Christian Wibisono, Ade Romadhony, Karissa Vincentio, Jennifer Santoso, David Moeljadi, Cahya Wirawan, Frederikus Hudi, Muhammad Satrio Wicaksono, Ivan Parmonangan, Ika Alfina, Ilham Firdausi Putra, Samsul Rahmadani, Yulianti Oenang, Ali Septiandri, James Jaya, Kaustubh Dhole, Arie Suryani, Rifki Afina Putri, Dan Su, Keith Stevens, Made Nindyatama Nityasya, Muhammad Adilazuarda, Ryan Hadiwijaya, Ryandito Diandaru, Tiezheng Yu, Vito Ghifari, Wenliang Dai, Yan Xu, Dyah Damapuspita, Haryo Wibowo, Cuk Tho, Ichwanul Karo Karo, Tirana Fatyanosa, Ziwei Ji, Graham Neubig, Timothy Baldwin, Sebastian Ruder, Pascale Fung, Herry Sujaini, Sakriani Sakti, and Ayu Purwarianti. 2023a.  NusaCrowd: Open source initiative for Indonesian NLP resources.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 13745–13818, Toronto, Canada. Association for Computational Linguistics.
  * Cahyawijaya et al. (2023b) Samuel Cahyawijaya, Holy Lovenia, Fajri Koto, Dea Adhista, Emmanuel Dave, Sarah Oktavianti, Salsabil Akbar, Jhonson Lee, Nuur Shadieq, Tjeng Wawan Cenggoro, Hanung Linuwih, Bryan Wilie, Galih Muridan, Genta Winata, David Moeljadi, Alham Fikri Aji, Ayu Purwarianti, and Pascale Fung. 2023b.  NusaWrites: Constructing high-quality corpora for underrepresented and extremely low-resource languages.  In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 921–945, Nusa Dua, Bali. Association for Computational Linguistics.
  * Cahyawijaya et al. (2024b) Samuel Cahyawijaya, Holy Lovenia, Fajri Koto, Rifki Afina Putri, Emmanuel Dave, Jhonson Lee, Nuur Shadieq, Wawan Cenggoro, Salsabil Maulana Akbar, Muhammad Ihza Mahendra, Dea Annisayanti Putri, Bryan Wilie, Genta Indra Winata, Alham Fikri Aji, Ayu Purwarianti, and Pascale Fung. 2024b.  Cendol: Open instruction-tuned generative large language models for indonesian languages.  _Preprint_ , arXiv:2404.06138.
  * Cahyawijaya et al. (2021) Samuel Cahyawijaya, Genta Indra Winata, Bryan Wilie, Karissa Vincentio, Xiaohong Li, Adhiguna Kuncoro, Sebastian Ruder, Zhi Yuan Lim, Syafri Bahar, Masayu Khodra, Ayu Purwarianti, and Pascale Fung. 2021.  IndoNLG: Benchmark and resources for evaluating Indonesian natural language generation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8875–8898, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Catapang and Visperas (2023) Jasper Kyle Catapang and Moses Visperas. 2023.  Emotion-based morality in Tagalog and English scenarios (EMoTES-3K): A parallel corpus for explaining (im)morality of actions.  In _Proceedings of the Joint 3rd International Conference on Natural Language Processing for Digital Humanities and 8th International Workshop on Computational Linguistics for Uralic Languages_ , pages 1–6, Tokyo, Japan. Association for Computational Linguistics.
  * Communication et al. (2023) Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenzek, Ethan Ye, Bapi Akula, Peng-Jen Chen, Naji El Hachem, Brian Ellis, Gabriel Mejia Gonzalez, Justin Haaheim, Prangthip Hansanti, Russ Howes, Bernie Huang, Min-Jae Hwang, Hirofumi Inaguma, Somya Jain, Elahe Kalbassi, Amanda Kallet, Ilia Kulikov, Janice Lam, Daniel Li, Xutai Ma, Ruslan Mavlyutov, Benjamin Peloquin, Mohamed Ramadan, Abinesh Ramakrishnan, Anna Sun, Kevin Tran, Tuan Tran, Igor Tufanov, Vish Vogeti, Carleigh Wood, Yilin Yang, Bokai Yu, Pierre Andrews, Can Balioglu, Marta R. Costa-jussà, Onur Celebi, Maha Elbayad, Cynthia Gao, Francisco Guzmán, Justine Kao, Ann Lee, Alexandre Mourachko, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Paden Tomasello, Changhan Wang, Jeff Wang, and Skyler Wang. 2023.  Seamlessm4t: Massively multilingual & multimodal machine translation.  _Preprint_ , arXiv:2308.11596.
  * Conneau et al. (2021) Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. 2021.  Unsupervised Cross-Lingual Representation Learning for Speech Recognition.  In _Proc. Interspeech 2021_ , pages 2426–2430.
  * Conneau et al. (2022) Alexis Conneau, Ankur Bapna, Yu Zhang, Min Ma, Patrick von Platen, Anton Lozhkov, Colin Cherry, Ye Jia, Clara Rivera, Mihir Kale, Daan van Esch, Vera Axelrod, Simran Khanuja, Jonathan Clark, Orhan Firat, Michael Auli, Sebastian Ruder, Jason Riesa, and Melvin Johnson. 2022.  XTREME-S: Evaluating Cross-lingual Speech Representations.  In _Proc. Interspeech 2022_ , pages 3248–3252.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
  * Costa-jussà et al. (2024) Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Jeff Wang, and N. L. L. B. Team. 2024.  Scaling neural machine translation to 200 languages.  _Nature_.
  * Dabre et al. (2022) Raj Dabre, Himani Shrotriya, Anoop Kunchukuttan, Ratish Puduppully, Mitesh Khapra, and Pratyush Kumar. 2022.  IndicBART: A pre-trained model for indic natural language generation.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1849–1863, Dublin, Ireland. Association for Computational Linguistics.
  * Dac Lai et al. (2023) Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen. 2023.  Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback.  _arXiv e-prints_ , pages arXiv–2307.
  * Dai et al. (2024) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2024.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  _Advances in Neural Information Processing Systems_ , 36.
  * Dorfman (1979) Robert Dorfman. 1979.  A formula for the gini coefficient.  _The review of economics and statistics_ , pages 146–149.
  * Dou et al. (2024) Longxu Dou, Qian Liu, Guangtao Zeng, Jia Guo, Jiahui Zhou, Wei Lu, and Min Lin. 2024.  Sailor: Open language models for south-east asia.  _Preprint_ , arXiv:2404.03608.
  * Dryer and Haspelmath (2013) Matthew S. Dryer and Martin Haspelmath, editors. 2013.  _WALS Online (v2020.3)_.  Zenodo.
  * Durmus et al. (2023) Esin Durmus, Karina Nguyen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, et al. 2023.  Towards measuring the representation of subjective global opinions in language models.  _arXiv preprint arXiv:2306.16388_.
  * Eberhard et al. (2021) David M. Eberhard, Gary F. Simons, and Charles D. Fennig. 2021.  _Ethnologue: Languages of the World. Twenty-fourth edition._ Dallas, Texas: SIL International.
  * Ebrahimi et al. (2022) Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir Meza Ruiz, Gustavo Giménez-Lugo, Elisabeth Mager, Graham Neubig, Alexis Palmer, Rolando Coto-Solano, Thang Vu, and Katharina Kann. 2022.  AmericasNLI: Evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resource languages.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6279–6299, Dublin, Ireland. Association for Computational Linguistics.
  * Elias (2018) Alexander Elias. 2018.  Lio and the central flores languages.  _Leiden: Leiden University Master thesis_.
  * Evans et al. (2020) Leanne M Evans, Crystasany R Turner, and Kelly R Allen. 2020.  " good teachers" with" good intentions": Misappropriations of culturally responsive pedagogy.  _Journal of Urban Learning, Teaching, and Research_ , 15(1):51–73.
  * Federmann et al. (2022) Christian Federmann, Tom Kocmi, and Ying Xin. 2022.  NTREX-128 – news test references for MT evaluation of 128 languages.  In _Proceedings of the First Workshop on Scaling Up Multilingual Evaluation_ , pages 21–24, Online. Association for Computational Linguistics.
  * Geigle et al. (2023) Gregor Geigle, Abhay Jain, Radu Timofte, and Goran Glavaš. 2023.  mblip: Efficient bootstrapping of multilingual vision-llms.  _arXiv_ , abs/2307.06930.
  * Gemma Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. 2024.  Gemma: Open models based on gemini research and technology.  _Preprint_ , arXiv:2403.08295.
  * Glotov (2023) Sergei Glotov. 2023.  Intercultural film literacy education against cultural misrepresentation: Finnish visual art teachers’ perspectives.  _Journal of Media Literacy Education_ , 15(1):31–43.
  * Gong et al. (2023) Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. 2023.  Multimodal-gpt: A vision and language model for dialogue with humans.  _arXiv preprint arXiv:2305.04790_.
  * Hammarström et al. (2024) Harald Hammarström, Robert Forkel, Martin Haspelmath, and Sebastian Bank. 2024.  Glottolog 5.0. leipzig: Max planck institute for evolutionary anthropology.
  * Hasan et al. (2021) Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021.  XL-sum: Large-scale multilingual abstractive summarization for 44 languages.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4693–4703, Online. Association for Computational Linguistics.
  * He et al. (2022) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2022.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.  In _The Eleventh International Conference on Learning Representations_.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020.  Deberta: Decoding-enhanced bert with disentangled attention.  In _International Conference on Learning Representations_.
  * Huang et al. (2021) Po-Yao Huang, Mandela Patrick, Junjie Hu, Graham Neubig, Florian Metze, and Alexander Hauptmann. 2021.  Multilingual multimodal pre-training for zero-shot cross-lingual transfer of vision-language models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2443–2459, Online. Association for Computational Linguistics.
  * Huynh et al. (2022) Tin Van Huynh, Kiet Van Nguyen, and Ngan Luu-Thuy Nguyen. 2022.  ViNLI: A Vietnamese corpus for studies on open-domain natural language inference.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 3858–3872, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Ichsan (2023) Muhammad Ichsan. 2023.  Merak-7b: The llm for bahasa indonesia.  _Hugging Face Repository_.
  * Imperial et al. (2019) Joseph Marvin Imperial, Jeyrome Orosco, Shiela Mae Mazo, and Lany Maceda. 2019.  Sentiment analysis of typhoon related tweets using standard and bidirectional recurrent neural networks.  _arXiv preprint arXiv:1908.01765_.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Jiang et al. (2022) Shengyi Jiang, Sihui Fu, Nankai Lin, and Yingwen Fu. 2022.  Pretrained models and evaluation data for the khmer language.  _Tsinghua Science and Technology_ , 27(4):709–718.
  * Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020.  The state and fate of linguistic diversity and inclusion in the NLP world.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6282–6293, Online. Association for Computational Linguistics.
  * Juan et al. (2015) Sarah Samson Juan, Laurent Besacier, Benjamin Lecouteux, and Mohamed Dyab. 2015.  Using resources from a closely-related language to develop asr for a very under-resourced language: A case study for iban.  In _Proceedings of INTERSPEECH_ , Dresden, Germany.
  * Kabra et al. (2023) Anubha Kabra, Emmy Liu, Simran Khanuja, Alham Fikri Aji, Genta Winata, Samuel Cahyawijaya, Anuoluwapo Aremu, Perez Ogayo, and Graham Neubig. 2023.  Multi-lingual and multi-cultural figurative language understanding.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 8269–8284, Toronto, Canada. Association for Computational Linguistics.
  * Kakwani et al. (2020) Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020.  IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4948–4961, Online. Association for Computational Linguistics.
  * Karo et al. (2022) Ichwanul Muslim Karo Karo, Mohd Farhan Md Fudzee, Shahreen Kasim, and Azizul Azhar Ramli. 2022.  Sentiment analysis in karonese tweet using machine learning.  _Indonesian Journal of Electrical Engineering and Informatics (IJEEI)_ , 10(1):219–231.
  * Khanuja et al. (2023) Simran Khanuja, Sebastian Ruder, and Partha Talukdar. 2023.  Evaluating the diversity, equity, and inclusion of NLP technology: A case study for Indian languages.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 1763–1777, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Koto et al. (2023a) Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. 2023a.  Large language models only pass primary school exams in Indonesia: A comprehensive test on IndoMMLU.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , Singapore. Association for Computational Linguistics.
  * Koto et al. (2023b) Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. 2023b.  Large language models only pass primary school exams in Indonesia: A comprehensive test on IndoMMLU.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 12359–12374, Singapore. Association for Computational Linguistics.
  * Koto et al. (2022) Fajri Koto, Timothy Baldwin, and Jey Han Lau. 2022.  Cloze evaluation for deeper understanding of commonsense stories in Indonesian.  In _Proceedings of the First Workshop on Commonsense Representation and Reasoning (CSRR 2022)_ , pages 8–16, Dublin, Ireland. Association for Computational Linguistics.
  * Koto and Koto (2020) Fajri Koto and Ikhwan Koto. 2020.  Towards computational linguistics in Minangkabau language: Studies on sentiment analysis and machine translation.  In _Proceedings of the 34th Pacific Asia Conference on Language, Information and Computation_ , pages 138–148, Hanoi, Vietnam. Association for Computational Linguistics.
  * Koto et al. (2024) Fajri Koto, Rahmad Mahendra, Nurul Aisyah, and Timothy Baldwin. 2024.  Indoculture: Exploring geographically-influenced cultural commonsense reasoning across eleven indonesian provinces.  _Preprint_ , arXiv:2404.01854.
  * Kudugunta et al. (2024) Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2024.  Madlad-400: a multilingual and document-level large audited dataset.  In _Proceedings of the 37th International Conference on Neural Information Processing Systems_ , NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.
  * Kumar et al. (2022) Aman Kumar, Himani Shrotriya, Prachi Sahu, Amogh Mishra, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, Mitesh M. Khapra, and Pratyush Kumar. 2022.  IndicNLG benchmark: Multilingual datasets for diverse NLG tasks in Indic languages.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5363–5394, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Laurençon et al. (2024) Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. 2024.  What matters when building vision-language models? _Preprint_ , arXiv:2405.02246.
  * Le and Luu (2023) Thang Le and Anh Luu. 2023.  A parallel corpus for Vietnamese central-northern dialect text transfer.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 13839–13855, Singapore. Association for Computational Linguistics.
  * Leong et al. (2022) Colin Leong, Joshua Nemecek, Jacob Mansdorfer, Anna Filighera, Abraham Owodunni, and Daniel Whitenack. 2022.  Bloom library: Multimodal datasets in 300+ languages for a variety of downstream tasks.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8608–8621, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Leong et al. (2023) Wei Qi Leong, Jian Gang Ngui, Yosephine Susanto, Hamsawardhini Rengarajan, Kengatharaiyer Sarveswaran, and William Chandra Tjhi. 2023.  Bhasa: A holistic southeast asian linguistic and cultural evaluation suite for large language models.  _arXiv preprint arXiv:2309.06085_.
  * Li et al. (2023a) Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023a.  Bactrian-x: A multilingual replicable instruction-following model with low-rank adaptation.  _arXiv preprint arXiv:2305.15011_.
  * Li et al. (2023b) Zejun Li, Zhihao Fan, Jingjing Chen, Qi Zhang, Xuanjing Huang, and Zhongyu Wei. 2023b.  Unifying cross-lingual and cross-modal modeling towards weakly supervised multilingual vision-language pre-training.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5939–5958, Toronto, Canada. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Lin et al. (2022) Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2022.  Few-shot learning with multilingual generative language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * List et al. (2022) Johann-Mattis List, Robert Forkel, Simon J. Greenhill, Christoph Rzymski, Johannes Englisch, and Russell D. Gray. 2022.  Lexibank, a public repository of standardized wordlists with computed phonological and lexical features.  _Scientific Data_ , 9(1):316.
  * Liu et al. (2024) Chen Cecilia Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2024.  Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings.  _Preprint_ , arXiv:2309.08591.
  * Liu et al. (2023a) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a.  Improved baselines with visual instruction tuning.
  * Liu et al. (2023b) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b.  Visual instruction tuning.  In _NeurIPS_.
  * Longpre et al. (2021) Shayne Longpre, Yi Lu, and Joachim Daiber. 2021.  MKQA: A linguistically diverse benchmark for multilingual open domain question answering.  _Transactions of the Association for Computational Linguistics_ , 9:1389–1406.
  * Longpre et al. (2023) Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, et al. 2023.  The data provenance initiative: A large scale audit of dataset licensing & attribution in ai.  _arXiv preprint arXiv:2310.16787_.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Mager et al. (2021) Manuel Mager, Arturo Oncevay, Annette Rios, Ivan Vladimir Meza Ruiz, Alexis Palmer, Graham Neubig, and Katharina Kann, editors. 2021.  _Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas_. Association for Computational Linguistics, Online.
  * Mahendra et al. (2021) Rahmad Mahendra, Alham Fikri Aji, Samuel Louvan, Fahrurrozi Rahman, and Clara Vania. 2021.  IndoNLI: A natural language inference dataset for Indonesian.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10511–10527, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023.  Crosslingual generalization through multitask finetuning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15991–16111, Toronto, Canada. Association for Computational Linguistics.
  * Muennighoff et al. (2022) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022.  Crosslingual generalization through multitask finetuning.  _arXiv preprint arXiv:2211.01786_.
  * Muzad and Rahutomo (2016) Aad Muzad and Faisal Rahutomo. 2016.  Korpus berita daring bahasa indonesia dengan depth first focused crawling.  _Prosiding Sentrinov (Seminar Nasional Terapan Riset Inovatif)_ , 2(1):11–20.
  * Neubig and Hu (2018) Graham Neubig and Junjie Hu. 2018.  Rapid adaptation of neural machine translation to new languages.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 875–880, Brussels, Belgium. Association for Computational Linguistics.
  * Nguyen et al. (2020) Kiet Nguyen, Vu Nguyen, Anh Nguyen, and Ngan Nguyen. 2020.  A Vietnamese dataset for evaluating machine reading comprehension.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 2595–2605, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Nguyen et al. (2024) Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2024.  CulturaX: A cleaned, enormous, and multilingual dataset for large language models in 167 languages.  In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_ , pages 4226–4237, Torino, Italia. ELRA and ICCL.
  * Nguyen et al. (2023) Xuan-Phi Nguyen, Wenxuan Zhang, Li Xin, Mahani Aljunied, Weiwen Xu, Hou Pong Chan, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. 2023.  Seallms - large language models for southeast asia.  _Preprint_ , arXiv:arXiv:2312.00738.
  * OpenAI et al. (2024) OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024.  Gpt-4 technical report.  _Preprint_ , arXiv:2303.08774.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Palen-Michel and Lignos (2023) Chester Palen-Michel and Constantine Lignos. 2023.  LR-sum: Summarization for less-resourced languages.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 6829–6844, Toronto, Canada. Association for Computational Linguistics.
  * Phatthiyaphaibun et al. (2023) Wannaphong Phatthiyaphaibun, Korakot Chaovavanich, Charin Polpanumas, Arthit Suriyawongkul, Lalita Lowphansirikul, Pattarawat Chormai, Peerat Limkonchotiwat, Thanathip Suntorntip, and Can Udomcharoenchaikit. 2023.  PyThaiNLP: Thai natural language processing in python.  In _Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)_ , pages 25–36, Singapore. Association for Computational Linguistics.
  * Phatthiyaphaibun et al. (2024) Wannaphong Phatthiyaphaibun, Surapon Nonesung, Patomporn Payoungkhamdee, Peerat Limkonchotiwat, Can Udomcharoenchaikit, Jitkapat Sawatphol, Chompakorn Chaksangchaichot, Ekapol Chuangsuwanich, and Sarana Nutanong. 2024.  Wangchanlion and wangchanx mrc eval.  _Preprint_ , arXiv:2403.16127.
  * Ponti et al. (2020) Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020.  XCOPA: A multilingual dataset for causal commonsense reasoning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2362–2376, Online. Association for Computational Linguistics.
  * Popović (2015) Maja Popović. 2015.  chrF: character n-gram F-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  * Popović (2017) Maja Popović. 2017.  chrF++: words helping character n-grams.  In _Proceedings of the Second Conference on Machine Translation_ , pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics.
  * Pratap et al. (2024) Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. 2024.  Scaling speech technology to 1,000+ languages.  _Journal of Machine Learning Research_ , 25(97):1–52.
  * Project (2024) The Joshua Project. 2024.  The joshua project.
  * Purwarianti and Crisdayanti (2019) Ayu Purwarianti and Ida Ayu Putu Ari Crisdayanti. 2019.  Improving bi-lstm performance for indonesian sentiment analysis using paragraph vector.  In _2019 International Conference of Advanced Informatics: Concepts, Theory and Applications (ICAICTA)_ , pages 1–5. IEEE.
  * Purwarianti et al. (2007) Ayu Purwarianti, Masatoshi Tsuchiya, and Seiichi Nakagawa. 2007.  A machine learning approach for Indonesian question answering system.  In _Artificial Intelligence and Applications_ , pages 573–578.
  * Putra et al. (2024) I Made Suwija Putra, Daniel Siahaan, and Ahmad Saikhu. 2024.  Snli indo: A recognizing textual entailment dataset in indonesian derived from the stanford natural language inference dataset.  _Data in Brief_ , 52:109998.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8748–8763. PMLR.
  * Radford et al. (2023) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. 2023.  Robust speech recognition via large-scale weak supervision.  In _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 28492–28518. PMLR.
  * Riccosan and Saputra (2023) Riccosan and Karen Etania Saputra. 2023.  Multilabel multiclass sentiment and emotion dataset from indonesian mobile application review.  _Data in Brief_ , 50:109576.
  * Riley et al. (2020) Parker Riley, Isaac Caswell, Markus Freitag, and David Grangier. 2020.  Translationese as a language in “multilingual” NMT.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7737–7746, Online. Association for Computational Linguistics.
  * Rizqullah et al. (2023) Muhammad Razif Rizqullah, Ayu Purwarianti, and Alham Fikri Aji. 2023.  Qasina: Religious domain question answering using sirah nabawiyah.  In _2023 10th International Conference on Advanced Informatics: Concept, Theory and Application (ICAICTA)_ , pages 1–6. IEEE.
  * Rouditchenko et al. (2023) Andrew Rouditchenko, Sameer Khurana, Samuel Thomas, Rogerio Feris, Leonid Karlinsky, Hilde Kuehne, David Harwath, Brian Kingsbury, and James Glass. 2023.  Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages.  In _Proc. INTERSPEECH 2023_ , pages 2268–2272.
  * Ruder et al. (2023) Sebastian Ruder, Jonathan H Clark, Alexander Gutkin, Mihir Kale, Min Ma, Massimo Nicosia, Shruti Rijhwani, Parker Riley, Jean-Michel Sarr, Xinyi Wang, et al. 2023.  Xtreme-up: A user-centric scarce-data benchmark for under-represented languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 1856–1884.
  * Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. 2021.  Multitask prompted training enables zero-shot task generalization.  _Preprint_ , arXiv:2110.08207.
  * Sani et al. (2012) Auliya Sani, Sakriani Sakti, Graham Neubig, Tomoki Toda, Adi Mulyanto, and Satoshi Nakamura. 2012.  Towards language preservation: Preliminary collection and vowel analysis of indonesian ethnic speech data.  In _2012 International Conference on Speech Database and Assessments_ , pages 118–122.
  * Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ , 35:25278–25294.
  * Setya and Mahendra (2018) Ken Nabila Setya and Rahmad Mahendra. 2018.  Semi-supervised textual entailment on indonesian wikipedia data.  In _International Conference on Computational Linguistics and Intelligent Text Processing_ , pages 416–427. Springer.
  * Singapore (2023) AI Singapore. 2023.  Sea-lion (southeast asian languages in one network): A family of large language models for southeast asia.  https://github.com/aisingapore/sealion.
  * Singh et al. (2024) Shivalika Singh, Freddie Vargus, Daniel D’souza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, et al. 2024.  Aya dataset: An open-access collection for multilingual instruction tuning.  _arXiv preprint arXiv:2402.06619_.
  * Søgaard (2022) Anders Søgaard. 2022.  Should we ban English NLP for a year? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5254–5260, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Sutoyo et al. (2022) Rhio Sutoyo, Said Achmad, Andry Chowanda, Esther Widhi Andangsari, and Sani M. Isa. 2022.  Prdect-id: Indonesian product reviews dataset for emotions classification tasks.  _Data in Brief_ , 44:108554.
  * Täckström et al. (2012) Oscar Täckström, Ryan McDonald, and Jakob Uszkoreit. 2012.  Cross-lingual word clusters for direct transfer of linguistic structure.  In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 477–487, Montréal, Canada. Association for Computational Linguistics.
  * Talat et al. (2022) Zeerak Talat, Aurélie Névéol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luccioni, Maraim Masoud, Margaret Mitchell, Dragomir Radev, Shanya Sharma, Arjun Subramonian, Jaesung Tae, Samson Tan, Deepak Tunuguntla, and Oskar Van Der Wal. 2022.  You reap what you sow: On the challenges of bias evaluation under multilingual settings.  In _Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models_, pages 26–41, virtual+Dublin. Association for Computational Linguistics.
  * Thapliyal et al. (2022) Ashish V. Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut. 2022.  Crossmodal-3600: A massively multilingual multimodal evaluation dataset.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 715–729, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Tran et al. (2021) Khanh Quoc Tran, Phap Ngoc Trinh, Khoa Nguyen-Anh Tran, An Tran-Hoai Le, Luan Van Ha, and Kiet Van Nguyen. 2021.  An empirical investigation of online news classification on an open-domain, large-scale and high-quality dataset in vietnamese.  In _New Trends in Intelligent Software Methodologies, Tools and Techniques_ , pages 367–379. IOS Press.
  * Üstün et al. (2024) Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. 2024.  Aya model: An instruction finetuned open-access multilingual language model.  _arXiv preprint arXiv:2402.07827_.
  * Van Nguyen et al. (2022) Kiet Van Nguyen, Tin Van Huynh, Duc-Vu Nguyen, Anh Gia-Tuan Nguyen, and Ngan Luu-Thuy Nguyen. 2022.  New vietnamese corpus for machine reading comprehension of health news articles.  _ACM Trans. Asian Low-Resour. Lang. Inf. Process._ , 21(5).
  * Vedantam et al. (2015) Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015.  Cider: Consensus-based image description evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4566–4575.
  * Wang et al. (2023) Bin Wang, Zhengyuan Liu, Xin Huang, Fangkai Jiao, Yang Ding, Ai Ti Aw, and Nancy F Chen. 2023.  Seaeval for multilingual foundation models: From cross-lingual alignment to cultural reasoning.  _arXiv preprint arXiv:2309.04766_.
  * Wang et al. (2024) Bin Wang, Zhengyuan Liu, Xin Huang, Fangkai Jiao, Yang Ding, Ai Ti Aw, and Nancy F. Chen. 2024.  Seaeval for multilingual foundation models: From cross-lingual alignment to cultural reasoning.  _NAACL_.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_.
  * Wibowo et al. (2023) Haryo Akbarianto Wibowo, Erland Hilman Fuadi, Made Nindyatama Nityasya, Radityo Eko Prasojo, and Alham Fikri Aji. 2023.  Copal-id: Indonesian language reasoning with local culture and nuances.  _arXiv preprint arXiv:2311.01012_.
  * Wilie et al. (2020) Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti. 2020.  IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pages 843–857, Suzhou, China. Association for Computational Linguistics.
  * Winata et al. (2023) Genta Indra Winata, Alham Fikri Aji, Samuel Cahyawijaya, Rahmad Mahendra, Fajri Koto, Ade Romadhony, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Pascale Fung, Timothy Baldwin, Jey Han Lau, Rico Sennrich, and Sebastian Ruder. 2023.  NusaX: Multilingual parallel sentiment dataset for 10 Indonesian local languages.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 815–834, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Winata et al. (2024) Genta Indra Winata, Ruochen Zhang, and David Ifeoluwa Adelani. 2024.  Miners: Multilingual language models as semantic retrievers.  _arXiv preprint arXiv:2406.07424_.
  * Workshop et al. (2022) BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Yong et al. (2023) Zheng Xin Yong, Ruochen Zhang, Jessica Forde, Skyler Wang, Arjun Subramonian, Holy Lovenia, Samuel Cahyawijaya, Genta Winata, Lintang Sutawika, Jan Christian Blaise Cruz, Yin Lin Tan, Long Phan, Long Phan, Rowena Garcia, Thamar Solorio, and Alham Aji. 2023.  Prompting multilingual large language models to generate code-mixed texts: The case of south East Asian languages.  In _Proceedings of the 6th Workshop on Computational Approaches to Linguistic Code-Switching_ , pages 43–63, Singapore. Association for Computational Linguistics.
  * Zhang et al. (2023a) Ruochen Zhang, Samuel Cahyawijaya, Jan Christian Blaise Cruz, Genta Winata, and Alham Aji. 2023a.  Multilingual large language models are not (yet) code-switchers.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 12567–12582, Singapore. Association for Computational Linguistics.
  * Zhang et al. (2023b) Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023b.  M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models.  In _Advances in Neural Information Processing Systems_ , volume 36, pages 5484–5505. Curran Associates, Inc.
  * Zhang et al. (2024) Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2024.  M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models.  _Advances in Neural Information Processing Systems_ , 36.
  * Zhu et al. (2024) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2024.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _ICLR_.

##  Appendix A Summary of SEACrowd

Benchmark | # Languages | # Indigenous SEA Languages | # Datasets | # Tasks
---|---|---|---|---
SEACrowd (ours)† | 39 | 38 | 254 | 13 (11 text, 1 speech, 1 vision)
NusaCrowd† Cahyawijaya et al. (2023a) | 19 | 19 | 137 | 12 (11 text, 1 speech)
BUFFET Asai et al. (2023) | 54 | N/A | 15 | 8 (8 text)
XTREME-UP Ruder et al. (2023) | 88 | 11 | 269 | 9 (7 text, 1 speech, 1 vision)

Table 3: Benchmark comparison. †The numbers in SEACrowd and NusaCrowd are the
numbers of datasets included in the evaluation.

Addressing the resource gaps and challenges in AI development for Southeast
Asian (SEA) languages is essential for our region’s sustainable and prosperous
future. The lack of representation of SEA languages in machine learning pre-
training models severely impacts their quality. Additionally, the scarcity of
high-quality datasets and evaluation tools further hinders progress in AI for
SEA languages. The dominance of English-centric training data introduces
cultural biases and fails to capture the local values and nuances of SEA
cultures. To overcome these obstacles, SEACrowd provides a comprehensive and
standardized resource center, along with evaluation tasks, for nearly 1,000
SEA indigenous and non-indigenous languages across various modalities.
SEACrowd closes the resource and evaluation gaps, enabling researchers and
developers to improve the performance of AI models for SEA languages.

The journey does not end here. Concrete next steps are essential to drive AI
advancement in Southeast Asia. Strategic investments in research and
development, collaborations with local communities, and efforts toward
language preservation are imperative. Governments, industry leaders, and
stakeholders must prioritize the development of national and under-resourced
local languages to ensure resource equity and promote inclusivity in AI
technology. By taking bold actions, such as funding initiatives for data
collection and model training, establishing partnerships with local
communities, and focusing on language preservation, we can unlock the full
potential of Southeast Asian languages. Not only this will spur economic
growth but also preserve the region’s rich cultural heritage.

In conclusion, developing AI for Southeast Asian languages is not a mere
necessity but an opportunity to create a sustainable and prosperous future. By
addressing resource gaps, accurately evaluating models, and fostering
inclusive AI development, we can harness the power of SEA languages to drive
long-term economic growth while preserving our region’s cultural diversity.

##  Appendix B Related Work

#### SEA data resources

LLM research efforts for SEA languages are limited by the lack of available
datasets and benchmarks. Up to this day, resources for SEA NLP tasks are
concentrated on relatively higher-resource SEA indigenous languages, such as
Indonesian (Mahendra et al., 2021; Wilie et al., 2020; Cahyawijaya et al.,
2021, 2023a) and Vietnamese (Nguyen et al., 2020; Huynh et al., 2022; Le and
Luu, 2023; Van Nguyen et al., 2022). NusaCrowd Cahyawijaya et al. (2023a)
introduce the first multimodal benchmark for Indonesian languages, including
text and speech. Ruder et al. (2023) introduce a multimodal benchmark
encompassing 11 indigenous languages from SEA, spanning a wide array of
languages totaling 88.

Additionally, Asai et al. (2023) present an LLM benchmark for cross-lingual
few-shot transfer, comprising 15 distinct tasks and 54 languages sourced from
varied multilingual datasets. Furthermore, Dou et al. (2024) find that
publicly available pre-training data for SEA languages suffer from quality
issues such as textual duplicates and excessive occurrences of Unicode
escapes. On the other hand, pre-trained LLMs specifically for SEA languages
suffer from limited language coverage; for instance, Cendol Cahyawijaya et al.
(2024b), Sailor Dou et al. (2024), SEA-LION (Singapore, 2023), and SeaLLMs
Nguyen et al. (2023) have only covered up to 11 different SEA languages,
including English and Chinese.

#### Open-source Community Initiatives in NLP

Open-source and open-science communities play a crucial role in engaging
native speakers to curate large-scale multilingual NLP resources. In the past,
collaborative efforts have been organized to collect data and train
multilingual language models either on a global scale (Workshop et al., 2022;
Singh et al., 2024; Üstün et al., 2024) or on a regional level, e.g.,
Masakhane for African languages (Adelani et al., 2021, 2022b, 2022a, 2023),
AI4Bharat for Indian languages (Kakwani et al., 2020; Kumar et al., 2022;
Dabre et al., 2022, inter alia), and AmericasNLP for Latin American languages
Mager et al. (2021); Ebrahimi et al. (2022). In the SEA region, there have
been community-based initiatives, e.g., IndoNLP, PyThaiNLP, and RojakNLP, to
study NLP on Indonesian languages (Aji et al., 2022; Wilie et al., 2020;
Cahyawijaya et al., 2021, 2023a), Thai language (Phatthiyaphaibun et al.,
2023), and the code-switching phenomenon in SEA (Aji et al., 2023; Yong et
al., 2023; Winata et al., 2024), respectively.

Submission | Points | Max points
---|---|---
Public datasheet | 2+bonus | 6
Dataloader | 3 | 6 if difficult
Private datasheet | 1 | -
Access to private data | 4+bonus | 10 if high-quality
Datasheet review | 1 | 1
Dataloader review | 2 | 4 if difficult
Private datasheet review | 0.5 | -
Private data contact | 1 | 5 if succeeds

Table 4: Amount of points obtained for contributions related to datasheet,
dataloader, and private data.

##  Appendix C Contributing to SEACrowd

###  C.1 Open Contributions

We identify four tasks for open contribution in SEACrowd.121212Landing page:
https://github.com/SEACrowd. These tasks and the workflow of SEACrowd are
heavily influenced by and extended upon NusaCrowd Cahyawijaya et al. (2023a,
2022), a collaborative effort to pool data resources for Indonesian NLP.

Figure 7: A glimpse of SEACrowd Catalogue.

  * •

Submitting Metadata for Existing Public Datasets. Contributors can submit
detailed datasheets for existing datasets through this form.131313Public
datasheet form: https://form.jotform.com/team/232952680898069/seacrowd-sea-
datasets. Contributors must provide important information such as data
license, size, language and dialect, annotation method, and so on. The
approved datasheets, as well as under review datasheets, will show up and be
indexed in a monitor spreadsheet and the SEACrowd Catalogue (Figure 7).

  * •

Building a Dataloader. From the approved datasheets from the previous task,
contributors can further contribute by building a HuggingFace dataset loader
to ensure that all datasets in SEACrowd are standardized in terms of
formatting and usage. Contributors can follow a dataloader guide and examples
available141414Dataloader guide: https://github.com/SEACrowd/seacrowd-
datahub/blob/master/DATALOADER.md. in the SEACrowd Data Hub. Dataloader
maintainers and reviewers also monitor the self-assigned dataloader issues
after 2 weeks of inactivity and ping contributors in case of a blocking
impediment.

  * •

Identifying Private AI Datasets for SEA Languages, Cultures, and/or Regions.
Unfortunately, a number of prior works involving SEA languages are still not
publicly available. These may be due to several different reasons, including
(but not limited to): non-release contracts related to funding, inclusion of
private and personally identifiable data, and the use of explicitly private
data such as those used by for-profit companies.

In this task, contributors can search for works that contain private data and
fill out a corresponding record form.151515Papers with private dataset form:
https://form.jotform.com/team/232952680898069/seacrowd-paper-with-private-
dataset. The SEACrowd team then attempts to contact the original data owners
and negotiate the open-sourcing of their resources.

  * •

Opening a Private AI Dataset of SEA. If a contributor has previous work with
closed data (or has been contacted by the SEACrowd team regarding closed-
source data), they can decide to release their resources and register them in
the collection via the public datasheet form. The resource will still be owned
by the original contributor and is still tied to the contributor’s previous
work, as SEACrowd simply catalogs it and records its now open-source license.

###  C.2 Measuring Contributions

Figure 8: The timeline of SEACrowd’s entire run.

To be considered as a co-author, 20 contribution points are
required.161616Submissions past the deadlines (see Appendix D.1) are still
recorded, but contribution points are no longer given. To monitor how many
points the contributors have obtained, the contribution point tracking is
provided and updated regularly. The purpose of the point system is not to
barrier collaboration but to reward rare and high-quality dataset entries.
Table 4 describes the contribution points.171717Contribution point guidelines:
https://github.com/SEACrowd/seacrowd-datahub/blob/master/POINTS.md. A bonus of
1 point is given if the dataset modality is speech or vision. We also provide
a bonus based on the language rarity in terms of available resources as
defined by Joshi et al.
(2020)181818https://microsoft.github.io/linguisticdiversity/assets/lang2tax.txt,
consisting of 1 point for languages in level 1 and 2, and 2 points for
languages in level 0 or absent from the list. For other contributions not
mentioned in Table 4 (e.g., maintenance, design, experiment, paper writing,
etc.), the amount of contribution points is adjusted to the bulk and the
complexity of the relevant work.

##  Appendix D Progression of SEACrowd

###  D.1 Timeline

SEACrowd released the open call for contributions on 1 November 2023. This
lasted until 31 March 2024, for datasheet submissions, and until 15 May 2024
for both dataloaders and private dataset submissions. SEACrowd contributors
have a biweekly discussion regarding the challenges they face while
contributing, the next steps they should take to proceed, and/or experiment
and research ideas for the paper. The detailed timeline can be seen in Figure
8.

###  D.2 Contribution Progress

Figure 9 shows the number of submissions for public datasheets, dataloader
pull requests, and papers with private datasets in SEACrowd.

##  Appendix E Reviewing SEACrowd’s Submissions

We provide the complete reviewing guidelines in our Data Hub.191919Reviewer
SOP: https://github.com/SEACrowd/seacrowd-datahub/blob/master/REVIEWING.md

###  E.1 Datasheet Reviewing

The datasheet reviewing standard operating procedure (SOP) ensures the
integrity and completeness of datasets submitted to SEACrowd. It outlines
procedures for verifying dataset availability, avoiding duplicates, and
ensuring correctness and relevance to the SEA region. The SOP includes FAQs
addressing common issues such as dataset duplicates and incorrect information,
along with an approval checklist covering aspects like data availability,
dataset splits, and licensing. Reviewers are instructed on how to handle
various scenarios, including correcting errors and determining points
allocation for multiple contributors. For instance, if the datasheet submitted
has incorrect or missing information, the reviewer can either ask the
contributor to fix it (with some guidance) or fix it themself. Upon completion
of the review, reviewers update the status, add notes and points, and await
the generation of a GitHub issue for the approved datasheet.

Figure 9: Weekly status update of the cumulative number of submissions in
SEACrowd.

###  E.2 Dataloader Reviewing

The dataloader reviewing SOP governs the review process for dataloaders in
SEACrowd, ensuring adherence to the data structure and seacrowd schema and
config standards. It specifies checks for metadata correctness, subset
implementation, test script passing, and adherence to coding conventions.
Additionally, it outlines dataloader config rules based on dataset types and
provides guidelines for multilingual datasets. The SOP emphasizes the
importance of reviewer collaboration, with each dataloader requiring two
reviewers per submitted pull request, and outlines the approval and reviewer
assignment process, either by allocation or by self-assignment based on
availability and promptness.

##  Appendix F Schemas in SEACrowd

Schemas define and format the attributes of the dataset returned by a
dataloader. For each dataloader, we implement 2 schema types: the source
schema and the seacrowd schema. The source schema presents the dataset in a
format similar to its original structure, while the seacrowd schema
standardizes the data structure across similar tasks.

The following subsections define the seacrowd schemas in NLP (F.1), speech
(F.2), and VL (F.3).

Subset ID | Language | Region | # Samples
---|---|---|---
Sentiment Analysis $\rightarrow$ *_seacrowd_text
lazada_review_filipino | fil | Philippines | 1001
gklmip_sentiment | mya | Myanmar | 716
indolem_sentiment | ind | Indonesia | 1011
id_sentiment_analysis | ind | Indonesia | 10806
karonese_sentiment | btx | Indonesia | 1000
wisesight_thai_sentiment | tha | Thailand | 2671
wongnai_reviews | tha | Thailand | 6203
typhoon_yolanda_tweets | fil | Philippines | 153
smsa | ind | Indonesia | 500
prdect_id_sentiment | ind | Indonesia | 5400
id_sent_emo_mobile_apps_sentiment | ind | Indonesia | 21696
shopee_reviews_tagalog | fil | Philippines | 2250
nusatranslation_senti_abs | abs | Indonesia | 500
nusatranslation_senti_btk | btx | Indonesia | 1200
nusatranslation_senti_bew | bew | Indonesia | 1200
nusatranslation_senti_bhp | bhp | Indonesia | 500
nusatranslation_senti_jav | jav | Indonesia | 1200
nusatranslation_senti_mad | mad | Indonesia | 1200
nusatranslation_senti_mak | mak | Indonesia | 1200
nusatranslation_senti_min | min | Indonesia | 1200
nusatranslation_senti_mui | mui | Indonesia | 500
nusatranslation_senti_rej | rej | Indonesia | 500
nusatranslation_senti_sun | sun | Indonesia | 1200
nusax_senti_ind | ind | Indonesia | 400
nusax_senti_ace | ace | Indonesia | 400
nusax_senti_jav | jav | Indonesia | 400
nusax_senti_sun | sun | Indonesia | 400
nusax_senti_min | min | Indonesia | 400
nusax_senti_bug | bug | Indonesia | 400
nusax_senti_bbc | bbc | Indonesia | 400
nusax_senti_ban | ban | Indonesia | 400
nusax_senti_nij | nij | Indonesia | 400
nusax_senti_mad | mad | Indonesia | 400
nusax_senti_bjn | bjn | Indonesia | 400
nusax_senti_eng | eng | Non-indigenous | 400
indonglish | ind | Indonesia | 1011

Table 5: Sentiment analysis data subsets used in SEACrowd NLU evaluation.

Subset ID | Language | Region | # Samples
---|---|---|---
NLI $\rightarrow$ *_seacrowd_pairs
indonli | ind | Indonesia | 5183
wrete | ind | Indonesia | 100
snli_indo | ind | Indonesia | 9823
myxnli | mya | Myanmar | 5010
xnli.tha | tha | Thailand | 5010
xnli.vie | vie | Vietnam | 5010

Table 6: NLI data subsets used in SEACrowd NLU evaluation.

Subset ID | Language | Region | # Samples
---|---|---|---
Topic Classification $\rightarrow$ *_seacrowd_text
gklmip_newsclass | khm | Cambodia | 1436
indonesian_news_dataset | ind | Indonesia | 2627
uit_vion | vie | Vietnam | 26000
sib_200_ace_Arab | ace | Indonesia | 204
sib_200_ace_Latn | ace | Indonesia | 204
sib_200_ban_Latn | ban | Indonesia | 204
sib_200_bjn_Arab | bjn | Indonesia | 204
sib_200_bjn_Latn | bjn | Indonesia | 204
sib_200_bug_Latn | bug | Indonesia | 204
sib_200_ceb_Latn | ceb | Philippines | 204
sib_200_ilo_Latn | ilo | Philippines | 204
sib_200_ind_Latn | ind | Indonesia | 204
sib_200_jav_Latn | jav | Indonesia | 204
sib_200_kac_Latn | kac | Myanmar | 204
sib_200_khm_Khmr | khm | Cambodia | 204
sib_200_lao_Laoo | lao | Laos | 204
sib_200_lus_Latn | lus | Myanmar | 204
sib_200_min_Arab | min | Indonesia | 204
sib_200_min_Latn | min | Indonesia | 204
sib_200_mya_Mymr | mya | Myanmar | 204
sib_200_pag_Latn | pag | Philippines | 204
sib_200_shn_Mymr | shn | Myanmar | 204
sib_200_sun_Latn | sun | Indonesia | 204
sib_200_tgl_Latn | fil | Philippines | 204
sib_200_tha_Thai | tha | Thailand | 204
sib_200_vie_Latn | vie | Non-indigenous | 204
sib_200_war_Latn | war | Philippines | 204
sib_200_zsm_Latn | zsm | Malaysia | 204
nusaparagraph_topic_btk | btx | Indonesia | 500
nusaparagraph_topic_bew | bew | Indonesia | 800
nusaparagraph_topic_bug | bug | Indonesia | 300
nusaparagraph_topic_jav | jav | Indonesia | 800
nusaparagraph_topic_mad | mad | Indonesia | 700
nusaparagraph_topic_mak | mak | Indonesia | 700
nusaparagraph_topic_min | min | Indonesia | 800
nusaparagraph_topic_mui | mui | Indonesia | 400
nusaparagraph_topic_rej | rej | Indonesia | 350
nusaparagraph_topic_sun | sun | Indonesia | 900

Table 7: Topic classification data subsets used in SEACrowd NLU evaluation.

Subset ID | Language | Region | # Samples
---|---|---|---
Commonsense Reasoning $\rightarrow$ *_seacrowd_text/qa
emotes_3k_tgl | fil | Philippines | 2905
emotes_3k_eng | eng | Non-indigenous | 2905
indo_story_cloze | ind | Indonesia | 1135
xstorycloze_id | ind | Indonesia | 1511
xstorycloze_my | mya | Myanmar | 1511

Table 8: Commonsense reasoning data subsets used in SEACrowd NLU evaluation.

Subset ID | Language | Region | # Samples
---|---|---|---
Standard Testing QA $\rightarrow$ *_seacrowd_qa
indommlu_ind | ind | Indonesia | 14979
indommlu_ban | ban | Indonesia | 14979
indommlu_mad | mad | Indonesia | 14979
indommlu_mak | mak | Indonesia | 14979
indommlu_sun | sun | Indonesia | 14979
indommlu_jav | jav | Indonesia | 14979
indommlu_bjn | bjn | Indonesia | 14979
indommlu_abl | abl | Indonesia | 14979
indommlu_nij | nij | Indonesia | 14979
seaeval_cross_mmlu_ind | ind | Indonesia | 150
seaeval_cross_mmlu_vie | vie | Vietnam | 150
seaeval_cross_mmlu_zlm | zsm | Malaysia | 150
seaeval_cross_mmlu_fil | fil | Philippines | 150
seaeval_cross_logiqa_ind | ind | Indonesia | 176
seaeval_cross_logiqa_vie | vie | Vietnam | 176
seaeval_cross_logiqa_zlm | zsm | Malaysia | 176
seaeval_cross_logiqa_fil | fil | Philippines | 176
m3exam_jav | jav | Indonesia | 371
m3exam_tha | tha | Thailand | 2168
m3exam_vie | vie | Vietnam | 1789
okapi_m_arc_ind | ind | Indonesia | 1170
okapi_m_arc_vie | vie | Vietnam | 1170
Cultural QA $\rightarrow$ *_seacrowd_qa
copal_colloquial | ind | Indonesia | 559
xcopa_tha | tha | Thailand | 500
xcopa_vie | vie | Vietnam | 500
xcopa_ind | ind | Indonesia | 500
seaeval_sg_eval_eng | eng | Non-indigenous | 103
seaeval_ph_eval_eng | eng | Non-indigenous | 100
mabl_ind | ind | Indonesia | 1140
mabl_jav | jav | Indonesia | 600
mabl_sun | sun | Indonesia | 600
Reading Comprehension QA $\rightarrow$ *_seacrowd_qa
belebele_ceb_latn | ceb | Philippines | 900
belebele_ilo_latn | ilo | Philippines | 900
belebele_ind_latn | ind | Indonesia | 900
belebele_jav_latn | jav | Indonesia | 900
belebele_kac_latn | kac | Myanmar | 900
belebele_khm_khmr | khm | Cambodia | 900
belebele_lao_laoo | lao | Laos | 900
belebele_mya_mymr | mya | Myanmar | 900
belebele_shn_mymr | shn | Myanmar | 900
belebele_sun_latn | sun | Indonesia | 900
belebele_tgl_latn | fil | Philippines | 900
belebele_tha_thai | tha | Thailand | 900
belebele_vie_latn | vie | Vietnam | 900
belebele_war_latn | war | Philippines | 900
belebele_zsm_latn | zsm | Malaysia | 900

Table 9: Multiple-choice QA data subsets used in SEACrowd NLU evaluation.

Subset ID | Language | Region | # Samples
---|---|---|---
Extractive & Abstractive QA $\rightarrow$ *_seacrowd_qa
facqa | ind | Indonesia | 311
iapp_squad | tha | Thailand | 739
qasina | ind | Indonesia | 500
mkqa_khm | khm | Cambodia | 10000
mkqa_zsm | zsm | Malaysia | 10000
mkqa_tha | tha | Thailand | 10000
mkqa_vie | vie | Vietnam | 10000

Table 10: Extractive and abstractive QA subsets used in SEACrowd NLG
evaluation.

Subset ID | Language | Region | # Samples
---|---|---|---
Summarization $\rightarrow$ *_seacrowd_t2t
lr_sum_ind | ind | Indonesia | 500
lr_sum_vie | vie | Vietnam | 1460
lr_sum_lao | lao | Laos | 1496
lr_sum_tha | tha | Thailand | 500
lr_sum_khm | khm | Cambodia | 486
lr_sum_mya | mya | Myanmar | 990
xl_sum_mya | mya | Myanmar | 570
xl_sum_ind | ind | Indonesia | 4780
xl_sum_tha | tha | Thailand | 826
xl_sum_vie | vie | Vietnam | 4013

Table 11: Summarization data subsets used in SEACrowd NLG evaluation.

###  F.1 NLP

  * •

Unlabeled text (SSP). This schema could be used for language modeling in self-
supervised pre-training. It consists of (id, text), where id denotes a unique
row identifier of the dataset and text denotes an input text.

  * •

Single-label text classification (TEXT). This schema could be used for
sentiment analysis, emotion classification, legal classification, and others.
It consists of (id, text, label), where id denotes a unique row identifier of
the dataset, text denotes an input text, and label denotes a deterministic
target variable.

  * •

Multi-label text classification (TEXT MULTI). This schema could be used for
hate speech detection and aspect-based sentiment analysis. It consists of (id,
text, labels), where id denotes a unique row identifier of the dataset, text
denotes an input text, and labels denotes a list of deterministic target
variables.

  * •

Text-to-text (T2T). This schema could be used for machine translation,
summarization, and paraphrasing. It consists of (id, text_1, text_2,
text_1_name, text_2_name), where id denotes a unique row identifier of the
dataset, text_1 and text_2 denote an input text pair, and text_1_name and
text_2_name denote the names of the input text pair (e.g., ind and jav for
translation input text pairs, or document and summary for summarization input
text pairs).

  * •

Sequence labeling (SEQ LABEL). This schema could be used for named entity
recognition (NER), POS tagging, and others. It consists of (id, tokens,
labels), where id denotes a unique row identifier of the dataset, tokens
denotes a list of tokens of an input text, and labels denotes a list of
targets for the tokens.

  * •

Question answering (QA). This schema could be used for extractive QA,
multiple-choice QA, and others. It consists of (id, question_id, document_id,
question, type, choices, context, answer), where id denotes a unique row
identifier of the dataset, question_id denotes a unique identifier of the
question, document_id denotes a unique identifier of the context document,
question denotes an input question to be answered, type denotes the type of
the QA task (e.g., extractive, multiple-choice, open-generative, closed-
generative, etc.), choices denotes a list of answer choices (if required),
context denotes a passage that serves as the background information of the
question (if required), and answer denotes the gold answer to the question (if
required).

  * •

Single-label text pair classification (PAIRS). This could be used for textual
entailment and next-sentence prediction. It consists of (id, text_1, text_2,
label), where id denotes a unique row identifier of the dataset, text_1 and
text_2 denote an input text pair, and label denotes the target variable.

  * •

Single-label text pair classification with continuous values or regression
(PAIRS SCORE). This could be used for answer grading and semantic textual
similarity. It consists of (id, text_1, text_2, label), where id denotes a
unique row identifier of the dataset, text_1 and text_2 denote an input text
pair, and label denotes a target variable as a continuous value.

  * •

Multi-label text pair classification (PAIRS MULTI). This could be used for
morphological inflection. It consists of (id, text_1, text_2, labels), where
id denotes a unique row identifier of the dataset, text_1 and text_2 denote an
input text pair, and labels denotes a list of target variables.

  * •

Knowledge base (KB). This schema could be used for constituency parsing,
dependency parsing, coreference resolution, dialogue systems, and other tasks
with complex structures. It consists of (id, passages, entities, events,
coreferences, relations). Considering its intricate structure, we encourage
readers to take a look at the implementation of the knowledge base schema.

  * •

Tree (TREE). This schema could be used for constituency parsing, this schema
assumes a document with subnode elements and a tree hierarchy. It consists of
(id, passage, nodes), where id denotes a unique row identifier of the dataset,
passage denotes the passage to that particular id; this passage consist of
(id, type, text, offsets), nodes denotes the nodes to that particular id; this
nodes consists of (id, type, text, offsets, subnodes).

  * •

Conversational Chat (CHAT). This schema could be used for conversational chat
and/or multi-turn conversation. It consists of (id, input, output, meta),
where id denotes a unique row identifier of the dataset, input denotes a
sequence that consists of content and role as an input prompt and the role of
the entity inputting the prompt, output denotes an answer from that input
prompt, and meta denotes relevant details to allow some flexibility of the
schema (if required).

  * •

End-to-end Task Oriented Dialogue (TOD). This schema could be used for end-to-
end task-oriented dialogue. It consists of (dialogue_idx, dialogue), where
dialogue_idx denotes a unique row identifier of the dialogue, dialogue denotes
some core details such as turn label, system utterance, turn idx, belief state
(consist of slots and act), user utterance, and system acts.

Subset ID | Language | Region | # Samples
---|---|---|---
Image Captioning $\rightarrow$ *_seacrowd_imtext
xm3600_fil | fil | Philippines | 2760
xm3600_id | ind | Indonesia | 2775
xm3600_th | tha | Thailand | 2798
xm3600_vi | vie | Vietnam | 2855

Table 12: Image captioning data subsets used in SEACrowd VL evaluation.

###  F.2 Speech

  * •

Speech-text (SPTEXT). This could be used for speech recognition, text-to-
speech (TTS) or speech synthesis, and speech-to-text translation. It consists
of (id, path, audio, text, speaker_id, metadata), where id denotes a unique
row identifier of the dataset, path denotes the file path to an input audio
source, audio denotes the audio data loaded from the corresponding path, text
denotes an input text, speaker_id denotes a unique identifier of the speaker,
metadata denotes relevant details such as the age and gender of the speaker
(if required).

  * •

Speech-to-speech (S2S). This could be used for speech-to-speech translation.
It consists of (id, path_1, audio_1, text_1, metadata_1, path_2, audio_2,
text_2, metadata_2), where id denotes a unique row identifier of the dataset,
path_1 and path_2 denote the file path to a respective input audio source,
audio_1 and audio_2 denote the audio data loaded from the corresponding path,
text_1 and text_2 denote input texts, and metadata_1 and metadata_2 denote
relevant details such as the age of the speaker and their gender (if
required).

  * •

Speech Classification (SPEECH). This schema could be used for speech
classification, speech-language identification, and speech-emotion recognition
for single-label use only. It consists of (id, path, audio, speaker_id,
labels, metadata), where id denotes a unique row identifier of the dataset,
path denotes the file path to an input audio source, audio denotes the audio
data loaded from the corresponding path, speaker_id denotes a unique
identifier of the speaker, labels denotes the label of that particular speech
(only can be single-label), metadata denotes relevant details such as the age
and gender of the speaker (if required).

  * •

Speech Classification for Multilabel (SPEECH MULTILABEL). This schema could be
used for speech classification, speech-language identification, and speech-
emotion recognition for multi-label use only. It consists of (id, path, audio,
speaker_id, labels, metadata), where id denotes a unique row identifier of the
dataset, path denotes the file path to an input audio source, audio denotes
the audio data loaded from the corresponding path, speaker_id denotes a unique
identifier of the speaker, labels denotes the sequence of labels of that
particular speech (only can be multi-label), metadata denotes relevant details
such as the age and gender of the speaker (if required).

Subset ID | Language | Region | # Samples
---|---|---|---
ASR $\rightarrow$ *_seacrowd_sptext
asr_ibsc | iba | Brunei | 473
commonvoice_120_ind | ind | Indonesia | 3647
commonvoice_120_tha | tha | Thailand | 10964
commonvoice_120_cnh | cnh | Myanmar | 763
commonvoice_120_vie | vie | Vietnam | 1302
fleurs_ind | ind | Indonesia | 687
fleurs_jav | jav | Indonesia | 728
fleurs_tha | tha | Thailand | 1021
fleurs_lao | lao | Laos | 405
fleurs_mya | mya | Myanmar | 880
fleurs_khm | khm | Cambodia | 771
fleurs_vie | vie | Vietnam | 857
fleurs_zlm | zlm | Malaysia | 749
fleurs_fil | fil | Philippines | 964
fleurs_ceb | ceb | Philippines | 541
indspeech_newstra_ethnicsr_nooverlap_jav | jav | Indonesia | 1000
indspeech_newstra_ethnicsr_nooverlap_sun | sun | Indonesia | 1000
indspeech_newstra_ethnicsr_nooverlap_ban | ban | Indonesia | 1000
indspeech_newstra_ethnicsr_nooverlap_btk | btx | Indonesia | 1000

Table 13: ASR data subsets used in SEACrowd speech evaluation.

Subset ID | Language | Region | # Samples
---|---|---|---
Eng $\rightarrow$ XX | XX $\rightarrow$ Eng
MT (Eng $\Leftrightarrow$ XX) $\rightarrow$ *_seacrowd_t2t
lio_and_central_flores_eng_ljl | lio_and_central_flores_ljl_eng | ljl | Indonesia | 1658
flores200_eng_Latn_ace_Latn | flores200_ace_Latn_eng_Latn | ace | Indonesia | 1012
flores200_eng_Latn_ban_Latn | flores200_ban_Latn_eng_Latn | ban | Indonesia | 1012
flores200_eng_Latn_bjn_Latn | flores200_bjn_Latn_eng_Latn | bjn | Indonesia | 1012
flores200_eng_Latn_bug_Latn | flores200_bug_Latn_eng_Latn | bug | Indonesia | 1012
flores200_eng_Latn_ceb_Latn | flores200_ceb_Latn_eng_Latn | ceb | Philippines | 1012
flores200_eng_Latn_ilo_Latn | flores200_ilo_Latn_eng_Latn | ilo | Philippines | 1012
flores200_eng_Latn_ind_Latn | flores200_ind_Latn_eng_Latn | ind | Indonesia | 1012
flores200_eng_Latn_jav_Latn | flores200_jav_Latn_eng_Latn | jav | Indonesia | 1012
flores200_eng_Latn_kac_Latn | flores200_kac_Latn_eng_Latn | kac | Myanmar | 1012
flores200_eng_Latn_khm_Khmr | flores200_khm_Khmr_eng_Latn | khm | Cambodia | 1012
flores200_eng_Latn_lao_Laoo | flores200_lao_Laoo_eng_Latn | lao | Laos | 1012
flores200_eng_Latn_lus_Latn | flores200_lus_Latn_eng_Latn | lus | Myanmar | 1012
flores200_eng_Latn_min_Latn | flores200_min_Latn_eng_Latn | min | Indonesia | 1012
flores200_eng_Latn_mya_Mymr | flores200_mya_Mymr_eng_Latn | mya | Myanmar | 1012
flores200_eng_Latn_pag_Latn | flores200_pag_Latn_eng_Latn | pag | Philippines | 1012
flores200_eng_Latn_shn_Mymr | flores200_shn_Mymr_eng_Latn | shn | Myanmar | 1012
flores200_eng_Latn_sun_Latn | flores200_sun_Latn_eng_Latn | sun | Indonesia | 1012
flores200_eng_Latn_tha_Thai | flores200_tha_Thai_eng_Latn | tha | Thailand | 1012
flores200_eng_Latn_vie_Latn | flores200_vie_Latn_eng_Latn | vie | Vietnam | 1012
flores200_eng_Latn_war_Latn | flores200_war_Latn_eng_Latn | war | Philippines | 1012
flores200_eng_Latn_zsm_Latn | flores200_zsm_Latn_eng_Latn | zsm | Malaysia | 1012
ntrex_128_eng-US_ind | ntrex_128_ind_eng-US | ind | Indonesia | 1997
ntrex_128_eng-US_mya | ntrex_128_mya_eng-US | mya | Myanmar | 1997
ntrex_128_eng-US_fil | ntrex_128_fil_eng-US | fil | Philippines | 1997
ntrex_128_eng-US_khm | ntrex_128_khm_eng-US | khm | Cambodia | 1997
ntrex_128_eng-US_lao | ntrex_128_lao_eng-US | lao | Laos | 1997
ntrex_128_eng-US_zlm | ntrex_128_zlm_eng-US | zsm | Malaysia | 1997
ntrex_128_eng-US_tha | ntrex_128_tha_eng-US | tha | Thailand | 1997