#  DialectBench:
A NLP Benchmark for Dialects, Varieties, and Closely-Related Languages

Fahim Faisal$\alpha$, Orevaoghene Ahia$\beta$,* Aarohi Srivastava$\gamma$
Kabir Ahuja$\beta$
David Chiang$\gamma$ Yulia Tsvetkov$\beta$ Antonios
Anastasopoulos$\alpha,\delta$
$\alpha$George Mason University $\beta$University of Washington
$\gamma$University of Notre Dame
$\delta$ Archimedes Research Unit, RC Athena, Greece
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>Equal contribution.

###### Abstract

Language technologies should be judged on their usefulness in real-world use
cases. An often overlooked aspect in natural language processing (NLP)
research and evaluation is language variation in the form of non-standard
dialects or language varieties (hereafter, varieties). Most NLP benchmarks are
limited to standard language varieties. To fill this gap, we propose
DialectBench, the first-ever large-scale benchmark for NLP on varieties, which
aggregates an extensive set of task-varied variety datasets (10 text-level
tasks covering 281 varieties). This allows for a comprehensive evaluation of
NLP system performance on different language varieties. We provide substantial
evidence of performance disparities between standard and non-standard language
varieties, and we also identify language clusters with larger performance
divergence across tasks. We believe DialectBench provides a comprehensive view
of the current state of NLP for language varieties and one step towards
advancing it further. 111More information can be found at the following:
Code/data:  https://github.com/ffaisal93/DialectBench Website:
https://fahimfaisal.info/DialectBench.io

##  1 Introduction

Benchmarking is important for tracking the progress the field of natural
language processing (NLP) has made in various tasks. In the past few years,
large-scale multilingual benchmarks like XTREME Hu et al. (2020), XTREME-R
Ruder et al. (2021), and XGLUE Liang et al. (2020) have played a pivotal role
in evaluating the multilingual capabilities of NLP models. These efforts have
sought to make model evaluation more accessible to researchers and
representative of a variety of languages Song et al. (2023). However, most
benchmarks have focused on the standard varieties of languages, largely
neglecting non-standard dialects and language varieties Blasi et al. (2022).

Figure 1: DialectBench Evaluation Suite.

We refer to non-standard dialects and language varieties simply as _varieties_
, and sometimes include low-resource related languages, writing system
variants, and other kinds of variation. Varieties contain subtle but notable
variations in vocabulary, pronunciation, orthography and grammar, reflecting
regional, social, and cultural differences Chambers and Trudgill (1998). The
non-standard nature of these language varieties oftentimes contributes to the
scarcity of substantial datasets that accurately capture these variations
Hedderich et al. (2021). As a result, they have often been absent from widely
adopted benchmarks, even from admirable efforts like XTREME-up Ruder et al.
(2023), GLUECoS Khanuja et al. (2020) and CreoleVal Lent et al. (2023), which
focus on under-resourced, code-switched, and creole languages, respectively.

Figure 2: DialectBench language clusters with their variety counts per task.
"Other" encompasses 18 clusters (full cluster list in Appendix
LABEL:lang_list).

It is currently challenging to accurately test the robustness of multilingual
models on a large suite of varieties without establishing an NLP evaluation
framework covering multiple language clusters (each of which contains standard
languages alongside its closely related varieties).

To this end, we create DialectBench, a large-scale benchmark covering 40
language clusters with 281 varieties, spanning 10 NLP tasks. We observe that
the performance disparity between different varieties of the same language
cluster becomes more pronounced when we shift from zero-shot evaluation to
fine-tuning on variety data, because of uneven data availability across
varieties. Certain language clusters exhibit varying performance across
downstream tasks within the same category, due to low-resource limitations.
Additionally, we improve the dialectal task coverage for natural language
inference by constructing a translate-test evaluation dataset. Putting these
all together, DialectBench serves as a comprehensive suite that attains a
multifaceted purpose: identifying broader limitations in dialectal NLP, while
reflecting on potential areas for improvement.

##  2 DialectBench

Category |  Task |  Metric |  Source Dataset
---|---|---|---
Structured |  DEP parsing |  UAS |  Universal Dependency Zeman et al. (2021), TwitterAAE Blodgett et al. (2018), Singlish Wang et al. (2017)
Prediction |  POS tagging |  F1 |  Universal Dependency Zeman et al. (2021), Singlish Wang et al. (2017), Noisy Dialects Blaschke et al. (2023)
|  NER |  F1 |  Wikiann Pan et al. (2017); Rahimi et al. (2019), Norwegian NER Johansen (2019)
Classification |  DId |  F1 |  MADAR Bouamor et al. (2018), DMT Jauhiainen et al. (2019), Greek Sababa and Stassopoulou (2018), DSL-TL Zampieri et al. (2023), Swiss Germans Scherrer et al. (2019)
|  SA |  F1 |  TSAC Medhaffar et al. (2017), TUNIZI Fourati et al. (2021), DzSentiA Abdelli et al. (2019), SaudiBank Alqahtani et al. (2022), MAC Garouani and Kharroubi (2022), ASTD Nabil et al. (2015), AJGT Alomari et al. (2017), OCLAR Al Omari et al. (2019)
|  TC |  F1 |  SIB-200 Adelani et al. (2023)
|  NLI |  F1 |  XNLI Conneau et al. (2018) translate-test
Question |  MRC |  F1 |  Belebele Bandarkar et al. (2023)
Answering |  EQA |  Span F1 |  SDQA Faisal et al. (2021)
Generation |  MT |  BLEU |  CODET Alam et al. (2023), TIL-MT Mirzakhalov (2021)
Table 1: The tasks and data sources of DialectBench (Detailed discussion:
LABEL:{sec:tasks_benchmark}).

DialectBench is a benchmark created to unify dialectal datasets across
different tasks to foster research on language varieties and non-standard
dialects. Below we describe the design choices we undertook to achieve this
goal. This includes our language variety and task selection procedures, data
collation methods, and evaluation principles.

#### Variety Selection

We first looked through papers published in the ACL
Anthology222https://aclanthology.org from the last 10 years to find usable
language resources, as well as commonly used online data repositories Littauer
(n.d.). We selected languages that have well-established, high-resourced
varieties. Varieties may vary by location, ethnicity, or other factors. We
also found instances where varieties are classified by writing system or even
by genre (e.g., Twitter). When varying by location, varieties may be
classified by different datasets at different levels of _granularity_ ,
sometimes country, region, or city. In some cases, we found resources with two
or more varieties within one dataset (e.g., the UD_Portuguese-Bosque
dependency treebank Rademaker et al. (2017) includes examples from both
European and Brazilian Portuguese variants. To incorporate all these cases
under one paradigm, we formulate a cluster-variety mapping procedure.

####  Cluster-Variety Mapping

We construct several language clusters comprising of both high-resourced
varieties and their low-resourced counterparts. We use the Glottolog language
database Nordhoff (2012) to define clusters and assign varieties as outlined
in Figure 1. This design choice enables us to keep varieties that are closely
related in terms of either mutual intelligibility, phylogenetic similarity or
geographic proximity within the same cluster. Hence, all cluster varieties
always root back to the closest common linguistic ancestor and the whole
cluster maps to an established phylogenetic subtree. For example, Fiji Hindi
and Hindi, with
Hindustani333https://glottolog.org/resource/languoid/id/hind1270 as their
closest common ancestor, are placed in the Hindustani cluster.

We primarily use the Glottocode language identification scheme Hammarström and
Forkel (2022), ensuring a standardized naming scheme across all varieties. For
instance, AAE variety from TwitterAAE Blodgett et al. (2018) dependency
parsing dataset is renamed as African American Vernacular English with a
corresponding Glottocode afri1276. In cases where Glottocodes are unavailable,
like for spoken English from South India, we substitute with the immediate
ancestor Glottocode (indi1255) and further categorize the varieties using the
following metadata identifiers:

  1. 1.

Area (a): the region where the variety is spoken or where its dataset was
collected.

  2. 2.

Language register (r): frozen, formal, consultative, casual, and intimate.

  3. 3.

Language mode (m): written, spoken, and signed language.

  4. 4.

Orthography (o): In DialectBench this is only specific to Sinitic varieties.
This could be either traditional or simplified.

  5. 5.

Identifier (i): Dataset-specific metadata, could be domain (eg. twitter).

We encapsulate all this information, into a naming convention, and use the
template: {Glottocode name}-(a:{},r:{},m:{},o:{},i:{}). 444For example,
mandarin chinese (a:mainland, o:simplified refers to Mandarin Chinese
(mand1415) spoken in Mainland China and written in simplified characters.

####  Cluster Representative

Each cluster will often have a high-resourced variety usually with the largest
speaker population. We choose this high-resourced variety as the cluster
representative. This selection might vary across downstream tasks depending on
the data availability. We primarily utilize this representative variety to
evaluate the performance gap across cluster varieties, and also rely on it for
transfer-learning in resource-scarce settings. Sometimes, the members of a
cluster are considered closely related languages, and sometimes dialects; to
avoid making this distinction, we refer to all the members of a cluster simply
as _varieties_ of the cluster representative.

#### Task and Dataset Selection

In selecting tasks, we maintain a balanced approach, promoting task diversity
while also including tasks that require diverse levels of textual
understanding. In the end, our complete list of tasks are as follows:

  1. 1.

Dependency parsing (DEP parsing)

  2. 2.

Parts-of-speech tagging (POS tagging)

  3. 3.

Named entity recognition (NER)

  4. 4.

Dialect identification (DId)

  5. 5.

Sentiment analysis (SA)

  6. 6.

Topic classification (TC)

  7. 7.

Natural language inference (NLI)

  8. 8.

Multiple-choice machine reading comprehension (MRC)

  9. 9.

Extractive question answering (EQA)

  10. 10.

Machine translation (MT)

In Table 1, we present the task and dataset details. We mostly keep the
datasets in their originally published form (except for varieties renaming).
For NLI, we use the existing English test set of XNLI Conneau et al. (2018)
and construct a multilingual dialect-focused translated evaluation dataset. We
refer to this as translate-test NLI.

#### Evaluation Principles

On the ground level, we evaluate existing NLP systems on text-based tasks
using standard evaluation metrics (e.g., UAS for parsing, F1 for
classification tasks, BLEU for translation). At a global level, we believe a
sustainable NLP system should be user-focused while providing substantial (i)
linguistic utility and (ii) demographic utility Song et al. (2023); Blasi et
al. (2022). Blasi et al. (2022) defined the utility of a task and language, as
the corresponding performance normalized by the best possible performance
(usually human-level performance). Demographic utility considers the demand
for a language technology within a specific language, where the demand is
proportional to the number of speakers of that language. Linguistic utility,
on the other hand, asserts that “all languages are created equal” regardless
of the number of speakers, and hence all languages in the world should receive
identical weights.

Overall, we want to capture the performance gap between language clusters
(e.g., Anglic vs. Italian Romance) as well as within language clusters (e.g.,
Norwegian Bokmål vs. Nynorsk). To attain this, we define the performance gap
metrics in §3.3 . We also vary the experimental settings in §3.2, including
zero-shot and few-shot cross-lingual transfer, as well as fine-tuning with
similar high-resource languages. This is essential given that we lack clean
annotated data in many varieties.

##  3 Experiments

Here, we report the entire process involved in creating and evaluating
baselines for all of the tasks and varieties in DialectBench. Additionally, we
define a _dialectal gap_ metric to analyze performance disparities within and
across clusters.

###  3.1 Models

We evaluate using two multilingual models: mBERT Devlin et al. (2019) and
XLM-R Conneau et al. (2020) for all tasks except MT. For MT, we do zero-shot
evaluation with NLLB NLLB Team et al. (2022), using both the 600M and 1.3B
variants. In addition, we use Mistral 7B Jiang et al. (2023) to evaluate the
current capability of LLMs on multilingual and dialectal understanding tasks.
Our main goal is collating dialectal data across different languages and tasks
under a single platform, hence we do not optimize for the best model
performance. Rather, we focus on understanding and reporting the current state
of performance on all DialectBench varieties.

###  3.2 Training and Evaluation

Training and evaluation procedures are largely determined by the availability
of training or evaluation data for each task.

For any cluster $C$, let
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{C}}$
be the highest-resourced variety (which is usually the cluster representative)
of $C$. In addition, for any variety $v\in C$, we write
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}}$
for the highest-resourced variety, that is,
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}}={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{C}}$.
For any varieties $t$ and $v$, let
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{S}_{t}(v)}$
be the raw evaluation score of a system fine-tuned on $t$ and tested on $v$
(higher is better).

We use six general approaches for task-specific model training:

  1. 1.

In-variety fine-tuning
(${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{S}_{v}(v)}$):
In cases where there is available training data for a variety $v$, we fine-
tune the base model on $v$. This primarily applies to tasks such as POS
tagging and dependency parsing.

  2. 2.

In-cluster fine-tuning
(${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{S}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}}}(v)}$):
In-variety fine-tuning is quite resource-intensive when we have a large number
of varieties within a cluster $C$. In such cases, we fine-tune the base model
on
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{C}}$.
Then we evaluate this model on each variety $v\in C$. This is the most common
setting in our experiments, as it allows us to evaluate the dialectal
performance gap without increasing the computation cost. For the DId task, we
typically use a dataset of sentences annotated with variety labels to fine-
tune one dialect-identification model for each language cluster.

  3. 3.

Combined fine-tuning
(${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{S}_{\mathcal{L}}(v)}$):
Unlike in the previous two methods, where each training set contains data from
a single language cluster, we fine-tune our baseline question-answering (EQA
and MRC) models using the SD-QA Faisal et al. (2021) and Belebele Bandarkar et
al. (2023) datasets respectively, both of which contain training data in
multiple standard varieties only and test data in other varieties. The SD-QA
training data ($\mathcal{L}$ in the notation) contains questions in 9 standard
varieties ($\mathcal{L}=\\{\text{eng, ara, ben, fin, ind, swa, kor, rus,
tel}\\}$), while Belebele assembles data from 6 distinct multiple-choice QA
datasets in standard English ($\mathcal{L}=\\{\text{eng}\\}$).

Task |  In-variety FT |  In-cluster FT |  Combined FT |  Zero-shot |  No ref-erence |  In-context learning
---|---|---|---|---|---|---
DEP |  $\checkmark$ |  |  |  $\checkmark$ |  |
POS |  $\checkmark$ |  |  |  $\checkmark$ |  |
NER |  |  $\checkmark$ |  |  $\checkmark$ |  |
EQA |  |  |  $\checkmark$ |  $\checkmark$ |  |  $\checkmark$
MRC |  |  |  $\checkmark$ |  |  |
NLI |  |  |  |  $\checkmark$ |  |
TC |  |  $\checkmark$ |  |  $\checkmark$ |  |
SA |  |  $\checkmark$ |  |  |  |  $\checkmark$
DI |  |  $\checkmark$ |  |  |  |
MT |  |  |  |  $\checkmark$ |  $\checkmark$ |
Table 2: Task-specific training and evaluation procedure.

  4. 4.

Zero-shot evaluation
(${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{S}_{\text{eng}}(v)}$):
For certain varieties, obtaining training data even for in-cluster fine-tuning
can be a challenge. Fortunately, English training data is always available for
the datasets we study, so we use English to fill the gaps when we lack in-
variety, in-cluster, or combined training data. At the same time, we aim to
assess the feasibility of using this zero-shot cross-lingual transfer in
reducing any existing performance gap across varieties. So we perform zero-
shot cross-lingual transfer from English to each variety for 6 tasks in total.
We only leave out those tasks such as dialect identification that explicitly
require in-cluster training data.

  5. 5.

In-context learning
(${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{S}_{\text{icl}}(v)}$):
When evaluating large language models, we do not fine-tune them but instead
rely on prompting and in-context learning. For this, we provide instructions
and 5 examples in English as exemplars, followed by a prompt for predicting
the test examples. Employing Mistral 7B Jiang et al. (2023), we assess the
present effectiveness of a close-to-state-of-the-art LLM on language
varieties. The task-specific example prompts are reported in Appendix I.

Table 2 summarizes the task-specific training procedures that we employ based
on data availability. Note that, for MT, we perform zero-shot evaluation
specifically in the translation direction, (standard variety to English)
tested on (dialectal variety to English). But evaluation is a challenge
because human-created reference translations into or out of non-standard
varieties are usually very limited. Therefore, we adopt an evaluation protocol
from previous work (Alam et al., 2023) that uses pseudo-references. Given $x$,
a sentence in a variety and
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{x}}$,
the translation of $x$ into the standard variety, let $y$ be the output of the
MT system on input $x$ and
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{y}}$
be the output on input
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{x}}$.
Then we measure the quality of $y$ (using, e.g., BLEU) compared against
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{y}}$
as a pseudo-reference.

###  3.3 Quantifying the Dialectal Gap

To quantify the performance disparity across various resource-specific
settings, language clusters and varieties, we introduce a _dialect performance
gap_ metric
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{t}(u,v)}$:
the relative decrease in performance of a system fine-tuned on variety $t$,
tested on variety $v$ compared to a baseline variety $u$:

$\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{t}(u,v)}=\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{S}_{t}(u)}-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{S}_{t}(v)}}{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{S}_{t}(u)}}$
with a special case for in-variety fine-tuning:
$\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\text{in-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}variety}}}(u,v)}=\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{S}_{u}(u)}-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{S}_{v}(v)}}{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{S}_{u}(u)}}.$

For the baseline score, we use either the score on the standard variety for
each cluster
($u={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}}$),
or, in the zero-shot setting, the score on the language used for fine-tuning,
namely English ($u=t=\text{eng}$). Rather than computing an absolute gap, we
opt for a relative gap (i.e., dividing by the baseline score). We also
indicate whether the training setting is zero-shot ($t=\text{eng}$) or fine-
tuned on in-variety, in-cluster, or assembled data. Putting all these
together, we compute the following three variations of dialectal inequality.

  1. 1.

${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\text{eng}}(\text{eng},v)}$:
We calculate this metric to get a comprehensive measurement of global
disparity across all varieties in a resource-limited environment (zero-shot
transfer from English).

  2. 2.

${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\text{eng}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}},v)}$:
Using this variation, we keep the setting fixed as zero-shot and calculate the
gap between the representative variety and any other variety.

  3. 3.

${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{t}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}},v)}$:
The two aforementioned metrics shed light on the extent of the variety
performance gap in a resource-limited setting. To gain a more comprehensive
perspective, we additionally compute another metric, this time utilizing the
availability of resources. The computation approach remains as straightforward
as before. We just use fine-tuning on a variety $t$ instead of zero-shot
transfer from Standard English:
$t=\text{in-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}variety}}$
for in-variety fine-tuning,
$t={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}}$
for in-cluster fine-tuning, or $t$ is some set of varieties for combined fine-
tuning.

For all three
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}}$
metrics, we compute them at the variety level and then average them at the
cluster level:

$\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{t}(u,C)}$
$\displaystyle=\frac{1}{|C|}\sum_{v\in
C}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{t}(u,v)}$
$\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\text{in-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}variety}}}(u,C)}$
$\displaystyle=\frac{1}{|C|}\sum_{v\in
C}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\text{in-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}variety}}}(u,v)}.$

##  4 Results

|  | num. | num. | avg. |  |  |  |
---|---|---|---|---|---|---|---|---
Category | Task | cl. | var. | score | Max-score cluster/variety | Min-score cluster/variety
Structured prediction | DEP parsing | 16 | 40 | 64.3 | southwestern shifted romance/brazilian portuguese | 94.4 | tupi-guarani subgroup i.a/mbyá guaraní (a:brazil) | 9.0
POS tagging | 17 | 51 | 72.1 | norwegian/norwegian bokmål (m:written) | 98.7 | tupi-guarani subgroup i.a/mbyá guaraní (a:brazil) | 1.9
NER | 27 | 85 | 70.1 | eastern romance/romanian | 94.2 | anglic/jamaican creole english | 0.0
Sequence classification | NLI | 15 | 38 | 64.2 | anglic/english | 83.4 | sotho-tswana (s.30)/southern sotho | 34.6
TC | 15 | 38 | 77.7 | sinitic/classical-middle-modern sinitic (o:traditional) | 89.8 | kurdish/central kurdish | 19.4
DId | 6 | 49 | 67.0 | sinitic/mandarin chinese (a:taiwan, o:simplified) | 98.6 | southwestern shifted romance/portuguese (m:written) | 17.4
SA | 1 | 9 | 80.3 | arabic/tunisian arabic | 94.6 | arabic/south levantine arabic | 58.9
Qusetion Answering | MRC | 4 | 11 | 40.9 | anglic/english | 53.4 | sotho-tswana (s.30)/southern sotho | 29.0
EQA | 5 | 24 | 74.2 | arabic/arabic (a:saudi-arabia) | 77.9 | swahili/swahili (a:tanzania) | 63.5
Generation | MT-dialect | 12 | 73 | 25.2 | arabic/gulf arabic (a:riy) | 43.1 | common turkic/sakha | 2.5
MT-region | 2 | 41 | 33.0 | high german/central alemannic (a:ur) | 44.1 | italian romance/italian (a:sardegna) | 13.0
Table 3: Task specific result summary using Maximum Obtainable Score. The
varieties with the minimum scores exhibit a noticeable lag in performance
across various tasks when compared to the average task performance.

We, first of all, discuss results by highlighting the highest possible score
per variety, aka the maximum obtainable evaluation scores regardless of
evaluation method or training data. Next, we extend our discussion further by
reporting the existing dialectal disparity across clusters and varieties.

###  4.1 Maximum Obtainable Scores

Here we provide key findings from our evaluation on each task. A task-specific
summary is reported in Table 3. Detailed results comprising all tasks, models,
language clusters and varieties are reported in Tables 10 to LABEL:zero-shot-
mt in Appendix E.

Figure 3: Maximum scores (max. UAS) in Dep. Parsing task. Yellow-shaded
region: Komi is the only cluster having no varieties seen during mBERT
pertaining. Colored bars with diagonal stripes: the cluster representative
variety. Low-resourced cluster varieties score lower compared to high-resource
Germanic clusters.

#### Structured prediction

We present visualizations for the task-specific maximum scores. We show the
one for Dependency Parsing in Fig. 3, where we observe that low-resource
varieties from language clusters such as Tupi-Guarani (indigenous South
American cluster), Saami and Komi (low-resource Uralic language clusters) have
the lowest performance compared to Standard English and other closely related
Germanic clusters. These low-resource varieties are also not included in the
pretraining stage of our base language models (eg. mBERT). Furthermore, this
trend is evident across all three structured prediction tasks. On the other
hand, high-resource Indo-European languages such as Portuguese, French, and
Norwegian usually perform better.

#### Sequence classification

For DId and SA, we generally collate different datasets for each language
cluster and therefore, report the comparative classification results together.
As a result, the locality level (e.g. city/region/country) also varies across
cluster. For example, we report city-level DId results for Arabic and High
German but country-level results for Portuguese, Spanish and English. In the
case of SA, we have region/country-level results for Arabic varieties. For TC
and NLI, we have the same set of clusters and varieties. However, we only
report the zero-shot transfer performance from Standard English for NLI using
the newly created translate-test NLI dataset.

For TC and NLI, we observe the largest in-cluster disparity in the Kurdish
cluster, with Northern Kurdish outperforming all others. The Sotho varieties
consistently perform significantly lower compared to other clusters. For all
three sequence classification tasks, we generally find the Chinese cluster
performing on par with high-resource Latin counterparts.

#### Question answering

We generally do not see large gaps in performance within varieties in each
language cluster. In EQA zero-shot experiments, English and its varieties have
the highest performance overall and Korean varieties score the lowest.
Combined fine-tuning boosts performance on all language clusters except in
English. For MRC the performance threshold peaks at 53.4 for Standard English
and the minimum we get is 29 for Southern Sotho. More detailed results are
presented in Tables 19 and 15.

#### Machine translation

The performance gap here varies widely across and within language varieties.
Performance is similar within the Swiss-German cluster, with higher
performance (see Figure 5) across regions in Northern Switzerland, which is
geographically closer to Germany. The performance gap for Norwegian dialects
(Figure 10(b)) is surprising as we perform zero-shot transfer from Norwegian
Nynorsk (a Western dialect) but obtain better performance on the Eastern
dialect. Within Arabic, Riyadh is the highest performer while Sfax performs
the worst. For the Bengali cluster, Jessore has the highest performance –this
is not surprising since it is one of the dialects from which standard Bengali
originated Alam et al. (2023). The Ethiopian variety of Tigrinya exhibits a
higher performance than the Eritrean one, even though Tigrinya is more
commonly spoken in Eritrea 555https://en.wikipedia.org/wiki/Tigrinya_language.
Amongst the clusters within the Basque cluster, Barkoxe and Maule have the
lowest score while Azkaine scores the highest.

###  4.2 Dialectal Gap Across Language Clusters

(a) Dependency parsing

(b) Topic classification

(c) Extractive question answering

Figure 4: Dialectal gap visualization for language clusters utilizing zero-
shot cross-lingual transfer from Standard English. In the x-axis, values far
from zero have a larger performance gap from English whereas, in the y-axis,
values far from zero have a larger within cluster gap. Ideally, we want both
of them to be close to zero.

In Fig. 4, we plot the zero-shot dialectal gap for three tasks. In the
$x$-axis, we report the aggregated cluster-level gap
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\text{eng}}(\text{eng},v)}$,
compared against the fine-tuning variety (standard English) while in $y$-axis
we report
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\text{eng}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}},v)}$,
the gap compared against the representative variety of a cluster. In an ideal
scenario, we would want both of these gap values to be close to zero. However,
this is certainly not the case. The general observed trend is that the low-
resource clusters have higher gaps of both
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\text{eng}}(\text{eng},C)}$
and
$\mathcal{G}_{\text{eng}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}},C)$,
whereas high-resource Germanic and Sinitic language clusters consistently
exhibit low dialectal gaps. That said, certain specific high-resource
varieties, such as Standard German and its dialectal counterparts like Swiss
German, showcase significant within-cluster dialectal gaps (Fig. 4(a)).

Task | Gap metric | Avg. val |  cluster (max) |  cluster (min)
---|---|---|---|---
DEP parsing | ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\text{eng}}(\text{eng},C)}$ | 34.0 | arabic, 50.8 | sw. shift. romance, 19.4
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\text{eng}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}},C)}$ | 15.7 | anglic, 34.2 | sw. shift. romance, $-0.9$
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\text{in-variety}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}},C)}$ | 26.4 | arabic, 93.8 | italian romance, 0.6
POS tagging | ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\text{eng}}(\text{eng},C)}$ | 27.4 | arabic, 58.5 | norwegian, 14.7
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\text{eng}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}},C)}$ | 6.7 | anglic, 20.9 | ew. armenian, $-2.1$
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\text{in-variety}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}},C)}$ | 6.2 | arabic, 29.1 | neva, $-0.5$
NER | ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\text{eng}}(\text{eng},C)}$ | 31.7 | kurdish, 77.1 | modern dutch, 12.3
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\text{eng}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}},C)}$ | 22.3 | kurdish, 78.2 | hindustani, $-23.6$
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}},C)}$ | $-28.6$ | kurdish, 91.5 | sorbian, -1162.7
TC | ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\text{eng}}(\text{eng},C)}$ | 22.4 | kurdish, 74.2 | sinitic, 0.4
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\text{eng}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}},C)}$ | 12.5 | kurdish, 60.6 | norwegian, $-2.2$
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}},C)}$ | $-1.9$ | latvian, 21.0 | kurdish, $-61.3$
Table 4: Comparative cluster-level dialectal gap across tasks. In general, the
average disparity is larger for zero-shot transfer
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\text{eng}}(\text{eng},C)}$.
However, when we move from zeroshot to finetune (i.e.
$\mathcal{G}_{\text{eng}}\rightarrow\mathcal{G}_{\bar{v}/\text{in-variety}}$)
and compute the distance from a cluster representative
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}}$,
we observe increased dialectal disparity
$|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}},C)}|$.

We primarily report dialectal gaps using zero-shot transfer because the
finetuning data available across task and cluster is very disproportionate.
Often the in-cluster/variety data is not good enough in terms of data quality
and quantity. For example, we have 37 varieties in 13 clusters for dependency
parsing but out of these, only 20 varieties have data available for in-variety
fine-tuning. This lacking becomes more apparent when we compare the statistics
of two types of within-cluster dialectal gaps: zero-shot
$\mathcal{G}_{\text{eng}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}},C)$
against fine-tuning $\mathcal{G}_{\bar{v}/\text{in-
variety}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}},C)$
in Table 4. In general, the within-cluster dialectal disparity is smaller for
zero-shot transfer (i.e.
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\text{eng}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}},C)}\leq|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{G}_{\bar{v}/\text{in-
variety}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bar{v}},C)}|$).
Here, the in-cluster/variety fine-tuning results in a higher performance
deviation primarily due to the inconsistent variety-specific finetuning data
quality.

##  5 Discussion

#### High resource vs. low resource varieties

The highest-performing varieties are mostly standard high-resource languages
and a few high-resource dialects (Norwegian dialects) whereas, the majority of
the lowest-performing language variants are low-resourced varieties. This
clear distinction of language varieties points towards the large existence of
in-cluster dialectal gaps. Furthermore, this finding correlates with language
script differences. We observe that 77.2% top-10 varieties in terms of maximum
obtainable score are written with Latin script. Another finding is the
performance instability of low-resource varieties across tasks. For instance,
Old Guarani performs better in DEP parsing whereas, Mbyá Guarani (Paraguay)
surpasses it in POS tagging even though the dataset remains the same (i.e.
UD). For more detailed comparisons, in Appendix Table 21, we report the top-10
highest and lowest-scoring varieties across different tasks.

| zero-shot | few-shot / FT
---|---|---
Task | mBERT | XLM-R | mBERT | XLM-R
DEP. Parsing | 61.6 | 61.3 | 76.2 | 64.3
POS Tagging | 69.5 | 69.7 | 89.8 | 89.1
NER | 59.7 | 57.8 | 65.8 | 61.4
NLI | 56.9 | 62.5 | — | —
TC | 72.3 | 71.4 | 73.1 | 68.9
MRC | — | — | 39.4 | 40.3
EQA | 53.9 | 51.9 | 69.2 | 67.2
SA | — | — | 78.8 | 80.1
DId | — | — | 65.8 | 59.3
win | 4/6 | 2/6 | 6/8 | 2/8
Table 5: Base model comparison. We found mBERT was easier to fine-tune using
the default hyperparameter setting thus, resulting in a higher winning rate.

#### Model hyperparameter tuning

In Table 5, we compare the baseline multilingual models. mBERT was
comparatively easier to train than XLM-R using the default learning rates
reported in the earlier task experiments. Often for tasks such as MRC, we
needed to tune hyperparameters (e.g. learning rate, max. sequence length) in
case of XLM-R. However, once we identify a hyperparameter configuration that
converges in a zero-shot setting, we use the same to train all the language-
variant training for that specific task. As a result, for some low-resource
languages, XLM-R does not converge compared to mBERT. For example, XLM-R
dependency parsing UAS score for Norwegian-NynorskLIA is 56.08 (zeroshot) and
8.25 (in-variety FT) whereas, we get 78.39 for in-variety mBERT fine-tuning.
We suspect this hyperparameter tuning issue is one of the contributing factors
toward a lower winning rate of XLM-R in few-shot / fine-tuning settings.
However, this could be improved further with an extensive parameter grid
search and settings specifically tailored for each language cluster.

#### Positive zero-shot transfer for Latin varieties

Low-resource varieties written in Latin script receive greater benefit in
zero-shot because of effective transfer from high-resource Standard English.
With the presence of in-cluster/variety finetuning data, we effectively
diminish this script effect to some extent. For example in the Hindi cluster,
Fiji performs better than its Latin non-standard counterparts with in-cluster
finetuning for NER (LABEL:tab:ner_full). In summary, if the standard variety
of a cluster is non-latin but high-resource, then the success rate of in-
cluster/variety fine-tuning tends to be higher. However, where all dialects
are low-resource, Latin script varieties utilizing zero-shot transfer,
eventually surpass others in the performance hierarchy. As an example, we
report the zero-shot NER instances where the low-resource varieties perform
better than the representative ones in Appendix Table 22 (most of these use
Latin script).

#### LLM evaluation via In-context learning

For SA and EQA tasks, we have in-context learning results using the Mistral7B
LLM. Comparing the performance against zero-shot and fine-tuning using our
encoder-based models, we find dialect performance of LLM is better than zero-
shot transfer but falls behind the finetuned results. On top of that, data
contamination Ahuja et al. (2023) during LLM evaluation is another existing
issue while considering these few available dialectal resources. Creating
translation-based comparable data might be a solution to perform a fair
benchmarking of LLMs on low-resource varieties.

#### Misinterpreting evaluation metrics

We also report the cluster-level population-weighted average (i.e. demographic
utility) which rewards a system more when it provides increased linguistic
utility (eg. raw F1 score) for varieties, spoken by a larger population
compared to varieties spoken by a smaller population. Alone, this metric could
be misleading if we consider the fact that the performance gap among all
varieties from a particular cluster should be minimal. On the other hand,
solely looking into the linguistic utility average does not give a clear
picture either (e.g. often overshadows the larger performance deviation of
certain varieties having extreme scores). So for all clusters and tasks, we
report the linguistic utility average as well as the demographic utility
average, the minimum score of a cluster, and the standard deviation in Tables
23 to 30.

##  6 Conclusion

We propose DialectBench, the first-ever inclusive Dialectal NLP benchmark
reporting performance evaluation and disparity across standard and non-
standard varieties. This is one step towards the effort of bringing more and
more language under the paradigm of NLP technology. We would like to further
improve the benchmark, constructing high-quality comparable data and expanding
the task coverage to speech-based NLP technologies.

## Limitations

The data quality and quantity, variety coverage vary significantly across
tasks because of data scarcity issues. We avoid full-scale llm-evaluation
consciously because of the uncertain data-contamination issue and their well-
known lower performance threshold compared to smaller masked-language-
modeling-based fine-tuned models. In addition, we focus on text-based NLP
tasks for this current iteration. Moreover, we do not claim the representative
varieties of each language clusters to be any kind of superior or standardized
forms over the other varieties. These varieties are chosen to perform a well-
informed comparison among the perceived well-resourced linguistic variety and
its counterparts having lesser data availability. At the same time, the mutual
intelligibility and phylogenetic similarity of the similar cluster varieties
also vary across cluster and this was not selected in a numerically
quantifiable manner.

#### Evaluation Limitations

To further improve the evaluation fairness of the current version of
DialectBench, we need (i) Parallel corpus utilization to prepare task-specific
data (ii) Translation-based task data generation to perform comparable
analysis (iii) Quantifying the resource-supply and demand as well as
population-coverage Song et al. (2023) to identify where a variety stands in
the global landscape of linguistic utility. Here, we have accumulated data for
diverse varieties across tasks that vary significantly in terms of quality,
example count, and domain. However, to perform a perfectly fair comparison of
dialectal inequality, we should consider high-quality comparable data (e.g.
parallel corpus, similar varieties across tasks) which is not available at
this point.

#### Continuity of DialectBench

Despite our best effort, this benchmark does not include every one of the
already published task-specific dialectal datasets. So, our next steps on this
project involve hosting the benchmark on the website that displays the current
statistics of the datasets in DialectBench. We will also encourage researchers
to add new and existing datasets for tasks, language clusters that might be
currently missing alongside the respective baselines.

#### Space Limitations

Our study encompasses a large set of evaluation result tables, their
corresponding visualizations and findings analysis. Due to space limitations,
we have to move the detailed reports (Appendix E) and the rest of the
visualizations (Appendix D) in the Appendix. To better assist, we include an
Appendix Table of Content (Table 6) at the introductory section of Appendix
(Appendix).

## Ethics Statement

This work is a compilation of existent dialectal datasets across different
tasks, including structured prediction and generative tasks. Our experiments
do not particularly optimize for the best model performance of these tasks.
Therefore we acknowledge that for some of the tasks, the baseline models might
not be robust enough to handle dialectal text hence resulting in wrong
predictions and generations. We believe that this underscores the need for
building models robust to different language variations and future work should
focus on this.

## References

  * Abdelli et al. (2019) Adel Abdelli, Fayçal Guerrouf, Okba Tibermacine, and Belkacem Abdelli. 2019.  Sentiment analysis of Arabic Algerian dialect using a supervised method.  In _2019 International Conference on Intelligent Systems and Advanced Computing Sciences (ISACS)_ , pages 1–6.
  * Abdul-Mageed et al. (2021) Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021.  ARBERT & MARBERT: Deep bidirectional transformers for Arabic.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7088–7105, Online. Association for Computational Linguistics.
  * Adelani et al. (2023) David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, and Annie En-Shiun Lee. 2023.  SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects.
  * Ahuja et al. (2023) Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. 2023.  Megaverse: Benchmarking large language models across languages, modalities, models and tasks.
  * Al Omari et al. (2019) Marwan Al Omari, Moustafa Al-Hajj, Nacereddine Hammami, and Amani Sabra. 2019.  Sentiment classifier: Logistic regression for Arabic services’ reviews in Lebanon.  In _2019 International Conference on Computer and Information Sciences (ICCIS)_ , pages 1–5.
  * Alam et al. (2023) Md Mahfuz Ibn Alam, Sina Ahmadi, and Antonios Anastasopoulos. 2023.  CODET: A benchmark for contrastive dialectal evaluation of machine translation.
  * Alomari et al. (2017) Khaled Mohammad Alomari, Hatem M. ElSherif, and Khaled Shaalan. 2017.  Arabic tweets sentimental analysis using machine learning.  In _Advances in Artificial Intelligence: From Theory to Practice_ , pages 602–610, Cham. Springer International Publishing.
  * Alqahtani et al. (2022) Dhuha Alqahtani, Lama Alzahrani, Maram Bahareth, Nora Alshameri, Hend Al-Khalifa, and Luluh Aldhubayi. 2022.  Customer sentiments toward Saudi banks during the Covid-19 pandemic.  In _Proceedings of the 5th International Conference on Natural Language and Speech Processing (ICNLSP 2022)_ , pages 251–257, Trento, Italy. Association for Computational Linguistics.
  * Bandarkar et al. (2023) Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2023.  The Belebele benchmark: a parallel reading comprehension dataset in 122 language variants.  _arXiv preprint arXiv:2308.16884_.
  * Blaschke et al. (2023) Verena Blaschke, Hinrich Schütze, and Barbara Plank. 2023.  Does manipulating tokenization aid cross-lingual transfer? A study on POS tagging for non-standardized languages.  In _Proceedings of the Tenth Workshop on NLP for Similar Languages, Varieties and Dialects_ , pages 40–54, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Blasi et al. (2022) Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2022.  Systematic inequalities in language technology performance across the world’s languages.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5486–5505, Dublin, Ireland. Association for Computational Linguistics.
  * Blodgett et al. (2018) Su Lin Blodgett, Johnny Wei, and Brendan O’Connor. 2018.  Twitter Universal Dependency parsing for African-American and mainstream American English.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1415–1425, Melbourne, Australia. Association for Computational Linguistics.
  * Bouamor et al. (2018) Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018.  The MADAR Arabic dialect corpus and lexicon.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Chambers and Trudgill (1998) Jack K. Chambers and Peter Trudgill. 1998.  _Dialectology_.  Cambridge University Press.
  * Clark et al. (2020) Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.  TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages.  _Transactions of the Association for Computational Linguistics_ , 8:454–470.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Elmadany et al. (2023) AbdelRahim Elmadany, ElMoatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023.  ORCA: A challenging benchmark for Arabic language understanding.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 9559–9586, Toronto, Canada. Association for Computational Linguistics.
  * Faisal et al. (2021) Fahim Faisal, Sharlina Keshava, Md Mahfuz Ibn Alam, and Antonios Anastasopoulos. 2021.  SD-QA: Spoken dialectal question answering for the real world.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 3296–3315, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Fourati et al. (2021) Chayma Fourati, Hatem Haddad, Abir Messaoudi, Moez BenHajhmida, Aymen Ben Elhaj Mabrouk, and Malek Naski. 2021.  Introducing a large Tunisian Arabizi dialectal dataset for sentiment analysis.  In _Proceedings of the Sixth Arabic Natural Language Processing Workshop_ , pages 226–230, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  * Garouani and Kharroubi (2022) Moncef Garouani and Jamal Kharroubi. 2022.  MAC: An open and free Moroccan Arabic corpus for sentiment analysis.  In _Innovations in Smart Cities Applications Volume 5_ , pages 849–858, Cham. Springer International Publishing.
  * Hämäläinen et al. (2021) Mika Hämäläinen, Khalid Alnajjar, Niko Partanen, and Jack Rueter. 2021.  Finnish dialect identification: The effect of audio and text.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8777–8783, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Hammarström and Forkel (2022) Harald Hammarström and Robert Forkel. 2022.  Glottocodes: Identifiers linking families, languages and dialects to comprehensive reference information.  _Semantic Web Journal_ , 13(6):917–924.
  * Hedderich et al. (2021) Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Strötgen, and Dietrich Klakow. 2021.  A survey on recent approaches for natural language processing in low-resource scenarios.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2545–2568, Online. Association for Computational Linguistics.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 4411–4421. PMLR.
  * Jauhiainen et al. (2022) Tommi Jauhiainen, Heidi Jauhiainen, and Krister Lindén. 2022.  Italian language and dialect identification and regional French variety detection using adaptive naive Bayes.  In _Proceedings of the Ninth Workshop on NLP for Similar Languages, Varieties and Dialects_ , pages 119–129, Gyeongju, Republic of Korea. Association for Computational Linguistics.
  * Jauhiainen et al. (2019) Tommi Jauhiainen, Krister Lindén, and Heidi Jauhiainen. 2019.  Discriminating between Mandarin Chinese and Swiss-German varieties using adaptive language models.  In _Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects_ , pages 178–187, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7B.  arXiv:2310.06825.
  * Johansen (2019) Bjarte Johansen. 2019.  Named-entity recognition for Norwegian.  In _Proceedings of the 22nd Nordic Conference on Computational Linguistics, NoDaLiDa_.
  * Kantharuban et al. (2023) Anjali Kantharuban, Ivan Vulić, and Anna Korhonen. 2023.  Quantifying the dialect gap and its correlates across languages.
  * Khanuja et al. (2020) Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, and Monojit Choudhury. 2020.  GLUECoS: An evaluation benchmark for code-switched NLP.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3575–3585, Online. Association for Computational Linguistics.
  * Lent et al. (2023) Heather Lent, Kushal Tatariya, Raj Dabre, Yiyi Chen, Marcell Fekete, Esther Ploeger, Li Zhou, Hans Erik Heje, Diptesh Kanojia, Paul Belony, Marcel Bollmann, Loïc Grobol, Miryam de Lhoneux, Daniel Hershcovich, Michel DeGraff, Anders Søgaard, and Johannes Bjerva. 2023.  CreoleVal: Multilingual multitask benchmarks for creoles.
  * Liang et al. (2020) Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020.  XGLUE: A new benchmark dataset for cross-lingual pre-training, understanding and generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6008–6018, Online. Association for Computational Linguistics.
  * Littauer (n.d.) Richard Littauer. n.d.  Low resource languages: Resources for conservation, development, and documentation of low resource (human) languages.  https://github.com/RichardLitt/low-resource-languages.  [Accessed 15-11-2023].
  * Medhaffar et al. (2017) Salima Medhaffar, Fethi Bougares, Yannick Estève, and Lamia Hadrich-Belguith. 2017.  Sentiment analysis of Tunisian dialects: Linguistic ressources and experiments.  In _Proceedings of the Third Arabic Natural Language Processing Workshop_ , pages 55–61, Valencia, Spain. Association for Computational Linguistics.
  * Mirzakhalov (2021) Jamshidbek Mirzakhalov. 2021.  _Turkic Interlingua: A Case Study of Machine Translation in Low-resource Languages_.  Ph.D. thesis, University of South Florida.
  * Mirzakhalov et al. (2021a) Jamshidbek Mirzakhalov, Anoop Babu, Duygu Ataman, Sherzod Kariev, Francis Tyers, Otabek Abduraufov, Mammad Hajili, Sardana Ivanova, Abror Khaytbaev, Antonio Laverghetta Jr., Bekhzodbek Moydinboyev, Esra Onal, Shaxnoza Pulatova, Ahsan Wahab, Orhan Firat, and Sriram Chellappan. 2021a.  A large-scale study of machine translation in Turkic languages.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5876–5890, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Mirzakhalov et al. (2021b) Jamshidbek Mirzakhalov, Anoop Babu, Duygu Ataman, Sherzod Kariev, Francis Tyers, Otabek Abduraufov, Mammad Hajili, Sardana Ivanova, Abror Khaytbaev, Antonio Laverghetta Jr, et al. 2021b.  A large-scale study of machine translation in turkic languages.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5876–5890.
  * Nabil et al. (2015) Mahmoud Nabil, Mohamed Aly, and Amir Atiya. 2015.  ASTD: Arabic sentiment tweets dataset.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 2515–2519, Lisbon, Portugal. Association for Computational Linguistics.
  * NLLB Team et al. (2022) NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022.  No Language Left Behind: Scaling human-centered machine translation.  arXiv:2207.04672.
  * Nordhoff (2012) Sebastian Nordhoff. 2012.  Linked data for linguistic diversity research: Glottolog/langdoc and asjp.  In Christian Chiarcos, Sebastian Nordhoff, and Sebastian Hellmann, editors, _Linked Data in Linguistics. Representing and Connecting Language Data and Language Metadata_ , pages 191–200. Springer, Heidelberg.
  * Pan et al. (2017) Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017.  Cross-lingual name tagging and linking for 282 languages.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1946–1958, Vancouver, Canada. Association for Computational Linguistics.
  * Park et al. (2021) Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Ji Yoon Han, Jangwon Park, Chisung Song, Junseong Kim, Youngsook Song, Taehwan Oh, Joohong Lee, Juhyun Oh, Sungwon Lyu, Younghoon Jeong, Inkwon Lee, Sangwoo Seo, Dongjun Lee, Hyunwoo Kim, Myeonghwa Lee, Seongbo Jang, Seungwon Do, Sunkyoung Kim, Kyungtae Lim, Jongwon Lee, Kyumin Park, Jamin Shin, Seonghyun Kim, Lucy Park, Lucy Park, Alice Oh, Jung-Woo Ha (NAVER AI Lab), Kyunghyun Cho, and Kyunghyun Cho. 2021.  KLUE: Korean language understanding evaluation.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks_ , volume 1. Curran.
  * Rademaker et al. (2017) Alexandre Rademaker, Fabricio Chalub, Livy Real, Cláudia Freitas, Eckhard Bick, and Valeria de Paiva. 2017.  Universal Dependencies for Portuguese.  In _Proceedings of the Fourth International Conference on Dependency Linguistics (Depling)_ , pages 197–206, Pisa, Italy.
  * Rahimi et al. (2019) Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019.  Massively multilingual transfer for NER.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 151–164, Florence, Italy. Association for Computational Linguistics.
  * Rios (2020) Anthony Rios. 2020.  FuzzE: Fuzzy fairness evaluation of offensive language classifiers on African-American English.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(01):881–889.
  * Ruder et al. (2023) Sebastian Ruder, Jonathan H. Clark, Alexander Gutkin, Mihir Kale, Min Ma, Massimo Nicosia, Shruti Rijhwani, Parker Riley, Jean-Michel A. Sarr, Xinyi Wang, John Wieting, Nitish Gupta, Anna Katanova, Christo Kirov, Dana L. Dickinson, Brian Roark, Bidisha Samanta, Connie Tao, David I. Adelani, Vera Axelrod, Isaac Caswell, Colin Cherry, Dan Garrette, Reeve Ingle, Melvin Johnson, Dmitry Panteleev, and Partha Talukdar. 2023.  XTREME-UP: A user-centric scarce-data benchmark for under-represented languages.
  * Ruder et al. (2021) Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021.  XTREME-R: Towards more challenging and nuanced multilingual evaluation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10215–10245, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Sababa and Stassopoulou (2018) Hanna Sababa and Athena Stassopoulou. 2018.  A classifier to distinguish between Cypriot Greek and Standard Modern Greek.  In _2018 Fifth International Conference on Social Networks Analysis, Management and Security (SNAMS)_ , pages 251–255.
  * Scherrer et al. (2019) Yves Scherrer, Tanja Samardžić, and Elvira Glaser. 2019.  Digitising Swiss German: how to process and study a polycentric spoken language.  _Language Resources and Evaluation_ , 53.
  * Seelawi et al. (2021) Haitham Seelawi, Ibraheem Tuffaha, Mahmoud Gzawi, Wael Farhan, Bashar Talafha, Riham Badawi, Zyad Sober, Oday Al-Dweik, Abed Alhakim Freihat, and Hussein Al-Natsheh. 2021.  ALUE: Arabic language understanding evaluation.  In _Proceedings of the Sixth Arabic Natural Language Processing Workshop_ , pages 173–184, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  * Song et al. (2023) Yueqi Song, Catherine Cui, Simran Khanuja, Pengfei Liu, Fahim Faisal, Alissa Ostapenko, Genta Indra Winata, Alham Fikri Aji, Samuel Cahyawijaya, Yulia Tsvetkov, Antonios Anastasopoulos, and Graham Neubig. 2023.  GlobalBench: A benchmark for global progress in natural language processing.
  * Wang et al. (2017) Hongmin Wang, Yue Zhang, GuangYong Leonard Chan, Jie Yang, and Hai Leong Chieu. 2017.  Universal Dependencies parsing for colloquial Singaporean English.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1732–1744, Vancouver, Canada. Association for Computational Linguistics.
  * Wang et al. (2022) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022.  Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5085–5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Wilie et al. (2020) Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti. 2020.  IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pages 843–857, Suzhou, China. Association for Computational Linguistics.
  * Zampieri et al. (2023) Marcos Zampieri, Kai North, Tommi Jauhiainen, Mariano Felice, Neha Kumari, Nishant Nair, and Yash Bangera. 2023.  Language variety identification with true labels.
  * Zeman et al. (2021) Daniel Zeman, Joakim Nivre, Mitchell Abrams, Elia Ackermann, Noëmi Aepli, Hamid Aghaei, Željko Agić, Amir Ahmadi, Lars Ahrenberg, Ajede, and et al. 2021.  Universal Dependencies 2.9.  LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University.
  * Ziems et al. (2022) Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Anderson, and Diyi Yang. 2022.  VALUE: Understanding dialect disparity in NLU.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3701–3720, Dublin, Ireland. Association for Computational Linguistics.
  * Ziems et al. (2023) Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023.  Multi-VALUE: A framework for cross-dialectal English NLP.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 744–768, Toronto, Canada. Association for Computational Linguistics.

## Appendix

In this supplementary material, we provide the following: (i) Relevant
literature review (ii) Overall results and dataset details that we could not
fit into the main body of the paper.

Section | Desctiption
---|---
Appendix A | Related Work
Appendix B | Tasks of DialectBench
• Translate-Test NLI Dataset Statistics (LABEL:{tab:nli_data})
Appendix C |  Varieties and Clusters of DialectBench
• DialectBench Variety list (LABEL:lang_list)
• Language Clusters and Representative Varieties (LABEL:stdv)
Appendix D | Result Visualizations
| Regional maps with aggregated Machine Translation scores (Figs. 5 to 6)
---
• Map of Switzerland with aggregated BLEU scores of Swiss-German variety per
region (Fig. 5)
• Map of Italy with aggregated BLEU scores of Italian variety per region (Fig.
6)
Task Specific Plot for Maximum Scores (Figs. 7 to 10)
• Parts-of-Speech Tagging (Fig. 7(a))
• Named entity recognition (Fig. 7(b))
• Topic classification (Fig. 7(c))
• Natural language inference (Fig. 8(a))
• Extractive question answering (Fig. 8(b))
• Multiple-choice machine reading comprehension (Fig. 8(c))
• Sentiment analysis (Fig. 9(a))
• Dialect identification (Fig. 9(b))
• Machine translation (MT-region) (Fig. 10(a))
• Machine translation (MT-dialect) (Fig. 10(b))
Dialectal Gap visualization for language clusters utilizing zero-shot cross-
lingual transfer from Standard English.
• Parts-of-Speech Tagging (Fig. 11(a))
• Named entity recognition (Fig. 11(b))
• Dialect identification (Fig. 11(c))
Appendix E | Task Specific Evaluation Result Tables
|  • Dependency parsing (Table 10)
---
• Parts-of-Speech tagging (Table 11)
• Named entity recognition (LABEL:tab:ner_full)
• Natural language inference (Table 13)
• Extractive question answering (Tables 14 to 15)
• Dialect identification (Table 16)
• Topic classification (Table 17)
• Sentiment analysis (Table 18)
• Multiple-choice machine reading comprehension (Table 19)
• Machine translation (LABEL:zero-shot-mt)
Appendix F | Highest performing and lowest performing varieties (LABEL:{tab:high-lang})
Appendix G | Low-resource Variety performing better in zero-shot NER (Table 22)
Appendix H |  Cluster level Result Summaries with Demographic Utility and Standard Deviation report (Tables 23 to 31)
Appendix I | In-Context Learning Details
Table 6: Table of contents for supplementary material.

##  Appendix A Related Work

The majority of multilingual benchmarks Hu et al. (2020); Ruder et al. (2023,
2021); Liang et al. (2020); Wilie et al. (2020); Park et al. (2021) have
heavily focused on dominant language varieties. However, the performance
disparity between standard languages and their dialectal counterparts has been
studied Kantharuban et al. (2023); Ziems et al. (2022); Rios (2020); Ziems et
al. (2023) and shown to be significantly large across different tasks. Still,
the large-scale generalization of this finding comprising numerous task
categories is yet to be done. There have been previous efforts to bridge this
gap though. Some work has focused on curating dialectal datasets across
several tasks within one language cluster, while others have focused on single
tasks across many language clusters. For instance, ORCA Elmadany et al.
(2023), ARLUE Abdul-Mageed et al. (2021) and ALUE Seelawi et al. (2021) are
dedicated natural language understanding benchmarks focusing on Arabic
varieties alone. Multi-VALUE Ziems et al. (2023) was developed for
benchmarking NLP tasks in English varieties. Mirzakhalov et al. (2021a) is a
suite of resources for benchmarking MT in several Turkic languages. There has
also been a large body of work on dialect identification across several
languages Jauhiainen et al. (2022); Hämäläinen et al. (2021). Recently CODET
Alam et al. (2023) was released as a contrastive dialectal MT benchmark
covering 882 different variations from nine different languages. To the best
of our knowledge, we are the first to do a large scale aggregation of
dialectal data across several language clusters and tasks.

##  Appendix B Tasks of DialectBench

DialectBench includes 10 NLP tasks falling into four broader categories:
structured prediction, sentence classification, question answering, and text
generation. In Table 1, we present statistics for the datasets for each task
and briefly discuss each task below.

#### Dependency parsing

For the dependency parsing task, we include only those Universal Dependencies
(UD) Zeman et al. (2021) languages that have dialectal data. Beyond the data
available in UD 2.12, we incorporate two additional datasets for African-
American English (AAVE) Twitter data Blodgett et al. (2018) and Singlish Wang
et al. (2017). To make these two datasets compatible with the UD processing
pipeline, we replace the original dependency labels with the labels
corresponding to the official UD formalism.

#### Part of speech (POS) tagging

We use the same UD languages for POS tagging that we used for dependency
parsing. At the same time, we use the POS data instances from Singlish Wang et
al. (2017). Moreover, we include six Finnish dialects, four Arabic dialects
and Occitan through the UPOS label standardized pipeline proposed by Blaschke
et al. (2023).

#### Named entity recognition (NER)

We use data from the 176-language version of the Wikiann dataset processed by
Rahimi et al. (2019). All these languages provide both training and test data.
In addition, we include dialectal data from the original Wikiann dataset (282
languages) Pan et al. (2017) for evaluation. Moreover, we include three
Norwegian dialects Johansen (2019) with train, test and validation datasets
that use a slightly different set of NER tags (GEO, ORG, OTH, PER) compared to
the one we use in Wikiann (LOC, ORG, PER). We leave these levels as it is and
do not convert to the Wikiann tags.

#### Dialect identification

We include dialect identification experiments on Arabic, Greek, Portuguese,
English, Spanish, and Swiss German dialectal datasets. In these datasets, we
find large variations in the level of granularity with which dialects are
classified. For instance, the MADAR corpus differentiates Arabic varieties at
the city level Bouamor et al. (2018), whereas our English and Spanish datasets
are labeled with country names Zampieri et al. (2023).

#### Sentiment classification

Here we include several different Arabic varieties. Like other dialectal
datasets, these datasets do not follow one standard labeling process. However,
all datasets contain two main sentiment types: positive and negative. A number
of datasets contain additional labels such as “objective” or “neutral.” In our
setting, we perform a further split of data to provide validation data for
each dialect. However, we do not remove these extra labeled data for
information preservation.

#### Topic classification

We use the SIB-200 dataset Adelani et al. (2023) for topic classification task
SIB-200 was constructed from the FLORES-200 translation datasets. The authors
annotated the English dataset of FLORES-200 with 6 topic labels and then
further propagated these labels to the translated instances for all other
languages. For our case of benchmarking dialectal segments, we use the
dialectal and regional varieties from SIB-200.

|  | Variety | # Sentences
---|---|---|---
cluster | Language code |  |
anglic | eng_Latn | english | 5010
arabic | acm_Arab | north mesopotamian arabic | 5010
acq_Arab | ta’izzi-adeni arabic | 5010
aeb_Arab | tunisian arabic | 5010
ajp_Arab | south levantine arabic | 5010
apc_Arab | levantine arabic (a:north) | 5010
arb_Arab | standard arabic | 5010
ars_Arab | najdi arabic | 5010
ary_Arab | moroccan arabic | 5010
arz_Arab | egyptian arabic | 5010
common turkic | azb_Arab | south azerbaijani | 5010
azj_Latn | north azerbaijani | 5010
tur_Latn | central oghuz (m:spoken) | 5010
gallo-italian | lij_Latn | ligurian | 5010
lmo_Latn | lombard | 5010
vec_Latn | venetian | 5010
gallo-rhaetian | fur_Latn | friulian | 5010
high german | lim_Latn | limburgan | 5010
ltz_Latn | luxemburgish | 5010
italian romance | ita_Latn | italian | 5010
scn_Latn | sicilian | 5010
kurdish | ckb_Arab | central kurdish | 5010
kmr_Latn | northern kurdish | 5010
latvian | ltg_Latn | east latvian | 5010
lvs_Latn | latvian | 5010
modern dutch | nld_Latn | dutch | 5010
norwegian | nno_Latn | norwegian nynorsk (m:written) | 5010
nob_Latn | norwegian bokmål (m:written) | 5010
sardo-corsican | srd_Latn | sardinian | 5010
sinitic | yue_Hant | cantonese | 5010
zho_Hans | classical-middle-modern sinitic (o:simplified) | 5010
zho_Hant | classical-middle-modern sinitic (o:traditional) | 5010
sotho-tswana (s.30) | nso_Latn | northern sotho | 5010
sot_Latn | southern sotho | 5010
southwestern shifted romance | glg_Latn | galician | 5010
oci_Latn | occitan | 5010
por_Latn | portuguese (a:european) | 5010
spa_Latn | spanish | 5010
Table 7: Data statistics for newly created translate-test natural language
inference (NLI) dataset. We prepare this translate-test NLI dataset by
translating XNLI Conneau et al. (2018) english evaluation dataset.

#### Natural language inference

For the natural language inference task, there is no existing dataset with
varieties. So we use the existing English test set of XNLI Conneau et al.
(2018) and construct a multilingual dialect-focused translated evaluation
dataset. We use a state-of-the-art machine translation model (NLLB-200 3B) to
translate the English test set to 12 language clusters encompassing 40
varieties (Complete data statistics are reported in Table 7). After that, we
perform zero-shot cross-lingual transfer from the English finetuned NLI model.
We refer to this setting as _translate-test_.

#### Multiple-choice machine reading comprehension

This task aims to evaluate the capability of multiple-choice question
answering given a context passage. The question could be answered by
understanding the context passage while the right answer is given at one of
the multiple choices. We use the Chinese, Sotho and Arabic clusters data from
the recently released Belebele MRC dataset Bandarkar et al. (2023). This is an
evaluation-only dataset.

#### Extractive question answering

This task predicts the answer span given a question and context passage. We
use the SD-QA dialectal question-answering dataset Faisal et al. (2021). SD-QA
is an evaluation dataset built on top of TyDiQA Clark et al. (2020), another
well-known typologically diverse question-answering dataset. SD-QA contains
the spoken utterances and transcription of the original TyDiQA question from
speakers of English, Bengali, Arabic, Korean, and Swahili. We only use the
textual part that contains the transcription of the dialectal spoken question
matching the original TyDiQA question text. Note that since the transcriptions
of the questions are obtained through automatic speech recognition, they may
include both dialectal variations and noise due to ASR transcription errors.

#### Machine translation

We evaluate variety translation using the CODET benchmark Alam et al. (2023)
and the TIL MT corpus Mirzakhalov et al. (2021b). CODET contains a contrastive
dataset of 882 different varieties from nine different languages. We evaluate
dialects here at city level for all languages except Italian and Swiss-German
and aggregate dialects at the region level for them. The TIL corpus contains
parallel translations across 22 Turkic languages, but in our evaluations we
only include 8 turkic languages (Turkic, Sakha, Kazakh, Karakalpak, Bashkir,
Azerbaijani, Kyrgyz) that have parallel English translations.

##  Appendix C Varieties and Clusters of DialectBench

###  C.1 DialectBench Variety list

Table 8: Varieties represented in DialectBench. Language cluster |  Variety name | Glottocode | mBERT seen | UDP | POS | NER | SDQA | RCMC | NLI | TC | SC | DI | MT
---|---|---|---|---|---|---|---|---|---|---|---|---|---
albanian |  albanian | alba1267 | ✓ | ✓ | ✓ | - | - | - | - | - | - | - | -
gheg albanian | gheg1238 | - | ✓ | ✓ | - | - | - | - | - | - | - | -
anglic |  african american vernacular english | afri1276 | - | ✓ | - | - | - | - | - | - | - | - | -
australian english | aust1314 | - | - | - | - | ✓ | - | - | - | - | - | -
english | stan1293 | ✓ | ✓ | ✓ | ✓ | - | ✓ | ✓ | ✓ | - | ✓ | -
english (a:scotland) | stan1293 | - | - | - | - | ✓ | - | - | - | - | - | -
english (a:uk) | stan1293 | - | - | - | - | - | - | - | - | - | ✓ | -
english (o:controlled) | stan1293 | - | - | - | ✓ | - | - | - | - | - | - | -
indian english (a:north) | indi1255 | - | - | - | - | ✓ | - | - | - | - | - | -
indian english (a:south) | indi1255 | - | - | - | - | ✓ | - | - | - | - | - | -
irish english | iris1255 | - | - | - | - | ✓ | - | - | - | - | - | -
jamaican creole english | jama1262 | - | - | - | ✓ | - | - | - | - | - | - | -
kenyan english | keny1281 | - | - | - | - | ✓ | - | - | - | - | - | -
new zealand english | newz1240 | - | - | - | - | ✓ | - | - | - | - | - | -
nigerian english | nige1260 | - | - | - | - | ✓ | - | - | - | - | - | -
north american english | nort3314 | - | - | - | - | - | - | - | - | - | ✓ | -
old english (ca. 450-1100) | olde1238 | - | - | - | ✓ | - | - | - | - | - | - | -
philippine english | phil1246 | - | - | - | - | ✓ | - | - | - | - | - | -
singlish | sing1272 | - | ✓ | ✓ | - | - | - | - | - | - | - | -
southeast american english | sout3300 | - | - | - | - | ✓ | - | - | - | - | - | -
southern african english | sout3331 | - | - | - | - | ✓ | - | - | - | - | - | -
arabic |  aleppo | alep1241 | - | - | - | - | - | - | - | - | - | ✓ | ✓
algerian arabic | alge1239 | - | - | - | - | ✓ | - | - | - | ✓ | ✓ | ✓
arabian peninsula arabic (a:yemen) | arab1393 | - | - | - | - | - | - | - | - | - | ✓ | ✓
arabic (a:bahrain) | - | - | - | - | - | ✓ | - | - | - | - | - | -
|  arabic (a:jordan) | - | - | - | - | - | ✓ | - | - | - | ✓ | - | -
|  arabic (a:saudi-arabia) | - | - | - | - | - | ✓ | - | - | - | ✓ | - | -
|  egyptian arabic | egyp1253 | - | - | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | - | -
|  egyptian arabic (a:alx) | egyp1253 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  egyptian arabic (a:asw) | egyp1253 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  egyptian arabic (a:cai) | egyp1253 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  egyptian arabic (a:kha) | egyp1253 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  fez. meknes | fezm1238 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  gilit mesopotamian arabic | meso1252 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  gulf arabic | gulf1241 | - | - | ✓ | - | - | - | - | - | - | - | -
|  gulf arabic (a:doh) | gulf1241 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  gulf arabic (a:jed) | gulf1241 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  gulf arabic (a:mus) | gulf1241 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  gulf arabic (a:riy) | gulf1241 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  levantine arabic | nort3139 | - | - | ✓ | - | - | - | - | - | - | - | -
|  levantine arabic (a:north) | nort3139 | - | - | - | - | - | ✓ | ✓ | ✓ | - | - | -
|  levantine arabic (a:north-dam) | nort3139 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  libyan arabic | liby1240 | - | - | - | - | - | - | - | - | - | - | -
|  libyan arabic (a:ben) | liby1240 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  moroccan arabic | moro1292 | - | - | - | - | ✓ | ✓ | ✓ | ✓ | ✓ | - | -
|  najdi arabic | najd1235 | - | - | - | - | - | ✓ | ✓ | ✓ | - | - | -
|  north african arabic | nort3191 | - | ✓ | ✓ | - | - | - | - | - | - | - | -
|  north mesopotamian arabic | nort3142 | - | - | - | - | - | ✓ | ✓ | ✓ | - | - | -
|  north mesopotamian arabic (a:bas) | nort3142 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  north mesopotamian arabic (a:mos) | nort3142 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  rabat-casablanca arabic | raba1252 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  sfax | sfax1238 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  south levantine arabic | sout3123 | - | ✓ | ✓ | - | - | - | ✓ | ✓ | ✓ | - | -
|  south levantine arabic (a:south-amm) | sout3123 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  south levantine arabic (a:south-jer) | sout3123 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  south levantine arabic (a:south-sal) | sout3123 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  standard arabic | stan1318 | ✓ | ✓ | ✓ | ✓ | - | ✓ | ✓ | ✓ | ✓ | ✓ | -
|  sunni beiruti arabic | sunn1238 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  ta’izzi-adeni arabic | taiz1242 | - | - | - | - | - | - | ✓ | ✓ | - | - | -
|  tripolitanian arabic | trip1239 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  tunisian arabic | tuni1259 | - | - | - | - | ✓ | - | ✓ | ✓ | ✓ | - | -
|  tunisian arabic (a:tun) | tuni1259 | - | - | - | - | - | - | - | - | - | ✓ | ✓
|  tunisian arabic (r:casual) | tuni1259 | - | - | - | - | - | - | - | - | ✓ | - | -
basque |  basque (a:amenduze) | basq1248 | ✓ | - | - | - | - | - | - | - | - | - | ✓
basque (a:azkaine) | basq1248 | - | - | - | - | - | - | - | - | - | - | ✓
basque (a:baigorri) | basq1248 | - | - | - | - | - | - | - | - | - | - | ✓
basque (a:barkoxe) | basq1248 | - | - | - | - | - | - | - | - | - | - | ✓
basque (a:donibane) | basq1248 | - | - | - | - | - | - | - | - | - | - | ✓
basque (a:garruze) | basq1248 | - | - | - | - | - | - | - | - | - | - | ✓
basque (a:iholdi) | basq1248 | - | - | - | - | - | - | - | - | - | - | ✓
basque (a:jatsu) | basq1248 | - | - | - | - | - | - | - | - | - | - | ✓
basque (a:jutsi) | basq1248 | - | - | - | - | - | - | - | - | - | - | ✓
basque (a:larzabale) | basq1248 | - | - | - | - | - | - | - | - | - | - | ✓
basque (a:luhuso) | basq1248 | - | - | - | - | - | - | - | - | - | - | ✓
basque (a:sara) | basq1248 | - | - | - | - | - | - | - | - | - | - | ✓
basque (a:senpere) | basq1248 | - | - | - | - | - | - | - | - | - | - | ✓
basque (a:suhuskune) | basq1248 | - | - | - | - | - | - | - | - | - | - | ✓
basque (a:uharte) | basq1248 | - | - | - | - | - | - | - | - | - | - | ✓
navarro-labourdin basque (a:behorlegi) | basq1249 | - | - | - | - | - | - | - | - | - | - | ✓
navarro-labourdin basque (a:bidarrai) | basq1249 | - | - | - | - | - | - | - | - | - | - | ✓
navarro-labourdin basque (a:helete) | basq1249 | - | - | - | - | - | - | - | - | - | - | ✓
navarro-labourdin basque (a:mugerre) | basq1249 | - | - | - | - | - | - | - | - | - | - | ✓
navarro-labourdin basque (a:urruna) | basq1249 | - | - | - | - | - | - | - | - | - | - | ✓
souletin (a:maule) | basq1250 | - | - | - | - | - | - | - | - | - | - | ✓
bengali |  vanga (a:barisal) | vang1242 | - | - | - | - | - | - | - | - | - | - | ✓
vanga (a:dhaka) | vang1242 | ✓ | - | - | - | ✓ | - | - | - | - | - | ✓
vanga (a:jessore) | vang1242 | - | - | - | - | - | - | - | - | - | - | ✓
vanga (a:khulna) | vang1242 | - | - | - | - | - | - | - | - | - | - | ✓
vanga (a:kushtia) | vang1242 | - | - | - | - | - | - | - | - | - | - | ✓
vanga (a:west bengal) | vang1242 | ✓ | - | - | - | ✓ | - | - | - | - | - | -
circassian |  adyghe | adyg1241 | - | - | - | ✓ | - | - | - | - | - | - | -
kabardian | kaba1278 | - | - | - | ✓ | - | - | - | - | - | - | -
common turkic |  bashkir | bash1264 | ✓ | - | - | - | - | - | - | - | - | - | ✓
central oghuz | azer1255 | - | - | - | - | - | - | - | - | - | - | ✓
central oghuz (m:spoken) | azer1255 | - | - | - | - | - | - | ✓ | ✓ | - | - | -
crimean tatar | crim1257 | - | - | - | ✓ | - | - | - | - | - | - | -
kara-kalpak | kara1467 | - | - | - | - | - | - | - | - | - | - | ✓
kazakh | kaza1248 | ✓ | - | - | - | - | - | - | - | - | - | ✓
kirghiz | kirg1245 | ✓ | - | - | - | - | - | - | - | - | - | ✓
north azerbaijani | nort2697 | ✓ | - | - | ✓ | - | - | ✓ | ✓ | - | - | -
sakha | yaku1245 | - | - | - | - | - | - | - | - | - | - | ✓
south azerbaijani | sout2697 | ✓ | - | - | ✓ | - | - | ✓ | ✓ | - | - | -
turkish | nucl1301 | ✓ | - | - | ✓ | - | - | - | - | - | - | ✓
uzbek | uzbe1247 | ✓ | - | - | - | - | - | - | - | - | - | ✓
eastern romance |  aromanian | arom1237 | - | - | - | ✓ | - | - | - | - | - | - | -
moldavian | mold1248 | - | - | - | ✓ | - | - | - | - | - | - | -
romanian | roma1327 | ✓ | - | - | ✓ | - | - | - | - | - | - | -
eastern-western armenian |  eastern armenian | nucl1235 | ✓ | ✓ | ✓ | - | - | - | - | - | - | - | -
western armenian | homs1234 | - | ✓ | ✓ | - | - | - | - | - | - | - | -
farsic |  dari | dari1249 | - | - | - | - | - | - | - | - | - | - | ✓
frisian |  ems-weser frisian | sate1242 | - | - | - | ✓ | - | - | - | - | - | - | -
northern frisian | nort2626 | - | - | - | ✓ | - | - | - | - | - | - | -
western frisian | west2354 | ✓ | - | - | ✓ | - | - | - | - | - | - | -
gallo-italian |  emiliano-romagnolo | emil1243 | - | - | - | ✓ | - | - | - | - | - | - | -
ligurian | ligu1248 | - | ✓ | ✓ | ✓ | - | - | ✓ | ✓ | - | - | -
lombard | lomb1257 | ✓ | - | - | ✓ | - | - | ✓ | ✓ | - | - | -
piemontese | piem1238 | ✓ | - | - | ✓ | - | - | - | - | - | - | -
venetian | vene1258 | - | - | - | ✓ | - | - | ✓ | ✓ | - | - | -
gallo-rhaetian |  anglo-norman | angl1258 | - | - | - | ✓ | - | - | - | - | - | - | -
arpitan | fran1260 | - | - | - | ✓ | - | - | - | - | - | - | -
french | stan1290 | ✓ | ✓ | ✓ | ✓ | - | - | - | - | - | - | -
french (a:paris) | stan1290 | - | ✓ | ✓ | - | - | - | - | - | - | - | -
friulian | friu1240 | - | - | - | ✓ | - | - | ✓ | ✓ | - | - | -
old french (842-ca. 1400) | oldf1239 | - | ✓ | ✓ | - | - | - | - | - | - | - | -
romansh | roma1326 | - | - | - | ✓ | - | - | - | - | - | - | -
walloon | wall1255 | - | - | - | ✓ | - | - | - | - | - | - | -
greater panjabic |  eastern panjabi | panj1256 | ✓ | - | - | ✓ | - | - | - | - | - | - | -
western panjabi | west2386 | ✓ | - | - | ✓ | - | - | - | - | - | - | -
greek |  apulian greek | apul1237 | - | - | - | - | - | - | - | - | - | - | ✓
cretan | cret1244 | - | - | - | - | - | - | - | - | - | - | ✓
cypriot greek | cypr1249 | - | - | - | - | - | - | - | - | - | - | -
cypriot greek (r:casual, m:written, i:fb) | cypr1249 | - | - | - | - | - | - | - | - | - | ✓ | -
cypriot greek (r:casual, m:written, i:other) | cypr1249 | - | - | - | - | - | - | - | - | - | ✓ | -
cypriot greek (r:casual, m:written, i:twitter) | cypr1249 | - | - | - | - | - | - | - | - | - | ✓ | -
modern greek | mode1248 | ✓ | - | - | ✓ | - | - | - | - | - | - | -
modern greek (r:casual, m:written, i:fb) | mode1248 | - | - | - | - | - | - | - | - | - | ✓ | -
modern greek (r:casual, m:written, i:other) | mode1248 | - | - | - | - | - | - | - | - | - | ✓ | -
modern greek (r:casual, m:written, i:twitter) | mode1248 | - | - | - | - | - | - | - | - | - | ✓ | -
pontic | pont1253 | - | - | - | ✓ | - | - | - | - | - | - | -
high german |  bavarian | bava1246 | ✓ | - | - | ✓ | - | - | - | - | - | - | -
central alemannic | swis1247 | - | - | - | ✓ | - | - | - | - | - | - | -
central alemannic (a:ag) | swis1247 | - | - | - | - | - | - | - | - | - | - | ✓
central alemannic (a:ai) | swis1247 | - | - | - | - | - | - | - | - | - | - | ✓
central alemannic (a:ar) | swis1247 | - | - | - | - | - | - | - | - | - | - | ✓
central alemannic (a:be) | swis1247 | - | - | - | - | - | - | - | - | - | ✓ | ✓
central alemannic (a:bl) | swis1247 | - | - | - | - | - | - | - | - | - | - | ✓
central alemannic (a:bs) | swis1247 | - | - | - | - | - | - | - | - | - | ✓ | ✓
central alemannic (a:fr) | swis1247 | - | - | - | - | - | - | - | - | - | - | ✓
central alemannic (a:gl) | swis1247 | - | - | - | - | - | - | - | - | - | - | ✓
central alemannic (a:gr) | swis1247 | - | - | - | - | - | - | - | - | - | - | ✓
central alemannic (a:lu) | swis1247 | - | - | - | - | - | - | - | - | - | ✓ | ✓
central alemannic (a:nw) | swis1247 | - | - | - | - | - | - | - | - | - | - | ✓
central alemannic (a:ow) | swis1247 | - | - | - | - | - | - | - | - | - | - | ✓
central alemannic (a:sg) | swis1247 | - | - | - | - | - | - | - | - | - | - | ✓
central alemannic (a:sh) | swis1247 | - | - | - | - | - | - | - | - | - | - | ✓
central alemannic (a:so) | swis1247 | - | - | - | - | - | - | - | - | - | - | ✓
central alemannic (a:sz) | swis1247 | - | - | - | - | - | - | - | - | - | - | ✓
central alemannic (a:tg) | swis1247 | - | - | - | - | - | - | - | - | - | - | ✓
central alemannic (a:ur) | swis1247 | - | - | - | - | - | - | - | - | - | - | ✓
central alemannic (a:vs) | swis1247 | - | - | - | - | - | - | - | - | - | - | ✓
central alemannic (a:zg) | swis1247 | - | - | - | - | - | - | - | - | - | - | ✓
central alemannic (a:zh) | swis1247 | - | ✓ | ✓ | - | - | - | - | - | - | ✓ | ✓
central bavarian | cent1967 | - | - | - | - | - | - | - | - | - | - | ✓
german | stan1295 | ✓ | ✓ | ✓ | ✓ | - | - | - | - | - | - | -
kölsch | kols1241 | - | - | - | ✓ | - | - | - | - | - | - | -
limburgan | limb1263 | - | - | - | ✓ | - | - | ✓ | ✓ | - | - | -
luxemburgish | luxe1243 | ✓ | - | - | ✓ | - | - | ✓ | ✓ | - | - | -
pennsylvania german | penn1240 | - | - | - | ✓ | - | - | - | - | - | - | -
pfaelzisch-lothringisch | pala1330 | - | - | - | ✓ | - | - | - | - | - | - | -
upper saxon | uppe1465 | - | - | - | - | - | - | - | - | - | - | ✓
hindustani |  fiji hindi | fiji1242 | - | - | - | ✓ | - | - | - | - | - | - | -
hindi | hind1269 | ✓ | - | - | ✓ | - | - | - | - | - | - | -
inuit |  alaskan inupiaq | inup1234 | - | - | - | ✓ | - | - | - | - | - | - | -
kalaallisut | kala1399 | - | - | - | ✓ | - | - | - | - | - | - | -
italian romance |  continental southern italian | neap1235 | - | ✓ | ✓ | ✓ | - | - | - | - | - | - | -
italian | ital1282 | ✓ | ✓ | ✓ | ✓ | - | - | ✓ | ✓ | - | - | -
italian (a:abruzzo) | ital1282 | - | - | - | - | - | - | - | - | - | - | ✓
italian (a:basilicata) | ital1282 | - | - | - | - | - | - | - | - | - | - | ✓
italian (a:calabria) | ital1282 | - | - | - | - | - | - | - | - | - | - | ✓
italian (a:campania) | ital1282 | - | - | - | - | - | - | - | - | - | - | ✓
italian (a:emilia-romagna) | ital1282 | - | - | - | - | - | - | - | - | - | - | ✓
italian (a:friuli-venezia giulia) | ital1282 | - | - | - | - | - | - | - | - | - | - | ✓
italian (a:lazio) | ital1282 | - | - | - | - | - | - | - | - | - | - | ✓
italian (a:liguria) | ital1282 | - | - | - | - | - | - | - | - | - | - | ✓
italian (a:lombardia) | ital1282 | - | - | - | - | - | - | - | - | - | - | ✓
italian (a:marche) | ital1282 | - | - | - | - | - | - | - | - | - | - | ✓
italian (a:molise) | ital1282 | - | - | - | - | - | - | - | - | - | - | ✓
italian (a:piemonte) | ital1282 | - | - | - | - | - | - | - | - | - | - | ✓
italian (a:puglia) | ital1282 | - | - | - | - | - | - | - | - | - | - | ✓
italian (a:sardegna) | ital1282 | - | - | - | - | - | - | - | - | - | - | ✓
italian (a:sicilia) | ital1282 | - | - | - | - | - | - | - | - | - | - | ✓
italian (a:toscana) | ital1282 | - | - | - | - | - | - | - | - | - | - | ✓
italian (a:trentino-alto adige/südtirol) | ital1282 | - | - | - | - | - | - | - | - | - | - | ✓
italian (a:umbria) | ital1282 | - | - | - | - | - | - | - | - | - | - | ✓
italian (a:unknown) | ital1282 | - | - | - | - | - | - | - | - | - | - | ✓
italian (a:veneto) | ital1282 | - | - | - | - | - | - | - | - | - | - | ✓
italian (r:casual, m:written, i:tweet) | ital1282 | ✓ | ✓ | ✓ | - | - | - | - | - | - | - | -
italian (r:formal, m:written, i:essay) | ital1282 | ✓ | ✓ | ✓ | - | - | - | - | - | - | - | -
italian romance (a:apulia, m:spoken, i:tarantino) | - | - | - | - | ✓ | - | - | - | - | - | - | -
sicilian | sici1248 | ✓ | - | - | ✓ | - | - | ✓ | ✓ | - | - | -
komi |  komi | komi1267 | - | - | - | ✓ | - | - | - | - | - | - | -
komi-permyak | komi1269 | - | ✓ | ✓ | ✓ | - | - | - | - | - | - | -
komi-zyrian (m:spoken) | komi1268 | - | ✓ | ✓ | - | - | - | - | - | - | - | -
komi-zyrian (m:written) | komi1268 | - | ✓ | ✓ | - | - | - | - | - | - | - | -
korean |  korean (a:south-eastern, m:spoken) | kore1280 | - | - | - | - | ✓ | - | - | - | - | - | -
seoul (m:spoken) | seou1239 | ✓ | - | - | - | ✓ | - | - | - | - | - | -
kurdish |  central kurdish | cent1972 | - | - | - | ✓ | - | - | ✓ | ✓ | - | - | -
kurdish | kurd1259 | - | - | - | ✓ | - | - | - | - | - | - | -
northern kurdish | nort2641 | - | - | - | - | - | - | ✓ | ✓ | - | - | -
sine’i | sine1239 | - | - | - | - | - | - | - | - | - | - | ✓
sorani | sora1257 | - | - | - | - | - | - | - | - | - | - | ✓
latvian |  east latvian | east2282 | - | - | - | ✓ | - | - | ✓ | ✓ | - | - | -
latvian | latv1249 | ✓ | - | - | ✓ | - | - | ✓ | ✓ | - | - | -
mari |  eastern mari | east2328 | - | - | - | ✓ | - | - | - | - | - | - | -
western mari | west2392 | - | - | - | ✓ | - | - | - | - | - | - | -
modern dutch |  dutch | dutc1256 | ✓ | - | - | ✓ | - | - | ✓ | ✓ | - | - | -
western flemish | vlaa1240 | - | - | - | ✓ | - | - | - | - | - | - | -
zeeuws | zeeu1238 | - | - | - | ✓ | - | - | - | - | - | - | -
neva |  central and north pohjanmaa (a:ostrobothnian) | cent1985 | - | - | ✓ | - | - | - | - | - | - | - | -
estonian | esto1258 | ✓ | - | ✓ | - | - | - | - | - | - | - | -
finnish | finn1318 | ✓ | - | ✓ | - | - | - | - | - | - | - | -
häme (a:tavastian) | hame1240 | - | - | ✓ | - | - | - | - | - | - | - | -
neva (a:south-west trans) | - | - | - | ✓ | - | - | - | - | - | - | - | -
savo (a:savonian) | savo1254 | - | - | ✓ | - | - | - | - | - | - | - | -
southeastern finnish (a:south-east) | sout1743 | - | - | ✓ | - | - | - | - | - | - | - | -
southwestern finnish (a:south-west) | sout2677 | - | - | ✓ | - | - | - | - | - | - | - | -
norwegian |  norwegian | norw1258 | ✓ | - | - | - | - | - | - | - | - | - | -
norwegian (a:eastern) | norw1258 | - | - | - | - | - | - | - | - | - | - | ✓
norwegian (a:setesdal) | norw1258 | - | - | - | - | - | - | - | - | - | - | ✓
norwegian (a:southwestern) | norw1258 | - | - | - | - | - | - | - | - | - | - | ✓
norwegian (m:written, i:samnorsk) | - | - | - | - | ✓ | - | - | - | - | - | - | -
norwegian bokmål (m:written) | norw1259 | ✓ | ✓ | ✓ | ✓ | - | - | ✓ | ✓ | - | - | -
norwegian nynorsk (m:written) | norw1262 | ✓ | ✓ | ✓ | ✓ | - | - | ✓ | ✓ | - | - | -
norwegian nynorsk (m:written, i:old) | norw1262 | - | ✓ | ✓ | - | - | - | - | - | - | - | -
saami |  north saami | nort2671 | - | ✓ | ✓ | - | - | - | - | - | - | - | -
skolt saami | skol1241 | - | ✓ | ✓ | - | - | - | - | - | - | - | -
sabellic |  umbrian | umbr1253 | - | ✓ | ✓ | - | - | - | - | - | - | - | -
sardo-corsican |  corsican | cors1241 | - | - | - | ✓ | - | - | - | - | - | - | -
sardinian | sard1257 | - | - | - | ✓ | - | - | ✓ | ✓ | - | - | -
serbian-croatian-bosnian |  bosnian standard | bosn1245 | ✓ | - | - | ✓ | - | - | - | - | - | - | -
croatian standard | croa1245 | ✓ | - | - | ✓ | - | - | - | - | - | - | -
serbian standard | serb1264 | ✓ | - | - | ✓ | - | - | - | - | - | - | -
serbian-croatian-bosnian | sout1528 | ✓ | - | - | ✓ | - | - | - | - | - | - | -
sinitic |  cantonese | cant1236 | - | - | - | ✓ | - | - | ✓ | ✓ | - | - | -
classical chinese | lite1248 | - | ✓ | ✓ | ✓ | - | - | - | - | - | - | -
classical-middle-modern sinitic (a:hongkong, o:traditional) | clas1255 | - | ✓ | ✓ | - | - | - | - | - | - | - | -
classical-middle-modern sinitic (o:simplified) | clas1255 | ✓ | ✓ | ✓ | - | - | ✓ | ✓ | ✓ | - | - | -
classical-middle-modern sinitic (o:traditional) | clas1255 | ✓ | - | - | - | - | ✓ | ✓ | ✓ | - | - | -
hakka chinese | hakk1236 | - | - | - | ✓ | - | - | - | - | - | - | -
mandarin chinese | mand1415 | - | - | - | ✓ | - | - | - | - | - | - | -
mandarin chinese (a:mainland, o:simplified) | mand1415 | - | - | - | - | - | - | - | - | - | ✓ | -
mandarin chinese (a:mainland, o:traditional, i:synthetic) | mand1415 | - | - | - | - | - | - | - | - | - | ✓ | -
mandarin chinese (a:taiwan, o:simplified) | mand1415 | - | - | - | - | - | - | - | - | - | ✓ | -
mandarin chinese (a:taiwan, o:traditional, i:synthetic) | mand1415 | - | - | - | - | - | - | - | - | - | ✓ | -
min nan chinese | minn1241 | - | - | - | ✓ | - | - | - | - | - | - | -
wu chinese | wuch1236 | - | - | - | ✓ | - | - | - | - | - | - | -
sorbian |  lower sorbian | lowe1385 | - | - | - | ✓ | - | - | - | - | - | - | -
upper sorbian | uppe1395 | - | - | - | ✓ | - | - | - | - | - | - | -
sotho-tswana (s.30) |  northern sotho | nort3233 | - | - | - | ✓ | - | ✓ | ✓ | ✓ | - | - | -
southern sotho | sout2807 | - | - | - | ✓ | - | ✓ | ✓ | ✓ | - | - | -
southwestern shifted romance |  brazilian portuguese | braz1246 | - | ✓ | ✓ | - | - | - | - | - | - | ✓ | -
extremaduran | extr1243 | - | - | - | ✓ | - | - | - | - | - | - | -
|  galician | gali1258 | ✓ | - | - | ✓ | - | - | ✓ | ✓ | - | - | -
|  latin american spanish | amer1254 | - | - | - | - | - | - | - | - | - | ✓ | -
|  mirandese | mira1251 | - | - | - | ✓ | - | - | - | - | - | - | -
|  occitan | occi1239 | ✓ | - | ✓ | ✓ | - | - | ✓ | ✓ | - | - | ✓
|  portuguese (a:european) | port1283 | - | ✓ | ✓ | - | - | - | ✓ | ✓ | - | ✓ | -
|  portuguese (i:mix) | port1283 | - | ✓ | ✓ | - | - | - | - | - | - | - | -
|  portuguese (m:written) | port1283 | ✓ | ✓ | ✓ | - | - | - | - | - | - | ✓ | -
|  spanish | stan1288 | ✓ | - | - | ✓ | - | - | ✓ | ✓ | - | ✓ | -
|  spanish (a:europe) | stan1288 | - | - | - | - | - | - | - | - | - | ✓ | -
swahili |  swahili | swah1253 | ✓ | - | - | - | - | - | - | - | - | - | -
swahili (a:kenya) | swah1253 | - | - | - | - | ✓ | - | - | - | - | - | -
swahili (a:tanzania) | swah1253 | - | - | - | - | ✓ | - | - | - | - | - | -
tigrinya |  tigrinya (a:eritrea) | tigr1271 | - | - | - | - | - | - | - | - | - | - | ✓
tigrinya (a:ethiopia) | tigr1271 | - | - | - | - | - | - | - | - | - | - | ✓
tupi-guarani subgroup i.a |  mbyá guaraní (a:brazil) | mbya1239 | - | ✓ | ✓ | - | - | - | - | - | - | - | -
mbyá guaraní (a:paraguay) | mbya1239 | - | ✓ | ✓ | - | - | - | - | - | - | - | -
old guarani | oldp1258 | - | ✓ | ✓ | - | - | - | - | - | - | - | -
west low german |  west low german | west2357 | ✓ | ✓ | ✓ | ✓ | - | - | - | - | - | - | -
yoruba |  yoruba (a:central nigeria) | yoru1245 | ✓ | - | - | - | - | - | - | - | - | - | ✓
Table 8: Varieties represented in DialectBench.

###  C.2 Language Clusters and Representative Varieties

Table 9: Language clusters and their standard varieties. Task |  Cluster Name |  Cluster Representative
---|---|---
DEP. Parsing | albanian | albanian
arabic | standard arabic
eastern-western armenian | western armenian
sinitic | classical-middle-modern sinitic (o:simplified)
anglic | english
gallo-rhaetian | french
high german | german
tupi-guarani subgroup i.a | old guarani
italian romance | italian (r:formal, m:written, i:essay)
komi | komi-zyrian (m:spoken)
norwegian | norwegian bokmål (m:written)
southwestern shifted romance | brazilian portuguese
saami | north saami
POS Tagging | albanian | albanian
anglic | english
arabic | standard arabic
eastern-western armenian | western armenian
gallo-rhaetian | french
high german | german
italian romance | italian (r:formal, m:written, i:essay)
komi | komi-zyrian (m:spoken)
neva | finnish
norwegian | norwegian bokmål (m:written)
saami | north saami
sinitic | classical-middle-modern sinitic (o:simplified)
southwestern shifted romance | brazilian portuguese
tupi-guarani subgroup i.a | mbyá guaraní (a:paraguay)
NER | anglic | english
arabic | standard arabic
circassian | adyghe
common turkic | turkish
eastern romance | romanian
frisian | western frisian
gallo-italian | piemontese
gallo-rhaetian | french
greater panjabic | western panjabi
greek | modern greek
high german | german
hindustani | hindi
inuit | alaskan inupiaq
italian romance | italian
komi | komi
kurdish | kurdish
latvian | latvian
mari | eastern mari
modern dutch | dutch
norwegian | norwegian bokmål (m:written)
sardo-corsican | sardinian
serbian-croatian-bosnian | croatian standard
sinitic | mandarin chinese
sorbian | lower sorbian
sotho-tswana (s.30) | southern sotho
southwestern shifted romance | spanish
EQA | anglic | southeast american english
arabic | arabic (a:saudi-arabia)
bengali | vanga (a:dhaka)
korean | seoul (m:spoken)
swahili | swahili (a:kenya)
MRC | anglic | english
arabic | standard arabic
sinitic | classical-middle-modern sinitic (o:simplified)
sotho-tswana (s.30) | northern sotho
TC | anglic | english
arabic | standard arabic
common turkic | north azerbaijani
gallo-italian | venetian
high german | luxemburgish
italian romance | italian
kurdish | northern kurdish
latvian | latvian
norwegian | norwegian bokmål (m:written)
sinitic | classical-middle-modern sinitic (o:simplified)
sotho-tswana (s.30) | northern sotho
southwestern shifted romance | spanish
SC | arabic | standard arabic
MT-Dialect | arabic | gulf arabic (a:riy)
basque | basque (a:azkaine)
bengali | vanga (a:dhaka)
high german | central bavarian
kurdish | sorani
norwegian | norwegian (a:eastern)
tigrinya | tigrinya (a:ethiopia)
common turkic | turkish
greek | cretan
MT-Region | italian romance | italian (a:umbria)
high german | central alemannic (a:zh)
DId | anglic | english (a:uk)
arabic | standard arabic
greek | modern greek (r:casual, m:written, i:twitter)
high german | central alemannic (a:zh)
sinitic | mandarin chinese (a:mainland, o:simplified)
southwestern shifted romance | brazilian portuguese
NLI | anglic | english
arabic | standard arabic
common turkic | north azerbaijani
gallo-italian | venetian
high german | luxemburgish
italian romance | italian
kurdish | northern kurdish
latvian | latvian
norwegian | norwegian bokmål (m:written)
sinitic | classical-middle-modern sinitic (o:simplified)
sotho-tswana (s.30) | northern sotho
southwestern shifted romance | spanish
Table 9: Language clusters and their clusters representatives.

##  Appendix D Result Visualizations

###  D.1 Regional maps with aggregated Machine Translation scores

Figure 5: Map of Switzerland with aggregated BLEU scores of Swiss-German
variety per region Figure 6: Map of Italy with aggregated BLEU scores of
Italian variety per region

###  D.2 Task Specific Plot for Maximum Scores

(a) Parts of Speech Tagging

(b) Named Entity Recognition

(c) Topic Classification

Figure 7: Task specific plot of maximum obtainable score for all varieties.
The yellow-shaded region represents language clusters having no varieties seen
during mBERT pertaining. The bars with colored stripes represent the standard
variety of a cluster

(a) Natural Language Inference

(b) Extractive Question Answering

(c) Multiple Choice Machine Reading Comprehension

Figure 8: Task specific plot of maximum obtainable Linguistic Utility for all
varieties. The yellow-shaded region represents language clusters having no
varieties seen during mBERT pertaining. The bars with colored stripes
represent the standard variety of a cluster. The dialect with the Rawlsian
score in each cluster is that with the leftmost bar.

(a) Sentiment Analysis

(b) Dialect Identification

Figure 9: Task specific plot of maximum obtainable Linguistic Utility for all
varieties. The yellow-shaded region represents language clusters having no
varieties seen during mBERT pertaining. The bars with colored stripes
represent the standard variety of a cluster. The dialect with the Rawlsian
score in each cluster is that with the leftmost bar.

(a) Machine Translation (Region level aggregation)

(b) Machine Translation (Variety level)

Figure 10: Task specific plot of maximum obtainable Linguistic Utility for all
varieties. The yellow-shaded region represents language clusters having no
varieties seen during mBERT pertaining. The bars with colored stripes
represent the standard variety of a cluster. The dialect with the Rawlsian
score in each cluster is that with the leftmost bar.

###  D.3 Dialectal Gap visualizations (zeroshot)

(a) Parts-of-speech (POS) tagging

(b) Named entity recognition (NER)

(c) Natural language inference (NLI)

Figure 11: Dialectal Gap visualization for language clusters utilizing zero-
shot cross-lingual transfer from Standard English. The x-axis is for cluster
gap while comparing against Standard English variety. Values far from zero
have a larger performance gap from English. The y-axis is for aggregated
cluster gap while comparing against standard cluster variety and values far
from zero have larger within cluster gap. Ideally, we want both of them to be
close to zero.

##  Appendix E Evaluation results

cluster |  variety |  UD-code |  Zeroshot (mBERT) |  Zeroshot (XLM-R) |  Finetune (mBERT) |  Finetune (XLM-R)
---|---|---|---|---|---|---
albanian |  albanian |  UD_Albanian-TSA |  81.78 |  83.08 |  - |  -
gheg albanian |  UD_Gheg-GPS |  38.14 |  43.50 |  - |  -
anglic |  english |  UD_English-EWT |  91.55 |  91.10 |  91.55 |  91.1
singlish |  singlish |  69.88 |  68.55 |  82.36 |  11.0
african american vernacular english |  TwitterAAE |  50.53 |  52.53 |  - |  -
arabic |  standard arabic |  UD_Arabic-PADT |  50.87 |  55.91 |  88.47 |  2.48
south levantine arabic |  UD_South_Levantine_Arabic-MADAR |  49.94 |  45.75 |  - |  -
north african arabic |  UD_Maghrebi_Arabic_French-Arabizi |  34.33 |  28.04 |  5.46 |  54.7
eastern-western armenian |  western armenian |  UD_Western_Armenian-ArmTDP |  54.63 |  59.71 |  88.84 |  89.29
eastern armenian |  UD_Armenian-ArmTDP |  53.27 |  62.63 |  85.55 |  87.01
gallo-italian |  ligurian |  UD_Ligurian-GLT |  50.22 |  43.78 |  13.06 |  9.1
gallo-rhaetian |  french |  UD_French-ParTUT |  80.87 |  82.17 |  93.93 |  92.09
french (a:paris) |  UD_French-ParisStories |  57.64 |  61.36 |  77.54 |  77.54
old french (842-ca. 1400) |  UD_Old_French-SRCMF |  56.12 |  46.52 |  91.51 |  89.78
high german |  german |  UD_German-LIT |  72.65 |  76.50 |  - |  -
central alemannic (a:zh) |  UD_Swiss_German-UZH |  36.77 |  34.70 |  - |  -
italian romance |  italian |  UD_Italian-PUD |  78.99 |  78.58 |  - |  -
italian (r:formal, m:written, i:essay) |  UD_Italian-MarkIT |  76.83 |  79.38 |  86.34 |  83.81
italian (r:casual, m:written, i:tweet) |  UD_Italian-PoSTWITA |  61.88 |  62.13 |  85.82 |  86.32
continental southern italian |  UD_Neapolitan-RB |  30.00 |  50.00 |  - |  -
komi |  komi-zyrian (m:spoken) |  UD_Komi_Zyrian-IKDP |  26.89 |  32.14 |  - |  -
komi-permyak |  UD_Komi_Permyak-UH |  26.12 |  30.91 |  - |  -
komi-zyrian (m:written) |  UD_Komi_Zyrian-Lattice |  21.01 |  27.55 |  - |  -
norwegian |  norwegian bokmål (m:written) |  UD_Norwegian-Bokmaal |  79.55 |  82.95 |  93.57 |  93.49
norwegian nynorsk (m:written) |  UD_Norwegian-Nynorsk |  76.45 |  76.76 |  93.15 |  93.14
norwegian nynorsk (m:written, i:old) |  UD_Norwegian-NynorskLIA |  56.58 |  56.08 |  78.39 |  8.25
saami |  north saami |  UD_North_Sami-Giella |  23.58 |  16.87 |  67.56 |  5.96
skolt saami |  UD_Skolt_Sami-Giellagas |  19.41 |  28.28 |  - |  -
sabellic |  umbrian |  UD_Umbrian-IKUVINA |  33.21 |  28.75 |  - |  -
sinitic |  classical-middle-modern sinitic (a:hongkong, o:traditional) |  UD_Chinese-HK |  58.97 |  61.78 |  - |  -
classical-middle-modern sinitic (o:simplified) |  UD_Chinese-GSDSimp |  53.35 |  52.21 |  87.85 |  87.37
classical chinese |  UD_Classical_Chinese-Kyoto |  46.72 |  35.14 |  19.65 |  18.11
southwestern shifted romance |  portuguese (a:european) |  UD_Portuguese-PUD |  76.67 |  77.28 |  - |  -
portuguese (i:mix) |  UD_Portuguese-Bosque |  75.89 |  77.96 |  92.64 |  92.15
brazilian portuguese |  UD_Portuguese-GSD |  73.30 |  74.21 |  94.37 |  93.75
portuguese (m:written) |  UD_Portuguese-CINTIL |  69.34 |  72.94 |  83.35 |  84.31
tupi-guarani subgroup i.a |  old guarani |  UD_Guarani-OldTuDeT |  22.58 |  29.03 |  - |  -
mbyá guaraní (a:paraguay) |  UD_Mbya_Guarani-Thomas |  13.51 |  11.15 |  - |  -
mbyá guaraní (a:brazil) |  UD_Mbya_Guarani-Dooley |  8.95 |  4.23 |  - |  -
west low german |  west low german |  UD_Low_Saxon-LSDC |  40.75 |  37.38 |  - |  -
Table 10: Dependency parsing evaluation report comprising zeroshot score and in-language finetuning. We report UAS as evaluation score. Zeroshot scores are evaluated using model finetuned on Standerd English. If training data is not available, we skip those languages (mentioned as ’-’) for in-language finetuning. cluster |  variety | Dataset-code | dataset |  Zeroshot (mBERT) |  Zeroshot (XLM-R) |  Finetune (mBERT) |  Finetune (XLM-R)
---|---|---|---|---|---|---|---
albanian |  albanian | UD_Albanian-TSA | ud |  75.80 |  84.41 |  - |  -
gheg albanian | UD_Gheg-GPS | ud |  48.96 |  55.84 |  - |  -
anglic |  english | UD_English-EWT | ud |  96.41 |  97.16 |  96.41 |  97.16
singlish | singlish | ud |  76.27 |  77.55 |  87.38 |  87.96
arabic |  south levantine arabic | UD_South_Levantine_Arabic-MADAR | ud |  51.99 |  61.84 |  - |  -
standard arabic | UD_Arabic-PADT | ud |  39.74 |  56.67 |  95.72 |  96.11
gulf arabic | dar-glf | noisy |  38.84 |  50.12 |  - |  -
egyptian arabic | dar-egy | noisy |  36.14 |  51.39 |  - |  -
levantine arabic | dar-lev | noisy |  32.66 |  43.76 |  - |  -
north african arabic | UD_Maghrebi_Arabic_French-Arabizi | ud |  28.30 |  26.01 |  67.89 |  59.29
eastern-western armenian |  eastern armenian | UD_Armenian-ArmTDP | ud |  71.78 |  82.63 |  91.31 |  92.36
western armenian | UD_Western_Armenian-ArmTDP | ud |  70.27 |  75.31 |  94.86 |  95.14
gallo-italian |  ligurian | UD_Ligurian-GLT | ud |  58.90 |  52.78 |  13.16 |  5.09
gallo-rhaetian |  french | UD_French-ParTUT | ud |  84.36 |  85.47 |  96.88 |  96.42
french (a:paris) | UD_French-ParisStories | ud |  81.37 |  82.77 |  96.70 |  96.76
old french (842-ca. 1400) | UD_Old_French-SRCMF | ud |  64.70 |  59.41 |  95.64 |  95.64
high german |  german | UD_German-LIT | ud |  87.08 |  88.36 |  - |  -
central alemannic (a:zh) | UD_Swiss_German-UZH | ud |  62.56 |  47.18 |  - |  -
italian romance |  italian | UD_Italian-PUD | ud |  81.09 |  83.12 |  - |  -
italian (r:formal, m:written, i:essay) | UD_Italian-MarkIT | ud |  80.00 |  81.87 |  93.35 |  92.70
italian (r:casual, m:written, i:tweet) | UD_Italian-PoSTWITA | ud |  73.71 |  76.45 |  95.80 |  96.83
continental southern italian | UD_Neapolitan-RB | ud |  30.00 |  57.14 |  - |  -
komi |  komi-zyrian (m:spoken) | UD_Komi_Zyrian-IKDP | ud |  41.25 |  46.66 |  - |  -
komi-permyak | UD_Komi_Permyak-UH | ud |  29.52 |  43.67 |  - |  -
komi-zyrian (m:written) | UD_Komi_Zyrian-Lattice | ud |  20.40 |  35.12 |  - |  -
neva |  finnish | UD_Finnish-TDT | ud |  81.29 |  86.21 |  96.05 |  97.76
estonian | UD_Estonian-EDT | ud |  80.34 |  85.17 |  96.49 |  97.20
hame (a:tavastian) | murre-HAAM | noisy |  55.63 |  70.08 |  - |  -
central and north pohjanmaa (a:ostrobothnian) | murre-POH | noisy |  55.09 |  69.68 |  - |  -
savo (a:savonian) | murre-SAV | noisy |  54.70 |  69.42 |  - |  -
southeastern finnish (a:south-east) | murre-KAA | noisy |  51.68 |  68.71 |  - |  -
southwestern finnish (a:south-west) | murre-LVA | noisy |  49.80 |  68.54 |  - |  -
neva (a:south-west trans) | murre-LOU | noisy |  42.57 |  61.72 |  - |  -
norwegian |  norwegian bokmål (m:written) | UD_Norwegian-Bokmaal | ud |  88.53 |  89.55 |  98.19 |  98.67
norwegian nynorsk (m:written) | UD_Norwegian-Nynorsk | ud |  85.06 |  85.81 |  97.83 |  98.32
norwegian nynorsk (m:written, i:old) | UD_Norwegian-NynorskLIA | ud |  73.25 |  79.29 |  95.47 |  95.72
saami |  north saami | UD_North_Sami-Giella | ud |  35.92 |  32.13 |  78.89 |  71.50
skolt saami | UD_Skolt_Sami-Giellagas | ud |  20.26 |  34.15 |  - |  -
sabellic |  umbrian | UD_Umbrian-IKUVINA | ud |  11.90 |  5.44 |  - |  -
sinitic |  classical-middle-modern sinitic (a:hongkong, o:traditional) | UD_Chinese-HK | ud |  68.99 |  35.49 |  - |  -
classical-middle-modern sinitic (o:simplified) | UD_Chinese-GSDSimp | ud |  58.26 |  30.92 |  94.72 |  95.02
classical chinese | UD_Classical_Chinese-Kyoto | ud |  35.80 |  20.85 |  89.62 |  89.49
southwestern shifted romance |  portuguese (a:european) | UD_Portuguese-PUD | ud |  80.08 |  81.38 |  - |  -
brazilian portuguese | UD_Portuguese-GSD | ud |  78.63 |  80.12 |  98.19 |  98.50
portuguese (i:mix) | UD_Portuguese-Bosque | ud |  78.48 |  79.85 |  97.71 |  97.81
occitan | ROci | noisy |  76.84 |  65.80 |  - |  -
portuguese (m:written) | UD_Portuguese-CINTIL | ud |  76.19 |  78.76 |  97.67 |  97.87
tupi-guarani subgroup i.a |  mbyá guaraní (a:paraguay) | UD_Mbya_Guarani-Thomas | ud |  27.89 |  28.77 |  - |  -
old guarani | UD_Guarani-OldTuDeT | ud |  8.96 |  10.30 |  - |  -
mbyá guaraní (a:brazil) | UD_Mbya_Guarani-Dooley | ud |  1.94 |  0.59 |  - |  -
west low german |  west low german | UD_Low_Saxon-LSDC | ud |  69.65 |  54.93 |  - |  -
Table 11: Parts-of-speech evaluation report comprising zeroshot score and in-language finetuning. We report F1 as evaluation score. Zeroshot scores are evaluated using model finetuned on Standerd English. If training data is not available, we skip those languages (mentioned as ‘-’) for in-language finetuning. Table 12: Named entity recognition (NER) evaluation report comprising zeroshot score and in-group finetuning. We report F1 as evaluation score. Zeroshot scores are evaluated using model finetuned on Standerd English. If training data is not available, we skip those languages (mentioned as ‘-’) for in-language finetuning. cluster |  variety |  target-code |  source | dataset |  support |  Zeroshot (mBERT) |  Zeroshot (XLM-R) |  Finetune (mBERT) |  Finetune (XLM-R)
---|---|---|---|---|---|---|---|---|---
anglic |  english (o:controlled) |  simple |  en | wikiann |  1000 |  89.07 |  86.03 |  89.07 |  86.03
english |  en |  en | wikiann |  10000 |  84.15 |  82.11 |  84.15 |  82.11
old english (ca. 450-1100) |  ang |  en | wikiann |  100 |  54.41 |  55.94 |  54.41 |  55.94
jamaican creole english |  jam |  en | wikiann |  0 |  0.00 |  0.00 |  0.00 |  0.00
arabic |  egyptian arabic |  arz |  ar | wikiann |  100 |  43.82 |  50.58 |  67.77 |  73.33
standard arabic |  ar |  ar | wikiann |  10000 |  41.12 |  41.76 |  89.10 |  87.85
circassian |  adyghe |  ady |  en | wikiann |  693 |  67.33 |  54.03 |  - |  -
kabardian |  kbd |  en | wikiann |  1482 |  47.51 |  34.79 |  - |  -
common turkic |  turkish |  tr |  tr | wikiann |  10000 |  73.56 |  75.71 |  92.90 |  91.80
north azerbaijani |  az |  az | wikiann |  1000 |  67.31 |  61.01 |  89.17 |  88.26
crimean tatar |  crh |  tr | wikiann |  100 |  47.81 |  40.57 |  57.99 |  52.67
south azerbaijani |  azb |  az | wikiann |  2567 |  31.67 |  11.16 |  30.22 |  22.14
eastern romance |  romanian |  ro |  ro | wikiann |  10000 |  74.62 |  70.82 |  94.17 |  93.47
aromanian |  roa-rup |  el | wikiann |  732 |  64.78 |  62.66 |  68.92 |  63.33
moldavian |  mo |  ro | wikiann |  345 |  63.31 |  56.53 |  60.60 |  64.56
frisian |  western frisian |  fy |  nl | wikiann |  1000 |  80.17 |  71.96 |  81.06 |  78.21
ems-weser frisian |  stq |  nl | wikiann |  1085 |  59.45 |  57.47 |  64.55 |  55.15
northern frisian |  frr |  nl | wikiann |  100 |  46.67 |  46.22 |  54.78 |  52.63
gallo-italian |  piemontese |  pms |  it | wikiann |  100 |  79.53 |  71.10 |  87.88 |  78.72
lombard |  lmo |  it | wikiann |  100 |  72.49 |  68.17 |  79.37 |  78.66
venetian |  vec |  it | wikiann |  100 |  62.71 |  55.20 |  73.73 |  67.51
ligurian |  lij |  it | wikiann |  100 |  45.36 |  34.75 |  56.51 |  47.79
emiliano-romagnolo |  eml |  it | wikiann |  100 |  33.55 |  33.23 |  42.80 |  45.45
gallo-rhaetian |  french |  fr |  fr | wikiann |  10000 |  79.16 |  76.52 |  90.96 |  89.00
anglo-norman |  nrm |  fr | wikiann |  1281 |  66.78 |  88.56 |  71.98 |  71.92
arpitan |  frp |  it | wikiann |  2358 |  63.30 |  63.67 |  68.38 |  71.13
romansh |  rm |  it | wikiann |  100 |  56.88 |  55.19 |  69.69 |  67.58
friulian |  fur |  it | wikiann |  100 |  51.41 |  50.75 |  64.12 |  56.30
walloon |  wa |  fr | wikiann |  100 |  46.33 |  41.27 |  45.19 |  42.50
greater panjabic |  western panjabi |  pnb |  pa | wikiann |  100 |  64.46 |  53.78 |  17.82 |  0.00
eastern panjabi |  pa |  pa | wikiann |  100 |  32.90 |  45.25 |  30.63 |  0.00
greek |  modern greek |  el |  el | wikiann |  10000 |  71.76 |  72.96 |  91.18 |  90.68
pontic |  pnt |  el | wikiann |  291 |  66.37 |  69.79 |  68.45 |  72.58
high german |  german |  de |  de | wikiann |  10000 |  79.08 |  75.67 |  89.97 |  87.80
central alemannic |  als |  de | wikiann |  100 |  75.36 |  65.15 |  84.84 |  79.02
luxemburgish |  lb |  nl | wikiann |  1000 |  71.61 |  49.22 |  79.22 |  58.71
limburgan |  li |  nl | wikiann |  100 |  63.03 |  63.72 |  78.03 |  73.15
bavarian |  bar |  de | wikiann |  100 |  56.62 |  55.96 |  76.36 |  68.84
kölsch |  ksh |  nl | wikiann |  100 |  54.80 |  39.42 |  62.50 |  48.51
pfaelzisch-lothringisch |  pfl |  de | wikiann |  1092 |  49.40 |  47.19 |  59.12 |  50.14
pennsylvania german |  pdc |  de | wikiann |  100 |  41.76 |  41.79 |  39.39 |  38.76
hindustani |  fiji hindi |  hif |  hi | wikiann |  715 |  81.29 |  79.67 |  74.28 |  85.15
hindi |  hi |  hi | wikiann |  1000 |  65.74 |  65.77 |  88.11 |  85.83
inuit |  alaskan inupiaq |  ik |  en | wikiann |  431 |  76.27 |  70.56 |  - |  -
kalaallisut |  kl |  en | wikiann |  1403 |  63.20 |  60.20 |  - |  -
italian romance |  italian |  it |  it | wikiann |  10000 |  81.44 |  78.35 |  92.21 |  90.22
italian romance (a:apulia, m:spoken, i:tarantino) |  roa-tara |  it | wikiann |  3811 |  62.97 |  66.02 |  60.34 |  64.45
sicilian |  scn |  it | wikiann |  100 |  60.26 |  54.46 |  72.25 |  60.87
continental southern italian |  nap |  it | wikiann |  100 |  57.35 |  54.81 |  61.48 |  58.33
komi |  komi |  kv |  en | wikiann |  2464 |  56.78 |  41.78 |  - |  -
komi-permyak |  koi |  en | wikiann |  2798 |  53.62 |  47.80 |  - |  -
kurdish |  kurdish |  ku |  ku | wikiann |  100 |  31.67 |  59.48 |  13.70 |  0.00
central kurdish |  ckb |  ku | wikiann |  1000 |  6.90 |  35.49 |  1.17 |  0.00
latvian |  latvian |  lv |  lv | wikiann |  10000 |  69.36 |  71.07 |  93.24 |  92.30
east latvian |  ltg |  lv | wikiann |  1036 |  48.27 |  48.86 |  50.10 |  49.95
mari |  eastern mari |  mhr |  mhr | wikiann |  100 |  46.67 |  40.94 |  32.70 |  0.00
western mari |  mrj |  mhr | wikiann |  6036 |  38.29 |  58.46 |  5.51 |  0.00
modern dutch |  dutch |  nl |  nl | wikiann |  10000 |  82.03 |  80.42 |  91.96 |  90.51
western flemish |  vls |  nl | wikiann |  100 |  73.36 |  72.08 |  77.66 |  79.85
zeeuws |  zea |  nl | wikiann |  100 |  65.98 |  66.20 |  79.55 |  76.09
norwegian |  norwegian nynorsk (m:written) |  nynorsk |  nynorsk | norwegian_ner |  1511 |  87.58 |  87.86 |  87.58 |  87.86
norwegian (m:written, i:samnorsk) |  samnorsk |  samnorsk | norwegian_ner |  3450 |  86.96 |  90.55 |  86.96 |  90.55
norwegian bokmål (m:written) |  bokmaal |  bokmaal | norwegian_ner |  1939 |  85.82 |  90.54 |  85.82 |  90.54
sardo-corsican |  sardinian |  sc |  it | wikiann |  917 |  67.32 |  65.26 |  80.87 |  81.17
corsican |  co |  it | wikiann |  100 |  56.65 |  56.41 |  70.59 |  66.06
serbian-croatian-bosnian |  croatian standard |  hr |  hr | wikiann |  10000 |  77.59 |  78.12 |  92.40 |  91.02
bosnian standard |  bs |  hr | wikiann |  1000 |  69.93 |  74.88 |  87.50 |  88.86
serbian standard |  sr |  hr | wikiann |  10000 |  64.38 |  60.71 |  63.68 |  65.86
serbian-croatian-bosnian |  sh |  hr | wikiann |  10000 |  38.92 |  69.14 |  85.05 |  85.43
sinitic |  wu chinese |  wuu |  zh | wikiann |  100 |  71.89 |  35.80 |  74.15 |  63.73
min nan chinese |  zh-min-nan |  zh | wikiann |  100 |  44.68 |  47.30 |  21.40 |  15.08
cantonese |  zh-yue |  zh | wikiann |  10000 |  43.73 |  26.55 |  76.19 |  71.97
mandarin chinese |  zh |  zh | wikiann |  10000 |  42.86 |  24.71 |  81.12 |  77.22
classical chinese |  zh-classical |  zh | wikiann |  100 |  28.03 |  16.78 |  67.28 |  62.39
hakka chinese |  hak |  zh | wikiann |  100 |  27.43 |  31.84 |  36.55 |  40.00
sorbian |  lower sorbian |  dsb |  hsb | wikiann |  862 |  71.04 |  68.80 |  3.03 |  0.00
upper sorbian |  hsb |  hsb | wikiann |  100 |  65.44 |  65.48 |  38.20 |  0.00
sotho-tswana (s.30) |  southern sotho |  st |  en | wikiann |  339 |  64.36 |  69.26 |  - |  -
northern sotho |  nso |  en | wikiann |  720 |  19.08 |  29.66 |  - |  -
southwestern shifted romance |  galician |  gl |  es | wikiann |  10000 |  81.98 |  80.27 |  87.99 |  86.14
spanish |  es |  es | wikiann |  10000 |  72.80 |  70.73 |  92.17 |  90.32