# DiaNet: BERT and Hierarchical Attention Multi-Task Learning of Fine-Grained
Dialect

Muhammad Abdul-Mageed1 Chiyu Zhang1 AbdelRahim Elmadany1 Arun Rajendran1 Lyle
Ungar2
1Natural Language Processing Lab, University of British Columbia
2Computer and Information Science, University of Pennsylvania
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Prediction of language varieties and dialects is an important language
processing task, with a wide range of applications. For Arabic, the native
tongue of $\sim 300$ million people, most varieties remain unsupported. To
ease this bottleneck, we present a very large scale dataset covering 319
cities from all 21 Arab countries. We introduce a hierarchical attention
multi-task learning (HA-MTL) approach for dialect identification exploiting
our data at the city, state, and country levels. We also evaluate use of BERT
on the three tasks, comparing it to the MTL approach. We benchmark and release
our data and models.

##  1 Introduction

Language identification (LID) is a critical first step for multilingual NLP.
Especially for processing social media such as Twitter text in global
settings, the ability to identify languages, language varieties, and dialects
is indispensable. In addition to classical applications of LID as an enabling
technology in tasks such as machine translation, web data collection and
search, and pedagogical applications Jauhiainen et al. (2018), LID has
essential real-time applications as a source of information for tracking
health and well-being trends (Paul and Dredze, 2011). However, of the world’s
currently known 7,111 living languages, 111Source: https://www.ethnologue.com.
the great majority are yet to be supported by NLP tools such as LID. As
technology continues to play an increasingly impactful role in our lives,
access to nuanced NLP tools (including LID) becomes an issue of equity Jurgens
et al. (2017).

Figure 1: Hierarchical Attention MTL of city, state, and country. All models
share one BiGRU layer of 1,000 units. Layers 2-4 are also BiGRU layers, with
multi-head attention. Left: City network supervised at layer 2, state at layer
3, and country at layer 4. Right: Supervision is reversed from left network.

In spite of this key role of LID, it is still challenging to find tools for
closely related languages and varieties, including those that are widely
spoken. We focus on one such situation for the Arabic language, a large
collection of similar varieties with $\sim$ 300 million native speakers. For
Arabic, currently available NLP tools are limited to the standard variety of
the language, Modern Standard Arabic (MSA), and a small set of dialects such
as Egyptian, Levantine, and Iraqi. Arabic dialects differ amongst themselves
and from MSA at various levels, including phonological and morphological
Watson (2007), lexical Salameh et al. (2018); Abdul-Mageed et al. (2018);
Qwaider et al. (2018), syntactic Benmamoun (2011), and sociological Bassiouney
(2009, 2017). A major limitation to developing robust and equitable LID
technologies for Arabic has been absence of large, diverse data. A number of
pioneering efforts, including shared tasks (Zampieri et al., 2014; Malmasi et
al., 2016; Zampieri et al., 2018), have been invested to bridge this gap by
collecting datasets. However, these works either depend on automatic geocoding
of user profiles (Abdul-Mageed et al., 2018), which is not quite accurate, as
we show in Section 2, use a small set of dialectal seed words as a basis for
the collection (Zaghouani and Charfi, 2018; Qwaider et al., 2018), which
limits text diversity, or are based on translation of a small dataset of
sentences rather than naturally-occurring text Salameh et al. (2018).

To alleviate this bottleneck, we use location as a surrogate for dialect to
build a very large scale Twitter dataset ($\sim$ 6 billion tweets), and (1)
automatically label a subset of it ($\sim$ 500M tweets) with coverage for all
21 Arab countries at the nuanced levels of state and city (i.e., micro-
dialects). We also (2) manually label another subset ($\sim$ 2M tweets from
$\sim$ 5,000 users). We then develop highly effective supervised and weakly-
supervised models exploiting the data at all three nuanced levels of city,
state, and country. For modeling, we introduce a novel hierarchical attention
multi-task learning (HA-MTL) network that is suited to our task (shown in
Figure 1), which proves highly successful. We further investigate the newly-
proposed Bidirectional Encoder Representations from Transformers (BERT)
(Devlin et al., 2018) and show its effectiveness.

Concretely, we make the following contributions: (1) We collect a large-scale
dataset covering all Arabic varieties; (2) we introduce supervised and weakly-
supervised HA-MTL models exploiting our data at fine-grained levels; (3) we
empirically evaluate BERT on our tasks, showing its effectiveness; and (4) we
benchmark and release our data and models. The rest of the paper is organized
as follows: In Section 2, we introduce our Twitter data, quality assurance
methods, and the external data we use for comparisons. Section 3 describes our
methods. We present our supervised models in Section 4 and weakly-supervised
models in Section 5. We compare to other works in Section 6, evaluate our
models at the user level in Section 7, review related works in Section 8, and
conclude in Section 9.

##  2 Data

###  2.1 Creating a Large User-Level Collection

To develop a large scale dataset of Arabic varieties, we extracted $\sim$ 7.5
million Twitter user ids from several in-house Arabic Twitter corpora. The
corpora were collected with the Twitter streaming API, including using
bounding boxes around the Arab world. The data span $\sim$ 10 years
(2009-2019). We then use the Twitter API to crawl up to 3,200 tweets from a
random sample of $\sim$ 2.7 million users from the collection. Overall, we
acquired $\sim$ 6 billion tweets.

Figure 2: A map of all 21 Arab countries. States are demarcated in thin black
lines within each country. A total of 319 cities (from our user location
validation study, in colored circles) are overlayed within corresponding
countries.

###  2.2 Automatic City Tagging

We use the Python geocoding library geopy 222https://github.com/geopy. to
identify the user countries (e.g., Morocco) and cities (e.g., Beirut). Geopy
is a client for several popular geocoding web services aiming at locating the
coordinates of addresses, cities, countries, and landmarks across the world
using third-party geocoders. In particular, we use the Nominatim geocoder for
OpenStreetMap data 333https://nominatim.openstreetmap.org.. With Nominatim,
Geopy depends on user-provided geographic information in Twitter profiles such
as names of countries or cities to assign user location. Out of the 2.7
million users, we acquired both ‘city’ and ‘country’ label for 233,105 users
who contribute 507,318,355 tweets. The total number of cities initially tagged
was 705, but we manually map them to only 646 as we explain next.

###  2.3 Correction of City and State Tags

City-Level. Investigating examples of the geolocated data, we observed geopy
made some mistakes. To solve the issue, we decided to manually verify the
information returned from geopy on all the 705 assumed ‘cities’. For this
purpose of manual verification, we use Wikipedia, Google maps, and web search
as sources of information while checking city names. We found that geopy made
mistakes in 7 cases as a result of misspelled city names in the queries we
sent (as coming from user profiles). We also found that 44 cases were not
assigned the correct city name as the first ‘solution’. Geopy provided us with
a maximum of 7 solutions for a query, with best solutions sometimes being
names of hamlets, villages, etc., rather than cities. In many cases, we found
the correct solution to fall between the 2nd and 4th solutions. A third
problem was that some city names (as coming from user profiles) were written
in non-Arabic (e.g., English or French). We solved this issue by requiring
geopy to also return the English version of a city name, and exclusively using
that English version. Ultimately, we acquired a total of 646 cities.

State-Level. Geopy also returned to us a total of 192 states/provinces that
correspond to the 646 cities. We manually verified all the state names, and
their correspondence to the cities and countries and found no issues.

###  2.4 Data Pre-processing

To keep only high-quality data, we apply the following procedures: First, we
remove all re-tweets, decreasing the collection to 318,174,122 tweets. Second,
we normalize the tweets by reducing 2 or more consecutive sequences of the
same character to only 2, replace usernames with $<USER>$ and URLs with
$<URL>$. Finally, we remove all tweets with less than three actual Arabic
words. This further reduces the collection to 277,430,807 tweets. Since for
most Arabic varieties there are no available tokenizers, we tokenize input
text only lightly by splitting off punctuation.

###  2.5 Validation of User Location

After manually correcting the city and state names, we needed to verify that a
given user actually belongs to the automatically assigned location labels
(city, state, and country). To achieve this, we first excluded cities that
have $<500$ tweets and users with $<30$ tweets from the data. This gave us 319
cities. We then ask two native Arabic annotators to label the data. Their job
was to consider the automatic label for each task (city and country) 444Note
that we have already manually established the link between states and their
corresponding cities and countries. and assign one label from the set {true,
false, unknown} per task for each user in the collection. We trained the
annotators and instructed them to examine the profile information of each user
on Twitter, providing a link to the profile. We asked them to consider various
sources of information as a basis for their decisions, including (1) the
profile picture, (2) profile textual description (including user-provided
location), (3) the actual name of the user (if available), (4) at least 10
tweets, (5) the followers and followees of the user, and (5) user’s network
behavior such as the ‘likes’. Each annotator was responsible for $\sim 50\%$
of the usernames and was given a random sample of 20 users for each city along
with the Twitter handles and the automatically assigned city and country
labels. We asked the users to label the first 10 accounts in each city, and
only add more if the city proves specially challenging (as we observed to be
the case in a pilot analysis of a few cities). Annotators ended up labeling a
total of 4,953 accounts, of whom 4,012 users were verified for both country
and city locations. We found that 81.00% of geopy tags for country are
correct, but only 62.29% for city. As a final sanity check, a third annotator
reviewed the labels for a random sample of 20 users from each annotator and
agreed fully. Figure 4 shows a map of all 21 Arab countries, each divided into
its states with cities overlayed as colored small circles. We now describe the
external datasets we use for comparisons.

Country | %vld_cntry | %vld_city | #tweets
---|---|---|---
Algeria | 77.49 | 69.74 | 185,854
Bahrain | 83.95 | 39.51 | 25,495
Djibouti | 68.42 | 68.42 | 3,939
Egypt | 92.66 | 64.02 | 463,695
… | … | … | …
Yemen | 72.41 | 56.32 | 47,450
Avg/Total | 81.00 | 62.29 | 2,025,013
Table 1: A subset of our gold data from manually verified users.

###  2.6 External Data

Arap-Tweet Zaghouani and Charfi (2018) comprises 17 countries collected from
1,100 manually-verified Twitter users based on a seed-word approach. The
dataset totals 2.4M tweets. In comparison, our dataset covers more countries,
has more nuanced tags (on cities and states), and is extracted from more
users, thus making it more diverse (since we also do not use seed words to
find our users). Zaghouani and Charfi (2018) do not perform classification
exploiting their data. We split Arap-Tweet into 80% TRAIN, 10% DEV, and 10%
TEST. SHAMI Qwaider et al. (2018) is a Twitter and web fora dataset of
Jordanian, Lebanese, Palestinian, and Syrian Arabic collected with a seed-word
approach. It has 66,249 manually labeled tweets. In comparison, our dataset is
much larger, covers more countries, and is more diverse. We split SHAMI into
TRAIN (80%), DEV (10%), and TEST (10%) for our experiments, thus using less
training data than Qwaider et al. (2018) who employ cross-validation.

MADAR Shared Task-2 Bouamor et al. (2019) is a dataset released for the MADAR
Twitter User Dialect Identification Shared Task 2. The dataset is distributed
as train, dev, and test (without labels) with user and tweet ids. We were able
to crawl the data for a total of 2,311 users, acquiring $193,086$,$26,588$,
and $43,909$ tweets for the three splits, respectively. We call training data
TRAIN-I as we also create another training set (TRAIN-II) that is a
concatenation of task 2 and task 1 data. 555Task 1 is also organized by
Bouamor et al. (2019).

##  3 Methods

We perform dialect identification at the country, state, and city levels. We
use two main classification methods, Gated Recurrent Units (GRUs) Cho et al.
(2014), a variation of recurrent neural networks (RNN), and Google’s
bidirectional masked language model based on transformers (BERT) Devlin et al.
(2018). We now describe each of these methods.

###  3.1 GRU

A Gated Recurrent Unit (GRU) Cho et al. (2014) is a type of cell proposed to
simplify recurrent neural network (RNN) learning. It makes use of an update
gate $\textbf{{z}}^{(t)}$ and a reset gate $\textbf{{r}}^{(t)}$. The
activation of GRU at time step $t$ is a linear interpolation of the previous
activation hidden state $\textbf{{h}}^{(t-1)}$ and the candidate activation
hidden state $\textbf{{$\widetilde{h}$}}^{(t)}$. The update state
$\textbf{{z}}^{(t)}$ decides how much the unit updates its content, and the
candidate activation makes use of a reset gate $\textbf{{r}}^{(t)}$. When its
value is close to zero, the reset gate allows the unit to forget the
previously computed state.

###  3.2 Multi-Task Learning

Figure 3: Our MTL network for city, state, and country. The three tasks share
2 hidden layers, with each task having its independent attention layer.

We investigate the utility of multi-task learning (MTL) for language ID. The
intuition behind MTL is that many real-world tasks involve making predictions
about closely related labels or outcomes. For related tasks, MTL helps achieve
inductive transfer between the various tasks by leveraging additional sources
of information from some of the tasks to improve performance on the target
task Caruana (1993). By using training signals for related tasks, MTL allows a
learner to prefer hypotheses that explain more than one task Caruana (1997)
and also helps regularize models.

In single task learning, an independent network is trained in isolation for
each task. In contrast, in MTL, a number of tasks are learned together in a
single network, with each task having its own output. An MTL network has a
shared input, and one or more hidden layers that are shared between all the
tasks. Backpropagation is then applied in parallel on all outputs. In our
case, we train a single network for our city, state, and country tasks with
one output for each of the three tasks. Figure 3 is an illustration of an MTL
network for our 3 tasks, with 2 shared hidden BiGRU layers and a task-specific
(i.e., independent) BiGRU attention layer. In our current work, each of the
three tasks has its own loss function, with the MTL loss computed as:

$\begin{multlined}\mathcal{L}(\theta_{MTL})\\\
=\left(\mathcal{L}(\theta_{city})+\mathcal{L}(\theta_{state})+\mathcal{L}(\theta_{country})\right)/3\\\
\end{multlined}\mathcal{L}(\theta_{MTL})\\\
=\left(\mathcal{L}(\theta_{city})+\mathcal{L}(\theta_{state})+\mathcal{L}(\theta_{country})\right)/3\\\
$ (1)

We now introduce the Transformer Vaswani et al. (2017), since both our
attention mechanism and BERT Devlin et al. (2018) are based on it.

###  3.3 Transformer

The Transformer (Vaswani et al., 2017) is based solely on attention. Similar
to most other sequence transduction models Bahdanau et al. (2014); Cho et al.
(2014); Sutskever et al. (2014), it is an encoder-decoder architecture. It
takes a sequence of symbol representations $x^{(i)}\dots x^{(n)}$, maps them
into a sequence of continuous representations $z^{(i)}\dots x^{(n)}$ that are
then used by the decoder to generate an output sequence $y^{(i)}\dots
y^{(n)}$, one symbol at a time. This is performed using self-attention, where
different positions of a single sequence are related to one another. The
Transformer employs an attention mechanism based on a function that operates
on queries, keys, and values. The attention function maps a query and a set of
key-value pairs to an output, where the output is a weighted sum of the
values. For each value, a weight is computed as a compatibility function of
the query with the corresponding key. We implement and apply the multi-head
attention function to our BiGRU models.

Encoder of the Transformer in Vaswani et al. (2017) has 6 attention layers,
each of which is composed of two sub-layers: (1) multi-head attention where,
rather than performing a single attention function with queries, keys, and
values, these are projected h times into linear, learned projections and
ultimately concatenated; and (2) fully-connected feed-forward network (FFN)
that is applied to each position separately and identically. Decoder of the
Transformer also employs 6 identical layers, similar to the encoder, yet with
an extra/third sub-layer that performs multi-head attention over the encoder
stack. As mentioned, the Transformer is the core learning component in BERT
Devlin et al. (2018), which we now introduce.

###  3.4 BERT

BERT Devlin et al. (2018) stands for Bidirectional Encoder Representations
from Transformers. It is an approach for pre-training language representations
that involves two unsupervised learning tasks, (1) masked language models
(Masked LM) and (2) next sentence prediction. Since BERT uses bidirectional
conditioning, a given percentage of random input tokens are masked and the
model attempts to predict these masked tokens. . Devlin et al. (2018) mask 15%
of the tokens (the authors use word pieces) and feed the final hidden vectors
of these masked tokens to an output softmax over the vocabulary. The next
sentence prediction task of BERT is also straightforward. Devlin et al. (2018)
simply cast the task as binary classification. For a given sentence S, two
sentences A and B are generated where A (positive class) is an actual sentence
from the corpus and B is a randomly chosen sentence (negative class). Once
trained on an unlabeled dataset, BERT can then be fine-tuned with supervised
data.

##  4 Gold-Supervised Models

Setting | City | State | Country
---|---|---|---
Eval Metric | acc | F1 | acc | F1 | acc | F1
Baseline I (majority in TRAIN) | 1.313 | 0.008 | 3.110 | 0.032 | 9.191 | 0.802
Baseline II (single task Attn-BiGRU) | 2.740 | 0.880 | 4.450 | 0.910 | 27.170 | 12.820
MTL (common-attn) | 4.036 | 1.693 | 5.693 | 2.195 | 28.255 | 13.362
MTL (spec-attn) | 4.000 | 1.479 | 5.956 | 2.085 | 28.946 | 13.858
HA-MTL (city first) | 12.295 | 11.736 | 13.728 | 12.836 | 40.349 | 29.869
HA-MTL (country first) | 11.265 | 10.588 | 13.577 | 12.869 | 41.250 | 29.763
BERT | 19.329 | 19.452 | 19.329 | 19.452 | 47.743 | 38.122
Table 2: Performance on TEST. Highest results for MTL are underlined. BERT
results (best) are in bold.

In this section, we explain how we split our data and specify our baseline and
evaluation metrics. We then present our gold-supervised models, namely (i) our
single-task models (Section 4.2), (ii) multi-task models (Section 4.3) and
(iii) our BERT model (Section 3.4).

###  4.1 Data Splits, Baseline, and Evaluation

We randomly split our own manually-verified dataset into 80% training (TRAIN),
10% development (DEV), and 10% test (TEST). To limit the GPU hours needed for
processing, we cap the number of tweets in our TRAIN in any given country at
100K. This reduces the TRAIN size from 1,620,436 to 1,099,711. Our DEV set has
202,509 tweets, and our TEST has 202,068 tweets. 666The distribution of
classes in our splits is in the supplementary material. For all our
experiments, we remove diacritics from the input text. We use two baselines:
the majority class in TRAIN (Baseline I) and a single-task BiGRU (Baseline II,
described in Section 4.2). For all our experiments, we tune model hyper-
parameters and identify best architectures on DEV. We run all models for 15
epochs, with early stopping ‘patience’ value of 5 epochs, choosing the model
that performs highest on DEV as our best model. We then run each best model on
TEST, and report accuracy and macro $F_{1}$ score. 777We include a table with
results on DEV in the supplementary material.

###  4.2 Single-Task BiGRUs

As a second baseline (Baseline II), we build an independent network for each
of the 3 tasks using the same architecture and model capacity. Each network
has 3 hidden BiGRU layers, 888We also ran single-task networks with 4 hidden
layers, but we find them to overfit quickly even when we regularize with
dropout at $0.7$ on all layers. with 1,000 units each (500 units from left to
right and 500 units from right to left). We add multi-head attention only to
the third hidden layer. We trim each sequence at 50 words, 999In initial
experiments, we found a maximum sequence of $30$ words to perform slightly
worse. and use a batch size of 8. Each word in the input sequence is
represented as a vector of 300 dimensions that are learned directly from the
data. Word vectors weights $W$ are initialized with a standard normal
distribution, with $\mu=0$, and $\sigma=1$, i.e., $W\sim N(0,1)$. For
optimization, we use Adam Kingma and Ba (2014) with a fixed learning rate of
$1e-3$. For regularization, we use dropout Srivastava et al. (2014) with a
value of $0.5$ on each of the 3 hidden layers. Table 2 presents our results on
TEST.

###  4.3 MTL

With MTL, we design a single network to learn the 3 tasks simultaneously. In
addition to our hierarchical attention MTL (HA-MTL) network, we design two
architectures that differ as to how we endow the network with the attention
mechanism. We describe these next.

####  4.3.1 Shared and Task-Specific Attention

We first design networks with attention at the same level in the architecture.
Note that we use the same hyper-parameters as the single-task networks. We
have two configurations:

Shared Attention: In this configuration, we design a network with 3 hidden
BiGRU layers, each of which has 1,000 units per layer (500 in each direction).
101010Again, 4 hidden-layered network for both the shared and task-specific
attention settings were sub-optimal and so we do not report their results
here. All the 3 layers are shared across the 3 tasks, including the third
layer. Only the third layer has attention applied. We refer to this setting as
MTL-common-attn.

Task-Specific Attention: This network is similar to the previous one in that
the first two hidden layers are shared, but differs in that the third layer
(attention layer) is task-specific (i.e., independent for each task). We refer
to this setting as MTL-spec-attn. Figure 3 illustrates our MTL network for
learning city, with task-specific attention. This architecture allows each
task to specialize its own attention within the same network. As Table 2
shows, both MTL-common-attn and MTL-spec-attn improve over each of the two
baselines, and are consistently complimentary: While the first acquires better
acc, the latter is slightly better in $F_{1}$ score.

###  4.4 Hierarchical Attention MTL (HA-MTL)

We design a single network for the 3 tasks, but with supervision at different
layers. Overall, the network has 4 BiGRU layers (each with a total of 1,000
units), the bottom-most of which has no attention. Layers 2, 3, and 4 each has
multi-head attention applied, followed directly by one task-specific fully-
connected layer with softmax for class prediction. This is the architecture
depicted in Figure 1. On the left side of Figure 1, we show the city-first
hierarchical attention network, with city supervised at the second hidden
layer. On the right side, we have the country-first network, where country is
supervised earlier (at the second layer). In the two scenarios, state is
supervised at the middle layer. These two architectures allow information flow
with different granularity: While the city-first network tries to capture what
is in the physical world a more fine-grained level (city), the country-first
network does the opposite. Again, we use the same hyper-parameters as the
single-task and MTL networks, but we use a dropout rate of 0.70 since we find
it to work better. As Table 2 shows, our proposed HA-MTL models significantly
outperform single-task and other MTL models. They outperform our Baseline II
with 9.555%, 9.277%, and 14.079% acc on city, state, and country prediction
respectively, thus demonstrating their effectiveness on the task.

###  4.5 BERT Models

We use the BERT-Base, Multilingual Cased model released by the authors
111111https://github.com/google-research/bert/blob/master/multilingual.md..
The model is trained on 104 languages, including Arabic, with 12 layer, 768
hidden units each, 12 attention heads, and has 110M parameters. The model has
119,547 word pieces for each language. For fine-tuning, we use a maximum
sequence size of 50 words and a batch size of 32. We set the learning rate to
2e-5. We train for 15 epochs, as mentioned earlier. As Table 2 shows, BERT
performs consistently better on the three tasks. It outperforms the best of
our two HA-MTL networks with an acc of 7.034% (city), 5.601% (state), 6.493%
(country). And $F-_{1}$ of 7.716, 6.583, and 8.253 for the 3 tasks,
respectively.

##  5 Learning From Noisy Labels

Supervision | acc | F1
---|---|---
Baseline I (majority in TRAIN) | 9.207 | 0.843
Baseline II (Gold) | 46.844 | 37.643
Weak | 41.166 | 23.697
Weak+Gold | 49.768 | 38.254
Weak_then_Gold | 47.862 | 38.560
Table 3: Results on TEST with models exploiting noisy labels on 20 countries
(with Djibouti excluded). For comparison, our gold (BERT trained on human-
labeled TRAIN) is re-trained with 20 classes.

In contrast to our gold-supervised models (Section 4), this set of experiments
is focused on learning from noisy labels. We only perform experiments on
predicting country labels. Our goal is to answer the question “To what extent
can automatically acquired labels in our dataset be beneficial for learning?”.
To this end, we remove human annotated users from our larger automatically
labeled pool and use only data tagged with any of the 319 cities whose users
we manually labeled. Keeping tweets with at least 3 Arabic words, we acquire
1,161,651 tweets from 3,195 users, across all countries except Djibouti (all
whose users were already in our human annotation round). As such, we have 20
countries in this dataset and refer to it simply as Auto-Tagged. We exploit
Auto-Tagged in 3 experimental settings, reporting results on our gold TEST in
all 3 cases. The 3 settings are: (1) Weakly Supervised: Where fine-tune BERT
exclusively on Auto-Tagged; (2) Weak+Gold: Where concatenate Auto-Tagged and
our TRAIN (gold), shffle the dataset, and fine-tune BERT on it; and (3) Weak-
Then-Gold: Where fine-tune BERT on Auto-Tagged first, then resume fine-tuning
on our human labeled data (TRAIN). Table 3 shows Weak+Gold to improve 2.923%
acc over our Gold model (Baseline II), establishing the utility of using noisy
labels on the country level.

##  6 Comparisons to Other Models

Since the existing data described in Section 2.6 have varying numbers of
classes (different from our data), we train BERT on their respective TRAIN
splits (as described in Section 2.6). While Qwaider et al. (2018) use linear
classifiers to model their data, there are no models we know of for Arap-Tweet
Zaghouani and Charfi (2018) nor MADAR Bouamor et al. (2019). As such, we
publish the first results on these two datasets. As a baseline, we run a
unidirectional 1-layered GRU, with 500 units, on each of Arap-Tweet and MADAR.
121212We evaluate on MADAR DEV set.

As Table 4 shows, our models outperform Qwaider et al. (2018) on SHAMI. We
also establish new results on both Arap-Tweet and MADAR. Note that we do not
report on the dataset described in Abdul-Mageed et al. (2018) since it is
automatically labeled, and so is noisy. We also do not compare to the dataset
in Salameh et al. (2018) since it is small and not naturally occurring (2,000
translated sentences per class). 131313 Salameh et al. (2018) report that
linear classifiers outperform deep learning models due to small data set size.

Dataset | Model | #cls | acc | F1
---|---|---|---|---
ARAB-TWT | GRU-500 | 17 | 38.787 | 39.171
| Ours | 17 | 54.606 | 55.066
MADAR | GRU-500 | 21 | 46.810 | 29.840
| Ours, TRAIN-I | 21 | 48.499 | 33.929
| Ours, TRAIN-II | 21 | 49.394 | 35.931
SHAMI | Qwaider et al.18 | 4 | 70.000 | 71.000
| Ours | 4 | 86.065 | 85.464
Table 4: Results on external data. Best performance and new results where
there are no models to compare to are bolded.

##  7 User-Level Evaluation

Our models are not designed to directly detect the dialect of a user, but
rather takes a single tweet input at a time. However, we test how the model
will fare on detecting user-level dialect given a certain number of tweets
from a random user. For the purpose, we crawl up to 500 tweets from each of
500 users from the MADAR Bouamor et al. (2019) user base and extract the
following number of tweets from each user: {10, 25, 50, 75, 100, 500}. We run
our best performing BERT model (from the 21 countries in Table 2) on these
user tweets, one tweet at a time. Taking the majority class on each user’s
tweets, we find that with 100 tweets, for example, the model can reach 65.171%
acc and with 500 tweets, it can reach 66.787% acc. 141414We provide the full
results table for user-level evaluation in supplementary material. 151515In
our 233,105 automatically tagged users, 94.85% have $>=$100 tweets, suggesting
a model based on only 100 tweets would have very high coverage.

##  8 Related Work

Arabic Dialects. Most of the early categorizations of Arabic dialects
arbitrarily depended on cross-country geographical divisions Habash (2010);
Versteegh (2014). More recent treatments such as Abdul-Mageed et al. (2018),
Salameh et al. (2018), Qwaider et al. (2018),and Zaghouani and Charfi (2018)
focus on more fine-grained levels of dialectness, e.g., country and city
levels. These works are more aligned with sociolinguistic work, e.g., Labov
(1964) and Trudgill (1974), showing language can vary at smaller regions such
as different parts of the same city, thus creating micro-dialects within the
same dialect. The finest Arabic variations treated in the literature cover 25
to 29 cities Salameh et al. (2018); Abdul-Mageed et al. (2018). To the best of
our knowledge, our work constitutes the most fine-grained attempt to classify
Arabic varieties, including micro-dialects. We also use a much larger dataset
than previous works.

Dialectal Arabic Data and Models. Once primarily spoken, Arabic varieties came
into written form with the proliferation of social media. Much of the early
work focused on collecting data for main varieties such as Egyptian and
Levantine Diab et al. (2010); Elfardy and Diab (2012); Al-Sabbagh and Girju
(2012); Sadat et al. (2014); Zaidan and Callison-Burch (2011). Many works
developed models for detecting 2-3 dialects Elfardy and Diab (2013); Zaidan
and Callison-Burch (2011, 2014); Cotterell and Callison-Burch (2014). These
works, e.g., Elfardy and Diab (2013) and Tillmann et al. (2014), mostly
exploit AOC Zaidan and Callison-Burch (2011). Larger datasets, mainly based on
Twitter, were recently introduced Mubarak and Darwish (2014); Abdul-Mageed et
al. (2018); Zaghouani and Charfi (2018). Our dataset (labeled and unlabeled)
is orders of magnitude than available datasets, and by far the most fine-
grained.

Geolocation. Relevant to our work is also research on geolocation Han et al.
(2016); Do et al. (2018). Rather than predicting geolocation, we focus on
urban locations such as cities and states as surrogates for micro-dialects.

MTL. MTL has been successfully applied to many NLP problems, including MT and
syntactic parsing Luong et al. (2015), sequence labeling Søgaard and Goldberg
(2016); Rei (2017), and text classification Liu et al. (2016). As we have
shown, MTL is well-suited to fine-grained dialect prediction and, to the best
to our knowledge, we are the first to apply it to this problem.

##  9 Conclusion

We proposed an approach for using location as a surrogate for dialect aiming
at building a very large scale Twitter dataset of Arabic varieties. Our data
and models cover varieties from all 21 Arab countries, including the nuanced
levels of city and state. We also introduced an effective hierarchical
attention multi-task learning (HA-MTL) approach for modeling varieties and
micro-dialects. Furthermore, we empirically demonstrated the utility of BERT
on our tasks. In addition, we benchmarked our data and models for release and
reported new state-of-the-art results on a number of external datasets.
Ultimately, our work has the potential to open up opportunities for
investigating variants of Arabic that remain largely understudied. The work is
also a first step toward deployment of Arabic NLP technologies in real-world
applications, such as in disaster and emergency situations where diverse
varieties are in actual use.

## References

  * Abdul-Mageed et al. (2018) Muhammad Abdul-Mageed, Hassan Alhuzali, and Mohamed Elaraby. 2018.  You tweet what you speak: A city-level dataset of arabic dialects.  In _LREC_ , pages 3653–3659.
  * Al-Sabbagh and Girju (2012) Rania Al-Sabbagh and Roxana Girju. 2012.  Yadac: Yet another dialectal arabic corpus.  In _LREC_ , pages 2882–2889.
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_.
  * Bassiouney (2009) Reem Bassiouney. 2009.  _Arabic sociolinguistics_.  Edinburgh University Press.
  * Bassiouney (2017) Reem Bassiouney. 2017.  _Identity and dialect performance: A study of communities and dialects_.  Routledge.
  * Benmamoun (2011) Elabbas Benmamoun. 2011.  Agreement and cliticization in arabic varieties from diachronic and synchronic perspectives.  _al-’Arabiyya_ , pages 137–150.
  * Bouamor et al. (2019) Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019.  Madar twitter user dialect identification (shared task 2).  In _Proceedings of the The Fourth Workshop for Arabic Natural Language Processing (WANLP2019)_ , pages 00–00.
  * Caruana (1993) R. Caruana. 1993.  Multitask learning: A knowledge-based source of inductive bias.  In _Proceedings of the 10th International Conference on Machine Learning_.
  * Caruana (1997) Rich Caruana. 1997.  Multitask learning.  _Machine learning_ , 28(1):41–75.
  * Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_.
  * Cotterell and Callison-Burch (2014) Ryan Cotterell and Chris Callison-Burch. 2014.  A multi-dialect, multi-genre corpus of informal written arabic.  In _LREC_ , pages 241–245.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Diab et al. (2010) Mona Diab, Nizar Habash, Owen Rambow, Mohamed Altantawy, and Yassine Benajiba. 2010\.  Colaba: Arabic dialect annotation and processing.  In _Lrec workshop on semitic language processing_ , pages 66–74.
  * Do et al. (2018) Tien Huu Do, Duc Minh Nguyen, Evaggelia Tsiligianni, Bruno Cornelis, and Nikos Deligiannis. 2018.  Twitter user geolocation using deep multiview learning.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6304–6308. IEEE.
  * Elfardy and Diab (2013) Heba Elfardy and Mona Diab. 2013.  Sentence level dialect identification in arabic.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , volume 2, pages 456–461.
  * Elfardy and Diab (2012) Heba Elfardy and Mona T Diab. 2012.  Simplified guidelines for the creation of large scale dialectal arabic annotations.  In _LREC_ , pages 371–378.
  * Habash (2010) Nizar Y Habash. 2010.  Introduction to arabic natural language processing.  _Synthesis Lectures on Human Language Technologies_ , 3(1):1–187.
  * Han et al. (2016) Bo Han, Afshin Rahimi, Leon Derczynski, and Timothy Baldwin. 2016.  Twitter geolocation prediction shared task of the 2016 workshop on noisy user-generated text.  In _Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT)_ , pages 213–217.
  * Jauhiainen et al. (2018) Tommi Jauhiainen, Marco Lui, Marcos Zampieri, Timothy Baldwin, and Krister Lindén. 2018.  Automatic language identification in texts: A survey.  _arXiv preprint arXiv:1804.08186_.
  * Jurgens et al. (2017) David Jurgens, Yulia Tsvetkov, and Dan Jurafsky. 2017.  Incorporating dialectal variability for socially equitable language identification.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , volume 2, pages 51–57.
  * Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Labov (1964) William Labov. 1964.  _he social stratification of En- glish in New York City_.  Ph.D. thesis, Columbia university.
  * Liu et al. (2016) Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016.  Recurrent neural network for text classification with multi-task learning.  _arXiv preprint arXiv:1605.05101_.
  * Luong et al. (2015) Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015\.  Multi-task sequence to sequence learning.  _arXiv preprint arXiv:1511.06114_.
  * Malmasi et al. (2016) Shervin Malmasi, Marcos Zampieri, Nikola Ljubešic, Preslav Nakov, Ahmed Ali, and Jörg Tiedemann. 2016.  Discriminating between similar languages and arabic dialect identification: A report on the third dsl shared task.  _VarDial 3_ , page 1.
  * Mubarak and Darwish (2014) Hamdy Mubarak and Kareem Darwish. 2014.  Using twitter to collect a multi-dialectal corpus of arabic.  In _Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP)_ , pages 1–7.
  * Paul and Dredze (2011) Michael J Paul and Mark Dredze. 2011.  You are what you tweet: Analyzing twitter for public health.  In _Fifth International AAAI Conference on Weblogs and Social Media_.
  * Qwaider et al. (2018) Chatrine Qwaider, Motaz Saad, Stergios Chatzikyriakidis, and Simon Dobnik. 2018\.  Shami: A corpus of levantine arabic dialects.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018)_.
  * Rei (2017) Marek Rei. 2017.  Semi-supervised multitask learning for sequence labeling.  _arXiv preprint arXiv:1704.07156_.
  * Sadat et al. (2014) Fatiha Sadat, Farnazeh Kazemi, and Atefeh Farzindar. 2014.  Automatic identification of arabic language varieties and dialects in social media.  _Proceedings of SocialNLP_ , page 22.
  * Salameh et al. (2018) Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018.  Fine-grained arabic dialect identification.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1332–1344.
  * Søgaard and Goldberg (2016) Anders Søgaard and Yoav Goldberg. 2016.  Deep multi-task learning with low level tasks supervised at lower layers.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , volume 2, pages 231–235.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _The Journal of Machine Learning Research_ , 15(1):1929–1958.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014.  Sequence to sequence learning with neural networks.  In _NIPS_.
  * Tillmann et al. (2014) Christoph Tillmann, Saab Mansour, and Yaser Al-Onaizan. 2014.  Improved sentence-level arabic dialect classification.  In _Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects_ , pages 110–119.
  * Trudgill (1974) Peter Trudgill. 1974.  Linguistic change and diffusion: Description and explanation in sociolinguistic dialect geography.  _Language in society_ , 3(2):215–246.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , pages 6000–6010.
  * Versteegh (2014) Kees Versteegh. 2014.  _The arabic language_.  Edinburgh University Press.
  * Watson (2007) Janet CE Watson. 2007.  _The phonology and morphology of Arabic_.  Oxford University Press.
  * Zaghouani and Charfi (2018) Wajdi Zaghouani and Anis Charfi. 2018.  Arap-tweet: A large multi-dialect twitter corpus for gender, age and language variety identification.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018)_.
  * Zaidan and Callison-Burch (2011) Omar F Zaidan and Chris Callison-Burch. 2011.  The arabic online commentary dataset: an annotated dataset of informal arabic with high dialectal content.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2_ , pages 37–41. Association for Computational Linguistics.
  * Zaidan and Callison-Burch (2014) Omar F Zaidan and Chris Callison-Burch. 2014.  Arabic dialect identification.  _Computational Linguistics_ , 40(1):171–202.
  * Zampieri et al. (2018) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Ahmed Ali, Suwon Shon, James Glass, Yves Scherrer, Tanja Samardžić, Nikola Ljubešić, Jörg Tiedemann, et al. 2018.  Language identification and morphosyntactic tagging: The second vardial evaluation campaign.  In _Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects_. Association for Computational Linguistics.
  * Zampieri et al. (2014) Marcos Zampieri, Liling Tan, Nikola Ljubešic, and Jörg Tiedemann. 2014\.  A report on the dsl shared task 2014.  In _Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects_ , pages 58–67.

##  Appendix A Summary of Supplementart Material

We provide the following supplementary items:

  1. 1.

Figure 4 is a bigger version of the map of all 21 Arab countries, with
corresponding states and cities provided in the manuscript.

  2. 2.

Table 5 shows statistics of the dataset for 233,105 users for which we
acquired geotags.

  3. 3.

Table 6 provides statistics across the 21 countries of our gold data from the
manually verified users.

  4. 4.

Table 5 provides statistics across the TRAIN, DEV, and TEST splits in our gold
dataset, after capping dominant classes at 100,000 tweets each.

  5. 5.

Tables 8 and 9 show results of our supervised models, in both DEV and TEST
data. We re-produce TEST results here for convenience.

  6. 6.

Tables 10 and 11 show our results from experiments exploiting noisy labels.
Again, we re-produce TEST results here for convenience.

  7. 7.

Table 12 shows results on user-level evaluation, with different sizes of
tweets per user.

Figure 4: A bigger-sized map of all 21 Arab countries. States are demarcated in thin black lines within each country. A total of 319 cities (from our user location validation study, in colored circles) are overlayed within corresponding countries. Countries |  | #Tweets |  |
---|---|---|---|---
Name | Code | #Users | Collected | -Retweets | Normalized | #States | #Cities
Algeria | dz | 1,960 | 3,939,411 | 2,889,447 | 2,324,099 | 47 | 200
Bahrain | bh | 1,080 | 2,801,399 | 1,681,337 | 1,385,533 | 4 | 4
Djibouti | dj | 6 | 11,901 | 9,173 | 8,790 | 1 | 1
Egypt | eg | 42,858 | 92,804,863 | 61,264,656 | 47,463,301 | 27 | 56
Iraq | iq | 4,624 | 7,514,750 | 4,922,553 | 4,318,523 | 18 | 62
Jordan | jo | 3,806 | 7,796,794 | 5,416,413 | 4,209,815 | 4 | 5
KSA | sa | 136,455 | 297,264,647 | 177,751,985 | 165,036,420 | 11 | 31
Kuwait | kw | 4,466 | 11,461,531 | 7,984,758 | 6,628,689 | 4 | 14
Lebanon | lb | 1,364 | 3,036,432 | 1,893,089 | 1,160,167 | 6 | 19
Libya | ly | 2,083 | 4,227,802 | 3,109,355 | 2,655,180 | 21 | 32
Mauritania | mr | 102 | 209,131 | 148,261 | 129,919 | 4 | 4
Morocco | ma | 1,729 | 3,407,741 | 2,644,733 | 1,815,947 | 17 | 117
Oman | om | 4,260 | 8,139,374 | 4,866,813 | 4,259,780 | 8 | 17
Palestine | ps | 2,854 | 6,004,791 | 4,820,335 | 4,263,491 | 2 | 12
Qatar | qa | 5,047 | 11,824,490 | 7,891,425 | 6,867,304 | 2 | 2
Somalia | so | 78 | 168,136 | 131,944 | 104,946 | 8 | 9
Sudan | sd | 1,162 | 2,348,325 | 1,522,274 | 1,171,866 | 14 | 27
Syria | sy | 1,630 | 2,992,106 | 2,184,715 | 1,889,455 | 12 | 19
Tunisia | tn | 227 | 460,268 | 362,806 | 239,769 | 10 | 10
UAE | ae | 14,923 | 36,121,319 | 23,309,788 | 18,484,296 | 7 | 15
Yemen | ye | 2,391 | 4,783,144 | 3,368,262 | 3,013,517 | 8 | 8
Total | 233,105 | 507,318,355 | 318,174,122 | 277,430,807 | 235 | 664
Table 5:  Statistics of our data representing 233,105 users from 664 cities and 21 countries. We process more than half a billion tweets, from a larger pool of $\sim$ 6 billion tweets, to acquire our final dataset. Note that the number of states and cities is further reduced after our manual user verification. Eventually, we acquire data for 319 cities, belonging to 192. The data represent all 21 Arab countries. Country | %vld_cntry | %vld_city | #tweets
---|---|---|---
Algeria | 77.49 | 69.74 | 185,854
Bahrain | 83.95 | 39.51 | 25,495
Djibouti | 68.42 | 68.42 | 3,939
Egypt | 92.66 | 64.02 | 463,695
Iraq | 51.50 | 37.61 | 59,287
Jordan | 83.61 | 54.10 | 17,958
KSA | 96.37 | 62.88 | 353,057
Kuwait | 84.30 | 34.88 | 65,036
Lebanon | 92.42 | 56.06 | 37,273
Libya | 75.48 | 72.03 | 128,152
Maurit. | 45.00 | 35.00 | 3,244
Morocco | 75.59 | 62.42 | 140,341
Oman | 90.25 | 77.97 | 108,846
Palestine | 87.50 | 82.35 | 87,446
Qatar | 85.00 | 77.50 | 29,445
Somalia | 52.73 | 45.45 | 9,640
Sudan | 56.88 | 41.28 | 23,642
Syria | 76.28 | 71.63 | 79,649
Tunisia | 78.95 | 75.94 | 26,300
UAE | 85.31 | 82.49 | 129,264
Yemen | 72.41 | 56.32 | 47,450
Avg/Total | 81.00 | 62.29 | 2,025,013
Table 6: Our gold data, from manually verified users. Country | TRAIN | DEV | TEST
---|---|---|---
Algeria | 100,000 | 18,700 | 18,572
Bahrain | 20,387 | 2,556 | 2,552
Djibouti | 3,158 | 408 | 373
Egypt | 100,000 | 46,136 | 46,325
Iraq | 47,395 | 5,903 | 5,989
Jordan | 14,413 | 1,826 | 1,719
KSA | 100,000 | 35,312 | 35,106
Kuwait | 52,127 | 6,416 | 6,493
Lebanon | 29,821 | 3,641 | 3,811
Libya | 100,000 | 12,847 | 12,803
Maurit. | 2,579 | 338 | 327
Morocco | 100,000 | 14,118 | 13,862
Oman | 87,048 | 10,807 | 10,991
Palestine | 69,834 | 8,668 | 8,944
Qatar | 23,624 | 2,968 | 2,853
Somalia | 7,678 | 1,023 | 939
Sudan | 18,929 | 2,334 | 2,379
Syria | 63,668 | 7,987 | 7,994
Tunisia | 21,164 | 2,599 | 2,537
UAE | 100,000 | 13,089 | 12,768
Yemen | 37,886 | 4,833 | 4,731
Total | 1,099,711 | 202,509 | 202,068
Table 7: Distribution of classes in our data splits. Setting | City | State | Country
---|---|---|---
Eval Metric | acc | F1 | acc | F1 | acc | F1
Baseline I (majority in TRAIN) | 1.313 | 0.008 | 3.110 | 0.032 | 9.191 | 0.802
Baseline II (single task Attn-BiGRU) | 2.110 | 0.450 | 3.840 | 0.360 | 21.250 | 7.390
MTL (common-attn) | 4.070 | 1.714 | 5.634 | 2.152 | 28.297 | 13.404
MTL (spec-attn) | 4.083 | 1.593 | 5.921 | 2.196 | 29.082 | 14.058
HA-MTL (city first) | 12.384 | 11.791 | 13.696 | 12.894 | 40.784 | 30.090
HA-MTL (country first) | 11.214 | 10.289 | 13.460 | 12.696 | 40.942 | 30.179
BERT | 19.528 | 19.818 | 21.199 | 21.671 | 47.567 | 38.297
Table 8: Performance on DEV. Highest results for MTL are underlined. BERT results (best) are in bold. Setting | City | State | Country
---|---|---|---
Eval Metric | acc | F1 | acc | F1 | acc | F1
Baseline I (majority in TRAIN) | 1.313 | 0.008 | 3.110 | 0.032 | 9.191 | 0.802
Baseline II (single task Attn-BiGRU) | 2.740 | 0.880 | 4.450 | 0.910 | 27.170 | 12.820
MTL (common-attn) | 4.036 | 1.693 | 5.693 | 2.195 | 28.255 | 13.362
MTL (spec-attn) | 4.000 | 1.479 | 5.956 | 2.085 | 28.946 | 13.858
HA-MTL (city first) | 12.295 | 11.736 | 13.728 | 12.836 | 40.349 | 29.869
HA-MTL (country first) | 11.265 | 10.588 | 13.577 | 12.869 | 41.250 | 29.763
BERT | 19.329 | 19.452 | 19.329 | 19.452 | 47.743 | 38.122
Table 9: Performance on TEST. Highest results for MTL are underlined. BERT results (best) are in bold. Supervision | acc | F1
---|---|---
Baseline (majority in TRAIN) | 9.207 | 0.843
Gold | 46.808 | 37.863
Weak | 41.195 | 23.560
Weak+Gold | 49.700 | 38.651
weak_then_Gold | 47.862 | 38.560
Table 10: Results on DEV with models exploiting noisy labels on 20 countries (with Djibouti excluded). For comparison, our gold (BERT trained on human-labeled TRAIN) is re-trained with 20 classes. Supervision | acc | F1
---|---|---
Baseline (majority in TRAIN) | 9.207 | 0.843
Gold | 46.844 | 37.643
Weak | 41.166 | 23.697
Weak+Gold | 49.768 | 38.254
Weak_then_Gold | 47.862 | 38.560
Table 11: Results on TEST with models exploiting noisy labels on 20 countries (with Djibouti excluded). For comparison, our gold and small-GRU are re-trained with 20 classes. #tweets | acc | thresh | F1 | thresh
---|---|---|---|---
10 | 45.132 | 0.62 | 39.131 | 0.83
25 | 54.872 | 0.70 | 48.211 | 0.72
50 | 62.006 | 0.65 | 54.833 | 0.75
75 | 64.012 | 0.75 | 56.809 | 0.85
100 | 65.171 | 0.95 | 60.335 | 0.95
500 | 66.787 | 0.57 | 58.661 | 0.67
Table 12: User-level evaluation on external data (crawled from the MADAR user
base). Note that we take a thresholded majority class of predicted tweets as a
user-level tag. For thresholding, we use the per-class softmax value in the
model’s output layer.


# VinaLLaMA: LLaMA-based Vietnamese Foundation Model

Quan Nguyen, Huy Pham and Dung Dao Virtual Interactive,
<EMAIL_ADDRESS>Interactive<EMAIL_ADDRESS>Interactive,
<EMAIL_ADDRESS>
###### Abstract

In this technical report, we present VinaLLaMA, an open-weight, state-of-the-
art (SOTA) Large Language Model for the Vietnamese language, built upon
LLaMA-2 with an additional 800 billion trained tokens. VinaLLaMA not only
demonstrates fluency in Vietnamese but also exhibits a profound understanding
of Vietnamese culture, making it a truly indigenous model. VinaLLaMA-7B-chat,
trained on 1 million high-quality synthetic samples, achieves SOTA results on
key benchmarks, including VLSP, VMLU, and Vicuna Benchmark Vietnamese, marking
a significant advancement in the Vietnamese AI landscape and offering a
versatile resource for various applications.

##  1 Introduction

The surge in Large Language Models (LLMs) such as ChatGPT and GPT-4 has
significantly advanced the field of artificial intelligence (AI), particularly
in language processing. In 2023, Vietnam’s AI sector witnessed a notable
development with the introduction of several Vietnamese-centric LLMs,
including BLOOMZ’s Vietcuna, URA-LLaMA, PhoGPT, and dama-2. Amidst this
progression, we introduce VinaLLaMA, a foundational LLM designed specifically
for the Vietnamese language. VinaLLaMA, built on top of LLaMA-2, represents a
vital stride towards linguistic inclusivity in AI, adeptly addressing the
syntactic and semantic intricacies of Vietnamese.

Embracing the spirit of collaboration and open innovation, we are pleased to
announce VinaLLaMA, an open-weight Foundation Language Model and its chat
variant. These models are now accessible on HuggingFace, ensuring
compatibility with all ’transformers’ framework-supported libraries. This
endeavor not only contributes to the global AI research landscape but also
provides a specialized tool for exploring and enhancing Vietnamese language
processing, encouraging a wider engagement and application in AI-driven NLP
research.

##  2 Related Work

###  2.1 Large Language Models

The development of large language models (LLMs) began with the Transformer
[VSP+17], the foundational architecture that enabled the pre-training of BERT
[CMCC23] and GPT [RNSS] using large-scale unsupervised data. This advancement
led to the emergence of popular foundation models such as RoBERTa [LOG+19], T5
[RSR+20], and BART [LLG+20].

Subsequently, GPT-3 [BMR+20] demonstrated capabilities in few-shot and zero-
shot learning through prompt engineering and in-context learning. This was
followed by significant advancements with ChatGPT (OpenAI, 2022) and GPT-4
(OpenAI, 2023), which have made substantial contributions not only in language
modeling but also in the broader field of artificial intelligence. These
models exhibit a diverse range of skills and produce high-quality outputs,
leading to the rise of newer LLMs such as Llama [RSR+20], Llama-2 [TMS+23],
Bloom [WSF+22], Falcon [AAA+23], Qwen [BBC+23], and Mistral [JSM+23].

###  2.2 Vietnamese Large Language Models

The Vietnamese LLM landscape has seen significant advancements with models
like Vietcuna [NPD23b], which underwent further pre-training on BLOOMZ
[WSF+22], followed by fine-tuning with the OpenOrca-Viet [NPD23a] dataset.
Another model, URA-Llama [NTN+23], was developed through fine-tuning on a
corpus of Vietnamese articles, building upon Llama-2’s architecture. DopikAI’s
ViGPT™ [NTNV23] also made strides by continuing pre-training and supervised
fine-tuning atop GPT-J-Vietnamese-News [Vie21], pre-trained by VietAI.
Additionally, PhoGPT [NNT+23] represents a significant development, initially
pre-trained on a 41GB text corpus, and later fine-tuned with a dataset of
150,000 samples, enhancing its proficiency in Vietnamese text understanding
and generation.

###  2.3 Alignment

Instruction tuning is a critical aspect of aligning large language models
(LLMs) with human interaction patterns, enhancing their ability in in-context
learning, reasoning, and instruction learning. Flan [WBZ+21] introduced this
approach with a comprehensive dataset featuring a wide array of tasks and
templates, closely mimicking human instructional styles. This approach
involves refining the model’s responses to align more closely with human
instructional patterns, thereby making the model’s output more intuitive and
user-friendly.

InstructGPT [OWJ+22] extended this approach by incorporating human feedback,
refining the models to produce less toxic and higher quality text. This method
of emulating human interaction in instruction tuning has been pivotal in
improving the performance and user-friendliness of LLMs.

Further advancements include the integration of chat data from open-assistant
datasets in models like Vicuna and Guanaco, and the use of synthetic datasets
generated by GPT-3.5 and GPT-4, such as OpenOrca [WKM+22] and Platypus
[LHR23]. These developments have further enhanced the conversational
capabilities of LLMs, making them more adept at handling real-world
interaction scenarios.

##  3 Training Procedure

###  3.1 Pretraining

LLaMA-2, a highly regarded pre-trained language model in English, shows a
significant gap in handling Vietnamese-related content due to limited relevant
tokens in its training set. Additionally, its original tokenizer falls short
in multilingual applications. To address these issues, we compiled a new
pretraining dataset combining public and synthetic in-house data. We selected
BKAI’s LLaMA-2-chat tokenizer [Lab23] for the tokenizer, which has been
specifically made for Vietnamese. This tokenizer shows enhanced performance in
processing the Vietnamese language, making it a suitable choice for our
VinaLLaMA model.

####  3.1.1 Public Data

Books. The dataset encompassing Vietnamese literature in our study is
comprehensive, comprising 250,000 volumes. This extensive collection spans
various domains, including science, history, finance, and philosophy, as well
as fiction genres like novels and science fiction, in addition to traditional
Vietnamese literature. The distribution of these categories is methodically
illustrated in Figure 1.

Public News. The dataset under examination is derived from two principal
Vietnamese news sources, VnExpress 111VnExpress and BaoMoi 222BaoMoi. This
compilation encompasses an exhaustive collection of articles disseminated by
these entities from January 1, 2010, through September 30, 2023. To align with
ethical guidelines and content appropriateness standards, a filtration process
was implemented, systematically excluding articles that contained keywords
indicative of harmful or violent content. The inclusion of this public news
data is crucial, as it helps the model better understand and represent
important aspects related to Vietnam and its people.

Figure 1: Distribution of Book Topics Used in VinaLLaMA training.

Finally, we also include a subset of Vietnamese from CulturaX [NNL+23] and an
additional 100B tokens in English. The final public dataset has a total of
roughly 330 billion tokens.

####  3.1.2 In-house Data

Influenced by the concept of synthetic textbooks for pretraining [GZA+23], our
approach incorporates a mechanism that selects random text segments from a
publicly available dataset. These segments are then integrated into over 80
bespoke prompt templates, each meticulously crafted to facilitate the
rewriting task. The prompts, when fed into GPT-4, result in roughly 100,000
samples. These samples represent a high-quality synthetic dataset,
specifically tailored for pretraining purposes. The methodology and workflow
of this process are illustrated in Figure 2.

Figure 2: The first stage of generating synthetic data for pretraining Figure
3: Our second stage of generating synthetic data for pretraining.
Vietcuna-3B-v2 was deployed to save cost while maintaining the same
performance with distillation from GPT-4

Additionally, we employed Vietcuna-3B-v2, our successor smaller-scale language
model, to train on the synthetic samples generated in Step 1. This training
process utilized the rewriting task-specific prompts previously developed (as
outlined in Step 2.1), which is detailed in Figure 3. Subsequently, we
replicated the procedure from Step 1 using this newly trained model. This
iteration resulted in the generation of over 500 billion synthetic tokens, a
process detailed in Step 2.2. The final result of this process is more than
500 billion of high-quality Vietnamese tokens ready to be used to continue
pretraining on the base LLaMA 2 with the expanded tokenizer.

###  3.2 Supervised Instruction Fine-tuning

In collaboration with Nous Research 333Nous Research, we employed proprietary
methodologies to create 500,000 Vietnamese synthetic samples. These samples
encompassed both instructional and conversational formats. To enhance the
dataset, an additional 500,000 English samples were sourced from the
OpenHermes-2.5 [tek23] and Capybara [DS23] datasets, also provided by Nous
Research. This integration of datasets culminated in a comprehensive
collection of 1 million samples, paving the way for the development of a
bilingual language model. The comprehensive final dataset encompasses an array
of tasks, including reasoning, role-playing, poem writing, coding, function
calling, and agent prompting. This diversity ensures a broad scope of
capabilities in the final model. We conducted a full fine-tuning of our pre-
trained model over four epochs, which led to the creation of a state-of-the-
art foundation language model specifically tailored for the Vietnamese
language, more on the result in Section 4.

###  3.3 VinaLLaMA-2.7B

Adopting the structured pruning procedure outlined in [XGZC23], we
successfully reduced our model to a more compact variant with 2.7 billion
parameters. This process involved strategically pruning the network while
retaining its core functional capabilities. Following the reduction in size,
we applied the same supervised fine-tuning methodology to this smaller variant
as was used for the 7B model, ensuring consistency in the training approach
across different model scales.

###  3.4 Training Details

For our pretraining phase, we utilized a cluster consisting of eight nodes,
each equipped with 8x Intel Habana Gaudi2 Processors. This phase was completed
over the course of one week. In contrast, the fine-tuning phase was conducted
more rapidly, utilizing a single node of Google Cloud TPU v5e, and completed
within a single day.

Additionally, our smaller 2.7 billion parameters variant, underwent an easier
process. This variant was both continued pre-trained and fully fine-tuned over
a period of five days. This process was carried out on a single node, which
was provisioned with 8x NVIDIA A100 80GB GPUs. The use of such high-
performance hardware facilitated the efficient handling of the substantial
computational demands associated with training and fine-tuning a model of this
scale.

##  4 Evaluation

In our research, we utilized three distinct evaluation benchmarks: VLSP, VMLU,
and the Vicuna Benchmark, with the latter being translated into Vietnamese by
VinAI Research. For each of these benchmarks, we implemented a dual-evaluation
approach. Specifically, we conducted separate experiments for two categories
of models: those in their pre-trained state and those that had undergone
instructional fine-tuning. This methodology allowed us to compare the baseline
capabilities of the pre-trained models against their performance post-
instructional fine-tuning, providing insights into the effectiveness of fine-
tuning in enhancing model proficiency within the context of these Vietnamese
language benchmarks.

For the pre-training benchmarks, we selected VLSP’s hoa-7b, BKAI-LLaMA-2,
ViGPT™, BLOOM-7B, and PhoGPT-7B5 as the base models. In contrast, the
instructional fine-tuned benchmarks comprised BLOOMZ-7B, ViGPT™-170K,
Vietcuna-7B-v3, PhoGPT-7B5-Instruct, URA-LLaMA-13B and SeaLLM-7B-chat. This
diverse set of models provided a comprehensive perspective, enabling us to
assess both the inherent capabilities of these models in their original form
and their enhanced performance post-fine-tuning, specifically tailored to the
Vietnamese language benchmarks.

###  4.1 VLSP

The first benchmark employed is the VLSP-LLM 2023 [CHC+23]. This benchmark
parallels the OpenLLM Leaderboard by HuggingFace, tailored specifically for
the Vietnamese language. It encompasses four distinct assessments: ARC
Challenge, HellaSwag, MMLU, and TruthfulQA, each meticulously adapted to
Vietnamese. This comprehensive benchmark suite allows for a robust evaluation
of language models in understanding and generating Vietnamese text across
various domains and complexities. The results are reported in Table 1 and
Table 2. In the pretrained segment of our study, the VinaLLaMA-7B model
demonstrated superior performance compared to other open-source Large Language
Models (LLMs) that support Vietnamese. This was evident in its outperformance
on three of the four benchmarks, leading to it achieving state-of-the-art
results based on the average score across these benchmarks. This signifies the
robustness and effectiveness of the VinaLLaMA-7B model in handling a variety
of tasks in the Vietnamese language context.

In the supervised fine-tuning domain, our results continued to reflect a high
level of excellence. Notably, VinaLLaMA-7B-chat, comparable in scale,
astonishingly outperformed larger models, including those with 13 billion
parameters, in terms of average score. This impressive achievement highlights
the efficacy of the fine-tuning process and the model’s adeptness at
leveraging its training to deliver outstanding performance, even against much
larger counterparts. Adding to this, the VinaLLaMA-2.7B, a significantly
smaller model, showcased remarkable performance. It not only competed closely
with larger 7B variants but also exceeded PhoGPT-7B5-Instruct in performance
while achieving inferencing speeds 60% faster. This showcases the model’s
optimized balance between size, speed, and performance. The success of
VinaLLaMA-7B-chat, and the noteworthy performance of VinaLLaMA-2.7B,
underscore the considerable potential of well-implemented fine-tuning
strategies with synthetic data in elevating the capabilities of Large Language
Models, especially in contexts requiring specialized language processing.

Model | arc_vi | hellaswag_vi | mmlu_vi | truthfulqa_vi | Average
---|---|---|---|---|---
| (25-shot) | (10-shot) | (0-shot) | (0-shot) |
hoa-7b | 0.2855 | 0.4329 | 0.2536 | 0.4542 | 0.3566
BKAI-LLaMA-2 | 0.2975 | 0.4402 | 0.3039 | 0.4542 | 0.3740
BLOOM-7B | 0.3255 | 0.4830 | 0.2810 | 0.4701 | 0.3899
ViGPT™ | 0.2596 | 0.3877 | 0.2482 | 0.4612 | 0.3392
PhoGPT-7B5 | 0.2496 | 0.2577 | 0.2474 | 0.4677 | 0.3056
VinaLLaMA-2.7B | 0.2906 | 0.4337 | 0.2490 | 0.4608 | 0.3585
VinaLLaMA-7B | 0.3350 | 0.4956 | 0.3168 | 0.4552 | 0.4007

Table 1: VLSP Benchmark Scores of Pretrained Models

Model | arc_vi | hellaswag_vi | mmlu_vi | truthfulqa_vi | Average
---|---|---|---|---|---
| (25-shot) | (10-shot) | (0-shot) | (0-shot) |
URA-LLaMA-13B | 0.3752 | 0.4830 | 0.3973 | 0.4574 | 0.4282
BLOOMZ-7B | 0.3205 | 0.4930 | 0.3975 | 0.4523 | 0.4158
ViGPT™-170K | 0.3651 | 0.4777 | 0.3412 | 0.4691 | 0.4133
PhoGPT-7B5-Instruct | 0.2470 | 0.2578 | 0.2413 | 0.4759 | 0.3055
SeaLLM-7B-chat | 0.3607 | 0.5112 | 0.3339 | 0.4948 | 0.4252
Vietcuna-7b-v3 | 0.3419 | 0.4939 | 0.3354 | 0.4807 | 0.4130
VinaLLaMA-2.7B-chat | 0.3274 | 0.4814 | 0.3051 | 0.4972 | 0.4028
VinaLLaMA-7B-chat | 0.4239 | 0.5407 | 0.3932 | 0.5251 | 0.4707

Table 2: VLSP Benchmark Scores of Supervised Fine-tuning Models

###  4.2 VMLU

VMLU [AoST23], a benchmark suite tailored for evaluating foundation models in
the Vietnamese language, comprises 10,880 multiple-choice questions across 58
subjects in domains like STEM, Humanities, and Social Sciences. Its diverse
range of difficulty levels tests models from basic knowledge to advanced
problem-solving. We selected VMLU for benchmarking due to its comprehensive
coverage of Vietnam-related questions, providing a relevant and challenging
environment for assessing model capabilities in a context-specific manner. We
conducted our experiments on the validation set of VMLU since the answers to
the test set are not publicly available, URA-LLaMA-13B is also not being
tested due to the lack of availability at testing time. We reported the
results under two few-shot settings: 0-shot and 5-shot, which can be viewed in
Table 3 and 4.

Model | (0-shot) | (5-shot)
---|---|---
ViGPT™ | 0.2379 | 0.2769
hoa-7b | 0.2513 | 0.2903
BKAI-LLaMA-2 | 0.2678 | 0.2513
BLOOM-7B | 0.2527 | 0.2312
PhoGPT-7B5 | 0.2352 | 0.2325
VinaLLaMA-2.7B | 0.2245 | 0.2688
VinaLLaMA-7B | 0.3199 | 0.3414

Table 3: VMLU Benchmark Scores of Pretrained Models

Model | (0-shot) | (5-shot)
---|---|---
ViGPT™-170K | 0.2419 | 0.2567
BLOOMZ-7B | 0.3945 | 0.3831
PhoGPT-7B5-Instruct | 0.2298 | 0.2379
SeaLLM-7B-chat | 0.2903 | 0.3508
Vietcuna-7b-v3 | 0.3441 | 0.3199
VinaLLaMA-2.7B-chat | 0.2702 | 0.2567
VinaLLaMA-7B-chat | 0.4046 | 0.4140

Table 4: VMLU Benchmark Scores of Supervised Fine-tuning Models

###  4.3 Vicuna Benchmark Vietnamese

The Vicuna Benchmark [ZCS+23], translated into Vietnamese by VinAI, serves as
our final benchmark. This comprehensive benchmark is composed of 80 distinct
instructions spanning 9 diverse areas, providing a broad spectrum for
assessing model capabilities. Uniquely, the evaluation of the results from all
participating models is conducted using GPT-4, which introduces an innovative
approach to performance assessment. This methodology employs an ELO ranking
system, traditionally used in chess and other competitive games, to rate the
models. Such a system offers a dynamic and relative measure of model
performance, allowing for a nuanced and comparative analysis of each model’s
proficiency in handling a variety of tasks and instructions within the
benchmark. This ELO-based evaluation provides a clear and quantifiable ranking
of the models, reflecting their effectiveness and adaptability in the context
of the Vietnamese language and the specific challenges presented by the Vicuna
Benchmark.

In our Vicuna Benchmark evaluation, responses from models were assessed using
a detailed five-point scale: 0 (’very bad’), 1 (’bad’), 2 (’ok’), 3 (’good’),
and 4 (’very good’). This granular scoring system allows for an in-depth
evaluation of the quality of each model’s response. The final ELO score for
each model is computed by aggregating these individual ratings, providing a
holistic measure of a model’s overall performance across the benchmark’s
varied tasks.

To ensure language relevance, we implemented a strict rule: any response not
in Vietnamese is automatically assigned a score of 0. This criterion
underscores the importance of language-specific accuracy in our evaluation.
The collective results, reflecting model performances across different tasks,
are visually represented in Figure 4.

In the interest of transparency and further research, we have made VinaLLaMA’s
responses and our evaluation code publicly available 444Viet-Eval-GPT-4
555vilm-ai/viet-llm-eval. This not only allows for independent verification of
our results but also facilitates further advancements in the field by
providing valuable resources to other researchers.

Figure 4: Score Distributions Assigned by GPT-4 Across Different Models: VinaLLaMA’s Exceptional Performance Competing Closely with ChatGPT-3.5-Turbo in Multiple Benchmark Assessments Model | Coding | Common-Sense | Counterfactual | Fermi
---|---|---|---|---
PhoGPT-7B5-Instruct | 0.000 | 3.600 | 3.100 | 0.000
ViGPT™-170K | 0.500 | 2.500 | 1.600 | 2.400
URA-LLaMA-13B | 0.714 | 0.600 | 0.200 | 0.000
URA-LLaMA-7B | 0.000 | 0.200 | 1.100 | 0.000
ChatGPT | 4.000 | 4.000 | 4.000 | 3.700
VinaLLaMA-7B-chat | 2.714 | 4.000 | 4.000 | 3.500
Table 5: Average Scores of Different Models Across Tasks in Vicuna
Benchmark(Part 1)

The benchmark results revealed that VinaLLaMA showcased commendable
performance in Vietnamese, closely trailing behind ChatGPT-3.5-Turbo in some
benchmarks. This indicates that VinaLLaMA is highly effective in Vietnamese
language tasks, with only a slight margin separating it from the more advanced
ChatGPT-3.5-Turbo, as shown in Table 5 and Table 6.

PhoGPT also demonstrated better outcomes compared to other Vietnamese
benchmarks, yet it still fell significantly behind in some areas, highlighting
potential avenues for improvement.

In contrast, URA-LLaMA-7B and 13B models performed less effectively, with some
of their responses not aligning with Vietnamese language requirements. This
points to a critical need for targeted enhancements or language-specific
training for these models to better cater to Vietnamese language processing.

Model | Generic | Knowledge | Math | Roleplay | Writing
---|---|---|---|---|---
PhoGPT-7B5-Instruct | 3.500 | 3.900 | 0.000 | 3.800 | 3.900
ViGPT™-170K | 3.500 | 3.300 | 0.000 | 2.900 | 2.900
URA-LLaMA-13B | 0.500 | 0.200 | 1.333 | 0.000 | 0.000
URA-LLaMA-7B | 0.200 | 0.300 | 0.000 | 0.000 | 0.600
ChatGPT | 4.000 | 4.000 | 4.000 | 4.000 | 4.000
VinaLLaMA-7B-chat | 4.000 | 4.000 | 4.000 | 4.000 | 4.000
Table 6: Average Scores of Different Models Across Tasks in Vicuna
Benchmark(Part 2)

###  4.4 HuggingFace OpenLLM Leaderboard (English)

Our approach in developing VinaLLaMA involved continued pretraining and fine-
tuning with both English and Vietnamese data, aiming to establish it as a
bilingual large language model. This strategy is key to enhancing its
linguistic versatility and adaptability.

We have documented VinaLLaMA’s performance metrics on the HuggingFace OpenLLM
Leaderboard, providing a comparative analysis with other open-source models.
These results, detailed in Table 7, offer insights into VinaLLaMA’s standing
in the realm of bilingual language models. This information is crucial for
understanding its bilingual proficiency and for benchmarking its capabilities
against existing models in the field.

Model | arc | hellaswag | mmlu | truthfulqa | winogrande | GSM8K | Average
---|---|---|---|---|---|---|---
| (25-shot) | (10-shot) | (5-shot) | (0-shot) | (5-shot) | (5-shot) |
LLaMA-2 7B-chat-hf | 0.5290 | 0.7855 | 0.4832 | 0.4557 | 0.7174 | 0.0735 | 0.5074
BLOOMZ-7B | 0.4078 | 0.6209 | 0.3613 | 0.4522 | 0.6535 | 0.0008 | 0.4161
MPT-7B-chat | 0.4625 | 0.7587 | 0.3762 | 0.4056 | 0.6843 | 0.0409 | 0.4547
Nous-Capybara-7B | 0.5520 | 0.7876 | 0.4880 | 0.4907 | 0.7340 | 0.0690 | 0.5202
VinaLLaMA-2.7B-chat | 0.3891 | 0.6556 | 0.3323 | 0.4838 | 0.5904 | 0.1350 | 0.4310
VinaLLaMA-7B-chat | 0.5000 | 0.7293 | 0.4753 | 0.4969 | 0.6346 | 0.3002 | 0.5227

Table 7: HuggingFace OpenLLM Leaderboard Benchmark Scores of Supervised Fine-
tuning Models

The performance of both the VinaLLaMA-7B-chat and 2.7B-chat models was
particularly noteworthy, not only achieving the highest overall scores but
also showing remarkable strength in mathematical benchmarks, specifically the
GSM8K. This highlights their capability in complex problem-solving and
analytical reasoning, areas often challenging for language models.

Remarkably, VinaLLaMA-7B-chat exhibited exceptional performance, surpassing
even the reinforcement learning-enhanced variants of Meta’s LLaMA-2-chat. This
achievement is significant, considering the advanced nature of reinforcement
learning models and their typically strong performance in such tasks. The
success of VinaLLaMA-7B-chat in this regard underscores its advanced
capabilities and positions it as a leading model in the domain of mathematics
and logic-based problem-solving.

##  5 Acknowledgement

We are also immensely grateful to the teams at Nous Research, LAION, and
Symato Team. Their assistance in the early stages of VinaLLaMA’s evaluation
was crucial, providing insights that guided further improvements and
enhancements of the model. We also want to send our appreciation to Nam Pham
666Nam Pham, HuggingFace, Nhan Nguyen 777Nhan Nguyen, HuggingFace, @Teknium1
888@Teknium, X and @ldjconfirmed 999@ldjconfirmed, X for their contribution to
our dataset creation.

Lastly, our profound appreciation is extended to Google Cloud and StabilityAI.
Their generous computational support was a cornerstone in bringing VinaLLaMA
to fruition, enabling the intensive training and development processes
required for such a sophisticated large language model. Their contribution was
vital in transforming our vision for VinaLLaMA into a reality.

##  6 Conclusion

In conclusion, the development of VinaLLaMA marks a significant milestone in
the realm of language models, particularly in the context of Vietnamese
language processing. Achieving state-of-the-art (SOTA) scores across all
Vietnamese benchmarks, VinaLLaMA has demonstrated exceptional proficiency and
adaptability. While its performance in English benchmarks was slightly less
dominant, it still showed considerable competence, underscoring its
effectiveness as a bilingual model.

A key factor in VinaLLaMA’s success is the strategic use of carefully crafted
synthetic data in its training regimen. This approach has proven to be highly
effective, yielding impressive results and highlighting the importance of
diverse and well-designed training datasets in the development of robust
language models.

VinaLLaMA’s achievements not only set a new standard for language models in
Vietnamese but also contribute valuable insights into the broader field of
natural language processing. It exemplifies how meticulous data preparation
and comprehensive training can significantly enhance the capabilities of
language models, paving the way for future advancements in this dynamic and
rapidly evolving domain.

## References

  * [AAA+23] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, et al.  The falcon series of open language models.  arXiv preprint arXiv:2311.16867, 2023.
  * [AoST23] Zalo AI, Japan Advanced Institute of Science, and Technology.  Vmlu: A vietnamese multitask language understanding benchmark suite for large language models.  2023\.
  * [BBC+23] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.  Qwen technical report.  arXiv preprint arXiv:2309.16609, 2023.
  * [BMR+20] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [CHC+23] Le Anh Cuong, Nguyen Trong Hieu, Nguyen Viet Cuong, Nguyen Ngoc Quoc, Le-Minh Nguyen, and Cam-Tu Nguyen.  Vlsp 2023 challenge on vietnamese large language models.  2023\.
  * [CMCC23] Llinet Benavides Cesar, Miguel-Ángel Manso-Callejo, and Calimanut-Ionut Cira.  Bert (bidirectional encoder representations from transformers) for missing data imputation in solar irradiance time series.  Engineering Proceedings, 39(1):26, 2023.
  * [DS23] Luigi Daniele and Suphavadeeprasit.  Amplify-instruct: Synthetically generated diverse multi-turn conversations for effecient llm training.  arXiv preprint arXiv:(comming soon), 2023.
  * [GZA+23] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li.  Textbooks are all you need, 2023.
  * [JSM+23] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  arXiv preprint arXiv:2310.06825, 2023.
  * [Lab23] BKAI-HUST Foundation Models Lab.  Llama-2-bk, 2023.
  * [LHR23] Ariel N Lee, Cole J Hunter, and Nataniel Ruiz.  Platypus: Quick, cheap, and powerful refinement of llms.  arXiv preprint arXiv:2308.07317, 2023.
  * [LLG+20] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, 2020.
  * [LOG+19] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  2019\.
  * [NNL+23] Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen.  Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages, 2023.
  * [NNT+23] Dat Quoc Nguyen, Linh The Nguyen, Chi Tran, Dung Ngoc Nguyen, Nhung Nguyen, Thien Huu Nguyen, Dinh Phung, and Hung Bui.  Phogpt: Generative pre-training for vietnamese.  arXiv preprint arXiv:2311.02945, 2023.
  * [NPD23a] Quan Nguyen, Huy Pham, and Dung Dao.  Openorca-viet.  2023\.
  * [NPD23b] Quan Nguyen, Huy Pham, and Dung Dao.  Vietcuna.  2023\.
  * [NTN+23] Duc Q. Nguyen, Sang T. Truong, Toan D. V. Nguyen, Dong D. Le, Nhi N. Truong, and Tho Quan.  Ura-llama: Universal adapted large language model for vietnamese.  2023\.
  * [NTNV23] Minh Thuan Nguyen, Khanh Tung Tran, Nhu Van Nguyen, and Xuan-Son Vu.  ViGPTQA - state-of-the-art LLMs for Vietnamese question answering: System overview, core models training, and evaluations.  In Mingxuan Wang and Imed Zitouni, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 754–764, Singapore, December 2023. Association for Computational Linguistics.
  * [OWJ+22] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27730–27744. Curran Associates, Inc., 2022.
  * [RNSS] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training.
  * [RSR+20] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
  * [tek23] teknium.  teknium/openhermes-2.5-mistral-7b.  2023\.
  * [TMS+23] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [Vie21] VietAI.  Gpt-j 6b on vietnamese news, 2021.
  * [VSP+17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [WBZ+21] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.  Finetuned language models are zero-shot learners.  In International Conference on Learning Representations, 2021.
  * [WKM+22] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi.  Openorca.  arXiv preprint arXiv:2212.10560, 2022.
  * [WSF+22] BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al.  Bloom: A 176b-parameter open-access multilingual language model.  arXiv preprint arXiv:2211.05100, 2022.
  * [XGZC23] Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen.  Sheared llama: Accelerating language model pre-training via structured pruning, 2023.
  * [ZCS+23] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.  Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.