# Cross-Lingual Text Classification of Transliterated
Hindi and Malayalam

Jitin Krishnan Antonios Anastasopoulos Hemant Purohit Huzefa Rangwala
George Mason University
Fairfax, VA, USA
<EMAIL_ADDRESS>
###### Abstract

Transliteration is very common on social media, but transliterated text is not
adequately handled by modern neural models for various NLP tasks. In this
work, we combine data augmentation approaches with a Teacher-Student training
scheme to address this issue in a cross-lingual transfer setting for _fine-
tuning_ state-of-the-art pre-trained multilingual language models such as
mBERT and XLM-R. We evaluate our method on transliterated Hindi and Malayalam,
also introducing new datasets for benchmarking on real-world scenarios: one on
sentiment classification in transliterated Malayalam, and another on crisis
tweet classification in transliterated Hindi and Malayalam (related to the
2013 North India and 2018 Kerala floods). Our method yielded an average
improvement of $+5.6\%$ on mBERT and $+4.7\%$ on XLM-R in F1 scores over their
strong baselines.111Datasets and implementation available at
https://github.com/jitinkrishnan/Transliteration-Hindi-Malayalam

##  1 Introduction

A significant number of native Hindi or Malayalam speakers use Latin script
instead of Devanagari or other Brahmic scripts for a wide range of social
media interactions such as posting tweets, updating Facebook status,
commenting on YouTube videos, and writing reviews for restaurants/movies. This
behavior occurs because keyboard optimizations focus primarily on English Bi
et al. (2012) and languages such as Hindi or Malayalam can be very time
consuming to type on small devices. If users prefer the original script, the
current solution is to back-transliterate the romanized text to the original
language ClusterDev (2020). Examples of this transliteration process for Hindi
and Malayalam are shown in Figure 2. In the first row, English and Hindi are
translations of each other, while Latin-transliterated (or romanized) Hindi is
phonetically identical to the Devanagari version but written using Latin
characters.

In this context, we explore state-of-the-art language models such as
multilingual BERT (Devlin et al., 2019, mBERT) and XLM-R Conneau et al. (2020)
to improve their multilingual generalizability through inclusion of romanized
Hindi and Malayalam, as shown in Figure 1. Previous work Pires et al. (2019)
has shown that existing transformer-based representations may exhibit
systematic deficiencies for certain language pairs. This deficiency also
appears in transliterated sentences, as shown in the left panels of Figure 5
(in Section §5), where the t-SNE plot of 3-way parallel datasets consisting of
sentences in source (English), target (Malayalam/Hindi), and their
romanizations are clearly separated in their own clusters, even though they
match semantically. We also provide a t-SNE plot of mBERT/XLM-R
representations showing this deficiency across various transformer layers in
Appendix A. Our work aims to provide a general solution towards alleviating
this issue, by designing a generic and extensible architecture that can be
used for aligning cross-lingual sentence representations.

Figure 1: Overview of our method to enhance the multilinguality of transformer
models such as mBERT or XLM-R to include Latin-transliterations
(romanizations) for two Indic languages: Hindi and Malayalam. Figure 2:
Examples of English sentences and corresponding translations and
transliterations.

Our problem setting is related to language alignment works where static Smith
et al. (2017); Conneau et al. (2017) or contextual Aldarmaki and Diab (2019);
Schuster et al. (2019); Cao et al. (2019) word representations from different
languages are aligned to a shared vector space. Such methods primarily use a
parallel word corpus and either design an alignment loss or explicitly perform
rotations (transformations) on the representations. Our work deviates from
these methods in three aspects: (a) we focus on sentence-level representation
(in this case as produced by the [CLS] classification BERT token; (b) we
create a synthetic 3-way parallel corpus out of the source data using machine
translations and transliterations, and (c) by using a Teacher-Student training
scheme, the final representations of the 3 language variants (source, target,
and romanized target) are aligned to the same source language space as
produced by the original pre-trained model.

Our Teacher-Student model is inspired from knowledge distillation methods
Hinton et al. (2015) that are intended to transfer knowledge from a complex
model (Teacher) to a simpler model (Student). This approach has been utilized
for various tasks, e.g. for reducing the dimension of word embeddings Shin et
al. (2019), distilling BERT for text generation Chen et al. (2019), self-
knowledge distillation Hahn and Choi (2019), or contrastive learning Chen et
al. (2020); Fang et al. (2020). We take a similar approach where the Teacher
model acts as an anchor by freezing all its layers, while the Student model is
fine-tuned based on our optimization procedure to align sentences in English
with their target translations and transliterations. Such an approach is
necessary, as opposed to other methods like pre-training or self-supervised
learning which are outside the scope of this work, because large unlabeled
transliterated datasets are not available and collecting them is non-trivial.
Thus, we focus on _fine-tuning_ and aligning representations of already pre-
trained models.

The practical implications of our work are demonstrated by applying it on
naturally-occurring transliterated datasets of tweets posted during the North
India and Kerala flood crises. A model that can immediately handle
transliterated tweets by producing embeddings in the same space as that of
English tweets can immensely benefit information systems for emergency
services, by utilizing the vast amount of crisis response models trained in
English Lewis et al. (2011); (2020) (Unocha); Nguyen et al. (2017); Krishnan
et al. (2020).

#### Contributions:

a) We propose a novel Teacher-Student method to address the alignment problem
for contextual representations of transliterated text produced by multilingual
language models (and show its efficacy on Hindi and Malayalam). b) We release
two newly labeled datasets; a binary sentiment dataset of Malayalam movie
reviews and a binary relevancy dataset of tweets posted during North India and
Kerala flood crises.

##  2 Methodology

We first describe the problem of cross-lingual transfer for transliterations,
followed by our Teacher-Student model.

###  2.1 Problem Definition

Given a source ($S$) dataset in language $\sigma$, e.g. in English (en), the
goal is to train a classifier such that it can be used to predict examples
from a target ($T$) dataset that consists of transliterations of the target
language ($\tau$). To tackle the lack of training data in the transliterated
target, as well as the lack of representation alignment between the source
sentences $X^{\sigma}$ and the transliterated target space $X^{\tau}$, we
propose data augmentation. Specifically, we first create translations
$X^{\operatorname{tr}(\sigma,\tau)}$ of the source language sentences in the
target language. Then, we also create transliterations of those translated
sentences into the source language’s script:
$X^{\operatorname{tl}(\operatorname{tr}(\sigma,\tau),\sigma)}$.222$\operatorname{tr}(a,b)$
represents translation from language $a$ to $b$, and $\operatorname{tl}(a,b)$
represents transliteration from language $a$ to $b$. All of these are matched
with the correct labels $y^{\sigma}$ from the original dataset. Briefly
outlined:

  * Input: $\mathcal{S}=X^{\sigma}$, $y^{\sigma}$

  * Augmented Input:
$\mathcal{S^{\prime}}=X^{\sigma}$, $X^{\operatorname{tr}(\sigma,\tau)}$,
$X^{\operatorname{tl}(\operatorname{tr}(\sigma,\tau),\sigma)}$, $y^{\sigma}$

  * Goal: $\mathcal{T}=y^{\tau}\leftarrow\operatorname{predict}(X^{\tau})$.

For example, $X^{\operatorname{tl}(\operatorname{tr}(en,hi),en)}$ represents
the data that are the result of first translating from English to Hindi, then
romanizing the result. The augmentation process can be performed using any
existing machine translation and transliteration tool. An example of this
process is shown in Figure 2. In the first row, second column is the result of
$\operatorname{tr}(\text{en,hi})$ and the third column is the result of
$\operatorname{tl}(\operatorname{tr}(\text{en,hi}),\text{en})$.

###  2.2 Teacher-Student Model & Joint Training

Figure 3: Teacher-Student Model & Joint Training. Figure 4: Data samples from
new datasets. Malayalam and Transliterated-Malayalam movie sentiment data
samples (Top). North India and Kerala floods data samples (Bottom).

The base component of our proposed model is straightforward: it obtains
sentence representations from a pre-trained language model (mBERT, XLM-R, or
similar) and uses the [CLS] token to classify the utterance. Our Teacher-
Student method uses two such models, jointly trained, as outlined in Figure 3.
In this setup, the Teacher model acts as an anchor that does not change, i.e.
all its multi-head attention layers are frozen. The goal is that the
representations produced from training the Student model for the translated
and transliterated data will eventually align with the original (Teacher)
model’s source language pre-trained representations.

Training consists of two tasks, an unsupervised alignment task and a
classification task. The goal of the alignment task is to ensure that the
three variants (source, target, and transliterated target) end up with similar
representations. As this only requires a 3-way parallel corpus with no labels,
it is trained in an unsupervised fashion. The goal of the classification task
is to train the model for the given class labels. Joint training on both tasks
is necessary for tackling our problem.

Since the two tasks we are interested in only require the classification token
for making a prediction, we build our loss functions on top of the [CLS]
token. We treat its representation as the encoder’s output ($h$), which is
directly used to compute the unsupervised alignment loss and passed through
linear layers to compute the classification output
($p=\operatorname{softmax}(Wh+b)$). As shown in Figure 3, $h_{\sigma}$,
$h_{\operatorname{tr}}$, and $h_{\operatorname{tl}}$ denote the
representations of $X^{\sigma}$, $X^{\operatorname{tr}(\sigma,\tau)}$, and
$X^{\operatorname{tl}(\operatorname{tr}(\sigma,\tau),\sigma)}$ respectively.

The unsupervised alignment loss consists of three components:
$\mathcal{L}_{ts}$, $\mathcal{L}_{tr}$, and $\mathcal{L}_{tl}$. The Teacher-
Student Loss ($\mathcal{L}_{ts}$) is defined as the difference between output
embeddings produced using source language ($X^{\sigma}$) by the Student ($s$)
versus the Teacher ($t$). The Translation Loss ($\mathcal{L}_{tr}$) is defined
as the difference between output embeddings produced using
$X^{\operatorname{tr}(\sigma,\tau)}$ on the Student versus $X^{\sigma}$ on the
Teacher. Similarly, the Transliteration Loss ($\mathcal{L}_{tl}$) is defined
as the difference between output embeddings produced using
$X^{\operatorname{tl}(\operatorname{tr}(\sigma,\tau),\sigma)}$ on the Student
versus $X^{\sigma}$ on the Teacher.

All distances are measured using the cosine similarity between the two
vectors.333We use $d(a,b)=1-\frac{a\cdot b}{|a||b|}$. Essentially, all three
losses penalize moving away from the Teacher’s representation of the source
language:

$\displaystyle\mathcal{L}_{ts}=\frac{1}{N}\sum$ $\displaystyle
d(h_{\sigma}^{t},h_{\sigma}^{s}),\hskip
5.69046pt\mathcal{L}_{tr}=\frac{1}{N}\sum d(h_{\sigma}^{t},h_{tr}^{s})$ (1)
$\displaystyle\mathcal{L}_{tl}=\frac{1}{N}\sum d(h_{\sigma}^{t},h_{tl}^{s}),$

where $N$ represents the number of samples. Thus, the final unsupervised
alignment loss ($\mathcal{L}_{u}$) defined over the Teacher-Student model can
be defined as the weighted sum of the three losses:

$\mathcal{L}_{u}=\beta_{1}\mathcal{L}_{ts}+\beta_{2}\mathcal{L}_{tr}+\beta_{3}\mathcal{L}_{tl}$
(2)

Meanwhile, the loss function ($\mathcal{J}_{joint}$) for the sentiment task is
a sum of binary cross-entropy
(BCE444$BCE=-\frac{1}{N}\sum_{i=1}^{N}y_{i}\log\hat{y_{i}}+(1-y_{i})\log(1-\hat{y_{i}})$)
losses, defined similarly for the three variants of parallel data (Source
($\sigma$), Translated ($tr$), and Transliterated ($tl$)):

$\mathcal{J}_{joint}=\sum_{k\in[\sigma,tr,tl]}BCE_{k}\vspace{-.3em}$ (3)

The overall loss is simply the combination of the unsupervised and the
classification loss.

$\mathcal{L}_{joint\\_ts}=\mathcal{J}_{joint}+\alpha\mathcal{L}_{u}\vspace{-.3em}$
(4)

where $\alpha$ is the hyperparameter that controls the unsupervised alignment
loss. To summarize, the Teacher-Student model takes the augmented data
($\mathcal{S^{\prime}}$) as the input and optimizes over a joint loss function
that comprises of an unsupervised component that aligns the translated and
transliterated representation into the same vector space as the source
language and a supervised component that learns to classify for the task at
hand.

Dataset | $+$ | $-$ | Avg. # of words | Avg. # of chars. | Avg. # of chars.
---|---|---|---|---|---
per sentence | per word | per sentence
Movie Review Datasets - Sentiment
Hindi Movie Reviews | 335 | 293 | 154.5 | 4.0 | 613.8
Romanized Hindi ($\operatorname{\text{hi}}_{\operatorname{ro}})$ | 335 | 293 | 154.5 | 5.0 | 775.8
Malayalam Movie Reviews | 501 | 451 | 10.4 | 9.3 | 108.2
Romanized Malayalam ($\operatorname{\text{ml}}_{\operatorname{ro}})$ | 501 | 451 | 10.4 | 10.8 | 122.6
Crisis Tweet Datasets - Relevancy
North India Floods ($\operatorname{\text{hi}}_{nf}$) | 206 | 250 | 20.2 | 3.9 | 78.1
Kerala Floods ($\operatorname{\text{ml}}_{kf}$) | 109 | 132 | 19.1 | 5.4 | 103.6
Table 1: Statistics for the test datasets.

##  3 Datasets

In this section, we outline the datasets we use in our experiments.

###  3.1 English Movie Reviews

English movie reviews are sampled from the large IMDb movie review dataset
Maas et al. (2011) with randomly sampled balanced counts of $5000$ positive
and $5000$ negative reviews for training and $500$ each for validation. During
training, this dataset is translated to Hindi and Malayalam, and subsequently
transliterated. Ground truth Hindi and Malayalam datasets (described in the
following sections), are used only for evaluation. The ‘$\operatorname{ro}$’
notation used in the following section for representing datasets signify
$\operatorname{tl}(\bullet,\text{en})$; i.e. romanization or Latin-
transliteration using transliteration tools.

###  3.2 Hindi Movie Reviews ($\operatorname{\text{hi}}$,
$\operatorname{\text{hi}}_{\operatorname{ro}}$)

The original Hindi movie review
dataset555https://www.kaggle.com/disisbig/hindi-movie-reviews-dataset is
constructed from various News Websites. This dataset consists of positive,
negative, and neutral movie reviews. We select only positive and negative
reviews from both training and validation datasets to construct the test
dataset for our cross-lingual setup. From this, we construct the romanized
Hindi dataset using the Indic-nlp transliteration tool Kunchukuttan
(2020).666https://pypi.org/project/indic-transliteration/

###  3.3 Malayalam Movie Reviews ($\operatorname{\text{ml}}$,
$\operatorname{\text{ml}}_{\operatorname{ro}}$) - New Dataset I

For Malayalam movie reviews, we construct a new human-labeled dataset from the
Samayam News website.777https://malayalam.samayam.com/malayalam-cinema/movie-
review/articlelist/48225004.cms Reviews are lengthy, in general, with plenty
of neutral text. So, a native Malayalam speaker was tasked with extracting a
few sentences from each movie review such that each positive or negative
example in our dataset is highly polar. From this, we construct the romanized
Malayalam dataset using the ml2en tool.888https://pypi.org/project/ml2en/ A
few samples from the dataset are shown in Figure 4 and dataset statistics are
available in Table 1.

###  3.4 Crisis Tweets ($\operatorname{\text{en}}$)

Appen999https://appen.com/datasets/combined-disaster-response-data/ provides a
labeled collection of tweets posted during various natural disasters such as
earthquakes, floods, and hurricanes. The three most common languages in this
dataset are English, Spanish, and Haitian Creole. For our experiments, we
focus on the related label column signifying the relevancy of tweets. The
dataset also contains English translations of non-English tweets. Training and
validation datasets are constructed using only the English tweets (message
column in the Appen dataset). Statistics are shown in Table 1.

###  3.5 Crisis Tweets: North India ($\operatorname{\text{hi}}_{nf}$) and
Kerala ($\operatorname{\text{ml}}_{kf}$) Floods - New Dataset II

To construct our test datasets for the crisis tweet classification
experiments, we collected tweets from the 2013 North India and 2018 Kerala
Floods using Twitter API. We filtered these tweets to restrict only to
naturally-occurring transliterated Hindi and Malayalam sentences, using a set
of transliterated crisis-related keywords such as madad, toofan, baarish,
sahayta, floods, etc., for Hindi and pralayam, vellapokkam, vellam, sahayam,
durantham, veedukal, etc., for Malayalam. With the help of a native language
expert for each language who are also proficient in English, the tweets are
labeled based on contextual-relevancy or relatedness; similar to the related
label for the English tweets in the Appen dataset. Dataset statistics are
shown in Table 1 and a few tweet samples are shown in Figure 4.

##  4 Experimental Setup

Since we are interested in cross-lingual transfer, we only use the English
datasets (IMDb reviews for Sentiment Analysis and Appen Crisis Dataset for
Tweet Classification) for training, augmented with their automatic
translations and transliterations. We evaluate on all other datasets (c.f.
Table 1).

#### Monolingual LM Baselines.

Our first baseline uses pre-trained language models from Hugging Face Wolf et
al. (2020) that are monolingually trained on
Hindi101010https://huggingface.co/monsoon-nlp/hindi-tpu-electra and
Malayalam.111111https://huggingface.co/eliasedwin7/MalayalamBERT

#### mBERT/XLM-R Baselines.

We also consider baselines using multilingual masked LMs (MLM), specifically
mBERT Devlin et al. (2019) and XLM-R Conneau et al. (2020). We compare our
models with the following MLM baselines: 1) a model trained using only in
English, 2) a model trained using English translated to the target language
using MarianMT Junczys-Dowmunt et al. (2018), 3) a transliterated model which
is trained using the target-transliterated version of target-translated
English data, and 4) a combination of the three. These baselines use our
augmented datasets but do not use the Teacher-Student training scheme, e.g.
$\text{mBERT}_{tl}$ refers to mBERT trained using target-transliterations of
target-translated English data.

Test Data $\rightarrow$ | $\mathbf{hi_{ro}}$ | $\mathbf{ml_{ro}}$ | $\mathbf{hi_{nf}}$ | $\mathbf{ml_{kf}}$ | AVG
---|---|---|---|---|---
Models $\downarrow$ | Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1
Monolingual LM♢ | 50.38 | 37.99 | 48.76 | 47.55 | 54.39 | 49.06 | 63.35 | 63.33 | 54.22 | 49.48
mBERT Baselines |  |
mBERTen | 47.55 | 41.23 | 48.23 | 41.63 | 56.67 | 55.27 | 57.84 | 57.72 | 52.57 | 48.96
mBERTtr | 51.81 | 48.21 | 51.32 | 45.62 | 52.89 | 47.89 | 58.26 | 57.54 | 53.57 | 49.82
mBERTtl | 55.29 | 54.18 | 61.72 | 56.47 | 56.14 | 55.78 | 58.09 | 57.79 | 57.81 | 56.06
mBERTen+tr+tl | 55.16 | 54.75 | 61.72 | 61.25 | 56.71 | 56.03 | 63.74 | 63.42 | 59.33 | 58.86
Our mBERT models |  |
mBERT-Joint | 55.57 | 55.56 | 64.29 | 63.58 | 57.75 | 56.73 | 51.85 | 53.98 | 57.37 | 57.46
mBERT-Joint-TS | 57.37♠ | 57.36♠ | 65.15♠ | 65.78♠ | 63.22♠ | 63.14♠ | 62.55 | 62.40 | 62.07 | 62.17
XLM-R Baselines |  |
XLM-Ren | 50.57 | 45.76 | 50.86 | 47.13 | 58.11 | 56.77 | 61.74 | 60.98 | 55.32 | 52.66
XLM-Rtr | 49.52 | 47.67 | 51.72 | 50.16 | 57.11 | 56.95 | 61.74 | 61.72 | 55.02 | 54.13
XLM-Rtl | 54.81 | 53.72 | 51.67 | 54.71 | 51.45 | 51.24 | 59.84 | 59.23 | 54.44 | 54.73
XLM-Ren+tr+tl | 55.57 | 54.19 | 62.46 | 61.52 | 56.40 | 55.67 | 63.24 | 63.10 | 59.42 | 58.62
Our XLM-R Models |  |
XLM-R-Joint | 56.09 | 55.40 | 62.90 | 63.14 | 53.68 | 52.81 | 62.79 | 62.77 | 58.87 | 58.53
XLM-R-Joint-TS | 57.70♠ | 57.03♠ | 65.93♠ | 65.71♠ | 58.39 | 57.86 | 64.87♠ | 64.87♠ | 61.72 | 61.37
Table 2: Performance evaluation (weighted F1) on various romanized datasets shows that our Teacher-Student model outperforms the baselines. $\diamondsuit$: Dedicated Hindi and Malayalam language models (LM) from Hugging Face; results reflect the best performing model by varying the training data. ♠: The difference is significant with p $<$ 0.05 using Tukey HSD (compared against the best baseline model). Train Language $\rightarrow$ | $\mathbf{en}$ | $\mathbf{hi}$ | $\mathbf{ml}^{\dagger}$
---|---|---|---
Model | (Baseline) | (Joint-TS Model)
mBERT | 81.06 | 82.35 | 71.25
XLM-R | 83.47 | 84.00 | 74.16
Table 3: Evaluation (weighted F1) on IMDb English test data showing that our model preserves the performance on the source language for Hindi but not for Malayalam. The results in $\operatorname{\text{hi}}$ and $\operatorname{\text{ml}}$ are with our modifications. $\dagger$: See discussion. Test Language $\rightarrow$ | $\mathbf{hi}$ | $\mathbf{ml}$
---|---|---
mBERT${}_{\operatorname{\text{en}}}$ | 54.34 | 54.07
mBERT${}_{\operatorname{tr}}$ | 60.11 | 66.20
mBERT-Joint-TS | 61.35 | 66.24
XLM-R${}_{\operatorname{\text{en}}}$ | 60.82 | 78.11
XLM-R${}_{\operatorname{tr}}$ | 59.72 | 71.40
XLM-R-Joint-TS | 62.23 | 76.46†
Table 4: Evaluation (weighted F1) on Hindi and Malayalam original script
showing that our model preserves or outperforms their performance on the
baselines except for Malayalam on XLM-R. $\dagger$: See discussion.

#### Our Proposed Models.

Joint-TS represents our Teacher-Student model shown in Figure 3 and Eq. 4. We
also perform an ablation (the Joint model) without the Teacher model (setting
$\alpha=0$ in Eq. 4), i.e. this model does not have an anchored Teacher, which
means that there is no penalty for representations of parallel sentences being
different.

#### Implementation Details.

Our implementation is in PyTorch Paszke et al. (2019) with the transformers
library Wolf et al. (2020). We use the pre-trained cased multilingual BERT and
the pre-trained xlm-roberta-base with a standard sequence classification
architecture. Maximum epoch is set to $40$ with an early stopping patience of
$10$, batch size to $32$, and we use the Adam optimizer Kingma and Ba (2014)
with a learning rate of $5\mathrm{e}{-5}$. We select the best model based on
the validation set, using both accuracy and weighted F1 as performance
measures.

##  5 Results & Discussion

#### Performance on Transliterated Target.

Table 2 shows the classification performance121212Model runtime shown in
Appendix B. on transliterated datasets. Averaging the scores, we see a total
boost of $+5.6\%$ on mBERT and $+4.7\%$ on XLM-R in weighted F1 performance
across the 4 romanized datasets.131313With similar trends on accuracy as well.
Our top baseline models in each of the masked language models are
mBERTen+tr+tl and XLM-Ren+tr+tl, which combine the augmented data with the
original data to fine-tune the model for the classification task. The
improvement produced by our model is due to the fact that the pre-trained
models have likely not seen that many transliterations in these languages.
mBERT is trained using data from Wikipedia, while XLM-R uses Common Crawl. As
such, XLM-R likely has been trained on at least some code-switched or
transliterated data. This is also supported by the scores in Tables 2, 3, and
4, where XLM-Ren outperforms mBERTen on all datasets, i.e. XLM-R’s English
representations are much more generalizable than mBERT’s for these two
languages.

Figure 5: The visualization of final layer representations (from the movie
review test data) shows the effectiveness of unsupervised alignment training
in bringing the representations of the 3 variants in a shared space.

#### Performance on Original-Script Target.

Beyond improving on transliterated text, we need our model to handle original,
not-transliterated text equally well. Evaluating on the original script (Table
4) we find that our models do preserve or even outperform the baselines
trained only with English or non-romanized text, especially in Hindi. The
exception is Malayalam in XLM-R, which could be attributed to the power of
XLM-R pre-training in producing a more generalizable English representation
than the original script ($\text{XLM-R}_{\operatorname{\text{en}}}$ vs.
$\text{XLM-R}_{tr}$).

#### Performance on English.

Ideally, our Teacher-Student model will preserve the performance on the source
language (in this case English). Table 3 shows the performance evaluation on
English data ($1000$ randomly selected positive and negative reviews from the
IMDb dataset Maas et al. (2011)). We observe that fine-tuning on Hindi
preserves and slightly improves performance in English, but Malayalam does not
($\dagger$ in Table 3). We speculate that, for Malayalam, this is because the
transliterated embeddings (red) are not blended in well with the others
compared to those for Hindi as shown in Figure 5 (right panels).

#### Representation Alignment.

Figure 5 shows a visual reason behind our model’s aforementioned performance
improvements. While the t-SNE projection of the (3-way parallel) sentence
representations for the original model are quite distinct based on
language/script (left), our model brings all representations in the same
vector space (right). This method of unsupervised training ($L_{u}$ in Eq. 2)
is very generic in nature that it can be adapted for any alignment scenarios
where different variants or dialects of the same language need to be aligned.

#### Ablation Studies.

We conduct $8$ ablations in total. First four mBERT/XLM-R models in Table 2
are ablations that show how the three language variants and their combination
perform. The joint model without Teacher-student in Table 2 represents the
version without the alignment training. Table 3 shows performance on English
and Table 4 shows performance on original script. The impact of unsupervised
alignment is shown in Figure 5 showing where the boost is coming from.
Ablation of the Joint-TS model without $tr$ is not shown as our goal is to
construct a single model that preserves performance across the three variants.

Figure 6: Freezing up to layers $8$ to $10$ of mBERT showed to be optimal for
our task.

#### Unsupervised Domain Adaptation.

In addition to being cross-lingual, our approach also falls under the paradigm
of unsupervised domain adaptation. Our sources and targets do not strictly
fall in the same domain, even though the classification tasks are similar. For
example, the training data for classifying tweets consists of not only floods,
but also other events such as hurricanes, fires, earthquakes, etc. This leads
to another application of our method, which can be deployed at the onset of
crisis events in any region, with minimal requirement to collect labeled data
from the new crisis, or converting the data to native script or English, which
might save precious time for crisis response.

#### Hyperparameter Tuning.

Primarily, we tune two hyperparameters: a) $\alpha$ \- the weight that is
given to unsupervised alignment loss in Eq. 4 and b) number of layers to
freeze to identify the appropriate amount of pre-trained information to be
preserved without alteration. $\alpha$ values are tuned using a simple grid
search from a range of [$0.01$-$1.0$]. All $\beta$ values (Eq. 2) are set to
$1$ to prioritize the three variants (source, target, transliterated target)
equally. For Joint-TS (Eq. 4), best hyperparameters for mBERT are
$\alpha_{\operatorname{\text{hi}}_{\operatorname{ro}}}$= $0.3$ and best
$\alpha_{\operatorname{\text{ml}}_{\operatorname{ro}}}$= $0.05$ and for XLM-R
are $\alpha_{\operatorname{\text{hi}}_{\operatorname{ro}}}$= $0.5$ and best
$\alpha_{\operatorname{\text{ml}}_{\operatorname{ro}}}$= $0.01$. Intuitively,
we find that giving the classification loss primary focus and the unsupervised
alignment loss secondary focus produced better results. On the other hand, we
found that freezing bottom layers leads to better results, as shown in Figure
6. The optimal amount of layers to freeze was empirically between $8$ to $10$.

##  6 Related Work

Sentiment analysis in Hindi spans a variety of tasks such as analysis of movie
reviews Nanda et al. (2018), building subjective lexica for product reviews
and blogs Arora (2013), analysis on tweets Sharma et al. (2015), aspect-based
sentiment Akhtar et al. (2016), predicting elections Sharma and Moh (2016),
and analysis on code-mixed text Joshi et al. (2016). Code-mixing is another
related phenomenon where multilingual speakers alternate between languages,
often in the same sentence. Both code-mixing and transliteration are studied
for Hindi and Marathi texts using supervised learning methods by Ansari and
Govilkar (2018). We restrict our analysis to transliteration, although our
dataset may contain code-mixed text. Recently, KhudaBukhsh et al. have
proposed a pipeline to sample code-mixed documents using minimal supervision.
In cross-lingual context, researchers have used linked WordNets Balamurali et
al. (2012) and cross-lingual word embeddings Singh and Lefever (2020) using
MUSE Conneau et al. (2017) and VecMap Artetxe et al. (2018) to bridge the
language gap, later addressing code-mixing and transliteration. With the
advent of large pre-trained language models, we take a step further in this
direction to enhance mBERT/XLM-R to cover transliterations for fine-tuning it
to the downstream task of sentiment analysis.

Malayalam has also seen several works on sentiment analysis Nair et al.
(2015); Kumar et al. (2017); Nair et al. (2014); Ashna and Sunny (2017).
Recently, a new Malayalam-English code-mixed corpus Chakravarthi et al. (2020)
has been constructed by scraping YouTube comments. This corpus primarily
consists of romanized sentences with some code-mixing. After converting this
dataset into its original script to obtain the parallel corpus, this can also
be used as an additional dataset for our model evaluation.

Translate and train has been a popular methodology Shah et al. (2010);
Yarowsky et al. (2001); Ni et al. (2017); Xu et al. (2020) that utilizes the
power of existing Machine Translation tools Wu et al. (2016); Junczys-Dowmunt
et al. (2018) to perform cross-lingual tasks by augmenting the original source
dataset with the target-translated data before training. This kind of training
could enhance the performance of multilingual representations by fine-tuning
the pre-trained models such as mBERT or XLM-R, creating a pseudo-supervised
environment where the model now has access to data in the target language. We
follow the same approach to create strong baselines as well as for the
Teacher-Student model.

##  7 Future Work

We acknowledge that existing Hindi and Malayalam translation and
transliteration tools pose limitations and may cause cascading errors. For
example, the Indic transliteration tool adds an extra, unnecessary ‘a’ for
some Hindi words (eg., ‘sandarbh’ vs ‘sandarbha’) which may not reflect how
native users tend to write. Despite its limitations, using it is still
beneficial to construct a strong baseline. We expect that improved
transliteration systems would further improve downstream accuracy. This is
also the case for the many Indic languages where extensive datasets are not
available. We plan to expand our model to other Indic languages such that
their translations and transliterations are aligned within the language models
such as mBERT/XLM-R. Another future direction is to perform an in-depth
sensitivity analysis over the $\beta_{2}$ and $\beta_{3}$ parameters that tune
the unsupervised alignment loss ($L_{u}$) in Eq. 2, which might also address
some of the exceptions shown in Tables 3 and 4, to ensure that the model
preserves performance on English or non-romanized target data.

##  8 Conclusion

We propose a Teacher-Student model to enhance the multilinguality of language
models such as mBERT/XLM-R so that it can be adapted to perform cross-lingual
text classification tasks for transliterated Hindi and Malayalam. Experiments
show that our model outperforms traditional fine-tuning and other baselines
built on the state-of-the-art. Furthermore, we release two human-annotated
datasets: a highly polar Malayalam movie review dataset for sentiment analysis
and a dataset of Hindi and Malayalam romanized tweets posted during North
India and Kerala floods. Additionally, our method presents a generic and
extensible architecture that could be adapted to any language alignment
scenarios where large pre-trained multilingual models may fall short.

##  9 Acknowledgement

We thank U.S. National Science Foundation grants IIS-1815459 and IIS-1657379
for partially supporting this research. We thank Ming Sun, Alexis Conneau,
Sneha Mehta, and Raj Patel for giving valuable insights on language models,
machine translation, and multilingual model training. We thank Chandini
Narayan and Sujay Das for data annotation. We also acknowledge ARGO team as
the experiments were run on ARGO, a research computing cluster provided by the
Office of Research Computing at George Mason University.

## References

  * Akhtar et al. (2016) Md Shad Akhtar, Asif Ekbal, and Pushpak Bhattacharyya. 2016.  Aspect based sentiment analysis in hindi: resource creation and evaluation.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 2703–2709.
  * Aldarmaki and Diab (2019) Hanan Aldarmaki and Mona Diab. 2019.  Context-aware cross-lingual mapping.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3906–3911.
  * Ansari and Govilkar (2018) Mohammed Arshad Ansari and Sharvari Govilkar. 2018.  Sentiment analysis of mixed code for the transliterated hindi and marathi texts.  _International Journal on Natural Language Computing (IJNLC) Vol_ , 7.
  * Arora (2013) Piyush Arora. 2013.  _Sentiment Analysis For Hindi Language_.  Ph.D. thesis, International Institute of Information Technology Hyderabad.
  * Artetxe et al. (2018) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018.  A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 789–798.
  * Ashna and Sunny (2017) MP Ashna and Ancy K Sunny. 2017.  Lexicon based sentiment analysis system for malayalam language.  In _2017 International Conference on Computing Methodologies and Communication (ICCMC)_ , pages 777–783. IEEE.
  * Balamurali et al. (2012) AR Balamurali, Aditya Joshi, and Pushpak Bhattacharyya. 2012.  Cross-lingual sentiment analysis for indian languages using linked wordnets.  In _Proceedings of COLING 2012: Posters_ , pages 73–82.
  * Bi et al. (2012) Xiaojun Bi, Barton A Smith, and Shumin Zhai. 2012.  Multilingual touchscreen keyboard design and optimization.  _Human–Computer Interaction_ , 27(4):352–382.
  * Cao et al. (2019) Steven Cao, Nikita Kitaev, and Dan Klein. 2019.  Multilingual alignment of contextual word representations.  In _International Conference on Learning Representations_.
  * Chakravarthi et al. (2020) Bharathi Raja Chakravarthi, Navya Jose, Shardul Suryawanshi, Elizabeth Sherly, and John P McCrae. 2020.  A sentiment analysis dataset for code-mixed malayalam-english.  _arXiv preprint arXiv:2006.00210_.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR.
  * Chen et al. (2019) Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu. 2019.  Distilling the knowledge of bert for text generation.  _arXiv preprint arXiv:1911.03829_.
  * ClusterDev (2020) ClusterDev. 2020.  Malayalam keyboard.  _clusterdev.com_.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451.
  * Conneau et al. (2017) Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017.  Word translation without parallel data.  _arXiv preprint arXiv:1710.04087_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Fang et al. (2020) Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. 2020.  Cert: Contrastive self-supervised learning for language understanding.  _arXiv preprint arXiv:2005.12766_.
  * Hahn and Choi (2019) Sangchul Hahn and Heeyoul Choi. 2019.  Self-knowledge distillation in natural language processing.  In _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)_ , pages 423–430.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_.
  * Joshi et al. (2016) Aditya Joshi, Ameya Prabhu, Manish Shrivastava, and Vasudeva Varma. 2016.  Towards sub-word level compositions for sentiment analysis of hindi-english code mixed text.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 2482–2491.
  * Junczys-Dowmunt et al. (2018) Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018.  Marian: Fast neural machine translation in C++.  In _Proceedings of ACL 2018, System Demonstrations_ , pages 116–121, Melbourne, Australia. Association for Computational Linguistics.
  * KhudaBukhsh et al. (2020) Ashiqur R KhudaBukhsh, Shriphani Palakodety, and Jaime G Carbonell. 2020.  Harnessing code switching to transcend the linguistic barrier.  _arXiv preprint arXiv:2001.11258_.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Krishnan et al. (2020) Jitin Krishnan, Hemant Purohit, and Huzefa Rangwala. 2020.  Unsupervised and interpretable domain adaptation to rapidly filter social web data for emergency services.  In _ASONAM_.
  * Kumar et al. (2017) S Sachin Kumar, M Anand Kumar, and KP Soman. 2017.  Sentiment analysis of tweets in malayalam using long short-term memory units and convolutional neural nets.  In _International Conference on Mining Intelligence and Knowledge Exploration_ , pages 320–334. Springer.
  * Kunchukuttan (2020) Anoop Kunchukuttan. 2020.  The IndicNLP Library.  https://github.com/anoopkunchukuttan/indic_nlp_library/blob/master/docs/indicnlp.pdf.
  * Lewis et al. (2011) William Lewis, Robert Munro, and Stephan Vogel. 2011.  Crisis mt: Developing a cookbook for mt in crisis situations.  In _Proceedings of the Sixth Workshop on Statistical Machine Translation_ , pages 501–511.
  * Maas et al. (2011) Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies_ , pages 142–150.
  * Nair et al. (2015) Deepu S Nair, Jisha P Jayan, RR Rajeev, and Elizabeth Sherly. 2015.  Sentiment analysis of malayalam film review using machine learning techniques.  In _2015 international conference on advances in computing, communications and informatics (ICACCI)_ , pages 2381–2384. IEEE.
  * Nair et al. (2014) Deepu S Nair, Jisha P Jayan, Elizabeth Sherly, et al. 2014.  Sentima-sentiment extraction for malayalam.  In _2014 International Conference on Advances in Computing, Communications and Informatics (ICACCI)_ , pages 1719–1723. IEEE.
  * Nanda et al. (2018) Charu Nanda, Mohit Dua, and Garima Nanda. 2018.  Sentiment analysis of movie reviews in hindi language using machine learning.  In _2018 International Conference on Communication and Signal Processing (ICCSP)_ , pages 1069–1072. IEEE.
  * Nguyen et al. (2017) Dat Tien Nguyen, Kamla Al-Mannai, Shafiq R Joty, Hassan Sajjad, Muhammad Imran, and Prasenjit Mitra. 2017.  Robust classification of crisis-related data on social networks using convolutional neural networks.  _ICWSM_ , 31(3):632–635.
  * Ni et al. (2017) Jian Ni, Georgiana Dinu, and Radu Florian. 2017.  Weakly supervised cross-lingual named entity recognition via effective annotation and representation projection.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1470–1480.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’ Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc.
  * Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019.  How multilingual is multilingual bert?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4996–5001.
  * Schuster et al. (2019) Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019.  Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1599–1613.
  * Shah et al. (2010) Rushin Shah, Bo Lin, Anatole Gershman, and Robert Frederking. 2010.  Synergy: a named entity recognition system for resource-scarce languages such as swahili using online machine translation.  In _Proceedings of the Second Workshop on African Language Technology (AfLaT 2010)_ , pages 21–26.
  * Sharma and Moh (2016) Parul Sharma and Teng-Sheng Moh. 2016.  Prediction of indian election using sentiment analysis on hindi twitter.  In _2016 IEEE international conference on big data (big data)_ , pages 1966–1971. IEEE.
  * Sharma et al. (2015) Yakshi Sharma, Veenu Mangat, and Mandeep Kaur. 2015.  A practical approach to sentiment analysis of hindi tweets.  In _2015 1st International Conference on Next Generation Computing Technologies (NGCT)_ , pages 677–680. IEEE.
  * Shin et al. (2019) Bonggun Shin, Hao Yang, and Jinho D Choi. 2019.  The pupil has become the master: teacher-student model-based word embedding distillation with ensemble learning.  _arXiv preprint arXiv:1906.00095_.
  * Singh and Lefever (2020) Pranaydeep Singh and Els Lefever. 2020.  Sentiment analysis for hinglish code-mixed tweets by means of cross-lingual word embeddings.  In _Proceedings of the The 4th Workshop on Computational Approaches to Code Switching_ , pages 45–51.
  * Smith et al. (2017) Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. 2017.  Offline bilingual word vectors, orthogonal transformations and the inverted softmax.  _arXiv preprint arXiv:1702.03859_.
  * (43) UNITED NATIONS OFFICE FOR THE COORDINATION OF HUMANITARIAN AFFAIRS (Unocha). 2020\.  Global humanitarian response plan covid-19.
  * Wolf et al. (2020) Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_.
  * Xu et al. (2020) Weijia Xu, Batool Haider, and Saab Mansour. 2020.  End-to-end slot alignment and recognition for cross-lingual nlu.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5052–5063.
  * Yarowsky et al. (2001) David Yarowsky, Grace Ngai, and Richard Wicentowski. 2001.  Inducing multilingual text analysis tools via robust projection across aligned corpora.  Technical report, Johns Hopkins Univ Baltimore MD Dept of Computer Science.

##  Appendix A Alignment Deficiency

Continuing the discussion from the introduction (Section 1), Figure 7
describes the representational deficiency exhibited by mBERT and XLM-R on
Hindi and Malayalam. The t-SNE plot of 3-way parallel datasets consisting of
sentences in source (English), target (Malayalam/Hindi), and their
romanizations are clearly clustered in their own groups, even though they
match semantically. Our goal is to address this issue and bring them into a
comparable vector space as shown in Figure 5 using a Teacher-Student training
scheme.

Figure 7: t-SNE plot of embeddings across the multi-head attention layers
shows that the alignment deficiency identified in previous works Pires et al.
(2019) also extends to transliteration. XLM-R on Hindi (top). mBERT on
Malayalam (bottom). English (blue), Original Script (green), and Latin-
Transliteration (red).

##  Appendix B Model Runtime

As an addendum to the performance evaluation shown in Table 2, we also provide
the runtime of our Joint-TS model and key baselines. The
$\text{mBERT}_{\operatorname{\text{en}}}$ and
$\text{XLM-R}_{\operatorname{\text{en}}}$ model are trained only on English
data without any augmentation while
$\text{mBERT}_{\operatorname{\text{en}}+tr+tl}$,
$\text{XLM-R}_{\operatorname{\text{en}}+tr+tl}$, and the Joint-TS models are
trained using translated and transliterated target in addition to English
data. The additional latency is caused due to the augmented data (two times
more data). Our Joint-TS model also consists of unsupervised optimization for
alignment, in addition to the augmented data. An interesting observation is
that the Teacher-Student model based on mBERT converges faster than its
${\operatorname{\text{en}}+tr+tl}$ counterpart and XLM-R, while also having a
comparable performance as shown in Table 2.

Model $\rightarrow$ | mBERTen | XLM-Ren | mBERTen+tr+tl | XLM-Ren+tr+tl | mBERT-Joint-TS | XLM-R-Joint-TS
---|---|---|---|---|---|---
Runtime $\rightarrow$ | 00:35:55 | 00:55:53 | 02:38:39 | 02:40:48 | 01:30:48 | 04:39:16
Table 5: Run time on a single K80 GPU in HH:MM:SS for training a Malayalam
model on Crisis data.

##  Appendix C Ethical Considerations

The tweets extraction procedure followed the Twitter Terms of Service and did
not violate privacy policies of individual users. Also, the datasets we share
include only Tweet IDs in the public domain. Data statement that includes
annotator guidelines for the labeling jobs and other dataset information will
be provided with the implementation. From a broader impact perspective, our
code is open-source and allows NLP technology to be accessible to information
systems for emergency services and social scientists in studying a large
population in India who use transliterated text for communication in everyday
life.