# Model-Agnostic Meta-Learning for Multilingual
Hate Speech Detection

Md Rabiul Awal, Roy Ka-Wei Lee, Eshaan Tanwar, Tanmay Garg, Tanmoy Chakraborty
Md Rabiul Awal was with University of Saskatchewan, Canada (e-mail:
mda219@usask.ca).Roy Ka-Wei Lee was with Singapore University of Technology
and Design, Singapore (e-mail: roy_lee@sutd.edu.sg).Tanmoy Chakraborty was
with IIT Delhi, India (email: tanchak@iitd.ac.in)Tanmay Garg was with IIIT-
Delhi, India (e-mail: tanmay17061@gmail.com).Eshaan Tanwar was with DTU-Delhi,
India (e-mail: eshaantanwar_2k18co125@dtu.ac.in).Tanmoy Chakraborty would like
to thank the support of the Ramanujan Fellowship (SB/S2/RJN-073/2017), funded
by the Science and Engineering Research Board (SERB), India.

###### Abstract

Hate speech in social media is a growing phenomenon, and detecting such toxic
content has recently gained significant traction in the research community.
Existing studies have explored fine-tuning language models (LMs) to perform
hate speech detection, and these solutions have yielded significant
performance. However, most of these studies are limited to detecting hate
speech only in English, neglecting the bulk of hateful content that is
generated in other languages, particularly in low-resource languages.
Developing a classifier that captures hate speech and nuances in a low-
resource language with limited data is extremely challenging. To fill the
research gap, we propose HateMAML, a model-agnostic meta-learning-based
framework that effectively performs hate speech detection in low-resource
languages. HateMAML utilizes a self-supervision strategy to overcome the
limitation of data scarcity and produces better LM initialization for fast
adaptation to an unseen target language (i.e., cross-lingual transfer) or
other hate speech datasets (i.e., domain generalization). Extensive
experiments are conducted on five datasets across eight different low-resource
languages. The results show that HateMAML outperforms the state-of-the-art
baselines by more than 3% in the cross-domain multilingual transfer setting.
We also conduct ablation studies to analyze the characteristics of HateMAML.

###### Index Terms:

Hate speech detection, cross-lingual transfer, meta-learning.

##  I Introduction

Motivation. Online hate speech that expresses hate or encourages violence
towards a person or a group based on characteristics such as race, religion,
or gender is a growing global concern. The hateful content has broken the
cohesiveness of online social communities and resulted in violent hate
crimes111https://thewire.in/law/hate-speech-what-it-is-and-why-it-matters.
Therefore, detecting and moderating hate speech in online social media is a
pressing issue.

The gravity of the situation has motivated social media platforms and academic
researchers to propose traditional machine learning and deep learning
solutions to detect online hate speech automatically [1, 2, 3] and early [4,
5]. Among the solutions, large pre-trained language models (LMs), which have
demonstrated their superiority in many NLP tasks [6], have also shown
excellent performance in the hate speech detection task [7]. However, existing
studies have predominantly focused on detecting hate speech in English and
content from Western cultures. This neglects the large portion of online hate
speech from other languages and cultures. Low-resource languages often have
very limited or no training samples available. This can make it challenging to
develop supervised classifiers. We need models that can efficiently adapt with
few training data or methods that can transfer knowledge from another dataset.
In such a case, a high-resource language can augment the low-resource language
by knowledge transfer. For example, Aluru et al. [8] proposed a method to
fine-tune a multilingual BERT model for hate speech detection in low-resource
languages that leverages cross-lingual transfer from high-resource pre-
training.

There are very few works that deal with multilingual hate speech detection. A
viable approach is to fine-tune pre-trained LMs, which is explored in existing
studies [7, 9, 8]. The underlying intuition is that the large LMs generate
shared embeddings in many languages, enabling cross-lingual transfer from
supervised training in the high-resource languages such as English. This
cross-lingual transfer can be achieved as follows: (i) training only in the
source language and evaluating in target language, known as ‘zero-shot’, or
(ii) training in the target language only or joint training of source and
target languages, known as ‘few-shot’ learning. In multilingual hate speech
detection, English and a few high-resource languages are considered “sources”,
and low-resource languages of interest are considered “targets”. A source
language is usually a magnitude larger than a target language. However, the
fine-tuning of pre-trained LMs also has several limitations [10]. An obvious
issue is the data scarcity in target low-resource languages; it is challenging
to capture hate speech and its nuances with insufficient observations. The
performance of cross-lingual transfer learning may adversely be affected if
source and target languages are from distant language families [11, 10].

Hate speech datasets can come from many domains, which can be divided into two
levels: one comes from language shift, and the other is content in each
language. For instance, a dataset in one language may be collected from the
Twitter platform, while another could be collected from Youtube. Standard
fine-tuning overfits the training languages and will face the difficulty of
adapting to a new domain or an unseen language. Conneau et al. [12] show that
adding more languages improves the performance on low-resource languages but
hurts the performance on high-resource languages. We adopt a meta-training
strategy for low-resource hate speech detection to address these concerns. Our
proposed model is based on a meta-learning approach that can be used for low-
resource hate speech detection. The model is capable of handling domain
adaptation and preventing negative transfer. Meta-learning has been successful
in low-resource settings and can be used to learn a good initial point for
fine-tuning a new task, making the fine-tuning process more efficient.

Research objectives. In this paper, we aim to tackle the challenges of
multilingual hate speech detection and address the limitations of existing
fine-tuned pre-trained LMs in hate speech detection. We propose HateMAML, a
model-agnostic meta-learning-based framework that effectively performs hate
speech detection in low-resource languages. Unlike existing hate speech
detection approaches that fine-tune LMs on the source and target only, we
focus on resource maximization by using resources in other “auxiliary”
languages beyond source and target. More specifically, we adopt the model-
agnostic meta-learning (MAML) [13] strategy, in which simulated training in
few-shot meta-tasks produces a meta-learner that can learn a target task
quickly.

Recent studies have explored adopting MAML in low-resource cross-lingual
adaptation for NLP tasks [14, 15]. The underlying intuition of HateMAML is to
train on samples from auxiliary languages and adapt it to detect hate speech
in a target language in which there are few or no datasets available [15]. To
address diversity and nuances in detection of hate speech across languages and
domains, we also modify the meta-learning loss function in HateMAML and add a
domain adaptation loss. Intuitively, task-specific learner evaluates their own
training samples and samples from another language. This mimics the scenario
we are interested in: training in one language while evaluating an unseen
language during test time. Such training objective forces the meta-learner to
generalize to a target language domain while learning from source domain
samples.

Considering the data scarcity of non-English languages, we further propose a
self-guided meta-learning-based fine-tuning mechanism. This mechanism utilizes
unlabeled examples of the low-resource target language and generates silver
labels for fine-tuning the target. We first train a multilingual predictor
model using training data in a high-resource language, e.g., English. The
predictor model predicts unlabeled samples in the target, e.g., Spanish. A
small portion of quality predictions are filtered out using a threshold value
to gather silver labels and enhance the training data in both the source and
target languages. Finally, we adapt HateMAML into an iterative self-refining
loop that replaces and refines the current predictor model and improves the
quality of the adapted model in the target.

In early studies, cross-lingual meta-training was mostly limited to language
pairs [15]. This setting is, however, not truly multilingual since we need
separate models for each target language. We study the applicability of meta-
training when training data is available in more than one low-resource
language. To this end, we investigate whether meta-training is better than
standard multilingual fine-tuning in two scenarios: when training examples are
only available for some languages and when both high and low-resource
languages have sufficient training data. The first scenario is a stress test
for cross-lingual domain adaptation, evaluating the trained model on a set of
reserved low-resource languages. The other scenario explores meta-training
benefits over standard fine-tuning. Though the early investigation was limited
to one auxiliary and target language, the current configuration allows us to
investigate cross-lingual meta-training for hate speech detection at scale. In
other words, we adopt meta-training as a proxy for general multilingual fine-
tuning. Our results show that when training data is available from many
languages, meta-training HateMAML yields better performance than standard
fine-tuning.

Contributions. We present HateMAML, a meta-learning framework for multilingual
hate speech detection with limited data at scale. Our major contributions are
summarized below:

  1. 1.

We propose HateMAML222https://gitlab.com/bottle_shop/safe/HateMAML, a domain-
adaptive MAML algorithm for hate speech detection in low-resource languages.
Our algorithm efficiently exploits the available resources, outperforming
state-of-the-art cross-lingual fine-tuning baselines. One of the novelties of
HateMAML lies in the efficient utilization of available samples in the
validation set of source and training data in auxiliary languages.

  2. 2.

To mimic an extreme data-scarce scenario in the target language (zero-shot),
we introduce a self-refining variant of HateMAML. We adopt an iterative meta-
learning approach to generate silver labels from unlabeled data in the target
language.

  3. 3.

We conduct extensive experiments in zero-shot and fine-tuning settings across
eight different low-resource languages. Our experiment results show that
HateMAML consistently outperforms state-of-the-art methods by more than 3% in
the cross-lingual transfer settings.

  4. 4.

We perform ablation studies, in which we carefully examine the contribution of
the auxiliary languages across each auxiliary- target language pair. We also
assess the algorithm’s robustness by varying (i) the amount of meta-training
samples, (ii) cross-domain adaptation, and (iii) one model to support all
language configurations.

  5. 5.

We conduct cross-domain hate speech detection to address data set diversity in
various languages and show that HateMAML achieves a significant domain
adaptation. Our experiments on meta-training over all available language
training data suggest that meta-learning could be an alternative to standard
fine-tuning that yields superior performance.

##  II Related Work

Multilingual hate speech detection In the last few years, hate speech
detection has gained significant attention, with the majority of work
dedicated towards monolingual hate speech [1, 2]. This skew in attention has
naturally led to the majority of datasets being in English [16, 17, 18].
However, hate speech is a global phenomenon, and increased diversity in
available resources is critical for developing automated systems. Recent
efforts addressed this issue, including multiple shared tasks (such as SemEval
2020 [19], HASOC 2020 [20], Evalita 2018 [21] and HateEval 2019 [22]). Such
efforts helped in bringing much-needed traction toward multilingual hate
speech research. Additionally, recent development in transformer-based large
multilingual LMs (mBERT [6], XLM-R [12]), pre-trained on more than 100
languages, has aided in developing state-of-the-art classifier in resource-
lean languages. Prior studies on multilingual hate speech detection have
explored: (i) resource development such as creating datasets [19, 23], shared
tasks and organizing workshops, (ii) cross-lingual transfer learning with
multilingual shared embeddings and pre-trained LMs [9, 8], (iii) utilization
of additional features from relevant domains [24] (e.g., emotion, sentiment),
and (iv) data augmentation [25, 26, 9] techniques including use of external
service (e.g., translation APIs) for supervised training.

Cross-lingual and multilingual transfer. Knowledge transfer from a high-
resource language (i.e., English) to a low-resource one (i.e., Hindi) has been
shown to be an effective resource optimization technique. Ranasinghe and
Zampieri [7] used a two-step training method: first training multilingual
transformer models on English data, and then the resulting model is fine-tuned
on the low resource target language. Such multi-phase training was found to be
a better model initialization for fine-tuning the target. Aluru et al. [8]
conducted an extensive study on state-of-the-art multilingual embeddings
including LASER [27] and MUSE [28]. Another line of studies [9, 29, 30]
utilized translation-based solutions to alleviate the shortage of data in low-
resource languages in hate speech. These models rely heavily on translation
APIs, which might be susceptible to producing low-quality translations.
Moreover, the translation of a large source corpus can be expensive.

Majority of the studies on cross-lingual and multilingual hate detection fall
into pre-trained model fine-tuning [31, 32, 29]. However, Nozza [10] found
that due to the domain shift of hate speech across languages, standard fine-
tuning methods are less effective for both zero-shot and few-shot settings. To
address the limitation of low-resource language fine-tuning, we develop
HateMAML for fine-tuning hate classifiers with limited data and an improved
mechanism for domain-adaptive cross-lingual transfer.

Meta-learning. Meta-learning, also known as “learning to learn”, is a few-shot
learning technique geared toward learning few-shot tasks and fast adaptation.
Unlike standard supervised fine-tuning on downstream tasks that result in a
final classifier, meta-learning focuses on producing a rich initialization
point from where an unseen target task can be learned quickly. Some common
meta-learning approaches include (i) optimization-based [13, 33], and (ii)
metric-based [34, 35]. Finn et al. [13] introduced model-agnostic meta-
learning (MAML), an optimization-based meta-learning framework for few-shot
tasks. Several recent studies have explored gradient-based meta-learning for
few-shot text classification [36, 37, 38]. Meta-learning for domain
generalization is also studied to handle domain shift during training in
diverse problems, including supervised learning and reinforcement learning
[39]. Meta-learning has also been explored in zero-shot and few-shot cross-
lingual settings. Gu et al. successfully adapted MAML for low-resource NMT
tasks using auxiliary languages, resulting in competitive performance with
only a fraction of the training data [14]. X-MAML was then proposed by [15]
for meta-training in cross-lingual NLU tasks, similar to our interest. X-MAML
explores various auxiliary languages to identify the optimal composition for
zero-shot cross-lingual transfer. Meta-learning has also been applied in the
detection of offensive language in cross-lingual and code-mixed texts [40, 41]
and other harmful content such as multilingual rumours [42]. However, the
limited availability of multilingual hate speech datasets, comprising of only
two or three languages, presents a challenge in finding an effective auxiliary
language.

Data augmentation/self-training. Data augmentation approaches can be divided
into three categories: (i) rule-based [wei-zou-2019-eda], (ii) interpolation-
based [43, 44, 45], and (iii) model-based [46, 47, 48]. In cross-lingual hate
speech, translation techniques [9, 29, 30] are commonly used for data
augmentation during training. Rather than collecting and annotating new hate
speech data, bootstrapping on unlabeled samples to create pseudo-labeled data
provides a way for data augmentation and fine-tuning on low-resource
languages. We draw inspiration from [26] to device a bootstrapping-based self-
training approach for HateMAML.

How is our approach different? We explore cross-lingual meta-training in the
domain of hate speech for both zero-shot and fine-tuning configurations. Our
proposed method of zero-shot meta-training, HateMAML, is a novel idea that can
be further adapted for languages with no availability of labeled data. We
focus on resource maximization and domain generalization while transferring
task-specific knowledge to low-resource languages. We carry out a large-scale
study in multilingual hate speech detection across diverse domains on
available hate speech datasets.

##  III Methodology

We devise a gradient-based meta-learning algorithm named HateMAML for
multilingual hate speech detection. Our proposed algorithm is tailored to
improve cross-lingual transfer in target low-resource languages. We create a
set of meta-tasks from samples for both high- and low-resource languages and
simulate episodic meta-training similar to MAML [13]. Our goal is to produce
better initialization of model parameters that can adapt to a low-resource
target language task using – (i) none (i.e., zero-shot) or (ii) only a small
number (i.e., few-shot) of labeled examples. We select an auxiliary language
to (meta-)train the model in the zero-shot setting. In the few-shot setting,
we meta-train the model on the target language. We assume some labeled data
(e.g., 2024 samples) is available for supervised fine-tuning on the target
language. Hate speech datasets from different languages often comprise several
domains and topics. For example, while many English datasets capture the
domain of racism-related hate, hate speech datasets in Hindi are more
religion/politics-oriented. Here, the term ‘domain’ is loosely used, referring
to different datasets and languages. Therefore, the task of cross-lingual hate
speech detection implicitly encompasses cross-domain adaptation. We aim to
fast adapt to both target language and domains. To account for this, our meta-
adaptation model includes a domain generalization loss. The idea is to produce
a good initialization that can perform well on diverse domains.

###  III-A Model Agnostic Meta-Learning (MAML)

MAML assumes a distribution $p(\mathcal{T})$ of tasks
$\\{\mathcal{T}_{1},\mathcal{T}_{2},\cdots,\mathcal{T}_{m}\\}$. A meta-model
$f_{\theta}(\cdot)$ with parameters $\theta$ is learned through episodic meta-
training over sample tasks $\mathcal{T}\sim p(\mathcal{T})$. MAML has two
loops – (i) an inner loop for task-specific adaptation and (ii) an outer loop
to meta-learn that fast adapt to an unseen novel task.

MAML fine-tunes the meta model $f_{\theta}$ to a particular task
$\mathcal{T}_{i}$ through gradient descent:

$\theta_{i}^{\prime}\leftarrow\theta-\alpha\nabla\mathcal{L}_{\mathcal{T}_{i}}(f_{\theta})$
(1)

where $\alpha$ is the step size, and $\mathcal{L}$ is the classification loss.
The task-specific training outcome is evaluated on the associated test set.

The meta-learner optimization objective is to minimize the meta loss computed
from all the training tasks:

$\min_{\theta}\sum_{i=1}^{m}\mathcal{L}_{\mathcal{T}_{i}}(f_{\theta^{\prime}})=\sum_{i=1}^{m}\mathcal{L}_{\mathcal{T}_{i}}(f_{\theta-\alpha\nabla_{\theta}\mathcal{L}_{\mathcal{T}_{i}}(f_{\theta})}).$
(2)

The meta parameter $\theta$ is then updated to:

$\theta=\theta-\beta\nabla_{\theta}\sum_{i=1}^{m}\mathcal{L}_{\mathcal{T}_{i}}(f_{\theta_{i}^{\prime}})$
(3)

where $\beta$ is the meta-learner learning rate. We can accumulate multiple
episodes of tasks to update $\theta$. Though MAML training requires double
gradient descent optimization, a first-order approximation is used in
practice. Different from standard fine-tuning, meta-training does not result
in a final model. However, it is a reasonably good starting point
(initialization) from which learning a new task can be executed quickly.
Intuitively, training across a set of meta-tasks can be seen as auxiliary; and
fast adaptation on the unseen target task is the main goal. A few recent
studies build on this motivation [15, 14] where supports from auxiliary
languages are used in cross-lingual meta-training. This is also a key
inspiration for our model.

Figure 1: Proposed HateMAML model sketch. (Left) Model training in zero-shot
and few-shot setups. We use an auxiliary language validation set in zero-shot
learning, and target language labeled data is used in few-shot. In both cases,
we include a source language validation set and create a (virtual) domain set
that helps us to mimic target domain performance evaluation during meta-
training. We sample a batch of triplets
$(\mathcal{S},\mathcal{Q},\mathcal{Q}^{\prime})$ and simulate episodic
training. (Right) Self-training procedure with HateMAML, which produces silver
labels from unlabeled data in the target language, model training on those,
replace base model. Repeat the procedure for $N$ times.

###  III-B Proposed Model: HateMAML

The training of our proposed HateMAML model facilitates hate classifier
adaptation and cross-lingual transfer in target languages while requiring only
a small number of samples for model fine-tuning. HateMAML can be used for
training with/without labeled data, only requiring access to training data in
the source language for the initial fine-tuning. Any multilingual encoders
that have shared embeddings, such as mBERT and XLM-R can be used as the base
model. To further improve cross-lingual transfer, we use a few labeled samples
in the target language (few-shot training), which helps quick adaptation.
Additionally, HateMAML benefits from task-specific training evaluation on a
(virtual) domain test (query) set at each episode that mimics target domain
evaluation during meta-training [39]. The intuition of the domain query set is
to achieve training generalization in unseen target languages and domains. We
discuss training datasets accumulation, meta-tasks generation, and domain loss
in HateMAML and present a concise training procedure in detail below. A rough
sketch of HateMAML can be found in Figure 1(right).

HateMAML Training Data. We need a set of support and query batches sampled
from $\mathcal{D}$, where $\mathcal{D}$ refers to available training data from
training languages, generally small in size. Suppose we have samples in source
$(\mathcal{D}_{src})$, auxiliary $(\mathcal{D}_{aux})$ and target
$(\mathcal{D}_{tgt})$ languages. Our training data $\mathcal{D}$ consists of
the tuple: (i) $(\mathcal{D}_{src},\mathcal{D}_{aux})$ in the zero-shot
setting, and (ii) $(\mathcal{D}_{src},\mathcal{D}_{aux})$ in the few-shot
setting. To this end, we split the training set for each language into a
support ($\mathcal{D}_{lang}^{support}$) and query
($\mathcal{D}_{lang}^{query}$), where $lang\in\\{src,aux,tgt\\}$. We randomly
sample a (virtual) domain set $\mathcal{D}^{domain-query}$ from either
$\mathcal{D}_{aux}^{query}$ or $\mathcal{D}_{tgt}^{query}$. By doing so, we
can mimic real train-test domain shifts so that over many iterations, we can
train a model to achieve better generalization on the target language’s final
test set.

Meta-Tasks Generation. HateMAML requires episodic training on meta-tasks
containing support (meta-training) and query (meta-testing) sets. For each
episode, three subsets of examples are sampled separately: the support set
$\mathcal{S}$, the query set $\mathcal{Q}$, and the (virtual) domain query set
$\mathcal{Q}^{\prime}$.

$\displaystyle S\leftarrow S\cup\wedge(X,Y)\in\mathcal{D}$ (4) $\displaystyle
Q\leftarrow Q\cup\wedge(X,Y)\smallsetminus S\in\mathcal{D}$ $\displaystyle
Q^{\prime}\leftarrow Q^{\prime}\cup\wedge(X,Y)\smallsetminus(S\cup
Q)\in\mathcal{D}$

We repeat this process multiple times. Suppose we have a total of $D$ training
samples in a given language and $K$ samples used for support, and $L$ for both
query sets in each episode. In total, we will have $D/K+L$ episodes. Note that
$\mathcal{Q}^{\prime}$ is virtual and selected randomly.

(Virtual) Domain Query Loss. We add query loss term in Equation 3 that
accounts for domain generalization. We now compute meta-update in Equation 3
as follows:

$\theta=\theta-\beta\nabla_{\theta}\sum_{i=1}^{m}\mathcal{L}_{\mathcal{T}_{i}}^{(\mathcal{Q}_{i},\mathcal{Q}_{i}^{\prime})}(f_{\theta_{i}^{\prime}})$
(5)

It is a variation of standard MAML training that addresses supervised training
limitations such as domain shift in low-resource languages. We compare our
proposed training procedure with MAML to show the effectiveness.

###  III-C HateMAML Algorithm

HateMAML training requires a base classifier. A base classifier can be any
multilingual encoder, such as mBERT, XLM-R, fine-tuned on a source language
such as English. Next, we decide between zero-shot or few-shot training based
on the availability of training data. The training choice is passed to
Algorithm 1 along with the base model parameters. For the zero-shot scenario,
we select an auxiliary language for model training, and the resulting meta-
model will be evaluated on the selected target language. Meta-learner
parameters $\theta$ are initialized from a fine-tuned base model (line 2).
Now, we sample a batch of pseudo-tasks $\mathcal{T}_{i}$. Each task contains a
triplet $(\mathcal{S},\mathcal{Q},\mathcal{Q}\prime)$ representing the
support, query, and domain query for model training, respectively. We take one
inner loop gradient step using training loss on $\mathcal{S}$ (line 10) and
adapt model parameters to $\theta^{\prime}$. Now, we evaluate task-specific
training outcomes using $\mathcal{Q}$ and $\mathcal{Q}^{\prime}$, and save it
for outer loop update (line 12). Each task has its own task-adaptive model
parameters $\theta_{i}^{\prime}$. At the end of iterations of all tasks, we
aggregate saved gradients and update the meta-model $\theta$ (line 13). At
this step, we have completed one outer loop. We repeat lines 8-13 till model
training is not finished. After training completion, based on the earlier
choice of zero-shot or few-shot, we evaluate the resulting model’s performance
on the target language’s final test set.

Input: Training choice C, high-resource language h, set of low-resource
languages L, Meta-learner M, fast lr $\alpha$, meta lr $\beta$

1

2Fine-tune $\theta$ on h (base model)

3 Initialize $\theta\leftarrow\theta$

4 if  _C is zero-shot_ then

5 Select auxiliary from L to get
$\mathcal{D}=(\mathcal{D}_{src},\mathcal{D}_{aux})$

6 else

7 Select target language get L to form
$\mathcal{D}=(\mathcal{D}_{src},\mathcal{D}_{tgt})$

8while  _not done_ do

9 Sample batch of tasks
$\mathcal{T}=\\{\mathcal{T}_{1},\mathcal{T}_{2}..,\mathcal{T}_{m}\\}\sim\mathcal{D}$

10 for  _all $\mathcal{T}_{i}=(\mathcal{S},\mathcal{Q},\mathcal{Q}^{\prime})$
in $\mathcal{T}$_ do

11 Compute $\nabla_{\theta}\mathcal{L}_{\mathcal{T}_{i}}(\texttt{M}_{\theta})$
on $S$

12 Compute adapted parameters with gradient descent:
$\theta^{{}^{\prime}}=\theta-\alpha\nabla_{\theta}\mathcal{L}_{\mathcal{T}_{i}}(\texttt{M}_{\theta})$

13 Compute $\mathcal{L}_{\mathcal{T}_{i}}(\texttt{M}_{\theta^{{}^{\prime}}})$
using $\mathcal{Q}$, $\mathcal{Q}_{d}$ for outer loop meta-update

14

15 Update
$\theta\leftarrow\theta-\beta\nabla_{\theta}\sum_{i}\mathcal{L}_{\mathcal{T}_{i}}(\texttt{M}_{\theta^{{}^{\prime}}})$

16

17

18Perform test set evaluation on the target language using
$\texttt{M}_{\theta}$

Algorithm 1 HateMAML

###  III-D Self-refinement and Zero-shot Learning

We now present a semi-supervised self-training procedure shown in Figure 1
(left) for hate speech classifier adaptation while no target or auxiliary
language is available. Our proposed approach requires the availability of the
source language. We first develop a ‘base model’ by fine-tuning source
language training data. Now, we can use the trained model to make predictions
on unlabeled samples from the target language. We want to use these predicted
samples for supervised training. Directly including all the predicted samples
in training may not result in a good classifier, as training samples are noisy
and do not represent ground-truth labels. We filter out samples with low
prediction confidence using a threshold value to keep it balanced. We call the
filtered outcomes silver labels and prepare a new training dataset. We use
HateMAML to carry out meta-training since it requires a few samples and can
adapt quickly. As we use a small number of samples, noise injection is less
than standard fine-tuning. We randomly take filtered samples as silver labels
and run HateMAML using few-shot training shown in Algorithm 1. After
completing one full training on the silver training data, the new classifier
should expectantly have better target language knowledge. To make it work in
practice requires self-refinement with multiple iterations. We repeat this
procedure by treating the resulting model as the base for generating silver
labels again, replacing the old base model. In our experiments, approximately
$5$ refinement iterations with HateMAML are enough for producing a quality
classifier on the target test data.

TABLE I: A summary of the multilingual hate speech datasets.

Dataset Language Size of training set Size of test set Domain Founta-EN[18]
English (EN) 80000 — Twitter stream HatEval19 [22] English (EN), Spanish (ES)
EN: 9000, ES: 4500 EN: 2971, ES: 1600 Immigrant, Woman SemEval20 [19] Arabic
(AR), Danish (DA), Greek (EL), Turkish (TR) AR: 8000, DA: 2961, EL: 8743, TR:
31756 AR: 200, DA: 329, EL: 1544, TR: 3528 Twitter, Facebook, Reddit, News
HASOC20 [23] English (EN), Hindi (HI), German (DE) EN: 3708, HI: 2963, DE:
2373 EN: 1628, HI: 1326, DE: 1052 Twitter stream HaSpeeDe [49] Italian (IT)
IT: 6839 $\bf{IT_{tweeter}}$: 1263, $\bf{IT_{news}}$: 500 Immigration, Roma,
Muslims

##  IV Hate speech detection datasets

We train and evaluate the baselines and HateMAML using five publicly-available
multilingual hate speech datasets. These datasets contain a variety of low-
resource languages. Table I summarizes the datasets.

  * •

Founta-EN [18] is a large English dataset of about 80,000 tweets, collected
through crowd-sourcing. These tweets were labeled into – normal, spam,
abusive, and hateful. For our study, we removed the tweets labeled spam.

  * •

HatEval19 [22], released in the shared task of SemEval-2019, is a multilingual
hate speech dataset in English and Spanish, specifically targeted hate against
immigrants and women.

  * •

HASOC20 [23] is a multilingual hate speech corpus of three languages –
English, German, and Hindi. The authors first created an extensive archive of
tweets and subsequently used weakly trained binary classifiers to extract
potential hateful tweets. The extracted ones are finally labeled by
annotators. The main aim of the dataset was to create an unbiased multilingual
hate speech corpus.

  * •

HaSpeedDe20 [49] is a monolingual corpus in Italian. It focuses on hate speech
against immigrants, Roma and Muslims. The dataset is collected from two
platforms – Twitter and news headlines. It consists of test sets from Twitter
and news.

  * •

SemEval20 [19] covers five languages – English, Arabic, Danish, Greek, and
Turkish. Native speakers annotated the dataset. Arabic, Greek, and Turkish
were collected from Twitter, while the Danish dataset is from Facebook,
Reddit, and a local newspaper. The task also produced a weakly-supervised
English dataset.

##  V Experiments

###  V-A Experiment Settings

We consider three training setups: (i) zero-shot with no fine-tuning in the
target, (ii) domain adaptation in which we consider training on a set of
auxiliary languages and test on a held-out language set, and (iii) full fine-
tuning using the training samples from all languages. Our experimental setup
also requires a base model, acquired from a pre-trained model, which is fine-
tuned on the source (English) language.

For the cross-lingual analysis, we experiment on eight low-resource languages
and one high-resource English. For the experiments on HASOC20 and HatEval19,
we utilize the English samples in the training set to train the base model.
For the experiments on SemEval2020 and HaSpeedDe20, we use Founta-EN as
English training data is not available.

The meta-training samples are retrieved from the source languages’ validation
sets as well as the auxiliary and target languages’ training sets. The source
languages’ training sets are used to train the base model. Across all the
experiments, we aggregate samples from each language and then generate meta-
tasks to be used in meta-training. We sample task triplets with a specified
number of shots, $K=L$, where $K=32$.

We utilize two multilingual transformer-based encoders as base models: (i)
$\bf mBERT$ [6] (bert-base-multilingual-uncased), and (ii) $\bf XLM-R$ [12]
(xlm-roberta-base). For both models, the output from the pooler layer is fed
to a 2-layer FFN classification head.

###  V-B Baselines

HateMAML is a model-agnostic algorithm for low-resource cross-lingual
adaptation. Therefore, we explore the following baselines for our experiments:

  * •

Fine-tuning: Standard LM fine-tuning is adapted as a baseline for both zero-
shot and few-shot scenarios. Here, we fine-tune on mBERT and XLM-R.

  * •

X-MAML [15]: Cross-lingual episodic meta-training using auxiliary language
composition, including pairs. We initialize the meta-learner from the base
model obtained from English fine-tuning. The meta-learned model is then
evaluated on the test set in the target language. Our implementation only
considers one auxiliary language.

###  V-C Variations of HateMAML

Besides benchmarking the baselines, we formulate and evaluate different
variations of HateMAML:

  * •

$\texttt{HateMAML}_{\textit{zero}}$: Simulated meta-training on a batch of
tasks created from aggregated samples in source language ‘validation set’ and
‘training set’ from auxiliary language. It provides zero-shot as training
choice in Algorithm 1 (§ III-C). This is similar to X-MAML[15], but it uses
both source and auxiliary languages for meta-training, while X-MAML only uses
auxiliary language samples.

  * •

$\texttt{HateMAML}_{\textit{self-training}}$: Choose a target language for
self-training on silver data. First, we generate silver labels and keep only
300 samples. Next, we apply HateMAML in an iterative manner to the silver-
labeled data from the target language as explained in Section III-D.

  * •

$\texttt{HateMAML}_{ft}$: We generate meta-task from a set of available
languages and apply HateMAML. This is very similar to fine-tuning aggregated
training examples from the selected language set.

###  V-D Implementation Details

For implementing the transformer-base models, we use the HuggingFace
transformers library333https://huggingface.co/transformers/. The chosen pre-
trained model is initialized from pre-trained weights provided in the
transformers library. The classification heads are implemented using the
pytorch444https://pytorch.org/ linear layers, initialized randomly. For
implementing the meta-learning features (for example, first-order
approximation), we use the learn2learn
library555https://github.com/learnables/learn2learn.

TABLE II: Zero-shot evaluation on target low-resource languages in several
multilingual hate speech datasets. Here $\clubsuit$ indicates that the pre-
trained model is initialized with BERT, and $\spadesuit$ refers to XLM-R
initialization. All the experiments (rows containing either $\clubsuit$ or
$\spadesuit$ symbol) require a base model that is created using standard fine-
tuning on a high-resource (EN) training set. Empty cells $(-)$ indicate that
meta-learning experiments are impossible due to the unavailability of the
required auxiliary language. In each cell ($x_{y}$), $x$ and $y$ denote the
mean and standard deviation of macro-F1 values across five runs by varying the
random seed initialization.

Model Aux lang HatEval19 SemEval20 HaSpeedDe20 HASOC20 AVG ES AR DA EL TR
$IT_{news}$ $IT_{tweets}$ HI DE mBERT - $0.499_{0.025}$ $0.448_{0.000}$
$0.650_{0.000}$ $0.457_{0.000}$ $0.452_{0.010}$ $0.436_{0.023}$ $0.489_{0.25}$
$0.426_{0.000}$ $0.414_{0.000}$ $0.473_{0.007}$ $\clubsuit X-MAML$ ✓ -
$0.562_{0.024}$ $0.654_{0.017}$ $0.501_{0.040}$ $0.585_{0.002}$ -
$0.455_{0.015}$ $0.629_{0.032}$ $0.564_{0.021}$
$\clubsuit\texttt{HateMAML}_{zero}$ ✓ - $0.583_{0.010}$ $0.668_{0.017}$
$0.532_{0.004}$ $0.601_{0.005}$ - $0.448_{0.073}$ $0.745_{0.046}$ $\bf
0.592_{0.026}$ $\clubsuit\texttt{HateMAML}_{self\\_training}$ - $\bf
0.525_{0.033}$ $0.529_{0.019}$ $0.657_{0.046}$ $0.547_{0.021}$ $0.586_{0.002}$
$0.569_{0.031}$ $\bf 0.662_{0.056}$ $0.509_{0.029}$ $0.572_{0.048}$
$0.562_{0.028}$ XLM-R - $0.421_{0.064}$ $0.524_{0.000}$ $0.683_{0.000}$
$0.529_{0.000}$ $0.478_{0.000}$ $0.457_{0.028},0.593_{0.039}$ $0.448_{0.000}$
$0.576_{0.000}$ $0.523_{0.014}$ $\spadesuit X-MAML$ $\checkmark$ -
$0.647_{0.007}$ $0.699_{0.007}$ $0.604_{0.023}$ $0.527_{0.014}$ -
$0.473_{0.010}$ $0.672_{0.007}$ $0.604_{0.011}$
$\spadesuit\texttt{HateMAML}_{zero}$ $\checkmark$ - $0.651_{0.007}$
$0.735_{0.006}$ $0.619_{0.003}$ $0.580_{0.028}$ - $0.497_{0.017}$
$0.650_{0.013}$ $\bf 0.622_{0.012}$
$\spadesuit\texttt{HateMAML}_{self\\_training}$ - $\bf 0.436_{0.0846}$
$0.621_{0.048}$ $0.688_{0.036}$ $0.635_{0.042}$ $0.600_{0.016}$
$0.492_{0.020}$ $0.637_{0.021}$ $0.598_{0.003}$ $0.661_{0.005}$
$0.591_{0.031}$

TABLE III: Domain Adaptation experiments on both fine-tuning and meta-
training. We use languages = {EN,ES,IT,DA,DE} for training and evaluate on all
eight languages. Here, first block means the pre-trained model is initialized
with BERT and the second block refers to XLM-R initialization.

Model data HatEval19 SemEval20 HaSpeedDe20 HASOC20 AVG ES AR DA EL TR
$IT_{news}$, $IT_{tweets}$ HI DE $\clubsuit FT$ 1024 $0.684_{0.022}$
$0.490_{0.035}$ $0.711_{0.020}$ $0.533_{0.012}$ $0.504_{0.027}$
$0.666_{0.035},0.703_{0.014}$ $0.537_{0.023}$ $0.735_{0.046}$ $0.618_{0.098}$
$\clubsuit FT$ 2048 $0.727_{0.049}$ $0.517_{0.041}$ $0.726_{0.018}$
$0.515_{0.022}$ $0.541_{0.024}$ $0.667_{0.035},0.695_{0.031}$ $0.464_{0.038}$
$0.781_{0.010}$ $0.626_{0.115}$ $\clubsuit FT$ 4096 $0.753_{0.024}$
$0.493_{0.037}$ $0.753_{0.021}$ $0.510_{0.035}$ $0.546_{0.030}$
$0.658_{0.041},0.736_{0.010}$ $0.462_{0.041}$ $0.805_{0.005}$ $0.635_{0.130}$
$\clubsuit\texttt{HateMAML}_{ft}$ 1024 $0.709_{0.006}$ $0.495_{0.059}$
$0.723_{0.022}$ $0.545_{0.015}$ $0.539_{0.032}$ $0.607_{0.058},0.711_{0.004}$
$0.498_{0.030}$ $0.766_{0.013}$ $0.622_{0.106}$
$\clubsuit\texttt{HateMAML}_{ft}$ 2048 $0.737_{0.019}$ $0.546_{0.009}$
$0.724_{0.019}$ $0.559_{0.017}$ $0.556_{0.014}$ $0.688_{0.031},0.722_{0.026}$
$0.474_{0.021}$ $0.795_{0.011}$ $0.645_{0.108}$
$\clubsuit\texttt{HateMAML}_{ft}$ 4096 $0.746_{0.018}$ $0.556_{0.020}$
$0.748_{0.025}$ $0.559_{0.008}$ $0.567_{0.009}$ $0.695_{0.017},0.702_{0.035}$
$0.475_{0.009}$ $0.802_{0.015}$ $\bf 0.650_{0.109}$ $\spadesuit FT$ 1024
$0.521_{0.154}$ $0.517_{0.088}$ $0.574_{0.111}$ $0.484_{0.047}$
$0.517_{0.079}$ $0.521_{0.135},0.492_{0.164}$ $0.492_{0.072}$ $0.580_{0.162}$
$0.522_{0.113}$ $\spadesuit FT$ 2048 $0.561_{0.180}$ $0.482_{0.045}$
$0.623_{0.149}$ $0.507_{0.047}$ $0.527_{0.077}$ $0.540_{0.145},0.507_{0.188}$
$0.438_{0.025}$ $0.638_{0.207}$ $0.536_{0.137}$ $\spadesuit FT$ 4096
$0.579_{0.191}$ $0.468_{0.024}$ $0.626_{0.146}$ $0.513_{0.060}$
$0.500_{0.060}$ $0.570_{0.167},0.536_{0.182}$ $0.431_{0.016}$ $0.646_{0.211}$
$0.541_{0.142}$ $\spadesuit\texttt{HateMAML}_{ft}$ 1024 $0.725_{0.011}$
$0.608_{0.014}$ $0.716_{0.030}$ $0.582_{0.027}$ $0.624_{0.004}$
$0.664_{0.042},0.732_{0.013}$ $0.552_{0.018}$ $0.769_{0.008}$ $0.663_{0.075}$
$\spadesuit\texttt{HateMAML}_{ft}$ 2048 $0.726_{0.022}$ $0.637_{0.035}$
$0.749_{0.019}$ $0.609_{0.023}$ $0.614_{0.029}$ $0.663_{0.031},0.681_{0.046}$
$0.510_{0.032}$ $0.806_{0.010}$ $\bf 0.666_{0.088}$
$\spadesuit\texttt{HateMAML}_{ft}$ 4096 $0.738_{0.021}$ $0.562_{0.061}$
$0.745_{0.043}$ $0.595_{0.019}$ $0.586_{0.017}$ $0.702_{0.009},0.700_{0.034}$
$0.481_{0.025}$ $0.805_{0.007}$ $0.657_{0.104}$

TABLE IV: Fine-tuning and meta-training evaluation on multilingual hate speech
datasets. We report F1 score. Here, first block means the pre-trained model is
initialized with BERT and the second block refers to XLM-R initialization.

Model data HatEval19 SemEval20 HaSpeedDe20 HASOC20 AVG ES AR DA EL TR
$IT_{news}$, $IT_{tweets}$ HI DE $\clubsuit FT$ 1024 $0.663_{0.009}$
$0.721_{0.016}$ $0.712_{0.027}$ $0.700_{0.080}$ $0.610_{0.053}$
$0.629_{0.039},0.686_{0.009}$ $0.590_{0.029}$ $0.760_{0.017}$ $0.675_{0.064}$
$\clubsuit FT$ 2048 $0.718_{0.024}$ $0.765_{0.020}$ $0.737_{0.028}$
$0.717_{0.041}$ $0.652_{0.050}$ $0.652_{0.037},0.708_{0.021}$ $0.609_{0.054}$
$0.796_{0.012}$ $0.706_{0.065}$ $\clubsuit FT$ 4096 $0.717_{0.017}$
$0.785_{0.024}$ $0.747_{0.037}$ $0.722_{0.007}$ $0.670_{0.035}$
$0.680_{0.045},0.684_{0.068}$ $0.584_{0.071}$ $0.803_{0.004}$ $0.712_{0.077}$
$\clubsuit\texttt{HateMAML}_{ft}$ 1024 $0.693_{0.015}$ $0.726_{0.025}$
$0.692_{0.018}$ $0.685_{0.044}$ $0.626_{0.018}$ $0.593_{0.026},0.702_{0.020}$
$0.583_{0.020}$ $0.743_{0.023}$ $0.671_{0.059}$
$\clubsuit\texttt{HateMAML}_{ft}$ 2048 $0.735_{0.013}$ $0.779_{0.011}$
$0.716_{0.013}$ $0.732_{0.026}$ $0.665_{0.007}$ $0.670_{0.019},0.730_{0.011}$
$0.627_{0.006}$ $0.789_{0.013}$ $0.716_{0.052}$
$\clubsuit\texttt{HateMAML}_{ft}$ 4096 $0.757_{0.018}$ $0.816_{0.012}$
$0.752_{0.010}$ $0.742_{0.022}$ $0.689_{0.012}$ $0.643_{0.042},0.741_{0.010}$
$0.627_{0.016}$ $0.793_{0.009}$ $\bf 0.729_{0.063}$ $\spadesuit FT$ 1024
$0.630_{0.063}$ $0.680_{0.106}$ $0.774_{0.028}$ $0.595_{0.064}$
$0.593_{0.077}$ $0.605_{0.081},0.643_{0.029}$ $0.621_{0.036}$ $0.735_{0.059}$
$0.653_{0.085}$ $\spadesuit FT$ 2048 $0.684_{0.019}$ $0.730_{0.043}$
$0.782_{0.022}$ $0.673_{0.061}$ $0.671_{0.032}$ $0.683_{0.039},0.629_{0.058}$
$0.641_{0.043}$ $0.800_{0.006}$ $0.699_{0.067}$ $\spadesuit FT$ 4096
$0.704_{0.015}$ $0.806_{0.021}$ $0.774_{0.008}$ $0.705_{0.021}$
$0.684_{0.013}$ $0.651_{0.033},0.679_{0.023}$ $0.632_{0.022}$ $0.806_{0.011}$
$0.716_{0.064}$ $\spadesuit\texttt{HateMAML}_{ft}$ 1024 $0.728_{0.004}$
$0.781_{0.016}$ $0.736_{0.001}$ $0.703_{0.004}$ $0.695_{0.001}$
$0.608_{0.015},0.741_{0.018}$ $0.664_{0.010}$ $0.789_{0.000}$ $0.716_{0.056}$
$\spadesuit\texttt{HateMAML}_{ft}$ 2048 $0.742_{0.012}$ $0.803_{0.002}$
$0.733_{0.003}$ $0.692_{0.018}$ $0.695_{0.024}$ $0.652_{0.015},0.748_{0.007}$
$0.672_{0.005}$ $0.823_{0.009}$ $0.729_{0.057}$
$\spadesuit\texttt{HateMAML}_{ft}$ 4096 $0.755_{0.005}$ $0.832_{0.010}$
$0.736_{0.022}$ $0.711_{0.019}$ $0.706_{0.033}$ $0.694_{0.036},0.738_{0.022}$
$0.657_{0.016}$ $0.812_{0.008}$ $\bf 0.738_{0.056}$

##  VI Results and analysis

###  VI-A Zero-shot Experiments

Experiments in the zero-shot setting can be found in Table II. The mBERT
initialization uses the $\clubsuit$ sign, and $\spadesuit$ refers to XLM-R
initialization. The reported results of all the experiments are average values
across five random runs. We report the average and standard deviation of the
F1-score (macro). $\texttt{HateMAML}_{zero}$ outperforms baseline models
including X-MAML with both initializations ($\clubsuit$ and $\spadesuit$). We
achieve the best overall score with an average of 11% over mBERT, 7% over
XLM-R, 7% over $\clubsuit$ X-MAML. Baseline models, mBERT and XLM-R, have poor
zero-shot performance. The use of auxiliary language to obtain the best meta-
learner produces rewarding outcomes for all the experiments. Even though
X-MAML uses a similar cross-lingual, auxiliary language-based, meta-training
setting to ours, HateMAML gives superior performance, likely due to the
domain-adaptive training in HateMAML. To summarize, in most cases, HateMAML’s
auxiliary to target transfer improves the results of zero-shot learning
substantially compared to the standard fine-tuning baselines.

We observe that the XLM-R-initialized models perform better than mBERT in
zero-shot prediction. Even the best performing $\clubsuit\texttt{HateMAML}$
model has a small gap between XLM-R zero-shot performance. One explanation is
that XLM-R is trained on a relatively large corpus and a deeper transformer
architecture, giving it a strong pre-training benefit for zero-shot transfer
learning. We observe similar trends in performance with HateMAML when
initialized with XLM-R base models across all the datasets. For the SemEval20
Danish (DA) test set, it is noted that the XLM-R model performs the best
compared to other meta-learning models. One reason could be that this dataset
was created from Facebook, Reddit, and news sources, unlike the other datasets
that have been created from Twitter.

Language similarity and cross-lingual transfer. We carefully analyze the
support that each target language receives from different auxiliary languages.
The SemEval20 dataset, with the largest number of possible (auxiliary, target)
language pairs, shows the benefit of meta-training on auxiliary language for
both X-MAML and $\texttt{HateMAML}_{zero}$ models. Meta-training helps in
increasing the zero-shot performance for SemEval20 from 0.473 avg. F1 (mBERT)
to 0.592 F1 ($\clubsuit\texttt{HateMAML}$). X-MAML also shows substantial
improvement for both initializations. We notice that SemEval20 Turkish (TR)
tends to be the best auxiliary language dataset in almost all cases for both
base models (see Table VI). For HASOC20 experiments, we notice small gain in
F1-score while training on auxiliary samples. The languages – German (DE) and
Hindi (HI), do not seem to be a good mix, coming from two distant language
families (details of family languages is provided in Table V).

TABLE V: List of languages and their ISO codes used in our experiments. Language | ISO 639-1 code | Family
---|---|---
Arabic | AR | Afro-Asiatic
Danish | DA | IE: Germanic
German | DE | IE: Germanic
Greek | EL | IE: Greek
English | EN | IE: Germanic
Hindi | HI | IE: Indo-Aryan
Spanish | ES | IE: Italic
Turkish | TR | Turkic
TABLE VI: Zero-shot experiments and results using auxiliary languages on
SemEval2020 dataset. All the results are produced from XLM-R initialization
and average over 5 runs.

Model Auxiliary lang Target lang AR TR DA EL $\texttt{HateMAML}_{zero}$ AR -
$0.553_{0.026}$ $0.718_{0.011}$ $0.591_{0.008}$ $\texttt{HateMAML}_{zero}$ TR
$0.651_{0.007}$ - $0.738_{0.012}$ $0.619_{0.003}$ $\texttt{HateMAML}_{zero}$
DA $0.628_{0.022}$ $0.580_{0.028}$ - $0.553_{0.026}$
$\texttt{HateMAML}_{zero}$ EL $0.530_{0.017}$ $0.494_{0.021}$ $0.657_{0.044}$
-

Self-training improves zero-shot performance. For datasets that have no
auxiliary language, i.e., HatEval19 and HaSpeedDe20, self-training HateMAML is
a convenient algorithm to maximize the performance. It mitigates the need for
auxiliary language or a labeled target language training set. We find that
$\texttt{HateMAML}_{self-training}$ produces comparable performance to meta-
training in auxiliary $\texttt{HateMAML}_{zero}$ (see Table II). The gain is
achieved from meta-training on silver labels of the target using multiple
iterations of self-refinement. We also utilize source language validation set
while self-training ($\texttt{HateMAML}_{\textit{self-training}}$). The
intuition is to have some gold-labeled samples to balance between noise and
ground truth. We observe a slight improvement given the source ground-truth
labels in training.

###  VI-B Domain Adaptation Experiments

Table III reports the results of domain adaptation experiments. We first
select two language families for training: Germanic and Romance. The languages
in the selected languages families, namely EN, DA, DE, ES, and IT, are
considered auxiliary and used in the training set. The trained model is
evaluated on the held-out languages. We use training samples in increasing
order to evaluate the model performance in the order of available data. Each
language’s varying training data size has three values, i.e., $1024$, $2048$,
and $4096$. However, if a language does not contain the selected number of
samples, i.e., DE has only $2373$ training samples, we capped it to the
maximum of found training samples. It can be seen that HateMAML produces
superior results on average compared to fine-tuning. We find that domain-
adaptive meta-training in $\texttt{HateMAML}_{ft}$ ($\clubsuit$ or
$\spadesuit$) has a consistently better F1 score than fine-tuning. We achieve
an overall improvement of $3\%$ over mBERT, $24\%$ over XLM-R while training
$2048$ samples per language. Both baseline models face a performance drop in
held-out languages, in which XLM-R faces a significant performance drop. To
summarize, standard fine-tuning does not generalize well during training on
the held-out target languages. On the other hand, HateMAML’s domain-adaptive
training helps retain consistent performance on both auxiliary and target
languages.

###  VI-C Full Fine-tuning vs Meta-training

To make one model support all languages, we evaluate a meta-training strategy
while training on data available in eight languages. We train a model varying
the training data size and evaluate performance under increasing data
availability. We create a set of meta-tasks by gathering support and query
sets from eight languages. We also fine-tune aggregated training data. In
Table IV, we show how $\texttt{HateMAML}_{ft}$ performs in comparison to mBERT
and XLM-R fine-tuning. In general, we notice an upward trend while increasing
the number of training data for all languages. As expected, the meta-learned
initialized $\texttt{HateMAML}_{ft}$ models perform better than the standard
fine-tuning. $\texttt{HateMAML}_{ft}$ outperforms on average for increasing
the order of available data. We observe that $\texttt{HateMAML}_{ft}$ gains in
F1 score for all languages – 2.4% and 3% improvement over baselines mBERT and
XLM-R using $4096$ samples. To improve performance further, we can increase
training data, if available. This suggests that meta-training can be an
alternative to standard fine-tuning for hate speech detection.

##  VII Conclusion

Our extensive experiments show that HateMAML is able to perform well in both
zero-shot and few-shot hate speech detection by improving cross-lingual
transfer in target languages. HateMAML can be trained with or without labeled
data. This model-agnostic approach supports multilingual encoders with shared
embeddings such as mBERT and XLM-R as base learners. The proposed zero-shot
self-refining technique adapts a base learner to be an effective predictor in
the target, reducing the need for ground-truth labels in fine-tuning. To
further improve cross-lingual transfer, we use a few labeled samples in the
target language (few-shot training), which helps to adapt fast and boosts
fine-tuning performance. Additionally, our method benefits from task-specific
learner evaluation on a (virtual) domain test (query) set at each episode that
mimics target domain evaluation during cross-lingual meta-training. The
intuition of the domain query set is to achieve training generalization in
unseen target languages and domains. We also found that meta-learner
adaptation effectively supports all available languages using a single
predictor model, making it highly suitable for detecting hate speech in
multiple languages and domains.

In this paper, our major contributions are two-fold. (i) We proposed HateMAML,
a novel model-agnostic cross-lingual meta-training algorithm for hate speech
detection that supports resource maximization. We evaluated HateMAML on five
benchmark datasets across eight languages and varying domains. We showed that
cross-lingual meta-training outperforms state-of-the-art fine-tuning baselines
in both zero-shot and fine-tuning settings. We also found that meta-adaptation
is effective in cross-domain evaluation, making it highly suitable for
detecting cross-domain hate speech. (ii) We devised a self-training procedure
that aids in hate speech detection in extreme data-scarce scenarios. In
future, we aim to explore multi-class hate speech detection and meta-training
on a set of heterogeneous tasks such as aggression, target identification,
code-mixing, etc. We hope our work will motivate modeling fast adaptation for
cross-lingual training and zero-shot hate speech detection.

## References

  * [1] J. H. Park and P. Fung, “One-step and two-step classification for abusive language detection on Twitter,” in _Workshop on Abusive Language Online_ , 2017, pp. 41–45.
  * [2] R. Cao, R. K.-W. Lee, and T.-A. Hoang, “Deephate: Hate speech detection via multi-faceted text representations,” in _WEBSCI_ , 2020, pp. 11–20.
  * [3] M. R. Awal, R. Cao, R. K.-W. Lee, and S. Mitrović, “Angrybert: Joint learning target and emotion for hate speech detection,” in _PAKDD_. Springer, 2021, pp. 701–713.
  * [4] K.-Y. Lin, R. K.-W. Lee, W. Gao, and W.-C. Peng, “Early prediction of hate speech propagation,” in _ICDMW_. IEEE, 2021, pp. 967–974.
  * [5] Q. Meng, T. Suresh, R. K.-W. Lee, and T. Chakraborty, “Predicting hate intensity of twitter conversation threads,” _arXiv preprint arXiv:2206.08406_ , 2022.
  * [6] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in _NAACL_ , 2019, pp. 4171–4186.
  * [7] T. Ranasinghe and M. Zampieri, “Multilingual offensive language identification with cross-lingual embeddings,” in _EMNLP_ , 2020, pp. 5838–5844.
  * [8] S. S. Aluru, B. Mathew, P. Saha, and A. Mukherjee, “A deep dive into multilingual hate speech classification,” in _ECML/PKDD_ , 2020.
  * [9] E. W. Pamungkas and V. Patti, “Cross-domain and cross-lingual abusive language detection: A hybrid approach with deep learning and a multilingual lexicon,” in _ACL-Student research workshop_ , 2019, pp. 363–370.
  * [10] D. Nozza, “Exposing the limits of zero-shot cross-lingual hate speech detection,” in _ACL-IJCNLP_ , Aug. 2021, pp. 907–914.
  * [11] H. Al Kuwatly, M. Wich, and G. Groh, “Identifying and measuring annotator bias based on annotators’ demographic characteristics,” in _Workshop on Online Abuse and Harms_ , 2020, pp. 184–190.
  * [12] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, É. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Unsupervised cross-lingual representation learning at scale,” in _ACL_ , 2020, pp. 8440–8451.
  * [13] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _ICML_ , 2017, pp. 1126–1135.
  * [14] J. Gu, Y. Wang, Y. Chen, V. O. K. Li, and K. Cho, “Meta-learning for low-resource neural machine translation,” in _EMNLP_ , 2018, pp. 3622–3631.
  * [15] F. Nooralahzadeh, G. Bekoulis, J. Bjerva, and I. Augenstein, “Zero-shot cross-lingual transfer with meta learning,” in _EMNLP_ , 2020, pp. 4547–4562.
  * [16] T. Davidson, D. Warmsley, M. W. Macy, and I. Weber, “Automated hate speech detection and the problem of offensive language,” in _ICWSM_ , 2017, pp. 512–515.
  * [17] Z. Waseem and D. Hovy, “Hateful symbols or hateful people? predictive features for hate speech detection on Twitter,” in _NAACL: Student Research Workshop_ , Jun. 2016, pp. 88–93.
  * [18] A.-M. Founta, C. Djouvas, D. Chatzakou, I. Leontiadis, J. Blackburn, G. Stringhini, A. Vakali, M. Sirivianos, and N. Kourtellis, “Large scale crowdsourcing and characterization of twitter abusive behavior,” in _ICWSM_ , 2018.
  * [19] M. Zampieri, P. Nakov, S. Rosenthal, P. Atanasova, G. Karadzhov, H. Mubarak, L. Derczynski, Z. Pitenis, and Ç. Çöltekin, “SemEval-2020 task 12: Multilingual offensive language identification in social media (OffensEval 2020),” in _SemEVal_ , Dec. 2020, pp. 1425–1447.
  * [20] T. Mandl, S. Modha, P. Majumder, D. Patel, M. Dave, C. Mandlia, and A. Patel, “Overview of the hasoc track at fire 2019: Hate speech and offensive content identification in indo-european languages,” in _FIRE_ , 2019, pp. 14–17.
  * [21] E. Fersini, D. Nozza, and P. Rosso, “Overview of the evalita 2018 task on automatic misogyny identification (ami),” in _EVALITA@CLiC-it_ , 2018.
  * [22] V. Basile, C. Bosco, E. Fersini, D. Nozza, V. Patti, F. M. Rangel Pardo, P. Rosso, and M. Sanguinetti, “SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter,” in _SemEval_ , 2019, pp. 54–63.
  * [23] T. Mandl, S. Modha, A. Kumar M, and B. R. Chakravarthi, “Overview of the hasoc track at fire 2020: Hate speech and offensive language identification in tamil, malayalam, hindi, english and german,” in _Forum for Information Retrieval Evaluation_ , ser. FIRE 2020, 2020, p. 29–32.
  * [24] I. Markov, N. Ljubešić, D. Fišer, and W. Daelemans, “Exploring stylometric and emotion-based features for multilingual cross-domain hate speech detection,” in _Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis_ , 2021, pp. 149–159.
  * [25] T. Wullach, A. Adler, and E. Minkov, “Fight fire with fire: Fine-tuning hate detectors using large samples of generated hate speech,” 2021.
  * [26] I. Bigoulaeva, V. Hangya, and A. Fraser, “Cross-lingual transfer learning for hate speech detection,” in _Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion_ , 2021, pp. 15–25.
  * [27] M. Artetxe and H. Schwenk, “Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond,” _TACL_ , vol. 7, pp. 597–610, 2019.
  * [28] A. Conneau, G. Lample, M. Ranzato, L. Denoyer, and H. Jégou, “Word translation without parallel data,” _arXiv preprint arXiv:1710.04087_ , 2017\.
  * [29] A. Jiang and A. Zubiaga, “Cross-lingual capsule network for hate speech detection in social media,” in _HYPERTEXT_ , 2021, p. 217–223.
  * [30] H. Sohn and H. Lee, “Mc-bert4hate: Hate speech detection using multi-channel bert for different languages and translations,” in _ICDMW_ , 2019, pp. 551–559.
  * [31] P. Saha, B. Mathew, P. Goyal, and A. Mukherjee, “Hatemonitors: Language agnostic abuse detection in social media,” _CoRR_ , vol. abs/1909.12642, 2019\.
  * [32] S. Mishra, “3idiots at hasoc 2019: Fine-tuning transformer neural networks for hate speech identification in indo-european languages,” in _FIRE_ , 2019\.
  * [33] J. Gordon, J. Bronskill, M. Bauer, S. Nowozin, and R. E. Turner, “Meta-learning probabilistic inference for prediction,” _arXiv preprint arXiv:1805.09921_ , 2018.
  * [34] G. Koch, R. Zemel, R. Salakhutdinov _et al._ , “Siamese neural networks for one-shot image recognition,” in _ICML deep learning workshop_ , 2015\.
  * [35] O. Vinyals, C. Blundell, T. Lillicrap, k. kavukcuoglu, and D. Wierstra, in _NIPS_ , D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, Eds., vol. 29, 2016.
  * [36] M. Yu, X. Guo, J. Yi, S. Chang, S. Potdar, Y. Cheng, G. Tesauro, H. Wang, and B. Zhou, “Diverse few-shot text classification with multiple metrics,” in _NAACL_ , 2018, pp. 1206–1215.
  * [37] C. Han, Z. Fan, D. Zhang, M. Qiu, M. Gao, and A. Zhou, “Meta-learning adversarial domain adaptation network for few-shot text classification,” in _ACL-IJCNLP_ , Aug. 2021, pp. 1664–1673.
  * [38] Y. Bao, M. Wu, S. Chang, and R. Barzilay, “Few-shot text classification with distributional signatures,” in _ICLR_ , 2020.
  * [39] D. Li, Y. Yang, Y.-Z. Song, and T. Hospedales, “Learning to generalize: Meta-learning for domain generalization,” in _AAAI_ , 2018.
  * [40] M. Mozafari, R. Farahbakhsh, and N. Crespi, “Cross-lingual few-shot hate speech and offensive language detection using meta learning,” _IEEE Access_ , vol. 10, pp. 14 880–14 896, 2022.
  * [41] G. Vadakkekara Suresh, B. R. Chakravarthi, and J. P. McCrae, “Meta-learning for offensive language detection in code-mixed texts,” in _FIRE_. Association for Computing Machinery, 2022, p. 58–66.
  * [42] M. R. Awal, M. D. Nguyen, R. K.-W. Lee, and K. T. W. Choo, “Muscat: Multilingual rumor detection in social media conversations,” in _IEEE International Conference on Big Data (Big Data)_ , 2022, pp. 455–464.
  * [43] X. Wang, H. Pham, Z. Dai, and G. Neubig, “Switchout: an efficient data augmentation algorithm for neural machine translation,” in _EMNLP_ , 2018, pp. 856–861.
  * [44] D. Guo, Y. Kim, and A. Rush, “Sequence-level mixed sample data augmentation,” in _EMNLP_ , 2020, pp. 5547–5552.
  * [45] A. Jindal, A. Ghosh Chowdhury, A. Didolkar, D. Jin, R. Sawhney, and R. R. Shah, “Augmenting NLP models using latent feature interpolations,” in _International Conference on Computational Linguistics_ , 2020, pp. 6931–6936.
  * [46] R. Sennrich, B. Haddow, and A. Birch, “Improving neural machine translation models with monolingual data,” in _ACL_ , 2016, pp. 86–96.
  * [47] S. Y. Feng, A. W. Li, and J. Hoey, “Keep calm and switch on! preserving sentiment and fluency in semantic text exchange,” in _EMNLP-IJCNLP_ , 2019, pp. 2701–2711.
  * [48] S. Kobayashi, “Contextual augmentation: Data augmentation by words with paradigmatic relations,” in _NAACL_ , M. A. Walker, H. Ji, and A. Stent, Eds., 2018, pp. 452–457.
  * [49] S. Manuela, C. Gloria, E. Di Nuovo, S. Frenda, M. A. Stranisci, C. Bosco, C. Tommaso, V. Patti, R. Irene _et al._ , “Haspeede 2@ evalita2020: Overview of the evalita 2020 hate speech detection task,” in _EVALITA_ , 2020, pp. 1–9.

|  Rabiul Awal earned a Bachelor’s degree in Computer Science from Noakhali
Science and Technology University and a Master’s degree from the University of
Saskatchewan. His research focuses on the intersection of social NLP, deep
learning, and language-vision. Currently, he is a research intern at Mila,
Montreal.
---|---
|  Roy Ka-Wei Lee is an Assistant Professor at the Information Systems
Technology and Design Pillar, Singapore University of Technology and Design.
He is a faculty of the transformative Design and Artificial Intelligence
programme. His research lies in the intersection of data mining, machine
learning, social computing, and natural language processing. He is leading the
Social AI Studio, a research group that focuses on designing next-generation
social artificial intelligence systems. He has published in top-tier venues in
data mining and computation linguistics domains. He serves on the program
committees of multiple top data mining and natural language processing
conferences.
---|---
|  Eshaan Tanwar Eshaan Tanwar received his bachelor of technology from Delhi
Technological University, New Delhi, India. He is currently a Research
Associate in the Laboratory for Computational Social Systems (LCS2). His
research interests include Social Computing, Natural Language Processing, and
Machine Learning.
---|---
|  Tanmay Garg Tanmay Garg completed his Master’s in Technology from
Indiraprastha Institute of Information Technology, Delhi in 2019. He is
currently working as a Research Engineer. His research interests include Hate
Speech, Bias in Hate Speech and Natural Language Processing.
---|---
|  Tanmoy Chakraborty is an Associate Professor and a Ramanujan Fellow in the
Department of Electrical Engineering at the Indian Institute of Technology
Delhi (IIT Delhi), India since September 2022. Prior to joining IIT Delhi, he
held positions as an Assistant Professor (May 2017 - Dec 2021) and an
Associate Professor (Jan 2022 - Aug 2022) in the Department of CSE at IIIT
Delhi, India. He also served as the head of the Infosys Centre for AI (CAI) at
IIIT Delhi. Tanmoy Chakraborty earned his PhD as a Google PhD scholar from the
Department of CSE at IIT Kharagpur, India in September 2015. Following this,
he worked as a Postdoctoral Researcher at the University of Maryland, College
Park. He leads LCS2, a research group that broadly works in Natural Language
Processing, Graph Neural Networks, and Social Computing. He is a senior IEEE
member.
---|---