# Transferring Knowledge via Neighborhood-Aware Optimal Transport
for Low-Resource Hate Speech Detection

Tulika Bose Irina Illina Dominique Fohr
Universite de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France
{tulika.bose, illina<EMAIL_ADDRESS>

###### Abstract

_Warning: this paper contains content that may be offensive and distressing._

The concerning rise of hateful content on online platforms has increased the
attention towards automatic hate speech detection, commonly formulated as a
supervised classification task. State-of-the-art deep learning-based
approaches usually require a substantial amount of labeled resources for
training. However, annotating hate speech resources is expensive, time-
consuming, and often harmful to the annotators. This creates a pressing need
to transfer knowledge from the existing labeled resources to low-resource hate
speech corpora with the goal of improving system performance. For this,
neighborhood-based frameworks have been shown to be effective. However, they
have limited flexibility. In our paper, we propose a novel training strategy
that allows flexible modeling of the relative proximity of neighbors retrieved
from a resource-rich corpus to learn the amount of transfer. In particular, we
incorporate neighborhood information with Optimal Transport, which permits
exploiting the geometry of the data embedding space. By aligning the joint
embedding and label distributions of neighbors, we demonstrate substantial
improvements over strong baselines, in low-resource scenarios, on different
publicly available hate speech corpora.

##  1 Introduction

With the alarming spread of Hate Speech (HS) in social media, Natural language
Processing techniques have been used to develop automatic HS detection
systems, typically to aid manual content moderation. Although deep learning-
based approaches (Mozafari et al., 2019; Badjatiya et al., 2017) have become
state-of-the-art in this task, their performance depends on the size of the
labeled resources available for training (Lee et al., 2018; Alwosheel et al.,
2018).

Annotating a large corpus for HS is considerably time-consuming, expensive,
and harmful to human annotators (Schmidt and Wiegand, 2017; Malmasi and
Zampieri, 2018; Poletto et al., 2019; Sarwar et al., 2022). Moreover, models
trained on existing labeled HS corpora have shown poor generalization when
evaluated on new HS content (Yin and Zubiaga, 2021; Arango et al., 2019; Swamy
et al., 2019; Karan and Šnajder, 2018). This is due to the differences across
these corpora, such as sampling strategies (Wiegand et al., 2019), varied
topics of discussion (Florio et al., 2020; Saha and Sindhwani, 2012), varied
vocabularies, and different victims of hate. Thus, to address these
challenges, here we aim to devise a strategy that can effectively transfer
knowledge from a resource-rich source corpus with a higher amount of annotated
content to a low-resource target corpus with fewer labeled instances.

One popular way to address this is transfer learning. For instance, Mozafari
et al. (2019) fine-tune a large-scale pre-trained language model, BERT (Devlin
et al., 2019), on the limited training examples in HS corpora. Further, a
sequential transfer, following Garg et al. (2020), can be performed where a
pre-trained model is first fine-tuned on a resource-rich source corpus and
subsequently fine-tuned on the low-resource target corpus. Since this may risk
forgetting knowledge from the source, the source and target corpora can be
mixed for training (Shnarch et al., 2018). Besides, to learn target-specific
patterns without forgetting the source knowledge, Meftah et al. (2021) augment
pre-trained neurons from the source model with randomly initialized units for
transferring knowledge to low-resource domains.

Recently, Sarwar et al. (2022) argue that traditional transfer learning
strategies are not systematic. Therefore, they model the relationship between
a source and a target corpus with a neighborhood framework and show its
effectiveness in transfer learning for content flagging. They model the
interaction between a query instance from the target and its neighbors
retrieved from the source. This interaction is modeled based on their label
agreement – whether the query and its neighbors have the same labels – while
using a fixed neighborhood size. However, different neighbors may have varying
levels of proximity to the queried instance based on their pair-wise cosine
similarities in a sentence embedding space. Therefore, intuitively, the
neighbors should also be weighted according to these similarity scores.

We hypothesize that simultaneously modeling the pair-wise distances between
instances from the low-resource target and their respective neighbors from the
resource-rich source, along with their label distributions should result in a
more flexible and effective transfer. With this aim, we propose a novel
training strategy where the model learns to assign varying importance to the
neighbors corresponding to different target instances by optimizing the amount
of pair-wise transfer. This transfer is learned without changing the
underlying model architecture. Such optimization can be efficiently performed
using Optimal Transport (OT) (Peyré and Cuturi, 2019; Villani, 2009;
Kantorovich, 2006) due to its ability to find correspondences between
instances while exploiting the underlying geometry of the embedding space. Our
contributions are summarised as follows:

  * •

We address HS detection in low-resource scenarios with a flexible and
systematic transfer learning strategy.

  * •

We propose novel incorporation of neighborhood information with joint
distribution Optimal Transport. This enables learning of the amount of
transfer between pairs of source and target instances considering both (i) the
similarity scores of the neighbors and (ii) their associated labels. To the
best of our knowledge, this is the first work that introduces Optimal
Transport for HS detection.

  * •

We demonstrate the effectiveness of our approach through considerable
improvements over strong baselines, along with quantitative and qualitative
analysis on different HS corpora from varied platforms.

##  2 Related Works

###  2.1 Hate Speech Detection

Deep Neural Networks, especially the transformer-based models, such as the
pre-trained BERT, have dominated the field of HS detection in the past few
years (Alatawi et al., 2021; D’Sa et al., 2020; Glavaš et al., 2020; Mozafari
et al., 2019).

Wiegand et al. (2019); Arango et al. (2019) raise concerns about data bias
present in most HS corpora, which results in overestimated within-corpus
performance. They, therefore, recommend cross-corpus evaluations as more
realistic settings. Bigoulaeva et al. (2021); Bose et al. (2021); Pamungkas et
al. (2021) perform such cross-corpus evaluations in this task with no access
to labeled instances from the target. However, Yin and Zubiaga (2021); Wiegand
et al. (2019) report fluctuating or degraded performance across corpora. As
pointed out by Sarwar et al. (2022), in real-life scenarios, most online
platforms could invest in obtaining at least some labeled training instances
for deploying an HS detection system. Thus, we study a more realistic setting
where a limited amount of labeled content is available in the target corpus.

###  2.2 Neighborhood Framework

$k$-Nearest Neighbors (${k}$NN)-based approaches have been successfully used
in the literature for an array of tasks such as language modeling (Khandelwal
et al., 2020), question answering (Kassner and Schütze, 2020), dialogue
generation (Fan et al., 2021), etc. Besides, ${k}$NN classifiers have been
used for HS detection (Prasetyo and Samudra, 2022; Briliani et al., 2019),
which typically predict the class of an input instance through a simple
majority voting using its neighbors in the training data.

Recently, Sarwar et al. (2022) propose a neighborhood framework $k$NN+ for
transfer learning in cross-lingual low-resource settings. They show that a
simple kNN classifier is prone to prediction errors as the neighbors may have
similar meanings, but opposite labels. They, instead, model the interactions
between the target corpus instances, treated as queries, and their nearest
neighbors retrieved from the source. This neighborhood interaction is modeled
based on whether a query and its neighbors have the same or different labels.
In their best performing framework (in cross-lingual setting) of Cross-Encoder
$k$NN+, Sarwar et al. (2022) obtain representations of concatenated query-
neighbor pairs to learn such neighborhood interactions.

However, Sarwar et al. (2022) do not consider the varying levels of the
proximity of different neighbors to the query. Besides, a mini-batch in their
framework comprises a query and all its neighbors. For fine-tuning large
language models like BERT, the batch size needs to be kept small due to
resource constraints. This could limit the neighborhood size in their
framework. This is different from our approach, where the neighborhood size is
scalable.

###  2.3 Optimal Transport

Optimal Transport (OT) has become increasingly popular in diverse NLP
applications, as it allows comparing probability distributions in a
geometrically sound manner. These include machine translation (Xu et al.,
2021), interpretable semantic similarity (Lee et al., 2022), rationalizing
text matching (Swanson et al., 2020), etc. Moreover, OT has been successfully
used for domain adaptation in audio, images, and text (Olvera et al., 2021;
Damodaran et al., 2018; Chen et al., 2020). In this work, we perform novel
incorporation of nearest neighborhood information with OT. Besides, to the
best of our knowledge, this is the first work that introduces OT to the HS
detection task.

##  3 Proposed Approach

Our problem setting involves a low-resource target corpus $X^{t}$ with a
limited amount of labeled training data
$(X^{t}_{train},Y^{t}_{train})=\\{x_{i}^{t},y_{i}^{t}\\}_{i=1}^{n_{t}}$ and a
resource-rich source corpus $X^{s}$ from a different distribution with a large
number of annotated data
$(X^{s}_{train},Y^{s}_{train})=\\{x_{i}^{s},y_{i}^{s}\\}_{i=1}^{n_{s}}$. Given
such a setting, we hypothesize that transferring knowledge from the nearest
neighbors in the source should improve the performance on the insufficiently
labeled target. Furthermore, to provide additional control to the model, we
propose a systematic transfer. With this transfer mechanism, a model can learn
different weights assigned to the neighbors in $X^{s}_{train}$ based on their
proximity to the instances in $X^{t}_{train}$ simultaneously in a sentence
embedding space and the label space. For this, we incorporate neighborhood
information with Optimal Transport (OT), as OT can learn correspondences
between instances from $X^{s}_{train}$ and $X^{t}_{train}$ by exploiting the
underlying embedding space geometry.

###  3.1 Joint Distribution Optimal Transport

In this work, we use the joint distribution optimal transport (JDOT) framework
(Courty et al., 2017) following the works of Damodaran et al. (2018); Fatras
et al. (2021), proposed for unsupervised domain adaptation in deep embedding
spaces. The framework aligns the joint distribution $P(Z,Y)$ of the source and
the target domains, where $Z$ is the embedding space through a mapping
function $g(.)$, and $Y$ is the label space. For a discrete setting, let
$\mu_{s}=\sum_{i}^{n_{s}}a_{i}\,\delta_{g(x_{i}^{s}),y_{i}^{s}}$ and
$\mu_{t}=\sum_{i}^{n_{t}}b_{i}\,\delta_{g(x_{i}^{t}),y_{i}^{t}}$ be two
empirical distributions on the product space of $Z\times Y$. Here
$\delta_{g(x_{i}),y_{i}}$ is the Dirac function at the position
$(g(x_{i}),y_{i})$, and $a_{i}$, $b_{i}$ are uniform probability weights, i.e.
$\sum_{i}^{n_{s}}a_{i}=\sum_{i}^{n_{t}}b_{i}=1$.

The ‘balanced’ OT problem ($OT_{b}$), as defined by Kantorovich (2006), seeks
for a transport plan $\gamma$ in the space of the joint probability
distribution $\Pi(\mu_{s},\mu_{t})$, with marginals $\mu_{s}$ and $\mu_{t}$,
that minimizes the cost of transport from $\mu_{s}$ to $\mu_{t}$, as:

$\begin{aligned}
{OT_{b}}(\mu_{s},\mu_{t})&=\underset{\gamma\in\Pi(\mu_{s},\mu_{t})}{\mathrm{min}}\quad\underset{i,j}{\sum}\gamma_{i,j}c_{i,j}\\\
&s.t.\quad\gamma{\mathbf{1}}_{n_{t}}=\mu_{s},\gamma^{T}{\mathbf{1}}_{n_{s}}=\mu_{t}\end{aligned}$

(1)

Here $c_{i,j}$ is an entry in a cost matrix $C\in R^{n_{s}\times n_{t}}$,
representing the pair-wise cost (see Section 3.2), and $\mathbf{1}_{n}$ is a
vector of ones with dimension $n$. Each entry $\gamma_{i,j}$ indicates the
amount of transfer from location $i$ in the source to $j$ in the target.

The constraint on $\gamma$ requires that all mass from $\mu_{s}$ is
transported to $\mu_{t}$. However, this can be alleviated through relaxation,
leading to the ‘unbalanced’ OT ($OT_{u}$) (Benamou, 2003), as:

$\begin{aligned}
{OT_{u}}(\mu_{s},\mu_{t})&=\underset{\gamma\in\Pi(\mu_{s},\mu_{t})}{\mathrm{min}}\quad\underset{i,j}{\sum}\gamma_{i,j}c_{i,j}+\Lambda\,;\\\
&\text{where}\quad\Lambda=\epsilon\>\Omega(\gamma)+\lambda\left(\text{KL}(\gamma\mathbf{1}_{n_{t}},\mu_{s})+\text{KL}(\gamma^{T}\mathbf{1}_{n_{s}},\mu_{t})\right)\\\
&s.t.\quad\gamma\geq 0\end{aligned}$

(2)

KL is the Kullback-Leibler divergence that allows the relaxation of the
marginal constraint on $\gamma$. $\lambda$ is the marginal relaxation
coefficient. $\Omega(\gamma)=\sum_{i,j}\gamma_{i,j}log(\gamma_{i,j})$
corresponds to the entropic regularization term, which allows fast computation
of the OT distances (Cuturi, 2013). $\epsilon$ is the entropy coefficient.

For models with a high-dimensional embedding space like ours, Fatras et al.
(2021) propose to make the computation of OT losses scalable using the mini-
batch OT. Thus, for every mini-batch, we sample an equal number of instances,
given by the batch size $m$, from $X^{s}_{train}$ and $X^{t}_{train}$, which
makes $C\in R^{m\times m}$ and $\gamma$ square matrices. As discussed by
Fatras et al. (2021), since the transport plan at the mini-batch level is much
less sparse, it may result in undesired pairings between instances if computed
by Equation 1. To counteract this effect, we rely on the more robust version
of OT as formulated in Equation 2. Thus, we adopt the joint distribution
entropy regularized unbalanced mini-batch OT for our framework, henceforth
simply referred to as OT. Note that this framework does not modify the
underlying model architecture used for classification, but only introduces a
new training strategy.

###  3.2 Neighborhood-aware OT (OTNN)

In the above joint distribution framework, the cost matrix $C$ is expressed as
the weighted combination of the costs in the embedding and the label spaces:

$\textit{c}_{i,j}(g(x_{i}^{s}),y_{i}^{s};g(x_{j}^{t}),y_{j}^{t})=\alpha\>d(g(x_{i}^{s}),g(x_{j}^{t}))+\beta\>L(y_{i}^{s},y_{j}^{t})$

(3)

$d(.,.)$ denotes the embedding distance (ED), which is a squared $l_{2}$
distance between the corresponding embeddings. $L(.,.)$ is label-consistency
loss (LC), which is a cross-entropy loss that enforces a match between the
label of the $i^{th}$ source instance and that of the $j^{th}$ target
instance. $\alpha$ and $\beta$ are scalar values. Minimizing the cost in
Equation 3 results in aligning instances from the source and the target that
simultaneously share similar representations and common labels.

We adapt $C$ to account for $k$ nearest neighbors of the target instances in
$X^{t}_{train}$ from the source $X^{s}_{train}$. Since BERT is not optimal for
semantic similarity search (Reimers and Gurevych, 2019), we extract the
neighbors using the Sentence-BERT (SBERT) model (Reimers and Gurevych, 2019).
SBERT provides sentence embeddings that can be easily compared using cosine
similarity. We hypothesize that allowing transfers to occur only from the
corresponding neighbors in the source to the target should result in more
effective learning.

For this, we explicitly assign the value $\mathrm{max}(C)$ to $c_{i,j}$ in $C$
whenever the $i^{th}$ source and $j^{th}$ target instances are not neighbors,
considering the nearest neighborhood space of $k$ neighbors. Besides, we use
the SBERT distances as the embedding distance in Equation 3. This distance, in
addition to the label consistency term, ensures that $\gamma$ is learned to
allow a higher amount of transfer from neighbors in $X^{s}_{train}$ that are
simultaneously (i) closer in the SBERT space and (ii) share the same label
with an instance in $X^{t}_{train}$, compared to the neighbors that are
further away and/or have opposite labels.

Note that even though we use a neighborhood size of $k$, the target instances
do not attend equally to all of their $k$ neighbors. This is because if the
distance between a target instance $x_{j}^{t}$ and its top $n^{th}$ neighbor
($x_{i}^{s}$) from the source, within the neighborhood size of $k$ (i.e. $n$ <
$k$) is comparatively large, their corresponding $(i,j)\mbox{-}th$ entry in
$C$ would have a larger value. This would comparatively reduce the transfer
even if they share common labels. Thus, for a neighbor with the same label as
the target instance, the higher its SBERT distance from the target instance,
the lower the amount of transfer. This results in more flexibility where the
model can learn from the relevant neighbors corresponding to every target
instance.

In addition to the OT loss from Equation 2, we introduce the cross-entropy
losses for the training instances from both $X^{t}_{train}$ and
$X^{s}_{train}$ in the final loss function, as required by our classification
task. Our final loss function is given by Equation 4. Here $g(.)$ encodes a
given input using the pre-trained BERT encoder to the BERT embedding space by
extracting the fine-tuned [CLS] token representation of the last hidden layer.
$f(.)$ denotes the classifier, which is one fully connected layer.
$\theta_{s}$ and $\theta_{t}$ are the weights assigned to the source and the
target cross-entropy losses, respectively.

$\begin{split}\mathrm{OT}^{NN}=\underset{\gamma,f,g}{\mathrm{min}}\quad&\theta_{s}\frac{1}{m}\underset{i}{\sum}L_{s}\left(y_{i}^{s},f(g(x_{i}^{s}))\right)+\underset{i,j}{\sum}\gamma_{i,j}c_{i,j}\\\
&+\Lambda+\theta_{t}\frac{1}{m}\underset{j}{\sum}L_{t}\left(y_{j}^{t},f(g(x_{j}^{t}))\right)\end{split}$

(4)

#### Solving the optimization problem:

Following Damodaran et al. (2018), we adopt a two-step procedure to solve the
above optimization problem at the mini-batch level. We first compute the
optimal $\gamma$ by fixing the model parameters of $f$ and $g$.

$\underset{\gamma}{\mathrm{min}}\underset{i,j}{\sum}\gamma_{i,j}\left(\alpha\,d({g_{sbert}}(x_{i}^{s}),{g_{sbert}}(x_{j}^{t}))+\beta\,L(y_{i}^{s},y_{j}^{t})\right)+\Lambda$

(5)

We use the SBERT embeddings through the mapping function $g_{sbert}(.)$ here
instead of the learned BERT embeddings to compute the ED loss. This is done so
that the $\gamma$ is updated based on the semantic proximity in the SBERT
space. $y_{i}^{s}$ and $y_{j}^{t}$ are the ground truth labels for the
instances $x_{i}^{s}$ and $x_{j}^{t}$ from $X^{s}_{train}$ and
$X^{t}_{train}$, respectively. In the next step, the model parameters of $f$
and $g$ are learned while fixing $\gamma$ obtained from Equation 5, denoted as
$\hat{\gamma}$.

$\begin{aligned}
\underset{f,g}{\mathrm{min}}\quad&\underset{i,j}{\sum}\hat{\gamma}_{i,j}\left(\alpha\>d(g(x_{i}^{s}),g(x_{j}^{t}))+\beta\>L(f(g(x_{i}^{s})),y_{j}^{t})\right)\\\
&+\theta_{s}\frac{1}{m}\underset{i}{\sum}L_{s}\left(y_{i}^{s},f(g(x_{i}^{s}))\right)+\theta_{t}\frac{1}{m}\underset{j}{\sum}L_{t}\left(y_{j}^{t},f(g(x_{j}^{t}))\right)\end{aligned}$

(6)

The first part of Equation 6 allows the model to learn from the instances in
$X^{s}_{train}$ that are consistent in terms of both the embedding space (ED
loss) and the label space (LC loss) with the instances in $X^{t}_{train}$.
Here we use $g(.)$, instead of $g_{sbert}(.)$, to compute ED so that $g$
learns from the SBERT space through $\hat{\gamma}$. For the LC loss, we use
the predicted labels for $x_{i}^{s}$ from the source and the actual labels
$y_{j}^{t}$ corresponding to $x_{j}^{t}$ from the target. This is done to
update the model parameters of $f$ and $g$ based on the target labels and
bring source instances that have common labels closer to the target instances.
We have provided an illustration of the training strategy of OTNN in Figure 3
of Appendix A.

We propose different variants of OTNN:

####  $\bm{\mathrm{OT}}^{\bm{NN}}$:

In this variant, we do not use the source cross-entropy loss term in Equation
4, thus effectively having $\theta_{s}=0$.

####  $\bm{\mathrm{OT}^{NN}}_{\bm{\mathrm{pre\mbox{-}select}}}$:

Prior to the training, we pre-select the $k$ nearest neighbors from
$X^{s}_{train}$ corresponding to every instance in $X^{t}_{train}$, instead of
training with all the source instances. Here also $\theta_{s}=0$.

####  $\bm{\mathrm{OT}}^{\bm{NN}}$ \+ sloss:

This is OTNN with source cross-entropy loss (sloss), thus having
$\theta_{s}=1$.

####  $\bm{\mathrm{OT}^{NN}}_{\bm{\mathrm{pre\mbox{-}select}}}$ \+ sloss:

This is similar to the second variant, with $\theta_{s}=1$. Here, sloss is
computed only on the pre-selected source instances.

##  4 Experimental Settings

###  4.1 Corpus Description

We perform experiments with three standard HS corpora, namely, Waseem (Waseem
and Hovy, 2016), Vidgen (Vidgen et al., 2021), and Ethos (Mollas et al.,
2022), as they are collected using different sampling strategies across varied
platforms. Following Wiegand et al. (2019); Swamy et al. (2019), we use the
labels of hate and non-hate, where the former involves all forms of hate.

Waseem is a Twitter corpus comprising hate against women and ethnic
minorities. We obtain 10.9K tweets in total from the tweet IDs, of which 26.8%
instances belong to the hate class. Vidgen is collected using a human-and-
model-in-the-loop process aimed at making the corpus robust. It covers hate
against diverse social groups, like blacks, women, muslims, immigrants, etc.
with a total of 41144 instances, of which 53.9% is labeled as hate. Ethos
comprises 998 instances from YouTube and Reddit, of which 43.4% are hate
instances. Even with fewer instances, it is made diverse with an active
learning-based sampling strategy, ensuring a balance with respect to different
hateful aspects. See Appendix B for further details on the corpora.

For our experiments, we create two different versions of every corpus
depending on its use as the source or the target, as presented in Table 1.

Corpus |  Number of comments
---|---
Source setting
| Train
Waseemsrc | 8720
Vidgensrc | 32924
Ethossrc | 998
Target setting
|  Train |  Validation |  Test
Waseemtar |  400 |  100 |  1090
Vidgentar |  400 |  100 |  4120
Ethostar |  400 |  100 |  200

Table 1:  Corpus statistics.

.

#### Source setting:

In the absence of available standard splits, we randomly sample 80% of Waseem
as the train set, resulting in 8720 instances. For Vidgen, we use the original
corpus-provided train split of 32924 instances. Since Ethos has a relatively
small size, we use the entire corpus for training, when used as the source. We
call the source versions of these corpora as $\textit{Waseem}_{\mathrm{src}}$,
$\textit{Vidgen}_{\mathrm{src}}$ and $\textit{Ethos}_{\mathrm{src}}$. Note
that the source corpus is only used for training, while its validation set is
not used for our experiments. Instead, we use the corresponding validation and
test sets of the low-resource target corpus.

#### Target setting:

In order to simulate a low-resource scenario for the target, we down-sample
the original training instances of the corpora to 500 instances. This yields
three low-resource target corpora, namely, $\textit{Waseem}_{\mathrm{tar}}$,
$\textit{Vidgen}_{\mathrm{tar}}$ and $\textit{Ethos}_{\mathrm{tar}}$.
Furthermore, we split each of them in the 80-20 ratio to obtain their
respective low-resource train (400) and validation (100) sets. For the test
set from $\textit{Waseem}_{\mathrm{tar}}$, we sample 10% of the original data,
disjoint from the train and validation sets, given by 1090 instances. We use
the original test split of 4120 instances for
$\textit{Vidgen}_{\mathrm{tar}}$. For $\textit{Ethos}_{\mathrm{tar}}$, we
randomly sample 20% of the data, disjoint from the previous set of 500
instances, as the test set.

###  4.2 Baselines

We compare our approach with the following baseline approaches:

#### Target-FT:

We fine-tune the pre-trained BERT on the train set of the low-resource target
corpus.

#### Seq-FT:

Here, we sequentially fine-tune the BERT model first on the resource-rich
source corpus and then on the low-resource target corpus.

#### Mixed-FT:

Here, we fine-tune BERT on a mix of the source and target corpora. Since the
target instances are limited, we first over-sample them. Then, for every mini-
batch of size $m$, we randomly sample $m$ training instances each from the
source and the target. We then combine their cross-entropy losses for updating
the model parameters, as:

$\underset{f,g}{\mathrm{min}}\quad\theta_{s}\frac{1}{m}\underset{i}{\sum}L_{s}(y_{i}^{s},f(g(x_{i}^{s})))+\theta_{t}\frac{1}{m}\underset{j}{\sum}L_{t}(y_{j}^{t},f(g(x_{j}^{t})))$

(7)

This is similar to Equation 4 without the OTNN losses.

####  kNN-FT:

For every target instance, we retrieve top-$k$ neighbors from the source,
ranked with cosine similarities over SBERT embeddings. This yields a subset of
source instances that are neighbors to the target instances. We then fine-tune
the BERT model with the strategy used for Mixed-FT.

####  kNN ranking:

Here, we predict the labels of the target instances using a majority voting
strategy. This voting is done over the labels associated with the top-k
retrieved neighbors from the source based on their cosine similarities.

#### Weighted kNN:

This uses a weighted voting of the top-k neighbors. Here we compute the sum of
cosine similarities of neighbors associated with every class. The class with
the highest score is returned as the predicted label of the target instance.

#### CE kNN+ \+ SRC:

This is the Cross-Encoder-based neighborhood framework kNN+, proposed by
Sarwar et al. (2022), as discussed in Section 2.2. For a fair comparison, we
use the pre-trained BERT as the base representation. We first train CE kNN+ on
the source (SRC) and then with the target instances and their neighbors from
the source.

#### PretRand:

This is a transfer learning strategy proposed by Meftah et al. (2021) for low-
resource domain adaptation. They jointly learn a pre-trained branch in the
target model with a normalized, weighted, and randomly initialized branch.
This is done so that the model can learn target-specific patterns while
retaining the source knowledge. For a fair comparison, we use the pre-trained
BERT as the base model, which is first fine-tuned on the source. For the
random branch, following the approach, we add a BiLSTM layer and a Fully
Connected layer over the final hidden layer from BERT. The final predictions
are obtained using an element-wise sum of the predictions from the two
branches.

#### OT:

Finally, we use OT to transfer knowledge from the source to the target using
both the ED and LC losses, similar to Equation 4. However, this is done
without incorporating any neighborhood information in both the cost matrix and
the computation of $\gamma$.

Target corpus | Waseemtar | Vidgentar | Ethostar
---|---|---|---
Target-FT | 64.0$\pm$2.1 | 68.8$\pm$3.2 | 69.6$\pm$6.4
Source corpus |  Vidgensrc |  Ethossrc |  Waseemsrc |  Ethossrc |  Vidgensrc |  Waseemsrc
Seq-FT |  63.2$\pm$2.1 |  65.0$\pm$1.1 |  67.0$\pm$2.2 |  70.8$\pm$3.9 |  79.8$\pm$0.7 |  70.2$\pm$3.1
Mixed-FT |  61.2$\pm$2.7 |  66.6$\pm$2.2 |  69.8$\mbox{*}\pm$1.6 |  71.4$\pm$3.9 |  77.6$\pm$2.1 |  71.8$\pm$3.5
kNN-FT |  62.2$\pm$1.2 |  65.6$\pm$0.8 |  69.4$\mbox{*}\pm$2.3 |  70.8$\pm$1.9 |  77.2$\pm$1.5 |  70.6$\pm$3.4
kNN ranking |  57.0 |  60.0 |  40.0 |  73.0* |  77.0 |  49.0
Weighted kNN |  57.0 |  60.0 |  37.0 |  73.0* |  77.0 |  47.0
CE kNN+ \+ SRC |  59.8$\pm$1.8 |  68.4$\mbox{*}\pm$0.8 |  65.6$\pm$1.6 |  68.8$\pm$3.9 |  76.8$\pm$0.7 |  67.6$\pm$2.8
PretRand |  59.6$\pm$5.1 |  63.2$\pm$2.9 |  71.0$\mbox{*}\pm$0.6 |  72.2$\mbox{*}\pm$2.0 |  77.6$\pm$2.2 |  71.4$\pm$3.7
OT |  65.4$\mbox{*}\pm$1.5 |  66.6$\pm$1.0 |  70.0$\mbox{*}\pm$2.8 |  71.4$\pm$5.2 |  73.6$\pm$3.6 |  74.6$\mbox{*}\pm$2.9
OTNN |  65.6$\mbox{*}\pm$2.9 |  67.4$\mbox{*}\pm$1.6 |  71.6$\mbox{*}\pm$1.4 |  73.2$\mbox{*}\pm$0.7 |  73.8$\pm$2.3 |  72.6$\mbox{*}\pm$3.1
OT${}^{NN}_{\mathrm{pre\mbox{-}select}}$ |  64.2$\pm$1.5 |  67.0$\pm$2.1 |  71.6$\mbox{*}\pm$2.7 |  72.6$\mbox{*}\pm$1.0 |  75.4$\pm$1.4 |  73.2$\mbox{*}\pm$1.9
OTNN \+ sloss |  62.8$\pm$2.2 |  68.4$\mbox{*}\pm$0.8 |  69.2$\mbox{*}\pm$3.2 |  73.8$\mbox{*}\pm$1.6 |  76.8$\pm$1.9 |  73.4$\mbox{*}\pm$0.8
OT${}^{NN}_{\mathrm{pre\mbox{-}select}}$ \+ sloss |  65.2$\mbox{*}\pm$1.7 |  66.6$\pm$1.6 |  70.2$\mbox{*}\pm$3.7 |  72.2$\mbox{*}\pm$1.3 |  77.2$\pm$1.3 |  74.6$\mbox{*}\pm$2.5
Table 2:  F1 score ($\pm$std-dev) on the target corpus. The last four are the
proposed OTNN variants. Bold denotes the best, underline denotes the second-
best scores in each column. * denotes the significantly improved scores
compared to Seq-FT using the McNemar test (Dror et al., 2018; McNemar, 1947).

###  4.3 Hyper-parameters

We train all the models for 10 epochs initialized with the pre-trained BERT-
base (Devlin et al., 2019) uncased model (Wolf et al., 2020), with a maximum
sequence length of 128 tokens. We use the Adam optimizer with a learning rate
of $5\times 10^{-5}$. Besides, we perform hyper-parameter tuning for $k$ and
model selection using the best F1 scores over the respective target corpus
validation sets. After the preliminary experiments, we set $\alpha$ = 0.05,
$\beta$ = 10, $\epsilon$ = 0.2, $\lambda$ = 0.5, and $\theta_{t}$ = 10 for all
our experiments. We use a batch size of 32 for the OTNN and the baselines,
except CE-kNN+. The latter inherently requires the batch size to be equal to
the neighborhood size, as it provides query-neighborhood pairs as inputs to
the model. See Appendix D for further details on the hyper-parameter tuning.

##  5 Results

###  5.1 Discussion

Table 2 shows the performance obtained with the baselines and the OTNN
variants across the test sets of three low-resource target corpora using
different resource-rich source corpora. We also present the performance with
Target-FT for reference. Following the prior work on HS detection (Sarwar et
al., 2022; Attanasio et al., 2022), we use the F1 score of the hate class to
report the performance, with an average F1 computed over five runs of the same
experiments with different random initializations.

The results show that transferring knowledge from a resource-rich corpus to a
low-resource corpus is generally helpful. The best scores in the six
respective settings of Table 2 are substantially higher than those from
Target-FT. Furthermore, while the baseline methods show inconsistent
performance across different settings, the proposed OTNN variants yield the
best performance in five out of six cases and the second-best in three cases.
The baselines of Mixed-FT, $k$NN variants and CE $k$NN+ achieve significant
improvements compared to the vanilla Seq-FT for only 1 case, and PretRand
achieves it for 2 cases. OTNN variants, on the other hand, yield significant
improvements in most cases; for instance, OTNN has significantly improved
scores in 5 out of 6 cases. Besides, the best scores from OTNN variants
improve over OT in 5 settings, while staying on par with OT in the remaining
setting. This demonstrates that incorporating neighborhood information results
in a more effective transfer.

When $\textit{Vidgen}_{\mathrm{src}}$ is used for transferring knowledge to
$\textit{Ethos}_{\mathrm{tar}}$, Seq-FT yields the highest score (79.8). This
is apparently because $\textit{Vidgen}_{\mathrm{src}}$ comprises a wide range
of hateful forms directed towards different social groups. Since
$\textit{Ethos}_{\mathrm{tar}}$ also involves hate against a variety of social
groups, pre-training on all the source instances from
$\textit{Vidgen}_{\mathrm{src}}$ for transfer learning, instead of training
with the nearest neighbors, seems to be more helpful in this case. However,
this is not the case when the transfer occurs from
$\textit{Ethos}_{\mathrm{src}}$ to $\textit{Vidgen}_{\mathrm{tar}}$. This is
likely because the Vidgen corpus involves adversarial instances that can
easily fool an HS detection system trained on a different corpus. Besides,
$\textit{Ethos}_{\mathrm{src}}$ has a subset of hateful forms and social
groups covered by Vidgen. Therefore, a nearest neighborhood framework for
transferring knowledge from $\textit{Ethos}_{\mathrm{src}}$ to
$\textit{Vidgen}_{\mathrm{tar}}$ yields an improved performance, the highest
score being 73.8 obtained by OTNN \+ sloss, compared to 70.8 from Seq-FT.

240 (300)400 (500)560 (700)720 (900)$60$$65$$70$$75$$80$(a) Ethossrc to
VidgentarF1 scoreTarget-FTMixed-FTOTNN \+ sloss

240 (300)400 (500)560 (700)$60$$65$$70$$75$$80$(b) Waseemsrc to EthostarF1
scoreTarget-FTMixed-FTOT${}^{NN}_{\mathrm{pre\mbox{-}select}}$ \+ sloss

Figure 1:  Performance with different sizes of the target train set. The total number of labeled instances available from the target is mentioned within the brackets, where the remaining instances are used as the target validation set. Target corpus | Waseemtar | Vidgentar | Ethostar
---|---|---|---
Source corpus |  Vidgensrc |  Ethossrc |  Waseemsrc |  Ethossrc |  Vidgensrc |  Waseemsrc
OTNN \+ sloss |  62.8$\pm$2.2 |  68.4$\pm$0.8 |  69.2$\pm$3.2 |  73.8$\pm$1.6 |  76.8$\pm$1.9 |  73.4$\pm$0.8
OTNN \+ sloss (without ED) |  63.8$\pm$1.3 |  65.8$\pm$1.7 |  68.0$\pm$0.0 |  70.0$\pm$2.4 |  76.4$\pm$0.8 |  71.8$\pm$2.5
OTNN \+ sloss (without LC) |  62.0$\pm$2.1 |  66.4$\pm$2.2 |  67.6$\pm$2.7 |  72.4$\pm$1.4 |  75.2$\pm$2.6 |  67.8$\pm$3.9
OT${}^{NN}_{\mathrm{pre\mbox{-}select}}$ \+ sloss |  65.2$\pm$1.7 |  66.6$\pm$1.6 |  70.2$\pm$3.7 |  72.2$\pm$1.3 |  77.2$\pm$1.3 |  74.6$\pm$2.5
OT${}^{NN}_{\mathrm{pre\mbox{-}select}}$ \+ sloss (without ED) |  64.4$\pm$1.5 |  67.6$\pm$1.4 |  67.6$\pm$4.3 |  70.8$\pm$2.3 |  75.6$\pm$2.7 |  74.2$\pm$5.6
OT${}^{NN}_{\mathrm{pre\mbox{-}select}}$ \+ sloss (without LC) |  62.2$\pm$2.6 |  63.8$\pm$1.5 |  67.2$\pm$5.0 |  71.8$\pm$1.5 |  74.6$\pm$4.1 |  67.2$\pm$5.2
Table 3:  Ablation study without the Embedding Distance (ED) /Label
Consistency (LC) losses. F1 ($\pm$std-dev) on low-resource target corpus. Bold
denotes the best, underline denotes the second-best score for each OTNN
variant.

#### Varying the size of $\bm{X^{t}}$:

We vary the size of the labeled target corpus available for training. We
illustrate the cases of transferring knowledge from
$\textit{Ethos}_{\mathrm{src}}$ to $\textit{Vidgen}_{\mathrm{tar}}$ in Figure
1(a), and from $\textit{Waseem}_{\mathrm{src}}$ to
$\textit{Ethos}_{\mathrm{tar}}$ in Figure 1(b), with different OTNN variants.
For $\textit{Vidgen}_{\mathrm{tar}}$, we sample 300, 500, 700, and 900
instances. We use 80% for training, given by 240, 400, 560, and 720 instances,
respectively, and the remaining 20% for validation. Since the $Ethos$ corpus
is small, we sample only 300, 500, and 700 instances as
$\textit{Ethos}_{\mathrm{tar}}$, with the same proportions for training and
validation. The target test set remains the same as in Table 1 for different
training sizes. We observe that the OTNN variants consistently improve the
performance, with larger improvements obtained when the size of available
target instances is lower. Mixed-FT, on the other hand, is inconsistent, and
in some cases performs worse than Target-FT.

The improvements with OTNN can be attributed to the fact that it can
systematically learn the amount of transfer based on both the embedding
distance and label consistency.

Target instance: rt i am not sexist but girls who smoke are a big turn off
even if your supermodel status its still a turn off.
---
Instances | GT
Nearest neighbors from source using SBERT
i do not actually care much for looks but the inflated sense of self importance that girls seem to have nowadays is rather obnoxious. |  hate
totally agree. i could never find a woman or man attractive to go out with. …. i could nt be in a relationship with them as i am aromantic. |  non-hate
every guy i know at work smoke at least 10 fags a day and i call them out. do not like it then do not do it. |  non-hate
wow the truth is i did not get turned on until you mentioned cigar smoking men pissing on other men |  non-hate
women have no redeeming features |  hate
i am not sexist but women are dumb |  hate
weed is stoner trash. if you are not a stoner guy, smoking this estrogen-filled garbage makes you a honorary one. |  non-hate
maybe i am sexist because women are dumb |  hate
women are less attracted to looks. they are far more attracted to high status and a dominant personality. the good news is you can acquire both of these. |  non-hate
women without big t*ts are not attractive |  hate
Nearest neighbors from source using OTNN
i am not sexist but women are dumb |  hate
maybe i am sexist because women are dumb |  hate
thrilled to have been born as the superior male gender |  hate
men are superior than women in every single aspect |  hate
women of color are not pleasant to look at |  hate
leadership positions should only be occupied by men. that is the way to get things done. |  hate
all women are created as equal b*tches |  hate
all women should be suppressed |  hate
this is the fact feminism is girlf*rts you cant unsmell |  hate
women are devious |  hate
Table 4:  Qualitative analysis of an example with its top 10 nearest neighbors
extracted using the SBERT and the learned OTNN representations, where the
source is Vidgen${}_{\text{src}}$ and the target is Waseem${}_{\text{tar}}$;
GT: Ground Truth class.

$20$$40$$60$$80$$100$$69$$70$$71$$72$$73$$\\#$neighborsF1
score$\textit{Ethos}_{\text{src}}$ to
$\textit{Vidgen}_{\text{tar}}$$20$$40$$60$$80$$100$$50$$60$$70$$\\#$neighborsF1
score$\textit{Waseem}_{\text{src}}$ to
$\textit{Ethos}_{\text{tar}}$$20$$40$$60$$80$$100$$50$$55$$60$$65$$\\#$neighborsF1
score$\textit{Vidgen}_{\text{src}}$ to $\textit{Waseem}_{\text{tar}}$SBERT
${k}$NN rankingOTNN ${k}$NN ranking

Figure 2:  F1 using the majority voting of the $k$-Nearest Neighbors retrieved
from SBERT and OTNN representations.

###  5.2 Ablation Study

To analyze the importance of the constituent losses in OTNN, we present an
ablation study in Table 3 by removing the ED and the LC losses from the OTNN
variants. The best performances for each variant are obtained in 5 out of 6
cases when both the ED and the LC losses are incorporated. Besides, the
second-best performances are obtained, in most cases, when we remove the ED
loss. This suggests that while both losses are essential for an effective
transfer, the LC loss contributes more towards the final performance than the
ED loss.

###  5.3 Analysis of OTNN Representations

We analyze the effect of training with OTNN on the representation space by
extracting the nearest neighbors of target instances. We rank these neighbors
with cosine similarity over the learned OTNN representations and check their
ground truth classes. We compare them with the nearest neighbors obtained
using SBERT representations. Table 4 contains an example of a hateful instance
from Waseem${}_{\text{tar}}$, and its top 10 nearest neighbors from
Vidgen${}_{\text{src}}$. We observe that the neighbors retrieved using the
SBERT representations belong to both hate and non-hate classes. This is
because SBERT is optimized mainly for semantic similarity, while they are sub-
optimal in differentiating hateful instances from non-hateful ones. On the
other hand, the neighbors obtained from OTNN representations indicate that
OTNN brings instances across corpora, which are both semantically similar (the
topic of women) and belong to the same class closer in the representation
space, compared to those belonging to the opposite class.

In addition, we study the effect of the OTNN representations by performing a
simple majority voting of the top $k$ nearest neighbors retrieved from the
source with SBERT versus OTNN. Figure 2 demonstrates the performance obtained
on the target test set. Here the neighbors from the two representation spaces
are ranked using cosine similarities. We can see that majority voting using
the OTNN representations achieves higher performance compared to that using
the SBERT representations for different numbers of neighbors.

##  6 Conclusion and Future Work

In this work, we proposed a framework for transferring knowledge to a low-
resource HS corpus by incorporating neighborhood information with Optimal
Transport. It allowed the model to flexibly learn the amount of transfer from
the nearest neighbors based both on their proximity in a sentence embedding
space and label consistency. Our framework yielded substantial improvements
across HS corpora from varied platforms in low-resource settings. Besides, the
qualitative analysis of its learned representations demonstrated that they
incorporate both semantic and label similarities. This is different from
sentence embedding representations, where semantically similar instances may
have opposite labels.

Since our framework uses neighborhood information for transferring knowledge,
it relies on the degree of proximity of the neighbors. However, if all of the
source and target instances are very distant semantically, all the nearest
neighbors from the source may have very low cosine similarity to the
corresponding target instances. In such scenarios, the framework may yield
limited improvements over the vanilla fine-tuning as the available
neighborhood information would be much weaker. In such cases, the performance
would mainly depend on the label consistency of the neighbors.

For future work, our framework can be explored for transferring knowledge from
resource-rich languages, such as English, to low-resource languages. This can
be done by extracting the cross-lingual neighbors using multilingual sentence
embedding models like LaBSE (Feng et al., 2022). Besides, the framework can be
applied for transferring knowledge in other text classification tasks, such as
sentiment classification, bragging detection (Jin et al., 2022), etc., as the
methodology is not restricted to only hate speech detection.

## Ethical Considerations

The proposed approach intends to support more robust detection of online hate
speech that can use the existing annotated resources for transferring
knowledge to a resource with limited annotations. We acknowledge that
annotating hateful content can have negative effects on the mental health of
the annotators. The corpora used in this work are publicly available and cited
appropriately in this paper. The authors of the respective corpora have
provided detailed information about the sampling strategies, data collection
process, annotation guidelines, and annotation procedure in peer-reviewed
articles. Besides, the hateful terms and slurs presented in the work are only
intended to give better insights into the models for research purposes.

## Acknowledgements

This work was supported partly by the french PIA project “Lorraine Université
d’Excellence”, reference ANR-15-IDEX-04-LUE. Experiments presented in this
article were carried out using the Grid’5000 testbed, supported by a
scientific interest group hosted by Inria and including CNRS, RENATER and
several Universities as well as other organizations (see
https://www.grid5000.fr). We are extremely grateful to Claire Gardent for
taking time out to review the paper internally and Michel Olvera for his very
helpful feedbacks regarding the work and for internally reviewing the paper.
We would also like to thank the anonymous reviewers for their valuable
feedbacks and suggestions.

## References

  * Alatawi et al. (2021) Hind S. Alatawi, Areej Alhothali, and Kawthar Moria. 2021.  Detection of hate speech using bert and hate speech word embedding with deep model.  _ArXiv_ , abs/2111.01515.
  * Alwosheel et al. (2018) Ahmad Alwosheel, Sander van Cranenburgh, and Caspar G Chorus. 2018.  Is your dataset big enough? Sample size requirements when using artificial neural networks for discrete choice analysis.  _Journal of Choice Modelling_ , 28:167–182.
  * Arango et al. (2019) Aymé Arango, Jorge Pérez, and Barbara Poblete. 2019.  Hate speech detection is not as easy as you may think: A closer look at model validation.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR’19, page 45–54, New York, NY, USA. Association for Computing Machinery.
  * Attanasio et al. (2022) Giuseppe Attanasio, Debora Nozza, Dirk Hovy, and Elena Baralis. 2022.  Entropy-based attention regularization frees unintended bias mitigation from lists.  In _Findings of the Association for Computational Linguistics: ACL2022 (Forthcoming)_. Association for Computational Linguistics.
  * Badjatiya et al. (2017) Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, and Vasudeva Varma. 2017.  Deep learning for hate speech detection in tweets.  In _Proceedings of the 26th International Conference on World Wide Web Companion_ , WWW ’17 Companion, page 759–760, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  * Benamou (2003) Jean-David Benamou. 2003.  Numerical resolution of an “unbalanced” mass transport problem.  _ESAIM: Mathematical Modelling and Numerical Analysis - Modélisation Mathématique et Analyse Numérique_ , 37(5):851–868.
  * Bigoulaeva et al. (2021) Irina Bigoulaeva, Viktor Hangya, and Alexander Fraser. 2021.  Cross-lingual transfer learning for hate speech detection.  In _Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion_ , pages 15–25, Kyiv. Association for Computational Linguistics.
  * Bose et al. (2021) Tulika Bose, Irina Illina, and Dominique Fohr. 2021.  Generalisability of topic models in cross-corpora abusive language detection.  In _Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda_ , pages 51–56, Online. Association for Computational Linguistics.
  * Briliani et al. (2019) Annisa Briliani, Budhi Irawan, and Casi Setianingsih. 2019.  Hate speech detection in indonesian language on instagram comment section using k-nearest neighbor classification method.  _2019 IEEE International Conference on Internet of Things and Intelligence System (IoTaIS)_ , pages 98–104.
  * Chen et al. (2020) Liqun Chen, Zhe Gan, Yu Cheng, Linjie Li, Lawrence Carin, and Jingjing Liu. 2020\.  Graph optimal transport for cross-domain alignment.  In _International Conference on Machine Learning_ , pages 1542–1553. PMLR.
  * Courty et al. (2017) Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. 2017\.  Joint distribution optimal transportation for domain adaptation.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, page 3733–3742, Red Hook, NY, USA. Curran Associates Inc.
  * Cuturi (2013) Marco Cuturi. 2013.  Sinkhorn distances: Lightspeed computation of optimal transport.  In _Advances in Neural Information Processing Systems_ , volume 26. Curran Associates, Inc.
  * Damodaran et al. (2018) Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nicolas Courty. 2018.  DeepJDOT: Deep joint distribution optimal transport for unsupervised domain adaptation.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 447–463.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dror et al. (2018) Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018.  The hitchhiker’s guide to testing statistical significance in natural language processing.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1383–1392, Melbourne, Australia. Association for Computational Linguistics.
  * D’Sa et al. (2020) Ashwin Geet D’Sa, Irina Illina, and D. Fohr. 2020.  Bert and fastText embeddings for automatic detection of toxic speech.  In _2020 International Multi-Conference on: “Organization of Knowledge and Advanced Technologies” (OCTA)_ , pages 1–5.
  * Fan et al. (2021) Angela Fan, Claire Gardent, Chloé Braud, and Antoine Bordes. 2021.  Augmenting transformers with KNN-based composite memory for dialog.  _Transactions of the Association for Computational Linguistics_ , 9:82–99.
  * Fatras et al. (2021) Kilian Fatras, Thibault Séjourné, Rémi Flamary, and Nicolas Courty. 2021\.  Unbalanced minibatch optimal transport; Applications to domain adaptation.  In _International Conference on Machine Learning_ , pages 3186–3197. PMLR.
  * Feng et al. (2022) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022\.  Language-agnostic BERT sentence embedding.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 878–891, Dublin, Ireland. Association for Computational Linguistics.
  * Flamary et al. (2021) Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, Ievgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. 2021.  POT: Python optimal transport.  _Journal of Machine Learning Research_ , 22(78):1–8.
  * Florio et al. (2020) Komal Florio, Valerio Basile, Marco Polignano, Pierpaolo Basile, and Viviana Patti. 2020.  Time of your hate: The challenge of time in hate speech detection on social media.  _Applied Sciences_ , 10(12).
  * Garg et al. (2020) Siddhant Garg, Thuy Vu, and Alessandro Moschitti. 2020.  TANDA: Transfer and adapt pre-trained transformer models for answer sentence selection.  In _34th AAAI Conference on Artificial Intelligence_.
  * Glavaš et al. (2020) Goran Glavaš, Mladen Karan, and Ivan Vulić. 2020.  XHate-999: Analyzing and detecting abusive language across domains and languages.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6350–6365, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Jin et al. (2022) Mali Jin, Daniel Preotiuc-Pietro, A. Seza Doğruöz, and Nikolaos Aletras. 2022.  Automatic identification and classification of bragging in social media.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3945–3959, Dublin, Ireland. Association for Computational Linguistics.
  * Kantorovich (2006) Leonid V Kantorovich. 2006.  On the translocation of masses.  _Journal of Mathematical Sciences_ , 133(4):1381–1382.
  * Karan and Šnajder (2018) Mladen Karan and Jan Šnajder. 2018.  Cross-domain detection of abusive language online.  In _Proceedings of the 2nd Workshop on Abusive Language Online (ALW2)_ , pages 132–137, Brussels, Belgium. Association for Computational Linguistics.
  * Kassner and Schütze (2020) Nora Kassner and Hinrich Schütze. 2020.  BERT-kNN: Adding a kNN search component to pretrained language models for better QA.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3424–3430, Online. Association for Computational Linguistics.
  * Khandelwal et al. (2020) Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020\.  Generalization through memorization: Nearest neighbor language models.  In _International Conference on Learning Representations_.
  * Lee et al. (2022) Seonghyeon Lee, Dongha Lee, Seongbo Jang, and Hwanjo Yu. 2022.  Toward interpretable semantic textual similarity via optimal transport-based contrastive sentence learning.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5969–5979, Dublin, Ireland. Association for Computational Linguistics.
  * Lee et al. (2018) Younghun Lee, Seunghyun Yoon, and Kyomin Jung. 2018.  Comparative studies of detecting abusive language on Twitter.  In _Proceedings of the 2nd Workshop on Abusive Language Online (ALW2)_ , pages 101–106, Brussels, Belgium. Association for Computational Linguistics.
  * Malmasi and Zampieri (2018) Shervin Malmasi and Marcos Zampieri. 2018.  Challenges in discriminating profanity from hate speech.  _Journal of Experimental & Theoretical Artificial Intelligence_, 30:187 – 202.
  * McNemar (1947) Quinn McNemar. 1947.  Note on the sampling error of the difference between correlated proportions or percentages.  _Psychometrika_ , 12(2):153–157.
  * Meftah et al. (2021) Sara Meftah, Nasredine Semmar, Youssef Tamaazousti, Hassane Essafi, and Fatiha Sadat. 2021.  Neural supervised domain adaptation by augmenting pre-trained models with random units.  _ArXiv_ , abs/2106.04935.
  * Mollas et al. (2022) Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. 2022\.  ETHOS: A multi-label hate speech detection dataset.  _Complex & Intelligent Systems_.
  * Mozafari et al. (2019) Marzieh Mozafari, Reza Farahbakhsh, and Noel Crespi. 2019.  A BERT-based transfer learning approach for hate speech detection in online social media.  In _International Conference on Complex Networks and Their Applications_ , pages 928–940. Springer.
  * Olvera et al. (2021) Michel Olvera, Emmanuel Vincent, and Gilles Gasso. 2021.  Improving sound event detection with auxiliary foreground-background classification and domain adaptation.  In _DCASE 2021-6th Workshop on Detection and Classification of Acoustic Scenes and Events_.
  * Pamungkas et al. (2021) Endang Wahyu Pamungkas, Valerio Basile, and Viviana Patti. 2021.  A joint learning approach with knowledge injection for zero-shot cross-lingual hate speech detection.  _Information Processing & Management_, 58(4):102544.
  * Peyré and Cuturi (2019) Gabriel Peyré and Marco Cuturi. 2019.  Computational optimal transport: With applications to data science.  _Foundations and Trends® in Machine Learning_ , 11(5-6):355–607.
  * Poletto et al. (2019) Fabio Poletto, Valerio Basile, Cristina Bosco, Viviana Patti, and Marco Antonio Stranisci. 2019.  Annotating hate speech: Three schemes at comparison.  In _6th Italian Conference on Computational Linguistics, CLiC-it_.
  * Prasetyo and Samudra (2022) Vincentius Riandaru Prasetyo and Anton Hendrik Samudra. 2022.  Hate speech content detection system on twitter using k-nearest neighbor method.  In _AIP Conference Proceedings_ , volume 2470, page 050001. AIP Publishing LLC.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Saha and Sindhwani (2012) Ankan Saha and Vikas Sindhwani. 2012.  Learning evolving and emerging topics in social media: A dynamic NMF approach with temporal regularization.  In _Proceedings of the Fifth ACM International Conference on Web Search and Data Mining_ , WSDM ’12, page 693–702, New York, NY, USA. Association for Computing Machinery.
  * Sarwar et al. (2022) Sheikh Muhammad Sarwar, Dimitrina Zlatkova, Momchil Hardalov, Yoan Dinkov, Isabelle Augenstein, and Preslav Nakov. 2022.  A neighborhood framework for resource-lean content flagging.  _Transactions of the Association for Computational Linguistics_ , 10:484–502.
  * Schmidt and Wiegand (2017) Anna Schmidt and Michael Wiegand. 2017.  A survey on hate speech detection using natural language processing.  In _Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media_ , pages 1–10, Valencia, Spain. Association for Computational Linguistics.
  * Shnarch et al. (2018) Eyal Shnarch, Carlos Alzate, Lena Dankin, Martin Gleize, Yufang Hou, Leshem Choshen, Ranit Aharonov, and Noam Slonim. 2018.  Will it blend? Blending weak and strong labeled data in a neural network for argumentation mining.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 599–605, Melbourne, Australia. Association for Computational Linguistics.
  * Swamy et al. (2019) Steve Durairaj Swamy, Anupam Jamatia, and Björn Gambäck. 2019.  Studying generalisability across abusive language detection datasets.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pages 940–950, Hong Kong, China. Association for Computational Linguistics.
  * Swanson et al. (2020) Kyle Swanson, Lili Yu, and Tao Lei. 2020.  Rationalizing text matching: Learning sparse alignments via optimal transport.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5609–5626, Online. Association for Computational Linguistics.
  * Vidgen et al. (2021) Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021.  Learning from the worst: Dynamically generated datasets to improve online hate detection.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1667–1682, Online. Association for Computational Linguistics.
  * Villani (2009) Cédric Villani. 2009.  Optimal transport: Old and new.  volume 338. Springer.
  * Waseem and Hovy (2016) Zeerak Waseem and Dirk Hovy. 2016.  Hateful symbols or hateful people? Predictive features for hate speech detection on Twitter.  In _Proceedings of the NAACL Student Research Workshop_ , pages 88–93, San Diego, California. Association for Computational Linguistics.
  * Wiegand et al. (2019) Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019.  Detection of abusive language: The problem of biased datasets.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 602–608, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xu et al. (2021) Jingjing Xu, Hao Zhou, Chun Gan, Zaixiang Zheng, and Lei Li. 2021.  Vocabulary learning via optimal transport for neural machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7361–7373, Online. Association for Computational Linguistics.
  * Yin and Zubiaga (2021) Wenjie Yin and Arkaitz Zubiaga. 2021.  Towards generalisable hate speech detection: a review on obstacles and solutions.  _PeerJ Computer Science_ , 7.

##  Appendix A Illustration of OTNN

Figure 3 presents an illustration of the proposed $OT^{NN}$ training strategy.

Figure 3: Illustration of the training strategy in OTNN. Even though the BERT
encoder $g$ and the classifier $f$ are shared by both corpora, they are
illustrated twice for better clarity by representing the two corpora
separately. The presented softmax values obtained from $f$ are simply examples
provided for illustration. The figure is inspired by Damodaran et al. (2018).

##  Appendix B Corpus Details

The corpora used in our experiments are collected during different time
periods, with different sampling strategies across varied online platforms.
Following are some additional details about the corpora discussed in Section
4.1.

Waseem: This Twitter corpus, provided by Waseem and Hovy (2016), is sampled
mainly using keywords containing common terms and slurs associated with hate
against sexual, gender, religious, and ethnic minorities. It originally has
three classes: ‘sexism’, ‘racism’, and ‘none’. However, since it is made
available as tweet IDs, we observe that a major portion of the racist comments
is removed by Twitter, and we could retrieve only 20 instances labeled as
‘racism’. Due to the difference in the labeling schemes across corpora, we
follow other prior work for cross-corpus evaluations (Wiegand et al., 2019;
Swamy et al., 2019), and merge the original ‘racism’ and ‘sexism’ classes into
the class hate. In total, we obtain 10.9K tweets, of which 26.8% instances
belong to the hate class. A major portion of hateful comments in this corpus
is subtle and do not involve profane words (Wiegand et al., 2019).

Vidgen: The Vidgen corpus (Vidgen et al., 2021) is dynamically generated using
a human-and-model-in-the-loop process that is aimed at making the corpus
robust to different forms of hate. These hateful forms are derogation,
animosity, threatening language, support for hateful entities, and
dehumanization. It covers hate directed against diverse social groups like
blacks, women, jews, muslims, immigrants, arabs, disabled people, trans
people, and others. It has adversarial content that can mislead the HS
detection systems trained on other corpora, such as ‘all women are created
equal, except chinks’, ‘the tea cup is bigger than the f*cking teapot’. We use
its version v0.2.3 containing a total of 41144 instances, where duplicates are
removed, of which 53.9% is labeled as hate.

Ethos: This corpus (Mollas et al., 2022) consists of 998 comments from YouTube
and Reddit, sampled using an active learning strategy, which ensures both
diversity and balance with respect to different hateful aspects defined. One
of the ways they ensure this is by keeping only one instance of multiple
comments with similar meanings. For example, the comments ‘I hate white
people’ and ‘I hate whites’ (Mollas et al., 2022) are similar, and only one of
them is added. It includes hate directed towards diverse identities, such as
gender, race, national origin, disability, religion, and sexual orientation.
In this work, we use the binary version of this corpus with 43.4% hate
instances.

Target corpus | Waseemtar | Vidgentar | Ethostar
---|---|---|---
Source corpus |  Vidgensrc |  Ethossrc |  Waseemsrc |  Ethossrc |  Vidgensrc |  Waseemsrc
Seq-FT |  63.2$\pm$2.1 |  65.0$\pm$1.1 |  67.0$\pm$2.2 |  70.8$\pm$3.9 |  79.8$\pm$0.7 |  70.2$\pm$3.1
$k$ = 10 |  59.8$\pm$1.8 |  68.4$\pm$0.8 |  65.6$\pm$1.6 |  68.8$\pm$3.9 |  76.8$\pm$0.7 |  67.6$\pm$2.8
$k$ = 20 |  61.2$\pm$1.5 |  67.6$\pm$1.5 |  64.8$\pm$1.6 |  69.2$\pm$3.2 |  76.8$\pm$1.0 |  67.4$\pm$3.3
$k$ = 30 |  60.3$\pm$1.6 |  68.1$\pm$1.0 |  64.4$\pm$1.9 |  69.9$\pm$2.8 |  76.8$\pm$0.5 |  68.5$\pm$1.7
$k$ = 40 |  61.6$\pm$1.6 |  68.6$\pm$1.4 |  64.6$\pm$1.0 |  70.8$\pm$3.5 |  76.2$\pm$1.2 |  68.2$\pm$2.6
$k$ = 50 |  60.8$\pm$2.0 |  68.8$\pm$0.7 |  62.8$\pm$2.6 |  68.4$\pm$4.8 |  75.8$\pm$0.4 |  68.4$\pm$0.5
Table 5:  Performance of CE $k$NN+ \+ SRC with different neighborhood sizes,
compared with Seq-FT. F1 score ($\pm$std-dev) is reported on the low-resource
target corpus with 400 labeled training instances (total 500 labeled instances
from the target) available.

##  Appendix C Data Preprocessing

We pre-process the corpora by removing the URLs, splitting the hashtags into
constituent words using CrazyTokenizer111https://redditscore.readthedocs.io,
expanding contractions (e.g. i’ll to i will), and removing the rarely
occurring Twitter handles and numbers. We finally convert the instances into
lower case.

##  Appendix D Implementation Details

For implementing the proposed OTNN framework, we fine-tune the pre-trained
BERT-base uncased model, implemented by Hugging Face (Wolf et al., 2020),
having 110 million parameters, with the joint distribution OT
framework222https://github.com/bbdamodaran/deepJDOT. We encode an instance
into the embedding space by obtaining the representations of the [CLS] token
from the last hidden layer of BERT, which is a 768-dimensional vector in the
BERT-base. We fine-tune the BERT model end-to-end for the classification task.
Therefore, the [CLS] representations are the fine-tuned BERT representations.
For incorporating the neighborhood information, we use the pre-trained SBERT
sentence embeddings from ‘all-mpnet-
base-v2’333https://huggingface.co/sentence-transformers/all-mpnet-base-v2
model, which is a sentence transformer model. For computing $\gamma$, we use
the entropic regularized unbalanced OT solver using the Python Optimal
Transport
package444https://pythonot.github.io/gen_modules/ot.unbalanced.html#ot.unbalanced.sinkhorn_unbalanced
(Flamary et al., 2021) at the mini-batch level.

For the baselines of ${k}$NN-FT, $k$NN ranking, weighted $k$NN and the OTNN
variants, we select the number of neighbors ($k$) from the range {10, 30, 50,
70, 100, 200, 300, 400, 500} through tuning over the corresponding target
validation sets with respect to the F1 score of the hate class with a random
seed. We set $\alpha$ = 0.05 and $\beta$ = 10 in Equation 3 and 5, and
$\theta_{s}$ = 1 for OTNN / $\mathrm{OT}_{\mathrm{pre-select}}^{NN}$ \+ sloss
and $\theta_{t}$ = 10 in Equation 4, 6 and 7 for all the experiments. For OTNN
without sloss, we set $\theta_{s}$ = 0.

For CE $k$NN+ \+ SRC, we perform experiments with the implementation provided
to us by the authors and report the results for the neighborhood size of 10 in
Table 2. Even though Sarwar et al. (2022) use 10 as the neighborhood size in
their task of transfer learning in a cross-lingual set-up, we experiment with
different neighborhood sizes ($k$ values). The results are reported in Table
5. However, we could not increase the neighborhood size beyond 50 because of
resource constraints. This is because a mini-batch in their framework
comprises a query instance from the target and all its $k$ neighbors from the
source. Thus, the number of neighbors is limited by the mini-batch size, which
usually needs to be kept small when fine-tuning large language models like
BERT. We can observe from Table 5 that the performances obtained with
different neighborhood sizes are similar.

Target corpus | Waseemtar | Vidgentar | Ethostar
---|---|---|---
Source corpus |  Vidgensrc |  Ethossrc |  Waseemsrc |  Ethossrc |  Vidgensrc |  Waseemsrc
Mixed-FT |  17.8 m |  0.4 m |  4.7 m |  0.5 m |  14.0 m |  4.7 m
OTNN |  18.9 m |  0.4 m |  5.1 m |  0.6 m |  14.2 m |  5.0 m
OT${}^{NN}_{\mathrm{pre\mbox{-}select}}$ |  3.7 m |  0.3 m |  1.1 m |  0.6 m |  6.5 m |  3.4 m
OTNN \+ sloss |  18.9 m |  0.4 m |  5.0 m |  0.6 m |  14.5 m |  4.9 m
OT${}^{NN}_{\mathrm{pre\mbox{-}select}}$ \+ sloss |  11.7 m |  0.4 m |  3.8 m |  0.6 m |  5.5 m |  3.9 m
Table 6:  Per epoch training time in minutes for different settings.

We implement PretRand ourselves following the description provided by Meftah
et al. (2021). This approach is evaluated by the authors on the tasks of part-
of-speech tagging, chunking, named entity recognition, and morphosyntactic
tagging. Therefore, the approach uses a sequence labeling model with pre-
trained word embeddings and a BiLSTM-based feature extractor. However, for a
fair comparison with our approach, we use the pre-trained BERT model as the
feature extractor instead of the BiLSTM model for the pre-trained units. For
the randomly initialized units, we follow the approach and add a BiLSTM layer
over the last hidden layer of the BERT model. We first fine-tune the pre-
trained BERT model, without the randomly initialized units, on the source
corpus. We then fine-tune the model with the additional randomly initialized
units on the target corpus. We use the Adam optimizer with a learning rate of
$5\times 10^{-5}$ for the pre-trained BERT parameters. For the randomly
initialized units, we use the Adam optimizer with a learning rate of
$1.5\times 10^{-2}$ following Meftah et al. (2021).

##  Appendix E Computational Efficiency

We present the per epoch training time of Mixed-FT and OTNN variants for
different settings of the source and target corpora in Table 6. Mixed-FT is a
baseline that involves training the pre-trained BERT model on the combination
of the source and target corpora. For every mini-batch of size $m$, there are
$m$ instances sampled from each of the source and target corpora (Equation 7).
This is the same mini-batch sampling that is followed in OTNN. We use one
Nvidia GTX 1080 Ti GPU for our experiments. We can observe that OTNN results
in approximately the same computation time as taken by Mixed-FT in most of the
settings as it does not change the model architecture, but only introduces a
new training strategy. With the ‘pre-select’ variant, the computation time
gets further reduced in a few settings. This is because, in this variant, the
model only gets trained on a subset of pre-selected source instances based on
the neighborhood size.