# Overlapping Word Removal is All You Need: Revisiting Data Imbalance in Hope
Speech Detection

Hariharan RamakrishnaIyer LekshmiAmmal a,Manikandan Ravikiran b,Gayathri Nisha
a,Navyasree Balamuralidhar a,Adithya Madhusoodanana,Anand Kumar
Madasamya,Bharathi Raja Chakravarthic CONTACT Hariharan R L. Email:
<EMAIL_ADDRESS>a Department of Information Technology,
National Institute of Technology, Karnataka Surathkal, India  bGeorgia
Institute of technology, Atlanta, Georgia  cInsight SFI Research Centre for
Data Analytics, Data Science Institute, National University of Ireland Galway,
Galway, Ireland

###### Abstract

Hope Speech Detection, a task of recognizing positive expressions, has made
significant strides recently. However, much of the current works focus on
model development without considering the issue of inherent imbalance in the
data. Our work revisits this issue in hope-speech detection by introducing
focal loss, data augmentation, and pre-processing strategies. Accordingly, we
find that introducing focal loss as part of Multilingual-BERT’s (M-BERT)
training process mitigates the effect of class imbalance and improves overall
F1-Macro by 0.11. At the same time, contextual and back-translation-based word
augmentation with M-BERT improves results by 0.10 over baseline despite
imbalance. Finally, we show that overlapping word removal based on pre-
processing, though simple, improves F1-Macro by 0.28. In due process, we
present detailed studies depicting various behaviors of each of these
strategies and summarize key findings from our empirical results for those
interested in getting the most out of M-BERT for hope speech detection under
real-world conditions of data imbalance. Our codes and models are made
available on GitHub 222https://github.com/hariharanrl/lre_hope_2021.

###### keywords:

Hope Speech Detection; Language modeling; Text Classification; Data imbalance;
Focal loss

##  1 Introduction

Hope Speech Detection is the task of identifying expressions that is positive,
encouraging, supportive, and inspiring promise of the future (Chakravarthi,,
2020). One such example of Hope and Non-Hope speech is shown below.

Hope Speech: I’m so proud for her. Non-Hope Speech: Yup it’s definitely in
your blood…your evil demonic blood…shes pure satanist wake up ppl.

With growing social media applications identifying hope speech is more vital,
especially in encouraging marginalized communities such as people from
Lesbian, Gay, Bisexual, Transgender, and Queer (LGBTQ) communities, racial
minorities, women in the fields of Science, Technology, Engineering, and
Management (STEM) and people with disabilities (Wang and Jurgens,, 2018).
Moreover, with 84% of social media traffic generated by teenagers (Auxier and
Anderson,, 2021), enforcing positivity is indispensable. Besides, due to the
natural state of social media with the presence of online harassment, violent
content, and abuse, inspiring people with positive comments have become an
imperative need across social media platforms. Further, showing positive
content tends to reflect more positive behavior by the people in social media
(Kramer et al.,, 2014). Besides, with the recent pandemic resulting in a lack
of social interaction and depression, there is a need to reinforce positivity
and hope. Considering these needs, coupled with the availability of sizeable
pretrained language models and release of open datasets, research on hope
speech detection has seen meaningful traction where many of the works exhibit
remarkable accuracy gains (Chakravarthi,, 2020). However, despite such recent
success, much of the study on hope speech detection is mainly directed towards
fine-tuning large language models without examining the following inherent
issues.

  * •

Data Imbalance: Firstly, due to the natural process of data collection and the
current natural skew of social media users towards writing Non-Hope Speech,
the dataset is highly imbalanced (See Table • ‣ 1). Meanwhile, Deep Neural
Networks (DNN’s), such as BERT and its alternatives which are used currently,
often encounter generalization issues under conditions of data imbalance (Dong
et al.,, 2019). Unfortunately, such imbalance problems are ignored by current
research works on hope speech detection. More specifically, a survey of
existing literature exhibits that out of 22 papers, 16 of them use a variant
of BERT (Devlin et al.,, 2019), 3 of them employ simpler DNN architectures
with word embeddings, with all of them not accounting for the data imbalance
issue.

Training Data Distribution (%) in Hope Speech Detection Dataset.  Language
Hope Non-Hope Other English 8.61 91.28 0.11

  * •

Word Overlap Issue: Secondly, to the best of our knowledge, none of the
existing works consider the issue of word overlap between the Hope and Non-
Hope speech comments. Our analysis reveals that out of 33458 words in the
training vocabulary of the hope speech detection dataset, 5084 words are
present in both Hope and Non-Hope classes. Such overlap may lead to over-
estimating models’ performance, effectively rendering much of the current
benchmarks useless. Moreover, such an issue often leads to bias in model and
parameter selection towards overfitting, impacting model generalization.
Furthermore, the issue is more prevalent when combined with the problem of
data imbalance.

  * •

Overlooked Preprocessing Stages: Also, much of the research on hope speech
detection overlooks pre-processing stages, with none of them explicitly
mapping the relationship between data pre-processing techniques and their
effect on imbalanced data distributions like the one seen in the hope speech
detection problem. Specifically, our survey of existing works reveals that
only two of the studies focus on some form of pre-processing..

  * •

Inaccurate Accuracy Measure: To date, all of the works on hope speech
detection employ a weighted average F1 score. While this measure is
reasonable, it is unsuitable for practical applications where the less
expressed Hope Speech class does not have equal importance like Non-Hope
class, making it more unreliable when evaluating the model performance,
considering the aim is to develop a model that performs well on all classes,
including minority.

Accordingly, we revisit the problem of hope speech detection by making the
following contributions.

  * •

Firstly, we introduce a baseline benchmark for M-BERT on the English hope
speech detection dataset. To this end, we report macro averaged results
suitable for practical applications and empirically verify previously
mentioned issues.

  * •

Secondly, to account for data imbalance in the context of hope speech
detection, we study (a) Focal loss (Lin et al.,, 2020) instead of cross-
entropy loss to train M-BERT, (b) Contextual (Kobayashi, 2018a, ) and Back-
Translation (Sennrich et al.,, 2016) data augmentation to mitigate data
imbalance. While the former reshapes the standard cross-entropy loss by
penalizing the loss assigned to well-classified instances, accounting for
imbalance, the latter generates new samples using existing samples from the
Hope class. To this end, we find (a) focal loss when coupled with M-BERT
improves F1-Macro by 0.1126 and 0.1173% and (b) data augmentation improves
F1-Macro by a maximum of 0.1131 and 0.1043 on validation and test splits of
hope speech detection dataset.

  * •

Finally, to account for word overlap issues, we propose a simplistic word
removal algorithm as a preprocessing step, which focuses on removing
conflicting context, improving F1-Macro by 0.2786 and 0.2828, respectively, on
validation and test sets.

The rest of the paper is organized as follows. First, in section 2, we briefly
review the literature on hope speech detection, imbalanced text
classification, preprocessing, and data augmentation strategies. Followed by
dataset and experimental setup are explained in section 3. Then, in section 4,
we present focal loss, data augmentation strategy, and word removal
preprocessing algorithm post which the baseline results and other experiments
are presented in section 5. Finally, we conclude with hints on possible future
works in section 6.

##  2 Related Work

In this section, we present literature on approaches for hope speech detection
(section 2.1), handling data imbalance (section 2.2) and preprocessing
approaches to improve results (section 2.3).

###  2.1 Approaches for Hope Speech Detection

The currently published approaches for hope speech detection could be broadly
divided into three categories as explained.

  * •

Classical Approaches: Classical approaches have seen very few works in hope
speech detection, with notable of them by (Dave et al.,, 2021) using TF-IDF
character n-grams coupled with pre-trained MuRIL embeddings (Khanuja et al.,,
2021) for text representation and Logistic Regression and Linear SVM for
classification.

  * •

Hybrid Approaches: Hybrid approaches typically employ some form of deep
learning architecture involving Convolutional Neural Network (CNN) or
Recurrent Neural Network (RNN). These are coupled with one or more variants of
embedding representations. In this line, there are works by (M K and A P,,
2021) using RNN with context-aware string embeddings and pooled document
embeddings for word representations. Following this, there is work by
(Balouchzahi et al.,, 2021) which revealed three models, namely, CoHope-ML,
CoHope-NN, and CoHope-TL based on an ensemble of classifiers, neural network
(NN), and Bi-directional Long Short Term Memory (Bi-LSTM) with 1-D CNN model.
Moreover, there are more simplistic variants of CNN-LSTM by (Saumya and
Mishra,, 2021) which uses GloVe (Pennington et al.,, 2014) and Word2Vec
(Mikolov et al.,, 2013) embeddings.

  * •

Language Modelling: Among the methods in the literature, language models seem
to enjoy a special place due to their wide usage and ease of training due to
open source tools. Almost all the works use language models being used for
hope speech detection, beginning with an exhaustive benchmark by (Puranik et
al.,, 2021) using variants of BERT. There are also individual works on using
ALBERT (Chi et al.,, 2021), DistilBERT (Sanh et al.,, 2019), XLM-RoBERTa
(Conneau et al.,, 2020), MuRIL (Khanuja et al.,, 2021) to classify dataset for
English, Malayalam and Tamil languages. Alternatively, there are a subset of
works that employ M-BERT directly (Puranik et al.,, 2021; S et al.,, 2021) and
then few which use it for representation learning (Ziehe et al.,, 2021;
Mahajan et al.,, 2021; Que,, 2021). Moreover, some works coupled Language
models with classification approaches like (Huang and Bai,, 2021) which
consolidated the XLM-RoBERTa language model and the TF-IDF. Finally, there are
few works (Awatramani,, 2021; Sharma and Arora,, 2021) that employ some form
of preprocessing and augmentation using back-transliteration of the comments.
However, they do not apply to the English Language.

Overall much of the current works focus on using language models either
directly or in the form of feature extractors without considering the issue of
data imbalance. In our work, we study M-BERT by explicitly focusing on the
impact of data imbalance, and in the due process to accommodate data
imbalance, we propose training M-BERT with Focal Loss.

###  2.2 Approaches for Handling Data Imbalance

Handling data-imbalance is done either through data-level methods and
algorithm-level methods with former create more data to balance the minority
classes and later proposing algorithmic modifications. Fundamental of data
level technique is Synthetic Minority Oversampling Technique (SMOTE) (Chawla
et al.,, 2002) and its variants SMOTE-SVM (Nguyen et al.,, 2011). Many more
methods exists, including usage of synonym list (Wang et al.,, 2020) ,
heuristic rules (Kafle et al.,, 2017), encoder-decoder models (Xia et al.,,
2017) to reduce data imbalance by generating new samples. Recently there are
methods that generate additional data samples using Generative Adversarial
Nets (Goodfellow et al.,, 2014; Fedus et al.,, 2018), Majority-Minority
translating (Kim et al.,, 2020), Easy data augmentation techniques (EDA) (Wei
and Zou,, 2019), MixUp Augmentation (Guo et al.,, 2019), Contextual
Augmentation (Fadaee et al.,, 2017; Kobayashi, 2018b, ) all of which describes
forming new examples through some modification to existing examples.

Alternately, there are also algorithm-level approaches. Primitive of these
typically employ some form of cost variable and modify the classifier to
output with reduced bias (Krawczyk,, 2016). Then there works that change the
loss functions to naturally account for data imbalance (Lin et al.,, 2020).
Typically, these loss functions manage the class imbalance problem by
allotting more weights to complex or easily miss-classified examples.

In line with the above works, we focus on reducing imbalance in the context of
hope speech detection. To this end, we contrast data level approaches like
contrastive and back-translation-based augmentation with algorithmic
strategies like Focal Loss while training the M-BERT model.

###  2.3 Preprocessing Approaches

While models and architectures play a crucial role in achieving high results,
preprocessing tends to hold its unique position as it impacts extrinsic
performance when integrated into the neural network architecture. There are a
plethora of works beginning with (Karlgren and Sahlgren,, 2001) studying the
impact of morphological analysis on creating word representations for synonymy
detection, (Bullinaria and Levy,, 2012) present benefits of stemming and
lemmatization before training word representations to show improvement in
results. Similarly, (Angiani et al.,, 2016) analyzes various preprocessing
methods such as stopword removal, stemming, negation detection, emoticon
preprocessing to find stemming to be most effective for the task of sentiment
analysis. Similarly, (Trask et al.,, 2015) employ POS-disambiguated targets to
show an improvement in the performance of a variety of tasks. In contrast,
(Saif et al.,, 2014) shows the negative impact on the classification
performance of using the precompiled stoplist for the problem of sentiment
analysis.

Recently there are works by (Ebert et al.,, 2016) which introduce LAMB that
show the advantage of word normalization on word similarity benchmarks. (Zhao
and Gui,, 2017) observed that removing stopwords, numbers, and URLs can reduce
noise but does not affect performance, whereas replacing negation and
expanding acronyms can improve the classification accuracy. Alternatively,
(Kuznetsov and Gurevych,, 2018) examines the effect of lemmatization and POS
typing on word embedding performance. (Pecar et al.,, 2018) also highlights
the importance of preprocessing when using user-generated content, with
emoticons processing being the most effective. Finally, (Camacho-Collados and
Pilehvar,, 2018) shows an extensive evaluation on standard benchmarks of text
categorization and sentiment analysis, highlighting significant variations in
results across preprocessing techniques.

More recently, there are works (Babanejad et al.,, 2020) that comprehensively
analyze the role of preprocessing techniques in affective analysis based on
the word vector model. However, to the best of our knowledge, there are no
works on preprocessing explicitly for language models and hope speech
detection, which often takes in raw sentences as input. Accordingly, we
introduce and study a simple overlapping word removal preprocessing algorithm
that caters to the behavior of language models for the task of hope speech
detection.

##  3 Dataset and Experimental Setup

###  3.1 Dataset

In this work, we employ the Hope-speech dataset (Chakravarthi,, 2020) was
released as a part of a shared task challenge on Language Technology for
Equality, Diversity, and Inclusion (Chakravarthi and Muralidaran,, 2021). The
dataset is originally multilingual and consists of three subsets, namely
English, Tamil, and Malayalam languages, with the latter two consisting of
code-mixed text. In this work, we focus only on the English language. The
English language subset consists of comments/posts from YouTube that offer
support, reassurance, suggestions, inspiration, and insight divided into three
classes, namely Hope-speech, Non-Hope-speech or Not-English. Overall dataset
statistics are as shown in Table 3.1. As mentioned previously in Table • ‣ 1,
the English dataset is highly imbalanced, with $91.28\%$ belonging to Non-
Hope, in turn, exhibiting skew in the dataset. Further, considering that the
proportion of samples in Not-English class is only 20, we remove Not-English
class and focus on Hope-Speech and Non-Hope-Speech classes, respectively. In
the rest of these documents, for convenience, the dataset will be referred to
as HSDv2, and the classes will be referred to as Hope Class and Non-Hope
Class, respectively.

Statistics of Hope Speech Detection Dataset  Language Class Training
Validation Testing English Hope 1962 272 250 Non-hope 20778 2569 2593 Not-
English 22 2 3

###  3.2 Evaluation Metrics

As mentioned previously in section 1, currently in this work, we focus on to
accurately classifying comments of Hope class. Accordingly, we use the
following accuracy measures.

  1. 1.

Precision: is the proportion of correct classification to the number of
incorrect classification.

$Precision=\frac{TP}{TP+FP}$ (1)

  2. 2.

Recall: is the proportion of correct classification to the number of missed
entries.

$Recall=\frac{TP}{TP+FN}$ (2)

  3. 3.

F1-Score: is the harmonic mean of precision and recall.

$F1-Score=2\times\frac{Precision\times Recall}{Precision+Recall}$ (3)

where True Positive (TP) corresponds to the number of comments correctly
classified to Hope Class, True Negative (TN) accounts for the number of
comments correctly classified to Non-Hope Class, False Positive (FN) considers
the number of Hope-Speech comments wrongly classified to Non-Hope Class and
False Negative (FN) reflects the number of Non-Hope class comments incorrectly
assigned to Hope class.

Upon computing these measures for both Hope and NonHope class, we calculate
macro and weighed average using equations 4 and 5, where Score is replaced
with either of precision, recall or F1-score and $W_{1},W_{2}$ is calculated
to the proportion of total sample belonging to each class.

$Macro\ Average=0.5\times(Score_{Hope}+Score_{Non-Hope})$ (4)

$Weighed\ Average=W_{1}\times Score_{Hope}+W_{2}\times Score_{Non-Hope}$ (5)

##  4 Methodology

In this section, we present our baseline Multilingual-BERT model (section
4.1), Focal Loss (section 4.2), Contextual Data Augmentation (section 4.3),
Back-Translation Data Augmentation (section 4.4) and Word Removal Pre-
Processing Algorithm (section 4.5).

###  4.1 Model

Multilingual-BERT (M-BERT) is widely used due to its ability to learn deep
multilingual representation independent of overlap of vocabulary between train
and test datasets (Pires et al.,, 2019). Accordingly, in this work, we employ
M-BERT as our baseline.

Given, a sentence $S$ consisting of word sequence
$\\{w_{1},w_{2},....,w_{n}\\}$ with a masked word $w_{i}$ ($i\in[1...n]$) in
position $i$. Let $H^{0}$ be the input layer and $H^{j}$ represent
intermediate layers defined as shown in Equations 6 and 7 respectively.

$H^{0}=[r_{1};...;r{i-1},r_{i},r{i+1};....r_{n}]+W_{p}\\\ $ (6)

$H^{j}=\textbf{SA}(H^{j-1}),\ j\in\ [1...K]$ (7)

where $r_{i}$ is embedding representation corresponding to word $w_{i}$,
$W_{p}$ weights for the position embedding and SA corresponds to Self-
Attention (Vaswani et al.,, 2017) encoder that encodes input with $K$ Self-
Attention layers. Originally, M-BERT is trained in form of Masked Language
Model (MLM), where it predicts $w_{i}$ using Equation 8.

$P(w_{i})=Softmax(Wh_{i}^{K})$ (8)

where $P(w_{i})$ denotes
$P(w_{i}\|w_{1},....,w_{i-1},<MASK>,w_{i+1},...w_{n})$ and $W$ is the model
parameter. For set if unlabelled text, $D=\\{W^{i}\\}$, M-BERT is originally,
trained on Wikipedia dump of 100 languages by maximizing

$J=\sum_{i=1}^{N}\sum_{j=1}^{D}P[w_{i}^{j}]\\\ $ (9)

In this work, we finetune M-BERT for hope speech detection as follows. Let
$\textbf{h}_{i}$ $(i\in[1,...F])$ be the output of final hidden states of
pretrained M-BERT for input hope speech word sequence $W$ and $\textbf{h}_{p}$
represent output from last hidden state. Let CE represent be the Cross-Entropy
loss function that uses hope speech detection class labels $y\in[Hope,Non-
Hope]$ to generate prediction probability $P(C_{i})$ over all the classes.
Then classification is achieved by minimizing the following objective $J^{*}$
where $W_{h}$ corresponds to weight of fully connected layer.

$P(C_{i})=Softmax(W_{h}h_{p})$ (10)

$J^{*}=CE(y,P(C_{i}))$ (11)

###  4.2 Focal Loss

As discussed in section 4.1, M-BERT is originally trained on Cross-Entropy
loss. For a classification problem like Hope Speech Detection, Cross-Entropy
(CE) is defined as shown in Equation 12.

$CE(y,p)=\begin{cases}-log(p)&\text{if y=Hope}\\\
-log(1-p)&\text{otherwise}\end{cases}$ (12)

In the above equation y corresponds ground-truth class and $p$ corresponds to
probability class wise probability in range of $[0-1]$ indicating models
estimate of class probability with label $y=Hope$.For simplicity let us define
CE(p,y) = CE($p_{t}$), where $p_{t}$ is as shown in Equation 13.

$p_{t}=\begin{cases}p&\text{if y=1}\\\ 1-p&\text{otherwise}\end{cases}$ (13)

While cross-entropy accounts for errors across the classes, it weighs
predictions of all the classes equally, and it views weak ($p_{t}\approx 0.5)$
and strong classifications ($p_{t}\geq 0.7$) equally, in turn assuming the
error contribution to be similar.

However, for the task of hope speech detection, CE loss is expected to be
overwhelmed due to the usage of overlapping vocabulary in both Hope and Non-
Hope classes. Moreover, wrong predictions of classes with minority samples are
often ignored during training, resulting in limited generalization during
actual testing. Additionally, Hope class usually consists of complex examples,
which again would impact if trained with CE loss.

Focal loss addresses the imbalance issue and other side effects of class
imbalance during training by using a modulating term to CE loss (Lin et al.,,
2020). Such a modulating term helps CE focus on learning hard samples (samples
belonging to minority classes and weakly classified). To this end, focal loss
injects the approach of dynamic scaling policy, where the scaling factor
($\gamma$) decays to zero as confidence in the correct class increases for all
the samples. Intuitively, $\gamma$ weighs down the contribution of easy
examples during training and concentrates the model on hard examples.
Formally, Focal Loss (FL) is defined as shown in 15,

$FL(p_{t})=-(1-p_{t})^{\gamma}log(p_{t})$ (14)

where $(1-p_{t})^{\gamma}$ is called modulating factor with $\gamma$ as a
tunable parameter. From Equation 15 we can see that a high value of $\gamma$
(ex: $\gamma>=2$) weights down all samples and gradients that are
approximately correct while converging on the examples which are yielding poor
performance. On the other hand, having a too low $\gamma$ (ex: $\gamma$ = 0)
reduces the impact of the weighting factor. However, the network will be
incompetent to learn the problem for complex tasks like hope speech detection.
Especially when the data is imbalanced, the network can easily be overwhelmed
by the dominant data. In summary, $\gamma$ decides which training samples the
network is most focused on at any given point during the training. As such, in
this work, we restrict $\gamma$=$\\{1,2\\}$, thereby balancing learning of
both poorly classified hate speech comments and easier ones, respectively.
More details are presented in section 5.

###  4.3 Contextual Data Augmentation

In this work, we also study contextual word augmentation (Kobayashi, 2018a, ),
where the words predicted by language models given the context surrounding the
original sentence to be augmented. More specifically, we use the language
model to calculate the word probability at a position $i$ based on its
surrounding context. Let $S_{org}$ be the sentence of size $N$ with words
$w_{i}=\\{w_{1},....,w_{N}\\}$ to be augmented, $LM$ be the language model,
$K$ be hyperparameter to select number of candidate words to be selected for
given word $w_{i}$ at position $i$ and $\\{A_{min},A_{max}\\}$ be minimum and
maximum number of words to be augmented for $S_{org}$. Then the contextual
data augmentation procedure is as shown in Algorithm 1.

Algorithm 1 Contextual Data Augmentation

Input: $S_{org},LM,K,A_{min},A_{max}$
Output: : Augmented Sentence $S_{aug}$
Procedure:

  * Set $A_{cur}=0$

  * for each word $w_{i}$ in $S_{org}$

  * Mask the word $w_{i}$ from position i

  * $WL$= Predict $K$ possible words for position $i$ using $LM$

  * if $A_{cur}\in\\{A_{min},A_{max}\\}$

  * replace $w_{i}$ in $S$ by $WL[0]$

  * $S_{aug}=S_{aug}\cup S_{j}$

  * end for

  * return $S_{aug}$

More specifically, in contextual data augmentation, we exploit Masked Language
Modeling, which acts as the task of fill in the blank, where we first mask a
word and use a language model which exploits context words surrounding a mask
to try to predict what the mask should be. To this end, for each position ’i’,
we mask the word and find new substitutes for word $w_{i}$ by sampling from a
given probability distribution using a pre-trained language model. An example
of contextual word augmentation is shown below with augmented words. In this
study, we use the original BERT uncased model for augmentation and set
$A_{min}=3$ and $A_{max}=10$, respectively.

Original Comment: Im only 11 but i love science and math i want to be a trauma
surgeon specializing in nuero surgery. I definitely think and that all this
will impower young girls like me or younger. Contextually Augmented Comment:
I’m only 11 months but first i love science and math i want to be a true
trauma rehabilitation surgeon specializing in neuro stress surgery. I
definitely into think and that all this will impower young girls around like
me or younger.

###  4.4 Back Translation based Data Augmentation

Previously we saw in section 2 that few works use back-translation in the
context of hope speech detection. To contrast with Focal Loss effectively, we
also employ back translation based data augmentation in line with (Sennrich et
al.,, 2016). Back translation based augmentation focuses on using the fact
that since LM’s are encoder-decoder architectures, they already condition the
probability distribution of the following target word on the previous target
words. We can create additional synthetic data by automatically translating
the original sentence to the target language and the target sentence back into
the source language. This target language is called the
intermediary/intermediate language.

Let $S={S_{1},....S_{L}}$ be the set sentence of size $L$ with words
$w_{i}=\\{w_{1},....,w_{N}\\}$, $LM=\\{LM_{i},.....,LM_{K}\\}$ be the K
language models corresponding K different languages, then back translation
based data augmentation is given as shown in Algorithm 2.

Algorithm 2 Back Translation based Data Augmentation

Input: $S={S_{1},....S_{L}}$, $LM=\\{LM_{i},.....,LM_{K}\\}$
Output: : Augmented Sentences $S_{aug}$
Procedure:

  * $S_{aug}=\\{\\}$

  * for each $S_{i}$ in $S$

  * for each $LM_{i}$ in $LM$

  * $S_{intermediate,i}=LM_{i}(S_{i})$

  * $S_{j}=LM_{j}(S_{intermediate,i})\ \forall\ \\{i\leq k,j\leq k,i\neq j\\}$

  * $S_{aug}=S_{aug}\cup S_{j}$

The example below shows back-translated augmentation with augmented words.

Original Comment: If you want to walk into a room and do not look like an
engineer, you have to prove it what does that have to do with being a woman?
If I walked in and claimed to be a surgeon no one would believe me either.
Most things have nothing to do with sexism…nnMany things would not have been
possible without female engineers? Why are we ok with that statement. Back
translated Comment: If you want to walk into a room and not look like an
engineer, you have to prove it, what does that have to do with being a woman?
If I walked in and said I’m a surgeon, no one would believe me either. Most
things have nothing to do with sexism … nn Many things would not have been
possible without the engineers? Why do we agree with that statement?

###  4.5 Word Removal Preprocessing Algorithm

Multilingual-BERT shows multiple advantages, as mentioned in section 4.1.
Besides, it is reasonably stable against out-of-vocabulary words, mainly due
to learning of semantic co-occurrence as part of its multi-head attention (Gan
et al.,, 2020). However, in the case of hope speech detection, we find a
significant overlap of vocabulary across hope-speech and non-hope-speech
classes, respectively, i.e., specific tokens that are present for non-hope-
speech are also present in hope-speech. More specifically, for English
languages, the overlap statistics are as shown in Table 4.5.

Word Overlap Statistics for Hope Speech Detection Dataset  Data Words in Hope
Words in Non-Hope Overlapping Words Train 7292 33458 5084 Validation 2070 8645
1288

Such irregularity in word occurrences often impacts M-BERT’s performance, as
embeddings can be biased towards duplicated words’ frequency. Furthermore,
words that occur less frequently with Hope class, have less impact on the
learned semantic meaning as they are sparsely distributed in the
representation space (Li et al.,, 2020). Besides such overlapping words, it
leads to under-estimate model accuracy and ignores models’ bias towards
certain words. As such, we employ a word removal pre-processing algorithm.

Let $S_{c1}=\\{S_{(1,c1)},....S_{(N,c1)}\\}$ and
$S_{c2}=\\{S_{(1,c2)},....S_{(M,c2)}\\}$ set of sentences from two subsets of
data of sizes $N$ and $M$ respectively belonging to classes $c1$ and $c2$
respectively with each sentence consisting of words
$w_{i}=\\{w_{1},w_{2}....\\}$. Then, we generate a inter class word-word
occurrence matrix $WO$ $\in$ $R$ of size $L$x$Q$ using Equation 15.

$WO_{i,j}=\frac{1}{L\times Q}\sum_{i=0}^{L}\sum_{j=0}^{Q}1\ \text{if}\
w_{i}=w_{j}$ (15)

where $L$ is the number of unique words in $S_{c1}$ and $Q$ is the number of
the word in $S_{c2}$. To remove irrelevant words, remove words from such that
$WO_{i,j}\geq(\tau\times\frac{1}{L\times Q})$ words, and use the rest of the
words. The summary of algorithmic steps is presented in Algorithm 3.

Algorithm 3 Word Removal Preprocessing

Input: $S_{c1}=\\{S_{(1,c1)},....S_{(N,c1)}\\}$,
$S_{c2}=\\{S_{(1,c2)},....S_{(M,c2)}\\}$ , $\tau$
Output: : $S_{(c1,cleaned)},S_{(c2,cleaned)}$
Procedure:

  * –

Extract $L,Q$ unique words from $S_{c1},S_{c2}$ respectively.

  * –

Compute $WO_{i,j}=\frac{1}{L\times Q}\sum_{i=0}^{L}\sum_{j=0}^{Q}1\
\text{iff}\ w_{i}=w_{j}$

  * –

$WO_{i,j}=\begin{cases}WO_{i,j}\ \text{if}\
WO_{i,j}\geq(\tau\times\frac{1}{L\times Q})\\\ 0\ \text{otherwise}\end{cases}$

  * –

Remove words $w_{i},w_{j}$ if $WO_{i,j}=0$ from $S_{c1},S_{c2}$

  * –

$S_{(c1,cleaned)},S_{(c2,cleaned)}$ = $S_{c1},S_{c2}$

##  5 Experiments and Results

In this section, we start by explaining the baseline experiments on M-BERT,
followed by improvements and characteristics of each of the strategies
proposed in section 4.

###  5.1 Baseline

We start with the development of the baseline model of M-BERT, for which we
fine-tune using the HSDv2 from section 3.1. To this end, we train for ten
epochs using cross-entropy loss with 1k warmup steps, batch size of 8, the
learning rate of $3.0\times 10^{-}5$, the sequence length of 160, and ADAM
(Kingma and Ba,, 2015) optimizer with clip threshold of 1.0 and epsilon value
of $1\times 10^{-8}$. Detailed results on both validation and test splits are
as shown in Table 5.1 with respective Precision, Recall, and F1-Score with
Macro Average results. Analysis of results shows various interesting details
leading towards the requirement of methods in section 4.

Baseline Results of M-BERT for English Hope Speech Detection  Language Data
Class Precision Recall F1-Score English Validation Data Hope (271) 0.6512
0.6020 0.6257 Non-hope(2569) 0.7002 0.7425 0.7207 Macro Average(2840) 0.6757
0.6722 0.6732 Weighted Average 0.9275 0.9306 0.9289 Test Data Hope (250)
0.6658 0.6088 0.6360 Non-hope(2593) 0.6762 0.7278 0.7011 Macro Average(2843)
0.6710 0.6683 0.6686 Weighted Average 0.9277 0.9307 0.9290

Figure 1: LIME Predictions for hard example.

Figure 2: LIME predictions showcasing word bias.

  * •

Firstly, we can see that the overall results are quite low, with 0.6732 and
0.6686 for both validation and test sets.

  * •

Furthermore, we can see that between the classes of Hope and Non-Hope, the net
results differ by a large margin ($\geq 0.10$ and $\geq 0.07$% in F1-Score).
To this end, we can attribute such a large difference to imbalanced samples
for the Hope class.

  * •

At the same time, despite the large difference in F1, we can argue that M-BERT
being a strong baseline as it shows balanced F1, despite having 100$\times$
fewer samples for Hope class.

  * •

We further explicitly examine results on each class individually using input
perturbation-based explanation method involving Local Interpretable Model
Agnostic Explanations (LIME) (Ribeiro et al.,, 2016). Examples are as shown in
Figures 1 and 2 respectively. From a manual analysis of such an explanation,
we find that many of the problems that we defined earlier in section 1 are
valid. More specifically, we find that.

    * –

Sheer imbalance on dataset often flips the hard samples (See Figure 1), which
suggests incorporating strategies for handling imbalance as part of the
overall training pipeline.

    * –

Besides, we can also see some words impacting the final predictions, leading
to erroneous results. For example, the word this in Figure 2 is shown to
contribute to the final results, despite having no semantically meaningful
information. Such results acknowledge the need to revisit and add
preprocessing stages in hope speech detection.

To summarize from our baseline, we find

  * •

Overall, F1-Scores differ by a large margin for Hope and Non-Hope classes.

  * •

M-BERT despite having 100$\times$ lesser sample for Hope class, shows descent
performance, suggesting its suitability as a strong baseline for Hope Speech
Detection.

  * •

Our initial postulation of issues related to data imbalance and word overlap
is true and often results in flipping predictions towards the wrong class.

###  5.2 Effect of Focal Loss

Having examined the baseline using M-BERT in section 5.1 where we touched upon
the issues due to imbalance via LIME interpretations, we now start by focusing
on establishing an approach to reduce the impact of data imbalance. While data
imbalance may explain low F1 with M-BERT, to establish this quantitatively, we
start by replacing cross-entropy with focal loss in M-BERT’s training
pipeline. We begin by examining Precision, Recall, and F1-Score with both
Macro and Weighed Average measures as shown in Table 5.2 with $\gamma=2$.

To start with, we can see that compared to baseline with focal loss, the
F1-Macro has improved to 0.7860 and 0.7859 on validation and test sets,
respectively. This is approximately 0.12 improvement over baseline. The
following shows examples of various comments which initially were wrongly
identified as Non-Hope by cross-entropy loss but were corrected when coupled
with focal loss.

Example 1: This was so wonderful! Example 2: I watched this to become educated
on this matter Example 3: This made me cry. Thank you for posting it! This
gives me hopes for the future.

Results of M-BERT with Focal Loss HSDv2 Dataset  Model Data Class Precision
Recall F1-Score M-BERT with Focal Loss (Gamma=1) (binary class) Validation
Data Hope 0.6125 0.6125 0.6125 Non-hope 0.9591 0.9591 0.9591 Macro average
0.7858 0.7858 0.7858 Weighted average 0.9261 0.9261 0.9261 Test Data Hope
0.5809 0.5600 0.5703 Non-hope 0.9577 0.9610 0.9594 Macro average 0.7693 0.7605
0.7648 Weighted average 0.9246 0.9258 0.9252 M-BERT with Focal Loss (Gamma=2)
(binary class) Validation Data Hope 0.6423 0.5830 0.6112 Non-hope 0.9561
0.9657 0.9609 Macro average 0.7992 0.7744 0.7860 Weighted average 0.9262
0.9292 0.9275 Test Data Hope 0.6052 0.5640 0.5839 Non-hope 0.9571 0.9645
0.9608 Macro average 0.8031 0.7710 0.7859 Weighted average 0.9662 0.9292
0.9277

The above examples are not straightforward when it comes to identification as
Hope Speech. This is because some of the words so, educated, cry etc., which
are vital in deciding context, are prone to non-uniformities which can skew
the results. This moderately supports our hypothesis that penalizing complex
examples is critical in hope speech detection to sufficiently understand the
context when data is imbalanced.

Although the results are significantly higher, we can see much of the
improvement in results are concerning Non-Hope class, where for Non-Hope
class, the results increase by approximately 0.22 with the drop in results for
the Hope class by 0.01. We believe this because, despite handling imbalance,
the $\gamma$ factor plays a crucial lesson in generalization.

More specifically, we can see that with $\gamma=2$, there is an additional
improvement by addition 0.01 for the Hope class, which begs to question if
this should be increased further. In this regard, we believe that a higher
$\gamma$ may focus on complex examples. Fixing this value to a static number
is not helpful as this may lead to convergence issues as mentioned in section
4.2.

Moreover, with the low gamma value, the network will not learn any complex
examples at all. Thus, at the beginning of this study, we examined to strike a
balance by studying both $\gamma=\\{1,2\\}$. However, we can see that much of
focal loss tried to improve overall F1 by focusing on complex examples of Non-
Hope Speech. Following are some of the samples from the Hope class that were
miss-classified as Non-Hope.

Example 1: Im only 11 but I LOVE science and math I want to be a trauma
surgeon specializing in nuero surgery. I definitely think that this will
empower young girls like me or younger. Example 2: its not common for women to
work as blue-collar job in construction or plumbing.. Most of ’em are in the
professional Example 3: why can someone not agree with something? She is not
hurting anyone with her opinion as long as she is not harassing people.

Comparing the miss classified ones to the comments that were corrected, we can
see one exact problem; namely, sentences of larger length, predominantly
filled with words that are often part of Non-Hope class such as trauma,
plumbing, construction, hurting, etc. In our work, upon analysis, we find that
more than 20% of total errors introduced by focal loss for hope speech class
have these characteristics. We believe such an issue is because of fixed
$\gamma$, whereas when words that frequent with Non-Hope class appear with
Hope class, the classification gets penalized. Such repetitive penalization
for Hope class texts due to Non-hope class leads to delay in convergence,
causing performance drop. To summarize, our findings are

  * •

Focal Loss with $\gamma=1$ has an overall result of 0.7858, and 0.7648 and
$\gamma=2$ had 0.7860 and 0.7859 across validation and test sets,
respectively, indeed highlighting the complexity of the Hope class.

  * •

Further, we can see that while focal loss increases the overall F1, we can see
that the Hope class results are lower than the baseline. To this end, we
believe this may be attributed to fixed constant penalization upon miss-
classification due to words frequenting with Non-Hope classes.

  * •

Also, we find, despite lower than initial results, Focal loss still helps in
correcting errors in Hope class.

###  5.3 Effect of Data Augmentation

While focal loss tackles data imbalance in the context of training, data
augmentation reduces imbalance by generating more data. Accordingly, as
mentioned 4.3 and 4.4, we employ two data augmentation strategies by coupling
them with both cross-entropy loss and focal loss. Though some existing works
in hope speech detection aim to increase performance through using back
translation, the results obtained are fairly limited. So, for back
translation, we also include two languages, Spanish and French, as an
intermediary, owing to their word order being similar to English. While back
translation is a fairly straightforward idea, we believe that comments from
social media often suffer from the micro-domain issue, where each has its
style, impacting overall back translation. In particular, one or more back
translations may also lead to a drop in results. The dataset statistics post
each of the augmentation strategies is as shown in Table 5.3.

Statistics of HSDv2 after Data Augmentation  Train Original Contextual
Augmented Back-translation Augmented Train 1962 3924 5886

To begin with, we experiment using M-BERT with cross-entropy loss function and
employ contextual word embedding augmentation. The results are as shown in
Table 5.3. Comparing these results with a baseline from Table 5.1, we can see
that contextual augmentation improves the overall F1-Macro by 0.1. However, we
can see that net improvement post augmentation is seen for only the Non-Hope
class. Meanwhile, when trained with Focal loss, the net results are a bit on
the lower side compared to that of cross-entropy loss (See table 5.2), yet
with the overall improvement of 0.07 F1 again significant change observable
only for Non-Hope class.

While the reason for the drop in Hope class performance is in line with that
of discussion from section 5.2, we believe that much of augmentation
introduces words or phrases that are often associated with Non-Hope class. To
this end, we analyze the statistics of augmentation and some of the words used
to find that upon augmentation. We introduce 2$\times$ new sentences out of
which close to 70% augmented words can be associated with non-hope than hope.
For example, after using contextual word augmentation in the below statement,
the predictions of Hope class were mispredicted with both M-BERT and M-BERT
with focal loss.

Original Comment: It’s pretty simple. A large portion of the world is anti-gay
and the parades are a way to show that gay people aren’t going anywhere and
demand the same rights that everyone else has. It’s especially useful for the
gay teenagers that are put out onto the street by their christian families and
feel like they dont belong anywhere. Being able to see that people like you
are celebrating and proud of who they are when everyone around you wants to
make you feel ashamed for being gay is something that saves lives. Hope that
answer explains part of it Contextual Word Augmented Comment: this it’s pretty
simple. a large portion of the world is anti - gay and the parades are a way
to mock show that no gay people aren’t going anywhere and demand the same
rights that everyone else has. it’s especially useful for the gay teenagers
that are now put over out onto the street by their private christian families
and feel like they dont belong anywhere. being able to see that people like
you are celebrating and proud of who they are when everyone passing around you
wants to make you feel ashamed for being secretly gay is something online that
saves lives. hope that answer explains part of it

Results of M-BERT with HSDv2 with various Data Augmentation approaches  Model
Data Class Precision Recall F1-Score M-BERT with Contextual embedding
Validation Data Hope 0.6667 0.5092 0.5774 Non-hope 0.9495 0.9731 0.9612 Macro
average 0.8081 0.7412 0.7693 Weighted average 0.9225 0.9289 0.9245 Test Data
Hope 0.6308 0.5400 0.5819 Non-hope 0.9563 0.9695 0.9628 Macro average 0.7935
0.7548 0.7724 Weighted average 0.9276 0.9318 0.9294 M-BERT with Back
translation (French) Validation Data Hope 0.6314 0.5941 0.6122 Non-hope 0.9574
0.9634 0.9604 Macro average 0.7944 0.7788 0.7863 Weighted average 0.9263
0.9282 0.9272 Test Data Hope 0.6188 0.5520 0.5835 Non-hope 0.9573 0.9672
0.9622 Macro average 0.7880 0.7596 0.7729 Weighted average 0.9275 0.9307
0.9289 M-BERT with Contextual embedding and Back translation Validation Data
Hope 0.6429 0.5314 0.5818 Non-hope 0.9515 0.9689 0.9601 Macro average 0.7972
0.7501 0.7709 Weighted average 0.9220 0.9271 0.9240 Test Data Hope 0.5822
0.4960 0.5356 Non-hope 0.9521 0.9657 0.9588 Macro average 0.7671 0.7308 0.7472
Weighted average 0.9196 0.9244 0.9216 M-BERT with Back translation (Spanish)
Validation Data Hope 0.6652 0.5572 0.6064 Non-hope 0.9541 0.9704 0.9622 Macro
average 0.8096 0.7638 0.7843 Weighted average 0.9265 0.9310 0.9282 Test Data
Hope 0.6308 0.5400 0.5819 Non-hope 0.9563 0.9695 0.9628 Macro average 0.7935
0.7548 0.7724 Weighted average 0.9276 0.9318 0.9294

Back translations are often beneficial as intermediary languages may introduce
new words and, in fact, wider contexts. Corresponding experiments and results
are as shown in Table 5.3 and 5.3 with both French and Spanish as intermediate
languages. Comparing these results against 5.1 and 5.2, we can see the overall
results are significantly higher. Specifically, we notice that back
translation introduces an improvement of 0.11 over M-BERT. However, compared
to M-BERT with focal loss, we find that net improvement to be limited.
Besides, we can see that in line with contextual word augmentation, the
results are improved for Non-Hope class and dropped for the Hope class. Such
an issue is similar to the one mentioned earlier, with examples below showing
such an impact in prediction. Especially, we could see that in the dataset,
the words such as practically, believing, our flipped predictions towards Non-
Hope class.

Original Comment: I’m a Buddhist…! ALL LIVES MATTER…! Back-translation
Augmented: I’m practically a believing buddhist…! our all lives forever
matter…!

Also, comparing across different languages used, we find that French is more
useful for back-translation-based augmentation than Spanish. This we believe
is because French word tends to be identified by their surface forms similar
to English. Specifically, we can see that using French as back translation
tends to improve overall results by 0.03 and 0.01 respectively on M-BERT and
M-BERT with focal loss. Finally, the net performance when all of them are put
together, i.e., both augmentations used together the results are intermediate
between the respective baselines, and the results of back translation with
french as an intermediate language.

Results of M-BERT with focal loss various Data Augmentation approaches on
HSDv2.  Model Data Class Precision Recall F1-Score M-BERT with Word embedding
(Focal Loss) Validation Data Hope 0.6667 0.5092 0.5774 Non-hope 0.9495 0.9731
0.9612 Macro average 0.8081 0.7412 0.7693 Weighted average 0.9225 0.9289
0.9245 Test Data Hope 0.6105 0.4640 0.5273 Non-hope 0.9495 0.9715 0.9604 Macro
average 0.7800 0.7177 0.7438 Weighted average 0.9197 0.9268 0.9223 M-BERT with
Word embedding and Back translation (Focal Loss) Validation Data Hope 0.6329
0.5535 0.5906 Non-hope 0.9535 0.9661 0.9598 Macro average 0.7932 0.7598 0.7752
Weighted average 0.9229 0.9268 0.9246 Test Data Hope 0.5965 0.5440 0.5690 Non-
hope 0.9564 0.9645 0.9604 Macro average 0.7764 0.7543 0.7647 Weighted average
0.9248 0.9275 0.9260 M-BERT with Back translation (Spanish) (Focal Loss)
Validation Data Hope 0.6652 0.5572 0.6064 Non-hope 0.9541 0.9704 0.9622 Macro
average 0.8096 0.7638 0.7843 Weighted average 0.9265 0.9310 0.9282 Test Data
Hope 0.6308 0.5400 0.5819 Non-hope 0.9563 0.9695 0.9628 Macro average 0.7935
0.7548 0.7724 Weighted average 0.9276 0.9318 0.9294 M-BERT with Back
translation (French) (Focal Loss) Validation Data Hope 0.6667 0.5830 0.6220
Non-hope 0.9566 0.9692 0.9629 Macro average 0.8116 0.7761 0.7925 Weighted
average 0.9289 0.9324 0.9304 Test Data Hope 0.6650 0.5480 0.6009 Non-hope
0.9571 0.9734 0.9652 Macro average 0.8111 0.7607 0.7830 Weighted average
0.9315 0.9360 0.9332

To summarize, we find the following

  * •

Both contextual and back-translation based word augmentation improves the
F1-Macro by 10% and 11% respectively, yet they both show the issue of
improving only the Non-Hope class.

  * •

Also, both augmentation show errors due to introducing words that are commonly
associated with Non-Hope class.

  * •

Focal loss, when coupled with Augmented data, has a substantial impact on the
results.

  * •

Among Spanish and French the latter seems more beneficial for back-translation
based augmentation with 3% more improvement in F1-Macro.

  * •

Finally, with both strategies together, the results are higher than baseline
M-BERT and M-BERT with focal loss but lower than back-translation.

###  5.4 Effect of Word Removal

As seen in section 5.1, word overlap often biases the results towards the
majority class, especially in the case of significant class imbalance.
Previously in section 4.5, we saw that between the classes of Hope and Non-
Hope, 70% of words present in Hope class is present in Non-Hope class.

We hypothesize that overlapping word removal across classes should not hurt
model generalization and is expected to have no worse result compared to that
of the baseline model in section 4.1. As such, we start by executing a word
removal algorithm with $\tau=50$. The results so obtained are as shown in
Table 5.4 using M-BERT trained on both cross-entropy loss and focus loss,
respectively.

Results of M-BERT with overlapping word removal.  Loss function Data Class
Precision Recall F1-Score Cross Entropy Validation Data Hope(271) 0.9449
0.9876 0.9663 Non-hope(2569) 0.8824 0.9946 0.9385 Macro average(2840) 0.9125
0.9911 0.9518 Weighted average(2840) 0.9835 0.9838 0.9836 Test Data Hope(250)
0.9362 0.9885 0.9623 Non-hope(2593) 0.8800 0.9942 0.9371 Macro average(2843)
0.9072 0.9913 0.9493 Weighted average(2843) 0.9839 0.9842 0.9839 Focal Loss
Validation Data Hope(271) 0.9555 0.9861 0.9708 Non-hope(2569) 0.8676 0.9957
0.9317 Macro average(2840) 0.9094 0.9909 0.9502 Weighted average(2840) 0.9832
0.9835 0.9831 Test Data Hope(250) 0.9442 0.9885 0.9664 Non-hope(2593) 0.8800
0.9950 0.9375 Macro average(2843) 0.9110 0.9917 0.9514 Weighted average(2843)
0.9846 0.9849 0.9846

As can be seen from Table 5.4, with the removal of overlapping words, the
overall results tend to improve significantly where the test corpus with a low
percentage of seen words tend to have a high F1-Macro. This suggests the
system can recognize necessary context and efficiently classify the contents
across different classes.

Also, comparing these results from Table 5.1, the F1-Macro with seen words on
both classes tend to lower results significantly with cross-entropy loss and
suggest some form of memorization and poor generalization. In particular, we
can see the Precision, Recall and F1-Score to have almost 0.3 higher results
for the Hope class. Of the two classes, we can see that the Hope class
improves by an additional margin of 10% over Non-hope. As a result, both cross
the 0.9 in F1. This may be because overlapping words across classes has the
effect of biasing the pre-trained language models towards specific types of
examples. This results in lower generalization with overlapping words.

Example Predictions after using Overlapping Word Removal Algorithm with M-BERT
coupled with Cross-Entropy and Focal Loss  Original Tweet Modified Tweet Label
Word Removal Prediction CE Loss Word Removal Prediction Focal Loss “God gave
us a choice and my choice is to love” nnPROTECT THIS CHILD “God love”nn
PROTECT THIS CHILD Hope Non_Hope Hope I love how kids can react so positively
to this kids react positively Hope Non_Hope Hope Good thing my country never
had slavery Good thing country slavery Hope Hope Non_Hope

Meanwhile, we can also see focal loss with $\gamma=2$ to show similar results
like cross-entropy loss, except focal loss tends to add an additional 0.2%
improvement in overall results for Test data. We believe there are two
significant reasons for focal failure not having any further progress in
results. Firstly, using overlap word removal, we see that, on average, we see
a reduction in sentence length by five words. As a result, examples that
previously were hard to classify and required focal loss have limited context,
hence having no effect with focal loss. Secondly, we can see that the 5% of
comments that are wrongly classified include a lack of correct context, as
shown in Table 5.4.

Finally, in our study, we used a $\tau=25$ as a fixed value and did not
perform any ablation study for a range of values. This is because initial
experiments with a large value of $\tau=30$ show a minor decrease in results
which is obtained with $\tau=25$ as the network lacks sufficient input context
for learning in line with 5% of the cases mentioned earlier. Meanwhile, a
lower value $\tau=20$ also shows lower results, especially for the Hope class.
This is because, initially, higher context words lead to memorizing
overlapping words. However, we do believe a more in-depth analysis is needed
in this regard.

To summarize, our findings are

  * •

The overall results for both validation set and test set are well improved
with scores of 0.9518 and 0.9493 for cross-entropy loss compared to baseline.

  * •

Meanwhile, similar behavior could be observed with focal loss, except with a
minor drop in results.

  * •

Large value of $\tau$ causes limitation in context information often leading
to dropping in performance, meanwhile lower value of $\tau$ biases results
towards Non-Hope class.

###  5.5 Comparison of State of the Arts

We compare our strong baseline with some state-of-the-art methods in Table
5.5. To this end, since all the existing approaches present only weighed
average F1-Score, we compare our strategies against state-of-the-art using the
same. However, we do explicitly highlight the macro average of F1-Score. We
have divided benchmarks into three categories inline with section 2.1. The
consolidated influence of each of the proposed methods from section 4 is as
shown in Table 5.5 along with a comparison of state-of-the-art methods to
date. To this end, word removal is a clear winner where the net macro average
is 0.17 higher than the closest state-of-the-art methods.

Consolidated results with different proposed strategies and comparison over
state-of-the-art. Sorted in descending order of weighed average F1-Score.a
F1-Score Macro Average Weighed Average M-BERT + Focal Loss + word removal
0.9514 0.9846 M-BERT + word removal 0.9493 0.9839 M-BERT + Focal Loss + Back
Translation v2 0.7830 0.9332 (Mahajan et al.,, 2021; Huang and Bai,, 2021;
Awatramani,, 2021) NA 0.93 M-BERT + Focal Loss + Back Translation v1 0.7724
0.9294 M-BERT + Cross Entropy Loss 0.6686 0.9290 M-BERT + Focal Loss 0.7859
0.9277 M-BERT + Focal Loss + Contextual Augmentation 0.7438 0.9223 (Zhao and
Tao,, 2021) NA 0.92 aWe compare using Weighed Average F1 as Macro F1 is
unavailable.

Meanwhile, without any pre-processing, we can see that M-BERT with focal loss
produces results close to works of (Awatramani,, 2021; Huang and Bai,, 2021;
Mahajan et al.,, 2021) all of which some of the transformers. Our approach of
using Focal Loss with M-BERT still outperforms results of (Zhao and Tao,,
2021) which does complex processing, where it uses the attention mechanism to
adjust the weight of all the output layers. Among all the published works to
date on using M-BERT, our work outperforms them by a large margin, primarily
upon coupling with one or more strategies to handle data imbalance. To the
best of our knowledge, ours is the only model that achieves the best
performance with a minor change in pre-processing and doesn’t require the
addition of any attention modification, local features, or semantic
information. Besides, the best-performing model is away by a large margin
compared to the current state-of-the-art.

##  6 Conclusion and Outlook

In this work, we studied the problem of hope speech detection by revisiting
the issue of data imbalance from multiple perspectives. To this end, we
introduce formulated M-BERT baseline (section 5.1) and studied focal loss,
data augmentation strategies, and word removal pre-processing algorithms
(section 4). In the due process, we find each of these strategies to improve
the F1-Macro by 0.11, 0.10, and 0.17, respectively (section 5). This result is
the best performance achieved by a variant of BERT without complex
modifications to the best of our knowledge. We further evaluated each of these
strategies in detail to find various advantages with potential drawbacks.
Firstly, while we see focal loss to improve the results when coupled with
M-BERT, we find $\gamma$ to be highly influential, wherewith optimal values
the results are significantly enhanced (section 5.2). However, we do notice
that this also affects convergence and training time.

Meanwhile, we saw, the augmentation approaches have little impact on overall
results when coupled with focal loss. But with regular M-BERT, the results
seem to improve but are not comparable with Focal loss. Moreover, we find that
compared to contextual augmentation, back translation to helpful F1-Macro by
additional 2% and also french language is more suitable as an intermediary for
back-translation (section 5.3). We also emphasized the potential bias that is
introduced due to issues of word overlap in section 4.1, for which we
introduced a word removal algorithm. We find this method to outperform all the
other strategies by a considerable margin of 0.17 in F1-Macro and also helped
with both focal loss and cross-entropy loss (section 5.4). Finally, we
compared our work extensively with multiple current state-of-the-art methods.

Nevertheless, multiple aspects were unexplored, namely (a) ablation study of
the effect of features from one or more layers of M-BERT, (b) convergence
issues of focal loss and its relative contribution to errors in hope speech,
(c) impact of the number of words augmented in contextual data augmentation,
(d) relationship of intermediary language used for back translation, (e)
effect of word removal on context which we plan to visit in upcoming works. In
the future, we will also explore additional languages and code mixed data to
verify the effectiveness of suggested strategies. In addition, some confusion
remains, such as why loss function has no impact on performance when we use
word removal. Points wherein the conclusions are unclear are also planned to
be explored.

## Acknowledgement

Hariharan RamakrishnaIyer LekshmiAmmal was supported by the Ministry of Human
Resources and Development Ph.D. Fellowship, Government of India. Bharathi Raja
Chakravarthi was supported in part by a research grant from Science Foundation
Ireland (SFI) under Grant Number SFI/12/RC/2289$\\_$P2 (Insight$\\_$2), co-
funded by the European Regional Development Fund and Irish Research Council
grant IRCLA/2017/129 (CARDAMOM-Comparative Deep Models of Language for
Minority and Historical Languages).

Any opinions, findings, and conclusion or recommendations expressed in this
material are those of the authors only and does not reflect the view of their
employing organization or graduate schools. Manikandan Ravikiran did part of
the work on word overlap and focal loss using language models as part of the
complex assignment project in CS6460 (Spring 2020) at the OMSCS Program,
Georgia Institute of Technology, which was extended for the task of hope
speech detection by members of Department of Information Technology, National
Institute of Technology Karnataka, Surathkal, Mangalore, India.

## Conflict of Interest

The authors have no conflicts of interest to declare that are relevant to the
content of this article.

## Code Availability

The code is made available in https://github.com/hariharanrl/lre_hope_2021.

## References

  * Angiani et al., (2016) Angiani, G., Ferrari, L., Fontanini, T., Fornacciari, P., Iotti, E., Magliani, F., and Manicardi, S. (2016).  A comparison between preprocessing techniques for sentiment analysis in twitter.  In Armano, G., Bozzon, A., Cristani, M., and Giuliani, A., editors, Proceedings of the 2nd International Workshop on Knowledge Discovery on the WEB, KDWeb 2016, Cagliari, Italy, September 8-10, 2016, volume 1748 of CEUR Workshop Proceedings. CEUR-WS.org.
  * Auxier and Anderson, (2021) Auxier, B. and Anderson, M. (2021).  Social media use in 2021.
  * Awatramani, (2021) Awatramani, V. (2021).  Hopeful NLP@LT-EDI-EACL2021: Finding hope in YouTube comment section.  In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 164–167, Kyiv. Association for Computational Linguistics.
  * Babanejad et al., (2020) Babanejad, N., Agrawal, A., An, A., and Papagelis, M. (2020).  A comprehensive analysis of preprocessing for word representation learning in affective tasks.  In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. R., editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 5799–5810. Association for Computational Linguistics.
  * Balouchzahi et al., (2021) Balouchzahi, F., B K, A., and Shashirekha, H. L. (2021).  MUCS@LT-EDI-EACL2021:CoHope-hope speech detection for equality, diversity, and inclusion in code-mixed texts.  In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 180–187, Kyiv. Association for Computational Linguistics.
  * Bullinaria and Levy, (2012) Bullinaria, J. A. and Levy, J. (2012).  Extracting semantic representations from word co-occurrence statistics: stop-lists, stemming, and svd.  Behavior Research Methods, 44:890–907.
  * Camacho-Collados and Pilehvar, (2018) Camacho-Collados, J. and Pilehvar, M. T. (2018).  On the role of text preprocessing in neural network architectures: An evaluation study on text categorization and sentiment analysis.  In Linzen, T., Chrupala, G., and Alishahi, A., editors, Proceedings of the Workshop: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2018, Brussels, Belgium, November 1, 2018, pages 40–46. Association for Computational Linguistics.
  * Chakravarthi, (2020) Chakravarthi, B. R. (2020).  HopeEDI: A multilingual hope speech detection dataset for equality, diversity, and inclusion.  In Proceedings of the Third Workshop on Computational Modeling of People’s Opinions, Personality, and Emotion’s in Social Media, pages 41–53, Barcelona, Spain (Online). Association for Computational Linguistics.
  * Chakravarthi and Muralidaran, (2021) Chakravarthi, B. R. and Muralidaran, V. (2021).  Findings of the shared task on hope speech detection for equality, diversity, and inclusion.  In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 61–72, Kyiv. Association for Computational Linguistics.
  * Chawla et al., (2002) Chawla, N. V., Bowyer, K. W., Hall, L. O., and Kegelmeyer, W. P. (2002).  SMOTE: synthetic minority over-sampling technique.  J. Artif. Intell. Res., 16:321–357.
  * Chi et al., (2021) Chi, P., Chung, P., Wu, T., Hsieh, C., Chen, Y., Li, S., and Lee, H. (2021).  Audio albert: A lite bert for self-supervised learning of audio representation.  In IEEE Spoken Language Technology Workshop, SLT 2021, Shenzhen, China, January 19-22, 2021, pages 344–350. IEEE.
  * Conneau et al., (2020) Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2020).  Unsupervised cross-lingual representation learning at scale.  In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. R., editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 8440–8451. Association for Computational Linguistics.
  * Dave et al., (2021) Dave, B., Bhat, S., and Majumder, P. (2021).  IRNLP_DAIICT@LT-EDI-EACL2021: Hope speech detection in code mixed text using TF-IDF char n-grams and MuRIL.  In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 114–117, Kyiv. Association for Computational Linguistics.
  * Devlin et al., (2019) Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2019).  BERT: pre-training of deep bidirectional transformers for language understanding.  In Burstein, J., Doran, C., and Solorio, T., editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics.
  * Dong et al., (2019) Dong, Q., Gong, S., and Zhu, X. (2019).  Imbalanced deep learning by minority class incremental rectification.  IEEE Trans. Pattern Anal. Mach. Intell., 41(6):1367–1381.
  * Ebert et al., (2016) Ebert, S., Müller, T., and Schütze, H. (2016).  LAMB: A good shepherd of morphologically rich languages.  In Su, J., Carreras, X., and Duh, K., editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 742–752. The Association for Computational Linguistics.
  * Fadaee et al., (2017) Fadaee, M., Bisazza, A., and Monz, C. (2017).  Data augmentation for low-resource neural machine translation.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 567–573, Vancouver, Canada. Association for Computational Linguistics.
  * Fedus et al., (2018) Fedus, W., Goodfellow, I. J., and Dai, A. M. (2018).  Maskgan: Better text generation via filling in the _______.  In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
  * Gan et al., (2020) Gan, L., Teng, Z., Zhang, Y., Zhu, L., Wu, F., and Yang, Y. (2020).  Semglove: Semantic co-occurrences for glove from BERT.  CoRR, abs/2012.15197.
  * Goodfellow et al., (2014) Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. C., and Bengio, Y. (2014).  Generative adversarial nets.  In NIPS.
  * Guo et al., (2019) Guo, H., Mao, Y., and Zhang, R. (2019).  Augmenting data with mixup for sentence classification: An empirical study.  CoRR, abs/1905.08941.
  * Huang and Bai, (2021) Huang, B. and Bai, Y. (2021).  TEAM HUB@LT-EDI-EACL2021: Hope speech detection based on pre-trained language model.  In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 122–127, Kyiv. Association for Computational Linguistics.
  * Kafle et al., (2017) Kafle, K., Yousefhussien, M. A., and Kanan, C. (2017).  Data augmentation for visual question answering.  In Alonso, J. M., Bugarín, A., and Reiter, E., editors, Proceedings of the 10th International Conference on Natural Language Generation, INLG 2017, Santiago de Compostela, Spain, September 4-7, 2017, pages 198–202. Association for Computational Linguistics.
  * Karlgren and Sahlgren, (2001) Karlgren, J. and Sahlgren, M. (2001).  From words to understanding.  In Foundations of Real-World Intelligence :, pages 294–308. CSLI Publications, 1 edition.
  * Khanuja et al., (2021) Khanuja, S., Bansal, D., Mehtani, S., Khosla, S., Dey, A., Gopalan, B., Margam, D. K., Aggarwal, P., Nagipogu, R. T., Dave, S., Gupta, S., Gali, S. C. B., Subramanian, V., and Talukdar, P. P. (2021).  Muril: Multilingual representations for indian languages.  CoRR, abs/2103.10730.
  * Kim et al., (2020) Kim, J., Jeong, J., and Shin, J. (2020).  M2m: Imbalanced classification via major-to-minor translation.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 13893–13902. Computer Vision Foundation / IEEE.
  * Kingma and Ba, (2015) Kingma, D. P. and Ba, J. (2015).  Adam: A method for stochastic optimization.  In Bengio, Y. and LeCun, Y., editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
  * (28) Kobayashi, S. (2018a).  Contextual augmentation: Data augmentation by words with paradigmatic relations.  In Walker, M. A., Ji, H., and Stent, A., editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), pages 452–457. Association for Computational Linguistics.
  * (29) Kobayashi, S. (2018b).  Contextual augmentation: Data augmentation by words with paradigmatic relations.  CoRR, abs/1805.06201.
  * Kramer et al., (2014) Kramer, A. D. I., Guillory, J. E., and Hancock, J. T. (2014).  Experimental evidence of massive-scale emotional contagion through social networks.  Proceedings of the National Academy of Sciences, 111(24):8788–8790.
  * Krawczyk, (2016) Krawczyk, B. (2016).  Learning from imbalanced data: open challenges and future directions.  Prog. Artif. Intell., 5(4):221–232.
  * Kuznetsov and Gurevych, (2018) Kuznetsov, I. and Gurevych, I. (2018).  From text to lexicon: Bridging the gap between word embeddings and lexical resources.  In Proceedings of the 27th International Conference on Computational Linguistics, pages 233–244, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Li et al., (2020) Li, B., Zhou, H., He, J., Wang, M., Yang, Y., and Li, L. (2020).  On the sentence embeddings from pre-trained language models.  In Webber, B., Cohn, T., He, Y., and Liu, Y., editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 9119–9130. Association for Computational Linguistics.
  * Lin et al., (2020) Lin, T., Goyal, P., Girshick, R. B., He, K., and Dollár, P. (2020).  Focal loss for dense object detection.  IEEE Trans. Pattern Anal. Mach. Intell., 42(2):318–327.
  * M K and A P, (2021) M K, J. and A P, A. (2021).  KU_NLP@LT-EDI-EACL2021: A multilingual hope speech detection for equality, diversity, and inclusion using context aware embeddings.  In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 79–85, Kyiv. Association for Computational Linguistics.
  * Mahajan et al., (2021) Mahajan, K., Al-Hossami, E., and Shaikh, S. (2021).  TeamUNCC@LT-EDI-EACL2021: Hope speech detection using transfer learning with transformers.  In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 136–142, Kyiv. Association for Computational Linguistics.
  * Mikolov et al., (2013) Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013).  Efficient estimation of word representations in vector space.  In Bengio, Y. and LeCun, Y., editors, 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings.
  * Nguyen et al., (2011) Nguyen, H. M., Cooper, E. W., and Kamei, K. (2011).  Borderline over-sampling for imbalanced data classification.  Int. J. Knowl. Eng. Soft Data Paradigms, 3(1):4–21.
  * Pecar et al., (2018) Pecar, S., Farkas, M., Simko, M., Lacko, P., and Bieliková, M. (2018).  NL-FIIT at IEST-2018: emotion recognition utilizing neural networks and multi-level preprocessing.  In Balahur, A., Mohammad, S. M., Hoste, V., and Klinger, R., editors, Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, WASSA@EMNLP 2018, Brussels, Belgium, October 31, 2018, pages 217–223. Association for Computational Linguistics.
  * Pennington et al., (2014) Pennington, J., Socher, R., and Manning, C. D. (2014).  Glove: Global vectors for word representation.  In Moschitti, A., Pang, B., and Daelemans, W., editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532–1543. ACL.
  * Pires et al., (2019) Pires, T., Schlinger, E., and Garrette, D. (2019).  How multilingual is multilingual bert?  In Korhonen, A., Traum, D. R., and Màrquez, L., editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4996–5001. Association for Computational Linguistics.
  * Puranik et al., (2021) Puranik, K., Hande, A., Priyadharshini, R., Thavareesan, S., and Chakravarthi, B. R. (2021).  IIITT@LT-EDI-EACL2021-hope speech detection: There is always hope in transformers.  In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 98–106, Kyiv. Association for Computational Linguistics.
  * Que, (2021) Que, Q. (2021).  Simon @ LT-EDI-EACL2021: Detecting hope speech with BERT.  In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 175–179, Kyiv. Association for Computational Linguistics.
  * Ribeiro et al., (2016) Ribeiro, M. T., Singh, S., and Guestrin, C. (2016).  ”why should I trust you?”: Explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 1135–1144.
  * S et al., (2021) S, A., Ramakrishnan, A., Balaji, A., D., T., and B, S. K. (2021).  ssn_diBERTsity@LT-EDI-EACL2021:hope speech detection on multilingual YouTube comments via transformer based approach.  In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 92–97, Kyiv. Association for Computational Linguistics.
  * Saif et al., (2014) Saif, H., Fernández, M., He, Y., and Alani, H. (2014).  On stopwords, filtering and data sparsity for sentiment analysis of twitter.  In Calzolari, N., Choukri, K., Declerck, T., Loftsson, H., Maegaard, B., Mariani, J., Moreno, A., Odijk, J., and Piperidis, S., editors, Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, Reykjavik, Iceland, May 26-31, 2014, pages 810–817. European Language Resources Association (ELRA).
  * Sanh et al., (2019) Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019).  Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter.  CoRR, abs/1910.01108.
  * Saumya and Mishra, (2021) Saumya, S. and Mishra, A. K. (2021).  IIIT_DWD@LT-EDI-EACL2021: Hope speech detection in YouTube multilingual comments.  In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 107–113, Kyiv. Association for Computational Linguistics.
  * Sennrich et al., (2016) Sennrich, R., Haddow, B., and Birch, A. (2016).  Improving neural machine translation models with monolingual data.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
  * Sharma and Arora, (2021) Sharma, M. and Arora, G. (2021).  Spartans@LT-EDI-EACL2021: Inclusive speech detection using pretrained language models.  In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 188–192, Kyiv. Association for Computational Linguistics.
  * Trask et al., (2015) Trask, A., Michalak, P., and Liu, J. (2015).  sense2vec - A fast and accurate method for word sense disambiguation in neural word embeddings.  CoRR, abs/1511.06388.
  * Vaswani et al., (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017).  Attention is all you need.  CoRR, abs/1706.03762.
  * Wang et al., (2020) Wang, T., Liu, L., Zhang, H., Zhang, L., and Chen, X. (2020).  Joint character-level convolutional and generative adversarial networks for text classification.  Complex., 2020:8516216:1–8516216:11.
  * Wang and Jurgens, (2018) Wang, Z. and Jurgens, D. (2018).  It’s going to be okay: Measuring access to support in online communities.  In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J., editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 33–45. Association for Computational Linguistics.
  * Wei and Zou, (2019) Wei, J. W. and Zou, K. (2019).  EDA: easy data augmentation techniques for boosting performance on text classification tasks.  In Inui, K., Jiang, J., Ng, V., and Wan, X., editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 6381–6387. Association for Computational Linguistics.
  * Xia et al., (2017) Xia, Y., Qin, T., Chen, W., Bian, J., Yu, N., and Liu, T. (2017).  Dual supervised learning.  CoRR, abs/1707.00415.
  * Zhao and Gui, (2017) Zhao, J. and Gui, X. (2017).  Comparison research on text pre-processing methods on twitter sentiment analysis.  IEEE Access, 5:2870–2879.
  * Zhao and Tao, (2021) Zhao, Y. and Tao, X. (2021).  ZYJ@LT-EDI-EACL2021:XLM-RoBERTa-based model with attention for hope speech detection.  In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 118–121, Kyiv. Association for Computational Linguistics.
  * Ziehe et al., (2021) Ziehe, S., Pannach, F., and Krishnan, A. (2021).  GCDH@LT-EDI-EACL2021: XLM-RoBERTa for hope speech detection in English, Malayalam, and Tamil.  In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 132–135, Kyiv. Association for Computational Linguistics.