# BioALBERT: A Simple and Effective
Pre-trained Language Model for
Biomedical Named Entity Recognition

Usman Naseem1, Matloob Khushi2 Vinay Reddy3, Sakthivel Rajendran4, Imran
Razzak5 and Jinman Kim6  12356School of Computer Science, University of
Sydney, Australia
4School of Information Technology, Deakin University, Australia
Email:12346FirstName.SecondName<EMAIL_ADDRESS>
###### Abstract

In recent years, with the growing amount of biomedical documents, coupled with
advancement in natural language processing algorithms, the research on
biomedical named entity recognition (BioNER) has increased exponentially.
However, BioNER research is challenging as NER in the biomedical domain are:
(i) often restricted due to limited amount of training data, (ii) an entity
can refer to multiple types and concepts depending on its context and, (iii)
heavy reliance on acronyms that are sub-domain specific. Existing BioNER
approaches often neglect these issues and directly adopt the state-of-the-art
(SOTA) models trained in general corpora which often yields unsatisfactory
results. We propose biomedical ALBERT (A Lite Bidirectional Encoder
Representations from Transformers for Biomedical Text Mining) – bioALBERT – an
effective domain-specific language model trained on large-scale biomedical
corpora designed to capture biomedical context-dependent NER. We adopted a
self-supervised loss used in ALBERT that focuses on modelling inter-sentence
coherence to better learn context-dependent representations and incorporated
parameter reduction techniques to lower memory consumption and increase the
training speed in BioNER. In our experiments, BioALBERT outperformed
comparative SOTA BioNER models on eight biomedical NER benchmark datasets with
four different entity types. We trained four different variants of BioALBERT
models which are available for the research community to be used in future
research.

##  I Introduction

The growing volume of the published biomedical literature, such as clinical
reports [1] and health literacy [2], are fuelling the advancements in the
development of text mining algorithms. Biomedical named entity recognition
(BioNER) intends to automatically identify biomedical entities such as
diseases, chemicals, genes and proteins, etc., from the biomedical literature.
So, a crucial step towards this aim is to build better and effective methods
which can automatically recognize and extract biomedical entities. BioNER is
an essential building block of many downstream text mining applications such
as extracting drug-to-drug interactions [3] and disease-treatment [4]
relationships. Traditionally, BioNER relies on feature engineering methods
(e.g., lexicon-based, rules-based and statistics-based). However, feature
engineering is dependent on domain-specific knowledge which does not perform
well on BioNER [5].

Deep learning (DL) with its ability to automatically extract features have
become common in BioNER recently [6]. For instance, Long Short-Term Memory
(LSTM) is usually employed to learn vector representations of each word in a
sentence, and then as the input to conditional random fields (CRF) have
greatly improved the performance in BioNER [6]. Recently state-of-the-art
(SOTA) DL based language models such as Embeddings from Language Models (ELMo)
[7], Bidirectional Encoder Representations from Transformers (BERT) [8] and (A
Lite Bidirectional Encoder Representations from Transformers (ALBERT) [9]
obtained SOTA best performance on many NLP tasks.

Although these models show promising results, but applying them on BioNER has
multiple challenges and limitations: (i) a limited amount of training data;
(ii) an entity could represent multiple entity types depending on its textual
context, i.e., a “BRCA1” can be referred as gene name as well as a disease
entity depending on its context. Similarly, “heart attack” and “myocardial
infarction” refer to the same concept and, (iii) the heavy use of acronyms in
biomedical texts makes it challenging to identify concepts, i.e., the
abbreviation ”RA” may refer to “right atrium,” “rheumatoid arthritis,” or one
of several other concepts, where the resolution of the abbreviation is,
therefore, context-dependent. Therefore, current models in BioNER rely on
various context-independent and transformer-based context-dependent language
models which are trained on biomedical corpora [10, 11, 12].

Figure 1:  Overview of the pre-training and fine-tuning of BioALBERT on NER

To overcome the identified limitations, we present Biomedical ALBERT –
bioALBERT - a context-dependent, fast and effective language model that
addresses the shortcomings of recently proposed domain-specific language
models. BioALBERT is trained on large biomedical corpora which address the
limitation of limited training data. We also innovate in the adoption of
cross-layer parameter sharing by learning parameters for the first block and
reuse the block in the remaining layers instead of learning unique parameters
for each of the layers and sentence-order-prediction (SOP) technique as a
measure of coherence loss between sentences. SOP takes two consecutive
sentences from training data and creates a random pair from different
sentences which helps the model to learn better representations and finally,
in BERT based models the size of the embedding was linked to the hidden layer
sizes of the transformer blocks. These embeddings are projected directly to
the hidden space of the hidden layer whereas in our model we use factorized
embedding parameterization which decomposes embedding matrix into two small
matrices which separate the size of the hidden layers from the size of
vocabulary embeddings. This allows for increasing the hidden size without
significantly increasing the parameter size of the vocabulary embeddings.
BioALBERT is simple and efficient when fine-tuned for BioNER task as compared
to other SOTA models. We evaluate our model on eight biomedical named entity
recognition benchmark datasets. Our pre-trained BioNER models, along with the
source code, will be publicly available.

##  II Related Work

###  II-A Language Model

In biomedical text mining research, there is a long history of using shared
language representations to capture the semantics of the text. One established
trend is a form of word-embeddings [13] that represent syntactic and semantic
meaning, and map words into low-dimensional vectors. Similar methods also have
been derived for improving embeddings of word sequences by introducing
sentence embeddings [14]. These context-independent word embeddings approach
such as word2vec [13] were trained on biomedical corpora that contain terms
and expressions that are usually not included in a general domain corpus [10].
These methods always require complex neural networks to be effectively used
and model context-independent representations.

Another common trend, particularly in recent years, is text representation
based on context [7, 8, 9]. Unlike traditional word-embeddings, this enables a
word to change its meaning depending on the context in which it occurs.
Several other works have investigated the usefulness of contextual models in
clinical and biomedical domains. Several researchers trained ELMo on
biomedical corpora and presented BioELMo and found that BioELMo beats ELMo on
BioNER tasks [11, 16]. A pre-trained BioELMo model was released along with
their work, allowing more clinical research in the area of BioNER. Beltagy et
al. [15] released Scientific BERT (SciBERT), where BERT was trained on the
scientific texts. BERT has typically been superior and better than ELMo to
non-contextual embeddings.

Si et al. [17], trained the BERT on clinical notes corpora, using complex
task-specific models to improve both traditional embedding and ELMo embedding
on the i2b2 2010 and 2012 BioNER. Similarly in another study, a new domain-
specific language model, BioBERT [12], trained a BERT model on a corpus of
biomedical articles from PMC abstracts111https://www.ncbi.nlm.nih.gov/pmc/ as
well as full texts sourced from PubMed222https://www.ncbi.nlm.nih.gov/pubmed/
which gave rise to enhanced performance on BioNER. Peng et al [18] introduced
Biomedical Language Understanding Evaluation (BLUE), a collection of resources
for evaluating and analysing biomedical natural language representation
models. Their study also confirmed that the BERT models pre-trained on PubMed
abstracts and clinical notes see better performance than most state-of-the-art
models.

Despite this success, BERT has some limitations such as BERT has a huge number
of parameters which is the cause for problems like degraded pre-training time,
memory management issues and model degradation etc [9]. These issues are very
well addressed in ALBERT, which is modified based on the architecture of BERT
and proposed by Lan et al. [9]. ALBERT incorporates two-parameter reduction
techniques that lift the significant obstacles in scaling pre-trained models.
(i) Factorized embedding parameterization - decomposes the large vocabulary
embedding matrix into two small matrices, (ii) replaces the NSP loss by SOP
loss; and (iii) Cross-layer parameter sharing- prevents the parameter from
growing with the depth of the network. These techniques significantly reduce
the number of parameters used when compared with BERT without significantly
affecting the performance of the model, thus improving parameter-efficiency.
An ALBERT configuration similar to BERT-large has 18x fewer parameters and can
be trained about 1.7x faster.

###  II-B Fine tuning

Fine-tuning has been successfully used to transfer pre-trained weights as
initialisation for parameters for various downstream tasks [19]. This improves
the efficiency of the target task when we have limited data and similar tasks
[20]. Recently, fine-tuning of pre-trained language models have been widely
used in various text mining tasks [21]. Liu et al. [22] pre-trained LSTM with
a language model and fine-tuned it, and this has contributed to improved
performance for various text classification tasks.

Universal Language Model Fine-tuning (ULMFiT) used general-domain pre-training
and fine-tuning techniques to avoid over-fitting and achieved SOTA performance
on the datasets with less samples [19]. Similarly, authors of BERT, ALBERT,
BioBERT and others tested the performance of their model for a wide range of
tasks with minimum fine-tuning efforts and achieved good performance.
Recently, multi-task fine-tuning has led to improvements even with many target
tasks [23]. In this paper, we fine-tuned our pre-trained language model
BioALBERT, which is trained on biomedical corpora for BioNER. BioALBERT offers
better performance on BioNER as it addresses the shortcomings of BERT used in
BioBERT. However, this is not a trivial task as it requires several
optimisations which are discussed in the next section. Weights of our pre-
trained model along with complete source code, will be available online.

TABLE I: Summary of parameters used in the Pre-training Summary of All
parameters used: (Pre-Training)
---
Name | BioALBERT 1.0 | BioALBERT 1.1
Architecture | ALBERT Base | ALBERT Large
Activation function | GeLU | GeLU
Number of Attention Heads | 12 | 16
Number of layers | 12 | 24
Hidden Size | 768 | 1024
Embedding Size | 128 | 128
Vocab Size | 30000 | 30000
Optimizer | LAMB | LAMB
Train Batch Size | 1024 | 256
Eval Batch Size | 16 | 16
Max Seq Length | 512 | 512
Max Predictions per Seq | 20 | 20
Learning Rate | 0.00176 | 0.00062
Training Steps (PubMed) | 200K | 200K
Training Steps (PMC) | 270K | 270K
Warmup Steps | 3125 | 3125

##  III Methodology

First, we initialized BioALBERT with weights from ALBERT. BioALBERT is pre-
trained on biomedical domain corpora (PubMed abstracts and PMC full-text
articles). To show the effectiveness of our approach, BioALBERT is fine-tuned
on BioNER. We tested various pre-training strategies with different
combinations and sizes of biomedical corpora. The overview of our methodology
is shown in Fig. 1. Below we present details of each step involved in pre-
training and fine-tuning of BioALBERT.

###  III-A Pre-training of BioALBERT

In this section, we present steps involved in the pre-training of BioALBERT,
which has the same architecture as ALBERT, which makes it simple. BioALBERT is
trained on PubMed abstracts and PMC full-text articles which contains
biomedical terms and enables to pre-train the ALBERT model, which was pre-
trained on the general text on biomedical corpora. As we cannot use the raw
text biomedical corpora (PubMed and PMC) to pre-train the model, so we
converted raw text to structured format by converting raw text files into a
single sentence in which; (i) all blank lines are removed within a document
and converted it into on single paragraph; (ii) Any line less than
20-character length is removed and (iii) there will be a blank line between
each document (when combining the multiple files) for training the model.
Overall, PubMed contains approximately 4.5 Billion words, and PMC contains
about 13.5 Billion words. As an initial step, we have initialized the model
weights from ALBERT to create BioALBERT model by pre-training on biomedical
corpora and kept the original vocabulary of ALBERT for pre-training.

TABLE II: BioALBERT Pre-trained Models

Model Version | Model | Trained On | # of words | Machine Used | Batch Size | Steps
---|---|---|---|---|---|---
BioALBERT 1.0 | Base | PubMed | 4.3B | GCP TPU v3-8 | 1024 | 200K
BioALBERT 1.0 | Base | PubMed+PMC | 18.8B | GCP TPU v3-8 | 1024 | 470K
BioALBERT 1.1 | Large | PubMed | 4.3B | GCP TPU v3-8 | 256 | 200K
BioALBERT 1.1 | Large | PubMed+PMC | 18.8B | GCP TPU v3-8 | 256 | 470K

Sentence embeddings are used for tokenization, and for that, we pre-processed
the data as a sentence text. So every line in the input text document is a
sentence, and an empty line separates every document. We set the maximum
length of each sentence to 512 words. Shorter sentences were padded to make
512 whereas longer sentences were truncated. Learning rate of 0.00176 and
number of warm-up steps as 3125 for all the pre-trained versions of the model,
except for BioALBERT 1.1 (PubMed +PMC) where the learning rate has been re-
adjusted to 0.00062 (by re-scaling the current learning rate with scaled
$1/2^{1.5}$ times to avoid the problem of exploding loss. Summary of
parameters used in the training process is given in Table I.

We experimented with different settings and found out that both base and large
models were successful with larger batch size on V3-8 TPU compute instances.
We have used two different embedding sizes, i.e., 128 and 256. The 128
embedding size creates a base model with 12 Million parameters, whereas a
large model has 16 Million parameters with 256 embedding size. With these
combinations, we have presented a total of four models, given in Table II.

###  III-B Fine-tuning of BioALBERT

Fine-tuning of BioALBERT on BioNER task is presented in this section. BioNER
involves annotating words in a sentence as named-entities. The labelled
datasets that were used for this task include four categories representing
Disease, Species, Drug/Proteins, and Drugs/Chemicals. The objective is to
train and make a prediction on the labels, which are the proper nouns within
the domain area. More formally, given an input sentence
$S=\\{x_{1},x_{2},...,x_{z}\\}$, where $x_{i}$ is the $i$-th word and $z$
represents the length of the sentence. The goal of BioNER is to classify each
word in $S$ and assign it to a corresponding label $y\in Y$, where $Y$ is a
predefined list of all possible label.

Fine-tuning is simple as compared to the pre-training, and the computational
requirements are also not that significant. BioALBERT which takes less
physical memory and improvised parameter sharing techniques. The BioNER fine-
tuning is trained to learn the word embeddings using the sentence piece
tokenization while the BioBERT model was based on the word piece embeddings.
For each of the pre-trained models, we constructed a fine-tuning task by using
the specific dataset.

The model setup uses the weights of the pre-trained model that are created
previously. We have used 1e-5 learning rate, batch size as 32 and lower case
texts and finally fine-tuned for 5336 steps. Pre-trained BioALBERT models were
pre-trained using TPUv3-8. All of the hyper-parameters used are same as
default ALBERT unless stated otherwise. All the tested datasets contain a list
of words along with a label “B”, “I”, and “O” where “B” denotes the beginning
of an entity, “I” stands for “inside” and is used for all words comprising the
entity except the first one, and “O” means the absence of an entity. For our
experiments, we used these datasets as-is and passed to it our pre-trained
models for the downstream task. The adamw optimiser was used with evaluation
checkpoint so that the model will be evaluated at different time intervals
using the holdout development dataset to identify the best model for final
predictions. The predictions were performed on the test datasets, and the
performance is compared with baseline models that were established previously
by calculating the F1 Score, Precision and Recall. The summary of all
parameters used in fine-tuning is given in Table III.

TABLE III: Summary of parameters used in fine-tuning Summary of All parameters
used: (Fine Tuning)
---
Name | BioNER
Optimizer | adamw
Train Batch Size | 32
Eval Batch Size | 16
Save Checkpoint | 200
Max Seq Length | 512
Learning Rate | 1.00E-05
Training Steps | 5336
Warmup Steps | 320

##  IV Experimental Analysis

In this section, we present the dataset used, baselines and results to show
the effectiveness of our model.

###  IV-A Datasets

Our model is evaluated on eight biomedical NER benchmark datasets which
contain four types of entities and provided by Lee et al. [12]. Table IV shows
the statistics of the datasets used. Below we briefly explained each dataset:

  * •

BC5CDR: The Bio-creative community challenge for the chemical-disease relation
extraction task (BC5CDR) the corpus was made available in a Bio-creative
workshop [24]. The two sub-tasks of BC5CDR are identifying; (i) chemical and
(ii) disease entities from Medline abstracts.

  * •

BC4CHEMD: This dataset is provided by Bio-Creative community challenge IV for
the development and evaluation of tools for Chemical NER [25]. BC4CHEMDNER was
used for the recognition of chemical compounds and drugs from Pubmed
abstracts.

  * •

NCBI Disease: To promote disease NER-system research, American National
Institutes of Health released the NCBI disease corpus for disease NER-
research. The NCBI disease corpus is large-scale and high-quality; it is based
on the corpus released by Leaman et al. [26].

  * •

JNLPBA: We also used the JNLPBA corpus in our experiments which are provided
by Kim et al. [27]. This corpus contains five entity types, including DNA,
RNA, Cell Type, Cell Line and Protein.

  * •

BC2GM: BC2GM is provided by Ando [28], the state-of-the-art system in the Bio-
Creative II gene mention recognition task is a semi-supervised learning method
using alternating structure optimization.

  * •

LINNAEUS: The LINNAEUS corpus was provided by Gerner et al. [29] which
consists of 100 full-text documents from the PMC Open access document set
which were randomly selected. All mentions of species terms were manually
annotated and normalized to the NCBI taxonomy IDs of the intended species.

  * •

Species-800: Species-800, which is also known as S800 [30], is a novel
abstract-based manually annotated corpus. S800 comprises 800 PubMed abstracts
in which organism mentions were identified and mapped to the corresponding
NCBI taxonomy identifiers.

TABLE IV: Statistics of the BioNER datasets Entitity Type | Dataset | # of Annotations
---|---|---
Disease | NCBI Disease | 6,881
BC5CDR | 12,694
Drug/Chem | BC5CDR | 15,411
BC4CHEMD | 79,842
Drug/Protein | BC2GN | 20,703
JNLPBA | 35,460
SPECIES | LINNAEUS | 4,077
Species-800 | 3,708
TABLE V: Comparison of performance in biomedical named entity recognition
(BioNER) task.

Type | Datasets | Metrics | SOTA | BioBERT v1.0 | BioBERT v1.0 | BioBERT v1.0 | BioBERTv1.1 | BioALBERT 1.0 | BioALBERT 1.0 | BioALBERT 1.1 | BioALBERT 1.1
---|---|---|---|---|---|---|---|---|---|---|---
(PubMed) | (PMC) | (PubMed+PMC) | (PubMed) | (PubMed) | (PubMed+PMC) | (PubMed) | (PubMed+PMC)
Base | Base | Base | Base | Base | Base | Large | Large
Disease | NCBI Disease | p | 88.30 | 86.76 | 86.16 | 89.04 | 88.22 | 97.45 | 96.84 | 97.18 | 97.38
R | 89.00 | 88.02 | 89.48 | 89.69 | 91.25 | 94.39 | 94.40 | 97.18 | 94.37
F | 88.60 | 87.38 | 87.79 | 89.36 | 89.71 | 95.89 | 95.61 | 97.18 | 95.85
BC5CDR | p | 89.61 | 85.80 | 84.67 | 85.86 | 86.47 | 99.69 | 99.11 | 99.27 | 99.39
R | 83.09 | 86.60 | 85.87 | 87.27 | 87.84 | 95.72 | 96.17 | 96.33 | 95.85
F | 86.23 | 86.2 | 85.27 | 86.56 | 87.15 | 97.66 | 97.62 | 97.78 | 97.61
Drug/Chem | BC5CDR | p | 94.26 | 92.52 | 92.46 | 93.27 | 93.68 | 99.99 | 99.99 | 99.99 | 99.99
R | 92.38 | 92.76 | 92.63 | 93.61 | 93.26 | 95.89 | 96.24 | 95.62 | 95.68
F | 93.31 | 92.64 | 92.54 | 93.44 | 93.47 | 97.9 | 98.08 | 97.76 | 97.79
BC4CHEMD | p | 92.29 | 91.77 | 91.65 | 92.23 | 92.80 | 97.76 | 97.71 | 97.71 | 97.88
R | 90.01 | 90.77 | 90.30 | 90.61 | 91.92 | 94.22 | 94.83 | 94.83 | 94.63
F | 91.14 | 91.26 | 90.97 | 91.41 | 92.36 | 95.96 | 96.25 | 96.25 | 96.23
Drug/Protein | BC2GM | p | 81.81 | 81.72 | 82.86 | 85.16 | 84.32 | 97.86 | 97.84 | 98.26 | 98.02
R | 81.57 | 83.38 | 84.21 | 83.65 | 85.12 | 94.87 | 94.27 | 95.72 | 94.70
F | 81.69 | 82.54 | 83.53 | 84.40 | 84.72 | 96.34 | 96.02 | 96.97 | 96.33
JNLPBA | p | 74.43 | 71.11 | 71.17 | 72.68 | 72.24 | 85.14 | 85.60 | 86.23 | 85.56
R | 83.22 | 83.11 | 82.76 | 83.21 | 83.56 | 80.43 | 80.98 | 81.90 | 81.49
F | 78.58 | 76.65 | 76.53 | 77.59 | 77.49 | 82.72 | 83.22 | 84.01 | 83.53
Species | LINNAEUS | p | 92.80 | 91.83 | 91.62 | 93.84 | 90.77 | 99.95 | 99.98 | 99.98 | 99.92
R | 94.29 | 84.72 | 85.48 | 86.11 | 85.83 | 99.47 | 99.44 | 99.48 | 99.55
F | 93.54 | 88.13 | 88.45 | 89.81 | 88.24 | 99.71 | 99.72 | 99.73 | 99.73
Species-800 | p | 74.34 | 70.60 | 71.54 | 72.84 | 72.80 | 99.17 | 98.93 | 99.10 | 98.75
R | 75.96 | 75.75 | 74.71 | 77.97 | 75.36 | 98.34 | 98.04 | 98.95 | 98.69
F | 74.98 | 73.08 | 73.09 | 75.31 | 74.06 | 98.76 | 98.49 | 99.02 | 98.72

Notes: Precision (P), Recall (R) and F1 (F) scores on each dataset are
reported. The best scores are in bold, and the second best scores are
underlined. We list the scores of the state-of-the-art (SOTA) models on
different datasets as follows: scores of Xu et al. [31] on NCBI Disease,
scores of Sachan et al. [32] on BC2GM, scores of Lou et al. [33] on BC5CDR-
disease, scores of Luo et al. [34] on BC4CHEMD, scores of Yoon et al. [35] on
BC5CDR-chemical and JNLPBA and scores of Giorgi and Bader [36] on LINNAEUS and
Species-800

###  IV-B Baselines

To assess the performance of the proposed method, an exhaustive comparison is
performed with several advanced SOTA methodologies along with their published
results333The published results were acquired from respective original
publication. Our model is compared with the following methods.

Yoon et al. [35] introduced CollaboNet, which consists of multiple BiLSTM-CRF
models, for BioNER. While existing models were only able to handle datasets
with a single entity type, CollaboNet leverages multiple datasets and achieves
the highest F1 scores. CollaboNet is built upon multiple single-task NER
models (STMs) that send information to each other for more accurate
predictions.

Lou et al. [33] proposed a transition-based model for joint disease entity-
recognition and normalization, based on transition-based structured prediction
framework using structured perceptron with early-update training and beam-
search decoding. In another study, Lou et al. [34] proposed a neural network
approach, i.e. attention-based bidirectional Long Short-Term Memory with a
conditional random field layer (Att-BiLSTM-CRF), to document level chemical
NER. The method leverages document-level global information obtained by
attention mechanism to enforce tagging consistency across multiple instances
of the same token in a document. It achieves better performances with little
feature engineering.

Xu et al. [31] proposed a novel dictionary-based and document-level attention
mechanism with a deep neural network NER method, named as DABLC. DABLC tags
the consistency of multiple instances in a document at the document level and
combines an external disease dictionary that is constructed with five disease
resources containing a rich collection of disease entities. The authors
adopted the efficient exact string matching method for dictionary matching;
this method can effectively and accurately match the disease names.

Lee et al. [12] introduced BioBERT, which is a pre-trained language model for
biomedical text mining. Authors showed that pre-training BERT on biomedical
corpora is crucial in applying it to the biomedical domain. BioBERT
outperforms previous models on biomedical text mining tasks such as NER, RE
and QA. We compare BioALBERT with both BioBERT (v1.0 and V1.1) models and
other SOTA models used in BioNER task. We selected those methods because they
are the SOTA, and based on the conducted meta-analysis exhibit the highest
performance among the techniques so far developed.

###  IV-C Results

Figure 2: Comparison of a) BioBERT v1.0 (Pubmed)-Base v/s BioALBERT 1.0
(Pubmed)-Base ; b) BioBERT v1.0(Pubmed+PMC)-Base v/s BioALBERT
1.0(Pubmed+PMC)-Base

Table V presents the performance of all the variants of BioALBERT and
contrasts them to baseline methodologies. Our model outperforms all other
methods on all eight datasets and entity types. For, (i) Disease-type
datasets, BioALBERT improved the performance by 7.47% and 10.63% for NCBI-
Disease and BC5CDR-Disease datasets respectively; (ii) Drug/Chem type datasets
increase in performance by 4.61% and 3.89% for BC5CDR-Chem and BC4CHEMD
datasets respectively; (iii) Gene/Protein type datasets increase in
performance by 12.25% and 6.42% for BC2GM and JNLPBA datasets respectively
and; (iv) Species type datasets increase in performance by 6.19% and 23.71%
respectively is observed.

We have performed multiple comparisons to analyse the effectiveness of
BioALBERT. Fig. 2 presents, the performance comparison of the same versions
(trained on same corpora and for the same number of steps) of both BioALBERT
and BioBERT. We can see that in Fig. (2a), we compared BioBERT v1.0-base model
which is trained on PubMed with BioALBERT 1.0-base model trained on Pubmed and
similarly in Fig. (2b), we compared BioBERT v1.0-base model trained on Pubmed
and PMC with BioALBER 1.0-base model trained on PubMed and PMC biomedical
corpora. In both cases, BioALBERT outperformed BioBERT on all eight datasets.
We also compared the training time of BioALBERT with BioBERT. We found that
all models BioALBERT beat BioBERT with a considerable margin. The run time
statistics of both pre-trained models are given in Fig. 3.

Figure 3: Comparison of run-time statistics of BioALBERT v/s BioBERT

###  IV-D Discussions

BioALBERT gives better performance and addresses the previously mentioned
challenges in the biomedical domain. We attribute this to the BioALBERT built
on top of the transformer-based language model that learns contextual
relations between words (or subwords) in a text. As opposed to directional
models, which read the text input sequentially (left-to-right or right-to-
left), the transformer encoder reads the entire sequence of words at once.
This characteristic allows the model to learn the context of a word based on
all of its surroundings (left and right of the word) and address the issue of
contextual representation.

Our model addresses the shortcomings of BERT based biomedical models. At
first, BioALBERT uses cross-layer parameter sharing and reduces 110 million
parameters of 12-layer BERT-base model to 31 million parameters while keeping
the same number of layers and hidden units by learning parameters for the
first block and reuse the block in the remaining 11 layers. Secondly, our
model uses the SOP, which takes two segments from the training corpus that
appear consecutively and constructs a random pair of segments from different
documents. This enables the model to learn about discourse-level coherence
characteristics from a finer-grained distinction and leads to better learning
representation in downstream tasks. Thirdly, our model uses factorized
embedding parameterization in which a smaller size layer vocabulary and hidden
layer to decompose the embedding matrix into two small matrices which reduce
the number of parameters between vocabulary and the first hidden layer whereas
in BERT based biomedical models embedding size is equal to the size of the
hidden layer. Furthermore, finally, our model is trained on massive biomedical
corpora to be effective on BioNER to address the issue of the shift of word
distribution from general domain corpora to biomedical corpora. All these,
when combined, address all the issues associated with BioNER earlier. As our
model offers a consistent improvement over all other methods for all tested
datasets, we can conclude that it is a robust solution for BioNER.

To extend our analysis, we analysed the performance of different pre-trained
models of BioALBERT. We found out that the performance of all BioALBERT models
are almost equally good, but BioALBERT 1.1-large trained on PubMed works
better than others (shown in Fig. 4). BioALBERT 1.1-large model, which is
trained on PubMed with dup-factor as five performs better on most of the
datasets. This shows that the relevance of duplication data in NLP tasks.

Figure 4: Comparison of different variants BioALBERT models

##  V Conclusion

In this study, we presented BioALBERT, which is a pre-trained language model
for biomedical named entity recognition. We presented four different variants
of BioALBERT models which are trained on huge biomedical corpora for a
different number of steps. We showed that training ALBERT on biomedical
corpora is a crucial step in applying it to BioNER. As future works, we plan
to pre-trained other versions which include hybrid of general and biomedical
corpora of ALBERT on biomedical corpora with more training steps and fine-tune
on biomedical text mining task. We also plan to fine-tune BioALBERT on other
text mining tasks to show the effectiveness of our model.

## References

  * [1] Stéphane M Meystre, Guergana K Savova, Karin C Kipper-Schuler, and John F Hurdle.  Extracting information from textual documents in the electronic health record: a review of recent research.  Yearbook of medical informatics, 17(01):128–144, 2008.
  * [2] Lena Mårtensson and Gunnel Hensing.  Health literacy–a heterogeneous phenomenon: a literature review.  Scandinavian journal of caring sciences, 26(1):151–160, 2012.
  * [3] Sangrak Lim, Kyubum Lee, and Jaewoo Kang.  Drug drug interaction extraction from the literature using a recursive neural network.  PLOS ONE, 13:e0190926, 01 2018.
  * [4] Barbara Rosario and Marti Hearst.  Classifying semantic relations in bioscience texts.  In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 430–437, Barcelona, Spain, July 2004\.
  * [5] Aaron M Cohen and William R Hersh.  A survey of current work in biomedical text mining.  Briefings in bioinformatics, 6(1):57–71, 2005.
  * [6] Vikas Yadav and Steven Bethard.  A survey on recent advances in named entity recognition from deep learning models, 2019.
  * [7] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  Deep contextualized word representations.  CoRR, abs/1802.05365, 2018.
  * [8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [9] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations, 2019.
  * [10] Sampo Pyysalo, Filip Ginter, Hans Moen, Tapio Salakoski, and Sophia Ananiadou.  Distributional semantics resources for biomedical text processing.  2013\.
  * [11] Qiao Jin, Bhuwan Dhingra, William W. Cohen, and Xinghua Lu.  Probing biomedical embeddings from language models, 2019.
  * [12] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang.  Biobert: a pre-trained biomedical language representation model for biomedical text mining, 2019.
  * [13] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  In Advances in neural information processing systems, pages 3111–9, 2013.
  * [14] Qingyu Chen, Yifan Peng, and Zhiyong Lu.  Biosentvec: creating sentence embeddings for biomedical texts.  2019 IEEE International Conference on Healthcare Informatics (ICHI), pages 1–5, 2018.
  * [15] Iz Beltagy, Kyle Lo, and Arman Cohan.  Scibert: A pretrained language model for scientific text, 2019.
  * [16] Henghui Zhu, Ioannis C. Paschalidis, and Amir M. Tahmasebi.  Clinical concept extraction with contextual word embedding.  NIPS Machine Learning for Health Workshop, Dec 2018.
  * [17] Yuqi Si, Jingqi Wang, Hua Xu, and Kirk Roberts.  Enhancing clinical concept extraction with contextual embeddings.  Journal of the American Medical Informatics Association, 26(11):1297–1304, Jul 2019.
  * [18] Yifan Peng, Shankai Yan, and Zhiyong Lu.  Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets, 2019.
  * [19] Jeremy Howard and Sebastian Ruder.  Universal language model fine-tuning for text classification.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia, July 2018. Association for Computational Linguistics.
  * [20] Dani Yogatama, Cyprien de Masson d’Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al.  Learning and evaluating general linguistic intelligence.  arXiv preprint arXiv:1901.11373, 2019.
  * [21] Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang.  Pre-trained models for natural language processing: A survey.  arXiv preprint arXiv:2003.08271, 2020.
  * [22] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang.  Recurrent neural network for text classification with multi-task learning.  arXiv preprint arXiv:1605.05101, 2016.
  * [23] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao.  Multi-task deep neural networks for natural language understanding.  arXiv preprint arXiv:1901.11504, 2019.
  * [24] Jiao Li, Yueping Sun, Robin J. Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Thomas C. Wiegers, and Zhiyong Lu.  Biocreative v cdr task corpus: a resource for chemical disease relation extraction.  Database : the journal of biological databases and curation, 2016, 2016.
  * [25] Martin Krallinger, Obdulia Rabal, Florian Leitner, Miguel Vazquez, David Salgado, Zhiyong lu, Robert Leaman, Yanan Lu, Donghong Ji, Daniel Lowe, Roger Sayle, Riza Batista-Navarro, Rafal Rak, Torsten Huber, Tim Rocktäschel, Sérgio Matos, David Campos, Buzhou Tang, Hua Xu, and Alfonso Valencia.  The chemdner corpus of chemicals and drugs and its annotation principles.  Journal of Cheminformatics, 7:S2, 03 2015.
  * [26] Rezarta Islamaj Doundefinedan, Robert Leaman, and Zhiyong Lu.  Ncbi disease corpus.  J. of Biomedical Informatics, 47(C):1â€“10, February 2014.
  * [27] Jin-Dong Kim, Tomoko Ohta, Yoshimasa Tsuruoka, Yuka Tateisi, and Nigel Collier.  Introduction to the bio-entity recognition task at jnlpba.  In Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and Its Applications, JNLPBA â€™04, page 70â€“75, USA, 2004. Association for Computational Linguistics.
  * [28] Rie Kubota Ando.  Biocreative ii gene mention tagging system at ibm watson.  2007\.
  * [29] Martin Gerner, Goran Nenadic, and Casey M Bergman.  Linnaeus: a species name identification system for biomedical literature.  BMC bioinformatics, 11(1):85, 2010.
  * [30] Evangelos Pafilis, Sune P. Frankild, Lucia Fanini, Sarah Faulwetter, Christina Pavloudi, Aikaterini Vasileiadou, Christos Arvanitidis, and Lars Juhl Jensen.  The species and organisms resources for fast and accurate identification of taxonomic names in text.  PLOS ONE, 8(6):1–6, 06 2013.
  * [31] Kai Xu, Zhenguo Yang, Peipei Kang, Qi Wang, and Wenyin Liu.  Document-level attention-based bilstm-crf incorporating disease dictionary for disease named entity recognition.  Computers in biology and medicine, 108:122–132, 2019.
  * [32] Devendra Singh Sachan, Pengtao Xie, Mrinmaya Sachan, and Eric P Xing.  Effective use of bidirectional language modeling for transfer learning in biomedical named entity recognition, 2017.
  * [33] Yinxia Lou, Yue Zhang, Tao Qian, Fei Li, Shufeng Xiong, and Donghong Ji.  A transition-based joint model for disease named entity recognition and normalization.  Bioinformatics (Oxford, England), 33, 03 2017.
  * [34] Ling Luo, Zhihao Yang, Pei Yang, Yin Zhang, Lei Wang, Hongfei Lin, and Jian Wang.  An attention based bilstm crf approach to document level chemical named entity recognition.  Bioinformatics, 34:1381â€“1388, 2018.
  * [35] Wonjin Yoon, Chan Ho So, Jinhyuk Lee, and Jaewoo Kang.  Collabonet: collaboration of deep neural networks for biomedical named entity recognition.  BMC Bioinformatics, 20(S10), May 2019.
  * [36] John M Giorgi and Gary D Bader.  Transfer learning for biomedical named entity recognition with neural networks.  Bioinformatics, 34(23):4087–4094, 2018.


utf8

# Ensemble of Pre-trained Language Models and Data Augmentation for Hate
Speech Detection from Arabic Tweets ††thanks: Citation: Authors. Title.
Pages…. DOI:000000/11111.

Kheir Eddine Daouadi, Yaakoub Boualleg, Kheir Eddine Haouaouchi
Laboratory of Vision and Artificial Intelligence (LAVIA)
Echahid Cheikh Larbi Tebessi University
Tebessa, Algeria
{kheireddine.daouadi, yaakoub.boualleg, Kheireddine.haouaouchi}@univ-
tebbesa.dz

Today, hate speech classification from Arabic tweets has drawn the attention
of several researchers. Many systems and techniques have been developed to
resolve this classification task. Nevertheless, two of the major challenges
faced in this context are the limited performance and the problem of
imbalanced data. In this study, we propose a novel approach that leverages
ensemble learning and semi-supervised learning based on previously manually
labeled. We conducted experiments on a benchmark dataset by classifying Arabic
tweets into 5 distinct classes: non-hate, general hate, racial, religious, or
sexism. Experimental results show that: (1) ensemble learning based on pre-
trained language models outperforms existing related works; (2) Our proposed
data augmentation improves the accuracy results of hate speech detection from
Arabic tweets and outperforms existing related works. Our main contribution is
the achievement of encouraging results in Arabic hate speech detection.

_K_ eywords Twitter $\cdot$ Hate Speech Detection $\cdot$ Arabic Tweet
Classification $\cdot$ Ensemble Learning $\cdot$ Pre-traine language model
$\cdot$ Fine-tuning

##  1 Introduction

Nowadays, the use of social media has substantially increased in Arab
countries, which has allowed more freedom for speech in different domains.
Twitter for example is one of the leading social media where users can share
short text of up to 280 characters optionally followed by a link, video, or
photo, known as a tweet. This free micro-blogging allows users to subscribe,
follow other users, share content, like other tweets, repost another tweet and
reply to another tweet. Today, Twitter is booming, the service has more than
hundreds of millions of users producing over five hundred million tweets per
day 111https://www.omnicoreagency.com/twitter-statistics/. The population of
Saudi Arabia and Oman are the most countries that used the platform. The Arab
users generate 27.4 million tweets per day Al-Hassan et al. (2021). From that
big number, we can assume that hate speech can spread easily and quickly
through this platform.

The General Policy Recommendation no. 15 of the European
Commission222http://hudoc.ecri.coe.int/eng?i=REC-15-2016-015-ENG has described
Hate Speech (HS) as ”the advocacy, promotion or incitement, in any form, of
the denigration, hatred or vilification of a person or group of persons, as
well as any harassment, insult, negative stereotyping, stigmatization or
threat in respect of such a person or group of persons and the justification
of all the preceding types of expression, on the ground of race, color,
descent, national or ethnic origin, age, disability, language, religion or
belief, sex, gender, gender identity, sexual orientation, and other personal
characteristics or status”.

Today, through Twitter, researchers propose approaches for Arabic HS
detection. Nevertheless, two major challenges faced in this context are the
limited performance and the problem of imbalanced data, which make them suffer
from severe over-fitting Founta et al. (2018). Automatic HS detection from
Arabic tweets using traditional machine learning classifiers like Naïve Bayes
(NB), Support Vector Machine (SVM) and Random Forest (RF) have shown
reasonable accuracy results. However, they are based on the handcrafted
features calculated based on some pre-defined methods like Term Frequency
(TF), Term Frequency-Inverse Document Frequency (TF-IDF), Bag of Word (BoW),
etc. Recently, Long Short Term Memory (LSTM), Convolution Neural Network
(CNN), and Bidirectional Encoder Representations from Transformers (BERT) have
already shown good results for HS classification from Arabic tweets.

In our study, we leveraged ensemble learning that relies on pre-trained
language models for the Arabic language Antoun et al. (2020); Abdul-Mageed et
al. (2020) to perform HS classification. Specifically, we fine-tuned various
Arabic BERT models trained on various Twitter data. In addition, we proposed a
semi-supervised learning method based on previously labeled data. Our research
contributions can be summarized as follows:

  * •

We leveraged various Arabic BERT models via transfer learning and fine-tuning
to build our baseline classifier.

  * •

We evaluated ensemble learning based on the leveraged Arabic BERT models.

  * •

We propose a data augmentation method based on semi-supervised learning and
previously labeled data.

The rest of this paper is organized as follows: Section 2 reviews the most
recent related works on Arabic hate speech detection. In Section 3, we provide
details of our methodology. Section 4 presents and discusses the results of
the conducted experiments. Conclusion and future works are presented in
Section 5.

##  2 Related Works

The emergence of the Twitter platform has encouraged a multitude of research
avenues including topic detection Daouadi et al. (2023, 2021, 2024),
organization detection Daouadi et al. (2019a, 2018a, 2018b), and bot detection
Daouadi et al. (2019b, 2020). Thanks to its importance, HS detection from
Arabic tweets has drawn the attention of several researchers. In the
literature, many systems have been developed to resolve this classification
problem. They follow two main principal approaches: a traditional approach,
and a deep learning approach.

###  2.1 Traditional Approaches

In this context, researchers have focused on manual features engineering based
on local metadata and rely on the presence of tweet features like hashtags.
Some examples of traditional approaches are briefly described in the
following.

Besides, authors in Mubarak et al. (2021) focus on the problem of detecting
offensive tweets. They use features from the Arabic word2vec model with SVM,
this yielded an Accuracy result of 88.6%.

Likewise, authors in Husain et al. (2020a) classify tweets being offensive or
not. The best experimental results are obtained using the trained ensemble
learning in offensive language detection, and this yielded F1 score of 88%,
which exceeds the score obtained by the best single learner classifier by 6%.

Furthermore, authors in Husain et al. (2020c) examine the effect of the
preprocessing step on offensive language and HS classification. They claimed
that an intensive preprocessing technique demonstrates its significant impact
on the classification rate. The optimal experimental results are obtained
using BoW and SVM, yielding F1 score results of 89% and 95% for offensive
language and HS classification.

In a different strategy, authors in Mubarak et al. (2017) classify user
accounts being abusive or not. The best experimental results are obtained
using list-based methods (Sweed Words + Log Odds Ratio), and this yielded F1
score of 60%.

###  2.2 Deep Learning Approaches

In this context, researchers follow two principal approaches are deep learning
from scratch approaches and fine-tuning approaches.

####  2.2.1 Deep Learning from Scratch Approaches

In this context, researchers have focused on automatic feature engineering.
They used deep learning models such as CNN and LSTM. Some examples of deep
learning approaches are briefly described in the following.

Authors in Alharbi et al. (2020) classify Arabic tweets being offensive or
not. They use features from n-gram and word embedding model. The best
experimental results are obtained using LSTM. This yielded Accuracy result of
74%.

In a similar, authors in Farha et al. (2020) investigate the impact of using
Multitask learning on offensive and HS detection. The showed that Multitask
learning based on CNN-LSTM improve the accuracy results, yielded macro
F1-score of 90.4% and 73.7% for classifying offensive and HS tweets,
respectively.

Likewise, authors in Rachid et al. (2020) classify tweets being cyberbullying
or not. They use features from the Arabic Word2vec model achieving. The best
experimental result is obtained using a simple CNN-LSTM model, yielded
F1-score of 84%.

Furthermore, authors in Husain et al. (2020b) use features based on TF-IDF to
classify tweets from those of offensive or not. The best experimental results
are obtained using a Bidirectional gated recurrent unit classifier, and this
yielded a macro F1-score of 83%.

In a similar, authors in Abuzayed et al. (2020) classify tweets being HS or
not based on the Arabic Word2vec model. They conducted a series of experiments
using CNN, RNN and SVM. This yielded a macro F1-score of 73%.

Besides, authors in Faris et al. (2020) classify tweets from those of hateful
and normal ones. They used CNN-LSTM with different versions of Arabic
Word2vec, which yielded an F1-score of 71.68%.

Likewise, authors in Alsafari et al. (2020c) investigate the effect of word
embedding models and neural network architectures on the accuracy rate using
(Offensive and Hate vs. Clean), (Hate vs. Offensive vs. Clean) and
(Nationality hate vs. Clean vs. Religion Hate vs. Gender Hate vs. ethnicity
hate vs. Offensive) classification tasks. The best results are achieved using
CNN and word2vec based on the Skip-gram model, which yielded F1-score results
of 70.80% 75.16% 87.22%, and for six-class, three-class, and two-class
classification tasks, respectively.

Besides, authors in Alsafari et al. (2020a) use the AraBERT embedding model
with an ensemble of CNNs and Bidirectional LSTM (BiLSTMs) classifiers. The
best results are achieved using the average-based ensemble approach, which
yielded F1-score results of 91.12% (CNNs), 84.01% (CNNs), and 80.23%(BiLSTMs),
for two-class, three-class, and six-class classification tasks, respectively.

In a similar, authors in Alsafari et al. (2020c) use CNN with the Multilingual
BERT embedding model, which yielded F1 score results of 87.03% 78.99%, and
75.51% for two-class, three-class, and six-class classification tasks,
respectively.

Furthermore, authors in Duwairi et al. (2021) conducted a series of
experiments using CNN, BiLSTM-CNN, and CNN-LSTM. They reported three types of
experiment: Binary classification (Hate or Normal), Ternary classification
(Hate, Abusive, or Normal), and Multi-class classification (Misogyny, Racism,
Religious Discrimination, Abusive, and Normal). In the binary classification
task, the CNN model outperformed other models and achieved an accuracy of 81%.
In the ternary classification task, both the CNN, and BiLSTM-CNN models
achieved the best accuracy of 74%. While in multi-class classification task,
the best results are achieved by CNN-LSTM, and BiLSTM-CNN, yielded an Accuracy
of 73%.

In similar, authors in Alshaalan et al. (2020) classify tweets being hate or
not, they compared four models: CNN, GRU, CNN-GRU, and BERT. Their
experimental results show that CNN model gives the best performance with an
F1-score of 79%.

Similarly, authors in Al-Hassan et al. (2021) classify tweets into five
distinct classes: none, religious, racial, sexism and general hate. They used
four deep learning models: LSTM, CNN-LSTM, GRU, and CNN-GRU. Their
experimental results show that the hybrid CNN-LSTM model gives the best
performance with an F1-score of 73%.

Besides, authors in Hassan et al. (2020) evaluate the combination of SVM, CNN,
CNN-BiLSTM on offensive language and HS classification tasks. These models
showed a significant performance with 90.51% macro F1-score for offensive
language detection and 80.63% macro F1-score for hate speech detection.

In a different strategy, authors in Alghanmi et al. (2020) use both Arabic
BERT and Arabic Word2vec) embedding models classify tweets being normal,
hateful, or abusive. The best accuracy result is achieved using CNN, which
yielded an F1 score of 72.1%.

####  2.2.2 Transfer Learning Approaches

In this context, researchers have focused on transfer learning based on fine-
tuning. The majority of transfer learning approaches focus mainly on the
Arabic versions of BERT. Some examples of transfer learning approaches are
briefly described in the following:

Authors in Elmadany et al. (2020) evaluate Arabic BERT on offensive language
and HS classification tasks. This yielded a macro F1-score of 82.31% and
70.51% for classifying HS and offensive language classification task.

Likewise, authors in Abdellatif et al. (2020) evaluate the Universal Language
Model Fine-tuning on the offensive language and HS classification task, this
yielded macro F1-score of 77% and 58% for classifying offensive and HS tweets.

Besides, authors in Aldjanabi et al. (2021) studied the impact multitask
learning model built on top of a pre-trained Arabic BERT on offensive and HS
classification tasks, which yielded F1-score of 92.34% and 88.73% for
classifying offensive and HS tweets.

###  2.3 Data Augmentation Methods

In this context, researchers have focused on generating synthetic data in
order to face the challenge of imbalanced learning. Some examples of data
augmentation methods are briefly described in the following:

Authors in Husain et al. (2020c) investigate the impact of upsampling
technique (duplicate tweets of the minority class to balance the class label)
on offensive and HS classification tasks. They showed that upsampling decrease
the result of F1-score results.

In similar, the authors in Haddad et al. (2020) propose two data augmentation
methods for offensive and HS classification. The first one uses an external
augmenting technique by adding some offensive tweets from another labeled
data. While the second one uses the random oversampling method by shuffling
the words into HS tweets to create new samples.

Likewise, authors in Elmadany et al. (2020) investigate the impact of seed
words and tweet emotion for automatic annotate tweets for offensive and HS
classification tasks. They showed that the examined method has a positive and
negative impact on the offensive and HS classification tasks, respectively.

In a different strategy, authors in Alsafari et al. (2021) explore the role of
semi-supervised built on unlabeled tweets for classifying tweets being
offensive, hateful, and normal. They showed that an improvement of up to 7%
was achieved from using additional pseudo-labeled tweets.

##  3 Proposed Approach

Figure 1: The process of transfer learning Corpus (NH: Normal, G: General hate
speech, Re: Religious, S: Sexism, Ra: Racism).

To overcome the challenges previously mentioned in our related works, we
propose a novel approach for classifying hate speech from Arabic tweets. As
shown in Figure 1, our proposed approach consists of four main steps are data
augmentation, tweets pre-processing, transfer learning and ensemble learning.

###  3.1 Data Augmentation

The imbalance nature in data often has been disregarded in the major related
works, which might have a substantial impact on the classification results.
This problem has occurred in the HS detection where most labeled datasets are
highly imbalanced. To face this issue, we propose semi-supervised learning
built on previously manually labeled tweets. Firstly, we train the model using
the state-of-the-arts datasets Al-Hassan et al. (2021) labeled as (Non-hate,
General HS, Sexism, Racial and Religious HS). The tweets labeled as religious
HS in Albadi et al. (2019); Alsafari et al. (2020c) are added directly to the
to the datasest of Al-Hassan et al. (2021). While the tweets previously
labeled as HS in Sun et al. (2019); Mulki et al. (2019); Haddad et al. (2019);
Alshaalan et al. (2020); Alsafari et al. (2020c); Ousidhoum et al. (2019) are
classified with the trained model, and the new label is used balance the first
datasets.

###  3.2 Tweets Preprocessing

The inputs of our proposition are the textual content of tweets composed of
raw tweet content. In this step, our main objective is to clean the textual
content produce a more consistent and standard tweet. We performed some pre-
processing tasks as follows:

  * •

Removing the tweet features: user mentions ’@’, URLs, the word RT, #,
punctuation, special characters (emoticons), and numerical characters.

  * •

Removing repeated characters, Arabic stop words, non-Arabic letters, new lines
as well as diacritics.

  * •

Arabic letters normalization: in the Arabic language, they are different
variations for representing some letters which are:

    * –

Letter ة which can be mistaken and written as ه, we normalized it to ه.

    * –

Letter أ which has the forms ا, آ, إ, أ ), all these four letters are
normalized into ا.

    * –

The Arabic dash that is used to expand the word مرحبا>) to مرحبـــــا>) has
been removed.

    * –

Letter ى has been normalized to ي.

All tweets are padded to the length of the longest tweet and will be used as
an input for the following step.

###  3.3 Transfer Learning

The second step aims to use the pre-trained language models (AraBert-Large,
AraBert-Base, and MarBert).

Traditional machine learning technology has matured to a point where it may be
used in a variety of practical applications with considerable success.
However, it have certain limits in some real-world circumstances. For example,
gathering sufficient training data is prohibitively expensive, time-consuming,
or impossible, necessitating the use of transfer learning methods. Transfer
learning is a method of training artificial neural networks that depend on
pre-trained models on specific tasks and data. Transfer learning assumes that
if a pre-trained network solves one issue well, it may be utilized to tackle a
similar but distinct problem with a little extra training. The reliance on a
large amount of training data can be reduced in this way.

Transfer learning is undeniably one of the most important aspects of language
models. In the Natural Language Processing, the advent of Bidirectional
Encoder Representation from Transformers (BERT) Devlin et al. (2018) sparked a
revolution. BERT is a deep learning model that has produced best-in-class
performance on a wide range of natural language processing tasks. BERT was
taught using a technique known as Masked Language Modeling (MLM). The MLM
operates by randomly concealing part of the existing tokens from the input. As
a result, the masked word’s original vocabulary id may be anticipated from the
word’s context. As a result, the MLM will be able to merge the left and right
contexts. In addition to the MLM, BERT employs the Next Sentence Prediction
task to train a competent language model that recognizes and understands
sentence relationships. The BERT model architecture is a multi-layered
bidirectional transformer encoder based on the Devlin et al. (2018) original
implementation. The transformers, it is suggested, are a collection of
numerous nested layers (or blocks). An ’attention’ layer is present in each
layer/block. For each block, BERT uses twelve (12) different attention
mechanisms, allowing tokens from the input sequence (e.g., sentences made up
of word or sub-word tokens) to focus on the other token. In their paper,
Devlin et al. (2018) Vaswani et al. (2017) offered the following two
architectures:

  * •

BERT Base: 12 layers (Transformer blocks), 12 attention heads, and 110 million
parameters.

  * •

BERT Large: 24 layers, 16 attention heads, and 340 million parameters.

A preprocessing phase, consisting of tokenization, should be conducted before
feeding a raw sentence to BERT. The sub-word tokenization technique WordPiece
employs a vocabulary initialization to cover all characters in the training
data. WordPiece added the appropriate merge rules for tokenization later on.
WordPiece learns positional embeddings with a sequence length of up to 512
tokens and creates embeddings with a 30,000 character vocabulary. English and
BooksCorpus are used to train BERT models. The single sentence categorization
using the BERT model is shown in Figure 2. In Figure 2, E stands for input
embedding, Ti for contextual representation of token i (Tok I), and [CLS] for
classification output, which summarizes all the hidden states outputted from
all tokens in the input sentence.

Figure 2: Single sentence classification using BERT.

###  3.4 Ensemble Learning

Ensemble learning is a broad machine learning meta-approach that tries to
improve predictive performance by mixing predictions from several models. In
our research, we compared two ensemble learning methods are majority voting
and average voting described as follows.

Majority voting (also known as Hard Voting), each classifier votes for a
class, and the class with the most votes win. The ensemble’s anticipated
target label is the mode of the distribution of individually predicted labels
in statistical terms. This strategy is intentionally used to increase model
performance, to out performs any one model in the ensemble.

Average voting (also known as Soft voting), where each classifier assigns a
probability value to each data point that it belongs to. The predictions are
totaled and weighted according to the relevance of the classifier. The vote is
then cast for the target label with the largest sum of weighted probability.

##  4 Experimental results and evaluation

To fine-tune the transfer learning model, authors in Sun et al. (2019) have
recommended selecting from the values of the following parameters: learning
rate, batch size, number of epochs described as follows:

  * •

The learning rate is a hyper-parameter that controls how much the model
changes each time.

  * •

The number of epochs is a hyper-parameter that specifies how many times the
learning algorithm will iterate over the whole training dataset.

  * •

The batch size is a hyper-parameter that specifies how many samples must be
processed before the internal model parameters are updated.

The tenfold cross-validation approach was used to evaluate the performance of
our proposed approach. The dataset was split into ten equal-sized segments
while maintaining the balance of each class in the corresponding datasets. One
of these parts was used as a testing and the remainder were used as training.
This procedure was repeated 10 times and the averaging of the performance
results was obtained across the ten repetitions of cross-validation. The
performance measures used to evaluate our proposed approach are as described
follows.

  * •

Recall (R), because it concentrates on the good examples, we should pay
greater attention to that measure. For each class label, recall refers to the
proportion of properly classified tweets to the number of tweets that belonged
to that class but were mistakenly classified by the model. For example, the
Recall of the Racial class is calculated as described in the following
equation:

$R_{Racial}=\frac{correctly\\_classified\\_as\\_Racial}{Total\\_number\\_of\\_Racial\\_tweets}$
(1)

  * •

Precision (P), this metric is likewise useful in our instance; it reflects the
proportion of relevant retrieved tweets from a specific class. For example,
the Precision of religious class is calculated as described in the following
equation:

$P_{Religious}=\frac{correctly\\_classified\\_as\\_Religious}{Tweets\\_classified\\_as\\_"Religious"}$
(2)

  * •

F1-Score is more useful than accuracy in our instance of unequal class
distribution since it is the weighted average of precision and recall.

$F1-Score=2\times\frac{Precision\times Recal}{Precision+Recall}$ (3)

    * –

Micro : Calculate measures globally by counting the total true positives,
false negatives, and false positives.

    * –

Macro : Calculate measures for each label, and find their unweighted mean,
this does not take class imbalance into account.

    * –

Weighted : Calculate measures for each label and find their average weighted
by support (the number of true instances for each label). This alters ‘macro’
to account for label imbalance.

To validate our proposition, we used different Arabic HS datasets. In the
following, a brief description of the corresponding datasets is presented.

We use the dataset annotated by Al-Hassan et al. (2021) to evaluate our
proposed classification approach. Table 1 shows an overview of the dataset we
utilized in our research. The authors utilized the Especial library with a
list of hashtags that activate and attract Twitter nasty content to collect
the tweets. Because of this, balancing the amount of non-hate and hate tweets
is unrealistic and does not reflect the true situation. To maintain a
realistic and natural environment, the writers simply picked a list of
hashtags that undoubtedly contains both non-hateful and hateful information.
After that, two annotators manually annotated the obtained tweets to eliminate
any annotator bias. A guide was supplied to the annotators to follow to
separate the tweet classifications, resulting in a dataset of 11634 tagged
tweets out of 37 k collected tweets. Table 1 shows that the tiny subset of
tweets belongs to the racial hate speech category, whereas the large subset
belongs to the non-hate category.

Table 1: Overview of the datasets used to evaluate our proposed classification approach (NH: Non-Hate speech, Se: Sexism hate speech, Re: Religious hate speech, GH: General hate speech, Ra: Racism hate speech, k denotes one thousand). | NH | GH | Re | Ra | Se | Total
---|---|---|---|---|---|---
Number of tweets | 8332 | 1397 | 722 | 526 | 657 | 11634
Word count | 96.9 K | 17.2 K | 9.2 K | 6.6 K | 8.3 K | 138.3 K
Unique words | 29 K | 8.7 K | 4.9 K | 4 K | 4.4 K | 37.9 K
Average words per tweet | 11.6 | 12.3 | 12.7 | 12.7 | 12.6 | 11.9

The statistics about the datasets used to evaluate our proposed method for
data augmentation are presented in Table 2 and described as follows:

  * •

OSACT Sun et al. (2019): This dataset comprises 10K tweets that have been
labeled for offensiveness (OFF or NOT OFF) and hate speech (HS or NOT HS).

  * •

L-HSAB Mulki et al. (2019): L-HSAB is a collection of 5,846 Syrian and
Lebanese political tweets categorized as normal, abusive, or hateful.

  * •

T-HSAB Haddad et al. (2020): T-HSAB brings together 6,024 Tunisian comments
that are classified as normal, abusive, or hateful.

  * •

GHDS Alshaalan et al. (2020): A total of 9,316 tweets were classed as hateful,
abusive, or normal in the dataset (not hateful or abusive).

  * •

RHS Albadi et al. (2019): These include 6000 Arabic tweets, 1000 for each of
the six religious groups.

  * •

MCH Alsafari et al. (2020c): This dataset contains 1100 tweets that were
gathered utilizing contentious accounts and hashtags.

  * •

MLMA Ousidhoum et al. (2019): This consists of multi-language and multi-
aspect, the Arabic tweets include 3353 of hate and non-hate speech.

Table 2: Overview of the datasets used to evaluate our proposed method for data augmentation. Datast | Hate tweets
---|---
MLMA | 460
GHDS | 180
RHS | 2759
MCH | 1417
OSACT | 491
T-HSAB | 1078
L-HSAB | 468
All data | 6853

###  4.1 Fine-tuning

The first set of our experiment is dedicated to determining the best hyper-
parameter. The initial parameters used in our experiment are 2 epochs, 8 batch
size, and 1e-5 learning rate.

The experiment presented in Table 3 shows the effect of the number of epochs.
The F1-score metric is increased when the number of epochs is increased from 2
to 4 by 4.99% in bert-base-arabertv02-twitter, but it is decreased when the
number of epochs is further increased. Then, using bert-large-
arabertv02-twitter, The F1-score metric is increased from 2 epochs, but it is
decreased when the number of epochs is further increased. While when using
MARBERT, the F1-score is increased by 0.16 when the number of epochs is
increased from 2 epochs to 4 epochs, but it is decreased when the number of
epochs is further increased

Table 3: The effect of the number of epochs on micro F1-score. Epochs | 2 | 3 | 4 | 5 | 10
---|---|---|---|---|---
bert-base-arabertv02-twitter | 79.65 | 84.60 | 84.64 | 84.28 | 84.14
bert-large-arabertv02-twitter | 84.59 | 83.78 | 81.74 | 83.24 | 79.68
MARBERT | 83.98 | 84.14 | 83.91 | 83.58 | 83.11

The experiment presented in Table 4 shows the effect of the number of batch
size. The F1-score is increased when the number of batch size is increased
from 8 to 16 by 0.02 using bert-base-arabertv02-twitter, but it is decreased
when the number of batch size is further increased. However, the F1-score
metric is decreased when the number of batch size is further increased from 8
to 64 using bert-large-arabertv02-twitter and MARBET.

Table 4: The effect of the number of Batch size on micro F1-score. Batch size | 8 | 16 | 32 | 64
---|---|---|---|---
bert-base-arabertv02-twitter | 84.64 | 84.66 | 84.11 | 83.76
bert-large-arabertv02-twitter | 84.59 | 84.08 | 83.66 | 81.88
MARBERT | 84.14 | 83.87 | 84.07 | 82.92

The experiment presented in Table 5 shows the effect of the learning rate. The
F1 score is decreased when the value of learning rate is increased from 1e-5
to 5e-5.

Table 5: The effect of the number of learning rate on micro F1-score. Learning rate | 1e-5 | 2e-5 | 3e-5 | 4e-5 | 5e-5
---|---|---|---|---|---
bert-base-arabertv02-twitter | 84.66 | 84.62 | 84.53 | 84.20 | 83.17
bert-large-arabertv02-twitter | 84.59 | 82.25 | 78.29 | 76.83 | 83.66
MARBERT | 84.14 | 83.42 | 81.83 | 80.55 | 79.68

In the summary, the optimal parameters of the corresponding models are shown
in table 6.

Table 6: Optimal parameters of the corresponding pre-trained language model (E: Epochs, BS: Batch Size, LR: Learning Rate). | E | BS | LR
---|---|---|---
bert-base-arabertv02-twitter | 4 | 16 | 1e-5
bert-large-arabertv02-twitter | 2 | 8 | 1e-5
MARBERT | 3 | 8 | 1e-5

###  4.2 The Effect of Ensemble Learning

The second set of our experiments is dedicated to evaluating the ensemble
learning methods (majority voting & average voting). Table 7 present a
comparison between single and ensemble models. It is important to note that
ensemble learning models achieve the highest results, yielding weighted-
average F1-score results of 85.48 and 85.10 using majority voting and averaged
voting, respectively. In a summary, the Macro-Averaged, Micro-Averaged, and
F1-score are reached 72.66%, 85.47% and 85.48%, respectively.

Table 7: The effect of ensemble learning (Ma: Macro-Average, Mi: Micro-Average, W: Weighted-Average). | NH | Se | Re | GH | Ra | Ma | Mi | W
---|---|---|---|---|---|---|---|---
bert-base-arabertv02-twitter | 92.42 | 70.26 | 79.89 | 60.72 | 50.48 | 70.76 | 84.66 | 84.69
bert-large-arabertv02-twitter | 92.47 | 67.60 | 79.74 | 60.82 | 47.81 | 69.69 | 84.59 | 84.46
MARBERT | 92.17 | 68.83 | 78.42 | 59.23 | 50.35 | 69.80 | 84.14 | 84.15
Average voting | 92.60 | 70.86 | 81.53 | 61.21 | 52.37 | 71.71 | 85.08 | 85.10
Majority voting | 92.73 | 72.27 | 82.56 | 61.89 | 53.85 | 72.66 | 85.47 | 85.48

###  4.3 The Effect of Data Augmentation

The third set of our experiments is delegated to evaluating our proposed
method for data augmentation. Firstly, the tweets labeled as religious hate
speech by Albadi et al. (2019); Alsafari et al. (2020c) are directly added to
the training data. Second, we used semi-supervised learning using the trained
model with the datasets presented in Table 2. Table 8 present the results of
the classification task.

Table 8: Prediction results. Predicted class | GH | Re | Se | Ra
---|---|---|---|---
Number tweet | 1034 | 950 | 863 | 926

The experiments presented in Table 9 shows a comparison of the proposed
approach using data augmentation and without data augmentation. However, the
F1-score result is increased from 85.48 to 85.65 when the training data is
augmented through or proposed method.

Table 9: The effect of our proposed data augmentation method. | NH | Se | Re | GH | Ra | Ma | Mi | W
---|---|---|---|---|---|---|---|---
Without Data Augmentation | 92.73 | 72.27 | 82.56 | 61.89 | 53.85 | 72.66 | 85.47 | 85.48
With Data Augmentation | 92.41 | 72.58 | 83.50 | 63.27 | 57.25 | 73.80 | 85.60 | 85.65

###  4.4 Comparison with Existing Data Augmentation Methods

The fourth set of our experiments is dedicated to comparing our proposed
methods with the relevant related works. Notably, from the experiment shown in
Table 10, the F1-score results of different classes is presented as follows:

  * •

Normal: The F1-score of the latest state-of-the-art data augmentation methods
is ranging from [91.65% – 92.36%] while the F1-score of our proposed data
augmentation methods yielded 92.41%.

  * •

General hate speech: The achieved results from our proposed data augmentation
method is increased by about 3.19% when compared with the latest state-of-the-
art.

  * •

Sexism: The F1-score of the latest state-of-the-art data augmentation methods
ranges from [66.38% – 70.52%], whereas our proposed data augmentation method
achieved 72.58%.

  * •

Racial: Our proposed data augmentation method increases the F1-score from
[44.15% - 50.99%] in comparison with the latest state-of-the-art, which
yielded 57.25%.

  * •

Religious hate speech: The F1-score of the latest state-of-the-art data
augmentation methods is ranging from [78.72% - 79.89%] while our proposed data
augmentation method yielded F1-score of 83.50%.

In a summary, the Macro-Averaged, Micro-Averaged, and Micro-Averaged F1-scores
are reached 73.80%, 85.60%, and 85.65%, respectively.

Table 10: Comparison of our proposed data augmentation method with the latest state-of-the-art methods. | NH | S | Re | G | Ra | Ma | Mi | W
---|---|---|---|---|---|---|---|---
Without Data Augmentation | 92.73 | 72.27 | 82.56 | 61.89 | 53.85 | 72.66 | 85.47 | 85.48
Husain et al. (2020c) | 92.04 | 70.12 | 79.89 | 59.45 | 50.99 | 70.50 | 84.20 | 84.28
Alsafari et al. (2021) | 92.21 | 70.52 | 79.42 | 60.08 | 50.60 | 70.57 | 84.29 | 84.45
Liu et al. (2020) | 92.36 | 66.38 | 78.72 | 59.48 | 44.15 | 68.22 | 84.08 | 83.92
Cao et al. (2020) | 91.65 | 67.50 | 79.20 | 57.01 | 49.07 | 68.89 | 83.52 | 83.43
Ours | 92.41 | 72.58 | 83.50 | 63.27 | 57.25 | 73.80 | 85.60 | 85.65

###  4.5 Comparison with Existing Classification Approaches

The last set of our experiment is dedicated to comparing our proposed
classification approach with the latest related works.

While comparing our proposed approach with the other baselines in Table 11,
the accuracy rate of our proposition is the highest. This is due to our use of
ensemble learning and the the data augmentation method. When compared with
CNN, LSTM, current DL hate speech classification models require to huge amount
of labeled data to achieve good performance results. In contrast, does not
require a huge amount of labeled data to achieve good performance results.
When compared to SVM, NB, and Bagging, ours has the benefit of not requiring
handcrafted features to be designed and extracted.

Table 11: Comparison between our proposed classification approach and the latest related works. Work | NH | Se | Re | GH | Ra | Ma | W
---|---|---|---|---|---|---|---
Mulki et al. (2019) | 0.85 | 0.21 | 0.10 | 0.12 | 0.09 | 0.27 | 0.64
Husain et al. (2020a) | 0.86 | 0.42 | 0.50 | 0.25 | 0.24 | 0.45 | 0.71
Chowdhury et al. (2020) | 0.87 | 0.41 | 0.54 | 0.30 | 0.29 | 0.48 | 0.73
Al-Hassan et al. (2021) | 0.86 | 0.43 | 0.59 | 0.31 | 0.25 | 0.49 | 0.73
Alsafari et al. (2020c) | 0.85 | 0.42 | 0.56 | 0.45 | 0.20 | 0.50 | 0.73
Haddad et al. (2020) | 0.87 | 0.50 | 0.64 | 0.49 | 0.27 | 0.55 | 0.76
Alghanmi et al. (2020) | 0.86 | 0.44 | 0.59 | 0.47 | 0.22 | 0.52 | 0.74
Faris et al. (2020) | 0.87 | 0.37 | 0.47 | 0.28 | 0.26 | 0.45 | 0.72
Alsafari et al. (2020a) | 0.87 | 0.49 | 0.56 | 0.47 | 0.24 | 0.53 | 0.75
Mubarak et al. (2021) | 0.88 | 0.49 | 0.67 | 0.41 | 0.37 | 0.57 | 0.77
Ensemble Learning | 0.93 | 0.72 | 0.83 | 0.62 | 0.54 | 0.73 | 0.85
Ensemble Learning + Data Agmentation | 0.92 | 0.73 | 0.84 | 0.63 | 0.57 | 0.74 | 0.86

##  5 Conclusion

Today, the increase in the spread of HS on social media is a global alarm.
Efforts are now taking place to control and avoid HS. This research
investigates the impact of ensemble learning that incorporates three different
pre-trained Arabic language models with semi-supervised learning built on
previously labeled datasets. The leveraged models are used to cover classical
Arabic texts and their dialects. Moreover, using pre-trained language model
will facilitate the process of learning content representations. Experiments
results show that: (1) ensemble learning based on pre-trained language models
outperforms existing related works; (2) Our proposed data augmentation
improves the accuracy results of hate speech detection from Arabic tweets and
outperforms existing related works.

As future works, we plan to follow numerous directions. First, we want to
focus on the contextual embedding model and try to modify its vocabulary to
support the HS detection task. Second, we plan to address the task of
classifying HS in the Algerian dialect.

From a research perspective, we will leverage our proposed systems to classify
and analyze Arabic Twitter discussions to determine the extent of HS
conversations with public discourse and to understand how their sophistication
and capabilities evolve over the time.

## References

  * Al-Hassan et al. (2021) A. Al-Hassan, H. Al-Dossari, Detection of hate speech in Arabic tweets using deep learning. Multimedia Systems (2021). https://doi.org/10.1007/s00530-020-00742-w.
  * Duwairi et al. (2021) R. Duwairi, A. Hayajneh, M. Quwaider, A Deep Learning Framework for Automatic Detection of Hate Speech Embedded in Arabic Tweets. Arabian Journal for Science and Engineering, 46(4), 4001-4014, (2021).
  * Mubarak et al. (2021) H. Mubarak, A. Rashed, K. Darwish, Y. Samih, A. Abdelali, Arabic offensive language on twitter: Analysis and experiments. arXiv preprint arXiv:2004.02192, (2021).
  * Daouadi et al. (2018b) K. E. Daouadi, R. Z. Rebaï, I. Amous, Organization vs. Individual: Twitter User classification. In International Workshop on Language Processing and Knowledge Management, (2018).
  * Elmadany et al. (2020) Elmadany, A., Zhang, C., Abdul-Mageed, M., & Hashemi, A. (2020). Leveraging affective bidirectional transformers for offensive language detection. arXiv preprint arXiv:2006.01266.
  * Antoun et al. (2020) W. Antoun, F. Baly, H. Hajj, Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104, (2020).
  * Abdul-Mageed et al. (2020) Abdul-Mageed, M., Elmadany, A., & Nagoudi, E. M. B. (2020). ARBERT & MARBERT: deep bidirectional transformers for Arabic. arXiv preprint arXiv:2101.01785.
  * Alshaalan et al. (2020) Alshaalan, R., & Al-Khalifa, H. (2020, December). Hate speech detection in saudi twittersphere: A deep learning approach. In Proceedings of the Fifth Arabic Natural Language Processing Workshop (pp. 12-23).
  * Keleg et al. (2020) Keleg, A., El-Beltagy, S. R., & Khalil, M. (2020, May). ASU_OPTO at OSACT4-Offensive Language Detection for Arabic text. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection (pp. 66-70).
  * Abdellatif et al. (2020) Abdellatif, M., & Elgammal, A. (2020, May). Offensive language detection in Arabic using ULMFiT. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection (pp. 82-85).
  * Daouadi et al. (2018a) K. E. Daouadi, R. Z. Rebaï, I. Amous, Towards a Statistical Approach for User Classification in Twitter. In International Conference on Machine Learning for Networking, (2018) 33-43.
  * Aldjanabi et al. (2021) Aldjanabi, W., Dahou, A., Al-qaness, M. A., Elaziz, M. A., Helmi, A. M., & Damaševičius, R. (2021, October). Arabic Offensive and Hate Speech Detection Using a Cross-Corpora Multi-Task Learning Model. In Informatics (Vol. 8, No. 4, p. 69). MDPI.
  * Hassan et al. (2020) Hassan, S., Samih, Y., Mubarak, H., Abdelali, A., Rashed, A., & Chowdhury, S. A. (2020, May). ALT submission for OSACT shared task on offensive language detection. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection (pp. 61-65).
  * Mubarak et al. (2017) Mubarak, H., Darwish, K., & Magdy, W. (2017, August). Abusive language detection on Arabic social media. In Proceedings of the first workshop on abusive language online (pp. 52-56).
  * Husain et al. (2020c) F. Husain, OSACT4 Shared Task on Offensive Language Detection: Intensive Preprocessing-Based Approach. arXiv preprint arXiv:2005.07297, (2020).
  * Saeed et al. (2020) Saeed, H. H., Calders, T., & Kamiran, F. (2020, May). OSACT4 shared tasks: Ensembled stacked classification for offensive and hate speech in Arabic tweets. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection (pp. 71-75).
  * Mulki et al. (2019) H. Mulki, H. Haddad, C. B. Ali, H. Alshabani, L-hsab: A levantine twitter dataset for hate speech and abusive language. In Proceedings of the third workshop on abusive language online (pp. 111-118), (2019).
  * Rachid et al. (2020) Rachid, B. A., Azza, H., & Ghezala, H. H. B. (2020, July). Classification of cyberbullying text in arabic. In 2020 International Joint Conference on Neural Networks (IJCNN) (pp. 1-7). IEEE.
  * Husain et al. (2020b) Husain, F., Lee, J., Henry, S., & Uzuner, O. (2020, December). SalamNET at SemEval-2020 Task 12: deep learning approach for Arabic offensive language detection. In Proceedings of the Fourteenth Workshop on Semantic Evaluation (pp. 2133-2139).
  * Founta et al. (2018) Founta, A. M., Djouvas, C., Chatzakou, D., Leontiadis, I., Blackburn, J., Stringhini, G., Athena Vakali, A., Sirivianos, M., Kourtellis, N. (2018, June). Large scale crowdsourcing and characterization of twitter abusive behavior. In Twelfth International AAAI Conference on Web and Social Media.
  * Daouadi et al. (2019a) K. E. Daouadi, R. Z. Rebaï, I.Amous, Organization, bot, or human: Towards an efficient twitter user classification. Computación y Sistemas, 23(2), 273-279,(2019).
  * Alsafari et al. (2020a) S. Alsafari, S. Sadaoui, M. Mouhoub, Effect of Word Embedding Models on Hate and Offensive Speech Detection. arXiv preprint arXiv:2012.07534, (2020).
  * Husain et al. (2020a) F. Husain, Arabic offensive language detection using machine learning and ensemble machine learning approaches. arXiv preprint arXiv:2005.08946, (2020).
  * Abuzayed et al. (2020) Abuzayed, A., & Elsayed, T. (2020, May). Quick and simple approach for detecting hate speech in Arabic tweets. In Proceedings of the 4th workshop on open-source Arabic Corpora and processing tools, with a shared task on offensive language detection (pp. 109-114).
  * Chowdhury et al. (2020) S. A. Chowdhury, H. Mubarak, A. Abdelali, S. G. Jung, B. J. Jansen, J. Salminen, A multi-platform Arabic news comment dataset for offensive language detection. In Proceedings of the 12th Language Resources and Evaluation Conference, (pp. 6203-6212), (2020).
  * Aljarah et al. (2020) I. Aljarah, M. Habib, N. Hijazi, H. Faris, R. Qaddoura, B. Hammo, M. Abushariah, M. Alfawareh, Intelligent detection of hate speech in Arabic social network: A machine learning approach. Journal of Information Science, 0165551520917651, (2020).
  * Daouadi et al. (2019b) K. E. Daouadi, R. Z. Rebaï, I. Amous, Bot detection on online social networks using deep forest. In Computer science on-line conference (2019) 307-315.
  * Abozinadah et al. (2017) E. A. Abozinadah, J. H. Jones Jr, A statistical learning approach to detect abusive twitter accounts. In Proceedings of the International Conference on Compute and Data Analysis, (pp. 6-13), (2017).
  * Haddad et al. (2019) Haddad, H., Mulki, H., & Oueslati, A. (2019, October). T-HSAB: A Tunisian hate speech and abusive dataset. In International Conference on Arabic Language Processing (pp. 251-263). Springer, Cham.
  * Alsafari et al. (2021) Alsafari, S., & Sadaoui, S. (2021). Semi-Supervised Self-Training of Hate and Offensive Speech from Social Media. Applied Artificial Intelligence, 35(15), 1621-1645.
  * Faris et al. (2020) H. Faris, I. Aljarah, M. Habib, P. A. Castillo, Hate Speech Detection using Word Embedding and Deep Learning in the Arabic Language Context. In Proceedings of the 9th International Conference on Pattern Recognition Applications and Methods (pp. 453-460), (2020).
  * Farha et al. (2020) Farha, I. A., & Magdy, W. (2020, May). Multitask learning for Arabic offensive language and hate-speech detection. In Proceedings of the 4th workshop on open-source Arabic corpora and processing tools, with a shared task on offensive language detection (pp. 86-90).
  * Alharbi et al. (2020) Alharbi, A. I., & Lee, M. (2020, May). Combining character and word embeddings for the detection of offensive language in Arabic. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection (pp. 91-96).
  * Alghanmi et al. (2020) I. Alghanmi, L. E. Anke, S. Schockaert, Combining BERT with Static Word Embeddings for Categorizing Social Media. In Proceedings of the Sixth Workshop on Noisy User-generated Text, (pp. 28-33), (2020).
  * Alsafari et al. (2020b) S. Alsafari, S. Sadaoui, M. Mouhoub, Deep Learning Ensembles for Hate Speech Detection. In IEEE 32nd International Conference on Tools with Artificial Intelligence, (pp. 526-531), (2020).
  * Alsafari et al. (2020c) S. Alsafari, S. Sadaoui, M. Mouhoub, Hate and offensive speech detection on arabic social media. Online Social Networks and Media, 19, 100096, (2020).
  * Alsafari et al. (2020a) Alsafari, S., Sadaoui, S., & Mouhoub, M. (2020, November). Deep learning ensembles for hate speech detection. In 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI) (pp. 526-531). IEEE.
  * Ousidhoum et al. (2019) Ousidhoum, N., Lin, Z., Zhang, H., Song, Y., & Yeung, D. Y. (2019). Multilingual and multi-aspect hate speech analysis. arXiv preprint arXiv:1908.11049.
  * Liu et al. (2020) Liu, R., Xu, G., & Vosoughi, S. (2020). Enhanced offensive language detection through data augmentation. arXiv preprint arXiv:2012.02954.
  * Cao et al. (2020) Cao, R., & Lee, R. K. W. (2020, December). Hategan: Adversarial generative-based data augmentation for hate speech detection. In Proceedings of the 28th International Conference on Computational Linguistics (pp. 6327-6338).
  * Daouadi et al. (2023) K. E. Daouadi, Y. Boualleg, & O. Guehairia, Deep Random Forest and AraBert for Hate Speech Detection from Arabic Tweets. Journal of Universal Computer Science, 29(11) (2023), 1319-1335.
  * Daouadi et al. (2021) K. E. Daouadi, R. Z. Rebaï, I. Amous, Optimizing Semantic Deep Forest for tweet topic classification. Information Systems, 101 (2021) 101801-101811.
  * Haddad et al. (2020) B. Haddad, Z. Orabe, A. Al-Abood, N. Ghneim, Arabic offensive language detection with attention-based deep neural networks. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection (pp. 76-81), (2020).
  * Husain et al. (2021) F. Husain, O. Uzuner, Transfer Learning Approach for Arabic Offensive Language Detection System–BERT-Based Model. arXiv preprint arXiv:2102.05708, (2021).
  * Daouadi et al. (2024) K. E. Daouadi, Y. Boualleg, & O. Guehairia, Systematic Investigation of Recent Pre-trained Language Model for Hate Speech Detection in Arabic Tweets. ACM Transactions on Asian and Low-Resource Language Information Processing. (2024). https://doi.org/10.1145/3674970.
  * Albayari et al. (2021) R. ALBayari, S. Abdullah, S. A. Salloum, Cyberbullying Classification Methods for Arabic: A Systematic Review. In The International Conference on Artificial Intelligence and Computer Vision (pp. 375-385), (2021).
  * Albadi et al. (2019) Albadi, N., Kurdi, M., & Mishra, S. (2019). Investigating the effect of combining GRU neural networks with handcrafted features for religious hatred detection on Arabic Twitter space. Social Network Analysis and Mining, 9(1), 1-19.
  * Antoun et al. (2020b) W. Antoun, F. Baly, H. Hajj, Araelectra: Pre-training text discriminators for arabic language understanding. arXiv preprint arXiv:2012.15516, (2020).
  * Conneau et al. (2019) A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Edouard Grave, M. Ott, M. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116, (2019).
  * Devlin et al. (2018) J. Devlin, M. W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, (2018).
  * Daouadi et al. (2020) K. E. Daouadi, R. Z. Rebaï, I. Amous, Real-Time Bot Detection from Twitter Using the Twitterbot+ Framework. Journal of Universal Computers Science, 26 (4)(2020) 496-507.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser,Ł & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
  * Sun et al. (2019) Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019, October). How to fine-tune bert for text classification?. In China national conference on Chinese computational linguistics (pp. 194-206). Springer, Cham.