Ginwidth=totalheight=keepaspectratio

# Visualizing Music Genres using a Topic Model

Swaroop Panda  Vinay P. Namboodiri  Shatarupa Thakurta Roy

(Indian Institute of Technology Kanpur)

###### Abstract

Music Genres serve as an important meta-data in the field of music information
retrieval and have been widely used for music classification and analysis
tasks. Visualizing these music genres can thus be helpful for music
exploration, archival and recommendation. Probabilistic topic models have been
very successful in modelling text documents. In this work, we visualize music
genres using a probabilistic topic model. Unlike text documents, audio is
continuous and needs to be sliced into smaller segments. We use simple MFCC
features of these segments as musical words. We apply the topic model on the
corpus and subsequently use the genre annotations of the data to interpret and
visualize the latent space.

##  1 Introduction

It is suggested by [Wells(1992)] that musicologists identify music genres as
pieces of music that have a similar musical language or structure. Music genre
analysis tasks are widely pursued in the music information retrieval
community. However, genre visualizations have not caught enough attention.
Music Genre Visualization can be very helpful in modern technologies such as
custom music playlists and recommendation systems. Probabilistic Topic Models
[Blei(2012)], have found wide applications in the field of Natural Language
processing. We use an unsupervised topic model on music genres data for
visualization. For our work we use raw music files, in .wav format. Unlike
text documents, raw music data has no discrete components such as words. To
create a text-like corpus, we slice the audio data into smaller segments. We
use MFCC features of these smaller slices as the representation. Further, to
build a corpus, we create a feature dictionary by using the k-means algorithm.
Also, in text documents, the inferred topics form a collection of words and
hence are straightforward to interpret. In our case, musical words, which are
mere MFCC feature arrays, lack inherent meaning and cannot be interpreted. We
interpret the latent space of the topic model using genre annotations in the
dataset.

##  2 Related Work

The work in [Cooper et al.(2006)Cooper, Foote, Pampalk, and Tzanetakis]
presented a summary of and had discussed some audio visualization techniques
in MIR which are mostly signal processing based. Topic Models have also been
applied on audio in [Kim et al.(2012)Kim, Georgiou, and Narayanan] for audio
classification and [Kim et al.(2009)Kim, Narayanan, and Sundaram] for audio
information retrieval. The work in [Shalit et al.(2013)Shalit, Weinshall, and
Chechik] uses a topic model to model musical influence. The work in [Hu and
Saul()] applies a topic model on musical notes for key analysis.The work in
[Hirai et al.(2016)Hirai, Doi, and Morishima] and [Hirai et al.(2018)Hirai,
Doi, and Morishima] also use this models on music in the context of DJ
performance. Similarly, the work in [Hariri et al.(2012)Hariri, Mobasher, and
Burke] uses topic models on music tags for use in recommendation systems. We
use the fault-filtered GTZAN [Tzanetakis and Cook(2002)] dataset for genre
analysis which is popular in MIR community.

##  3 The Probabilistic Topic Model

Probabilistic topic models are based upon an idea that documents are mixture
of topics and topics are probability distributions over words. These words
come from a fixed size vocabulary. To make a new document, one chooses a
distribution of topics, then chooses a topic from this distribution and
finally draws a word from the chosen topic. The Latent Dirichlet Allocation
(LDA) inverts this generative process and thus infers the set of topics that
were that useful in generating the document. The plate notation also helps
understand the generative process.

Using some mathematical notations, the generative process can be defined as
follows. Let $K$ be the specified number of topics, $V$ the size of the
vocabulary, $\alpha$ a positive $K$-vector and $\eta$ a scalar. Let
${Dir_{V}}$($\alpha$) denote a $V$-dimensional dirichlet with a vector
parameter $\vec{\alpha}$ and Let ${Dir_{K}}$($\eta$) denote a K-dimensional
dirichlet with a scalar parameter $\eta$.

  1. 1.

For each topic

     * •

Draw a distribution over words $\beta_{K}$ over ${Dir_{V}}$($\eta$)

  2. 2.

For each document

     * •

Draw a vector of topic proportions $\theta_{d}$ over $Dir{(\alpha)}$

     * •

For each word

       * –

Draw a topic assignment ${Z_{d,n}}$ over
Mult(${\theta_{d}}$),${Z_{d,n}}$ $\epsilon$ ${1,2,...,K}$

       * –

Draw a word ${W_{d,n}}$ over
Mult($\beta_{Z_{d,n}}$), ${W_{d,n}}$ $\epsilon$ ${1,2,...,V}$

Figure 1: Plate Notation for a Topic Model

The above figure describes the generative process of the LDA algorithm using a
what is called as a plate notation. The shaded variable ${W_{d,n}}$ is
observable while all other variables are latent. The direction of the arrows
help in understanding the generative process of the algorithm.

##  4 Applying Topic Models on Music

The main challenge with the application of topic models in the music (with raw
audio files) is to represent the audio in a text-document like corpus. The
intent of the work is to interpret the latent space of the topic model using
music genres. This interpretation would help giving a probabilistic genre
annotation to a song. For example a song may belong 60 % to Blues, 15 % to
Jazz and 25 % to Pop genres. To enable such a probabilistic assignment, we
build basic genre buckets consisting of at least 3 genres. We do this since a
mixture containing all the 10 genres would be very large and obfuscating for
the listener to meaningfully interpret. The rationale used to bucket the
genres is roughly based on the histories and the musical form of these genres.
The first bucket consists of Rock, Metal and Pop genres; the second of Blues,
Jazz and Country genres and the final bucket consists of Reggae, Disco and
Hip-Hop genres. The songs were clipped down to 0.10 seconds clips. The MFCC
features of these clips were then calculated. We then use a K means clustering
algorithm on the MFCC features to build the dictionary. It partitions the data
into k clusters, where each data point belongs to a cluster and the cluster
mean serves as its prototype. The librosa package [McFee et al.(2015)McFee,
Raffel, Liang, Ellis, McVicar, Battenberg, and Nieto] was used for the
computation and the pre-processing while the models are implemented using
gensim package. An algorithmic description of the process is given as below.

  1. 1.

For each song

     * •

Discretize into clips of 0.10 secs

     * •

Calculate MFCC features of these clips.

  2. 2.

For all songs in a genre bucket

     * •

Calculate K-means (K=3) of the MFCC feature vectors

     * •

Map each MFCC feature vector (and thus the song) to any of the K-clusters

     * •

Apply the textual probabilistic topic model

##  5 Interpretation of the Latent Space

Unlike text documents, where topics are interpreted as a mixture of words; the
acoustic topic model has topics which are mixtures of cluster means. These
cluster means are prototypes of the nearest datapoints(the audiofiles) and
thus lack meaning. It is thus essential to assign a suitable meaning to these
cluster means. The first part of the interpretation involves understanding the
cluster means in terms of music genre. The cluster means are constructed from
the MFCC arrays. The cluster means hence can be mapped to and from these audio
files and linked with the genre annotations. For instance, lets say that 3
audio files, audio1, audio4 and audio7 make up a cluster mean. We get back to
the dataset and find out that audio1 belongs to the Blues genre, audio4
belongs to the Country genre and the audio7 belongs to the Blues genre. Hence,
the genres associated with cluster means becomes Blues, Country,Blues. In
math, the cluster centers (or terms) can be described as the following,

$\displaystyle clustermean1$ $\displaystyle=\\{Blues,Country,Blues\\}$ (1)
$\displaystyle=0.67Blues+0.33Country$ $\displaystyle clustermean2$
$\displaystyle=\\{Blues,Jazz,Jazz,Country\\}$
$\displaystyle=0.25Blues+0.5Jazz+0.25Country$

Once we interpret cluster means in terms of music genres, we can conveniently
represent the topics in terms of music genres. The topic space consists of
cluster means and an associated probability value. The cluster means can now
be defined as music genres with their proportions.

$\begin{split}Topic1&=prob1*clustermean1+prob2*clustermean2\\\
&=(prob1*0.67+prob2*0.25)Blues+\\\ &(prob1*0.33+prob2*0.25)Country\\\
&+(prob2*0.5)Jazz\\\ \end{split}$ (2)

###  5.1 Document-Topic Proportions

Once the topic space has been interpreted, the document-topic proportions can
also be made sense of. The document topic proportions from the topic model are
probability values of the inferred topics present in each document. In this
context, the document topic proportions can provide with the proportions of
different music genres present within the musical document, that is, a song.

$\begin{split}Doc1&=prob1*topic1+prob2*topic2\\\
&=prob1*(0.33Blues+0.33Country+0.34Jazz)\\\ &+prob2*(0.5Jazz+0.5Country)\\\
&=(prob1*0.33)Blues+(prob1*0.34+prob2*0.5)Jazz\\\
&+(prob1*0.33+prob2*0.5)Country\\\ \end{split}$ (3)

###  5.2 The Term-Topic Proportions

The term-topic proportions suggest the topic distributions in each term. So
the term-topic distribution represents the genres present in each term, or
cluster center. This is represented in an equation as,

$\begin{split}Term1&=prob1*topic1+prob2*topic4\\\
&=prob1*(0.33Blues+0.50Country+0.17Jazz)\\\ &+prob2*(0.5Blues+0.5Jazz)\\\
&=(prob1*0.33+prob2*0.5)Blues+(prob1*0.17+prob2*0.5)Jazz\\\
&+(prob2*0.5)Country\\\ \end{split}$ (4)

The term-topic proportions is an interesting insight derived from the topic
models. The terms are cluster centers derived from MFCC feature arrays. These
feature arrays are in turn derived from small segments of a song. So the term-
topic proportion suggest the genre proportions of each small segment of a
song. For instance, 3 small segments of a song can belong to Blues-Classical,
Jazz-Pop-Classical or Disco-Rock genres in some proportions.

##  6 Evaluating the Topic Model

We evaluate the model using a genre classification task. We use the model to
get the document-topic proportions of every document(song). We use these
document-topic proportions as a representation for each song. We use genre
labels from the fault-filtered GTZAN dataset, divide the data into train-test
sets and perform the classification task using a SVM. We also test our model
with different number of topics to look for the optimal number of topics that
best capture the genre bucket.

| 2 | 3 | 4 |  5
---|---|---|---|---
1 | 0.47 | 0.53 | 0.58 |  0.53
2 | 0.36 | 0.38 | 0.35 |  0.40
3 | 0.53 | 0.46 | 0.48 |  0.50
Table 1: Accuracies obtained for different number of topic terms. The rows
represent the genre bucket, while the columns represent the number of topics

##  7 Music Genre Visualization

Using the topic model, we can get a probabilistic genre labels of different
songs(from document-topic proportions) along with progressive genre
visualizations(from term-topic proportions).

Figure 2: The above is a visualization of two topics emerging out of genre
buckets 1 and 2 respectively. The colours represent the music genres and the
doughnut partitions show genre proportions within a topic. Figure 3: The
progressive viz of a song segment. The x-axis denotes time and the y-axis the
genre proportions. The above figure displays the changing genre proportions as
the song progresses.

##  8 Conclusion

In this work, we applied a Probabilistic Topic Model on raw Music data and
used the available genre annotations to interpret the latent space. We used
the inferred parameters of the Topic Model to visualize the Music Data and
also proposed some genre visualizations. Further, it would be interesting to
explore other audio features and genre data for visualization using the Topic
Model.

## References

  * [Blei(2012)] David M Blei.  Probabilistic topic models.  _Communications of the ACM_ , 55(4):77–84, 2012\.
  * [Cooper et al.(2006)Cooper, Foote, Pampalk, and Tzanetakis] Matthew Cooper, Jonathan Foote, Elias Pampalk, and George Tzanetakis.  Visualization in audio-based music information retrieval.  _Computer Music Journal_ , 30(2):42–62, 2006\.
  * [Hariri et al.(2012)Hariri, Mobasher, and Burke] Negar Hariri, Bamshad Mobasher, and Robin Burke.  Context-aware music recommendation based on latenttopic sequential patterns.  In _Proceedings of the sixth ACM conference on Recommender systems_ , pages 131–138. ACM, 2012.
  * [Hirai et al.(2016)Hirai, Doi, and Morishima] Tatsunori Hirai, Hironori Doi, and Shigeo Morishima.  Musicmixer: Automatic dj system considering beat and latent topic similarity.  In _International Conference on Multimedia Modeling_ , pages 698–709. Springer, 2016.
  * [Hirai et al.(2018)Hirai, Doi, and Morishima] Tatsunori Hirai, Hironori Doi, and Shigeo Morishima.  Latent topic similarity for music retrieval and its application to a system that supports dj performance.  _Journal of Information Processing_ , 26:276–284, 2018\.
  * [Hu and Saul()] Diane J Hu and Lawrence K Saul.  A probabilistic topic model for music analysis.
  * [Kim et al.(2009)Kim, Narayanan, and Sundaram] Samuel Kim, Shrikanth Narayanan, and Shiva Sundaram.  Acoustic topic model for audio information retrieval.  In _Applications of Signal Processing to Audio and Acoustics, 2009\. WASPAA’09. IEEE Workshop on_ , pages 37–40. IEEE, 2009.
  * [Kim et al.(2012)Kim, Georgiou, and Narayanan] Samuel Kim, Panayiotis Georgiou, and Shrikanth Narayanan.  Latent acoustic topic models for unstructured audio classification.  _APSIPA Transactions on Signal and Information Processing_ , 1, 2012\.
  * [McFee et al.(2015)McFee, Raffel, Liang, Ellis, McVicar, Battenberg, and Nieto] Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto.  librosa: Audio and music signal analysis in python.  In _Proceedings of the 14th python in science conference_ , pages 18–25, 2015.
  * [Shalit et al.(2013)Shalit, Weinshall, and Chechik] Uri Shalit, Daphna Weinshall, and Gal Chechik.  Modeling musical influence with topic models.  In _International Conference on Machine Learning_ , pages 244–252, 2013.
  * [Tzanetakis and Cook(2002)] George Tzanetakis and Perry Cook.  Musical genre classification of audio signals.  _IEEE Transactions on speech and audio processing_ , 10(5):293–302, 2002.
  * [Wells(1992)] Paul F Wells.  Origins of the popular style: The antecedents of twentieth-century popular music, 1992.


# A Theoretical Analysis of Logistic Regression and Bayesian Classifiers

Roman V. Kirin
Department of Applied Economics
Higher School of Economics
Pokrovsky blvd. 11, Moscow
<EMAIL_ADDRESS>
###### Abstract

This study aims to show the fundamental difference between logistic regression
and Bayesian classifiers in the case of exponential and unexponential families
of distributions, yielding the following findings. First, the logistic
regression is a less general representation of a Bayesian classifier. Second,
one should suppose distributions of classes for the correct specification of
logistic regression equations. Third, in specific cases, there is no
difference between predicted probabilities from correctly specified generative
Bayesian classifier and discriminative logistic regression.

_JEL Classification:_ C13, C18, C25, C35.

_Keywords:_ Bayes’ theorem, distribution families, logistic regression, method
of moments.

##  1 Introduction

The comparison of generative and discriminative models has been a perennial
topic of discussion. An example is a comparison of naive Bayes classifier and
logistic regression in terms of a classification quality or an error (Ng and
Jordan, 2002; L. Mary Gladence and Anu, 2015; Tsangaratos and Ilia, 2016;
Prabhat and Khullar, 2017; Pranckevicius and Marcinkevičius, 2017; Aborisade
and Anwar, 2018; Helmi Setyawan et al., 2018; Hasanli and Rustamov, 2019; Seka
et al., 2019; Itoo et al., 2020). However, what has been unreasonably ignored
in this debate is the fundamental relationship between the models, when both
classes have a multivariate normal distribution with similar covariance
matrices (Efron, 1975; Rubinstein and Hastie, 1997).

Efron (1975) stated that the conditional likelihood for logistic regression ‘…
is valid under general exponential family assumptions …’ However, this study
shows that it is correct only under the linearity by the features assumption
of the discriminant function.

The novelty of this study is that it offers:

  * •

a generalization for the multiclass task;

  * •

a solution without a likelihood maximization;

  * •

a covariance dissimilarity for classes with multivariate normal distribution;

  * •

an analysis in terms of families of distributions, not prediction accuracy.

##  2 Problem statement

###  2.1 General case

Suppose we have a sample $\left\\{y_{i\ },\ x_{i}\right\\}_{i=1}^{N}$ where
$y_{i}\in\left\\{1,\ 2,\ ...,\ M\right\\}$ denotes the correct class label,
and $x_{i}$ denotes the feature vector of size $d$ for observation $i$. There
are two well-known approaches to estimate parameters with maximum likelihood:
generative (informative) and discriminative (Rubinstein and Hastie, 1997).
Instead of likelihood maximization, let us directly draw the conditional
probability for an observation.

Suppose $m$ is a target class, whose conditional probability
$P\left(y_{i}=m\mid x_{i}\right)$ we want to estimate. According to Bayes’
theorem, the estimated probability can be written as a classifier (1):

$P\left(y_{i}=m\middle|x_{i}\right)=\\\
\frac{P\left(x_{i}\middle|y_{i}=m\right)\times
P(y_{i}=m)}{P\left(x_{i}\middle|y_{i}=m\right)\times
P\left(y_{i}=m\right)+\sum_{s\neq m}P\left(x_{i}\middle|y_{i}=s\right)\times
P(y_{i}=s)}=\\\ \frac{1}{1+\sum_{s\neq
m}\frac{P\left(x_{i}\middle|y_{i}=s\right)\times
P\left(y_{i}=s\right)}{P\left(x_{i}\middle|y_{i}=m\right)\times
P\left(y_{i}=m\right)}}$ (1)

The unconditional probability for any class $s$ and observation $i$ equals
$P\left(y_{i}=s\right)=\frac{n_{s}}{N}$, where $n_{s}$ is the number of
objects with class $s$. The conditional probability
$P\left(x_{i}\middle|y_{i}=s\right)$ depends on a specific distribution and a
family of the distributions.

###  2.2 Exponential families

Consider that the conditional probability for any class $s$ and any
observation $i$ belongs to the exponential family of distributions. Hence,
$P\left(x_{i}\middle|y_{i}=s\right)$ can be represented as
$P(x_{i};\theta_{s})=\exp\left(\left\langle
t_{s}\left(x_{i}\right),\theta_{s}\right\rangle-
F_{s}\left(\theta_{s}\right)+k_{s}\left(x_{i}\right)\right)$, where
$t_{s}(x_{i})$ is the sufficient statistic, $\theta_{s}$ denotes the natural
parameters, $\langle.,.\rangle$ is the inner product (commonly called dot
product), $F\left(\cdot\right)$ is the log-normalizer, and
$k_{s}\left(x_{i}\right)$ is the carrier measure (Nielsen and Garcia, 2011).

Under this consideration equation (2) is appropriate, where log-odds ratio
equals (3) and the log-fraction of unconditional probabilities equals (4).

$P\left(y_{i}=m\middle|x_{i}\right)=\frac{1}{1+\sum_{s\neq
m}e^{-(\ln{\rho_{m,s}^{cond}(x_{i})}+\ln{\rho_{m,s}^{uncond}})}}$ (2)

$\ln{\rho_{m,s}^{cond}\left(x_{i}\right)}=\ln{\frac{P\left(x_{i}\middle|y_{i}=m\right)}{P\left(x_{i}\middle|y_{i}=s\right)}}=\left(\langle
t_{m}\left(x_{i}\right),\theta_{m}\rangle-\langle
t_{s}\left(x_{i}\right),\theta_{s}\rangle\right)-\left(F_{m}\left(\theta_{m}\right)-F_{s}\left(\theta_{s}\right)\right)+\left(k_{m}\left(x_{i}\right)-k_{s}\left(x_{i}\right)\right)$
(3)

$\ln\rho_{m,s}^{uncond}=\ln\frac{P\left(y_{i}=m\right)}{P\left(y_{i}=s\right)}=\ln\frac{n_{m}}{n_{s}}$
(4)

As a result, any Bayesian classifier can be represented as a logistic
regression, which was known (Efron, 1975). But there is no guarantee that the
discriminant function
$z_{m,s}\left(x_{i}\right)=\ln\rho_{m,s}^{cond}\left(x_{i}\right)+\ln\rho_{m,s}^{uncond}$
can be represented as $\beta_{0}+\beta^{T}x_{i}$.

##  3 Particular solutions

###  3.1 Examples for exponential families

####  3.1.1 Univariate normal distribution

According to the given specification, there are $M$ normally distributed
classes, with only one feature ($x_{i}$ is 1-dimensional). Then, for each
class $s$ a number of equations (5) are valid (Nielsen and Garcia, 2011).

$\displaystyle\begin{split}\theta_{s}&=\left(\theta_{s,1},\theta_{s,2}\right)=\left(\frac{\mu_{s}}{\sigma^{2}_{s}},-\frac{1}{2\sigma^{2}_{s}}\right)\\\
t_{s}\left(x_{i}\right)&=\left(x_{i},x_{i}^{2}\right)\\\
F_{s}\left(\theta_{s}\right)&=-\frac{\theta_{s,1}^{2}}{4\theta_{s,2}}+\frac{1}{2}\ln{\left(-\frac{\pi}{\theta_{s,2}}\right)}=\\\
&=\frac{1}{2}\left(\frac{\mu^{2}_{s}}{\sigma^{2}_{s}}\right)+\frac{1}{2}\ln{\left(2\pi\right)}+\frac{1}{2}\ln{\left(\sigma^{2}_{s}\right)}\\\
k_{s}\left(x_{i}\right)&=0\end{split}$ (5)

The discriminant function for a couple and will take this form (6).

$\displaystyle\begin{split}z_{m,s}(x_{i})&=\alpha_{m,s}+\beta_{m,s}\times
x_{i}+\gamma_{m,s}\times x_{i}^{2}\\\
\alpha_{m,s}&=\ln{\left(\frac{n_{m}}{n_{s}}\right)}-\frac{1}{2}\left(\ln{\left(\frac{\sigma_{m}^{2}}{\sigma_{s}^{2}}\right)}+\left(\frac{\mu_{m}^{2}}{\sigma_{m}^{2}}-\frac{\mu_{s}^{2}}{\sigma_{s}^{2}}\right)\right)\\\
\beta_{m,s}&=\left(\frac{\mu_{m}}{\sigma_{m}^{2}}-\frac{\mu_{s}}{\sigma_{s}^{2}}\right)\\\
\gamma_{m,s}&=-\frac{1}{2}\left(\frac{1}{\sigma_{m}^{2}}-\frac{1}{\sigma_{s}^{2}}\right)\end{split}$
(6)

The solution for the specific case leads to the following findings. First, the
discriminant function is not linear by feature, in general. Second, if feature
variances are the same for both classes, then the logistic regression
equations are linear.

####  3.1.2 Multivariate normal distribution

This section presents the solution for a set of multivariate normally
distributed classes. It describes the generalization of the previous
subsection when $x_{i}$ is $d$-dimensional. According to the current task,
equations (7) are valid.

$\displaystyle\begin{split}\mathbf{\Theta_{s}}&=\left(\theta_{s},\Theta_{s}\right)=\left(\Sigma^{-1}_{s}\mu_{s},\frac{1}{2}\Sigma^{-1}_{s}\right)\\\
t_{s}\left(x_{i}\right)&=\left(x_{i},-x_{i}x_{i}^{T}\right)\\\
F_{s}\left(\mathbf{\Theta_{s}}\right)&=\frac{1}{4}tr\left(\Theta_{s}^{-1}\theta_{s}\theta_{s}^{T}\right)-\frac{1}{2}\ln{\left(\det{\Theta_{s}}\right)}+\frac{d}{2}\ln{\left(\pi\right)}\\\
&=\frac{1}{2}\mu_{s}^{T}\Sigma_{s}^{-1}\mu_{s}+\frac{1}{2}\ln{\left(\det{\Sigma_{s}}\right)}+\frac{d}{2}\ln{\left(2\pi\right)}\\\
k_{s}\left(x_{i}\right)&=0\end{split}$ (7)

For simplicity, let $\Omega_{s}=\Sigma_{s}^{-1}$ for any class $s$ and
$\omega_{s,j,h}$ be the corresponding element of the matrix $\Omega_{s}$. Then
the discriminant function can be represented as (8).

$\displaystyle\begin{split}z_{m,s}(x_{i})&=\alpha_{m,s}+\sum_{j=1}^{d}\beta_{m,s,j}x_{i,j}+\sum_{j=1}^{d}{\sum_{h=1}^{d}\gamma_{m,s,j,h}}x_{i,j}x_{i,h}\\\
\alpha_{m,s}&=\ln{\left(\frac{n_{m}}{n_{s}}\right)}-\frac{1}{2}\left(\ln{\left(\frac{\det{\Sigma_{m}}}{\det{\Sigma_{s}}}\right)}+\left(\mu_{m}^{T}\Sigma_{m}^{-1}\mu_{m}-\mu_{s}^{T}\Sigma_{s}^{-1}\mu_{s}\right)\right)\\\
\beta_{m,s,j}&=\sum_{h=1}^{d}\left(\omega_{m,j,h}\mu_{m,h}-\omega_{s,j,h}\mu_{s,h}\right)\\\
\gamma_{m,s,j,h}&=-\frac{1}{2}\left(\omega_{m,j,h}-\omega_{s,j,h}\right)\\\
\end{split}$ (8)

The analytical solution in this section yields two additional findings. First,
if features are not independent (at least for one class), then feature
interactions (including squares) should be added to the regression equations.
Second, if the features are independent and their variance is the same (for
all classes), then the regression equations are linear.

###  3.2 Example for unexponential families

####  3.2.1 Univariate uniform distribution

Consider two classes: $x_{i}\mid y_{i}=1\sim U\left[a,c\right]$ and $x_{i}\mid
y_{i}=2\sim U\left[b,d\right]$, where $a<b<c<d$. Then according to the
Bayesian classifier (1), the estimated probability is that the observation i
drawn from the class $m$ equals (9).

$P(y_{i}=1\mid x_{i})=\begin{cases}0,&\text{if }x_{i}<a\\\ 1,&\text{if }a\leq
x_{i}<b\\\ \frac{1}{1+\frac{P\left(x_{i}\middle|y_{i}=2\right)\times
P\left(y_{i}=2\right)}{P\left(x_{i}\middle|y_{i}=1\right)\times
P\left(y_{i}=1\right)}},&\text{if }b\leq x_{i}\leq c\\\ 0,&\text{if
}x_{i}>c\end{cases}$ (9)

Obviously, the task cannot be solved by logistic regression with a simple
linear equation, especially for probability 1. As a result, standard logistic
regression does not work on an unexponential family, unlike a Bayesian
classifier.

##  4 Conclusion

As models, Bayesian classifiers are more general than simple logistic
regression. They are appropriate for both exponential and unexponential
distribution families (in the cases considered). Under Bayesian classifiers,
there always exists a distribution assumption and there can be an additional
restriction on parameters, but they should not be naive by default. The main
assumptions of classical logistic regression are: a) exponential distribution
family of classes, and b) linearity by features of the discriminant function.
Both assumptions do not always hold, but if the first is appropriate and the
regression equations are correctly specified, then logistic regression is a
useful representation (especially in econometrics) and leads to a correct
solution. Moreover, learning Bayesian classifier is more computationally
efficient because moment’s method can be used. For logistic regression, only
variations of gradient ascent are available.

## Conflict of Interest

None.

## Funding

This research did not receive any specific grants from funding agencies in the
public, commercial, or not-for-profit sectors.

## References

  * Ng and Jordan [2002] Andrew Ng and Michael Jordan.  On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes.  _Adv. Neural Inf. Process. Sys_ , 2, 04 2002.
  * L. Mary Gladence and Anu [2015] M. Karthi L. Mary Gladence and V. Maria Anu.  A statistical comparison of logistic regression and different bayes classification methods for machine learning.  _ARPN Journal of Engineering and Applied Sciences_ , 10, 2015.
  * Tsangaratos and Ilia [2016] Paraskevas Tsangaratos and Ioanna Ilia.  Comparison of a logistic regression and Naïve Bayes classifier in landslide susceptibility assessments: The influence of models complexity and training dataset size.  _CATENA_ , 145:164–179, 2016.  ISSN 0341-8162.  doi:https://doi.org/10.1016/j.catena.2016.06.004.  URL https://www.sciencedirect.com/science/article/pii/S0341816216302090.
  * Prabhat and Khullar [2017] Anjuman Prabhat and Vikas Khullar.  Sentiment classification on big data using Naïve bayes and logistic regression.  In _2017 International Conference on Computer Communication and Informatics (ICCCI)_ , pages 1–5, 2017.  doi:10.1109/ICCCI.2017.8117734.
  * Pranckevicius and Marcinkevičius [2017] Tomas Pranckevicius and Virginijus Marcinkevičius.  Comparison of Naive Bayes, Random Forest, Decision Tree, Support Vector Machines, and Logistic Regression Classifiers for Text Reviews Classification.  _Baltic Journal of Modern Computing_ , 5, 01 2017.  doi:10.22364/bjmc.2017.5.2.05.
  * Aborisade and Anwar [2018] Opeyemi Aborisade and Mohd Anwar.  Classification for Authorship of Tweets by Comparing Logistic Regression and Naive Bayes Classifiers.  In _2018 IEEE International Conference on Information Reuse and Integration (IRI)_ , pages 269–276, 2018.  doi:10.1109/IRI.2018.00049.
  * Helmi Setyawan et al. [2018] Muhammad Yusril Helmi Setyawan, Rolly Maulana Awangga, and Safif Rafi Efendi.  Comparison Of Multinomial Naive Bayes Algorithm And Logistic Regression For Intent Classification In Chatbot.  In _2018 International Conference on Applied Engineering (ICAE)_ , pages 1–5, 2018.  doi:10.1109/INCAE.2018.8579372.
  * Hasanli and Rustamov [2019] Huseyn Hasanli and Samir Rustamov.  Sentiment Analysis of Azerbaijani twits Using Logistic Regression, Naive Bayes and SVM.  In _2019 IEEE 13th International Conference on Application of Information and Communication Technologies (AICT)_ , pages 1–7, 2019.  doi:10.1109/AICT47866.2019.8981793.
  * Seka et al. [2019] D. Seka, B.S. Bonny, A.N. Yoboué, S.R. Sié, and B.A. Adopo-Gourène.  Identification of maize (Zea mays L.) progeny genotypes based on two probabilistic approaches: Logistic regression and naïve Bayes.  _Artificial Intelligence in Agriculture_ , 1:9–13, 2019\.  ISSN 2589-7217.  doi:https://doi.org/10.1016/j.aiia.2019.03.001.  URL https://www.sciencedirect.com/science/article/pii/S2589721719300030.
  * Itoo et al. [2020] Fayaz Itoo, Meenakshi, and Satwinder Singh.  Comparison and analysis of logistic regression Naïve Bayes and KNN machine learning algorithms for credit card fraud detection.  _International Journal of Information Technology_ , feb 2020.  doi:10.1007/s41870-020-00430-y.  URL https://doi.org/10.1007%2Fs41870-020-00430-y.
  * Efron [1975] Bradley Efron.  The Efficiency of Logistic Regression Compared to Normal Discriminant Analysis.  _Journal of the American Statistical Association_ , 70(352):892–898, 1975.  doi:10.1080/01621459.1975.10480319.
  * Rubinstein and Hastie [1997] Y. Dan Rubinstein and Trevor Hastie.  Discriminative vs Informative Learning.  In _Proceedings of the Third International Conference on Knowledge Discovery and Data Mining_ , KDD’97, page 49–53. AAAI Press, 1997.
  * Nielsen and Garcia [2011] Frank Nielsen and Vincent Garcia.  Statistical exponential families: A digest with flash cards, 2011.


# BRUMS at SemEval-2020 Task 12 : Transformer based Multilingual Offensive
Language Identification in Social Media

Tharindu Ranasinghe1, Hansi Hettiarachchi2
1Research Group in Computational Linguistics, University of Wolverhampton, UK
2School of Computing and Digital Technology, Birmingham City University, UK
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

In this paper, we describe the team BRUMS entry to OffensEval 2: Multilingual
Offensive Language Identification in Social Media in SemEval-2020. The
OffensEval organizers provided participants with annotated datasets containing
posts from social media in Arabic, Danish, English, Greek and Turkish. We
present a multilingual deep learning model to identify offensive language in
social media. Overall, the approach achieves acceptable evaluation scores,
while maintaining flexibility between languages.

##  1 Introduction

00footnotetext:  This work is licensed under a Creative Commons Attribution
4.0 International Licence. Licence details:
http://creativecommons.org/licenses/by/4.0/.

Social media has become a normal medium of communication for people these days
as it provides the convenience of sending messages fast from a variety of
devices. Unfortunately, social networks also provide the means for
distributing abusive and aggressive content. Given the amount of information
generated every day on social media, it is not possible for humans to identify
and remove such messages manually, instead it is necessary to employ automatic
methods. As offensive language becomes pervasive in social media, scholars and
companies have been working on developing systems capable of identifying
offensive posts, which can be set aside for human moderation or permanently
deleted [Risch and Krestel, 2018].

Along with these studies, a few shared tasks have been organized on detecting
offense in social media, such as HatEval [Basile et al., 2019], HASOC [Mandl
et al., 2019], TRAC [Kumar et al., 2018] and OffensEval [Zampieri et al.,
2019b] co-located with SemEval 2019. In semeval 2020, OffensEval returns for
the second time with multilingual offensive language identification in social
media. OffensEval 2 focuses on several languages Arabic, Danish, English,
Greek and Turkish motivating participants to submit multilingual offensive
language identification systems.

This paper revisits the problem of offensive language identification
describing our submission to the SemEval-2020 Task 1 [Zampieri et al., 2020].
The remainder of this paper is structured as follows: Section 2 describes the
related work done in the field of aggression detection, Section 3 has a
description of the dataset. Section 4 describes the system that was submitted,
split into a description of how the data was processed and the architectures
of the classifiers that were used. Section 5 presents an analysis of the
results of our evaluation of the different architectures, as well as of the
final submission. Finally, Section 6 offers some final remarks and a
conclusion.

##  2 Relevant Work

The majority of text classification approaches in early research work were
supported by traditional or feature-based supervised learning methods.
Following this tradition, Malmasi and Zampieri [Malmasi and Zampieri, 2017]
proposed a linear Support Vector Machine (SVM) classifier for hate speech
detection in social media. The best accuracy of this approach was obtained by
using character n-grams as model features. Another research suggested stacked
ensemble system using logistic regression and random forest-based classifiers
[Montani, 2018]. For initial layer of classifiers, they used character
n-grams, token n-grams, important tokens and word2vec embeddings. The Outputs
of initial layer classifiers were used as the inputs of next classifier layer
which produces the final result. This approach obtained the best results in
GermEval 2018 [Wiegand et al., 2018] by outperforming the deep neural network
(DNN) models such as convolutional neural networks (CNN) [von Grünigen et al.,
2018] and combination of bi-directional Long-Short term memory and CNN
(BiLSTM-CNN) [Wiedemann et al., 2018].

However, most of the recent research conducted in this area prove that DNN-
based methods can outperform traditional supervised learning methods [Zampieri
et al., 2019a, Modha et al., 2018]. But, unlike the traditional methods, DNNs
require sufficient amount of training data to adjust their weights properly
without overfitting. Focusing on this issue, Modha et al. [Modha et al., 2018]
found that LSTM with higher dropout ($\approx 0.5$) can handle overfitting
problem. In order to increase the model performance using a high number of
training instances in a simple manner, the available data sets related to
aggression and cyberbullying detection can be merged [Fortuna et al., 2018].
But, when focused on a particular area, it cannot guarantee the availability
of multiple data sets. Another research suggested data augmentation and pseudo
labelling approaches to increase the amount of training data [Aroyehun and
Gelbukh, 2018]. As the data augmentation strategy, they proposed to translate
each instance into an intermediate language and back to the original language.
For pseudo labelling approach, they trained some initial models using
available training data and picked the best model to label new data. A LSTM
model trained using the augmented data set received the first place in
Trolling, Aggression and Cyberbullying (TRAC-2018) shared task designed for
Facebook English data set. In the same shared task, for the platform unnamed
English data set, third place could be obtained by a CNN-LSTM model trained
using the combination of augmented and pseudo labelled data sets.

In addition to approaching data set increasing methods, transfer learning
techniques also used by previous research to overcome the data limitation
issue. Wiedemann et al. [Wiedemann et al., 2018] showed that transfer learning
can be applied using some labelled data set with relevant labels or unlabelled
data set with unsupervised user clusters or topics. Further, as mentioned in
SemEval-2019 OffensEval leader-board, the top 5 solutions for offensive
language identification are BERT-based and they proved the effectiveness in
transfer learning as well as Transformers [Zampieri et al., 2019b]. The top
solution of this shared task fined tuned a BERT model using the training data
and found that it outperforms a LSTM model with higher dropout [Liu et al.,
2019a].

In summary, majority of recent research in this area were supported by DNN
models and among them, CNN, LSTM and BERT-based models were found to be most
effective and commonly used. When focused on the input data representation,
there was a tendency to use fastText embeddings to handle the high
availability of spelling mistakes and new words in social media [Aroyehun and
Gelbukh, 2018, Modha et al., 2018, Wiedemann et al., 2018, Zampieri et al.,
2019a]. Following this tendency, in this research, we further experimented the
effectiveness of DNN models including some recently published transformers for
offensive language identification.

##  3 Dataset

OffensEval 2020, the SemEval 2020 Offensive Language Detection Shared Task,
does not provide a new manually labeled training set for the English language
[Zampieri et al., 2020]. The competitors were recommended to use the Offensive
Language Identification Dataset (OLID) which was released for the the SemEval
2019 Offensive Language Detection Shared Task [Zampieri et al., 2019b]. The
OLID is a hierarchical dataset to identify the type and the target of
offensive texts in social media. The dataset is collected on Twitter and
publicly available. There are 14,100 tweets in total, in which 13,240 are in
the training set, and 860 are in the test set. For each tweet,there are three
levels of labels:(A) Offensive/Not-Offensive, (B) Targeted-Insult/Untargeted,
(C) Indi-vidual/Group/Other. For English subtask organisers provided an
additional dataset similar to OLID [Zampieri et al., 2019b], it still has
three levels, but this time only confidence scores, generated by different
models, are provided rather than human annotated labels. In addition, the data
in level A is separated from levels B and C. In level A, there are 9,089,140
tweets,in levels B and C, there are different 188,973 tweets.

For non-English languages the organisers released Multilingual Offensive
Language Identification Dataset (MOLID). It contains four languages: Arabic
[Mubarak et al., 2020], Danish [Sigurbergsson and Derczynski, 2020], Greek
[Pitenis et al., 2020], and Turkish [Çöltekin, 2020]. Table 1 contains
information about number of rows that the training, dev and test datasets had
in each language [Zampieri et al., 2020].

Language | training | dev | test
---|---|---|---
Arabic | 7000 | 1000 | 2000
Danish | 2961 | - | 330
Greek | 8744 | - | 1545
Turkish | 31757 | - | 3529
Table 1: Number of rows in Training set(training), Development set(dev) and
Test set(test) in each language. ”-” denotes that no data was available.

##  4 Methodology

In this section we present the methodology employed for the shared task. All
the languages followed a common methodology which had two steps: Data
Preprocessing and Machine Learning.

###  4.1 Data Preprocessing

As mentioned previously, the data preprocessing for this task was kept fairly
minimal to make it portable for all the languages. More specifically, we
perform only four specialised tasks for this data, followed by tokenisation.
The tasks include removing usernames, removing URLs, converting all the emojis
to text and converting all tokens to lower case.

First, we completely remove all usernames from the tweets, without inserting a
placeholder. In the SemEval-2020 dataset, mention of a certain user is
represented by @USER. We removed all strings containing @USER using a regular
expression. The reasoning behind this step is mainly to remove noisy text.
Then we remove all URL mentions in the tweets. In the SemEval-2020 dataset,
mention of a certain URL is represented by URL. We removed all strings
containing URL using a regular expression. The reasoning behind this step too
is mainly to remove noisy text. Emojis play an important role in showing
aggression in social media [Hettiarachchi and Ranasinghe, 2019]. Since we
can’t guarantee that the pretrained embedding models we use will have
embeddings for emjois, we converted all the emojis to text using a third party
python library named Emoji 111https://github.com/carpedm20/emoji/. We only
conducted this step for English, since the Emoji library does not support
other languages.

For Arabic language, we used a separate preprocessing step. Twitter users
writing in Arabic can either write in Modern Standard Arabic (MSA) or in their
particular dialects. If the tweet is written in MSA, it may or may not include
diacritics whereas dialectical Arabic does not include any. Offensive tweets
can be formulated in any of these different versions of the Arabic transcript.
Thus, in order to avoid false classifications due to a non-defining feature,
we used regular expressions to delete diacritics.

Final prepossessing step is only applied to the architectures that used
character embeddings. The fastText pretrained character embedding models that
we used only contain lower-cased letters. Therefore, we convert the text to
lower case letters. However, for the Transformer architecture, we did this
preprocessing step only if the model that we used is uncased.

###  4.2 Machine Learning

In order to determine the most suitable neural network architecture for the
task, we experimented with three different neural architectures: Convolutional
Neural Network (CNN) [Kim, 2014], Recurrent Neural Network (RNN) [Cui et al.,
2018] and Transformer [Devlin et al., 2018].

####  4.2.1 Convolutional Neural Network (CNN)

CNN is a class of deep, feed-forward artificial neural networks that uses a
variation of multilayer perceptrons [Kim, 2014]. CNNs are generally used in
computer vision, however they have recently been applied to various NLP tasks
and the results were promising [Kim, 2014, Hughes et al., 2017]. In a CNN
result of each convolution will fire when a special pattern is detected. By
varying the size of the kernels and concatenating their outputs, we can detect
patterns of multiples sizes (2, 3, or 5 adjacent words) [Kim, 2014]. Patterns
could be expressions like “fuck it”, “stupid cunt” and CNNs can identify them
in the sentence regardless of their position. Since offense is mostly a word
pattern, we assumed that CNNs would be a good architecture to detect offensive
sentences.

For this architecture we used fasttext character embeddings [Bojanowski et
al., 2016]. After the embedding layer we used four convolution layers with
1,2,3 and kernal sizes followed by a max pooling layer, [Scherer et al.,
2010]. Finally the outputs of the max pooling layers were concatenated and
passed through a linear layer. For all the languages in the task, we used
fasttext character embeddings released by facebook [Grave et al., 2018]
222https://fasttext.cc/docs/en/crawl-vectors.html.

####  4.2.2 Recurrent Neural Network (RNN)

RNNs are designed to make use of sequential data, when the current step has
some kind of relation with the previous steps. [Cui et al., 2018]. This makes
them ideal for applications with a time component (audio, time-series data)
and natural language processing [Pitenis et al., 2020]. RNNs perform very well
for applications where sequential information is clearly important, because
the meaning could be misinterpreted if sequential information is not used.
Since sequential information is important to identify offensive sentences, we
used the RNN architecture for this task. For this architecture too we used
fasttext character embeddings [Bojanowski et al., 2016] trained on English
Wikipedia text. After the embedding layer we used a RNN layer. We experimented
with LSTM [Hochreiter and Schmidhuber, 1997], bi-directional LSTM [Graves et
al., 2005], GRU [Chung et al., 2014] and bi-directional GRU [Chung et al.,
2014]. The output of the RNN layer was followed a linear layer finally. For
this architecture also we used fasttext character embeddings released by
facebook.

####  4.2.3 Transformers

With the introduction of BERT [Devlin et al., 2018] transformer architectures
have shown a massive success in wide range of NLP tasks. Transformer
architectures have been trained on general tasks like language modelling and
then fine-tuned for classification tasks. [Sun et al., 2019, Ranasinghe et
al., 2019].

Transformer models take an input of a sequence and outputs the representation
of the sequence. The sequence has one or two segments that the first token of
the sequence is always [CLS] which contains the special classification
embedding and another special token [SEP] is used for separating segments. For
text classification tasks, Transformer models take the final hidden state h of
the first token [CLS] as the representation of the whole sequence [Sun et al.,
2019]. A simple softmax classifier is added to the top of the transformer
model to predict the probability of label c as shown in Equation 1 where W is
the task-specific parameter matrix .

$p(c|\textbf{h})=softmax(W\textbf{h})$ (1)

We fine-tuned all the parameters from transformer models as well as W jointly
by maximising the log-probability of the correct label. For English language,
there exists many pretrained transformer models released by Hugging
Face333https://huggingface.co. Therefore, we experimented several transformer
architectures like BERT [Devlin et al., 2018], XLNet [Yang et al., 2019], XLM
[Conneau et al., 2019], RoBERTa [Liu et al., 2019b] and DistilBERT [Sanh et
al., 2019]. We used the HuggingFace’s implementation of the transformer models
[Wolf et al., 2019] and the pre-trained models available in the HuggingFace’s
model repository444https://huggingface.co/models. However, for non-English
languages, Google only provides a single BERT multilinugal model that supports
104 languages555https://github.com/google-
research/bert/blob/master/multilingual.md. DistilBERT also has a multilingual
model that supports the same 104 languages that multilingual BERT supports. We
used these two models for all the non-English languages. To experiment more,
we also experimented several community built transformer models.

For Arabic, other than the multilingual BERT model and multilingual DistilBERT
we used AraBERT [Antoun et al., 2020] 666https://github.com/aub-mind/arabert.
AraBERT is an Arabic pretrained lanaguage model based on Google’s BERT
architechture. The model was trained on 70M sentences or 23GB of Arabic text
with 3B words. The training corpora are a collection of publicly available
large scale raw Arabic text (Arabic Wikidumps, The 1.5B words Arabic Corpus
[El-Khair, 2016], The OSIAN Corpus [Zeroual et al., 2019], Assafir news
articles, and 4 other manually crawled news websites (Al-Akhbar, Annahar, AL-
Ahram, AL-Wafd). There are two version off the model AraBERTv0.1 and
AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text
where prefixes and suffixes were splitted using the Farasa Segmenter [Abdelali
et al., 2016]. We used AraBERTv1 since it performed better from the two models
in most of the downstream NLP tasks [Antoun et al., 2020].

For Danish language, we used Danish-BERT
777https://github.com/botxo/nordic_bert, a Danish pretrained language model
based on BERT architecture. The Danish corpus used to train the Danish-BERT
was compiled by combining multiple sources: All Danish language text from
Common Crawl, The Danish Wikipedia, Custom scraped data from the two biggest
Danish debate forums (dindebat.dk and hestenettet.dk) and Danish
OpenSubtitles.

We used GreekBERT888https://github.com/nlpaueb/greek-bert for Greek Language.
This model too has been trained on combining multiple text sources: Greek
Wikidumps, Greek part of European Parliament Proceedings Parallel Corpus and
Greek language text from OSCAR [Ortiz Suárez et al., 2019], a cleansed version
of Common Crawl.Common Crawl. Unlike the transformer models we used for other
languages GreekBERT is uncased. Therefore, we had to convert all tokens to
lowercase before feeding them to the model.

For Turkish we used BERTurk999https://github.com/stefan-it/turkish-bert which
is also based on the BERT architecture. The current version of the model was
trained on a filtered and sentence segmented version of the Turkish OSCAR
corpus [Ortiz Suárez et al., 2019], a recent Wikipedia dump and various OPUS
corpora [Tiedemann, 2012].

##  5 Results

In this section we present the evaluation results that were obtained during
testing for each language. We also provide a brief look at the final
submission results of the shared task. In all the languages multilingual
DistilBERT does not perform better than multilingual BERT model we used.
Therefore, we have omitted it from the results.

###  5.1 Arabic

For Arabic language, we trained our system on the training set and evaluated
the system on development data that the organisers provided. Table 2 shows the
results we obtained for the development set. As shown Transformer models
outperform traditional word embedding models. Also it should be noted that
AraBERT slightly outperformed multilingual BERT. On the test set our system
achieved 0.788 macro F1 score and ranked 41st out of 53 teams.

| Not Offensive | Offensive | Weighted Average |
---|---|---|---|---
Model | P | R | F1 | P | R | F1 | P | R | F1 | F1 Macro
CNN | 0.80 | 0.82 | 0.81 | 0.71 | 0.66 | 0.69 | 0.82 | 0.82 | 0.78 | 0.70
RNN-BILSTM | 0.76 | 0.78 | 0.77 | 0.67 | 0.62 | 0.65 | 0.78 | 0.78 | 0.74 | 0.66
BERT-multilingual-cased | 0.82 | 0.84 | 0.83 | 0.74 | 0.69 | 0.72 | 0.86 | 0.87 | 0.86 | 0.77
AraBERT | 0.84 | 0.86 | 0.85 | 0.76 | 0.71 | 0.74 | 0.88 | 0.89 | 0.88 | 0.79
Table 2: Results for English Task For each model, Precision (P), Recall (R),
and F1 are reported on all classes, and weighted averages. Macro-F1 is also
listed (best in bold).

###  5.2 Danish

Since the organisers didn’t provide a separate development set for Danish
language, we separated 20% of the training data and treated it as the
development set to evaluate the models. Table 3 shows the results. In Danish
language also transformer models outperformed traditional word embedding
models. However, Danish-BERT model could not outperform the BERT Multilingual
model. We suspect that this is mainly due to the fact that Danish-BERT model
we used was uncased. For the test data our best system scored 0.656 Macro F1
score ranking 32nd out of 39 teams.

| Not Offensive | Offensive | Weighted Average |
---|---|---|---|---
Model | P | R | F1 | P | R | F1 | P | R | F1 | F1 Macro
CNN | 0.91 | 0.92 | 0.93 | 0.71 | 0.67 | 0.67 | 0.85 | 0.85 | 0.85 | 0.72
RNN-BILSTM | 0.90 | 0.91 | 0.92 | 0.70 | 0.66 | 0.66 | 0.84 | 0.84 | 0.84 | 0.71
BERT-multilingual-cased | 0.95 | 0.96 | 0.97 | 0.75 | 0.75 | 0.75 | 0.93 | 0.92 | 0.93 | 0.80
DANISH-BERT | 0.92 | 0.93 | 0.93 | 0.71 | 0.71 | 0.71 | 0.89 | 0.89 | 0.89 | 0.76
Table 3: Results for English Task For each model, Precision (P), Recall (R),
and F1 are reported on all classes, and weighted averages. Macro-F1 is also
listed (best in bold).

###  5.3 English

For English language also we separated an evaluation set which was 20% from
the training set and evaluated our models on that. Since there were many
pretrained transformer models for English, we were able to experiment with
various transformer architectures. Table 4 shows the results for the
evaluation set. XLNET large cased transformer model outperformed all the other
architectures. Our best system scored 0.90056 Macro F1 score for the test set
and ranked 62nd out of 86 teams.

| Not Offensive | Offensive | Weighted Average |
---|---|---|---|---
Model | P | R | F1 | P | R | F1 | P | R | F1 | F1 Macro
CNN | 0.87 | 0.88 | 0.87 | 0.81 | 0.77 | 0.77 | 0.81 | 0.83 | 0.82 | 0.79
RNN-BILSTM | 0.86 | 0.87 | 0.86 | 0.80 | 0.76 | 0.76 | 0.80 | 0.81 | 0.81 | 0.77
BERT-large-cased | 0.91 | 0.92 | 0.91 | 0.85 | 0.81 | 0.81 | 0.85 | 0.85 | 0.86 | 0.82
XLNet-large-cased | 0.92 | 0.93 | 0.92 | 0.84 | 0.83 | 0.83 | 0.87 | 0.87 | 0.88 | 0.85
Table 4: Results for English Task For each model, Precision (P), Recall (R),
and F1 are reported on all classes, and weighted averages. Macro-F1 is also
listed (best in bold).

###  5.4 Greek

For Greek we prepared a separate evaluation set as we did with Danish and
English since the organisers did not provide a development set. In Greek,
GreekBERT performed best outperforming multilingual BERT model slightly. Table
5 shows the results for the evaluation set. For the test set it had 0.814
Macro F1 score and ranked 15th out of 37 teams.

| Not Offensive | Offensive | Weighted Average |
---|---|---|---|---
Model | P | R | F1 | P | R | F1 | P | R | F1 | F1 Macro
CNN | 0.81 | 0.82 | 0.83 | 0.74 | 0.75 | 0.74 | 0.86 | 0.85 | 0.84 | 0.80
RNN-BILSTM | 0.80 | 0.80 | 0.81 | 0.73 | 0.72 | 0.73 | 0.77 | 0.77 | 0.75 | 0.71
BERT-multilingual-cased | 0.87 | 0.87 | 0.88 | 0.79 | 0.79 | 0.79 | 0.83 | 0.83 | 0.82 | 0.77
GREEKBERT | 0.89 | 0.89 | 0.90 | 0.81 | 0.81 | 0.81 | 0.85 | 0.85 | 0.85 | 0.81
Table 5: Results for Greek subtask For each model, Precision (P), Recall (R),
and F1 are reported on all classes, and weighted averages. Macro-F1 is also
listed (best in bold).

###  5.5 Turkish

For Turkish also we prepared a separate evaluation set as we did with Danish
and English. For Turkish, the transformer model with BERTURK performed best
surpassing BERT multilingual as shown in table 6. Our best system had 0.7858
Macro F1 score and ranked 7th out of 46 teams.

| Not Offensive | Offensive | Weighted Average |
---|---|---|---|---
Model | P | R | F1 | P | R | F1 | P | R | F1 | F1 Macro
CNN | 0.80 | 0.81 | 0.79 | 0.70 | 0.59 | 0.69 | 0.75 | 0.76 | 0.75 | 0.67
RNN-BILSTM | 0.79 | 0.80 | 0.78 | 0.69 | 0.58 | 0.68 | 0.74 | 0.75 | 0.74 | 0.66
BERT-multilingual-cased | 0.83 | 0.84 | 0.82 | 0.73 | 0.62 | 0.72 | 0.78 | 0.79 | 0.78 | 0.70
BERTURK | 0.92 | 0.93 | 0.91 | 0.82 | 0.82 | 0.81 | 0.87 | 0.88 | 0.87 | 0.79
Table 6: Results for Turkish subtask For each model, Precision (P), Recall
(R), and F1 are reported on all classes, and weighted averages. Macro-F1 is
also listed (best in bold).

##  6 Conclusion

In this paper, we have presented the team BRUMS system for identifying
offensive language in social media posts in Arabic, Danish,English, Greek and
Turkish. The system uses minimal preprocessing, and relies on word and context
embeddings. We experimented with different deep neural network architectures
in order to determine the most suitable for this task. Our implementation has
been made available on
Github.101010https://github.com/tharindudr/offenseval_2020. According to our
evaluation, and the results provided by the task organisers, it is clear that
fine tuning transformer architectures score highest overall. Also we
discovered that pretrained monolingual transformer models perform better than
the bert-multilingual model. In future, we are hoping to experiment more with
transformer architectures in different languages.

## References

  * [Abdelali et al., 2016]  Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak.  2016\.  Farasa: A fast and furious segmenter for Arabic.  In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16, San Diego, California, June. Association for Computational Linguistics.
  * [Antoun et al., 2020]  Wissam Antoun, Fady Baly, and Hazem Hajj.  2020\.  Arabert: Transformer-based model for arabic language understanding.
  * [Aroyehun and Gelbukh, 2018]  Segun Taofeek Aroyehun and Alexander Gelbukh.  2018\.  Aggression detection in social media: Using deep neural networks, data augmentation, and pseudo labeling.  In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), pages 90–97.
  * [Basile et al., 2019]  Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti.  2019\.  Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter.  In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 54–63.
  * [Bojanowski et al., 2016]  Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov.  2016\.  Enriching word vectors with subword information.  arXiv preprint arXiv:1607.04606.
  * [Chung et al., 2014]  Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio.  2014\.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  In NIPS 2014 Workshop on Deep Learning, December 2014.
  * [Çöltekin, 2020]  Çağrı Çöltekin.  2020\.  A corpus of Turkish offensive language on social media.  In Proceedings of The 12th Language Resources and Evaluation Conference, pages 6174–6184, Marseille, France, May. European Language Resources Association.
  * [Conneau et al., 2019]  Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  2019\.  Unsupervised cross-lingual representation learning at scale.  arXiv preprint arXiv:1911.02116.
  * [Cui et al., 2018]  Jianjing Cui, Jun Long, Erxue Min, Qiang Liu, and Qian Li.  2018\.  Comparative study of cnn and rnn for deep learning based intrusion detection system.  In Xingming Sun, Zhaoqing Pan, and Elisa Bertino, editors, Cloud Computing and Security, pages 159–170, Cham. Springer International Publishing.
  * [Devlin et al., 2018]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  2018\.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805.
  * [El-Khair, 2016]  Ibrahim Abu El-Khair.  2016\.  1.5 billion words arabic corpus.  ArXiv, abs/1611.04033.
  * [Fortuna et al., 2018]  Paula Fortuna, José Ferreira, Luiz Pires, Guilherme Routar, and Sérgio Nunes.  2018\.  Merging datasets for aggressive text identification.  In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), pages 128–139.
  * [Grave et al., 2018]  Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov.  2018\.  Learning word vectors for 157 languages.  In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018).
  * [Graves et al., 2005]  Alex Graves, Santiago Fernández, and Jürgen Schmidhuber.  2005\.  Bidirectional lstm networks for improved phoneme classification and recognition.  In Proceedings of the 15th International Conference on Artificial Neural Networks: Formal Models and Their Applications - Volume Part II, ICANN’05, page 799–804, Berlin, Heidelberg. Springer-Verlag.
  * [Hettiarachchi and Ranasinghe, 2019]  Hansi Hettiarachchi and Tharindu Ranasinghe.  2019\.  Emoji powered capsule network to detect type and target of offensive posts in social media.  In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 474–480, Varna, Bulgaria, September. INCOMA Ltd.
  * [Hochreiter and Schmidhuber, 1997]  Sepp Hochreiter and Jürgen Schmidhuber.  1997\.  Long short-term memory.  Neural Comput., 9(8):1735–1780, November.
  * [Hughes et al., 2017]  M Hughes, I Li, S Kotoulas, and T Suzumura.  2017\.  Medical text classification using convolutional neural networks.  Studies in health technology and informatics, 235:246.
  * [Kim, 2014]  Yoon Kim.  2014\.  Convolutional neural networks for sentence classification.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar, October. Association for Computational Linguistics.
  * [Kumar et al., 2018]  Ritesh Kumar, Atul Kr. Ojha, Shervin Malmasi, and Marcos Zampieri.  2018\.  Benchmarking aggression identification in social media.  In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), pages 1–11, Santa Fe, New Mexico, USA, August. Association for Computational Linguistics.
  * [Liu et al., 2019a]  Ping Liu, Wen Li, and Liang Zou.  2019a.  Nuli at semeval-2019 task 6: transfer learning for offensive language detection using bidirectional transformers.  In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 87–91.
  * [Liu et al., 2019b]  Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  2019b.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692.
  * [Malmasi and Zampieri, 2017]  Shervin Malmasi and Marcos Zampieri.  2017\.  Detecting Hate Speech in Social Media.  In Proceedings of the International Conference Recent Advances in Natural Language Processing (RANLP), pages 467–472.
  * [Mandl et al., 2019]  Thomas Mandl, Sandip Modha, Prasenjit Majumder, Daksh Patel, Mohana Dave, Chintak Mandlia, and Aditya Patel.  2019\.  Overview of the hasoc track at fire 2019: Hate speech and offensive content identification in indo-european languages.  In Proceedings of the 11th Forum for Information Retrieval Evaluation, FIRE ’19, page 14–17, New York, NY, USA. Association for Computing Machinery.
  * [Modha et al., 2018]  Sandip Modha, Prasenjit Majumder, and Thomas Mandl.  2018\.  Filtering aggression from the multilingual social media feed.  In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), pages 199–207.
  * [Montani, 2018]  Joaquın Padilla Montani.  2018\.  Tuwienkbs at germeval 2018: German abusive tweet detection.  In 14th Conference on Natural Language Processing KONVENS, volume 2018, page 45.
  * [Mubarak et al., 2020]  Hamdy Mubarak, Ammar Rashed, Kareem Darwish, Younes Samih, and Ahmed Abdelali.  2020\.  Arabic offensive language on twitter: Analysis and experiments.
  * [Ortiz Suárez et al., 2019]  Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary.  2019\.  Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures.  In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7), Cardiff, United Kingdom, July.
  * [Pitenis et al., 2020]  Zeses Pitenis, Marcos Zampieri, and Tharindu Ranasinghe.  2020\.  Offensive language identification in greek.  In Proceedings of the Twelfth International Conference on Language Resources and Evaluation (LREC 2020), Marseille, France, May. European Language Resources Association (ELRA).
  * [Ranasinghe et al., 2019]  Tharindu Ranasinghe, Marcos Zampieri, and Hansi Hettiarachchi.  2019\.  Brums at hasoc 2019: Deep learning models for multilingual hate speech and offensive language identification.  In Proceedings of the 11th annual meeting of the Forum for Information Retrieval Evaluation (December 2019).
  * [Risch and Krestel, 2018]  Julian Risch and Ralf Krestel.  2018\.  Delete or not Delete? Semi-Automatic Comment Moderation for the Newsroom.  In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), pages 166–176.
  * [Sanh et al., 2019]  Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  2019\.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  arXiv preprint arXiv:1910.01108.
  * [Scherer et al., 2010]  Dominik Scherer, Andreas Müller, and Sven Behnke.  2010\.  Evaluation of pooling operations in convolutional architectures for object recognition.  In Konstantinos Diamantaras, Wlodek Duch, and Lazaros S. Iliadis, editors, Artificial Neural Networks – ICANN 2010, pages 92–101, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * [Sigurbergsson and Derczynski, 2020]  Gudbjartur Ingi Sigurbergsson and Leon Derczynski.  2020\.  Offensive language and hate speech detection for Danish.  In Proceedings of The 12th Language Resources and Evaluation Conference, pages 3498–3508, Marseille, France, May. European Language Resources Association.
  * [Sun et al., 2019]  Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang.  2019\.  How to fine-tune bert for text classification?  In Maosong Sun, Xuanjing Huang, Heng Ji, Zhiyuan Liu, and Yang Liu, editors, Chinese Computational Linguistics, pages 194–206, Cham. Springer International Publishing.
  * [Tiedemann, 2012]  Jörg Tiedemann.  2012\.  Parallel data, tools and interfaces in OPUS.  In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2214–2218, Istanbul, Turkey, May. European Language Resources Association (ELRA).
  * [von Grünigen et al., 2018]  Dirk von Grünigen, Fernando Benites, Pius von Däniken, Mark Cieliebak, Ralf Grubenmann, and AG SpinningBytes.  2018\.  spmmmp at germeval 2018 shared task: Classification of offensive content in tweets using convolutional neural networks and gated recurrent units.  In 14th Conference on Natural Language Processing KONVENS 2018, page 130.
  * [Wiedemann et al., 2018]  Gregor Wiedemann, Eugen Ruppert, Raghav Jindal, and Chris Biemann.  2018\.  Transfer learning from lda to bilstm-cnn for offensive language detection in twitter.  arXiv preprint arXiv:1811.02906.
  * [Wiegand et al., 2018]  Michael Wiegand, Melanie Siegel, and Josef Ruppenhofer.  2018\.  Overview of the GermEval 2018 Shared Task on the Identification of Offensive Language.  In Proceedings of GermEval.
  * [Wolf et al., 2019]  Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew.  2019\.  Huggingface’s transformers: State-of-the-art natural language processing.  ArXiv, abs/1910.03771.
  * [Yang et al., 2019]  Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.  2019\.  Xlnet: Generalized autoregressive pretraining for language understanding.  In Advances in neural information processing systems, pages 5754–5764.
  * [Zampieri et al., 2019a]  Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar.  2019a.  Predicting the type and target of offensive posts in social media.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1415–1420, Minneapolis, Minnesota, June. Association for Computational Linguistics.
  * [Zampieri et al., 2019b]  Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar.  2019b.  SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval).  In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 75–86, Minneapolis, Minnesota, USA, June. Association for Computational Linguistics.
  * [Zampieri et al., 2020]  Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin.  2020\.  SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020).  In Proceedings of SemEval.
  * [Zeroual et al., 2019]  Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja.  2019\.  OSIAN: Open source international Arabic news corpus - preparation and integration into the CLARIN-infrastructure.  In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175–182, Florence, Italy, August. Association for Computational Linguistics.