# A Unified System for Aggression Identification in English Code-Mixed and
Uni-Lingual Texts

Anant Khandelwal<EMAIL_ADDRESS>M.Tech., IIT DelhiBangaloreIndia
and  Niraj Kumar<EMAIL_ADDRESS>Conduent Labs, PhD, IIIT-
HyderabadBangaloreIndia

(2020)

###### Abstract.

Wide usage of social media platforms has increased the risk of aggression,
which results in mental stress and affects the lives of people negatively like
psychological agony, fighting behavior, and disrespect to others. Majority of
such conversations contains code-mixed languages(Kumar et al., 2018d).
Additionally, the way used to express thought or communication style also
changes from one social media platform to another platform (e.g.,
communication styles are different in twitter and Facebook). These all have
increased the complexity of the problem. To solve these problems, we have
introduced a unified and robust multi-modal deep learning architecture which
works for English code-mixed dataset and uni-lingual English dataset both. The
devised system, uses psycho-linguistic features and very basic linguistic
features. Our multi-modal deep learning architecture contains, Deep Pyramid
CNN, Pooled BiLSTM, and Disconnected RNN(with Glove and FastText embedding,
both). Finally, the system takes the decision based on model averaging. We
evaluated our system on English Code-Mixed
TRAC111https://sites.google.com/view/trac1/home 2018 dataset and uni-lingual
English dataset obtained from
Kaggle222https://www.kaggle.com/dataturks/dataset-for-detection-of-
cybertrolls. Experimental results show that our proposed system outperforms
all the previous approaches on English code-mixed dataset and uni-lingual
English dataset.

English code-mixed Text, Deep Pyramid CNN, Disconnected RNN, Pooled BiLSTM,
Model Averaging

††journalyear: 2020††copyright: acmcopyright††conference: 7th ACM IKDD CoDS
and 25th COMAD; January 5–7, 2020; Hyderabad, India††booktitle: 7th ACM IKDD
CoDS and 25th COMAD (CoDS COMAD 2020), January 5–7, 2020, Hyderabad,
India††price: 15.00††doi: 10.1145/3371158.3371165††isbn:
978-1-4503-7738-6/20/01

##  1\. Introduction

The exponential increase of interactions on the various social media platforms
has generated the huge amount of data on social media platforms like Facebook
and Twitter, etc. These interactions resulted not only positive effect but
also negative effect over billions of people owing to the fact that there are
lots of aggressive comments (like hate, anger, and bullying). These cause not
only mental and psychological stress but also account deactivation and even
suicide(Hinduja and Patchin, 2010). In this paper we concentrate on problems
related to aggressiveness.
The fine-grained definition of the aggressiveness/aggression identification is
provided by the organizers of TRAC-2018 (Kumar et al., 2018d, a). They have
classified the aggressiveness into three labels (Overtly aggressive(OAG),
Covertly aggressive(CAG), Non-aggressive(NAG)). The detailed description for
each of the three labels is described as follows:

  1. (1)

Overtly Aggressive(OAG) \- This type of aggression shows direct verbal attack
pointing to the particular individual or group. For example, ”Well said
sonu..you have courage to stand against dadagiri of Muslims”.

  2. (2)

Covertly Aggressive(CAG) \- This type of aggression the attack is not direct
but hidden, subtle and more indirect while being stated politely most of the
times. For example, ”Dear India, stop playing with the emotions of your people
for votes.”

  3. (3)

Non-Aggressive(NAG) \- Generally these type of text lack any kind of
aggression it is basically used to state facts, wishing on occasions and
polite and supportive.

The additional discussion on aggressiveness task can be found in Kaggle task
333https://www.kaggle.com/dataturks/dataset-for-detection-of-cybertrolls,
which just divided the task into two classes - i.e., presence or absence of
aggression in tweets.
The informal setting/environment of social media often encourage multilingual
speakers to switch back and forth between languages when speaking or writing.
These all resulted in “code-mixing” and “code-switching”. Code-mixing refers
to the use of linguistic units from different languages in a single utterance
or sentence, whereas code-switching refers to the co-occurrence of speech
extracts belonging to two different grammatical systems(Gumperz, 1982). This
language interchange makes the grammar more complex and thus it becomes tough
to handle it by traditional algorithms. Thus the presence of high percentage
of code-mixed content in social media text has increased the complexity of the
aggression detection task. For example, the dataset provided by the organizers
of TRAC-2018 (Kumar et al., 2018d, a) is actually a code-mixed dataset.
The massive increase of the social media data rendered the manual methods of
content moderation difficult and costly. Machine Learning and Deep Learning
methods to identify such phenomena have attracted more attention to the
research community in recent years(Kumar et al., 2018b).
Based on the current context, we can divide the problem into three sub-
problems: (a) detection of aggression levels, (b) handling code-mixed data and
(c) handling styles (due to differences in social media platforms and text
entry rules/restrictions).
A lot of the previous approaches(Kumar et al., 2018c) have used an ensemble
model for the task. For example, some of them uses ensemble of statistical
models(Arroyo-Fernández et al., 2018; Fortuna et al., 2018; Samghabadi et al.,
2018; Orasan, 2018) some used ensemble of statistical and deep learning
models(Risch and Krestel, 2018; Tommasel et al., 2018; Ramiandrisoa and Mothe,
2018) some used ensemble of deep learning models (Madisetty and Desarkar,
2018). There are approaches which proposed unified architecture based on deep
learning(Aroyehun and Gelbukh, 2018; Golem et al., 2018; Orabi et al., 2018;
Raiyani et al., 2018; Nikhil et al., 2018; Galery et al., 2018) while some
proposed unified statistical model(Fortuna et al., 2018). Additionally, there
are some approaches uses data augmentation either through translation or
labeling external data to make the model generalize across domains(Aroyehun
and Gelbukh, 2018; Risch and Krestel, 2018; Fortuna et al., 2018).
Most of the above-discussed systems either shows high performance on (a)
Twitter dataset or (b) Facebook dataset (given in the TRAC-2018), but not on
both English code-mixed datasets. This may be due to the text style or level
of complexities of both datasets. So, we concentrated to develop a robust
system for English code-mixed texts, and uni-lingual texts, which can also
handle different writing styles. Our approach is based on three main ideas:

  * •

Deep-Text Learning. The goal is to learn long range associations, dependencies
between regions of text, N-grams, key-patterns, topical information, and
sequential dependencies.

  * •

Exploiting psycho-linguistic features with basic linguistic features as meta-
data. The main aim is to minimize the direct dependencies on in-depth
grammatical structure of the language (i.e., to support code-mixed data). We
have also included emoticons, and punctuation features with it. We use the
term ”NLP Features” to represent it in the entire paper.

  * •

Dual embedding based on FastText and Glove. This dual embedding helps in high
vocabulary coverage and to capture the rare and partially incorrect words in
the text (specially by FastText (Mikolov et al., 2018)).

Our ”Deep-text architecture” uses model averaging strategy with three
different deep learning architectures. Model averaging belongs to the family
of ensemble learning techniques that uses multiple models for the same problem
and combines their predictions to produce a more reliable and consistent
prediction accuracy (Ju et al., 2018). This is the simplest form of weighted
average ensemble based prediction(Pawlikowski and Chorowska, 2019) where, each
ensemble member contribute equally to predictions. Specifically in our case,
three different models have been used. The following contains the intuition
behind the selection of these three models:

  1. (1)

Deep Pyramid CNN (Johnson and Zhang, 2017) being deeper helps to learn long
range associations between temporal regions of text using two-view embeddings.

  2. (2)

Disconnected RNN (Wang, 2018) is very helpful in encoding the sequential
information with temporal key patterns in the text.

  3. (3)

Pooled BiLSTM In this architecture the last hidden state of BiLSTM is
concatenated with mean and max-pooled representation of the hidden states
obtained over all the time steps of Bi-LSTM. The idea of using mean and max
pooling layers together is taken from (Howard and Ruder, 2018) to avoid the
loss of information in longer sequences of texts and max-pooling is taken to
capture the topical information(Shen et al., 2014).

  4. (4)

NLP Features In each of the individual models, the NLP features are
concatenated with last hidden state before the softmax classification layer as
meta-data. The main aim is to provide additional information to the deep
learning network.

The intuition behind the NLP features are the following:

  * •

Emotion Sensor Dataset We have introduced to use of emotion sensor features,
as a meta-data information. We have obtained the word sensor dataset from
Kaggle444https://www.kaggle.com/iwilldoit/emotions-sensor-data-set. In this
dataset each word is statistically classified into 7 distinct classes
(Disgust, Surprise, Neutral, Anger, Sad, Happy and Fear) using Naive Bayes,
based on sentences collected from twitter and blogs.

  * •

Controlled Topical Signals from Empath555http://empath.stanford.edu/. Empath
can analyse the text across 200 gold standard topics and emotions.
Additionally, it uses neural embedding to draw connotation among words across
more than 1.8 billion words. We have used only selected categories like
violence, hate, anger, aggression, social media and dispute from 200 Empath
categories useful for us unlike(Ramiandrisoa and Mothe, 2018) which takes 194
categories.

  * •

Emoticons frequently used on social media indicates the sense of
sentence(Raiyani et al., 2018; Galery et al., 2018; Orasan, 2018).

  * •

Normalized frequency of POS tags According to (Ramiandrisoa and Mothe, 2018;
Tommasel et al., 2018; Fortuna et al., 2018; Golem et al., 2018) POS Tags
provide the degree of target aggressiveness. Like(Ramiandrisoa and Mothe,
2018), we have used only four tags (a) adjective (JJ, JJR, JJS), (b) adverb
(RB, RBR, RBS), (c) verb (VB, VBD, VBG, VBN, VBP, VBZ) and (d) noun (NN, NNS,
NNP, NNPS) (See “Penn-Treebank POS
Tags”666https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html
for abbreviations and the full list). The main reason behind the selection of
these four tags is to just identify words related to persons, activities,
quality, etc, in the text.

  * •

Sentiment polarity obtained from VADER Sentiment Analysis (Hutto and Gilbert,
2014) (positive, negative and neutral) like used in (Golem et al., 2018; Risch
and Krestel, 2018; Tommasel et al., 2018; Fortuna et al., 2018). It helps to
demarcate aggressiveness with non-aggressiveness in the text.

Figure 1. Block diagram of the proposed system

The block diagram of the proposed system is shown in Figure 1. The proposed
system does not use any data augmentation techniques like (Aroyehun and
Gelbukh, 2018), which is the top performer in TRAC (in English code-mixed
Facebook data). This means the performance achieved by our system totally
depends on the training dataset provided by TRAC. This also proves the
effectiveness of our approach. Our system outperforms all the previous state
of the art approaches used for aggression identification on English code-mixed
TRAC data, while being trained only from Facebook comments the system
outperforms other approaches on the additional Twitter test set. The remaining
part of this paper is organized as follows: Section 2 is an overview of
related work. Section 3 presents the methodology and algorithmic details.
Section 4 discusses the experimental evaluation of the system, and Section 5
concludes this paper.

##  2\. Related work

There are several works for aggression identification submitted at TRAC 2018
among them some approaches use the ensemble of multiple statistical
models(Arroyo-Fernández et al., 2018; Fortuna et al., 2018; Samghabadi et al.,
2018; Orasan, 2018). Similarly, some of the models like(Risch and Krestel,
2018; Tommasel et al., 2018; Ramiandrisoa and Mothe, 2018) have used ensemble
of statistical and deep learning models. In these models the statistical part
of the model uses additional features from text analysis like parts-of-speech
tags, punctuation, emotion, emoticon etc. Model like: (Madisetty and Desarkar,
2018) has used the ensemble of deep learning models based on majority voting.
Some other models like: (Majumder et al., 2018; Ramiandrisoa and Mothe, 2018;
Orasan, 2018) have used different models for Facebook and twitter. While
approaches like:(Aroyehun and Gelbukh, 2018; Golem et al., 2018; Orabi et al.,
2018; Raiyani et al., 2018; Nikhil et al., 2018; Galery et al., 2018) have
proposed unified architecture based on deep learning. Systems like(Aroyehun
and Gelbukh, 2018; Risch and Krestel, 2018; Fortuna et al., 2018) have used
data augmentation either through translation or labelling external data to
make the model generalize across domains. While (Fortuna et al., 2018) has
proposed a unified statistical model.
Among approaches like(Arroyo-Fernández et al., 2018) extracted features from
TF-IDF of character n-grams while(Majumder et al., 2018) uses LSTM with pre-
trained embeddings from FastText. (Golem et al., 2018) have used the BiLSTM
based model and the SVM metaclassifier model for the Facebook and Twitter test
sets, respectively. While (Madisetty and Desarkar, 2018) tried ensembling of
CNN, LSTM, and BILSTM.
Some approaches like:(Ramiandrisoa and Mothe, 2018) has used emotions
frequency as one of the features, while some others use sentiment emotion as
feature(Tommasel et al., 2018). Also,(Raiyani et al., 2018; Galery et al.,
2018) have converted emoticons to their description. (Orasan, 2018) have used
TF-IDF of emoticons per-class as one of the features. Compared to all these
approaches, we have concentrated to capture multiple linguistic/pattern based
relations, key-terms and key-patters (with their association in text) through
a combination of deep learning architectures with model averaging. We have
also used NLP features as additional features with our deep learning
architecture, obtained from psycho-linguistic and basic linguistic features.

##  3\. Methodology

In this section, we describe our system architecture for aggressiveness
classifier. In section 3.1 we describe data preprocessing applied on the input
text before feeding it to each of the classification models. Section 3.2
describes the computation of NLP features. In Sections 3.3, 3.4 and 3.5 we
have described the architecture of different deep learning models like Deep
Pyramid CNN, Disconnected RNN and Pooled BiLSTM respectively. Finally, in
Section 3.6, we describe model averaging based classification model which
combines the prediction probabilities from three deep learninig architectures
discussed above. (see Figure 1. for block diagram of system architecture).

###  3.1. Data Preprocessing

We consider the text to be well formatted before applying the text to the
embedding layer. First, we detect non-English text(which are few) and
translate all of them to English using Google
Translate777https://pypi.org/project/googletrans/. Still, there is some code
mixed words like ”mc”, ”bc” and other English abbreviations and spelling
errors like ”nd” in place of ”and”, ”u” in place of ”you” causes deep learning
model to confuse with sentences of the same meaning. We follow the strategy of
preprocessor as in(Raiyani et al., 2018) to normalize the abbreviations and
remove spelling errors, URLs and punctuation marks, converting emojis to their
description.

Feature Name |  Description | Feature Count
---|---|---
| Emotion Sensor Feature
---
Emotion Sensor Features are developed by classifying the words statistically using Naive Bayes algorithm into 7 different categories (Disgust, Surprise, Neutral, Anger, Sad, Happy and Fear) using sentences collected from twitter or blogs. | 7
Parts-of-Speech(POS) |  Like(Ramiandrisoa and Mothe, 2018) we have used the normalized frequencies of noun, adjective, verb, and adverb which serve as a rich feature for exaggerating type of aggressiveness in the text. We used the Spacy888https://spacy.io/usage/linguistic-features#pos-tagging POS tagger. | 4
Punctuation |  Like in “Communist parties killed lacks of opponents in WB in 35 years ruling????? ?” the presence of multiple question marks linked with the aggressiveness content in the text. Similarly ”!” also has the same effect. We have used the count of “!” & “?” in the text as a feature. | 1
Sentiment analysis |  The percentage of positive, negative, and neutral can indicate the amount of aggressiveness in the text. We have used VADER Sentiment Analysis(Hutto and Gilbert, 2014) to extract the weight for positive, negative & neutral class. | 3
Topic Signals from text |  We have used open source library Empath[2] introduced in (Fast et al., 2016) which has categories highly correlated to LIWC(Pennebaker et al., 2007). Although it has a rich number of categories we particularly identified selected categories useful for our case these are aggressiveness, violence, hate, anger, dispute & social_media. We have used the normalized weight of each of these categories as separate features. | 6
| TF-IDF Emoticon Feature
---
Since emoticons are widely used to convey the meaning on social media platform analyzing the data emoticons are an important feature for classifying aggressive behavior and tf-idf feature for each class is calculated as in (Orasan, 2018) | 3
| Total Features
---
| 24
Table 1. Details of NLP features

###  3.2. NLP Features

We have identified a novel combination of features which are highly effective
in aggression classification when applied in addition to the features obtained
from the deep learning classifier at the classification layer. We have
introduced two new features in addition to the previously available features.
The first one is the Emotion Sensor
Feature999https://www.kaggle.com/iwilldoit/emotions-sensor-data-set which use
a statistical model to classify the words into 7 different classes based on
the sentences obtained from twitter and blogs which contain total 1,185,540
words. The second one is the collection of selected topical signal from text
collected using Empath101010http://empath.stanford.edu/ (see Table 1.).
Different from previous approaches(Samghabadi et al., 2018; Ramiandrisoa and
Mothe, 2018) where (Ramiandrisoa and Mothe, 2018) have used Emotion features
in the form of frequency while (Samghabadi et al., 2018) have used emotion
feature vector obtained from LIWC 2007(Pennebaker et al., 2007).
Unlike(Ramiandrisoa and Mothe, 2018) we have used only 6 topical signals from
Emapth(Fast et al., 2016). We have borrowed the idea of using other features
like punctuation features and parts-of-speech tags from (Ramiandrisoa and
Mothe, 2018). The Table 1. lists and describes features, tools used to obtain
them and the number of features resulted from each type.

Figure 2. DPCNN

###  3.3. Deep Pyramid CNN(DPCNN)

Since it has been proved that CNNs are great feature extractors for text
classification(Yin et al., 2017; Kim, 2014; Johnson and Zhang, 2014, 2015,
2016, 2017) while deeper networks(whether RNNs or CNN’s) has been proven for
learning long-range association like deeper character level CNN’s(Zhang et
al., 2015; Conneau et al., 2016), and complex combination of RNN and CNN(Peng
et al., 2019; Shen et al., 2019; Jiao et al., 2019; Yu et al., 2018; Yang et
al., 2016). Deep Pyramid CNN (DPCNN)(Johnson and Zhang, 2017) has 15 layers of
word-level CNN’s and contains similar pre-activation as proposed in improved
Resnet(He et al., 2016b). DPCNN outperforms the 32-layer character CNN(Conneau
et al., 2016) and Hierarchical attention networks(Yang et al., 2016) it has
added advantage that due to its pyramid structure it does not require
dimension matching in shortcut connections defined as z + h(z) as in(He et
al., 2016b) where h(z) represents the skipped layers essentially contains two
convolutional layers with pre-activation. It uses enhanced region embedding
which consumes pre-trained embeddings (in our case it is FastText+Glove based
dual embedding).
Enhanced Region Embedding. The current DPCNN(Johnson and Zhang, 2017), uses
two view type enhanced region embedding. For the text categorization, it
defines a region of text as view-1 and its adjacent regions as view-2. Then
using unlabeled data, it trains a neural network of one hidden layer with an
artificial task of predicting view-2 from view-1. The obtained hidden layer,
which is an embedding function that takes view-1 as input, serves as an
unsupervised embedding function in the model for text categorization. The
detailed architecture has been shown in Figure 2.
Let each word input $x_{j}\in R^{d}$ be the d-dimensional vector for the
$j^{th}$ word $w_{j}$ and the sentence $s_{i}$ contains sequence of $n$ words
$\\{w_{1},w_{2},w_{3},......,w_{n}\\}$ as shown in Figure 2. In comparision to
conventional convolution layer, DPCNN proposes to use pre-activation, thus
essentially the convolutional layer of DPCNN is
$\textbf{W}\sigma(\textbf{x})+\textbf{b}$, where W and b(unique to each layer)
are the weights matrix and bias respectively, we use $\sigma$ as PReLU(He et
al., 2015). During implementation we use kernel size of 3(represented by x to
denote the small overlapping regions of text.), The number of filters(number
of feature maps denoted by the number of rows of W) is 128 as depicted in
Figure 2. With the number of filters same in each convolution layer and max-
pooling with stride 2 makes the computation time halved, and doubles the net
coverage of convolution kernel. Thus the deeper layers cause to learn long-
range associations between regions of text. Let’s say $h_{dpcnn}\in R^{p_{1}}$
be the hidden state obtained from DPCNN just before the classification layer
and $f_{nlp}\in R^{24}$ be the NLP features computed from the text. Lets
$z_{1}\in R^{p_{1}+24}$ be another hidden state obtained as

(1) $z_{1}=h_{dpcnn}\oplus f_{nlp}$

where, $\oplus$ denotes concatenation. The vector $z_{1}$ obtained, then fed
to the fully connected layer with softmax activation. Let $y_{i1}^{*}$ be the
softmax probabilities, specifically for class label $k$ is given as:

(2)
$y_{i1,k}^{*}=p(y_{i}=k|s_{i})=softmax(W_{dpcnn}^{T}z_{1}+b_{dpcnn})[k]\textrm{
}\forall k\in[1...K]$

where $K$ is the number of classes, $W_{dpcnn}$ and $b_{dpcnn}$ are the weight
matrix and bias respectively.

Figure 3. DRNN

###  3.4. Disconnected RNN(DRNN)

Given a sequence $s_{i}=[x_{1},x_{2},x_{3},....x_{n}]$ where $x_{j}\in R^{d}$
represents the d-dimensional word vector for word $w_{j}$ and $n$ is the
length of input text applied to a variant of RNN called Long Short-Term Memory
(LSTM)(Hochreiter and Schmidhuber, 1997) as shown in Figure 3. It is widely
used for sequential modelling with long-term dependencies. For sequence
modelling it keeps on updating the memory cell with current input using an
adaptive gating mechanism. At time step $t$ the memory $c_{t}$ and the hidden
state $h_{t}$ are updated as follows:

(3) $\begin{bmatrix}i_{t}\\\ f_{t}\\\ o_{t}\\\
\hat{c}_{t}\end{bmatrix}=\begin{bmatrix}\sigma\\\ \sigma\\\ \sigma\\\
tanh\end{bmatrix}=W.[h_{t-1},x_{t}]$ (4) $c_{t}=f_{t}\odot
c_{t-1}+i_{t}\odot\hat{c}_{t}$

where $\hat{c}_{t}$ is the current cell state obtained from current input
$x_{t}$ and previous hidden state $h_{t-1}$, $i_{t}$, $f_{t}$ and $o_{t}$ are
the activation corresponding to input gate, forget gate and output gate
respectively, $\sigma$ denotes the logistic sigmoid function and $\odot$
denotes the element-wise multiplication. Hence the hidden state representation
at time step $t$ depends on all the previous input vectors given as

(5) $h_{t}=LSTM(x_{t},....,x_{2},x_{1})\textrm{ }\forall t\in[1...n]$

Specifically we have used Bi-directional LSTM (Hochreiter and Schmidhuber,
1997) to capture both past and future context. It provides $h_{t}$ from both
directions(forward & backward). The forward LSTM takes the natural order of
words from $x_{1}$ to $x_{n}$ to obtain $\overrightarrow{h_{t}}$, while
backward-LSTM $x_{n}$ to $x_{1}$ to obtain $\overleftarrow{h_{t}}$. then
$h_{t}$ is calculated as

(6) $h_{t}=\overrightarrow{h_{t}}\oplus\overleftarrow{h_{t}}\in R^{2L}$

where $\oplus$ is the concatenation and $L$ is the size for one-directional
LSTM. Therefore we denote the hidden state in equation 5 with BiLSTM as

(7) $h_{t}=BiLSTM(x_{t},,......x_{2},x_{1})\in R^{2L}\textrm{ }\forall
t\in[1..n]$

To avoid handling of long sequence and to capture local information for each
word we define the window size $k$ for each word such that the BiLSTM only
sees the the previous $k-1$ words with the current word, where $k$ is a
hyperparameter(Wang, 2018). We use padding ¡PAD¿ to make the slices of fixed
size k(as shown in Figure 3). It provides each hidden state $h_{t}$ with
sequence of $k$ previous words. Since the phrase of $k$ words can lie anywhere
in the text it helps to model the position invariant phrase representation due
to which the it identifies key phrases important for identifying particular
category. In this case, the equation of $h_{t}$ is given as

(8) $h_{t}=BiLSTM(x_{t},x_{t-1},x_{t-2},......,x_{t-k+1})$

The output hidden vectors, $H=[h_{1},h_{2},h_{3},......h_{n}]\in R^{n\times
2L}$ are converted to fixed-length vector $h_{drnn}\in R^{2L}$ with max
pooling over time:

(9) $h_{drnn}^{l}=\underset{t}{max}\textrm{ }h_{t}^{l},t\in[1,...n],\forall
l\in[1,.....2L]$

Let’s say $f_{nlp}\in R^{24}$ be the NLP features computed from the text.
Let’s $z_{2}\in R^{2L+24}$ be another hidden state obtained as

(10) $z_{2}=h_{drnn}\oplus f_{nlp}$

where $\oplus$ denotes concatenation. The vector $z_{2}$ obtained, then fed to
the fully connected layer with softmax activation. Let $y_{i2}^{*}$ be the
softmax probabilities, specifically for class label $k$ is given as:

(11)
$y_{i2,k}^{*}=p(y_{i}=k|s_{i})=softmax(W_{drnn}^{T}z_{2}+b_{drnn})[k]\textrm{
}\forall k\in[1...K]$

where $K$ is the number of classes, $W_{drnn}$ is the weight matrix, and
$b_{drnn}$ is the bias.

Figure 4. Pooled BiLSTM

###  3.5. Pooled BiLSTM

The architecture has been shown in Figure 4. Given a sequence
$s_{i}=[x_{1},x_{2},x_{3},.....x_{j}]$, where $x_{j}\in R^{d}$ is the
d-dimensional word vector for word $w_{j}$, the hidden state obtained after
BiLSTM is given as

(12) $h_{t}=BiLSTM(x_{t},x_{t-1},....x_{1})\forall t\in[1...n]$

To avoid the loss of information because of modelling the entire sequence, we
have concatenated the max-pooled($c_{max}$) and mean-pooled($c_{mean}$)
representation of hidden states calculated over all time steps (Howard and
Ruder, 2018). We have also concatenated the nlp features, $f_{nlp}\in R^{24}$
the final feature vector $z_{3}$ is given as

(13) $z_{3}=h_{n}\oplus c_{max}\oplus c_{mean}\oplus f_{nlp}$

where $\oplus$ denotes concatenation. The final feature $z_{3}$ vector is fed
to the fully connected layer with softmax activation. Let $y_{i3}^{*}$ be the
softmax probablities, specifically for class label $k$ given as:

(14)
$y_{i3,k}^{*}=p(y_{i}=k|s_{i})=softmax(W_{bilstm}^{T}z_{3}+b_{bilstm})[k]\textrm{
}\forall k\in[1...K]$

where $K$ is the number of classes and $W_{bilstm}$ and $b_{bilstm}$ are the
weight matrix and bias respectively.

###  3.6. Classification Model

According to deep learning literature (Simonyan and Zisserman, 2014; Szegedy
et al., 2015; He et al., 2016a), unweighted averaging might be a reasonable
ensemble for similar base learners of comparable performance. Now, similar to
the information discussed in (Ju et al., 2018), we can compute the model
averaging (unweighted) by combining the softmax probabilities of three
different classification models obtained from equations 2, 11, 14. The
averaged class probabilities are computed as:

(15) $y_{i,k}^{*}=\frac{y_{i1,k}^{*}+y_{i2,k}^{*}+y_{i3,k}^{*}}{3}\forall
k\in[1....K]$ (16) $\hat{y}_{i}=\operatorname*{arg\,max}_{k}(y_{i}^{*})$

where K is the number of classes, and $\hat{y_{i}}$ is the predicted label for
sentence $s_{i}$.

##  4\. Experiment and Evaluation

###  4.1. Dataset Description

We have used two datasets in our experimental evaluations: (1) TRAC 2018
Dataset111111https://sites.google.com/view/trac1/shared-task and (2) Kaggle
Dataset121212https://www.kaggle.com/dataturks/dataset-for-detection-of-
cybertrolls.

TRAC 2018 Dataset: We have used the English code-mixed dataset provided by
TRAC 2018. This dataset contains three labels, (a) Non-Aggressive(NAG), (b)
Overtly-Aggressive (OAG) and (c) Covertly-Aggressive(CAG). The distribution of
training, validation and test sets are described in Table 3.

Kaggle Dataset: This dataset contains 20001 tweets which are manually labeled.
The labels are divided into two categories (indicating presence or absence of
aggression in tweets) AGG(Aggressive) or NAG(Non-Aggressive). We have used the
same test split available in the baseline
code131313https://www.kaggle.com/alisaeidi92/a-very-simple-nlp-
model-0-75-accuracy. The distribution for each of the training and test is
given in Table 3.

Number | Training | Validation | Test
---|---|---|---
Facebook | Twitter
Texts | 11999 | 3001 | 916 | 1257
Overtly-aggressive | 2708 | 711 | 144 | 361
Covertly aggressive | 4240 | 1057 | 142 | 413
Non-aggressive | 5051 | 1233 | 630 | 483

Table 2. TRAC 2018, Details of English Code-Mixed Dataset

Number | Training | Test
---|---|---
Texts |  15000 |  5001
Aggressive(AGG) |  5867 |  1955
Non-Aggressive(NAG) |  9133 |  3046
Table 3. Kaggle, Uni-lingual(English) Dataset

| Rank
---
in
Trac
2018
System |  | F1 (Weighted)
---
Facebook
1 | saroyehun(Aroyehun and Gelbukh, 2018) | 0.6425
2 | EBSI-LIA-UNAM(Arroyo-Fernández et al., 2018) | 0.6315
3 | DA-LD-Hildesheim(Majumder et al., 2018) | 0.6178
4 | TakeLab(Golem et al., 2018) | 0.6161
5 | sreeIN(Madisetty and Desarkar, 2018) | 0.6037
Our Models
A | DPCNN | 0.6147
B | DRNN | 0.6228
C | Pooled BiLSTM | 0.6190
D | DPCNN + NLP Features | 0.6530
E | DRNN + NLP Features | 0.6201
F | Pooled BiLSTM + NLP Features | 0.6671
| Model Averaging(A + B + C) | 0.6298
|  | Our proposed method
---
Model Averaging(D + E + F)
0.6770

Table 4. Facebook Test results(TRAC 2018 Dataset)

| Rank
---
in
Trac
2018
System |  | F1 (Weighted)
---
Twitter
1 | vista.ue(Raiyani et al., 2018) | 0.6008
2 | Julian(Risch and Krestel, 2018) | 0.5994
3 | saroyehun(Aroyehun and Gelbukh, 2018) | 0.5920
4 | EBSI-LIA-UNAM(Arroyo-Fernández et al., 2018) | 0.5716
5 | uOttawa(Orabi et al., 2018) | 0.5690
Our Models
A | DPCNN | 0.5364
B | DRNN | 0.5272
C | Pooled BiLSTM | 0.5513
D | DPCNN + NLP Features | 0.5529
E | DRNN + NLP Features | 0.6189
F | Pooled BiLSTM + NLP Features | 0.6227
| Model Averaging(A + B + C) | 0.5967
|  | Our proposed method
---
Model Averaging(D + E + F)
0.6480

Table 5. Twitter Test results(TRAC 2018 Dataset)

S.No. | System | ACCURACY |  | PRECISION
---
(Weighted)
| RECALL
---
(Weighted)
| F1
---
(Weighted)
| Baseline | 0.7546 | 0.7660 | 0.7546 | 0.7380
A | DPCNN | 0.8478 | 0.8720 | 0.8478 | 0.8497
B | DRNN | 0.8620 | 0.8666 | 0.8620 | 0.8630
C | Pooled BiLSTM | 0.8646 | 0.8726 | 0.8646 | 0.8659
D | DPCNN + NLP Features | 0.7798 | 0.8372 | 0.7798 | 0.7812
E | DRNN + NLP Features | 0.8820 | 0.8885 | 0.8820 | 0.8830
F | Pooled BiLSTM + NLP Features | 0.8728 | 0.8855 | 0.8728 | 0.8742
| Model Averaging(A + B + C)) | 0.8666 | 0.8712 | 0.8666 | 0.8676
| Our proposed Mehtod(Model Averaging(D + E + F)) | 0.9016 | 0.9052 | 0.9016 | 0.9023

Table 6. Results on Kaggle Test Dataset

###  4.2. Experimental Setup

We have used Glove Embeddings(Pennington et al., 2014) concatenated with
FastText Embeddings(Mikolov et al., 2018) in all the three classification
models presented in this paper. Specifically, we used Glove pre-trained
vectors obtained from Twitter corpus containing 27 billion tokens and 1.2
million vocabulary entries where each word is represented using
100-dimensional vector. In the case of FastText the word is represented using
300-dimensional vector. Also, we have applied spatial dropout(Tompson et al.,
2015) of 0.3 at embedding layer for DPCNN(in section 3.3) and Pooled BiLSTM(in
section 3.5). For DPCNN model(in 3.3) we have learnt 128-dimensional vector
representation for unsupervised embeddings implicitly for task specific
representation as in (Johnson and Zhang, 2017). Additionally, for DPCNN all
the convolutional layers used 128 filters, kernel size of 3 and max-pooling
stride 2. Additionally, in the case of DPCNN we have used kernel and bias
regularizer of value 0.00001 for all convolutional kernels. The pre-activation
function used in DPCNN is Parametric ReLU (PReLU) proposed in (He et al.,
2015) while the activation at each of the convolutional kernel is linear. For,
DRNN(in section 3.4) we have used the window size of 8 and rest of the
parameters related to LSTM units are same as given in(Wang, 2018). For, Pooled
BiLSTM(in section 3.5) we have used LSTM hidden units size as 256. The maximum
sequence length is 200 in all three models. In each of the classification
model the classification layer contains the fully connected layer with softmax
activation with output size of 3 equal to number of classes in case of TRAC
2018 dataset and its 2 in case of Kaggle dataset. Training has been done using
ADAM optimizer(Kingma and Ba, 2014) for DPCNN and RMSPROP(Tieleman and Hinton,
2012) for DRNN and Pooled Bi-LSTM models. All the models are trained end-to-
end using softmax cross entropy loss(Chollet et al., 2018) for TRAC 2018
dataset and binary cross entropy loss(Chollet et al., 2018) for Kaggle
dataset.
To train our model for TRAC 2018 dataset, we merged the training and
validation dataset and then used 10% split from shuffled dataset to save the
best model, for all classifiers. We have used only 20 NLP features (except TF-
IDF Emoticon feature and Punctuation feature as given in Table 1) for Kaggle
dataset (as these are not present in the Kaggle dataset).

(a) Confusion Matrix for Facebook Test Set(TRAC 2018)

(b) Confusion Matrix for Twitter Test Set(TRAC 2018)

(c) Confusion Matrix for Kaggle Test Set

Figure 5. Confusion Matrix for Facebook, Twitter and Kaggle Datasets.

###  4.3. Evaluation Strategy

To compare our experimental results we have used top-5 systems from the
published results of TRAC-2018(Kumar et al., 2018c). To compare our results on
Kaggle dataset, we have used the last & the best published result on Kaggle
website as a baseline. We have conducted the separate experiments, to properly
investigate the performance of (a) each of the classifiers (used in our model
averaging based system), (b) impact of the NLP features on each of these
classifiers and finally, (c) the performance of our proposed system. In Table
6, 6 and 6, models, named as DPCNN(ref 3.3), DRNN (ref 3.4) and Pooled
BiLSTM(ref 3.5) are corresponding models without NLP features. Similarly,
DPCNN+NLP Features, DRNN + NLP Features and Pooled BiLSTM + NLP Features are
corresponding models with NLP features. The Model Averaging (A+B+C) is the
ensemble of three models (i.e., model averaging of DPCNN, DRNN and Pooled
BiLSTM) without NLP features. Finally, Our Proposed Method, which represents
the model averaging of three models with NLP features.

###  4.4. Results and Discussion

In this paper, we have evaluated our model using weighted macro-averaged
F-score. The measure is defined as in (See (Kumar et al., 2018c, a)). It
weights the F-score computed per class based on the class composition in the
test set and then takes the average of these per-class F-score gives the final
F-score. Table 6, 6 and 6. presents the comparative experimental results for
the proposed method in this paper with respect to the state-of-the-art. The
top 5 models(Kumar et al., 2018c) given in Table 6 and 6. are the best
performing models for Facebook and Twitter test dataset respectively on TRAC
2018. We have followed all the experimental guidelines as discussed in TRAC
contest guideline paper(Kumar et al., 2018a, c). From the results given in
Table 6, 6 and 6 it is clear that our proposed model shows the best
performance among all of the approaches. These results also state that all the
deep learning architectures with NLP features, perform better than individual
corresponding deep learning architectures. This means NLP features, adds some
value to the architectures, even if it is not very high.

##  5\. Conclusion and Future Work

In this paper, we have briefly described the approach we have taken to solve
the aggressive identification on online social media texts which is very
challenging since the dataset is noisy and code-mixed. We presented an
ensemble of deep learning models which outperform previous approaches by
sufficient margin while having the ability to generalize across domains.
In future, we will explore other methods to increase the understanding of deep
learning models on group targeted text, although the categories are well
defined we will look after if we further fine-tune the categories with more
data. In the future, we are planning to pay attention on a generalized
language model for code-mixed texts which can also handle Hindi-code-mixed and
other multi-lingual code-mixed datasets (i.e., trying to reduce the
dependencies on language-specific code-mixed resources).

## References

  * (1)
  * Aroyehun and Gelbukh (2018) Segun Taofeek Aroyehun and Alexander Gelbukh. 2018.  Aggression detection in social media: Using deep neural networks, data augmentation, and pseudo labeling. In _Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)_. 90–97.
  * Arroyo-Fernández et al. (2018) Ignacio Arroyo-Fernández, Dominic Forest, Juan-Manuel Torres-Moreno, Mauricio Carrasco-Ruiz, Thomas Legeleux, and Karen Joannette. 2018\.  Cyberbullying Detection Task: the EBSI-LIA-UNAM System (ELU) at COLING‘18 TRAC-1. In _Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)_. 140–149.
  * Chollet et al. (2018) François Chollet et al. 2018\.  Keras: The python deep learning library.  _Astrophysics Source Code Library_ (2018).
  * Conneau et al. (2016) Alexis Conneau, Holger Schwenk, Loıc Barrault, and Yann Lecun. 2016.  Very deep convolutional networks for natural language processing.  _arXiv preprint arXiv:1606.01781_ 2 (2016).
  * Fast et al. (2016) Ethan Fast, Binbin Chen, and Michael S Bernstein. 2016\.  Empath: Understanding topic signals in large-scale text. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_. ACM, 4647–4657.
  * Fortuna et al. (2018) Paula Fortuna, José Ferreira, Luiz Pires, Guilherme Routar, and Sérgio Nunes. 2018.  Merging datasets for aggressive text identification. In _Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)_. 128–139.
  * Galery et al. (2018) Thiago Galery, Efstathios Charitos, and Ye Tian. 2018.  Aggression identification and multi lingual word embeddings. In _Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)_. 74–79.
  * Golem et al. (2018) Viktor Golem, Mladen Karan, and Jan Šnajder. 2018\.  Combining shallow and deep learning for aggressive text detection. In _Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)_. 188–198.
  * Gumperz (1982) John J Gumperz. 1982\.  _Discourse strategies_. Vol. 1.  Cambridge University Press.
  * He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015\.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In _Proceedings of the IEEE international conference on computer vision_. 1026–1034.
  * He et al. (2016a) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016a.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * He et al. (2016b) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016b.  Identity mappings in deep residual networks. In _European conference on computer vision_. Springer, 630–645.
  * Hinduja and Patchin (2010) Sameer Hinduja and Justin W Patchin. 2010.  Bullying, cyberbullying, and suicide.  _Archives of suicide research_ 14, 3 (2010), 206–221.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal Language Model Fine-tuning for Text Classification. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_. 328–339.  https://doi.org/10.18653/v1/P18-1031
  * Hutto and Gilbert (2014) Clayton J Hutto and Eric Gilbert. 2014.  Vader: A parsimonious rule-based model for sentiment analysis of social media text. In _Eighth international AAAI conference on weblogs and social media_.
  * Jiao et al. (2019) Wenxiang Jiao, Haiqin Yang, Irwin King, and Michael R Lyu. 2019\.  HiGRU: Hierarchical Gated Recurrent Units for Utterance-level Emotion Recognition.  _arXiv preprint arXiv:1904.04446_ (2019).
  * Johnson and Zhang (2014) Rie Johnson and Tong Zhang. 2014.  Effective use of word order for text categorization with convolutional neural networks.  _arXiv preprint arXiv:1412.1058_ (2014).
  * Johnson and Zhang (2015) Rie Johnson and Tong Zhang. 2015.  Semi-supervised convolutional neural networks for text categorization via region embedding. In _Advances in neural information processing systems_. 919–927.
  * Johnson and Zhang (2016) Rie Johnson and Tong Zhang. 2016.  Convolutional neural networks for text categorization: Shallow word-level vs. deep character-level.  _arXiv preprint arXiv:1609.00718_ (2016).
  * Johnson and Zhang (2017) Rie Johnson and Tong Zhang. 2017.  Deep pyramid convolutional neural networks for text categorization. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 562–570.
  * Ju et al. (2018) Cheng Ju, Aurélien Bibaut, and Mark van der Laan. 2018\.  The relative performance of ensemble methods with deep convolutional neural networks for image classification.  _Journal of Applied Statistics_ 45, 15 (2018), 2800–2818.
  * Kim (2014) Yoon Kim. 2014.  Convolutional neural networks for sentence classification.  _arXiv preprint arXiv:1408.5882_ (2014).
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Kumar et al. (2018a) Ritesh Kumar, Guggilla Bhanodai, Rajendra Pamula, and Maheshwar Reddy Chennuru. 2018a.  Trac-1 shared task on aggression identification: Iit (ism)@ coling‘18. In _Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)_. 58–65.
  * Kumar et al. (2018b) Ritesh Kumar, Atul Kr Ojha, Shervin Malmasi, and Marcos Zampieri. 2018b.  Benchmarking aggression identification in social media. In _Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)_. 1–11.
  * Kumar et al. (2018c) Ritesh Kumar, Atul Kr Ojha, Marcos Zampieri, and Shervin Malmasi. 2018c.  Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018). In _Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)_.
  * Kumar et al. (2018d) Ritesh Kumar, Aishwarya N. Reganti, Akshit Bhatia, and Tushar Maheshwari. 2018d.  Aggression-annotated Corpus of Hindi-English Code-mixed Data. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018)_. European Languages Resources Association (ELRA), Miyazaki, Japan.  https://www.aclweb.org/anthology/L18-1226
  * Madisetty and Desarkar (2018) Sreekanth Madisetty and Maunendra Sankar Desarkar. 2018\.  Aggression detection in social media using deep neural networks. In _Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)_. 120–127.
  * Majumder et al. (2018) Prasenjit Majumder, Thomas Mandl, et al. 2018\.  Filtering aggression from the multilingual social media feed. In _Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)_. 199–207.
  * Mikolov et al. (2018) Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018\.  Advances in Pre-Training Distributed Word Representations. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018)_.
  * Nikhil et al. (2018) Nishant Nikhil, Ramit Pahwa, Mehul Kumar Nirala, and Rohan Khilnani. 2018.  Lstms with attention for aggression detection.  _arXiv preprint arXiv:1807.06151_ (2018).
  * Orabi et al. (2018) Ahmed Husseini Orabi, Mahmoud Husseini Orabi, Qianjia Huang, Diana Inkpen, and David Van Bruwaene. 2018\.  Cyber-aggression detection using cross segment-and-concatenate multi-task learning from text. In _Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)_. 159–165.
  * Orasan (2018) Constantin Orasan. 2018\.  Aggressive language identification using word embeddings and sentiment features. In _Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)_. 113–119.
  * Pawlikowski and Chorowska (2019) Maciej Pawlikowski and Agata Chorowska. 2019.  Weighted ensemble of statistical models.  _International Journal of Forecasting_ (2019).
  * Peng et al. (2019) Hao Peng, Jianxin Li, Qiran Gong, Senzhang Wang, Lifang He, Bo Li, Lihong Wang, and Philip S Yu. 2019.  Hierarchical Taxonomy-Aware and Attentional Graph Capsule RCNNs for Large-Scale Multi-Label Text Classification.  _arXiv preprint arXiv:1906.04898_ (2019).
  * Pennebaker et al. (2007) James W Pennebaker, Roger J Booth, and Martha E Francis. 2007\.  Linguistic inquiry and word count: LIWC [Computer software].  _Austin, TX: liwc. net_ 135 (2007).
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_. 1532–1543.
  * Raiyani et al. (2018) Kashyap Raiyani, Teresa Gonçalves, Paulo Quaresma, and Vítor Nogueira. 2018.  Fully connected neural network with advance preprocessor to identify aggression over facebook and twitter.  (2018).
  * Ramiandrisoa and Mothe (2018) Faneva Ramiandrisoa and Josiane Mothe. 2018.  Irit at trac 2018. In _Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)_. 19–27.
  * Risch and Krestel (2018) Julian Risch and Ralf Krestel. 2018.  Aggression identification using deep learning and data augmentation. In _Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)_. 150–158.
  * Samghabadi et al. (2018) Niloofar Safi Samghabadi, Deepthi Mave, Sudipta Kar, and Thamar Solorio. 2018.  RiTUAL-UH at TRAC 2018 shared task: Aggression identification.  _arXiv preprint arXiv:1807.11712_ (2018).
  * Shen et al. (2019) Dinghan Shen, Asli Celikyilmaz, Yizhe Zhang, Liqun Chen, Xin Wang, Jianfeng Gao, and Lawrence Carin. 2019.  Towards Generating Long and Coherent Text with Multi-Level Latent Variable Models.  _arXiv preprint arXiv:1902.00154_ (2019).
  * Shen et al. (2014) Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014.  A latent semantic model with convolutional-pooling structure for information retrieval. In _Proceedings of the 23rd ACM international conference on conference on information and knowledge management_. ACM, 101–110.
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ (2014).
  * Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015.  Going deeper with convolutions. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 1–9.
  * Tieleman and Hinton (2012) Tijmen Tieleman and Geoffrey Hinton. 2012.  _Lecture 6.5—RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012_.  Technical Report. TRITA-MAT-E 2017: 81 ISRN-KTH/MAT/E–17/81–SE.
  * Tommasel et al. (2018) Antonela Tommasel, Juan Manuel Rodriguez, and Daniela Godoy. 2018.  Textual aggression detection through deep learning. In _Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)_. 177–187.
  * Tompson et al. (2015) Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler. 2015.  Efficient object localization using convolutional networks. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 648–656.
  * Wang (2018) Baoxin Wang. 2018\.  Disconnected Recurrent Neural Networks for Text Categorization. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 2311–2320.
  * Yang et al. (2016) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016\.  Hierarchical attention networks for document classification. In _Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies_. 1480–1489.
  * Yin et al. (2017) Wenpeng Yin, Katharina Kann, Mo Yu, and Hinrich Schütze. 2017\.  Comparative study of CNN and RNN for natural language processing.  _arXiv preprint arXiv:1702.01923_ (2017).
  * Yu et al. (2018) Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. 2018.  Mattnet: Modular attention network for referring expression comprehension. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1307–1315.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification. In _Advances in neural information processing systems_. 649–657.


# Fast and Accurate Non-Linear Predictions of Universes with Deep Learning

Renan Alves de Oliveira
PPGCosmo/Flatiron Institute
<EMAIL_ADDRESS>
&Yin Li
Flatiron Institute
<EMAIL_ADDRESS>
&Francisco Villaescusa-Navarro
Princeton University
<EMAIL_ADDRESS>
&Shirley Ho
Flatiron Institute
<EMAIL_ADDRESS>
&David N. Spergel
Flatiron Institute
<EMAIL_ADDRESS>

###### Abstract

Cosmologists aim to model the evolution of initially low amplitude Gaussian
density fluctuations into the highly non-linear "cosmic web" of galaxies and
clusters. They aim to compare simulations of this structure formation process
with observations of large-scale structure traced by galaxies and infer the
properties of the dark energy and dark matter that make up 95% of the
universe. These ensembles of simulations of billions of galaxies are
computationally demanding, so that more efficient approaches to tracing the
non-linear growth of structure are needed. We build a V-Net based model that
transforms fast linear predictions into fully nonlinear predictions from
numerical simulations. Our NN model learns to emulate the simulations down to
small scales and is both faster and more accurate than the current state-of-
the-art approximate methods. It also achieves comparable accuracy when tested
on universes of significantly different cosmological parameters from the one
used in training. This suggests that our model generalizes well beyond our
training set.

##  1 Introduction

Cosmology is at critical stage: the 2019 Nobel Prize recognized that a simple
model with only five basic parameters fits a host of astronomical observations
on large-scales where the fluctuations are in the linear regime. This model
implies that atoms make up only 5% of the universe with the remaining 95% in
the form of dark matter and dark energy. Astronomers have embarked on several
large-scale surveys and are launching multiple satellites that aim to collect
data to understand these mysteries. Analysis of this new data is limited by
our computational abilities as numerical simulations are essential for
comparing theoretical predictions to observations on the small non-linear
scales that contain most of the cosmological information [1, 2, 3, 4, 5].
Unfortunately, the computational cost of these simulations is high; a single
simulation may require from thousands to millions of CPU hours.

In this work, we aim to build a neural network (NN) model that can learn to
emulate these numerical simulations both accurately and efficiently. This
paper builds on the work of He et al. [6], which showed that neural networks
can indeed learn to run _fast approximate_ simulations. Our main contributions
is improvement of the model by training on _full N-body_ simulations and
extending to much smaller scales, where the learning task becomes more
difficult due to nonlinearity.

##  2 Methods

### N-body Simulation.

An $N$-body simulation evolves a large number of massive dark matter particles
interacting with each other only through Newtonian gravity. Since our Universe
began with matter distributed almost uniformly, the simulation starts with
particles only slightly perturbed from a uniform grid and nearly at rest.
During the simulation, a particle moves from its initial location
${\mathbf{q}}$ to its final position
${\mathbf{x}}={\mathbf{q}}+{\bm{\Psi}}({\mathbf{q}})$, where ${\bm{\Psi}}$ is
the displacement vector. $N$-body simulations usually end at our current time,
i.e. “today”; this is the time at which we compare them to our ML predictions.
The left panel of Fig. 1 shows an example of a simulation output.

We use 210 $N$-body simulations from the Quijote suite [7] for training (180),
validation (20), and testing (10). Each simulation contains $512^{3}$
particles in a box of 1 $(\mathrm{Gpc}/h)^{3}$ volume (nearly 5 billion light
years in size). In contrast to [6], the training set data was fast
approximation simulations using FastPM [8] (see next section), with $32^{3}$
particles in a volume of 128 $(\mathrm{Mpc}/h)^{3}$. The simulated universes
resemble our own, and can be characterized by the value of five cosmological
parameters: matter density parameter $\Omega_{\mathrm{m}}=0.3175$ (this means
31.7% of the Universe is made of dark matter), baryon density parameter
$\Omega_{\mathrm{b}}=0.049$, dark energy density parameter
$\Omega_{\Lambda}=0.6825$, Hubble parameter $h=0.6711$, matter fluctuation
within a 8 Mpc/h sphere $\sigma_{8}=0.834$, primordial spectral index
$n_{\mathrm{s}}=0.9624$. In addition, we also test our model, trained on the
above simulations, on three Quijote simulations with different values of the
cosmological parameters, but the same configuration otherwise.

### Fast Approximation.

$N$-body simulations can be very computationally expensive, as they typically
solve the dynamics of billions of particles for thousands of time steps. Fast
approximate simulations [9, 8] are usually adopted when a large number of
simulations are needed. These methods save computation time by only
integrating tens of time steps; they are thus less accurate than full $N$-body
simulations. We aim to build an NN model that is more accurate and faster than
these approximators. In this work we use the widely used method COLA (COmoving
Lagrangian Acceleration) [9], as implemented in the publicly available package
L-PICOLA [10], as a benchmark. COLA solves the particle motions relative to
predictions by the second order Lagrangian perturbation theory [11]. We setup
L-PICOLA using the same configurations as the full $N$-body simulations,
running it for only 10 time steps.

### Linear Theory.

Fast and accurate solutions exist when matter distribution is close to
uniform, and density fluctuations are small. For instance, at linear order,
particles move along straight lines, with the distance determined by the
growth function $D$ of time:
${\bm{\Psi}}_{\mathrm{lin}}({\mathbf{q}},t)=D(t)/D(t_{0}){\bm{\Psi}}_{\mathrm{lin}}({\mathbf{q}},t_{0})$.
At early times $t_{0}\to 0$, such as the starting time of the simulations,
${\bm{\Psi}}_{\mathrm{lin}}(t_{0})$ agrees very well with the simulation
prediction for ${\bm{\Psi}}(t_{0})$ due to uniformity.

The linear theory prediction is a very good approximation on large scales,
where density fluctuations are small. However, on small scales, the density
contrast increases drastically and structure formation becomes non-linear,
limiting the validity of linear theory predictions. Therefore, we use
${\bm{\Psi}}_{\mathrm{lin}}({\mathbf{q}},t)$ as the input to our NN model,
which predicts the fully non-linear target ${\bm{\Psi}}({\mathbf{q}},t)$ given
by the $N$-body simulations at the same $t$. By design, our NN will make
accurate predictions on large scales. This is true even if we test the model
on universes with cosmological parameters different from the one used for
training, as we show below.

### Neural Network Model.

Both the input (linear theory) ${\bm{\Psi}}_{\mathrm{lin}}$’s and target
($N$-body simulation) ${\bm{\Psi}}$’s of our NN model are functions of
${\mathbf{q}}$’s, that form a uniform grid. So each of them is a displacement
field that can be viewed as a 3D image, with three channels being the three
Cartesian components of the displacement vectors. This allows us to apply many
computer vision models to our problem.

In this work we adopt a simple U-Net / V-Net [12, 13] type architecture
similar to that in [6]. The model works on 3 levels of resolution connected in
a “V” shape by 2 downsampling layers and 2 upsampling layers, achieved by
stride-2 $2^{3}$ convolutions and stride-$\nicefrac{{1}}{{2}}$ $2^{3}$
transposed convolutions, respectively. Blocks of 2 $3^{3}$ convolutions
connect the input, the resampling, and the output layers. As in V-Net, a
residual connection, in this case, $1^{3}$ convolutions instead of identity,
are added over each of these convolution blocks. We add batch normalization
after every convolution except the first one and the last two, and leaky ReLU
activation with negative slope $0.01$ after every batch normalization, as well
as the first and the second to last convolutions. As in the original ResNet
[14], the last activation in each residual block acts after the summation. And
as in U-Net / V-Net, at all except the bottom resolution levels, the inputs to
the downsampling layers are concatenated to the outputs of the upsampling
layers. All layers have a channel size of 64, except for the input and the
output (3), as well as those after concatenations (128). Finally, an important
difference from the original U-Net / V-Net is that we also add the input
${\bm{\Psi}}_{\mathrm{lin}}$ directly to the output, so effectively the
network is learning the corrections to match the target ${\bm{\Psi}}$.

Given a displacement field, we can compute the particle positions
${\mathbf{x}}$ and calculate their density distribution, characterized by the
overdensity field $\delta({\mathbf{x}})\equiv n({\mathbf{x}})/\bar{n}-1$,
where $n({\mathbf{x}})$ is the particle number in voxel ${\mathbf{x}}$ and
$\bar{n}$ is its mean value. See Fig. 1 for an example of $1+\delta$ at
“today”. Because in cosmology the density field is closely related to
observables, e.g. galaxies, we compose a loss function that involves both
${\bm{\Psi}}$ and $n$, and combine them as
$L=\ln(L_{\delta}L_{\bm{\Psi}}^{\lambda})$, with $L_{\delta}$ and
$L_{\bm{\Psi}}$ being the MSE losses on $n({\mathbf{x}})$ and
${\bm{\Psi}}({\mathbf{q}})$ respectively. We compute $n$ from ${\bm{\Psi}}$
using the second order B-spline kernel (known as cloud-in-cell), so that $n$
is differentiable. By combining the two losses with logarithm rather than
summation, we can ignore their absolute magnitudes and trade between their
relative values. $\lambda$ serves as a weight on this trade-off of relative
losses. Through experiments, we find that a value of $\lambda$ from 2 to 5
yields the lowest $\ln(L_{\delta}L_{\bm{\Psi}})$, and in this work, we use
$\lambda=3$.

Limited by the GPU memory, the entire input, ${\bm{\Psi}}_{\mathrm{lin}}$
($3\times 512^{3}$), cannot be feed all at once to the model and needs to be
cropped into smaller cubes of size $3\times 128^{3}$. To preserve the physical
translational equivariance, we do not use any padding in the $3^{3}$
convolutions, resulting in a smaller output than the input in spatial size. We
compensate this by periodically padding 20 voxels per side to the input cubes.
To preserve the rotational equivariance of the simulation box, we implement
data augmentation that rotates and reflects the displacement fields as in Ref.
[6]. We use the Adam optimizer [15] with learning rate $0.0001$,
$\beta_{1}=0.9$, $\beta_{2}=0.999$, and reduce the learning rate by half when
loss does not improve for 3 epochs.

##  3 Results

### Accuracy

We quantify model accuracy by compare the simulation power spectra with the
power spectra estimated by the NN and by the benchmark method. The density
power spectrum quantifies the correlation of density fluctuations as a
function of scale: $P_{\delta}(k_{i})=(N_{i}V)^{-1}\sum_{k_{i}<k\leq
k_{i+1}}\delta({\mathbf{k}})\delta(-{\mathbf{k}})$, where $V$ is the volume,
$\delta({\mathbf{k}})$ is the Fourier transform of $\delta({\mathbf{x}})$,
$k_{i}$’s are bin edges for the wavevector ${\mathbf{k}}$, and $N_{i}$ is the
number of ${\mathbf{k}}$’s falling in $(k_{i},k_{i+1}]$. The wavenumber
$k\equiv\lvert{\mathbf{k}}\rvert$ describes the scale, with low/high $k$
representing large/small scales. The cross-power spectrum,
$\delta_{\mathrm{pred}}({\mathbf{k}})\delta_{\mathrm{true}}(-{\mathbf{k}})$
measures the covariance between the predicted and target density fields. We
define a transfer function, $T(k)=\sqrt{P_{\rm{pred}}(k)/P_{\rm{true}}(k)}$,
where $P_{\rm{pred}}(k)$ and $P_{\rm{true}}(k)$ are the power spectra of the
prediction and the simulation, respectively. If $T(k)=1$, the predictions
accurate capture the amplitude of the density field. The cross-correlation
coefficient, $r$, measures the phase correlations:
$r(k)=P_{\rm{pred}\times\rm{true}}(k)/\sqrt{P_{\rm{pred}}(k)P_{\rm{true}}(k)}$,
with the numerator being the cross-power spectrum between the prediction and
the simulation. $T$ and $r$ are good estimators to quantify the accuracy of
the model, because when they are both 1 the prediction and target are
identical as proved in Ref. [6]. Instead of $r$, we use $1-r^{2}$, which gives
the amount of unexplained variance between the two fields. Similar to
$\delta$, we can also compute $P(k)$, $T(k)$, and $r(k)$ for ${\bm{\Psi}}$ by
simply replacing the scalar product with the dot vector product.

Figure 1: The left panel shows an example of the projected density field in a
Universe as predicted by an $N$-body simulation. We quantify the accuracy of
the NN against the fast approximator and the N-body simulation using the power
spectrum of the displacement field (middle panel) and the density constrast
(right panel). The NN model outperforms the fast simulator in all cases.

We consider ten Quijote simulations from the test set (see the left panel of
Fig. 1 for an example), together with their fast approximate and NN
counterparts. From each of them, we compute their displacement and density
power spectra, as well as the corresponding $T$ and $1-r^{2}$. We show the
results in the central and right panels of Fig. 1. We find that the NN
outperforms the benchmark model in accuracy on all scales and for all
considered quantities. We emphasize that our NN model produces percent-level
accurate results down to scales as small as $k\simeq 1~{}h/{\rm Mpc}$.

### Generalization to Different Universes.

We test the NN model on 2000 universes with drastically different cosmological
parameters, where structure formation proceeds in a very different way from
that in the training simulations. In particular, we varied all five
cosmological parameters: $\\{\Omega_{\rm m},\Omega_{\rm
b},h,n_{s},\sigma_{8}\\}$ that are relevant to the simulations. We show the
prediction comparisons for the density field in Fig. 2, and find that the NN
generalizes very well for cosmologies with low
$\Omega_{\mathrm{b}}/\Omega_{\mathrm{m}}$ and achieves a similar accuracy to
that in Fig. 1 (same conclusion holds for the displacement field). It also
outperforms the fast approximator COLA, even though COLA depends explicitly on
the varied cosmological parameters. This result is in concordance with that in
Ref. [6], where $\Omega_{\mathrm{m}}$ and $\sigma_{8}$ were varied
individually.

Figure 2: Accuracy comparison between predictions by the fast approximator
(blue dot-dashed) and our NN (green dashed) for universes with cosmological
parameters very different to those used in training: $\\{\Omega_{\rm
m},\Omega_{\rm b},h,n_{s},\sigma_{8}\\}$ equal to
$\\{0.1755,0.0668,0.7737,0.8849,0.6641\\}$ (left),
$\\{0.3889,0.0363,0.6275,0.9513,0.9137\\}$ (middle),
$\\{0.4291,0.0337,0.5511,1.1687,0.6619\\}$ (right). This demonstrates that our
NN generalizes very well.

### Efficiency.

We now compare the computational cost of the NN versus the fast approximation
benchmark, as well as that of the $N$-body simulations. We show in Table 1 the
total inference time to obtain the whole $512^{3}$ displacement field using 64
$128^{3}$ cubes.

The fast approximation uses 20 CPU cores (Intel Xeon CPU E5-2640 v4), and the
NN inference uses PyTorch [16] on 1 GPU (NVIDIA Tesla P100-PCIE-16GB). The
$N$-body simulation requires 500 CPU hours on 48 cores, or $\sim 10^{5}$
seconds if we scale it to 20 cores to compare with the fast approximation. Our
NN model is two times faster than the fast approximation benchmark, and more
than $10^{3}$ times faster than the $N$-body simulation.

Table 1: Runtime benchmark. The NN model is $2\times$ faster than the fast approximator, and more than $10^{3}\times$ faster than the $N$-body simulation. | Quijote | L-PICOLA | PyTorch-GPU
---|---|---|---
Wall Time (s) | $10^{5}$ | 112 | 59

##  4 Conclusions

In this paper, we have shown that neural networks are able to accurately
emulate the output of expensive N-body simulations. We have shown that our
model reproduces the results of full $N$-body simulations down to scales as
small as $k=1~{}h/{\rm Mpc}$ at present time to within $\sim 1\%$ accuracy.
When compared to the state-of-the-art N-body simulation approximator in
cosmology, our V-Net based network achieves better accuracy in less time.
Furthermore, we have demonstrated that our network generalizes extremely well,
by showing a comparable level of accuracy for universes that it has not been
trained on. Our method represents a big step forward in the direction of
reducing the computational time needed to provide theory predictions in the
non-linear regime. This paves the way to maximizing the scientific return of
current and upcoming billion dollar astronomical missions.

## Broader Impact

We think the ML methods used in this work may be useful to motivate and
enlighten students from different fields, especially those who are interested
in astronomy, and to use new tools to help us for a better understanding of
our Universe. Moreover, because of the upcoming astronomical probes for the
next ten years, we believe that the outcomes of this work might help in future
scientists run fast simulations of universes with high precision using a
regular computer.

## Acknowledgments and Disclosure of Funding

RAO thanks Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES)
and Simons Foundation. YL thanks the Simons Foundation for support of
fellowship. The Flatiron Institute is supported by the Simons Foundation. We
acknowledge that our work was partly performed using the Princeton Research
Computing resources at Princeton University which is consortium of groups led
by the Princeton Institute for Computational Science and Engineering (PICSciE)
and Office of Information Technology’s Research Computing, and made use of the
CHE cluster, managed and funded by the COSMO/CBPF/MCTI, with financial support
from FINEP and FAPERJ, and operating at Javier Magnin Computing Center/CBPF.

## References

  * [1] ChangHoon Hahn, Villaescusa-Navarro Francisco, Castorina Emanuele, and Scoccimarro Roman.  Constraining $M_{\nu}$ with the Bispectrum I: Breaking Parameter Degeneracies.  arXiv e-prints, page arXiv:1909.11107, Sep 2019.
  * [2] Elena Massara, Francisco Villaescusa-Navarro, Shirley Ho, Neal Dalal, and David N. Spergel.  Using the Marked Power Spectrum to Detect the Signature of Neutrinos in Large-Scale Structure.  arXiv e-prints, page arXiv:2001.11024, January 2020.
  * [3] Cora Uhlemann, Oliver Friedrich, Francisco Villaescusa-Navarro, Arka Banerjee, and Sand rine Codis.  Fisher for complements: Extracting cosmology and neutrino mass from the counts-in-cells PDF.  arXiv e-prints, page arXiv:1911.11158, Nov 2019.
  * [4] E. Allys, T. Marchand, J. F. Cardoso, F. Villaescusa-Navarro, S. Ho, and S. Mallat.  New Interpretable Statistics for Large Scale Structure Analysis and Generation.  arXiv e-prints, page arXiv:2006.06298, June 2020.
  * [5] Arka Banerjee and Tom Abel.  Nearest Neighbor distributions: new statistical measures for cosmological clustering.  arXiv e-prints, page arXiv:2007.13342, July 2020.
  * [6] Siyu He, Yin Li, Yu Feng, Shirley Ho, Siamak Ravanbakhsh, Wei Chen, and Barnabás Póczos.  Learning to predict the cosmological structure formation.  Proceedings of the National Academy of Sciences, 116(28):13825–13832, 2019.
  * [7] Francisco Villaescusa-Navarro, ChangHoon Hahn, Elena Massara, Arka Banerjee, Ana Maria Delgado, Doogesh Kodi Ramanah, Tom Charnock, Elena Giusarma, Yin Li, Erwan Allys, Antoine Brochard, Cora Uhlemann, Chi-Ting Chiang, Siyu He, Alice Pisani, Andrej Obuljen, Yu Feng, Emanuele Castorina, Gabriella Contardo, Christina D. Kreisch, Andrina Nicola, Justin Alsing, Roman Scoccimarro, Licia Verde, Matteo Viel, Shirley Ho, Stephane Mallat, Benjamin Wandelt, and David N. Spergel.  The quijote simulations.  The Astrophysical Journal Supplement Series, 250(1):2, aug 2020\.
  * [8] Yu Feng, Man-Yat Chu, Uroš Seljak, and Patrick McDonald.  Fastpm: a new scheme for fast simulations of dark matter and haloes.  Monthly Notices of the Royal Astronomical Society, 463(3):2273–2286, 2016.
  * [9] Svetlin Tassev, Matias Zaldarriaga, and Daniel Eisenstein.  Solving Large Scale Structure in Ten Easy Steps with COLA.  JCAP, 06:036, 2013.
  * [10] Cullan Howlett, Marc Manera, and Will J. Percival.  L-PICOLA: A parallel code for fast dark matter simulation.  Astron. Comput., 12:109–126, 2015.
  * [11] Francis Bernardeau, S Colombi, E Gaztanaga, and R Scoccimarro.  Large-scale structure of the universe and cosmological perturbation theory.  Physics reports, 367(1-3):1–248, 2002.
  * [12] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi, editors, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Lecture Notes in Computer Science, pages 234–241, Cham, 2015. Springer International Publishing.
  * [13] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation.  In 2016 Fourth International Conference on 3D Vision (3DV), pages 565–571, October 2016.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [15] Diederik P. Kingma and Jimmy Ba.  Adam: A Method for Stochastic Optimization.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.  arXiv: 1412.6980.
  * [16] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8026–8037. Curran Associates, Inc., 2019.