# Molecule Property Prediction and Classification with Graph Hypernetworks

Eliya Nachmani  Facebook AI Research and Tel Aviv University
<EMAIL_ADDRESS>Lior Wolf  Facebook AI Research and Tel Aviv University
<EMAIL_ADDRESS>
###### Abstract

Graph neural networks are currently leading the performance charts in
learning-based molecule property prediction and classification. Computational
chemistry has, therefore, become the a prominent testbed for generic graph
neural networks, as well as for specialized message passing methods. In this
work, we demonstrate that the replacement of the underlying networks with
hypernetworks leads to a boost in performance, obtaining state of the art
results in various benchmarks.

A major difficulty in the application of hypernetworks is their lack of
stability. We tackle this by combining the current message and the first
message. A recent work has tackled the training instability of hypernetworks
in the context of error correcting codes, by replacing the activation function
of the message passing network with a low-order Taylor approximation of it. We
demonstrate that our generic solution can replace this domain-specific
solution.

##  I Introduction

The field of learning-based prediction of molecule properties holds the
promise of delivering accurate predictions at a fraction of the complexity
that is required by the Density Functional Theory (DFT) models, while not
being tied to the assumptions and approximations of this theory. This order of
magnitude reduction in runtime supports not only the rapid screening of
molecule banks, with important applications in medicine, manufacturing, and
environmental science, but also the automatic design of new materials.

Molecules are often represented as graphs. Similarly to other application
fields, such as computer vision, computational chemistry has benefited both
from the development of powerful generic (graph) neural networks, as well as
the development of specialized methods that are developed for the specific
prediction tasks. For example, the state of the art NMP-Edge method of [1] is
a sophisticated domain-specific method, with many significant algorithmic
choices, which generalizes the method of [2] and incorporates ideas from the
work of [3] and [4].

In this work, we propose a generic way to improve graph neural networks and
demonstrate that it is able to improve molecule property prediction and
classification in both specialized networks and in more generic methods that
are applied to computational chemistry datasets. The value of our scheme stems
from its ability to improve upon a diverse set of already optimized state of
the art methods.

Our method employs hypernetworks, also known as dynamic networks. In such
neural networks, the weights of at least some of the layers vary dynamically
based on the input. A hypernetwork can be seen as a composite network in which
one network predicts the weights of another network. In our case, both
networks receive the messages that are passed in the graph neural network as
inputs.

Since the weights of the message generating network in the hypernetworks
change dynamically during inference, training it is a challenge. We tackle
this with a specific way of incorporating the incoming messages into the
hypernetwork. Instead of passing the current message, we pass a linear
combination of the current message and the first message. This simple
modification is enough to ensure an improvement in performance. Without it,
the hypernetwork would not typically outperform the original network.

Our experiments show that the same scheme is able to improve the predictions
provided by three state of the art methods: the NMP-Edge network, the
Invariant Graph Network of [5], and the Graph Isomorphism Network of [6]. In
addition, we evaluate our method in the domain of error correcting codes, in
which a very recent contribution by [7] employed hypernetworks to improve the
accuracy of a message passing scheme. We are able to show that our
modification of the input messages is able to replace the method used there to
stabilize the network. Taking both methods together, the results further
improve.

##  II Related Work

Graph Networks  The topic of graph neural networks has drawn considerable
attention, both in the context of specific applications, such as text analysis
[8] and computer vision [9] and as a generic tool. Earlier work employed
recursive neural networks [10, 11, 12, 13], where the information flows once
in a network that is generated based on the graph. Most current methods,
including graph spectra methods [14, 15, 16] can be cast as message passing
algorithms [3, 17, 13, 18, 4, 19, 6].

The generic message passing methods employ three networks: one that pools the
hidden states from the neighborhood graph vertices, another that updates the
hidden states based on the aggregated representation of the neighbouring
vertices, and one that reads the information from the entire graph in order to
generate the final classification. In such a network, the messages are the
hidden states of the nodes. In molecule prediction, conditioning the messages
also on the receiving graph node and storing a hidden state for the linking
edge improves performance [4, 3, 1].

An alternative to message passing techniques, is presented by permutation
equivariant operators on the tensors that represent $k$-order interactions
between graph nodes [20, 21, 22, 5]. We are able to demonstrate that applying
our method to both message passing methods and equivariant operator methods
improved performance.

Molecule property prediction  While in the past, feature engineering was the
main route to applying machine learning in chemistry [23, 24, 25, 26, 27],
neural networks have become increasingly popular. The NMP-Edge model of [1]
described in Sec. III-A is an example of a specialized model. It follows the
basic architecture of [3], in which the messages are conditioned on the nodes
across both sides of the edge and on the hidden representation of the edge. In
the NMP-Edge model, however, similar to SchNet [2], the message is an
elementwise product of a network that encodes the sending node and a network
that encodes the edge (not encoding the receiving edge directly). Also similar
to SchNet, an RBF initialization and a soft-plus activation are used. Unlike
SchNet, the edge embedding is being updated in time, following the Weave
network proposed by [4].

An example of a generic graph network solution that also excels on the popular
QM9 benchmark [28, 29] is the Invariant Graph Network (IGN) of [5], presented
in Sec. III-B.

Hypernetworks  Dynamic layers, also known as gating layers, are layers in
which the weights are determined by a separate neural network. Such networks
were introduced by [30, 31] for visual tasks that require an adaptation of the
input image. More recently, the term hypernetworks was coined to refer to a
composite neural network in which a network $f$ is trained to predict the
weights $\theta_{g}$ of another network $g$. The shift from specific layers to
entire networks was presented by [32], who employed hypernetworks for video
frames and stereo views prediction. The usage of hypernetworks for recurrent
neural networks was presented by [33]. [34] have presented a Bayesian
formulation of hypernetworks, and such networks have become prominent in meta-
learning following [35], who studied transfer learning between multiple few-
shot learning tasks.

Since the weights of network $g$ are generated instantaneously by network $f$,
[36] have used hypernetworks for searching over the space of possible network
architectures. In this case, a lengthy backpropagation optimization is
replaced by the feed forward prediction of network $f$. More related to our
work is that of [37], who use hypernetworks on graphs, also in the domain of
network architecture search. In this work, the weight generating network $f$
is a graph network that operates on the graph that captures the generated
architecture.

Another recent application of hypernetworks to graphs is the work of [7],
where an MLP generates the weights of a message passing network that decodes
error correcting codes. This generalizes earlier attempts in the domain of
network decoders including [38, 39, 40, 41, 42, 43] and is shown to improve
performance. The input to both the weight generating network $f$ and the
message generation network $g$ is the incoming message, where for the first
network, the absolute value is used. It is shown that training hypernetworks
suffers from severe initialization challenges and would often lead to the
explosion of the weights. [7], therefore, present a new activation function
that is more stable than the $arctanh$ activation typically used in message
passing decoders. In our work, we employ conventional activations, and do not
employ the absolute value for molecule prediction. We demonstrate that a
combination of the initial message (from the first iteration) with the last
message is an effective way to stabalize the training of the graph
hypernetwork and do not employ dedicated activation functions.

##  III Graph Hypernetworks

We extend three leading architectures for graph neural networks that were
either designed for the molecule inference task or shown to excel on it. In
each case, we add a hypernetwork scheme in which the input is a linear
combination of the first message passed in the network and the current
message. As our experiments show, in all three cases, sizable gains in
performance are obtained, in comparison to the underlying method. In addition,
in order to compare ourselves with a recent hypernetwork message passing
scheme, we modify the decoding method of [7].

###  III-A Extending the NMP-Edge network by [1]

In order to describe how hypernetworks are applied to the NMP-Edge network, we
rely on the original notation of [1]. Let $h^{t}_{v}$ be the hidden state of a
node associated with a specific atoms at iteration $t$, and $e^{t}_{vw}$ be
the hidden state representation of an edge, which denotes either a chemical
link between atom or spatial proximity. The hidden states of the atoms
$h^{0}_{v}$ are initialized using a look-up table and the hidden state of the
edges $e^{0}_{vw}$ are initialized using an RBF function with multiple scales,
following [1, 2].

The message passing scheme of the original NMP-Edge network takes the form:

$\displaystyle m_{v}^{t+1}$ $\displaystyle=\sum_{w\in
N(v)}M_{t}(h_{w}^{t},e_{vw}^{t}),$ (1) $\displaystyle h_{v}^{t+1}$
$\displaystyle=S_{t}\left(h_{v}^{t},m_{v}^{t+1}\right),$ (2)

where $m_{v}^{t}$ are the messages aggregated at node $v$ at time $t$, and
$M_{t}$, $S_{t}$ are the message and transition networks for iteration $t$.
These networks are dynamic (vary between iterations) but are independent of
the inputs. The earlier work by [19] uses a similar set of networks which do
not change between the iterations.

In our modified network, we replace the state transition function with the
hypernetwork $f$ and $g$ as follows:

$\theta^{t}_{g}=f\left(c\cdot h^{0}_{v}+(1-c)\cdot h^{t}_{v}\right)$ (3)
$h^{t+1}_{v}=h^{t}_{v}+g_{\theta^{t}_{g}}\left(m^{t+1}_{v}\right)$ (4)

where $c$ is a learned damping factor, which is clipped to be in the range
[0,1] and is initialized with a uniform distribution, and the weights of
network $g$ are given by $\theta^{t}_{g}$. For $t=0$ , Eq. 3 becomes
$\theta^{0}_{g}=f\left(h^{0}_{v}\right)$. Note that $f$ is a fixed function.
However, $g_{\theta^{t}_{g}}$ vary in time, since the set of weights
$\theta^{t}_{g}$ change as the input to $f$ changes.

The readout function is the same as in [1], which is two layer neural network
that pools from all of the network atoms. The network $M_{t}$, the readout
network, and network $S_{t}$ of the original architecture employ a shifted-
soft-plus network, following [2], while $f,g$ employ $tanh$. Bias terms are
not used. The number of layers is two in both $g$ and $S_{t}$. $f$ has four
layers (we believe that the architecture of $S_{t}$ is locally optimal and
adding layers to it did not improve the accuracy in our experiments).

###  III-B Extending the Invariant Graph Network of [5]

We follow the original notations of IGN, in order to enable a quick reference
to the original IGN work. The original model with $d$ blocks has the form
$F=m\circ h\circ B_{d}\circ...\circ B_{2}\circ B_{1}$ where, $h$ is an
invariant layer [44], $m$ is a MLP, and the blocks $B_{i}$ are defined as:

$Y_{i}=B_{i}(X_{i})=[m_{3}(X_{i}),m_{1}(X_{i})\odot m_{2}(X_{i})]$ (5)

where $X_{i}\in\mathbb{R}^{n\times n\times a}$ denote the input tensor to the
block, $\odot$ denotes element-wise multiplication, the square brackets denote
concatenation along the last tensor dimension, and the three MLPs
$m_{1},m_{2}:\mathbb{R}^{a}\rightarrow\mathbb{R}^{b}$ and
$m_{3}:\mathbb{R}^{a}\rightarrow\mathbb{R}^{b^{\prime}}$ are applied to each
of the $n\times n$ elements of the input tensor individually along the third
tensor dimension. The dimension of the block’s output $Y_{i}$ is, therefore,
$n\times n\times(b^{\prime}+b)$.

The modified IGN network has the form:

$F=m\circ h\circ B_{d}\circ H_{d-1}\circ...\circ B_{2}\circ H_{1}\circ B_{1},$
(6)

where $H_{1},...,H_{d-1}$ are hyper blocks. Let $Y_{i}=B_{i}(X_{i})$ be the
input tensor to the hyper block $H_{i}$. The hyper block $H_{i}$ performs the
following computation:

$\theta^{i}_{g}=f\left(c\cdot Y_{0}+(1-c)\cdot Y_{i}\right)$ (7)
$H_{i}(Y_{i})=g_{\theta^{i}_{g}}\left(c\cdot Y_{0}+(1-c)\cdot Y_{i}\right)$
(8)

where the damping factor $c$ is a learned parameter, initialized from the
uniform [0,1] distribution and clipped to remain in this range. The input
tensors for $f$ and $g$ are an aggregation of $Y_{0}$ and $Y_{i}$ with the
damping factor $c$. Each layer in $g$ is applied to each feature of the input
tensor independently along the third dimension. As before, $f$ and $g$ are
neural networks with the $tanh$ activation with $f$ having four layers and $g$
two.

We use the same suffix networks per benchmark as [5]. For QM9 $h$ of Eq. 6 is
an invariant max pooling, which is followed by a MLP $m$ with three layers.
For the classification datasets, $h$ is an invariant max pooling layer from
every block $B_{i}$ output $H_{i}(Y_{i})$ ($Y_{i}$ in the original work)
followed by a single layer. These outputs are then summed to produce the
network output.

###  III-C Extending the Graph Isomorphism Network of [6]

We now turn to the notation used in [6] to introduce the modified GIN model.
$G(V,E)$ is the graph with node feature vector $X_{v}$ for vertices $v\in V$
and edges $E$. In the graph classification problem, one is given a set of
graphs with matching labels
$\left\\{(G_{1},y_{1}),(G_{2},y_{2}),...,(G_{N},y_{N})\right\\}$. The GNN
model calculates representation vectors $h_{v}$ for each node $v$ in an
iterative manner. After convergence, a readout function calculates the global
graph embedding $h_{G}$, from which the label is predicted
$\bar{y}_{G}=M(h_{G})$ for a MLP $M$. In GIN, the readout takes the form of a
summation followed by concatenation:

$h_{G}=\left[\sum_{v\in G}h^{(1)}_{v},\sum_{v\in
G}h^{(2)}_{v},\dots,\sum_{v\in G}h^{(K)}_{v}\right]$ (9)

where $K$ is number of message passing iterations. The update function of the
hidden node representation for iteration $k$ is given by a MLP that is
specific to this iteration:

$h_{v}^{(k)}={\rm MLP}^{(k)}\left(\left(1+\epsilon^{(k)}\right)\cdot
h_{v}^{(k-1)}+\sum\nolimits_{u\in\mathcal{N}(v)}h_{u}^{(k-1)}\right)$ (10)

where $\epsilon^{(k)}$ is either a learned parameter or a fixed scalar,
depending on the experiment.

The modified GIN model we propose modifies the update step, without changing
the final readout:

$\theta^{k}_{g}=f\left(c\cdot
h^{0}_{v}+(1-c)\cdot\left(h_{v}^{(k-1)}+\sum\nolimits_{u\in\mathcal{N}(v)}h_{u}^{(k-1)}\right)\right)$
(11) $h_{v}^{(k)}=g_{\theta^{k}_{g}}\left(c\cdot
h^{0}_{v}+(1-c)\cdot\left(h_{v}^{(k-1)}+\sum\nolimits_{u\in\mathcal{N}(v)}h_{u}^{(k-1)}\right)\right)$
(12)

where $h^{0}_{v}$ is calculated from Eq. 10 with $k=0$ as $h_{v}^{(0)}={\rm
MLP}^{(0)}\left(\left(1+\epsilon^{(0)}\right)\cdot
x_{v}+\sum\nolimits_{u\in\mathcal{N}(v)}x_{u}\right)$, where $x_{v}$ is the
vector of input features of node $v$. $c$ is a damping factor that is learned
during training. $f$ and $g$ are neural networks with three and two hidden
layers respectively with the $tanh$ activation. Note that the network $g$
changes between iterations and across nodes, depending on the input to $f$.
However, the entire hypernetwork is fixed between the iterations.

###  III-D Extending the Decoding Hypernetwork of [7]

Since [7] have proposed to extend an existing network using a hypernetwork, we
modify their work in order to compare our way of converting a graph network to
a hypernetwork with theirs. Specifically, it is reported that hypernetworks
cannot train for the task of decoding error correcting codes, unless a
dedicated activation function is used, since any other activation function
attempted in their experiments leads to a divergence of the weights.

[7] modify the belief propagation algorithm of [38], which is given by:

$x^{j}_{e}=x^{j}_{(c,v)}=\left\\{\begin{matrix}\tanh\left(\frac{1}{2}\left(l_{v}+\sum_{e^{\prime}\in
N(v)\setminus\\{e\\}}w_{e^{\prime}}x^{j-1}_{e^{\prime}}\right)\right),&\\\
\hskip 142.26378pt$j$\textup{ is odd}\\\ 2arctanh\left(\prod_{e^{\prime}\in
N(c)\setminus\\{e\\}}{x^{j-1}_{e^{\prime}}}\right),&\\\ \hskip
142.26378pt$j$\textup{ is even}\end{matrix}\right.$ (13)

where $l_{v}$ is the log likelihood ratios of the input bits, $x^{j}_{e}$ is
the computed edge message for the edge $e=(c,v)$ in a Tanner graph, which is a
bidirectional graph that has variable nodes $v$ on one side and check nodes
$c$ on the other. Let $H$ be the parity check matrix. Each variable node is
indexed by an edge $e=(c,v)$ on the Tanner graph and
$N(v)=\\{(c,v)|H(c,v)=1\\}$, i.e, the set of all edges in which $v$
participates. $x^{j-1}_{e}$ is the message from the previous iteration.

The hypernetwork model of [7] has the following update equations for odd $j$:

$\theta_{g}^{j}=f(|x^{j-1}|,\theta_{f})$ (14)
$x^{j}_{e}=x^{j}_{(c,v)}=g(l_{v},x^{j-1}_{N(v,\hbox{ \leavevmode\hbox
to1.7pt{\vbox to3.2pt{\pgfpicture\makeatletter\hbox{\hskip
0.09999pt\lower-0.09999pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{{}{}}{{}}{}
{{}{}}{}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@setlinewidth{0.2pt}\pgfsys@invoke{ }\pgfsys@roundcap\pgfsys@invoke{
}{}\pgfsys@moveto{1.5pt}{0.0pt}\pgfsys@lineto{0.0pt}{3.0pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}c)},\theta_{g}^{j}),$
(15)

where $N(v,\hbox{ \leavevmode\hbox to1.7pt{\vbox
to3.2pt{\pgfpicture\makeatletter\hbox{\hskip 0.09999pt\lower-0.09999pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{{}{}}{{}}{}
{{}{}}{}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@setlinewidth{0.2pt}\pgfsys@invoke{ }\pgfsys@roundcap\pgfsys@invoke{
}{}\pgfsys@moveto{1.5pt}{0.0pt}\pgfsys@lineto{0.0pt}{3.0pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}c)$ is a vector that
contains the elements of $x^{j}$ that correspond to the indices
$N(v)\setminus\\{(c,v)\\}$. For even $j$ the update equation takes the form:

$x^{j}_{e}=x^{j}_{(c,v)}=2\sum_{m=0}^{q}\frac{1}{2m+1}\left(\prod_{e^{\prime}\in
N(c)\setminus\\{(c,v)\\}}{x_{e^{\prime}}^{j-1}}\right)^{2m+1}$ (16)

where $q$ is the degree of the Taylor approximation of $arctanh$.

We modified the model of [7] with the following update equations. For odd $j$:

$\theta_{g}^{j}=f(|c\cdot x^{0}+(1-c)\cdot x^{j-1}|,\theta_{f})$ (17)
$x^{j}_{e}=x^{j}_{(c,v)}=g(l_{v},c\cdot x^{0}+(1-c)\cdot x^{j-1}_{N(v,\hbox{
\leavevmode\hbox to1.7pt{\vbox to3.2pt{\pgfpicture\makeatletter\hbox{\hskip
0.09999pt\lower-0.09999pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{{}{}}{{}}{}
{{}{}}{}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@setlinewidth{0.2pt}\pgfsys@invoke{ }\pgfsys@roundcap\pgfsys@invoke{
}{}\pgfsys@moveto{1.5pt}{0.0pt}\pgfsys@lineto{0.0pt}{3.0pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}c)},\theta_{g}^{j}),$
(18)

where $x^{0}$ is the output of one iteration from Eq. 13, and $c$ is the
damping factor which is learned during training.

For an even $j$ we either use Eq. 16 (Taylor approximated arctanh), or
consider the conventional $arctanh$ activation, as in Eq. 13. The readout
function is not modified.

##  IV Experiments

We evaluate our model on regression and classification for predicting molecule
proprieties and for decoding linear block codes. For regression we use the
Quantum Machines 9 (QM9) dataset [28, 29] and Open Quantum Materials Database
(OQMD) [45, 46]. The QM9 dataset has $133,885$ molecules. Each molecule has
$12$ properties for predicting. When comparing our results to NMP-Edge and IGN
methods, we use the same train-validation-test split as [1] or [5],
respectively. While based on the same dataset, these two benchmarks cannot be
directly compared for many of the properties. The OQMD dataset contain
$435,582$ inorganic structures. We use the same train-validation-test split as
[1].

For classification we employ the four bioinformatics dataset of [47], which
contains protein structures or chemical compounds: MUTAG, PROTEINS, PTC and
NCI1. We use the original train folds, which are also used by [5] and [6].

For decoding error correcting codes, we use the parity check metrics of [48].
We use three classes of linear block codes: Low Density Parity Check (LDPC)
codes [49], Polar codes [50] and Bose-Chaudhuri-Hocquenghem (BCH) codes [51].

We compare with various baseline method on top of the methods that we modify.
For the QM9 and OQMD datasets we compare with V-RF [52], SchNet [2], enn-s2s
[3], Cormorant [53], Incidence [54], 123-gnn [55] and the two methods by [56]:
DTNN and MPNN. The baseline method for the classification datasets are WL
subtree [57], DCNN [58], PATCHY-SAN[59], DGCNN [60], AWL [61], GCN [16], and
GraphSAGE [62].

Implementation details The various hyperparameters were selected based on the
validation set (where vary between experiments) or set arbitrarily based on
the underlying architecture (where fixed). NMP-edge network For QM9, we
trained the models with the ADAM optimizer, with a learning rate set to
$1e-4$. The number of iterations was $4$. The number of neurons in network $f$
was $64$ and the number of neurons in network $g$ was $128$. The learning rate
decreases by a factor of $0.96$ every $100,000$ gradient steps. The minibatch
size was $32$. The node embedding size was $256$ to all the parameters. For
OQMD, we use an ADAM optimizer, with a learning rate set to $1e-4$. The
learning rate decreases by a factor of $0.96$ every $400,000$ gradient steps.
We use a minibatch of $32$ examples. The number of iterations was $3$. The
number of neurons per layer in network $f$ was $64$ and that number in network
$g$ was $128$. Invariant graph network For QM9, we use the same configuration
as [5], except for the following hyper parameters. When training one model to
predict all molecule parameters $f$ has four layers with $128$ neurons,
whereas $g$ has two layers with $128$ neurons. When we trained a separate
model for each molecule parameter, which calls for a smaller capacity, $f$ has
four layers with $64$ neurons and $g$ has two layers with $64$ neurons. For
the classification datasets we trained the models with the following
hyperparameters, learning rate was $5e-5$ for MUTAG, PTC and NCI1, and was
$1e-3$ for PROTEINS. The number of channels in blocks $B_{i}$ was $400$ for
all datasets except for PROTEINS which has $128$ channels. The number of
layers in the MLP $m$ was $2$ for all datasets. For all datasets the number of
neurons in network $f$ was $64$ and the number of neurons in network $g$ was
$64$, except for PROTEINS which has $32$ neurons for $f$ and $g$. Graph
isomorphism network For the classification datasets, we use the same training
procedure as [6], who train the model for 10 folds and choose the number of
epochs based on the cross validation accuracy over the folds. We train the
models with the following hyper-parameters. Learning rate was $5e-3$, the
models run for $5$ iterations and the number of epochs was $180$ for all
datasets. The minibatch sizes were $512$, $256$, $32$ and $16$ for NCI1, PTC,
MUTAG and PROTEINS, respectively. In all datasets, $f$ has three layers and
$64$ neurons, $g$ has two layers with $64$ neurons each. Decoding hyper-
network We use the same hyper-parameters as in [7].

###  IV-A Results

#### Graph regression

The results for the QM9 dataset are reported in Tab. I for the NMP-Edge
compatible splits and units and in Tab. III for the benchmark version used by
IGN. Our model based on the NMP-Edge architecture achieves state of the art
performance on 9 out of 12 parameters, and in only one parameter it is
outperformed by the original NMP-Edge model (and another tie).

The result for formation energy predictions OQMD, based on the NMP-Edges
architecture, is provided in Tab. II. We obtain state of the art performance
in this benchmark as well.

The QM9 model that is based on IGN obtains state of the art performance on 7
out of 12 parameters when training the model for each parameter. Furthermore,
we improve the results of 9 out of 12 parameters when comparing to IGN model
that is trained for each parameter separately. When training one model to
predict all the parameters, we improve 12 out of 12 parameters, compared to
the IGN model.

#### Graph classification

The results for the classification datasets are provided in Tab. IV. As can be
observed, our modified versions of IGN and GIN improve the baseline IGN and
GIN models in almost all cases (in one case we tie). Note that the GIN model
has many variants and our modification is based on the GIN-$\epsilon$ model.
There is no Graph Neural Network model that outperforms our results, and for
the PTC and PROTEINS datasets, our method outperforms all literature
baselines.

#### Error Correcting Codes results

In Fig. 1 we provide the BER-SNR results for multiple linear block codes. Our
method improves on [7] across codes. We get an improvement range between
$0.09dB$ and $0.12dB$ for large SNR. Moreover, in all three cases, we are able
to improve the baseline results, even without the Taylor approximation of [7].
Since [7] fail to train without the approximation (using $arctanh$ their runs
always diverge), this shows that our method stabilizes the training process
for error correcting codes. We can also observe that our results with and
without this approximation are almost identical.

###  IV-B Ablation Analysis

In Tab. V we provide an ablation analysis on the QM9 benchmark. For the NMP-
Edge model, we can observe degradation of 11 out of 12 parameters when
training without $h^{0}_{v}$ in Eq. 3 and the associated damping factor.
Moreover, when training without the hypernetwork, but with $h^{0}_{v}$ (Eq. 2
becomes $h_{v}^{t+1}=c\cdot h_{v}^{0}+(1-c)\cdot
S_{t}\left(h_{v}^{t},m_{v}^{t+1}\right)$) we get a degradation of 7 out of 12
parameters.

For the IGN model, we can observe degradation in 9 out of the 12 parameters
when training without the damping factor and $Y_{0}$ in Eq. 7, 8. Moreover,
when training without the hypernetwork but with the added first message
($X_{i}$ become $c\cdot X_{0}+(1-c)\cdot X_{i}$ in Eq. 5), we get a
degradation of 8 out of the 12 parameters.

##  V Conclusions

Graph neural networks are becoming the dominant tool in molecule prediction
and classification tasks. Here we show that by employing hypernetworks with a
stabilization mechanism, significant performance gains are obtained. In order
to demonstrate the advantage of our stabilizing mechanism over a recently
proposed hypernetwork scheme, we also show improved performance in the field
of decoding linear block codes.

## References

  * [1] P. B. Jørgensen, K. W. Jacobsen, and M. N. Schmidt, “Neural message passing with edge updates for predicting properties of molecules and materials,” _arXiv preprint arXiv:1806.03146_ , 2018.
  * [2] K. Schütt, P.-J. Kindermans, H. E. S. Felix, S. Chmiela, A. Tkatchenko, and K.-R. Müller, “SchNet: A continuous-filter convolutional neural network for modeling quantum interactions,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 991–1001.
  * [3] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl, “Neural message passing for quantum chemistry,” in _Proceedings of the 34th International Conference on Machine Learning-Volume 70_. JMLR. org, 2017, pp. 1263–1272.
  * [4] S. Kearnes, K. McCloskey, M. Berndl, V. Pande, and P. Riley, “Molecular graph convolutions: moving beyond fingerprints,” _Journal of computer-aided molecular design_ , vol. 30, no. 8, pp. 595–608, 2016.
  * [5] H. Maron, H. Ben-Hamu, H. Serviansky, and Y. Lipman, “Provably powerful graph networks,” _arXiv preprint arXiv:1905.11136_ , 2019.
  * [6] K. Xu, W. Hu, J. Leskovec, and S. Jegelka, “How powerful are graph neural networks?” in _International Conference on Learning Representations_ , 2019\. [Online]. Available: https://openreview.net/forum?id=ryGs6iA5Km
  * [7] E. Nachmani and L. Wolf, “Hyper-graph-network decoders for block codes,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * [8] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” in _Proceedings of the 2013 conference on empirical methods in natural language processing_ , 2013, pp. 1631–1642.
  * [9] J. Johnson, A. Gupta, and L. Fei-Fei, “Image generation from scene graphs,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1219–1228.
  * [10] C. Goller and A. Kuchler, “Learning task-dependent distributed representations by backpropagation through structure,” in _Proceedings of International Conference on Neural Networks (ICNN’96)_ , vol. 1. IEEE, 1996, pp. 347–352.
  * [11] M. Gori, G. Monfardini, and F. Scarselli, “A new model for learning in graph domains,” in _Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005._ , vol. 2. IEEE, 2005, pp. 729–734.
  * [12] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini, “The graph neural network model,” _IEEE Transactions on Neural Networks_ , vol. 20, no. 1, pp. 61–80, 2008.
  * [13] Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel, “Gated graph sequence neural networks,” in _ICLR_ , 2016.
  * [14] J. Bruna, W. Zaremba, A. Szlam, and Y. LeCun, “Spectral networks and locally connected networks on graphs,” _arXiv preprint arXiv:1312.6203_ , 2013.
  * [15] M. Defferrard, X. Bresson, and P. Vandergheynst, “Convolutional neural networks on graphs with fast localized spectral filtering,” in _Advances in neural information processing systems_ , 2016, pp. 3844–3852.
  * [16] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” _arXiv preprint arXiv:1609.02907_ , 2016.
  * [17] D. K. Duvenaud, D. Maclaurin, J. Iparraguirre, R. Bombarell, T. Hirzel, A. Aspuru-Guzik, and R. P. Adams, “Convolutional networks on graphs for learning molecular fingerprints,” in _Advances in neural information processing systems_ , 2015, pp. 2224–2232.
  * [18] P. Battaglia, R. Pascanu, M. Lai, D. J. Rezende _et al._ , “Interaction networks for learning about objects, relations and physics,” in _Advances in neural information processing systems_ , 2016, pp. 4502–4510.
  * [19] K. T. Schütt, F. Arbabzadah, S. Chmiela, K. R. Müller, and A. Tkatchenko, “Quantum-chemical insights from deep tensor neural networks,” _Nature communications_ , vol. 8, p. 13890, 2017.
  * [20] R. Kondor, H. T. Son, H. Pan, B. Anderson, and S. Trivedi, “Covariant compositional networks for learning graphs,” _arXiv preprint arXiv:1801.02144_ , 2018.
  * [21] H. Maron, H. Ben-Hamu, N. Shamir, and Y. Lipman, “Invariant and equivariant graph networks,” _arXiv preprint arXiv:1812.09902_ , 2018.
  * [22] R. L. Murphy, B. Srinivasan, V. Rao, and B. Ribeiro, “Relational pooling for graph representations,” _arXiv preprint arXiv:1903.02541_ , 2019.
  * [23] D. Rogers and M. Hahn, “Extended-connectivity fingerprints,” _Journal of chemical information and modeling_ , vol. 50, no. 5, pp. 742–754, 2010.
  * [24] M. Rupp, A. Tkatchenko, K.-R. Müller, and O. A. Von Lilienfeld, “Fast and accurate modeling of molecular atomization energies with machine learning,” _Physical review letters_ , vol. 108, no. 5, p. 058301, 2012.
  * [25] G. Montavon, K. Hansen _et al._ , “Learning invariant representations of molecules for atomization energy prediction,” in _Advances in Neural Information Processing Systems_ , 2012, pp. 440–448.
  * [26] K. Hansen, F. Biegler _et al._ , “Machine learning predictions of molecular properties: Accurate many-body potentials and nonlocality in chemical space,” _The journal of physical chemistry letters_ , vol. 6, no. 12, pp. 2326–2331, 2015.
  * [27] B. Huang and O. A. Von Lilienfeld, “Communication: Understanding molecular representations in machine learning: The role of uniqueness and target similarity,” 2016.
  * [28] R. Ramakrishnan, P. O. Dral, M. Rupp, and O. A. Von Lilienfeld, “Quantum chemistry structures and properties of 134 kilo molecules,” _Scientific data_ , vol. 1, p. 140022, 2014.
  * [29] L. Ruddigkeit, R. Van Deursen, L. C. Blum, and J.-L. Reymond, “Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17,” _Journal of chemical information and modeling_ , vol. 52, no. 11, pp. 2864–2875, 2012.
  * [30] B. Klein, L. Wolf, and Y. Afek, “A dynamic convolutional layer for short range weather prediction,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 4840–4848.
  * [31] G. Riegler, S. Schulter, M. Rüther, and H. Bischof, “Conditioned regression models for non-blind single image super-resolution,” in _2015 IEEE International Conference on Computer Vision (ICCV)_ , Dec 2015, pp. 522–530.
  * [32] X. Jia, B. De Brabandere, T. Tuytelaars, and L. V. Gool, “Dynamic filter networks,” in _Advances in Neural Information Processing Systems_ , 2016, pp. 667–675.
  * [33] D. Ha, A. Dai, and Q. V. Le, “Hypernetworks,” _arXiv preprint arXiv:1609.09106_ , 2016.
  * [34] D. Krueger, C.-W. Huang, R. Islam, R. Turner, A. Lacoste, and A. Courville, “Bayesian hypernetworks,” _arXiv preprint arXiv:1710.04759_ , 2017.
  * [35] L. Bertinetto, J. F. Henriques, J. Valmadre, P. Torr, and A. Vedaldi, “Learning feed-forward one-shot learners,” in _Advances in Neural Information Processing Systems_ , 2016, pp. 523–531.
  * [36] A. Brock, T. Lim, J. Ritchie, and N. Weston, “SMASH: One-shot model architecture search through hypernetworks,” in _International Conference on Learning Representations_ , 2018. [Online]. Available: https://openreview.net/forum?id=rydeCEhs-
  * [37] C. Zhang, M. Ren, and R. Urtasun, “Graph hypernetworks for neural architecture search,” in _International Conference on Learning Representations_ , 2019\. [Online]. Available: https://openreview.net/forum?id=rkgW0oA9FX
  * [38] E. Nachmani, Y. Be’ery, and D. Burshtein, “Learning to decode linear codes using deep learning,” in _2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton)_. IEEE, 2016, pp. 341–346.
  * [39] H. Kim, Y. Jiang, S. Kannan, S. Oh, and P. Viswanath, “Deepcode: Feedback codes via deep learning,” in _Advances in Neural Information Processing Systems (NIPS)_ , 2018, pp. 9436–9446.
  * [40] T. Gruber, S. Cammerer, J. Hoydis, and S. ten Brink, “On deep learning-based channel decoding,” in _2017 51st Annual Conference on Information Sciences and Systems (CISS)_. IEEE, 2017, pp. 1–6.
  * [41] C.-F. Teng, C.-C. Liao, C.-H. Chen, and A.-Y. A. Wu, “Polar feature based deep architectures for automatic modulation classification considering channel fading,” in _2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP)_. IEEE, 2018, pp. 554–558.
  * [42] S. Cammerer, T. Gruber, J. Hoydis, and S. ten Brink, “Scaling deep learning-based decoding of polar codes via partitioning,” in _GLOBECOM 2017-2017 IEEE Global Communications Conference_. IEEE, 2017, pp. 1–6.
  * [43] B. Vasić, X. Xiao, and S. Lin, “Learning to decode ldpc codes with finite-alphabet message passing,” in _2018 Information Theory and Applications Workshop (ITA)_. IEEE, 2018, pp. 1–9.
  * [44] H. Maron, E. Fetaya, N. Segol, and Y. Lipman, “On the universality of invariant networks,” in _International Conference on Machine Learning_ , 2019, pp. 4363–4371.
  * [45] J. E. Saal, S. Kirklin, M. Aykol, B. Meredig, and C. Wolverton, “Materials design and discovery with high-throughput density functional theory: the open quantum materials database (oqmd),” _Jom_ , vol. 65, no. 11, pp. 1501–1509, 2013.
  * [46] S. Kirklin, J. E. Saal, B. Meredig, A. Thompson, J. W. Doak, M. Aykol, S. Rühl, and C. Wolverton, “The open quantum materials database (oqmd): assessing the accuracy of dft formation energies,” _npj Computational Materials_ , vol. 1, p. 15010, 2015.
  * [47] P. Yanardag and S. Vishwanathan, “Deep graph kernels,” in _Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. ACM, 2015, pp. 1365–1374.
  * [48] M. Helmling, S. Scholl, F. Gensheimer, T. Dietz, K. Kraft, S. Ruzika, and N. Wehn, “Database of Channel Codes and ML Simulation Results,” www.uni-kl.de/channel-codes, 2019.
  * [49] R. Gallager, “Low-density parity-check codes,” _IRE Transactions on information theory_ , vol. 8, no. 1, pp. 21–28, 1962.
  * [50] E. Arikan, “Channel polarization: A method for constructing capacity-achieving codes,” in _2008 IEEE International Symposium on Information Theory_. IEEE, 2008, pp. 1173–1177.
  * [51] R. C. Bose and D. K. Ray-Chaudhuri, “On a class of error correcting binary group codes,” _Information and control_ , vol. 3, no. 1, pp. 68–79, 1960\.
  * [52] L. Ward, R. Liu, A. Krishna, V. I. Hegde, A. Agrawal, A. Choudhary, and C. Wolverton, “Including crystal structure attributes in machine learning models of formation energies via voronoi tessellations,” _Physical Review B_ , vol. 96, no. 2, p. 024104, 2017.
  * [53] B. Anderson, T.-S. Hy, and R. Kondor, “Cormorant: Covariant molecular neural networks,” _arXiv preprint arXiv:1906.04015_ , 2019.
  * [54] M. Albooyeh, D. Bertolini, and S. Ravanbakhsh, “Incidence networks for geometric deep learning,” _arXiv preprint arXiv:1905.11460_ , 2019.
  * [55] C. Morris, M. Ritzert, M. Fey, W. L. Hamilton, J. E. Lenssen, G. Rattan, and M. Grohe, “Weisfeiler and leman go neural: Higher-order graph neural networks,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 4602–4609.
  * [56] Z. Wu, B. Ramsundar, E. N. Feinberg, J. Gomes, C. Geniesse, A. S. Pappu, K. Leswing, and V. Pande, “Moleculenet: a benchmark for molecular machine learning,” _Chemical science_ , vol. 9, no. 2, pp. 513–530, 2018.
  * [57] N. Shervashidze, P. Schweitzer, E. J. v. Leeuwen, K. Mehlhorn, and K. M. Borgwardt, “Weisfeiler-lehman graph kernels,” _Journal of Machine Learning Research_ , vol. 12, no. Sep, pp. 2539–2561, 2011.
  * [58] J. Atwood and D. Towsley, “Diffusion-convolutional neural networks,” in _Advances in Neural Information Processing Systems_ , 2016, pp. 1993–2001.
  * [59] M. Niepert, M. Ahmed, and K. Kutzkov, “Learning convolutional neural networks for graphs,” in _International conference on machine learning_ , 2016, pp. 2014–2023.
  * [60] M. Zhang, Z. Cui, M. Neumann, and Y. Chen, “An end-to-end deep learning architecture for graph classification,” in _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.
  * [61] S. Ivanov and E. Burnaev, “Anonymous walk embeddings,” in _Proceedings of the 35th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. Stockholmsmässan, Stockholm Sweden: PMLR, 10–15 Jul 2018, pp. 2186–2195. [Online]. Available: http://proceedings.mlr.press/v80/ivanov18a.html
  * [62] W. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 1024–1034.

TABLE I: Mean absolute error for QM9 molecule parameters prediction, using the NMP-Edge splits, units and our modified architecture. The lowest error is in bold. The results of the Incidence and Cormorant networks are from [54] and [53], respectively. The rest of the results from [1]. Target | Unit | SchNet | enn-s2s | NMP-Edge | Cormorant | Incidence | Ours
---|---|---|---|---|---|---|---
$\varepsilon_{HOMO}$ | $\mathrm{m}\mathrm{e}\mathrm{V}$ | 41 | 43 | 36.7 | 36 | 89 | 26.56
$\varepsilon_{LUMO}$ | $\mathrm{m}\mathrm{e}\mathrm{V}$ | 34 | 37 | 30.8 | 36 | 49 | 23.47
$\Delta\varepsilon$ | $\mathrm{m}\mathrm{e}\mathrm{V}$ | 63 | 69 | 58.0 | 60 | 68 | 41.91
ZPVE | $\mathrm{m}\mathrm{e}\mathrm{V}$ | 1.7 | 1.5 | 1.49 | 1.982 | 8 | 1.49
$\mu$ | $\mathrm{D}\mathrm{e}\mathrm{b}\mathrm{y}\mathrm{e}$ | 0.033 | 0.030 | 0.029 | 0.130 | 0.04 | 0.031
$\alpha$ | $\mathrm{B}\mathrm{o}\mathrm{h}\mathrm{r}^{3}$ | 0.235 | 0.092 | 0.077 | 0.092 | 0.03 | 0.066
$\langle R^{2}\rangle$ | $\mathrm{B}\mathrm{o}\mathrm{h}\mathrm{r}^{2}$ | 0.073 | 0.180 | 0.072 | 0.673 | 0.017 | 0.057
$U_{0}$ | $\mathrm{m}\mathrm{e}\mathrm{V}$ | 14 | 19 | 10.5 | 28 | 8 | 7.27
$U$ | $\mathrm{m}\mathrm{e}\mathrm{V}$ | 19 | 19 | 10.6 | - | 7 | 6.99
$H$ | $\mathrm{m}\mathrm{e}\mathrm{V}$ | 14 | 17 | 11.3 | - | 8 | 7.17
$G$ | $\mathrm{m}\mathrm{e}\mathrm{V}$ | 14 | 19 | 12.2 | - | 8 | 7.99
$C_{v}$ | $\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{/}\mathrm{m}\mathrm{o}\mathrm{l}\mathrm{K}$ | 0.033 | 0.040 | 0.032 | 0.031 | 0.028 | 0.026
TABLE II: OQMD - NMP-Edge. Mean absolute error for formation energy predictions. The results of the various baselines are from [1] Method: | V-RF | SchNet | NMP-Edge | Ours
---|---|---|---|---
$\mathrm{m}\mathrm{e}\mathrm{V}\mathrm{/}\mathrm{a}\mathrm{t}\mathrm{o}\mathrm{m}$ | 74.5 | 27.5 | 14.9 | 13.5
TABLE III: Mean absolute error for QM9 molecule parameters prediction for the IGN splits, units, and our modified architecture of it. The results of the various datasets are taken from [5]. For IGN and our modifications, we show results of a single network predicting all values and of dedicated networks. Target | DTNN | MPNN | 123-gnn | IGN - all | IGN - single | Ours - all | Ours - single
---|---|---|---|---|---|---|---
$\mu$ | 0.244 | 0.358 | 0.476 | 0.231 | 0.0934 | 0.157 | 0.0883
$\alpha$ | 0.95 | 0.89 | 0.27 | 0.382 | 0.318 | 0.325 | 0.303
$\epsilon_{homo}$ | 0.00388 | 0.00541 | 0.00337 | 0.00276 | 0.00174 | 0.00203 | 0.00178
$\epsilon_{lumo}$ | 0.00512 | 0.00623 | 0.00351 | 0.00287 | 0.0021 | 0.00228 | 0.0020
$\Delta_{\epsilon}$ | 0.0112 | 0.0066 | 0.0048 | 0.00406 | 0.0029 | 0.00306 | 0.0027
$\langle R^{2}\rangle$ | 17 | 28.5 | 22.9 | 16.07 | 3.78 | 13.9 | 7.56
$ZPVE$ | 0.00172 | 0.00216 | 0.00019 | 0.00064 | 0.000399 | 0.00049 | 0.000396
$U_{0}$ | 2.43 | 2.05 | 0.0427 | 0.234 | 0.022 | 0.093 | 0.018
$U$ | 2.43 | 2 | 0.111 | 0.234 | 0.0504 | 0.092 | 0.0174
$H$ | 2.43 | 2.02 | 0.0419 | 0.229 | 0.0294 | 0.093 | 0.0193
$G$ | 2.43 | 2.02 | 0.0469 | 0.238 | 0.024 | 0.093 | 0.017
$C_{v}$ | 0.27 | 0.42 | 0.0944 | 0.184 | 0.144 | 0.180 | 0.146
TABLE IV: Test set classification accuracy (%) for MUTAG, PROTEINS, PTC, NCI1 datasets. The highest results on bold. The results of IGN are by [5], the rest of the results are from [6].  | Datasets | MUTAG | PROTEINS | PTC | NCI1 |
---|---|---|---|---|---|---
Datasets | # graphs | 188 | 1113 | 344 | 4110 |
# classes | 2 | 2 | 2 | 2 |
Avg # nodes | 17.9 | 39.1 | 25.5 | 29.8 |
Baselines | WL subtree | 90.4 $\pm$ 5.7 | 75.0 $\pm$ 3.1 | 59.9 $\pm$ 4.3 |  86.0 $\pm$ 1.8 ∗ |
DCNN | 67.0 | 61.3 | 56.6 | 62.6 |
PatchySan |  92.6 $\pm$ 4.2 ∗ | 75.9 $\pm$ 2.8 | 60.0 $\pm$ 4.8 | 78.6 $\pm$ 1.9 |
DGCNN | 85.8 | 75.5 | 58.6 | 74.4 |
AWL | 87.9 $\pm$ 9.8 | – | – | – |
GNN variants | Ours based on GIN | 90.55 $\pm$ 5.4 | 76.90 $\pm$ 2.24 | 69.68 $\pm$ 5.5 | 82.7 $\pm$ 2.0 |
GIN-0 | 89.4 $\pm$ 5.6 | 76.2 $\pm$ 2.8 | 64.6 $\pm$ 7.0 | 82.7 $\pm$ 1.7 |
GIN-$\epsilon$ | 89.0 $\pm$ 6.0 | 75.9 $\pm$ 3.8 | 63.7 $\pm$ 8.2 | 82.7 $\pm$ 1.6 |
GIN-Sum–1-Layer | 90.0 $\pm$ 8.8 | 76.2 $\pm$ 2.6 | 63.1 $\pm$ 5.7 | 82.0 $\pm$ 1.5 |
GCN-Mean–1-Layer | 85.6 $\pm$ 5.8 | 76.0 $\pm$ 3.2 | 64.2 $\pm$ 4.3 | 80.2 $\pm$ 2.0 |
GraphSAGE-Max–1-Layer | 85.1 $\pm$ 7.6 | 75.9 $\pm$ 3.2 | 63.9 $\pm$ 7.7 | 77.7 $\pm$ 1.5 |
Ours based on IGN | 91.66 $\pm$ 6.54 | 77.8 $\pm$ 5.93 | 68.23 $\pm$ 10.07 | 81.99 $\pm$ 2.08 |
IGN | 90.55 $\pm$ 8.7 | 77.2 $\pm$ 4.73 | 66.17 $\pm$ 6.54 | 83.19 $\pm$ 1.11 |
TABLE V: Ablation analysis on the two QM9 dataset views. Mean absolute error is reported in both.  | NMP-Edge split,units, and architecture | IGN split, units, and architecture
---|---|---
Target | Full | No $h_{v}^{0}$ | No hypernetwork | Full | No $X_{0}$ | No hypernetwork
$\mu$ | 0.031 | 0.033 | 0.024 |  0.157 | 0.163 | 0.171
$\alpha$ | 0.066 | 0.455 | 0.065 | 0.325 | 0.332 | 0.318
$\varepsilon_{HOMO}$ | 26.5 | 26.7 | 25.9 |  0.00203 | 0.00223 | 0.00218
$\varepsilon_{LUMO}$ | 23.4 | 23.99 | 22.5 |  0.00228 | 0.00229 | 0.00244
$\Delta\varepsilon$ | 41.916 | 34.44 | 42.934 |  0.00306 | 0.00311 | 0.00341
$\langle R^{2}\rangle$ | 0.057 | 0.183 | 0.197 | 13.9 | 13.6 | 13.8
ZPVE | 1.49 | 2.16 | 1.51 | 0.00049 | 0.00045 | 0.00045
$U_{0}$ | 7.27 | 7.71 | 9.09 |  0.093 | 0.099 | 0.102
$U$ | 6.99 | 7.73 | 9.64 |  0.092 | 0.0986 | 0.103
$H$ | 7.17 | 7.40 | 9.36 |  0.093 | 0.096 | 0.104
$G$ | 7.99 | 9.05 | 9.58 |  0.093 | 0.0997 | 0.102
$C_{v}$ | 0.026 | 0.043 | 0.025 | 0.180 | 0.163 | 0.180

$\begin{array}[]{ccc}\includegraphics[width=151.76964pt]{BCH_63_51_ICLR2020.png}&\includegraphics[width=151.76964pt]{POLAR_64_48_ICLR2020.png}&\includegraphics[width=151.76964pt]{LDPC_121_80_ICLR2020.png}\\\
(a)&(b)&(c)\\\ \end{array}$

Figure 1: BER for various values of SNR for various codes. (a) BCH (63,51),
(b) POLAR(64,48), (c) LDPC ARRAY (121,80).


# XD at SemEval-2020 Task 12: Ensemble Approach to Offensive Language
Identification in Social Media Using Transformer Encoders

Xiangjue Dong
Computer Science
Emory University
Atlanta, GA, USA
<EMAIL_ADDRESS>
&Jinho D. Choi
Computer Science
Emory University
Atlanta, GA, USA
<EMAIL_ADDRESS>

###### Abstract

This paper presents six document classification models using the latest
transformer encoders and a high-performing ensemble model for a task of
offensive language identification in social media. For the individual models,
deep transformer layers are applied to perform multi-head attentions. For the
ensemble model, the utterance representations taken from those individual
models are concatenated and fed into a linear decoder to make the final
decisions. Our ensemble model outperforms the individual models and shows up
to 8.6% improvement over the individual models on the development set. On the
test set, it achieves macro-F1 of 90.9% and becomes one of the high performing
systems among 85 participants in the sub-task A of this shared task. Our
analysis shows that although the ensemble model significantly improves the
accuracy on the development set, the improvement is not as evident on the test
set.

##  1 Introduction

00footnotetext:  This work is licensed under a Creative Commons Attribution
4.0 International License.
License details: http://creativecommons.org/licenses/by/4.0/.

With the development of IT, social media has become more and more popular for
people to express their views and exchange ideas publicly. However, some
people may take advantage of the anonymity in social media platform to express
their comments rudely, and attack other people verbally with offensive
language. To keep a healthy online environment for the adolescences [Chen et
al., 2012] and to filter offensive messages for the users [Razavi et al.,
2010], it is necessary and significant for technology companies to develop an
efficient and effective computational methods to identify offensive language
automatically.

Transformer-based contextualized embedding approaches such as BERT [Devlin et
al., 2019a], XLNet [Yang et al., 2019], RoBERTa [Liu et al., 2019], ALBERT
[Lan et al., 2020] or ELECTRA [Clark et al., 2020] have re-established the
state-of-the-art for many natural language classification tasks especially the
GLUE Dataset [Wang et al., 2018]. Their pre-trained models were pre-trained on
different large datasets, for example, BERT was pre-trained on the BookCorpus
[Zhu et al., 2015] and English Wikipedia, and RoBERTa was pre-trained on CC-
News [Nagel, 2016], OpenWebText [Gokaslan and Cohen, 2019], and Stories [Trinh
and Le, 2018] which enable their models to learn different language features.

This paper presents six transformer-based offensive language identification
models that learn different features from the target utterance. To combine the
distinctive learned language features, we introduce an ensemble strategy which
concatenates the representations of the individual models and feed them into
the linear decoder to make binary classification (Section 4.2). It largely
improves the performance over the baseline on our dev set (Section 4.4).

##  2 Related Work

Offensive language in Twitter [Wiegand et al., 2018], Facebook [Kumar et al.,
2018], and Wikipedia [Georgakopoulos et al., 2018] has been widely studied. In
addition, different aspects of offensive language have been studied, like the
type and target of offensive posts [Zampieri et al., 2019], cyberbullying
[Dinakar et al., 2011, Huang et al., 2014], aggression [Kumar et al., 2018],
toxic comments [Georgakopoulos et al., 2018] and hate speech [Badjatiya et
al., 2017, Davidson et al., 2017, Malmasi and Zampieri, 2017, Malmasi and
Zampieri, 2018].

Many deep learning approaches have been used to address the task. The
Convolutional Neural Networks (CNNs), Long Short-Term Memory Networks (LSTMs)
and FastText were applied on the hate speech detection task [Badjatiya et al.,
2017]. Gamback and Sikdar [Gambäck and Sikdar, 2017] used four Convolutional
Neural Network (CNN) models with random word vectors, word2vec word vectors,
character n-gram, and concatenation of word2vec word embeddings and character
n-grams as feature embeddings separately to categorize each tweet into four
classes: racism, sexism, both (racism and sexism) and non-hate-speech.

##  3 Data Description

The datasets we use are Offensive Language Identification Dataset (OLID)
[Zampieri et al., 2019] and Semi-Supervised Offensive Language Identification
Dataset (SOLID) [Rosenthal et al., 2020]. Given a tweet, the task is to
predict whether the content involves offensive language. Table 1 shows the
examples of offensive and non-offensive tweets in these two datasets.

Id | Tweet | Label
---|---|---
09 | @USER Buy more icecream!!! | NOT
71 | @USER That’s because you are an old man. | OFF
(a) Examples from OLID.

Id | Tweet | AVG_CONF | CONF_STD
---|---|---|---
167 | @USER Pre-ordered your book, received in July, started last night and cannot put it down! | 0.215 | 0.188
524 | a combination of innocence and corruption | 0.691 | 0.142

(b) Examples from SOLID.

Table 1: Examples in OLID and SOLID. NOT: not offensive, OFF: offensive,
AVG_CONF: average of the confidences to be offensive, CONF_STD: confidences’
standard deviation

OLID is a collection of 14,100 English tweets annotated as OFF or NOT. It is
divided into a training set of 13,240 tweets and test set of 860 tweets
[Zampieri et al., 2019]. SOLID is a collection of about 9 million English
tweets labeled in a semi-supervised manner [Rosenthal et al., 2020]. The data
are annotated with AVG_CONF and CONF_STD predicted by several supervised
models [Zampieri et al., 2020]. The test set provided by organizers this year
has 3887 tweets. Table 2 shows the statistics of OLID and SOLID.

| OLID | SOLID
---|---|---
TRN | 13240 | 9089140
TST | 860 | 3887
Table 2: Statistics of OLID and SOLID. TRN: training set, TST: test set.

##  4 Experiments

###  4.1 Data Split

For our experiments, a combination of OLID and SOLID (Section 3) is used. We
find that about 1.0% of SOLID are duplicates, which have been removed before
data splitting. For the dataset used for fine-tuning classification model, we
set threshold of AVG_CONF (Section 3) to be 0.5 in SOLID, which means the data
with AVG_CONF above 0.5 is labelled as OFF. 90% of the TRN of OLID is combined
with the whole SOLID as the new training set TRN for default transformer-based
models fine-tuning (FT). The remaining 10% of the TRN and the TST of OLID is
used as the development set DEV of FT. All the existed datasets are combined
together as the training set TRN for model pre-training (PT). After pre-
training, 99.5% of the SOLID is randomly selected as the training set TRN and
0.5% of the SOLID is randomly selected to create the development set DEV for
fine-tuning our pre-trained models into classification models and regression
models (PT-C and PT-R). In PT-C, the data with AVG_CONF above 0.5 is labelled
as OFF and in PT-R, original value of AVG_CONF is used. Furthermore, 90% of
TRN in OLID is randomly selected as the new training set TRN, and 10% of TRN
in OLID is combined with the TST of the OLID and become the development set
DEV for classification models and regression models’ further fine-tuning (PT-
C-C and PT-R-C). The ensemble model is fine-tuned on the same dataset as PT-
C-C. Table 3 shows the detailed statistics of the data split in our
experiments.

| FT | PT | PT-R | PT-C | PT-R-C | PT-C-C | E
---|---|---|---|---|---|---|---
TRN | 8,963,663 | 9,107,127 | 8,951,747 | 8,951,747 | 11,916 | 11,916 | 11,916
DEV | 2,184 | - | 44,983 | 44,983 | 2,184 | 2,184 | 2,184
Table 3: Statistics of the data split used for our experiments. TRN: training
set, DEV: development set, FT: dataset used for default model fine-tuning, PT:
dataset used for default model pre-training, PT-R: dataset used for fine-
tuning our pre-trained models into regression models, PT-C: dataset used for
fine-tuning our pre-trained models into classification models, PT-R-C: dataset
used for fine-tuning regression model into classification models, PT-C-C:
dataset used for further fine-tuning classification models, E: dataset used
for fine-tuning ensemble models.

###  4.2 Models

In general, default transformer-based models are fine-tuned as baseline
models. The sequence of embeddings of input generated from the transformer
encoder is fed into linear decoder to gain the output vector that makes the
binary classification. Then we pre-train these default models and choose the
models with lowest perplexity. Next, we fine-tune the pre-trained models into
regression models and classification models based on corresponding dataset,
respectively. Furthermore, the regression models and classification models are
fine-tuned again into classification models. In the end, sentence presentation
of individual models are concatenated and fed into linear decoder to generate
the output vector that makes the binary decision of whether or not this tweet
is offensive.

Figure 1: Overview of the individual models and the ensemble model

In our experiments, two types of transformer-based models are used as the
default models, BERT-Base model [Devlin et al., 2019b] and RoBERTa-Base model
[Liu et al., 2020]. For the default model fine-tuning part, the default BERT-
Base and RoBERTa-Base are fine-tuned on FT (Section 4.1) as baseline models.
For the pre-training part, the BERT-Base and RoBERTa-Base are pre-trained on
PT (Section 4.1). Then, the two pre-trained models which have the lowest
perplexity are fine-tuned into regression models and classification models
separately on PT-R and PT-C. Next, the fine-tuned pre-trained models are
further fine-tuned into classification models on PT-R-C and PT-C-C. Finally,
sentence presentation of six individual models are concatenated to form the
ensemble model which is fine-tuned on E. Figure 1 shows the overview of the
six individual models and the ensemble model.

###  4.3 Experimental Setup

According to our experiments, the data preprocessing doesn’t contribute
significantly to the final prediction results on such huge dataset. Thus, we
skip the data preprocessing. According to the analysis of sentence length in
the dataset, we set max_length of the models to be $128$. After an extensive
hyper-parameter search, we set learning_rate to be $2e-5$, seed_value to be
$42$, and epochs to be $10$ for our six individual models and ensemble model.
After that, we also experiment more on the ensemble model and find that the
best result is gained by changing learning_rate to $1e-5$ and dropout to
$0.5$.

###  4.4 Results

Table 4 shows the results achieved by our individual models and ensemble
model. The selected pre-trained BERT-base model and pre-trained RoBERTa-base
model have the lowest perplexities, which are 21.3 and 47.5. Our fine-tuned
pre-trained classification-classificaion BERT and RoBERTa models outperform
their counterpart baseline by about 1.7% and 1.1%, respectively. In addition,
our fine-tuned pre-trained regression-classification BERT and RoBERTa models
show 2.1% and 1.8% improvements over their baselines. The ensemble model with
learning_rate of $1e-5$ and dropout of 0.5 (E_2) achieves significantly
improvement on development set. It outperforms the BERT baseline and RoBERTa
baseline by 8.5% and 8.6%, respectively. As a result, we use this ensemble
model as our final model and submit the prediction results to the shared
task’s CodaLab page.111https://competitions.codalab.org/competitions/23285 We
achieve a macro-F1 score of 90.901% on the test set and rank 36th among 85
participants in sub-task A. After the release of the gold labels, we also
calculate our other models’ performance on test set (Table 4) and make
detailed comparison and analysis among them (Section 4.5.1).

Model | ACC_DEV | ACC_TST | P_TST | R_TST | F1_TST | Epochs
---|---|---|---|---|---|---
B-FT | 83.784 | 92.153 | 88.990 | 94.510 | 90.933 | 6
R-FT | 83.692 | 92.102 | 88.933 | 94.503 | 90.882 | 10
B-PT-C-C | 85.204 | 90.610 | 88.402 | 88.115 | 88.256 | 1
B-PT-R-C | 85.845 | 92.102 | 88.933 | 94.532 | 90.885 | 2
R-PT-C-C | 84.654 | 92.102 | 88.933 | 94.532 | 90.885 | 1
R-PT-R-C | 85.158 | 88.552 | 85.129 | 87.886 | 86.299 | 2
E | 88.548 | 92.153 | 88.990 | 94.510 | 90.933 | 2
E_1 | 90.701 | 92.153 | 88.992 | 94.396 | 90.917 | 1
E_2 | 90.884 | 92.128 | 88.962 | 94.464 | 90.901 | 2
Table 4:  Results of individual models and ensemble model on dev set and test
set. B-FT: fine-tuned default BERT-base, R-FT: fine-tuned default RoBERTa-
base, B-PT-C-C: fine-tuned our pre-trained BERT-base classification-
classification model, R-PT-C-C: fine-tuned our pre-trained RoBERTa-base
classification-classification model, B-PT-R-C: fine-tuned our pre-trained
BERT-base regression-classification model, R-PT-R-C: fine-tuned our pre-
trained RoBERTa-base regression-classification model, E: ensemble model with
default learning_rate of $2e-5$, E_1: ensemble model with lower learning_rate
of $1e-5$, E_2: submitted ensemble model with higher dropout of $0.5$.

###  4.5 Analysis

####  4.5.1 Ablation Analysis

When we fine-tuned our pre-trained models, B-PT-C, B-PT-R, R-PT-C, and R-PT-R
on only 10% of the PT-R and PT-C (Section 4.4) separately, the accuracy of
models, B-PT-C-C, B-PT-R-C, R-PT-C-C, and R-PT-R-C we get is 82.822%, 83.326%,
83.280%, and 83.646%, which is lower than the results using total data (Table
4). It indicates that deep learning models which are trained on larger dataset
perform better. For the ensemble model, when we decrease the learning_rate
from $2e-5$ (E) to $1e-5$ (E_LL), the performance improves from 88.548% to
90.701%, which shows that the ensemble model is sensitive to the change in
learning rates. By changing the default dropout from 0.1 (E_LL) to 0.5 (E_HD),
the model performance increase to 90.884%, which indicates the influence of
the dropout rate. After comparing the predicted labels from our unsubmitted
models with the released gold labels (Table 4), we can see the model which
achieves the highest accuracy on the development set doesn’t perform best on
the test set. which may be caused by overfitting. Pure fine-tuned BERT-base
model (B_FT) achieves the same accuracy as other two ensemble models. In
addition, higher accuracy can’t guarantee the higher f1-score due to the data
imbalance.

####  4.5.2 Error Analysis

The confusion matrix in Figure 2 further displays the error pattern of our
classifier on test set. As we can see, there are only three instances labeled
with OFF are misclassified to NOT while more data labeled with NOT are
classified to OFF. Table 5 shows these three misclassified offensive examples
and other misclassified not offensive tweets.

Figure 2: Confusion matrix of the ensemble model

One explanation of the results may be that the imbalance of the dataset leads
to the classifier’s preference for the majority class. It is possible that our
classifier may not capture some of the subtle nuances in meaning and contexts,
and our system still needs some improvement for these subtle details.

Id | Tweet | PL | TL
---|---|---|---
304 | Can someone please jump her ass. | NOT | OFF
2333 | @USER I don’t So far as you can recognize your Dad aa trash there’s no need to be talking to you. | NOT | OFF
2825 | @USER Wings over and it’s not even a question (sweet chili & Jamaican jerk hanger). | NOT | OFF

(a) Misclassified offensive examples.

Id | Tweet | PL | TL
---|---|---|---
3564 | @USER @USER @USER Do not engage with idiots, they’ll bring you down to their level and beat you with experience. | OFF | NOT
3725 | This heartburn is disgusting. | OFF | NOT

(b) Misclassified not offensive examples.

Table 5: Misclassified examples. PL: predicted label, TL: true label

###  4.6 Conclusion

This paper explores the performance of six individual transformer-based models
and their ensemble model for the task of offensive language identification in
social media. Default BERT-Base and RoBERTa-Base individual fine-tuning models
are adapted to establish the strong baselines for the ensemble model. Sentence
representations from six individual models are concatenated and fed into the
linear decoder to make binary decision for the ensemble model. Our ensemble
model with higher dropout shows significant improvements on accuracy, up to
8.6%, on the dev set than baseline models. However, it performs worse than the
baseline model B-FT and original ensemble model E on the test set, which has a
92.153% accuracy. It may be caused by model overfitting and data imbalance,
which are the problems we need to take into consideration in future
experiments.

## Acknowledgments

We gratefully acknowledge the support of the AWS Machine Learning Research
Awards (MLRA). Any contents in this material are those of the authors and do
not necessarily reflect the views of AWS.

## References

  * [Badjatiya et al., 2017]  Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, and Vasudeva Varma.  2017\.  Deep Learning for Hate Speech Detection in Tweets.  In Proceedings of the 26th International Conference on World Wide Web Companion, WWW ’17 Companion, page 759–760, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  * [Chen et al., 2012]  Y. Chen, Y. Zhou, S. Zhu, and H. Xu.  2012\.  Detecting Offensive Language in Social Media to Protect Adolescent Online Safety.  In 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Conference on Social Computing, pages 71–80, Sep.
  * [Clark et al., 2020]  Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning.  2020\.  ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators.  In International Conference on Learning Representations.
  * [Davidson et al., 2017]  Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber.  2017\.  Automated Hate Speech Detection and the Problem of Offensive Language.  In International AAAI Conference on Web and Social Media.
  * [Devlin et al., 2019a]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  2019a.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June. Association for Computational Linguistics.
  * [Devlin et al., 2019b]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  2019b.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171–4186.
  * [Dinakar et al., 2011]  Karthik Dinakar, Roi Reichart, and Henry Lieberman.  2011\.  Modeling the Detection of Textual Cyberbullying.  In International AAAI Conference on Web and Social Media.
  * [Gambäck and Sikdar, 2017]  Björn Gambäck and Utpal Kumar Sikdar.  2017\.  Using Convolutional Neural Networks to Classify Hate-Speech.  In Proceedings of the First Workshop on Abusive Language Online, pages 85–90, Vancouver, BC, Canada, August. Association for Computational Linguistics.
  * [Georgakopoulos et al., 2018]  Spiros V. Georgakopoulos, Sotiris K. Tasoulis, Aristidis G. Vrahatis, and Vassilis P. Plagianakos.  2018\.  Convolutional Neural Networks for Toxic Comment Classification.  CoRR, abs/1802.09957.
  * [Gokaslan and Cohen, 2019]  Aaron Gokaslan and Vanya Cohen, 2019.  OpenWebText Corpus.
  * [Huang et al., 2014]  Qianjia Huang, Vivek Kumar Singh, and Pradeep Kumar Atrey.  2014\.  Cyber Bullying Detection Using Social and Textual Analysis.  In Proceedings of the 3rd International Workshop on Socially-Aware Multimedia, SAM ’14, page 3–6, New York, NY, USA. Association for Computing Machinery.
  * [Kumar et al., 2018]  Ritesh Kumar, Atul Kr. Ojha, Shervin Malmasi, and Marcos Zampieri.  2018\.  Benchmarking Aggression Identification in Social Media.  In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), pages 1–11, Santa Fe, New Mexico, USA, August. Association for Computational Linguistics.
  * [Lan et al., 2020]  Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  2020\.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations.  In International Conference on Learning Representations.
  * [Liu et al., 2019]  Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  2019\.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.
  * [Liu et al., 2020]  Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  2020\.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  In Proceedings of the International Conference on Learning Representations.
  * [Malmasi and Zampieri, 2017]  Shervin Malmasi and Marcos Zampieri.  2017\.  Detecting Hate Speech in Social Media.  In RANLP.
  * [Malmasi and Zampieri, 2018]  Shervin Malmasi and Marcos Zampieri.  2018\.  Challenges in Discriminating Profanity from Hate Speech.  Journal of Experimental & Theoretical Artificial Intelligence, 30(2):187–202.
  * [Nagel, 2016]  Sebastian Nagel, 2016.  News Dataset Available.
  * [Razavi et al., 2010]  Amir H. Razavi, Diana Inkpen, Sasha Uritsky, and Stan Matwin.  2010\.  Offensive Language Detection Using Multi-level Classification.  In Atefeh Farzindar and Vlado Kešelj, editors, Advances in Artificial Intelligence, pages 16–27, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * [Rosenthal et al., 2020]  Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Marcos Zampieri, and Preslav Nakov.  2020\.  A Large-Scale Weakly Supervised Dataset for Offensive Language Identification.  In arxiv.
  * [Trinh and Le, 2018]  Trieu H. Trinh and Quoc V. Le.  2018\.  A Simple Method for Commonsense Reasoning.  arXiv, 1806.02847.
  * [Wang et al., 2018]  Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  2018\.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.  In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium, November. Association for Computational Linguistics.
  * [Wiegand et al., 2018]  Michael Wiegand, Melanie Siegel, and Josef Ruppenhofer.  2018\.  Overview of the GermEval 2018 Shared Task on the Identification of Offensive Language.  In Proceedings of GermEval 2018, 14th Conference on Natural Language Processing (KONVENS 2018), Vienna, Austria – September 21, 2018, pages 1 – 10. Austrian Academy of Sciences, Vienna, Austria.
  * [Yang et al., 2019]  Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.  2019\.  XLNet: Generalized Autoregressive Pretraining for Language Understanding.  In Advances in Neural Information Processing Systems 32, pages 5754–5764. Curran Associates, Inc.
  * [Zampieri et al., 2019]  Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar.  2019\.  Predicting the Type and Target of Offensive Posts in Social Media.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1415–1420, Minneapolis, Minnesota, June. Association for Computational Linguistics.
  * [Zampieri et al., 2020]  Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin.  2020\.  SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020).  In Proceedings of SemEval.
  * [Zhu et al., 2015]  Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler.  2015\.  Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books.  In The IEEE International Conference on Computer Vision (ICCV), December.