HUGE  $\BODY$

# Deep Learning Methods for Protein Family Classification on PDB Sequencing
Data

Aaron Wang
Department of Computer Science
Brown University
Providence, RI 02912
<EMAIL_ADDRESS>

###### Abstract

Composed of amino acid chains that influence how they fold and thus dictating
their function and features, proteins are a class of macromolecules that play
a central role in major biological processes and are required for the
structure, function, and regulation of the body’s tissues. Understanding
protein functions is vital to the development of therapeutics and precision
medicine, and hence the ability to classify proteins and their functions based
on measurable features is crucial; indeed, the automatic inference of a
protein’s properties from its sequence of amino acids, known as its primary
structure, remains an important open problem within the field of
bioinformatics, especially given the recent advancements in sequencing
technologies and the extensive number of known but uncategorized proteins with
unknown properties. In this work, we demonstrate and compare the performance
of several deep learning frameworks, including novel bi-directional LSTM and
convolutional models, on widely-available sequencing data from the Protein
Data Bank (PDB) of the Research Collaboratory for Structural Bioinformatics
(RCSB), as well as benchmark this performance against classical machine
learning approaches, including k-nearest neighbors and multinomial regression
classifiers, trained on experimental data. Our results show that our deep
learning models deliver superior performance to classical machine learning
methods, with the convolutional architecture providing the most impressive
inference performance.

_Keywords_ computational biology $\cdot$ proteomics $\cdot$ statistical
modeling $\cdot$ deep learning $\cdot$ machine learning

## Introduction

Proteins are macromolecules responsible for the function and regulation of the
body’s tissues and organs. At their most basic level, proteins are composed of
chains of amino acids, called polypeptide chains, that themselves are
determined by the sequence of nucleotides in a gene. Two proteins that
diverged from a common ancestral family are known as homologous and tend to
have similar structures, functions, and general behavior; when a new protein
is discovered, its functional and structural properties can be suggested based
on what group of prior labelled proteins the new proteins is most similar to.
This is the goal of protein classification. Classification techniques have
been used ubiquitously in statistics, computer science, biology, and the
social sciences, and are powerful for classifying samples within data into
classes with similar patterns; this branch of machine learning, known as
supervised learning, has proven useful for accurate prediction of group labels
in protein classification where proteins in the same class likely share
similar functionality and structure. The application of classification methods
to protein homology modeling is an important task since millions of proteins
have been studied but unsuccessfully classified into protein families. For
instance, well-curated Swiss Prot dataset contains 560,000 annotated proteins,
whereas the sparsely annotated TrEMBL dataset contains over 150 million
proteins.

## Approach

### Deep Learning

Deep learning is a specific family of models within machine learning that
relies on artificial neural networks, an algorithmic system of artificial
neurons that processes information and learns complex patterns in data, and is
broadly inspired by biological systems in the human brain; indeed, these
models have become exceptionally popular in the bioinformatics community in
recent years because of their ability to extract conclusive and often complex
patterns from large datasets. [27] first used feed forward neural network for
protein structure prediction, and since then neural networks have been used
for a number of applications in the field of genomics, which include
predicting the outcome of alternative splicing in exon skipping events [12],
annotating coding and non-coding genetic variants for identifying pathogenic
variants [13], and identifying active cis-regulatory regions in the human
genome ( [14], [15]). Many of these methods report improved predictive
performance over methods such as linear regression, k-nearest neighbors
clustering, and support vector machines. In this work, we consider two types
of neural network architectures that differ in how the data is processed
within the models: recurrent neural networks (RNN) and convolutional neural
networks (CNN).

#### Recurrent Neural Networks

Recurrent neural networks, also known as RNNs, are a class of neural networks
that introduce state variables to store past information, together with the
current inputs, to determine outputs. As such, RNNs are are designed for
sequential or time-series data, as hidden layers of the RNN can be viewed as
memory states that retain information from the sequence previously observed
and are updated at each time step. More specifically, RNNs process the input
sequentially, such that at each time step, the RNN takes as input the current
subsequence of the input and output of the past layer; what results from these
operations is a context matrix contains the information of the sequence, which
can be used as input to another model, such as a feedforward neural network.
This architecture suffers from the vanishing gradient problem, wherein the
weights of the model are unable to learn long-term sequential patterns due to
the exponential shrinking of the gradient per time step. To remediate this we
utilize a variation of the RNN, known as a bidirectional LSTM, that make use
of gating procedures that regulate the neural network’s ability to model long-
term dependencies.

In genomics, RNNs have been used in applications including prediction of
single-cell DNA methylation states from CpG coverage [16] and prediction of
non-coding function de novo from DNA sequences [17].

#### Convolutional Neural Networks

In a convolutional neural network (CNN), a convolutional filter, or kernel, is
scanned across the entire input matrix and computes the local weighted sum of
each region. Each convolutional layer scans the input with several of these
filters, each producing a scalar value at every position that denotes the
"match" between the input and the kernel- that is, each kernel is initialized
separately and attempts to learn a distinct spatial feature of the data. After
the convolutional layer, a nonlinear activation function is applied, often in
conjunction with dropout and batch normalization to prevent overfitting.
Furthermore, a pooling layer, which reduces the spatial size of the input with
the purpose of denoising the data, is applied. This set of layers is known as
a convolutional block, and are repeated in a network architecture to enhance
the signal produced by the CNN; the output of the final block can be used as
input to a multi-layer perception for downstream classification.

DeepBind [23], Basset [24], and DeepSea [25] were the first to apply CNN
methods genomic datasets. The authors of DeepBind trained several single-task
models for binary classification to predict binding affinities of
transcription factors, showing that their deep learning approach performed
consistently better than existing state-of-the-art classical machine learning
approaches. Similarly, the authors of Basset utilize a CNN to predict DNA
accessibility features, again modeling the task as binary classification. The
DeepSEA model, the largest model of the three with over 50 million trainable
parameters, predicts the binary presence or absence of chromatin features,
such as transcription factor binding, DNA accessibility, and histone
modification. In this work, we model the task as a multi-class classification
problem, as opposed to binary classification. We transform our sequential data
by embedding each token in the sentence and using this transformed matrix as
input to the first convolutional layer and passing in the out of the final
convolutional block as input to the feed forward neural network for
prediction. This technique of convolution across the encoded matrix is also
seen in DeepChrome [26], where the model’s kernels slide across the one-hot
encoded DNA sequences to predict gene expression on histone modification data.

### Classical Machine Learning

Machine learning (ML) is a branch of artificial intelligence (AI) where
computer systems are able to automatically learn and improve from patterns and
observations in data without being explicitly programmed [12]. Machine
learning methods lend themselves well to tasks that require the compilation of
a list of sequence elements into a set of classes, and thus machine learning
techniques have extensively been applied to areas of genomics and genetics.
For example, machine learning methods have been used to identify splice sites
[18], enhancers [20], promoters [19], and positioned nucleosomes [21], as well
as used similar techniques to identify locations of transcription start sites
in genomic sequences [22]. The effectiveness of machine learning on these
tasks are only enabled by the presence of large, complex datasets, many of
which have only recently become available through international collaborative
projects, such as the NIH’s 4D Nucleome Initiative, ENCODE, the Roadmap
Epigenomics Project, and the 100000 Genomes Project. In this work, we utilize
data from the Protein Data Bank to predict protein sequences into a set of
protein family classes from experimental data using two well-studied machine
learning models, multinomial regression and k-nearest neighbors, as a form of
comparison against our deep learning models.

#### Multinomial Regression

Multinomial regression is a generalization of logistic regression to allow for
more than two possible categorical labels. In this section, we provide brief
analysis and intuition for the underlying mathematics behind the model.

Let $X=\\{x_{1},x_{2},...,x_{n}\\}$ in $\mathbb{R}^{n\times d}$ for $d$
features and $Y=\\{y_{1},y_{2},...,y_{n}\\}$ in $\mathbb{R}^{n\times 1}$ where
$y_{i}\in\\{1,2,...,k\\}$ for $k$ classes. We would like to utilize
statistical machine learning for multi-class classification such that
$X\xrightarrow{model}Y$, and multinomial regression is an intuitive approach,
since the generated prediction is the label with the maximum predicted
probability, itself coming from the softmax function. The parameters of the
model are optimized to maximize the likelihood of the observed data.

Figure 1: Figure 1: Multinomial Regression Overview 222the omission of
$\alpha$ is done to conserve space for the figure and does not affect the Big
O runtime, space complexity, or inherent logic of the model.

Similar to logistic regression, multinomial regression begins by calculating
the logit matrix $Z$ in $\mathbb{R}^{n\times k}$. Note that this is a matrix
since with $K$ classes, whereas logistic regression calculates a logit vector
for $k=2$ and
$\operatorname*{\mathbb{P}}(Y=0|X=x)=1-\operatorname*{\mathbb{P}}(Y=1|X=x)$.
Outputting the logits directly is not possible since the probabilities
outputted by our model before $argmax$ must be non-negative and sum to 1, as
defined by the axioms of probability; instead, the model exponentiates each
logit to ensure non-negativity and divides by their sum to ensure a summation
of 1. This is the intuition behind the softmax function as first defined by
Robert Duncan Luce in 1959 [8] and is what is used to calculate
$\operatorname*{\mathbb{P}}(Y_{i}=k|X=x_{i})$. The multinomial regression is
still a linear model despite the softmax function being non-linear, as the
outputs of multinomial regression are determined by an affine transformation
of the feature set.

$\operatorname*{\mathbb{P}}(Y_{i}=k|X=x_{i})=softmax(Z_{i,k})=\frac{e^{Z_{i,k}}}{\sum_{k=1}^{K}e^{Z_{i,k}}}$
(1)

When $K=2$,
$\operatorname*{\mathbb{P}}(Y_{i}=1|X=x_{i})=\frac{e^{Z_{i,1}}}{e^{Z_{i,1}}+e^{Z_{i,2}}}=\frac{e^{Z_{i,1}}}{e^{Z_{i,1}}(1+e^{Z_{i,2}-Z_{i,1}})}=\frac{1}{1+e^{-Z^{\prime}}}$,
which is of the same form as the sigmoid function for logistic
regression333$Z^{\prime}=Z_{i,1}-Z_{i,2}$. There are an infinite number of
combinations of $\beta_{1}$ and $\beta_{2}$ that will produce the same
probabilities for $Y_{1}$ and $Y_{2}$, so $\beta_{2}$ can be set to $0$ such
that $Z^{\prime}=Z_{i,1}$., the only difference being that multinomial is
generalized to $K>2$ classes.
With regards to defining a loss function to quantify the performance of the
model’s parameter choices, cross-entropy loss is most often used in literature
with multinomial regression since it is derived from MLE [3], which is
straight-forward and only requires some knowledge in information theory; a
more interesting derivation of cross-entropy loss comes from KL divergence, a
statistical measure of how different two probability distributions are [6]. We
can apply this to the true class distribution (denoted
$\operatorname*{\mathbb{P}}(y|x_{i})$) and the predicted class distribution
(denoted $\operatorname*{\mathbb{P}}(\hat{y}|x_{i})$) and attempt to minimize
the KL divergence.

$D_{KL}(\operatorname*{\mathbb{P}}(y|x_{i})||\operatorname*{\mathbb{P}}_{\theta}(\hat{y}|x_{i}))\stackrel{{\scriptstyle\triangle}}{{=}}\sum_{i=1}^{n}\operatorname*{\mathbb{P}}(y|x_{i})\times\log\frac{\operatorname*{\mathbb{P}}(y|x_{i})}{\operatorname*{\mathbb{P}}_{\theta}(\hat{y}|x_{i})}$
(2)

After simplification, only
$-\sum_{i=1}^{n}\operatorname*{\mathbb{P}}(y|x_{i})\times\log\operatorname*{\mathbb{P}}_{\theta}(\hat{y}|x_{i})$
depends on parameter $\theta$. Thus,
$\operatorname*{argmin}\limits_{\theta}D_{KL}(\operatorname*{\mathbb{P}}(y|x_{i})||\operatorname*{\mathbb{P}}_{\theta}(\hat{y}|x_{i}))=\operatorname*{argmin}\limits_{\theta}-\sum_{i=1}^{n}\operatorname*{\mathbb{P}}(y|x_{i})\log\operatorname*{\mathbb{P}}_{\theta}(\hat{y}|x_{i})$.
Naturally, this result hints that maximizing likelihood is equivalent to
minimizing KL divergence, a notion shown by [6]; that is, maximizing the
likelihood of our observed data under our model is equivalent to minimizing
the difference between our model’s prediction and the true data distribution.

Calculating the root of the derivative of cross-entropy loss with respect to
the parameters is difficult [9], so iterative optimization algorithms are
deployed to approximate values of the parameters, such as
Broyden–Fletcher–Goldfarb–Shanno algorithm, the Newton–Raphson method, and
quasi versions of the latter. In this work, we use Newton-CG which applies the
linear conjugate gradients to the second-order Taylor-series approximations of
$\operatorname*{\mathbb{P}}$ around $\theta_{i}$; it has shown to converge
well and in reasonable time complexity as each iteration requires the
computation of the Hessian Matrix. One drawback of the Newton-CG method is
that saddle points are easier to fall into [5], but the cross entropy loss
function is convex, so this will not be an issue. Our results show that the
choice of optimizer does not affect our experiment significantly.
Multinomial regression is appropriate for protein classification because the
explanatory variables of the data are all ordinal and the response variable is
categorical, which is what the model expects as input and outputs,
respectively. Furthermore, the proteins were independently sampled from one
another and all redundant samples are removed from the data. The features
themselves do not show perfect multicollinearity, which is also an assumption
of the model that is met. In addition to these assumptions of the model that
are met, the model specificically does not make several assumptions which also
lends itself well to the protein classification task: Multinomial regression
does not require homoscedasticity or that the residuals be normally
distributed, an assumption that OLS regression requires [4].

#### K-Nearest Neighbors

The K-Nearest Neighbor (KNN) algorithm operates on the basis that similar
samples exist in close proximity to one another in Euclidean space; that is, a
data point is classified as the label that is most frequent among its k
nearest neighbors, where k is a hyperparameter tuned by the researcher. More
specifically, let $C$ be the set of $k$ samples closest to point $X_{i}$:

$\operatorname*{\mathbb{P}}(Y_{i}=y|X=x)=\frac{1}{K}\sum_{X_{j}\in
C}\mathbbm{1}(Y_{j}=y)$ (3)

The $k$ "closest" samples are determined by how we define distance, though
common metrics include Euclidean, Manhattan, Chebyshev, and Hamming distance.
The choice of distance metric depends on the data, such as the frequency of
outliers you expect, the dispersion of the data, etc.; this work utilizes
Euclidean distance since the dimensionality of the data is not high, in which
we may look to another metric like Manhattan distance, and Euclidean distance
is by far the most popular [10]. The most optimal choice of k depends on the
data and the task, although smaller values of k increase the effect of the
variance of the samples on the classification. In this work, we tested values
of k up to 30 and found that k=3 works best for this task.

Figure 2: Figure 2: Parameter Tuning for k

Note that no distinct training is necessary, since the label of a sample is
generated by Euclidean distance between the sample and every point. The
precomputing of these distances, as well as the use of a k-d tree data
structure, are options to increase the efficiency of this algorithm. The
inferencing pseudocode for this algorithm is outlined below.

Load the data $(x_{1},y_{1}),(x_{2},y_{2}),...,(x_{n},y_{n})$, initialize the
value $K=k$, and define a distance metric $d$. To determine the label $y_{l}$
of a new sample $x_{l}$:

  1. Step 1:

Compute and store $d(x_{l},x_{j})\forall x_{j}\in X$ in $L$

  2. Step 2:

Sort $L$ and take the first $k$ samples with smallest distances

  3. Step 3:

Let $y_{l}$ be the majority class of the $k$ samples

Figure 3: Pseudocode of Inferencing with KNN

## Experimental Setup

### Protein Data Bank

The Protein Data Bank (PDB) is a freely accessible, interoperating archive of
structured data and metadata of biological macromolecules (proteins and
nucleic acids) for research and education across the sciences funded by the
National Science Foundation, the National Institutes of Health, and the
Department of Energy. The experimental data, obtained by biologists using
methods such as X-ray crystallography, NMR spectroscopy, and cryo-electron
microscopy, are open-source on the Internet through its member organizations.
The PDB is set as the gold standard for many areas of biology, including
structural genomics, and other databases use protein structures from the PDB.
In fact, many scientific journals and funding agencies require researchers to
submit their data to the PDB. Hence, the database is rich with thousands of
samples of proteins, each with features that may be useful as covariates for
classification. These features include the primary structure of each protein,
which will be of interest when building our sequence-based deep learning
models.

Figure 3: Figure 3: Top Frequencies of Protein Families in Protein Data Bank

The raw PDB dataset includes 60757 non-unique samples of proteins, DNA, and
RNA with 16 total features including publication data and sequencing
information. After removing redundant and non-protein samples and reducing the
features to only include experimental and primary protein structure data,
proteins with NaN data points or that were not a part of the ten most frequent
protein families were also excluded from the analyses. Of the remaining 36,411
proteins, 7,282 are left for testing, while the remaining 29,129 samples are
used for training and validation with a 15% split. The data pipeline for this
work is of the form:

$f(X)=f_{\text{model}}(f_{\text{onehot}}(f_{\text{tokenization}}(f_{\text{kmer}}(X))))$
(4)

### Deep Learning Models

#### RNN

In this work, the RNN implemented is a bidirectional LSTM architecture using
the Tensorflow framework. This architecture is well-studied and academic
literature on the topic is broad, so we will instead focus on how we transform
the protein sequencing data into a format usable for these types of models.
Recall that RNN models require a sequence of tokens, where each token is
composed of letters of an alphabet, as input. The current data does not
satisfy this condition as long sequences of proteins. Hence, we can attempt to
create a language for our model by breaking up a protein sequence into tokens
using the following trick. First, let us define the alphabet of our language
to be the set of amino acids that comprise a protein; the length of our
alphabet is thus twenty. Next, we define tokens as equal-length k-mers that
are generated by sliding a window of size $k$ across the sequence, where each
step $s$ constitutes a new word in the sentence. Tada! Our protein sequence
has transformed into a sentence of amino-acid words that hold intrinsic
information about the protein in both the amino acids and the ordering of the
tokens. With a tokenized sequence, we now move onto embedding. The motivation
behind this step is that our language model requires a matrix-representation
of the inherent meaning of the tokens that can be processed by the hidden
cells of the RNN. Essentially, we would like to project the information of a
string token into euclidean space. To do this, we initialize each token as a
random $1xd$ vector, where $d$ is the number of dimensions of the euclidean
space, and train a feed forward neural network to learn the true value of
these vectors. We can reformat our protein sequence into the form $[(a,c),b]$,
where $b$ represents the current token, $a$ represents the $t$ tokens before
it, and $b$ represents the $t$ tokens after it. We call $(a,c)$ the context
vectors around $b$ for which the "lemma" neural network model will attempt to
learn the vector representation of. Formally, this form of learning is known
as word2vec, as we find a mapping of words to vectors, and this particular
context-based algorithm is known as the skip-gram architecture [29].
Our character-based language model operates on protein sequence data tokenized
on the level of amino acids. Each protein sequence can now be represented as
an $nxd$ input matrix, where each $n$ represents the length of the sequence
and $d$ represents the dimension of the embedding space. The architecture of
this model is of the form:

$f(X)=f_{\text{mlp}}(f_{\text{BDLSTM}}(f_{\text{embedding}}(X)))$ (5)

#### CNN

In this work, we implement a CNN for protein family classification using the
Tensorflow framework. Our model is composed of five stages: convolution,
rectification, pooling, dropout, and a multi-layer perceptron. Each
convolution stage consists of 128 filters of varying sizes between four and
six such that each filter learns a different pattern of the data. The size and
number of filters is a hyperparameter at the discretion of the authors, and we
found that these particular sizes and number of filters worked well for this
task. The rectification stage pairs with the prior stage to apply a non-
linearity function called rectified linear unit (ReLU), which is an
elementwise operation that sets all negative values to approach zero. Here, we
utilize leaky ReLU instead of the traditional ReLU to avoid vanishing gradient
issues that may persist with the size of our model.

$f_{\text{leaky ReLU}}(z)=\begin{cases}\alpha x&\text{if $x\leq 0$}\\\
x&\text{if $x>0$}\end{cases}$ (6)

Next, we use temporal maxpooling, which selects the maximum values in a
certain window, to learn translational invariant features and denoise the
output of prior stages. For this CNN model, we set a window size of 2, which
conservatively allows for both denoising of the output of prior stages while
also allowing the model to have enough data to make an informed prediction.
Following pooling is dropout [28], which helps to prevent overfitting by
randomly zeroes the inputs to the next layer during training with a set
probability of 0.2. It should be noted that this method closely resembles
ensemble techniques in classical machine learning like bagging or model
averaging which are heavily utilized in bioinformatics. Finally, this output
is fed into a feedforward neural network to learn a classification function
mapping to protein family labels. Each layer in the MLP maps its input to a
hidden feature space, known as hidden layers, and the final layer learns to
map from the hidden space of the last hidden layer to the class label space
through a softmax function. This model is trained using the Adam optimizer to
minimize our categorical cross-entropy loss function. The architecture of this
model is of the form:

$f(X)=f_{\text{mlp}}(f_{\text{dropout}}(f_{\text{pool}}(f_{\text{relu}}(f_{\text{conv}}(X)))))$
(7)

### Baseline Models

The baseline models utilize the experimental data available, as opposed to the
sequencing data, as the experimental data matches the input parameters for the
kNN classifier and the multinomial regression model. As such, the 36,411
protein samples are feature-normalized to a standard normal distribution. Both
models are implemented in Python. The kNN classifier was run to produce an
elbow graph for the best value of $k$. In Figure 2, we see that the optimal
value of $k$ is 3. No hyperparameter tuning is needed for multinomial
regression.

#### Feature Selection

Extremely Randomized Trees Classifiers (Extra-Treess Classifier) are an
ensemble learning technique that aggregates the results of multiple collected
de-correlated decision trees to find features that are most important in the
data set, a technique first utilized by [11]. During the construction of the
forest, the normalized total reduction of feature split decision for each
feature is computed, also called Gini importance of the feature, describes how
"important" the particular feature is to the outcome variable. Our results
from running the Extra-Trees Classifier on our experimental data features are
presented in Figure 5. We found that the Gini index values for each feature is
approximately 0.11, except for the residue count and molecular weight features
which have Gini indices values closer to 0.20, we utilize all features for the
classical machine learning models.

Figure 5: Gini Indices for Feature Selection

## Results

Three metrics were utilized for evaluating the performance of the classifiers
on the data, each with their own scoring schemes: accuracy, precision, and
recall. Accuracy measures the ratio of correct classifications (true positives
and true negatives) to the total number of samples. In other words, accuracy =
$\frac{TP+TN}{TP+TN+FP+FN}$. Precision is the proportion of the
classifications of proteins to a particular protein family that actually
belong to that family. In other words, precision = $\frac{TP}{TP+FP}$. Recall
is the proportion of proteins in a protein family that were classified into
the right family. In other words, recall = $\frac{TP}{TP+FN}$. Because these
evaluation metrics are used on multi-class classification models, accuracy,
precision, and recall are calculated for each class and averaged to get the
final metrics of that particular model.

| Accuracy | Precision | Recall
---|---|---|---
Multinomial Regression | 0.29 | 0.23 | 0.29
K-Nearest Neighbors | 0.40 | 0.41 | 0.40
Bi-Directional LSTM | 0.79 | 0.75 | 0.77
CNN | 0.83 | 0.79 | 0.81
Figure 4: Figure 6: Performance Metrics of Classifiers

Due to the contrast in results between the deep learning models and classical
ML models, we begin our discussion of the results of the two separately. Of
the ML models, the KNN classifier had the higher accuracy of 0.40, which
indicates that proteins within the same family are closer in Euclidean space
than proteins in other families. Looking back at our data, this result makes
sense because the number of samples in our data is high, which means that
there are more proteins in the same family near each other, resulting in a
higher accuracy. The KNN classifier also had the higher precision of 0.41.
This shows that the data’s relatively lower dimensionality also aided the
performance of the KNN, for if the dimensionality of the data were high, the
model would be more susceptible to adding $x_{j}$’s of a different protein
family into $x_{i}$’s $k$ nearest neighbors since the curse of dimensionality
makes it more difficult for the model to differentiate between different
points in space using distance [7]. Not surprisingly, the recall of the KNN
classifier was also the highest. Again, the sheer number of samples played a
large role in this; since there are more proteins of the same family
neighboring each other, a higher proportion of the proteins within the same
family will be classified correctly, thus leading to a higher precision.
The multinomial regression model was unable to perform as well as the KNN
classifier. The first reason for this is the class imbalance of the data. Of
the 36411 proteins, 10190 were from a single protein family, hydrolase, which
is more than the frequencies of the bottom six proteins combined. Thus, the
model learned to guess that any protein it saw most likely came from the
hydrolase family. In fact, this is clear from the model’s accuracy of 0.29,
which is just slightly above the relative hydrolase frequency in the data of
0.28. Indeed, this also explains the precision score of 0.23; the overall
model precision is an average of the precision of each protein family, most of
which will be lower than $0.29$ and thus lower the average. The precision for
hydrolase is $0.30$, which means that the model is guessing for a majority of
the proteins that they belong to the hydrolase protein family, for if this
weren’t the case, then the hydrolase precision would be much higher. This is
supported by the near-zero precision scores of the other 9 protein families.

(a) CNN Confusion Matrix

(b) LSTM Confusion Matrix

Figure 5: Figure 7: Deep learning models confusion matrices

The deep learning models presented in this work perform exceptionally better
than both of the classical machine learning approaches. The CNN performed the
best with an accuracy of 0.83, while the LSTM lagged only slightly with an
accuracy of 0.79. In fact, the evaluation metric of each deep learning model
is greater than the two classical machine learning models’ evaluation metrics
combined. There are several reasons for this discrepancy. The deep learning
models are severely more complex than the classical machine learning models,
and because protein family classification is such a difficult task, it is
intuitive that classical machine learning models do not have the complexity
bandwidth to learn all of the intrinsic patterns within the data. In
comparison to the classical machine learning models, which only have a handful
of trainable parameters (kNN does not have even have a training phase, whereas
multinomial regression has only $f\cdot k=70$ trainable parameters), the
convolutional neural network has almost 4 million trainable parameters and the
bi-directional LSTM model has over 41 million trainable parameters. The added
complexities of the deep learning models allow for ordered representation of
the data, whether spatial or sequential, that is not seen in the classical
machine learning models. Furthermore, it should be noted that the models use
different data as input. While the classical ML models used experimental data,
the deep learning models used protein sequence data; this matters because the
protein sequence data has an inherent order to it, unlike the experimental
data, that conveys information.

(a) CNN Accuracy

(b) LSTM Accuracy

Figure 6: Figure 8: Deep learning models training and validation accuracy per
epoch

## Discussion

In this work, we have presented a comparison of several deep learning and
machine learning models, including a recurrent neural network, a convolutional
neural network, multinomial regression, and k-nearest neighbor, for protein
classification on sequential and experimental data from the Protein Data Bank.
Our findings show that despite multinomial regression fails to provide an edge
over the k-nearest neighbors classifier. There are several reasons why this
may be the case, including class imbalance, low feature space, and high number
of samples. Hyperparameter tuning of the multinomial regression model, such as
trying different optimizers, did not make any significant change in the
performance of the model. Another observation is that the data points
collected may not have enough significant figures to differentiate the samples
well enough. For example, the observations within the crystallization
temperature feature (in Kelvin) only contain one decimal place, which
certainly is not enough to differentiate 36000 proteins into 10 classes,
especially since deviation outside of the optimal temperature leads to protein
denaturation. This aspect of the data will certainly improve the performance
of the KNN since neighbors of a sample may even have the same measurements,
but will decrease the performance of the multinomial regression model since a
high number of the same sample measurements leads to guessing the majority
class. This is supported by the plot of the effect of $k$ on KNN accuracy,
which shows that as $k$ increases, the accuracy of the classifier decreases;
in fact, $k=1$ has nearly the same accuracy as the top accuracy, which shows
that looking at the nearest neighbor is equally a good choice for determining
classifications. A biological underpinning for our results may also be that
the protein families we attempted to classify the samples into may be too
similar. These proteins were chosen since they were the most frequent in the
Protein Data Bank, so it is feasible that there exists underlying relations
between the families that skew the performance of the models. For example, if
some of the 10 protein families are themselves evolutionarily or functionally
related, then it makes sense that the multinomial regression model outputs
guesses since it is impossible to distinguish between classes. Comparing the
two deep learning models is difficult. This is because there is no intuitive
reasons for the difference in results between the two; that is, what gains are
made in inference performance are lost in interpretability of the results.
What can be said, however, is that it may be possible that both models overfit
the protein sequence data. This is suggested by the near immediate convergence
of the models to their best validation accuracy scores within the first five
epochs of around 0.80. Yet, during training, the CNN averaged an accuracy of
0.95 while the LSTM averaged an accuracy of 0.94. This may indicate that the
models overfit the protein sequence data with their millions of parameters and
thus is why we see this immediate convergence. To combat this, future work
should be done on regularization of RNNs and CNNs on bioinformatics data, as
well as on the interpretability of these deep learning models such that the
inference procedure of these "black-box" models can be better understood: we
know not the underlying biological mechanisms within the sequence that allow
for these classifications, which make inference from deep learning models,
however accurate, not useful. We hypothesize that the CNN models perform
better than the RNN models on bioinformatics data because of the varying
degree and flexibility of the kernels, in their size, stride, padding, and
pooling techniques. From this flexibility, CNNs are able to learn short-term
and long-term dependencies across the sequence that RNN models generally
struggle with, including LSTMs. The introduction of transformers in recent
years seem like an interesting improvement to RNNs that may level the playing
field. For the machine learning models, we would like to examine the effects
of data synthesis and random over-sampling of the minority classes. We
hypothesize that this will greatly boost the performance of the model since a
class-balanced dataset may prevent guessing of the majority class. We would
also like to better understand how the data is collected, what machinery is
used, and how precise and specific these measurements are.
In summary, we have presented an analysis of several deep learning and
classical machine learning models for protein classification on Protein Data
Bank Data and shown that deep learning models, in particular convolutional
neural networks, are superior over classical machine learning techniques due
to their added complexity and ability to understand sequential information on
PDB data. Despite the power of these models, constraints of interpretability
leave much to be desired in the way of meaningful classifications, a possible
direction for future work.

## Acknowledgements

This work was done as part of graduate coursework for CSCI 2820: Algorithmic
Foundations of Computational Biology, under the instruction of distinguished
Professor Sorin Istrail. I would like to thank him for an exceptional semester
and appreciate the feedback from fellow students during the project’s
development. I look forward to learning more under his guidance in the future.

## References

  * [1] Dongardive, Jyotshna, and Siby Abraham. “Protein Sequence Classification Based on N-Gram and K-Nearest Neighbor Algorithm.” Computational Intelligence in Data Mining—Volume 2, edited by Himansu Sekhar Behera and Durga Prasad Mohapatra, vol. 411, Springer India, 2016, pp. 163–71, doi:10.1007/978-81-322-2731-1_15.
  * [2] Strodthoff, Nils, et al. “UDSMProt: Universal Deep Sequence Models for Protein Classification.” Bioinformatics, edited by Yann Ponty, vol. 36, no. 8, Apr. 2020, pp. 2401–09, doi:10.1093/bioinformatics/btaa003.
  * [3] Czepiel, Scott A. Maximum Likelihood Estimation of Logistic Regression Models: Theory and Implementation. p. 23.
  * [4] Böhning, Dankmar. “Multinomial Logistic Regression Algorithm.” Annals of the Institute of Statistical Mathematics, vol. 44, no. 1, Mar. 1992, pp. 197–200, doi:10.1007/BF00048682.
  * [5] Royer, Clément W., et al. “A Newton-CG Algorithm with Complexity Guarantees for Smooth Unconstrained Optimization.” Mathematical Programming, vol. 180, no. 1, Mar. 2020, pp. 451–88, doi:10.1007/s10107-019-01362-7.
  * [6] Murphy, K. P. Machine Learning: A Probabilistic Perspective. MIT Press, 2012, https://books.google.com/books?id=RC43AgAAQBAJ.
  * [7] Jiang, Shengyi & Pang, Guansong & Wu, Meiling & Kuang, Limin. (2012). An Improved k-Nearest Neighbor Algorithm for Text Categorization. Expert Systems with Applications. 39. 1503-1509. 10.1016/j.eswa.2011.08.040.
  * [8] R. Duncan Luce. Individual Choice Behavior: A Theoretical Analysis. John Wiley & Sons, 1959.
  * [9] Jurafsky, Daniel & Martin, James. (2008). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition.
  * [10] Abu Alfeilat, Haneen Arafat, et al. “Effects of Distance Measure Choice on K-Nearest Neighbor Classifier Performance: A Review.” Big Data, vol. 7, no. 4, Dec. 2019, pp. 221–48, doi:10.1089/big.2018.0175.
  * [11] Geurts, Pierre, et al. “Extremely Randomized Trees.” Machine Learning, vol. 63, no. 1, Apr. 2006, pp. 3–42, doi:10.1007/s10994-006-6226-1.
  * [12] Jha, Anupama, Matthew R. Gazzara, and Yoseph Barash. "Integrative deep models for alternative splicing." Bioinformatics 33.14 (2017): i274-i282.
  * [13] Quang, Daniel, Yifei Chen, and Xiaohui Xie. "DANN: a deep learning approach for annotating the pathogenicity of genetic variants." Bioinformatics 31.5 (2015): 761-763.
  * [14] Liu, Feng, et al. "PEDLA: predicting enhancers with a deep learning-based algorithmic framework." Scientific reports 6.1 (2016): 1-14.
  * [15] Li, Yifeng, Wenqiang Shi, and Wyeth W. Wasserman. "Genome-wide prediction of cis-regulatory regions using supervised deep learning methods." BMC bioinformatics 19.1 (2018): 1-14.
  * [16] Angermueller, Christof, et al. "DeepCpG: accurate prediction of single-cell DNA methylation states using deep learning." Genome biology 18.1 (2017): 1-13.
  * [17] Quang, Daniel, and Xiaohui Xie. "DanQ: a hybrid convolutional and recurrent deep neural network for quantifying the function of DNA sequences." Nucleic acids research 44.11 (2016): e107-e107.
  * [18] Degroeve, Sven, et al. "Feature subset selection for splice site prediction." Bioinformatics 18.suppl_2 (2002): S75-S83.
  * [19] Bucher, P. “Weight matrix descriptions of four eukaryotic RNA polymerase II promoter elements derived from 502 unrelated promoter sequences.” Journal of molecular biology vol. 212,4 (1990): 563-78. doi:10.1016/0022-2836(90)90223-9
  * [20] Heintzman, N., Stuart, R., Hon, G. et al. Distinct and predictive chromatin signatures of transcriptional promoters and enhancers in the human genome. Nat Genet 39, 311–318 (2007).
  * [21] Segal, Eran et al. “A genomic code for nucleosome positioning.” Nature vol. 442,7104 (2006): 772-8. doi:10.1038/nature04979
  * [22] Ohler, U., Liao, Gc., Niemann, H. et al. Computational analysis of core promoters in the Drosophila genome. Genome Biol 3, research0087.1 (2002)
  * [23] Alipanahi, B., Delong, A., Weirauch, M. et al. Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning. Nat Biotechnol 33, 831–838 (2015).
  * [24] Kelley, David R., Jasper Snoek, and John L. Rinn. "Basset: learning the regulatory code of the accessible genome with deep convolutional neural networks." Genome research 26.7 (2016): 990-999.
  * [25] Zhou, J., Troyanskaya, O. Predicting effects of noncoding variants with deep learning–based sequence model. Nat Methods 12, 931–934 (2015).
  * [26] Ritambhara Singh, Jack Lanchantin, Gabriel Robins, Yanjun Qi, DeepChrome: deep-learning for predicting gene expression from histone modifications, Bioinformatics, Volume 32, Issue 17, 1 September 2016.
  * [27] Qi, Yanjun et al. “A unified multitask architecture for predicting local protein properties.” PloS one vol. 7,3 (2012): e32235. doi:10.1371/journal.pone.0032235
  * [28] Srivastava, Nitish & Hinton, Geoffrey & Krizhevsky, Alex & Sutskever, Ilya & Salakhutdinov, Ruslan. (2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research. 15. 1929-1958.
  * [29] Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed Representations of Words and Phrases and their Compositionality. In C. J. Burges, L. Bottou, M. Welling, Z. Ghahramani, & K. Q. Weinberger (Eds.), Advances in Neural Information Processing Systems (Vol. 26). Curran Associates, Inc.

## Supplementary Data

Figure 7: Figure 9: Multinomial Regression Confusion Matrix


# Time to Take Emoji Seriously: They Vastly Improve Casual Conversational
Models

Pieter Delobelle and Bettina Berendt
KU Leuven, Department of Computer Science
Celestijnenlaan 200A box 2402, 3000 Leuven, Belgium
{pieter.delobelle<EMAIL_ADDRESS>
###### Abstract

Graphical emoji are ubiquitous in modern-day online conversations. So is a
single thumbs-up emoji able to signify an agreement, without any words. We
argue that the current state-of-the-art systems are ill-equipped to correctly
interpret these emoji, especially in a conversational context. However, in a
casual context, the benefits might be high: a better understanding of users’
utterances and more natural, emoji-rich responses.

With this in mind, we modify BERT to fully support emoji, both from the
Unicode Standard and custom emoji. This modified BERT is then trained on a
corpus of question-answer (QA) tuples with a high number of emoji, where we’re
able to increase the _1-of-100 accuracy_ from 12.7% for the current state-of-
the-art to 17.8% for our model with emoji support.

††footnotetext: Copyright © 2019 for this paper by its authors. Use permitted
under Creative Commons License Attribution 4.0 International (CC BY 4.0).

##  1 Introduction

The prevalent use of emoji—and their text-based precursors—is mostly
unaddressed in current natural language processing (NLP) tasks. The support of
the Unicode Standard [31] for emoji characters in 2010 ushered in a wide-
spread, international adoption of these graphical elements in casual contexts.
Interpreting the meaning of these characters has been challenging however,
since they take on multiple semantic roles [24].

Whether or not emoji are used depends on the context of a text or
conversation, with more formal settings generally being less tolerating. So is
the popular aligned corpus Europarl [16] naturally devoid of emoji. Technical
limitations, like no Unicode support, also limit its use. This in turn affects
commonly used corpora, tokenizers, and pre-trained networks.

Take for example the Ubuntu Dialog Corpus by Lowe et al. [20], a commonly used
corpus for multi-turn systems. This dataset was collected from an Internet
Relay Chat (IRC) room casually discussing the operating system Ubuntu. IRC
nodes usually support the ASCII text encoding, so there’s no support for
graphical emoji. However, in the 7,189,051 utterances, there are only 9946
happy emoticons (i.e. :-) and the cruelly denosed :) version) and 2125 sad
emoticons.

Word embeddings are also handling emoji poorly: Word2vec [22] with the
commonly used pre-trained Google News vectors doesn’t support the graphical
emoji at all and vectors for textual emoticons are inconsistent. As another
example with contextualized word embeddings, there are also no emoji or
textual emoticons in the vocabulary list of BERT [6] by default and support
for emoji is only recently added to the tokenizer. The same is true for GPT-2
[26]. As all downstream systems, ranging from multilingual résumé parsing to
fallacy detection [5], rely on the completeness of these embeddings, this lack
of emoji support can affect the performance of some of these systems.

Another challenge is that emoji usage isn’t static. Think of shifting
conventions, different cultures, and newly added emoji to the Unicode list.
Several applications also use their own custom emoji, like chat application
Slack and streaming service Twitch. This becomes an issue for methods that
leverage the Unicode description [7] or that rely on manual annotations [32].

Our contribution with this paper is two-fold: firstly, we argue that the
current use—or rather non-existing use—of emoji in the tokenizing, training,
and the datasets themselves is insufficient. Secondly, we attempt to quantify
the significance of incorporating emoji-based features by presenting a fine-
tuned model. We then compare this model to a baseline, but without special
attention to emoji.

Section 2 will start with an overview of work on emoji representations, emoji-
based models and analysis of emoji usage. A brief introduction in
conversational systems will also be given. Section 3 will then look into
popular datasets with and without emoji and then introduce the dataset we
used.

Our model will then be discussed in Section 4, including the tokenization in
Subsection 4.1, training setup in Subsection 4.2 and evaluation in Subsection
4.3. This brings us to the results of our experiment, which is discussed in
Section 5 and finally our conclusion and future work are presented in Section
6.

##  2 Related work

Inspired by the work on word representations, Eisner et al. [7] presented
Emoji2vec. This system generates a vector representation that’s even
compatible with the Word2vec representations, so they can be used together.
This compatibility makes it easy to quickly incorporate Emoji2vec in existing
systems that use Word2vec.

The main drawback is that the emoji representations are trained on the Unicode
descriptions. As a consequence, the representations only capture a limited
meaning and do not account for shifting or incorrect use of emoji in the real
world. For example, a peach emoji could be considered a double entendre, due
to the resemblance to a woman’s posterior. This is of course mentioned nowhere
in the Unicode description. Which shows that the meaning of an emoji is how
users interpret it, so also accidental incorrect use can cause issues [23].

In spirit, Felbo et al. [8] is similar to our work. Their system, DeepMoji,
illustrates the importance of emoji for sentiment, emotion, and sarcasm
classification. For these tasks, they used a dataset of 1246 million tweets
containing at least one emoji. However, the authors use the emoji in those
tweets not for the DeepMoji model input, but as an target label. With a
slightly better agreement score than humans on the sentiment task, this
supports our hypothesis that emoji carry the overall meaning of an utterance.

Barbieri et al. [2] focus on a predicting one emoji based on the textual
content. Interestingly, they looked into both English and Spanish tweets and
compared a range of systems for a shared task at SemEval 2018: Multilingual
Emoji Prediction. This shared task shows that emoji are getting more
attention, but how their task is set up also highlights the current lack of
high quality datasets with emoji.

The same shared task was tackled by [4] and a year later by Huang et al. [13],
which made use of a pre-processor and tokenizer from Baziotis et al. [3]. This
tokenizer replaces some emoji and emoticons by tokens related to their
meaning. So is \o/ replaced with <happy>. Naturally, this approach suffers
from the same issues as described before. And even though it’s really useful
to have some basic, out-of-the-box support for emoticons thanks to this work,
we think that this strategy is too reducing to capture subtle nuances.

An analysis on the use of emoji on a global scale is done by Ljubešić and
Fišer [19]. For this, the authors used geo-tagged tweets, which also allowed
them to correlate the popularity of certain emoji with development indicators.
This shows that the information encoded by emoji—and of course the
accompanying tweet—is not limited to sentiment or emotion. Also Ai et al. [1]
analyze the uses of emoji on social networks. Their approach consists of
finding information networks between emoji and English words with LINE [30].

An interesting aspect of emoji usage is analyzed by Robertson et al. [28]. In
this work, the correlation between the use of Fitzpatrick skin tone [9]
modifiers and the perceived skin tone of the user. This research shows that
users are inclined to use representing emoji for themselves. Robertson et al.
[28] reported that no negative sentiment was associated with specific skin
tone modifiers.

Table 1: Overview of the support for both graphical emoji and textual emoticons for common corpora. No support is indicated with an em bar (—). Otherwise, we provide the fraction of utterances with at least one emoji or emoticon. Dataset | Size |  Language |  Emoji |  Emoticons
---|---|---|---|---
Ubuntu [20] | 7,189,051 | EN | — | 0.17%
Amazon QA [33] | 1,396,896 | EN | — | 0.14%
OpenSubtitles [18] | 316,891,717 | EN | — | —
ConvAI2 Persona-Chat [36] | 47,234 | EN | — | 1.25%
Twitter customer support [12] | 2,000 | EN | 8.75% | 7.50%

#### Conversational AI systems

The research on conversational AI has been focussing on various aspects,
including building high-quality datasets [20, 17, 18, 36, 11, 27], adding
customizable personalities [36, 37, 21] or conjoining the efforts with regard
to different datasets, models and evaluation practices [11]. With these
combined efforts, businesses and the general public quickly began developing
ambitious use-cases, like customer support agents on social networks.

The proposed models in this field are diverse and largely depending on how the
problem is formulated. When considering free-form responses, generative models
like GPT [25], GPT-2 [26] or seq2seq [29] are appropriate. When the
conversational task is modeled as a response selection task to pick the
correct response out of $N$ candidates [10, 11, 35], this can be a language
model like BERT [6] with a dedicated head.

##  3 Emoji-rich datasets are hard to find

Emoji are commonly used in casual settings, like on social media or in casual
conversations. In conversations—as opposed to relatively context-free social
media posts—an emoji alone can be an utterance by itself. And with a direct
impact for some applications, like customer support, we focus on
conversational datasets. We hope the conversational community has the most
direct benefit from these emoji-enabled models. Of course, the conclusions
we’ll draw don’t have to be limited to this field.

Table 1 gives an overview of frequently used and interesting conversational
datasets. The lacuna of emoji-rich reference datasets was already mentioned in
Section 1 and is in our opinion one of the factors that emoji remain fairly
underutilized.

For our models, we’ll use a customer support dataset with a relatively high
usage of emoji. The dataset contains 2000 tuples collected by Hori and Hori
[12] that are sourced from Twitter. They provide conversations, which consist
of at least one question and one free-form answer. Some conversations are
longer, in this case we ignored the previous context and only looked at the
last tuple. This dataset illustrates that even when contacting companies,
Twitter users keep using emoji relatively often, 8.75% of all utterances.

The tweets were filtered on hyper links and personal identifiers, but Unicode
emoji characters were preserved. As emoji are frequently used on Twitter, this
resulted in a dataset with 170 of the 2000 tuples containing at least one
emoji character.

Figure 1: Illustration of the use of BERT for a response selection task.

##  4 Fine-tuning BERT with emoji support

We continue training of a multilingual BERT model [6] with new tokens for
emoji and fine-tune this model and a baseline on the dataset discussed in
Section 4. This approach is explained in Subsection 4.1 and the training
itself is discussed in Subsection 4.2. At last, the evaluation is then
discussed in Subsection 4.3.

###  4.1 Tokenizing emoji

We add new tokens to the BERT tokenizer for 2740 emoji from the Unicode Full
Emoji List [31, v12.0], as well as some aliases (in the form of :happy: as is
a common notation for emoji). In total, 3627 emoji tokens are added to the
vocabulary.

We converted all UTF-8 encoded emoji to a textual alias for two reasons.
First, this mitigates potential issues with text encodings that could drop the
emoji. Second, this is also a common notation format for custom emoji, so we
have one uniform token format. Aside from this attention to emoji, we use
WordPiece embeddings [34] in the same manner as Devlin et al. [6].

Figure 2: Illustration the tokenization process for BERT. The illustrated
input is an actual sample from the test set. Notice the emoji is replaced by a
descriptive token.

###  4.2 Training and fine-tuning

We start from 12-headed multilingual BERT (bert-base-multilingual-cased),
which has 110M parameters. For the model with emoji support, the number of
tokens is increased, so new vectors are appended at the end of the embeddings
matrix. We then continue training on the language modeling task. We use the
default configuration as is also used by Devlin et al. [6] where randomly
selected tokens are replaced by:

  * •

a mask token: 80% chance,

  * •

another random word: 10% chance,

  * •

the original word: 10% chance.

This model is trained for 100 epochs with the Adam [15] optimizer. The
learning rate is set to the commonly used $lr=5\cdot 10^{-5}$ and
$\epsilon=10^{-8}$. No hyper-parameter tuning was done, as the results are
acceptable on their own and are sufficient to allow conclusions for this
paper. The loss is cross entropy [14].

We then fine-tune both models, with and without emoji tokenization, on the
sentence prediction task with a training set of 70%. We again use the Adam
optimizer with the same settings and with binary cross entropy. In this case,
the training was limited to 10 epochs. To mitigate the need for weighting and
other class imbalance issues, we trained with pairs of positive and negative
candidates. This is in contract to the evaluation, where 99 negative
candidates are used. However, since each candidate is considered on its own
merit during evaluation, this discrepancy won’t affect the performance.

For the formulation of the fine-tuning task, we use the same approach as
Devlin et al. [6]. The first input sentence is joined with the second
sentence, separated by a special [SEP] token, as can be seen in Figure 2. The
model, with a specialized head for _next sentence prediction_ , then outputs a
correlation score.

###  4.3 evaluation metrics

Finally, our model is compared against the pre-trained version of BERT without
special emoji tokens. We evaluate both this baseline and our model as a
response selection task. In this case, the system has to select the most
appropriate response out $N=100$ candidates. This is a more restricted
problem, where the _1-of-100 accuracy_ [11] is a popular evaluation metric.

Note that 1-in-100 accuracy gives a summary of the model performance for a
particular dataset. Since not all 99 negative responses are necessarily bad
choices, the resulting score is in part dependent on the prior distribution of
a dataset. For example, Henderson et al. [11] compares models for three
datasets, where the best performing model has a score of 30.6 for
OpenSubtitles [18] and 84.2 for AmazonQA [33].

Aside from the 1-of-100 accuracy, we also present the mean rank of the correct
response. Since the Twitter dataset is focussed on customer service, the
correct response is sometimes similar to others. The mean rank, also out of
$N=100$, can differentiate whether or not the model is still selecting good
responses. For each input sentence, a rank of 1 means the positive response is
ranked highest and is thus correctly selected and a rank of $N$ signifies the
positive response was—incorrectly—considered the worst-matching candidate.

##  5 Emoji provide additional context to response selection models

After training of the language model with additional tokens for all Unicode
emoji, we achieved a final perplexity of 2.0281. For comparison, the BERT
model with 16 heads achieved a perplexity of 3.23 [6], but this is on a
general dataset.

For the sentence prediction task, Table 2 shows the results of the baseline
and our model with additional emoji tokens. For each of the 600 utterance
pairs of the held-out test set, we added 99 randomly selected negative
candidates, as described in Subsection 4.3. The _1-out-of-100 accuracy_
measures how often the true candidate was correctly selected and the _mean
rank_ gives an indication of how the model performs if it fails to correctly
select the positive candidate.

Table 2: 1-of-100 accuracy and mean ranking out of 100 candidates on the test
set of 600 utterance tuples.

Model | 1-of-100 ACC | Mean rank
---|---|---
bert-base-multilingual-cased | 12.7% | 33
bert-base-multilingual-cased with emoji tokens | 17.8% | 31

The baseline correctly picks 12.7% of all candidate responses, out of 100.
Given that the dataset is focussed on support questions and multiple responses
are likely to be relevant, this baseline already performs admirable. For
reference, a BERT model on the OpenSubtitles dataset [18] achieves a _1-of-100
accuracy_ between 12.2% and 17.5%, depending on the model size [11].

Our model improves on this baseline with a _1-of-100 accuracy_ of 17.8%. The
mean rank remains almost the same. This indicates that the emoji tokens do
help with with picking the correct response, but don’t really aide when
selecting alternative suitable candidates. One possible explanation is that
when emoji are used (this is the case for 8.75% of all utterances), including
those tokens helps matching those based on those emoji and their meaning. When
there are no emoji present, our model might be just as clueless as the
baseline.

##  6 Conclusion and future work

In this paper we discussed the current state of emoji usage for conversational
systems, which mainly lacks large baseline datasets. When looking at public
datasets, conversational AI makers have to choose between dataset size and
emoji support, with some datasets at least containing a few textual emoticons.
We argued that this duality results in systems that fail to capture some
information encoded in those emoji and in turn fail to respond adequately.

Based on this premise, we investigated how a response selection system based
on BERT can be modified to support emoji. We proposed a format and
tokenization method that’s indifferent to current Unicode specifications, and
thus also works for datasets containing custom emoji.

Evaluation of this emoji-aware system increased the _1-of-100 accuracy_ from
12.7% for the baseline to 17.8%. Thus showing that supporting emoji correctly
can help increasing performance for more casual systems, without having to
rely on labeling or external descriptions for those emoji.

However, the lack of high-quality, general datasets with emoji limits our
conversational model. Working towards larger casual conversational datasets
would help both for our model, and for the conversational NLP community in
general.

We investigated the impact of emoji for conversational models and one could
argue that these conclusions—or even the BERT model—can be generalized. We
didn’t investigate whether other tasks also benefited from our fine-tuned BERT
model with the additional emoji tokens.

During evaluation, we also observed utterances with only emoji characters.
Even with our model that supports emoji, it could still be difficult to
extract information like the subject of a conversation. Some of these
utterances—but not all—were part of a larger conversation, so an interesting
question could be how additional information affects the model.

## Acknowledgements

This work was supported by the Research Foundation - Flanders under EOS No.
30992574.

## References

  * Ai et al. [2017] Wei Ai, Xuan Lu, Xuanzhe Liu, Ning Wang, Gang Huang, and Qiaozhu Mei.  Untangling emoji popularity through semantic embeddings.  In _Eleventh International AAAI Conference on Web and Social Media_ , 2017.
  * Barbieri et al. [2016] Francesco Barbieri, Francesco Ronzano, and Horacio Saggion.  What does this Emoji Mean? A Vector Space Skip-Gram Model for Twitter Emojis.  In _LREC_ , 2016.
  * Baziotis et al. [2017] Christos Baziotis, Nikos Pelekis, and Christos Doulkeridis.  DataStories at SemEval-2017 Task 4: Deep LSTM with Attention for Message-level and Topic-based Sentiment Analysis.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 747–754, Vancouver, Canada, August 2017. Association for Computational Linguistics.  doi: 10.18653/v1/S17-2126.
  * Chen et al. [2018] Jing Chen, Dechuan Yang, Xilian Li, Wei Chen, and Tengjiao Wang.  Peperomia at SemEval-2018 Task 2: Vector Similarity Based Approach for Emoji Prediction.  In _Proceedings of The 12th International Workshop on Semantic Evaluation_ , pages 428–432, New Orleans, Louisiana, June 2018\. Association for Computational Linguistics.  doi: 10.18653/v1/S18-1067.
  * Delobelle et al. [2019] Pieter Delobelle, Murilo Cunha, Eric Massip Cano, Jeroen Peperkamp, and Bettina Berendt.  Computational Ad Hominem Detection.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics: Student Research Workshop_ , pages 203–209. Association for Computational Linguistics, July 2019.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, June 2019.  doi: 10.18653/v1/N19-1423.
  * Eisner et al. [2016] Ben Eisner, Tim Rocktäschel, Isabelle Augenstein, Matko Bošnjak, and Sebastian Riedel.  Emoji2vec: Learning Emoji Representations from their Description.  In _Proceedings of The Fourth International Workshop on Natural Language Processing for Social Media_ , pages 48–54, Austin, TX, USA, November 2016. Association for Computational Linguistics.  doi: 10.18653/v1/W16-6208.
  * Felbo et al. [2017] Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann.  Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1615–1625, Copenhagen, Denmark, 2017. Association for Computational Linguistics.  doi: 10.18653/v1/D17-1169.
  * Fitzpatrick [1988] Thomas B. Fitzpatrick.  The Validity and Practicality of Sun-Reactive Skin Types I Through VI.  _Archives of Dermatology_ , 124(6):869–871, June 1988.  ISSN 0003-987X.  doi: 10.1001/archderm.1988.01670060015008.
  * Henderson et al. [2017] Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-hsuan Sung, Laszlo Lukacs, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil.  Efficient Natural Language Response Suggestion for Smart Reply.  _arXiv:1705.00652 [cs]_ , May 2017.
  * Henderson et al. [2019] Matthew Henderson, Paweł Budzianowski, Iñigo Casanueva, Sam Coope, Daniela Gerz, Girish Kumar, Nikola Mrkšić, Georgios Spithourakis, Pei-Hao Su, Ivan Vulić, and Tsung-Hsien Wen.  A Repository of Conversational Datasets.  _arXiv:1904.06472 [cs]_ , April 2019.
  * Hori and Hori [2017] Chiori Hori and Takaaki Hori.  End-to-end Conversation Modeling Track in DSTC6.  _arXiv:1706.07440 [cs]_ , June 2017.
  * Huang et al. [2019] Chenyang Huang, Amine Trabelsi, and Osmar R. Zaïane.  ANA at SemEval-2019 Task 3: Contextual Emotion detection in Conversations through hierarchical LSTMs and BERT.  _arXiv:1904.00132 [cs]_ , March 2019.
  * Janocha and Czarnecki [2017] Katarzyna Janocha and Wojciech Marian Czarnecki.  On Loss Functions for Deep Neural Networks in Classification.  _arXiv:1702.05659 [cs]_ , February 2017.
  * Kingma and Ba [2014] Diederik P. Kingma and Jimmy Ba.  Adam: A Method for Stochastic Optimization.  _arXiv:1412.6980 [cs]_ , December 2014.
  * Koehn [2012] Philipp Koehn.  Europarl: A Parallel Corpus for Statistical Machine Translation.  page 8, 2012.
  * Kummerfeld et al. [2019] Jonathan K. Kummerfeld, Sai R. Gouravajhala, Joseph J. Peper, Vignesh Athreya, Chulaka Gunasekara, Jatin Ganhotra, Siva Sankalp Patel, Lazaros C Polymenakos, and Walter Lasecki.  A Large-Scale Corpus for Conversation Disentanglement.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3846–3856, Florence, Italy, July 2019. Association for Computational Linguistics.
  * Lison and Tiedemann [2016] Pierre Lison and Jorg Tiedemann.  OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles.  page 7, 2016.
  * Ljubešić and Fišer [2016] Nikola Ljubešić and Darja Fišer.  A Global Analysis of Emoji Usage.  In _Proceedings of the 10th Web as Corpus Workshop_ , pages 82–89, Berlin, August 2016. Association for Computational Linguistics.  doi: 10.18653/v1/W16-2610.
  * Lowe et al. [2015] Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau.  The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems.  _arXiv:1506.08909 [cs]_ , June 2015.
  * Mazaré et al. [2018] Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes.  Training Millions of Personalized Dialogue Agents.  _arXiv:1809.01984 [cs]_ , September 2018.
  * Mikolov et al. [2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_ , 2013.
  * Miller et al. [2017] Hannah Miller, Daniel Kluver, Jacob Thebault-Spieker, Loren Terveen, and Brent Hecht.  Understanding Emoji Ambiguity in Context: The Role of Text in Emoji-Related Miscommunication.  _Proceedings of the Eleventh International AAAI Conference on Web and Social Media_ , page 10, 2017.
  * Na’aman et al. [2017] Noa Na’aman, Hannah Provenza, and Orion Montoya.  Varying Linguistic Purposes of Emoji in (Twitter) Context.  In _Proceedings of ACL 2017, Student Research Workshop_ , pages 136–141, Vancouver, Canada, July 2017. Association for Computational Linguistics.
  * Radford et al. [2018] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving Language Understanding by Generative Pre-Training.  page 12, June 2018.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1(8), 2019.
  * Reddy et al. [2019] Siva Reddy, Danqi Chen, and Christopher D. Manning.  CoQA: A Conversational Question Answering Challenge.  _Transactions of the Association for Computational Linguistics_ , 7:249–266, March 2019.  doi: 10.1162/tacl˙a˙00266.
  * Robertson et al. [2018] Alexander Robertson, Walid Magdy, and Sharon Goldwater.  Self-representation on twitter using emoji skin color modifiers.  In _Twelfth International AAAI Conference on Web and Social Media_ , 2018.
  * Sutskever et al. [2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le.  Sequence to Sequence Learning with Neural Networks.  _arXiv:1409.3215 [cs]_ , September 2014.
  * Tang et al. [2015] Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei.  Line: Large-scale information network embedding.  In _Proceedings of the 24th International Conference on World Wide Web_ , pages 1067–1077. International World Wide Web Conferences Steering Committee, 2015.
  * The Unicode Consortium [2010] The Unicode Consortium.  The Unicode Standard, 2010.
  * Tigwell and Flatla [2016] Garreth W. Tigwell and David R. Flatla.  Oh that’s what you meant!: Reducing emoji misunderstanding.  In _Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct - MobileHCI ’16_ , pages 859–866, Florence, Italy, 2016. ACM Press.  ISBN 978-1-4503-4413-5.  doi: 10.1145/2957265.2961844.
  * Wan and McAuley [2016] Mengting Wan and Julian McAuley.  Modeling Ambiguity, Subjectivity, and Diverging Viewpoints in Opinion Question Answering Systems.  In _2016 IEEE 16th International Conference on Data Mining (ICDM)_ , pages 489–498, Barcelona, Spain, December 2016. IEEE.  ISBN 978-1-5090-5473-2.  doi: 10.1109/ICDM.2016.0060.
  * Wu et al. [2016] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean.  Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation.  _arXiv:1609.08144 [cs]_ , September 2016.
  * Yu et al. [2016] Zhou Yu, Ziyu Xu, Alan W Black, and Alexander Rudnicky.  Strategy and policy learning for non-task-oriented conversational systems.  In _Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 404–412, Los Angeles, September 2016. Association for Computational Linguistics.  doi: 10.18653/v1/W16-3649.
  * Zhang et al. [2018a] Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston.  Personalizing Dialogue Agents: I have a dog, do you have pets too?  _arXiv:1801.07243 [cs]_ , January 2018a.
  * Zhang et al. [2018b] Zhuosheng Zhang, Jiangtong Li, Pengfei Zhu, Hai Zhao, and Gongshen Liu.  Modeling Multi-turn Conversation with Deep Utterance Aggregation.  _arXiv:1806.09102 [cs]_ , June 2018b.