# Towards a Robust Out-of-the-box Neural Network Model for Genomic Data

Zhaoyi Zhang<EMAIL_ADDRESS>
Songyang Cheng∗<EMAIL_ADDRESS>
Department of Computer Science
University of Wisconsin-Madison
Madison, WI 53715, USA Claudia Solís-Lemus<EMAIL_ADDRESS>
Wisconsin Institute for Discovery
and Department of Plant Pathology
University of Wisconsin-Madison
Madison, WI 53715, USA Joint first authors with equal contribution and
randomly chosen order (reproducible script in Supplementary
Material)Corresponding author

###### Abstract

The accurate prediction of biological features from genomic data is paramount
for precision medicine, sustainable agriculture and climate change research.
For decades, neural network models have been widely popular in fields like
computer vision, astrophysics and targeted marketing given their prediction
accuracy and their robust performance under big data settings. Yet neural
network models have not made a successful transition into the medical and
biological world due to the ubiquitous characteristics of biological data such
as modest sample sizes, sparsity, and extreme heterogeneity. Here, we
investigate the robustness, generalization potential and prediction accuracy
of widely used convolutional neural network and natural language processing
models with a variety of heterogeneous genomic datasets. While the perspective
of a robust out-of-the-box neural network model is out of reach, we identify
certain model characteristics that translate well across datasets and could
serve as a baseline model for translational researchers.

Keywords: Generalization error, phenotype prediction, convolutional, natural
language processing

##  1 Introduction

Background. The ability to accurately predict phenotypes from genomic data is
one of the most coveted goals of modern-day medicine and biology. Examples
abound: from precision medicine where researchers want to predict a patient’s
disease susceptibility based on the genetic information (Ashley, 2015; Rost et
al., 2016; Katuwal and Chen, 2016; Krittanawong et al., 2017; Lee et al.,
2018; Ho et al., 2019) to prediction of antibiotic-resistant bacterial strains
based on the genomes of pathogenic microbes (Fjell et al., 2009; Coelho et
al., 2013; Pesesky et al., 2016; Kavvas et al., 2018; Li et al., 2018).
Examples extend beyond human health into soil and plant health such as the
prediction of crops yield (or plant disease susceptibility) based on soil
microbiome metagenomic data (Chang et al., 2017; Bokulich et al., 2018;
Carrieri et al., 2019) and the prediction of pesticide-resistant microbial
strains from plant bacterial pathogen genomes (Yang and Guo, 2017; Ip et al.,
2018; Maino et al., 2018; Duarte-Carvajalino et al., 2018). Our ability to
anticipate outcomes from data is at our scientific core when we face human
disease, environmental challenges, and climate change.

Naturally, biologists and medical researchers have turned to the machine-
learning community for answers given the great success of machine-learning
methods in a plethora of applications such as computer vision (Hjelmås and
Low, 2001; Egmont-Petersen et al., 2002) and astrophysics (Kucuk et al., 2017;
Jonas et al., 2018) to name a few.

However, the success of machine-learning methods on other fields has not been
easily translated to the biological realm (Chen et al., 2019; Ekins et al.,
2019; Teschendorff, 2019; Dacrema et al., 2019). Indeed, the complexity of
biological omics data has hampered the adoption of machine-learning models,
especially neural networks. Among the main challenges of genomic data in
neural network models are 1) smaller sample sizes compared to other fields and
2) heterogeneity of training samples and testing samples.

First, despite the advance in high-throughput sequencing technologies,
extracting whole genomes remains a time-consuming and expensive task when
sample sizes must be in the order of thousands. In addition, data privacy and
restrictions on data sharing in medical research restrict scientists’ ability
to combine multiple smaller datasets into larger ones suitable for neural
network modeling.

Second, more important than sample size, the weak link of deep learning in
biological applications is the assumption of homogeneity between training and
testing samples. This assumption is violated, for example, in microbial
datasets where laboratory samples (training data) and environmentally or
clinically collected samples (testing data) can be intricately heterogeneous.
This data heterogeneity can cause lack of robustness and generalization errors
in neural network models. Robustness is the key ingredient that is needed for
neural network models to translate into medical practice and into the
phenotype prediction in the agricultural or environmental field.

Main contributions. In literature, there are multiple examples of successfully
fit neural network models on biological or medical genomic data (Agarwal et
al., 2019; Nguyen et al., 2016; Zeng et al., 2016; Shadab et al., 2020;
Trabelsi et al., 2019). However, it remains uncertain whether the proposed
models could be translated to other similar datasets with comparable
performance. That is, we ask whether the neural network models proposed in
literature are robust across heterogeneous (but similar in nature) datasets.

In addition, we approach the existing neural network models with the mindset
of a biological or medical user. A biological researcher would see the neural
network model in an existing publication and then would try to apply a similar
model to their own dataset. First, we explore how easy it is to replicate the
analysis on existing publications. Second, by making incremental changes to
the model characteristics, we gauge the effect of each model component on the
overall performance.

We learn mainly three things: 1) in multiple instances, we are not able to
replicate the performance in existing publications either because data is not
available, code is not available, or code is corrupted, incomplete or not
well-documented; 2) most of the times the good performance of existing models
does not translate to alternative datasets, yet we do encounter some model
characteristics (in particular, related to convolutional neural networks) that
are generally robust across datasets and that could serve as a potential
baseline model – albeit with modest performance – to start the prediction
process from a user perspective, and 3) we find that accurate prediction is a
balancing game between underfitting and overfitting, and that small changes in
the architecture can have unpredictable outcomes.

The quest for robust neural network models that could tackle the complexities
of biological data (and its intrinsic heterogeneity) is imminent. Neural
network models cannot be fully applicable in informed patient care, medical or
agricultural framework if they cannot guarantee some level of generalization
potential given that genomic data are not static but constantly evolving.

The difficulty of the prediction problem in biology or medicine is such that
it would be naive to believe that there will exist an out-of-the-box model
that will be fully transferable. Yet, from a user perspective, it is desirable
to know if there are certain model characteristics that perform modestly under
scrutiny from a variety of different datasets.

While we advise biological or medical users against using out-of-the-box
strategies, we conclude from our study that simple convolutional neural
networks (small number of layers) are relatively robust across genomics
datasets and generally not affected by the type of data encoding or the
sparsity level. Overfitting is an issue on more complex CNN models (as
expected), but it is relatively controlled via regularization schemes. We also
find that a general LSTM layer for embedding performed relatively well across
datasets and this performance was robust to batch sizes and choice of
optimizer. Finally, our work raises awareness to the importance of
reproducibility and replicability. As machine-learning scientists, it is
crucial to accompany our work with reproducible scripts that are relatively
easy to follow by the scientific community so that our findings have an impact
across fields, in particular, into the biological and medical community.

Structure of the paper. We focus on two widely used neural network models
found in literature for genomic data: convolutional neural networks (Section
2.2) and natural language processing (Section 2.3). For each of these models,
we explored the role of specific layers, robustness to alternative datasets,
the role of data encoding, the role of the optimizer, the role of batch size,
and the role of embedding size. We present our findings for convolutional
neural networks in Section 3.1 and for natural language processing in Section
3.2.

##  2 Methods

We focus on convolutional neural networks (Zeng et al., 2016; Nguyen et al.,
2016; Shadab et al., 2020; Trabelsi et al., 2019) and natural language
processing (Le and Mikolov, 2014; Agarwal et al., 2019; Dutta et al., 2018).

Whenever available, we first compare the performance of the models on the same
dataset used in the publication, and then, on alternative similar datasets. In
addition, we modify the model characteristics incrementally and test the
effect of the change on the model performance. More details on Section 2.2 for
convolutional neural networks and on Section 2.3 for natural language
processing.

###  2.1 Available Datasets

We focus on five datasets from the available ones in the papers under study
(Table 1):

Histone data. Nguyen et al. (2016) included 10 datasets about DNA sequences
wrapping around histone proteins. We will focus on the H3 occupancy from the
histone dataset that has 14965 sequences of length 500 bp. The H3 indicates
the histone type, and the dataset has two classes: the positive class are DNA
sequences that contains regions wrapping around histone proteins and the
negative class does not.

Splice data. Nguyen et al. (2016) included a splice dataset (also in the UCI
machine learning repository (Dua and Graff, 2017)). Splice junctions are
points on a DNA sequence at which superfluous DNA is removed during the
process of protein creation in higher organisms. This dataset has 3190
sequences of length 60 bp and are classified into three classes: exon/intron
boundaries (EI), intron/exon boundaries (IE), and non-splice (N).

Promoter data. Nguyen et al. (2016) included a promoter dataset (also in the
UCI machine learning repository (Dua and Graff, 2017)). A promoter is a
sequence of DNA to bind proteins that initiate transcription from downstream
DNA. This dataset has 106 sequences of length 57 bp and are classified into
two classes: promoters and non-promoters.

Motif discovery data. Zeng et al. (2016) included two ChIP-seq datasets: motif
discovery and motif occupancy. These datasets contain the labels of the
binding affinity of transcription factors to DNA sequence in 690 different
ChIP-seq experiments. We will only focus on a subset of 269100 sequences from
the motif discovery data (out of 20464149) of length 101.

ChIP-seq toy data. This is a subset of the ChIP-seq data provided by Zeng et
al. (2016) of 97718 sequences of length 101 bp. It is unclear if this subset
was taken from the motif discovery data or the motif occupancy as the authors
do not provide this information. However, given the sequence length, it
appears that this subset comes from the motif discovery data.

Dataset | Sample size | Sequence Length (bp) | Reference
---|---|---|---
Histone | 14965 | 500 | Nguyen et al. (2016)
Splice | 3190 | 60 | Nguyen et al. (2016)
Promoter | 106 | 58 | Nguyen et al. (2016)
Motif discovery | 269100 | 101 | Zeng et al. (2016)
ChIP-seq toy | 97718 | 101 | Zeng et al. (2016)
Table 1: Datasets used to test the neural network models.

###  2.2 Convolutional Neural Networks

We test the performance of four convolutional neural network (CNN) models
found in literature (Zeng et al., 2016; Nguyen et al., 2016; Shadab et al.,
2020; Trabelsi et al., 2019) that have been successful on genomic-based
prediction. We assess their performance on their own datasets (when available)
and on alternative similar datasets, as well as under incremental
modifications of the model characteristics such as data encoding, window size
and number of layers. See Table 2 for a summary of the performance tests and
models.

|  Model |  Own datasets |  Outside datasets |  Performance tests
---|---|---|---|---
CNN-Nguyen (Nguyen et al., 2016) |  Figure 1(a) |  histone, splice, promoter |  motif discovery |  number of layers, data robustness, dimension
CNN-Zeng (Zeng et al., 2016) |  Figure 1(c) |  motif discovery |  |  number of layers
DeepDBP (Shadab et al., 2020) |  Figure 1(d) |  |  motif discovery |  data robustness
DeepRAM (Trabelsi et al., 2019) |  Figures 1(e), 1(f), 1(g) |  |  motif discovery |  data encoding
Table 2: CNN models along with the datasets used and the performance tests

(a) Original CNN-Nguyen (Nguyen et al., 2016)

(b) Modified CNN-Nguyen (Nguyen et al., 2016)

(c) Original CNN-Zeng (2 layers), modified (3,4 layers) (Zeng et al., 2016)

(d) DeepDBP (Shadab et al., 2020)

(e) DeepRAM-CNN (Trabelsi et al., 2019)

(f) DeepRAM-RNN (Trabelsi et al., 2019)

(g) DeepRAM-CNN-RNN (Trabelsi et al., 2019)

Figure 1: Convolutional Neural Network models

####  2.2.1 CNN-Nguyen (Nguyen et al., 2016)

Datasets. We use the original datasets of the paper: histone, splice and
promoter (Nguyen et al., 2016). The histone dataset is split into 70% for
training, 15% for validation, and 15% for testing. The splice and promoter
datasets are relatively small, so they are only split into 75% for training
and 25% for testing.

Data encoding. As in Nguyen et al. (2016), we use a sliding window of fixed
size across the sequence. At each step, a kmer is read from the window and
added to the kmer sequence. For example, if a sequence looks like “ACTGG”,
then its 3-mers are [“ACT”, “CTG”, “TGG”]. The process is similar to how
ngrams are created from text. After that, one-hot encoding is applied to the
kmers. To also include to spatial information of the sequences, we concatenate
the encoding of k-mers within a fixed region size. As in Nguyen et al. (2016),
we choose a window size of 3 and a region of 2.

Neural network model. We implement the simple neural network in Nguyen et al.
(2016) as our baseline model (Figure 1(a)). The model contains two 2D
convolutional layers, each followed by a pooling layer, then the output of the
convolutional layers are connected to a fully connected layers. The fully
connected layer has a dropout rate of 0.5 to reduce the effect of overfitting.
Finally, we use a softmax layer to predict the labels of the input sequences.

Performance tests. We perform three tests with this model: 1) we replicate the
results in (Nguyen et al., 2016), 2) we compare the performance of the model
with a different dimension (1D) and increasing number of layers (Figure 1(b)),
and 3) we test the robustness of the model (with and without transfer
learning) when fitted to the alternative motif discovery dataset (Zeng et al.,
2016).

####  2.2.2 CNN-Zeng (Zeng et al., 2016)

Datasets. We used a subset of the original dataset: motif discovery data (Zeng
et al., 2016). The subset is further split into 48.7% for training, 2.6% for
validation, and 48.7% for testing.

Data encoding. We use the same encoding method as in the CNN-Nguyen model. The
sequences are converted into kmers, one-hot encoded, and then concatenated
with neighbouring kmers. Again, for consistency, we choose a window size of 3
and a region of 2.

Neural network model. We implement the neural network model in Zeng et al.
(2016) (Figure 1(c)) that contains two 2D convolutional layers with unknown
hyperparameters, so we also use a model that contains two 2D convolutional
layers, each followed by a batch-normalization and max-pooling layer. The
output of the convolutional layers is connected to a fully connected layer.
This layer has a dropout rate of 0.5 to prevent overfitting. Finally, a
softmax layer is used to predict the class of the input sequence.

Performance tests. To explore the effect of the number of layers, we add 2D
convolutional, batch-normalization, and max-pooling layers to the end of the
convolutional network (Figure 1(c)). The number of filters in convolutional
layer doubles each time we add a new set of these layers (e.g. 16 filters in
the first convolutional layer, 32 in the second, 64 in the third).

####  2.2.3 DeepDBP (Shadab et al., 2020)

Dataset. This paper did not provide any datasets, so we use the motif
discovery dataset (Zeng et al., 2016).

Data encoding. To give fair comparison between all CNN models we tested, we
use the same encoding method applied to the motif discovery dataset. The
sequences are converted into kmers, one-hot encoded, and then concatenated
with neighbouring kmers. Again, for consistency, we choose a window size of 3
and a region of 2.

Neural network model. The source code of this paper is not well structured and
contains many different models that are not properly documented, so we are
unable to identify the architecture of the DeepDBP model. However, we
implemented a model based on the paper description that contains a embedding
layer, a convolutional layer, max-pooling layer, followed by fully connected
layers and the output layer (Figure 1(d)).

Performance tests. We test the robustness of the proposed architecture to the
motif discovery dataset (Zeng et al., 2016) and see if the achieved prediction
accuracy is comparable to the one reported in the paper.

####  2.2.4 DeepRAM (Trabelsi et al., 2019)

Dataset. This paper did not provide any datasets, so we use the motif
discovery dataset (Zeng et al., 2016).

Data encoding. We experiment with two different ways of encoding the
sequences. One way is, as described before, to use one-hot encoding with word
size 3 and region size 2. The other way is to convert sequences into
overlapping kmers, and embed kmers into dense vectors using embedding layers.
Note that this embedding is different from the one used in Section 2.3 because
of the unit used for encoding. Here, we use the kmer as the unit for encoding
while in LSTM-AE, we use the nucleotide as the unit.

Neural network model. We implement the three models in Trabelsi et al. (2019):
1D convolutional neural networks (Figure 1(e)), recurrent neural networks
(Figure 1(f)), and a mixture of 1D convolutional and recurrent neural networks
(Figure 1(g)). For convolutional neural networks, we use two 1D convolutional
layers, with each followed by a max-pooling layer, and finally fully connected
layers and output layer. For recurrent neural networks, we use two Long Short-
Term Memory (LSTM) layers followed by fully connected layers and output layer.
For the hybrid neural networks, we use two 1D convolutional layers and two
Long short-term memory layers, and finally fully connected layers and output
layers. We note that the DeepRAM-RNN and the DeepRAM-CNN-RNN models are not
entirely CNN models and share many characteristics with the models presented
in Section 2.3. However, we present these models in this section given that
they are all part of the DeepRAM paper (Trabelsi et al., 2019) and we follow
the comparisons and analyses highlighted in this work.

Performance tests. We compare the changes in performance based on the data
encoding.

###  2.3 Natural Language Processing Neural Networks

Traditionally, genomic data is stored as a collection of long strings
comprised of the four nucleotides: A,C,G,T. It is thus intuitive to turn to
Natural Language Processing (NLP) theory for solid ways to embed the sequences
in a latent space. Furthermore, NLP methods naturally overcome one of the main
drawbacks of CNN models which is the sparsity of the input vectors.

Here, we focus on two widely used NLP tools: doc2vec (Le and Mikolov, 2014)
and Long Short Term Memory (LSTM) (Agarwal et al., 2019; Kimothi et al.,
2016). Both methods share the same objective: represent the input sequence
with a low dimensional dense vector yet the specifics differ as is explained
below.

We first clarify that the NLP methods are not performing prediction (as the
CNN models were in Section 2.2). Since the purpose of this work is to compare
the performance of neural network models on the prediction of phenotypes from
genomes, we need to add a neural network model to the NLP model that will
perform the prediction of labels (Figure 2(d)).

Table 3 presents a summary of the performance tests and models. In addition to
those tests, we compare the prediction accuracy of the three models and we
compare the auto-encoding potential of doc2vec and LSTM-AE.

|  Model |  Own datasets |  Outside datasets |  Performance tests
---|---|---|---|---
LSTM-layer |  Figure 2(a) |  |  ChIP-seq toy, splice |  optimizer
doc2vec+NN (Le and Mikolov, 2014) |  Figure 2(d) |  |  ChIP-seq toy, splice |  embedding size
LSTM-AE+NN (Agarwal et al., 2019) |  Figures 2(b) and 2(d) |  |  ChIP-seq toy, splice |  batch size
Table 3: NLP models along with the datasets used and the performance tests

(a) LSTM-layer

(b) LSTM-AE (encoder) (Agarwal et al., 2019)

(c) LSTM-AE (decoder) (Agarwal et al., 2019)

(d) Simple NN for prediction after LSTM-AE (encoder) and doc2vec

Figure 2: Natural Language Processing Neural Network models

####  2.3.1 LSTM-layer

Datasets. We use the ChIP-seq toy dataset which is a subset of the motif
discovery data from Zeng et al. (2016) which had already been separated into
training/validation/test sets (77174/1000/19544 sequences respectively). The
length of each sequence is 101. As described in the motif discovery data, the
sequences have binary classes and the class distribution in each set is fairly
even. In addition, we use the splice data in Nguyen et al. (2016) which we
split into training set, validation set, and test set with 2041, 511, and 638
samples respectively.

Data encoding. Each nucleotide is converted to a label number. For example,
[“A”, “C”, “G”, “T”], are encoded as [3, 2, 1, 0] in descending
lexicographical order. The LSTM-layer is crucial given the intractable growth
in dimension of the input vector. That is, a sequence containing 6000
nucleotides would be represented by a sequence of 6000 numbers.

Neural network model. We implement the neural network model (Figure 2(a)) that
contains, after the input layer, an embedding layer followed by an LSTM layer
with size of 30 for both datasets. There are four dense layers with size
decaying by a factor of 2 (128-64-32-16). There is one Dropout layer between
any two Dense layers with dropout rate of 0.2.

Performance tests. We compare the prediction accuracy of this model with the
other two NLP models: doc2vec+NN and LSTM-AE+NN. In addition, we study the
changes in performance when using different optimizers.

####  2.3.2 doc2vec+NN (Le and Mikolov, 2014)

Datasets. We use the same two datasets already described in LSTM-layer: ChIP-
seq toy (Zeng et al., 2016) and splice (Nguyen et al., 2016).

Data encoding. We encode the sequences based on 3-mers with window size of 1.
For example, for the “ACTGG” sequence, the 3-mers are [“ACT”, “CTG”, “TGG”].
We construct a dictionary with all the 3-mers in the training set. While it is
unlikely for 3-mers in the test set to not appear in the dictionary, we
categorize these instances as out-of-vocabulary (OOV) with a unique encoding.

Neural network model. The nature of the doc2vec sequence representation as a
semantic vector preserves similarity of sequences in terms of frequency and
location of ngrams. We apply the distributed memory mode (DV-PM) as in Le and
Mikolov (2014), and then, we use the simple fully connected neural network in
Figure 2(d) containing two dense layer with size shrinking by a factor of 2
with a dropout layer in between.

Performance tests. We compare the prediction accuracy of this model with the
other two NLP models: LSTM-layer and LSTM-AE+NN. In addition, we study the
effect of embedding size in the performance of the model. Finally, we compare
the encoding potential (ability to recover the original sequences) of doc2vec
and LSTM-AE.

####  2.3.3 LSTM-AE+NN (Agarwal et al., 2019)

Datasets. We use the same two datasets already described in LSTM-layer: ChIP-
seq toy (Zeng et al., 2016) and splice (Nguyen et al., 2016).

Data encoding. We use the same encoding as in the LSTM-layer model. Each
nucleotide is converted to a label number. For example, [“A”, “C”, “G”, “T”],
are encoded as [3, 2, 1, 0] in descending lexicographical order.

Neural network model. A LSTM autoencoder model (LSTM-AE) aims to represent a
sequence by a dense vector that can be converted back to the original
sequence. Indeed, LSTM-AE is comprised of two parts: an encoder network
(Figure 2(b)) that compresses the original sequence into a low dimensional
dense vector, and a decoder network (Figure 2(c)) that converts the vector
back to the original sequence. The encoder reads as input an encoded DNA
sequence and outputs a dense vector as the embedding for this sequence whose
length is a hyper parameter to tune. The decoder reads as input the dense
vector produced by the encoder and produces a reconstructed sequence. The
accuracy of the autoencoder is measured by comparing the reconstructed
sequence to the original sequence.

We implement a LSTM-AE following Agarwal et al. (2019) based on the
description in their publication given that no reproducible script was
available. The LSTM-AE model is trained to achieve maximum reconstruction
accuracy of the sequences. Then, since LSTM-AE is not performing
classification, we add a simple fully connected neural network (Figure 2(d))
containing two dense layer with size shrinking by a factor of 2 with a dropout
layer in between for the prediction of class labels. The size of the first
dense layer is adjusted, as a rule of thumb, to match 1 to 4 times the
embedding dimension. We denote this model as LSTM-AE+NN. We note that only the
weights corresponding to the simple fully connected neural network are
optimized for classification which is different to the LSTM-layer model whose
embedding indeed changes during training. We highlight that the LSTM-layer and
LSTM-AE models differ on how an embedding is evaluated. The embedding produced
by the LSTM-layer model aims to better classify the sequences into the right
category while the embedding produced by the LSTM-AE model aims to better
capture the sequence itself.

Performance tests. We compare the prediction accuracy of this model with the
other two NLP models: LSTM-layer and doc2vec+NN. In addition, we study the
effect of the batch size in the performance of the model. Finally, we compare
the encoding potential (ability to recover the original sequences) of LSTM-AE
and doc2vec.

##  3 Results

###  3.1 Convolutional Neural Networks

####  3.1.1 Replication of Results

First, DeepDBP (Shadab et al., 2020) did not provide data or reproducible
scripts of the analyses, and thus, we were unable to replicate the results.
DeepRAM (Trabelsi et al., 2019) does provide open-source software
(https://github.com/MedChaabane/deepRAM), but they did not provide details or
public links on the datasets used. Thus, we decided to use existing data
(Table 1) to test the models in both DeepDBP and DeepRAM.

The CNN-Nguyen model (Nguyen et al., 2016) shows good performance on the
histone data (78.2% training accuracy, 77.1% validation accuracy in Figure 3)
and there is little evidence of overfitting. The accuracy that we obtain is
slightly different to the one reported in the paper. Despite the lack of
evidence of overfitting, we add an L2 regularization which increases the loss
and decreases the accuracy (75.2% training accuracy, 76.2% validation accuracy
in Figure 3).

For splice and promoter datasets, an even simpler neural network model (only
one 2D convolutional layer) performs quite well (100% and 100% validation
accuracy respectively, figure in the Supplementary Material). The original
paper used a model with two 2D convolutional layers, and has got an accuracy
of 96.1% and 99.1% for splice and promoter dataset respectively.

For the CNN-Zeng model (Zeng et al., 2016), we observe clear underfitting to
their own motif discovery data (64.8% training accuracy, 64.1% validation
accuracy, Figure 5 first column). These results do not match the results in
the original paper, yet we are considering a subset of the original data which
could be the source of the difference in performance.

Figure 3: Learning dynamics for the original CNN-Nguyen model with two 2D
convolutional layers (Figure 1(a)) on the histone data. Color represents
training (blue) or validating (green) accuracy or loss. Even without
regularization, there is not any evidence of overfitting. With L2
regularization, the validation accuracy does not change much, but the overall
loss increases.

####  3.1.2 The Role of Convolutional Layers and Dimension.

We add one additional convolutional 1D layer to CNN-Nguyen (Nguyen et al.,
2016), and compare the performance with the original model. The rationale for
the added layer is that the sequences in the histone data are relatively long
(500 bp), so in order to shorten the input, we add a 1D convolutional layer
before the 2D convolutional layers to shorten the input into equal width and
height (e.g. $192\times 192\times n_{filters}$). We find that this model
suffers from severe overfitting (97.5% training accuracy, 79.5% validation
accuracy in Figure 4). Adding an L2 regularization reduces the effect of
overfitting (90.5% training accuracy, 78.4% validation accuracy in Figure 4),
but does not improves validation accuracy significantly.

Figure 4: Learning dynamics for modified CNN-Nguyen with one extra 1D
convolutional layer (Figure 1(b)) on the histone data. Color represents
training (blue) or validating (green) accuracy or loss. There is evidence of
overfitting without regularization which is somewhat alleviated by an L2
regularization.

Regarding the dimension, we compare the performance of the one 2D
convolutional model of CNN-Nguyen on the splice and promoters datasets
(already quite accurate with 93.9% and 100% validation accuracy, see Section
3.1.1) to a model with one 1D convolutional layer instead. We find that the
performance is comparable (96.1% and 100% validation accuracy respectively,
figure in the Supplementary Material) which gives us evidence that in this
case, the dimension is not playing an important role in prediction.

With the CNN-Zeng model Zeng et al. (2016), we test if more 2D layers improve
the prediction accuracy given that the original model with two convolutional
layers suffers from underfitting in their own motif discovery data (64.8%
training accuracy, 64.1% validation accuracy). When adding a new layer (total
of three convolutional layers), the accuracy is improved (73.7% training
accuracy, 69.7% validation accuracy in Figure 5).

Figure 5: Learning dynamics for the CNN-Zeng models with two (the original),
three, or four 2D convolutional layers (Figure 1(c)) in the motif discovery
data. Color represents training (blue) or validating (green) accuracy or loss.
The model with only 2 layers (left) exhibits underfitting. The model with 3
layers (middle) exhibits a certain degree of overfitting which can be argued
as acceptable. The model with 4 layers (right) exhibits clear overfitting. The
pattern is consistent with the knowledge that more complex models tend to
overfit.

The problem of overfitting emerges when we use four convolutional layers
(80.1% training accuracy, 68.2% validation accuracy in Figure 5). Therefore,
we apply L2 regularization on the CNN-Zeng model with four layers which indeed
reduces the effect of overfitting (69.0% training accuracy, 69.6% validation
accuracy in Figure 6), but it also limits the learning power of the model by
yielding a lower prediction accuracy overall. It appears that having 3
convolutional layers was the sweet spot between underfitting and overfitting
for this particular dataset.

Figure 6: Learning dynamics for the CNN-Zeng model with four 2D convolutional
layers (Figure 1(c)) in the motif discovery data. Color represents training
(blue) or validating (green) accuracy or loss. The model suffers from
overfitting which is improved via L2 regularization.

####  3.1.3 Robustness across Datasets

We test if the model in CNN-Nguyen (Nguyen et al., 2016) and the DeepDBP model
(Shadab et al., 2020) could work successfully on the motif discovery data from
Zeng et al. (2016).

When we train the CNN-Nguyen model Nguyen et al. (2016) on the motif discovery
data (Zeng et al., 2016), the validation accuracy is not great (59.3%), so we
try transfer learning by first training the model on the splice dataset
(validation accuracy 93.9%) and then training the pre-trained model on the
motif discovery data. The validation accuracy is improved, but not
dramatically (60.3%). We conclude that the model is not robust across datasets
even with transfer learning. One possible explanation for the lack of
improvement with transfer learning is that the pre-trained model is only
trained on a very small dataset (3190), and the new dataset is fairly large
(269100), so the model did not learn anything that is transferable.

Finally, we find that the DeepDBP model (Shadab et al., 2020) does not perform
very well on the motif discovery dataset (Zeng et al., 2016). The accuracy
does not change very significantly during the training time, fluctuating
around 50%.

####  3.1.4 The Role of Data Encoding

We use the architectures of DeepRAM in Trabelsi et al. (2019) to test
performance changes due to data encoding.

We find that all the combinations of the two encoding methods and three types
of models give reasonable performance (around 75% validation accuracy) and
rarely suffer from overfitting (Figure 7). In particular, the combination of
using one-hot encoding and 1D convolutional and recurrent neural network gives
the highest validation accuracy (77.8%). As a point of comparison, the
accuracy presented in original DeepRAM paper ranges from 83.6% to 99.4%

Figure 7: Learning dynamics for the DeepRAM models (Figures 1(e), 1(f), 1(g))
on the motif discovery data. Rows correspond to different data encodings
(embedding and one-hot encoding). Columns correspond to different
architectures (convolutional neural networks, convolutional plus recurrent
neural networks, and recurrent neural networks). Color represents training
(blue) or validating (green) accuracy. Plots for loss function are in the
Supplementary Material.

####  3.1.5 The Role of Sparsity in Overfitting

We wonder if the overfitting on CNN-Zeng (4 CNN layers) is related to sparsity
of genomic data. We train a model with four convolutional and pooling layers
on both the motif discovery dataset and the CIFAR-10 dataset (Krizhevsky et
al., ). We observe that loss curves show similar overfitting pattern (Figure
8), so it appears that the overfitting is not related to the sparsity of the
one-hot encoded sequence data.

Figure 8: Learning dynamics for the four 2D convolutional layered model
trained on the CIFAR-10 data and motif discovery data. Color represents
training (blue) or validating (green) accuracy or loss. We perceive
overfitting on both data.

###  3.2 Natural Language Processing Neural Networks

####  3.2.1 Comparison of Classification Accuracy

Table 4 shows the classification accuracy of LSTM-layer (Figure 2(a)) and
LSTM-AE+NN (Figures 2(b) and 2(d)) models on the two datasets. LSTM-layer
outperforms LSTM-AE+NN in the ChIP-seq data, while LSTM-AE+NN outperforms
LSTM-layer on the splice data. The performance of doc2vec+NN is not shown
since the classification accuracy was never far above 50% for these datasets.
Figures for the learning dynamics of all models can be found in the
Supplementary Material.

Model | Dataset | Training | Validation | Testing
---|---|---|---|---
LSTM-layer | ChIP-seq toy | 92.64% | 91.20% | 90.62%
LSTM-AE+NN | ChIP-seq toy | 67.60% | 67.50% | 66.50%
LSTM-layer | splice | 90.98% | 85.52% | 87.77%
LSTM-AE+NN | splice | 91.50% | 89.80% | 91.10%
Table 4: Classification accuracy for the NLP models. The classification
accuracy of doc2vec+NN was always close to 50% for both datasets.

One conjecture for the poor performance of doc2vec is the short length of the
sequences (60 in the splice data and 101 in the ChIP-seq data). We therefore
decide to test the performance of doc2vec on a dataset of 281 sequences of
length 4132 classified as Potyvirus and Picornavirus. This dataset belongs to
our collaborators and is soon to be published, but it remains unpublished at
the time of submission of this work (see Acknowledgements).

The doc2vec+NN model performs well on longer sequences with training
prediction accuracy of 97.32% and testing prediction accuracy of 96.49%. The
LSTM-layer model, however, performs worse on longer sequences with training
prediction accuracy of 56.42%/ and testing prediction accuracy of 56.14%
(LSTM-AE+NN has similar poor performance, results not shown).

To investigate the reasons for the poor performance of LSTM-layer (and LSTM-
AE+NN) on the longer sequences, we reduce the sequence length on the virus
dataset to 2449 bp based on three criteria: 1) using a phylogenetic-based
approach denoted MDL (Ané, 2011), 2) keeping the first 2449 nucleotides in the
sequence and 3) keeping the last 2449 nucleotides in the sequence. The number
2449 was determined by MDL.

Table 5 shows that doc2vec+NN outperforms LSTM-layer on all cases of reduced
sequence length, yet the performance of LSTM-layer does not appear to depend
exclusively on the sequence length. Indeed, LSTM-layer has its best
performance when keeping the first 2449 nucleotides of the sequence. This
pattern is not explained either by the percentage of missing nucleotides
(labeled “N”) in the sequences. Biologically, DNA sequences contain important
information about translation at the beginning of the sequence which could in
part explain why the performance is improved when keeping this region. We
conclude that doc2vec+NN seems to provide a better embedding for the accurate
classification for long sequences compared to LSTM-layer (or LSTM-AE+NN).

Model | Subset | Length | Missingness | Training | Testing
---|---|---|---|---|---
doc2vec+NN | complete | 4132 | 90.58% | 97.32% | 96.49%
LSTM-layer | complete | 4132 | 90.58% | 56.42% | 56.14%
LSTM-layer | MDL | 2449 | 84.82% | 62.01% | 59.65%
LSTM-layer | front | 2449 | 91.50% | 93.85% | 96.49%
LSTM-layer | rear | 2449 | 88.48% | 54.19% | 54.39%
Table 5: Classification accuracy of different truncated versions of the virus
dataset on doc2vec+NN and LSTM-layer models. The doc2vec+NN model outperforms
LSTM-layer on every case, but it is evident that the performance of LSTM-layer
does not depend exclusively on the length of the sequences.

####  3.2.2 The Role of the Embedding Size on Classification Accuracy of
doc2vec+NN

We compare the performance of the doc2vec+NN classification accuracy on the
virus dataset on different embedding sizes (Table 6 and Figure in the
Supplementary Material). Embedding size of 100 produces the best performance
of all, but results do not differ much among the sizes selected for the
experiment.

Embedding size | Training | Testing
---|---|---
50 | 90.63% | 87.72%
100 | 97.32% | 96.49%
150 | 95.98% | 94.74%
200 | 94.20% | 92.98%
Table 6: Classification accuracy of doc2vec+NN model on the virus dataset for
different embedding sizes. Embedding size of 100 produces the best
performance.

####  3.2.3 The Role of the Optimizer in the Classification Accuracy of the
LSTM-layer Model

Table 7 and Figure 9 show that the Stochastic Gradient Descent (SGD) optimizer
outperforms the Adam optimizer in the training of the LSTM-layer model (Figure
2(a)) for the splice data (and in the Supplementary Material for the CHiP-seq
toy data). Reddi et al. (2019) have already discussed the convergence issues
of the Adam optimizer. We had to be careful, however, when using Early Stop
Callback with the SGD optimizer as training would stop much earlier than
desirable and thus, Adam would outperform SGD. While we used Early Stop
Callback on both Adam and SGD optimizers, the patience parameter was quite
different. The patience parameter is a threshold to stop the training if the
loss stops decreasing further after a certain number of epochs. We set the
patience at 100 for Adam and at 400 for SGD which moves slower compared to
Adam.

Dataset | Optimizer | Training | Validation | Testing
---|---|---|---|---
ChIP-seq toy | Adam | 92.64% | 91.20% | 90.62%
ChIP-seq toy | SGD | 94.08% | 90.40% | 91.70%
splice | Adam | 86.82% | 78.28% | 80.25%
splice | SGD | 90.98% | 85.82% | 87.77%
Table 7: Classification accuracy of the LSTM-layer model on different
optimizers and different datasets. SGD seems to outperform Adam on the cases
compared.

Figure 9: Learning dynamics for LSTM-layer model trained on the splice data
with two optimizers: Adam and SGD. Color represents training (blue) or
validating (green) accuracy or loss. Note that there are different number of
epochs on each optimizer due to the speed of convergence.

####  3.2.4 The Role of the Batch Size in the Classification Accuracy of the
LSTM-AE+NN Model

While training the LSTM-AE model (Agarwal et al., 2019) on the ChIP-seq toy
data, we notice a strange pattern of sudden huge increments in loss (or
decrements in accuracy). The pattern appears to be related to the batch size
and learning rate. Increasing the batch size has very similar effects as
decreasing the learning rate. A larger batch size enables the optimizer to
better estimate the gradient for each step.

We compare different batch sizes in Table 8 and in Figure 10. We use the Adam
optimizer with learning rate of 0.001.

Figure 10 shows that the sudden increments in loss (corresponding to sudden
drops in accuracy) can be largely explained by the randomness of small batches
as increasing the batch size alleviates this behavior with batch size of 256
showing the best performance. Interestingly, the effectiveness of the
embedding is not affected by the erratic behavior of the learning dynamic
(Table 8). Larger batch sizes have the added benefit of reducing the training
time per epoch in the encoder network.

Figure 10: Learning dynamics of the LSTM-AE+NN classification model on the ChIP-seq toy data by different batch sizes (columns): 32, 256 and 1024. Color represents training (blue) or validating (green) accuracy or loss. Batch size | Training time per epoch | Training | Validating | Testing
---|---|---|---|---
32 | 38s | 58.50% | 57.70% | 57.30%
256 | 15s | 66.10% | 67.60% | 65.40%
1024 | 13s | 67.60% | 67.50% | 66.50%
Table 8: Classification accuracy per batch size for the LSTM-AE+NN model on
the ChIP-seq toy data. Training time corresponds to the encoder network.

####  3.2.5 Comparison of Sequence Embedding in doc2vec and LSTM-AE

We compare the embedding potential of doc2vec and LSTM-AE without the
prediction part of the model. That is, we explore the strengths and weaknesses
of each embedding strategy. Given that doc2vec and LSTM-AE correspond to quite
different embedding processes, we cannot compare them directly, but we
investigate reconstruction accuracy for LSTM-AE and Silhoutte score
(Rousseeuw, 1987; Pedregosa et al., 2011) for doc2vec.

For the LSTM-AE model (Figures 2(b) and 2(c)), we compare the reconstruction
accuracy for different batch sizes using the sparse categorical cross entropy
metric with nucleotides coded as integers. As mentioned already, the accuracy
is not affected by the batch size, but the training time is reduced
dramatically.

Batch size | Training time per epoch | Training (reconstruction) | Testing (reconstruction)
---|---|---|---
32 | 38s | 48.12% | 48.10%
256 | 15s | 45.96% | 46.24%
1024 | 13s | 47.33% | 47.46%
Table 9: Reconstruction accuracy of the LSTM-AE model for different batch
sizes on the ChIP-seq toy data. The accuracy is not improved by batch size,
but the training time per epoch is indeed reduced.

For the doc2vec embedding, we cannot calculate any reconstruction accuracy
since the model is not reconstructing the input sequences. We instead project
the embedded sequences on a low dimensional space (tSNE) and assess whether
there is a clear separation of classes. Intuitively, if the embedding is able
to separate the classes in a low dimensional space, the embedding aids in the
prediction task of the neural network model. We evaluate the separation of
classes for different embedding sizes with the Silhouette Score (Rousseeuw,
1987; Pedregosa et al., 2011). A higher Silhouette Score corresponds to an
embedding with better defined clusters. The Silhouette Score is defined as
$(b-a)/\max(a,b)$ for $a$, the mean distance between a sample and all other
points in the same class, and $b$, the mean distance between a sample and all
other points in the next nearest cluster.

Table 10 shows the Silhouette Score by embedding sizes for doc2vec on the
virus dataset.

| Full dimension | Low dimension
---|---|---
Embedding size | Training SS | Testing SS | Training SS | Testing SS
50 | 0.22 | 0.11 | 0.29 | 0.12
100 | 0.37 | 0.23 | 0.41 | 0.23
150 | 0.05 | 0.03 | 0.26 | 0.04
200 | 0.13 | 0.07 | 0.22 | 0.01
Table 10: Silhouette Score (SS) on the doc2vec embedding of the virus dataset
by embedding size. Full dimension corresponds to the score computed on the
doc2vec embedding before any projection to a low dimensional space. Low
dimension corresponds to the score after project to a low dimensional space
via tSNE. The Silhouette Score is the highest for the embedding size of 100
which agrees our conclusion in Table 6.

Finally, we compute the Silhouette Score on the embedding produced by the
LSTM-AE model on the virus dataset merely as a point of comparison. For the
full dimensional embedding, the training score is 0.029 and the testing score
is 0.063, while the training/testing scores for the tSNE projection of the
embedding are 0.003/0.034. These scores are much smaller than those obtained
with the doc2vec embedding which agrees with our conclusion that doc2vec is a
better embedding strategy for long sequences.

##  4 Discussion

Neural network models provide endless opportunities for prediction and
classification in biological applications (Peng et al., 2020; Alber et al.,
2019), yet much remains unknown regarding the transferability of the
performance across datasets. Robustness across datasets of similar nature is a
key ingredient to translate neural network models into medical practice and
into the agricultural or environmental field.

Here, we study the performance on genomic data of two widely used classes of
neural network models: convolutional neural networks (CNN) and natural
language processing (NLP). We find that simple CNN (few layers) are robust
across different genomic datasets with moderately good performance that is
transferable and generalizable. As expected, more convolutional layers cause
overfitting which can be overcome by regularization in most cases. In the
situations we study, more than 3 convolutional layers are not justified for
the sake of robustness and for control of overfitting, and simpler models are
preferable. Finally, we suspect that the overfitting pattern that we observe
with increasingly complex CNN could be due to the inherent sparsity of genomic
data. Yet we observe the same overfitting pattern when the input data are
images providing evidence that it is not the data, but the model what causes
the nature of overfitting.

Regarding embedding of the sequences, we identify two conclusions. First, we
find that doc2vec embedding is more powerful with longer sequences while LSTM-
AE is better for shorter sequences. Second, embedding within the neural
network architecture (LSTM-layer) has better prediction accuracy than
embedding first, and then predicting with a simple NN (doc2vec+NN and LSTM-
AE+NN). However, when this embedding is done within a CNN (DeepRAM), it can
lead to overfitting. Furthermore, the optimization problem of embedding layers
can become intractable for large datasets.

We conclude by raising awareness to the importance of reproducibility in
science. In many instances, it was impossible to replicate the results of
existing publications given the lack of reproducible well-documented scripts
and available data. Reproducibility is crucial not just for the sake of open
science, but to maximize the applicability of our machine-learning findings
into a biological or medical community who might not have a strong programming
background.

###  4.1 Reproducibility

All the scripts are publicly available in the Github repository
https://github.com/solislemuslab/dna-nn-theory.

Acknowledgments

This work is supported by the Department of Energy [DE-SC0021016 to CSL]. We
thank Dr. Aurelie Rakotondrafara and Helena Jaramillo Mesa for the virus data
which will be published soon as part of their own research. Finally, we
acknowledge the work in Hotaling (2020) which helped us improve the scientific
writing of this manuscript.

## Appendix A.

Figure 11: Learning dynamics for the simple CNN models that contain one 2D
convolutional layer on splice and promoter data. Color represents training
(blue) or validating (green) accuracy or loss.

Figure 12: Learning dynamics for the simple CNN models that contain one 1D
convolutional layer on splice and promoter data. Color represents training
(blue) or validating (green) accuracy or loss. Figure 13: Learning dynamics
for the DeepRAM models (Figures 1(e), 1(f), 1(g)) on the motif discovery data.
Rows correspond to different data encodings (embedding and one-hot encoding).
Columns correspond to different architectures (convolutional neural networks,
convolutional plus recurrent neural networks, and recurrent neural networks).
Color represents training (blue) or validating (green) loss.

Figure 14: Learning dynamics on the LSTM-layer model on the ChIP-seq toy data
by optimizer. Color represents training (blue) or validating (green) accuracy
or loss.

Figure 15: Learning dynamics of the LSTM-AE+NN classification model on ChIP-
seq toy dataset and splice dataset. Color represents training (blue) or
validating (green) accuracy or loss.

Figure 16: Learning dynamics of the doc2vec+NN classification model on ChIP-
seq toy dataset and splice dataset. Color represents training (blue) or
validating (green) accuracy or loss.

Figure 17: Learning dynamics of the doc2vec+NN classification model on the
virus dataset depending on the embedding size: 50, 100, 150, and 200. Color
represents training (blue) or validating (green) accuracy or loss.

### Reproducible Julia Script for Random Order of First Authors

    using Random
    s = 313627913
    Random.seed!(s)
    people = ["songyang","zhaoyi"]
    people[randperm(length(people))]

## References

  * Agarwal et al. (2019) Vishal Agarwal, N Jayanth Kumar Reddy, and Ashish Anand.  Unsupervised representation learning of DNA sequences.  _arXiv_ , (1906.03087), June 2019.
  * Alber et al. (2019) Mark Alber, Adrian Buganza Tepole, William R Cannon, Suvranu De, Salvador Dura-Bernal, Krishna Garikipati, George Karniadakis, William W Lytton, Paris Perdikaris, Linda Petzold, et al.  Integrating machine learning and multiscale modeling—perspectives, challenges, and opportunities in the biological, biomedical, and behavioral sciences.  _NPJ digital medicine_ , 2(1):1–11, 2019.
  * Ané (2011) Cécile Ané.  Detecting Phylogenetic Breakpoints and Discordance from Genome-Wide Alignments for Species Tree Reconstruction.  _Genome Biology and Evolution_ , 3:246–258, 02 2011.  ISSN 1759-6653.  doi: 10.1093/gbe/evr013.  URL https://doi.org/10.1093/gbe/evr013.
  * Ashley (2015) Euan A Ashley.  The precision medicine initiative: a new national effort.  _Jama_ , 313(21):2119–2120, 2015.
  * Bokulich et al. (2018) Nicholas A Bokulich, Matthew R Dillon, Evan Bolyen, Benjamin D Kaehler, Gavin A Huttley, and J Gregory Caporaso.  q2-sample-classifier: machine-learning tools for microbiome classification and regression.  _Journal of open research software_ , 3(30), 2018.
  * Carrieri et al. (2019) Anna Paola Carrieri, Will PM Rowe, Martyn Winn, and Edward O Pyzer-Knapp.  A fast machine learning workflow for rapid phenotype prediction from whole shotgun metagenomes.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 9434–9439, 2019.
  * Chang et al. (2017) Hao-Xun Chang, James S Haudenshield, Charles R Bowen, and Glen L Hartman.  Metagenome-wide association study and machine learning prediction of bulk soil microbiome and crop productivity.  _Frontiers in Microbiology_ , 8:519, 2017.
  * Chen et al. (2019) Po-Hsuan Cameron Chen, Yun Liu, and Lily Peng.  How to develop machine learning models for healthcare.  _Nature Materials_ , 18(5):410–414, 2019.
  * Coelho et al. (2013) Joana Rosado Coelho, João André Carriço, Daniel Knight, Jose-Luis Martínez, Ian Morrissey, Marco Rinaldo Oggioni, and Ana Teresa Freitas.  The use of machine learning methodologies to analyse antibiotic and biocide susceptibility in staphylococcus aureus.  _PLoS One_ , 8(2):e55582, 2013.
  * Dacrema et al. (2019) Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach.  Are we really making much progress? a worrying analysis of recent neural recommendation approaches.  In _Proceedings of the 13th ACM Conference on Recommender Systems_ , RecSys ’19, page 101–109, New York, NY, USA, 2019. Association for Computing Machinery.  ISBN 9781450362436.  doi: 10.1145/3298689.3347058.  URL https://doi.org/10.1145/3298689.3347058.
  * Dua and Graff (2017) Dheeru Dua and Casey Graff.  UCI machine learning repository, 2017.  URL http://archive.ics.uci.edu/ml.
  * Duarte-Carvajalino et al. (2018) Julio M Duarte-Carvajalino, Diego F Alzate, Andrés A Ramirez, Juan D Santa-Sepulveda, Alexandra E Fajardo-Rojas, and Mauricio Soto-Suárez.  Evaluating late blight severity in potato crops using unmanned aerial vehicles and machine learning algorithms.  _Remote Sensing_ , 10(10):1513, 2018.
  * Dutta et al. (2018) T. Dutta, A.and Dubey, K. K. Singh, and A. Anand.  Splicevec: Distributed feature representations for splice junction prediction.  _Computational biology and chemistry_ , 74:434–441, 2018\.
  * Egmont-Petersen et al. (2002) Michael Egmont-Petersen, Dick de Ridder, and Heinz Handels.  Image processing with neural networks: a review.  _Pattern recognition_ , 35(10):2279–2301, 2002\.
  * Ekins et al. (2019) Sean Ekins, Ana C. Puhl, Kimberley M. Zorn, Thomas R. Lane, Daniel P. Russo, Jennifer J. Klein, Anthony J. Hickey, and Alex M. Clark.  Exploiting machine learning for end-to-end drug discovery and development.  _Nature Materials_ , 18(5):435–441, 2019.
  * Fjell et al. (2009) Christopher D Fjell, Håvard Jenssen, Kai Hilpert, Warren A Cheung, Nelly Pante, Robert EW Hancock, and Artem Cherkasov.  Identification of novel antibacterial peptides by chemoinformatics and machine learning.  _Journal of medicinal chemistry_ , 52(7):2006–2015, 2009.
  * Hjelmås and Low (2001) Erik Hjelmås and Boon Kee Low.  Face detection: A survey.  _Computer vision and image understanding_ , 83(3):236–274, 2001.
  * Ho et al. (2019) Daniel Sik Wai Ho, William Schierding, Melissa Wake, Richard Saffery, and Justin O’Sullivan.  Machine learning snp based prediction for precision medicine.  _Frontiers in Genetics_ , 10:267, 2019.
  * Hotaling (2020) Scott Hotaling.  Simple rules for concise scientific writing.  _Limnology and Oceanography Letters_ , 5(6):379–383, 2020.  doi: https://doi.org/10.1002/lol2.10165.  URL https://aslopubs.onlinelibrary.wiley.com/doi/abs/10.1002/lol2.10165.
  * Ip et al. (2018) Ryan HL Ip, Li-Minn Ang, Kah Phooi Seng, JC Broster, and JE Pratley.  Big data and machine learning for crop protection.  _Computers and Electronics in Agriculture_ , 151:376–383, 2018.
  * Jonas et al. (2018) Eric Jonas, Monica Bobra, Vaishaal Shankar, J Todd Hoeksema, and Benjamin Recht.  Flare prediction using photospheric and coronal image data.  _Solar Physics_ , 293(3):48, 2018.
  * Katuwal and Chen (2016) Gajendra Jung Katuwal and Robert Chen.  Machine learning model interpretability for precision medicine.  _arXiv preprint arXiv:1610.09045_ , 2016.
  * Kavvas et al. (2018) Erol S Kavvas, Edward Catoiu, Nathan Mih, James T Yurkovich, Yara Seif, Nicholas Dillon, David Heckmann, Amitesh Anand, Laurence Yang, Victor Nizet, et al.  Machine learning and structural analysis of mycobacterium tuberculosis pan-genome identifies genetic signatures of antibiotic resistance.  _Nature communications_ , 9(1):1–9, 2018.
  * Kimothi et al. (2016) Dhananjay Kimothi, Akshay Soni, Pravesh Biyani, and James M. Hogan.  Distributed representations for biological sequence analysis.  _arXiv_ , (1608.05949), 2016.
  * Krittanawong et al. (2017) Chayakrit Krittanawong, HongJu Zhang, Zhen Wang, Mehmet Aydar, and Takeshi Kitai.  Artificial intelligence in precision cardiovascular medicine.  _Journal of the American College of Cardiology_ , 69(21):2657–2664, 2017.
  * (26) Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.  Cifar-10 (canadian institute for advanced research).  URL http://www.cs.toronto.edu/~kriz/cifar.html.
  * Kucuk et al. (2017) Ahmet Kucuk, Juan M Banda, and Rafal A Angryk.  A large-scale solar dynamics observatory image dataset for computer vision applications.  _Scientific data_ , 4:170096, 2017.
  * Le and Mikolov (2014) Quoc Le and Tomas Mikolov.  Distributed representations of sentences and documents.  In Eric P. Xing and Tony Jebara, editors, _Proceedings of the 31st International Conference on Machine Learning_ , volume 32 of _Proceedings of Machine Learning Research_ , pages 1188–1196, Bejing, China, 22–24 Jun 2014. PMLR.  URL http://proceedings.mlr.press/v32/le14.html.
  * Lee et al. (2018) Su-In Lee, Safiye Celik, Benjamin A Logsdon, Scott M Lundberg, Timothy J Martins, Vivian G Oehler, Elihu H Estey, Chris P Miller, Sylvia Chien, Jin Dai, et al.  A machine learning approach to integrate big data for precision medicine in acute myeloid leukemia.  _Nature communications_ , 9(1):1–13, 2018.
  * Li et al. (2018) Li-Guan Li, Xiaole Yin, and Tong Zhang.  Tracking antibiotic resistance gene pollution from different sources using machine-learning classification.  _Microbiome_ , 6(1):1–12, 2018.
  * Maino et al. (2018) James L Maino, Paul A Umina, and Ary A Hoffmann.  Climate contributes to the evolution of pesticide resistance.  _Global Ecology and Biogeography_ , 27(2):223–232, 2018.
  * Nguyen et al. (2016) Ngoc Giang Nguyen, Vu Anh Tran, Duc Luu Ngo, Dau Phan, Favorisen Rosyking Lumbanraja, Mohammad Reza Faisal, Bahriddin Abapihi, Mamoru Kubo, and Kenji Satou.  DNA sequence classification by convolutional neural network.  _JBiSE_ , 09(05):280–286, 2016.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011\.
  * Peng et al. (2020) Grace C. Y. Peng, Mark Alber, Adrian Buganza Tepole, William R. Cannon, Suvranu De, Savador Dura-Bernal, Krishna Garikipati, George Karniadakis, William W. Lytton, Paris Perdikaris, Linda Petzold, and Ellen Kuhl.  Multiscale modeling meets machine learning: What can we learn?  _Archives of Computational Methods in Engineering_ , 2020.
  * Pesesky et al. (2016) Mitchell W Pesesky, Tahir Hussain, Meghan Wallace, Sanket Patel, Saadia Andleeb, Carey-Ann D Burnham, and Gautam Dantas.  Evaluation of machine learning and rules-based approaches for predicting antimicrobial resistance profiles in gram-negative bacilli from whole genome sequence data.  _Frontiers in microbiology_ , 7:1887, 2016.
  * Reddi et al. (2019) Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar.  On the convergence of adam and beyond.  _arXiv_ , (1904.09237), 2019.
  * Rost et al. (2016) Burkhard Rost, Predrag Radivojac, and Yana Bromberg.  Protein function in precision medicine: deep understanding with machine learning.  _FEBS letters_ , 590(15):2327–2341, 2016.
  * Rousseeuw (1987) Peter J. Rousseeuw.  Silhouettes: A graphical aid to the interpretation and validation of cluster analysis.  _Journal of Computational and Applied Mathematics_ , 20:53 – 65, 1987.  ISSN 0377-0427.  doi: https://doi.org/10.1016/0377-0427(87)90125-7.  URL http://www.sciencedirect.com/science/article/pii/0377042787901257.
  * Shadab et al. (2020) S Shadab, M T A Khan, N A Neezi, S Adilina, and others.  DeepDBP: Deep neural networks for identification of DNA-binding proteins.  _Informatics in Medicine_ , 2020.
  * Teschendorff (2019) Andrew E Teschendorff.  Avoiding common pitfalls in machine learning omic data science.  _Nature Materials_ , 18(5):422–427, 2019.
  * Trabelsi et al. (2019) Ameni Trabelsi, Mohamed Chaabane, and Asa Ben-Hur.  Comprehensive evaluation of deep learning architectures for prediction of DNA/RNA sequence binding specificities.  _Bioinformatics_ , 35(14):i269–i277, July 2019\.
  * Yang and Guo (2017) Xin Yang and Tingwei Guo.  Machine learning in plant disease research.  _European Journal of BioMedical Research_ , 3(1):6–9, 2017.
  * Zeng et al. (2016) Haoyang Zeng, Matthew D Edwards, Ge Liu, and David K Gifford.  Convolutional neural network architectures for predicting DNA-protein binding.  _Bioinformatics_ , 32(12):i121–i127, June 2016\.