# Predicting Transcription Factor Binding Sites using Transformer based
Capsule Network

Nimisha Ghosh and Daniele Santoni and Indrajit Saha and Giovanni Felici
Nimisha Ghosh is with the Department of Computer Science and Information
Technology, Siksha ‘O’ Anusandhan (Deemed to be) University, Bhubaneswar,
India and the Institute of System Analysis and Informatics of the Italian
National Research Council (CNR-IASI), Rome, Italy. Email:
<EMAIL_ADDRESS>Santoni and Giovanni Felici are with
theInstitute of System Analysis and Informatics of the Italian National
Research Council (CNR-IASI), Rome, Italy. Email<EMAIL_ADDRESS><EMAIL_ADDRESS>Saha is with National Institute of
Technical Teachers Training and Research, Kolkata, India. Email:
<EMAIL_ADDRESS>received April 19, xxxx; revised August 16,
xxxx.

###### Abstract

Prediction of binding sites for transcription factors is important to
understand how they regulate gene expression and how this regulation can be
modulated for therapeutic purposes. Although in the past few years there are
significant works addressing this issue, there is still space for improvement.
In this regard, a transformer based capsule network viz. DNABERT-Cap is
proposed in this work to predict transcription factor binding sites mining
ChIP-seq datasets. DNABERT-Cap is a bidirectional encoder pre-trained with
large number of genomic DNA sequences, empowered with a capsule layer
responsible for the final prediction. The proposed model builds a predictor
for transcription factor binding sites using the joint optimisation of
features encompassing both bidirectional encoder and capsule layer, along with
convolutional and bidirectional long-short term memory layers. To evaluate the
efficiency of the proposed approach, we use a benchmark ChIP-seq datasets of
five cell lines viz. A549, GM12878, Hep-G2, H1-hESC and Hela, available in the
ENCODE repository. The results show that the average area under the receiver
operating characteristic curve score exceeds 0.91 for all such five cell
lines. DNABERT-Cap is also compared with existing state-of-the-art deep
learning based predictors viz. DeepARC, DeepTF, CNN-Zeng and DeepBind, and is
seen to outperform them.

###### Index Terms:

Capsule Layer, Deep Learning, DNABERT, DNA sequences, Transcription Factor
Binding Sites (TFBSs)

##  I Introduction

Transcription Factors (TFs) are proteins that bind to certain genomic
sequences and influence a wide range of cellular functions [1, 2]. TFs bind to
DNA-regulatory sequences which are known as Transcription Factor Binding Sites
(TFBSs), typically of size 4-30 bp [3, 4, 5] and modulate the gene
transcription along with playing important role in cellular processes [6, 7,
8]. The accurate prediction of TFBSs is indeed crucial to characterise certain
functional aspects of the genome and explain how programmes of specific
sequence expression are organised in complex organisms [9, 10, 11]. High-
throughput sequencing technology has led to a large amount of experimental
data about TFBS, like JASPAR [12], TRANSFAC [13] etc. However, identifying
TFBSs using experimental methods is very time consuming and expensive. This
led to the development of computational methods to identify TFBSs encompassing
large amount of data.

Initially, many researchers proposed machine learning methods to identify
TFBSs. In this regard, Wong et al. [14] put forth kmerHMM to identify TFBS. In
this method, Hidden Markov Model (HMM) is trained for the underlying motif
representation and subsequently belief propagation is used to extract multiple
motifs from HMM. To predict DNA binding site, Ghandi et al. [15] proposes gkm-
SVM which uses a tree data structure to calculate the kernel matrix. However,
traditional machine learning models usually rely on manual feature extraction
and they have problem processing large-scale datasets. In recent times, there
have been a number of deep learning models specifically developed for natural
language processing [16] and computer vision [17, 18]. Similar models have
also been applied to solve problems in computational biology and
bioinformatics [19, 20, 21]. Deep neural network based methods such as
DeepBind [22] and DeepSEA [23] show competitively better results as compared
to traditional methods like Markov model, support vector machines,
hierarchical mixture models, discriminative maximum conditional likelihood and
random forests. DeepBind [22] demonstrates the capabilities of deep learning
to assess sequence specificity from experimental data. It offers a scalable,
adaptable and integrated calculating technique for finding patterns. DeepBind
is also the first technology to ever address the demand for precise modelling
of protein target binding motifs. A long-short term recurrent convolutional
network called DeeperBind [24] is used to anticipate the specificities of how
proteins will bind to DNA probes. In order to effectively account for the
contributions provided by various sub-regions in DNA sequences, DeeperBind can
describe the positional dynamics of probe sequences. It can also be trained
and evaluated on datasets with sequences of different lengths. Quang et al.
[25] propose DanQ that combines CNNs and bidirectional long short-term memory
network (BiLSTM) to predict binding sites. Zeng et al. [23] perform systematic
exploration of CNN architectures for the prediction of DNA sequence binding
using an extensive collection of transcription factor datasets. They identify
the best performing architectures by varying CNN width, depth and pooling
designs. In order to split the DNA binding sequence into overlapping pieces
and predict TFBS, Farrel et al. [26] present an effective pentamer approach.
In [27], Qin et al. propose TFImpute to predict cell-specific TFBS on ChIP-seq
data. TFImpute incorporates TFs and cell lines into continuous vectors that
are used as model input. DeepSNR as proposed by Salekin et al. [28] uses CNN-
Deconvolutional model to predict transcription factor binding location at
single nucleotide resolution. DeepFinder [29] is an enhanced three-stage DNA
motif prediction for the large-scale pattern analysis that uses TFBS-
associated deep learning neural networks to build the motif model. For data on
imbalanced DNA-protein binding sites, Zhang et al. [30] suggest a new
prediction approach. This technique employs Bootstrap algorithm to undersample
the negative data, and adaptive synthesis sampling to oversample the positive
data. To further capture long-term relationships between DNA sequence motifs,
DeepSite [31] uses CNN and bidirectional long short-term memory. Apart from
considering sequence dependencies, filtering out valid information in huge
data and precisely locating motif information in the imbalanced data are also
essential questions addressed in [31]. Yang et al. [32] use deep neural
networks and binomial distribution to enhance motif prediction in the human
genome to help with TFBS identification and motif prediction accuracy. In
[33], Chen et al. use deep learning to develop TF binding prediction tool
known as DeepGRN. The first part of the model is a convolutional layer while
the BiLSTM nodes are recurrent units. Multi-scale convolution along with LSTM
(MCNN-LSTM) are the choice made in [34] to accurately predict TFBSDeepTF.
Their results show that MCNN-LSTM outperforms several existing TFBS
predictors. Zhang et al. [35] propose a new architecture by combining a
convolutional autoencoder with convolutional neural network (CAE-CNN) to
predict TFBS, employing a highway connection layer to better understand the
features of DNA nucleotides through a gated unit. Their primary contribution
lies in the integration of supervised and unsupervised learning methods to
predict TFBS. In [36], Jing et al. use meta learning based CNN method (MLCNN)
to predict TFBS. The performance of MLCNN shows that it is competitively
better than other state-of-the-art CNN methods. A hybrid convolutional
recurrent neural network (CNN/RNN) architecture known as CRPTS is proposed in
[37] to predict TFBSs by combining DNA sequence and DNA shape features. Cao et
al. [38] propose DeepARC which combines convolutional neural network (CNN) and
recurrent neural network (RNN) to predict TFBS. To extract the hidden
embedding from DNA sequences, DeepARC uses a positional embedding method that
incorporates the positional data from OneHot encoding and the distributed
embedding from DNA2Vec. With an average area under the receiver operating
characteristic curve (AUC) score of 0.908 on five cell lines (A549, GM12878,
Hep-G2, H1-hESC and Hela) in the benchmark dataset, DeepARC displays promising
capabilities. However, DeepARC is lacking in adopting an efficient encoding
policy.

Thus, it can be very well concluded that deep learning is widely applied in
the prediction of TFBSs. Although, many aspects of deep learning are well
explored in the context of TFBS prediction, works based on transformers are
quite limited. Motivated by the literature, in this work we develop DNABERT-
Cap built on top of pre-trained DNABERT model to predict TFBSs. Initially, we
consider the positive set which are cell line specific DNA sequences with
TFBSs while the negative set is positive shuffled sequences conserving
dinucleotide frequency. These DNA sequences are divided into k-mers which are
then sequentially passed through DNABERT, a Convolutional Layer (CL), a
bidirectional Long-Short Term Memory (LSTM) layer and finally to a capsule
layer, in order to make effective prediction of TFBSs.

This transformer based capsule network viz. DNABERT-Cap uses DNABERT to
provide the embedding of DNA sequences and apply CL with bidirectional LSTM
along with capsule layer to create an effective prediction model. DNABERT is a
bidirectional encoder pre-trained with a large number of DNA sequences. It
uses positional embedding to understand the semantic relationship among the
k-mers of a DNA sequence. Along with DNABERT, the convolutional layer provides
a further abstract representation of the k-mers taking spatial correlations
into account while the bidirectional LSTM layer captures the context of k-mers
to learn long term dependencies in a DNA sequence. Finally, the addition of a
capsule layer improves the efficiency of a predictor by encoding spatial
patterns to process the relationship between a part and the whole of the
sequence through capsules and dynamic routing. Extensive experimental results
show that the proposed method performs better than existing state-of-the-art
methodologies. It is worth mentioning over here that capsule network is
already used in natural language processing (NLP) for text [39, 40] as well as
tweet act classification [41] with competitive results.

##  II Materials and Methods

In this section, data preparation is elaborated, followed by the discussion of
the pipeline of the proposed work.

###  II-A Data Preparation

In this work, the benchmark dataset from Encyclopedia of DNA Elements (ENCODE)
[42] is used to acquire the TFBS data analysed by ChIPseq method. This data is
used to train and test the proposed model. Data preprocessing is the same as
considered in [23], where the positive samples with 101 bps are generated in
the centre of each ChIP-seq peak, while the negative samples are obtained by
recombining the positive sequence conserving dinucleotide sequences. The
positive and negative samples are then distinguished based on the presence or
absence of TFBSs in a sequence. For experimental purposes and based on
availability of resources, 500,000 sequences for each of the five cell lines
viz. A549, GM12878, Hep-G2, H1-hESC and Hela are selected randomly from the
ChIP-seq datasets. This random selection takes care of a fair balance between
positive and negative sequences. Each dataset is divided into 70% training,
20% test and 10% validation sets. All the experiments are conducted on
machines with NVIDIA GA100 GPUs. The dataset used in this work along with code
are available at https://github.com/NimishaGhosh/DNABERT-Cap/tree/main.

###  II-B DNABERT and Capsule Network

Before delving into the pipeline of the work, a brief discussion on DNABERT
model and capsule network is put forth.

####  II-B1 DNABERT

DNABERT [43] is a pre-trained bidirectional encoder representation which
captures intricacies of DNA sequences based on up and downstream nucleotide
contexts. A set of sequences represented as k-mers are provided as input to
DNABERT where each sequence is represented as a matrix $\mathcal{M}$ by
embedding each token into a numerical vector. It captures contextual
information of a sequence by performing multi-head self-attention mechanism on
$\mathcal{M}$:

$multihead\mathcal{M}=Concat(head_{1},\dots,head_{h})\mathcal{W}^{O}$ (1)

where

$head_{i}=softmax\left(\frac{\mathcal{M}\mathcal{W}^{Q}_{i}\mathcal{M}\mathcal{W}^{K}_{i}T}{\sqrt{d_{k}}}\right).\mathcal{M}\mathcal{W}^{V}_{i}$
(2)

Here, all $\mathcal{W}$s are learned parameters for linear projection.
Equations (1) and (2) are performed $L$ times, where $L$ is the number of
layers. The model architecture is similar to BERT base with 12 transformer
layers and 768 hidden units as well as 12 attention heads in each layer.

####  II-B2 Capsule Networks

In neural network based methods, spatial patterns obtained at lower levels are
responsible for representation of higher level concepts. In order to extract
local patterns from a vector sequence, CNN builds convolutional feature
detectors. The most noticeable patterns are then chosen using max-pooling.
However, CNN may lose many important information during such pooling process
and perform poorly on problems characterised by positional invariance. In
contrast, approaches that do not take spatial relationships into account
perform flawlessly when making inferences, regardless of the order of a token
or local patterns; however, they cannot encode rich structures that may be
present in a sequence. In this regard, capsule networks [44] help to improve
the efficiency to encode spatial patterns to include knowledge about the part-
whole relationships. Each capsule is a group of neurons where the input and
output are both vectors. These group of neurons work together to recognise
specific features or patterns. They have an iterative dynamic routing
algorithm [45] which helps to decide the credit attribution between nodes from
lower to higher layers. As a result, capsule networks generalise a particular
class instead of memorising every viewpoint variant of the class, thereby
becoming invariant to the viewpoint and showing improved performance as
compared to CNN.

###  II-C Pipeline of the Work

The pipeline of the work is depicted in Fig. 1. Initially, the DNA sequences
for each cell line are fed to the DNABERT model in the form of k-mer tokens
(k=6 in this case as [43] reports that 6-mers show the best performance). For
each input sequence of tokens, DNABERT returns an embedding.

Figure 1: Depiction of the pipeline of the methodology of DNABERT-Cap. The
first row represents the main sequences of the components, while the yellow
box below is an explosion of the DNABERT sequence. The last column of boxes,
in green, is where the capsule layer is out in force (two capsules are
represented with blue dots in the output layer).

Let $W_{A}\in\mathbb{R}^{l\times d}$ be the weight matrix for each such
embedding, where $l$ is the length of a sequence and $d$ is the dimension of
each token representation. Each weight matrix is then passed through a series
of hierarchical layers to obtain the optimal sequence representation to
classify such sequences as Transcription Factor Binding Site. The subsequent
layers are described as follows:

  1. 1.

Convolutional Layer: The output from DNABERT is provided as an input to the
convolutional layer to extract the N-gram feature map. This feature map is
produced by the convolution layer by sheafing the inner convolution layers and
then by pooling which can be represented as:

$\alpha_{i}=W_{A}*d_{i}+b$ (3) $\hat{\alpha}_{i}=pool(\alpha_{i})$ (4)

Here, output feature map $\alpha_{i}$ is produced by kernel $d_{i}$ with bias
$b$ by applying convolution and pooling $pool(.)$. $\eta$ such feature maps
are then combined to form a $\eta$-channel layer as

$A=[\hat{\alpha}_{1},\hat{\alpha}_{2},\dots,\hat{\alpha}_{\eta}]$ (5)

This N-gram convolutional layer depicts an abstract representation of the
tokens, thereby reflecting their importance at different positions that
finally span over the entire sequence.

  2. 2.

Bidirectional LSTM Layer: In order to learn semantic dependency,
$\eta$-channel feature vector $A$ as obtained from the previous layer is
passed through a bidirectional Long Short Term Memory (LSTM). The long term
dependencies in a sequence is captured using such a bidirectional LSTM by
sequentially encoding the feature maps into hidden states as

$\overrightarrow{h}_{\eta}=\overrightarrow{LSTM}(\hat{\alpha}_{\eta},\overrightarrow{h}_{\eta-1})$
(6)
$\overleftarrow{h}_{\eta}=\overleftarrow{LSTM}(\hat{\alpha}_{\eta},\overleftarrow{h}_{\eta+1})$
(7)

Here, each feature map $\hat{\alpha}_{\eta}$ is mapped to forward and backward
hidden states $\overrightarrow{h}_{\eta}$ and $\overleftarrow{h}_{\eta}$ by
respectively invoking forward $\overrightarrow{LSTM}$ and backward
$\overleftarrow{LSTM}$. Subsequently, $\overrightarrow{h}_{\eta}$ and
$\overleftarrow{h}_{\eta}$ are merged to get a single hidden state $h_{\eta}$.
The final hidden state matrix is defined as:

$H=[h_{1},h_{2},\dots,h_{\eta}]$ (8)

where $H\in\mathbb{R}^{\eta\times 2D}$ and $D$ is the number of hidden states
in each LSTM. Thus, this layer is important for capturing the context of
k-mers in a DNA sequence.

  3. 3.

Primary Capsule Layer: The primary capsule layer was originally introduced to
mitigate the flaws of conventional CNN by replacing the scalar-output features
with vector-output capsules to preserve instantiated parameters, such as the
local order and semantic representations of tokens. In this regard, primary
capsules hold a group of neurons to represent each element in the feature maps
as opposed to a scalar. This feature is important as, apart from considering
the intensity of the activation, it also maintains the details of the
instantiated parts in the input of this layer as well. So, the capsule can be
viewed as a short representation of the instantiated parts that are captured
by the convolution kernel. By sliding over the LSTM generated hidden states
$H$, each kernel $d_{i}$ generates a sequence of capsules $c_{i}$ of dimension
D. These sequence of capsules comprise of a channel $C_{i}$ which belong to
the primary capsule layer.

$C_{i}=S(d_{i}*H+b)$ (9)

Here, $S$ is the non-linear squash function and $b$ represents the bias weight
parameter of the capsule. Therefore, all the $n$ channels can be merged as:

$C=[C_{1},C_{2},\dots,C_{n}]$ (10)

Similar to spatial orientation in images, this layer captures the local
ordering of k-mers in a sequence and its corresponding semantic
representations.

  4. 4.

Dynamic Routing Between Capsules: The primary idea behind dynamic routing is
building non-linear map in an iterative way. This ensures that a lower level
capsule has strong connection to an appropriate capsule in the subsequent
layer. Moreover, pooling function of the traditional convolution layer which
normally removes the location information is replaced with dynamic routing
technique leading to a robust network. The length of a capsule is fixed in the
range of [0,1] by applying a nonlinear squashing function as given in Equation
(11)

$v_{j}=\frac{||s_{j}||^{2}}{1+||s_{j}||}\frac{s_{j}}{|||s_{j}||^{2}}$ (11)

Here, $v_{j}$ is the output vector of capsule $j$ and $s_{j}$ is its total
input. $s_{j}$ is a weighted sum over all prediction vectors $\hat{u}_{j|i}$
made by capsule $i$ and passed to capsule $j$. $\hat{u}_{j|i}$ is calculated
by multiplying output $u_{i}$ of a capsule in the previous layer by a weight
matrix $W_{ij}$. This process helps a capsule network to process the
relationship between a part and the whole of a sequence.

$\hat{u}_{i|j}=W_{ij}u_{i}$ (12) $s_{j}=\sum_{i}c_{ij}\hat{u}_{j|i}$ (13)

Here, $c_{ij}$ are the coupling coefficients as determined by the iterative
dynamic routing algorithm. $c_{ij}$ is computed as a softmax $b_{ij}$ which is
a log prior probabilities between capsules $i$ and $j$.

$c_{ij}=\frac{exp(b_{ij})}{\sum_{k}exp(b_{ik})}$ (14)

The initial coupling coefficients are refined iteratively, based on $b_{ij}$,
measuring the agreement between $v_{j}$ and $\hat{u}_{j|i}$. Such agreement is
given by the scalar product $a_{ij}=v_{j}.\hat{u}_{j|i}$. $b_{ij}$ is updated
as:

$b_{ij}=b_{ij}+\hat{u}_{j|i}.v_{j}$ (15)

This entire procedure reflects the dynamic routing for all capsule $i$ in
layer $L$ and capsule $j$ in layer $L+1$. In this work, dynamic routing
algorithm helps in deciding the importance and agreement of tokens for a
specific task by learning which k-mer tokens are important in a sequence or
which ones to ignore.

  5. 5.

TFBS Capsule Layer: In this layer, the TFBS capsules are responsible for
detecting the transcription factor binding sites of a given DNA sequence. The
vector sequence from the primary capsule is carried forward to the TFBS
capsule layer which generates one vector for each of the TFBS class capsule;
one for the class which exhibits the presence of TFBS; and another one
depicting the absence of the same.

  6. 6.

Output: The output from TFBS capsule layer has two class capsules (denoted by
two blue circles in Fig. 1) generating two vector outputs whose elements
encode various properties of features and the lengths represent the
probability that the corresponding class is present in the input data. To
increase the difference between the lengths of two class capsules and for
better generalisation, separate margin loss [45] $L_{b}$ is used in this work.

$\small
L_{b}=G_{b}max(0,m^{+}-||v_{b}||)^{2}+\lambda(1-G_{b})max(0,m^{-}-||v_{b}||)^{2}$
(16)

where $v_{b}$ is the capsule for class $b$. $G_{b}=1$ iff class $b$ is the
ground truth and $m^{+}$ = 0.9 and $m^{-}$ = 0.1. $\lambda$ is used to tune
the weight of an absent class.

###  II-D Hyperparameters

The pre-trained DNABERT [43] model has 12 transformer layers and 768 hidden
units along with 12 attention heads in each layer. For the convolutional
layer, the number of units is 64 while the kernel size used is 2. The number
of units used in the bidirectional LSTM layer is 64 as well. Furthermore, a
dropout value of $0.3$ and a capsule length of 16 are considered. The dynamic
routing algorithm with 3 iterations provided the optimum results and Adam
optimiser [46] is used for all the experiments. All these parameters are
considered after conducting thorough experiments.

##  III Results and Discussions

DNABERT-Cap is devised for predicting TFBSs in DNA sequences. The reported
results are an average of multiple runs on 500,000 randomly chosen sequences
for each cell line viz. A549, GM12878, Hep-G2, H1-hESC and Hela. The reported
results are conducted on several such randomly generated datasets and they
confirm the efficacy of DNABERT-Cap. We also have conducted similar
experiments on such other datasets which show similar results (available on
request). To show the effectiveness of the proposed model, it is compared with
some baselines as well as other state-of-the-art approaches. The performance
measures for such comparisons are mentioned next.

###  III-A Performance Metrics

The different performance parameters as used in this work are Accuracy,
Recall, Specificity Mathew’s Correlation Coefficient (MCC) and area under the
receiver operating characteristic curve (AUC) [38].

###  III-B Performance Comparison with Baselines

To show the efficacy of the proposed model, it is compared with the following
baseline models:

  1. 1.

Fine-tuned DNABERT model (Baseline-1): For comparison purposes, the original
DNABERT model is fine-tuned with our dataset where we have added a
classification layer on top of DNABERT.

  2. 2.

DNABERT+CL+LSTM+CE (Baseline-2): The usage of capsule layer to calculate the
loss is removed from this baseline and instead categorical cross entropy is
used as the loss function.

  3. 3.

DNABERT+CL+Capsule Layer (Baseline-3): In this baseline, the bidirectional
LSTM layer is removed to provide the comparison.

The results in terms of accuracy, recall, specificity, MCC and AUC are
reported in Table I.

TABLE I: Summary of the performances of the DNABERT-Cap and the selected baselines Model | Cell line | Accuracy (%) | Recall (%) | Specificity (%) | MCC | AUC
---|---|---|---|---|---|---
Fine-tuned DNABERT model (Baseline-1) | A549 | 79.14 | 78.55 | 79.72 | 0.609 | 0.873
GM12878 | 78.13 | 76.87 | 78.43 | 0.608 | 0.854
Hep-G2 | 79.56 | 78.20 | 79.88 | 0.609 | 0.876
H1-hESC | 78.11 | 77.14 | 78.13 | 0.603 | 0.858
Hela | 78.19 | 77.98 | 78.56 | 0.604 | 0.873
DNABERT+CL+LSTM+CE (Baseline-2) | A549 | 80.45 | 80.63 | 80.01 | 0.629 | 0.890
GM12878 | 80.32 | 78.73 | 80.67 | 0.625 | 0.877
Hep-G2 | 81.65 | 80.66 | 81.37 | 0.639 | 0.896
H1-hESC | 80.51 | 78.01 | 80.59 | 0.631 | 0.892
Hela | 81.64 | 69.02 | 81.55 | 0.643 | 0.905
DNABERT+CL+Capsule Layer(Baseline-3) | A549 | 83.58 | 83.03 | 79.58 | 0.681 | 0.915
GM12878 | 82.65 | 77.57 | 82.65 | 0.659 | 0.903
Hep-G2 | 84.76 | 79.92 | 84.68 | 0.698 | 0.920
H1-hESC | 82.62 | 78.85 | 82.59 | 0.654 | 0.904
Hela | 83.09 | 80.68 | 83.00 | 0.662 | 0.908
DNABERT-Cap | A549 | 84.66 | 81.57 | 84.65 | 0.696 | 0.925
GM12878 | 83.52 | 81.11 | 83.52 | 0.671 | 0.913
Hep-G2 | 85.49 | 83.43 | 85.46 | 0.710 | 0.930
H1-hESC | 83.43 | 78.39 | 83.50 | 0.674 | 0.914
Hela | 83.94 | 80.46 | 83.89 | 0.680 | 0.917

As can be observed from Table I, DNABERT-Cap provides competitive results if
compared to all the other baselines. When compared to the fine-tuned DNABERT
model, DNABERT-Cap shows an improvement of around 5% in terms of accuracy for
cell line A549. Similar improvements can be observed for other metrics as well
for other cell lines. Baseline 2 which includes DNABERT, convolutional and
bidirectional LSTM layer without the advantage of capsule layer also shows
quite competitive results as compared to baseline-1. This is true for all the
considered cell lines considered in this work. As compared to baseline-2,
where the advantage of capsule layer is not taken into account - the proposed
model shows improvement in terms of accuracy, specificity, MCC and AUC of
around 4% for A549. However, in terms of recall, baseline-3 shows better
performance. In this regard, to determine which baseline has the best
performance, AUC is a better parameter. It is a summary statistic for the
goodness of a classifier in a binary classification task and may resolve the
ambiguity that different metrics may produce. As can be seen from the results,
DNABERT-Cap has an improved performance in terms of AUC. This improved
performance is reflected for all the other cell lines as well. It is worth
noting here that as compared to baseline-3, the overall improved performance
of the proposed model can be attributed to the addition of a bidirectional
LSTM layer. Fig. 2 shows the area under the curve for all the five cell lines
for one of the multiple runs of the proposed DNABERT-Cap where the AUC values
of A549, GM12878, Hep-G2, H1-hESC and Hela are 0.925, 0.913, 0.930, 0.914 and
0.917 respectively. In the figures, no skill is represented at (0.5,0.5). A
model with no skill at each threshold is represented by a diagonal line from
the bottom left of the plot to the top right and has an AUC of 0.5. Please
note that for the ease of understanding and explanation, out of multiple runs,
curves with similar AUC values to those as reported in Table I are provided
over here.

(a) (b)

(c) (d)

(e)

Figure 2: ROC-AUC curve of the five cell lines analysed (a) A549 (AUC = 0.925)
(b) GM12878 (AUC = 0.913) (c) Hep-G2 (AUC = 0.930) (d) H1-hESC (AUC = 0.914)
(e) Hela (AUC = 0.917)

###  III-C Performance Comparison with State-of-the-Art Predictors

In order to analyse the performance of the proposed DNABERT-Cap, we compare it
with several other prediction methods viz. DeepARC [38], DeepTF [34], CNN-Zeng
[47] and DeepBind [22] considering the same cell lines as mentioned
previously. Table II reports the results of these comparisons considering the
average of the five cell lines. As evident from the table, the proposed model
has the best predictive performance among all the other state-of-the-art
approaches in terms of accuracy, specificity, MCC and AUC. With respect to
recall, DeepARC shows the best performance. As compared to DeepARC, DNABERT-
Cap has an improved performance of 1.11%, 0.01%, 2.2% and 1.10% for accuracy,
specificity, MCC and AUC respectively. Fig. 3 reports the AUC of each method
for the five cell lines. Statistical significance is also tested to analyse
the results as obtained while comparing DNABERT-Cap against DeepARC as it
shows the best results among all the compared classifiers. In this regard, the
Wilcoxon non-parametric test rank-sum test for significance assessment is
used. It is observed that all pairwise p-values are less than 0.05 (1.95E-02,
1.97E-02, 1.95E-02, 1.93E-02 and 1.93E-02 respectively for A549, GM12878,
Hep-G2, H1-hESC and Hela), thereby leading to the conclusion that median of
these distributions differ and these are not random occurrences.

TABLE II: SUMMARY OF PERFORMANCES OF DNABERT-CAP AND STATE-OF-THE-ART PREDICTORS Model | Accuracy (%) | Recall (%) | Specificity (%) | MCC
---|---|---|---|---
DNABERT-Cap | 84.21 | 80.99 | 84.20 | 0.686
DeepARC | 83.10 | 82.02 | 84.19 | 0.664
DeepTF | 80.98 | 77.44 | 81.36 | 0.632
CNN-Zeng | 79.92 | 72.12 | 81.96 | 0.619
DeepBind | 79.82 | 72.64 | 81.44 | 0.609

Figure 3: Chart depicting the area under the ROC curve of DNABERT-Cap and the
four selected state-of-the-art predictors

The improved performance of the proposed DNABERT-Cap can be attributed to the
following:

  1. 1.

The ability of DNABERT embeddings to generate rich bidirectional contextual
representations by having multiple attention heads focusing on various input
sections concurrently;

  2. 2.

The ability of capsule network in keeping information about the location of an
object as opposed to a traditional convolutional network that loses track of
it when the pooling layers only extract the most important information from
data;

  3. 3.

The proposed model benefits from the joint optimisation of the aforementioned
DNABERT and capsule layers along with convolutional and bidirectional LSTM
layers to learn important attributes and features for TFBSs. Similar to how
spatial correlation is crucial for correctly identifying objects in images,
such kind of orientation in terms of local ordering of k-mers and its semantic
representation are important for DNA sequences as well. The proposed model
seems to identify such relationships and thus perform better predictions.

##  IV Conclusion

In this work we propose DNABERT-Cap for the identification of transcription
factor binding sites in DNA sequences. We show here that such combination may
have an important role in exploiting data of high complexity. In order to
learn characteristics specific to DNA sequences and TFBSs, DNABERT-Cap relies
on the beneficial strength of combining a pre-trained DNABERT model and a
capsule layer. The proposed model is also compared with state-of-the-art
approaches and shows an accuracy of more than 83% and an AUC of more than 0.91
for all the five cell lines as considered in this work. As a future research
work, attempts can be made to further improve the performance of the model by
not only considering DNA sequences for feature embedding but also other
parameters as well. Moreover, DNABERT-Cap can be used to address other
prediction problems such as predicting RNA-binding sites and DNA-protein
binding sites from protein sequences. We hope that this work will be useful
for the researchers working on applications of deep learning models in
computational biology.

## Acknowledgments

The work presented in this paper has been supported by PNRR MUR project
PE0000013-FAIR. The authors show their gratitude to the Italian CINECA
Consortium for graciously providing the required computational resources.

## References

  * [1] D. S. Latchman, Transcription factors: An overview, The International Journal of Biochemistry & Cell Biology 29 (12) (1997) 1305–1312.  doi:10.1016/S1357-2725(97)00085-X.
  * [2] M. Karin, Too many transcription factors: positive and negative interactions, The New biologist 2 (2) (1990) 126–131.
  * [3] M. Tompa, N. Li, T. L. Bailey, et al., Assessing computational tools for the discovery of transcription factor binding sites, Nature Biotechnology 23 (2005) 137–144.  doi:10.1038/nbt1053.
  * [4] G. Tan, B. Lenhard, Tfbstools: an r/bioconductor package for transcription factor binding site analysis, Bioinformatics 32 (10) (2016) 1555–1556.  doi:10.1093/bioinformatics/btw024.
  * [5] K. Qu, L. Wei, Q. Zou, A review of dna-binding proteins prediction methods, Current Bioinformatics, 14 (3) (2019) 246–254.  doi:10.2174/1574893614666181212102030.
  * [6] B. S. Alexandrov, V. Gelev, S. W.Yoo, et al., Dna dynamics play a role as a basal transcription factor in the positioning and regulation of gene transcription initiation, Nucleic Acids Research 38 (6) (2010) 1790–1795.  doi:10.1093/nar/gkp1084.
  * [7] J. Li, J. h. Ou, Differential regulation of hepatitis b virus gene expression by the sp1 transcription factor, Journal of Virology 75 (18) (2001) 8400–8406.  doi:10.1128/JVI.75.18.8400-8406.2001.
  * [8] A. C. Wilkinson, H. Nakauchi, B. Göttgens, Mammalian transcription factor networks: Recent advances in interrogating biological complexity, Cell Systems 5 (4) (2017) 319–331.  doi:10.1016/j.cels.2017.07.004.
  * [9] S. A. Lambert, A. Jolma, L. F. Campitelli, et al., The human transcription factors, Cell 172 (4) (2018) 650–665.  doi:10.1016/j.cell.2018.01.029.
  * [10] S. Basith, B. Manavalan, T. H. Shin, et al., iGHBP: Computational identification of growth hormone binding proteins from sequences using extremely randomised tree, Computational and Structural Biotechnology Journal 16 (2018) 412–420.  doi:10.1016/j.csbj.2018.10.007.
  * [11] Z. Shen, Y. Lin, Q. Zou, Transcription factors–DNA interactions in rice: identification and verification, Briefings in Bioinformatics 21 (3) (2019) 946–956.  doi:10.1093/bib/bbz045.
  * [12] O. Fornes, J. A. Castro-Mondragon, A. Khan, et al., JASPAR 2020: update of the open-access database of transcription factor binding profiles, Nucleic Acids Research 48 (D1) (2019) D87–D92.  doi:10.1093/nar/gkz1001.
  * [13] V. Matys, O. V. Kel-Margoulis, E. Fricke, et al., TRANSFAC and its module TRANSCompel: transcriptional gene regulation in eukaryotes, Nucleic Acids Research 34 (2006) D108–110.  doi:10.1093/nar/gkj143.
  * [14] K.-C. Wong, T.-K. Chan, C. Peng, et al., DNA motif elucidation using belief propagation, Nucleic Acids Research 41 (16) (2013) e153–e153.  doi:10.1093/nar/gkt574.
  * [15] M. Ghandi, D. Lee, M. M. Mohammad-Noori, et al., Enhanced regulatory sequence prediction using gapped k-mer features, PLOS Computational Biology 10 (7) (2014) 1–15.  doi:10.1371/journal.pcbi.1003711.
  * [16] J. Devlin, M.-W. Chang, K. Lee, et al., Bert: Pre-training of deep bidirectional transformers for language understanding (2019).  arXiv:1810.04805.
  * [17] K. He, X. Zhang, S. Ren, et al., Identity mappings in deep residual networks, in: Computer Vision – ECCV 2016, 2016, pp. 630–645.
  * [18] K. He, X. Zhang, S. Ren, et al., Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [19] H. Zhao, Z. Tu, Y. Liu, et al., PlantDeepSEA, a deep learning-based web service to predict the regulatory effects of genomic variants in plants, Nucleic Acids Research 49 (W1) (2021) W523–W529.  doi:10.1093/nar/gkab383.
  * [20] S. Min, H. Kim, B. Lee, et al., Protein transfer learning improves identification of heat shock protein families, PLOS ONE 16 (5) (2021) 1–14.  doi:10.1371/journal.pone.0251865.
  * [21] Y. Liu, Y.-H. Zhu, X. Song, et al., Why can deep convolutional neural networks improve protein fold recognition? a visual explanation by interpretation, Briefings in Bioinformatics 22 (5) (2021).  doi:10.1093/bib/bbab001.
  * [22] B. Alipanahi, A. Delong, M. T. Weirauch, et al., Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning, Nature Biotechnology 33 (2015) 831–838.  doi:10.1038/nbt.3300.
  * [23] H. Zeng, M. D. Edwards, G. Liu, et al., Convolutional neural network architectures for predicting dna–protein binding, Bioinformatics 32 (12) (2016) i121–i127.  doi:10.1093/bioinformatics/btw255.
  * [24] H. R. Hassanzadeh, M. D. Wang, DeeperBind: Enhancing prediction of sequence specificities of DNA binding proteins, in: 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2016, pp. 178–183.
  * [25] D. Quang, X. Xie, DanQ: a hybrid convolutional and recurrent deep neural network for quantifying the function of DNA sequences, Nucleic Acids Research 44 (11) (2016) e107–e107.  doi:10.1093/nar/gkw226.
  * [26] A. Farrel, J. t. Guo, An efficient algorithm for improving structure-based prediction of transcription factor binding sites, BMC Bioinformatics volume 18 (342) (2017).  doi:10.1186/s12859-017-1755-0.
  * [27] Q. Qin, J. Feng, Imputation for transcription factor binding predictions based on deep learning, PLOS Computational Biology 13 (2) (2017) 1–20.  doi:10.1371/journal.pcbi.1005403.
  * [28] S. Salekin, J. M. Zhang, Y. Huang, Base-pair resolution detection of transcription factor binding site by deep deconvolutional network, Bioinformatics 34 (20) (2018) 3446–3453.  doi:10.1093/bioinformatics/bty383.
  * [29] N. K. Lee, F. L. Azizan, Y. S. Wong, et al., DeepFinder: An integration of feature-based and deep learning approach for DNA motif discovery, Biotechnology & Biotechnological Equipment 32 (3) (2018) 759–768.  doi:10.1080/13102818.2018.1438209.
  * [30] Y. Zhang, S. Qiao, S. Ji, et al., Identification of DNA–protein binding sites by bootstrap multiple convolutional neural networks on sequence information, Engineering Applications of Artificial Intelligence 79 (2019) 58–66.  doi:10.1016/j.engappai.2019.01.003.
  * [31] Y. Zhang, S. Qiao, S. Ji, et al., DeepSite: bidirectional LSTM and CNN models for predicting DNA–protein binding, International Journal of Machine Learning and Cybernetics volume 11 (2020) 841–851.  doi:10.1007/s13042-019-00990-x.
  * [32] J. Yang, A. Ma, A. D. Hoppe, et al., Prediction of regulatory motifs from human Chip-sequencing data using a deep learning framework, Nucleic Acids Research 47 (15) (2019) 7809–7824.  doi:10.1093/nar/gkz672.
  * [33] C. Chen, J. Hou, X. Shi, et al., DeepGRN: prediction of transcription factor binding site across cell-types using attention-based deep neural networks, BMC Bioinformatics 22 (38) (2021).  doi:10.1186/s12859-020-03952-1.
  * [34] X.-R. Bao, Y.-H. Zhu, D.-J. Yu, DeepTF: Accurate prediction of transcription factor binding sites by combining multi-scale convolution and long short-term memory neural network, in: Intelligence Science and Big Data Engineering. Big Data and Machine Learning, 2019, pp. 126–138.
  * [35] Y. Zhang, S. Qiao, Y. Zeng, et al., CAE-CNN: Predicting transcription factor binding site with convolutional autoencoder and convolutional neural network, Expert Systems with Applications 183 (2021) 115404.  doi:10.1016/j.eswa.2021.115404.
  * [36] F. Jing, S.-W. Zhang, S. Zhang, Prediction of the transcription factor binding sites with meta-learning, Methods 203 (2022) 207–213.  doi:10.1016/j.ymeth.2022.04.010.
  * [37] S. Wang, Q. Zhang, Z. Shen, et al., Predicting transcription factor binding sites using DNA shape features based on shared hybrid deep learning architecture, Molecular Therapy - Nucleic Acids 24 (2021) 154–163.  doi:10.1016/j.omtn.2021.02.014.
  * [38] L. Cao, P. Liu, J. Chen, et al., Prediction of transcription factor binding sites using a combined deep learning approach, Frontiers in Oncology (06 2022).  doi:10.3389/fonc.2022.893520.
  * [39] J. Kim, S. Jang, E. Park, et al., Text classification using capsules, Neurocomputing 376 (2020) 214–221.  doi:10.1016/j.neucom.2019.10.033.
  * [40] B. Chen, Z. Xu, X. Wang, et al., Capsule network-based text sentiment classification, IFAC-PapersOnLine 53 (5) (2020) 698–703.  doi:10.1016/j.ifacol.2021.04.160.
  * [41] T. Saha, S. R. Jayashree, S. Saha, et al., Bert-caps: A transformer-based capsule network for tweet act classification, IEEE Transactions on Computational Social Systems 7 (5) (2020) 1168–1179.  doi:10.1109/TCSS.2020.3014128.
  * [42] The encode project consortium. an integrated encyclopedia of dna elements in the human genome, Nature 489 (2012) 57–74.  doi:10.1038/nature11247.
  * [43] Y. Ji, Z. Zhou, H. Liu, et al., DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome, Bioinformatics 37 (15) (2021) 2112–2120.  doi:10.1093/bioinformatics/btab083.
  * [44] G. E. Hinton, S. Sabour, N. Frosst, Matrix capsules with em routing, in: International conference on learning representations, 2018.
  * [45] S. Sabour, N. Frosst, G. E. Hintons, Dynamic routing between capsules, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017, p. 3859–3869.
  * [46] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, in: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [47] H. Zeng, M. D. Edwards, G. Liu, et al., Convolutional neural network architectures for predicting dna–protein binding, Bioinformatics 32 (12) (2016) i121–i127.  doi:10.1093/bioinformatics/btw255.