Advance Access Publication Date: Day Month Year Manuscript Category

∗To whom correspondence should be addressed. ${\dagger}$ These authors
contributed equally to this work.

# Drug-Target Interaction Prediction with Graph Attention networks

Subject Section

Haiyang Wang ${}^{\text{\sfb$\dagger$,1}}$ Guangyu Zhou
${}^{\text{\sfb$\dagger$,2}}$ Siqi Liu ${}^{\text{\sfb 2}}$ Jyun-Yu Jiang
${}^{\text{\sfb 2}}$ and Wei Wang ${}^{\text{\sfb 2,}*}$ ${}^{\text{\sf
1}}$Zhiyuan College, Shanghai Jiao Tong University, Shanghai, 200240, China
and
${}^{\text{\sf 2}}$Department of Computer Science, University of California,
Los Angeles, 90095, USA

(2020; 2020)

###### Abstract

Motivation: Predicting Drug-Target Interaction (DTI) is a well-studied topic
in bioinformatics due to its relevance in the fields of proteomics and
pharmaceutical research. Although many machine learning methods have been
successfully applied in this task, few of them aim at leveraging the inherent
heterogeneous graph structure in the DTI network to address the challenge. For
better learning and interpreting the DTI topological structure and the
similarity, it is desirable to have methods specifically for predicting
interactions from the graph structure.
Results: We present an end-to-end framework, DTI-GAT (Drug-Target Interaction
prediction with Graph ATtention networks) for DTI predictions. DTI-GAT
incorporates a deep neural network architecture that operates on graph-
structured data with the attention mechanism, which leverages both the
interaction patterns and the features of drug and protein sequences. DTI-GAT
facilitates the interpretation of the DTI topological structure by assigning
different attention weight to each node with the self-attention mechanism.
Experimental evaluations show that DTI-GAT outperforms various state-of-the-
art systems on the binary DTI prediction problem. Moreover, the independent
study results further demonstrate that our model can be generalized better
than other conventional methods.
Availability: The source code and all datasets are available at
https://github.com/Haiyang-W/DTI-GRAPH
Contact: wanghaiyang<EMAIL_ADDRESS>weiwang<EMAIL_ADDRESS>

##  1 Introduction

Detecting drug–target interactions (DTIs) potentially facilitates therapeutic
target identification (Xia et al., 2010; Petta et al., 2016) and novel drug
design (Skrabanek et al., 2008; AY et al., 2007; Janga and Tzakos, 2009; Kuhn
et al., 2008). Until quite recently, pharmacological effects were often
discovered using primitive trial and error procedures, such as applying plant
extracts on living systems and observing the outcomes (Singh et al., 2016).
However, experiment-based methods remain expensive, labor-intensive and time-
consuming (Dickson and Gagnon, 2004; Kola and Landis, 2004; Kapetanovic,
2008). Evidently, there is an immense need for reliable computational
approaches to identify and characterize DTIs, hoping to accelerate the pace
and reduce the cost of drug development.

With the rapid development of machine learning techniques, various
computational prediction approaches have been proposed to predict drug-target
interactions. Yamanishi et al. (2010) integrated the relationship among the
pharmacological space, the chemical space, and the topology of drug–target
interaction networks to predict the associations between drugs and targets,
and their experimental results have demonstrated that drug–target interactions
are more correlated with pharmacological effect similarity than with chemical
structure similarity. According to the similarity of chemical information,
Keiser et al. (2009) proposed a method to explore the associations between
drugs and targets. They selected 30 of predicted results for biological
experiments and finally confirmed 23 with interrelationships. Wang et al.
(2010) used supervised machine learning methods to predict the relationship
between drugs and targets. To solve the problem of sample imbalance, they are
collecting the positive samples from the database, and the negative samples
using the random selection method. The input features of the classifier
consist of the chemical structure of the drug and the sequence information of
the protein. Chen et al. (2012) developed a novel method of Network-based
Random Walk with Restart on the Heterogeneous (NRWRH) network to predict
potential drug–target interactions on a large scale. The excellent
experimental results show that the proposed method can discover new potential
drug–target interactions for drug development. These approaches provide
feasible solutions to the problem. However, the extracted features used in
these approaches only have limited coverage on interaction information, since
they are dedicated to specific facets of the protein and the drug profiles.

To alleviate the inadequacy of statistical learning methods, deep learning
algorithms provide the powerful functionality to represent large-scale raw
data for different tasks and thus facilitate the learning of latent patterns
in the data (LeCun et al., 2015). Recently, deep learning architectures have
produced powerful systems to address several estimation problems related to
protein sequences, such as protein-protein interaction (Chen et al., 2019),
protein binding affinity upon mutation (Zhou et al., 2020), and protein
structural changes (Senior et al., 2020) estimation. These works typically use
convolutional neural networks (CNNs) for automatically selecting local
features, recurrent neural networks (RNN) that aim at preserving the
contextualized and long-term ordering information or the combination of both
CNNs and RNNs. In contrast, fewer efforts have been made to capture the
pairwise interactions of the protein drug interaction with deep learning,
which remains a non-trivial problem with the following challenges: (i)
Characterization of the proteins and drugs requires a model to effectively
filter and aggregate their local features, while preserving significant
contextualized and sequential information of the amino acids and drug
fingerprints; (ii) Constructing a deep neural architecture without biological
insights often suffers from the interpretation issue; (iii) An effective
mechanism is also needed to apprehend the mutual influence of protein-drug
pairs in DTI prediction. Moreover,the framework needs to be scalable to large
data. Corresponding methods, including DeepDTI (Wen et al., 2017), RFDT (Wang
et al., 2018a), DeepDTA (Öztürk et al., 2018) and DeepConv-DTI (Lee et al.,
2019), employed deep neural networks on DTI prediction tasks. Specifically,
DeepDTI built by Wen et al. (2017) used the deep belief network (DBN), with
features such as the composition of amino acids, dipeptides, and tripeptides
for proteins and fingerprints for drugs. RFDT by Wang et al. (2018a) employed
stacked Auto-Encoder (AE) to abstract original features into a latent
representation with a small dimension. With latent representation, they
trained a Random Forest (RF), which performed better than previous methods.
DeepDTA (Öztürk et al., 2018) and DeepConv-DTI (Lee et al., 2019) both used
CNN to extract local residue patterns to predict the binding affinity between
drugs and targets. They performed convolution on various lengths of the
subsequences’ amino acids to capture local residue patterns of generalized
protein classes.

The previous DTI prediction methods can be mainly separated into two
categories: similarity-based methods and feature-based methods. Similarity-
based methods assume that similar drugs or proteins may have similar
interaction patterns. These methods use many different similarity measures
based on fingerprint, chemical structure, sequence data and so on to identify
drug-target interaction. Feature-based methods solve drug–target interaction
prediction as a binary classification problem, such as DeepConv-DTI (Lee et
al., 2019), use known drug-target pairs as positive sample. To combine both
the similarity and feature vectors, the neural network models require a
mechanism to represent both the interaction patterns and the features of drug
and protein sequences.

In this paper, we introduce DTI-GAT (Drug-Target Interaction prediction with
Graph ATtention networks), a deep neural network architecture that operates on
graph-structured data with attention mechanism. First, it converts the
position-specific scoring matrix (PSSM) and the drug fingerprint as feature
vectors for all the target proteins and drugs. Secondly, a DTI graph is
constructed based on the similarity of feature vectors. Each node represents a
protein or a drug and the nodes are lined by edges representing interactions
between protein-drug, protein-protein and drug-drug separately. The graph
attention network is then applied on the built graph to generate embeddings
for each protein and drug, followed by a final decoder architecture to predict
the interaction result. Notably, based on the assumption that different
neighbors’ importance are different and the DTI graph is heterogeneous, we
don’t know which neighbor or domain knowledge (protein or drug) is more
important for a given node in the DTI prediction task. The Graph attention
(GAT) layer applies self-attention mechanism, which allows for assigning
different attentional weight to neighbor nodes, enabling a leap in model
capacity. Furthermore, analyzing the learned weights with attention can lead
to benefits in interpreting the DTI topological structure and similarity.

Our contributions are 3-fold. First, we provide an approach to transform the
feature representations of proteins and drugs into a protein-drug interaction
graph. We emphasize the need to extend the bipartite graph to a heterogeneous
graph, by adding drug-drug and protein-protein similarities. Second, we
demonstrate that the attention mechanism can automatically extract the
important high-level relationships among proteins and drugs by assigning
different weights to each edge and drop all structural information. Third, we
provide an extensive analysis to show the better interpretability of learned
attention weights for representing the topological structure and similarity.
Last, DTI-GAT is highly efficient, as the operation of the self-attentional
layer can be parallelized across all edges, and the computation of output
features can be parallelized across all nodes. DTI-GAT significantly
outperforms various state-of-the-art approaches on the DTI binary prediction
task, which confirms the effectiveness of the graph attention strategy in
identifying drug-target interactions.

##  2 Methods

###  2.1 Preliminary

####  2.1.1 Protein feature representation

In this work, protein sequences are represented as pseudo-position specific
scoring matrix (PsePSSM) features to encode the evolution and sequential
information for proteins with different lengths of sequences. Note that this
setting is consistent with previous studies (Shi et al., 2019).

For a target protein sequence $p_{m}$ with $L$ amino acid residues, we use the
position-specific scoring matrix (PSSM) as its descriptor introduced by Jones
(1999). The PSSM with a dimension of L $\times$ 20 can be expressed as:

$PSSM(p_{m})=\begin{pmatrix}E_{1\rightarrow{1}}&E_{1\rightarrow{2}}&\cdots&E_{1\rightarrow{20}}\\\
E_{2\rightarrow{1}}&E_{2\rightarrow{2}}&\cdots&E_{2\rightarrow{20}}\\\
\vdots&\vdots&\ddots&\vdots\\\
E_{L\rightarrow{1}}&E_{L\rightarrow{2}}&\cdots&E_{L\rightarrow{20}}\end{pmatrix},$

where each element $E_{i\rightarrow{j}}$ in the PSSM matrix is then normalized
to the interval (0, 1) as:

$\overline{E}_{i\rightarrow{j}}=\frac{1}{1+exp(E_{i\rightarrow{j}})}$ (1)

To make the PSSM descriptor a uniform representation despite proteins with
different lengths correspond to different numbers of rows, we represent the
uniformed PSSM of $p_{m}$ as:

$\overline{PSSM}(p_{m})=[\overline{E}_{1},\overline{E}_{2},...,\overline{E}_{20}]^{T},$
(2)

where $T$ is the transpose operator. Here,
$\overline{E}_{j}=\frac{1}{L}\sum_{i=1}^{L}\overline{E}_{i\rightarrow{j}}$
computes the average score of the residue in protein $p_{m}$ during the
process of evolution, which is the mutation from amino acid type $i$ to $j$.
To retain the sequence information after Eq. 2, the pseudo-position specific
scoring matrix (PsePSSM) for protein $p_{m}$ is computed as:

$P_{\text{Pse}(p_{m})}^{\lambda}=[\overline{E}_{1},...,\overline{E}_{20},G_{1}^{1},...,G_{20}^{1},...,G_{1}^{\lambda},...G_{20}^{\lambda}]^{T},$
(3)

where

$G_{j}^{\lambda}=\frac{1}{L-\lambda}\sum_{i=1}^{L-\lambda}[\overline{E}_{i\rightarrow{j}}-\overline{E}_{(i+\lambda)\rightarrow{j}}]^{2},(j=1,...,20;0\leq\lambda\leq
L).$ (4)

Here, $G_{j}^{\lambda}$ is the correlation factor of $j^{th}$ amino acid and
$L$ is the continues distance along the protein sequence. Therefore, a protein
sequence can be expressed as Eq. 3 using PsePSSM and generates a
$20+20\times\lambda$ dimensional feature vector.

####  2.1.2 Drug feature representation

It has been indicated in some studies that descriptors as molecular structure
fingerprints can effectively represent the drug (Ding et al., 2017; Yamanishi
et al., 2011). For drug molecules, we use the chemical structure of the
molecular substructure fingerprints from the PubChem database (Kim et al.,
2019). For each drug molecule, it defines an 881- dimensional binary vector
$Q$ to represent the molecular substructure, where the corresponding bits of
the vector are encoded as 1s for existence of substructures and 0s for
absence. Therefore, given a drug $d_{n}$, its fingerprint feature is
calculated as $Q(d_{n})=[q1(d_{n}),q2(d_{n}),...q881(d_{n})]$ Fingerprints
property is ‘‘PUBCHEM_CACTVS_SUBSKEYS’’ in PubChem SDF files and is Base64
encoded, which provides a textual description by the binary data. The drug
feature representations in this work are also consistent with the previous
study (Wang et al., 2018a).

####  2.1.3 Problem statement

Given a set of $N$ proteins $P=(p_{1},p_{2},...,p_{N})$ and a set of $M$ drugs
$D=(d_{1},d_{2},...,d_{M})$, our goal is to predict the interaction ($I$)
between $p_{m}$ and $d_{n}$ based on the the protein’s PsePSSM:
$P_{\text{Pse}(p_{m})}^{\lambda}$ and the drug’s fingerprint: $Q(d_{n})$,
where $p_{m}\in P$ and $d_{n}\in D$.

$I(p_{m},d_{n})=\left\\{\begin{array}[]{lr}1,\quad interaction&\\\ 0,\quad
no\quad interaction&\\\ \end{array}\right.$ (5)

###  2.2 DTI-GAT

We introduce DTI-GAT, a deep neural network architecture that operates on
graph-structured data with attention mechanism for the feature-based DTI
prediction task. The overall learning architecture is illustrated in Figure 1.

Figure 1: Architecture of DTI-GAT.

DTI-GAT consists of the following main components: First, given the pseudo-
position specific scoring matrix (PsePSSM) of each protein and the fingerprint
of each drug, a DTI graph is constructed based on the information about
interaction target and similarity of local structure. Then, the graph
attention network is applied to the built graph to integrate higher-level
information and generate embeddings for each protein and drug. A final decoder
architecture takes a pair of protein-drug embeddings and predicts the final
interaction probability.

####  2.2.1 Graph construction

Given a protein set $P=(p_{1},p_{2},...,p_{N}),p_{i}\in R^{p}$, drug set
$D=(d_{1},d_{2},...,d_{M}),d_{i}\in R^{d}$, and interaction targets set
$T=(t_{1},t_{2},...,t_{k}),t_{i}\in\\{1,-1\\}$, where $p$ is the PsePSSM
feature dimension, $d$ is the fingerprint dimension, we construct a DTI-Graph
$G(V,E)$.

First, we transform the PsePSSM and the fingerprint to the same dimension, so
that nodes can be easily aggregated. In our experiment, we use two weight
matrices, $W_{p}\in R^{p\times F}$ and $W_{d}\in R^{d\times F}$ for
transforming protein and drug separately, where F is the feature dimension,
and $ReLU$ nonlinearity to transform the feature dimension.

$p^{{}^{\prime}}_{i}=ReLU(W_{p}\cdot p_{i}),\ p^{{}^{\prime}}_{i}\in R^{F}$
(6) $d^{{}^{\prime}}_{i}=ReLU(W_{d}\cdot d_{i}),\ d^{{}^{\prime}}_{i}\in
R^{F}$ (7)

Once obtained, then we combine the new protein set
$P^{{}^{\prime}}=(p^{{}^{\prime}}_{1},p^{{}^{\prime}}_{2},...,p^{{}^{\prime}}_{N}),p^{{}^{\prime}}_{i}\in
R^{F}$ and the new drug set
$D=(d^{{}^{\prime}}_{1},d^{{}^{\prime}}_{2},...,d^{{}^{\prime}}_{M}),d^{{}^{\prime}}_{i}\in
R^{F}$ to the graph node set $H=(h_{1},h_{2},...,h_{K}),h_{i}\in R^{F}$, where
each node is represented as a vector. These nodes are linked by undirected
edges, $e\in\\{1,-1\\}$, which are generated from the drug-protein
interaction, drug-drug and protein-protein similarity.

The edge set, $E$, contains two different components: interaction edge and
similarity edge. The interaction edge, $e_{i}$, is either $1$ or $-1$,
referring positive relation and negative relation of two nodes. Since the
positive relation is from the ground truth interaction targets set, $T$, the
initial graph of the DTI is very sparse and has an imbalanced issue. To solve
the above issues, the negative samples are selected randomly from the
unidentified drug–target pairs. We assume that the positive sample is a small
percentage of all possible samples, so there is a low probability that real
interaction will be selected as a negative sample. Actually, the proportions
of positive samples detected in each dataset are $0.99\%$(Enzymes),
$3.49\%$(Ion channels), $2.99\%$(GPCR) and $6.40\%$(Nuclear receptors). In the
experiment, we choose a negative sample with the same number of positive
samples. However, the initial interaction edges are only between the drug-
protein pairs, which construct a bipartite graph that limit the information
flow. In order to aggregate more information, we transform the bipartite graph
to a heterogeneous graph, by adding drug-drug and protein-protein
similarities. Similarity edge, $e_{s}$, is based on the DTI bipartite graph
and its common neighbor information. If the number of two nodes’ common
positive neighbor or negative neighbor is greater than a threshold, the two
nodes will be connected by $1$, which means they are similar.

$e_{s}(i,j)=\left\\{\begin{array}[]{lr}1,\quad if\
com\\_n(h_{i},h_{j})\textgreater\theta&\\\ 0,\quad if\
com\\_n(h_{i},h_{j})\leq\theta&\\\ \end{array}\right.$ (8)

where $com\\_n$ is the function to compute the common neighbor of two nodes
and $\theta$ is the threshold. In our experiment, we use adjacent matrix $A\in
R^{K\times K}$ to represent $E$, where $K$ is the number of node.

The interaction edge connects the protein-drug domain, similarity edge
connects protein-protein domain and drug-drug domain separately, that allows
information to flow across the entire DTI graph. What’s more, positive edge
(e.g. 1) means to pull two nodes closer and negative (e.g. -1) means to push
them apart.

####  2.2.2 Graph attention network

Our approach uses Graph Attention Network (GAT)(Veličković et al., 2017) to
adaptively learn weights for each edge and represent each node by message
passing.

The input of GAT is a set of node feature $H=(h_{1},h_{2},...,h_{K}),h_{i}\in
R^{F}$ and DTI adjacent matrix $A$. $H$ contains
$H_{p}=(h_{p_{1}},h_{p_{2}},...,h_{p_{m}})$ and
$H_{d}=(h_{d_{1}},h_{d_{2}},...,h_{d_{n}})$ and $A$ is generated by 2.2.1.

The output is a new set of node feature,
$H^{{}^{\prime}}=(h^{{}^{\prime}}_{1},h^{{}^{\prime}}_{2},...,h^{{}^{\prime}}_{K}),h^{{}^{\prime}}_{i}\in
R^{F^{{}^{\prime}}}$.

In order to transform input features to higher level features, we apply a
weight matrix, $W\in R^{F^{{}^{\prime}}\times F}$, to each node.

$h^{{}^{\prime}}_{i}=\sigma(W\cdot h_{i}),\ h^{{}^{\prime}}_{i}\in
R^{F^{{}^{\prime}}}$ (9)

Then we perform a self-attention mechanism on node pair,
$a:R^{F^{{}^{\prime}}}\times R^{F^{{}^{\prime}}}\rightarrow R$, to compute
attention weight, which indicates the importance of $n_{j}$ to $n_{i}$,
$n_{j}\in N_{i}$, where $N_{i}$ is the neighborhood of $n_{i}$ and itself in
DTI graph.

$w_{ij}=a(h^{{}^{\prime}}_{i},h^{{}^{\prime}}_{j}),\ w_{ij}\in R^{1}$ (10)

In GAT layer, the attention mechanism $a$ is a single-layer neural network,
parametrized by a weight matrix $\widetilde{a}\in R^{2F^{{}^{\prime}}}$. Then
the LeakyReLU nonlinearity is applied.

$w_{ij}=LeakyReLU(\widetilde{a}^{T}[h^{{}^{\prime}}_{i}||h^{{}^{\prime}}_{j}])$
(11)

where $.^{T}$ is transposition and $||$ is the concatenation operation.

In the general formulation, attention mechanism allows every node to attend on
every other node, dropping all structural information. To add graph structure
information, we perform mask attention according to the DTI adjacent matrix
$A$, which enables only the neighbor nodes to be attended. Then we normalize
the attention weight across all choices of $j$ using the softmax function to
make it comparable of different nodes.

$\alpha_{ij}=Softmax_{j}(w_{ij})=\frac{exp(w_{ij})}{\sum_{k\in
N_{i}}exp(w_{ik})}$ (12)

where $N_{i}$ is the set of $h_{i}$’s neighborhood and itself.

After obtaining the normalized attention score, we use message passing to
compute a linear combination of the node features and output the final
aggregated features of each node.

$h_{i}^{{}^{\prime}}=\sigma(\sum_{j\in N_{i}}\alpha_{ij}Wh_{j})$ (13)

where $\sigma$ is a nonlinearity, $H^{{}^{\prime}}$ is final aggregated
features.

####  2.2.3 Final decoder

The final decoder is a sample neural network, parametrized by a weight matrix
$W\in R^{2F^{{}^{\prime}}}$. It takes pairs of drug-protein embeddings,
generated by GAT layer(e.g. $h_{i}$ and $h_{j}$), as input. Then the two node
vector do an element-wise multiplication, $p\odot d\rightarrow v,\ v,p,d\in
R^{F}$. Finally, through a layer of neural network $v^{F}\rightarrow R^{1}$
and a Sigmoid activate function, produce a probability score indicating
whether they interact:

$s_{ij}=Sigmoid(ReLU(W(h^{{}^{\prime}}_{i}\odot h^{{}^{\prime}}_{j})))$ (14)

####  2.2.4 Loss function

For training DTI-GAT, we assign a binary class label ${0,1}$ (interact or not)
to each identified drug-protein pair. We assign positive labels to interacted
drug-protein pairs and negative labels to non-interacted drug-protein pairs.
The output is the interaction probability. The main learning objective is to
minimize the following binary cross entropy loss (BCELoss) between the target
$X=(x_{1},x_{2},...,x_{N}),x_{i}\in\\{0,1\\}$ and the output
$Y=(y_{1},y_{2},...,y_{N}),y_{i}\in(0,1)$. The loss can be described as:

$l(x,y)=\\{l_{1},l_{2},...,l_{N}\\}^{T}$ (15) $l_{n}=-w_{n}[y_{n}\cdot log\
x_{n}+(1-y_{n})\cdot log\ (1-x_{n})]$ (16)

####  2.2.5 Implementation details

In this experiment, we use PyTorch (Paszke et al., 2019) for efficient GPU-
based implementation and PyTorch geometric (Fey and Lenssen, 2019) for a
sparse Graph Attention Layer.

The DTI-GAT contains the feature (PsePSSM, Fingerprint) encoder, the GAT
embedding module and a final interaction decoder. Feature encoder is
implemented as two MLP layers for protein and drug respectively. The input
dimensions are 220 (PsePSSM) and 881 (fingerprint) and the output dimensions
are both 256. The GAT embedding module consists of a Multi-head GAT layer,
based on PyTorch geometric (Fey and Lenssen, 2019). The hidden dimension and
output dimension both are set to 256 for all datasets. In building the
adjacency matrix, the setting of the common neighbor threshold is different in
different datasets, it is set to 1 in Nuclear Receptor, 3 in GPCR, Ion Channel
and Enzyme dataset. The final interaction decoder includes three MLP layers,
which input dimension $2\times 256=512$ and the hidden dimension is also 256.
The output is a scalar, then be computed by a sigmoid function to get the
final interaction probability.

In our paper, the DTI-GAT are trained end-to-end using gradient descent based
on Adam optimizer (Kingma and Ba, 2014). We perform batch gradient descent
using the full dataset for every training iteration, which is a viable option
as long as datasets fit in memory. Base on the implementation of PyTorch
Geometric (Fey and Lenssen, 2019), we use a sparse representation for A. The
memory requirement is $\mathcal{O}(|\varepsilon|)$, which is linear in the
number of edges. The learning rate $\alpha$ is 0.0005 and the weight decay is
$5e-4$. Stochasticity in the training process is by dropout (Srivastava et
al., 2014), which is set to 0.3. The model is trained until converging for
each fold of the cross-validation, for which we set the epoch number to be
4000, 8000, 10000, 15000 for Nuclear Receptor, GPCR, Ion Channel and Enzyme,
respectively.

##  3 Results

Table 1: Number of drugs, target proteins, and their interactions for 5 datasets used for cross-validation. | Enzyme | IC | GPCR | NR | Drugbank_approved
---|---|---|---|---|---
Drugs | 445 | 210 | 223 | 54 | 1555
Targets | 663 | 204 | 95 | 26 | 1591
Interactions | 2925 | 1476 | 635 | 90 | 5831

(a) Enzyme

(b) Ion Channel

(c) GPCR

(d) Nuclear Receptor

Figure 2: Evaluation of DTI prediction on the four datasets based on 5-fold cross-validation under 6 evaluation metrics. We report the mean and standard deviation for the test sets for all methods. Table 2: Evaluation of DTI prediction on the drugbank_approved dataset based on 5-fold cross-validation under 6 evaluation metrics. We report the mean and standard deviation for the test sets. |  | Acc | Sen | Prec | Spec | Mcc | Auc
---|---|---|---|---|---|---|---
SAE+RF* | mean | 82.60 | 82.75 | 82.39 | 82.54 | 66.05 | 88.31
| std | 1.24 | 1.35 | 1.16 | 1.07 | 1.29 | 0.31
DeepDTA* | mean | 82.13 | 83.61 | 81.29 | 80.65 | 64.34 | 87.36
| std | 0.96 | 1.76 | 2.15 | 2.94 | 1.92 | 0.77
DeepConv-DTI | mean | 84.54 | 83.44 | 85.31 | 85.65 | 69.02 | 91.73
| std | 0.65 | 0.72 | 0.83 | 0.98 | 1.21 | 0.34
LRF* | mean | 86.11 | 83.45 | 88.18 | 88.78 | 72.38 | 92.20
| std | 0.45 | 2.40 | 1.13 | 1.56 | 0.77 | 0.75
DTI-GAT | mean | 87.67 | 81.12 | 93.34 | 94.21 | 76.00 | 92.30
| std | 0.95 | 2.24 | 0.60 | 0.65 | 1.69 | 0.83
Figure 3: Performance evaluation on using different hyperparameters for PDI prediction on the Enzyme dataset. The accuracy is reported. Table 3: Evaluation of DTI predictions by different methods on the independent study set. | Acc | Sen | Prec | Spec | Mcc | Auc
---|---|---|---|---|---|---
SAE-RF* | 64.20 | 55.30 | 69.30 | 77.30 | 32.10 | 71.20
DeepDTA* | 64.82 | 67.33 | 59.80 | 62.75 | 29.95 | 73.04
DeepConv-DTI | 68.60 | 58.50 | 73.20 | 78.60 | 37.90 | 74.30
LRF* | 71.38 | 45.89 | 73.19 | 92.37 | 44.02 | 75.70
DTI-GAT | 72.90 | 67.36 | 75.54 | 78.39 | 46.04 | 78.42

(a) Protein nodes grouped by top connected drugs

(b) Drug nodes grouped by top connected proteins

Figure 4: t-SNE visualization of the protein (a) and drug (b) nodes embeddings
generated by DTI-GAT on the Enzyme dataset.

We present the experimental evaluation of the proposed framework on the binary
DTI prediction task. The experiments are conducted on the following datasets.

###  3.1 Datasets

We use the interaction data between drugs and target proteins collected by
Yamanishi et al. (2008), available at
http://web.kuicr.kyoto-u.ac.jp/supp/yoshi/drugtarget. The dataset is collected
from various databases like SuperTarget (Günther et al., 2007), DrugBank
(Wishart et al., 2008), KEGG BRITE (Kanehisa et al., 2006), and BRENDA
(Schomburg et al., 2004). This dataset includes four main subsets: enzymes,
ion channels (IC), G-protein-coupled receptors (GPCR) and nuclear receptors
(NR). The statics of the interaction are shown in Table 1. We obtained the
protein sequence information from UniProt (Consortium, 2019), and the drug
fingerprint data from PubChem (Kim et al., 2019). Note that there are a few
proteins and drugs that have been removed in the newer version of databases,
so we did not include them in our training.

In addition to the above four benchmark datasets, we also use a dataset called
drugbank_approved (Ba-Alawi et al., 2016), which contains all FDA-approved
drugs and their corresponding protein targets in the DrugBank database
(Wishart et al., 2008). After removing the non-existing proteins and drugs,
the processed dataset contains 1555 drugs and 1591 targets.

###  3.2 Evaluation protocol

Following the settings in previous works (Wang et al., 2018b, a; Shi et al.,
2019), we conduct 5-fold cross-validation (CV) on the five datasets of the
enzyme, ion channel, GPCR, nuclear receptor and the drugbank_approved. Under
the 5-fold CV setting, the data is equally divided into 5 non-overlapping
subsets, and each subset has a chance to train and to test the model so as to
ensure an unbiased evaluation. We repeat the prediction model with 10 trials
and record the mean and standard deviation of results. We aggregate fix
metrics on the test cases of each fold, i.e. the overall _accuracy (ACC)_ ,
_precision (PR)_ , _sensitivity (SE)_ , _specificity (SP)_ , _Matthews
correlation coefficient (MCC)_ and _Area under the curve (AUC)_. All these
metrics are preferred to be higher to indicate better performance.

###  3.3 Baseline methods

We compare DTI-GAT with the following two groups of baselines:

  1. 1.

Classical statistical learning models: SVM(Cao et al., 2012), Silico DVM(Li et
al., 2017), and LRF(Shi et al., 2019).

  2. 2.

Deep learning models: SAE+RF (Wang et al., 2018a), DeepDTA Öztürk et al.
(2018), and DeepConv-DTI Lee et al. (2019).

Note that LRF has been modified to perform data augmentation on the training
set only, to ensure validation and training samples are fully separated. Also,
note that DeepDTA is designed for Drug-Target Affinity Regression instead. We
adapt it by using a threshold of 0.5 to do a binary classification in order to
obtain the interaction.

###  3.4 Experimental results

####  3.4.1 Comparison with other methods

For the prediction of drug-target interactions, many prediction methods have
been proposed. Figure 2 details the comparison between DTI-GAT with other
baseline methods on enzymes, ion channels, GPCRs, nuclear receptors and Table
2 shows the results on the larger drugbank_approved datasets.

As shown in Figure 2, in general, the statistical baselines perform better
than the deep learning methods. This shows that deep learning methods are not
able to obtain enough information to generalize. The best performing model
among them is Silico DVM, a statistical learning model that trains a
discriminative vector machine classifier with a local binary pattern generated
from PSSM. Our model DTI-GAT can generalize using additional information from
the graph, thus achieving better performance than the baseline models. DTI-GAT
outperforms Silico DVM on Enzyme, Ion channel and GPCR by 0.014, 0.054, 0.011
on MCC, but Silico DVM outperforms DTI-GAT on nuclear receptor by 0.018 on
MCC. This attribute to the fact that DTI-GAT is able to extract more
generalizable information from the graph of interaction, especially on a
large, dense graph.

Specifically, we can see that for the dataset of the enzyme, the accuracy rate
of DTI-GAT reaches 93.75%, which is higher than other prediction methods. In
addition, the AUC of the prediction model reaches 96.32%, comparable with the
best performing model LRF of 96.64%. In the ion channel dataset, the accuracy
of prediction in this paper is 94.38%, which is also higher than other
prediction methods. Our AUC of 96.51%is the highest among other baselines.
Similarly in the GPCR dataset, DTI-GAT outperforms all baselines in accuracy,
precision, specificity, MCC and AUC. As for the nuclear receptor dataset, our
prediction method accuracy rate is 91.11% and our AUC is 90.49%, slightly
lower than the best performing Silico DVM. This is mainly due to the fact that
the nuclear receptor dataset only contains a very limited number of samples
for training and all the deep learning based methods suffers from it. However,
we still drastically outperform other neural network based methods including
SAE+RF, DeepDTA and DeepConv-DTI.

The results on the drugbank_approved dataset are shown in Table 2. With this
larger dataset, we focus on comparing our model with four other methods
including LRF, DeepConv-DTI, DeepDTA and SAE+RF. We can see from the table
that the performances for all the methods drop by 5% to 10% comparing to the
previous cross-validation results. This is because the larger
drugbank_approved dataset contains more complicated and new protein-drug
interactions, which make the learning process harder. However, the accuracy
rate of DTI-GAT reaches 87.67% and the AUC of DTI-GAT reaches 92.30%, which
are still higher than all the baseline methods and relatively robust in
identifying new interactions.

####  3.4.2 Independent Study

To show the generalizability of DTI-GAT, we evaluate the performance of
different methods on an independent dataset. We construct the independent
dataset using the five datasets we discussed in Table 1. This independent
study dataset consists of a training set of enzyme, IC, GPCR and NR. The
testing set comprises of the drugbank_approve dataset, with the protein-drug
pairs that appear in the training set removed. Since the proteins and drugs
have different names in different sources, we perform the matching and removal
by directly comparing the PsePSSM and the fingerprints. After removing the
similar sequences, we obtain the independent dataset with 1883 drugs and 2048
proteins. Among them, 5113 interactions are used for training and the rest
5315 interactions are used for independent testing. Using this dataset, we can
examine the ability of DTI-GAT to generalize the prediction to a larger
protein network from a subset of networks.

We compare DTI-GAT with four baselines including SAE+RF, DeepDTA, DeepConv-
DTI, and LRF. As shown in Table 3, DTI-GAT outperforms other methods by
achieving the highest accuracy of 72.9%, sensitivity of 67.36%, MCC of 46.04%
and AUC of 78.42%. Notice that LRF achieves high specificity with the cost of
getting very low sensitivity, meaning that there are many false negatives
(predicting many interactions to be non-interactions), which is not optimal
for the DTI prediction task. The comparative results of DTI-GAT on the
independent set demonstrate that DTI-GAT can generalize better than other
methods to drugs and targets that share low sequence similarity with the
training set.

####  3.4.3 Parameter analysis

In our work, the main hyperparameter is the common neighbor threshold,
$\theta$. It is the key factor of similarity edge, $e_{s}$, for which links
the protein and drug domain.

To further discuss the influence of common neighbors, we choose the Enzyme
dataset, which has the most nodes. The result of different settings is
illustrated by Figure 3. $\theta=1$ means once two nodes have common
neighbors, they will be linked. In our observation, there are some clustered
graphs, many different drug nodes are linked to the same protein node. Once
$\theta=1$, these drug nodes will be joined into a fully connected graph, the
DTI graph structure and similarity information will be destroyed. That’s why
the performance of $\theta=1$ is lower than the others. As the $\theta$
increases, the DTI graph includes more and more useful similarity edges, leads
to higher accuracy. What’s more, if we delete all the $e_{s}$, means $\theta$
is infinite, no similarity edge, the score is only 87.01, much lower than
others, that proves it is very important.

####  3.4.4 t-SNE visualization of the node embeddings by DTI-GAT

The effectiveness of the learned feature representations for each node may
also be investigated qualitatively. For this purpose, we provide a
visualization of the t-SNE (Maaten and Hinton, 2008) transformed feature
representations extracted by the output layer of the GAT model on the Enzyme
dataset. The t-SNE visualizations are generated for both the view of protein
nodes and the view of drug nodes as shown in Figure 4(a) and Figure 4(b),
respectively.

To show the 2D-projection of protein features, we choose the top 4 drugs as
labels ordered by their number of connections to all proteins. We then filter
out proteins that are not connecting to any of the selected drugs or
ambiguously connecting to multiple selected drugs. The feature vectors of the
rest proteins outputted by GAT are then transformed into 2D space by t-SNE.
Similarly, we choose the top 4 proteins as labels and conduct the same
filtering process on the drugs to visualize the drug node features.

As shown in Figure 4(a), each dot represents a protein and each color
represents the drug that the protein is interacting with. Most proteins except
the two blue ones are forming discernible clusters. Note that these clusters
correspond to the labels of the common interacting drugs. Similarly in Figure
4(b), each dot represent a drug and each color represent the protein that the
drug is interacting with. We can also see four clusters are formed based on
the learned features of each drug. These two different views of visualizations
in 2D space by t-SNE demonstrate that our learned attended vector can capture
the important interactions while preserving the properties of each node.

To demonstrate the biological interpretability of the learned features, in
addition to the internal distance within each cluster, we also examine the
distance among different clusters. The distance also correlates with the
pathway of the labeled proteins. For example, in Figure 4(b), the blue hsa:759
cluster has only metabolic pathways, and the green hsa:43 and red hsa:4129
clusters both participate in metabolism and other functions. These 3 clusters
share the metabolic pathway. The purple one, hsa:1636 instead has pathways in
the Renin system. Therefore, hsa:759 is closer to hsa:43 and hsa:4129
comparing to the distance between hsa:759 and hsa:1636.

####  3.4.5 Attention weight analysis

Figure 5: Attention graph of DTI-GAT on NUC.

To further illustrate the effectiveness of the graph attention approach, we
also compute an attention graph from the training, shown in Figure 5. We
generate this figure using the attention value of the edges trained from the
Nuclear Receptor dataset. From this graph, the proteins and drugs are able to
form groups based on their similarities in the functions. We focus on
analyzing two main groups.

First, we analyze the large group around proteins hsa:2099 (estrogen receptor
alpha), hsa:2100 (estrogen receptor beta), and hsa:5241 (progesterone
receptor), shown in the purple circle in the figure. The functions of these
proteins are very similar. Therefore they are shown as connected and interact
with many drugs. The drugs to the right of these proteins consist of a few
progesterone receptor agonists (D00182, D00951, D01294, D00066, D00950,
D00954, D01217, D02367). For some of the drugs, they only act as progesterone
receptors, such as D00951 and D01217, and therefore have a large weight
connecting to the progesterone receptor. On the other hand, drugs like
Progesterone (D00066) also activate the estrogen pathway and therefore are
connected to the estrogen receptor pathway. Because of there similarity, these
drugs also give attention to other drugs with a similar function, lowering the
individual attention weight in the cluster. On the smaller cluster on the left
of the estrogen receptors, we can also see a similar cluster of estrogen
receptor agonists (D00554, D00105, D00067, D00312, D00577, D00898) and
estrogen receptor antagonists (D01161). These proteins do not activate
progesterone receptors, with a few of them only activate the alpha receptor
(D00962, D02217).

In addition, we analyze the smaller cluster on the top right of the graph,
shown in the green box. This cluster is disjoint from the rest of the graph,
centered around Vitamin D Receptor (hsa:7421). There are 5 drugs around this
receptor, which are the following:

  * •

Vitamin D2 (Ergocalciferol, D00187)

  * •

activated Vitamin D2 (Paricalcitol, D00930)

  * •

Vitamin D3 (Cholecalciferol, D00188)

  * •

activated Vitamin D3 (Calcitriol, D00129)

  * •

a synthetic vitamin D analog (Dihydrotachysterol, D00299)

For these drugs, we can see that the receptor has placed more attention
weights on the activated versions of the vitamin D (D00930 and D00129). And
for these two drug, there are less attention on the receptor. This is because
they pay more attention on itself, lowering their attention weight to the
receptor. However, our graph is not able to predict the connection between the
vitamins and their activated version due to insufficient information.

We can see that DTI-GAT is able to learn the importance of each interaction
from the interaction graph of the dataset. This allows the model to
effectively predict the interaction between the drug and the protein.

##  4 Conclusion

In this paper, we introduce a novel and comprehensive learning framework to
predict the drug-target interactions. Our proposed framework, DTI-GAT,
operates on the graph-structured data with the attention mechanism based on
the deep neural network architecture. We provide an approach to transform the
feature representations of proteins and drugs into a protein-drug interaction
graph. We also emphasize the importance of adding the drug-drug and protein-
protein similarities to the graph. With the attention mechanism, DTI-GAT can
automatically extract the important high-level relationships by assigning
different weights to each edge. Extensive experiments of both cross-validation
and independent tests conducted on the five datasets demonstrate the promising
performance of DTI-GAT. Moreover, we provide case studies to show the
interpretability of DTI-GAT. As a future direction, we plan to explore other
external knowledge including gene ontology and PPI networks, which can be
integrated into DTI-GAT. To further improve the performance, we seek to
incorporate the knowledge graph representation learning framework (Hao et al.,
2019).

## References

  * AY et al. (2007) AY, M. et al. (2007).  Drug–target network.  Nature biotechnology, 25(10), 1119–1127.
  * Ba-Alawi et al. (2016) Ba-Alawi, W. et al. (2016).  Daspfind: new efficient method to predict drug–target interactions.  Journal of cheminformatics, 8(1), 15.
  * Cao et al. (2012) Cao, D.-S. et al. (2012).  Large-scale prediction of drug–target interactions using protein sequences and drug topological structures.  Analytica chimica acta, 752, 1–10.
  * Chen et al. (2019) Chen, M. et al. (2019).  Multifaceted protein–protein interaction prediction based on siamese residual rcnn.  Bioinformatics, 35(14), i305–i314.
  * Chen et al. (2012) Chen, X. et al. (2012).  Drug–target interaction prediction by random walk on the heterogeneous network.  Molecular BioSystems, 8(7), 1970–1978.
  * Consortium (2019) Consortium, U. (2019).  Uniprot: a worldwide hub of protein knowledge.  Nucleic acids research, 47(D1), D506–D515.
  * Dickson and Gagnon (2004) Dickson, M. and Gagnon, J. P. (2004).  Key factors in the rising cost of new drug discovery and development.  Nature reviews Drug discovery, 3(5), 417–429.
  * Ding et al. (2017) Ding, Y. et al. (2017).  Identification of drug-target interactions via multiple information integration.  Information Sciences, 418, 546–560.
  * Fey and Lenssen (2019) Fey, M. and Lenssen, J. E. (2019).  Fast graph representation learning with PyTorch Geometric.  In ICLR Workshop on Representation Learning on Graphs and Manifolds.
  * Günther et al. (2007) Günther, S. et al. (2007).  Supertarget and matador: resources for exploring drug-target relationships.  Nucleic acids research, 36(suppl_1), D919–D922.
  * Hao et al. (2019) Hao, J. et al. (2019).  Universal representation learning of knowledge bases by jointly embedding instances and ontological concepts.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1709–1719.
  * Janga and Tzakos (2009) Janga, S. C. and Tzakos, A. (2009).  Structure and organization of drug-target networks: insights from genomic approaches for drug discovery.  Molecular BioSystems, 5(12), 1536–1548.
  * Jones (1999) Jones, D. T. (1999).  Protein secondary structure prediction based on position-specific scoring matrices.  Journal of molecular biology, 292(2), 195–202.
  * Kanehisa et al. (2006) Kanehisa, M. et al. (2006).  From genomics to chemical genomics: new developments in kegg.  Nucleic acids research, 34(suppl_1), D354–D357.
  * Kapetanovic (2008) Kapetanovic, I. (2008).  Computer-aided drug discovery and development (caddd): in silico-chemico-biological approach.  Chemico-biological interactions, 171(2), 165–176.
  * Keiser et al. (2009) Keiser, M. J. et al. (2009).  Predicting new molecular targets for known drugs.  Nature, 462(7270), 175–181.
  * Kim et al. (2019) Kim, S. et al. (2019).  Pubchem 2019 update: improved access to chemical data.  Nucleic acids research, 47(D1), D1102–D1109.
  * Kingma and Ba (2014) Kingma, D. P. and Ba, J. (2014).  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980.
  * Kola and Landis (2004) Kola, I. and Landis, J. (2004).  Can the pharmaceutical industry reduce attrition rates?  Nature reviews Drug discovery, 3(8), 711–716.
  * Kuhn et al. (2008) Kuhn, M. et al. (2008).  Large-scale prediction of drug–target relationships.  FEBS letters, 582(8), 1283–1290.
  * LeCun et al. (2015) LeCun, Y. et al. (2015).  Deep learning.  nature, 521(7553), 436–444.
  * Lee et al. (2019) Lee, I. et al. (2019).  Deepconv-dti: Prediction of drug-target interactions via deep learning with convolution on protein sequences.  PLoS computational biology, 15(6), e1007129.
  * Li et al. (2017) Li, Z. et al. (2017).  In silico prediction of drug-target interaction networks based on drug chemical structure and protein sequences.  Scientific reports, 7(1), 1–13.
  * Maaten and Hinton (2008) Maaten, L. v. d. and Hinton, G. (2008).  Visualizing data using t-sne.  Journal of machine learning research, 9(Nov), 2579–2605.
  * Öztürk et al. (2018) Öztürk, H. et al. (2018).  Deepdta: deep drug–target binding affinity prediction.  Bioinformatics, 34(17), i821–i829.
  * Paszke et al. (2019) Paszke, A. et al. (2019).  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc.
  * Petta et al. (2016) Petta, I. et al. (2016).  Modulation of protein–protein interactions for the development of novel therapeutics.  Molecular Therapy, 24(4), 707–718.
  * Schomburg et al. (2004) Schomburg, I. et al. (2004).  Brenda, the enzyme database: updates and major new developments.  Nucleic acids research, 32(suppl_1), D431–D433.
  * Senior et al. (2020) Senior, A. W. et al. (2020).  Improved protein structure prediction using potentials from deep learning.  Nature, pages 1–5.
  * Shi et al. (2019) Shi, H. et al. (2019).  Predicting drug-target interactions using lasso with random forest based on evolutionary information and chemical structure.  Genomics, 111(6), 1839–1852.
  * Singh et al. (2016) Singh, P. et al. (2016).  Biological synthesis of nanoparticles from plants and microorganisms.  Trends in biotechnology, 34(7), 588–599.
  * Skrabanek et al. (2008) Skrabanek, L. et al. (2008).  Computational prediction of protein–protein interactions.  Molecular biotechnology, 38(1), 1–17.
  * Srivastava et al. (2014) Srivastava, N. et al. (2014).  Dropout: A simple way to prevent neural networks from overfitting.  Journal of Machine Learning Research, 15(56), 1929–1958.
  * Veličković et al. (2017) Veličković, P. et al. (2017).  Graph attention networks.
  * Wang et al. (2018a) Wang, L. et al. (2018a).  A computational-based method for predicting drug–target interactions by using stacked autoencoder deep neural network.  Journal of Computational Biology, 25(3), 361–373.
  * Wang et al. (2018b) Wang, L. et al. (2018b).  Rfdt: A rotation forest-based predictor for predicting drug-target interactions using drug structure and protein sequence information.  Current Protein and Peptide Science, 19(5), 445–454.
  * Wang et al. (2010) Wang, Y.-C. et al. (2010).  Computationally probing drug-protein interactions via support vector machine.  Letters in Drug Design & Discovery, 7(5), 370–378.
  * Wen et al. (2017) Wen, M. et al. (2017).  Deep-learning-based drug–target interaction prediction.  Journal of proteome research, 16(4), 1401–1409.
  * Wishart et al. (2008) Wishart, D. S. et al. (2008).  Drugbank: a knowledgebase for drugs, drug actions and drug targets.  Nucleic acids research, 36(suppl_1), D901–D906.
  * Xia et al. (2010) Xia, Z. et al. (2010).  Semi-supervised drug-protein interaction prediction from heterogeneous biological spaces.  In BMC systems biology, volume 4, page S6. BioMed Central.
  * Yamanishi et al. (2008) Yamanishi, Y. et al. (2008).  Prediction of drug–target interaction networks from the integration of chemical and genomic spaces.  Bioinformatics, 24(13), i232–i240.
  * Yamanishi et al. (2010) Yamanishi, Y. et al. (2010).  Drug-target interaction prediction from chemical, genomic and pharmacological data in an integrated framework.  Bioinformatics, 26(12), i246–i254.
  * Yamanishi et al. (2011) Yamanishi, Y. et al. (2011).  Extracting sets of chemical substructures and protein domains governing drug-target interactions.  Journal of chemical information and modeling, 51(5), 1183–1194.
  * Zhou et al. (2020) Zhou, G. et al. (2020).  Mutation effect estimation on protein–protein interactions using deep contextualized representation learning.  NAR Genomics and Bioinformatics, 2(2), lqaa015.