# Molecular geometric deep learning

Cong Shen  College of Computer Science and Electronic Engineering, Hunan
University, Changsha 410000, China  School of Physical and Mathematical
Sciences, Nanyang Technological University, 637371, Singapore  Jiawei Luo
Kelin Xia

###### Abstract

Geometric deep learning (GDL) has demonstrated huge power and enormous
potential in molecular data analysis. However, a great challenge still remains
for highly efficient molecular representations. Currently, covalent-bond-based
molecular graphs are the de facto standard for representing molecular topology
at the atomic level. Here we demonstrate, for the first time, that molecular
graphs constructed only from non-covalent bonds can achieve similar or even
better results than covalent-bond-based models in molecular property
prediction. This demonstrates the great potential of novel molecular
representations beyond the de facto standard of covalent-bond-based molecular
graphs. Based on the finding, we propose molecular geometric deep learning
(Mol-GDL). The essential idea is to incorporate a more general molecular
representation into GDL models. In our Mol-GDL, molecular topology is modeled
as a series of molecular graphs, each focusing on a different scale of atomic
interactions. In this way, both covalent interactions and non-covalent
interactions are incorporated into the molecular representation on an equal
footing. We systematically test Mol-GDL on fourteen commonly-used benchmark
datasets. The results show that our Mol-GDL can achieve a better performance
than state-of-the-art (SOTA) methods. Source code and data are available at
https://github.com/CS-BIO/Mol-GDL.

##  1 Introduction

Artificial intelligence (AI) based models have demonstrated huge power and
enormous potential for the prediction of various molecular properties. In
particular, AI-based drug design has achieved great success in various steps
in virtual screening and has the potential to revolutionize the drug industry
[1, 2, 3, 4]. However, even with the great progress, to design efficient
molecular representations and featurization remains a great challenge. In
general, all AI-based molecular models can be classified into two types, i.e.,
molecular descriptor-based machine learning models and end-to-end deep
learning models.

The first type is to use handcrafted molecular descriptors or fingerprints as
input features for machine learning models. The generation of the handcrafted
descriptors is known as featurization or feature engineering. Other than
physical, chemical, and biological properties, such as atomic partial charge,
hydrophobicity, electronic properties, steric properties, etc, the majority of
the molecular features are obtained from molecular structural properties. In
fact, more than 5000 molecular structural descriptors have been developed, and
can be generalized as one-dimensional (1D), two-dimensional (2D), three-
dimensional (3D), and four-dimensional (4D) features [5, 6]. The 1D molecular
descriptors include atom counts, bond counts, molecular weight, fragment
counts, functional group counts, and other summarized general properties. The
2D molecular descriptors include topological indices, graph properties,
combinatorial properties, molecular profiles, autocorrelation coefficients,
etc. The 3D molecular descriptors include molecular surface properties, volume
properties, autocorrelation descriptors, substituent constants, quantum
chemical descriptors, etc. A related higher computational cost is usually
required for the generation of 3D molecular descriptors. The 4D chemical
descriptors characterize configuration changes in a dynamic process. These
structural descriptors are widely used in quantitative structure-activity
relationships (QSAR), quantitative structure-property relationships (QSPR),
and machine learning models. Recently, deep learning models, including
autoencoder, convolutional neural network (CNN), and graph neural network
(GNN), have also been used in molecular fingerprint generation [7, 8, 9, 10,
11]. For all machine learning models, their performance is directly related to
the efficiency of these molecular descriptors.

The second type is end-to-end geometric deep learning (GDL) models. In these
GDLs, molecules are represented as molecular graphs, density functions, or
molecular surfaces, and various deep learning models, such as (3D)
convolutional neural networks (CNNs), graph neural networks (GNNs), recurrent
neural networks (RNNs), etc, can be used to automatically learn the molecular
properties [12, 13, 14, 15, 16]. Among these different molecular
representations, molecular graphs are the most popular one. In particular,
covalent-bond-based molecular graphs are the de facto standard for
representing molecular topology at atomic level. Based on them, various GDL
models have been proposed, including graph recurrent neural networks
(GraphRNN) [17], graph convolutional networks (GCN) [18], graph autoencoders
[19], graph transformers [20], and others. These GDLs have been widely used in
molecular data analysis, in particular drug design [21, 22, 23]. Recently,
non-covalent-interaction-based molecular descriptors have achieved great
performance in the prediction of binding affinities of protein-ligand and
protein-protein [24, 25], demonstrating potential new molecular graph
representations beyond the de facto standard of covalent-bond-based graphs. In
fact, the incorporation of geometric information, such as bond angles,
periodicity, symmetry, rotation-translation invariance, equivalence, etc, into
GDL models, can help to significantly improve the learning performance [26,
27, 28, 29], especially for the machine learned based force field models.
Efficient representations beyond covalent-bond-based molecular graphs provide
a great promise for a better representation of molecular geometric
information.

Here we show that molecular representations with only non-covalent
interactions can achieve similar or even better results than the de facto
standard of covalent-bond-based models in molecular property prediction, for
the first time. More specifically, we systematically compare the performance
of GDL models using two types of molecular representations, i.e., covalent
interaction graphs and non-covalent interaction graphs, in several most-
commonly-used benchmark datasets, including BACE, ClinTox, SIDER, Tox21, HIV,
ESOL, etc. It has been found that GDL models using only non-covalent
interactions have comparable or even superior performance than the de facto
standard models. Further, we propose molecular geometric deep learning (Mol-
GDL) to incorporate a more general molecular representation into GDLs. In our
Mol-GDL, molecular topology is modeled as a series of molecular graphs, each
focusing on a different scale of atomic interactions. In this way, both
covalent interactions and non-covalent interactions are incorporated into the
molecular representation on an equal footing. We systematically test Mol-GDL
on fourteen commonly-used benchmark datasets. The results show that our Mol-
GDL can achieve a better performance than state-of-the-art (SOTA) methods.

##  2 Results

Figure 1:  Illustration of different molecular graph representations and the
performance of their GDLs in six commonly-used datasets. The de facto standard
of covalent-bond based molecular graph representation has clear limitations
and is inferior to models with only non-covalent interactions. a Molecular
graph representations for Monobenzone molecule. The de facto standard of
covalent-bond model is represented in orange color and the other four non-
covalent-interaction-based graphs are in blue colors. b The performance of
GDLs with five different molecular representations on six most commonly-used
datasets. The color of the bar (for each model) is the same as that of the
corresponding molecular graph. For instance, the orange bars are for GDLs with
covalent-bond-based molecular graph.

###  2.1 Is covalent-bond based molecular graph a de facto standard for GDL?

Currently, the de facto standard for molecular representation at the atomic
level is the covalent-bond-based molecular graphs. Here we show that, in
molecular property prediction, GDLs with molecular graphs constructed only
from non-covalent interactions can achieve similar or even better results. We
consider five different molecular graph representations and their GDL
performance on the six most-commonly used datasets, including BACE, ClinTox,
SIDER, Tox21, HIV, and ESOL, as illustrated in Figure 1. Of the five different
molecular graphs, one is the de facto standard of the covalent-bond-based
model and the other four are all constructed using only non-covalent
interactions. Stated differently, all the edges in these four molecular graphs
represent only non-covalent information and none of them are generated from
the covalent bonds. Mathematically, these four non-covalent molecular graphs
are constructed by defining the edges only between atoms within a certain pre-
defined Euclidean distance (larger than the covalent bond distances). We
specify a certain domain $I$ for each graph in a way that the edge in this
graph exists if and only if the distance between the corresponding two atoms
is in the domain $I$. For instance, a graph $G(I)$ with $I=[4\AA,6\AA)$ means
all the edges in graph $G(I)$ have lengths between 4Å and 6Å. Details for the
molecular graph representation can be found in Section 2.2.

It can be seen from Figure 1 that GDL with the de facto standard of covalent-
bond-based molecular graph does not have the best performance. In contrast,
GDLs with non-covalent molecular graphs can not only have comparable results
but even outperform the covalent-bond-based model. In fact, for all six
datasets, the non-covalent model with $I=[4\AA,6\AA)$ has superior performance
than the de facto standard model. More interestingly, even for the highly non-
traditional molecular graph representation with $I=[8\AA,\infty)$, in which
edges only form between atoms that have Euclidean distances larger than 8 Å,
the corresponding GDL model can still have a comparably good performance and
even outperform the de facto standard model in the four test datasets,
including BACE, ClinTox, SIDER, and HIV. This demonstrates the great potential
of novel molecular representations beyond the de facto standard of covalent-
bond-based molecular graphs.

###  2.2 Molecular geometric deep learning (Mol-GDL)

#### Molecular graph representation for Mol-GDL

For a molecule with $N$ atoms and atom coordinates denoted as ${\bf
r}_{1},{\bf r}_{2},...,{\bf r}_{N}$, its molecular graph representation can be
expressed as $G(I)=(V,E(I))$. Here $I$ is a certain interaction region we used
to define our graph representation. For simplicity, we only consider region in
terms of $I=[x_{\rm min},x_{\rm max})$ with $x_{\rm min}$ and $x_{\rm max}$
real values (that satisfy $0\leq x_{\rm min}<x_{\rm max}$). The set of nodes
(or atoms) is denoted as $V$ and the set of edges is denoted $E(I)$. Note that
each atom is modeled as a single node (or vertex) in our molecular graph
representation. The adjacent matrix $A(I)=\left(a(I)_{ij}\right)_{1\leq i\leq
N;1\leq j\leq N}$ and its element $a(I)_{ij}$ is represented as,

$a(I)_{ij}=\begin{cases}1,&x_{\rm min}\leq\|{\bf r}_{i}-{\bf r}_{j}\|<x_{\rm
max}~{}{\rm and}~{}i\neq j\\\ 0,&\rm{others}.\end{cases}$ (1)

Geometrically, it means that an edge is formed between the $i$-th and $j$-th
atom if their Euclidean distance $\|{\bf r}_{i}-{\bf r}_{j}\|$ is within the
domain of $I$, in molecular graph $G(I)$. For instance, we can set
$I_{0}=[0\AA,2\AA)$, the corresponding molecular graph $G(I_{0})$ is exactly
the de facto standard molecular graph, which is constructed with only covalent
bonds, as their bond lengths are all within the region of $I_{0}=[0\AA,2\AA)$.
Mathematically, by the variation of the domain $I$, graph representations with
dramatically different topologies can be generated. For instance, if we let
$x_{\rm min}>2\AA$ (thus $x_{\rm max}>2\AA$), the corresponding molecular
graph will contain only non-covalent interactions.

In our Mol-GDL, instead of using only one molecular graph, a series of graphs
$G(I_{k})$ are systematically generated by selecting different regions $I_{k}$
$(k>0)$ (See Method for details). Geometrically, for a certain graph
$G(I_{k})$, an edge is formed between two atoms only when their Euclidian
distance is within the region of $I_{k}$. In this way, we have great
flexibility to construct molecular graphs.

#### Geometric node features for Mol-GDL

Figure 2:  Illustration of geometric node features for Mol-GDL. Different from
all previous models, our geometric node features are solely determined by atom
types and Euclidean distances between atoms. a The illustration of a carbon
atom (in pink) and its neighboring Carbon atoms (in dark black) and Hydrogen
atoms (in light grey) from a molecular graph for Monobenzone. b In our
geometric node features, the neighboring atoms are grouped based on their atom
types. Here the two Carbon atoms are classified into one group and the four
Hydrogen atoms are classified into the other. For each group, the Euclidian
distances between all the neighboring atoms to the Carbon atoms are classified
into several intervals. For each interval, we count the total number (or
frequency) of distances within it. Here three equal-sized intervals are
considered. For Carbon atoms, their frequencies in these intervals are (2, 1,
0), and for hydrogen atoms, their number is (2, 0, 2). These frequency numbers
are then concatenated (in a predefined order according to atom types) into a
fixed-length long vector, i.e., our geometric node feature.

The other important setting for our Mol-GDL is the distance-related node
features. In contrast with traditional node features, our node features
contain only atom types and distance information.

Mathematically, the node feature for the $i$-th vertex of molecular graph
$G(I)$ is denoted as
$f_{i}(I)=[f_{i}(I,\alpha_{1}),f_{i}(I,\alpha_{2}),...,f_{i}(I,\alpha_{m})]$.
Note that here $\alpha_{j}$ (with $1\leq j\leq m$) means the type of atoms,
such as carbon (C), nitrogen (N), oxygen (O), hydrogen (H), sulfur (S), etc.
The element $f_{i}(I,\alpha_{j})$ is defined as the number (or frequency) of
edges in $G(I)$ that are formed between the $i$-th node and any other nodes
that are of atom type $\alpha_{j}$. Mathematically, it is defined as follows,

$f_{i}(I,\alpha_{j})=\sum_{T_{j}=\alpha_{j}}\chi(x_{\rm min}\leq||{\bf
r}_{i}-{\bf r}_{j}||<x_{\rm max}),$ (2)

here $T_{j}$ is the atom type of $j$-th atom and the value of the indicator
function $\chi$ is 1 if the following condition is satisfied and 0 otherwise.
Geometrically, the node feature contains $m$ descriptors and each descriptor
represents the total number of edges connecting to atoms of a specific atom
type. Figure 2 illustrates the node features using an carbon atom as an
example.

Further, a refined node feature can be generated through the subdivision of
domain $I$. More specifically, we can divide $I$ into $L$ intervals
$\\{I^{l}=[x_{l-1},x_{l});l=1,2,..,L\\}$ with $x_{0}=x_{\rm min}$ and
$x_{L}=x_{\rm max}$. The node feature element $f_{i}(I,\alpha_{j})$ will be
extended into a vector
$[f_{i}(I^{1},\alpha_{j}),f_{i}(I^{2},\alpha_{j}),...,f_{i}(I^{L},\alpha_{j})]$.
Similar, the scale element $f_{i}(I^{1},\alpha_{j})$ is defined as

$f_{i}(I^{l},\alpha_{j})=\sum_{T_{j}=\alpha_{j}}\chi(x_{l-1}\leq||{\bf
r}_{i}-{\bf r}_{j}||<x_{l}).$ (3)

Note that our node features are solely determined by atom types and Euclidean
distances between atoms.

Figure 3:  The flowchart of our Mol-GDL model. A set of molecular graphs are
systematically constructed for each molecule. The geometric node features goes
through the same message passing module. The two pooling operations, one is
done within graphs and the other between groups, are used to aggregate
individual node features into a single molecular feature vector. A multi-layer
perceptron (MLP) is employed on the molecular feature vector to generate the
final prediction.

#### The general framework for Mol-GDL

Our Mol-GDL provides a geometric deep learning architecture that can
comprehensively learn the multiscale information within a molecule. Figure 3
illustrates the flowchart of our Mol-GDL. Different from all previous GDLs, a
series of molecular graphs focusing on different scales of interactions are
systematically constructed. A common message passing (MP) module is employed
on each individual molecular graph. After that, two pooling operations are
used. The first pooling is done at atomic level. For each molecular graph,
node features are aggregated together into a single molecular feature vector.
Second pooling is done at molecular graph level. Each molecular feature vector
from the first pooling goes through a single-layer perceptron and then they
are concatenated into a single feature vector. Finally, the single feature
vector, which contains information from all the molecular graphs, goes through
multi-layer perceptron (MLP) to generate the final prediction.

###  2.3 Performance of Mol-GDL for molecular data analysis

In this section, we present the comparison results between our Mol-GDLs and
SOTAs on different types of molecular data analysis, including classification
tasks on the molecular property, regression tasks on the molecular property,
and tasks on molecular interaction prediction. The detailed information of
datasets, SOTA models, and evaluation metrics can be found in Supplementary
Note 1 and Table S1.

#### Mol-GDL for classification tasks on molecular property

In the classification task, seven commonly used datasets[30] are considered.
These datasets can be roughly classified into two categories, one for
biophysical properties (BACE, HIV, and MUV) and the other for physiological
properties (BBBP, Tox21, SIDER, and ClinTox). More specifically, the BACE
dataset provides quantitative binding results for inhibitors of human
$\beta$-secretase 1(BACE-1)[31]. Its average number of atoms per molecule is
around 65, which is the largest of the seven data sets. The HIV data set is
for the study of the molecule’s ability to inhibit HIV replication. The MUV
data set, which is screened from PubChem BioAssay by applying a nearest
neighbor analysis[32], is devoted to the validation of virtual screening
techniques. The BBBP dataset is for barrier permeability[33], while SIDER is a
database of marketed drugs and adverse drug reactions[34]. Both Tox21 and
ClinTox datasets[35] are related to the toxicity of compounds and they all
contain multiply classification tasks. The measurement of area under curve
(AUC) is used for the evaluation of the results of the models. The detailed
setting of our Mol-GDL parameters can be found in Supplementary Note 2.

The overall performance of Mol-GDL on classification benchmarks along with
SOTAs is shown in Table 1. It can be seen that our Mol-GDL has achieved the
best results, and consistently outperform the SOTAs in all the tasks except
only one from MUV. Our Mol-GDL on BACE is 0.863, which is better than the
previous best result (AUC=0.856). The AUC results of the Mol-GDL model on the
two datasets of HIV and MUV are 0.808 and 0.675, respectively. The results of
Mol-GDL on BBBP and SIDER are significantly better than SOTAs. Among them, the
performance on the SIDER dataset is the most prominent, and the AUC value of
the Mol-GDL model is 20.68% higher than the previous SOTA results. The
performance of the Mol-GDL model on these two datasets of Tox21 and ClinTox is
also much better than the comparison methods. It is worth noting that our
model not only has the largest AUC value on the ClinTox dataset (AUC=0.966)
but also has a much lower standard deviation (std=0.002) than all other
existing methods.

Table 1: The comparison of Mol-GDL with SOTAs on seven commonly-used datasets,
which contain only classification tasks on molecular properties.  Note that
the subindex indicates standard deviation values. For instance, the element
$65_{(19)}$ means the number of average atoms in BACE is 65 with 19 as its
standard deviation.

Dataset | BACE | BBBP | ClinTox | SIDER | tox21 | HIV | MUV
---|---|---|---|---|---|---|---
No.molecule | 1513 | 2039 | 1478 | 1427 | 7831 | 41127 | 93087
No.average atoms | 65(19) | 46(21) | 50.58(31) | 65(93) | 36(23) | 46(24) | 43(10)
No.tasks | 1 | 1 | 2 | 27 | 12 | 1 | 17
D-MPNN[36] | 0.809(0.006) | 0.710(0.003) | 0.906(0.006) | 0.570(0.007) | 0.759(0.007) | 0.771(0.005) | 0.786(0.014)
AttentiveFP[37] | 0.784(0.022) | 0.643(0.018) | 0.847(0.003) | 0.606(0.032) | 0.761(0.005) | 0.757(0.014) | 0.766(0.015)
N-GramRF[38] | 0.779(0.015) | 0.697(0.006) | 0.775(0.040) | 0.668(0.007) | 0.743(0.004) | 0.772(0.001) | 0.769(0.007)
N-GramXGB[38] | 0.791(0.013) | 0.691(0.008) | 0.875(0.027) | 0.655(0.007) | 0.758(0.009) | 0.787(0.004) | 0.748(0.002)
PretrainGNN[39] | 0.845(0.007) | 0.687(0.013) | 0.726(0.015) | 0.627(0.008) | 0.781(0.006) | 0.799(0.007) | 0.813(0.021)
GROVEbase[40] | 0.826(0.007) | 0.700(0.001) | 0.812(0.030) | 0.648(0.006) | 0.743(0.001) | 0.625(0.009) | 0.673(0.018)
GROVElarge[40] | 0.810(0.014) | 0.695(0.001) | 0.762(0.037) | 0.654(0.001) | 0.735(0.001) | 0.682(0.011) | 0.673(0.018)
GEM[41] | 0.856(0.011) | 0.724(0.004) | 0.901(0.013) | 0.672(0.004) | 0.781(0.001) | 0.806(0.009) | 0.817(0.005)
Mol-GDL | 0.863(0.019) | 0.728(0.019) | 0.966(0.002) | 0.831(0.002) | 0.794(0.005) | 0.808(0.007) | 0.675(0.014)

#### Mol-GDL for regression tasks on the molecular property

The commonly used datasets for the regression task on the molecular property
are mainly divided into two categories [30]. One is for predicting the
physical and chemical properties of molecules, including ESOL, FreeSolv, and
Lipo. The other is in the field of quantum chemistry, including QM7, QM8, and
QM9. The detailed setting of our Mol-GDL parameters can be found in
Supplementary Note 2.

For a better comparison with SOTAs [30, 41], we consider two metrics, root
mean squre error (RMSE) and mean absolute error (MAE) respectively for the two
types of regression tasks. Table 2 presents the results of the Mol-GDL model
and SOTAs. Generally speaking, our Mol-GDL can achieve similar or even better
results than SOTA. In particular, our Mol-GDL has the smallest RMSE values on
the ESOL and FreeSolv tasks. On the QM7 dataset, the MAE value of the Mol-GDL
model is 62.2, second only to GEM[41]. Although the performance of Mol-GDL on
the three datasets (Lipo, QM8, and QM9) did not rank among the top, it still
beats several classical prediction methods of molecular properties, such as
$N$-GramRF[38], $N$-GramXGB[38], GROVEbase[40] and GROVElarge[40]. The reason
why Mol-GDL performs poorly on some regression tasks may be that the graph
constructed by 3D coordinates is incomplete. As described in the previous
literature[49, 50], there are some structures in the 3D atomic cloud that
cannot be distinguished by the distance-based GNN model, which leads to the
suboptimal performance of Mol-GDL on these tasks.

Table 2: The comparison of Mol-GDL with SOTAs on six commonly-used datasets,
which contain only region tasks on molecular properties.Note that the subindex
indicates standard deviation values.

| RMSE | MAE
---|---|---
Dataset | ESOL | FreeSolv | Lipo | QM7 | QM8 | QM9
No.molecules | 1128 | 642 | 4200 | 6830 | 21786 | 133885
No.average atoms | 26(13) | 18(7) | 49(15) | 16(3) | 16(3) | 18(3)
No.tasks | 1 | 1 | 1 | 1 | 12 | 12
D-MPNN[36] | 1.050(0.008) | 2.082(0.082) | 0.683(0.016) | 103.5(8.6) | 0.0190(0.0001) | 0.00814(0.00001)
AttentiveFP[37] | 0.877(0.029) | 2.073(0.183) | 0.721(0.001) | 72.0(2.7) | 0.0179(0.0001) | 0.00812(0.00001)
N-GramRF[38] | 1.074(0.107) | 2.688(0.085) | 0.812(0.028) | 92.8(4.0) | 0.0236(0.0006) | 0.01037(0.00016)
N-GramXGB[38] | 1.083(0.082) | 5.061(0.744) | 2.072(0.030) | 81.9(1.9) | 0.0215(0.0005) | 0.00964(0.00031)
PretrainGNN[39] | 1.100(0.006) | 2.764(0.002) | 0.739(0.003) | 113.2(0.6) | 0.0200(0.0001) | 0.00922(0.00004)
GROVEbase[40] | 0.983(0.090) | 2.176(0.052) | 0.817(0.008) | 94.5(3.8) | 0.0218(0.0004) | 0.00984(0.00055)
GROVElarge[40] | 0.895(0.017) | 2.272(0.051) | 0.823(0.010) | 92.0(0.9) | 0.0224(0.0003) | 0.00986(0.00025)
GEM[41] | 0.798(0.029) | 1.877(0.094) | 0.660(0.008) | 58.9(0.8) | 0.0171(0.0001) | 0.00746(0.00001)
Mol-GDL | 0.798(0.024) | 1.809(0.100) | 0.779(0.007) | 62.2(0.4) | 0.0205(0.0001) | 0.00952(0.00013)

#### Mol-GDL for molecular interaction analysis

In this task, we use the dataset from DeepDDS[42], which contains 36
anticancer drugs, 31 human cancer cell lines, and 12,415 unique drug pair-cell
line combinations. Six different validation metrics are used to measure the
performance of models, including AUC, AUPR, Recall, Precision and F1-score.
Note that a slightly different architecture and parameter setting is used and
the details cane be found in Supplementary Note 3 and Table S4.

Five deep learning-based drug synergy prediction methods, including
TranSynergy[43], DeepSynergy[44], Deep Tensor Factorization(DTF)[45],
DeepDDSGAT[42], and DeepDDSGCN[42], and two machine learning methods, i.e.,
XGBoost[46] and Random Forest (RF)[47], are used as baselines. Since the task
in the dataset is drug combination prediction between two drugs, a slightly
different learning architecture is considered in our Mol-GDL. Table 3 shows
the overall performance of the Mol-GDL model in predicting synergistic drug
combinations. It can be seen that our Mol-GDL model outperforms all the other
methods on five validation metrics.

Table 3: The comparison of the performance of Mol-GDL and SOTAs on synergistic drug combination dataset. Note that the subindex indicates standard deviation values. Methods | Metrics |
---|---|---
AUC | AUPR | REC | PREC | F1
XGBoost[46] | 0.92(0.01) | 0.92(0.01) | 0.84(0.01) | 0.84(0.01) | 0.84(0.01)
RF[47] | 0.86(0.01) | 0.85(0.02) | 0.74(0.01) | 0.78(0.02) | 0.76(0.01)
TranSynergy[43] | 0.90(0.01) | 0.89(0.01) | 0.80(0.01) | 0.84(0.01) | 0.82(0.01)
DTF[45] | 0.89(0.01) | 0.88(0.01) | 0.77(0.03) | 0.82(0.01) | 0.80(0.02)
DeepSynergy[44] | 0.88(0.01) | 0.87(0.01) | 0.75(0.01) | 0.81(0.01) | 0.78(0.01)
DeepDDSGAT[42] | 0.93(0.01) | 0.93(0.01) | 0.84(0.07) | 0.85(0.07) | 0.85(0.07)
DeepDDSGCN[42] | 0.93(0.01) | 0.92(0.01) | 0.84(0.01) | 0.85(0.01) | 0.84(0.01)
Mol-GDL | 0.94(0.01) | 0.94(0.01) | 0.86(0.01) | 0.86(0.01) | 0.86(0.01)

##  3 Discussion

###  3.1 Incorporation of non-covalent-bond information enhances performance

The role of non-covalent bonds in predicting molecular properties has long
been underappreciated. In this work, we systematically compare the performance
of GDLs from non-covalent-bond-based molecular graphs and covalent-bond-based
molecular graphs in 13 datasets. The detailed comparison setting is discussed
in Supplementary Note 4 and the results are listed in Table S5. It can be seen
that the performance of our Mol-GDLs is better than that of the models that
consider only covalent-bond information on most datasets, indicating that non-
covalent bonds provide a positive impact on improving the prediction
performance of the model. Therefore, these results show the importance of non-
covalent interactions in molecular property prediction.

Further, we consider some extreme cases to explore the effects of covalent and
non-covalent bonds in molecular representations for GDLs. Three types of
neighboring relations are used for the construction of molecular graphs, i.e.,
$n$-nearest neighbor, $n$-farthest neighbor, and $n$-random neighbor. Here $n$
is an integer number and can be taken from 1 to 7. The results are shown in
Tables S6 and S7, and we have the following observations. First, as the number
of neighbor nodes increases, the performance will get better and then tend to
be stable. Second, with only a few farthest neighbors or random neighbors,
i.e., $n$=1 or 2, the model can still have comparably good prediction, in
particular the classification tasks. Third, on some datasets, the prediction
performance of molecular graphs constructed by the $n$ farthest neighbors is
better than that of the $n$ nearest neighbors, in particular for BBBP,
ClinTox, and QM7 datasets. These observations again demonstrate the importance
of non-covalent interactions and the potential novel molecular representations
beyond the de facto standard of covalent-bond-based molecular graphs.

###  3.2 Geometric node features enhance GDL performance

Note features are of key importance for GDLs. Here we propose the geometric
node features that are related only to atomic types and Euclidean distances.
To verify the efficiency of our geometric node feature approach, we compare it
with three different feature engineering methods from the learning models of
AttentiveFP[37], D-MPNN[36, 48] and DeepDDS[42]. Note that the features used
in these three models are derived from the structural, physical, chemical, and
biological properties, such as the number of atoms, normal charge, chirality,
hybridization, aromaticity, etc. For a fair comparison, the same graph neural
network architecture from our Mol-GDL is used. That is the same Mol-GDL
architecture equipped with four types of different initial node features on 11
datasets (MUV and QM9 datasets are omitted due to memory issues). The detailed
setting can be found in Supplementary Note 5 and the results are listed in
Figure 4 and Table S8. It can be seen that our geometric node features are
superior to the three feature engineering approaches in almost all the
datasets. In particular, our geometric node feature approach has obvious
advantages in BACE, BBBP, freesolv, and QM7 datasets.

Figure 4:  Comparison of different methods of calculating node feature.

It should be noted that our geometric node feature approach is developed based
on strong relations between molecular structures and their functions. Even
though our approach contains only the atomic types and Euclidean distances,
various physical, chemical, and biological information has been implicitly
incorporated into it. In fact, the positions of all the atoms within a
molecule are the direct reflection of the overall interactions within the
molecule. Stated differently, two atoms are close to each other only when the
overall interactions (from all the physical, chemical, and biological effects)
between them are relatively strong. Since this interaction information is
implicitly embodied into its 3D structure through the structure-function
relationships, an efficient molecular representation can be obtained as long
as we fully explore the structural information of the molecule. From another
point of view, the length of chemical bonds is used as the node feature in our
Mol-GDL model. This is essentially the process of storing the information of
non-covalent bonds in the initial representation. The great performance of our
approach again indicates the importance of non-covalent interactions in
molecular representation.

###  3.3 General properties of Mol-GDL

With the highly efficient molecular representation, our Mol-GDL model achieves
a better performance than SOTAs on most datasets. The performance of our Mol-
GDL model is highly consistent and does not depend on the size of the
datasets. In Tables 1 and 2, our Mol-GDL achieves the best performance in 8 of
the 13 datasets. Especially on the classification task, the average
performance is improved by about 3% compared to the best model GEM in the SOTA
methods, and 6 of the 7 classification datasets are optimal. Note that the
number of samples varies widely in datasets for these classification tasks.
There are only about 1,400 molecules in ClinTox and SIDER datasets, while more
than 40,000 molecules are included in the HIV dataset. Even with the huge size
difference in the datasets, the performance of our Mol-GDL model is
consistently better than SOTAs.

Further, the performance of our Mol-GDL model can be related to the size of
the molecules. In particular, Mol-GDL may have slightly inferior results than
SOTAs for small-sized molecules. In the regression task, although Mol-GDL
model has better results than most of the models, such as N-GramRF, N-GramXGB,
GROVEbase and GROVElarge, it has slightly inferior results than GEM on QM7,
QM8, and QM9. The average number of atoms per molecule for these three
datasets is about 16, 16, and 18, respectively. However, it should be noticed
that GEM is a pre-trained self-supervised model and it uses a much larger
dataset for the pre-training process. In contrast, our Mol-GDL uses only the
regular training set, which is significantly smaller in size than the dataset
for pre-training. It is worth mentioning that the other three models, i.e.,
PretainGNN, GROVERbase, and GROVERlarge, also use the pre-training process,
but their performance is still inferior to our Mol-GDL for all the tasks.

Moreover, other than molecular property analysis, our Mol-GDL model can also
achieve great performance in molecular interaction prediction. As shown in
Table 3, our Mol-GDL outperforms SOTA methods for predicting synergistic drug
combinations on all validation metrics. This indicates that efficient
molecular graph representations are important to the analysis of not only
properties at the single molecular level but also interactions between two or
more molecules.

Finally, our Mol-GDL models can be further generalized or extended from
several different aspects. First, we can use our Mol-GDL framework to generate
a series of molecular fingerprints [7, 8, 9, 10, 11]. More specifically, a
series of molecular descriptors can be obtained from the aggregation of node
features. Second, Mol-GDL based multi-task learning models can be constructed
to further improve the performance of Mol-GDL. Third, distance GNN models have
some intrinsic limitations to distinguish some special molecular structures
[49]. Many-body descriptors alone are also incomplete for molecular structure
representation [50]. The generalization of Mol-GDL into higher-order
topological models, such as simplicial complex, polyhedron complex,
hypergraph, etc, may provide ways for a more efficient and complete molecular
representation.

##  4 Conclusion

Molecular representation learning plays an important role in molecular
property prediction. Existing molecular property prediction work often relies
on the de facto standard of covalent-bond-based molecular graphs for
representing molecular topology at the atomic level, and totally ignores the
non-covalent interactions within the molecule. In this study, we propose a
molecular geometric deep learning(Mol-GDL) model to predict the properties of
molecules, which aims to comprehensively consider the information of covalent
and non-covalent interactions of molecules. Extensive tests have demonstrated
the effectiveness of the Mol-GDL model, and the important role of non-covalent
interactions in molecular property prediction has also been validated through
multiple tests. In future work, we consider multi-task learning or
incorporating more effective information other than non-covalent bonds to
further improve the performance of Mol-GDL.

##  5 Methods

###  5.1 Molecular graph representation for Mol-GDL

Graph neural networks can encode molecular structural information into a low-
dimensional representation vector from a high-dimensional space. It is a
popular method to regard the atoms in the molecule as nodes and the chemical
bonds as edges in the graph, and then use the method of message passing to
predict the properties of the molecule[48][51]. However, ignoring the
quantitative distances between atoms in traditional graphs may result in the
loss of a great deal of critical physical and chemical information in
molecules. Molecular graphs reconstructed from specific distance intervals can
capture some different physicochemical and biophysical patterns, such as
hydrogen bonding and van der Waals interactions between different
atoms[25][52].

A molecule can be represented as a graph $G(I)=(V,E(I))$, where $E(I)$ is an
edge set and $V$ is a node set. $I$ is a certain interaction region.
Physically, the covalent bonds are usually within the interaction region of
$[0\AA,2\AA]$. Non-covalent interactions are comparably weak and are within a
much larger range. For instance, the distance between donor and acceptor atoms
in hydrogen bonds, is roughly between $[2.0\AA,4.0\AA]$. Van der Waals forces
can cover a much larger range and play an very important role in the range
$[4.0\AA,6.0\AA]$. The electrostatic forces can go further to $[6.0\AA,8\AA]$.
To fully explore the physicochemical and biophysical patterns in non-covalent
interactions, we divide the interval region greater than 2$\AA$ into several
non-overlapping regions. For each interaction region, a corresponding
molecular graph is generated and its associated adjacency matrix is defined as
Eq. (1).

To balance the computational costs and model accuracy, a total of 4 segments
are considered for non-covalent interactions, namely
$I_{1}=[2\AA,4\AA),~{}I_{2}=[4\AA,6\AA),~{}I_{3}=[6\AA,8\AA),~{}I_{4}=[8\AA,\infty)$.
Note that the selection of interaction regions to build molecular graphs is
not unique. For convenience, we let $I_{0}=[0\AA,2\AA)$ denotes the interval
of covalent interactions. In this way, total 5 interaction regions are
considered and a total of 5 molecular graphs, denoted as
$G=\left\\{G_{1},G_{2},G_{3},G_{4},G_{5}\right\\}$, are generated for each
molecular structure, in all of our three types of tasks except those from QM7,
QM8, and QM9. Since the number of atoms in the three datasets QM7, QM8, and
QM9 is relatively small, the number of non-covalent interactions greater than
$6\AA$ is very small. Therefore, we divide the non-covalent interaction
interval for these three datasets into three segments, namely
$I_{1}=[2\AA,4\AA),~{}I_{2}=[4\AA,6\AA),~{}I_{3}=[6\AA,\infty)$. Similarly,
let $I_{0}=[0\AA,2\AA)$ represent the interval of covalent bonds. In this way,
total 4 interaction regions are considered and a total of 4 molecular graphs,
denoted as $G=\left\\{G_{1},G_{2},G_{3},G_{4}\right\\}$, are generated for
each molecular structure from QM7, QM8, and QM9.

###  5.2 Graph neural network architecture in Mol-GDL

For a molecular $G(I_{k})$, denoted as $G_{k}$ for short, we use message
passing to learn the feature representation of each node,

$\displaystyle\bm{h}_{i}^{G_{k},(t)}=\sigma\left({\sum\limits_{j\in\mathcal{N}{(i)}\cup{\\{i\\}}}{\frac{1}{\sqrt{d(i)}\cdot\sqrt{d(j)}}\cdot\left({W_{G_{k}}\cdot\bm{h}_{j}^{G_{k},{({t-1})}}}\right)}}\right).$
(4)

In the $t$-th iteration, the $i$-th node feature vector $h_{i}^{G_{k},(t)}$ of
a graph $G_{k}$ is obtained by gathering node feature vectors of its
neighbors, denoted as $\mathcal{N}{(i)}$, and itself, from $(t-1)$-th
iteration. Here $d(\cdot)$ represents the node degree and $W_{G_{k}}$ is the
weight matrix (to be learned). Computationally, we usually repeat the process
1 to 3 times, and the final node feature is denoted as $\bm{h}_{i}^{G_{k}}$.

After the message passing, we aggregate all node features in a molecular graph
through a pooling process,

$\displaystyle\bm{h}_{G_{k}}=Pooling\left(\bm{h}_{i}^{G_{k}}\middle|i\in[1,2,...,N]\right)$
(5)

where $Pooling(\cdot)$ is a pooling function, such as max pooling, mean
pooling, etc. Note that this pooling process is within the single molecular
graph.

In our Mol-GDL, each molecule will have more than one molecular graph, we
aggregate all the features from its molecular graphs,

$\displaystyle\bm{h}_{G_{k}}^{{}^{\prime}}=\sigma\left(W_{G_{k}}^{{}^{\prime}}\bm{h}_{G_{k}}+b_{G_{k}}^{{}^{\prime}}\right)$
(6)
$\displaystyle\bm{h}_{G}=READOUT\left(\bm{h}_{G_{k}}^{{}^{\prime}}\middle|k\in\left\\{{1,2,...,5}\right\\}\right)$
(7)

where $READOUT(\cdot)$ denotes a pooling function, and we choose concatenation
(or mean) in this study.

Finally, a multiply layer perceptron (MLP) is utilized for the final
prediction,

$\displaystyle\hat{y}=\delta\left(W^{1}\sigma\left({W^{0}\bm{h}_{G}+b^{0}}\right)+b^{1}\right)$
(8)

where $\sigma\left(\cdot\right)$ and $\delta(\cdot)$ are
$Relu\left(\cdot\right)$ and $Sigmoid\left(\cdot\right)$, respectively. For
regression tasks, $\delta(\cdot)$ is a linear activation function. The
$\ell_{1}$ loss and cross-entropy loss are implemented for regression and
classification tasks, respectively.

###  5.3 Geometric node features for Mol-GDL

The geometric node features are of great importance for GDLs. In our Mol-GDL
geometric node features that contain only atomic types and Euclidean distance
information. Computationally, we consider a total of 12 types of atoms,
including $C$, $H$, $O$, $N$, $P$, $Cl$, $F$, $Br$, $S$, $S_{i}$, $I$ and all
the rest atoms as one type. These atoms are chosen due to their high
frequencies in the molecules in our datasets. For the $i$-th atom, atom type
$\alpha_{j}$ will contribute a component $f_{i}(I,\alpha_{j})$ in the
geometric node feature
$f_{i}(I)=[f_{i}(I,\alpha_{1}),f_{i}(I,\alpha_{2}),...,f_{i}(I,\alpha_{1}2)]$.
Here $f_{i}(I,\alpha_{j})$ means the number (or frequency) of all the
neighboring atoms of type $\alpha_{j}$ for the $i$-th atom. Further, different
subdivision of interaction region $I$ is done for different types of atoms.
More specifically, for $C$ (and $H$), its node feature vector components are
of the size 19 after subdivision. That is the component $f_{i}(I,C)$ has been
extended to a vector $[f_{i}(I^{1},C),f_{i}(I^{2},C),...,f_{i}(I^{19},C)]$.
Here $f_{i}(I^{1},C)$ is the number of all the neighboring atoms of type
$\alpha_{j}$ within the interaction subregion $I^{1}$ for the $i$-th atom. For
$O$ and $N$, their node feature components are 4 after subdivision. All the
rest types of atoms have the same node feature components of size 2 after
subdivision. The detailed information for the subdivision of the interaction
region (based on different atom types) is listed in Supplementary Note 6 and
Table S9.

##  6 Funding

This work was supported in part by the Natural Science Foundation of China
(NSFC grant no. 61873089, 62032007), Nanyang Technological University Startup
Grant (grant no. M4081842), Singapore Ministry of Education Academic Research
fund (grant no. Tier 1 RG109/19, MOE-T2EP20120-0013, MOE-T2EP20220-0010) and
China Scholarship Council (CSC grant no. 202006130147).

## References

  * [1] L. Zhang, J. Tan, D. Han, and H. Zhu, “From machine learning to deep learning: progress in machine intelligence for rational drug discovery,” _Drug discovery today_ , vol. 22, no. 11, pp. 1680–1685, 2017.
  * [2] H. Chen, O. Engkvist, Y. Wang, M. Olivecrona, and T. Blaschke, “The rise of deep learning in drug discovery,” _Drug discovery today_ , vol. 23, no. 6, pp. 1241–1250, 2018.
  * [3] K.-K. Mak and M. R. Pichika, “Artificial intelligence in drug development: present status and future prospects,” _Drug discovery today_ , vol. 24, no. 3, pp. 773–780, 2019.
  * [4] H. S. Chan, H. Shan, T. Dahoun, H. Vogel, and S. Yuan, “Advancing drug discovery via artificial intelligence,” _Trends in pharmacological sciences_ , vol. 40, no. 8, pp. 592–604, 2019.
  * [5] T. Puzyn, J. Leszczynski, and M. T. Cronin, _Recent advances in QSAR studies: methods and applications_. Springer Science & Business Media, 2010, vol. 8.
  * [6] Y. C. Lo, S. E. Rensi, W. Torng, and R. B. Altman, “Machine learning in chemoinformatics and drug discovery,” _Drug discovery today_ , vol. 23, no. 8, pp. 1538–1546, 2018.
  * [7] C. Merkwirth and T. Lengauer, “Automatic generation of complementary descriptors with molecular graph networks,” _Journal of chemical information and modeling_ , vol. 45, no. 5, pp. 1159–1168, 2005.
  * [8] D. K. Duvenaud, D. Maclaurin, J. Iparraguirre, R. Bombarell, T. Hirzel, A. Aspuru-Guzik, and R. P. Adams, “Convolutional networks on graphs for learning molecular fingerprints,” in _Advances in neural information processing systems_ , 2015, pp. 2224–2232.
  * [9] C. W. Coley, R. Barzilay, W. H. Green, T. S. Jaakkola, and K. F. Jensen, “Convolutional embedding of attributed molecular graphs for physical property prediction,” _Journal of chemical information and modeling_ , vol. 57, no. 8, pp. 1757–1772, 2017.
  * [10] Y. Xu, J. Pei, and L. Lai, “Deep learning based regression and multiclass models for acute oral toxicity prediction with automatic chemical feature extraction,” _Journal of chemical information and modeling_ , vol. 57, no. 11, pp. 2672–2685, 2017.
  * [11] R. Winter, F. Montanari, F. Noé, and D.-A. Clevert, “Learning continuous and data-driven molecular descriptors by translating equivalent chemical representations,” _Chemical science_ , vol. 10, no. 6, pp. 1692–1701, 2019\.
  * [12] O. Wieder, S. Kohlbacher, M. Kuenemann, A. Garon, P. Ducrot, T. Seidel, and T. Langer, “A compact review of molecular property prediction with graph neural networks,” _Drug Discovery Today: Technologies_ , vol. 37, pp. 1–12, 2020.
  * [13] Z. Yu and H. Gao, “Molecular graph representation learning via heterogeneous motif graph construction,” _arXiv preprint arXiv:2202.00529_ , 2022.
  * [14] K. Atz, F. Grisoni, and G. Schneider, “Geometric deep learning on molecular representations,” _Nature Machine Intelligence_ , vol. 3, no. 12, pp. 1023–1032, 2021.
  * [15] S. Li, J. Zhou, T. Xu, D. Dou, and H. Xiong, “GeomGCL: geometric graph contrastive learning for molecular property prediction,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 4, 2022, pp. 4541–4549.
  * [16] Y. Wang, J. Wang, Z. Cao, and A. Barati Farimani, “Molecular contrastive learning of representations via graph neural networks,” _Nature Machine Intelligence_ , vol. 4, no. 3, pp. 279–287, 2022.
  * [17] J. You, R. Ying, X. Ren, W. Hamilton, and J. Leskovec, “GraphRNN: Generating realistic graphs with deep auto-regressive models,” in _Proceedings of the 35th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. PMLR, 10–15 Jul 2018, pp. 5708–5717. [Online]. Available: https://proceedings.mlr.press/v80/you18a.html
  * [18] M. Welling and T. N. Kipf, “Semi-supervised classification with graph convolutional networks,” in _J. International Conference on Learning Representations (ICLR 2017)_ , 2016.
  * [19] T. N. Kipf and M. Welling, “Variational graph auto-encoders,” _Advances in neural information processing systems_ , 2016.
  * [20] S. Yun, M. Jeong, R. Kim, J. Kang, and H. J. Kim, “Graph transformer networks,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [21] P.-C. Kotsias, J. Arús-Pous, H. Chen, O. Engkvist, C. Tyrchan, and E. J. Bjerrum, “Direct steering of de novo molecular generation with descriptor conditional recurrent neural networks,” _Nature Machine Intelligence_ , vol. 2, no. 5, pp. 254–265, 2020.
  * [22] J. Wang, C.-Y. Hsieh, M. Wang, X. Wang, Z. Wu, D. Jiang, B. Liao, X. Zhang, B. Yang, Q. He _et al._ , “Multi-constraint molecular generation based on conditional transformer, knowledge distillation and reinforcement learning,” _Nature Machine Intelligence_ , vol. 3, no. 10, pp. 914–922, 2021\.
  * [23] J. Vamathevan, D. Clark, P. Czodrowski, I. Dunham, E. Ferran, G. Lee, B. Li, A. Madabhushi, P. Shah, M. Spitzer _et al._ , “Applications of machine learning in drug discovery and development,” _Nature reviews Drug discovery_ , vol. 18, no. 6, pp. 463–477, 2019.
  * [24] M. Wang, Z. Cang, and G.-W. Wei, “A topology-based network tree for the prediction of protein–protein binding affinity changes following mutation,” _Nature Machine Intelligence_ , vol. 2, no. 2, pp. 116–123, 2020.
  * [25] Z. Meng and K. Xia, “Persistent spectral–based machine learning (PerSpect ML) for protein-ligand binding affinity prediction,” _Science Advances_ , vol. 7, no. 19, p. eabc5329, 2021.
  * [26] K. T. Schütt, H. E. Sauceda, P.-J. Kindermans, A. Tkatchenko, and K.-R. Müller, “SchNet–a deep learning architecture for molecules and materials,” _The Journal of Chemical Physics_ , vol. 148, no. 24, p. 241722, 2018.
  * [27] W. Gong and Q. Yan, “Graph-based deep learning frameworks for molecules and solid-state materials,” _Computational Materials Science_ , vol. 195, p. 110332, 2021.
  * [28] K. Kim, S. Kang, J. Yoo, Y. Kwon, Y. Nam, D. Lee, I. Kim, Y.-S. Choi, Y. Jung, S. Kim _et al._ , “Deep-learning-based inverse design model for intelligent discovery of organic molecules,” _npj Computational Materials_ , vol. 4, no. 1, p. 67, 2018.
  * [29] I. Batatia, D. P. Kovacs, G. Simm, C. Ortner, and G. Csányi, “MACE: Higher order equivariant message passing neural networks for fast and accurate force fields,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 11 423–11 436, 2022.
  * [30] Z. Wu, B. Ramsundar, E. N. Feinberg, J. Gomes, C. Geniesse, A. S. Pappu, K. Leswing, and V. Pande, “MoleculeNet: a benchmark for molecular machine learning,” _Chemical science_ , vol. 9, no. 2, pp. 513–530, 2018.
  * [31] G. Subramanian, B. Ramsundar, V. Pande, and R. A. Denny, “Computational modeling of $\beta$-secretase 1 (bace-1) inhibitors using ligand based approaches,” _Journal of chemical information and modeling_ , vol. 56, no. 10, pp. 1936–1949, 2016.
  * [32] S. G. Rohrer and K. Baumann, “Maximum unbiased validation (MUV) data sets for virtual screening based on pubchem bioactivity data,” _Journal of chemical information and modeling_ , vol. 49, no. 2, pp. 169–184, 2009.
  * [33] I. F. Martins, A. L. Teixeira, L. Pinheiro, and A. O. Falcao, “A bayesian approach to in silico blood-brain barrier penetration modeling,” _Journal of chemical information and modeling_ , vol. 52, no. 6, pp. 1686–1697, 2012.
  * [34] M. Kuhn, I. Letunic, L. J. Jensen, and P. Bork, “The sider database of drugs and side effects,” _Nucleic acids research_ , vol. 44, no. D1, pp. D1075–D1079, 2016.
  * [35] K. M. Gayvert, N. S. Madhukar, and O. Elemento, “A data-driven approach to predicting successes and failures of clinical trials,” _Cell chemical biology_ , vol. 23, no. 10, pp. 1294–1301, 2016.
  * [36] K. Yang, K. Swanson, W. Jin, C. Coley, P. Eiden, H. Gao, A. Guzman-Perez, T. Hopper, B. Kelley, M. Mathea _et al._ , “Analyzing learned molecular representations for property prediction,” _Journal of chemical information and modeling_ , vol. 59, no. 8, pp. 3370–3388, 2019.
  * [37] Z. Xiong, D. Wang, X. Liu, F. Zhong, X. Wan, X. Li, Z. Li, X. Luo, K. Chen, H. Jiang _et al._ , “Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism,” _Journal of medicinal chemistry_ , vol. 63, no. 16, pp. 8749–8760, 2019.
  * [38] S. Liu, M. F. Demirel, and Y. Liang, “N-gram graph: Simple unsupervised representation for graphs, with applications to molecules,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [39] W. Hu, B. Liu, J. Gomes, M. Zitnik, P. Liang, V. Pande, and J. Leskovec, “Strategies for pre-training graph neural networks,” _arXiv preprint arXiv:1905.12265_ , 2019.
  * [40] Y. Rong, Y. Bian, T. Xu, W. Xie, Y. Wei, W. Huang, and J. Huang, “Self-supervised graph transformer on large-scale molecular data,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 12 559–12 571, 2020.
  * [41] X. Fang, L. Liu, J. Lei, D. He, S. Zhang, J. Zhou, F. Wang, H. Wu, and H. Wang, “Geometry-enhanced molecular representation learning for property prediction,” _Nature Machine Intelligence_ , vol. 4, no. 2, pp. 127–134, 2022.
  * [42] J. Wang, X. Liu, S. Shen, L. Deng, and H. Liu, “DeepDDS: deep graph neural network with attention mechanism to predict synergistic drug combinations,” _Briefings in Bioinformatics_ , vol. 23, no. 1, p. bbab390, 2022.
  * [43] Q. Liu and L. Xie, “TranSynergy: Mechanism-driven interpretable deep neural network for the synergistic prediction and pathway deconvolution of drug combinations,” _PLoS computational biology_ , vol. 17, no. 2, p. e1008653, 2021.
  * [44] K. Preuer, R. P. Lewis, S. Hochreiter, A. Bender, K. C. Bulusu, and G. Klambauer, “DeepSynergy: predicting anti-cancer drug synergy with deep learning,” _Bioinformatics_ , vol. 34, no. 9, pp. 1538–1546, 2018.
  * [45] Z. Sun, S. Huang, P. Jiang, and P. Hu, “DTF: deep tensor factorization for predicting anticancer drug synergy,” _Bioinformatics_ , vol. 36, no. 16, pp. 4483–4489, 2020.
  * [46] T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” in _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_ , 2016, pp. 785–794.
  * [47] T. K. Ho, “Random decision forests,” in _Proceedings of 3rd international conference on document analysis and recognition_ , vol. 1. IEEE, 1995, pp. 278–282.
  * [48] J. M. Stokes, K. Yang, K. Swanson, W. Jin, A. Cubillos-Ruiz, N. M. Donghia, C. R. MacNair, S. French, L. A. Carfrae, Z. Bloom-Ackermann _et al._ , “A deep learning approach to antibiotic discovery,” _Cell_ , vol. 180, no. 4, pp. 688–702, 2020.
  * [49] S. N. Pozdnyakov and M. Ceriotti, “Incompleteness of graph neural networks for points clouds in three dimensions,” _Machine Learning: Science and Technology_ , vol. 3, no. 4, p. 045020, 2022.
  * [50] S. N. Pozdnyakov, M. J. Willatt, A. P. Bartók, C. Ortner, G. Csányi, and M. Ceriotti, “Incompleteness of atomic structure representations,” _Physical Review Letters_ , vol. 125, no. 16, p. 166001, 2020.
  * [51] S. Li, F. Wan, H. Shu, T. Jiang, D. Zhao, and J. Zeng, “MONN: a multi-objective neural network for predicting compound-protein interactions and affinities,” _Cell Systems_ , vol. 10, no. 4, pp. 308–322, 2020.
  * [52] D. Chen, K. Gao, D. D. Nguyen, X. Chen, Y. Jiang, G.-W. Wei, and F. Pan, “Algebraic graph-assisted bidirectional transformers for molecular property prediction,” _Nature Communications_ , vol. 12, no. 1, pp. 1–9, 2021.