2022

These authors contributed equally to this work.

These authors contributed equally to this work.

These authors contributed equally to this work.

[1]Ferdinand Molnár, [4]Vsevolod Peshkov, [3]Siamac Fazli,

[1]Department of Biology, Nazarbayev University, Nur-Sultan, Kazakhstan
[2]Computer Technologies Laboratory, ITMO University, St. Petersburg, Russia
[3]Department of Computer Science, Nazarbayev University, Nur-Sultan,
Kazakhstan [4]Department of Chemistry, Nazarbayev University, Nur-Sultan,
Kazakhstan

# A biologically-inspired multi-modal evaluation of molecular generative
machine learning

Elizaveta Vinogradova  Abay Artykbayev  Alisher Amanatay  Mukhamejan Karatayev
Maxim Mametkulov  Albina Li  Anuar Suleimenov  Abylay Salimzhanov  Karina Pats
Rustam Zhumagambetov<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>*  *  *  *

###### Abstract

While generative models have recently become ubiquitous in many scientific
areas, less attention has been paid to their evaluation. For molecular
generative models, the state-of-the-art examines their output in isolation or
in relation to its input. However, their biological and functional properties,
such as ligand-target interaction is not being addressed. In this study, a
novel biologically-inspired benchmark for the evaluation of molecular
generative models is proposed. Specifically, three diverse reference datasets
are designed and a set of metrics are introduced which are directly relevant
to the drug discovery process. In particular we propose a recreation metric,
apply drug-target affinity prediction and molecular docking as complementary
techniques for the evaluation of generative outputs.

While all three metrics show consistent results across the tested generative
models, a more detailed comparison of drug-target affinity binding and
molecular docking scores revealed that unimodal predictiors can lead to
erroneous conclusions about target binding on a molecular level and a multi-
modal approach is thus preferrable. The key advantage of this framework is
that it incorporates prior physico-chemical domain knowledge into the
benchmarking process by focusing explicitly on ligand-target interactions and
thus creating a highly efficient tool not only for evaluating molecular
generative outputs in particular, but also for enriching the drug discovery
process in general.

###### keywords:

generative ML, drug discovery, benchmark

##  1 Introduction

Drug discovery and development is a highly cost-intensive and time consuming
process that may take decades and according to some estimates requires up to
$2.6 billion Hughes et al (2011) per marketed drug. One of the main
difficulties of drug discovery is related to the size of the so-called
chemical space, i.e. the space of all theoretically possible molecules
adhering to the physico-chemical rules. Even after limiting this space to all
molecules obeying Lipinski’s “rule of 5” Lipinski et al (2001), its size is
still in the range of 1023-1060 Reymond (2015), which is so vast that an
exhaustive search for new molecules is practically impossible. Thus, a wide
range of generative models for drug discovery have recently been proposed and
attracted the attention of the scientific community Gebauer et al (2019);
Segler et al (2018); Kadurin et al (2016, 2017); Prykhodko et al (2019);
Maziarka et al (2020); Zhumagambetov et al (2021); Skalic et al (2019); Harel
and Radinsky (2018); Olivecrona et al (2017); Blaschke et al (2020); Moret et
al (2020); Dollar et al (2021); Zhumagambetov et al (2020). Such models can
explore and navigate the chemical space efficiently and thereby facilitate the
discovery and generation of new drugs.

To standardize and evaluate the performance of these generative models, a
number of benchmarking metrics and platforms have been proposed Preuer et al
(2018); Brown et al (2019); Polykovskiy et al (2020). For example, Fréchet
ChemNet Distance Preuer et al (2018) is a metric that measures the distance
between two distributions of molecules, generated and real, in terms of their
chemical and biological properties. GuacaMol Brown et al (2019) and Molecular
sets (MOSES) Polykovskiy et al (2020) are two other widely used benchmarking
platforms that are composed of multiple metrics, such as validity, novelty,
uniqueness, Fréchet ChemNet Distance, among others. These metrics assess the
generated molecular output in terms of their validity, diversity, novelty,
uniqueness, required physico-chemical properties, as well as their similarity
to the training set. In other words, their focus is to provide quantitative
and standardized descriptors to characterize the outputs of the generative
methods. While they provide extensive characteristics, they do not evaluate
the molecular output in a biological context. In particular, the ligand-target
interaction process, which is of paramount importance to the drug discovery
process, is currently not given sufficient attention in these benchmarks.

To this end, we propose to extend the current set of benchmark metrics with a
number of tools that also take biological properties into account such as
ligand-target interaction, which is named _affinity binding and recreation
applied heuristic for the analysis of molecules (ABRAHAM)_. The recreation of
originator molecules (ROOM) is such a novel metric that we introduce. It aims
at quantifying the likelihood of a given generative model to recreate known
ligands for a specific protein target. After a set of ligands is assembled for
a specific target, the model trains on a ligand subset and is tasked to
generate an analogous output which is then compared to the true binders and
their overlap is recorded. Next, we propose to incorporate machine learning
(ML)-based drug-target affinity (DTA) binding predictors Chen et al (2018);
Nascimento et al (2016); Öztürk et al (2018); Nguyen et al (2021); Huang et al
(2020); Hu et al (2021); Wang et al (2021) for the evaluation of the
generative models. In addition, we propose to apply molecular docking, which
is the leading methodology of structure-based virtual screening Jastrzębski et
al (2020), to the outputs of the generative models in order to calculate
ligand docking scores. These scores are subsequently utilized as an
alternative and independent verification of the two above mentioned metrics.

To establish our proposed metrics we applied our pipeline to three important
drug targets, namely vitamin D (VDR) and $\gamma$-aminobutyric acid type A
(GABAA) receptors as well as the mammalian target of rapamycin (mTOR).

In short, this work introduces a novel biologically-inspired benchmark for
evaluating _de novo_ molecular generation frameworks. A graphical
representation of the workflow is depicted in Figure 1.

Figure 1: Graphical overview of the workflow for the ABRAHAM metrics.

##  2 Results

###  2.1 ROOM metric results

Table 1 shows the overall results of the proposed ROOM recreation metric for
all three reference datasets and the following seven generative models which
have been used for this study: Conditional Diversity Network (CDN) Harel and
Radinsky (2018); LigDream Skalic et al (2019); Generative Molecular Design in
Low Data Regime (GMDLDR) Moret et al (2020); REINVENT Olivecrona et al (2017);
REINVENT 3.0 Blaschke et al (2020); Transmol Zhumagambetov et al (2021) and
TransVAE Dollar et al (2021). In particular, the average number of recreated
ligands across splits, the total number of recreated ligands (including the
percentage of recreated ligands vs all test set ligands), the total number of
unique generated molecules, the average number of generated molecules across
splits and the ratio of recreated ligands by total number of generated
molecules is shown. Values for best performing models are highlighted. Box-
plots in Figure 2 depict the average number of recreated ligands across
splits.

Table 1: Recreation metrics for protein target datasets

|  average recreated ligands across 10 splits (Mean and std) |  total # unique recreated molecules |  # of valid unique generated molecules |  average generated molecules across 10 splits (Mean and std) |  ratio of recreated ligands to generated molecules ($10^{-6}$)
---|---|---|---|---|---
VDR
CDN |  0.1±0.3 |  1 (0.27%) |  2468 |  257.8±101.9 |  405
GMDLDR |  23.5±3.5 |  100 (27%) |  22458 |  2697.0±164.4 |  4453
LigDream |  0.0±0.0 |  0 |  16717 |  2216.4±136.0 |  0
REINVENT |  0.0±0.0 |  0 |  1176687 |  118943.2±350.8 |  0
REINVENT 3.0 |  13.9±3.2 |  56 (15%) |  6908 |  782.4±27.1 |  8107
Transmol |  9.3±3.9 |  63 (17%) |  205969 |  15131.6±1,458.0 |  306
TransVAE |  0.0±0.0 |  0 |  112512 |  12363.5±389.8 |  0
GABAA
CDN |  0.0±0.0 |  0 |  955 |  110.9±59.9 |  0
GMDLDR |  35.9±6.8 |  120 (47%) |  10872 |  1498.2±108.9 |  11038
LigDream |  0.0±0.0 |  0 |  20512 |  2790.9±135.9 |  0
REINVENT |  0.0±0.0 |  0 |  1192523 |  120735.4±186.3 |  0
REINVENT 3.0 |  29.2±3.2 |  110 (43%) |  5632 |  731.6±20.8 |  19531
Transmol |  24.0±5.2 |  129 (51%) |  168991 |  19245.6±3,779.7 |  767
TransVAE |  0.0±0.0 |  0 |  415591 |  47894.9±379.7 |  0
mTOR
CDN |  35.6±11.3 |  225 (4.86%) |  20413 |  2238.6±220.7 |  11022
GMDLDR |  1.6±1.5 |  13 (0.28%) |  23271 |  2340.8±1,215.7 |  559
LigDream |  0.0±0.0 |  0 |  453602 |  53509.9±517.0 |  0
REINVENT |  0.0±0.0 |  0 |  1188355 |  119769.3±365.5 |  0
REINVENT 3.0 |  95.6±7.7 |  645 (14.0%) |  16947 |  1899.4±8.5 |  38060
Transmol |  20.7±6.2 |  177 (3.82%) |  149068 |  15131.6±1,458.0 |  1187
TransVAE |  0.0±0.0 |  0 |  111855 |  12354.9±151.0 |  0

According to the ROOM metric LigDream, REINVENT and TransVAE failed to
recreate any test set ligands (see Table 1). While CDN showed inadequate
results generating 1 and 0 ligands for VDR and GABAA, respectively, the
performance is drastically improved for mTOR, where 225 ligands are recreated,
which corresponds to a 4.86% recreation rate of all test set ligands (see
Table 1). GMDLDR recreates 100 (27%), 120 (47%) and 13 (0.28%) ligands, for
VDR, GABAA and mTOR, respectively. REINVENT 3.0 performs well for all three
targets. In terms of the ratio of recreated ligands by total number of
generated molecules the model scores highest for each target. Transmol is also
able to recreate ligands for all datasets. In particular, for GABAA the
recreation rate is 51%, which is the highest among all seven generative
models.

Figure 2: The average number of recreated ligands. The box-plots show the
average number of recreated ligands for the 7 generative models across splits.

Figure 3 shows three Venn diagrams for a) VDR, b) GABAA and c) mTOR datasets,
which indicate the intersections of ligand recreation. Only those generative
models are depicted that were able to recreate at least one test-set ligand,
namely CDN, GMDLDR, REINVENT 3.0 and Transmol. While some ligands are jointly
recreated by multiple generative models, others are uniquely recreated. For
example in VDR, 137 ligands are recreated by a single generative model whereas
38 ligands are recreated simultaneously by at least two models. This indicates
that the generative models complement each other for the ligand recreation
process.

Figure 3: Overlap in the ligand recreation metrics. The Venn diagram shows the
relationships among the recreated ligands, in particular the overlap for
recreated ligands by the four best-performing generative models CDN, GMDLDR,
REINVENT 3.0 and Transmol, and three targets VDR (A) GABAA (B) and mTOR (C).

###  2.2 DTA prediction metric results

Table 2 gives a summary for the out-of-sample DTA predictions for the support
vector machine (SVM), GraphDTA and multitask label encoding model (MLT-LE)
models. The DTA predictions are initially performed for whole set of 400
molecules. Subsequently, predictions are sorted and only the top 10% highest
classification/regression outputs are shown. Additionally, box-plot
representations of these SVM predictions are depicted in Figure 4. In Table 2,
highest classification/regression outputs for a given model and reference
dataset are highlighted in bold. Similar to the ROOM metric, also here CDN,
GMDLDR, REVINVENT 3.0 as well as Transmol are the best performing models, i.e.
their generative output is most likely to create ligands for the reference
targets.

Figure 4: The SVM classification outputs. The box-plot shows the SVM
classification outputs for the 7 generative models and three reference targets
VDR, GABAA and mTOR.

Figure 5 shows kernel density estimates (KDE) for similarity, SVM, GraphDTA,
and MLT-LE scores for all generative methods. KDE for similarity shows that
Transmol creates focused libraries with high similarity to the reference set.
GMDLDR and REINVENT 3.0 also show this type of behaviour, albeit to a lesser
degree (left column of Figure 5). While KDE estimates of SVM outputs show bi-
modal distributions across generative models, GraphDTA and MLT-LE exhibit
mostly uni-modal distributions.

Figure 5: Distribution of the properties for the DTA-based prediction metrics.
Kernel density estimation (KDE) of similarity between input and generative
output (first column), SVM classification outputs (second column), GraphDTA
(third column) and MLT-LE pKD predictions (fourth columns) for the 7
generative models and three targets VDR (first row), GABAA (second row) and
mTOR (third row).

In order to create a ranking of generative models (and molecular docking), 10%
of the output molecules with highest DTA predictions (respectively absolute
docking scores for VDR) were selected and their predictions averaged.
Generative models were then ranked according to the average prediction score
of their generative output. Figure 6 shows the ranking of all considered
generative methods. The highest rank corresponds to the highest average
binding affinity, i.e. to high probability of ligand creation. As can be seen
in Figure 6 ranking across generative models is mostly conserved for mTOR,
while more variability in the ranking across methods is observed for VDR and
more so for GABAA. The top row shows the ranking according to the docking
scores for VDR. Interestingly, the ranking concurs with the DTA prediction
results.

Figure 6: Ranking of generative models based on the DTA predictors and docking
scores. The figure shows the ranking of the 7 generative models and three
targets VDR GABAAand mTOR based on DTA predictors and docking scores. The
ranking in dark red color (7) shows the best-performing generative models,
while the blue color (1) the worst.

Figures 7, 8 and 9 show the top five structures sorted by their SVM DTA
prediction score for each generative method and target. For VDR, a detailed
inspection of the top five molecular structures generated by CDN, GMDLDR,
REINVENT 3.0 and Transmol revealed that all of them contain the key structural
features of VDR binding molecules of the secosteroidal type. For GABAA, four
out of the top five molecules generated by CDN exhibit classical
benzodiazepine (BZD) scaffold, consisting of a fused benzene and diazepine
ring, while one molecule featured an expanded benzodiazocine core.
Interestingly, all the top molecules generated by GMDLDR, REINVENT 3.0 and
Transmol contain the polycyclic benzo[f]imidazo[1,5-a][1,4]diazepine scaffold
in which the benzodiazepine moiety is fused with an additional imidazole ring.
Finally, one molecule containing the BZD scaffold is found among the top five
structures produced by TransVAE. Due to the large variability of the mTOR
ligands, unique scaffolds or structural fragments are more difficult to
identify. Furthermore, while the overlap of scaffolds between molecules across
the outputs of different generative models does occur, it is not as extensive
as for VDR and GABAA. In particular, the most frequently occurring
1-(4-(1H-pyrazolo[3,4-d]pyrimidin-6-yl)phenyl)urea fragment appears in five,
four and one of the top five molecules generated by CDN, REINVENT 3.0 and
GMDLDR, respectively. In addition, a structurally related
1-(4-(pyrimidin-2-yl)phenyl)urea fragment is found in three out of the top
five molecules produced by GMDLDR. Another frequently found fragment,
5-((4-phenylpyrimidin-5-yl)ethynyl)pyridin-2-amine, appears in two and one of
the top five molecules stemming from Transmol and GMDLDR, respectively. It
should be noted that different scaffold distributions can be obtained if the
top five molecules are selected based on Tanimoto similarity score or on
binding strength indicators as opposed to SVM score. The respective molecular
structures produced by each generative model for all three reference targets
can be found in the supplementary material.

Figure 7: Top 5 generated molecules sorted by SVM output for VDR. The two
values under each structure represent the SVM and similarity scores,
respectively. Figure 8: Top 5 generated molecules sorted by SVM output for
GABAA. The two values under each structure represent the SVM and similarity
scores, respectively. Figure 9: Top 5 generated molecules sorted by SVM output
for mTOR. The two values under each structure represent the SVM and similarity
scores, respectively. Table 2: The mean value of the highest 10% of the scores
for each generative method

DTA method | SVM (is $\mu M<1$) | GCNNet (*pKD) | MLT-LE (*pKD)
---|---|---|---
| VDR | GABAA | mTOR | VDR | GABAA | mTOR | VDR | GABAA | mTOR
CDN | 0.947 | 0.979 | 0.998 | 8.901 | 7.285 | 7.479 | 8.728 | 7.087 | 7.70
GMDLDR | 0.961 | 0.984 | 0.999 | 8.447 | 7.284 | 7.341 | 8.692 | 7.385 | 7.788
LigDream | 0.272 | 0.902 | 0.533 | 6.634 | 7.09 | 6.323 | 6.068 | 6.971 | 5.784
REINVENT | 0.355 | 0.749 | 0.449 | 6.29 | 6.923 | 5.914 | 5.89 | 6.055 | 5.434
REINVENT 3.0 | 0.942 | 0.983 | 0.999 | 8.44 | 7.299 | 7.338 | 8.692 | 7.539 | 7.793
Transmol | 0.936 | 0.981 | 0.986 | 8.656 | 7.173 | 7.143 | 8.656 | 7.667 | 6.801
TransVAE | 0.278 | 0.881 | 0.432 | 6.446 | 7.222 | 5.677 | 6.043 | 6.639 | 5.643
* $pK\textit{\textsubscript{D}}=-log_{10}(\frac{K\textit{\textsubscript{D}}}{10^{9}}+1e-10)$

###  2.3 Molecular docking results

A comparison of heavy atom count between the VDR reference and generated
molecules revealed similar distributions for five of the seven generated
output datasets. REINVENT and TransVAE had smaller average heavy atom counts
compared to the reference dataset and other generative models (Figure 10). In
addition, the generated molecules from REINVENT and TransVAE had lower average
heavy atom count as compared to the reference dataset. Based on this analysis
and the low number of identified outliers, the Glide SP molecular docking was
performed without further filtering.

The validation of the Glide SP docking procedure using the top-5 poses
confirmed the feasibility of this approach. In particular, the conformation of
the docked molecule was in agreement with their orientation compared to the
original PDB structures or calcitriol in case of C2$\alpha$-butyl-calcitriol
analog (see supplementary data).

The results of molecular docking showed that ligands from four generative
models, namely CDN, GMDLDR, REINVENT 3.0 and Transmol have their distribution
of docking scores in the same range as the VDR reference dataset (Figure 10,
Table 3). Furthermore, superimposition based on the most common substructure
of the "best" (Figure 11A-D, highlighted in green) and the "worst" (Figure
11A-D, highlighted in red) pose, from the top performing generative models, to
the best calcitriol pose, from the VDR reference dataset, was further explored
and visualized (Figure 11). Root-mean square distance (RMSD), the measure of
similarity for the spatial conformation, indicates the lowest value 1.1 Å for
Transmol vs. 1.31 Å for the best pose of calcitriol (Figure 11D). The
calculated binding energies for the best poses are in the range of -14
kcal/mol. In contrast, the best pose of calcitriol has a binding energy of
-12.03 kcal/mol. For the four generative models, the best poses show a lower
docking score than the best pose for calcitriol, which indicates a more
favorable conformation and higher affinity. With respect to the "worst" poses
the position of the CD-ring is significantly shifted out of any alignment and
in some cases such as for Transmol the spatial orientation of the molecule is
reversed as compared to calcitriol (Figure 11D).

Table 3: Docking score statistics for VDR reference and generative datasets

| mean ± std | min | 25% | 50% | 75% | max
---|---|---|---|---|---|---
VDR reference | -10.87 ± 1.27 | -15.97 | -11.68 | -10.94 | -10.01 | -4.16
CDN | -10.20 ± 1.05 | -14.5 | -10.87 | -10.21 | -9.52 | -5.98
GMDLDR | -10.45 ± 1.06 | -13.78 | -11.19 | -10.47 | -9.77 | -6.33
LigDream | -8.14 ± 1.00 | -11.44 | -8.79 | -8.06 | -7.54 | -2.05
REINVENT | -6.70 ± 1.53 | -11.12 | -7.68 | -6.86 | -5.74 | -2.09
REINVENT 3.0 | -10.52 ± 1.07 | -14.39 | -11.22 | -10.49 | -9.81 | -6.75
Transmol | -10.43 ± 1.02 | -13.37 | -11.15 | -10.51 | -9.77 | -6.44
TransVAE | -7.14 ± 1.25 | -11.1 | -7.99 | -7.14 | -6.4 | -2.71

Figure 10: Distribution of ligand heavy atom counts and docking scores. The
box-plots show the heavy atom count (A) and docking scores (B) for the VDR
reference dataset (in red) and the outputs of 7 generative models. Lower
docking scores mean higher binding affinities. Figure 11: Illustrative docking
results for the four best-performing generative models. Comparison of the best
(green) and the worst (red) poses for the top compounds generated by the four
best-performing algorithms CDN (A), GMDLDR (B), REINVENT 3.0 (C) and Transmol
(D) to the reference calcitriol with its best re-docking pose (gray). The
chemical structure of the ligands is depicted under their three-dimensional
representation. The RMSD was calculated in-place against the reference (-12.03
kcal/mol, rmsd = 1.31Å).

###  2.4 Divergence between DTA-based predictions and molecular docking

Calcitriol (see Figure 12A molecule ‘i’), bottom-right panel) is regarded as
the most active natural metabolite of vitamin D3 and contains a secosteroid
backbone consisting of A-, C- and D-rings. The residual B-ring from the
sterane molecule manifests in the form of a conjugated diene linker, having
two exocyclic carbon atoms and connecting the A- to the C/D-rings, and a
methylene group on the A-ring (see Figure 12A molecule ‘i’). The carefully
positioned two hydroxyl (OH) groups on the A-ring at carbons C1 and C3 as well
as the third OH at C25 at the very end of an aliphatic chain fused onto the
D-ring have co-evolved with VDR for optimal receptor activation and subsequent
regulation of target genes Carlberg et al (2012).

Figure 12A depicts a scatter plot, where SVM-based DTA scores (x-axis) are
plotted against absolute docking scores (y-axis) for all VDR generated
molecules. The molecular outputs of the 7 generative models are color-coded.
For reference, these scores are also plotted for calcitriol and highlighted as
a red circle (Figure 12A compound ‘i’). Overall, DTA-based prediction scores
correlate well with docking scores of potential VDR ligands (concordance index
$=-0.77$). Individual molecules are distributed in two large clusters: a
cluster of unlikely binders having low docking and DTA scores; and a cluster
of likely binders for which both scores are relatively high indicating that
the two scores correlate well for the majority of generated molecules.

However, there are also several scarcely populated regions where the two
scores diverge. In particular, we were interested in examining two cases,
where molecules exhibit a high docking score but a low DTA-based prediction
score and vice versa. Close structural evaluation of molecules belonging to
both of these regions identified molecules that on first sight seem to
resemble the ideal secosteroid backbone inherent to calcitriol, yet they
represent chimeras that differ from the classical secosteroid structure in
various details. To illustrate this, the deviation from the secosteroid
backbone in these compounds is highlighted in orange color and shown in Figure
12 panel B.

In the first row of panel B the four molecules exhibit high docking but low
SVM scores (Figure 12B molecules ’a-d’). Interestingly, all these molecules
deviate from the classical secosteroid backbone in both, the ring system as
well as their aliphatic chains. The aliphatic chains are usually branched and
have a missing or incorrectly positioned OH group. For molecules ‘b’ and ‘d’,
the C-rings are smaller while the linker between the two ring systems is
longer compared to calcitriol. For molecule ‘c’, the D-ring is enlarged while
the linker is shortened. All these deviations may have led to the DTA-based
prediction assigning lower scores. With respect to molecular docking results,
all molecules were compared to the VDR-calcitriol crystal structure (PBDID
1DB1), where calcitriol has a “boat-shaped” conformation in the ligand-binding
pocket (LBP) of VDR. Only ‘b’ and ‘d’ showed comparable conformation to
calcitriol and the correct orientation in the LBP. While their correct
conformation may have led to high docking scores, it should be noted that the
position of OH groups as well as the orientation of these molecules are
incorrect. As described above the increase in the length and/or size of the
molecules due to a longer linker and branched aliphatic chain did not prevent
the molecules to take the correct boat-shaped conformation. Molecule ’c’
represents an interesting analog that is very similar to calcitriol in terms
of its size and although docked in an incorrect orientation some of the OH
group were positioned rather similarly compared to calcitriol. Molecule ’a’
has the longest aliphatic chain from all the four compounds (about 3.6 Å),
nonetheless the size of the LBP can easily accommodate it. Even though the
orientation compared to calcitriol is reversed, it fully follows the boat-
shaped confirmation.

The second row contains examples of molecules that show low docking but high
SVM-scores (Figure 12B molecules ‘e-h’). From the four molecules that we
examine here, ‘e’ and ‘g’ were generated by CDN. Both exhibit the
incorporation of larger rings. The tendency of CDN to incorporate larger rings
into the generated molecules can also be observed for two of the top five
molecules with the highest SVM-based DTA prediction scores (see Figure 7).

Molecules ‘e’, ‘g’ and ‘h’ show incorrect orientation as well as some
additional steric issues in their conformations. For ‘e’, which has a shorter
aliphatic chain than calcitriol, all the ring sub-structures are shifted away
from the residue W286, which creates a critical hydrophobic interaction with
the C/D-rings of calcitriol. As expected, a very similar feature can be
observed for molecule ‘g’. The most notable structural misalignment can be
noticed for ‘h’, which has the lowest docking score from all four molecules.
With compensation for the long linker that contains one of the OH groups the
molecule is shifted more than 3.5 Å leading to a deformed boat-shaped
confirmation. Although molecule ‘f’ shows rather high structural conservation
with calcitriol and a correct orientation in the LBP, the docking algorithm
tried to compensate for the additional ring and length of the aliphatic chain
by displacing the ring structures leading to a shift in the position of the
molecule. The alignment of OH groups is also incorrect, which may have also
contributed to the low docking score. For the limited examples that have been
examined here, it may seem that DTA predictions assign high scores to
molecules with a structurally similar backbone and overall size, whereas a
high docking score can be expected for molecules that follow calcitriol’s
evolutionary conserved boat-shaped conformation and positioning OH groups.
Thus in order for an individual ML-generated molecule to display appropriate
structural features inherent of likely binders, both SVM-based DTA prediction
and docking scores should be high. In addition to these scores, we believe
that the manual inspection of the docking results reinforces the judgement on
the correct conformation of the generated molecule for the respective protein
target.

Figure 12: Comparison of generated ligands, based DTA prediction and molecular
docking scores. The scatter plot depicts molecular outputs from all 7
generative models and compares SVM (x-axis) and absolute docking scores
(y-axis). The outputs from generative models are color-coded. For
illustration, calcitriol is shown as a red circle (molecule ’i’) (A).
Illustrative molecules with high docking, but low SVM scores (’a-d’; first
row) and molecules with low docking, but high SVM scores (’e-h’; second row)
are depicted. Their deviations from reference calcitriol are highlighted by
orange color and the missing functional or incorrectly positioned hydroxyl
group by blue circles (B).

###  2.5 MOSES metrics

In addition to ABRAHAM, MOSES metrics were calculated and are given in Table
4. Most of the generative models achieve high scores of around 0.8-1.0 with
standard metrics such as validity, uniqueness, novelty, etc. Moreover, models
that have good ABRAHAM scores, i.e. CDN, GMDLDR, REINVENT 3.0, and Transmol,
performed well for MOSES metrics that measure similarities between the
generated and testing sets of molecules. These metrics include fragment
similarity (Frag/Test), scaffold similarity (Scaf/Test), similarity to a
nearest neighbour (SNN/Test), and Fréchet ChemNet distance (FCD/Test).

Table 4: MOSES metrics for generated datasets: VDR; GABAA; and mTOR

VDR
---
Method | valid | unique@1000 | Filters | Novelty | IntDiv
CDN | 0.922 $\pm$ 0.059 | 0.783 $\pm$ 0.091 | 0.737 $\pm$ 0.090 | 1.000 $\pm$ 0.000 | 0.761 $\pm$ 0.048
GMDLDR | 1.000 $\pm$ 0.000 | 0.970 $\pm$ 0.011 | 0.939 $\pm$ 0.020 | 0.982 $\pm$ 0.002 | 0.763 $\pm$ 0.013
Ligdream | 1.000 $\pm$ 0.000 | 0.939 $\pm$ 0.011 | 0.955 $\pm$ 0.008 | 1.000 $\pm$ 0.000 | 0.817 $\pm$ 0.001
REINVENT | 0.970 $\pm$ 0.003 | 1.000 $\pm$ 0.001 | 0.830 $\pm$ 0.011 | 1.000 $\pm$ 0.000 | 0.842 $\pm$ 0.002
REINVENT 3.0 | 0.996 $\pm$ 0.002 | 0.786 $\pm$ 0.026 | 0.969 $\pm$ 0.006 | 0.952 $\pm$ 0.004 | 0.747 $\pm$ 0.018
TransVAE | 0.600 $\pm$ 0.006 | 1.000 $\pm$ 0.000 | 0.925 $\pm$ 0.004 | 1.000 $\pm$ 0.000 | 0.865 $\pm$ 0.001
Transmol | 1.000 $\pm$ 0.000 | 0.827 $\pm$ 0.028 | 0.797 $\pm$ 0.029 | 0.999 $\pm$ 0.000 | 0.816 $\pm$ 0.018
| FCD/Test | SNN/Test | Frag/Test | Scaf/Test | IntDiv2
CDN | 17.615 $\pm$ 2.737 | 0.376 $\pm$ 0.053 | 0.539 $\pm$ 0.096 | 0.123 $\pm$ 0.168 | 0.713 $\pm$ 0.048
GMDLDR | 5.518 $\pm$ 0.566 | 0.597 $\pm$ 0.011 | 0.941 $\pm$ 0.024 | 0.865 $\pm$ 0.033 | 0.723 $\pm$ 0.011
Ligdream | 36.024 $\pm$ 0.993 | 0.231 $\pm$ 0.002 | 0.472 $\pm$ 0.015 | 0.006 $\pm$ 0.005 | 0.804 $\pm$ 0.001
REINVENT | 45.934 $\pm$ 1.589 | 0.223 $\pm$ 0.004 | 0.473 $\pm$ 0.028 | 0.060 $\pm$ 0.008 | 0.835 $\pm$ 0.002
REINVENT 3.0 | 10.717 $\pm$ 0.851 | 0.579 $\pm$ 0.012 | 0.763 $\pm$ 0.031 | 0.441 $\pm$ 0.041 | 0.709 $\pm$ 0.016
TransVAE | 48.917 $\pm$ 0.950 | 0.200 $\pm$ 0.001 | 0.422 $\pm$ 0.018 | 0.020 $\pm$ 0.009 | 0.859 $\pm$ 0.001
Transmol | 10.021 $\pm$ 0.960 | 0.531 $\pm$ 0.016 | 0.862 $\pm$ 0.050 | 0.735 $\pm$ 0.087 | 0.779 $\pm$ 0.016
GABAA
Method | valid | unique@1000 | Filters | Novelty | IntDiv
CDN | 0.880 $\pm$ 0.106 | 0.419 $\pm$ 0.196 | 0.440 $\pm$ 0.245 | 0.996 $\pm$ 0.005 | 0.629 $\pm$ 0.124
GMDLDR | 1.000 $\pm$ 0.000 | 0.873 $\pm$ 0.017 | 0.778 $\pm$ 0.036 | 0.955 $\pm$ 0.005 | 0.766 $\pm$ 0.004
Ligdream | 1.000 $\pm$ 0.000 | 0.995 $\pm$ 0.004 | 0.934 $\pm$ 0.005 | 1.000 $\pm$ 0.000 | 0.847 $\pm$ 0.002
REINVENT | 0.985 $\pm$ 0.002 | 1.000 $\pm$ 0.000 | 0.763 $\pm$ 0.019 | 1.000 $\pm$ 0.000 | 0.835 $\pm$ 0.002
REINVENT 3.0 | 0.989 $\pm$ 0.004 | 0.740 $\pm$ 0.020 | 0.826 $\pm$ 0.027 | 0.886 $\pm$ 0.009 | 0.781 $\pm$ 0.006
TransVAE | 0.599 $\pm$ 0.003 | 1.000 $\pm$ 0.000 | 0.925 $\pm$ 0.003 | 1.000 $\pm$ 0.000 | 0.865 $\pm$ 0.000
Transmol | 1.000 $\pm$ 0.000 | 0.687 $\pm$ 0.051 | 0.665 $\pm$ 0.032 | 0.999 $\pm$ 0.000 | 0.808 $\pm$ 0.013
| FCD/Test | SNN/Test | Frag/Test | Scaf/Test | IntDiv2
CDN | 28.748 $\pm$ 5.008 | 0.381 $\pm$ 0.118 | 0.048 $\pm$ 0.034 | 0.202 $\pm$ 0.303 | 0.577 $\pm$ 0.110
GMDLDR | 6.911 $\pm$ 0.490 | 0.577 $\pm$ 0.013 | 0.872 $\pm$ 0.021 | 0.952 $\pm$ 0.014 | 0.746 $\pm$ 0.004
Ligdream | 29.804 $\pm$ 0.877 | 0.230 $\pm$ 0.002 | 0.612 $\pm$ 0.037 | 0.000 $\pm$ 0.000 | 0.838 $\pm$ 0.002
REINVENT | 38.210 $\pm$ 0.996 | 0.217 $\pm$ 0.003 | 0.498 $\pm$ 0.026 | 0.002 $\pm$ 0.002 | 0.828 $\pm$ 0.002
REINVENT 3.0 | 7.153 $\pm$ 0.282 | 0.573 $\pm$ 0.012 | 0.890 $\pm$ 0.014 | 0.967 $\pm$ 0.007 | 0.756 $\pm$ 0.007
TransVAE | 31.555 $\pm$ 0.591 | 0.205 $\pm$ 0.001 | 0.478 $\pm$ 0.018 | 0.000 $\pm$ 0.001 | 0.860 $\pm$ 0.000
Transmol | 11.525 $\pm$ 0.928 | 0.553 $\pm$ 0.027 | 0.827 $\pm$ 0.160 | 0.785 $\pm$ 0.119 | 0.785 $\pm$ 0.011
mTOR
Method | valid | unique@1000 | Filters | Novelty | IntDiv
CDN | 0.976 $\pm$ 0.016 | 0.594 $\pm$ 0.092 | 0.872 $\pm$ 0.035 | 0.975 $\pm$ 0.003 | 0.780 $\pm$ 0.024
GMDLDR | 1.000 $\pm$ 0.000 | 0.996 $\pm$ 0.003 | 0.844 $\pm$ 0.134 | 0.999 $\pm$ 0.001 | 0.828 $\pm$ 0.003
Ligdream | 1.000 $\pm$ 0.000 | 0.999 $\pm$ 0.001 | 0.953 $\pm$ 0.002 | 1.000 $\pm$ 0.000 | 0.845 $\pm$ 0.000
REINVENT | 0.976 $\pm$ 0.003 | 1.000 $\pm$ 0.000 | 0.819 $\pm$ 0.011 | 1.000 $\pm$ 0.000 | 0.840 $\pm$ 0.003
REINVENT 3.0 | 0.986 $\pm$ 0.003 | 0.980 $\pm$ 0.003 | 0.916 $\pm$ 0.008 | 0.885 $\pm$ 0.006 | 0.836 $\pm$ 0.002
TransVAE | 0.594 $\pm$ 0.006 | 1.000 $\pm$ 0.000 | 0.924 $\pm$ 0.006 | 1.000 $\pm$ 0.000 | 0.865 $\pm$ 0.001
Transmol | 1.000 $\pm$ 0.000 | 0.749 $\pm$ 0.104 | 0.729 $\pm$ 0.043 | 0.998 $\pm$ 0.000 | 0.840 $\pm$ 0.012
| FCD/Test | SNN/Test | Frag/Test | Scaf/Test | IntDiv2
CDN | 10.617 $\pm$ 1.700 | 0.664 $\pm$ 0.043 | 0.855 $\pm$ 0.031 | 0.344 $\pm$ 0.125 | 0.737 $\pm$ 0.031
GMDLDR | 9.678 $\pm$ 4.301 | 0.362 $\pm$ 0.011 | 0.901 $\pm$ 0.025 | 0.057 $\pm$ 0.043 | 0.818 $\pm$ 0.007
Ligdream | 15.756 $\pm$ 0.182 | 0.292 $\pm$ 0.001 | 0.803 $\pm$ 0.003 | 0.001 $\pm$ 0.000 | 0.839 $\pm$ 0.000
REINVENT | 25.530 $\pm$ 0.681 | 0.310 $\pm$ 0.003 | 0.832 $\pm$ 0.008 | 0.005 $\pm$ 0.002 | 0.833 $\pm$ 0.003
REINVENT 3.0 | 1.819 $\pm$ 0.072 | 0.613 $\pm$ 0.005 | 0.995 $\pm$ 0.001 | 0.824 $\pm$ 0.016 | 0.819 $\pm$ 0.002
TransVAE | 20.256 $\pm$ 0.215 | 0.270 $\pm$ 0.001 | 0.833 $\pm$ 0.004 | 0.003 $\pm$ 0.001 | 0.859 $\pm$ 0.001
Transmol | 13.894 $\pm$ 1.740 | 0.534 $\pm$ 0.025 | 0.789 $\pm$ 0.056 | 0.209 $\pm$ 0.096 | 0.805 $\pm$ 0.013

##  3 Discussion

In this study, we proposed a new biologically inspired benchmark for
evaluating the performance of _de novo_ molecular generation methods and for
assessing their ability to learn structural features of ligands responsible
for the interaction with a specific target.

For our benchmark we carefully designed three distinct target-based reference
datasets for the evaluation of generative models: VDR; GABAA; and mTOR. Each
of the selected targets is well known and widely studied, and thus a
significant number of ligands have been identified for all of them. At the
same time, there are a number of key differences _between_ the reference
datasets that we have extracted and curated. Firstly, the number of ligands in
the datasets varies substantially: $N=370$ for VDR; $N=256$ for GABAA; and
$N=4625$ for mTOR (see Table 5 for a quantitative description of reference
datasets including MOSES metrics). This difference in ligand number directly
affects the training of the generative models: While for some generative
models a small number of samples may be sufficient, others may need more data
to train a reasonable model. Secondly, the variability of molecular weights
differs between ligands of specific targets: While VDR and GABAA ligands have
large variability, mTOR ligand variability is comparatively small. Thirdly,
the IntDiv within ligands of a specific target differs. While VDR and GABAA
ligands contain similar molecular scaffolds within their respective datasets,
mTOR ligands show greater diversity and may thus require a more complex
generative model to capture this variability, which in turn required more
training data. In summary, this non-exhaustive list of differences between
ligand-classes adds additional constraints for a given generative model to be
successful at creating a meaningful output across all benchmark datasets.

Once the generative model has been trained on a subset of ligands for a given
target and tasked to create a focused library, this output allows us to: i)
calculate the number of recreated ligands; ii) calculate DTA predictions; and
iii) perform molecular docking.

The recreation metric that has been proposed here allows to not only estimate
the ability of creating focused libraries for a given generative model but
also examines its output from a functional perspective that has direct
relevance for the drug discovery process. A generative model that succeeds in
recreating a large fraction of known ligands in an out-of-sample fashion is
clearly able to model the variability of this compound class accurately. As a
result, this makes the generation of other so far unknown binders more likely.

The second metric we proposed is based on the calculation of the average
binding affinity of generated molecules. The underlying assumption here is
that if a DTA model is able to accurately predict whether an unseen molecule
acts as a ligand for a specific target, then it can be employed as a validator
of the molecular outputs of generative models. A large range of DTA predictors
have been proposed to date. Some of them being target-specific, while others
are target-invariant. In this work we have employed a target-specific SVM, a
number of related graph-based architectures, such as GraphDTA and in addition
proposed a multi-task learning label encoding approach. Predictions of these
models were used to rank generative models according to the predicted average
binding affinities of their generative outputs. This ranking showed a high
level of homogeneity across generative models and can thus be seen as a
legitimization of this approach (see also Figure 6). In addition, the DTA
prediction metric led to results, which are in line with those of the
recreation metric.

Molecular docking is _de facto_ the most accurate computational tool available
for predicting the interaction of two molecules and is being used to evaluate
ML-based methods for predicting binding affinities Maziarka et al (2020).
Since molecular docking provides insight into the quality of drug-target
interactions, docking results are a viable approach to further support the
reliability of the proposed alternative metrics. Molecular docking was
performed for the VDR reference data as well as the generated output sets. As
expected, the ligands from the VDR reference set exhibited the lowest average
docking score (i.e. strongest affinity binding). However and more
interestingly, the same four generative models (CDN, GMDLDR, REINVENT 3.0 and
TransMOL) that performed well in the ROOM and DTA prediction metrics also
demonstrated low average docking scores for their generated molecules. Please
note that the best poses produced by these generative models showed even the
lower docking scores than for calcitriol and had very similar conformations.
In summary, molecular docking not only served as an alternative and
independent validation of the generative models’ output, but was also able to
concur the results of the two other proposed metrics.

Additionally, we have examined possible relations between ABRAHAM and MOSES
metrics. As stated earlier, fragment similarity (Frag/Test), scaffold
similarity (Scaf/Test), similarity to a nearest neighbour (SNN/Test) and
Fréchet ChemNet distance (FCD/Test) metrics show a similar trend. They thus
confirm the recreation metric, which is not surprising considering the fact
that these metrics measure various types of similarities between two sets of
molecules. For instance, Frag/Test measures the similarity of BRICS fragments
Degen et al (2008) present in reference and generated sets. Since training and
testing sets used in our study belong to the same ligand set, e.g. VDR, GABAA,
or mTOR, they contain similar fragments. The more molecules from the testing
set a generative model is able to recreate, the higher the similarity between
fragments present in the testing set and the generated set will be, as the
number of overlapping molecules, hence overlapping fragments, increases.

As mentioned earlier, CDN, GMDLDR, REINVENT 3.0 and Transmol were able to
perform well in all three considered metrics. However, LigDream, REINVENT and
TransVAE did not perform optimally for the given tasks. One potential reason
for the failure of LigDream and REINVENT are the lengths of the generated
SMILES strings. Both methods seem to have difficulty in creating molecules
with appropriate numbers of heavy atoms (see upper part of Figure 10). For
LigDream, this may be linked to the shortcomings of the long short-term memory
(LSTM). For REINVENT the failure in ligand recreation may be explained by the
design of the reward function (termed scoring function), which has sparse
rewards and thus making the training unstable. Their reward function design
takes single molecules as a target instead of using a sequence of molecules.
As a result, the scoring function may fail to navigate the agent through
chemical space meaningfully. There are multiple potential reasons for the
failure of TransVAE. First, training sets might have been too small to
estimate entropy values accurately enough and/or to fine-tune the pre-trained
models sufficiently. Second, the dataset that the TransVAE model was pre-
trained on and our datasets might have conflicting distributions of molecules.
In particular, some atoms present in our dataset were absent in the
vocabularies of the pre-trained models.Additionally, TransVAE uses training
sets to calculate entropies and select dimensions of the latent space that
have high entropy, but this may not be sufficient for extracting meaningful
information from the given training sets. For a more in-depth discussion of
how models were trained and why some of them have performed sub-optimally for
the given metrics, we would like to refer the reader to the supplementary
material.

The unique combination of reference datasets and metrics that we proposed for
the evaluation of generative models are more closely related to the drug
discovery process since ligand-target relations are taken into account.
However, our benchmark currently suffers from a number of limitations that
need to be addressed: i) While we have carefully designed three diverse
reference datasets, there may be other targets, which may enrich the proposed
framework. Some of our future work will focus on creating a more comprehensive
set of targets, which will add additional constraints for the creation of
novel and more powerful generative models; ii) At this time, only the
recreation metric as well as multiple ML-based DTA prediction models are
available via our GitHub repository111https://gitlab.com/cheml.io/abraham to
test any generative model. The reason for molecular docking not being included
is that the required software is proprietary and that some expert knowledge is
required to run these types of simulations accurately and interpret the
results. We are planning to incorporate this feature into our repository in
the future using freely available software, such as AutoDock Morris et al
(2009); iii) A further limitation of this study lies in the _in silico_ nature
of the benchmark. While generative models are able to provide meaningful
outputs and have shown their potential of actually being able to assist and
speedup the drug discovery process Merk et al (2018b, a); Bruns et al (2019);
Zhavoronkov et al (2019); Grisoni et al (2021); Friedrich et al (2020, 2021);
Chenthamarakshan et al (2022), to date, only wet-lab experiments can
ultimately decide on whether a given target candidate is effective and safe.

During our analysis, we compared the generative outputs in terms of ML-based
DTA predictions and molecular docking. While the comparison of these two
metrics generally exhibited high concordance, a number of cases were
identified where specific molecules showed diverging scores. Examining these
molecules in terms of their structural composure revealed that relying on
either DTA predictions or molecular docking alone may lead to inaccurate
conclusions about potential binding properties of given molecules. We believe
this to be a highly important result, in particular, since molecular docking
is generally accepted as the gold standard methodology for high-throughput
screening during the early stages of the drug discovery process. Instead of
trusting a single methodology, we thus propose to rather employ a multi-modal
approach, comprised of a range of metrics, such as the ones that have been
utilized in this work.

Machine learning-based generative models are currently receiving considerable
attention from the scientific community and their usage has led to many
interesting insights as well as novel applications in a whole variety of
fields, such as computer science, material science, medical science and drug
discovery among many others Aggarwal et al (2021); Abukmeil et al (2021); Gm
et al (2020). However to date, while of paramount importance for their
evaluation, benchmarks have not yet received the necessary attention. In this
paper we argue that when designing benchmarks for generative models,
structural domain knowledge which is functionally task-relevant needs to be
incorporated as an evaluation tool. Here, this has been achieved by creating a
diverse collection of benchmark datasets, which are composed of ligand sets
and their associated targets. By leveraging these reference datasets and their
associated metrics, a direct measure of relevance of a generative output can
be obtained and thus a new task-associated descriptor of the model itself can
be constructed going well beyond a mere successful data generation by the
model in a functionally targeted manner. While having successfully adapted
this type of framework for applications in drug discovery, the idea of not
only examining generative outputs in isolation (or with respect to its input),
but rather paying attention to its _functional properties_ is applicable to
studying and evaluating generative models in general. Such novel benchmarking
approaches including structural and domain prior knowledge which focus on
functionality will likely lead to generative models which are enabled to
create a more directed and thus more practically useful output.

## Conclusions

In summary, we propose a unique combination of reference datasets and metrics
for evaluating generative ML models for molecular generation from a viewpoint
of their potential applications in the drug discovery process. As a first
indicator, we assessed the ability of models to recreate known binders from
three reference molecular datasets associated with their specific biological
targets, namely VDR, GABAA and mTOR. As a further indicator, we analyzed ML-
based predictions of drug-target affinity between the molecular outputs
generated by the models which were trained and fine-tuned on our reference
datasets and associated biological targets. Importantly, these two indicators
were found to correlate well between each other. In other words, generative
models that achieved a high recreation rate were also likely to achieve high
ranking results for DTA-based predictions. Furthermore, in the case of VDR,
the recreation and DTA metrics were generally found to be in agreement with
the results of molecular docking. However, we also identified a number of
molecules, where DTA predictions diverged from molecular docking scores. By
analyzing of these particular cases, we conclude that for high DTA prediction
scores, generated molecules need to exhibit a structurally similar backbone
and overall size, while for high molecular docking scores the 3D confirmation
and the position of critical interacting functional groups is crucial. In
summary, we show that both modalities complement each other and thus both need
to be considered to make well-informed judgements about the quality of
potential drug-like compounds during the discovery phase of the drug design
process.

In addition, this study shows that incorporating structural and domain prior
knowledge which focus on functionality when designing benchmarks for
generative models can not only help with the evaluation of generative models
but may also lead to models with more directed and thus more practically
useful output.

##  4 Methods

###  4.1 Generative models

To ensure wide applicability and compatibility of our proposed metrics, seven
different generative models have been chosen for assessment: CDN; LigDream;
GMDLDR; REINVENT; REINVENT 3.0; Transmol and TransVAE. These models use
diverse ML architectures including transformers, recurrent neural networks,
autoencoders and reinforcement learning. A brief explanation, graphical
overview of each generative model’s architecture and how they were trained can
be found in the supplementary material, while more detailed descriptions can
be found in the original articles.

###  4.2 Reference datasets for metric evaluation

For training and testing generative models we utilized three distinct chemical
datasets that contained sets of ligands for VDR, GABAA $\alpha$ subunit and
mTOR, all of which are well studied and established targets.

Calcitriol, the most active secosteroid vitamin D metabolite, exerts its
ligand-dependent agonistic gene regulatory effects through VDR which is a
member of the nuclear receptor superfamily of Zn-finger transcription factors
Molnár (2014). VDR’s ubiquitous tissue expression pattern Wang et al (2012)
indicates its crucial role in nearly all physiological processes from
calcium/phosphate mineral homeostasis, cellular proliferation and
differentiation, as well as in the regulation of immune responses Norman
(2006). Most of the 4000-5000 synthetic VDR analogs are modifications of the
aromatic secosteroid scaffold and/or aliphatic side chain Carlberg and Molnár
(2015). The VDR ligand dataset, which has been manually curated for matching
assay type and activity, consists of 370 molecules extracted from various
chemical databases including ChEMBL, PubChem and ZINC.

BZDs are not naturally occurring ligands, but are a class of synthesized
psychoactive drugs. Their chemical scaffold results from fusing a benzene and
a diazepine ring Möhler et al (2002). Many of the representatives of this
class act by binding to an alternative BZD binding site of the GABAA receptor,
located between the $\alpha 1$ and $\gamma 2$ subunits and are acting as
agonists Möhler et al (2002). Thus, BZDs primarily act as anxiolytics,
hypnotics, anticonvulsants and muscle relaxants. A targeted substructure
search of the BZD scaffold resulted in 256 molecules with structurally highly
similar compounds. While the vast majority of the extracted BZDs are binders
of the GABAA site, only a small fraction lacks binding data.

mTOR inhibitors, modulate the function of an evolutionary conserved
Serine/Threonine kinase and its protein complexes play a central role in
controlling transcription, translation Chen and Zhou (2020), ribosomal
synthesis as well as cell growth and size. They can be categorized into four
groups: i) derivatives of the originally identified inhibitor - rapamycin
(first generation); ii) smaller ATP-competitive inhibitors (second
generation); iii) mTOR/PI3K dual inhibitors (second generation); and iv) all
newer inhibitors (third generation) Chen and Zhou (2020). mTOR was chosen as a
target with a large number of modulators. While these modulators may not
directly bind to mTOR, they nevertheless modulate its activity. For
simplicity, we will use the term ligand for all reference compounds throughout
the manuscript. The dataset which contains 4770 unique human-specific mTOR
compounds was extracted from PubChem, ZINC and ChEMBL.

Reference dataset representation and its quantitative description: All
molecules were represented in the simplified molecular-input line-entry system
(SMILES) format Weininger (1988) and required to pass RDKit validity Landrum
(2016). Subsequently, all molecule’s isomeric information was removed, they
were canonicalized via RDKit and finally all duplicates were removed. For each
reference dataset (VDR, GABAA, mTOR), 10 random 50:50 splits were created to
obtain training and testing sets. To quantitatively characterize all three
datasets, averages of MOSES metrics Polykovskiy et al (2020) were calculated
across splits. By comparing the averaged metrics between the train and test
splits, the differences and similarities between datasets can be evaluated.
Table 5 provides a summary of the MOSES metrics.

Table 5: Quantitative description of reference datasets including MOSES
metrics. N. molecules - number of molecules, N. heavy atoms - average number
of atoms, N. clusters - number of clusters, FCD/Test - Fréchet ChemNet
Distance to the test set, Scaf/Test - scaffold similarity to test set,
SNN/Test - nearest neighbour similarity to test set, IntDiv - internal
diversity, logP(W1D) - the octanol-water partition coefficient’s 1D
Wasserstein-1 distance to the test set, MW(W1D) - 1D Wasserstein-1 distance of
molecular weights.

Metric | VDR | GABAA | mTOR
---|---|---|---
N. molecules | 370 | 256 | 4625
N. heavy atoms | 33.9 | 23.1 | 33.1
N. clusters | 50 | 89 | 949
FCD/Test | 2.9309 $\pm$ 0.2228 | 5.8417 $\pm$ 0.1589 | 1.0353 $\pm$ 0.04
SNN/Test | 0.7779 $\pm$ 0.0055 | 0.6693 $\pm$ 0.008 | 0.7506 $\pm$ 0.0026
Frag/Test | 0.981 $\pm$ 0.0077 | 0.8831 $\pm$ 0.0109 | 0.9976 $\pm$ 0.0004
Scaf/Test | 0.9455 $\pm$ 0.0174 | 0.9687 $\pm$ 0.0097 | 0.8939 $\pm$ 0.0049
IntDiv | 0.7517 $\pm$ 0.0055 | 0.743 $\pm$ 0.0027 | 0.8284 $\pm$ 0.0006
logP(W1D) | 0.0121 $\pm$ 0.0158 | 0.044 $\pm$ 0.0346 | 0.0038 $\pm$ 0.0038
MW(W1D) | 62.6983 $\pm$ 54.4703 | 73.217 $\pm$ 37.1869 | 21.7956 $\pm$ 16.4942

Internal diversity (IntDiv) measures the dissimilarity _within_ a given
molecular set using Tanimoto similarity Tanimoto (1958). While all three
datasets are diverse, mTOR exhibits the highest diversity. Since VDR ligands
and BZDs adhere to a limited number of scaffolds, these datasets are expected
to have lower IntDiv. While having similar IntDiv, VDR ligands have higher
fragment similarity (Frag/Test) and BZDs are more conserved in terms of
scaffolds (Scaf/Test). Apart from similarity considerations _within_ datasets,
an effort was made to construct datasets that also vary _between_ each other.
This can be assessed by the average number of atoms, molecular weight (MW)
distributions and other chemical properties as described by the average 1D
Wasserstein-1 distance (W1D) of logP, among others.

###  4.3 Generation of molecules

For generating molecules, a diverse set of seven generative models were
employed and the same pipeline was applied for each of them. For each split,
the model was trained and/or fine-tuned on the corresponding ligand training
subset. This process resulted in 10 sets of generated molecules per model.
Output sets of generated molecules, were reduced in number by filtering them
according to their similarity using the combination of Morgan fingerprint
representations with radius 2 and length 2048 and Tanimoto similarity.

Instead of calculating the similarity between each reference compound with
respect to the generated ones, the Butina clustering algorithm Butina (1999),
with a similarity cutoff parameter of 0.4 Olivecrona et al (2017) was applied
to the reference compounds and the obtained cluster centroids were used for
similarity calculation. The maximum similarity for each generated compound was
then calculated with respect to the centroids. Table 5 shows the number
clusters used for the similarity computation.

This process led to same size subsets of generated compounds for each
generative method. The following steps summarize the above described
procedure:

  1. i)

All 10 splits for a given reference dataset and generative model were
concatenated;

  2. ii)

Invalid molecules were removed using RDKit;

  3. iii)

All generated molecules were canonicalized using RDKit;

  4. iv)

Duplicates were removed;

  5. v)

Generated molecules also present in the training set were removed;
This data was subsequently used for the recreation metric (see Sec. 4.4.1)

  6. vi)

The molecules were sorted by their average similarity with respect to
reference cluster centroids; and

  7. vii)

Top 400 molecules with the highest similarity were kept for each generative
model;
This data was subsequently used for DTA predictions and Molecular Docking (see
Sec. 4.4.2 and 4.4.3)

###  4.4 Proposed metrics

####  4.4.1 Recreation metric (ROOM)

In this study, we propose a new metric for evaluating the performance of _de
novo_ molecular generation models. This metric is based on the assumption that
the number of recreated ligands is related to the ability of the model to
generate molecules with properties similar to those of known binders. Thus
high values of this metric suggest the ability of a given generative model to
create molecules with a high probability of being ligands for a specific
target.

To calculate this metric, two non-overlapping datasets are required: a
training and a test dataset, both of which contain known binders. After
training the generative model on the training set and generating a set of
output molecules, the test set is used to calculate the overlap between the
generated and test set molecules. This procedure must be performed several
times to ensure consistency, with varying molecular sets allocated to the test
and training sets each time. A detailed workflow for examining the recreation
rate of known ligands for a given generative model is shown in Algorithm 1.

Algorithm 1 ROOM

1:Average recreation rate

2:Perform 10 splits of known binders into two equal subsets (50:50)

3:Use the training molecules of the first split for fine tuning the generative
model

4:Bring molecules into canonical form and discard the stereo-isomeric
information

5:Use canonical form of known binders as a seed for the generative model to
create a set of generated molecules

6:Compare the generated molecules to the known binders in the test set and
record the overlap

7:Repeat steps 2 to 5 for ten splits and report average recreation rate

8:Repeat steps 2 to 6 for each generative model

####  4.4.2 DTA prediction-based metric

Another metric we propose in this benchmark focuses on ML-based DTA
prediction. Binding affinity is an important characteristic of protein-ligand
interaction and a popular metric in virtual screening. Here, DTA predictions
are computed between generative molecular outputs and three predefined
reference targets (i.e. VDR, GABAA, mTOR). In particular, we use a number of
ML-based methods for DTA prediction: an SVM Cortes and Vapnik (1995), state-
of-the-art GraphDTA Nguyen et al (2021) and a newly proposed multitask label
encoding model (MLT-LE). In addition, we also use three models from the
GraphDTA library, namely GAT-GCN, GCNNet and GINConvNet. All models were
trained on subsets extracted from BindingDB Liu et al (2007), which contains
2.3 million binding records.

There are two main approaches for ML-based DTA prediction: i) A classifier is
trained for a specific target. Here, all known ligands are used for model
estimation and subsequent DTA prediction. For well studied and characterized
targets such as VDR, GABAA, and mTOR, where a lot of experimental data is
available, it is possible to train small but accurate models. ii) Here a
target-independent classifier is trained. These types of DTA prediction models
are generally more complex, require vast amounts of data for training, but are
able to generalize well to new, unseen drug-target pairs with unknown
affinity. In this work, we use both approaches to rank the generated
molecules.

##### SVM

To binarize the extracted binding data, we used a threshold of $<1\mu M$ to
distinguish between active and inactive ligands (as has previously been done
in Moret et al (2020)). All datasets were class-balanced using the random
oversampling technique from the imbalanced-learn package Lemaître et al
(2017). For the VDR subset, 4.7% of the data were excluded, and for GABAA and
mTOR, 25.7% and 27.7%, respectively. After the preprocessing, the VDR, GABAA,
and mTOR datasets totaled 742, 678, and 7072 records, respectively. Three SVMs
with RBF kernels and automatic per-class parameter balancing were trained in
scikit-learn v1.0.1 Pedregosa et al (2011), using the Morgan fingerprint
representation with a radius of 2 and a length of 2048. The regularization
parameters of these models were tuned using a grid search with 3-fold cross-
validation, with $C=1,4,16$ for the VDR, GABAA, and mTOR datasets,
respectively. The value of $\gamma$ was set to $\frac{1}{N*Var}$, where $N$ is
the number of features and $Var$ is the total variation in the feature data.
Cross-validation of VDR, GABAA, and mTOR classifiers resulted in an accuracy
of 88.5%, 89.4% and 93.7%, respectively, and an F1 score of 88.6%, 88.9% and
93.6%. After cross-validation, the classifiers were trained on all data with
hyperparameters obtained from cross-validation.

##### GraphDTA and MLT-LE

The training datasets for GraphDTA and MLT-LE were obtained from TDC BindingDB
Huang et al (2021) and BindingDB v2022m3 Liu et al (2007), respectively. Only
human target-specific information from both datasets were retrieved. Both
datasets contained multiple binding affinity records for the same drug-target
pairs due to different experimental assays. All records for drug-target pairs
with a variance of more than two standard deviations were removed from the TDC
BindingDB data. Additionally, all invalid SMILES strings from the BindingDB
v2022m3 were removed and canonicalized using RDkit. The median was calculated
for each drug-target pair with multiple records Tang et al (2014). For the TDC
BindingDB 42,236, 296,685, 766,904 and 0 unique drug-target pairs were
obtained for KD, KI, IC50, and EC50, respectively. For BindingDB v2022m3 the
following unique drug-target pairs were obtained: 39,379, 214,052, 610,502 and
84,487. Three GraphDTA models were then trained separately for KD, KI, and
IC50 using the TDC BindingDB, while one MLT-LE model was trained on all data
at once using BindingDB v2022m3. Further details on the training procedure can
be found in the supplementary material. The data were subsequently divided
into training and test sets with 80:20 splits.

GraphDTA represents small molecules as a graph and target are encoded by 1D
convolutional neural networks (CNNs) Nguyen et al (2021). GraphDTA models were
implemented using PyTorch Geometric Fey and Lenssen (2019). In addition, we
incorporate a novel multitask label encoding approach termed MLT-LE, that
allows all binding affinity constants, namely KD, KI, IC50, and EC50, to be
used for training simultaneously and in turn predicts them at the same time.

Table 6 provides an overview of the dataset, the affinity measures and a
summary of all employed architectures for DTA prediction.

Table 6: Standard affinity measures and DTA prediction method reference

Dataset |  |
---|---|---
BindingDB |  BindingDB is a public, web-accessible database of measured binding affinities.
Biological Activity Measures |  |
pKD |  The equilibrium dissociation constant (KD) measures the propensity of a ligand-target complex to dissociate. In other words, (KD) measures the equilibrium between the ligand-target complex and the dissociated components. pKD is $log$ transformed KD.
pKI |  The KI inhibition constant also represents a dissociation constant, but more narrowly for the binding of an inhibitor to an enzyme target. An inhibitor is a ligand whose binding reduces the catalytic activity of the enzyme. KI can thus be used as an indicator of how potent the inhibitor is. KI is $log$ transformed KI.
pIC50 |  pIC50 stands for a 50 % inhibitory concentration. It is the concentration of inhibitor required to reduce the biological activity of interest (typically the rate of the enzymatic reaction) to half of the maximum uninhibited value. pIC50 is $log$ transformed pIC50
pEC50 |  EC50 refers to 50 % effective concentration. It is the ligand concentration at which 50 % of its maximum effect is exerted on the biological target. The term is rather general and can be applied regardless of whether the ligand inhibits or enhances (induces) the biological function of the target. pEC50 is $log$ transformation of EC50.
Proba. |  Probability estimate of biological activity based on labeling where all measurements $<1(\mu M)$ were classified as _biologically active_.
Architectures |  |
GATNet |  Models form state-of-the-art DTA library - GraphDTA |  Graph Attention Network Nguyen et al (2021).
GAT-GCN |  Combined Graph Attention Network and Graph Convolutional network Nguyen et al (2021).
GINConvNet |  Graph Iisomorphism Network Nguyen et al (2021).
MLT-LE |  New deep-learning model |  Multi-task Network with Label Encoding.
SVM |  Classical Support Vector Machine Classifier from scikit-learn package |  Classical machine learning classification algorithm.

####  4.4.3 Molecular docking

Molecular docking, a well established tool in virtual screening, was used as a
validation methodology for outputs of the generative models as well as for
confirming results of ROOM and DTA predictions. The entire VDR reference
dataset was used for docking. Prior to docking, the outputs of each generative
model were ranked according to their calcitriol Tanimoto similarity and the
top 400 molecules were chosen for docking. After docking, these molecules were
ranked using the docking scoring function. To check the suitability of the
generated compounds for molecular docking, the number of heavy atoms was
calculated. Based on this analysis only a few outliers were identified and
thus the preparation of ligands using the LigPrep module and the subsequent
Glide SP molecular docking was performed without further filtering. All
procedures were done in Schrödinger Release 2021-2: Maestro, Schrödinger, LLC,
New York, NY, 2021. Prior to docking the molecules from the generative models,
the validation of the Glide SP docking procedure was performed by analyzing
the top-5 poses produced from the docking of the reference VDR dataset. The
analyzed subset contained four nonsecosteridal aryl acetic acid substitution
analogs of calcitriol: two isomers of CID11577808 ligand (PDBID 3W0C) and two
isomers of CID57432986 ligand (PDBID 3W0A). The last, fifth pose was an isomer
of C2$\alpha$-butyl-calcitriol (CID45484898). A more detailed description of
the docking procedure can be found in the supplementary material.

## Acknowledgments

The co-authors would like to acknowledge the support of Nazarbayev University
Research Grants funding to SF, VP and FM (240919FD3926, 11022021FD2903,
110119FD4520). We are also immensely grateful to Prof. Klaus-Robert Müller for
his valuable comments on a earlier versions of the manuscript. They have
increased the quality and scope of the paper substantially. We would also like
to thank Alexa Etheridge for proofreading the manuscript.

## References

  *   * Abukmeil et al (2021) Abukmeil M, Ferrari S, Genovese A, et al (2021) A Survey of Unsupervised Generative Models for Exploratory Data Analysis and Representation Learning. ACM Comput Surv 54(5):1–40. 10.1145/3450963
  * Aggarwal et al (2021) Aggarwal A, Mittal M, Battineni G (2021) Generative adversarial network: An overview of theory and applications. Int J Inf Manag 1(1):100,004. 10.1016/j.jjimei.2020.100004
  * Blaschke et al (2020) Blaschke T, Arús-Pous J, Chen H, et al (2020) REINVENT 2.0: An AI Tool for De Novo Drug Design. J Chem Inf Model 60(12):5918–5922. 10.1021/acs.jcim.0c00915
  * Brown et al (2019) Brown N, Fiscato M, Segler MH, et al (2019) GuacaMol: Benchmarking Models for de Novo Molecular Design. J Chem Inf Model 59(3):1096–1108. 10.1021/acs.jcim.8b00839
  * Bruns et al (2019) Bruns D, Merk D, Santhana Kumar K, et al (2019) Synthetic Activators of Cell Migration Designed by Constructive Machine Learning. ChemistryOpen 8(10):1303–1308. 10.1002/open.201900222
  * Butina (1999) Butina D (1999) Unsupervised data base clustering based on daylight’s fingerprint and Tanimoto similarity: A fast and automated way to cluster small and large data sets. J Chem Inf Comput Sci 39(4):747–750
  * Carlberg and Molnár (2015) Carlberg C, Molnár F (2015) Vitamin D receptor signaling and its therapeutic implications: Genome-wide and structural view. Can J Physiol Pharmacol 93(5):311–318. 10.1139/cjpp-2014-0383
  * Carlberg et al (2012) Carlberg C, Molnár F, Mouriño A (2012) Vitamin d receptor ligands: the impact of crystal structures. Expert Opinion on Therapeutic Patents 22(4):417–435. 10.1517/13543776.2012.673590
  * Chen et al (2018) Chen H, Engkvist O, Wang Y, et al (2018) The rise of deep learning in drug discovery. Drug Discov Today 23(6):1241–1250. 10.1016/j.drudis.2018.01.039
  * Chen and Zhou (2020) Chen Y, Zhou X (2020) Research progress of mTOR inhibitors. Eur J Med Chem 208:112,820. 10.1016/j.ejmech.2020.112820
  * Chenthamarakshan et al (2022) Chenthamarakshan V, Hoffman SC, Owen CD, et al (2022) Accelerating Inhibitor Discovery for Multiple SARS-CoV-2 Targets with a Single, Sequence-Guided Deep Generative Framework Preprint at https://arxiv.org/abs/2204.09042
  * Cortes and Vapnik (1995) Cortes C, Vapnik V (1995) Support-vector networks. Machine learning 20(3):273–297
  * Degen et al (2008) Degen J, Wegscheid-Gerlach C, Zaliani A, et al (2008) On the Art of Compiling and Using ’Drug-Like’ Chemical Fragment Spaces. ChemMedChem 3(10):1503–1507. 10.1002/cmdc.200800178
  * Dollar et al (2021) Dollar O, Joshi N, Beck DAC, et al (2021) Attention-based generative models for de novo molecular design. Chem Sci 12(24):8362–8372. 10.1039/D1SC01050F
  * Fey and Lenssen (2019) Fey M, Lenssen JE (2019) Fast Graph Representation Learning with PyTorch Geometric 10.48550/ARXIV.1903.02428
  * Friedrich et al (2020) Friedrich L, Byrne R, Treder A, et al (2020) Shape Similarity by Fractal Dimensionality: An Application in the de novo Design of (-)-Englerin A Mimetics. ChemMedChem 15(7):566–570. 10.1002/cmdc.202000017
  * Friedrich et al (2021) Friedrich L, Cingolani G, Ko YH, et al (2021) Learning from Nature: From a Marine Natural Product to Synthetic Cyclooxygenase-1 Inhibitors by Automated De Novo Design. Adv Sci 8(16):2100,832. 10.1002/advs.202100832
  * Gebauer et al (2019) Gebauer N, Gastegger M, Schütt K (2019) Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules. In: Wallach H, Larochelle H, Beygelzimer A, et al (eds) Advances in Neural Information Processing Systems, vol 32. Curran Associates, Inc., URL https://proceedings.neurips.cc/paper/2019/file/a4d8e2a7e0d0c102339f97716d2fdfb6-Paper.pdf
  * Gm et al (2020) Gm H, Gourisaria MK, Pandey M, et al (2020) A comprehensive survey and analysis of generative models in machine learning. Computer Science Review 38:100,285. 10.1016/j.cosrev.2020.100285
  * Grisoni et al (2021) Grisoni F, Huisman BJH, Button AL, et al (2021) Combining generative artificial intelligence and on-chip synthesis for de novo drug design. Sci Adv 7(24):eabg3338. 10.1126/sciadv.abg3338
  * Harel and Radinsky (2018) Harel S, Radinsky K (2018) Accelerating Prototype-Based Drug Discovery using Conditional Diversity Networks. Preprint at https://arxiv.org/abs/1804.02668
  * Hu et al (2021) Hu F, Jiang J, Wang D, et al (2021) Multi-PLI: interpretable multi-task deep learning model for unifying protein–ligand interaction datasets. J Cheminformatics 13(1):1–14. 10.1186/s13321-021-00510-6
  * Huang et al (2020) Huang K, Fu T, Glass LM, et al (2020) DeepPurpose: a deep learning library for drug–target interaction prediction. Bioinformatics 36(22-23):5545–5547. 10.1093/bioinformatics/btaa1005
  * Huang et al (2021) Huang K, Fu T, Gao W, et al (2021) Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development Preprint at http://arxiv.org/abs/2102.09548
  * Hughes et al (2011) Hughes J, Rees S, Kalindjian S, et al (2011) Principles of early drug discovery. Br J Pharmacol 162(6):1239–1249. 10.1111/j.1476-5381.2010.01127.x
  * Jastrzębski et al (2020) Jastrzębski S, Szymczak M, Pocha A, et al (2020) Emulating Docking Results Using a Deep Neural Network: A New Perspective for Virtual Screening. J Chem Inf Model 60(9):4246–4262. 10.1021/acs.jcim.9b01202
  * Kadurin et al (2016) Kadurin A, Aliper A, Kazennov A, et al (2016) The cornucopia of meaningful leads: Applying deep adversarial autoencoders for new molecule development in oncology. Oncotarget 8(7):10,883–10,890. 10.18632/oncotarget.14073
  * Kadurin et al (2017) Kadurin A, Nikolenko S, Khrabrov K, et al (2017) druGAN: An Advanced Generative Adversarial Autoencoder Model for de Novo Generation of New Molecules with Desired Molecular Properties in Silico. Mol Pharm 14(9):3098–3104. 10.1021/acs.molpharmaceut.7b00346
  * Landrum (2016) Landrum G (2016) RDKit: Open-Source Cheminformatics Software. URL https://www.rdkit.org/
  * Lemaître et al (2017) Lemaître G, Nogueira F, Aridas CK (2017) Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning. J Mach Learn Res 18(1):559–563. 10.5555/3122009.3122026
  * Lipinski et al (2001) Lipinski CA, Lombardo F, Dominy BW, et al (2001) Experimental and computational approaches to estimate solubility and permeability in drug discovery and development settings. Adv Drug Deliv Rev 46(1):3–26. 10.1016/S0169-409X(00)00129-0
  * Liu et al (2007) Liu T, Lin Y, Wen X, et al (2007) BindingDB: a web-accessible database of experimentally determined protein–ligand binding affinities. Nucleic Acids Res 35(suppl_1):D198–D201. 10.1093/nar/gkl999
  * Maziarka et al (2020) Maziarka Ł, Pocha A, Kaczmarczyk J, et al (2020) Mol-CycleGAN: a generative model for molecular optimization. J Cheminformatics 12(1):1–18. 10.1186/s13321-019-0404-1
  * Merk et al (2018a) Merk D, Friedrich L, Grisoni F, et al (2018a) De Novo Design of Bioactive Small Molecules by Artificial Intelligence. Mol Inform 37(1-2):1700,153. 10.1002/minf.201700153
  * Merk et al (2018b) Merk D, Grisoni F, Friedrich L, et al (2018b) Tuning artificial intelligence on the de novo design of natural-product-inspired retinoid X receptor modulators. Commun Chem 1(1):68. 10.1038/s42004-018-0068-1
  * Molnár (2014) Molnár F (2014) Structural considerations of vitamin D signaling. Front Physiol 5:191. 10.3389/fphys.2014.00191
  * Moret et al (2020) Moret M, Friedrich L, Grisoni F, et al (2020) Generative molecular design in low data regimes. Nat Mach Intell 2(3):171–180. 10.1038/s42256-020-0160-y
  * Morris et al (2009) Morris GM, Huey R, Lindstrom W, et al (2009) AutoDock4 and AutoDockTools4: Automated docking with selective receptor flexibility. J Comput Chem 30(16):2785–2791. 10.1002/jcc.21256
  * Möhler et al (2002) Möhler H, Fritschy JM, Rudolph U (2002) A New Benzodiazepine Pharmacology. J Pharmacol Exp Ther 300(1):2–8. 10.1124/jpet.300.1.2
  * Nascimento et al (2016) Nascimento ACA, Prudêncio RBC, Costa IG (2016) A multiple kernel learning algorithm for drug-target interaction prediction. BMC Bioinformatics 17(1):1–16. 10.1186/s12859-016-0890-3
  * Nguyen et al (2021) Nguyen T, Le H, Quinn TP, et al (2021) GraphDTA: predicting drug–target binding affinity with graph neural networks. Bioinformatics 37(8):1140–1147. 10.1093/bioinformatics/btaa921
  * Norman (2006) Norman AW (2006) Vitamin D Receptor: New Assignments for an Already Busy Receptor. Endocrinology 147(12):5542–5548. 10.1210/en.2006-0946
  * Olivecrona et al (2017) Olivecrona M, Blaschke T, Engkvist O, et al (2017) Molecular de-novo design through deep reinforcement learning. J Cheminformatics 9(1):1–14. 10.1186/s13321-017-0235-x
  * Pedregosa et al (2011) Pedregosa F, Varoquaux G, Gramfort A, et al (2011) Scikit-learn: Machine Learning in Python. J Mach Learn Res 12(85):2825–2830. 10.5555/1953048.2078195
  * Polykovskiy et al (2020) Polykovskiy D, Zhebrak A, Sanchez-Lengeling B, et al (2020) Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models. Front Pharmacol 11:565,644. 10.3389/fphar.2020.565644
  * Preuer et al (2018) Preuer K, Renz P, Unterthiner T, et al (2018) Fréchet ChemNet Distance: A Metric for Generative Models for Molecules in Drug Discovery. J Chem Inf Model 58(9):1736–1741. 10.1021/acs.jcim.8b00234
  * Prykhodko et al (2019) Prykhodko O, Johansson SV, Kotsias PC, et al (2019) A de novo molecular generation method using latent vector based generative adversarial network. J Cheminformatics 11(1):1–13. 10.1186/s13321-019-0397-9
  * Reymond (2015) Reymond JL (2015) The Chemical Space Project. Acc Chem Res 48(3):722–730. 10.1021/ar500432k
  * Segler et al (2018) Segler MHS, Kogej T, Tyrchan C, et al (2018) Generating Focused Molecule Libraries for Drug Discovery with Recurrent Neural Networks. ACS Cent Sci 4(1):120–131. 10.1021/acscentsci.7b00512
  * Skalic et al (2019) Skalic M, Jiménez J, Sabbadin D, et al (2019) Shape-Based Generative Modeling for de Novo Drug Design. J Chem Inf Model 59(3):1205–1214. 10.1021/acs.jcim.8b00706
  * Tang et al (2014) Tang J, Szwajda A, Shakyawar S, et al (2014) Making Sense of Large-Scale Kinase Inhibitor Bioactivity Data Sets: A Comparative and Integrative Analysis. J Chem Inf Model 54(3):735–743. 10.1021/ci400709d
  * Tanimoto (1958) Tanimoto T (1958) An Elementary Mathematical Theory of Classification and Prediction. IBM
  * Wang et al (2021) Wang S, Shan P, Zhao Y, et al (2021) GanDTI: A multi-task neural network for drug-target interaction prediction. Comput Biol Chem 92:107,476. 10.1016/j.compbiolchem.2021.107476
  * Wang et al (2012) Wang Y, Zhu J, DeLuca HF (2012) Where is the vitamin D receptor? Arch Biochem Biophys 523(1):123–133. 10.1016/j.abb.2012.04.001
  * Weininger (1988) Weininger D (1988) SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules. J Chem Inf Comput Sci 28(1):31–36. 10.1021/ci00057a005
  * Zhavoronkov et al (2019) Zhavoronkov A, Ivanenkov YA, Aliper A, et al (2019) Deep learning enables rapid identification of potent DDR1 kinase inhibitors. Nat Biotechnol 37(9):1038–1040. 10.1038/s41587-019-0224-x
  * Zhumagambetov et al (2020) Zhumagambetov R, Kazbek D, Shakipov M, et al (2020) cheml. io: an online database of ml-generated molecules. RSC advances 10(73):45,189–45,198
  * Zhumagambetov et al (2021) Zhumagambetov R, Molnár F, Peshkov VA, et al (2021) Transmol: repurposing a language model for molecular generation. RSC Adv 11(42):25,921–25,932. 10.1039/D1RA03086H
  * Öztürk et al (2018) Öztürk H, Özgür A, Ozkirimli E (2018) DeepDTA: deep drug–target binding affinity prediction. Bioinformatics 34(17):i821–i829. 10.1093/bioinformatics/bty593