These authors contributed equally to this work.

These authors contributed equally to this work.

These authors contributed equally to this work.

[1]Bo Huang [1]Wenbiao Zhou

1]Beijing StoneWise Technology Co Ltd., Haidian Street #15, Beijing, 100080,
China

2]Innovation Center for Pathogen Research, Guangzhou Laboratory, Guangzhou,
510320, China

# Lingo3DMol: Generation of a Pocket-based 3D Molecule using a Language Model

Lvwei Wang  Zaiyun Lin  Yanhao Zhu  Rong Bai  Wei Feng  Huting Wang  Jielong
Zhou  Wei Peng  huangbo@stonewise.cn;orcid.org/0000-0003-3822-9110
zhouwenbiao@stonewise.cn;orcid.org/0000-0002-7168-3676 [  [

###### Abstract

Structure-based drug design is a fundamental aspect of pharmaceutical
research, but generating target-aware molecules using artificial intelligence
faces numerous challenges. Current generative models lack the ability to
capture important 3D spatial interactions and often produce undesirable
molecular structures. To address these issues, we introduce Lingo3DMol, a
novel approach that combines a Fragment-based SMILES (FSMILES) encoding with
both local and global coordinate predictions. Our method constrains the
generatable molecule space, preventing the formation of unusual structures and
enabling efficient traversal of drug-like spaces. Additionally, we train a
separate non-covalent interaction (NCI) model that provides essential binding
pattern information for the generation model. We propose a new evaluation
standard, incorporating the DUD-E dataset and additional metrics, to better
assess model performance. Lingo3DMol outperforms state-of-the-art methods in
various metrics, including drug-likeness, synthetic accessibility, binding
affinity, and is significantly faster in generating valid molecules. The model
is available to academic users as an online service at
https://sw3dmg.stonewise.cn.

###### keywords:

fragment-based SMILES, pocket-based 3D molecule generation, NCI

##  1 Introduction

Structure-based drug design, which involves crafting molecules that can
specifically bind to a desired target protein, is a fundamental and
challenging aspect of pharmaceutical research[2]. In recent years, there has
been a growing interest in using artificial intelligence (AI) for de novo
molecular generation to aid in drug discovery. However, this approach faces
several obstacles, including high computational demands, an expansive
synthetically feasible space, a large number of degrees of freedom, and the
scarcity of high-quality training data.

Earlier molecular generative models relied on either molecular string
representations [4, 24, 41, 52] or graph representations [28, 30, 21, 42].
However, both representations fail to account for 3D spatial interactions,
rendering them suboptimal for target-aware molecule generation. Recent
advancements in structural biology and geometric deep learning, coupled with
an increase in available structural data [11], have opened new doors for
machine learning algorithms to directly design drugs within 3D binding
complexes [44, 15]. Methods using 3D CNN[33] struggle to capture 3D inductive
bias and are not rotationally equivariant. For more recent networks such as
[31, 29, 36, 27], they have proposed to represent pocket and molecule as 3D
graphs and using graph neural network to better encode them. These models use
an auto-regressive generation process which linearizes the graph into a
sequence of decisions. The most recent method, Targetdiff [17], employs a
graph based diffusion model for non-auto-regressive molecule generation.

However these graph based methods shares a common drawback: the generated
molecule contains many problematic, non drug-like, not synthetic available
structures such as very large rings (ring of seven or more atoms), honeycomb-
like array of parallel, juxtaposed rings, molecule with excessive number of
rings or molecule with no rings present. One cause of the problem specific to
auto-regressive sampling methods is that they can hinder the accurate
prediction of molecular structures (both 2D and 3D) due to improper semantics.
For example, as mentioned by [17], the model can readily position the
$n\text{th}$ atom to create a benzene ring when the $(n-1)\text{th}$ carbon
atoms are already in the same plane. However, due to insufficient context
information, accurate placement of the initial atoms is problematic, resulting
in unrealistic fragments. Although Targetdiff [17] avoids this problem, but it
still generate a large portion of undesirable structure. The model’s weak
perception of molecular topology may be due to its failure to directly encode
or predict bonds. The adoption of a post-processing process adds further room
for error.

Beyond the aforementioned issues, the primary reason underlying the challenge
of generating drug-like molecules is the vast space of generatable molecules’
2D and 3D topology. The 2D space includes numerous structures that are non-
drug-like and not synthetically accessible. When examining 3D structures,
similar issues emerge. Current generation methods encompasses far more degrees
of freedom than are relevant in molecular conformation generation. Bond
lengths, angles, and small rings in the ligand are essentially rigid, and
ligand flexibility mainly resides in the dihedral angles at rotatable bonds
[6]. An effective method to constrain the model, reduce the dimensionality of
state space, and enable efficient traversal only within the drug-like molecule
space is imperative. The second limitation of graph representations is their
inferior ability to capture 2D topological patterns that is apparent in
sequence representations like SMILES. This issue manifests particularly in low
2D similarity to the reference active molecule. These problem combined lead to
poorer performance on all metrics including drug-like metrics such as QED[3]
and SA score[10], as detailed in the subsequent Evaluation sections 3.2.

Additionally, generating valid molecules using state-of-the-art methods can be
time-consuming. For instance, AR[31], Targetdiff[17], and Pocket2Mol[36] take
an average of 7785s, 3428s, and 2544s, respectively, to generate 100 valid
molecules. Generating molecules that meet multiple criteria simultaneously
requires generating a large number of molecules (on the order of hundreds of
thousands) for each target. Lastly the popular evaluation benchmark/training
set CrossDocked[11] is unreliable. This dataset primarily consists of docked,
rather than experimental complex structures, as well as unvalidated ligand-
pocket pairs that have been artificially generated; in addition, there are a
significant number of non-drug-like molecules present in the dataset.
Moreover, the current train/test splitting strategy only considers pocket
similarity, resulting in the problem of ligand data leakage. As a result,
evaluating models on this dataset leads to unreliable metrics and poorly
reflects real-world performance.

To address the problems mentioned above, we propose Lingo3DMol and a new
evaluation standard, which adopts the DUD-E[35] dataset as evaluation set and
adds additional metrics. To constrain the generatable molecule space to a more
meaningful drug-like space[37], we introduce a novel semi-fragmented encoding
method for molecules, called Fragment-based SMILES (FSMILES). FSMILES
restricts the model to generate only specific molecular fragments, which
effectively prevents the generation of some unusual structures, significantly
reduces the search space and improves drug-likeness. Although we adopt an
auto-regressive approach, the FSMILES directly express the ring size in all
tokens within the ring, providing the necessary contextual information.
Therefore, this resolves the aforementioned ”limited context” issue.
Furthermore, we incorporate both local spherical coordinates[15]and global
coordinates into the model. By directly predicting the bond length and bond
angle which is a rather trivial task for the model, enables the generation of
3D molecules with reasonable local structures and good global spatial
perception. Due to the efficient traversal of the chemical and geometric
space, we can use sampling method instead of relying on beam search. This
leads to a speed advantage of several tens of times faster. Moreover, it is
crucial to include non-covalent interactions (NCI) and ligand-protein binding
patterns in the model, as emphasized by Ding et al. [8]. To achieve this, we
trained a separate NCI/Anchor model that provides this information to the
generation model.

To train our model, we used the PDBbind[48] instead of CrossDocked due to its
high-quality data. However, the dataset is small, so we introduce a molecular
pretraining model similar to BART[26] and Chemformer[19] to improve the
generalization ability of the model. Finally, we evaluate Lingo3DMol on both
DUD-E and CrossDocked dataset and compare it with state-of-the-art (SOTA)
methods. Lingo3DMol outperformed existing methods on various metrics. We
believe that our proposed method and evaluation standard can significantly
advance the field of de novo molecular generation and facilitate drug
discovery.

Our main contributions can be summarized as follow:

  * •

Introducing a novel FSMILES encoding that constrains the generatable molecule
space, effectively preventing the generation of unusual and undesired
structures. FSMILES also provides the necessary semantics for auto-regressive
generation.

  * •

Incorporating both local and global coordinates in the method, enabling the
generation of 3D molecules with reasonable local structures and global spatial
perception.

  * •

Introducing a 3D molecule denoising pretraining method to overcome the problem
of limited data.

  * •

Training an independent NCI/Anchor model for identifying potential non-
covalent interaction binding sites and incorporating this information into the
generation model.

  * •

Proposing a new evaluation benchmark, including the DUDE benchmark and
additional metrics, to provide a more reliable and comprehensive assessment of
performance.

  * •

Demonstrating superior performance over state-of-the-art methods on various
metrics, including drug-likeness, synthetic accessibility, and binding
affinity, and performing several tens of times faster than previous state-of-
the-art methods.

Figure 1: Overall molecular growth process for Lingo3DMol

##  2 Method

Figure 2: The overall architecture of Lingo3DMol comprises three separate
models: the pre-training model, the fine-tuning model, and the NCI/Anchor
prediction model. These models share the same architecture with slightly
different inputs and outputs.

In this section, we present an overview of the Lingo3DMol architecture and its
key attributes. The methodology comprises two models: the Generation
Model(GM), which is the central component, and the NCI/Anchor prediction
model(NPM), an essential auxiliary module. These models share the same
Transformer-based architecture. The complete architecture is illustrated in
Fig.2. In the following text, unless specifically mentioned, it is generally
referring to the Generation Model.

###  2.1 Definations and Notations

The Lingo3DMol learns $M\sim P(M|Pocket;\mu)$, where $\mu$ is the parameter of
the model, $Pocket=(p_{1},p_{2}...p_{n})$ is the set of atoms in the pocket,
and
$p_{i}=(type_{i},main/side_{i},residue_{i},coords_{i},hbd/hba_{i},NCI/Anchor_{i})$
indicates the information of the $i$th atom in the pocket, where $type$
denotes the element type of the atom, $main/side$ denotes an atom on the
main/side chain, $residue$ denotes the residue type of the atom, $coords$ is
the coordinates of the atom, $hbd/hba$ denotes whether an element is a
hydrogen bond donor or acceptor, and $NCI/Anchor$ records whether it is a
possible NCI site or anchor point where a potential molecular atom exists
within a 4$\AA$ range. Further details are provided in the Section 2.3.3.
$M=(FSMILES,\\{(r_{i})\\}^{K}_{i=1})$ is the representation of the ligand, and
$r_{i}$ is the coordinates of the $i$th atom of the ligand, K is the number of
atoms in the ligand.

FSMILES. Unlike SMILES and SELFIES[23], which cover almost all the chemical
universe of small molecule, we suggests to only allow a limited but prefered
chemical space. Hence we developed the Fragment-based SMILES(FSMILES),
reorganizing SMILES[51] with fragments as units. Within a fragment, the normal
SMILES syntax is used, and the whole molecule is composed of fragments using a
specific syntax. Under this representation, the whole process generates a
complete molecule token by token, fragment by fragment, in a depth-first
manner, as illustrated in Fig.1. We contend that this approach offers two key
benefits: (A) Limits the generation space to a more desired space, preventing
the generation of problematic structure and (B) Provides the necessary
semantics for auto-regressive generation and enables a modified depth-first
traversal algorithm that prioritizes the closure of rings, as demonstrated by
[27], resulting in a significant improvement in generation quality.

The molecule fragment cutting process involves selecting individual bonds that
meet the following criteria: (1) the single bond is not part of a ring, (2)
the single bond does not connect hydrogen atoms, and (3) at least one end of
the single bond is attached to a ring. Fig.3 illustrates the two-step FSMILES
construction process. First, the ligand is divided into fragments according to
the cutting rule. Second, ring information is embedded in each FSMILES token,
with the number following the element type’s underscore indicating the count
of atoms in the element’s ring.

The symbol ”$*$” denotes a fragment’s connection points, with the preceding
atom indicating the connection position. In our depth-first growth model, each
subsequent fragment connects to the preceding asterisk. To facilitate this, we
push every asterisk into a stack. Upon encountering a new fragment, we pop the
topmost asterisk from the stack to establish a connection with this new
fragment.

Figure 3: FSMILES construction example; the same color corresponds to the same
fragment.

###  2.2 Effective pre-training and fine-tuning method

Here, we introduce the specific pre-training and fine-tuning scheme of
Lingo3DMol.

####  2.2.1 Pre-training strategy

Figure 4: Illustration of our pre-training perturbation strategy. (a) Original
molecular state. (b) Removal of edge information during pre-training. (c)
Perturbation of the molecular structure by randomly deleting 25% of the atoms.
(d) Perturbation of the coordinates using a uniform distribution within the
range $[-0.5\text{\AA},0.5\text{\AA}]$. (e) Perturbation of 25% of the carbon
element types. These perturbations are applied in no particular order, and the
pre-training task aims to restore the molecular structure from (e) to (a).

We used a large dataset of small molecules to develop a pre-training method
compatible with subsequent fine-tuning tasks. To implement our pre-training
strategy, we perturbed the 3D molecule and fed it into an encoder. We then
attempted to reconstruct the perturbed molecule in its original state in both
2D and 3D. By employing pre-training in this manner, we optimized our
subsequent fine-tuning tasks, to achieve highly effective training with a
limited amount of annotated data.

In the pre-training phase, as illustrated in Fig.4, we introduced
perturbations into the 3D molecular structure and fed the perturbed molecule
into the encoder. This model, which employs an autoregressive approach, aims
to reconstruct the perturbed molecule back to its original state in both 2D
and 3D representations.

####  2.2.2 Fine-tuning

For the fine-tuning phase, we utilized the pre-trained model and further fine-
tuned it on the protein-small molecule complex data. The primary task during
this phase continued to be autoregressive molecule generation. To circumvent
overfitting of the finetuning dataset, the initial three encoder layers were
fixed during fine-tuning.

###  2.3 Model Architecture

The Generation Model and NCI/Anchor Prediction Model are built on one
Transformer-based model structure with additional graph structural information
encoded into the model similar to work [54] . The Generation Model(GM) is
trained by pre-training and fine-tuning. The NCI/Anchor Prediction Model(NPM)
is trained based on GM’s pre-trained parameters and fine-tuned additionally by
its specific prediction task. The overall architecture is shown in Fig.2. In
the following sections, we first discuss the Encoder and Decoder components of
the Generation Model, followed by an introduction to the NCI/Anchor Prediction
Model.

####  2.3.1 Encoder

Pre-training Stage Encoder Input. In the pre-training stage, the input of the
encoder is a perturbed 3D molecule, which includes the element type and
Euclidean coordinates. We can define the molecule as
$M^{enc}=(m^{enc}_{1},m^{enc}_{2}...m^{enc}_{n})$,
$m_{i}=(type_{i},coords_{i})$, where $coords_{i}=(x_{i},y_{i},z_{i})$, and $n$
is the number of atoms. Input feature $f_{pre}$ can then be defined as
follows:

$f_{coords,i}=MLP([E_{coords}(x_{i}),E_{coords}(y_{i}),E_{coords}(z_{i})])$
(1)

$f_{pre,i}=E_{type}(type_{i})+f_{coords,i}$ (2)

where

$E_{type}(type_{i})\in\mathbb{R}^{H},f_{coords,i}\in\mathbb{R}^{H}$
$E_{coords}(x_{i})\in\mathbb{R}^{H},\ E_{coords}(y_{i})\in\mathbb{R}^{H},\
E_{coords}(z_{i})\in\mathbb{R}^{H}$

where, $E_{type}$ and $E_{coords}$ represent the embedding functions for the
element type, and the coordinates, respectively. $H$ is the number of
dimensions after embedding.

Fine-tuning Stage Encoder Input. In the fine-tuning stage, the input is
changed from perturbed molecules to pockets. The input feature $E_{fine}$ can
be defined as follows:

$\begin{split}f_{\text{fine},i}=&E_{\text{type}}(\text{type}_{i})+E_{\text{main/side}}(\text{main/side}_{i})+E_{\text{residue}}(\text{residue}_{i})\\\
&+f_{\text{coords},i}+E_{\text{hbd/hba}}(\text{hbd/hba}_{i})+E_{\text{NCI/Anchor}}(\text{NCI/Anchor}_{i})\end{split}$
(3)

where

$\begin{split}&E_{main/side}(main/side_{i})\in\mathbb{R}^{H},E_{residue}(residue_{i})\in\mathbb{R}^{H},\\\
&E_{hbd/hba}(hbd/hba_{i})\in\mathbb{R}^{H},E_{NCI/Anchor}(NCI/Anchor_{i})\in\mathbb{R}^{H}\end{split}$
(4)

where, $E_{main/side}$, $E_{residue}$, $E_{hbd/hba}$, and $E_{NCI/Anchor}$
represent the embedding functions for the main/side chain, residue type,
hydrogen bond donor/acceptor, and NCI/Anchor point, respectively.

####  2.3.2 Decoder

The molecule generation process is implemented by two decoders: one 2D
topology decoder ($D_{2D}$) generates FSMILES tokens and local coordinates,
and the other decoder generates 3D global coordinates ($D_{3D}$). First, the
next token is predicted using $D_{2D}$. The latest 2D token is then input to
the 3D global coordinates decoder. A 3D global coordinates decoder predicts
the global coordinates of the molecule. Both decoders simultaneously predict
the local coordinates $r$, $\theta$, and $\phi$, particularly the radial
distance ($r$), bond angle ($\theta$), and dihedral angle ($\phi$) of the
molecule. The local coordinates prediction by $D_{3D}$ only serves as an
auxiliary training task.

Decoder Input Feature. The input to the decoder network for a molecule can be
defined as $M^{dec}=(m^{dec}_{1},m^{dec}_{2}...m^{dec}_{n})$,
$m_{i}=(token_{i},global\\_coords_{i})$ and $M_{bias}=(D,J)$, where $D$ is a
distance matrix of size $n\times n$ and $J$ is an edge vector matrix of size
$n\times n$, where $n$ is the length of the sequence. $M^{dec}$ is transformed
into embeddings using the same process as in Equation 2. These embeddings are
then fed into the 2D topology decoder and global coordinates decoders,
respectively. For non-atom tokens, we assigned the same coordinates as those
of the most recently generated atom.

Attention Bias. Bias terms $B_{D}$ and $B_{J}$ are derived from the distance
and edge vector matrices, $D$ and $J$, respectively. Modified attention scores
were calculated by incorporating the following bias terms:

$A_{\text{biased}}=\text{softmax}\left(\frac{QK^{\top}}{\sqrt{d_{k}}}+B_{D}+B_{J}\right)V.$
(5)

Predicting FSMILES, Local Coordinates For FSMILES, we used an MLP projection
head to predict the next token based on $D_{2D}$’s output. To predict the
local coordinates, we established a local coordinates system with three atomic
reference points: root1 is current position’s parent atom , root1’s parent
atom is root2, and root2’s parent atom is root3. Parent atom is the atom that
connects to the child atom.

To predict the radial distance $r$, we utilized features of root1
($h_{\text{root1}}$) extracted from the $D_{2D}$’s hidden representation, the
current FSMILES token ($E_{type}(cur)$), and the molecule’s hidden
representation ($H_{\text{topo}}$) from $D_{2D}$,
$H_{topo}=(h_{1},h_{2}...h_{i})$, where each $h$ represent a FSMILES token’s
hidden representation, and $i$ represent the number of the generated tokens.
For the polar angle $\theta$, we concatenate the hidden representations
$h_{\text{root1}}$ and $h_{\text{root2}}$. To predict the dihedral angle
$\phi$, we concatenated the representations of all three roots.

We used MLPs as projection heads to predict the local coordinates
$(r,\theta,\phi)$. The mathematical representations of these processes are as
follows:

distance prediction $r$:

$r=\text{argmax}\left(\text{softmax}\left(\text{MLP}_{1}\left([E_{type}(cur),H_{\text{topo}},h_{\text{root1}}]\right)\right)\right)$
(6)

angle prediction $\theta$:

$\theta=\text{argmax}\left(\text{softmax}\left(\text{MLP}_{2}\left([E_{type}(cur),H_{\text{topo}},h_{\text{root1}},h_{\text{root2}}]\right)\right)\right)$
(7)

dihedral angle prediction $\phi$:

$\phi=\text{argmax}\left(\text{softmax}\left(\text{MLP}_{3}\left([E_{type}(cur),H_{\text{topo}},h_{\text{root1}},h_{\text{root2}},h_{\text{root3}}]\right)\right)\right)$
(8)

The predicted local coordinates $(r,\theta,\phi)$ are obtained by taking the
argmax of the respective softmax outputs.

Predicting Global Coordinates. As illustrated in Fig.2, $D_{3D}$ receives the
hidden representaion of $D_{2D}$, and concatenate the predicted FSMILES token,
and then predicts the global coordinate $(x,y,z)$.

Pre-training/Fine-tuning Loss. In our model, the loss function is a
combination of multiple components that evaluates different aspects of the
predicted molecule. The overall loss function is defined as follows:

$L=L_{\text{FSMILES}}+L_{\text{abs\\_coord}}+L_{r}+L_{\theta}+L_{\phi}+L_{r\\_aux}+L_{\theta\\_aux}+L_{\phi\\_aux}$
(9)

where:

  * •

$L_{\text{FSMILES}}$ measures the discrepancy between the predicted and
ground-truth molecular topology.

  * •

$L_{\text{abs\\_coord}}$ evaluates the difference between the predicted and
ground-truth atomic coordinates.

  * •

$L_{r}$ and $L_{r\\_aux}$ measure the error between the predicted and ground-
truth radial distances.

  * •

$L_{\theta}$ and $L_{\theta\\_aux}$ assess the discrepancy between the
predicted and ground-truth bond angles.

  * •

$L_{\phi}$ and $L_{\phi\\_aux}$ evaluate the difference between the predicted
and ground-truth dihedral angles.

All the prediction tasks are treated as classification tasks. Therefore, the
cross entropy loss is used for each individual loss component. Auxiliary
prediction tasks $r\\_aux,\theta\\_aux$, and $\phi\\_aux$ are used only during
training. They are not utilized during the actual inference process. For
further details, please refer to the section 2.4

####  2.3.3 NCI/Anchor Prediction Model

In this work, we aim to enhance the Generation Model by integrating Non-
Covalent Interaction (NCI) and anchor point information during fine-tuning and
inference stage. To achieve this, we employ an NCI/Anchor Prediction Model,
which mirrors the Generation Model’s architecture. This Prediction Model is
initialized using the Generation Model’s pre-trained parameters. Equiped with
a specialized output head, the Encoder can predict whether a pocket atom will
form a NCI with the ligand or act as an anchor point.

This approach allows us to enhance the Generation Model in two significant
ways. Firstly, we enrich the model’s input by incorporating the predicted NCI
and anchor point data as distinct features of the pocket atoms; Sencondly, we
can start the molecule generation process by predicting the first atom of the
molecule near a chosen NCI site. Specifically, we sample an NCI site within a
pocket and generate the first small molecule atom within a $4.5\AA$ radius of
this site.

There are three main implications of our approach:

1\. We can enhance the perception of NCI and pocket shape which is critical
for generating 3D molecules that can effectively interact with the target
protein.

2\. By positioning the first atom near the atoms within the NCI pocket, we can
increase the likelihood of obtaining a correct NCI pair with a high degree of
certainty. However, that although our model is designed to generate molecules
that are prone to forming NCI pairs, it cannot ensure the exact positioning of
the generated molecule in relation to the NCI pocket atoms. This is because
the model does not explicitly enforce the coverage of all predicted NCI pocket
positions, thereby allowing for some degree of variability in the positioning
of the generated molecule.

3\. Obtaining a good starting position: the auto-regressive generation can be
seen as a sequential decision-making problem. If the initial step is not
chosen appropriately, it can affect every subsequent step. Our empirical study
suggests that the NCI position as a starting point is a better choice than a
random sample from the predicted 3D coordinates distribution or by selecting
the coordinates with highest possibility.

NCI/Anchor Prediction Loss. We define two loss functions: The NCI loss
measures the difference between the predicted NCI sites and the ground truth
NCI sites, and the anchor loss measures the difference between the predicted
anchor points and the ground truth anchor points. Both loss functions use
binary cross entropy.

The total loss for the model is the sum of the NCI loss, anchor loss, and all
other auxiliary losses from the original 3D generation task.

$L_{\text{total}}=L_{\text{nci}}+L_{\text{anchor}}+L_{\text{gen}},$

where $L_{\text{nci}}$ and $L_{\text{anchor}}$ represent the NCI and anchor
losses, respectively, and $L_{\text{gen}}$ corresponds to the losses from the
original 3D generation task which served as only an auxiliary training
objective.

###  2.4 Generation Process

Figure 5: Intuitive visualization of the 3D molecule generation process. In
Step 1, based on the precomputed pocket NCI information, we select an NCI as
the starting position. Within a radius $r$, we select the position with the
highest global coordinate probability. In each subsequent step, we predict the
local coordinates $r$, $\theta$, and $\phi$, and combine them with the global
coordinates to determine the final position.

Here, we present the process of generating the final 3D molecule, as shown in
Fig.5. First, the NCI/Anchor Prediction Model predicts the NCI sites and
anchor atoms. We then sample one NCI site from these predictions, and generate
the first ligand atom. This atom is positioned at the global coordinates
$(x,y,z)$ with the highest predicted joint probability, provided it lies
within a $4.5\AA$ radius of the sampled NCI site. The subsequent atomic
positions were generated iteratively. For each step $i$, $D_{2D}$ predicts the
$i+1$’s FSMILES token and local coordinates $(r,\theta,\phi)$ . Based on
$i+1$’s FSMILES token, we identified the indices of the root1, root2, and
root3 atoms. The 3D global coordinates decoder $D_{3D}$ was employed to
predict the probability distribution of the $(i+1)^{\text{th}}$ 3D coordinates
$(x,y,z)$ by incorporating information from the $(i+1)^{\text{th}}$ FSMILES
token and the local coordinates.

Leveraging Local and Global Coordinates. Within a molecule, bond lengths and
angles are largely fixed, and FSMILES fragments are consistently rigid and
replicable. As a result, the prediction of local spatial positions will get
easier by using local coordinates, which include bond length, bond angle, and
dihedral angle. In contrast, the global coordinates offers a robust global 3D
perception, which is essential for assessing the overall structural context.

We propose a fusion method that combines the local and global coordinates.
Particularly, this method defines a flexible search space around the predicted
local coordinates, then selects the global coordinates with the highest
probability.

The search space is defined as follows:

$r\pm 0.1$ Å (angstroms) for distance, $\theta\pm 2^{\circ}$ (degrees) for the
angle, and $\phi\pm 10^{\circ}$ (degrees) for the dihedral angle.

Within this search space, we determine the position with the highest joint
probability in the predicted global coordinate distributions. The generation
process was repeated to extend the molecular structure progressively. The
pseudocode is shown as Algorithm 1

Algorithm 1 Generation of the Final 3D Molecular Structure

1:Initialize first atom position:

2: Select random NCI site in molecular pocket

3: Choose the highest predicted joint probability for initial $(x,y,z)$ within
radius $r$

4:for each generation step $i$ do

5: Predicts the $i+1$’s FSMILES token

6: Identify the indices of the $(i+1)^{\text{th}}$’s root1, root2, and root3
atoms based on FSMILES codes

7: Predicts local coordinates $(r,\theta,\phi)$ based on roots features.

8: $D_{3D}$ predicts the $(i+1)^{\text{th}}$ 3D coordinates $(x,y,z)$ using
the FSMILES token

9: Define search space around predicted local coordinates for final atom
coordinates:

10: Distance: $r\pm 0.1$ Å

11: Angle: $\theta\pm 2^{\circ}$

12: Dihedral Angle: $\phi\pm 10^{\circ}$

13: Find global coordinate with highest joint probability within search space

14: Set final predicted location for the current generation step

15:end for

##  3 Results

###  3.1 Data

The pre-training dataset was derived from 20 million commercially available
compounds, with a low-energy state conformer generated for each molecule using
ligprep [40]. To ensure the generation of drug-like molecules, we filtered the
dataset, excluding complex ring structures such as large, screw, and bridge
rings, and only retaining molecules with less than three consecutive flexible
bonds. This filtering process resulted in 12 million molecules.

The fine-tuning dataset was sourced from PDBbind[48]. We cropped pockets by
including complete protein residues within a 6Å radius of each ligand atom.
Two distinct test sets were created for fine-tuning: a CrossDocked test
set[11] and DUD-E test set [35]. To avoid information leakage, we filtered out
training set pockets with a homology[34] over 30% to the test set, as
determined by MMseqs2[34]. This resulted in final training sample sizes of
11.8k for the DUD-E test set and 6.5k for the CrossDocked test set.

The NCI training set samples were aligned with the fine-tuneing samples,
yielding training sample sizes of 11.8k for the DUD-E test set and 6.5k for
the CrossDocked test set. The NCI of hydrogen bond, halogen bond, salt Bridge
and pi-pi stacking in the PDBbind were labeled using Open Drug Discovery
Toolkit(ODDT)[32]. The anchors were marked as the atoms in the pocket that are
less than $4\AA$ away from any atom in the ligand. These labeled samples were
used for the NCI/Anchor Prediction Model, averaging $4.1$ NCI atoms and $32.1$
anchor atoms per pocket sample.

###  3.2 Evaluation

Benchmarks To evaluate our results, we used two benchmark datasets. First we
used the DUD-E[35] dataset, which includes 102 targets and an average of 224
active ligands per target. This dataset is more drug-like than CrossDocked,
and spans diverse protein categories like Kinase, Protease, GPCRs, and ion
channels. Hence, it allows us to determine if our model can reproduce a
positive molecule across various protein targets. Additionally, we generated a
control group for each target by randomly selecting 1000 molecules from a
commercial library of 12 million molecules. The results are summarized in
Table 1. To further evaluate our model’s performance, we utilized the widely
adopted benchmark which contains 100 targets split from the CrossDocked
dataset[11].

Baseline In the DUD-E evaluation, we compared Lingo3DMol with previous SOTA
methods: Pocket2Mol[36], and a diffusion based model Targetdiff[17]. In the
CrossDocked evaluation, we further compared it with liGAN[33], AR[31],
GraphBP[29]. AR, GraphBP and Pocket2Mol are auto-regressive generative GNN
models. liGAN is a 3D CNN based VAE model.

####  3.2.1 Evaluation with the DUD-E dataset

|  Random Test |  Pocket2 Mol |  Targetdiff |  Lingo3DMol (ours)
---|---|---|---|---
ECFP$>$0.5 ($\uparrow$) | 0.168 | 0.109 | 0.050 | 0.327
Docking Score ($\downarrow$) | -6.42 | -7.195 | -6.82 | -7.403
QED($\uparrow$) | 0.691 | 0.486 | 0.512 | 0.615
SA Score($\downarrow$) | 2.555 | 3.858 | 4.88 | 2.885
Diversity ($\uparrow$) | 0.852 | 0.831 | 0.8775 | 0.824
RMSD vs. low-energy conformer ($\downarrow$) | 3.977 | 1.192 | 1.357 | 0.861
RMSD vs. docking pose ($\downarrow$) | - | 5.488 | 5.165 | 4.697
RMSD vs. docking pose $<$ 2Å ($\uparrow$) | - | 0.137 | 0.162 | 0.165
Table 1:  Comparison on the DUD-E dataset. In this experiment, we generated
1000 molecules per target for each methods. The docking scores and docked
conformers are calculated using Glide[13, 12]

For the ECFP$>$0.5 metrics, a target is considered successful if at least one
molecule pair has a Tanimoto similarity[46] above $0.5$. Lingo3DMol notably
outperformed all methods demonstrating its ability to generate higher-quality
molecules resembling positive ligands. The value 0.327 suggests our method can
reproduce positive ligands in about 33% of protein pockets.

Docking score is another crucial metric for molecular generation. Table 1
shows that the molecules generated by Lingo3DMol have the best docking scores,
indicating a better binding relationship between the generated molecules and
the pockets. Importantly, Pock2Mol and Targetdiff was trained on the
CrossDocked dataset, exhibiting high homology with DUD-E targets, potentially
causing information leakage. Our model was trained on a low-homology ($<$0.3)
dataset, mitigating this issue. We assessed performance by categorizing DUD-E
targets based on homology with Pocket2Mol training targets into severe
($>$0.9), moderate (0.3–0.9), and limited ($<$0.3) information leakage groups.
As shown in Fig.6(a), Pock2Mol matches our model’s performance only in the
severe leakage group. Our method excels with limited leakage.

Figure 6: (a) Comparison of the docking score.(b) Comparison of the percentage
of RMSD vs docking pose $<$ 2Å in different stages based on homology
similarity.

QED[3] is an abbreviation for the quantitative estimate of drug likeness. SA
Score[10] is an abbreviation for synthetic accessibility score. Lingo3DMol
demonstrated superior QED and SA Score compared to other methods. Targetdiff
and Pocket2Mol exhibit worse QED, primairly because of the large portion of
molecules with problematic structure.

RMSD[5] is a crucial metric about conformation quality and binding affinities
in drug discovery as the docking score has relative weak correlation with the
experimental binding affinities [49]. We devised several RMSD-based evaluation
metrics, including RMSD vs. low-energy conformer—calculated by generating 20
low-energy conformers in vacuum from SMILES using ConfGen[50] and determining
the minimum RMSD between generated and low-energy conformers. Lingo3DMol can
generate lower-energy conformers while satisfying the requirements for binding
to a specific pocket. RMSD vs. docking pose is for calculating the RMSD
between the originally generated 3D molecule and the docked one using Glide.
We also introduced RMSD vs. docking pose $<$ 2Å, assessing the proportion of
generated molecules with RMSD below 2Å, with higher percentages indicating
more reasonable poses. The result demonstrates Lingo3DMol’s ability to
generate lower-energy conformers suitable for specific pocket binding.
Moreover, Fig.6(b) provides detailed proof that as the degree of information
leakage decreases, the metric of Lingo3DMol is higher than that of Pocket2Mol.
The detail results for the different stages of the homology are presented in
Section A.4

####  3.2.2 Evaluation with the CrossDocked test set

|  Test Set | liGAN | AR | GraphBP |  Pocket2 Mol |  Targetdiff |  Lingo3DMol (ours)
---|---|---|---|---|---|---|---
QVina Score (kcal/mol, $\downarrow$) | -7.16 | -6.14 | -6.22 | -4.80 | -7.29 | - | -7.51
Vina Score (kcal/mol, $\downarrow$) | -7.45 | -6.75 | - | - | -7.15 | -7.80 |  -7.3
High Affinity ($\uparrow$) | - | 0.238 | 0.267 | 0.142 | 0.542 | 0.513 | 0.585
QED ($\uparrow$) | 0.484 | 0.369 | 0.502 | 0.43 | 0.563 | 0.48 | 0.623
SA ($\uparrow$) | 0.732 | 0.590 | 0.675 | 0.49 | 0.765 | 0.58 | 0.776
LogP | 0.947 | -0.140 | 0.257 | - | 1.586 |  | 2.160
Diversity ($\uparrow$) | - | 0.654 | 0.742 | 0.79 | 0.688 | 0.72 | 0.822
Time (s,$\downarrow$) | - | - | 19569 | 105 | 2504 | 3428 | 114
Table 2: Comparison of several baselines on the CrossDocked test set. We used
both QuickVina 2[1](QVina) and AutoDock Vina[9] for docking. We generated 100
molecules for each target, as with the other methods. The high affinity
metrics were calculated using QVina’s docking score. The metric data for other
methods are copied from corresponding original papers or Targetdiff and
Pocket2Mol’s paper. Targetdiff provides the data for the high affinity metric
of the QVina version but it does not provide the QVina docking score

We further evaluate our method using widely adopted benchmarks[38, 17, 36].
The results are summarized in Table 2.

Metrics include:

  * •

Vina Score, assessing binding affinity between a ligand and a receptor
calculated by AutoDock Vina.

  * •

QVina Score similar to Vina Score, calculated by QVina, a faster but less
accurate version of AutoDock Vina.

  * •

High Affinity, indicating the proportion of generated ligands which have
higher QVina scores than actual ligands in the test set.

  * •

SA, measuring synthetic accessibility (distinct from SA in Section 3.2.1) with
values between 0 and 1.

  * •

LogP, the logarithm of a molecule’s partition coefficient (P) between aqueous
and lipophilic phases. Values should be between -0.4 and 5.6 to be good drug
candidates[16].

  * •

Diversity, representing the average Tanimoto similarities[46] of molecules
belonging to the same target, calculated using Moses[38].

  * •

Time, denoting the generation time for 100 molecules.

Lingo3DMol achieves best or competitive scores on Vina and Qvina Score
indicates that Lingo3DMol effectively captures binding affinity between
receptors and ligands, generating higher-quality molecules.

The difference in the QVina and Vina metrics is hard to interpret as there is
a relatively weak correlation between the docking scores and experimental
binding affinities, which indicates that current scoring functions are still
not reliable and universal enough. The pose generated by the Autodock Vina has
only a 49.0% (Glide 57.8%) success rates to match the native pose[49].
Moreover, the complications arising from cross-docking pocket quality -
particularly the presence of numerous non-drug target pockets - serve to
further undermine the dependability of this metric. We recommend to focus on
the DUD-E benchmark’s result for more reliable interpretation and we present
ten cases, each comparing three methods with three random molecules per method
for more intuitive assessment in A.5. We obtained the best scores for QED and
SA. Lingo3DMol achieved the best diversity score, suggesting its ability to
generate various molecules targeting specific pockets. Remarkably, Lingo3DMol
required only 114 seconds to generate 100 molecules on an NVIDIA V100 GPU,
making it dozen times faster than other methods. (Targetdiff’s speed is
calculated using NVIDIA GeForce GTX 3090 GPU111https://www.nvidia.com/en-
gb/geforce/graphics-cards/30-series/rtx-3090-3090ti/ , which is a faster GPU
than V100222https://www.nvidia.com/en-gb/data-center/v100/)

###  3.3 Case Analysis

For case study, we selected the generated molecules from two perspectives:
ECFP fingerprint similarity with known active compounds and docking score.
Particularly, a high similarity of ECFP fingerprints with known active
compounds indicated the replication of positive molecules, and a high docking
score indicated a stronger fit with the pocket.

As shown in Figs.7(a) and 7(b), the molecules with ”high similarity, high
docking score” groups were similar to positive molecules in terms of structure
and binding mode, demonstrating the ability of our model to reproduce active
compounds. However, this is not enough for this study, because drug design
researchers in the real world are more interested in retrieving the following
two types of molecules: 1) active compounds missed by the docking program or
other virtual screen tools and 2) molecules with novel scaffolds and good
binding affinity to the pocket. The first issue, “active compounds missed by
the docking program”, is often caused by insufficient sampling of the binding
pose, which is closely related to the docking score. Our 3D molecule
generation model provides a potential solution to such issue. As shown in
Figs.7(c) and 7(d), the generated molecules in the ”high similarity, low
docking score” groups are highly similar to the positive molecules but have
low docking scores (i.e., -6.8 and -6.4, respectively) when docking program
was used for binding pose sampling. On the contrary, when the binding pose
generated by our model was evaluated by Glide Scorer without conformation
sampling, we obtained good scores of -10.2 and -8.8, respectively for the two
compounds. This case demonstrates the effectiveness of our generated 3D
conformation for retrieving good molecules with poor docking scores. For the
second issue, “obtaining molecules with a novel scaffold and good binding
mode,” we present two cases in Figs.7(e) and 7(f) to demonstrate that our
model can generate molecules with these characteristics.

Figure 7:  Comparison of positive ligand conformation, conformation generated
by Lingo3DMol, docking conformation of Lingo3DMol SMILES by glide, and 2D
conformation of most similar positive ligand with Lingo3DMol molecule.

###  3.4 Ablation Analysis

####  3.4.1 Effective pre-training and fine-tuneing analysis

Pre-training plays an important role in model training; specifically, the
molecules generated by the model with pre-training on the DUD-E dataset and
those generated by the model without pre-training on the same dataset were
compared with the molecules in the pre-training set, respectively. We
demonstrated that the molecules generated by the pre-training model were more
similar to the molecules in the pre-training set than to those generated by
the model without pre-training, indicating that the model retained the effect
of pre-training after the fine-tuning. The comparison of these methods is
described in Appendix A.3. As shown in Table 3, pre-training significantly
improves the docking score and enhances the Dice Similarity Coefficient(a
statistical measure used to compare the similarity between two sets) which
measures the degree of overlay of the generated molecule’s position and the
actual ligand’s position in 3D space.

Table 3: Comparison of the results of each group on the DUD-E test set in the ablation experiment |  Lingo3DMol (standard) |  Lingo3DMol (without pre-training) |  Lingo3DMol (without NCI)
---|---|---|---
Docking Score ($\downarrow$) | -7.403 | -6.77 | -7.096
QED($\uparrow$) | 0.615 | 0.619 | 0.611
SAS($\downarrow$) | 2.885 | 2.81 | 2.952
molWeight | 337 | 336 | 335
Dice ($\uparrow$) | 0.222 | 0.069 | 0.187
RMSD vs. low-energy conformer ($\downarrow$) | 0.861 | 0.831 | 0.806
Table 4: Precision/Recall effect of the NCI model on the DUD-E test set |  Precision | Recall
---|---|---
NCI atoms on pocket | 0.629 | 0.622
Anchor atoms on pocket | 0.739 | 0.756

####  3.4.2 NCI/Anchor Prediction Model analysis

We trained the NCI/Anchor Prediction Model on our labeled PDBbind[48] dataset
and evaluated it on DUD-E test set, the results is in Table 4. We found that
adding NCI restraints significantly improved the docking results, but at the
same time caused a certain degree of RMSD deterioration, as depicted in Table
3. This problem is difficult to avoid completely in multi-restraint
optimization, but fortunately the overall benefits obtained with a small
decrease in RMSD are more significant.

##  4 Discussion

Generating pocket-based 3D molecules is a complex task. Our study shows that
Lingo3DMol effectively produces molecules with high binding potential and
novelty due to several factors:

  1. 1.

We used a large commercial molecule library for pre-training, designed a
specific perturbation and recovery task, and fine-tuned the model to improve
performance.

  2. 2.

We employed an additional NCI/Anchor Prediction Model to extract key
interaction knowledge from PDBbind and incorporated this data into the input
pocket’s feature, boosting docking performance.

  3. 3.

Fragment-based SMILES allows the model to generate molecules incrementally,
resulting in more reasonable local topology and laying the groundwork for
future research through quantifiable evaluation and guidance in molecular
sampling.

Our evaluations reveal that our generated ligands capture strong non-covalent
interactions, achieve better docking scores, have lower strain-energy, and
possess drug-like properties similar to commercial molecules. As an integral
part of our virtual screening pipeline, this deep generative learning-based
method consistently produces novel, synthesizable molecules and holds promise
for future drug discovery efforts.

However, there are still challenges ahead. Due to the autoregressive
generation process, capturing all NCIs within a single molecule is not easy,
and we will investigate this issue further in the future. Moreover, our model
tends to generate similar structures in similar pocket environments.

Moreover, the equivariance property is a critical aspect of 3D molecule
generation[18, 39, 53]. There are many works on rotational and translational
equivariant models, such as GVP [22] and Vector Neurons [7]. Currently, we use
rotation and translation augmentation to enhance the model, and we employ
SE(3) invariant features like distance matrices and local coordinates [43] to
alleviate the problem. In future work, we plan to re-train the generation
model using a reward function to encourage it to sample more diverse
structures. Lastly, although we’ve assessed drug-like properties through case
analysis and cheminformatics tools such as QED[3] and SA Score[10] from
RDKit[25], a comprehensive and systematic evaluation of these properties is
still needed in the next step.

Author Contributions
W.Z. and B.H. conceived the study. W.Z. and J.Z. provided instructions for AI
modeling. B.H. and H.W. provided instructions on evaluation framework. L.W.,
Z.L., Y.Z., R.B., and W.F. developed the model. L.W. and R.B. prepared
evaluation data. W.P. supported molecular docking tests.
Acknowledgements
This study was funded by the National Key R&D Program of China (grant number
2022YFF1203004). This work was also supported by the Beijing Municipal Science
and Technology Commission (No. Z211100003521001).

## References

  * [1] Amr Alhossary, Stephanus Daniel Handoko, Yuguang Mu, and Chee-Keong Kwoh.  Fast, accurate, and reliable molecular docking with quickvina 2.  Bioinformatics, 31(13):2214–2216, 2015.
  * [2] Amy C Anderson.  The process of structure-based drug design.  Chemistry & biology, 10(9):787–797, 2003.
  * [3] G Richard Bickerton, Gaia V Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L Hopkins.  Quantifying the chemical beauty of drugs.  Nature chemistry, 4(2):90–98, 2012.
  * [4] Esben Jannik Bjerrum and Richard Threlfall.  Molecular generation with recurrent neural networks (rnns).  arXiv preprint arXiv:1705.04612, 2017.
  * [5] Fred E Cohen and Michael JE Sternberg.  On the prediction of protein structure: the significance of the root-mean-square deviation.  Journal of molecular biology, 138(2):321–333, 1980.
  * [6] Gabriele Corso, Hannes Stärk, Bowen Jing, Regina Barzilay, and Tommi Jaakkola.  Diffdock: Diffusion steps, twists, and turns for molecular docking.  arXiv preprint arXiv:2210.01776, 2022.
  * [7] Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J Guibas.  Vector neurons: A general framework for so (3)-equivariant networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12200–12209, 2021.
  * [8] Kang Ding, Shiqiu Yin, Zhongwei Li, Shiju Jiang, Yang Yang, Wenbiao Zhou, Yingsheng Zhang, and Bo Huang.  Observing noncovalent interactions in experimental electron density for macromolecular systems: a novel perspective for protein–ligand interaction research.  Journal of Chemical Information and Modeling, 62(7):1734–1743, 2022\.
  * [9] Jerome Eberhardt, Diogo Santos-Martins, Andreas F Tillack, and Stefano Forli.  Autodock vina 1.2. 0: New docking methods, expanded force field, and python bindings.  Journal of chemical information and modeling, 61(8):3891–3898, 2021\.
  * [10] Peter Ertl and Ansgar Schuffenhauer.  Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions.  Journal of cheminformatics, 1:1–11, 2009.
  * [11] Paul G Francoeur, Tomohide Masuda, Jocelyn Sunseri, Andrew Jia, Richard B Iovanisci, Ian Snyder, and David R Koes.  Three-dimensional convolutional neural networks and a cross-docked data set for structure-based drug design.  Journal of chemical information and modeling, 60(9):4200–4215, 2020\.
  * [12] Richard A Friesner, Jay L Banks, Robert B Murphy, Thomas A Halgren, Jasna J Klicic, Daniel T Mainz, Matthew P Repasky, Eric H Knoll, Mee Shelley, Jason K Perry, et al.  Glide: a new approach for rapid, accurate docking and scoring. 1. method and assessment of docking accuracy.  Journal of medicinal chemistry, 47(7):1739–1749, 2004.
  * [13] Richard A Friesner, Robert B Murphy, Matthew P Repasky, Leah L Frye, Jeremy R Greenwood, Thomas A Halgren, Paul C Sanschagrin, and Daniel T Mainz.  Extra precision glide: Docking and scoring incorporating a model of hydrophobic enclosure for protein- ligand complexes.  Journal of medicinal chemistry, 49(21):6177–6196, 2006.
  * [14] Vettigli G.  Minisom: minimalistic and numpy-based implementation of the self organizing map.  2018\.
  * [15] Niklas Gebauer, Michael Gastegger, and Kristof Schütt.  Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules.  Advances in neural information processing systems, 32, 2019.
  * [16] Arup K Ghose, Vellarkad N Viswanadhan, and John J Wendoloski.  A knowledge-based approach in designing combinatorial or medicinal chemistry libraries for drug discovery. 1. a qualitative and quantitative characterization of known drug databases.  Journal of combinatorial chemistry, 1(1):55–68, 1999.
  * [17] Jiaqi Guan, Wesley Wei Qian, Xingang Peng, Yufeng Su, Jian Peng, and Jianzhu Ma.  3d equivariant diffusion for target-aware molecule generation and affinity prediction.  arXiv preprint arXiv:2303.03543, 2023.
  * [18] Emiel Hoogeboom, Víctor Garcia Satorras, Clément Vignac, and Max Welling.  Equivariant diffusion for molecule generation in 3D.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 8867–8887. PMLR, 17–23 Jul 2022.
  * [19] Ross Irwin, Spyridon Dimitriadis, Jiazhen He, and Esben Jannik Bjerrum.  Chemformer: a pre-trained transformer for computational chemistry.  Machine Learning: Science and Technology, 3(1):015022, 2022.
  * [20] Dong J et al.  a python library for various molecular representations of chemicals, proteins and dnas and their interactions.  Journal of Cheminfomatics, 2018.
  * [21] Wengong Jin, Regina Barzilay, and Tommi Jaakkola.  Junction tree variational autoencoder for molecular graph generation.  In International conference on machine learning, pages 2323–2332. PMLR, 2018.
  * [22] Bowen Jing, Stephan Eismann, Pratham N Soni, and Ron O Dror.  Equivariant graph neural networks for 3d macromolecular structure.  arXiv preprint arXiv:2106.03843, 2021.
  * [23] Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alan Aspuru-Guzik.  Self-referencing embedded strings (selfies): A 100% robust molecular string representation.  Machine Learning: Science and Technology, 1(4):045024, 2020.
  * [24] Matt J Kusner, Brooks Paige, and José Miguel Hernández-Lobato.  Grammar variational autoencoder.  In International conference on machine learning, pages 1945–1954. PMLR, 2017.
  * [25] Greg Landrum et al.  Rdkit: Open-source cheminformatics software. 2016.  URL http://www. rdkit. org/, https://github. com/rdkit/rdkit, 149(150):650, 2016.
  * [26] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  arXiv preprint arXiv:1910.13461, 2019.
  * [27] Yibo Li, Jianfeng Pei, and Luhua Lai.  Structure-based de novo drug design using 3d deep generative models.  Chemical science, 12(41):13664–13675, 2021.
  * [28] Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia.  Learning deep generative models of graphs.  arXiv preprint arXiv:1803.03324, 2018.
  * [29] Meng Liu, Youzhi Luo, Kanji Uchino, Koji Maruhashi, and Shuiwang Ji.  Generating 3d molecules for target protein binding.  arXiv preprint arXiv:2204.09410, 2022.
  * [30] Qi Liu, Miltiadis Allamanis, Marc Brockschmidt, and Alexander Gaunt.  Constrained graph variational autoencoders for molecule design.  Advances in neural information processing systems, 31, 2018.
  * [31] Shitong Luo, Jiaqi Guan, Jianzhu Ma, and Jian Peng.  A 3d generative model for structure-based drug design.  Advances in Neural Information Processing Systems, 34:6229–6239, 2021.
  * [32] Pawel Siedlecki Maciej Wójcikowski, Piotr Zielenkiewicz.  Open drug discovery toolkit (oddt): a new open-source player in the drug discovery field.  Journal of cheminformatics, 2015.
  * [33] Tomohide Masuda, Matthew Ragoza, and David Ryan Koes.  Generating 3d molecular structures conditional on a receptor binding site with deep generative models.  arXiv preprint arXiv:2010.14442, 2020.
  * [34] Milot Mirdita, Martin Steinegger, F Breitwieser, Johannes Söding, and E Levy Karin.  Fast and sensitive taxonomic assignment to metagenomic contigs.  Bioinformatics, 37(18):3029–3031, 2021.
  * [35] Michael M Mysinger, Michael Carchia, John J Irwin, and Brian K Shoichet.  Directory of useful decoys, enhanced (dud-e): better ligands and decoys for better benchmarking.  Journal of medicinal chemistry, 55(14):6582–6594, 2012.
  * [36] Xingang Peng, Shitong Luo, Jiaqi Guan, Qi Xie, Jian Peng, and Jianzhu Ma.  Pocket2mol: Efficient molecular sampling based on 3d protein pockets.  In International Conference on Machine Learning, pages 17644–17655. PMLR, 2022.
  * [37] Pavel G Polishchuk, Timur I Madzhidov, and Alexandre Varnek.  Estimation of the size of drug-like chemical space based on gdb-17 data.  Journal of computer-aided molecular design, 27:675–679, 2013.
  * [38] Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, et al.  Molecular sets (moses): a benchmarking platform for molecular generation models.  Frontiers in pharmacology, 11:565644, 2020.
  * [39] Vıctor Garcia Satorras, Emiel Hoogeboom, and Max Welling.  E (n) equivariant graph neural networks.  In International conference on machine learning, pages 9323–9332. PMLR, 2021.
  * [40] New York Schrödinger, Inc.  Ligprep.
  * [41] Marwin HS Segler, Thierry Kogej, Christian Tyrchan, and Mark P Waller.  Generating focused molecule libraries for drug discovery with recurrent neural networks.  ACS central science, 4(1):120–131, 2018.
  * [42] Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang.  Graphaf: a flow-based autoregressive model for molecular graph generation.  arXiv preprint arXiv:2001.09382, 2020.
  * [43] Gregor NC Simm, Robert Pinsler, Gábor Csányi, and José Miguel Hernández-Lobato.  Symmetry-aware actor-critic for 3d molecular design.  arXiv preprint arXiv:2011.12747, 2020.
  * [44] Miha Skalic, Davide Sabbadin, Boris Sattarov, Simone Sciabola, and Gianni De Fabritiis.  From target to drug: generative modeling for the multimodal structure-based ligand design.  Molecular pharmaceutics, 16(10):4282–4291, 2019.
  * [45] G. Takacs, M. Sandor, Z. Szalai, R. Kiss, and G.T. Balogh.  Analysis of the uncharted, druglike property space by self-organizing maps.  Molecular Diversity, 2021.
  * [46] Taffee T Tanimoto.  Elementary mathematical theory of classification and prediction.  1958\.
  * [47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [48] Renxiao Wang, Xueliang Fang, Yipin Lu, and Shaomeng Wang.  The pdbbind database:collection of binding affinities for protein-ligand complexes with known three-dimensional structures.  Journal of Medicinal Chemistry, 47(12):2977–2980, 2004.
  * [49] Zhe Wang, Huiyong Sun, Xiaojun Yao, Dan Li, Lei Xu, Youyong Li, Sheng Tian, and Tingjun Hou.  Comprehensive evaluation of ten docking programs on a diverse set of protein–ligand complexes: the prediction accuracy of sampling power and scoring power.  Physical Chemistry Chemical Physics, 18(18):12964–12975, 2016.
  * [50] K Shawn Watts, Pranav Dalal, Robert B Murphy, Woody Sherman, Rich A Friesner, and John C Shelley.  Confgen: a conformational search method for efficient generation of bioactive conformers.  Journal of chemical information and modeling, 50(4):534–546, 2010\.
  * [51] David Weininger.  Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules.  Journal of chemical information and computer sciences, 28(1):31–36, 1988.
  * [52] Mingyuan Xu, Ting Ran, and Hongming Chen.  De novo molecule design through the molecular generative model conditioned by 3d information of protein binding sites.  Journal of Chemical Information and Modeling, 61(7):3240–3254, 2021\.
  * [53] Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang.  Geodiff: A geometric diffusion model for molecular conformation generation.  arXiv preprint arXiv:2203.02923, 2022.
  * [54] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu.  Do transformers really perform badly for graph representation?  Advances in Neural Information Processing Systems, 34:28877–28888, 2021.

##  Appendix A Supplementary

###  A.1 Encoder/Decoder backbone

The encoder and decoder use the standard transformer[47] architecture with
attention bias in decoder.

Encoder. The encoder has N identical layers, each with multi-head self-
attention and a position-wise fully connected feed-forward network. Residual
connections and layer normalization are employed, with all outputs having
dimension $d_{model}$ [47].

Decoder. The decoder also has N identical layers, with an extra multi-head
cross-attention sub-layer for the encoder’s output. Residual connections and
layer normalization are used, and a masked self-attention sub-layer ensures
position dependencies.[47]

Self-Attention. Let $Q$, $K$, and $V$ represent the query, key, and value
matrices, respectively. The original attention scores are calculated as:

$A=\text{softmax}\left(\frac{QK^{\top}}{\sqrt{d_{k}}}\right)V$ (10)

where $d_{k}$ is the dimension of the key vectors.

###  A.2 HyperParameters

We employ a consistent set of hyperparameters for all three stages of our
model, namely pre-training, finetuning, and NCI prediction.

Hyperparameter | Value
---|---
Batch size | 50
Learning rate ($\alpha$) | 1e-4
Hidden dimension | 512
Feed-forward network (FFN) dimension | 2048
Encoder layers | 6
Attention head size | 8
Topological decoder layers | 3
3D global decoder layers | 3
Table 5: Hyperparameters used for the 3D molecule generation model.

###  A.3 Validation method for pre-training effect

The method is to generate the molecule set A on DUD-E dataset by the model
with pre-training, and then generate the molecule set B on the same dataset by
the model without pre-training. Ensure that the number of molecules generated
on the same pocket remains the same for sets A and B. Then four properties
including carbon-scaled atomic mass, carbon-scaled atomic van der Waals
volume, carbon-scaled atomic Sanderson electronegativity, and carbon-scaled
atomic polarizability of Set A and B are calculated by PyBioMed[20], and the
above properties are mapped into two-dimensional planar (120*120) squares by
self-organizing-map (SOM)[45] which is implemented using MiniSom library[14]
for dimensionality reduction, each molecule will be mapped into 1 square, and
each square will be statistically superimposed on the number of samples that
are mapped in. We can prove that Set A is more similar with Pre-training set
than set B by calculating the total variation divergence and KL divergence, as
shown in Table 6.

$D_{TV}(P\|Q)=\frac{1}{2}\sum\limits_{i}\left|P_{i}-Q_{i}\right|$ (11)

$D_{KL}\left(P\|Q\right)=E_{x\sim
P}log\left[\frac{P\left(x\right)}{Q\left(x\right)}\right]$ (12)

Table 6: The Comparison of total variation divergence and KL divergence |  Total variation divergence | KL divergence
---|---|---
Model with pretrain vs. pretrain set | 0.4360 | 2.5772
Model without pretrain vs. pretrain set | 0.4787 | 3.4514

###  A.4 Detailed Results for Different Stages of Homology

The Table 7 presents a performance comparison between Lingo3DMol and
Pocket2Mol on the DUD-E dataset under varying degrees of information leakage.
This leakage is assessed based on the homology between proteins in the
training set and those in the DUD-E dataset.

Table 7: The Detailed Comparison of Lingo3DMol and Pocket2Mol in different stages based on homology |  $>$0.9 | 0.3-0.9 |  $<$0.3
---|---|---|---
|  Pocket2 Mol |  Lingo3DMol (Ours) |  Pocket2 Mol |  Lingo3DMol (Ours) |  Pocket2 Mol |  Lingo3DMol (Ours)
ECFP$>$0.5 ($\uparrow$) | 0.095 | 0.310 | 0.105 | 0.474 | 0.250 | 0.250
Docking Score ($\downarrow$) | -7.205 | -7.274 | -7.292 | -7.844 | -6.640 | -7.096
QED($\uparrow$) | 0.487 | 0.624 | 0.488 | 0.562 | 0.459 | 0.687
SAS($\downarrow$) | 3.886 | 2.935 | 3.796 | 2.767 | 3.728 | 2.759
molWeight | 421.30 | 329.73 | 427.58 | 369.73 | 435.54 | 314.78
RMSD vs. low-energy conformer ($\downarrow$) | 1.1878 | 0.8582 | 1.1397 | 0.8930 | 1.374 | 0.8124
RMSD vs. docking pose ($\downarrow$) | 5.434 | 4.737 | 5.426 | 4.495 | 6.141 | 4.814
RMSD vs. docking pose $<$ 2A ($\uparrow$) | 0.134 | 0.142 | 0.181 | 0.254 | 0.063 | 0.172

###  A.5 Random Cases

Figure 8: Random Cases of 1djy in CrossDock. Figure 9: Random Cases of 3daf in
CrossDock. Figure 10: Random Cases of 2cy0 in CrossDock. Figure 11: Random
Cases of 3u5y in CrossDock. Figure 12: Random Cases of 5w2g in CrossDock.
Figure 13: Random Cases of 3bgs in DUD-E. Figure 14: Random Cases of 3lq8 in
DUD-E. Figure 15: Random Cases of 2hv5 in DUD-E. Figure 16: Random Cases of
2am9 in DUD-E. Figure 17: Random Cases of 1e66 in DUD-E.