# Improving Molecular Contrastive Learning via Faulty Negative Mitigation and
Decomposed Fragment Contrast

Yuyang Wang  [  Rishikesh Magar  [  Chen Liang  [  Amir Barati Farimani
<EMAIL_ADDRESS>[

###### Abstract

Deep learning has been a prevalence in computational chemistry and widely
implemented in molecule property predictions. Recently, self-supervised
learning (SSL), especially contrastive learning (CL), gathers growing
attention for the potential to learn molecular representations that generalize
to the gigantic chemical space. Unlike supervised learning, SSL can directly
leverage large unlabeled data, which greatly reduces the effort to acquire
molecular property labels through costly and time-consuming simulations or
experiments. However, most molecular SSL methods borrow the insights from the
machine learning community but neglect the unique cheminformatics (e.g.,
molecular fingerprints) and multi-level graphical structures (e.g., functional
groups) of molecules. In this work, we propose iMolCLR: improvement of
Molecular Contrastive Learning of Representations with graph neural networks
(GNNs) in two aspects, (1) mitigating faulty negative contrastive instances
via considering cheminformatics similarities between molecule pairs; (2)
fragment-level contrasting between intra- and inter-molecule substructures
decomposed from molecules. Experiments have shown that the proposed strategies
significantly improve the performance of GNN models on various challenging
molecular property predictions. In comparison to the previous CL framework,
iMolCLR demonstrates an averaged 1.3% improvement of ROC-AUC on 7
classification benchmarks and an averaged 4.8% decrease of the error on 5
regression benchmarks. On most benchmarks, the generic GNN pre-trained by
iMolCLR rivals or even surpasses supervised learning models with sophisticated
architecture designs and engineered features. Further investigations
demonstrate that representations learned through iMolCLR intrinsically embed
scaffolds and functional groups that can reason molecule similarities.

###### keywords:

American Chemical Society, LaTeX

meche] Department of Mechanical Engineering, Carnegie Mellon University,
Pittsburgh, PA, USA meche] Department of Mechanical Engineering, Carnegie
Mellon University, Pittsburgh, PA, USA cheme] Department of Chemical
Engineering, Carnegie Mellon University, Pittsburgh, PA, USA meche] Department
of Mechanical Engineering, Carnegie Mellon University, Pittsburgh, PA, USA
[cheme] Department of Chemical Engineering, Carnegie Mellon University,
Pittsburgh, PA, USA IR,NMR,UV

##  1 Introduction

Recent years have witnessed the development of computational molecule design
and property prediction driven by deep learning (DL) 1, 2, owing to the
ability to perform fast and accurate computation 3, 4, 5, 6. Several works
build deep neural networks on top of cheminformatics fingerprints to predict
molecular properties 7, 8, 9. DL methods have been also implemented on string-
based molecular embeddings, like SMILES 10 and SELFIES 11, for molecule design
12, 13. However, both fingerprints and string embeddings can neglect important
structural information of molecules. Recently, graph neural networks (GNNs)
14, 15 are developed to learn representations from non-Euclidean graphs of
chemical structures, where each node in molecule graphs are defined as an atom
and each edge represent chemical bond or adjacency of atoms 16. Modern GNNs
rely on message-passing to aggregate neighboring node information within the
graph and have been introduced to predict various properties from molecule
graphs 17, 18. Aggregation based upon a continuous filter is also developed
for GNNs to model quantum interactions within molecules 19, 20. Attention
mechanism 21 has also been leveraged in node aggregation for better prediction
accuracy and model interpretability 22, 23. Instead of only considering
molecules as two-dimensional (2D) graphs, several works have built GNNs in use
of 3D molecular conformations with equivariant aggregation 24, 25, 26, 27.

Despite the success of DL in computational chemistry, the potential is greatly
limited by the availability of labeled data, as the collection of molecule
properties usually requires time-consuming and costly lab experiments or
simulations 28. Moreover, it is challenging for DL models trained on such
limited data to generalize among the gigantic chemical space 29, which
significantly restricts real-world applications like drug discovery and
material design. To address this, self-supervised learning (SSL) 30, 31 have
been investigated to utilize the large unlabelled molecule data and learn
representations generalizable to various down-stream applications. Motivated
by the success of SSL in language models, transformer-based models 21, like
BERT 32, have been implemented to learn representations from large SMILES
database 33, 34, 35, 36, 37. Apart from language models, SSL has also been
developed for representation learning from molecule graphs. Liu et al. 38
embed molecules to N-gram representations by assembling the vertex embedding
in short walks. Hu et al. 39 propose both node-level and graph-level GNN pre-
training strategies. The former includes self-supervised context prediction
and attribute masking, while the latter is based on supervised property
prediction, which is still limited by label availability. Following the
insights, Rong et al. 40 propose contextual property and graph-level motif
predictions as SSL tasks combined with a transformer-based model. Besides
node-level pre-training, Zhang et al. 41 introduce a motif-level SSL strategy,
which builds motif trees from molecules and performs motif generative pre-
training. Additionally, contrastive learning (CL) 42, 43, 44, 45, 46, which
learns representation through contrasting positive pairs against negative
pairs, has been a prevalence in representation learning 47 and implemented to
graphical data 48. Wang et al. 49 propose MolCLR, a CL framework for molecular
representation learning, as well as three augmentation strategies to generate
contrastive pairs. Zhang et al. 50 further leverage frequently-occurring
subgraph patterns and perform CL on subgraph level. Liu et al. 51 and Stärk et
al. 52 perform contrastive training between 2D topological structures and 3D
geometric views to learn molecular representations with 3D information
embedded. Also, Zhu et al. 53 develop multi-view CL between SMILES strings and
molecule graphs, encoded by transformer and GNN, respectively. Although CL has
demonstrated the effectiveness in molecular representation learning, these
methods assume all the other molecules are equal negative pairs in contrast
with a given anchor, which introduces faulty negatives. Faulty negatives are
instances that are supposed to be similar with the anchor while considered as
negative instances in CL 54. Such faulty negative instances harm the
robustness and performance of CL pre-trained model on downstream property
prediction tasks. Additionally, the previous motif-level CL learns a motif
dictionary and trains a sampler to sample subgraphs within each molecule 50,
which may ignore unique chemical substructure patterns. Chemical substructures
of molecules contain functional groups that are critical to various molecular
properties, which can provide multi-level information for representation
learning.

Figure 1: Overview of the proposed iMolCLR framework. (a) CL pre-training
pipeline for molecular representation learning. (b) Fine-tuning of pre-trained
model for various property predictions. (c) An example of faulty negative
introduced by ordinary CL methods due to treating all negative instances
equivalently. (d) In iMolCLR, latent vectors of molecules are repelled based
upon the similarities between different molecule pairs to mitigate faulty
negatives. (e) Fragment-level contrast on decomposed molecular substructures
in iMolCLR.

In this work, we propose iMolCLR: improvement of molecular contrastive
learning through mitigating faulty negative instances and contrasting
decomposed chemical fragments as shown in Figure 1. A GNN encoder is first
trained on large unlabeled data to learn expressive molecular representations
via contrasting positive pairs against negative pairs in a self-supervised
manner (Figure 1a). The pre-trained GNN is fine-tuned on downstream datasets
to predict a wide variety of molecular properties (Figure 1b). Unlike ordinary
CL framework that introduces faulty negatives (Figure 1c), iMolCLR does not
treat all negative pairs equivalently. On the contrary, similar molecules are
encouraged to have closer representations than dissimilar ones (Figure 1d).
Apart from molecule-level contrast, different substructures decomposed via
breaking retrosynthetically interesting chemical substructures (BRICS) 55 are
also considered as contrastive negative pairs (Figure 1e). Such a
decomposition strategy maintains major structural features of compounds and
molecular representations are forced to distinguish important functional
groups within molecules. Experiments show that iMolCLR pre-training
significantly improves the performance of GNN models on challenging molecular
property prediction benchmarks. Through fine-tuning, iMolCLR rivals and even
surpasses strong supervised learning baselines on multiple classification and
regression tasks. Additionally, it shows an overall advantage over other SSL
baselines. In particular, iMolCLR outperforms the original CL framework by an
average 1.3% improvement of ROC-AUC on 7 classification benchmarks and an
average 4.8% reduction of the error on 5 regression benchmarks. Additional
investigations demonstrate that the proposed CL framework effectively learns
intrinsic relations between atoms without meticulously engineered input
descriptors. The representations learned through iMolCLR also show an
advantage to reason molecule similarities in consideration of scaffold and
functional groups in comparison to previous SSL methods.

##  2 Methods

###  2.1 Contrastive Learning Framework

CL aims at learning representations through contrasting positive pairs against
negative pairs. We develop the CL framework 43, 49 containing four components:
molecule graph augmentation, GNN-based encoder, non-linear projection head,
and contrastive loss, as shown in Figure 1a. Given a batch of $N$ molecules
$\\{m_{1},\dots,m_{N}\\}$, each molecule $m_{n}$ is augmented into two graphs
$G_{i}$ and $G_{j}$ through augmentations, $t_{i}$ and $t_{j}$, sampled from
$\mathcal{T}$, where $i=2n-1$ and $j=2n$. We implement $\mathcal{T}$ as random
atom masking of 25% and random bond deletion of 25% following widely-used
graph augmentation strategies 49, 56. The masked atoms are colored by orange
and deleted bonds are colored by dark blue in Figure 1a. Two graphs augmented
from the same molecule compose a positive pair while those from different
molecules are negative pairs. The GNN-based encoder $f(\cdot)$ takes in an
augmented graph $G_{i}$ and encodes it to the representation
$\boldsymbol{h}_{i}$, followed by the non-linear projection head $g(\cdot)$
which maps $\boldsymbol{h}_{i}$ into a latent vector $\boldsymbol{z}_{i}$.
Contrastive loss is applied on the $2N$ latent vectors from the projection
head to maximize the agreement between positive pair vectors (e.g.,
$\boldsymbol{z}_{i}$ and $\boldsymbol{z}_{j}$) while minimizing the agreement
between negative ones (e.g., $\boldsymbol{z}_{i}$ and $\boldsymbol{z}_{k}$,
$\boldsymbol{z}_{j}$ and $\boldsymbol{z}_{k}$). After pre-training, the model
$f(\cdot)$ is fine-tuned to predict various molecular properties of interest
as shown in Figure 1b. During fine-tuning, only the GNN encoder $f(\cdot)$ is
preserved followed by a randomly initialized prediction head $p(\cdot)$ to map
the representation to the target property.

###  2.2 Graph Neural Network

A molecule graph $G$ is defined as $G=(V,E)$, where each node $v\in V$
represents an atom and each edge $e_{uv}\in E$ represents a chemical bond
between atoms $u$ and $v$ 57. Each node is featurized as $\boldsymbol{x}_{v}$
and each edge is featurized as $\boldsymbol{\epsilon}_{uv}$, which contain
unambiguous input vectors to denote each node and edge like atomic number and
covalent bond type. Modern GNNs 14 updates the feature of each node layer-wise
through iterative combination and aggregation operations. The update rule for
the feature of node $v$ at $k$-th graph convolutional layer,
$\boldsymbol{h}_{v}^{(k)}$, is given in Equation 1:

$\boldsymbol{a}_{v}^{(k)}=\text{AGGREGATE}^{(k)}\Big{(}\Big{\\{}\boldsymbol{h}_{u}^{(k-1)}:u\in\mathcal{N}(v)\Big{\\}}\Big{)},\;\boldsymbol{h}_{v}^{(k)}=\text{COMBINE}^{(k)}\Big{(}\boldsymbol{h}_{v}^{(k-1)},\boldsymbol{a}_{v}^{(k)}\Big{)},$
(1)

where $\mathcal{N}(v)$ denotes the set of all the neighbors of node $v$.
Aggregation passes the information of neighboring nodes to $v$ and combination
updates the aggregated feature. Each $\boldsymbol{h}_{v}^{(0)}$ is initialized
by the node feature $\boldsymbol{x}_{v}$. After $K$ layers of node updates,
readout operation integrates all the node features within the graph $G$ to a
graph-level feature $\boldsymbol{h}$ as shown in Equation 2:

$\boldsymbol{h}_{G}=\text{READOUT}\Big{(}\Big{\\{}\boldsymbol{h}_{u}^{(k)}:v\in
G\Big{\\}}\Big{)}.$ (2)

In this work, we develop our GNN encoder based on Graph Isomorphism Network
(GIN) 15, a widely-used generic model. To consider edge features, we follow Hu
et al. 39 to extend node aggregation as
$\boldsymbol{a}_{v}^{(k)}=\sum_{u\in\mathcal{N}(v)}\sigma(\boldsymbol{h}_{u}^{(k-1)}+\boldsymbol{\epsilon}_{uv})$
where $\sigma(\cdot)$ is a non-linear activation function. Combination
operation is modeled by summation followed by an MLP as
$\boldsymbol{h}_{v}^{(k)}=\texttt{MLP}(\boldsymbol{h}_{v}^{(k-1)}+\boldsymbol{a}_{v}^{(k)})$.
Readout operation is implemented as an average pooling over all nodes to
obtain a graph-level representation for each molecule.

###  2.3 Mitigating Faulty Negatives

Contrastive loss 58, like the Normalized Temperature-scaled Cross-Entropy (NT-
Xent) loss 43, aims at representation learning through maximizing the
agreement between positive pairs while minimizing the agreement between
negative pairs. Given $2N$ latent vectors
$\\{\boldsymbol{z}_{1},\dots,\boldsymbol{z}_{2N}\\}$ from a batch of $N$
molecules, NT-Xent for a positive pair
$(\boldsymbol{z}_{i},\boldsymbol{z}_{j})$ is given in Equation 3:

$\mathcal{L}_{i,j}=-\log\frac{\exp(\cos(\boldsymbol{z}_{i},\boldsymbol{z}_{j})/\tau)}{\sum_{k=1}^{2N}\mathbbm{1}_{k\neq
i}\exp(\cos(\boldsymbol{z}_{i},\boldsymbol{z}_{k})/\tau)},$ (3)

where $\tau$ is the temperature parameter and
$\cos(\boldsymbol{z}_{i},\boldsymbol{z}_{j})=\frac{\langle\boldsymbol{z}_{i},\boldsymbol{z}_{j}\rangle}{\|\boldsymbol{z}_{i}\|\|\boldsymbol{z}_{j}\|}$
measures the cosine similarity between two latent vectors. However, such a
contrastive loss assumes all negative pairs are equally negative against the
anchor $\boldsymbol{z}_{i}$, which leads to faulty negatives. Faulty negatives
are instances that are similar with the anchor yet are treated as negative
instances in contrastive training 54. An example of faulty negatives
introduced in original molecular CL framework is illustrated in Figure 1c.
When NT-Xent loss is applied, both the molecule $M_{A}$ (o-xylene, CID 7237)
and molecule $M_{C}$ (CID 89970782) are trained as equivalent negative
instances against the anchor molecule $M_{B}$ (toluene, CID 1140). However,
$M_{B}$ has much more similar molecular properties to $M_{A}$ than $M_{C}$,
since $M_{A}$ and $M_{B}$ share a similar structure and functional groups. In
this case, $M_{A}$ is a “faulty negative” as it should not be far away from
the anchor in the representation domain as other negative samples like
$M_{C}$. Faulty negatives strongly repel the anchor and the negative sample,
even though they should preferably be close in representation domain. 59, 60.

To mitigate the effect of faulty negatives in CL, $M_{A}$ and $M_{B}$ should
be “less negative” comparing to $M_{B}$ and $M_{C}$ as illustrated in Figure
1d. Namely, the latent vector of $M_{A}$ is not pushed too far away from
$M_{B}$, while the agreement of $M_{B}$ and $M_{C}$ is still minimized during
training. In particular, we propose a weighted NT-Xent loss
$\mathcal{L}_{i,j}^{\text{w}}$ which penalizes each negative instance against
the anchor via molecular similarities. The similarity measurement between two
latent vectors $(\boldsymbol{z}_{i},\boldsymbol{z}_{k})$ from a negative
molecule pair $(M_{i},M_{k})$ is penalized by a weight coefficient $w_{ik}$,
as given in Equation LABEL:eq:weighted_ntxent:

$\displaystyle\mathcal{L}_{i,j}^{\text{w}}=-\log\frac{\exp(\cos(\boldsymbol{z}_{i},\boldsymbol{z}_{j})/\tau)}{\sum_{k=1}^{2N}\mathbbm{1}_{k\neq
i}\exp(w_{ik}\cos(\boldsymbol{z}_{i},\boldsymbol{z}_{k})/\tau)},$ (4)

where $w_{ik}\in[0,1]$. To identify the faulty negative instances,
cheminformatic fingerprint is leveraged to evaluate the similarity between
molecule pairs, as shown in Equation 5:

$w_{ik}=1-\lambda_{1}\texttt{FPSim}(M_{i},M_{k}),$ (5)

where $\texttt{FPSim}(M_{i},M_{k})$ evaluates the fingerprint similarity of
the given two molecules $(M_{i},M_{k})$ and $\lambda_{1}$ is the
hyperparameter that determines the scale of penalty for faulty negatives. In
this work, we model $\texttt{FPSim}(\cdot,\cdot)$ as the Tanimoto similarity
61 of extended-connectivity fingerprint (ECFP) 62, since it has been
demonstrated to be efficient and effective measurement of molecule
similarities in multiple domains 63. Through the weighted NT-Xent loss, faulty
negative molecule pairs, i.e., those with high fingerprint similarities, are
forced to be closer in representation domain, this greatly mitigates faulty
negatives and benefits the prediction of molecular properties.

###  2.4 Fragment Contrast

Most CL frameworks for molecular representation learning perform contrastive
training on the whole molecule graph level 49, 53, 51, 52. Few works have
investigated motif-level CL for molecules, which learns a table of frequently-
occurring motif embeddings and trains a sampler to generate informative
subgraphs for CL 50. Though such a method shows performance enhancement on
various benchmarks, the learned sampler may not cover all the unique
substructures in the large molecule dataset. Additionally, the previous work
performs contrast across molecule graphs and motif subgraphs, which can lead
to the ignorance of unsampled substructures while each molecule relies on a
combination of different motifs to function. Driven by the insight, we
leverage a widely used systematic fragmentation method, BRICS decomposition
55, to break each molecule into fragments, which preserves major structural
features of chemical compounds 64, and perform CL on substructures
individually from molecule-level contrast. Figure 1e illustrates the proposed
fragment contrast, where different colors indicate fragments obtained from
BRICS. Instead of pooling over the entire graph, we conduct pooling on the
decomposed subgraphs to obtain fragment representations. Different fragments,
either from the same molecule or different molecules, are treated as negative
pairs, like chlorobenzene (colored by purple) and benzofuran (colored by dark
yellow). Notably, some fragments may share similar structures, however, they
are still considered as negative instances. This is because different
fragments have different neighbors to aggregate from. By this means,
representations of fragments are forced to embed their unique neighboring
information through layer-wise aggregation in GNNs. Such neighboring
information can benefit molecular property prediction as molecules function
differently due to the categorical and positional combination of various
functional groups.

Assume $2M$ fragments are created given a batch of $2N$ augmented molecule
graphs. Notice that when conducting augmentations, fragments within molecules
are also randomly transformed (i.e., atom masking and bond deletion). Thus, no
extra augmentation is required to generate positive fragment pairs. A positive
fragment pair $(F_{i},F_{j})$ is mapped to latent vectors
$(\boldsymbol{z}^{\text{frag}}_{i},\boldsymbol{z}^{\text{frag}}_{j})$ through
the same GNN encoder $f(\cdot)$ except the readout is conducted on the
fragment subgraph instead of the whole molecule graph. The contrastive loss on
the fragment level is given in Equation 6:

$\mathcal{L}_{i,j}^{\text{frag}}=-\log\frac{\exp(\cos(\boldsymbol{z}_{i}^{\text{frag}},\boldsymbol{z}_{j}^{\text{frag}})/\tau)}{\sum_{k=1}^{2M}\mathbbm{1}_{k\neq
i}\exp(\cos(\boldsymbol{z}_{i}^{\text{frag}},\boldsymbol{z}_{k}^{\text{frag}})/\tau)}.$
(6)

Eventually, the total loss given in Equation 7 is a combination of the
weighted contrastive loss on the whole molecule graph level shown in Equation
LABEL:eq:weighted_ntxent, and fragment level constrastive loss shown in 6 :

$\mathcal{L}_{i,j}^{total}=\mathcal{L}_{i,j}^{\text{w}}+\lambda_{2}\mathcal{L}_{i,j}^{\text{frag}},$
(7)

where $\lambda_{2}\in(0,1]$ is a hyperparameter that controls the scale of
fragment-level contrast during pre-training.

###  2.5 Datasets

The model is pre-trained on approximately 10 million unique unlabeled
molecules from PubChem 65 collected and cleaned by Chithrananda et al 34. The
pre-training dataset is randomly split into training and validation sets by
the ratio of 95/5. The GNN model is pre-trained on the training set and tested
on the validation set to select the best-performing model.

To evaluate the performance of iMolCLR framework, we fine-tune the pre-trained
GNN model on 12 benchmarks from MoleculeNet 28, including 7 classification and
5 regression benchmarks. These benchmarks contain a wide variety of molecular
properties covering physiology, biophysics, physical chemistry, and quantum
mechanics. During fine-tuning, each dataset is split into
train/validation/test sets through scaffold split by the ratio of 80/10/10
following previous molecule SSL works 39, 40, 49, 53. Comparing to random
split, scaffold split provides a more challenging yet more realistic setting
to benchmark molecular property predictions 28. During fine-tuning, the model
is only trained on the train set and leverages the validation set to select
the best-performing model. The performance of the selected model on the test
set is reported in this work. More details of molecular property benchmarks
can be found in Supplementary Table 1.

###  2.6 Training Details

In iMolCLR pre-training, the GNN encoder embeds each molecule graph into a
512-dimension representation $\boldsymbol{h}$. The projection head is modeled
by an MLP with one hidden layer maps $\boldsymbol{h}$ into 256-dimensional
latent vector $\boldsymbol{z}$. ReLU 66 is implemented as the non-linear
activation function. The whole model is pre-trained for 50 epochs with batch
size 512. We use Adam optimizer 67 with an initial learning rate $5\times
10^{-4}$ and the weight decay $1\times 10^{-5}$. Additionally. cosine learning
rate decay 68 is performed during pre-training.

During fine-tuning, we replace the projection head with a randomly initialized
MLP which maps the representation $\boldsymbol{h}$ into the desired property
prediction while keeping the pre-trained GNN encoder. The pre-trained model is
trained individually for 100 epochs on each task from the benchmarks. We
perform a random search of hyperparameters on validation sets and report the
results on test sets. For each benchmark, we run three individual runs and
report the average and standard deviation of three trials. The whole model is
implemented on PyTorch Geometric 69. More details of fine-tuning
hyperparameters can be found in Supplementary Table 2.

###  2.7 Baselines

To demonstrate the effectiveness of iMolCLR, we compare its performance with
various supervised GNN models. GCN 14 and GIN 15, 39, as prevalent GNN models
for general graphical tasks, are implemented for performance comparison.
Additionally, GNN models that are designed for molecular property prediction
and have achieved state-of-the-art (SOTA) performance on certain benchmarks
are included. D-MPNN 18 leverages a message-passing architecture that is
invariant to molecule graph. SchNet 19 and MGCN 20 model the quantum
interactions within molecules in the graph aggregation. Additionally,
attention-based model, AttentiveFP 22, is included in baselines.

We further compare our proposed method with other pre-training and SSL models.
N-gram 38 obtains molecular representation through assembling the vertex
embedding in short walks. Hu et al. 39 is included, which contains both a
self-supervised node-level and a supervised graph-level pre-training. MolCLR
49 proposes a general CL framework for molecular representation learning.
Notably, Hu et al., MolCLR, and our proposed iMolCLR are all implemented based
on the GIN encoder. Thus, a comparison of these models, as well as supervised-
learning GIN, well reflects the effectiveness of different SSL methods on
various molecular property predictions.

##  3 Results and Discussion

###  3.1 Molecular Property Predictions

Molecular SSL methods are commonly evaluated by their predictive performance
on various molecular properties. It is expected that good molecular
representations learned through SSL greatly boost the prediction performance
39, 40, 49. Table 1 demonstrate the test compute area under the receiver
operating characteristic curve (ROC-AUC) on classification benchmarks of our
CL pre-training model in comparison to a wide variety of supervised (the first
6 models) and self-supervised (the last 4 models) baselines. Both best
performing supervised and self-supervised models for each benchmark are
highlighted in bold. The last column lists the averaged performance over all
the classification benchmarks for each model. As shown in Table 1, iMolCLR
pre-training significantly boosts the performance of the GIN model by 14.2%
comparing to mere supervised learning. Built upon GIN, a generic GNN
architecture, iMolCLR rivals strong supervised learning models and even
exceeds them on 4 out of 7 classification benchmarks, where the latter develop
sophisticated graph convolutional operations or engineered descriptors. For
example, on SIDER and MUV, iMolCLR prevails over the best-performing
supervised models by 6.7% and 13.2%, respectively. Furthermore, iMolCLR
demonstrates an overall preferable performance of 83.1% ROC-AUC on average for
all the datasets that we considered. Particularly, in comparison to MolCLR,
the original CL framework with neither faulty negative mitigation nor fragment
contrast, iMolCLR outperforms by 1.3% ROC-AUC which demonstrates the
effectiveness of improvement for CL in these challenging benchmarks.

Dataset | BBBP | Tox21 | ClinTox | HIV | BACE | SIDER | MUV | Avg.
---|---|---|---|---|---|---|---|---
GCN 14 |  71.8$\pm$0.9 |  70.9$\pm$2.6 |  62.5$\pm$2.8 |  74.0$\pm$3.0 |  71.6$\pm$2.0 |  53.6$\pm$3.2 |  71.6$\pm$4.0 | 68.0
GIN 15 |  65.8$\pm$4.5 |  74.0$\pm$0.8 |  68.2$\pm$3.7 |  75.3$\pm$1.9 |  70.1$\pm$5.4 |  57.3$\pm$1.6 |  71.8$\pm$2.5 | 68.9
SchNet 19 |  84.8$\pm$2.2 |  77.2$\pm$2.3 |  71.5$\pm$3.7 |  70.2$\pm$3.4 |  76.6$\pm$1.1 |  53.9$\pm$3.7 |  71.3$\pm$3.0 | 72.2
MGCN 20 |  85.0$\pm$6.4 |  70.7$\pm$1.6 |  63.4$\pm$4.2 |  73.8$\pm$1.6 |  73.4$\pm$3.0 |  55.2$\pm$1.8 |  70.2$\pm$3.4 | 70.2
D-MPNN 18 |  81.2$\pm$3.8 |  78.9$\pm$1.3 |  90.5$\pm$5.3 |  75.0$\pm$2.1 |  85.3$\pm$5.3 | 63.2$\pm$2.3 |  76.2$\pm$2.8 | 78.7
AttentiveFP 22 | 90.8$\pm$5.0 | 80.7$\pm$2.0 | 93.3$\pm$2.0 | 82.9$\pm$2.2 | 86.3$\pm$1.5 |  60.5$\pm$6.0 | 77.6$\pm$3.1 | 81.7
N-Gram 38 | 91.2$\pm$3.0 |  76.9$\pm$2.7 |  85.5$\pm$3.7 | 83.0$\pm$1.3 |  87.6$\pm$3.5 |  63.2$\pm$0.5 |  81.6$\pm$1.9 | 81.3
Hu et al. 39 |  70.8$\pm$1.5 |  78.7$\pm$0.4 |  78.9$\pm$2.4 |  80.2$\pm$0.9 |  85.9$\pm$0.8 |  65.2$\pm$0.9 |  81.4$\pm$2.0 | 77.3
MolCLR 49 |  73.6$\pm$0.5 | 79.8$\pm$0.7 |  93.2$\pm$1.7 |  80.6$\pm$1.1 | 89.0$\pm$0.3 |  68.0$\pm$1.1 |  88.6$\pm$2.2 | 81.8
iMolCLR |  76.4$\pm$0.7 | 79.9$\pm$0.6 | 95.4$\pm$1.1 |  80.8$\pm$0.1 |  88.5$\pm$0.5 | 69.9$\pm$1.5 | 90.8$\pm$1.7 | 83.1
Table 1: Mean and standard deviation of test ROC-AUC (%) of iMolCLR in
comparison to different supervised and self-supervised learning models on
classification benchmarks.

Additionally, we test the performance of our CL pre-training model and
baselines on 5 regression benchmarks as demonstrated in Table 2. The last
column of the table shows the average of scaled error over all the regression
benchmarks for each model, which is calculated by dividing the error by the
range of each database label. We report root mean square error (RMSE) for
FreeSolv, ESOL, and Lipo and mean absolute error (MAE) for QM7 and QM8
following MoleculeNet 28. Similar to the performance on classification
benchmarks, iMolCLR also demonstrates a rival or even superior prediction
accuracy over supervised baseline models on challenging regression benchmarks.
In terms of the averaged scaled error, iMolCLR outperforms the best-supervised
model, D-MPNN, by 0.0019, which is a nontrivial improvement as regression
tasks are more challenging than classifications. In comparison with the
original CL pre-training, iMolCLR decreases the prediction errors on 4 out of
5 benchmarks and shows competitive performance on the remaining ESOL dataset.
Other SSL baselines also cannot emulate iMolCLR on most datasets. In
particular, iMolCLR shows an advantage of 0.0157 and 0.0239 on scaled error
over N-Gram 38 and Hu et al. 39, respectively.

Dataset | FreeSolv | ESOL | Lipo | QM7 | QM8 | Scaled avg.
---|---|---|---|---|---|---
GCN 14 |  2.87$\pm$0.14 |  1.43$\pm$0.05 |  0.85$\pm$0.08 |  122.9$\pm$2.2 |  0.0366$\pm$0.0011 | 0.1002
GIN 15 |  2.76$\pm$0.18 |  1.45$\pm$0.02 |  0.85$\pm$0.07 |  124.8$\pm$0.7 |  0.0371$\pm$0.0009 | 0.1002
SchNet 19 |  3.22$\pm$0.76 |  1.05$\pm$0.06 |  0.91$\pm$0.10 | 74.2$\pm$6.0 |  0.0204$\pm$0.0021 | 0.0861
MGCN 20 |  3.35$\pm$0.01 |  1.27$\pm$0.15 |  1.11$\pm$0.04 |  77.6$\pm$4.7 |  0.0223$\pm$0.0021 | 0.0982
D-MPNN 18 |  2.18$\pm$0.91 |  0.98$\pm$0.26 | 0.65$\pm$0.05 |  105.8$\pm$13.2 | 0.0143$\pm$0.0022 | 0.0699
AttentiveFP 22 | 2.03$\pm$0.42 | 0.85$\pm$0.06 | 0.65$\pm$0.03 |  126.7$\pm$4.0 |  0.0282$\pm$0.0010 | 0.0755
N-Gram 38 |  2.51$\pm$0.19 | 1.10$\pm$0.03 |  0.88$\pm$0.12 |  125.6$\pm$1.5 |  0.0320$\pm$0.0032 | 0.0919
Hu et al. 39 |  2.83$\pm$0.12 |  1.22$\pm$0.02 |  0.74$\pm$0.00 |  110.2$\pm$6.4 |  0.0191$\pm$0.0003 | 0.0837
MolCLR 49 |  2.20$\pm$0.20 | 1.11$\pm$0.01 | 0.65$\pm$0.08 |  87.2$\pm$2.0 |  0.0174$\pm$0.0013 | 0.0714
iMolCLR | 2.09$\pm$0.03 |  1.13$\pm$0.02 | 0.64$\pm$0.00 | 66.3$\pm$2.0 | 0.0170$\pm$0.0002 | 0.0680
Table 2: Mean and standard deviation of test RMSE (for FreeSolv, ESOL, Lipo)
or MAE (for QM7, QM8) of iMolCLR in comparison to different supervised and
self-supervised learning models on regression benchmarks. The scaled error is
calculated by dividing RMSE/MAE by the range of each benchmark labels.

Overall, experiments on a wide variety of challenging molecular property
benchmarks demonstrate that our proposed iMolCLR is an effective SSL strategy
that greatly improves the performance of pre-trained GNN models. Further,
iMolCLR shows superiority over the original CL framework on both
classification and regression tasks, which validates the effectiveness of
faulty negative mitigation and fragment contrast. The following section
investigates how the two techniques impact the molecular CL methods in detail.

###  3.2 Influence of Faulty Negative Mitigation and Fragment Contrast in
Molecular CL

We further probe the impact of the two improvement strategies: faulty negative
mitigation and fragment contrast on molecular CL. Specifically, whether
contrastive training benefits from each method solely or the combination of
both strategies. To this end, we consider four pre-training strategies:
original CL pre-training, CL with only weighted NT-Xent for faulty negative
mitigation, CL with only fragment contrast, and CL with both improvement
methods, as shown in Figure 2. The four strategies are illustrated by green,
blue, orange, and purple bars, respectively. Figure 2a shows ROC-AUC on
classification tasks and Figure 2b shows scaled error on regression tasks. The
height of each bar denotes the averaged performance while the length of each
error bar represents the standard deviation over three individual runs. It is
illustrated that the integration of weighted NT-Xent and fragment contrast
demonstrates the best performance on 6 out 7 classification benchmarks and 3
out of 5 regression benchmarks. On average, iMolCLR with both improvement
strategies applied surpasses weighted NT-Xent and fragment contrast solely by
0.9% and 1.5% on classifications, respectively. Similarly, iMolCLR shows a
decrease of the averaged scaled error by 0.0003 and 0.0025 over each strategy
alone on regressions. Also, weighted NT-Xent solely improves the property
prediction over original CL on most benchmarks except for Tox21, BACE, and
ESOL. Interestingly, fragment contrast alone shows a limited advantage over
original CL pre-training. This could be because when applying only fragment
contrast, decomposed substructures are considered as negative pairs, this may
exacerbate the faulty negatives in contrastive training. On the other hand,
with faulty negative mitigated through weighted NT-Xent loss, through fragment
contrast the backbone model can easily identify the functional groups or
motifs within each molecule. Thus, the weighted NT-Xent mitigates the faulty
negative instances not only on the molecule level, but also on the decomposed
fragment level. Overall, the combination of faulty negative mitigation
together with fragment contrast greatly improves the molecular CL for property
prediction and it demonstrates an advantage over applying each strategy
solely. More test results of different pre-training strategies can be found in
Supplementary Tables 3 and 4.

Figure 2: Investigation of the impact of weighted NT-Xent and fragment
contrast on molecular CL pre-training. (a) Mean and standard deviation of test
ROC-AUC ($\uparrow$) for different CL strategies on classification benchmarks.
(b) Mean and standard deviation of scaled error ($\downarrow$) for different
CL strategies on regression benchmarks.

###  3.3 Empirical Study of Hyperparameters

To better investigate the proposed strategies for molecular CL, we conduct
empirical study of hyperparameters of contrastive loss given in Equation
LABEL:eq:weighted_ntxent, 5, 6, and 7. Different combinations of
$\lambda_{1}$, $\lambda_{2}$, and $\tau$ are tested on molecular property
prediction benchmarks. $\lambda_{1}$ controls the scale of penalty on faulty
negative instances and $\lambda_{2}$ weighs the magnitude of fragment-level
contrast. Besides, the selection of the appropriate $\tau$ benefits learning
from hard negative samples 43. Table 3 shows the performance of iMolCLR pre-
trained GNNs with different combinations of the three hyperparameters on both
classification and regression benchmarks. $\lambda_{1}$ and $\lambda_{2}$ are
selected from $\\{0.3,0.5,0.7\\}$, and $\tau$ is selected from
$\\{0.05,0.1,0.5\\}$. iMolCLR achieves the best overall performance on
classifications under $\lambda_{1}=0.5,\lambda_{2}=0.5,\tau=0.1$, while on
regressions, using $\lambda_{1}=0.5,\lambda_{2}=0.5,\tau=0.5$ obtains the best
results. Tuning each hyperparameter may have an opposite impact on different
benchmarks. For instance, decreasing $\lambda_{1}$ from 0.5 to 0.3 causes a
drop of 0.8% ROC-AUC on classifications, while a gain of 0.0008 is observed on
regressions. It indicates that the best combination of hyperparameters is
task-dependent on the target property and data distribution. However, it
should be pointed out that though the selection of hyperparameters affects the
performance on downstream molecular property predictions as demonstrated in
Table 3, all the listed hyperparameter combinations still significantly boost
GNNs in comparison to supervised learning. This reflects the robustness of our
proposed molecular CL framework in learning expressive representations.
Detailed test results of property prediction on each benchmark can be found in
Supplementary Tables 3 and 4.

Table 3: Evaluation for iMolCLR pre-trained GNNs with different combinations of $\lambda_{1}$, $\lambda_{2}$, and $\tau$. Both averaged ROC-AUC over classification benchmarks and averaged error over regression benchmarks are reported. $\lambda_{1}$ | 0.5 | 0.3 | 0.7 | 0.5 | 0.5 | 0.5 | 0.5
---|---|---|---|---|---|---|---
$\lambda_{2}$ | 0.5 | 0.5 | 0.5 | 0.3 | 0.7 | 0.5 | 0.5
$\tau$ | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.05 | 0.5
Avg. ROC-AUC (%) ($\uparrow$) | 82.8 | 82.0 | 81.6 | 82.3 | 82.0 | 81.0 | 81.4
Avg. scaled error ($\downarrow$) | 0.0733 | 0.0725 | 0.0730 | 0.0726 | 0.0723 | 0.0736 | 0.0712

###  3.4 Does Extra Features Benefit Molecular CL Pre-training?

We implement simple yet distinguishable node and edge features via RDKit 70 to
model 2D molecule graphs following previous pre-training frameworks 39, 49. In
particular, node features include atomic number and chirality type, and edge
features consist of covalent bond type and direction. However, extra features
can also be considered in molecule graphs 22. In supervised learning, rich
input features are expected to benefit molecular property predictions as more
information is provided. This leads to the question: whether enriched input
benefits molecular CL pre-training for property predictions? To this end, we
introduce more node and edge features to iMolCLR pre-training as shown in
Table 4. Besides the original features, degree, charge, hybridization,
aromatic, and number of hydrogens are included in the node feature, meanwhile,
the stereotype of bond is added to edge features. The extra features are fed
into embedding layers and added to the embedding from original features before
being sent to graph convolutional layers. During augmentation, each feature of
masked nodes is set to a unique code. For example, atomic is set to 0 when
masked. Through adding richer features, more molecular information is provided
during pre-training as well as downstream fine-tuning.

Type | Name | Description | Range
---|---|---|---
Node | atomic | Atomic number | $\\{x:1\leq x\leq 119,x\in\mathbbm{Z}\\}$
chirality | Chirality type | {unspecified, CW, CCW, other}
degree | Number of bonded neighbors | $\\{x:0\leq x\leq 10,x\in\mathbbm{Z}\\}$
charge | Formal charge of the atom | $\\{x:-5\leq x\leq 5,x\in\mathbbm{Z}\\}$
hybrization | Hybrization type |  {sp, sp2, sp3, sp3d, sp3d2, other}
aromatic | Whether on a aromatic ring | $\\{0,1\\}$
hydrogen | Number of bonded hydrogens | $\\{x:0\leq x\leq 5,x\in\mathbbm{Z}\\}$
Edge | bond_type | Type of covalent bonds | {single, double, triple, aromatic}
bond_dir | Direction of covalent bonds | {none, end-upright, end-downright}
stereo | Stereotype | {None, Any, Z, E, Cis, Trans}
Table 4: List of extended node and edge features for molecule graphs.

We compare the results of GNN models pre-trained and fine-tuned with original
and enriched features. As shown in Figure 3, we implement molecular CL pre-
training with weighted NT-Xent loss, where green bars represent training using
original input features while purple bars represent enriched features. Both
the mean and standard deviation of performance on each benchmark are
illustrated in Figure 3. On most benchmarks, adding extra input features
benefits the downstream molecular property prediction. For instance, enriched
features improve ROC-AUC by 1.6% on SIDER and reduce MAE by 3.9 on QM7.
Although adding extra features demonstrates better performances on several
benchmarks, the improvements are limited comparing to the various categories
of extra features included. In few benchmarks, CL with enriched features even
slightly falls behind the original model like on HIV and Tox21. This reveals
that our molecular CL framework effectively learns the intrinsic relationships
between atoms without heavily relying on engineered features. Extra node
features, like degree (i.e., number of neighbors) and hydrogen (i.e., number
of neighboring hydrogen atoms), can be inherently learned by pre-trained graph
aggregations without explicit provided. Other features such as hybrization and
charge are strongly related with the atomic feature, providing a redundant
input for CL. The major takeaway is that our proposed molecular CL, as a self-
supervised pre-training strategy, learns expressive representations from
graphs even, which does not require engineered and enriched features to
achieve better performance.

Figure 3: Comparison of molecular CL pre-training with original and enriched
input features. (a) Test ROC-AUC ($\uparrow$) of pre-trained GNNs on
classification benchmarks. (b) Test scaled error ($\downarrow$) of pre-trained
GNNs on regression benchmarks.

###  3.5 Case Study of iMolCLR Representations

To further evaluate iMolCLR, we compare the molecular representations learned
by iMolCLR with those learned by original CL together with cheminformatics
fingerprints. Given the query molecule (CID: 132820209) shown in Figure 4a, we
compute the Tanimoto similarities 61 of cheminformatics fingerprints between
the query and all the molecules in the $\sim$10M pre-training dataset. Figure
4b and 4c exhibit the distribution of similarities on extended-connectivity
fingerprint (ECFP) 62 and RDKit-specific fingerprint (RDKFP) 70, respectively.
ECFP is a topological fingerprint for structure-activity modeling while RDKFP
identifies subgraphs of different sizes. Overall, ECFP leads to lower
similarity scores than RDKFP due to the different features and algorithms
implemented. Within the database, molecules are considered very similar to the
query with respect to ECFP if their similarity is greater than 0.6 (Figure
4b). While with regard to RDKFP, similar molecules are expected to have
similarities greater than 0.9 (Figure 4c).

Figure 4: Case study of CL learned molecular representations. (a) Query
molecule of CID 132820209. (b) Distribution of Tanimoto similarities on ECFP
between molecules from the pre-training dataset and the query. (c)
Distribution of Tanimoto similarities on RDKFP between molecules from the pre-
training dataset and the query. (d) 8 molecules that are closest to the query
on the original CL learned representation domain with Tanimoto similarities on
ECFP and RDKFP denoted. (e) 8 molecules that are closest to the query on the
iMolCLR learned representation domain with Tanimoto similarities on ECFP and
RDKFP denoted.

We then select 8 molecules that are closest to the query in the representation
domain learned by either original CL (Figure 4d) or iMolCLR (Figure 4e).
Cosine similarity is used to measure the distances between learned
representations. The 8 molecules selected by original CL have averaged ECFP
similarity of 0.540 and RDKFP of 0.839, while those selected by iMolCLR have
remarkably higher averaged ECFP similarity of 0.655 and RDKFP of 0.893. It is
indicated that through faulty negative mitigation with weighted NT-Xent,
iMolCLR embeds molecular cheminformatics in the learned representations, which
can reason molecular similarities. Additionally, benefiting from the fragment
contrast, substructure-level topology is also embedded in iMolCLR
representations. For instance, almost all the closest molecules found by
iMolCLR share the phenanthroline-like substructure of three fused rings with
the query molecule, whereas the original CL retrieves molecules with the
substructure of only two rings fused. Besides, the C-O-C substructure of the
query is captured by iMolCLR and shared among all the selected molecules.
Through the case study, iMolCLR demonstrates the improvement over original CL
on the learned representations, which better embed cheminformatics and
substructure topology. More examples of molecule retrieval through iMolCLR can
be found in Supplementary Figures 1, 2, and 3.

##  4 Conclusions

In this work, we propose iMolCLR, an improvement of molecular contrastive
learning of representations with GNNs. Specifically, two strategies are
introduced in iMolCLR: (1) the weighted NT-Xent loss to mitigate faulty
negative instances during contrastive pre-training, (2) fragment-level
contrast on substructures from BRICS decomposition. The former considers
cheminformatics such that learned representations are related to molecular
similarities, which are neglected by previous molecular CL methods. The
latter, on the other hand, encourages the pre-trained GNNs to embed functional
groups and motif information which are vital to molecular properties.
Benefiting from the two strategies, iMolCLR outperforms other SSL baseline
models, including the original molecular CL, on a wide variety of molecular
property prediction benchmarks. Further investigation demonstrates that
iMolCLR is an effective and robust pre-training framework, which learns
expressive representations from limited input features. iMolCLR, an SSL method
that can leverage large unlabeled data, bears a promise for accurate molecular
property prediction, which can greatly benefit applications like drug and
material discovery.

## References

  * LeCun et al. 2015 LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. _nature_ 2015, _521_ , 436–444
  * Schmidhuber 2015 Schmidhuber, J. Deep learning in neural networks: An overview. _Neural networks_ 2015, _61_ , 85–117
  * Duvenaud et al. 2015 Duvenaud, D.; Maclaurin, D.; Aguilera-Iparraguirre, J.; Gómez-Bombarelli, R.; Hirzel, T.; Aspuru-Guzik, A.; Adams, R. P. Convolutional Networks on Graphs for Learning Molecular Fingerprints. Proceedings of the 28th International Conference on Neural Information Processing Systems. 2015
  * Butler et al. 2018 Butler, K. T.; Davies, D. W.; Cartwright, H.; Isayev, O.; Walsh, A. Machine learning for molecular and materials science. _Nature_ 2018, _559_ , 547–555
  * Wang et al. 2021 Wang, Y.; Cao, Z.; Barati Farimani, A. Efficient water desalination with graphene nanopores obtained using artificial intelligence. _npj 2D Materials and Applications_ 2021, _5_ , 1–9
  * AlQuraishi and Sorger 2021 AlQuraishi, M.; Sorger, P. K. Differentiable biology: using deep learning for biophysics-based and data-driven modeling of molecular mechanisms. _Nature Methods_ 2021, _18_ , 1169–1180
  * Unterthiner et al. 2014 Unterthiner, T.; Mayr, A.; Klambauer, G.; Steijaert, M.; Wegner, J. K.; Ceulemans, H.; Hochreiter, S. Deep learning as an opportunity in virtual screening. Proceedings of the deep learning workshop at NeurIPS. 2014; pp 1–9
  * Ma et al. 2015 Ma, J.; Sheridan, R. P.; Liaw, A.; Dahl, G. E.; Svetnik, V. Deep neural nets as a method for quantitative structure–activity relationships. _Journal of chemical information and modeling_ 2015, _55_ , 263–274
  * Vamathevan et al. 2019 Vamathevan, J.; Clark, D.; Czodrowski, P.; Dunham, I.; Ferran, E.; Lee, G.; Li, B.; Madabhushi, A.; Shah, P.; Spitzer, M., et al. Applications of machine learning in drug discovery and development. _Nature Reviews Drug Discovery_ 2019, _18_ , 463–477
  * Weininger 1988 Weininger, D. SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules. _Journal of chemical information and computer sciences_ 1988, _28_ , 31–36
  * Krenn et al. 2020 Krenn, M.; Häse, F.; Nigam, A.; Friederich, P.; Aspuru-Guzik, A. Self-Referencing Embedded Strings (SELFIES): A 100% robust molecular string representation. _Machine Learning: Science and Technology_ 2020, _1_ , 045024
  * Xu et al. 2017 Xu, Z.; Wang, S.; Zhu, F.; Huang, J. Seq2seq fingerprint: An unsupervised deep molecular embedding for drug discovery. Proceedings of the 8th ACM international conference on bioinformatics, computational biology, and health informatics. 2017; pp 285–294
  * Gómez-Bombarelli et al. 2018 Gómez-Bombarelli, R.; Wei, J. N.; Duvenaud, D.; Hernández-Lobato, J. M.; Sánchez-Lengeling, B.; Sheberla, D.; Aguilera-Iparraguirre, J.; Hirzel, T. D.; Adams, R. P.; Aspuru-Guzik, A. Automatic chemical design using a data-driven continuous representation of molecules. _ACS central science_ 2018, _4_ , 268–276
  * Kipf and Welling 2016 Kipf, T. N.; Welling, M. Semi-supervised classification with graph convolutional networks. _arXiv preprint arXiv:1609.02907_ 2016,
  * Xu et al. 2019 Xu, K.; Hu, W.; Leskovec, J.; Jegelka, S. How Powerful are Graph Neural Networks? International Conference on Learning Representations. 2019
  * Jiang et al. 2021 Jiang, D.; Wu, Z.; Hsieh, C.-Y.; Chen, G.; Liao, B.; Wang, Z.; Shen, C.; Cao, D.; Wu, J.; Hou, T. Could graph neural networks learn better molecular representation for drug discovery? A comparison study of descriptor-based and graph-based models. _Journal of cheminformatics_ 2021, _13_ , 1–23
  * Gilmer et al. 2017 Gilmer, J.; Schoenholz, S. S.; Riley, P. F.; Vinyals, O.; Dahl, G. E. Neural message passing for quantum chemistry. International conference on machine learning. 2017; pp 1263–1272
  * Yang et al. 2019 Yang, K.; Swanson, K.; Jin, W.; Coley, C.; Eiden, P.; Gao, H.; Guzman-Perez, A.; Hopper, T.; Kelley, B.; Mathea, M., et al. Analyzing learned molecular representations for property prediction. _Journal of chemical information and modeling_ 2019, _59_ , 3370–3388
  * Schütt et al. 2018 Schütt, K. T.; Sauceda, H. E.; Kindermans, P.-J.; Tkatchenko, A.; Müller, K.-R. SchNet–A deep learning architecture for molecules and materials. _The Journal of Chemical Physics_ 2018, _148_ , 241722
  * Lu et al. 2019 Lu, C.; Liu, Q.; Wang, C.; Huang, Z.; Lin, P.; He, L. Molecular property prediction: A multilevel quantum interactions modeling perspective. Proceedings of the AAAI Conference on Artificial Intelligence. 2019; pp 1052–1060
  * Vaswani et al. 2017 Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. Advances in neural information processing systems. 2017; pp 5998–6008
  * Xiong et al. 2019 Xiong, Z.; Wang, D.; Liu, X.; Zhong, F.; Wan, X.; Li, X.; Li, Z.; Luo, X.; Chen, K.; Jiang, H., et al. Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism. _Journal of medicinal chemistry_ 2019, _63_ , 8749–8760
  * Ying et al. 2021 Ying, C.; Cai, T.; Luo, S.; Zheng, S.; Ke, G.; He, D.; Shen, Y.; Liu, T.-Y. Do Transformers Really Perform Badly for Graph Representation? Advances in Neural Information Processing Systems. 2021
  * Klicpera et al. 2020 Klicpera, J.; Groß, J.; Günnemann, S. Directional Message Passing for Molecular Graphs. International Conference on Learning Representations. 2020
  * Fuchs et al. 2020 Fuchs, F.; Worrall, D.; Fischer, V.; Welling, M. SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks. Advances in Neural Information Processing Systems. 2020; pp 1970–1981
  * Liu et al. 2021 Liu, Y.; Wang, L.; Liu, M.; Zhang, X.; Oztekin, B.; Ji, S. Spherical Message Passing for 3D Graph Networks. 2021
  * Jing et al. 2021 Jing, B.; Eismann, S.; Suriana, P.; Townshend, R. J. L.; Dror, R. Learning from Protein Structure with Geometric Vector Perceptrons. International Conference on Learning Representations. 2021
  * Wu et al. 2018 Wu, Z.; Ramsundar, B.; Feinberg, E. N.; Gomes, J.; Geniesse, C.; Pappu, A. S.; Leswing, K.; Pande, V. MoleculeNet: a benchmark for molecular machine learning. _Chemical science_ 2018, _9_ , 513–530
  * Oprea and Gottfries 2001 Oprea, T. I.; Gottfries, J. Chemography: the art of navigating in chemical space. _Journal of combinatorial chemistry_ 2001, _3_ , 157–166
  * Hadsell et al. 2006 Hadsell, R.; Chopra, S.; LeCun, Y. Dimensionality reduction by learning an invariant mapping. 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06). 2006; pp 1735–1742
  * Doersch and Zisserman 2017 Doersch, C.; Zisserman, A. Multi-task self-supervised visual learning. Proceedings of the IEEE International Conference on Computer Vision. 2017; pp 2051–2060
  * Devlin et al. 2018 Devlin, J.; Chang, M.-W.; Lee, K.; Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_ 2018,
  * Wang et al. 2019 Wang, S.; Guo, Y.; Wang, Y.; Sun, H.; Huang, J. SMILES-BERT: large scale unsupervised pre-training for molecular property prediction. Proceedings of the 10th ACM international conference on bioinformatics, computational biology and health informatics. 2019; pp 429–436
  * Chithrananda et al. 2020 Chithrananda, S.; Grand, G.; Ramsundar, B. ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction. _arXiv preprint arXiv:2010.09885_ 2020,
  * Fabian et al. 2020 Fabian, B.; Edlich, T.; Gaspar, H.; Segler, M.; Meyers, J.; Fiscato, M.; Ahmed, M. Molecular representation learning with language models and domain-relevant auxiliary tasks. _arXiv preprint arXiv:2011.13230_ 2020,
  * Flam-Shepherd et al. 2021 Flam-Shepherd, D.; Zhu, K.; Aspuru-Guzik, A. Keeping it Simple: Language Models can learn Complex Molecular Distributions. _arXiv preprint arXiv:2112.03041_ 2021,
  * Ross et al. 2021 Ross, J.; Belgodere, B.; Chenthamarakshan, V.; Padhi, I.; Mroueh, Y.; Das, P. Do Large Scale Molecular Language Representations Capture Important Structural Information? _arXiv preprint arXiv:2106.09553_ 2021,
  * Liu et al. 2019 Liu, S.; Demirel, M. F.; Liang, Y. N-Gram Graph: Simple Unsupervised Representation for Graphs, with Applications to Molecules. NeurIPS. 2019
  * Hu et al. 2020 Hu, W.; Liu, B.; Gomes, J.; Zitnik, M.; Liang, P.; Pande, V.; Leskovec, J. Strategies for Pre-training Graph Neural Networks. International Conference on Learning Representations. 2020
  * Rong et al. 2020 Rong, Y.; Bian, Y.; Xu, T.; Xie, W.; WEI, Y.; Huang, W.; Huang, J. Self-Supervised Graph Transformer on Large-Scale Molecular Data. Advances in Neural Information Processing Systems. 2020; pp 12559–12571
  * Zhang et al. 2021 Zhang, Z.; Liu, Q.; Wang, H.; Lu, C.; Lee, C.-K. Motif-based Graph Self-Supervised Learning for Molecular Property Prediction. _Advances in Neural Information Processing Systems_ 2021, _34_
  * He et al. 2020 He, K.; Fan, H.; Wu, Y.; Xie, S.; Girshick, R. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020; pp 9729–9738
  * Chen et al. 2020 Chen, T.; Kornblith, S.; Norouzi, M.; Hinton, G. A simple framework for contrastive learning of visual representations. International conference on machine learning. 2020; pp 1597–1607
  * Caron et al. 2020 Caron, M.; Misra, I.; Mairal, J.; Goyal, P.; Bojanowski, P.; Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. _Advances in Neural Information Processing Systems_ 2020, _33_ , 9912–9924
  * Chen and He 2021 Chen, X.; He, K. Exploring simple siamese representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021; pp 15750–15758
  * Zbontar et al. 2021 Zbontar, J.; Jing, L.; Misra, I.; LeCun, Y.; Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. International Conference on Machine Learning. 2021; pp 12310–12320
  * Bengio et al. 2021 Bengio, Y.; Lecun, Y.; Hinton, G. Deep learning for AI. _Communications of the ACM_ 2021, _64_ , 58–65
  * You et al. 2020 You, Y.; Chen, T.; Sui, Y.; Chen, T.; Wang, Z.; Shen, Y. Graph contrastive learning with augmentations. _Advances in Neural Information Processing Systems_ 2020, _33_ , 5812–5823
  * Wang et al. 2021 Wang, Y.; Wang, J.; Cao, Z.; Farimani, A. B. MolCLR: Molecular contrastive learning of representations via graph neural networks. _arXiv preprint arXiv:2102.10056_ 2021,
  * Zhang et al. 2020 Zhang, S.; Hu, Z.; Subramonian, A.; Sun, Y. Motif-driven contrastive learning of graph representations. _arXiv preprint arXiv:2012.12533_ 2020,
  * Liu et al. 2021 Liu, S.; Wang, H.; Liu, W.; Lasenby, J.; Guo, H.; Tang, J. Pre-training Molecular Graph Representation with 3D Geometry. _arXiv preprint arXiv:2110.07728_ 2021,
  * Stärk et al. 2021 Stärk, H.; Beaini, D.; Corso, G.; Tossou, P.; Dallago, C.; Günnemann, S.; Liò, P. 3D Infomax improves GNNs for Molecular Property Prediction. _arXiv preprint arXiv:2110.04126_ 2021,
  * Zhu et al. 2021 Zhu, J.; Xia, Y.; Qin, T.; Zhou, W.; Li, H.; Liu, T.-Y. Dual-view Molecule Pre-training. _arXiv preprint arXiv:2106.10234_ 2021,
  * Morgado et al. 2021 Morgado, P.; Misra, I.; Vasconcelos, N. Robust Audio-Visual Instance Discrimination. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021; pp 12934–12945
  * Degen et al. 2008 Degen, J.; Wegscheid-Gerlach, C.; Zaliani, A.; Rarey, M. On the art of compiling and using ‘drug-like’ chemical fragment spaces. _ChemMedChem_ 2008, _3_ , 1503
  * Magar et al. 2021 Magar, R.; Wang, Y.; Lorsung, C.; Liang, C.; Ramasubramanian, H.; Li, P.; Farimani, A. B. AugLiChem: Data Augmentation Library of Chemical Structures for Machine Learning. _arXiv preprint arXiv:2111.15112_ 2021,
  * Bronstein et al. 2017 Bronstein, M. M.; Bruna, J.; LeCun, Y.; Szlam, A.; Vandergheynst, P. Geometric deep learning: going beyond euclidean data. _IEEE Signal Processing Magazine_ 2017, _34_ , 18–42
  * Oord et al. 2018 Oord, A. v. d.; Li, Y.; Vinyals, O. Representation learning with contrastive predictive coding. _arXiv preprint arXiv:1807.03748_ 2018,
  * Robinson et al. 2021 Robinson, J. D.; Chuang, C.-Y.; Sra, S.; Jegelka, S. Contrastive Learning with Hard Negative Samples. International Conference on Learning Representations. 2021
  * Huynh et al. 2020 Huynh, T.; Kornblith, S.; Walter, M. R.; Maire, M.; Khademi, M. Boosting contrastive self-supervised learning with false negative cancellation. _arXiv preprint arXiv:2011.11765_ 2020,
  * Chen and Reynolds 2002 Chen, X.; Reynolds, C. H. Performance of similarity measures in 2D fragment-based similarity searching: comparison of structural descriptors and similarity coefficients. _Journal of chemical information and computer sciences_ 2002, _42_ , 1407–1414
  * Rogers and Hahn 2010 Rogers, D.; Hahn, M. Extended-connectivity fingerprints. _Journal of chemical information and modeling_ 2010, _50_ , 742–754
  * Bajusz et al. 2015 Bajusz, D.; Rácz, A.; Héberger, K. Why is Tanimoto index an appropriate choice for fingerprint-based similarity calculations? _Journal of cheminformatics_ 2015, _7_ , 1–13
  * Liu et al. 2017 Liu, T.; Naderi, M.; Alvin, C.; Mukhopadhyay, S.; Brylinski, M. Break down in order to build up: decomposing small molecules for fragment-based drug design with e MolFrag. _Journal of chemical information and modeling_ 2017, _57_ , 627–631
  * Kim et al. 2019 Kim, S.; Chen, J.; Cheng, T.; Gindulyte, A.; He, J.; He, S.; Li, Q.; Shoemaker, B. A.; Thiessen, P. A.; Yu, B., et al. PubChem 2019 update: improved access to chemical data. _Nucleic acids research_ 2019, _47_ , D1102–D1109
  * Maas et al. 2013 Maas, A. L.; Hannun, A. Y.; Ng, A. Y. Rectifier nonlinearities improve neural network acoustic models. Proc. icml. 2013; p 3
  * Kingma and Ba 2014 Kingma, D. P.; Ba, J. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_ 2014,
  * Loshchilov and Hutter 2016 Loshchilov, I.; Hutter, F. Sgdr: Stochastic gradient descent with warm restarts. _arXiv preprint arXiv:1608.03983_ 2016,
  * Fey and Lenssen 2019 Fey, M.; Lenssen, J. E. Fast Graph Representation Learning with PyTorch Geometric. ICLR Workshop on Representation Learning on Graphs and Manifolds. 2019
  * Landrum 2006 Landrum, G. RDKit: Open-source cheminformatics. https://www.rdkit.org/, 2006