2021

These authors contributed equally to this work.

These authors contributed equally to this work.

These authors contributed equally to this work.

[1,2]Amir Barati Farimani

1]Department of Mechanical Engineering, Carnegie Mellon University,
Pittsburgh, PA 15213, USA

2]Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh,
PA 15213, USA

# AugLiChem: Data Augmentation Library of Chemical Structures for Machine
Learning

Rishikesh Magar  Yuyang Wang  Cooper Lorsung  Chen Liang  Hariharan
Ramasubramanian  Peiyuan Li<EMAIL_ADDRESS>[  [

###### Abstract

Machine learning (ML) has demonstrated the promise for accurate and efficient
property prediction of molecules and crystalline materials. To develop highly
accurate ML models for chemical structure property prediction, datasets with
sufficient samples are required. However, obtaining clean and sufficient data
of chemical properties can be expensive and time-consuming, which greatly
limits the performance of ML models. Inspired by the success of data
augmentations in computer vision and natural language processing, we developed
AugLiChem: the data augmentation library for chemical structures. Augmentation
methods for both crystalline systems and molecules are introduced, which can
be utilized for fingerprint-based ML models and Graph Neural Networks (GNNs).
We show that using our augmentation strategies significantly improves the
performance of ML models, especially when using GNNs. In addition, the
augmentations that we developed can be used as a direct plug-in module during
training and have demonstrated the effectiveness when implemented with
different GNN models through the AugliChem library. The Python-based package
for our implementation of Auglichem: Data augmentation library for chemical
structures, is publicly available at: https://github.com/BaratiLab/AugLiChem.

##  1 Introduction

Machine learning (ML) models, especially deep neural networks (DNNs)
lecun2015deep , capable of learning representative features from data, have
gathered growing attention in computational chemistry and material science
compchem . Moreover, advances from areas, such as computer vision (CV) and
natural language processing (NLP), have often had synergistic effect on ML
research in computational chemistry. This has lead to development of graph
neural networks (GNNs) based ML models capable for accurate and efficient
crystalline and molecular property prediction fung2021benchmarking ;
wu2018moleculenet ; MolPeter2019  and fingerprint (FP) based ML models that
take chemical FPs PhysRevB.87.184115_SOAP ; huo2017unified_MBTR ;
lam2017machine_OFM ; botu2017machine ; rogers2010extended  as input and
predict target properties. However, development of the FPs requires manual
design and is sometimes time-consuming to calculate, which limits the
performance of ML models leading to increased popularity of GNN based models.
Recently, GNNs kipf2017semi ; xu2018how  have been a prevalence in
computational chemistry, as GNNs are capable of learning representations from
non-Euclidean graphs of chemical structures directly without manually
engineered descriptors schutt2018schnet ; klicpera_dimenet_2020 . GNNs
developed for crystalline systems and molecules to predict various target
properties and have shown advantage over other ML methods justin2017neural ;
yang2019analyzing ; wang2021molclr ; xie2018cgcnn ; karamad2020orbital  and a
remarkably high performance. However, ML models, including GNNs and FP-based
models, require large amounts of data for training hestness2017deep .
Additionally, the availability of large and clean data is an important pre-
requisite as the performance of ML models scales with the magnitude of data
zhu2016we . However, generating large scale properties datasets for
crystalline systems or molecules usually requires sophisticated and time-
consuming computational or lab experiments brown2019guacamol . The
experimental data can also be noisy and hard to utilize directly. Thus, the
lack of data of has often prevented the use of deep learning models as they
suffer from lower reliability in the small data regime bengio_limit_deep .

The issue of data availability also plagues the areas of computer vision (CV)
and natural language processing(NLP). To address the issue of data
availability, data augmentation techniques have been introduced in CV and NLP,
which generates new instances from available training data via some
transformations to increase the amount and variance of the data
bitton2021augly . Such techniques have been investigated for multiple domains,
like images and text, and has demonstrated the effectiveness in improving the
generalization, performance, and robustness of ML models shorten2019survey ;
kobayashi2018contextual ; coulombe2018text ; pmlr-v119-chen20j . For images in
CV, the fundamental augmentation transformations include cropping, rotation,
color distortion, etc. Following the transformations, some works manually
design transformations devries2017improved  or learn the desired mixture of
image transformations zhang2017mixup ; cubuk2018autoaugment ; yin2019fourier
during training. Augmentations have also been investigated on the
representation domain instead of directly on input, including adding noise,
interpolating, and extrapolating of image representations devries2017dataset ;
konno2018icing . In addition, data augmentations have been widely used for
texts in NLP feng2021survey , like random deletion, insertion, and swap. One
common method is to replace words with their synonyms zhang2015character  or
relative word embeddings wang2015s ; fadaee2017data . Other augmentations
focus on manipulation of the whole sentence, such as back translation
sennrich2015improving  and generative models kafle2017data . Recent works have
also found that simple augmentations (e.g., cropping, resizing, color
distortion for images and token masking for texts) significantly benefit
representation learning through self-supervised pre-training devlin2018bert ;
chen2020simple ; tian2020makes ; gao2021simcse . Motivated by the success of
data augmentation on CV and NLP, some works in the area of molecular machine
learning investigate augmentation for string-based molecular descriptors
(e.g., SMILES weininger1988smiles ) bjerrum2017smiles ; lambard2020smiles  or
conformational oversampling hemmerich2020cover . However, data augmentation is
still under-explored for chemical structures, especially for graph-based ML
models. Unlike images or text, crystalline and molecular systems form
graphical structures and follow chemical rules, like composition of motifs and
boundary conditions. Therefore, direct leverage of existing augmentations,
such as image resizing and cropping, does not work for chemical structures.
This leads us to the question: can we develop data augmentation techniques for
crystalline systems and molecules to boost the performance of ML models,
especially GNNs?

Figure 1: Overview of the augmentations for chemical structures. (a) Framework
of AugLiChem package. (b) Augmentations of crystalline materials for both
conventional ML models and GNNs. (c) Augmentations of molecules for GNNs (top
row) and augmentations for fingerprint-based ML models (bottom row).

We develop AugLiChem: data augmentation library for chemical structures, which
boost the data availability of crystals and molecules and helps improve the
performance of ML models. The augmentations can be implemented as simple plug-
ins to FP-based ML models as well as GNNs. The whole pipeline of AugLiChem is
illustrated in Figure 1(a), which first takes in the chemical structures,
either crystalline or molecular, and creates augmented data instances. The
augmented data is then assigned with the same label as the original instance.
The original chemical instances, together with the augmented ones, are fed
into ML models for property prediction. For crystals, five augmentation
methods are proposed: random perturbation, random rotation, swap axes, random
translation, and supercell transformation, which can be direct plug-ins for
both conventional ML models and graph-based GNNs as shown in Figure 1(b).
Similarly for molecules, we introduce three molecule graph augmentations: atom
masking, bond deletion, and substructure removal. Since fingerprint (FP)
calculation are different between molecules and crystalline materials,
molecule graph augmentations cannot be directly utilized for FP-based ML
models. We consider two extra augmentation methods for molecule FPs. Figure
1(c) illustrates both the graph and FP augmentations for molecules.
Experiments on various benchmarks have demonstrated the effectiveness of our
data augmentations for both crystals and molecules. Especially for GNNs,
augmentation significantly boosts the performance of chemical property
prediction tasks. Finally, for convenient use of our data augmentation
techniques, we develop an open-source library for chemical structure
augmentation based on Python as part of this work. Both crystal and molecule
augmentations can be implemented with a few lines of code. Different GNN
models and widely-used benchmarks are also included in the package. The
package is largely self contained with tutorials describing the functionality
of different package features and example codes to run them.

##  2 Methods

###  2.1 Crystalline Augmentations

We treat a crystalline system as an undirected graph with atoms as the nodes
and the bonds as the edges. The crystal graph can be denoted by
$G_{cr}=(V_{cr},E_{cr})$, where $V_{cr}$ are the nodes of the graph denoting
the atom sites in a crystal structure. The idea of a bond in crystalline
systems is not as concretely defined as in molecular graphs. Therefore, to
construct the edges between the nodes, all atoms within a radius are
considered to be bonded. The strength of the bond is then determined by the
amount of distance from the central atom and edge features are encoded
accordingly. After constructing the graph, we treat as a input feature to the
GNN model, the GNN model then learns from the graph and predicts the material
property.

We implement 5 different types of crystal augmentations as a part of this
work. The augmentations we implement include random perturbation, random
rotation, swap axes, random translate, supercell transformation. Each of the
augmentation that we implement creates a different view of the crystal
enabling the machine learning models particularly GNNs to take advantage of
the large data set and generate accurate representations the crystal. We show
that using augmentations we can improve the prediction performance of GNNs on
all the data sets and also improve performance of the shallow machine learning
models for a majority of the data sets. We would like to note that these
augmentation strategies have been applied only for the training set and not
for the validation and test sets.

####  2.1.1 Random Perturbation

In the random perturbation augmentation all the sites in the crystalline
system are randomly perturbed by a distance between 0 to 0.5 Å. This
augmentation is especially useful in breaking the symmetries that exist
between the sites in the crystals.

####  2.1.2 Random rotation

In the random rotation transform, we randomly rotate the sites in the crystal
between 0 to 360 degrees. To generate the augmentation we initially use the
random perturbation augmentation to generate the initial structure which is
then rotated randomly.

####  2.1.3 Swap Axes

In the swap axes augmentation strategy, we swap the coordinates of the sites
in the crystal between two axes. For example, we may swap the x and the y axes
coordinates or the y and z axes coordinates. The swap axes transform greatly
displaces the locations of all the sites in the crystal. This augmentation
strategy has been inspired from work done by Kim et al.kim2020generative_gan

####  2.1.4 Random Translate

The random translate transform randomly displaces a site in the crystal by a
distance between 0 to 0.5 Å. In this work, we randomly select 25% of the sites
in the crystal and displace them. This creates an augmentation different from
the random perturb augmentation as not all the sites in the crystal are
displaced.

####  2.1.5 Supercell Transformation

The supercell transformation produces a supercell of the crystalline system.
The distinct feature of the supercell of the crystal is that after
transformation the supercell represents the same crystal with a larger volume.
There exists a linear mapping between the basis vectors of crystal and the
basis vectors of the supercell.

###  2.2 Molecule Augmentations

For molecules, we introduce augmentations that can be applied when using FP
based models and GNNs. A molecule can be embedded to a unique FP, like ECFP
rogers2010extended  and RDKFP. We introduce two molecular fingerprint (FP)
augmentation techniques, FP Break and FP Concatenation (FP Concat for
brevity), are proposed for FP-based ML models.

In addition, for GNN models, a molecular graph $G_{m}=(V_{m},E_{m})$ is
defined where nodes $V_{m}$ denote atoms and edges $E_{m}$ denote chemical
bonds between atom pairs xiong2019pushing ; Hu2020Strategies . For $v_{i}\in
V_{m}$, its node feature consists of two components, one-hot embedding of the
atomic type $a_{i}$ and the chirality $c_{i}$, namely $v_{i}=[a_{i},c_{i}]$.
For $e_{ij}\in E_{m}$, its edge feature also consists of two components, one-
hot embedding of the bond type $b_{ij}^{t}$ and one-hot embedding of the bond
directional $b_{ij}^{d}$, namely $e_{ij}=[b_{ij}^{t},b_{ij}^{d}]$. Both node
and edge features can be enlarged with more attributes, like valency and
aromaticity. Three molecular graph augmentation techniques, node masking, bond
deletion, and substructure removal, are also introduced and can be direct
plugins to various GNN models you2020graphcontrstive ; wang2021molclr .

####  2.2.1 Fingerprint Break

A molecule is broken into multiple-level fragments through Breaking
Retrosynthetically Interesting Chemical Substructures (BRICS) degen2008art
decomposition, where each fragment reserves one or more functional groups from
the original chemical structure. Notice that BRICS decomposition follows a
tree-structure such that high-level fragments can contain low-level fragments.
Fragments obtained from BRICS along with the molecule are featurized through
molecular fingerprints, like RDK topological fingerprint (RDKFP) greg2006rdkit
and extended-connectivity fingerprint (ECFP) rogers2010extended . We calculate
the Tanimoto similarity score bajusz2015tanimoto  between fragments and the
molecule and only keep the fragments with score greater than a threshold $S$.
This means fragments which contain important functional groups and share
similar topology with the original molecule are retained. Since the property
of a molecule is heavily determined by the functional groups, fragments and
the molecule are assumed to share similar properties mcnaught1997compendium ;
smith2020march . In FP Break, each fragment is assigned with the same label as
the molecule in the training set. In the validation and test set, only the
original molecules are used.

####  2.2.2 Fingerprint Concatenation

On the other hand, FP Concat follows the same BRICS decomposition as the FP
Break, except that the former concatenates randomly picked $K$ FPs from
fragments and the molecule, and assigns the same label as the original
molecule to each concatenated FP. Additionally, FP Concat includes a
replicated FP that repeats the FP of the original molecule $K$-times, so that
the original FP can be used with the trained models. In the testing phase,
models are tested only on the replicated FPs from the validation and test
sets. In our implementation of both FP Break and FP Concat, $S$ is set as 0.6
and $K$ as 4. In atom masking, a node $v_{i}$ is randomly masked by replacing
the node feature $a_{i}$ with the mask vector $m$. Bond deletion removes an
edge $e_{ij}$ from the molecular graph with a certain probability. And
substructure removal follows the BRICS decomposition in FP augmentations,
where molecular graphs are created based on the decomposed fragments from
BRICS and are assigned with the same label as the original molecule.

####  2.2.3 Atom Masking

In atom masking, a node $v_{i}$ is randomly masked by replacing the node
feature $a_{i}$ with the mask vector $m$, which is a unique vector different
from all the node features in the database. Thus, the GNNs lose the atomic
information concerning the masked atoms during training, which forces the
model to learn the correlation between atoms through messages passing from the
adjacencies.

####  2.2.4 Bond Deletion

Bond deletion randomly removes an edge $e_{ij}$ from the molecular graph given
a certain probability, so that the node pair $(v_{i},v_{j})$ are not directly
connected within the molecular graph. This means the model is forced to
aggregate information from atoms that are not adjacent in the molecular graph.
Bond deletion relates to the breaking of chemical bonds which prompts the
model to learn correlations between the involvements of one molecule in
various reactions.

####  2.2.5 Substructure Removal

Substructure removal follows the BRICS decomposition in FP augmentations,
where molecular graphs are created based on the decomposed fragments from
BRICS and are assigned with the same label as the original molecule. Fragment
graphs contain the one or more important functional groups from the molecule,
and GNNs trained on such augmented data learns to correlate target properties
with functional groups.

###  2.3 Graph Neural Networks for Chemical Structures

Modern GNNs updates node-level representations through the aggregated message
passing between nodes and extract graph-level representations through readout
wu2020comprehensive . Recently, GNNs have been widely investigated in modeling
chemical structures zhou2020graph . GNNs have been succesfully used on the
molecular graphs to learn FPs duvenaud2015conv ; kearnes2016molecular ,
predict molecular properties justin2017neural ; yang2019analyzing ;
huang2020skipgnn ; wang2021molclr , and generate target molecules de2018molgan
; you2018graph ; imrie2020deep . Chemical structures, both organic molecules
and inorganic structures, are represented as graphs. In this work, each node
represents an atom Hu2020Strategies  and edges can be defined as covalent
bonds (molecules) xiong2019pushing ; wang2021molclr ; duvenaud2015conv  or
connection to adjacent atoms (crystals) xie2018cgcnn ; klicpera_dimenet_2020 .
This makes molecular graphs naturally translationally and rotationally
invariant. Examples of building a graph from a crystalline system and a
molecule is illustrated in Figure 2.

(a)

(b)

Figure 2: GNNs for crystalline systems and molecules. (a) In the crystal ML
pipeline, we start with an input crystal. The graph representation is
constructed with atoms representing the nodes $V$ and the bonds representing
the edges $E$ connecting adjacent nodes. Once the graph is constructed, the
graph convoluational operations (shown in green) are conducted followed by the
fully-connected layers to predict the target property. (b) Similarly to the
crystalline graph, a molecule graph contains nodes representing atoms, while
edges are defined as chemical bonds between atoms. The built molecule graph is
then fed to the GNNs and fully-connected layers to predict molecular
properties.

GNNs utilize a neighborhood aggregation operation, which update the node
representation iteratively. The update of each node in a GNN layer is given in
Equation 1:

$\displaystyle\begin{split}\boldsymbol{a}_{v}^{(k)}&=\text{AGGREGATE}^{(k)}(\\{\boldsymbol{h}_{u}^{(k-1)}:u\in\mathcal{N}(v)\\}),\\\
\boldsymbol{h}_{v}^{(k)}&=\text{COMBINE}^{(k)}(\boldsymbol{h}_{v}^{(k-1)},\boldsymbol{a}_{v}^{(k)}),\end{split}$
(1)

where $\boldsymbol{h}_{v}^{(k)}$ is the feature of node $v$ at the $k$-th
layer and $\boldsymbol{h}_{v}^{(0)}$ is initialized by node feature
$\boldsymbol{x}_{v}$. $\mathcal{N}(v)$ denotes the set of all the neighbors of
node $v$. GNNs implemented in this work have multiple aggregation and
combination functions, including mean or max pooling kipf2017semi ;
hamilton2017inductive , fully-connected (FC) layers xu2018how , recurrent
neural network (RNN) hamilton2017inductive ; li2019deepgcns , Gaussian weighed
summation xie2018cgcnn ; karamad2020orbital , and attention mechanism
velickovic2018graph ; xiong2019pushing . Other works exploits edge features in
the message passing to leverage more information gong2019exploiting ;
Hu2020Strategies . To extract a graph-level feature $\boldsymbol{h}_{G}$ from
the node features, readout operation integrates all the node features among
the graph $G$ as given in Equation 2:

$\boldsymbol{h}_{G}=\text{READOUT}(\\{\boldsymbol{h}_{u}^{(k)}:v\in G\\}).$
(2)

Common readout operations can be the summation, averaging, or max pooling over
all the nodes xu2018how . More recently, differentiable pooling readout has
been devlopedying2018diffpool ; gao2019graph . Given $\boldsymbol{h}_{G}$,
either FC layers Hu2020Strategies  or RNNs xiong2019pushing  are developed to
map the graph feature to the predicted property.

Unlike two-dimensional (2D) molecular graphs, crystal graphs usually require
the 3D positional data as the input. Crystal graphs treat the atoms in the
crystal as nodes and edges in crystal graphs are defined as relationship
between neighboring atomsschmidt2019recent ; schleder2019dft . Following the
setting, various aggregation operations xie2018cgcnn ; chen2019graph  and
crystalline features st2019message ; karamad2020orbital ; park2020developing
have been investigated in predicting crystal properties using GNNs.
Additionally, some works also consider 3D positional features in molecular
GNNs for property predictions concerning quantum mechanics schutt2018schnet ;
klicpera_dimenet_2020 .

In this work, we use data augmentation strategies for multiple GNN backones
for both the crystalline and molecular systems. For the crystalline systems we
use the CGCNN xie2018cgcnn , MEGNETchen2019graph , SchNetschutt2017schnet ,
GINE modelsHu2020Strategies . The CGCNN model builds a crystal graph by taking
the atoms as the nodes and the modelling the interaction between the atoms as
the edges. To describe the edge features the CGCNN model uses the Gaussian
featurization. The model uses 3 GCN layers to aggregate node and edge features
and does the material property prediction. The MEGNET model takes as input the
atom features, bond features and the current state, the state is
representative of the global feature vector, in the update the bond
information is updated first followed by atom feature vector and global state
vector. The update strategy is performed in the MEGNET blocks introduced in
the architecture and fully connected layers are subsequently used to perform
material property prediction. The SchNet model uses the continuous
convolutional filter layers to model the molecular structures. The SchNet
model uses a feature map to represent the atom feature and takes into account
the atomic positions of the atoms and their nuclear charges. The model uses a
a combination of interaction block and the atom-wise layers to generate the
representations that can be used for prediction tasks. Finally, the GINE model
utilizes an MLP to perform the weighted aggregation of node and edge features.
We use mean pooling as the readout function for the GIN and use the pooled
vector extract the representations of the crystalline system that can be used
used for property prediction.

For molecules, we implement GCN kipf2017semi , GIN xu2018how ;
Hu2020Strategies , DeeperGCN li2019deepgcns ; li2020deepergcn , and
AttentiveFP xiong2019pushing  as GNN backbones to predict molecular
properties. These models follow the message passing framework but differ in
detailed aggregation and combination operations. GCN aggregates all the
neighbors through mean pooling followed by a linear mapping with ReLU
activation. GIN, on the other hand, sums up all the neighboring node features
and develops an MLP afterwards. However, such GNN models suffer from the
vanishing gradient problem when number of graph convolutional layers increases
li2019deepgcns ; li2020deepergcn . To this end, DeeperGCN introduces skip
connections between layers. AttentiveFP utilizes the attention mechanism and
RNNs to aggregate neighbor information and update node features, which reaches
SOTA on several challenging molecular property benchmarks. These models cover
a wide variety of GNN frameworks and greatly demonstrate the effectiveness of
our molecule augmentation techniques.

###  2.4 Training Details

We benchmark our augmentation methods on different crystalline and molecule
datasets. For the five crystalline benchmarks, instances are first randomly
split into train set and test set. The ratio of train set and test set is 4:1.
We further split the training data into training and validation set in the
ratio of 4:1. Finally, we have 64% of the data as training data, 16% of the
data as validation data and the remaining 20% as the test set. The percentages
of train/validation/test sets have been kept the same baseline models which
are trained without augmented data and the models trained with augmented data.
Only the data in the training set has been augmented, we do not augment the
validation and test set data.

For the twelve molecule benchmarks, instances are split into
train/validation/test by the ratio of 8:1:1 via scaffold splitting, which
provides a challenging yet realistic setting Hu2020Strategies ; wang2021molclr
. Augmentations are implemented on the training sets only while validation and
test sets stay intact. Each benchmark is split into the same
training/validation/test sets, and same augmentations are implemented on
instances on the training set if applied. So that all the ML models are
trained and test on the same instances for fair comparison. For example, for
Band Gap benchmark containing 26,709 crystalline instances, it is split into
train/validation/test sets and each includes 17,093/4,274/5,342 instances. If
five augmented instances are created for each crystalline training data, this
gives 85,465 augmented instances in addition to 17,093 instances in the
original training set, which adds up to total 102,558 instances in the
augmented training set. Trained ML models are test on validation sets to
select the best performing model and results on the test set are reported in
this work.

##  3 Results and Discussions

###  3.1 Investigation of Crystalline Augmentations

To improve the performance of ML models, especially GNNs, we introduce five
data augmentation strategies that enhance the amount of data available for
training, namely random perturbation, rotation, random swap axes, random
translation and supercell transformation. To evaluate the effects of using
these augmentations when training GNN models, we benchmark the performance of
the different models with data augmentation on five different datasets and
observe performance gains in all the cases. The five datasets that we include
in our study are: Band Gapdoi:10.1063/1.4812323_MP , Fermi Energy
doi:10.1063/1.4812323_MP , Formation Energy doi:10.1063/1.4812323_MP ,
Lanthanideslam2017machine_OFM , PerovskitesC1EE02717D_castelli . We
implemented 3-fold cross validation strategy to have a better estimate of the
model performance, the MAE and the standard deviation of our experiments are
demonstrated in 1. The models trained on the augmented data are denoted with
subscript ”aug”. For example, the CGCNN model trained on the augmented data is
denoted by CGCNNaug. We observe performance gains on all the data sets for all
the models validating the usefulness of data set augmentations to improve the
performance of the models. We observe a gain between 15% - 33% for the five
data sets with CGCNN. Similarly, a performance gain between 6% - 46% is
observed for the MEGNET model. For the SchNet model the improvements lie
between 6% - 40% and GIN model shows improvement between 2% to - 36%. These
improvements indicate the gain in performance of models when trained with
augmentations. Additional information on datasets and training details for the
GNN model are available in supplementary information (Section - D and Section
- F ). We also use the augmentations with shallow machine learning algorithms
and examine the performance of these models on the five data sets. To
featurize the crystalline systems, we use the AGNI Fingerprints from Botu et
albotu2017machine . The AGNI fingerprints is a vector of size 32 that
represents the crystalline systems. The vector can then be used as feature for
the conventional machine learning algorithms like Random Forest and SVM. The
results obtained for SVM and RF, however, do not show improvement for all the
data sets as in the case of GNN algorithms. We can conclude empirically the
effects of data augmentation are more pronounced for for deep learning based
architectures than the conventional machine learning algorithms.

Table 1: Results of different GNN models on benchmark datasets with and without crystalline systems data augmentation. Only the data points in the training set have been augmented. Both the average and standard deviation of the MAEs are reported.  Dataset | Band Gap | Fermi Energy | Formation Energy | Lanthanides | Perovskites
---|---|---|---|---|---
# of data points | 26709 | 27779 | 26078 | 4133 | 18928
# of Aug data points | 68372 | 71112 | 66756 | 10580 | 48452
Models | Prediction on test set (Random Split)
RF | 0.9276(0.006) | 1.128(0.012) | 0.4474(0.008) | 0.4676(0.01) | 0.4858(0.004)
RFaug | 0.9382(0.009) | 1.1417 (0.011) | 0.4516(0.006) | 0.4603(0.01) | 0.4823 (0.004)
SVM | 1.0520(0.008) | 1.2484(0.012) | 0.5167(0.006) | 0.8885(0.021) | 0.5280(0.005)
SVMaug | 1.0474(0.008) | 1.2674 (0.012) | 0.5225 (0.006) | 0.8688(0.02) | 0.5216(0.004)
CGCNN | 0.4588(0.007) | 0.5054(0.010) | 0.0565(0.001) | 0.109(0.002) | 0.0743(0.0005)
CGCNNaug |  0.3692(0.001) | 0.4271(0.006) | 0.0380(0.001) | 0.0750(0.002) | 0.0496(0.001)
GIN | 0.6006(0.038) | 0.7978(0.014) | 0.1090(0.007) | 0.1969(0.038) | 0.3800(0.008)
GINaug | 0.5314(0.046) | 0.6054(0.15) | 0.0771(0.003) | 0.1243(0.003) | 0.3719(0.002)
MegNet | 0.4774(0.017) | 0.4872(0.010) | 0.0672(0.002) | 0.1859(0.038) | 0.1162(0.004)
MegNetaug | 0.3682 (0.009) | 0.4542(0.021) | 0.0410(0.001) | 0.1041(0.009) | 0.0627(0.007)
SchNet | 0.7769(0.003) | 0.8257(0.021) | 0.3332(0.020) | 0.5644(0.026) | 0.0733(0.003)
SchNetaug | 0.6672(0.018) | 0.7717(0.026) | 0.1986 (0.019) | 0.3706 (0.026) | 0.0521(0.0003)
|  |  |  |  |

###  3.2 Investigation of Molecule Augmentations

We consider two molecular fingerprint (FP) augmentations for FP-based ML
models and three molecular graph augmentations you2020graphcontrstive ;
wang2021molclr  for GNNs. As shown in Table 2, Random Forest (RF) and Support
Vector Machine (SVM) are implemented on top of RDKFP greg2006rdkit  and ECFP
rogers2010extended  for 7 classification benchmarks from MoleculeNet
wu2018moleculenet . All the benchmarks are split through scaffold splitting to
provide a more challenging yet realistic way of splitting Hu2020Strategies .
Within each benchmark, a model is trained for 3 individual runs and both the
mean and standard deviation of compute areas under the receiver operating
characteristic curve (ROC-AUC) on test sets are reported. All the experimental
results for molecules are reported following this setting unless further
mentioning. Models with an ”aug” subscript indicate models are trained with an
augmented data set. In almost all the classification benchmarks, both RF and
SVM trained with augmented data sets outperform baseline models trained only
on original data sets. For instance, FP augmentations improve the ROC-AUC of
SVM by more 6% on ClinTox and by 3% on BACE. It is demonstrated that assigning
the same label for FP augmented data enlarges the available training set and
sharpens the decision boundary of ML models. Also, performances of FP
augmentations on 5 regression benchmarks: FreeSolv, ESOL, Lipo, QM7, QM8, are
investigated as shown in Table 3. Similar with classifications, all benchmarks
are scaffold split. The mean and standard deviation of rooted mean square
errors (RMSEs) in three individual runs are reported. In 4 out of 5 regression
benchmarks, augmentations help boost the performance of SVM models. However,
RFs trained with augmented data fail to improve the RMSE in comparison to
baseline results except on ESOL benchmark. This could because unlike
classification tasks where labels are usually abstract and sparse, labels for
regression are more sensitive and shallow ML models trained on augmented data
can be distracted from the ground truth labels. More detailed results of RF
and SVM can be found in Supplementary Information (Table S1 and Table S2).

Table 2: Results of different ML models on multiple molecular classification benchmarks with and without molecular data augmentations. Both the average and standard deviation of the test ROC-AUC scores are reported.  Dataset | BBBP | BACE | ClinTox | HIV | MUV | Tox21 | SIDER
---|---|---|---|---|---|---|---
# of data points | 2039 | 1513 | 1478 | 41127 | 93087 | 7831 | 1427
# of tasks | 1 | 1 | 2 | 1 | 17 | 12 | 27
Models | ROC-AUC on test set (scaffold split)
RF | 53.34 (1.19) | 71.11 (0.46) | 50.00 (0.00) | 54.59 (1.43) | 50.00 (0.00) | 53.19 (0.49) | 52.13 (1.27)
RFaug | 54.29 (0.45) | 74.87 (1.83) | 50.83 (1.18) | 55.48 (0.19) | 50.62 (0.00) | 54.68 (0.78) | 53.80 (1.42)
SVM | 58.12 (0.00) | 73.55 (0.00) | 60.35 (0.00) | 58.75 (0.00) | 49.98 (0.00) | 55.78 (0.00) | 56.00 (0.00)
SVMaug | 58.62 (0.00) | 76.70 (0.00) | 66.49 (0.00) | 62.73 (0.00) | 53.74 (0.00) | 57.83 (0.00) | 56.52 (0.00)
GCN | 71.82 (0.94) | 75.63 (1.95) | 67.42 (7.62) | 74.05 (3.03) | 71.64 (4.00) | 70.86 (2.58) | 53.60 (3.21)
GCNaug | 73.66 (1.13) | 82.32 (1.87) | 75.20 (2.60) | 75.76 (0.98) | 84.68 (3.63) | 74.63 (1.47) | 68.99 (2.42)
GINE | 71.69 (1.08) | 53.66 (5.98) | 61.40 (1.22) | 68.48 (1.51) | 67.99 (5.70) | 70.24 (1.72) | 58.84 (4.19)
GINEaug | 73.64 (1.17) | 80.27 (2.26) | 76.81 (6.29) | 77.31 (0.32) | 80.01 (3.54) | 75.15 (1.29) | 67.97 (2.01)
DeeperGCN | 61.01 (3.92) | 75.73 (5.29) | 57.56 (1.40) | 65.67 (1.71) | 67.99 (5.70) | 70.78 (1.54) | 53.60 (3.34)
DeeperGCNaug | 70.00 (0.51) | 86.51 (0.39) | 64.23 (3.06) | 75.00 (0.75) | 81.01 (5.30) | 74.80 (1.13) | 65.45 (1.69)
AttentiveFP | 68.56 (1.16) | 81.39 (1.00) | 69.97 (4.53) | 75.46 (2.32) | 63.60 (7.83) | 70.04 (2.63) | 57.57 (4.13)
AttentiveFPaug | 73.58 (0.65) | 83.32 (0.53) | 76.07 (5.98) | 77.96 (1.27) | 78.22 (3.06) | 75.89 (1.31) | 67.90 (2.16)
|  |  |  |  |  |  |
Table 3: Results of different ML models on multiple molecular regression benchmarks with and without molecular graph data augmentations. Both the average and standard deviation of the RMSEs are reported. Dataset | FreeSolv | ESOL | Lipo | QM7 | QM8
---|---|---|---|---|---
# of data points | 642 | 1128 | 4200 | 6830 | 21786
# of tasks | 1 | 1 | 1 | 1 | 12
Models | RMSE on test set (scaffold split)
RF | 4.049 (0.240) | 1.591 (0.020) | 0.9613 (0.0088) | 166.8 (0.4) | 0.03677 (0.00040)
RFaug | 4.140 (0.007) | 1.528 (0.021) | 0.9950 (0.0041) | 168.4 (1.2) | 0.03741 (0.00011)
SVM | 3.143 (0.000) | 1.496 (0.000) | 0.8186 (0.0000) | 156.9 (0.0) | 0.05445 (0.00000)
SVMaug | 3.092 (0.000) | 1.433 (0.000) | 0.8148 (0.0000) | 169.1 (0.0) | 0.05352 (0.00000)
GCN | 2.847 (0.685) | 1.433 (0.067) | 0.8423 (0.0569) | 123.0 (0.9) | 0.03660 (0.00112)
GCNaug | 2.311 (0.198) | 1.358 (0.039) | 0.8069 (0.0104) | 120.8 (1.8) | 0.03480 (0.00081)
GIN | 2.760 (0.180) | 1.450 (0.021) | 0.8500 (0.0722) | 124.8 (0.7) | 0.03708 (0.00092)
GINaug | 2.434 (0.051) | 1.325 (0.028) | 0.7914 (0.0310) | 118.6 (1.1) | 0.03430 (0.00066)
DeeperGCN | 3.107 (0.462) | 1.433 (0.030) | 1.0008 (0.0150) | 128.1 (2.9) | 0.04224 (0.00115)
DeeperGCNaug | 2.368 (0.106) | 1.431 (0.057) | 0.8095 (0.0102) | 124.9 (1.1) | 0.03969 (0.00256)
AttentiveFP | 2.292 (0.424) | 0.884 (0.070) | 0.7682 (0.0429) | 119.4 (0.6) | 0.03466 (0.0006)
AttentiveFPaug | 1.779 (0.081) | 0.857 (0.016) | 0.7518 (0.0092) | 117.8 (1.6) | 0.03419 (0.0008)
|  |  |  |  |

Besides FP augmentation, we introduce three molecular graph augmentation
techniques, node masking, bond deletion, and substructure removal, focusing on
GNN models you2020graphcontrstive ; wang2021molclr . To demonstrate how
molecular graph augmentation benefits the molecular property prediction, we
implement multiple GNN models, including GCN kipf2017semi , GIN xu2018how ;
Hu2020Strategies , DeeperGCN li2019deepgcns ; li2020deepergcn , and
AttentiveFP xiong2019pushing , and compare the results trained with and
without augmentations. For more details of the GNN implementation and
training, please refer to Supplementary Information (Section - G). Table 2
shows the test ROC-AUC of GNNs model on multiple classification benchmarks. It
is demonstrated that on all the 7 classification benchmarks containing 61
tasks, GNN models trained with augmented data surpasses baseline results by
significant amounts. For example, GIN models trained with augmentations
achieve an averaged 11.27% improvement comparing to GIN with no augmentation.
Also, molecular graph augmentation improves the test RMSE results on all the
regression benchmarks as shown in Table 3. On FreeSolv data set, a small yet
challenging benchmark concerning molecular hydrogen free energy in water,
augmentations reduce RMSE by 0.575 in average of the 4 GNN models, which is a
20.55% improvement with respect to the non-augmented baseline models. Such
augmentation techniques not only benefit training with limited data, but also
improves performance on large data sets with tens of thousands molecules, like
HIV, MUV, and QM8. Results shown in Table 2 and 3 demonstrate that our
molecular graph augmentations improve the performance of challenging
classification and regression property prediction tasks on various GNN models,
regardless of different message passing and aggregation functions.

###  3.3 Combination of Different Graph Augmentations

Figure 3: Comparison of molecular graph augmentations and their combinations.
(a) Test ROC-AUC on classification benchmarks. (b) Test normalized RMSE on
regression benchmarks.

(a)

(b)

Figure 4: Comparison of Crystalline system augmentations and their
combinations. The Bar plot represents the normalized MAE of the five different
datasets.

To understand the effect of augmentations and evaluate the best strategy for
using these augmentations, we conduct a systematic ablation study for both
crystals and molecules. For crystalline systems, we trained using all the five
augmentation and evaluate the performance of CGCNN model using MAE as a
performance metric. The plot for the normalized MAE for all the 5 datasets
with different augmentation strategies has been shown in Figure 3. To find the
optimal augmentation strategy we experiment with different augmentation
strategies: no augmentation (None), random perturbation/random
rotation(RP+RR), random perturbation/ random rotation/random
translation(RP+RR+RT), random perturbation/ random rotation/random
translation/swap axes (RP+RR+RT+SA), all augmentations(RP+RR+RT+SA+ST). To
conduct the ablation all the training parameters were kept the same to ensure
fair comparison among the augmentation strategies. Finally, we select the
random perturbation, random rotation and swap axes along with original data as
our final augmentation strategy to ensure computational efficiency and higher
performance when training our GNNs and fingerprint based ML models. However,
we would like to note that an optimal augmentation strategy is task specific
and data dependent.

To investigate the optimal augmentation strategy in molecules, we compare the
performance of different molecular graph augmentations as well as their
combinations. Figure 4 shows the test results, where mean and standard
deviation of 3 individual runs are illustrated by the height of each column
and error bar respectively. In particular, 6 augmentation strategies are
considered: no augmentation (None), node masking (NM), bond deletion (BD),
substructure removal (SR), node masking with bond deletion (NM+BD), and all
augmentations combined (NM+BD+SR). For comparison, AttentiveFP models with the
same settings are implemented for each benchmark with different augmentations.
More implementation details and results concerning augmentation combinations
can be found in Supplementary Information (Table S4 and Table S5).

Test ROC-AUC of classification tasks are illustrated in Figure 4(a).
Augmentations, regardless of the specific strategies, improve the performance
on most benchmarks. When applying one augmentation solely, NM and BD are
generally better choices comparing to the SR. Further, the combination of NM
and BD shows rival improvement as NM or BD, and even surpasses the single
augmentation strategy in multiple benchmarks, including BBBP, HIV, Tox21, and
SIDER. Figure 4(b) demonstrates the test normalized RMSE of different
augmentations on regression tasks. Normalized RMSE is calculated by dividing
the RMSE by the range of label in each dataset. BD achieves the best
performance among NM, BD, and SR in most benchmarks. Results from NM are also
close to BD and even better in Lipo dataset. Similarly, NM with BD obtains
competitive performance and surpasses other strategies in 3 out of all 5
benchmarks. In general, NM, BD, and the combination of these two are the best
performing augmentation strategies in most classification and regression
benchmarks. However, it should be pointed out that the optimal augmentation
strategy is task-dependent. For instance, though SR struggles to compete with
other augmentations in most cases, in BACE and MUV, SR still achieves
comparable improvement to NM and BD. Also, combination of all 3 molecular
graph augmentations does not perform well compared to other strategies. This
maybe due to such combinations generating augmented graphs that are hard to
match with the original graph.

###  3.4 AugLiChem Package

The package is designed with ease of use in mind for all aspects. Publicly
available data is downloaded and cleaned automatically, significantly reducing
the time to start running experiments. Implementations of popular models are
available for easy comparison. Both random splitting and k-fold cross
validation can be done for crystal data sets. Single and multi-target training
can be done with molecule data sets, where handling missing data is done
automatically. These training styles are included for ease of experimentation.
Both molecule and crystal data handling is also designed so only the training
set is augmented, ensuring evaluation validity. Examples can be found in
Supplementary Information (Section - H). Detailed usage guides and
documentation for all relevant functions and models:
https://baratilab.github.io/AugLiChem/.

##  4 Conclusion

The work introduces a systematic methodology for augmentation of molecular
data sets. We design augmentation strategies for crystalline systems and
molecules. Using these augmentation, we are able to significantly boost the
amount of data available for the machine learning models. For crystalline
systems, we design five different augmentation strategies for crystalline
systems and select the most effective ones to train ML models. Using the
augmented data we observe significant gains in performance of four different
graph neural networks on five different data sets consisting a wide range of
properties of crystalline systems. Similarly, for molecules we investigate two
augmentation strategies for fingerprint-based ML and three strategies for GNN
models. We observe performance gains on various popular benchmarks (seven
classification and five regression data sets in total) when compared to
performance of baseline GNNs without augmentation, indicating the
effectiveness of augmentation strategies. The performance gains observed when
training GNN models empirically proves the effectiveness of data augmentation
strategies on crystalline and molecular data sets. Finally, we also develop an
open source python package that implements all the augmentations introduced in
the paper and can be imported directly into any machine learning workflow.

##  5 Data Availability Statement

The data that support the findings of this study are openly available in
Materials Project (reference number: 79), Molecule Net Benchmark(reference
number: 4), Perovskites dataset (reference number: 80), Lanthanides dataset
(reference number: 8).

## References

  *   * (1) LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. nature 521(7553), 436–444 (2015)
  * (2) Keith, J.A., Vassilev-Galindo, V., Cheng, B., Chmiela, S., Gastegger, M., Müller, K.-R., Tkatchenko, A.: Combining machine learning and computational chemistry for predictive insights into chemical systems. Chemical Reviews 121(16), 9816–9872 (2021) https://doi.org/10.1021/acs.chemrev.1c00107. https://doi.org/10.1021/acs.chemrev.1c00107. PMID: 34232033
  * (3) Fung, V., Zhang, J., Juarez, E., Sumpter, B.G.: Benchmarking graph neural networks for materials chemistry. npj Computational Materials 7(1), 1–8 (2021)
  * (4) Wu, Z., Ramsundar, B., Feinberg, E.N., Gomes, J., Geniesse, C., Pappu, A.S., Leswing, K., Pande, V.: Moleculenet: a benchmark for molecular machine learning. Chemical science 9(2), 513–530 (2018)
  * (5) Elton, D.C., Boukouvalas, Z., Fuge, M.D., Chung, P.W.: Deep learning for molecular design—a review of the state of the art. Mol. Syst. Des. Eng. 4, 828–849 (2019). https://doi.org/10.1039/C9ME00039A
  * (6) Bartók, A.P., Kondor, R., Csányi, G. Phys. Rev. B 87, 184115 (2013)
  * (7) Huo, H., Rupp, M.: Unified Representation of Molecules and Crystals for Machine Learning (2017)
  * (8) Lam Pham, T., Kino, H., Terakura, K., Miyake, T., Tsuda, K., Takigawa, I., Chi Dam, H.: Machine learning reveals orbital interaction in materials. Science and technology of advanced materials 18(1), 756–765 (2017)
  * (9) Botu, V., Batra, R., Chapman, J., Ramprasad, R.: Machine learning force fields: construction, validation, and outlook. The Journal of Physical Chemistry C 121(1), 511–522 (2017)
  * (10) Rogers, D., Hahn, M.: Extended-connectivity fingerprints. Journal of chemical information and modeling 50(5), 742–754 (2010)
  * (11) Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: International Conference on Learning Representations (ICLR) (2017)
  * (12) Xu, K., Hu, W., Leskovec, J., Jegelka, S.: How powerful are graph neural networks? In: International Conference on Learning Representations (2019)
  * (13) Schütt, K.T., Sauceda, H.E., Kindermans, P.-J., Tkatchenko, A., Müller, K.-R.: Schnet–a deep learning architecture for molecules and materials. The Journal of Chemical Physics 148(24), 241722 (2018)
  * (14) Klicpera, J., Groß, J., Günnemann, S.: Directional message passing for molecular graphs. In: International Conference on Learning Representations (ICLR) (2020)
  * (15) Gilmer, J., Schoenholz, S.S., Riley, P.F., Vinyals, O., Dahl, G.E.: Neural message passing for quantum chemistry. In: Proceedings of the 34th International Conference on Machine Learning - Volume 70. ICML’17, pp. 1263–1272 (2017)
  * (16) Yang, K., Swanson, K., Jin, W., Coley, C., Eiden, P., Gao, H., Guzman-Perez, A., Hopper, T., Kelley, B., Mathea, M., et al.: Analyzing learned molecular representations for property prediction. Journal of chemical information and modeling 59(8), 3370–3388 (2019)
  * (17) Wang, Y., Wang, J., Cao, Z., Farimani, A.B.: Molclr: Molecular contrastive learning of representations via graph neural networks. arXiv preprint arXiv:2102.10056 (2021)
  * (18) Xie, T., Grossman, J.C.: Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Phys. Rev. Lett. 120, 145301 (2018). https://doi.org/10.1103/PhysRevLett.120.145301
  * (19) Karamad, M., Magar, R., Shi, Y., Siahrostami, S., Gates, I.D., Farimani, A.B.: Orbital graph convolutional neural network for material property prediction. Physical Review Materials 4(9), 093801 (2020)
  * (20) Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M., Ali, M., Yang, Y., Zhou, Y.: Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409 (2017)
  * (21) Zhu, X., Vondrick, C., Fowlkes, C.C., Ramanan, D.: Do we need more training data? International Journal of Computer Vision 119(1), 76–92 (2016)
  * (22) Brown, N., Fiscato, M., Segler, M.H., Vaucher, A.C.: Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling 59(3), 1096–1108 (2019)
  * (23) Bengio, Y., Lecun, Y., Hinton, G.: Deep learning for ai. Commun. ACM 64(7), 58–65 (2021). https://doi.org/10.1145/3448250
  * (24) Bitton, J., Papakipos, Z.: AugLy: A data augmentations library for audio, image, text, and video. https://github.com/facebookresearch/AugLy (2021). https://doi.org/10.5281/zenodo.5014032
  * (25) Shorten, C., Khoshgoftaar, T.M.: A survey on image data augmentation for deep learning. Journal of Big Data 6(1), 1–48 (2019)
  * (26) Kobayashi, S.: Contextual augmentation: Data augmentation by words with paradigmatic relations. arXiv preprint arXiv:1805.06201 (2018)
  * (27) Coulombe, C.: Text data augmentation made simple by leveraging nlp cloud apis. arXiv preprint arXiv:1812.04718 (2018)
  * (28) Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: III, H.D., Singh, A. (eds.) Proceedings of the 37th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 119, pp. 1597–1607. PMLR, ??? (2020). https://proceedings.mlr.press/v119/chen20j.html
  * (29) DeVries, T., Taylor, G.W.: Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 (2017)
  * (30) Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017)
  * (31) Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V., Le, Q.V.: Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501 (2018)
  * (32) Yin, D., Lopes, R.G., Shlens, J., Cubuk, E.D., Gilmer, J.: A fourier perspective on model robustness in computer vision. arXiv preprint arXiv:1906.08988 (2019)
  * (33) DeVries, T., Taylor, G.W.: Dataset augmentation in feature space. arXiv preprint arXiv:1702.05538 (2017)
  * (34) Konno, T., Iwazume, M.: Icing on the cake: An easy and quick post-learnig method you can try after deep learning. arXiv preprint arXiv:1807.06540 (2018)
  * (35) Feng, S.Y., Gangal, V., Wei, J., Chandar, S., Vosoughi, S., Mitamura, T., Hovy, E.: A survey of data augmentation approaches for nlp. arXiv preprint arXiv:2105.03075 (2021)
  * (36) Zhang, X., Zhao, J., LeCun, Y.: Character-level convolutional networks for text classification. Advances in neural information processing systems 28, 649–657 (2015)
  * (37) Wang, W.Y., Yang, D.: That’s so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using# petpeeve tweets. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2557–2563 (2015)
  * (38) Fadaee, M., Bisazza, A., Monz, C.: Data augmentation for low-resource neural machine translation. arXiv preprint arXiv:1705.00440 (2017)
  * (39) Sennrich, R., Haddow, B., Birch, A.: Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709 (2015)
  * (40) Kafle, K., Yousefhussien, M., Kanan, C.: Data augmentation for visual question answering. In: Proceedings of the 10th International Conference on Natural Language Generation, pp. 198–202 (2017)
  * (41) Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  * (42) Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607 (2020). PMLR
  * (43) Tian, Y., Sun, C., Poole, B., Krishnan, D., Schmid, C., Isola, P.: What makes for good views for contrastive learning? arXiv preprint arXiv:2005.10243 (2020)
  * (44) Gao, T., Yao, X., Chen, D.: Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821 (2021)
  * (45) Weininger, D.: Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences 28(1), 31–36 (1988)
  * (46) Bjerrum, E.J.: Smiles enumeration as data augmentation for neural network modeling of molecules. arXiv preprint arXiv:1703.07076 (2017)
  * (47) Lambard, G., Gracheva, E.: Smiles-x: autonomous molecular compounds characterization for small datasets without descriptors. Machine Learning: Science and Technology 1(2), 025004 (2020)
  * (48) Hemmerich, J., Asilar, E., Ecker, G.F.: Cover: conformational oversampling as data augmentation for molecules. Journal of cheminformatics 12(1), 1–12 (2020)
  * (49) Kim, S., Noh, J., Gu, G.H., Aspuru-Guzik, A., Jung, Y.: Generative adversarial networks for crystal structure prediction. ACS central science 6(8), 1412–1420 (2020)
  * (50) Xiong, Z., Wang, D., Liu, X., Zhong, F., Wan, X., Li, X., Li, Z., Luo, X., Chen, K., Jiang, H., et al.: Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism. Journal of medicinal chemistry 63(16), 8749–8760 (2019)
  * (51) Hu*, W., Liu*, B., Gomes, J., Zitnik, M., Liang, P., Pande, V., Leskovec, J.: Strategies for pre-training graph neural networks. In: International Conference on Learning Representations (2020)
  * (52) You, Y., Chen, T., Sui, Y., Chen, T., Wang, Z., Shen, Y.: Graph contrastive learning with augmentations. In: Advances in Neural Information Processing Systems, vol. 33, pp. 5812–5823 (2020)
  * (53) Degen, J., Wegscheid-Gerlach, C., Zaliani, A., Rarey, M.: On the art of compiling and using’drug-like’chemical fragment spaces. ChemMedChem 3(10), 1503 (2008)
  * (54) Landrum, G.: RDKit: Open-source Cheminformatics
  * (55) Bajusz, D., Rácz, A., Héberger, K.: Why is tanimoto index an appropriate choice for fingerprint-based similarity calculations? Journal of cheminformatics 7(1), 1–13 (2015)
  * (56) McNaught, A.D., Wilkinson, A., et al.: Compendium of Chemical Terminology vol. 1669. Blackwell Science Oxford, ??? (1997)
  * (57) Smith, M.B.: March’s Advanced Organic Chemistry: Reactions, Mechanisms, and Structure. John Wiley & Sons, ??? (2020)
  * (58) Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., Philip, S.Y.: A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems 32(1), 4–24 (2020)
  * (59) Zhou, J., Cui, G., Hu, S., Zhang, Z., Yang, C., Liu, Z., Wang, L., Li, C., Sun, M.: Graph neural networks: A review of methods and applications. AI Open 1, 57–81 (2020)
  * (60) Duvenaud, D.K., Maclaurin, D., Iparraguirre, J., Bombarell, R., Hirzel, T., Aspuru-Guzik, A., Adams, R.P.: Convolutional networks on graphs for learning molecular fingerprints. In: Advances in Neural Information Processing Systems, vol. 28 (2015)
  * (61) Kearnes, S., McCloskey, K., Berndl, M., Pande, V., Riley, P.: Molecular graph convolutions: moving beyond fingerprints. Journal of computer-aided molecular design 30(8), 595–608 (2016)
  * (62) Huang, K., Xiao, C., Glass, L.M., Zitnik, M., Sun, J.: Skipgnn: predicting molecular interactions with skip-graph networks. Scientific reports 10(1), 1–16 (2020)
  * (63) De Cao, N., Kipf, T.: Molgan: An implicit generative model for small molecular graphs. arXiv preprint arXiv:1805.11973 (2018)
  * (64) You, J., Liu, B., Ying, R., Pande, V., Leskovec, J.: Graph convolutional policy network for goal-directed molecular graph generation. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 6412–6422 (2018)
  * (65) Imrie, F., Bradley, A.R., van der Schaar, M., Deane, C.M.: Deep generative models for 3d linker design. Journal of chemical information and modeling 60(4), 1983–1995 (2020)
  * (66) Hamilton, W., Ying, Z., Leskovec, J.: Inductive representation learning on large graphs. In: Advances in Neural Information Processing Systems (2017)
  * (67) Li, G., Muller, M., Thabet, A., Ghanem, B.: Deepgcns: Can gcns go as deep as cnns? In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9267–9276 (2019)
  * (68) Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., Bengio, Y.: Graph attention networks. In: International Conference on Learning Representations (2018)
  * (69) Gong, L., Cheng, Q.: Exploiting edge features for graph neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9211–9219 (2019)
  * (70) Ying, R., You, J., Morris, C., Ren, X., Hamilton, W.L., Leskovec, J.: Hierarchical graph representation learning with differentiable pooling. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. NIPS’18, pp. 4805–4815 (2018)
  * (71) Gao, H., Ji, S.: Graph u-nets. In: International Conference on Machine Learning, pp. 2083–2092 (2019). PMLR
  * (72) Schmidt, J., Marques, M.R., Botti, S., Marques, M.A.: Recent advances and applications of machine learning in solid-state materials science. npj Computational Materials 5(1), 1–36 (2019)
  * (73) Schleder, G.R., Padilha, A.C., Acosta, C.M., Costa, M., Fazzio, A.: From dft to machine learning: recent approaches to materials science–a review. Journal of Physics: Materials 2(3), 032001 (2019)
  * (74) Chen, C., Ye, W., Zuo, Y., Zheng, C., Ong, S.P.: Graph networks as a universal machine learning framework for molecules and crystals. Chemistry of Materials 31(9), 3564–3572 (2019)
  * (75) St. John, P.C., Phillips, C., Kemper, T.W., Wilson, A.N., Guan, Y., Crowley, M.F., Nimlos, M.R., Larsen, R.E.: Message-passing neural networks for high-throughput polymer screening. The Journal of chemical physics 150(23), 234111 (2019)
  * (76) Park, C.W., Wolverton, C.: Developing an improved crystal graph convolutional neural network framework for accelerated materials discovery. Physical Review Materials 4(6), 063801 (2020)
  * (77) Schütt, K.T., Kindermans, P.-J., Sauceda, H.E., Chmiela, S., Tkatchenko, A., Müller, K.-R.: Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. arXiv preprint arXiv:1706.08566 (2017)
  * (78) Li, G., Xiong, C., Thabet, A., Ghanem, B.: Deepergcn: All you need to train deeper gcns. arXiv preprint arXiv:2006.07739 (2020)
  * (79) Jain, A., Ong, S.P., Hautier, G., Chen, W., Richards, W.D., Dacek, S., Cholia, S., Gunter, D., Skinner, D., Ceder, G., Persson, K.A.: Commentary: The materials project: A materials genome approach to accelerating materials innovation. APL Materials 1(1), 011002 (2013) https://doi.org/10.1063/1.4812323. https://doi.org/10.1063/1.4812323
  * (80) Castelli, I.E., Olsen, T., Datta, S., Landis, D.D., Dahl, S., Thygesen, K.S., Jacobsen, K.W.: Computational screening of perovskite metal oxides for optimal solar light capture. Energy Environ. Sci. 5, 5814–5819 (2012). https://doi.org/10.1039/C1EE02717D