# Graph neural networks for materials science and chemistry

Patrick Reiser  Institute of Theoretical Informatics, Karlsruhe Institute of
Technology, Am Fasanengarten 5, 76131 Karlsruhe, Germany  Institute of
Nanotechnology, Karlsruhe Institute of Technology, Hermann-von-Helmholtz-Platz
1, 76344 Eggenstein-Leopoldshafen, Germany  Marlen Neubert  Institute of
Theoretical Informatics, Karlsruhe Institute of Technology, Am Fasanengarten
5, 76131 Karlsruhe, Germany  André Eberhard  Institute of Theoretical
Informatics, Karlsruhe Institute of Technology, Am Fasanengarten 5, 76131
Karlsruhe, Germany  Luca Torresi  Institute of Theoretical Informatics,
Karlsruhe Institute of Technology, Am Fasanengarten 5, 76131 Karlsruhe,
Germany  Chen Zhou  Institute of Theoretical Informatics, Karlsruhe Institute
of Technology, Am Fasanengarten 5, 76131 Karlsruhe, Germany  Chen Shao
Institute of Theoretical Informatics, Karlsruhe Institute of Technology, Am
Fasanengarten 5, 76131 Karlsruhe, Germany  Houssam Metni  Institute of
Theoretical Informatics, Karlsruhe Institute of Technology, Am Fasanengarten
5, 76131 Karlsruhe, Germany  ECPM, Université de Strasbourg, 25 Rue Becquerel,
67087 Strasbourg, France  Clint van Hoesel  Institute of Theoretical
Informatics, Karlsruhe Institute of Technology, Am Fasanengarten 5, 76131
Karlsruhe, Germany  Department of Applied Physics, Eindhoven University of
Technology, Groene Loper 19, 5612 AP Eindhoven, The Netherlands  Henrik
Schopmans  Institute of Theoretical Informatics, Karlsruhe Institute of
Technology, Am Fasanengarten 5, 76131 Karlsruhe, Germany  Institute of
Nanotechnology, Karlsruhe Institute of Technology, Hermann-von-Helmholtz-Platz
1, 76344 Eggenstein-Leopoldshafen, Germany  Timo Sommer  Institute of
Theoretical Informatics, Karlsruhe Institute of Technology, Am Fasanengarten
5, 76131 Karlsruhe, Germany  Institute for Theory of Condensed Matter,
Karlsruhe Institute of Technology, Wolfgang-Gaede-Str. 1, 76131 Karlsruhe,
Germany  Pascal Friederich  Institute of Theoretical Informatics, Karlsruhe
Institute of Technology, Am Fasanengarten 5, 76131 Karlsruhe, Germany
Institute of Nanotechnology, Karlsruhe Institute of Technology, Hermann-von-
Helmholtz-Platz 1, 76344 Eggenstein-Leopoldshafen, Germany

###### Abstract

Machine learning plays an increasingly important role in many areas of
chemistry and materials science, e.g. to predict materials properties, to
accelerate simulations, design new materials, and to predict synthesis routes
of new materials. Graph neural networks (GNNs) are one of the fastest growing
classes of machine learning models. They are of particular relevance for
chemistry and materials science, as they directly work on a graph or
structural representation of molecules and materials and therefore have full
access to all relevant information required to characterize materials. In this
review article, we provide an overview of the basic principles of GNNs, widely
used datasets and state-of-the-art architectures, followed by a discussion of
a wide range of recent applications of GNNs in chemistry and materials
science, and concluding with a road-map for the further development and
application of GNNs.

##  1 Introduction

Data science and machine learning have become an integral part of natural
sciences, discussed as the fourth pillar in science, next to experiment,
theory, and simulation 1. Machine learning methods are increasingly applied in
all steps of the materials development cycle, from finding initial candidate
materials using property prediction 2, 3, database screening 4, 5 or even
inverse materials design 6, 7, over the detailed analysis of materials in
machine learning accelerated simulations 8, 9, to the prediction of synthesis
conditions 10, 11 and automated experimental data analysis 12, 13 and
experimental planning 14. Machine learning models applied in chemistry and
materials science cover a wide spectrum of methods, ranging from classical
machine learning models such as decision tree ensembles to modern deep
learning methods such as convolutional neural networks 15 and sequence models
16 originally developed for challenges in computer vision and natural language
processing.

A recent addition to the toolbox of machine learning models for chemistry and
materials science are graph neural networks (GNNs), which operate on graph-
structured data and have strong ties to the field of geometric deep learning
17, 18, 19. Aside from research on social and citation networks as well as
knowledge graphs, chemistry has been one of the main drivers in the
development of GNNs 20, 21. Graph neural networks can be interpreted as the
generalization of convolutional neural networks to irregular-shaped graph
structures. While other machine learning methods, e.g. convolutional neural
networks are at the peak of publication activity, GNNs are still rising
exponentially, with hundreds of papers per year since 2019 (see Figure 1).
Their architecture allows them to directly work on natural input
representations of molecules and materials, which are chemical graphs of atoms
and bonds, or even 3D structures or point clouds of atoms. Therefore, GNNs
have access to a complete representation of materials on the atomic level 22,
with a lot of flexibility to incorporate physical laws 23, as well as
phenomena on larger scales, such as doping and disorder. Using that
information, GNNs can learn internal materials representations that are useful
and informative for specific tasks such as the prediction of given materials’
properties. Therefore, GNNs can complement or even replace hand-crafted
feature representations which were and are widely used in the context of
natural sciences in general. A similar trend toward representation learning
methods has also been observed in other application areas during the last
years, where end-to-end trainable models show a systematic advantage over
traditional feature-based methods 24. However, despite promising recent
developments toward higher sample efficiency 25, 26, this often comes at the
cost of higher data requirements, potentially limiting the applicability of
existing GNNs to applications where large amounts of data are available.
Overall, GNNs outperformed conventional machine learning models in predicting
molecular properties throughout the last years 22, 27, 28. While GNNs are not
as widely applied (yet) in materials science as they are in chemistry, there
are advantages and the potential to outperform other machine learning methods
and thus boost virtual materials design and materials science in general,
which will be discussed in this article.

In Section 2, we will introduce the general formalism of GNNs and discuss the
way they transform the atomic structure of materials and molecules and use it
to predict materials’ properties. We will present and compare state-of-the-art
architectures and benchmark datasets, as well as summarize initial efforts
towards inverse materials design based on GNNs. Section 3 covers a wide range
of current application areas but also open challenges for GNNs in chemistry
and materials science. Section 4 concludes with a perspective on necessary and
expected future developments and so far unused potential of GNNs in materials
science.

##  2 Graph neural networks in materials science and chemistry

###  2.1 Basic principles

In the most general sense, graphs are used to describe abstract structures
consisting of entities or objects represented as vertices (or nodes) and their
connections, called edges. Formally, a graph is a tuple $G=(V,E)$ of a set of
vertices $v\in V$ and a set edges $e_{v,w}=(v,w)\in E$, which defines the
connection between vertices. Potential tasks that can be solved using graph
neural networks (GNNs) include classification or regression of graph
properties on graph level (molecular property prediction), node level
(classification of members, i.e. nodes, of a social graph), or edge level
(prediction of relations, i.e. edges, between objects in a scene graph). In
materials science and chemistry, most tasks involve graph-level predictions,
which will be the focus of this paper.

The concept of graphs is used in mathematical chemistry to represent the
structure of compounds. The molecular structure is represented by an
undirected graph, where nodes correspond to atoms and edges correspond to
chemical bonds. In fact, chemical graphs were first considered as early as in
1874 29 and their idea traces back further 30, which may place them even
before the advent of the term graph in modern graph theory 31. The description
of molecules as graphs can also be transferred to solid-state materials, even
though bonds might not be uniquely defined in crystals, and the exact three-
dimensional arrangement of atoms plays a more decisive role.

Since their proposal 17, 18, 19, GNNs have become a popular machine learning
method for processing irregularly shaped data encoded as graphs. They can be
seen as an alternative to approaches, where predefined feature representations
of molecules or materials are used as input to conventional machine learning
models such as densely connected neural networks, random forest models, or
Gaussian process regression models. In the case of GNNs, the full molecular
structure or even geometry is used as input and the GNN itself learns
informative molecular representations to predict given target properties. Due
to their popularity and wide applicability, a large number of different GNN
architectures have been proposed 32, 33, 18, 20, 34, 21. While the exact
architecture type can notably differ, ranging from the initially proposed
recursive GNNs 18 to spectral neural filters 35, 34 and finally to spatial or
convolutional GNNs 33, most GNNs designed for chemistry and materials science
can be summarized under the framework of Message Passing Graph Neural Networks
(MPNN) as suggested by Gilmer et al. 21. In this section, we give an overview
of ideas of the message passing framework and discuss how learned graph- or
node-level embeddings can be used for materials property prediction.

For MPNNs, associated node or edge information (e.g. atom and bond types) is
commonly provided by node attributes $h_{v}^{0}\in\mathbb{R}^{d}$ and edge
attributes $h_{e}^{0}\in\mathbb{R}^{c}$. Details about feature and structure
representations are discussed in Section 2.2. Using node and edge features in
combination with the graph’s structure, GNNs are capable of deriving a node-
level embedding of the graph, i.e. learned vectors representing each atom
including its individual chemical environment. This is done in the so-called
message passing phase, in which node information is propagated in form of
messages $m_{v}$ through edges to neighboring nodes. The embedding of each
node is then updated based on all incoming messages. The locality of the
message passing is sought to be alleviated by repeating the message passing
phase $t=1\dots K$ times, in principle allowing information to travel longer
distances, i.e. within the K-hop neighborhood. In practice, however,
information from long-range dependencies can be distorted in node bottlenecks,
referred to as over-squashing36, or be washed out, leaving indistinguishable
representations of neighboring nodes, known as over-smoothing37. Note that for
typical (not fully linear) molecules and crystal unit cells with $n$ atoms,
only approximately $\log n$ message passing steps are required to pass
information to all other atoms. The information processing is facilitated by
the learnable functions $U_{t}(\cdot)$ for node update and $M_{t}(\cdot)$ for
the message generation. Finally, in the readout phase, a graph-level embedding
$y$ is obtained by pooling node embeddings of the entire graph via a
parametric readout function $R(\cdot)$. The final representation of the graph
is used for training both regression and classification tasks. In summary, the
MPNN scheme reads 21:

$\displaystyle m^{t+1}_{v}$ $\displaystyle=\sum_{w\in
N(v)}M_{t}(h^{t}_{v},h^{t}_{w},e_{vw})$ (1) $\displaystyle h^{t+1}_{v}$
$\displaystyle=U_{t}(h^{t}_{v},m^{t+1}_{v})$ (2) $\displaystyle y$
$\displaystyle=R(\\{h^{K}_{v}|v\in G\\})\;,$ (3)

were $N(v)=\\{u\in V|(v,u)\in E\\}$ denotes the set of neighbors of node $v$.
Note that readout and aggregation can be in principle any mathematical
operation that is permutation-invariant, e.g. a sum, mean or maximum operation
similar to Equation 1 or learnable such as the Set2Set encoder proposed by
Vinyals et al. 38, which was originally used for the readout $R(\cdot)$. The
learnable functions are mostly neural networks and eventually determine the
performance characteristics of the GNN, both in prediction accuracy and
computational cost. Figure 1b shows a schematic of the message passing scheme
for the example of a molecular graph. Message passing can also be understood
as a convolution operation running over each node in a graph. Different
extensions and modifications of the message passing schemes are discussed in
Section 2.3 and include edge updates 39, skip connections 40, and geometric
information 26, 41, 42.

A main open research question of GNNs revolves around their limited expressive
performance for specific tasks 43 and how GNNs compare with Weisfeiler-Lehman
hierarchy for graph isomorphism testing 44, 45. With regard to this topic,
there are many promising extensions to GNNs proposed in literature, such as
hypergraph representations 46, 47, universal equivariant models 48 or higher-
order graph networks 49. Furthermore, the challenge of over-smoothing due to
commonly used aggregation functions 37, transfer and multitask learning 50, as
well as training (in)stability 51 are subject of current research.

Figure 1: a) Analysis of the keywords "convolutional neural network", "graph
neural network", and "deep learning" based on Google Trends (March 2022) b)
Schematic depiction of the message passing operation for molecules and
crystalline materials. c) QM9 benchmark. Mean absolute error of the prediction
of internal, HOMO and LUMO energies for different GNN models since 2017.

###  2.2 Structure representation

Many graph networks directly use the chemical graph as input, representing
both molecules 21 and inorganic compounds 52, 24, and offering advantages over
compositional or fixed-sized vector representations in terms of flexibility
and scalability. Consequently, GNNs can be applied for tasks such as drug
design or material screening 53, which require knowledge about functional
groups, scaffolds 54 or the full chemical structure and its topology. In
molecular applications, the chemical graph is often extracted from SMILES
codes and augmented with features that can be readily obtained from
cheminformatics software such as RDkit 55 or OpenBabel 56. Common features for
atoms, bonds, and the molecule itself are listed in Table 1. Besides hand-
crafted input features, learned embeddings of molecules and materials
motivated by word embedding techniques in natural language processing have
been explored which can be used for downstream tasks 47, 57, 58, 59. For
specific tasks in chemistry, the connectivity of atoms in molecules (i.e. the
molecular graph) contains sufficient and complete information to predict given
molecular properties which do not depend on the exact geometry. Geometry or
stereochemical information can be taken into account e.g. in form of
additional edge features representing the distance between atoms 60. In
contrast to that, in materials applications, atom connectivity is not well
defined in most cases (apart from e.g. covalently linked frameworks) and
graphs have to be extracted from crystal structures based on distance
heuristics.

Graph-level | Attributes | Description
---|---|---
nodes | atom-type | type of atoms (one-hot)
| chirality | R or S (one-hot or null)
| degree | number of covalent bonds (one-hot)
| radical | number of radical electrons (integer)
| hybridization | sp, sp2, sp${}^{3}\dots$ (one-hot)
| aromaticity | part of an aromatic system (binary)
| charge | formal charge (integer)
edges | bond-type | single, double, $\dots$ (one-hot)
| conjugation | is conjugated (binary)
| ring | bond is part of a ring (binary)
| stereo | None, Any, Z, E (one-hot)
graph | weight | average atomic weight (float)
| bonds | average bonds per atom (float)
Table 1: Table of typical (molecular) graph features used in literature 53,
61. They can be further combined with geometric features 60.

The sole chemical graph and its connectivity are often not sufficient to
accurately predict quantum-mechanical or electronic-structure properties 62
that strongly depend on the exact molecular geometry, even though ground-state
or equilibrium geometries can in principle be inferred from the molecular
graph alone. In tasks that intrinsically involve geometric dependencies, e.g.
predicting the potential energy surface of molecules and materials, it becomes
obvious that geometric information is required. The representation of
positional and geometric information to learn quantum properties has been
explored among others in the work of Lilienfeld et al. 63 and Behler et al. 64
and lead to a large variety of descriptors. Some examples of descriptors are
atomic centered symmetry functions (ACSF) 65, 66, angular Fourier series (AFS)
67, the smooth overlap of atomic orbitals (SOAP) 67, partial radial
distribution function (PRDF) 68, many-body tensor (MBTR) 69, Spectral London
Axilrod-Teller-Muto (SLATM) 70 and the Faber–Christensen–Huang–Lilienfeld
(FCHL) 71 representation. Many of those descriptors expand geometric
information into symmetry or basis functions. The resulting vector
representation is typically used as input for conventional machine learning
models such as neural networks or Gaussian Processes. Geometric information
can also be used for node or edge representations in graph neural networks.
Graph networks have been adopting distances 60, bond 72 and even dihedral
angles 26, 73, motivated by the comparison to force fields 74. Angles or
distances are similarly expanded into Gaussian-like 22, radial 23 and
spherical Fourier-Bessel functions 27. Although architectures such as the
Behler-Parinello (BP) neural network potentials 8 or SchNet 22 are not
strictly graph networks in terms of the chemical graph, and often do not refer
to themselves as such, they can be summarized within the term geometric deep
learning 75, 76.

Under the term geometric deep learning, architectures and descriptors are
summarized that focus on manifolds, geometric data or structured data 77, 78.
This includes the work on 3D point clouds 79, which aims at learning
segmentation and object detection of a large number of 3D points. In the case
of PointNet++ 80 a graph is constructed which reduces the point set from
learned descriptors using the points’ features. Commonly, adjacency matrices
are defined by using distance cutoffs between points in 3D clouds, while edges
carry explicit information about distances between nodes, i.e. points. Graph
pooling or coarsening algorithms 81, 82 that reduce the input representation
and condense structure information are also promising for GNNs to tackle
larger molecules such as proteins or polymers.

Eventually, the representation of materials for graph networks can be
structural or geometric but must follow certain symmetry considerations 83,
84. For example molecules without external fields have rotational and
translation symmetries. If they are incorporated into the model and its
representation, less data is required and overall performance can be improved.
This concept can be extended to equivariant representations 85, 86, which are
equivariant under translation, rotation, and permutation operations, and which
can enable the prediction of tensorial properties 28.

For solid crystals and periodic structures, the periodicity and space group
symmetries are additional symmetries to be added to the representation for
GNNs. Periodic extensions of the crystal graph 52, 2 of the unit cell have
been introduced 60 and their representation builds on the idea of a k-point
mesh of Monkhorst-Pack grid points to sample the Brillouin zone 87.

###  2.3 State-of-the-art architectures and benchmarks

Different architectures have been proposed in the literature to improve the
performance of GNNs on typical tasks arising in chemistry and materials
sciences. Table 2 shows a list of popular benchmark datasets for different
materials classes, i.e. molecules or crystals, and respective supervised
tasks, i.e. regression or classification. While some datasets contain
experimental data, the largest datasets typically use computational methods to
generate labels. Most datasets in this table can be downloaded from data
collections such as TUDatasets 88 and MoleculeNet 89. While a large variety of
datasets and tasks exist for chemistry, there are only a few large datasets
for materials, limited to crystalline structures. Recent datasets were
constructed by filtering the Materials Project (MP) 90 and Open Quantum
Materials Database (OQMD) 91 for specific targets such as electronic band-gap
or formation energy while removing incomplete data 87.

Table 2: Table of common benchmark datasets for graph learning tasks. Note that this list is not complete and merely serves as an overview of different sizes and supervised learning tasks, which is either regression (R) or classification (C).  Molecules | Size | Tasks | Type | Description
---|---|---|---|---
QM7 92 | 7,165 | 1 | R | DFT quantum calculations
QM7b 93 | 7,211 | 13 | R | DFT quantum calculations
QM9 62 | 133,885 | 12 | R | DFT quantum calculations
PDBBind 94 | 23,496 | 1 | R | protein binding affinity
MD17 95, 96 |  $>$ 100,000 |  $\geq$1 | R | molecular dynamics trajectories
FreeSolv 97 | 643 | 1 | R | solvation free energy
Lipop 89 | 4,200 | 1 | R | lipophilicity
Tox21 89 | 8,014 | 12 | C | qualitative toxicity measurement
ToxCast 98 | 8,615 | 617 | C | qualitative toxicity measurement
BBBP 99 | 2,053 | 1 | C | blood-brain barrier penetration
HIV 89 | 41,913 | 1 | C | inhibition to virus HIV
SIDER 100, 101 | 1,427 | 27 | C | adverse drug reaction
Crystals | Size | Tasks | Type | Description
MP 90 |  $\sim$ 144,595 |  $\geq$1 | R,C | Materials Project (MP)
OQMD 91 |  $\sim$ 1,022,603 |  $\geq$1 | R,C | Open Quantum Materials Database
OC20 102 |  $\sim$ 133,934,018 |  $\geq$1 | R | Open Catalyst Project

In Section 2.1, the message passing framework for GNNs has been illustrated.
Here, we will discuss modified and extended GNN models, which are relevant for
materials science and chemistry. However, listing all graph network
architectures would be beyond the scope of this review.

Some of the earliest work on neural networks for molecular graphs dates back
to the 90s and 2000s, without explicitly referring to the term graph neural
network 8, 32. In 2017, a graph convolutional network was proposed by Kipf et
al. 33 for semi-supervised learning, which can be interpreted as a first-order
approximation of spectral graph convolutions 35, 34. The addition of more
complicated node aggregation functions such as gated recurrent units 103 or
long short-term memories 104 has been employed by GraphSAGE for inductive
learning 105. For graph embedding tasks, a state- or super-node 21, which is
connected to all nodes, extends the message passing framework to help extract
global graph information, in addition to the final node aggregation step. A
message passing neural network (MPNN) with edge features capturing bond
information was applied to molecular graphs 21 and crystal graphs 52. A
variant of the original MPNN involves directed edge embeddings and message
passing between edges in D-MPNN 39. Known from models in natural language
processing 106, masked self-attention layers which attend over the node’s
neighborhood have been suggested for graph attention networks 107 and used
explicitly for molecules in Attentive Fingerprint models 53.

Besides graph models which focus on the chemical graph, there is a large group
of models explicitly designed for learning quantum properties. They commonly
take atomic numbers and positions as input and train on data derived from
(approximate) solutions of the steady-state Schrödinger equation. A popular
benchmark dataset is the QM9 dataset 62 with 13 quantum properties of small
molecules with up to nine atoms apart from hydrogen. The improvement of graph
networks on QM9 property prediction over the past few years is highlighted in
Figure 1c. Among the first graph networks that reached chemical accuracy on
QM9 is SchNet 22, which makes use of convolutional filters for inter-atomic
distances and applies skip connections between node updates. One improvement
to SchNet was to update the positional features along the graph edges as seen
in 108. The application of GNNs to crystals using geometric information has
been explored by MEGNET 60, which further leverages global properties such as
temperature, which is of importance for solid-state crystalline systems. The
potential energy of molecules depends on bond angles and therefore, in DimeNet
27, 72, edge embedding uses messages passing steps from atomic triplets and
bond pairs in order to incorporate angular features. This formalism has been
adopted in other recent GNNs 109, 110 and can be further extended to include
dihedral (or torsion) angles 111, 26, 112.

For explicit angle plus node information in directed edge updates as in
DimeNet 27, 72, the message passing essentially operates on higher order paths
73 or $k$-pairs of atoms 49. This is unfeasible for fully connected larger
graphs because the number of multi-node interactions that need to be computed
is dramatically increasing. To reduce the computational costs, models like
MXMNet 110 make use of multiplex graphs, which selectively consider only
specific edges when going to higher order pathways for calculating bond angles
109.

Note that the GNNs mentioned previously are invariant to the translation and
rotation of the molecules throughout space. Recently, equivariant GNNs have
been proposed 28, 85, 42, 113, which transform equivariant under symmetry
operations of its (positional) input, meaning the GNN’s features or its output
undergoes the same operations as well. This enables efficient consideration of
angular information between atoms 28 without higher-order pathways and enables
the prediction of tensor features of general rank 114, 113.

Further adapted message passing steps allow for the determination of the
molecular orbitals 115, 116, 117, 2. Molecular orbital interactions can in
turn be used for improving the prediction performance 2. Lastly, the mapping
of atoms to non-Euclidean space such as in the proposed hyperbolic GNNs 118
can lead to gains in representational efficiency. For a more in-depth
discussion of graph variants 119 and graph taxonomy 120 that goes beyond Table
3, we refer to more general articles about GNNs 121, 122, e.g Zhou et al. 120
and Wu et al. 123.

With regard to the QM9 benchmark in Figure 1c some models have slightly lower
performance for the total energy but can be superior in other QM9 properties
or achieve similar results with much less computational effort. Some other
factors that complicate a stringent comparison are differences in train-test
splits, cleaning steps, e.g. of ill-converged molecules in QM9, multi-task vs.
single task settings, where a separate model for each QM9 target is usually
trained 60, and differences in used loss metrics (a mean absolute error loss
was found to yield lower overall test errors 23 than the mean squared error
loss used in previous models 22, although the mean absolute error is typically
given as a benchmark reference). It has to be noted that hyperparameters are
generally very important and are often not exhaustively optimized for GNNs
which can cause differences in performance apart from the model architecture
124, 125, 126.

Table 3: Table of GNN models are sorted by categories. It is to note, that some models can also fall into more than one category and that this table can not list all relevant models but only give a grouping of a few popular models mentioned in the text. There is no strict distinction between categories spatial convolution, message passing, and 3D geometric message passing.  Categories | GNN architectures
---|---
Spectral convolution | LanczosNet 127, SpecConv 128, 129, CayleyNet 130, ChebNet 34
Spatial convolution and | GCN 33, 123-GNN or k-GNN 49, R-GCN 131, GIN 45
| PatchySan 132, C-SGEL 133, GraphSAGE 105, OGCNN 2
| CGCNN and iCGCNN 134, 135
Message passing | MPNN 21, D-MPNN 39, MPSN 43, MGN 32
| G-MPNN and MPNN-R 136, PMP 137
3D geometric message passing | MEGNET 60, DimeNet 27, 72, PhysNet 23, MolNet 83, 138
| PointNet++ 80, MXMNet 110, SchNet 22, 108, ForceNet 139,
| GemNet 26, Geomol 112, ALIGNN 109 and ALIGNN-d 111,
| GNNFF 140, GeoCGNN 87, SphereNet 141, HGCN 118
Attention and graph transformer | GAT 107, GATv2 142, MAT 50, AGNN 143, AMPNN 144
| CapsGNN 145, RGAT 146, AttentiveFP 53, AGN 51
| GACNN 147, MEGAN 148, SAMPN 149, HamNet 150
Equivariant models | PAiNN 28, NequIP 25, TFN 151, CGNet 152,
| Cormorant 113, LieConv 42, EGNN 85, UNiTE 114
| SEGNN 153, SE(3)T 154, CNN-G 155, 156, 157
Graph pooling | DiffPool 158, EdgePool 159, gPool 160
| HGP-SL82, SAGPool 81, iPool 161, EigenPool 162
Generative graph models | CGVAE 163, JT-VAE 164, GCPN 165, GeoMol 112
| GraphGAN 166, DCGAN 167

###  2.4 GNNs in generative models and reinforcement learning

An important challenge in materials science is inverse materials design,
aiming to generate new materials or molecules that possess required properties
and fulfill specific criteria168. GNN-based generative methods have been
suggested to deal with this challenge in the context of chemistry, e.g. for
drug discovery 169 and retrosynthesis 148. In most cases, only the chemical
structure of molecules is generated, i.e. the connectivity of the molecular
graph, without additional information on specific 3D geometry.

Initial graph generative models were designed to generate graphs based on
simplified theoretical assumptions, such as the random Erdös-Renyi (ER) model
170, and improvements thereof using small-world approaches 171 or the
Kronecker graph model 172. While these traditional approaches attempt to model
real-world graphs, they are based on several assumptions about the nature of
graphs generated and are thus inflexible for many data-related applications.
Machine learning approaches for graph generation are promising because they
can directly learn to generate realistic graphs from the distribution of
observed data while accommodating goal-directed tasks such as property
optimization. Examples include variational autoencoders (VAEs) 173, generative
adversarial networks (GANs) 174, reinforcement learning 175, recurrent neural
networks (RNNs) 176 and flow-based generative models 177, 178, 179.

Several architectures of VAEs have been developed to work with different types
of input data, such as images 180, text-based data 181, 182, or graphs 183,
184. Kipf et al. introduced a variational graph auto-encoder (VGAE) to learn
latent representations of undirected graphs and applied it to a link
prediction task in citation networks 183. Liu et al. introduced a Constrained
Graph Variational Autoencoder (CGVAE), in which node type information is
learned from and added to the latent vectors 163. Starting from a set of these
latent node representations, CGVAE iteratively forms valid molecules following
hard valency constraints derived from the explicit node types. Jin et al.
introduced Junction Tree VAE (JT-VAE) to work directly on molecular graphs and
achieved an improvement over baseline methods in molecular design tasks 164.
The JT-VAE approach encodes and decodes molecules in two steps: First, tree-
structured objects called junction trees are generated which represent trees
of molecular subgraphs and their arrangements. GNN-based encoders and decoders
are used to generate latent embeddings of the junction trees. In parallel,
molecular graph embeddings are generated using GNNs, and junction trees are
decoded into molecular representations 21. These are then encoded to a latent
vector and decoded back to their original representations using graph and
tree-based encoders and decoders. While this scheme works well for molecules,
it is hard to adapt for crystalline materials, where the graphs are less tree-
like and the definition of scaffolds is not as straightforward.

GANs have shown promising results in a number of fields, such as image 185 or
sequence 186 generation, and have also been applied to 3D grid representations
of materials 167 and graphs166. De Cao et al. introduced MolGAN 187 as a
framework for generating molecular graphs using GNNs. The generator learns to
directly output the graph’s representation. While the standard GAN loss forces
the generator to generate molecules following a particular prior distribution,
the authors add a reinforcement learning (RL) objective to generate molecules
with optimized properties. The generation of invalid molecules is avoided by
the assignment of zero reward to them. While direct prediction of outputs is
appealing in methods using VAEs or GANs, they usually predict outputs of
small, fixed sizes. Therefore, another branch of deep graph generative models
employs sequential decision-making procedures to overcome these limitations.

You et al. identify three challenging aspects when graphs are generated
directly 176. First, as these methods need to model the adjacency matrix in
some form, the output size grows as $\mathcal{O}(n^{2})$ for graphs with a
maximum number of $n$ nodes. This is especially undesirable for large and
sparse graphs as the model dedicates much of its capacity to learn which nodes
are not connected. Second, graph isomorphism complicates the calculation of
reconstruction losses for VAEs and usually involves expensive graph matching
procedures. Third, VAEs assume the outputs, e.g., the entries of an adjacency
matrix, to be i.i.d., which is not true in practice. As a solution, the
authors propose GraphRNN, a framework in which graphs are represented as
sequences of node and edge additions. By using recurrent networks, graph
constituents are generated conditioned on previous additions, thus taking into
account the history of modifications. Another sequential graph generation
scheme was proposed by Li et al. 188. In this framework, the generation
process is decomposed into modular steps, e.g., whether to add a node or which
nodes to connect. Each module is a fully-connected neural network modeling
probabilities of executing particular types of modifications.

You et al. suggested a purely RL-based approach based on Graph Convolutional
Policy Networks (GCPN) 165 (see Figure 2b). In this setting, the agent
explores the chemical space and generates new molecular graphs by modifying
the starting molecules according to the reward function, representing the
molecular property to optimize. Atance et al. recently introduced a similar RL
approach based on Gated GNNs189, outperforming other GNN-based approaches in
molecular graph generation tasks190. Another sequential approach based on
conditional graph generative models has been used by Li et al. on drug design
tasks191. The previous two works inspired recent molecular graph generation
frameworks such as GraphINVENT190.

More recently, attention has turned to generative modeling based on
normalizing flows (NFs)192, capable of modeling complex target distributions
by directly mapping them to simpler ones using revertible bijective
transformations. NFs potentially offer significant improvements in the field
of graph generation because they allow exact evaluation and inference of the
probability distribution. This approach has been applied in a number of
molecular tasks193, 169, 194, 195.

Overall, graph generative models have been extensively applied for molecular
materials and stayed up to date with recent developments in the field of graph
generation. However, these remain under-explored for crystalline materials,
mainly due to graph representation challenges196. While finding such a
reliable graph representation is still an open question168 and will likely
remain case-specific, we believe that using generative models based on GNNs is
a promising research direction in inverse design, especially given current
breakthroughs such as normalizing flows.

##  3 Applications

After introducing the basic principles of GNNs as well as selected GNN
architectures and benchmark datasets, we will provide a structured overview of
GNN applications in chemistry and materials science. GNNs were successfully
applied to a rich variety of different challenges, ranging from property
prediction of molecules and materials over accelerated atomistic simulations
to predicting reactivity and synthesis routes of molecules and materials.
While other machine learning models such as densely connected neural networks
were successfully applied to these tasks as well, state-of-the-art GNNs in
many cases currently outperform other models. However, there exists a range of
open challenges, including data requirements and data efficiency, as well as a
lack of fully GNN-based generative models for molecular and materials design.

Figure 2: Overview of GNN applications for molecules and materials. a)
prediction of ADMET properties (adapted with permission from Feiberg et
al.197, Copyright 2020 American Chemical Society), GNNs accounting for
environment effects of molecules (reproduced from Ref. 198 with permission
from the Royal Society of Chemistry.), GNNs to predict the toxicity of
molecules for bees (this illustration was published in Ref. 147, Copyright
Elsevier), b) RL-based approach for inverse molecular design based on Graph
Convolutional Policy Networks (GCPN) (adapted from Ref. 165), c) template-free
retrosynthesis (adapted from Ref. 199), d) transferable excited states
dynamics (reproduced from Ref. 200 with permission from Springer Nature),
coarse graining (reproduced from Ref. 201 with permission from the Royal
Society of Chemistry), e) explainable GNNs (adapted from Ref. 202), f) Crystal
GNN to predict methane adsorption volumes in metal organic frameworks (MOFs)
(this illustration was published in 203, Copyright Elsevier), doped structures
(this illustration was published in 204, Copyright Elsevier), point defects
(adapted with permission from Frey et al.205, Copyright 2020 American Chemical
Society) g) reactions of Al2O3 surface in contact with HF gas (reproduced from
Ref. 140 with permission from Springer Nature), GNNs to predict
magnetostriction of polycrystalline systems (reproduced from Ref. 206 with
permission from Springer Nature), h) a GNN classifier to predict if a system
is in a liquid or a glassy phase only by the positions of the atoms
(reproduced from Ref. 207 with permission from the Royal Society of
Chemistry).

###  3.1 Molecular systems

Some of the first applications of GNNs and probably also one of the main
driving forces for the ongoing development of GNN models are challenges in the
area of molecular chemistry. Most prevalent is the task of predicting
molecular properties, i.e. a regression or classification task which is
challenging to solve with conventional machine learning models, as they
typically require predefined molecular representations (e.g. molecular feature
vectors or molecular fingerprints) which are informative for the label to
predict. GNNs have access to the full chemical graph or even molecular
geometry and learn to extract feature representations, which yields an
advantage over other ML models. Compared to domain knowledge-informed feature
representations combined with conventional ML models, e.g. Gaussian process
regression, GNNs often have comparably high data requirements but outperform
conventional models when enough data is available. Once trained, accurate ML
models can then be used to accelerate the high-throughput virtual screening of
molecules 208, 209 to find promising candidate molecules for many different
applications. However, property prediction is not the only application of
GNNs. They were also successfully applied to provide trainable interatomic
potentials to accelerate costly ab initio molecular dynamics simulations, as
well as to predict the outcome of chemical reactions and synthetic routes.

####  3.1.1 Molecular property Prediction

Among the most relevant molecular properties in the area of drug discovery are
the ADMET (absorption, distribution, metabolism, exclusion, and toxicity)
properties of potential drug-like molecules (see Figure 2a) 197, 210, 211,
212. A review on GNNs for drug design can be found in Xiong et al. 213. In
recent years, one application focus were Covid 19 related challenges, where
GNNs were used for e.g. finding new drug candidates 214 or detecting
infections in medical images 215, 216. Similar methods are also applicable to
other challenges in drug design and medicine.

Furthermore, GNNs were applied to predict electronic and optical properties of
molecules. For many applications such as organic electronics, organic
photovoltaics, and organic light-emitting diodes, the energy of the highest
occupied molecular orbital (HOMO), the lowest unoccupied MO (LUMO) and the
optical gap are of high importance for the device efficiency. These properties
can therefore be found in numerous databases 62, 217, 93, 218, 219, 220, 221.
Related properties include (transition) dipole moment 62, 222, ionization
potential and electron affinity 93. In devices, these properties often depend
on the molecular environment, which can be modeled and accounted for with GNNs
(see Figure 2a) 198. For certain applications such as opto-electronic
materials 218, 219, GNNs have been proposed to complement scalar molecular
properties with spectroscopic properties 111.

The aforementioned properties are often determined using computationally
expensive simulation methods. Cheaper, mostly semi-empirical methods can
provide fast but potentially less accurate estimates. GNNs have been used to
represent Kohn-Sham wavefunctions and energy levels in a minimal basis
representation 223, as well as for delta-learning from semi-empirical methods
to DFT computed molecular properties 221. For applications where not enough
data is available to train GNNs, the representations learned by GNNs on large
generic datasets can also be transferred to supervised tasks with little data
20, 224, 225, where they are used as input for other ML models such as
gradient-boosted models 226.

Further application areas of GNNs spread across all application domains of
molecular materials, including the prediction of toxicity of molecules to bees
(see Figure 2a),147 determining the quality of a material for fuel ignition
227 and the classification of different phases of materials, in particular
water 228. It should be noted that another review paper on molecular property
prediction utilizing GNNs exists by Wieder et al. 119. In many application
areas, tools such as GNNExplainer 229 are used to validate and analyze GNN
predictions, e.g. in the prediction of scents230 and for porous organic
cages231.

####  3.1.2 Dynamics simulations

Molecular dynamics simulations are an important tool for understanding dynamic
processes and mechanisms on a microscopic level in various areas of chemistry,
biology, and materials science. Besides the prediction of equilibrium
properties of molecules and materials, they also offer the possibility to
simulate excited states and non-equilibrium dynamics, as well as slow
processes and rare events.

In molecular dynamics simulations, total energy and forces are needed in every
time step to propagate the system. Computationally demanding ab initio methods
that calculate the energy and forces of a particular atomic configuration of a
system at every time step are therefore often too costly. ML methods can
replace ab initio calculations to speed up simulations while ideally retaining
their accuracy 9. Therefore, long and highly accurate MD simulations can be
performed based on ML potentials, which have not been possible using classical
force fields nor ab initio methods. GNNs are perfectly suited for this task,
as atomic forces depend on the (local) atomic environment and global
aggregation is not needed.

The concept of integrating ML models in atomistic simulations was demonstrated
multiple times using for example SchNet 22, PhysNet 23, DimeNet 27 or
DimeNet$++$ 72. However, there are several open challenges that need to be
overcome in order to move to larger systems, longer time scales, higher data
efficiency, better generalization and transferability, and eventually more
accurate and realistic applications. Usually, ML models learn the potential
energy surface and calculate forces using derivatives of the energy
predictions. This ensures that energy predictions and forces are consistent.
Since only forces are required in MD simulations, architectures are being
developed in which these forces are predicted directly - so that the costly
derivative calculations are omitted. In the GNN framework (GNNFF 140), a
message passing step builds upon an embedding step in which node and edge
features include atom type and interatomic distances respectively. Force
magnitudes per atom are then calculated from the sum of the forces of the
neighboring atoms. Evaluation on the ISO17 database reveals higher force
accuracies compared to SchNet while being 1.6$\times$ faster. The approach is
also shown to be scalable to larger systems. Due to the direct prediction of
the forces, the energy of the system is however not necessarily preserved,
making the model not suitable to predict energy-related properties.

ForceNet 139 is based on an encoder-decoder architecture and tries to
completely capture 3D geometry through a specific design of the message
passing structure. In contrast to models such as SchNet and DimeNet, ForceNet
encodes physical information without constraining the model architecture to
enforce physical invariances. Instead, physics-based data augmentation is
performed on the data level to achieve rotational invariance. The evaluation
was performed on the OC20 dataset which contains DFT calculated energies and
per-atom forces of more than 200 million large atomic structures (20-200
atoms) including non-equilibrium structures from optimization trajectories.
The resulting mean absolute force errors are comparable to DimeNet++, while
being faster in training and prediction.

A promising approach to encoding more physical information about a system is
the design of equivariant models. Models that are based on equivariant message
passing, e.g. PaiNN 28, NequIP 25, NewtonNet 232, are shown to significantly
increase data efficiency and predictive performance compared to models that
are based on invariant convolutions. The Neural Equivariant Interatomic
Potential (NequIP 25) predicts both energy and forces utilizing
E(3)-equivariant convolutions over geometric tensors. Evaluated on the MD17
data set its accuracy exceeds those of existing models while needing up to
three orders of magnitude less training data. Due to its data efficiency, it
was also used with coupled cluster (CCSD(T)) based training data, showing
great potential for applications where a prediction accuracy beyond DFT is
needed. In order to further improve data efficiency, GNNFF140 and NewtonNet
232 introduce more physical priors in the form of latent force vectors as well
as operators containing physical information. This leads to good prediction
accuracy with higher computational efficiency at only 1-10% of the training
data compared to other models.

To improve generalization, hybrid models such as SpookyNet 233 explicitly
include electronic structure information such as total charge or spin state,
not included in most ML potentials, by applying self-attention in a
transformer architecture. Empirical augmentations to include non-local and
long-range contributions such as electrostatic interaction and dispersion
improve transferability and at the same time enable interpretability 233.

GNN for large-scale MD simulations. Many applications require models that
scale to large system sizes. For example, simulations of biological systems,
e.g. to study protein dynamics or drug binding mechanisms involve orders of
magnitude more atoms than many other applications, while configuration changes
occur on much longer timescales ($10^{-3}-10^{3}\mathrm{s}$) than a typical MD
timestep ($10^{-15}\mathrm{s}$). One way to address this enormous challenge is
the development of models in a QM/MM-inspired approach, where only a small
relevant subsystem, e.g. a reaction site, needs to be simulated at ab initio
accuracy, while the rest of the system can be described using classical force
fields.

GNNs also have the potential to support (adaptive) coarse-graining methods.
They were shown to be useful in mapping atoms of a molecule into coarse-
grained groups needed for large-scale simulations. The Deep Supervised Graph
Partitioning Model (DSGPM) 201 treats mapping operators as a graph
segmentation problem. It predicted a coarse-grained mapping nearly
indistinguishable from human annotations (see Figure 2d). Furthermore, the ML
framework by Wang et al. 234, which generates a coarse-grained force field,
was further improved by Husic et al. 235 replacing manual input features with
a GNN architecture making the models transferable across different molecular
systems such as small proteins.

Excited states dynamics. GNNs were also shown as a very promising tool to
tackle the challenging task of simulating excited state dynamics of complex
systems 236. Unlike ground-state dynamics, multiple potential energy surfaces
as well as their crossings and couplings must be considered, leading to a
higher dimensionality and complexity of the problem. Furthermore, even the
generation of reliable training data using quantum mechanical calculations is
challenging.

Westermayr et al. developed SchNarc 237 for photodynamics simulations by
adapting SchNet for excited states potentials, forces, and couplings,
combining it with the MD framework SHARC (surface hopping including arbitrary
couplings). While SchNarc is molecule specific and was applied to two
compounds, CH2NH${}_{2}^{+}$ and CSH2, Axelrod et al. developed the first
transferable excited state potential (see Figure 2d)200. The diabatic
artificial neural network (DANN) is based on PaiNN combined with a physics-
informed diabatic model for photodynamic simulations for virtual screening.
The resulting ML potential is transferable among azobenzene derivatives and
estimated to be multiple orders of magnitude faster than the underlying
quantum mechanical calculation method, even considering computational effort
for transfer which required additional training data.

####  3.1.3 Reaction Prediction and Retrosynthesis

While reliable property prediction and simulation methods are crucial for
virtual molecular design, synthesis is often one of the main bottlenecks in
the overall development process of new molecules. Progress in reaction
prediction and retrosynthesis, i.e. the prediction of a reaction outcome and
the design of synthetic routes for a desired product, can help to accelerate
and also automate 238 the development of new molecules. However, the two
problems are still considered challenging due to the vast chemical space and
currently require skills and experience from well-trained chemists. Therefore,
many machine learning algorithms, e.g. seq2seq models and transformers, have
been proposed for synthesis prediction and retrosynthesis, aiming at reducing
manual effort. In many cases, molecules are embedded as SMILES codes, and
reaction predictions as well as retrosynthesis predictions are formulated as
natural language processing tasks 239, 240, 241, 242. Furthermore,
fingerprints are widely used as structure encodings and neural networks
trained on them are able to predict the most probable transformations for
given molecules 243, 244. GNN based graph embeddings have recently attracted
growing attention, due to the natural representation of molecular structures
as graphs.

Prediction of reactivity with GNNs has been formalized into reaction center
identification and graph edit tasks. Jin et al. developed a GNN-based approach
to scoring each pair of connected atoms with bond change likelihood, followed
by the selection of bonds with the highest score and transformation to
potential products using a Weisfeiler-Lehman Difference Network (WLDN) 245,
246. The WLN architecture is also adopted by Struble and coworkers 247 to
perform multitask prediction of aromatic C-H functionalization reactions with
site selectivity, while Guan et al. predict reaction outcomes with regio-
selectivity through combining molecule representations learned by the WLN with
on-the-fly calculated quantum mechanical descriptors 248. In 2020, Nikitin et
al. introduced a strategy to treat reaction prediction as a node
classification problem, where the role of each node in the reactant graphs is
predicted by a GNN with a self-attention mechanism and pseudo-global nodes
249.

Furthermore, predicting reaction products as the result of a sequence of graph
editing actions on the reactant molecules has also been investigated. One
example is the work by Do et al. where graph editing actions are predicted by
a reinforcement learning algorithm based on reactant and reagent molecule
representations generated by GNNs 250. In 2019, Bradshaw et al. developed a
generative model that uses GNNs to predict a series of electron movements for
reactions, through which the products and reaction mechanisms are predicted at
the same time 251. Apart from product prediction, GNNs are also employed to
predict important reaction properties including bond dissociation energies
252, 253, transition states 254, and activation energies 255.

For the retrosynthesis task, recent studies can be divided into template-based
and template-free approaches. The former matches the graph embedding of the
product molecule to a large number of reaction templates, which determine bond
changes and thus predict possible reactants, while the latter bypass the
templates and directly modifies input graphs to generate synthetic precursors.
Examples of template-based retrosynthesis prediction include the work by Dai
and coworkers, who predict the probability distribution of reaction templates
to be fitted to the reaction outcomes by a Conditional Graph Logic Network
(GLN) 256. The reactants are generated by the GLN as the result of a
probability prediction given the reaction outcome and the selected template.
Another example by Ishida et al. uses GCNs to classify single retrosynthesis
steps into reaction templates, where integrated gradients are applied to
visualize atom contributions toward the GCN prediction 257. More recently,
Chen and coworkers proposed a framework based on MPNNs with a global attention
mechanism to predict the reaction center of a target molecule, as well as the
corresponding “local” reaction template based on atoms and bonds in the
reaction center, which is then applied to generate reactants 258.

For template-free retrosynthesis, Yan et al. use an edge-enhanced graph
attention network to locate the reaction center in product molecules, which
are transformed through bond dissociation into molecular fragments called
synthons 199. The synthon graphs are converted to SMILES and expanded to
reactants by a sequence-to-sequence algorithm. At the same time, Somnath et
al. proposed an approach that uses GNNs to predict a series of graph edits
that transform a product into synthons that are further completed into
reactants with leaving groups predicted by another GNN (see Figure 2d) 259. A
similar strategy is adopted by Shi et al., where synthons are expanded to
reactants by a variational graph translation 260.

In 2021, Sacha et al. proposed a model that formulates both retrosynthesis and
forward synthesis tasks as a sequence of graph edit actions that are predicted
by an encoder-decoder structure constructed by stacking graph convolutional
layers 148.

###  3.2 Crystalline and solid state systems

Compared to molecules, crystal structures and solid-state materials have some
additional challenges, such as periodic boundary conditions for crystals and
multiple kinds of disorder, either in form of perturbations in the crystal
structure itself or in the (lack of) long-range ordering of atoms. We will
present different recent applications of GNNs in solid state systems, from
predicting global properties of crystal structures with and without disorder,
over driving atomistic simulations, to the design of new materials aided by
materials synthesis prediction, active learning, and inverse design. Table 4
gives an overview of datasets used in the following applications. We will also
discuss approaches in which the trained GNN models have been analyzed to
enable further insight into specific scientific questions, which is often
equally important as accurate numerical predictions.

####  3.2.1 Materials property prediction and materials design

GNNs can be used to predict a multitude of materials properties, ranging from
formation energies 261, 24, 135, 262, 263, 264 and synthesizability prediction
265 over band-gaps 266, 267, 268 and other functional properties to mechanical
properties 134, 60. The most straightforward application of property
prediction models is the screening of large databases of crystals, where
exhaustive screening using conventional simulation techniques (e.g. DFT) is
often non-feasible. Screening using GNNs only requires labels from simulation
or experiment for the training set while providing fast predictions on the
rest of the database - provided the model generalizes well.

Wang et al. use a crystal GNN to predict methane adsorption volumes in metal-
organic frameworks (MOFs) 203. The pooling function leverages domain knowledge
by additionally including structural properties (see Figure 2f), e.g. the pore
limiting diameter, achieving better performance than previous work 269. They
apply the model to the screening of a hypothetical MOF database by Wilmer et
al. 270 and find several high-performing candidates 203. Gu et al. use an
ensemble of attention crystal GNNs to screen alloy catalysts for CO2 reduction
271. Only bulk-relaxed structures without the adsorbate (e.g. from the
Materials Project (MP) 90 database) are needed as input, removing the costly
DFT relaxation from the screening process. The performance approaches that of
base-line models trained on fully relaxed structures 271.

An interesting application of GNNs was presented by Goodall et al., who use
GNNs for representation learning of chemical formulas 272. The chemical
formula is represented as a dense weighted graph, in which each node
corresponds to a chemical element weighted by the fraction of this element in
the chemical formula. This graph representation was used to train a GNN in a
supervised way to obtain a mapping from the chemical formula to an embedding.
It was demonstrated that this representation has a better sample efficiency
than other structure agnostic approaches.

Schmidt et al. use a crystal graph attention network for stability-screening
of non-relaxed hypothetical materials, predicting the convex hull distance 24.
Only graph distances are included, making the model usable for hypothetical
materials where the exact coordinates might be unknown. A vast database is
used, combining MP, AFLOW 273 and group-internal datapoints. Transfer learning
then allows the screening of 15 million tetragonal perovskites of composition
ABCD2. Dai et al. use GNNs to predict magnetostriction of polycrystalline
systems (see Figure 2g) 206. Instead of using GNNs to model the atoms in the
unit cell, individual grains and their interactions to neighboring grains are
represented by nodes and edges in the GNN. This shows that GNNs can be used on
different scales to predict materials properties.

Typically, screening of materials databases using ML models is only possible
if training data is available which covers the target materials distribution,
i.e. which adequately allows generalization to the rest of the database.
Active learning offers a promising solution when the available data does not
fulfill this criterion. Based on uncertainty quantification, the training
dataset can then be iteratively extended to include previously uncertain data
points and thereby efficiently explore the chemical space. Lu et al. use
active learning to search for 2D ferromagnetic materials with a custom crystal
graph multilayer descriptor in a small dataset 274. Further applications of
active learning based GNNs in materials science are promising and can be
expected in the future.

####  3.2.2 Disordered systems and defects

Disordered materials are a wide and particularly interesting field of study,
as the effects of disorder can influence or even dominate material properties.
Disorder ranges from weak disorder (defects, dislocations, grain boundaries)
to strong disorder (e.g. glassy systems and inhomogeneous systems such as
porous materials) and includes topological/structural disorder, orientational
disorder (spins, dipoles), or substitutional disorder (chemical doping,
compositional disorder) 275. Due to its inherent multi-scale nature, disorder
poses severe challenges not only to materials modeling and simulation, but
also to materials synthesis, characterization, and fundamental description.
Graph neural networks can be extended to model various forms of disordered
systems and predict their local and global properties. In contrast to quantum
mechanical methods such as DFT, crystalline systems with substitutional
disorder can be modeled using larger, more representative unit cells or by
defining atoms/nodes with mixed occupations. Furthermore, amorphous systems
can be modeled explicitly by using representative simulation boxes and
aggregating over learned atom representations characteristic of the properties
of the amorphous system. In the following, we will present seminal work in
that direction.

One important challenge for machine learning models is how to represent
substitutional disorder such as doping and fractional occupancies in the
crystal graph. Chen et al. addressed this question with a MEGNet model with
trainable 16-dimensional embeddings for each atom which were pre-trained on a
dataset with only ordered structures 276. The pre-trained embeddings were used
to represent doped crystal sites by doing a weighted average of the elemental
embeddings, weighted with (logarithmic or appropriately scaled) occupancies of
the elements on the crystal sites. Additionally, it was also demonstrated how
to perform multi-fidelity training with band gaps of different levels of DFT
by encoding the DFT level in the global state of the MEGNet. Similarly, Wang
et al. have trained a crystal GNN model on predicting the phase stability of
materials (see Figure 2f), classifying materials as metal or insulator and
predicting the band gap of semiconductors 204. They train a crystal GNN on
undoped crystal structures and use this trained model to predict the
properties of doped crystal structures, which they validate with DFT
calculations. They find that the crystal GNN and the DFT usually predict the
same trend, despite the crystal GNN not being trained on doped crystal
structures 204. Frey et al. use a crystal GNN model to predicted the
properties of 2D materials with point defects (see Figure 2f) 205. However,
the crystal GNN is only used to screen the properties of ordered structures to
find promising host structures. The properties of the disordered structures
were then partially calculated with DFT to train a random forest model on
physics-based features which was used to screen additional structures. Another
study predicted the properties of bcc iron structures with point defects 277.

A very special application of GNNs is glassy systems. There have been works
that predicted the properties of glasses 278 and which used inverse design to
find glasses with new properties 279. Swanson et al. trained a classifier on
predicting if a system is in a liquid or a glassy phase only by the positions
of the atoms (see Figure 2h) 207. They verified that a GNN was significantly
better than a CNN for this task and used self-attention to interpret the
reasoning of the trained classifier. Consequently, the trained GNN was
interpreted to find previously unknown relationships. Three simple and
previously unknown formulas were developed, which describe the reasoning of
the classifier and which could be used to differentiate the two phases.
Initial work on predicting polymer properties using GNNs was based on learning
representations of single monomers 280 or of unit cells of crystalline
polymers 281.

####  3.2.3 Aiding simulations

Analogous to atomistic dynamics simulations of molecular systems (see Section
3.1.2, GNNs can also be used to simulate the dynamic behavior of crystals,
i.e. to predict potential energy, forces, and partial charges, in order to
drive molecular dynamics simulations. To predict forces for MD of crystals,
most approaches use conventional ML methods 282, but there are first examples
that use GNNs 140, 283. Raza et al. use a GNN with node-level readouts to
predict partial charges of MOFs for molecular simulations of gas adsorption
284. For each atom, a probability distribution of charge is learned and
optimized globally using the maximum likelihood principle under a charge
neutrality constraint. Park et al. predict forces for MD directly using a GNN
with edge-level readouts that are used to predict the magnitude of the force
between atoms 140 (see also Section 3.1). This avoids calculating the
derivative of the potential energy and speeds up the simulation, yielding good
accuracy in chemical reactions occurring at the surface of a test system of
Al2O3 in contact with HF gas (see Figure 2g). Also, good scaling accuracy
transferring a model trained on a 1x2x1 supercell to a 1x2x2 supercell
(Li7P3S11) is achieved. This approach allows simulations of a larger scale
than possible using ab initio methods with similar accuracy and might make it
possible to simulate mesoscopic phenomena such as grain boundaries in the
future.

####  3.2.4 Solid state synthesis prediction

While predicting synthesizability overall 265 is important in the design
process of new materials, predicting the products of solid state synthesis is
a challenging task and interesting application area of GNNs. Malik et al. have
developed a pipeline to predict the major outcome of solid-state reactions
285. In this pipeline, GNNs were used to represent the precursors of a
reaction in a set-like fashion as a dense graph while a long short-term memory
layer is used to represent the series of processing actions in the reaction.
Current limitations in the availability of systematic (experimental) synthesis
data, including reaction conditions 11, hinder further progress in this area.
The use of electronic lab notebooks and repositories specifically designed for
chemistry and materials science has a large potential to alleviate that
challenge 286, 287.

####  3.2.5 Periodic graph generation

A major requirement for the graph-based inverse design of crystal structures
is the possibility to generate new periodic graphs based on a vector
representation. Recently, this problem has been addressed with a new
architecture called PGD-VAE 196, a variational autoencoder capable of
generating new periodic graph structures. Another work using a VAE focuses on
predicting stable crystal structures using GNNs 288. New structures are
generated in a three-step approach: First, the composition, lattice, and
number of atoms of a sampled point in the latent space are decoded using an
MLP. Afterward, a random structure with these properties is assembled and a
GNN trained on moving atoms to their equilibrium positions is used to generate
stable structures. Despite these promising efforts, the design of solids state
materials and crystal structures using GNNs is only at the beginning. Multiple
challenges need to be solved to achieve wide applicability and transferability
of methods to multiple classes of materials, ranging from densely packed
crystals over porous materials to amorphous systems need to be solved.

Table 4: Selected application-specific datasets for solid-state systems. Dataset |  Size
---|---
AFLOW 273 \- calculated properties of materials |  $>$ 3,400,000
Inorganic Crystal Structure Database (ICSD)289 \- extensive and well curated experimental database |  $\approx$ 210,000
Pure carbon and C-H-N-O structures at different pressures290 |  $\approx$ 200,000
Materials Project 90 \- calculated properties of materials |  $\approx$ 145,000
Hypothetical MOF database 270 |  137,953
NREL Materials Database (NRELMatDB)291 \- computational materials database focused on renewable energy applications |  $\approx$ 60,000
CO and H surface binding energy dataset 292 |  $\approx$ 40,000
Inorganic materials synthesis recipes293 |  19,488
Perovskite structures and energies294 |  18,928
CoRE MOF database 295 \- Experimental MOF database |  $>$ 14,000
bcc iron structures with energies and various kinds of defects296 |  12,193
MOF methane adsorption volume of CoRE MOFs (from GCMC) 203 |  10,102
Elemental boron structures with energies297 |  5,038
Computational 2D Materials Database (C2DB)298, 299 |  $\approx$ 4000
DDEC MOF point charges 300 |  2,932

##  4 Outlook

GNNs became a very versatile and important tool very quickly. A lot has been
achieved already, not only in terms of fundamental method development tailor-
made for requirements of materials science and chemistry (see Section 2), but
also in terms of applications (see Section 3), where GNNs were successfully
applied for materials simulation, screening, design, and synthesis. However,
there is a wide range of open questions and challenges which need to be solved
in order to leverage the full potential of GNNs in materials science and
chemistry.

Despite the growing amount of research and publications on GNN model
development and application, GNN models remain expert tools, i.e. they are
comparably hard to implement, adapt, transfer, train, and apply to given
datasets and applications. Libraries such as PyTorch Geometric 301, DGL 302 or
the Keras based KGCNN 303 implement a selection of state-of-the-art GNN layers
and models. However, the use of such libraries in many cases requires expert
knowledge which goes beyond the knowledge needed for the successful
application of more established machine learning models. One of the reasons
for this is certainly related to the fast development of new GNN variants
which are partially hard to compare. The widespread use and further
development of common benchmarks (e.g. the QM9 dataset) as well as open
communication of models and training workflows, including open-source code,
inspired by common practice in the machine learning community, are essential
for a reliable quantitative comparison of future developments. However, non-
optimal hyperparameters and the (prohibitively) high computational cost of
hyperparameter optimization is an open challenge. A further challenge
hindering the transfer of state-of-the-art models to applications is the
discrepancy in the data distribution between widely used benchmark datasets
(e.g. QM9) and actually relevant datasets, which typically contain larger
molecules (i.e. larger graphs), which are more diverse, e.g. in terms of the
chemical elements used and in the size variance, and which sample the chemical
or materials space less densely then QM9.304

Generally, there is more research and model development activity for molecules
and chemistry, compared to materials science. As a consequence, the transfer
of existing GNNs to new application areas (e.g. porous or amorphous materials)
might be challenging and requires further development of GNN models. To some
extent, this can be attributed to a lack of generic benchmark datasets for
crystalline materials and, even more importantly, (partially) disordered
structures and amorphous materials. OC20 is one of the few examples of
datasets covering both materials science and chemistry 102. Nonetheless, there
are many promising application areas of GNNs for solid-state materials. In
contrast to quantum mechanical simulation methods such as DFT, GNNs are
particularly suited for representing disordered systems and predicting their
properties, e.g. systems with compositional disorder such as high-entropy
alloys. However, there is a lack of large datasets of disordered systems,
particularly labeled datasets with consistent measured or simulated
properties. Overall, screening of (hypothetical) materials spaces with
predictive ML models allows the rapid discovery of new materials with
optimized target properties, especially when combined with transfer and active
learning to improve data efficiency and reduce computational cost.

The first steps towards GNN-based generative models exist, but there are many
open challenges of reliability and transferability. Despite the high potential
of GNN-based algorithms for the inverse design of new molecules and materials,
convincing success stories of ML-designed materials are rare. To make
generative models more application relevant, new methods are required that
e.g. allow to include constraints in the design process, in the simplest case
symmetries of generated molecules and materials, or in more complex scenarios
additional (empirical or analytical) objectives such as synthesizability. A
large step in that direction is new representations, not only for organic
molecules but also for (3D) materials 305, 306, 307.

Finally, more research on the explainability and interpretability of GNNs and
machine learning in general will help to better understand underlying
correlations and eventually causal relations in large and complex datasets,
eventually contributing to scientific understanding and progress 308, 309,
310.

## References

  * 1 von Lilienfeld, O. A.  Introducing machine learning: Science and technology.  _Machine Learning: Science and Technology_ 1, 010201 (2020).  URL https://doi.org/10.1088/2632-2153/ab6d5d.
  * 2 Karamad, M. _et al._ Orbital graph convolutional neural network for material property prediction.  _Phys. Rev. Materials_ 4, 093801 (2020).  URL https://link.aps.org/doi/10.1103/PhysRevMaterials.4.093801.
  * 3 Vamathevan, J. _et al._ Applications of machine learning in drug discovery and development.  _Nature reviews Drug discovery_ 18, 463–477 (2019).
  * 4 Jorissen, R. N. & Gilson, M. K.  Virtual screening of molecular databases using a support vector machine.  _Journal of Chemical Information and Modeling_ 45, 549–561 (2005).  URL https://doi.org/10.1021/ci049641u.  PMID: 15921445, https://doi.org/10.1021/ci049641u.
  * 5 Ekins, S. _et al._ Exploiting machine learning for end-to-end drug discovery and development.  _Nature materials_ 18, 435–441 (2019).
  * 6 Noh, J., Gu, G. H., Kim, S. & Jung, Y.  Machine-enabled inverse design of inorganic solid materials: promises and challenges.  _Chem. Sci._ 11, 4871–4881 (2020).  URL http://dx.doi.org/10.1039/D0SC00594K.
  * 7 Zunger, A.  Inverse design in search of materials with target functionalities.  _Nature Reviews Chemistry_ 2, 1–16 (2018).
  * 8 Behler, J. & Parrinello, M.  Generalized neural-network representation of high-dimensional potential-energy surfaces.  _Phys. Rev. Lett._ 98, 146401 (2007).  URL https://link.aps.org/doi/10.1103/PhysRevLett.98.146401.
  * 9 Friederich, P., Häse, F., Proppe, J. & Aspuru-Guzik, A.  Machine-learned potentials for next-generation matter simulations.  _Nature Materials_ 20, 750–761 (2021).
  * 10 Shields, B. J. _et al._ Bayesian reaction optimization as a tool for chemical synthesis.  _Nature_ 590, 89–96 (2021).
  * 11 Luo, Y. _et al._ Mof synthesis prediction enabled by automatic data mining and machine learning.  _Angewandte Chemie International Edition_ e202200242 (2021).
  * 12 Kalinin, S. V. _et al._ Machine learning in scanning transmission electron microscopy.  _Nature Reviews Methods Primers_ 2, 1–28 (2022).
  * 13 Velasco, L. _et al._ Phase–property diagrams for multicomponent oxide systems toward materials libraries.  _Advanced Materials_ 33, 2102301 (2021).
  * 14 Häse, F., Roch, L. M. & Aspuru-Guzik, A.  Next-generation experimentation with self-driving laboratories.  _Trends in Chemistry_ 1, 282–291 (2019).  URL https://www.sciencedirect.com/science/article/pii/S258959741930019X.
  * 15 LeCun, Y., Kavukcuoglu, K. & Farabet, C.  Convolutional networks and applications in vision.  In _Proceedings of 2010 IEEE International Symposium on Circuits and Systems_ , 253–256 (2010).
  * 16 Schwaller, P. _et al._ Molecular transformer: a model for uncertainty-calibrated chemical reaction prediction.  _ACS central science_ 5, 1572–1583 (2019).
  * 17 Sperduti, A. & Starita, A.  Supervised neural networks for the classification of structures.  _IEEE Transactions on Neural Networks_ 8, 714–735 (1997).
  * 18 Gori, M., Monfardini, G. & Scarselli, F.  A new model for learning in graph domains.  In _Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005._ , vol. 2, 729–734 vol. 2 (2005).
  * 19 Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M. & Monfardini, G.  The graph neural network model.  _IEEE Transactions on Neural Networks_ 20, 61–80 (2009).
  * 20 Duvenaud, D. _et al._ Convolutional networks on graphs for learning molecular fingerprints.  In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M. & Garnett, R. (eds.) _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , 2224–2232 (2015).  URL https://proceedings.neurips.cc/paper/2015/hash/f9be311e65d81a9ad8150a60844bb94c-Abstract.html.
  * 21 Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O. & Dahl, G. E.  Neural message passing for quantum chemistry.  In _Proceedings of the 34th International Conference on Machine Learning - Volume 70_ , ICML’17, 1263–1272 (JMLR.org, 2017).
  * 22 Schütt, K. T., Sauceda, H. E., Kindermans, P.-J., Tkatchenko, A. & Müller, K.-R.  SchNet – A deep learning architecture for molecules and materials.  _The Journal of Chemical Physics_ 148, 241722 (2018).  URL https://aip.scitation.org/doi/full/10.1063/1.5019779.  Publisher: American Institute of Physics.
  * 23 Unke, O. T. & Meuwly, M.  Physnet: A neural network for predicting energies, forces, dipole moments, and partial charges.  _Journal of Chemical Theory and Computation_ 15, 3678–3693 (2019).  URL http://dx.doi.org/10.1021/acs.jctc.9b00181.
  * 24 Schmidt, J., Pettersson, L., Verdozzi, C., Botti, S. & Marques, M. A. L.  Crystal graph attention networks for the prediction of stable materials.  _Science Advances_ 7, eabi7948 (2021).  URL https://www.science.org/doi/abs/10.1126/sciadv.abi7948.  https://www.science.org/doi/pdf/10.1126/sciadv.abi7948.
  * 25 Batzner, S. _et al._ E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials.  _Nature communications_ 13, 1–11 (2022).
  * 26 Klicpera, J., Becker, F. & Günnemann, S.  Gemnet: Universal directional graph neural networks for molecules.  In Beygelzimer, A., Dauphin, Y., Liang, P. & Vaughan, J. W. (eds.) _Advances in Neural Information Processing Systems_ (2021).  URL https://openreview.net/forum?id=HS_sOaxS9K-.
  * 27 Klicpera, J., Groß, J. & Günnemann, S.  Directional message passing for molecular graphs.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_ (OpenReview.net, 2020).  URL https://openreview.net/forum?id=B1eWbxStPH.
  * 28 Schütt, K., Unke, O. & Gastegger, M.  Equivariant message passing for the prediction of tensorial properties and molecular spectra.  In _International Conference on Machine Learning_ , 9377–9388 (PMLR, 2021).
  * 29 F.R.S., P. C.  Lvii. on the mathematical theory of isomers.  _The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science_ 47, 444–447 (1874).  URL https://doi.org/10.1080/14786447408641058.  https://doi.org/10.1080/14786447408641058.
  * 30 Bonchev, D. & Rouvray, D. H.  _Chemical Graph Theory: Introduction and Fundamentals_ (Routledge, London, 1991).  URL https://doi.org/10.1201/9781315139104.
  * 31 Biggs, N., Lloyd, E. K. & Wilson, R. J.  _Graph Theory, 1736-1936_ (Clarendon Press, USA, 1986).
  * 32 Merkwirth, C. & Lengauer, T.  Automatic generation of complementary descriptors with molecular graph networks.  _Journal of Chemical Information and Modeling_ 45, 1159–1168 (2005).  URL https://doi.org/10.1021/ci049613b.  PMID: 16180893, https://doi.org/10.1021/ci049613b.
  * 33 Kipf, T. N. & Welling, M.  Semi-supervised classification with graph convolutional networks.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_ (OpenReview.net, 2017).  URL https://openreview.net/forum?id=SJU4ayYgl.
  * 34 Defferrard, M., Bresson, X. & Vandergheynst, P.  Convolutional neural networks on graphs with fast localized spectral filtering.  In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I. & Garnett, R. (eds.) _Advances in Neural Information Processing Systems_ , vol. 29 (Curran Associates, Inc., 2016).  URL https://proceedings.neurips.cc/paper/2016/file/04df4d434d481c5bb723be1b6df1ee65-Paper.pdf.
  * 35 Estrach, J. B., Zaremba, W., Szlam, A. & LeCun, Y.  Spectral networks and deep locally connected networks on graphs.  In _2nd international conference on learning representations, ICLR_ , vol. 2014 (2014).
  * 36 Alon, U. & Yahav, E.  On the bottleneck of graph neural networks and its practical implications.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_ (OpenReview.net, 2021).  URL https://openreview.net/forum?id=i80OPhOCVH2.
  * 37 Li, Q., Han, Z. & Wu, X.  Deeper insights into graph convolutional networks for semi-supervised learning.  In McIlraith, S. A. & Weinberger, K. Q. (eds.) _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , 3538–3545 (AAAI Press, 2018).  URL https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16098.
  * 38 Vinyals, O., Bengio, S. & Kudlur, M.  Order matters: Sequence to sequence for sets.  In Bengio, Y. & LeCun, Y. (eds.) _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_ (2016).  URL http://arxiv.org/abs/1511.06391.
  * 39 Yang, K. _et al._ Analyzing learned molecular representations for property prediction.  _Journal of Chemical Information and Modeling_ 59, 3370–3388 (2019).  URL https://doi.org/10.1021/acs.jcim.9b00237.  PMID: 31361484, https://doi.org/10.1021/acs.jcim.9b00237.
  * 40 Schütt, K. T. _et al._ SchNetPack: A deep learning toolbox for atomistic systems.  _Journal of Chemical Theory and Computation_ 15, 448–455 (2018).  URL https://doi.org/10.1021/acs.jctc.8b00908.
  * 41 Shui, Z. & Karypis, G.  Heterogeneous molecular graph neural networks for predicting molecule properties.  In Plant, C., Wang, H., Cuzzocrea, A., Zaniolo, C. & Wu, X. (eds.) _20th IEEE International Conference on Data Mining, ICDM 2020, Sorrento, Italy, November 17-20, 2020_ , 492–500 (IEEE, 2020).  URL https://doi.org/10.1109/ICDM50108.2020.00058.
  * 42 Finzi, M., Stanton, S., Izmailov, P. & Wilson, A. G.  Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data.  In III, H. D. & Singh, A. (eds.) _Proceedings of the 37th International Conference on Machine Learning_ , vol. 119 of _Proceedings of Machine Learning Research_ , 3165–3176 (PMLR, 2020).  URL https://proceedings.mlr.press/v119/finzi20a.html.
  * 43 Bodnar, C. _et al._ Weisfeiler and lehman go topological: Message passing simplicial networks.  In _ICLR 2021 Workshop on Geometrical and Topological Representation Learning_ (2021).  URL https://openreview.net/forum?id=RZgbB-O3w6Z.
  * 44 Maron, H., Ben-Hamu, H., Serviansky, H. & Lipman, Y.  Provably powerful graph networks.  In Wallach, H. _et al._ (eds.) _Advances in Neural Information Processing Systems_ , vol. 32 (Curran Associates, Inc., 2019).  URL https://proceedings.neurips.cc/paper/2019/file/bb04af0f7ecaee4aae62035497da1387-Paper.pdf.
  * 45 Xu, K., Hu, W., Leskovec, J. & Jegelka, S.  How powerful are graph neural networks?  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_ (OpenReview.net, 2019).  URL https://openreview.net/forum?id=ryGs6iA5Km.
  * 46 Feng, Y., You, H., Zhang, Z., Ji, R. & Gao, Y.  Hypergraph neural networks.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 33, 3558–3565 (2019).  URL https://ojs.aaai.org/index.php/AAAI/article/view/4235.
  * 47 Jo, J. _et al._ Edge representation learning with hypergraphs.  _Advances in Neural Information Processing Systems_ 34, 7534–7546 (2021).
  * 48 Dym, N. & Maron, H.  On the universality of rotation equivariant point cloud networks.  In _International Conference on Learning Representations_ (2021).  URL https://openreview.net/forum?id=6NFBvWlRXaG.
  * 49 Morris, C. _et al._ Weisfeiler and leman go neural: Higher-order graph neural networks.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 33, 4602–4609 (2019).  URL https://ojs.aaai.org/index.php/AAAI/article/view/4384.
  * 50 Maziarka, L. _et al._ Molecule attention transformer.  _CoRR_ abs/2002.08264 (2020).  URL https://arxiv.org/abs/2002.08264.  2002.08264.
  * 51 Chen, Y., Tang, X., Qi, X., Li, C.-G. & Xiao, R.  Learning graph normalization for graph neural networks.  _Neurocomputing_ 493, 613–625 (2022).  URL https://www.sciencedirect.com/science/article/pii/S0925231222000030.
  * 52 Xie, T. & Grossman, J. C.  Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties.  _Phys. Rev. Lett._ 120, 145301 (2018).  URL https://link.aps.org/doi/10.1103/PhysRevLett.120.145301.
  * 53 Xiong, Z. _et al._ Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism.  _Journal of Medicinal Chemistry_ 63, 8749–8760 (2020).  URL https://doi.org/10.1021/acs.jmedchem.9b00959.  PMID: 31408336, https://doi.org/10.1021/acs.jmedchem.9b00959.
  * 54 Kruger, F., Stiefl, N. & Landrum, G. A.  rdscaffoldnetwork: The scaffold network implementation in rdkit.  _Journal of Chemical Information and Modeling_ 60, 3331–3335 (2020).  URL https://doi.org/10.1021/acs.jcim.0c00296.  PMID: 32584031, https://doi.org/10.1021/acs.jcim.0c00296.
  * 55 RDKit: Open-source cheminformatics.  http://www.rdkit.org.  [Online; accessed 11-April-2013].
  * 56 O’Boyle, N. M. _et al._ Open Babel: An open chemical toolbox.  _Journal of Cheminformatics_ 3, 33 (2011).  URL https://doi.org/10.1186/1758-2946-3-33.
  * 57 Chen, Z.-H. _et al._ Prediction of drug–target interactions from multi-molecular network based on deep walk embedding model.  _Frontiers in Bioengineering and Biotechnology_ 8 (2020).  URL https://www.frontiersin.org/article/10.3389/fbioe.2020.00338.
  * 58 Perozzi, B., Al-Rfou, R. & Skiena, S.  Deepwalk: Online learning of social representations.  In _Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’14, 701–710 (Association for Computing Machinery, New York, NY, USA, 2014).  URL https://doi.org/10.1145/2623330.2623732.
  * 59 Grover, A. & Leskovec, J.  Node2vec: Scalable feature learning for networks.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’16, 855–864 (Association for Computing Machinery, New York, NY, USA, 2016).  URL https://doi.org/10.1145/2939672.2939754.
  * 60 Chen, C., Ye, W., Zuo, Y., Zheng, C. & Ong, S. P.  Graph networks as a universal machine learning framework for molecules and crystals.  _Chemistry of Materials_ 31, 3564–3572 (2019).
  * 61 Pocha, A., Danel, T., Podlewska, S., Tabor, J. & Maziarka, L.  Comparison of atom representations in graph neural networks for molecular property prediction.  In _International Joint Conference on Neural Networks, IJCNN 2021, Shenzhen, China, July 18-22, 2021_ , 1–8 (IEEE, 2021).  URL https://doi.org/10.1109/IJCNN52387.2021.9533698.
  * 62 Ramakrishnan, R., Dral, P. O., Rupp, M. & von Lilienfeld, O. A.  Quantum chemistry structures and properties of 134 kilo molecules.  _Scientific Data_ 1, 140022 (2014).  URL https://doi.org/10.1038/sdata.2014.22.
  * 63 von Lilienfeld, O. A., Müller, K.-R. & Tkatchenko, A.  Exploring chemical compound space with quantum-based machine learning.  _Nature Reviews Chemistry_ 4, 347–358 (2020).  URL https://doi.org/10.1038/s41570-020-0189-9.
  * 64 Behler, J.  Atom-centered symmetry functions for constructing high-dimensional neural network potentials.  _The Journal of Chemical Physics_ 134, 074106 (2011).  URL https://doi.org/10.1063/1.3553717.  https://doi.org/10.1063/1.3553717.
  * 65 Seko, A., Takahashi, A. & Tanaka, I.  Sparse representation for a potential energy surface.  _Phys. Rev. B_ 90, 024101 (2014).  URL https://link.aps.org/doi/10.1103/PhysRevB.90.024101.
  * 66 Behler, J.  Perspective: Machine learning potentials for atomistic simulations.  _The Journal of Chemical Physics_ 145, 170901 (2016).  URL https://doi.org/10.1063/1.4966192.  https://doi.org/10.1063/1.4966192.
  * 67 Bartók, A. P., Kondor, R. & Csányi, G.  On representing chemical environments.  _Phys. Rev. B_ 87, 184115 (2013).  URL https://link.aps.org/doi/10.1103/PhysRevB.87.184115.
  * 68 Schütt, K. T. _et al._ How to represent crystal structures for machine learning: Towards fast prediction of electronic properties.  _Phys. Rev. B_ 89, 205118 (2014).  URL https://link.aps.org/doi/10.1103/PhysRevB.89.205118.
  * 69 Huo, H. & Rupp, M.  Unified representation of molecules and crystals for machine learning (2018).  1704.06439.
  * 70 Huang, B. & von Lilienfeld, O. A.  Quantum machine learning using atom-in-molecule-based fragments selected on the fly.  _Nature Chemistry_ 12, 945–951 (2020).
  * 71 Christensen, A. S., Bratholm, L. A., Faber, F. A. & Anatole von Lilienfeld, O.  Fchl revisited: Faster and more accurate quantum machine learning.  _The Journal of Chemical Physics_ 152, 044107 (2020).  URL https://doi.org/10.1063/1.5126701.  https://doi.org/10.1063/1.5126701.
  * 72 Klicpera, J., Giri, S., Margraf, J. T. & Günnemann, S.  Fast and uncertainty-aware directional message passing for non-equilibrium molecules (2020).  2011.14115.
  * 73 Flam-Shepherd, D., Wu, T. C., Friederich, P. & Aspuru-Guzik, A.  Neural message passing on high order paths.  _Machine Learning: Science and Technology_ 2, 045009 (2021).  URL https://doi.org/10.1088/2632-2153/abf5b8.
  * 74 Pukrittayakamee, A. _et al._ Simultaneous fitting of a potential-energy surface and its corresponding force fields using feedforward neural networks.  _The Journal of Chemical Physics_ 130, 134101 (2009).  URL https://doi.org/10.1063/1.3095491.  https://doi.org/10.1063/1.3095491.
  * 75 Bronstein, M. M., Bruna, J., LeCun, Y., Szlam, A. & Vandergheynst, P.  Geometric deep learning: Going beyond euclidean data.  _IEEE Signal Processing Magazine_ 34, 18–42 (2017).
  * 76 Atz, K., Grisoni, F. & Schneider, G.  Geometric deep learning on molecular representations.  _Nature Machine Intelligence_ 3, 1023–1032 (2021).  URL https://doi.org/10.1038/s42256-021-00418-8.
  * 77 Cao, W., Yan, Z., He, Z. & He, Z.  A comprehensive survey on geometric deep learning.  _IEEE Access_ 8, 35929–35949 (2020).
  * 78 Monti, F. _et al._ Geometric deep learning on graphs and manifolds using mixture model cnns.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ (2017).
  * 79 Nguyen, A. & Le, B.  3d point cloud segmentation: A survey.  In _2013 6th IEEE Conference on Robotics, Automation and Mechatronics (RAM)_ , 225–230 (2013).
  * 80 Qi, C. R., Yi, L., Su, H. & Guibas, L. J.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space.  In Guyon, I. _et al._ (eds.) _Advances in Neural Information Processing Systems_ , vol. 30 (Curran Associates, Inc., 2017).  URL https://proceedings.neurips.cc/paper/2017/file/d8bf84be3800d12f74d8b05e9b89836f-Paper.pdf.
  * 81 Lee, J., Lee, I. & Kang, J.  Self-attention graph pooling.  In Chaudhuri, K. & Salakhutdinov, R. (eds.) _Proceedings of the 36th International Conference on Machine Learning_ , vol. 97 of _Proceedings of Machine Learning Research_ , 3734–3743 (PMLR, 2019).  URL https://proceedings.mlr.press/v97/lee19c.html.
  * 82 Zhang, Z. _et al._ Hierarchical graph pooling with structure learning.  _CoRR_ abs/1911.05954 (2019).  URL http://arxiv.org/abs/1911.05954.  1911.05954.
  * 83 Li, C. _et al._ 3dmol-net: Learn 3d molecular representation using adaptive graph convolutional network based on rotation invariance.  _IEEE Journal of Biomedical and Health Informatics_ 1–1 (2021).
  * 84 Montavon, G. _et al._ Learning invariant representations of molecules for atomization energy prediction.  In Pereira, F., Burges, C. J. C., Bottou, L. & Weinberger, K. Q. (eds.) _Advances in Neural Information Processing Systems_ , vol. 25 (Curran Associates, Inc., 2012).  URL https://proceedings.neurips.cc/paper/2012/file/115f89503138416a242f40fb7d7f338e-Paper.pdf.
  * 85 Satorras, V. G., Hoogeboom, E. & Welling, M.  E(n) equivariant graph neural networks.  In Meila, M. & Zhang, T. (eds.) _Proceedings of the 38th International Conference on Machine Learning_ , vol. 139 of _Proceedings of Machine Learning Research_ , 9323–9332 (PMLR, 2021).  URL https://proceedings.mlr.press/v139/satorras21a.html.
  * 86 Nigam, J., Willatt, M. J. & Ceriotti, M.  Equivariant representations for molecular hamiltonians and n-center atomic-scale properties.  _The Journal of Chemical Physics_ 156, 014115 (2022).  URL https://doi.org/10.1063/5.0072784.  https://doi.org/10.1063/5.0072784.
  * 87 Cheng, J., Zhang, C. & Dong, L.  A geometric-information-enhanced crystal graph network for predicting properties of materials.  _Communications Materials_ 2, 92 (2021).  URL https://doi.org/10.1038/s43246-021-00194-3.
  * 88 Morris, C. _et al._ Tudataset: A collection of benchmark datasets for learning with graphs.  _CoRR_ abs/2007.08663 (2020).  URL https://arxiv.org/abs/2007.08663.  2007.08663.
  * 89 Wu, Z. _et al._ Moleculenet: a benchmark for molecular machine learning.  _Chemical science_ 9, 513–530 (2018).
  * 90 Jain, A. _et al._ Commentary: The Materials Project: A materials genome approach to accelerating materials innovation.  _APL Materials_ 1, 011002 (2013).
  * 91 Kirklin, S. _et al._ The Open Quantum Materials Database (OQMD): assessing the accuracy of DFT formation energies.  _npj Computational Materials_ 1, 15010 (2015).  URL https://doi.org/10.1038/npjcompumats.2015.10.
  * 92 Blum, L. C. & Reymond, J.-L.  970 million druglike small molecules for virtual screening in the chemical universe database GDB-13.  _J. Am. Chem. Soc._ 131, 8732 (2009).
  * 93 Montavon, G. _et al._ Machine learning of molecular electronic properties in chemical compound space.  _New Journal of Physics_ 15, 095003 (2013).  URL http://stacks.iop.org/1367-2630/15/i=9/a=095003.
  * 94 Wang, R., Fang, X., Lu, Y., Yang, C.-Y. & Wang, S.  The PDBbind database: methodologies and updates.  _Journal of Medicinal Chemistry_ 48, 4111–4119 (2005).  URL https://doi.org/10.1021/jm048957q.
  * 95 Chmiela, S. _et al._ Machine learning of accurate energy-conserving molecular force fields.  _Science Advances_ 3, e1603015 (2017).  URL https://www.science.org/doi/abs/10.1126/sciadv.1603015.  https://www.science.org/doi/pdf/10.1126/sciadv.1603015.
  * 96 Chmiela, S., Sauceda, H. E., Müller, K.-R. & Tkatchenko, A.  Towards exact molecular dynamics simulations with machine-learned force fields.  _Nature Communications_ 9, 3887 (2018).  URL https://doi.org/10.1038/s41467-018-06169-2.
  * 97 Mobley, D. L. & Guthrie, J. P.  FreeSolv: a database of experimental and calculated hydration free energies, with input files.  _Journal of Computer-Aided Molecular Design_ 28, 711–720 (2014).  URL https://doi.org/10.1007/s10822-014-9747-x.
  * 98 Richard, A. M. _et al._ Toxcast chemical landscape: Paving the road to 21st century toxicology.  _Chemical Research in Toxicology_ 29, 1225–1251 (2016).  URL https://doi.org/10.1021/acs.chemrestox.6b00135.  PMID: 27367298, https://doi.org/10.1021/acs.chemrestox.6b00135.
  * 99 Martins, I. F., Teixeira, A. L., Pinheiro, L. & Falcao, A. O.  A bayesian approach to in silico blood-brain barrier penetration modeling.  _Journal of Chemical Information and Modeling_ 52, 1686–1697 (2012).  URL https://doi.org/10.1021/ci300124c.  PMID: 22612593, https://doi.org/10.1021/ci300124c.
  * 100 Kuhn, M., Letunic, I., Jensen, L. J. & Bork, P.  The SIDER database of drugs and side effects.  _Nucleic Acids Research_ 44, D1075–D1079 (2015).  URL https://doi.org/10.1093/nar/gkv1075.  https://academic.oup.com/nar/article-pdf/44/D1/D1075/16661270/gkv1075.pdf.
  * 101 Altae-Tran, H., Ramsundar, B., Pappu, A. S. & Pande, V.  Low data drug discovery with one-shot learning.  _ACS Central Science_ 3, 283–293 (2017).  URL https://doi.org/10.1021/acscentsci.6b00367.  PMID: 28470045, https://doi.org/10.1021/acscentsci.6b00367.
  * 102 Chanussot, L. _et al._ Open catalyst 2020 (oc20) dataset and community challenges.  _ACS Catalysis_ 11, 6059–6072 (2021).  URL https://doi.org/10.1021/acscatal.0c04525.  https://doi.org/10.1021/acscatal.0c04525.
  * 103 Cho, K., van Merrienboer, B., Bahdanau, D. & Bengio, Y.  On the properties of neural machine translation: Encoder-decoder approaches.  In Wu, D., Carpuat, M., Carreras, X. & Vecchi, E. M. (eds.) _Proceedings of SSST@EMNLP 2014, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, Doha, Qatar, 25 October 2014_ , 103–111 (Association for Computational Linguistics, 2014).  URL https://aclanthology.org/W14-4012/.
  * 104 Hochreiter, S. & Schmidhuber, J.  Long short-term memory.  _Neural Computation_ 9, 1735–1780 (1997).
  * 105 Hamilton, W., Ying, Z. & Leskovec, J.  Inductive representation learning on large graphs.  _Advances in neural information processing systems_ 30 (2017).
  * 106 Vaswani, A. _et al._ Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * 107 Velickovic, P. _et al._ Graph attention networks.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_ (OpenReview.net, 2018).  URL https://openreview.net/forum?id=rJXMpikCZ.
  * 108 Jørgensen, P. B., Jacobsen, K. W. & Schmidt, M. N.  Neural message passing with edge updates for predicting properties of molecules and materials (2018).  1806.03146.
  * 109 Choudhary, K. & DeCost, B.  Atomistic line graph neural network for improved materials property predictions.  _npj Computational Materials_ 7 (2021).  URL https://doi.org/10.1038/s41524-021-00650-1.
  * 110 Zhang, S., Liu, Y. & Xie, L.  Molecular mechanics-driven graph neural network with multiplex graph for molecular structures (2020).  2011.07457.
  * 111 Hsu, T. _et al._ Efficient, interpretable atomistic graph neural network representation for angle-dependent properties and its application to optical spectroscopy prediction (2021).  2109.11576.
  * 112 Ganea, O. _et al._ Geomol: Torsional geometric generation of molecular 3d conformer ensembles.  _Advances in Neural Information Processing Systems_ 34, 13757–13769 (2021).
  * 113 Anderson, B., Hy, T. S. & Kondor, R.  Cormorant: Covariant molecular neural networks.  _Advances in neural information processing systems_ 32 (2019).
  * 114 Qiao, Z. _et al._ Unite: Unitary n-body tensor equivariant network with applications to quantum chemistry (2021).  2105.14655.
  * 115 Schütt, K. T., Gastegger, M., Tkatchenko, A., Müller, K.-R. & Maurer, R. J.  Unifying machine learning and quantum chemistry with a deep neural network for molecular wavefunctions.  _Nature Communications_ 10, 5024 (2019).  URL https://doi.org/10.1038/s41467-019-12875-2.
  * 116 Qiao, Z., Welborn, M., Anandkumar, A., Manby, F. R. & Miller, T. F.  Orbnet: Deep learning for quantum chemistry using symmetry-adapted atomic-orbital features.  _The Journal of Chemical Physics_ 153, 124111 (2020).  URL https://aip.scitation.org/doi/full/10.1063/5.0021955.
  * 117 Unke, O. _et al._ Se (3)-equivariant prediction of molecular wavefunctions and electronic densities.  _Advances in Neural Information Processing Systems_ 34, 14434–14447 (2021).
  * 118 Liu, Q., Nickel, M. & Kiela, D.  Hyperbolic graph neural networks.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * 119 Wieder, O. _et al._ A compact review of molecular property prediction with graph neural networks.  _Drug Discovery Today: Technologies_ 37 (2020).  URL https://www.sciencedirect.com/science/article/pii/S1740674920300305.
  * 120 Zhou, J. _et al._ Graph neural networks: A review of methods and applications.  _AI Open_ 1, 57–81 (2020).  URL https://www.sciencedirect.com/science/article/pii/S2666651021000012.
  * 121 Sun, M. _et al._ Graph convolutional networks for computational drug development and discovery.  _Briefings in Bioinformatics_ 21, 919–935 (2019).  URL https://doi.org/10.1093/bib/bbz042.  https://academic.oup.com/bib/article-pdf/21/3/919/33227266/bbz042.pdf.
  * 122 Zhang, S., Tong, H., Xu, J. & Maciejewski, R.  Graph convolutional networks: a comprehensive review.  _Computational Social Networks_ 6, 1–23 (2019).
  * 123 Wu, Z. _et al._ A comprehensive survey on graph neural networks.  _IEEE Transactions on Neural Networks and Learning Systems_ 32, 4–24 (2021).
  * 124 Yuan, Y., Wang, W. & Pang, W.  A systematic comparison study on hyperparameter optimisation of graph neural networks for molecular property prediction.  _Proceedings of the Genetic and Evolutionary Computation Conference_ (2021).  URL http://dx.doi.org/10.1145/3449639.3459370.
  * 125 Jiang, D. _et al._ Could graph neural networks learn better molecular representation for drug discovery? a comparison study of descriptor-based and graph-based models.  _Journal of cheminformatics_ 13, 1–23 (2021).
  * 126 Banitalebi-Dehkordi, A. & Zhang, Y.  ML4CO: is GCNN all you need? graph convolutional neural networks produce strong baselines for combinatorial optimization problems, if tuned and trained properly, on appropriate data.  _CoRR_ abs/2112.12251 (2021).  URL https://arxiv.org/abs/2112.12251.  2112.12251.
  * 127 Liao, R., Zhao, Z., Urtasun, R. & Zemel, R.  Lanczosnet: Multi-scale deep graph convolutional networks.  In _International Conference on Learning Representations_ (2019).  URL https://openreview.net/forum?id=BkedznAqKQ.
  * 128 Bruna, J., Zaremba, W., Szlam, A. & LeCun, Y.  Spectral networks and locally connected networks on graphs (2013).  URL https://arxiv.org/abs/1312.6203.
  * 129 Henaff, M., Bruna, J. & LeCun, Y.  Deep convolutional networks on graph-structured data.  _CoRR_ abs/1506.05163 (2015).  URL http://arxiv.org/abs/1506.05163.  1506.05163.
  * 130 Levie, R., Monti, F., Bresson, X. & Bronstein, M. M.  Cayleynets: Graph convolutional neural networks with complex rational spectral filters.  _IEEE Transactions on Signal Processing_ 67, 97–109 (2019).
  * 131 Schlichtkrull, M. _et al._ Modeling relational data with graph convolutional networks.  In Gangemi, A. _et al._ (eds.) _The Semantic Web_ , 593–607 (Springer International Publishing, Cham, 2018).
  * 132 Niepert, M., Ahmed, M. & Kutzkov, K.  Learning convolutional neural networks for graphs.  In Balcan, M. F. & Weinberger, K. Q. (eds.) _Proceedings of The 33rd International Conference on Machine Learning_ , vol. 48 of _Proceedings of Machine Learning Research_ , 2014–2023 (PMLR, New York, New York, USA, 2016).  URL https://proceedings.mlr.press/v48/niepert16.html.
  * 133 Wang, X. _et al._ Molecule property prediction based on spatial graph embedding.  _Journal of Chemical Information and Modeling_ 59, 3817–3828 (2019).  URL https://doi.org/10.1021/acs.jcim.9b00410.  PMID: 31438677, %****␣ms.bbl␣Line␣1225␣****https://doi.org/10.1021/acs.jcim.9b00410.
  * 134 Xie, T. & Grossman, J. C.  Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties.  _Physical review letters_ 120, 145301 (2018).
  * 135 Park, C. W. & Wolverton, C.  Developing an improved crystal graph convolutional neural network framework for accelerated materials discovery.  _Physical Review Materials_ 4, 063801 (2020).
  * 136 Yadati, N.  Neural message passing for multi-relational ordered and recursive hypergraphs.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. & Lin, H. (eds.) _Advances in Neural Information Processing Systems_ , vol. 33, 3275–3289 (Curran Associates, Inc., 2020).  URL https://proceedings.neurips.cc/paper/2020/file/217eedd1ba8c592db97d0dbe54c7adfc-Paper.pdf.
  * 137 Strathmann, H., Barekatain, M., Blundell, C. & Veličković, P.  Persistent message passing.  In _ICLR 2021 Workshop on Geometrical and Topological Representation Learning_ (2021).  URL https://openreview.net/forum?id=HhOJZT--N23.
  * 138 Kim, Y. _et al._ Molnet: A chemically intuitive graph neural network for prediction of molecular properties.  _Chemistry–An Asian Journal_ (2022).
  * 139 Hu, W. _et al._ Forcenet: A graph neural network for large-scale quantum calculations (2021).  2103.01436.
  * 140 Park, C. _et al._ Accurate and scalable graph neural network force field and molecular dynamics with direct force architecture.  _npj Computational Materials_ 7 (2021).
  * 141 Coors, B., Condurache, A. P. & Geiger, A.  Spherenet: Learning spherical representations for detection and classification in omnidirectional images.  In Ferrari, V., Hebert, M., Sminchisescu, C. & Weiss, Y. (eds.) _Computer Vision – ECCV 2018_ , 525–541 (Springer International Publishing, Cham, 2018).
  * 142 Brody, S., Alon, U. & Yahav, E.  How attentive are graph attention networks?  _CoRR_ abs/2105.14491 (2021).  URL https://arxiv.org/abs/2105.14491.  2105.14491.
  * 143 Thekumparampil, K. K., Oh, S., Wang, C. & Li, L.-J.  Attention-based graph neural network for semi-supervised learning (2018).  URL https://openreview.net/forum?id=rJg4YGWRb.
  * 144 Withnall, M., Lindelöf, E., Engkvist, O. & Chen, H.  Building attention and edge message passing neural networks for bioactivity and physical–chemical property prediction.  _Journal of cheminformatics_ 12, 1–18 (2020).
  * 145 Xinyi, Z. & Chen, L.  Capsule graph neural network.  In _International Conference on Learning Representations_ (2019).  URL https://openreview.net/forum?id=Byl8BnRcYm.
  * 146 Busbridge, D., Sherburn, D., Cavallo, P. & Hammerla, N. Y.  Relational graph attention networks.  _CoRR_ abs/1904.05811 (2019).  URL http://arxiv.org/abs/1904.05811.  1904.05811.
  * 147 Wang, F. _et al._ Graph attention convolutional neural network model for chemical poisoning of honey bees’ prediction.  _Science Bulletin_ 65, 1184–1191 (2020).  URL https://www.sciencedirect.com/science/article/pii/S2095927320302176.
  * 148 Sacha, M. _et al._ Molecule edit graph attention network: modeling chemical reactions as sequences of graph edits.  _Journal of Chemical Information and Modeling_ 61, 3273–3284 (2021).
  * 149 Tang, B. _et al._ A self-attention based message passing neural network for predicting molecular lipophilicity and aqueous solubility.  _Journal of cheminformatics_ 12, 1–9 (2020).
  * 150 Li, Z., Yang, S., Song, G. & Cai, L.  Conformation-guided molecular representation with hamiltonian neural networks.  In _International Conference on Learning Representations_ (2021).  URL https://openreview.net/forum?id=q-cnWaaoUTH.
  * 151 Thomas, N. _et al._ Tensor field networks: Rotation- and translation-equivariant neural networks for 3d point clouds.  _CoRR_ abs/1802.08219 (2018).  URL http://arxiv.org/abs/1802.08219.  1802.08219.
  * 152 Kondor, R., Lin, Z. & Trivedi, S.  Clebsch-gordan nets: a fully fourier space spherical convolutional neural network.  In Bengio, S. _et al._ (eds.) _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , 10138–10147 (2018).  URL https://proceedings.neurips.cc/paper/2018/hash/a3fc981af450752046be179185ebc8b5-Abstract.html.
  * 153 Brandstetter, J., Hesselink, R., van der Pol, E., Bekkers, E. & Welling, M.  Geometric and physical quantities improve E(3) equivariant message passing.  _CoRR_ abs/2110.02905 (2021).  URL https://arxiv.org/abs/2110.02905.  2110.02905.
  * 154 Fuchs, F., Worrall, D., Fischer, V. & Welling, M.  Se(3)-transformers: 3d roto-translation equivariant attention networks.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. & Lin, H. (eds.) _Advances in Neural Information Processing Systems_ , vol. 33, 1970–1981 (Curran Associates, Inc., 2020).  URL https://proceedings.neurips.cc/paper/2020/file/15231a7ce4ba789d13b722cc5c955834-Paper.pdf.
  * 155 Cohen, T. & Welling, M.  Group equivariant convolutional networks.  In Balcan, M. F. & Weinberger, K. Q. (eds.) _Proceedings of The 33rd International Conference on Machine Learning_ , vol. 48 of _Proceedings of Machine Learning Research_ , 2990–2999 (PMLR, New York, New York, USA, 2016).  URL https://proceedings.mlr.press/v48/cohenc16.html.
  * 156 Lu, Y. _et al._ Cnn-g: Convolutional neural network combined with graph for image segmentation with theoretical analysis.  _IEEE Transactions on Cognitive and Developmental Systems_ 13, 631–644 (2021).
  * 157 Bekkers, E. J.  B-spline cnns on lie groups.  In _International Conference on Learning Representations_ (2020).  URL https://openreview.net/forum?id=H1gBhkBFDH.
  * 158 Ying, R. _et al._ Hierarchical graph representation learning with differentiable pooling.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , NIPS’18, 4805–4815 (Curran Associates Inc., Red Hook, NY, USA, 2018).
  * 159 Diehl, F.  Edge contraction pooling for graph neural networks (2019).  URL https://arxiv.org/abs/1905.10990.
  * 160 Gao, H. & Ji, S.  Graph u-nets.  In Chaudhuri, K. & Salakhutdinov, R. (eds.) _Proceedings of the 36th International Conference on Machine Learning_ , vol. 97 of _Proceedings of Machine Learning Research_ , 2083–2092 (PMLR, 2019).  URL https://proceedings.mlr.press/v97/gao19a.html.
  * 161 Gao, X., Xiong, H. & Frossard, P.  ipool - information-based pooling in hierarchical graph neural networks.  _IEEE transactions on neural networks and learning systems_ PP (2021).
  * 162 Ma, Y., Wang, S., Aggarwal, C. C. & Tang, J.  Graph convolutional networks with eigenpooling.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD ’19, 723–731 (Association for Computing Machinery, New York, NY, USA, 2019).  URL https://doi.org/10.1145/3292500.3330982.
  * 163 Liu, Q., Allamanis, M., Brockschmidt, M. & Gaunt, A.  Constrained graph variational autoencoders for molecule design.  _Advances in neural information processing systems_ 31 (2018).
  * 164 Jin, W., Barzilay, R. & Jaakkola, T.  Junction tree variational autoencoder for molecular graph generation.  In Dy, J. & Krause, A. (eds.) _Proceedings of the 35th International Conference on Machine Learning_ , vol. 80 of _Proceedings of Machine Learning Research_ , 2323–2332 (PMLR, 2018).  URL https://proceedings.mlr.press/v80/jin18a.html.
  * 165 You, J., Liu, B., Ying, Z., Pande, V. & Leskovec, J.  Graph convolutional policy network for goal-directed molecular graph generation.  _Advances in neural information processing systems_ 31 (2018).
  * 166 Wang, H. _et al._ Graphgan: Graph representation learning with generative adversarial nets.  In McIlraith, S. A. & Weinberger, K. Q. (eds.) _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , 2508–2515 (AAAI Press, 2018).  URL https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16611.
  * 167 Long, T. _et al._ Constrained crystals deep convolutional generative adversarial network for the inverse design of crystal structures.  _npj Computational Materials_ 7, 1–7 (2021).
  * 168 Sanchez-Lengeling, B. & Aspuru-Guzik, A.  Inverse molecular design using machine learning: Generative models for matter engineering.  _Science_ 361, 360–365 (2018).  URL https://www.science.org/doi/abs/10.1126/science.aat2663.  https://www.science.org/doi/pdf/10.1126/science.aat2663.
  * 169 Zang, C. & Wang, F.  Moflow: An invertible flow model for generating molecular graphs.  _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_ (2020).  URL http://dx.doi.org/10.1145/3394486.3403104.
  * 170 Gilbert, E. N.  Random graphs.  _The Annals of Mathematical Statistics_ 30, 1141–1144 (1959).
  * 171 Watts, D. J. & Strogatz, S. H.  Collective dynamics of ‘small-world’ networks.  _Nature_ 393, 440–442 (1998).
  * 172 Leskovec, J., Chakrabarti, D., Kleinberg, J., Faloutsos, C. & Ghahramani, Z.  Kronecker graphs: An approach to modeling networks.  _Journal of Machine Learning Research_ 11, 985–1042 (2010).  URL http://jmlr.org/papers/v11/leskovec10a.html.
  * 173 Kingma, D. P. & Welling, M.  Auto-encoding variational bayes.  In Bengio, Y. & LeCun, Y. (eds.) _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_ (2014).  URL http://arxiv.org/abs/1312.6114.
  * 174 Goodfellow, I. _et al._ Generative Adversarial Nets.  In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. & Weinberger, K. Q. (eds.) _Advances in Neural Information Processing Systems_ , vol. 27 (Curran Associates, Inc., 2014).  URL https://proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf.
  * 175 Sutton, R. S. & Barto, A. G.  _Reinforcement learning: An introduction_ (MIT press, 2018).
  * 176 You, J., Ying, R., Ren, X., Hamilton, W. & Leskovec, J.  GraphRNN: Generating realistic graphs with deep auto-regressive models.  In Dy, J. & Krause, A. (eds.) _Proceedings of the 35th International Conference on Machine Learning_ , vol. 80 of _Proceedings of Machine Learning Research_ , 5708–5717 (PMLR, 2018).  URL https://proceedings.mlr.press/v80/you18a.html.
  * 177 Dinh, L., Krueger, D. & Bengio, Y.  NICE: non-linear independent components estimation.  In Bengio, Y. & LeCun, Y. (eds.) _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings_ (2015).  URL http://arxiv.org/abs/1410.8516.
  * 178 Dinh, L., Sohl-Dickstein, J. & Bengio, S.  Density estimation using real NVP.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_ (OpenReview.net, 2017).  URL https://openreview.net/forum?id=HkpbnH9lx.
  * 179 Kingma, D. P. & Dhariwal, P.  Glow: Generative flow with invertible 1x1 convolutions.  _Advances in neural information processing systems_ 31 (2018).
  * 180 Pu, Y. _et al._ Variational autoencoder for deep learning of images, labels and captions.  In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I. & Garnett, R. (eds.) _Advances in Neural Information Processing Systems_ , vol. 29 (Curran Associates, Inc., 2016).  URL https://proceedings.neurips.cc/paper/2016/file/eb86d510361fc23b59f18c1bc9802cc6-Paper.pdf.
  * 181 Kusner, M. J., Paige, B. & Hernández-Lobato, J. M.  Grammar variational autoencoder.  In Precup, D. & Teh, Y. W. (eds.) _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ , vol. 70 of _Proceedings of Machine Learning Research_ , 1945–1954 (PMLR, 2017).  URL http://proceedings.mlr.press/v70/kusner17a.html.
  * 182 Dai, H., Tian, Y., Dai, B., Skiena, S. & Song, L.  Syntax-directed variational autoencoder for structured data.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_ (OpenReview.net, 2018).  URL https://openreview.net/forum?id=SyqShMZRb.
  * 183 Kipf, T. N. & Welling, M.  Variational graph auto-encoders (2016).  1611.07308.
  * 184 Simonovsky, M. & Komodakis, N.  Graphvae: Towards generation of small graphs using variational autoencoders.  In Kurková, V., Manolopoulos, Y., Hammer, B., Iliadis, L. S. & Maglogiannis, I. (eds.) _Artificial Neural Networks and Machine Learning - ICANN 2018 - 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part I_ , vol. 11139 of _Lecture Notes in Computer Science_ , 412–422 (Springer, 2018).  URL https://doi.org/10.1007/978-3-030-01418-6_41.
  * 185 Denton, E. L., Chintala, S., Fergus, R. _et al._ Deep generative image models using a laplacian pyramid of adversarial networks.  _Advances in neural information processing systems_ 28 (2015).
  * 186 Yu, L., Zhang, W., Wang, J. & Yu, Y.  Seqgan: Sequence generative adversarial nets with policy gradient.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 31 (2017).  URL https://ojs.aaai.org/index.php/AAAI/article/view/10804.
  * 187 Cao, N. D. & Kipf, T.  Molgan: An implicit generative model for small molecular graphs (2018).  1805.11973.
  * 188 Li, Y., Vinyals, O., Dyer, C., Pascanu, R. & Battaglia, P.  Learning deep generative models of graphs (2018).  URL https://arxiv.org/abs/1803.03324.
  * 189 Romeo Atance, S., Viguera Diez, J., Engkvist, O., Olsson, S. & Mercado, R.  De novo drug design using reinforcement learning with graph-based deep generative models.  _ChemRxiv_ (2021).
  * 190 Mercado, R. _et al._ Graph networks for molecular design.  _Machine Learning: Science and Technology_ 2 (2021).
  * 191 Li, Y., Zhang, L. R. & ming Liu, Z.  Multi-objective de novo drug design with conditional graph generative model.  _Journal of Cheminformatics_ 10 (2018).
  * 192 Papamakarios, G., Nalisnick, E. T., Rezende, D. J., Mohamed, S. & Lakshminarayanan, B.  Normalizing flows for probabilistic modeling and inference.  _J. Mach. Learn. Res._ 22, 1–64 (2021).
  * 193 Madhawa, K., Ishiguro, K., Nakago, K. & Abe, M.  Graphnvp: An invertible flow model for generating molecular graphs (2019).  1905.11600.
  * 194 Bengio, E., Jain, M., Korablyov, M., Precup, D. & Bengio, Y.  Flow network based generative models for non-iterative diverse candidate generation.  In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P. & Vaughan, J. W. (eds.) _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , 27381–27394 (2021).  URL https://proceedings.neurips.cc/paper/2021/hash/e614f646836aaed9f89ce58e837e2310-Abstract.html.
  * 195 Frey, N. C., Gadepally, V. & Ramsundar, B.  Fastflows: Flow-based models for molecular graph generation (2022).  2201.12419.
  * 196 Wang, S., Guo, X. & Zhao, L.  Deep Generative Model for Periodic Graphs.  _arXiv:2201.11932 [cs]_ (2022).  2201.11932.
  * 197 Feinberg, E. N., Joshi, E., Pande, V. S. & Cheng, A. C.  Improvement in admet prediction with multitask deep featurization.  _Journal of Medicinal Chemistry_ 63, 8835–8848 (2020).  URL https://doi.org/10.1021/acs.jmedchem.9b02187.  PMID: 32286824, https://doi.org/10.1021/acs.jmedchem.9b02187.
  * 198 Gastegger, M., Schütt, K. T. & Müller, K.-R.  Machine learning of solvent effects on molecular spectra and reactions.  _Chemical science_ 12, 11473–11483 (2021).
  * 199 Yan, C. _et al._ Retroxpert: Decompose retrosynthesis prediction like A chemist.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. & Lin, H. (eds.) _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ (2020).  URL https://proceedings.neurips.cc/paper/2020/hash/819f46e52c25763a55cc642422644317-Abstract.html.
  * 200 Axelrod, S., Shakhnovich, E. & Gómez-Bombarelli, R.  Excited state non-adiabatic dynamics of large photoswitchable molecules using a chemically transferable machine learning potential.  _Nature communications_ 13, 1–11 (2022).
  * 201 Li, Z. _et al._ Graph neural network based coarse-grained mapping prediction.  _Chem. Sci._ 11, 9524–9531 (2020).  URL http://dx.doi.org/10.1039/D0SC02458A.
  * 202 Gao, Y. _et al._ Gnes: Learning to explain graph neural networks.  In _2021 IEEE International Conference on Data Mining (ICDM)_ , 131–140 (IEEE, 2021).
  * 203 Wang, R. _et al._ Combining crystal graphs and domain knowledge in machine learning to predict metal-organic frameworks performance in methane adsorption.  _Microporous and Mesoporous Materials_ 331 (2022).
  * 204 Wang, Z., Han, Y., Cai, J., Wu, S. & Li, J.  DeepTMC: A deep learning platform to targeted design doped transition metal compounds.  _Energy Storage Materials_ 45, 1201–1211 (2022).
  * 205 Frey, N. C., Akinwande, D., Jariwala, D. & Shenoy, V. B.  Machine Learning-Enabled Design of Point Defects in 2D Materials for Quantum and Neuromorphic Information Processing.  _ACS Nano_ 14, 13406–13417 (2020).
  * 206 Dai, M., Demirel, M., Liang, Y. & Hu, J.-M.  Graph neural networks for an accurate and interpretable prediction of the properties of polycrystalline materials.  _npj Computational Materials_ 7 (2021).
  * 207 Swanson, K., Trivedi, S., Lequieu, J., Swanson, K. & Kondor, R.  Deep learning for automated classification and characterization of amorphous materials.  _Soft Matter_ 16, 435–446 (2020).
  * 208 Pham, T.-H., Qiu, Y., Zeng, J., Xie, L. & Zhang, P.  A deep learning framework for high-throughput mechanism-driven phenotype compound screening and its application to covid-19 drug repurposing.  _Nature Machine Intelligence_ 3, 247–257 (2021).  URL https://doi.org/10.1038/s42256-020-00285-9.
  * 209 St. John, P. C. _et al._ Message-passing neural networks for high-throughput polymer screening.  _The Journal of Chemical Physics_ 150, 234111 (2019).  URL http://dx.doi.org/10.1063/1.5099132.
  * 210 Peng, Y. _et al._ Enhanced graph isomorphism network for molecular admet properties prediction.  _IEEE Access_ 8, 168344–168360 (2020).
  * 211 Montanari, F., Kuhnke, L., Ter Laak, A. & Clevert, D.-A.  Modeling physico-chemical admet endpoints with multitask graph convolutional networks.  _Molecules_ 25 (2020).  URL https://www.mdpi.com/1420-3049/25/1/44.
  * 212 Yaowen, G., Bowen, Z., Si, Z., Fengchun, Y. & Jiao, L.  Predicting drug admet properties based on graph attention network.  _Data Analysis and Knowledge Discovery_ 5, 76–85 (2021).
  * 213 Xiong, J., Xiong, Z., Chen, K., Jiang, H. & Zheng, M.  Graph neural networks for automated de novo drug design.  _Drug Discovery Today_ 26, 1382–1393 (2021).  URL https://www.sciencedirect.com/science/article/pii/S1359644621000787.
  * 214 Cheung, M. & Moura, J. M.  Graph neural networks for covid-19 drug discovery.  In _2020 IEEE International Conference on Big Data (Big Data)_ , 5646–5648 (IEEE, 2020).
  * 215 Yu, X., Lu, S., Guo, L., Wang, S.-H. & Zhang, Y.-D.  Resgnet-c: A graph convolutional neural network for detection of covid-19.  _Neurocomputing_ 452, 592–605 (2021).  URL https://www.sciencedirect.com/science/article/pii/S0925231220319184.
  * 216 Kumar, A., Tripathi, A. R., Satapathy, S. C. & Zhang, Y.-D.  Sars-net: Covid-19 detection from chest x-rays by combining graph convolutional network and convolutional neural network.  _Pattern Recognition_ 122, 108255 (2022).  URL https://www.sciencedirect.com/science/article/pii/S0031320321004350.
  * 217 Nakata, M. & Shimazaki, T.  PubChemQC Project: A Large-Scale First-Principles Electronic Structure Database for Data-Driven Chemistry.  _Journal of Chemical Information and Modeling_ 57, 1300–1308 (2017).  URL https://doi.org/10.1021/acs.jcim.7b00083.  Publisher: American Chemical Society.
  * 218 Lee, C.-K. _et al._ Transfer learning with graph neural networks for optoelectronic properties of conjugated oligomers.  _The Journal of Chemical Physics_ 154, 024906 (2021).  URL https://doi.org/10.1063/5.0037863.
  * 219 Lu, C. _et al._ Deep learning for optoelectronic properties of organic semiconductors.  _The Journal of Physical Chemistry C_ 124, 7048–7060 (2020).  URL https://doi.org/10.1021/acs.jpcc.0c00329.  https://doi.org/10.1021/acs.jpcc.0c00329.
  * 220 Pronobis, W., Schütt, K. T., Tkatchenko, A. & Müller, K.-R.  Capturing intensive and extensive dft/tddft molecular properties with machine learning.  _The European Physical Journal B_ 91, 178 (2018).  URL https://doi.org/10.1140/epjb/e2018-90148-y.
  * 221 Atz, K., Isert, C., Böcker, M. N., Jiménez-Luna, J. & Schneider, G.  $\delta$-quantum machine learning for medicinal chemistry.  _ChemRxiv_ (2021).  URL https://chemrxiv.org/engage/chemrxiv/article-details/61c02f7e7f367e306759a0fd.
  * 222 Nakata, M., Shimazaki, T., Hashimoto, M. & Maeda, T.  PubChemQC PM6: Data Sets of 221 Million Molecules with Optimized Molecular Geometries and Electronic Properties.  _Journal of Chemical Information and Modeling_ 60, 5891–5899 (2020).  URL https://doi.org/10.1021/acs.jcim.0c00740.  Publisher: American Chemical Society.
  * 223 Gastegger, M., McSloy, A., Luya, M., Schütt, K. T. & Maurer, R. J.  A deep neural network for molecular wave functions in quasi-atomic minimal basis representation.  _The Journal of Chemical Physics_ 153, 044123 (2020).  URL http://dx.doi.org/10.1063/5.0012911.
  * 224 Burkholz, R., Quackenbush, J. & Bojar, D.  Using graph convolutional neural networks to learn a representation for glycans.  _Cell Reports_ 35, 109251 (2021).  URL https://doi.org/10.1016/j.celrep.2021.109251.
  * 225 Li, S. _et al._ MutagenPred-GCNNs: A graph convolutional neural network-based classification model for mutagenicity prediction with data-driven molecular fingerprints.  _Interdisciplinary Sciences: Computational Life Sciences_ 13, 25–33 (2021).  URL https://doi.org/10.1007/s12539-020-00407-2.
  * 226 Deng, D. _et al._ Xgraphboost: Extracting graph neural network-based features for a better prediction of molecular properties.  _Journal of Chemical Information and Modeling_ 61, 2697–2705 (2021).  URL https://doi.org/10.1021/acs.jcim.0c01489.  PMID: 34009965, https://doi.org/10.1021/acs.jcim.0c01489.
  * 227 Schweidtmann, A. M. _et al._ Graph neural networks for prediction of fuel ignition quality.  _Energy & Fuels_ 34, 11395–11407 (2020).  URL https://doi.org/10.1021/acs.energyfuels.0c01533.  https://doi.org/10.1021/acs.energyfuels.0c01533.