linear _gating_. This means that we apply a nonlinearity to a, possibly
learnable, combination of scalars434343This can be a simple MLP of the scalars
$\vec{\mathbf{M}}_{i}^{(0)}$ for instance. in
$\vec{\mathbf{M}}_{i}^{(0:l_{\text{max}})}$ and then use this as a weight to
scale the tensors of a given degree $l$. In equations,

$\underline{\textsc{Gated-
nl}}\left(\vec{\mathbf{M}}_{i}^{(0:l_{\text{max}})}\right)=\eta_{0}(\vec{\mathbf{M}}_{i}^{(0)})\oplus\left(\bigoplus_{l=1}^{l_{\text{max}}}\eta_{l}\left(\underline{\textsc{Gate}}\left(\vec{\mathbf{M}}_{i}^{(0)}\right)\right)\
\vec{\mathbf{M}}_{i}^{(l)}\right)$ (45)

Here, $\eta_{l}$ are non-linear functions and the above equation is understood
to apply channel-wise and $\underline{\textsc{Gate}}:\mathbb{R}\to\mathbb{R}$
is an arbitrary function with parameters, such as an MLP, that is applied
channel-wise.

##### Putting the pieces together.

Finally let us put the pieces together and write out the equations for a full
layer of our spherical EGNN.

$\underline{\textsc{Upd}}\left(\vec{\mathbf{H}}^{(0:l_{\text{max}})}_{i},\vec{\mathbf{M}}^{(0:l_{\text{max}})}_{i}\right)=\vec{\mathbf{H}}^{(0:l_{\text{max}})}_{i}+\underline{\textsc{Gated-
nl}}\left(\underline{\textsc{Mix}}\left(\vec{\mathbf{M}}^{(0:l_{\text{max}})}_{i}\right)\right)$
(46)

The update function performs a residual update of the spherical tensor
features at each node $i$ by first performing the graph convolution with the
learnable filter (Equation 43) to construct and aggregate messages. Next, we
mix the channels within a message and apply gated nonlinearities.

The updated node feature then becomes

$\vec{\mathbf{H}}^{(0:l_{\text{max}})}_{i}\leftarrow\underline{\textsc{Upd}}\left(\vec{\mathbf{H}}^{(0:l_{\text{max}})}_{i},\vec{\mathbf{M}}^{(0:l_{\text{max}})}_{i}\right).$
(47)

This completes a single layer of our example spherical EGNN. As with the
scalar-vector EGNN in LABEL:subsec:equivariant-gnns-cartesian, we can then
apply pooling and read out only the scalar channel at the output, in case we
are interested in an invariant prediction. As an added bonus, we can now also
read out higher equivariant types, in case we wanted to predict equivariant
quantities such as an atomic systems dipole moment (an $(l=1)$ quantity) or
quadrupole moment (an $(l=2)$ quantity) for instance.

(a) PaiNN

(b) TFN

Figure 19: Equivariant message passing.  $\mathscr{{G}}$-equivariant layers
such as PaiNN and TFN propagated geometric quantities such as vectors,
relative positions, or tensors.

##### Relations to Tensor Field Networks.

Tensor Field Network (Thomas et al., 2018) is one of the earliest approaches
proposing this type of $\mathscr{{G}}$-equivariant GNN with spherical tensors,
and is very similar to the exemplary spherical EGNN we just built. It has the
benefits of processing higher-order tensors directly and in an equivariant
fashion, resulting in SO(3)-equivariant model predictions. Despite its well
devised theoretical framework, TFN suffers from a few shortcomings. For
instance, retaining all the non-zero tensor products $\underline{\otimes}$ up
to degree $l_{\text{max}}$ becomes computationally unfeasible as we scale
$l_{\text{max}}$ since it requires $O(l_{\text{max}}^{3}C)$ multiplications.
Therefore most e3nn networks are limited to the computation of
$l_{\text{max}}=1$ or $2$ and $l_{f}\leq 2$ or $3$ for the filter functions
$l$. Notably, when restricting the tensor product to only scalars (up to
$l=0$), we obtain updates of the form similar to Section 4. Similarly, when
using only scalars and vectors (up to $l=1$), we obtain updates of the form
similar to Equation 16, Equation 17 and Equation 18.

####  5.3.2 Optimisations and improvements.

Several issues highlighted in TFNs have been tackled by subsequent approaches.
Below we give a brief selection of a few papers that use interesting ideas for
optimisations and improvements. We only give the key ideas, and refer to the
cited papers for details.

Fuchs et al. (2020) proposed SE(3)-Transformers, which adapt the TFN framework
to use an equivariant variant of the self-attention operation during
aggregation and operate on large point clouds while preserving equivariance.
SEGNN (Brandstetter et al., 2022) proposes equivariant non-linear convolutions
by introducing steerable MLPs that transform equivariant representations.
SEGNN follows a standard MPNN framework and uses steerable MLPs for the
message as well as update steps, essentially providing a recipe for for
building equivariant MPNNs with Spherical tensor features. The Equiformer
model (Liao and Smidt, 2023) successfully combines these ideas by interleaving
equivariant self-attention aggregation with equivariant non-linear updates
into Transformer-style blocks.

MACE (Batatia et al., 2022b) attempts to incorporate many-body interaction
terms444444Many-body effects refer to the collective behaviour of a large
number of interacting constituents. They are needed for an accurate
description of both the structure and dynamics of large chemical systems. by
relying on a clever factorization of higher-order terms into products of two-
body representation, which builds on the popular Atomic Cluster Expansion
(ACE) formalism (Drautz, 2019). The key idea here is to exchange summation and
multiplication454545This is referred to as _density-trick_ in the physics-ML
literature and comes from (Drautz, 2019). to reduce the number of
multiplication operations: As a simple example without tensors, think of the
expression $(a+b)^{2}$. This contains the terms $a^{2},ab,ba,b^{2}$ but
required us to perform only one multiplication464646The cost of this is that
the coefficients of these terms are coupled.. MACE essentially introduces an
efficient algorithm to compute a parameterised tensor-product of expressions
of the form
$\underline{\bigotimes}_{b=1}^{B}(\sum_{i=1}^{n_{i}}\vec{A}^{(l)}_{i})$. The
same idea is also used in Allegro (Musaelian et al., 2022), but without
message-passing to allow easier parallelisation of large material simulation
systems across multiple GPUs.

eSCN (Passaro and Zitnick, 2023) tackles the computational scalability issue
by reducing the SO(3) equivariant convolutions to mathematically equivalent
convolutions in $SO(2)$, making the tensor product easier to compute. This is
achieved by aligning the node embeddings’ primary axis with the edge
vector474747This trick sometimes called as _point-and-shoot_ in computational
electromagnetics., which reduces the rotational symmetry to rotations around
that axis, making the problem effectively 2 dimensional. While coming at the
cost of two extra Wigner D-matrix rotations to align and unalign the spherical
tensors to the given axis, this trick effectively sparsifies the Clebsch-
Gordan coefficients and thereby leads to computational speedups for $l>1$.

##  6 Unconstrained Geometric GNNs

Key idea Unlike other methods, architecturally unconstrained GNNs do not
‘bake’ symmetries into their architecture, leading to greater flexibility in
model design and more diverse optimization paths. Instead, they let the model
learn approximate symmetries, encourage approximate symmetries through loss
terms or data augmentation, or enforce symmetries through alternate strategies
such as (global or local) canonicalization.

Overview.  We have seen that previous Geometric GNN families, by design,
confine the set of learnable functions to equivariant ones, aligning with the
goal of accurately modeling equivariance. However, this constraint raises
concern about potential impediments to the neural network optimization
process. The idea is that a more unconstrained model, i.e. not bound to
equivariance, may traverse more diverse optimization paths (ultimately
converging to equivariant functions). As motivated by Duval et al. (2023),
this increased flexibility could empower the model to capture the intricacies
of the data more effectively. In contrast, the strict adherence to equivariant
constraints may limit the optimization paths available, or change the
optimization landscape in a way that hinders our algorithms, for example, by
altering the conditioning or prevalence of local minima484848The two arguments
can converge if the impairment corresponds to a spectrum distribution of the
eigenvalues of the Hessian around the fixed points, significantly limiting
escape possibilities (e.g., negative eigenvalues) around these fixed points..
This raises the question of whether the benefits of enforcing Euclidean
equivariance as an inductive bias truly offset a potential reduction in
optimization diversity within constrained learning spaces.

A useful parallel may be Deep Learning regularization, where equivariant
models, akin to a regularization paradigm, enforce specific constraints on the
functional learning space. While these constraints are intended to promote
desirable properties, such as provable or guaranteed equivariance,
unconstrained GNNs are concerned by the potential downside — the risk that
such constraints may overly regularize the model, hindering its capacity to
fully express the intricacies of the data. A few approaches (Wang et al.,
2022b; Hu et al., 2021; Zitnick et al., 2022; Duval et al., 2023) make this
argument, proposing a data-driven view of symmetries as opposed to the usual
model-based view.

Data augmentation and soft constraints. To explore the potential of this idea,
let us consider image classification. While most modern Convolutional Neural
Networks and Visual Transformers (Dosovitskiy et al., 2021) are not scale- or
rotation-equivariant (Weiler et al., 2018), they can still learn approximate
equivariance through rotation and scale diversity in the training data. This
adaptability extends to invariant prediction tasks on 3D images, where Gerken
et al. (2022) showed that data augmentation methods match invariant networks
in accuracy with significantly lower computational cost at inference time.
While the results are less successful for equivariant tasks, their work
suggest that leveraging unconstrained GNNs holds promise as an effective
approach.

ForceNet (Hu et al., 2021) was one of the first unconstrained GNN
architectures to explore data augmentation as a soft symmetry constraint. This
adaptation of Sanchez-Gonzalez et al. (2020) to 3D atomic systems proposes to
implicitly learn symmetries via data augmentation procedures such as adding
diverse rotations of the same geometric graph to the training data. Despite
showing promising results, ForceNet’s accuracy vs. scalability gains were not
significant enough to constitute a true Pareto optimal improvement.

Interestingly, architectures from the parallel field of 3D point cloud
processing for natural scenes and shapes (Guo et al., 2020) also rely heavily
on data augmentation. Models are generally trained with random rotations,
translations, and crops of the point cloud, along with corruptions to its
categorical values (Qi et al., 2017). Additional loss terms can also be added
as regularisation or soft constraints that encourage the model to preserve
symmetries without strictly enforcing it.

Another Geometric GNN with soft constraints is the SCN model (Zitnick et al.,
2022), an equivariant architecture that initially utilizes spherical harmonics
to represent channel embeddings with explicit orientation information.
However, it later relaxes the equivariant constraint to enable more expressive
non-linear transformations. Specifically, it projects spherical channels onto
a grid and conducts pointwise convolutions followed by a non-linearity,
facilitating intricate mixing of various degrees of spherical harmonics at the
expense of strict equivariance. When released, it reached state-of-the-art
performance on the Open Catalyst dataset OC20 (Chanussot et al., 2021).

Globally and locally canonicalized GNNs.  Iterating on the data augmentation
and soft constraint perspective, FAENet (Duval et al., 2023) proposes a model
architecture completely free of all design constraints, which directly
processes atom relative positions $\vec{{\bm{x}}}_{ij}$ using non-linear
functions (MLPs):

$\displaystyle{\bm{s}}_{i}^{(t+1)}$
$\displaystyle=\underline{f_{1}}\bigg{(}{\bm{s}}_{i}^{(t)},\sum_{j\in\mathcal{N}_{i}}{\bm{s}}_{j}^{(t)}\odot\underline{f_{2}}\big{(}\
\vec{x}_{ij},\ {\bm{s}}_{i}^{(t)},\ {\bm{s}}_{j}^{(t)}\big{)}\bigg{)},$ (48)

where $\underline{f_{2}}(\cdot)$ is the convolution filter encoding the 3D
geometric information (MLPs with swish activation) and
$\underline{f_{1}}(\cdot)$ the message passing update function. This inner
working is “possible” because FAENet outsources equivariance to the data
representation, allowing the model to process geometric information with
complete freedom. Building upon Frame Averaging (Puny et al., 2022), FAENet
projects data points into a canonical space via Principal Components Analysis
(PCA), offering a unique representation of all Euclidean transformations. This
canonicalization, illustrated in Figure 20, enables to rigorously (or
empirically) preserve symmetries while retaining maximal expressiveness.

Figure 20: (Stochastic) Frame Averaging pipeline utilized by FAENet to enforce
explicit (or approximate) equivariance. Input data is mapped to a global
reference frame, offering a unique representation of all Euclidean
transformations. It is then passed to the Geometric GNN model called FAENet
before being aggregated to yield invariant or equivariant predictions.

Let us briefly highlight the mechanisms of FAENet. We first need to compute
the centroid of an atomic graph with $n$ nodes
$\vec{t}=\frac{1}{n}\vec{\mathbf{x}}^{\top}\mathbf{1}\in\mathbb{R}^{3}$ and
the centred covariance matrix
$\Sigma=(\vec{\mathbf{x}}-\mathbf{1}\vec{t})^{\top}(\vec{\mathbf{x}}-\mathbf{1}\vec{t})$.
Solving for $(\vec{u},\lambda):\Sigma\vec{u}=\lambda\vec{u}$ and assuming
distinct eigenvalues $\lambda_{1}>\lambda_{2}>\lambda_{3}$ we can then create
the frame $\mathcal{F}(\vec{\mathbf{x}})=\left\\{(\vec{\mathbf{u}},\vec{t})\
|\ \vec{\mathbf{u}}=[\pm\vec{u}_{1},\pm\vec{u}_{2},\pm\vec{u}_{3}]\right\\}$,
For any given function (including GNNs) $\underline{f}$,
$\langle\underline{f}\rangle_{\mathcal{F}}(\vec{\mathbf{x}})$ is
$\mathscr{{G}}$-equivariant or invariant depending on $\rho_{2}$:

$\displaystyle\langle\underline{f}\rangle_{\mathcal{F}}(\vec{\mathbf{x}})$
$\displaystyle\vcentcolon=\frac{1}{|\mathcal{F}(\vec{\mathbf{x}})|}\sum_{g\in\mathcal{F}(\vec{\mathbf{x}})}\rho_{2}(g)\underline{f}(\rho_{1}^{-1}(g)\vec{\mathbf{x}})$
(49) $\displaystyle\rho_{1}(g)\vec{\mathbf{x}}$
$\displaystyle\vcentcolon=\vec{\mathbf{x}}\vec{\mathbf{u}}^{\top}+\mathbf{1}\vec{t}^{\top}$
(50) $\displaystyle\rho_{2}(g)\vec{\mathbf{y}}$
$\displaystyle\vcentcolon=\begin{cases}\vec{\mathbf{y}}\vec{\mathbf{u}}^{\top}\quad\text{for
equivariant predictions}\ \vec{\mathbf{y}}\in{\mathbb{R}}^{n\times 3}\\\ \
\vec{\mathbf{y}}\quad\quad\hskip 3.1298pt\text{for invariant predictions}\\\
\end{cases}$ (51)

The intuition here is that one can choose appropriate representatives of a
group, act on the data ($\rho_{1}$), infer with a function ($\underline{f}$),
act back to the original space ($\rho_{2}$) and finally average those
predictions to produce invariant or equivariant outputs. One downside of this
approach is that it requires $|\mathcal{F}(\vec{\mathbf{x}})|$ inferences (8
for the group E$(3)$, 4 for SE$(3)$) through $\underline{f}$, which can be an
expensive (graph-)neural network. To mitigate this effect, FAENet uses
Stochastic Frame-Averaging (SFA) where members of a frame are sampled instead
of being exhaustively averaged over. In different terms, we apply the function
$\underline{f}$ on a single frame element, selected randomly at inference and
at each training epoch, instead of all members. This approach does not
guarantee invariance/equivariance but allows the model to learn it. In this
perspective, it is akin to a geometry-informed data augmentation procedure
over a restricted set of $|\mathcal{F}(\vec{\mathbf{x}})|$ frames. It is
interesting to note how the frame averaging paradigm relates to existing
families of invariant and equivariant GNNs, defined in Section 4 and Section
5. FAENet is similar to invariant GNNs as it treats geometric information as
scalar quantities and uses a standard message passing scheme (i.e.
unconstrained) to update node representations non-linearly. But unlike them,
it can leverage relative atom positions directly instead of a pre-defined
scalarisation of geometric information. Besides, by avoiding the application
of symmetry-preserving equivariant operations on internal representations,
FAENet ‘breaks’ the expressivity limits of invariant and equivariant GNNs and
trivially distinguishes between all known counterexamples proposed by
Pozdnyakov et al. (2020) and Joshi et al. (2023a). Since it aggregates
relative atom positions using any non-linear function, it can be considered as
a many-body approach that iteratively incorporates information coming from
higher-order neighborhoods.

Similar to FAENet, Pozdnyakov and Ceriotti (2023) proposed an unconstrained
Geometric Transformer architecture based on an alternative to the frame
averaging protocol, termed Equivariant Coordinate System Ensemble, which
defines local coordinate systems at each atom and averages over the
predictions of a non-equivariant network for each coordinate system.
Similarly, Kaba et al. (2023) proposed to learn the canonicalization function
using a neural network instead of designing it by hand like in Frame
Averaging.

Summary.  Geometric GNNs not explicitly enforcing symmetries into the model
architecture are an emerging and under-explored line of work. Relaxing
symmetry constraints in the model design allows for the implementation of more
expressive architectures allowing for an interesting accuracy vs. scalability
trade-off at inference time. However, the partial or approximate enforcement
of symmetries may come at the cost of precision and physical violations in the
model’s prediction. For instance, a recent benchmark by Fu et al. (2023) found
ForceNet-based molecular simulators to be unstable over long timesteps and for
large systems. Thus, understanding when to impose strict equivariance in the
model as opposed to letting the model learn flexibly is an open question.
Similarly, looking at new ways to implicitly learn exact symmetries from data
(Dehmamy et al., 2021) could lead to significant improvements in the field.
The discussion is continued in Section 8.

Opinion Unconstrained GNNs do not state or show that building symmetries into
neural networks is not worthwhile. They simply tackle the interplay between
the space of functions that a model can learn and the ease of optimisation of
ML algorithms. Whether we should rigorously enforce symmetries or not is an
open question that probably depends on the application and scale of data
available. Here, SE(3) is a small group in terms of the degrees of freedom (a
rigid transformation is uniquely determined by 6 parameters), so not
rigorously enforcing equivariance may be tolerable and, perhaps, desirable
from the optimisation perspective. However, in other cases, incorporating such
inductive bias into the model will reduce sample complexity and improve
generalization. Finally, note that unconstrained GNNs still enforce node
permutation symmetry via permutation equivariant message passing.

##  7 Applications

Overview.  Geometric GNNs have shown promising results across a range of
application areas494949In this survey, we purposefully focus on applications
of Geometric GNNs to atomic systems. We refer the reader to Xiao et al.
(2020); Cao et al. (2020); Gerken et al. (2023) for other perspectives and
applications of Geometric Deep Learning, including computer vision., spanning
structural biology, biochemistry, and materials science (Zhang et al., 2023;
Wang et al., 2023a). These applications can be broadly categorised in Figure
21 as (1) property prediction, (2) molecular dynamics simulations, (3)
generative modeling, and (4) structure prediction. This section concisely
describes the utility of Geometric GNNs for each task, followed by a detailed
discussion of relevant datasets.

###  7.1 Tasks

(a) Property prediction

(b) Dynamics simulation

(c) Generative modeling &
structure prediction

Figure 21: Geometric GNN applications.  Representations learnt by Geometric
GNNs are used as part of task-specific pipelines in property prediction,
molecular dynamics simulation, generative modeling, and structure prediction.

####  7.1.1 Property Prediction

The most common application of Geometric GNNs is to predict functional and
physical properties of geometric graphs, ranging from quantum mechanical
properties of molecules and materials (Gasteiger et al., 2021) to the outcomes
of experimental assays in drug discovery (Stokes et al., 2020).

Conventional approaches to determining properties of 3D atomic systems are
known to be resource-intensive, both for simulation and experimental
approaches. For instance, quantum mechanical properties are computed using
simulation techniques such as Density Functional Theory (DFT)✜ (Kohn et al.,
1996) which often require supercomputers. Similarly, obtaining experimental
data usually requires specialized equipment tailored to a particular
application, with different requirements for biological systems, chemical
systems and complex materials. Geometric GNNs have emerged as a fast
alternative by learning to predict properties of new systems from large
annotated datasets generated by historical simulations or experiments. This
expedites the screening of large libraries of molecules and materials in order
to discover new candidates with specific desired properties. Notable use cases
of Geometric GNNs for property predictions include:

  1. 1.

Drug discovery – where Geometric GNNs help identify promising drug candidates
by modeling relevant properties of potential drugs, including proteins and
small molecules interacting with the human body (Huang et al., 2022). This
type of accelerated screening can help practitioners understand a drug’s
potential efficacy and safety profiles, and prioritize compounds for further
experimental testing.

  2. 2.

Material discovery – where, by accurately predicting properties such as
energy, stability, bandgap and conductivity, Geometric GNNs can help
researchers optimize material compositions and structures for specific
applications. Similar to drug discovery, this can help researchers discover
new materials for a diversity of applications (Chanussot et al., 2021; Lee et
al., 2023b).

  3. 3.

Environmental impact assessment – where they help understand the behavior of
substances in the environment, evaluating their toxicity, persistence, and
bioaccumulation potential. This enables to make informed decisions regarding
their use and disposal (Feinstein et al., 2021; Epa, 2020).

  4. 4.

Process optimization – where, by accurately predicting reactivity,
selectivity, and solubility, Geometric GNNs can help researchers optimize
reaction conditions, design efficient separation processes, and minimize waste
generation leading to more sustainable and cost-effective manufacturing
processes (Kearnes et al., 2021; Mercado et al., 2023).

Formally, the property prediction task is formulated as a regression or
classification problem, most often targeted to the full graph (e.g. energy,
band gaps, thermal conductivity, stability) although it can also apply at a
node level (e.g. atomic charges). A Geometric GNN model is trained to minimize
the loss function $\sum_{i}\mathcal{L}(\ \texttt{GNN}\ (\ \mathcal{{G}}_{i})\
,\ y_{i}\ )$ over a dataset of $n$ 3D geometric graphs $\mathcal{{G}}_{i}$,
paired with their corresponding property values $y_{i}$. The objective is to
optimize the model’s ability to predict a specific property by minimizing the
difference between the predicted value and the ground truth property value.
Commonly used loss functions include mean absolute error (MAE) for regression
tasks or cross-entropy for classification tasks. Geometric GNNs are very
effective for predicting materials and molecules’ properties due to their
ability to leverage atomic and geometric attributes while respecting data
symmetries.

####  7.1.2 Interatomic Potentials for Molecular Dynamics Simulation

Molecular Dynamics (MD) simulations, also known as atomistic simulations,
predict how every atom in a 3D system will move over time based on a general
model of the physics governing interatomic interactions (Karplus and McCammon,
2002). MD simulations are used to model a diversity of materials, including
periodic crystal structures, molecules and large-scale protein structures,
providing useful insights into various properties and behaviours. MD
simulations also represent atomistic systems as a set of $N$ atoms with
position vectors $\vec{\mathbf{x}}$ and atomic types $\mathbf{{S}}$ and aim to
solve Newton’s equation of motion for all atoms in the system. As such, the
system behaviour is governed by the potential energy $U$, which depends on the
interaction of the various atoms in the system. Concretely, $U$ is the
summation of of one-body $U(\vec{x}_{i})$, two-body
$U(\vec{x}_{i},\vec{x}_{j})$, three-body
$U(\vec{x}_{i},\vec{x}_{j},\vec{x}_{k})$, up to $N$-body interaction terms:

$U=\sum_{i=1}^{N}U(\vec{x}_{i})+\sum_{\begin{subarray}{c}i,j=1;\\\ i\neq
j\end{subarray}}^{N}U(\vec{x}_{i},\vec{x}_{j})+\sum_{\begin{subarray}{c}i,j,k=1;\\\
i\neq j\neq k\end{subarray}}^{N}U(\vec{x}_{i},\vec{x}_{j},\vec{x}_{k})+\cdots$
(52)

MD simulations also have the capability to include desired thermodynamic
conditions, such as temperature and pressure, by including relevant thermostat
and barostat settings found in common MD simulation packages (Thompson et al.,
2022; Van Der Spoel et al., 2005; Larsen et al., 2017). Once the potential
energy $U$ of the system is known, one can obtain the forces on each atom
using differentiation leading to forces on each atom: $\vec{F}_{i}=-\partial
U/\partial\vec{x}_{i}$. The acceleration of each atom is then calculated by
dividing the forces by the atomic mass $m_{i}$. Next, the updated atomic
positions are computed by numerically integrating the equations of motion
enabling one to study the dynamics of atomic systems.

Geometric GNNs can be infused into MD simulations by using GNN-based
interatomic potentials, meaning that
$U=\Phi^{1}(\vec{\mathbf{x}},\mathbf{{S}})$ with the forces being determined
by differentiation of the GNN-based potential (Batzner et al., 2022; Batatia
et al., 2022b, a). Alternatively, one can perform direct force prediction for
each atom with another GNN (head):
$\vec{\mathbf{F}}=\Phi^{2}(\vec{\mathbf{x}},\mathbf{{S}})$ (Gasteiger et al.,
2021). Note that, in both cases, we minimize a loss function that measures the
discrepancy (e.g. MAE) between the predicted and the ground truth values. A
thorough analysis of the advantages and shortcomings of MD potentials is
available in Bihani et al. (2023) and Fu et al. (2023). These studies suggest
that MAE-based regression training methods for GNN-based MD simulations are
not sufficient to guarantee MD simulation stability likely to due distribution
shift. One promising alternative may be training on the dynamics of the
trajectory data itself as explored by Bhattoo et al. (2023). Nonetheless, work
by Schaarschmidt et al. (2022) suggests that Geometric GNNs can avoid local
minima that classical quantum mechanical simulators often get trapped in,
showcasing that further research is needed to better understand the
capabilities and limitations of using Geometric GNNs in MD simulations.

####  7.1.3 Generative Modeling and Design

Geometric GNNs are emerging as a powerful data encoder tool to facilitate the
synthesis of new complex geometric structures including molecules, crystals,
and 3D objects. Their utilization in the generation pipeline is quite recent
and does not seem to have reached its full potential yet: most existing
generative approaches still use traditional GNN models (De Cao and Kipf, 2018;
Ragoza et al., 2020). To be more specific about their utilisation within the
generative pipeline, let’s first distinguish two main categories of generative
methods where they are (or could be) used:

Full-graph methods generate all attributes at once (i.e. scalar features,
geometric features and adjacency matrix), building on top of Variational Auto
Encoders (VAE) (Ren et al., 2022; Pakornchote et al., 2023), Generative
Adversarial Networks (GANs) (Nouira et al., 2018; Long et al., 2021),
Normalizing Flows (Satorras et al., 2021a; Ahmad and Cai, 2022) and Diffusion
models (Zheng et al., 2023; Xu et al., 2022). In general, these methods are
trained to reconstruct the training data from a latent distribution by
maximizing its likelihood, or by minimizing the discrepancy between generated
samples and the real data distribution. Once trained, they can generate new
instances by sampling from the learned latent space, transforming latent
representations into valid geometric graphs. But what about Geometric GNNs?
They are natural data encoders. Since they preserve data symmetries, they are
useful to preserve the likelihood of generated samples when the 3D atomic
system is rotated, reflected or translated. For VAE, they are used at train
time to create latent representations of the input geometric graph before
using an MLP or a GCN to (re-)produce a 3D graph (Xie et al., 2022). Regarding
diffusion models, which is a very active area of research, Hoogeboom et al.
(2022) employs an equivariant network jointly operating on continuous (atom
coordinates) and categorical features (atom types) in the denoising phase.
Finally, GAN based methods could use a Geometric GNN discriminator to predict
if the input geometric graph comes from the generator or from the training
dataset.

Iterative methods sequentially generate geometric graphs, selecting actions at
each step. G-Schnet and G-SphereNet (Gebauer et al., 2019; Luo and Ji, 2022),
for example, are auto-regressive models that generate 3D molecules by
performing atom-by-atom completion using invariant Geometric GNNs (Schütt et
al., 2017; Liu et al., 2021) on the current structure. Alternatively, the
generation of 3D atomic systems can be decomposed into compositional objects,
constructed step by step using Reinforcement Learning (RL). In this case,
Geometric GNNs are used to steer the action choices of the RL algorithm,
acting as a reward function that optimises the generation of
materials/molecules with targeted desirable properties. AI4Science et al.
(2023) employs a GFlowNet (Bengio et al., 2021) to sequentially sample 3D
crystals through the selection of the composition, space group and lattice
parameters, with any property prediction model as an objective function. This
domain-inspired approach enables the flexible incorporation of physical and
geometrical constraints and allows to search efficiently through the entire
material space.

Overall, applying Geometric GNNs to generative modeling has significant
implications for accelerating progress in fields like protein-conditioned
molecule generation (Corso et al., 2023; Schneuing et al., 2022), de novo
protein design (Ingraham et al., 2019; Dauparas et al., 2022), and
electrocatalyst discovery (Zitnick et al., 2020; Kolluru et al., 2022).

####  7.1.4 Structure Prediction

##### Protein Structure Prediction:

For biomolecules, the structure prediction task entails predicting or
generating a plausible set of 3D coordinates of a biomolecule given its 1D
sequence representation (AlQuraishi, 2021) (for proteins: the sequence of
amino acid residues, for nucleic acids: the sequence of nucleotides).
Geometric GNNs play a central role in modern structure prediction systems like
AlphaFold (Jumper et al., 2021) and RosettaFold (Baek et al., 2021, 2022). At
a high level, structure prediction systems consist of two modules applied
sequentially:

  1. 1.

The sequence module which constructs and updates latent representations of
each residue, generally via a standard Transformer (Lin et al., 2023) or a
specialised variant for processing multiple sequence alignments (Rao et al.,
2021; Jumper et al., 2021) to capture evolutionary relationships among
homologous sequences.

  2. 2.

The structure module which initialises a fully connected graph with nodes
representing the 3D positions of residues. Node representations are then
updated via a Geometric GNN with invariant (Jumper et al., 2021) or
equivariant message passing (Baek et al., 2021; Lee et al., 2023a) among all
the residues to iteratively refine the predicted 3D structure.

Models are trained via minimizing a loss function that quantifies the
difference between the predicted structure and the ground truth 3D structure.
The training data usually consists of experimentally determined 3D structures
from the Protein Data Bank, so models essentially aim to find the most
energetically favorable configuration among multiple possible conformational
states (Lane, 2023).

Note that, although closely related to generative modeling, structure
prediction involves predicting only the 3D positions of a given input
sequence. On the other hand, generative modeling involves learning the
distribution of a dataset of molecules and materials, and sampling new systems
from the underlying distribution. Interestingly, protein structure prediction
models can be repurposed as generative models for protein design (Watson et
al., 2023). This line of research has historical roots in physics-based
approaches which attempted to build energy functions grounded in a biophysical
understanding of protein folding (Alford et al., 2017).

##### Molecular Conformer Prediction:

In the case of molecular conformer prediction one aims to predict the 3D
atomic positions from a molecular formula, usually given a molecular string
representation (Weininger, 1988; Krenn et al., 2020; Cheng et al., 2023b).
Some of the most successful methods rely on predicting torsion angles from a
graph representation constructed from the molecular string representation
where Geometric GNNs serve as an encoder for learning effective
representations (Xu et al., 2022).

##### Crystal Structure Prediction:

Similar to molecular structure prediction, crystal structure prediction
involves the prediction of 3D crystal structures from two-dimensional crystal
compositions. Early work by Chen and Ong (2022) and Choudhary and DeCost
(2021) applied Geometric GNNs specifically designed for predicting 3D crystal
structures. Using a different approach, Jiao et al. (2023) applied an
equivariant diffusion to predict 3D crystal structures from atomic
compositions, taking into account the periodic structure of crystals as well
as relevant symmetries. A detailed review of various ML methods, including
Geometric GNNs, for crystal structure prediction, is given in Riebesell et al.
(2023).

##### Structure Prediction from Experimental Data:

An emerging application of geometric GNNs is in hybrid experimental and
computational pipelines for molecular structure determination. Notable
examples include CryoEM protein structure determination (Jamali et al., 2023)
and NMR chemical shift prediction (Guan et al., 2021; Yang et al., 2021).
Along similar lines, Cheng et al. (2023a) explored the use of equivariant
diffusion models to predict 3D molecular structures based on incomplete
information from real-world characterization instruments.

Opinion It is interesting to note the contrast in how the input geometric
graph is defined across the four tasks. Property prediction and dynamics
simulation tend to use local radial cutoff graphs. This is presumably due to
locality being a strong inductive bias, e.g. quantum mechanical properties or
forces of a system are unlikely to be influenced by long-range interactions.
On the other hand, generative modeling and structure prediction require models
to develop globally coherent representations and generally necessitate
operating on fully connected graphs. An interesting exception to this
observation is the random long-range graph construction scheme in Chroma
(Ingraham et al., 2022), a generative model for protein design which was
validated in the wet lab.

###  7.2 Datasets

In the realm of Geometric GNNs, the backbone of success lies in the
availability and quality of the data. With numerous datasets proposed to date,
it can be challenging to navigate through the landscape of options. Although
it is not the primary focus on the paper, we provide a selected list of
existing datasets for Geometric GNNs (see Table 1). A more exhaustive list is
accessible on a dedicated GitHub repository, which we hope the community will
keep up to date.

Our objective in this section goes beyond mere enumeration of available
datasets. Since data construction is essential to improve model performance
and to allow for true progress in subsequent years, we also discuss promising
data directions for the field. By addressing these considerations, we aspire
to foster the growth of the Geometric GNN community and empower researchers to
build more powerful and robust models. Overall, we encourage practitioners to
utilize (and construct) benchmark repositories containing several task-
specific datasets, similarly to the Open Catalyst Project (Chanussot et al.,
2021; Tran et al., 2023), Matbench (Dunn et al., 2020) and the Open MatSci ML
Toolkit (Lee et al., 2023b; Miret et al., 2023) for materials research as well
as to the Therapeutic Data Commons (Huang et al., 2022) for drug discovery;
all of which provide valuable ways to understand GNN model performance. Here
are a few reasons why;

  1. 1.

Splits and Leaderboards. These benchmarks are endowed with thorough and
transparent evaluation protocols, including carefully defined train/val/test
splits, a visible leaderboard with fixed evaluation metrics and some open-
source guides describing how to use each dataset (with some baseline methods
implemented). This enables easy utilisation as well as fair evaluation and
comparison across methods, which is essential. Besides, the validation
datasets contain both In-Domain (ID) and Out-of-Domain (OOD) split, allowing
the assessment of model generalisation.

  2. 2.

Domain experts. These datasets are constructed as the fruit of a collaboration
between domain experts and the machine learning community, bridging the
knowledge gap between scientific domains (e.g. physics, chemistry, materials
science, biology) and machine learning communities. This ensures that ML tasks
are consistent with underlying practical applications and that the available
datasets account for the subtleties of the application domain. As the datasets
continue to mature and the underlying problems become more and more complex,
the range of domain experts should concurrently expand to include
representation from government and corporations in addition to academic
researchers.

  3. 3.

Continual dataset updates. These datasets have shown regular updates and
expansion through the collection or generation of new data, such as when
enough new samples are obtained using DFT or experimental methods. The
continual dataset expansions allow ML models to solve more complex and
relevant challenges motivated by the underlying application. Ultimately the
goal is to create more robust ML models, including both highly specialized
models for a given application as well as general ML models that can tackle a
wide range of modeling problems.

  4. 4.

Diversity of tasks for generalized learning. In well-maintained benchmarks, it
is often possible to pre-train a model on a specific task and fine-tune it on
another. As datasets grow, a greater diversity of tasks enables the community
to build towards generalist foundation models for scientific applications in
3D atomic systems. Given the success of foundation models in the natural
language and vision domains, these developments have the potential to unlock
tremendous future research opportunities.

Software and libraries. On a slightly different note, one crucial factor for
the applicability and development of Geometric GNNs to any dataset is the
existence of handy code bases. We provide a list of useful repositories on
github, and encourage the community to maintain it up-to-date.

Opinion Navigating datasets of 3D atomic systems is currently difficult due to
major differences at every level: ground truth simulations, subsample
selection, splits, benchmarks vs single dataset, etc. We advocate for
additional work providing structure in this domain.

Task | Dataset | Benchmark | # Samples | # Tasks | Domain | Split | Metric | Date | Source
---|---|---|---|---|---|---|---|---|---
PP | Open MatSci ML Toolkit | Open MatSci ML Toolkit | 1.5M | Varied | Crystal Structures | Stratified/Random | MAE | 2023* | [Paper][github]
QM7-b | MoleculeNet | 7k | 14 | Molecule | Random | MAE | 2014 | [GDB-13][info][info]
QM9 | MoleculeNet | 130-134k | 12-19 | Molecule | Random | MAE | 2012 | [GDB-17][info][PyG]
PDBbind | MoleculeNet | 5k-13k | 1 | Protein-ligand | Time | RMSE | 2004* | [PDB][info]
Alchemy | Alchemy Contest | 120k | 12 | Molecule | Stratified/Size† | MAE | 2019 | [GDB MedChem] [link][HF]
Matbench | Mathbench | 1k-100k | 13 | Crystal Structures | StratifiedKFold† | MAE, ROC-AUC | 2019 | [Materials Project][data]
Atom3D | Atom3D | Varied | 8 | Mol., RNA, Prot. | Varied | Various | 2021 | [Varied][github]
Jarvis | Nist-Jarvis | 1k-800k | Varied | Molecules | Random† | MAE | 2020* | [Documentation][Paper]
Therapeutic Data Commons | TDC | Varied | Varied | Molecules & Proteins | Stratified† | Varied | 2022 | [Documentation][github]
TorchProtein | TorchProtein | Varied | Varied | Proteins | Stratified† | Varied | 2022 | [Paper][github]
TorchDrug | TorchDrug | Varied | Varied | Molecules | Stratified† | Varied | 2022 | [Paper][github]
PP and MD | OC20 | OpenCatalyst Project | 560k-133M | 3 | Catalyst | Extrapolation† | MAE, EwT | 2020 | [Paper][github]
OC22 | OpenCatalyst Project | 50k-10M | 3 | Catalyst | Extrapolation† | MAE, EwT | 2022 | [Paper][github]
ODAC23 | OpenCatalyst Project |  $\leq$40M | 3 | Catalyst & MOF | Extrapolation† | MAE, EwT | 2023* | [Paper][github]
MD | MD17 | – | 50k-1M | 10 | Molecule | Extrapolation | MAE | 2017* | [Paper] [HF] [PyG]
ISO17 | – | 645K | 1 | Molecule | Extrapolation | MAE | 2016 | [info] [Paper]
Gen.Mod | GEOM | GEOM | 37M | 1 | Molecule | Random | MAE, RMSD | 2021 | [Paper][github]
Open MatSci ML Toolkit | Open MatSci ML Toolkit | 25k | 1 | Crystal Structures | Random | MAE | 2023* | [Paper][github]
TorchDrug | TorchDrug | Varied | Varied | Molecules | Stratified† | MAE | 2022* | [Paper][github]
Struct.Pred | TorchProtein | TorchProtein | Varied | 3 | Proteins | Stratified† | Varied | 2022* | [Paper][github]
SPICE | – | 1.1 M | 6 | Molecules | Random | MAE | 2023 | [Paper][github]
MatBench Discovery | MatBench Discovery | 150k-250k | 2 | Crystal Structures | Varied† | F1, Accuracy, MAE | 2023 | [Paper] [github]
ProteinNet | – | 35k-105k | 7 | Proteins | Stratified | Varied | 2019 | [Paper][github]
Molecule3D | – | 3.9M | 4 | Molecules | Random | MAE, RMSE, validity | 2021 | [Paper][link]

Table 1: Summary of benchmark datasets for Geometric GNNs. We categorize each
dataset with respect to the application task detailed in Section 7.1. We
display various properties (from left to right): the number of samples per
dataset, the number of properties which can be predicted, the input data
domain, the desired symmetry property, the dataset split method, the metric
used to measure performance and the source. * signifies that the dataset has
been updated recently. † means that there is an active leaderboard to submit
test predictions and compare one’s results.

##  8 Conclusion and Future Research Directions

This opinionated survey aims to provide both newcomers and experienced
researchers with a pedagogical understanding of Geometric Graph Neural
Networks for 3D atomic systems. Throughout its course, we delve into the
foundations, motivations, and distinctive features that set Geometric GNNs
apart from traditional GNNs. We hope that our taxonomy of Geometric GNN
architectures – invariant, equivariant with Cartesian basis, equivariant with
spherical basis, and unconstrained GNNs – establishes clear links between
different models, granting the reader with a deeper comprehension of the
available methods. In addition, we have summarised a wide range of
applications and highlighted the high potential for impact of Geometric GNNs.
We hope to inspire further developments in Geometric GNN modeling towards
socially beneficial applications such as the discovery of novel medicine
(Stokes et al., 2020), energy-efficient materials (Miret et al., 2023), and
green chemistry (Anastas and Eghbali, 2010).

Our ultimate aspiration is to contribute to the organization and advancement
of this emerging field while igniting the curiosity of newcomers to embark on
their own journey into the captivating world of Geometric GNNs. By combining
scholarly rigour with a comprehensive overview, we aspire to make this survey
the natural go-to paper for the Geometric GNN community, facilitating
knowledge dissemination and fostering future advancements.

As we conclude our survey, the following section reflects on the promising
research directions that lie ahead and that we believe are worthy of interest.

###  8.1 To what extent should physics and symmetry be ‘baked in’ to Geometric
GNNs?

Enforcing symmetries: The choice between invariant GNNs, equivariant GNNs, or
unconstrained GNNs is an important consideration. Exploring the trade-off
between (1) an unconstrained local message passing approach that
(approximately) preserves global equivariance (e.g. FAENet (Duval et al.,
2023)), and (2) a strictly equivariant method that passes local equivariant
messages between atoms (e.g. MACE (Batatia et al., 2022b)), is an interesting
open question. Discussing the implications of these design choices and
rigorous empirical benchmarking would contribute to a deeper understanding of
the trade-offs involved. For instance, rigorously enforcing symmetries can
provide greater data efficiency and generalization abilities for model
architectures, which is particularly interesting when getting more high-
quality data is costly, or when stronger generalisation guarantees are needed.
On the other hand, relaxing these constraints may be desirable if enough data
is available, enabling greater expressivity and efficiency. Section 6 contains
additional arguments.

Energy conservation: On a related note, for molecular dynamics applications,
the debate between predicting forces using the gradient of energy versus
predicting forces directly from node representations is crucial. While the
former improves stability when using Geometric GNNs to run dynamics
simulations (Fu et al., 2023), the latter offers memory, runtime, and
sometimes performance gains (Gasteiger et al., 2021). Considering the
scalability vs. simulation stability trade-off can help practitioners decide
on the appropriate approach for their datasets and tasks. Additionally,
developing better metrics to quantify the importance of energy conservation
would be valuable with initial proposals emerging in the literature (Bihani et
al., 2023). Given the ambitions to scale molecular dynamics simulations to
trillions of atoms at longer and longer timescales, research on both improving
stability and compute efficiency at scale are needed.

Opinion We postulate that strict equivariance may be critical for tasks
related to precise geometric prediction, such as structure prediction or
molecular simulation, where small errors may compound and lead to unstable or
unphysical geometries if we use approximately symmetric models (Fu et al.,
2023; Bihani et al., 2023). On the other hand, tasks akin to property
prediction may benefit from approximately symmetric models which are optimised
for a particular dataset, without the hard constraints that come with
enforcing physical symmetries (Duval et al., 2023). Similarly, these
unconstrained models constitute a promising direction for applications where a
unique canonical ordering and orientation of data is possible, such as for
antibody proteins (Martinkus et al., 2023).

Deeper theoretical characterisation: A first step towards theoretical
understanding of Geometric GNNs was based on their ability to solve the
geometric graph isomorphism problem (Joshi et al., 2023a)✜ , i.e. mapping
unique geometric graphs to unique representations. A deeper understanding of
the expressive power, optimisation behaviour, and generalisation capacity of
Geometric GNNs will complement a growing landscape of empirical work, while
abstracting away domain-specific implementation details. For instance,
developing a provably universal, equivariant GNN on sparse graphs with finite
tensor and body order remains an open question (Batatia et al., 2022b). The
emergence of new equivariant architectures based on Clifford algebras also
presents opportunities for theoretical advancement and unifications (Brehmer
et al., 2023; Ruhe et al., 2023). These efforts have the potential to further
understand the generalization ability of equivariant models, which can help
practitioners choose what models may be appropriate for their desired use
case.

Opinion While equivariant GNNs operating on higher-order tensors have
favorable theoretical expressivity, their practical utility remains unclear.
Notably, prominent models such as RosettaFold (Baek et al., 2021) and DiffDock
(Corso et al., 2023), which use the e3nn framework, _do not_ use higher-order
tensors (they are restricted to tensor order $=1$). In practice, we believe
that the theoretical advantages of higher-order tensors are circumvented by
the (currently) high GPU memory requirements and slow speed of tensor product
operations. Thus, theoretical studies must be supplemented by empirical
benchmarks which fairly compare architectures under compute and time budgets
(Jamasb et al., 2023) as well as domain-specific evaluation beyond empirical
accuracy (Harris et al., 2023; Buttenschoen et al., 2023).

###  8.2 How to construct geometric graphs?

Graph creation and coarse-graining. There are various approaches to
constructing a geometric graph as input to a Geometric GNN, including radial
cutoffs, $k$-nearest neighbours, and long range connections. While optimal
graph construction heuristics are often highly domain-specific, exploring how
this construction could be modified to alleviate common structural bottlenecks
for GNNs, such as the over-squashing problem (Di Giovanni et al., 2023;
Giraldo et al., 2023), may hold great promise. Further studies are also needed
to better understand the importance of the local neighborhood in an atomic
system compared to long-range interactions that could be effectively modeled
with targeted graph constuction.

Additionally, the choice of entities included in graph construction needs
careful consideration. An obvious example is that all current Geometric GNN
applications assume a coarse-grained implicit solvent system and do not
explicitly include entities such as water molecules and ions which play an
important role in molecular structure and function (Bellissent-Funel et al.,
2016).

When working with coarse-grained representations of atomic systems, it becomes
critical to analyse the need for atomic-level precision and completeness of
the representation (whether there is a one-to-one mapping back to the all-atom
scale) (Badaczewska-Dawid et al., 2020). Imagine analyzing a molecular system
where each atom’s position is recorded with ultra-high accuracy. This level of
detail might be unnecessary for certain research objectives, and it may even
introduce undue computational complexity or overfitting to artefacts from the
structure determination process (Dauparas et al., 2022). In such cases, it
raises inquiries about the deliberate introduction of controlled noise to
coordinate data. The key question becomes: should this noise be intentionally
motivated by physical principles, and if so, what is the optimal amount of
noise to inject?

Opinion We suspect that optimizing the way information flows from the
perspective of graph machine learning may be promising for boosting
performance beyond the physics-based perspective of radial cutoffs and
interaction thresholds.

Temporal dynamics and conformational flexibility: Ideal computational
representations of molecules and materials should account for both geometric
structures as well as temporal dynamics. Looking beyond learning from static
structures, dynamics and conformational flexibility is key to the
functionality of several classes of proteins important for drug discovery,
such as antibodies and membrane proteins (Carugo and Djinović-Carugo, 2023;
Lane, 2023), as well as the dynamic behavior of materials in diverse
applications (Bihani et al., 2023; Fu et al., 2023). Learning from molecular
conformational ensembles will be the next frontier for advancing geometric
representation learning of small molecules (Axelrod and Gomez-Bombarelli,
2023), crystals (Lee et al., 2023b; Riebesell et al., 2023), proteins (Noé et
al., 2019; Janson et al., 2023), and RNA (Joshi et al., 2023b).

###  8.3 How to scale up Geometric GNNs?

Foundation models: Drawing inspiration from the success of self-supervised
learning for large pre-trained foundation models (Bommasani et al., 2021), the
question of why we do not yet have Geometric GNNs which can generalise across
the range of atomic systems is worth contemplating. Exploring new self-
supervised learning tasks to understand the dynamics of general-purpose atomic
interactions (so-called _universal potentials_) could significantly enhance
downstream performance across a wide range of application domains. On the
other hand, while it is pedagogical to group application domains together
under the umbrella of ‘atomic systems’, it may be the case that interactions
governing small molecules are semantically different than those in proteins or
materials. While not taking a definitive stance, this question presents an
interesting research direction for the field. What architectures are
expressive and scalable, and how can we efficiently train them?

Recently, Krishna et al. (2023) and DeepMind-Isomorphic (2023) generalised
protein structure prediction models to full biological assemblies including
proteins, small molecules, nucleic acids, and other ligands. These works show
promising results, even outperforming specialised models for certain tasks,
and represent an exciting first step towards foundation models for structural
biology. Geometric GNNs specialised for biomolecular complexes and
supramolecular systems (Steed and Atwood, 2022; Gallego et al., 2022) may
require deeper consideration of higher-order symmetries present in self-
assembling biological systems. For instance, naturally occurring DNA, perhaps
the best-known self-assembling structure, exists in a double helical form.

Large-scale datasets and software infrastructure: Training foundation models
necessitates large datasets and associated software infrastructure. The
availability of large-scale collections of predicted protein structures
derived from both AlphaFold2 (Jumper et al., 2021) and ESMFold (Lin et al.,
2023) is an extremely promising data source towards this end. The AlphaFold
Database (Varadi et al., 2021) and ESM Atlas (MGnify 2023 release (Richardson
et al., 2022)) contain over 200M and 772M predicted structures, respectively.
Bespoke software tools for predicted protein structures such as FoldSeek (van
Kempen et al., 2023) for efficient clustering and FoldComp (Kim et al., 2023)
for compression have started being used as part of pipelines to scale up
Geometric GNNs for protein structure annotation (Jamasb et al., 2023).

Similarly to the Open Catalyst Project (Zitnick et al., 2020; Tran et al.,
2023) and Open MatSci ML Toolkit (Miret et al., 2023; Lee et al., 2023b) for
materials discovery, the release of large public datasets and benchmarks
targeted to specific use cases and specially curated for advancing deep
learning architecture development are much needed. Such efforts may
necessitate rethinking the data generation process to be tailored towards
training large deep learning models, as motivated by recent initiatives for
small molecules (Mathiasen et al., 2023; Beaini et al., 2023). We hope to see
more community-driven and public initiatives in this direction, as motivated
in Section 7.2.

## References

  * Adams et al. [2022] K. Adams, L. Pattanaik, and C. W. Coley.  Learning 3d representations of molecular chirality with invariance to bond rotations.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net, 2022.
  * Ahmad and Cai [2022] R. Ahmad and W. Cai.  Free energy calculation of crystalline solids using normalizing flows.  _Modelling and Simulation in Materials Science and Engineering_ , 30(6):065007, 2022.
  * AI4Science et al. [2023] M. AI4Science, A. Hernandez-Garcia, A. Duval, A. Volokhova, Y. Bengio, D. Sharma, P. L. Carrier, M. Koziarski, and V. Schmidt.  Crystal-gfn: sampling crystals with desirable properties and constraints.  _arXiv preprint arXiv:2310.04925_ , 2023.
  * Alford et al. [2017] R. F. Alford, A. Leaver-Fay, J. R. Jeliazkov, M. J. O’Meara, F. P. DiMaio, H. Park, M. V. Shapovalov, P. D. Renfrew, V. K. Mulligan, K. Kappel, et al.  The rosetta all-atom energy function for macromolecular modeling and design.  _Journal of chemical theory and computation_ , 13(6):3031–3048, 2017.
  * AlQuraishi [2021] M. AlQuraishi.  Machine learning in protein structure prediction.  _Current opinion in chemical biology_ , 65:1–8, 2021.
  * Anastas and Eghbali [2010] P. Anastas and N. Eghbali.  Green chemistry: principles and practice.  _Chemical Society Reviews_ , 39(1):301–312, 2010\.
  * Anderson et al. [2019] B. Anderson, T. S. Hy, and R. Kondor.  Cormorant: Covariant molecular neural networks.  _Advances in neural information processing systems_ , 32, 2019.
  * Atz et al. [2021] K. Atz, F. Grisoni, and G. Schneider.  Geometric deep learning on molecular representations.  _Nature Machine Intelligence_ , 3(12):1023–1032, 2021.
  * Axelrod and Gomez-Bombarelli [2023] S. Axelrod and R. Gomez-Bombarelli.  Molecular machine learning with conformer ensembles.  _Machine Learning: Science and Technology_ , 4(3):035025, 2023.
  * Badaczewska-Dawid et al. [2020] A. E. Badaczewska-Dawid, A. Kolinski, and S. Kmiecik.  Computational reconstruction of atomistic protein structures from coarse-grained models.  _Computational and structural biotechnology journal_ , 18:162–176, 2020.
  * Baek et al. [2021] M. Baek, F. DiMaio, I. Anishchenko, J. Dauparas, et al.  Accurate prediction of protein structures and interactions using a three-track neural network.  _Science_ , 2021.
  * Baek et al. [2022] M. Baek, R. McHugh, I. Anishchenko, D. Baker, and F. DiMaio.  Accurate prediction of nucleic acid and protein-nucleic acid complexes using rosettafoldna.  _bioRxiv_ , pages 2022–09, 2022.
  * Bapst et al. [2020] V. Bapst, T. Keck, A. Grabska-Barwińska, C. Donner, E. D. Cubuk, S. Schoenholz, A. Obika, A. Nelson, T. Back, D. Hassabis, et al.  Unveiling the predictive power of static structure in glassy systems.  _Nature Physics_ , 16(4):448–454, 2020.
  * Bartók et al. [2013] A. P. Bartók, R. Kondor, and G. Csányi.  On representing chemical environments.  _Physical Review B_ , 2013.
  * Batatia et al. [2022a] I. Batatia, S. Batzner, D. P. Kovács, A. Musaelian, G. N. Simm, R. Drautz, C. Ortner, B. Kozinsky, and G. Csányi.  The design space of e (3)-equivariant atom-centered interatomic potentials.  _arXiv preprint_ , 2022a.
  * Batatia et al. [2022b] I. Batatia, D. P. Kovács, G. N. C. Simm, C. Ortner, and G. Csányi.  MACE: higher order equivariant message passing neural networks for fast and accurate force fields.  In _NeurIPS_ , 2022b.
  * Battaglia et al. [2018] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al.  Relational inductive biases, deep learning, and graph networks.  _arXiv preprint_ , 2018.
  * Batzner et al. [2022] S. Batzner, A. Musaelian, L. Sun, M. Geiger, J. P. Mailoa, M. Kornbluth, N. Molinari, T. E. Smidt, and B. Kozinsky.  E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials.  _Nature communications_ , 13(1):1–11, 2022.
  * Beaini et al. [2023] D. Beaini, S. Huang, J. A. Cunha, G. Moisescu-Pareja, O. Dymov, S. Maddrell-Mander, C. McLean, F. Wenkel, L. Müller, J. H. Mohamud, et al.  Towards foundational models for molecular learning on large-scale multi-task datasets.  _arXiv preprint arXiv:2310.04292_ , 2023.
  * Behler [2016] J. Behler.  Perspective: Machine learning potentials for atomistic simulations.  _The Journal of chemical physics_ , 145(17):170901, 2016.
  * Bellissent-Funel et al. [2016] M.-C. Bellissent-Funel, A. Hassanali, M. Havenith, R. Henchman, P. Pohl, F. Sterpone, D. Van Der Spoel, Y. Xu, and A. E. Garcia.  Water determines the structure and dynamics of proteins.  _Chemical reviews_ , 116(13):7673–7697, 2016\.
  * Benamira et al. [2020] A. Benamira, B. Devillers, E. Lesot, A. K. Ray, M. Saadi, and F. D. Malliaros.  Semi-supervised learning and graph neural networks for fake news detection.  In _ASONAM_ , 2020.
  * Bengio et al. [2021] E. Bengio, M. Jain, M. Korablyov, D. Precup, and Y. Bengio.  Flow network based generative models for non-iterative diverse candidate generation.  _Advances in Neural Information Processing Systems_ , 34:27381–27394, 2021.
  * Bhattoo et al. [2023] R. Bhattoo, S. Ranu, and N. A. Krishnan.  Learning the dynamics of particle-based systems with lagrangian graph neural networks.  _Machine Learning: Science and Technology_ , 4(1):015003, 2023.
  * Bihani et al. [2023] V. Bihani, U. Pratiush, S. Mannan, T. Du, Z. Chen, S. Miret, M. Micoulaut, M. M. Smedskjaer, S. Ranu, and N. Krishnan.  Egraffbench: Evaluation of equivariant graph neural network force fields for atomistic simulations.  _arXiv preprint arXiv:2310.02428_ , 2023.
  * Bishop [1995] C. M. Bishop.  Training with noise is equivalent to tikhonov regularization.  _Neural Computation_ , 7:108–116, 1995.  doi: 10.1162/neco.1995.7.1.108.
  * Blum-Smith and Villar [2022] B. Blum-Smith and S. Villar.  Machine learning and invariant theory.  _arXiv preprint arXiv:2209.14991_ , 2022.
  * Bodnar et al. [2021] C. Bodnar, F. Frasca, N. Otter, Y. Wang, P. Lio, G. F. Montufar, and M. Bronstein.  Weisfeiler and lehman go cellular: Cw networks.  _NeurIPS_ , 2021.
  * Bogatskiy et al. [2022] A. Bogatskiy, S. Ganguly, T. Kipf, R. Kondor, D. W. Miller, D. Murnane, J. T. Offermann, M. Pettee, P. Shanahan, C. Shimmin, et al.  Symmetry group equivariant architectures for physics.  _arXiv preprint_ , 2022.
  * Bommasani et al. [2021] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Brandstetter et al. [2021] J. Brandstetter, R. Hesselink, E. van der Pol, E. Bekkers, and M. Welling.  Geometric and physical quantities improve E(3) equivariant message passing.  _International Conference on Learning Representations_ , 2021.
  * Brandstetter et al. [2022] J. Brandstetter, R. Hesselink, E. van der Pol, E. J. Bekkers, and M. Welling.  Geometric and physical quantities improve e(3) equivariant message passing.  In _ICLR_ , 2022.
  * Brehmer et al. [2023] J. Brehmer, P. De Haan, S. Behrends, and T. Cohen.  Geometric algebra transformers.  _arXiv preprint arXiv:2305.18415_ , 2023.
  * Bronstein et al. [2017] M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst.  Geometric deep learning: Going beyond euclidean data.  _IEEE Signal Process. Mag._ , 34(4):18–42, 2017\.  doi: 10.1109/MSP.2017.2693418.
  * Bronstein et al. [2021] M. M. Bronstein, J. Bruna, T. Cohen, and P. Veli$\mathbf{c}$ković.  Geometric deep learning: Grids, groups, graphs, geodesics, and gauges.  _arXiv preprint_ , 2021.
  * Buttenschoen et al. [2023] M. Buttenschoen, G. M. Morris, and C. M. Deane.  Posebusters: Ai-based docking methods fail to generate physically valid poses or generalise to novel sequences.  _arXiv preprint arXiv:2308.05777_ , 2023.
  * Cao et al. [2020] W. Cao, Z. Yan, Z. He, and Z. He.  A comprehensive survey on geometric deep learning.  _IEEE Access_ , 8:35929–35949, 2020.  doi: 10.1109/ACCESS.2020.2975067.  URL https://ieeexplore.ieee.org/document/9003285.
  * Carugo and Djinović-Carugo [2023] O. Carugo and K. Djinović-Carugo.  Structural biology: A golden era.  _PLoS Biology_ , 21(6):e3002187, 2023.
  * Chanussot et al. [2021] L. Chanussot, A. Das, S. Goyal, T. Lavril, M. Shuaibi, M. Riviere, K. Tran, J. Heras-Domingo, C. Ho, W. Hu, et al.  Open catalyst 2020 (oc20) dataset and community challenges.  _ACS Catalysis_ , 11(10):6059–6072, 2021.
  * Chen and Ong [2022] C. Chen and S. P. Ong.  A universal graph deep learning interatomic potential for the periodic table.  _Nature Computational Science_ , 2(11):718–728, 2022.
  * Cheng et al. [2023a] A. Cheng, A. Lo, S. Miret, B. Pate, and A. Aspuru-Guzik.  Reflection-equivariant diffusion for 3d structure determination from isotopologue rotational spectra in natural abundance, 2023a.
  * Cheng et al. [2023b] A. H. Cheng, A. Cai, S. Miret, G. Malkomes, M. Phielipp, and A. Aspuru-Guzik.  Group selfies: a robust fragment-based molecular string representation.  _Digital Discovery_ , 2023b.
  * Chmiela et al. [2017] S. Chmiela, A. Tkatchenko, H. E. Sauceda, I. Poltavsky, K. T. Schütt, and K.-R. Müller.  Machine learning of accurate energy-conserving molecular force fields.  _Science advances_ , 3(5):e1603015, 2017.
  * Chmiela et al. [2018] S. Chmiela, H. E. Sauceda, K.-R. Müller, and A. Tkatchenko.  Towards exact molecular dynamics simulations with machine-learned force fields.  _Nature communications_ , 9(1):1–10, 2018.
  * Choudhary and DeCost [2021] K. Choudhary and B. DeCost.  Atomistic line graph neural network for improved materials property predictions.  _npj Computational Materials_ , 7(1):185, 2021\.
  * Cohen and Welling [2016] T. Cohen and M. Welling.  Group equivariant convolutional networks.  In _International conference on machine learning_ , pages 2990–2999. PMLR, 2016.
  * Corso et al. [2023] G. Corso, H. Stärk, B. Jing, R. Barzilay, and T. S. Jaakkola.  Diffdock: Diffusion steps, twists, and turns for molecular docking.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_ , 2023.
  * Dauparas et al. [2022] J. Dauparas, I. Anishchenko, N. Bennett, H. Bai, R. J. Ragotte, L. F. Milles, B. I. Wicky, A. Courbet, R. J. de Haas, N. Bethel, et al.  Robust deep learning based protein sequence design using proteinmpnn.  _Science_ , 2022.
  * De Cao and Kipf [2018] N. De Cao and T. Kipf.  Molgan: An implicit generative model for small molecular graphs.  _arXiv preprint arXiv:1805.11973_ , 2018.
  * DeepMind-Isomorphic [2023] DeepMind-Isomorphic.  Performance and structural coverage of the latest, in-development alphafold model.  2023\.
  * Dehmamy et al. [2021] N. Dehmamy, R. Walters, Y. Liu, D. Wang, and R. Yu.  Automatic symmetry discovery with lie algebra convolutional network.  _Advances in Neural Information Processing Systems_ , 34:2503–2515, 2021.
  * Derrow-Pinion et al. [2021] A. Derrow-Pinion, J. She, D. Wong, O. Lange, T. Hester, L. Perez, M. Nunkesser, S. Lee, X. Guo, B. Wiltshire, et al.  Eta prediction with graph neural networks in google maps.  In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, pages 3767–3776, 2021.
  * Di Giovanni et al. [2023] F. Di Giovanni, T. K. Rusch, M. M. Bronstein, A. Deac, M. Lackenby, S. Mishra, and P. Veli$\mathbf{c}$ković.  How does over-squashing affect the power of gnns?  _arXiv preprint arXiv:2306.03589_ , 2023.
  * Dosovitskiy et al. [2021] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.
  * Drautz [2019] R. Drautz.  Atomic cluster expansion for accurate and transferable interatomic potentials.  _Physical Review B_ , 99(1):014104, 2019.
  * Du et al. [2022] W. Du, H. Zhang, Y. Du, Q. Meng, W. Chen, N. Zheng, B. Shao, and T.-Y. Liu.  Se (3) equivariant graph neural networks with complete local frames.  In _International Conference on Machine Learning_ , pages 5583–5608. PMLR, 2022.
  * Dugas et al. [2001] C. Dugas, Y. Bengio, F. Bélisle, C. Nadeau, and R. Garcia.  Incorporating second-order functional knowledge for better option pricing.  _Advances in neural information processing systems_ , pages 472–478, 2001.
  * Dunn et al. [2020] A. Dunn, Q. Wang, A. Ganose, D. Dopp, and A. Jain.  Benchmarking materials property prediction methods: the matbench test set and automatminer reference algorithm. npj comput mater 6, 1 (september 2020), 1-10.  _DOI: https://doi. org/10.1038/s41524-020-00406-3_ , 2020.
  * Duval et al. [2022] A. Duval, V. Schmidt, S. Miret, Y. Bengio, A. Hernández-García, and D. Rolnick.  PhAST: Physics-aware, scalable, and task-specific GNNs for accelerated catalyst design.  _Preprint arXiv:2211.12020_ , 2022.
  * Duval et al. [2023] A. Duval, V. Schmidt, A. Hernández-García, S. Miret, F. D. Malliaros, Y. Bengio, and D. Rolnick.  Faenet: Frame averaging equivariant GNN for materials modeling.  In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 9013–9033. PMLR, 2023\.
  * Duvenaud et al. [2015] D. K. Duvenaud, D. Maclaurin, J. Iparraguirre, R. Bombarell, T. Hirzel, A. Aspuru-Guzik, and R. P. Adams.  Convolutional networks on graphs for learning molecular fingerprints.  _Advances in neural information processing systems_ , 28, 2015.
  * Dwivedi et al. [2023] V. P. Dwivedi, C. K. Joshi, A. T. Luu, T. Laurent, Y. Bengio, and X. Bresson.  Benchmarking graph neural networks.  _JMLR_ , 2023.
  * Dym and Maron [2020] N. Dym and H. Maron.  On the universality of rotation equivariant point cloud networks.  In _ICLR_ , 2020.
  * Epa [2020] U. Epa.  User’s guide for test (version 5.1)(toxicity estimation software tool): a program to estimate toxicity from molecular structure.  _Chemical Characterization and Exposure Division Cincinnati O, ed_ , 2020.
  * Feinstein et al. [2021] J. Feinstein, G. Sivaraman, K. Picel, B. Peters, Á. Vázquez-Mayagoitia, A. Ramanathan, M. MacDonell, I. Foster, and E. Yan.  Uncertainty-informed deep transfer learning of perfluoroalkyl and polyfluoroalkyl substance toxicity.  _Journal of chemical information and modeling_ , 61(12):5793–5803, 2021.
  * Frank et al. [2022] T. Frank, O. T. Unke, and K. R. Muller.  So3krates: Equivariant attention for interactions on arbitrary length-scales in molecular systems.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Fu et al. [2023] X. Fu, Z. Wu, W. Wang, T. Xie, S. Keten, R. Gomez-Bombarelli, and T. S. Jaakkola.  Forces are not enough: Benchmark and critical evaluation for machine learning force fields with molecular simulations.  _Transactions on Machine Learning Research_ , 2023.
  * Fuchs et al. [2020] F. Fuchs, D. Worrall, V. Fischer, and M. Welling.  SE(3)-transformers: 3D roto-translation equivariant attention networks.  _Advances in Neural Information Processing Systems_ , 33:1970–1981, 2020.
  * Gallego et al. [2022] L. Gallego, J. F. Woods, and M. Rickhaus.  Recent concepts for supramolecular 2d materials.  _Organic Materials_ , 4(03):137–145, 2022.
  * Gasteiger et al. [2020] J. Gasteiger, J. Groß, and S. Günnemann.  Directional message passing for molecular graphs.  In _ICLR_ , 2020.
  * Gasteiger et al. [2021] J. Gasteiger, F. Becker, and S. Günnemann.  Gemnet: Universal directional graph neural networks for molecules.  _Advances in Neural Information Processing Systems_ , 34:6790–6802, 2021.
  * Gasteiger et al. [2022] J. Gasteiger, M. Shuaibi, A. Sriram, S. Günnemann, Z. W. Ulissi, C. L. Zitnick, and A. Das.  Gemnet-oc: Developing graph neural networks for large and diverse molecular simulation datasets.  _Trans. Mach. Learn. Res._ , 2022, 2022.
  * Gebauer et al. [2019] N. Gebauer, M. Gastegger, and K. Schütt.  Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules.  _Advances in neural information processing systems_ , 32, 2019.
  * Geiger and Smidt [2022] M. Geiger and T. Smidt.  e3nn: Euclidean neural networks.  _arXiv preprint_ , 2022.
  * Gerken et al. [2022] J. Gerken, O. Carlsson, H. Linander, F. Ohlsson, C. Petersson, and D. Persson.  Equivariance versus augmentation for spherical images.  In _International Conference on Machine Learning_ , pages 7404–7421. PMLR, 2022.
  * Gerken et al. [2023] J. E. Gerken, J. Aronsson, O. Carlsson, H. Linander, F. Ohlsson, C. Petersson, and D. Persson.  Geometric deep learning and equivariant neural networks.  _Artificial Intelligence Review_ , 56:14605–14662, 2023\.  doi: 10.1007/s10462-023-10502-7.  URL https://link.springer.com/article/10.1007/s10462-023-10502-7/fulltext.html.
  * Gilmer et al. [2017] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl.  Neural message passing for quantum chemistry.  In _ICML_ , 2017.
  * Giraldo et al. [2023] J. H. Giraldo, K. Skianis, T. Bouwmans, and F. D. Malliaros.  On the trade-off between over-smoothing and over-squashing in deep graph neural networks.  In _CIKM_ , 2023.
  * Glorot et al. [2011] X. Glorot, A. Bordes, and Y. Bengio.  Deep sparse rectifier neural networks.  In _aistats_ , pages 315–323, 2011.
  * Godwin et al. [2021] J. Godwin, M. Schaarschmidt, A. Gaunt, A. Sanchez-Gonzalez, Y. Rubanova, P. Veli$\mathbf{c}$ković, J. Kirkpatrick, and P. Battaglia.  Simple gnn regularisation for 3d molecular property prediction & beyond.  _arXiv preprint arXiv:2106.07971_ , 2021.
  * Goller and Kuchler [1996] C. Goller and A. Kuchler.  Learning task-dependent distributed representations by backpropagation through structure.  In _Proceedings of International Conference on Neural Networks (ICNN’96)_ , volume 1, pages 347–352. IEEE, 1996.
  * Gori et al. [2005] M. Gori, G. Monfardini, and F. Scarselli.  A new model for learning in graph domains.  In _Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005._ , volume 2, pages 729–734. IEEE, 2005.
  * Guan et al. [2021] Y. Guan, S. S. Sowndarya, L. C. Gallegos, P. C. S. John, and R. S. Paton.  Real-time prediction of 1 h and 13 c chemical shifts with dft accuracy using a 3d graph neural network.  _Chemical Science_ , 12(36):12012–12026, 2021\.
  * Guo et al. [2020] Y. Guo, H. Wang, Q. Hu, H. Liu, L. Liu, and M. Bennamoun.  Deep learning for 3d point clouds: A survey.  _IEEE transactions on pattern analysis and machine intelligence_ , 43(12):4338–4364, 2020.
  * Halgren [1996] T. Halgren.  Merck molecular force field. i. basis, form, scope, parameterization, and performance of mmff94.  _Comput. Chem._ , 17:490–519, 1996.
  * Hamilton et al. [2017] W. L. Hamilton, R. Ying, and J. Leskovec.  Inductive representation learning on large graphs.  In _NeurIPS_ , 2017.
  * Han et al. [2022] J. Han, Y. Rong, T. Xu, and W. Huang.  Geometrically equivariant graph neural networks: A survey.  _arXiv preprint arXiv:2202.07230_ , 2022.
  * Harris et al. [2023] C. Harris, K. Didi, A. Jamasb, C. Joshi, S. Mathis, P. Lio, and T. Blundell.  Posecheck: Generative models for 3d structure-based drug design produce unrealistic poses.  In _NeurIPS 2023 Generative AI and Biology (GenBio) Workshop_ , 2023\.
  * [89] K. He, X. Zhang, S. Ren, and J. Sun.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778.  doi: 10.1109/CVPR.2016.90.
  * Hoja et al. [2021] J. Hoja, L. Medrano Sandonas, B. G. Ernst, A. Vazquez-Mayagoitia, R. A. DiStasio Jr, and A. Tkatchenko.  Qm7-x, a comprehensive dataset of quantum-mechanical properties spanning the chemical space of small organic molecules.  _Scientific data_ , 8(1):43, 2021.
  * Hoogeboom et al. [2022] E. Hoogeboom, V. G. Satorras, C. Vignac, and M. Welling.  Equivariant diffusion for molecule generation in 3d.  In _International conference on machine learning_ , pages 8867–8887. PMLR, 2022.
  * Hu et al. [2021] W. Hu, M. Shuaibi, A. Das, S. Goyal, A. Sriram, J. Leskovec, D. Parikh, and C. L. Zitnick.  Forcenet: A graph neural network for large-scale quantum calculations.  _Preprint arXiv:2103.01436_ , 2021.
  * Huang et al. [2022] K. Huang, T. Fu, W. Gao, Y. Zhao, Y. Roohani, J. Leskovec, C. W. Coley, C. Xiao, J. Sun, and M. Zitnik.  Artificial intelligence foundation for therapeutic science.  _Nature chemical biology_ , 18(10):1033–1036, 2022.
  * Ingraham et al. [2019] J. Ingraham, V. Garg, R. Barzilay, and T. Jaakkola.  Generative models for graph-based protein design.  _NeurIPS_ , 2019.
  * Ingraham et al. [2022] J. Ingraham, M. Baranov, Z. Costello, V. Frappier, A. Ismail, S. Tie, W. Wang, V. Xue, F. Obermeyer, A. Beam, et al.  Illuminating protein space with a programmable generative model.  _BioRxiv_ , 2022.
  * Jamali et al. [2023] K. Jamali, L. Käll, R. Zhang, A. Brown, D. Kimanius, and S. H. Scheres.  Automated model building and protein identification in cryo-em maps.  _bioRxiv_ , 2023.
  * Jamasb et al. [2023] A. R. Jamasb, A. Morehead, Z. Zuobai, C. K. Joshi, K. Didi, S. V. Mathis, C. Harris, J. Tang, J. Cheng, P. Liò, and T. L. Blundell.  Evaluating representation learning on the protein structure universe.  _NeurIPS 2023 Workshop on Machine Learning for Structural Biology_ , 2023.
  * Janson et al. [2023] G. Janson, G. Valdes-Garcia, L. Heo, and M. Feig.  Direct generation of protein conformational ensembles via machine learning.  _Nature Communications_ , 2023.
  * Jegelka [2022] S. Jegelka.  Theory of graph neural networks: Representation and learning.  _arXiv preprint arXiv:2204.07697_ , 2022.
  * Jiao et al. [2023] R. Jiao, W. Huang, P. Lin, J. Han, P. Chen, Y. Lu, and Y. Liu.  Crystal structure prediction by joint equivariant diffusion.  _arXiv preprint arXiv:2309.04475_ , 2023.
  * Jing et al. [2020] B. Jing, S. Eismann, P. Suriana, R. J. L. Townshend, and R. Dror.  Learning from protein structure with geometric vector perceptrons.  In _ICLR_ , 2020.
  * Joshi [2020] C. Joshi.  Transformers are graph neural networks.  _The Gradient_ , 2020.
  * Joshi et al. [2023a] C. K. Joshi, C. Bodnar, S. V. Mathis, T. Cohen, and P. Liò.  On the expressive power of geometric graph neural networks.  In _International Conference on Machine Learning_ , 2023a.
  * Joshi et al. [2023b] C. K. Joshi, A. R. Jamasb, R. Viñas, C. Harris, S. Mathis, and P. Liò.  Multi-state rna design with geometric multi-graph neural networks.  _arXiv preprint_ , 2023b.
  * Jumper et al. [2021] J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. $\mathbf{Z}$ídek, A. Potapenko, et al.  Highly accurate protein structure prediction with alphafold.  _Nature_ , 2021.
  * Kaba and Ravanbakhsh [2022] S.-O. Kaba and S. Ravanbakhsh.  Equivariant networks for crystal structures.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Kaba et al. [2023] S.-O. Kaba, A. K. Mondal, Y. Zhang, Y. Bengio, and S. Ravanbakhsh.  Equivariance with learned canonicalization functions.  In _International Conference on Machine Learning_ , pages 15546–15566. PMLR, 2023.
  * Karplus and McCammon [2002] M. Karplus and J. A. McCammon.  Molecular dynamics simulations of biomolecules.  _Nature structural biology_ , 9(9):646–652, 2002\.
  * Kearnes et al. [2021] S. M. Kearnes, M. R. Maser, M. Wleklinski, A. Kast, A. G. Doyle, S. D. Dreher, J. M. Hawkins, K. F. Jensen, and C. W. Coley.  The open reaction database.  _Journal of the American Chemical Society_ , 143(45):18820–18826, 2021.
  * Kim et al. [2023] H. Kim, M. Mirdita, and M. Steinegger.  Foldcomp: a library and format for compressing and indexing large protein structure sets.  _Bioinformatics_ , 39(4), Mar. 2023.  doi: 10.1093/bioinformatics/btad153.
  * Kipf and Welling [2017] T. N. Kipf and M. Welling.  Semi-supervised classification with graph convolutional networks.  In _International Conference on Learning Representations_ , 2017.
  * Klicpera et al. [2020] J. Klicpera, J. Groß, and S. Günnemann.  Directional message passing for molecular graphs.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_ , 2020.
  * Kohn et al. [1996] W. Kohn, A. D. Becke, and R. G. Parr.  Density functional theory of electronic structure.  _The Journal of Physical Chemistry_ , 100(31):12974–12980, 1996.
  * Kolluru et al. [2022] A. Kolluru, M. Shuaibi, A. Palizhati, N. Shoghi, A. Das, B. Wood, C. L. Zitnick, J. Kitchin, and Z. W. Ulissi.  Open challenges in developing generalizable large scale machine learning models for catalyst discovery.  _ACS Catalysis_ , 2022.  doi: 10.1021/acscatal.2c02291.
  * Kondor et al. [2018] R. Kondor, Z. Lin, and S. Trivedi.  Clebsch-gordan nets: a fully fourier space spherical convolutional neural network.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Kosmala et al. [2023] A. Kosmala, J. Gasteiger, N. Gao, and S. Günnemann.  Ewald-based long-range message passing for molecular graphs.  In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 17544–17563. PMLR, 2023\.
  * Krenn et al. [2020] M. Krenn, F. Häse, A. Nigam, P. Friederich, and A. Aspuru-Guzik.  Self-referencing embedded strings (selfies): A 100% robust molecular string representation.  _Machine Learning: Science and Technology_ , 1(4):045024, 2020.
  * Krishna et al. [2023] R. Krishna, J. Wang, W. Ahern, P. Sturmfels, P. Venkatesh, I. Kalvet, G. R. Lee, F. S. Morey-Burrows, I. Anishchenko, I. R. Humphreys, et al.  Generalized biomolecular modeling and design with rosettafold all-atom.  _bioRxiv_ , pages 2023–10, 2023.
  * Lam et al. [2023] R. Lam, A. Sanchez-Gonzalez, M. Willson, P. Wirnsberger, M. Fortunato, F. Alet, S. Ravuri, T. Ewalds, Z. Eaton-Rosen, W. Hu, et al.  Learning skillful medium-range global weather forecasting.  _Science_ , page eadi2336, 2023.
  * Lane [2023] T. J. Lane.  Protein structure prediction has reached the single-structure frontier.  _Nature Methods_ , 20(2):170–173, 2023.
  * Larsen et al. [2017] A. H. Larsen, J. J. Mortensen, J. Blomqvist, I. E. Castelli, R. Christensen, M. Du$\underline{a}$k, J. Friis, M. N. Groves, B. Hammer, C. Hargus, E. D. Hermes, P. C. Jennings, P. B. Jensen, J. Kermode, J. R. Kitchin, E. L. Kolsbjerg, J. Kubal, K. Kaasbjerg, S. Lysgaard, J. B. Maronsson, T. Maxson, T. Olsen, L. Pastewka, A. Peterson, C. Rostgaard, J. Schiøtz, O. Schütt, M. Strange, K. S. Thygesen, T. Vegge, L. Vilhelmsen, M. Walter, Z. Zeng, and K. W. Jacobsen.  The atomic simulation environment—a python library for working with atoms.  _Journal of Physics: Condensed Matter_ , 29(27):273002, 2017.
  * Le et al. [2022] T. Le, F. Noe, and D.-A. Clevert.  Representation learning on biomolecular structures using equivariant graph attention.  In _Learning on Graphs Conference_ , pages 30–1. PMLR, 2022.
  * Lee et al. [2019] J. Lee, Y. Lee, J. Kim, A. Kosiorek, S. Choi, and Y. W. Teh.  Set transformer: A framework for attention-based permutation-invariant neural networks.  In _International conference on machine learning_ , pages 3744–3753. PMLR, 2019.
  * Lee et al. [2023a] J. H. Lee, P. Yadollahpour, A. Watkins, N. C. Frey, A. Leaver-Fay, S. Ra, K. Cho, V. Gligorijević, A. Regev, and R. Bonneau.  Equifold: Protein structure prediction with a novel coarse-grained structure representation.  _bioRxiv_ , 2023a.  doi: 10.1101/2022.10.07.511322.
  * Lee et al. [2023b] K. L. K. Lee, C. Gonzales, M. Nassar, M. Spellings, M. Galkin, and S. Miret.  Matsciml: A broad, multi-task benchmark for solid-state materials modeling.  _arXiv preprint arXiv:2309.05934_ , 2023b.
  * Li et al. [2023] Z. Li, X. Wang, Y. Huang, and M. Zhang.  Is distance matrix enough for geometric deep learning?  _arXiv preprint arXiv: 2302.05743_ , 2023.
  * Liao and Smidt [2023] Y. Liao and T. E. Smidt.  Equiformer: Equivariant graph attention transformer for 3d atomistic graphs.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_ , 2023.
  * Lin et al. [2023] Z. Lin, H. Akin, R. Rao, B. Hie, Z. Zhu, W. Lu, N. Smetanin, R. Verkuil, O. Kabeli, Y. Shmueli, et al.  Evolutionary-scale prediction of atomic-level protein structure with a language model.  _Science_ , 379(6637):1123–1130, 2023.
  * Liu et al. [2023] S. Liu, W. Du, Y. Li, Z. Li, Z. Zheng, C. Duan, Z. Ma, O. Yaghi, A. Anandkumar, C. Borgs, et al.  Symmetry-informed geometric representation for molecules, proteins, and crystalline materials.  _arXiv preprint arXiv:2306.09375_ , 2023.
  * Liu et al. [2021] Y. Liu, L. Wang, M. Liu, X. Zhang, B. Oztekin, and S. Ji.  Spherical message passing for 3D graph networks.  _Preprint arXiv:2102.05013_ , 2021.
  * Liu et al. [2022] Y. Liu, L. Wang, M. Liu, Y. Lin, X. Zhang, B. Oztekin, and S. Ji.  Spherical message passing for 3d molecular graphs.  In _ICLR_ , 2022.
  * Long et al. [2021] T. Long, N. M. Fortunato, I. Opahle, Y. Zhang, I. Samathrakis, C. Shen, O. Gutfleisch, and H. Zhang.  Constrained crystals deep convolutional generative adversarial network for the inverse design of crystal structures.  _npj Computational Materials_ , 7(1):66, 2021\.
  * Luo and Ji [2022] Y. Luo and S. Ji.  An autoregressive flow model for 3d molecular geometry generation from scratch.  In _International Conference on Learning Representations (ICLR)_ , 2022.
  * Maas et al. [2013] A. L. Maas, A. Y. Hannun, A. Y. Ng, et al.  Rectifier nonlinearities improve neural network acoustic models.  In _Proc. icml_ , volume 30, page 3. Atlanta, GA, 2013.
  * MacRobert [1947] T. M. MacRobert.  Spherical harmonics: an elementary treatise on harmonic functions with applications.  1947\.
  * Martinkus et al. [2023] K. Martinkus, J. Ludwiczak, W.-C. Liang, J. Lafrance-Vanasse, I. Hotzel, A. Rajpal, Y. Wu, K. Cho, R. Bonneau, V. Gligorijevic, et al.  Abdiffuser: full-atom generation of in-vitro functioning antibodies.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * Mathiasen et al. [2023] A. Mathiasen, H. Helal, P. Balanca, K. Klaeser, J. Dean, C. Luschi, D. Beaini, A. W. Fitzgibbon, and D. Masters.  Repurposing density functional theory to suit deep learning.  In _1st Workshop on the Synergy of Scientific and Machine Learning Modeling @ ICML2023_ , 2023.
  * Mercado et al. [2023] R. Mercado, S. M. Kearnes, and C. W. Coley.  Data sharing in chemistry: Lessons learned and a case for mandating structured reaction data.  _Journal of Chemical Information and Modeling_ , 63(14):4253–4265, 2023.
  * Miret et al. [2023] S. Miret, K. L. K. Lee, C. Gonzales, M. Nassar, and M. Spellings.  The open matsci ML toolkit: A flexible framework for machine learning in materials science.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.
  * Monti et al. [2019] F. Monti, F. Frasca, D. Eynard, D. Mannion, and M. M. Bronstein.  Fake news detection on social media using geometric deep learning.  _arXiv preprint arXiv:1902.06673_ , 2019.
  * Morehead and Cheng [2022] A. Morehead and J. Cheng.  Geometry-complete perceptron networks for 3d molecular graphs.  _arXiv preprint arXiv:2211.02504_ , 2022.
  * Morris et al. [2019] C. Morris, M. Ritzert, M. Fey, W. L. Hamilton, J. E. Lenssen, G. Rattan, and M. Grohe.  Weisfeiler and leman go neural: Higher-order graph neural networks.  In _AAAI_ , 2019.
  * Musaelian et al. [2022] A. Musaelian, S. L. Batzner, A. Johansson, L. Sun, C. J. Owen, M. Kornbluth, and B. Kozinsky.  Learning local equivariant representations for large-scale atomistic dynamics.  _Nature Communications_ , 2022.  doi: 10.1038/s41467-023-36329-y.
  * Noé et al. [2019] F. Noé, S. Olsson, J. Köhler, and H. Wu.  Boltzmann generators: Sampling equilibrium states of many-body systems with deep learning.  _Science_ , 365(6457):eaaw1147, 2019.
  * Nouira et al. [2018] A. Nouira, J. Crivello, and N. Sokolovska.  Crystalgan: Learning to discover crystallographic structures with generative adversarial networks.  _AAAI Spring Symposium Combining Machine Learning with Knowledge Engineering_ , 2018.
  * Nusrat and Jang [2018] I. Nusrat and S.-B. Jang.  A comparison of regularization techniques in deep neural networks.  _Symmetry_ , 10:648, 2018.
  * Pakornchote et al. [2023] T. Pakornchote, N. Choomphon-anomakhun, S. Arrerut, C. Atthapak, S. Khamkaeo, T. Chotibut, and T. Bovornratanaraks.  Diffusion probabilistic models enhance variational autoencoder for crystal structure generative modeling.  _arXiv preprint arXiv:2308.02165_ , 2023.
  * Passaro and Zitnick [2023] S. Passaro and C. L. Zitnick.  Reducing so (3) convolutions to so (2) for efficient equivariant gnns.  _arXiv preprint arXiv:2302.03655_ , 2023.
  * Pozdnyakov and Ceriotti [2022] S. N. Pozdnyakov and M. Ceriotti.  Incompleteness of graph convolutional neural networks for points clouds in three dimensions.  _arXiv preprint_ , 2022.
  * Pozdnyakov and Ceriotti [2023] S. N. Pozdnyakov and M. Ceriotti.  Smooth, exact rotational symmetrization for deep learning on point clouds.  _arXiv preprint arXiv:2305.19302_ , 2023.
  * Pozdnyakov et al. [2020] S. N. Pozdnyakov, M. J. Willatt, A. P. Bartók, C. Ortner, G. Csányi, and M. Ceriotti.  Incompleteness of atomic structure representations.  _Physical Review Letters_ , 2020.
  * Puny et al. [2022] O. Puny, M. Atzmon, E. J. Smith, I. Misra, A. Grover, H. Ben-Hamu, and Y. Lipman.  Frame averaging for invariant and equivariant network design.  In _International Conference on Learning Representations_ , 2022.
  * Qi et al. [2017] C. R. Qi, L. Yi, H. Su, and L. J. Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  _Advances in neural information processing systems_ , 30, 2017.
  * Ragoza et al. [2020] M. Ragoza, T. Masuda, and D. R. Koes.  Learning a continuous representation of 3d molecular structures with deep generative models.  _arXiv preprint arXiv:2010.08687_ , 2020.
  * Ramachandran et al. [2017] P. Ramachandran, B. Zoph, and Q. V. Le.  Swish: a self-gated activation function.  _arXiv preprint arXiv:1710.05941_ , 7, 2017.
  * Ramakrishnan et al. [2014] R. Ramakrishnan, P. O. Dral, M. Rupp, and O. A. Von Lilienfeld.  Quantum chemistry structures and properties of 134 kilo molecules.  _Scientific data_ , 1(1):1–7, 2014.
  * Rao et al. [2021] R. M. Rao, J. Liu, R. Verkuil, J. Meier, J. Canny, P. Abbeel, T. Sercu, and A. Rives.  Msa transformer.  In _International Conference on Machine Learning_ , pages 8844–8856. PMLR, 2021.
  * Ren et al. [2022] Z. Ren, S. I. P. Tian, J. Noh, F. Oviedo, G. Xing, J. Li, Q. Liang, R. Zhu, A. G. Aberle, S. Sun, et al.  An invertible crystallographic representation for general inverse design of inorganic crystals with targeted properties.  _Matter_ , 5(1):314–335, 2022.
  * Richardson et al. [2022] L. Richardson, B. Allen, G. Baldi, M. Beracochea, M. L. Bileschi, T. Burdett, J. Burgin, J. Caballero-Pérez, G. Cochrane, L. J. Colwell, T. Curtis, A. Escobar-Zepeda, T. A. Gurbich, V. Kale, A. Korobeynikov, S. Raj, A. B. Rogers, E. Sakharova, S. Sanchez, D. J. Wilkinson, and R. D. Finn.  MGnify: the microbiome sequence data analysis resource in 2023.  _Nucleic Acids Research_ , 51(D1):D753–D759, Dec. 2022.  doi: 10.1093/nar/gkac1080.
  * Riebesell et al. [2023] J. Riebesell, R. E. Goodall, A. Jain, P. Benner, K. A. Persson, and A. A. Lee.  Matbench discovery-an evaluation framework for machine learning crystal stability prediction.  _arXiv preprint arXiv:2308.14920_ , 2023.
  * Rowley et al. [1998] H. A. Rowley, S. Baluja, and T. Kanade.  Rotation invariant neural network-based face detection.  In _Proceedings. 1998 IEEE computer society conference on computer vision and pattern recognition (Cat. No. 98CB36231)_ , pages 38–44. IEEE, 1998.
  * Ruhe et al. [2023] D. Ruhe, J. Brandstetter, and P. Forré.  Clifford group equivariant neural networks.  _arXiv preprint arXiv:2305.11141_ , 2023.
  * Sanchez-Gonzalez et al. [2020] A. Sanchez-Gonzalez, J. Godwin, T. Pfaff, R. Ying, J. Leskovec, and P. Battaglia.  Learning to simulate complex physics with graph networks.  In _International conference on machine learning_ , pages 8459–8468. PMLR, 2020.
  * Satorras et al. [2021a] V. G. Satorras, E. Hoogeboom, F. Fuchs, I. Posner, and M. Welling.  E(n) equivariant normalizing flows.  _Neural Information Processing Systems_ , 2021a.
  * Satorras et al. [2021b] V. G. Satorras, E. Hoogeboom, and M. Welling.  E(n) equivariant graph neural networks.  In _International conference on machine learning_ , pages 9323–9332. PMLR, 2021b.
  * Scarselli et al. [2008] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini.  The graph neural network model.  _IEEE transactions on neural networks_ , 20(1):61–80, 2008.
  * Schaarschmidt et al. [2022] M. Schaarschmidt, M. Rivière, A. Ganose, J. Spencer, A. Gaunt, J. Kirkpatrick, S. Axelrod, P. Battaglia, and J. Godwin.  Learned force fields are ready for ground state catalyst discovery.  _ARXIV.ORG_ , 2022.  doi: 10.48550/arXiv.2209.12466.
  * Schneuing et al. [2022] A. Schneuing, Y. Du, C. Harris, A. Jamasb, I. Igashov, W. Du, T. Blundell, P. Lió, C. Gomes, M. Welling, et al.  Structure-based drug design with equivariant diffusion models.  _arXiv preprint_ , 2022.
  * Schütt et al. [2017] K. Schütt, P.-J. Kindermans, H. E. Sauceda Felix, S. Chmiela, A. Tkatchenko, and K.-R. Müller.  Schnet: A continuous-filter convolutional neural network for modeling quantum interactions.  _Advances in neural information processing systems_ , 30, 2017.
  * Schütt et al. [2021a] K. Schütt, O. Unke, and M. Gastegger.  Equivariant message passing for the prediction of tensorial properties and molecular spectra.  In _International Conference on Machine Learning_ , pages 9377–9388. PMLR, 2021a.
  * Schütt et al. [2021b] K. Schütt, O. Unke, and M. Gastegger.  Equivariant message passing for the prediction of tensorial properties and molecular spectra.  In _ICML_ , 2021b.
  * Schütt et al. [2018] K. T. Schütt, H. E. Sauceda, P.-J. Kindermans, A. Tkatchenko, and K.-R. Müller.  Schnet-a deep learning architecture for molecules and materials.  _The Journal of Chemical Physics_ , 148(24):241722, 2018.
  * Sherstinsky [2020] A. Sherstinsky.  Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network.  _Physica D: Nonlinear Phenomena_ , 404:132306, 2020.
  * Shuaibi et al. [2021] M. Shuaibi, A. Kolluru, A. Das, A. Grover, A. Sriram, Z. Ulissi, and C. L. Zitnick.  Rotation invariant graph neural networks using spin convolutions.  _Preprint arXiv:2106.09575_ , 2021.
  * Sperduti and Starita [1997] A. Sperduti and A. Starita.  Supervised neural networks for the classification of structures.  _IEEE Transactions on Neural Networks_ , 8(3):714–735, 1997.
  * Srivastava et al. [2014] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15(56):1929–1958, 2014.
  * Steed and Atwood [2022] J. W. Steed and J. L. Atwood.  _Supramolecular chemistry_.  John Wiley & Sons, 2022.
  * Stokes et al. [2020] J. M. Stokes, K. Yang, K. Swanson, W. Jin, A. Cubillos-Ruiz, N. M. Donghia, C. R. MacNair, S. French, L. A. Carfrae, Z. Bloom-Ackermann, et al.  A deep learning approach to antibiotic discovery.  _Cell_ , 2020.
  * Thölke and De Fabritiis [2022] P. Thölke and G. De Fabritiis.  Torchmd-net: Equivariant transformers for neural network based molecular potentials.  _Preprint arXiv:2202.02541_ , 2022.
  * Thomas et al. [2018] N. Thomas, T. Smidt, S. Kearnes, L. Yang, L. Li, K. Kohlhoff, and P. Riley.  Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds.  _Preprint arXiv:1802.08219_ , 2018.
  * Thompson et al. [2022] A. P. Thompson, H. M. Aktulga, R. Berger, D. S. Bolintineanu, W. M. Brown, P. S. Crozier, P. J. in’t Veld, A. Kohlmeyer, S. G. Moore, T. D. Nguyen, et al.  Lammps-a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales.  _Computer Physics Communications_ , 271:108171, 2022.
  * Tran et al. [2023] R. Tran, J. Lan, M. Shuaibi, B. M. Wood, S. Goyal, A. Das, J. Heras-Domingo, A. Kolluru, A. Rizvi, N. Shoghi, et al.  The open catalyst 2022 (oc22) dataset and challenges for oxide electrocatalysts.  _ACS Catalysis_ , 13(5):3066–3084, 2023.
  * Uhrin [2021] M. Uhrin.  Through the eyes of a descriptor: Constructing complete, invertible descriptions of atomic environments.  _Physical Review B_ , 104(14):144110, 2021.
  * Unke and Meuwly [2019] O. T. Unke and M. Meuwly.  PhysNet: A neural network for predicting energies, forces, dipole moments, and partial charges.  _Journal of chemical theory and computation_ , 15(6):3678–3693, 2019.
  * Van Der Spoel et al. [2005] D. Van Der Spoel, E. Lindahl, B. Hess, G. Groenhof, A. E. Mark, and H. J. Berendsen.  Gromacs: fast, flexible, and free.  _Journal of computational chemistry_ , 26(16):1701–1718, 2005.
  * van Kempen et al. [2023] M. van Kempen, S. S. Kim, C. Tumescheit, M. Mirdita, J. Lee, C. L. M. Gilchrist, J. Söding, and M. Steinegger.  Fast and accurate protein structure search with foldseek.  _Nature Biotechnology_ , May 2023.  doi: 10.1038/s41587-023-01773-0.
  * Varadi et al. [2021] M. Varadi, S. Anyango, M. Deshpande, S. Nair, C. Natassia, G. Yordanova, D. Yuan, O. Stroe, G. Wood, A. Laydon, A. $\mathbf{Z}$ídek, T. Green, K. Tunyasuvunakool, S. Petersen, J. Jumper, E. Clancy, R. Green, A. Vora, M. Lutfi, M. Figurnov, A. Cowie, N. Hobbs, P. Kohli, G. Kleywegt, E. Birney, D. Hassabis, and S. Velankar.  AlphaFold protein structure database: massively expanding the structural coverage of protein-sequence space with high-accuracy models.  _Nucleic Acids Research_ , 50(D1):D439–D444, Nov. 2021.  doi: 10.1093/nar/gkab1061.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Velic  ković et al. [2018] P. Veli$\mathbf{c}$ković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio.  Graph Attention Networks.  _ICLR_ , 2018.
  * Velic  $kovi\'{c}et~{}al.[2018]$ P. Veli$\mathbf{c}$ković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio.  Graph attention networks.  In _ICLR_ , 2018.
  * Wang et al. [2023a] H. Wang, T. Fu, Y. Du, W. Gao, K. Huang, Z. Liu, P. Chandak, S. Liu, P. Van Katwyk, A. Deac, et al.  Scientific discovery in the age of artificial intelligence.  _Nature_ , 620(7972):47–60, 2023a.
  * Wang et al. [2022a] L. Wang, Y. Liu, Y.-C. Lin, H. Liu, and S. Ji.  ComENet: Towards complete and efficient message passing for 3D molecular graphs.  _Neural Information Processing Systems_ , 2022a.  doi: 10.48550/arXiv.2206.08515.
  * Wang et al. [2022b] L. Wang, Y. Zhou, Y. Wang, X. Zheng, X. Huang, and H. Zhou.  Regularized molecular conformation fields.  _Advances in Neural Information Processing Systems_ , 35:18929–18941, 2022b.
  * Wang et al. [2023b] L. Wang, H. Liu, Y. Liu, J. Kurtin, and S. Ji.  Learning hierarchical protein representations via complete 3d graph networks.  In _ICLR_ , 2023b.
  * Wang and Zhang [2022] X. Wang and M. Zhang.  Graph neural network with local frame for molecular potential energy surface.  In B. Rieck and R. Pascanu, editors, _Learning on Graphs Conference, LoG 2022, 9-12 December 2022, Virtual Event_ , volume 198 of _Proceedings of Machine Learning Research_ , page 19. PMLR, 2022.
  * Wang et al. [2022c] Y. Wang, Z. Li, and A. B. Farimani.  Graph neural networks for molecules.  _arXiv preprint arXiv:2209.05582_ , 2022c.
  * Wang et al. [2023c] Y. Wang, K. Yi, X. Liu, Y. G. Wang, and S. Jin.  ACMP: Allen-cahn message passing with attractive and repulsive forces for graph neural networks.  In _The Eleventh International Conference on Learning Representations_ , 2023c.  URL https://openreview.net/forum?id=4fZc_79Lrqs.
  * Watson et al. [2023] J. L. Watson, D. Juergens, N. R. Bennett, B. L. Trippe, J. Yim, H. E. Eisenach, W. Ahern, A. J. Borst, R. J. Ragotte, L. F. Milles, et al.  De novo design of protein structure and function with rfdiffusion.  _Nature_ , 620(7976):1089–1100, 2023.
  * Weiler et al. [2018] M. Weiler, M. Geiger, M. Welling, W. Boomsma, and T. S. Cohen.  3d steerable cnns: Learning rotationally equivariant features in volumetric data.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Weininger [1988] D. Weininger.  Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules.  _Journal of Chemical Information and Computer Sciences_ , 28(1):31–36, 1988.
  * Xiao et al. [2020] Y.-P. Xiao, Y.-K. Lai, F.-L. Zhang, C. Li, and L. Gao.  A survey on deep geometry learning: From a representation perspective.  _Computational Visual Media_ , 6:113–133, 2020.  doi: 10.1007/s41095-020-0174-8.  URL https://link.springer.com/article/10.1007/s41095-020-0174-8/fulltext.html.
  * Xie and Grossman [2018] T. Xie and J. C. Grossman.  Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties.  _Phys. Rev. Lett._ , 2018.
  * Xie et al. [2022] T. Xie, X. Fu, O. Ganea, R. Barzilay, and T. S. Jaakkola.  Crystal diffusion variational autoencoder for periodic material generation.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net, 2022.
  * Xu et al. [2019] K. Xu, W. Hu, J. Leskovec, and S. Jegelka.  How powerful are graph neural networks?  In _ICLR_ , 2019.
  * Xu et al. [2022] M. Xu, L. Yu, Y. Song, C. Shi, S. Ermon, and J. Tang.  Geodiff: A geometric diffusion model for molecular conformation generation.  In _International Conference on Learning Representations_ , 2022.
  * Yan et al. [2022] K. Yan, Y. Liu, Y. Lin, and S. Ji.  Periodic graph transformers for crystal material property prediction.  In _NeurIPS_ , 2022.
  * Yang et al. [2021] Z. Yang, M. Chakraborty, and A. D. White.  Predicting chemical shifts with graph neural networks.  _Chemical science_ , 12(32):10802–10809, 2021\.
  * Ying et al. [2021] C. Ying, T. Cai, S. Luo, S. Zheng, G. Ke, D. He, Y. Shen, and T.-Y. Liu.  Do transformers really perform badly for graph representation?  _Advances in Neural Information Processing Systems_ , 34:28877–28888, 2021.
  * Yu and Koltun [2016] F. Yu and V. Koltun.  Multi-scale context aggregation by dilated convolutions.  In Y. Bengio and Y. LeCun, editors, _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_ , 2016.
  * Zee [2016] A. Zee.  _Group theory in a nutshell for physicists_.  Princeton University Press, 2016.
  * Zhang et al. [2023] X. Zhang, L. Wang, J. Helwig, Y. Luo, C. Fu, Y. Xie, M. Liu, Y. Lin, Z. Xu, K. Yan, et al.  Artificial intelligence for science in quantum, atomistic, and continuum systems.  _arXiv preprint arXiv:2307.08423_ , 2023.
  * Zheng et al. [2023] S. Zheng, J. He, C. Liu, Y. Shi, Z. Lu, W. Feng, F. Ju, J. Wang, J. Zhu, Y. Min, et al.  Towards predicting equilibrium distributions for molecular systems with deep learning.  _arXiv preprint arXiv:2306.05445_ , 2023.
  * Zitnick et al. [2020] C. L. Zitnick, L. Chanussot, A. Das, S. Goyal, J. Heras-Domingo, C. Ho, W. Hu, T. Lavril, A. Palizhati, M. Riviere, et al.  An introduction to electrocatalyst design using machine learning for renewable energy storage.  _Preprint arXiv:2010.09435_ , 2020.
  * Zitnick et al. [2022] L. Zitnick, A. Das, A. Kolluru, J. Lan, M. Shuaibi, A. Sriram, Z. Ulissi, and B. Wood.  Spherical channels for modeling atomic interactions.  _Advances in Neural Information Processing Systems_ , 35:8054–8067, 2022.

##  Appendix A Lexicon

In this section, we provide background material on a variety of essential
concepts to Geometric GNNs for 3D atomic systems.

###  A.1 Geometric vocabulary

  1. 1.

Scalars: are quantities that do not have any direction or orientation and
consequently remain unchanged under rotations or reflections. For example, the
temperature or the energy of a system.

  2. 2.

(Geometric) vectors: here we use the word vector to refer to an object that
transforms with ‘simple’ rotation matrices $\mathbf{{R}}$, i.e. in the
standard representation of $SO(3)$. In contrast to the general machine
learning literature, where vectors are simply one-dimensional lists of values,
vectors in our usage of the word are objects that have both magnitude and
direction. In the context of 3D atomic systems, a geometric vector can
represent geometric attributes such as position, velocity, or forces acting on
atoms. They are often expressed as Cartesian vectors, that is, geometric
vectors that are described within a Cartesian coordinate system (i.e. they
have components along the coordinate axes $(x,y,z)$ in 3D space). They have
the same dimension as the space they exist in.

  3. 3.

Geometric Tensors: are mathematical objects that generalize geometric vectors
to higher-dimensional spaces. In contrast to general machine learning, in
which tensor just means a multi-dimensional list of numbers, we use the word
tensor to describe objects that transform in a consistent way under under
group actions (rotations in $SO(3)$ for us). Scalars and vectors are two
special subclasses of tensors that are so common that we gave them their extra
name. Objects that still transform consistently under rotations, but not
invariantly (i.e. with the identity) or through simple rotation matrices are
tensors. An example of a tensor would be the moment of inertia tensor in
physics. Tensors are essential in many areas of physics, for example in
differential geometry and general relativity, where they describe quantities
in curved spacetimes. In the context of 3D atomic systems, geometric tensors
can be used to represent higher-order geometric attributes or relationships.
For instance, the stress and strain505050 The strain tensor in a material
describes how a material’s shape deforms when subjected to mechanical loads.
In a 3D Cartesian coordinate system, the strain tensor has components
corresponding to the changes in lengths and angles in different directions.
tensors in a material are second-order geometric tensors, capturing how the
material deforms under external forces.

     * •

Cartesian Tensors: are geometric tensors whose components aligned with the
coordinate directions of a Cartesian coordinate system. These tensors are used
to represent quantities that are directly related to the physical axes of the
coordinate system.

     * •

Spherical Tensors: are used to describe quantities that are invariant under
rotations and have components aligned with spherical harmonics. They are
particularly useful in problems involving spherical symmetry and rotational
invariance.

  4. 4.

The tensor order (of a feature): refers to the way a Cartesian tensor
transforms under rotation. It indicates the level of complexity or the number
of components needed to fully describe the feature. In the context of
Geometric GNNs, features are often represented as tensors, where each
dimension corresponds to a specific aspect or property of the feature. The
tensor order determines the number of indices needed to access and manipulate
the feature’s values.

     * •

Scalar features are zero-order tensors because they do not have any additional
dimensions or indices.

     * •

Geometric vectors are first-order tensors because they have one index to
represent their components along different axes.

     * •

Geometric tensors have tensor order equal or greater than two. The strain
tensor, for instance, has a tensor order of two because it requires two
indices to access its entries. Higher-order tensors (e.g. spherical tensors)
can have a tensor order greater than 2 as they involve more complex structures
with multiple indices. These tensors can represent more intricate features,
such as higher-order interactions or relationships between elements in a
system.

  5. 5.

The type of a tensor: is often used interchangeably with the tensor order, but
they can have slightly different meaning depending on the context. In
particular, the type of a feature tells us how it changes under symmetry
transformation, i.e. how the tensor components change when the coordinate
system or basis vectors are transformed.

  6. 6.

The body order of a message, feature or model layer: refers to the number of
atoms involved in the related computations. It is often used to describe the
level of complexity (i.e. number of atoms considered) in a specific
interaction model. For example, a two-body message involves pairwise
interactions between two atoms (e.g. distances) while a three-body message
involves interactions among three atoms (e.g. bond angles). We often use many-
body interactions to describe interactions with more than 3-4 atoms [Batatia
et al., 2022b].

  7. 7.

The frame of reference, in the context of 3D atomic systems, refers to the
specific spatial arrangement and atom ordering used to represent the system.
Essentially, it is the viewpoint from which the researcher reads the atomic
system.

Going further → An Introduction to Tensors for Students of Physics and
Engineering (Kolecki, 2002) articlearticle article video article blog article
book article technical article visual article [] technicalarticle technical
video technical blog technical book technical technical technical visual
technical [] → The Tensor Product, Demystified (math3ma, 2018) blogarticle
blog video blog blog blog book blog technical blog visual blog []
visualarticle visual video visual blog visual book visual technical visual
visual visual [] → How to Conquer Tensorphobia  (Kun, 2014) blogarticle blog
video blog blog blog book blog technical blog visual blog [] technicalarticle
technical video technical blog technical book technical technical technical
visual technical []

###  A.2 Group theory

Group theory is a branch of mathematics that studies the symmetries and
transformations of objects. In the context of 3D Geometric GNN models, group
theory is particularly relevant because it helps us capture and exploit the
symmetries present in atomic systems.

Indeed, atoms in a molecule or in a material exhibit specific spatial
arrangements and undergo transformations such as rotations, reflections, and
translations. These transformations preserve the overall structure and
properties of the system. Group theory allows us to formally describe and
analyze these transformations, enabling us to uncover hidden relationships and
patterns.

Formally, a group $(\mathscr{{G}},\star)$ is a set of elements $\mathscr{{G}}$
together with a binary operation
$\star:\mathscr{{G}}\times\mathscr{{G}}\rightarrow\mathscr{{G}}$ satisfying
the following three conditions:

  1. 1.

Associativity:
$\forall\mathfrak{{g}}_{1},\mathfrak{{g}}_{2},\mathfrak{{g}}_{3}\in\mathscr{{G}},$
we have
$(\mathfrak{{g}}_{1}\star\mathfrak{{g}}_{2})\star\mathfrak{{g}}_{3}=\mathfrak{{g}}_{1}\star(\mathfrak{{g}}_{2}\star\mathfrak{{g}}_{3}).$

  2. 2.

Identity: there exists an identity element $\mathfrak{{e}}\in\mathscr{{G}}$
such that $\forall\mathfrak{{g}}\in\mathscr{{G}}$, we have
$\mathfrak{{e}}\star\mathfrak{{g}}=\mathfrak{{g}}\star\mathfrak{{e}}=\mathfrak{{g}}$

  3. 3.

Inverse: each element has an inverse - that is,
$\forall\mathfrak{{g}}\in\mathscr{{G}}$,
$\exists\mathfrak{{h}}\in\mathscr{{G}}$ such that
$\mathfrak{{g}}\star\mathfrak{{h}}=\mathfrak{{h}}\star\mathfrak{{g}}=\mathfrak{{e}}$.

We denote by $|\mathscr{{G}}|$ the size of a group $\mathscr{{G}}$, and call
this the order of $\mathscr{{G}}$. If $(\mathscr{{G}},\star)$ is a group and
$\mathscr{{H}}\in\mathscr{{G}}$ is a subset such that $(\mathscr{{H}},\star)$
satisfies the above group axioms, then we call $\mathscr{{H}}$ a subgroup of
$\mathscr{{G}}$, which we write as $\mathscr{{H}}\leq\mathscr{{G}}$.

Now that we have seen what a group is, let’s see the different symmetry groups
of interest for 3D atomic systems, illustrated in Figure 22.

  1. 1.

$\text{E}(d)$: the Euclidean group includes translations, rotations, and
reflections. It represents all possible transformations in d-dimensional
Euclidean space.

  2. 2.

$\text{O}(d)$: the orthogonal group represents rotations and reflections.
$\text{O}(d)\leq\text{E}(d)$.

  3. 3.

$\text{SO}(d)$: the Special Orthogonal group consists of rotations without
reflections, in the d-dimensional space. $\text{SO}(d)\leq\text{O}(d)$.

  4. 4.

$\text{SE}(d)$: the Special Euclidean group combines translations and
rotations. $\text{SE}(d)\leq\text{E}(d)$.

  5. 5.

$\text{T}(d)$: the Translation group represents the symmetry transformations
of pure spatial translations in d-dimensions, without any rotation or
reflection. $\text{T}(d)\leq\text{SE}(d)$.

  6. 6.

$\text{S}_{n}$: the symmetric group is the group of all permutations of the
set of $n$ atoms.

For example, for a rotation angle $\theta$, a rotation matrix around the
z-axis around would be written:

$R=\begin{bmatrix}\cos(\theta)&-\sin(\theta)&0\\\
\sin(\theta)&\cos(\theta)&0\\\ 0&0&1\\\ \end{bmatrix}\in\text{SO}(3)$

Figure 22: Illustration of the different euclidean symmetries for 3D atomic
systems. Source: [Atz et al., 2021].

What it means for Geometric GNNs? Typically, we would like Geometric GNNs to
exhibit $\text{E}(3)$ or $\text{SE}(3)$-equivariance. While rotations and
translations do not pose a concern, reflections do. In isolation, a molecule
and its mirror image share the same internal features and properties,
regardless of its chirality. Since ML datasets often showcase molecules in
isolation, E$(3)$-equivariance is desirable. However, molecular functionality
is most often conferred by intermolecular interactions with surrounding
components, meaning that a molecule’s properties may differ from those of its
mirror image. In such cases, we no longer require equivariance to reflections,
making $\text{SE}(3)$-equivariance desirable.

###  A.3 Data structures

All data structures defined below: molecules, proteins and material, are
encapsulated under the term “atomic systems“. Alternatively, we refer to them
as “molecules and materials“.

(a) Small molecule

(b) Protein

(c) Material

Figure 23: Examples of different data structures.

####  A.3.1 (Small) Molecules

A molecule is a group of atoms bonded together in a non-periodic manner,
forming the basic unit of a substance. Molecules can exist in different forms
like gases, liquids, or solids. Atoms, which are the building blocks of
molecules, combine in specific ways to create molecules with unique structures
and properties. Scientists analyze molecular structures to understand how they
interact, participate in chemical reactions, and contribute to the properties
of substances. This knowledge leads to advancements in various domains
including medicine,, technology, environmental science, etc.

In the field of machine learning and computational chemistry, molecules are
typically described using different representations (see Figure 24).
Traditional approaches have focused on 1D descriptions such as molecular
fingerprints or SMILES strings, and 2D topology graphs, where atoms and bonds
are represented by nodes and edges, respectively. To construct meaningful
representations that capture the molecule’s topology, scientists have opted
for Graph Neural Networks on 2D graphs due to their ability to efficiently
account for atomic interactions through message passing mechanisms.

Going beyond 2D graphs, there has been an increasing recognition of the
importance of considering the 3D geometric conformations515151conformations
represent the 3D structure of an atomic system: the specific arrangement of
atoms, their bond lengths, bond angles, and torsion angles. Different
conformations often impact molecular properties. of molecules for property
prediction tasks. This highlights the necessity of leveraging 3D structures to
enhance the understanding and prediction of molecular properties.

Figure 24: Exemplary molecular representations for a targeted molecule (i.e.,
the penam substructure of penicillin). a. 2D Kekulé structure. b. 2D molecular
graph. c. SMILES string Weininger [1988], in which atom type, bond type and
connectivity are specified by alphanumerical characters. d. 3D graph with atom
positions. e. Molecular surface represented as a mesh colored according to the
respective atom types. Source: [Atz et al., 2021].

Going further → Molecular representations for machine learning applications in
chemistry (Raghunathan, 2021) articlearticle article video article blog
article book article technical article visual article [] visualarticle visual
video visual blog visual book visual technical visual visual visual []
technicalarticle technical video technical blog technical book technical
technical technical visual technical []

####  A.3.2 Proteins

A protein is a large molecule that plays a crucial role in various biological
processes in living organisms. Protein sequences are made up of smaller units
called amino acids, which are connected together in a specific order to form a
long chain called a polypeptide.

Proteins have a unique 3D structure that is essential for their function. This
structure can be divided into different levels. The primary structure refers
to the linear sequence of amino acids in the polypeptide chain. The secondary
structure describes the local folding patterns that arise from interactions
between neighboring amino acids. Examples of secondary structures include
helices and sheets. The tertiary structure represents the overall 3D
arrangement of the protein, including its folds and twists. Finally, in some
cases, multiple protein chains can come together to form a complex known as
the quaternary structure.

Scientists can use ML to modelize and analyse the 3D structure of proteins.
Doing so, they can gain insights on how it interacts with other molecules and
performs specific tasks in the body. This understanding is crucial for various
fields including medicine, buichemistry and drug discovery.

Going further → Protein–protein interaction prediction with deep learning: A
comprehensive review (Soleymani, 2022) articlearticle article video article
blog article book article technical article visual article [] → Paper list
github.com/lirongwu/awesome-protein-representation-learning

####  A.3.3 Solid-State Materials

Solid-state materials are characterised by their composition and structure
leading to unique properties making them amendable to a diverse set of
applications. Solid-state materials lose their fixed structure when they
transition to liquid or gas phases, which have less restrictions on what
spaces atoms can occupy. The solid, or condensed matter, state lends itself to
a a fixed structure and definite volume. Solids exhibiting a very regular,
periodic structure are referred to as crystals while solids with no such
positional large scale order are called amorphous.

Crystals, also known as crystalline materials, are solid substances
characterized by a periodic arrangement of atoms and molecules in all spatial
directions. A crystal structure includes atom types and positions as well as
the translational axes along which the structure repeats. Since crystal
structures extend infinitely, scientists define the smallest part needed to
fully define the crystal as the unit cell. This unit cell describes the
crystal structure and composition, as well as how the crystal is repeated in
space. This repeated ordered structure gives crystals unique shapes and
properties, e.g. transparency or high melting points, making it crucial for
the development of new materials with customised properties. As a result, ML
methods often attempt to harness this periodicity to be as accurate as
possible in their predictions.

Going further → Self-supervised learning of materials concepts from crystal
structures via deep neural networks (Suzuki, 2022) articlearticle article
video article blog article book article technical article visual article []
technicalarticle technical video technical blog technical book technical
technical technical visual technical [] → Equivariant Networks for Crystal
Structures (Kaba, 2023) articlearticle article video article blog article book
article technical article visual article []

###  A.4 Periodic boundary conditions

Periodic boundary conditions (pbc) are commonly used to represent the infinite
repeating nature of crystals and other periodic systems. Instead of
considering just a single 3D unit cell where all atoms would lie, one imagine
an infinite array of identical unit cells extending in all directions.

To incorporate this periodicity into simulations, one imposes pbc on the
simulation cell. This means that if an atom or molecule exits the cell on one
side, it re-enters on the opposite side as if it were moving through a
periodic lattice. By doing this, scientists can simulate the behavior of an
infinite crystal using a finite computational domain.

In practice, we implement pbc by defining a simulation cell $\vec{\mathbf{c}}$
and a set of cell offsets $\vec{\mathbf{o}}_{ij}=[a,b,c]$ where
$a,b,c\in\\{0,1,-1\\}$ are associated with each edge $(i,j)$ of the graph to
encode edges across neighboring unit cells. These cell offsets thus help
determine distances between pairs of atoms while considering the periodic
repetition of the crystal lattice. Distances under pbc are written
$d_{ij}=||(\mathbf{x}_{i}-\mathbf{x}_{j})+\vec{o}_{ij}\cdot\vec{\mathbf{c}}||$.
Figure 8 provides a visual illustration.

In different terms, when two atoms are adjacent across a cell boundary, we
calculate their distance by accounting for the periodicity of the lattice. For
instance, if the cells are on top of each other (wrt z-axis), the cell offset
vector would be $[0,0,-1]$ or $[0,0,1]$. By considering these pbc, we can more
accurately model the behavior of crystals and other periodic systems.

Note that, in the context of PBCs, the Euclidean group actions must be
extended to account for rotations not only within the primary unit cell but
also across its periodic replicas. This means that the Geometric GNN must also
exhibit equivariance to group actions that extend across neighboring unit
cells, allowing for a robust representation of euclidean symmetries in the
presence of periodicity.

Going further → Molecular Dynamics: Periodic Boundary Conditions (Mcelfresh,
2020) blogarticle blog video blog blog blog book blog technical blog visual
blog [] visualarticle visual video visual blog visual book visual technical
visual visual visual [] → Periodic Boundary Conditions  (Digital Research
Alliance of Canada, 2023) blogarticle blog video blog blog blog book blog
technical blog visual blog []visualarticle visual video visual blog visual
book visual technical visual visual visual []

###  A.5 Quantum chemistry background

Quantum chemistry serves as the foundation for understanding the behavior of
atoms and molecules at the atomic scale. It provides insights into the
interactions of atoms and electron leading to a better understanding of
electronic structures, molecular interactions, and the fundamental mechanisms
governing chemical processes. This section outlines some key concepts that
form the basis for machine learning projects applied to physics and chemistry.

In quantum chemistry, electrons govern how atoms form interatomic (i.e.
between atoms) bonds and thereby interact with each other. Their behaviour is
modeled using wave functions, meaning, by probability distributions expressing
the likelihood of finding an electron at a specific location around an atom.
Having knowledge about electrons’ wave functions is essential because they
hold the key to understanding how atoms in a given structure behave, therefore
unlocking insights into chemical reactions, material properties and behaviors
that are critical to understanding current materials systems and potential new
designs.

The Schrödinger equation is the fundamental equation in quantum mechanics that
governs the wavefunctions’ behavior. The equation incorporates a Hamiltonian
operator, which represents the total energy of the system. Solving the
Schrödinger equation for a given system yields the wavefunctions and
corresponding energy levels for its particles. However, it is mathematically
intractable beyond the Hydrogen atom.

The Density Functional Theory (DFT) is a computational approach which
determines the electronic structure of molecules and solids by approximating
electron density, i.e. the Schrödinger equation using diverse sets of
functionals for different atomic systems. While computationally tractable for
many systems, running DFT remains computationally expensive and becomes
impractical for intricate systems, we propose the use ML to efficiently
approximate DFT calculations. ML models hold the potential to significantly
reduce computational costs while maintaining accuracy, making them invaluable
tools for researchers.

For instance, ML models that efficiently and accurately approximate DFT-based
energy calculations could be used to model the relationship between the
potential energy of a molecule and its nuclear coordinates, called the
potential energy surface (PES). The PES provides valuable insights into
molecular stability, chemical reactions, and the geometry of molecules.
Analyzing it helps us understand how molecules interact, react, and transition
between different energy states.

Opinion While a deep understanding of quantum chemistry is not “required” to
use and develop Geometric GNNs, being familiar with basic quantum chemistry
principles may enhance your comprehension of the field as well as your ability
to design more meaningful models. Besides, running DFT (or any other quantum
chemistry numerical method) remains essential to construct bigger and more
versatile databases to train machine learning models on.

Going further → How do we model atoms? videoarticle video video video blog
video book video technical video visual video []

###  A.6 Energy conservation

The potential energy of a system represents the energy stored within it. It
arises from the interplay of attractive and repulsive forces between atoms,
which are determined by factors such as atom types and atom positions. The
potential energy represents the energy that can be released when these