# Directional Message Passing on Molecular Graphs
via Synthetic Coordinates

Johannes Gasteiger, Chandan Yeshwanth, Stephan Günnemann
Technical University of Munich, Germany
<EMAIL_ADDRESS>

###### Abstract

Graph neural networks that leverage coordinates via directional message
passing have recently set the state of the art on multiple molecular property
prediction tasks. However, they rely on atom position information that is
often unavailable, and obtaining it is usually prohibitively expensive or even
impossible. In this paper we propose synthetic coordinates that enable the use
of advanced GNNs without requiring the true molecular configuration. We
propose two distances as synthetic coordinates: Distance bounds that specify
the rough range of molecular configurations, and graph-based distances using a
symmetric variant of personalized PageRank. To leverage both distance and
angular information we propose a method of transforming normal graph neural
networks into directional MPNNs. We show that with this transformation we can
reduce the error of a normal graph neural network by $55\text{\,}\mathrm{\char
37\relax}$ on the ZINC benchmark. We furthermore set the state of the art on
ZINC and coordinate-free QM9 by incorporating synthetic coordinates in the SMP
and DimeNet++ models. Our implementation is available online.
111https://www.daml.in.tum.de/synthetic-coordinates

##  1 Introduction

Graph neural networks (GNNs) have set the state of the art on many tasks of
molecular machine learning, such as the prediction of quantum mechanical
properties (Gilmer et al., 2017), solubility (Wu et al., 2018), or the
generation of new molecules (Jin et al., 2020). Thanks to their fast inference
time, good generalization and scalability, GNNs are thus promising to
revolutionize large parts of chemistry, from ab-initio quantum mechanical
simulations and reaction kinetics to synthesis planning and drug discovery.
Atom positions are central to many of these tasks, but unavailable in most
cases. Many tasks in chemistry instead use a more coarse-grained
representation: The molecular graph. Unfortunately, this representation makes
many predictive tasks substantially harder, and GNNs have performed
significantly better when they have access to the exact molecular
configuration (Gilmer et al., 2017). Missing atom positions furthermore
preclude the use of many advanced GNNs that were developed with coordinates in
mind.

In this work we aim to fill in this information with well-defined coordinates
constructed purely from the molecular graph. Regular approximation methods for
generating atom positions often do not benefit model performance, due to the
fundamental ambiguity of molecular configurations. The energy landscape of
molecules can have multiple local minima, and a molecule can be in any of
multiple different minima, known as conformers. In this work, we propose to
circumvent this problem by incorporating the conformational ambiguity via
empirical distance _bounds_. Instead of yielding a potentially wrong
configuration, these bounds only estimate the range of viable molecular
geometries. They are thus valid regardless of which state the molecule was in
when generating the data.

A molecular configuration is fully specified by the pairwise distances between
all atoms, due to rotational, translational, and reflectional invariance. We
can thus obtain a molecular geometry from any method that provides pairwise
distances between atoms. Since directional message passing does not require
the full molecular geometry, these distances do not need to correspond to an
actual three-dimensional configuration. We leverage this generality and
propose purely graph-based distances calculated from a symmetric variant of
personalized PageRank (PPR) as a second set of coordinates. This distance
performs surprisingly well, despite incorporating no chemical knowledge. Both
the distance bounds and the symmetric PPR distance require no hand-tuning and
can be calculated efficiently, even for large molecules.

We leverage these two variants of _synthetic coordinates_ to transform regular
GNNs into directional message passing, as illustrated in Fig. 1. We first
calculate the synthetic, pairwise distances for the given molecular graph.
Based on these, we calculate the edge distances and angles between edges.
Finally, we compute the molecule’s line graph. Executing a GNN on the line
graph improves its expressivity (Garg et al., 2020) and allows us to
incorporate angular information. We use the original node and edge attributes
together with the distances as node attributes, and the obtained angles as
edge attributes. The GNN is then executed on this featurized line graph
instead of the original graph. Our experiments show this transformation can
significantly improve the performance of the underlying GNN, across multiple
models and datasets. Incorporating synthetic coordinates reduces the error of
a normal GNN by $55\text{\,}\mathrm{\char 37\relax}$, putting it on par with
the current state of the art. Our enhanced version of the SMP model (Vignac et
al., 2020) improves upon the current state of the art on ZINC by
$21\text{\,}\mathrm{\char 37\relax}$, and DimeNet++ (Gasteiger et al., 2020a)
with synthetic coordinates outcompetes previous methods on coordinate-free QM9
by $20\text{\,}\mathrm{\char 37\relax}$. In summary, our core contributions
are:

  * •

Well-defined synthetic coordinates based on node distances and simple
molecular bounds, which significantly improve the performance of GNNs for
molecules.

  * •

A general scheme of converting a normal GNN into a directional MPNN, which can
improve performance and allows incorporating both distance and angular
information.

Molecular graphPairwise distancesDistances and anglesFeaturized line
graphGNNDirectional MPNN Figure 1: Illustration of transforming a regular
molecular graph (ethanol) to a line graph with synthetic coordinates. We first
calculate all (bounds of) pairwise distances using our synthetic coordinates.
We then calculate the (bounds of) distances and angles for the molecular
graph. Finally, we convert the molecular graph to its line graph and embed the
distances and angles as features. This process allows us to convert a regular
GNN to a directional MPNN, which improves its accuracy and allows to
incorporate angular information.

##  2 Directional message passing

Graph neural networks. To use GNNs for molecules we represent them as graphs
${\mathcal{G}}=({\mathcal{V}},{\mathcal{E}})$, where the atoms define the node
set ${\mathcal{V}}$ and the interactions the edge set ${\mathcal{E}}$. These
interactions are usually the bonds of the molecular graph, but they can also
be all atoms pairs within a given cutoff of e.g.
$5\text{\,}\mathrm{\SIUnitSymbolAngstrom}$. In this work we focus on an
extension of message passing neural networks (MPNNs) (Gilmer et al., 2017).
MPNNs embed each atom $u$ separately as ${\bm{h}}_{u}\in\mathbb{R}^{H}$, and
can additionally use interaction embeddings
${\bm{e}}_{(uv)}\in\mathbb{R}^{H_{\text{e}}}$. These embeddings are updated in
each layer using messages passed between neighboring nodes, starting with the
atom features ${\bm{h}}_{u}^{(0)}={\bm{x}}_{u}^{({\mathcal{V}})}$ (e.g. its
type) and the interaction features
${\bm{e}}_{(uv)}^{(0)}={\bm{x}}_{(uv)}^{({\mathcal{E}})}$ (e.g. the bond type
or a distance representation). Extended MPNNs can be expressed via the
following two equations:

$\displaystyle{\bm{h}}_{u}^{(l+1)}$
$\displaystyle=f_{\text{update}}({\bm{h}}_{u}^{(l)},\operatorname*{Agg}_{v\in{\mathcal{N}}_{u}}[f_{\text{msg}}({\bm{h}}_{u}^{(l)},{\bm{h}}_{v}^{(l)},{\bm{e}}_{(uv)}^{(l)})]),$
(1) $\displaystyle{\bm{e}}_{(uv)}^{(l+1)}$
$\displaystyle=f_{\text{edge}}({\bm{h}}_{u}^{(l+1)},{\bm{h}}_{v}^{(l+1)},{\bm{e}}_{(uv)}^{(l)}).$
(2)

The atom and interaction update functions $f_{\text{node}}$ and
$f_{\text{edge}}$ and the message function $f_{\text{msg}}$ are learnable
functions, such as simple linear layers or arbitrarily complex neural
networks. The aggregation $\operatorname*{Agg}$ over the atom’s neighbors
${\mathcal{N}}_{u}$ is usually a simple summation.

Line graph. The directed line graph
${\mathcal{L}}({\mathcal{G}})=({\mathcal{V}}_{{\mathcal{L}}},{\mathcal{E}}_{{\mathcal{L}}})$
expresses the adjacencies between the directed edges in ${\mathcal{G}}$. Its
nodes are the directed edges of the original graph
${\mathcal{V}}_{{\mathcal{L}}}=\\{(u,v)\mid
u\in{\mathcal{V}},v\in{\mathcal{N}}_{u}\\}$. For undirected graphs like
molecular graphs, every undirected edge $\\{u,v\\}$ is split into two directed
edges $(u,v)$ and $(v,u)$. Two nodes in ${\mathcal{L}}({\mathcal{G}})$ are
connected if the corresponding edges in ${\mathcal{G}}$ share a node, i.e.
${\mathcal{E}}_{{\mathcal{L}}}=\\{((u,v),(v,w))\mid(u,v),(v,w)\in{\mathcal{V}}_{{\mathcal{L}}}\\}$.
We obtain node features for the line graph by embedding the original node and
edge features as
${\bm{x}}_{(uv)}^{({\mathcal{V}}_{{\mathcal{L}}})}=f_{\text{emb}}({\bm{x}}_{u}^{({\mathcal{V}})},{\bm{x}}_{v}^{({\mathcal{V}})},{\bm{x}}_{(uv)}^{({\mathcal{E}})})$.
The line graph can furthermore incorporate additional features for atom
triplets as edge features
${\bm{x}}_{(uvw)}^{({\mathcal{E}}_{{\mathcal{L}}})}$, such as the angle
between bonds or interactions.

Directional message passing. Directional MPNNs improve upon regular MPNNs in
two ways. First, they embed the directed messages instead of the nodes in the
graph, essentially operating on the directed line graph. Models using only
this first step are also known as directed MPNNs or line graph neural networks
(Dai et al., 2016; Yang et al., 2019; Chen et al., 2019). Directed MPNNs are
strictly more expressive than regular MPNNs (Morris et al., 2020). We can
transform any MPNN to a directed MPNN simply by executing it on the directed
line graph instead of the original graph.
Second, for graphs with nodes that are embedded in an inner product space
(such as molecules in 3D space) the directed edges correspond to directions in
that space, via
${\bm{x}}^{({\mathcal{V}}_{{\mathcal{L}}})}_{(uv)}={\bm{x}}^{({\mathcal{V}})}_{u}-{\bm{x}}^{({\mathcal{V}})}_{v}$.
Directional MPNNs leverage this connection to better represent the molecular
configuration, usually by using the angles in
${\bm{x}}_{(uvw)}^{({\mathcal{E}}_{{\mathcal{L}}})}$ (Gasteiger et al.,
2020b). To fully leverage both aspects of directional MPNNs we therefore need
some form of coordinates.

Expressivity of GNNs. A central limitation of GNNs is their inability of
distinguishing between certain non-isomorphic graphs. For example, GNNs are
not able to distinguish between a hexagon and two triangles if all nodes and
edges have the same features. More specifically, Xu et al. (2019); Morris et
al. (2019) have shown that GNNs are only as powerful as the 1-Weisfeiler-
Lehman (WL) test of isomorphism. While it is still possible to construct
indistinguishable examples for directional MPNNs, this is significantly more
difficult (Garg et al., 2020). Dym & Maron (2021) have shown that MPNNs using
SO(3) group representations and atom positions are even universal, i.e. able
to approximate any continuous function to arbitrary precision. This
demonstrates that coordinates can alleviate and even solve this central
limitation of GNNs.

##  3 Molecular configurations

To prevent any pitfalls when constructing synthetic coordinates for GNNs based
on chemical knowledge we first need to consider the properties of atomic
positions in a molecule and how they are obtained. At first glance these
positions might seem like an obvious and straightforward property. However,
molecular configurations are actually ambiguous and difficult to obtain, even
for small molecules. This misconception has even led some works to suggest
semi-supervised learning methods leveraging positions, effectively treating
them as abundant input features (Hao et al., 2020). To clarify this issue we
will next describe the complexity behind molecular configurations and how to
approximate them efficiently.

Finding molecular configurations. The atoms of a molecule can in principle be
at any arbitrary position. However, most of these configurations will lead to
an extremely high energy and are thus very unlikely to be observed in nature.
A molecular configuration thus usually refers to the atom positions at or
close to equilibrium, i.e. at the molecule’s energy minimum. To find these
positions we have to search the molecule’s energy landscape and solve a non-
convex optimization problem. This is in fact a bilevel optimization problem,
where the atom positions are optimized in the outer and the electron
wavefunctions in the inner task. These wave functions can then be ignored in
the outer task; they only influence the energy and the forces
${\bm{F}}_{i}=-\frac{\partial E}{\partial{\bm{x}}_{i}}$ acting on each atomic
nucleus. We can then use these forces for gradient-based optimization, and
avoid saddle points by using quasi-Newton methods.

Difficulties. The above optimization process is very expensive due to the
quantum mechanical (QM) computations required for optimizing the electron
wavefunction at each gradient step. It is orders of magnitude more expensive
than calculating the energy of a _given_ molecular configuration, since we
need to calculate the energy’s gradient for each optimization step.
Furthermore, the optimization will only converge to a local, and not the
global minimum. And in fact, the global minimum is not the only state of
interest — _any_ reasonably low local minimum of the energy landscape is a
valid configuration, known as a conformer. A molecule thus does not have a
unique configuration; it can be in any of these states. Their statistical
distribution and the interaction between them is central for many molecular
properties. This ambiguity of atom positions poses a fundamental limit on how
precise molecular predictions can be without knowing the exact (ensemble of)
configurations. For example, without knowing the molecule’s conformer we can
not reasonably predict its energy at a precision below roughly
$60\text{\,}\mathrm{meV}$ (Grimme, 2019) — except for small, rigid molecules
that do not have multiple conformers (e.g. benzene).

Approximating energies and forces. The most prominent way of accelerating the
process of finding a valid molecular configuration is by approximating its
most expensive part: The quantum mechanical optimization of the electron
wavefunction. There is a large hierarchy of methods with various runtime
versus accuracy trade-offs (Folmsbee & Hutchison, 2021). The cheapest class of
methods are force fields. They allow running molecular dynamics simulations
with millions of atoms, and can estimate the equilibrium structure of a small
molecule in less than one second. Force fields approximate the quantum-
mechanical interactions via a closed-form, differentiable function that only
depends on the atom positions. One common example is the Merck Molecular Force
Field (MMFF94) (Halgren, 1996). MMFF94 calculates the molecular energy based
on interatomic distances, angles, dihedral angles, and long-range interaction
terms. Each term is approximated using an analytic equation with empirically
chosen coefficients that depend on the involved atom types. Forces are
obtained via the analytical gradients ${\bm{F}}_{i}=-\frac{\partial
E}{\partial{\bm{x}}_{i}}$, and conformers via gradient-based optimization.
Generating configurations with force fields is fast enough to even generate a
large ensemble of conformers. However, the resulting conformers are highly
biased and require corrections based on expensive QM-based methods for
reasonably approximating the molecule’s true distribution (Ebejer et al.,
2012; Kanal et al., 2018).

Directly predicting the configuration. There are multiple methods that
circumvent the optimization process to quickly generate low-energy conformers
for a given molecular graph. Distance geometry methods generate conformers
using an experimental database of ideal bond lengths, bond angles, and
torsional angles (Havel, 2002). The ETKDG method combines this with empirical
torsional angle preferences (Riniker & Landrum, 2015). Multiple machine
learning methods for generating conformers have also recently been proposed
(Weinreich et al., 2021; Lemm et al., 2021).

Restrictions for ML. All of the above methods yield reasonable molecular
configurations. However, they often require many initializations and a
considerable amount of hand-tuning to yield a good result for every molecule
in a dataset. Furthermore, the obtained conformer might not even be the
correct one for the data of interest. The data could have been generated by a
different conformer or by a statistical ensemble of multiple conformers. The
configuration of a wrong conformer can cause our model to overfit to the false
training data and cause bad generalization (see Sec. 6). To solve this issue
we could try to generate an ensemble of conformers and embed their
distribution. However, cheap generation methods yield strongly biased
ensembles and would thus require expensive post-processing, defeating the
purpose of fast and scalable machine learning (ML) methods (Ebejer et al.,
2012; Kanal et al., 2018). We propose to instead solve this issue by using
less precise _synthetic_ coordinates that are easier and cheaper to obtain.

##  4 Synthetic coordinates

Molecular distance bounds. To circumvent the issues associated with
conformational ambiguity, we propose to use pairwise distance _bounds_ instead
of simple coordinates, i.e. minimum and maximum distances $d_{(\min)}$ and
$d_{(\max)}$ for every pair of atoms. These bounds only provide the chemical
information we are certain of, without being falsely accurate. Specifically,
we use the distance bounds provided by RDKit (RDKit, 2021). These bounds
provide different estimates depending on how the atoms are bonded in the
molecular graph. The edges in the molecular graph correspond to directly
bonding atoms, whose bounds are calculated as the equilibrium distance (as
parametrized in the universal force field (UFF) (Rappe et al., 1992)) plus or
minus a tolerance of $0.01\text{\,}\mathrm{\SIUnitSymbolAngstrom}$. The angles
between triplets of atoms are estimated based on bond hybridization and
whether an atom is part of a ring. The distance bounds between two-hop
neighbors are then calculated based on this angle, the bond length, and a
tolerance of $0.04\text{\,}\mathrm{\SIUnitSymbolAngstrom}$, or
$0.08\text{\,}\mathrm{\SIUnitSymbolAngstrom}$ for atoms larger than Aluminium.
Pairwise distances between higher-order neighbors are not relevant for our
method, since we only use the distances and angles of the molecular graph. The
distance bounds are then refined using the triangle inequality. Note that
these bounds depend almost exclusively on the directly involved atoms. They
thus only provide local structural information.

Based on these distance bounds we calculate three different angles for
directional MPNNs: The maximally and minimally realizable angles, and the
center angle. We obtain them using standard trigonometry, via

$\alpha_{ijk}^{(\text{a})}=\arccos\left(\frac{d_{(\text{b}),ij}^{2}+d_{(\text{b}),jk}^{2}-d_{(\text{a}),ik}^{2}}{2d_{(\text{b}),ij}d_{(\text{b}),jk}}\right),$
(3)

where $(\text{a})=(\max)$ and $(\text{b})=(\min)$ for the maximally realizable
angle, $(\text{a})=(\min)$ and $(\text{b})=(\max)$ for the minimally
realizable angle, and $(\text{a})=(\text{b})=(\text{center})$ for the center
angle, with the center distance
$d_{(\text{center})}=(d_{(\min)}+d_{(\max)})/2$. These distance and angle
bounds hold for all reasonable molecular structures and thus provide valuable,
general information for our model. Their calculation requires no hand-tuning,
takes only a few milliseconds, and worked out-of-the-box for every molecule we
investigated.

Graph-based distances. Directional MPNNs only use the distances of
interactions and the angles between interactions; they do not require a full
three-dimensional geometry. We leverage this generality to propose a second
distance based on a common graph-based proximity measure: Personalized
PageRank (PPR) (Page et al., 1998), also known as random walks with restart.
PPR measures how close two atoms in the molecular graph are by calculating the
probability that a random walker starting at atom $i$ ends up at atom $j$. At
each step, the random walker jumps to any neighbor of the current atom with
equal probability, and teleports back to the original atom $i$ with
probability $\alpha$. To satisfy the symmetry property of a metric we use a
variant of PPR that uses the symmetrically normalized transition matrix, i.e.

${\bm{\Pi}}^{\text{sppr}}=\alpha({\bm{I}}_{N}-(1-\alpha){\bm{D}}^{-1/2}{\bm{A}}{\bm{D}}^{-1/2})^{-1},$
(4)

with the teleport probability $\alpha\in(0,1]$, the adjacency matrix
${\bm{A}}$, and the diagonal degree matrix
${\bm{D}}_{ij}=\sum_{k}{\bm{A}}_{ik}\delta_{ij}$. We found that this method
works well even without considering any bond type information in ${\bm{A}}$.
We convert ${\bm{\Pi}}^{\text{sppr}}$ to a distance via

$d_{\text{sppr},ij}=\sqrt{{\bm{\Pi}}^{\text{sppr}}_{ii}+{\bm{\Pi}}^{\text{sppr}}_{jj}-2{\bm{\Pi}}^{\text{sppr}}_{ij}}.$
(5)

Figure 2: $d_{\text{sppr}}$ distance between direct neighbors on ethanol.

Note that ${\bm{\Pi}}^{\text{sppr}}$ defines a positive definite kernel, and
this is the induced distance in its reproducing kernel Hilbert space. It
therefore satisfies all properties of a metric, i.e. identity of
indiscernibles, symmetry, and the triangle inequality (Berg et al., 1984,
Chapter 3, §3). However, $d_{\text{sppr},ij}$ is a general metric and does
_not_ yield atom positions in 3D. This is a purely graph-based measure that
does not incorporate any chemical knowledge. It reflects how central an atom
is in the molecular graph, and how important another atom is to this one,
based on the overall network of bonds. It thus only helps the GNN better
reflect and process the molecular graph structure. Fig. 2 shows an example of
$d_{\text{sppr}}$ on ethanol. Since the law of cosines holds for any inner
product space we can calculate the angles for directional message passing via

$\alpha_{ijk}=\arccos\left(\frac{d_{ij}^{2}+d_{jk}^{2}-d_{ik}^{2}}{2d_{ij}d_{jk}}\right).$
(6)

Note that the bounds- and graph-based distances encode orthogonal information.
The former is solely based on the global molecular graph structure, while the
latter provides purely local chemical knowledge. Instead of just choosing one
or the other we can therefore combine both to obtain the benefits of both.

Representing distances and angles. The additional structural information can
directly be incorporated into existing models as edge features. For this
purpose, we propose to first represent the distances using $N_{\text{RBF}}$
Gaussian radial basis functions (RBF), i.e.

$h_{\text{RBF},n}(d_{ij})=\exp^{-1/2(d_{ij}-c_{n})^{2}/\sigma^{2}},$ (7)

where the Gaussian centers $c_{n}$ are set uniformly between 0 and the overall
maximum distance, $n\in[0,N_{\text{RBF}}]$, and $\sigma=c_{1}-c_{0}$ is set as
the distance between two neighboring centers. The angles are similarly
represented using $N_{\text{ABF}}$ cosine angular basis functions (ABF), i.e.

$h_{\text{ABF},n}(\alpha_{ijk})=\cos(n\alpha_{ijk}),$ (8)

with $n\in[0,N_{\text{ABF}}]$. We then transform these features using two
linear layers. The first layer is global and uses a small output dimension to
force the model to learn a well-generalizing intermediate representation. The
second layer is specific to each GNN layer, enabling more flexibility.
Overall, we obtain the distance-based edge features ${\bm{e}}_{ij}$ and angle-
based triplet features ${\bm{a}}_{ijk}$ in layer $l$ via

$\displaystyle{\bm{e}}_{ij}^{(l)}$
$\displaystyle={\bm{W}}^{(l)}_{\text{RBF}2}{\bm{W}}_{\text{RBF}1}({\bm{h}}_{\text{RBF}}(d_{ij})\|{\bm{x}}_{ij}^{({\mathcal{E}})}),$
(9) $\displaystyle{\bm{a}}_{ijk}^{(l)}$
$\displaystyle={\bm{W}}^{(l)}_{\text{ABF}2}{\bm{W}}_{\text{ABF}1}{\bm{h}}_{\text{ABF}}(\alpha_{ijk}),$
(10)

where ${\bm{W}}^{(l)}_{\text{RBF}2}$ and ${\bm{W}}^{(l)}_{\text{ABF}2}$ are
layer-wise learned weight matrices, ${\bm{W}}_{\text{RBF}1}$ and
${\bm{W}}_{\text{ABF}1}$ are global learned weight matrices, $\|$ denotes
concatenation, and ${\bm{x}}_{ij}^{({\mathcal{E}})}$ are bond (edge) features.
We can furthermore combine multiple synthetic coordinates by concatenating
their representations ${\bm{h}}_{\text{RBF}}$ and ${\bm{h}}_{\text{ABF}}$.
Note that for DimeNet++ we use the original basis transformation instead of
the one described here.

##  5 Related work

Graph neural networks. Sperduti & Starita (1997); Baskin et al. (1997)
proposed the first models resembling modern GNNs. Gori et al. (2005);
Scarselli et al. (2009) were the first to use the name GNN, but these models
are quite different to current GNNs, as described in Sec. 2. GNNs became
widely adopted after their potential in a wide range of graph-related tasks
was shown by Kipf & Welling (2017); Veličković et al. (2018); Gasteiger et al.
(2019); Defferrard et al. (2016); Bruna et al. (2014). Notably, Beaini et al.
(2021) use the Laplacian eigenvectors of a graph to enable anisotropic
aggregation in MPNNs. This approach is related to our synthetic coordinates.
However, it is not rotationally invariant w.r.t. the directions induced by the
eigenvectors, and unsuited for enabling existing directional MPNNs.

GNNs for molecules. Molecules have always played a central role in the
development of GNNs, both for the very first GNNs (Baskin et al., 1997) and
during the modern era of GNNs (Duvenaud et al., 2015; Gilmer et al., 2017).
GNNs have been particularly successful when leveraging coordinates (Schütt et
al., 2017; Unke & Meuwly, 2019), but many variants only rely on the molecular
graph (Fey et al., 2020).

Directionality in GNNs. Incorporating directionality in molecular MPNNs is
currently a very active and successful area of research. These methods can
roughly be divided into two classes: Models based on SO(3) group
representations (Thomas et al., 2018; Anderson et al., 2019), and models
incorporating directional information directly (Gasteiger et al., 2020b).
Multiple promising models have recently been proposed for both classes (Fuchs
et al., 2020; Batzner et al., 2021; Schütt et al., 2021; Liu et al., 2021;
Satorras et al., 2021). While we focus on directional message passing in this
paper, all of these methods can benefit from synthetic coordinates.

Molecular representations. Molecular fingerprints are a useful tool for
comparing molecules, e.g. for machine learning. Popular examples include
extended connectivity fingerprints (ECFP), also known as Morgan or circular
fingerprints (Rogers & Hahn, 2010), MACCS keys (Durant et al., 2002), MHFP
(Probst & Reymond, 2018), the subgraph-based RDKit fingerprint (RDKit, 2021),
and the SELFIES string representation (Krenn et al., 2020). These can be
viewed as an alternative or supplement to synthetic coordinates. However,
unlike synthetic coordinates they do not leverage the peculiarities of
directional MPNNs, and can usually only be used with regressors that are not
graph-based. Another class of molecular representations aims at better
encoding the geometry of a molecule (Faber et al., 2017). Examples include
FCHL (Faber et al., 2018; Christensen et al., 2020), smooth overlap of atomic
positions (SOAP) (Bartók et al., 2013), and atomic spectrum of London and ATM
potential (aSLATM) (Huang & von Lilienfeld, 2020). OrbNet is an example of a
GNN that enhances its input with such a representation (Qiao et al., 2020).
Obviously, none of these can be used without access to the molecular
configuration.

##  6 Experiments

###  6.1 Experimental setup

We use three common benchmarks to evaluate the proposed synthetic coordinates:
Coordinate-free QM9 (Ramakrishnan et al., 2014, CC0 license), ZINC (Irwin et
al., 2012), and ogbg-molhiv (Hu et al., 2020, MIT license). QM9 contains
various quantum mechanical properties of equilibrium conformers of small
molecules with up to nine heavy atoms. To exclude effects from regular
chemical information we use all available edge (bond types) and node features
(acceptor/donor, aromaticity, hybridization). However, unlike previous work we
do not use the Mulliken partial charges. These are computed by quantum
mechanical calculations that use the molecule’s configuration. They thus lead
to information leakage and defeat the purpose of QM9’s regression task. We use
the same data split as Brockschmidt (2020) for QM9, i.e. $10\,000$ molecules
for the validation and test sets, and the remaining $\sim 110\,000$ molecules
for training. Note that the properties in QM9 fundamentally depend on the
molecular configuration. The predictions in coordinate-free QM9 should thus be
viewed as estimates for the equilibrium configurations. There are fundamental
limits to the accuracy achievable in this setup, as discussed in Sec. 3. The
goal in ZINC is to predict the penalized logP (also called “constrained
solubility” in some works), given by
$y=\operatorname{logP}-\operatorname{SAS}-\operatorname{cycles}$ (Jin et al.,
2018), where $\operatorname{logP}$ is the water-octanol partition coefficient,
$\operatorname{SAS}$ is the synthetic accessibility score (Ertl &
Schuffenhauer, 2009), and $\operatorname{cycles}$ denotes the number of cycles
with more than six atoms. Penalized logP is a score commonly used for training
molecular generation models (Kusner et al., 2017). We use $10\,000$ training,
$1000$ validation, and $1000$ test molecules, as established by Dwivedi et al.
(2020) and provided by PyTorch Geometric (Fey & Lenssen, 2019). For ogbg-
molhiv we need to predict whether a molecule inhibits HIV virus replication.
It contains $41\,127$ graphs, out of which $80\text{\,}\mathrm{\char
37\relax}$ are training samples, and $10\text{\,}\mathrm{\char 37\relax}$ each
are validation and test samples, as provided by the official data splits. We
report the mean and standard deviation across five runs for ZINC and ogbg-
molhiv. Due to computational constraints we only report single results on QM9.
The experiments were run on GPUs using an internal cluster equipped mainly
with NVIDIA GeForce GTX 1080Ti.

We aim to answer two questions with our experiments: 1. Do synthetic
coordinates improve the performance of existing GNNs? 2. Does transforming
existing GNNs to directional MPNNs improve accuracy? To answer these questions
we investigate three GNNs: DeeperGCN (Li et al., 2020, MIT license),
structural message passing (SMP) (Vignac et al., 2020, MIT license), and
DimeNet++ (Gasteiger et al., 2020a, Hippocratic license). We show step-by-step
how the changes affect the resulting error of each model. For easier
comparison we also provide published results of other state-of-the-art GNNs:
Gated GCN, MPNN-JT (Fey et al., 2020), GIN (Xu et al., 2019; Fey et al.,
2020), PNA (Corso et al., 2020), DGN (Beaini et al., 2021), SMP (Vignac et
al., 2020), GNN-FiLM (Brockschmidt, 2020), and GNN-FiLM+FA (Alon & Yahav,
2021).

###  6.2 Model hyperparameters

To prevent overfitting we use the SMP and DimeNet++ models and hyperparameters
largely as-is, without any further optimization. Similarly, we chose the
DeeperGCN variant and hyperparameters based on the ogbg-molhiv dataset, and
did not further tune on ZINC. More specifically, we use the DeeperGCN (Li et
al., 2020) with 12 ResGCN+ blocks, mean aggregation in the graph convolution,
and average pooling to obtain the graph embedding. For SMP (Vignac et al.,
2020) we use 12 layers, 8 towers, an internal representation of size 32 and no
residual connections. For both DeeperGCN and SMP we use an embedding size of
256, and distance and angle bases of size $16$ and $18$, respectively, with a
bottleneck dimension of $4$ between the global basis embedding and the local
embedding in each layer. We train all models on ZINC with the same training
hyperparameters as SMP, particularly the same learning rate schedule with a
patience of 100 and minimum learning rate of $1\text{\times}{10}^{-5}$.

For DimeNet++ we use a cutoff of $2.5\text{\,}\mathrm{\SIUnitSymbolAngstrom}$,
radial and spherical bases of size 12, embedding size 128, output embedding
size 256, basis embedding size 8 and 4 blocks. We use the same optimization
parameters - learning rate $0.001$, $3000$ warmup steps and a decay rate of
$0.01$.

###  6.3 Results

Table 1: Ablation study for transforming DeeperGCN and SMP to directional MPNNs (MAE on ZINC). Every step improves the error of DeeperGCN, resulting in a $55\text{\,}\mathrm{\char 37\relax}$ improvement. The combined bounds+PPR encoding performs best. *Replicated using the reference implementation. |  | SMP | DeeperGCN
---|---|---|---
Basic |  |  $0.159$ p m 0.028* |  $0.317$ p m 0.021
+distance | Bounds |  $0.124$ p m 0.002 |  $0.264$ p m 0.003
PPR |  $0.151$ p m 0.008 |  $0.227$ p m 0.006
Bounds+PPR |  $0.121$ p m 0.006 |  $0.228$ p m 0.005
+distance | Bounds |  $0.112$ p m 0.004 |  $0.212$ p m 0.008
& line graph | PPR |  $0.150$ p m 0.003 |  $0.194$ p m 0.009
+distance, | Bounds |  $0.113$ p m 0.003 |  $0.180$ p m 0.007
line graph | PPR |  $0.153$ p m 0.005 |  $0.158$ p m 0.005
& angle | Bounds+PPR |  $0.109$ p m 0.004 |  $0.142$ p m 0.006

Table 2: MAE on ZINC. SMP with synthetic coordinates outcompetes previous models by $21\text{\,}\mathrm{\char 37\relax}$, without any hyperparameter tuning. Model | MAE
---|---
Gated GCN | $0.282$
GIN | $0.252$
PNA | $0.188$
DGN | $0.168$
MPNN-JT | $0.151$
SMP | $0.138$
DeeperGCN-SC |  $0.142$ p m 0.006
SMP-SC |  $0.109$ p m 0.004

Transforming existing GNNs. Table 2 shows that DeeperGCN’s errors improve for
each step of the transformation: Adding distance information, switching to the
line graph, and adding angles. Interestingly, the PPR distance reduces the
error more than molecular distance bounds do. This suggests that this
structural information is more relevant for the GNN than the rough bounds. SMP
benefits less from using the line graph and angles. This is likely due to SMP
already encoding structural information as part of its architecture. Using
both the PPR distance and molecular distance bounds improves the performance
further for both models. Table 3 shows that using more expensive methods of
generating conformers yields a higher error than our simple and fast methods.
As discussed in Sec. 3, this can be attributed to the ambiguities of different
molecular conformers. DeeperGCN with synthetic coordinates performs similarly
well to the best models proposed previously, while the enhanced SMP sets a new
state of the art on this dataset, as shown in Table 2.

Table 3: Comparison of different distance generation methods for DeeperGCN on
ZINC (MAE). Our simpler, faster, and more principled methods (bounds, PPR)
perform better than more sophisticated conformer generation methods.

MMFF94 ETKDG Bounds PPR Bounds+PPR Distance $0.324$ p m 0.012 $0.329$ p m
0.022 $0.264$ p m 0.003 $0.227$ p m 0.006 $0.228$ p m 0.005 Distance & line
graph $0.232$ p m 0.008 $0.234$ p m 0.007 $0.212$ p m 0.008 $0.194$ p m 0.009
$0.178$ p m 0.009 Distance, line graph & angle $0.236$ p m 0.011 $0.274$ p m
0.012 $0.180$ p m 0.007 $0.158$ p m 0.005 $0.142$ p m 0.006

Table 4: MAE on coordinate-free QM9 ($\mathrm{meV}$) for DimeNet++ with synthetic coordinates. Both synthetic distances and angles yield significant improvements, together reducing the error by $24\text{\,}\mathrm{\char 37\relax}$ on average. | $\epsilon_{\text{HOMO}}$ | $U_{0}$
---|---|---
No dist/angle | $74.1$ | $41.9$
No angle | 63.5 | 32.1
(bounds+PPR)
distance & angle:
Bounds | $63.6$ | $29.4$
PPR | $63.0$ | $29.5$
Bounds+PPR | $61.7$ | $28.7$

Table 5: MAE on coordinate-free QM9. DimeNet++ with synthetic coordinates outperforms previous models by $20\text{\,}\mathrm{\char 37\relax}$, without any hyperparameter tuning. *Uses Mulliken partial charges. | Unit | GNN-FiLM* | GNN-FiLM+FA* | DimeNet++-SC
---|---|---|---|---
$\mu$ | $\mathrm{D}$ | 0.238 | 0.226 | 0.303
$\alpha$ | $\mathrm{\text{ $a_{0}$ }}^{3}$ | 0.375 | 0.193 | 0.171
$\epsilon_{\text{HOMO}}$ | $\mathrm{meV}$ | 52.5 | 47.7 | 61.7
$\epsilon_{\text{LUMO}}$ | $\mathrm{meV}$ | 55.9 | 52 | 54.3
$\Delta\epsilon$ | $\mathrm{meV}$ | 84.3 | 77 | 86.2
$\left<R^{2}\right>$ | $\mathrm{\text{ $a_{0}$ }}^{2}$ | 18.7 | 14.3 | 12.7
ZPVE | $\mathrm{meV}$ | 13.2 | 5.62 | 2.98
$U_{0}$ | $\mathrm{meV}$ | 233 | 68.8 | 28.7
$U$ | $\mathrm{meV}$ | 256 | 75.2 | 29.6
$H$ | $\mathrm{meV}$ | 240 | 83 | 29.6
$G$ | $\mathrm{meV}$ | 222 | 76.1 | 28.2
$c_{\text{v}}$ | $\frac{\mathrm{cal}}{\mathrm{mol}\text{\,}\mathrm{K}}$ | 0.173 | 0.082 | 0.076

Enhancing DimeNet++. DimeNet was originally developed for molecular dynamics
and other use cases that provide the true atom positions, such as the full QM9
dataset. Despite this, we can still use it as-is without available positional
information by setting the used distance and angle embeddings to constants.
DimeNet++ still performs surprisingly well in this form, as shown in Table 5.
However, its performance increases significantly if we provide it with the
proposed synthetic coordinates. Notably, the PPR distance again causes a
larger improvement than the molecular distance bounds. Combining both
distances still performs best, though. Table 5 furthermore shows that
DimeNet++ sets the state of the art for coordinate-less QM9 on eight out of
twelve targets — without any further hyperparameter optimization.
Interestingly, the achieved energy error lies significantly below the limit of
$60\text{\,}\mathrm{meV}$ we mentioned in Sec. 3. This is likely due to two
reasons. First, QM9 only contains small molecules, many of which are very
rigid. These molecules do not have multiple conformers and their energy will
thus be more deterministic. Second, QM9’s data was generated by initializing
each molecule’s position with a fast force field method. This can bias the
final conformer towards a deterministic state, which might be learnable by a
GNN.

Table 6: Ablation of DeeperGCN on QM9 $U_{0}$ (MAE, $\mathrm{meV}$) and ogbg-molhiv (ROC-AUC). Using the line graph does not always provide benefits. However, synthetic coordinates help even in these cases. |  | QM9, $U_{0}$ | ogbg-molhiv
---|---|---|---
Basic |  | $106$ |  $0.728$ p m 0.008
+distance | Bounds | $114$ |  $0.724$ p m 0.014
PPR | $88$ |  $0.734$ p m 0.014
Bounds+PPR | $100$ |  $0.733$ p m 0.024
+distance | Bounds | $233$ |  $0.705$ p m 0.011
& line graph | PPR | $204$ |  $0.697$ p m 0.009
+distance, | Bounds | $205$ |  $0.703$ p m 0.021
line graph | PPR | $164$ |  $0.700$ p m 0.014
& angle | Bounds+PPR | $186$ |  $0.767$ p m 0.016

Synthetic coordinates without directional message passing. In some cases we
found that using synthetic coordinates yields performance improvements while
transforming the model to a directional MPNN does not. Table 6 demonstrates
this using DeeperGCN on QM9 and ogbg-molhiv. Using the line graph
significantly impairs performance on both datasets. Whether directional MPNNs
provide a benefit thus seems to depend on both the underlying model and the
dataset. This is likely due to the directional MPNN’s different training
dynamics, which require further architectural and hyperparameter changes.
Moreover, directional MPNNs are likely more prone to overfitting due to their
better expressivity. This affects ogbg-molhiv in particular, since it uses a
scaffold split for the test set. However, the additional information provided
by synthetic coordinates still yields improvements in both cases.

##  7 Limitations and societal impact

Limitations. Converting a GNN to a directional MPNN incurs significant
computational overhead, since the line graph is usually substantially larger
than the molecular graph. However, just incorporating the information provided
by graph distances or molecular distance bounds without transforming to
directional message passing can also provide benefits, with almost no
computational overhead. We furthermore found that transforming a GNN to a
directional MPNN does not yield improvements in many cases, while synthetic
coordinates still do (see Sec. 6 for details). There are likely also cases
where synthetic coordinates lead to overfitting and do not improve accuracy.
Directional MPNNs appear to be most successful when the molecular
configuration is directly relevant.

Societal impact. Improving the predictions of molecular models can positively
affect various applications in chemistry, biology, and medicine. Our research
is general and not focused on a field where malicious use should be expected.
However, similar to most methodological research, our improvements can be
misused to accelerate the development of chemical agents and biological
weapons. We do not think that this potential for harm goes beyond regular
research in theoretical chemistry and related fields. Still, to slightly
reduce these negative effects our code will be published under the Hippocratic
license (Ehmke, 2020).

##  8 Conclusion

We proposed two methods for providing synthetic coordinates: Molecular
distance bounds based on the interacting atom types, and graph-based distances
based on personalized PageRank scores. Both of these methods provide well-
defined pairwise distances, which can then be used to calculate distances for
edge features in the molecular graph, and angles for edge features in its line
graph. These synthetic coordinates improve GNN performance for various models
and datasets, and allow transforming a regular GNN into a directional MPNN.
This transformation leads to substantial improvements, resulting in state-of-
the-art accuracies on multiple datasets.

## Acknowledgments and Disclosure of Funding

We would like to thank Johannes T. Margraf, Nicholas Gao, and Oleksandr Shchur
for their invaluable advice and comments.

This research was supported by the Deutsche Forschungsgemeinschaft (DFG)
through the Emmy Noether grant GU 1409/2-1 and the TUM International Graduate
School of Science and Engineering (IGSSE), GSC 81.

## References

  * Alon & Yahav (2021) Uri Alon and Eran Yahav.  On the Bottleneck of Graph Neural Networks and its Practical Implications.  In _ICLR_ , 2021.
  * Anderson et al. (2019) Brandon M. Anderson, Truong-Son Hy, and Risi Kondor.  Cormorant: Covariant Molecular Neural Networks.  In _NeurIPS_ , 2019.
  * Bartók et al. (2013) Albert P. Bartók, Risi Kondor, and Gábor Csányi.  On representing chemical environments.  _Physical Review B_ , 87(18):184115, 2013.
  * Baskin et al. (1997) Igor I. Baskin, Vladimir A. Palyulin, and Nikolai S. Zefirov.  A Neural Device for Searching Direct Correlations between Structures and Properties of Chemical Compounds.  _Journal of Chemical Information and Computer Sciences_ , 37(4):715–721, 1997.
  * Batzner et al. (2021) Simon Batzner, Tess E. Smidt, Lixin Sun, Jonathan P. Mailoa, Mordechai Kornbluth, Nicola Molinari, and Boris Kozinsky.  SE(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials.  _arXiv_ , 2101.03164, 2021.
  * Beaini et al. (2021) Dominique Beaini, Saro Passaro, Vincent Létourneau, William L. Hamilton, Gabriele Corso, and Pietro Liò.  Directional Graph Networks.  In _ICML_ , 2021.
  * Berg et al. (1984) Christian Berg, Jens Peter Reus Christensen, and Paul Ressel.  _Harmonic Analysis on Semigroups_.  Number 100 in Graduate Texts in Mathematics. 1984.
  * Brockschmidt (2020) Marc Brockschmidt.  GNN-FiLM: Graph Neural Networks with Feature-wise Linear Modulation.  In _ICML_ , 2020.
  * Bruna et al. (2014) Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun.  Spectral Networks and Deep Locally Connected Networks on Graphs.  In _ICLR_ , 2014.
  * Chen et al. (2019) Zhengdao Chen, Lisha Li, and Joan Bruna.  Supervised Community Detection with Line Graph Neural Networks.  In _ICLR_ , 2019.
  * Christensen et al. (2020) Anders S. Christensen, Lars A. Bratholm, Felix A. Faber, and O. Anatole von Lilienfeld.  FCHL revisited: Faster and more accurate quantum machine learning.  _The Journal of Chemical Physics_ , 152(4):044107, 2020.
  * Corso et al. (2020) Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković.  Principal Neighbourhood Aggregation for Graph Nets.  In _NeurIPS_ , 2020.
  * Dai et al. (2016) Hanjun Dai, Bo Dai, and Le Song.  Discriminative Embeddings of Latent Variable Models for Structured Data.  In _ICML_ , 2016.
  * Defferrard et al. (2016) Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering.  In _NeurIPS_ , 2016.
  * Durant et al. (2002) Joseph L. Durant, Burton A. Leland, Douglas R. Henry, and James G. Nourse.  Reoptimization of MDL Keys for Use in Drug Discovery.  _Journal of Chemical Information and Computer Sciences_ , 42(6):1273–1280, 2002.
  * Duvenaud et al. (2015) David K. Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Gómez-Bombarelli, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams.  Convolutional Networks on Graphs for Learning Molecular Fingerprints.  In _NeurIPS_ , 2015.
  * Dwivedi et al. (2020) Vijay Prakash Dwivedi, Chaitanya K. Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson.  Benchmarking Graph Neural Networks.  _arXiv_ , 2003.00982, 2020.
  * Dym & Maron (2021) Nadav Dym and Haggai Maron.  On the Universality of Rotation Equivariant Point Cloud Networks.  In _ICLR_ , 2021.
  * Ebejer et al. (2012) Jean-Paul Ebejer, Garrett M. Morris, and Charlotte M. Deane.  Freely Available Conformer Generation Methods: How Good Are They?  _Journal of Chemical Information and Modeling_ , 52(5):1146–1158, 2012.
  * Ehmke (2020) Coraline Ada Ehmke.  The Hippocratic License 2.1: An Ethical License for Open Source., 2020.  URL https://firstdonoharm.dev.
  * Ertl & Schuffenhauer (2009) Peter Ertl and Ansgar Schuffenhauer.  Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions.  _Journal of Cheminformatics_ , 1(1):8, 2009.
  * Faber et al. (2017) Felix A. Faber, Luke Hutchison, Bing Huang, Justin Gilmer, Samuel S. Schoenholz, George E. Dahl, Oriol Vinyals, Steven Kearnes, Patrick F. Riley, and O. Anatole von Lilienfeld.  Prediction Errors of Molecular Machine Learning Models Lower than Hybrid DFT Error.  _Journal of Chemical Theory and Computation_ , 13(11):5255–5264, 2017.
  * Faber et al. (2018) Felix A. Faber, Anders S. Christensen, Bing Huang, and O. Anatole von Lilienfeld.  Alchemical and structural distribution based representation for universal quantum machine learning.  _The Journal of Chemical Physics_ , 148(24):241717, 2018.
  * Fey & Lenssen (2019) Matthias Fey and Jan E. Lenssen.  Fast Graph Representation Learning with PyTorch Geometric.  In _Workshop on Representation Learning on Graphs and Manifolds, ICLR_ , 2019.
  * Fey et al. (2020) Matthias Fey, Jan-Gin Yuen, and Frank Weichert.  Hierarchical Inter-Message Passing for Learning on Molecular Graphs.  In _Workshop on Graph Representation Learning and Beyond, ICML_ , 2020.
  * Folmsbee & Hutchison (2021) Dakota Folmsbee and Geoffrey Hutchison.  Assessing conformer energies using electronic structure and machine learning methods.  _International Journal of Quantum Chemistry_ , 121(1):e26381, 2021.
  * Fuchs et al. (2020) Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling.  SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks.  In _NeurIPS_ , 2020.
  * Garg et al. (2020) Vikas Garg, Stefanie Jegelka, and Tommi Jaakkola.  Generalization and Representational Limits of Graph Neural Networks.  In _ICML_ , 2020.
  * Gasteiger et al. (2019) Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann.  Predict then Propagate: Graph Neural Networks Meet Personalized PageRank.  In _ICLR_ , 2019.
  * Gasteiger et al. (2020a) Johannes Gasteiger, Shankari Giri, Johannes T. Margraf, and Stephan Günnemann.  Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Molecules.  In _Machine Learning for Molecules Workshop, NeurIPS_ , 2020\.
  * Gasteiger et al. (2020b) Johannes Gasteiger, Janek Groß, and Stephan Günnemann.  Directional Message Passing for Molecular Graphs.  In _ICLR_ , 2020.
  * Gilmer et al. (2017) Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl.  Neural Message Passing for Quantum Chemistry.  In _ICML_ , 2017.
  * Gori et al. (2005) M. Gori, G. Monfardini, and F. Scarselli.  A new model for learning in graph domains.  In _IEEE International Joint Conference on Neural Networks_ , 2005.
  * Grimme (2019) Stefan Grimme.  Exploration of Chemical Compound, Conformer, and Reaction Space with Meta-Dynamics Simulations Based on Tight-Binding Quantum Chemical Calculations.  _Journal of Chemical Theory and Computation_ , 15(5):2847–2862, 2019.
  * Halgren (1996) Thomas A. Halgren.  Merck molecular force field. I. Basis, form, scope, parameterization, and performance of MMFF94.  _Journal of Computational Chemistry_ , 17(5-6):490–519, 1996.
  * Hao et al. (2020) Zhongkai Hao, Chengqiang Lu, Zhenya Huang, Hao Wang, Zheyuan Hu, Qi Liu, Enhong Chen, and Cheekong Lee.  ASGN: An Active Semi-supervised Graph Neural Network for Molecular Property Prediction.  In _KDD_ , 2020.
  * Havel (2002) Timothy F. Havel.  Distance Geometry: Theory, Algorithms, and Chemical Applications.  In _Encyclopedia of Computational Chemistry_. 2002.
  * Hu et al. (2020) Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec.  Open Graph Benchmark: Datasets for Machine Learning on Graphs.  In _NeurIPS_ , 2020.
  * Huang & von Lilienfeld (2020) Bing Huang and O. Anatole von Lilienfeld.  Quantum machine learning using atom-in-molecule-based fragments selected on the fly.  _Nature Chemistry_ , 12(10):945–951, 2020.
  * Irwin et al. (2012) John J. Irwin, Teague Sterling, Michael M. Mysinger, Erin S. Bolstad, and Ryan G. Coleman.  ZINC: A Free Tool to Discover Chemistry for Biology.  _Journal of Chemical Information and Modeling_ , 52(7):1757–1768, 2012.
  * Jin et al. (2018) Wengong Jin, Regina Barzilay, and Tommi Jaakkola.  Junction Tree Variational Autoencoder for Molecular Graph Generation.  In _ICML_ , 2018.
  * Jin et al. (2020) Wengong Jin, Dr Regina Barzilay, and Tommi Jaakkola.  Hierarchical Generation of Molecular Graphs using Structural Motifs.  In _ICML_ , 2020.
  * Kanal et al. (2018) Ilana Y. Kanal, John A. Keith, and Geoffrey R. Hutchison.  A sobering assessment of small-molecule force field methods for low energy conformer predictions.  _International Journal of Quantum Chemistry_ , 118(5):e25512, 2018.
  * Kipf & Welling (2017) Thomas N. Kipf and Max Welling.  Semi-Supervised Classification with Graph Convolutional Networks.  In _ICLR_ , 2017.
  * Krenn et al. (2020) Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alan Aspuru-Guzik.  Self-referencing embedded strings (SELFIES): A 100% robust molecular string representation.  _Machine Learning: Science and Technology_ , 1(4):045024, 2020.
  * Kusner et al. (2017) Matt J. Kusner, Brooks Paige, and José Miguel Hernández-Lobato.  Grammar Variational Autoencoder.  In _ICML_ , 2017.
  * Lemm et al. (2021) Dominik Lemm, Guido Falk von Rudorff, and O. Anatole von Lilienfeld.  Machine learning based energy-free structure predictions of molecules, transition states, and solids.  _Nature Communications_ , 12(1):4468, 2021.
  * Li et al. (2020) Guohao Li, Chenxin Xiong, Ali Thabet, and Bernard Ghanem.  DeeperGCN: All You Need to Train Deeper GCNs.  _arXiv_ , 2006.07739, 2020.
  * Liu et al. (2021) Yi Liu, Limei Wang, Meng Liu, Xuan Zhang, Bora Oztekin, and Shuiwang Ji.  Spherical Message Passing for 3D Graph Networks.  _arXiv_ , 2102.05013, 2021.
  * Morris et al. (2019) Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe.  Weisfeiler and Leman Go Neural: Higher-Order Graph Neural Networks.  In _AAAI_ , 2019.
  * Morris et al. (2020) Christopher Morris, Gaurav Rattan, and Petra Mutzel.  Weisfeiler and Leman go sparse: Towards scalable higher-order graph embeddings.  In _NeurIPS_ , 2020.
  * Page et al. (1998) Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd.  The pagerank citation ranking: Bringing order to the web.  Report, Stanford InfoLab, 1998.
  * Probst & Reymond (2018) Daniel Probst and Jean-Louis Reymond.  A probabilistic molecular fingerprint for big data settings.  _Journal of Cheminformatics_ , 10(1):66, 2018\.
  * Qiao et al. (2020) Zhuoran Qiao, Matthew Welborn, Animashree Anandkumar, Frederick R. Manby, and Thomas F. Miller.  OrbNet: Deep learning for quantum chemistry using symmetry-adapted atomic-orbital features.  _The Journal of Chemical Physics_ , 153(12):124111, 2020.
  * Ramakrishnan et al. (2014) Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp, and O. Anatole von Lilienfeld.  Quantum chemistry structures and properties of 134 kilo molecules.  _Scientific Data_ , 1(1):1–7, 2014.
  * Rappe et al. (1992) A. K. Rappe, C. J. Casewit, K. S. Colwell, W. A. Goddard, and W. M. Skiff.  UFF, a full periodic table force field for molecular mechanics and molecular dynamics simulations.  _Journal of the American Chemical Society_ , 114(25):10024–10035, 1992.
  * RDKit (2021) RDKit.  RDKit: Open-Source Cheminformatics Software, 2021.  URL https://rdkit.org/.
  * Riniker & Landrum (2015) Sereina Riniker and Gregory A. Landrum.  Better Informed Distance Geometry: Using What We Know To Improve Conformation Generation.  _Journal of Chemical Information and Modeling_ , 55(12):2562–2574, 2015.
  * Rogers & Hahn (2010) David Rogers and Mathew Hahn.  Extended-Connectivity Fingerprints.  _Journal of Chemical Information and Modeling_ , 50(5):742–754, 2010.
  * Satorras et al. (2021) Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling.  E(n) Equivariant Graph Neural Networks.  In _ICML_ , 2021.
  * Scarselli et al. (2009) F. Scarselli, M. Gori, Ah Chung Tsoi, M. Hagenbuchner, and G. Monfardini.  The Graph Neural Network Model.  _IEEE Transactions on Neural Networks_ , 20(1):61–80, 2009.
  * Schütt et al. (2017) Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller.  SchNet: A continuous-filter convolutional neural network for modeling quantum interactions.  In _NeurIPS_ , 2017.
  * Schütt et al. (2021) Kristof T. Schütt, Oliver T. Unke, and Michael Gastegger.  Equivariant message passing for the prediction of tensorial properties and molecular spectra.  In _ICML_ , 2021.
  * Sperduti & Starita (1997) A. Sperduti and A. Starita.  Supervised neural networks for the classification of structures.  _IEEE Transactions on Neural Networks_ , 8(3):714–735, 1997.
  * Thomas et al. (2018) Nathaniel Thomas, Tess Smidt, Steven M. Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley.  Tensor Field Networks: Rotation- and Translation-Equivariant Neural Networks for 3D Point Clouds.  _arXiv_ , 1802.08219, 2018.
  * Unke & Meuwly (2019) Oliver T. Unke and Markus Meuwly.  PhysNet: A Neural Network for Predicting Energies, Forces, Dipole Moments, and Partial Charges.  _Journal of Chemical Theory and Computation_ , 15(6):3678–3693, 2019.
  * Veličković et al. (2018) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio.  Graph Attention Networks.  In _ICLR_ , 2018.
  * Vignac et al. (2020) Clement Vignac, Andreas Loukas, and Pascal Frossard.  Building powerful and equivariant graph neural networks with structural message-passing.  In _NeurIPS_ , 2020.
  * Weinreich et al. (2021) Jan Weinreich, Nicholas J. Browning, and O. Anatole von Lilienfeld.  Machine learning of free energies in chemical compound space using ensemble representations: Reaching experimental uncertainty for solvation.  _The Journal of Chemical Physics_ , 154(13):134113, 2021.
  * Wu et al. (2018) Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande.  MoleculeNet: a benchmark for molecular machine learning.  _Chemical Science_ , 9(2):513–530, 2018.
  * Xu et al. (2019) Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka.  How Powerful are Graph Neural Networks?  In _ICLR_ , 2019.
  * Yang et al. (2019) Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, Andrew Palmer, Volker Settels, Tommi Jaakkola, Klavs Jensen, and Regina Barzilay.  Analyzing Learned Molecular Representations for Property Prediction.  _Journal of Chemical Information and Modeling_ , 59(8):3370–3388, 2019.

##  Appendix A Choosing hyperparameters

Table 7: ROC-AUC on ogbg-molhiv for various numbers of DeeperGCN layers. We use 12 layers. Layers | AUC-ROC
---|---
$7$ |  $0.754$ p m 0.028
$12$ |  $0.756$ p m 0.026
$15$ |  $0.742$ p m 0.028

Table 8: ROC-AUC on ogbg-molhiv for various hidden layer sizes in DeeperGCN. We choose the largest, i.e. 256. Hidden size | AUC-ROC
---|---
$64$ |  $0.764$ p m 0.009
$128$ |  $0.760$ p m 0.026
$256$ |  $0.756$ p m 0.026

Table 9: Different methods of embedding the angle for chemical distance bounds (ROC-AUC on ogbg-molhiv). Jointly using all 3 proposed components (Min+Max+Center) works best. Angle mode | MAE
---|---
Min |  $0.754$ p m 0.048
Max |  $0.754$ p m 0.013
Center |  $0.744$ p m 0.012
Min+Max |  $0.745$ p m 0.018
Center+Min+Max |  $0.756$ p m 0.026

Table 10: ROC-AUC on ogbg-molhiv. Different parameter sharing variants for the two layers used for embedding the distance and angle. Sharing the parameters of the first layer globally and using separate parameters per message passing step for the second layer (“Mixed”) performs slightly better. Embedding method | MAE
---|---
Local |  $0.754$ p m 0.010
Global |  $0.753$ p m 0.025
Mixed |  $0.756$ p m 0.026

Table 11: Different bottleneck and basis sizes for embedding the distance and angle (ROC-AUC on ogbg-molhiv). We choose a 4-dimensional bottleneck, a 16-dimensional distance and a 18-dimensional angle embedding. Note that the latter numbers are the sum of all components, i.e. we use 8 dimensions for the minimum and 8 for the maximum distance. | Basis size |
---|---|---
Bottleneck | Distance  | Angle | MAE
2 | $4$ | $6$ |  $0.762$ p m 0.021
$8$ | $9$ |  $0.766$ p m 0.019
$16$ | $18$ |  $0.751$ p m 0.031
4 | $4$ | $6$ |  $0.743$ p m 0.016
$8$ | $9$ |  $0.756$ p m 0.026
$16$ | $18$ |  $0.767$ p m 0.016
8 | $4$ | $6$ |  $0.741$ p m 0.024
$8$ | $9$ |  $0.771$ p m 0.015
$16$ | $18$ |  $0.743$ p m 0.012

In this section we highlight the best results as well as the chosen
hyperparameters and model variants via bold font. We tune DeeperGCN on ogbg-
molhiv to prevent selection bias and overfitting. Tables 9 and 9 show its
performance for various choices of depth and width. Many of these results are
not statistically significant. We chose a depth of 12 layers and a hidden size
of 256.

Table 9 compares the three ways of representing the angle bounds described in
Sec. 4. We see that simply using all three of them performs the best. Note
that we keep the total basis size constant, i.e. we either use one
18-dimensional, two 9-dimensional, or three 6-dimensional angle bases.

We embed the information provided by synthetic coordinates using two linear
layers with a small “bottleneck” layer in between and without non-linearities.
Table 11 compares using separate local layers per message passing step against
using a single global layer, i.e. sharing these parameters. Mixing these two
variants by using one global and one local layer performs best. Table 11
furthermore compares different bottleneck and basis sizes for representing the
distances and angles. A 4- or even 2-dimensional bottleneck performs best —
which is surprisingly small compared to the used hidden size of 256. The basis
size on the other hand is similar to the size used e.g. by Gasteiger et al.
(2020b).


#  Mqt-Tz: Secure MQTT Broker for Biomedical Signal Processing on the Edge

Carlos Segarra Corresponding Author: Carlos Segarra; E-mail:
<EMAIL_ADDRESS>Ricard Delgado-Gonzalo  Valerio Schiavoni
CSEM, Neuchâtel, Switzerland  Université de Neuchâtel, Neuchâtel, Switzerland

###### Abstract

Physical health records belong to healthcare providers, but the information
contained within belongs to each patient. In an increasing manner, more
health-related data is being acquired by wearables and other IoT devices
following the ever-increasing trend of the Quantified Self. Even though data
protection regulations (e.g., GDPR) encourage the usage of privacy-preserving
processing techniques, most of the current IoT infrastructure was not
originally conceived for such purposes. One of the most used communication
protocols, MQTT, is a lightweight publish-subscribe protocol commonly used in
the Edge and IoT applications. In MQTT, the broker must process data on clear
text, hence exposing a large attack surface for a malicious agent to
steal/tamper with this health-related data. In this paper, we introduce Mqt-
Tz, a secure MQTT broker leveraging Arm TrustZone, a popular Trusted Execution
Environment (TEE). We define a mutual TLS-based handshake and a two-layer
encryption for end-to-end security using the TEE as a trusted proxy. We
provide quantitative evaluation of our open-source PoC on streaming ECGs in
real time and highlight the trade-offs.

###### keywords:

wearablesmHealthsecure brokerMQTTmosquittoTrustZone

, and

##  1 Introduction

Personalized health and medicine has the potential of being the next
revolution in healthcare. It is also referred as the P4 medicine (Predictive,
Preventive, Personalized, and Participatory), and provides the opportunity to
benefit from more targeted and effective diagnoses and treatments [1]. One of
the driving forces behind this tendency is the increasing medicalization of
wearable technology [2] and mobile health (mHealth) apps [3]. In order to
enable these technologies, complex processing IoT pipelines are gradually
being deployed or repurposed. When the data-in-motion are vital signs,
protecting user’s privacy becomes a topic of crucial importance. Recent data
protection regulations (e.g., GDPR [4]) stress the importance of protecting
sensitive information against malicious attackers or untrusted cloud
providers.

Message Queuing Telemetry Transport (MQTT) [5] is one of the most commonly-
used communication protocols in IoT. In spite of that, it is not included in
some of the most extended Medical-Grade data exchange standards [6, 7]. It
follows a publish-subscribe architecture specially designed for environments
with limited memory and reduced network bandwidth. In such circumstances, MQTT
has proven to be more adapted to the IoT than classical protocols such as HTTP
[8]. In MQTT, a client publishes data to a topic and the broker forwards it to
each client that previously subscribed to it. The protocol is currently used
in a variety of settings: data generation by sensors, pre-processing on the
edge, and forwarding to the cloud. Examples include live heart-rate data [9,
10], smart-grids [11], or building management systems [12]. Most MQTT
implementations support TLS for transport security in the client-broker link,
preventing malicious actors from spoofing application data. However, the
broker itself still exposes a great attack surface [13].

In order to protect the privacy of health-related data, we present Mqt-Tz, a
secure implementation of the MQTT broker leveraging Arm TrustZone, a Trusted
Execution Environments (TEE), widely available on edge devices [14]. TrustZone
is a security feature available in recent Arm processors that enables system-
wide hardware isolation for trusted software [15]. Our prototype builds atop
mosquitto (https://mosquitto.org), a popular MQTT broker implementation, and
includes persistent storage of client’s keys in Arm’s tamper-proof secure
storage, as well as TEE-protected re-encryption of application data. These
security enhancements make our ecosystem compliant with the ”Services Secure
Interface” [6] described by the Personal Connected Health Alliance, and
address several attack vectors listed [7] by the IHE. We also consider linking
our secure broker to a larger storage utility where data-at-rest is encrypted
and its origin authenticated by Mqt-Tz.

The paper is organized as follows. In Section 2, we describe the technical
architecture and implementation of Mqt-Tz. Then, in Section 3, we evaluate its
performance and robustness at processing 1-lead ECGs in real time. Finally, in
Section 4 we expose our main conclusions and propose further lines of
research.

MQT-TZ Message BrokerACLMQTTDBRe-encryptionTA (REE)Re-encryption TA (TEE)4
857Secure Monitor ModeLightweight Secure World CacheTrusted Application Heap
MemoryCache Page Miss (LRU)66Tamper Proof Secure Storage1239Trusted Execution
EnvironmentRich Execution EnvironmentTLS & App. Layer EncryptionApplication
Layer EncryptionData in Clear

Figure 1: Mqt-Tz Architecture and data flow.

##  2 Mqt-Tz: Securing the MQTT Broker

###  2.1 Architecture & Component Description

TrustZone splits the system in a hardware-protected trusted part (the TEE) and
an untrusted one (also called Rich Execution Environment, or REE). We add an
encryption layer in MQTT’s payload using client-specific keys stored in Arm’s
secure storage. This way, application data is only processed in clear inside
the TEE. For the additional key-provisioning, we redefine the client
authentication in the mutual TLS handshake to prevent the REE from gaining
access to clients’ keys.

The overall workflow looks as follows. Data travels two-fold encrypted from
the client to the broker (Fig.1-❶). Once the client access is confirmed,
Fig.1-❷, the subscribers for the given topic are retrieved and the payload
forwarded (Fig.1-❸). Then, encrypted data is transferred to the TEE (Fig.1-❹).
The origin and destination client keys are retrieved (❺-❼), the payload is re-
encrypted, and sent back to the REE (Fig.1-❽) and to the subscriber (Fig.1-❾).

Two-Step Handshake. Mqt-Tz defines and uses a two-step handshake that realizes
broker and client authentication with end-to-end encryption from the client to
the TEE. The handshake protocol requires minimal pre-provisioned cryptographic
material. The broker (server in TLS nomenclature) authentication is done
through TLS’ handshake, supported by default in mosquitto. The client
authentication is done through MQTT. It publishes its symmetric key, encrypted
with the broker’s TEE public key, to a specific write-only topic. This TEE
key-pair is generated at device start-up time (secure boot) and derived from a
Hardware Unique Key (HUK).

Layered Encryption & Access Control Mechanisms. Once the initial handshake is
finished, Mqt-Tz uses a two-layer encryption mechanism. First, the client-
broker link is protected by TLS within MQTT. Second, MQTT’s payload field is
encrypted using the clients’ symmetric key. Then, data is re-encrypted in the
TEE (explained next) and sent to destination over MQTT-TLS. Doing so, we
achieve end-to-end security relying on TrustZone as a secure proxy.

Payload Re-encryption. The core secure functionality implemented in Mqt-Tz is
the payload re-encryption. We link MQTT with a Trusted Application (TA)
running inside the TEE that transfers the encrypted data to the Secure World,
retrieves the origin and destination keys from secure storage, and re-encrypts
the information. Currently, topic subscription lists and MQTT metadata are
stored in a dedicated database (MQTT DB) in the REE. We plan on shadowing
these structures and keeping them in the TEE.

Lightweight Cache. Mqt-Tz embeds a lightweight cache that keeps the most
recent keys in the TA’s heap memory, and evicts the least used to persistent
secure storage.

###  2.2 Implementation Details

Mqt-Tz is implemented in C. The current version of Mqt-Tz adds 400 SLOC to
mosquitto and the TA amounts to 1184 SLOC. The Mqt-Tz TA relies on Op-Tee
(https://www.optee.org), an open-source framework with native support for
TrustZone. Our implementation will be publicly available
(https://github.com/mqttz).

Client and Server Authentication. The server-side authentication is done
through vanilla TLS. We deploy Mqt-Tz’s secure broker in a device with a
static IP address. Then, we bound the address to a domain name and use a
certificate. We rely on Let’s Encrypt (https://letsencrypt.org/) to get one
and to authenticate the broker. The client-side authentication uses MQTT as
communication layer, and openssl (v1.1.1a) for cryptographic primitives and
operations. The integration with mosquitto exploits custom callbacks for each
packet processing. In addition, we use MQTT Request/Response (RR) features
(since v5) for the client’s key exchange. To control access and R/W
permissions to topics, we use mosquitto’s ACLs.

Trusted Application. We use Op-Tee to implement the payload re-encryption TA.
Code developed within this framework has two parts: _(1)_ , a host app that
runs in the REE and acts as entry point and bridge to the TEE, and _(2)_ a
trusted API in the TEE that exposes different functions. Mqt-Tz intercepts all
MQTT packets being forwarded to the recipients, and feeds our host app with
both client’s IDs, and the encrypted data. We then perform the payload re-
encryption using Op-Tee’s storage and cryptographic libraries. TrustZone not
only provides isolation between worlds, but also between different TAs. Hence,
we use the same secure API to store new keys during the handshake. For the key
retrieval, we plan to implement a small LRU cache to store the most frequently
used keys in the TA’s heap, and the rest in persistent secure storage.

##  3 Evaluation and Results

In this section, we perform an evaluation of Mqt-Tz. First, we benchmark the
TA re-encryption with random data in order to understand the overhead
introduced by the re-encryption; and then, we analyze the CPU and network
throughput when monitoring vital signs in a real setting. For all experiments,
we virtualize a Raspberry Pi 3 using Qemu-v8 (https://www.qemu.org/) running
mosquitto v1.6.3 and Op-Tee v3.5.0.

###  3.1 TA Re-encryption

In Fig. 2, we show the breakdown of the time required to re-encrypt a single
block of data for different sizes. The time is split in the time to retrieve
each key (retrieve_dec_key, retriev_enc_key), encrypt, and decrypt. We can
observe that AES is two orders of magnitude slower in the TEE. This is a
consequence of Op-Tee not using hardware accelerators in contrast to openssl
in the REE. Moreover, we observe sensible slowdowns when switching from in-
memory to secure persistent storage.

Figure 2: Re-encryption TA microbenchmark.

###  3.2 Real-time ECG Processing

Figure 3: Workload test: Network throughput (top) and CPU usage (bottom)

In this case, we test the resilience of Mqt-Tz at sustaining the workloads
that can be encountered in a hospital. For the experiments, we use the LTMS-S
[16] platform developed by CSEM for the European Space Agency (ESA). In
particular, we simulate 50 patients streaming in real-time 1-lead
electrocardiograms (ECGs) at a frequency of 321.25 Hz. All ECGs are streamed
toward a single Mqt-Tz broker. In Fig. 3, we depict the outbound throughput
generated by each publisher measured using nethogs
(https://github.com/raboof/nethogs). We observe that at any given time only a
subset of the publishers actually emits data. A single subscriber streams at
350 Bytes/s in the worst case, and the full collective generates between 3 to
5 kBytes per second. During the experiment, we recorded using dstat
(https://github.com/dagwieers/dstat) the CPU load at the broker, shown in Fig.
3. We observe that after the initial peak, the overall CPU usage (both for usr
and sys processes) stabilizes at 60%.

##  4 Conclusion and Future Work

Motivated by the lack of secure-by-design communication protocols for the
Edge, we presented Mqt-Tz, our secure implementation of the MQTT broker using
TrustZone and showed its direct application in a in-hospital setting. The
proposed system is robust and capable of managing 50 patients in real-time
with a CPU usage of 60%. We plan to extend this work along the following
directions. First, we will compare Mqt-Tz against other publish-subscribe
protocols and messaging queues. Second, we will study the performance overhead
of Mqt-Tz when deployed on large-scale scenarios. Finally, we intend to look
into the energy footprint, an aspect of paramount relevance for edge
deployments.

## References

  * [1] G. P. Cumming, “Connecting & collaborating - Healthcare for the 21st century,” in PAHI’2014, 2014.
  * [2] J. Dunn, R. Runge, and M. Snyder, “Wearables and the medical revolution,” Pers. Med., vol. 15, no. 5, pp. 429–448, 2018.
  * [3] M.-P. Gagnon, P. Ngangue, J. Payne-Gagnon, and M. Desmartis, “m-Health adoption by healthcare professionals: A systematic review,” J. Am. Med. Inform. Assn., vol. 23, pp. 212–220, June 2015.
  * [4] The European Parliment and the Council of the European Union, “Regulation (EU) 2016/679,” 2016.
  * [5] A. Banks and R. Gupta, “MQTT version 3.1.1,” software, OASIS, Oct. 2014.
  * [6] Personal Connected Health Alliance, “Fundamentals of medical-grade data exchange,” white paper, Continua, Sept. 2018.
  * [7] IHE PCD Technical Committee, “Medical equipment management (MEM): Medical device cyber security,” white paper, IHE International, Inc., Oct. 2015\.
  * [8] T. Yokotani and Y. Sasaki, “Comparison with HTTP and MQTT on required network resources for IoT,” in ICCEREC’2016, pp. 1–6, Sept. 2016.
  * [9] K. Chooruang and P. Mangkalakeeree, “Wireless heart rate monitoring system using MQTT,” Procedia Comput. Sci., vol. 86, pp. 160–163, 2016.
  * [10] C. Segarra, R. Delgado-Gonzalo, M. Lemay, P.-L. Aublin, P. Pietzuch, and V. Schiavoni, “Using trusted execution environments for secure stream processing of medical data,” in Lect. Notes Comput. Sc., vol. 11534, pp. 91–107, 2019.
  * [11] A. Krylovskiy, M. Jahn, and E. Patti, “Designing a smart city internet of things platform with microservice architecture,” in FiCloud’2015, pp. 25–30, Aug. 2015.
  * [12] Y. Lee, H. Hsiao, C. Huang, and S. T. Chou, “An integrated cloud-based smart home management system with community hierarchy,” IEEE. T Consum. Electr., vol. 62, pp. 1–9, Feb. 2016.
  * [13] Teserakt AG, “Is MQTT secure? (a report),” 2019.
  * [14] R. Liu and M. Srivastava, “VirtSense: Virtualize sensing through ARM TrustZone on Internet-of-Things,” in SysTEX’2018, (New York, NY, USA), pp. 2–7, ACM, 2018.
  * [15] J. Amacher and V. Schiavoni, “On the performance of ARM TrustZone,” in DAIS’2019, pp. 133–151, 2019.
  * [16] O. Chételat, D. Ferrario, M. Proença, J.-A. Porchet, A. Falhi, O. Grossenbacher, R. Delgado-Gonzalo, N. Della Ricca, and C. Sartori, “Clinical validation of LTMS-S: A wearable system for vital signs monitoring,” in EMBC’2015, pp. 3125–3128, 2015.


Let us present the proof of the theorem without the second item. The group $G$ acts trivially on $H^2 (X, \ZZ)$, so it acts symplectically on $X$. Hence $p$ is a symplectic quotient. The ramified locus of $p$ is contained in the union of the fixed loci $\bigcup_{f \in G} X^f$, which is of codimension $\ge 2$. This means $p$ is quasi-étale.

The fundamental group is invariant under codimension $\ge 2$ modifications. Hence $\bar X = X/G$ has the fundamental group $G$. If $\bar X$ becomes smooth then it a smooth irreducible symplectic variety. By <cit.>, $\bar X$ becomes a hyper-Kähler manifold. In particular, $\bar X$ is simply connected and hence $G$ becomes trivial.

The last item follows directly from <cit.>. Since $G$ acts on $\mathcal X$ holomorphically and trivially on $H^2 (X, \ZZ)$, $\mathcal X \to \Def(X)$ equipped with a $G$-action is the universal deformation of the pair $(X, G)$. Once we have a universal deformation of the pair $(X, G)$, the quotient $\mathcal X/G \to \Def(X)$ is the locally trivial universal family of $X/G$.

Let $(X, \sigma)$ be a compact symplectic variety and $f : X' \to X$ a finite quasi-étale morphism. Then $(X', f^* \sigma)$ is a compact symplectic variety.

By <cit.> or <cit.>, $X'$ is Gorenstein and canonical. Therefore, it has rational singularities by <cit.>. Now $f^* \sigma \in H^0 (X', \Omega_{X'}^{[2]})$ is a symplectic form in codimension $1$ as $f$ is étale in codimension $1$. The claim follows.

We prove the second item here. As a finite quotient of a smooth variety $X$, the space $\bar X$ is certainly $\QQ$-factorial and has quotient singularities. If the additional assumption on the fixed loci holds, then $\codim X_{\operatorname{sing}} \ge 4$ and it becomes Namikawa symplectic.

To prove $\bar X$ is irreducible symplectic, we follow the argument of Matsushita <cit.>. Let $f : \bar Y \to \bar X$ be an arbitrary finite quasi-étale morphism. Consider the diagram

\[\begin{tikzcd}
Y \arrow[r, "g"] \arrow[d, "q"] & X \arrow[d, "p"] \\
\bar Y \arrow[r, "f"] & \bar X
\end{tikzcd},\]

where $Y$ is the normalization of the fiber product $X \times_{\bar X} \bar Y$. We claim $g$ and $q$ are finite quasi-étale. The finiteness is clear, so we concentrate on their quasi-étaleness. Notice that the quasi-étale property is stable under base change, so we need to prove the normalization in this case is quasi-étale. But notice that $X$ is smooth and $f$ is quasi-étale, so that $X \times_{\bar X} \bar Y$ is smooth in codimension $1$. Hence the normalization of it is in fact isomorphism in codimension $1$. This proves $g$ and $q$ are quasi-étale.

Now $X$ is smooth, $Y$ is normal, and $g : Y \to X$ is finite quasi-étale. By the Zariski–Nagata purity theorem of the branch locus (e.g., <cit.>), this forces $g$ to be étale. The hyper-Kähler manifold $X$ is simply connected, so this means $Y$ must be a disjoint union of several isomorphic copies of $X$. Let us fix a connected component $Y_0$ of $Y$. It is a hyper-Kähler manifold isomorphic to $X$.

Consider the morphism $q$ restricted to the connected component $q : Y_0 \to \bar Y$. It is a finite quasi-étale morphism. Note that the target $\bar Y$ is canonical (<Ref>), so <cit.> guarantees the existence of a reflexive pullback $q^* : H^0 (\bar Y, \Omega_{\bar Y}^{[k]}) \to H^0 (Y_0, \Omega_{Y_0}^{[k]})$. Since $q$ is quasi-étale, this morphism is injective. But recall that $Y_0 \cong X$ is a hyper-Kähler manifold, so this forces $\bar Y$ to satisfy the dimension condition of the definition of irreducible symplectic varieties. This proves $\bar X$ is an irreducible symplectic variety.

The following sequence of identities proves $q_{\bar X}$ is the Beauville–Bogomolov form with the Fujiki constant $c_{\bar X} = c_X/|G|$:

\[ \int_{\bar X} x^{2n} = \frac{1}{|G|} \int_X (p^*x)^{2n} = \frac{c_X}{|G|} q_X (p^*x)^n = \frac{c_X}{|G|} q_{\bar X} (x)^n .\]

Since $\bar X$ is a compact Kähler orbifold, its rational singular cohomology admits a well-behaved pure Hodge structure (e.g., <cit.>) and $p^* : H^* (\bar X, \QQ) \to H^* (X, \QQ)$ is an injective Hodge structure homomorphism with the image $H^* (X, \QQ)^G$. In particular, $p^*$ is an isomorphism in degree $2$.

To prove $H^* (\bar X, \QQ) = H^* (X, \QQ)^G$ is closed under the $\mathfrak g$-action, it is enough to prove the $G$-action and $\mathfrak g$-action on $H^* (X, \QQ)$ commutes. Recall that the LLV structure is diffeomorphism invariant. In other words, if $f : X_1 \to X_2$ is a diffeomorphism between two compact hyper-Kähler manifolds then we have

\[ f^* (L_x (\xi)) = L_{f^* x} (f^* \xi), \qquad f^* (\Lambda_x (\xi)) = \Lambda_{f^* x} (f^* \xi) \]

for any $x \in H^2 (X_2, \QQ)$ and $\xi \in H^* (X_2, \QQ)$. Here $L_x$ and $\Lambda_x$ are Lefschetz and inverse Lefschetz operators associated to $x$. If we set $X_1 = X_2 = X$ and $f \in G$ to be an $H^2$-trivial automorphism then this means $f^*$ commutes with the operators $L_x$ and $\Lambda_x$. That is, $G$ commutes with $\mathfrak g$.

To obtain the results about the Mumford–Tate algebras one imitates the method used in <cit.> and deduces $f \in \mathfrak g$ for $f$ a Weil operator on the cohomology $H^* (\bar X, \QQ)$ (which is the restriction of Weil operator on $H^* (X, \QQ)$). This proves all the special Mumford–Tate algebra of $H^k (\bar X, \QQ)$ are the same and even same for that of $H^2 (X, \QQ)$.