# Which Hyperparameters to Optimise? An Investigation of Evolutionary
Hyperparameter Optimisation in Graph Neural Network For Molecular Property
Prediction

Yingfang Yuan  Heriot-Watt UniversityFirst GaitEdinburghUnited KingdomEH14 4AS
<EMAIL_ADDRESS>,  Wenjun Wang  Heriot-Watt UniversityEdinburghUnited
KingdomEH14 4AS<EMAIL_ADDRESS>and  Wei Pang  Heriot-Watt
UniversityEdinburghUnited KingdomEH14 4AS<EMAIL_ADDRESS>
(2021)

###### Abstract.

Recently, the study of graph neural network (GNN) has attracted much attention
and achieved promising performance in molecular property prediction. Most GNNs
for molecular property prediction are proposed based on the idea of learning
the representations for the nodes by aggregating the information of their
neighbor nodes (e.g. atoms). Then, the representations can be passed to
subsequent layers to deal with individual downstream tasks. Therefore, the
architectures of GNNs can be considered as being composed of two core parts:
graph-related layers and task-specific layers. Facing real-world molecular
problems, the hyperparameter optimization for those layers are vital.
Hyperparameter optimization (HPO) becomes expensive in this situation because
evaluating candidate solutions requires massive computational resources to
train and validate models. Furthermore, a larger search space often makes the
HPO problems more challenging. In this research, we focus on the impact of
selecting two types of GNN hyperparameters, those belonging to graph-related
layers and those of task-specific layers, on the performance of GNN for
molecular property prediction. In our experiments. we employed a state-of-the-
art evolutionary algorithm (i.e., CMA-ES) for HPO. The results reveal that
optimizing the two types of hyperparameters separately can gain the
improvements on GNNs’ performance, but optimising both types of
hyperparameters simultaneously will lead to predominant improvements.
Meanwhile, our study also further confirms the importance of HPO for GNNs in
molecular property prediction problems.

Graph Neural Networks, Molecular Property Prediction, Hyperparameter
Optimisation, Evolutionary Computation

††copyright: acmcopyright††journalyear: 2021††doi:
10.1145/nnnnnnn.nnnnnnn††conference: the Genetic and Evolutionary Computation
Conference 2021; July 10–14, 2021; Lille, France††price: 15.00††isbn:
978-x-xxxx-xxxx-x/YY/MM††ccs: Computing methodologies Neural networks††ccs:
Computing methodologies Search methodologies††ccs: Applied computing††ccs:
General and reference Experimentation

##  1\. Introduction

Graph neural networks (GNNs) have been applied to solve a wide range of
problems, such as social recommendation (Fan et al., 2019), citation trend
prediction (Cummings and Nassar, 2020), and molecular property prediction
(Yuan et al., 2021b; Duvenaud et al., 2015; Gilmer et al., 2017; Jiang et al.,
2021). One advantage of GNNs is that they can be directly operated on graphs
in an end-to-end manner for real-world problems, and task-specific
representations can be learned automatically between latent layers. In
contrast, traditional machine learning methods require handcrafted features
from graph as input. For example, the work presented in (Jiang et al., 2021)
mentioned that support vector machine, random forest, and XGBoost took
handcrafted descriptors and/or fingerprints from molecular structures as
inputs to predict molecular properties. When GNNs are applied to the same
problems, given a molecular graph, the vector representations for all atoms
are learned at first by aggregating and updating the information of neighbor
atoms. Thereafter, the readout operator (Wu et al., 2020) (i.e., mean, sum) is
employed to collapse all atom representations into a molecular (graph-level)
representation, which can be passed to subsequent task-specific layers to make
predictions. In our experiments, we employed graph convolution (Duvenaud et
al., 2015) because it has been proposed considering molecular domain
knowledge.

The study presented in (Yuan et al., 2021b, c, a) indicated that
hyperparameter optimization (HPO) by evolutionary computation can improve the
GNN’s performance for predicting molecular properties. However, HPO for GNNs
is often an expensive task. Evaluating hyperparameter settings on GNNs needs
to train models which cost a lot of computational resources. Some methods have
been proposed to alleviate this issue such as using surrogate models (Bergstra
et al., 2011), successive halving (Karnin et al., 2013; Jamieson and
Talwalkar, 2016). From another angle to consider this issue, the predefined
hyperparameter search space may directly affect the HPO results. For example,
it is very challenging to search for optimal solutions given a very large
hyperparameter space and a relatively limited computational budget. The HPO
search space is defined by selected hyperparameters (e.g., batch size,
learning rate), the ranges of these hyperparameter values, and the step sizes
(i.e, intervals) which determine the resolutions of the hyperparameter values
to be searched. From our perspective, the ranges and step sizes depend on
individual practical cases to some degree. This research focuses on
investigating the impact of selecting different types of hyperparameters to
optimize on the GNN’s performance for molecular property predictions. In
concrete, we investigated the differences of optimizing hyperparameters
related to graph layers and those from task-specific layers. Therefore, we
grouped the hyperparameters according to the layer-wise neural architectures
of GNNs, which can be considered as being the combination of graph layers and
task-specific layers. In this way, we expect to discover that optimizing which
types of the hyperparameters may bring more expected gains.

Additionally, CMA-ES (Hansen, 2016) as a state-of-the-art, derivative-free,
and evolutionary black-box optimization method has been employed to optimize
hyperparamters. Therefore, we use it as our evolutionary search strategy for
HPO. This will allow us to focus on the impact of the two types of
hyperparameters with the same search strategy. However, we also point out that
more evolutionary HPO approaches could be used in the future to test the
impact of hyperparameters, but this is beyond the scope of this research. .

The rest of this paper is organized as follows. Section 2 introduces related
work. In Section 3, the experiments are reported and the results are analysed.
Finally, Section 4 concludes the paper and explores some directions in future
work.

##  2\. Related Work

###  2.1. Graph Neural Networks

Figure 1. The Neural Architecture of GNN in Molecular Property Prediction
The input is SMILES (Weininger, 1988) which can be used to describe molecular
structure in the form of a line notation

Many types of GNNs have been proposed based on the idea of aggregating
neighborhood recursively (Xu et al., 2018) (or message passing (Gilmer et al.,
2017)). Each node aggregates feature vectors of its neighbors to compute its
new feature representation. To generate the representation of the entire
graph, many schemes have been proposed such as graph pooling (Ying et al.,
2018), attention sum (Li et al., 2015), mean (Atwood and Towsley, 2016), max
(Gori et al., 2005). When GNNs are applied in molecular property prediction,
the representation of the entire graph is then passed to subsequent layers
(Jiang et al., 2021). For example, graph convolution model (GC) (Duvenaud et
al., 2015) makes use of neural networks to imitate circular fingerprint (a
representation of molecular structures by encoding the information of atom
neighborhoods.) (Glen et al., 2006) to generate molecular representation, the
so called neural fingerprint, which can be passed to a simple neural network
to predict drug efficiency and photovoltaic efficiency.

Therefore, in general, the architectures of GNNs are classified into graph
layers and task-specific layers (Fig. 1). The former denote those layers which
take responsibility of processing the structured data by aggregations, and
generate vector representations for all the nodes in graph. Task-specific
layers are exploited to deal with individual problems (e.g., classification,
regression). In Fig. 1, task-specific layers are implemented by fully-
connected layers which takes the molecular representation as input to output a
value. We also point out that the types of layers such as pooling, readout
depend on the individual situations. For example, readout-equivalent operation
has been implicitly defined within the graph convolution algorithm (Duvenaud
et al., 2015) because it aims to generate a molecular representations (graph-
level), and in this case the readout operation is classified into graph layer.
In contrast, graph convolutional networks (Kipf and Welling, 2016) was
originally proposed to learn each node representation for node classification.
When it is applied to solve graph-level tasks, the node representations have
to be collapsed into a representation by readout, and in this case the readout
is classified into task-specific layer.

Furthermore, there is an increasing number of research works about applying
GNN into molecular property prediction. The work presented in (Wieder et al.,
2020) taxonomies the problems of molecular property prediction into four main
types: quantum chemistry, physical chemistry, biophysics, and biological
effect. MPNN (Gilmer et al., 2017) makes use of the idea of message-passing to
update atom hidden states recursively, and the final hidden states of atoms
are processed by readout function to predict the multiple properties in
quantum chemistry originally. Thereafter, directed MPNN (Yang et al., 2019),
MPNN with SELU activation function (Withnall et al., 2020), MPNN with
attention and edge memory mechanisms (Withnall et al., 2020) have been
proposed as the variants of MPNN (Gilmer et al., 2017) and applied to predict
a wider range of molecular properties. Message-passing is a kind of
propagating information in spatial manner. Similarly, there are some GNNs
which have been proposed based on the idea of performing the spatial-like
convolution operation on molecular graphs (Coley et al., 2017; Duvenaud et
al., 2015; Wang et al., 2019; Xinyi and Chen, 2018). In contrast, the number
of the applications of spectral GNNs in molecular property prediction is
relatively small. LanczosNet (Liao et al., 2019), GCN (Henaff et al., 2015),
AGCN (Li et al., 2018), GCN with eigen-pooling (Ma et al., 2019) which have
been proposed based on the spectral decomposition. Furthermore, the work
presented in (Gilmer et al., 2017) used MPNN framework to unify message-
passing-based, spatial, and spectral GNNs.

###  2.2. CMA-ES

CMA-ES(Hansen, 2016) is one of the state-of-the-art evolutionary approaches
for HPO. It regularly dominates the black-box optimization benchmarking (BBOB)
challenge (Hutter et al., 2019). The core idea of CMA-ES is maintaining a
multivariate normal distribution (Eq. 1) to sample new solutions. The
multivariate normal distribution consists of a covariance matrix $C$ and mean
$m$, and step size $\sigma$. The update of these parameters is inspired by
biological evolution. Specifically, $g$ denotes the generation in Eq. 1, CMA-
ES always follows the principle of ‘survival of the fittest’, the number of
$\mu$ promising individuals are selected to update $m^{(g)}$ with different
weights $w$ by Eq. 2. To update $C^{(g)}$, CMA-ES combines the
Rank-$\mu$-Update with Rank-One-Update to keep the balance between
exploitation and exploration. Finally, to control the scale of step size and
speed up learning proccess, CMA-ES introduces $\sigma$ which is updated by
cumulative path length control.

(1)
$\boldsymbol{x}_{k}^{(g+1)}\sim\boldsymbol{m}^{(g)}+\sigma^{(g)}\mathcal{N}\left(\mathbf{0},\boldsymbol{C}^{(g)}\right)\quad\text{
for }k=1,\ldots,\lambda$

(2)
$\boldsymbol{m}^{(g+1)}=\sum_{i=1}^{\mu}w_{i}\boldsymbol{x}_{i:\lambda}^{(g+1)}$

The most surprising discovery about CMA-ES is that the learning of the
distribution parameters is similar to the descent in direction of a sampled
natural gradient of the expected objective function value (Akimoto et al.,
2010). Additionally, the using of convariance matrix in the method provides
promising future for dealing with multi-dimensions HPO problems (Nomura et
al., 2020). In concrete, there are some hyperparameters are not independent
for all deep learning models, for example, the learning rate and batch size.
Convariance matrix helps to automatically build the relation between them and
coordinate them smoothly.

##  3\. Experiments

ESOL (Delaney, 2004), FreeSolv (Mobley and Guthrie, 2014), and Lipophilicity
(Hersey, 2015) are three representative molecular benchmark datasets used in
our experiments, and they respectively correspond the tasks of predicting
solubility, hydration free energy, and octanol/water distribution coefficient.
The sizes of them are 1,128, 642, 4,200 respectively. Each dataset is split
into training, validation, and test sets with the ratio $80\%/10\%/10\%$. The
training set is used to train GNNs, the validation set is used for guiding
HPO, and test set is used to do final evaluations. Compared with the common
designs of HPO experiments, we made the modification that the evaluation of a
solution (hyperparameter setting) is repeated three times, and the mean of the
root mean squared errors (RMSE) is used to score the solutions.

Meanwhile, we employed the graph convolution model (GC) (Duvenaud et al.,
2015) to predict these properties in the above mentioned three datasets. GC
was proposed with the molecular domain knowledge which fits our research
problem (molecular property prediction) compared with other GNNs. For ease of
implementation, we leveraged the DeepChem (Python toolkit for deep learning in
drug discovery, materials science, quantum chemistry, and biology) (Ramsundar
et al., 2019) to preprocess molecular data and implement GC. In addition, we
make use of Optuna (Akiba et al., 2019) to conduct the HPO experiments with
CMA-ES.

To assess the impact of optimizing different types of hyperparameters on the
performance of GNNs, we first look at the graph-related hyperparameters, and
for GC, they include the number of graph convolution layer $n_{g}$, the sizes
of the graph convolution layer $s_{g}$, and the size of dense layer $s_{d}$
which is defined in GC to generate molecular representations. The range of
$n_{g}$ is $1\sim 6$ with the step size 1. $s_{g}$ and $s_{d}$ are in the
ranges of $32\sim 512$ (step size 32) and $64\sim 1024$ (step size 64),
respectively. These ranges are set according to the default value provided in
(Wu et al., 2018), while the step sizes are selected following (Yuan et al.,
2021c). As for the task-specific hyperparameters (i.e., the hyperparameters in
task-specific layers), in order to predict molecular properties $\mathbb{R}$,
we employed a simple feedforward neural network which consists of a few fully-
connected layers. So the task-specific hyperparameters include the number of
fully-connected layers $n_{f}$ (excluding the output layer), and the sizes of
those layers $s_{f}$, and the activation function $a$. The options of $a$ are
$Sigmoid$, $Tanh$, and $ReLU$. Meanwhile, the ranges of $s_{f}$ referring to
$s_{d}$ are from $64\sim 1024$ with the step size 64. The arrangement of
$n_{f}$ is the same as $n_{g}$ for facilitating analysis. The aboved
hyperparameters are summarized in Table 1. Furthermore, $n_{g}$ and $n_{f}$
will determine the number of $s_{g}$ and $s_{f}$, so our search space is
dynamic. The dynamic feature of the search space will change the dimensions of
the problem during the search, which makes HPO more challenging.

Types | Hyperparameters | Ranges | Step Sizes
---|---|---|---
Graph Layer | $n_{g}$ | $1\sim 6$ | $1$
$s_{g}$ | $32\sim 512$ | $32$
$s_{d}$ | 64-1024 | $64$
Fully-connected Layer | $n_{f}$ | 1-6 | $1$
$s_{f}$ | $64\sim 1024$ | $64$
$a$ | $sigmoid,relu,tanh$ | $1$

Table 1. Hyperparameters Summary

###  3.1. Pseudo-dynamic Search Space

However, it is noted that CMA-ES does not support dynamic search space (Akiba
et al., 2019). Therefore we turn to implement the pseudo-dynamic process, and
the process of HPO is shown in Algorithm 1. In Algorithm 1, $\mathbf{\Lambda}$
denotes the entire hyperparameter space, and $|\mathbf{\Lambda}|=N$ means the
number of hyperparameters. Furthermore, $\Lambda_{n}^{I}$ is used to define a
dynamic hyperparameter. For example, in our experiments, $s_{g}$ is a list in
which each element represents the size of the corresponding graph convolution
layer. Meanwhile, the number of elements in $s_{g}$ is dynamic as it is
determined by $n_{g}$. It is not possible to decrease/increase the dimensions
of the multivariate normal distribution after initialization, so that we keep
the CMA-ES to sample the maximum number of elements. Regarding this, we make
use of $n_{g}$ to decide how many elements will be used to instantiate the
model. In this way, the search space maintained by CMA-ES is not changed, but
in practical it affects the generation of models.

input : the hyperparameter space
$\mathbf{\Lambda}=\Lambda_{1}\times\Lambda_{2}\times\ldots\Lambda_{N}$, where
$\Lambda_{n}$ denotes the $n-$th hyperparameter; if $\Lambda_{n}$ is a dynamic
hyperparameter,
$\Lambda_{n}=\\{\Lambda_{n}^{1},\Lambda_{n}^{2},\ldots\Lambda_{n}^{i}\\}$,
where $i\in\Lambda_{m}$, $(\Lambda_{m},\Lambda_{n})$ are paired, $\Lambda_{m}$
will determines the number of elements in $\Lambda_{n}$ ; a GNN $\mathcal{M}$;
the total number of trials $T$

1 initializing CMA-ES; $\lambda_{best}\text{ current best hyperparameter
setting}$;

sort($\Lambda$) ;

// move all dynamic hyperparameters backward

2 t = 1;

3 while  _$t <T$_ do

$\lambda$ = [] ;

// null list for collecting sampled hyperparameters

4 for  _$n$ = $1$ to $N$_ do

5 if  _$\Lambda_{n}$ is a dynamic hyperparameter_ then

6 lookup ($(\Lambda_{m},\Lambda_{n})$);

7 for  _$z$ = $1$ to $I$_ do

8 $v$ = CMA-ES.suggest($\Lambda_{n}^{z}$);

9 if  _$z <=i$_ then

10 $\lambda$.$\text{append}(v)$;

11

12

13 end for

14

15 else

16 $v$ = CMA-ES.$\text{suggest}(\Lambda_{n})$;

17 $\lambda$.$\text{append}(v)$;

18

19 end if

20

21 end for

22 $\mathcal{M}$ is instantiated with $\lambda$;

23 evaluate($\mathcal{M}$);

24 update($\lambda_{best}$);

25 $t+=1$;

26

27 end while

Algorithm 1 HPO on Pseudo-dynamic Search Space

Overall, we designed four sets of experiments. The first set of experiments
take the default values of hyperparameters from DeepChem (Ramsundar et al.,
2019) to train a GC model for thirty times on ESOL, FreeSolv, and
Lipophilicity datasets, respectively. The average of root mean squared errors
(Mean RMSE) and standard deviations (Mean Std) are shown in Table 2. Batch
size $s_{b}$, the number of training epochs $n_{e}$, and learning rate
$l_{r}$, which are not considered to optimize because this study focuses on
discovering the impact of optimizing the two different types of GNNs’
hyperparameters. Meanwhile, $s_{b}=128$, $n_{e}=100$, and $l_{r}=0.0005$ are
set for all our experiments. In Table 2, $n_{f}$, $s_{f}$ and $a$ are 0 or
none because the task-specific layer only has a single output layer (i.e., no
hidden layers) according to the orginal settings in DeepChem (Ramsundar et
al., 2019). The purpose of this group of experiments is to set a baseline for
our following experiments.

Datasets | ESOL | FreeSolv | Lipophilicity
---|---|---|---
Hyperparameters |  |  $n_{g}=2$, $s_{g}=[128,128]$,
---
$s_{d}=256$, $n_{f}=0$,
$s_{f}=none$, $a=none$,
$s_{b}=128$, $n_{e}=100$,
$l_{r}=0.0005$,
Train | Mean RMSE | 0.3202 | 0.7782 | 0.2146
Mean Std | 0.0487 | 0.0785 | 0.0259
Valid | Mean RMSE | 1.1145 | 2.1353 | 0.7245
Mean Std | 0.0479 | 0.2326 | 0.0247
Test | Mean RMSE | 1.1570 | 1.8482 | 0.7424
Mean Std | 0.0700 | 0.1679 | 0.0203
Table 2. The Results by Using the Hyperparameters Provided by MoleculeNet

The other three sets of experiments are conducted the HPO for GC on ESOL,
FreeSolv, and Lipophilicity, respectively. In Section 3.2, we will discuss the
process of HPO on the three datasets (Figs. 2$\sim$4). Thereafter, we will
start to analyse the results in details for each dataset (Tables 3$\sim$5).

###  3.2. The Process of Hyperparameter Optimization

Figure 2. The Optimization Process on the ESOL Dataset Figure 3. The
Optimization Process on the FreeSolv Dataset Figure 4. The Optimization
Process on the Lipophilicity Dataset

In Figs. 2$\sim$4, the x-axis denotes the index of each trial. One trial
represents a process of sampling and evaluating a new hyperparameter setting.
CMA-ES was assigned with 200 trials for each dataset. The y-axis denotes the
metric of RMSE for evaluating each trial; each trial in our experiments are
evaluated thrice, and the mean values are drawn in these figures. The reason
for using the mean value of multiple evaluations is that we observed the
results for multiple evaluations are not stable, which may mislead the HPO. In
Fig. 2$\sim$4, the blue points represent the RMSEs for optimizing graph-
related hyperparameters, the green points represent the RMSEs of optimizing
task-specified hyperparameters, and the orange points denote the results of
optimizing both graph-related and task-specified hyperparameters
simultaneously. The nine lines are used to represent the trends of performing
HPO on different types of hyperparameters, and these lines are drawn by
connecting all current best points in time sequence. It is easy to observe
that most of the lines hold obviously decreasing trends, which indicates that
CMA-ES works for optimizing hyperparameters. Furthermore, in Figs. 2 and 4,
the decreasing trends of RMSEs for red and purple lines are less significant,
compared with those for the blue lines. From these observations, we can see
that, Fig. 2 implies that appropriate settings for both graph and task-
specific layers are needed together, and they may complement with each other
to achieve better performance in molecular prediction tasks. Overall, the
three figures indicate that optimizing both types of hyperparameters can get
more gains given the same number of trials, even the search space becomes
larger because the number of possible combinations of different
hyperparameters increases.

###  3.3. The Results

The best hyperparamter values obtained from Section 3.2 are used to
instantiate GCs, and these GCs are trained respectively on ESOL, FreeSolv, and
Lipophilicity datasets. The detailed results are shown in Tables 3 $\sim$5.

In general, the models configured with the CMA-ES optimized hyperparameters on
the three datasets achieved better performances than the original ones (Tab
2). For example, in ESOL, the RMSE of the GC with default hyperparameters
(Table 2) is 1.1570 on the test set. The models with HPO on graph layers,
task-specific layers, and both of them have the Mean RMSE of 1.0854, 0.9505,
and 0.8824, respectively (Table 3). To statistically analyse the improvements,
we conducted the $t$-test for the RMSEs on the test dataset between Table 2
and Table 3 with the $t$ values of 4.0000, 12.7625, and 18.1311. When the
significance level $\alpha=0.001$, their performance are all significantly
better than original ones??

In Tables 3$\sim$5, we can see that HPO on graph and fully connected (task-
specific) layers outperforms HPO on either graph or fully connected layers.
With the same number of trials in HPO, optimizing graph or fully connected
layers face a large search space, but it achieved promising performance.
Meanwhile, in Tables 4 and 5, we observed that only optimizing fully-connected
layers has relatively more serious over-fitting problem compared with the
optimizing both types of hyperparameters, since it always obtained less RMSE
values on the training datasets, and inversely had larger RMSE values on
validation and test sets. In this case, it indicates optimizing fully-
connected layers only would help to fit the problems, but without optimizing
graph layers, the molecular representations may not be better learnt, which
leads to reduced performance of GNNs in test set. Conducting HPO on graph
layers only achieved lower performance than performing HPO on task-specific
layers and the both in the three datasets. We believe the reason is that the
default setting of GC only provides a output layer without hidden layers; this
means molecular representations are only passed to a linear layer (without
non-linear transformation) which dramatically restricts the learning
capability. Interestingly, after HPO, the hyperparameter $a$ was assigned to
$ReLU$ in all experiments which is the same choice as described in (Duvenaud
et al., 2015), where the $ReLU$ activation function was manually selected.

In summary, although graph layers and task-specific layers play different
roles in GNNs, they need to be optimized together when solving practical
problems. The reason is as follows: a better graph representation learned from
graph layers needs to be supported by tailored task-specific layers to
accomplish tasks. Similarly, task-specific layers also need appropriate graph
representations to achieved good performances.

From the above analysis, we can conclude that when only limited computational
budget is available, we should still optimize the hyperparameters of both
types of layers, rather than focusing on one of them.

ESOL | Graph Layers | Fully Connected Layers | Graph and Fully Connected Layers
---|---|---|---
Hyperparameters |  $n_{g}=3$, $s_{g}=[320,384,128]$, $s_{d}=448$ |  $n_{f}=5$, $s_{f}=[512,768,832,320,192]$, $a=relu$ |  $n_{g}=1$, $s_{g}=512$, $s_{d}=640$, $n_{f}=5$, $s_{f}=[1024,832,640,192,832]$, $a=relu$
Train | Mean RMSE | 0.3554 | 0.1917 | 0.2328
Mean Std | 0.0368 | 0.0372 | 0.0651
Valid | Mean RMSE | 1.0108 | 0.8678 | 0.7718
Mean Std | 0.0554 | 0.0318 | 0.0371
Test | Mean RMSE | 1.0854 | 0.9508 | 0.8824
Mean Std | 0.0660 | 0.0515 | 0.0417

Table 3. HPO on the ESOL Dataset

FreeSolv | Graph Layers | Fully Connected Layers | Graph and Fully Connected Layers
---|---|---|---
Hyperparameters |  $n_{g}=1$, $s_{g}=[256]$, $s_{d}=192$ |  $n_{f}=2$, $s_{f}=[1024,448]$, $a=relu$ |  $n_{g}=2$, $s_{g}=[512,352]$, $s_{d}=128$, $n_{f}=5$, $s_{f}=[192,640,320,320,768]$, $a=relu$
Train | Mean RMSE | 0.7743 | 0.4565 | 0.4926
Mean Std | 0.1361 | 0.2301 | 0.1510
Valid | Mean RMSE | 1.9128 | 1.6702 | 1.4779
Mean Std | 0.2172 | 0.1464 | 0.1191
Test | Mean RMSE | 1.6983 | 1.4118 | 1.2285
Mean Std | 0.1891 | 0.1461 | 0.0727

Table 4. HPO on the FreeSolv Dataset

Lipophilicity | Graph Layers | Fully Connected Layers | Graph and Fully Connected Layers
---|---|---|---
Hyperparameters |  $n_{g}=6$, $s_{g}=[416,256,512,320,384,128]$, $s_{d}=768$ |  $n_{f}=4$, $s_{f}=[1024,896,832,64]$, $a=relu$ |  $n_{g}=5$, $s_{g}=[480,512,256,192,224]$, $s_{d}=960$, $n_{f}=4$, $s_{f}=[704,320,128,768]$, $a=relu$
Train | Mean RMSE | 0.2148 | 0.1369 | 0.1701
Mean Std | 0.0206 | 0.0201 | 0.0361
Valid | Mean RMSE | 0.6655 | 0.6656 | 0.6239
Mean Std | 0.0171 | 0.0144 | 0.0154
Test | Mean RMSE | 0.7014 | 0.6786 | 0.6472
Mean Std | 0.0148 | 0.0165 | 0.0187

Table 5. HPO on the Lipophilicity Dataset

##  4\. Conclusions and Future Work

With the rapid development of GNNs, applying them in molecular machine
learning problems becomes increasingly compelling and meaningful. For example,
accurate molecular property prediction can significantly facilitate the entire
process of drug discovery in a faster and cheaper way. However, the
performance of GNNs are largely affected by hyperparameter selection, so the
research of HPO on GNNs is of extremely important.

In this paper, we elaborated the problem of HPO on GNNs for molecular property
prediction, and investigated in depth that which types of hyperparameters
should be selected to optimize when computational resources are limited. Based
on our experiments, we conclude that both graph-related hyperparameters and
task-specific hyperparameters should be optimised simultaneously, and leaving
any one out will result in reduced performance. Even doing this means a larger
search space, which seems to be more challenging given the same number of
trials (limited computational resources), such a strategy can surprisingly
achieve better performance.

Finally, we acknowledge that our experiments are based on one type of GNN
model and one evolutionary strategy. However, we believe that our conclusion
can be further generalised, because we have selected the representative GNN
model, used state-of-the-art evolutionary HPO approach, and the benchmark
datasets used for experiments are also representative in molecular property
prediction problems.

Still, we propose two future directions to carry out our research in the next
step. First, there exist various GNNs, and most of them comply with the rule
of aggregating the neighbor information to learn the node representations.
However, they can be classified into spectral and spatial GNNs (Balcilar et
al., 2020). In this research, we have extensively investigated the impact of
HPO on GC, which is a representative of spatial GNNs. Therefore, it would be
interesting and worthwhile to investigate whether the same conclusion holds
for HPO of spectral GNNs. Second, we employed CMA-ES as the HPO strategy
because it is a state-of-the-art evolutionary HPO method. However, it does not
support the dynamic search space, which constrains its scalability. In our
future work, other evolutionary HPO approaches can be applied to explore their
effectiveness on optimizing hyperparamters with dynamic search space for GNNs.

## References

  * (1)
  * Akiba et al. (2019) Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019.  Optuna: A next-generation hyperparameter optimization framework. In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_. 2623–2631.
  * Akimoto et al. (2010) Youhei Akimoto, Yuichi Nagata, Isao Ono, and Shigenobu Kobayashi. 2010.  Bidirectional relation between CMA evolution strategies and natural evolution strategies. In _International Conference on Parallel Problem Solving from Nature_. Springer, 154–163.
  * Atwood and Towsley (2016) James Atwood and Don Towsley. 2016.  Diffusion-convolutional neural networks. In _Advances in neural information processing systems_. 1993–2001.
  * Balcilar et al. (2020) Muhammet Balcilar, Guillaume Renton, Pierre Héroux, Benoit Gauzere, Sebastien Adam, and Paul Honeine. 2020.  Bridging the gap between spectral and spatial domains in graph neural networks.  _arXiv preprint arXiv:2003.11702_ (2020).
  * Bergstra et al. (2011) James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011.  Algorithms for hyper-parameter optimization. In _25th annual conference on neural information processing systems (NIPS 2011)_ , Vol. 24. Neural Information Processing Systems Foundation.
  * Coley et al. (2017) Connor W Coley, Regina Barzilay, William H Green, Tommi S Jaakkola, and Klavs F Jensen. 2017\.  Convolutional embedding of attributed molecular graphs for physical property prediction.  _Journal of chemical information and modeling_ 57, 8 (2017), 1757–1772.
  * Cummings and Nassar (2020) Daniel Cummings and Marcel Nassar. 2020.  Structured Citation Trend Prediction Using Graph Neural Networks. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 3897–3901.
  * Delaney (2004) John S Delaney. 2004\.  ESOL: estimating aqueous solubility directly from molecular structure.  _Journal of chemical information and computer sciences_ 44, 3 (2004), 1000–1005.
  * Duvenaud et al. (2015) David Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Gómez-Bombarelli, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams. 2015.  Convolutional networks on graphs for learning molecular fingerprints.  _arXiv preprint arXiv:1509.09292_ (2015).
  * Fan et al. (2019) Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. 2019\.  Graph neural networks for social recommendation. In _The World Wide Web Conference_. 417–426.
  * Gilmer et al. (2017) Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017\.  Neural message passing for quantum chemistry. In _International Conference on Machine Learning_. PMLR, 1263–1272.
  * Glen et al. (2006) Robert C Glen, Andreas Bender, Catrin H Arnby, Lars Carlsson, Scott Boyer, and James Smith. 2006\.  Circular fingerprints: flexible molecular descriptors with applications from physical chemistry to ADME.  _IDrugs_ 9, 3 (2006), 199.
  * Gori et al. (2005) Marco Gori, Gabriele Monfardini, and Franco Scarselli. 2005\.  A new model for learning in graph domains. In _Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005._ , Vol. 2. IEEE, 729–734.
  * Hansen (2016) Nikolaus Hansen. 2016\.  The CMA evolution strategy: A tutorial.  _arXiv preprint arXiv:1604.00772_ (2016).
  * Henaff et al. (2015) Mikael Henaff, Joan Bruna, and Yann LeCun. 2015.  Deep convolutional networks on graph-structured data.  _arXiv preprint arXiv:1506.05163_ (2015).
  * Hersey (2015) Anne Hersey. 2015\.  _ChEMBL Deposited Data Set - AZ_dataset_.  Technical Report. EMBL-EBI.  https://doi.org/10.6019/CHEMBL3301361
  * Hutter et al. (2019) Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. 2019\.  _Automated machine learning: methods, systems, challenges_.  Springer Nature.
  * Jamieson and Talwalkar (2016) Kevin Jamieson and Ameet Talwalkar. 2016.  Non-stochastic best arm identification and hyperparameter optimization. In _Artificial Intelligence and Statistics_. PMLR, 240–248.
  * Jiang et al. (2021) Dejun Jiang, Zhenxing Wu, Chang-Yu Hsieh, Guangyong Chen, Ben Liao, Zhe Wang, Chao Shen, Dongsheng Cao, Jian Wu, and Tingjun Hou. 2021.  Could graph neural networks learn better molecular representation for drug discovery? A comparison study of descriptor-based and graph-based models.  _Journal of cheminformatics_ 13, 1 (2021), 1–23.
  * Karnin et al. (2013) Zohar Karnin, Tomer Koren, and Oren Somekh. 2013\.  Almost optimal exploration in multi-armed bandits. In _International Conference on Machine Learning_. PMLR, 1238–1246.
  * Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_ (2016).
  * Li et al. (2018) Ruoyu Li, Sheng Wang, Feiyun Zhu, and Junzhou Huang. 2018\.  Adaptive graph convolutional neural networks. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 32.
  * Li et al. (2015) Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. 2015\.  Gated graph sequence neural networks.  _arXiv preprint arXiv:1511.05493_ (2015).
  * Liao et al. (2019) Renjie Liao, Zhizhen Zhao, Raquel Urtasun, and Richard S Zemel. 2019.  Lanczosnet: Multi-scale deep graph convolutional networks.  _arXiv preprint arXiv:1901.01484_ (2019).
  * Ma et al. (2019) Yao Ma, Suhang Wang, Charu C Aggarwal, and Jiliang Tang. 2019\.  Graph convolutional networks with eigenpooling. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 723–731.
  * Mobley and Guthrie (2014) David L Mobley and J Peter Guthrie. 2014.  FreeSolv: a database of experimental and calculated hydration free energies, with input files.  _Journal of computer-aided molecular design_ 28, 7 (2014), 711–720.
  * Nomura et al. (2020) Masahiro Nomura, Shuhei Watanabe, Youhei Akimoto, Yoshihiko Ozaki, and Masaki Onishi. 2020\.  Warm Starting CMA-ES for Hyperparameter Optimization.  _arXiv preprint arXiv:2012.06932_ (2020).
  * Ramsundar et al. (2019) Bharath Ramsundar, Peter Eastman, Patrick Walters, Vijay Pande, Karl Leswing, and Zhenqin Wu. 2019\.  _Deep Learning for the Life Sciences_.  O’Reilly Media.  https://www.amazon.com/Deep-Learning-Life-Sciences-Microscopy/dp/1492039837.
  * Wang et al. (2019) Xiaofeng Wang, Zhen Li, Mingjian Jiang, Shuang Wang, Shugang Zhang, and Zhiqiang Wei. 2019\.  Molecule property prediction based on spatial graph embedding.  _Journal of chemical information and modeling_ 59, 9 (2019), 3817–3828.
  * Weininger (1988) David Weininger. 1988\.  SMILES, a chemical language and information system. 1\. Introduction to methodology and encoding rules.  _Journal of chemical information and computer sciences_ 28, 1 (1988), 31–36.
  * Wieder et al. (2020) Oliver Wieder, Stefan Kohlbacher, Mélaine Kuenemann, Arthur Garon, Pierre Ducrot, Thomas Seidel, and Thierry Langer. 2020.  A compact review of molecular property prediction with graph neural networks.  _Drug Discovery Today: Technologies_ (2020).
  * Withnall et al. (2020) Michael Withnall, Edvard Lindelöf, Ola Engkvist, and Hongming Chen. 2020.  Building attention and edge message passing neural networks for bioactivity and physical–chemical property prediction.  _Journal of Cheminformatics_ 12, 1 (2020), 1–18.
  * Wu et al. (2020) Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. 2020\.  A comprehensive survey on graph neural networks.  _IEEE transactions on neural networks and learning systems_ (2020).
  * Wu et al. (2018) Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. 2018\.  MoleculeNet: a benchmark for molecular machine learning.  _Chemical science_ 9, 2 (2018), 513–530.
  * Xinyi and Chen (2018) Zhang Xinyi and Lihui Chen. 2018.  Capsule graph neural network. In _International conference on learning representations_.
  * Xu et al. (2018) Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018\.  How powerful are graph neural networks?  _arXiv preprint arXiv:1810.00826_ (2018).
  * Yang et al. (2019) Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et al. 2019\.  Analyzing learned molecular representations for property prediction.  _Journal of chemical information and modeling_ 59, 8 (2019), 3370–3388.
  * Ying et al. (2018) Rex Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L Hamilton, and Jure Leskovec. 2018\.  Hierarchical graph representation learning with differentiable pooling.  _arXiv preprint arXiv:1806.08804_ (2018).
  * Yuan et al. (2021c) Yingfang Yuan, Wenjun Wang, George M Coghill, and Wei Pang. 2021c.  A novel genetic algorithm with hierarchical evaluation strategy for hyperparameter optimisation of graph neural networks.  _arXiv preprint arXiv:2101.09300_ (2021).
  * Yuan et al. (2021a) Yingfang Yuan, Wenjun Wang, and Wei Pang. 2021a.  A Genetic Algorithm with Tree-structured Mutation for Hyperparameter Optimisation of Graph Neural Networks.  _arXiv preprint arXiv:2102.11995_ (2021).
  * Yuan et al. (2021b) Yingfang Yuan, Wenjun Wang, and Wei Pang. 2021b.  A Systematic Comparison Study on Hyperparameter Optimisation of Graph Neural Networks for Molecular Property Prediction.  _arXiv preprint arXiv:2102.04283_ (2021).