# FINETUNA: Fine-tuning Accelerated Molecular Simulations

Joseph Musielewicz  Department of Chemical Engineering, Carnegie Mellon
University  These authors contributed equally to this work  Xiaoxiao Wang
Department of Chemical Engineering, Carnegie Mellon University  These authors
contributed equally to this work  Tian Tian  Department of Chemical
Engineering, Carnegie Mellon University  Zachary Ulissi  Department of
Chemical Engineering, Carnegie Mellon University  Corresponding author:
Zachary Ulissi<EMAIL_ADDRESS>
###### Abstract

Progress towards the energy breakthroughs needed to combat climate change can
be significantly accelerated through the efficient simulation of atomistic
systems. However, simulation techniques based on first principles, such as
Density Functional Theory (DFT), are limited in their practical use due to
their high computational expense. Machine learning approaches have the
potential to approximate DFT in a computationally efficient manner, which
could dramatically increase the impact of computational simulations on real-
world problems. However, they are limited by their accuracy and the cost of
generating labeled data. Here, we present an online active learning framework
for accelerating the simulation of atomic systems efficiently and accurately
by incorporating prior physical information learned by large-scale pre-trained
graph neural network models from the Open Catalyst Project. Accelerating these
simulations enables useful data to be generated more cheaply, allowing better
models to be trained and more atomistic systems to be screened. We also
present a method of comparing local optimization techniques on the basis of
both their speed and accuracy. Experiments on 30 benchmark adsorbate-catalyst
systems show that our method of transfer learning to incorporate prior
information from pre-trained models accelerates simulations by reducing the
number of DFT calculations by 91%, while meeting an accuracy threshold of 0.02
eV 93% of the time. Finally, we demonstrate a technique for leveraging the
interactive functionality built in to VASP to efficiently compute single point
calculations within our online active learning framework without the
significant startup costs. This allows VASP to work in tandem with our
framework while requiring 75% fewer self-consistent cycles than conventional
single point calculations. The online active learning implementation, and
examples using the VASP interactive code, are available in the open source
FINETUNA package on Github.

Keywords— Active Learning, DFT, Fine Tuning, Graph Potential Energy Surface

##  1 Introduction

Global population growth and climate change have greatly raised clean energy
demand. Heterogeneous catalysis plays a crucial role in the development of
renewable energy sources and sustainable chemical production processes [1, 2].
Examples include hydrogen generation from a renewable source, carbon dioxide
conversion into liquid fuel, and ammonia synthesis for fertilization [3, 4, 5,
6, 7, 8]. Catalyst discovery for each application is time consuming due to the
enormous design space. The numerous combinations of compositions are
impossible to be searched fully even with high throughput experimentation [9].
Alternatively, based on Brønsted-Evans-Polanyi relationships, computational
screening uses adsorption energy as a descriptor of catalyst activity and
allows a larger number of systems to be screened [10]. Generating accurate
adsorption energies for a variety of catalyst systems is a rate limiting step.
Adsorption energies are usually obtained using computational modeling methods
like Density Functional Theory (DFT) via geometric optimization [11]. Starting
with an initial atomistic structure, in each step, the energy and forces are
evaluated by DFT code. The structure is updated by an optimizer iteratively,
until a local minimum energy is found. Some common choices of optimizers
include the Conjugate Gradient (CG) and Broyden-Fletcher-Goldfarb-Shanno
(BFGS) algorithms. In this process, a series of DFT single point calculations
are performed, which requires extensive computing resources.

With broad excitement for Machine Learning (ML) in molecular simulation
community, there is now a push towards applying ML models to accelerate local
relaxations and predict adsorption energies [12, 13]. Garijo del Río et al.
developed the Gaussian Process Minimizer (GPMin), which builds a Gaussian
Process (GP) model of the potential energy surface for fast convergence [14].
Given the nature of GP s, GPMin is limited by the number of atoms in the
atomistic systems. Behler-Parrinello Neural Network paves the way for Machine
Learning potential (MLP) applications in atomistic simulations [15]. Over the
past decade, many MLP s have been developed to substitute expensive DFT
calculations. For example, ANI-1 potential and its extensions are accurate and
transferable models for organic molecules [16, 17, 18, 19]. Recently, a number
of Graph Neural Network (GNN) architectures, e.g.: SchNet [20], DimeNet [21],
DimeNet++ [22], SpinConv [23], and GemNet [24], are developed and have shown
increasing accuracy in energy and forces predictions. Usually the MLP s need a
training data set consists of a variety of catalysts systems and their
adsorption energies, which is expensive to generate with DFT. The current
state-of-the-art catalyst data set, the Open Catalyst Project 2020 (OC20) data
set, is composed of over 1.2 million DFT relaxations [25]. It opens up
possibilities for building ML models for catalysis applications. In the case
where a comprehensive training data set is not available, active learning
strategy has been utilized to train the MLP s with limited labeled data [26,
27, 28, 29, 30, 31, 32, 33]. With an active learning framework, at each step
the configuration is evaluated by the surrogate model as long as the
uncertainty of the prediction is within a threshold. Otherwise, a DFT single
point calculation is triggered, and the model is retrained with the new DFT
data. Vandermause et al. built a GP based model that can be trained on-the-fly
for molecular dynamic simulations [31, 30]. Yang et al. and Shuaibi et al.
demonstrated online and offline active learning applications to accelerate
simulation tasks [32, 33]. However, these methods usually start from scratch,
meaning the surrogate model has no prior information of the systems.

In this work, we propose an active learning scheme that leverages the prior
information to accelerate geometry optimizations [34]. More specifically, we
use a GemNet model pretrained on OC20 data set, and we retrain only a small
portion of the model for the new optimization tasks. The acceleration is
achieved by the active learning framework, where it queries sparsely from the
potential energy surface and avoids unnecessary DFT calculations. To evaluate
the performance of the proposed scheme, 30 catalyst systems from OC20 data are
randomly chosen as benchmark systems, and we compare local relaxations on the
benchmark systems using different optimization methods. We show that the fine-
tuning process improves the prediction performance of a large GNN model on
each individual system. Our best performing active learning strategy reduces
DFT single point calculations by 91%, with 93% of the final relaxed energies
are close to or lower than the reference DFT relaxed energy. We also discuss
an efficient implementation of the interactive mode of the the Vienna Ab
initio Simulation Package (VASP) calculator in the Atomistic Simulation
Environment Python package (ASE), which speeds up the calculation process by
reducing overhead costs and the number Self-Consistent Field (SCF) cycles [35,
36, 37, 38, 39, 40].

##  2 Methods

###  2.1 Online Learner Framework

Figure 1: GemNet model and online active learner workflow. (a) Examples of
systems in the OC20 data set that GemNet model has been pretrained on. (b)
Fine-tune GemNet. All the embedding, interaction and output blocks except for
the final layer output block are frozen during fine-tuning. (c) Online active
learning workflow. (d) Demonstration of different querying strategies.

We implement an online active learning framework, illustrated in figure 1,
similar to other successful active learners in this domain [29, 31, 32, 33].
The framework accelerates geometric optimizations of atomistic systems using a
machine learning potential for force estimation. The machine learning
potential, depicted in figure 1b, improves by fitting to the data generated
during the optimization. The framework also uses a parent calculator, which is
the role filled by VASP in this work, to compute the DFT energy and forces
which serve as the ground-truth labels for the generated data [41, 42].

The online active learning workflow, seen in figure 1c, starts by using the
pre-trained surrogate model to predict the energy and forces of an initial
atomistic structure. Then, the optimizer uses the forces, as the gradients of
the energy, to update the positions of the atoms to reduce the overall energy
of the atomistic system. The energy and forces of the new structure are again
predicted by the surrogate model. With each surrogate model prediction, a
querying criteria is used to accept or reject the prediction. If the
prediction is accepted, the optimization continues as normal. If it is
rejected, then the parent calculator is queried instead. The parent calculator
is used to compute the ground-truth energy and forces for the system in its
current state, and this calculation is sent to the optimizer instead of the
surrogate prediction. Additionally, the parent-calculated data point,
consisting of positions labeled with the energy and forces, is added to a set
of parent-data. It is immediately used to fine-tune the surrogate model, to
make future predictions on the atomistic system more accurate. Since the BFGS
optimizer is a second order optimizer with a numerical estimate of the
hessian, we also reset its hessian estimate to use only the set of parent-
data, so that it does not retain a hessian estimate which relies on a previous
version of the surrogate model.

The querying criteria is a key decision within the active learning framework
which affects both its performance and accuracy. We assess three strategies
for querying within our active learning framework. Relaxed-only querying
occurs only when the surrogate model predicts the forces on the atomic
structure to be below the relaxed threshold, based on the definition used by
the Open Catalyst Project (OCP) paper [25]. Regardless of the other querying
criteria used, in all three approaches we find that querying each point
predicted to be below the relaxed threshold is necessary for convergence of
the framework. K-steps querying occurs whenever the surrogate model has taken
number of optimization steps, k, since the last query. Uncertainty querying
occurs whenever a measure of the surrogate model’s uncertainty climbs above
some heuristic threshold. Uncertainty metrics are commonly used in other
active learning approaches, they are often derived from kernel methods and are
used to ensure the surrogate model does not make unsafe predictions [30].
However we expect the prior information encoded in the pre-trained surrogate
model to help alleviate this concern, potentially allowing relaxed-only or
K-steps querying to be potentially effective. We also cannot use a kernel
method approach to measuring uncertainty because our surrogate model is a
graph neural network. Therefore we use the disagreement between the members of
an ensemble of GemNet-T models to compute the uncertainty [43]. To obtain
disagreement between models we fine-tune a different output block of each
model, and then compute the standard deviation of their force predictions.

The active learning approach serves as an alternative to traditional numerical
optimization approaches used for adsorbate-catalyst systems, such as BFGS and
CG. While it still makes use of a numerical optimizer, it accelerates
optimization by substituting a number of the expensive parent calculations
with ML model inference calculations. We expect that using a pre-trained ML
model will allowed it to incorporate the prior information learned by the ML
model to further accelerate this process. However, a more obvious and perhaps
more simple method of incorporating this prior information into the
optimization is to simply perform an optimization on the initial adsorbate-
catalyst structure with the pre-trained model acting as the force calculator
until it predicts it is fully optimized, and then to check that final
structure with DFT. If the structure is not fully optimized, then finish the
optimization using DFT as the force calculator. We refer to this method as the
GemNet Warm Start method of optimization, and it is one of the benchmarks by
which we must measure the active learning approach against.

###  2.2 Fine-Tuning a Pre-Trained Model

The advantage of this framework over a more standard optimization approach,
and over other active learning frameworks, is the incorporation of prior
information learned by the pre-trained machine learning potential. In this
work, we use a pre-trained graph neural network called GemNet-T [24] as the
basis for the machine learning potential. Graph neural networks have recently
shown great success at approximating DFT calculations for adsorbate-catalyst
systems, and GemNet-T is the current state of the art among them, according to
the OCP leaderboard. Making an energy and force prediction with GemNet-T is
inexpensive relative to DFT, on the order of seconds on a CPU core compared to
hours on a CPU core [25]. Still, GemNet-T lacks the accuracy required to
reliably find a structure with a local minimum energy, and it is expensive to
train from scratch, so we fine-tune it on-the-fly using the parent data from
the relaxation. During fine-tuning we start with the pre-trained model and
perform training only on one of the GemNet-T output blocks using the AdamW
optimizer, with similar settings to the ones used by the authors of the
GemNet-T paper [24]. The goal of our training approach is to preserve the
underlying physical information learned by the GemNet-T model while improving
the model’s accuracy specifically for the atomistic system it is helping to
optimize. This process of fine-tuning large pre-trained models to "transfer"
their knowledge between domains has a track record of success in other domains
of machine learning [44]. Although in this work we employ this strategy for an
extremely low data regime to improve specific predictions on-the-fly, instead
of using a larger data set to transfer on more general tasks.

In our on-the-fly training approach, we retrain the model sequentially on each
new data point queried from the parent calculator. We take an incremental
approach to fine-tuning, where the previous state of the model is retrained on
only the most recent point for 400 epochs. We intend for this incremental
approach to tune the model to match the most recent state of the atomistic
system as it changes throughout the optimization. We use an initial learning
rate of 0.0003 and a loss-based learning rate scheduler to reduce the learning
rate by a factor of 0.9 when the loss fails to improve for at least 3 training
steps. In this work, the fine-tuned state of the model is only used during the
optimization of a specific system, and is not reused for other systems.
However the model could be preserved for use in other active learning
optimizations if similar systems needed to be optimized. Details on the
effects of fine-tuning the GemNet model can be found in the supplementary
information.

###  2.3 Experiment Setup

To test each online active learning method we perform geometric optimizations
on a set of 30 atomistic systems, and then compare their performance in terms
of cost and accuracy metrics to some baseline optimization methods. We sample
the set of 30 atomistic systems from the OC20 validation set. We use systems
only from the set of out-of-domain materials for both adsorbates and metal
surfaces. This means each of these systems consist of an adsorbate placed
above a metal substrate, and the pretrained GemNet-T model we use was never
explicitly trained on any systems with the same adsorbate molecules or metal
substrate. These 30 systems can be found in the FINETUNA GitHub repository
with Google Colab ASE examples [34, 45]. We used VASP as our parent DFT
calculator, with the settings specified by the Open Catalyst Project [25].

We compare each method to CG, an optimization method with a native
implementation in VASP. We choose this for the baseline since it is the
default optimization method for VASP, and it was used to generate all of the
relaxation data for the OCP [25] data sets. We perform geometric optimizations
using the following techniques for comparison to ASE CG and to each other:

  * •

VASP CG: VASP with built-in Conjugate Gradient optimization.

  * •

ASE BFGS: VASP ASE calculator with ASE BFGS optimization.

  * •

GemNet Warm Start: Pretrained GemNet-T calculator with ASE BFGS optimization
until relaxed, followed by ASE VASP calculator with ASE BFGS optimization from
that point.

  * •

ASE GPMin: An optimization strategy built in to ASE which uses VASP as a
parent calculator and trains a Gaussian process on-the-fly to smooth out the
potential energy surface of VASP [14].

  * •

Relax-only Online Learner: Online learner with VASP as the parent calculator
and GemNet-T as the surrogate model. Querying the parent calculator and
retraining the surrogate model only when the surrogate model predicts a
relaxed structure.

  * •

K-steps Online Learner: Same as Relax-only Online Learner while also querying
whenever a number, k, steps have been taken since the last query.

  * •

Uncertainty Online Learner: Same as Relax-only Online Learner while also
querying whenever the uncertainty metric for the surrogate model predictions
rises above some threshold.

We introduce a framework for comparing geometric optimization methods in terms
of speed and accuracy. As a proxy for speed we compute the ratio of the number
of DFT singlepoint calculations (DFT calls) made during the optimization of a
given system to the number of DFT calls made when using ASE CG to optimize
that system. As a proxy for accuracy we compute the fraction of optimizations
which satisfy two conditions. First, the parent calculator (VASP) calculates
that the final point is relaxed, meaning the maximum force on any atom is
$\text{F}_{\text{max}}<=0.03$ eV/Å. And second, the parent calculator (VASP)
calculates the energy of final point is less than or equal to the energy of
the final point of the ASE CG optimization, plus a $0.02$ eV threshold for
error. We choose the $\text{F}_{\text{max}}$ threshold of 0.03 eV/Åto match
the OCP criteria for relaxation [25]. We choose the $0.02$ eV threshold for
energy heuristically, based on experience working with these calculations, and
the distribution of final energy differences between VASP CG and ASE BFGS. We
expect $0.02$ eV to be a reasonable guess for the energy threshold, but it
could easily be changed depending on one’s expectation for similarity between
optimization strategies.

##  3 Results & Discussion

###  3.1 Mapping Trajectories with PCA

The relaxation process of a randomly selected test system is illustrated in
figure 2a. In the GemNet Warm Start and active learning methods, the steps
before the first DFT point are identical because they rely solely on the
pretrained GemNet predictions. The active learning strategy shown in this
figure is the K-steps, and a more detailed comparison of different querying
strategies can be found in section 3.2. Figure 2b shows the DFT energy
relative to the relaxed energy from VASP CG at each DFT call. In the GemNet
Warm Start and active learning methods, DFT relaxation starts from an energy
0.18 eV above the relaxed energy, suggesting that the pretrained GemNet model
has relaxed the structure close to the final structure. In the region near the
local minimum (energy less than 0.2 eV), active learning converges much faster
than other methods. Therefore, the overall acceleration of the relaxation
process is a combined effect of the large pretrained GNN model and the active
learning framework.

To visualize and qualitatively compare relaxation trajectories from each
method, we perform Principal Component Analysis (PCA). We adapt the B2
descriptor from Vandermause et al. [31] to represent each atomic configuration
in the trajectory. The descriptors are vectors which we linearly transform
into principle components which describe the majority of the variation in the
positions. The first two components are plotted in figure 2. It is clear that
ASE BFGS, GemNet Warm Start, and the online active learner take similar
relaxation routes as they possess similar curves. This is reasonable because
all three of these methods use the same BFGS algorithm as their underlying
optimizer. VASP CG and ASE GPMin deviate further from the BFGS-based
trajectories, but they end up at the same relaxed configuration in PCA space.
This suggests that the choice of optimizer could affect the path optimization
takes.

Figure 2: Detailed comparison of a randomly chosen system. Solid markers
represent DFT single point calculations, and hollow markers represents
pretrained GemNet predictions and fine-tune GemNet predictions. (a) Relaxation
steps of DFT-only runs and MLP involved runs. For each strategy we see how the
DFT calls are dispersed along the trajectory. (b) Energy relative to the
relaxed energy from VASP CG at each DFT steps, we see here that each strategy
converges to a similar final energy. (c) Principal component analysis of
relaxation trajectories, and here we see each strategy follows a similar path.

###  3.2 Comparing Optimization Methods

We find the K-steps, single GemNet online learner to be a significant
improvement over other methods in terms of speed, without sacrificing
accuracy. In figure 3 we see that it performs the set of optimizations with an
average of 10 times fewer parent calls when compared to the baseline, and at
the same overall accuracy threshold as its underlying optimizer, BFGS. The
addition of the K-steps querying criteria results in only a slight improvement
over the relax-only querying strategy alone. This is due to the majority of
parent calls being caused by relaxation queries, with only one or two calls at
the beginning of the optimization being caused by K-steps . Occasionally the
K-steps querying results in some speed-up, but in most cases has little impact
on the optimization, leading to a small difference. The use of ensembling hurt
the accuracy of the online learner strategy, and the use of uncertainty
calculated from the ensemble failed to outperform the relax-only strategy
alone in both speed and accuracy. We speculate that the uncertainty method of
querying failed to show improvement because measuring the disagreement within
an ensemble of graph models is a poor surrogate for uncertainty in this case.
It may instead be more analogous to a simple measurement of the distance of a
structure from most recent parent call, similar to the K-steps method of
querying. Regardless of the reason, both online learning strategies using
ensembling underperformed the single model strategies in terms of accuracy
without a significant gain in speed to make up for the increased training
time. Therefore we favor the use of single model strategies, along with the
K-steps querying strategy, going forward.

Figure 3: Accuracy versus speed outcomes on the test set of 30 systems for the
different optimization strategies. The accuracy metric is the fraction of test
outcomes with an energy less than or equal to the base-case optimizer (ASE CG)
relaxed energy for that system plus a margin of error of 0.02 eV. The speed
metric is the average ratio of parent (DFT) calls made by the optimizer to
parent calls made by the base-case optimizer.

Looking more closely at the distribution of accuracy and speed, we compare the
distribution of outcomes of the K-steps online active learning strategy to the
other methods in figure 4. Here we see the online active learning consistently
improves speed when compared to all other methods, with rare exceptions where
it is narrowly beaten by ASE GPMin or GemNet Warm Start techniques on a
particular system. It always beats traditional numerical methods like BFGS and
CG, which is expected. In terms of accuracy, the online active learner often
finds the same energy outcome as VASP CG and the other methods. However, it is
more likely to find a lower minimum energy than any of the others. We
speculate that this could be due to the noise introduced by the regular fine-
tuning of the GemNet-T model. Since the optimization algorithm is searching
for a local minimum over a changing function, it may be occasionally forced
out of shallow local minima, and therefore more likely to find deeper local
minima. Regardless, the online active learner was just as likely as ASE BFGS
and ASE GPMin to reach a local minimum below the threshold set by VASP CG
which makes it equivalent to ASE BFGS and ASE GPMin by this accuracy metric,
but with consistently superior speed-up.

Figure 4: Comparison of all 30 systems with different optimization strategies.
(a) The top plot shows the number of DFT single points calculations required
for each strategy as a percentage of VASP CG. The bottom plot shows the
relaxed energy difference with respect to VASP CG. The relaxed energy from
different strategies is similar if it is within 0.02 eV of VASP CG reference
energy. (b) Visualizations of the reference relaxed structures from VASP CG.
Table 1: Results for different optimization methods and online active learning
strategies, averaged across 30 test systems.

Algorithm | DFT calls (%) $\downarrow$ |  E - EVASP CG reference
---|---|---
$\leq$ \- 0.02 eV (%) |  $\in$[- 0.02, 0.02] eV (%) |  $\leq$ 0.02 eV (%) $\uparrow$
Vasp CG | 100 | 0 | 100 | 100
ASE BFGS |  60 $\pm$ 4 | 10 | 83 | 93
ASE GPMin |  34 $\pm$ 3 | 10 | 83 | 93
GemNet Warm Start |  29 $\pm$ 3 | 20 | 70 | 90
Online Learner | Uncertainty (ensemble) |  10 $\pm$ 1 | 33 | 53 | 86
Relax-only (ensemble) |  9 $\pm$ 1 | 30 | 60 | 90
Relax-only (single) |  10 $\pm$ 1 | 30 | 63 | 93
K-steps (single) |  9 $\pm$ 1 | 30 | 63 | 93

###  3.3 Optimizing Run-time Performance of Online Learners

The speed-up of online learners by means of parent DFT calls is not
essentially the run-time speed-up in real-world applications, due to the
following factors: i) steps to reach electronic convergence vary in each
single point calculation, and ii) the GNN training time is non-negligible. In
this section, we discuss the software engineering efforts we have made to
address the above issues for performance optimization and take full advantage
of built-in interactive features of VASP.

Like other DFT packages, VASP uses an iterative scheme [41] to solve the Kohn-
Sham equation, making the time consumed by DFT calculations proportional to
the number of SCF loops. In this respect, using the standard ASE VASP
calculator has several drawbacks: i) more SCF loops to achieve convergence due
to random wavefunction initialization, and ii) system overhead caused by start
/ stop of the VASP program and file I/O. Here we design a new calculator
interface for VASP by leveraging its interactive mode  (VASPInteractive) [40]
to avoid shutting down the VASP program, and reduce the of SCF loops by an
average of $\sim{}75\%$ when compared with standard ASE VASP calculator.
Details of the implementation can be found in the supplementary information.

To systematically study the impact of system overhead and SCF loops on the
run-time performance of DFT single point, we compare three different
implementations of calculator interfaces to VASP, as schematically shown in
figure 5a:

  1. i)

Standard ASE VASP (M1): cold-start VASP process on each single point

  2. ii)

ASE VASP with wavefunction cache (M2): use local file (WAVECAR) to store and
pass wavefunction between single points.

  3. iii)

VASPInteractive (M3): stream-based calculator maintaining a long-running VASP
process

Figure 5: Software engineering approaches for optimizing online learner run-
time performance. a. Schematic comparison between three implementations
(M1-M3) of DFT calculators used in active learning framework. b. Total DFT SCF
loops of different optimization strategies (ASE BFGS, ASE GPMin and MLP) and
VASP calculator interfaces (M1-M3) compared with that of VASP CG. c. Walltime
speed-up of the whole relaxation process for different optimization strategies
(ASE BFGS, ASE GPMin, fine-tuning on CPU and fine-tuning on GPU), and VASP
calculator interfaces (M1-M3) compared with that of VASP CG. Using online
learner with MLP fine-tuned on GPU and VASPInteractive as the VASP calculator
(M3), the mean walltime speed-up approaches the ideal speed-up of parent DFT
calls as shown in figure 3. All MLP examples are using best strategy in
section 3.2 (K-step online learner with single GemNet model and 400 epochs).

Figure 5b shows the total SCF loops compared with VASP CG, when choosing
different optimization strategies (ASE BFGS, ASE GPMin or online learner) and
VASP interfaces (M1-M3). We find that independent of the optimization
strategy, ASE VASP with wavefunction cache (M2) and VASPInteractive (M3)
reduce the total SCF loops compared with ASE VASP by roughly 50% and 75% on
average, respectively. We note although in both M2 and M3 the wavefunction
information is passed between subsequent DFT single points, M2 requires more
steps for converging the orbitals when read from a locally-cached WAVECAR file
[46]. As a result, the choice of VASP interface has huge impact on the
performance: using ASE VASP (M1 or M2) as calculator, ASE BFGS and ASE GPMin
optimizers may have computational cost similar or even greater than VASP CG,
despite reduction of parent DFT calls. On the other hand, the total SCF
reduction when combining online learner and VASPInteractive is consistent with
its reduction of parent DFT calls as shown in figure 3. We note that all of
these metrics are dependent on precise computer hardware/architecture, and
should only be viewed as rough estimates.

Further run-time performance optimization of the online learner framework
requires to reduce the walltime ratio between training and DFT. This can be
achieved by moving the ML training workload from CPU to GPU device. For the
best fine-tuning strategy in section 3.2 (K-steps, single GemNet, 400 epochs),
single training steps requires O($10^{2}$ s) on a 32-core CPU, similar to DFT
single point on the same CPU architecture (O($10^{2}$ s) $\sim$ O($10^{3}$
s)). On the other hand, the training cost drastically drops to O($10^{0}$ s)
$\sim$ O($10^{1}$ s) on a GPU device. Figure 5c shows rough walltime speed-ups
of the whole relaxation process as compared with VASP CG, when choosing
different optimization strategies and VASP interfaces, but precise speedups
are of course dependent on precise computer hardware. When taking full
advantage of VASP’s interactive capabilities in the online learner, simply by
switching training from CPU to GPU, we can improve the mean walltime speed-up
from $\sim$5x to $\sim$10x, approaching the ideal speed-up by means of parent
DFT calls as shown in figure 3. We note that all of this is possible only due
to the much-appreciated functionality of VASP to run interactively.

##  4 Conclusion

The costly simulation of atomic systems to generate data is often a bottleneck
holding back machine learning methods from being effectively applied to new
materials discovery problems. We have developed a comprehensive approach for
performing and evaluating the acceleration of atomic simulations. We applied
this approach to the local optimization of adsorbate-catalyst structures using
VASP. We have shown that a transfer learning technique can be used to
incorporate prior information from large-scale OCP graph models into an active
learning framework. We also provided a method to evaluate its performance in
terms of speed and accuracy. Using this method, we have shown that our active
learning framework provides significant speed increases over other techniques
for geometric optimizations of atomic structures, while maintaining equivalent
accuracy, even without the use of an uncertainty metric. Further, we have
demonstrated a method to run the VASP code in an interactive mode to mitigate
the real-world computational cost of performing single point calculations
during the optimization, ensuring the computational time does not suffer from
inefficiencies associated with repeated cold restarts of the VASP code.

There are three main ways in which this work ought to be expanded. The first
is to use the active learning framework on distinct atomic simulation tasks,
to determine what limitations exist in terms of transferring the physical
information learned by the pre-trained OCP model. The nudged elastic band
method of finding transition states might be an especially good use case for
this method, since the fine-tuned model could be reused for each new
relaxation [47, 48]. The second is to test more sophisticated methods of fine-
tuning GemNet-T or other models on small amounts of data. In other domains of
machine learning there exists evidence that additional adapter layers may be
more effective for fine-tuning in very low data regimes [49]. Finally, it
seems advantageous to adapt this method to other types of simulations where
pre-trained OCP models are less likely to be effective, such as within high-
temperature molecular dynamics simulations. In these simulations the same
method for active learning could be applied, using a similar fine-tuning
technique on a similar model, pre-trained on data specific to the molecular
dynamics simulation instead.

In addition to these areas of exploration, improvements or alternatives to the
online active learning method of accelerating simulations described in this
work should be compared using the framework of accuracy and speed we have
described. This should result in faster and more useful techniques which can
be adopted more confidently in practice. Methods for accelerating geometric
optimizations performed by VASP involving single point calculations should
also make use of the VASPInteractive code we have provided, to improve their
computational efficiency and take full advantage of VASP’s interactive
interface. Future dataset generation and screening tasks normally performed
using VASP should consider using online active learning and VASPInteractive in
tandem to reduce computational costs and increase throughput. We also
recommend that GPU-enabled machines be used to achieve the full acceleration
enabled by the online active learning, but CPU-only machines will still
achieve significant speed-up.

##  5 Acknowledgements

We acknowledge the support from the U.S. Department of Energy, Office of
Science, Basic Energy Sciences Award #DE-SC0019441, and Office of Energy
Efficiency and Renewable Energy under grant #DE-0008822. We also thank AJ
Medford (Georgia Tech), Andrew Peterson (Brown), Muhammed Shuaibi and Adeesh
Kolluru (CMU), as well as C. Lawrence Zitnick, Aditya Grover, Anuroop Sriram,
Janice Lan, Nima Shoghi, and Siddharth Goyal (FAIR) for their insightful
discussions.

This report was prepared as an account of work sponsored by an agency of the
United States Government. Neither the United States Government nor any agency
thereof, nor any of their employees, makes any warranty, express or implied,
or assumes any legal liability or responsibility for the accuracy,
completeness, or usefulness of any information, apparatus, product, or process
disclosed, or represents that its use would not infringe privately owned
rights. Reference herein to any specific commercial product, process, or
service by trade name, trademark, manufacturer, or otherwise does not
necessarily constitute or imply its endorsement, recommendation, or favoring
by the United States Government or any agency thereof. The views and opinions
of authors expressed herein do not necessarily state or reflect those of the
United States Government or any agency thereof.

##  6 Data Availability Statement

All data that support the findings of this study are included within the
Github repository at https://github.com/ulissigroup/finetuna_manuscript.

## References

  * [1] C. M. Friend and B. Xu, “Heterogeneous Catalysis: A Central Science for a Sustainable Future,” Accounts of Chemical Research, vol. 50, pp. 517–521, mar 2017.
  * [2] H. Liu, L. Wei, F. Liu, Z. Pei, J. Shi, Z. J. Wang, D. He, and Y. Chen, “Homogeneous, Heterogeneous, and Biological Catalysts for Electrochemical N2 Reduction toward NH3 under Ambient Conditions,” ACS Catalysis, pp. 5245–5267, 2019.
  * [3] I. Yuranov, N. Autissier, K. Sordakis, A. F. Dalebrook, M. Grasemann, V. Orava, P. Cendula, L. Gubler, and G. Laurenczy, “Heterogeneous Catalytic Reactor for Hydrogen Production from Formic Acid and Its Use in Polymer Electrolyte Fuel Cells,” ACS Sustainable Chemistry and Engineering, vol. 6, pp. 6635–6643, may 2018.
  * [4] R. P. Ye, J. Ding, W. Gong, M. D. Argyle, Q. Zhong, Y. Wang, C. K. Russell, Z. Xu, A. G. Russell, Q. Li, M. Fan, and Y. G. Yao, “CO2 hydrogenation to high-value products via heterogeneous catalysis,” Nature Communications 2019 10:1, vol. 10, pp. 1–15, dec 2019.
  * [5] S. De, A. Dokania, A. Ramirez, and J. Gascon, “Advances in the Design of Heterogeneous Catalysts and Thermocatalytic Processes for CO2 Utilization,” ACS Catalysis, vol. 10, pp. 14147–14185, dec 2020.
  * [6] S. L. Foster, S. I. Bakovic, R. D. Duda, S. Maheshwari, R. D. Milton, S. D. Minteer, M. J. Janik, J. N. Renner, and L. F. Greenlee, “Catalysts for nitrogen reduction to ammonia,” Nature Catalysis 2018 1:7, vol. 1, pp. 490–500, jul 2018.
  * [7] Y. Kobayashi, Y. Tang, T. Kageyama, H. Yamashita, N. Masuda, S. Hosokawa, and H. Kageyama, “Titanium-Based Hydrides as Heterogeneous Catalysts for Ammonia Synthesis,” J. Am. Chem. Soc, vol. 139, p. 37, 2017.
  * [8] V. S. Marakatti and E. M. Gaigneaux, “Recent Advances in Heterogeneous Catalysis for Ammonia Synthesis,” ChemCatChem, vol. 12, pp. 5838–5857, dec 2020.
  * [9] K. McCullough, T. Williams, K. Mingle, P. Jamshidi, and J. Lauterbach, “High-throughput experimentation meets artificial intelligence: A new pathway to catalyst discovery,” Physical Chemistry Chemical Physics, vol. 22, no. 20, pp. 11174–11196, 2020.
  * [10] T. Bligaard, J. K. Nørskov, S. Dahl, J. Matthiesen, C. H. Christensen, and J. Sehested, “The Brønsted-Evans-Polanyi relation and the volcano curve in heterogeneous catalysis,” Journal of Catalysis, vol. 224, pp. 206–217, may 2004.
  * [11] W. Kohn, A. D. Becke, and R. G. Parr, “Density functional theory of electronic structure,” Journal of Physical Chemistry, vol. 100, pp. 12974–12980, aug 1996.
  * [12] B. R. Goldsmith, J. Esterhuizen, J. X. Liu, C. J. Bartel, and C. Sutton, “Machine learning for heterogeneous catalyst design and discovery,” AIChE Journal, vol. 64, pp. 2311–2323, jul 2018.
  * [13] T. Williams, K. McCullough, and J. A. Lauterbach, “Enabling Catalyst Discovery through Machine Learning and High-Throughput Experimentation,” Chemistry of Materials, vol. 32, pp. 157–165, jan 2020.
  * [14] E. G. Del Río, J. J. Mortensen, and K. W. Jacobsen, “Local Bayesian optimizer for atomic structures,” Physical Review B, vol. 100, no. 10, p. 104103, 2019.
  * [15] J. Behler and M. Parrinello, “Generalized neural-network representation of high-dimensional potential-energy surfaces,” Physical Review Letters, vol. 98, no. 14, 2007.
  * [16] J. S. Smith, O. Isayev, and A. E. Roitberg, “ANI-1: an extensible neural network potential with DFT accuracy at force field computational cost,” Chemical Science, vol. 8, pp. 3192–3203, mar 2017.
  * [17] X. Gao, F. Ramezanghorbani, O. Isayev, J. S. Smith, and A. E. Roitberg, “TorchANI: A Free and Open Source PyTorch-Based Deep Learning Implementation of the ANI Neural Network Potentials,” Journal of Chemical Information and Modeling, vol. 60, pp. 3408–3415, jul 2020.
  * [18] C. Devereux, J. S. Smith, K. K. Davis, K. Barros, R. Zubatyuk, O. Isayev, and A. E. Roitberg, “Extending the Applicability of the ANI Deep Learning Molecular Potential to Sulfur and Halogens,” Journal of Chemical Theory and Computation, vol. 16, pp. 4192–4202, jul 2020.
  * [19] J. S. Smith, R. Zubatyuk, B. Nebgen, N. Lubbers, K. Barros, A. E. Roitberg, O. Isayev, and S. Tretiak, “The ANI-1ccx and ANI-1x data sets, coupled-cluster and density functional theory properties for molecules,” Scientific Data 2020 7:1, vol. 7, pp. 1–10, may 2020.
  * [20] K. T. Schütt, H. E. Sauceda, P. J. Kindermans, A. Tkatchenko, and K. R. Müller, “SchNet - A deep learning architecture for molecules and materials,” Journal of Chemical Physics, vol. 148, p. 241722, mar 2018\.
  * [21] J. Klicpera, J. Groß, and S. Günnemann, “Directional message passing for molecular graphs,” arXiv preprint arXiv:2003.03123, 2020.
  * [22] J. Klicpera, S. Giri, J. T. Margraf, and S. Günnemann, “Fast and uncertainty-aware directional message passing for non-equilibrium molecules,” arXiv preprint arXiv:2011.14115, 2020.
  * [23] M. Shuaibi, A. Kolluru, A. Das, A. Grover, A. Sriram, Z. Ulissi, and C. L. Zitnick, “Rotation invariant graph neural networks using spin convolutions,” arXiv:2106.09575, 2021.
  * [24] J. Gasteiger, F. Becker, and S. Günnemann, “Gemnet: Universal directional graph neural networks for molecules,” arXiv:2106.08903, 2021\.
  * [25] L. Chanussot, A. Das, S. Goyal, T. Lavril, M. Shuaibi, M. Riviere, K. Tran, J. Heras-Domingo, C. Ho, W. Hu, A. Palizhati, A. Sriram, B. Wood, J. Yoon, D. Parikh, C. L. Zitnick, and Z. Ulissi, “Open Catalyst 2020 (OC20) Dataset and Community Challenges,” ACS Catalysis, vol. 11, pp. 6059–6072, may 2021.
  * [26] W. Wang, T. Yang, W. H. Harris, and R. Gómez-Bombarelli, “Active learning and neural network potentials accelerate molecular screening of ether-based solvate ionic liquids,” Chemical Communications, vol. 56, pp. 8920–8923, aug 2020.
  * [27] T. Lookman, P. V. Balachandran, D. Xue, and R. Yuan, “Active learning in materials science with emphasis on adaptive sampling using uncertainties for targeted design,” npj Computational Materials, vol. 5, pp. 1–17, feb 2019\.
  * [28] K. Tran and Z. W. Ulissi, “Active learning across intermetallics to guide discovery of electrocatalysts for CO2 reduction and H2 evolution,” Nature Catalysis, vol. 1, no. 9, pp. 696–703, 2018.
  * [29] M. Zhong, K. Tran, Y. Min, C. Wang, Z. Wang, C. T. Dinh, P. De Luna, Z. Yu, A. S. Rasouli, P. Brodersen, S. Sun, O. Voznyy, C. S. Tan, M. Askerka, F. Che, M. Liu, A. Seifitokaldani, Y. Pang, S. C. Lo, A. Ip, Z. Ulissi, and E. H. Sargent, “Accelerated discovery of CO2 electrocatalysts using active machine learning,” Nature 2020 581:7807, vol. 581, pp. 178–183, may 2020\.
  * [30] J. Vandermause, Y. Xie, J. S. Lim, C. J. Owen, and B. Kozinsky, “Active learning of reactive bayesian force fields: Application to heterogeneous hydrogen-platinum catalysis dynamics,” arXiv:2106.01949, 2021.
  * [31] J. Vandermause, S. B. Torrisi, S. Batzner, Y. Xie, L. Sun, A. M. Kolpak, and B. Kozinsky, “On-the-fly active learning of interpretable Bayesian force fields for atomistic rare events,” npj Computational Materials, vol. 6, no. 1, 2020.
  * [32] Y. Yang, O. A. Jiménez-Negrón, and J. R. Kitchin, “Machine-learning accelerated geometry optimization in molecular simulation,” Journal of Chemical Physics, vol. 154, no. 23, p. 234704, 2021.
  * [33] M. Shuaibi, S. Sivakumar, R. Q. Chen, and Z. W. Ulissi, “Enabling robust offline active learning for machine learning potentials using simple physics-based priors,” Machine Learning: Science and Technology, vol. 2, no. 2, p. 025007, 2021.
  * [34] “FINETUNA: Fine-Tuning Accelerated Molecular Simulations.” https://github.com/ulissigroup/finetuna, 2022.  Accessed on 03.29.2022.
  * [35] G. Kresse and J. Furthmüller, “Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set,” Computational Materials Science, vol. 6, pp. 15–50, jul 1996.
  * [36] G. Kresse, “Ab initio molecular dynamics for liquid metals,” Journal of Non-Crystalline Solids, vol. 192-193, pp. 222–229, 1995.
  * [37] G. Kresse and J. Hafner, “Ab initio molecular-dynamics simulation of the liquid-metalamorphous- semiconductor transition in germanium,” Physical Review B, vol. 49, no. 20, pp. 14251–14269, 1994.
  * [38] G. Kresse and J. Furthmüller, “Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set,” Physical Review B - Condensed Matter and Materials Physics, vol. 54, no. 16, pp. 11169–11186, 1996.
  * [39] A. Hjorth Larsen, J. JØrgen Mortensen, J. Blomqvist, I. E. Castelli, R. Christensen, M. Dułak, J. Friis, M. N. Groves, B. Hammer, C. Hargus, E. D. Hermes, P. C. Jennings, P. Bjerre Jensen, J. Kermode, J. R. Kitchin, E. Leonhard Kolsbjerg, J. Kubal, K. Kaasbjerg, S. Lysgaard, J. Bergmann Maronsson, T. Maxson, T. Olsen, L. Pastewka, A. Peterson, C. Rostgaard, J. SchiØtz, O. Schütt, M. Strange, K. S. Thygesen, T. Vegge, L. Vilhelmsen, M. Walter, Z. Zeng, and K. W. Jacobsen, “The atomic simulation environment - A Python library for working with atoms,” Journal of Physics Condensed Matter, vol. 29, p. 273002, jun 2017.
  * [40] “VASPInteractive: Interactive Vasp calculator.” https://github.com/ulissigroup/vasp-interactive, 2022.  Accessed on 03.04.2022.
  * [41] G. Kresse and J. Furthmüller, “Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set,” Physical Review B - Condensed Matter and Materials Physics, vol. 54, no. 16, pp. 11169–11186, 1996.
  * [42] G. Kresse and J. Hafner, “Ab initio molecular dynamics for liquid metals,” Physical Review B, vol. 47, no. 1, pp. 558–561, 1993.
  * [43] B. Settles, “Active learning literature survey,” Computer Sciences Technical Report 1648, University of Wisconsin–Madison, 2009.
  * [44] C. Sun, X. Qiu, Y. Xu, and X. Huang, “How to Fine-Tune BERT for Text Classification?,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11856 LNAI, pp. 194–206, oct 2019.
  * [45] “FINETUNA: Fine-Tuning Accelerated Molecular Simulations manuscript.” https://github.com/ulissigroup/finetuna_manuscript, 2022.  Accessed on 04.05.2022.
  * [46] “NELMDL: non-self consistent steps in electronic minimization from The VASP Manual.” https://www.vasp.at/wiki/index.php/NELMDL, 2022\.  Accessed on 03.04.2022.
  * [47] J. A. Garrido Torres, P. C. Jennings, M. H. Hansen, J. R. Boes, and T. Bligaard, “Low-Scaling Algorithm for Nudged Elastic Band Calculations Using a Surrogate Machine Learning Model,” Physical Review Letters, vol. 122, no. 15, 2019.
  * [48] A. A. Peterson, “Acceleration of saddle-point searches with machine learning,” The Journal of Chemical Physics, vol. 145, p. 074106, aug 2016\.
  * [49] N. Houlsby, A. Giurgiu, S. Jastrzçbski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for NLP,” 36th International Conference on Machine Learning, ICML 2019, vol. 2019-June, pp. 4944–4953, feb 2019.