# Open Challenges in Developing Generalizable Large Scale Machine Learning
Models for Catalyst Discovery

Adeesh Kolluru  Co-first Author  Muhammed Shuaibi  Co-first Author  Aini
Palizhati  Department of Chemical Engineering, Carnegie Mellon University
Nima Shoghi  Fundamental AI Research at Meta AI  Abhishek Das  Fundamental AI
Research at Meta AI  Brandon Wood  Fundamental AI Research at Meta AI  C.
Lawrence Zitnick  Fundamental AI Research at Meta AI  John R Kitchin
Department of Chemical Engineering, Carnegie Mellon University  Zachary W
Ulissi  Department of Chemical Engineering, Carnegie Mellon University
<EMAIL_ADDRESS>
###### Abstract

The development of machine learned potentials for catalyst discovery has
predominantly been focused on very specific chemistries and material
compositions. While effective in interpolating between available materials,
these approaches struggle to generalize across chemical space. The recent
curation of large-scale catalyst datasets has offered the opportunity to build
a universal machine learning potential, spanning chemical and composition
space. If accomplished, said potential could accelerate the catalyst discovery
process across a variety of applications (CO2 reduction, NH3 production, etc.)
without additional specialized training efforts that are currently required.
The release of the Open Catalyst 2020 Dataset (OC20) 1 has begun just that,
pushing the heterogeneous catalysis and machine learning communities towards
building more accurate and robust models. In this perspective, we discuss some
of the challenges and findings of recent developments on OC20. We examine the
performance of current models across different materials and adsorbates to
identify notably underperforming subsets. We then discuss some of the modeling
efforts surrounding energy-conservation, approaches to finding and evaluating
the local minima, and augmentation of off-equilibrium data. To complement the
community’s ongoing developments, we end with an outlook to some of the
important challenges that have yet to be thoroughly explored for large-scale
catalyst discovery.

Department of Chemical Engineering, Carnegie Mellon University Department of
Chemical Engineering, Carnegie Mellon University

##  1 Introduction

Figure 1: Summary of challenges associated with training on large dataset with
large ML potentials discussed in the paper. Top left Trade offs in direct and
gradient GNN force predictions. Top right An example system for a case where
the distance metrics are relatively good for the direct approach but the force
metrics are worse. Bottom left Demonstration of inconsistent error across a
metallic surface and a non-metal through an example. Bottom right Augmenting
existing relaxation datasets with off-equlibrium data can aid in relaxation
performance.

Catalysts have played a key role in the synthesis of everyday chemicals and
fuels necessary for a 21st century society. As renewable energy prices
continue to decrease, traditional chemical synthesis processes are being
revisited for more sustainable alternatives. At the center of this, catalyst
discovery plays a key role in the advancement of renewable energy processes
and sustainable chemical production, i.e. ammonia for fertilizer and hydrogen
production. Unfortunately, the search space for catalyst materials is enormous
for even high-throughput experiments 2. This presents a need for computational
tools to simulate systems through quantum mechanical (QM) models like Density
Functional Theory (DFT). QM approaches have made notable advancements in
bridging computational results to experimental findings 3, 4, 5, 6, 7, 8.
While effective, QM tools scale very poorly, $O(N^{3})$ or worse in the number
of electrons. The computational cost associated with QM tools render them
infeasible to the scale of the systems and search space desired for catalyst
discovery. As a result, the catalysis community has moved towards a more data
driven approach 9, 10, 11, 12, 13. With the QM data available, researchers are
often interested in building machine learning surrogates for a particular
chemical property 14, 15, 16, 17. Such efforts, however, were limited to the
finite data available, often for a very specific chemistry or system, limiting
the generalizability ability of such models 10, 18. Fortunately, as the
community continues to curate larger, and more diverse datasets, machine
learning models will continue to improve as they move towards larger, and more
sophisticated architectures.

In the field of small molecules, a vast collection of datasets have been
developed for varying use cases, including molecular dynamics simulations
(MD1719, ANI-120, COLL21) and quantum mechanical properties (QM922 Alchemy23).
These datasets are often limited to a few (5-10) unique elements, on average
10-20 atoms per system, and training set sizes in the range of 10k-1M samples.
In the field of heterogeneous catalysis, datasets are often much more limited
with training set sizes between 100 - 50k 24, 25, 26, 27. These datasets were
often created for very specific applications involving a handful of small
adsorbates (i.e. hydrogen containing adsorbates on transition metal surfaces,
CO2 reduction catalysts, etc.). The release of OC20 marks a push towards a
large, sparse collection of the material space. OC20 spans 55 unique elements,
82 adsorbates and includes a collection of unary, binary and ternary
materials. A total of 1.28 million DFT relaxations were performed, comprising
$\sim$260M single point evaluations of system energy and per-atom forces.

OC20 presented several practical tasks for the community to work towards. The
most general of the tasks, Structure to Energy and Forces (S2EF) evaluates a
model’s ability to serve as a surrogate to DFT - predicting a configuration’s
energy and per-atom forces. Initial Structure to Relaxed Energy (IS2RE) asks
to predict the relaxed state energy, given only the initial structure. Initial
Structure to Relaxed Structure (IS2RS) explores how well the relaxed structure
can be predicted given only the initial configuration. In the scope of OC20,
all energies were referenced to represent adsorption energy. For more details,
we refer readers to the original manuscript 1.

In this perspective we shed light on the challenges of training Graph Neural
Networks (GNNs) on large-scale datasets spanning material and composition
space, illustrated in Figure 1. We begin with a quick overview on the current
state of the community’s progress and share some takeaways from what we have
observed. We then discuss some telling trends on the performance of models
across different adsorbates and material types. We discuss how different
approaches and modeling decisions impact the prediction tasks and highlight
the challenges associated with each. Further, we explain what the accuracies
in various proposed metrics mean and some of the challenges in analyzing them.
Finally, we share our outlook on the direction the community is headed and
what still remains to achieve a large scale, generalizable potential for
catalyst discovery.

##  2 Community progress in developing ML models for catalysis

Figure 2: Community progress on the OC20 dataset since release. Left: IS2RE
performance for both direct and relaxation based approaches. The current error
target of 0.10 eV would make these models more practically useful for
researchers’ applications. Right: S2EF performance as evaluated by mean
absolute error of the forces. IS2RE and S2EF MAEs for their median baselines
are 1.756 eV and 0.084 eV/Å , respectively.

Molecular modeling has progressed at an incredible rate over the past few
decades. Simple linear models, neural networks, and kernel methods were
originally developed relying on hand-crafted atomic representations, or
descriptors 28, 29, 30, 31, 32 as inputs to the models. Descriptors capture
invariant geometric information in the form of bonds and angles of the local
environment of an atom. While effective, the parameterization of such
descriptors has been a challenging and non-trivial task. The past few years
has seen a shift towards deep learning approaches. Rather than relying on hand
crafted representations, models are being developed to learn similar or more
expressive representations, specifically by exploiting the graphical nature of
molecules using Graph Neural Networks (GNNs) 33, 34, 35, 36, 37, 38. Such
models only take in 3D atomic coordinates and atomic numbers. A graph is then
generated, where atoms are treated as nodes, and the distance between them as
edges. Once a graph has been constructed, GNNs will undergo several rounds of
message passing in which node representations are updated based off messages
sent between neighboring nodes. While models may differ in their exact
architecture, the update and message functions often include a series of
multi-layer perceptrons and nonlinearities. Unlike traditional descriptor
based models, GNNs end up learning node representations as part of the
training process. Learned representations proceed through a final output block
where a final prediction is made. In recent years, GNNs have come to surpass
traditional descriptor based models 33, 34, 35, 36, 37, 38. While typically
data hungry, recent models like NequIP 36 are demonstrating great performance
with as little as 100 samples. GNNs continue to gain traction as models
continue to demonstrate state of the art performance on molecular datasets.

Since the release of OC20, the community has been rapidly developing new
approaches to improve existing baselines. Models being developed range from
traditional descriptor-style models 39 to complex and large GNN architectures
35, 40, 41, 42, 43. Godwin, et al. present a simple, but effective GNN
regularization technique to improve graph-level predictions, namely IS2RE.
Liu, et al. use a similar technique in addition to a graph-based transformer
to win $1^{st}$ place in the NeurIPS 2021 Open Catalyst Challenge 44 for
direct IS2RE predictions. Klicpera, et al.35, 45 and Shuaibi, et al.40 explore
various higher order representations (i.e., triplets and quadruplets) and
leverage training on the entire OC20 to achieve impressive performance on the
S2EF task, with GemNet-OC45 holding the current state of the art across all
tasks. Sriram, et al.41 introduces Graph Parallelism, allowing them to scale
GemNet to nearly a billion parameters across multiple GPUs. The scale and
diversity of OC20 has additionally enabled transfer learning approaches to
smaller datasets. Kolluru, et al. 46 propose a transfer learning technique to
use OC20 pretrained models to improve performance on smaller, out-of-
distribution datasets. Similar work has also been demonstrated for other big
material datasets 47.

As the community continues to improve performance (Figure 2), it’s important
to understand some of the challenges, trends, and pitfalls in developing a
generalizable potential.

##  3 Where are molecular GNNs still erroneous?

Figure 3: Analysis of GemNet-dT errors on the OC20 validation sets. (a) The
categorization of OC20 elements into intermetallics, nonmetals, metalloids and
halides for analysis. (b) Model performance across the different distributions
and material types. (c) Errors averaged across all validation splits for
specific adsorbate containing systems. (d) Errors averaged across all
validation splits for adsorabtes containing certain elements.

Most of the independent work done in developing ML potentials has been
confined to datasets built for certain applications. For example, ML
potentials for the applications of CO2RR are usually just trained with CO and
H adsorbates 48, 27, 49, 50. While this approach might interpolate well across
materials, extrapolation to different adsorbates or more complicated materials
will likely suffer in performance. A universal ML potential, if possible,
would first require a large, diverse dataset that spans material and chemical
space. OC20 dataset was created to build ML potentials that cover a large and
diverse space of heterogeneous catalysts.

Errors across material types: With over 300k unique surfaces, OC20 spans a
vast range of material compositions. When training large GNNs on the entire
OC20 dataset, we observe that the accuracies are not uniform across element
and adsorbate types. To analyze this, we divide the validation set into four
different material types: intermetallics, metalloids, non-metals and halides,
Figure 3(a). The distribution of data across these classes of materials is not
the same, we have significantly more intermetallics and relatively fewer
halides. We observe that the performance on non-metals is significantly worse,
although both nonmetals and metalloids contribute to similar percentage of
training data (Figure 3(b)). On the other hand, models tend to do much better
across the board for intermetallics. Inaccuracies coming from non-metals
disproportionately contribute to the overall errors, leading to worse
performance for both force and energy predictions.

Errors across adsorbates: Large adsorbates are inherently more complicated as
the degrees of freedom increases with the number of atoms. However, we observe
no correlation with our model’s performance and the size of the adsorbate.
Model accuracies are poor for bidentate adsorbates like *CH*COH, *N*NO,
*CH2*O, shown in Figure 3(c). Figure 3(d) also shows that adsorbates with N
and O are generally more erroneous.

Model | Energy MAE (eV) $\downarrow$ | Force MAE (eV/Å ) $\downarrow$
---|---|---
ID | OOD Ads. | OOD Cat. | OOD Both | ID | OOD Ads. | OOD Cat. | OOD Both
Median | 2.04 | 2.42 | 1.99 | 2.58 | 0.081 | 0.080 | 0.079 | 0.098
| Gradient forces
SpinConv40 | - | - | - | - | 0.031 | 0.035 | 0.032 | 0.042
GemNet-dT35 | 0.36 | 0.39 | 0.48 | 0.58 | 0.030 | 0.034 | 0.033 | 0.042
| Direct forces
SpinConv40 | 0.26 | 0.29 | 0.38 | 0.47 | 0.027 | 0.030 | 0.029 | 0.037
GemNet-dT35 | 0.23 | 0.25 | 0.35 | 0.41 | 0.021 | 0.024 | 0.025 | 0.032

Table 1: Results on the OC20 S2EF task via gradient-derived or direct force
predictions. All models were trained on the OC20 S2EF All dataset. Results
reported for the validation set. Energy metrics are unavailable for the
gradient based SpinConv model due to being optimized only on forces.

##  4 Modeling trade-offs

###  4.1 Energy-conserving forces

|  |  | Energy MAE [eV] $\downarrow$ |  Energy within Threshold (EwT) $\uparrow$
---|---|---|---|---
Model | Approach | Dataset Size | ID | OOD Ads | OOD Cat | OOD Both | ID | OOD Ads | OOD Cat | OOD Both
Median baseline | - | - | $1.75$ | $1.88$ | $1.71$ | $1.66$ | $0.71\%$ | $0.72\%$ | $0.89\%$ | $0.74\%$
DimeNet${+}{+}$ 34 | Direct | 460,328 | $0.56$ | $0.73$ | $0.58$ | $0.66$ | $4.25\%$ | $2.07\%$ | $4.10\%$ | $2.41\%$
SpinConv40 | Direct | 460,328 | 0.56 | 0.72 | 0.57 | 0.67 | 4.08% | 2.26% | 3.82% | 2.33%
NoisyNodes43 | Direct | 460,328 | 0.42 | 0.57 | 0.44 | 0.47 | 9.12% | 3.49% | 8.01% | 4.64%
Graphormer42 | Direct | 460,328 | 0.40 | 0.57 | 0.42 | 0.50 | 8.97% | 3.45% | 8.18% | 3.79%
DimeNet${+}{+}$ – LF + LE34, 1, 51 | Relaxation | 2,000,000 | $0.53$ | $0.57$ | $0.56$ | $0.52$ | $6.79\%$ | $4.71\%$ | $6.49\%$ | $4.54\%$
SpinConv40, 51 | Relaxation | 2,000,000 | $0.46$ | $0.51$ | $0.47$ | $0.44$ | $7.38\%$ | $4.82\%$ | $7.05\%$ | $5.31\%$
GemNet-dT35 | Relaxation | 2,000,000 | $0.44$ | $0.44$ | $0.45$ | $0.42$ | $9.37\%$ | $6.59\%$ | $8.42\%$ | $6.40\%$
GemNet-OC45 | Relaxation | 2,000,000 | 0.41 | 0.42 | 0.42 | 0.39 | 11.02% | 8.68% | 10.10% | 7.82%
DimeNet${+}{+}$ – LF + LE 34, 1 | Relaxation | 133,934,018 | $0.50$ | $0.54$ | $0.58$ | $0.61$ | $6.57\%$ | $4.34\%$ | $5.09\%$ | $3.93\%$
SpinConv40 | Relaxation | 133,934,018 | $0.42$ | $0.44$ | $0.46$ | $0.42$ | $9.37\%$ | $7.47\%$ | $8.16\%$ | $6.56\%$
GemNet-dT35 | Relaxation | 133,934,018 | 0.39 | 0.39 | 0.43 | 0.38 | $12.37\%$ | $9.11\%$ | $10.09\%$ | $7.87\%$
GemNet-OC45 | Relaxation | 133,934,018 | 0.35 | 0.35 | 0.38 | 0.34 | 16.06% | 12.62% | 13.17% | 11.06%

Table 2: Results on the OC20 IS2RE task using one of two approaches. Direct
Directly predicting the relaxed state energy and Relaxation Training a model
for energy and force predictions, followed by an iterative ML-based geometry
optimization to arrive at a relaxed structure and energy. Relaxation results
on the 2M subset suggest that competitive results are still possible with a
limited compute budget. Results reported for the test set.

Force predictions play an important role in the applications of ML models for
catalyst discovery. While some tasks may only be interested in property
predictions like adsorption or formation energy 48, 52, 53, forces are
necessary to study dynamics such as structural relaxations, molecular
dynamics, and transition state calculations 1, 36, 33, 54.

Physically, energy-conserving forces are derived as the gradient of energy
with respect to atomic positions:

$\boldsymbol{F_{i}}=-\frac{dE}{dx_{i}}$ (1)

Energy-conservation is critical in studying molecular dynamics accurately. ML
models estimating energy-conserving forces must ensure the architecture is
continuous and differentiable, often satisfied by appropriate non-linear
activation functions 33, 34, 35. Geometrically, forces derived in an energy-
conserving manner ensures forces are rotationally equivariant, a necessary
physical relation of molecular systems 55. Unfortunately, a gradient
calculation increases model overhead in both memory usage and computational
time by a factor of 2-4 40, 56. For datasets like MD17, calculating forces as
a gradient is known to help in model accuracies as that is an important
physical prior to the model 36, 35, 40. Models trained on MD17 are often used
to run molecular dynamics, further necessitating the need for energy-
conservation 36. However, for the OC20 dataset, particularly in the task of
geometric optimization, we observe that the gradient approach for calculating
forces to perform worse than direct prediction of forces for GemNet-dT 35 and
Spinconv 40. Dimenet ++ 34 and ForceNet 56 were built for gradient and direct
approach respectively. The gradient approach could also make the training
unstable in certain cases, which has been observed for ForceNet56 and GemNet-
OC45. Table 1 compares performance on the S2EF task for two recent top
performing models, GemNet-dT 35 and SpinConv 40. Not only are the force
accuracies worse for the gradient approach, but the corresponding relaxed
structure and relaxed energy metrics calculated via optimization are also
significantly worse 40.

While energy-conservation plays a critical role in many molecular
applications, we observe that direct force computations brings efficiency and
performance advantages 56, 40. Models trained for direct force predictions are
limited to applications where strict enforcement of energy-conservation can
reasonably be ignored, i.e. OC20’s structural relaxations. Here, atomic
positions are updated solely from force estimates 1, 57. If necessary, DFT, or
a subsequent ML model, can then be used to make reliable energy predictions on
the ML optimized structure. Similarly, transition states or saddle points can
be derived in a similar manner with direct-force models. We want to emphasize
that although unorthodox, direct-force models still prove to be useful in
certain catalyst applications, i.e. OC20-like tasks.

###  4.2 Prediction of relaxed energy and structure

Adsorption energy is one of many properties that helps inform catalyst
performance58. Computationally, this is computed via a series of QM structural
relaxations. The relaxed energy is then referenced to represent the adsorption
energy, see Chanussot et al.1, García-Muelas et al.59 for more details. From a
data-driven approach, we can predict the relaxed energy or the relaxed
structure of an atomic system usually via two methods. First, we can build a
surrogate to DFT, approximating system energy and per-atom structures, and
running ML optimizations to find the minimum energy, a common approach within
the field. Alternatively, given a large enough dataset of relaxed structures
and energies, we can try to predict these properties directly using a ML model
instead of optimizing via an iterative loop. The advantage of the direct
method over the relaxation approach is that it requires only a single call to
the ML model, whereas the relaxation approach could require on average 200-300
calls for a single relaxation. Direct approaches are particularly advantageous
when we talk about the computational cost of approaching large scale inference
on the order of hundreds of millions to billions of systems.

The community has made tremendous progress in predicting adsorption energy as
evaluated by the OC20 IS2RE task (Figure 2). Direct approaches, despite using
300x less data, are approaching the competitive relaxation based approaches of
GemNet-XL and GemNet-OC. Inference time aside, models trained on the full 133M
dataset for the relaxation based approaches are typically compute intensive,
using between 128-512 GPUs 40, 56, 35, 1. While this is certainly a small
price to pay if the models developed accelerate the discovery process, it does
make it difficult for the community to engage in and aid in development. This
has been particularly observed in the NeurIPS 2021 Open Catalyst Challenge 44,
where of the 30 submissions, 0 were made via the relaxation approach. Here, we
show that models trained on a 2M subset of the full dataset are still able to
provide competitive results and even, averaged across all splits, out perform
direct approaches. Given the trends in the 2M dataset correlate well with the
full 133M dataset 45, this should help incentivize the community to explore
other approaches even with resource limitations. Although the relaxation
approach is computationally expensive for both training and inference, we have
observed that the models trained through this approach tend to generalize
better on out-of-distribution (OOD) data, Table 2.

Direct relaxed energy predictions are an easier ML problem than direct
structure predictions. For a system of size N, energy predictions require a
single scalar output, while structure predictions require 3N components. We
find that for relaxed energy prediction tasks, metrics are closer for direct
and relaxation approach whereas for structure prediction task the metrics are
worse. The OC20 paper provides a baseline for relaxed structure prediction
only via the relaxation approach 1. In Table 3 we provide baselines for direct
relaxed structure prediction. A considerable gap exists between the direct and
relaxation based approaches (especially in the DFT based metrics).

###  4.3 Metrics for finding local minima

Relaxed structure prediction is less straightforward than some of the other
common energy and force prediction tasks. Given a dataset like OC20 where
relaxed structures are not necessarily global minima, a model trained on such
a dataset could either (1) predict and arrive at the same local minima, (2)
arrive at a different, but still suitable minima, or (3) fail to arrive at any
sort of minima.

To account for this, two main metrics have been presented in the OC20 paper.
Average Distance within Threshold (ADwT) is a distance based metric and
measures how close the predicted structure compares to the actual structure.
This is similar to the Global Distance Test (GDT) metric in the protein
folding task 60, 61. ADwT takes an average across different thresholds varying
from 0.1 to 0.5Å to ensure a signal is captured. For the OC20 dataset, we
evaluate this metric for the input initial structures for an accuracy of
21.18% on the in-domain validation set 1. Models, at the bare minimum, should
perform better than this baseline. To ensure invariance to arbitrary
coordinate reference frames, we predict the difference between initial and
final positions instead of the final position Cartesian coordinates.
Predicting the delta difference helps simplify this task and results in
improved ADwT accuracies.

Table 3: Baseline metrics for IS2RS direct task in comparison with the
relaxation approach. Metrics are reported on a 2k subset of the validation
set, across all splits. DwT is evaluated at a threshold of 0.04 Å. For compute
reasons, DFT-based metrics were evaluated on a 200 system subset of the 2k, 50
systems from each split.

| Model | DwT (at 0.04 Å ) $\uparrow$ | ADwT $\uparrow$ | FbT* $\uparrow$ | AFbT* $\uparrow$
---|---|---|---|---|---
Direct | ForceNet56 | 0.70 | 45.69% | 0.00% | 0.00%
SpinConv40 | 1.05 | 47.76% | 0.00% | 0.00%
GemNet-dT35 | 1.75 | 45.87% | 0.00% | 0.08%
Relaxation | ForceNet56 | 1.45 | 46.51% | 0.00% | 7.64%
SpinConv40 | 8.20 | 55.81% | 0.00% | 12.55%
GemNet-dT35 | 13.95 | 60.88% | 0.00% | 20.35%

A model that predicts a relaxed structure that is not identical to its DFT
reference may still be considered successful for two reasons. (1) the model
could have predicted a symmetrically identical site on the surface and (2) the
model predicted a different, but still suitable local minima. The former is
more a concern surrounding the distance-based metric, as ADwT, although
accounts for periodic-boundary conditions, does not consider symmetrically
identical sites. While it is rather unlikely an adsorbate initialized over a
particular site will hop several sites over to a symmetrically identical site,
it is worth raising awareness to the possibility. On the other hand, a model
that arrives at a different relaxed structure entirely will fail according to
ADwT. However, to verify whether the model has predicted a different suitable
minima, we can evaluate the DFT forces corresponding to the ML predicted
structures. This metric is called Average Force below Threshold (AFbT) and it
measures the percent of structures having their forces close to zero 1. Since
models are expected to predict relaxed structures, DFT forces should be close
to zero. This is a stricter metric as compared to ADwT. However, this is far
more expensive due to the additional DFT calculations. A more practically
useful metric would be number of DFT calculations required to find the relaxed
structure starting from the ML relaxed structure. This would give us an idea
of the percent of DFT calculations that the current ML models can reduce.
Although useful, this is a significantly more expensive metric than AFbT
calculations. While it is not something Open Catalyst Project’s (OCP) tracks
on their public leaderboard, we bring awareness to it as there could be
instances where models do poorly on ADwT and AFbT but resulting structures are
only a few DFT steps away from the relaxed structure.

In Table 3 we compare relaxed structure prediction via a direct and relaxation
approach. We observe that direct methods, although having competitive ADwT
metrics, have AFbT metrics that are significantly worse. This suggests that
direct models do a reasonable job at getting close to the relaxed structure
but are in high-force configurations, failing to capture repulsive physical
interactions 43. We speculate models struggle with this since small
perturbations distances can have large consequences on forces, e.g. moving two
atoms at an equilibrium bond length fractions of an angstrom towards each
other. Relaxed structure prediction via the relaxation approach avoids this
issue by using ML forces to drive a geometric optimizer.

We observe that distance metrics at tighter thresholds correlate better with
force based metrics, however, going below 0.04 Å  does not give sufficient
signal and the accuracies for most systems fall to zero. Moreover, the
Distance within Threshold (DwT) at 0.04 Å  isn’t a good enough signal that can
replace AFbT. For example, DwT (at 0.04Å ) for ForceNet relaxation approach
and GemNet-dT direct approach are similar, however, the AFbT metrics still
differ by 7.56% (as shown in Table 3). We believe that finding non DFT-based
metrics that correlate well with DFT-based metrics is still an open and
important question in the community which would make model evaluation
computationally less expensive.

###  4.4 Additional data

The OC20 paper 1 released two additional data subsets generated with ab-initio
molecular dynamics (‘MD’) and structural perturbations (‘Rattled’). These
provide $38M$ and $17M$ additional S2EF training data points respectively.

Table 4 presents results for GemNet-OC45 models trained on S2EF, Rattled, and
MD data compared against similar analysis from the OC20 paper for DimeNet++
34, 21. First, on the force MAE metric, addition of MD data hurts DimeNet++
while it improves GemNet-OC. We speculate this to be another artifact of
modeling forces as negative gradients of energy (as in DimeNet++) vs. direct
prediction (as in GemNet-OC). Second, consistent with the OC20 paper, adding
MD data to the training set provides a useful signal for IS2RS structure
relaxations as per the AFbT metric. Finally, adding Rattled data helps with
IS2RS metrics, but did not help or marginally hurt the S2EF force MAE. This
could be due to a variety of reasons – random perturbations being too large /
small to be useful, intermediate structures along a trajectory being less
useful compared to closer to the local minimum (as in MD initial structures),
etc. A promising direction here could be active learning approaches to
optimally query additional training data points.

|  | S2EF Val ID | IS2RS Test
---|---|---|---
| Training Data (# samples) | Force MAE $\downarrow$ | ADwT $\uparrow$ | AFbT $\uparrow$
DN++ $\begin{dcases}\\\ \\\ \end{dcases}$ |  $20M$ ($20M$) | $0.0511$ | $34.37\%$ | $2.67\%$
$20M$ $+$ MD ($58M$) | $0.0594$ | $47.69\%$ | $17.09\%$
$20M$ $+$ Rattled ($37M$) | $0.0614$ | $43.94\%$ | $12.51\%$
GN-OC $\begin{dcases}\\\ \\\ \end{dcases}$ | All ($133M$) | $0.0179$ | $60.33\%$ | $35.27\%$
All $+$ MD ($172M$) | $0.0173$ | $60.77\%$ | $38.05\%$
All $+$ MD $+$ Rattled ($189M$) | $0.0174$ | - | -

Table 4: Results with DimeNet++ (DN++) and GemNet-OC (GN-OC) trained on MD and
Rattled. S2EF results reported for the validation in-distribution set. IS2RS
results reported on the test set.

##  5 Summary and Outlook

The development of generalizable or universal ML models has only recently been
seriously considered with the emergence of large-scale datasets like OC20 1.
Since its release, the catalysis and ML communities have both made tremendous
progressive in developing models for catalyst applications. As the community
continues to grow and as more datasets emerge that span material and
composition space, the prospect of large-scale generalizable models is within
reason. Progress thus far has demonstrated several challenges in accomplishing
this feat: classes of materials and adsorbates with inconsistent errors,
energy-conserving forces, relaxed vs direct approaches, DFT metrics, and data
augmentation strategies. In this perspective, we discussed these challenges in
detail and provided some insights as to how and why they are important.
Although these challenges were discussed in the context of OC20, we anticipate
similar challenges to future datasets of its kind.

Datasets like OC20 has offered new ways to how we think about building large,
generalizable, and reliable models. While model performance has been the focal
point of community progress thus far, we provide an outlook of other important
challenges that we hope the community to engage in.

Training strategies. OC20 was released with predefined training, validation,
and test sets. Its splits were curated in a manner to tackle the problem of
building a single generalizable model for catalysis. However, it could be the
case that multiple models for different subsets of the data, e.g. adsorbates,
compositions, materials, do better. In the case of nonmetals, for instance, we
have shown that this actually hurts performance - a possible consequence of
the reduced dataset size.

Uncertainty and active learning. While model performance is a necessary step
for the discovery process, it is not always sufficient. A practical ML-aided
catalyst discovery pipeline will ultimately turn to experiments to validate
whether the ML predicted “great” catalyst is at all effective. Having
confidence in these predictions is particularly important to avoid wasted
expensive experiments. Uncertainty quantification has been a particularly
popular topic within the catalysis community, often focused on the small data
regime and active learning 62, 63, 64, 65, 66, 67, 68. The effectiveness of
traditional uncertainty estimation techniques on large datasets like OC20 is a
necessary and important step for the future of this work. Similarly, how to
best leverage active learning for either dataset generation and/or
augmentation 69 or online active learning 65, 64 at the scale of OC20 will be
an exciting future direction.

Model efficiency. In addition to model performance and reliability, model
efficiency will continue to be critical for all applications. For training,
faster, more data efficient models can help attract the community to tackle
some of the bigger challenges like a surrogate to DFT, i.e. OC20’s S2EF task.
Progress so far has shown that the best models are also the largest models.
From an inference perspective, this poses obvious challenges of slower speeds
and ultimately reduced screening throughput. While models still remain orders
of magnitude faster than DFT, when considering the possibility of screening
billions of systems, computational costs add up. Recent models encoding
equivariant representations 36, 70 have shown incredible scaling and
efficiency gains that could be promising to explore. Moving forward, efficient
architectures and model distillation71 will be an important contribution to
reduce the computational cost of large-scale inference, even if it means
sacrificing some accuracy.

Data augmentation. The scale of OC20 makes data augmentation a non-trivial
challenge. With 130M+ training data points, randomly adding 10-100k data
points will likely have negligible impact on the models. We observed that
models using the additional MD data are able to perform the best, while the
rattled data has little impact. Identifying strategies to combine and train
large molecular and material datasets like ANI-120 and OQMD72 with OC20 could
help improve models even further. The biggest challenge surrounding this comes
from combining datasets of varying levels of DFT theory.

Energy-conserving forces. In the context of OC20, we have observed that the
best performing models make a direct force-prediction. While this may be
suitable for some applications, the more physically motivated gradient
approach to force prediction is desired for other applications like MD. The
same direct models applied to MD17 observe the opposite effect, better
performance via the gradient method 45. It remains an open question why this
is the case, and we encourage others to investigate this observation.

Physics-based modeling. The majority of models submitted to OC20 have followed
a purely data-driven approach, only taking in atomic numbers and positions as
inputs. Exploring ways to leverage OC20 charge density or Bader charge data
111To be made publically available at https://github.com/Open-Catalyst-
Project/ocp/blob/main/DATASET.md could prove useful, particularly in the low
data regime. Additionally, models like UNiTE 73 or OrbNet74 that leverage
tight binding DFT75 for featurization could be interesting to explore for
catalyst applications.

## Acronyms

ADwT

    Average Distance within Threshold
AFbT

    Average Force below Threshold
DFT

    Density Functional Theory
DwT

    Distance within Threshold
EwT

    Energy within Threshold
GNNs

    Graph Neural Networks
IS2RE

    Initial Structure to Relaxed Energy
IS2RS

    Initial Structure to Relaxed Structure
OC20

    Open Catalyst 2020 Dataset
OCP

    Open Catalyst Project’s
S2EF

    Structure to Energy and Forces

## Glossary

## References

  * Chanussot et al. 2021 Chanussot, L.; Das, A.; Goyal, S.; Lavril, T.; Shuaibi, M.; Riviere, M.; Tran, K.; Heras-Domingo, J.; Ho, C.; Hu, W., et al. Open Catalyst 2020 (OC20) Dataset and Community Challenges. _ACS Catalysis_ 2021, _11_ , 6059–6072
  * Ren et al. 2018 Ren, F.; Ward, L.; Williams, T.; Laws, K. J.; Wolverton, C.; Hattrick-Simpers, J.; Mehta, A. Accelerated discovery of metallic glasses through iteration of machine learning and high-throughput experiments. _Science advances_ 2018, _4_ , eaaq1566
  * Jose et al. 2006 Jose, R.; Zhanpeisov, N. U.; Fukumura, H.; Baba, Y.; Ishikawa, M. Structure- property correlation of CdSe clusters using experimental results and first-principles DFT calculations. _Journal of the American Chemical Society_ 2006, _128_ , 629–636
  * Rahali et al. 2021 Rahali, S.; Ben Aissa, M. A.; Khezami, L.; Elamin, N.; Seydou, M.; Modwi, A. Adsorption behavior of Congo red onto barium-doped ZnO nanoparticles: correlation between experimental results and DFT calculations. _Langmuir_ 2021, _37_ , 7285–7294
  * Xu et al. 2019 Xu, L.; Meng, X.; Li, M.; Li, W.; Sui, Z.; Wang, J.; Yang, J. Dissolution and degradation of nuclear grade cationic exchange resin by Fenton oxidation combining experimental results and DFT calculations. _Chemical Engineering Journal_ 2019, _361_ , 1511–1523
  * Ktari et al. 2015 Ktari, N.; Fourati, N.; Zerrouki, C.; Ruan, M.; Seydou, M.; Barbaut, F.; Nal, F.; Yaakoubi, N.; Chehimi, M.; Kalfat, R. Design of a polypyrrole MIP-SAW sensor for selective detection of flumequine in aqueous media. Correlation between experimental results and DFT calculations. _RSC advances_ 2015, _5_ , 88666–88674
  * Rode et al. 2018 Rode, N. D.; Abdalghani, I.; Arcadi, A.; Aschi, M.; Chiarini, M.; Marinelli, F. Synthesis of 2-Acylindoles via Ag-and Cu-Catalyzed anti-Michael Hydroamination of $\beta$-(2-Aminophenyl)-$\alpha$, $\beta$-ynones: Experimental Results and DFT Calculations. _The Journal of organic chemistry_ 2018, _83_ , 6354–6362
  * Pause et al. 2001 Pause, L.; Robert, M.; Heinicke, J.; Kühl, O. Radical anions of carbenes and carbene homologues. DFT study and preliminary experimental results. _Journal of the Chemical Society, Perkin Transactions 2_ 2001, 1383–1388
  * Agrawal and Choudhary 2016 Agrawal, A.; Choudhary, A. Perspective: Materials informatics and big data: Realization of the “fourth paradigm” of science in materials science. _Apl Materials_ 2016, _4_ , 053208
  * Guan et al. 2022 Guan, Y.; Chaffart, D.; Liu, G.; Tan, Z.; Zhang, D.; Wang, Y.; Li, J.; Ricardez-Sandoval, L. Machine learning in solid heterogeneous catalysis: Recent developments, challenges and perspectives. _Chemical Engineering Science_ 2022, _248_ , 117224
  * Rosen et al. 2022 Rosen, A. S.; Notestein, J. M.; Snurr, R. Q. Realizing the data-driven, computational discovery of metal-organic framework catalysts. _Current Opinion in Chemical Engineering_ 2022, _35_ , 100760
  * Himanen et al. 2019 Himanen, L.; Geurts, A.; Foster, A. S.; Rinke, P. Data-driven materials science: status, challenges, and perspectives. _Advanced Science_ 2019, _6_ , 1900808
  * Xu et al. 2021 Xu, J.; Cao, X.-M.; Hu, P. Perspective on computational reaction prediction using machine learning methods in heterogeneous catalysis. _Physical Chemistry Chemical Physics_ 2021, _23_ , 11155–11179
  * Tran and Ulissi 2018 Tran, K.; Ulissi, Z. W. Active learning across intermetallics to guide discovery of electrocatalysts for CO2 reduction and H2 evolution. _Nature Catalysis_ 2018, _1_ , 696–703
  * Back et al. 2019 Back, S.; Tran, K.; Ulissi, Z. W. Toward a design of active oxygen evolution catalysts: insights from automated density functional theory calculations and machine learning. _Acs Catalysis_ 2019, _9_ , 7651–7659
  * Ying et al. 2021 Ying, Y.; Fan, K.; Luo, X.; Qiao, J.; Huang, H. Unravelling the origin of bifunctional OER/ORR activity for single-atom catalysts supported on C2N by DFT and machine learning. _Journal of Materials Chemistry A_ 2021, _9_ , 16860–16867
  * Ge et al. 2020 Ge, L.; Yuan, H.; Min, Y.; Li, L.; Chen, S.; Xu, L.; Goddard III, W. A. Predicted optimal bifunctional electrocatalysts for the hydrogen evolution reaction and the oxygen evolution reaction using chalcogenide heterostructures based on machine learning analysis of in silico quantum mechanics based high throughput screening. _The Journal of Physical Chemistry Letters_ 2020, _11_ , 869–876
  * Toniato et al. 2022 Toniato, A.; Vaucher, A. C.; Laino, T. Grand challenges on accelerating discovery in catalysis. _Catalysis Today_ 2022, _387_ , 140–142
  * Chmiela et al. 2017 Chmiela, S.; Tkatchenko, A.; Sauceda, H. E.; Poltavsky, I.; Schütt, K. T.; Müller, K.-R. Machine learning of accurate energy-conserving molecular force fields. _Science advances_ 2017, _3_ , e1603015
  * Smith et al. 2017 Smith, J. S.; Isayev, O.; Roitberg, A. E. ANI-1: an extensible neural network potential with DFT accuracy at force field computational cost. _Chemical science_ 2017, _8_ , 3192–3203
  * Klicpera et al. 2020 Klicpera, J.; Giri, S.; Margraf, J. T.; Günnemann, S. Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Molecules. _arXiv preprint arXiv:2011.14115_ 2020,
  * Ramakrishnan et al. 2014 Ramakrishnan, R.; Dral, P. O.; Rupp, M.; Von Lilienfeld, O. A. Quantum chemistry structures and properties of 134 kilo molecules. _Scientific data_ 2014, _1_ , 1–7
  * Chen et al. 2019 Chen, G.; Chen, P.; Hsieh, C.-Y.; Lee, C.-K.; Liao, B.; Liao, R.; Liu, W.; Qiu, J.; Sun, Q.; Tang, J., et al. Alchemy: A quantum chemistry dataset for benchmarking ai models. _arXiv preprint arXiv:1906.09427_ 2019,
  * Andersen et al. 2019 Andersen, M.; Levchenko, S. V.; Scheffler, M.; Reuter, K. Beyond scaling relations for the description of catalytic materials. _ACS Catalysis_ 2019, _9_ , 2752–2759
  * Abild-Pedersen et al. 2007 Abild-Pedersen, F.; Greeley, J.; Studt, F.; Rossmeisl, J.; Munter, T. R.; Moses, P. G.; Skulason, E.; Bligaard, T.; Nørskov, J. K. Scaling properties of adsorption energies for hydrogen-containing molecules on transition-metal surfaces. _Physical review letters_ 2007, _99_ , 016105
  * Ma and Xin 2017 Ma, X.; Xin, H. Orbitalwise coordination number for predicting adsorption properties of metal nanocatalysts. _Physical review letters_ 2017, _118_ , 036101
  * Noh et al. 2018 Noh, J.; Back, S.; Kim, J.; Jung, Y. Active learning with non-ab initio input features toward efficient CO2 reduction catalysts. _Chemical science_ 2018, _9_ , 5152–5159
  * Behler and Parrinello 2007 Behler, J.; Parrinello, M. Generalized neural-network representation of high-dimensional potential-energy surfaces. _Physical review letters_ 2007, _98_ , 146401
  * Lorenz et al. 2004 Lorenz, S.; Groß, A.; Scheffler, M. Representing high-dimensional potential-energy surfaces for reactions at surfaces by neural networks. _Chemical Physics Letters_ 2004, _395_ , 210–215
  * Chen et al. 2013 Chen, J.; Xu, X.; Xu, X.; Zhang, D. H. A global potential energy surface for the H2 + OH -¿ H2O + H reaction using neural networks. _The Journal of Chemical Physics_ 2013, _138_ , 154301
  * Bartók and Csányi 2015 Bartók, A. P.; Csányi, G. G aussian approximation potentials: A brief tutorial introduction. _International Journal of Quantum Chemistry_ 2015, _115_ , 1051–1057
  * Bartók et al. 2010 Bartók, A. P.; Payne, M. C.; Kondor, R.; Csányi, G. Gaussian approximation potentials: The accuracy of quantum mechanics, without the electrons. _Physical review letters_ 2010, _104_ , 136403
  * Schütt et al. 2017 Schütt, K.; Kindermans, P.-J.; Felix, H. E. S.; Chmiela, S.; Tkatchenko, A.; Müller, K.-R. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. NeurIPS. 2017; pp 991–1001
  * Klicpera et al. 2020 Klicpera, J.; Groß, J.; Günnemann, S. Directional message passing for molecular graphs. ICLR. 2020
  * Klicpera et al. 2021 Klicpera, J.; Becker, F.; Günnemann, S. GemNet: Universal Directional Graph Neural Networks for Molecules. _arXiv preprint arXiv:2106.08903_ 2021,
  * Batzner et al. 2021 Batzner, S.; Musaelian, A.; Sun, L.; Geiger, M.; Mailoa, J. P.; Kornbluth, M.; Molinari, N.; Smidt, T. E.; Kozinsky, B. Se (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. _arXiv preprint arXiv:2101.03164_ 2021,
  * Schütt et al. 2021 Schütt, K. T.; Unke, O. T.; Gastegger, M. Equivariant message passing for the prediction of tensorial properties and molecular spectra. _arXiv preprint arXiv:2102.03150_ 2021,
  * Liu et al. 2021 Liu, Y.; Wang, L.; Liu, M.; Zhang, X.; Oztekin, B.; Ji, S. Spherical message passing for 3d graph networks. _arXiv preprint arXiv:2102.05013_ 2021,
  * Lei and Medford 2021 Lei, X.; Medford, A. J. A universal framework for featurization of atomistic systems. _arXiv preprint arXiv:2102.02390_ 2021,
  * Shuaibi et al. 2021 Shuaibi, M.; Kolluru, A.; Das, A.; Grover, A.; Sriram, A.; Ulissi, Z.; Zitnick, C. L. Rotation Invariant Graph Neural Networks using Spin Convolutions. _arXiv preprint arXiv:2106.09575_ 2021,
  * Sriram et al. 2021 Sriram, A.; Das, A.; Wood, B. M.; Zitnick, C. L. Towards Training Billion Parameter Graph Neural Networks for Atomic Simulations. International Conference on Learning Representations. 2021
  * Ying et al. 2021 Ying, C.; Cai, T.; Luo, S.; Zheng, S.; Ke, G.; He, D.; Shen, Y.; Liu, T.-Y. Do Transformers Really Perform Badly for Graph Representation? _Advances in Neural Information Processing Systems_ 2021, _34_
  * Godwin et al. 2021 Godwin, J.; Schaarschmidt, M.; Gaunt, A. L.; Sanchez-Gonzalez, A.; Rubanova, Y.; Veličković, P.; Kirkpatrick, J.; Battaglia, P. Simple GNN Regularisation for 3D Molecular Property Prediction and Beyond. International Conference on Learning Representations. 2021
  * ocp 2021 Open Catalyst Project Challenge. https://opencatalystproject.org/challenge.html, 2021
  * Gasteiger et al. 2022 Gasteiger, J.; Shuaibi, M.; Sriram, A.; Günnemann, S.; Ulissi, Z.; Zitnick, C. L.; Das, A. How Do Graph Networks Generalize to Large and Diverse Molecular Systems? 2022; https://arxiv.org/abs/2204.02782
  * Kolluru et al. 2022 Kolluru, A.; Shoghi, N.; Shuaibi, M.; Goyal, S.; Das, A.; Zitnick, L.; Ulissi, Z. W. Transfer Learning using Attentions across Atomic Systems with Graph Neural Networks (TAAG). _The Journal of Chemical Physics_ 2022,
  * Chen and Ong 2021 Chen, C.; Ong, S. P. AtomSets as a hierarchical transfer learning framework for small and large materials datasets. _npj Computational Materials_ 2021, _7_ , 1–9
  * Back et al. 2019 Back, S.; Yoon, J.; Tian, N.; Zhong, W.; Tran, K.; Ulissi, Z. W. Convolutional neural network of atomic surface structures to predict binding energies for high-throughput screening of catalysts. _The journal of physical chemistry letters_ 2019, _10_ , 4401–4408
  * Yoon and Ulissi 2020 Yoon, J.; Ulissi, Z. W. Differentiable optimization for the prediction of ground state structures (DOGSS). _Physical Review Letters_ 2020, _125_ , 173001
  * Chen et al. 2020 Chen, A.; Zhang, X.; Chen, L.; Yao, S.; Zhou, Z. A machine learning model on simple features for CO2 reduction electrocatalysts. _The Journal of Physical Chemistry C_ 2020, _124_ , 22471–22478
  * dis 2021 IS2RE Leaderboard Concerns. https://discuss.opencatalystproject.org/t/is2re-leaderboard-concerns/66, 2021
  * Li et al. 2017 Li, Z.; Wang, S.; Chin, W. S.; Achenie, L. E.; Xin, H. High-throughput screening of bimetallic catalysts enabled by machine learning. _Journal of Materials Chemistry A_ 2017, _5_ , 24131–24138
  * Mamun et al. 2019 Mamun, O.; Winther, K. T.; Boes, J. R.; Bligaard, T. High-throughput calculations of catalytic properties of bimetallic alloy surfaces. _Scientific data_ 2019, _6_ , 1–9
  * Peterson 2016 Peterson, A. A. Acceleration of saddle-point searches with machine learning. _The Journal of chemical physics_ 2016, _145_ , 074106
  * Christensen and von Lilienfeld 2020 Christensen, A. S.; von Lilienfeld, O. A. On the role of gradients for machine learning of molecular energies and forces. _Machine Learning: Science and Technology_ 2020, _1_ , 045018
  * Hu et al. 2021 Hu, W.; Shuaibi, M.; Das, A.; Goyal, S.; Sriram, A.; Leskovec, J.; Parikh, D.; Zitnick, C. L. Forcenet: A graph neural network for large-scale quantum calculations. _arXiv preprint arXiv:2103.01436_ 2021,
  * del Río et al. 2019 del Río, E. G.; Mortensen, J. J.; Jacobsen, K. W. Local Bayesian optimizer for atomic structures. _Physical Review B_ 2019, _100_ , 104103
  * Wang et al. 2018 Wang, Y.; Qiu, W.; Song, E.; Gu, F.; Zheng, Z.; Zhao, X.; Zhao, Y.; Liu, J.; Zhang, W. Adsorption-energy-based activity descriptors for electrocatalysts in energy storage applications. _National Science Review_ 2018, _5_ , 327–341
  * García-Muelas and López 2019 García-Muelas, R.; López, N. Statistical learning goes beyond the d-band model providing the thermochemistry of adsorbates on transition metals. _Nature communications_ 2019, _10_ , 1–7
  * Jumper et al. 2021 Jumper, J.; Evans, R.; Pritzel, A.; Green, T.; Figurnov, M.; Ronneberger, O.; Tunyasuvunakool, K.; Bates, R.; Žídek, A.; Potapenko, A., et al. Highly accurate protein structure prediction with AlphaFold. _Nature_ 2021, _596_ , 583–589
  * Zemla 2003 Zemla, A. LGA: a method for finding 3D similarities in protein structures. _Nucleic acids research_ 2003, _31_ , 3370–3374
  * Tran et al. 2020 Tran, K.; Neiswanger, W.; Yoon, J.; Zhang, Q.; Xing, E.; Ulissi, Z. W. Methods for comparing uncertainty quantifications for material property predictions. _Machine Learning: Science and Technology_ 2020, _1_ , 025006
  * Busk et al. 2021 Busk, J.; Jørgensen, P. B.; Bhowmik, A.; Schmidt, M. N.; Winther, O.; Vegge, T. Calibrated uncertainty for molecular property prediction using ensembles of message passing neural networks. _Machine Learning: Science and Technology_ 2021, _3_ , 015012
  * Shuaibi et al. 2020 Shuaibi, M.; Sivakumar, S.; Chen, R. Q.; Ulissi, Z. W. Enabling robust offline active learning for machine learning potentials using simple physics-based priors. _Machine Learning: Science and Technology_ 2020, _2_ , 025007
  * Vandermause et al. 2020 Vandermause, J.; Torrisi, S. B.; Batzner, S.; Xie, Y.; Sun, L.; Kolpak, A. M.; Kozinsky, B. On-the-fly active learning of interpretable Bayesian force fields for atomistic rare events. _npj Computational Materials_ 2020, _6_ , 1–11
  * Jinnouchi et al. 2019 Jinnouchi, R.; Lahnsteiner, J.; Karsai, F.; Kresse, G.; Bokdam, M. Phase transitions of hybrid perovskites simulated by machine-learning force fields trained on the fly with Bayesian inference. _Physical review letters_ 2019, _122_ , 225701
  * Torres et al. 2019 Torres, J. A. G.; Jennings, P. C.; Hansen, M. H.; Boes, J. R.; Bligaard, T. Low-scaling algorithm for nudged elastic band calculations using a surrogate machine learning model. _Physical review letters_ 2019, _122_ , 156001
  * Scalia et al. 2020 Scalia, G.; Grambow, C. A.; Pernici, B.; Li, Y.-P.; Green, W. H. Evaluating scalable uncertainty estimation methods for deep learning-based molecular property prediction. _Journal of chemical information and modeling_ 2020, _60_ , 2697–2717
  * Smith et al. 2020 Smith, J. S.; Zubatyuk, R.; Nebgen, B.; Lubbers, N.; Barros, K.; Roitberg, A. E.; Isayev, O.; Tretiak, S. The ANI-1ccx and ANI-1x data sets, coupled-cluster and density functional theory properties for molecules. _Scientific data_ 2020, _7_ , 1–10
  * Musaelian et al. 2022 Musaelian, A.; Batzner, S.; Johansson, A.; Sun, L.; Owen, C. J.; Kornbluth, M.; Kozinsky, B. Learning Local Equivariant Representations for Large-Scale Atomistic Dynamics. _arXiv preprint arXiv:2204.05249_ 2022,
  * Frankle and Carbin 2018 Frankle, J.; Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. _arXiv preprint arXiv:1803.03635_ 2018,
  * Kirklin et al. 2015 Kirklin, S.; Saal, J. E.; Meredig, B.; Thompson, A.; Doak, J. W.; Aykol, M.; Rühl, S.; Wolverton, C. The Open Quantum Materials Database (OQMD): assessing the accuracy of DFT formation energies. _npj Computational Materials_ 2015, _1_ , 1–15
  * Qiao et al. 2021 Qiao, Z.; Christensen, A. S.; Welborn, M.; Manby, F. R.; Anandkumar, A.; Miller III, T. F. Unite: Unitary n-body tensor equivariant network with applications to quantum chemistry. _arXiv preprint arXiv:2105.14655_ 2021,
  * Qiao et al. 2020 Qiao, Z.; Welborn, M.; Anandkumar, A.; Manby, F. R.; Miller III, T. F. OrbNet: Deep learning for quantum chemistry using symmetry-adapted atomic-orbital features. _The Journal of Chemical Physics_ 2020, _153_ , 124111
  * Bannwarth et al. 2019 Bannwarth, C.; Ehlert, S.; Grimme, S. GFN2-xTB—An accurate and broadly parametrized self-consistent tight-binding quantum chemical method with multipole electrostatics and density-dependent dispersion contributions. _Journal of chemical theory and computation_ 2019, _15_ , 1652–1671


11institutetext: Technische Universität München, Munich, Germany
11email<EMAIL_ADDRESS>22institutetext: Reutlingen University,
Reutlingen, Germany
22email<EMAIL_ADDRESS>22email:
<EMAIL_ADDRESS>33institutetext: University of
Surrey, Guildford, United Kingdom
33email<EMAIL_ADDRESS>
# A 3D Face Modelling Approach for Pose-Invariant Face Recognition in a Human-
Robot Environment

Michael Grupp 11 Philipp Kopp 22 Patrik Huber 33 Matthias Rätsch 22

###### Abstract

Face analysis techniques have become a crucial component of human-machine
interaction in the fields of assistive and humanoid robotics. However, the
variations in head-pose that arise naturally in these environments are still a
great challenge.

In this paper, we present a real-time capable 3D face modelling framework for
2D in-the-wild images that is applicable for robotics. The fitting of the 3D
Morphable Model is based exclusively on automatically detected landmarks.
After fitting, the face can be corrected in pose and transformed back to a
frontal 2D representation that is more suitable for face recognition. We
conduct face recognition experiments with non-frontal images from the MUCT
database and uncontrolled, in the wild images from the PaSC database, the most
challenging face recognition database to date, showing an improved
performance.

Finally, we present our SCITOS G5 robot system, which incorporates our
framework as a means of image pre-processing for face analysis.

##  1 Introduction

Among the technologies for biometric identification and verification, face
recognition has become a widely-used method, as it is a non-intrusive and
reliable method.

However, the robustness of a facial recognition system is constrained by the
degree of head pose that is involved and the recognition rates of state of the
art systems drop significantly for large pose angles. Especially for tasks
where a great variation in head pose has to be expected, like in human-robot
interaction, pose-invariant face recognition is crucial. The rise of
collaborative and assistive robots in industrial and home environments will
increase the demand for algorithms that can adapt to changing settings and
uncontrolled conditions. A humanoid robot’s cognitive ability to interpret
non-verbal communication during conversations relies also heavily on the face
of the human counterpart.

The impact of head pose on the face analysis performance can be minimised by
using normalisation techniques that transform non-frontal faces into frontal
representations. This type of image pre-processing can loosely be classified
into two categories:

  * •

2D methods (cylindrical warping, AAM (active appearance models) [7], 2D
warping [2])

  * •

3D methods (3DMM, GEM (generic elastic models) [22], mean face shape [3])

As the 2D methods are trained on 2D data, the warping will only be an
approximation of the underlying 3D rotation. They also do not model imaging
parameters like the pose of the face and illumination, and face parameters
like expressions, explicitly. Instead, these parameters are inherent and
implicitly encoded in the model parameters. When dealing with larger pose
ranges than around $\pm$40∘ in yaw angle, a single 2D model is often no longer
sufficient and a multi-model approach must be used (e.g. [8], [13]).

The use of a 3D face model has several advantages, especially when further
analysis involving the shape of the face is required, like in emotion
detection. To correct the pose for face recognition, a 3D representation of
the face can be rotated easily into a frontal view. At close ranges, the use
of a 3D or RGB-D sensor can provide the additional depth information for the
model. However, since the error of depth measurements increases quadratically
with increasing distance when using RGB-D sensors like Microsoft Kinect [17],
they are not suitable for acquiring image data from a distance. On the other
hand, 2D cameras are more widely-used in existing platforms and cost-
efficient, so that a method combining the advantages of 2D image acquisition
and the analysis with a 3D model is needed.

The 3D Morphable Model (3DMM) [6] satifies this need by providing a
parameterised Principal Component Analysis (PCA) model for shape and albedo,
which can be used for the synthesis of 3D faces from 2D images. A Morphable
Model can be used to reconstruct a 3D representation from a 2D image through
fitting. Fitting is the process of adapting the 3D Morphable Model in such a
way that the difference to a 2D input image is as small as possible. Depending
on the implementation, the fitter’s cost function uses different model
parameters to iteratively minimise the difference between the modeled image
and the original input image.

Existing fitting algorithms solve a complex cost function, using the image
information to perform shape from shading, edge detection, and solve for both
shape and texture coefficients ([24, 25, 26, 15, 11]). Due to the complexity
of the cost functions, these algorithms require several minutes to fit a
single image. While the execution time may not be crucial for offline
applications, real-time robotics applications require faster solutions. More
recently, new methods were introduced that use local features [16] or
geometric features (landmarks, edges) with a fitting algorithm similar to the
iterative closest point algorithm [4].

In this paper, we present and evaluate a complete, fully automatic face
modelling framework consisting of landmark detection, landmark-based 3D
Morphable Model fitting, and pose-normalisation for face recognition purposes.
The whole process is designed to be lightweight enough to run in real-time on
a standard PC, but is also especially intended to be used for human-robot
interaction in uncontrolled, in the wild environments. The theory is explained
in Section 2. In Section 3, we show how the approach improves the recognition
rates of a regular COTS (commercial off-the-shelf) face recognition system
when it is used as a pre-processing method for pose-normalisation on non-
frontal and uncontrolled images from two image databases. Finally, we show how
our approach is suitable for real-time robotics applications by presenting its
integration into the HMI framework of our SCITOS robot platform in Section 4.
Section 5 concludes the paper and gives an outlook to future work.

All parts of the framework are publicly available to support other
researchers: the C++ implementations of the landmark detection
111https://github.com/patrikhuber/superviseddescent and the fitting algorithm
and the face model 222https://github.com/patrikhuber/eos, as well as a demo
app that combines both 333http://www.4dface.org.

##  2 Landmark-based 3D Morphable Model Fitting

In this section, we will give a brief introduction to the 3D Morphable Model
and then introduce our algorithm to recover pose and shape from a given 2D
image. We will then present our pose-invariant texture representation in form
of a so-called isomap.

###  2.1 The 3D Morphable Model

A 3D Morphable Model consists of a shape and albedo (colour) PCA model
constructed from 169 3D scans of real faces. These meshes first have to be
brought in dense correspondence, that is, vertices with the same index in the
mesh correspond to the same semantic point on each face. The model used for
this implementation consists of 3448 vertices. A 3D shape is expressed in the
form of
$\mathbf{v}=[x_{1},y_{1},z_{1},\dots,x_{{}_{V}},y_{{}_{V}},z_{{}_{V}}]^{\mathrm{T}}$,
where $[x_{v},y_{v},z_{v}]^{\mathrm{T}}$ are the coordinates of the $v$th
vertex and $V$ is the number of mesh vertices. The RGB colour values are
stacked in a similar manner. PCA is then applied to both the vertex- and
colour data matrices separately, each consisting of $m$ stacked 3D face
meshes, resulting in $m-1$ shape eigenvectors $\mathbf{S}_{i}$, their
variances $\sigma_{S,i}^{2}$, and a mean shape $\mathbf{\bar{s}}$, and
similarly for the colour model ($\mathbf{T}_{i}$, $\sigma_{T,i}^{2}$ and
$\mathbf{\bar{t}}$). A face can then be approximated as a linear combination
of the respective basis vectors:

$\mathbf{s}=\mathbf{\bar{s}}+\sum_{i=1}^{m-1}\alpha_{i}\mathbf{S}_{i},~{}~{}~{}~{}\mathbf{t}=\mathbf{\bar{t}}+\sum_{i=1}^{m-1}\beta_{i}\mathbf{T}_{i},$
(1)

where $\boldsymbol{\alpha}=[\alpha_{1},\dots,\alpha_{m-1}]^{\mathrm{T}}$ and
$\boldsymbol{\beta}=[\beta_{1},\dots,\beta_{m-1}]^{\mathrm{T}}$ are vectors of
shape and colour coefficients respectively.

###  2.2 Shape and Pose Reconstruction

Given an image with a face, we would like fit the 3D Morphable Model to that
face and obtain a pose invariant representation of the subject. At the core of
our fitting algorithm is an affine camera model, shape reconstruction from
landmarks, and a pose invariant textural representation of the face. We thus
only use the shape PCA model from the 3DMM and not the colour PCA model, and
instead use the original texture from the image to obtain the best possible
image quality for subsequent face analysis steps.

Similar to Aldrian and Smith [1], we obtain a linear solution by decomposing
the problem into two steps which can be alternated. The first step in our
framework is to estimate the pose of the face. Given a set of 2D landmark
locations and their known correspondences in the 3D Morphable Model, we
compute an affine camera matrix. The detected 2D landmarks
$x_{i}\in\mathbb{R}^{3}$ and the corresponding 3D model points
$X_{i}\in\mathbb{R}^{4}$ (both represented as homogeneous coordinates) are
normalised by similarity transforms that translate the centroid of the image
and model points to the origin and scale them so that the Root-Mean-Square
distance from their origin is $\sqrt{2}$ for the landmark and $\sqrt{3}$ for
the model points respectively: $\tilde{x}_{i}=\text{U}x_{i}$ with
$\text{U}\in\mathbb{R}^{3\times 4}$, and $\tilde{X}_{i}=\text{W}X_{i}$ with
$\text{W}\in\mathbb{R}^{4\times 4}$. Using $\geq 4$ landmark points, we then
compute a normalised camera matrix $\tilde{\text{C}}$ using the Gold Standard
Algorithm [14] and obtain the final camera matrix after denormalising:
$\text{C}=\text{U}^{-1}\tilde{\text{C}}\text{W}$.

The second step in our framework consists of reconstructing the 3D shape using
the estimated camera matrix. We estimate the vector of PCA shape coefficients
$\boldsymbol{\alpha}_{s}$:

$\underset{\boldsymbol{\alpha}_{s}}{\operatorname{arg}\,\operatorname{min}}\;\sum_{i=1}^{3N}(x_{m,i}-x_{i})^{2}+\lambda\lVert\boldsymbol{\alpha}_{s}\rVert^{2}\,,$
(2)

where $N$ is the number of landmarks and $\lambda$ is a weighting parameter
for the regularisation that is needed to only allow plausible shapes. $x_{i}$
are the detected landmark locations and $x_{m,i}$ is the projection of the 3D
Morphable Model shape to 2D using the estimated camera matrix. Subsequently,
the camera matrix can then be re-estimated using the now obtained identity
specific face shape, instead of only the mean face. Both steps are iterated
for a few times - each of them only involves solving a small linear system of
equations.

In the experiments with automatically detected landmarks, we use a cascaded
regression based approach similar to Feng et al. [12]. After an initial
estimate, i.e. placing the landmarks in the region found by a face detector,
we extract local features (HOG [9]) to update the landmark locations until
convergence towards their actual position. The detected 2D landmarks have
known corresponding points in the 3D Morphable Model, and these points are
then used to fit the model. The combination of the regression based landmark
detection and the landmark-based 3D Morphable Model fitting results in a
lightweight framework that is feasible to run on videos (all components run in
the order of milliseconds on a standard CPU).

###  2.3 Pose-independent Face Representation

After fitting the pose and shape, a correspondence between the 3D mesh and the
face in the 2D image is known for each point in the image. We use this
correspondence to remap the original face texture from the image onto a pose-
invariant 2D surface that covers the face from all angles. We create such a
generic representation with the isomap algorithm [27]: it finds a projection
from the 3D vertices to a 2D plane that preserves the geodesic distance
between the mesh vertices. Our mapping is computed with the algorithm from
Tena [23].

The isomap of different persons are in dense correspondence with each other,
meaning each location in the map corresponds to the same physical point in the
face of every subject (for example, a hypothetical point $x=[100,120]$ is
always the center of the right eye). It can consequently show the accuracy of
the fitting and is therefore a plausible representation of the result.

Figure 1 shows an isomap and a frontal rendering side by side. The isomap
captures the whole face, while the rendering only shows the frontal parts of
the face. As the face in the input image is frontal, with most parts of the
face visible, there is little self occlusion. In case of large poses, a major
part of the isomap can not be filled with face texture, and these regions will
be marked black (for example, a small black spot next to the nose can be
observed in the figure).

Figure 1: Example picture from PaSC (left), frontal rendering (middle) and
pose invariant isomap representation (right).

##  3 Face Recognition Experiments

To analyse the performance of our approach, we have run experiments on two
different image databases (MUCT and Point and Shoot Challenge (PaSC)). Our
experiments are face verification experiments, which means that the system has
to verify the identity of probe images compared to a set of gallery images via
matching. The results are then statistically analysed regarding the ratio of
the two possible error types, the false acceptance rate (FAR) and false
rejection rate (FRR) and plotted into detection error tradeoff (DET) curves
[19], using the ground-truth subject-ID information provided in the metadata
of the image databases. In the experiments on the PaSC database (section 3.2),
we filter the probe and gallery images according to different head poses.

We used a market-leading, commercial face recognition engine for the process
of enrolment and matching444Our results obtained with commercial systems
should not be construed as the vendors’ full-capability results.. As a
reference measure, all experiments were also conducted using the original,
unprocessed images from the databases (denoted as org). With this reference,
we are able to analyse the impact of using our approach compared to a
conventional face recognition framework without 3D modelling.

###  3.1 Experiments on the MUCT Image Database

The MUCT Database consists of 3755 faces of 276 subjects and is published by
the University Of Cape Town [20]. For our experiments only persons without
glasses and their mouths closed were used. After this filtering, 1221 pictures
were left. The pictures are taken from 5 different cameras that cause
different pose angles and there are in total 10 different lighting schemes
applied to the pictures. Moreover, the size of the faces within the images is
quite large and the subjects are placed in a controlled lab environment. MUCT
comes with 76 manually labeled landmarks given for every image, from which we
used 16 for the initialisation of our landmark-based 3DMM fitting (LM).

In total, we used the unprocessed images (org) and the pose normalised
renderings coming from our fitting method (LM). We ran a verification
experiment on these images, leading to similarity matrices with 1 490 841
scores for both methods. Using these scores, a DET curve was plotted, of which
a relevant part is depicted in Figure 2.

Figure 2: DET curve showing the fewer recognition errors when using our
landmark-based fitting method (LM) on the MUCT image database. The dotted line
shows the result for the experiment with the original images without pose
normalisation (org).

It is clear to see that the verification errors of the face recognition drop
significantly when our landmark-based modelling approach is used to correct
the head pose. This experiment shows that when pose is the major uncontrolled
factor, like on MUCT, the usage of our LM aproach leads to a clearly improved
face recognition performance.

###  3.2 Experiments on the Point and Shoot (PaSC) Image Database

For our proposed use case in assistive or humanoid robotics, MUCT is not
representative of typical in the wild images we encounter. To verify the
performance in heavily uncontrolled settings, we decided to do further
investigations on a larger, more challenging database. The Point and Shoot
Challenge (PaSC) is a performance evaluation challenge for developers of face
recognition systems initiated by the Colorado State University [5].

As the name depicts, the images and videos used in the challenge are taken
with consumer point and shoot cameras and not with the help of professional
equipment. The database offers 9376 still images of varying resolution up to
4000$\times$3000 pixels and labeled metadata. The arrangement of the image
sceneries raises some major challenges for the automated recognition of faces.
The pictures were taken in various locations, indoors and outdoors, with
complex backgrounds and harsh lighting conditions, accompanied by a low image
quality due to blur, noise or incorrect white balance. Additionally, the
regions of interest for face recognition are also relatively small and of low
resolution because the photographed people are not the main subject of the
scenery.

In a direct comparison, the verification rate on PaSC equals only 0.22 at
0.001 FAR, in contrast to the more controlled databases MBE (0.997), GBU (0.8)
and LFW (0.54) [5].

Because of these harsh, uncontrolled conditions, which are similar to a real
life scenario, we chose this image database for our experiments and applied
our fitting algorithm (LM) to the images. Like the previous experiment on MUCT
(sec. 3.1), it was a verification experiment using the same COTS face
recognition algorithm.

####  3.2.1 Influence of the Number of Landmarks

(a)

(b)

Figure 3: Evaluation of different sets of facial landmarks. (a) visualises the
landmark sets on the mean face of the 3D Morphable Model. red: 7; red+blue:
13; red+blue+green: 49. (b) shows a DET curve for face recognition on PaSC
when using the different sets of landmarks (7, 13 and 49) for our LM fitting
algorithm.

Our experiments on PaSC are intended to be as near to a realistic scenario as
possible. Therefore, we used the automatic landmark detection of [12] instead
of using manually annotated facial landmarks. The detector and our fitting
algorithm (LM) allow the use of different landmark schemes, which can vary in
their amount. We tested three different sets of 7, 13 and 49 landmarks, which
are visualised on the 3D Morphable Model in Figure 3 (a). To compare the
different amounts of landmarks, their corresponding results for the face
recognition experiment on PaSC are shown in Figure 3 (b). It can be observed
that a higher number of facial landmarks leads to a higher quality 3DMM
fitting, which then leads to a better verification performance.

####  3.2.2 Performance Evaluation across Head Pose

One of the key questions of this work is whether our approach can improve the
performance of face recognition under uncontrolled conditions, especially when
dealing with head pose.

To allow a more detailed interpretation of the results, we generated head pose
annotations for the PaSC database. We used OpenCV’s implementation of the
POSIT algorithm [10] to calculate the yaw angles for each image in the
database. POSIT estimates the pose of an object from an image using a 3D
reference model and corresponding feature points of the object in the image.
In our case, the 3D model was the 3D Morphable Model and the feature points
were facial landmarks.

We used this yaw angle annotation to filter the scores into different groups
of yaw angles. The results obtained with our method were then compared to the
results without any face modelling. For comparison, we also tested the
performance of a commercial face modelling system (CFM) using the same
workflow. The CFM uses additional modelling capabilities like mirroring (to
reconstruct occluded areas of the face) and deillumination. Figure 4 depicts
frontalised renderings of the commercial system and our fitter. In total, we
compared all images of PaSC against each other and the yaw angle filter
separated the scores into 10∘ bins that span from -70∘ to +70∘.

Figure 4: From left to right: example images from PaSC (cropped to area of
face for this figure) and their frontalised renderings from the commercial
face modelling solution and our landmark-based fitter. Figure 5: Face
recognition improvement over the baseline COTS system of two face modelling
approaches. Different probe angles are compared against a frontal gallery.
_(blue):_ our 3DMM landmark fitting (LM), _(black):_ commercial face modelling
system (CFM).

Figure 6: Face recognition improvement over the baseline COTS system of two
face modelling approaches. Different probe angles are compared against a 20∘
and 40∘ gallery. _(blue):_ our 3DMM landmark fitting (LM), _(black):_
commercial face modelling system (CFM).

To better see how the pose normalisation methods improve the performance, the
resulting curves show the absolute difference of the false rejection rate to
the baseline without face modelling (org) at a specific operating point. This
type of diagram allows a more simple and clearly arranged representation of
the results at different head poses.

In Figure 5, such a curve from the experiment is shown. A set of gallery
images of 0∘ yaw is compared to sets of probe images from 0∘ to 70∘. As
gallery databases in face recognition applications often contain frontal
images, this experiment reflects a common use case. At the operating point
0.01 FAR, we then plot the improvement for each group of yaw angles. The
commercial face modelling solution shows its first improvements over the
conventional face recognition starting at 30∘ yaw. Surprisingly, our 3D
Morphable Model based method is even able to slightly improve the matching
performance for frontal images. Although the COTS face recognition system was
used outside of the specification for the large yaw angles, we are able to
improve the performance for the whole pose range. By using the original
image’s texture, our fitter (LM) does not alter the face characteristics and
corrects only rotations in the pitch or roll angles.

In the next two experiments, we take a look at how well the system operates
when the gallery is also not frontal. In Figure 6, the results for the
experiment with a 20∘ and a 40∘ gallery can be seen. In this case, matching
the faces is a much more difficult task, especially when the subjects look in
different directions and large areas of the faces are not visible due to self-
occlusion. While we are still able to improve the face recognition
capabilities in the experiment with a 20∘ gallery, our method LM struggles to
keep up with the commercial solution for a 40∘ gallery. The ability to
reconstruct parts of the non-visible texture is a key component of the
commercial modelling system (CFM) in such a use case which our system
currently doesn’t offer. Although, for the robotics application, which we
present in section 4 of this paper, it is likely that frontal images are used
for the gallery enrolment.

##  4 Application on a Robot System

Our fast and accurate 3D face modelling method is particularly suitable for
face analysis tasks in assistive or humanoid robotics. In the following, we
demonstrate our efforts in this field by illustrating how we integrate our
approach into a mobile robot’s software framework.

Figure 7: Schematic overview of the framework for human-robot interaction on
our SCITOS G5 platform. Our proposed landmark based 3D Morphable Model fitter
acts as a pre-processing method for various face analysis tasks.

A mobile robot platform combined with an HMI system is the basis for our
research on human-robot interaction. The robot is based on a SCITOS G5 drive
system with onboard industrial PC which is able to navigate autonomously in
indoor buildings. For mapping and localisation ability, as well as collision
avoidance, two laser scanners are attached to the base (SICK S300 and Hokuyo
URG-04LX). Using this base platform, the robot is able to approach humans
safely. The HMI part of the robot consists of a touch-based display and a head
with moveable, human-like eyes. On top of the robot, we mounted a pan-tilt-
unit (PTU; Directed Perception D46-17), where additional sensors and cameras
can be installed. The PTU adds an additional degree of freedom which is used
for camera positioning to enhance the tracking process.

To align the cameras to the region of interest, we track people by applying an
adaptive approach which uses a particle filter to track the position and size
of the target and estimates the target motion using an optical flow based
prediction model. Furthermore, an SVM-based learning strategy is implemented
to calculate the particle weights and to prevent bad updates while staying
adaptive to changes in object pose [21, 18]. At this point, the 3DMM pose
normalisation of our approach can be used for the image of the tracked face.
It can be either applied continuously or if the pose of the subject is greater
than a certain threshold (using the knowledge obtained in the experiments of
section 3.2).

Figure 8: Frames from a video of a subject turning from left profile to right
profile view. The bottom row shows the pose-invariant isomap representations
after fitting. Black regions depict self-occlusion/invisible parts of the face
and thus cannot be drawn. On the two most extreme pose images, the
initialisation with the face detector fails.

Key frames of an example video taken from our robot system are presented in
the upper row of Figure 8. The OpenCV object detection with a face cascade
[28] was used to initialise the regression based landmark detector, which then
automatically found 15 facial landmarks. The fitting results, represented in
isomaps, are shown in the second row of Figure 8. On the robot, we use the
frontal renderings as an input for the same face recognition engine that was
also used for the experiments. To distinguish the person’s identity properly,
the system should generate a rather high score for this positive match. The
pose normalised renderings allow the face recognition algorithm to achieve a
clearly higher score compared to the original, unprocessed images. Possible
use cases for a robot of this kind are industrial mobile robotics, shop or
museum tour guides or a surveillance system (e.g. in a supermarket). In all
these cases, the robot has to interact naturally with technically unskilled
people. It is therefore convenient to be able to analyse people independent of
their pose, without requiring them to look at the robot directly.
Identification, age and gender estimation on pose normalised renderings offer
the possibility to personalise the robot’s behaviour.

##  5 Conclusion and Further Work

We proposed a powerful method for head pose correction using a fully automatic
landmark-based approach for fitting a 3D Morphable Model. By using an
efficient fitting algorithm, our approach can be used for tasks which require
a fast real-time solution that can be also used on live video streams. In
contrast to existing work, we focus on an evaluation of our approach with
automatically found landmarks and in-the-wild images and publish the whole
3DMM fitting pipeline as open source software (see section 1).

An experimental evaluation on the two commonly-used image databases MUCT and
PaSC showed the significance of our approach as a means of image processing
for facial recognition in both controlled and heavily unconstrained settings.
For recognition algorithms that are mostly trained for frontal faces, a
rotation of the head means a loss of information that results in a lower
matching score. We showed that our methodology is capable of improving the
recognition rates for larger variations in head pose. Compared to a commercial
face modelling solution, our approach keeps up well and even outperforms it in
certain scenarios. The addition of 3DMM pose normalisation certainly brings
advantages compared to a conventional face recognition framework when the
setting is uncontrolled - like in robotics, surveillance or consumer
electronics.

Furthermore, we presented our software framework on a mobile robot platform
which uses our pose normalisation approach to allow a more reliable face
analysis. We are currently doing further research on age- and emotion-
estimation systems to expand the abilities of our robot. In the future, we
also plan to statistically evaluate the effect on their performance when using
our landmark-based pose normalisation, like we did for face recognition in
this paper.

## Acknowledgements

The authors would like to thank Huan Fui Lee and the RT-LIONS robocup team of
Reutlingen University. We would also like to thank CyberExtruder, Inc. for
supporting our research.

## References

  * [1] Aldrian, O., Smith, W.A.: Inverse rendering of faces with a 3D morphable model. Pattern Analysis and Machine Intelligence, IEEE Transactions on 35(5), 1080–1093 (2013)
  * [2] Asthana, A., Jones, M.J., Marks, T.K., Tieu, K.H., Goecke, R.: Pose normalization via learned 2d warping for fully automatic face recognition. In: BMVC. pp. 1–11 (2011)
  * [3] Asthana, A., Marks, T.K., Jones, M.J., Tieu, K.H., Rohith, M.: Fully automatic pose-invariant face recognition via 3D pose normalization. In: Computer Vision (ICCV), IEEE International Conference on. pp. 937–944 (2011)
  * [4] Bas, A., Smith, W.A., Bolkart, T., Wuhrer, S.: Fitting a 3D morphable model to edges: A comparison between hard and soft correspondences. arXiv preprint arXiv:1602.01125 (2016)
  * [5] Beveridge, J.R., Phillips, P.J., Bolme, D.S., Draper, B.A., Givens, G.H., Lui, Y.M., Teli, M.N., Zhang, H., Scruggs, W.T., Bowyer, K.W., et al.: The challenge of face recognition from digital point-and-shoot cameras. In: Biometrics: Theory, Applications and Systems (BTAS), IEEE Sixth International Conference on. pp. 1–8 (2013)
  * [6] Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th annual conference on Computer graphics and interactive techniques. pp. 187–194. ACM Press/Addison-Wesley Publishing Co. (1999)
  * [7] Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. IEEE Transactions on pattern analysis and machine intelligence 23(6), 681–685 (2001)
  * [8] Cootes, T.F., Taylor, C.J.: Statistical models of appearance for computer vision. Technical report, University of Manchester (2004)
  * [9] Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society Conference on. vol. 1, pp. 886–893. IEEE (2005)
  * [10] Dementhon, D.F., Davis, L.S.: Model-based object pose in 25 lines of code. International journal of computer vision 15(1-2), 123–141 (1995)
  * [11] Egger, B., Schönborn, S., Forster, A., Vetter, T.: Pose normalization for eye gaze estimation and facial attribute description from still images. In: 36th German Conference on Pattern Recognition (Sep 2014)
  * [12] Feng, Z.H., Huber, P., Kittler, J., Christmas, W., Wu, X.J.: Random cascaded-regression copse for robust facial landmark detection. Signal Processing Letters 22(1), 76–80 (2015)
  * [13] Feng, Z.H., Kittler, J., Christmas, W., Wu, X.J.: Feature level multiple model fusion using multilinear subspace analysis with incomplete training set and its application to face image analysis. Lecture Notes in Computer Science 7872, 73–84 (2013)
  * [14] Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, ISBN: 0521540518, second edn. (2004)
  * [15] Hu, G., Chan, C.H., Kittler, J., Christmas, B.: Resolution-aware 3D morphable model. In: BMVC. pp. 1–10 (2012)
  * [16] Huber, P., Feng, Z.H., Christmas, W., Kittler, J., Rätsch, M.: Fitting 3D morphable models using local features. ICIP (2015)
  * [17] Khoshelham, K.: Accuracy analysis of kinect depth data. In: ISPRS workshop laser scanning. vol. 38, pp. 133–138 (2011)
  * [18] Kopp, P., Grupp, M., Poschmann, P., Böhme, H.J., Rätsch, M.: Tracking system with pose-invariant face analysis for human-robot interaction. In: Informatics Inside (2015)
  * [19] Martin, A., Doddington, G., Kamm, T., Ordowski, M., Przybocki, M.: The DET curve in assessment of detection task performance. Tech. rep. (1997)
  * [20] Milborrow, S., Morkel, J., Nicolls, F.: The MUCT Landmarked Face Database. Pattern Recognition Association of South Africa (2010)
  * [21] Poschmann, P., Huber, P., Rätsch, M., Kittler, J., Böhme, H.J.: Fusion of tracking techniques to enhance adaptive real-time tracking of arbitrary objects. Conference on Intelligent Human Computer Interaction (IHCI) (2014)
  * [22] Prabhu, U., Heo, J., Savvides, M.: Unconstrained pose-invariant face recognition using 3D generic elastic models. IEEE Transactions on Pattern Analysis and Machine Intelligence 33(10), 1952–1961 (2011)
  * [23] Rodríguez, J.R.T.: 3D Face Modelling for 2D+3D Face Recognition. Ph.D. thesis, University of Surrey (2007)
  * [24] Romdhani, S., Vetter, T.: Estimating 3D shape and texture using pixel intensity, edges, specular highlights, texture constraints and a prior. In: Computer Vision and Pattern Recognition, IEEE Computer Society Conference on. vol. 2, pp. 986–993 (2005)
  * [25] van Rootseler, R., Spreeuwers, L., Veldhuis, R.: Using 3D morphable models for face recognition in video. In: Proceedings of the 33rd WIC Symposium on Information Theory in the Benelux (2012)
  * [26] Tena, J.R., Smith, R.S., Hamouz, M., Kittler, J., Hilton, A., Illingworth, J.: 2D face pose normalisation using a 3D morphable model. In: Advanced Video and Signal Based Surveillance, IEEE Conference on. pp. 51–56 (2007)
  * [27] Tenenbaum, J.B., de Silva, V., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290, 2319–2323 (2000)
  * [28] Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Computer Vision and Pattern Recognition, IEEE Computer Society Conference on. vol. 1, pp. I–511 – I–518 (2001)