#  Atrous Space Bender U-Net (ASBU-Net/LogiNet)

Anurag Bansal Oleg Ostap Miguel Maestre Trueba Kristopher Perry
Logitech Inc.
{anuragb, oostap, mmaestretrueba<EMAIL_ADDRESS>
###### Abstract

With recent advances in CNNs, exceptional improvements have been made in
semantic segmentation of high resolution image in terms of accuracy and
latency. However, challenges still remain in detecting objects in crowded
scenes, large scale variations, partial occlusion, and distortions, while
still maintaining mobility and latency. We introduce a fast and efficient
convolutional neural network, ASBU-Net, for semantic segmentation of high
resolution images that addresses these problems and uses no novelty layers for
ease of quantization and embedded hardware support. ASBU-Net is based on a new
feature extraction module, atrous space bender layer (ASBL), which is
efficient in terms of computation and memory. The ASB layers form a building
block that is used to make ASBNet. Since this network does not use any special
layers it can be easily implemented, quantized and deployed on FPGAs and other
hardware with limited memory. We present experiments on resource and accuracy
trade-offs and show strong performance compared to other popular models.

##  I Introduction

Ever since AlexNet [1] popularized deep convolutional neural networks by
winning the ImageNet Challenge: ILSVRC 2012 [2], everyone is trying to
incorporate convolutional neural networks in their products as an end to end
solution or a part of the state machine. The general research has been focused
to improve accuracy of detection by making the networks deeper and more
complicated [3][4][5][6][7][8]. With the advances in the accuracy of
detection, the need and demand for CNNs have been increasing everyday. In the
recent years, there has been a slight shift in paradigm where focus moved
towards having networks that can run on actual hardware with limited memory
and compute capabilities [9][10][11][12][13][14][15]. Thus CNNs are no longer
just running on cutting-edge GPUs in university research labs, but in most of
the electronic devices being used for video collaboration, autonomous driving,
warehouse automation, augmented reality, etc.

In recent years, visual data has grown volumetrically and qualitatively. The
resolution of images being captured by the mobile cameras in smartphones,
webcams or remote/in-office conference solutions is improving substantially.
Which means more dense information is available at our disposal to break down,
extract and utilize rich spatial information.

This paper describes our work in developing ASBU-net to further improve these
on device high accuracy efficient neural network models. Some of the key
challenges that we face in general and with most of these efficient models are
detecting objects - in a crowded environments, partially/heavily occluded
objects, significantly deformed objects, and most importantly the dynamic
scale of objects. To address most of these problems it is very important to
utilize the rich spatial and contextual information available in the image.
There has also been a lot of work on detecting small objects, crowded
environments and combination of both, but most of them are computationally
extensive. Some of these challenges can be illustrated with the example images
in figure below Fig 1.

Figure 1: Challenging Scenarios

Atrous/dilated convolutions [39] have successfully demonstrated capturing the
spatial and contextual information, and simultaneously also reducing the
computational complexity of deep CNNs with wider receptive fields. The
popularity of atrous convolutions has led to its adoption in various
efficient/mobile existing architectures like atrous spatial pyramid pooling,
efficient spatial pyramid, etc. In our approach we utilize the benefits of
using atrous convolutions too, but instead of applying aggressive parallel
dilated convolution which might cause a loss in spatial information or
aggressive series of dilated convolution, we propose a method which combines a
parallel and series combination of atrous convolutional layers in network-in-
network blocks.

Our goal here is to have a model that is optimizes the accuracy-latency trade
off on mobile devices and is easily deployable using most of standard
frameworks. We tackle all the above mentioned problems along with keeping the
model light by introducing the atrous space bender layers, which constitutes a
major building block of the atrous space bender network. And then we use the
technique from U-Net architecture [37] which seeks to retain high spatial
frequency information by directly adding skip connections between early and
late layers. The paper walks through the steps that were taken to solve each
problem and experiments that demonstrate the efficacy and value of each
technique evaluated with apple to apple comparison.

The paper is organized as follows. After the introduction of the topic and
problem statement we discuss the related work and some overlap in Section 2.
Later, in Section 3 we briefly introduce each concept and building blocks for
the model construction. Section 4 reviews the experimentation results and
benchmark results. Section 5 contains conclusions and future work.

##  II Related Work

###  II-A Contextual Models

Contextual information is important not just to “guess” small/blurry objects
based on a prior, but to make sense of the world i.e. resolve ambiguity, and
notice ”unusual” things. Current context based networks show that global
features or contextual interactions [41][42][43][44][45][46] are beneficial in
correctly classifying pixels for semantic segmentation. When training models
there has been an increase in the use of transfer learning that was originally
designed for classification networks. But the issue with this is that the
details in the downsampled features map are lost, which has been addressed by
learning the upsampling filters i.e. use of fractionally strided convolutions
[40]. Another popular work adopts dense CRF which takes the FCN output and
refines the segmentation boundaries. One of the other popular techniques that
has been used widely is atrous/dilated convolution based, which has shown to
preserve large receptive fields and give dense predictions. However, some of
these strategies cause drop in efficiency, for example, PSPNet (Zhao et al.)
[48] uses convolution on flat feature maps after pyramid pooling and
upsampling. The other popular technique, DeepLab [39] uses the ASPP module
that uses a parallel combination of dilated convolutions which causes two
problems. First, it might cause failure in aggregating the local features due
to sparsity of the kernels, and that is detrimental to small objects. Second,
aggressive dilation rate also reduces to 1x1 convolution in extreme cases. The
Atrous Space Bender-Net mitigates this by keeping a close to linear increase
in receptive field rather than exponential.

###  II-B Skip Connections

Skip connection has gained a lot of traction since the success of ResNet
architecture. However, the concept of skip connections has been around prior
to residual networks, for example, Highway Networks (Srivastava et al.) had
skip connections with gates that controlled and learned the flow of
information to deeper layers. Since then a lot of variations of skip
connections have been designed like skip connections via addition in ResNet by
He et al. in 2015 to solve the image classification problem, dense skip
connections via concatenation by Huang et al. in 2017 and one of the most
popular use cases long skip connections introduced in U-Net by Ronneberger et
al. for biomedical image segmentation. The reason why skip connections are so
popular because they alleviate the vanishing-gradient problem, strengthen
feature propagation, encourage feature reuse, and substantially reduce the
number of parameters. We leverage the long skip connections like a partial
U-Net architecture to retain high spatial frequency information by directly
adding skip connections between early and late layers. We dive into more
details in the section III-D

###  II-C Network Compression

In the recent years, there has been a slight shift in the paradigm where the
focus moved towards having networks that can run on actual hardware with
limited memory and compute capabilities [9][10][11][12][13][14][15]. Several
approaches around network compression and small architectures have been
researched and reported. However, there has always been a trade off between
accuracy, size, latency and covering edge cases. This paper proposes an
approach that has a configurable network to match the resource restriction,
latency and size, still collecting contextual information and giving
relatively better overall performance than existing architectures.

SqueezeNet [9] uses a bottle neck design approach with extensive 1x1
convolutions with squeeze and expand layers. MobileNets, [14][15][29] which
are built primarily from depthwise separable convolutions, improve upon the
number of operations using inverted residuals and linear bottlenecks.
ShuffleNet [30] utilizes group convolution and channel shuffle operations.
MnasNet [31] introduces lightweight attention modules based on squeeze and
excitation into the bottleneck structure. ShiftNet [32] proposes the shift
operation interleaved with point-wise convolutions to replace expensive
spatial convolutions. The other popular ways of smaller networks are based on
quantization (compressing), shrinking and factorizing [33][34][35]. Denton et
al. apply singular value decomposition (SVD) to a pretrained CNN model and Han
et al. developed network pruning. We also use 8-bit integer quantization to
further reduce the size, which is optional when a 32 bit float version is used
or a full precision model is run.

##  III Model Architecture And it’s Building Blocks

In this section we will go over the basic building blocks that are used to
make the ASBU-Net/LogiNet, which are atrous/dilated convolution, Atrous Space
Bender Layer, and Atrous Space Bender Network. We later combined all the
building blocks to create the final model design for the ASBU-Net. We will see
how effective use of atrous/dilated convolution helps in collecting dense
spatial features and thereby context around the objects. We will also discuss
how even though we use the concept of feature pyramid network to extract
smaller objects but still exploit the pyramidal hierarchy to construct feature
pyramids with marginal extra cost. And we also go over the reasons behind
design considerations.

Figure 2: ASB-Net

###  III-A Atrous/Dilated Convolution

Before we jump into describing the model modules, we will briefly discuss the
concept of atrous convolution which we leverage to extract contextual
information about the object and its surroundings. The core idea of atrous
convolution is to increase the receptive field of different scales by changing
the dilation rate while maintaining the size of the convolution kernel and
obtain dense feature maps. It injects holes into the standard convolution map
to expand the receptive field without increasing the parameters and channels.
A 1D atrous convolution can be defined as follows:

$y[i]=\sum_{k=1}^{K}x[i+r.k]w[k]$ (1)

where $y[i]$ is output signal, $x[i]$ is input signal with a filter $w[k]$ of
length $K$, $r$ corresponds to the dilation rate to sample $x[i]$, and
standard convolution is a special case for the rate $r=1$.

We will discuss in further detail as to how using atrous convolution with
proper dilation rate plays an important role in avoiding some known problems
and how we mitigate it by managing the receptive field growth close to linear
rather than exponential.

###  III-B Atrous Space Bender Layer (ASBL)

Figure 3: Atrous Space Bender Layer

The Atrous Space Bender layer forms the basic building block of the network.
It consists of a squeeze 3x3 convolution layer connected to a parallel
combination of 1x1 convolutions, 3x3 convolutions and 3x3 atrous convolutions,
the output of these layers are concatenated and then fed to the next layer. We
use a bottleneck like architecture with squeeze and expand layers to
accumulate the features.

The 3x3 convolution layer helps squeeze the feature maps channelwise while
still retaining the spatial information which is important to retain the
contextual information. This is then followed by a parallel combination of -

  * •

pointwise convolution that looks at 1 pixel at a time which mimics fully
connected behavior along the channel dimension and thereby helps retain
relationship amongst channels

  * •

3x3 convolution which as described earlier helps capture the spatial
information from the immediate neighbors i.e. it captures more local
information

  * •

3x3 atrous convolution with increasing and then decreasing dilation rates
which helps to increase the receptive field size, avoid downsampling, and
generate a multiscale framework for segmentation i.e. it acts as a good trade-
off between accurate localization (small field-of-view) and context
assimilation (large field-of-view)

The hyperparameters or the tunable parameters for this layer are the number of
filters in squeeze layer for ss3x3 (spatial squeeze), e1x1 (expand), se3x3
(spatial expand), ase3x3 (atrous spatial expand) and dilation rate for ase3x3.
The number of filters in e1x1 are set to be less than
$(e_{1}x1+se_{3}x3+ase_{3}x3)$ to limit the number of channels to the expand
layer. We maintain the squeeze:expand filter ratio below 1.

Figure 4: ASBU-Net

###  III-C Atrous Space Bender Network (ASB-Net)

In this section we go over the architecture design and strategies for the
entire ASB-Net which forms the backbone architecture to the ASBU-Net/LogiNet.
As shown in the figure 2 the ASB-Net begins with a primary/standalone (conv1)
3x3 convolution layer, followed by 7 ASB layers along with a couple of max
pooling operations inserted in between to scale down the feature maps. The max
pooling operations are performed to half the scale from the higher layer and
we reduce the scale until $\frac{1}{16}^{th}$ of the input image dimension.
There is also an option to reduce the scale until $\frac{1}{8}^{th}$ for a
smaller model. The max pooling operations with a stride of 2 are placed after
conv1, ASBL3, and ASBL5 for a network with $\frac{1}{16}^{th}$ scaling, and
after conv1 and ASBL3 for a network with $\frac{1}{8}^{th}$ scaling. We
minimize the use of max pooling operation to retain the information and also
downsample late in the network to retain maximum amount of information using
the strategy from SqueezeNet [9]. This network forms the encoder part of the
network which learns the feature representations by use of convolutional and
downsampling operations.

We gradually increase the number of filters per ASB-layer from the beginning
to the end of the network. Another important aspect of this network is the
selection of the dilation rates for each subsequent ASB-layer. Atrous layers
when applied aggressively causes the problem where the local spatial
information may be lost in the deeper layers and spatial inconsistency between
neighboring units. Which means the purpose of using the atrous convolution
which is to extract dense contextual information and decrease blurring in the
segmentation masks is defeated. There are two solutions that we propose to
combat this problem -

  * a.

increase the field of view of the ASB-layers almost linearly rather than
exponentially

  * b.

use a 3x3 and 1x1 convolution in parallel to atrous convolution, thereby
retaining the local spatial information

To achieve an almost linear increase in receptive field, we deploy a strategy
where we gradually increase the dilation/atrous rates in the ASB-layers and
then decrease it from top to bottom. As we know the atrous convolution
introduces holes in a convolutional kernel, while still maintaining the
weights in the kernel. When the dilation rate for a kernel is set to some
value let’s say $\alpha$ that means we introduce a stride of $\alpha$ between
each sample, thereby the receptive field of the kernel ($k>0$) is increased to
$(\alpha(k-1)+1)$ [36]. From this we can compute the receptive field, for the
dilation rate of 1 the atrous conv behaves like a regular convolution
operation i.e. the receptive field being equal to the size of the filter.
Similarly, for dilation rates for 2,3,4, and 5 the receptive field for a 3x3
atrous convolution can be computed as - $2(3-1)+1=5$, 7, 9, and 11
respectively. Now when these atrous blocks are placed in series combination,
the effective receptive field grows exponentially. The effective receptive
field at each layer can be calculated using the formula from - Computing
Receptive Fields of Convolutional Neural Networks [36].

$r_{0}=\sum_{l=1}^{L}\left((k_{l}-1)\prod_{i=1}^{l-1}s_{i}\right)+1\\\ $ (2)

where,
$k_{l}$: kernel size (positive integer)
$s_{l}$: stride (positive integer)
effective stride $S_{l}$ = $\prod_{i=l+1}^{L}s_{i}$ : the stride between a
given feature map $f_{l}$ and the output feature map $f_{L}$

For the atrous convolution we use the above equation by replacing kernel size
$k$ by $(\alpha(k-1)+1)$. From the model diagram, dilation rates we use to
define the local receptive field of ASB-layer keeps the effective receptive
field in check. When we calculate the effective receptive field for each layer
we see that the layerwise growth is almost linear and the flow of information
of neighboring units is connected again. Hence, we show that our approach
gradually recovers consistency between neighboring pixels and extracts local
structure in higher layer.

###  III-D Atrous Space Bender U-Net/LogiNet

The ASBU-Net as the name suggests utilises the above backbone to not only
improve the results but also to solve the problem of detecting objects at
scale, by improving the contextual information gathered by atrous convolution
blocks. As seen in figure 4, ASBU-Net is broken down into 2 major blocks -
feature extraction block, and feature decoder block. The feature decoder block
is further divided into 2 blocks - long connect block, which is a combination
of upscale connector blocks and ASB layers, and decoder block.

A couple of important things that are different from traditional U-Net [37],
are the 1x1 convolution layer introduced in the skip connection and long
connect blocks to which the skip connections feed into. The 3 reasons why we
do this are -

  * a.

To reduce the memory footprint from earlier layers

  * b.

To capture additional spatial information and reduce semantic gap in the
information

  * c.

To retain information along the channels thereby reducing the gap from the
encoding layers

The number of filters from the higher level layers in the encoder block if
passed directly to the decoder layer increases the memory footprint quite
significantly. In order to keep the memory requirement to a minimum without
losing information, we introduce 1x1 convolution to the skip connection to
reduce the number of channels being passed down yet retaining the dense
information.

It is important to note that using the U-Net like architecture helps us to
have feature reusability by connecting features from encoder to decoder.
Thereby, helping recover information lost during the downsampling and
encouraging the model to learn both the finer information (from layers close
to input) and the more generic information (from layers close to output) of
the objects. However, the shorter connections help in stabilizing the gradient
updates. Significant problem with connecting the encoder directly with decoder
is that, the layers in encoder contain low level feature information and as we
go deeper into the network into the decoder block the feature information is
supposed to be of much higher level. When these 2 layers are connected
directly, there may be a semantic gap that gets introduced and thereby
affecting the gradient propagation in the backward pass, which thereby also
affects the accuracy. Inorder to solve this, we introduce what we call long
connect blocks. For each skip connection a series combination of upscale
connector block and ASB-layer with dilation rate of 1 (i.e. regular 3x3
convolution) is added to the long connect block. The gradual merging of the
high level and low level features with upscale connector block and ASB-layer
helps retain the information that gets lost in the sudden gap.

Post the long connect block, we add a combination of 1x1 convolution and 3x3
convolution, followed by bilinear resizing and sigmoid activation to obtain
the semantic segmentation mask. Our goal while designing this network
architecture was to keep it simple and use simplistic layers that can be
easily quantized and run on embedded platforms.

##  IV Experimental Setup and Hyperparameters

In this section we discuss the experimental setup, training hyperparameters
and validation metrics which we used to achieve competitive results compared
to mobilenet-v2-unet and efficientnet-unet in quantitative evaluation on
Logitech Validation dataset. We have also tried to provide a sufficient level
of ablation study of the architecture and loss functions with qualitative
results. Finally, we demonstrate that ASBU-net is an adaptable and extensible
model for various use cases and can be easily deployed on almost any hardware
without dramatic changes.

###  IV-A Data and Augmentation

We perform experiments on Logitech Video Collaboration proprietary
segmentation dataset, containing 156k training images, which is split 80:20
during training for training and testing, and 4k validation images. Each image
is annotated with segmentation polygons. We also use various data augmentation
techniques to enhance the dataset during the runtime, like auto cropping,
image flipping, adding gaussian noise, gamma variation, shadow augmentation,
and color/saturation modification to name a few. For pre-processing operation
we normalize the images before forward pass.

###  IV-B Training Details

Our entire training pipeline was implemented using Tensorflow backend [38].
The experiments were conducted on a desktop computer with Intel core i9-9900K
processor (3.6 GHz, 16 MB cache) CPU, 64 GB RAM, and NVIDIA GeForce RTX 2080
Ti (11 GB, 1545 MHz) GPU.

For training the ASBU-net we used a momentum optimizer setting the momentum to
0.9, with a linear decay initialized to $10^{-2}$ , multiplying the decay rate
by 0.1 at every 0.3 epoch. We use a very low value of $10^{-12}$ for l2
regularization. All convolution weights are initialized with Kaiming-He init
[49], and use batch-normalization layers with average decay of 0.99 for
training higher batch sizes. As part of training,and test data, we have split
the data-set into 80-20. The task of semantic segmentation is basically a
problem of pixel wise classification, hence for the loss function we use
weighted binary cross entropy. For convenience and ease of experimentation we
have trained the network on a single class of segmenting humans in the frame.

Figure 5: Results Heatmap Comparison

###  IV-C Evaluation Metric

It is important to decide a good evaluation metrics when validating the
results of segmentation models. The metrics itself is not just enough. The
labels need to be precise or closely encompassing the boundary of the actual
object. Since most of the evaluation metrics consider IOU of the segmentation
mask with the label mask, which means per pixel comparison, we usually end up
penalizing the network for its performance around the edges if the labels are
not accurate. Relative comparison between different models is not affected
when the validation data remains same, but the overall score is what can be
different. Inorder to combat this issue, we use ignore band which is
calculated as inverse of bitwise XOR of dilated and eroded label around a
label. This operation is performed based on the scale of the object to make
sure size of object does not affect the original mask around smaller objects.
Mathematically the ignore boundary band is given as follows -

$\displaystyle ll=Y\oplus J_{osf}$ (3) $\displaystyle sl=Y\ominus J_{osf}$
$\displaystyle b_{ignore}=\neg(ll\veebar sl)$ $\displaystyle
Y_{new}=Y\&b_{ignore}$ $\displaystyle Y_{pred\\{new\\}}=Y_{pred}\&b_{ignore}$

where,
osf = Objectwise scale factor
Y = original label mask
$Y_{pred}$ = predicted mask
ll = dilated label
sl = eroded label
$b_{ignore}$ = boundary to ignore

Once the new prediction map and label map are obtained using the above we then
calculate the Jaccard index i.e. ratio of the intersection and union. In
addition to this, we also add extra penalization for false detections by
reducing the score by 1.0 for each misdetection.

Figure 6: Results - Accuracy and Computation Time Comparison

Figure 7: Results - Total Params and Model Size Comparison

###  IV-D Results

In this section, we go over the results from experimenting with ASBU-
Net/LogiNet on the validation images from Logitech Video Collaboration
dataset. A large-scale dataset containing high quality pixel-level annotations
of 160,000 images. We use the 4000 images reserved for validation/test for
obtaining the validation scores using the above explained evaluation metrics.
It is also important to note that inorder to maintain the sanctity of the
experiments we used the exact same setup for training and testing for all the
3 models which are ASBU-Net/LogiNet, Mobilenet-v2-Unet, and Efficient Net
Unet.

Let’s first look at the results subjectively on the heatmaps in the figure 5.
We can see that most of these scenarios are pretty complex to detect humans
with complex lighting conditions, crowding, heavy camouflage, and heavy
occlusions. The subjective analysis shows that mobilenetv2 [15] based models
have higher false positives and weak detections in uneven lighting conditions,
with objects at different scales. The efficientnet[47] based models however,
perform well with varied lighting conditions but have weak detections for
objects that are closer and heavily occluded. Both mobilenetv2 [15] and
efficientnet [47] based models fail to detect heavily camouflaged people of
color at the end of the table. Under all these and more conditions we see that
the ASBU-net/Loginet performs consistently well.

As seen in the results table in figure 6 and 7, we compare the ASBU-
Net/LogiNet in terms of accuracy, total params, model size and computation
time. We show that the ASBU-Net/LogiNet is pretty consistent in terms of
performance in all these areas and still computationally inexpensive as
compared to the other popular models. The validation accuracy is almost 3%
better than Mobilenet v2 U-Net with a multiplier 1.4, and almost 1.5% better
than EfficientNet U-Net version B2. We also show that the computation time is
almost 3x faster for best performing efficient net and 1.5x$\sim$2x faster
than mobilenet backend models. The computation times are on a workstation
running Intel(R) Core(TM) i9-9900K CPU 3.60GHz with NVIDIA GeForce RTX 2080
SUPER graphic card. ASBU-Net has 1.25x less parameters as compared to
efficientnet [47] based B2 model, and $\sim$1.1x less parameters than
mobilenetv2 [15] backed model, which means it will have lower power
consumption and lower battery discharge on embedded platforms.

##  V Conclusion

In this paper we present a novel model architecture based on dilated
convolutions that utilizes the contextual information to perform precise image
segmentation tasks. We also show that the general problems associated with the
dilated convolutions can be handled by using the ASB-layer which is
lightweight, demands less memory and computation, and can extract complex
contextual feature representations. We show that the network architecture is
more efficient, consistent and robust in comparison to some famous existing
architectures. Furthermore, we also share our validation method which helps us
ignore the labeling inefficiencies around the edges of the image and focus on
the actual detections. We show that using the ASBU-Net you can detect objects
that are at different scales, crowded, deformed, or partially occluded, and
other complex conditions, and still maintain mobility and latency. We are
certain that ASBU-net/Loginet can be used for a wide variety of tasks and more
ablation studies can be performed to test more performance boost.

## References

  * [1] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012.
  * [2] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211–252,2015.
  * [3] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014
  * [4] C. Szegedy, S. Ioffe, and V. Vanhoucke. Inception-v4, inception-resnet and the impact of residual connections on learning. arXiv preprint arXiv:1602.07261, 2016.
  * [5] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. arXiv preprint arXiv:1512.00567, 2015.
  * [6] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385, 2015
  * [7] Xiang, Y., Fox, D.: DA-RNN: Semantic mapping with data associated recurrent neural networks. Robotics: Science and Systems (RSS) (2017)
  * [8] R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR, 2014.
  * [9] F. N. Iandola, M. W. Moskewicz, K. Ashraf, S. Han, W. J. Dally, and K. Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 1mb model size. arXiv preprint arXiv:1602.07360, 2016.
  * [10] J. Jin, A. Dundar, and E. Culurciello. Flattened convolutional neural networks for feedforward acceleration. arXiv preprint arXiv:1412.5474, 2014
  * [11] M. Wang, B. Liu, and H. Foroosh. Factorized convolutional neural networks. arXiv preprint arXiv:1608.04337, 2016.
  * [12] J. Wu, C. Leng, Y. Wang, Q. Hu, and J. Cheng. Quantized convolutional neural networks for mobile devices.
  * [13] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. Xnornet: Imagenet classification using binary convolutional neural networks. arXiv preprint arXiv:1603.05279, 2016.
  * [14] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, abs/1704.04861, 2017
  * [15] Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. mobile networks for classification, detection and segmentation. CoRR, abs/1801.04381, 2018 Quantization
  * [16] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
  * [17] Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. CoRR, abs/1806.08342, 2018.
  * [18] Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices. CoRR, abs/1512.06473, 2015.
  * [19] Daniel Soudry, Itay Hubara, and Ron Meir. Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger, editors, NIPS, pages 963–971, 2014.
  * [20] Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantization: Towards lossless cnns with low-precision weights. CoRR, abs/1702.03044, 2017
  * [21] Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu,and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. CoRR, abs/1606.06160, 2016.
  * [22] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. CoRR, abs/1603.05279, 2016.
  * [23] P. Hu and D. Ramanan. Finding Tiny Faces. arXiv preprint arXiv:1612.04402, 2016.
  * [24] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A., et al.: Going deeper with convolutions. In: CVPR. (2015)
  * [25] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: CVPR. (2016)
  * [26] Szegedy, C., Ioffe, S., Vanhoucke, V.: Inception-v4, inception-resnet and the impact of residual connections on learning. CoRR (2016)
  * [27] Xie, S., Girshick, R., Dollar, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: CVPR. (2017)
  * [28] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. CVPR (2017)
  * [29] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for MobileNetV3. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
  * [30] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. CoRR, abs/1707.01083, 2017.
  * [31] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V. Le. Mnasnet: Platform-aware neural architecture search for mobile. CoRR, abs/1807.11626, 2018.
  * [32] Bichen Wu, Alvin Wan, Xiangyu Yue, Peter H. Jin, Sicheng Zhao, Noah Golmant, Amir Gholaminejad, Joseph Gonzalez, and Kurt Keutzer. Shift: A zero flop, zero parameter alternative to spatial convolutions. CoRR, abs/1711.08141,2017.
  * [33] Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. ICLR, 2016.
  * [34] He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., and Han, S. Amc: Automl for model compression and acceleration on mobile devices. ECCV, 2018.
  * [35] Yang, T.-J., Howard, A., Chen, B., Zhang, X., Go, A., Sze, V., and Adam, H. Netadapt: Platform-aware neural network adaptation for mobile applications. ECCV, 2018.
  * [36] André Araujo, Wade Norris, Jack Sim: Computing Receptive Fields of Convolutional Neural Networks. 10.23915/distill.00021
  * [37] Ronneberger O., Fischer P., Brox T. (2015, October). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention (pp. 234-241). Springer, Cham.
  * [38] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Rafal Jozefowicz, Yangqing Jia, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Mike Schuster, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org.
  * [39] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI (2017)
  * [40] V. Dumoulin and F. Visin, “A guide to convolution arithmetic for deep learning,” arXiv preprint arXiv:1603.07285, 2016.
  * [41] X. He, R. S. Zemel, and M. Carreira-Perpindn. Multiscale conditional random fields for image labeling. In CVPR, 2004.
  * [42] J. Shotton, J. Winn, C. Rother, and A. Criminisi. Textonboost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context. IJCV, 2009.
  * [43] P. Kohli, P. H. Torr, et al. Robust higher order potentials for enforcing label consistency. IJCV, 82(3):302–324, 2009.
  * [44] L. Ladicky, C. Russell, P. Kohli, and P. H. Torr. Associative hierarchical crfs for object class image segmentation. In ICCV, 2009.
  * [45] S. Gould, R. Fulton, and D. Koller. Decomposing a scene into geometric and semantically consistent regions. In ICCV. IEEE, 2009.
  * [46] J. Yao, S. Fidler, and R. Urtasun. Describing the scene as a whole: Joint object detection, scene classification and semantic segmentation. In CVPR, 2012
  * [47] Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ICML, 2019.
  * [48] Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J., 2017. Pyramid scene parsing network, in: Proc. of Computer Vision and Pattern Recognition (CVPR).
  * [49] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015

##  VI Supplementary

###  VI-A Model Layer Chart

Figure 8: ASBU-Net Encoder Layer Specifications


# Automated Structure Discovery for Scanning Tunneling Microscopy

Lauri Kurki  [  Niko Oinonen  [  Adam S. Foster<EMAIL_ADDRESS>[

###### Abstract

Scanning tunnelling microscopy (STM) with a functionalized tip apex reveals
the geometric and electronic structure of a sample within the same experiment.
However, the complex nature of the signal makes images difficult to interpret
and has so far limited most research to planar samples with a known chemical
composition. Here, we present automated structure discovery for STM (ASD-STM),
a machine learning tool for predicting the atomic structure directly from an
STM image, by building upon successful methods for structure discovery in non-
contact atomic force microscopy (nc-AFM). We apply the method on various
organic molecules and achieve good accuracy on structure predictions and
chemical identification on a qualitative level, while highlighting future
development requirements to ASD-STM. This method is directly applicable to
experimental STM images of organic molecules, making structure discovery
available for a wider SPM audience outside of nc-AFM. This work also opens
doors for more advanced machine learning methods to be developed for STM
structure discovery.

Aalto University] Department of Applied Physics, Aalto University, 00076
Aalto, Espoo, Finland Aalto University] Department of Applied Physics, Aalto
University, 00076 Aalto, Espoo, Finland Nanolayers Research Computing Ltd.,
London N12 0HL, United Kingdom Aalto University] Department of Applied
Physics, Aalto University, 00076 Aalto, Espoo, Finland [Kanazawa University]
WPI Nano Life Science Institute (WPI-NanoLSI), Kanazawa University, Kakuma-
machi, Kanazawa 920-1192, Japan

Keywords: scanning probe microscopy, scanning tunnelling microscopy, machine
learning

## Introduction

Scanning probe microscopy (SPM) methods are powerful tools for studying
nanoscale systems with atomic resolution. As the basis of SPM methods,
scanning tunnelling microscopy (STM) 1 and atomic force microscopy (AFM) 2
have been widely utilized in the characterization of various systems, such as
biological samples, hybrid inorganic-organic interfaces and individual steps
of on-surface reactions 3, 4, 5, 6, 7, 8, 9. To enhance the spatial accuracy
in the characterization, the probe tip can be functionalized using a
chemically inert, flexible apex (often a CO molecule) to allow scanning at a
very close tip-sample distance where Pauli repulsion is the dominating
interaction 10, 11. With respect to the detailed characterization of atomic
structures, there exists many demonstrations of the improved spatial
resolution of AFM scanning with functionalized tips 9, 12, 13, but STM is now
also being increasingly used in this bond-resolved mode 13, 8, 14, 15, 16, 17,
18. STM in particular benefits significantly from tip functionalization as
sharp sub-molecular features appear in the image, revealing both the molecular
skeleton and the electronic structure in high detail within the same
experiment – this is impossible using a bare metal tip 19 or by high-
resolution AFM. In addition to atomic structural characterization, STM with a
functionalized tip offers interesting approaches for electronic structure
characterization via frontier orbitals of the tip apex 20 by improving the
visibility of molecular orbitals 21 and by distinguishing nearby molecular
states 17.

In terms of structure characterization, the main limitations of STM are its
incapability to see beyond the closest atom to the tip and the insurmountable
problem of chemical identification of atoms. In fact, research using bond-
resolved STM has so far been mostly limited to planar molecules consisting of
only a priori known atomic species, and the recognition of an unknown sample
can require an extensive search through all possible molecules and
configurations. The same problems have been experienced in the field of non-
contact AFM where recent advances in machine learning image analysis have been
proven effective in structure discovery and chemical identification of single
molecules and ice structures 22, 23, 24, 25, 26, 27. For STM, advanced image
analysis methods have so far not been developed for bond-resolved imaging but
instead the focus has been on, e.g., defect detection 28, molecule keypoint
detection 29, surface characterization 30 and atom manipulation 31 as well as
autonomous experiments 32. Machine learning methods have also been used for
automating the tip conditioning 33 and tip functionalization 34 in STM. It is
clear that the focus in general has been on larger scale systems and
processes, and there is a need for automated characterization methods for
bond-resolved imaging with a CO tip.

In this work, we present automated structure discovery for STM (ASD-STM), a
machine learning approach for structure characterization directly from
experimental bond-resolved STM images. With ASD-STM, we offer a solution to
the problems in STM structure characterization while bridging the gap between
image analysis methods for high-resolution AFM and STM. As STM with a
functionalized tip is a more easily accessible characterization method
compared to non-contact AFM 35, this work also makes automated structure
discovery available for a wider SPM audience and opens a door to more
sophisticated methods to be developed for sample recognition in STM.

## Methods

### Simulated STM images

Figure 1: Scheme of ASD-STM. (A) Starting from a large set of organic
molecules, (B-E) we simulate STM and AFM for different rotations of the
molecules and calculate corresponding physical descriptors. (F) The images are
stored in a database and (G) used as training material for a machine learning
model for STM structure discovery.

The proposed method starting from a data set of molecule geometries and ending
in a machine learning model that can predict atomic structure directly from
experimental images is outlined in Fig. 1. The initial data set contains the
optimized geometries and atomic point charges of approximately 81,000 small
organic molecules with chemical species lighter than bromine ($Z\leq 35$) with
most emphasis on hydrogen, carbon, nitrogen and oxygen atoms (Fig. 1A) 25.
Simulating STM images requires that we first obtain the densities of states of
all molecules in the data set, and since the molecular structures are already
optimized, a single-point calculation is performed for all molecules. All ab-
initio calculations are performed using the FHI-aims code 36, 37 with the PBE
functional 38 considering only the $\Gamma$ $k$-point. The unit cell is
constructed by padding the molecule with $7$ Å of empty space on each side to
ensure that the periodic images do not interact and affect the electronic
structure.

The STM simulations for all molecules are done using the PPSTM code 39. PPSTM
is based on the Bardeen tunnelling theory 40 and it simulates constant height
raster scanning by calculating the tunnelling current at each point of a
three-dimensional grid of tip positions allowing for a quick simulation of
multiple constant height scans at different tip-sample distances (Fig. 1C,E).
In PPSTM, the tunnelling current is composed of contributions from individual
states of the tip and the sample. The sample states are obtained from DFT
calculations as mentioned, and they are represented by Lorentzian
distributions with $0.1$ eV broadening. The tip orbitals are considered as $s$
and $p$ orbitals located at the tip apex and they provide independent
tunnelling channels. The tunnelling channels are given a weight between $0$
and $1$ and the correct weights for a certain tip apex are found by matching
with experiments (e.g. approx. $10$ % $s$ and $90$ % $p$ orbitals for CO tip
41, 8). Parallel to the STM simulations, we run the PPAFM 42, 43 code to
calculate relaxation of the CO molecule near the sample and the relaxed
positions are, in turn, used in the STM simulation to achieve good resemblance
to experimental images revealing the molecular skeleton. PPAFM calculates the
CO relaxation by considering the CO molecule as a probe particle (PP) and
using a mechanical model taking into account the Lennard-Jones and
electrostatic potentials between the PP and the sample. The position of the PP
approaching the sample is relaxed by minimizing the total force acting upon
the PP, and while calculating the relaxed tip positions, we simultaneously
obtain AFM images of the molecules with little additional cost by integrating
the forces affecting the PP over its path 44. The AFM images are included in
the data set of synthetic images and they could be used as additional signal
in a method developed for simultaneous AFM/STM imaging.

To maximize the amount of information in the SPM images, we emphasized flat
regions normal to the scanning direction and also assured a more even
distribution of elements in the resulting images. To this end, we subjected
the molecules to a set of rotations calculated separately for each molecule
(Fig. 1B). The rotations were obtained by computing the convex hull of the
atomic coordinates which defines planar segments of the molecule, and the
rotations were weighted to include an even distribution of elements within
$0.7$ Å of the plane. After applying the rotations, our data set consists of
synthetic SPM images of approx. 48,000 unique molecules and 235,000 unique
images (Fig. 1F). This procedure and the full composition of the data set are
described in full detail in a previous work by Oinonen et al. 25.

One example item from the generated data set is shown in Fig. 1D-E. All items
in the data set include a stack of 10 constant height STM and AFM images taken
at $0.1$ Å separation, over a range of tip-sample distances starting from a
close distance scan where the molecular skeleton is revealed to a distance
where only the electronic structure at the Fermi level is seen. A stack of
images is created as opposed to just one slice because the real tip-sample
distance in the experiment is not known, and we want to avoid overfitting to
an exact scanning height by letting the model see images taken at a range of
different scanning heights. This is achieved by randomly selecting one 2D
slice from the 3D stack of images during training. Consequently, our model
does not require a full stack of images at prediction time, making structure
discovery easily accessible by using only one image. The data set also
includes three image descriptors for each molecule – atomic disks, van der
Waals spheres and height map 22 (Fig. 1D) – which we have chosen as visual and
two-dimensional representations of the molecules that we can calculate
analytically from the atomic coordinates. Image descriptors are a convenient
tool here as they are not only visually intuitive for people but also suitable
for many machine learning model designs. In all results, we focus on the
atomic disks descriptor as it provides the most discrete predictions for the
atomic positions and encodes the chemical information in the size of each
disk, proportional to the covalent radius of the atom. The relative height of
each atom is represented as brightness of the disk (background is zero), and
atoms deeper than $1.2$ Å relative to the top atom are not considered.

### Machine learning

In this work, the structure discovery task is formulated as an image-to-image
problem, for which we have developed and trained a machine learning model
which translates the STM image into a descriptor by using an Attention U-Net-
type model which utilizes an encoder-decoder architecture and an attention-
gating mechanism 45. On a conceptual level, the encoder compresses the
information of the input image (STM image) into a latent space vector, which
in turn is translated into a spatially larger representation (image
descriptor) by the decoder. The encoder consists of 4 convolutional blocks
containing two 2D convolutional layers with a LeakyReLU0.1 46 activation
function in between, followed by batch normalization 47 and a second LeakyReLU
activation. Each of the convolutional blocks is preceded by a downscaling
block which downsamples the image by using strided convolutions to increase
the receptive field. The same activation and batch normalization policies are
used in the downscaling. The decoder section consists of four blocks that
include an upscaling block, an attention gate, and a convolutional block
identical with the encoder. Upsampling is achieved by using a transposed 2D
convolutional layer, and in the block, the same activation and batch
normalization policies are used as in the encoder. The attention gate
generates highlighted regions in the input for the model to focus on by using
a type of skip connection where additional query signal comes from the
upscaled feature map. The attention map is concatenated with the upsampled
feature map and used as input for the convolutional block. The output layer of
the model is a $1\times 1$ convolutional layer followed by a ReLU activation
(see full details in Fig. LABEL:fig:attunet).

The ML model is implemented in PyTorch 48 and it is trained in a supervised
setting for which the images were divided into training (180,000 images),
validation (20,000) and testing (35,554) sets. The model was trained for 50
epochs and a batch size of 30 was used. The parameters of the model were
optimized using the Adam optimizer 49 to minimize the mean squared error.
During training, the simulated STM images are normalized to 0 mean and unit
variance and we also add white noise and cut-outs to the images, representing
electronic noise and tip artifacts to make the model not rely on pristine
simulated images but instead force it to make predictions on imperfect images.
After training, inference on the model can be done using true experimental
images to obtain a prediction of the structure with very little computational
cost.

## Results

### Predictions on simulated images

To benchmark our method and assess the improvement of prediction accuracy, we
use the trained model to predict atomic structures from simulated images not
included in the training data. The obvious benefit from analysing simulated
predictions is that the atomic coordinates of the reference, and therefore the
image descriptor, are known exactly. Fig. 2 shows three example predictions
from simulated images. The three examples have been taken at varying scanning
heights and the sample molecules are different in size and structure, and they
contain different chemical species and functional groups to demonstrate the
versatility of the method.

Molecule 1 (Fig. 2A-D) is the largest of the three and it has two benzene
groups and a 5-ring lactone group with an additional nitrogen heteroatom. The
molecule is planar and all atoms are visible to the tip resulting in the
carbon backbone skeleton being revealed with the exception of the benzene
rings which are not prominent in the STM image (Fig. 2B). Even though the
skeleton is not completely visible, the prediction (Fig. 2C) is very close to
the reference (Fig. 2D). All atoms, including hydrogens which are not visible
to the eye, are correctly located and the associated chemical identification
distinguishes hydrogen atoms from carbon, nitrogen and oxygen atoms. It is
also promising that the model correctly identifies the oxygen bonded to the
5-ring as not hydrogen and that it does not blindly add hydrogen atoms to
nitrogen and oxygen atoms of the 5-ring. Overall, the mean absolute error
relative to the range of values in the reference is $1.1$ %.

The second example shows a smaller molecule containing a nitro group and a
chlorine atom (Fig. 2E-H). The molecule is slightly tilted and it includes a
non-planar part in which the branch containing a carbonyl group is bending
downwards. This is projected in the reference descriptor by a barely visible
carbon and two of the bonded hydrogen atoms are too deep to be considered
(Fig. 2H). Again, the prediction resembles the reference very closely (Fig.
2G). The chlorine atom is distinguished from hydrogen atoms and the relative
height of the oxygen in the carbonyl group is correctly identified. The
location of the rightmost carbon atom is predicted correctly in the scanning
plane but the disk is too bright, meaning that the vertical position
prediction is wrong. Since the oxygen atom is approximately $1.0$ Å higher
than the deepest carbon atom, the disk representing it is very bright in
comparison and it appears to shadow the nearby deep carbon atom meaning it is
not surprising that the model struggles most in this region. The relative
error is $0.4$ %.

The final simulated example contains a sulphur atom, a hydroxy group and two
carbon 6-rings, one of which has two carbonyl groups (Fig. 2J-M). In the STM
image, the sulphur atom appears as the brightest area (Fig. 2K). This molecule
is particularly tricky for chemical identification due to the carbonyl groups
which could theoretically be hydrogen atoms of a benzene ring, and the sulphur
atom bridging the two carbon structures could be e.g., an oxygen atom forming
an ether group. Regardless of the apparent challenges, the model predicts all
these properties correctly (Fig. 2L). The disks representing oxygen atoms have
a larger radius than the hydrogen atoms of the lower ring, and the sulphur
atom has an even bigger radius distinguishing it from oxygen and carbon atoms.
The carbon chain originating from the sulphur is not considered in the
descriptor due to the depth threshold. Finally, all hydrogen atoms included in
the reference are predicted and the tilting of the molecule is captured in the
prediction. The relative error for this example is $0.6$ %. Overall, the
prediction accuracy is very good on simulated STM images and the trained model
predicts atomic structures of different size, distinguishes various chemical
species and even appears to predict seemingly hidden atoms. However, the final
validation of the model has to be done on experimental data and to this end,
we first consider four benchmark molecules.

Figure 2: Example predictions of simulated STM images of (A-D) C18H13NO2,
(E-H) C8H6FNO3 and (J-M) C13H12O3S molecules from the testing set. Each
molecule has a column for the geometry, STM image, predicted structure and
reference. Scale bar is $2$ Å.

### Experimental validation

The first experimental images we apply ASD-STM on are STM images obtained by
Song et al. 18 that reveal the molecular skeletons of four hydrocarbons in
high detail making them excellent for benchmarking purposes (Fig. 3A, 3D, 3G,
3K). The molecules consist of exclusively carbon and hydrogen atoms
constructing five and six-membered rings barring the fourth example in which a
carbon atom links two ring structures (Fig. 3M). In all STM images, the rings
appear quite distorted, and in many cases it is not immediately clear how many
carbon atoms constitute the particular ring and therefore along with general
structure discovery, we consider our objective to be to distinguish these
rings. The first example consists of nine carbon-rings of which seven are six-
membered. Most of the rings, seven out of nine, are correctly identified in
the prediction (Fig. 3B), characterization of one ring is unclear, and one
ring is misclassified. The model also seems to have learned how rings of
different size connect with each other. Additionally, the prediction suggests
that the bottom and top parts of the molecule are slightly closer to the tip
than the central parts but these predictions are difficult to validate.

The second example is an STM image of a symmetric molecule with eight rings
(Fig. 3D-F). The five-rings are fairly large in the image, and the rings in
all ends of the molecule appear brighter than the center. The prediction is
very good in terms of the carbon structure with all atoms predicted correctly
(Fig. 3E). In this case, even many hydrogen atoms are included in the
prediction. The third example we consider is smaller than the previous
molecules (Fig. 3G-J) and again, the model performs well in predicting the
atomic positions of carbon atoms. In this case, the bottom row of rings
proposes a challenge for the model as the six-rings in the corners are
misclassified as five-rings. A possible explanation for the wrong prediction
is that the rings are almost circular and lack the characteristic corners.
This is supported by the first example which exhibited similar features in the
STM image, and where similar problems were encountered. It is also worth
noting that the STM image contains considerable noise in the form of
horizontal lines but the model succeeds in the prediction regardless. We
attribute this to a successful augmentation process during training phase
where noise and cut-outs were applied to the initially pristine simulated
images.

The final example is different from the others in that it includes two
discrete ring structures that are connected by a carbon atom (Fig. 3K-M). It
is clear that the model struggles with the molecule and does not predict the
correct structure in this case. The rightmost part is mostly correct, although
the model does not seem to fully recognize the rings, as some of the predicted
atoms are blurry and slightly deformed. The leftmost part of the molecule has
more errors, and only the central five-ring is correctly predicted (Fig. 3L).
A general problem with many machine learning models is that their
explainability is poor and it is often difficult to explain the reasons behind
a certain prediction 50. One possible reason would be the particular structure
of the molecule – molecules containing two separate but connected carbon ring
structures are rare in the data set, meaning the model has not had proper
experience and thus fails in the prediction. Overall, the predictions
correspond well to the molecular structures and in total 106 out of 116
($91.4$ %) carbon atoms and 23 out of 31 ($74.2$ %) carbon rings are predicted
correctly.

Figure 3: Predictions from four experimental STM images. (A, D, G, K) The
first column of each example is the STM image used as input for ASD-STM. (B,
E, H, L) The predictions are shown in the second column and (C, F, J, M) the
third column contains the structural formula. Scale bar is $4$ Å. STM images
and structural formulas adapted with permission from 18. ©2020 ACS.

Predictions on the hydrocarbon examples are promising but the molecular
structures were rather simple, and we want to test the limits of the model on
a more challenging example. To this end, we use STM images of the TOAT
molecule (1,5,9-trioxo-13-azatriangulene) (Fig. 4A) 51 and phthalocyanine
(2H-Pc) (Fig. 4D) 21. Predicting the atomic structure of TOAT is particularly
interesting since the simulation software we used to generate the training
data is known for not being able to properly reproduce experimental STM images
of TOAT 39. Regardless, the prediction is in very good agreement with the true
molecular structure (4B). The triangulene backbone is correctly identified and
all oxygen atoms are distinguished from hydrogen atoms, indicated by larger
disks in the prediction. In this example, even most hydrogen atoms are located
with the only exception of the top right ring missing one hydrogen. The disk
representing the center nitrogen seems to have the same radius as the
neighboring carbon atoms suggesting an incorrect prediction but this
classification is inconclusive. Definitive chemical identification has been
explored previously for AFM 23, 24 but for STM it remains a future challenge.

The most challenging example we test here is an STM image of 2H-Pc (Fig. 4D)
and this prediction (Fig. 4E) reveals the limitations of our approach. 2H-Pc
is a cyclic molecule consisting of four isoindole units connected by nitrogen
atoms (Fig. 4F). In this example, a perfect prediction would reveal the
configuration of hydrogen atoms in the central moiety. In the STM image, the
inner five-rings are bright and clearly exhibit the pentagon-like motif
whereas the outer six-rings are less pronounced. The nitrogen atoms connecting
the units appear as cones decaying towards the center. In general, the
molecular skeleton is not very prominent in the image and the signal
originating from the electronic structure is intersecting significantly with
the mechanical bending of the tip apex. It is apparent that these factors
severely affect the ability of ASD-STM to make accurate predictions of the
structure. In the prediction, the five-rings apart from the top left ring are
correctly identified and all nitrogen atoms connecting the rings are located.
The biggest errors are the missing halves of the six-rings and also the
central hydrogens that are not included in the prediction. It is worth noting
that a similar phenomenon was observed in the case of the fourth hydrocarbon
(Fig. 3L) where the top half of a ring was missing in the prediction.

Figure 4: Prediction on experimental of image of the (A-C) TOAT
(1,5,9-trioxo-13-azatriangulene) and (D-F) phthalocyanine (2H-Pc) molecules.
For both molecules the columns contain, from left to right, the STM image,
structure prediction and geometry of the molecule. Location of the pyrrolic
hydrogen atoms in (F) as proposed in 21. Scale bar is $2$ Å. STM images in (A)
reprinted with permission from 51 ©2016 ACS and in (D) from 21 ©2023 ACS.

## Discussion

While predictions on most experimental examples were good, it is clear there
is room for improvement. Since the quality of data included in the training
process is vital to the accuracy of predictions, rare structures affecting the
accuracy should be discussed as it raises the important question of how to
select the molecules in the data set. In the scope of SPM structure discovery,
there are two main approaches to data set generation. The first option, which
this work highlights, is to create a large, diverse and descriptive molecular
data set of various chemical species and structures and to train a versatile
model with the aim of predicting the structure of almost any small organic
molecule. This approach has also been chosen in most previous sample
characterization efforts (e.g. 22, 52) but recently another method has been
used in ice structure discovery 26, 53 where instead of a diverse data set, a
tailored data set is utilized and perfected to make very accurate predictions
possible in a constrained problem domain. That is, if the goal was to only
predict the geometries of different hydrocarbons or triangulene-based
molecules, the model would benefit from a tailored data set with a heavy
emphasis on such structures. As methods for quick generation of molecular
structures to create vast molecular data sets are becoming more readily
available 54, 55, 56, the balance and cost effectiveness of tailored versus
diverse data sets should be the focus of future research.

Second, ASD-STM has been trained on synthetic STM images of molecules in
isolation, without the substrate and as such, the generated image data set
should not be inferred as a source for accurate electronic structures of
adsorbed organic molecules. Moreover, as we know that PPSTM cannot accurately
reproduce STM images of the TOAT molecule, it was expected that ASD-STM would
struggle with the experimental STM image of TOAT. However, the accuracy in the
prediction was excellent which suggests that even though images in the
training set are not entirely representative of true images in terms of
electronic structure, they capture the characteristic sharp lines coming from
CO tip bending accurately, and this seems to be critical for structure
discovery. Also, synthetic AFM images of isolated molecules in general
correspond well to experimental AFM images where the substrate is naturally
present, and since for some samples it is possible to simultaneously gather
STM and AFM signals, the possibility of incorporating AFM data into the
training process should be explored.

Third, we note that the appearance of the molecule in STM is sensitive to
frontier orbitals of the CO tip and consequently to the tip height as at
closer tip-sample distances the tip exhibits a strong $p$-wave character which
diminishes to a more $s$-wave character at longer distances 21. The $s$-wave
character is also increased at higher absolute bias voltages 57. In this work
ASD-STM has been trained using a constant ratio of $s$ and $p$-wave
contributions but the different orbital ratios should be accounted for if ASD-
STM is used for a larger range of tip-sample distances or bias voltages.
Finally, although it is inevitable that the accuracy of predictions on
experimental images is worse than on simulated images, we will investigate
this further in the future with particular focus in improving the robustness
of the model against experimental noise and artifacts.

## Conclusions

This work presents ASD-STM, a method for predicting the atomic structure of a
sample molecule directly from a bond-resolved STM image. We present the
workflow and the software required to generate a STM image data set and to
train the ML model. Images included in the training set were synthesized by
considering the molecules in isolation, and even though the electronic
structure can be affected by the substrate, we showed that the approximation
of molecules in isolation is reasonable for structure discovery on bond-
resolved STM. The model was validated by applying it to experimental images of
six different systems and on most samples the accuracy was good in terms of
atomic structure – additionally we achieved a qualitative level of chemical
identification by distinguishing between hydrogen, carbon, and oxygen atoms.
On the other hand, the example of 2H-Pc clearly demonstrated some of the
challenges. The main limitation of the method is the reliance on the sharp
submolecular features in the images which restrict the method to high quality
images and to a range of short tip-sample distances. While the simulated
images are augmented with noise and cut-outs during training, we anticipate
that these augmentations are not the only difference between simulated and
experimental images and advanced augmentation strategies should be explored to
make the model more robust against experimental conditions. Also, we discussed
possible further improvements to ASD-STM by varying the orbital contributions
in the tunnelling calculation, by tailoring the composition of the initial
data set of molecules to the problem at hand, and by including simultaneously
gathered AFM data in the training process.

Despite the challenges, ASD-STM is readily applicable to STM images of various
small organic molecules and with this work, we open doors for more
sophisticated methods to be developed for STM structure discovery. Finally, as
bond-resolved STM images exhibit similar sub-molecular features as non-contact
AFM images at a significantly reduced acquisition time, ASD-STM demonstrates a
promising start for accelerating molecular structure discovery in SPM in
general.

The authors acknowledge funding from the Academy of Finland (project no.
346824). A.S.F. was supported by the World Premier International Research
Center Initiative (WPI), MEXT, Japan. L.K. acknowledges funding from the
Finnish Cultural Foundation. The authors acknowledge the computational
resources provided by the Aalto Science-IT project and CSC, Helsinki.

Details of the machine learning model.

## References

  * Binnig et al. 1982 Binnig, G.; Rohrer, H.; Gerber, C.; Weibel, E. Tunneling through a controllable vacuum gap. _Appl. Phys. Lett._ 1982, _40_ , 178–180
  * Binnig et al. 1986 Binnig, G.; Quate, C. F.; Gerber, C. Atomic force microscope. _Phys. Rev. Lett._ 1986, _56_ , 930–933
  * Forsburg et al. 1990 Forsburg, S.; Guarente, L. A.; Johnson, P. R. .; Mcknight, S. A.; Mitchell, P. J.; Tjian, R.; Buratowski, S. .; Hahn, S.; Guarente, L.; Sharp, P. . M. W.; Roeder, R. G.; Sawadogo, M.; Buratowski, S.; Hahn, S. .; Sharp, P.; Eisenmann, D. M.; Dollard, C.; Winston, F. Atomic-scale imaging of DNA using scanning tunnelling microscopy. _Nature_ 1990, _346_ , 294–296
  * Pawlak et al. 2019 Pawlak, R.; Vilhena, J. G.; Hinaut, A.; Meier, T.; Glatzel, T.; Baratoff, A.; Gnecco, E.; Pérez, R.; Meyer, E. Conformations and cryo-force spectroscopy of spray-deposited single-strand DNA on gold. _Nat. Commun._ 2019, _10_ , 1–7
  * Dufrêne et al. 2017 Dufrêne, Y. F.; Ando, T.; Garcia, R.; Alsteens, D.; Martinez-Martin, D.; Engel, A.; Gerber, C.; Müller, D. J. Imaging modes of atomic force microscopy for application in molecular and cell biology. _Nat. Nanotechnol._ 2017, _12_ , 295–307
  * Müller and Dufrêne 2008 Müller, D. J.; Dufrêne, Y. F. Atomic force microscopy as a multifunctional molecular toolbox in nanobiotechnology. _Nat. Nanotechnol._ 2008, _3_ , 261–269
  * Gobbi et al. 2018 Gobbi, M.; Orgiu, E.; Samorì, P.; Gobbi, M.; Orgiu, . E.; Samorì, . P. When 2D Materials Meet Molecules: Opportunities and Challenges of Hybrid Organic/Inorganic van der Waals Heterostructures. _Adv. Mater._ 2018, _30_ , 1706103
  * Sun et al. 2022 Sun, K.; Silveira, O. J.; Ma, Y.; Hasegawa, Y.; Matsumoto, M.; Kera, S.; Krejčí, O.; Foster, A. S.; Kawai, S. On-surface synthesis of disilabenzene-bridged covalent organic frameworks. _Nat. Chem._ 2022, _15_ , 136–142
  * Gross et al. 2018 Gross, L.; Schuler, B.; Pavliček, N.; Fatayer, S.; Majzik, Z.; Moll, N.; Peña, D.; Meyer, G. Atomic Force Microscopy for Molecular Structure Elucidation. _Angew. Chem. Int. Ed._ 2018, _57_ , 3888–3908
  * Gross et al. 2009 Gross, L.; Mohn, F.; Moll, N.; Liljeroth, P.; Meyer, G. The chemical structure of a molecule resolved by atomic force microscopy. _Science_ 2009, _325_ , 1110–1114
  * Bartels et al. 1997 Bartels, L.; Meyer, G.; Rieder, K. H. Controlled vertical manipulation of single CO molecules with the scanning tunneling microscope: A route to chemical contrast. _Appl. Phys. Lett._ 1997, _71_ , 213–215
  * Zhong et al. 2020 Zhong, Q.; Li, X.; Zhang, H.; Chi, L. Noncontact atomic force microscopy: Bond imaging and beyond. _Surf. Sci. Rep._ 2020, _75_ , 100509
  * Jelinek 2017 Jelinek, P. High resolution SPM imaging of organic molecules with functionalized tips. _J. Condens. Matter Phys._ 2017, _29_ , 343002
  * Pavliček et al. 2017 Pavliček, N.; Mistry, A.; Majzik, Z.; Moll, N.; Meyer, G.; Fox, D. J.; Gross, L. Synthesis and characterization of triangulene. _Nat. Nanotechnol._ 2017, _12_ , 308–311
  * Seibel et al. 2023 Seibel, J.; Fittolani, G.; Mirhosseini, H.; Wu, X.; Rauschenbach, S.; Anggara, K.; Seeberger, P. H.; Delbianco, M.; Kühne, T. D.; Schlickum, U.; Kern, K. Visualizing Chiral Interactions in Carbohydrates Adsorbed on Au(111) by High-Resolution STM Imaging. _Angew. Chem. Int. Ed._ 2023, _62_ , e202305733
  * Vilas-Varela et al. 2023 Vilas-Varela, M.; Romero-Lara, F.; Vegliante, A.; Calupitan, J. P.; Martínez, A.; Meyer, L.; Uriarte-Amiano, U.; Friedrich, N.; Wang, D.; Schulz, F.; Koval, N. E.; Sandoval-Salinas, M. E.; Casanova, D.; Corso, M.; Artacho, E.; Peña, D.; Pascual, J. I.; Vilas-Varela, M.; Martínez, A.; Peña, D. et al. On-Surface Synthesis and Characterization of a High-Spin Aza-[5]-Triangulene. _Angew. Chem. Int. Ed._ 2023, _62_ , e202307884
  * Martinez-Castro et al. 2022 Martinez-Castro, J.; Bolat, R.; Fan, Q.; Werner, S.; Arefi, H. H.; Esat, T.; Sundermeyer, J.; Wagner, C.; Gottfried, J. M.; Temirov, R.; Ternes, M.; Tautz, F. S. Disentangling the electronic structure of an adsorbed graphene nanoring by scanning tunneling microscopy. _Commun. Mater._ 2022, _3_ , 1–9
  * Song et al. 2020 Song, S.; Guo, N.; Li, X.; Li, G.; Haketa, Y.; Telychko, M.; Su, J.; Lyu, P.; Qiu, Z.; Fang, H.; Peng, X.; Li, J.; Wu, X.; Li, Y.; Su, C.; Koh, M. J.; Wu, J.; Maeda, H.; Zhang, C.; Lu, J. Real-Space Imaging of a Single-Molecule Monoradical Reaction. _J. Am. Chem. Soc._ 2020, _142_ , 13550–13557
  * Weiss et al. 2010 Weiss, C.; Wagner, C.; Temirov, R.; Tautz, F. S. Direct imaging of intermolecular bonds in scanning tunneling microscopy. _J. Am. Chem. Soc._ 2010, _132_ , 11864–11865
  * Gross et al. 2011 Gross, L.; Moll, N.; Mohn, F.; Curioni, A.; Meyer, G.; Hanke, F.; Persson, M. High-Resolution Molecular Orbital Imaging Using a p-Wave STM Tip. _Phys. Rev. Lett._ 2011, _107_ , 86101
  * Néel and Kröger 2023 Néel, N.; Kröger, J. Orbital and Skeletal Structure of a Single Molecule on a Metal Surface Unveiled by Scanning Tunneling Microscopy. _J. Phys. Chem. Lett._ 2023, _14_ , 3946–3952
  * Alldritt et al. 2020 Alldritt, B.; Hapala, P.; Oinonen, N.; Urtev, F.; Krejci, O.; Canova, F. F.; Kannala, J.; Schulz, F.; Liljeroth, P.; Foster, A. S. Automated structure discovery in atomic force microscopy. _Sci. Adv._ 2020, _6_ , eaay6913
  * Carracedo-Cosme et al. 2021 Carracedo-Cosme, J.; Romero-Muñiz, C.; Pérez, R. A Deep Learning Approach for Molecular Classification Based on AFM Images. _Nanomater._ 2021, _11_ , 1658
  * Oinonen et al. 2022 Oinonen, N.; Kurki, L.; Ilin, A.; Foster, A. S. Molecule graph reconstruction from atomic force microscope images with machine learning. _MRS Bull._ 2022, _47_ , 895–905
  * Oinonen et al. 2022 Oinonen, N.; Xu, C.; Alldritt, B.; Canova, F. F.; Urtev, F.; Cai, S.; Krejčí, O.; Kannala, J.; Liljeroth, P.; Foster, A. S. Electrostatic Discovery Atomic Force Microscopy. _ACS Nano_ 2022, _16_ , 89–97
  * Tang et al. 2022 Tang, B.; Song, Y.; Qin, M.; Tian, Y.; Wu, Z. W.; Jiang, Y.; Cao, D.; Xu, L. Machine learning aided atomic structure identification of interfacial ionic hydrates from AFM images. _Natl. Sci. Rev._ 2022, _10_ , nwac282
  * Carracedo-Cosme et al. 2023 Carracedo-Cosme, J.; Romero-Muñiz, C.; Pou, P.; Pérez, R. Molecular Identification from AFM Images Using the IUPAC Nomenclature and Attribute Multimodal Recurrent Neural Networks. _ACS Appl. Mater. Interfaces_ 2023, _15_ , 22692–22704
  * Wang et al. 2020 Wang, C.; Li, H.; Hao, Z.; Li, X.; Zou, C.; Cai, P.; Wang, Y.; You, Y. Z.; Zhai, H. Machine learning identification of impurities in the STM images. _Chin. Physics B_ 2020, _29_ , 116805
  * Yuan et al. 2023 Yuan, S.; Zhu, Z.; Lu, J.; Zheng, F.; Jiang, H.; Sun, Q. Applying a Deep-Learning-Based Keypoint Detection in Analyzing Surface Nanostructures. _Molecules_ 2023, _28_ , 5387
  * Choudhary et al. 2021 Choudhary, K.; Garrity, K. F.; Camp, C.; Kalinin, S. V.; Vasudevan, R.; Ziatdinov, M.; Tavazza, F. Computational scanning tunneling microscope image database. _Sci Data_ 2021, _8_ , 1–9
  * Chen et al. 2022 Chen, I. J.; Aapro, M.; Kipnis, A.; Ilin, A.; Liljeroth, P.; Foster, A. S. Precise atom manipulation through deep reinforcement learning. _Nat. Commun._ 2022, _13_ , 1–8
  * Rashidi and Wolkow 2018 Rashidi, M.; Wolkow, R. A. Autonomous Scanning Probe Microscopy in Situ Tip Conditioning through Machine Learning. _ACS Nano_ 2018, _12_ , 5185–5189
  * Wang et al. 2021 Wang, S.; Zhu, J.; Blackwell, R.; Fischer, F. R. Automated tip conditioning for scanning tunneling spectroscopy. _J. Phys. Chem. A_ 2021, _125_ , 1384–1390
  * Alldritt et al. 2022 Alldritt, B.; Urtev, F.; Oinonen, N.; Aapro, M.; Kannala, J.; Liljeroth, P.; Foster, A. S. Automated tip functionalization via machine learning in scanning probe microscopy. _Comp. Phys. Comp._ 2022, _273_ , 108258
  * Gross 2011 Gross, L. Recent advances in submolecular resolution with scanning probe microscopy. _Nat. Chem._ 2011, _3_ , 273–278
  * Blum et al. 2009 Blum, V.; Gehrke, R.; Hanke, F.; Havu, P.; Havu, V.; Ren, X.; Reuter, K.; Scheffler, M. Ab initio molecular simulations with numeric atom-centered orbitals. _Comp. Phys. Comm._ 2009, _180_ , 2175–2196
  * Havu et al. 2009 Havu, V.; Blum, V.; Havu, P.; Scheffler, M. Efficient O(N) integration for all-electron electronic structure calculation using numeric basis functions. _J. Comput. Phys._ 2009, _228_ , 8367–8379
  * Perdew et al. 1996 Perdew, J. P.; Burke, K.; Ernzerhof, M. Generalized Gradient Approximation Made Simple. _Phys. Rev. Lett._ 1996, _77_ , 3865
  * Krejčí et al. 2017 Krejčí, O.; Hapala, P.; Ondráček, M.; Jelínek, P. Principles and simulations of high-resolution STM imaging with a flexible tip apex. _Phys. Rev. B_ 2017, _95_
  * Bardeen 1961 Bardeen, J. Tunnelling from a Many-Particle Point of View. _Phys. Rev. Lett._ 1961, _6_ , 57
  * Torre et al. 2017 Torre, B. D. L.; Švec, M.; Foti, G.; Krejčí, O.; Hapala, P.; Garcia-Lekue, A.; Frederiksen, T.; Zbořil, R.; Arnau, A.; Vázquez, H.; Jelínek, P. Submolecular Resolution by Variation of the Inelastic Electron Tunneling Spectroscopy Amplitude and its Relation to the AFM/STM Signal. _Phys. Rev. Lett._ 2017, _119_ , 166001
  * Hapala et al. 2014 Hapala, P.; Kichin, G.; Wagner, C.; Tautz, F. S.; Temirov, R.; Jelínek, P. Mechanism of high-resolution STM/AFM imaging with functionalized tips. _Phys. Rev. B_ 2014, _90_ , 1–9
  * Hapala et al. 2014 Hapala, P.; Temirov, R.; Tautz, F. S.; Jelínek, P. Origin of high-resolution IETS-STM images of organic molecules with functionalized tips. _Phys. Rev. Lett._ 2014, _113_ , 226101
  * Giessibl 2001 Giessibl, F. J. A direct method to calculate tip-sample forces from frequency shifts in frequency-modulation atomic force microscopy. _Appl. Phys. Lett._ 2001, _78_ , 123–125
  * Oktay et al. 2018 Oktay, O.; Schlemper, J.; Folgoc, L. L.; Lee, M.; Heinrich, M.; Misawa, K.; Mori, K.; McDonagh, S.; Hammerla, N. Y.; Kainz, B.; Glocker, B.; Rueckert, D. Attention U-Net: Learning Where to Look for the Pancreas. 2018; arXiv:1804.03999
  * Xu et al. 2015 Xu, B.; Wang, N.; Kong, H.; Chen, T.; Li, M. Empirical Evaluation of Rectified Activations in Convolutional Network. 2015; arXiv:1505.00853v2
  * Ioffe and Szegedy 2015 Ioffe, S.; Szegedy, C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. _ICML_ 2015, _1_ , 448–456
  * Paszke et al. 2019 Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; Desmaison, A.; Köpf, A.; Yang, E.; DeVito, Z.; Raison, M.; Tejani, A.; Chilamkurthy, S.; Steiner, B.; Fang, L.; Bai, J. et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. _NeurIPS_ 2019, _32_
  * Kingma and Ba 2015 Kingma, D. P.; Ba, J. L. Adam: A method for stochastic optimization. 2015; arXiv:1412.6980v9
  * Linardatos et al. 2021 Linardatos, P.; Papastefanopoulos, V.; Kotsiantis, S. Explainable AI: A Review of Machine Learning Interpretability Methods. _Entropy_ 2021, _23_ , 1–45
  * Heijden et al. 2016 Heijden, N. J. V. D.; Hapala, P.; Rombouts, J. A.; Lit, J. V. D.; Smith, D.; Mutombo, P.; Švec, M.; Jelinek, P.; Swart, I. Characteristic Contrast in $\Delta$fmin Maps of Organic Molecules Using Atomic Force Microscopy. _ACS Nano_ 2016, _10_ , 8517–8525
  * Carracedo-Cosme et al. 2022 Carracedo-Cosme, J.; Iz, C. R.-M.; Pou, P.; Pérez, R. QUAM-AFM: A Free Database for Molecular Identification by Atomic Force Microscopy. _J. Chem. Inf. Model_ 2022, _62_ , 38
  * Priante et al. 2023 Priante, F.; Oinonen, N.; Tian, Y.; Guan, D.; Xu, C.; Cai, S.; Liljeroth, P.; Jiang, Y.; Foster, A. S. Structure discovery in Atomic Force Microscopy imaging of ice. 2023; arXiv:2310.17161v1
  * Bilodeau et al. 2022 Bilodeau, C.; Jin, W.; Jaakkola, T.; Barzilay, R.; Jensen, K. F. Generative models for molecular discovery: Recent advances and challenges. _Wiley Interdiscip. Rev. Comput. Mol. Sci._ 2022, _12_ , e1608
  * Hoogeboom et al. 2022 Hoogeboom, E.; Satorras, V. G.; Vignac, C.; Welling, M. Equivariant Diffusion for Molecule Generation in 3D. 2022; arXiv:2203.17003
  * Daigavane et al. 2023 Daigavane, A.; Kim, S.; Geiger, M.; Smidt, T. Symphony: Symmetry-Equivariant Point-Centered Spherical Harmonics for Molecule Generation. 2023; arXiv:2311.16199
  * Pavliček et al. 2013 Pavliček, N.; Swart, I.; Niedenführ, J.; Meyer, G.; Repp, J. Symmetry dependence of vibration-assisted tunneling. _Phys. Rev. Lett._ 2013, _110_ , 136101