# Warwick Electron Microscopy Datasets

Jeffrey M. Ede  University of Warwick, Department of Physics, Coventry, CV4
7AL, UK<EMAIL_ADDRESS>
###### Abstract

Large, carefully partitioned datasets are essential to train neural networks
and standardize performance benchmarks. As a result, we have set up new
repositories to make our electron microscopy datasets available to the wider
community. There are three main datasets containing 19769 scanning
transmission electron micrographs, 17266 transmission electron micrographs,
and 98340 simulated exit wavefunctions, and multiple variants of each dataset
for different applications. To visualize image datasets, we trained
variational autoencoders to encode data as 64-dimensional multivariate normal
distributions, which we cluster in two dimensions by t-distributed stochastic
neighbor embedding. In addition, we have improved dataset visualization with
variational autoencoders by introducing encoding normalization and
regularization, adding an image gradient loss, and extending t-distributed
stochastic neighbor embedding to account for encoded standard deviations. Our
datasets, source code, pretrained models, and interactive visualizations are
openly available at https://github.com/Jeffrey-Ede/datasets.

##  1 Introduction

We have set up new repositories[1] to make our large new electron microscopy
datasets available to both electron microscopists and the wider community.
There are three main datasets containing 19769 experimental scanning
transmission electron microscopy[2] (STEM) images, 17266 experimental
transmission electron microscopy[2] (TEM) images and 98340 simulated TEM exit
wavefunctions[3]. Experimental datasets represent general research and were
collected by dozens of University of Warwick scientists working on hundreds of
projects between January 2010 and June 2018. We have been using our datasets
to train artificial neural networks (ANNs) for electron microscopy[4, 5, 3, 6,
7], where standardizing results with common test sets has been essential for
comparison. This paper provides details of and visualizations for datasets and
their variants, and is supplemented by source code, pretrained models, and
both static and interactive visualizations[8].

Machine learning is increasingly being applied to materials science[9, 10],
including to electron microscopy[11]. Encouraging scientists, ANNs are
universal approximators[12] that can leverage an understanding of physics to
represent[13] the best way to perform a task with arbitrary accuracy. In
theory, this means that ANNs can always match or surpass the performance of
contemporary methods. However, training, validating and testing requires
large, carefully partitioned datasets[14, 15] to ensure that ANNs are robust
to general use. To this end, our datasets are partitioned so that each subset
has different characteristics. For example, TEM or STEM images can be
partitioned so that subsets are collected by different scientists, and
simulated exit wavefunction partitions can correspond to Crystallography
Information Files[16] (CIFs) for materials published in different journals.

Most areas of science are facing a reproducibility crisis[17], including
artificial intelligence[18]. Adding to this crisis, natural scientists do not
always benchmark ANNs against standardized public domain test sets; making
results difficult or impossible to compare. In electron microscopy, we believe
this is a symptom of most datasets being small, esoteric or not having default
partitions for machine learning. For example, most datasets in the Electron
Microscopy Public Image Archive[19, 20] are for specific materials and are not
partitioned. In contrast, standard machine learning datasets such as
CIFAR-10[21, 22], MNIST[23], and ImageNet[24] have default partitions for
machine learning and contain tens of thousands or millions of examples. By
publishing our large, carefully partitioned machine learning datasets, and
setting an example by using them to standardize our research, we aim to
encourage higher standardization of machine learning research in the electron
microscopy community.

There are many popular algorithms for high-dimensional data visualization[25,
26, 27, 28, 29, 30, 31, 32] that can map $N$ high-dimensional vectors of
features
$\\{\textbf{x}_{1},...,\textbf{x}_{N}\\},\;\textbf{x}_{i}\in\mathbb{R}^{u}$ to
low-dimensional vectors
$\\{\textbf{y}_{1},...,\textbf{y}_{N}\\},\;\textbf{y}_{i}\in\mathbb{R}^{v}$. A
standard approach for data clustering in $v\in\\{1,2,3\\}$ dimensions is
t-distributed stochastic neighbor embedding[33, 34] (tSNE). To embed data by
tSNE, Kullback-Leibler (KL) divergence,

$L_{\text{tSNE}}=\sum\limits_{i}\sum\limits_{j\neq
i}p_{ij}\log\left(\frac{p_{ij}}{q_{ij}}\right)\,,$ (1)

is minimized by gradient descent[35] for normally distributed pairwise
similarities in real space, $p_{ij}$, and heavy-tailed Student t-distributed
pairwise similarities in an embedding space, $q_{ij}$. For symmetric tSNE[33],

$\displaystyle p_{i|j}$
$\displaystyle=\frac{\exp\left(-||\textbf{x}_{i}-\textbf{x}_{j}||_{2}^{2}/2\alpha_{j}^{2}\right)}{\sum\limits_{k\neq
j}\exp\left(-||\textbf{x}_{k}-\textbf{x}_{j}||_{2}^{2}/2\alpha_{j}^{2}\right)}\,,$
(2) $\displaystyle p_{ij}$ $\displaystyle=\frac{p_{i|j}+p_{j|i}}{2N}\,,$ (3)
$\displaystyle q_{ij}$
$\displaystyle=\frac{\left(1+||\textbf{y}_{i}-\textbf{y}_{j}||_{2}^{2}\right)^{-1}}{\sum\limits_{k\neq
i}\left(1+||\textbf{y}_{k}-\textbf{y}_{i}||_{2}^{2}\right)^{-1}}\,.$ (4)

To control how much tSNE clusters data, perplexities of $p_{i|j}$ for
$j\in\\{1,...,N\\}$ are adjusted to a user-provided value by fitting
$\alpha_{j}$. Perplexity, $\exp(H)$, is an exponential function of entropy,
$H$, and most tSNE visualizations are robust to moderate changes to its value.

Feature extraction is often applied to decrease input dimensionality,
typically to $u\lesssim 100$, before clustering data by tSNE. Decreasing input
dimensionality can decrease data noise and computation for large datasets, and
is necessary for some high-dimensional data as distances,
$||\textbf{x}_{i}-\textbf{x}_{j}||_{2}$, used to compute $p_{ij}$ are affected
by the curse of dimensionality[36]. For image data, a standard approach[33] to
extract features is probabilistic[37, 38] or singular value decomposition[39]
(SVD) based principal component analysis[40] (PCA). However, PCA is limited to
linearly separable features. Other hand-crafted feature extraction methods
include using a histogram of oriented gradients[41], speeded-up robust
features[42], local binary patterns[43], wavelet decomposition[44] and other
methods[45]. The best features to extract for a visualization depend on its
purpose. However, most hand-crafted feature extraction methods must be tuned
for different datasets. For example, Minka’s algorithm[46] is included in the
scikit-learn[47] implementation of PCA by SVD to obtain optimal numbers of
principal components to use.

To increase representation power, nonlinear and dataset-specific features can
be extracted with deep learning. For example, by using the latent space of an
autoencoder[48, 49] (AE) or features before logits in a classification
ANN[50]. Indeed, we have posted AEs for electron microscopy with pre-trained
models[51, 52] that could be improved. However, AE latent vectors can exhibit
inhomogeneous dimensional characteristics and pathological semantics, limiting
correlation between latent features and semantics. To encode well-behaved
latent vectors suitable for clustering by tSNE, variational autoencoders[53,
54] (VAEs) can be trained to encode data as multivariate probability
distributions. For example, VAEs are often regularized to encode multivariate
normal distributions by adding KL divergence of encodings from a standard
normal distribution to its loss function[53]. The regularization homogenizes
dimensional characteristics and sampling noise correlates semantics with
latent features.

##  2 Dataset Visualization

To visualize datasets presented in this paper, we trained VAEs shown in figure
1 to embed 96$\times$96 images in $u=64$ dimensions before clustering in $v=2$
dimensions by tSNE. Our VAE consists of two convolutional neural networks[55,
56] (CNNs): an encoder and a generator. The encoder embeds batches of $B$
input images, $I$, as mean vectors,
$\\{\boldsymbol{\mu}_{1},...,\boldsymbol{\mu}_{B}\\}$, and standard deviation
vectors, $\\{\boldsymbol{\sigma}_{1},...,\boldsymbol{\sigma}_{B}\\}$, to
parameterize multivariate normal distributions. During training, input images
are linearly transformed to have minimum and maximum values of 0 and 1,
respectively, and we apply a random combination of flips and 90$\degree$
rotations to augment training data by a factor of eight. The generator, $G$,
is trained to cooperate with the encoder to output encoder inputs by sampling
latent vectors,
$\textbf{z}_{i}=\boldsymbol{\mu}_{i}+\boldsymbol{\sigma}_{i}\boldsymbol{\epsilon}_{i}$,
where $\boldsymbol{\mu}_{i}=\\{\mu_{i1},...,\mu_{iu}\\}$,
$\boldsymbol{\sigma}_{i}=\\{\sigma_{i1},...,\sigma_{iu}\\}$, and
$\boldsymbol{\epsilon}_{i}=\\{\epsilon_{i1},...,\epsilon_{iu}\\}$ are random
variates sampled from standard normal distributions, $\epsilon_{ij}\sim
N(0,1)$. Each convolutional or fully connected layer is followed by batch
normalization[57] then ReLU[58] activation, except the output layers of the
encoder and generator. An absolute nonlinearity, $f(x)=|x|$, is applied to
encode positive standard deviations.

Figure 1:  Simplified VAE architecture. a) An encoder outputs means,
$\boldsymbol{\mu}$, and standard deviations, $\boldsymbol{\sigma}$, to
parameterize multivariate normal distributions,
$\textbf{z}\sim\text{N}(\boldsymbol{\mu},\boldsymbol{\sigma})$. b) A generator
predicts input images from z.  Figure 2:  Images at 500 randomly selected
images in two-dimensional tSNE visualizations of 19769 96$\times$96 crops from
STEM images for various embedding methods. Clustering is best in a) and gets
worse in order a)$\rightarrow$b)$\rightarrow$c)$\rightarrow$d).

Traditional VAEs are trained to optimize a balance, $\lambda_{\text{MSE}}$,
between mean squared errors (MSEs) of generated images and KL divergence of
encodings from a multivariate standard normal distribution[53],

$L_{\text{trad}}=\lambda_{\text{MSE}}\text{MSE}(G(\textbf{z}),I)+\frac{1}{2Bu}\sum\limits_{i=1}^{B}\sum\limits_{j=1}^{u}\mu_{ij}^{2}+\sigma_{ij}^{2}-\log(\sigma_{ij}^{2})-1.$
(5)

However, traditional VAE training is sensitive to $\lambda_{\text{MSE}}$[59]
and other hyperparameters[60]. If $\lambda_{\text{MSE}}$ is too low, the
encoder will learn learn to consistently output $\sigma_{ij}\simeq 1$,
limiting regularization. Else if $\lambda_{\text{MSE}}$ is too high, the
encoder will learn to output $\sigma_{ij}\ll|\mu_{ij}|$, limiting
regularization. As a result, traditional VAE hyperparameters must be carefully
tuned for different ANN architectures and datasets. To improve VAE
regularization and robustness to different datasets, we normalize encodings
parameterizing normal distributions to

$\displaystyle\mu_{ij}$
$\displaystyle\leftarrow\frac{\lambda_{\mu}(\mu_{ij}-\mu_{\text{avg},j})}{\mu_{\text{std},j}}\,,$
(6) $\displaystyle\sigma_{ij}$
$\displaystyle\leftarrow\frac{\sigma_{ij}}{2\sigma_{\text{std},j}}\,,$ (7)

where batch means and standard deviations are

$\displaystyle\mu_{\text{avg},j}$
$\displaystyle=\frac{1}{B}\sum\limits_{k=1}^{B}\mu_{kj}\,,$ (8)
$\displaystyle\mu_{\text{std},j}^{2}$
$\displaystyle=\frac{1}{B}\sum\limits_{k=1}^{B}\mu_{kj}^{2}-\left(\frac{1}{B}\sum\limits_{k=1}^{B}\mu_{kj}\right)^{2}\,,$
(9) $\displaystyle\sigma_{\text{std},j}^{2}$
$\displaystyle=\frac{1}{B}\sum\limits_{k=1}^{B}\sigma_{kj}^{2}-\left(\frac{1}{B}\sum\limits_{k=1}^{B}\sigma_{kj}\right)^{2}\,.$
(10)

Encoding normalization is a modified form of batch normalization[57] for VAE
latent spaces. As part of encoding normalization, we introduce a new
hyperparameter, $\lambda_{\mu}$, to scale the ratio of expectations
$\text{E}(|\mu_{ij}|)/\text{E}(|\sigma_{ij}|)$. We use $\lambda_{\mu}=2.5$ in
this paper; however, we confirm that training is robust to values
$\lambda_{\mu}\in\\{1.0,2.0,2.5\\}$ for a range of datasets and ANN
architectures. Batch means are subtracted from $\boldsymbol{\mu}$ and not
$\boldsymbol{\sigma}$ so that $\sigma_{ij}\geq 0$. In addition, we multiply
$\sigma_{\text{std},j}$ by an arbitrary factor of 2 so that
$\text{E}(|\mu_{ij}|)\approx\text{E}(|\sigma_{ij}|)$ for $\lambda_{\mu}=1$.

Encoding normalization enables the KL divergence loss in equation 5 to be
removed as latent space regularization is built into the encoder architecture.
However, we find that removing the KL loss can result in VAEs encoding either
very low or very high $\sigma_{ij}$. In effect, an encoder can learn to use
$\boldsymbol{\sigma}$ apply a binary mask to $\boldsymbol{\mu}$ if a generator
learns that latent features with very high absolute values are not meaningful.
To prevent extreme $\sigma_{ij}$, we add a new encoding regularization loss,
$\text{MSE}(\boldsymbol{\sigma},1)$, to the encoder. Human vision is sensitive
to edges[61], so we also add a gradient-based loss to improve realism. Adding
a gradient-based loss is a computationally inexpensive alternative to training
a variational autoencoder generative adversarial network[62] (VAE-GAN) and
often achieves similar performance. Our total training loss is

$L=\lambda_{\text{MSE}}\text{MSE}(G(\textbf{z}),I)+\lambda_{\text{Sobel}}\text{MSE}(S(G(\textbf{z})),S(I))+\text{MSE}(\boldsymbol{\sigma},1)\,,$
(11)

where we chose $\lambda_{\text{MSE}}=\lambda_{\text{Sobel}}=50$, and $S(x)$
computes a concatenation of horizontal and vertical Sobel derivatives[63] of
$x$. We found that training is robust to choices of
$\lambda_{\text{MSE}}=\lambda_{\text{Sobel}}$ where
$\lambda_{\text{MSE}}\text{MSE}(G(\textbf{z}),I)+\lambda_{\text{Sobel}}\text{MSE}(S(G(\textbf{z})),S(I))$
is in $[0.5,25.0]$, and have not investigated losses outside this interval.

We trained VAEs to minimize $L$ by ADAM[64] optimized stochastic gradient
descent[35, 65]. At training iteration $t\in[1,T]$, we used a stepwise
exponentially decayed learning rate[66],

$\eta=\eta_{\text{start}}a^{\text{floor}(bt/T)}\,,$ (12)

and a DEMON[67] first moment of the momentum decay rate,

$\beta_{1}=\frac{\beta_{\text{start}}(1-t/T)}{(1-\beta_{\text{start}})+\beta_{\text{start}}(1-t/T)}\,,$
(13)

where we chose initial values $\eta_{\text{start}}=0.001$ and
$\beta_{\text{start}}=0.9$, exponential base $a=0.5$, $b=8$ steps, and
$T=600000$ iterations. We used a batch size of $B=64$ and emphasize that a
large batch size decreases complication of encoding normalization by varying
batch statistics. Training our VAEs takes about 12 hours on a desktop computer
with an Nvidia GTX 1080 Ti GPU and an Intel i7-6700 CPU.

To use VAE latent spaces to cluster data, means are often embedded by tSNE.
However, this does not account for highly varying $\sigma$ used to calculate
latent features. To account for uncertainty, we modify calculation of pairwise
similarities, $p_{ij}$, in equation 2 to include both $\boldsymbol{\mu}_{i}$
and $\boldsymbol{\sigma}_{i}$ encoded for every example, $i\in[1,N]$, in our
datasets,

$\displaystyle p_{i|j}$
$\displaystyle=\exp\left(-\frac{1}{2\alpha_{j}^{2}}\sum\limits_{k}w_{ijk}(\mu_{ik}-\mu_{jk})^{2}\right)\left(\sum\limits_{m\neq
j}\exp\left(-\frac{1}{2\alpha_{j}^{2}}\sum\limits_{k}w_{mjk}(\mu_{mk}-\mu_{jk})^{2}\right)\right)^{-1}\,,$
(14)

where we chose weights

$\displaystyle w_{ijk}$
$\displaystyle=\frac{1}{\sigma_{ik}^{2}+\sigma_{jk}^{2}+\epsilon}\left(\sum\limits_{l}\frac{1}{\sigma_{il}^{2}+\sigma_{jl}^{2}+\epsilon}\right)^{-1}\,.$
(15)

We add $\epsilon=0.01$ for numerical stability, and to account for uncertainty
in $\boldsymbol{\sigma}$ due to encoder imperfections or variation in batch
statistics. Following Oskolkov[68], we fit $\alpha_{j}$ to perplexities given
by $N^{1/2}$, where $N$ is the number of examples in a dataset, and confirm
that changing perplexities by $\pm 100$ has little effect on visualizations
for our $N\simeq 20000$ TEM and STEM datasets. To ensure convergence, we run
tSNE computations for 10000 iterations. In comparison, KL divergence is stable
by 5000 iterations for our datasets. In preliminary experiments, we observe
that tSNE with $\boldsymbol{\sigma}$ results in comparable visualizations to
tSNE without $\boldsymbol{\sigma}$, and think that tSNE with
$\boldsymbol{\sigma}$ may be a slight improvement. For comparison, pairs of
visualizations with and without $\boldsymbol{\sigma}$ are indicated in
supplementary information.

Our improvements to dataset visualization by tSNE are showcased in figure 2
for various embedding methods. The visualizations are for a new dataset
containing 19769 96$\times$96 crops from STEM images, which will be introduced
in section 3. To suppress high-frequency noise during training, images were
blurred by a 5$\times$5 symmetric Gaussian kernel with a 2.5 px standard
deviation. Clusters are most distinct in figure 2a for encoding normalized VAE
training with a gradient loss described by equation 11. Ablating the gradient
loss in figure 2b results in similar clustering; however, the VAE struggles to
separate images of noise and fine atom columns. In contrast, clusters are not
clearly separated in figure 2c for a traditional VAE described by equation 5.
Finally, embedding the first 50 principal components extracted by a scikit-
learn[69] implementation of probabilistic PCA in figure 2d does not result in
clear clustering.

Figure 3:  Two-dimensional tSNE visualization of 64-dimensional VAE latent spaces for 19769 STEM images that have been downsampled to 96$\times$96\. The same grid is used to show a) map points and b) images at 500 randomly selected points.  \begin{overpic}[scale={1.3}]{0-0.png} \put(50.0,34.0){\makebox[0.0pt]{Dark Field Atom Columns \cite[cite]{[\@@bibref{}{van2016unscrambling}{}{}]}}} \end{overpic} |  \begin{overpic}[scale={1.3}]{0-1.png} \put(50.0,34.0){\makebox[0.0pt]{Bright Field Atom Columns \cite[cite]{[\@@bibref{}{zhou2016sample}{}{}]}}} \end{overpic}
---|---
\begin{overpic}[scale={1.3}]{0-2.png} \put(50.0,34.0){\makebox[0.0pt]{Nanowires \cite[cite]{[\@@bibref{}{bu2016surface}{}{}]}}} \end{overpic} |  \begin{overpic}[scale={1.3}]{0-3.png} \put(50.0,34.0){\makebox[0.0pt]{Atomic Resolution Bands}} \end{overpic}
\begin{overpic}[scale={1.3}]{0-4.png} \put(50.0,34.0){\makebox[0.0pt]{Incomplete Scans}} \end{overpic} |  \begin{overpic}[scale={1.3}]{0-5.png} \put(50.0,34.0){\makebox[0.0pt]{Multilayer Materials \cite[cite]{[\@@bibref{}{monclus2018effect}{}{}]}}} \end{overpic}
\begin{overpic}[scale={1.3}]{0-6.png} \put(50.0,34.0){\makebox[0.0pt]{Atomic Boundaries \cite[cite]{[\@@bibref{}{pyrz2010atomic}{}{}]}}} \end{overpic} |  \begin{overpic}[scale={1.3}]{0-7.png} \put(50.0,34.0){\makebox[0.0pt]{Lacey Carbon Supports \cite[cite]{[\@@bibref{}{mcgilvery2012contamination}{}{}]}}} \end{overpic}
Table 1:  Examples and descriptions of STEM images in our datasets. References
put some images into context to make them more tangible to unfamiliar readers.

##  3 Scanning Transmission Electron Micrographs

We curated 19769 STEM images from University of Warwick electron microscopy
dataservers to train ANNs for compressed sensing[5, 7]. Atom columns are
visible in roughly two-thirds of images, and similar proportions are bright
and dark field. In addition, most signals are noisy[76] and are imaged at
several times their Nyquist rates[77]. To reduce data transfer times for large
images, we also created variant containing 161069 non-overlapping
512$\times$512 crops from full images. For rapid development, we have also
created new variants containing 96$\times$96 images downsampled or cropped
from full images. In this section we give details of each STEM dataset,
referring to them using their names in our repositories.

STEM Full Images: 19769 32-bit TIFFs containing STEM images taken with a
University of Warwick JEOL ARM 200F electron microscope by dozens of
scientists working on hundreds of projects. Images were originally saved in
DigitalMicrograph DM3 or DM4 files created by Gatan Microscopy Suite[78]
software and have their original sizes and intensities. The dataset is
partitioned into 14826 training, 1977 validation, and 2966 test set images.
The dataset was made by concatenating contributions from different scientists,
so partitioning the dataset before shuffling also partitions scientists.

STEM Crops: 161069 32-bit TIFFs containing 512$\times$512 non-overlapping
regions cropped from STEM Full Images. The dataset is partitioned into 110933
training, 21259 validation, and 28877 test set images. This dataset is biased
insofar that larger images were divided into more crops.

STEM 96$\times$96: A 32-bit NumPy[79, 80] array with shape [19769, 96, 96, 1]
containing 19769 STEM Full Images area downsampled to 96$\times$96 with MATLAB
and default antialiasing.

STEM 96$\times$96 Crops: A 32-bit NumPy array with shape [19769, 96, 96, 1]
containing 19769 96$\times$96 regions cropped from STEM Full Images. Each crop
is from a different image.

Variety of STEM 96$\times$96 images is shown in figure 3 by clustering means
and standard deviations of VAE latent spaces in two dimensions by tSNE.
Details are in section 2. An interactive visualization that displays images
when map points are hovered over is also available[8]. This paper is aimed at
a general audience so readers may not be familiar with STEM. Subsequently,
example images are tabulated with references and descriptions in table 1 to
make them more tangible.

Figure 4:  Two-dimensional tSNE visualization of 64-dimensional VAE latent spaces for 17266 TEM images that have been downsampled to 96$\times$96\. The same grid is used to show a) map points and b) images at 500 randomly selected points.  \begin{overpic}[scale={1.3}]{1-0.png} \put(50.0,34.0){\makebox[0.0pt]{Lacey Carbon Supports \cite[cite]{[\@@bibref{}{karlsson2001thickness}{}{}]}}} \end{overpic} |  \begin{overpic}[scale={1.3}]{1-1.png} \put(50.0,34.0){\makebox[0.0pt]{Apertures Blocking Electrons}} \end{overpic}
---|---
\begin{overpic}[scale={1.3}]{1-2.png} \put(50.0,34.0){\makebox[0.0pt]{Block Copolymers \cite[cite]{[\@@bibref{}{inam20171d}{}{}]}}} \end{overpic} |  \begin{overpic}[scale={1.3}]{1-3.png} \put(50.0,34.0){\makebox[0.0pt]{Diffraction Patterns \cite[cite]{[\@@bibref{}{bendersky2001electron}{}{}]}}} \end{overpic}
\begin{overpic}[scale={1.3}]{1-4.png} \put(50.0,34.0){\makebox[0.0pt]{Vacuum at Specimen Edges}} \end{overpic} |  \begin{overpic}[scale={1.3}]{1-5.png} \put(50.0,34.0){\makebox[0.0pt]{Nanowires \cite[cite]{[\@@bibref{}{wu2001superconducting}{}{}]}}} \end{overpic}
\begin{overpic}[scale={1.3}]{1-6.png} \put(50.0,34.0){\makebox[0.0pt]{Multilayer Materials \cite[cite]{[\@@bibref{}{pang2017microstructural}{}{}]}}} \end{overpic} |  \begin{overpic}[scale={1.3}]{1-7.png} \put(50.0,34.0){\makebox[0.0pt]{Particles \cite[cite]{[\@@bibref{}{dong2016individual}{}{}]}}} \end{overpic}
Table 2:  Examples and descriptions of TEM images in our datasets. References
put some images into context to make them more tangible to unfamiliar readers.

##  4 Transmission Electron Micrographs

We curated 17266 2048$\times$2048 high-signal TEM images from University of
Warwick electron microscopy dataservers to train ANNs to improve signal-to-
noise[4]. However, our dataset was only available upon request. It is now
openly available[1]. For convenience, we have also created a new variant
containing 96$\times$96 images that can be used for rapid ANN development. In
this section we give details of each TEM dataset, referring to them using
their names in our repositories.

TEM Full Images: 17266 32-bit TIFFs containing 2048$\times$2048 TEM images
taken with University of Warwick JEOL 2000, JEOL 2100, JEOL 2100+, and JEOL
ARM 200F electron microscope by dozens of scientists working on hundreds of
projects. Images were originally saved in DigitalMicrograph DM3 or DM4 files
created by Gatan Microscopy Suite[78] software and have been cropped to
largest possible squares and area resized to 2048$\times$2048 with MATLAB and
default antialiasing. Images with at least 2500 electron counts per pixel were
then linearly transformed to have minimum and maximum values of 0 and 1,
respectively. We discarded images with less than 2500 electron counts per
pixel as images were curated to train an electron micrograph denoiser[4]. The
dataset is partitioned into 11350 training, 2431 validation, and 3486 test set
images. The dataset was made by concatenating contributions from different
scientists, so each partition contains data collected by a different subset of
scientists.

TEM 96$\times$96: A 32-bit NumPy array with shape [17266, 96, 96, 1]
containing 17266 TEM Full Images area downsampled to 96$\times$96 with MATLAB
and default antialiasing. Training, validation, and test set images are
concatenated in that order.

Variety of TEM 96$\times$96 images is shown in figure 4 by clustering means
and standard deviations of VAE latent spaces in two dimensions by tSNE.
Details are in section 2. An interactive visualization that displays images
when map points are hovered over is also available[8]. This paper is aimed at
a general audience so readers may not be familiar with TEM. Subsequently,
example images are tabulated with references and descriptions in table 2 to
make them more tangible.

##  5 Exit Wavefunctions

We simulated 98340 TEM exit wavefunctions to train ANNs to reconstruct phases
from amplitudes[3]. Half of wavefunction information is undetected by
conventional TEM as only the amplitude, and not the phase, of an image is
recorded. Wavefunctions were simulated at 512$\times$512 then centre-cropped
to 320$\times$320 to remove simulation edge artefacts. Wavefunctions have been
simulated for real physics where Kirkland potentials[87] for each atom are
summed from $n=3$ terms, and by truncating Kirkland potential summations to
$n=1$ to simulate an alternative universe where atoms have different
potentials. Wavefunctions simulated for an alternate universe can be used to
test ANN robustness to simulation physics. For rapid development, we also
downsampled $n=3$ wavefunctions from 320$\times$320 to 96$\times$96\. In this
section we give details of each exit wavefunction dataset, referring to them
using their names in our repositories.

CIFs: 12789 CIFs downloaded from the Crystallography Open Database[88, 89, 90,
91, 92, 93] (COD). The CIFs are for materials published in inorganic chemistry
journals. There are 150 New Journal of Chemistry, 1034 American Mineralogist,
1998 Journal of the American Chemical Society and 5457 Inorganic Chemistry
CIFs used to simulate training set wavefunctions, 1216 Physics and Chemistry
of Materials CIFs used to simulate validation set wavefunctions, and 2927
Chemistry of Materials CIFs used to simulate test set wavefunctions. In
addition, the CIFs have been preprocessed to be input to clTEM wavefunction
simulations.

URLs: COD Uniform Resource Locators[94] (URLs) that CIFs were downloaded from.

Wavefunctions: 36324 complex 64-bit NumPy files containing 320$\times$320
wavefunctions. The wavefunctions are for a large range of materials and
physical hyperparameters. The dataset is partitioned into 24530 training, 3399
validation, and 8395 test set wavefunctions. Metadata Javascript Object
Notation[95] (JSON) files link wavefunctions to CIFs and contain some
simulation hyperparameters.

Wavefunctions Unseen Training: 1544 64-bit NumPy files containing
320$\times$320 wavefunctions. The wavefunctions are for training set CIFs and
are for a large range of materials and physical hyperparameters. Metadata
JSONs link wavefunctions to CIFs and contain some simulation hyperparameters.

Wavefunctions Single: 4825 complex 64-bit NumPy files containing
320$\times$320 wavefunctions. The wavefunctions are for a single material,
In1.7K2Se8Sn2.28[96], and a large range of physical hyperparameters. The
dataset is partitioned into 3861 training, and 964 validation set
wavefunctions. Metadata JSONs link wavefunctions to CIFs and contain some
simulation hyperparameters.

Wavefunctions Restricted: 11870 complex 64-bit NumPy files containing
320$\times$320 wavefunctions. The wavefunctions are for a large range of
materials and a small range of physical hyperparameters. The dataset is
partitioned into 8002 training, 1105 validation, and 2763 test set
wavefunctions. Metadata JSON files link wavefunctions to CIFs and contain some
simulation hyperparameters.

Wavefunctions 96$\times$96: A 32-bit NumPy array with shape [36324, 96, 96, 2]
containing 36324 wavefunctions. The wavefunctions were simulated for a large
range of materials and physical hyperparameters, and bilinearly downsampled
with skimage[47] from 320$\times$320 to 96$\times$96 using default
antialiasing. In Python[97], Real components are at index […,0], and imaginary
components are at index […,1]. The dataset can be partitioned in 24530
training, 3399 validation, and 8395 test set wavefunctions, which have been
concatenated in that order. To be clear, the training subset is at Python
indexes [:24530].

Wavefunctions 96$\times$96 Restricted: A 32-bit NumPy array with shape [11870,
96, 96, 2] containing 11870 wavefunctions. The wavefunctions were simulated
for a large range of materials and a small range of physical hyperparameters,
and bilinearly downsampled with skimage from 320$\times$320 to 96$\times$96
using default antialiasing. The dataset can be partitioned in 8002 training,
1105 validation, and 2763 test set wavefunctions, which have been concatenated
in that order.

Wavefunctions 96$\times$96 Single: A 32-bit NumPy array with shape [4825, 96,
96, 2] containing 11870 wavefunctions. The wavefunctions were simulated for
In1.7K2Se8Sn2.28 and a large range of physical hyperparameters, and bilinearly
downsampled with skimage from 320$\times$320 to 96$\times$96 using default
antialiasing. The dataset can be partitioned in 3861 training, and 964
validation set wavefunctions, which have been concatenated in that order.

Wavefunctions $n=1$: 37457 complex 64-bit NumPy files containing
320$\times$320 wavefunctions. The wavefunctions are for a large range of
materials and physical hyperparameters. The dataset is partitioned into 25352
training, 3569 validation, and 8563 test set wavefunctions. These
wavefunctions are for an alternate universe where atoms have different
potentials.

Wavefunctions $n=1$ Unseen Training: 1501 64-bit NumPy files containing
320$\times$320 wavefunctions. The wavefunctions are for training set CIFs and
are for a large range of materials and physical hyperparameters. Metadata
JSONs link wavefunctions to CIFs and contain some simulation hyperparameters.
These wavefunctions are for an alternate universe where atoms have different
potentials.

Wavefunctions $n=1$ Single: 4819 complex 64-bit NumPy files containing
320$\times$320 wavefunctions. The wavefunctions are for a single material,
In1.7K2Se8Sn2.28, and a large range of physical hyperparameters. The dataset
is partitioned into 3856 training, and 963 validation set wavefunctions.
Metadata JSONs link wavefunctions to CIFs and contain some simulation
hyperparameters. These wavefunctions are for an alternate universe where atoms
have different potentials.

Experimental Focal Series: 1000 experimental focal series. Each series
consists of 14 32-bit 512$\times$512 TEM images, area downsampled from
4096$\times$4096 with MATLAB and default antialiasing. The images are in
TIFF[98] format. All series were created with a common, quadratically
increasing[99] defocus series. However, spatial scales vary and would need to
be fitted as part of wavefunction reconstruction.

In detail, exit wavefunctions for a large range of physical hyperparameters
were simulated with clTEM[100, 101] for acceleration voltages in
$\\{80,200,300\\}$ kV, material depths uniformly distributed in $[5,100)$ nm,
material widths in $[5,10)$ nm, and crystallographic zone axes $(h,k,l)$
$h,k,l\in\\{0,1,2\\}$. Materials were padded on all sides with vacuum 0.8 nm
wide and 0.3 nm deep to reduce simulation artefacts. Finally, crystal tilts
were perturbed by zero-centred Gaussian random variates with 0.1$\degree$
standard deviations. We used default values for other clTEM hyperparameters.
Simulations for a small range of physical hyperparameters used lower upper
bounds that reduced simulation hyperparameter ranges by factors close to 1/4.
All wavefunctions are linearly transformed to have a mean amplitude of 1.

All wavefunctions show atom columns, so tSNE visualizations are provided in
supplementary information to conserve space. The visualizations are for
Wavefunctions 96$\times$96, Wavefunctions 96$\times$96 Restricted and
Wavefunctions 96$\times$96 Single.

##  6 Discussion

The best dataset variant varies for different applications. Full-sized
datasets can always be used as other dataset variants are derived from them.
However, loading and processing full-sized examples may bottleneck training,
and it is often unnecessary. Instead, smaller 512$\times$512 crops, which can
be loaded more quickly the full-sized images, can often be used to train ANNs
to be applied convolutionally[102] to or tiled across[4] full-sized inputs. In
addition, our 96$\times$96 datasets can be used for rapid initial development
before scaling up to full-sized datasets, similar to how ANNs might be trained
with CIFAR-10 before scaling up to ImageNet. However, subtle application- and
dataset-specific considerations may also influence the best dataset choice.
For example, an ANN trained with downsampled 96$\times$96 inputs may not
generalize to 96$\times$96 crops from full-sized inputs as downsampling may
introduce artifacts[103] and change noise or other data characteristics.

In practice, electron microscopists image most STEM and TEM signals at several
times their Nyquist rates[77]. This eases visual inspection, decreases sub-
Nyquist aliasing[104], improves display on computer monitors, and is easier
than carefully tuning sampling rates to capture the minimum data needed to
resolve signals. High sampling may also reveal additional high-frequency
information when images are inspected after an experiment. However, this
complicates ANN development as it means that information per pixel is often
higher in downsampled images. For example, partial scans across STEM images
that have been dowsampled to 96$\times$96 require higher coverages than scans
across 96$\times$96 crops for ANNs to learn to complete images with equal
performance[5]. It also complicates the comparison of different approaches to
compressed sensing. For example, we suggested that sampling 512$\times$512
crops at a regular grid of probing locations outperforms sampling along spiral
paths as a subsampling grid can still access most information[5].

Test set performance should be calculated for a standardized dataset partition
to ease comparison with other methods. Nevertheless, training and validation
partitions can be varied to investigate validation variance for partitions
with different characteristics. Default training and validation sets for STEM
and TEM datasets contain contributions from different scientists that have
been concatenated or numbered in order, so new validation partitions can be
selected by concatenating training and validation partitions and moving the
window used to select the validation set. Similarly, exit wavefunctions were
simulated with CIFs from different journals that were concatenated or numbered
sequentially. There is leakage[105, 106] between training, validation and test
sets due to overlap between materials published in different journals and
between different scientists’ work. However, further leakage can be minimized
by selecting dataset partitions before any shuffling and, for wavefunctions,
by ensuring that simulations for each journal are not split between
partitions.

Experimental STEM and TEM image quality is variable. Images were taken by
scientists with all levels of experience and TEM images were taken on multiple
microscopes. This means that our datasets contain images that might be omitted
from other datasets. For example, the tSNE visualization for STEM in figure 3
includes incomplete scans, $\sim$50 blank images, and images that only contain
noise. Similarly, the tSNE visualization for TEM in figure 4 revealed some
images where apertures block electrons, and that there are small number of
unprocessed standard diffraction and convergent beam electron diffraction[107]
patterns. Although these conventionally low-quality images would not normally
be published, they are important to ensure that ANNs are robust for live
applications. In addition, inclusion of conventionally low-quality images may
enable identification of this type of data. We encourage readers to try our
interactive tSNE visualizations[8] for detailed inspection of our datasets.

In this paper, we present tSNE visualizations of VAE latent spaces to show
image variety. However, our VAEs can be directly applied to a wide range of
additional applications. For example, successful tSNE clustering of latent
spaces suggests that VAEs could be used to create a hash table[108, 109] for
an electron micrograph search engine. VAEs can also be applied to semantic
manipulation[110], and clustering in tSNE visualizations may enable subsets of
latent space that generate interesting subsets of data distributions to be
identified. Other applications include using clusters in tSNE visualizations
to label data for supervised learning, data compression, and anomaly
detection[111, 112]. To encourage further development, we have made our source
code and pretrained VAEs openly available[8].

##  7 Conclusion

We have presented details of and visualizations for large new electron
microscopy datasets that are openly available from our new repositories.
Datasets have been carefully partitioned into training, validation, and test
sets for machine learning. Further, we provide variants containing
512$\times$512 crops to reduce data loading times, and examples downsampled to
96$\times$96 for rapid development. To improve dataset visualization with
VAEs, we introduce encoding normalization and regularization, and add an image
gradient loss. In addition, we propose extending tSNE to account for encoded
standard deviations. Source code, pretrained VAEs, precompiled tSNE binaries,
and interactive dataset visualizations are provided in supplementary
repositories to help users become familiar with our datasets and
visualizations. By making our datasets available, we aim to encourage
standardization of performance benchmarks in electron microscopy and increase
participation of the wider computer science community in electron microscopy
research.

##  8 Supplementary Information

Note: Content in this section will be provided in a supplementary document
with the published version of this preprint.

Ten additional tSNE visualizations are provided as supplementary information.
They are for:

  * •

Extracting 50 principal components by probabilistic PCA for the STEM
96$\times$96, STEM 96$\times$96 Crops, TEM 96$\times$96, Wavefunctions
96$\times$96, Wavefunctions 96$\times$96 Restricted and Wavefunctions
96$\times$96 Single datasets. PCA is a quick and effective method to extract
features. As a result, we think that visualizations for PCA are interesting
benchmarks.

  * •

VAE latent spaces with $\boldsymbol{\sigma}$ propagation for the STEM
96$\times$96 Crops dataset. Crops show smaller features than downsampled
images.

  * •

VAE latent spaces without $\boldsymbol{\sigma}$ propagation for the STEM
96$\times$96, STEM 96$\times$96 Crops and TEM 96$\times$96 datasets. They are
comparable to visualizations created with $\boldsymbol{\sigma}$ propagation.

Interactive versions of tSNE visualizations that display data when map points
are hovered over are available[8] for every figure. In addition, we propose an
algorithm to increase whitespace utilization in tSNE visualizations by
uniformly separating points.

##  9 Data Availability

The data that support the findings of this study are openly available at
https://doi.org/10.5281/zenodo.3834197. For additional information contact the
corresponding author (J.M.E.).

## References

  * [1] Ede, J. M.  Electron Microscopy Datasets.  Online: https://github.com/Jeffrey-Ede/datasets/wiki (2020).
  * [2] FEI Company.  An Introduction to Electron Microscopy.  Online: https://www.fei.com/documents/introduction-to-microscopy-document (2010).
  * [3] Ede, J. M., Peters, J. J. P., Sloan, J. & Beanland, R.  Exit Wavefunction Reconstruction from Single Transmission Electron Micrographs with Deep Learning.  _arXiv preprint arXiv:2001.10938_ (2020).
  * [4] Ede, J. M. & Beanland, R.  Improving Electron Micrograph Signal-to-Noise with an Atrous Convolutional Encoder-Decoder.  _Ultramicroscopy_ 202, 18–25, DOI: 10.1016/j.ultramic.2019.03.017 (2019).
  * [5] Ede, J. M. & Beanland, R.  Partial Scanning Transmission Electron Microscopy with Deep Learning.  _Scientific Reports_ 10, DOI: 10.1038/s41598-020-65261-0 (2020).
  * [6] Ede, J. M. & Beanland, R.  Adaptive Learning Rate Clipping Stabilizes Learning.  _Machine Learning: Science and Technology_ 1, 015011, DOI: 10.1088/2632-2153/ab81e2 (2020).
  * [7] Ede, J. M.  Deep Learning Supersampled Scanning Transmission Electron Microscopy.  _arXiv preprint arXiv:1910.10467_ (2019).
  * [8] Ede, J. M.  Visualization of Electron Microscopy Datasets with Deep Learning.  Online: https://github.com/Jeffrey-Ede/datasets, DOI: 10.5281/zenodo.3834197 (2020).
  * [9] Schmidt, J., Marques, M. R., Botti, S. & Marques, M. A.  Recent Advances and Applications of Machine Learning in Solid-State Materials Science.  _npj Computational Materials_ 5, 1–36 (2019).
  * [10] von Lilienfeld, O. A.  Introducing Machine Learning: Science and Technology.  _Machine Learning: Science and Technology_ 1, 010201 (2020).
  * [11] Belianinov, A. _et al._ Big Data and Deep Data in Scanning and Electron Microscopies: Deriving Functionality from Multidimensional Data Sets.  _Advanced Structural and Chemical Imaging_ 1, 1–25 (2015).
  * [12] Hornik, K., Stinchcombe, M. & White, H.  Multilayer Feedforward Networks are Universal Approximators.  _Neural Networks_ 2, 359–366 (1989).
  * [13] Lin, H. W., Tegmark, M. & Rolnick, D.  Why does Deep and Cheap Learning Work so Well?  _Journal of Statistical Physics_ 168, 1223–1247 (2017).
  * [14] Raschka, S.  Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning.  _arXiv preprint arXiv:1811.12808_ (2018).
  * [15] Roh, Y., Heo, G. & Whang, S. E.  A Survey on Data Collection for Machine Learning: A Big Data-AI Integration Perspective.  _IEEE Transactions on Knowledge and Data Engineering_ (2019).
  * [16] Hall, S. R., Allen, F. H. & Brown, I. D.  The Crystallographic Information File (CIF): A New Standard Archive File for Crystallography.  _Acta Crystallographica Section A: Foundations of Crystallography_ 47, 655–685 (1991).
  * [17] Baker, M.  Reproducibility Crisis?  _Nature_ 533, 353–66 (2016).
  * [18] Hutson, M.  Artificial Intelligence Faces Reproducibility Crisis (2018).
  * [19] Iudin, A., Korir, P. K., Salavert-Torres, J., Kleywegt, G. J. & Patwardhan, A.  EMPIAR: A Public Archive for Raw Electron Microscopy Image Data.  _Nature Methods_ 13, 387 (2016).
  * [20] Hey, T., Butler, K., Jackson, S. & Thiyagalingam, J.  Machine Learning and Big Scientific Data.  _Philosophical Transactions of the Royal Society A_ 378, 20190054 (2020).
  * [21] Krizhevsky, A., Nair, V. & Hinton, G.  The CIFAR-10 Dataset.  Online: http://www.cs.toronto.edu/k̃riz/cifar.html (2014).
  * [22] Krizhevsky, A. & Hinton, G.  Learning Multiple Layers of Features from Tiny Images.  Tech. Rep., Citeseer (2009).
  * [23] LeCun, Y., Cortes, C. & Burges, C.  MNIST Handwritten Digit Database.  AT&T Labs, Online: http://yann.lecun.com/exdb/mnist (2010).
  * [24] Russakovsky, O. _et al._ ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision_ 115, 211–252 (2015).
  * [25] Tenenbaum, J. B., De Silva, V. & Langford, J. C.  A Global Geometric Framework for Nonlinear Dimensionality Reduction.  _Science_ 290, 2319–2323 (2000).
  * [26] Roweis, S. T. & Saul, L. K.  Nonlinear Dimensionality Reduction by Locally Linear Embedding.  _science_ 290, 2323–2326 (2000).
  * [27] Zhang, Z. & Wang, J.  MLLE: Modified Locally Linear Embedding Using Multiple Weights.  In _Advances in Neural Information Processing Systems_ , 1593–1600 (2007).
  * [28] Donoho, D. L. & Grimes, C.  Hessian Eigenmaps: Locally Linear Embedding Techniques for High-Dimensional Data.  _Proceedings of the National Academy of Sciences_ 100, 5591–5596 (2003).
  * [29] Belkin, M. & Niyogi, P.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation.  _Neural Computation_ 15, 1373–1396 (2003).
  * [30] Zhang, Z. & Zha, H.  Principal Manifolds and Nonlinear Dimensionality Reduction via Tangent Space Alignment.  _SIAM Journal on Scientific Computing_ 26, 313–338 (2004).
  * [31] Buja, A. _et al._ Data Visualization with Multidimensional Scaling.  _Journal of Computational and Graphical Statistics_ 17, 444–472 (2008).
  * [32] Van Der Maaten, L.  Accelerating t-SNE Using Tree-Based Algorithms.  _The Journal of Machine Learning Research_ 15, 3221–3245 (2014).
  * [33] Maaten, L. v. d. & Hinton, G.  Visualizing Data Using t-SNE.  _Journal of Machine Learning Research_ 9, 2579–2605 (2008).
  * [34] Wattenberg, M., Viégas, F. & Johnson, I.  How to Use t-SNE Effectively.  _Distill_ 1, e2 (2016).
  * [35] Ruder, S.  An Overview of Gradient Descent Optimization Algorithms.  _arXiv preprint arXiv:1609.04747_ (2016).
  * [36] Schubert, E. & Gertz, M.  Intrinsic t-Stochastic Neighbor Embedding for Visualization and Outlier Detection.  In _International Conference on Similarity Search and Applications_ , 188–203 (Springer, 2017).
  * [37] Halko, N., Martinsson, P.-G. & Tropp, J. A.  Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions.  _SIAM Review_ 53, 217–288 (2011).
  * [38] Martinsson, P.-G., Rokhlin, V. & Tygert, M.  A Randomized Algorithm for the Decomposition of Matrices.  _Applied and Computational Harmonic Analysis_ 30, 47–68 (2011).
  * [39] Wall, M. E., Rechtsteiner, A. & Rocha, L. M.  Singular Value Decomposition and Principal Component Analysis.  In _A Practical Approach to Microarray Data Analysis_ , 91–109 (Springer, 2003).
  * [40] Jolliffe, I. T. & Cadima, J.  Principal Component Analysis: A Review and Recent Developments.  _Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences_ 374, 20150202 (2016).
  * [41] Dalal, N. & Triggs, B.  Histograms of Oriented Gradients for Human Detection.  In _2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)_ , vol. 1, 886–893 (IEEE, 2005).
  * [42] Bay, H., Ess, A., Tuytelaars, T. & Van Gool, L.  Speeded-Up Robust Features (SURF).  _Computer Vision and Image Understanding_ 110, 346–359 (2008).
  * [43] Ojala, T., Pietikainen, M. & Maenpaa, T.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns.  _IEEE Transactions on pattern analysis and machine intelligence_ 24, 971–987 (2002).
  * [44] Mallat, S. G.  A Theory for Multiresolution Signal Decomposition: The Wavelet Representation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 11, 674–693 (1989).
  * [45] Latif, A. _et al._ Content-Based Image Retrieval and Feature Extraction: A Comprehensive Review.  _Mathematical Problems in Engineering_ 2019 (2019).
  * [46] Minka, T. P.  Automatic Choice of Dimensionality for PCA.  In _Advances in Neural Information Processing Systems_ , 598–604 (2001).
  * [47] Van der Walt, S. _et al._ scikit-image: Image Processing in Python.  _PeerJ_ 2, e453 (2014).
  * [48] Tschannen, M., Bachem, O. & Lucic, M.  Recent Advances in Autoencoder-Based Representation Learning.  _arXiv preprint arXiv:1812.05069_ (2018).
  * [49] Kramer, M. A.  Nonlinear Principal Component Analysis Using Autoassociative Neural Networks.  _AIChE Journal_ 37, 233–243 (1991).
  * [50] Marcelino, P.  Transfer Learning From Pre-trained Models.  Towards Data Science, online: https://towardsdatascience.com/transfer-learning-from-pre-trained-models-f2393f124751 (2018).
  * [51] Ede, J. M.  Kernels, MLPs and Autoencoders.  Online: https://github.com/Jeffrey-Ede/Denoising-Kernels-MLPs-Autoencoders (2018).
  * [52] Ede, J. M.  Autoencoders, Kernels, and Multilayer Perceptrons for Electron Micrograph Restoration and Compression.  _arXiv preprint arXiv:1808.09916_ (2018).
  * [53] Kingma, D. P. & Welling, M.  Auto-Encoding Variational Bayes.  _arXiv preprint arXiv:1312.6114_ (2014).
  * [54] Kingma, D. P. & Welling, M.  An Introduction to Variational Autoencoders.  _arXiv preprint arXiv:1906.02691_ (2019).
  * [55] McCann, M. T., Jin, K. H. & Unser, M.  Convolutional Neural Networks for Inverse Problems in Imaging: A Review.  _IEEE Signal Processing Magazine_ 34, 85–95 (2017).
  * [56] Krizhevsky, A., Sutskever, I. & Hinton, G. E.  ImageNet Classification with Deep Convolutional Neural Networks.  In _Advances in Neural Information Processing Systems_ , 1097–1105 (2012).
  * [57] Ioffe, S. & Szegedy, C.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.  _arXiv preprint arXiv:1502.03167_ (2015).
  * [58] Nair, V. & Hinton, G. E.  Rectified Linear Units Improve Restricted Boltzmann Machines.  In _Proceedings of the 27th International Conference on Machine Learning (ICML-10)_ , 807–814 (2010).
  * [59] Higgins, I. _et al._ beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework.  _International Conference on Learning Representations_ 2, 6 (2017).
  * [60] Hu, Q. & Greene, C. S.  Parameter Tuning is a Key Part of Dimensionality Reduction via Deep Variational Autoencoders for Single Cell RNA Transcriptomics.  In _Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing_ , vol. 24, 362 (NIH Public Access, 2019).
  * [61] McIlhagga, W.  Estimates of Edge Detection Filters in Human Vision.  _Vision research_ 153, 30–36 (2018).
  * [62] Larsen, A. B. L., Sønderby, S. K., Larochelle, H. & Winther, O.  Autoencoding Beyond Pixels using a Learned Similarity Metric.  _arXiv preprint arXiv:1512.09300_ (2015).
  * [63] Vairalkar, M. K. & Nimbhorkar, S.  Edge Detection of Images using Sobel Operator.  _International Journal of Emerging Technology and Advanced Engineering_ 2, 291–293 (2012).
  * [64] Kingma, D. P. & Ba, J.  ADAM: A Method for Stochastic Optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * [65] Zou, D., Cao, Y., Zhou, D. & Gu, Q.  Stochastic Gradient Descent Optimizes Over-Parameterized Deep ReLU Networks.  _arXiv preprint arXiv:1811.08888_ (2018).
  * [66] Ge, R., Kakade, S. M., Kidambi, R. & Netrapalli, P.  The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure.  _arXiv preprint arXiv:1904.12838_ (2019).
  * [67] Chen, J. & Kyrillidis, A.  Decaying Momentum Helps Neural Network Training.  _arXiv preprint arXiv:1910.04952_ (2019).
  * [68] Oskolkov, N.  How to Tune Hyperparameters of tSNE.  Towards Data Science, Online: https://towardsdatascience.com/how-to-tune-hyperparameters-of-tsne-7c0596a18868 (2019).
  * [69] Pedregosa, F. _et al._ Scikit-learn: Machine Learning in Python.  _Journal of Machine Learning Research_ 12, 2825–2830 (2011).
  * [70] Van den Bos, K. H. _et al._ Unscrambling Mixed Elements Using High Angle Annular Dark Field Scanning Transmission Electron Microscopy.  _Physical Review Letters_ 116, 246101 (2016).
  * [71] Zhou, D. _et al._ Sample Tilt Effects on Atom Column Position Determination in ABF-STEM Imaging.  _Ultramicroscopy_ 160, 110–117 (2016).
  * [72] Bu, L. _et al._ Surface Engineering of Hierarchical Platinum-Cobalt Nanowires for Efficient Electrocatalysis.  _Nature Communications_ 7, 1–10 (2016).
  * [73] Monclús, M. _et al._ Effect of Layer Thickness on the Mechanical Behaviour of Oxidation-Strengthened Zr/Nb Nanoscale Multilayers.  _Journal of Materials Science_ 53, 5860–5878 (2018).
  * [74] Pyrz, W. D. _et al._ Atomic-Level Imaging of Mo-VO Complex Oxide Phase Intergrowth, Grain Boundaries, and Defects Using HAADF-STEM.  _Proceedings of the National Academy of Sciences_ 107, 6152–6157 (2010).
  * [75] McGilvery, C. M., Goode, A. E., Shaffer, M. S. & McComb, D. W.  Contamination of Holey/Lacey Carbon Films in STEM.  _Micron_ 43, 450–455 (2012).
  * [76] Seki, T., Ikuhara, Y. & Shibata, N.  Theoretical Framework of Statistical Noise in Scanning Transmission Electron Microscopy.  _Ultramicroscopy_ 193, 118–125 (2018).
  * [77] Landau, H.  Sampling, Data Transmission, and the Nyquist Rate.  _Proceedings of the IEEE_ 55, 1701–1706 (1967).
  * [78] Gatan Microscopy Suite.  Online: www.gatan.com/products/tem-analysis/gatan-microscopy-suite-software (2019).
  * [79] NPY Format.  Online: https://docs.scipy.org/doc/numpy/reference/generated/numpy.lib.format.html (2019).
  * [80] Kern, R.  NEP 1 — A Simple File Format for NumPy Arrays.  Online: https://numpy.org/neps/nep-0001-npy-format.html (2007).
  * [81] Karlsson, G.  Thickness Measurements of Lacey Carbon Films.  _Journal of Microscopy_ 203, 326–328 (2001).
  * [82] Inam, M. _et al._ 1D vs. 2D Shape Selectivity in the Crystallization-Driven Self-Assembly of Polylactide Block Copolymers.  _Chemical Science_ 8, 4223–4230 (2017).
  * [83] Bendersky, L. A. & Gayle, F. W.  Electron Diffraction Using Transmission Electron Microscopy.  _Journal of Research of the National Institute of Standards and Technology_ 106, 997 (2001).
  * [84] Wu, Y., Messer, B. & Yang, P.  Superconducting MgB2 Nanowires.  _Advanced Materials_ 13, 1487–1489 (2001).
  * [85] Pang, B. _et al._ The Microstructural Characterization of Multiferroic LaFeO3-YMnO3 Multilayers Grown on (001)-and (111)-SrTiO3 Substrates by Transmission Electron Microscopy.  _Materials_ 10, 839 (2017).
  * [86] Dong, Z. _et al._ Individual Particles of Cryoconite Deposited on the Mountain Glaciers of the Tibetan Plateau: Insights into Chemical Composition and Sources.  _Atmospheric Environment_ 138, 114–124 (2016).
  * [87] Kirkland, E. J.  _Advanced Computing in Electron Microscopy_ (Springer Science & Business Media, 2010).
  * [88] Quirós, M., Gražulis, S., Girdzijauskaitė, S., Merkys, A. & Vaitkus, A.  Using SMILES Strings for the Description of Chemical Connectivity in the Crystallography Open Database.  _Journal of Cheminformatics_ 10, DOI: 10.1186/s13321-018-0279-6 (2018).
  * [89] Merkys, A. _et al._ COD::CIF::Parser: An Error-Correcting CIF Parser for the Perl Language.  _Journal of Applied Crystallography_ 49, DOI: 10.1107/S1600576715022396 (2016).
  * [90] Gražulis, S., Merkys, A., Vaitkus, A. & Okulič-Kazarinas, M.  Computing Stoichiometric Molecular Composition From Crystal Structures.  _Journal of Applied Crystallography_ 48, 85–91, DOI: 10.1107/S1600576714025904 (2015).
  * [91] Gražulis, S. _et al._ Crystallography Open Database (COD): An Open-Access Collection of Crystal Structures and Platform for World-Wide Collaboration.  _Nucleic Acids Research_ 40, D420–D427, DOI: 10.1093/nar/gkr900 (2012).  http://nar.oxfordjournals.org/content/40/D1/D420.full.pdf+html.
  * [92] Gražulis, S. _et al._ Crystallography Open Database – An Open-Access Collection of Crystal Structures.  _Journal of Applied Crystallography_ 42, 726–729, DOI: 10.1107/S0021889809016690 (2009).
  * [93] Downs, R. T. & Hall-Wallace, M.  The American Mineralogist Crystal Structure Database.  _American Mineralogist_ 88, 247–250 (2003).
  * [94] Berners-Lee, T., Masinter, L. & McCahill, M.  RFC1738: Uniform Resource Locators (URL).  _RFC_ DOI: 10.17487/RFC1738 (1994).
  * [95] ISO/IEC JTC 1/SC 22.  International Standard ISO/IEC21778: Information Technology - The JSON Data Interchange Syntax.  Online: https://www.iso.org/standard/71616.html (2017).
  * [96] Hwang, S.-J., Iyer, R. G., Trikalitis, P. N., Ogden, A. G. & Kanatzidis, M. G.  Cooling of Melts: Kinetic Stabilization and Polymorphic Transitions in the KInSnSe4 System.  _Inorganic chemistry_ 43, 2237–2239 (2004).
  * [97] Python Software Foundation.  Python 3.6.  Online: http://www.python.org (2020).
  * [98] Adobe Developers Association _et al._ TIFF Revision 6.0.  Online: www.adobe.io/content/dam/udp/en/open/standards/tiff/TIFF6.pdf (1992).
  * [99] Haigh, S., Jiang, B., Alloyeau, D., Kisielowski, C. & Kirkland, A.  Recording Low and High Spatial Frequencies in Exit Wave Reconstructions.  _Ultramicroscopy_ 133, 26–34 (2013).
  * [100] Peters, J. J. P. & Dyson, M. A.  clTEM.  Online: https://github.com/JJPPeters/clTEM (2019).
  * [101] Dyson, M. A.  _Advances in Computational Methods for Transmission Electron Microscopy Simulation and Image Processing_.  Ph.D. thesis, University of Warwick (2014).
  * [102] Zhu, J.-Y., Park, T., Isola, P. & Efros, A. A.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 2223–2232 (2017).
  * [103] MicroImages.  Resampling Methods.  Technical Guide, Online: https://www.microimages.com/documentation/TechGuides/77resampling.pdf (2010).
  * [104] Amidror, I.  Sub-Nyquist Artefacts and Sampling Moiré Effects.  _Royal Society Open Science_ 2, 140550 (2015).
  * [105] Open Data Science.  How to Fix Data Leakage - Your Model’s Greatest Enemy.  Towards Data Science, online: https://medium.com/@ODSC/how-to-fix-data-leakage-your-models-greatest-enemy-e34fa26abac5 (2019).
  * [106] Bussola, N., Marcolini, A., Maggio, V., Jurman, G. & Furlanello, C.  Not Again! Data Leakage in Digital Pathology.  _arXiv preprint arXiv:1909.06539_ (2019).
  * [107] Tanaka, M.  Convergent-Beam Electron Diffraction.  _Acta Crystallographica Section A: Foundations of Crystallography_ 50, 261–286 (1994).
  * [108] Patterson, N. & Wang, Y.  Semantic Hashing with Variational Autoencoders.
  * [109] Jin, G., Zhang, Y. & Lu, K.  Deep Hashing Based on VAE-GAN for Efficient Similarity Retrieval.  _Chinese Journal of Electronics_ 28, 1191–1197 (2019).
  * [110] Klys, J., Snell, J. & Zemel, R.  Learning Latent Subspaces in Variational Autoencoders.  In _Advances in Neural Information Processing Systems_ , 6444–6454 (2018).
  * [111] Yao, R., Liu, C., Zhang, L. & Peng, P.  Unsupervised Anomaly Detection Using Variational Auto-Encoder based Feature Extraction.  In _2019 IEEE International Conference on Prognostics and Health Management (ICPHM)_ , 1–7 (IEEE, 2019).
  * [112] Xu, H. _et al._ Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications.  In _Proceedings of the 2018 World Wide Web Conference_ , 187–196 (2018).

##  10 Acknowledgements

Funding: J.M.E. acknowledges EPSRC EP/N035437/1 and EPSRC Studentship 1917382.

##  11 Competing Interests

The author declares no competing interests.