# Partial Scanning Transmission Electron Microscopy with Deep Learning

Jeffrey M. Ede  University of Warwick, Department of Physics, Coventry, CV4
7AL, UK<EMAIL_ADDRESS>Richard Beanland  University of Warwick,
Department of Physics, Coventry, CV4 7AL, UK

###### Abstract

Compressed sensing algorithms are used to decrease electron microscope scan
time and electron beam exposure with minimal information loss. Following
successful applications of deep learning to compressed sensing, we have
developed a two-stage multiscale generative adversarial neural network to
complete realistic 512$\times$512 scanning transmission electron micrographs
from spiral, jittered gridlike, and other partial scans. For spiral scans and
mean squared error based pre-training, this enables electron beam coverage to
be decreased by 17.9$\times$ with a 3.8% test set root mean squared intensity
error, and by 87.0$\times$ with a 6.2% error. Our generator networks are
trained on partial scans created from a new dataset of 16227 scanning
transmission electron micrographs. High performance is achieved with adaptive
learning rate clipping of loss spikes and an auxiliary trainer network. Our
source code, new dataset, and pre-trained models have been made publicly
available at https://github.com/Jeffrey-Ede/partial-STEM.

##  1 Introduction

Aberration corrected scanning transmission electron microscopy (STEM) can
achieve imaging resolutions below 0.1 nm, and locate atom columns with pm
precision[1, 2]. Nonetheless, the high current density of electron probes
produces radiation damage in many materials, limiting the range and type of
investigations that can be performed[3, 4]. A number of strategies to minimize
beam damage have been proposed, including dose fractionation[5] and a variety
of sparse data collection methods[6]. Perhaps the most intensively
investigated approach to the latter is sampling a random subset of pixels,
followed by reconstruction using an inpainting algorithm[7, 8, 9, 4, 10, 6].
Poisson random sampling of pixels is optimal for reconstruction by compressed
sensing algorithms[11]. However, random sampling exceeds the design parameters
of standard electron beam deflection systems, and can only be performed by
collecting data slowly[12, 13], or with the addition of a fast deflection or
blanking system[4, 14].

Sparse data collection methods that are more compatible with conventional beam
deflection systems have also been investigated. For example, maintaining a
linear fast scan deflection whilst using a widely-spaced slow scan axis with
some small random ‘jitter’[12, 9]. However, even small jumps in electron beam
position can lead to a significant difference between nominal and actual beam
positions in a fast scan. Such jumps can be avoided by driving functions with
continuous derivatives, such as those for spiral and Lissajous scan paths[15,
13, 16, 4]. Sang[13, 16] considered a variety of scans including Archimedes
and Fermat spirals, and scans with constant angular or linear displacements,
by driving electron beam deflectors with a field-programmable gate array
(FPGA) based system. Spirals with constant angular velocity place the least
demand on electron beam deflectors. However, dwell times, and therefore
electron dose, decreases with radius. Conversely, spirals created with with
constant spatial speeds are prone to systematic image distortions due to lags
in deflector responses. In practice, fixed doses are preferable as they
simplify visual inspection and limit the dose dependence of STEM noise[17].

Deep learning has a history of successful applications to image infilling,
including image completion[18], irregular gap infilling[19] and
supersampling[20]. This has motivated applications of deep learning to the
completion of sparse, or ‘partial’, scans, including supersampling of scanning
electron microscopy[21] (SEM) and STEM images[22, 23]. Where pre-trained
models are unavailable for transfer learning[24], artificial neural networks
(ANNs) are typically trained, validated and tested with large, carefully
partitioned machine learning datasets[25, 26] so that they are robust to
general use. In practice, this often requires at least a few thousand
examples. Indeed, standard machine learning datasets such as CIFAR-10[27, 28],
MNIST[29], and ImageNet[30] contain tens of thousands or millions of examples.
To train an ANN to complete STEM images from partial scans, an ideal dataset
might consist of a large number of pairs of partial scans and corresponding
high-quality, low noise images, taken with an aberration-corrected STEM. Such
a dataset does not exist. As a result, we have collated a new dataset of STEM
raster scans from which partial scans can be selected. Selecting partial scans
from full scans is less expensive than collecting image pairs, and individual
pixels selected from experimental images have realistic noise characteristics.

Examples of spiral and jittered gridlike partial scans investigated in this
paper are shown in Fig. 1. Continuous spiral scan paths that extend to image
corners cannot be created by conventional scan systems without going over
image edges. However, such a spiral can be cropped from a spiral with radius
at least $2^{-1/2}$ times the minimum image side, at the cost of increased
scan time and electron beam damage to the surrounding material. We use
Archimedes spirals, where $r\propto\theta$, and $r$ and $\theta$ are polar
radius and angle coordinates, as these spirals have the most uniform spatial
coverage. Jittered gridlike scans would also be difficult to produce with a
conventional system, which would suffer variations in dose and distortions due
to limited beam deflector response. Nevertheless, these idealized scan paths
serve as useful inputs to demonstrate the capabilities of our approach. We
expect that other scan paths could be used with similar results.

Figure 1: Examples of Archimedes spiral (top) and jittered gridlike (bottom)
512$\times$512 partial scan paths for 1/10, 1/20, 1/40, and 1/100 px coverage.

We fine-tune our ANNs as part of generative adversarial networks[31] (GANs) to
complete realistic images from partial scans. A GAN consists of sets of
generators and discriminators that play an adversarial game. Generators learn
to produce outputs that look realistic to discriminators, while discriminators
learn to distinguish between real and generated examples. Limitedly,
discriminators only assess whether outputs look realistic; not if they are
correct. This can result in a neural network only generating a subset of
outputs, referred to as mode collapse[32]. To counter this issue, generator
learning can be conditioned on an additional distance between generated and
true images[33]. Meaningful distances can be hand-crafted or learned
automatically by considering differences between features imagined by
discriminators for real and generated images[34, 35].

##  2 Training

In this section we introduce a new STEM images dataset for machine learning,
describe how partial scans were selected from images in our data pipeline, and
outline ANN architecture and learning policy. Detailed ANN architecture,
learning policy, and experiments are provided as supplementary information,
and source code is available[36].

###  2.1 Data Pipeline

To create partial scan examples, we collated a new dataset containing 16227
32-bit floating point STEM images collected with a JEOL ARM200F atomic
resolution electron microscope. Individual micrographs were saved to
University of Warwick data servers by dozens of scientists working on hundreds
of projects as Gatan Microscopy Suite[37] generated dm3 or dm4 files. As a
result, our dataset has a diverse constitution. Atom columns are visible in
two-thirds of STEM images, with most signals imaged at several times their
Nyquist rates[38], and similar proportions of images are bright and dark
field. The other third of images are at magnifications too low for atomic
resolution, or are of amorphous materials. The Digital Micrograph image format
is rarely used outside the microscopy community. As a result, data has been
transferred to the widely supported TIFF[39] file format in our publicly
available dataset[40].

Micrographs were split into 12170 training, 1622 validation, and 2435 test set
examples. Each subset was collected by a different subset of scientists and
has different characteristics. As a result, unseen validation and test sets
can be used to quantify the ability of a trained network to generalize. To
reduce data read times, each micrograph was split into non-overlapping
512$\times$512 sub-images, referred to as ‘crops’, producing 110933 training,
21259 validation and 28877 test set crops. For convenience, our crops dataset
is also available[40]. Each crop, $I$, was processed in our data pipeline by
replacing non-finite electron counts, i.e. NaN and $\pm\infty$, with zeros.
Crops were then linearly transformed to have intensities
$I_{\text{N}}\in[-1,1]$, except for uniform crops satisfying
$\max(I)-\min(I)<10^{-6}$ where we set $I_{\text{N}}=0$ everywhere. Finally,
each crop was subject to a random combination of flips and 90$\degree$
rotations to augment the dataset by a factor of eight.

Partial scans, $I_{\text{scan}}$, were selected from raster scan crops,
$I_{\text{N}}$, by multiplication with a binary mask $\Phi_{\text{path}}$,

$I_{\text{scan}}=\Phi_{\text{path}}I_{\text{N}},$ (1)

where $\Phi_{\text{path}}=1$ on a scan path, and $\Phi_{\text{path}}=0$
otherwise. Raster scans are sampled at a rectangular lattice of discrete
locations, so a subset of raster scan pixels are experimental measurements. In
addition, although electron probe position error characteristics may differ
for partial and raster scans, typical position errors are small[41, 42]. As a
result, we expect that partial scans selected from raster scans with binary
masks are realistic.

We also selected partial scans with blurred masks to simulate varying dwell
times and noise characteristics. These difficulties are encountered in
incoherent STEM[43, 44], where STEM illumination is detected by a transmission
electron microscopy (TEM) camera. For simplicity, we created non-physical
noise by multiplying $I_{\text{scan}}$ with
$\eta(\Phi_{\text{path}})=\Phi_{\text{path}}+(1-\Phi_{\text{path}})U$, where
$U$ is a uniform random variate distributed in [0, 2). ANNs are able to
generalize[45, 46], so we expect similar results for other noise
characteristics. A binary mask, with values in $\\{0,1\\}$, is a special case
where no noise is applied i.e. $\eta(1)=1$, and $\Phi_{\text{path}}=0$ is not
traversed. Performance is reported for both binary and blurred masks.

The noise characteristics in our new STEM images dataset vary. This is
problematic for mean squared error (MSE) based ANN training losses, as
differences are higher for crops with higher noise. In effect, this would
increase the importance of noisy images in the dataset, even if they are not
more representative. Although adaptive ANN optimizers that divide parameter
learning rates by gradient sizes[47] can partially mitigate weighting by
varying noise levels, this restricts training to a batch size of 1 and limits
momentum. Consequently, we low-passed filtered ground truth images, $I_{N}$,
to $I_{\text{blur}}$ by a 5$\times$5 symmetric Gaussian kernel with a 2.5 px
standard deviation, to calculate MSEs for ANN outputs.

###  2.2 Network Architecture

Figure 2:  Simplified multiscale generative adversarial network. An inner
generator produces large-scale features from inputs. These are mapped to half-
size completions by a trainer network and recombined with the input to
generate full-size completions by an outer generator. Multiple discriminators
assess multiscale crops from input images and full-size completions.

To generate realistic images, we developed a multiscale conditional GAN with
TensorFlow[48]. Our network can be partitioned into the six convolutional[49,
50] subnetworks shown in Fig. 2: an inner generator, $G_{\text{inner}}$, outer
generator, $G_{\text{outer}}$, inner generator trainer, $T$, and small, medium
and large scale discriminators, $D_{1}$, $D_{2}$ and $D_{3}$. We refer to the
compound network
$G(I_{\text{scan}})=G_{\text{outer}}(G_{\text{inner}}(I_{\text{scan}}),I_{\text{scan}})$
as the generator, and to $D$ = {$D_{1}$, $D_{2}$, $D_{3}$} as the multiscale
discriminator. The generator is the only network needed for inference.

Following recent work on high-resolution conditional GANs[34], we use two
generator subnetworks. The inner generator produces large scale features from
partial scans bilinearly downsampled from 512$\times$512 to 256$\times$256\.
These features are then combined with inputs embedded by the outer generator
to output full-size completions. Following Inception[51, 52], we introduce an
auxiliary trainer network that cooperates with the inner generator to output
256$\times$256 completions. This acts as a regularization mechanism, and
provides a more direct path for gradients to backpropagate to the inner
generator. To more efficiently utilize initial generator convolutions, partial
scans selected with a binary mask are nearest neighbour infilled before being
input to the generator.

Multiscale discriminators examine real and generated STEM images to predict
whether they are real or generated, adapting to the generator as it learns.
Each discriminator assesses different-sized crops selected from 512$\times$512
images, with sizes 70$\times$70, 140$\times$140 or 280$\times$280\. After
selection, crops are bilinearly downsampled to 70$\times$70 before
discriminator convolutions. Typically, discriminators are applied at fractions
of the full image size[34] e.g. $512/2^{2}$, $512/2^{1}$ and $512/2^{0}$.
However, we found that discriminators that downsample large fields of view to
70$\times$70 are less sensitive to high-frequency STEM noise characteristics.
Processing fixed size image regions with multiple discriminators has been
proposed[53] to decrease computation for large images, and extended to
multiple region sizes[34]. However, applying discriminators to arrays of non-
overlapping image patches[54] results in periodic artefacts[34] that are often
corrected by larger-scale discriminators. To avoid these artefacts and reduce
computation, we apply discriminators to randomly selected regions at each
spatial scale.

###  2.3 Learning Policy

Training has two halves. In the non-adversarial first half, the generator and
auxiliary trainer cooperate to minimize mean squared errors (MSEs). This is
followed by an optional second half of training, where the generator is fine-
tuned as part of a GAN to produce realistic images. Our ANNs are trained by
ADAM[55] optimized stochastic gradient descent[47, 56] for up to 2$\times$106
iterations, which takes a few days with an Nvidia GTX 1080 Ti GPU and an
i7-6700 CPU. The objectives of each ANN are codified by their loss functions.

In the non-adversarial first half of training, the generator, $G$, learns to
minimize the MSE based loss

$L_{\text{MSE}}=\text{ALRC}(\lambda_{\text{cond}}\text{MSE}\left(G\left(I_{\text{scan}}),I_{\text{blur}}\right)\right),$
(2)

where $\lambda_{\text{cond}}=200$, and adaptive learning rate clipping[57]
(ALRC) limits loss spikes to stabilize learning. To compensate for varying
noise levels, ground truth images were blurred by a 5$\times$5 symmetric
Gaussian kernel with a 2.5 px standard deviation. In addition, the inner
generator, $G_{\text{inner}}$, cooperates with the auxiliary trainer, $T$, to
minimize

$L_{\text{aux}}=\text{ALRC}\left(\lambda_{\text{trainer}}\text{MSE}\left(T\left(G_{\text{inner}}\left(I_{\text{scan}}^{\text{half}}\right)\right)\right),I_{\text{blur}}^{\text{half}}\right),$
(3)

where $\lambda_{\text{trainer}}=200$, and $I_{\text{scan}}^{\text{half}}$ and
$I_{\text{blur}}^{\text{half}}$ are 256$\times$256 inputs bilinearly
downsampled from $I_{\text{scan}}$ and $I_{\text{blur}}$, respectively.

In the optional adversarial second half of training, we use $N=3$
discriminator scales with numbers, $N_{1}$, $N_{2}$ and $N_{3}$, of
discriminators, $D_{1}$, $D_{2}$ and $D_{3}$, respectively. There many popular
GAN loss functions and regularization mechanisms[58, 59]. In this paper, we
use spectral normalization[60] with squared difference losses[61] for the
discriminators,

$L_{D}=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{N_{i}}[D_{i}(G(I_{\text{scan}}))^{2}+(D_{i}(I_{N})-1)^{2}],$
(4)

where discriminators try to predict 1 for real images and 0 for generated
images. We found that $N_{1}=N_{2}=N_{3}=1$ is sufficient to train the
generator to produce realistic images. However, higher performance might be
achieved with more discriminators e.g. 2 large, 8 medium and 32 small
discriminators. The generator learns to minimize the adversarial squared
difference loss,

$L_{\text{adv}}=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{N_{i}}D_{i}(G(I_{\text{scan}})-1)^{2},$
(5)

by outputting completions that look realistic to discriminators.

Discriminators only assess the realism of generated images; not if they are
correct. To the lift degeneracy and prevent mode collapse, we condition
adversarial training on non-adversarial losses. The total generator loss is

$L_{G}=\lambda_{\text{adv}}L_{\text{adv}}+L_{\text{MSE}}+\lambda_{\text{aux}}L_{\text{aux}},$
(6)

where we found that $\lambda_{\text{aux}}=1$ and $\lambda_{\text{adv}}=5$ is
effective. We also tried conditioning the second half of training on
differences between discriminator imagination[34, 35]. However, we found that
MSE guidance converges to slightly lower MSEs and similar structural
similarity indexes[62] for STEM images.

Figure 3:  Adversarial and non-adversarial completions for 512$\times$512 test
set 1/20 px coverage blurred spiral scan inputs. Adversarial completions have
realistic noise characteristics and structure whereas non-adversarial
completions are blurry. The bottom row shows a failure case where detail is
too fine for the generator to resolve. Enlarged 64$\times$64 regions from the
top left of each image are inset to ease comparison, and the bottom two rows
show non-adversarial generators outputting more detailed features nearer scan
paths.  Figure 4:  Non-adversarial generator outputs for 512$\times$512 1/20
px coverage blurred spiral and gridlike scan inputs. Images with predictable
patterns or structure are accurately completed. Circles accentuate that
generators cannot reliably complete unpredictable images where there is no
information.  Figure 5:  Generator mean squared errors (MSEs) at each output
pixel for 20000 512$\times$512 1/20 px coverage test set images. Systematic
errors are lower near spiral paths for variants of MSE training, and are less
structured for adversarial training. Means, $\mu$, and standard deviations,
$\sigma$, of all pixels in each image are much higher for adversarial outputs.
Enlarged 64$\times$64 regions from the top left of each image are inset to
ease comparison, and to show that systematic errors for MSE training are
higher near output edges.  Figure 6:  Test set root mean squared (RMS)
intensity errors for spiral scans in $[0,1]$ selected with binary masks. a)
RMS errors decrease with increasing electron probe coverage, and are higher
than deep learning supersampling[23] (DLSS) errors. b) Frequency distributions
of 20000 test set RMS errors for 100 bins in $[0,0.224]$ and scan coverages in
the legend.

##  3 Performance

To showcase ANN performance, example applications of adversarial and non-
adversarial generators to 1/20 px coverage partial STEM completion are shown
in Fig. 3. Adversarial completions have more realistic high-frequency spatial
information and structure, and are less blurry than non-adversarial
completions. Systematic spatial variation is also less noticeable for
adversarial completions. For example, higher detail along spiral paths, where
errors are lower, can be seen in the bottom two rows of Fig. 3 for non-
adversarial completions. Inference only requires a generator, so inference
times are the same for adversarial and non-adversarial completions. Single
image inference time during training is 45 ms with an Nvidia GTX 1080 Ti GPU,
which is fast enough for live partial scan completion.

In practice, 1/20 px scan coverage is sufficient to complete most spiral
scans. However, generators cannot reliably complete micrographs with
unpredictable structure in regions where there is no coverage. This is
demonstrated by example applications of non-adversarial generators to 1/20 px
coverage spiral and gridlike partial scans in Fig. 4. Most noticeably, a
generator invents a missing atom at a gap in gridlike scan coverage. Spiral
scans have lower errors than gridlike scans as spirals have smaller gaps
between coverage. Additional sheets of examples for spiral scans selected with
binary masks are provided for scan coverages between 1/17.9 px and 1/87.0 px
as supplementary information

To characterize generator performance, MSEs for output pixels are shown in
Fig. 5. Errors were calculated for 20000 test set 1/20 px coverage spiral
scans selected with blurred masks. Errors systematically increase with
increasing distance from paths for non-adversarial training, and are less
structured for adversarial training. Similar to other generators[63, 23],
errors are also higher near the edges of non-adversarial outputs where there
is less information. We tried various approaches to decrease non-adversarial
systematic error variation by modifying loss functions. For examples: by ALRC;
multiplying pixel losses by their running means; by ALRC and multiplying pixel
losses by their running means; and by ALRC and multiplying pixel losses by
final mean losses of a trained network. However, we found that systematic
errors are similar for all variants. This is a limitation of partial STEM as
information decreases with increasing distance from scan paths. Adversarial
completions also exhibit systematic errors that vary with distance from spiral
paths. However, spiral variation is dominated by other, less structured,
spatial error variation. Errors are higher for adversarial training than for
non-adversarial training as GANs complete images with realistic noise
characteristics.

Spiral path test set intensity errors are shown in Fig. 6a, and decrease with
increasing coverage for binary masks. Test set errors are also presented for
deep learning supersampling[23] (DLSS) as they are the only results that are
directly comparable. DLSS is an alternative approach to compressed sensing
where STEM images are completed from a sublattice of probing locations. Both
DLSS and partial STEM results are for the same neural network architecture,
learning policy and training dataset. We find that DLSS errors are lower than
spiral errors at all coverages. In addition, spiral errors exponentially
increase above DLSS errors at low coverages where minimum distances from
spiral paths increase. Although this comparison may appear unfavourable for
partial STEM, we expect that this is a limitation of training signals being
imaged at several times their Nyquist rates.

Distributions of 20000 spiral path test set root mean squared (RMS) intensity
errors for spiral data in Fig. 6a are shown in Fig. 6b. The coverages listed
in Fig. 6 are for infinite spiral paths with 1/16, 1/25, 1/36, 1/49, 1/64,
1/81, and 1/100 px coverage after paths are cut by image boundaries; changing
coverage. All distributions have a similar peak near an RMS error of 0.04,
suggesting that generator performance remains similar for a portion of images
as coverage is varied. As coverage decreases, the portion of errors above the
peak increases as generators have difficulty with more images. In addition,
there is a small peak close to zero for blank or otherwise trivial
completions.

##  4 Discussion

Partial STEM can decrease scan coverage and total electron electron dose by
10-100$\times$ with 3-6% test set RMS errors. These errors are small compared
to typical STEM noise. Decreased electron dose will enable new STEM
applications to beam-sensitive materials, including organic crystals[64],
metal-organic frameworks[65], nanotubes[66], and nanoparticle dispersions[67].
Partial STEM can also decrease scan times in proportion to decreased coverage.
This will enable increased temporal resolution of dynamic materials, including
polar nanoregions in relaxor ferroelectrics[68, 69], atom motion[70],
nanoparticle nucleation[71], and material interface dynamics[72]. Faster scans
could also reduce delay for experimenters, decreasing microscope time.

Our generators are trained for fixed coverages and 512$\times$512 inputs.
However, recent research has introduced loss function modifications that can
be used to train a single generator for multiple coverages with minimal
performance loss[23]. Using a single GAN improves portability as each of our
GANs requires 1.3 GB of storage space with 32 bit model parameters, and limits
technical debt that may accompany a large number of models. Although our
generator input sizes are fixed, they can be tiled across larger images;
potentially processing tiles in a single batch for computational efficiency.
To reduce higher errors at the edge of generator outputs, tiles can be
overlapped so that edges may be discarded[63]. Smaller images could be padded.
Alternatively, dedicated generators can be trained for other output sizes.

There is an effectively infinite number of possible partial scan paths for
512$\times$512 STEM images. In this paper, we focus on spiral and gridlike
partial scans. For a fixed coverage, we find that the most effective method to
decrease errors is to minimize maximum distances from input information. The
less information there is about an output region, the more information that
needs to be extrapolated, and the higher the error. For example, we find that
errors are lower for spiral scans than gridlike scans than gridlike scans as
maximum distances from input information are lower. Really, the optimal scan
shape is not static: It is specific to a given image and generator
architecture. As a result, we are actively developing an intelligent partial
scan system that adapts to inputs as they are scanned.

Partial STEM has a number of limitations relative to DLSS. For a start,
partial STEM may require a custom scan system. Even if a scan system supports
or can be reprogrammed to support custom scan paths, it may be insufficiently
responsive. In contrast, DLSS can be applied as a postprocessing step without
hardware modification. Another limitation of partial STEM is that errors
increase with increasing distance from scan paths. Distances from continuous
scan paths cannot be decreased without increasing coverage. Finally, most
features in our new STEM crops dataset are sampled at several times their
Nyquist rates. Electron microscopists often record images above minimum
sufficient resolutions and intensities to ease visual inspection and limit the
effects of drift[73], shot[17], and other noise. This means that a DLSS
lattice can still access most high frequency information in our dataset.

Test set DLSS errors are lower than partial STEM errors for the same
architecture and learning policy. However, this is not conclusive as
generators were trained for a few days; rather than until validation errors
diverged from training errors. For example, we expect that spirals need more
training iterations than DLSS as nearest neighbour infilled spiral regions
have varying shapes, whereas infilled regions of DLSS grids are square. A lack
of high frequency information also limits one of the key strengths of partial
STEM that DLSS lacks: access to high-frequency information from neighbouring
pixels. As a result, we expect that partial STEM performance would be higher
for signals imaged closer to their Nyquist rates.

To generate realistic images, we fine-tuned partial STEM generators as part of
GANs. GANs generate images with more realistic high-frequency spatial
components and structure than MSE training. However, GANs focus on semantics;
rather than intensity differences. This means that although adversarial
completions have realistic characteristics, such as high-frequency noise,
individual pixel values differ from true values. GANs can also be difficult to
train[74, 75], and training requires additional computation. Nevertheless,
inference time is the same for adversarial and non-adversarial generators
after training.

Encouragingly, ANNs are universal approximators[76] that can represent[77] the
optimal mapping from partial scans with arbitrary accuracy. This overcomes the
limitations of traditional algorithms where performance is fixed. If ANN
performance is insufficient or surpassed by another method, training or
development can be continued to achieve higher performance. Indeed, validation
errors did not diverge from training errors during our experiments, so we are
presenting lower bounds for performance. In this paper, we compare spiral STEM
performance against DLSS. It is the only method that we can rigorously and
quantitatively compare against as it used the same test set data. This yielded
a new insight into how signals being imaged above their Nyquist rates may
affect performance discussed two paragraphs earlier, and highlights the
importance of standardized datasets like our new STEM images dataset. As
machine learning becomes more established in the electron microscopy
community, we hope that standardized datasets will also become established to
standardize performance benchmarks.

Detailed neural network architecture, learning policy, experiments, and
additional sheets of examples are provided as supplementary information.
Further improvements might be made with AdaNet[78], Ludwig[79], or other
automatic machine learning[80] algorithms, and we encourage further
development. In this spirit, we have made our source code[36], a new dataset
containing 16227 STEM images[40], and pre-trained models publicly available.
For convenience, new datasets containing 161069 non-overlapping 512$\times$512
crops from STEM images used for training, and 19769 antialiased 96$\times$96
area downsampled STEM images created for faster ANN development, are also
available.

##  5 Conclusions

Partial STEM with deep learning can decrease electron dose and scan time by
over an order of magnitude with minimal information loss. In addition,
realistic STEM images can be completed by fine-tuning generators as part of a
GAN. Detailed MSE characteristics are provided for multiple coverages,
including MSEs per output pixel for 1/20 px coverage spiral scans. Partial
STEM will enable new beam sensitive applications, so we have made our source
code, new STEM dataset, pre-trained models, and details of experiments
available to encourage further investigation. High performance is achieved by
the introduction of an auxiliary trainer network, and adaptive learning rate
clipping of high losses. We expect our results to be generalizable to SEM and
other scan systems.

##  6 Data Availability

New STEM datasets are available on our publicly accessible dataserver[40].
Source code is in a GitHub repository with links to pre-trained models[36].
For additional information contact the corresponding author (J.M.E.).

## References

  * [1] Yankovich, A. B., Berkels, B., Dahmen, W., Binev, P. & Voyles, P. M.  High-Precision Scanning Transmission Electron Microscopy at Coarse Pixel Sampling for Reduced Electron Dose.  _Advanced Structural and Chemical Imaging_ 1, 2 (2015).
  * [2] Peters, J. J. P., Apachitei, G., Beanland, R., Alexe, M. & Sanchez, A. M.  Polarization Curling and Flux Closures in Multiferroic Tunnel Junctions.  _Nature Communications_ 7, 13484 (2016).
  * [3] Egerton, R. F., Li, P. & Malac, M.  Radiation Damage in the TEM and SEM.  _Micron_ 35, 399–409 (2004).
  * [4] Hujsak, K., Myers, B. D., Roth, E., Li, Y. & Dravid, V. P.  Suppressing Electron Exposure Artifacts: An Electron Scanning Paradigm with Bayesian Machine Learning.  _Microscopy and Microanalysis_ 22, 778–788 (2016).
  * [5] Jones, L. _et al._ Managing Dose-, Damage- and Data-Rates in Multi-Frame Spectrum-Imaging.  _Microscopy_ 67, i98–i113 (2018).
  * [6] Trampert, P. _et al._ How Should a Fixed Budget of Dwell Time be Spent in Scanning Electron Microscopy to Optimize Image Quality?  _Ultramicroscopy_ 191, 11–17 (2018).
  * [7] Anderson, H. S., Ilic-Helms, J., Rohrer, B., Wheeler, J. & Larson, K.  Sparse Imaging for Fast Electron Microscopy.  In _Computational Imaging XI_ , vol. 8657, 86570C (International Society for Optics and Photonics, 2013).
  * [8] Stevens, A., Yang, H., Carin, L., Arslan, I. & Browning, N. D.  The Potential for Bayesian Compressive Sensing to Significantly Reduce Electron Dose in High-Resolution STEM Images.  _Microscopy_ 63, 41–51 (2013).
  * [9] Stevens, A. _et al._ A Sub-Sampled Approach to Extremely Low-Dose STEM.  _Applied Physics Letters_ 112, 043104 (2018).
  * [10] Hwang, S., Han, C. W., Venkatakrishnan, S. V., Bouman, C. A. & Ortalan, V.  Towards the Low-Dose Characterization of Beam Sensitive Nanostructures via Implementation of Sparse Image Acquisition in Scanning Transmission Electron Microscopy.  _Measurement Science and Technology_ 28, 045402 (2017).
  * [11] Candes, E. & Romberg, J.  Sparsity and Incoherence in Compressive Sampling.  _Inverse Problems_ 23, 969 (2007).
  * [12] Kovarik, L., Stevens, A., Liyu, A. & Browning, N. D.  Implementing an Accurate and Rapid Sparse Sampling Approach for Low-Dose Atomic Resolution STEM Imaging.  _Applied Physics Letters_ 109, 164102 (2016).
  * [13] Sang, X. _et al._ Dynamic Scan Control in STEM: Spiral Scans.  _Advanced Structural and Chemical Imaging_ 2, 6 (2017).
  * [14] Béché, A., Goris, B., Freitag, B. & Verbeeck, J.  Development of a Fast Electromagnetic Beam Blanker for Compressed Sensing in Scanning Transmission Electron Microscopy.  _Applied Physics Letters_ 108, 093103 (2016).
  * [15] Li, X., Dyck, O., Kalinin, S. V. & Jesse, S.  Compressed Sensing of Scanning Transmission Electron Microscopy (STEM) with Nonrectangular Scans.  _Microscopy and Microanalysis_ 24, 623–633 (2018).
  * [16] Sang, X. _et al._ Precision Controlled Atomic Resolution Scanning Transmission Electron Microscopy using Spiral Scan Pathways.  _Scientific Reports_ 7, 43585 (2017).
  * [17] Seki, T., Ikuhara, Y. & Shibata, N.  Theoretical Framework of Statistical Noise in Scanning Transmission Electron Microscopy.  _Ultramicroscopy_ 193, 118–125 (2018).
  * [18] Wu, X. _et al._ Deep Portrait Image Completion and Extrapolation.  _IEEE Transactions on Image Processing_ (2019).
  * [19] Liu, G. _et al._ Image Inpainting for Irregular Holes using Partial Convolutions.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 85–100 (2018).
  * [20] Yang, W. _et al._ Deep Learning for Single Image Super-Resolution: A Brief Review.  _IEEE Transactions on Multimedia_ (2019).
  * [21] Fang, L. _et al._ Deep Learning-Based Point-Scanning Super-Resolution Imaging.  _bioRxiv_ 740548 (2019).
  * [22] de Haan, K., Ballard, Z. S., Rivenson, Y., Wu, Y. & Ozcan, A.  Resolution Enhancement in Scanning Electron Microscopy using Deep Learning.  _Scientific Reports_ 9, 12050, DOI: 10.1038/s41598-019-48444-2 (2019).
  * [23] Ede, J. M.  Deep Learning Supersampled Scanning Transmission Electron Microscopy.  _arXiv preprint arXiv:1910.10467_ (2019).
  * [24] Tan, C. _et al._ A Survey on Deep Transfer Learning.  In _International Conference on Artificial Neural Networks_ , 270–279 (Springer, 2018).
  * [25] Raschka, S.  Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning.  _arXiv preprint arXiv:1811.12808_ (2018).
  * [26] Roh, Y., Heo, G. & Whang, S. E.  A Survey on Data Collection for Machine Learning: A Big Data-AI Integration Perspective.  _IEEE Transactions on Knowledge and Data Engineering_ (2019).
  * [27] Krizhevsky, A., Nair, V. & Hinton, G.  The CIFAR-10 Dataset.  Online: http://www.cs.toronto.edu/k̃riz/cifar.html (2014).
  * [28] Krizhevsky, A., Hinton, G. _et al._ Learning Multiple Layers of Features from Tiny Images.  Tech. Rep., Citeseer (2009).
  * [29] LeCun, Y., Cortes, C. & Burges, C.  MNIST Handwritten Digit Database.  AT&T Labs, online: http://yann.lecun.com/exdb/mnist (2010).
  * [30] Russakovsky, O. _et al._ ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision_ 115, 211–252 (2015).
  * [31] Goodfellow, I. _et al._ Generative Adversarial Nets.  In _Advances in Neural Information Processing Systems_ , 2672–2680 (2014).
  * [32] Bang, D. & Shim, H.  MGGAN: Solving Mode Collapse using Manifold Guided Training.  _arXiv preprint arXiv:1804.04391_ (2018).
  * [33] Mirza, M. & Osindero, S.  Conditional Generative Adversarial Nets.  _arXiv preprint arXiv:1411.1784_ (2014).
  * [34] Wang, T.-C. _et al._ High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 8798–8807 (2018).
  * [35] Larsen, A. B. L., Sønderby, S. K., Larochelle, H. & Winther, O.  Autoencoding Beyond Pixels using a Learned Similarity Metric.  _arXiv preprint arXiv:1512.09300_ (2015).
  * [36] Ede, J. M.  Partial STEM Repository.  Online: https://github.com/Jeffrey-Ede/partial-STEM, DOI: 10.5281/zenodo.3662481 (2019).
  * [37] Gatan.  Gatan Microscopy Suite.  Online: www.gatan.com/products/tem-analysis/gatan-microscopy-suite-software (2019).
  * [38] Landau, H.  Sampling, Data Transmission, and the Nyquist Rate.  _Proceedings of the IEEE_ 55, 1701–1706 (1967).
  * [39] Adobe Developers Association _et al._ TIFF Revision 6.0.  Online: www.adobe.io/content/dam/udp/en/open/standards/tiff/TIFF6.pdf (1992).
  * [40] Ede, J. M.  STEM Datasets.  Online: https://warwick.ac.uk/fac/sci/physics/research/condensedmatt/microscopy/research/machinelearning (2019).
  * [41] Ophus, C., Ciston, J. & Nelson, C. T.  Correcting Nonlinear Drift Distortion of Scanning Probe and Scanning Transmission Electron Microscopies from Image Pairs with Orthogonal Scan Directions.  _Ultramicroscopy_ 162, 1–9 (2016).
  * [42] Sang, X. & LeBeau, J. M.  Revolving Scanning Transmission Electron Microscopy: Correcting Sample Drift Distortion Without Prior Knowledge.  _Ultramicroscopy_ 138, 28–35 (2014).
  * [43] Krause, F. F. _et al._ ISTEM: A Realisation of Incoherent Imaging for Ultra-High Resolution TEM Beyond the Classical Information Limit.  In _European Microscopy Congress 2016: Proceedings_ , 501–502 (Wiley Online Library, 2016).
  * [44] Hartel, P., Rose, H. & Dinges, C.  Conditions and Reasons for Incoherent Imaging in STEM.  _Ultramicroscopy_ 63, 93–114 (1996).
  * [45] Neyshabur, B., Bhojanapalli, S., McAllester, D. & Srebro, N.  Exploring Generalization in Deep Learning.  In _Advances in Neural Information Processing Systems_ , 5947–5956 (2017).
  * [46] Kawaguchi, K., Kaelbling, L. P. & Bengio, Y.  Generalization in Deep Learning.  _arXiv preprint arXiv:1710.05468_ (2017).
  * [47] Ruder, S.  An Overview of Gradient Descent Optimization Algorithms.  _arXiv preprint arXiv:1609.04747_ (2016).
  * [48] Abadi, M. _et al._ TensorFlow: A System for Large-Scale Machine Learning.  In _OSDI_ , vol. 16, 265–283 (2016).
  * [49] McCann, M. T., Jin, K. H. & Unser, M.  Convolutional Neural Networks for Inverse Problems in Imaging: A Review.  _IEEE Signal Processing Magazine_ 34, 85–95 (2017).
  * [50] Krizhevsky, A., Sutskever, I. & Hinton, G. E.  ImageNet Classification with Deep Convolutional Neural Networks.  In _Advances in Neural Information Processing Systems_ , 1097–1105 (2012).
  * [51] Szegedy, C. _et al._ Going Deeper with Convolutions.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 1–9 (2015).
  * [52] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. & Wojna, Z.  Rethinking the Inception Architecture for Computer Vision.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2818–2826 (2016).
  * [53] Durugkar, I., Gemp, I. & Mahadevan, S.  Generative Multi-Adversarial Networks.  _arXiv preprint arXiv:1611.01673_ (2016).
  * [54] Isola, P., Zhu, J.-Y., Zhou, T. & Efros, A. A.  Image-to-Image Translation with Conditional Adversarial Networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 1125–1134 (2017).
  * [55] Kingma, D. P. & Ba, J.  ADAM: A Method for Stochastic Optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * [56] Zou, D., Cao, Y., Zhou, D. & Gu, Q.  Stochastic Gradient Descent Optimizes Over-Parameterized Deep ReLU Networks.  _arXiv preprint arXiv:1811.08888_ (2018).
  * [57] Ede, J. M. & Beanland, R.  Adaptive Learning Rate Clipping Stabilizes Learning.  _arXiv preprint arXiv:1906.09060_ (2019).
  * [58] Wang, Z., She, Q. & Ward, T. E.  Generative Adversarial Networks: A Survey and Taxonomy.  _arXiv preprint arXiv:1906.01529_ (2019).
  * [59] Dong, H.-W. & Yang, Y.-H.  Towards a Deeper Understanding of Adversarial Losses.  _arXiv preprint arXiv:1901.08753_ (2019).
  * [60] Miyato, T., Kataoka, T., Koyama, M. & Yoshida, Y.  Spectral Normalization for Generative Adversarial Networks.  _arXiv preprint arXiv:1802.05957_ (2018).
  * [61] Mao, X. _et al._ Least Squares Generative Adversarial Networks.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 2794–2802 (2017).
  * [62] Wang, Z., Bovik, A. C., Sheikh, H. R. & Simoncelli, E. P.  Image Quality Assessment: From Error Visibility to Structural Similarity.  _IEEE Transactions on Image Processing_ 13, 600–612 (2004).
  * [63] Ede, J. M. & Beanland, R.  Improving Electron Micrograph Signal-to-Noise with an Atrous Convolutional Encoder-Decoder.  _Ultramicroscopy_ 202, 18–25 (2019).
  * [64] S’ari, M., Cattle, J., Hondow, N., Brydson, R. & Brown, A.  Low Dose Scanning Transmission Electron Microscopy of Organic Crystals by Scanning Moiré Fringes.  _Micron_ 120, 1–9 (2019).
  * [65] Mayoral, A., Mahugo, R., Sánchez-Sánchez, M. & Díaz, I.  $C_{s}$-Corrected STEM Imaging of Both Pure and Silver-Supported Metal-Organic Framework MIL-100 (Fe).  _ChemCatChem_ 9, 3497–3502 (2017).
  * [66] Gnanasekaran, K., de With, G. & Friedrich, H.  Quantification and Optimization of ADF-STEM Image Contrast for Beam-Sensitive Materials.  _Royal Society Open Science_ 5, 171838 (2018).
  * [67] Ilett, M., Brydson, R., Brown, A. & Hondow, N.  Cryo-Analytical STEM of Frozen, Aqueous Dispersions of Nanoparticles.  _Micron_ 120, 35–42 (2019).
  * [68] Kumar, A., Dhall, R. & LeBeau, J. M.  In Situ Ferroelectric Domain Dynamics Probed with Differential Phase Contrast Imaging.  _Microscopy and Microanalysis_ 25, 1838–1839 (2019).
  * [69] Xie, L. _et al._ Static and Dynamic Polar Nanoregions in Relaxor Ferroelectric Ba(Ti1-xSnx)O3 System at High Temperature.  _Physical Review B_ 85, 014118 (2012).
  * [70] Aydin, C. _et al._ Tracking Iridium Atoms with Electron Microscopy: First Steps of Metal Nanocluster Formation in One-Dimensional Zeolite Channels.  _Nano Letters_ 11, 5537–5541 (2011).
  * [71] Hussein, H. E. _et al._ Tracking Metal Electrodeposition Dynamics from Nucleation and Growth of a Single Atom to a Crystalline Nanoparticle.  _ACS Nano_ 12, 7388–7396 (2018).
  * [72] Chen, S. _et al._ Atomic Structure and Migration Dynamics of MoS2/LixMoS2 Interface.  _Nano Energy_ 48, 560–568 (2018).
  * [73] Jones, L. & Nellist, P. D.  Identifying and Correcting Scan Noise and Drift in the Scanning Transmission Electron Microscope.  _Microscopy and Microanalysis_ 19, 1050–1060 (2013).
  * [74] Salimans, T. _et al._ Improved Techniques for Training GANs.  In _Advances in Neural Information Processing Systems_ , 2234–2242 (2016).
  * [75] Liang, K. J., Li, C., Wang, G. & Carin, L.  Generative Adversarial Network Training is a Continual Learning Problem.  _arXiv preprint arXiv:1811.11083_ (2018).
  * [76] Hornik, K., Stinchcombe, M. & White, H.  Multilayer Feedforward Networks are Universal Approximators.  _Neural Networks_ 2, 359–366 (1989).
  * [77] Lin, H. W., Tegmark, M. & Rolnick, D.  Why does Deep and Cheap Learning Work so Well?  _Journal of Statistical Physics_ 168, 1223–1247 (2017).
  * [78] Weill, C. _et al._ AdaNet: A Scalable and Flexible Framework for Automatically Learning Ensembles.  _arXiv preprint arXiv:1905.00080_ (2019).
  * [79] Molino, P., Dudin, Y. & Miryala, S. S.  Ludwig: A Type-Based Declarative Deep Learning Toolbox.  _arXiv preprint arXiv:1909.07930_ (2019).
  * [80] He, X., Zhao, K. & Chu, X.  AutoML: A Survey of the State-of-the-Art.  _arXiv preprint arXiv:1908.00709_ (2019).
  * [81] Salimans, T. & Kingma, D. P.  Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks.  In _Advances in Neural Information Processing Systems_ , 901–909 (2016).
  * [82] Hoffer, E., Banner, R., Golan, I. & Soudry, D.  Norm Matters: Efficient and Accurate Normalization Schemes in Deep Networks.  In _Advances in Neural Information Processing Systems_ , 2160–2170 (2018).
  * [83] Chen, L.-C., Papandreou, G., Schroff, F. & Adam, H.  Rethinking Atrous Convolution for Semantic Image Segmentation.  _arXiv preprint arXiv:1706.05587_ (2017).
  * [84] Arjovsky, M., Chintala, S. & Bottou, L.  Wasserstein Generative Adversarial Networks.  In _International Conference on Machine Learning_ , 214–223 (2017).
  * [85] Nair, V. & Hinton, G. E.  Rectified Linear Units Improve Restricted Boltzmann Machines.  In _Proceedings of the 27th International Conference on Machine Learning (ICML-10)_ , 807–814 (2010).
  * [86] Maas, A. L., Hannun, A. Y. & Ng, A. Y.  Rectifier Nonlinearities Improve Neural Network Acoustic Models.  In _Proceedings of the International Conference on Machine Learning_ , vol. 30, 3 (2013).
  * [87] Ioffe, S. & Szegedy, C.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.  _arXiv preprint arXiv:1502.03167_ (2015).
  * [88] Pfau, D. & Vinyals, O.  Connecting Generative Adversarial Networks and Actor-Critic Methods.  _arXiv preprint arXiv:1610.01945_ (2016).
  * [89] Shrivastava, A. _et al._ Learning from Simulated and Unsupervised Images through Adversarial Training.  _arXiv preprint arXiv: 161207828_ (2016).
  * [90] Schaul, T., Quan, J., Antonoglou, I. & Silver, D.  Prioritized Experience Replay.  _arXiv preprint arXiv:1511.05952_ (2015).
  * [91] He, K., Zhang, X., Ren, S. & Sun, J.  Deep Residual Learning for Image Recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 770–778 (2016).
  * [92] Mao, X.-J., Shen, C. & Yang, Y.-B.  Image Restoration using Convolutional Auto-encoders with Symmetric Skip Connections.  _arXiv preprint arXiv:1606.08921_ (2016).
  * [93] Casas, L., Navab, N. & Belagiannis, V.  Adversarial Signal Denoising with Encoder-Decoder Networks.  _arXiv preprint arXiv:1812.08555_ (2018).
  * [94] Badrinarayanan, V., Kendall, A. & Cipolla, R.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 39, 2481–2495 (2017).
  * [95] Zheng, H., Yao, J., Zhang, Y. & Tsang, I. W.  Degeneration in VAE: In the Light of Fisher Information Loss.  _arXiv preprint arXiv:1802.06677_ (2018).
  * [96] Graham, B.  Spatially-Sparse Convolutional Neural Networks.  _arXiv preprint arXiv:1409.6070_ (2014).
  * [97] Sutskever, I., Martens, J., Dahl, G. & Hinton, G.  On the Importance of Initialization and Momentum in Deep Dearning.  In _International Conference on Machine Learning_ , 1139–1147 (2013).
  * [98] Nesterov, Y.  A Method of Solving a Convex Programming Problem with Convergence Rate O(1/$k^{2}$).  In _Soviet Mathematics Doklady_ , vol. 27, 372–376 (1983).
  * [99] Hinton, G., Srivastava, N. & Swersky, K.  Neural Networks for Machine Learning Lecture 6a Overview of Mini-Batch Gradient Descent.  _Cited on_ 14 (2012).

##  7 Acknowledgements

J.M.E. and R.B. acknowledge EPSRC grant EP/N035437/1 for financial support. In
addition, J.M.E. acknowledges EPSRC Studentship 1917382.

##  8 Author Contributions

J.M.E. proposed this research, wrote the code, collated training data,
performed experiments and analysis, created repositories, and co-wrote this
paper. R.B. supervised and co-wrote this paper.

##  9 Competing Interests

The authors declare no competing interests.

##  10 Detailed Architecture

Figure 7:  Discriminators examine random $w$$\times$$w$ crops to predict
whether complete scans are real or generated. Generators are trained by
multiple discriminators with different $w$.  Figure 8:  Two-stage generator
that completes 512$\times$512 micrographs from partial scans. A dashed line
indicates that the same image is input to the inner and outer generator. Large
scale features developed by the inner generator are locally enhanced by the
outer generator and turned into images. An auxiliary trainer network restores
images from inner generator features to provide direct feedback.

Discriminator architecture is shown in Fig. 7. Generator and inner generator
trainer architecture is shown in Fig. 8. The components in our networks are

Bilinear Downsamp, wxw: This is an extension of linear interpolation in one
dimension to two dimensions. It is used to downsample images to
$w$$\times$$w$.

Bilinear Upsamp, xs: This is an extension of linear interpolation in one
dimension to two dimensions. It is used to upsample images by a factor of $s$.

Conv d, wxw, Stride, x: Convolutional layer with a square kernel of width,
$w$, that outputs $d$ feature channels. If the stride is specified,
convolutions are only applied to every $x$th spatial element of their input,
rather than to every element. Striding is not applied depthwise.

Linear, d: Flatten input and fully connect it to $d$ feature channels.

Random Crop, wxw: Randomly sample a $w$$\times$$w$ spatial location using an
external probability distribution.

+

: Circled plus signs indicate residual connections where incoming tensors are
added together. These help reduce signal attenuation and allow the network to
learn perturbative transformations more easily.

All generator convolutions are followed by running mean-only batch
normalization then ReLU activation, except output convolutions. All
discriminator convolutions are followed by slope 0.2 leaky ReLU activation.

##  11 Learning Policy

Optimizer: Training is ADAM[55] optimized and has two halves. In the first
half, the generator and auxiliary trainer learn to minimize mean squared
errors between their outputs and ground truth images. For the quarter of
iterations, we use a constant learning rate $\eta_{0}=0.0003$ and a decay rate
for the first moment of the momentum $\beta_{1}=0.9$. The learning rate is
then stepwise decayed to zero in eight steps over the second quarter of
iterations. Similarly, $\beta_{1}$ is stepwise linearly decayed to 0.5 in
eight steps. In an optional second half, the generator and discriminators play
an adversarial game conditioned on MSE guidance. For the third quarter of
iterations, we use $\eta=0.0001$ and $\beta_{1}=0.9$ for the generator and
discriminators. In the final quarter of iterations, the generator learning
rate is decayed to zero in eight steps while the discriminator learning rate
remains constant. Similarly, generator and discriminator $\beta_{1}$ is
stepwise decayed to 0.5 in eight steps.

Experiments with GAN training hyperparameters show that $\beta_{1}=0.5$ is a
good choice[60]. Our decision to start at $\beta_{1}=0.9$ aims to improve the
initial rate of convergence. In the first stage, generator and auxiliary
trainer parameters are both updated once per training step. In the second
stage, all parameters are updated once per training step. In most of our
initial experiments with burred masks, we used a total of $10^{6}$ training
iterations. However, we found that validation errors do not diverge if
training time is increased to $2\times 10^{6}$ iterations, and used this
number for experiments with binary masks. These training iterations are in-
line with with other GANs, which reuse datasets containing a few thousand
examples for 200 epochs[34]. The lack of validation divergence suggests that
performance may be substantially improved, and means that our results present
lower bounds for performance. All training was performed with a batch size of
1 due to the large model size needed to complete 512$\times$512 scans.

Adaptive learning rate clipping: To stabilize batch size 1 training, adaptive
learning rate clipping[57] (ALRC) was developed to limit high MSEs. ALRC
layers were initialized with first raw moment $\mu_{1}=25$, second raw moment
$\mu_{2}=30$, exponential decay rates $\beta_{1}=\beta_{2}=0.999$, and $n=3$
standard deviations.

Input normalization: Partial scans, $I_{\text{scan}}$, input to the generator
are linearly transformed to $I_{\text{scan}}^{\prime}=(I_{\text{scan}}+1)/2$,
where $I_{\text{scan}}^{\prime}\in[0,1]$. The generator is trained to output
ground truth crops in $[0,1]$, which are linearly transformed to $[-1,1]$.
Generator outputs and ground truth crops in $[-1,1]$ are directly input to
discriminators.

Weight normalization: All generator parameters are weight normalized[81].
Running mean-only batch normalization[81, 82] is applied to the output
channels of every convolutional layer, except the last. Channel means are
tracked by exponential moving averages with decay rates of 0.99. Running mean-
only batch normalization is frozen in the second half of training to improve
stability[83].

Spectral normalization: Spectral normalization[60] is applied to the weights
of each convolutional layer in the discriminators to limit the Lipschitz norms
of the discriminators. We use the power iteration method with one iteration
per training step to enforce a spectral norm of 1 for each weight matrix.

Spectral normalization stabilizes training, reduces susceptibility to mode
collapse and is independent of rank, encouraging discriminators to use more
input features to inform decisions[60]. In contrast, weight normalization[81]
and Wasserstein weight clipping[84] impose more arbitrary model distributions
that may only partially match the target distribution.

Activation: In the generator, ReLU[85] non-linearities are applied after
running mean-only batch normalization. In the discriminators, slope 0.2 leaky
ReLU[86] non-linearities are applied after every convolutional layer.
Rectifier leakage encourages discriminators to use more features to inform
decisions. Our choice of generator and discriminator non-linearities follows
recent work on high-resolution conditional GANs[34].

Initialization: Generator weights were initialized from a normal distribution
with mean 0.00 and standard deviation 0.05. To apply weight normalization, an
example scan is then propagated through the network. Each layer output is
divided by its L2 norm and the layer weights assigned their division by the
square root of the L2 normalized output’s standard deviation. There are no
biases in the generator as running mean-only batch normalization would allow
biases to grow unbounded c.f. batch normalization[87].

Discriminator weights were initialized from a normal distribution with mean
0.00 and standard deviation 0.03. Discriminator biases were zero initialized.

Experience replay: To reduce destabilizing discriminator oscillations[75], we
used an experience replay[88, 89] with 50 examples. Prioritizing the replay of
difficult examples can improve learning[90], so we only replayed examples with
losses in the top 20%. Training examples had a 20% chance to be sampled from
the replay.

##  12 Experiments

In this section, we present learning curves for some of our non-adversarial
architecture and learning policy experiments. During training, each training
set example was reused $\sim$8 times. In comparison, some generative
adversarial networks (GANs) are trained on the same data hundreds of
times[34]. As a result, we did not experience noticeable overfitting. In cases
where final errors are similar; so that their difference is not significant
within the error of a single experiment, we choose the lowest error approach.
In practice, choices between similar errors are unlikely to have a substantial
effect on performance. Each experiment took a few days with an Nvidia GTX 1080
Ti GPU. All learning curves are 2500 iteration boxcar averaged. In addition,
the first $10^{4}$ iterations before dashed lines in figures, where losses
rapidly decrease, are not shown.

Following previous work on high-resolution GANs[34], we used a multi-stage
training protocol for our initial experiments. The outer generator was trained
separately; after the inner generator, before fine-tuning the inner and outer
generator together. An alternative approach uses an auxiliary loss network for
end-to-end training, similar to Inception[51, 52]. This can provide a more
direct path for gradients to back-propagate to the start of the network and
introduces an additional regularization mechanism. Experimenting, we connected
an auxiliary trainer to the inner generator and trained the network in a
single stage. As shown by Fig. 9a, auxiliary network supported end-to-end
training is more stable and converges to lower errors.

Figure 9:  Learning curves. a) Training with an auxiliary inner generator
trainer stabilizes training, and converges to lower than two-stage training
with fine tuning. b) Concatenating beam path information to inputs decreases
losses. Adding symmetric residual connections between strided inner generator
convolutions and transpositional convolutions increases losses. c) Increasing
sizes of the first inner and outer generator convolutional kernels does not
decrease losses. d) Losses are lower after more interations, and a learning
rate (LR) of 0.0004; rather than 0.0002. Labels indicate inner generator
iterations - outer generator iterations - fine tuning iterations, and k
denotes multiplication by 1000 e) Adaptive learning rate clipped quartic
validation losses have not diverged from training losses after $10^{6}$
iterations. f) Losses are lower for outputs in [0, 1] than for outputs in [-1,
1] if leaky ReLU activation is applied to generator outputs.  Figure 10:
Learning curves. a) Making all convolutional kernels 3$\times$3, and not
applying leaky ReLU activation to generator outputs does not increase losses.
b) Nearest neighbour infilling decreases losses. Noise was not added to low
duration path segments for this experiment. c) Losses are similar whether or
not extra noise is added to low-duration path segments. d) Learning is more
stable and converges to lower errors at lower learning rates (LRs). Losses are
lower for spirals than grid-like paths, and lowest when no noise is added to
low-intensity path segments. e) Adaptive momentum-based optimizers, ADAM and
RMSProp, outperform non-adaptive momentum optimizers, including Nesterov-
accelerated momentum. ADAM outperforms RMSProp; however, training
hyperparameters and learning protocols were tuned for ADAM. Momentum values
were 0.9. f) Increasing partial scan pixel coverages listed in the legend
decreases losses.

In encoder-decoders, residual connections[91] between strided convolutions and
symmetric strided transpositional convolutions can be used to reduce
information loss. This is common in noise removal networks where the output is
similar to the input[92, 93]. However, symmetric residual connections are also
used in encoder-decoder networks for semantic image segmentation[94] where the
input and output are different. Consequently, we tried adding symmetric
residual connections between strided and transpositional inner generator
convolutions. As shown by Fig. 9b, extra residuals accelerate initial inner
generator training. However, final errors are slightly higher and initial
inner generator training converged to similar errors withand without symmetric
residuals. Taken together, this suggests that symmetric residuals initially
accelerate training by enabling the final inner generator layers to generate
crude outputs though their direct connections to the first inner generator
layers. However, the symmetric connections also provide a direct path for low-
information outputs of the first layers to get to the final layers, obscuring
the contribution of the inner generator’s skip-3 residual blocks (section 10)
and lowering performance in the final stages of training.

Figure 11:  Adaptive learning rate clipping stabilizes learning, accelerates
convergence and results in lower errors than Huberisation. Weighting pixel
errors with their running or final mean errors is ineffective.

Path information is concatenated to the partial scan input to the generator.
In principle, the generator can infer electron beam paths from partial scans.
However, the input signal is attenuated as it travels through the network[95].
In addition, path information would have to be deduced; rather than informing
calculations in the first inner generator layers, decreasing efficiency. To
compensate, paths used to generate partial scans from full scans are
concatenated to inputs. As shown by Fig. 9b, concatenating path information
reduces errors throughout training. Performance might be further improved by
explicitly building sparsity into the network[96].

Large convolutional kernels are often used at the start of neural networks to
increase their receptive field. This allows their first convolutions to be
used more efficiently. The receptive field can also be increased by increasing
network depth, which could also enable more efficient representation of some
functions[77]. However, increasing network depth can also increase information
loss[95] and representation efficiency may not be limiting. As shown by Fig.
9c, errors are lower for small first convolution kernels; 3$\times$3 for the
inner generator and 7$\times$7 for the outer generator or both 3$\times$3,
than for large first convolution kernels; 7$\times$7 for the inner generator
and 17$\times$17 for the outer generator. This suggests that the generator
does not make effective use of the larger 17$\times$17 kernel receptive field
and that the variability of the extra kernel parameters harms learning.

Learning curves for different learning rate schedules are shown in Fig. 9d.
Increasing training iterations and doubling the learning rate from 0.0002 to
0.0004 lowers errors. Validation errors do not plateau for $10^{6}$ iterations
in Fig. 9e, suggesting that continued training would improve performance. In
our experiments, validation errors were calculated after every 50 training
iterations.

The choice of output domain can affect performance. Training with a [0, 1]
output domain is compared against $[-1,1]$ for slope 0.01 leaky ReLU
activation after every generator convolution in Fig. 9f. Although $[-1,1]$ is
supported by leaky ReLUs, requiring orders of magnitude differences in scale
for $[-1,0)$ and $(0,1]$ hinders learning. To decrease dependence on the
choice output domain, we do not apply batch normalization or activation after
the last generator convolutions in our final architecture.

The $[0,1]$ outputs of Fig. 9f were linearly transformed to $[-1,1]$ and
passed through a $\tanh{}$ non-linearity. This ensured that $[0,1]$ output
errors were on the same scale as $[-1,1]$ output errors, maintaining the same
effective learning rate. Initially, outputs were clipped by a tanh non-
linearity to limit outputs far from the target domain from perturbing
training. However, Fig. 10a shows that errors are similar without end non-
linearites so they were removed. Fig. 10a also shows that replacing slope 0.01
leaky ReLUs with ReLUs and changing all kernel sizes to 3$\times$3 has little
effect. Swapping to ReLUs and 3$\times$3 kernels is therefore an option to
reduce computation. Nevertheless, we continue to use larger kernels throughout
as we think they would usefully increase the receptive field with more stable,
larger batch size training.

To more efficiently use the first generator convolutions, we nearest neighbour
infilled partial scans. As shown by Fig. 10b, infilling reduces error.
However, infilling is expected to be of limited use for low-dose applications
as scans can be noisy, making meaningful infilling difficult. Nevertheless,
nearest neighbour partial scan infilling is a computationally inexpensive
method to improve generator performance for high-dose applications.

To investigate our generator’s ability to handle STEM noise[17], we combined
uniform noise with partial scans of Gaussian blurred STEM images. More noise
was added to low intensity path segments and low-intensity pixels. As shown by
Fig. 10c, ablating extra noise for low-duration path segments increases
performance.

Fig. 10d shows that spiral path training is more stable and reaches lower
errors at lower learning rates. At the same learning rate, spiral paths
converge to lower errors than grid-like paths as spirals have more uniform
coverage. Errors are much lower for spiral paths when both intensity- and
duration-dependent noise is ablated.

To choose a training optimizer, we completed training with stochastic gradient
descent, momentum, Nesterov-accelerated momentum[97, 98], RMSProp[99] and
ADAM[55]. Learning curves are in Fig. 10e. Adaptive momentum optimizers, ADAM
and RMSProp, outperform the non-adaptive optimizers. Non-adaptive momentum-
based optimizers outperform momentumless stochastic gradient decent. ADAM
slightly outperforms RMSProp; however, architecture and learning policy were
tuned for ADAM. This suggests that RMSProp optimization may also be a good
choice.

Learning curves for 1/10, 1/20, 1/40 and 1/100 px coverage spiral scans are
shown in Fig. 10f. In practice, 1/20 px coverage is sufficient for most STEM
images. On average, a non-adversarial generator can complete test set 1/20 px
coverage partial scans with a 2.6% root mean squared intensity error.
Nevertheless, higher coverage is needed to resolve fine detail in some images.
Likewise, lower coverage may be appropriate for images without fine detail.
Consequently, we are developing an intelligent scan system that adjusts
coverage based on micrograph content.

Training is performed with a batch size of 1 due to the large network size
needed for 512$\times$512 partial scans. However, MSE training is unstable and
large error spikes destabilize training. To stabilize learning, we developed
adaptive learning rate clipping[57] (ALRC) to limit magnitudes of high losses
while preserving their initial gradient distributions. ALRC is compared
against MSE, Huberised MSE, and weighting each pixel’s error by its Huberised
running mean, and fixed final errors in Fig. 11. ALRC results in more stable
training with the fastest convergence and lowest errors. Similar improvements
have been confirmed for CIFAR-10 and STEM supersampling with ALRC[57].

##  13 Additional Examples

Note: Additional sheets of examples are not included in this preprint. They
will be in the published version.

Sheets of examples comparing non-adversarial generator outputs and true images
are not shown in this preprint for 512$\times$512 spiral scans selected with
binary masks. True images are blurred by a 5$\times$5 symmetric Gaussian
kernel with a 2.5 px standard deviation so that they are the same as the
images that generators were trained output. Examples are presented for 1/17.9,
1/27.3, 1/38.2, 1/50.0, 1/60.5, 1/73.7, and 1/87.0 px coverage, in that order,
so that higher errors become apparent for decreasing coverage with increasing
page number. Quantitative performance characteristics for each generator are
provided in the main article.