# Review: Deep Learning in Electron Microscopy

Jeffrey M. Ede  University of Warwick, Department of Physics, Coventry, CV4
7AL, UK<EMAIL_ADDRESS>
###### Abstract

Deep learning is transforming most areas of science and technology, including
electron microscopy. This review paper offers a practical perspective aimed at
developers with limited familiarity. For context, we review popular
applications of deep learning in electron microscopy. Afterwards, we discuss
hardware and software needed to get started with deep learning and interface
with electron microscopes. We then review neural network components, popular
architectures, and their optimization. Finally, we discuss future directions
of deep learning in electron microscopy.

Keywords: deep learning, electron microscopy, review.

##  1 Introduction

Following decades of exponential increases in computational capability[1] and
widespread data availability[2, 3], scientists can routinely develop
artificial neural networks[4, 5, 6, 7, 8, 9, 10, 11] (ANNs) to enable new
science and technology[12, 13, 14, 15, 16, 17]. The resulting deep learning
revolution[18, 19] has enabled superhuman performance in image
classification[20, 21, 22, 23], games[24, 25, 26, 27, 28, 29], medical
analysis[30, 31], relational reasoning[32], speech recognition[33, 34] and
many other applications[35, 36]. This introduction focuses on deep learning in
electron microscopy and is aimed at developers with limited familiarity. For
context, we therefore review popular applications of deep learning in electron
microscopy. We then review resources available to support researchers and
outline electron microscopy. Finally, we review popular ANN architectures and
their optimization, or “training”, and discuss future trends in artificial
intelligence (AI) for electron microscopy.

Deep learning is motivated by universal approximator theorems[37, 38, 39, 40,
41, 42, 43, 44, 45], which state that sufficiently deep and wide[37, 46, 40]
ANNs can approximate functions to arbitrary accuracy. It follows that ANNs can
always match or surpass the performance of methods crafted by humans. In
practice, deep neural networks (DNNs) reliably[47] learn to express[48, 49,
50, 51] generalizable[52, 53, 54, 55, 56, 57, 58, 59] models without a prior
understanding of physics. As a result, deep learning is freeing physicists
from a need to devise equations to model complicated phenomena[60, 61, 13, 14,
16]. Many modern ANNs have millions of parameters, so inference often takes
tens of milliseconds on graphical processing units (GPUs) or other hardware
accelerators[62]. It is therefore unusual to develop ANNs to approximate
computationally efficient methods with exact solutions, such as the fast
Fourier transform[63, 64, 65] (FFT). However, ANNs are able to leverage an
understanding of physics to accelerate time-consuming or iterative
calculations[66, 67, 68, 69], improve accuracy of methods[70, 30, 31], and
find solutions that are otherwise intractable[24, 71].

Figure 1:  Example applications of a noise-removal DNN to instances of Poisson
noise applied to 512$\times$512 crops from TEM images. Enlarged 64$\times$64
regions from the top left of each crop are shown to ease comparison. This
figure is adapted from our earlier work[72] under a Creative Commons
Attribution 4.0[73] license.

###  1.1 Improving Signal-to-Noise

A popular application of deep learning is to improve signal-to-noise[74, 75].
For example, of medical electrical[76, 77], medical image[78, 79, 80], optical
microscopy[81, 82, 83, 84], and speech[85, 86, 87, 88] signals. There are many
traditional denoising algorithms that are not based on deep learning[89, 90,
91], including linear[92, 93] and non-linear[94, 95, 96, 97, 98, 99, 100, 101,
102] spatial domain filters, Wiener[103, 104, 105] filters, non-linear[106,
107, 108, 109, 110, 111] wavelet domain filters, curvelet transforms[112,
113], contourlet transforms[114, 115], hybrid algorithms[116, 117, 118, 119,
120, 121, 122] that operate in both spatial and transformed domains, and
dictionary-based learning[123, 124, 125, 126, 127]. However, traditional
denoising algorithms are limited by features (often laboriously) crafted by
humans and cannot exploit domain-specific context. In perspective, they
leverage an ever-increasingly accurate representation of physics to denoise
signals. However, traditional algorithms are limited by the difficulty of
programmatically describing a complicated reality. As a case in point, an ANN
was able to outperform decades of advances in traditional denoising algorithms
after training on two GPUs for a week[70].

Definitions of electron microscope noise can include statistical noise[128,
129, 130, 131, 132, 133, 134, 135], aberrations[136], scan distortions[137,
138, 139, 140], specimen drift[141], and electron beam damage[142].
Statistical noise is often minimized by either increasing electron dose or
applying traditional denoising algorithms[143, 144]. There are a variety of
denoising algorithms developed for electron microscopy, including algorithms
based on block matching[145], contourlet transforms[114, 115], energy
minimization[146], fast patch reorderings[147], Gaussian kernel density
estimation[148], Kronecker envelope principal component analysis[149] (PCA),
non-local means and Zernike moments[150], singular value thresholding[151],
wavelets[152], and other approaches[153, 154, 141, 155, 156]. Noise that is
not statistical is often minimized by hardware. For example, by using
aberration correctors[136, 157, 158, 159], choosing scanning transmission
electron microscopy (STEM) scan shapes and speeds that minimize
distortions[138], and using stable sample holders to reduce drift[160]. Beam
damage can also be reduced by using minimal electron voltage and electron
dose[161, 162, 163], or dose-fractionation across multiple frames in multi-
pass transmission electron microscopy[164, 165, 166] (TEM) or STEM[167].

Deep learning is being applied to improve signal-to-noise for a variety of
applications[168, 169, 170, 171, 172, 173, 174, 175, 176]. Most approaches in
electron microscopy involve training ANNs to either map low-quality
experimental[177], artificially deteriorated[70, 178] or synthetic[179, 180,
181, 182] inputs to paired high-quality experimental measurements. For
example, applications of a DNN trained with artificially deteriorated TEM
images are shown in figure 1. However, ANNs have also been trained with
unpaired datasets of low-quality and high-quality electron micrographs[183],
or pairs of low-quality electron micrographs[184, 185]. Another approach is
Noise2Void[168], ANNs are trained from single noisy images. However,
Noise2Void removes information by masking noisy input pixels corresponding to
target output pixels. So far, most ANNs that improve electron microscope
signal-to-noise have been trained to decrease statistical noise[183, 177, 70,
186, 181, 182, 184, 179, 180, 181] as other approaches have been developed to
correct electron microscope scan distortions[187, 188] and specimen drift[189,
188, 141]. However, we anticipate that ANNs will be developed to correct a
variety of electron microscopy noise as ANNs have been developed for
aberration correction of optical microscopy[190, 191, 192, 193, 194, 195] and
photoacoustic[196] signals.

Figure 2:  Example applications of DNNs to restore 512$\times$512 STEM images
from sparse signals. Training as part of a generative adversarial network[197,
198, 199, 200] yields more realistic outputs than training a single DNN with
mean squared errors. Enlarged 64$\times$64 regions from the top left of each
crop are shown to ease comparison. a) Input is a Gaussian blurred 1/20
coverage spiral[201]. b) Input is a 1/25 coverage grid[202]. This figure is
adapted from our earlier works under Creative Commons Attribution 4.0[73]
licenses.

###  1.2 Compressed Sensing

Compressed sensing[203, 204, 205, 206, 207] is the efficient reconstruction of
a signal from a subset of measurements. Applications include faster medical
imaging[208, 209, 210], image compression[211, 212], increasing image
resolution[213, 214], lower medical radiation exposure[215, 216, 217], and
low-light vision[218, 219]. In STEM, compressed sensing has enabled electron
beam exposure and scan time to be decreased by 10-100$\times$ with minimal
information loss[201, 202]. Thus, compressed sensing can be essential to
investigations where the high current density of electron probes damages
specimens[161, 220, 221, 222, 223, 224, 225, 226]. Even if the effects of beam
damage can be corrected by postprocessing, the damage to specimens is often
permanent. Examples of beam-sensitive materials include organic crystals[227],
metal-organic frameworks[228], nanotubes[229], and nanoparticle
dispersions[230]. In electron microscopy, compressed sensing is especially
effective due to high signal redundancy[231]. For example, most electron
microscopy images are sampled at 5-10$\times$ their Nyquist rates[232] to ease
visual inspection, decrease sub-Nyquist aliasing[233], and avoid
undersampling.

Perhaps the most popular approach to compressed sensing is upsampling or
infilling a uniformly spaced grid of signals[234, 235, 236]. Interpolation
methods include Lancsoz[234], nearest neighbour[237], polynomial
interpolation[238], Wiener[239] and other resampling methods[240, 241, 242].
However, a variety of other strategies to minimize STEM beam damage have also
been proposed, including dose fractionation[243] and a variety of sparse data
collection methods[244]. Perhaps the most intensively investigated approach to
the latter is sampling a random subset of pixels, followed by reconstruction
using an inpainting algorithm[245, 244, 246, 247, 248, 249]. Random sampling
of pixels is nearly optimal for reconstruction by compressed sensing
algorithms[250]. However, random sampling exceeds the design parameters of
standard electron beam deflection systems, and can only be performed by
collecting data slowly[251, 138], or with the addition of a fast deflection or
blanking system[247, 252].

Sparse data collection methods that are more compatible with conventional STEM
electron beam deflection systems have also been investigated. For example,
maintaining a linear fast scan deflection whilst using a widely-spaced slow
scan axis with some small random ‘jitter’[251, 245]. However, even small jumps
in electron beam position can lead to a significant difference between nominal
and actual beam positions in a fast scan. Such jumps can be avoided by driving
functions with continuous derivatives, such as those for spiral and Lissajous
scan paths[201, 253, 138, 254, 247]. Sang[138, 254] considered a variety of
scans including Archimedes and Fermat spirals, and scans with constant angular
or linear displacements, by driving electron beam deflectors with a field-
programmable gate array[255] (FPGA) based system[138]. Spirals with constant
angular velocity place the least demand on electron beam deflectors. However,
dwell times, and therefore electron dose, decreases with radius. Conversely,
spirals created with constant spatial speeds are prone to systematic image
distortions due to lags in deflector responses. In practice, fixed doses are
preferable as they simplify visual inspection and limit the dose dependence of
STEM noise[129].

Deep learning can leverage an understanding of physics to infill images[256,
257, 258]. Example applications include increasing scanning electron
microscopy[259, 178, 260] (SEM), STEM[202, 261] and TEM[262] resolution, and
infilling continuous sparse scans[201]. Example applications of DNNs to
complete sparse spiral and grid scans are shown in figure 2. However, caution
should be used when infilling large regions as ANNs may generate artefacts if
a signal is unpredictable[201]. A popular alternative to deep learning for
infilling large regions is exemplar-based infilling[263, 264, 265, 266].
However, exemplar-based infilling often leaves artefacts[267] and is usually
limited to leveraging information from single images. Smaller regions are
often infilled by fast marching[268], Navier-Stokes infilling[269], or
interpolation[238].

###  1.3 Labelling

Deep learning has been the basis of state-of-the-art classification[270, 271,
272, 273] since convolutional neural networks (CNNs) enabled a breakthrough in
classification accuracy on ImageNet[71]. Most classifiers are single
feedforward neural networks (FNNs) that learn to predict discrete labels. In
electron microscopy, applications include classifying image region
quality[274, 275], material structures[276, 277], and image resolution[278].
However, siamese[279, 280, 281] and dynamically parameterized[282] networks
can more quickly learn to recognise images. Finally, labelling ANNs can learn
to predict continuous features, such as mechanical properties[283]. Labelling
ANNs are often combined with other methods. For example, ANNs can be used to
automatically identify particle locations[284, 285, 186, 286] to ease
subsequent processing.

Figure 3:  Example applications of a semantic segmentation DNN to STEM images
of steel to classify dislocation locations. Yellow arrows mark uncommon
dislocation lines with weak contrast, and red arrows indicate that fixed
widths used for dislocation lines are sometimes too narrow to cover defects.
This figure is adapted with permission[287] under a Creative Commons
Attribution 4.0[73] license.

###  1.4 Semantic Segmentation

Semantic segmentation is the classification of pixels into discrete
categories. In electron microscopy, applications include the automatic
identification of local features[288, 289], such as defects[290, 291],
dopants[292], material phases[293], material structures[294, 295], dynamic
surface phenomena[296], and chemical phases in nanoparticles[297]. Early
approaches to semantic segmentation used simple rules. However, such methods
were not robust to a high variety of data[298]. Subsequently, more adaptive
algorithms based on soft-computing[299] and fuzzy algorithms[300] were
developed to use geometric shapes as priors. However, these methods were
limited by programmed features and struggled to handle the high variety of
data.

To improve performance, DNNs have been trained to semantically segment
images[301, 302, 303, 304, 305, 306, 307, 308]. Semantic segmentation DNNs
have been developed for focused ion beam scanning electron microscopy[309,
310, 311] (FIB-SEM), SEM[312, 313, 314, 311], STEM[315, 287], and TEM[286,
316, 317, 310, 318, 311, 319]. For example, applications of a DNN to semantic
segmentation of STEM images of steel are shown in figure 3. Deep learning
based semantic segmentation also has a high variety of applications outside of
electron microscopy, including autonomous driving[320, 321, 322, 323, 324],
dietary monitoring[325, 326], magnetic resonance images[327, 328, 329, 330,
331], medical images[332, 333, 334] such as prenatal ultrasound[335, 336, 337,
338], and satellite image translation[339, 340, 341, 342, 343]. Most DNNs for
semantic segmentation are trained with images segmented by humans. However,
human labelling may be too expensive, time-consuming, or inappropriate for
sensitive data. Unsupervised semantic segmentation can avoid these
difficulties by learning to segment images from an additional dataset of
segmented images[344] or image-level labels[345, 346, 347, 348]. However,
unsupervised semantic segmentation networks are often less accurate than
supervised networks.

Figure 4:  Example applications of a DNN to reconstruct phases of exit
wavefunction from intensities of single TEM images. Phases in $[-\pi,\pi)$ rad
are depicted on a linear greyscale from black to white, and Miller indices
label projection directions. This figure is adapted from our earlier work[349]
under a Creative Commons Attribution 4.0[73] license.

###  1.5 Exit Wavefunction Reconstruction

Electrons exhibit wave-particle duality[350, 351], so electron propagation is
often described by wave optics[352]. Applications of electron wavefunctions
exiting materials[353] include determining projected potentials and
corresponding crystal structure information[354, 355], information storage,
point spread function deconvolution, improving contrast, aberration
correction[356], thickness measurement[357], and electric and magnetic
structure determination[358, 359]. Usually, exit wavefunctions are either
iteratively reconstructed from focal series[360, 361, 362, 363, 364] or
recorded by electron holography[352, 363, 365]. However, iterative
reconstruction is often too slow for live applications, and holography is
sensitive to distortions and may require expensive microscope modification.

Non-iterative methods based on DNNs have been developed to reconstruct optical
exit wavefunctions from focal series[69] or single images[366, 367, 368].
Subsequently, DNNs have been developed to reconstruct exit wavefunctions from
single TEM images[349], as shown in figure 4. Indeed, deep learning is
increasingly being applied to accelerated quantum mechanics[369, 370, 371,
372, 373, 374]. Other examples of DNNs adding new dimensions to data include
semantic segmentation described in section 1.4, and reconstructing 3D atomic
distortions from 2D images[375]. Non-iterative methods that do not use ANNs to
recover phase information from single images have also been developed[376,
377]. However, they are limited to defocused images in the Fresnel
regime[376], or to non-planar incident wavefunctions in the Fraunhofer
regime[377].

##  2 Resources

Access to scientific resources is essential to scientific enterprise[378].
Fortunately, most resources needed to get started with machine learning are
freely available. This section provides directions to various machine learning
resources, including how to access deep learning frameworks, a free GPU or
tensor processing unit (TPU) to accelerate tensor computations, platforms that
host datasets and source code, and pretrained models. To support the ideals of
open science embodied by Plan S[379, 378, 380], we focus on resources that
enhance collaboration and enable open access[381]. We also discuss how
electron microscopes can interface with ANNs and the importance of machine
learning resources in the context of electron microscopy. However, we expect
that our insights into electron microscopy can be generalized to other
scientific fields.

###  2.1 Hardware Acceleration

A DNN is an ANN with multiple layers that perform a sequence of tensor
operations. Tensors can either be computed on central processing units (CPUs)
or hardware accelerators[62], such as FPGAs[382, 383, 384, 385], GPUs[386,
387, 388], and TPUs[389, 390, 391]. Most benchmarks indicate that GPUs and
TPUs outperform CPUs for typical DNNs that could be used for image
processing[392, 393, 394, 395, 396] in electron microscopy. However, GPU and
CPU performance can be comparable when CPU computation is optimized[397]. TPUs
often outperform GPUs[394], and FPGAs can outperform GPUs[398, 399] if FPGAs
have sufficient arithmetic units[400, 401]. Typical power consumption per
TFLOPS[402] decreases in order CPU, GPU, FPGA, then TPU, so hardware
acceleration can help to minimize long-term costs and environmental
damage[403].

For beginners, Google Colab[404, 405, 406, 407] and Kaggle[408] provide
hardware accelerators in ready-to-go deep learning environments. Free compute
time on these platforms is limited as they are not intended for industrial
applications. Nevertheless, the free compute time is sufficient for some
research[409]. For more intensive applications, it may be necessary to get
permanent access to hardware accelerators. If so, many online guides detail
how to install[410, 411] and set up an Nvidia[412] or AMD[413] GPU in a
desktop computer for deep learning. However, most hardware comparisons for
deep learning[414] focus on Nvidia GPUs as most deep learning frameworks use
Nvidia’s proprietary Compute Unified Device Architecture (CUDA) Deep Neural
Network (cuDNN) primitives for deep learning[415], which are optimized for
Nvidia GPUs. Alternatively, hardware accelerators may be accessible from a
university or other institutional high performance computing (HPC) centre, or
via a public cloud service provider[416, 417, 418, 419].

Framework | License | Programming Interfaces
---|---|---
Apache SINGA[420] |  Apache 2.0[421] | C++, Java, Python
BigDL[422] |  Apache 2.0[423] | Python, Scala
Caffe[424, 425] |  BSD[426] | C++, MATLAB, Python
Chainer[427] |  MIT[428] | Python
Deeplearning4j[429] |  Apache 2.0[430] | Clojure, Java, Kotlin, Python, Scala
Dlib[431, 432] |  BSL[433] | C++
Flux[434] |  MIT[435] | Julia
MATLAB Deep Learning Toolbox[436] |  Proprietary[437] | MATLAB
Microsoft Cognitive Toolkit[438] |  MIT[439] | BrainScript, C++, Python
Apache MXNet[440] |  Apache 2.0[441] | C++, Clojure, Go, JavaScript, Julia, Matlab, Perl, Python, R, Scala
OpenNN[442] |  GNU LGPL[443] | C++
PaddlePaddle[444] |  Apache 2.0[445] | C++
PyTorch[446] |  BSD[447] | C++, Python
TensorFlow[448, 449] |  Apache 2.0[450] | C++, C#, Go, Haskell, Julia, MATLAB, Python, Java, JavaScript, R, Ruby, Rust, Scala, Swift
Theano[451, 452] |  BSD[453] | Python
Torch[454] |  BSD[455] | C, Lua
Wolfram Mathematica[456] |  Proprietary[457] | Wolfram Language
Table 1:  Deep learning frameworks with programming interfaces. Most
frameworks have open source code and many support multiple programming
languages.

###  2.2 Deep Learning Frameworks

A deep learning framework[458, 459, 460, 461, 462, 9, 463, 464] (DLF) is an
interface, library or tool for DNN development. Features often include
automatic differentiation[465], heterogeneous computing, pretrained models,
and efficient computing[466] with CUDA[467, 468, 469], cuDNN[415, 470],
OpenMP[471, 472], or similar libraries. Popular DLFs tabulated in table 1
often have open source code and support multiple programming interfaces.
Overall, TensorFlow[448, 449] is the most popular DLF[473]. However,
PyTorch[446] is the most popular DLF at top machine learning conferences[473,
474]. Some DLFs also have extensions that ease development or extend
functionality. For example, TensorFlow extensions[475] that ease development
include Keras[476], Sonnet[477], Tensor2Tensor[478] and TFLearn[479, 480], and
extensions that add functionality include Addons[481], Agents[482],
Dopamine[483], Federated[484, 485, 486], Probability[487], and TRFL[488]. In
addition, DLFs are supplemented by libraries for predictive data analysis,
such as scikit-learn[489].

A limitation of the DLFs in table 1 is that users must use programming
interfaces. This is problematic as many electron microscopists have limited,
if any, programming experience. To increase accessibility, a range of
graphical user interfaces (GUIs) have been created for ANN development. For
example, ANNdotNET[490], Create ML[491], Deep Cognition[492], Deep Network
Designer[493], DIGITS[494], ENNUI[495], Expresso[496], Neural Designer[497],
Waikato Environment for Knowledge Analysis[498, 499, 500] (WEKA) and
ZeroCostDL4Mic[501]. The GUIs offer less functionality and scope for
customization than programming interfaces. However, GUI-based DLFs are rapidly
improving. Moreover, existing GUI functionality is more than sufficient to
implement popular FNNs, such as image classifiers[272] and encoder-
decoders[502, 503, 305, 306, 307, 308, 504].

###  2.3 Pretrained Models

Training ANNs is often time-consuming and computationally expensive[403].
Fortunately, pretrained models are available from a range of open access
collections[505], such as Model Zoo[506], Open Neural Network Exchange[507,
508, 509, 510] (ONNX) Model Zoo[511], TensorFlow Hub[512, 513], and TensorFlow
Model Garden[514]. Some researchers also provide pretrained models via project
repositories[70, 349, 201, 231, 202]. Pretrained models can be used
immediately or to transfer learning[515, 516, 517, 518, 519, 520, 521] to new
applications. For example, by fine-tuning and augmenting the final layer of a
pretrained model[522]. Benefits of transfer learning can include decreasing
training time by orders of magnitude, reducing training data requirements, and
improving generalization[520, 523].

Using pretrained models is complicated by ANNs being developed with a variety
of DLFs in a range of programming languages. However, most DLFs support
interoperability. For example, by supporting the saving of models to a common
format or to formats that are interoperable with the Neural Network Exchange
Format[524] (NNEF) or ONNX formats. Many DLFs also support saving models to
HDF5[525, 526], which is popular in the pycroscopy[527, 528] and HyperSpy[529,
530] libraries used by electron microscopists. The main limitation of
interoperability is that different DLFs may not support the same
functionality. For example, Dlib[431, 432] does not support recurrent neural
networks[531, 532, 533, 534, 535, 536] (RNNs).

###  2.4 Datasets

Randomly initialized ANNs[537] must be trained, validated, and tested with
large, carefully partitioned datasets to ensure that they are robust to
general use[538]. Most ANN training starts from random initialization, rather
than transfer learning[515, 516, 517, 518, 519, 520, 521], as:

  1. 1.

Researchers may be investigating modifications to ANN architecture or ability
to learn.

  2. 2.

Pretrained models may be unavailable or too difficult to find.

  3. 3.

Models may quickly achieve sufficient performance from random initialization.
For example, training an encoder-decoder based on Xception[539] to improve
electron micrograph signal-to-noise[70] can require less training than for
PASCAL VOC 2012[540] semantic segmentation[305].

  4. 4.

There may be a high computing budget, so transfer learning is unnecessary[541,
542].

There are millions of open access datasets[543, 544] and a range of platforms
that host[545, 546, 547, 548, 549] or aggregate[550, 551, 552, 553] machine
learning datasets. Openly archiving datasets drives scientific enterprise by
reducing need to repeat experiments[554, 555, 556, 557, 558], enabling new
applications through data mining[559, 560], and standardizing performance
benchmarks[561]. For example, popular datasets used to standardize image
classification performance benchmarks include CIFAR-10[562, 563], MNIST[564]
and ImageNet[565]. A high range of both domain-specific and general platforms
that host scientific data for free are listed by the Open Access
Directory[566] and Nature Scientific Data[567]. For beginners, we recommend
Zenodo[568] as it is free, open access, has an easy-to-use interface, and will
host an unlimited number of datasets smaller than 50 GB for at least 20
years[569].

There are a range of platforms dedicated to hosting electron microscopy
datasets, including the Caltech Electron Tomography Database[570] (ETDB-
Caltech), Electron Microscopy Data Bank[571, 572, 573, 574, 575, 576]
(EMDataBank), and the Electron Microscopy Public Image Archive[577] (EMPIAR).
However, most electron microscopy datasets are small, esoteric or are not
partitioned for machine learning[231]. Nevertheless, a variety of large
machine learning datasets for electron microscopy are being published in
independent repositories[231, 578, 579], including Warwick Electron Microscopy
Datasets[231] (WEMD) that we curated. In addition, a variety of databases host
information that supports electron microscopy. For example, crystal structure
databases provide data in standard formats[580, 581], such as Crystallography
Information Files[582, 583, 584, 585] (CIFs). Large crystal structure
databases[586, 587, 588] containing over $10^{5}$ crystal structures include
the Crystallography Open Database[589, 590, 591, 592, 593, 594] (COD),
Inorganic Crystal Structure Database[595, 596, 597, 598, 599] (ICSD), and
National Institute of Standards and Technology (NIST) Crystal Data[600, 601].

Platform | Website | For Machine Learning
---|---|---
Amazon Mechanical Turk | https://www.mturk.com | General tasks
Appen | https://appen.com | Machine learning data preparation
Clickworker | https://www.clickworker.com | Machine learning data preparation
Fiverr | https://www.fiverr.com | General tasks
Hive | https://thehive.ai | Machine learning data preparation
iMerit | https://imerit.net | Machine learning data preparation
JobBoy | https://www.jobboy.com | General tasks
Minijobz | https://minijobz.com | General tasks
Microworkers | https://www.microworkers.com | General tasks
OneSpace | https://freelance.onespace.com | General tasks
Playment | https://playment.io | Machine learning data preparation
RapidWorkers | https://rapidworkers.com | General tasks
Scale | https://scale.com | Machine learning data preparation
Smart Crowd | https://thesmartcrowd.lionbridge.com | General tasks
Trainingset.ai | https://www.trainingset.ai | Machine learning data preparation
ySense | https://www.ysense.com | General tasks
Table 2: Microjob service platforms. The size of typical tasks varies for
different platforms and some platforms specialize in preparing machine
learning datasets.

To achieve high performance, it may be necessary to curate a large dataset for
ANN training[2]. However, large datasets like DeepMind Kinetics[602],
ImageNet[565], and YouTube 8M[603] may take a team months to prepare. As a
result, it may not be practical to divert sufficient staff and resources to
curate a high-quality dataset, even if curation is partially automated[604,
605, 606, 607, 608, 609, 603, 610]. To curate data, human capital can be
temporarily and cheaply increased by using microjob services[611]. For
example, through microjob platforms tabulated in table 2. Increasingly,
platforms are emerging that specialize in data preparation for machine
learning. Nevertheless, microjob services may be inappropriate for sensitive
data or tasks that require substantial domain-specific knowledge.

###  2.5 Source Code

Software is part of our cultural, industrial, and scientific heritage[612].
Source code should therefore be archived where possible. For example, on an
open source code platform such as Apache Allura[613], AWS CodeCommit[614],
Beanstalk[615], BitBucket[616], GitHub[617], GitLab[618], Gogs[619], Google
Cloud Source Repositories[620], Launchpad[621], Phabricator[622],
Savannah[623] or SourceForge[624]. These platforms enhance collaboration with
functionality that helps users to watch[625] and contribute improvements[626,
627, 628, 629, 630, 631, 632] to source code. The choice of platform is often
not immediately important for small electron microscopy projects as most
platforms offer similar functionality. Nevertheless, functionality comparisons
of open source platforms are available[633, 634, 635]. For beginners, we
recommend GitHub as it is actively developed, scalable to large projects and
has an easy-to-use interface.

###  2.6 Finding Information

Most web traffic[636, 637] goes to large-scale web search engines[638, 639,
640, 641, 642] such as Bing, DuckDuckGo, Google, and Yahoo. This includes
searches for scholarly content[643, 644, 645]. We recommend Google for
electron microscopy queries as it appears to yield the best results for
general[646, 647, 648], scholarly[645, 644] and other[649] queries. However,
general search engines can be outperformed by dedicated search engines for
specialized applications. For example, for finding academic literature[650,
651, 652], data[653], jobs[654, 655], publication venues[656], patents[657,
658, 659, 660], people[661, 662, 663], and many other resources. The use of
search engines is increasingly political[664, 665, 666] as they influence
which information people see. However, most users appear to be satisfied with
their performance[667].

Introductory textbooks are outdated[668, 669] insofar that most information is
readily available online. We find that some websites are frequent references
for up-to-date and practical information:

  1. 1.

Stack Overflow[670, 671, 672, 673, 674, 675] is a source of working code
snippets and a useful reference when debugging code.

  2. 2.

Papers With Code State-of-the-Art[561] leaderboards rank the highest
performing ANNs with open source code for various benchmarks.

  3. 3.

Medium[676] and its subsidiaries publish blogs with up-to-date and practical
advice about machine learning.

  4. 4.

The Machine Learning subreddit[677] hosts discussions about machine learning.
In addition, there is a Learn Machine Learning subreddit[678] aimed at
beginners.

  5. 5.

Dave Mitchell’s DigitalMicrograph Scripting Website[679, 680] hosts a
collection of scripts and documentation for programming electron microscopes.

  6. 6.

The Internet Archive[681, 682] maintains copies of software and media,
including webpages via its Wayback Machine[683, 684, 685].

  7. 7.

Distill[686] is a journal dedicated to providing clear explanations about
machine learning. Monetary prizes are awarded for excellent communication and
refinement of ideas.

This list enumerates popular resources that we find useful, so it may
introduce personal bias. However, alternative guides to useful resources are
available[687, 688, 689]. We find that the most common issues finding
information are part of an ongoing reproducibility crisis[690, 691] where
machine learning researchers do not publish their source code or data.
Nevertheless, third party source code is sometimes available. Alternatively,
ANNs can reconstruct source code from some research papers[692].

###  2.7 Scientific Publishing

The number of articles published per year in reputable peer-reviewed[693, 694,
695, 696, 697] scientific journals[698, 699] has roughly doubled every nine
years since the beginning of modern science[700]. There are now over 25000
peer-reviewed journals[699] with varying impact factors[701, 702, 703], scopes
and editorial policies. Strategies to find the best journal to publish in
include using online journal finders[704], seeking the advice of learned
colleagues, and considering where similar research has been published.
Increasingly, working papers are also being published in open access preprint
archives[705, 706, 707]. For example, the arXiv[708, 709] is a popular
preprint archive for computer science, mathematics, and physics. Advantages of
preprints include ensuring that research is openly available, increasing
discovery and citations[710, 711, 712, 713, 714], inviting timely scientific
discussion, and raising awareness to reduce unnecessary duplication of
research. Many publishers have adapted to the popularity of preprints[705] by
offering open access publication options[715, 716, 717, 718] and allowing, and
in some cases encouraging[719], the prior publication of preprints. Indeed,
some journals are now using the arXiv to host their publications[720].

A variety of software can help authors prepare scientific manuscripts[721].
However, we think the most essential software is a document preparation
system. Most manuscripts are prepared with Microsoft Word[722] or similar
software[723]. However, Latex[724, 725, 726] is a popular alternative among
computer scientists, mathematicians and physicists[727]. Most electron
microscopists at the University of Warwick appear to prefer Word. A 2014
comparison of Latex and Word found that Word is better at all tasks other than
typesetting equations[728]. However, in 2017 it become possible to use Latex
to typeset equations within Word[727]. As a result, Word appears to be more
efficient than Latex for most manuscript preparation. Nevertheless, Latex may
still be preferable to authors who want fine control over typesetting[729,
730]. As a compromise, we use Overleaf[731] to edit Latex source code, then
copy our code to Word as part of proofreading to identify issues with grammar
and wording.

Figure 5:  Reciprocity of TEM and STEM electron optics.

##  3 Electron Microscopy

An electron microscope is an instrument that uses electrons as a source of
illumination to enable the study of small objects. Electron microscopy
competes with a large range of alternative techniques for material
analysis[732, 733, 734], including atomic force microscopy[735, 736, 737]
(AFM); Fourier transformed infrared (FTIR) spectroscopy[738, 739]; nuclear
magnetic resonance[740, 741, 742, 743] (NMR); Raman spectroscopy[744, 745,
746, 747, 748, 749, 750]; and x-ray diffraction[751, 752] (XRD),
dispersion[753], fluorescence[754, 755] (XRF), and photoelectron
spectroscopy[756, 757] (XPS). Quantitative advantages of electron microscopes
can include higher resolution and depth of field, and lower radiation damage
than light microscopes[758]. In addition, electron microscopes can record
images, enabling visual interpretation of complex structures that may
otherwise be intractable. This section will briefly introduce varieties of
electron microscopes, simulation software, and how electron microscopes can
interface with ANNs.

###  3.1 Microscopes

Figure 6:  Numbers of results per year returned by Dimensions.ai abstract
searches for SEM, TEM, STEM, STM and REM qualitate their popularities. The
number of results for 2020 is extrapolated using the mean rate before 14th
July 2020.

There are a variety of electron microscopes that use different illumination
mechanisms. For example, reflection electron microscopy[759, 760] (REM),
scanning electron microscopy[761, 762] (SEM), scanning transmission electron
microscopy[763, 764] (STEM), scanning tunnelling microscopy[765, 766] (STM),
and transmission electron microscopy[767, 768, 769] (TEM). To roughly gauge
popularities of electron microscope varieties, we performed abstract searches
with Dimenions.ai[770, 771, 651, 772] for their abbreviations followed by
“electron microscopy” e.g. “REM electron microscopy”. Numbers of results per
year in figure 6 qualitate that popularity increases in order REM, STM, STEM,
TEM, then SEM. It may be tempting to attribute the popularity of SEM over TEM
to the lower cost of SEM[773], which increases accessibility. However, a range
of considerations influence the procurement of electron microscopes[774] and
hourly pricing at universities[775, 776, 777, 778, 779] is similar for SEM and
TEM.

In SEM, material surfaces are scanned by sequential probing with a beam of
electrons, which are typically accelerated to 0.2-40 keV. The SEM detects
quanta emitted from where the beam interacts with the sample. Most SEM imaging
uses low-energy secondary electrons. However, reflection electron
microscopy[759, 760] (REM) uses elastically backscattered electrons and is
often complimented by a combination of reflection high-energy electron
diffraction[780, 781, 782] (RHEED), reflection high-energy electron loss
spectroscopy[783, 784] (RHEELS) and spin-polarized low-energy electron
microscopy[785, 786, 787] (SPLEEM). Some SEMs also detect Auger electrons[788,
789]. To enhance materials characterization, most SEMs also detect light. The
most common light detectors are for cathodoluminescence and energy dispersive
r-ray[790, 791] (EDX) spectroscopy. Nonetheless, some SEMs also detect
Bremsstrahlung radiation[792].

Alternatively, TEM and STEM detect electrons transmitted through specimens. In
conventional TEM, a single region is exposed to a broad electron beam. In
contrast, STEM uses a fine electron beam to probe a series of discrete probing
locations. Typically, electrons are accelerated across a potential difference
to kinetic energies, $E_{k}$, of 80-300 keV. Electrons also have rest energy
$E_{\text{e}}=m_{\text{e}}c^{2}$, where $m_{\text{e}}$ is electron rest mass
and $c$ is the speed of light. The total energy, $E_{t}=E_{\text{e}}+E_{k}$,
of free electrons is related to their rest mass energy by a Lorentz factor,
$\gamma$,

$\displaystyle E_{t}$ $\displaystyle=\gamma m_{\text{e}}c^{2}\,,$ (1)
$\displaystyle\gamma$ $\displaystyle=(1-v^{2}/c^{2})^{1/2}\,,$ (2)

where $v$ is the speed of electron propagation in the rest frame of an
electron microscope. Electron kinetic energies in TEM and STEM are comparable
to their rest energy, $E_{\text{e}}=511$ keV[793], so relativistic
phenomena[794, 795] must be considered to accurately describe their dynamics.

Electrons exhibit wave-particle duality[350, 351]. Thus, in an ideal electron
microscope, the maximum possible detection angle, $\theta$, between two point
sources separated by a distance, $d$, perpendicular to the electron
propagation direction is diffraction-limited. The resolution limit for imaging
can be quantified by Rayleigh’s criterion[796, 797, 798]

$\theta\simeq 1.22\frac{\lambda}{d},$ (3)

where resolution increases with decreasing wavelength, $\lambda$. Electron
wavelength decreases with increasing accelerating voltage, as described by the
relativistic de Broglie relation[799, 800, 801],

$\lambda=hc\left(E_{k}^{2}+2E_{\text{e}}E_{k}\right)^{-1/2}\,,$ (4)

where $h$ is Planck’s constant[793]. Electron wavelengths for typical
acceleration voltages tabulated by JEOL are in picometres[802]. In comparison,
Cu K-$\alpha$ x-rays, which are often used for XRD, have wavelengths near 0.15
nm[803]. In theory, electrons can therefore achieve over 100$\times$ higher
resolution than x-rays. Electrons and x-rays are both ionizing; however,
electrons often do less radiation damage to thin specimens than x-rays[758].
Tangentially, TEM and STEM often achieve over 10 times higher resolution than
SEM[804] as transmitted electrons in TEM and STEM are easier to resolve than
electrons returned from material surfaces in SEM.

In practice, TEM and STEM are also limited by incoherence[805, 806, 807]
introduced by inelastic scattering, electron energy spread, and other
mechanisms. TEM and STEM are related by an extension of Helmholtz
reciprocity[808, 809] where the source plane in a TEM corresponds to the
detector plane in a STEM[810], as shown in figure 5. Consequently, TEM
coherence is limited by electron optics between the specimen and image,
whereas STEM coherence is limited by the illumination system. For conventional
TEM and STEM imaging, electrons are normally incident on a specimen[811].
Advantages of STEM imaging can include higher contrast and resolution than TEM
imaging, and lower radiation damage[812]. As a result, STEM is increasing
being favoured over TEM for high-resolution studies. However, we caution that
definitions of TEM and STEM resolution can be disparate[813].

In addition to conventional imaging, TEM and STEM include a variety of
operating modes for different applications. For example, TEM operating
configurations include electron diffraction[814]; convergent beam electron
diffraction[815, 816, 817] (CBED); tomography[818, 819, 820, 821, 822, 823,
824, 825, 826]; and bright field[827, 828, 768, 829], dark field[768, 829] and
annular dark field[830] imaging. Similarly, STEM operating configurations
include differential phase contrast[831, 832, 833, 834]; tomography[818, 820,
822, 823]; and bright field[835, 836] or dark field[837] imaging. Further,
electron cameras[838, 839] are often supplemented by secondary signal
detectors. For example, elemental composition is often mapped by EDX
spectroscopy, electron energy loss spectroscopy[840, 841] (EELS) or wavelength
dispersive spectroscopy[842, 843] (WDS). Similarly, electron backscatter
diffraction[844, 845, 846] (EBSD) can detect strain[847, 848, 849] and
crystallization[850, 851, 852].

###  3.2 Contrast Simulation

The propagation of electron wavefunctions though electron microscopes can be
described by wave optics[136]. Further, the most popular approach to modelling
measurement contrast is multislice simulation[853, 854], where an electron
wavefunction is iteratively perturbed as it travels through a model of a
specimen. Multislice software for electron microscopy includes ACEM[854, 855,
856], clTEM[857, 858], cudaEM[859], Dr. Probe[860, 861], EMSoft[862, 863],
JEMS[864], JMULTIS[865], MULTEM[866, 867, 868], NCEMSS[869, 870], NUMIS[871],
Prismatic[872, 873, 874], QSTEM[875], SimulaTEM[876], STEM-CELL[877],
Tempas[878], and xHREM[879, 880, 881, 882, 883, 884]. We find that most
multislice software is a recreation and slight modification of common
functionality, possibly due to a publish-or-perish culture in academia[885,
886, 887]. Bloch-wave simulation[888, 889, 890, 891, 854, 892] is an
alternative to multislice simulation that can reduce computation time and
memory requirements for crystalline materials[893].

###  3.3 Automation

Most modern electron microscopes support Gatan Microscopy Suite (GMS)
Software[894]. GMS enables electron microscopes to be programmed by
DigitalMicrograph Scripting, a propriety Gatan programming language akin to a
simplified version of C++. A variety of DigitalMicrograph scripts, tutorials
and related resources are available from Dave Mitchell’s DigitalMicrograph
Scripting Website[679, 680], FELMI/ZFE’s Script Database[895] and Gatan’s
Script library[896]. Some electron microscopists also provide
DigitalMicrograph scripting resources on their webpages[897, 898, 899].
However, DigitalMicrograph scripts are slow insofar that they are interpreted
at runtime, and there is limited native functionality for parallel and
distributed computing. As a result, extensions to DigitalMicrograph scripting
are often developed in other programming languages that offer more
functionality.

Historically, most extensions were developed in C++[900]. This was problematic
as there is limited documentation, the standard approach used outdated C++
software development kits such as Visual Studio 2008, and programming
expertise required to create functions that interface with DigitalMicrograph
scripts limited accessibility. To increase accessibility, recent versions of
GMS now support python[901]. This is convenient as it enables ANNs developed
with python to readily interface with electron microscopes. For ANNs developed
with C++, users have the option to either create C++ bindings for
DigitalMicrograph script or for python. Integrating ANNs developed in other
programming languages is more complicated as DigitalMicrograph provides almost
no support. However, that complexity can be avoided by exchanging files from
DigitalMicrograph script to external libraries via a random access memory
(RAM) disk[902] or secondary storage[903].

Increasing accessibility, there are collections of GMS plugins with GUIs for
automation and analysis[904, 897, 898, 899]. In addition, various individual
plugins are available[905, 906, 907, 908, 909]. Some plugins are open source,
so they can be adapted to interface with ANNs. However, many high-quality
plugins are proprietary and closed source, limiting their use to automation of
data collection and processing. Plugins can also be supplemented by a variety
of libraries and interfaces for electron microscopy signal processing. For
example, popular general-purpose software includes ImageJ[910],
pycroscopy[527, 528] and HyperSpy[529, 530]. In addition, there are
directories for tens of general-purpose and specific electron microscopy
programs[911, 912, 913].

##  4 Components

Most modern ANNs are configured from a variety of DLF components. To take
advantage of hardware accelerators[62], most ANNs are implemented as sequences
of parallelizable layers of tensor operations[914]. Layers are often
parallelized across data and may be parallelized across other dimensions[915].
This section introduces popular nonlinear activation functions, normalization
layers, convolutional layers, and skip connections. To add insight, we provide
comparative discussion and address some common causes of confusion.

###  4.1 Nonlinear Activation

In general, DNNs need multiple layers to be universal approximators[37, 38,
39, 40, 41, 42, 43, 44, 45]. Nonlinear activation functions[916, 917] are
therefore essential to DNNs as successive linear layers can be contracted to a
single layer. Activation functions separate artificial neurons, similar to
biological neurons[918]. To learn efficiently, most DNNs are tens or hundreds
of layers deep[47, 919, 920, 921]. High depth increases representational
capacity[47], which can help training by gradient descent as DNNs evolve as
linear models[922] and nonlinearities can create suboptimal local minima where
data cannot be fit by linear models[923]. There are infinitely many possible
activation functions. However, most activation functions have low polynomial
order, similar to physical Hamiltonians[47].

Most ANNs developed for electron microscopy are for image processing, where
the most popular nonlinearities are rectifier linear units[924, 925] (ReLUs).
The ReLU activation, $f(x)$, of an input, $x$, and its gradient,
$\partial_{x}f(x)$, are

$f(x)=\max(0,x)$ (5a) |  |  $\frac{\partial f(x)}{\partial x}=\begin{cases}0,&\text{if }x\leq 0\\\ 1,&\text{if }x>0\end{cases}$ (5b)
---|---|---

Popular variants of ReLUs include Leaky ReLU[926],

$f(x)=\max(\alpha x,x)$ (6a) |  |  $\frac{\partial f(x)}{\partial x}=\begin{cases}\alpha,&\text{if }x\leq 0\\\ 1,&\text{if }x>0\end{cases}$ (6b)
---|---|---

where $\alpha$ is a hyperparameter, parametric ReLU[22] (PreLU) where $\alpha$
is a learned parameter, dynamic ReLU where $\alpha$ is a learned function of
inputs[927], and randomized leaky ReLU[928] (RReLU) where $\alpha$ is chosen
randomly. Typically, learned PreLU $\alpha$ are higher the nearer a layer is
to ANN inputs[22]. Motivated by limited comparisons that do not show a clear
performance difference between ReLU and leaky ReLU[929], some blogs[930] argue
against using leaky ReLU due to its higher computational requirements and
complexity. However, an in-depth comparison found that leaky ReLU variants
consistently slightly outperform ReLU[928]. In addition, the non-zero gradient
of leaky ReLU for $x\leq 0$ prevents saturating, or “dying”, ReLU[931, 932,
933], where the zero gradient of ReLUs stops learning.

There are a variety of other piecewise linear ReLU variants that can improve
performance. For example, ReLU$h$ activations are limited to a threshold[934],
$h$, so that

$f(x)=\min(\max(0,x),h)$ (7a) |  |  $\frac{\partial f(x)}{\partial x}=\begin{cases}0,&\text{if }x\leq 0\\\ 1,&\text{if }0<x\leq h\\\ 0,&\text{if }x>h\end{cases}$ (7b)
---|---|---

Thresholds near $h=6$ are often effective, so popular choice is ReLU6. Another
popular activation is concatenated ReLU[935] (CReLU), which is the
concatenation of $\text{ReLU}(x)$ and $\text{ReLU}(-x)$. Other ReLU variants
include adaptive convolutional[936], bipolar[937], elastic[938], and
Lipschitz[939] ReLUs. However, most ReLU variants are uncommon as they are
more complicated than ReLU and offer small, inconsistent, or unclear
performance gains. Moreover, it follows from the universal approximator
theorems[37, 38, 39, 40, 41, 42, 43, 44, 45] that disparity between ReLU and
its variants approaches zero as network depth increases.

In shallow networks, curved activation functions with non-zero Hessians often
accelerate convergence and improve performance. A popular activation is the
exponential linear unit[940] (ELU),

$f(x)=\begin{cases}\alpha(\exp(x)-1),&\text{if }x\leq 0\\\ x,&\text{if }x\geq 0\end{cases}$ (8a) |  |  $\frac{\partial f(x)}{\partial x}=\begin{cases}\alpha\exp(x),&\text{if }x\leq 0\\\ 1,&\text{if }x\geq 0\end{cases}$ (8b)
---|---|---

where $\alpha$ is a learned parameter. Further, a scaled ELU[941] (SELU),

$f(x)=\begin{cases}\lambda\alpha(\exp(x)-1),&\text{if }x\leq 0\\\ \lambda x,&\text{if }x\geq 0\end{cases}$ (9a) |  |  $\frac{\partial f(x)}{\partial x}=\begin{cases}\lambda\alpha\exp(x),&\text{if }x\leq 0\\\ \lambda,&\text{if }x\geq 0\end{cases}$ (9b)
---|---|---

with fixed $\alpha=1.67326$ and scale factor $\lambda=1.0507$ can be used to
create self-normalizing neural networks (SNNs). A SNN cannot be derived from
ReLUs or most other activation functions. Activation functions with curvature
are especially common in ANNs with only a couple of layers. For example,
activation functions in radial basis function (RBF) networks[942, 943, 944,
945], which are efficient universal approximators, are often Gaussians,
multiquadratics, inverse multiquadratics, or square-based RBFs[946].
Similarly, support vector machines[947, 948, 949] (SVMs) often use RBFs, or
sigmoids,

$f(x)=\frac{1}{1+\exp(-x)}$ (10a) |  |  $\frac{\partial f(x)}{\partial x}=f(x)\left(1-f(x)\right)$ (10b)
---|---|---

Sigmoids can also be applied to limit the support of outputs. Unscaled, or
“logistic”, sigmoids are often denoted $\sigma(x)$ and are related to $\tanh$
by $\tanh(x)=2\sigma(2x)-1$. To avoid expensive $\exp(-x)$ in the computation
of tanh, we recommend K-tanH[950], LeCun tanh[951], or piecewise linear
approximation[952, 953].

The activation functions introduced so far are scalar functions than can be
efficiently computed in parallel for each input element. However, functions of
vectors, $\textbf{x}=\\{x_{1},x_{2},...\\}$, are also popular. For example,
softmax activation[954],

$f(\textbf{x})=\frac{\exp(\textbf{x})}{\text{sum}(\exp(\textbf{x}))}$ (11a) |  |  $\frac{f(\textbf{x})}{\partial x_{j}}=\sum_{i}f(\textbf{x})_{i}(\delta_{ij}-f(\textbf{x})_{j})$ (11b)
---|---|---

is often applied before computing cross-entropy losses for classification
networks. Similarly, L$n$ vector normalization,

$f(\textbf{x})=\frac{\textbf{x}}{||\textbf{x}||_{n}}$ (12a) |  |  $\frac{f(\textbf{x})}{\partial x_{j}}=\frac{1}{||\textbf{x}||_{n}}\left(1-\frac{x_{j}^{n}}{||\textbf{x}||_{n}^{n}}\right)$ (12b)
---|---|---

with $n=2$ is often applied to vectors to ensure that they lie on a unit
sphere[349]. Finally, max pooling[955, 956],

$f(\textbf{x})=\max(\textbf{x})$ (13a) |  |  $\frac{f(\textbf{x})}{\partial x_{j}}=\begin{cases}1,&\text{if }j=\text{argmax}(\textbf{x})\\\ 0,&\text{if }j\neq\text{argmax}(\textbf{x})\end{cases}$ (13b)
---|---|---

is another popular multivariate activation function that is often used for
downsampling. However, max pooling has fallen out of favour as it is often
outperformed by strided convolutional layers[957]. Other vector activation
functions include squashing nonlinearities for dynamic routing by agreement in
capsule networks[958] and cosine similarity[959].

There are many other activation functions that are not detailed here for
brevity. Further, finding new activation functions is an active area of
research[960, 961]. Notable variants include choosing activation functions
from a set before training[962, 963] and learning activation functions[962,
964, 965, 966, 967]. Activation functions can also encode probability
distributions[968, 969, 970] or include noise[953]. Finally, there are a
variety of other deterministic activation functions[971, 961]. In electron
microscopy, most ANNs enable new or enhance existing applications.
Subsequently, we recommend using computationally efficient and established
activation functions unless there is a compelling reason to use a specialized
activation function.

###  4.2 Normalization

Normalization[972, 973, 974] standardizes signals, which can accelerate
convergence by gradient descent and improve performance. Batch
normalization[975, 976, 977, 978, 979, 980] is the most popular normalization
layer in image processing DNNs trained with minibatches of $N$ examples.
Technically, a “batch” is an entire training dataset and a “minibatch” is a
subset; however, the “mini” is often omitted where meaning is clear from
context. During training, batch normalization applies a transform,

$\displaystyle\mu_{B}$
$\displaystyle=\frac{1}{N}\sum\limits_{i=1}^{N}x_{i}\,,$ (14)
$\displaystyle\sigma_{B}^{2}$
$\displaystyle=\frac{1}{N}\sum\limits_{i=1}^{N}(x_{i}-\mu_{B})^{2}\,,$ (15)
$\displaystyle\hat{\textbf{x}}$
$\displaystyle=\frac{\textbf{x}-\mu_{B}}{(\sigma_{B}^{2}+\epsilon)^{1/2}}\,,$
(16) $\displaystyle\text{BatchNorm}(\textbf{x})$
$\displaystyle=\gamma\hat{\textbf{x}}+\beta\,,$ (17)

where $\textbf{x}=\\{x_{1},...,x_{N}\\}$ is a batch of layer inputs, $\gamma$
and $\beta$ are a learnable scale and shift, and $\epsilon$ is a small
constant added for numerical stability. During inference, batch normalization
applies a transform,

$\displaystyle\text{BatchNorm}(\textbf{x})=\frac{\gamma}{(\text{Var}[x]+\epsilon)^{1/2}}\textbf{x}+\left(\beta-\frac{\gamma\text{E}[x]}{(\text{Var}[x]+\epsilon)^{1/2}}\right)\,,$
(18)

where E[x] and Var[x] are expected batch means and variances. For convenience,
E[x] and Var[x] are often estimated with exponential moving averages that are
tracked during training. However, E[x] and Var[x] can also be estimated by
propagating examples through an ANN after training.

Increasing batch size stabilizes learning by averaging destabilizing loss
spikes over batches[261]. Batched learning also enables more efficient
utilization of modern hardware accelerators. For example, larger batch sizes
improve utilization of GPU memory bandwidth and throughput[981, 391, 982].
Using large batches can also be more efficient than many small batches when
distributing training across multiple CPU clusters or GPUs due to
communication overheads. However, the performance benefits of large batch
sizes can come at the cost of lower test accuracy as training with large
batches tends to converge to sharper minima[983, 984]. As a result, it is
often best not to use batch sizes higher than $N\approx 32$ for image
classification[985]. However, learning rate scaling[541] and layer-wise
adaptive learning rates[986] can increase accuracy of training with fixed
larger batch sizes. Batch size can also be increased throughout training
without compromising accuracy[987] to exploit effective learning rates being
inversely proportional to batch size[987, 541]. Alternatively, accuracy can be
improved by creating larger batches from replicated instances of training
inputs with different data augmentations[988].

There are a few caveats to batch normalization. Originally, batch
normalization was applied before activation[976]. However, applying batch
normalization after activation often slightly improves performance[989, 990].
In addition, training can be sensitive to the often-forgotten $\epsilon$
hyperparameter[991] in equation 16. Typically, performance decreases as
$\epsilon$ is increased above $\epsilon\approx 0.001$; however, there is a
sharp increase in performance around $\epsilon=0.01$ on ImageNet. Finally, it
is often assumed that batches are representative of the training dataset. This
is often approximated by shuffling training data to sample independent and
identically distributed (i.i.d.) samples. However, performance can often be
improved by prioritizing sampling[992, 993]. We observe that batch
normalization is usually effective if batch moments, $\mu_{B}$ and
$\sigma_{B}$, have similar values for every batch.

Batch normalization is less effective when training batch sizes are small, or
do not consist of independent samples. To improve performance, standard
moments in equation 16 can be renormalized[994] to expected means, $\mu$, and
standard deviations, $\sigma$,

$\displaystyle\hat{\textbf{x}}$ $\displaystyle\leftarrow
r\hat{\textbf{x}}+d\,,$ (19) $\displaystyle r$
$\displaystyle=\text{clip}_{[1/r_{\text{max}},r_{\text{max}}]}\left(\frac{\sigma_{B}}{\sigma}\right)\,,$
(20) $\displaystyle d$
$\displaystyle=\text{clip}_{[-d_{\text{max}},d_{\text{max}}]}\left(\frac{\mu_{B}-\mu}{\sigma}\right)\,,$
(21)

where gradients are not backpropagated with respect to (w.r.t.) the
renormalization parameters, $r$ and $d$. Moments, $\mu$ and $\sigma$ are
tracked by exponential moving averages and clipping to $r_{\text{max}}$ and
$d_{\text{max}}$ improves learning stability. Usually, clipping values are
increased from starting values of $r_{\text{max}}=1$ and $d_{\text{max}}=0$,
which correspond to batch normalization, as training progresses. Another
approach is virtual batch normalization[995] (VBN), which estimates $\mu$ and
$\sigma$ from a reference batch of samples and does not require clipping.
However, VBN is computationally expensive as it requires computing a second
batch of statistics at every training iteration. Finally, online[996] and
streaming[974] normalization enable training with small batch sizes by replace
$\mu_{B}$ and $\sigma_{B}$ in equation 16 with their exponential moving
averages.

There are alternatives to the $L_{2}$ batch normalization of equations 14-18
that standardize to different Euclidean norms. For example, $L_{1}$ batch
normalization[997] computes

$\displaystyle s_{1}$
$\displaystyle=\frac{1}{N}\sum\limits_{i=1}^{N}|x_{i}-\mu_{B}|\,,$ (22)
$\displaystyle\hat{\textbf{x}}$
$\displaystyle=\frac{\textbf{x}-\mu_{B}}{C_{L_{1}}s_{1}}\,,$ (23)

where $C_{L_{1}}=(\pi/2)^{1/2}$. Although the $C_{L_{1}}$ factor could be
learned by ANN parameters, its inclusion accelerates convergence of the
original implementation of $L_{1}$ batch normalization[997]. Another
alternative is $L_{\infty}$ batch normalization[997], which computes

$\displaystyle s_{\infty}$
$\displaystyle=\text{mean}(\text{top}_{k}(|\textbf{x}-\mu_{B}|))\,,$ (24)
$\displaystyle\hat{\textbf{x}}$
$\displaystyle=\frac{\textbf{x}-\mu_{B}}{C_{L_{\infty}}s_{\infty}}\,,$ (25)

where $C_{L_{\infty}}$ is a scale factor, and $\text{top}_{k}(\textbf{x})$
returns the $k$ highest elements of x. Hoffer et al suggest $k=10$[997]. Some
$L_{1}$ batch normalization proponents claim that $L_{1}$ batch normalization
outperforms[975] or achieves similar performance[997] to $L_{2}$ batch
normalization. However, we found that $L_{1}$ batch normalization often lowers
performance in our experiments. Similarly, $L_{\infty}$ batch normalization
often lowers performance[997]. Overall, $L_{1}$ and $L_{\infty}$ batch
normalization do not appear to offer a substantial advantage over $L_{2}$
batch normalization.

Figure 7:  Visual comparison of various normalization methods highlighting
regions that they normalize. Regions can be normalized across batch, feature
and other dimensions, such as height and width.

A variety of layers normalize samples independently, including layer,
instance, and group normalization. They are compared with batch normalization
in figure 7. Layer normalization[998, 999] is a transposition of batch
normalization that is computed across feature channels for each training
example, instead of across batches. Batch normalization is ineffective in
RNNs; however, layer normalization of input activations often improves
accuracy[998]. Instance normalization[1000] is an extreme version of layer
normalization that standardizes each feature channel for each training
example. Instance normalization was developed for style transfer[1001, 1002,
1003, 1004, 1005] and makes ANNs insensitive to input image contrast. Group
normalization[1006] is intermediate to instance and layer normalization
insofar that it standardizes groups of channels for each training example.

The advantages of a set of multiple different normalization layers, $\Omega$,
can be combined by switchable normalization[1007, 1008], which standardizes to

$\displaystyle\hat{\textbf{x}}$
$\displaystyle=\frac{\textbf{x}-\sum\limits_{z\in\Omega}\lambda_{z}^{\mu}\mu_{z}}{\sum\limits_{z\in\Omega}\lambda_{z}^{\sigma}\sigma_{z}}\,,$
(26)

where $\mu_{z}$ and $\sigma_{z}$ are means and standard deviations computed by
normalization layer $z$, and their respective importance ratios,
$\lambda_{z}^{\mu}$ and $\lambda_{z}^{\sigma}$, are trainable parameters that
are softmax activated to sum to unity. Combining batch and instance
normalization statistics outperforms batch normalization for a range of
computer vision tasks[1009]. However, most layers strongly weighted either
batch or instance normalization, with most preferring batch normalization.
Interestingly, combining batch, instance and layer normalization
statistics[1007, 1008] results in instance normalization being preferred in
earlier layers, whereas layer normalization was preferred in the later layers,
and batch normalization was preferred in the middle layers. Smaller batch
sizes lead to a preference towards layer normalization and instance
normalization. Limitingly, using multiple normalization layers increases
computation. To limit expense, we therefore recommend either defaulting to
batch normalization, or progressively using single instance, batch, or layer
normalization layers.

A significant limitation of batch normalization is that it is not effective in
RNNs. This is a limited issue as most electron microscopists are developing
CNNs for image processing. However, we anticipate that RNNs may become more
popular in electron microscopy following the increasing popularity of
reinforcement learning[1010]. In addition to general-purpose alternatives to
batch normalization that are effective in RNNs, such as layer normalization,
there are a variety of dedicated normalization schemes. For example, recurrent
batch normalization[1011, 1012] uses distinct normalization layers for each
time step. Alternatively, batch normalized RNNs[1013] only have normalization
layers between their input and hidden states. Finally, online[996] and
streaming[974] normalization are general-purpose solutions that improve the
performance of batch normalization in RNNs by applying batch normalization
based on a stream of past batch statistics.

Normalization can also standardize trainable weights, w. For example, weight
normalization[1014],

$\text{WeightNorm}(\textbf{w})=\frac{g}{||\textbf{w}||_{2}}\textbf{w}\,,$ (27)

decouples the L2 norm, $g$, of a variable from its direction. Similarly,
weight standardization[1015] subtracts means from variables and divides them
by their standard deviations,

$\text{WeightStd}(\textbf{w})=\frac{\textbf{w}-\text{mean}(\textbf{w})}{\text{std}(\textbf{w})}\,,$
(28)

similar to batch normalization. Weight normalization often outperforms batch
normalization at small batch sizes. However, batch normalization consistently
outperforms weight normalization at larger batch sizes used in practice[1016].
Combining weight normalization with running mean-only batch normalization can
accelerate convergence[1014]. However, similar final accuracy can be achieved
without mean-only batch normalization at the cost of slower convergence, or
with the use of zero-mean preserving activation functions[937, 997]. To
achieve similar performance to batch normalization, norm-bounded weight
normalization[997] can be applied to DNNs with scale-invariant activation
functions, such as ReLU. Norm-bounded weight normalization fixes $g$ at
initialization to avoid learning instability[1016, 997], and scales outputs
with the final DNN layer.

Limitedly, weight normalization encourages the use of a small number of
features to inform activations[1017]. To encourage higher feature utilization,
spectral normalization[1017],

$\text{SpectralNorm}(\textbf{w})=\frac{\textbf{w}}{\sigma(\textbf{w})}\,,$
(29)

divides tensors by their spectral norms, $\sigma(\textbf{w})$. Further,
spectral normalization limits Lipschitz constants[1018], which often improves
generative adversarial network[197, 198, 199, 200] (GAN) training by bounding
backpropagated discriminator gradients[1017]. The spectral norm of v is the
maximum value of a diagonal matrix, $\boldsymbol{\Sigma}$, in the singular
value decomposition[1019, 1020, 1021, 1022] (SVG),

$\textbf{v}=\textbf{U}\boldsymbol{\Sigma}\textbf{V}^{*}\,,$ (30)

where U and V are orthogonal matrices of orthonormal eigenvectors for
$\textbf{v}\textbf{v}^{T}$ and $\textbf{v}^{T}\textbf{v}$, respectively. To
minimize computation, $\boldsymbol{\sigma}(\textbf{w})$ is often approximated
by the power iteration method[1023, 1024],

$\displaystyle\hat{\textbf{v}}$
$\displaystyle\leftarrow\frac{\textbf{w}^{\text{T}}\hat{\textbf{u}}}{||\textbf{w}^{\text{T}}\hat{\textbf{u}}||_{2}}\,,$
(31) $\displaystyle\hat{\textbf{u}}$
$\displaystyle\leftarrow\frac{\textbf{w}\hat{\textbf{v}}}{||\textbf{w}\hat{\textbf{v}}||_{2}}\,,$
(32) $\displaystyle\sigma(\textbf{w})$
$\displaystyle\simeq\hat{\textbf{u}}^{T}\textbf{w}\hat{\textbf{v}}\,,$ (33)

where one iteration of equations 31-32 per training iteration is usually
sufficient.

Parameter normalization can complement or be combined with signal
normalization. For example, scale normalization[1025],

$\text{ScaleNorm}(\textbf{x})=\frac{g}{||\textbf{x}||_{2}}\textbf{x}\,,$ (34)

learns scales, $g$, for activations, and is often combined with weight
normalization[1026, 1014] in transformer networks. Similarly, cosine
normalization[959],

$\text{CosineNorm}(\textbf{x})=\frac{\textbf{w}}{||\textbf{w}||_{2}}\cdot\frac{\textbf{x}}{||\textbf{x}||_{2}}\,,$
(35)

computes products of L2 normalized parameters and signals. Both scale and
cosine normalization can outperform batch normalization.

Figure 8:  Visualization of convolutional layers. a) Traditional convolutional
layer where output channels are sums of biases and convolutions of weights
with input channels. b) Depthwise separable convolutional layer where
depthwise convolutions compute one convolution with weights for each input
channel. Output channels are sums of biases and pointwise convolutions weights
with depthwise channels.

###  4.3 Convolutional Layers

A convolutional neural network[1027, 1028, 1029, 1030] (CNN) is trained to
weight convolutional kernels to exploit local correlations, such as spatial
correlations in electron micrographs[231]. Historically, the development of
CNNs was inspired by primate visual cortices[1031], where partially
overlapping neurons are only stimulated by visual stimuli within their
receptive fields. Based on this idea, Fukushima published his
Neocognitron[1032, 1033, 1034, 1035] in 1980. Convolutional formulations were
then published by Atlas et al in 1988 for a single-layer CNN[1036], and LeCun
et al in 1998 for a multi-layer CNN[1037, 1038]. Subsequently, GPUs were
applied to accelerate convolutions in 2010[1039], leading to a breakthrough in
classification performance on ImageNet with AlexNet in 2012[71]. Indeed, the
deep learning era is often partitioned into before and after AlexNet[19]. Deep
CNNs are now ubiquitous. For example, there are review papers on applications
of CNNs to action recognition in videos[1040], cytometry[1041], image and
video compression[1042, 1043], image background subtraction[1044], image
classification[272], image style transfer[1001], medical image analysis[1045,
1046, 1047, 1048, 334, 1049, 1050, 332, 333, 1051, 1052], object
detection[1053, 1054], semantic image segmentation[304, 334, 333, 332], and
text classification[1055].

In general, the convolution of two functions, $f$ and $g$, is

$(f*g)(x)\coloneqq\int\limits_{s\in\Omega}f(s)g(x-s)\mathop{}\\!\mathrm{d}s\,,$
(36)

and their cross-correlation is

$(f\circ
g)(x)\coloneqq\int\limits_{s\in\Omega}f(s)g(x+s)\mathop{}\\!\mathrm{d}s\,,$
(37)

where integrals have unlimited support, $\Omega$. In a CNN, convolutional
layers sum convolutions of feature channels with trainable kernels, as shown
in figure 8. Thus, $f$ and $g$ are discrete functions and the integrals in
equations 36-37 can be replaced with limited summations. Since cross-
correlation is equivalent to convolution if the kernel is flipped in every
dimension, and CNN kernels are usually trainable, convolution and cross-
correlation is often interchangeable in deep learning. For example, a
TensorFlow function named “tf.nn.convolution” computes cross-
correlations[1056]. Nevertheless, the difference between convolution and
cross-correlation can be source of subtle errors if convolutional layers from
a DLF are used in an image processing pipeline with static asymmetric kernels.

Figure 9:  Two 96$\times$96 electron micrographs a) unchanged, and filtered by
b) a 5$\times$5 symmetric Gaussian kernel with a 2.5 px standard deviation, c)
a 3$\times$3 horizontal Sobel kernel, and d) a 3$\times$3 vertical Sobel
kernel. Intensities in a) and b) are in [0, 1], whereas intensities in c) and
d) are in [-1, 1].

Kernels designed by humans[1057] are often convolved in image processing
pipelines. For example, convolutions of electron micrographs with Gaussian and
Sobel kernels are shown in figure 9. Gaussian kernels compute local averages,
blurring images and suppressing high-frequency noise. For example, a
5$\times$5 symmetric Gaussian kernel with a 2.5 px standard deviation is

$\begin{bmatrix}0.1689\\\ 0.2148\\\ 0.2326\\\ 0.2148\\\
0.1689\end{bmatrix}\begin{bmatrix}0.1689&0.2148&0.2326&0.2148&0.1689\end{bmatrix}=\begin{bmatrix}0.0285&0.0363&0.0393&0.0363&0.0285\\\
0.0363&0.0461&0.0500&0.0461&0.0363\\\ 0.0393&0.0500&0.0541&0.0500&0.0393\\\
0.0363&0.0461&0.0500&0.0461&0.0363\\\
0.0285&0.0363&0.0393&0.0363&0.0285\end{bmatrix}\,.$ (38)

Alternatives to Gaussian kernels for image smoothing[1058] include mean,
median and bilateral filters. Sobel kernels compute horizontal and vertical
spatial gradients that can be used for edge detection[1059]. For example,
3$\times$3 Sobel kernels are

$\begin{bmatrix}1\\\ 2\\\ 1\end{bmatrix}\begin{bmatrix}1&0&-1\end{bmatrix}=\begin{bmatrix}1&0&-1\\\ 2&0&-2\\\ 1&0&-1\end{bmatrix}$ (39a) |  |  $\begin{bmatrix}1\\\ 0\\\ -1\end{bmatrix}\begin{bmatrix}1&2&1\end{bmatrix}=\begin{bmatrix}1&2&1\\\ 0&0&0\\\ -1&-2&-1\end{bmatrix}$ (39b)
---|---|---

Alternatives to Sobel kernels offer similar utility, and include extended
Sobel[1060], Scharr[1061, 1062], Kayyali[1063], Roberts cross[1064] and
Prewitt[1065] kernels. Two-dimensional Gaussian and Sobel kernels are examples
of linearly separable, or “flattenable”, kernels, which can be split into two
one-dimensional kernels, as shown in equations 38-39b. Kernel separation can
decrease computation in convolutional layers by convolving separated kernels
in series, and CNNs that only use separable convolutions are effective[1066,
1067, 1068]. However, serial convolutions decrease parallelization and
separable kernels have fewer degrees of freedom, decreasing representational
capacity. Thus, separated kernels are usually at least 5$\times$5, and
separated 3$\times$3 kernels are unusual. Even-sized kernels, such as
2$\times$2 and 4$\times$4, are rare as symmetric padding is needed to avoid
information erosion caused by spatial shifts of feature maps[1069].

A traditional 2D convolutional layer maps inputs, $x^{\text{input}}$, with
height $H$, width, $W$, and depth, $D$, to

$x_{kij}^{\text{output}}=b_{k}+\sum\limits_{d=1}^{D}\sum\limits_{m=1}^{M}\sum\limits_{n=1}^{N}w_{dkmn}x_{d(i+m-1)(j+n-1)}^{\text{input}}\,,i\in[1,H-M+1]\,,j\in[1,W-N+1]\,,$
(40)

where $K$ output channels are indexed by $k\in[1,K]$, is the sum of a bias,
$b$, and convolutions of each input channel with $M\times N$ kernels with
weights, $w$. For clarity, a traditional convolutional layer is visualized in
figure 8a. Convolutional layers for 1D, 3D and higher-dimensional
kernels[1070] have a similar form to 2D kernels, where kernels are convolved
across each dimension. Most inputs to convolutional layers are padded[1071,
1072] to avoid reducing spatial resolutions by kernel sizes, which could
remove all resolution in deep networks. Padding is computationally inexpensive
and eases implementations of ANNs that would otherwise combine layers with
different sizes, such as FractalNet[1073], Inception[1074, 1075, 1076],
NASNet[1077], recursive CNNs[1078, 1079], and ResNet[1080]. Pre-padding inputs
results in higher performance than post-padding outputs[1081]. Following
AlexNet[71], most convolutional layers are padded with zeros for simplicity.
Reflection and replication padding achieve similar results to zero
padding[1072]. However, padding based on partial convolutions[1082]
consistently outperforms other methods[1072].

Convolutional layers are similar to fully connected layers used in multilayer
perceptrons[1083, 1084] (MLPs). For comparison with equation 40, a fully
connected, or “dense”, layer in a MLP computes

$x_{k}^{\text{output}}=b_{k}+\sum\limits_{d=1}^{D}w_{dk}x_{d}^{\text{input}}\,,$
(41)

where every input element is connected to every output element. Convolutional
layers reduce computation by making local connections within receptive fields
of convolutional kernels, and by convolving kernels rather than using
different weights at each input position. Intermediately, fully connected
layers can be regularized to learn local connections[1085]. Fully connected
layers are sometimes used at the middle of encoder-decoders[1086]. However,
such fully connected layers can often be replaced by multiscale atrous, or
“holey”, convolutions[955] in an atrous spatial pyramid pooling[306, 305]
(ASPP) module to decrease computation without a significant decrease in
performance. Alternatively, weights in fully connected layers can be
decomposed into multiple smaller tensors to decrease computation without
significantly decreasing performance[1087, 1088].

Convolutional layers can perform a variety of convolutional arithmetic[955].
For example, strided convolutions[1089] usually skip computation of outputs
that are not at multiples of an integer spatial stride. Most strided
convolutional layers are applied throughout CNNs to sequentially decrease
spatial extent, and thereby decrease computational requirements. In addition,
strided convolutions are often applied at the start of CNNs[1074, 539, 1075,
1076] where most input features can be resolved at a lower resolution than the
input. For simplicity and computational efficiency, stride is typically
constant within a convolutional layer; however, increasing stride away from
the centre of layers can improve performance[1090]. To increase spatial
resolution, convolutional layers often use reciprocals of integer
strides[1091]. Alternatively, spatial resolution can be increased by combining
interpolative upsampling with an unstrided convolutional layer[1092, 1093],
which can help to minimize output artefacts.

Convolutional layers couple the computation of spatial and cross-channel
convolutions. However, partial decoupling of spatial and cross-channel
convolutions by distributing inputs across multiple convolutional layers and
combining outputs can improve performance. Partial decoupling of convolutions
is prevalent in many seminal DNN architectures, including FractalNet[1073],
Inception[1074, 1075, 1076], NASNet[1077]. Taking decoupling to an extreme,
depthwise separable convolutions[539, 1094, 1095] shown in figure 8b compute
depthwise convolutions,

$\displaystyle x_{dij}^{\text{depth}}$
$\displaystyle=\sum\limits_{m=1}^{M}\sum\limits_{n=1}^{N}u_{dmn}x_{d(i+m-1)(j+n-1)}^{\text{input}}\,,i\in[1,H-M+1]\,,j\in[1,W-N+1]\,,$
(42)

then compute pointwise 1$\times$1 convolutions for $D$ intermediate channels,

$\displaystyle x_{kij}^{\text{output}}$
$\displaystyle=b_{k}+\sum\limits_{d=1}^{D}v_{dk}^{\text{point}}x_{dij}^{\text{depth}}\,,$
(43)

where $K$ output channels are indexed by $k\in[1,K]$. Depthwise convolution
kernels have weights, $u$, and the depthwise layer is often followed by extra
batch normalization before pointwise convolution to improve performance and
accelerate convergence[1094]. Increasing numbers of channels with pointwise
convolutions can increase accuracy[1094], at the cost of increased
computation. Pointwise convolutions are a special case of traditional
convolutional layers in equation 40 and have convolution kernel weights, $v$,
and add biases, $b$. Naively, depthwise separable convolutions require fewer
weight multiplications than traditional convolutions[1096, 1097]. However,
extra batch normalization and serialization of one convolutional layer into
depthwise and pointwise convolutional layers mean that depthwise separable
convolutions and traditional convolutions have similar computing times[539,
1097].

Most DNNs developed for computer vision use fixed-size inputs. Although fixed
input sizes are often regarded as an artificial constraint, it is similar to
animalian vision where there is an effectively constant number of retinal rods
and cones[1098, 1099, 1100]. Typically, the most practical approach to handle
arbitrary image shapes is to train a DNN with crops so that it can be tiled
across images. In some cases, a combination of cropping, padding and
interpolative resizing can also be used. To fully utilize unmodified variable
size inputs, a simple is approach to train convolutional layers on variable
size inputs. A pooling layer, such as global average pooling, can then be
applied to fix output size before fully connected or other layers that might
require fixed-size inputs. More involved approaches include spatial pyramid
pooling[1101] or scale RNNs[1102]. However, typical electron micrographs are
much larger than 299$\times$299, which often makes it unfeasible for electron
microscopists with a few GPUs to train high-performance DNNs on full-size
images. For comparison, Xception was trained on 299$\times$299 images with 60
K80 GPUs for over one month.

The Fourier transform[1103], $\hat{f}(k_{1},...,k_{N})$, at an $N$-dimensional
Fourier space vector, $\\{k_{1},...,k_{N}\\}$, is related to a function,
$f(x_{1},...,x_{N})$, of an $N$-dimensional signal domain vector,
$\\{x_{1},...,x_{N}\\}$, by

$\displaystyle\hat{f}(k_{1},...,k_{N})$
$\displaystyle=\left(\frac{|b|}{(2\pi)^{1-a}}\right)^{N/2}\int\limits_{-\infty}^{\infty}...\int\limits_{-\infty}^{\infty}f(x_{1},...,x_{N})\exp(+ibk_{1}x_{i}+...+ibk_{N}x_{N})\mathop{}\\!\mathrm{d}x_{1}...\mathop{}\\!\mathrm{d}x_{N}\,,$
(44) $\displaystyle f(x_{1},...,x_{N})$
$\displaystyle=\left(\frac{|b|}{(2\pi)^{1+a}}\right)^{N/2}\int\limits_{-\infty}^{\infty}...\int\limits_{-\infty}^{\infty}\hat{f}(k_{1},...,k_{N})\exp(-ibk_{1}x_{i}-...-ibk_{N}x_{N})\mathop{}\\!\mathrm{d}k_{1}...\mathop{}\\!\mathrm{d}k_{N}\,,$
(45)

where $\pi=3.141...$, and $i=(-1)^{1/2}$ is the imaginary number. Two
parameters, $a$ and $b$, can parameterize popular conventions that relate the
Fourier and inverse Fourier transforms. Mathematica documentation nominates
conventions[1104] for general applications $(a,b)$, pure mathematics $(1,-1)$,
classical physics $(-1,1)$, modern physics $(0,1)$, systems engineering
$(1,-1)$, and signal processing $(0,2\pi)$. We observe that most electron
microscopists follow the modern physics convention of $a=0$ and $b=1$;
however, the choice of convention is arbitrary and does not matter if it is
consistent within a project. For discrete functions, Fourier integrals are
replaced with summations that are limited to the support of a function.

Discrete Fourier transforms of uniformly spaced inputs are often computed with
a fast Fourier transform (FFT) algorithm, which can be parallelized for
CPUs[1105] or GPUs[65, 1106, 1107, 1108]. Typically, the speedup of FFTs on
GPUs over CPUs is higher for larger signals[1109, 1110]. Most popular FFTs are
based on the Cooley-Turkey algorithm[1111, 1112], which recursively divides
FFTs into smaller FFTs. We observe that some electron microscopists consider
FFTs to be limited to radix-2 signals that can be recursively halved; however,
FFTs can use any combination of factors for the sizes of recursively smaller
FFTs. For example, clFFT[1113] FFT algorithms support signal sizes that are
any sum of powers of 2, 3, 5, 7, 11 and 13.

Convolution theorems can decrease computation by enabling convolution in the
Fourier domain[1114]. To ease notation, we denote the Fourier transform of a
signal, I, by $\text{FT}(\textbf{I})$, and the inverse Fourier transform by
$\text{FT}^{-1}(\textbf{I})$. Thus, the convolution theorems for two signals,
$\textbf{I}_{1}$ and $\textbf{I}_{2}$, are[1115]

$\displaystyle\text{FT}(\textbf{I}_{1}*\textbf{I}_{2})$
$\displaystyle=\text{FT}(\textbf{I}_{1})\cdot\text{FT}(\textbf{I}_{2})\,,$
(46) $\displaystyle\text{FT}(\textbf{I}_{1}\cdot\textbf{I}_{2})$
$\displaystyle=\text{FT}(\textbf{I}_{1})*\text{FT}(\textbf{I}_{2})\,,$ (47)

where the signals can be feature channels and convolutional kernels. Fourier
domain convolutions,
$\textbf{I}_{1}*\textbf{I}_{2}=\text{FT}^{-1}\left(\text{FT}(\textbf{I}_{1})\cdot\text{FT}(\textbf{I}_{2})\right)$,
are increasingly efficient, relative to signal domain convolutions, as kernel
and image sizes increase[1114]. Indeed, Fourier domain convolutions are
exploited to enable faster training with large kernels in Fourier CNNs[1116,
1114]. However, Fourier CNNs are rare as most researchers use small 3$\times$3
kernels, following University of Oxford Visual Geometry Group (VGG)
CNNs[1117].

Figure 10:  Residual blocks where a) one, b) two, and c) three convolutional
layers are skipped. Typically, convolutional layers are followed by batch
normalization then activation.

###  4.4 Skip Connections

Residual connections[1080] add a signal after skipping ANN layers, similar to
cortical skip connections[1118, 1119]. Residuals improve DNN performance by
preserving gradient norms during backpropagation[1120, 537] and avoiding bad
local minima[1121] by smoothing DNN loss landscapes[1122]. In practice,
residuals enable DNNs to behave like an ensemble of shallow networks[1123]
that learn to iteratively estimate outputs[1124]. Mathematically, a residual
layer learns parameters, $\textbf{w}_{l}$, of a perturbative function,
$f_{l}(\textbf{x}_{l},\textbf{w}_{l})$, that maps a signal, $\textbf{x}_{l}$,
at depth $l$ to depth $l+1$,

$\textbf{x}_{l+1}=\textbf{x}_{l}+f_{l}(\textbf{x}_{l},\textbf{w}_{l})\,.$ (48)

Residuals were developed for CNNs[1080], and examples of residual connections
that skip one, two and three convolutional layers are shown in figure 10.
Nonetheless, residuals are also used in MLPs[1125] and RNNs[1126, 1127, 1128].
Representational capacity of perturbative functions increases as the number of
skipped layers increases. As result, most residuals skip two or three layers.
Skipping one layer rarely improves performance due to its low representational
capacity[1080].

There are a range of residual connection variants that can improve
performance. For example, highway networks[1129, 1130] apply a gating function
to skip connections, and dense networks[1131, 1132, 1133] use a high number of
residual connections from multiple layers. Another example is applying a
1$\times$1 convolutional layer to $x_{l}$ before addition[1080, 539] where
$f_{l}(x_{l},w_{l})$ spatially resizes or changes numbers of feature channels.
However, resizing with norm-preserving convolutional layers[1120] before
residual blocks can often improve performance. Finally, long additive[1134]
residuals that connect DNN inputs to outputs are often applied to DNNs that
learn perturbative functions.

A limitation of preserving signal information with residuals[1135, 1136] is
that residuals make DNNs learn perturbative functions, which can limit
accuracy of DNNs that learn non-perturbative functions if they do not have
many layers. Feature channel concatenation is an alternative approach that is
not perturbative, and that supports combination of layers with different
numbers of feature channels. In encoder-decoders, a typical example is
concatenating features computed near the start with layers near the end to
help resolve output features[316, 305, 306, 308]. Concatenation can also
combine embeddings of different[1137, 1138] or variants of[366] input features
from multiple DNNs. Finally, peephole connections in RNNs can improve
performance by using concatenation to combine cell state information with
other cell inputs[1139, 1140].

##  5 Architecture

There is a high variety of ANN architectures[4, 5, 6, 7] that are trained to
minimize losses for a range of applications. Many of the most popular ANNs are
also the simplest, and information about them is readily available. For
example, encoder-decoder[502, 503, 305, 306, 307, 308, 504] or classifier[272]
ANNs usually consist of single feedforward sequences of layers that map inputs
to outputs. This section introduces more advanced ANNs used in electron
microscopy, including actor-critics, GANs, RNNs, and variational autoencoders
(VAEs). These ANNs share weights between layers or consist of multiple
subnetworks. Other notable architectures include recursive CNNs[1078, 1079],
Network-in-Networks[1141] (NiNs), and transformers[1142, 1143]. Although they
will not be detailed in this review, their references may be good starting
points for research.

Figure 11:  Actor-critic architecture. An actor outputs actions based on input
states. A critic then evaluates action-state pairs to predict losses.

###  5.1 Actor-Critic

Most ANNs are trained by gradient descent using backpropagated gradients of a
differentiable loss function cf. section 6.1. However, some losses are not
differentiable. Examples include losses of actors directing their vision[1144,
1145], and playing competitive[24] or score-based[1146, 1147] computer games.
To overcome this limitation, a critic[1148] can be trained to predict
differentiable losses from action and state information, as shown in figure
11. If the critic does not depend on states, it is a surrogate loss
function[1149, 1150]. Surrogates are often fully trained before actor
optimization, whereas critics that depend on actor-state pairs are often
trained alongside actors to minimize the impact of catastrophic
forgetting[1151] by adapting to changing actor policies and experiences.
Alternatively, critics can be trained with features output by intermediate
layers of actors to generate synthetic gradients for backpropagation[1152].

Figure 12:  Generative adversarial network architecture. A generator learns to
produce outputs that look realistic to a discriminator, which learns to
predict whether examples are real or generated.

###  5.2 Generative Adversarial Network

Generative adversarial networks[197, 198, 199, 200] (GANs) consist of
generator and discriminator subnetworks that play an adversarial game, as
shown in figure 12. Generators learn to generate outputs that look realistic
to discriminators, whereas discriminators learn to predict whether examples
are real or generated. Most GANs are developed to generate visual media with
realistic characteristics. For example, partial STEM images infilled with a
GAN are less blurry than images infilled with a non-adversarial generator
trained to minimize MSEs[201] cf. figure 2. Alternatively, computationally
inexpensive loss functions designed by humans, such as structural similarity
index measures[1153] (SSIMs) and Sobel losses[231], can improve generated
output realism. However, it follows from the universal approximator
theorems[37, 38, 39, 40, 41, 42, 43, 44, 45] that training with ANN
discriminators can often yield more realistic outputs.

There are many popular GAN loss functions and regularization mechanisms[1154,
1155, 1156, 1157, 1158]. Traditionally, GANs were trained to minimize
logarithmic discriminator, $D$, and generator, $G$, losses[1159],

$\displaystyle L_{D}$ $\displaystyle=-\log
D(\textbf{x})-\log(1-D(G(\textbf{z})))\,,$ (49) $\displaystyle L_{G}$
$\displaystyle=\log(1-D(G(\textbf{z})))\,,$ (50)

where z are generator inputs, $G(\textbf{z})$ are generated outputs, and x are
example outputs. Discriminators predict labels, $D(\textbf{x})$ and
$D(G(\textbf{z}))$, where target labels are 0 and 1 for generated and real
examples, respectively. Limitedly, logarithmic losses are numerically unstable
for $D(\textbf{x})\rightarrow 0$ or $D(G(\textbf{z}))\rightarrow 1$, as the
denominator, $f(x)$, in $\partial_{x}\log f(x)=\partial_{x}f(x)/f(x)$
vanishes. In addition, discriminators must be limited to $D(\textbf{x})>0$ and
$D(G(\textbf{z}))<1$, so that logarithms are not complex. To avoid these
issues, we recommend training discriminators with squared difference
losses[1160, 1161],

$\displaystyle L_{D}$
$\displaystyle=(D(\textbf{x})-1)^{2}+D(G(\textbf{z}))^{2}\,,$ (51)
$\displaystyle L_{G}$ $\displaystyle=(D(G(\textbf{z}))-1)^{2}\,.$ (52)

However, there are a variety of other alternatives to logarithmic loss
functions that are also effective[1154, 1155].

A variety of methods have been developed to improve GAN training[1162, 995].
The most common issues are catastrophic forgetting[1151] of previous learning,
and mode collapse[1163] where generators only output examples for a subset of
a target domain. Mode collapse often follows discriminators becoming Lipschitz
discontinuous. Wasserstein GANs[1164] avoid mode collapse by clipping
trainable variables, albeit often at the cost of 5-10 discriminator training
iterations per generator training iteration. Alternatively, Lipschitz
continuity can be imposed by adding a gradient penalty[1165] to GAN losses,
such as differences of L2 norms of discriminator gradients from unity,

$\displaystyle\tilde{x}$ $\displaystyle=G(\textbf{z})\,,$ (53)
$\displaystyle\hat{\textbf{x}}$
$\displaystyle=\epsilon\textbf{x}+(1-\epsilon)\tilde{\textbf{x}}\,,$ (54)
$\displaystyle L_{D}$
$\displaystyle=D(\tilde{\textbf{x}})-D(\textbf{x})+\lambda(||\partial_{\hat{\textbf{x}}}D(\hat{\textbf{x}})||_{2}-1)^{2}\,,$
(55) $\displaystyle L_{G}$ $\displaystyle=-D(G(\textbf{z}))\,,$ (56)

where $\epsilon\in[0,1]$ is a uniform random variate, $\lambda$ weights the
gradient penalty, and $\tilde{\textbf{x}}$ is an attempt to generate $x$.
However, using a gradient penalty introduces additional gradient
backpropagation that increases discriminator training time. There are also a
variety of computationally inexpensive tricks that can improve training, such
as adding noise to labels[995, 1075, 1166] or balancing discriminator and
generator learning rates[349]. These tricks can help to avoid discontinuities
in discriminator output distributions that can lead to mode collapse; however,
we observe that these tricks do not reliably stabilize GAN training.

Instead, we observe that spectral normalization[1017] reliably stabilizes GAN
discriminator training in our electron microscopy research[201, 349, 202].
Spectral normalization controls Lipschitz constants of discriminators by
fixing the spectral norms of their weights, as introduced in section 4.2.
Advantages of spectral normalization include implementations based on the
power iteration method[1023, 1024] being computationally inexpensive, not
adding a regularizing loss function that could detrimentally compete[1167,
1168] with discrimination losses, and being effective with one discriminator
training iterations per generator training iteration[1017, 1169]. Spectral
normalization is popular in GANs for high-resolution image synthesis, where it
is also applied in generators to stabilize training[1170].

There are a variety of GAN architectures[1171]. For high-resolution image
synthesis, computation can be decreased by training multiple discriminators to
examine image patches at different scales[1172, 201]. For domain translation
characterized by textural differences, a cyclic GAN[1004, 1173] consisting of
two GANs can map from one domain to the other and vice versa. Alternatively,
two GANs can share intermediate layers to translate inputs via a shared
embedding domain[1174]. Cyclic GANs can also be combined with a siamese
network[279, 280, 281] for domain translation beyond textural
differences[1175]. Finally, discriminators can introduce auxiliary losses to
train DNNs to generalize to examples from unseen domains[1176, 1177, 1178].

Figure 13:  Architectures of recurrent neural networks with a) long short-term
memory (LSTM) cells, and b) gated recurrent units (GRUs).

###  5.3 Recurrent Neural Network

Recurrent neural networks[531, 532, 533, 534, 535, 536] reuse an ANN cell to
process each step of a sequence. Most RNNs learn to model long-term
dependencies by gradient backpropagation through time[1179] (BPTT). The
ability of RNNs to utilize past experiences enables them to model partially
observed and variable length Markov decision processes[1180, 1181] (MDPs).
Applications of RNNs include directing vision[1144, 1145], image
captioning[1182, 1183], language translation[1184], medicine[77], natural
language processing[1185, 1186], playing computer games[24], text
classification[1055], and traffic forecasting[1187]. Many RNNs are combined
with CNNs to embed visual media[1145] or words[1188, 1189], or to process RNN
outputs[1190, 1191]. RNNs can also be combined with MLPs[1144], or text
embeddings[1192] such as BERT[1193, 1192], continuous bag-of-words[1194, 1195,
1196] (CBOW), doc2vec[1197, 1198], GloVe[1199], and word2vec[1200, 1194].

The most popular RNNs consist of long short-term memory[1201, 1202, 1203,
1204] (LSTM) cells or gated recurrent units[1202, 1205, 1206, 1207] (GRUs).
LSTMs and GRUs are popular as they solve the vanishing gradient problem[1208,
1209, 537] and have consistently high performance[1210, 1211, 1212, 1213,
1214, 1215]. Their architectures are shown in figure 13. At step $t$, an LSTM
outputs a hidden state, $h_{t}$, and cell state, $C_{t}$, given by

$\displaystyle\textbf{f}_{t}$
$\displaystyle=\sigma(\textbf{w}_{f}\cdot[\textbf{h}_{t-1},\textbf{x}_{t}]+\textbf{b}_{f})\,,$
(57) $\displaystyle\textbf{i}_{t}$
$\displaystyle=\sigma(\textbf{w}_{i}\cdot[\textbf{h}_{t-1},\textbf{x}_{t}]+\textbf{b}_{i})\,,$
(58) $\displaystyle\tilde{\textbf{c}}_{t}$
$\displaystyle=\tanh(\textbf{w}_{C}\cdot[\textbf{h}_{t-1},\textbf{x}_{t}]+\textbf{b}_{C})\,,$
(59) $\displaystyle\textbf{C}_{t}$
$\displaystyle=\textbf{f}_{t}\textbf{C}_{t-1}+\textbf{i}_{t}\tilde{\textbf{C}_{t}}\,,$
(60) $\displaystyle\textbf{o}_{t}$
$\displaystyle=\sigma(\textbf{w}_{o}\cdot[\textbf{h}_{t-1},\textbf{x}_{t}]+\textbf{b}_{o})\,,$
(61) $\displaystyle\textbf{h}_{t}$
$\displaystyle=\textbf{o}_{t}\tanh(\textbf{C}_{t})\,,$ (62)

where $\textbf{C}_{t-1}$ is the previous cell state, $\textbf{h}_{t-1}$ is the
previous hidden state, $\textbf{x}_{t}$ is the step input, and $\sigma$ is a
logistic sigmoid function of equation 10a, $[\textbf{x},\textbf{y}]$ is the
concatenation of x and y channels, and $(\textbf{w}_{f},\textbf{b}_{f})$,
$(\textbf{w}_{i},\textbf{b}_{i})$, $(\textbf{w}_{C},\textbf{b}_{C})$ and
$(\textbf{w}_{o},\textbf{b}_{o})$ are pairs of weights and biases. A GRU
performs fewer computations than an LSTM and does not have separate cell and
hidden states,

$\displaystyle\textbf{z}_{t}$
$\displaystyle=\sigma(\textbf{w}_{z}\cdot[\textbf{h}_{t-1},\textbf{x}_{t}]+\textbf{b}_{z})\,,$
(63) $\displaystyle\textbf{r}_{t}$
$\displaystyle=\sigma(\textbf{w}_{r}\cdot[\textbf{h}_{t-1},\textbf{x}_{t}]+\textbf{b}_{r})\,,$
(64) $\displaystyle\tilde{\textbf{h}}_{t}$
$\displaystyle=\tanh(\textbf{w}_{h}\cdot[\textbf{r}_{t}\textbf{h}_{t-1},\textbf{x}_{t}]+\textbf{b}_{h})\,,$
(65) $\displaystyle\textbf{h}_{t}$
$\displaystyle=(1-\textbf{z}_{t})\textbf{h}_{t-1}+\textbf{z}_{t}\tilde{\textbf{h}}_{t}\,,$
(66)

where $(\textbf{w}_{z},\textbf{b}_{z})$, $(\textbf{w}_{r},\textbf{b}_{r})$,
and $(\textbf{w}_{h},\textbf{b}_{h})$ are pairs of weights and biases. Minimal
gated units (MGUs) can further reduce computation[1216]. A large-scale
analysis of RNN architectures for language translation found that LSTMs
consistently outperform GRUs[1210]. GRUs struggle with simple languages that
are learnable by LSTMs as the combined hidden and cell states of GRUs make it
more difficult for GRUs to perform unbounded counting[1214]. However, further
investigations found that GRUs can outperform LSTMs on tasks other than
language translation[1211], and that GRUs can outperform LSTMs on some
datasets[1217, 1212, 1213]. Overall, LSTM performance is usually comparable to
that of GRUs.

There are a variety of alternatives to LSTM and GRUs. Examples include
continuous time RNNs[1218, 1219, 1220, 1221, 1222] (CTRNNs), Elman[1223] and
Jordan[1224] networks, independently RNNs[1225] (IndRNNs), Hopfield
networks[1226], recurrent MLPs[1227] (RMLPs). However, none of the variants
offer consistent performance benefits over LSTMs for general sequence
modelling. Similarly, augmenting LSTMs with additional connections, such as
peepholes[1139, 1140] and projection layers[1228], does not consistently
improve performance. For electron microscopy, we recommend defaulting to LSTMs
as we observe that their performance is more consistently high than
performance of other RNNs. However, LSTM and GRU performance is often
comparable, so GRUs are also a good choice to reduce computation.

There are a variety of architectures based on RNNs. Popular examples include
deep RNNs[1229] that stack RNN cells to increase representational ability,
bidirectional RNNs[1230, 1231, 1232, 1233] that process sequences both
forwards and in reverse to improve input utilization, and using separate
encoder and decoder subnetworks[1205, 1234] to embed inputs and generate
outputs. Hierarchical RNNs[1235, 1236, 1237, 1238, 1239] are more complex
models that stack RNNs to efficiently exploit hierarchical sequence
information, and include multiple timescale RNNs[1240, 1241] (MTRNNs) that
operate at multiple sequence length scales. Finally, RNNs can be augmented
with additional functionality to enable new capabilities. For example,
attention[1242, 1243, 1244, 1182] mechanisms can enable more efficient input
utilization. Further, creating a neural Turing machine (NTM) by augmenting a
RNN with dynamic external memory[1245, 1246] can make it easier for an agent
to solve dynamic graphs.

Figure 14:  Architectures of autoencoders where an encoder maps an input to a
latent space and a decoder learns to reconstruct the input from the latent
space. a) An autoencoder encodes an input in a deterministic latent space,
whereas a b) traditional variational autoencoder encodes an input as means,
$\mu$, and standard deviations, $\sigma$, of Gaussian multivariates,
$\mu+\sigma\cdot\epsilon$, where $\epsilon$ is a standard normal multivariate.

###  5.4 Autoencoders

Autoencoders[1247, 1248, 1249] (AEs) learn to efficiently encode inputs, I,
without supervision. An AE consists of a encoder, $E$, and decoder, $D$, as
shown in figure 14a. Most encoders and decoders are jointly trained[1250] to
restore inputs from encodings, $E(\textbf{I})$, to minimize a MSE loss,

$L_{\text{AE}}=\text{MSE}(D(E(\textbf{I})),\textbf{I})\,,$ (67)

by gradient descent. In practice, DNN encoders and decoders yield better
compression[1248] than linear techniques, such as principal component
analysis[1251] (PCA), or shallow ANNs. Indeed, deep AEs can outperform JPEG
image compression[1252]. Denoising autoencoders[1253, 1254, 1255, 1256, 1257]
(DAEs) are a popular AE variant that can learn to remove artefacts by
artificially corrupting inputs inside encoders. Alternatively, contractive
autoencoders[1258, 1259] (CAEs) can decrease sensitivity to input values by
adding a loss to minimize gradients w.r.t. inputs. Most DNNs that improve
electron micrograph signal-to-noise are DAEs.

In general, semantics of AE outputs are pathological functions of encodings.
To generate outputs with well-behaved semantics, traditional VAEs[969, 1260,
1261] learn to encode means, $\boldsymbol{\mu}$, and standard deviations,
$\boldsymbol{\sigma}$, of Gaussian multivariates. Meanwhile, decoders learn to
reconstruct inputs from sampled multivariates,
$\boldsymbol{\mu}+\boldsymbol{\sigma}\cdot\boldsymbol{\epsilon}$, where
$\boldsymbol{\epsilon}$ is a standard normal multivariate. Traditional VAE
architecture is shown in figure 14b. Usually, VAE encodings are regularized by
adding Kullback-Leibler (KL) divergence of encodings from standard
multinormals to an AE loss function,

$L_{\text{VAE}}=\text{MSE}(D(\boldsymbol{\mu}+\boldsymbol{\sigma}\cdot\boldsymbol{\epsilon}),\textbf{I})+\frac{\lambda_{\text{KL}}}{2Bu}\sum\limits_{i=1}^{B}\sum\limits_{j=1}^{u}\mu_{ij}^{2}+\sigma_{ij}^{2}-\log(\sigma_{ij}^{2})-1\,,$
(68)

where $\lambda_{\text{KL}}$ weights the contribution of the KL divergence loss
for a batch size of $B$, and a latent space with $u$ degrees of freedom.
However, variants of Gaussian regularization can improve clustering[231], and
sparse autoencoders[1262, 1263, 1264, 1265] (SAEs) that regularize encoding
sparsity can encode more meaningful features. To generate realistic outputs, a
VAE can be combined with a GAN to create a VAE-GAN[1266, 1267, 1268]. Adding a
loss to minimize differences between gradients of generated and target outputs
is computationally inexpensive alternative that can generate realistic outputs
for some applications[231].

A popular application of VAEs is data clustering. For example, VAEs can encode
hash tables[1269, 1270, 1271, 1272, 1273] for search engines, and we use VAEs
as the basis of our electron micrograph search engines[231]. Encoding clusters
visualized by tSNE can be labelled to classify data[231], and encoding
deviations from clusters can be used for anomaly detection[1274, 1275, 1276,
1277, 1278]. In addition, learning encodings with well-behaved semantics
enables encodings to be used for semantic manipulation[1279, 1278]. Finally,
VAEs can be used as generative models to create synthetic populations[1280,
1281], develop new chemicals[1282, 1283, 1284, 1285], and synthesize
underrepresented data to reduce imbalanced learning[1286].

##  6 Optimization

Training, testing, deployment and maintenance of machine learning systems is
often time-consuming and expensive[1287, 1288, 1289, 1290]. The first step is
usually preparing training data and setting up data pipelines for ANN training
and evaluation. Typically, ANN parameters are randomly initialized for
optimization by gradient descent, possibly as part of an automatic machine
learning algorithm. Reinforcement learning is a special optimization case
where the loss is a discounted future reward. During training, ANN components
are often regularized to stabilize training, accelerate convergence, or
improve performance. Finally, trained models can be streamlined for efficient
deployment. This section introduces each step. We find that electron
microscopists can be apprehensive about robustness and interpretability of
ANNs, so we also provide subsections on model evaluation and interpretation.

Figure 15:  Gradient descent. a) Arrows depict steps across one dimension of a
loss landscape as a model is optimized by gradient descent. In this example,
the optimizer traverses a small local minimum; however, it then gets trapped
in a larger sub-optimal local minimum, rather than reaching the global
minimum. b) Experimental DNN loss surface for two random directions in
parameter space showing many local minima[1122]. The image in part b) is
reproduced with permission under an MIT license[1291].  Algorithm 1
Optimization by gradient descent.

Initialize a model, $f(\textbf{x})$, with trainable parameters,
$\boldsymbol{\theta}_{1}$.

for training step $t=1,T$ do

Forwards propagate a randomly sampled batch of inputs, x, through the model to
compute outputs, $\textbf{y}=f(\textbf{x})$.

Compute loss, $L_{t}$, for outputs.

Use the differentiation chain rule[1292] to backpropagate gradients of the
loss to trainable parameters, $\boldsymbol{\theta}_{t-1}$.

Apply an optimizer to the gradients to update $\boldsymbol{\theta}_{t-1}$ to
$\boldsymbol{\theta}_{t}$.

end for

Vanilla SGD[1293, 1294] $[\eta]$

$\displaystyle\theta_{t}=\theta_{t-1}-\eta\partial_{\theta}L_{t}$ (69)

Momentum[1295] $[\eta,\gamma]$

$\displaystyle v_{t}$ $\displaystyle=\gamma
v_{t-1}+\eta\partial_{\theta}L_{t}$ (70) $\displaystyle\theta_{t}$
$\displaystyle=\theta_{t-1}-v_{t}$ (71)

Nesterov momentum[1296, 1297, 1298] $[\eta,\gamma]$

$\displaystyle\phi$ $\displaystyle=\theta_{t-1}+\eta\gamma v_{t-1}$ (72)
$\displaystyle v_{t}$ $\displaystyle=\gamma v_{t-1}+\partial_{\theta}L_{t}$
(73) $\displaystyle\theta_{t}$ $\displaystyle=\phi-\eta v_{t}(1+\gamma)$ (74)

Quasi-hyperbolic momentum[1299] $[\eta,\beta,\nu]$

$\displaystyle g_{t}$ $\displaystyle=\beta
g_{t-1}+(1-\beta)\partial_{\theta}L_{t}$ (75) $\displaystyle\theta_{t}$
$\displaystyle=\theta_{t-1}-\eta(vg_{t}+(1-v)\partial_{\theta}L_{t})$ (76)

AggMo[1300] $[\eta,\beta^{(1)},...,\beta^{(K)}]$

$\displaystyle v_{t}^{(i)}$
$\displaystyle=\beta^{(i)}v_{t-1}^{(i)}-(\partial_{\theta}L_{t})$ (77)
$\displaystyle\theta_{t}$
$\displaystyle=\theta_{t-1}+\frac{\eta}{K}\sum\limits_{i=1}^{K}v_{t}^{(i)}$
(78)

RMSProp[1301] $[\eta,\beta,\epsilon]$

$\displaystyle v_{t}$ $\displaystyle=\beta
v_{t-1}+(1-\beta)(\partial_{\theta}L_{t})^{2}$ (79) $\displaystyle\theta_{t}$
$\displaystyle=\theta_{t-1}-\frac{\eta}{(v_{t}+\epsilon)^{1/2}}\partial_{\theta}L_{t}$
(80)

ADAM[1302] $[\eta,\beta_{1},\beta_{2},\epsilon]$

$\displaystyle m_{t}$
$\displaystyle=\beta_{1}m_{t-1}+(1-\beta_{1})\partial_{\theta}L_{t}$ (81)
$\displaystyle v_{t}$
$\displaystyle=\beta_{2}v_{t-1}+(1-\beta_{2})(\partial_{\theta}L_{t})^{2}$
(82) $\displaystyle\hat{m}_{t}$ $\displaystyle=\frac{m_{t}}{1-\beta_{1}^{t}}$
(83) $\displaystyle\hat{v}_{t}$ $\displaystyle=\frac{v_{t}}{1-\beta_{2}^{t}}$
(84) $\displaystyle\theta_{t}$
$\displaystyle=\theta_{t-1}-\frac{\eta}{\hat{v}_{t}^{1/2}+\epsilon}\hat{m}_{t}$
(85)

AdaMax[1302] $[\eta,\beta_{1},\beta_{2}]$

$\displaystyle m_{t}$
$\displaystyle=\beta_{1}m_{t-1}+(1-\beta_{1})\partial_{\theta}L_{t}$ (86)
$\displaystyle u_{t}$
$\displaystyle=\max(\beta_{2}u_{t-1},|\partial_{\theta}L_{t}|)$ (87)
$\displaystyle\hat{m}_{t}$ $\displaystyle=\frac{m_{t}}{1-\beta_{1}^{t}}$ (88)
$\displaystyle\theta_{t}$
$\displaystyle=\theta_{t-1}-\frac{\eta}{u_{t}}\hat{m}_{t}$ (89)

Algorithms 1:  Update rules of various gradient descent optimizers for a
trainable parameter, $\theta_{t}$, at iteration $t$, gradients of losses
w.r.t. the parameter, $\partial_{\theta}L_{t}$, and learning rate, $\eta$.
Hyperparameters are listed in square brackets.

###  6.1 Gradient Descent

Most ANNs are iteratively trained by gradient descent[1303, 1304, 1305, 465,
1306, 1307], as described by algorithm 1 and shown in figure 15. To minimize
computation, results at intermediate stages of forward propagation, where
inputs are mapped to outputs, are often stored in memory. Storing the forwards
pass in memory enables backpropagation memoization by sequentially computing
gradients w.r.t. trainable parameters. To reduce memory costs for large ANNs,
a subset of intermediate forwards pass results can be saved as starting points
to recompute other stages during backpropagation[1308, 1309]. Alternatively,
forward pass computations can be split across multiple devices[1310].
Optimization by gradient descent plausibly models learning in some biological
systems[1311]. However, gradient descent is not generally an accurate model of
biological learning[1312, 1313, 1314].

There are many popular gradient descent optimizers for deep learning[1303,
1304, 1305]. Update rules for eight popular optimizers are summarized in
figure 1. Other optimizers include AdaBound[1315], AMSBound[1315],
AMSGrad[1316], Lookahead[1317], NADAM[1318], Nostalgic Adam[1319], Power
Gradient Descent[1320], Rectified ADAM[1321] (RADAM), and trainable
optimizers[1322, 1323, 1324, 1325, 1326]. Gradient descent is effective in the
high-dimensional optimization spaces of overparameterized ANNs[1327] as the
probability of getting trapped in a sub-optimal local minima decreases as the
number of dimensions increases. The simplest optimizer is “vanilla” stochastic
gradient descent (SGD), where a trainable parameter perturbation,
$\Delta\theta_{t}=\theta_{t}-\theta_{t-1}$, is the product of a learning rate,
$\eta$, and derivative of a loss, $L_{t}$, w.r.t. the trainable parameter,
$\partial_{\theta}L_{t}$. However, vanilla SGD convergence is often limited by
unstable parameter oscillations as it a low-order local optimization
method[1328]. Further, vanilla SGD has no mechanism to adapt to varying
gradient sizes, which vary effective learning rates as
$\Delta\theta\propto\partial_{\theta}L_{t}$.

To accelerate convergence, many optimizers introduce a momentum term that
weights an average of gradients with past gradients[1329, 1296, 1330].
Momentum-based optimizers in figure 1 are momentum, Nesterov momentum[1296,
1297], quasi-hyperbolic momentum[1299], AggMo[1300], ADAM[1302], and
AdaMax[1302]. To standardize effective learning rates for every layer,
adaptive optimizers normalize updates based on an average of past gradient
sizes. Adaptive optimizers in figure 1 are RMSProp[1301], ADAM[1302], and
AdaMax[1302], which usually result in faster convergence and higher accuracy
than other optimizers[1331, 1332]. However, adaptive optimizers can be
outperformed by vanilla SGD due to overfitting[1333], so some researchers
adapt adaptive learning rates to their variance[1321] or transition from
adaptive optimization to vanilla SGD as training progresses[1315]. For
electron microscopy we recommend adaptive optimization with Nadam[1318], which
combines ADAM with Nesterov momentum, as it is well-established and a
comparative analysis of select gradient descent optimizers found that it often
achieves higher performance than other popular optimizers[1334]. Limitingly,
most adaptive optimizers slowly adapt to changing gradient sizes e.g. a
default value for ADAM $\beta_{2}$ is 0.999[1302]. To prevent learning being
destabilized by spikes in gradient sizes, adaptive optimizers can be combined
with adaptive learning rate[261, 1315] or gradient[1335, 1208, 1336] clipping.

For non-adaptive optimizers, effective learning rates are likely to vary due
to varying magnitudes of gradients w.r.t. trainable parameters. Similarly,
learning by biological neurons varies as stimuli usually activate a subset of
neurons[1337]. However, all neuron outputs are usually computed for ANNs.
Thus, not effectively using all weights to inform decisions is computational
inefficient. Further, inefficient weight updates can limit representation
capacity, slow convergence, and decrease training stability. A typical example
is effective learning rates varying between layers. Following the chain rule,
gradients backpropagated to the $i$th layer of a DNN from its start are

$\frac{\partial
L_{t}}{\partial\textbf{x}_{i}}=\left(\prod\limits_{l=i}^{L-1}\frac{\partial\textbf{x}_{l+1}}{\partial\textbf{x}_{l}}\right)\frac{\partial
L_{t}}{\partial\textbf{x}_{L}}\,,$ (90)

for a DNN with $L$ layers. Vanishing gradients[1209, 537, 1208] occur when
many layers have $\partial x_{l+1}/\partial x_{l}\ll 1$. For example, DNNs
with logistic sigmoid activations often exhibit vanishing gradients as their
maximum gradient is $1/4$ cf. equation 10b. Similarly, exploding
gradients[1209, 537, 1208] occur when many layers have $\partial
x_{l+1}/\partial x_{l}\gg 1$. Adaptive optimizers alleviate vanishing and
exploding gradients by dividing gradients by their expected sizes.
Nevertheless, it is essential to combine adaptive optimizers with appropriate
initialization and architecture to avoid numerical instability.

Optimizers have a myriad of hyperparameters to be initialized and varied
throughout training to optimize performance[1338] cf. figure 1. For example,
stepwise exponentially decayed learning rates are often theoretically
optimal[1339]. There are also various heuristics that are often effective,
such as using a DEMON decay schedule for an ADAM first moment of the momentum
decay rate[1340],

$\beta_{1}=\frac{1-t/T}{(1-\beta_{\text{init}})+\beta_{\text{init}}(1-t/T)}\beta_{\text{init}}\,,$
(91)

where $\beta_{\text{init}}$ is the initial value of $\beta_{1}$, $t$ is the
iteration number, and $T$ is the final iteration number. Developers often
optimize ANN hyperparameters by experimenting with a range of heuristic
values. Hyperparameter optimization algorithms[1341, 1342, 1343, 1344, 1345,
1346] can automate optimizer hyperparameter selection. However, automatic
hyperparameter optimizers may not yield sufficient performance improvements
relative to well-established heuristics to justify their use, especially in
initial stages of development.

Alternatives to gradient descent[1347] are rarely used for parameter
optimization as they are not known to consistently improve upon gradient
descent. For example, simulated annealing[1348, 1349] has been applied to CNN
training[1350, 1351], and can be augmented with momentum to accelerate
convergence in deep learning[1352]. Simulated annealing can also augment
gradient descent to improve performance[1353]. Other approaches include
evolutionary[1354, 1355] and genetic[1356, 1357] algorithms, which can be a
competitive alternative to deep reinforcement learning where convergence is
slow[1358]. Indeed, recent genetic algorithms have outperformed a popular deep
reinforcement learning algorithm[1359]. Another direction is to augment
genetic algorithms with ANNs to accelerate convergence[1360, 1361, 1362,
1363]. Other alternatives to backpropagation include direct search[1364], the
Moore-Penrose Pseudo Inverse[1365]; particle swarm optimization[1366, 1367,
1368, 1369] (PSO); and echo-state networks[1370, 1371, 1372] (ESNs) and
extreme learning machines[1373, 1374, 1375, 1376, 1377, 1378, 1379] (ELMs),
where some randomly initialized weights are never updated.

###  6.2 Reinforcement Learning

Reinforcement learning[1380, 1381, 1382, 1383, 1384, 1385, 1386] (RL) is where
a machine learning system, or “actor”, is trained to perform a sequence of
actions. Applications include autonomous driving[1387, 1388, 1389],
communications network control[1390, 1391], energy and environmental
management[1392, 1393], playing games[24, 25, 26, 27, 1394, 1146, 28, 29], and
robotic manipulation[1395, 1396]. To optimize a MDP[1180, 1181], a discounted
future reward, $Q_{t}$, at step $t$ in a MDP with $T$ steps is usually
calculated from step rewards, $r_{t}$, with Bellman’s equation,

$Q_{t}=\sum\limits_{t^{\prime}=t}^{T}\gamma^{t^{\prime}-t}r_{t^{\prime}},$
(92)

where $\gamma\in[0,1)$ discounts future step rewards. To be clear, multiplying
$Q_{t}$ by $-1$ yields a loss that can be minimized using the methods in
section 6.1.

In practice, many MDPs are partially observed or have non-differentiable
losses that may make it difficult to learn a good policy from individual
observations. However, RNNs can often learn a model of their environments from
sequences of observations[1147]. Alternatively, FNNs can be trained with
groups of observations that contain more information than individual
observations[1394, 1146]. If losses are not differentiable, a critic can learn
to predict differentiable losses for actor training cf. section 5.1.
Alternatively, actions can be sampled from a differentiable probability
distribution[1397, 1144] as training losses given by products of losses and
sampling probabilities are differentiable. There are also a variety of
alternatives to gradient descent introduced at the end of section 6.1 that do
not require differentiable loss functions.

There are a variety of exploration strategies for RL[1398, 1399]. Adding
Ornstein-Uhlenbeck[1400] (OU) noise to actions is effective for continuous
control tasks optimized by deep deterministic policy gradients[1146] (DDPG) or
recurrent deterministic policy gradients[1147] (RDPG) RL algorithms. Adding
Gaussian noise achieves similar performance for optimization by TD3[1401] or
D4PG[1402] RL algorithms. However, a comparison of OU and Gaussian noise
across a variety of tasks[1403] found that OU noise usually achieves similar
performance to or outperforms Gaussian noise. Similarly, exploration can be
induced by adding noise to ANN parameters[1404, 1405]. Other approaches to
exploration include rewarding actors for increasing action entropy[1406, 1407,
1405] and intrinsic motivation[1408, 1409, 1410], where ANNs are incentified
to explore actions that they are unsure about.

RL algorithms are often partitioned into online learning[1411, 1412], where
training data is used as it is acquired; and offline learning[1413, 1414],
where a static training dataset has already been acquired. However, many
algorithms operate in an intermediate regime, where data collected with an
online policy is stored in an experience replay[1415, 1416, 1417] buffer for
offline learning. Training data is often sampled at random from a replay.
However, prioritizing the replay of data with high losses[993] or data that
results in high policy improvements[992] often improves actor performance. A
default replay buffer size of around $10^{6}$ examples is often used; however,
training is sensitive to replay buffer size[1418]. If the replay is too small,
changes in actor policy may destabilize training; whereas if the replay is too
large, convergence may be slowed by delays before the actor learns from policy
changes.

###  6.3 Automatic Machine Learning

There are a variety of automatic machine learning[1419, 1420, 1421, 1422,
1423] (AutoML) algorithms that can create and optimize ANN architectures and
learning policies for a dataset of input and target output pairs. Most AutoML
algorithms are based on RL or evolutionary algorithms. Examples of AutoML
algorithms include AdaNet[1424, 1425], Auto-DeepLab[1426], AutoGAN[1427],
Auto-Keras[1428], auto-sklearn[1429], DARTS+[1430], EvoCNN[271], H2O[1431],
Ludwig[1432], MENNDL[1433, 1434], NASBOT[1435], XNAS[1436], and others[1437,
1438, 1439, 1440, 1441]. AutoML is becoming increasingly popular as it can
achieve higher performance than human developers[1442, 1077] and enables human
developer time to be traded for potentially cheaper computer time.
Nevertheless, AutoML is currently limited to established ANN architectures and
learning policies. Consequently, we recommend that researchers either focus on
novel ANN architectures and learning policies or developing ANNs for novel
applications.

###  6.4 Initialization

How ANN trainable parameters are initialized[537, 1443] is related to model
capacity[1444]. Further, initializing parameters with values that are too
small or large can cause slow learning or divergence[537]. Careful
initialization can also prevent training by gradient descent being
destabilized by vanishing or exploding gradients[1209, 537, 1208], or high
variance of length scales across layers[537]. Finally, careful initialization
can enable momentum to accelerate convergence and improve performance[1296].
Most trainable parameters are multiplicative weights or additive biases.
Initializing parameters with constant values can result in every parameter in
a layer receiving the same updates by gradient descent, reducing model
capacity. Thus, weights are often randomly initialized. Followingly, biases
are often initialized with constant values due to symmetry breaking by the
weights.

Consider the projection of $n_{\text{in}}$ inputs,
$\textbf{x}^{\text{input}}=\\{x_{1}^{\text{input}},...,x_{n_{\text{in}}}^{\text{input}}\\}$,
to $n_{\text{out}}$ outputs,
$\textbf{x}^{\text{output}}=\\{x_{1}^{\text{output}},...,x_{n_{\text{out}}}^{\text{output}}\\}$,
by an $n_{\text{in}}\times n_{\text{out}}$ weight matrix, w. The expected
variance of an output element is[1443]

$\displaystyle\text{Var}(\textbf{x}^{\text{output}})=n_{\text{in}}\text{E}(\textbf{x}^{\text{input}})^{2}\text{Var}(\textbf{w})+n_{\text{in}}\text{E}(\textbf{w})^{2}\text{Var}(\textbf{x}^{\text{input}})+n_{\text{in}}\text{Var}(\textbf{w})\text{Var}(\textbf{x}^{\text{input}})\,,$
(93)

where $\text{E}(\textbf{x})$ and $\text{Var}(\textbf{x})$ denote the expected
mean and variance of elements of x, respectively. For similar length scales
across layers, $\text{Var}(\textbf{x}^{\text{output}})$ should be constant.
Initially, similar variances can be achieved by normalizing ANN inputs to have
zero mean, so that $\text{E}(\textbf{x}^{\text{input}})=0$, and initializing
weights so that $\text{E}(\textbf{w})=0$ and
$\text{Var}(\textbf{w})=1/n_{\text{in}}$. However, parameters can shift during
training, destabilizing learning. To compensate for parameter shift, popular
normalization layers like batch normalization often impose
$\text{E}(\textbf{x}^{\text{input}})=0$ and
$\text{Var}(\textbf{x}^{\text{input}})=1$, relaxing need for
$\text{E}(\textbf{x}^{\text{input}})=0$ or $\text{E}(\textbf{w})=0$.
Nevertheless, training will still be sensitive to the length scale of
trainable parameters.

There are a variety of popular weight initializers that adapt weights to ANN
architecture. One of the oldest methods is LeCun initialization[951, 941],
where weights are initialized with variance,

$\text{Var}(\textbf{w})=\frac{1}{n_{\text{in}}}\,,$ (94)

which is argued to produce outputs with similar length scales in the previous
paragraph. However, a similar argument can be made for initializing with
$\text{Var}(\textbf{w})=1/n_{\text{out}}$ to produce similar gradients at each
layer during the backwards pass[1443]. As a compromise, Xavier
initialization[1445] computes an average,

$\text{Var}(\textbf{w})=\frac{2}{n_{\text{in}}+n_{\text{out}}}\,.$ (95)

However, adjusting weights for $n_{\text{out}}$ is not necessary for adaptive
optimizers like ADAM, which divide gradients by their length scales, unless
gradients will vanish or explode. Finally, He initialization[22] doubles the
variance of weights to

$\text{Var}(\textbf{{w}})=\frac{2}{n_{\text{in}}}\,,$ (96)

and is often used in ReLU networks to compensate for activation functions
halving variances of their outputs[22, 1446, 1443]. Most trainable parameters
are initialized from either a zero-centred Gaussian or uniform distribution.
For convenience, the limits of such a uniform distribution are
$\pm(3\text{Var}(\textbf{w}))^{1/2}$. Uniform initialization can outperform
Gaussian initialization in DNNs due to Gaussian outliers harming
learning[1443]. However, issues can be avoided by truncating Gaussian
initialization, often to two standard deviations, and rescaling to its
original variance.

Some initializers are mainly used for RNNs. For example, orthogonal
initialization[1447] often improves RNN training[1448] by reducing
susceptibility to vanishing and exploding gradients. Similarly, identity
initialization[1449, 1450] can help RNNs to learn long-term dependencies. In
most ANNs, biases are initialized with zeros. However, the forget gates of
LSTMs are often initialized with ones to decrease forgetting at the start of
training[1211]. Finally, the start states of most RNNs are initialized with
zeros or other constants. However, random multivariate or trainable variable
start states can improve performance[1451].

There are a variety of alternatives to initialization from random
multivariates. Weight normalized[1014] ANNs are a popular example of data-
dependent initialization, where randomly initialized weight magnitudes and
biases are chosen to counteract variances and means of an initial batch of
data. Similarly, layer-sequential unit-variance (LSUV) initialization[1452]
consists of orthogonal initialization followed by adjusting the magnitudes of
weights to counteract variances of an initial batch of data. Other approaches
standardize the norms of backpropagated gradients. For example, random walk
initialization[1453] (RWI) finds scales for weights to prevent vanishing or
exploding gradients in deep FNNs, albeit with varied success[1452].
Alternatively, MetaInit[1454] scales the magnitudes of randomly initialized
weights to minimize changes in backpropagated gradients per iteration of
gradient descent.

###  6.5 Regularization

There are a variety of regularization mechanisms[1455, 1456, 1457, 1458] that
modify learning algorithms to improve ANN performance. One of the most popular
is L$X$ regularization, which decays weights by adding a loss,

$L_{X}=\lambda_{X}\sum\limits_{i}\frac{|\theta_{i}|^{X}}{X}\,,$ (97)

weighted by $\lambda_{X}$ to each trainable variable, $\theta_{i}$. L2
regularization[1459, 1460, 1461] is preferred[1462] for most DNN optimization
as subtraction of its gradient,
$\partial_{\theta_{i}}L_{2}=\lambda_{2}\theta_{i}$, is equivalent to
computationally-efficient multiplicative weight decay. Nevertheless, L1
regularization is better at inducing model sparsity[1463] than L2
regularization, and L1 regularization achieves higher performance in some