# Correcting for the overabundance of low-mass quiescent galaxies in semi-
analytic models

Jimi E. Harrold1, Omar Almaini1, Frazer R. Pearce1, Robert M. Yates2
1School of Physics & Astronomy, University of Nottingham, Nottingham, NG7 2RD,
UK
2Centre for Astrophysics Research, University of Hertfordshire, Hatfield, AL10
9AB, UK

(Accepted XXX. Received YYY; in original form ZZZ)

###### Abstract

We compare the L-Galaxies semi-analytic model to deep observational data from
the UKIDSS Ultra Deep Survey (UDS) across the redshift range $0.5<z<3$. We
find that the over-abundance of low-mass, passive galaxies at high redshifts
in the model can be attributed solely to the properties of ‘orphan’ galaxies,
i.e. satellite galaxies where the simulation has lost track of the host dark
matter subhalo. We implement a simple model that boosts the star-formation
rates in orphan galaxies by matching them to non-orphaned satellite galaxies
at a similar evolutionary stage. This straightforward change largely addresses
the discrepancy in the low-mass passive fraction across all redshifts. We find
that the orphan problem is somewhat alleviated by higher resolution
simulations, but the preservation of a larger gas reservoir in orphans is
still required to produce a better fit to the observed space density of low-
mass passive galaxies. Our findings are also robust to the precise definition
of the passive galaxy population. In general, considering the vastly different
prescriptions used for orphans in semi-analytic models, we recommend that they
are analysed separately from the resolved satellite galaxy population,
particularly with JWST observations reigniting interest in the low-mass regime
in which they dominate.

###### keywords:

methods: analytical – methods: numerical – galaxies: abundances – galaxies:
evolution.

††pubyear: 2023††pagerange: Correcting for the overabundance of low-mass
quiescent galaxies in semi-analytic models–Correcting for the overabundance of
low-mass quiescent galaxies in semi-analytic models

##  1 Introduction

Our view of galaxy formation has evolved rapidly in recent decades, driven
largely by deep observational surveys, which allow us to study large numbers
of galaxies at different slices through cosmic history. To gain a deeper
understanding of the underlying physics, it is also vital to compare our
models of galaxy formation with these observations. A key test is to reproduce
the galaxy stellar mass function (SMF) and its evolution. When the SMFs
between theory and observations agree over a wide range in redshift and
stellar mass it provides a level of confidence that our understanding is
likely to be broadly correct. In contrast, if we cannot explain the evolution
of the SMF it may indicate a misunderstanding of the underlying physical
processes. Further splitting SMFs by redshift, star formation rate (SFR),
environment, galaxy class, etc., can help to diagnose likely problems or
missing ingredients in our models (Henriques et al., 2015; Croton et al.,
2016; Cattaneo et al., 2017; Cora et al., 2018; Lagos et al., 2023). Large
numbers of galaxies are required in order to directly compare the SMFs between
observations and models accurately over a range of different parameters, and
large volumes of the Universe should be modelled to overcome the influence of
cosmic variance (Somerville et al., 2004). Until recently, the difficulty in
obtaining a statistically significant sample of both simulated and observed
galaxies across a range of environments and redshfits has proven to be
prohibitive, preventing such direct comparisons.

Wide-field deep near-infrared surveys provide a powerful method for probing
high redshift ($z>1$) galaxy populations. Deep near-infrared studies are vital
for probing rest-frame optical light at high redshift, and to ensure that we
are not biased against old or dusty galaxies (Cimatti et al., 2004). In
addition to infrared depth, a wide field is also required to accurately
determine the space density of rare massive galaxies, and to study galaxy
clustering and large-scale structure (e.g. Hartley et al., 2013). In
particular, surveys such as the UKIDSS Ultra Deep Survey (UDS) and UltraVISTA
can probe the SMFs of typical (sub-L∗) galaxy populations to redshifts of
around $z=3$ (e.g., Muzzin et al., 2013; Taylor et al., 2023). Such surveys
are ideal for studying galaxy evolution around "cosmic noon" ($z=1-3$), at the
peak epoch of both star formation and quenching (e.g. Ilbert et al., 2013;
Förster Schreiber & Wuyts, 2020). Comparisons between observations and models
of galaxy evolution at these epochs are therefore ideal for testing our
understanding of galaxy formation and evolution.

Semi-analytical models (SAMs) provide a potent method for comparing galaxy
populations directly to observations (White & Frenk, 1991; Springel et al.,
2001a; Baugh, 2006; Somerville & Davé, 2015). By opting to replace the full
hydrodynamical prescriptions typically used to model a galaxy with simple
analytical counterparts, and exploiting the massive computational speed gains
achieved by doing so, SAMs can simulate substantial volumes and produce large
catalogues of galaxies. These catalogues can be tuned to produce similar
populations of galaxies to those observed in deep wide-field surveys such as
the UDS, and thus can be directly compared against them on observables not
included in the tuning process. Direct tracers of the underlying evolution of
the galaxy population, such as the SMF, can then be used to test our
contemporary understanding of galaxy evolution.

While it has been shown that SAMs reconstruct a range of local galaxy
properties with a reasonable degree of accuracy, historically they have found
it harder to simultaneously tune to observations at higher redshifts (Asquith
et al., 2018; Fontanot et al., 2009; Weinmann et al., 2012; Knebe et al.,
2015). This difficulty is exacerbated as observational surveys probe to lower
masses and even higher redshifts. Increasingly observations point to a so-
called ‘anti-hierarchical’ form of growth, otherwise known as ‘mass assembly
downsizing’, where high mass galaxies formed rapidly in the early Universe,
with low-mass galaxies forming at late times (Cimatti et al., 2006; Lee & Yi,
2013). Many SAMs, such as L-Galaxies have been tuned to reproduce the overall
SMF across the redshift range $0.5<z<3.0$. However, one particular tension
between SAMs and observations still exists: SAMs overproduce the number of
passive low-mass galaxies, with the overproduction becoming worse at higher
redshifts (Asquith et al., 2018; Donnari et al., 2021). Previous works such as
Donnari et al. (2021) have shown that while some differences in the passive
fraction can be attributed to the selection method used, this mainly alters
the fraction at stellar masses above $\mathrm{10^{10.5}~{}M_{\odot}}$.
Overall, it has become clear that SAMs overproduce the space density of low-
mass passive galaxies by a factor of $\sim 3$ at low redshift, rising to up to
over an order of magnitude more than is observed at high redshifts (Asquith et
al., 2018). In this letter we identify a significant subset of galaxies,
orphan galaxies, that are causing this tension.

SAMs are built on top of an underlying N-body simulation within which
collapsed structures are identified using a halo finder and connected from one
output to the next using a tree builder (Springel et al., 2001b; Behroozi et
al., 2013; Srisawat et al., 2013). In isolation the evolution and growth of
dark matter haloes is relatively straightforward to follow. However, when
haloes merge the density field can become difficult to disentangle. In
particular, sub-haloes orbiting within a larger host dark matter halo can
become lost and/or disrupted. Baryonic objects, where gas has cooled,
collapsed and formed stars at the centre of these haloes can survive this
process. Within SAMs, this process produces so-called orphan (type-II)
galaxies (Hopkins et al., 2010). These are satellite galaxies where the
underlying simulation has lost track of the host dark matter halo, either due
its tidal disruption or simply falling below the resolution limit of the
simulation. The treatment of orphan galaxies varies across different SAMs, and
has been a topic of historical disagreement: some SAMs simply remove orphan
galaxies completely, stating this improves the fit to the observational data
used to tune the model (Croton et al., 2016). A more common approach involves
stripping away the gas supply to the galaxy and implementing a merger
timescale between the central galaxy and the orphan (Springel et al., 2001a;
Guo et al., 2011). This concept is physically motivated by the idea that
without a dark matter halo the potential well of the galaxy is too shallow to
retain gas against external forces such as ram-pressure stripping, which act
on a short dynamical timescale (McCarthy et al., 2008). Older models such as
De Lucia & Blaizot (2007) typically stripped all satellite galaxies (both
orphans and resolved satellites) instantly. This idea has been revised in
works such as Guo et al. (2011), where satellites strip slower and
instantaneous stripping only applies to the hot gas that remains once a galaxy
becomes an orphan. As this quenching happens rapidly these galaxies quickly
become identified as quiescent. In the local universe this model appears to
work well down to moderate masses, especially when compounded with the fact
that it is difficult to separate orphans and satellites observationally. As
observations have probed deeper and mass functions become complete to lower
stellar masses the tension between the observed passive galaxy counts and the
models has increased (Asquith et al., 2018; Donnari et al., 2021). Some
attempts to circumnavigate the orphan problem utilise the ‘excellent
convergence’ in numerous metrics between the two millennium simulations shown
in works such as Boylan-Kolchin et al. (2009), employing a hybrid of two
separate dark matter simulations. This solution was first attempted in Guo et
al. (2011), which used Millennium-I for high and intermediate mass galaxies,
combined with a run on Millennium-II for lower-mass galaxies. These results
show improvements to the lower-mass passive fraction at the cost of increased
runtime due to the use of merger trees from two cosmological simulations.
Additionally, such models fail to directly address the underlying problem
highlighted in this letter: the way orphans are currently modelled in SAMs is
imprecise and there is a lack of consistency between the physics applied to
orphans and resolved satellites. SAMs such as GAEA have attempted to address
the low-mass passive population by slowing the quenching in satellites in
works such as Xie et al. (2020), but still fail to rectify the issue (see De
Lucia et al. 2024; figure 11). Luo et al. (2016) implemented a resolution
independent treatment of orphans and satellites in an older version of
L-Galaxies, finding an under-abundance of quenched central galaxies in the
model (in addition to an over-abundance of quenched satellites), indicating
that issues with quenching are not solely of environmental origin. However,
adjustments made by Luo et al. (2016) were performed on the 2013 version of
L-Galaxies, which lacked the significant alterations to gas reincorporation
and AGN feedback that the 2015 model provides, which aimed to address many of
the discrepancies with central galaxies and how they quench (Henriques et al.,
2015).

In this letter, we apply L-Galaxies through a lightcone designed to mimic the
properties of galaxies observed in the UKIDSS Ultra-Deep Survey (UDS), which
is the deepest near-infrared ($K$-band selected) survey to date over such a
large area of sky. By separating galaxies into star-forming and passive
subsets, we find that many of the discrepancies can be attributed to the
treatment of the orphan population in the models. We also show that by
retroactively adjusting the properties of the orphan galaxies to follow a more
gradual quenching prescription, the agreement between the model and survey
dramatically improves, with a 4-fold reduction of the passive SMF $\chi^{2}$
values when compared with no adjustment. We show this result remains true when
increasing the resolution of the underlying dark matter simulation, and when
applying different standard cuts to define the passive fraction, including:
specific SFR, UVJ cuts, and an L-Galaxies specific cut. We argue that while
L-Galaxies is an excellent model for the luminosity evolution of the whole
galaxy population it can be significantly improved for the passive population
if the quenching of these galaxies is somewhat altered. We suggest that such a
model should be implemented self-consistently into L-Galaxies.

Section 2 covers the usage of a catalogue derived from the 11th data release
from the UKIDSS Ultra-Deep Survey, and how the standard output of L-Galaxies
was processed in order to closely match the observational properties of this
survey. Section 3 covers the direct comparison of galaxy SMFs as a result of
this processing, highlighting the initial overabundance of low-mass passives,
followed by implementation and comparison of the toy model. Section 4
discusses the conclusions we derive from applying this toy model, and possible
routes for more permanent alterations to L-Galaxies. This letter assumes a
flat $\Lambda$CDM cosmology of
$\text{h}=0.7,\Omega_{m}=0.3,\Omega_{\Lambda}=0.7$; the standard cosmology of
L-Galaxies ($\text{h}=0.673,\Omega_{m}=0.315,\Omega_{\Lambda}=0.685$) has been
scaled in the figures shown to match this cosmology.

##  2 The Data

###  2.1 Observational Data

For comparison to observations, we use a catalogue derived from the 11th data
release of the UKIDSS Ultra-Deep Survey (UDS; Lawrence et al. 2007; Almaini et
al., in preparation). The UDS covers 0.77 square degrees, reaching near-
infrared depths of $J=25.6$, $H=25.1$ and $K=25.3$. Additional imaging of the
UDS in various wavebands has been performed using several instruments (all
magnitudes given in AB, $5\sigma$ S/N over 2 arcsec diameter): Subaru Suprime-
CAM ($B=27.6$, $V=27.2$, $R=27.0$ and $z^{\prime}=26.0$; Furusawa et al.
2008); CFHT MegaCam ($u^{\prime}=26.75$); VISTA ($Y=24.5$; Jarvis et al.
2013); Spitzer IRAC from SpUDS (PI: Dunlop) at $3.6\mu m$ and $4.5\mu m$,
reaching magnitudes of $24.2$ and $24.0$ respectively, supplemented by deeper
imaging from SEDS (Ashby et al., 2013). The full area covered by all 12 bands
is 0.63 sq degrees, after the masking of artefacts and bright stars.

Photometric redshifts were calculated following the method of Simpson et al.
(2013), using the with EAzY code (Brammer et al., 2008), as described in
Taylor et al. (2023). The approach is based on the default EAzY setup, with 12
Flexible Stellar Population Synthesis (FSPS) SED components (Conroy & Gunn,
2010), augmented by three simple stellar population (SSP) templates. These
additional SSP templates, with ages of 20, 50, and 150 Myr, use a Chabrier IMF
and sub-solar metallicity (0.2 solar) to represent recent bursts of star
formation alongside the continuous star-formation histories in the FSPS
templates. Photometric redshifts were calibrated using approximately 8000
sources with secure spectroscopic redshifts, yielding a mean absolute
dispersion of $\sigma_{NMAD}$ = 0.019 and an outlier fraction of $\sim$ 3$\%$,
where outliers are defined when |$\Delta$z|/(1 + z) > 0.15. Further details of
the photometric redshift determination can be obtained from Taylor et al.
(2023).

Stellar masses for both the UDS and the mock lightcone were calculated using
the Bayesian analysis technique described in Wild et al. (2016), based on a
PCA analysis. A library of around 44000 stochastic burst model SEDs were built
from Bruzual & Charlot (2003) stellar population synthesis models, accounting
for a wide range metallicity, dust content, and star-formation histories. When
fitting to the model SEDs, this principal component analysis finds only three
distinct eigenvectors are needed to reproduce over 99.7% of the observed
variance, with the 3 respective eigenvalues ("supercolours") being known as
SC1, SC2, and SC3. Each of these SCs can be mapped to physical properties of
the observed galaxies: SC1 correlates with the $R-$band weighted mean stellar
age of a galaxy and with dust content; SC2 correlates with the fraction of
stars formed in the last Gyr, and helps to break the degeneracy between dust
and age; SC3 helps to break the degeneracy between the fraction of stellar
mass formed and metallicity seen in SC2. Galaxies are classified using their
position in SC space to separate star-forming, dusty, and passive galaxies, in
addition to rarer classes such as post-starburst galaxies. This technique has
been verified spectroscopically (e.g. see Maltby et al., 2016; Wilkinson et
al., 2021), finding very low rates of contamination of star-forming galaxies
in the passive class. The resulting separation of passive and star-forming
galaxies is also in very good agreement with the UVJ rest-frame technique
(Wuyts et al., 2007), as demonstrated in Wild et al. (2014) and Almaini et al.
(2017). The influence of the precise selection method will be discussed
further in section 4. We estimate stellar mass errors are typically $\pm 0.1$
dex (see Almaini et al., 2017). The same PCA technique is used to classify
galaxies and determine stellar masses for the simulated galaxies within the
lightcone (see below).

Stellar mass completeness limits at a given redshift are calculated using the
method described in Pozzetti et al. (2010) using a 95% completeness limit. We
impose an additional minimum stellar mass limit of
$\mathrm{\log(M_{\ast}/M_{\odot})=9.5}$ to both the UDS and the mock galaxies,
reflecting the minimum mass resolution of L-Galaxies when run on the
Millennium simulation (Henriques et al., 2015).

###  2.2 Simulated Data

The model used to compare to observations was obtained using the 2015 version
of the Munich Galaxy Formation Model known as L-Galaxies, described in
Henriques et al. (2015). The underlying DM simulation used is the Millennium
simulation (Springel et al., 2005), with the cosmology retroactivly re-scaled
to the standard UDS cosmology of
$\text{h}=0.7,\Omega_{m}=0.3,\Omega_{\Lambda}=0.7$. While this re-scaling does
not fully account for adjustments detailed in Angulo & White (2010), we argue
any changes to our conclusions as a result of this will be negligible. We use
standard input parameters described in Henriques et al. (2015) and originally
constrained via a Markov chain Monte Carlo approach. Additional dust
attenuation was included via the prescription described by Charlot & Fall
(2000). On top of the L-Galaxies output we construct a lightcone. This
lightcone is designed to cover a field-of-view of 1 square degree over a
redshift range of $0.5<z<3.0$. This closely matches the field-of-view of the
UDS survey. Galaxies along the lightcone were then selected by sampling their
data from the simulation snapshot that best matched the redshift at which they
are being sampled. To prevent galaxies from being detected multiple times, the
progenitor and descendants of a galaxy are marked in order to ensure they are
not re-used in the construction of the lightcone. A scatter of 0.08(1 + z) was
applied to the stellar mass values before processing, derived from Conroy et
al. (2009), to represent the systematic uncertainty in observational stellar
mass determination. An additional scatter of 0.0189(1 + z) is applied to the
redshift values of the catalogue, reflecting the estimated uncertainty derived
by comparing spectroscopic and photometric redshift values for sources within
the UDS (Wilkinson et al., 2021).

Galaxy SEDs were generated in a similar way to our observational sample, by
matching each galaxy to a library of around 44000 stochastic burst model SEDs
built from Bruzual & Charlot (2003) stellar population synthesis models,
accounting for a wide range of metallicities, dust content, and star-formation
histories. Each galaxy SED is then convolved with the 13 UDS filters at
various redshifts, stepping by $\Delta z=0.1$ through a redshift range of
$0.5<z<3.0$, giving expected flux values for the mock galaxy as if it was
observed at each of the various redshifts. Using the 13 flux values generated
for each mock galaxy, the lightcone is then run though the same principal
component analysis described above.

Once the SC values have been generated, stellar masses and SFRs can be
calculated for the galaxy using the technique outlined in Wild et al. (2016).
All properties then calculated are obtained in precisely the same way as the
real catalogue. We find our constructed lightcone to be in strong agreement
with typical L-Galaxies output and production tests.

Figure 1: The mass functions of star-forming (blue) and passive (red) galaxies
in three redshift ranges. The observational data from the UDS is shown by the
solid dots. Three variants of the L-Galaxies model output are shown:
unadjusted (solid lines), orphans removed (dashed lines), and our toy model
with modified orphans (black, dot-dashed lines). The vertical line denotes the
95% completeness limit, determined using the method of Pozzetti et al. (2010).
The bottom row shows the difference between the models and the UDS mass
functions.

##  3 Comparison of galaxy mass functions

Figure 2: Contours for star formation rate (SFR) for both the star forming
(blue) and passive (red) populations. In all four panels the observational
data is displayed as the background shading. The contours are taken from the
models, respectively L-Galaxies unaltered (first row), and L-Galaxies with the
modified orphans (second row). Similar to the SMFs, the star forming contours
show very little change when adjusting for orphans, however the passive
contours show a significant improvement in fit.

We compare the galaxy SMFs produced by the mock lightcone and the UDS across
the redshift range $0.5<z<3.0$ in Figure 1. Galaxies are split into star
forming and quiescent bins via a supercolour cut defined by Wild et al. (2014)
and further refined in Wilkinson et al. (2021). The SMF for the mock is
generated both with and without orphans included. Excluding the orphans from
the star-forming SMF (blue lines) makes no appreciable difference across all
redshifts, while the quiescent SMF (red lines) loses the large tail of low-
mass galaxies seen in both upper panels. While removing the orphans shows an
improvement in the overall shape of the quiescent SMF, the density of low-mass
passive galaxies drops by as much as an order of magnitude below that seen in
the UDS. This drop is mostly due to the fact that orphans make up at least 35%
of the total quiescent population produced by L-Galaxies. We explore the
effects of various changes in selection method, the DM model used, and the
version of L-Galaxies used in section 4

While removing orphans improves the general shape of the SMF, the loss of
around a third of the quiescent population itself produces problems: the low-
mass passive population is now under- rather than over-represented. While this
underproduction could be addressed by re-calibrating L-Galaxies , we argue
that simply removing orphan galaxies without a physical mechanism is likely to
be problematic.

###  3.1 Feeding orphans

In an attempt to address this point we implement a toy model that aims to
rectify the issues presented above without requiring new modules and re-
calibration of L-Galaxies. Our model aims to substitute orphan galaxies with
higher SFR and higher mass satellite galaxies which have otherwise comparable
properties. This is motivated by the idea that while the growth of the disc
region of a galaxy closely follows the growth of the host halo (and thus is
interrupted when this halo is no longer present in an orphan) the central
regions of the galaxy present a deep enough potential well to retain some of
the gas reservoirs that are currently considered stripped. This allows star
formation to continue for longer as the would-be orphan falls into the central
cluster. This model has some observational motivation, e.g. Cortese et al.
(2021) discuss that stripping is not sufficient to completely remove cold gas
discs in cluster galaxies above masses of $\mathrm{M_{*}>10^{8}M_{\odot}}$,
allowing star formation to continue centrally for several Gyr. Bamford et al.
(2007) found star formation is more central in satellite galaxies compared to
their field counterparts, with Weinmann et al. (2009) finding little evidence
for rapid quenching effecting satellites, preferring a gradual quenching path
created by starvation. Kawinwanichakij et al. (2017) finds a "lack of
significant environmental quenching of low mass galaxies at $z>1$", supporting
a model where gas is depleted through over-consumption and outflows as opposed
to environmental effects, but note that effects such as disk fading are needed
to explain the transition from star forming to quiescent. Woo et al. (2017)
also notes a consistent increase in the surface density of satellites as they
transition through the green valley, suggesting either an enhancement of star
formation in the central regions or quenching effects in the outermost regions
of a galaxy. Allowing the continuation of star formation in the models will
have three key effects on the overall galaxy population: (1) many of the low-
mass orphans will be moved to the star forming region of the SMF, alleviating
the overproduction of passive galaxies while having little impact on the star-
forming SMF (since star-forming galaxies are far more numerous); (2) once the
orphans do quench, they do so at a higher mass, which acts to help build the
’knee’ of both the star forming and quiescent SMFs; (3) low-mass galaxies with
low or decreasing halo mass should retain more star formation and grow past
the typical halo mass to stellar mass relationship, allowing slightly higher-
mass galaxies to exist before stripping and reclassification as orphans takes
place.

In order to provide adjusted properties for these modified orphans we use the
properties of satellite galaxies with equivalent redshifts and supercolour
values. Galaxies were binned using a k-d tree to ensure a minimum population
of equivalent satellite galaxies in each colour bin without sacrificing
binning resolution in high density regions. Using standard 2d-histogram bins
over a range of bin widths produces similar results. Using these bins, the
L-Galaxies output is modified; each orphan has a random satellite substitute
selected from within the same bin. The properties of this satellite are then
used in place of the orphan’s when processing the lightcone. This simplistic
approach allows orphans to assume the properties of galaxies undergoing
similar expected evolutionary pathways. Using different properties to
initially bin the galaxies such as $U-V$ and $V-J$ rest-frame colours, has
little effect on the conclusion. Further robustness tests are presented in
section 4.

The result of feeding the orphans can be seen in the lower panels of Figures 1
& 2. We observe an improvement in the overall fit of the passive fraction
across all redshifts, while having a negligible impact of the star forming
SMFs. SFRs show a similar improvement, matching closely with the SFRs produced
by the UDS observations.

This improvement of the quiescent fraction may complement recent works such as
those of Murphy et al. (2022), who employ gradual tidal stripping of the cold
gas supplies of satellites, including orphans, improving the overall fit to
the observations used. This adjustment differs from our proposal, which
primarily effects the size of available gas reservoirs before tidal stripping
takes place, but similarly promotes a less violent treatment of orphan
galaxies. While no direct comparison of the passive fraction is made in the
paper, it is likely that the low-mass passive fraction will increase under the
Murphy et al. (2022) prescription, due to an earlier onset of cold gas
stripping in satellites, exacerbating the tension. Combining such a model with
our prescription, where orphans have access to a larger quantity of gas to
allow for boosted levels of star formation, may allow the stated improvements
while still reproducing the passive fraction at high redshifts.

##  4 Robustness Tests

###  4.1 Quiescent Selection

In order to ensure that our primary conclusions are a robust feature of the
model and not an artifact of selection methods, we apply several different
quiescent selection techniques to the mock lightcone and the observational
data. We find a similar overproduction due to orphans when selecting via a
standard sSFR (e.g. $\log(\text{sSFR}/\text{yr}^{-1})<-11$), UVJ colour
selection (e.g., as outlined in Wuyts et al. 2007, Muzzin et al. 2013), and
the adjusted UVJ technique outlined in Henriques et al. (2015). We note that
the discrepancies we find largely agree with those found in section 5.1 of
Henriques et al. (2015), which also notes an excess of low-mass passive
galaxies at increasing redshifts despite calibrating to the passive fraction.
This problem may be exacerbated in our results due the increased selection
accuracy of the supercolour technique, combined with the close matching of the
mock data to our observational data set.

We also compared our results with observational data from the
COSMOS/UltraVISTA deep survey (Wilkinson et al., 2021) at $z<1$, finding
consistent results and the same discrepancy in the predicted fraction of low-
mass passive galaxies.

###  4.2 Model Selection

We also obtain similar results if SMFs are generated without using a
lightcone, and are instead produced directly from individual snapshots with
observational limits applied. The 2020 version of L-Galaxies (Henriques et
al., 2020) also shows the same problem and subsequent improvement, which is to
be expected as the underlying treatment of orphans did not change.

###  4.3 Millenium-II

Our findings are also independent of the details of lightcone construction and
the particular version of L-Galaxies used. As orphans are a resolution
dependent phenomenon, it is expected that the effect would be lessened in
higher resolution simulations. We confirmed this by running partial analysis
on halo trees produced by Millennium-II Boylan-Kolchin et al. (2009), which
provides 5 times better spatial resolution, down to a softening length of 1
kpc/h, and 125 times better mass resolution of $\mathrm{6.9\times
10^{6}\,h^{-1}M_{\odot}}$. We find the results produced by Millennium-II show
an improved initial shape of the passive galaxy SMF, but the low-mass tail was
still improved with orphan galaxies removed, suggesting that while orphans are
indeed a resolution dependent feature, their problematic prescriptions
persist. Due to the precise integration of the lightcone with L-Galaxies 2015
and the requirement for large simulation box sizes to accurately capture the
complete cosmology represented in the observations used, it was not possible
to run a direct lightcone comparison to Millennium-II. Instead we apply a
$K$-band magnitude cut at $25.3$, matching the $5\sigma$ $K$-band limit from
data release 11 of the UDS, to act as a basic proxy to the more sophisticated
detection criteria imposed by the lightcone.

###  4.4 Selection of satellite replacements

In our simple model we match orphans to alternative non-orphaned satellite
galaxies with similar redshifts and PCA parameters. We investigated the impact
of additional matching to (i) the distance of the satellite galaxy from the
central galaxy, (ii) the virial halo mass before infall, and (iii) the SFH
before infall. We find no significant changes to the resulting mass functions.
This is likely due to our focus on quiescent galaxies which, by definition,
show little change in mass once quenched and thus do not vary significantly in
their properties with parameters such as distance from the central.
Additionally, many of these parameters are partially fit simply by fitting to
the PCA parameters before infall, which will likely account for secondary
dependencies on properties such as halo mass.

##  5 Conclusions

We construct a mock lightcone from the semi-analytic model L-Galaxies,
designed to closely match the observational constraints of a galaxy population
derived from the UKIDSS UDS. We show that this mock lightcone overproduces
low-mass passive galaxies and that this excess is robust to changes in the
mock catalogue construction process.

In the context of a semi-analytic model, orphan galaxies are defined as those
galaxies where the surrounding dark-matter halo can no longer be tracked.
Typically the host halo has been stripped as the galaxy orbits within a larger
galaxy cluster or group. In these cases the galaxy itself may survive but its
location is no longer known accurately. By isolating the orphan galaxy
population we show that the majority of the discrepancy seen in the stellar
mass functions across our redshift range of $0.5<z<3.0$ can be explained by an
improper treatment of this population. We show that this result holds
regardless of the choice of the underlying simulation, the selection method
used to identify quiescent galaxies, or the version of L-Galaxies used.

We argue that a more realistic treatment of orphan galaxies can be obtained
within semi-analytic models, such as L-Galaxies, by not quenching their star
formation as rapidly. Our suggestion is that the central regions of these
galaxies should retain an increased gas reservoir, either directly via a
larger supply of cold gas, or indirectly by retaining a hot gas reservoir out
to the virial radius for longer than the current stripping prescription
allows. By producing a toy model implementing this idea, we demonstrate a
version of L-Galaxies that shows a remarkable alignment with the observed
passive galaxy SMFs across the entire studied redshift range out to $z=3$.

Within the complex and challenging environment of a galaxy cluster the
extensive dark matter haloes of galaxies are naturally stripped away and
merged into the general halo of the cluster itself. In such an atmosphere some
galaxies will lose their entire surrounding dark matter envelope. Within a
simulation a galaxy’s host dark matter halo can disappear and be untraceable.
What to do with the galaxy in these cases is an ongoing and well-known problem
for semi-analytic modellers and there have been a variety of solutions
implemented. This population exists in all semi-analytic models, even if it is
sometimes neglected. The low-mass end of the passive population is
particularly sensitive to this orphan population and alignment with
observational results is much improved if some ongoing residual star formation
is allowed.

## Acknowledgements

This work was supported by the Science and Technology Facilities Council grant
numbers ST/X000982/1 and ST/X006581/1. We thank David Maltby, Will Hartley,
Kate Rowlands, and Vivienne Wild for valuable discussions.

## Data Availability

The data specifically shown in this paper will be shared upon request to the
corresponding author.

## References

  * Almaini et al. (2017) Almaini O., et al., 2017, Monthly Notices of the Royal Astronomical Society, 472, 1401
  * Angulo & White (2010) Angulo R. E., White S. D. M., 2010, Monthly Notices of the Royal Astronomical Society, 405, 143
  * Ashby et al. (2013) Ashby M. L. N., et al., 2013, The Astrophysical Journal, 769, 80
  * Asquith et al. (2018) Asquith R., et al., 2018, Monthly Notices of the Royal Astronomical Society, 480, 1197
  * Bamford et al. (2007) Bamford S. P., Milvang-Jensen B., Aragón-Salamanca A., 2007, Monthly Notices of the Royal Astronomical Society: Letters, 378, L6
  * Baugh (2006) Baugh C. M., 2006, Reports on Progress in Physics, 69, 3101
  * Behroozi et al. (2013) Behroozi P. S., Wechsler R. H., Conroy C., 2013, The Astrophysical Journal, 770, 57
  * Boylan-Kolchin et al. (2009) Boylan-Kolchin M., Springel V., White S. D. M., Jenkins A., Lemson G., 2009, Monthly Notices of the Royal Astronomical Society, 398, 1150
  * Brammer et al. (2008) Brammer G. B., Dokkum P. G. v., Coppi P., 2008, The Astrophysical Journal, 686, 1503
  * Bruzual & Charlot (2003) Bruzual G., Charlot S., 2003, Monthly Notices of the Royal Astronomical Society, 344, 1000
  * Cattaneo et al. (2017) Cattaneo A., et al., 2017, Monthly Notices of the Royal Astronomical Society, 471, 1401
  * Charlot & Fall (2000) Charlot S., Fall S. M., 2000, The Astrophysical Journal, 539, 718
  * Cimatti et al. (2004) Cimatti A., et al., 2004, Nature, 430, 184
  * Cimatti et al. (2006) Cimatti A., Daddi E., Renzini A., 2006, Astronomy & Astrophysics, 453, L29
  * Conroy & Gunn (2010) Conroy C., Gunn J. E., 2010, The Astrophysical Journal, 712, 833
  * Conroy et al. (2009) Conroy C., White M., Gunn J. E., 2009, The Astrophysical Journal, 708, 58
  * Cora et al. (2018) Cora S. A., et al., 2018, Monthly Notices of the Royal Astronomical Society, 479, 2
  * Cortese et al. (2021) Cortese L., Catinella B., Smith R., 2021, Publications of the Astronomical Society of Australia, 38, e035
  * Croton et al. (2016) Croton D. J., et al., 2016, The Astrophysical Journal Supplement Series, 222, 22
  * De Lucia & Blaizot (2007) De Lucia G., Blaizot J., 2007, Monthly Notices of the Royal Astronomical Society, 375, 2
  * De Lucia et al. (2024) De Lucia G., Fontanot F., Xie L., Hirschmann M., 2024, Tracing the Quenching Journey across Cosmic Time, doi:10.48550/arXiv.2401.06211, %****␣mnras_template.bbl␣Line␣125␣****https://ui.adsabs.harvard.edu/abs/2024arXiv240106211D
  * Donnari et al. (2021) Donnari M., Pillepich A., Nelson D., Marinacci F., Vogelsberger M., Hernquist L., 2021, Monthly Notices of the Royal Astronomical Society, 506, 4760
  * Fontanot et al. (2009) Fontanot F., De Lucia G., Monaco P., Somerville R. S., Santini P., 2009, Monthly Notices of the Royal Astronomical Society, 397, 1776
  * Furusawa et al. (2008) Furusawa H., et al., 2008, The Astrophysical Journal Supplement Series, 176, 1
  * Förster Schreiber & Wuyts (2020) Förster Schreiber N. M., Wuyts S., 2020, Annual Review of Astronomy and Astrophysics, 58, 661
  * Guo et al. (2011) Guo Q., et al., 2011, Monthly Notices of the Royal Astronomical Society, 413, 101
  * Hartley et al. (2013) Hartley W. G., et al., 2013, Monthly Notices of the Royal Astronomical Society, 431, 3045
  * Henriques et al. (2015) Henriques B. M. B., White S. D. M., Thomas P. A., Angulo R., Guo Q., Lemson G., Springel V., Overzier R., 2015, Monthly Notices of the Royal Astronomical Society, 451, 2663
  * Henriques et al. (2020) Henriques B. M. B., Yates R. M., Fu J., Guo Q., Kauffmann G., Srisawat C., Thomas P. A., White S. D. M., 2020, Monthly Notices of the Royal Astronomical Society, 491, 5795
  * Hopkins et al. (2010) Hopkins P. F., et al., 2010, The Astrophysical Journal, 724, 915
  * Ilbert et al. (2013) Ilbert O., et al., 2013, Astronomy & Astrophysics, 556, A55
  * Jarvis et al. (2013) Jarvis M. J., et al., 2013, Monthly Notices of the Royal Astronomical Society, 428, 1281
  * Kawinwanichakij et al. (2017) Kawinwanichakij L., et al., 2017, The Astrophysical Journal, 847, 134
  * Knebe et al. (2015) Knebe A., et al., 2015, Monthly Notices of the Royal Astronomical Society, 451, 4029
  * Lagos et al. (2023) Lagos C., et al., 2023, Quenching massive galaxies across cosmic time with the semi-analytic model SHARK v2.0
  * Lawrence et al. (2007) Lawrence A., et al., 2007, Monthly Notices of the Royal Astronomical Society, 379, 1599
  * Lee & Yi (2013) Lee J., Yi S. K., 2013, The Astrophysical Journal, 766, 38
  * Luo et al. (2016) Luo Y., Kang X., Kauffmann G., Fu J., 2016, Monthly Notices of the Royal Astronomical Society, 458, 366
  * Maltby et al. (2016) Maltby D. T., et al., 2016, Monthly Notices of the Royal Astronomical Society: Letters, 459, L114
  * McCarthy et al. (2008) McCarthy I. G., Frenk C. S., Font A. S., Lacey C. G., Bower R. G., Mitchell N. L., Balogh M. L., Theuns T., 2008, Monthly Notices of the Royal Astronomical Society, 383, 593
  * Murphy et al. (2022) Murphy G. G., Yates R. M., Mohamed S. S., 2022, Monthly Notices of the Royal Astronomical Society, 510, 1945
  * Muzzin et al. (2013) Muzzin A., et al., 2013, The Astrophysical Journal, 777, 18
  * Pozzetti et al. (2010) Pozzetti L., et al., 2010, Astronomy and Astrophysics, 523, A13
  * Simpson et al. (2013) Simpson C., Westoby P., Arumugam V., Ivison R., Hartley W., Almaini O., 2013, Monthly Notices of the Royal Astronomical Society, 433, 2647
  * Somerville & Davé (2015) Somerville R. S., Davé R., 2015, Annual Review of Astronomy and Astrophysics, 53, 51
  * Somerville et al. (2004) Somerville R. S., Lee K., Ferguson H. C., Gardner J. P., Moustakas L. A., Giavalisco M., 2004, The Astrophysical Journal, 600, L171
  * Springel et al. (2001a) Springel V., Yoshida N., White S. D. M., 2001a, New Astronomy, 6, 79
  * Springel et al. (2001b) Springel V., White S. D. M., Tormen G., Kauffmann G., 2001b, Monthly Notices of the Royal Astronomical Society, 328, 726
  * Springel et al. (2005) Springel V., et al., 2005, Nature, 435, 629
  * Srisawat et al. (2013) Srisawat C., et al., 2013, Monthly Notices of the Royal Astronomical Society, 436, 150
  * Taylor et al. (2023) Taylor E., Almaini O., Merrifield M., Maltby D., Wild V., Hartley W. G., Rowlands K., 2023, Monthly Notices of the Royal Astronomical Society, 522, 2297
  * Weinmann et al. (2009) Weinmann S. M., Kauffmann G., van den Bosch F. C., Pasquali A., McIntosh D. H., Mo H., Yang X., Guo Y., 2009, Monthly Notices of the Royal Astronomical Society, 394, 1213
  * Weinmann et al. (2012) Weinmann S. M., Pasquali A., Oppenheimer B. D., Finlator K., Mendel J. T., Crain R. A., Macciò A. V., 2012, Monthly Notices of the Royal Astronomical Society, 426, 2797
  * White & Frenk (1991) White S. D. M., Frenk C. S., 1991, The Astrophysical Journal, 379, 52
  * Wild et al. (2014) Wild V., et al., 2014, Monthly Notices of the Royal Astronomical Society, 440, 1880
  * Wild et al. (2016) Wild V., Almaini O., Dunlop J., Simpson C., Rowlands K., Bowler R., Maltby D., McLure R., 2016, Monthly Notices of the Royal Astronomical Society, 463, 832
  * Wilkinson et al. (2021) Wilkinson A., Almaini O., Wild V., Maltby D., Hartley W. G., Simpson C., Rowlands K., 2021, Monthly Notices of the Royal Astronomical Society, 504, 4533
  * Woo et al. (2017) Woo J., Carollo C. M., Faber S. M., Dekel A., Tacchella S., 2017, Monthly Notices of the Royal Astronomical Society, 464, 1077
  * Wuyts et al. (2007) Wuyts S., et al., 2007, The Astrophysical Journal, 655, 51
  * Xie et al. (2020) Xie L., De Lucia G., Hirschmann M., Fontanot F., 2020, Monthly Notices of the Royal Astronomical Society, 498, 4327


Machine learning refinement of in situ images acquired by low electron dose
LC-TEM
Running title: Machine learning refinement on low dose LC-TEM

Hiroyasu Katsuno1,∗, Yuki Kimura2, Tomoya Yamazaki2 and Ichigaku Takigawa3,4

1Emerging Media Initiative, Kanazawa University, Kakuma-machi, Kanazawa,
920-1192, Ishikawa, Japan

2Institute of Low Temperature Science, Hokkaido University, Kita-19, Nishi-8,
Kita-ku, Sapporo, 060-0819, Hokkaido, Japan

3Institute for Liberal Arts and Sciences, Kyoto University, 302 Konoe-kae, 69
Konoe-cho, Sakyo-ku, Kyoto, 606-8315, Kyoto, Japan

4Institute for Chemical Reaction Design and Discovery, Hokkaido University,
N21 W10, Kita-ku, Sapporo, 001-0021, Hokkaido, Japan

∗Corresponding author: E-mail<EMAIL_ADDRESS>TEL
+81-76-234-6935, FAX +81-76-234-6918

###### Abstract

We study a machine learning (ML) technique for refining images acquired during
in situ observation using liquid-cell transmission electron microscopy (LC-
TEM). Our model is constructed using a U-Net architecture and a ResNet
encoder. For training our ML model, we prepared an original image dataset that
contained pairs of images of samples acquired with and without a solution
present. The former images were used as noisy images and the latter images
were used as corresponding ground truth images. The number of pairs of image
sets was $1,204$ and the image sets included images acquired at several
different magnifications and electron doses. The trained model converted a
noisy image into a clear image. The time necessary for the conversion was on
the order of $10$ ms, and we applied the model to in situ observations using
the software Gatan DigitalMicrograph (DM). Even if a nanoparticle was not
visible in a view window in the DM software because of the low electron dose,
it was visible in a successive refined image generated by our ML model.

Key words: Transmission electron microscopy, Machine learning, in situ
observation, Liquid cell

## Introduction

Transmission electron microscopy (TEM) has been a powerful tool in the
research field of materials science because TEM has revealed phenomena at the
atomic scale such as an epitaxial Si(111) CoSi2 interface (Haider et al.,
1998), Au nanoparticles (Azubel et al., 2014), and order-disorder transition
(Jeon et al., 2021). Recent advances in the TEM technique have enabled the
acquisition of images of a sample in a solution as well as under vacuum
occurrence (liquid cell TEM, LC-TEM) (De Jonge and Ross, 2011; De Jonge et
al., 2019). Because of interactions between electrons and a solution, some
inevitable problems have arisen, such as blurred images (Holtz et al., 2013;
Nakajima et al., 2020; Yamazaki and Kimura, 2021) and the occurrence of
reactions due to the radiolysis of solutions (Schneider et al., 2014). Hence,
LC-TEM observations and the corresponding analysis of the resultant data
should be performed with these negative effects in mind.

One of the interesting nanoscale phenomena that occur in a solution is
nonclassical nucleation. The nucleation of crystals is the origin of all
materials and the general understanding of this phenomenon is summarized in
the classical nucleation theory (Markov, 2003). The classical nucleation
theory has only a single nucleation pathway in which molecules coalesce into
crystals. The justification of the classical nucleation theory has been
confirmed by experiments on the epitaxial growth of metal or semiconductor
materials (Michely and Krug, 2004). At the beginning of this century, new
nucleation phenomena (called nonclassical nucleation phenomena) attracted many
researchers, and various nucleation pathways have since been proposed (Yoreo
et al., 2015; Lee et al., 2016; Ishizuka et al., 2017; Driessche et al.,
2018). To clarify the nonclassical nucleation phenomena, it is necessary to
directly visualize phenomena at the nanoscale; LC-TEM is a candidate tool for
this purpose. However, TEM observation for an event in a solution is difficult
because achieving higher magnification, which results in a high electron dose,
unless another observation condition is used, such as ptychography (Gao et
al., 2017) and a higher accentuation voltage or an aberration corrector (Rose
and Urban, 2020). The radiolysis of a solution by the electron beam of a TEM
can induce unexpected reactions. In observations of samples that are sensitive
to electron beams, only a low electron dose can be used, resulting in an
unclear image. Recent advances in machine learning methods have enabled the
conversion of unclear microscopy images into clear images. In the case of TEM,
machine learning has also been successfully used for, for example, sparse
coding (Olshausen and Field, 1996; Leary et al., 2013; Stevens et al., 2014)
and convolutional neural networks (CNNs) (Madsen et al., 2018; Katsuno et al.,
2022a, b; Kimura et al., 2022; Krishna et al., 2023). The first CNN approach
for pictures acquired by optical cameras mimics a sparse coding method, and
its correspondence has been discussed (Dong et al., 2015). A CNN can convert
images in a short time, and real-time conversion has been realized in situ
observations by scanning TEM (STEM), where the STEM system was built to
support remote autonomous experiments using edge computing and storage systems
(Kalinin et al., 2023).

In this paper, we investigate whether CNN can be used for image refinement
during in situ observations with standard equipment. We prepared a handmade
dataset for the training of a machine learning model. The developed model was
evaluated by qualitative evaluation for some model encoders, and the time
necessary for image improvement was measured for certain image sizes. We
demonstrate visualization using a low electron dose for in situ observation.
Using a standard function of the software Gatan DM, which is the industry
standard software for image capture and analysis in (S)TEM, real-time
conversion is achieved without the addition of any system.

## Scheme of machine learning

### Collection of training data

The dataset for training is the basis of machine learning and plays a crucial
role and a good dataset will ensure that the model is properly trained
(Kotsiantis et al., 2006). The model trains patterns from the dataset and uses
these learned patterns to make predictions on new data. The size and the
diversity of the dataset are essential for managing overfitting and
underfitting. One of the data collection methods for image improvement is to
fabricate images using an image processing technique. In training a model for
super resolution, the dataset can be created by resizing the ground truth
image. Recently, the generation of simulated LC-TEM images was reported (Yao
et al., 2020). The training dataset was fabricated by not only using a
Gaussian filter to blur the image but also by using parameters on the
experimental equipment. Here, we adopted the method of making the dataset
directly by using a transmission electron microscope because there are some
unclear factors such as liquid thickness. Our transmission electron microscope
was equipped with a field-emission gun (JEM-2100F, JEOL, Tokyo) operated at an
acceleration voltage of $200$ kV, and a CMOS camera (One View IS, Gatan,
Pleasanton, CA, USA). The liquid cell consisted of a pair of silicon chips
with an amorphous silicon nitride membrane of $50$ nm thickness as an
observation window. The LC-TEM holder (Poseidon, Protochips, Morrisville, NC,
USA) is equipped with liquid injection ports, which were open in our
operation. A solution was injected into the liquid cell using a syringe when
necessary. Using our transmission electron microscope, we obtained a pair of
images: one acquired in a solution as a noisy image and one acquired without a
solution as a ground truth. First, a sample was placed in the liquid cell (no
injection of a solution) and images were acquired without a solution. The
space between the amorphous membranes was likely filled with air. A solution
was then introduced into the liquid cell with a spacer thickness of $500$ nm,
which controlled liquid thickness, and images were acquired in a solution. The
actual liquid thickness was much greater than the thickness of the spacer (see
Supplementary Material). This method is valid when the sample does not react
with the solution. We chose Au nanoparticles as a sample, and water as a
solution. When nanoparticles are encapsulated in a liquid cell, the particle
positions change dramatically before and after the liquid injection. That is,
it is not possible to obtain corresponding image pairs without and with a
solution. This problem was avoided by placing the sample outside the silicon
nitride membrane at the bottom side of the liquid cell. It is known that
images are different wether samples are placed on the top or bottom membrane
(Zaluzec, 2015). In our case, images of nanoparticles placed on the top
membrane were too blurry to be appropriate as a training dataset.
Nevertheless, automatic data collection was difficult because of the slight
difference in the position of the nanoparticles. The difference was probably a
consequence of the bulging of the membrane with a solution being suppressed by
the capillary force compared with the bulging of the membrane without a
solution. The pairs of acquired images were manually clipped to the region of
interest, and the number of image pairs in the training dataset was $1,204$.
90% of the pairs (1,083 pairs) were used for training and 10% (121 pairs) were
used for validation (Katsuno et al., 2023b).

The size of the acquired images was $4,096\times 4,096$ pixels. The typical
magnifications used in the present study were $10,000\times$, $20,000\times$,
and $100,000\times$. The typical exposure time was $0.04$–$0.5$ s for images
acquired in a solution and $1$–$5$ s for images acquired in the absence of a
solution. The lowest exposure time for $4,096\times 4,096$ images in the
software Gatan Digital Micrograph (DM) was $0.04$ s, that is, our dataset
included the image that the user observed during in situ observation and its
total dose is approximately $1$–$1,000\,$e-/nm2 for solution samples. The
distribution of the total dose of each whole image in our dataset is shown in
Figure 1a. The brightness differs by more than two orders of magnitude between
pairs of images. Figure 1b shows the structural similarity index measure
(SSIM) values [see equation 1] of paired images in the training (blue) and
validation (red) dataset. About half of the paired images are between 0.1 and
0.2 and the other half of paired images are uniformly distributed between
$0.2$ and $0.7$ in the training and validation datasets.

Fig. 1:  (a) Number distribution of the total dose of whole images acquired in
the absence of a solution (blue) and in a solution (red). (b) Number
distribution of SSIM of pair images in training (blue) and validation (red)
dataset.

### Machine learning model

Our machine learning model was constructed with the U-Net architecture and the
ResNet encoder using the segmentation package in Pytorch (Iakubovskii, 2019).
U-Net is a simple network used in segmentation and includes a skip connection
for maintaining the position of objects in the CNN (Ronneberger et al., 2015;
Falk et al., 2019). The ResNet encoder was developed for image recognition and
has been demonstrated to effectively avoid the problem of vanishing gradients
during training (He et al., 2016). After the final convolution layer of the
U-Net architecture, a sigmoid function was applied so that the intensity value
of each pixel was in the range $0$–$1$ because the output images were
quantitatively compared to images with their data range of $0$–$1$.

Fig. 2:  Schematics of making dataset and training of our machine learning
model for predicted images (P image) from noisy images (S image) acquired in a
solution. The model parameters are updated by using the loss function SSIM
shown in equation (1) so that the P image is identical to the corresponding T
image.

### Training

A schematic of the training procedure is shown in Figure 2. Before a solution-
acquired image of the training dataset was input into the machine learning
model, the image was resized to $512\times 512$ by using the resize function
with the option cv2.INTER_AREA provided by the OpenCV package (Bradski, 2000)
and its intensity data were rescaled to $0$–$1$. By comparing the case of
$512\times 512$ images of the camera output and the case of converting
$4,096\times 4,096$ images of camera output to $512\times 512$ images, we
confirmed that no artifacts are generated by the preprocess. Hereafter, we
refer to a preprocessed image as an S image. The S images were then randomly
flipped for data augmentations (DAs). In addition, a single image was
sometimes created from four images resized to $256\times 256$. The model
parameters were obtained using the Adam optimizer (Kingma and Ba, 2015) with a
learning rate of less than $10^{-4}$. The loss function was constructed using
the SSIM (Wang et al., 2004) with a predicted image (called P image) and
corresponding ground truth image (called T image). The SSIM is one of the
methods for evaluating the perceived quality of digital images. The SSIM can
be calculated from two images $x$ and $y$ of a common region,

$\displaystyle\textrm{SSIM}(x,y)=\frac{(2\mu_{x}\mu_{y}+c_{1})(2\sigma_{xy}+c_{2})}{(\mu_{x}^{2}+\mu_{y}^{2}+c_{1})(\sigma_{x}^{2}+\sigma_{y}^{2}+c_{2})},$
(1)

where $\mu_{x}$ and $\mu_{y}$ are the averages of $x$ and $y$, respectively.
$\sigma_{x}^{2}$ and $\sigma_{y}^{2}$ represent the variances of $x$ and $y$,
respectively and $\sigma_{xy}$ represents the covariance of $x$ and $y$.
Parameters $c_{1}$ and $c_{2}$ are introduced to stabilize the division and
are set to $0.01$ and $0.03$, respectively. Both values are commonly used such
as default values in the scikit-learn package (Pedregosa et al., 2011). The
SSIM of a pair of images is obtained over the entire image with the common
region shifted and indicates the average of equation 1. If two images are
identical, the SSIM becomes $1$, which is the maximum value. If there is no
similarity between the two images, the SSIM value is $0$. Using the SSIM, the
loss function is defined as

$\displaystyle\textrm{Loss
function}=1-\frac{1}{N}\sum_{\textrm{region}}\textrm{SSIM}(x,y),$ (2)

where $N$ represents the number of common regions whose size is $11\times 11$,
which is the standard value in the scikit-image package (Pedregosa et al.,
2011). We also used $3\times 3$ and $39\times 39$ as the common size, which
resulted in no significant change.

All training was conducted on a Linux machine with a 10-core Intel i9-9900X
3.5GHz CPU and an NVIDIA Quadro RTX $8000$ graphics card. The training was
almost converged at $1,000$ epochs, but the training was performed up to
$2,000$ epochs. The overfitting is a topic of ongoing discussion in the field
of machine learning (Belkin et al., 2018; Bartlett et al., 2020; Advani et
al., 2020). To suppress the overfitting, during the training, the data
augmentation with the randomness was performed and we confirmed that the value
of the loss function for validation did not increase. The training took about
$50$ minutes when using the ResNet-$18$ encoder without a pre-trained model
such as the ImageNet data. The code was available on the website (Katsuno et
al., 2023a).

## Results and discussion

### Examples of refined images

Fig. 3:  Examples of S, P, and T images. These images are not used for
training. S and T images were preprocessed images of the original images. All
objects are Au nanoparticles with a diameter of $100$ nm. White boxes in each
T image represent a $500$ nm scale for (c), (f), (i) and (l), and a $100$ nm
scale for (o), (r), (u) and (x). The magnitude of the scale of the width
differs from that of the height because clipped images are not square. The
number on the right side of the label on the S and T images represents the
total dose expressed in e-/nm2.

Figure 3 shows examples of preprocessed images of a sample in a solution (S
images), output images improved with our machine learning model (P images),
and images of a sample in the absence of a solution (T images). These images
were not used for training. All of the samples in the images are Au
nanoparticles with a diameter of $100$ nm. The number shown on the right side
of the label in each figure is the total dose. Even with an electron dose of
$1/1000$, we obtained P images comparable to T images. In the low-
magnification images in Figures 3a and 3d, a nanoparticle becomes relatively
small in the image; distinguishing between nanoparticles and noise in this
case is difficult. Our model successfully reproduces the positions of the
particles without missing them (Figures 3b and 3e). The refined images
(Figures 3b and 3e) are comparable to the corresponding T images (Figures 3c
and 3f) and do not produce extra particles. The P images (Figures 3h and 3k)
corresponding to the images in Figures 3g and 3h, in which many particles are
observed but the boundaries of particles are blurred and unclear, are
equivalent to the T images in Figures 3h and 3k, respectively. Image
acquisition with high magnification brings additional benefits. In the S
images, the particles appear spherical because of the blurring caused by the
solution (Figures 3m, 3p, 3s and 3v). In the P images, objects with polygonal
shapes are observed (Figures 3n, 3q, 3t and 3w), in agreement with the images
acquired in the absence of a solution (Figures 3o, 3r, 3u and 3x,
respectively). We found that the shape restoration is effective even when
particles are located adjacent to each other.

### Quality of improvement

Table 1:  PSNR and SSIM of pairs of a P image and T image for the combination of loss functions and ResNet encoders. ‘Original’ represents the pairs of an S image and a corresponding T image.  Loss function | Encoder | PSNR | SSIM
---|---|---|---
SSIM | ResNet-$18$ | $29.52$ | $0.85$
SSIM | ResNet-$34$ | $29.45$ | $0.85$
SSIM | ResNet-$50$ | $29.27$ | $0.85$
SSIM | ResNet-$101$ | $30.12$ | $0.85$
SSIM | ResNet-$152$ | $29.87$ | $0.85$
$L_{1}$ | ResNet-$18$ | $29.99$ | $0.85$
$L_{2}$ | ResNet-$18$ | $29.65$ | $0.85$
(Original) | $8.48$ | $0.08$

Table 1 shows the quantitative evaluations of our model in terms of the peak
signal-to-noise ratio (PSNR) and the SSIM. Both parameters are one of the most
frequently used evaluations in machine learning models such as (Chen et al.,
2018). The PSNR is a logarithm of the inverse of the mean square error at the
pixel level. A high PSNR value indicates that the two images are similar to
each other. The SSIM is the same as defined in equation 1, and the model
parameters are trained such that the SSIM is equal to $1$. PSNR and SSIM are
widely used as image quality degradation evaluation methods.

The S images are very noisy, and finding Au nanoparticles at low magnification
is difficult. After our model is applied, S images are converted to P images.
When the encoder ResNet-$18$ is used, the average PSNR and SSIM for P images
and the corresponding T images are $29.52$ and $0.85$, respectively. Because
noise was reduced, the P images are very similar to the corresponding T
images. We have evaluated other ResNet encoders listed in Table 1. Although
large encoders are expected to obtain better results, we observed no
significant change compared with the results obtained using the smallest
encoder, ResNet-$18$. Our results show that the smallest encoder is sufficient
in our case of improving grayscale images with $16$-bit expression in each
pixel.

We also evaluated different types of the loss function. The $L_{1}$ ($L_{2}$)
loss function is the mean absolute (squared) error and the absolute (squared)
difference in the intensity at each pixel. The obtained model parameters were
comparable to those in the case where the SSIM was used. As already pointed
out in previous studies (Chen et al., 2018), changing the loss function makes
no significant difference in this framework although it affects the magnitude
of the gradient during training.

The ’Original’ column represents the values of the pairs of S and
corresponding T images. The average PSNR and SSIM in the dataset of S and T
images are obtained as $8.48$ and $0.08$, respectively. Our model is able to
improve the image beyond simple data rescaling performed in the preprocessing.

### Time necessary

Table 2:  Conversion time of one image for various sizes of an image. The time for a GPU includes the data transfer between the CPU and GPU.  Processor | $256\times 256$ | $512\times 512$ | $1,024\times 1,024$
---|---|---|---
Intel i7-9700 |  $106$ ms |  $240$ ms |  $787$ ms
NVIDIA Geforce GTX 1650 |  $12.1$ ms |  $31.4$ ms |  $133$ ms
NVIDIA RTX A2000 |  $8.0$ ms |  $13.3$ ms |  $40.8$ ms
NVIDIA Quadro RTX 8000 |  $4.3$ ms |  $5.9$ ms |  $19.1$ ms

We investigated the conversion time required for the conversion of an S image
by our machine learning model. The model encoder for this test was
ResNet-$18$. For the comparison, one CPU (Intel i7-9700) and three different
GPUs (NVIDIA Geforce GTX 1650, NVIDIA A2000, and NVIDIA Quadro RTX 8000) were
used. The results are summarized in Table 2. When the image size is smaller,
the conversion time becomes shorter. When an Intel i7-9700 was used, the
conversion time was greater than $100$ ms even for the $256\times 256$ images.
Although there is no problem for normal use, this long conversion time reduces
the temporal resolution of in situ observation using DM. When GPUs were used,
the conversion time for the $1,024\times 1,024$ image was four times longer
than that for the $512\times 512$ image. This result is reasonable because the
image data is four times larger. When the most powerful processor was used,
the time to process a $256\times 256$ image was reduced to only $70\%$ of the
time to process a $512\times 512$ image. By contrast, for the least powerful
processors, the processing time for a $256\times 256$ image was less than
$50$% of the time to process a $512\times 512$ image. That is, the more
powerful processors tended to offer less benefit from a reduction of the image
size. We chose an image size of $512\times 512$ for the conversion under in
situ observation.

The time required depends largely on machine power. Therefore, the image might
be improved within a short time if a machine dedicated to image improvement is
used. Even in the current state, we found that the refinement of images of in
situ observation is possible by simply integrating a reasonably priced GPU
into a PC used to control a transmission electron microscope, without building
a dedicated machine for image refinement.

### Test for in situ observation

Fig. 4:  Examples of images for P images under in situ observation using the
software Gatan DM. (a): Time change of the total dose (filled circles) and
PSNR (crosses) with respect to (l). for each image. (b)–(e): Images acquired
with an exposure time of $0.01$ s at $t=0,10,13$ and $23$ s, as shown by down
arrows in (a). (f)–(i): Images refined from (b)–(e), respectively, by our
machine learning model. (j)–(m): Each of these images was created by combining
the images (b)–(e) and nine images preceding them, and then refining them by
our machine learning model. The scale bar shown in (b) represents $500$ nm.

To test whether our machine learning model is effective for in situ
observations, we observed Au nanoparticles in a solution. When an image with
the current maximum size of $4,096\times 4,096$ was acquired, the system
became slow when outputting acquired images to the view window. When a
$2,048\times 2,048$ image was acquired with an exposure time of $0.01$ s, the
appearance in the view window showed the same temporal resolution, as if the
machine learning model did not work. When a $2,048\times 2,048$ image with a
magnification of $20,000\times$ was acquired, the images were converted to
$512\times 512$ and the data were rescaled, corresponding to the preprocessing
shown in Figure 2. Preprocessed images were input to the machine learning
model, and the output images were displayed. All processes were performed on
the software Gatan DM and NVIDIA RTX A2000 GPU. Although the time necessary
for the conversion is $13.3$ ms from Table 2, the time necessary from a camera
output to display a refined image is $63$ ms. Therefore, the time necessary
for drawing in Gatan DM with python script is about $50$ ms. Although the
recording was made with $100$ fps, the images are displayed about $10$ Hz, we
had no problems with the slow display in actual in situ TEM observation.

Because our goal is to recognize the presence or absence of samples in a
solution at very low electron doses, we used the Gatan DM software to compare
the raw view images and refined images collected at various electron doses.
The change in the electron dose is shown in Figure 4a, where the magnitude of
the electron dose was measured in the background. The left $y$-axis shows the
dose rate per image. The right $y$-axis shows the PSNR of each image when
Figure 4l is used instead of the S image. Some of the raw images viewed in
Gatan DM are shown in Figures 4b–4e. The time corresponding to each image is
indicated by the arrow in Figure 4a. The electron dose represented in Figure
4b is approximately 1 e-/nm2, where the dose rate corresponds to 100 e-/nm2 s.
We could not find Au nanoparticles in Figure 4b. In the refined image in
Figure 4f, an object is observed at the center. The electron dose represented
in Figure 4c is about 10 e-/nm2, and an object is visible. In the refined
images in Figure 4g, an object is clearly observed but the shape restoration
of the samples could not be realized. The electron dose represented in Figure
4d is approximately 80 e-/nm2, which is in the range of the electron dose used
in our training dataset (see Figure 1a); some particles of well-defined size
are visible although the image is very noisy. The refined image in Figure 4h
provides the same information for samples but without noise. After acquiring
the image in Figure 4d, we again acquired the image in Figure 4e whose refined
image in Figure 4i shows the object at the center as dose Figure 4f. Because
the electron dose rate we usually use during the search for nanoparticles in
samples is approximately $10,000$ e-/nm2 s, we can detect an object with an
electron dose of $1/100$.

Our model was trained well without over-fitting because the model can refine
the images in Figures 4c and d. This behavior was expected, as shown in Figure
3. In these dark images, we observe nanoparticles. Using this model, we
succeeded not only in brightening but also in visualizing images that are
difficult to judge with the human eye (Figures 4b and 4e). In an actual
adjustment, the presence of an object can be seen more clearly if the previous
nine images are integrated and refined (Figures 4j–4m) rather than simply
refined one by one. The total electron dose in the input images corresponding
to Figures 4j–4m is approximately ten times greater than those in Figures
4b–4e. More discriminative images can be obtained by adding the preceding
images that were already acquired for the image.

We here consider the other aspect of the effect of electron-beam irradiation
of a sample. After the TEM observation begins, water in the sample will reach
a steady state within 10-3 s under the TEM observation conditions. The action
of ionization is inevitable. In particular, it affects the pH of the water
(Schneider et al., 2014). The dose rate during an ordinary TEM observation is
approximately 107 Gy/s, and the pH does not change when the initial pH of a
liquid is less than 4. Because our work has reduced the total dose required
for TEM visualization by two orders of magnitude, we can visualize phenomena
in a solution with a pH less than 6 without substantially changing its pH.

Currently, our method can reveal where objects are located but cannot
reproduce their structure in detail. Our model tends to binarize the image. We
were not sure why output images would be binarized. One possibility is a
sigmoid function put just before the output to avoid negative values. However,
the sigmoid function is probably not the direct cause, since another study
(Katsuno et al., 2022b) has shown that even cases without a sigmoid function
also tend to be binarized. Another possibility is the small size of the
dataset. Although we assume that a larger dataset is necessary to reproduce a
detailed structure, large datasets are difficult to create in a
straightforward manner. It would be necessary to create a pseudo-dataset in a
way that uses empirical functions such as the work (Yao et al., 2020). If
large datasets could be created, other models such as generative adversarial
networks (GANs) (Goodfellow et al., 2014, 2020) would be better suited for use
with our method. We attempted to use GAN, but the result was unstable under
the current situation with a small dataset; we therefore did not adopt it in
the present work.

The Au nanoparticles in this manuscript are located on the outside of the
lower membrane of the cell. We have tested this model on several images in
which the particles are placed inside the cell, with moderately good results.
This result means that our proposed machine learning model can be used for
ordinary LC-TEM observation.

### Summary

We developed a method to improve in situ TEM images using machine learning
techniques. Our model was constructed with the U-Net architecture and the
ResNet-$18$ encoder. The training dataset was prepared from experimental sets
of noisy images acquired in a solution and corresponding truth images acquired
in the absence of a solution. The PSNR and the SSIM of the improved images
were $29.52$ and $0.85$, respectively. Our model was applied using the
software Gatan DM, which controls the CMOS camera of a transmission electron
microscope. The refined images enabled us to observe nanoparticles that could
not be observed in the view window without almost loss of temporal resolution.
Although our dataset was the image of particles outside of the cell, machine
learning model trained using this dataset would be applicable to ordinary LC-
TEM observation in which samples were located inside the cell.

## Acknowledgments

This work was supported by JSPS KAKENHI Grant Numbers 20H05657 and 21K03379,
the Grant for Joint Research Program of the Institute of Low Temperature
Science, Hokkaido University (23G016), and the Grant for the Public Foundation
of Chubu Science and Technology Center.

## References

  * Advani et al. [2020] M. S. Advani, A. M. Saxe, and H. Sompolinsky.  High-dimensional dynamics of generalization error in neural networks.  _Neural Networks_ , 132:428–446, 2020.
  * Azubel et al. [2014] M. Azubel, J. Koivisto, S. Malola, D. Bushnell, G. L. Hura, A. L. Koh, H. Tsunoyama, T. Tsukuda, M. Pettersson, H. Häkkinen, et al.  Electron microscopy of gold nanoparticles at atomic resolution.  _Science_ , 345(6199):909–912, 2014.
  * Bartlett et al. [2020] P. L. Bartlett, P. M. Long, G. Lugosi, and A. Tsigler.  Benign overfitting in linear regression.  _Proceedings of the National Academy of Sciences_ , 117(48):30063–30070, 2020.
  * Belkin et al. [2018] M. Belkin, D. J. Hsu, and P. Mitra.  Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate.  _Advances in neural information processing systems_ , 31, 2018.
  * Bradski [2000] G. Bradski.  The OpenCV Library.  _Dr. Dobb’s Journal of Software Tools_ , 2000.
  * Chen et al. [2018] C. Chen, Q. Chen, J. Xu, and V. Koltun.  Learning to see in the dark.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3291–3300, 2018.
  * De Jonge and Ross [2011] N. De Jonge and F. M. Ross.  Electron microscopy of specimens in liquid.  _Nature nanotechnology_ , 6(11):695–704, 2011\.
  * De Jonge et al. [2019] N. De Jonge, L. Houben, R. E. Dunin-Borkowski, and F. M. Ross.  Resolution and aberration correction in liquid cell transmission electron microscopy.  _Nature Reviews Materials_ , 4(1):61–78, 2019\.
  * Dong et al. [2015] C. Dong, C. C. Loy, K. He, and X. Tang.  Image super-resolution using deep convolutional networks.  _IEEE transactions on pattern analysis and machine intelligence_ , 38(2):295–307, 2015.
  * Driessche et al. [2018] A. E. S. V. Driessche, N. V. Gerven, P. H. H. Bomans, R. R. M. Joosten, H. Friedrich, D. Gil-Carton, N. A. J. M. Sommerdijk, and M. Sleutel.  Molecular nucleation mechanisms and control strategies for crystal polymorph selection.  _Nature_ , 556:89–96, 2018.
  * Falk et al. [2019] T. Falk, D. Mai, R. Bensch, Ö. Çiçek, A. Abdulkadir, Y. Marrakchi, A. Böhm, J. Deubner, Z. Jäckel, K. Seiwald, et al.  U-net: deep learning for cell counting, detection, and morphometry.  _Nat. Methods_ , 16(1):67–70, 2019.
  * Gao et al. [2017] S. Gao, P. Wang, F. Zhang, G. T. Martinez, P. D. Nellist, X. Pan, and A. I. Kirkland.  Electron ptychographic microscopy for three-dimensional imaging.  _Nature communications_ , 8(1):163, 2017.
  * Goodfellow et al. [2020] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial networks.  _Communications of the ACM_ , 63(11):139–144, 2020.
  * Goodfellow et al. [2014] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  _Advances in Neural Information Processing Systems_ , pages 2672–2680, 2014.
  * Haider et al. [1998] M. Haider, S. Uhlemann, E. Schwan, H. Rose, B. Kabius, and K. Urban.  Electron microscopy image enhanced.  _Nature_ , 392(6678):768–769, 1998.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Holtz et al. [2013] M. E. Holtz, Y. Yu, J. Gao, H. D. Abruña, and D. A. Muller.  In situ electron energy-loss spectroscopy in liquids.  _Microscopy and Microanalysis_ , 19(4):1027–1035, 2013.
  * Iakubovskii [2019] P. Iakubovskii.  Segmentation models pytorch, 2019.  URL https://github.com/qubvel/segmentation\\_models.pytorch.
  * Ishizuka et al. [2017] S. Ishizuka, Y. Kimura, S. Yokoi, T. Yamazaki, R. Sato, and T. Hama.  Self-assembly of MoO3 needles in gas current for cubic formation pathway.  _Nanoscale_ , 9:10109–10116, 2017.
  * Jeon et al. [2021] S. Jeon, T. Heo, S.-Y. Hwang, J. Ciston, K. C. Bustillo, B. W. Reed, J. Ham, S. Kang, S. Kim, J. Lim, et al.  Reversible disorder-order transitions in atomic crystal nucleation.  _Science_ , 371(6528):498–503, 2021.
  * Kalinin et al. [2023] S. V. Kalinin, D. Mukherjee, K. M. Roccapriore, B. Blaiszik, A. Ghosh, M. A. Ziatdinov, A. Al-Najjar, C. Doty, S. Akers, N. S. Rao, J. C. Agar, and S. R. Spurgeon.  Deep learning for automated experimentation in scanning transmission electron microscopy, 2023.
  * Katsuno et al. [2022a] H. Katsuno, Y. Kimura, T. Yamazaki, and I. Takigawa.  Early detection of nucleation events from solution in LC-TEM by machine learning.  _Frontiers in Chemistry_ , 10:818230, 2022a.
  * Katsuno et al. [2022b] H. Katsuno, Y. Kimura, T. Yamazaki, and I. Takigawa.  Fast improvement of tem images with low-dose electrons by deep learning.  _Microscopy and microanalysis_ , 28(1):138–144, 2022b.
  * Katsuno et al. [2023a] H. Katsuno, Y. Kimura, T. Yamazaki, and I. Takigawa.  Machine-learning-refinement-of-images-acquired-by-LC-TEM, 2023a.  URL https://github.com/hiroyasukatsuno/Machine-learning-refinement-of-images-acquired-by-LC-TEM.
  * Katsuno et al. [2023b] H. Katsuno, Y. Kimura, T. Yamazaki, and I. Takigawa.  Dataset for machine learning refinement of in situ images acquired by low electron dose LC-TEM, 2023b.  URL https://doi.org/10.5281/zenodo.10042733.
  * Kimura et al. [2022] Y. Kimura, H. Katsuno, and T. Yamazaki.  Possible embryos and precursors of crystalline nuclei of calcium carbonate observed by liquid-cell transmission electron microscopy.  _Faraday Discussions_ , 235:81–94, 2022.
  * Kingma and Ba [2015] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  In Y. Bengio and Y. LeCun, editors, _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.
  * Kotsiantis et al. [2006] S. B. Kotsiantis, I. D. Zaharakis, and P. E. Pintelas.  Machine learning: a review of classification and combining techniques.  _Artificial Intelligence Review_ , 26:159–190, 2006.
  * Krishna et al. [2023] K. M. Krishna, R. Madhavan, M. V. Pantawane, R. Banerjee, and N. B. Dahotre.  Machine learning based de-noising of electron back scatter patterns of various crystallographic metallic materials fabricated using laser directed energy deposition.  _Ultramicroscopy_ , page 113703, 2023.
  * Leary et al. [2013] R. Leary, Z. Saghi, P. A. Midgley, and D. J. Holland.  Compressed sensing electron tomography.  _Ultramicroscopy_ , 131:70–91, 2013.
  * Lee et al. [2016] S. Lee, H. S. Wi, W. Jo, Y. C. Cho, H. H. Lee, S.-Y. Jeong, Y.-I. Kim, and G. W. Lee.  Multiple pathways of crystal nucleation in an extremely supersaturated aqueous potassium dihydrogen phosphate (KDP) solution droplet.  _Proc. Natl. Acad. Sci. USA_ , 113(48):13618–13623, 2016.
  * Madsen et al. [2018] J. Madsen, P. Liu, J. Kling, J. B. Wagner, T. W. Hansen, O. Winther, and J. Schiøtz.  A deep learning approach to identify local structures in atomic-resolution transmission electron microscopy images.  _Advanced Theory and Simulations_ , 1(8):1800037, 2018.
  * Markov [2003] I. V. Markov.  _Crystal Growth for Beginners 2nd ed._ World Scientific Publishing Co. Ltd., 27 Warren Street, Suite 401–402, Hackensack, NJ 07601, 2003.
  * Michely and Krug [2004] T. Michely and J. Krug.  _Island, Mounds and Atoms —Patterns and Processes in Crystal Growth Far from Equilibrium—_.  Springer–Verlag Berlin Heidelberg, 2004.
  * Nakajima et al. [2020] K. Nakajima, T. Yamazaki, Y. Kimura, M. So, Y. Goto, and H. Ogi.  Time-resolved observation of evolution of amyloid-$\beta$ oligomer with temporary salt crystals.  _The Journal of Physical Chemistry Letters_ , 11(15):6176–6184, 2020.
  * Olshausen and Field [1996] B. A. Olshausen and D. J. Field.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images.  _Nature_ , 381(6583):607–609, 1996.
  * Pedregosa et al. [2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011\.
  * Ronneberger et al. [2015] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Rose and Urban [2020] M. H. Rose and K. Urban.  In recognition of aberration-corrected tem.  _Nat. Nanotechnol._ , 15:417, 2020.
  * Schneider et al. [2014] N. M. Schneider, M. M. Norton, B. J. Mendel, J. M. Grogan, F. M. Ross, and H. H. Bau.  Electron–water interactions and implications for liquid cell electron microscopy.  _The Journal of Physical Chemistry C_ , 118(38):22373–22382, 2014.
  * Stevens et al. [2014] A. Stevens, H. Yang, L. Carin, I. Arslan, and N. D. Browning.  The potential for bayesian compressive sensing to significantly reduce electron dose in high-resolution stem images.  _Microscopy_ , 63(1):41–51, 2014.
  * Wang et al. [2004] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ , 13(4):600–612, 2004.
  * Yamazaki and Kimura [2021] T. Yamazaki and Y. Kimura.  Radiolysis-induced crystallization of sodium chloride in acetone by electron beam irradiation.  _Microscopy and microanalysis_ , 27(3):459–465, 2021.
  * Yao et al. [2020] L. Yao, Z. Ou, B. Luo, C. Xu, and Q. Chen.  Machine learning to reveal nanoparticle dynamics from liquid-phase tem videos.  _ACS central science_ , 6(8):1421–1430, 2020\.
  * Yoreo et al. [2015] J. J. D. Yoreo, P. U. P. A. Gilbert, N. A. J. M. Sommerdijk, R. L. Penn, S. Whitelam, D. Joester, H. Zhang, J. D. Rimer, A. Navrotsky, J. F. Banfield, A. F. Wallace, F. M. Michel, F. C. Meldrum, H. Cőelfen, and P. M. Dove.  Crystallization by particle attachment in synthetic, biogenic, and geologic environments.  _Science_ , 349:aaa6760, 2015.
  * Zaluzec [2015] N. J. Zaluzec.  The influence of cs/cc correction in analytical imaging and spectroscopy in scanning and transmission electron microscopy.  _Ultramicroscopy_ , 151:240–249, 2015.