# Automated image acquisition for low-dose STEM at atomic resolution

###### Abstract

Beam damage is a major limitation in electron microscopy that becomes
increasingly severe at higher resolution. One possible route to circumvent
radiation damage, which forms the basis for single-particle electron
microscopy and related techniques, is to distribute the dose over many
identical copies of an object. For the acquisition of low-dose data, ideally
no dose should be applied to the region of interest prior to the acquisition
of data. We present an automated approach that can collect large amounts of
data efficiently by acquiring images in an user-defined area-of-interest with
atomic resolution. We demonstrate that the stage mechanics of the Nion
UltraSTEM, combined with an intelligent algorithm to move the sample, allows
the automated acquisition of atomically resolved images from micron-sized
areas of a graphene substrate. Moving the sample stage automatically in a
regular pattern over the area-of-interest enables the collection of data from
pristine sample regions without exposing them to the electron beam before
recording an image. Therefore, it is possible to obtain data with minimal dose
(no prior exposure from focusing), which is only limited by the minimum signal
needed for data processing. This enables us to prevent beam induced damage in
the sample and to acquire large datasets within a reasonable amount of time.

Andreas Mittelberger, Christian Kramberger, Christoph Hofer, Clemens Mangler,
Jannik C. Meyer*

University of Vienna, Faculty of Physics, Boltzmanngasse 5, 1090 Vienna,
Austria

*corresponding author<EMAIL_ADDRESS>
## Introduction

Automating data acquisition tasks in transmission electron microscopy (TEM)
started with the availability of CCD cameras and computer-controlled TEMs back
in the early 90’s [Dierksen et al., 1992, Koster et al., 1992]. One of the
main goals behind automation was, and still is [Mastronarde, 2005], reducing
the dose needed in electron tomography as well as making tilt series
acquisition less prone to errors introduced by manual re-centering and re-
focusing. Another area where automated data acquisition is important is
nowadays in single-particle electron microscopy [Suloway et al., 2005, Shi et
al., 2008, Zhang et al., 2001, Zhang et al., 2003, Zhang et al., 2010, Cheng,
2015, Bartesaghi et al., 2015, Frank et al., 1978]. This technique uses low-
dose images of many copies of the same object to reduce beam damage in highly
radiation-sensitive specimens such as biological samples. After collecting
hundreds to thousands of images containing a total of up to a few ten thousand
particles, the dataset has to be aligned and classified. Classification means
that particle images are divided into groups with similar orientation and
conformation which makes it possible to average them in the next step and
therefore to greatly increase the signal-to-noise ratio. The resulting dataset
consists now of different two-dimensional projections of the imaged particles
which can be used to calculate a three-dimensional model in the next step
[Cheng et al., 2015].

Besides using small doses for the actual image acquisition, “low-dose” also
means separating the focusing and sample-tracking part from the actual
recording of image data. In the ideal case, every snapshot should be recorded
from a “fresh” region of the sample, i.e., no dose should be applied to the
image area for focusing and the image acquisition should start with the first
electron in the respective field of view. Our approach is closely related to
automated methods used in single-particle analysis with the difference that we
aim for much smaller structures, and atomic resolution. Our test sample is a
graphene sheet suspended over the holes of a Quantifoil(TM) grid, which serves
very well for judging the alignment and focus precision. Directly conceivable
applications would be the imaging of radiation-sensitive defects in 2D
materials, or the analysis of small molecules deposited on the graphene
support or embedded in a graphene sandwich. However, applications to other
ultra-thin samples, such as molecules embedded in thin films also do not seem
far-fetched as long as the membranes are sufficiently flat and thin. This type
of data acquisition is also relevant for our recently published algorithm for
structure recovery from low-dose (S)TEM images, which requires up to a few
thousand images for a successful reconstruction [Meyer et al., 2014,
Kramberger and Meyer, 2016]. We further demonstrate that we can reconstruct
defects created by electron beam irradiation in the graphene lattice by
feeding low-dose data acquired with the method described here into this
algorithm.

## Materials and Methods

### Overview of the mapping procedure

Electron doses low enough to image for example individual organic molecules
have to be below $10^{0}-10^{3}\,e{}^{-}/\mathring{A}^{2}$ for typical (S)TEM
high-voltages [Egerton, 2012]. In contrast, atomically resolved images of
light elements such as carbon require doses of at least
$10^{5}\,e{}^{-}/\mathring{A^{2}}$. This means that tuning and focusing have
to take place in sample areas other than the area-of-interest. We achieve this
by manually setting a focus reference on the four corners of a quadrangle that
surrounds the area of interest. This focus references and all further focus
corrections are only set electronically without adjustment of the mechanical
height (z) drive. After the user has set the imaging parameters for the map,
our program fills the quadrangle with a regular array of image positions and
interpolates the focus at each point (see figure 1). Now, the aim is to move
the stage to each position and acquire an image. Remarkably, the deviation of
the stage from the desired in-plane (x,y) coordinates is quite small
(typically below 25 nm for movements within $\sim 1\mu\textrm{m}$) and it
appears that stage movements in x- and y-direction do not lead to any
detectable random changes in sample height (z). More importantly, the
deviation in (x,y) from the target position is highly reproducible if the
sequence of (x,y) positions is driven in a specific sequence. Hence, it
becomes possible to measure and compensate the small remaining inaccuracies of
the stage drive. So far, it appears that a “reference” measurement for this
purpose is needed only once for this sample stage and (x,y) position pattern.
This procedure of course requires the microscope to provide stable imaging
conditions during the whole data acquisition, which is typically up to a few
hours. With the present parameters we can acquire about 1000 images per hour.

Figure 1: (a) Schematic of the approach. Four positions are used as focus
reference (green dots) and then the central part of the membrane is scanned by
moving the stage and interpolating the focus value. (b) Example of a high-dose
image of a graphene membrane, recorded by moving to a fresh area automatically
and using focus interpolation without further adjustment.

### Details of the implementation

Our microscope is controlled via Nion’s own user software, Swift, which is
python-based and also provides an API for custom extensions. Therefore all
programs used for this paper were also written in python as plug-ins for Nion
Swift. In the following paragraph, some details of the mapping process and its
implementation will be explained.

We chose a serpentine-path as mapping pattern e.g. the stage moves from left
to right in odd and from right to left in even lines. The reason for this is
that while the sample stage suffers only very little from backlash and drift,
it is still favorable to avoid large stage movements. To account for the still
present sample drift after moving the stage, we use a delay of typically 2 s
between moving the stage and acquiring the image. The electron beam is blanked
during that time to prevent additional dose put into the area-of-interest.
Moreover, the serpentine pattern is oriented to be aligned with the natural
stage coordinates (i.e., movement along a line uses only the x or y drive, and
not a combination of x and y). As the four user-selected points usually form
an arbitrary quadrangle, the largest-possible rectangle that is aligned with
the natural stage coordinates is inscribed into them. This area is then filled
with an evenly spaced grid of coordinates, based on the image size and the
chosen offset between the images. For each stage position the correct focus is
calculated via a bilinear interpolation function. An image is then acquired at
each stage position and saved on hard disk. Also an overview image is taken
after the map has been completed.

With this simple approach it is possible to acquire maps consisting of up to
about 1000 images with reasonable quality. As a measure for judging the focus
precision, we use Fourier transforms (FTs) of the acquired images. For
periodic structures like graphene, the peaks in the FTs are still present even
in low-dose exposures where the lattice is hidden in the noise. For an
incoherent (annular dark-field, ADF) image, the FT peak intensities are a
simple but accurate measure for the quality of atomically resolved images:
Figure 2 shows the sum of the first- and second-order peaks of a graphene
lattice in a simulated focus series. Especially the first-order peaks show a
sharp maximum at zero defocus. Because all our samples are either pure
graphene or at least graphene supported, the 12 peaks of the graphene lattice
can be used for judging image quality.

Figure 2: Plot of the sum of all visible FT spot intensities of a simulated
ADF-STEM focus series of graphene. The peak intensity sum is strongly
dependent on the amount of defocus and can therefore be used to judge image
quality. Jumps in the curve occur when the number of peaks included in the
intensity sum decreases (because they become too weak to be found). The number
of peaks included in the sum is visualized by a color code in the image.

## Results and Discussion

To test our approach, we recorded atomically resolved high-dose maps from
graphene. This is no option for beam-sensitive samples, but for pure graphene,
which is stable at 60 kV, it provides a way to verify that atomic resolution
is reached in the single images. In figure 3 such a map is shown where all
images were aligned on an overview image. For this purpose, the random
contamination pattern that is typically found on graphene samples provides a
perfect reference structure that is well visible in both the low-magnification
overview and the high-resolution snapshots. Zooming into a small area of this
map reveals the graphene lattice in those images and shows that we can reach
atomic resolution in almost all exposures. In addition to judging the single
image quality, also the accuracy of stage movement can be seen via this
analysis as discussed later in more detail.

Figure 3: Zoom into a part of an automatically acquired high-dose map
consisting of 12x17 images. (a) The atomically resolved images were aligned on
an overview image in order to find their actual positions on the sample. (b)
An about 10 times magnified part of the map highlighted by the blue rectangle
in (a). The contamination pattern in the background stems from the overview
image and was used to align the small images with respect to the overview. (c)
An about 5 times zoom into the area of (b) marked with a red rectangle reveals
the atomic lattice in the image.

As already mentioned earlier, the FT peak intensity sum in an image provides a
good measure for its tuning. This provides the means to quantify the overall
focus precision in our automatically acquired maps. Figure 4 shows a histogram
of all peak intensity sums of one high-dose map. In panel a, b and c of figure
4 there are three images that correspond to a poorly focused, an averagely
focused and a very well focused image of this map, respectively. According to
the histogram in figure 4 d, most images of a map suffer from slight defocus
which is, however, still small enough to reach atomic resolution (c.f. figure
4 b). Assuming that the best images of a map are acquired at zero defocus, an
estimate for the amount of defocus can be made by assigning a defocus to the
relative peak intensity sum via figure 2. The maximum peak intensity sum for
an image of the map shown in figure 4 is about 16,000 which we assume to be at
zero defocus. Therefore, images with a peak intensity sum of 8,000 (which is
half of the maximum value) are, according to figure 2, recorded with a defocus
of $\pm$5 nm. A closer look to such an image (see figure 4 b) still clearly
shows the atomic lattice, indicating that a slight defocus does not
significantly decrease image quality. This finding corresponds well with the
expected depth-of-field for our microscope which is about 10 nm at an
acceleration voltage of 60 kV and an aperture half-angle of 35 mrad [Fultz and
Howe, 2012].

Figure 4: Histogram of the peak intensity sums of the map in figure 3. (a) One
of the worst focused images of the map. (b) An image with average focus
precision. (c) One of the best focused images. (d) Histogram of the peak
intensity sums. All peak intensities are normalized with respect to the clean
graphene area in an image to make the intensity sum independent from
contamination coverage.

Even though there is a certain focus tolerance it is still desirable that the
actual image positions match the target positions as good as possible to
enable accurate focus interpolation, and to avoid overlapping exposures.
Figure 5 shows two examples of maps, where the single frames were aligned on
the overview image. To make it easier to see both, the overview and the single
images on top, the different images were put into the blue and green channel
of an RGB-image, respectively. This allows also to see immediately if a
position was not found correctly by the alignment algorithm because the colors
would not mix to light-blue in this case but the small image would still
appear green. The image numbers are shown in red in this figure.

Figure 5: 561 automatically acquired images placed on an overview image to
find their actual positions. (a) Frame positions when feeding a regular grid
as stage coordinates into the mapping program. Inset: Deviation between actual
and ideal x-positions as gray-scale color code. Each square corresponds to one
image. The strong misplacement, especially between even and odd lines is
clearly visible. (b) Another map recorded in the same area as in (a) but with
corrected stage positions. This was done by analyzing frame positions in 10
maps acquired earlier and using the data to create the corrected target stage
positions here. Inset: The x-deviations clearly show the improvement in
comparison with (a). The intensity range was set to $\pm$50 nm in both insets
to make them directly comparable. (c and d) Zoom to the top-left corner of the
map in (a) and (b). The grid of images in (d) is much more regular than in
(c). Its root mean square displacement from the target positions could be
reduced from 36 to 18 nm.

The first look at figure 5a reveals that the mapped area is almost a perfect
rectangle which matches exactly the target area. This is due to the fact that
the sample stage moves are highly reproducible, meaning that after going up to
1 µm to the right and 1 µm back in the next line it arrives again at the left
margin of the rectangle with a precision of a few nm. A closer look, however,
shows that the frames are not distributed in a perfectly regular pattern
although these were the target positions. After the stage movement direction
changed, the first few frames in each line lie very close to each other and
sometimes even overlap because there is still some backlash present in the
stage mechanics. Towards the end of a line the distance between the images
increases. Also in y-direction there is a periodic deviation from the target
positions, although it is not as pronounced as in x-direction. Another
artifact can be seen in the very first line: The first image is located ca. 30
nm outside of the rectangle formed by all others and the spacing between the
first and second line is smaller than the distance between following lines.
Some of these artifacts are quantified in the inset in figure 5a, which shows
the x-deviation of the actual image positions from their target positions.
Especially the almost-perfectly aligned borders of the mapped area and the
strong deviation (backlash) between even and odd lines are visible there.

Overlapping areas are not desirable when using this automatic acquisition
procedure for low-dose imaging because beam-sensitive structures would have
already been exposed when the second snapshot is taken. Increasing the offset
between the images is also not a good solution because it would lead to a big
amount of unused sample area. We can significantly increase the precision of
the (x,y) positions by compensating the aforementioned systematic deviations
in a subsequent mapping. We have used the data of the deviation between the
desired and the actual frame positions from 10 earlier maps for a correction
that feeds “wrong” coordinates into the program such that the stage will
actually move to the correct position. This will not only suppress overlapping
images but also improve image quality, since the focus interpolation is
carried out assuming perfect frame positioning. In figure 5b the same map as
in 5a was acquired again, but with stage error correction enabled. There is
much less overlap in the single images and also the inset, which shows again
the x-deviations as a gray-scale image, reveals that the positions are much
more precise. Especially the deviations between even and odd lines could be
reduced significantly. Originally, deviations of up to 61 nm from the ideal
image positions were found, whereas in the corrected case this could be
reduced to a maximum of 40 nm. Similarly the root mean square displacement
could be reduced from 36 to 18 nm between the uncorrected and the corrected
case.

An example where we used the software we have developed to acquire an actual
low-dose map is shown in figure 6. Although the dose is too small to directly
see the graphene lattice, it can still be shown that the lattice information
is there by analyzing the image’s FTs. In all images of the map in figure 6b
at least the first 6 FT peaks are visible, in most of them even the first 12
which corresponds to a resolution of 2.13 Å and 1.23 Å, respectively. Hence,
the focus interpolation and stage movement can deliver atomically resolved
images also in the low-dose exposures.

Figure 6: Montage of a low-dose map consisting of 22x22 images. The total
mapped area is about 0.1 µm². (a) Overview image of a hole in the Quantifoil
support film covered with monolayer graphene. (b) Montage of low-dose images
acquired from the highlighted region in (a). (c) Single low-dose image. The
graphene lattice is not directly visible, but the typical reflections are
still present in the FT.

Finally, we used the automated low-dose acquisition together with the maximum-
likelihood reconstruction that we described in [Meyer et al., 2014] and
[Kramberger and Meyer, 2016]. The results presented here are the first
demonstration of this new algorithm with real experimental data (Ref. [Meyer
et al., 2014] used only calculations and [Kramberger and Meyer, 2016] used
experimental data that was artificially resampled to emulate low-dose
acquisition). Our test structures were defects in a graphene membrane, which
are not particularly radiation sensitive, but have the advantage that they are
well known from previous works [Meyer et al., 2008, Kotakoski et al., 2011,
Robertson et al., 2012, Kotakoski et al., 2014]. For this experiment we used
the microscope at an acceleration voltage of 100 kV, where graphene is no
longer stable and defects are created during imaging [Meyer et al., 2012].
Because the dose in single images is too low to generate defects, we scanned
up to 80 times at each stage position in order to ensure a sufficient defect
density. The dose per exposure in this case is $1.8\cdot 10^{4}e^{-}/A^{2}$
and the total accumulated dose per location up to $1.5\cdot
10^{6}e^{-}/A^{2}$. An example exposure is shown in figure 7a. The dose is too
low to discern the lattice and especially any defects, but the location and
orientation of the lattice can still be obtained from a FT of the image. Also,
the contaminated region of the graphene can still be recognized. In this data
set, the contaminated areas of the graphene sheet are now masked out, as
illustrated in figure 7b. For the reconstruction, the remaining data has to be
cut into small hexagonal cells. The size of one cell is indicated by a red
hexagon in figure 7b. From the 3403 raw images recorded for this experiment,
about 90,000 data cells could be extracted by dividing the entire clean region
of graphene into slightly overlapping cells.

Figure 7: Reconstruction of point defects in graphene from low-dose data. (a)
One raw data frame. Inset: FT of the image. Although the noise level in the
image is too high to directly see the graphene lattice, the presence of the 12
typical spots in the FT indicates that the imaging conditions are excellent.
(b) The same image as in (a) but with the contamination masked. The red
hexagon shows how the data is cut into data cells before feeding it into the
reconstruction algorithm. (c) The sum of ~90,000 data cells shows a perfect
and noise-free graphene lattice. (d-f) Result of the reconstruction. The
models are optimized in parallel and show frequently appearing features in the
raw data. The scale bar for (c-f) is 0.25 nm.

The starting point of the reconstruction is an empty graphene lattice as shown
in figure 7c, obtained by simply averaging all data cells. From here, the
maximum likelihood reconstruction seeks for a set of models that best
describes the actually observed data. We use the following formula for
calculating the likelihood of obtaining our snapshots under the assumption of
the model images:

$L=\prod_{f=1}^{F}\sum_{m=1}^{M}{\frac{w_{m}}{S}}\sum_{s=1}^{S}{P_{m,f,s}}$
(1) $P_{m,f,s}=\prod_{i=1}^{I}{P(k_{f,i},\lambda_{m,s(i)})}$ (2)

In this formula, $k_{f,i}$ denote the raw data values of every frame $f$ in
each pixel $i$. $\lambda_{m,s(i)}$ are the expectation values of every model
$m$ in each pixel $s(i)$ under a lattice symmetry operation $s$. The models
are weighted with $w_{m}$. Every $P_{m,f,s}$ is the probability to observe a
frame $f$ for a given model $m$ under a symmetry operation $s$. $P(k,\lambda)$
is the probability to observe $k$ counts for a corresponding expectation value
$\lambda$. The reconstruction algorithm works with any probability density
function $P(\lambda,k)$. It is, however, important to match the actual
distribution function of the raw data as well as possible. We therefore use
empirical histograms that can be collected from equivalent pixels for the
results presented here. The implementation of the reconstruction is described
in more detail in [Kramberger and Meyer, 2016]. The resulting images
effectively correspond to high dose views of all deviations from the periodic
lattice that occur sufficiently often in the data (the pristine hexagonal
lattice is also always among the set of reconstructed models, since it
accounts for the overwhelming majority of pristine graphene). From the present
dataset, two prevalent defect structures could be recovered, namely the single
585 divacancy (DV) and a pair of aligned 585 DVs. These two models as well as
the remaining empty lattice are shown in figure 7d-f.

It is interesting to consider why the 585 DV and also the double-585 DV were
observed, while the well known “reconstructed” versions of the di-vacancy in
graphene (the 555777 and 55556777 [Kotakoski et al., 2011]) are absent. These
defect structures all originate from two missing carbon atoms in the hexagonal
graphene lattice: Removing two adjacent atoms leads to an octagon, flanked by
two pentagons (585). By beam-induced bond rotations this defect can be
transformed into a circular arrangement of three pentagons and three heptagons
(555777). Further bond rotations lead to a hexagon surrounded by four
pentagons and four heptagons (555567777). More details on the creation,
migration and transformation of defects in graphene can be found in [Kotakoski
et al., 2011] and [Robertson et al., 2012]. Hence, we can speculate that in
our low-dose experiment, the irradiation dose was high enough to form DVs in
large numbers (possibly by a clustering of mono-vacancies which are highly
mobile under the beam [Banhart et al., 2011]), but not high enough to convert
a significant number into their reconstructed shapes.

## Conclusions

In summary we developed a method to automatically acquire a large number (few
hundred to a few thousand) atomically resolved images with the Nion UltraSTEM.
We find that the stage and lenses are stable enough to obtain atomically
resolved images from an area up to 1 µm2 in a time span of up to a few hours
without intermediate re-tuning by the user. The low-dose data acquisition
makes it possible to obtain atomically resolved images from sample areas that
are not exposed to the electron beam before the actual image acquisition. We
expect that this will enable the study of beam-sensitive structures such as
individual organic molecules deposited on graphene, defects in low-dimensional
materials that are too mobile under the beam for a conventional analysis, or
other ultra-thin films with embedded objects. It may further be useful for
studies that simply require larger amounts of images than can be conveniently
obtained manually, e.g. a search for rarely occurring defects or a meaningful
statistical analysis of structural details. To our knowledge, this is the
first time that a data acquisition scheme that is commonplace in the
biological single-particle analysis was applied to a non-biological material
and was achieved with sufficient precision that individual atoms can be seen
in almost all exposures. It may provide a bridge to use methods from
structural biology in the study of radiation-sensitive materials, or vice
versa.

## Acknowledgements

A.M., C.K., C.H., C.M., and J.C.M. acknowledge funding from the European
Research Council (ERC) Project No. 336453- PICOMAT.

## References

  * [Banhart et al., 2011] Banhart, F., Kotakoski, J., and Krasheninnikov, A. V. (2011).  Structural Defects in Graphene.  ACS Nano, 5(1):26–41.
  * [Bartesaghi et al., 2015] Bartesaghi, A., Merk, A., Banerjee, S., Matthies, D., Wu, X., Milne, J. L. S., and Subramaniam, S. (2015).  2.2 Åresolution cryo-EM structure of $\beta$-galactosidase in complex with a cell-permeant inhibitor.  Science (New York, N.Y.), 348(6239):1147–1151.
  * [Cheng, 2015] Cheng, Y. (2015).  Single-particle Cryo-EM at crystallographic resolution.  Cell, 161(3):450–457.
  * [Cheng et al., 2015] Cheng, Y., Grigorieff, N., Penczek, P., and Walz, T. (2015).  A Primer to Single-Particle Cryo-Electron Microscopy.  Cell, 161(3):438–449.
  * [Dierksen et al., 1992] Dierksen, K., Typke, D., Hegerl, R., Koster, A., and Baumeister, W. (1992).  Towards automatic electron tomography.  Ultramicroscopy, 40(1):71–87.
  * [Egerton, 2012] Egerton, R. F. (2012).  Mechanisms of radiation damage in beam-sensitive specimens, for TEM accelerating voltages between 10 and 300 kV.  Microscopy research and technique, 75(11):1550–6.
  * [Frank et al., 1978] Frank, J., Goldfarb, W., Eisenberg, D., and Baker, T. S. (1978).  Reconstruction of glutamine synthetase using computer averaging.  Ultramicroscopy, 3(C):283–290.
  * [Fultz and Howe, 2012] Fultz, B. and Howe, J. M. (2012).  Transmission electron microscopy and diffractometry of materials.  Springer Science & Business Media.
  * [Koster et al., 1992] Koster, A., Chen, H., Sedat, J., and Agard, D. (1992).  Automated microscopy for electron tomography.  Ultramicroscopy, 46(1-4):207–227.
  * [Kotakoski et al., 2011] Kotakoski, J., Krasheninnikov, A. V., Kaiser, U., and Meyer, J. C. (2011).  From Point Defects in Graphene to Two-Dimensional Amorphous Carbon.  Physical Review Letters, 106(10):105505.
  * [Kotakoski et al., 2014] Kotakoski, J., Mangler, C., and Meyer, J. C. (2014).  Imaging atomic-level random walk of a point defect in graphene.  Nature communications, 5:3991.
  * [Kramberger and Meyer, 2016] Kramberger, C. and Meyer, J. C. (2016).  Progress in structure recovery from low dose exposures: Mixed molecular adsorption, exploitation of symmetry and reconstruction from the minimum signal level.  Ultramicroscopy, 170:60–68.
  * [Mastronarde, 2005] Mastronarde, D. N. (2005).  Automated electron microscope tomography using robust prediction of specimen movements.  Journal of structural biology, 152(1):36–51.
  * [Meyer et al., 2012] Meyer, J. C., Eder, F., Kurasch, S., Skakalova, V., Kotakoski, J., Park, H. J., Roth, S., Chuvilin, A., Eyhusen, S., Benner, G., Krasheninnikov, A. V., and Kaiser, U. (2012).  Accurate Measurement of Electron Beam Induced Displacement Cross Sections for Single-Layer Graphene.  Physical Review Letters, 108(19):196102.
  * [Meyer et al., 2008] Meyer, J. C., Kisielowski, C., Erni, R., Rossell, M. D., Crommie, M. F., and Zettl, A. (2008).  Direct Imaging of Lattice Atoms and Topological Defects in Graphene Membranes.  Nano Letters, 8(11):3582–3586.
  * [Meyer et al., 2014] Meyer, J. C., Kotakoski, J., and Mangler, C. (2014).  Atomic structure from large-area, low-dose exposures of materials: a new route to circumvent radiation damage.  Ultramicroscopy, 145:13–21.
  * [Robertson et al., 2012] Robertson, A. W., Allen, C. S., Wu, Y. a., He, K., Olivier, J., Neethling, J., Kirkland, A. I., and Warner, J. H. (2012).  Spatial control of defect creation in graphene at the nanoscale.  Nature Communications, 3:1144.
  * [Shi et al., 2008] Shi, J., Williams, D. R., and Stewart, P. L. (2008).  A Script-Assisted Microscopy (SAM) package to improve data acquisition rates on FEI Tecnai electron microscopes equipped with Gatan CCD cameras.  Journal of structural biology, 164(1):166–9.
  * [Suloway et al., 2005] Suloway, C., Pulokas, J., Fellmann, D., Cheng, A., Guerra, F., Quispe, J., Stagg, S., Potter, C. S., and Carragher, B. (2005).  Automated molecular microscopy: the new Leginon system.  Journal of structural biology, 151(1):41–60.
  * [Zhang et al., 2001] Zhang, P., Beatty, A., Milne, J. L., and Subramaniam, S. (2001).  Automated data collection with a Tecnai 12 electron microscope: applications for molecular imaging by cryomicroscopy.  Journal of structural biology, 135(3):251–61.
  * [Zhang et al., 2003] Zhang, P., Borgnia, M. J., Mooney, P., Shi, D., Pan, M., O’Herron, P., Mao, A., Brogan, D., Milne, J. L., and Subramaniam, S. (2003).  Automated image acquisition and processing using a new generation of 4Kx4K CCD cameras for cryo electron microscopic studies of macromolecular assemblies.  Journal of Structural Biology, 143(2):135–144.
  * [Zhang et al., 2010] Zhang, X., Jin, L., Fang, Q., Hui, W. H., and Zhou, Z. H. (2010).  3.3 ÅCryo-EM Structure of a Nonenveloped Virus Reveals a Priming Mechanism for Cell Entry.  Cell, 141(3):472–482.


###### Abstract

Adaptive mobile device-based health interventions often use machine learning
models trained on non-mobile device data, such as social media text, due to
the difficulty and high expense of collecting large text message (SMS) data.
Therefore, understanding the differences and generalization of models between
these platforms is crucial for proper deployment. We examined the psycho-
linguistic differences between Facebook and text messages, and their impact on
out-of-domain model performance, using a sample of 120 users who shared both.
We found that users use Facebook for sharing experiences (e.g., leisure) and
SMS for task-oriented and conversational purposes (e.g., plan confirmations),
reflecting the differences in the affordances. To examine the downstream
effects of these differences, we used pre-trained Facebook-based language
models to estimate age, gender, depression, life satisfaction, and stress on
both Facebook and SMS. We found no significant differences in correlations
between the estimates and self-reports across 6 of 8 models. These results
suggest using pre-trained Facebook language models to achieve better accuracy
with just-in-time interventions.

## Introduction

Language reflects users’ psychology and can be used to understand and predict
mental health conditions (i.e., de2013predicting). While language from social
media such as Facebook has been widely used (eichstaedt2018facebook;
jaidka2018facebook; liu2022linguistic), text messaging (Short Message Service
or SMS) is emerging as a new platform for detecting mental health conditions
and delivering interventions (e.g., depression: liu2021relationship,
loneliness: liu2022head). This also opens possibilities for Just-in-Time
Adaptive Interventions (JITAIs) to deliver physical and mental health support
based on an individual’s changing state and environment (nahum2018just).

Most JITAIs are designed for smartphones, but current NLP models are primarily
trained on social media, not SMS. Acquiring large-scale SMS for fine-tuning
models is difficult and expensive. Transferring pre-trained Facebook models to
SMS is thus common (liu2021relationship). But the impact of cross-platform
language differences is unclear. Our evaluation aims to scientifically
quantify the differences and ensure the successful transfer of current NLP
models trained on social media sites into potential JITAIs using SMS.

Figure 1: For the same users, we: (1) compared differences between Facebook
and SMS language, and (2) evaluated the in- vs. out-of-domain efficacy of
language models in predicting users’ self-reported psychological traits.

Our research aims to explore: RQ1 the distinctions in language between
Facebook and SMS, and RQ2 to evaluate the efficacy of language models derived
from Facebook data in predicting psychological traits when applied to SMS. To
achieve this, we utilize _the same cohort of users_ who have provided their
Facebook language, SMS, and psychometric self-reports (e.g., demographics,
depression; Figure 1).

### Contributions

Our contributions are: (1) showing clear Facebook vs. SMS distinctions in
language use for the same users; (2) evaluating the two platforms by
training/validating within and across domains; and (3) laying the foundation
for NLP model transfer to SMS with within-user comparisons.

## Background

Language use varies across contexts. How can the language use of the same
person differ in SMS and Facebook? Although, to our knowledge, no study has
compared these two, some research has compared Facebook status updates with
direct messages (DM), a private message on Facebook resembles SMS.
doi:10.1177/0261927X12456384 observed that sharing positive emotions is
associated with self-presentational concerns in Facebook status updates but
not DM, noting the difference between communication on public and private
channels. 10.1111/jcom.12106 also identified various goals and motivations for
self-disclosures in Facebook status updates and DMs. Status updates relate to
more social validation, self-expression, and relief, while DMs relate to
relationship development and social maintenance.

If individuals use Facebook and SMS for different functions, it is still
unclear how well models trained on Facebook posts will perform when applied to
SMS data. Preliminary work showed that linguistic model predictions change
across platforms. For example, seabrook2018predicting examined the association
between depression and emotional word expressions on Facebook and Twitter and
found different patterns. On Facebook, the instability of negative emotion
words only predicts depression, whereas on Twitter, the variability of
negative emotion words reflects the severity of depression. However, as most
cross-platform comparisons are made across user groups; individual differences
(e.g., demographic) may cause these variations more than language choices. As
cross-platform generalizations are expected to lead to model performance
degradation, only a few studies have conducted same-user cross-platform
comparisons (jaidka2018facebook; guntuku2019understanding), no previous
research has quantified these differences across Facebook and SMS within the
same users. Our paper aim to draw more attention to these differences and
provide possibilities for actionable improvements to conduct more precise
predictions for future JITAIs plans.

## Data

### Participants

Participants were recruited online via Qualtrics as part of a larger national
survey (tao2023covid). Each consenting participant (1) lived in the U.S., (2)
was over 18 years old, (3) shared Facebook status updates, (4) installed the
open-source mobile sensing application AWARE (ferreira2015aware) on their
Android phones, (5) wrote at least 500 words across platforms (Facebook and
SMS apps), and (6) completed a survey which contains questions on age,
gender111We only analyze binary male/female gender,a limited and problematic
sense of gender, due to limited data and the limitations of our gender
estimation model. Three participants reporting a non-binary gender were
excluded from the gender analysis., depression, life satisfaction, and stress.
Our final sample included 120 participants ($M_{age}$ = 36.46, 69%
female).222See Supplement for full details on participant recruitment,
demographics, survey-based measures, and text-based estimates, at
https://github.com/TTRUCurtis/Facebook-vs-SMS-language. Table 1 shows usage
differences between Facebook and SMS.333Extensive cleaning was automatically
applied (i.e., no human in the loop) to the keystroke data to remove any
sensitive PII data. See Supplement for details. To fairly compare the Facebook
data to the keystroke data, we applied the same cleaning pipeline to both.

| Words | Posts
---|---|---
| Med. | Mean | SD | Med. | Mean | SD
FB | 12,800 | 26,652 | 37,924 | 1,279 | 2,193 | 2,599
SMS | 3,607 | 7,881 | 11,693 | 331 | 711 | 961
Table 1: Posts and word count statistics per platform (Med. = median and SD =
standard deviation).

### Survey-Based Measures

For each participant, we collected self-reported age, gender, depression,
stress, and life satisfaction via surveys used as gold-standard measures. We
measured depression via the Patient Health Questionnaire (PHQ-9;
kroenke2001phq), life satisfaction via Cantril’s Ladder (cantril1965pattern),
and stress via Cohen’s Perceived Stress Scale (cohen1983global).supplement

### Text-Based Estimates

We employed off-the-shelf text-based models to estimate age, gender
(sap2014developing), depression (schwartz2017dlatk), stress
(guntuku2019understanding), and life satisfaction (jaidka2020estimating). All
models were developed in previous studies and trained on Facebook status
updates to predict survey-based self-reports via lexical features (i.e., bag-
of-words or bag-of-topics models).supplement

For this study, we also trained RoBERTa-based models (liu2019roberta) on the
data sets used in the original papers. These models were trained for
depression, life satisfaction, and stress only, as we cannot access the
original data used to train the age and gender models. Since this paper is not
aimed to build state-of-the-art classifiers, we used the same model pipeline
across depression, life satisfaction, and stress: (1) we extracted user-level
RoBERTa embeddings using the penultimate layer, (2) reduced the dimensions of
the resulting 768 dimension embedding (using non-negative matrix
factorization) to 128 dimensions (v-ganesan-etal-2021-empirical), and (3)
applied a $\ell_{2}$ regularized Ridge regression with $\alpha=1$ (chosen via
nested cross-validation). The RoBERTa-based models had similar accuracy to the
lexical models.supplement

## Methods

### RQ1: Cross-platform Differences

We first tokenized the Facebook status updates and SMS data, using a tokenizer
designed for social media data (schwartz2017dlatk). We considered both 1-to-3
grams and the Linguistic Inquiry and Word Count (LIWC) 2022 dictionary
(boyd2022development). LIWC has been widely used in psychological sciences
(e.g., eichstaedt2018facebook) and LIWC 2022 consists of 102 manually curated
categories by psychologists. From both Facebook and SMS data, we extracted
1-to-3 grams and created a binary outcome variable for each participant to
indicate which platform they were on. We then calculated effect size using
Cohen’s _d_ values between platforms and conducted a logistic regression using
n-grams to predict the binary platform indicator in order to calculate
statistical significance (_p_ values). Next, we extracted all LIWC 22
categories from each user’s Facebook and SMS data. To calculate differences,
we computed paired sample $t$-tests for each LIWC category between Facebook
and SMS. All significance thresholds were adjusted using a Benjamini-Hochberg
False Discovery Rate (FDR) correction (benjamini1995controlling).

Facebook
---
Category | Top frequent words | _t_
Leisure | _fun, weekend, play_ | 14.65
Determiners | _the, a, my, this_ | 9.63
Quantities | _all, day, some, more_ | 7.91
Power | _own, order, power, president_ | 7.77
Emotion | _love, good, happy, :), fun_ | 7.35
SMS
Category | Top frequent words | _t_
Auxiliary verbs | _is, have, be, was_ | -20.26
Communication | _thank, say, thanks, said, tell_ | -17.92
Discrepancy | _can, want, would_ | -14.90
Assent | _yes, ok, yeah, okay_ | -14.82
2nd person | _you, your, you’re, u_ | -13.27

Table 2: Paired _t_ -tests results of LIWC 2022 categories, showing top
categories which differ between Facebook and SMS. All results are
statistically significant at $p<0.001$ after Benjamini-Hochberg FDR
correction.

### RQ2: In vs. Out of Domain Estimates

Here we performed three tasks to answer this from two different approaches:
First, _Task 1_ applied off-the-shelf models to both the Facebook and SMS data
to evaluate in-and across-domain estimates and their generalization, and _Task
2_ examined which linguistic features were driving the differences in
estimates in _Task 1_. _Task 3_ opted not to use off-the-shelf models in _Task
1_ and _2_. Instead, it involved training and assessing predictive models
within and across each domain.

_Task 1_ : For each participant, we estimated age, gender, depression, life
satisfaction, and stress from Facebook and SMS text using the text-based
models described above. We then correlated the estimates with the gold-
standard survey-based measures for both the lexical and embedding-based
models. A statistical bootstrap test was used to assess differences in
correlations between SMS-based estimates and Facebook-based estimates.

_Task 2_ : To identify features driving lexical-based model estimates in both
domains, we investigated feature importance $i$, which is defined as:

$i(f)=w_{f}\big{(}freq_{FB}(f)-freq_{SMS}(f)\big{)}.$ (1)

Here $w_{f}$ is the weight of the feature $f$ in the depression model,
$freq_{*}(f)$ is the frequency of feature $f$ in either the Facebook (FB) or
SMS domain.

_Task 3_ : Finally, instead of using off-the-shelf models, we trained and
evaluated predictive models within and across each data set. To do this, we
trained models to predict our five outcomes (age, gender, depression, life
satisfaction, and stress) using both text sources from the same person as
training and testing data sets: (1) train on FB / test on FB, (2) train on FB
/ test on SMS, (3) train on SMS / test on SMS, and (4) train on SMS / test on
FB. We used a leave-one-out cross-validation setup when training and testing
within the same text domain (1 and 3). When testing across text domains (2 and
4), we trained a model using one text source, applied the model to the other
text source (producing estimates of our 5 outcomes), and correlated those
estimates with self-reports.

## Results

### RQ1: Cross-platform Differences

As seen in Table 2 and Figure 2, people preferred to discuss leisure
activities, share pleasant feelings (LIWC positive emotion) and contents
(e.g., books, songs), and express their motivations (LIWC power) on Facebook.
People used more conversational language (LIWC communication), more second-
person pronouns, and were more task-oriented (e.g., actions, LIWC verbs, plan
confirmations) in SMS. See Supplementsupplement for additional n-gram
correlations that provided more context.

Figure 2: 1-to-3 grams most correlated with Facebook vs. SMS, statistically
significant at $p<0.05$ after Benjamini-Hochberg FDR correction. Cohen’s $d$ =
effect size measuring Facebook vs. SMS differences. Angle brackets: spaCy
annotated named entities (e.g., $<$work of art$>$).

| Lexical Models | Embedding Models
---|---|---
| Facebook | SMS | Facebook | SMS
Age | .68 | .45∗ | - | -
Gender† | .91 | .80∗ | - | -
Depression | .36 | .29 | .25 | .08
Life Satis. | .21 | .14 | .31 | .31
Stress | .21 | .18 | .21 | .23

Table 3: Pearson correlations (or † accuracy) between language estimates and
self-reports. ∗ Significant difference in bootstrapping test between SMS and
Facebook correlations.

### RQ2: In vs. Out-of-Domain Estimates

In Table 3, we found that in-domain estimates from Facebook data predicted
self-reports at rates similar to those in the original papers supplement from
which the models were built (_Task 1_). When predicting self-reports from SMS-
based estimates (i.e., out-of-domain), we observed a drop in prediction
accuracy across all lexical models and 1 out of 3 embedding models. However,
the differences between the Facebook correlation with self-report and the SMS
correlation with self-report were not statistically different (using a
bootstrapping test) except for those for age and gender (where SMS does not
perform as well as Facebook). In Figure 3, we further investigated the drop in
performance by examining feature importance (_Task 2_). Here we identified
features reflecting language style, such as more use of contractions (“i’ll”,
“i’m”, “they’re”, “she’s”, “haven’t”), driving the SMS depression estimates,
and features about content, experience, and life events (“family”, “sick”,
“anniversary”) driving the Facebook depression estimates. In Table 4, we
presented the results from training and evaluating models within and across
domains. Facebook-trained models have higher in-domain accuracy, and SMS-
trained models have higher out-of-domain accuracy. Again, using a
bootstrapping significance test, we did not see significant differences
between the correlation of Facebook and self-reports versus SMS and self-
reports (in both in- and cross-domain tasks).

Figure 3: Feature importance results, as defined by the product of the
depression model word weight and the difference in Facebook vs SMS word usage
frequency. Top row (red bars; A and B) are positively weighted words in the
depression model, while the bottom row (blue bars; C and D) are negatively
weighted words. Left column (A and C) is more frequency words in Facebook
(i.e., positive frequency difference), while the right column (B and D)
contains words more frequent on SMS (i.e., negative frequency difference).

| In Domain | Cross Domain (train/test)
---|---|---
| FB | SMS | FB/SMS | SMS/FB
Age | .61 | .52 | .40 | .50
Gender† | .75 | .74 | .63 | .73
Depression | .25 | .09 | .15 | .32
Life Satis. | .19 | .07 | .25 | .29
Stress | .24 | .12 | .32 | .38

Table 4: Within and across platform evaluation. Pearson correlations (or †
accuracy) between language estimates and self-reports. In Domain models are
evaluated using leave-one-out cross validation.

## Conclusion

Our study, based on data from the same users, shows: (1) individuals disclose
different aspects of their lived experiences on Facebook and SMS, (2) two
platforms generate similar mental health estimates, both within and across
domains, whether using off-the-shelf models trained on Facebook data (Table 3)
or models built specifically on the paper’s dataset (Table 4). Consistent with
past findings, Facebook usage reflects the need to belong and self-
presentation (nadkarni2012people), leading to more content sharing and opinion
expression; whereas SMS is used for phatic communication to maintain social
relationships and for informal discussions (fibaek2016s), leading to more
confirmations and conversational features. Our data, derived from the same
users, indicates that cross-platform differences can be attributed to language
rather than demographics. Despite the linguistic differences, our findings
suggest that predictions from both platforms are similar.

## Broader Impact

Our findings have important implications. Firstly, our research highlights the
variations in psycho-linguistic features between Facebook and SMS, thus
warranting further investigation of downstream applications. Secondly, future
researchers can build predictive models on large-scale social media language
and apply them to SMS, which may offer a new approach to address the cost-
accuracy trade-off in the context of just-in-time interventions on mobile
devices.

This study involves human subjects and was approved by the Institutional
Review Board (IRB). The data used in this study raise ethical concerns such as
handling sensitive personal information (PII) and thus, we have taken measures
to securely store, clean, and analyze the data, further data sharing is not
possibleclean. We use social media, SMS data, and machine learning methods to
estimate sensitive attributes like depression. Such estimates can have both
positive and negative implications, ranging from providing support to causing
discrimination. We must use them with caution.

## Acknowledgments

This study was supported by the Intramural Research Program of the NIH,
National Institute on Drug Abuse (ZIA-DA000632). The authors report no
conflict of interest.