[1]John S.Schreck [2]MatthewHayman [1]GabrielleGantos [3]AaronBansemer
[1]David JohnGagne

1]National Center for Atmospheric Research (NCAR), Computational and
Information Systems Laboratory, Boulder, CO, USA 2]National Center for
Atmospheric Research (NCAR), Earth Observing Laboratory, Boulder, CO, USA
3]National Center for Atmospheric Research (NCAR), Mesoscale and Microscale
Meteorology Laboratory, Boulder, CO, USA

<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Mimicking non-ideal instrument behavior for hologram processing using neural
style translation

###### Abstract

Holographic cloud probes provide unprecedented information on cloud particle
density, size and position. Each laser shot captures particles within a large
volume, where images can be computationally refocused to determine particle
size and shape. However, processing these holograms, either with standard
methods or with machine learning (ML) models, requires considerable
computational resources, time and occasional human intervention. Current ML
models are trained on simulated holograms obtained from the physical model of
the probe since real holograms have no absolute truth labels. Any attempt to
use another processing method to produce labels would be subject to errors
that the ML model would subsequently inherit. Trained models perform well on
real holograms only when image corruption is performed on the simulated images
during training, thereby mimicking non-ideal conditions in the actual probe
(Schreck et. al, 2022). Optimizing image corruption requires a cumbersome
manual labeling effort.

Here we demonstrate the feasibility of applying the neural style translation
approach (Gatys et. al, 2016) to the simulated holograms. With a standard,
pre-trained convolutional neural network (VGG-19), the simulated holograms are
“stylized” to resemble the real ones obtained from the probe, while at the
same time preserving the simulated image “content” (e.g. the particle
locations and sizes). Several image similarity metrics concur that the
stylized images are more like real holograms than the synthetic ones. With an
ML model trained to predict particle locations and shapes on the stylized data
sets, we observed comparable performance on both simulated and real holograms,
obviating the need to perform manual labeling. The approach described here is
not specific to hologram images and could be applied in other domains as a
means for capturing noise and imperfections in observational instruments to
make simulated data more like real world observations.

Holographic imaging enables 3D reconstruction of particle positions and sizes
within a large sample volume. Through this technique, ensembles of thousands
of particles can be captured and characterized in a single instantaneous
volume. This technique has been successfully applied to characterizing the
microphysical properties of clouds with airborne in situ instruments such as
the HOLODEC probe (Fugal et al., 2004; Spuler and Fugal, 2011) which is
operated by NCAR and has been successfully deployed on several field
campaigns.

The HOLODEC instrument captures inline holograms of a 15 $cm^{3}$ sample
volume. A laser operating at 355 nm transmits an expanded beam between two
probe arms. After the light passes through the sample volume, the light is
imaged onto a CCD with an effective pixel resolution of 3 $\mu m$. The laser
fires at a 3 Hz repetition rate, allowing for point-like captures of particle
size distributions as the aircraft passes through a cloud with spatial
separations of those points dictated by aircraft speed.

While holographic imaging captures a significant amount of information in a
single image, processing these holograms tends to be computationally
intensive, often requiring human intervention to tune the processing approach.
As a result, there has been substantial research into the use of machine
learning to accelerate or improve processing of holographic particle data
(Shimobaba et al., 2019; Shao et al., 2020; Zhang et al., 2022). However, many
of these efforts are focused on images with relatively small depth components
which is distinct from HOLODEC requirements. Another key distinction is in the
idyllic nature of the holograms that are processed. Processing solutions that
work well on controlled laboratory or simulated holograms tend to perform
suboptimally on operationally captured data from HOLODEC. One cannot expect
the same consistent clarity and quality from an instrument on the outside of
an aircraft, where a variety of operating and environmental conditions can
cause the captured images to contain a significant amount of imperfections.
The challenge of processing HOLODEC data necessitates tuning filters to
minimize both false positives and missing true particles in holograms with
non-negligible imperfections.

These imperfections are difficult to model in simulation because they often
depend on combinations of component imperfections within the instrument. For
example, small amounts of reflections from transmission optics can create
modulation patterns and focus spots in reconstruction, laser modes deviate in
amplitude and phase from a true Gaussian, vignetting can cause non-uniform
field response and depend on beam pointing that varies with vibrations and
temperature. Ultimately, developing processing techniques that can account for
these and other (sometimes unknown) effects is essential to delivering high
quality scientific observations.

In a previous work, we outlined a processing pipeline concept to accelerate
and improve processing of HOLODEC images through a combination of hardware
acceleration and machine learning (Schreck et al., 2022). As is common in
atmospheric sensing, actual hologram data lacked truth labels. In order to
achieve an improvement in processing capability over the existing solution, we
developed an approach for corrupting simulated holograms in order to emulate
the imperfections of actual HOLODEC data. A machine learning solution to
process actual HOLODEC data was trained using simulated holograms (where
labels are are known to near absolute accuracy) by adding random noise,
applying blurring and skewing, and adjusting the image contrast. While the
simulated training data did not visually appear similar to actual HOLODEC
data, these image corruptions were sufficient to train a model that still out
performed the particle recognition approaches used in the standard processing
solution (Schreck et al., 2022). However, to succeed, each of these transforms
needed to be tuned in hyperparameter optimization, which required manual
labeling of thousands of reconstructed holograms. This is problematic because
manual labeling requires considerable person-hours from well trained
personnel, and (as we discovered in the process) human determined labels have
some non-zero but unquantified error. What is more, during the optimization,
we simply guessed which types of corruptions to perform on the images.
Finally, there may be many sensor situations where these simple corruption
operations do not suffice in developing a higher performance processing
solution. In order to generalize to the broader domain of sensor signal
processing, a solution should be able to address the specialized behavior of
the sensor type and this mandates a more general ability to represent the
difficult to model, non-ideal behaviors of the instrument.

Here we introduce a new approach to bridging the gap between optimizing signal
processing/machine learning solutions and the realities of operational sensor
data. We show that the original neural style transfer method introduced by
(Gatys et al., 2016), which utilizes a convolutional neural network (CNN) to
perform image augmentation, can be used to produce realistic-looking, labeled
hologram examples without the need for manually labeling as a means for tuning
noise parameters. These stylized synethetic training data result in processing
performance similar to the previous method, but without the requirement of
human labeling and therefore represent a significant step toward producing an
operation solution for processing HOLODEC data.

##  1 Methodology

###  1.1 Neural Style Transfer

Figure 1: (a) NCAR’s Mesa Lab taken to be the content image (Commons, 2007).
(b) An image obtained from the James Webb telescope showing star formation is
the style image (NASA, 2022). (c) The synthesized image is the product of the
neural style transfer method that leverages the content and style images.

The original style transfer method leverages CNN representations of a
“content” image and a “style” image to produce a third image where the style
features are automatically synthesized with content features. For example,
Figure 1 illustrates how the method is used to create an image of NCAR’s Mesa
Lab shown in (a), stylized using a recent image from the James Webb telescope
shown in (b) to create a synthesized image shown in (c). The synthesized image
still clearly still shows the Mesa Lab, however, overall the image colors,
contrast, and line smoothnesses have been transformed relative to the original
to also resemble the style image.

The synthetic holograms come with labels and are free of noise, so they are
identified as possessing the “content” we wish to preserve — that is the
information about the particles within. The HOLODEC examples all contain
varying degrees of noise/imperfections that we wish to effectively impart onto
the synthetic examples, hence they are identified as the “style” examples. In
this way, synthetic images are transformed to create a realistic training
dataset with known labels.

Figure 2: Schematic illustration of the neural style transfer approach by
Gatys et al. (2016) using the VGG-19 neural model, where the model’s 18
convolution layers are stacked into five blocks (orange squares). In this
example, the output from the first convolution layer in each of the five
blocks were used to compute style and content losses. The blocks represent the
(same) model for inputs $p$, $x$, and $a$.

Figure 2 schematically illustrates the style transfer method using the VGG-19
network as the chosen CNN (Simonyan and Zisserman, 2015). The VGG model
contains 19 layers and was originally trained to perform object recognition
and localisation on the ImageNet dataset (Russakovsky et al., 2014). The
model’s 18 convolution layers utilize the ReLU activation and are sequentially
connected together into five blocks separated by 2D average Pooling layers, as
is illustrated schematically in Figure 2. The first two blocks contain two
convolution layers, while the remaining three contain four. The position of a
convolution layer in VGG is characterized by integer $l$. All convolution
layers in VGG-19 use a kernel size of $3x3$ and stride and padding sizes both
set to unity. Three fully-connected layers complete VGG-19. Thus, each column
in Figure 2 is a single (the same) VGG-19 model (with the fully-connected
layers discarded). Currently, a single color channel is used for the hologram
images, so the very first convolution layer was replaced with one having the
color channel equal to one.

The style transfer approach begins by sampling random values for $x$, and
freezing the weights of the all of the layers in the CNN. For the input
content image $p$ and generated image $x$, the output from convolution layer
$l$ are the feature matrix representations $P^{l}$ and $F^{l}$, where
$P_{ij}^{l}$ and $F_{ij}^{l}$ are the responses to the activation of the $ith$
filter at position $j$ in convolution layer at position $l$. A central
question with style-transfer is the choice of loss function, and which layer
outputs should be used to compute the loss. The choice will determine to what
extent we need the full VGG-19 model, and will greatly affect run-time
performance. We follow the original approach introduced by Gatys et al.
(2016), but other effective loss functions have been developed (Li and Wand,
2016b; Li et al., 2017; Wilmot et al., 2017). A new image that matches the
responses of the content image $p$ may be found by updating the values of $x$
via gradient descent using a squared-loss of the two feature representations

$\mathcal{L}_{content}(p,x,l)=\frac{1}{2}\sum_{i,j}(F_{ij}^{l}-P_{ij}^{l})^{2}$
(1)

for convolution layer at position $l$, as is illustrated in Figure 2 by a blue
hexagon. The “style” representation of an input image $x$ is defined using a
feature space for capturing texture information. This is taken to be the
correlation between the outputs across the spatial domain for layer $l$, so
that the expected correlation for style feature maps $i$ and $j$ are given by
the Gram matrix

$G_{ij}^{l}=\sum_{k}G_{ik}^{l}G_{jk}^{l}.$ (2)

Now the image $x$ may be updated via gradient descent so that it’s style
features $G^{l}$ match the style features $A^{l}$ of image a at layer $l$. The
contribution coming from layer $l$ is taken to be

$E_{l}=\frac{1}{4N_{l}^{2}M_{l}^{2}}\sum_{i,j}(G_{ij}^{l}-A_{ij}^{l})^{2}$ (3)

and the total style loss, illustrated by green circles in Figure 2, is

$\mathcal{L}_{style}(a,x)=\sum_{l}w_{l}E_{l}$ (4)

where $w_{l}$ represents a weight for each contribution coming from layer $l$.
Although not used in Gatys et al. (2016), the predicted images may sometimes
contain higher-frequency noise which we would prefer to smooth out. This can
be accomplished via a total variation term given by

$\mathcal{L}_{TV}(x)=\sum_{i,j}\left|x_{i,j}-x_{i+1,j}\right|+\left|x_{i,j}-x_{i,j+1}\right|$
(5)

for pixels $i$ and $j$. For content image $p$, style image $a$, and generated
image $x$, the total style transfer loss is then

$\mathcal{L}_{total}(p,a,x)=\alpha\mathcal{L}_{content}(p,x)+\beta\mathcal{L}_{style}(a,x)+\gamma\mathcal{L}_{TV}(x)$
(6)

where the parameters $\alpha$, $\beta$, and $\gamma$ are constant weights, and
only the output from a single layer is used to compute the content loss (see
Figure 2). In general, if more than one layer output is selected for computing
the content loss, the terms would be summed in Equation 6. We set $\alpha$ =
$\gamma=1$ and focus on the relationship between $\beta$ and the predicted
images below.

In addition to $\beta$, the choice of which layer(s) to use in content and
style representations are left as hyperparameters. For the content feature
layer choice, the output from convolution layers farther from the input image
should capture global features relative to those blocks nearer to the input
image, which should capture more details of the input. Similar arguments apply
to the style features, with the more distant convolution layers capturing more
global styles and vice versa. Gatys et al. (2016) selected the output from the
second convolution layer from the fourth block to compute the content loss
(e.g. the 12th convolution layer in VGG-19), and the output from the first
convolution layer in all five blocks to compute the style loss components
(convolution layers at positions 1, 3, 5, 9 and 13). These choices result in a
large CNN that greatly affects training performance when used with our
relatively large holograms. We found that a much smaller model was just as
capable as that used by Gatys et al. (2016) and was thus much faster to use.
Specifically, we selected the fourth convolution layer outputs in VGG-19 for
the content contributions and the first five convolution layer outputs in
VGG-19 for the style contributions (with all weights equal to unity in Eq. 4),
therefore only requiring layers in three of the five blocks in VGG-19. There
may be a more optimal, faster model that achieves similar results, but we did
not explore other combinations any further.

Figure 2 shows that content image $p$ and style image $x$ are passed through
the CNN and the outputs from selected feature layers are stored in each case.
Initially white noise image $x$ is then passed through the CNN and the outputs
from the content and feature layers are used to compute the total loss
according to Equation 6. The image pixels in $x$ can then be updated via back-
propagation and with successive iterations, $x$ can learn to match the $p$’s
content and $a$’s style features.

###  1.2 Metrics for comparing images

The choice of $\alpha$ and $\beta$ will determine how similar the generated
image is to the content and style images. There are numerous metrics available
for estimating the similarity between two images. For images $x_{0}$ and
$x_{1}$, we select the structural similarity index measure (SSIM) (Wang et
al., 2004) (Equation 7), a standard pixel-based metric that extracts
information on structure, contrast, and luminance of an image that serves as
the basis for comparison of two images. Additionally, a similarity metric
based on the AlexNet convolutional neural network model (Krizhevsky et al.,
2017), referred to as $d_{\text{Alex}}\equiv d_{\text{Alex}}(x_{0},x_{1})$, is
used to capture the distance between two images, $x_{0}$ and $x_{1}$, in deep
feature space (Zhang et al., 2018). The SSIM ranges from -1 to 1, with larger
values indicating more similarity between images. The AlexNet metric ranges
from 0 to 1 with smaller value indicating greater similarity; we use one minus
the value so that a higher value indicates greater similarity in both metrics.
For images $x_{0}$ and $x_{1}$, both metrics are computed by averaging over
local differences between “patches” of size $11\times 11$ for SSIM, and size
$3\times 3$ (the filter size used) for the AlexNet metric.

###  1.3 Image segmentation with a U-Net

Figure 3: (a) U-Net segmentation model. (i) Example synthetic holograms (solid
red lines) are passed into the encoder-head of the U-Net shown in (ii). (iii)
During training, noise may optionally be added to the holograms. (iv)
Predicted masks for the input examples, with a dark circle enclosing the in-
focus particle. (b) (i) Synthetic images are (ii) stylized using HOLODEC
holograms (dashed blue lines), which are then (iii) used to train a U-Net
model. (c) In operation, HOLODEC holograms are passed through the U-Net
trained on style-transferred images to obtain a mask prediction. In the
example no particles were identified.

Figure 3 shows a neural segmentation model, for example a “U-Net” (Ronneberger
et al., 2015; Shelhamer et al., 2017), that is used in the current study to
process holograms and has been used in other studies (Shao et al., 2020). The
model contains two CNN “heads” as illustrated in the figure. An encoder head
takes in the input image and compresses it into a smaller dimensional
representation of a fixed size. The decoder head up-samples the latent
representation of the image and outputs a prediction the same size as the
input image. Here, we use a sigmoid activation function on the decoder’s last
layer that rescales the predicted outputs for all the pixels to lie within (0,
1). In other words, the model is trained to predict a probability that an in-
focus particle occupies a pixel. If the probability exceeds a certain value,
often 0.5 for binary tasks, the pixel is labeled 1 and 0 otherwise. The
convolutional layers in the U-Net encoder head may also leverage pre-trained
weights from the layers in other convolutional models, such as ResNet (He et
al., 2016) trained on the ImageNet dataset, which frequently helps to speed
training and also boost overall performance.

###  1.4 Datasets

There are essentially four datasets we use in this work. The first two
originate from full size images captured by HOLODEC or simulations of HOLODEC.
These initial images consist of 4872-by-3248 pixel images which are processed
and segmented to create images that can be readily processed by the U-Net.
These images are also sampled to ensure a balance between in-focus particles
and out-of-focus particles in the U-Net training process. This training data
from the two sources are then used by style transfer to generate a realistic
generated dataset (with content from the synthetic images and style from the
HOLODEC images) which is used to train the U-Net. Here, style transfer is
performed using reconstructed synthetic and HOLODEC images at different
positions along $z$ (see below). Finally a set of manually labeled HOLODEC
holograms are used to evaluate the performance of the U-Net, and therefore the
effectiveness of the style transfer solution.

####  1.4.1 HOLODEC and synthetic data

A physical model of the instrument was used to generate 120 synthetic
holograms which had optical settings identical to the HOLODEC instrument
(Fugal et al., 2009). One hundred holograms were randomly selected for
training, ten for validation, and the remaining ten for testing. Each hologram
contained 500 spherically shaped particles randomly positioned along the $x$
and $y$ directions, positioned between minimum and maximum values of 14.072
and 158.928 millimeters along the z-direction, respectively. The diameter
sizes were sampled with a gamma distribution.

The HOLODEC data were selected from a set of several hundred holograms
obtained from the RF07 subset from the Cloud Systems Evolution in the Trades
(CSET) project that was originally obtained from June 1 to Aug 15, 2015
(Albrecht et al., 2019). They primarily contain liquid water particles of
varying numbers and shape, were largely free of ice but contain other
unidentified “artifacts” and image perturbations not captured by the physical
model of the instrument. Overall, the known particle numbers range from none
up to hundreds, with the average particle diameter less than 50 $\textmu m$.

In accordance with the approach described in (Schreck et al., 2022), each
hologram is initially processed by reconstructing 1,000 planes along the
z-axis (axis of laser propagation). This produces 1,000, 12 megapixel size
images that must scanned for in-focus particles. These images are too large to
be processed efficiently with a CNN. Therefore, following our previous
approach (Schreck et al., 2022), each hologram reconstruction is broken up
into 38-by-28 tiles each of 512 by 512 pixels. These tiles have an overlap
stride of 128 pixels leaving 828 total tiles for each full image. This means
that any one pixel will show up in more than 1 tile.

For training using synthetic holograms, each was selected and then refocused
along z for all of the 500 particles, from which a random grid tile containing
the in-focus particle was selected and saved, resulting in 50,000 images (for
details on refocusing, see Schreck et al. (2022)). Additionally, 25,000 tiles
were selected where a particle was just out of focus by no more than 120
micrometers. Finally, 25,000 tiles were randomly selected which did not
contain in-focus particles. A comparable procedure was repeated to produce
validation and test sets of tile images where half the images contained in-
focus particles.

For the RF07 subset used here, a hologram was drawn at random from several
hundred, excluding 20 that were set aside previously for manual examination
(and described below). This raw data can be accessed at
https://doi.org/10.5281/zenodo.6347222. As the true particle positions are not
known (and they are not needed as they will be used only for style transfer),
the hologram was refocused at a random position between 14 and 160 millimeters
along $z$, from which 10 tiles were randomly selected and saved from the
refocused hologram. This process was repeated until about 100,000 image tiles
were selected for training, 10,000 for validation, and 10,000 for testing the
mask prediction performance. Most of the tiles in each split did not have
particles directly in-focus, but occasionally were pretty close.

####  1.4.2 Generated data sets with style transfer

The labeled synthetic training splits are used as the content images while
RF07 holograms, which come without labels, are the style images. Training,
validation, and testing splits for 20 different values of $\beta$ sampled
evenly from approximately $10^{-5}$ to $10^{15}$, with $\alpha=1$ and
$\gamma=1$ were created to find the optimal value of $\beta$. Once obtained, a
data set containing training, validation, and testing splits was created with
$\beta\equiv\beta_{0}=10^{9}$, which we identify below as an optimal choice,
and $\alpha$ and $\gamma$ both equal to one. This is created to compare
against the model trained on synthetic training split only and the model
described in Schreck et al. (2022) where optimized noise (based on hand
labeled data) was added during training.

For each selection of $\beta$ including $\beta_{0}$, a randomly selected
content (synthetic) image tile was selected and paired off with a randomly
selected style (HOLODEC) image tile. Then, both content and style images were
used to generate a training sample with the style-transfer method, with the
mask labels for the generated example are inherited from the content image.
For the $\beta_{0}$ data set, 100,000, 20,000, and 20,000 style-generated
images were created for training, validation, and testing the ability of U-Net
model’s to reproduce the mask labels. For the other 20 values of $\beta$, a
total of 20,000 training, 2,000 validation, and 2,000 testing samples were
produced in each case for the same purpose. In the latter data sets, we also
used the same random seed so that all the pairs selected in each split for the
different $\beta$ choices (excluding $\beta_{0}$) were the same.

####  1.4.3 Manually labeled HOLODEC data set

The 20 manually labeled HOLODEC examples resulted in 2,356 total images of
size 512 pixels by 512 pixels, where prospective in-focus particles were
positioned at the image center. The examples were manually labeled as
containing an in-focus particle or not, along with an average reviewer
confidence score ranging from zero to five (higher means more confident). The
binary output of the U-Net is simply taken to be unity if any predicted value
is larger than 0.5 and 0 otherwise. Many of the examples were subjective, and
the total of number of particles we could identify was limited by previous
modeling attempts, hence there could be more particles in these holograms
still undiscovered.

The total number of labeled examples was split into validation and testing
data sets. The validation set contained 1,204 total images with 367 containing
at least one in-focus particle. The testing set contained 1,154 images, where
874 contained at least one in-focus particle. The validation set was used to
help guide the optimization of noise in our previous study that we compare
with here (Schreck et al., 2022), while the test set represents the hold-out
set of manually labeled examples.

##  2 Results

###  2.1 Style transfer with holograms

Figure 4: In (a-c), the same synthetic hologram is paired with a different
HOLODEC hologram to produce a generated hologram using style transfer. The
HOLODEC examples illustrate (a) an in-focus particle, (b) interference
patterns and successive dark spots, and (c) small artefacts. U-Nets are
observed to perform poorly on (b) and (c) when trained with synthetic images
only. The content weight $\alpha$ = 1, while the style weight $\beta$ =
$\beta_{0}$ = $10^{9}$. Figure 5: (a) The style, content, TV, and total loss
are plotted versus the number of training epochs (the number of times the
image was updated). (b) The initial random image and selected images at
different training updates are shown. The pixel values shown ranged from 50 to
90, as in Figure 4 for the HOLODEC and generated images.

Figure 4 shows a synthetic hologram (content image) that contains a single in-
focus particle, and different HOLODEC hologram examples (style images) that
were used to generate the style transferred image, for $\alpha$ = 1, and
$\beta$ = $10^{9}$. The HOLODEC example shown in Figure 4(a) was selected
because it contains an in-focus particle. Note that while there appears to be
a transfer of fine scale texture from the style images, particles from these
style images are not transferred generated images. Thus there is no need to
locate empty holograms for the style transfer images or perform careful QC in
selecting the appropriate images for this process.

The images in Figure 4(b-c) are typical examples where a U-Net trained only on
synthetic images (left column) performs poorly on actual HOLODEC images due to
a high false-positive rate. The interference fringes seen in the HOLODEC image
in Figure 4(b) often produce patterns that resemble in-focus particles, while
(c) shows an example where many small artefacts are present and have a round
shape. In all three examples the in-focus particle in the synthetic example is
still present in the generated examples, and the overall generated image
contrast is much more comparable to the HOLODEC images.

Figure 5(a) shows the training curve for the generated image shown in Figure
4(a). Figure 5(b) illustrates the progression of the predicted image at
selected epochs. Starting initially with a random noise image for $x$, a very
steep drop in the style loss is observed over the first few epochs. The
content and TV losses also drop quickly after the first few epochs, then the
curves mostly flatten after about 10 epochs, and the content contribution
comes to dominate the total loss thereafter. In these examples, the choice of
$\alpha/\beta$ determined the similarity between the generated image and
content and style images used to create it. Here, according to the AlexNet
metric, all the generated images are more similar to the HOLODEC images than
to the synthetic image.

Figure 6: (a) The average SSIM and (b) average $d_{\text{Alex}}$ scores
computed using the test data sets, is shown versus the style weight $\beta$ in
Equation 6 with $\alpha$ and $\gamma$ = 1.

Figure 6(a) and (b) show the SSIM and $d_{\text{Alex}}$ metrics versus $\beta$
with $\alpha$ = 1. These metrics are computed for test data of HOLODEC-
synthetic, HOLODEC-generated, and synthetic-generated pairs. Note that these
metrics are constant for the HOLODEC-synthetic pairs because these images are
unaffected by $\beta$.

In both figures, four regions of interest are approximately identified with
vertical lines. Region A corresponds with values of approximately $\beta\leq
10$ where the content loss dominates the total loss. Accordingly, both metrics
show that the style-transferred images strongly resemble the original
synthetic images (see also Figure 4(a,b)). The main difference between the two
metrics is that $d_{\text{Alex}}$ scores HOLODEC-generated pairs higher
relative to HOLODEC-synthetic pairs than does SSIM. However, the curves are
all qualitatively similar between the two metrics for all values of $\beta$.

Examples of images from each region in Figure 6 are shown in Figure 7 using
the content and style images from Figure 4(a) with $\alpha=1$ and varying
$\beta$.

Region B is identified approximately when $10^{2}$ < $\beta\leq 10^{7}$, which
is when both metrics for synthetic-generated and HOLODEC-generated pairs drop
significantly. The examples in Figure 4(c,d) show that the generated image
starts to look significantly different compared with the original synthetic
image as the style term begins to contribute. Although these examples may give
the impression that the generated images still look like the synthetic images,
they are in reality a mixed representation which does not on average resemble
the synthetic or HOLODEC hologram (for example, the average contrast
difference for examples like those in Figure 4(c,d) is less than that for the
synthetic images but still greater than what is observed in the HOLODEC
examples). Around $\beta=10^{4}$ both metrics reach a minimum before beginning
to rise with increasing $\beta$.

Region C is identified approximately when $10^{7}<\beta\leq 10^{12}$. Both
SSIM and $d_{\text{Alex}}$ continue to increase for synthetic-generated pairs
and peak near $\beta=10^{9}$, though the peak is more pronounced for SSIM.
Both scores for HOLODEC-generated pairs also increase and reach plateau
values, which are higher compared against that in regions A and B, indicating
the images have been stylized to some degree while the the Synthetic-Generated
scores indicate the image still retains content features. Clearly, the neural
$d_{\text{Alex}}$ determines the generated holograms are more similar to
HOLODEC holograms compared to SSIM, but overall both metrics indicate the
predicted images have struck a particular balance between retained content and
learned style features. Figure 4(e-g) show that the in-focus particle remains
in place and roughly the same shape, while, for example, the image contrast is
adjusted to be more like that of the HOLODEC images.

Finally, region D is identified by $\beta>10^{12}$, which is when the style
term dominates the loss. The $d_{\text{Alex}}$ score for style-generated pairs
remains high while that for content-generated pairs converges to the same
value for content-style pairs, which is to say the metric cannot differentiate
the generated image from style images. However, Figure 4(h,i) show that the
in-focus particle and other content from the synthetic image has largely been
removed. Furthermore, what appear to be fake particles are now being
introduced into the generated images. In other words, the original image was
over-stylized.

Figure 7: (a-i) Examples of generated holograms using the content and style
images from Figure 4(a) and different style weights. The region is listed in
the bottom right.

Region A represents the instances where output has very little style imposed
on the content images and therefore most closely resembles idealized synthetic
data, which we previously established in Schreck et al. (2022) makes a poor
U-Net training data set when the objective is to process actual HOLODEC data.
Images from region D represent instances where content is largely absent from
output. Clearly we cannot expect to train a U-Net to recognize particles that
are not in the input images. Without having to resort to manual labeled
HOLODEC images, the obvious choice would be to select the value of $\beta$
around $\beta_{0}=10^{9}$ in region C, when both metrics for synthetic-
generated pairs peak, and the generated examples appear to be more similar to
HOLODEC examples according to $d_{\text{Alex}}$.

The performance of the U-Net trained on the generated images is then expected
to peak at mask reproduction when $\beta$ is around $\beta_{0}=10^{9}$, as the
predicted images are the most similar to the content images in region C. Also,
because the generated images are similar to the HOLODEC images in region C
relative to A and B, we might expect U-Net models trained on them to perform
the best on HOLODEC images. If this is the case, one can select the value of
$\beta$ by finding the value where the generated images are more similar to
the HOLODEC images according to $d_{Alex}$ and SSIM. This selection should
result in the best generated dataset for training the U-Net.

###  2.2 Performance on synthetic and HOLODEC images

Figure 8: (a) Several binary metrics for the U-Net’s mask prediction
performance are plotted versus style weight $\beta$ for synthetic or style-
generated images. (b) The same binary metrics are shown for the particle
detection task in manually labeled HOLODEC images versus $\beta$. As the
models did not rely on the manual examples during training, all 2,356 examples
were used in the figure. The x-axis labels "Syn" and "0" refer to U-Nets
trained on synthetic images and generated images with $\beta=0$, respectively.

In order to test the hypothesis that the optimal $\beta$ can be found
analytically from $d_{Alex}$ and SSIM, the generated images (and their
corresponding labels) at the different values of $\beta$ are used to train
U-Net models, as is shown schematically in Figure 3(b). The model architecture
and training hyperparameters were all identical for each different value of
$\beta$ (see Appendix .1 for more details). Post-training, each U-Net model is
run in operation mode on the test set of generated images and the manually
labeled test set of actual HOLODEC tiles. Figure 8(a) shows several binary
performance metrics for the U-Net at reproducing the mask labels accompanying
the generated images, while Figure 8(b) shows similar quantities for particle
detection in true HOLODEC tiles. Since none of the manually labeled tile
examples were used in the creation of the style-generated images, all 2,356
were used in the figure. The metrics are the the F1 score, the area-under-the-
ROC-curve (AUC), the probability of detection (POD), and the false alarm rate
(FAR).

Figure 8(a) shows overall gradually diminishing training performance with
increased $\beta$. This is expected as the pristine simulations are
increasingly stylized with non-ideal instrument noise and artifacts, making
particle detection more difficult. In region A, when the generated holograms
resemble the synthetic examples, the mask prediction performance is very high
and generally flat. But then in region B, when the generated holograms are
modified more by the style image, there is a clear drop in the AUC and POD
while the FAR rises. As $\beta$ continues to grow and the generated images
start to resemble both content (synthetic) and style (HOLODEC) examples, the
performance rises and reaches a peak around $\beta$ = $\beta_{0}$ before
dropping, first slowly and then very suddenly once in region D. We expect this
peak in region C to be less than that in A where the images lack the
characteristic noise of the instrument. In region D, the AUC is about 0.5
which should be expected as the over-stylized images and the mask labels
retain very little content from the actual particles. The trained U-Net
outputs all, or too many 1s when trained on over-stylized images, so POD
quickly approaches 1. We also computed diameters for the predicted masks and
found no significant deviation from the mean true value for the test set, for
all choices of $\beta$ except those in region D.

Overall, Figure 8(b) shows that the relevant range of $\beta$ may be selected
according to our hypothesis. Relative to the U-Net trained only on synthetic
images (labeled ‘Syn’ on the x-axis in Figure 8(b)), the performance on the
HOLODEC examples in region A is signified by lower PODs and higher FARs,
although the AUC values are all approximately around 0.6. Even the model
trained on generated images with $\beta=0$ showed a higher FAR relative to
’Syn’, indicating that the U-Net trained on idealized synthetic holograms
tends to under predict particles, to the benefit of a low FAR. We should note
here that the difference in performance between the synthetic trained and
$\beta=0$ cases indicates that setting $\beta=0$ does not perfectly reproduce
the content image, resulting in slightly lower U-Net performance. However,
neither case is relevant to the objective of this work since both cases fail
to perform well on HOLODEC data. In region B, the particle detection
performance generally rises even as the mask performance drops. However, the
particle detection performance zig-zags quite significantly making a selection
of $\beta$ unreliable without manual labeled examples. In Region C, the
detection performance reaches an approximate plateau, with high AUC values and
consistently low FAR values. The performance seems to slightly decrease as the
style weight approaches values in region D. In region D, the model is unable
to train effectively, resulting in an output of all 1s. This results in a high
POD and FAR with AUC at 0.5. Region C models also performed the best overall
on the HOLODEC images relative to A, B and D. As expected, models in Region D
do not perform at all on the manual examples, as signified by AUCs close to
0.5.

###  2.3 Comparison of optimized models

Metric | F1 | AUC | POD | FAR | CSI
---|---|---|---|---|---
| Syn/Gen | HOLO | Syn/Gen | HOLO | Syn/Gen | HOLO | Syn/Gen | HOLO | Syn/Gen | HOLO
Synthetic | 0.977 | 0.220 | 0.992 | 0.448 | 0.979 | 0.106 | 0.025 | 0.387 | 0.956 | 0.100
Optimized noise | 0.968 | 0.866 | 0.985 | 0.808 | 0.967 | 0.922 | 0.031 | 0.096 | 0.937 | 0.840
Stylized | 0.966 | 0.864 | 0.988 | 0.805 | 0.968 | 0.922 | 0.035 | 0.097 | 0.935 | 0.839
Table 1: Several metrics are listed for each hyperparameter optimized model
and were computed on the synthetic or generated (Syn/Gen), and HOLODEC (HOLO)
test data sets, for mask and particle detection (binary) predictions,
respectively. The validation set of labeled HOLODEC images was used to
optimize the noise parameters in the Optimized noise model.

Now that the style weight may be selected without relying on manual examples
for optimization, we selected $\beta=\beta_{0}=10^{9}$ and trained a model on
the style-transferred images. Table 1 compares this model (‘Stylized’) versus
a model trained on idealized synthetic images only (‘Synthetic’), and a model
trained on synthetic images with noise introduced during training (‘Optimized
noise’, where the optimization required manual labels). In all three cases we
performed extensive hyperparameter optimization to find optimal architectures
and training parameters using the mask performance on the validation set of
generated images as the optimization metric. The model which leveraged noise
during training additionally used the manually labeled validation HOLODEC
examples to guide optimization of the added noise. The same binary metrics in
Figure 8 are listed as well as the critical success index (CSI) for mask
performance on test images (either synthetic or generated) and detection
performance on test HOLODEC images. As the validation set of manually labeled
examples were used to select the noise added which resulted in the Optimized
noise, only the testing split is used in Table 1.

Table 1 shows that the Stylized model and the Optimized noise model gave
nearly the same performance on all metrics on both data sets even though the
Stylized approach does not require manual labeling. The Optimized noise model
outperforms Stylized by hundredths of a percent in each metric category. The
Synthetic model showed better mask performance on synthetic data. However, the
Synthetic demonstrated much worse performance at particle detection on real
HOLODEC data compared to the other two models. For example Table 1 shows that
it had a much higher FAR, and the CSI was more than 70% lower compared to the
other models.

Figure 9: (a-c) Histograms of computed particle diameters for the three models
for the test set of manually labeled HOLODEC images. The dashed vertical line
in each panel denotes the predicted average particular diameter $\langle
d\rangle$.

Lastly, as the Optimized noise and Stylized models produced nearly identical
performance, the diameters for the predicted particles in the test set of
manually labeled HOLODEC examples are computed to probe how the models
describe the particles shapes. Figure 9 shows histograms of the predicted
diameters for the three models. A particle diameter was computed for a
predicted mask if at least one in-focus particle was present. From the
predicted mask, any pixel labeled 1 was grouped with a neighboring pixel along
$x$ or $y$ (but not along the diagonal in the plane) if it was also labeled 1.
Therefore, breaks between groups defines multiple particles. The diameter of a
group was taken to be it’s maximum extent in either x or y.

Figure 9(a)(i) shows the Synthetic model expectantly under-predicting the true
particles compared with the other two models, and nearly as many false
positive predictions were made relative to true positive, as seen in Figure
9(a)(ii). Meanwhile, the Optimized noise and Stylized models predicted similar
distributions, but which differed from the Synthetic distribution. Both models
predicted two peaks centered approximately around 15 and 22 µm, respectively,
with the Stylized model having the more pronounced second peak. The Stylized
model also predicts slightly greater numbers of larger particles relative to
Optimized noise that are also sometimes observed by the Synthetic model. As a
result, the overall average diameter is predicted to be about 17.8 µm versus
15.9 µm for Stylized and Optimized noise models, respectively. Figure 9(b)(ii)
and (c)(ii) further show that both models mostly predict false diameters at 15
µm or smaller.

##  3 Discussion

Overall, the performances observed on both synthetic and HOLODEC images with
the Optimized noise and the Style model were comparable, with both models
performing well against models trained on the synthetic holograms only. This
shows the advantage of using the style-transfer method for hologram image
augmentation, which obviated the need to perform manual labeling as a means
for optimizing noise added during the training of U-Net models. Additionally,
we did not have to ambiguously choose the types of augmentations performed on
the images, as was required for the Optimized model. Furthermore, the
application of style transfer to images that were used for training, rather
than applied during operation, means that computational performance is also
comparable with the Optimized noise model.

The main computational bottleneck involving the style-transfer method occurs
during the creation of style-transferred training data sets. Many generated
images needed to be created, one at a time, from a unique content-style image
pair, that all required the iterative optimization described above. The image
sizes used were also quite large. That further slowed training times and made
it cumbersome to create large data sets used for exploring the different
weight combinations. Improved approaches could be focused on replacing the
optimization with a trained neural network that predicts the stylized image.
This can lead to several orders of magnitude speed-up over the original method
when feed-forward architectures are utilized (Li and Wand, 2016a; Ulyanov et
al., 2016; Huang and Belongie, 2017). These may include the StyleGAN method
(Karras et al., 2021), which integrates the concepts of style transfer with a
generative adversarial network (Goodfellow et al., 2020) and allows for fine-
tune control over the relative strength of image features at different scales.
For holograms, the StyleGAN could potentially be used to more selectively
control specific content or style features on demand (for example, adding more
features pertaining to artefacts).

We also only worked with CSET holograms, in particular the RF07 data split,
which had limited particle densities as well as sizes. How well a trained
model, which leveraged style images from one field campaign, works on other
campaigns remains to be tested in future work. Style-generated hologram data
sets, which do not resemble operational inputs, plausibly would not be
feasible as training targets, as we learned in Schreck et al. (2022). In such
a scenario, the holograms from the latest field campaign could be used as
style examples to generate the relevant training data set. The StyleGAN
approach again could potentially take advantage of all the different types of
holograms obtained from field campaigns (and all of the unique objects
contained in each one), since it may be capable of leveraging multiple styles
and content features arising in the data sets.

Finally, as the primary objective of this study was to remove the manual-
labeling step without sacrificing performance, we did not try to assess the
physical realness of the generated holograms since during operation only real
HOLODEC holograms are used. However, the question remains as to whether they
are physically reasonable. One way to test this assumption could be to perform
style-transfer using full-size holograms, which would be computationally
challenging but not impossible with current GPUs, and then refocus them via
wave-propagation along the focal plane. This would require faithfully
reconstructing the real and imaginary components of the electric field, rather
than just the intensity, as was done here since style-transfer was applied to
tiles selected from the full-sized holograms after they had been propagated.

##  4 Conclusions

In summary, we have shown that the style-transfer algorithm is an effective
approach for translating synthetic holograms, which were created using an
idealized physical model of the instrument, into holograms that resemble those
actually observed by the instrument. In principle, the application of style-
transfer to perform the image augmentations should not be limited to holograms
since non-ideal instrument behavior is a problem across many domains. When the
synthetic holograms are used in machine learning models to predict masks
around in-focus particles as was done here, noise had to be injected onto the
images during training so that the model performed well on both synthetic and
raw data. However, the choice of noise transformations and parameters can only
be selected after expensive manual labeling of raw images and hyperparameter
optimization. The style-transfer approach, which transformed the clean
hologram data set into one more resembling observed holograms, delivered the
same mask prediction performance without the cumbersome requirement of manual
labeling. Furthermore, both models had comparable performance on a small set
of manually labeled HOLODEC examples, and predicted similar distributions for
the particle diameters.

###  .1 Model and training hyperparameters

Parameter | Synthetic | Optimized noise | Stylized ($\beta_{0}$) |  $10^{-5}$ < $\beta$ $\leq$ $10^{15}$
---|---|---|---|---
Learning rate | $3.86\text{\times}{10}^{-4}\text{\,}$ | $2.46\text{\times}{10}^{-4}\text{\,}$ | $8.5\text{\times}{10}^{-5}\text{\,}$ | $1\text{\times}{10}^{-3}\text{\,}$
Training loss | Focal-Tyversky | Focal-Tyversky | IoU | Focal-Tyversky
Segmentation model | U-Net | LinkNet | U-Net | U-Net
Encoder model | EfficientNet-b0 | Xception | DenseNet121 | VGG11
Tile transform | 255 | Normalized | Standard | Symmetric
L2 regularization | $0.0\text{\,}$ | $0.0\text{\,}$ | $2.1\text{\times}{10}^{-7}\text{\,}$ | $6.0\text{\times}{10}^{-6}\text{\,}$
Gaussian blur $\sigma$ | - | 2.125 | - | -
Gaussian noise | - | 0.326 | - | -
Brightness factor | - | 1.270 | - | -
Table 2: The values of the best hyperparameters in the optimization studies
for the neural segmentation models for the three species. The batch size was
fixed at 16.

Table 2 lists the best parameters found for the segmentation models
investigated. See reference Schreck et al. (2022) for more details on the
hyperparameter optimization of segmentation models with hologram data sets.
The segmentation models found to be optimal in Table 2 were the U-Net
(Ronneberger et al., 2015) and LinkNet (Chaurasia and Culurciello, 2017),
while the pre-trained encoder model weights considered were DenseNet-121
(Huang et al., 2017), Xception (Chollet, 2017), EfficientNet-b0 (Tan and Le,
2019), and VGG-11 (Simonyan and Zisserman, 2014). See the package
segmentation-models-pytorch located at
https://github.com/qubvel/segmentation_models.pytorch for more details on the
segmentation and encoder models. The optimal training losses found were the
intersection over union (IOU) and Focal-Tyversky losses (Lin et al., 2017;
Salehi et al., 2017). For additional definitions of each loss function, see
the Holodec-ML software package located at https://github.com/NCAR/holodec-ml.
We also utilized pre-trained weights obtained from the ImageNet data set in
all trained models. The tile transforms were applied to a tile just before
being passed into a segmentation model include dividing all pixels by 255, re-
scaling all pixels to lie between 0 and 1 (Normalized) or -1 and 1
(Symmetric), or re-scaling each tile by subtracting the pixel mean and
dividing by the square of the variance (Standard).

###  .2 Structural similarity index metric (SSIM)

The global structure similarity definition for comparing images $x$ and $y$ is
given by

$SSIM(x,y)=\frac{(2\mu_{x}\mu_{y}+C_{1})(2\sigma_{xy}+C_{2})}{(\mu_{x}^{2}+\mu_{y}^{2}+C_{1})(\sigma_{x}^{2}+\sigma_{y}^{2}+C_{2})}\\\
$ (7)

where $C_{1}$ and $C_{2}$ are fixed constants, and $\mu$ and $\sigma$ are
defined as

$\mu_{x}=\frac{1}{N}\sum_{i=1}^{N}x_{i}\\\
\sigma_{x}^{2}=\frac{1}{N-1}\sum_{i=1}^{N}(x_{i}-\mu_{x})^{2}\\\
\sigma_{xy}=\frac{1}{N-1}\sum_{i=1}^{N}(x_{i}-\mu_{x})(y_{i}-\mu_{y}).$ (8)

In practice, we use the mean structural similarity index which averages over
an 11 by 11 circular-symmetric Gaussian Weighting function (Wang et al.,
2004).

###  .3 Data sets

The HOLODEC and synthetic data sets can be accessed at
https://doi.org/10.5281/zenodo.6347222. The labeled HOLODEC examples were a
subset of the RF07 data set (validation set ID 0-9, testing set ID 10-19),
while the synthetic holograms were generated with simulations. All holograms
in the RF07 data set were used for creating the style-transferred data sets.
See Section 1.4 for more details. The style generated data sets can be built
using the Holodec-ML software package located at
https://github.com/NCAR/holodec-ml.

###### Acknowledgements.

This material is based upon work supported by the National Center for
Atmospheric Research, which is a major facility sponsored by the National
Science Foundation under Cooperative Agreement No. 1852977. We would like to
acknowledge high-performance computing support from Cheyenne and Casper
Computational and Information Systems Laboratory, CISL (2020) provided by
NCAR’s Computational and Information Systems Laboratory, sponsored by the
National Science Foundation. The neural networks described here and simulation
code used to train and test the models are archived at
https://github.com/NCAR/holodec-ml. All HOLODEC and synthetic hologram data
sets created for this study are available at
https://doi.org/10.5281/zenodo.6347222

## References

  * Albrecht et al. (2019) Albrecht, B., Ghate, V., Mohrmann, J., Wood, R., Zuidema, P., Bretherton, C., Schwartz, C., Eloranta, E., Glienke, S., Donaher, S., et al.: Cloud System Evolution in the Trades (CSET): Following the evolution of boundary layer cloud systems with the NSF–NCAR GV, Bulletin of the American Meteorological Society, 100, 93–121, 2019.
  * Chaurasia and Culurciello (2017) Chaurasia, A. and Culurciello, E.: Linknet: Exploiting encoder representations for efficient semantic segmentation, in: 2017 IEEE Visual Communications and Image Processing (VCIP), pp. 1–4, IEEE, 2017.
  * Chollet (2017) Chollet, F.: Xception: Deep learning with depthwise separable convolutions, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251–1258, 2017.
  * Commons (2007) Commons, W.: National Center for Atmospheric Research, Boulder Colorado, URL https://upload.wikimedia.org/wikipedia/commons/3/31/National_Center_for_Atmospheric_Research_-_Boulder%2C_Colorado.jpg, 2007\.
  * Computational and Information Systems Laboratory, CISL (2020) Computational and Information Systems Laboratory, CISL: Cheyenne: HPE/SGI ICE XA System (NCAR Community Computing), Tech. rep., National Center for Atmospheric Research, URL https://doi.org/10.5065/D6RX99HX, 2020\.
  * Fugal et al. (2004) Fugal, J. P., Shaw, R. A., Saw, E. W., and Sergeyev, A. V.: Airborne digital holographic system for cloud particle measurements, Applied optics, 43, 5987–5995, 2004.
  * Fugal et al. (2009) Fugal, J. P., Schulz, T. J., and Shaw, R. A.: Practical methods for automated reconstruction and characterization of particles in digital in-line holograms, Measurement Science and Technology, 20, 075 501, 2009.
  * Gatys et al. (2016) Gatys, L. A., Ecker, A. S., and Bethge, M.: Image Style Transfer Using Convolutional Neural Networks, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2414–2423, IEEE Computer Society, 10.1109/CVPR.2016.265, 2016.
  * Goodfellow et al. (2020) Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. C., and Bengio, Y.: Generative adversarial networks, Commun. ACM, 63, 139–144, 10.1145/3422622, 2020.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.: Deep Residual Learning for Image Recognition, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 10.1109/CVPR.2016.90, 2016.
  * Huang et al. (2017) Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q.: Densely connected convolutional networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708, 2017.
  * Huang and Belongie (2017) Huang, X. and Belongie, S. J.: Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization, in: IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 1510–1519, IEEE Computer Society, 10.1109/ICCV.2017.167, 2017.
  * Karras et al. (2021) Karras, T., Laine, S., and Aila, T.: A Style-Based Generator Architecture for Generative Adversarial Networks, IEEE Trans. Pattern Anal. Mach. Intell., 43, 4217–4228, 10.1109/TPAMI.2020.2970919, 2021.
  * Krizhevsky et al. (2017) Krizhevsky, A., Sutskever, I., and Hinton, G. E.: ImageNet classification with deep convolutional neural networks, Commun. ACM, 60, 84–90, 10.1145/3065386, 2017.
  * Li and Wand (2016a) Li, C. and Wand, M.: Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks, in: Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III, edited by Leibe, B., Matas, J., Sebe, N., and Welling, M., vol. 9907 of Lecture Notes in Computer Science, pp. 702–716, Springer, 10.1007/978-3-319-46487-9_43, 2016a.
  * Li and Wand (2016b) Li, C. and Wand, M.: Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2479–2486, IEEE Computer Society, 10.1109/CVPR.2016.272, 2016b.
  * Li et al. (2017) Li, Y., Wang, N., Liu, J., and Hou, X.: Demystifying Neural Style Transfer, in: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, edited by Sierra, C., pp. 2230–2236, ijcai.org, 10.24963/ijcai.2017/310, 2017\.
  * Lin et al. (2017) Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P.: Focal loss for dense object detection, in: Proceedings of the IEEE international conference on computer vision, pp. 2980–2988, 2017.
  * NASA (2022) NASA: Star Forming Region Carina, URL https://www.nasa.gov/sites/default/files/styles/full_width/public/thumbnails/image/main_image_star-forming_region_carina_nircam_final-1280.jpg, 2022\.
  * Ronneberger et al. (2015) Ronneberger, O., Fischer, P., and Brox, T.: U-net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical image computing and computer-assisted intervention, pp. 234–241, Springer, 2015.
  * Russakovsky et al. (2014) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M. S., Berg, A. C., and Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge, CoRR, abs/1409.0575, URL http://arxiv.org/abs/1409.0575, 2014.
  * Salehi et al. (2017) Salehi, S. S. M., Erdogmus, D., and Gholipour, A.: Tversky loss function for image segmentation using 3D fully convolutional deep networks, in: International workshop on machine learning in medical imaging, pp. 379–387, Springer, 2017.
  * Schreck et al. (2022) Schreck, J. S., Gantos, G., Hayman, M., Bansemer, A., and Gagne, D. J.: Neural network processing of holographic images, Atmospheric Measurement Techniques, 15, 5793–5819, 2022.
  * Shao et al. (2020) Shao, S., Mallery, K., Kumar, S. S., and Hong, J.: Machine learning holography for 3D particle field imaging, Optics Express, 28, 2987–2999, 10.1364/OE.379480, 2020.
  * Shelhamer et al. (2017) Shelhamer, E., Long, J., and Darrell, T.: Fully Convolutional Networks for Semantic Segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence, 39, 640–651, 10.1109/TPAMI.2016.2572683, 2017.
  * Shimobaba et al. (2019) Shimobaba, T., Takahashi, T., Yamamoto, Y., Endo, Y., Shiraki, A., Nishitsuji, T., Hoshikawa, N., Kakue, T., and Ito, T.: Digital holographic particle volume reconstruction using a deep neural network, Appl. Opt., 58, 1900–1906, 10.1364/AO.58.001900, 2019.
  * Simonyan and Zisserman (2014) Simonyan, K. and Zisserman, A.: Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556, 2014.
  * Simonyan and Zisserman (2015) Simonyan, K. and Zisserman, A.: Very Deep Convolutional Networks for Large-Scale Image Recognition, in: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, edited by Bengio, Y. and LeCun, Y., URL http://arxiv.org/abs/1409.1556, 2015.
  * Spuler and Fugal (2011) Spuler, S. M. and Fugal, J.: Design of an in-line, digital holographic imaging system for airborne measurement of clouds, Applied optics, 50, 1405–1412, 2011\.
  * Tan and Le (2019) Tan, M. and Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks, in: International conference on machine learning, pp. 6105–6114, PMLR, 2019.
  * Ulyanov et al. (2016) Ulyanov, D., Lebedev, V., Vedaldi, A., and Lempitsky, V. S.: Texture Networks: Feed-forward Synthesis of Textures and Stylized Images, in: Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, edited by Balcan, M. and Weinberger, K. Q., vol. 48 of JMLR Workshop and Conference Proceedings, pp. 1349–1357, JMLR.org, URL http://proceedings.mlr.press/v48/ulyanov16.html, 2016.
  * Wang et al. (2004) Wang, Z., Bovik, A., Sheikh, H., and Simoncelli, E.: Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing, 13, 600–612, 10.1109/TIP.2003.819861, 2004.
  * Wilmot et al. (2017) Wilmot, P., Risser, E., and Barnes, C.: Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses, CoRR, abs/1701.08893, URL http://arxiv.org/abs/1701.08893, 2017.
  * Zhang et al. (2018) Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O.: The Unreasonable Effectiveness of Deep Features as a Perceptual Metric, in: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 586–595, Computer Vision Foundation / IEEE Computer Society, 10.1109/CVPR.2018.00068, 2018.
  * Zhang et al. (2022) Zhang, Y., Zhu, Y., and Lam, E. Y.: Holographic 3D particle reconstruction using a one-stage network, Applied Optics, 61, B111–B120, 2022.


longtable

# Real-Time Detection of Volatility in Liquidity Provision

Matt Brigida111SUNY Polytechnic Institute, 100 Seymour Rd., Utica, NY 13502,
<EMAIL_ADDRESS>
(November 5, 2020)

###### Abstract

Previous research has found that high-frequency traders will vary the bid or
offer price rapidly over periods of milliseconds. This is a benefit to fast
traders who can time thier trades with microsecond precision, however it is a
cost to the average market participant due to increased trade execution price
uncertainty. In this analysis we attempt to construct real-time methods for
determining whether the liquidity of a security is being altered rapidly. We
find a four-state Markov switching model identifies a state where liquidity is
being rapidly varied about a mean value. This state can be used to generate a
signal to delay market participant orders until the price volatility subsides.
Over our sample, the signal would delay orders, in aggregate, over 0 to 10% of
the trading day. Each individual delay would only last tens of milliseconds,
and so would not be noticable by the average market participant.

_JEL Codes:_ G10; G12; C24; C45

_Keywords:_ High-Frequency Trading; Liquidity; Markov-Switching Models

The goal of this analysis is to construct methods to determine, in real-time,
when the volatility of the liquidity provided is being rapidly changed around
a mean value, which is consistent with the effect of an algorithm or set of
algorithms. Such methods would allow the creation of orders which can be
canceled, or delayed, if the market switches to such a regime with unstable
liquidity. This is analogous to the crumbling quote signal from the Investors
Exchange (outlined in Bishop (2017)).

Such real-time detection is a difficult task, though identification does not
have to be perfect. The threshold is that investors choose to use the
order—that it is correlated enough with undesirable activity that it adds
value to the investor to submit the order type. For the order type to have
worth to investors algorithmic activity, or other processes which rapidly
change liquidity around a mean value, which is a cost to the average investor
must exist.

Hasbrouck (2018) found evidence for substantial volatility in the bid and
offer prices which was not due to fundamental changes in the asset value. The
cost of this volatility is not borne equally by traders. Faster traders are
able to choose the point (in microseconds) at which they trade. Slower
traders, however, will receive a trade price some time later (maybe seconds)
after they attempt to submit a marketable order. This trade price is a random
variable, and they are exposed to price risk which is a function of the
expected variation of the bid (or offer) price over the time from when they
submitted the order to when it is matched by the exchange.

So when fast traders change the bid/ask price quickly, slower traders still
_expect_ to receive/pay the same amount for each sell/buy order, however they
have increased uncertainty. This increased risk without increased compensation
should be avoided by any rational investor. The goal of our analysis is to
help investors find ways to delay their order until the execution price of
their order has more certainty. Since the volatility can occur in
milliseconds, the method of identification must itself be algorithmic.

Note, investors should attempt to avoid these periods of increased uncertainty
even if the source of the uncertainty is not high-frequency traders. We
therefore don’t attempt to determine the source of uncertainty, but rather, in
real time, identify when such variations in liquidity are occurring.

Both spread and depth pose substantial risk, particularly for institutional
investors who tend to trade in qualtities far larger than what is available at
the inside quotes. Despite this many seminal models of market making under
asymmetric information ignore market depth by assuming a unit size for all
trades (Copeland and Galai (1983); Glosten and Milgrom (1985); Easley (1992)).
Alternatively, in Kyle (1985) market depth is implicitly incorporated in the
model through requiring specialists to supply complete pricing functions. In
our analysis we will consider the time-series of liquidity available in the
orderbook within a set distance from the bid-ask midpoint.

Our algorithm will attempt to filter out the other various drivers of price
and market depth changes. For example, French and Roll (1986) found evidence
that stock price volatility is driven by private information being
incorporated into market prices via trading. Lee, Mucklow, and Ready (1993)
studied the relationship between spreads and depth around earnings
announcements. So we are attempting to find a state where price and market
depth are changing in a manner inconsistent with trading on private
information or around events. Notably, this first source of price and depth
change would impart a directional bias to prices, and in the case of Lee,
Mucklow, and Ready (1993) the spread widened. Alternatively, the high-
frequency trading we are attempting to identify does not change _mean_ price
or market depth as in these former cases.

##  1 Data

We use data for the heavily traded E-Mini S&P 500 Futures contract. Price
discovery in the equity market occurs in this contract (Hasbrouck (2003)).
Trading hours from Sunday–Friday from 6:00 p.m. to 5:00 p.m. Eastern Time
(ET). Contract value is $50 times the futures price. Cash delivery with
expirations every 3 months. Traded on the Chicago Mercantile Exchange (CME)
(pit and electronic (Globex)).

The reason we use CME Data ES is because, in addition to being the first place
that information is incorporated into prices and trading overnight, all trades
and quotes take place in this one central book. So there is no delay in orders
due to location.

Data are Market Depth
Data222http://www.cmegroup.com/confluence/display/EPICSANDBOX/Market%2BDepth
for E-Mini S&P 500 futures (Globex), for the trading week from November 7 to
November 11, 2016. The data were purchased directly from the CME. We focus our
results on November 9 2016 because it was the trading day where results of the
US Presidential election were released, and therefore there were high levels
of trade and quote volume, which makes the presence of algorithmic activity
more likely.

Market Depth Data contains all market messages (trade/limit order updates) to
and from the CME, and is time-stamped to the nanosecond. The data also
includes tags for aggressor side. Using this data we can recreate the ES
orderbook with nanosecond resolution and up to 10 levels deep. The data are
encoded in the CME’s FIX/FAST message
specification333http://www.cmegroup.com/confluence/download/attachments/62816403/Legacy/%2520FIX-
FAST/%2520Market/%2520Dta/%2520Message/%2520Specification.pdf?version=2&modificationDate=1452872661000&api=v2.
We have made the translation scripts used in this analysis freely
available444https://github.com/Matt-Brigida/CME-FIX-FAST-Translator.

In the following charts and analysis it is helpful to note the difference
between clock and market time. When considering the nanosecond (one-billionth
of a second) level, the market has long periods of inactivity interspersed
with periods of activity. Our data set only contains these periods of activity
(and of course the length of time since the previous period of activity).
Otherwise we would require a time series of 1 billion data points to analyze
each second.

##  2 Methods

Our challenge is that of _unsurpervised learning_ —we are attempting to
identify a state without training data providing the states for a sample of
data. A classic problem of this type in the economics literature is to
determine if the economy is in an expansion or recession. In this
expansion/recession analysis Markov regime-switching regressions are used (see
for example the method employed by the US Federal Reserve). We’ll use a
similar approach in our analysis to determine periods of stable, and unstable,
liquidity driven by algorithmic activity. Our exact model is outlined below.

We measure liquidity on each side of the book as the amount of ES that can be
bought within one point of the present bid-offer midpoint. One point is
equivalent to 4 ticks (so maximum the inside quote and 3 additional levels of
the book). Results below are for the November 9, 2016 trading day, which is
the most likely to exhibit algorithmic trading activity due to the large
public release of information, and the consequent portfolio rebalancing and
increased trade volume.

###  2.1 Markov-Switching Model

There is no test for the proper number of states in a multiple state model. We
thus estimate an increasing number of states and let the interpretation of the
results and standard tests of the residuals, in each state, to guide us to
finding a state consistent with algorithmic activity.

The two-state version of our model is:

$Liq_{t}=\begin{cases}\alpha_{1}+\beta_{11}Liq_{t-1}+\beta_{12}\Delta
BAM+\epsilon_{1},\ \ \ \epsilon_{1}\sim N(0,\sigma_{1})\\\
\alpha_{2}+\beta_{21}Liq_{t-1}+\beta_{22}\Delta BAM+\epsilon_{2},\ \ \
\epsilon_{2}\sim N(0,\sigma_{2})\\\ \end{cases}$

$P(s_{t}=j|s_{t-1}=i)=p_{ij}\\\ \text{for}\ \ i,j\in{1,2}\ and\
\sum_{j=1}^{2}p_{ij}=1$

where $Liq_{t-1}$ is the liquidity in the previous period and $\Delta BAM$ is
the most recent change in the bid-ask midpoint. There are two states, denoted
by $s_{1}$ and $s_{2}$, and $p_{ij}$ denotes the probability that the state is
$j$ given the state was $i$ in the previous period. We estimate the model via
the Hamilton Filter with a custom implementation in C++ due to the large
number of points in our time series.

Similar to the bid and ask volatility estimate in Hasbrouck (2018), we
estimate the model for the bid and ask sides of the book separately. This is
because the rapid deviations from a mean liquidity value, which we are
attempting to identify, largely affect one side of the book, and so are more
likely to be an artifact of the trading process rather than due to fundamental
information. Nonetheless, modeling the entire book (bid and ask sides jointly)
would include more information in the parameter estimates, such as spillover
effects. However this would increase the time required to estimate parameters
as well as the time it takes to create a state prediction. Since the algorithm
must be very quick to be useful, we err on the side of speed relative to the
benefit of the information in both sides of the spread.

####  2.1.1 Two States

The two-state model is picking up states of changing liquidity and stable
liquidity. In both the bid and offer models, the first state had a coefficient
of 1 on the previous liquidity, and a small residual standard deviation. This
state is consistent with no public or private information being incorporated
into prices, and little in the market changing.

The second state, which has a higher residual variance, exhibits evidence of
changing liquidity. However the coefficient on previous liquidity, and the
intercept are significantly different between the two models. Accordingly,
state 2 may be driven by liquidity changing for various reasons. These results
motivate a 3 state model where we differentiate the state with changing
liquidity into two states—one representing changing liquidity due to HFT
activity:

  * •

Stable liquidity

  * •

Normal changing liquidity

  * •

Changing liquidity due to HFT

#####  2.1.1.1 Bid

$Liq_{t}=\begin{cases}0.00+1.00Liq_{t-1}+0.09\Delta BAM+\epsilon_{2},\ \ \
\epsilon_{2}\sim N(0,0.002)\\\ -0.83+0.49Liq_{t-1}-0.06\Delta
BAM+\epsilon_{1},\ \ \ \epsilon_{1}\sim N(0,0.47)\\\ \end{cases}$

Figure 1: Two state Markov-Switching model of liquidity available at the bid.

#####  2.1.1.2 Offer

$Liq_{t}=\begin{cases}0.42+1.33Liq_{t-1}-0.12\Delta BAM+\epsilon_{1},\ \ \
\epsilon_{1}\sim N(0,0.42)\\\ 0.00+1.00Liq_{t-1}+0.16\Delta BAM+\epsilon_{2},\
\ \ \epsilon_{2}\sim N(0,0.001)\\\ \end{cases}$

Figure 2: Two state Markov-Switching model of liquidity available at the
offer.

####  2.1.2 Three States

The first state in the 3 state model again exhibits stable liquidity. The
following two states exhibit varying volatility which is driven by different
factors. In state 2 liquidity is driven by a change in the bid-ask midpoint.
This is consistent with liquidity provision in reaction to a movement in the
market—posssibly driven by new information.

In state 3, however, a change in the bid-ask midpoint has no effect on
liquidity. Similarly previous liquidity explains only a quarter to a third of
present liquidity, and the variance of the residual is the highest in state 3.
If there is HFT activity present, it is most likely within state 3. Note,
these results are consistent across both bids and asks. Lastly, we’ll estimate
the parameters of a four state model to see if state 3 is a composite of other
states.

#####  2.1.2.1 Bid

$Liq_{t}=\begin{cases}-0.00+1Liq_{t-1}-0.12\Delta BAM+\epsilon_{1}\\\
-0.09+0.22Liq_{t-1}+1.02\Delta BAM+\epsilon_{2}\\\
-0.01+0.32Liq_{t-1}+0.004\Delta BAM+\epsilon_{3}\\\ \end{cases}$

$\texttt{where}\begin{cases}\epsilon_{1}\sim N(0,0.004)\\\ \epsilon_{2}\sim
N(0,0.2929)\\\ \epsilon_{3}\sim N(0,0.40)\\\ \end{cases}$

Figure 3: Three state Markov-Switching model of liquidity available at the
bid.

#####  2.1.2.2 Offer

$Liq_{t}=\begin{cases}-0.00+1Liq_{t-1}-0.10\Delta BAM+\epsilon_{1}\\\
0.38+-0.03Liq_{t-1}+0.81\Delta BAM+\epsilon_{2}\\\
0.12+0.25Liq_{t-1}+0.01\Delta BAM+\epsilon_{3}\\\ \end{cases}$

$\texttt{where}\begin{cases}\epsilon_{1}\sim N(0,0.004)\\\ \epsilon_{2}\sim
N(0,0.078)\\\ \epsilon_{3}\sim N(0,0.90)\\\ \end{cases}$

Figure 4: Three state Markov-Switching model of liquidity available at the
offer.

####  2.1.3 Four States

Similar to the three-state equation, the first two states represent stable
liquidity, and changing liquidity driven by changes in the bid-ask midpoint.
State 3 exhibits negative relationships between previous and present
liquidity. The standard deviation of the error term is moderately high in this
state, however it is about a quarter to a third of the standard deviation of
the error term in state 4.

State 4 is most consistent with the type of HFT activity we are trying to
identify. In state 4 liquidity remains constant with substantial variability
around the stable mean liquidity amount.

#####  2.1.3.1 Bid

$Liq_{t}=\begin{cases}0.0024+0.9983Liq_{t-1}+0.1319\Delta BAM+\epsilon_{1}\\\
-0.0594-0.3211Liq_{t-1}+0.8524\Delta BAM+\epsilon_{2}\\\
0.3796-0.0636Liq_{t-1}-0.1802\Delta BAM+\epsilon_{3}\\\
-0.1626+0.9469Liq_{t-1}+0.0791\Delta BAM+\epsilon_{4}\\\ \end{cases}$

$\texttt{where}\begin{cases}\epsilon_{1}\sim N(0,0.0077)\\\ \epsilon_{2}\sim
N(0,0.2901)\\\ \epsilon_{3}\sim N(0,0.2409)\\\ \epsilon_{4}\sim N(0,0.6580)\\\
\end{cases}$

Figure 5: Four state Markov-Switching model of liquidity available at the bid.

#####  2.1.3.2 Offer

$Liq_{t}=\begin{cases}0.0000+1.0000Liq_{t-1}-0.2681\Delta BAM+\epsilon_{1}\\\
-0.0055+0.9949Liq_{t-1}-1.1200\Delta BAM+\epsilon_{2}\\\
-1.1325-0.3480Liq_{t-1}-0.0122\Delta BAM+\epsilon_{3}\\\
-0.0048+1.0051Liq_{t-1}-0.5034\Delta BAM+\epsilon_{4}\\\ \end{cases}$

$\texttt{where}\begin{cases}\epsilon_{1}\sim N(0,0.0000)\\\ \epsilon_{2}\sim
N(0,0.0153)\\\ \epsilon_{3}\sim N(0,0.1400)\\\ \epsilon_{4}\sim N(0,0.6207)\\\
\end{cases}$

Figure 6: Four state Markov-Switching model of liquidity available at the
offer.

Given the above estimates, and using 0.2 as our signal threshold for state 4,
the signal fires, on average, 0.636 times per second on the bid side of the
orderbook. On the ask side of the orderbook the signal fires 10.59 times per
second on average. Assuming a 10 millisecond delay each time the signal fires,
this implies the signal duration of 0.636% and 10.59% of the trading day on
the bid and ask side of the orderbook respectively. This duration range is
reasonable given anecdotal accounts of the pervasiveness of high-frequency
trading in markets, such as Hendershott, Jones, and Menkveld (2011) which
reported that as much as 73% of volume in US markets was due to high-frequency
trading.

In tables 3 and 4 in the appendix we provide parameter estimates for the 4
state model, along with signal duration estimates, for the entire week
(November 7 through 11, 2016). The parameter estimates are very similar across
days for each side of the orderbook. Further the signal durations are also
similar with the exception of the offer side of the book on the November 9th
trading day. The large release of public information occurred on November 9th,
and this orderbook asymmetry with regards to algorithmic activity is
consistent with Hasbrouck (2018).

Table 1: Bid side of the orderbook. Below are coefficient estimates from the Markov-switching regressions. The standard errors are next to the coefficient in parentheses. The coefficients were estimated using the nanosecond time-stamped orderbook ranging from 6:00 PM EST on November 8, 2016 to 5:00 PM EST on November 9, 2016. There are 9,965,673 changes to the orderbook for this period. Coefficient | Two-State | Three-State | Four-State
---|---|---|---
$\alpha_{1}$ | 0.00(0.0000) | -0.00(0.0000) | 0.00(0.0000)
$\alpha_{2}$ | -0.83(0.0007) | -0.09(0.0250) | -0.05(0.0033)
$\alpha_{3}$ |  | -0.01(0.0140) | 0.37(0.0025)
$\alpha_{4}$ |  |  | -0.16(0.0018)
$\beta_{11}$ | 1.00(0.0000) | 1.00(0.0000) | 0.99(0.0000)
$\beta_{12}$ | 0.09(0.1369) | -0.12(0.259) | 0.13(0.4226)
$\beta_{21}$ | 0.49(0.0004) | 0.22(0.003) | -0.32(0.0110)
$\beta_{22}$ | -0.06(0.0075) | 1.02(0.670) | 0.85(1.1350)
$\beta_{31}$ |  | 0.32(0.000) | -0.06(0.0050)
$\beta_{32}$ |  | 0.00(0.0000) | -0.18(0.8833)
$\beta_{41}$ |  |  | 0.94(0.0001)
$\beta_{42}$ |  |  | 0.07(0.0933)
$\sigma_{1}$ | 0.00(0.0490) | 0.00(0.0661) | 0.00(0.0000)
$\sigma_{2}$ | 0.47(0.0002) | 0.29(0.0024) | 0.29(0.0044)
$\sigma_{3}$ |  | 0.40(0.0001) | 0.24(0.0033)
$\sigma_{4}$ |  |  | 0.65(0.0008)
Table 2: Ask side of the orderbook. Below are coefficient estimates from the Markov-switching regressions. The standard errors are next to the coefficient in parentheses. The coefficients were estimated using the nanosecond time-stamped orderbook ranging from 6:00 PM EST on November 8, 2016 to 5:00 PM EST on November 9, 2016. There are 9,965,673 changes to the orderbook for this period. Coefficient | Two-State | Three-State | Four-State
---|---|---|---
$\alpha_{1}$ | 0.42(0.0000) | -0.00(0.0000) | 0.00(0.0000)
$\alpha_{2}$ | 0.00(0.0041) | 0.38(0.0141) | -0.00(0.0970)
$\alpha_{3}$ |  | 0.12(0.0196) | -1.13(0.7924)
$\alpha_{4}$ |  |  | -0.00(0.02269)
$\beta_{11}$ | 1.33(0.4078) | 1.00(0.0000) | 1.00(0.0083)
$\beta_{12}$ | -0.12(0.0059) | -0.10(0.2259) | -0.26(0.1421)
$\beta_{21}$ | 1.00(0.0000) | -0.03(0.0192) | 0.99(0.0001)
$\beta_{22}$ | 0.16(0.0009) | 0.81(1.5312) | 1.12(0.0018)
$\beta_{31}$ |  | 0.25(0.0027) | -0.34(0.0990)
$\beta_{32}$ |  | 0.01(1.3956) | -0.01(0.6147)
$\beta_{41}$ |  |  | 1.00(0.2876)
$\beta_{42}$ |  |  | -0.50(0.9778)
$\sigma_{1}$ | 0.42(0.0011) | 0.00(0.0000) | 0.00(0.0000)
$\sigma_{2}$ | 0.00(0.0327) | 0.07(0.0101) | 0.01(0.0626)
$\sigma_{3}$ |  | 0.90(0.0002) | 0.14(0.0115)
$\sigma_{4}$ |  |  | 0.62(0.0004)

##  3 Conclusion

In this analysis we have used Markov-Switching regression models to identify
the presence of high-frequency traders who are rapidly changing volatility.
Using a model with four states, we identify a state with a stable mean
liquidity, but substantial variability in liquidity around the mean. That is
there is rapidly changing liquidity, which does not affect overall liquidity
or the price.

Since trading in this state benefits high-frequency traders at the expense of
slower retail order flow, a transition to this state can serve as a signal to
delay slower traders’ orders. The delay being mere tens of milliseconds, it
will not be perceptible to the typical trader. And while this may save each
trade a small amount, in aggregate such a delayed order type would provide
substatial savings across all non-high-frequency traders. Delaying orders due
to the signal can be offered to retail traders through a particular order
type. A similar strategy is used by the IEX’s ‘crumbling quote’ order.

##  4 Appendix

In tables 3 and 4 below are parameter estimates from the following 4-state
Markov-Switching model.

Table 3: Parameter estimates from a 4-state Markov-switching model on the liquidity available on the bid side of the orderbook. There are 2,917,466 entries to the book over the Nov. 7 trading day. There are 3,502,097 book entries on Nov. 8. There are 9,965,673 book entries on Nov. 9, which is the trading day over which the results of the election were announced. There were 7,346,604 book entries on Nov. 10, and 4,905,882 on Nov. 11. The duration of the signal (Sig. Dur.) was calculated assuming a 10 millisecond delay for each signal, and a 0.2 threshold for the signal generation. Coefficient | Nov. 7 | Nov. 8 | Nov. 9 | Nov. 10 | Nov. 11
---|---|---|---|---|---
$\alpha_{1}$ | 0.0065 | 0.0431 | 0.0024 | -0.0656 | -0.0010
$\alpha_{2}$ | -0.1132 | -0.1694 | -0.0594 | -0.4849 | -0.4917
$\alpha_{3}$ | 0.1121 | 0.3509 | 0.3796 | 0.3917 | 0.2975
$\alpha_{4}$ | -0.2210 | -0.1783 | -0.1626 | -0.2966 | -0.3563
$\beta_{11}$ | 1.0004 | 0.8102 | 0.9983 | 0.9500 | 1.0057
$\beta_{12}$ | -0.1579 | 0.0754 | 0.1319 | 0.0174 | 0.2716
$\beta_{21}$ | 0.1741 | 0.0168 | -0.3211 | 0.2004 | 0.2565
$\beta_{22}$ | 0.9270 | 0.8738 | 0.8524 | 0.8375 | 0.4647
$\beta_{31}$ | 0.0628 | 0.1031 | -0.0636 | 0.0132 | 0.0101
$\beta_{32}$ | -0.1324 | -0.1707 | -0.1802 | -0.1676 | -0.3864
$\beta_{41}$ | 0.6621 | 0.6239 | 0.9469 | 0.4445 | 1.1467
$\beta_{42}$ | 0.1151 | 0.0910 | 0.0791 | -0.0752 | -0.2919
$\sigma_{1}$ | 0.0221 | 0.0912 | 0.0077 | 0.0109 | 0.0219
$\sigma_{2}$ | 0.0920 | 0.1716 | 0.2901 | 0.4268 | 0.6963
$\sigma_{3}$ | 0.1701 | 0.0787 | 0.2409 | 0.0769 | 0.1083
$\sigma_{4}$ | 0.6386 | 0.6873 | 0.6580 | 0.4252 | 0.3709
Log Lik. | 4880164 | 117503.2 | 16693395 | 20395.45 | 249944.1
Sig. Dur. | 0.736% | 0.020% | 0.636% | 0.000% | 0.000%
Table 4: Parameter estimates from a 4-state Markov-switching model on the liquidity available on the offer side of the orderbook. There are 2,917,466 entries to the book over the Nov. 7 trading day. There are 3,502,097 book entries on Nov. 8. There are 9,965,673 book entries on Nov. 9, which is the trading day over which the results of the election were announced. There were 7,346,604 book entries on Nov. 10, and 4,905,882 on Nov. 11. The duration of the signal (Sig. Dur.) was calculated assuming a 10 millisecond delay for each signal, and a 0.2 threshold for the signal generation. Coefficient | Nov. 7 | Nov. 8 | Nov. 9 | Nov. 10 | Nov. 11
---|---|---|---|---|---
$\alpha_{1}$ | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000
$\alpha_{2}$ | -0.0007 | 0.0008 | -0.0055 | 0.0011 | -0.0051
$\alpha_{3}$ | -1.1325 | -1.1374 | -1.1325 | -0.1314 | -1.1329
$\alpha_{4}$ | -0.0042 | 0.0052 | -0.0048 | -0.0060 | 0.0015
$\beta_{11}$ | 1.0054 | 1.0051 | 1.0049 | 1.0059 | 0.9979
$\beta_{12}$ | -0.2681 | -0.2707 | -0.2681 | -0.2643 | -0.0033
$\beta_{21}$ | 0.9960 | 0.9977 | 0.9949 | 0.9991 | 0.9955
$\beta_{22}$ | -1.1161 | -1.1034 | -1.1200 | -1.1195 | -1.291
$\beta_{31}$ | 0.3465 | 0.3481 | 0.3480 | 0.3487 | 0.3414
$\beta_{32}$ | -0.0121 | -0.0153 | -0.0122 | -0.0082 | -0.0064
$\beta_{41}$ | 1.2411 | 1.2369 | 1.0086 | 1.2357 | 1.0148
$\beta_{42}$ | -0.5031 | -0.5057 | -0.5034 | -0.5006 | -0.5136
$\sigma_{1}$ | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000
$\sigma_{2}$ | 0.0016 | 0.0018 | 0.0153 | 0.0030 | 0.0039
$\sigma_{3}$ | 0.1400 | 0.1400 | 0.1400 | 0.1400 | 0.1400
$\sigma_{4}$ | 0.6217 | 0.6369 | 0.6207 | 0.6316 | 0.6165
Log Lik. | 841089.4 | 5872364 | 45918365 | 5637773 | 839429.5
Sig. Dur. | 0.403% | 0.797% | 10.59% | 0.704% | 1.206%

## References

Bishop, Allison. 2017. “The Evolution of the Crumbling Quote Signal.”
_Available at SSRN 2956535_.

Copeland, Thomas E., and Dan Galai. 1983. “Information Effects on the Bid-Ask
Spread.” _The Journal of Finance_ 38 (5): 1457–69.
https://doi.org/10.1111/j.1540-6261.1983.tb03834.x.

Easley, Maureen, David And O’Hara. 1992. “Time and the Process of Security
Price Adjustment.” _The Journal of Finance_ 47 (2): 577–605.
https://doi.org/10.1111/j.1540-6261.1992.tb04402.x.

French, Kenneth R., and Richard Roll. 1986. “Stock Return Variances.” _Journal
of Financial Economics_ 17 (1): 5–26.
https://doi.org/10.1016/0304-405x(86)90004-8.

Glosten, Lawrence R., and Paul R. Milgrom. 1985. “Bid, Ask and Transaction
Prices in a Specialist Market with Heterogeneously Informed Traders.” _Journal
of Financial Economics_ 14 (1): 71–100.
https://doi.org/10.1016/0304-405x(85)90044-3.

Hasbrouck, Joel. 2003. “Intraday Price Formation in Us Equity Index Markets.”
_The Journal of Finance_ 58 (6): 2375–2400.

———. 2018. “High-Frequency Quoting: Short-Term Volatility in Bids and Offers.”
_Journal of Financial and Quantitative Analysis_ 53 (2): 613–41.

Hendershott, Terrence, Charles M Jones, and Albert J Menkveld. 2011. “Does
Algorithmic Trading Improve Liquidity?” _The Journal of Finance_ 66 (1): 1–33.

Kyle, Albert S. 1985. “Continuous Auctions and Insider Trading.”
_Econometrica_ 53 (6): 1315. https://doi.org/10.2307/1913210.

Lee, Charles M. C., Belinda Mucklow, and Mark J. Ready. 1993. “Spreads,
Depths, and the Impact of Earnings Information: An Intraday Analysis.” _Review
of Financial Studies_ 6 (2): 345–74. https://doi.org/10.1093/rfs/6.2.345.