# SynCellFactory: Generative Data Augmentation for Cell Tracking

Moritz Sturm
<EMAIL_ADDRESS>Interdisciplinary Center for Scientific Computing (IWR),
Heidelberg University. Lorenzo Cerrone 11footnotemark: 1
<EMAIL_ADDRESS>Fred A. Hamprecht 11footnotemark: 1
<EMAIL_ADDRESS>
###### Abstract

Cell tracking remains a pivotal yet challenging task in biomedical research.
The full potential of deep learning for this purpose is often untapped due to
the limited availability of comprehensive and varied training data sets. In
this paper, we present SynCellFactory, a generative cell video augmentation.

At the heart of SynCellFactory lies the ControlNet architecture, which has
been fine-tuned to synthesize cell imagery with photorealistic accuracy in
style and motion patterns. This technique enables the creation of synthetic
yet realistic cell videos that mirror the complexity of authentic microscopy
time-lapses.

Our experiments demonstrate that SynCellFactory boosts the performance of
well-established deep learning models for cell tracking, particularly when
original training data is sparse.

##  1 Introduction

Digital time-lapse microscopy allows for large-scale observations of cells
over time, providing a deeper understanding of cellular processes [6, 19, 3].
However, to fully harness the potential of time-lapse imaging, automated cell
tracking approaches are needed, which can provide a quantitative analysis of
cell behavior for vast amounts of data.

Cell tracking is characterized by challenges such as variable image contrast,
intricate behaviors such as cell division and, in some examples,
indistinguishability of cells. Recent advancements in computer vision have
shown that neural networks are highly effective in multi-object tracking tasks
[12, 23, 9]. However, their application in cell tracking remains limited and
exploratory, primarily due to the scarcity of annotated cell tracking data.
This shortage hinders the development of robust neural network models for cell
tracking [22].

In recent years, although medium-scale annotated tracking data sets in the
order of several thousand timeframes have become available [32, 43, 22], they
are limited to specific cell styles. However, given the vast diversity of cell
types and imaging techniques, it is difficult for a single data set to cover
all these variations comprehensively. The reason for the lack of a broad range
of annotated training cell tracking data is the costly expert annotation
combined with cell tracking’s niche status in the broader field of machine
learning.

To address these challenges, this paper introduces SynCellFactory, a
generative data augmentation strategy specifically designed for cell tracking.
We embrace the power of conditioned 2D diffusion models [28, 44] to generate
high-quality synthetic cell videos that mimic the appearance and behavior of
real cell data sets.

Utilizing as little as one annotated cell video for training, SynCellFactory
can generate an extensive library of annotated videos in a consistent style,
effectively augmenting the available training data. This advancement holds the
potential to transform cell tracking by enabling the application of
sophisticated deep learning models previously hindered by data scarcity. Our
research focuses on evaluating the impact of this data augmentation on the
performance of state-of-the-art cell tracking models. The results indicate
that this novel approach addresses the data scarcity issue and enhances
tracking accuracy. We aim to open the door for large-scale deep learning
architectures in cell tracking.

To summarize, our contributions are:

  * •

SynCellFactory, a generative data augmentation pipeline for cell tracking.

  * •

The proposed pipeline demonstrates out-of-the-box robust results on a wide
variety of data sets without complex hyperparameter tuning or domain-specific
knowledge.

  * •

Empirical proof that the proposed data augmentation strategy can further
enhance accuracy of an already leading deep learning cell tracking method.

Figure 1: Showcase of real (top row) and synthetic (other rows) images
generated using SynCellFactory. The training data sets are a subset of the 2D
Cell Tracking Challenge [41, 22] and provide a broad spectrum of cell lines
and microscopy modalities. Figure 2: SynCellFactory is a data augmentation
pipeline designed to create unlimited high-quality synthetic raw video data
and corresponding pseudo ground truth. It trains three key components using a
small, and possibly sparsely labeled data set: Positional ControlNet (CN-Pos),
Movement ControlNet (CN-Mov), and a 2D movement engine for realistic
simulations. The process initiates in reverse, with the motion model
generating a conditioning image at time $T$ for CN-Pos. This image illustrates
the expected centers of cells using colored dots, where each color signifies a
specific cell state in the mitotic cycle: green during the interphase and blue
during the different phases of cell division. CN-Pos then employs this
information to generate a realistic frame for time $T$. Subsequently, CN-Mov
assumes the role of producing the next frame $T-1$, using as conditioning an
RGB image that combines the previously generated frame (in the red channel)
with the projected positions and movement patterns (in green and blue
channels). Derived from the motion model, these patterns represent each cell’s
trajectory from its current to its anticipated next position as a line
connecting the two. By iteratively applying CN-Mov, SynCellFactory can
efficiently produce time-lapse sequences of any desired length, suitable for
training deep learning pipelines in cell tracking.

###  1.1 Related Work

Generative Data Augmentation: Data augmentation is ubiquitous in deep learning
and is known to help enhance the robustness and generalization of deep
learning models [27, 34, 13, 35]. Most data augmentation strategies in
computer vision are based on simple transformations of the input data,
including rotations, non-rigid deformations, color alteration, and cropping of
images or videos. Recently, approaches based on generative machine learning
are gaining traction [1]. The idea is to use a generative model to sample a
new data point drawn from the estimated real data distribution. Until
recently, Generative Adversarial Networks (GANs) were the primary choice for
this task [2, 40, 46]. However, diffusion models advanced and eventually
surpassed GANs in image synthesis quality [7], making diffusion models the new
state of the art for generating synthetic image data [1, 11, 14]. Refs. [1,
14, 39, 30] demonstrate the effectiveness of generative data augmentation
using diffusion models in terms of achievable image classification accuracy.
These advances also start to impact the medical imaging domain, where they
show great promise [37, 25, 8, 26] and have already proven effective when used
to train deep learning models [5, 10]. The main motivation behind using
generative data augmentation in medical computer vision is the scarcity of
ground truth as well as privacy concerns.

Cell Synthesis: Several approaches to synthesizing artificial cell images have
been proposed since the late 90s [18]. Early methods relied heavily on
handcrafted parametric models. These models, informed by extensive biological
knowledge, simulated the appearance and temporal evolution of cells [38].

However, with the evolution of deep learning, an effort has been made to
reduce the domain-specific knowledge required to generate realistic cell
video. While [4] combines statistical shape and evolution models with
conditional GANs to generate 2D+t data, [33] employs video diffusion models to
create full cell videos directly from the training set. Most existing models
focus on generating a single cell in either 2D or 3D [16, 45, 42, 2, 31]. To
simulate a cell population, some methods first model single cell appearance
and temporal development and then incorporate simultaneous evolution of
multiple cells to model a population of cells [38, 4]. Our proposed method is
based on a similar premise to model cell appearance and temporal development
separately, but distinguishes itself by modeling the image of an entire cell
population jointly. Closely related to our work, [33] generates an entire
video at once using a 3D diffusion model guided by optical flow; crucially,
their approach does not produce pseudo ground truth labels, while
SynCellFactory does. Ref. [17] proposes a CycleGAN [47] based model which can
also generate raw data as well as lineage and segmentation pesudo ground
truth. An important difference is that said work relies on simulated ground
truth segmentation masks to condition the data generation; this is a much
harder problem to solve than the simple trajectory modeling employed here.

##  2 Method

SynCellFactory operates on the principle of decoupling cell dynamics from
their appearance. By controlling these two aspects independently, our model
can generate highly heterogeneous annotated cell videos. Our model comprises
two principal components:

  1. 1.

A simple 2D motion model that simulates the spatial distribution and dynamics
of cell cultures. It uses statistical parameters derived from real data,
enabling SynCellFactory to produce coherent and physically plausible
timelapses, irrespective of frame count. This simulation approach enhances the
diversity and realism of the generated data.

  2. 2.

Two distinct ControlNets, each with a specific function, are trained for
photorealistic rendering. The first, CN-Pos, is adept at inpainting cells with
lifelike appearance at accurate spatial positions. The second, CN-Mov, focuses
on the temporal displacement of individual cells across consecutive frames,
ensuring temporal consistency. Particularly crucial for our use case in data
augmentation, these ControlNets demonstrate efficient training capabilities,
even with limited data such as a single time-lapse sequence, making them well
suited for augmenting data sets in cell tracking applications.

In the following sections, we will detail the methodology of our model and
introduce an automated protocol for training SynCellFactory on new data sets.

###  2.1 Motion Model

Cell movement under a microscope is a process influenced by various factors
like cell type, environmental conditions, and cell-to-cell interactions.
Biologically accurate modeling of the cell behavior is outside the scope of
this manuscript. Therefore, we are not implementing intra-cellular forces and
complex behaviors such as amoeboid, ciliary, and flagellar motions. The goal
of our motion model is to generate plausible spatial cell configuration and
displacement, leaving the heavy lifting of visually representing cell
interactions to the generative neural network.

Our engine represents cells as 2D disks, capable of moving, splitting, and
interacting with one another. The motion model initializes a population of
cells with randomly sampled positions and sizes, drawn from the distribution
observed in real cells from our training data. This population dynamically
evolves following a stochastic Brownian motion, guided by statistics extracted
from annotated real cell videos. These key parameters include the mean and
standard deviation of cell area ($A_{c}$ and $\sigma_{A}$) and the cell
displacement distribution, which we model using a gamma distribution
$G(\alpha_{d},\Theta_{d})$. These statistical models are derived from
available segmentation and tracking ground truth data. A significant advantage
of our approach is its flexibility in adjusting the tracking difficulty of the
generated data sets. By manipulating variables such as the number of cells,
their movement speed, and the frequency of cell splitting events, we can
tailor the complexity of the tracking task. This allows us to create data sets
that either exceed or are comparable in complexity to the original data,
accommodating a range of research and application needs. Details on the
statistical methods, the specific equations governing cell movement, and the
pseudocode for the simulation are provided in supplementary Sec. A.

###  2.2 ControlNet

ControlNet [44] is a popular architecture for enabling the conditioning of
text-to-image generative models like stable diffusion [29].

Our ControlNet uses the standard pre-trained Stable Diffusion v1.5 backbone
trained on natural images. In contrast to the original ControlNet model that
maintains the stable diffusion decoder locked, we fine-tune it to the
appearance of biological images.

Morever, training a ControlNet in the context of latent text-to-image
diffusion models involves triplets of the form
$(c_{\text{txt}},c_{\text{img}},i_{\text{tgt}})$, where $c_{\text{txt}}$ is a
conditioning text, $c_{\text{img}}$ is the conditioning image and
$i_{\text{tgt}}$ is the target image. The text conditioning has been kept
fixed for all our experiments to the prompt “cell, microscopy, image” to
provide a good initialization during the early stages of training.

In the following sections, we describe the specific details and
$c_{\text{img}}$ conditioning the CN-Pos and CN-Move models.

Figure 3: Train-time computation flow in the ControlNet [44]. Initially, the
source image $I_{\text{tgt}}$, and the conditioning input $c_{\text{img}}$ are
transformed into $4\times 64\times 64$ embeddings through two convolutional
encoders: $\mathcal{E}_{\text{SD}}$ for the source and
$\mathcal{E}_{\text{CNet}}$ for conditioning. In the stable diffusion training
routine, the source embedding undergoes a forward diffusion process, where
Gaussian noise is incrementally added. Subsequently, a UNet encoder-decoder
attempts to estimate and revert this noise perturbation. This is accomplished
by applying the mean square error loss between the input noise
$\epsilon_{\mathcal{N}}$ and the decoder output $\epsilon_{\Theta}$. Unique to
ControlNet architecture, additional image conditioning is integrated via an
auxiliary branch. In the diagram, neural network blocks with solid lines
represent components with fixed parameters during training, while those with
dotted lines indicate blocks subject to finetuning. The described architecture
is used in both CN-Pos and CN-Mov.

####  2.2.1 Positional ControlNet

The positional ControlNet CN-Pos is tasked with drawing realistic looking
cells conditioned on a given position (see Fig. 2). This model plays two
fundamental roles in SynCellFactory: it is used to generate the last frame of
our synthetic videos and is the pre-trained backbone of the CN-Move model. At
train time, the position map is constructed by computing the center
coordinates for each cell given the corresponding detection ground truth (see
Fig. 2). Each detection is represented in the conditioning image
$c_{\text{img}}$ as a disk with a fixed radius of $r=\sqrt{{A_{c}}/{\pi}}/4$,
where $A_{c}$ is the data set average cell area in pixels. Fixing the radii of
the disks instead of varying them proportionally to the cell size empirically
improved the diversity of cell appearances. Moreover, the disks are colored
according to the stage in the cell cycle. The disk is green while the cell is
in the interphase, linearly changing between green and blue during the Mitosis
phases (Prophase, Metaphase, Anaphase, and Telophase) and linearly reverting
from blue to green after the cytokinesis phase and the split event is
complete. The cycle length is a hyperparameter that needs to be set according
to the specific cell type for each data set. At inference time, the
conditioning image $c_{\text{img}}$ is derived by the state simulated by the
motion model, where the center of the simulated cells are converted into
color-coded disks.

####  2.2.2 Movement ControlNet

CN-Mov is analogous to CN-Pos in its implementation details; the sole
difference lies in its conditioning.

CN-Mov is tasked with predicting frame $t-1$ conditioned on the frame at time
$t$, the position map at time $t-1$, and the displacement vectors. Frame $t$
is encoded in the red channel of $c_{\text{img}}$, while the position map at
time $t-1$ is encoded in the blue and green channels as described in Sec.
2.2.1. Displacement vectors describe the movement of the cells between frames
and are encoded as lines connecting the center position of cells. A more
detailed description of the conditionings can be found in supplementary Sec.
B.

###  2.3 SynCellFactory Inference

During the sampling process, we apply the trained ControlNets on the output of
our motion module to generate realistic-looking videos (see Fig. 2). The
choice to sample our video in reverse time is arbitrary, but simplifies
network learning: in backward time, cell merge events require the network to
predict only a single position from two merging cells. Conversely, in forward
time, the network would face the more complex task of generating two new cells
from one during a split event.

The inference process is initiated with CN-Pos generating frame $t$.
Subsequently, CN-Mov iteratively takes the generated frame $t$ and, in
conjunction with the motion model, samples a new frame $t-1$. This iterative
process continues until the desired video length is reached. In all our
experiments, we generated a fixed length of 12 frames.

###  2.4 Segmentation Pseudo Ground Truth

SynCellFactory produces only raw video, detection, and lineage ground truth,
but critically is not able to produce instance segmentation ground truth. This
is a significant challenge, as several deep learning methods [20, 43] require
full segmentation for effective training. To address this, we rely on Cellpose
[36, 24], a renowned deep learning framework for cell segmentation, to create
pseudo-ground truth segmentation and thereby mitigate this limitation.
Cellpose is particularly effective due to its range of pretrained models that
cater to various cell image styles, boasting high segmentation accuracy right
out of the box. For our cell data sets, we chose the pretrained model
recommended by Cellpose, and further finetuned it for 100 epochs using ground
truth segmentation masks from our training data. This finetuning allows the
model to more closely adapt to the unique characteristics of a particular data
set. While the segmentation labels produced by Cellpose are mostly accurate,
they occasionally miss or hallucinate cells. To improve upon this, we
integrated the motion model into the segmentation process. In cases where a
segmentation mask does not overlap with any part of the generated detection
ground truth, it is removed. Conversely, when a ground truth detection is
present without a corresponding segmentation mask, we generate one by drawing
a circle with a radius $r=\sqrt{{A_{c}}/{\pi}}$. This step is crucial to
ensure that small inaccuracies or hallucination artifacts produced by the
ControlNet do not receive segmentation masks, while cells that are challenging
to segment are still provided with an approximate mask. See supplementary Sec.
C for an illustrative example of the correction step.

###  2.5 Automated Training

To reduce the domain-specific expertise required to train SynCellFactory, we
propose a fully automated pipeline for training the ControlNets and the
sampling from the motion module.

The only required hyperparameters that have to be manually specified to
produce our synthetic videos are: the number of videos to be generated, the
number of frames per video, and the characteristic length of the mitosis
cycle.

Other parameters,such as movement statistics for the motion model, are
automatically inferred from the raw data and ground truth annotations. One of
the main challenges with training the ControlNet is defining a good stopping
criterion. We found that in our applications, the number of required steps to
convergence is correlated with the average cell number in each frame. Thus, we
can automatically set an appropriate number of training iterations for our
ControlNets (see supplementary Sec. D)

##  3 Experiments and Results

###  3.1 Data sets

To validate the proposed SynCellFactory we use the publicly available Cell
Tracking Challenge (CTC) [41, 22]
111http://celltrackingchallenge.net/2d-datasets/ data sets. The CTC is an open
benchmark to test new cell tracking and segmentation advancements on data sets
with a large variety of cell types, microscopy modalities, and different
styles, as well as inhomogeneous spatial and temporal resolution.

For our experiments, we focus on the seven 2D data sets enumerated in Fig. 1.
Each data set consists of two timelapses with full tracking annotations of
ground truth and only partial hand-curated segmentation. The tracking ground
truth includes detection and identity masks for each frame and the
corresponding cell lineages. The number of frames per video ranges from 30
(Fluo-C2DL-Huh7) to 300 (PhC-C2DL-PSC).

Our experiments used a single time-lapse for the training and validation and
one for testing tracking accuracy. Sample still frames from the mentioned data
sets can be found in Fig. 1. We have elided from the pool of data sets for our
experiments Fluo-N2DH-SIM+ because it is an already simulated cell video [38].
We excluded BF-C2DL-HSC and BF-C2DL-MuSC because the boundary conditions of
the petri dish could not be reproduced by our motion model and because these
data sets already provide several thousand annotated frames, rendering
augmentation less relevant.

###  3.2 Experimental Setup

The training of CN-Pos and CN-Move follows the standard procedure as presented
in [44]; the only major difference is that we also finetune the stable
diffusion UNet decoder block. In the original ControlNet architecture, this
module is fixed at train time. A detailed diagram of the training computation
is shown in Fig. 3.

Since we could only access a single annotated timelaps for training our
ControlNet, we relied on data augmentation. In particular, we found random
cropping of the images and random 90 degree rotations beneficial. The training
process is highly resource-intensive, roughly 20 hours on a Nvidia A100 with
40GB of VRAM. A more detailed explanation of the ControlNet training protocol
is reported in supplementary Sec. D.

To evaluate the usefulness of SynCellFactory as a data augmentation strategy,
we used it in conjunction with the state-of-the-art deep learning model
EmbedTrack [20]. EmbedTrack uses CNNs to predict cell segmentation and
tracking jointly. For training EmbedTrack, we use the default settings
suggested by the authors; in particular, for each experiment, we trained the
model for 15 epochs using the CyclingAdam optimizer [15]. For testing, we used
the best-performing model with respect to the validation intersection over
union.

####  3.2.1 Tracking Metric

To evaluate the tracking performance of our trained models, we use the
official tracking accuracy measure (TRA) provided by the Cell Tracking
Challenge [21].

The TRA score is based on the concept of Acyclic Oriented Graph Matching AOGM.
Suppose we represent the tracking predictions as an acyclic-oriented graph. In
that case, we can measure the accuracy of a predicted graph by counting the
number of operations required to transform the prediction into the ground
truth graph. The operations considered are node insertion, node deletion, edge
insertion, and edge deletion. Formally, the TRA score is defined as:

$\text{TRA}=1-\frac{\text{min}\left(\text{AOGM},\text{AOGM}_{0}\right)}{\text{AOGM}_{0}},$
(1)

where AOGM represents the weighted sum of operations required to build the
prediction graph and $\text{AOGM}_{0}$ represents the weighted sum of
operations required to build the ground truth graph. Higher TRA scores
indicate better tracking performance.

data set | W/o SynCellFactory | With SynCellFactory | $\alpha$
---|---|---|---
Fluo-C2DL-Huh7 | 0.960 $\pm$ 0.002 |  0.966 $\pm$ 0.003 | 0.66
Fluo-C2DL-MSC | 0.624 $\pm$ 0.060 |  0.685 $\pm$ 0.060 | 0.50
DIC-C2DH-HeLa | 0.968 $\pm$ 0.001 |  0.974 $\pm$ 0.001 | 0.80
Fluo-N2DH-GOWT1 | 0.980 $\pm$ 0.003 |  0.989 $\pm$ 0.002 | 0.48
Fluo-N2DL-HeLa | 0.939 $\pm$ 0.002 |  0.981 $\pm$ 0.002 | 0.87
PhC-C2DL-PSC | 0.958 $\pm$ 0.001 |  0.960 $\pm$ 0.001 | 0.20
PhC-C2DH-U373 |  0.938 $\pm$ 0.008 | 0.935 $\pm$ 0.007 | 0.87
Table 1: Tracking Accuracy Measure TRA (higher is better) obtained with and without our SynCellFactory. The proposed data augmentation increases tracking accuracy for all but one of the CTC [41, 22] tested data sets. The $\alpha$ mixing coefficient has been set to the optimal value for each data set. Error bars indicate the standard deviation over three runs. data set |  $\Delta$ Splits |  $\Delta$ FP E |  $\Delta$ FN E
---|---|---|---
Fluo-C2DL-Huh7 | 0 | -1 | -1
Fluo-C2DL-MSC | 0 | -3 | -126
DIC-C2DH-HeLa | -3 | -2 | -10
Fluo-N2DH-GOWT1 | -13 | -3 | -35
Fluo-N2DL-HeLa | -10 | 0 | -17
PhC-C2DL-PSC | -780 | -14 | -230
PhC-C2DH-U373 | 0 | 0 | +3
Table 2: Difference in numbers of tracking mistakes with and without SynCellFactory. $\Delta$Split is the difference in splitting detection errors, the $\Delta$FP E represents the difference in false positive edge predictions, and $\Delta$FN E represents the difference in false negative edge predictions. Any value lower than zero signifies an improvement by means of SynCellFactory. The most notable improvements from our data augmentation are observed in resolving splitting errors and in reducing missing tracking assignments ($\Delta$FN E).  data set | EmbedTrack [20] | Ours
---|---|---
Fluo-C2DL-Huh7 | - | 0.920
Fluo-C2DL-MSC | 0.693 | 0.703
DIC-C2DH-HeLa | 0.934 | 0.943
Table 3: Comparison of the official CTC results. We improved the performance
of EmbedTrack [20] by using our data generation for all three data sets.
Originally, EmbedTrack [20] was inapplicable to Fluo-C2DL-Huh7 due to a
shortage of segmentation masks for training. The generated data sets provided
a sufficient amount of segmentation masks, enabling EmbedTrack to track
Fluo-C2DL-Huh7. Figure 4: Quantitative results according to the Tracking
Accuracy Measure TRA (higher is better). We trained the EmbedTrack model
without data augmentation (black square) and with different real and synthetic
training data mixing ratios $\alpha$. Here, one can observe that although
SynCellFactory augmentation positively impacts the TRA score in all but one of
the tested data sets, the correct choice of $\alpha$ is critical for the model
we benchmarked. Error bars indicate the standard deviation over three runs.

###  3.3 Quantitative results

In all but one of the test data sets, our SynCellFactory data augmentation
improved the tracking quality as measured by the TRA score; the results can be
found in Tab. 1. As shown in previous work using generative models for data
augmentation [14, 39], the ratio between synthetic and real data

$\alpha=\frac{\\#\text{syn-frames}}{\\#\text{syn-frames}+\\#\text{real-
frames}}$ (2)

in the training set is crucial. This is also true for the proposed methods; in
Fig. 4, we show the tracking accuracy achieved using different mixing ratios
$\alpha$. Our experiments showed two behaviors between fluorescence microscopy
data sets (denominated as Fluo-*) and all other microscopy modalities. In
fluorescence datasets, TRA scores typically increased with the mixing ratio up
to an optimal $0.5<\alpha<0.7$, then dropped, except in one dataset where
improvement continued steadily. In Phase Contrast (PhC-*) and Differential
Interference Contrast (DIC-*) experiments, most datasets initially showed a
performance drop at low $\alpha$, followed by consistent improvement at high
$\alpha\sim 0.8$, with one exhibiting an initial increase, a subsequent drop,
and then improvement. In addition to the TRA score, we also report additional
performance metrics in Tab. 2 based on the number of correct tracking
predictions.

####  3.3.1 CTC Results

In addition to our standard experimental setup, we tested our strategy on the
official cell tracking challenge evaluation data set. The CTC organizers
evaluate the submissions on a private ground truth. We submitted the results
for three data sets. Here, we trained SynCellFactory and the EmbedTrack model
on all available training data and using the optimal mixing ratio $\alpha$.
The results are presented in Tab. 3 and show an improvement on all three data
sets compared to those in [20].

###  3.4 Qualitative Results

Overall, the results for all tested data sets look realistic from the human
perspective. We now present a summary of the qualitative characteristics we
observed in our generated images.

Mitosis: One of the most challenging aspects of automatizing cell tracking is
correctly identifying cell splitting. The process by which cells replicate is
called mitosis, and most cell types exhibit a drastic change in their
appearance during this stage. This process happens in subsequent stages
(typically classified into Prophase, Metaphase, Anaphase, and Telophase), each
with its distinct visual appearance. It is, therefore, important to correctly
mimic the appearance of cells during mitosis in case visual deep learning
models are trained on the generated data sets. For our data sets, DIC-C2DH-
HeLa is the data set with the strongest visual indication of a splitting
event. In Fig. 5, we compare generated and real HeLa cells undergoing mitosis
from the DIC-C2DH data set.

Figure 5: Comparison of real (top) and simulated mitosis (bottom). Shown here
are the cell division at time $t$, plus three frames before and after. Key
differences include the simulated cell being brighter and lacking the gradual
brightness change seen in the real cell before the split. However, the model
accurately simulates cell contraction at $t-1$ and realistic artifacts at the
split. Post-split, the simulated daughter cells transition realistically from
$t+1$ to $t+3$.

Hallucinations: Hallucinations of cells and other artifacts are a weak point
of the proposed pipeline (see Fig. 6 for an example). We observe two main
types of hallucinations in the generated data sets. The first mode of
hallucination is the appearance of additional cells near image borders. These
cells are usually inpainted from the beginning of the video and remain
stationary throughout the video. We hypothesize that these hallucinations are
derived from the lack of cell detections at the border in the CTC data sets.
Indeed, cells partially cut out of the field of view are often not included in
the ground truth. The second type of hallucination occurs more sporadically
and consists of artifacts and debris in the generated frames seldomly promoted
to cells by our ControlNet. We observed such hallucinations more prevalently
in data sets with much background noise, like Fluo-C2DL-MSC and PhC-C2DH-U373.

Figure 6: Example of cell hallucinations close to the image boundary. The
network inpaints cells marked in red which do not match the conditioning shown
in green.

##  4 Conclusion

Despite the positive results, SynCellFactory is not without limitations, which
also serve as directions for future work.

The current motion module in SynCellFactory is simplistic, focusing on basic
cell movements. This limits its applicability in capturing the full spectrum
of biological diversity and complexity. Future iterations of SynCellFactory
could benefit from a more sophisticated motion module that can accurately
model a broader range of biological behaviors and interactions. Moreover, the
model’s performance declines in scenarios with high cell density or dramatic
changes in cell count. Addressing this limitation would enhance the model’s
utility in more complex biological environments. While SynCellFactory does not
produce segmentation ground truth, the availability of advanced segmentation
tools partially mitigates this issue. However, it remains a potential source
of inaccuracies and could be an area for future integration. SynCellFactory
can sample videos of arbitrary length, but there is a noticeable drop in
quality for extended sequences; in our experiments, we found that after 30
frames, the quality is reduced to the point of losing most resemblance with
the original data. We hypothesize that this is due to the training procedure,
in which the CN-Mov is only trained on frames pair and real data conditioning.
Future developments could focus on maintaining consistent quality across
varying video lengths.

In conclusion, SynCellFactory represents a significant step forward in the
field of biological data augmentation. Its ability to generate realistic and
diverse data sets holds great potential for advancing deep learning-based cell
tracking pipelines. While there are several areas for improvement, the
foundation laid by SynCellFactory offers a solid base for the concrete
application of generative AI in biomedical computer vision.

##  5 Code Availability

The code supporting this work is published and freely available at the
following URL: https://github.com/mosturm/SynCellFactory.

## References

  * Azizi et al. [2023] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet.  Synthetic data from diffusion models improves imagenet classification.  _arXiv preprint arXiv:2304.08466_ , 2023.
  * Brock et al. [2019] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale GAN training for high fidelity natural image synthesis.  In _7th International Conference on Learning Representations, ICLR 2019_ , 2019.
  * Burke and Orth [2016] Russell T Burke and James D Orth.  Through the looking glass: Time-lapse microscopy and longitudinal tracking of single cells to study anti-cancer therapeutics.  _Journal of visualized experiments: JoVE_ , 2016.
  * Bähr et al. [2021] Dennis Bähr, Dennis Eschweiler, Anuk Bhattacharyya, Daniel Moreno-Andrés, Wolfram Antonin, and Johannes Stegmaier.  Cellcyclegan: Spatiotemporal microscopy image synthesis of cell populations using statistical shape models and conditional gans.  In _2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI)_ , 2021.
  * Chen et al. [2021] Richard J. Chen, Ming Y. Lu, Tiffany Y. Chen, Drew F. K. Williamson, and Faisal Mahmood.  Synthetic data in machine learning for medicine and healthcare.  _Nature Biomedical Engineering_ , 2021.
  * Collins et al. [2018] John L. Collins, Bart van Knippenberg, Kai Ding, and Alexander V. Kofman.  Time-lapse microscopy.  In _Cell Culture_ , chapter 3. IntechOpen, 2018.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Dorjsembe et al. [2023] Zolnamar Dorjsembe, Hsing-Kuo Pao, Sodtavilan Odonchimed, and Furen Xiao.  Conditional diffusion models for semantic 3d medical image synthesis.  _arXiv preprint arXiv:2305.18453_ , 2023.
  * Du et al. [2023] Yunhao Du, Zhicheng Zhao, Yang Song, Yanyun Zhao, Fei Su, Tao Gong, and Hongying Meng.  Strongsort: Make deepsort great again, 2023.
  * Fernandez et al. [2022] Virginia Fernandez, Walter Hugo Lopez Pinaya, Pedro Borges, Petru-Daniel Tudosiu, Mark S Graham, Tom Vercauteren, and M Jorge Cardoso.  Can segmentation models be trained with fully synthetically generated data?  In _International Workshop on Simulation and Synthesis in Medical Imaging_ , pages 79–90. Springer, 2022.
  * Gowal et al. [2021] Sven Gowal, Sylvestre-Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A Mann.  Improving robustness using generated data.  _Advances in Neural Information Processing Systems_ , 2021.
  * Hassan et al. [2023] Saif Hassan, Ghulam Mujtaba, Asif Rajput, and Noureen Fatima.  Multi-object tracking: a systematic literature review.  _Multimedia Tools and Applications_ , 2023.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_ , 2016.
  * He et al. [2023] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and XIAOJUAN QI.  IS SYNTHETIC DATA FROM GENERATIVE MODELS READY FOR IMAGE RECOGNITION?  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Kingma and Ba [2015] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.
  * Kozubek [2020] Michal Kozubek.  When deep learning meets cell image synthesis.  _Cytometry Part A_ , 2020.
  * Liu et al. [2021] Quan Liu, Isabella M Gaeta, Mengyang Zhao, Ruining Deng, Aadarsh Jha, Bryan A Millis, Anita Mahadevan-Jansen, Matthew J Tyska, and Yuankai Huo.  Asist: annotation-free synthetic instance segmentation and tracking by adversarial simulations.  _Computers in biology and medicine_ , 2021.
  * Lockett et al. [1998] Stephen J. Lockett, Damir Sudar, Curtis T. Thompson, Dan Pinkel, and Joe W. Gray.  Efficient, interactive, and three-dimensional segmentation of cell nuclei in thick tissue sections.  _Cytometry_ , 1998.
  * Loewke and Pera [2011] Kevin E. Loewke and Renee A. Reijo Pera.  _The Role of Time-Lapse Microscopy in Stem Cell Research and Therapy_ , pages 181–191.  2011\.
  * Löffler and Mikut [2022] Katharina Löffler and Ralf Mikut.  Embedtrack—simultaneous cell segmentation and tracking through learning offsets and clustering bandwidths.  _IEEE Access_ , 10:77147–77157, 2022.
  * Matula et al. [2015] Pavel Matula, Martin Maška, Dmitry V. Sorokin, Petr Matula, Carlos Ortiz-de Solórzano, and Michal Kozubek.  Cell tracking accuracy measurement based on comparison of acyclic oriented graphs.  _PLOS ONE_ , 2015.
  * Maška et al. [2023] Martin Maška, Vladimír Ulman, Pablo Delgado-Rodriguez, Estibaliz Gómez-de Mariscal, Tereza Nečasová, et al.  The cell tracking challenge: 10 years of objective benchmarking.  _Nature Methods_ , 2023.
  * Meinhardt et al. [2022] Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer.  Trackformer: Multi-object tracking with transformers.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022.
  * Pachitariu and Stringer [2022] M. Pachitariu and C. Stringer.  Cellpose 2.0: how to train your own model.  _Nature Methods_ , 2022.
  * Pinaya et al. [2022] Walter HL Pinaya, Petru-Daniel Tudosiu, Jessica Dafflon, Pedro F Da Costa, Virginia Fernandez, Parashkev Nachev, Sebastien Ourselin, and M Jorge Cardoso.  Brain imaging generation with latent diffusion models.  In _MICCAI Workshop on Deep Generative Models_ , 2022.
  * Pinaya et al. [2023] Walter HL Pinaya, Mark S Graham, Eric Kerfoot, Petru-Daniel Tudosiu, Jessica Dafflon, Virginia Fernandez, Pedro Sanchez, Julia Wolleb, Pedro F da Costa, Ashay Patel, et al.  Generative ai for medical imaging: extending the monai framework.  _arXiv preprint arXiv:2307.15208_ , 2023.
  * Rebuffi et al. [2021] Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy A Mann.  Data augmentation can improve robustness.  _Advances in Neural Information Processing Systems_ , 34:29935–29948, 2021.
  * Rombach et al. [2021] Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  Stable diffusion 1.5.  https://github.com/runwayml/stable-diffusion, 2022.
  * Sariyildiz et al. [2023] Mert Bulent Sariyildiz, Karteek Alahari, Diane Larlus, and Yannis Kalantidis.  Fake it till you make it: Learning transferable representations from synthetic imagenet clones.  In _CVPR 2023–IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Scalbert et al. [2019] Marin Scalbert, Florent Couzinie-Devy, and Riadh Fezzani.  Generic isolated cell image generator.  _Cytometry Part A_ , 2019.
  * Schwartz et al. [2023] Morgan Sarah Schwartz, Erick Moen, Geneva Miller, Tom Dougherty, Enrico Borba, Rachel Ding, William Graf, Edward Pao, and David Van Valen.  Caliban: Accurate cell tracking and lineage construction in live-cell imaging experiments with deep learning.  _bioRxiv_ , 2023.
  * Serna-Aguilera et al. [2022] Manuel Serna-Aguilera, Khoa Luu, Nathaniel Harris, and Min Zou.  Neural cell video synthesis via optical-flow diffusion, 2022.
  * Shorten and Khoshgoftaar [2019] Connor Shorten and Taghi M. Khoshgoftaar.  A survey on image data augmentation for deep learning.  _Journal of Big Data_ , 2019.
  * Simard et al. [2003] Patrice Y Simard, David Steinkraus, John C Platt, et al.  Best practices for convolutional neural networks applied to visual document analysis.  In _Icdar_. Edinburgh, 2003.
  * Stringer et al. [2021] C. Stringer, T. Wang, M. Michaelos, and M. Pachitariu.  Cellpose: a generalist algorithm for cellular segmentation.  _Nature Methods_ , 2021.
  * Sun et al. [2022] Li Sun, Junxiang Chen, Yanwu Xu, Mingming Gong, Ke Yu, and Kayhan Batmanghelich.  Hierarchical amortized gan for 3d high resolution medical image synthesis.  _IEEE journal of biomedical and health informatics_ , 2022.
  * Svoboda and Ulman [2017] David Svoboda and Vladimír Ulman.  Mitogen: A framework for generating 3d synthetic time-lapse sequences of cell populations in fluorescence microscopy.  _IEEE Transactions on Medical Imaging_ , 2017.
  * Trabucco et al. [2023] Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov.  Effective data augmentation with diffusion models.  _CoRR_ , 2023.
  * Tran et al. [2017] Toan Tran, Trung Pham, Gustavo Carneiro, Lyle Palmer, and Ian Reid.  A bayesian data augmentation approach for learning deep models.  In _Advances in Neural Information Processing Systems_ , 2017.
  * Ulman et al. [2017] Vladimír Ulman, Martin Maška, Klas E G Magnusson, Olaf Ronneberger, Carsten Haubold, Nathalie Harder, Pavel Matula, Petr Matula, David Svoboda, Miroslav Radojevic, Ihor Smal, Karl Rohr, Joakim Jaldén, Helen M Blau, Oleh Dzyubachyk, Boudewijn Lelieveldt, Pengdong Xiao, Yuexiang Li, Siu-Yeung Cho, Alexandre C Dufour, Jean-Christophe Olivo-Marin, Constantino C Reyes-Aldasoro, Jose A Solis-Lemus, Robert Bensch, Thomas Brox, Johannes Stegmaier, Ralf Mikut, Steffen Wolf, Fred A Hamprecht, Tiago Esteves, Pedro Quelhas, Ömer Demirel, Lars Malmström, Florian Jug, Pavel Tomancak, Erik Meijering, Arrate Muñoz-Barrutia, Michal Kozubek, and Carlos Ortiz-de Solorzano.  An objective comparison of cell-tracking algorithms.  _Nature Methods_ , 2017.
  * Wiesner et al. [2019] David Wiesner, Tereza Nečasová, and David Svoboda.  On generative modeling of cell shape using 3d gans.  In _Image Analysis and Processing – ICIAP 2019_ , 2019.
  * Zargari et al. [2023] Abolfazl Zargari, Gerrald A. Lodewijk, Najmeh Mashhadi, Nathan Cook, Celine W. Neudorf, Kimiasadat Araghbidikashani, Robert Hays, Sayaka Kozuki, Stefany Rubio, Eva Hrabeta-Robinson, Angela Brooks, Lindsay Hinck, and S. Ali Shariati.  Deepsea is an efficient deep-learning model for single-cell segmentation and tracking in time-lapse microscopy.  _Cell Reports Methods_ , 2023.
  * Zhang and Agrawala [2023] Lvmin Zhang and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  _CoRR_ , 2023.
  * Zhao and Murphy [2007] Ting Zhao and Robert F. Murphy.  Automated learning of generative models for subcellular location: Building blocks for systems biology.  _Cytometry Part A_ , 2007.
  * Zheng et al. [2017] Zhedong Zheng, Liang Zheng, and Yi Yang.  Unlabeled samples generated by gan improve the person re-identification baseline in vitro.  In _2017 IEEE International Conference on Computer Vision (ICCV)_ , 2017.
  * Zhu et al. [2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2223–2232, 2017.

Supplementary Material

##  Appendix A Motion Model

###  A.1 Statistics

In our study, we derive the motion module’s key statistics from both the
tracking ground truth and a limited set of segmentation ground truth data.

We use all segmentation masks available within a single data set to calculate
the mean $A_{c}$ and standard deviation $\sigma_{A}$ of pixel counts per
segmentation mask.

We estimate our cell displacement distribution given the tracking ground truth
by computing the displacements of individual cells between consecutive
timeframes. The distribution of all calculated displacements is then fitted
using a gamma distribution $G$ with respect to its parameters
$\alpha_{d},\Theta_{d}$:

$G(x;\alpha_{d},\Theta_{d})=\frac{x^{\alpha_{d}-1}\cdot
e^{-\frac{x}{\Theta_{d}}}}{\Theta_{d}^{\alpha_{d}}\cdot\Gamma(\alpha_{d})}$
(3)

Here, $\alpha_{d}$ represents the shape parameter, $\Theta_{d}$ the scale
parameter, and $\Gamma(\alpha_{d})$ denotes the gamma function. The choice of
a gamma distribution is empirically driven by its adaptability in modeling
data with skewed distributions.

We can generate new, plausible samples of cell population timelapses with
these statistical parameters determined. These samples take into account
variations in both cell area and motion characteristics.

###  A.2 Simulation

Initialisation: In the first timeframe we sample $n$ cell centers at random
positions $(x_{i},y_{i})\in(0,1)$ for $i\in\\{1,2,\ldots,n\\}$ with each cell
having an area $A_{i}\sim\mathcal{N}(A_{c},\frac{\sigma_{A}}{10})$. We model
each cell as a 2D disk with radius $r_{i}=\sqrt{\frac{A_{i}}{\pi}}$.

Movement of cells: In our simulation, cell movements are modeled as a random
walk.

Let $\mathbf{p}_{i,t}=(x_{i,t},y_{i,t})$ represent the position vector of cell
$i$ at timeframe $t$. The position at the next timeframe $t+1$ is updated
using:

$\displaystyle\phi$ $\displaystyle\sim\text{Uniform}(0,2\pi)$ (4)
$\displaystyle m_{v}$ $\displaystyle\sim G(\alpha_{d},\Theta_{d})$ (5)
$\displaystyle\mathbf{p}_{i,t+1}$
$\displaystyle=\mathbf{p}_{i,t}+m_{v}\begin{pmatrix}\cos(\phi)\\\
\sin(\phi)\end{pmatrix}$ (6)

Each cell has a splitting probability $p_{\text{split}}$ at each timeframe.
Upon splitting, a cell divides into two daughter cells along a random axis.
The position vectors of the daughter cells, denoted as $\mathbf{p}_{i,1}$ and
$\mathbf{p}_{i,2}$, are given by:

$\mathbf{p}_{i,1/2}=\mathbf{p}_{i}\pm d\cdot r_{i}\begin{pmatrix}\cos(\phi)\\\
\sin(\phi)\end{pmatrix}$ (7)

Here, $d\sim\text{Uniform}(0.7,0.8)$ introduces a slight randomness in the
splitting process, and $r_{i}$ is a scaling factor.

Resolving overlaps: The resolution of overlaps in our model is repeated in
successive iterations, until no overlaps remain.

To resolve the overlap of two circles, we first define the repulsion vector
$\mathbf{v}_{\text{repulsion}}$ between two overlapping circles $c_{i}$ and
$c_{j}$ with center coordinates $(x_{i},y_{i})$ and $(x_{j},y_{j})$ as
follows:

$\mathbf{v}_{\text{repulsion}}=\begin{pmatrix}x_{i}-x_{j}\\\
y_{i}-y_{j}\end{pmatrix}$ (8)

The angle of repulsion $\theta$ is calculated using the arctan function:

$\theta=\arctan((\mathbf{v}_{\text{repulsion}})_{y},(\mathbf{v}_{\text{repulsion}})_{x})+\theta_{\epsilon},$
(9)

where $\theta_{\epsilon}\sim\text{Uniform}(-0.1,0.1)$ is a small perturbation
to the angle to prevent infinite loops during the overlap resolution process.

Next, we calculate the repulsion displacement $F_{i,j}$, based on the overlap
distance between the circles, scaled by the average of their radii to account
for potentially larger overlap areas in bigger circles. The formula is given
as:

$F_{i,j}=\left(\left(r_{i}+r_{j}-\|\mathbf{v}_{\text{repulsion}}\|\right)\cdot\frac{r_{i}+r_{j}}{2}\right)\cdot\left(1+\epsilon\right)$
(10)

where $\epsilon\sim\text{Uniform}(0,0.2)$ is introduced in each iteration with
the same reasoning as for $\theta_{\epsilon}$.

The positions of the cells are then updated as follows:

$\displaystyle\mathbf{p}_{i}$
$\displaystyle=\mathbf{p}_{i}+F_{i,j}\cdot\begin{pmatrix}\cos(\theta)\\\
\sin(\theta)\end{pmatrix}$ (11) $\displaystyle\mathbf{p}_{j}$
$\displaystyle=\mathbf{p}_{j}-F_{i,j}\cdot\begin{pmatrix}\cos(\theta)\\\
\sin(\theta)\end{pmatrix}$ (12)

##  Appendix B ControlNet Conditioning

In Fig. 7 we showcase an example of the construction of position and movement
maps for training.

Figure 7: Construction of training pairs [$c_{\text{img}}$ (green squared
images), $i_{\text{tgt}}$ (blue squared images)] from real cell videos. The
network is then trained with [$c_{\text{img}}$, $i_{\text{tgt}}$] as depicted
in Fig. 3.
Top: Construction of a position map for training. Given a real timeframe, we
know the cell positions from the corresponding detection ground truth (GT). We
then transform the detection ground truth into our color-coded position map as
described in Sec. 2.2.1.
Bottom: Construction of a movement map for training. The goal is that the
network learns to produce a realistic-looking timeframe $t$, given the
appearance of timeframe $t+1$ and individual cell displacements. We construct
the movement map in the following way: We overlay timeframe $t+1$ with a
position map from timeframe $t$ and displacement vectors computed from the
tracking ground truth. With this setup, the network can learn to draw a
realistic timeframe $t$ by learning where to draw cells (position map of $t$),
where they were previously located in $t+1$ (displacement vectors), and how
they looked like in $t+1$.

##  Appendix C Segmentation Pseudo Ground Truth

The segmentation correction step is illustrated in Fig. 8.

Figure 8: Calculation of approximate segmentation masks for our generated
time-frames. We use finetuned Cellpose (CP) [36, 24] segmentation models to
predict segmentation masks. These masks are corrected using the corresponding
position map; for more details, see Sec. 2.4.

##  Appendix D ControlNet Training

To address the challenges of limited training data we found it beneficial to
choose a sequential training procedure, training in total four consecutive
ControlNets per dataset.

We start our training procedure by training a base model (BM) of CN-pos on
randomly selected patches of $c_{\text{img}}$ and $i_{\text{tgt}}$ measuring
$h/2\times w/2$. The initial phase focuses on enabling ControlNet to learn the
correlation between position maps and cell appearance.

After a fixed number of iteration steps (see Tab. 4) the training procedure of
CN-pos is stopped and we start training an initial version of CN-mov. This
phase involves using cropped movement maps and corresponding target images of
size $h/2\times w/2$, similarly to the base model of CN-pos.

CN-mov is initialized with the pre-trained weights from the CN-pos base model,
ensuring continuity in the learning process.

The final step in our training methodology involves fine-tuning both base
model ControlNets. This is achieved through limited optimization steps
conducted on the full-sized $h\times w$ training timeframes. After fine-
tuning, the full model (FM) ControlNets are optimized and ready to integrate
into the sampling process.

In Tab. 4 is a list of training parameters used to train our ControlNets.
Numbers for one A100 40 GB GPU.

Total Trainable Parameters | 1.2 B
---|---
Latent Representation | shape(z) = (1,4,64,64)
Diffusion Steps | 50
Optimizer | AdamW
Learning Rate | 5e-6
Batch Size | 4
Training BM CN-pos $n_{\text{cell}}:10^{2};10^{3}$ | 30000; 60000 Steps
Training BM CN-mov $n_{\text{cell}}:10^{2};10^{3}$ | 10000; 20000 Steps
Finetuning FM CN-pos $n_{\text{cell}}:10^{2};10^{3}$ | 3000; 7000 Steps
Finetuning FM CN-mov $n_{\text{cell}}:10^{2};10^{3}$ | 3000; 7000 Steps
Sampling Time (12 Timeframes) |  $\sim$ 3 min
Table 4: Parameters used for ControlNet training and sampling.
$n_{\text{cell}}$ denotes the number of cells expected in a single timeframe
and determines the number of optimization steps. For example, if
$n_{\text{cell}}$ is in the order of $10^{1}-10^{2}$, we start our training
procedure by training the CN-pos base model (BM) for 30000 optimizer steps. If
$n_{\text{cell}}$ is in the order of $10^{2}-10^{3}$, we train the CN-pos base
model for 60000 optimizer steps. The optimizer steps for consecutive
ControlNet training are denoted in the same way in this table.

##  Appendix E Additional Generated Time-Lapse

In Fig. 9 \- Fig. 15, we showcase more examples of generated time-lapses and
the corresponding tracking ground truth as simulated by the motion module.
Generated time-frames are shown in the top row, and tracking ground truth is
shown in the bottom row.

Figure 9: Fluo-C2DL-Huh7 Figure 10: Fluo-C2DL-MSC Figure 11: DIC-C2DH-HeLa
Figure 12: Fluo-N2DH-GOWT1 Figure 13: Fluo-N2DL-HeLa Figure 14: PhC-C2DL-PSC
Figure 15: PhC-C2DH-U373


preorders are a significant improvement on preorder $\preccurlyeq_{0}$.

It seems natural to define an ordering on $X$ by taking limits of the
positive-radius preorders $\preccurlyeq_{r}$ as $r\to 0$. As any binary
relation can be viewed as a subset of the Cartesian product $X\times X$, where
$(x,x^{\prime})\in\mathord{\preccurlyeq_{r}}$ precisely when
$x\preccurlyeq_{r}x^{\prime}$, we define some candidate limiting orderings
using set-theoretic limits of the net $(\preccurlyeq_{r})_{r>0}$. The
corresponding limit set need not be a preorder in general, but we show that
certain set-theoretic limits do always yield a preorder.

Indeed, the _set-theoretic limits inferior and superior_ of a net
$(A_{r})_{r>0}$ of subsets of $X$ are defined by

$\displaystyle\liminf_{r\to 0}A_{r}$
$\displaystyle\coloneqq\bigcup_{R>0}\bigcap_{r<R}A_{r}=\left\\{y\in
X\,\middle|\,y\in A_{r}\text{ for all $r<R(y)$}\right\\},$
$\displaystyle\limsup_{r\to 0}A_{r}$
$\displaystyle\coloneqq\bigcap_{R>0}\bigcup_{r<R}A_{r}=\left\\{y\in
X\,\middle|\,\text{for some null sequence $(r_{n})_{n\in\mathbb{N}}$, }y\in
A_{r_{n}}\text{ for all $n\in\mathbb{N}$}\right\\},$

and the _Kuratowski lower and upper limits_ of $(A_{r})_{r>0}$ are defined by

$\displaystyle\operatorname*{Li}_{r\to 0}A_{r}$
$\displaystyle\coloneqq\left\\{y\in X\,\middle|\,y\text{ is a limit point of
the net }(A_{r})_{r>0}\right\\},$ $\displaystyle\operatorname*{Ls}_{r\to
0}A_{r}$ $\displaystyle\coloneqq\left\\{y\in X\,\middle|\,y\text{ is a cluster
point of the net }(A_{r})_{r>0}\right\\}.$

The following is a useful equivalent characterisation of the Kuratowski
limits:

######  Lemma B.1 (Beer, 1993, Lemmas 5.2.7 and 5.2.8).

Let $X$ be any metric space and let $(A_{r})_{r>0}$ be a net of subsets of
$X$.

  1. (a)

$x\in\operatorname*{Li}_{r\to 0}A_{r}$ if and only if there exists a net
$(x_{r})_{r>0}$ converging to $x$ with $x_{r}\in A_{r}$.

  2. (b)

$x\in\operatorname*{Ls}_{r\to 0}A_{r}$ if and only if there exists a
decreasing null sequence $(r_{n})_{n\in\mathbb{N}}$ and a sequence
$(x_{r_{n}})_{n\in\mathbb{N}}$ converging to $x$ with $x_{r_{n}}\in
A_{r_{n}}$.

The set limits described above give four different approaches to taking the
limit of the sets $(\preccurlyeq_{r})_{r>0}$, which we denote

$\displaystyle\mathord{\preccurlyeq_{0}^{\liminf}}$
$\displaystyle\coloneqq\liminf_{r\to 0}\mathord{\preccurlyeq_{r}},$
$\displaystyle\mathord{\preccurlyeq_{0}^{\limsup}}$
$\displaystyle\coloneqq\limsup_{r\to 0}\mathord{\preccurlyeq_{r}},$
$\displaystyle\mathord{\preccurlyeq_{0}^{\operatorname*{Li}}}$
$\displaystyle\coloneqq\operatorname*{Li}_{r\to 0}\mathord{\preccurlyeq_{r}},$
$\displaystyle\mathord{\preccurlyeq_{0}^{\operatorname*{Ls}}}$
$\displaystyle\coloneqq\operatorname*{Ls}_{r\to 0}\mathord{\preccurlyeq_{r}}.$

######  Proposition B.2.

Let $X$ be a metric space and let $\mu\in\mathscr{P}(X)$.

  1. (a)

$\preccurlyeq_{0}^{\liminf}$ is a preorder;

  2. (b)

$\mathord{\preccurlyeq_{0}^{\liminf}}$ is a subset of
$\mathord{\preccurlyeq_{0}}$;

  3. (c)

$x\preccurlyeq_{0}^{\liminf}x^{\prime}\implies x\preccurlyeq_{0}x^{\prime}$.

###### Proof.

For (a), it is routine to check that $\preccurlyeq_{0}^{\liminf}$ is a
preorder: reflexivity is obvious, and if $x\preccurlyeq_{0}^{\liminf}y$ and
$y\preccurlyeq_{0}^{\liminf}z$ then there exists $R>0$ such that, for all
$r<R$, $x\preccurlyeq_{r}y$ and $y\preccurlyeq_{r}z$, giving
$x\preccurlyeq_{r}z$ by transitivity of $\preccurlyeq_{r}$.

For (b), $(x,x^{\prime})\in\mathord{\preccurlyeq_{0}^{\liminf}}$ implies that,
for some $R>0$ and all $r<R$, $x\preccurlyeq_{r}x^{\prime}$. Hence,

$\limsup_{r\to 0}\frac{\mu(B_{r}(x))}{\mu(B_{r}(x^{\prime}))}\leqslant 1,$

so $x\preccurlyeq_{0}x^{\prime}$ by definition. Claim (c) follows immediately
from (b): if $(x,x^{\prime})\in\mathord{\preccurlyeq_{0}^{\liminf}}$, then
$(x,x^{\prime})\in\mathord{\preccurlyeq_{0}}$, so
$x\preccurlyeq_{0}x^{\prime}$. ∎

As a consequence of (b), any $\preccurlyeq_{0}$-antichain is also a
$\preccurlyeq_{0}^{\liminf}$-antichain. Hence, Theorem 5.11 gives an example
of a countable dense $\preccurlyeq_{0}^{\liminf}$-antichain; this demonstrates
that $\preccurlyeq_{0}^{\liminf}$ does not have better behaviour in this
regard than $\preccurlyeq_{0}$.

The set-theoretic ordering $\preccurlyeq_{0}^{\liminf}$ can also be criticised
as unnecessarily strict in cases where $x^{\prime}\prec_{0}x$ for any $r>0$,
but

$\limsup_{r\to 0}\frac{\mu(B_{r}(x))}{\mu(B_{r}(x^{\prime}))}=1.$

Example 4.9 gives a measure where the $\preccurlyeq_{0}^{\liminf}$-greatest
elements and the $\preccurlyeq_{0}$-greatest elements differ: under
$\preccurlyeq_{0}^{\liminf}$ only $+1$ is greatest, whereas both $-1$ and $+1$
are $\preccurlyeq_{0}$-greatest. While $\preccurlyeq_{0}^{\liminf}$-greatest
elements are reasonable candidates for modes, they do not seem to correspond
exactly to any of the established definitions of modes. To be more precise,
while Proposition B.2(b) implies that they are always weak modes, it is not
clear whether or not they are strong modes, and not all weak modes are
$\preccurlyeq_{0}^{\liminf}$-greatest.

######  Example B.3 ($\preccurlyeq_{0}^{\limsup}$ is not necessarily
transitive).

The essential idea is even if $x\preccurlyeq_{r_{n}}y$ for some null sequence
$(r_{n})_{n\in\mathbb{N}}$, and $y\preccurlyeq_{r_{n}^{\prime}}z$ for some
null sequence $(r_{n}^{\prime})_{n\in\mathbb{N}}$, it is possible that
$x\not\preccurlyeq_{r}z$ for any $r>0$. For a concrete example of this
situation, let

$X\coloneqq\bigl{\\{}-2\pm
2^{-3n+2}\,\big{|}\,n\in\mathbb{N}\bigr{\\}}\cup\bigl{\\{}2\pm
2^{-3n+2}\,\big{|}\,n\in\mathbb{N}\bigr{\\}}\cup\bigl{\\{}\pm
2^{-3n+2}\,\big{|}\,n\in\mathbb{N}\bigr{\\}}$

with its usual Euclidean metric. Define the “target RCDFs”

$\displaystyle f(2^{-3n+2})$ $\displaystyle\coloneqq
2^{-3n+2},~{}~{}~{}~{}~{}g(2^{-3n+2})\coloneqq 2^{-3n+1},$ $\displaystyle
h(2^{-3n+2})$ $\displaystyle\coloneqq\begin{cases}2^{-3n+3},&\text{if $n$ is
odd,}\\\ 2^{-3n},&\text{if $n$ is even,}\end{cases}$

and let

$\displaystyle\mu$
$\displaystyle\coloneqq\frac{1}{Z}\sum_{n\in\mathbb{N}}\frac{1}{2}\bigl{(}f(2^{-3n+2})-f(2^{-3n-1})\bigr{)}\bigl{(}\delta_{-2-2^{-3n+2}}+\delta_{-2+2^{-3n+2}}\bigr{)}$
$\displaystyle\phantom{\coloneqq}\quad+\frac{1}{Z}\sum_{n\in\mathbb{N}}\frac{1}{2}\bigl{(}g(2^{-3n+2})-g(2^{-3n-1})\bigr{)}\bigl{(}\delta_{2-2^{-3n+2}}+\delta_{2+2^{-3n-1}}\bigr{)}$
$\displaystyle\phantom{\coloneqq}\quad+\frac{1}{Z}\sum_{n\in\mathbb{N}}\frac{1}{2}\bigl{(}h(2^{-3n+2})-h(2^{-3n-1})\bigr{)}\bigl{(}\delta_{-2^{-3n+2}}+\delta_{2^{-3n-1}}\bigr{)},$

where $Z$ is a normalisation constant chosen to ensure that
$\mu\in\mathscr{P}(X)$.

The construction of $\mu$ ensures that the RCDFs at $-2$, $+2$ and $0$ are
$\frac{1}{Z}f(r)$, $\frac{1}{Z}g(r)$ and $\frac{1}{Z}h(r)$ for $r\leqslant
2^{-1}$. Then

$\displaystyle\frac{\mu(B_{2^{-3n+2}}(-2))}{\mu(B_{2^{-3n+2}}(0))}$
$\displaystyle=\begin{cases}2^{-1},&\text{if $n$ is odd,}\\\ 2^{2},&\text{if
$n$ is even,}\end{cases}$
$\displaystyle\frac{\mu(B_{2^{-3n+2}}(0))}{\mu(B_{2^{-3n+2}}(+2))}$
$\displaystyle=\begin{cases}2^{2}&\text{if $n$ is odd},\\\ 2^{-1}&\text{if $n$
is even,}\end{cases}$
$\displaystyle\frac{\mu(B_{2^{-3n+2}}(-2))}{\mu(B_{2^{-3n+2}}(+2))}$
$\displaystyle=2.$

It follows that $-2\asymp_{0}^{\limsup}0$, because there are null sequences
$(r_{n})_{n\in\mathbb{N}}$ such that $-2\preccurlyeq_{r_{n}}0$ and vice versa;
the same argument shows that $0\asymp_{0}^{\limsup}+2$. But $-2\prec_{r}2$ for
any $r>0$, and hence $-2\not\preccurlyeq_{0}^{\limsup}2$. This violates
transitivity.

######  Example B.4 ($\preccurlyeq_{0}^{\operatorname*{Li}}$ and
$\preccurlyeq_{0}^{\operatorname*{Ls}}$ are not necessarily transitive).

Let $\mu\in\mathscr{P}(\mathbb{R})$ be the measure with Lebesgue density
$\rho(x)\coloneqq\mathds{1}\big{[}x\in[0,1]\big{]}$. We first verify that:

  1. (a)

$x\preccurlyeq_{0}^{\operatorname*{Li}}1$ for any $x\in\mathbb{R}$;

  2. (b)

$1\preccurlyeq_{0}^{\operatorname*{Li}}x$ for any $x\in\mathbb{R}$;

  3. (c)

$\nicefrac{{1}}{{2}}\not\preccurlyeq_{0}^{\operatorname*{Ls}}x$ for any
$x\in\mathbb{R}\setminus[0,1]$; and

  4. (d)

$\nicefrac{{1}}{{2}}\not\preccurlyeq_{0}^{\operatorname*{Li}}x$ for any
$x\in\mathbb{R}\setminus[0,1]$.

For (a), observe that $x\preccurlyeq_{r}1-r$ for all small $r$, so
$(x,1-r)\in\mathord{\preccurlyeq_{r}}$. Hence,
$(x,1)\in\mathord{\preccurlyeq_{0}^{\operatorname*{Li}}}$ by Lemma B.1.

For (b), use that $1+r\preccurlyeq_{r}x$ for all small $r$, so
$(1+r,x)\in\mathord{\preccurlyeq_{r}}$. This implies that
$(1,x)\in\mathord{\preccurlyeq_{0}^{\operatorname*{Li}}}$.

For (c), suppose that
$(x_{r_{n}}^{\prime},x_{r_{n}})\to(\nicefrac{{1}}{{2}},x)$, and
$(x_{r_{n}}^{\prime},x_{r_{n}})\in\mathord{\preccurlyeq_{r_{n}}}$. Let
$\varepsilon=\min\\{\lvert x\rvert,\lvert x-1\rvert\\}>0$. There exists
$N_{1}\in\mathbb{N}$ such that, for all $n\geqslant N_{1}$, $\min\\{\lvert
x_{r_{n}}\rvert,\lvert x_{r_{n}}-1\rvert\\}>\nicefrac{{\varepsilon}}{{2}}$. As
$(r_{n})_{n\in\mathbb{N}}$ is a decreasing null sequence, there exists
$N_{2}\in\mathbb{N}$ such that, for all $n\geqslant N_{2}$,
$r_{n}<\nicefrac{{\varepsilon}}{{2}}$. Picking
$N\coloneqq\max\\{N_{1},N_{2}\\}$, we have $\mu(B_{r_{n}}(x_{r_{n}}))=0$ for
$n\geqslant N$, because $B_{r_{n}}(x_{r_{n}})\cap[0,1]=\varnothing$. It is
easy to see that if $x_{r_{n}}^{\prime}\to\nicefrac{{1}}{{2}}$, then for
sufficiently large $n$ $\mu(B_{r_{n}}(x_{r_{n}}^{\prime}))>0$. Hence, for all
sufficiently large $n$, $x_{r_{n}}^{\prime}\not\preccurlyeq_{r_{n}}x_{r_{n}}$.
This is a contradiction.

Claim (d) is a corollary of (c), because
$\mathord{\preccurlyeq_{0}^{\operatorname*{Li}}}\subseteq\mathord{\preccurlyeq_{0}^{\operatorname*{Ls}}}$.

Now we prove that $\preccurlyeq_{0}^{\operatorname*{Li}}$ and
$\preccurlyeq_{0}^{\operatorname*{Ls}}$ are not transitive. Suppose for
contradiction that they are: then (a) and (b) imply that every point
$x\in\mathbb{R}$ is equivalent to $1$, and so all points in $\mathbb{R}$ are
equivalent by transitivity. As
$\mathord{\preccurlyeq_{0}^{\operatorname*{Li}}}\subseteq\mathord{\preccurlyeq_{0}^{\operatorname*{Ls}}}$,
this implies that all points are also
$\preccurlyeq_{0}^{\operatorname*{Ls}}$-equivalent. However, (c) and (d) show
that not all points in $\mathbb{R}$ are
$\preccurlyeq_{0}^{\operatorname*{Ls}}$-equivalent or
$\preccurlyeq_{0}^{\operatorname*{Li}}$-equivalent.

## Acknowledgements

This work has been partially supported by the Deutsche Forschungsgemeinschaft
through project 415980428. HL is supported by the Warwick Mathematics
Institute Centre for Doctoral Training and gratefully acknowledges funding
from the University of Warwick and the UK Engineering and Physical Sciences
Research Council (Grant number: EP/W524645/1). The authors would like to thank
David Bate, Adam Epstein, Ilja Klebanov, Florian Theil, and Philipp Wacker for
helpful discussions.

## References

  * Agapiou et al. (2018) S. Agapiou, M. Burger, M. Dashti, and T. Helin.  Sparsity-promoting and edge-preserving maximum a posteriori estimators in non-parametric Bayesian inverse problems.  _Inverse Probl._ , 34(4):045002, 37pp., 2018.  10.1088/1361-6420/aaacac.
  * Aliprantis and Border (2006) C. D. Aliprantis and K. C. Border.  _Infinite Dimensional Analysis: A Hitchhiker’s Guide_.  Springer, Berlin, third edition, 2006.  10.1007/3-540-29587-9.
  * Anderson (1955) T. W. Anderson.  The integral of a symmetric unimodal function over a symmetric convex set and some probability inequalities.  _Proc. Amer. Math. Soc._ , 6:170–176, 1955.  10.2307/2032333.
  * Ayanbayev et al. (2022a) B. Ayanbayev, I. Klebanov, H. C. Lie, and T. J. Sullivan.  $\Gamma$-convergence of Onsager–Machlup functionals: I. With applications to maximum a posteriori estimation in Bayesian inverse problems.  _Inverse Probl._ , 38(2):025005, 32pp., 2022a.  10.1088/1361-6420/ac3f81.
  * Ayanbayev et al. (2022b) B. Ayanbayev, I. Klebanov, H. C. Lie, and T. J. Sullivan.  $\Gamma$-convergence of Onsager–Machlup functionals: II. Infinite product measures on Banach spaces.  _Inverse Probl._ , 38(2):025006, 35pp., 2022b.  10.1088/1361-6420/ac3f82.
  * Beer (1993) G. Beer.  _Topologies on Closed and Closed Convex Sets_ , volume 268 of _Mathematics and Its Applications_.  Springer, Dordrecht, 1993.  10.1007/978-94-015-8149-3.
  * Björn and Björn (2011) A. Björn and J. Björn.  _Nonlinear Potential Theory on Metric Spaces_ , volume 17 of _EMS Tracts in Mathematics_.  European Mathematical Society (EMS), Zürich, 2011.  10.4171/099.
  * Chang and Pollard (1997) J. T. Chang and D. Pollard.  Conditioning as disintegration.  _Statist. Neerlandica_ , 51(3):287–317, 1997\.  10.1111/1467-9574.00056.
  * Clason et al. (2019) C. Clason, T. Helin, R. Kretschmann, and P. Piiroinen.  Generalized modes in Bayesian inverse problems.  _SIAM/ASA J. Uncertain. Quantif._ , 7(2):652–684, 2019.  10.1137/18M1191804.
  * Dashti et al. (2012) M. Dashti, S. Harris, and A. M. Stuart.  Besov priors for Bayesian inverse problems.  _Inverse Probl. Imaging_ , 6(2):183–200, 2012\.  10.3934/ipi.2012.6.183.
  * Dashti et al. (2013) M. Dashti, K. J. H. Law, A. M. Stuart, and J. Voss.  MAP estimators and their consistency in Bayesian nonparametric inverse problems.  _Inverse Probl._ , 29(9):095017, 27pp., 2013.  10.1088/0266-5611/29/9/095017.
  * Davey and Priestley (2002) B. A. Davey and H. A. Priestley.  _Introduction to Lattices and Order_.  Cambridge University Press, New York, second edition, 2002.  10.1017/CBO9780511809088.
  * Dembo and Zeitouni (1998) A. Dembo and O. Zeitouni.  _Large Deviations Techniques and Applications_ , volume 38 of _Applications of Mathematics (New York)_.  Springer-Verlag, New York, second edition, 1998.  10.1007/978-3-642-03311-7.
  * Dürr and Bach (1978) D. Dürr and A. Bach.  The Onsager–Machlup function as Lagrangian for the most probable path of a diffusion process.  _Comm. Math. Phys._ , 60(2):153–170, 1978.  10.1007/BF01609446.
  * E et al. (2004) W. E, W. Ren, and E. Vanden-Eijnden.  Minimum action method for the study of rare events.  _Comm. Pure Appl. Math._ , 57(5):637–656, 2004\.  10.1002/cpa.20005.
  * Fel′dman and Nesterenko (1998) N. I. Fel′dman and Yu. V. Nesterenko.  _Number Theory IV_ , volume 44 of _Encyclopaedia of Mathematical Sciences_.  Springer, Berlin, Heidelberg, 1998.  10.1007/978-3-662-03644-0.
  * Freidlin and Wentzell (1998) M. I. Freidlin and A. D. Wentzell.  _Random Perturbations of Dynamical Systems_ , volume 260 of _Grundlehren der mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]_.  Springer-Verlag, New York, second edition, 1998.  10.1007/978-1-4612-0611-8.  Translated from the 1979 Russian original by Joseph Szücs.
  * Grushka (2019) Ya. I. Grushka.  On monotonous separately continuous functions.  _Appl. Gen. Topol._ , 20(1):75–79, 2019.  10.4995/agt.2019.9817.
  * Hansson (1968) B. Hansson.  Choice structures and preference relations.  _Synthese_ , 18(4):443–458, 1968.  10.1007/BF00484979.
  * Helin and Burger (2015) T. Helin and M. Burger.  Maximum a posteriori probability estimates in infinite-dimensional Bayesian inverse problems.  _Inverse Probl._ , 31(8):085009, 22pp., 2015.  10.1088/0266-5611/31/8/085009.
  * Kaipio and Somersalo (2005) J. Kaipio and E. Somersalo.  _Statistical and Computational Inverse Problems_ , volume 160 of _Applied Mathematical Sciences_.  Springer, New York, 2005.  10.1007/b138659.
  * Klebanov and Wacker (2022) I. Klebanov and P. Wacker.  Maximum a posteriori estimators in $\ell^{p}$ are well-defined for diagonal Gaussian priors, 2022.  arXiv:2207.00640.
  * Kretschmann (2019) R. Kretschmann.  _Nonparametric Bayesian Inverse Problems with Laplacian Noise_.  PhD thesis, Universität Duisburg-Essen, 2019.  10.17185/duepublico/70452.
  * Lambley (2022) H. Lambley.  An order-theoretic perspective on modes and maximum a posteriori estimation in Bayesian inverse problems.  MA4K9 Research Project (Master’s Dissertation), University of Warwick, 2022.
  * Lasanen (2012) S. Lasanen.  Non-Gaussian statistical inverse problems. Part I: Posterior distributions.  _Inverse Probl. Imaging_ , 6(2):215–266, 2012\.  10.3934/ipi.2012.6.215.
  * Lassas et al. (2009) M. Lassas, E. Saksman, and S. Siltanen.  Discretization-invariant Bayesian inversion and Besov space priors.  _Inverse Probl. Imaging_ , 3(1):87–122, 2009\.  10.3934/ipi.2009.3.87.
  * Lie and Sullivan (2018) H. C. Lie and T. J. Sullivan.  Equivalence of weak and strong modes of measures on topological vector spaces.  _Inverse Probl._ , 34(11):115013, 22pp., 2018\.  10.1088/1361-6420/aadef2.
  * Malkowsky and Rakočević (2019) E. Malkowsky and V. Rakočević.  _Advanced Functional Analysis_.  CRC Press, Boca Raton, FL, 2019.  10.1201/9780429442599.
  * Stuart (2010) A. M. Stuart.  Inverse problems: A Bayesian perspective.  _Acta Numer._ , 19:451–559, 2010.  10.1017/S0962492910000061.
  * Sudakov (1959) V. N. Sudakov.  Linear sets with quasi-invariant measure.  _Dokl. Akad. Nauk SSSR_ , 127:524–525, 1959.
  * Sullivan (2017) T. J. Sullivan.  Well-posed Bayesian inverse problems and heavy-tailed stable quasi-Banach space priors.  _Inverse Probl. Imaging_ , 11(5):857–874, 2017\.  10.3934/ipi.2017040.
  * Szpilrajn (1930) E. Szpilrajn.  Sur l’extension de l’ordre partiel.  _Fund. Math._ , 16:386–389, 1930.  10.4064/fm-16-1-386-389.
  * Young (1910) W. Young.  A note on monotone functions.  _Q. J. Pure Appl. Math. (Oxf.)_ , 41:79–87, 1910.