# Fast spline detection in high density microscopy data

Albert Alonso  Niels Bohr Institute, University of Copenhagen, Blegdamsvej 17,
Copenhagen, Denmark Julius B. Kirkegaard  Niels Bohr Institute, University of
Copenhagen, Blegdamsvej 17, Copenhagen, Denmark

###### Abstract

Computer-aided analysis of biological microscopy data has seen a massive
improvement with the utilization of general-purpose deep learning techniques.
Yet, in microscopy studies of multi-organism systems, the problem of collision
and overlap remains challenging. This is particularly true for systems
composed of slender bodies such as crawling nematodes, swimming spermatozoa,
or the beating of eukaryotic or prokaryotic flagella. Here, we develop a novel
end-to-end deep learning approach to extract precise shape trajectories of
generally motile and overlapping splines. Our method works in low resolution
settings where feature keypoints are hard to define and detect. Detection is
fast and we demonstrate the ability to track thousands of overlapping
organisms simultaneously. While our approach is agnostic to area of
application, we present it in the setting of and exemplify its usability on
dense experiments of crawling Caenorhabditis elegans. The model training is
achieved purely on synthetic data, utilizing a physics-based model for
nematode motility, and we demonstrate the model’s ability to generalize from
simulations to experimental videos.

##  1 Introduction

Large-scale, high-throughput quantification of microscopy data has
increasingly become possible with the aid of computer vision [1, 2, 3, 4, 5,
6]. In particular, within the last decade, deep learning techniques [7, 8, 9]
have improved and enabled accurate image analysis of microscopy data in a
broad range of areas including cell counting [10, 11], cell segmentation [12,
13, 14], nucleus detection [6, 15], sub-cellular segmentation [16], drug
discovery [17], cancer detection [18, 19, 20], and the identification of
infectious diseases [21, 22]. Detection models serve as the fundamental
operation in tracking procedures and combined with suitable tracking
algorithms, these can achieve morphologically resolved organism tracks that
can accurately quantify organism motility [23], the application of which
ranges from fundamental neuroscience [24, 25, 26] and the circuitry of simple
organisms [27, 28, 29, 30] to drug discovery [31, 32, 33, 34, 35].

Multi-organism detection can be achieved at increasing levels of fidelity: at
the crudest, only center-of-mass locations or bounding boxes are predicted
[36] which does enable tracking of organisms but provide little morphological
information. In contrast, pixel-wise segmentation models [12] and pose
estimation using keypoints [37] reveal accurate shape dynamics when employed
on high-resolution data. However, these methods rely on high definition
objects, as segmentation and prediction are highly sensitive to noise. In
particular, for organisms that are long and slender, pixel-wise segmentation
fails at low resolution, as correct predictions require sub-pixel accuracy.
Moreover, at high densities, these methods may fail due to their inability to
properly handle overlap between organisms.

Here, we consider the problem of studying slender organisms at low resolution
and high density with the goal to enable both accurate identity tracking and
quantification of shape dynamics. This problem has traditionally been
approached by employing pixel-wise segmentation and subsequent skeletonization
procedures [38, 39, 40, 41, 42, 43], an approach that requires ad-hoc
procedures to solve the problem of correctly identifying overlapping organisms
[44], the combinatorial complexity of which blows up at high densities. To
this end we abandon pixel-wise output and instead construct a neural network
architecture that predicts, potentially overlapping, splines directly [45, 46,
47]. Our method enables both accurate shape prediction and tracking in dense
experiments of slender objects. This is applicable to a broad class of systems
[Fig. 1], including tracking of nematode worms [48, 49, 50], spiral or
elongated bacteria [51, 52, 53, 54], spermatozoa [55, 56], the flagella of
both eukaryotes [42, 43] and prokaryotes [57], and freely swimming flagella
such those of microgametes [58].

Figure 1: Microscopy images of different microorganisms whose slender
structure and frequent overlaps makes them hard to detect using classical
approaches. a. C. elegans motility experiment from the dataset of this paper.
b. Motile, flexuous, thin, spiral-shaped B. pilosicoli bacteria. Still from
Ref. [53]. c. Beating flagella of the green alga C. reinhardtii, provided by
Kirsty Wan, University of Exeter. d. Swimming human spermatozoa. From dataset
in Ref. [56].

Our method relies on recent advances in deep learning [59, 60, 61, 62, 63] and
extends these by a few simple ideas: We found that humans are better at
correctly resolving overlap between moving bodies when given access to videos
rather than still micrographs. Thus, to allow the neural network to encode the
identity of individual bodies as a function of their motion, the input to our
neural network is taken to be short video clips rather than single frames. Our
network outputs multiple independent predictions, and for each produces (1)
the spline representing the centre-line of an object, (2) an estimated
confidence score for the prediction, and (3) a latent vector, the space of
which we induce a metric on that measures whether two predictions are trying
to predict the same body. To train the network, each output quantity is
associated with a specific loss term, where, importantly, the spline loss term
is permutation-invariant in the labels. To resolve overlap, we do non-max
suppression [36], but rather than measuring distances between spline
predictions, we use the latent space output, which allows two predictions to
be kept even though they are close in physical space. This enables correct
predictions for data in which objects overlap very closely. Our method is
further tailored to support the subsequent tracking process, which must link
unique predictions from frame to frame. To that end, we not only predict the
object location at a single timepoint but also predict consecutive past and
future splines. Using these time-resolved predictions in the linking process
enables high-precision tracking even through dense regions.

Our method is principally applicable to all microscopy datasets that involve
slender bodies. In this paper, we focus on its applications for tracking dense
experiments of crawling C. elegans worms, a popular model system in
neuroscience [64], human diseases [65], drug discovery [32], motor control
[66], memory [67], and ageing [68]. Studies of C. elegans often rely on
phenotypic assays that measure the motility of the nematode worms as a
function of some environmental condition or treatment [69, 74, 75, 76, 77, 78,
79, 80, 81, 70, 35, 71, 72, 73], the throughput of which can be massively
increased if the overlap between organisms can be tolerated. Likewise,
resolving identities of organisms during overlap is crucial for studies of
interactions between organisms [82]. Previous work on tracking C. elegans have
generally employed classical computer vision approaches to accurately track
single or a few high-definition worms [83, 84, 85, 86, 39], or many low-
resolution worms at non-overlapping densities [87, 88, 40], in some cases by
utilizing a computational model of the worm motion for hypothesis tracking
[39, 83]. Recently, deep learning techniques have been utilized to track C.
elegans worms using e.g. bounding box predictions [89, 90, 91] and fully
resolved centre-line splines in the case of isolated worms [92], allowing for
detection also during periods of self-overlap.

With this paper, we publish a dataset of videos of motile C. elegans worms
imaged at a wide range of densities. The dataset includes $\sim 1,\\!500$
labelled splines that we use to evaluate, but not train, our detection model.
We demonstrate that our model can be trained exclusively using synthetically
generated data and yet generalizes well to real videos. Our method leverages
the parallel capabilities of convolutional neural networks and is thus able to
handle thousands of detections in a single pass, resulting in real-time
detection at $\sim 90$ Hz at $512\times 512$ resolution on a single GPU. The
code is open source and available at
`https://github.com/kirkegaardlab/deeptangle`.

##  2 Results

###  2.1 Architecture

Figure 2 illustrates the overall structure of our approach. Our model is based
on single-stage detection models [36, 59] that output many candidate
predictions per target in a single forward pass and rely on a score system to
prune until a single candidate is left for each target object. The performance
of such single-stage models has been shown to enable accurate real-time
bounding box detection [62]. The backbone of our neural network [Fig. 2a]
consists of convolutional residual networks [60] with the small modification
that we employ average pooling rather than max-pooling to avoid translational
invariance in the spline predictions, which need to be accurate to a sub-pixel
degree.

We take the input to our model to be a stack of consecutive frames in order to
provide the model with a temporal context [Fig. 2c]. This has previously been
shown to improve the detection of e.g. partially hidden objects [93]. In
particular, in the present case of motile slender objects where dynamic
crossings and overlap between objects are very common, a temporal context can
provide the necessary information to resolve the problem of correct
identification. Furthermore, the temporal context allows the output of our
model to include information on the motion of the splines, which we will
further exploit for tracking purposes.

The backbone of our neural network performs a $16^{2}$-fold reduction in
resolution when mapping the input images to feature space, from which the
network outputs multiple anchored predictions. We choose the resulting number
of candidates to be considerably larger than the number of objects in the
frame, thus ensuring that all objects have suggestions. The anchored approach
further means that the only restriction on input size is that its dimensions
be divisible by 16, and, in particular, it allows training at a certain
resolution $H\times W$ and subsequent inference at another $H^{\prime}\times
W^{\prime}$ without loss of accuracy.

The output of our model is composed of spline predictions, confidence scores
and latent vectors:

Figure 2:  (a) Structure of the detection method. Trainable neural networks
are colored in gray and represent the convolutional neural network
$f(I;\theta)$ and the latent space encoder $q(\lambda,x_{0};\phi)$. (b)
Procedure to prune unfiltered predictions to final detections with the use of
the encoded latent space vectors. (c) Method overview from the input clip $I$
(we use a stack of 11 frames in this work) to the final matrix of splines
${\boldsymbol{x}}$. The target frames $[I^{-},I,I^{+}]$ (center frames from
the clip, orange) are explicitly shown for both the synthetic and real videos.
Additionally, the training setup is represented using lighter color arrows;
from synthetic data to loss backpropagation. After detection, direct
visualization of the predicted splines ${\boldsymbol{x}}$ is possible. (d)
Diagram with a spline descriptor composed of $k$ equidistant points along the
skeleton of the nematode. (e) Visual representation of the two distances used
in Eq. (1), the minimum of which corresponds to correct head-tail alignment
and is the one that will be used in the model.

#### Spline predictions

We choose to represent the centre line of the slender bodies of interest by
arrays consisting of $k$ equidistant points [Fig. 2d]. These coordinate
arrays, which we refer to as splines, become high-precision descriptors even
for complex shapes when $k$ is chosen large. To reduce the complexity of
predicting $k$ points, we embed the spline representation with a principal
component (PCA) transform $\mathbf{A}$, the dimension $\kappa$ of which can be
much smaller than $k$ [94]. The PCA components ${\boldsymbol{\lambda}}$
represent the shape, and in addition hereto, the network also predicts the
offset ${\boldsymbol{x}}_{0}$ of the spline, the internal calculation of which
is done in a local coordinate system defined by the anchor points. Thus,
instead of predicting $2k$ floating point values per spline, the network needs
only output $\kappa+2$.

The temporal context of the input image stack permits output spline prediction
also for the non-central images. In our approach, we predict a set of three
splines
${\boldsymbol{z}}=[{\boldsymbol{x}}^{-},{\boldsymbol{x}},{\boldsymbol{x}}^{+}]$
corresponding to the three central frames $[I^{-},I,I^{+}]$ of the input stack
[Fig. 2c]. We consider the central spline ${\boldsymbol{x}}$ the main output,
whereas the past ${\boldsymbol{x}}^{-}$ and future ${\boldsymbol{x}}^{+}$
splines are considered auxiliary predictions whose main purpose lies in their
use during the latent space encoding as well as the tracking procedure.

We define the similarity measure between two splines by the standard Euclidean
distance. In the case of splines that look symmetric from either end, we
exploit this symmetry and employ the flip-invariant distance defined by

$\displaystyle
d^{2}\\!\left({\boldsymbol{x}},{\boldsymbol{x}}^{\prime}\right)=\min\Big{[}\sum_{i=1}^{k}(x_{i}-x^{\prime}_{i})^{2},\,\sum_{i=1}^{k}(x_{i}-x^{\prime}_{k-i+1})^{2}\Big{]},$
(1)

as illustrated in Figure 2e.

Likewise, we define a distance between two collections of consecutive splines
${\boldsymbol{z}}$, ${\boldsymbol{z}}^{\prime}$ by their weighted average
$d_{s}^{2}=\sum_{t}\omega_{t}\,d^{2}({\boldsymbol{z}}_{t},{\boldsymbol{z}}^{\prime}_{t})$,
where the weights can be adjusted to give focus to central predictions, and
for the present case we choose $\omega=2\omega^{-}=2\omega^{+}$.

The neural network is trained to minimize the distance $d_{s}^{2}$ between
predictions and labels. To do so, we let the independent predictors specialize
for different shapes. This is achieved by using a permutation-invariant loss
such that the total loss is computed as a sum over the labels only, each using
the predictor that best match the labels. Thus many spline predictions will
not contribute to the spline loss.

#### Confidence scores

Each independent prediction of the network includes a confidence score $s$,
which is used to filter out bad candidates. In bounding box or mask detection,
intersection over union (IoU) is commonly used to evaluate the accuracy of a
prediction, however, this metric does not generalize well to spline
predictions when there is overlap. Instead, we introduce a custom metric to
define the goodness of a spline set ${\boldsymbol{z}}$ by comparing it to its
label $\hat{{\boldsymbol{z}}}$,

$\displaystyle\hat{s}=\exp\\!{\left(-d_{s}^{2}({\boldsymbol{z}}_{,}\hat{\boldsymbol{z}})/\sigma_{s}^{2}\right)}.$
(2)

Here, $\sigma_{s}$ is a parameter that sets the scale over which the score
varies. The metric is sensitive to perturbations on accurate predictions, i.e.
predictions close to labels where $d_{s}\rightarrow 0$, but loses sensitivity
the worse the predictions are. This is a useful feature as correct scoring for
good predictions is crucial for choosing the best one, whereas low-scoring
predictions are discarded in any case and their relative scoring is therefore
unimportant.

The score prediction is trained using L2 loss. To avoid conflicting backwards
error propagation between this task and that of spline prediction (as scoring
bad predictions is easier), we stop the gradient flow in the computational
graph on the last layer of the score-predicting part of $f_{\theta}$ [Fig. 2a]
such that it does not interfere with the accuracy of the predicted splines.

#### Latent space for candidates suppression

Finally, we need to ensure that there is only one prediction per object.
Bounding box detectors let the user decide the fraction of overlap between
prediction boxes of the same class that should be considered to be targeting
the same object. As our method must work at high densities, this task is
complicated by the fact that two predictions might be very close, even
completely overlapping in the central frame, and yet represent different
objects. The task of choosing a suitable cutoff distance is therefore
difficult, and we make this a trainable task. We do so by embedding each
prediction in a low-dimensional latent space in which comparison between
predictions is cheap, thus allowing efficient and fast candidate suppression
also at high densities.

Our method computes the latent vectors ${\boldsymbol{p}}$ for predictions
using an auxiliary neural network, $q_{\phi}$ which acts directly on the
eigenvalues ${\boldsymbol{\lambda}}$ and offsets ${\boldsymbol{x}}_{0}$ rather
than the more redundant spline coordinate points. We induce a Euclidean metric
on the latent space with the interpretation that two predictions $i,j$ are
predicting the same object with probability

$\mathbb{P}(i\leftrightarrow
j)=\begin{cases}\exp\\!\left(-||{\boldsymbol{p}}_{i}-{\boldsymbol{p}}_{j}||^{2}\right)\quad&\text{if
}||{\boldsymbol{x}}_{0i}-{\boldsymbol{x}}_{0j}||\leq\sigma_{l},\\\
0\quad&\text{otherwise.}\\\ \end{cases}$ (3)

Here, $\sigma_{l}$ is a real-space visibility cutoff that prevents far
predictions to interact in the encoded space, thus avoiding the need to scale
the dimensionality of the latent space with the number of candidates or the
input size. We note that when using the flip-invariant metric $d_{s}$ on
splines, we explicitly construct the latent space encoder to likewise be flip-
invariant (see Methods).

To train the latent space, we make the assumption that during training
predictors are ‘trying’ to predict the label closest to the prediction spline.
Combined with the probability interpretation, this allows us to use binary
cross entropy as a loss function for the probability defined in Eq. (3). To
avoid wrong clustering between undefined close-by predictions, the loss
contribution of each prediction is scaled by the product of their real scores
$\hat{s}_{i}\hat{s}_{j}$, thus ensuring that the network focuses its attention
on good predictions that will not be filtered out. Finally, since the encoder
should not alter the performance of the spline suggestions, the loss on the
latent space representations only updates the weights $q_{\phi}$ of the
encoder but is trained concurrently with the main model.

We employ non-max suppression to choose the best prediction of each object,
but with distances measured in latent space, as illustrated in Fig. 2b.
Concretely: Once all the predictions whose score is lower than a threshold
$\tau_{s}$ have been discarded, multiple candidates are likely to still remain
for each target object. The lack of low score predictions exposes clusters in
the latent space that correspond to single objects. We sort the remaining
predictions by their score, automatically accepting the highest-scored one.
Once a prediction $i$ is accepted, all predictions $j$ that have a high
probability $\mathbb{P}(i\leftrightarrow j)>\tau_{o}$ of being the same object
are removed. This is equivalent to setting an exclusion radius $r_{l}$ in the
latent space as shown in Fig. 2b. We keep iterating on the remaining
predictions, pruning the latent space until all candidates have been iterated.
The final number of accepted predictions should equal the number of objects in
the frame.

### Detection on dense C. elegans experiments

Figure 3: Showcase of the capabilities of the method. (a) Detected splines
predicted on an entire densely populated well plate with a single forward pass
through the neural network. Inset shows a zoom-in section to demonstrate the
accuracy of detection across the entire plate (except near borders, where the
plate interferes). The total plate contains around $6,\\!000$ splines. (b)
Close-up evaluation of different experimental clips with different densities
of worms. Figure 4:  (a) Average number of overlaps counted on frames of pixel
size $512\times 512$ with different densities of worms ($N=90$). (b)
Illustration of the asymmetric dynamic time warping distance error
corresponding to the average value of the orange euclidean distances between
the prediction (green) and the labelled points (white). (c) Example frame with
manually labelled points (white) and models predictions (colored). The metric
is only evaluated in the lighter area of size $100\times 100$. (d) Quantified
accuracy of the detections by showing the distance to the manually labelled
splines. Distributions for different densities are shown. The violin plots
represent the 99 percentile of the data whereas outliers are plotted
individually. (e–f) Rates for True Positive and False Negative on the manually
annotated dataset. (g–i) Performance of the model with different combinations
of score ($\tau_{s}$) and overlap ($\tau_{o}$) thresholds. $N=1,\\!420.$

To evaluate our approach, we study microscopy videos of crawling C. elegans
worms. We are particularly interested in videos captured at much higher
densities than those typically used in motility experiments. Thus we evaluate
our model on wide-field videos captured under approximately uniform
illumination [40], exemplified in Fig. 3a. In our dataset, the number of
nematode worms varies ranging from $\sim 400$ with a small probability of
overlap occurring ( $\\!\approx 0.05$ average overlaps per worm) to extremely
densely packed plates with up to $\sim 6,\\!000$ nematodes, where there is, on
average, one overlap per worm. This means that in the dense plates, detection
methods that stop tracking after contact between worms happens are rendered
completely ineffective.

Defining worm density $\rho$ as the number of worms in a region per square
millimeter, we find, as expected, a linear relation between the average amount
of overlap per worm and the density [Fig. 4a]. Due to the spatial
heterogeneity of the worm distribution inside the plate, higher densities can
be observed when considering small regions. On $100\,\mathrm{mm}^{2}$ scales,
the highest density in the dataset is $\rho\sim 2.5\,\textrm{mm}^{-1}$, but
this jumps to an extreme $\rho\sim 3.5\,\textrm{mm}^{-1}$ when considering
$10\,\mathrm{mm}^{2}$ regions, where humans begin to struggle to correctly
identify worms. For quantitative evaluation of our model, $\sim 200$ random
regions of the videos were sampled and hand-labelled resulting in $\sim
1,\\!500$ labelled worm splines. A sample of frames is shown in Fig. 3b to
provide a sense of the different densities encountered in the evaluation
dataset, with the predictions of the model overlaid.

#### Simulation-based training.

To train our network, we implement a physics-based synthetic dataset generator
to exploit perfectly defined labels. This approach removes the need for a
supervised dataset, and also allows labelled videos in situations where manual
labelling may not be reliable, or where the subjectivity of the human
labellers can result in inconsistent labels. Physics-based synthetic datasets
have successfully been used to train systems on similar conditions, for
instance where manual labelling may introduce unnecessary noise or bias to the
model [16]. Our in-silico data generator has two main components: a physics-
based model for the organism and a synthetic frame generator.

In-silico worms are generated on demand every training step which removes the
possibility of overfitting to the generated frames. In order to train the
model to work effectively with a range of worm densities, we generate batches
with different numbers of worms in a uniform manner, without bias towards low
or high worm counts. This teaches the model to handle a variety of densities
without overfitting to any specific case. And to make the model more robust,
training also happens on densities whose manual annotation would be extremely
challenging. The simulation and video synthesis are implemented in a GPU
framework which enables fast end-to-end training without the performance
penalization of data transferring between the accelerator and the host
machine.

We base the worm simulation on resistive force theory, as it has previously
been shown to correctly predict the position of the skeleton for short spans
of time [95]. Since the network only perceives the frames surrounding the
target frames, we found the total duration of the clip to be short enough that
a linear crawling model approximation fits our needs. The physics-based model
should encapsulate all types of organism behavior. This can be achieved by
oversampling the behavior, i.e. by making the simulations more diverse in the
behavior than reality and thus hope to include all types of real behavior as
well. Details on the worm simulation and video synthesis can be found in the
methods section.

Despite the potential for physics-based simulations to be used for synthetic
training data, discrepancies with real data may lead to inaccuracies when
applied to real microscopy images. This reality gap can be the result of an
overly simplified motility model or physics model or the result of imprecise
video synthesis. The gap may be further increased by the fact that the model
relies on the PCA transformation matrix $\mathbf{A}$ obtained on synthetic
data, where the number of PCA components used have been chosen to accurately
reproduce all synthetic patterns, but not necessarily to generalize to out-of-
sample videos. Thus we find that our model is limited to accurate skeleton
predictions only on shapes that resemble those produced by our simulations,
and the goal of the simulations is therefore to reproduce a broad spectrum of
possible motility patterns. Likewise, we find that our model is susceptible to
the brightness of the videos, and accordingly, we adjust the real videos to
increase their resemblance to the training data (see Methods).

#### Metrics

Despite being trained exclusively on synthetic data, the model’s inference
performance is very good on real clips. From visual inspection, no immediate
discrepancies are observed between detections in low density clips and at high
density [Fig. 3b]. Likewise, per design, the network accuracy is independent
of the input clip dimensions, and the parallel structure of convolutions
permits the use of large videos covering thousands of nematodes to be
processed simultaneously in a single forward pass [Fig. 3a]. We note, however,
that even though no quality impact on detections is observed when using large
fields-of-view clips, there can be a dependency if non-uniform illumination is
used as different sections of the frame may have different requirements for
preprocessing.

For a quantitative assessment of the method accuracy, we compare to the
manually labelled dataset, an example of which alongside the model predictions
can be seen in Fig. 4c. As the predictions are densely defined splines (here,
$\sim 50$ points), we introduce a custom metric to suitably evaluate the
accuracy of the predictions using labels with lower fidelity. The metric used
must be shift-invariant, as having points anywhere along the spline should
yield zero error regardless of whether the label points precisely coincide
with the prediction points or not. Likewise, label points should be
monotonically assigned along the spline in order to avoid artificially
reducing the error for strongly bent or self-coiling worms. Finally, it must
be robust against the subjectivity of the labellers, as manual annotations
might miss or avoid spots where visibility is low such as the end-points of
the worms.

To satisfy all these requirements, we introduce a metric based on the
dynamical time warping (DTW) distance used to measure the similarity between
temporal curves. In our modified version, asymmetric DTW, summation only runs
over label points. Thus, the metric $\delta_{\mathrm{adtw}}$ is defined as
follows: Let $d(i,j)$ be the Euclidean distance between label point $i$ and
prediction line segment $j$, then

$\displaystyle\delta_{\mathrm{adtw}}=\min_{\alpha}\frac{1}{N}\sum_{i=1}^{N}d(i,\alpha(i)),$
(4)

where $\alpha:[1,N]\rightarrow[1,M]$ is a monotonic (non-decreasing or non-
increasing) assignment of the $N$ label points to the $M$ prediction line
segments. A visual representation of the metric is shown in Figure 4b, and the
$\mathcal{O}(NM)$ algorithm for its calculation is detailed in the Methods
section.

The results of evaluating the trained model on the labelled dataset are shown
in Fig. 4. For reliable comparisons, we first solve the assignment algorithm
for the label-prediction pairs. This means that in the case of two completely
overlapped worms, two predictions need to be present to not count as a miss,
and likewise, two predictions cannot be considered to target the same label.
We find an average error of $\delta_{\mathrm{adtw}}\approx 0.54\,\mathrm{px}$
with no strong dependency between accuracy and density of worms [Fig. 4d],
with the exception of a slight increase in error for extremely dense clips
($\sim 3.5\,\mathrm{mm}^{-2}$). The average error corresponds to less than the
width of a worm ($\approx 2\,\mathrm{px}\approx 50\,\upmu\mathrm{m}$), and
part of this can be attributed to the fact that humans accuracy is also near
the half-pixel level [Fig. 4c]. Some outliers can be seen, however, which can
mostly be attributed to an artefact of the model, where the network mistakes a
single long worm for two overlapping shorter predictions. This effect seems
particularly sensitive to incorrect intensity normalization of the videos.

Let $\sigma_{\epsilon}$ be a cutoff distance above which we no longer consider
the predictions to be targeting the closest label. For all the figures in
Figure 4, this cutoff is assumed to be $\sigma_{\epsilon}=3.0\,\mathrm{px}$,
and we observe no significant changes by tuning it within the range of
sensible values. We define the True Positive (TP) rate as the fraction of
predictions that both get assigned a label and this label is within the
distance $\sigma_{\epsilon}$ of the prediction. Figure 4e shows that the model
rarely predicts a spline where there is nothing with a TP rate of $0.999$.
Nevertheless, there are some predictions that do not get assigned a label
which can be attributed to the double-prediction artefacts just mentioned. The
likelihood of this happening decreases with density, but the rate is so low
that it is almost negligible. Similarly, we define the False Negative (FN)
rate as the fraction of labels that are not assigned a prediction closer than
$\sigma_{\epsilon}$. Fig. 4f shows that the model in general manages a low FN
rate at around $\sim 0.015$, but that this increases to a rate of $\sim 0.06$
at extreme densities such as $\rho\geq 3.0\ \mathrm{mm}^{-2}$, where clusters
tend to be densely packed and manual labelling likewise becomes challenging.

The filtering part of the model depends on the previously introduced
thresholds $\tau_{s}$ and $\tau_{o}$. The score threshold, $0<\tau_{s}<1$, is
used to prune low score predictions [Fig. 2b(1)], while the overlap threshold,
$0<\tau_{o}<1$, is used to decide the probability of two independent
predictions to be targeting the same object [Fig. 2b(2)]. Throughout this
paper, we have set these to $\tau_{s}=\tau_{o}=0.5$. However, due to their
relevance in modifying the filtering process, we evaluate how different
combinations of thresholds may alter the performance results. Figures 4g–i
show the average performance obtained across all densities when filtering the
predictions with variable thresholds. In spite of some dependency between worm
density and TP/FN rates, we consider the average metric to be a good indicator
of the performance in each case.

Fig. 4g shows the effect of the thresholds on accuracy. No significant
dependency on the thresholds is observed. This can be explained by the fact
that accuracy is determined by the best predictors only, which are not
discarded until a high $\tau_{s}$ is used, and once those are removed,
$\tau_{o}$ becomes irrelevant. Further, the fact that there is no notable
difference between different values of $\tau_{o}$ indicates that the clusters
are highly compact.

In contrast, Fig. 4h shows that the TP rate has a stronger dependency on
$\tau_{o}$ at low $\tau_{s}$ because low score predictions do not form compact
clusters, and therefore a larger exclusion radius is required to discard them.
Finally, Fig. 4i shows that misses only begin to occur once the best
predictions are discarded, and a strong dependence on the $\tau_{s}$ is not
observed before that point.

###  2.2 Tracking from consecutive detections

Figure 5:  (a) Illustration of the directed distance used to assign the same
identity to consecutive detections. The simplified drawing shows two
independent predictions at adjacent frames and showcases how the assignment
scheme computes the identity by comparing future-present and past-present
distances and choosing the assignment that minimizes their sum. (b) Diagram
showcasing how using a location cutoff simplifies the assignment problem.
Nodes represent independent detections at each frame whereas edge values are
given by the directed distance measure. The assignment happens by minimizing
the sum of edges at each timestep. (c) Comparison of using the straightforward
spline distance and the proposed directed approach. The accuracy is evaluated
by measuring the integrity of the tracks. In contrast to other metrics in this
paper, this plot has been obtained using synthetic worms as long-term,
accurate tracks are required to evaluate the tracking integrity (See Methods
for details on Tracking integrity). (d) Qualitative example of 30 s
trajectories of the center of mass of the nematodes in a dense experiment. The
still background image represents the last frame of the video. To improve the
visualization, a small subset of the trajectories is shown. In contrast, a
corner of the frame is used to display all the trajectories to showcase the
density of simultaneous tracks. (e) Two samples of the spline angle $\psi$ of
two randomly sampled nematodes from (d). (f) Undulations corresponding to 30 s
of the detections relative to the center of mass coordinate of nine randomly
sampled nematodes from (d). (g) Standard error value of the measurements of
the center of mass speed as a function of density. (h) Showcase the possible
throughput of the method, by simultaneously tracking more than $6,\\!000$
tracks from a full dense plate. A small window on the tracks is shown to
showcase their continuity.

Motility assays require not only accurate detections but also the ability to
link these across frames to form time-resolved tracks of individual organisms.
This is challenging at high densities where we have the breakdown of the
assumption that the closest detected object to the previous frame corresponds
to the same identity. In general, greedy approaches to particle tracking such
as assigning directly the closest particle in consecutive frames frequently
leads to failed tracks. Instead, the process of tracking can be efficiently
formulated as a set of linear assignment problems [96]. Naturally, here we can
expand upon particle tracking by using a metric that measures distances not
between the center-of-mass of the worms, but between the full splines as
defined in Eq. (1). This works well for most predictions but can fail for
fast-moving worms or in dense clusters.

A separate approach to tracking is Kalman filtering. This would require
separate detection of entry and exit events of worms, as well as a
probabilistic model for worm motility, which would most likely have to be
highly non-linear. Kalman filtering is viable for the tracking of few
organisms, but for present large-scale systems, we require a more efficient
approach. As previously mentioned, splines from adjacent frames are also
predicted in order to embed temporal information into the latent vector. We
propose a directed metric that leverages both past $x^{-}$ and future $x^{+}$
spline predictions [Fig. 5a]. Thus to find a mapping $\sigma$ from one frame
to the next, we solve

$\displaystyle\sigma=\arg\min_{\sigma}\left[\sum_{i}d({\boldsymbol{x}}_{i}(t),{\boldsymbol{x}}_{\sigma_{i}}^{-}(t^{\prime}))+d(\boldsymbol{x}^{+}_{i}(t),\boldsymbol{x}_{\sigma_{i}}(t^{\prime})\right].$
(5)

Identity assignment can be seen as a network flow global optimization where
nodes represent detections and edges carry the cost of assignment. To avoid
having to perform all possible combinations of assignments, we include a
physical distance threshold on the midpoint of the central spline. This
threshold significantly simplifies the assignment scheme and improves the
runtime of the filtering process [Fig. 5b].

To quantify the performance of these methods, we define the tracking integrity
$\iota$ as a scalar that indicates how consistent the assignment of a label to
a prediction is along the tracked video. Perfect tracks have $\iota=1$,
whereas labels that get assigned two different identities for half of the
duration of the video have $\iota=\frac{1}{2}$, and so on (see Methods for a
detailed definition). We evaluate this on synthetically generated videos of 10
seconds (200 frames) that have perfectly labelled tracks, the results of which
are shown in Fig. 5c. On videos with densities up to 2.0 $\mathrm{mm}^{-2}$,
we achieve an average integrity of $\iota\approx 0.97$. This is a $\sim$ 30 %
improvement of the error over using direct spline assignment defined in Eq.
(1). We observe that the integrity is almost perfect at low densities, but
drops to $\iota\approx 0.93$ at the highest densities.

When applied to high density videos of C. elegans, the tracking method is able
to keep track of individual worms as they pass through clusters of other worms
[Fig. 5d]. In contrast to pixel-level classification of worms, our approach
outputs splines directly, and thus subsequent analysis is straightforward. For
instance, one may directly study the worm undulations [Fig. 5f] or extract the
worm spline angle $\psi=\arctan(y(s,t)-y_{0}(t),\,x(s,t)-x_{0}(t))$ to provide
insight into the movement patterns and kinematics of the worm [Fig. 5e].

One of the key advantages of our methods is its ability to collect a larger
number of samples compared to traditional techniques, while still obtaining
reliable results. As the standard error decreases with the number of samples,
using our methods allows for metrics to be gathered with less uncertainty
while still requiring the same experimental setup. For instance, Figure 5g)
shows how the error of estimating the average speed of the center of mass of
the nematodes decreases with density. This advantage can be extended to
tracking large numbers of nematodes in crowded environments, such as extremely
dense petri dishes where more than $6,\\!000$ concurrent tracks can be
simultaneously computed [Fig. 5h]. Thus, with our method, we are able to
collect a larger number of samples and obtain more precise and reliable
results, even in challenging conditions.

##  3 Discussion

We have introduced a novel deep learning approach for detecting and tracking
slender bodies, such as crawling nematodes, in microscopy data. The presented
convolutional neural network architecture is capable of accurately detecting a
large number of overlapping organisms, a task that can be particularly
challenging for standard methods such as bounding boxes and pixel-level
classifiers due to the issue of occlusion and overlap. To address this, we
have implemented a latent space encoding which allows us to filter by non-
maximum suppression and effectively handle overlapping objects. Not only is
our method capable of accurately detecting and tracking slender bodies, but it
also demonstrates strong scalability, performing well across a range of input
frame sizes and densities of bodies. This makes it an ideal tool for a variety
of experimental settings where splines are useful descriptors, including
studies of crawling nematodes, swimming spermatozoa and beating eukaryotic or
prokaryotic flagella.

Besides a suitable detector model, labelled training data is also needed. We
have demonstrated that relying on a physics-based model to generate synthetic
data is adequate to train our network to perform well on real data. This is a
key achievement as it means that applications of our system for different
experimental studies do not require large datasets to be procured, but rather
the implementation of a suitable simulation. Our approach for synthetic data
generation relies on over-sampling the behavior of the worms. This is
naturally a trade-off as too extreme behavior can lead to datasets that are
too hard for the neural network to replicate. For our model, we found that we
slightly undersampled certain worm shapes such as strong coiling, which the
model therefore could struggle with identifying. Though we did not look into
this here, an interesting avenue for future research would be to bootstrap
synthetic motility models on small datasets of real organisms. In a similar
fashion, the frame-generator procedure should oversample the textures, pixel
intensities and noise of real videos. Here, it could be interesting to study
whether style transfer [15] or diffusion models [97] could be used to further
reduce the gap between training and inference data.

For tracking, we introduced a directed metric that employs past and future
spline predictions to link them across time. At very high densities this may
still fail, in particular, because the directed metric yields little advantage
if predictions are missing in some frames. A potential way to improve on this
could come from utilizing the latent space encoding as well. This would
require temporal continuity in the latent space representation, which is
achievable by modifying the associated loss function. This should enhance the
integrity of tracking, as it could potentially be used to resolve issues such
as switches by leveraging the separation of close physical predictions with
different temporal behaviour that characterises the latent encoding. We
believe that these suggestions might be fruitful avenues for further research
for improving deep learning models for dense detection of splines.

In this paper, we have proposed a new approach for fast and precise detection
and tracking of slender bodies in microscopy data. Its speed and accurate
performance across a range of densities and sizes, combined with the ability
to handle overlapping objects, make it a valuable tool for a variety of
experimental settings where precise tracking is essential for obtaining
quantitative metrics.

##  4 Methods

#### Convolutional neural network

Most of the weights of the network are at the feature detection convolutional
network whose backbone is made of four ResNet groups consisting of $2,4,4,2$
blocks with strides $1,2,1,2$, respectively. We modify the original ResNet
architecture by replacing the initial max-pooling layer with an average-pool
layer to avoid translational invariance. The final shape of the feature space
is $[H/16,W/16,C]$, with $C$ being the number of candidates each cell
proposes. We have set $C=8$ for this project in order to fulfil the condition
of $M\gg N$ even at high densities. All in all, there will always be $C$
candidates per cell regardless of input size, which leads to a large number of
candidates being sorted in the filtering process. The head of the
convolutional neural network is composed of two fully connected layers of 512
and $C\cdot(3(m+2)+1)$ cells, respectively, with batch normalization in
between. Due to the orientation invariance of the loss function on the spline
predictions, it is possible that the splines in the predicted set
${\boldsymbol{x}^{-}},{\boldsymbol{x}},{\boldsymbol{x}^{+}}$ are not aligned.
To remedy this, we aligned them by comparing them with the eigenvalues of the
flipped spline. In order to get the flipped eigenvalues $\lambda_{f}$, we use

$\displaystyle\lambda_{f}=\mathbf{A^{-1}}\mathbf{J}\mathbf{A}\lambda$ (6)

where $\mathbf{A}$ is the PCA transformation matrix and $\mathbf{J}$ is the
exchange matrix.

#### Latent space encoder

The encoder $q_{\phi}$ is composed of two fully connected layers with batch
normalization in-between. The input of the encoder is the vector of size
$3(m+2)$ characterizing the splines predictions and the output is $D$ floating
point values, corresponding to the coordinates of $p$ in the $D$-dimensional
latent space. We have found $D=8$ to be a well-performing dimension in our
experiments. Due to the orientation invariance of the splines predictions, we
need to construct the encoder to cluster those splines regardless of
orientations as well. To do so, the input values are expanded to include those
of the flipped splines $\lambda\to(\lambda,\lambda_{f})$ and both are fed to
the same layer. To ensure symmetry, the output is then summed before passing
through the last layer. In doing so, the encoder becomes independent of spline
orientation.

#### Input clips pre-processing

The images used to train the model have dark (small pixel intensity)
backgrounds, as we employ zero-padded convolutional layers. This is relevant
for real recordings, where a negative flip may be necessary to match the
network requirements. During training, generated clips are normalized using a
1–99 percentile normalization. For real clips, we have found that accuracy is
improved if we apply CLAHE (adaptive histogram equalization) before
prediction. Likewise, a simple intensity correction factor $\mu$ may need to
be applied to the videos in order to match the pixel profile of the simulated
data. For our dataset, we use correction factors of $\mu\approx 1.2$, to get
the best results. Note that we match real data to the synthetic as this avoids
the need to retrain the network for different experimental setups.

#### Loss functions

Spline descriptors are trained as a regression problem. Thus, the loss
contribution is given by the custom distance defined in Eq. (1). To enforce
specialization on the predictors, and due to the number of predictions $M$
being considerably larger than the number of bodies $N$, only the best
predictors are accounted for in the loss. Nevertheless, there may be labels
$\hat{{\boldsymbol{x}}}$ completely or partially outside the frame at $t_{c}$,
despite being inside at $t_{0}$. To make sure not to punish bad predictions at
the boundaries for not matching invisible splines, instead of using the number
of simulated bodies $N$, the subset of bodies completely inside the frame
$N_{v}$ is used and the final loss expression is given by:

$\displaystyle
l_{x}=\frac{1}{N_{v}}\sum_{i}^{N_{v}}\min_{m}d^{2}_{s}({\boldsymbol{z}}_{m},\hat{\boldsymbol{z}}_{i})$
(7)

The score L2 loss is computed as the difference between the values predicted
and the score the spline proposals should have. Thus, using Eq. (2), we train
the predicted score of all predictions using:

$\displaystyle
l_{s}=\frac{1}{M}\sum_{i}^{M}\left(\exp{\left(-\min_{n}\frac{{d}^{2}_{s}({\boldsymbol{z}}_{i},\hat{\boldsymbol{z}}_{n})}{\sigma_{s}}\right)}-s\right)^{2}$
(8)

Finally, the loss function for the latent space encoder is a modified cross
entropy loss scaled by the product of scores. Denote
$\mathbb{P}_{i,j}=\mathbb{P}(i\leftrightarrow j)$ as defined in Eq. (3), then
the encoder loss is defined as an average over all pairs of predictions
$\langle i,j\rangle$ that are physically within the cutoff $\sigma_{l}$,

$l_{p}=\frac{1}{S}\left\langle{\hat{s}_{i}\hat{s}_{j}}(t_{ij}\log{(\mathbb{P}_{i,j})}+(1-t_{i,j})\log{(1-\mathbb{P}_{i,j})})\right\rangle_{\langle
i,j\rangle},$ (9)

where $S=\sum\hat{s}_{i}\hat{s}_{j}$, and $t_{i,j}$ indicates whether $i$ and
$j$ are targeting the same label $k$, and is set by

$t_{ij}=\begin{cases}1\qquad\text{if }k_{i}=k_{j}\\\
0\qquad\text{otherwise}\end{cases}$ (10)

with $k_{i},k_{j}$ being the closest labels to the predictions
${\boldsymbol{z}}_{i},{\boldsymbol{z}}_{j}$ respectively.

#### Training details

Training has been done from scratch, i.e. without the use of a pretrained
backbone. During training, the frame size for the input clips used was
256$\times$256, but due to the anchored approach, this does not constrain
inference to happen at the same resolution. Synthetic input is generated on
demand and on device rather than using a fixed pre-generated dataset. Thus,
the network never sees the same frame twice and there is no host-to-device
data transfer. As mentioned in the main text, all networks are trained
simultaneously, despite the weights of each one depending on different cost
functions. The code has been written in Jax using Haiku and training has been
carried out on a cluster of 8 $\times$ NVIDIA A5000’s.

#### Inference

Inference happens at any resolution whose dimensions are multiple of 16. The
input frames need to be slightly pre-processed as described in the previous
sections. Candidate predictions are chosen using a score threshold, and non-
maximum suppression in latent space is used for filtering. Due to the
sequential nature of the filtering process, the implementation is written to
use the CPU using numba.

#### Worm simulation

Worm trajectories are computed by employing a resistive force theory crawling
model used to predict rigid body motions of C. elegans from the undulations
[95]. Thus, we ensure that from a given set of generated undulations, the
produced motions will match those of real worms. From empirical observations,
we propose a simple equation (Eq. (11)) to generate the undulation of the
worms. We define the motions by the spline angle $\psi_{k}(s)$ with
$s\in[0,1]$ [Fig. 2d], and decompose this into a linear combination:

$\psi(s)=\psi_{u}(s,t)+\psi_{s}(s,t).$ (11)

This logically separates the worm undulations into two types of motion: one
corresponding to a sinusoidal motion $\psi_{s}$ and one in which the whole
body bends $\psi_{u}$. These we define by

$\displaystyle\psi_{u}(s,t)$
$\displaystyle=A\cos\left(\frac{2\pi}{T}t+\rho_{1}\right)\cos\left(k_{u}s_{k}+\rho_{2}\right)$
(12) $\displaystyle\psi_{s}(s,t)$
$\displaystyle=\tilde{A}\cos\left(\frac{2\pi}{T}t+k_{s}s_{k}+\rho_{3}\right)$
(13)

where $\tilde{A}=\frac{1}{2}\left(1+|\sin\left(2\pi t\right)|\right)A$ and the
rest of parameters are sampled from random distributions. Although many
improvements for the above equations can be suggested, we prefer to keep the
model simple.

Once the values of the parameters for $\psi$ are generated all for the
timesteps of the simulation, the positional coordinates are obtained using

$\displaystyle\vec{x}(s,t)$
$\displaystyle=L\int_{0}^{s}\begin{pmatrix}\cos\left(\psi(s^{\prime},t)+\gamma\right)\\\
\sin\left(\psi(s^{\prime},t)+\gamma\right)\end{pmatrix}\mathrm{d}s^{\prime}$
(14)

where $\gamma$ is a random orientation and $L$ is the length of the worm (also
sampled). Once the skeleton is defined, the rigid body motions are predicted
by solving [95]

$\displaystyle\vec{F}$ $\displaystyle=\int_{0}^{L}\vec{f}\,\mathrm{d}s=0,$
(15) $\displaystyle\vec{\tau}$
$\displaystyle=\int_{0}^{L}(\vec{x}-\vec{x}_{\text{CoM}})\times\vec{f}\,\mathrm{d}s=0,$
(16)

where the force $\vec{f}$ can be calculated from the spline velocity
$\vec{U}=\partial_{t}\vec{x}+V+\Omega\times(\vec{x}-\vec{x}_{\text{CoM}})$ by

$\displaystyle\vec{f}=\alpha_{t}\,(\hat{t}\cdot\vec{U})\,\hat{t}+\alpha_{n}\,(\hat{n}\cdot\vec{U})\,\hat{n}.$
(17)

Here, $V$ and $\Omega$ are the center-of-mass velocity and rotational velocity
(that we are solving for), and $\alpha_{t}$ and
$\alpha_{n}=\alpha\,\alpha_{t}$ is the tangential and normal drag
coefficients, which is also sampled for ($\alpha>1$). We did not find a need
for using a non-linear force theory. The simulation is run with Python 3.9
using the Jax library.

#### Video synthesis

Given the labels for the splines positions, synthetic videos are generated to
be used as input during training. In order to add width to each worm, we vary
the local body radius $r$ by a function of the form

$r(s)=\tilde{R}\left|\sin\\!\left(\\!\arccos\\!\left(as+b\right)\right)\right|$
(18)

The pixel values of those circles are calculated with anti-aliasing. Once the
worms have been rendered, noise artefacts such as uneven background, blurring,
Gaussian noise, etc. are added to replicate the observed conditions of real
experiments. During training, standard augmentation techniques are applied as
well. In the same manner as the simulation of the motion and the neural
network training, frame generation is also written in Python using the Jax
library in order to leverage GPU capabilities.

#### Experimental dataset

Videos of crawling C. elegans were filmed using the protocol described in Ref.
[40].

#### Manually annotated dataset

The evaluation dataset is annotated using a custom tool that can be found at
`https://github.com/kirkegaardlab/deeptanglelabel`. Around $\sim 1,\\!500$
splines have been annotated and this dataset (videos and labels) is included
in the SI.

#### Asymmetric dynamic time-warped error distance

In order to evaluate the manually labelled dataset, we introduce an error
distance that compares the similarity between two curves by calculating the
distance between each point on one curve and the nearest segment on the other.
The error distance used is a variation of the dynamic time warping distance,
which is widely used for comparing time series data. We note that, just as is
the case for the dynamic time warping distance, this is not a true distance in
the mathematical sense.

Data: Label curve defined by $N$ points $\\{p_{i}\\}$, and prediction curve
defined by $M$ line segments $\\{s_{j}\\}$.

Result: The asymmetric dynamically time-warped distance from label to
prediction.

Initialize matrices $C,D$ with size $[N,M]$.

for  _$i=1$ to $N$_ do

for  _$j=1$ to $M$_ do

$D_{i,j}\leftarrow$ distance_from_point_to_segment(_$p_{i},s_{j}$_)

$C_{1,1}\leftarrow D_{1,1}$

for  _$i=2$ to $N$_ do

$C_{i,1}=C_{i-1,1}+D_{i,1}$

for  _$j=2$ to $M$_ do

$C_{1,j}=\min{(C_{1,j-1},D_{1,j-1})}$

for  _$i=2$ to $N$_ do

for  _$j=2$ to $M$_ do

$C_{i,j}=\min{\left(C_{i,j-1},C_{i-1,j}+D_{i,j}\right)}$

return _$C_{N,M}/N$_

Algorithm 1 Algorithm for asymmetric dynamic time warping

#### Tracking implementation

Tracking is done by sequentially predicting individual frames. For better
performance, batching of frames allows for parallel detections and can
drastically reduce execution time. Nevertheless, due to the requirement of
including surrounding frames for each detection, a considerable increase in
memory usage is observed. Once a collection of spline detections is obtained,
each prediction is adapted in order to make it work with the TrackPy Python
library. Due to the peculiarity of our distance metric, we implement a custom
neighbor strategy (see Code Availability) that avoids the assumption of a
symmetric distance function.

It may happen that some detection artefacts appear during the sequential
detection performed during tracking. We have implemented a quick check on the
resulting tracks to make sure not to have stubs, and fix the obvious branching
of tracks due to these artefacts. A slight increase in integrity is observed
on dense clips.

#### Tracking integrity

Given a true label of a track of length $N$, we associate to this track at
each time point $i$ a prediction identity $I_{i}$. We may then define the
integrity of the track as
$\iota=1/N^{2}\sum_{i=1}^{N}\sum_{j=1}^{N}[I_{i}=I_{j}]$. For instance, if a
label is given identities $I=[1,1,1,5,5,5,3,3,3]$ during the track, i.e. there
have been two identity swaps, we find $\iota=\frac{1}{3}$, which has the
interpretation that the track was correct for a third of the time. This
measure will in general scale like $\iota\sim N^{-1}$, as longer tracks will
have a higher likelihood of identity swaps.

#### Acknowledgments

Video of C. elegans were provided by Celia Raimondi, Sunehera Sarwat and
Michele Perni. This work was supported by the Novo Nordisk Foundation, Grant
Agreement NNF20OC0062047.

## References

  * [1] Nikhil R Pal and Sankar K Pal “A review on image segmentation techniques” Publisher: Elsevier In _Pattern recognition_ 26.9, 1993, pp. 1277–1294
  * [2] Dinesh D Patil and Sonal G Deore “Medical image segmentation: a review” In _International Journal of Computer Science and Mobile Computing_ 2.1, 2013, pp. 22–27
  * [3] Dzung L Pham, Chenyang Xu and Jerry L Prince “A survey of current methods in medical image segmentation” In _Annual review of biomedical engineering_ 2.3, 2000, pp. 315–337
  * [4] Anne E Carpenter et al. “CellProfiler: image analysis software for identifying and quantifying cell phenotypes” Publisher: Springer In _Genome biology_ 7.10, 2006, pp. 1–11
  * [5] Rainer Pepperkok and Jan Ellenberg “High-throughput fluorescence microscopy for systems biology” Publisher: Nature Publishing Group In _Nature reviews Molecular cell biology_ 7.9, 2006, pp. 690–696
  * [6] Juan C Caicedo et al. “Data-analysis strategies for image-based cell profiling” Publisher: Nature Publishing Group In _Nature methods_ 14.9, 2017, pp. 849–863
  * [7] Yann LeCun, Yoshua Bengio and Geoffrey Hinton “Deep learning” Publisher: Nature Publishing Group In _Nature_ 521.7553, 2015, pp. 436–444
  * [8] Fuyong Xing et al. “Deep learning in microscopy image analysis: A survey” Publisher: IEEE In _IEEE transactions on neural networks and learning systems_ 29.10, 2017, pp. 4550–4568
  * [9] Erick Moen et al. “Deep learning for cellular image analysis” Publisher: Nature Publishing Group In _Nature methods_ 16.12, 2019, pp. 1233–1246
  * [10] David A Van Valen et al. “Deep learning automates the quantitative analysis of individual cells in live-cell imaging experiments” Publisher: Public Library of Science San Francisco, CA USA In _PLoS computational biology_ 12.11, 2016, pp. e1005177
  * [11] Thorsten Falk et al. “U-Net: deep learning for cell counting, detection, and morphometry” Publisher: Nature Publishing Group In _Nature methods_ 16.1, 2019, pp. 67–70
  * [12] Olaf Ronneberger, Philipp Fischer and Thomas Brox “U-net: Convolutional networks for biomedical image segmentation” In _International Conference on Medical image computing and computer-assisted intervention_ Springer, 2015, pp. 234–241
  * [13] Carsen Stringer, Tim Wang, Michalis Michaelos and Marius Pachitariu “Cellpose: a generalist algorithm for cellular segmentation” Publisher: Nature Publishing Group In _Nature methods_ 18.1, 2021, pp. 100–106
  * [14] Noah F Greenwald et al. “Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning” Publisher: Nature Publishing Group In _Nature biotechnology_ 40.4, 2022, pp. 555–565
  * [15] Reka Hollandi et al. “nucleAIzer: a parameter-free deep learning framework for nucleus segmentation using image style transfer” Publisher: Elsevier In _Cell Systems_ 10.5, 2020, pp. 453–458
  * [16] Arif Ahmed Sekh et al. “Physics-based machine learning for subcellular segmentation in living cells” Number: 12 Publisher: Nature Publishing Group In _Nature Machine Intelligence_ 3.12, 2021, pp. 1071–1080 DOI: 10.1038/s42256-021-00420-0
  * [17] Paul Lang, Karen Yeow, Anthony Nichols and Alexander Scheer “Cellular imaging in drug discovery” Publisher: Nature Publishing Group In _Nature Reviews Drug Discovery_ 5.4, 2006, pp. 343–356
  * [18] Mitko Veta, Paul J van Diest and Josien PW Pluim “Cutting out the middleman: measuring nuclear area in histopathology slides without segmentation” In _International conference on medical image computing and computer-assisted intervention_ Springer, 2016, pp. 632–639
  * [19] Andre Esteva et al. “Dermatologist-level classification of skin cancer with deep neural networks” Publisher: Nature Publishing Group In _nature_ 542.7639, 2017, pp. 115–118
  * [20] Nicolas Coudray et al. “Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning” Publisher: Nature Publishing Group In _Nature medicine_ 24.10, 2018, pp. 1559–1567
  * [21] Mahdieh Poostchi et al. “Image analysis and machine learning for detecting malaria” Publisher: Elsevier In _Translational Research_ 194, 2018, pp. 36–55
  * [22] Vibor Laketa “Microscopy in infectious disease research—Imaging across scales” Publisher: Elsevier In _Journal of molecular biology_ 430.17, 2018, pp. 2612–2625
  * [23] Gordon J Berman “Measuring behavior across scales” Publisher: Springer In _BMC biology_ 16.1, 2018, pp. 1–11
  * [24] John W Krakauer et al. “Neuroscience needs behavior: correcting a reductionist bias” Publisher: Elsevier In _Neuron_ 93.3, 2017, pp. 480–490
  * [25] Gopal P Sarma et al. “OpenWorm: overview and recent advances in integrative biological simulation of Caenorhabditis elegans” Publisher: The Royal Society In _Philosophical Transactions of the Royal Society B_ 373.1758, 2018, pp. 20170382
  * [26] Kelsey M Hallinen et al. “Decoding locomotion from population neural activity in moving C. elegans” Publisher: eLife Sciences Publications Limited In _Elife_ 10, 2021, pp. e66135
  * [27] Linda Turner, Liam Ping, Marianna Neubauer and Howard C Berg “Visualizing flagella while tracking bacteria” Publisher: Elsevier In _Biophysical journal_ 111.3, 2016, pp. 630–639
  * [28] Marco Polin et al. “Chlamydomonas swims with two “gears” in a eukaryotic version of run-and-tumble locomotion” Publisher: American Association for the Advancement of Science In _Science_ 325.5939, 2009, pp. 487–490
  * [29] Liang Li, Edward C Cox and Henrik Flyvbjerg “‘Dicty dynamics’: Dictyostelium motility as persistent random motion” Publisher: IOP Publishing In _Physical biology_ 8.4, 2011, pp. 046006
  * [30] Andrew Adamatzky “Neuroscience without neurons” _eprint: https://aip.scitation.org/doi/pdf/10.1063/5.0082008 In _AIP Conference Proceedings_ 2425.1, 2022, pp. 390001 DOI: 10.1063/5.0082008
  * [31] David Kokel and Randall T Peterson “Using the zebrafish photomotor response for psychotropic drug screening” In _Methods in cell biology_ 105 Elsevier, 2011, pp. 517–524
  * [32] Linda P O’Reilly et al. “C. elegans in high-throughput drug discovery” Publisher: Elsevier In _Advanced drug delivery reviews_ 69, 2014, pp. 247–253
  * [33] Demetrio Raldua and Benjamin Pina “In vivo zebrafish assays for analyzing drug toxicity” Publisher: Taylor & Francis In _Expert opinion on drug metabolism & toxicology_ 10.5, 2014, pp. 685–697
  * [34] Adam Michael Stewart, Robert Gerlai and Allan V Kalueff “Developing highER-throughput zebrafish screens for in-vivo CNS drug discovery” Publisher: Frontiers Media SA In _Frontiers in behavioral neuroscience_ 9, 2015, pp. 14
  * [35] Michele Perni et al. “A natural product inhibits the initiation of α-synuclein aggregation and suppresses its toxicity” Publisher: National Acad Sciences In _Proceedings of the National Academy of Sciences_ 114.6, 2017, pp. E1009–E1017
  * [36] Joseph Redmon, Santosh Divvala, Ross Girshick and Ali Farhadi “You only look once: Unified, real-time object detection” In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 779–788
  * [37] Talmo D. Pereira et al. “Fast animal pose estimation using deep neural networks” Number: 1 Publisher: Nature Publishing Group In _Nature Methods_ 16.1, 2019, pp. 117–125 DOI: 10.1038/s41592-018-0234-5
  * [38] Wei Geng et al. “Automatic tracking, feature extraction and classification of C. elegans phenotypes” Publisher: IEEE In _IEEE transactions on biomedical engineering_ 51.10, 2004, pp. 1811–1820
  * [39] Nicolas Roussel, Christine A Morton, Fern P Finger and Badrinath Roysam “A computational model for C. elegans locomotory behavior: application to multiworm tracking” Publisher: IEEE In _IEEE transactions on biomedical engineering_ 54.10, 2007, pp. 1786–1797
  * [40] Michele Perni et al. “Massively parallel C. elegans tracking provides multi-dimensional fingerprints for phenotypic discovery” Publisher: Elsevier In _Journal of neuroscience methods_ 306, 2018, pp. 57–67
  * [41] Jirapat Likitlersuang, Greg Stephens, Konstantine Palanski and William S Ryu “C. elegans tracking and behavioral measurement” In _JoVE (Journal of Visualized Experiments)_ , 2012, pp. e4094
  * [42] Veikko F Geyer, Frank Jülicher, Jonathon Howard and Benjamin M Friedrich “Cell-body rocking is a dominant mechanism for flagellar synchronization in a swimming alga” Publisher: National Acad Sciences In _Proceedings of the National Academy of Sciences_ 110.45, 2013, pp. 18058–18063
  * [43] Kirsty Y Wan, Kyriacos C Leptos and Raymond E Goldstein “Lag, lock, sync, slip: the many ‘phases’ of coupled flagella” Publisher: The Royal Society In _Journal of the Royal Society Interface_ 11.94, 2014, pp. 20131160
  * [44] Nikzad Babaii Rizvandi, Aleksandra Pižurica, Filip Rooms and Wilfried Philips “Skeleton analysis of population images for detection of isolated and overlapped nematode C. elegans” In _2008 16th European signal processing conference_ IEEE, 2008, pp. 1–5
  * [45] Pascal Laube, Matthias O Franz and Georg Umlauf “Deep learning parametrization for B-spline curve approximation” In _2018 International Conference on 3D Vision (3DV)_ IEEE, 2018, pp. 691–699
  * [46] Jun Gao et al. “Deepspline: Data-driven reconstruction of parametric curves and surfaces” In _arXiv preprint arXiv:1901.03781_ , 2019
  * [47] Soham Mandal and Virginie Uhlmann “Splinedist: Automated cell segmentation with spline curves” In _2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI)_ IEEE, 2021, pp. 1082–1086
  * [48] Greg J Stephens, Bethany Johnson-Kerner, William Bialek and William S Ryu “Dimensionality and dynamics in the behavior of C. elegans” Publisher: Public Library of Science San Francisco, USA In _PLoS computational biology_ 4.4, 2008, pp. e1000028
  * [49] André EX Brown et al. “A dictionary of behavioral motifs reveals clusters of genes affecting Caenorhabditis elegans locomotion” Publisher: National Acad Sciences In _Proceedings of the National Academy of Sciences_ 110.2, 2013, pp. 791–796
  * [50] Tosif Ahamed, Antonio C Costa and Greg J Stephens “Capturing the continuous complexity of behaviour in Caenorhabditis elegans” Publisher: Nature Publishing Group In _Nature Physics_ 17.2, 2021, pp. 275–283
  * [51] Noel R Krieg, Joseph P Tomelty and J Scott Wells Jr “Inhibitio of Flagellar Coordination in Spirillum volutans” Publisher: Am Soc Microbiol In _Journal of Bacteriology_ 94.5, 1967, pp. 1431–1436
  * [52] Dhruv K Vig and Charles W Wolgemuth “Swimming dynamics of the Lyme disease spirochete” Publisher: APS In _Physical review letters_ 109.21, 2012, pp. 218104
  * [53] David J. Hampson “The Spirochete Brachyspira pilosicoli, Enteric Pathogen of Animals and Humans” In _Clinical Microbiology Reviews_ 31.1, 2017, pp. e00087–17 DOI: 10.1128/CMR.00087-17
  * [54] Navish Wadhwa and Howard C Berg “Bacterial motility: machinery and mechanisms” Publisher: Nature Publishing Group In _Nature Reviews Microbiology_ 20.3, 2022, pp. 161–173
  * [55] David M Woolley, Rachel F Crockett, William DI Groom and Stuart G Revell “A study of synchronisation between the flagella of bull spermatozoa, with related observations” Publisher: Company of Biologists In _Journal of Experimental Biology_ 212.14, 2009, pp. 2215–2223
  * [56] Trine B. Haugen et al. “VISEM: a multimodal video dataset of human spermatozoa” In _Proceedings of the 10th ACM Multimedia Systems Conference_ , MMSys ’19 New York, NY, USA: Association for Computing Machinery, 2019, pp. 261–266 DOI: 10.1145/3304109.3325814
  * [57] Linda Turner, William S Ryu and Howard C Berg “Real-time imaging of fluorescent flagellar filaments” Publisher: Am Soc Microbiol In _Journal of bacteriology_ 182.10, 2000, pp. 2793–2801
  * [58] Laurence G Wilson, Lucy M Carter and Sarah E Reece “High-speed holographic microscopy of malaria parasites reveals ambidextrous flagellar waveforms” Publisher: National Acad Sciences In _Proceedings of the National Academy of Sciences_ 110.47, 2013, pp. 18769–18774
  * [59] Joseph Redmon and Ali Farhadi “Yolov3: An incremental improvement” In _arXiv preprint arXiv:1804.02767_ , 2018
  * [60] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun “Deep residual learning for image recognition” In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778
  * [61] Kaiming He, Georgia Gkioxari, Piotr Dollár and Ross Girshick “Mask r-cnn” In _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969
  * [62] Tsung-Yi Lin et al. “Focal loss for dense object detection” In _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988
  * [63] Jiuxiang Gu et al. “Recent advances in convolutional neural networks” Publisher: Elsevier In _Pattern recognition_ 77, 2018, pp. 354–377
  * [64] Piali Sengupta and Aravinthan DT Samuel “Caenorhabditis elegans: a model system for systems neuroscience” Publisher: Elsevier In _Current opinion in neurobiology_ 19.6, 2009, pp. 637–643
  * [65] Maria Markaki and Nektarios Tavernarakis “Caenorhabditis elegans as a model system for human diseases” Publisher: Elsevier In _Current opinion in biotechnology_ 63, 2020, pp. 118–125
  * [66] Jesse M Gray, Joseph J Hill and Cornelia I Bargmann “A circuit for navigation in Caenorhabditis elegans” Publisher: National Acad Sciences In _Proceedings of the National Academy of Sciences_ 102.9, 2005, pp. 3184–3191
  * [67] Catherine H Rankin, Christine DO Beck and Catherine M Chiba “Caenorhabditis elegans: a new model system for the study of learning and memory” Publisher: Elsevier In _Behavioural brain research_ 37.1, 1990, pp. 89–92
  * [68] Michael R Klass “A method for the isolation of longevity mutants in the nematode Caenorhabditis elegans and initial results” Publisher: Elsevier In _Mechanisms of ageing and development_ 22.3, 1983, pp. 279–286
  * [69] CH Opperman and S Chang “Effects of Aldicarb and Fenamiphos on Acetycholinesterase and Motility of Caenorhabditis elegans” Publisher: Society of Nematologists In _Journal of Nematology_ 23.1, 1991, pp. 20
  * [70] Sheng Fong et al. “Energy crisis precedes global metabolic failure in a novel Caenorhabditis elegans Alzheimer Disease model” Publisher: Nature Publishing Group In _Scientific reports_ 6.1, 2016, pp. 1–9
  * [71] Amy L Lee, Hailey M Ung, L Paul Sands and Elise A Kikis “A new Caenorhabditis elegans model of human huntingtin 513 aggregation and toxicity in body wall muscles” Publisher: Public Library of Science San Francisco, CA USA In _PloS one_ 12.3, 2017, pp. e0173644
  * [72] Michele Perni et al. “Multistep inhibition of α-synuclein aggregation and toxicity in vitro and in vivo by trodusquemine” Publisher: ACS Publications In _ACS chemical biology_ 13.8, 2018, pp. 2308–2319
  * [73] D Dipon Ghosh et al. “C. elegans discriminates colors to guide foraging” Publisher: American Association for the Advancement of Science In _Science_ 371.6533, 2021, pp. 1059–1063
  * [74] James F Morley, Heather R Brignull, Jill J Weyers and Richard I Morimoto “The threshold for polyglutamine-expansion protein aggregation and cellular toxicity is dynamic and influenced by aging in Caenorhabditis elegans” Publisher: National Acad Sciences In _Proceedings of the National Academy of Sciences_ 99.16, 2002, pp. 10417–10422
  * [75] Jesse M Gray et al. “Oxygen sensation and social feeding mediated by a C. elegans guanylate cyclase homologue” Publisher: Nature Publishing Group In _Nature_ 430.6997, 2004, pp. 317–322
  * [76] Linjiao Luo et al. “Olfactory behavior of swimming C. elegans analyzed by measuring motile responses to temporal variations of odorants” Publisher: American Physiological Society In _Journal of neurophysiology_ 99.5, 2008, pp. 2617–2625
  * [77] Terence I. Moy et al. “High Throughput Screen for Novel Antimicrobials using a Whole Animal Infection Model” In _ACS chemical biology_ 4.7, 2009, pp. 527–533 DOI: 10.1021/cb900084v
  * [78] Annelie Persson et al. “Natural variation in a neural globin tunes oxygen sensing in wild Caenorhabditis elegans” Publisher: Nature Publishing Group In _Nature_ 458.7241, 2009, pp. 1030–1033
  * [79] Raphael Sznitman et al. “Multi-environment model estimation for motility analysis of Caenorhabditis elegans” Publisher: Public Library of Science San Francisco, USA In _PLoS One_ 5.7, 2010, pp. e11631
  * [80] Steven D Buckingham, Frederick A Partridge and David B Sattelle “Automated, high-throughput, motility analysis in Caenorhabditis elegans and parasitic nematodes: Applications in the search for new anthelmintics” Publisher: Elsevier In _International Journal for Parasitology: Drugs and Drug Resistance_ 4.3, 2014, pp. 226–232
  * [81] Jeong-Hoon Hahm et al. “C. elegans maximum velocity correlates with healthspan and is maintained in worms with an insulin receptor mutation” Publisher: Nature Publishing Group In _Nature communications_ 6.1, 2015, pp. 1–7
  * [82] He-Peng Zhang, Avraham Be’er, E-L Florin and Harry L Swinney “Collective motion and density fluctuations in bacterial colonies” Publisher: National Acad Sciences In _Proceedings of the National Academy of Sciences_ 107.31, 2010, pp. 13626–13630
  * [83] Nicolas Roussel, Jeff Sprenger, Susan J Tappan and Jack R Glaser “Robust tracking and quantification of C. elegans body shape and locomotion through coiling, entanglement, and omega bends” Issue: 4 In _Worm_ 3 Taylor & Francis, 2014, pp. e982437
  * [84] Sijie Jason Wang and Zhao-Wen Wang “Track-a-worm, an open-source system for quantitative assessment of C. elegans locomotory and bending behavior” Publisher: Public Library of Science San Francisco, USA In _PloS one_ 8.7, 2013, pp. e69653
  * [85] Zhaoyang Feng et al. “An imaging system for standardized quantitative analysis of C. elegans behavior” Publisher: Springer In _BMC bioinformatics_ 5.1, 2004, pp. 1–6
  * [86] Ebraheem Fontaine, Alan H Barr and Joel W Burdick “Tracking of multiple worms and fish for biological studies” In _ICCV Workshop on Dynamical Vision_ Citeseer, 2007
  * [87] Daniel Ramot et al. “The Parallel Worm Tracker: a platform for measuring average speed and drug-induced paralysis in nematodes” Publisher: Public Library of Science San Francisco, USA In _PloS one_ 3.5, 2008, pp. e2208
  * [88] Nicholas A Swierczek, Andrew C Giles, Catharine H Rankin and Rex A Kerr “High-throughput behavioral analysis in C. elegans” Publisher: Nature Publishing Group In _Nature methods_ 8.7, 2011, pp. 592–598
  * [89] Kathleen Bates, Kim N. Le and Hang Lu “Deep learning for robust and flexible tracking in behavioral studies for C. elegans” Publisher: Public Library of Science In _PLOS Computational Biology_ 18.4, 2022, pp. e1009942 DOI: 10.1371/journal.pcbi.1009942
  * [90] Shoubhik Chandan Banerjee, Khursheed Ahmad Khan and Rati Sharma “Deep-Worm-Tracker: Deep Learning Methods for Accurate Detection and Tracking for Behavioral Studies in C. elegans” Pages: 2022.08.18.504475 Section: New Results bioRxiv, 2022 DOI: 10.1101/2022.08.18.504475
  * [91] Anthony D Fouad et al. “High-throughput imaging of Caenorhabditis elegans aging using collective activity monitoring” Publisher: Cold Spring Harbor Laboratory In _bioRxiv_ , 2021
  * [92] Laetitia Hebert et al. “WormPose: Image synthesis and convolutional networks for pose estimation in C. elegans” Publisher: Public Library of Science San Francisco, CA USA In _PLoS computational biology_ 17.4, 2021, pp. e1008914
  * [93] Sara Beery et al. “Context R-CNN: Long Term Temporal Context for Per-Camera Object Detection”, 2020, pp. 13075–13085 URL: https://openaccess.thecvf.com/content_CVPR_2020/html/Beery_Context_R-CNN_Long_Term_Temporal_Context_for_Per-Camera_Object_Detection_CVPR_2020_paper.html
  * [94] B. Moghaddam and A. Pentland “Probabilistic visual learning for object detection” In _Proceedings of IEEE International Conference on Computer Vision_ , 1995, pp. 786–793 DOI: 10.1109/ICCV.1995.466858
  * [95] Eric E. Keaveny and André E.. Brown “Predicting path from undulations for C. elegans using linear and nonlinear resistive force theory” Publisher: IOP Publishing In _Physical Biology_ 14.2, 2017, pp. 025001 DOI: 10.1088/1478-3975/aa5ce6
  * [96] John C Crocker and David G Grier “Methods of digital video microscopy for colloidal studies” Publisher: Elsevier In _Journal of colloid and interface science_ 179.1, 1996, pp. 298–310
  * [97] Amirhossein Kazerouni et al. “Diffusion Models for Medical Image Analysis: A Comprehensive Survey” arXiv, 2022 DOI: 10.48550/arXiv.2211.07804