# Cell Tracking-by-detection using Elliptical Bounding Boxes

Lucas N. Kirsten
Institute of Informatics
Universidade Federal do Rio Grande do Sul
Porto Alegre, RS (Brazil)
<EMAIL_ADDRESS>
& Cláudio R. Jung
Institute of Informatics
Universidade Federal do Rio Grande do Sul
Porto Alegre, RS (Brazil)
<EMAIL_ADDRESS>

###### Abstract

Cell detection and tracking are paramount for bio-analysis. Recent approaches
rely on the tracking-by-model evolution paradigm, which usually consists of
training end-to-end deep learning models to detect and track the cells on the
frames with promising results. However, such methods require extensive amounts
of annotated data, which is time-consuming to obtain and often requires
specialized annotators. This work proposes a new approach based on the
classical tracking-by-detection paradigm that alleviates the requirement of
annotated data. More precisely, it approximates the cell shapes as oriented
ellipses and then uses generic-purpose oriented object detectors to identify
the cells in each frame. We then rely on a global data association algorithm
that explores temporal cell similarity using probability distance metrics,
considering that the ellipses relate to two-dimensional Gaussian
distributions. Our results show that our method can achieve detection and
tracking results competitively with state-of-the-art techniques that require
considerably more extensive data annotation. Our code is available at:
https://github.com/LucasKirsten/Deep-Cell-Tracking-EBB.

_Keywords_ Cell tracking $\cdot$ Cell detection $\cdot$ Oriented object
detection

##  1 Introduction

Detection and tracking of living cells in microscopy images is a crucial task
required in many biomedical applications, such as cell growth, migration,
invasion, morphological changes, and changes in the localization of molecules
within cells [1, 2, 3, 4]. The sheer amount of data produced by high-
throughput microscopy imaging imposes an analytical challenge for science
researchers, which can only be overcome with the appropriate computational
tools.

As with several other computer vision tasks, the state-of-the-art (SOTA) for
cell detection and tracking is based on deep learning approaches [5, 6]. These
techniques typically require manual cell annotations for training and
evaluating the models, and the annotation format has a significant impact on
both the time devoted to image labeling and the complexity of the network
itself. The most traditional object representation refers to using horizontal
bounding boxes (HBBs, a.k.a. BBs) to detect objects in a scene. Despite being
very simple to annotate, this representation is not adequate when dealing with
oriented elongated objects, since the HBB may contain large portions of the
background or other objects in clutter scenarios. On the other hand,
segmenting each object provides a fine-grained representation of the shape,
but it is a tedious and time-consuming task. Moreover, applications that use
multiple cell lineages from different sources (e.g., microscope, cell type)
may require several rounds of labeling data and retraining the models [7]. In
these cases, a fast and efficient method for quickly labeling the data is
crucial for the application continuity, since it is usually the most time-
consuming step.

In the context of cell detection and tracking, knowing the complete shape
representation might not be needed, while it can impose a real challenge in
cases where the cell contour is highly uncertain (e.g., when there is low
contrast between the cells and the background). Furthermore, methods used for
individually segmenting the cell masks usually require more complex and
computationally expensive algorithms, since they are typically developed in a
two-step manner (either detecting and then segmenting [8, 9], or segmenting
and then splitting the masks [10, 11, 12, 13, 14]). Oriented bounding boxes
(OBBs, a.k.a. rotated bounding boxes) are an intermediate representation
between segmentation masks and HBBs with a good compromise between simplicity
and completeness. However, the presence of roughly circular cells imposes
angular ambiguity on its representation, since its OBB representation will be
a square rotated at any angle (see the left cell on Figure 1).

In this work, we advocate using elliptical bounding boxes (EBBs), which can be
directly derived from OBBs and can capture oriented cells while mitigating the
angular ambiguity for circular objects. Fig. 1 shows a comparison of HBBs,
OBBs, segmentation masks, and EBBs for two different cells: the left one is
roughly circular, and right one is oriented. The EBB representation, shown in
blue, presents a good fit in both examples. As an additional advantage, the
orientation/shape of detected cells represented as EBBs can be explored in
tracking-by-detection approaches to provide a better spatio-temporal
association when time-lapse sequences are used, while weakly-supervised
segmentation methods such as [15] can be coupled to the detected OBBs/EBBs to
obtain a more detailed representation of the cell shape.

Figure 1: Comparison of different types of annotations. In green are the full
segmentation mask, in yellow the HBB representation, in red the OBB
representation, and in blue the EBB representation.

We propose a cell tracking-by-detection method that uses a deep learning model
to detect the cells as OBBs, then convert them to EBBs to fit the cell shapes
better. For the tracking part, we based our solution on the work of Bise et
al. [10], which describes an unsupervised (i.e., no tracking label is
necessary) long-term global data association algorithm. We adapted their
algorithm to rely only on the detection information provided by the object
detector model (e.g., position, confidence score). It is important to note
that our method only requires OBB cell annotations for isolated frames, and no
tracking annotation involving temporal sequences is needed.

##  2 Related work

###  2.1 Cell detection and segmentation

There are several approaches for cell counting, detection, and segmentation,
and the best results have been achieved by deep learning methods [16, 17].
These methods vary considerably regarding the underlying structure of the
network and also on the degree of supervision required to label training data.
For example, detecting just the cell nucleus requires one pixel-per-cell as
supervision; detecting the cell boundaries as an HBB requires two points (top-
left and bottom-right), while OBBs require an additional parameter related to
the orientation; finally, segmentation requires the identification of all
pixels belonging to a cell, which is very time-consuming. There are also some
intermediate shape representations such as the star-convex polygons [18].

The U-Net presented in [11] has become a popular segmentation approach in
biomedical applications when segmentation-level information is required. It is
based on an encoder-decoder U-shaped architecture with skip connections, and
focuses mostly on semantic segmentation tasks (i.e., segmenting all objects
for each class altogether). Although the connected components produced by
U-Net can also be used for _instance segmentation_ , dense scenarios or
situations with strongly overlapping cells are challenging. For instance
segmentation, most approaches produce an embedding vector for each image pixel
in such a way that similar vectors should relate to the same instance (object)
[19], and have been explored in the context of cell segmentation by Payer and
colleagues [20, 21] by using a cosine loss for estimating local embedding
distances. To mitigate the cost of clustering pixels embedding (that is
required to obtain the final instance segmentation), Zhao and colleagues [22]
proposed a fast Mean-Shift algorithm that works on GPUs. The panoptic
segmentation task combines both category- and instance-level information into
a single framework, and has been applied to biomedical imaging by Liu and
colleagues [23].

Although solutions that can return the full segmentation mask of cells have
the clear advantage of providing a complete shape representation, most
segmentation approaches are fully supervised, requiring time-consuming per-
cell annotations. In order to overcome this problem, weakly supervised
approaches explore partial annotations such as bounding boxes [24], center-
points [25] or user-defined scribbles [26] for the segmentation task.
Nevertheless, many applications (e.g., tissue engineering [27]) do not require
the full description of the cell shape, but rather a position, size, and
orientation descriptor are enough. This alleviates the annotation process,
making it faster, less tedious, and more scalable.

As noted in [18], a popular approach for cell detection in microscopy images
is to use generic-purpose object detectors based on HBBs, such as SSD [28] or
YOLO [29]. Despite the constant evolution of object detectors [30, 31, 32,
33], the representation of cells as HBBs presents limitations in denser
scenarios, particularly when oriented and elongated cells are present as
aforementioned. Intermediate representations between HBBs and full
segmentation masks have also been explored for cell detection. For example,
Schmidt et al. [18] presented a polygonal shape representation based on radial
sweeps with equidistant angles. A similar approach using splines instead of
polygonal representations was presented in [34], obtaining smoother cell
boundaries.

In this work, we advocate using EBB representations for cell detection, which
is a natural extension of the OBB representation, but usually provides a
better fit to the cell shape. In particular, roughly circular shapes induce a
naturally ambiguous angular representation when OBBs are used since any
rotated square fits equally to a circle. For these cases, the EBB would reduce
to a circle, mitigating the angular problem. Finally, it is worth mentioning
that the output of any OBB detector can be mapped to an EBB, and OBB detectors
have demonstrated exciting results in the past years [35, 36, 37, 38]. An OBB
annotation is almost as simple as an HBB, with a considerable gain in shape
representation, particularly regarding applications that involve elongated
cells.

###  2.2 Cell tracking

Firstly, it is important to emphasize that “cell tracking” presents a broad
spectrum of challenges, since each cell type and application might require a
tailored solution. As noted in [7], “there is no simple way to point out the
right algorithm for a given dataset”, hinting that finding an algorithm
capable of working for a broad spectrum of cell lineages is a huge challenge.
Nevertheless, the ISBI Challenge [39] has provided several different datasets
for benchmarking cell tracking algorithms, which usually are elaborated to
work on more than one cell type and lineage.

Recent cell tracking algorithms can be broadly divided into two categories
[40, 41, 14]: (i) tracking by model evolution and (ii) tracking-by-detection.
In tracking by model evolution methods (a.k.a. end-to-end tracking systems),
detection and tracking are solved simultaneously. In this context, Payer et
al. [20, 21] introduced temporal information for spatio-temporal learning of
the embeddings. They explored a cosine loss for estimating local embedding
distances, and used a convolutional Gated Recursive Unit (ConvGRU) to learn
temporal relationships. Nishimura et al. [42] presented a cell tracking
approach that works with weak annotation (cell centers in successive frames)
by exploring a co-detection CNN. More recently, Hayashida et al. [5] proposed
a complete pipeline that uses spatial-temporal context in multiple frames and
long-term motion estimation with an objective level warping loss that
addresses the problem of detecting and tracking highly dense cell images.
Although these methods are capable of achieving high performance, it is
important to emphasize that they all require full annotations for both the
detection/segmentation and tracking steps, which might be a strong limitation.

The tracking-by-detection paradigm consists of two stages: cell detection and
cell association. When segmentation masks are required, the detection stage
can either aid a segmentation step to extract the masks of each detected cell
[8], or it can directly infer the segmentation masks and then split those
wrongly joined cells using some algorithm such as the watershed [43, 44, 40,
41, 11, 12]. The methods for connecting the cells in subsequent frames usually
rely on graphs and linear integer programming [43, 44, 40, 41, 8] by defining
the costs of the cell events (e.g., movement, mitoses, and apoptosis). Other
strategies include the use of multi-Bernoulli random finite sets [45] or joint
particle filters based on Markov random field to model the dependency of the
target movements [46]. Akram et al. [8] uses a deep learning model that first
detects the cells using the HBB representation and then feeds these detections
to a segmentation model to further retrieve the segmentation masks of
individual cells. For associating the detection, they use random forests to
estimate the costs of the event graph. More recently, Wang et al. [14]
proposed a method that first segments the images using a U-Net [11] deep
learning model, and then uses deep reinforcement learning to associate the
detected targets between frames.

This work proposes a tracking-by-detection approach that solely uses spatial
information and the detection scores from an object detector to determine the
associations between cells. It alleviates the annotation requirements to use
only annotated cells as OBBs, and also eliminates the necessity of using other
features from the detections (e.g., image histograms) to compute the
associations.

##  3 The proposed tracking-by-detection method

Our approach follows the typical pipeline of a tracking-by-detection method.
First, an object detector is used to identify the cells in each frame using
OBB representations, which are then converted to elliptical representations
(EBBs). For tracking, we initially generate short tracklets by joining cells
in subsequent frames with an objective function that jointly explores the
shape and distance of cells. More precisely, we map the EBB representation to
a two-dimensional Gaussian distribution and explore the Helinger distance,
which directly correlates to the IoU metric [47]. Finally, a global data
association method based on the work of [10] associates the tracklets to
obtain the final cell trajectories and lineage trees. Figure 2 shows an
overview of our complete pipeline for cell tracking-by-detection, and the
steps are detailed next.

Figure 2: Overview of the proposed pipeline for cell-tracking-by-detection.
First, we detect the cells as OBBs and then convert them to the EBB
representation. Next, we join the cells with high overlap between two adjecent
frames. Finally, a global data association algorithm is used to identify all
the cell events (i.e., movement, mitoses and apoptosis), while filling gaps
generated by false negative detections and removing false positives ones, in
order to produce the final tracklets.

###  3.1 Cell detection

The first step of our method relies on identifying the cells for each frame
individually. We propose to use off-the-shelf OBB object detectors trained
with cell images and then convert the output to elliptical bounding boxes
(EBBs). For an OBB with center $(x,y)$, width $W$, height $H$, and orientation
$\theta$, we generate an ellipse with the same center and orientation, with
semi-axes $a=W/2$ and $b=H/2$. If the OBB is clearly oriented (i.e.,
$W>\\!\\!>H$ or $W<\\!\\!<H$), the EBB will preserve the orientation of the
OBB. On the other hand, if the OBB is roughly square (i.e., $H\approx W$), the
produced EBB will be roughly circular. In the case of a perfect square, the
EBB simplifies to a single circle regardless of the orientation of the OBB,
which mitigates the orientation ambiguity (recall the example shown in Fig.
1).

###  3.2 Detection filtering and suppression

In a typical deep object detector, only candidate detections with scores
larger than a pre-defined threshold $\tau_{s}$ are retrieved. Still, we
usually have several overlapping candidates related to the same object, and
Non-Maximum Suppression (NMS) is then used to retrieve only the candidate with
the highest score. In this step, it is crucial to define a geometrical
similarity measure between the detections for quantifying their “overlap”
degree.

The Intersection-over-Union (IoU) is the de facto standard metric for
computing the overlap in HBB or OBB detectors. However, computing the IoU for
OBBs is not trivial due to the several possibilities for two intersecting OBBs
[48]. Furthermore, the IoU is unreliable for OBB detections related to
circular cells, since angular discrepancies might artificially degrade the IoU
[49]. Using the IoU with EBBs mitigates the latter problem, because the
ellipse reduces to a circle. However, computing the intersection using EBBs is
even more complex than using OBBs since it involves the overlap of two
ellipses. In this work, we propose an alternative similarity metric based on
fuzzy object representations.

In [50, 38] and [47], the core idea is to use 2D Gaussian distributions
(denoted as GBBs – Gaussian Bounding Boxes) as intermediate representations
for oriented objects, and train OBB detectors using loss functions based on
similarity metrics between distributions. Here, we explore their developed
fuzzy representation to compute the distance/similarity, as explained next.
Following [47], an OBB with center $\bm{\mu}=(x_{c},y_{c})^{T}$, width $W$,
height $H$ and angle $\theta$ is mapped to a GBB described by the mean vector
$\bm{\mu}$ (which is the OBB center) and a covariance matrix

$\Sigma=\begin{bmatrix}a&c\\\
c&b\end{bmatrix}=\begin{bmatrix}\frac{W^{2}}{12}\cos^{2}\theta+\frac{H^{2}}{12}\sin^{2}\theta&\frac{1}{2}\left(\frac{W^{2}}{12}-\frac{H^{2}}{12}\right)\sin
2\theta\\\ \frac{1}{2}\left(\frac{W^{2}}{12}-\frac{H^{2}}{12}\right)\sin
2\theta&\frac{W^{2}}{12}\sin^{2}\theta+\frac{H^{2}}{12}\cos^{2}\theta\end{bmatrix}.$
(1)

Note that square OBBs, for which $H=W$, generate a diagonal covariance matrix
that does not involve the angular parameter $\theta$. Hence, squares that
differ only by angle are mapped to the exact same GBB.

In [50, 38] and [47], the Gaussian distributions are used to train OBB object
detectors as their loss function. However, the metrics proposed by [50, 38] do
not hold the mathematical properties of a similarity measure, which is the
goal for comparing two detection. Therefore, we define a similarity metric
based on the Hellinger Distance, as done by [47]. Let us consider that
$p\sim\mathcal{N}(\bm{\mu}_{1},\Sigma_{1})$ and
$q\sim\mathcal{N}(\bm{\mu}_{2},\Sigma_{2})$ are Gaussian distributions with

$\bm{\mu}_{1}=\begin{pmatrix}x_{1}\\\
y_{1}\end{pmatrix},~{}\Sigma_{1}=\begin{bmatrix}a_{1}&c_{1}\\\
c_{1}&b_{1}\end{bmatrix},~{}\bm{\mu}_{2}=\begin{pmatrix}x_{2}\\\
y_{2}\end{pmatrix},~{}\Sigma_{2}=\begin{bmatrix}a_{2}&c_{2}\\\
c_{2}&b_{2}\end{bmatrix}.$ (2)

The Bhattacharyya Distance [51] between distributions $p$ and $q$ is given by

$\displaystyle B_{D}(p,q)$
$\displaystyle=\frac{1}{8}\bm{\mu_{12}}^{T}\Sigma^{-1}\bm{\mu_{12}}+\frac{1}{2}\ln\left(\frac{\det\Sigma_{12}}{\sqrt{\det\Sigma_{1}\det\Sigma_{2}}}\right),$
(3) $\displaystyle\bm{\mu_{12}}$
$\displaystyle=\bm{\mu}_{1}-\bm{\mu}_{2},~{}\Sigma_{12}=\frac{1}{2}\left(\Sigma_{1}+\Sigma_{2}\right),$

and the Hellinger distance [52] between $p$ and $q$ is then

$H_{D}(p,q)=\sqrt{1-e^{-B_{D}(p,q)}}.$ (4)

Although both $B_{D}$ and $H_{D}$ are named “distances”, only $H_{D}$
satisfies the mathematical properties for a metric [53]. Moreover, we can see
that $0\leq H_{D}(p,q)\leq 1$ (with $0$ being the maximum similarity), meaning
that $H_{D}$ provides a normalized distance metric. In this work, we explore
$H_{D}$ to find overlapping EBBs to suppress non-maximum detections. Finally,
given a detection $p$ with the highest confidence score, we suppress all
detections $q\neq p$ for which $H_{D}(p,q)<\tau_{h}$. We illustrate this
procedure in Figure 3.

(a)

(b)

Figure 3: Illustration of detection filtering and NMS using the Hellinger
Distance on a frame. Observe that in (a) we have many duplicates and false
positives detection, while in (b) they are suppressed and eliminated,
resulting in only one detection per cell.

###  3.3 Tracklet generation

As stated by [10], long trajectories obtained via frame-by-frame association
may include more failures (such as drift and occlusions) than short
trajectories. Hence, it is better to first reliably associate cell detections
in adjacent frames, and then use some global data association algorithm to
obtain the final long-term tracklets.

In this work, “reliable tracklets” are obtained by computing the overlap
between all detections of subsequent frames with the Hellinger distance (Eq.
(4)), and the optimal association between detections is obtained by solving a
linear sum assignment problem with the Hungarian Algorithm [54]. Note that
$H_{D}$ jointly considers the centroid distance (typically used for tracklet
generation) and shape information – encoded in the covariance matrix. Hence,
nearby cells with distinct shape/orientations that generate strong ambiguity
when using only the centroid distance can be disambiguated through the
Hellinger distance (see Figure 4).

Figure 4: Illustration of using the Hellinger Distance for associating
detections in subsequent frames. The dashed ellipse (black) represents a
detection in frame $i$, while the two solid ellipses (red and blue) represent
two possible associations in frame $i\\!+\\!1$ (the stars mark the respective
centers of the detections). The Hellinger distance eliminates the imprecision
of joining the black and blue detections due to only relating their centroid
distances, while the red and black share more shape similarity.

In order to avoid bad associations caused by false positive detections, we
only associate pairs of cells $p$ and $q$ for which $H_{D}(p,q)<\tau_{o}$,
where $\tau_{o}$ is a similarity threshold. Note that we implicitly assume a
low cell displacement and shape change in adjacent frames, which is typically
the case for time-lapse microscopy imagery [10].

###  3.4 Global data association

In an ideal scenario, the combination of the associations in adjacent frames
would lead to the long-term track of each cell. However, there may be errors
either in cell detection or the association process itself. Furthermore, we
also have to consider cell division and death. Hence, we explore a global data
association step to fill the gaps between disjointed tracklets, remove false
positives and identify the mitoses events.

We based our method on the work by [10], who formulated a maximum-a-posteriori
problem (MAP) solved by linear programming that addresses the tree structure
association problem. We proceed to briefly describe their method and then
introduce our modifications. The MAP problem is solved by defining a set of
hypotheses associated with a likelihood score for combinations of the $N_{T}$
tracklets (i.e., each tracklet will respond to a set of possible hypotheses
with their likelihood). More precisely, they assume the following five
possible types of hypotheses that can be made for each tracklet: initiation,
termination, translation, mitoses, and false positive.

Mathematically, let $\bm{C}_{M\times 2N_{T}}$ be a binary matrix containing
the constraints for all possible hypotheses. Each row of $\bm{C}$ relates to a
single hypothesis, and it presents $2N_{T}$ columns that indicate the possible
tracklet associations, where the first $N_{T}$ columns indicate the index of
the source tracklet index and the following $N_{T}$ columns relate to the
target tracklet index (or indices). For example, the translation hypothesis
presents one source tracklet and one target tracklet. Meanwhile, the mitosis
hypothesis relates one source tracklet to a pair of target tracklets (its
children), and the false positive hypothesis relates one tracklet to itself
(i.e., the source index is the same as the target). There is also a likelihood
value for each hypothesis (i.e., for each row of $\bm{C}$), stored in a vector
$\bm{\rho}$ with $M$ elements (the number of generated hypotheses).

The solution of the global optimization problem is a subset of all non-
conflicting hypotheses (a subset of rows of $\bm{C}$) that maximizes the sum
of the corresponding likelihoods. This can be formulated as the following ILP
optimization problem:

$\bm{x}^{*}=\mathop{\text{argmax}}_{\bm{x}}\bm{\rho}^{T}\bm{x},~{}~{}s.t.~{},\bm{C}^{T}\bm{x}\leq\bm{1},$
(5)

where ${\bm{x}_{M\times 1}}$ is a binary vector, and an entry $x_{k}=1$ means
the ${k\text{-th}}$ hypothesis is selected in the global optimal solution. The
constraint ${\bm{C}^{T}\bm{x}\leq\bm{1}}$ guarantees that each tracklet ID
appears in only one associated tree or false positive tracklet.

The work developed by [10] allows a simple but efficient method for long-term
data association. Yet, their approach still presents some limitations that we
propose to address with the following modifications:

  1. 1.

In [10], they propose to use a specific algorithm [55] to identify the mitotic
cells (i.e., the cells that are more likely to suffer mitoses). This
algorithm, however, is specifically designed to work with images captured
under phase-contrast microscopy, which narrows its usage to other types of
images (e.g., fluorescence). We decided not to differentiate between mitotic
and non-mitotic cells, and consider all cells as potentially mitotic.
Furthermore, to address this choice, we employ two free parameters to adjust
the likelihood distribution of the translation and mitoses hypothesis;

  2. 2.

We removed the initiation and termination hypotheses and replaced them with a
_completeness_ one. In theory, these two hypotheses are essential to define
the tree structure of the MAP problem. However, we noted in our experiments
that they tend to generate a more complex MAP problem and do not provide
better tree structures (i.e., closer to the ground-truth one), as we
demonstrate in Section 5. In the original formulation, these hypotheses are
defined regarding the cell position towards the borders and their first (or
last) appearance, assuming that cells appear or disappear when they enter or
leave the area captured by the microscope. Although these assumptions work
well for an ideal detector, real detectors can fail to capture some cells
(e.g., due to the absence of contrast between cell and background). As a
consequence, the tracklets can start to be recognized only after some frames
and/or with the cells already located farther away from the image boundaries,
and imposing these boundary-related conditions can increase the number of
false negatives;

  3. 3.

We re-defined all the likelihood computations to use only information
regarding the detected cells (e.g., position, time-frame distance, and
confidence score returned by the object detector). In particular, we explore
the confidence score to discriminate between the true and false positive
hypotheses. For the translation and mitoses hypotheses, we used only the
center and time distance between detections, whereas generic feature matching
(e.g., based on histogram matching) was proposed by [10]. Our motivation is
that finding an adequate feature to individually discriminate cells in long-
term matching is challenging (no specific feature was mentioned in [10]), and
appearance-based features might change depending on the method used to capture
the images. Moreover, using only positional information and confidence score
allow using any cell detector. We also removed the true positive likelihood
that was originally proposed by [10] since our pre-processing step can remove
low-scored detections.

The proposed alternate hypotheses and corresponding likelihoods are explained
next.

  * •

Translation hypothesis:
If the time and center distances between the last detection of tracklet
$X_{k1}$ and the first detection of $X_{k2}$ are smaller than a pair of
thresholds, $X_{k1}\rightarrow X_{k2}$ is a candidate of a tracklet
translation. Considering that $h$ denotes the index of a new hypothesis, we
append a new row to $\bm{C}$ and a corresponding likelihood to $\bm{\rho}$ as:

$\displaystyle C(h,i)=\begin{cases}1,&\text{if $i=k_{1}$ or
$i=N_{T}+k_{2}$}\\\ 0,&\text{otherwise}\end{cases},$
$\displaystyle\rho(h)=P_{link}(X_{k2}|X_{k1}).$

  * •

Mitosis hypothesis:
If the time and center distances between the last detection of tracklets
$X_{p}$ and the first detection of $X_{c1}$ and $X_{c2}$ are smaller than a
threshold for the detection center and time,
$X_{p}\rightarrow\\{X_{c1},X_{c2}\\}$ is a candidate of a tracklet mitosis. We
define new entries for $\bm{C}$ and $\bm{\rho}$ as:

$\displaystyle C(h,i)=\begin{cases}1,&\text{if $i=p$ or $i=N_{T}+c_{1}$ or
$i=N_{T}+c_{2}$}\\\ 0,&\text{otherwise}\end{cases},$
$\displaystyle\rho(h)=P_{mit}(X_{c1},X_{c2}|X_{p}).$

  * •

False positive and Completeness hypothesis:
If the score of tracklet $X_{k}$ (i.e., the mean score of all its detections)
is smaller than threshold $\tau_{FP}$, $X_{k}$ is a candidate for both false
positive and completeness. We define two new entries for $\bm{C}$ ($h$ and
$h+1$) and $\bm{\rho}$ given by

$\displaystyle C(h,i)=C(h+1,i)=\begin{cases}1,&\text{if $i=k$ or
$i=N_{T}+k$}\\\ 0,&\text{otherwise}\end{cases},$

and for the $\bm{\rho}$ entries as:

$\displaystyle\rho(h)=P_{FP}(X_{k}),$
$\displaystyle\rho(h+1)=P_{cplt}(X_{k}).$

The completeness hypothesis aims to cover the cases in which a tracklet does
not translate or suffer from mitoses, i.e., the tracklet is, in fact, a full
long-term cell track. Hence, assuming only the false positive hypothesis would
wrongly eliminate those tracklets.

Now, we formalize the likelihoods for each hypothesis provided above. For the
link and mitosis hypotheses, we would like to maximize the likelihood (i.e.,
values closer to 1) when the time and space distance between two cell
tracklets are small, and minimize it (i.e., values closer to 0) when their
distance increase. For this purpose, we based the formulation of the link and
mitoses likelihoods on an exponential mapping that penalizes the time-space
distance between tracklets. More precisely, we propose to use

$P_{link}(X_{j}|X_{i})=\text{exp}\left(-\frac{(c_{i,j}+1)~{}t_{j,i}}{\Delta
t~{}\lambda_{link}}\right),$ (6)

where $\Delta t$ is the dataset capture time step in
$\frac{\text{frames}}{\text{hour}}$, $c_{i,j}$ is the Euclidean center
distance between the last and the first detection of tracklets $X_{i}$ and
$X_{j}$, respectively, $t_{i,j}$ is their time distance in frames, and
$\lambda_{link}$ is a parameter that controls the decay of the exponential.

The mitosis likelihood is defined in a similar way, but jointly considering
the space-time distance between the parent cell $p$ and the two candidate
children $c1$, $c2$. Formally, it is given by

$P_{mit}(X_{c1},X_{c2}|X_{p})=\text{exp}\left(-\frac{(c_{p,c1}+c_{p,c2}+1)(t_{p,c1}+t_{p,c2})}{4\Delta
t~{}\lambda_{mit}}\right),$ (7)

where $\lambda_{mit}$ controls the decay of the exponential. We do not use the
Helinger distance in these formulations because cell shapes can change
considerably in longer-term translations or during mitosis.

For the false positive likelihood, we also explore the precision $\alpha$ of
the object detector computed for a given dataset (e.g., the training or
validation set) and the tracklet score $s_{i}$, defined as the mean confidence
score of all its detection. We want low-confidence tracklets (w.r.t. to the
detector precision) to have an increased false positive likelihood, but longer
tracklets must have a smaller value since they tend to be related to actual
tracklets. Based on these assumptions, the false positive likelihood is
computed as

$P_{FP}(X_{i})=(1-\alpha)(1-s_{i}+\tau_{s})^{|X_{i}|},$ (8)

where $|X_{i}|$ is the number of total detection responses in the tracklet,
and $\tau_{s}$ is the threshold detection score defined in Section 3.2. Note
that detectors with low precision (${\alpha<\\!\\!<1}$) are prone to produce
false positives, and $P_{FP}$ should be increased in this case. On the other
hand, detectors with high precision (${\alpha\approx 1}$) are less prone to
produce false positives, and the likelihood $P_{FP}$ is decreased.

A similar rationale is used to define the completeness tracklet likelihood,
given by

$P_{cplt}(X_{i})=\alpha(s_{i}-\tau_{s}).$ (9)

Note that the likelihood of a tracklet being complete (i.e., it does not fit
any other hypotheses) is not directly related to the tracklet size, since a
cell can emerge and die very soon or appear in the visible field only in the
last frames, which might lead to very small tracklets. Hence, we chose to
define the likelihood based only on how precise the predictions of the object
detector are, so that the tracklet generation step can do most of the correct
associations between adjacent frames.

Figure 5 shows an example of the proposed hypotheses generation. From left to
right, it shows a set of tracklets; the possible hypotheses connecting the
tracklets; the likelihood vector $\bm{\rho}$; the matrix $\bm{C}$ indicating
the possible tracklet connections; and the vector $\bm{x}$ returned by the MAP
problem solution. The final tracks are then obtained by solving Eq. (5).

Figure 5: Example of hypotheses generation for a given set of tracklets. Given
a set of initial tracklets, we generate hypothesis for each of them, which are
associated with a likelihood value stored in vector $\bm{\rho}$, and the
constraints stored in the binary matrix $\bm{C}$. Solving the MAP problem
returns the binary vector $\bm{x}$ which selects a subset of rows in $\bm{C}$
that defines the optimal solution of the MAP problem. For example, the first
line defines a translation hypothesis of tracklet 1 to 2 with a likelihood of
0.5. The matrix $\bm{C}$ stores the ID values of the considered tracklets (in
each of its halves), and the value 0 in the vector $\bm{x}$ indicates that
this hypothesis was not chosen during the optimization solving.

##  4 Experiments

###  4.1 Datasets

We evaluated the proposed method on three publicly available cell microscopy
datasets provided from the ISBI 2015 Cell Tracking Challenge [39]: Fluo-N2DH-
GOWT1, PhC-C2DH-U373, and Fluo-N2DL-HeLa, illustrated in Fig. 6. Each dataset
contains two sequences in the training set (with ground truth annotations) and
two challenge sequences (without ground truth annotation), named with suffices
-01 and -02. The results for the challenge sequences are obtained by
submitting the results to the ISBI challenge server. We proceed to provide
details regarding each of the chosen datasets.

(a) Fluo-N2DH-GOWT1

(b) Fluo-N2DH-HeLa

(c) PhC-C2DH-U373

Figure 6: Example of cell images for each of the datasets used for evaluation.

Fluo-N2DH-GOWT1 contains GFP-transfected GOWT1 mouse embryonic stem cells
captured on fluorescence microscopy. Challenges with this dataset include low
contrast of some cells and few cells entering and exiting the imaged region
from the axial direction. The capture time step is $\Delta
t=12\frac{\text{frames}}{\text{hour}}$.

PhC-C2DH-U373 contains glioblastoma-astrocytoma U373 cells captured under
phase contrast microscopy. This dataset is challenging due to cells having
highly deformable shapes and parts of cell bodies having a similar appearance
to the background. The capture time step is $\Delta
t=4\frac{\text{frames}}{\text{hour}}$.

Fluo-N2DH-HeLa contains fluorescently labeled HeLa nuclei captured on
fluorescence microscopy. Challenges with this dataset include high cell
density, low contrast, a few irregularly shaped cells, various mitoses events,
and cells entering and exiting the imaged region. The capture time step is
$\Delta t=2\frac{\text{frames}}{\text{hour}}$.

All these datasets only contain ground truth (GT) annotations for cells within
a field of interest, which excludes a few pixels for cells close to the image
boundaries. There are two types of GT annotations: cell masks for the
segmentation evaluation, and cell markers for the detection and tracking
evaluation. For the cell masks, the annotations are provided as _silver_ and
_gold_ standards. The silver standard annotations refer to computer-origin
reference annotations, while the gold standard refers to human-origin ones.
Since only a few cells are annotated in the gold standard, we used only the
silver ones for both training and evaluation. The cell marks are “similar” to
the segmentation masks, but they have reduced size and serve solely as a
position descriptor of the cells. Moreover, they do not follow a standard
regarding their size or placement on the cell image. We illustrate these
annotation discrepancies in Fig. 7.

(a) GOWT1-01

(b) HeLa-01

(c) U373-01

(d) GOWT1-02

(e) HeLa-02

(f) U373-02

Figure 7: Illustration showing the difference between the two CTC [39]
annotations types: cell masks (green) and cell markers (red).

###  4.2 Data pre-processing

We use the same data pre-processing procedure for all datasets, except for
some specific adjustments in the HeLa, as described next. Since the HeLa
dataset contains cells with very different sizes compared to the other CTC
datasets [39], we followed a similar strategy to the one employed by [8] and
magnified all images by a factor of 2. This procedure allows us to use the
same network architecture without needing any adjustment in its head
parameters regressor (e.g., the anchors) to better fit the cell sizes in this
dataset.

For training the object detector, we extracted patches of full images in the
datasets using a $512\\!\times\\!512$px sliding window with $100$px stride.
The patches are extracted starting the window at the top-left corner of the
image and sliding it across the image horizontally and vertically according to
the specified stride. At each position, we extract a patch within the window
boundaries. This process resulted in, for each sequence of the CTC dataset,
$4,\\!508$ for the GOWT1 dataset, $690$ for U373, and $16,\\!314$ for HeLa.
During inference, we used the full-sized images, except for the HeLa dataset,
for which we used the patches with a $256$ stride (to provide some overlap in
the borders) and then divided the parameters of the detection by a factor of 2
to retrieve them with the expected original image sizes. We used a similar
strategy as in other works [43, 40, 41, 8, 14]: with dataset sequences that
contain GT annotations (i.e., not the one related to the challenge), we
trained the models with one sequence and used the other for evaluation. To
generate the OBBs (and then the EBBs) from the CTC cell masks, which are
required to train the object detectors, we employed the minimum-area rectangle
fitting algorithm available in the OpenCV [56] framework.

###  4.3 Implementation details

As the object detector, we chose to use the R2CNN [35] provided by the
AlphaRotate benchmark 111https://github.com/yangxue0827/RotationDetection,
because it has shown to have the best compromise between precision and recall
compared to other OBB detectors in our preliminary investigations using a
private cell dataset. Furthermore, as presented in [57], general-purpose two-
stage detectors usually provide better results when compared with one-stage
ones. R2CNN is a two-stage object detector (i.e. it has a region proposed
module (RPN) before the detection module) that was first proposed in text
detection problems. The model parameters used for all experiments are the same
as in the AlphaRotate default architecture. The trained model uses a ResNet50
[58] backbone with pre-trained weights on the ImageNet dataset [59]. Both
classification modules (RPN and head) use the categorical cross-entropy loss,
and both box regression modules use the smooth-$\ell_{1}$ loss. Weight decay
and momentum are set to $10^{-4}$ and 0.9, respectively. We employ Momentum
Optimizer over 1 GPU and eight images per mini-batch. All models are trained
for 100 epochs with a 0.1 reduction factor of the learning rate at epochs 12,
16, and 20, using random rotation and flips as data augmentation primitives.
The initial learning rate was $10^{-3}$ for all datasets except HeLa, for
which the model showed overfitting. In this dataset, we used $10^{-4}$ as the
initial learning rate and trained for 24 epochs only.

The hyper-parameters for the tracking system are the same for all datasets
(except the object detection precision $\alpha$ and capture time $\Delta t$),
and were chosen empirically to produce good results for all the evaluated
datasets (i.e., we used the training datasets in order to define the hyper-
parameters of the final tracking system that was evaluated on the CTC [39]
server with hidden GT). Nevertheless, we provide a sensitivity analysis of the
parameters in Section 5.3, and conclude that changing them has a small impact
on the results. For the inference of the detector, we used a score threshold
$\tau_{s}=0.5$, and an overlap threshold of $\tau_{h}=0.5$ for the filtering
and aggregation step. For the tracklet generation, we employed an overlap
threshold of $\tau_{o}=0.5$, which are classical thresholds for IoU-like
metrics.

For the parameters of the global data association algorithm, we used a time
threshold $t_{th}=3$ frames, and a false positive threshold $\tau_{FP}=0.9$.
The space threshold was set to $0.1\sqrt{W_{f}^{2}+H_{f}^{2}}$, where $W_{f}$
and $H_{f}$ are the width and height of the dataset frames, respectively. The
$\alpha$ value was computed using the precision value for each dataset
considering an overlap threshold of $\tau_{h}=0.5$ between the predicted and
ground-truth detections of the training images with no score threshold (i.e.,
considering any detection with a confidence score above zero), and are
available in Table 1. The free parameters for likelihood adjustment are set to
$\lambda_{link}=25$ and $\lambda_{mit}=50$ for all datasets. The MAP problem
was solved using the Cbc [60] mixed ILP solver provided in the
CVXPY222Available at: https://www.cvxpy.org/index.html Python 3 [61] library.

Table 1: Object detection precision $\alpha$ for each dataset. Dataset | GOWT1-01 | GOWT1-02 | U373-01 | U373-02 | HeLa-01 | HeLa-02
---|---|---|---|---|---|---
$\bm{\alpha}$ | 0.8993 | 0.8644 | 0.7673 | 0.6867 | 0.7930 | 0.7899

###  4.4 Evaluation Metrics

We evaluated our method using the metrics proposed by the ISBI 2015 Cell
Tracking Challenge [39] and standard literature metrics used for evaluating
general-purpose detection and tracking systems. The ISBI Challenge provides
the DET, SEG and TRA metrics 333A full description of the metrics can be found
at http://celltrackingchallenge.net/evaluation-methodology/. Both DET and TRA
metrics are designed to mirror the manual effort required to correct the
errors of a given detection and tracking algorithm, respectively, using
Acyclic Oriented Graph Matching; SEG measures the Jaccard similarity index
(a.k.a. IoU) between predicted and ground-truth segmentation masks. All ISBI
metrics return values from 0 to 1 (1 being the highest score).

Since our method provides only an approximation of the segmentation masks
through EBBs, we would also like to estimate how close both the GT EBBs and
the predicted EBBs are to the GT segmentation masks. The former can be
answered by evaluating the EBBs generated from the ground truth cell masks
with the SEG metric, obtaining the _EBB SEG_ metric. The latter is obtained by
simply evaluating our method with the SEG metric.

For the detection and tracking evaluation, the algorithm provided by the ISBI
challenge disregards detections that do not entirely overlap with the provided
ground-truth cell marks. As mentioned before, they do not follow a standard
for size and displacement, which might affect the quantitative metrics. For
the U373-02 dataset, in particular, Ulah and colleagues [8] propose simply
enlarging the predicted masks to avoid missing the cell marks. In our
approach, however, the variability of cell mark annotations can significantly
impact all the tested datasets, since the EBBs are only approximations of the
segmentation mask and might not completely overlap with a ground-truth mark –
they are not bounding representations as well. In order to overcome this
issue, we also evaluated our method enlarging the predicted cell masks with a
simple watershed algorithm, using the EBBs as guiding markers. An example of
EBB-guided watershed is shown in Fig. 8.

Figure 8: Visual comparison of the cell masks using EBBs (left) and post-
processed with watershed (right). Note that the watershed fills the voids
between the different detections allowing some degree of error when evaluating
with the DET and TRA metrics.

For the standard literature metrics, we evaluated our object detector using
the Average Precision (AP) metric, which relates the precision and recall for
a given threshold score and detection overlap (e.g., using IoU or the
Hellinger distance) between detections. For the tracking evaluation, we used
the CLEAR-MOT [62, 63] and ID-MEASURE [64] metrics444We used a Python [61]
implementation of these metrics available at: https://github.com/cheind/py-
motmetrics. Both metrics attempt to find a minimum-cost assignment between
ground truth objects and predictions. However, while CLEAR-MOT solves the
assignment problem on a local per-frame basis, ID-MEASURE solves the bipartite
graph matching by finding the minimum cost of objects and predictions over all
frames. For evaluating the object detector, we used the Hellinger distance to
compute the overlap between predicted and ground-truth detections, and set the
score and overlap thresholds to 0.5.

##  5 Results and Discussions

In this section, we present the results for evaluating our tracking-by-
detection method in the CTC [39] datasets. We evaluated our results in two
manners: one by using only the training datasets with GT and comparing with
other works directly (see Section 4.2 for more details), and the other by
submitting our codes to the CTC server555Available at:
http://celltrackingchallenge.net/ in order to retrieve the metrics scores and
rank compared to other submitted works. We begin this section by briefly
describing the baseline methods used in the first evaluation methodology.
Then, we present the results for both methodologies and discuss them.

###  5.1 Baseline methods

We proceed to briefly describe the methods used as baseline for comparison
with our proposal, highlighting the nature of required training data.

#### KTH

[43] segments cells using a bandpass filter followed by thresholding, and then
uses the watershed algorithm to split joined cells. The tracking graph is
created by connecting cell segmentations in adjacent frames, and then solved
by iteratively finding the lowest cost path in the graph using Viterbi
algorithm. It does not require any annotation.

#### EPFL

[40] detects cells by fitting ellipses to binary segmented regions. It joins
the detection on subsequent frames using a tracking graph solved using integer
linear programming (ILP). It requires annotation for segmentation, detection
and tracking.

#### HEID

[40] detects cells by merging super-pixels clustering segmentation, which are
obtained using watersheds. Then, the tracking is retrieved by finding the
global optimum of a graph model that represents cellular events using ILP. It
requires annotation for segmentation, detection, and tracking.

#### BLOB

[41] detect cells using multiple elliptical filter banks, and performs
tracking by iteratively finding the shortest path in a model graph. It
requires only tracking annotation, i.e., temporal cell associations.

#### CPN

[8] first generates cell region proposals using an HBB object detector, and
then finds the segmentation masks of these regions using a deep learning
segmentation network similar to the U-Net [11]. It performs tracking using ILP
to solve a graph for which the weights are set by training a random forest
classifier with several histogram features. It requires full annotation for
detection, segmentation, and tracking.

#### DRL

[14] uses the U-Net [11] model to produce the cell segmentation masks. Then,
it uses deep reinforcement learning to build the cost matrix that joins the
cells of subsequent frames. It requires full annotation for detection,
segmentation, and tracking.

#### ST-TCV

[44] detects cells using a joint spatio-temporal diffusion and region-based
level-set optimization approach [65]. Then, it uses motion prediction and
minimization of a global probabilistic function to join the cells of
subsequent frames. It does not require any annotation.

We did not include results for the methods U-Net [11], GC-ME [12], and U-Net S
[13] because they do not follow the same data split methodology (i.e., they
employ images from both sequences in the training phase), and hence it would
define an unfair baseline for comparison.

###  5.2 Results for the CTC datasets

The results for comparing our method with SOTA approaches on the ISBI
evaluation using separate sets are provided in Table 2. We include in this
table results for both using or not using the watershed algorithm for
enlarging the detection results.

Table 2: Results for the CTC [39] training datasets using separate sequences.
Our results with the watershed method are reported as _Ours-W_. Bold values
mark the best results, while underline values mark the second best. We also
report the annotation requirements (Ann. Req. column) of each technique
related to the detection (Det) and tracking (Tra)

.

|  | Ann. Req. |  |  |  |
---|---|---|---|---|---|---
Dataset | Method | Det. | Tra. | DET | SEG | EBB SEG | TRA
| ST-TCV | ✗ | ✗ | N/A | N/A | - | 0.913
| KTH | ✗ | ✗ | N/A | 0.6849 | - | 0.9462
| BLOB | ✗ | ✓ | N/A | 0.7415 | - | 0.9733
| CPN | ✓ | ✓ | N/A | 0.8506 | - | 0.9864
| DRL | ✓ | ✓ | N/A | 0.8585 | - | 0.9875
| Ours | ✓ | ✗ | 0.9916 | 0.8568 | 0.9268 | 0.9914
GOWT1-01 | Ours-W | ✓ | ✗ | 0.9940 | - | - | 0.9930
| ST-TCV | ✗ | ✗ | N/A | N/A | - | 0.914
| HEID | ✓ | ✓ | N/A | N/A | - | 0.95
| EPFL | ✓ | ✓ | N/A | N/A | - | 0.95
| KTH | ✗ | ✗ | N/A | 0.8942 | - | 0.9452
| BLOB | ✗ | ✓ | N/A | 0.9046 | - | 0.9628
| CPN | ✓ | ✓ | N/A | 0.8725 | - | 0.9719
| DRL | ✓ | ✓ | N/A | 0.9181 | - | 0.9575
| Ours | ✓ | ✗ | 0.9812 | 0.8509 | 0.9167 | 0.9817
GOWT1-02 | Ours-W | ✓ | ✗ | 0.9868 | - | - | 0.9853
| CPN | ✓ | ✓ | N/A | 0.7336 | - | 0.9594
| DRL | ✓ | ✓ | N/A | 0.8527 | - | 0.9919
| Ours | ✓ | ✗ | 0.9647 | 0.6307 | 0.7791 | 0.9671
U373-01 | Ours-W | ✓ | ✗ | 0.9748 | - | - | 0.9774
| CPN | ✓ | ✓ | N/A | 0.7376 | - |  0.9346*
| DRL | ✓ | ✓ | N/A | 0.7735 | - | 0.9318
| Ours | ✓ | ✗ | 0.8822 | 0.5626 | 0.7029 | 0.8737
U373-02 | Ours-W | ✓ | ✗ | 0.9634 | - | - | 0.9525
| ST-TCV | ✗ | ✗ | N/A | N/A | - | 0.816
| HEID | ✓ | ✓ | N/A | N/A | - | 0.80
| EPFL | ✓ | ✓ | N/A | N/A | - | 0.98
| KTH | ✗ | ✗ | N/A | 0.8018 | - | 0.9775
| CPN | ✓ | ✓ | N/A | 0.8313 | - | 0.9869
| BLOB | ✗ | ✓ | N/A | 0.7951 | - | 0.9803
| Ours | ✓ | ✗ | 0.9779 | 0.7264 | 0.8871 | 0.9758
HeLa-01 | Ours-W | ✓ | ✗ | 0.9863 | - | - | 0.9820
| ST-TCV | ✗ | ✗ | N/A | N/A | - | 0.845
| HEID | ✓ | ✓ | N/A | N/A | - | 0.85
| EPFL | ✓ | ✓ | N/A | N/A | - | 0.97
| KTH | ✗ | ✗ | N/A | 0.8366 | - | 0.9747
| CPN | ✓ | ✓ | N/A | 0.8445 | - | 0.9826
| BLOB | ✗ | ✓ | N/A | 0.8390 | - | 0.9771
| Ours | ✓ | ✗ | 0.9707 | 0.7618 | 0.8897 | 0.9664
HeLa-02 | Ours-W | ✓ | ✗ | 0.9796 | - | - | 0.9740

* denotes augmentation on the segmentation masks.

Regarding the approximation of the cell masks using EBBs, we can observe from
the _EBB SEG_ column that it provides a good fit for the GOWT1 and HeLa
datasets. On the other hand, the EBB approximation is not very good for the
U373 dataset since there is strong variability in the cell shapes, as
mentioned before. Finally, our method did not achieve SOTA scores on the SEG
metric, which is expected since we only approximate the cell masks through
EBBs. Nevertheless, it could reach close values to those on both the GOWT1 and
HeLa datasets, and even get the second best for the GOWT1-01.

For the DET and TRA metrics, we note that our approach achieves a considerable
boost using the watershed post-processing algorithm, particularly for the
U373-02 dataset. We believe that this behavior is mostly due to the GT
annotation of the cell markers in the dataset, which is sometimes located at
the boundary of the cells and might not overlap completely with the EBB.
Nevertheless, our method (without watershed) could reach SOTA results in both
GOWT1 datasets, while having the second-best result in U373-01. When applying
the watershed mask augmentation, our method reached SOTA scores on three
datasets, and second best on other two. Regarding the degree of annotation
required for each technique, our method was capable of outperforming fully
supervised methods (HEID [40], EPFL [40], CPN [8] and DRL [14]) in most of the
evaluated datasets. It also presented better results than the tracking-
supervised approach BLOB [41] and the unsupervised trackers KTH [43] and ST-
TCV [44] in all datasets, except for HeLa-02 compared to the KTH method.

Table 3 report the results for evaluating our method on the CTC server
666Submitted implementation available at:
http://celltrackingchallenge.net/participants/UFRGS-BR/ on the DET and TRA
metrics with and without the watershed method777Due to environment problems
related to CUDA instructions on the CTC server computers, we could not
reproduce the exact same code used on our side. This ended up slightly harming
the predicted EBB shapes and hence under-estimating the SEG metric, and the
DET and TRA metrics when the detections’ shapes are not augmented with the
watershed algorithm.. In this evaluation, our method was capable of achieving
the TOP 3 rank on the DET metric for the GOWT1 dataset using the watershed
algorithm. Although it could not overcome the SOTA in any dataset, we can
observe a small difference between the scores of our method with those ranked
as the top one. Furthermore, most of the top-ranked algorithms are end-to-end
trackers or use elaborated techniques to improve the predicted segmentation
masks from deep learning models (e.g., using model assemble or multiple
refinement stages). In contrast, our proposed method intends to provide a
simple yet efficient method for tracking-by-detection that requires only per-
frame OBB cell annotations.

Table 3: Results from the CTC [39] challenge evaluation server. We report the results for our technique both using and not the watershed mask augmentation (_W_ column), the rank position over all submissions, and the relative difference to the first rank method (_To TOP1_ column). Evaluation date: October 10, 2022. Dataset | W | DET | TRA
---|---|---|---
Score | Rank | To TOP1 | Score | Rank | To TOP1
GOWT1 | ✗ | 0.925 | 26/49 | 5.61% | 0.922 | 19/40 | 5.82%
✓ | 0.970 | 3/49 | 1.02% | 0.959 | 4/40 | 2.04%
U373 | ✗ | 0.914 | 33/38 | 7.68% | 0.909 | 26/31 | 7.72%
✓ | 0.979 | 17/38 | 1.11% | 0.976 | 12/31 | 0.91%
HeLa | ✗ | 0.986 | 15/48 | 0.80% | 0.984 | 11/39 | 0.91%
✓ | 0.989 | 10/48 | 0.50% | 0.988 | 8/39 | 0.50%

The results using standard detection and quality metrics are provided in Table
4. The detection results refer only to the detector performance itself, i.e.,
it does not use the global data association to further eliminate false
positive detections and/or add false negative ones. This table evaluates
different aspects of our method, enabling us to identify its strengths and
weakness better. Regarding the recall metric, we can observe that it could
achieve high values on all datasets for detection and tracking. However, it is
noticeable that our method fails to eliminate false positive detections, which
impact the precision in both detection and tracking, as noted for dataset
U373-01. On the other hand, the detector precision in the GOWT1-02 dataset was
also relatively low, but our global data association algorithm was capable of
eliminating most of the false positive detections and hence obtaining a higher
precision value on the tracking metrics. For U373-02, we can observe an
inconsistency between the detection and tracking precision, which might be
explained by the inconsistency of the cell mark annotations mentioned in
Section 4.4.

Table 4: Results using standard detection and tracking quality metrics (in %), as described in Section 4.4. Dataset | Tracking | Detection
---|---|---
ID-F1 | ID-P | ID-R | R | P | MOTA | R50 | P50 | F150 | AP50
GOWT1-01 | 97.9 | 96.7 | 99.0 | 99.9 | 97.5 | 97.0 | 99.4 | 97.0 | 98.2 | 90.9
GOWT1-02 | 95.2 | 91.3 | 99.6 | 100.0 | 91.7 | 90.7 | 99.9 | 83.7 | 91.1 | 90.5
U373-01 | 88.9 | 79.9 | 100.0 | 100.0 | 79.9 | 74.8 | 99.9 | 79.0 | 88.2 | 90.8
U373-02 | 81.5 | 79.6 | 83.6 | 96.0 | 91.4 | 86.3 | 93.6 | 90.1 | 91.8 | 89.5
HeLa-01 | 94.6 | 92.4 | 96.8 | 100.0 | 95.4 | 94.0 | 90.8 | 96.5 | 93.6 | 90.4
HeLa-02 | 93.1 | 89.0 | 97.6 | 100.0 | 91.2 | 89.1 | 95.0 | 93.5 | 94.2 | 90.1

Table 5 shows a comparison of our tracking pipeline using our data association
algorithm and a modified version using the approach by [10] for computing the
final tracks. We note that the proposed modifications only slightly improve
the DET and TRA metrics for most datasets, but they provide a significant
reduction in the number of generated hypotheses. As a consequence, it allows
faster solution computation and fewer hardware requirements.

Table 5: Results comparing our complete method and a modified version of [10] (Baseline). Dataset | Method | DET | TRA | Hypothesis | Time (s)
---|---|---|---|---|---
GOWT1-01 | Baseline | 0.9914 | 0.9913 | 133 | 0.1234
Ours | 0.9916 | 0.9914 | 47 | 0.0998
GOWT1-02 | Baseline | 0.9803 | 0.9805 | 312 | 0.2280
Ours | 0.9812 | 0.9817 | 161 | 0.2082
U373-01 | Baseline | 0.9655 | 0.9677 | 78 | 0.0891
Ours | 0.9647 | 0.9671 | 53 | 0.0763
U373-02 | Baseline | 0.8818 | 0.8733 | 83 | 0.0918
Ours | 0.8822 | 0.8737 | 24 | 0.0509
HeLa-01 | Baseline | 0.9772 | 0.9741 | 7989 | 30.681
Ours | 0.9779 | 0.9758 | 7061 | 25.337
HeLa-02 | Baseline | 0.9699 | 0.9652 | 25525 | 259.18
Ours | 0.9707 | 0.9664 | 22649 | 235.06

Finally, we provide visual detection results on the CTC [39] datasets on
Figure 9. We can observe that EBBs provide a good description of the cell
shapes for the GOWT1 and HeLa datasets, but not so much for the U373 datasets.
We can also note the high recall rate of the object detector, since almost all
cells are retrieved. Figure 10 presents the generated tracking trees888We also
provide animated images in our GitHub repository at:
https://github.com/LucasKirsten/Deep-Cell-Tracking-EBB/. Analyzing the results
for the GOWT1 and U373 datasets, which are less cluttered, we can observe that
our method could produce clear paths for most of the initial cells. These
datasets have almost no mitosis or apoptosis events, so the paths that seem to
emerge in later frames can be false positives or cells emerging from the image
borders.

(a) GOWT1-01

(b) GOWT1-02

(c) HeLa-01

(d) HeLa-02

(e) U373-01

(f) U373-02

Figure 9: Visual results of our method in the CTC [39] training datasets. In
green are the ground-truth segmentation masks, and in red are the predicted
EBBs.

(a) GOWT1-01

(b) GOWT1-02

(c) HeLa-01

(d) HeLa-02

(e) U373-01

(f) U373-02

Figure 10: Visualization of the generated tracking trees in the CTC [39]
training datasets.

###  5.3 Sensitivity analysis

In this section, we analyze the sensitivity of our tracking-by-detection
method regarding its hyper-parameters. We randomly sampled the parameter
values within an interval and evaluated the results on the CTC [39] tested
training datasets (i.e., with provided GT). The parameters were randomly
sampled in the following scheme: $\lambda_{link}$ and
$\lambda_{mit}\in\mathbb{R^{+}}$ linearly spaced between 5 and 1000 with step
25, $t_{th}\in\mathbb{N}$ linearly ranging from 1 to 8 with step 1,
$\tau_{s}\in\mathbb{R^{+}}$ linearly ranging from 0.4 to 0.9 with 0.05 step,
and $\tau_{FP}\in\mathbb{R^{+}}$ linearly ranging from 0.5 to 0.9 with 0.05
step. For consistently evaluating the impact of the parameters on the
different datasets, we subtracted the metrics values from the ones reported in
Table 2 when using the watershed method to show the relative gain or loss when
changing the hyper-parameters.

Figure 11 shows a boxplot with the relative changes of the DET and TRA metrics
for a set of $\sim\\!400$ random combinations of the hyper-parameters in each
individual dataset. We can observe that the impact of changing the parameters
is small for most of the datasets. The worst-case scenario occurs on the
U373-2 dataset, with a negative impact of $\sim\\!3.5\%$ on the TRA metric. On
the other hand, we note that some combination of hyper-parameters can actually
improve the results obtained with the default parameters.

Figure 11: Impacts on the individual datasets evaluation from randomly
sampling the hyper-parameters. The zero value refers to an output metric value
equal to the one reported in Table 2.

In order to access the impact of the individual hyper-parameters on the
method, we used the Shapley Additive Explanations (SHAP values) [66, 67] using
the DET and TRA evaluation metrics as the targets and averaging the results
among all the CTC tested datasets. SHAP values provide insights into feature
contributions, distributing credit among features to explain machine learning
predictions. They measure the impact of each feature compared to its absence
or average value, allowing a nuanced understanding of feature importance.
Positive values increase predictions, negative values decrease predictions,
and zero values have no impact. Examining SHAP values helps identify
influential features, aiding feature selection, model debugging, and
understanding model decisions. In our case, we adapted the method to work with
the hyper-parameters instead of the features. Figure 12 presents the violin
plots, where the color indicates the parameter value and the horizontal axis
denotes the corresponding SHAP value. The small range of the horizontal axis
and concentration at small SHAP values indicate that the method is robust to
the parameter choice. Furthermore, we note that most random combinations of
individual parameters lead to a positive impact on the metric scores.

(a) DET metric

(b) TRA metric

Figure 12: Individual impacts of the hyper-parameters on the method
performance using the SHAP values [66, 67] regarding the DET and TRA metrics.

##  6 Conclusions

We proposed a tracking-by-detection method that explores an OBB detector to
identify cells, represent them as ellipses, and then uses the detection
information in an unsupervised tracking algorithm based on tracklet
association. Our method alleviates the annotation efforts by representing the
cells as a 5-parameter oriented ellipses (that can be either annotated as an
OBB or EBB), and by defining an unsupervised tracking system oriented solely
on the detection information retrieved by the trained object detector. Our
results demonstrate that the cell elliptical representation presents a good
approximation for the full segmentation mask, particularly for lineages with a
regular shape. Furthermore, our tracking-by-detection method can achieve
results competitive to other state-of-the-art methods that require
considerably more annotated data. Moreover, our method reduces the hardware
requirements for training and predicting when compared to the current trend of
end-to-end trackers, since it requires training only one object detector and
does not rely on training a complete detection and association deep learning
architecture that needs both batches of frame images and their objects
associations. We believe that our method can be broadly used in applications
where there are limited resources or short deadlines for retrieving the full
annotations.

## References

  * [1] T. Q. Syed, V. Vigneron, S. Lelandais, G. Barlovatz-Meimon, M. Malo, C. Charrière-Bertrand, and C. Montagne, “Detection and counting of" in vivo" cells to predict cell migratory potential,” in 2008 First Workshops on Image Processing Theory, Tools and Applications, pp. 1–8, IEEE, 2008.
  * [2] M. R. C. Leite, I. A. Cestari, and I. N. Cestari, “Computational tool for morphological analysis of cultured neonatal rat cardiomyocytes,” in 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 3517–3520, IEEE, 2015.
  * [3] D. Di Giuseppe, F. Corsi, A. Mencattini, M. C. Comes, P. Casti, C. Di Natale, L. Ghibelli, and E. Martinelli, “Learning cancer-related drug efficacy exploiting consensus in coordinated motility within cell clusters,” IEEE Transactions on Biomedical Engineering, vol. 66, no. 10, pp. 2882–2888, 2019.
  * [4] D. Gradeci, A. Bove, G. Charras, A. R. Lowe, and S. Banerjee, “Single-cell approaches to cell competition: high-throughput imaging, machine learning and simulations,” in Seminars in cancer biology, vol. 63, pp. 60–68, Elsevier, 2020.
  * [5] J. Hayashida, K. Nishimura, and R. Bise, “Consistent cell tracking in multi-frames with spatio-temporal context by object-level warping loss,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1727–1736, 2022.
  * [6] N. Emami, Z. Sedaei, and R. Ferdousi, “Computerized cell tracking: Current methods, tools and challenges,” Visual Informatics, vol. 5, no. 1, pp. 1–13, 2021.
  * [7] V. Ulman, M. Maška, K. E. Magnusson, O. Ronneberger, C. Haubold, N. Harder, P. Matula, P. Matula, D. Svoboda, M. Radojevic, et al., “An objective comparison of cell-tracking algorithms,” Nature methods, vol. 14, no. 12, pp. 1141–1152, 2017.
  * [8] S. U. Akram, J. Kannala, L. Eklund, and J. Heikkilä, “Cell tracking via proposal generation and selection,” arXiv preprint arXiv:1705.03386, 2017.
  * [9] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in Proceedings of the IEEE international conference on computer vision, pp. 2961–2969, 2017.
  * [10] R. Bise, Z. Yin, and T. Kanade, “Reliable cell tracking by global data association,” in 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, pp. 1004–1010, IEEE, 2011.
  * [11] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention, pp. 234–241, Springer, 2015.
  * [12] R. Bensch and O. Ronneberger, “Cell segmentation and tracking in phase contrast images using graph cut with asymmetric boundary costs,” in 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI), pp. 1220–1223, IEEE, 2015.
  * [13] D. K. Gupta, N. de Bruijn, A. Panteli, and E. Gavves, “Tracking-assisted segmentation of biological cells,” arXiv preprint arXiv:1910.08735, 2019.
  * [14] J. Wang, X. Su, L. Zhao, and J. Zhang, “Deep reinforcement learning for data association in cell tracking,” Frontiers in Bioengineering and Biotechnology, vol. 8, p. 298, 2020.
  * [15] V. Kulharia, S. Chandra, A. Agrawal, P. Torr, and A. Tyagi, “Box2seg: Attention weighted loss and discriminative feature learning for weakly supervised segmentation,” in European Conference on Computer Vision, pp. 290–308, Springer, 2020.
  * [16] F. Xing, Y. Xie, H. Su, F. Liu, and L. Yang, “Deep learning in microscopy image analysis: A survey,” IEEE Transactions on Neural Networks and Learning Systems, vol. 29, no. 10, pp. 4550–4568, 2017.
  * [17] E. Moen, D. Bannon, T. Kudo, W. Graf, M. Covert, and D. Van Valen, “Deep learning for cellular image analysis,” Nature methods, vol. 16, no. 12, pp. 1233–1246, 2019.
  * [18] U. Schmidt, M. Weigert, C. Broaddus, and G. Myers, “Cell detection with star-convex polygons,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 265–273, Springer, 2018.
  * [19] A. Newell, Z. Huang, and J. Deng, “Associative embedding: End-to-end learning for joint detection and grouping,” in Advances in neural information processing systems, pp. 2277–2287, 2017.
  * [20] C. Payer, D. Štern, T. Neff, H. Bischof, and M. Urschler, “Instance segmentation and tracking with cosine embeddings and recurrent hourglass networks,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 3–11, Springer, 2018.
  * [21] C. Payer, D. Štern, M. Feiner, H. Bischof, and M. Urschler, “Segmenting and tracking cell instances with cosine embeddings and recurrent hourglass networks,” Medical image analysis, vol. 57, pp. 106–119, 2019.
  * [22] M. Zhao, A. Jha, Q. Liu, B. A. Millis, A. Mahadevan-Jansen, L. Lu, B. A. Landman, M. J. Tyska, and Y. Huo, “Faster mean-shift: Gpu-accelerated clustering for cosine embedding-based cell segmentation and tracking,” Medical Image Analysis, vol. 71, p. 102048, 2021.
  * [23] D. Liu, D. Zhang, Y. Song, H. Huang, and W. Cai, “Panoptic feature fusion net: a novel instance segmentation paradigm for biomedical and biological images,” IEEE Transactions on Image Processing, vol. 30, pp. 2045–2059, 2021.
  * [24] Z. Zhao, L. Yang, H. Zheng, I. H. Guldner, S. Zhang, and D. Z. Chen, “Deep learning based instance segmentation in 3d biomedical images using weak annotation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 352–360, Springer, 2018.
  * [25] T. Zhao and Z. Yin, “Weakly supervised cell segmentation by point annotation,” IEEE Transactions on Medical Imaging, vol. 40, no. 10, pp. 2736–2747, 2020.
  * [26] H.-J. Oh, K. Lee, and W.-K. Jeong, “Scribble-supervised cell segmentation using multiscale contrastive regularization,” in 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pp. 1–5, IEEE, 2022.
  * [27] K. Lu, Y. Qian, J. Gong, Z. Zhu, J. Yin, L. Ma, M. Yu, and H. Wang, “Biofabrication of aligned structures that guide cell orientation and applications in tissue engineering,” Bio-Design and Manufacturing, vol. 4, no. 2, pp. 258–277, 2021.
  * [28] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in European conference on computer vision, pp. 21–37, Springer, 2016.
  * [29] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779–788, 2016.
  * [30] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in European conference on computer vision, pp. 213–229, Springer, 2020.
  * [31] M. Tan, R. Pang, and Q. V. Le, “Efficientdet: Scalable and efficient object detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10781–10790, 2020.
  * [32] C.-Y. Wang, A. Bochkovskiy, and H.-Y. M. Liao, “Scaled-yolov4: Scaling cross stage partial network,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13029–13038, June 2021.
  * [33] J. Yang, C. Li, and J. Gao, “Focal modulation networks,” arXiv preprint arXiv:2203.11926, 2022.
  * [34] S. Mandal and V. Uhlmann, “Splinedist: Automated cell segmentation with spline curves,” in 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 1082–1086, IEEE, 2021.
  * [35] Y. Jiang, X. Zhu, X. Wang, S. Yang, W. Li, H. Wang, P. Fu, and Z. Luo, “R2cnn: Rotational region cnn for orientation robust scene text detection,” arXiv preprint arXiv:1706.09579, 2017.
  * [36] X. Yang, J. Yan, Z. Feng, and T. He, “R3det: Refined single-stage detector with feature refinement for rotating object,” in AAAI, 2021.
  * [37] G.-S. Xia, X. Bai, J. Ding, Z. Zhu, S. Belongie, J. Luo, M. Datcu, M. Pelillo, and L. Zhang, “Dota: A large-scale dataset for object detection in aerial images,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [38] X. Yang, X. Yang, J. Yang, Q. Ming, W. Wang, Q. Tian, and J. Yan, “Learning high-precision bounding box for rotated object detection via kullback-leibler divergence,” in Advances in Neural Information Processing Systems (M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, eds.), vol. 34, pp. 18381–18394, Curran Associates, Inc., 2021.
  * [39] M. Maška, V. Ulman, D. Svoboda, P. Matula, P. Matula, C. Ederra, A. Urbiola, T. España, S. Venkatesan, D. M. Balak, et al., “A benchmark for comparison of cell tracking algorithms,” Bioinformatics, vol. 30, no. 11, pp. 1609–1617, 2014.
  * [40] E. Türetken, X. Wang, C. Becker, C. Haubold, and P. Fua, “Globally optimal cell tracking using integer programming,” arXiv preprint arXiv:1501.05499, 2015.
  * [41] S. U. Akram, J. Kannala, L. Eklund, and J. Heikkilä, “Joint cell segmentation and tracking using cell proposals,” in 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), pp. 920–924, IEEE, 2016.
  * [42] K. Nishimura, J. Hayashida, C. Wang, D. F. E. Ker, and R. Bise, “Weakly-supervised cell tracking via backward-and-forward propagation,” in European Conference on Computer Vision, pp. 104–121, Springer, 2020.
  * [43] K. E. Magnusson and J. Jaldén, “A batch algorithm using iterative application of the viterbi algorithm to track cells and construct cell lineages,” in 2012 9th IEEE International Symposium on Biomedical Imaging (ISBI), pp. 382–385, IEEE, 2012.
  * [44] F. Boukari and S. Makrogiannis, “Automated cell tracking using motion prediction-based matching and event handling,” IEEE/ACM transactions on computational biology and bioinformatics, vol. 17, no. 3, pp. 959–971, 2018.
  * [45] B. Xu, J. Shi, M. Lu, J. Cong, L. Wang, and B. Nener, “An automated cell tracking approach with multi-bernoulli filtering and ant colony labor division,” IEEE/ACM transactions on computational biology and bioinformatics, vol. 18, no. 5, pp. 1850–1863, 2019.
  * [46] O. Hirose, S. Kawaguchi, T. Tokunaga, Y. Toyoshima, T. Teramoto, S. Kuge, T. Ishihara, Y. Iino, and R. Yoshida, “Spf-celltracker: Tracking multiple cells with strongly-correlated moves using a spatial particle filter,” IEEE/ACM transactions on computational biology and bioinformatics, vol. 15, no. 6, pp. 1822–1831, 2017.
  * [47] J. M. Llerena, L. F. Zeni, L. N. Kristen, and C. Jung, “Gaussian bounding boxes and probabilistic intersection-over-union for object detection,” arXiv preprint arXiv:2106.06072, 2021.
  * [48] Z. Chen, K. Chen, W. Lin, J. See, H. Yu, Y. Ke, and C. Yang, “Piou loss: Towards accurate oriented object detection in complex environments,” in European conference on computer vision, pp. 195–211, Springer, 2020.
  * [49] J. Murrugarra-Llerena, L. N. Kirsten, and C. R. Jung, “Can we trust bounding box annotations for object detection?,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 4813–4822, June 2022.
  * [50] X. Yang, J. Yan, M. Qi, W. Wang, Z. Xiaopeng, and T. Qi, “Rethinking rotated object detection with gaussian wasserstein distance loss,” in International Conference on Machine Learning (ICML), 2021.
  * [51] A. Bhattacharyya, “On a measure of divergence between two multinomial populations,” Sankhyā: the indian journal of statistics, pp. 401–406, 1946.
  * [52] E. Hellinger, “Neue begründung der theorie quadratischer formen von unendlichvielen veränderlichen.,” Journal für die reine und angewandte Mathematik, vol. 1909, no. 136, pp. 210–271, 1909.
  * [53] T. Kailath, “The divergence and bhattacharyya distance measures in signal selection,” IEEE transactions on communication technology, vol. 15, no. 1, pp. 52–60, 1967.
  * [54] H. W. Kuhn, “The hungarian method for the assignment problem,” Naval research logistics quarterly, vol. 2, no. 1-2, pp. 83–97, 1955.
  * [55] S. Huh, R. Bise, M. Chen, T. Kanade, et al., “Automated mitosis detection of stem cell populations in phase-contrast microscopy images,” IEEE transactions on medical imaging, vol. 30, no. 3, pp. 586–596, 2010.
  * [56] G. Bradski, “The OpenCV Library,” Dr. Dobb’s Journal of Software Tools, 2000.
  * [57] L. Liu, W. Ouyang, X. Wang, P. Fieguth, J. Chen, X. Liu, and M. Pietikäinen, “Deep learning for generic object detection: A survey,” International journal of computer vision, vol. 128, no. 2, pp. 261–318, 2020.
  * [58] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  * [59] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255, Ieee, 2009.
  * [60] J. Forrest, T. Ralphs, H. G. Santos, S. Vigerske, J. Forrest, L. Hafer, B. Kristjansson, jpfasano, EdwinStraver, M. Lubin, rlougee, jpgoncal1, Jan-Willem, h-i gassmann, S. Brito, Cristina, M. Saltzman, tosttost, B. Pitrus, F. MATSUSHIMA, and to st, “coin-or/cbc: Release releases/2.10.8,” May 2022.
  * [61] G. Van Rossum and F. L. Drake Jr, Python reference manual.  Centrum voor Wiskunde en Informatica Amsterdam, 1995.
  * [62] K. Bernardin and R. Stiefelhagen, “Evaluating multiple object tracking performance: the clear mot metrics,” EURASIP Journal on Image and Video Processing, vol. 2008, pp. 1–10, 2008.
  * [63] A. Milan, L. Leal-Taixé, I. Reid, S. Roth, and K. Schindler, “Mot16: A benchmark for multi-object tracking,” arXiv preprint arXiv:1603.00831, 2016.
  * [64] E. Ristani, F. Solera, R. Zou, R. Cucchiara, and C. Tomasi, “Performance measures and a data set for multi-target, multi-camera tracking,” in European conference on computer vision, pp. 17–35, Springer, 2016.
  * [65] F. Boukari and S. Makrogiannis, “Joint level-set and spatio-temporal motion detection for cell segmentation,” BMC Medical Genomics, vol. 9, no. 2, pp. 179–194, 2016.
  * [66] S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” in Advances in Neural Information Processing Systems 30 (I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, eds.), pp. 4765–4774, Curran Associates, Inc., 2017.
  * [67] S. M. Lundberg, B. Nair, M. S. Vavilala, M. Horibe, M. J. Eisses, T. Adams, D. E. Liston, D. K.-W. Low, S.-F. Newman, J. Kim, et al., “Explainable machine-learning predictions for the prevention of hypoxaemia during surgery,” Nature Biomedical Engineering, vol. 2, no. 10, p. 749, 2018.