# Contextual Fusion For Adversarial Robustness

Aiswarya Akumalla
Department of Medicine
University of California, San Diego
La Jolla, CA 92093
<EMAIL_ADDRESS>
Seth Haney
Department of Medicine
University of California, San Diego
La Jolla, CA 92093
<EMAIL_ADDRESS>
Maxim Bazhenov
Department of Medicine
University of California, San Diego
La Jolla, CA 92093
<EMAIL_ADDRESS>

###### Abstract

Mammalian brains handle complex reasoning tasks in a gestalt manner by
integrating information from regions of the brain that are specialised to
individual sensory modalities. This allows for improved robustness and better
generalisation ability. In contrast, deep neural networks are usually designed
to process one particular information stream and susceptible to various types
of adversarial perturbations. While many methods exist for detecting and
defending against adversarial attacks, they do not generalise across a range
of attacks and negatively affect performance on clean, unperturbed data. We
developed a fusion model using a combination of background and foreground
features extracted in parallel from Places-CNN and Imagenet-CNN. We tested the
benefits of the fusion approach on preserving adversarial robustness for human
perceivable (e.g., Gaussian blur) and network perceivable (e.g., gradient-
based) attacks for CIFAR-10 and MS COCO data sets. For gradient based attacks,
our results show that fusion allows for significant improvements in
classification without decreasing performance on unperturbed data and without
need to perform adversarial retraining. Our fused model revealed improvements
for Gaussian blur type perturbations as well. The increase in performance from
fusion approach depended on the variability of the image contexts; larger
increases were seen for classes of images with larger differences in their
contexts. We also demonstrate the effect of regularization to bias the
classifier decision in the presence of a known adversary. We propose that this
biologically inspired approach to integrate information across multiple
modalities provides a new way to improve adversarial robustness that can be
complementary to current state of the art approaches.

##  1 Introduction

###  1.1 Biological background

Current deep learning networks are designed to optimally solve specific
learning tasks for a particular category of inputs (e.g., convolutional neural
networks (CNNs) for visual pattern recognition), but are limited in their
ability to solve tasks that require combining different feature categories
(e.g., visual, semantic, auditory) into one coherent representation. Some of
the challenges include finding the right alignment of unimodal
representations, fusion strategy, and complexity measures for determining the
efficacy of fused representations[DBLP:journals/corr/BaltrusaitisAM17]. In
comparison, biological systems are excellent in their ability to form unique
and coherent object representations, which is usually done in the associative
cortex, by linking together different object features available from different
specialized cortical networks, e.g., primary visual or auditory cortices
[gisiger_computational_2000, pandya_association_1982, mars_412_2017,
rosen_role_2017]. This natural strategy has many advantages including better
discrimination performance, stability against adversarial attacks and better
scalability [gilad_spatiotemporal_2020]. Indeed, if a classification decision
is made based on a combination of features from different sensory categories,
a noise or lack of information in one category can be compensated by another
to make a correct decision. Furthermore, different types of sensory
information can complement each other by being available at different times
within a processing window. A good example may be human driving skill which
relies on a combination of visual and auditory processing that helps to avoid
mistakes and greatly enhances performance over only vision-based driving.
Another example is insect navigation that depends both on visual and olfactory
information to minimize classification error and to identify objects more
reliably across range of distances[strube-bloss_multimodal_2018].

Although it seems obvious that humans and animals base their classification
decisions on the complex mixture of features from different modalities using
specialized classifiers in each of them, this ability is still lacking in
current state of the art machine learning (ML) algorithms. Problems include
difficulty of training because of the lack of data sets combining different
types of information, and suboptimal performance of generic deep learning
networks vs specialized ones. Indeed, e.g., high performance of the CNNs
designed for visual processing depends on their architecture that makes
explicit assumptions that inputs are images, and the same network performs
poorly for other types of data. Thus, there is a need to develop approaches
that would combine strength of specialized networks with ability to integrate
information across multiple streams as human and animal brain can do
efficiently.

###  1.2 Multi-modal fusion

Multimodal fusion has been previously explored for hard classification
problems. Proposed methods in literature include learning joint
representations from unimodal representations that are derived using
VLAD[DBLP:journals/corr/GongWGL14], Fisher Vector
representations[dixit_scene_2015], and deep features locally extracted from
CNN’s with various configurations[DBLP:journals/corr/WuWWY15][yoo_multi-
scale_2015][shen_deep_2019]. These approaches have been successfully applied
for action, scene and event recognition, and object detection tasks.

In [zhou_learning_2014], the authors used features extracted from Alexnet
pretrained on Places365 and Imagenet to show how internal representations of
these networks perform for various scene and object centric datasets.
Performance was not significantly superior to using a unimodal approach,
however, some of the advantages were found to be related to reducing data
bias. In [herranz_scene_2016] the authors demonstrated this by using
combinations of object and scene features aggregated at different scales to
build a more efficient joint representation that helped to mitigate dataset
bias induced by scale. Our fusion approach aligns most closely with the
methods proposed in these papers.

###  1.3 Adversarial Attacks

Image processing using deep convolutional neural nets (CNNs) has made
historical leaps in the last decade [krizhevsky_imagenet_2012, he_deep_2016,
szegedy_going_2015]. However, the same convolutional networks are susceptible
to small perturbations in data, even imperceptible to humans, that can result
in misclassification. There have been two main approaches for investigating
ANN robustness: adversarial machine learning and training data manipulation
[DBLP:journals/corr/abs-1901-10513]. Although it has been proposed that
adversarial and manipulation robustness can be increased through various
mechanisms during the training phase, recent research has shown that these
methods are mostly ineffective or their effectiveness is inconclusive
[DBLP:journals/corr/abs-1802-05666] [DBLP:journals/corr/abs-1808-08750];
[DBLP:journals/corr/abs-1802-00420].

Fast Gradient Sign Method (FGSM)[goodfellow_explaining_2014] is a popular one-
step attack that is easier to defend compared to the iterative variants like
Basic Iterative Method(BIM)[noauthor_alexey_nodate] or Projected Gradient
Descent (PGD). Adversarial training and its variants are defense methods
commonly employed for dealing with adversarial attacks. In [tramr2017ensemble]
the authors found that adversarial training is more robust with adversarial
examples generated from white box attacks (attacks designed against the
specifics of the underlying CNN architecture and weights) but it remains
vulnerable to black box transferred examples (examples generated in
architecture agnostic manner). To combat this, an ensemble model was proposed
that combines adversarial examples created from different source models and
substitute pretrained networks. In general, adversarial training on one type
of attack does not generalise to other attacks and can compromise
classification accuracy on clean, unperturbed data. For example, in
[kurakin_adversarial_2017] the authors demonstrated that adversarial
retraining on one step attacks do not protect against iterative attacks like
PGD. Consequently, adversarial training with multi step attack is regarded as
the state of the art method used for improving adversarial robustness for
white box and black box attacks and was initially proposed in [aleks2017deep].
Recently, [wong_fast_2020] showed that single-step adversarial training with
an attack similar to FGSM successfully yields models robust to white-box
attacks, if the stepsizes of the attack’s random and gradient step are
appropriately tuned. Several other methods are proposed such as adversarial
example detection, reconstructing adversarial inputs, network distillation
etc. and are discussed in further detail in [yuan_adversarial_2018].

###  1.4 Summary of our approach

In this paper, we describe the fusion of two data streams, one focused on
background (context) and another focused on the foreground (object) image
information, and we use different types of adversarial perturbations to
evaluate the efficacy of the fused representation.

We explore the following main concepts:

  * •

Adversarial attacks can have divergent effects on context feature space and
object feature space.

  * •

Utilizing combination of multiple modalities for the information processing
can be an efficient method for combating adversarial attacks.

  * •

Context features provide additional information to object-oriented data, and
can be used to improve classification, especially during adversarial attacks.

##  2 Methodology

###  2.1 Contextual fusion

We developed three different image classifiers designed to extract foreground
features, background features, or a fused version of both. Below we refer to
these as the foreground, background, and joint classifiers, respectively. The
distinction between the various classifiers is based on the underlying
training data. The foreground classifier was trained on the object-centric
Imagenet database, whereas the background classifier was trained on the scene-
centric Places365 database similar to [zhou_learning_2014]. Each classifier
was built on the Resnet18 architecture [he_deep_2016] with a final fully
connected layer specific to the dataset (Figure 1). Only the 512 dimensional
fully connected layer was finetuned for MS COCO [lin_microsoft_2015] or
CIFAR-10 [krizhevsky_learning_nodate]. For the joint classifier, we adopted a
late fusion strategy and concatenated features obtained from the foreground
and background pre-trained networks. This 1024 dimensional representation was
finetuned for the specific dataset.

Figure 1: Cartoon of architecture: foreground, background, and joint
classifiers.

###  2.2 Datasets

MS COCO dataset is a large dataset with 1.6 million images that have multiple
objects and multiple instances with overlapping contexts. We pruned the MS
COCO dataset for images with fewer than two instances (<=2) of a single object
so as to minimise cases of co-occurring context. We used available bounding
boxes to constrain the percentage of foreground to be less than or equal to
fifty percent of the total image, where foreground was defined by the area
within the bounding box. This significantly reduced the dataset size but
helped create the conditions for learning a representation that had enough
information for the background classifier to utilize. Our final dataset
comprised of 24 classes with 7500 images. We used 75% of the dataset for
training and the rest for testing. We also used CIFAR-10, which is a standard
object recognition dataset with 10 classes. For experiments with CIFAR-10, we
used the entire dataset with 50000 images for training and tested with the
remaining 10000 images. All images from have been resized and cropped to
224x224 and normalized with CIFAR-10 mean and standard deviation.

###  2.3 Adversarial attacks

We tested the classifiers with two different types of adversarial attacks to
evaluate the benefits of fusion: Gaussian blur and FGSM. Gaussian blur was
applied to the test set by convolving a portion of the image within the object
bounding box given in the dataset with Gaussian kernel. Differing degrees of
blur were created by varying the standard deviation of the kernel. Adversarial
example attacks were generated using FGSM similar to [szegedy_intriguing_2013,
goodfellow_explaining_2014]. Here, small amounts of noise were added to the
test images based on the gradient of the loss function with respect to the
input. Following [goodfellow_explaining_2014] we used the equation

$\eta=\epsilon\cdot sign\left(\nabla_{x}J(\theta,x,y)\right)$

where $\epsilon$ is a small real number, $J$ is the loss as a function of the
parameters ($\theta$), the input ($x$), and the label $y$.

Figure 2: Effect of object blurring on background and foreground features. PCA
projection to 2D-space is shown. Small and bright colored dots represent raw
images while large and dark colored dots represent blurred images. Left,
entire subspace of the foreground features moved (up in this example) in
presence of blur. Right, after application of blur background features remain
within the statistical subspace created by raw images. Each color represents a
single image class. A filter with Gauss kernel and $\sigma=5$ was used to blur
images.

##  3 Results

###  3.1 Blur affects foreground and background channels differently

The impact of adversarial attacks can vary with the CNN architecture and the
underlying training data [dodge_understanding_2016]. Here, we used two
distinct CNNs with the same Resnet18 architecture to serve as the background
and foreground feature extractors. The background feature extractor was
trained on the Places365 database [zhou_learning_2014], a scene-centric
collection of images with 365 scenes as categories (e.g. abbey, bedroom, and
library). The foreground feature extractor was trained on the object-centric
Imagenet database with object categories (e.g. goldfish, English setter,
toaster). The two different feature extractors represent the same images in
different ways and have a different sensitivity to the same adversarial
attack. We use pretrained models readily available in Pytorch.

Blur is a natural artifact common to many real-world acquired images.
Therefore, it is critical for an image classifier to maintain robustness to
blur. We created blurred images by convolution of the foreground of each image
with a Gaussian kernel. This type of Gaussian blur is perceivable by humans,
and for small standard deviations (e.g. $\sigma=0.001$), the human visual
system is able to perform a classification task with minimal mistakes. As we
increase the amount of blur (e.g. $\sigma=45$), both humans and neural
networks tend towards chance level performance. In our approach, only the
foreground pixels are modified leaving the context of the images intact.

We first tested effect of application of the Gaussian blur on images from the
MS COCO dataset. The blur was applied to the bounding box area of the image,
where main object was located. Images were processed by each of the two CNNs
independently and features were extracted from the batch normalization layers.
To illustrate effect of blur on the high-level foreground and background
features, we visualized the feature space using PCA to reduce dimensionality
for few representative image classes (see Figure 2). We found that blur had
differing effects between the background (scene-centric) and foreground
(object-centric) features. For the foreground features, blur caused a shift of
the entire statistical representation subspace, i.e., all blurred images moved
conjointly away from the non-blurred images (compare small vs large dots in
Figure 2). However, for background features, blurred images were represented
in the same statistical subspace as clean images. This shows, as one can
expect, that blurring foreground (object) alone will have larger impact on the
features extracted by the network focused on the foreground than on the
features coming from the network that was train to recognize a background.
This finding supports an idea that combined use of the foreground and
background classifiers to process images in a multi-modal fashion may help to
defend against adversarial attacks designed against specific image components.
Specifically, if adversarial attacks affected only one channel of information
(i.e. the foreground channel), then multi-modal integration could overcome
these attacks.

(a) All

(b) Gaussian Blur on foreground pixels

(c) Similar

(d) Dissimilar

Figure 3: Effect of Gaussian Blur on classification performance for MS COCO
data. Panels a,c,d show classification performance for different levels of
blur ($\sigma$). ‘All’ refers to the 24 classes from different supercategories
that remain after downsizing the dataset. ‘Dissimilar’ and ‘Similar’ are eight
classes randomly selected from the 24 classes such that the supercategories
are either distinct or the same (see Methodology). b) Example of high Gaussian
blur ($\sigma$ = 45) on foreground pixels within bounding box.

###  3.2 Gaussian blur and the effect of contextual fusion

To directly test the hypothesis about increased robustness of the fusion based
classifier against blur, we compared classification performance of the
foreground, background, and fused classifiers on the images where foreground
object was blurred. We found an increase in classification for all levels of
$\sigma$ with the largest increase over foreground classifier for $\sigma>5$
(Figure 3(a)).

Because, our joint fusion method depends on associating context with object,
we also tested separately images with Similar and Dissimilar background
context. To do this, we used the "supercategories" given in the MS COCO
dataset to refine test sets into a "Similar test set" where all images came
from different categories but the same supercategory (e.g. "dog", "horse", and
"sheep" - all animals) and the "Dissimilar test set" where images come from
distinct supercategories (e.g. "dog"-animal, "airplane"-vehicles, and
"toilet"-indoor). This changes our problem to either a coarse grained
classification e.g. classes from different supercategories like “Animals”,
“Vehicles”, “Indoor”, “Outdoor”, or a much harder fine grained classification,
e.g. classes from the same supercategory of “Animals”.

An increase in classification was evident for each subset of the MS COCO
dataset (Similar or Dissimilar) and almost all values of $\sigma$ (Figure 3(c)
and Figure 3(d)). The largest improvements for the joint classifier were seen
for moderate values of $\sigma$ and while using the Dissimilar test set (at
$\sigma=10$ the joint classifier outperforms the foreground classifier by 10%,
Figure 3(d)). Thus, if all images in the test set contained different contexts
(e.g. some were indoor images and others were outdoor images), then the joint
classier performed much better than the individual ones. This finding was
expected because contexts in the dissimilar test set contain more information
specific to the underlying class. However, even when using the Similar test
set, we found significant increases in classification over foreground alone.
For example, classification with the joint classifier was 5% higher at
$\sigma=10$.

###  3.3 Gradient-based attacks

Adversarial attacks were constructed based on adding the right type of noise
that maximizes the increase in the loss function, gradient-based attacks
[szegedy_intriguing_2013]. The magnitude of the added noise was quantified by
$\epsilon$ (see Methodology and [goodfellow_explaining_2014]). Here, we
explored how well a fusion strategy can defend against them. Adversarial
images were created from the full original images using FGSM method
[goodfellow_explaining_2014] applied to the foreground network, and
performance was tested with the foreground, background and joint networks.

For MS COCO dataset with all 24 classes (the ’All’ test set in Figure 3), we
found that classification accuracy for the foreground classifier quickly
dropped to chance with increasing $\epsilon$ \- strength of the attack (Figure
4(a)). The background classifier degraded at a much slower rate compared to
the foreground classifier. A joint classifier revealed somewhat intermediate
level of performance across a range of the attack strengths. Importantly,
joint classifier revealed the same level of performance as the foreground one
for intact images, suggesting that our approach can overcome a common problem
of the adversarial defense - degraded performance for intact images (Figure
4(a)). To reveal relative contribution of two networks (object-centric and
scene-centric) to the joint classifier, we examined the weights of the trained
joint classifier (Figure 4(b)) and found that the foreground and background
networks influenced the joint decision almost equally. Qualitatively, this
suggests that since the MS COCO dataset has object information embedded with
semantically meaningful context, the background classifier was able to weigh
in on the joint decision, thereby retaining performance above foreground
network when it was affected by adversarial attack.

(a) FGSM on ’All’ classes of MS COCO

(b) Weights of MS COCO joint classifer

(c) FGSM on CIFAR-10

(d) Weights of CIFAR-10 joint classifier

Figure 4: Effect of FGSM on MS COCO and CIFAR-10. a,c) FGSM attack on
foreground classifier of ’All’ categories from MS COCO (a) and CIFAR-10 (c).
$\epsilon$ indicates strength of attack. b,d) Average absolute value of the
weights from the last layers of the foreground and background networks to the
joint classifier for MS COCO (b) and CIFAR-10 (d).

We next tested our fusion strategy on CIFAR-10 (Figure 4(c)). This dataset is
much larger than the subset of MS COCO dataset we used, and it has established
performance baselines in literature for different types of adversarial
defenses[wong_fast_2020], [yan2018deep]. For the foreground classifier, we
trained all layers of a Resnet18 on CIFAR-10, with weights initialized from a
Resnet 18 pretrained on Imagenet. For the background classifier, we train only
the last fully connected layer of Resnet18 with weights initialized from a
Resnet18 pretrained on the Places365 dataset. This was done to ensure the
background feature extractor represents scene-centric information, without
overwriting to the object-based features in CIFAR-10. The joint classifier was
a concatenated model of the foreground and background classifiers as described
above such that only the last fully connected layer was trained and the model
parameters of the foreground and background classifiers were retained. We
found that the adversarial examples crafted using the foreground classifier
also affected the joint classifier with its performance being only marginally
above the performance of the foreground classifier. We further examined the
weights of the joint classifier after training (Figure 4(d)) which revealed
that the foreground features dominated the joint decision. This was most
likely due to the fact that the background classifier significantly under-
performed the foreground classifier at $\epsilon=0$ ( 60% vs 85%). The last
finding was not surprising because of the object-centric nature of the
CIFAR-10 dataset, and the lack of meaningful background information. Thus,
without significant input from background features, multimodal fusion failed
to increase performance in FGSM attacks.

These results suggest that although adversarial examples do transfer across
classifiers, a principled way to combine contextual information can be useful
for adversarial robustness without the added cost of adversarial retraining
and compromising accuracy on unperturbed data. This usefulness depended on two
factors: a) the modalities needed to be sufficiently distinct; and b) the
fusion mechanism needed to balance contribution of the information streams
from different modalities. Below, we explored how this balance can be struck
in an optimal way.

###  3.4 Regularization on known adversary

When it is known that the foreground is targeted by the attack, considerable
gains can be made using the multi-modal method, even when compared to
conventional methods like adversarial retraining. Inspired by the finding that
improvement in the joint network is related to the relative weighting of
background information, we employed regularization of the targeted foreground
network to enhance the performance of the joint network. The equation below
regularized the foreground weights of the joint network with L2 penalty
determined by a tuneable hyperparameter $\alpha$.

$L(a_{i},t_{i})=-\sum_{i}^{N}t_{i}log(\frac{e^{a_{i}}}{\sum_{j}^{C}a_{j}})+\alpha|\theta_{fg}|^{2}$

The first term was the standard cross entropy loss between the activation
vector $a_{i}$ and the target vector $t_{i}$ for all N samples from C classes.
The second term was the L2 penalty on the foreground weights, $\theta_{fg}$.
This allowed the network to bias the classifier decision towards the
background or the foreground based on the value of $\alpha$.

We also compared the performance of the regularized joint network with a
standard method to combat FGSM attacks, adversarial retraining
[aleks2017deep]. We found that tuning $\alpha$ had a strong effect on the
performance of the joint network (Figure 5). The highest levels of
regularization ($\alpha$ = 10) enabled the joint classifier to perform close
to the levels of the background classifier while retaining high performance on
the clean test set. Since there was a known adversary on the foreground, the
biasing mechanism described in the loss function equation above, can be used
to weight the contextual decision of the background classifier more than the
foreground. Surprisingly, the joint network with higher regulation
outperformed the adversarial retraining strategy at all levels of $\epsilon$.
It suggests that foreground regularization allows the user to select the
degree of a more informative object-centric channel as opposed to a scene-
centric channel that is less sensitive to the attack.

(a) FGSM on Dissimilar MS COCO dataset

(b) FGSM on All MS COCO dataset

(c) Weights with Dissimilar MS COCO dataset

(d) Weights with All MS COCO dataset

Figure 5: Regularization on the foreground weights of joint classifier. a)-b)
Varying values of $\alpha$ on MS COCO Dissimilar and All categories are shown
for the joint classifier in green. The range of $\alpha$ was between 0.1 and
10. Darker green colors indicate higher levels of $\alpha$. Examples for
adversarial retraining were generated with an attack strength $\epsilon$ =0.3.
c)-d) Average absolute value of foreground and background weights as a
function of $\alpha$

##  4 Conclusion

We present a novel method using semantic data fusion to increase robustness to
adversarial attacks. By using features from distinct information streams,
object-centered foreground and scene-centered background, we were able to
maintain higher classification performance in the face of targeted adversarial
attacks. We found that the degree of success for this method partially depends
on the amount of variability in the background data. Thus, our method was more
successful at maintaining robustness when objects came from distinct super
categories of data with different and distinct backgrounds. Regularization of
the foreground network enhanced this performance far above standard
adversarial training strategies. In this work, we used a simple method of
fusion - a single fully connected layer integrating the outputs of the object-
centric and scene-centric networks. Our approach can be easily scaled by (a)
adding other richer contexts and modalities (e.g., auditory, text) and (b) by
implementing more sophisticated fusion layers inspired by neuro-scientific
computational principles (e.g. recurrent networks, etc).

Our work may lead to better understanding of how knowledge is extracted from
experience in biological networks and how brain dynamics are shaped by the
development of a rich internal model of the world, including the ability to
predict the outcomes of current situations and one’s own actions in that
context. The sophistication and complexity of the brain processing to combine
different types of information streams to create internal model of the world
is arguably the basis for "cognitive reserve", which is a significant factor
in protection from age and disease related dementia, and so a deep
understanding of how it is created and expressed is of high societal impact.
Equally important, our study may provide insights into the algorithms that
evolution has devised to make predictions about optimal behaviour based on the
multimodal rich input from the world. This is a main issue in machine learning
and using insight from biology will undoubtedly lead to advances in machine
learning algorithms.


1 m Swope | 2022-09-01T01:00 | 2459823.54167-2459823.91311 | 8.9 | 523 | pre-L3.3
1 m Swope | 2022-09-02T00:49 | 2459824.53435-2459824.91263 | 9.1 | 550 | pre-L3.3
0.6 m TS | 2022-09-04T02:23 | 2459826.59957-2459826.86723 | 6.4 | 153 | pre-L3.4
0.6 m TS | 2022-09-05T02:22 | 2459827.59877-2459827.69795 | 2.4 | 81 | pre-L3.4
0.6 m TS | 2022-09-06T02:33 | 2459828.60646-2459828.67713 | 1.7 | 52 | pre-L3.4
0.6 m TS | 2022-09-12T01:31 | 2459834.56367-2459834.74939 | 4.5 | 256 | pre-L4.1
1 m LCOGT | 2022-09-12T14:10 | 2459835.09086-2459835.27470 | 4.4 | 198 | pre-L4.1
1 m LCOGT | 2022-09-12T20:01 | 2459835.33419-2459835.51788 | 4.4 | 187 | pre-L4.1
0.6 m TS | 2022-09-14T06:33 | 2459836.77356-2459836.90962 | 3.3 | 147 | pre-L4.1
0.6 m TS | 2022-09-15T04:34 | 2459837.69065-2459837.81569 | 3.0 | 145 | pre-L4.1
1.5 m Danish | 2022-09-16T08:22 | 2459838.84882-2459838.91024 | 1.5 | 98 | pre-L4.1
0.6 m TS | 2022-09-18T07:49 | 2459840.82589-2459840.90635 | 1.9 | 76 | pre-L4.2
0.5 m T72 | 2022-09-20T01:44 | 2459842.57286-2459842.74630 | 4.2 | 162 | pre-L4.2
1.5 m Danish | 2022-09-20T02:38 | 2459842.61009-2459842.90694 | 7.1 | 463 | pre-L4.2
0.6 m TS | 2022-09-20T02:46 | 2459842.61532-2459842.70663 | 2.2 | 120 | pre-L4.2
0.6 m TS | 2022-09-20T07:16 | 2459842.80324-2459842.90456 | 2.4 | 139 | pre-L4.2
0.6 m TS | 2022-09-21T03:46 | 2459843.65757-2459843.90548 | 5.9 | 243 | pre-L4.2
0.6 m TS | 2022-09-22T08:53 | 2459844.87053-2459844.90641 | 0.9 | 45 | pre-L4.2
0.6 m TS | 2022-09-23T06:47 | 2459845.78307-2459845.90582 | 2.9 | 161 | pre-L4.3
0.6 m TS | 2022-09-24T02:03 | 2459846.58601-2459846.89954 | 7.5 | 264 | pre-L4.3
0.6 m TS | 2022-09-25T02:11 | 2459847.59151-2459847.89606 | 7.3 | 375 | pre-L4.3
1 m Swope | 2022-09-25T03:16 | 2459847.63677-2459847.89319 | 6.2 | 435 | pre-L4.3
1.5 m Danish | 2022-09-25T03:28 | 2459847.64511-2459847.89944 | 6.1 | 569 | pre-L4.3
1 m LCOGT | 2022-09-25T14:50 | 2459848.11830-2459848.24055 | 2.9 | 172 | pre-L4.3
1.5 m Danish | 2022-09-26T03:50 | 2459848.66035-2459848.70365 | 1.0 | 92 | pre-L4.3
1 m Swope | 2022-09-26T02:15 | 2459848.59377-2459848.88760 | 7.1 | 429 | pre-L4.3
0.6 m TS | 2022-09-26T07:41 | 2459848.82069-2459848.89597 | 1.8 | 113 | pre-L4.3
1 m Swope | 2022-09-28T02:33 | 2459850.60635-2459850.75643 | 3.6 | 237 | L0.1
1.5 m Danish | 2022-09-28T02:38 | 2459850.60989-2459850.88674 | 6.6 | 340 | L0.1
1 m Swope | 2022-09-29T02:40 | 2459851.61163-2459851.88686 | 6.6 | 433 | L0.2
1.5 m Danish | 2022-09-29T02:50 | 2459851.61825-2459851.89795 | 6.7 | 639 | L0.2
1 m LCOGT | 2022-09-29T04:52 | 2459851.70320-2459851.86630 | 3.9 | 212 | L0.2
0.6 m TN | 2022-09-30T02:09 | 2459852.58974-2459852.72890 | 3.3 | 94 | L0.3
1.5 m Danish | 2022-09-30T02:40 | 2459852.61163-2459852.90395 | 7.0 | 669 | L0.3
1 m Swope | 2022-09-30T02:52 | 2459852.61965-2459852.88542 | 6.4 | 420 | L0.3
1.5 m TCS | 2022-09-30T03:36 | 2459852.65033-2459852.73971 | 2.1 | 86 | L0.3
1 m LCOGT | 2022-09-30T03:52 | 2459852.66165-2459852.88666 | 5.4 | 319 | L0.3
0.6 m TS | 2022-09-30T07:03 | 2459852.79385-2459852.88887 | 2.3 | 87 | L0.3
1 m LCOGT | 2022-09-30T21:45 | 2459853.40690-2459853.55059 | 3.4 | 168 | L0.4
1 m Swope | 2022-10-01T03:28 | 2459853.64455-2459853.88504 | 5.8 | 376 | L0.4
0.6 m TS | 2022-10-01T03:29 | 2459853.64552-2459853.88815 | 5.8 | 246 | L0.4
1 m LCOGT | 2022-10-01T04:00 | 2459853.66732-2459853.88294 | 5.2 | 292 | L0.4
1.5 m Danish | 2022-10-01T06:33 | 2459853.77308-2459853.89512 | 2.9 | 278 | L0.4
1 m LCOGT | 2022-10-01T21:48 | 2459854.40896-2459854.63187 | 5.3 | 268 | L0.5
1.5 m Danish | 2022-10-02T03:05 | 2459854.62906-2459854.85671 | 5.5 | 530 | L0.5
1 m Swope | 2022-10-02T03:15 | 2459854.63565-2459854.88820 | 6.1 | 396 | L0.5
1 m LCOGT | 2022-10-02T04:00 | 2459854.66735-2459854.88319 | 5.2 | 269 | L0.5
1.1 m Hall | 2022-10-02T08:19 | 2459854.84674-2459854.95490 | 2.6 | 132 | L0.5
1 m LCOGT | 2022-10-02T22:00 | 2459855.41728-2459855.53916 | 2.9 | 136 | L0.6
1 m LCOGT | 2022-10-03T01:00 | 2459855.54228-2459855.62661 | 2.0 | 99 | L0.6
1 m Swope | 2022-10-03T03:29 | 2459855.64571-2459855.89431 | 6.0 | 385 | L0.6
0.6 m TS | 2022-10-03T03:33 | 2459855.64845-2459855.87218 | 5.4 | 275 | L0.6
1 m LCOGT | 2022-10-03T04:05 | 2459855.67017-2459855.86302 | 4.6 | 248 | L0.6
1.5 m Danish | 2022-10-03T04:27 | 2459855.68573-2459855.72804 | 1.0 | 98 | L0.6
0.6 m TN | 2022-10-04T02:13 | 2459856.59250-2459856.70134 | 2.6 | 93 | L0.7
0.6 m TS | 2022-10-04T03:43 | 2459856.65517-2459856.86429 | 5.0 | 92 | L0.7
1 m Swope | 2022-10-04T03:46 | 2459856.65705-2459856.85069 | 4.6 | 224 | L0.7
1 m LCOGT | 2022-10-04T04:15 | 2459856.67770-2459856.86631 | 4.5 | 248 | L0.7
1 m LCOGT | 2022-10-04T23:15 | 2459857.46932-2459857.62841 | 3.8 | 206 | L0.8
1.5 m Danish | 2022-10-05T03:40 | 2459857.65293-2459857.74436 | 2.2 | 151 | L0.8
0.6 m TS | 2022-10-05T03:47 | 2459857.65778-2459857.74651 | 2.1 | 94 | L0.8
1.1 m Hall | 2022-10-05T08:03 | 2459857.83609-2459858.00200 | 4.0 | 142 | L0.8
1 m Swope | 2022-10-05T03:45 | 2459857.65664-2459857.77734 | 2.9 | 181 | L0.8
1 m LCOGT | 2022-10-05T22:18 | 2459858.42973-2459858.58280 | 3.7 | 194 | L0.9
1 m Swope | 2022-10-06T03:49 | 2459858.65972-2459858.88938 | 5.5 | 346 | L0.9
1 m LCOGT | 2022-10-06T04:31 | 2459858.68822-2459858.83991 | 3.6 | 194 | L0.9
1.5 m Danish | 2022-10-06T05:03 | 2459858.71052-2459858.89416 | 4.4 | 370 | L0.9
0.6 m TS | 2022-10-06T05:16 | 2459858.71951-2459858.85093 | 3.2 | 110 | L0.9
1.5 m Danish | 2022-10-07T04:10 | 2459859.67378-2459859.85302 | 4.3 | 379 | L0.10
1.5 m TCS | 2022-10-07T02:49 | 2459859.61755-2459859.75465 | 3.3 | 114 | L0.10
1 m LCOGT | 2022-10-07T04:52 | 2459859.70316-2459859.88078 | 4.3 | 219 | L0.10
1 m LCOGT | 2022-10-07T22:31 | 2459860.43826-2459860.62727 | 4.5 | 246 | L0.11
0.6 m TS | 2022-10-08T03:52 | 2459860.66117-2459860.73576 | 1.8 | 88 | L0.11
1.5 m Danish | 2022-10-08T04:00 | 2459860.66732-2459860.89540 | 5.5 | 471 | L0.11
1 m LCOGT | 2022-10-08T22:37 | 2459861.44249-2459861.62691 | 4.4 | 200 | L0.12
0.6 m TS | 2022-10-09T04:08 | 2459861.67266-2459861.86428 | 4.6 | 165 | L0.12
1 m LCOGT | 2022-10-09T04:49 | 2459861.70073-2459861.88372 | 4.4 | 244 | L0.12
1.5 m Danish | 2022-10-09T06:41 | 2459861.77911-2459861.89363 | 2.7 | 244 | L0.12
0.6 m TS | 2022-10-10T04:13 | 2459862.67614-2459862.86424 | 4.5 | 246 | L0.13
1.5 m Danish | 2022-10-10T04:30 | 2459862.68801-2459862.88784 | 4.8 | 419 | L0.13
1 m LCOGT | 2022-10-10T04:49 | 2459862.70070-2459862.79332 | 2.2 | 102 | L0.13
1.5 m TCS | 2022-10-17T02:50 | 2459869.61814-2459869.75936 | 3.4 | 114 | L1.1
0.9 m Spacewatch | 2022-10-17T10:43 | 2459869.94658-2459870.02078 | 1.8 | 51 | L1.1
0.9 m Spacewatch | 2022-10-20T09:01 | 2459872.87591-2459873.00474 | 3.1 | 92 | L1.2
2.4 m MRO | 2022-10-21T08:27 | 2459873.85238-2459873.99883 | 3.5 | 202 | L1.2
0.9 m Spacewatch | 2022-10-21T08:40 | 2459873.86150-2459874.01733 | 3.7 | 81 | L1.2
1.1 m Hall | 2022-10-24T08:06 | 2459876.83777-2459877.00597 | 4.0 | 81 | L1.3
1.1 m Hall | 2022-10-25T08:04 | 2459877.83659-2459878.00060 | 3.9 | 88 | L1.3
0.9 m Spacewatch | 2022-10-25T09:36 | 2459877.90001-2459878.01528 | 2.8 | 72 | L1.3
1.1 m Hall | 2022-10-26T08:05 | 2459878.83737-2459878.99494 | 3.8 | 91 | L1.3
1.8 m VATT | 2022-10-26T09:01 | 2459878.87629-2459879.01152 | 3.2 | 114 | L1.3
0.9 m Spacewatch | 2022-10-26T09:43 | 2459878.90541-2459879.01919 | 2.7 | 57 | L1.3
1.1 m Hall | 2022-10-27T08:00 | 2459879.83398-2459880.00799 | 4.2 | 94 | L1.3
0.9 m Spacewatch | 2022-10-27T10:20 | 2459879.93084-2459880.02873 | 2.3 | 69 | L1.3
0.6 m Ondrejov | 2022-10-28T00:59 | 2459880.54132-2459880.68111 | 3.4 | 94 | L1.4
1 m LCOGT | 2022-10-28T02:21 | 2459880.59792-2459880.75474 | 3.8 | 162 | L1.4
0.6 m TS | 2022-10-28T05:45 | 2459880.73984-2459880.87327 | 3.2 | 56 | L1.4
1.5 m Danish | 2022-10-29T07:18 | 2459881.80431-2459881.87030 | 1.6 | 69 | L1.4
1.1 m Hall | 2022-10-28T07:58 | 2459880.83197-2459881.01225 | 4.3 | 120 | L1.4
1.1 m Hall | 2022-10-29T07:50 | 2459881.82682-2459882.01607 | 4.5 | 110 | L1.4
1.8 m VATT | 2022-10-29T08:49 | 2459881.86752-2459882.02402 | 3.8 | 100 | L1.4
1 m LCOGT | 2022-10-30T02:16 | 2459882.59474-2459882.75793 | 3.9 | 178 | L1.4
0.6 m Ondrejov | 2022-10-30T02:20 | 2459882.59771-2459882.68875 | 2.2 | 36 | L1.4
1.5 m Danish | 2022-10-30T05:16 | 2459882.71997-2459882.86674 | 3.5 | 151 | L1.4
1.1 m Hall | 2022-10-31T07:47 | 2459883.82451-2459883.99910 | 4.2 | 124 | L1.4
1.5 m Danish | 2022-11-01T05:24 | 2459884.72555-2459884.86775 | 3.4 | 94 | L1.4
1.5 m Danish | 2022-11-02T05:27 | 2459885.72764-2459885.86534 | 3.3 | 92 | L1.4
0.9 m Spacewatch | 2022-11-02T09:27 | 2459885.89397-2459886.01292 | 2.9 | 69 | L1.4
1.5 m TCS | 2022-11-17T04:56 | 2459900.70591-2459900.77040 | 1.5 | 37 | L2.1
4.3 m LDT | 2022-11-17T07:24 | 2459900.80843-2459901.05212 | 5.8 | 379 | L2.1
0.6 m TN | 2022-11-18T01:04 | 2459901.54456-2459901.75044 | 4.9 | 125 | L2.1
1 m LCOGT | 2022-11-18T01:31 | 2459901.56323-2459901.75770 | 4.7 | 174 | L2.1
4.3 m LDT | 2022-11-18T07:26 | 2459901.81037-2459902.05936 | 6.0 | 479 | L2.2
0.9 m Spacewatch | 2022-11-19T08:34 | 2459902.85749-2459903.03832 | 4.3 | 113 | L2.2
1.1 m Hall | 2022-11-19T09:18 | 2459902.88771-2459903.02975 | 3.4 | 52 | L2.2
0.6 m TN | 2022-11-20T01:04 | 2459903.54492-2459903.74612 | 4.8 | 87 | L2.2
1.1 m Hall | 2022-11-20T06:47 | 2459903.78303-2459904.04012 | 6.2 | 108 | L2.2
0.9 m Spacewatch | 2022-11-20T07:41 | 2459903.82040-2459904.03469 | 5.1 | 147 | L2.2
0.6 m TN | 2022-11-21T01:48 | 2459904.57533-2459904.74878 | 4.2 | 103 | L2.3
1.5 m Danish | 2022-11-21T05:16 | 2459904.71981-2459904.86194 | 3.4 | 77 | L2.3
1.1 m Hall | 2022-11-21T06:42 | 2459904.77930-2459905.03178 | 6.1 | 104 | L2.3
0.9 m Spacewatch | 2022-11-21T09:03 | 2459904.87735-2459905.02906 | 3.6 | 103 | L2.3
1.5 m Danish | 2022-11-22T05:15 | 2459905.71932-2459905.85242 | 3.2 | 86 | L2.3
1 m LCOGT | 2022-11-22T01:16 | 2459905.55338-2459905.76103 | 5.0 | 159 | L2.3
2.4 m MRO | 2022-11-22T06:02 | 2459905.75173-2459905.91849 | 4.0 | 229 | L2.3
1.1 m Hall | 2022-11-22T06:35 | 2459905.77496-2459905.85868 | 2.0 | 39 | L2.3
1.1 m Hall | 2022-11-23T06:29 | 2459906.77020-2459907.04413 | 6.6 | 123 | L2.3
1.5 m Danish | 2022-11-24T05:12 | 2459907.71703-2459907.85243 | 3.2 | 77 | L2.4
1.1 m Hall | 2022-11-24T06:24 | 2459907.76714-2459908.03644 | 6.5 | 111 | L2.4
0.9 m Spacewatch | 2022-11-24T07:43 | 2459907.82161-2459908.04074 | 5.3 | 124 | L2.4
1.5 m AZT-22 | 2022-11-24T20:49 | 2459908.36785-2459908.44936 | 2.0 | 76 | L2.4
1 m LCOGT | 2022-11-25T01:01 | 2459908.54258-2459908.76786 | 5.4 | 166 | L2.4
1.1 m Hall | 2022-11-25T06:30 | 2459908.77131-2459909.04076 | 6.5 | 117 | L2.4
1.5 m Danish | 2022-11-25T06:39 | 2459908.77729-2459908.85502 | 1.9 | 47 | L2.4
0.9 m Spacewatch | 2022-11-25T08:28 | 2459908.85347-2459909.03805 | 4.4 | 99 | L2.4
1.5 m Danish | 2022-11-26T05:56 | 2459909.74773-2459909.85536 | 2.6 | 47 | L2.5
1.1 m Hall | 2022-11-26T06:14 | 2459909.76004-2459910.04018 | 6.7 | 100 | L2.5
1.5 m Danish | 2022-11-27T05:36 | 2459910.73347-2459910.85399 | 2.9 | 61 | L2.5
1.1 m Hall | 2022-11-27T06:09 | 2459910.75637-2459911.04136 | 6.8 | 125 | L2.5
1.5 m AZT-22 | 2022-11-27T19:24 | 2459911.30869-2459911.55133 | 5.8 | 160 | L2.5
0.6 m Ondrejov | 2022-11-27T22:02 | 2459911.41807-2459911.53662 | 2.8 | 44 | L2.6
1.5 m Danish | 2022-11-28T05:36 | 2459911.73380-2459911.78206 | 1.2 | 28 | L2.6
0.6 m Stara Lesna | 2022-11-28T23:27 | 2459912.47749-2459912.63264 | 3.7 | 58 | L2.6
1.1 m Hall | 2022-11-29T06:05 | 2459912.75399-2459913.01913 | 6.4 | 117 | L2.6
1 m LCOGT | 2022-11-30T00:51 | 2459913.53587-2459913.76175 | 5.4 | 162 | L2.6
1.5 m Danish | 2022-11-30T04:54 | 2459913.70480-2459913.85738 | 3.7 | 78 | L2.6
1.1 m Hall | 2022-11-30T06:03 | 2459913.75269-2459914.04306 | 7.0 | 114 | L2.6
2.4 m MRO | 2022-11-30T06:19 | 2459913.76355-2459913.99664 | 5.6 | 353 | L2.6
0.9 m Spacewatch | 2022-11-30T06:54 | 2459913.78763-2459914.03500 | 5.9 | 133 | L2.6
1 m LCOGT | 2022-11-30T06:50 | 2459913.78537-2459913.90697 | 2.9 | 89 | L2.6
1 m LCOGT | 2022-12-01T02:46 | 2459914.61557-2459914.74010 | 3.0 | 88 | L2.7
1.1 m Hall | 2022-12-01T05:48 | 2459914.74174-2459914.93801 | 4.7 | 82 | L2.7
1.5 m Danish | 2022-12-01T06:06 | 2459914.75456-2459914.85574 | 2.4 | 51 | L2.7
2.4 m MRO | 2022-12-01T06:15 | 2459914.76075-2459914.97346 | 5.1 | 240 | L2.7
0.9 m Spacewatch | 2022-12-01T07:13 | 2459914.80119-2459915.02296 | 5.3 | 142 | L2.7
0.5 m Sugarloaf | 2022-12-02T05:32 | 2459915.73117-2459915.94429 | 5.1 | 44 | L2.7
2 m Faulkes-N | 2022-12-14T10:02 | 2459927.91809-2459928.13465 | 5.2 | 249 | L3.1
4.3 m LDT | 2022-12-15T07:06 | 2459928.79595-2459929.06886 | 6.5 | 488 | L3.1
1 m LCOGT | 2022-12-16T23:41 | 2459930.48726-2459930.65677 | 4.1 | 116 | L3.2
1.5 m Danish | 2022-12-17T04:07 | 2459930.67167-2459930.85421 | 4.4 | 64 | L3.2
1.1 m Hall | 2022-12-17T04:23 | 2459930.68277-2459931.02634 | 8.2 | 120 | L3.2
0.6 m TN | 2022-12-18T22:18 | 2459932.42977-2459932.72830 | 7.2 | 139 | L3.3
1.5 m Danish | 2022-12-19T03:42 | 2459932.65464-2459932.85511 | 4.8 | 120 | L3.3
1.1 m Hall | 2022-12-19T04:10 | 2459932.67420-2459933.00020 | 7.8 | 128 | L3.3
1 m Tian-Shan | 2022-12-19T16:09 | 2459933.17361-2459933.43726 | 6.3 | 148 | L3.3
1.5 m Danish | 2022-12-20T03:54 | 2459933.66318-2459933.82730 | 3.9 | 79 | L3.3
0.8 m IAC80 | 2022-12-20T04:05 | 2459933.67079-2459933.79256 | 2.9 | 46 | L3.3
1.1 m Hall | 2022-12-20T04:51 | 2459933.70215-2459933.83654 | 3.2 | 58 | L3.3
0.6 m TN | 2022-12-21T00:01 | 2459934.50107-2459934.64302 | 3.4 | 73 | L3.3
1.1 m Hall | 2022-12-21T04:15 | 2459934.67758-2459934.99702 | 7.7 | 117 | L3.3
0.5 m Sugarloaf | 2022-12-21T04:06 | 2459934.67103-2459934.95380 | 6.8 | 127 | L3.3
0.7 m AC-32 | 2022-12-21T18:51 | 2459935.28582-2459935.60357 | 7.6 | 120 | L3.4
1.5 m Danish | 2022-12-22T03:31 | 2459935.64693-2459935.85438 | 5.0 | 115 | L3.4
1.5 m Danish | 2022-12-23T04:03 | 2459936.66886-2459936.82079 | 3.6 | 86 | L3.4
0.7 m AC-32 | 2022-12-23T17:49 | 2459937.24273-2459937.56234 | 7.7 | 127 | L3.4
1.8 m BOAO | 2022-12-24T14:05 | 2459938.08730-2459938.37919 | 7.0 | 129 | L3.5
2.4 m MRO | 2022-12-25T03:37 | 2459938.65106-2459938.98469 | 8.0 | 465 | L3.5
1.1 m Hall | 2022-12-25T04:20 | 2459938.68114-2459938.77319 | 2.2 | 39 | L3.5
1.1 m Hall | 2022-12-26T04:46 | 2459939.69905-2459939.99006 | 7.0 | 124 | L3.5
0.5 m Sugarloaf | 2022-12-25T03:36 | 2459938.65060-2459938.94060 | 7.0 | 32 | L3.5
0.5 m Sugarloaf | 2022-12-26T04:02 | 2459939.66866-2459939.94683 | 6.7 | 96 | L3.5
1.8 m BOAO | 2022-12-26T13:04 | 2459940.04469-2459940.36915 | 7.8 | 160 | L3.5
0.6 m Ondrejov | 2022-12-27T22:33 | 2459941.43962-2459941.62416 | 4.4 | 73 | L3.6
1.5 m Danish | 2022-12-30T02:53 | 2459943.62060-2459943.85782 | 5.7 | 141 | L3.6
1.5 m Danish | 2022-12-31T04:15 | 2459944.67764-2459944.85225 | 4.2 | 103 | L3.6
1 m LCOGT | 2022-12-31T06:16 | 2459944.76126-2459944.89522 | 3.2 | 98 | L3.6
1.5 m Danish | 2023-01-11T01:50 | 2459955.57649-2459955.81858 | 5.8 | 103 | L4.1
1 m LCOGT | 2023-01-14T05:46 | 2459958.74075-2459958.90387 | 3.9 | 67 | L4.1
0.6 m TN | 2023-01-15T00:13 | 2459959.50919-2459959.65642 | 3.5 | 73 | L4.1
1.5 m Danish | 2023-01-15T01:32 | 2459959.56451-2459959.80226 | 5.7 | 96 | L4.1
1 m LCOGT | 2023-01-16T07:23 | 2459960.80828-2459960.94995 | 3.4 | 64 | L4.1
1.5 m Danish | 2023-01-18T01:26 | 2459962.55987-2459962.79337 | 5.6 | 98 | L4.1
1.5 m Danish | 2023-01-24T01:02 | 2459968.54363-2459968.69672 | 3.7 | 55 | L4.2
1 m LCOGT | 2023-01-24T20:06 | 2459969.33776-2459969.58969 | 6.0 | 102 | L4.2
4.3 m LDT | 2023-01-25T02:08 | 2459969.58954-2459970.01919 | 10.3 | 218 | L4.2
1 m LCOGT | 2023-01-26T01:57 | 2459970.58135-2459970.80766 | 5.4 | 93 | L4.2
1 m LCOGT | 2023-01-26T21:19 | 2459971.38821-2459971.55048 | 3.9 | 72 | L4.2
1.5 m Danish | 2023-01-28T01:01 | 2459972.54294-2459972.76243 | 5.3 | 73 | L4.2
2.4 m MRO | 2023-01-28T01:45 | 2459972.57311-2459972.75924 | 4.5 | 169 | L4.2
1.5 m Danish | 2023-01-29T01:30 | 2459973.56271-2459973.76053 | 4.7 | 80 | L4.2
2.4 m MRO | 2023-01-29T02:19 | 2459973.59682-2459973.74686 | 3.6 | 116 | L4.2
2.4 m MRO | 2023-01-30T02:24 | 2459974.60024-2459974.74168 | 3.4 | 77 | L4.2
2.4 m MRO | 2023-02-11T01:42 | 2459986.57115-2459986.90646 | 8.0 | 255 | L5.1
4.3 m LDT | 2023-02-17T02:32 | 2459992.60565-2459992.93700 | 8.0 | 80 | L5.2
2.4 m MRO | 2023-02-17T07:06 | 2459992.79600-2459992.93543 | 3.3 | 44 | L5.2
4.3 m LDT | 2023-02-21T02:15 | 2459996.59395-2459996.85968 | 6.4 | 117 | L5.2
2.4 m MRO | 2023-02-25T03:21 | 2460000.63968-2460000.82854 | 4.5 | 255 | L5.2

##  Appendix B Supplemental Material

All of the lightcurve measurements and associated decompositions are included
here as a supplemental data file. The columns in this file are: light time
corrected Julian Date, measured apparent magnitude, differential magnitudes,
decomposed residuals, and IDs for the observing runs and decompositions.

## References

  * Akhlaghi (2019) Akhlaghi, M. 2019, arXiv e-prints, arXiv:1909.11230.  https://arxiv.org/abs/1909.11230
  * Akhlaghi & Ichikawa (2015) Akhlaghi, M., & Ichikawa, T. 2015, ApJS, 220, 1, doi: 10.1088/0067-0049/220/1/1
  * Astropy Collaboration et al. (2013) Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., et al. 2013, A&A, 558, A33, doi: 10.1051/0004-6361/201322068
  * Astropy Collaboration et al. (2018) Astropy Collaboration, Price-Whelan, A. M., Sipőcz, B. M., et al. 2018, AJ, 156, 123, doi: 10.3847/1538-3881/aabc4f
  * Bagnulo et al. (2023) Bagnulo, S., Gray, Z., Granvik, M., et al. 2023, ApJ, 945, L38, doi: 10.3847/2041-8213/acb261
  * Barbary (2018) Barbary, K. 2018, SEP: Source Extraction and Photometry, Astrophysics Source Code Library, record ascl:1811.004.  http://ascl.net/1811.004
  * Bertin (2006) Bertin, E. 2006, in Astronomical Society of the Pacific Conference Series, Vol. 351, Astronomical Data Analysis Software and Systems XV, ed. C. Gabriel, C. Arviset, D. Ponz, & S. Enrique, 112
  * Bertin & Arnouts (1996) Bertin, E., & Arnouts, S. 1996, A&AS, 117, 393, doi: 10.1051/aas:1996164
  * Bessell (1990) Bessell, M. S. 1990, PASP, 102, 1181, doi: 10.1086/132749
  * Bowell et al. (1989) Bowell, E., Hapke, B., Domingue, D., et al. 1989, in Asteroids II, ed. R. P. Binzel, T. Gehrels, & M. S. Matthews, 524–556
  * Brown et al. (2013) Brown, T. M., Baliber, N., Bianco, F. B., et al. 2013, PASP, 125, 1031, doi: 10.1086/673168
  * Chabot et al. (2023) Chabot, N., Rivkin, A., & Cheng, A. e. a. 2023,
  * Cheng et al. (2016) Cheng, A. F., Michel, P., Jutzi, M., et al. 2016, Planet. Space Sci., 121, 27, doi: 10.1016/j.pss.2015.12.004
  * Clemens et al. (2004) Clemens, J. C., Crain, J. A., & Anderson, R. 2004, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 5492, Ground-based Instrumentation for Astronomy, ed. A. F. M. Moorwood & M. Iye, 331–340, doi: 10.1117/12.550069
  * Collins et al. (2017) Collins, K. A., Kielkopf, J. F., Stassun, K. G., & Hessman, F. V. 2017, AJ, 153, 77, doi: 10.3847/1538-3881/153/2/77
  * Daly et al. (2023) Daly, R. T., Ernst, C. M., Barnouin, O. S., et al. 2023, Nature, 616, 443, doi: 10.1038/s41586-023-05810-5
  * Dotto & Zinzi (2023) Dotto, E., & Zinzi, A. 2023, Nature Communications, 14, 3055, doi: 10.1038/s41467-023-38705-0
  * Dotto et al. (2021) Dotto, E., Della Corte, V., Amoroso, M., et al. 2021, Planet. Space Sci., 199, 105185, doi: 10.1016/j.pss.2021.105185
  * Dressler et al. (2011) Dressler, A., Bigelow, B., Hare, T., et al. 2011, PASP, 123, 288, doi: 10.1086/658908
  * Fahnestock et al. (2022) Fahnestock, E. G., Cheng, A. F., Ivanovski, S., et al. 2022, , 3, 206, doi: 10.3847/PSJ/ac7fa1
  * Fletcher et al. (2018) Fletcher, Z. J., Ryan, K. J., Maas, B. J., et al. 2018, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10698, Space Telescopes and Instrumentation 2018: Optical, Infrared, and Millimeter Wave, ed. M. Lystrup, H. A. MacEwen, G. G. Fazio, N. Batalha, N. Siegler, & E. C. Tong, 106981X, doi: 10.1117/12.2310136
  * Flewelling et al. (2020) Flewelling, H. A., Magnier, E. A., Chambers, K. C., et al. 2020, ApJS, 251, 7, doi: 10.3847/1538-4365/abb82d
  * Gaia Collaboration et al. (2018) Gaia Collaboration, Brown, A. G. A., Vallenari, A., et al. 2018, A&A, 616, A1, doi: 10.1051/0004-6361/201833051
  * Gaia Collaboration et al. (2021) —. 2021, A&A, 649, A1, doi: 10.1051/0004-6361/202039657
  * Gilliland & Brown (1988) Gilliland, R. L., & Brown, T. M. 1988, PASP, 100, 754, doi: 10.1086/132232
  * Ginsburg et al. (2019) Ginsburg, A., Sipőcz, B. M., Brasseur, C. E., et al. 2019, AJ, 157, 98, doi: 10.3847/1538-3881/aafc33
  * Graykowski et al. (2023) Graykowski, A., Lambert, R. A., Marchis, F., et al. 2023, Nature, 616, 461, doi: 10.1038/s41586-023-05852-9
  * Hasselmann et al. (2023) Hasselmann, P., Della Corte, V., & Pravec, P. e. a. 2023,
  * Hasselmann (2023) Hasselmann, P. e. a. 2023, in Asteroids, Comets, Meteors Conference, Asteroids, Comets, Meteors Conference
  * Huang et al. (2017) Huang, X.-J., Lu, X.-P., Li, J.-Y., et al. 2017, Research in Astronomy and Astrophysics, 17, 106, doi: 10.1088/1674-4527/17/10/106
  * Im et al. (2010) Im, M.-S., Ko, J.-W., Cho, Y.-S., et al. 2010, Journal of Korean Astronomical Society, 43, 75, doi: 10.5303/JKAS.2010.43.3.075
  * Jackson et al. (2022) Jackson, S. L., Rozitis, B., Dover, L. R., et al. 2022, MNRAS, 513, 3076, doi: 10.1093/mnras/stac1053
  * Jehin et al. (2011) Jehin, E., Gillon, M., Queloz, D., et al. 2011, The Messenger, 145, 2
  * Jester et al. (2005) Jester, S., Schneider, D. P., Richards, G. T., et al. 2005, AJ, 130, 873, doi: 10.1086/432466
  * Jewitt et al. (2023) Jewitt, D., Kim, Y., Li, J., & Mutchler, M. 2023, ApJ, 952, L12, doi: 10.3847/2041-8213/ace1ec
  * Kaasalainen & Torppa (2001) Kaasalainen, M., & Torppa, J. 2001, Icarus, 153, 24, doi: 10.1006/icar.2001.6673
  * Kareta et al. (2023) Kareta, T., Thomas, C., J.-Y., L., Knight, M., & Moskovitz, N. e. a. 2023, Nature
  * Kelley & Lister (2022) Kelley, M., & Lister, T. 2022, mkelley/calviacat: v1.3.0, v1.3.0, Zenodo, doi: 10.5281/zenodo.7019180
  * Kitazato et al. (2004) Kitazato, K., Abe, M., Mito, H., et al. 2004, in Lunar and Planetary Science Conference, ed. S. Mackwell & E. Stansbery, Lunar and Planetary Science Conference, 1623
  * Landolt (1992) Landolt, A. U. 1992, AJ, 104, 340, doi: 10.1086/116242
  * Lister et al. (2021) Lister, T. A., Gomez, E., Chatelain, J., et al. 2021, Icarus, 364, 114387, doi: 10.1016/j.icarus.2021.114387
  * Mahlke et al. (2021) Mahlke, M., Carry, B., & Denneau, L. 2021, Icarus, 354, 114094, doi: 10.1016/j.icarus.2020.114094
  * McCully et al. (2018) McCully, C., Volgenau, N. H., Harbeck, D. R., et al. 2018, in Proc. SPIE, Vol. 10707, Observatory Operations: Strategies, Processes, and Systems VII (SPIE), 107070K
  * Meyer et al. (2021) Meyer, A. J., Gkolias, I., Gaitanas, M., et al. 2021, , 2, 242, doi: 10.3847/PSJ/ac3bd1
  * Michel et al. (2022) Michel, P., Küppers, M., Bagatin, A. C., et al. 2022, , 3, 160, doi: 10.3847/PSJ/ac6f52
  * Mommert (2017) Mommert, M. 2017, Astronomy and Computing, 18, 47, doi: 10.1016/j.ascom.2016.11.002
  * Mommert et al. (2019) Mommert, M., Kelley, M., de Val-Borro, M., et al. 2019, The Journal of Open Source Software, 4, 1426, doi: 10.21105/joss.01426
  * Moskovitz et al. (2022) Moskovitz, N. A., Wasserman, L., Burt, B., et al. 2022, Astronomy and Computing, 41, 100661, doi: 10.1016/j.ascom.2022.100661
  * Mottola et al. (1995) Mottola, S., De Angelis, G., Di Martino, M., et al. 1995, Icarus, 117, 62, doi: 10.1006/icar.1995.1142
  * Muinonen et al. (2010) Muinonen, K., Belskaya, I. N., Cellino, A., et al. 2010, Icarus, 209, 542, doi: 10.1016/j.icarus.2010.04.003
  * Naidu et al. (2023) Naidu, S., Chesley, S. R., & Moskovitz, N. e. a. 2023,
  * Naidu et al. (2022) Naidu, S. P., Chesley, S. R., Farnocchia, D., et al. 2022, , 3, 234, doi: 10.3847/PSJ/ac91c0
  * Naidu et al. (2020) Naidu, S. P., Benner, L. A. M., Brozovic, M., et al. 2020, Icarus, 348, 113777, doi: 10.1016/j.icarus.2020.113777
  * Narita et al. (2019) Narita, N., Fukui, A., Kusakabe, N., et al. 2019, Journal of Astronomical Telescopes, Instruments, and Systems, 5, 015001, doi: 10.1117/1.JATIS.5.1.015001
  * Narita et al. (2020) Narita, N., Fukui, A., Yamamuro, T., et al. 2020, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 11447, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, 114475K, doi: 10.1117/12.2559947
  * Niels Bohr Institute et al. (2014) Niels Bohr Institute, Institute of Astronomy, C., & Real Instituto y Observatorio de La Armada. 2014, VizieR Online Data Catalog, I/327
  * Oszkiewicz et al. (2020) Oszkiewicz, D., Troianskyi, V., Föhring, D., et al. 2020, A&A, 643, A117, doi: 10.1051/0004-6361/202038062
  * Oszkiewicz et al. (2021) Oszkiewicz, D., Wilawer, E., Podlewska-Gaca, E., et al. 2021, Icarus, 357, 114158, doi: 10.1016/j.icarus.2020.114158
  * Oszkiewicz et al. (2023) Oszkiewicz, D., Troianskyi, V., Galád, A., et al. 2023, Icarus, 397, 115520, doi: 10.1016/j.icarus.2023.115520
  * Parrott (2020) Parrott, D. 2020, , 48, 262
  * Penttilä et al. (2016) Penttilä, A., Shevchenko, V. G., Wilkman, O., & Muinonen, K. 2016, Planet. Space Sci., 123, 117, doi: 10.1016/j.pss.2015.08.010
  * Pravec et al. (2012) Pravec, P., Harris, A. W., Kušnirák, P., Galád, A., & Hornoch, K. 2012, Icarus, 221, 365, doi: 10.1016/j.icarus.2012.07.026
  * Pravec et al. (2000) Pravec, P., Šarounová, L., Rabinowitz, D. L., et al. 2000, Icarus, 146, 190, doi: 10.1006/icar.2000.6375
  * Pravec et al. (2003) Pravec, P., Benner, L. A. M., Nolan, M. C., et al. 2003, IAU Circ., 8244, 2
  * Pravec et al. (2006) Pravec, P., Scheirich, P., Kušnirák, P., et al. 2006, Icarus, 181, 63, doi: 10.1016/j.icarus.2005.10.014
  * Pravec et al. (2022) Pravec, P., Thomas, C. A., Rivkin, A. S., et al. 2022, , 3, 175, doi: 10.3847/PSJ/ac7be1
  * Rivkin et al. (2021) Rivkin, A. S., Chabot, N. L., Stickle, A. M., et al. 2021, , 2, 173, doi: 10.3847/PSJ/ac063e
  * Sanchez et al. (2019) Sanchez, J. A., Reddy, V., Thirouin, A., et al. 2019, ApJ, 881, L6, doi: 10.3847/2041-8213/ab31ac
  * Scheirich & Pravec (2022) Scheirich, P., & Pravec, P. 2022, , 3, 163, doi: 10.3847/PSJ/ac7233
  * Scheirich et al. (2023) Scheirich, P., Pravec, P., & Meyer, A. J. e. a. 2023,
  * Stickle et al. (2022) Stickle, A. M., DeCoster, M. E., Burger, C., et al. 2022, , 3, 248, doi: 10.3847/PSJ/ac91cc
  * Thomas et al. (2023) Thomas, C. A., Naidu, S. P., Scheirich, P., et al. 2023, Nature, 616, 448, doi: 10.1038/s41586-023-05805-2
  * Tody (1986) Tody, D. 1986, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 627, Instrumentation in astronomy VI, ed. D. L. Crawford, 733, doi: 10.1117/12.968154
  * Tonry et al. (2012) Tonry, J. L., Stubbs, C. W., Lykke, K. R., et al. 2012, ApJ, 750, 99, doi: 10.1088/0004-637X/750/2/99
  * Tonry et al. (2018) Tonry, J. L., Denneau, L., Flewelling, H., et al. 2018, ApJ, 867, 105, doi: 10.3847/1538-4357/aae386


11institutetext:  $\ast\ $
$\ast\ $
$\ast\ $ 11institutetext: Computer Aided Medical Procedures, Technical
University of Munich, Germany
11email: {shervin.dehghani, b.busam, nassir.navab,
<EMAIL_ADDRESS>Computer Aided Medical Procedures, Johns
Hopkins University, Baltimore, USA 33institutetext: Augenklinik rechts der
Isar, Technical University of Munich, Germany

# BFS-Net: Weakly Supervised Cell Instance Segmentation from Bright-Field
Microscopy Z-Stacks

$\ast\ $
$\ast\ $ Shervin Dehghani 11 Benjamin Busam 11 Nassir Navab 11 2 2 M. Ali
Nasseri 11 3 3

###### Abstract

Despite its broad availability, volumetric information acquisition from
Bright-Field Microscopy (BFM) is inherently difficult due to the projective
nature of the acquisition process. We investigate the prediction of 3D cell
instances from a set of BFM Z-Stack images. We propose a novel two-stage
weakly supervised method for volumetric instance segmentation of cells which
only requires approximate cell centroids annotation. Created pseudo-labels are
thereby refined with a novel refinement loss with Z-stack guidance. Evaluation
shows that our approach can generalize not only to BFM Z-Stack data, but to
other 3D cell imaging modalities. A comparison of our pipeline against fully
supervised methods indicates that the significant gain in reduced data
collection and labelling results in minor performance difference.

###### Keywords:

Bright-Field Microscopy Z-Stack Cell Instance Segmentation Weak Supervision

##  1 Introduction

Figure 1: BFS-Net Data Flow. We learn from a Bright-Field Z-Stack of cells
(1st from left) to predict 3D Centres (2nd) which create Pseudo Segmentation
labels (3rd) to learn 3D Segmentation (4th). The estimates are processed to
provide 3D Instance Segmentations (right).

Automatic micromanipulation of biological cells in 3D space, gained
significant interests in the recent years for various applications [1]. The
level and precision of automation for micromanipulation tasks is highly
dependant on the performance of the 3D localization components in the
micromanipulation setup. Such a localization methods are often utilized to
plan the manipulator trajectory, preview the operation and execute the
manipulation tasks. One of the critical aspects for spatial manipulation of
cells is reasoning on the distribution of cells at different depths within the
situs which is a challenging task from a single 2D microscopy image due to the
projective nature of the acquisition process. Due to the inherent ambiguities
of such an estimation, one cannot uniquely determine the 3D location of the
cells from a single all-in-focus 2D image [2].

One way to address this is by using Electron Microscopy. While providing
highly accurate resolution, such hardware can be costly and is not broadly
available. Moreover, it needs to be used in a specific environmental setup
that requires an isolation chamber typically without access from the outside
during a scan [3]. In order to be able to manipulate cells reasonably fast and
with acceptable performance and speed, an open environment is essential such
that a manipulator can be integrated seamlessly. In practice, Bright-Field
Microscopy (BFM) is arguably the most widely spread modality for cellular
imaging with its comparably lower cost. While naturally being available in
many biology and medical labs, it also has the potential of volumetric imaging
by integration of a micron precision step motor for controlling the Z plane
[4]. The captured 3D volume from such an enhanced BFM is different than a
depth-aware 3D volume as can be seen in the example in Fig. 1. Due to the
effect of BFM-projection in Z direction, the cell image is composed of
information from in-focus and out-of-focus parts within the visible area.
Different regions are in focus, depending on the Z position within the stack.
While being of projective nature, the data provides focus cues that can be
leveraged to estimate depth together with the necessary morphological features
to spatially separate the cells [5].

Annotation of instance volumes can be intricate and time-consuming. We
therefore leverage BFM Z-Stacks and propose a weakly supervised approach for
segmentation of cells within the volume. We first train a cell centre
prediction network which we use to create pseudo labels. These labels are used
jontly with an image-guided refinement loss to train our BFS-Net segmentation
model. To this end, our main contributions are:

  * •

We propose the first weakly supervised instance segmentation method for cells
from BFM Z-Stack inputs that only needs cell centre annotations for training.

  * •

We propose a novel Z-Stack guided refinement loss which is used to train a 3D
segmentation network with centre coordinate induced pseudo labels.

  * •

We make the BFM Z-Stack cell dataset publically available to stimulate
research in this new domain.

##  2 Related Works

Z-Stack Cell Segmentation. Despite its wide availability surprisingly little
work investigates BFM Z-Stacks. The pioneering work of [4] investigates
instance segmentation from this modality. In comparison to our approach,
however, they (1) train with full supervision and (2) assign the same label to
each pixel coordinate in the entire stack. We believe that the 3D local
neighbourhood can be advantageous and estimate volumetric labels.

2D/3D Cell Segmentation. 2D cell instance segmentation methods incorporate
variants of U-Net [6] combined with post-processing [7] steps. The main
challenge for these pipelines is to separate the cells and achieve instance-
wise precision. 3D cell segmentation approaches similarly use V-Net [8] or
3DUNet [9] backbones. The recent segmentation work of [10] estimates instances
in a post post-processing stage taking advantage of each cell center.

Weakly Supervised Cell Segmentation. Weakly supervised methods can be
separated into two branches based on the used labels being either bounding
boxes or cell centroids. [11] uses point annotations as supervision and
classifies each pixel based on a combination of k-means and Voronoi
tessellation which results in imprecise predictions close to cell edges. [12]
attempts to solve this issue trough consecutive edge detection. Another branch
of works is based around the idea of [13], who introduce Peak Response Maps
(PRM). PRMs indicate relevant visual regions created by backpropagation of
peak responses from a classifier similar to saliency maps [14]. [15] and [16]
use PRMs with center points and [17] with bounding box annotation to generate
segmentation through these visual cues. This family of methods need a post-
processing step to refine the results, and require $\mathcal{O}(n)$ back-
propagations at inference which significantly hampers their efficiency for the
prediction of $n$ cells in one input volume. We utilize PRMs only in the first
stage of our training to create pseudo labels which are refined in a second
stage omitting the costly back-propagation at test time.

##  3 Methodology

Figure 2: Pipeline Overview. Creation of Pseudo Labels from centroid
annotations (Row 1), Training with Pseudo Labels to achieve refined
segmentation mask and creating the Instance Segmentations (Row 2). For
simplification only one slice of the volume is shown.

An overview of our pipeline is illustrated in Fig. 2. We first train a
regression model $S1$ to predict cell center likelihood maps. With $S1$, we
generate the Response Map of each of the local maxima of the likelihood map to
obtain pseudo segmentation labels. Previous methods such as [15] and [16]
improve upon back-propagated labels at test time. To omit this additional
computational cost and further improve the results, we use a second training
stage to train the model $S2$ that learns with these labels and refines the
results under Z-Stack image guidance. With the segmentation mask from $S2$ and
the predicted center maps from $S1$, we consecutively apply an efficient label
assignment to separate instances. We detail each step hereafter.

###  3.1 Creation of Cell Centre Maps

Similar to [18] we generate a Gaussian based voxel map around each center
point. In addition, we divide the space into Voronoi sub-spaces. The borders
of these sub-spaces are hard-negative cases, and we override the Gaussian
value with $0$. This helps learning the cell centers in a way which they are
separable, even if being adjacent. The voxel likelihood map can be formulated
as follows:

$p(x)=\begin{cases}0,&\text{if}\ x\in\mathcal{B_{V}}\\\
e^{-k\frac{\mathcal{D}(x)}{d_{m}}},&\text{else if}\ \mathcal{D}(x)\leq
d_{m}\\\ 0&\text{otherwise}\end{cases}$ (1)

where $\mathcal{B_{V}}$ is the set of voxels which are on Voronoi borders in
the volume $\mathcal{V}$, $\mathcal{D}(x)$ is the Euclidean distance of voxel
$x$ to the closest center point, and $d_{m}$ and $k$ are the parameters to
control the distribution.

###  3.2 Centroid Regression Network and Pseudo Labels

With the cell centre maps, we train a regression 3DUNet $S1$ for cell center
likelihood maps. The training loss for this stage is:

$\mathcal{L}_{S1}=\mathcal{L}_{bce}+\lambda\mathcal{L}_{focal}$ (2)

where $\mathcal{L}_{bce}$ is a Binary Cross Entropy loss and
$\mathcal{L}_{focal}$ is a focal loss, applied on voxels with $p(x)>0.7$. As
in [13], we are interested to have the PRMs of the model output. For each of
the local maxima stimulated from model’s output, we run a back-propagation
through $S1$ to get the visual cue from the input which results in a predicted
local peak. Each of these masks belong to a separate instance, but one
instance can have multiple masks. Similar to [16], we use a back-propagation
method in 3D which extends the 2D proposal of [17]. Although being very
informative, these response maps cannot be used as a final segmentation
result, since they lack sufficient precision. While others use these masks and
perform post-processing with e.g. Graph Cuts [15], we store them as pseudo
labels to train a second network $S2$ to directly predict refined peak
response maps in one single pass without the need for back-propagation at
inference time.

###  3.3 Segmentation Network

At this stage we train a model with the pseudo labels generated in 3.2, and a
self-supervised refinement loss with Z-Stack image guidance which aligns the
edges of the segmentation with cells borders. Our proposed loss reads as

$\mathcal{L}_{S2}=\mathcal{L}_{class}+\lambda_{BF}\mathcal{L}_{BF}$ (3)

where $\mathcal{L}_{class}$ is a Cross Entropy loss against our pseudo labels,
$\lambda_{BF}$ is a weighting coefficient and $\mathcal{L}_{BF}$ is a BF
refinement loss which we introduce to improve upon the pseudo labels.

####  3.3.1 BF Refinement Loss

Blob detection methods are frequently used in efficient image processing
pipelines to detect regions with an appearance that differs from their
surrounding [12]. The area information they capture is usually retrieved with
a discretized version of the differentiable operator given by
$\Delta=\nabla\cdot\nabla=\partial_{x}^{2}+\partial_{y}^{2}+\partial_{z}^{2}$.
Since the calculation of second order derivatives is very sensitive to noise,
we robustify the calculation by applying it on a Gaussian-smoothed version of
our input.[19]

Figure 3: BF Boundary Loss. An Z-Stack BFM image is shown (1st from left)
together with its gradient image (2nd) and its LoG (3rd). The plot (right)
indicates the value distribution along the dashed line for all three images.
It can be seen that the LoG peaks at the cell boundary.

Inspired by the weighting mask for colour similarities based on image
gradients proposed in [20], we design a smooth loss term to align the
segmentation mask with the visible cell membrane from the image using the LoG
operator. Inspection of the image gradient along the cell boundary as depicted
in Fig. 3 indicates that the segmentation boundary coincides with with the
image Laplacian. To enforce accurate segmentation boundaries, we
asymmetrically penalize a disagreement between the image $I$ and its
segmentation mask $S$ by

$\mathcal{L}_{BF}=\nabla\text{G}\left(S,\sigma_{1}\right)\cdot\exp\left(-\left\|\Delta\text{G}\left(I,\sigma_{2}\right)\right\|_{p}^{p}\right),$
(4)

where G describes Gaussian smoothing with standard deviation $\sigma$ and
$\nabla$ and $\Delta=\nabla\cdot\nabla$ are the first and second order
differential operators.

###  3.4 Instance Label Assignment

With the cell centers likelihood map from 3.2 and the segmentation mask from
$S2$, we first merge the peaks in the likelihood map with low Euclidean
distance by a weighted mean over their scores and then run a watershed
algorithm to assign instance labels.

##  4 Experiments and Results

Our models were trained using a GeForce GTX 1060 GPU with a learning rate of
$5\times 10^{-5}$ and weight decay of $10^{-6}$ using the Adam optimizer and a
batch size of one due to the memory restrictions. We trained all models until
convergence with a maximum of 40 epochs.

###  4.1 BFM Z-Stack

In absence of a publicly available dataset for this use case, we collected a
BFM Z-Stack dataset of SF-9 insect cells (ThermoFisher, Germany) which we
roughly annotated with geometric cell centers. In order to decrease the
sedimentation speed of the cells, cells were suspended in feeding medium
supplemented with 20% Fetal Calf Serum (FCS). BFM images were obtained using a
Zeiss Axiovert 40 CFL microscope with 20x magnification.

The dataset consists of 120 instances of $64\times 256\times 256$ with
z-spacing of 2 $\mu m$ per slice. We crop the instances to 16 slices for
training and resize each plane to a $128\times 128$ image. The data is split
to a 4:1 train/validation split. The dataset will be made publicly available
upon acceptance.

Figure 4: Example image from BFM Z-Stack. Shown are four instance segmentation
results with their corresponding BFM image.

We trained our model with the Euclidean norm $p=2$, $\lambda_{BF}=1$, and
$\sigma_{1}=\sigma_{2}=3$ in eq.4. Without instance labels in this dataset, we
evaluate our results on BFM Z-Stack qualitatively. (Fig. 4). While we see
aligned segmentation and clear class separation on the first three results,
the last indicates the limitation of our weakly supervised method in the
presence of cell conglomerates where the segmentation mask provides cell
regions for connecting cells whose instances labels become fused.

###  4.2 Fluorescent Z-Stacks

In the absence of another independent BFM Z-Stack dataset with cell
annotations and to demonstrate the generalization capabilities of our
approach, we used images from BBBC050 [10, 21] which is a 3D image dataset of
early mouse embryos with nuclei fluorescently labeled with mRFP1 and imaged
with an IX71 microscope.

This allows to evaluate the results also quantitatively in comparison to the
recent cell segmentation approach presented by [10]. We generate the weak
labels we need in our pipeline as average of each segmentation mask. Following
their evaluation protocol, we report on the standard metrics for intersection
over union (IoU) for segmentation as well as SEG [22] and MUCov [23] for
instance labels. While SEG is an indicator of the absence of false-negative
instances, MUCov indicates the absence of false-positives.

Since BBBC050 is a fluorescent microscopy dataset its appearance is different
from BFM. Cell edges are not high response regions for LoG and the image
statistics vary in comparison to BFM. To cope with this noticeable difference
of the fluorescent images, we change our asymmetric boundary loss to penalize
on gradient alignments instead of the Laplacian. The loss then reads as:

$\mathcal{L}_{FL}=\nabla\text{G}\left(S,\sigma_{1}\right)\cdot\exp\left(-\left\|\nabla\text{G}\left(I,\sigma_{2}\right)\right\|_{p}^{p}\right).$
(5)

Again we choose $p=2$, $\lambda_{FL}=1$, and $\sigma_{1}=\sigma_{2}=3$.

There is an overview of this loss and the areas it is effecting on a
segmentation mask on an example image in Fig. 5 together with our predictions.

Figure 5: Example image from BBBC050 dataset. An Image from the fluorescent
Z-Stack is shown (1st from left) together with an evaluation of the Loss
(2nd), the Segmentation Mask (3rd), our Instance Prediction (4th) and the
Ground Truth (right).

We train our architecture with weak labels from the centroid of instances. To
assess the influence of the second network and the boundary loss, we ablate
our full pipeline and compare it against the recent method QCANet [10] and a
3D Mask R-CNN [24] as proposed in [10]. (Results in Table. 1)

Table 1: Comparison on BBBC050. Method | Annotation Level | IoU | SEG | MUCov
---|---|---|---|---
Fully supervised methods |  |  |  |
3D Mask R-CNN | Full Mask | 0.558 | 0.476 | 0.607
QCANet [10] | Full Mask | 0.746 | 0.710 | 0.721
Our training with weak annotations |  |  |  |
Our Pseudo Labels | Cell Centers | 0.506 | 0.353 | 0.399
BFS-Net (+ $\mathcal{L}_{class}$) | Pseudo Labels | 0.603 | 0.441 | 0.506
BFS-Net full pipeline (+ $\mathcal{L}_{FL}$) | Pseudo Labels | 0.643 | 0.530 | 0.473

While the performance of our network trained with weak labels results in a
$0.1$ lower IoU compared to the fully supervised method QCANet, we are able to
outperform the fully supervised 2-stage pipeline of 3D Mask R-CNN in terms of
IoU as reported by [10]. The result of training with weak labels reaches
$86.2\%$ IoU performance in comparison to the fully supervised counterpart
QCANet while requiring significantly fewer annotation effort. However shows a
larger gap on the instance metrics. We believe that this is due to the
refinement loss refining the segmentation quality while not specifically
targeting the instance labels. As expected, the performance of using pseudo
labels leads to the lowest performance. If we train $S2$ without BF-Refinement
Loss, the performance increases. We believe that this is due to the training
helping to remove false positives. When $S2$ is trained with BF-Refinement
Loss, the result gets improves further in terms of segmentation alignment due
to the Z-Stack guided refinements.

##  5 Discussion

We have proposed the first weakly supervised approach for cell instance
segmentation of BFM Z-Stack together with a new dataset. Rough center points
annotations were used to create pseudo labels which we refined using a novel
loss. We demonstrated the qualitative and quantitative performance of our
method on two datasets where we reach an IoU score of $86.2\%$ compared to the
state-of-the-art method QCANet despite significantly fewer label effort. We
believe that our dataset can pave the way to more methods based on weak labels
in the domain of Bright-Field Microscopy.

## References

  * [1] I. Shishkin, H. Markovich, Y. Roichman, and P. Ginzburg, “Auxiliary optomechanical tools for 3d cell manipulation,” Micromachines, vol. 11, no. 1, p. 90, 2020.
  * [2] A. Saxena, J. Schulte, A. Y. Ng, et al., “Depth estimation using monocular and stereo cues.,” in IJCAI, vol. 7, pp. 2197–2203, 2007.
  * [3] P. J. Goodhew and J. Humphreys, Electron microscopy and analysis.  CRC Press, 2000.
  * [4] J.-B. Lugagne, S. Jain, P. Ivanovitch, Z. B. Meriem, C. Vulin, C. Fracassi, G. Batt, and P. Hersen, “Identification of individual cells from z-stacks of bright-field microscopy images,” Scientific reports, vol. 8, no. 1, pp. 1–5, 2018.
  * [5] M. Subbarao and G. Surya, “Depth from defocus: A spatial domain approach,” International Journal of Computer Vision, vol. 13, no. 3, pp. 271–294, 1994\.
  * [6] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention, pp. 234–241, Springer, 2015\.
  * [7] Y. Al-Kofahi, A. Zaltsman, R. Graves, W. Marshall, and M. Rusu, “A deep learning-based algorithm for 2-d cell segmentation in microscopy images,” BMC bioinformatics, vol. 19, no. 1, pp. 1–11, 2018.
  * [8] F. Milletari, N. Navab, and S.-A. Ahmadi, “V-net: Fully convolutional neural networks for volumetric medical image segmentation,” in 2016 fourth international conference on 3D vision (3DV), pp. 565–571, IEEE, 2016.
  * [9] Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3d u-net: learning dense volumetric segmentation from sparse annotation,” in International conference on medical image computing and computer-assisted intervention, pp. 424–432, Springer, 2016.
  * [10] Y. Tokuoka, T. G. Yamada, D. Mashiko, Z. Ikeda, N. F. Hiroi, T. J. Kobayashi, K. Yamagata, and A. Funahashi, “3d convolutional neural networks-based segmentation to acquire quantitative criteria of the nucleus during mouse embryogenesis,” NPJ systems biology and applications, vol. 6, no. 1, pp. 1–12, 2020.
  * [11] H. Qu, P. Wu, Q. Huang, J. Yi, G. M. Riedlinger, S. De, and D. N. Metaxas, “Weakly supervised deep nuclei segmentation using points annotation in histopathology images,” in International Conference on Medical Imaging with Deep Learning, pp. 390–400, PMLR, 2019.
  * [12] I. Yoo, D. Yoo, and K. Paeng, “Pseudoedgenet: nuclei segmentation only with point annotations,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 731–739, Springer, 2019.
  * [13] Y. Zhou, Y. Zhu, Q. Ye, Q. Qiu, and J. Jiao, “Weakly supervised instance segmentation using class peak response,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3791–3800, 2018.
  * [14] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in Proceedings of the IEEE international conference on computer vision, pp. 618–626, 2017.
  * [15] K. Nishimura, R. Bise, et al., “Weakly supervised cell instance segmentation by propagating from detection response,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 649–657, Springer, 2019.
  * [16] M. Dong, D. Liu, Z. Xiong, X. Chen, Y. Zhang, Z.-J. Zha, G. Bi, and F. Wu, “Instance segmentation from volumetric biomedical images without voxel-wise labeling,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 83–91, Springer, 2019.
  * [17] Z. Zhao, L. Yang, H. Zheng, I. H. Guldner, S. Zhang, and D. Z. Chen, “Deep learning based instance segmentation in 3d biomedical images using weak annotation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 352–360, Springer, 2018.
  * [18] P. Kainz, M. Urschler, S. Schulter, P. Wohlhart, and V. Lepetit, “You should use regression to detect cells,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 276–283, Springer, 2015\.
  * [19] T. Lindeberg, “Image matching using generalized scale-space interest points,” Journal of mathematical Imaging and Vision, vol. 52, no. 1, pp. 3–36, 2015\.
  * [20] P. Heise, S. Klose, B. Jensen, and A. Knoll, “Pm-huber: Patchmatch with huber regularization for stereo matching,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 2360–2367, 2013.
  * [21] V. Ljosa, K. L. Sokolnicki, and A. E. Carpenter, “Annotated high-throughput microscopy image sets for validation.,” Nature methods, vol. 9, no. 7, pp. 637–637, 2012.
  * [22] M. Maška, V. Ulman, D. Svoboda, P. Matula, P. Matula, C. Ederra, A. Urbiola, T. España, S. Venkatesan, D. M. Balak, et al., “A benchmark for comparison of cell tracking algorithms,” Bioinformatics, vol. 30, no. 11, pp. 1609–1617, 2014.
  * [23] N. Silberman, D. Sontag, and R. Fergus, “Instance segmentation of indoor scenes using a coverage loss,” in European Conference on Computer Vision, pp. 616–631, Springer, 2014.
  * [24] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in Proceedings of the IEEE international conference on computer vision, pp. 2961–2969, 2017.