# DETCID: Detection of Elongated Touching Cells with
Inhomogeneous Illumination using a Deep Adversarial Network

Ali Memariani and Ioannis A. Kakadiaris  Member, IEEE This work was supported
in part by the Hugh Roy and Lillie Cranz Cullen Endowment Fund. Ali Memariani
and Ioannis A. Kakadiaris are affiliated with the Computational Biomedicine
Lab, Department of Computer Science, University of Houston, Houston TX
(e-mail: {amemaria, ikakadia}@central.uh.edu). We would like to thank Dr.
Bradley T. Endress and Dr. Eugenie Basseres for correcting the primary set of
annotations and Dr. Jahangir Alam for providing the second set of annotations.

###### Abstract

Clostridioides difficile infection (C. diff) is the most common cause of death
due to secondary infection in hospital patients in the United States.
Detection of C. diff cells in scanning electron microscopy (SEM) images is an
important task to quantify the efficacy of the under-development treatments.
However, detecting C. diff cells in SEM images is a challenging problem due to
the presence of inhomogeneous illumination and occlusion. An Illumination
normalization pre-processing step destroys the texture and adds noise to the
image. Furthermore, cells are often clustered together resulting in touching
cells and occlusion. In this paper, DETCID, a deep cell detection method using
adversarial training, specifically robust to inhomogeneous illumination and
occlusion, is proposed. An adversarial network is developed to provide region
proposals and pass the proposals to a feature extraction network. Furthermore,
a modified IoU metric is developed to allow the detection of touching cells in
various orientations. The results indicate that DETCID outperforms the state-
of-the-art in detection of touching cells in SEM images by at least 20 percent
improvement of mean average precision.

Clostridioides difficile infection (C. diff), Cell detection, Cell
segmentation, Deep adversarial training, Illumination.

##  1 Introduction

Clostridioides difficile infection (C. diff) is the most common cause of death
due to infectious gastroenteritis in the USA and a significant source of
morbidity [1]. Phenotypic features (e.g., length, shape deformation) of C.
diff cells in scanning electron microscopy (SEM) images indicate critical
information about cell health in C. diff research studies. Therefore,
instance-level segmentation of the C. diff cells is an important task to
extract phenotypic information [2]. However, analysis of SEM images is
challenging due to inhomogeneous illumination and presence of touching cells.
Figure 1 depicts samples of C. diff cells in SEM images. In computer vision,
the terms detection, semantic segmentation, and instance-level segmentation
are defined differently. Nevertheless, in microscopy image analysis, these
terms are often used interchangeably referring to the latter where the task is
to provide a mask for every cell instance in the image.

Classical illumination normalization techniques could reduce the effect of
illumination as a pre-processing step. However, they destroy the texture and
add noise to the image [3, 4]. Furthermore, SEM images are limited in number
and expensive to obtain. Deep learning methods have achieved state-of-the-art
performance in many computer vision tasks such as object detection and
segmentation [5]. However, deep methods require a large number of training
samples. Therefore, preparing a large training set that includes all possible
variation of cells is an important task in cell segmentation. Exiting data
augmentation methods (e.g., flipping, rotation, and warping the entire image)
are limited in creating diverse training samples. In current practice,
increasing the training samples by an order of magnitude results in a highly
correlated training set.

|
---|---
(a) | (b)
|
(c) | (d)
Figure 1: Depiction of samples of C. diff cells with challenges of
Inhomogeneous illumination and touching cells. Inhomogeneous illumination
makes the segmentation of scanning electron microscopy images challenging: (a)
shadows on the cell body, (b) shadows in the periphery, (c) touching cells
with overlapping cell walls, and (d) touching cells with occluded cell bodies.
Figure 2: Depiction of DETCID Pipeline: (a) An adversarial region proposal
network (ARPN) selects the cell region to be classified, (b) an FEN extracts
the features from cell candidate regions, (c) detected region of interests
(ROI) are aligned using bilinear interpolation, and (d) a network head is
applied to produce the final mask.

In this paper, we present DETCID (Detection of Elongated Touching C. diff
Cells in the presence of Inhomogeneous illumInation using a Deep adversarial
network) a deep cell detection algorithm to detect C. diff cells in SEM
images. DETCID models the inhomogeneous illumination as an adversarial attack.
Our previous work, SoLid [6] demonstrated that adversarial training could
improve the semantic segmentation performance of U-net in presence of
inhomogeneous illumination by 44% where the adversarial loss penalizes the
segmentation output to be similar to the ground truth without increasing the
complexity of the network during deployment. DETCID improves the state-of-the-
art with the following contributions:

  1. 1.

Developed an adversarial region proposal network to provide candidate bounding
boxes for cell detection, reducing the effect of inhomogeneous illumination
(Section III.A)

  2. 2.

Developed a novel non-max suppression method based on mask overlaps to detect
touching cells in various orientations (Section III.B)

  3. 3.

Developed an image synthesis algorithm to generate a large training set of
microscopy images for training deep networks (Section III.C).

DETCID expands the semantic segmentation method in SoLiD to an instance-level
segmentation method where several bounding boxes are selected to extract
features relevant features from a deep feature pyramid network such as
ResNet50. Furthermore, the mask-based non-max suppression method detects the
clusters of touching cells in various orientations. An image synthesis
algorithm is developed to generate clustered cell images to train the network.

The rest of the paper is organized as follows: Section II presents a review of
the literature related to instance-level cell segmentation. Section III
describes DETCID algorithm. Section IV presents the experimental results
comparing the segmentation performance of DETCID with the state-of-the-art.
Finally, Section V draws the conclusions.

##  2 Related Work

This section categorizes the related work on cell detection into shallow and
deep detection methods. Then, presents the findings from a review of the
literature on deep segmentation methods addressing inhomogeneous illumination.

###  2.1 Shallow cell detection methods

Shallow methods select cell candidate regions using intensity thresholding or
energy minimization. Then, an optimization algorithm is applied [7, 8], a
machine learning regressor is trained over a set of hand-crafted features to
select the cell from a set of candidates [9, 10, 11, 12], or a level set
segmentation is used [13, 14]. Finally, size filtering or hole filling is
applied to refine the results. To address the inhomogeneous illumination, pre-
processing steps are proposed. However, these pre-processing steps may remove
texture information, making the detection more challenging [15]. Ulman et. al.
[16] performed an objective comparison of many shallow cell detection
algorithms with deep convolutional networks. Shallow methods do not require
expensive computing hardware to train and are interpretable. However, they are
outperformed by deep learning methods.

###  2.2 Deep cell detection methods

Deep learning algorithms have outperformed the state of the art in many
biomedical image processing tasks [5]. Shi et. al. [17] applied a cascade of
Quaternion Grassmann average layers to develop an unsupervised deep network
for segmentation of histology cells. Others have applied deep auto-encoders
for cell segmentation [18]. Shen et. al. [5] reviewed many unsupervised, or
CNN-based approaches combined with hand-crafted features for segmentation of
biomedical images. Roopa et. al. [19] trained a CNN with hand-craft features
as input to classify white blood cells in peripheral blood smear images. Hand-
crafted cell nuclei boundary masks are also used as shape prior to filter the
detection of CNNs [20]. Others applied CNNs for cell detection with pixel-
level classification for each patch in the images[21, 22, 23]. Hofener et. al.
[24] applied post-processing to smooth the scores derived by CNNs to improve
the cell nuclei detection in histology images. However, patch-based approaches
need to run the network for every patch resulting in redundant computations.

To reduce the computations by sharing the computations over the overlapping
patches, fully convolutional networks were introduced for image segmentation
[25]. Specifically, U-net is widely used for biomedical image segmentation
tasks [26]. Xie et. al. [27] evaluated the performance of U-net on multiple
pathology datasets. Ramesh et. al. [28] added an unsupervised pre-processing
layer with logistic sigmoid functions to U-net to separate clustered image
patches from each other. However, U-net is sensitive to inhomogeneous
illumination which increases the false positive for segmentation of SEM images
[6]. Xie et. al. [22] applied two U-shape network architecture without skip
connections to compute cell spatial density maps. Spatial densities are
applied to detect cell overlaps. However, the application is limited to round
shape objects only [22, 29].

Gu et. al.  [30] proposed a pyramid of residual blocks to capture spatial
information in multiple resolutions to detect histology cells in various
sizes. Li et. al. [31] replaced the head layers in Mask R-CNN with a
conditional random field (CRF) to impose smoothness on the boundaries of
segmented patches.

###  2.3 Deep methods addressing inhomogeneous illumination

Deep networks, such as U-net, are sensitive to inhomogeneous illumination. Wan
et. al. [32] proposed an iterative process where a U-net is applied to provide
a preliminary segmentation followed by a convolution layer to estimate the
bias field in magnetic resonance (MR) images. Next, the bias field corrected
image is again sent to the U-net for the next iteration, improving the
segmentation. However, the iterative process involves many passes through the
network, increasing the complexity of the method.

Generative adversarial networks (GANs) have been used to add robustness to
adversarial attacks to the deep networks [33]. Adversarial training can
improve image segmentation by producing label maps that are similar to a
target image [34]. Adversarial networks have been applied in the segmentation
of MR images [35, 36, 37] where the datasets and the annotations are
available. However, the application is limited since the adversarial training
requires a large training set to train both the segmenter and the
discriminator networks. Lee et. al. [38] proposed an unsupervised image
deconvolution method using a cycle-consistent adversarial network to improve
the quality of blurred and noisy fluorescence microscopy images without
labeled data. The adversarial network in DETCID models the inllumination as an
adversarial attack without during training without increasing the complexity
of the network during deployment.

##  3 Methods

DETCID compromises of two parts: a deep adversarial region proposal network
(ARPN) and a feature extraction network (FEN). Figure 2 depicts the overview
of the pipeline. The input to ARPN is a cell image and the output is a label
map. The FEN is fully convolutional and produces a probability map for
presence of cells in the image. An RoI alignment layer combines the output of
the two network and aligns the extracted features with the input. RoIs are
passed through two convolution layers to produce the final segmentation mask.

###  3.1 Adversarial region proposals network

The ARPN consists of two deep ConvNets, namely the segmenter and the
discriminator. The segmenter predicts a label map for the pixels while the
discriminator distinguishes between the predicted label maps and the ground
truth.

The input to the segmenter is a cell image. The segmenter includes a
convolution path and a deconvolution path similar to U-net. The convolution
path extracts a feature map for segmentation using convolution layers while
the deconvolution path increases the resolution, creating a label map. The
generated label map may differ significantly from the ground truth since the
segmenter does not consider the smoothness of the labels, resulting in a non-
continuous segmentation.

A second ConvNet (discriminator) is used to train the segmenter to produce
label maps similar to the ground truth. The discriminator is a regular ConvNet
classifier trained on the ground truth and predicted segmentations. During
training, it learns to classify the input image into two classes:
“artificially generated” or “ground truth”, and backpropagates the gradients.

####  3.1.1 Segmenter Network

The segmenter network consists of six convolutional units: the first three
units include a 3x3 convolution layer, a ReLU layer and a 2x2 max pooling
layer with a stride of two downsampling the image (contracting units). The
next three units (expanding units) include an upsampling of the features
followed by a 2x2 deconvolution. Each contracting unit doubles the number of
feature channels while each expanding unit halves the number of channels. The
segmenter minimizes a loss function $L_{S}$:

$L_{S}=\mathbf{w_{c}}*L_{C}\Big{(}\mathcal{S}(\mathbf{I}),\mathbf{G}\Big{)}+L_{C}\Big{(}\mathcal{D}\big{(}\mathcal{S}(\mathbf{I})\big{)},1\Big{)},$
(1)

where $L_{C}\Big{(}\mathcal{S}(\mathbf{I}),\mathbf{G}\Big{)}$ is a cross-
entropy term between the predicted labels $S$ corresponding to the image
$\mathbf{I}$ and the ground truth $\mathbf{G}$. The second term
$L_{C}\Big{(}\mathcal{D}\big{(}\mathcal{S}(\mathbf{I})\big{)},1\Big{)}$ is the
adversarial loss term, computed by the discriminator. The label map of image
$\mathbf{I}$ generated by the segmenter is denoted by
$\mathcal{S}(\mathbf{I})$ and $\mathcal{D}$ is the discriminator network
described in the next section. The adversarial loss forces the segmenter to
produce label maps that would be considered as ground truth by the
discriminator. To distinguish touching cells, the segmenter considers the
boundaries of the cells (cell wall) as a separate class. Hence, the segmenter
loss is one-hot encoded with three classes. The number of cell wall samples
are considerably less compared to the other classes. To compensate for the
bias in the training set, the segmenter cross-entropy loss is weighted
($\mathbf{w_{c}}$). The minority class receives a higher classification
weight.

The segmenter network may misclassify a large portion of cells due to
inhomogeneous illumination. The adversarial training is used to evaluate such
misclassifications and improve the segmenter. A discriminator ConvNet is used
to compute the likelihood of the predicted segmentation map being an actual
label map.

####  3.1.2 Discriminator Network

The discriminator improves the generated labels by sending feedback to the
generator if the segmentation labels are significantly different from the
ground truth. It does not increase the complexity of the network since it is
used only during training. It consists of five convolutional layers with valid
padding, followed by ReLU activations and average pooling. Furthermore, two
fully connected layers are placed at the end of the discriminator.

To avoid saturation, the last layer of the discriminator does not have a
thresholding operator so it produces an unscaled output. Computing scores
between 0 and 1 may cause the discriminator to generate values close to 0 for
generated label maps, in which case the gradient would be too small to update
the generator and eventually saturate the network [33].

The discriminator ($\mathcal{D}$) computes the cross-entropy of the ground
truth label maps ($\mathbf{G}$) and 1, and the cross-entropy of the generated
label maps ($S(\mathbf{I})$) and 0, minimizing the following loss function:

$L_{D}=L_{C}\Big{(}\mathcal{D}(\mathbf{G}),1\Big{)}+L_{C}\Big{(}\mathcal{D}(\mathcal{S}(\mathbf{I})),0\Big{)}.$
(2)

During the training, the discriminator improves the segmenter network,
penalizing the segmentation labels that do not look like manual labels.
Therefore, the adversarial result has properties such as smoothness and
robustness to inhomogeneous illumination.

|  |
---|---|---
|  |
Figure 3:  (T) Depiction of selected the synthesized SEM images of overlapping
C. diff cells with inhomogeneous illumination and (B) with the presence of
debris.

###  3.2 Instance-level cell segmentation

A ResNet50 architecture [39] is applied to extract features from images. The
computation is shared for all cell bounding boxes for efficiency. Generating a
large number of bounding box proposals is one of the major drawbacks of
algorithms such as MaskRCNN [40] and Faster R-CNN [41]. Even with feature
sharing using ResNet, region based methods computational complexity is not
comparable with one shot detection algorithms such as YOLO [42]. Unlike Mask
R-CNN, DETCID uses the APRN to generate the proposal bounding boxes to reduce
the number of the proposals. An average pooling is applied on the result of
the ARPN for each anchor type (i.e., horizontal, vertical, and square box). If
the average value is greater than a threshold t, then anchor boxes are
centered at that location. The values of the threshold is determine by the
size of the smallest cell in the training set.

Manual annotation assigns a single bounding box to a cell. However, small
variations of the bounding box in length and height may also get a high
classification score by the detection method due to smoothness property.
Setting all such variations to the background will be confusing for the
network. Furthermore, the region proposal network may not always propose the
finest bounding box around the object. Therefore, for each proposed region,
similar bounding boxes with variations in length and height are passed to a
fully-connected layer for refinement.

ResNet features are extracted for each anchor box and are fed to the ROI
alignment. A network head is applied similar to head in Mask-RCNN to compute
the masks, bounding boxes, and detection probabilities. The cells may appear
in various orientations. Therefore, applying non-max suppression based on the
bounding box overlaps is not suitable for detection of cells in SEM images.
DETCID modifies the computation of intersection over union (IoU) based on the
masks overlaps and the area of the cell masks. Non-max suppression is then
applied on the modified IoU values.

The loss function includes a classification term to detect the cells and a
regression term to identify the bounding box:

$L_{F}=L_{C}(p,p^{*})+p^{*}L_{R}(\mathbf{r},\mathbf{r^{*}})+L_{C}(\mathbf{M},\mathbf{G}),$
(3)

where, $p$ and $p^{*}$ denote the classification score and label respectively,
$r,r^{*}$ denote the prediction and annotated bounding box parameters, and
$\mathbf{M}$ and $\mathbf{G}$ are the predicted and the ground truth masks
respectively labeling the pixels as background or cell candidate.

Table 1: Definition of notations used in Algorithm 1.  Notation | Type | Definition
---|---|---
$\phi$ | Scalar | Angle between the major axes of a crossing pair of cells (degrees)
$\theta$ | Scalar | Angle of the major axis of the cell (degrees)
$(x,y)$ | Scalar | Centroid of a cell (pixels)
$e,f$ | Scalar | Constants
$\eta$ | Scalar | Height of a cell mask (pixels)
$\sigma$ | Scalar | Horizontal shear parameter
$(\rho,\kappa)$ | Scalar | Image size for acquired and synthetic images (pixels)
$\delta$ | Scalar | Maximum change of the random orientation of a cell (degrees)
$\chi$ | Scalar | Maximum horizontal shift of the second cell in a touching/crossing pair (pixels)
$\psi$ | Scalar | Maximum vertical shift of the second cell in a touching/crossing pair (pixels)
$a$ | Scalar | Number of acquired images
$n$ | Scalar | Number of cells in an acquired image
$c$ | Scalar | Number of crossing pairs of cells in a synthetic image
$o$ | Scalar | Number of isolated cells in a synthetic image
$t$ | Scalar | Number of touching pairs of cells in a synthetic image
$z$ | Scalar | Number of synthetic images
$\epsilon$ | Scalar | Random number in the range [0,1]
$\omega$ | Scalar | Width of a cell mask (pixels)
$w$ | Scalar | Window size for texture synthesis (same width and height) (pixels)
$\mathbf{A}^{k}$ | 2D tensor | Acquired image k
$\mathbf{B}^{k}_{i}$ | 2D tensor | Annotation mask for image k and cell i with size $(\rho,\kappa)$
$\mathbf{J}^{l}_{j}$ | 2D tensor | Synthetic ground truth mask for image l and cell j with size $(\rho,\kappa)$ (pixels)
$\mathbf{C}$ | 2D tensor | Image of a cell with size $(\rho,\kappa)$ (pixels)
$\mathbf{C}^{m}$ | 2D tensor | Binary mask of a cell with size $(\rho,\kappa)$ (pixels)
$\mathbf{I}^{l}$ | 2D tensor | The lth synthetic image
T | 2D tensor | 3$\times$3 geometric transformation matrix
$\mathbf{B}^{k}$ | 3D tensor | Annotation mask for acquired image k with size $(n,\rho,\kappa)$ (pixels)
$\mathbf{J}^{l}$ | 3D tensor | Synthetic ground truth mask for image l with size $(o+2t+2c,\rho,\kappa)$ (pixels)
$\boldsymbol{\mathscr{A}}^{m}$ | Set | Set of manually annotated masks
$\boldsymbol{\mathscr{A}}^{r}$ | Set | Set of acquired images
$\boldsymbol{\mathscr{I}}^{g}$ | Set | Set of synthetic ground truth masks
$\boldsymbol{\mathscr{I}}^{s}$ | Set | Set of synthetic cell images

1

2

Input : A set of $a$ acquired images
$\boldsymbol{\mathscr{A}}^{r}=\\{\mathbf{A}^{1},...,\mathbf{A}^{a}\\}$ and
their manually annotated masks
$\boldsymbol{\mathscr{A}}^{m}=\\{\mathbf{B}^{1},...,\mathbf{B}^{a}\\}$, number
of images to be synthesized $z$, window size $m$

Output : Synthetic cell images
$\boldsymbol{\mathscr{I}}=\\{\mathbf{I}^{1},...,\mathbf{I}^{z}\\}$, synthetic
ground truth masks
$\boldsymbol{\mathscr{J}}=\\{\mathbf{J}^{1},...,\mathbf{I}^{z}\\}$

3

4 Function  _AddCell(_$\mathbf{A}^{k}$ , $\mathbf{B}^{k}_{i}$,
$\mathbf{I}^{l}$, $\mathbf{J}^{l}_{j}$, $\theta$, $x$, $y$_)_:

  * 5•

Extract the $i$th cell in the $k$th acquired image $\mathbf{A}^{k}$ using its
annotated mask $\mathbf{B}^{k}_{i}$:
${\mathbf{C}\leftarrow\mathbf{A}^{k}\circ\mathbf{B}^{k}_{i}}$,
${\mathbf{C}^{m}\leftarrow\mathbf{B}^{k}_{i}}$;

* 6•

Translate the center of the annotation mask of the cell in $\mathbf{C}$ and
its mask $\mathbf{C}^{m}$ to the center of the image;

* •

Initialize a 3$\times$3 geometric transformation matrix T as an identity
matrix and add a random noise to the transformation
parameters (e.g., horizontal shear: $\sigma=f+e\epsilon$); 7• Warp the masked
cell image $\mathbf{C}$ and its annotation mask $\mathbf{C}^{m}$ with
transformation T and resize to size $(\rho,\kappa)$;

* 8•

Compute the angle $\phi$ of the major axis of the cell corresponding to the
horizontal axis;

* •

Rotate $\mathbf{C}$ and $\mathbf{C}^{m}$ to align with orientation $\theta$
and crop to the size $(\rho,\kappa)$; • Translate the object centroid in
$\mathbf{C}$ and $\mathbf{C}^{m}$ to location $(x,y)$ (move the object inside
if the cell is partially outside the image boundaries); 9• Overlay
$\mathbf{C}$ on $\mathbf{I}^{l}$ and
$\mathbf{J}^{l}_{j}\leftarrow\mathbf{C}^{m}$;

10 return $\mathbf{I}^{l}$, $\mathbf{J}^{l}_{j}$;

11

12

13

14begin

15 for  _$l=1,...,z$_ do

  * 16•

Randomly select an image $\mathbf{A}^{k}$ from the acquired images
$\boldsymbol{\mathscr{A}}^{r}$ and its annotation mask $\mathbf{B}^{k}$ from
$\boldsymbol{\mathscr{A}}^{m}$;

* 17•

Apply the image inpainting algorithm [43] to remove the cells from the image
and store the background to $\mathbf{I}^{l}$;

* 18•

Randomly select a background patch of size $w\times w$ from image
$\mathbf{I}^{l}$;

* 19•

Synthesize a background image with the same resolution as $\mathbf{I}^{l}$
using the texture synthesis algorithm [44] and
replace with $\mathbf{I}^{l}$;

* 20•

Randomly select the number of isolated cells $o$, touching pairs $t$, and
crossing pairs $c$ to be placed into
the image ($o,t,c\in\\{1,...,n\\}$);

* 21•

Initialize $\mathbf{J}^{l}$ with zeros as a 3D tensor of size
($o+2t+2c,\rho,\kappa$)

22 for  _$p=1,...,o$_ do

  * 23•

Randomly select a cell mask $i$ ($i\in{1,...,n}$) with annotation tensor
$\mathbf{B}^{k}_{i}$ in the acquired image $\mathbf{A}^{k}$;

* 24•

Randomly generate $\theta\in[0^{\circ},180^{\circ}]$,
$x_{1}\in[\frac{\omega}{2},...,\rho-\frac{\omega}{2}]$,
$y_{1}\in[\frac{\eta}{2},...,\kappa-\frac{\eta}{2}]$;

* 25•

Add the cell into the image: $\mathbf{I}^{l}$, $\mathbf{J}^{l}_{j}\leftarrow$
AddCell(_$\mathbf{A}^{k}$ , $\mathbf{B}^{k}_{i}$, $\mathbf{I}^{l}$,
$\mathbf{J}^{l}_{j}$, $\theta$, $x_{1}$, $y_{1}$_);

26

27 end for

28 for  _$q=1,...,t$_ do

  * 29•

Randomly select a cell mask $i$ ($i\in{1,...,n}$) with annotation tensor
$\mathbf{B}^{k}_{i}$ in the acquired image $\mathbf{A}^{k}$;

* 30•

Randomly generate $\theta\in[0^{\circ},180^{\circ}]$,
$x_{1}\in[\frac{\omega}{2},...,\rho-\frac{\omega}{2}]$,
$y_{1}\in[\frac{\eta}{2},...,\kappa-\frac{\eta}{2}]$;

* 31•

Add the first touching pair into the image the first output: $\mathbf{I}^{l}$,
$\mathbf{J}^{l}_{j}$ $\leftarrow$ AddCell(_$\mathbf{A}^{k}$ ,
$\mathbf{B}^{k}_{i}$, $\mathbf{I}^{l}$, $\mathbf{J}^{l}_{j}$, $\theta$,
$x_{1}$, $y_{1}$_);

* 32•

Update $(x_{1},y_{1})\leftarrow$ centroid of $\mathbf{J}^{l}_{j}$;

* 33•

Randomly select a cell mask $u$ with annotation tensor $\mathbf{B}^{k}_{u}$ in
the acquired image $\mathbf{A}^{k}$ as the second cell;

* 34•

Let $\omega$ be the width and $(x_{1},y_{1})$ the centroid of the cell mask
$\mathbf{C}^{m}$. Randomly select a location $(x_{2},y_{2})$:
$x_{2}\in\left[x_{1}-\omega,x_{1}-\frac{\omega}{2}\right]\cup\left[x_{1}+\frac{\omega}{2},x_{1}+\omega\right]$
and $y_{2}\in\left[y_{1}-\psi\epsilon,y_{1}+\psi\epsilon\right]$;

* 35•

Add the second touching pair into the image: $\mathbf{I}^{l}$,
$\mathbf{J}^{l}_{v}$ $\leftarrow$ AddCell(_$\mathbf{A}^{k}$ ,
$\mathbf{B}^{k}_{u}$, $\mathbf{I}^{l}$, $\mathbf{J}^{l}_{v}$, $\theta$,
$x_{2}$, $y_{2}$_);

36

37 end for

38 for  _$r=1,...,c$_ do

  * 39•

Randomly select a cell mask $i$ ($i\in{1,...,n}$) with annotation tensor
$\mathbf{B}^{k}_{i}$ in the acquired image $\mathbf{A}^{k}$;

* 40•

Randomly generate $\theta\in[0^{\circ},180^{\circ}]$,
$x_{1}\in[\frac{\omega}{2},...,\rho-\frac{\omega}{2}]$,
$y_{1}\in[\frac{\eta}{2},...,\kappa-\frac{\eta}{2}]$;

* 41•

Add the first crossing pair into the image: $\mathbf{I}^{l}$,
$\mathbf{J}^{l}_{j}$ $\leftarrow$ AddCell(_$\mathbf{A}^{k}$ ,
$\mathbf{B}^{k}_{i}$, $\mathbf{I}^{l}$, $\mathbf{J}^{l}_{j}$, $\theta$,
$x_{1}$, $y_{1}$_);

* 42•

Update $(x_{1},y_{1})\leftarrow$ centroid of $\mathbf{J}^{l}_{j}$;

* 43•

Randomly select a cell mask $u$ with annotation tensor $\mathbf{B}^{k}_{u}$ in
the acquired image $\mathbf{A}^{k}$ as the second cell;

* 44•

Let $\eta$ be the height and $(x_{1},y_{1})$ the centroid of the cell mask
$\mathbf{C}^{m}$. Randomly select a location $(x_{2},y_{2})$:
$y_{2}\in\left[y_{1}-\eta,y_{1}-\frac{\eta}{2}\right]\cup\left[y_{1}+\frac{\eta}{2},y_{1}+\eta\right]$
and $x_{2}\in\left[x_{1}-\chi\epsilon,x_{1}+\chi\epsilon\right]$;

* •

Randomly select an angle $\phi$: $\phi=\theta+90\pm\delta\epsilon$ 45• Add the
second crossing pair into the image: $\mathbf{I}^{l}$, $\mathbf{J}^{l}_{v}$
$\leftarrow$ AddCell(_$\mathbf{A}^{k}$ , $\mathbf{B}^{k}_{u}$,
$\mathbf{I}^{l}$, $\mathbf{J}^{l}_{v}$, $\phi$, $x_{2}$, $y_{2}$_);

46

47 end for

48

49 end for

50

51 end

52

Algorithm 1 Image synthesis algorithm generating images with isolated,
touching, and crossing cells. Notations are defined in Table 1

###  3.3 Data Augmentation

Deep networks require large numbers of training data. Specifically, deep
learning architectures proposed for biomedical image analysis are hindered by
the lack of large amounts of training data. Therefore, generating synthetic
images becomes important in the analysis of biomedical images, since their
acquisition is expensive and time-consuming. A simple solution would be to
avoid training by applying a pre-trained model. However, the solution is
limited to cases where a pre-trained network with similar training data exits
[5]. To train a deep segmentation model, Algorithm 1 is developed which is
capable of synthesizing cell images with inhomogeneous illumination where
cells could be isolated, touching, or crossing.

Deep learning frameworks such as TensorFlow provide simple image augmentation
functions such as translation, rotation, cropping, flipping, and scaling [45].
These basic augmentation techniques often create black areas in the image
which could be filled with interpolation or more complex image in-painting
techniques. The basic augmentation methods mentioned above have been
previously used to increase the number of training samples and add invariance
to challenges such as rotation of objects in the image [46, 35, 34, 36, 5, 47,
37].

Increasing the amount of data by orders of magnitude is essential in training
deep models to analyze SEM images. However, using basic augmentation methods
to increase the amount of data by orders of magnitude results in high
correlation between the images in the training dataset. U-net applied image
warping to the cell images, creating images with slightly different cell and
backgrounds [26]. Nevertheless, warping the whole image with the same warping
transformation would limit the number of synthesized images.

Algorithm 1 presents an image synthesis algorithm capable of synthesizing
large numbers of images with the same background texture and cell shapes in
the images captured by SEM. First, SEM-acquired images are manually annotated
to three classes, namely cell body, cell wall, and background. The image
quilting technique by Efros and Freeman is applied to synthesize similar
background images [44]. Then, the cells are randomly warped and placed into
the image. Warping the cells ensures that the training data are different from
the testing data. Figure 3 depicts samples of synthesized isolated images of
isolated, touching, and crossing cells in presence of inhomogeneous
illumination and debris.

###  3.4 Implementation details

The implementation of the adversarial region proposal network is based on the
previous work by the authors [6] and random patches of size 256$\times$256 are
passed to the ResNet50 [39] for training. The ResNet50 architrave applies
scale anchors of size $\\{32^{2},64^{2},128^{2}\\}$ to extract features and is
initialized with COCO pretrained weights. To allow finer ROIs, five percent
variations in length and height of the bounding boxes are considered and
bounding boxes with less than five percent of potential cell areas are
filtered before pooling ROIs. The above thresholds are found empirically. Adam
optimization is applied to optimize the overall loss function on a cluster
with 4 NVIDIA GeForce GTX GPU of 12 GB capacity.

|  |  |  |
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
(a) | (b) | (c) | (d) | (e)
Figure 4: Depiction of the segmentation results: (a) original image, (b)
ground truth labels, (c) Mask-RCNN, (d) FCRN, and (e) DETCID segmentation.
Mask-RCNN is more accurate in detecting isolated cells. However, Mask-RCNN
does not detect cells in presence of debris or cells clusters. FCRN is
sensitive to inhomogeneous illumination and presence of debris and results in
false positives. DETCID is able to detect cells when touching cells are
clustered together.

##  4 Results and discussion

To evaluate DETCID for cell detection and segmentation, an acquired SEM
dataset UH-A-cdiff1 [15], and a synthetic dataset UH-S-cdiff1 were used. Mean
average precision (mAP) and dice score metrics were used to evaluate the
performance of detection and segmentation, respectively.

###  4.1 Datasets

####  4.1.1 UH-A-cdiff1

UH-A-cdiff1 consists of 22 C. diff cell images (197 vegetative cells and 111
spores) acquired via scanning electron microscopy. Image dimensions are
411$\times$711 pixels with 10,000x magnification. Moreover, many cells are
touching or crossing each other with the existence of debris. Also, the cells
were partially deformed and cell walls are damaged due to a laboratory
treatment, making the detection challenging. Two set of annotations were
provided labeling every cell as a binary mask for every cell in the image.

####  4.1.2 UH-S-cdiff1

According to ImageNet [48] statistics, the rule of thumb for size of training
set is 1,000 samples. Algorithm 1 is applied to synthesize 6,000 images of
411$\times$711 pixel dimensions, considering at least 1,000 samples for
isolated, touching, and crossing cases of vegetative cells and spores. The
number of cases per image is chosen empirically.

The synthetic images include two to four pairs of the following scenarios: a
pair of two vegetative cells touching, a pair of two vegetative cells
crossing, and a vegetative cell touching a spore. Moreover, two to four single
isolated cells of each type is added into the image, creating a variety of
possible overlaps between cells in various orientations.

Table 2: Quantitative results between the performance of DETCID and the state-of-the-art in cell detection by Mask-RCNN and FCRN on the acquired (UH-A-cdiff1) and the synthetic (UH-S-cdiff1) images. A 10-fold cross validation is performed on the synthetic dataset and the result is reported with 95% CI.  Dataset | Method | Vegetative Cell | Spore | Overall
---|---|---|---|---
mAP | Dice | mAP | Dice | mAP | Dice
UH-A-cdiff1 | Mask RCNN | 0.52 | 0.88 | 0.69 | 0.88 | 0.54 | 0.88
FCRN | 0.41 | 0.60 | 0.13 | 0.46 | 0.23 | 0.60
DETCID | 0.65 | 0.85 | 0.65 | 0.87 | 0.65 | 0.85
UH-S-cdiff1 | Mask RCNN | 0.52 $\pm$ 0.03 | 0.86 $\pm$ 0.01 | 0.45 $\pm$ 0.05 |  0.90 $\pm$ 0.01 | 0.49 $\pm$ 0.02 | 0.83 $\pm$ 0.01
FCRN | 0.36 $\pm$ 0.1 | 0.62 $\pm$ 0.03 | 0.46 $\pm$ 0.11 | 0.62 $\pm$ 0.03 | 0.25 $\pm$ 0.05 | 0.60 $\pm$ 0.03
DETCID | 0.66 $\pm$ 0.02 | 0.88 $\pm$ 0.01 | 0.69 $\pm$ 0.02 | 0.89 $\pm$ 0.01 | 0.67 $\pm$ 0.01 | 0.88 $\pm$ 0.01

###  4.2 Baseline comparisons

The results were compared with two state-of-the-art methods in cell detection
and segmentation:

####  4.2.1 Mask-RCNN

Mask-RCNN [40] was developed by Facebook AI Research (FAIR) Lab as an
instance-based object segmentation method capable of providing mask and
bounding box for every object in the image. ResNet50 is used as the backbone
with COCO pre-trained weights to be consistent with the backbone used for
DETCID. A fully-connected region proposal network predicts rectangular
bounding box regions. The regions of interest were resized and passed to the
network head to compute the mask and refine the bounding box. We selected
Mask-RCNN as a baseline since Mask-RCNN has achieved the state-of-the-art in
instance-segmentation of cell nuclei in microscopy images [49] and
segmentation of epithelial cells in pathology images with overlapping cells
[31] using a TensorFlow implementation provided by Matterport which is used
for comparison with DETCID [50] and is initialized with COCO pretrained
weights.

|
---|---
(a) | (b)
|
(c) | (d)
Figure 5: Depiction of Bland-Altman plots, comparing DETCID performance with
the performance of the secondary set of human expert annotations: (a) the
agreement between the two set of annotations on the number of annotated cells
in image, (b) the agreement between the primary set of annotations and DETCID
on the number of annotated cells in image, (c) the agreement between the two
set of annotations on the annotated masks, and (d) the agreement between the
annotated in the primary set of annotations and the computed masks by DETCID.

####  4.2.2 Fully convolutional regression network (FCRN)

Fully convolutional regression network (FCRN) is a U-net based cell detection
method develop by Visual Geometry Group (VGG) at University of Oxford [22].
FCRN does not rely on a bounding box region of proposals and therefore is not
sensitive to IoU threshold for detecting overlapping cells in various
orientations. The network architecture includes a convolution and a
deconvolution path predicting a density map for the image. The ground truth
for every cell is defined as a 2D Gaussian where the pick is at the cell
center. The label images were obtained by filtering the binary masks where the
cell is white and the background is zero. The predicted local maxima in the
image were found using Laplacian of Gaussian (LoG) and the watershed algorithm
is used to detect the cell boundaries. We selected FCRN as a baseline because
FCRN is capable of detecting cell clumps in various orientations using only
synthetic training data.

###  4.3 Experimental results

UH-S-cdiff1 is used perform a 10-fold cross-validation to evaluate and compare
the performance. Furthermore, UH-S-cdiff1 is used to train DETCID to evaluate
on the acquired dataset UH-A-cdiff1. Mean average precision (mAP) and dice
score were computed to measure the performance of the detection and
segmentation respectively. Table 2 summarizes the quantitative performance
evaluations. The results indicates that DETCID outperforms the state-of-the-
art in detection of C. diff cells in the acquired SEM images UH-A-cdiff1
(P=0.04; 95% CI 0.001-Inf). However, DETCID achieves comparable results to the
state-of-the-art in detection of spores in UH-A-cdiff1 (P=0.36; 95%
CI-0.11-0.28) which is more challenging since they could be misclassified as
debris due to their smaller size. DETCID also achieved comparable results in
segmentation of the vegetative cells (P =0.63; 95% CI -0.071-0.11) and spores
(P=0.15; 95% CI -0.04-0.23).

Moreover, Table 2 indicates that DETCID achieved significant improvements in
detection (P$<$0.001, 95% CI -0.18-Inf) and comparable results in segmentation
(P=0.23; 95% CI -0.03-0.13) of UH-S-cdiff1 images where the number of touching
clustered cells are higher. Therefore, DETCID outperforms the state-of-the-art
in detection of C. diff cells where touching cells are clustered together.

Figure 4 depicts qualitative comparisons between DETCID performance and the
state-of-the-art. The qualitative results indicate that Mask-RCNN has better
performance in segmentation of isolated objects. However, DETCID outperforms
Mask-RCNN when multiple cells are touching and in presence of debris resulting
in lower mAP. FCRN is able to separate the touching cells. However, FCRN is
highly sensitive to inhomogeneous illumination and presence of debris,
resulting in poor mAP and dice score.

###  4.4 Bland–Altman analysis

A Bland-Altman analysis is performed to compute the agreement between the
result of DETCID and the primary manual annotation as well as the agreement
two set of annotations for cell detection and segmentation. Figure 5 depicts
the Blond-Altman plots for number of detected cells and the their segmentation
masks. The Bland-Altman plots reveal no evidence of proportional bias for cell
counts and segmentation differences. Furthermore, the number of detected cells
by DETCID correlated with the primary set of annotations (${R^{2}=0.79}$)
compared to the correlation between the two set of annotations ($R^{2}=0.81$).
Accordingly, DETCID performance differs from the primary set of manual
annotations in detection of CDI cells as the two set of manual annotations
differ from themselves. The area of the segmentation masks computed by DETCID
also correlated with the area of the annotated masks in the primary set
(${R^{2}=0.89}$) compared to the area of the masks between the two set of
annotations (${R^{2}=0.94}$).

##  5 Conclusion

In this paper, DETCID is proposed to detect and segment CDI cells in SEM
images. An adversarial region proposal network was implemented to address the
challenge of inhomogeneous illumination. Furthermore, a modified IoU metric is
used for non-max suppression for detecting clusters of touching cells. A data
augmentation algorithm was developed to provide a large number of training
images suitable for training deep feature extraction architectures such as
ResNet. The performance is compared to both deep region-based method and U-net
based methods. DETCID outperforms the state-of-the-art in detection of
touching cells and provide comparable result in segmentation of cells.

## References

  * [1] B. T. Endres, K. Begum, H. Sun, S. T. Walk, A. Memariani, C. Lancaster, A. J. Gonzales-Luna, K. M. Dotson, E. Bassères, C. Offiong, S. Tupy, K. Kuper, E. Septimus, R. Arafat, M. J. Alam, Z. Zhao, J. G. Hurdle, T. C. Savidge, and K. W. Garey, “Epidemic Clostridioides difficile ribotype 027 lineages: comparisons of Texas versus worldwide strains,” _Open forum infectious diseases_ , vol. 6, no. 2, pp. 1–13, 2019.
  * [2] B. T. Endres, E. Bassères, A. Memariani, L. Chang, M. J. Alam, R. J. Vickers, I. A. Kakadiaris, and K. W. Garey, “A novel method for imaging the pharmacological effects of antibiotic treatment on clostridium difficile,” _Anaerobe_ , vol. 40, pp. 10–14, 2016.
  * [3] H. Han, S. Shan, X. Chen, and W. Gao, “A comparative study on illumination preprocessing in face recognition,” _Pattern Recognition_ , vol. 46, no. 6, pp. 1691–1699, 2013.
  * [4] A. Memariani, C. Nikou, B. T. Endres, E. Bassères, K. W. Garey, and I. A. Kakadiaris, “DeTEC: Detection of touching elongated cells in SEM images,” in _Proc. Advances in Visual Computing_ , December 12-14 2016, pp. 288–297.
  * [5] D. Shen, G. Wu, and H.-I. Suk, “Deep learning in medical image analysis,” _Annual Review of Biomedical Engineering_ , vol. 19, pp. 221–248, 2017.
  * [6] A. Memariani and I. A. Kakadiaris, “SoLiD: Segmentation of Clostridioides Difficile Cells in the Presence of Inhomogeneous Illumination using a Deep Adversarial Network,” in _International Workshop on Machine Learning in Medical Imaging_ , September 16 2018, pp. 285–293.
  * [7] C. Arteta, V. Lempitsky, A. Noble, and A. Zisserman, “Detecting overlapping instances in microscopy images using extremal region trees,” _Medical Image Analysis_ , vol. 27, pp. 3–16, 2016.
  * [8] E. Türetken, X. Wang, C. J. Becker, C. Haubold, and P. Fua, “Network Flow Integer Programming to Track Elliptical Cells in Time-lapse Sequences,” _IEEE Transactions on Medical Imaging_ , vol. 36, no. 4, pp. 942–951, 2017.
  * [9] P. Kainz, M. Urschler, S. Schulter, P. Wohlhart, and V. Lepetit, “You should use regression to detect cells,” in _Proc. Medical Image Computing and Computer-Assisted Intervention_ , Munich, Germany, October 5-9 2015, pp. 276–283.
  * [10] A. Santamaria-Pang, J. Rittscher, M. Gerdes, and D. Padfield, “Cell segmentation and classification by hierarchical supervised shape ranking,” in _Proc. IEEE International Symposium on Biomedical Imaging_ , Brooklyn, NY, April 16-19 2015, pp. 1296–1299.
  * [11] J. Funke, F. A. Hamprecht, and C. Zhang, “Learning to segment: training hierarchical segmentation under a topological loss,” in _Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention_ , Munich, Germany, October 5-9 2015, pp. 268–275.
  * [12] A. Ahmad, A. Asif, N. Rajpoot, M. Arif, F. Amir, and A. Minhas, “Correlation filters for detection of cellular nuclei in histopathology images,” _Journal of Medical Systems_ , vol. 42, no. 1, pp. 7–15, 2018.
  * [13] S. Zhou, J. Wang, M. Zhang, Q. Cai, and Y. Gong, “Correntropy-based level set method for medical image segmentation and bias correction,” _Neurocomputing_ , vol. 234, pp. 216–229, 2017.
  * [14] X.-H. Zhi and H.-B. Shen, “Saliency driven region-edge-based top down level set evolution reveals the asynchronous focus in image segmentation,” _Pattern Recognition_ , vol. 80, pp. 241–255, 2018.
  * [15] A. Memariani, C. Nikou, B. T. Endres, E. Bassères, K. W. Garey, and I. A. Kakadiaris, “DETCIC: Detection of elongated touching cells with inhomogeneous illumination using a stack of conditional random fields.” in _Proc. International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications_ , Funchal Madeira Portugal, January 27-29 2018, pp. 574–580.
  * [16] V. Ulman, M. Maška, K. E. G. Magnusson, O. Ronneberger, C. Haubold, N. Harder, P. Matula, P. Matula, D. Svoboda, M. Radojevic, I. Smal, K. Rohr, J. Jaldén, H. M. Blau, O. Dzyubachyk, B. Lelieveldt, P. Xiao, Y. Li, S.-Y. Cho, A. C. Dufour, J.-C. Olivo-Marin, C. C. Reyes-Aldasoro, J. A. Solis-Lemus, R. Bensch, T. Brox, J. Stegmaier, R. Mikut, S. Wolf, F. A. Hamprecht, T. Esteves, P. Quelhas, Ö. Demirel, L. Malmström, F. Jug, P. Tomancak, E. Meijering, A. Muñoz-Barrutia, M. Kozubek, and C. Ortiz-de Solorzano, “An objective comparison of cell-tracking algorithms,” _Nature Methods_ , vol. 14, no. 12, p. 1141, 2017.
  * [17] J. Shi, X. Zheng, J. Wu, B. Gong, Q. Zhang, and S. Ying, “Quaternion Grassmann average network for learning representation of histopathological image,” _Pattern Recognition_ , vol. 89, pp. 67–76, 2019.
  * [18] L. Hou, V. Nguyen, A. B. Kanevsky, D. Samaras, T. M. Kurc, T. Zhao, R. R. Gupta, Y. Gao, W. Chen, D. Foran, and Others, “Sparse autoencoder for unsupervised nucleus detection and representation in histopathology images,” _Pattern Recognition_ , vol. 86, pp. 188–200, 2019.
  * [19] R. B. Hegde, K. Prasad, H. Hebbar, and B. M. K. Singh, “Comparison of traditional image processing and deep learning approaches for classification of white blood cells in peripheral blood smear images,” _Biocybernetics and Biomedical Engineering_ , vol. 39, no. 2, pp. 382–392, 2019.
  * [20] M. Tofighi, T. Guo, J. K. P. Vanamala, and V. Monga, “Deep networks with shape priors for nucleus detection,” in _Proc. International Conference on Image Processing_ , Athens, Greece, October 7-10 2018, pp. 719–723.
  * [21] Y. Xue, N. Ray, J. Hugh, and G. Bigras, “Cell counting by regression using convolutional neural network,” in _Proc. European Conference on Computer Vision_ , Amsterdam, The Netherlands, October 8-16 2016, pp. 274–290.
  * [22] W. Xie, J. A. Noble, and A. Zisserman, “Microscopy cell counting and detection with fully convolutional regression networks,” _Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization_, vol. 6, no. 3, pp. 283–292, 2018.
  * [23] P. Tiwari, J. Qian, Q. Li, B. Wang, D. Gupta, A. Khanna, J. J. P. C. Rodrigues, and V. H. C. de Albuquerque, “Detection of subtype blood cells using deep learning,” _Cognitive Systems Research_ , vol. 52, pp. 1036–1044, 2018\.
  * [24] H. Höfener, A. Homeyer, N. Weiss, J. Molin, C. F. Lundström, and H. K. Hahn, “Deep learning nuclei detection: A simple approach can deliver state-of-the-art results,” _Computerized Medical Imaging and Graphics_ , vol. 70, pp. 43–52, 2018.
  * [25] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proc. Computer Vision and Pattern Recognition_ , Boston, MA, June 8-10 2015, pp. 3431–3440.
  * [26] O. Ronneberger, P. Fischer, and T. Brox, “U-net: convolutional networks for biomedical image segmentation,” in _Proc. Medical Image Computing and Computer-Assisted Intervention_ , Munich, Germany, October 5-9 2015, pp. 234–241.
  * [27] Y. Xie, F. Xing, X. Shi, X. Kong, H. Su, and L. Yang, “Efficient and robust cell detection: A structured regression approach,” _Medical Image Analysis_ , vol. 44, pp. 245–254, 2018.
  * [28] N. Ramesh and T. Tasdizen, “Cell Segmentation using a Similarity Interface with a Multi-task Convolutional Neural Network,” _IEEE Journal of Biomedical and Health Informatics_ , vol. 23, no. 4, pp. 1457–1468, 2019.
  * [29] P. Naylor, M. Laé, F. Reyal, and T. Walter, “Segmentation of nuclei in histopathology images by deep regression of the distance map,” _IEEE Transactions on Medical Imaging_ , vol. 38, no. 2, pp. 448–459, 2018.
  * [30] J. Gu, Y. Zhu, B. Yang, J. J. Yang, J. W. Yang, J. Y. Yang, and W. Z. Yang, “A Multi-scale Pyramid of Fully Convolutional Networks for Automatic Cell Detection,” in _International Conference on Bioinformatics and Biomedicine_ , Madrid, Spain, December 3-6 2018, pp. 633–636.
  * [31] W. Li, J. Li, K. V. Sarma, K. C. Ho, S. Shen, B. S. Knudsen, A. Gertych, and C. W. Arnold, “Path R-CNN for prostate cancer diagnosis and gleason grading of histological images,” _IEEE Transactions on Medical Imaging_ , vol. 38, no. 4, pp. 945–954, 2018.
  * [32] F. Wan, O. Smedby, and C. Wang, “Simultaneous MR knee image segmentation and bias field correction using deep learning and partial convolution,” in _Proc. Medical Imaging: Image Processing_ , pp. 61–67.
  * [33] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Proc. Advances in neural information processing systems_ , Montréal, Canada, December 8-13 2014, pp. 2672–2680.
  * [34] P. Luc, C. Couprie, S. Chintala, and J. Verbeek, “Semantic segmentation using adversarial networks,” in _Proc. NIPS Workshop on Adversarial Training_ , Barcelona, Spain, December 5-10 2016, pp. 1–9.
  * [35] Z. Li, Y. Wang, and J. Yu, “Brain Tumor Segmentation using an Adversarial Network,” in _Proc. International MICCAI Brainlesion Workshop_ , Quebec City, QC, Canada, September 10-14 2017, pp. 123–132.
  * [36] P. Moeskops, M. Veta, M. W. Lafarge, K. A. J. Eppenhof, and J. P. W. Pluim, “Adversarial training and dilated convolutions for brain MRI segmentation,” in _Proc. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support_ , Quebec City, QC, Canada, September 14 2017, pp. 56–64.
  * [37] Y. Xue, T. Xu, H. Zhang, L. R. Long, and X. Huang, “SegAN: Adversarial network with multi-scale L-1 loss for medical image segmentation,” _Neuroinformatics_ , no. 3-4, pp. 383–392.
  * [38] S. Lee, S. Han, P. Salama, K. W. Dunn, and E. J. Delp, “Three Dimensional Blind Image Deconvolution for Fluorescence Microscopy using Generative Adversarial Networks,” in _Proc. International Symposium on Biomedical Imaging_ , Venice, Italy, April 8-11 2019, pp. 538–542.
  * [39] T.-Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proc. Computer Vision and Pattern Recognition_ , Honolulu, HI, July 21-26 2017.
  * [40] K. He, G. Gkioxari, P. Dollar, and R. Girshick, “Mask R-CNN,” in _Proc. International Conference on Computer Vision_ , Venice, Italy, October 22-29 2017, pp. 2961–2969.
  * [41] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” in _Proc. Advances in Neural Information Processing Systems_ , Montreal, Canada, October 22-29 2015, pp. 91–99.
  * [42] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You Only Look Once: Unified, Real-time Object Detection,” in _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , Las Vegas, NV, June 26 – July 1 2016\.
  * [43] S. Korman and S. Avidan, “Coherency sensitive hashing,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 38, no. 6, pp. 1099–1112, 2015.
  * [44] A. A. Proc. Efros and W. T. Freeman, “Image quilting for texture synthesis and transfer,” in _Computer Graphics and Interactive Techniques_ , New York, NY, August 2001, pp. 341–346.
  * [45] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, and Others, “TensorFlow: A system for large-scale machine learning,” in _Proc. Symposium on Operating Systems Design and Implementation_ , November 2–4 2016, pp. 265–283.
  * [46] W. Dai, J. Doyle, X. Liang, H. Zhang, N. Dong, Y. Li, and E. P. Xing, “SCAN: Structure Correcting Adversarial Network for Organ Segmentation in Chest X-rays,” in _Proc. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support_ , Granada, Spain, September 20 2018, pp. 263–273.
  * [47] D. A. Van Valen, T. Kudo, K. M. Lane, D. N. Macklin, N. T. Quach, M. M. DeFelice, I. Maayan, Y. Tanouchi, E. A. Ashley, and M. W. Covert, “Deep learning automates the quantitative analysis of individual cells in live-cell imaging experiments,” _PLoS Computational Biology_ , vol. 12, no. 11, p. e1005177, 2016.
  * [48] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _Proc. Computer Vision and Pattern Recognition_ , Miami Beach, FL, June 22-24 2009, pp. 248–255.
  * [49] L. Feng, J. H. Song, J. Kim, S. Jeong, J. S. Park, and J. Kim, “Robust nucleus detection with partially labeled exemplars,” _IEEE Access_ , vol. 7, pp. 162 169–162 178, 2019.
  * [50] W. Abdulla, “Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow,” Available: https://github.com/matterport/Mask-RCNN, 2017, last accessed: 2019-11-22.