# Unsupervised Learning of Object-Centric Embeddings
for Cell Instance Segmentation in Microscopy Images

Steffen Wolf1, Manan Lalit2, Henry Westmacott1, Katie McDole1, Jan Funke2
1MRC Laboratory of Molecular Biology, 2HHMI Janelia Research Campus
Corresponding Author<EMAIL_ADDRESS>Author,
<EMAIL_ADDRESS>
###### Abstract

Segmentation of objects in microscopy images is required for many biomedical
applications. We introduce object-centric embeddings (OCEs), which embed image
patches such that the spatial offsets between patches cropped from the same
object are preserved. Those learnt embeddings can be used to delineate
individual objects and thus obtain instance segmentations. Here, we show
theoretically that, under assumptions commonly found in microscopy images,
OCEs can be learnt through a self-supervised task that predicts the spatial
offset between image patches. Together, this forms an unsupervised cell
instance segmentation method which we evaluate on nine diverse large-scale
microscopy datasets. Segmentations obtained with our method lead to
substantially improved results, compared to state-of-the-art baselines on six
out of nine datasets, and perform on par on the remaining three datasets. If
ground-truth annotations are available, our method serves as an excellent
starting point for supervised training, reducing the required amount of
ground-truth needed by one order of magnitude, thus substantially increasing
the practical applicability of our method. Source code is available at
github.com/funkelab/cellulus.

##  1 Introduction

Determining whether two image regions belong to the same object is a
fundamental challenge in instance segmentation, albeit a simple task for
humans. A plausible hypothesis is that humans learn to recognize parts as
belonging to a whole by repeatedly observing them in each other’s vicinity. We
introduce object-centric embeddings (OCEs), which leverage this assumption for
unsupervised instance segmentation. OCEs map image patches in such a way that
the spatial offsets between patches cropped from the same object are preserved
in embedding space. We investigate the usage of OCEs in the domain of
microscopy imaging and introduce Cellulus, a method that identifies and
segments individual cells in microscopy images.

By relying on reasonable assumptions about microscopy images, namely that (i)
the objects in these images have a similar appearance and (ii) the objects in
these images are randomly distributed, we show that OCEs can be learnt in an
unsupervised fashion.

Cell instance segmentation is crucial for answering important life science
questions. In recent years, deep learning-based segmentation approaches [23,
12] have achieved the best performance on standard benchmarking datasets, but
these approaches rely on large amounts of annotated training data. Our
proposed unsupervised method Cellulus, in contrast, circumvents the problem of
acquiring these manual annotations.

With Cellulus, we provide an approach for employing the learnt object-centric
embedding locations per patch, identifying image patches that are part of the
same cell and thus segmenting cell instances in a unsupervised way (see Figure
1 for a few examples). We demonstrate that this unsupervised segmentation
pipeline achieves competitive results with respect to pre-trained baseline
models on a diverse set of nine microscopy image datasets (see Table 1).

Additionally, instance segmentations obtained through our proposed
unsupervised pipeline are excellent starting points to support supervised
training when very little manually generated ground truth annotations are
available. We show that we obtain comparable performance to supervised
segmentation methods, after fine-tuning on one order of magnitude less data
(see Figure 5).

More generally, supervised training supported by unsupervised segmentation is
at least as good as purely supervised learning on all investigated datasets,
demonstrating that our method dramatically reduces the amount of ground truth
annotations needed, and at times not requiring any.

Figure 1: Method overview and example segmentations on diverse datasets. Top
row: An unsupervised learning objective gives rise to object-centric
embeddings (OCEs), such that patches extracted from the same object (green
boxes) maintain their relative position to each other. Predicted densely,
these OCEs allow instance segmentation of cells in microscopy images, by using
a post-processing step such as mean-shift clustering. Bottom row: Example raw
images and dense OCEs/instance segmentations on four datasets spanning
different imaging modalities, cell sizes and shapes.

Reducing or eliminating the need for manual ground truth is of particular
importance to biological research, as new light-microscopy methods are capable
of generating terabytes of data in a single experiment. Manually annotating
even small regions of such datasets can take hundreds or thousands of human
hours. Thus, there is a tremendous need for self-supervised learning methods
to help cope with the vast amount of data generated by modern microscopes.
Cellulus is available at github.com/funkelab/cellulus.

##  2 Related Work

Currently, machine learning and deep learning-based methods dominate the field
of cell instance segmentation [26, 23, 12]. These cell segmentation methods
can be categorized by their intermediate (auxiliary) representation used to
derive the predicted segmentation.

StarDist [22], for example, represents objects as star-convex polygons (_i.e._
, distances from a center point to the cell boundary along sets of equi-
distant rays). On the other hand, Cellpose [23] encodes cells by vectors that
point inwards from the boundary. The representations of StarDist and Cellpose
are pre-defined and tailored to the tasks of cell segmentation.

Alternatively, pixel-level representations (here referred to as embeddings)
can be learnt from labels directly by pulling embeddings of pixels within
instances together and pushing embeddings across instances apart [3].
Initially developed for natural images, this concept was further developed
into a cell segmentation and tracking algorithm in the work by Payer _et al_.
[18], which established the state-of-the-art on six Cell Tracking Challenge
(CTC) datasets.

Recent submissions to the CTC further improved the segmentation and tracking
performance. While Arbelle _et al_. [1] and Scherr _et al_. [21] relied on
boundary classification to separate densely clustered cells, Löffler _et al_.
[15] used spatial embeddings.

Spatial embedding-based approaches learn a function which associates each
pixel at location $i$ in the raw image, to a relative spatial embedding
(offset vector) $r_{i}$, such that the resulting absolute spatial embedding
$e_{i}=i+r_{i}$ for all pixels belonging to an object instance point to a
common point (_e.g._ the instance centroid).

Typically, the embeddings are learnt using a regression loss function, either
minimizing the distance between absolute spatial embeddings of pairs of pixels
$i,j$ from the same instance
$\mathcal{L}_{\text{regr}}=\sum_{i,j}\sigma\left(e_{i}-e_{j}\right)$ or
equivalently by approaching the mean over the whole instance [16, 12]. Here,
$\sigma$ is a measure of distance, _e.g._ , $|\cdot|^{2}$. Recently, EmbedSeg
used spatial embeddings to establish the state-of-the-art on multiple 2D and
3D microscopy datasets [12].

We note that our learning approach has parallels with supervised learning of
pixel-wise spatial embeddings. In our work, self-supervised learning leads to
object-centric embeddings, which are post-processed using the mean-shift
clustering algorithm, analogous to De Brabandere _et al_. [3].

Figure 2:  Unsupervised Learning of Object-Centric Embeddings. During
learning, small image patches are randomly cropped from the raw image and
embedded through a learnable function $f_{\theta}$ into a 2D embedding space.
The objective of the loss $\mathcal{L}_{\text{{OCE}}}$ is to ensure that the
spatial offset between pairs of patches in the raw image (green arrows) is
preserved in the embedding space (see Equation 4).

Self-supervised learning methods learn representation by solving tasks that
predict an intentionally hidden part of the data. Predicting the spatial
arrangement of image patches provides a rich signal for learning meaningful
representations for downstream tasks. Spatial tasks include solving jigsaws
[17], predicting patch rotations [6], or classifying relative patch positions
from a grid-like pattern [5]. More recently, contrastive learning between
multiple views [8, 9, 25, 14] enabled learning of representations that
transfer well to downstream tasks. These learnt representations have been
shown to reduce the required amounts of annotated data in tasks such as image
classification [10] and semantic segmentation [13, 24].

###  2.1 Unsupervised Methods for Cell Segmentation

Recently, methods for cell instance segmentation have been proposed that do
not rely on human annotation.

The unsupervised segmentation pipeline proposed by Din and Yu [4] employed a
Convolutional Neural Network (CNN), which when centered on each cell nucleus
is tasked to predict a binary mask for each cell. The model is trained without
any ground-truth and is tasked to predict consistent masks that cover all
foreground pixels. However, this method still relied on pre-trained networks
for locating the nuclei using which the cell segmentations are predicted and
can therefore not be considered fully unsupervised.

Completely unsupervised instance separation has been proposed by Wolf  _et
al_. [27], where inpainting networks are used to determine which image regions
are independent. These independent regions are determined by a hierarchical
optimization strategy that continually subdivides the image until all
instances are separated. In contrast to our proposed method Cellulus, the
post-processing step of Wolf  _et al_. [27] is very computationally expensive
and does not provide a method for detecting background regions automatically.

Xie _et al_. [28] proposed a self-supervised method that employed two proxy
tasks of estimating nuclei size and ranking count of nuclei and this enabled
the model to mine instance-aware representations from raw data.

##  3 Method

We aim to learn an embedding of image patches that reflects the relative
spatial arrangement of these patches (_i.e._ the offset between the predicted
embeddings should be equal to their spatial offset), as if they were extracted
from the same object (see Figure 2). We refer to the spatial offset between
patches extracted from the same object as intra-object offset and the learnt
embeddings as _object-centric embeddings_ (OCEs).

###  3.1 Unsupervised Learning of OCEs

Under conditions that are commonly found in microscopy images, OCEs can be
learnt in an unsupervised manner, _i.e._ , without the provision of
segmentation ground-truth. Those conditions are:

  1. 1.

Objects in the image are similar

  2. 2.

Objects are randomly distributed in the image plane

  3. 3.

A patch cropped from an object contains enough information to identify its
position inside the object (_i.e._ , no two parts of an object look exactly
identical)

Under these conditions, the _expected_ offset between two image patches is
proportional to the _intra-object offset_ of those two patches, _i.e._ , the
spatial offset between those patches if they were part of the same object.

Let $a$ and $b$ be two different patches found on an object (_e.g._ , the
left- and the right-most patches of a cell, see Figure 3). If multiple similar
objects are present in an image, there will be multiple locations $i\in\Omega$
where the patch $a$ is visible, and distinct locations $j\in\Omega$ where the
patch $b$ is visible. Here, $\Omega$ is the set of all pixel locations and
$x:\Omega\mapsto\mathcal{R}$ is the image.

Figure 3:  Illustration of the expected offset between two example patches in
an idealized image: Black squares show all image locations where the two
patches  and  are found. The expected offset between those patches stems from
offsets observed within the _same_ object (_intra-object offsets_ , shown as
green arrows) and offsets observed between _different_ objects (_inter-object
offsets_ , shown as orange arrows for the center object only). Assuming a
random distribution of objects in large images, the average offset between
_different_ objects is zero, thus the expected offset $\overrightarrow{ij}$
between the given patches is proportional to the intra-object offset.

We will refer to the image patch at a location $i$ as $p(i)$ and denote the
set of all locations that contain a given patch $a$ as $\Omega_{a}$, _i.e._ ,
$\Omega_{a}=\left\\{i\in\Omega\;\;|\;\;p(i)=a\right\\}$.

Consider the expected observed offset $\overrightarrow{ij}=j-i$ between all
occurrences of patches $a$ and $b$: for each object contained in the image,
patches $a$ and $b$ are observed once with their _intra-object offset_ ,
_i.e._ , the offset they have to each other as being part of the same object.
For every pair of different objects, however, patches $a$ and $b$ will be
observed at random offsets, following the assumption that objects in the image
are randomly distributed. The key insight that allows unsupervised learning of
OCEs is that the observed offsets of patches from different objects have zero
mean.

Formally, the expected offset between all locations of two image patches $a$
and $b$ is given as

$\mathbb{E}\left[{\overrightarrow{ij}|a,b}\right]\approx\frac{1}{N}\sum_{i\in\Omega_{a}}\sum_{j\in\Omega_{b}}\overrightarrow{ij}\text{,}$
(1)

where $N=\left|\Omega_{a}\right|\cdot\left|\Omega_{b}\right|$ is the number of
pairs of image locations $i,j$, where patches $a$ and $b$ are observed.

This expectation can be rewritten to distinguish observed offsets from the
_same_ versus _different_ objects. For that, let $\Omega_{b}^{i}$ denote all
locations $j$ where patch $b$ appears and is part of the same object at
location $i$. Similarly, let $\overline{\Omega}_{b}^{i}$ be the set of
locations $j$ where patch $b$ appears, but is not part of the object at
location $i$. We can now rewrite the expected observed offset
$\overrightarrow{ij}$ as

$\displaystyle\mathbb{E}\left[{\overrightarrow{ij}|a,b}\right]$
$\displaystyle\approx\frac{1}{N}\sum_{i\in\Omega_{a}}\left[\sum_{j\in\Omega_{b}^{i}}\overrightarrow{ij}+\sum_{j\in\overline{\Omega}_{b}^{i}}\overrightarrow{ij}\right]$
(2)
$\displaystyle=\underbrace{\frac{1}{N_{\text{s}}}\sum_{i\in\Omega_{a}}\sum_{j\in\Omega_{b}^{a}}\overrightarrow{ij}}_{\text{intra-
object
offset}}+\underbrace{\frac{1}{N_{\text{d}}}\sum_{i\in\Omega_{a}}\sum_{j\in\overline{\Omega}_{b}^{a}}\overrightarrow{ij}}_{\approx
0}\text{,}$ (3)

where $N_{\text{s}}$ and $N_{\text{d}}$ denote the number of times that
patches $a$ and $b$ are observed in the same object and different objects,
respectively.

The first term in Equation 3 is, by definition, the intra-object offset,
_i.e._ , the quantity we aim to infer. The second term is the expected offset
between patches $a$ and $b$ if both are part of different objects. Under the
assumption that multiple similar objects are randomly distributed in the
image, this expectation is zero: observing patch $a$ relative to patch $b$
with offset $\overrightarrow{ij}$ is just as likely as observing them at the
inverse offset $\overrightarrow{ji}$. Without any supervision, the constants
$N_{\text{s}}$ and $N_{\text{d}}$ are not known. The expected offset,
calculated as in Equation 1, is thus proportional to the sought after intra-
object offset.

In conclusion, the expected offset given any two patches approximates the
offset between the patches extracted from the same object. We can leverage
this property to devise a loss function that minimizes the differences between
the spatial and embedding offsets between pairs of patches, and thus learn an
object-centric embedding in an unsupervised fashion.

Let $f_{\theta}:\mathcal{P}\mapsto\mathbb{R}^{2}$ be a parameterized embedding
function, mapping from the set of all image patches $\mathcal{P}$ to a 2D
embedding space. We denote a patch located at $i$ as $p(i)$ and its embedding
as $r(i)=f_{\theta}(p(i))$. We propose the following unsupervised loss,
minimizing the difference between $d(i,j)=i-j$ and $\hat{d}(i,j)=r(i)-r(j)$
for pairs of patches:

$\mathcal{L}_{\text{{OCE}}}=\sum_{i,j\in\Omega}\sigma\left(d(i,j)-\hat{d}(i,j)\right)\text{,}$
(4)

where $\sigma$ is a measure of distance, _e.g._ , $|\cdot|_{2}$ (we will
discuss our choice of $\sigma$ below).

###  3.2 Loss Implementation

Figure 4: Overview of the inference pipeline. Input image to the trained
object-centric embedding (OCE) network is augmented repeatedly with salt and
pepper noise, producing several noisy instances of the raw image (first
column). OCEs are predicted densely for each noisy instance of the input raw
image (second column). Next, the pixel-wise mean and variance of the predicted
OCEs is calculated (third column). Images locations with high variance are
treated as the background. The remaining foreground region is clustered into
individual object instances using mean-shift clustering (fourth column).

In practice, the embedding function will be implemented as a Convolutional
Neural Network (CNN) and its weights can be updated using stochastic gradient
descent. In this setting, strong gradient contributions resulting from pairs
of patches of different objects can be problematic due to their high variance,
even if they have zero mean. To address this, we dampen the effect of large
distances in our loss function by using a sigmoid distance function, _i.e._ ,
$\sigma(\delta)=\left(1+\exp(-\frac{\norm{\delta}^{2}_{2}}{\tau})\right)^{-1}$,
where $\tau$ is a hyperparameter controlling the rate of damping.

Furthermore, we limit the sampling of pairs of patches to have a maximal
distance $\kappa$ and add an L2 regularization term to obtain our final
unsupervised loss function as

$\mathcal{L}=\sum_{i,j\in
P}\sigma\left(d(i,j)-\hat{d}(i,j)\right)+\lambda_{\text{reg}}\norm{r(i)}_{2}\text{,}$
(5)

where $P\subset\\{i,j\in\Omega\;|\;|i-j|_{2}\leq\kappa\\}$. For more details,
see Appendix A.

###  3.3 Instance Segmentation from OCEs

An instance segmentation can be obtained from OCEs by firstly segmenting
foreground vs. background, followed by partitioning the foreground into
individual instances.

To address the background identification, we exploit the sensitivity of the
OCEs to noise in background: We observe that certain noise patterns in the
background (_e.g._ , single bright pixels) become the center point of locally
consistent embeddings, thus creating spurious objects (see Figure 4, first
column for an example). To identify background, we repeatedly introduce
artificial noise to the raw image and measure the variance of the predicted
embeddings (we found salt-and-pepper noise to be effective). We find that the
distribution of the variance of these embeddings over image locations is bi-
modal, such that a parameter-free thresholding method like Otsu’s is
sufficient to separate foreground from background.

After identifying the background, we segment individual instances in the
foreground through a mean-shift clustering on the dense OCE predictions [2,
19] (see Figure 4).

##  4 Experiments

Used Datasets. We test our method Cellulus on nine publicly available datasets
for which dense ground truth annotations are available. The datasets were
chosen to represent a diverse set of image modalities, cell/tissue types, and
imaging platforms.

TissueNet [7] is the largest of the analyzed datasets, with $1.3$ million
annotated cells. It covers six imaging platforms and includes histologically
normal and diseased tissue of humans, mice, and macaques. The included tissue
types (Immune, Lung, Pancreas, Skin cells) vary widely in cell appearance and
density. Therefore, we add evaluations where we restrict the dataset to the
four individual tissue types. For reference, constructing TissueNet required
$>4,000$ hours of human annotation time.

The nuclei and whole cell are labeled in TissueNet and both of these image
channels were used during training and inference. For evaluation purposes
during inference, the predicted instance segmentations are compared against
the ground truth labels for the whole cell image channel.

Cell Tracking Challenge (CTC) [26] provides diverse 2D and 3D datasets
111http://celltrackingchallenge.net/2d-datasets/. We select five 2D datasets
with distinct cell appearances: HSC, HU7, Simulated, FluoHela and PSC.

Each dataset comes with two sets of image sequences: $1$ and $2$. We used set
$1$ for training, while set $2$ is held out for evaluation. Images in the CTC
datasets contain only one channel.

Segmentation Metrics. We use two widely used cell segmentation scores: (i) SEG
score (used by CTC [26]) matches every ground truth object to a predicted
instance segmentation and measures the average intersection over union (IoU)
of all matches. (ii) F1 score (used by Greenwald  _et al_. [7]) matches all
predictions and ground truth objects with an IoU greater than or equal to a
fixed threshold ($0.5$ unless specified) and reports the F1 measure of
successfully found matches.

Table 1: Quantitative results when no annotations are available (fully
unsupervised setting).
The pretrained models of StarDist [22] and Cellpose [23] are compared with
Cellulus on nine diverse microscopy image datasets. Two instance segmentation
metrics F1 and SEG are evaluated by comparing the quality of predicted
instance segmentation with the ground truth instance segmentation. Best
performing method on each dataset is shown in bold. The last row TissueNet
(all) shows a weighted average (weights proportional to the number of images)
of results for Immune, Lung, Pancreas and Skin.

| Cellpose | StarDist | Cellulus
---|---|---|---
| F1 | SEG | F1 | SEG | F1 | SEG
HSC | $0.00$ | $0.00$ | 0.09 | $0.14$ | $0.06$ | 0.42
HU7 | $0.40$ | $0.27$ | $0.03$ | $0.02$ | 0.75 | 0.55
Simulated | $0.49$ | $0.34$ | $0.23$ | $0.35$ | 0.83 | 0.65
FluoHela | $0.36$ | $0.65$ | 0.38 | 0.79 | $0.34$ | $0.70$
PSC | 0.76 | 0.58 | $0.64$ | $0.47$ | $0.64$ | $0.51$
Immune | $0.44$ | $0.21$ | $0.66$ | $0.41$ | 0.69 | 0.57
Lung | $0.76$ | $0.53$ | 0.81 | 0.59 | $0.51$ | $0.51$
Pancreas | $0.56$ | $0.36$ | $0.58$ | $0.36$ | 0.67 | 0.49
Skin | $0.48$ | $0.26$ | $0.39$ | $0.24$ | 0.60 | 0.46
TissueNet (all) | 0.55 | 0.32 | 0.59 | 0.38 | 0.64 | 0.52

###  4.1 Unsupervised Segmentation

We compare Cellulus against two state-of-the-art pretrained segmentation
models that are widely used across datasets. We investigate the segmentation
performance under the condition that no ground truth annotations are
available.

Baseline Methods. StarDist [22] is a widely used cell/nucleus segmentation
method. It predicts, for each pixel, the distances to the boundary in a
predefined set of directions. Cellpose [23] uses a supervised network to
predict spatial embeddings and clusters pixels together using a diffusion-
based aggregation method.

Segmentation Performance.  For each dataset, we train an object-centric
embedding network. Raw images are intensity-normalized ($1$ percentile
intensity is mapped to $0$ while $99.8$ percentile intensity is mapped to $1$)
and input to the network to produce dense object-centric embeddings. During
inference, these object-centric embeddings are processed to obtain instance
segmentations and the F1 and SEG scores are computed with respect to the
ground truth masks for the set of images held-out for evaluation purposes (see
Table 1). An overview of the datasets and the predicted embeddings and
instance segmentations is shown in Figure 7.

Our method outperforms both baselines on real-world datasets HU7, HSC,
Simulated, Immune, Pancreas and Skin (according to SEG score). On the
Simulated dataset, our method performs exceptionally well (see Table 1 and
Appendix D). To highlight the success and failure modes of our method, we
measure the F1 score per image and report the [$0^{\text{th}}$,
$25^{\text{th}}$, $50^{\text{th}}$, $75^{\text{th}}$, and $100^{\text{th}}$]
percentile images for different tissue types in Figure 6.

Table 2:  Performance of Cellulus across a range of scale factors for two
datasets.
Two instance segmentation metrics F1 and SEG are evaluated by comparing the
quality of predicted instance segmentation obtained using Cellulus at
different scale factors, with the ground truth labels at scale factor = $1.0$.
Scale factor is inversely related to the employed patch size.

Scale Factor | $0.5$ | $0.6$ | $0.7$ | $0.8$ | $0.9$ | $1.0$ | $1.1$ | $1.2$ | $1.3$ | $1.4$ | $1.5$
---|---|---|---|---|---|---|---|---|---|---|---
Immune
F1 | 0.12 | 0.33 | 0.48 | 0.44 | 0.64 | 0.69 | 0.69 | 0.66 | 0.67 | 0.59 | 0.58
SEG | 0.20 | 0.28 | 0.38 | 0.36 | 0.48 | 0.57 | 0.55 | 0.57 | 0.57 | 0.56 | 0.54
Lung
F1 | 0.17 | 0.23 | 0.32 | 0.52 | 0.48 | 0.51 | 0.36 | 0.46 | 0.37 | 0.27 | 0.28
SEG | 0.27 | 0.25 | 0.35 | 0.49 | 0.44 | 0.51 | 0.40 | 0.52 | 0.48 | 0.38 | 0.44

We find that our method can compensate for some variations in object sizes.
Compare, for example, the small cells in the $75^{\text{th}}$ percentile image
of the Immune dataset with more voluminous cells in the $100^{\text{th}}$
percentile (see Figure 6). However, we also observe that larger outlier
objects (_e.g._ , the $0^{\text{th}}$ percentile Skin image) lead to
structural under-segmentation.

Background Detection Performance. We observe that our background detection
generally matches the ground truth in the datasets HU7, FluoHela, TissueNet,
PSC and Simulated, where no additional structure in the background is visible.
When objects are exceptionally dim, their embeddings may vary with the added
noise, which leads to them being treated as background (e.g. see Figure 6,
$25^{\text{th}}$ percentile in the Skin dataset).

The HSC dataset is exceptionally challenging, with a visible culture plate in
the background adding additional structure to the image. This leads to
additional segments in our method (see Figure 7). All studied methods struggle
to predict these segments accurately, with an F1 score below $0.1$.
Remarkably, our method receives a high SEG score compared to the other
methods. Further analysis of this dataset, including an evaluation of
segmentation metrics for all matching thresholds, can be found in Appendix C.

Figure 5: Supervised cell segmentation performance for varying amounts of
annotations. We compare a classical supervised learning approach (blue)
trained using only manual annotations, against using a mixture of manual
annotations and pseudo-ground truth derived from our unsupervised OCEs
(orange), on the four tissue types (Immune, Pancreas, Lung and Skin) in the
TissueNet dataset [7]. The results for All (left) are obtained by averaging
the results obtained individually on the four datasets (right).

Scale Informs All Parameter Choices. The size of the selected image patches
determines what object sizes can be detected. Patches should be smaller than
individual objects but still contain meaningful features. To keep all network
parameters and training setup constant, we resize the training data.
Specifically, datasets HU7, PSC and Simulated are re-scaled by a factor of
[$0.5$,$2.$, $\frac{2}{3}$], respectively. All other datasets were analyzed at
their native resolution.

We also explore the performance of Cellulus across a range of scale factors
for two datasets Immune and Lung (see Table 2). Note that predictions produced
for any scale factor are compared against ground truth labels at scale factor
= $1.0$ to obtain the F1 and SEG scores.

Implementation. For learning the object-centric embeddings, we use a U-Net
architecture with a limited field of view of $16\times 16$ (single $2$x down-
sampling layer, ReLU activation). For more details, see Appendix A.

After the training, a scale-appropriate bandwidth of mean-shift clustering has
to be chosen. We use the implementation of mean-shift clustering provided by
scikit-learn [19] and perform a line search to determine the optimal value.

When instances are tightly packed, our segmentation matches closely with the
ground truth without post-processing. However, when instances are surrounded
by background, we find that patches close to the object borders get mapped to
the object center. Therefore, we shrink our objects to correct for this halo
(see Appendix D). We pick the optimal shrinkage distance between $0$ and $6$
pixels for all datasets and report the best score.

###  4.2 Supporting Supervised Learning

In the following experiments, we investigate how our unsupervised segmentation
can be used to increase model performance when only a few objects are
annotated.

Supervised Training Setup. For the supervised training setup, we build two
sparsely annotated supervised datasets, which we call the sparse and pseudo
dataset by randomly sampling ground truth objects as a fixed percentage of
annotated cells. (1) The sparse dataset contains only the annotated samples.
(2) The pseudo dataset uses our predicted segmentations as a starting point
(pseudo ground truth) and utilizes the same sampled annotations to correct our
predictions. We mask all our predicted objects that overlap with the
annotations and use the annotations instead. We include annotations of
background pixels close to the labeled object ($<30$ pixels).

We use these datasets to train a U-Net using a supervised StarDist training
loss $\mathcal{L}_{\text{\mbox{{StarDist}}}}$ [22]. Mini-batches contain half
of the images from the sparse dataset and the other half from the pseudo
dataset. The StarDist loss is computed on each respective half
($\mathcal{L}_{\text{\mbox{{StarDist}}}}^{\text{sparse}}$ and
$\mathcal{L}_{\text{\mbox{{StarDist}}}}^{\text{pseudo}}$). The total loss
$\mathcal{L}_{\text{\mbox{{StarDist}}}}=(1-\alpha)\mathcal{L}_{\text{\mbox{{StarDist}}}}^{\text{sparse}}+\alpha\mathcal{L}_{\text{\mbox{{StarDist}}}}^{\text{pseudo}}$
is a linear combination, where $\alpha=0$ corresponds to classical supervised
training. For further details, see Appendix B.

Figure 6: Predicted OCEs and segmentations on the TissueNet dataset with
tissue types Immune (top-left), Skin (top-right), Lung (bottom-left) and
Pancreas (bottom-right). The F1-score is evaluated for each individual image
and the $0^{\text{th}}$, $25^{\text{th}}$, $50^{\text{th}}$, $75^{\text{th}}$
and $100^{\text{th}}$ percentile images and their respective F1 scores are
reported in each column. Rows (from the top) show the Raw Images, Dense
Prediction of OCEs, the Predicted Instance Segmentation and the Ground Truth
Instance Segmentation available for evaluation purposes.

We train StarDist models with varying amounts of annotations and compare the
performance of models trained only on annotated images ($\alpha=0$, blue in
Figure 5) with those supported by our segmentation ($\alpha=0.5$, orange in
Figure 5). Each experiment is repeated $3$ times with different annotation
samples. We evaluate the trained networks on the full TissueNet dataset as
well as subsets of tissue types Immune, Pancreas, Lung and Skin. All measured
performances and their standard deviations are visualized in Figure 5.

Supported supervision makes a significant improvement at $1$%. When $1$% of
cell annotations from the TissueNet dataset are used, F1 $=0.75\pm 0.03$ is
obtained which is significantly better than the performance of the purely
unsupervised segmentation at F1 $=0.64$ (see Table 1). This effect could be
due to the biases of the StarDist representation, which might help to refine
the unsupervised segmentation.

We additionally perform a training experiment using $0$% ground truth
annotations (_i.e._
$\mathcal{L}_{\text{\mbox{{StarDist}}}}=\mathcal{L}_{\text{\mbox{{StarDist}}}}^{\text{pseudo}}$)
and notice no improvement (F1 $=0.63$). In conclusion, the combination of
minimal annotations and the supporting pseudo ground truth significantly help.

Supported supervision substantially outperforms purely supervised training.
Our proposed supported supervision method can be used as a replacement for
training only on annotations without a performance compromise across all
annotation levels. Notably, at annotation levels $\leq 10\%$ our method
outperforms the baseline substantially.

##  5 Discussion

Figure 7: Qualitative results on a diversity of microscopy image datasets.
Sample raw images from six datasets (Simulated, HU7, Immune, HSC, PSC,
FluoHela) are shown in the first and fifth columns. OCEs are predicted from
patches extracted from these raw images and the spatial arrangement of these
OCEs are shown in the second and sixth columns. During inference, OCEs are
predicted densely for the input raw image (third and seventh columns). The
final cell instance segmentation can be derived from the dense prediction of
OCEs by using a post-processing step such as mean-shift clustering (fourth and
eighth columns).

We believe that this work offers a feasible way to accelerate the analysis of
microscopy image datasets of cells. As our experiments on nine large cell
segmentation datasets demonstrate (see Table 1), a surprisingly good
segmentation can often be achieved in a completely unsupervised fashion.
Depending on the biological question at hand, those results might already be
sufficient for downstream analysis. Furthermore, to obtain more accurate cell
segmentations, the segmentations generated in this unsupervised way can be
used to augment very small amounts of manual labels and thus increase their
efficacy without any additional costs (see Section 4.2). This will in turn
drastically reduce the amount of human effort required to analyze large
microscopy datasets, and provide a rich source of data for more quantitative
and reproducible analyses.

However, we also note some limitations of our method stemming from violated
assumptions: if objects are not randomly distributed (_e.g._ , if cells always
cluster together in pairs), there is no way to tell in a purely unsupervised
manner which structure is to be considered as one instance (either the pair of
cells, or individual cells). Similarly, if the objects in the image do not
resemble many other examples, the proposed method is unlikely to learn a
meaningful object-centric embedding. As such, cells with outlier morphologies
could result in degenerate segmentations. Furthermore, we note that the
proposed method is sensitive to the size of the objects to be segmented,
_i.e._ , the patch size has to be large enough to contain enough information
to predict the relative position of the patch compared to others, but small
enough to not contain entire objects. Although this introduces a hyper-
parameter that has to be adjusted for each dataset, we believe that this is of
little practical relevance since the size of cells in an image can easily be
estimated.

While the work discussed here focuses on segmenting cells in 2D datasets, it
is theoretically feasible to expand this method to 3D and even 4D datasets.
This capability would be particularly useful to biological research, where
cells and tissues are commonly imaged in 3D.

## Acknowledgments

S.W., H.W. and K.M. are supported by the Medical Research Council, as part of
UK Research and Innovation [MCUP1201/23].

## References

  * [1] Assaf Arbelle, Shaked Cohen, and Tammy Riklin Raviv.  Dual-Task ConvLSTM-UNet for Instance Segmentation of Weakly Annotated Microscopy Videos.  IEEE Transactions on Medical Imaging, 2022.
  * [2] Dorin Comaniciu and Peter Meer.  Mean shift: A robust approach toward feature space analysis.  IEEE Transactions on pattern analysis and machine intelligence, 24(5):603–619, 2002.
  * [3] Bert De Brabandere, Davy Neven, and Luc Van Gool.  Semantic Instance Segmentation with a Discriminative Loss Function.  arXiv preprint arXiv:1708.02551, 2017.
  * [4] Nizam Ud Din and Ji Yu.  Training a deep learning model for single-cell segmentation without manual annotation.  Scientific reports, 11(1):1–10, 2021.
  * [5] Carl Doersch, Abhinav Gupta, and Alexei A Efros.  Unsupervised Visual Representation Learning by Context Prediction.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1422–1430, 2015.
  * [6] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised Representation Learning by Predicting Image Rotations.  arXiv preprint arXiv:1803.07728, 2018.
  * [7] Noah F Greenwald, Geneva Miller, Erick Moen, Alex Kong, Adam Kagel, Thomas Dougherty, Christine Camacho Fullaway, Brianna J McIntosh, Ke Xuan Leow, Morgan Sarah Schwartz, et al.  Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning.  Nature Biotechnology, pages 1–11, 2021.
  * [8] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent: A new approach to self-supervised Learning.  Advances in Neural Information Processing Systems, 33:21271–21284, 2020.
  * [9] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum Contrast for Unsupervised Visual Representation Learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
  * [10] Olivier Henaff.  Data-Efficient Image Recognition with Contrastive Predictive Coding.  In International Conference on Machine Learning, pages 4182–4192. PMLR, 2020.
  * [11] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2014.
  * [12] Manan Lalit, Pavel Tomancak, and Florian Jug.  Embedding-based Instance Segmentation in Microscopy.  In Mattias Heinrich, Qi Dou, Marleen de Bruijne, Jan Lellmann, Alexander Schläfer, and Floris Ernst, editors, Proceedings of the Fourth Conference on Medical Imaging with Deep Learning, volume 143 of Proceedings of Machine Learning Research, pages 399–415. PMLR, 07–09 Jul 2021\.
  * [13] Yiwen Li, Gratianus Wesley Putra Data, Yunguan Fu, Yipeng Hu, and Victor Adrian Prisacariu.  Few-shot Semantic Segmentation with Self-supervision from Pseudo-classes.  arXiv preprint arXiv:2110.11742, 2021.
  * [14] Yijie Lin, Yuanbiao Gou, Zitao Liu, Boyun Li, Jiancheng Lv, and Xi Peng.  COMPLETER: Incomplete Multi-view Clustering via Contrastive Prediction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11174–11183, 2021.
  * [15] Katharina Löffler and Ralf Mikut.  EmbedTrack–Simultaneous Cell Segmentation and Tracking Through Learning Offsets and Clustering Bandwidths.  arXiv preprint arXiv:2204.10713, 2022.
  * [16] Davy Neven, Bert De Brabandere, Marc Proesmans, and Luc Van Gool.  Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8837–8845, 2019.
  * [17] Mehdi Noroozi and Paolo Favaro.  Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles.  In European conference on computer vision, pages 69–84. Springer, 2016.
  * [18] Christian Payer, Darko Štern, Marlies Feiner, Horst Bischof, and Martin Urschler.  Segmenting and tracking cell instances with cosine embeddings and recurrent hourglass networks.  Medical Image Analysis, 57:106–119, 2019.
  * [19] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830, 2011.
  * [20] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation, 2015\.
  * [21] Tim Scherr, Katharina Löffler, Moritz Böhland, and Ralf Mikut.  Cell segmentation and tracking using CNN-based distance predictions and a graph-based matching strategy.  PLOS One, 15(12):e0243219, 2020.
  * [22] Uwe Schmidt, Martin Weigert, Coleman Broaddus, and Gene Myers.  Cell detection with star-convex polygons.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 265–273. Springer, 2018.
  * [23] Carsen Stringer, Tim Wang, Michalis Michaelos, and Marius Pachitariu.  Cellpose: a generalist algorithm for cellular segmentation.  Nature Methods, 18(1):100–106, 2021.
  * [24] Aiham Taleb, Winfried Loetzsch, Noel Danz, Julius Severin, Thomas Gaertner, Benjamin Bergner, and Christoph Lippert.  3D Self-Supervised Methods for Medical Imaging.  Advances in Neural Information Processing Systems, 33:18158–18172, 2020.
  * [25] Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency.  Self-supervised Learning from a Multi-view Perspective.  arXiv preprint arXiv:2006.05576, 2020.
  * [26] Vladimír Ulman, Martin Maška, Klas EG Magnusson, Olaf Ronneberger, Carsten Haubold, Nathalie Harder, Pavel Matula, Petr Matula, David Svoboda, Miroslav Radojevic, et al.  An objective comparison of cell-tracking algorithms.  Nature Methods, 14(12):1141–1152, 2017.
  * [27] Steffen Wolf, Fred A. Hamprecht, and Jan Funke.  Instance Separation Emerges from Inpainting, 2020.
  * [28] Xinpeng Xie, Jiawei Chen, Yuexiang Li, Linlin Shen, Kai Ma, and Yefeng Zheng.  Instance-aware Self-supervised Learning for Nuclei Segmentation, 2020\.

##  Appendix A Self-Supervised Training

Architecture. Our self-supervised training requires a network (here referred
to as mini U-Net) with a field of view (FoV) smaller than the expected cell
diameter. Since our analyzed datasets contain cells with diameters as small as
$20$ pixels wide, we use a U-Net architecture [20] with an FoV of only
$16\times 16$. To increase the model’s capabilities without expanding the FoV,
we include additional $1\times 1$ convolutions. Each U-Net block is composed
of a series of [$3\times 3$, $1\times 1$, $1\times 1$, $3\times 3$] valid
convolution layers with ReLU activations. We use a downsampling factor of
$2\times 2$, a depth of $1$ and constant upsampling layers. In the first
layer, we use $64$ feature maps and increase it by a factor of $3$ after each
block.

Training. We train the mini U-Net on batches of $8$ randomly chosen images
with size $252\times 252$ pixels. We use the Adam optimizer [11] with an
initial learning rate of $4e^{-5}$ and train for $50$ epochs, reducing the
learning rate by a factor of $10$ after epochs $20$ and $30$. In our pairwise
loss, defined in Equation 5,

$\begin{split}\mathcal{L}&=\sum_{i,j\in
P}\sigma\left(d(i,j)-\hat{d}(i,j)\right)+\lambda_{\text{reg}}\norm{r(i)}_{2}\text{,}\end{split}$

we use
$\sigma(\delta)=\left(1+\exp(-\frac{\norm{\delta}^{2}_{2}}{\tau})\right)^{-1}$,
$\tau=10$, $\lambda_{\text{reg}}=1e^{-5}$ and reduce the amount of coordinate
pairs to $P$ to reduce the GPU memory footprint. We obtain $P$ by first
sampling $\mathcal{P}_{1}$ as $10$% of all pixels. For every sample in
$p_{1}\in\mathcal{P}_{1}$ we then sample $p_{2}\in\mathcal{P}_{2}$, a random
coordinate within radius $\kappa=10$ of $p_{1}$. In our loss we sample $i,j\in
P=\mathcal{P}_{1}\times\mathcal{P}_{2}$.

##  Appendix B Supervised Training

Architecture. We use the same architecture as the mini U-Net (see Appendix A )
but increase the depth of the network to $3$ which expands the network’s field
of view (FoV).

In conclusion, each U-Net block is composed of a series of [$3\times 3$,
$1\times 1$, $1\times 1$, $3\times 3$] valid convolution layers with ReLU
activations. We use a downsampling factor of $2\times 2$, a depth of $3$ and
constant upsampling layers. In the first layer, we use $64$ feature maps and
increase it by a factor of $3$ after each block.

Training. All models are trained with identical training setups. We optimize
the loss (see $\mathcal{L}_{\text{\mbox{{StarDist}}}}$) with the Adam
optimizer with learning rate $1e^{-5}$ for $200$ epochs, reducing the learning
rate by a factor of $10$ at epochs $30$, $80$ and $160$. We use batches of $8$
images with size $252\times 252$ pixels, sampling pairs of patches within
radius $\kappa=10$ (see Equation 5), and set the loss temperature $\tau=10$.

##  Appendix C HSC Dataset

The HSC dataset is especially challenging - a culture plate is visible in the
background, which causes all evaluated models to predict additional object
instances near the border of the plate.

We compute scores on a range of IoU thresholds to investigate robustness of
evaluated methods. At the matching IoU threshold of $0.5$, we obtain F1 scores
of $0.00$ for the pre-trained Cellpose model, $0.09$ for the pre-trained
StarDist model and $0.06$ for Cellulus (ours) (see Table 3). Additionally for
the same matching IoU threshold, we obtain Recall scores of $0.01$ for
Cellpose, $0.26$ for StarDist and $0.55$ for Cellulus. On the HSC dataset,
Cellulus is the most sensitive at detecting cells, but performs less favorably
with respect to F1 and Accuracy metrics. The high Recall scores obtained with
Cellulus also explains the high SEG scores where false-positive predictions
are not heavily penalized. Qualitative results on the HSC dataset can be seen
in Figure 7.

Table 3: Quantitative results on the HSC dataset in the Cell Tracking
Challenge [26], for selected matching IoU thresholds (fully unsupervised
setting). Pre-trained Cellpose and StarDist baseline models are compared with
Cellulus (ours). Best performing method on each threshold and metric, is shown
in bold.

| Accuracy | F1 | Recall
---|---|---|---
Threshold | Cellpose | StarDist | Cellulus | Cellpose | StarDist | Cellulus | Cellpose | StarDist | Cellulus
$0.1$ | 0.68 | $0.22$ | $0.06$ | 0.81 | $0.36$ | $0.11$ | $0.92$ | $0.98$ | 0.99
$0.2$ | 0.59 | $0.21$ | $0.06$ | 0.74 | $0.35$ | $0.11$ | $0.84$ | 0.97 | $0.96$
$0.3$ | 0.27 | $0.18$ | $0.05$ | 0.43 | $0.30$ | $0.10$ | $0.51$ | $0.83$ | 0.86
$0.4$ | $0.03$ | 0.11 | $0.04$ | $0.06$ | 0.20 | $0.08$ | $0.07$ | $0.55$ | 0.72
$0.5$ | $0.00$ | 0.05 | $0.03$ | $0.00$ | 0.09 | $0.06$ | $0.01$ | $0.26$ | 0.55
$0.6$ | $0.00$ | $0.01$ | 0.02 | $0.00$ | $0.03$ | 0.05 | $0.00$ | $0.07$ | 0.39
$0.7$ | $0.00$ | $0.00$ | 0.01 | $0.00$ | $0.00$ | 0.02 | $0.00$ | $0.01$ | 0.13
$0.8$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | 0.01
$0.9$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$

##  Appendix D Simulated Dataset

Figure 8: Qualitative results on the Simulated dataset in the Cell Tracking
Challenge [26]. The Simulated dataset comprises of two time-lapse videos
(Videos $01$ and $02$) which contain $64$ and $149$ image frames respectively.
Shown here are individual raw images from the two videos (first column),
predicted instance segmentations obtained using the pre-trained baseline
models StarDist (second column) and Cellpose (third column), dense prediction
of Object-Centric Embeddings (OCEs) obtained using Cellulus (fourth column),
intermediate instance segmentations obtained by applying mean-shift (MS)
clustering on the dense OCEs (fifth column), these intermediate instance
segmentations are further post-processed (sixth column, see more details in
the Section 5 \- Scale Informs All Parameter Choices) and the Ground Truth
Instance Segmentation available for evaluation purposes (seventh column).
Video $02$ in Simulated (see last two rows) contains cells with visible
granules which cause over-segmentation for both the evaluated, pre-trained
baseline models.


# Stride and translation invariance in CNNs

Coenraad Mouton, Johannes C. Myburgh, Marelie H. Davel
Multilingual Speech Technologies, North-West University, South Africa; and
CAIR, South Africa.
{moutoncoenraad<EMAIL_ADDRESS>

###### Abstract

Convolutional Neural Networks have become the standard for image
classification tasks, however, these architectures are not invariant to
translations of the input image. This lack of invariance is attributed to the
use of stride which ignores the sampling theorem, and fully connected layers
which lack spatial reasoning. We show that stride can greatly benefit
translation invariance given that it is combined with sufficient similarity
between neighbouring pixels, a characteristic which we refer to as local
homogeneity. We also observe that this characteristic is dataset-specific and
dictates the relationship between pooling kernel size and stride required for
translation invariance. Furthermore we find that a trade-off exists between
generalization and translation invariance in the case of pooling kernel size,
as larger kernel sizes lead to better invariance but poorer generalization.
Finally we explore the efficacy of other solutions proposed, namely global
average pooling, anti-aliasing, and data augmentation, both empirically and
through the lens of local homogeneity.

This is a preprint - the final authenticated publication is available online
at https://doi.org/10.1007/978-3-030-66151-9_17

_K_ eywords Translation invariance $\cdot$ Subsampling $\cdot$ Convolutional
Neural Network $\cdot$ Local homogeneity

##  1 Introduction

Traditional computer vision tasks such as image classification and object
detection have been revolutionized by the use of Convolutional Neural networks
(CNNs) [1]. CNNs are often assumed to be translation invariant, that is,
classification ability is not influenced by shifts of the input image. This is
a desirable characteristic for image recognition, as a specific object or
image must be correctly identified regardless of its location within the
canvas area. The assumption that CNNs exhibit translation invariance, however,
has been shown to be erroneous by multiple authors [2, 3, 4], who all show
that shifts of the input image can drastically alter network classification
accuracy. Correcting for image translation within CNNs is an active area of
study which has been partially addressed by methods such Spatial Transformer
Networks [5], Anti-Aliasing [3], and Global Average Pooling [2].

In this work, we provide an overview of the properties that are required for
translation invariance and show why they generally do not hold in the case of
CNNs. Furthermore we show that subsampling, commonly referred to as "stride"
[6], can greatly benefit translation invariance. However, we find that
subsampling is only beneficial given that it is combined with sufficient
similarity between neighbouring pixels, a characteristic that we refer to as
local homogeneity.

We empirically explore the relation between local homogeneity and subsampling
by measuring the effects of varying pooling kernel size and stride on both
invariance and generalization, repeated across several different architectures
and datasets. We show that the inter-pixel variance of a given dataset
dictates the necessary kernel size to ensure a degree of translation
invariance. In addition we examine the efficacy of solutions proposed by other
authors, namely global average pooling, anti-aliasing, and data augmentation
in terms of local homogeneity, translation invariance, and generalization.

Taken together, we clarify the various aspects that influence translation
invariance in CNNs in order to better understand how certain architectural
choices affect both invariance and generalization, and we further measure
these effects to see if our hypotheses hold true.

##  2 Background

In order to understand translation invariance, we must first define the terms
that play a role and their relation to one another.

###  2.1 Translation Invariance and Equivariance

Convolutional neural networks make use of convolution and pooling operators
which are inherently translational, as filter kernels are shifted over an
image to provide an output, commonly referred to as a feature map [6]. The
translational nature of CNNs leads to the erroneous assumption that these
systems are invariant to translations of the input image, or that the spatial
location of internal features are irrelevant for classification. However,
Azulay and Weiss [2] show that even for state of the art CNN architectures
(VGG16, ResNet50, InceptionResNetV2), a single pixel shift of the input image
can cause a severe change in the prediction confidence of the network.
Surprisingly the shift of the input is imperceptible to the human eye, but
drastically impedes the network’s ability to classify.

Translation invariance can be described as follows (as adapted from [7]):

######  Definition 1.

A function f can be said to be invariant to a group of translations G if for
any g element of G:

$f(g(I))=f(I)$ (1)

This implies that g has no effect on the output of function f, and the result
remains equal.

A second misconception is that whilst modern CNNs are not translation
invariant, they are translation equivariant. Translation equivariance (also
referred to as covariance by some authors) is the property by which internal
feature maps are shifted in a one-to-one ratio along with shifts of the input.
We define translation equivariance as follows (as adapted from [7]):

######  Definition 2.

A function f is equivariant to the translations of group G if for any g
element of G:

$f(g(I))=g(f(I))$ (2)

This implies that the output is shifted in accordance with the shift of the
input, or in other terms that the output of the function f can be translated
to produce the same result as translating the input I before f is applied
would.

Intuitively it would be expected that translation equivariance holds for both
convolution and pooling layers, and this intuition proves correct for dense
convolution and pooling, if edge effects are ignored. To illustrate this
property, an arbitrary one-dimensional filter is applied to a one-dimensional
input, as well as to shifted values of this input. Consider an input signal
$I[n]=[0,0,0,0,1,2,0,0,0,0]$ and a kernel $K[n]=[1,0,1]$, the result of the
convolution $I[n]\circledast K[n]$ is shown in the second column of Table 1.

Table 1: Dense and strided convolution of a one-dimensional input and shifted variants Input | Dense Convolution | Strided Convolution
---|---|---
[0,0,0,0,1,2,0,0,0,0] | [0,0,1,2,1,2,0,0] | [0,1,1,0]
[0,0,0,0,0,1,2,0,0,0] | [0,0,0,1,2,1,2,0] | [0,0,2,2]
[0,0,0,0,0,0,1,2,0,0] | [0,0,0,0,1,2,1,2] | [0,0,1,1]

As per this example, the equivariance property holds, as a shift of the input
results in an equal shift of the output, meaning $f(g(I))=g(f(I))$. However,
this intuition fails when considering subsampling.

###  2.2 Subsampling

In CNNs subsampling occurs when a convolution or pooling layer is used with a
kernel stride greater than one. Using strided filters results in intermediary
samples present in the input being skipped over and disregarded. Strided
filters are widely used in state-of-the-art architectures, which has both
benefits and drawbacks.

We first consider the disadvantages of subsampling. Azulay and Weiss [2]
correctly show that subsampling breaks translation equivariance, as the act of
subsampling ignores the Nyquist-Shannon sampling theorem [8]. This theorem
states that for a continuous signal to be fully reconstructed after sampling
the sampling rate must be at least twice that of the highest frequency present
in the input signal. In the case of CNNs, a digital input is sampled at a rate
dictated by the kernel stride, implying that information is disregarded. As
information is lost, shifts of the input are not guaranteed to result in
equivalent responses.

This effect can be shown by using the previous example and a stride of 2 when
calculating the convolution (thus a subsampling factor of 2) as the third
column of Table 1 illustrates.

Two important characteristics of stride are illustrated by this example:

  1. 1.

Signal information is lost. Compared to the output of dense convolution, it is
clear that subsampling disregards intermediary values. It is logical that this
property can have an adverse effect on a CNN’s ability to detect features in
an image, as translating the input signal can cause features present in the
input not to line up with the stride of the kernel.

  2. 2.

Translation Equivariance is lost. The output of the convolution operation is
no longer shifted in a one-to-one ratio with the input, meaning $f(g(I))\neq
g(f(I))$, as can be seen by comparing the output of $I[n]\circledast K[n]$ to
that of $I[n-1]\circledast K[n]$. This implies that shifts of the input signal
result in outputs that are not equivalent. The result of the strided filtering
of a translated input is not guaranteed to be sufficiently similar to that of
its untranslated counterpart if equivariance does not hold; this partly
explains why CNNs are so susceptible to small shifts.

The main benefit of subsampling is that it can greatly reduce the training
time of CNNs. As He and Sun [9] show, the time complexity of convolution
layers in a CNN is given by:

$O(\sum_{l=1}^{d}n_{l-1}\cdot s_{l}^{2}\cdot n_{l}\cdot m_{l}^{2})$ (3)

where $l$ is the index of the convolutional layer, $d$ is the number of
convolutional layers, and for any layer $l$, $n_{l}$ is the number of output
channels, $s_{l}$ the spatial size of the filter, and $m_{l}$ the size of the
output feature map. The subsequent output size of any given layer has a large
effect, given that $m_{l}$ is squared. It can also be shown that the output
width of a layer $W_{l}$ given the previous layer width $W_{l-1}$ is specified
by:

$W_{l}=\frac{W_{l-1}-k_{w}+2p}{s}+1$ (4)

where $k_{w}$ is the kernel width, p is padding, and s is stride (the same
equation holds for height). Given that stride acts as a divisor, we conclude
that subsampling drastically reduces the size of the output, and in doing so
substantially reduces the time complexity. Furthermore, through spatial
reduction, the size of the input layer to the fully connected layers is
greatly reduced, implying there are less learn-able parameters which further
reduces training time and memory consumption [9].

Whilst subsampling reduces spatial dimensions, it is worth noting that this
can also be achieved by downsampling in CNNs. We define downsampling as the
reduction in spatial size caused by the size of a kernel during a
convolution/pooling operation, where information along the edges of an input
is disregarded (commonly referred to as “edge effects"). Conversely,
subsampling causes spatial reduction by explicitly disregarding intermediary
samples. Whilst downsampling does have an effect on translation invariance
[10], we attempt to mitigate this effect through the use of adequate padding.

##  3 Signal Movement, Signal Similarity, and Local Homogeneity

In this section we show how subsampling plays a role in terms of translation
equivariance, and which characteristics must be accounted for when it is
present.

###  3.1 Shiftability

Whilst subsampling breaks the equivariance property, we propose that it can
greatly benefit translation invariance under certain circumstances due to a
third property we define as shiftability. Shiftability holds for systems that
make use of subsampling and is defined as follows:

######  Definition 3.

A function f with subsampling factor s is shiftable for a given translation if

$f(g(I))=g^{\prime}(f(I))$ (5)

where g is a translation function with translation vector $\vec{u}$ and input
X

$g(X)=t(\vec{u},X)$ (6)

and

$g^{\prime}(X)=t(\frac{\vec{u}}{s},X)$ (7)

Put otherwise, shiftability holds for translations that are factors of the
subsampling factor s of f. When subsampling shifted inputs, equivalence will
hold if a given translation is in accordance with the stride. To illustrate
this property, consider an arbitrary input signal that is subsampled by a
factor of four and various shifts of the signal, as shown in Table 2.

Table 2: Subsampling factor of four for an arbitrary input Input | 0 0 0 0 0 0 0 0 0 3 2 5 2 4 1 6 3 4 6 5 5 0 0 0 0 0 0 0 0 0
---|---
Shift | Subsampled Output
0 | 0 0 0 2 3 5 0 0
1 | 0 0 0 5 6 5 0 0
2 | 0 0 0 2 1 6 0 0
3 | 0 0 0 3 4 4 0 0
4 | 0 0 0 0 2 3 5 0
5 | 0 0 0 0 5 6 5 0
6 | 0 0 0 0 2 1 6 0
7 | 0 0 0 0 3 4 4 0
8 | 0 0 0 0 0 2 3 5

In this example, shiftability holds for translations that are factors of the
subsampling factor (shifts of 4 and 8), and so a scaled form of equivariance
is kept. It is further evident that subsampling scales shifts of the input
signal: in this example, a shift of four in the input results in only a shift
of one in the output.

The subsampling factor dictates how many versions of the output signal can
potentially exist after translation (again ignoring edge effects): In this
example four discrete outputs are present, where all other outputs are merely
shifted variants. In the case of two-dimensional filtering, inputs are
subsampled both vertically and horizontally, meaning $s^{2}$ output signals
can exist given a single input and a bounded translation. This further implies
that a given input will only be shiftable for $\frac{1}{s^{2}}$ of possible
translations [2].

To explain how shiftability benefits translation invariance, we must first
define two distinct characteristics that must be accounted for when comparing
outputs of translated inputs to that of untranslated inputs.

###  3.2 Signal Similarity and Signal Movement

We propose two characteristics that influence translation invariance when
subsampling is present:

  * •

Signal Similarity: How much of the untranslated signal’s output information is
preserved after translation.

  * •

Signal Movement: How far the translated output has been moved from the
original position of the untranslated output.

As an example, consider an output signal of [0,0,1,2,0,0] and another of
[0,0,0,1,2,0]. These signals are exactly similar except that the second is
shifted by one. Conversely, an input of [0,0,1,2,0,0] and [0,0,2,3,0,0] are
not exactly similar, but do not exhibit any shift (signal movement).

Due to the shiftability property it is evident that subsampling reduces signal
movement, but leads to the loss of signal similarity, conversely using no
subsampling leads to perfect signal similarity but allows the output to shift
in a one-to-one ratio with the input. For translation invariance in CNNs
signal similarity must be kept but signal movement must also be reduced to the
extent possible. Whilst the reduction of signal movement during translation is
fully dependent upon the subsampling factor, we propose that the degree of
signal similarity that is preserved during subsampling is dependent upon local
homogeneity.

###  3.3 Local Homogeneity

As subsampling disregards intermediary pixels in a given feature map,
translated versions will results in more equivalent outputs if neighbouring
pixels are more similar to each other in a given region. Put otherwise,
preserving signal similarity requires that the variance between intermediary
elements in a given window are sufficiently low enough, the size of which is
dictated by the subsampling factor. We call this property local homogeneity.

To illustrate the effect of local homogeneity, consider an input that is fully
locally homogeneous in accordance with the subsampling factor and the
resulting output when shifting the signal. This is shown in Table 3 where a
subsampling factor of 2 is used.

Table 3: Subsampling of a locally homogeneous signal Input | 0 0 0 0 0 0 2 2 3 3 1 1 2 2 0 0 0 0 0 0
---|---
Shift | Subsampled Output
0 | 0 0 0 2 3 1 2 0 0
1 | 0 0 0 2 3 1 2 0 0
2 | 0 0 0 0 2 3 1 2 0
3 | 0 0 0 0 2 3 1 2 0
4 | 0 0 0 0 0 2 3 1 2

Given that successive elements are similar in accordance with the subsampling
factor, subsampling results in equivalent responses, yet the initial shift of
the input is also scaled; implying that signal similarity is preserved and
signal movement is negated. Our hypothesis surrounding local homogeneity
brings about two logical conclusions:

  * •

Pooling improves local homogeneity: Dense pooling (max and average pooling
alike) reduces the variance of any given input, therefore providing more
similarity between neighbouring pixels.

  * •

Strided pooling can benefit translation invariance: Combining a pooling window
of sufficient size (and therefore sufficient local homogeneity) with
subsampling leads to the reduction of signal movement. This implies that
translated samples will result in outputs that are more equivalent to
untranslated samples.

In the following section we empirically explore our hypothesis surrounding
local homogeneity by observing the effects of stride and pooling on
translation invariance, as well as solutions proposed by other authors.

##  4 Analysis

In this section we empirically measure the translation invariance of different
architectures and explore how these results relate to local homogeneity.

###  4.1 Experimental Setup

####  4.1.1 Training

Two datasets are used for analysis, namely MNIST [11] and CIFAR10 [12]. For
prepossessing, MNIST networks are zero padded by 6 (meaning 6 rows of zeros
are added to every edge of the image) and 10 for CIFAR10, resulting in 40x40
and 52x52 sized images respectively. This allows space on the image canvas for
translation.

We use cross-entropy loss and ReLU activation functions, along with the Adam
optimizer [13]. Batch size is kept constant at 128 for MNIST networks, and a
smaller batch size of 64 is used for CIFAR10 due to a lack of video memory to
train exceptionally large networks. Validation set size is constant at 5 000
for both MNIST and CIFAR10.

All MNIST networks are trained for a minimum of 100 epochs and 200 epochs for
CIFAR10; furthermore, if a network has shown improvement in validation
accuracy within the last 10 epochs an additional 15 epochs are added to
training. Step-wise learning rate decay is also used and the starting learning
rate for a group of networks is chosen empirically, within the range of 0.001
to 0.0001. It is confirmed that all networks are trained to 100% train
accuracy.

Post-training, the epoch exhibiting the highest validation accuracy is chosen
to ensure the best generalization possible for the specific architecture.
Apart from early stopping, we do not make use of use any explicit
regularization methods such as dropout or batch-norm.

In terms of architectural choices, all convolution/pooling layers use padding
to adjust for the downsampling caused by the size of the kernel, but not to
mitigate the effects of subsampling, according to the following equation:

$p=\lfloor\frac{k_{w}-1}{2}\rfloor$ (8)

We use a standard three-layer architecture for both datasets, with 128
channels on the final layer and 3x3 dense convolution kernels, whilst pooling
size and stride is varied for each experiment. This architecture is useful as
it performs relatively well on both datasets, given that no explicit
regularization is used, but is still simple enough to interpret any subsequent
results.

####  4.1.2 Metrics

We measure translation invariance across different architectures by comparing
the activation value vectors at the network output layer of an original and
translated sample. Two measurements are used:

  1. 1.

Mean Cosine Similarity (MCS): As defined in Equation 9, MCS uses the angle
between any two vectors $a$ and $b$ as a similarity measure. Since only the
angle is used, differences in vector magnitudes are ignored.

$cos(\theta)=\frac{a\cdotp b}{\|a\|\|b\|}$ (9)

  2. 2.

Probability of top 1 change (PTop1): The probability of the top class
prediction of a given network changing after an image is translated, as
originally proposed by Azulay and Weiss [2]. This allows us to determine an
exact probability of a sample being incorrectly classified given a range of
translation. This is useful as it purely measures a change in prediction
accuracy and does not concern itself with other secondary effects.

####  4.1.3 Method of comparison

For each measurement, we report on the average value of the metric, across all
test samples that are translated according to randomly sampling from a
specified range, for both vertical and horizontal translation. This results in
a single scalar value specifying the network’s translation invariance for a
given maximum range of translation. Furthermore, when comparing several
networks, only the samples that are correctly classified (before translation)
by all networks present in the comparison are used. Finally, all results are
averaged over three training seeds. For results displayed with graphs, we
include error bars indicating the standard error.

###  4.2 Strided pooling and translation invariance

We explore the effects of strided pooling on translation invariance, using an
architecture without subsampling as baseline. Three other networks are trained
with subsampling factors of 2, 4, and 8 respectively for the MNIST dataset.
The subsampling factor is varied by consecutively setting the stride of the
2x2 max pooling filters in the network to 2, starting at the first layer. The
comparative MCS is shown in Figure 1.

Figure 1: MCS comparison for MNIST architectures with varying subsampling

This result shows that subsampling improves translation invariance for MNIST,
where networks using subsampling show a substantially higher MCS. This further
implies that 2x2 max pooling is sufficient to provide local homogeneity given
a stride of 2 for each layer, however samples in the MNIST dataset are
generally inherently highly homogeneous.

CIFAR10 is a more complex and detailed dataset, and is therefore generally
less homogeneous and would require a larger degree of filtering to ensure
translation invariance. To ascertain the required max pooling size for this
dataset, the previous experiment is repeated for kernel sizes ranging from 2x2
to 5x5 for each layer, as shown in Table 4.

Table 4: MCS for CIFAR10 networks with varying subsampling and max pooling kernel sizes (10 Pixel Range) Subsampling Factor | Kernel Size
---|---
| 2x2 | 3x3 | 4x4 | 5x5
1 | 0.630 | 0.598 | 0.595 | 0.618
2 | 0.554 | 0.635 | 0.683 | 0.731
4 | 0.622 | 0.674 | 0.759 | 0.789
8 | 0.610 | 0.660 | 0.762 | 0.791

We observe that in the case of CIFAR10 2x2 max pooling is not sufficient, and
a substantial increase in translation invariance following subsampling is only
observed at 3x3 pooling and larger.

For networks that make use of subsampling, we observe that larger kernel sizes
always result in greater invariance. Conversely, for networks that do not make
use of subsampling (the first row of Table 4) we observe a a significant
decrease in translation invariance as kernel-size is increased. Intuitively
one would expect larger kernels to always provide greater translation
invariance, but this intuition fails since these networks are fully
translation equivariant. Finally, we observe that greater subsampling always
results in greater invariance when adequately sized kernels are used (as in
the case of 4x4 and 5x5 pooling) which are aligned with our findings on MNIST.

These results support our proposal that stride can significantly increase the
translation invariance of a network, given that is combined with sufficient
local homogeneity. Furthermore we also find that the inherent homogeneity of a
given dataset dictates the required filtering for subsampling to be effective.

###  4.3 Anti-aliasing

In terms of ensuring local homogeneity, average pooling outperforms that of
max pooling, as an averaging kernel acts as an anti-aliasing (blurring) filter
which greatly benefits translation invariance during subsampling. However,
Scherer et al. [14] show that max pooling results in better generalization
than that of average pooling when used in CNNs.

Zhang [3] proposes a solution to this problem which allows the generalization
benefits of max pooling without compromising translation invariance. The
author alters strided max pooling by separating it into two distinct layers:
(1) Dense Max Pooling, and (2) Strided Anti-Aliasing. By applying an anti-
aliasing filter, local homogeneity is ensured and the subsequent subsampling
operation’s affect on signal similarity is strongly mitigated, which results
in a more translation invariant network.

The efficacy of this method is explored for both the MNIST and CIFAR10
datasets using the three layer 2x2 pooling networks from the previous section.
Each pooling layer present in the network is replaced with a dense max pooling
layer and a bin-5 anti-aliasing filter. These networks are then compared to
their baseline counterparts that do not make use of anti-aliasing. The
comparative MCS for a maximum shift of 10 pixels is shown in Table 5.

Table 5: Mean Cosine Similarity for MNIST and CIFAR10 networks with and without anti-aliasing for a maximum shift of 10 pixels Subsampling Factor | AA | MNIST | CIFAR
---|---|---|---
1 | No | 0.248 | 0.630
Yes | 0.329 | 0.518
4 | No | 0.383 | 0.620
Yes | 0.654 | 0.710
8 | No | 0.447 | 0.611
Yes | 0.638 | 0.690

For MNIST, anti-aliasing seems to always provide better translation invariance
regardless of whether subsampling is used or not. However, the greatest
increase in translation invariance occurs when subsampling is applied, as it
is not solely anti-aliasing that provides invariance, but its combination with
stride. For CIFAR10 the results are slightly different, anti-aliasing greatly
reduces the invariance of the architecture without subsampling. However, as is
the case with MNIST, a large increase is evident when it is combined with
subsampling.

In conclusion, these results confirm that both signal similarity must be
preserved and signal movement must be reduced to increase the network’s
invariance to translation, and that anti-aliasing is an effective solution for
ensuring local homogeneity.

###  4.4 No Subsampling and Global Average Pooling

Whilst we have shown that the use of subsampling can greatly benefit
translation invariance, Azulay and Weiss [2] propose a different approach that
makes use of Global Average Pooling (GAP) and avoiding any subsampling
throughout the network. Global pooling is not influenced by signal movement,
and with no subsampling, equivariance is kept, resulting in a perfectly
translation invariant system.

We verify this by adding a final global average pooling layer to our baseline
model without subsampling for CIFAR10, and we find that it has a 0% Ptop1
change for shifts within the canvas area. Put otherwise, the system is
completely translation invariant.

Although this might seem to be a complete solution, GAP is not without its
drawbacks. Ignoring the benefits of subsampling, the GAP operation disregards
a tremendous amount of information and could lower the classification ability
of a given architecture, and is therefore not necessarily a suitable solution
for every dataset. However, Fully Convolutional Neural Networks (FCNNs) do
make use of GAP, and withholding the use of subsampling in these architectures
could be a suitable solution for ensuring translation invariance.

###  4.5 Learned Invariance

We explore the effects of learned invariance by training our previous MNIST
architectures of Figure 1 on a translated data set. We keep the size of the
train set constant and randomly translate each sample up to a maximum of 8
pixels, furthermore we apply the same translation to the validation set (the
test set remains untranslated). In this way we explicitly optimize our models
for translation invariance, and then examine whether our previous findings
still hold true. The comparative MCS of these networks is shown in Figure 2.

Figure 2: MCS comparison for MNIST architectures with data translation

Observing this result, we find that the same pattern emerges as that of Figure
1, where greater subsampling leads to greater translation invariance. However,
these networks are much more translation invariant than those not trained on
translated data, with the lowest MCS at a staggering 0.97.

Whilst learned invariance is certainly a powerful tool, Azulay and Weiss point
out that this method can potentially result in models that are overly biased
to translations of the train set and it can not be expected to generalize well
to translations of unseen data in all cases. We also point out that MNIST is a
particularly easy problem compared to more complex datasets such as CIFAR100
or ImageNet [15], and usually data augmentation would be required for these
networks to achieve good performance. This implies that the training set must
be explicitly augmented with translated data, which leads to a substantial
increase in training time.

###  4.6 Translation Invariance and Generalization

We explore the relationship between translation invariance and generalization
by comparing the test accuracy of our architectures to their MCS. The test
accuracy of our CIFAR10 networks of varying kernel size and subsampling from
Table 4 is shown in Table 6.

Table 6: Test accuracy for CIFAR10 networks with varying subsampling and kernel size Subsampling Factor | Kernel Size
---|---
| 2x2 | 3x3 | 4x4 | 5x5
1 | 72.33 | 75.00 | 76.10 | 76.00
2 | 74.43 | 77.00 | 77.57 | 76.69
4 | 73.94 | 76.72 | 77.25 | 76.76
8 | 72.53 | 75.31 | 76.69 | 75.95

We observe that larger kernel sizes generally generalize better, but also that
kernels that are too large (such as 5x5 in this case) lead to a reduction in
test set accuracy. This is an expected result - larger kernels lower the
variance of a given sample and result in more locally homogeneous regions, but
also implies that more information is disregarded which negatively impacts the
model’s ability to generalize to samples not seen during training. Similarly,
some subsampling seems to always provide better generalization regardless of
kernel size, but too much subsampling leads to a reduction in model
performance. These differences suggest that there is a slight trade-off
between a model’s inherent invariance to translation and its generalization
ability.

For the anti-aliased models of section 4.3 we observe a very small overall
effect on generalization: Table 7 shows the test accuracy of these models with
and without the use of anti-aliasing filters.

Table 7: MNIST and CIFAR10 test accuracy with and without anti-aliasing Subsampling Factor | AA | MNIST | CIFAR10
---|---|---|---
1 | No | 99.36 | 72.33
Yes | 99.18 | 73.62
4 | No | 99.38 | 73.94
Yes | 99.35 | 74.71
8 | No | 99.3 | 72.53
Yes | 99.21 | 73.40

The combination of subsampling with anti-aliasing actually improves
generalization for the CIFAR10 dataset, and only slightly hampers accuracy for
that of MNIST. These results are aligned with that of Zhang which show a
slight improvement in generalization for state-of-the-art ImageNet networks
using anti-aliasing. These results, along with those of Section 4.3, show that
for these data sets anti-aliasing is effective at improving translation
invariance without reducing generalization ability.

##  5 Conclusion

We investigated the effect of stride and filtering on translation invariance
and generalization in CNNs. Our main findings are summarised below:

  * •

Subsampling can greatly benefit translation invariance, given that it is
combined with local homogeneity. If sufficient filtering is used, greater
subsampling leads to greater translation invariance.

  * •

The amount of filtering required depends on the inherent homogeneity of a
given data set.

  * •

Too much filtering or subsampling negatively affects generalization, as such,
a trade-off exists between translation invariance and generalization.

  * •

Anti-Aliasing performs well in ensuring local homogeneity and therefore
greatly increases an architecture’s invariance to translation; it also
performs better than large max pooling kernels in terms of generalization.

  * •

We find that data translation, or learned invariance, is very effective, but
can not be expected to perform well in all cases.

In terms of further research, we do not yet measure the effects of strided
convolution, which could also be beneficial to translation invariance, nor do
we quantify the effects of downsampling. Along with this, several other
methods that have been proposed, such as Spatial Transformer Networks [5] and
Capsule Networks [16] also require further exploration.

## References

  * [1] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel.  Backpropagation applied to handwritten zip code recognition.  Neural Computation, 1(4):541–551, 1989.
  * [2] Aharon Azulay and Yair Weiss.  Why do deep convolutional networks generalize so poorly to small image transformations?  CoRR, abs/1805.12177, 2018.
  * [3] Richard Zhang.  Making convolutional networks shift-invariant again.  CoRR, abs/1904.11486, 2019.
  * [4] Eric Kauderer-Abrams.  Quantifying translation-invariance in convolutional neural networks.  CoRR, abs/1801.01450, 2018.
  * [5] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu.  Spatial transformer networks.  CoRR, abs/1506.02025, 2015.
  * [6] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep Learning.  MIT Press, 2016.  http://www.deeplearningbook.org.
  * [7] Karel Lenc and Andrea Vedaldi.  Understanding image representations by measuring their equivariance and equivalence.  CoRR, abs/1411.5908, 2014.
  * [8] H. Nyquist.  Certain topics in telegraph transmission theory.  Transactions of the American Institute of Electrical Engineers, 47(2):617–644, 1928.
  * [9] Kaiming He and Jian Sun.  Convolutional neural networks at constrained time cost.  CoRR, abs/1412.1710, 2014.
  * [10] Osman Semih Kayhan and Jan C. van Gemert.  On translation invariance in cnns: Convolutional layers can exploit absolute spatial location, 2020.
  * [11] Y. LeCun, C Cortes, and C.J.C Burges.  MNIST database of handwritten digits.  http://yann.lecun.com/exdb/mnist/.
  * [12] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  University of Toronto, 05 2012.
  * [13] Diederik Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  International Conference on Learning Representations, 12 2014.
  * [14] Dominik Scherer, Andreas C. Müller, and Sven Behnke.  Evaluation of pooling operations in convolutional architectures for object recognition.  In ICANN, 2010.
  * [15] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 1097–1105. Curran Associates, Inc., 2012.
  * [16] Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton.  Dynamic routing between capsules.  CoRR, abs/1710.09829, 2017.