# Magnifying Networks for Images with Billions of Pixels

Neofytos Dimitriou Ognjen Arandjelović
University of St Andrews
KY16 9SX  United Kingdom
<EMAIL_ADDRESS>
###### Abstract

The shift towards end-to-end deep learning has brought unprecedented advances
in many areas of computer vision. However, there are cases where the input
images are excessively large, deeming end-to-end approaches impossible. In
this paper, we introduce a new network, the Magnifying Network (MagNet), which
can be trained end-to-end independently of the input image size. MagNets
combine convolutional neural networks with differentiable spatial
transformers, in a new way, to navigate and successfully learn from images
with billions of pixels. Drawing inspiration from the magnifying nature of an
ordinary brightfield microscope, a MagNet processes a downsampled version of
an image, and _without supervision_ learns how to identify areas that may
carry value to the task at hand, upsamples them, and recursively repeats this
process on each of the extracted patches. Our results on the publicly
available Camelyon16 and Camelyon17 datasets first corroborate to the
effectiveness of MagNets and the proposed optimization framework and second,
demonstrate the advantage of Magnets’ built-in transparency, an attribute of
utmost importance for critical processes such as medical diagnosis.

##  1 Introduction

Figure 1: An Illustration of the architecture of MagNet. The shown model
consists of 3 magnifying layers and a classification layer.

Convolutional neural networks (CNNs) have been the major drive behind the
paradigm shift of computer vision towards deep learning. Over the years, the
research community has come up with a cornucopia of convolution-based network
topologies with differences in their connectivity patterns, training
procedures, and building blocks [32, 15, 34, 39, 17, 6, 24, 18, 30]. As a
consequence of the continuous improvements and the widespread availability of
annotated data [9, 21, 23, 12, 26, 1, 8], end-to-end trained CNNs have
dominated the area of visual object recognition.

In parallel with the increase in data availability, hardware advancements have
created the possibility of both the capturing and storing of much larger
images. One of the most extreme yet practically important examples can be
found in Digital Pathology and in particular, in the task of analysing so-
called whole slide images (WSIs) [10, 4] – digitized microscope slides which
are often several gigabyte in size with a typical resolution of $100000\times
100000$ pixels [38, 10]. Conventional neural network optimization using such
images is practically infeasible considering the associated memory and compute
requirements.

In this paper, we introduce a new family of neural networks, henceforth
referred to as Magnifying Networks (MagNets), that can be optimized end-to-end
even when the input images have billions of pixels. MagNets use an attention
based mechanism to decide on a course to fine basis when to analyse
increasingly finer-grain detail as a way of processing extremely large images
(n.b. incidentally, this is conceptually similar to a pathologist’s knowledge
and attention based use of magnification with a brightfield miscroscope). A
microscope has multiple magnification settings that enable the user to view a
specimen at different scales. Starting at the lowest magnification setting,
the whole specimen can be observed. As the magnification is increased, the
specimen is proportionally magnified, thereby providing finer detail while at
the same time, a smaller part of the specimen is displayed. During a visual
examination, the clinician finds areas of interest at lower magnification
levels and then examines them further at higher and higher magnification
levels, accruing in the process information from all magnification levels.
Similarly, a MagNet starts at the lowest magnification level and recursively
identifies, magnifies and analyses areas of interest with more fine-grain
detail (see Figure 1). More importantly, as we show in our experiments,
MagNets can be optimized without extra supervision (e.g. boundary boxes) for
the task at hand.

MagNets provide a new way of solving both the “where” and “what” problems of
gigapixel image analysis in an end-to-end fashion [10]. Given the innate
transparency, no preprocessing requirements, and ability to perform both
localization and classification tasks with no additional information (only
slide-level information used [10]), it would be no exaggeration to say that
MagNets have the potential to revolutionize fields, such as Digital Pathology.

The rest of the paper is organised as follows. In Section 2 we provide further
context for gigapixel image analysis, and discuss related work. In Section 3,
we introduce the key novelty of the present work, that is, explain the
technical underpinnings of MagNets. In Section 4, we provide more details on
the data, as well as the optimization framework. Finally, in Section 5, we
discuss our results, and conclude our work with Section 6.

##  2 Related Work

Pathology is embracing the era of digitisation and, with the widespread
adoption of whole slide scanners, has become a research area which stands to
benefit greatly from advances in gigapixel image analysis on the one hand, and
which presents novel challenges to computer vision and deep learning
techniques, on the other.

Figure 2: An illustration of a single magnifying layer that outputs two
patches. The convolutional layers are independent between the two patches.
Note that if this was the last magnifying layer, the image size of the patches
would have been $224\times 224$.

##  3 Magnifying Networks

Large images typically require further preprocessing for any meaningful
analysis. While a number of different approaches have been described in the
literature, CNN based ones have emerged as the most effective and popular
choice [12, 1] as a way of automatically learning features rather than
handcrafting them. A CNN typically excels with image sizes of less than one
million pixels [32, 15, 34, 39, 17]. Consequently, the most popular
preprocessing approach to gigapixel image analysis involves the breaking up of
the original large image into its constituent sub-images, that is patches, but
also sometimes referred to as regions of interest (ROIs).

A challenge that comes with patch-based supervised training is the need for
patch-level annotations. Although patch-level labelling can sometimes be
provided [27, 37, 25, 22], it is generally either extremely laborious and
expensive to obtain or simply infeasible. In order to make use of unlabelled
patches, most of the existing approaches employ techniques from multiple
instance learning (MIL) [16, 5, 3], weakly supervised learning [41, 2] and
unsupervised learning [36].

In MIL, unlabelled instances belong to labelled bags of instances. The bag
label can be determined based on a single instance [5], or based collectively
on all the instances [16, 3]. In weakly supervised learning, mostly inexact
supervision is observed in which the image-level (slide-level) label is used
at some point as the patch-level label [41, 2]. Finally, there have also been
some approaches that employ unsupervised learning techniques either as an
early step in a pipeline [41] or as a way of compressing the gigapixel images
into smaller representations [36].

One of the primary limitations of the previously described methods emerges as
a consequence of their analysing a large input image by means of independent
analysis of smaller patches that is divided into. In particular, such
approaches are inherently unable to capture information distributed over
scales greater than the patch size. For example in the case of whole slide
images, although cell characteristics can be extracted based on isolated
patches, higher-level structural information, such as the shape or extend of a
tumour, can only be captured when analysing larger regions. Explicitly
modeling the spatial correlations between the patches has been proposed as a
potential solution [25, 22, 40]. A different approach involves the use of
multi-resolution patches [11, 16, 27, 5] which seems to provide better
empirical results.

Another major limitation of patch-based analysis comes from the initial patch
extraction procedure. There are generally two ways that the WSIs are turned
into patches: (a) by tiling the whole image or (b) by random sampling. On the
one hand, the brute force nature of (a) is computational prohibiting which is
why most methodologies resort to intensive preprocessing steps. On the other
hand, relevant information can be missed out with (b), especially since we are
dealing with gigapixel images. A more sophisticated approach to selecting the
right patches involve the use of attention models [2, 29, 28].

The framework we introduce in this paper addresses all of the aforementioned
limitations. It allows for a multi-resolution, multi-field of view neural
network to be optimized in an end-to-end fashion with gigapixel images, and
without the need for any type of preprocessing. For the dynamic patch
localization, we employ the Spatial Transformer Network (STN) [18] in a new
way as described in the following section. The most prominent use cases of
STNs are spatial invariance [31] and supervised semantic segmentation [20, 7,
14]. Existing research on MIL of STNs is scarce and, relative to our approach,
simple [33, 13].

A MagNet consists of $N$ magnifying layers followed by a classification layer.
Consider a single gigapixel image $I_{0}$ that will pass through a MagNet.

###  3.1 Magnifying Layer

#### Resize and pad.

As we subsequently employ convolutional layers expecting $56\times 56$ pixel
images, input $I$, either as a single input image (e.g. $I_{0}$) or a set of
images, is resized to a $56\times h_{i}$ or $w_{i}\times 56$ resolution with
the smaller side, $h_{i}$ or $w_{i}$, then padded (new pixels are black) so
that $h_{i}=56$ or $w_{i}=56$ accordingly. For the purpose of up-sampling (see
“Sampling” paragraph), a larger version $I^{\prime}$ ($112\times 112$
resolution) is also generated.

#### Convolutional layers.

The salient parts in each image (e.g. areas with tissue at the lowest
magnification level) vary significantly in size. This is illustrated in Figure
3 wherein each row depicts a different WSI. Therefore, the right kernel size
for the convolutional operations varies depending on $I$. As such, we stack
convolutional operations with different kernel sizes similarly to
InceptionNet–v3 [35]. Further information is provided in the Appendix. Our
MagNets employ three stacked convolutional layers for each patch
independently, e.g. a 2-layer MagNet with two patches at each magnifying layer
has six of these layers (two at the first layer, and four at the second).

#### Spatial Transformer.

A STN consists of three parts; a localization network, a grid generator, and a
sampler [18].

The _localization network_ is typically a FCNN or a recurrent neural network
[33], and its role is to output the parameters of a spatial transformation.
With scalability in mind, this approach was too prohibitive (GPU VRAM-wise)
for MagNet as for each ST there would have been a large amount of extra
parameters that needed to be optimized.

Instead, MagNet utilizes a spatial sparsemax at the last convolutional layer
whose output can be used to infer the affine transformation parameters ($s$,
$t_{x}$, $t_{y}$) directly (more information in Appendix). Note how the output
of sparsemax has a $56\times 56$ resolution and can therefore be thought as a
mask highlighting the regions of interest of the input image.

Given the transformation parameters $s$ for isotropic scaling and $t_{x}$,
$t_{y}$ for translation on each axis, we further constrain the parameters
accordingly:

$\displaystyle s$ $\displaystyle=max(s,0.05)$ (1) $\displaystyle t_{x}$
$\displaystyle=tanh(t_{x})$ (2) $\displaystyle t_{y}$
$\displaystyle=tanh(t_{y})$ (3)

for spatial (affine) transformation $\theta$,

$\displaystyle\theta=\begin{bmatrix}s&0&t_{x}\\\ 0&s&t_{y}\\\ \end{bmatrix}$
(4)

The _grid generator_ then creates the desired grid by applying $\theta$ on a
meshgrid with dimensions $56\times 56$. Based on a differentiable _sampler_ ,
e.g. billinear sampling, an image can be transformed by $\theta$ by
interpolating it onto the grid.

#### Sampler

Early on while experimenting with bilinear interpolation, it became obvious
that it was a poor choice of a sampler for our work. For more information we
direct the reader to the excellent work of Jiang et al. [19] that later came
out corroborating to our empirical analysis and providing an alternative
sampler, called Linearized Multi-Sampling, whose gradients are not affected by
the amount of scaling (i.e. how much the network has zoomed-in) that is needed
(part of our empirical analysis is included in the Appendix). We use the
original implementation of this sampler as provided to the authors by Jiang et
al. [19].

#### Sampling (The “heart” of the MagNet)

This is the part that makes each layer ”magnifying”. Given $\theta$ and $I$,
we can transform $I$ based on $\theta$ (simple matrix operation) to retrieve
an image containing only the detected region of interest. MagNet applies the
transformation $\theta$ on $I^{\prime}$ instead, thereby allowing the output
to contain information (finer-grain) that was not present in $I$. An example
of a magnifying layer that outputs two patches is shown in Figure 2. The
introduction of new information by transforming images that came from a higher
magnification level is what enabled the idea of stacking multiple
magnification layers together.

###  3.2 Classification Layer

The images of the last magnifying layer are sampled using a grid with
$224\times 224$ pixel resolution. These images are forwarded through a CNN
(InceptionNet-v3) that outputs a feature map into a Gated Recurrent Unit (GRU)
network. The output of the GRU can be passed through a FCNN (two layers with
512 and 256 hidden neurons respectively) to output the $\hat{y}$ estimate of
whether the given WSI contains cancer or not.

###  3.3 Auxiliary classifiers

A form of self-supervision is introduced by using two auxiliary classifiers.
These are ImageNet–pretrained ResNets–18 networks [15] that output a slide-
level prediction using the extracted images from magnifying layer 1 and layer
3 respectively. The premise is that information presented at layer 3 images
should provide an equally good, or better, prediction than that from layer 1.

###  3.4 Configurations

The network consists of $L$ magnifying layers each of which is concerned with
a different magnification scale, shifted dynamically based on the degree of
zooming (i.e. $s$) thus far. At each layer $l$, $P_{l}$ number of patches are
extracted (ROIs).

A consequence of the recurrent nature of MagNets is that an exponential number
of patches are extracted and analysed from a single gigapixel image, if more
than 1 patch is extracted per layer. In particular, given a constant $P_{l}$
across the layers:

$\textrm{Total patches}=\begin{cases}L,&\text{if $P=1$}.\\\
P^{L},&\text{otherwise}.\end{cases}$

We find that a combination of $[2,3]$ for $P_{l}$ provides a balance between a
sufficient rate of expansion (breadth), while allowing for up to 4-layer
MagNets (depth) to be trained on a GPU with $24$ GB of VRAM.

##  4 Evaluation

###  4.1 Camelyon data sets

The Camelyon data sets contain WSIs from surgically resected lymph nodes of
breast cancer patients. These WSIs were independently curated across multiple
hospitals [12, 26]. Camelyon16 includes images from 238 normal and 160
cancerous tissue sections whereas the publicly available portion of Camelyon17
has a total of 500 WSIs (318 normal, 182 cancerous). In addition, in the case
of metastasis, metadata is available as to the extend of the metastasis
(macrometastasis, micrometastasis, or isolated tumour cells (ITC)). Since only
a small portion of the data set contains the much more difficult ITC cases, we
exclude such cases from the training data set.

We follow the protocol described in the Camelyon competition website, and in
addition set aside $25\%$ of Camelyon17 as a testing set. We shuffle the
remaining WSIs from Camelyon17 with the Camelyon16 WSIs, and train on the
$80\%$ and validate the better models from the remaining $20\%$. The best
MagNets (based on the validation set) are retrained on both the training and
validation data, and evaluated on the testing set.

The pixel-level annotations that are available for some of the WSIs of
Camelyon are not used in our work. Instead, we only use the binary slide-level
label that indicates the presence, or lack thereof, of cancerous cells
somewhere in the gigapixel image.

Figure 3: Three WSIs from different hospitals showcasing the differences in
the scanning process. The left side shows the unfiltered version of the WSI,
whereas the right ride shows the WSIs after the “grey” filter has been
applied.  Figure 4: Examples from the testing set against the 3–layer MagNet.
WSIs in the top square (blue) do not contain any malignancies, whereas the
WSIs on the bottom (red) contain macro-, micro-metastasis, and ITC in order
from top to bottom. The red circles show the cancer based on the annotations
provided by the pathologists. Note that only 10 out of the last layer’s 18
patches are shown for space efficiency.

Figure 5: A visualization of a forward pass of a WSI with minor metastasis
(from the testing set) through a 4-layer MagNet.

###  4.2 Data Augmentation

For any given image, we apply a filter that removes the “grey” pixels of the
image. This includes any white background, as well as gradients and other
artifacts that have red, green, and blue channel values close together
(threshold set to $15$). The effects of the filter are illustrated in Figure
3.

We employ neither colour normalization nor random colour perturbation [27].
Synthetic data augmentation we perform is based on horizontal and vertical
mirroring, and rotation by 90, 180, and 270 degrees.

###  4.3 Training

The networks were trained using the Adam optimizer for 200 epochs. A batch
size of $16$ and $8$ was employed for MagNet networks with 3 and 4 layers
respectively. The initial learning rate was set to $3\times 10^{-}5$ and was
decayed using a cosine annealing scheduler. Both ResNet and InceptioNet
networks are initialized using pretrained networks on ImageNet. However, the
ST convolutional layers are randomly initialized.

#### “Frozen” patch

Some WSIs have already been preprocessed so that they only contain regions
with tissue, whereas others depict all of the tissue slide (see Figure 3).
This diversity comes as a consequence of the differences in the clinical
pipelines leading to the creation of WSIs, e.g. due to different scanning
profiles. In order to mitigate for the above, we freeze the 1st patch of the
2nd layer so that it always attends to the whole input image. This allows for
the image to catch-up in quality in the cases where a large amount of zooming
was required at the first magnifying layer, i.e. when the WSI shows the whole
tissue slide. Examples are included in the Appendix.

#### Loss Functions

In order to impose the aforementioned self-supervision with the auxiliary
classifiers, we employ the paradoxical loss function as described by Maksoud
et al. [28]. In addition, cross-entropy is used between the slide-level labels
and both the last output of the GRU as well as the ResNet-18 outputs.

Table 1: The results of MagNets on the testing set containing $25\%$ of the publicly available Camelyon17 data set.  | Macro- | Micro- | ITC | Macro $+$ Micro | All
---|---|---|---|---|---
Mean RGB Baseline [36] | $59\%$ | $57\%$ | - | $58\%$ | -
Baseline w/ patch–level supervision [36] | $91\%$ | $63\%$ | - | $77\%$ | -
NIC [36] | $72\%$ | $68\%$ | - | $70\%$ | -
3-layer MagNet | $\boldsymbol{95\%}$ | $71\%$ | $57\%$ | $\boldsymbol{84\%}$ | $71\%$
4-layer MagNet | $91\%$ | $\boldsymbol{76\%}$ | $\boldsymbol{63\%}$ | $\boldsymbol{84\%}$ | $\boldsymbol{75\%}$

##  5 Results & Discussion

To evaluate the proposed method, using the optimization framework described in
the previous section we trained 3-layer and 4-layer MagNets on the task of
cancer detection from WSIs. A summary of the results is presented in Table 1
which shows the area under the receiver operating characteristic curve (AUROC)
– the standard evaluation metric used in the related literature [37, 36, 12,
26] – obtained using different configurations of the proposed method and the
state of the art in the literature. Considering that to the best of our
knowledge there is no existing work which utilizes only slide-level
information on the Camelyon17 data set, the comparison in the table is made
with the best reported competitor performances on Camelyon16 [36] (which would
be fair to say overall provides a lesser challenge, thus, if anything, making
the comparison disadvantageous to our method).

Interrogating our findings further, we note that although the 3–layer MagNet
achieves better separation between macro-metastasis and normal cases, the
4–layer MagNet does better on micro-metastasis, as well as on the ITC cases.
The latter is unsurprising given that a 4-layer MagNet makes use of finer-
grain detail (cells are visible in Figure 5). Perhaps more importantly, the
4–layer MagNet performs well on ITC cases with a $63\%$ AUROC despite the lack
of ITC examples in our training set.

MagNets exhibit robust and effective exploration capabilities, namely
attending to image content in an attention driven manner, exploring slides at
varying magnification levels best suited to the task at hand and learning how
to fuse relevant information both within the same WSI region and across
different regions at different magnification levels. In addition, the
classifier (in the form of InceptionNet) demonstrates an excellent ability to
distinguish normal from cancerous tissue across all magnifications. Examples
corroborating this are shown in Figure 4. It is particularly remarkable to
observe cases such as those shown in rows 3–4 of the bottom group of examples
in Figure 4, wherein the classifier can be seen to change its decision from
“normal” to “cancerous” when sufficient evidence for cancer is accumulated at
any magnification level. These examples were verified by using annotations
provided by pathologists.

Figure 5 shows the forward pass of a single example through a 4-layer MagNet.
The Figure is made up of three images. Image A contains $10\times 4$ sub-
images wherein the first column contains the input $I$ to a magnifying layer
along red squares indicating the patches to be magnified next, as selected by
the MagNet. The rest of the images visualize the last magnifying layer along
with its magnified patches of $224\times 224$ pixel resolution that are
forwarded to the classification layer. Since a GRU is employed in the
classification layer, we are able to output a prediction with each new patch,
allowing for trivial post-processing analysis. For instance, in the example
shown, we observe that the model did not find anything that could be
considered as a malignancy for the better part of the last layer’s patches.
The patches responsible for the change in malignancy probability (from $0.19$
to $0.30$ and then $0.43$) contain a region that was not seen in previous
patches and which contains unusual looking tissue. As the final patches are
forwarded through the GRU, we observe that the model successfully classifies
the WSI as cancerous.

##  6 Conclusions

In this work, we introduced the MagNet – a neural network consisting of fully-
connected, convolutional, and recurrent layers that employ spatial
transformers (ST) in a novel and ingenious manner so as to facilitate
attention and data driven recurrent exploration and, ultimately, end-to-end
learning over gigapixel images. Our experimental results and analysis
demonstrate the manifold novel capabilities of both the MagNets and the
proposed framework in gigapixel image analysis.

The built-in hard attention mechanism of MagNets makes them perfectly suited
for clinical use. In particular, any machine learning system that is deployed
in clinical practice must generate accurate explanations for its decisions
that are easily interpretable by a medical expert. The explanations generated
by MagNets are visually intuitive for a domain-specific expert to interpret
and can be generated on the go without any additional overhead. Moreover,
MagNets can be optimized without extra supervision (e.g. bounding boxes) for
the task at hand. This is of high significance since for most clinical tasks,
collecting ground truth data required for a higher degree of supervision is
either extremely laborious and expensive, or simply not possible, e.g. in the
case of patient prognosis.

## References

  * [1] G. Aresta, T. Araújo, S. Kwok, S. S. Chennamsetty, M. Safwan, V. Alex, B. Marami, M. Prastawa, M. Chan, M. Donovan, G. Fernandez, J. Zeineh, M. Kohl, C. Walz, F. Ludwig, S. Braunewell, M. Baust, Q. Dang Vu, M. Nguyen Nhat To, and P. Aguiar.  Bach: Grand challenge on breast cancer histology images.  arXiv:1808.04277, 2018.
  * [2] A. BenTaieb and G. Hamarneh.  Predicting cancer with a recurrent visual attention model for histopathology images.  In A. F. Frangi, J. A. Schnabel, C. Davatzikos, C. Alberola-López, and G. Fichtinger, editors, Medical Image Computing and Computer Assisted Intervention, pages 129–137. Springer International Publishing, 2018.
  * [3] D. Bychkov, N. Linder, R. Turkki, S. Nordling, E. P. Kovanen, C. Verrill, M. Walliander, M. Lundin, C. Haglund, and J. Lundin.  Deep learning based tissue analysis predicts outcome in colorectal cancer.  Scientific Reports, 8(1), feb 2018.
  * [4] Peter D. Caie, Neofytos Dimitriou, and Ognjen Arandjelović.  Chapter 8 - precision medicine in digital pathology via image analysis and machine learning.  In Stanley Cohen, editor, Artificial Intelligence and Deep Learning in Pathology, pages 149–173. Elsevier, 2021.
  * [5] G. Campanella, W. K. V. Silva, and J. T. Fuchs.  Terabyte-scale deep multiple instance learning for classification and localization in pathology.  arXiv:1805.06983, 2018.
  * [6] Y. Chen, J. Li, H. Xiao, X. Jin, S. Yan, and J. Feng.  Dual path networks.  CoRR, abs/1707.01629, 2017.
  * [7] J. Dai, K. He, and J. Sun.  Instance-aware semantic segmentation via multi-task network cascades.  CoRR, abs/1512.04412, 2015.
  * [8] I. Demir, K. Koperski, D. Lindenbaum, G. Pang, J. Huang, S. Basu, F. Hughes, D. Tuia, and R. Raskar.  Deepglobe 2018: A challenge to parse the earth through satellite images.  arXiv:1805.06561, 2017.
  * [9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  * [10] Neofytos Dimitriou, Ognjen Arandjelović, and Peter D. Caie.  Deep learning for whole slide image analysis: An overview.  Frontiers in Medicine, 6:264, 2019.
  * [11] S. Doyle, M. Feldman, J. Tomaszewski, and A. Madabhushi.  A boosted bayesian multiresolution classifier for prostate cancer detection from digitized needle biopsies.  IEEE Transactions on Biomedical Engineering, 59(5):1205–1218, may 2012.
  * [12] B. B. Ehteshami, M. Veta, P. Johannes van Diest, and et al.  Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer.  JAMA, 318(22):2199–2210, 2017.
  * [13] S. Guo, L. Liu, W. Wang, S. Lao, and L. Wang.  An attention model based on spatial transformers for scene recognition.  In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 3757–3762, 2016.
  * [14] K. He, G. Gkioxari, P. Dollár, and B. R. Girshick.  Mask R-CNN.  CoRR, abs/1703.06870, 2017.
  * [15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [16] L. Hou, D. Samaras, M. T. Kurc, Y. Gao, E. J. Davis, and H. J. Saltz.  Patch-based convolutional neural network for whole slide tissue image classification.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, jun 2016.
  * [17] G. Huang, Z. Liu, and Q. K. Weinberger.  Densely connected convolutional networks.  CoRR, abs/1608.06993, 2016.
  * [18] M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu.  Spatial transformer networks.  CoRR, abs/1506.02025, 2015.
  * [19] Wei Jiang, Weiwei Sun, Andrea Tagliasacchi, Eduard Trulls, and Kwang Moo Yi.  Linearized multi-sampling for differentiable image transformation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
  * [20] J. Johnson, A. Karpathy, and F. F. Li.  Densecap: Fully convolutional localization networks for dense captioning.  CoRR, abs/1511.07571, 2015.
  * [21] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, and A. Zisserman.  The kinetics human action video dataset.  CoRR, abs/1705.06950, 2017.
  * [22] Bin Kong, Xin Wang, Zhongyu Li, Qi Song, and Shaoting Zhang.  Cancer metastasis detection via spatially structured deep network.  In Marc Niethammer, Martin Styner, Stephen Aylward, Hongtu Zhu, Ipek Oguz, Pew-Thian Yap, and Dinggang Shen, editors, Information Processing in Medical Imaging, pages 236–248. Springer International Publishing, 2017.
  * [23] A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, T. Duerig, and V. Ferrari.  The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.  arXiv:1811.00982, 2018.
  * [24] G. Larsson, M. Maire, and G. Shakhnarovich.  Fractalnet: Ultra-deep neural networks without residuals.  CoRR, abs/1605.07648, 2016.
  * [25] Y. Li and W. Ping.  Cancer metastasis detection with neural conditional random field.  arXiv:1806.07064, 2018.
  * [26] G. Litjens, P. Bandi, B. Ehteshami Bejnordi, O. Geessink, M. Balkenhol, P. Bult, A. Halilovic, M. Hermsen, R. van de Loo, R. Vogels, F. Q. Manson, N. Stathonikos, A. Baidoshvili, P. van Diest, C. Wauters, M. van Dijk, and J. van der Laak.  1399 h&e-stained sentinel lymph node sections of breast cancer patients: the CAMELYON dataset.  GigaScience, 7(6), May 2018.
  * [27] Y. Liu, K. Gadepalli, M. Norouzi, E. G. Dahl, T. Kohlberger, A. Boyko, S. Venugopalan, A. Timofeev, Q. P. Nelson, S. G. Corrado, D. J. Hipp, L. Peng, and C. M. Stumpe.  Detecting cancer metastases on gigapixel pathology images.  CoRR, abs/1703.02442, 2017.
  * [28] Sam Maksoud, Kun Zhao, Peter Hobson, Anthony Jennings, and Brian C. Lovell.  Sos: Selective objective switch for rapid immunofluorescence whole slide image classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [29] T. Qaiser and M. N. Rajpoot.  Learning where to see: A novel attention model for automated immunohistochemical scoring.  arXiv, 2019.
  * [30] S. Sabour, N. Frosst, and E. G. Hinton.  Dynamic routing between capsules.  CoRR, abs/1710.09829, 2017.
  * [31] C. Shu, X. Chen, Q. Xie, and H. Han.  Hierarchical spatial transformer network.  CoRR, abs/1801.09467, 2018.
  * [32] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  CoRR, abs/1409.1556, 2014.
  * [33] K. S. Sønderby, K. C. Sønderby, L. Maaløe, and O. Winther.  Recurrent spatial transformer networks.  CoRR, abs/1509.05329, 2015.
  * [34] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, E. R. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich.  Going deeper with convolutions.  CoRR, abs/1409.4842, 2014.
  * [35] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision, 2015.
  * [36] David Tellez, Geert Litjens, Jeroen van der Laak, and Francesco Ciompi.  Neural image compression for gigapixel histopathology image analysis.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(2):567–578, 2021.
  * [37] D. Wang, A. Khosla, R. Gargeya, H. Irshad, and H. A. Beck.  Deep learning for identifying metastatic breast cancer.  arXiv:1606.05718, 2016.
  * [38] X. Yue, N. Dimitriou, and O. Arandjelović.  Colorectal cancer outcome prediction from H&E whole slide images using machine learning and automatically inferred phenotype profiles.  BICOB, pages 139–149, 2019.
  * [39] S. Zagoruyko and N. Komodakis.  Wide residual networks.  CoRR, abs/1605.07146, 2016.
  * [40] G. F. Zanjani, S. Zinger, and H. N. P. de With.  Cancer detection in histopathology whole-slide images using conditional random fields on deep embedded spaces.  In Medical Imaging 2018: Digital Pathology. SPIE, mar 2018.
  * [41] X. Zhu, J. Yao, F. Zhu, and J. Huang.  Wsisa: Making survival prediction from whole slide histopathological images.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6855–6863, July 2017.