Department of Computer Science
TDT4173 - Method Paper

Deep Learning

_Deep_ Convolutional Neural Networks
A survey of the foundations, selected improvements, and some current
applications

_Authors:_

Lars Ankile (larslank)
Morgan Heggland (morganfh)
Kjartan Krange (kjartkra)

October 12, 2020

###### Abstract

Within the world of machine learning there exists a wide range of different
methods with respective advantages and applications. This paper seeks to
present and discuss one such method, namely Convolutional Neural Networks
(CNNs). CNNs are deep neural networks that use a special linear operation
called convolution. This operation represents a key and distinctive element of
CNNs, and will therefore be the focus of this method paper. The discussion
starts with the theoretical foundations that underlie convolutions and CNNs.
Then, the discussion proceeds to discuss some improvements and augmentations
that can be made to adapt the method to estimate a wider set of function
classes. The paper mainly investigates two ways of improving the method: by
using locally connected layers, which can make the network less invariant to
translation, and tiled convolution, which allows for the learning of more
complex invariances than standard convolution. Furthermore, the use of the
Fast Fourier Transform can improve the computational efficiency of
convolution. Subsequently, this paper discusses two applications of
convolution that have proven to be very effective in practice. First, the YOLO
architecture is a state of the art neural network for image object
classification, which accurately predicts bounding boxes around objects in
images. Second, tumor detection in mammography may be performed using CNNs,
accomplishing 7.2% higher specificity than actual doctors with only .3% less
sensitivity. Finally, the invention of technology that outperforms humans in
different fields also raises certain ethical and regulatory questions that are
briefly discussed.

###### Table of Contents

  1. 1 Introduction
  2. 2 Foundations
    1. 2.1 Core Idea of Convolution
    2. 2.2 Theoretical Foundation
    3. 2.3 Training and Optimization
  3. 3 Improvements to the Method
    1. 3.1 Improvements to the Basic Convolution
    2. 3.2 Improvements to the Computation of the Convolution
  4. 4 Current Applications
    1. 4.1 YOLO
    2. 4.2 Tumor Detection in Mammography
  5. 5 Conclusions and Further Work

##  1 Introduction

In the Machine Learning taxonomy, Convolutional neural networks (CNNs) are a
subset of deep learning algorithms, as displayed in Figure 1. CNNs can be both
supervised and unsupervised. An example of the former is image classification
and for the latter, an instance is word embedding (connecting words and their
meanings).

CNNs are a type of neural network which are used to analyze data with a grid-
like structure. A good example of this is image data, represented as two
dimensions of RGB values. For problems such as image classification and the
like, there are three main challenges with using a regular MLP (multi layer
perceptron), which CNNs solve:

  * •

_Parameter growth:_ Using one perceptron for each pixel causes the amount of
parameters to increase rapidly.

  * •

_Translations:_ A standard MLP would treat a picture and its slightly shifted
version as two entirely different images. Recognizing a car in a picture
should not depend on where in the image the car is located.

  * •

_Spatiality:_ MLPs do not account for spatial relations in images. The fact
that two pixels are in proximity to one another is meaningful information.

Thus, CNNs solve the problem of understanding images, using networks of more
manageable complexity. The special neural network takes into account that
physical closeness between pixels bears meaning and that elements of interest
can appear anywhere in a picture. This is accomplished through the use of a
linear convolution operation, which is discussed in section 2. The use of this
operation, in one or more layers, is what defines a CNN.

Sometimes, however, the assumptions underlying the design choices of CNNs must
be weakened or altered due to the nature of the input data. Translational
invariance is, for example, not always as desirable. Furthermore, even though
computational complexity is more manageable, modern networks tend to be very
deep. This makes clever algorithms for calculating the convolutions a must.
Both of these aspects are discussed in detail in section 3.

Today, the method enjoys a wide range of applications, spanning from object
recognition to natural language processing. CNNs like YOLO (section 4), are
used to find objects and are used in e.g. BMW cars. In 2018, Cisco®
prophesized that by 2022 84% of all data traffic would be video [Cisco, 2020].
Even more striking, this high percentage is not counting still images. Since
computers traditionally have had a notoriously hard time understanding visual
data, the growing amount of videos and images on the world wide web makes the
need for understanding this data apparent. The evolution CNNs have seen over
the last decade is a huge step in the direction of making visual data readable
by computers. In fact, convolutional neural networks have won every single
ILSVRC (ImageNet Large Scale Visual Classification Challenge) since 2012
indicating that CNNs are the best publicly available solution for image
classification problems.

Figure 1: CNNs are a subset of Deep Learning Neural Networks. Source: Soffer
et al. [2018]

##  2 Foundations

The use of convolution instead of matrix multiplication in at least one of the
layers in a neural network is what defines a convolutional neural network
[Goodfellow et al., 2016]. ”Use of convolution” refers to the _convolutional
operator_ , a linear operator which is an essential distinctive element of a
convolutional neural network, and will be the focus of the subsequent
sections. In this section, the core idea and theoretical foundations of the
convolutional operator are discussed.

###  2.1 Core Idea of Convolution

Prior knowledge of the structure or nature of the input data to a neural
network can in some cases be exploited to improve the performance of the
network. One such example is signal processing, which includes both temporal
(e.g. sound) and spatial (e.g. 2D images) signals [Kaelbling, 2019]. This data
can be thought of as grid-like: sound waves and other time-series data
resemble one dimensional grids where each measurement is an entry, images
resemble a two dimensional grid of pixels etc. This allows for optimization of
analysis of such data in a neural network. Compared to a traditional network,
runtime and space complexity may be improved, and the training data generally
required for feasible results may be reduced.

The main idea behind convolution is the identification of features in the
input data through the application of a _kernel_ (also known as _filter_)
across the input data. Both the input data and the kernel is of a grid-like
structure, and can be represented as _tensors_ , which are multidimensional
arrays. The kernel can be of any size, and is usually smaller than the input
data. Kernels are used to identify features in the input data, such as edges
in an image. The input data is _convolved_ with the kernel, meaning the kernel
is ”slid” across the input data, calculating the dot product or matrix product
(depending on dimensions) between the overlapping part of the input data and
the kernel. An illustrative example of the convolution operation can be seen
in Figure 2 and a mathematical description can be found in subsection 2.2.

Figure 2: Example of convolution of two dimensional input data, ignoring
outputs where the kernel is positioned partially outside the input data. The
outlined output indicate the result from applying convolution with the given
kernel on the outlined input. Source: Goodfellow et al. [2016]

When applying convolution, it is common practice to add _padding_ to the edges
of the input data to account for situations where the kernel is positioned
partially outside the original input values. The padding is commonly
implemented by extending the edges of the data with 0-valued entries (other
values may be used). Furthermore, it is common to increase _stride_ in order
to skip over indices when applying the kernel over the input, thus decreasing
the size of the layer output.

There are a number of reasons that make this approach effective for this
certain type of data. Firstly, it allows for the exploitation of a number of
_invariant_ properties of the data. One such property is _spatial locality_ :
the fact that the relevant information that makes up features are close to
each other in a topological sense (e.g the pixels considered to find a cat’s
ears are close to each other spatially). Another property is _translational
invariance_ : the fact that a feature can be identified independently of where
it is located in the topology (e.g it a cat is a cat regardless of where it is
located in an image). Both of these properties can be leveraged as a result of
the nature of the convolutional operation: sliding the kernel over the input
data to identify features. Informally speaking, the spatial locality is
leveraged through applying kernels to an enclosed region of the input data
each time, and translational invariance is leveraged through the movement of
the kernel across the input data. Additionally, as a result of reusing the
same kernel on the entirety of the input data, this method significantly
reduces the number of parameters required to store and tune compared to
traditional neural networks, which generally uses parameters only once. This
concept is called _parameter sharing_ and is an essential advantage of CNNs.
While traditional fully connected layers rely on a large number of parameters
describing the interaction between each input and output unit, applying a
kernel which is smaller sized than the input also significantly reduces the
number of parameters required. This is called _sparse connectivity_ , as
opposed to full connectivity. An illustration of this is shown in Figure 4,
and further discussion of these topics is included in subsection 3.1.

Normally in CNNs, the application of the convolution operation is followed by
a _non-linearity stage_ and then a _pooling stage_. This pipeline has proven
to be an effective constellation for identifying features in the data
[Goodfellow et al., 2016]. The main idea behind the non-linearity stage is the
application of a non-linear function to the input data, ensuring the non-
linearity properties of the network. The pooling stage further alters the
output by using a summary operation on regions of the data, reducing the
output in size while still highlighting key features, for instance by use of
maximization aggregation (known as _max-pooling_).

###  2.2 Theoretical Foundation

In a theoretical sense, the convolutional operator in its most primitive form
can be considered an operation on two functions of a real-valued input
[Goodfellow et al., 2016]. The convolution operation is defined as

$s(t)=(x*w)(t)=\int x(a)w(t-a)da$ (1)

where $x$ is the function mapping to a specific value in the input data, and
$w$ represents the kernel. This formulation can for example (with the right
kernel) be thought of as a smoothing average of $x$ across its entire domain,
giving highest weight to values closest to $t$. If the input values are
discrete, the convolution operation can be rewritten using summation:

$s(t)=(x*w)(t)=\sum_{a=-\infty}^{\infty}x(a)w(t-a).$ (2)

The input is commonly multidimensional. In that case, one can replace the
functions with multi-variable functions, operating on tensors. Consider an
example of applying convolution to a two-dimensional image $I$ as input. One
can then use a two-dimensional kernel $K$, and the operation can be written as
follows:

$S(i,j)=(I*K)(i,j)=\sum_{m}\sum_{n}I(m,n)K(i-m,j-n).$ (3)

That is, for a given pixel in the input, positioned in row $i$ and column $j$,
the convolution is computed by ”placing” the centre of the kernel over the
input pixel, and summing over the product of overlapping kernel parameters and
input pixels to produce the output value for $i$ and $j$.

The above operation forms the basis for the operation performed in the
convolutional stages or layers of neural networks. Depending on the values of
the parameters of the kernels, this operation is able to highlight and
identify different patterns and features in the input data.

###  2.3 Training and Optimization

The key component of convolution is the kernel with its corresponding
parameters. The values of the parameters dictate which features and patterns
the convolution operation identify. Thus, parameter selection is crucial in
order to achieve high accuracy in solving the machine learning problem at
hand. A method proven effective for setting these values is through training
of the neural network in its entirety using labeled training data. As network
size and complexity increase, fine-tuning a convolutional layer in isolation
is infeasible, and performance is more easily measured by the output of the
network as a whole. Only a brief overview of the training procedure is
included in this subsection, as this is not the core focus of the paper.

Training is efficiently conducted using _gradient descent_ and _back-
propagation_ [Kaelbling, 2019]. This method allows for automatic tuning of the
parameters through the application of labeled training data. In essence, a
_loss function_ or objective function calculates how well the network
performed on a single input example compared to the true output (from the
label). Based on this, one can calculate the gradient of the loss function
evaluated at the training example with respect to the parameters or weights in
each layer. This can be written as $\nabla_{W}Loss(NN(x:W),y)$, where $W$
represents all weights in all layers, $Loss$ is the loss function that takes a
predicted value generated by the neural network $NN$ with input training
example $x$ and weights $W$, and compares this to the actual value $y$. Using
the chain rule of differentiation, this gradient can be decomposed into parts
relevant to each layer. One can then apply the principle of back-propagation
to calculate the gradient with respect to the weights in each layer. This
represents the direction of the steepest ascent of the loss function with
respect to each weight. By updating the weight in the negative direction of
the gradient, taking a ”step” according to a _step size_ parameter, the output
loss is then reduced. This update can be written as
$\theta_{i}:=\theta_{i}-\eta\frac{\partial Loss}{\partial\theta_{i}}$, where
$\theta_{i}$ is a parameter and $\eta$ is the step size. This is the principle
of gradient descent and applies to kernel weights just as much as weights in a
traditional neural network layer. An example of the descent can be seen in
Figure 3.

Figure 3: An illustration of gradient descent on a three-dimensional objective
function. Source: Sanderson [2017]

As neural networks increase in size, the number of parameters grow
significantly. In order to produce reliable predictions and avoid
_overfitting_ , a large amount of diverse training data is required.
Overfitting is when the product of an analysis fits its input data or training
data ”too well”, in a sense that it will likely not generalize well to new
data. By leveraging convolution, less data can be used without the risk of
overfitting. This comes as a result of the benefits of convolution, such as
parameter sharing and sparse connectivity. On the other hand, increased use of
parameter sharing may cause _underfitting_. That is, the network is not able
to capture the full predictive power available in the training data.
Alternatives to help alleviate this issue are discussed in section 3.

##  3 Improvements to the Method

###  3.1 Improvements to the Basic Convolution

Convolutions and pooling impose an infinitely strong _prior_ on the model
parameters [Goodfellow et al., 2016]. A prior probability distribution over
the model parameters is a way to encode our beliefs about what kind of models
are likely to be good before the model is ever presented any data. The
strength of the prior says something about how strong our belief is. A weak
prior let the parameters roam relatively freely during training, while a
strong prior restricts the parameters. An infinitely strong prior sets the
probability for some of the parameters to be 0, i.e. they can never influence
the model, regardless of how much support the data provides.

In the case of convolutional networks, the infinitely strong prior says that
the function that the convolutional layer learns will only contain local
interactions and is invariant to translation [Goodfellow et al., 2016]. A
convolutional layer can be considered a fully connected layer where all non-
local connections are set to have a permanent zero-weight (this is not how it
is done in practice but is a useful conceptual tool). See Figure 4 for a
visual comparison of convolutional layers and fully connected ones.

Figure 4: Comparison of the connectedness of convolutional and fully connected
layers, as well as of degree of parameter sharing between different layer
architectures. All but the network in the bottom-right (fully connected layer)
are locally connected. The locally connected networks differ in the degree to
which they share parameters. In the top-right network (standard convolution,
kernel size 2), we see that the same two parameters are shared among all the
connections. In the top-left network (locally connected layer), there are a
separate parameters for every connection. In the bottom-left network (tiled
convolutional layer, kernel size 2, 2 distinct kernels), there is some
sharing, albeit with more separation and more parameters. Source: Goodfellow
et al. [2016]

For any prior to be useful, we would require the assumptions made by the prior
to be accurate. E.g. when inputs in distant locations influence the output,
the prior imposed by the convolution might be too strong and cause the network
to underfit. In this subsection, we will present two alternatives to the basic
convolution that weaken this prior slightly, _locally connected layers_ and
_tiled convolution_.

####  3.1.1 Locally Connected Layers

As stated above, the convolutional layer is invariant to translations.
Sometimes, however, approximate position information must be preserved. An
example can be seen in LeCun et al. [1989]. One solution to the problem is to
use locally connected layers. This form of layer preserves the local
connections to take advantage of the fact that local interactions is what
matters in images, but can treat different parts of the image differently.
This is useful when we know that features will be a function of a local area
of the input space, but have no reason to believe that we will find the same
features multiple places in the input [Goodfellow et al., 2016]. An example
could be in facial recognition, where the eyes usually would be located near
the top of the image. Locally connected layers can also be beneficial when
using very deep, zero-padded convolutions. In those networks, the inputs along
the edges will have less and less impact on the output as we go deeper into
the network. Because locally connected layers do not share parameters, they
can learn to amplify the edge inputs more than the ones in the center, if
useful given the training data.

In Figure 4, one can see an example of a layer with local connections in the
top row. Observe that each node is equally connected as in the convolutional
case (bottom), but each connection has a unique parameter associated with it.

Locally connected layers will require orders of magnitude fewer parameters
than the fully connected ones because of being locally connected from layer to
layer. However, they will require many more parameters than regular
convolutional layers because they do not utilize parameter sharing. This can
lead to a large performance-hit in use-cases where the properties of locally
connected layers are not required.

####  3.1.2 Tiled Convolution

Convolutional neural nets allow for hard coding of translational invariance
into the network. However, this makes for a rigid structure that is unable to
adapt to other kinds of invariances. Examples are invariance to scale and
rotation, both of which regular convolutions handle poorly. Ngiam et al.
[2010] proposes tiled convolutional neural nets, or Tiled CNNs, as an
alternative. The main benefit of this approach is that it allows the network
to learn invariances, instead of them having to be hardcoded into the network.
This architecture can learn complex invariances (e.g. rotation and scale)
while still keeping much of the advantages of CNNs’ low number of parameters,
which allows for faster training and easier learning (compared to fully
connected layers).

What sets tiled CNNs apart is that they only constrain parameters that are $k$
steps away from each other. The special case of $k=1$ yields the normal
convolution operation, and with $k$ equal to the number of input parameters we
get fully untied simple units (like locally connected layers discussed above).
When we vary $k$ along this spectrum we vary the trade-off between having few
learnable parameters and being able to learn complex invariances.

In Figure 4, one can see a comparison between locally connected layers, tiled
convolution, and standard convolution. In the example, the network in the
middle is a simple representation of tiled convolution, where each kernel is 2
pixels wide, and there are 2 distinct kernels. This means that it is just as
connected as the regular convolution (bottom), but there is twice the number
of parameters, and the parameters that are forced to be equal are more spaced
out.

Figure 5: Left: CNNs with local connections and shared weights. Right:
Partially unshared locally connected networks – Tiled CNNs. Units with the
same color belong to the same feature map, and within each map, units with the
same fill texture have tied weights. Source: Ngiam et al. [2010]

###  3.2 Improvements to the Computation of the Convolution

The deep CNNs of modern machine learning often contain millions of parameters,
making the use of parallel processing essential to be able to train the
networks in reasonable time [Goodfellow et al., 2016]. In some special cases,
it is also possible to speed up the convolution by utilizing different
convolution algorithms.

####  3.2.1 Using the Fast Fourier Transform to Convolve, Fast

In Mathieu et al. [2014], the Fast Fourier Transform (FFT) is utilized to
significantly increase efficiency of training and inference in CNNs. The work
builds on the Convolution Theorem, that states that convolutions in the space
domain are equivalent to point-wise product in the Fourier domain. Let
$\mathcal{F}$ be the Fourier transform and $\mathcal{F}^{-1}$ be the inverse.
We have that the convolution of two arbitrary functions $f$ and $g$ is

$\displaystyle f*g=\mathcal{F}^{-1}(\mathcal{F}(f)\cdot\mathcal{F}(g)).$ (4)

The increased efficiency is achieved by both utilizing that the dot product is
a faster operation than the convolution operation, as well as that the
transformed input features can be reused multiple times, lowering the initial
cost of transforming them.

There are mainly 3 equations containing convolutions that are important for
why this approach is efficient. Let $x_{f}$ be a set of input images indexed
by $f$, of dimensions $n\times n$ and $y_{f^{\prime}}$ the set of output
images indexed by $f^{\prime}$, and the layer $L$’s trainable parameters
$w_{f^{\prime}f}$, which are small kernels with dimensions $k\times k$, then
the equations are the following:

$\displaystyle y_{f^{\prime}}$ $\displaystyle=\sum_{f}x_{f}*w_{f^{\prime}f}$
(5) $\displaystyle\frac{\partial L}{\partial x_{f}}$
$\displaystyle=\frac{\partial L}{\partial y_{f^{\prime}}}*w^{T}_{f^{\prime}f}$
(6) $\displaystyle\frac{\partial L}{\partial w^{T}_{f^{\prime}f}}$
$\displaystyle=\frac{\partial L}{\partial y_{f^{\prime}}}*x_{f}.$ (7)

As mentioned, there are two things that make this approach fast. The first is
that computing the convolution directly in general requires more operations
than calculating the convolutions by going via the Fourier domain. In Mathieu
et al. [2014] they compute that direct convolution and FFT convolution,
respectively, for Equation 5 would require

direct convolution: $\displaystyle S\cdot f^{\prime}\cdot
f\cdot(n-k-1)^{2}\cdot k^{2}\quad\textnormal{and}$ (8) convolution with FFT:
$\displaystyle 2Cn^{2}\log n[f^{\prime}\cdot S+f\cdot S+f^{\prime}\cdot
f]+4S\cdot f^{\prime}\cdot f\cdot n^{2},$ (9)

operations respectively, with $S$ being mini-batch size and $C$ a constant.
The expression for direct convolutions, Equation 8, is a product of 5
variables, and will in general be larger than the expression for convolutions
via FFT, Equation 9, which is a product of 4 variables.

The second effect that provides the efficiency increase is that Fourier
transforms can be calculated once and reused [Mathieu et al., 2014]. That is
because in equations (5), (6), and (7), all the matrices indexed by $f$ is
convolved with each of the matrices indexed by $f^{\prime}$. Therefore, each
feature map can be processed by the FFT once, and all subsequent convolutions
can be performed as pair-wise products in the Fourier domain, without having
to recalculate the FFT. This makes for a large improvement in speed, and means
that even though a single convolution might be more computationally expensive
with FFT, the effective reuse of transformed inputs compensate for the
overhead.

##  4 Current Applications

###  4.1 YOLO

YOLO, You Only Look Once, is one of the most efficient object detection
algorithms that exist [Bochkovskiy et al., 2020]. Combined with some other
clever approaches, it is an example of a CNN used to analyze visual data. The
first version was published by Redmon et al. [2016], with the newest
implementation, YOLOv4 released in April 2020. For v4 and onward, Alexey
Bochkovskiy is leading the project.

The YOLO algorithm predicts both the position (represented as a bounding box)
and the classification of objects in images. YOLO aims to find the following
variables in a picture: $(bx,by)$ \- the center of a bounding box, $(bw,bh)$
\- the width and height of a bounding box, $c$ \- the class of the object,
$P_{c}$ \- the probability that there is an object of class $c$ in the box.
Often YOLO splits the image into a $19x19$ image, where each cell predicts $5$
bounding boxes of the form $y=(P_{c},bx,by,bw,bh,c)$. This gives $19x19x5$ =
$1805$ different bounding boxes per picture. Removing the boxes with a low
$P_{c}$ is called _non-max suppression_ as seen in Figure 6.

Figure 6: A subroutine of the YOLO algorithm – non-max suppression only keeps
best fit boundaries  Source: Sambasivarao [2020]

The main difference between YOLOv4 and earlier implementations is a focus on
speed. The new YOLO algorithm’s goal is that anyone having a decent GPU should
be able to apply the algorithm to achieve incredible accuracy for image
recognition running in real time. In Figure 7, the results from the comparison
between YOLOv4 and other architectures is shown, where it is apparent that
YOLOv4 outperforms most other neural networks in terms of average precision
and does so at more than twice the frame rate. This differentiates YOLOv4
greatly from the other algorithms, as it can be used in real time
classification of objects at a near human precision. For instance, the CNN can
differentiate between cars, bikes and trucks driving on a highway. With its
high speed and surprisingly good precision YOLO is widely adopted and as such
a success.

Figure 7:  Display of YOLOv4 vs. competing algorithms. v4 runs twice as fast
as EfficientDet with around the same performance. AP stands for average
precision and FPS stands for frames per second.  Source: Bochkovskiy et al.
[2020]

###  4.2 Tumor Detection in Mammography

One intriguing application of convolutional neural networks is within the
field of radiology. Each year millions of women go through treatment for
detection of cancers at an early stage through mammography. A paper published
in Nature Scientific Journal [Shen et al., 2019], shows that CNNs can already
detect cancers at an early stage with high accuracy. In fact, their algorithm
is in many ways already outperforming US doctors. The use of CNNs in diagnosis
can greatly reduce the cost for hospitals, making tests available to a wider
public and at the same time increase diagnostic precision.

The study presents a CNN that has the input of a mammographic picture and
outputs a location and presence of benign (not dangerous) or malignant (should
be treated) tumors. In medical diagnosis, _sensitivity_ refers to the ability
of correctly identifying a sick person, the _specificity_ refers to
identifying a healthy person correctly. In the US, the average sensitivity of
digital mammography screening is 86.9% with an average specificity of 88.9%.
As seen in Figure 8, some of the tumors being spotted are quite hard for even
a trained personnel to see.

Figure 8: A depiction of the neural network architecture applied, and three
mammography pictures, showing that the network quite clearly detects and
diagnoses tumors. Source: Shen et al. [2019]

The resulting CNN ran on multiple databases with an AUC (area under the ROC
curve) on the lowest quality pictures of 0.91. Trained on full-field high
resolution pictures of the INbreast database, their best model achieved a per-
image AUC of 0.95. Combining all of their models, this number was raised to
0.98 with a sensitivity of 86.7% and a specificity of 96.1%! This shows that
CNNs can perform tasks that are cost saving, but more importantly, lifesaving.
With the result of the paper in mind, this would mean that the CNN is
outperforming US medical personnel in detecting and classifying tumors. US
doctors are only .3% more likely to correctly spot a malignant tumor but have
a 7.2% less chance of correctly finding a patient to be healthy. This may
imply that US doctors tend to diagnose malignant tumors when in doubt. As a
last point, it is interesting to discuss the moral dilemma that arises from
this study. It is important to keep in mind that the 7.2% incorrectly
diagnosed with a malignant tumor are unnecessarily placed under a lot of
stress, chemotherapy and could potentially be scarred for life. Thus, the
paper indirectly touches on the moral dilemma – should technology replace
doctors as soon as it is better by some metric? As of March 2020, Savage
[2020] presents actual adoption of ML in hospitals as being in its infancy.

##  5 Conclusions and Further Work

CNNs are deep neural nets which utilize the convolutional operation to
materialize stunning results on problems of object detection and image
classification. The convolutional part of CNNs are what sets them apart, and
therefore the chosen focus of this paper. Furthermore, the paper laid out
theory behind why convolution is thought to be effective in signal processing,
and highlighted areas where improvements and optimizations are possible.

Of the many potential improvements, two have been discussed. First, locally
connected layers which keeps the local connectivity of convolutional nets, but
none of the parameter sharing. This results in a layer that can be adaptive to
cases where certain features are only relevant in certain parts of an input
map. Secondly, tiled convolution is in many ways a compromise between standard
convolution and locally connected layers. Tiled convolution brings back some
of the parameter sharing, but is still able to learn more complex invariances
in the input than the regular convolution. Use of the Fast Fourier Transform
can greatly improve computational efficiency in some cases because convolution
is just pair-wise multiplication in the frequency domain, which is a “cheaper”
operation. Also, many of the Fourier transforms can be computed once, and then
reused multiple times, which further increases the performance gains
attainable from using this approach.

Even though there are improvements to be made, applications of CNNs in
practice have proven to be very effective. In the world of fast image
recognition, the YOLO architecture is a state-of-the-art neural network which
accurately predicts bounding boxes around a variable number objects in near-
real time. Lastly, since tumor detection in mammography can be performed using
CNNs, and these are outperforming trained doctors by reporting fewer false
positives, one should already be asking the question if CNNs should replace
(or preferably aid) our medical experts. This demonstrates the high
proficiency of CNNs today, and that even though improvements can be made, CNNs
can already _help_ humanity solve some of its biggest challenges. In our
opinion, it is not an all-or-nothing issue. Doctors are still immensely
important, but technology can help them be more efficient by enabling them to
aid a larger number of people in a shorter amount of time, and with higher
quality. In our view, that is what Justin Bieber would call a _“No Brainer”_ ,
alternatively _“Best of Both Worlds”_ , as Hannah Montana would say.

The number of ways to apply CNNs seem to be endless, and new ideas and
improvements surface constantly. One idea could be to use a CNN, like YOLO, to
be an effective way to implement driving aids. YOLO’s ability to detect
objects like people or bicycles can be leveraged to increase safety on the
roads. A second application could be to use CNNs to automatically categorize
images (and the objects within them). Based on the output from the CNN, a
company like Instagram, could automatically generate hashtags and categorize
pictures of e.g. beaches or with dogs in them. The list of new applications
could go on and on, but to mention a couple more: unlocking doors with facial
recognition, automatic pet watching cameras, automatic pose detection in dance
games, fraud detection (by treating transactions as a signal, as a function of
time), and much more.

As a last note, we have some thoughts regarding the misuse of CNNs. As William
Lamb, member of the British Parliament, said in 1817: _“The possession of
great power necessarily implies great responsibility.”_ Joseph Redmon, the
creator of YOLO, left the project because he did not want to work on something
that he saw was used for military and aggressive marketing purposes. Another
example is OpenAI’s GPT language model (also utilizing deep neural nets, but
not convolutions specifically). The model became capable of producing e.g.
propaganda and hate speech at a much larger rate than humans, which could
potentially be weaponized. The question Redmon asked himself is important to
contemplate, even though one is easily awed and blinded by such brilliant
technology. It is also important to remember that technology, which is
basically a complex tool, is not evil, but it can be harnessed for evil
purposes. In a field that is as fast moving as AI and ML, it is very hard for
governments and regulatory agencies to be on top of all the developments,
which makes appropriate regulation lag behind the curve.

## References

  * Bochkovskiy et al. [2020] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao.  Yolov4: Optimal speed and accuracy of object detection, 2020.
  * Cisco [2020] Cisco.  Cisco annual internet report (2018–2023) white paper.  https://www.cisco.com/c/en/us/solutions/collateral/executive-perspectives/annual-internet-report/white-paper-c11-741490.html, 2020\.  Accessed: 18.09.2020.
  * Goodfellow et al. [2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  _Deep Learning_.  MIT Press, 2016.  http://www.deeplearningbook.org.
  * Kaelbling [2019] Leslie P. Kaelbling.  _MIT 6.036 - Introduction to Machine Learning - Lecture Notes_.  MIT, 2019.  https://phillipi.github.io/6.882/2020/notes/6.036_notes.pdf.
  * LeCun et al. [1989] Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jackel.  Backpropagation applied to handwritten zip code recognition.  _Neural Computation_ , 1:541–551, 1989.
  * Mathieu et al. [2014] Michaël Mathieu, Mikael Henaff, and Y. LeCun.  Fast training of convolutional networks through ffts.  _CoRR_ , abs/1312.5851, 2014.
  * Ngiam et al. [2010] Jiquan Ngiam, Zhenghao Chen, Daniel Chia, Pang W. Koh, Quoc V. Le, and Andrew Y. Ng.  Tiled convolutional neural networks.  In J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta, editors, _Advances in Neural Information Processing Systems 23_ , pages 1279–1287. Curran Associates, Inc., 2010.  URL http://papers.nips.cc/paper/4136-tiled-convolutional-neural-networks.pdf.
  * Redmon et al. [2016] Joseph Redmon, S. Divvala, Ross B. Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 779–788, 2016.
  * Sambasivarao [2020] K. Sambasivarao.  Non-maximum suppression (nms).  https://towardsdatascience.com/non-maximum-suppression-nms-93ce178e177c, 2020\.  Accessed: 18.09.2020.
  * Sanderson [2017] Grant Sanderson.  Gradient descent, how neural networks learn — deep learning, chapter 2\.  https://www.youtube.com/watch?v=IHZwWFHWa-w, 2017.  Screenshot from video, colors inverted for readability.
  * Savage [2020] N. Savage.  How ai is improving cancer diagnostics.  https://www.nature.com/articles/d41586-020-00847-2, 2020.  Accessed: 18.09.2020.
  * Shen et al. [2019] Li Shen, L. R. Margolies, J. H. Rothstein, E. Fluder, R. McBride, and W. Sieh.  Deep learning to improve breast cancer detection on screening mammography.  _Nature Research_ , 2019.
  * Soffer et al. [2018] S. Soffer, A. Ben-Cohen, O. Shimon, M. M. Amitai, H. Greenspan, and E. Klang.  Convolutional neural networks for radiologic images: A radiologist’s guide.  _Radiology_ , 2018.


# Histopathologic Cancer Detection

Varan Singh Rohila, Neeraj Lalwani, and Lochan Basyal  Stevens Institute of
Technology, Hoboken, U.S.A.
{vrohila, nlalwan1<EMAIL_ADDRESS>
###### Abstract

Early diagnosis of the cancer cells is necessary for making an effective
treatment plan and for the health and safety of a patient. Nowadays, doctors
usually use a histological grade that pathologists determine by performing a
semi-quantitative analysis of the histopathological and cytological features
of hematoxylin-eosin (HE) stained histopathological images. This research
contributes a potential classification model for cancer prognosis to
efficiently utilize the valuable information underlying the HE-stained
histopathological images. This work uses the PatchCamelyon benchmark datasets
and trains them in a multi-layer perceptron and convolution model to observe
the model’s performance in terms of precision, Recall, F1 Score, Accuracy, and
AUC Score. The evaluation result shows that the baseline convolution model
outperforms the baseline MLP model. Also, this paper introduced ResNet50 and
InceptionNet models with data augmentation, where ResNet50 is able to beat the
state-of-the-art model. Furthermore, the majority vote and concatenation
ensemble were evaluated and provided the future direction of using transfer
learning and segmentation to understand the specific features.

##  I Introduction

Cancer has become one of the major health concerns worldwide and early
diagnosis of cancer cells plays a vital role in the effective treatment
planning of the patient. Cancer screening using breast tissue biopsies aims to
distinguish between benign and malignant lesions. However, manual assessment
of large-scale histopathological images is a challenging task due to the
variance in appearance, heterogeneous structure, and textures[1]. Such a
manual analysis is laborious, time-consuming, and often dependent on
subjective human interpretation. For this reason, the concept of cancer
detection with the analysis of histopathological images with machine learning
algorithms provides a significant direction in the research of early cancer
diagnosis.

In recent years, deep learning outperformed state-of-the-art methods in
machine learning and medical image analysis tasks, including classification,
detection, segmentation, and computer-based diagnosis. The PatchCamelyon
benchmark dataset as histopathological images has been used for model training
and further computation in this research project. These datasets’ images are
extracted from the histopathologic scans of lymph nose sections. For
implementing the machine learning model, this paper mainly focuses on two
approaches: a multi-layer-perceptron model and a simple convolutional model
for image classification. The model’s performance is observed with the
Precision, Recall, F1 Score, Accuracy, and AUC Score.

The rest of the paper is organized so that related work has been discussed in
section II. Furthermore, the proposed solution includes the description of the
dataset, machine learning algorithms, and implementation details presented in
section III. Similarly, the model comparison and the future direction are
discussed in IV and V. The purpose of this research is concluded in section
VI.

Figure 1: Examples From the Dataset.

##  II Related Work

This section summarizes existing works in the field of histopathology cancer
detection and image classification in general.

###  II-A Histopathology Cancer Detection

Various image classification approaches have been proposed for automatic
cancer detection. The authors of[2] presented the concept of transfer learning
and deep feature extraction. The two models, AlexNet and Vgg16, are considered
for feature extraction, and AlexNet is used for further fine-tuning.
Furthermore, a support vector machine(SVM) is used for the classification of
the images into benign and malignant classes. This research used the BreaKHis
dataset for the experiment and calculated the accuracy scores for performance
evaluation.

Other authors[3] proposed an ensemble transfer learning (ETL) framework for
classifying well-differentiated, moderately differentiated, and poorly
differentiated cervical histopathological images. First, the author developed
a transfer learning structure based on Inception V3, Exception, VGG16, and
Resnet 50. Then, an ensemble learning strategy based on weighted voting was
introduced to improve classification performance. This research claims that
the problem of histopathological cancer detection can be solved by adopting
the concept of deep multiple instances learning that integrates deep
convolutional neural networks and multi-instance learning.

Continuing with the ensemble techniques, the authors of[4] proposed an
ensemble model that adapts three pre-trained CNNs, namely VGG19, MobileNet,
and DenseNet. The ensemble model was used for the feature extraction, and a
multi-layer perceptron classifier was used to perform the classification task.
Furthermore, diverse pre-processing and CNN tuning techniques, stain-
normalization, data augmentation, hyperparameter tuning, and fine-tuning were
used to train the model. The author of this research validated four publicly
available benchmark datasets, i;e., ICIAR, BreakHis, Patch-Cameleon, and
Bioimaging.

Lastly, this study[5] suggests a deep ensemble model for the binary
classification of breast histopathology pictures of benign and malignant
lesions based on image-level labeling. The training, validation, and test sets
of the BreaKHis dataset are randomly assigned. The numbers of benign and
cancerous samples are then balanced using data augmentation techniques. VGG16,
Xception, ResNet50, and DenseNet201 are chosen as base classifiers based on
their performance in transfer learning and the complementarity between
networks. Image-level binary classification in the ensemble network model is
accurate to 98.90%. This approach is experimentally evaluated on the same
dataset with the most recent MLP models to confirm its capabilities. In
classification tasks, the Ensemble model has a 5-20% advantage, highlighting
its extensive capabilities. Just as a few doctors should be referred to before
reaching any conclusion, these works showcase the importance of ensemble
learning in combining the prowess of different models.

To understand the scope and landscape of the issue at hand, this paper[6]
reviewed the detection methods of histopathological cancer cells and predicted
the future development trends to guide follow-up research. The paper focuses
on machine vision overcomes the disadvantages of traditional detection methods
in cancer detection and can help pathologists improve detection accuracy.
Furthermore, the author presented the workflow of the machine vision detection
system, image acquisition, pre-processing techniques, segmentation, feature
extraction, and classification. The system depicted the microscope-mounted
digital cameras or scanners used to obtain histopathological images. The pre-
processing of the images can be performed through enhancement and color
normalization techniques. The author reviewed the various segmentation
techniques, threshold segmentation, active contour, clustering, and watershed.
After segmentation, feature extraction can be performed with shape features,
HSV, and Gray symbiotic. In the last step of the machine vision detection
system, the author discussed the supervised and unsupervised classification
for finding Benign and Malignant. At every stage, a couple of unique
techniques can be applied for better overall results.

###  II-B Image Classification

Image classification refers to extracting useful information from an image and
then classifying it based on certain attributes. The most widely common image
classification task is the classification of cats and dogs. Earlier versions
of image classification models found even such trivial problems challenging.
The introduction of convolution operations and deep neural networks advanced
state-of-the-art image classification. Today, new architectures are evaluated
on the ImageNet dataset [7]. It has 1000 categories for image classification.
Recent state-of-the-art image classification models are:

  * $\bullet$

ResNet Deep convolution models have the issue of vanishing gradients where the
gradient becomes so small during backpropagation and weights aren’t updated.
To counter that issue, authors of [8] introduce skip connections in their
network, which adds the information from previous layers to the current layer,
thus reducing the problem of vanishing gradient descent.

  * $\bullet$

DenseNet Inspired by the skip connections introduced in [8], authors of [9]
use those skip connections by concatenation the previous layer with the
current layer. This has an increased performance from the addition of skip
layers of Resnet.

  * $\bullet$

InceptionNet Complexity of convolution models increases as the number of
filters increases. The authors of [10] introduce a 1x1 convolution operation
that helps change the network’s filter size. This reduces complexity and
offers greater control of the architecture.

  * $\bullet$

ViT or Vision Transformer Transformer models have proven successful in Natural
language problems. The authors of [11] utilize the transformer model in image
classification. To create the input embeddings, they divide the image into
small image patches, project them in linear space, and then encode their
position embeddings.

##  III Our Solution

###  III-A Description of Dataset

The dataset we considered to evaluate our approach is the PatchCamelyon
benchmark dataset [12]. It consists of 220,025 color images (96 x 96px)
extracted from histopathologic scans of lymph node sections. Each image is
annotated with a binary label indicating the presence of metastatic tissue. A
positive label indicates that the center 32x32px region of a patch contains at
least one pixel of tumor tissue. Tumor tissue in the outer region of the patch
does not influence the label. The images from the data are shown in 1. The
dataset is divided into the train, and test sets, with the test set being 25%
of the dataset. The data statistics are tabulated in Table I.

Statistics | Train Set | Test Set
---|---|---
Positive Labels | 66,837 | 22,280
Negative Labels | 98,181 | 32,727
Total | 165,018 | 55,007
TABLE I: Dataset Statistics Models | Precision | Recall | F1 Score | Accuracy | AUC Score
---|---|---|---|---|---
Baseline MLP | 0.677 | 0.567 | 0.617 | 0.715 | 0.774
Baseline Convolution | 0.791 | 0.731 | 0.760 | 0.812 | 0.875
ResNet50 | 0.950 | 0.932 | 0.941 | 0.952 | 0.988
InceptionNet | 0.929 | 0.920 | 0.925 | 0.925 | 0.983
Majority Vote | 0.905 | 0.969 | 0.936 | 0.946 | -
Concatenation Ensemble | 0.922 | 0.927 | 0.925 | 0.939 | 0.982
Model proposed in [4] | 0.957 | 0.952 | 0.955 | 0.946 | -
TABLE II: Baseline Results

###  III-B Machine Learning Algorithms

For our baseline models, we consider a multi-layer perceptron model and a
simple convolutional model for image classification. The input of both these
models is (96px, 96px, 3), where 3 corresponds to the RGB image channels.
Briefly describing the baseline models below:

  * $\bullet$

Multi-layer Perceptron Model: Simple ANN models have been proven to perform
well on image tasks such as the MNIST Handwriting Benchmark Dataset. Hence, we
consider a single-layer MLP model with 768 hidden nodes with relu activation.
A single node with sigmoid activation outputs the predictions.

  * $\bullet$

Convolution Model: A convolution is the simple application of a filter to an
input that results in activation. Repeated application of the same filter to
an input results in a map of activations called a feature map, indicating the
locations and strength of a detected feature in an input, such as an image. We
consider a single-layer convolutional model with 32 filters, (3, 3) filter
size with relu activation. After that, Max Pooling is applied with (2, 2)
filter size. After flattening and adding a single output node with sigmoid
activation, the output is calculated.

  * $\bullet$

ResNet50 Model: We utilize data augmentation with the ResNet50 model. To the
output of the ResNet50 model, we add a Global Max Pooling layer and a Global
Average Pooling Layer. Finally, these three outputs are concatenated and
attached to a single output node. A dropout layer of 0.2 is also added to the
model.

  * $\bullet$

Inception Model: Similar to the ResNet50 model, inception model is trained and
evaluated.

  * $\bullet$

Majority Vote Ensemble Model: We combine the ResNet50 and Inception model with
majority voting. This is a hard ensemble.

  * $\bullet$

Concatenation Ensemble Model: We combine the ResNet50 and Inception models
using concatenation and train them together as a joint Neural Network. This is
a soft ensemble.

The number of parameters of both the models is shown in figure 2 and figure 3,
respectively.

Figure 2: MLP Baseline Model.

Figure 3: Convolution Baseline Model.

Figure 4: ResNet50 Model.

Figure 5: InceptionNet Model.

###  III-C Implementation Details

The models are trained on the train set for 50 epochs and then evaluated on
the test set. Adam is the optimizer for both models, with a 0.0005 learning
rate for the baseline MLP and 0.001 for the baseline convolution model. For
the ResNet50, the Inception model, and the ensemble model, are trained with a
0.00003 learning rate. Early stopping with patience 5 is used to avoid
overfitting.

For data augmentation, random vertical and horizontal flips are applied to the
training data with a re-scale of 1/255.

##  IV Comparison

The models are tested on these metrics:

  * $\bullet$

Precision: Precision refers to the number of true positives divided by the
total number of positive predictions.

  * $\bullet$

Recall: The recall is the measure of our model correctly identifying True
Positives.

  * $\bullet$

F1 Score: It is the harmonic mean of Recall and Precision.

  * $\bullet$

Accuracy: Accuracy is the fraction of predictions our model got right.

  * $\bullet$

AUC Score: AUC score represents the degree or measure of separability.

The evaluation results are tabulated in table II. The baseline convolution
model outperforms the baseline MLP model in all metrics confirming the prowess
of simple convolution networks in image classification. Moreover, due to the
weight-sharing property of convolutions, the number of trainable parameters is
considerably less than in the baseline MLP model.

##  V Future Directions

In this work, we evaluate the performances of baseline models on the Patch-
Cameleyon dataset. The classification performances can be further increased by
using the following:

  * $\bullet$

Transfer Learning: Transfer learning refers to the technique of applying
already learned knowledge from one application area to another with shared
weights and retraining some layers of the model.

  * $\bullet$

Segmentation: In medical image applications, segmentation extracts useful
information about the subject, i.e., the tissue, which can help the model to
specific features and ensures the model’s focus.

##  VI Conclusion

Automatic detection of cancer tissues can help doctors identify tumor tissues
and act accordingly. This paper proposes and evaluates baseline models for
detecting cancer tissue in lymph nodes. The baseline convolution model
achieves an accuracy of 81.2%, which is quite good for a single-layer model
indicating bigger, better models equipped with other techniques such as data
augmentation, transfer learning, and segmentation can achieve state-of-the-art
performance on this benchmark dataset. As we advance with that idea, ResNet50
and InceptionNet models were trained with data augmentation. Finally, their
majority and Concatenation ensemble were also evaluated on the dataset.
Extensive experimentation showcases that the ResNet50 model trained with
proper data augmentation is able to beat the state-of-the-art model. In the
future, transfer learning and segmentation techniques can be utilized to
create a more general and efficient model.

## References

  * [1] Li, Chao, et al. ”Weakly supervised mitosis detection in breast histopathology images using concentric loss.” Medical image analysis 53 (2019): 165-178.
  * [2] Deniz, Erkan, et al. ”Transfer learning based histopathologic image classification for breast cancer detection.” Health information science and systems 6.1 (2018): 1-7.
  * [3] Udendhran, R., B. Sreedevi, and G. Sneha. ”A Unified and Semantic Model Approach for Histopathologic Cancer Detection Based on Deep Double Transfer Learning.” 2022 Second International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies (ICAECT). IEEE, 2022.
  * [4] Kassani, Sara Hosseinzadeh, et al. ”Classification of histopathological biopsy images using ensemble of deep learning networks.” arXiv preprint arXiv:1909.11870 (2019).
  * [5] Yuchao Zheng, Chen Li, Xiaomin Zhou, Haoyuan Chen, Hao Xu, Yixin Li, Haiqing Zhang, Xiaoyan Li, Hongzan Sun, Xinyu Huang, Marcin Grzegorzek. ”Application of Transfer Learning and Ensemble Learning in Image-level Classification for Breast Histopathology, Intelligent Medicine” (2022) ISSN 2667-1026.
  * [6] He, Wenbin, et al. ”Progress of Machine Vision in the Detection of Cancer Cells in Histopathology.” IEEE Access 10 (2022): 46753-46771.
  * [7] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (pp. 248–255).
  * [8] He, Kaiming, et al. ”Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
  * [9] Huang, G., et al. ”Densely Connected Convolutional Networks. arXiv. 2016 doi: 10.48550.” arXiv preprint arXiv.1608.06993 1608.
  * [10] Szegedy, Christian, et al. ”Going deeper with convolutions. 2014.” arXiv preprint arXiv:1409.4842 10 (2014).
  * [11] Dosovitskiy, Alexey, et al. ”An image is worth 16x16 words: Transformers for image recognition at scale.” arXiv preprint arXiv:2010.11929 (2020).
  * [12] Veeling, Bastiaan S., et al. ”Rotation equivariant CNNs for digital pathology.” International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2018.