Diagnosis and Analysis of Celiac Disease and Environmental Enteropathy on
Biopsy Images using Deep Learning Approaches

A Dissertation

Presented to

the faculty of the School of Engineering and Applied Science

University of Virginia

in partial fulfillment

of the requirements for the degree

Doctor of Philosophy

by

Kamran Kowsari

May

2020

Approval Sheet
This Dissertation is submitted in partial fulfillment of the requirements for
the degree of Doctor of Philosophy (Systems and Information Engineering)

Kamran Kowsari

This Dissertation has been read and approved by the Examining Committee:

Laura Barnes, Ph.D. (School of Engineering and Applied Science)

Donald Brown, Ph.D. (School of Engineering and Applied Science)

Michael Porter, Ph.D. (School of Engineering and Applied Science)

Sana Syed, MD, MS (School of Medicine)

Peter Beling, Ph.D. (School of Engineering and Applied Science)

Accepted for the School of Engineering and Applied Science:

Craig H. Benson, Dean, School of Engineering and Applied Science

May 2020

©Copyright by Kamran Kowsari 2020

All Rights Reserved

## Abstract

Celiac Disease (CD) and Environmental Enteropathy (EE) are common causes of
malnutrition and adversely impact normal childhood development. Both
conditions require a tissue biopsy for diagnosis and a major challenge of
interpreting clinical biopsy images to differentiate between these
gastrointestinal diseases is striking histopathologic overlap between them. In
the current study, we propose four diagnosis techniques for these diseases and
address their limitations and advantages.

First, the diagnosis between CD, EE, and Normal biopsies is considered, but
the main challenge with this diagnosis technique is the staining problem. The
dataset used in this research is collected from different centers with
different staining standards. To solve this problem, we use color balancing in
order to train our model with a varying range of colors.

Random Multimodel Deep Learning (RMDL) architecture has been used as another
approach to mitigate the effects of the staining problem. RMDL combines
different architectures and structures of deep learning and the final output
of the model is based on the majority vote.

CD is a chronic autoimmune disease that affects the small intestine
genetically predisposed children and adults. Typically, CD rapidly progress
from Marsh I to IIIa. Marsh III is sub-divided into IIIa (partial villus
atrophy), Marsh IIIb (subtotal villous atrophy) and Marsh IIIc (total villus
atrophy) to explain the spectrum of villus atrophy along with crypt
hypertrophy and increased intraepithelial lymphocytes. In the second part of
this study, we proposed two ways for diagnosing different stages of CD.

Finally, in the third part of this study, these two steps are combined as
Hierarchical Medical Image Classification (HMIC) to have a model to diagnose
the disease data hierarchically.

## Acknowledgements

It is a great pleasure to thank my committee members for their time and
support. First, I would like to express my sincere gratitude to my advisor,
Dr. Laura Barnes, for her constant encouragement and guidance throughout my
study at UVa. Without her support, I wouldn’t be able to complete this
dissertation. Also, I would like to thank my co-advisor Donald Brown for his
advice and help on my graduate study. To my wife, Sanam: thank you for
everything you did and said for the last five years. To my mother, Giti, your
support has been the driving motive for me to work hard and succeed. To my
siblings and family: thank you for your support.

###### Table of Contents

  1. Acknowledgements
  2. 1 Introduction
  3. 2 Literature Review
    1. 2.1 Diagnosis of Celiac Disease and Environmental Enteropathy on Biopsy Images
      1. 2.1.1 Automated Detection of Celiac Disease using Deep Learning
      2. 2.1.2 Automated Detection of Celiac Disease Using Traditional Technique
    2. 2.2 Image Classification Techniques
      1. 2.2.1 Logistic Regression
        1. 2.2.1.1 Basic Framework
      2. 2.2.2 K-Nearest Neighbor
        1. 2.2.2.1 Basic Concept of k-NN
      3. 2.2.3 Support Vector Machine (SVM)
      4. 2.2.4 Decision Tree
      5. 2.2.5 Ensemble techniques
        1. 2.2.5.1 Random Forest
        2. 2.2.5.2 Boosting
        3. 2.2.5.3 Bagging
      6. 2.2.6 Deep Learning:
    3. 2.3 Autoencoder
      1. 2.3.1 General Framework
      2. 2.3.2 Conventional Autoencoder Architecture
      3. 2.3.3 Recurrent Autoencoder Architecture
    4. 2.4 Medical Image Staining Problem for Machine Learning Algorithms
      1. 2.4.1 Medical Image Staining
      2. 2.4.2 Staining Problem for Machine Learning Algorithm
      3. 2.4.3 Color Normalization and Color Balancing
    5. 2.5 Evaluation Techniques
      1. 2.5.1 Macro-Averaging and Micro-Averaging
      2. 2.5.2 Fβ Score
      3. 2.5.3 Matthews Correlation Coefficient (MCC)
      4. 2.5.4 Receiver Operating Characteristics (ROC)
      5. 2.5.5 Area Under ROC Curve (AUC)
    6. 2.6 Summary
  4. 3 Diagnosis of Celiac Disease and Environmental Enteropathy on Biopsy Images Using Color Balancing on Convolutional Neural Networks
    1. 3.1 Introduction and Related Works
    2. 3.2 Data Source
    3. 3.3 Pre-Processing
      1. 3.3.1 Image Patching
      2. 3.3.2 Clustering
        1. 3.3.2.1 Autoencoder
        2. 3.3.2.2 K-Means
      3. 3.3.3 Color Balancing
    4. 3.4 Method
      1. 3.4.1 Convolutional Layer
      2. 3.4.2 Pooling Layer
      3. 3.4.3 Neuron Activation
      4. 3.4.4 Optimizer
      5. 3.4.5 Network Architecture
    5. 3.5 Empirical Results
      1. 3.5.1 Evaluation Setup
      2. 3.5.2 Experimental Setup
      3. 3.5.3 Experimental Results
    6. 3.6 Conclusion
  5. 4 Random Multimodel Deep Learning (RMDL) for Medical Image Classification
    1. 4.1 Introduction
    2. 4.2 Method
      1. 4.2.1 Feature Extraction and Data Pre-processing
        1. 4.2.1.1 Image and 3D Object Feature Extraction
      2. 4.2.2 Random Multimodel Deep Learning
      3. 4.2.3 Deep Learning in RMDL
        1. 4.2.3.1 Deep Neural Networks
        2. 4.2.3.2 Recurrent Neural Networks (RNN)
          1. 4.2.3.2.1 Long Short-Term Memory (LSTM)
          2. 4.2.3.2.2 Gated Recurrent Unit (GRU):
        3. 4.2.3.3 Convolutional Neural Networks (CNN)
      4. 4.2.4 Optimization
        1. 4.2.4.1 Stochastic Gradient Descent (SGD) Optimizer
        2. 4.2.4.2 Adam Optimizer
        3. 4.2.4.3 Multi Optimization rule
    3. 4.3 Limitation and Discussion
    4. 4.4 Results of RMDL for Medical Images
      1. 4.4.1 Data
        1. 4.4.1.1 Data Source
        2. 4.4.1.2 Pre-Processing
          1. 4.4.1.2.1 Image Patching
          2. 4.4.1.2.2 Patch Clustering
      2. 4.4.2 Empirical results
      3. 4.4.3 Hardware and Framework
    5. 4.5 Conclusion
  6. 5 Celiac Disease Severity Diagnosis on Duodenal Histopathological Images Using Deep Learning
    1. 5.1 Introduction
    2. 5.2 Severity Classes of Celiac Disease
    3. 5.3 Data Source
    4. 5.4 Data Pre-processing
      1. 5.4.1 Image Patching
      2. 5.4.2 Patch Clustering
      3. 5.4.3 Stain Normalization
    5. 5.5 Methodology
      1. 5.5.1 Model Development
      2. 5.5.2 Whole slide classification
    6. 5.6 Experimental results
      1. 5.6.1 Patch-level performance
      2. 5.6.2 Slide-level performance
      3. 5.6.3 Class Activation Mapping
      4. 5.6.4 Hardware and Framework
    7. 5.7 Conclusion
  7. 6 Hierarchical Medical Image Classification (HMIC)
    1. 6.1 Introduction and Related Works
    2. 6.2 Data Source
    3. 6.3 Pre-Processing
      1. 6.3.1 Image Patching
      2. 6.3.2 Clustering
        1. 6.3.2.1 Autoencoder
        2. 6.3.2.2 K-Means
      3. 6.3.3 Staining Problem
        1. 6.3.3.1 Color Balancing
        2. 6.3.3.2 Stain normalization
    4. 6.4 Baseline
      1. 6.4.1 Deep Convolutional Neural Networks
      2. 6.4.2 Deep Neural Networks
    5. 6.5 Method
      1. 6.5.1 Convolutional Neural Networks
        1. 6.5.1.1 Convolutional Layer
        2. 6.5.1.2 Pooling Layer
        3. 6.5.1.3 Neuron Activation
        4. 6.5.1.4 Optimizer
        5. 6.5.1.5 Network Architecture
      2. 6.5.2 Whole Slide Classification
      3. 6.5.3 Hierarchical Medical Image Classification
    6. 6.6 Results
      1. 6.6.1 Evaluation Setup
      2. 6.6.2 Experimental Setup
      3. 6.6.3 Empirical Results
      4. 6.6.4 Visualization
    7. 6.7 Conclusion
  8. 7 Conclusions & Future Directions
    1. 7.1 Summary of Contributions
      1. 7.1.1 Two approaches for diagnosing celiac disease and Environmental Enteropathy
      2. 7.1.2 Two Approaches for Diagnosis of Duodenal Histopathological images Celiac Disease
      3. 7.1.3 Hierarchical Medical Image Classification (HMIC)
    2. 7.2 Summary of Future Works

###### List of Tables

  1. 2.1 Metrics Pitfalls
  2. 3.1 The clustering results for all patches into two clusters
  3. 3.2 CNN Architecture for Diagnosis of Diseased Duodenal on Biopsy Images
  4. 3.3 F1-score for train on a set with color balancing of 0.001, 0.01, 0.1, and 1.0. Then, we evaluate test set with same color balancing
  5. 3.4 F1-score for train with color balancing of 0.001, 0.01, 0.1, and 1.0 and test with color balancing of 0.5, 1.0, 1.5 and 2.0
  6. 3.5 Comparison accuracy with different baseline methods
  7. 4.1 Results of RMDL per Class for 3, 9 and 15 Random models models
  8. 4.2 Results of RMDL in comparison with our baselines
  9. 5.1 Architecture of the model
  10. 5.2 Patch-level performance of model for celiac disease severity diagnosis Based on the two Deep learning techniques which are ResNet and CNN. The model performances for I and IIIa ResNet is more robust but for IIIb and IIIc
  11. 6.1 Population results of biopsies dataset
  12. 6.2 Dataset is used for Hierarchical Medical Image Classification (HMIC)
  13. 6.3 Result of parent level classifications for normal, environmental enteropathy, and Celiac disease
  14. 6.4 Results of HMIC with comparison with our baseline
  15. 6.5 Results per-classed of HMIC with comparison with our baseline

###### List of Figures

  1. 2.1 Wei et. al [1] works on celiac disease on whole‑slide biopsy images
  2. 2.2 S. Syed et. al [2] works on Celiac Disease and Environmental Enteropathy on biopsy images using CNN
  3. 2.3 A architecture of k-NN model for 2D dataset and three classes
  4. 2.4 This figure shows the linear and non-linear SVM for 2D dataset (for text data we have thousands of dimitions )
  5. 2.5 Random Forest
  6. 2.6 This figure is the Boosting technique architecture
  7. 2.7 This figure shows a simple model of Bagging technique
  8. 2.8 This figure shows how a simple autoencoder works. The model depicted contains the following layers: $Z$ is code and two hidden layers are used for encoding and two are used for decoding.
  9. 2.9 Hematoxylin and eosin (H&E) staining ith 0 (control, a and b), 50 mg/L (b and e) and 200 mg/L HgCl2 (c and f). Panels (a-c) are shown in low magnification (400×) while (d-f) in high magnification (1000×). [3].
  10. 2.10 Different H&E Color staining
  11. 2.11 Confusion matrix
  12. 3.1 Overview of methodology
  13. 3.2 Structure of clustering model with autoencoder and K-means combination
  14. 3.3 Some samples of clustering results - cluster 1 includes patches with useful information and cluster 2 includes patches without useful information (mostly created from background parts of WSIs)
  15. 3.4 Color Balancing samples for the three classes
  16. 3.5 Structure of Convolutional Neural Net using multiple 2D feature detectors and 2D max-pooling
  17. 3.6 Our Convolutional Neural Networks’ Architecture
  18. 3.7 Receiver operating characteristics (ROC) curves for three classes also the figure shows micro-average and macro-average of our classifier
  19. 4.1 Overview of RDML: Random Multimodel Deep Learning for classification that includes $n$ Random models which are $d$ random model of the Multilayer Perceptron classifiers, $c$ models of CNN classifiers, and $r$ RNN classifiers where $r+c+d=n$.
  20. 4.2 Random Multimodel Deep Learning (RDML) architecture for classification which includes $3$ Random models, a Multilayer Perceptron classifier at left, a Deep CNN classifier at middle, and a Deep RNN classifier at right (each unit could be LSTM or GRU).
  21. 4.3 left Figure is a cell of GRU, and right Figure is a cell of LSTM
  22. 4.4 This figure Shows multi SGD optimizer
  23. 4.5 This Figure indicates loss function and accuracy for 3 Random Deep Learning (RDL) model for 100 epochs.
  24. 4.6 This Figure indicates loss function and accuracy for 9 Random Deep Learning (RDL) model for 100 epoch
  25. 4.7 This Figure indicates loss function and accuracy for 15 Random Deep Learning (RDL) model for 100 epoch
  26. 5.1 CD severity classification based on modified Marsh score [4]
  27. 5.2 Color normalization artifacts when using the method proposed by Vahadane et al. [5]. Images in the first row represent the target image and some source images. Their associated normalized images are in second row
  28. 5.3 Building blocks of (left) a traditional CNN, (right) a ResNet
  29. 5.4 Comparison of F1-score results for four classes between ResNet and CNN
  30. 5.5 Patch-level ROC and AUC for different classes
  31. 5.6  Class activation mapping heat maps highlighting the most informative regions of patches relevant to different categories including goblet cells, inflammatory cells in lamina propria, crypt epithelium, fibrotic inflammatory debris, surface enterocytes, Paneth cells, Brunner’s glands, neuroendocrine cells, villus edge and apposed enterocytes. Area of attention is shown in blue color.
  32. 6.1 Hierarchical Medical Image Classification
  33. 6.2 Pipeline of patching and autoencoder to find useful patches for training model. The biopsy images are very large, so we need to divide into smaller patches to be used in the machine learning model. As you can see in the image, many of these patches do not contain any useful medical information. After using an autoencoder, we can apply a clustering algorithm to discard useless patches (green patches contain useful information, while red patches do not).
  34. 6.3 Hierarchical Medical Image classification (HMIC) to diagnosis Celiac Disease and Environmental Enteropathy and child level of Celiac Disease Severity Diagnosis on Duodenal Histopathological Images
  35. 6.4 Some samples of clustering results - cluster 1 includes patches with useful information and cluster 2 includes patches without useful information (mostly created from background parts of WSIs)
  36. 6.5 Structure of Convolutional Neural Net using multiple 2D feature detectors and 2D max-pooling
  37. 6.6 Overview of whole-slide inference process using aggregation of patch-level classifications
  38. 6.7 Results of Grad-CAM for showing feature importance.

##  Chapter 1 Introduction

Celiac Disease (CD) is a chronic autoimmune disease that affects the small
intestine in genetically predisposed children and adults [6, 7, 2]. Gluten
exposure triggers an inflammatory cascade which leads to compromised
intestinal barrier function. If this enteropathy is unrecognized, it can lead
to anemia, decreased bone density and in longstanding cases, intestinal
cancer. Celiac Disease and Environmental Enteropathy (EE) [8] are common
causes of malnutrition and adversely impact normal childhood development. CD
is an autoimmune disorder that is prevalent worldwide and is caused by an
increased sensitivity to gluten. Gluten exposure destructs the small
intestinal epithelial barrier, resulting in nutrient mal-absorption and
childhood under-nutrition. EE also results in barrier dysfunction but is
thought to be caused by an increased vulnerability to infections. EE has been
implicated as the predominant cause of under-nutrition, oral vaccine failure,
and impaired cognitive development in low-and-middle-income countries. Both
conditions require a tissue biopsy for diagnosis, and a major challenge of
interpreting clinical biopsy images to differentiate between these
gastrointestinal diseases is striking histopathologic overlap between them.
Studies have shown the ease of training Convolutional Neural Networks (CNNs)
for image recognition. These networks are an iterative family of machine
learning architectures. CNNs have proven to have superior performance over a
wide range of computer vision tasks such as classification and object
detection. Due to the wide availability of robust open-source software and
high-quality public datasets, these architectures are becoming the popular
choice for being selected as the backbone of many modern computer vision
technologies. Using large amounts of data, these models have shown to be
effective in solving many biomedical imaging challenges. Currently, CNNs have
been successfully applied to medical images such as MRI and X-rays [9, 10].
The CNNs have also shown promising performance on histopathological images [6,
11].

In this dissertation, we propose four techniques for diagnosis of celiac
disease and Environmental Enteropathy based on biopsy images. The first
technique uses Color Balancing (CB) on Convolutional Neural Networks (CNN).
The second method is using Random Multimodel Deep Learning (RMDL) for the
diagnosis of celiac disease and Environmental Enteropathy. The third technique
is used to diagnosis only in the stage of Celiac Disease Severity Diagnosis on
Duodenal Histopathological images using Deep Learning (DL). Finally, we use an
approach we call Hierarchical Medical Image classification (HMIC) to combine
these models as the hierarchical representations of theses biopsy images. In
this dissertation, two main ideas have been covered to the diagnosis of
diseased duodenal architecture on biopsy images. We proposed a data-driven
model for diagnosis of diseased duodenal architecture on biopsy images using
color balancing on convolutional neural networks. The validation results of
this model show that it can be utilized by pathologists in diagnostic
operations regarding CD and EE. The second approach called RMDL, is used for
the medical image classification. It combines multiple deep learning
architectures to produce random classification models. We investigated CD
severity by applying Shallow CNNs and residual neural network architecture to
histopathological images.

In this dissertation, we propose a technique called Hierarchical Medical Image
Classification (HMIC), that combines multiple deep learning approaches to
produce hierarchical classifications. The method that is presented in this
dissertation described here can be improved in multiple ways. Additional
training and testing with other hierarchically structured medical data sets
will continue to identify architectures that work best for these problems.
Also, it is possible to extend the hierarchy to more than two levels to
capture more of the complexity in the hierarchical classification.

The rest of the dissertation is organized into five chapters which includes a
literature review in Chapter 2, discussing related research on Diagnosis of
Celiac Disease and Environmental Enteropathy using Biopsy Images, the other
kind of existing classification techniques (traditional techniques). Chapter 2
also includes a summary of the existing medical image classification
techniques, and finally, evaluation techniques have been discussed.

In Chapter 3, we talk about the technique to the diagnosis of Celiac Disease
and Environmental Enteropathy on biopsy images using Color Balancing on
Convolutional Neural Networks (CNN).

In Chapter 4, Random Multimodel Deep Learning (RMDL) is introduced. We apply
this technique on resized biopsy images.

In Chapter 5, Celiac Disease Severity diagnosis on Duodenal Histopathological
Images is discussed with using Deep Learning approaches. In this Chapter, we
use two popular techniques call Shallow Convolutional Neural Networks (CNN)
and deep residual networks (ResNet) with color normalization. All of data in
this section is collected from one center so, we did not use color balancing
in this section.

In Chapter 6, Hierarchical Medical Image classification (HMIC) is introduced
which is a hierarchical deep learning approach. This Chapter in a combination
of Chapter 3 and Chapter 5 present hierarchical representations of biopsy
images.

##  Chapter 2 Literature Review

### 2.1 Diagnosis of Celiac Disease and Environmental Enteropathy on Biopsy
Images

Under-nutrition is the underlying cause of approximately $45$% of the $5$
million under $5$-year-old childhood deaths annually in low and middle-income
countries (LMICs) [12] and is a major cause of mortality in this population.
Linear growth failure (or stunting) is a major complication of under-
nutrition, and is associated with irreversible physical and cognitive
deficits, with profound developmental implications [13]. A common cause of
stunting in LMICs is EE, for which there are no universally accepted, clear
diagnostic algorithms or non-invasive biomarkers for accurate diagnosis [13],
making this a critical priority [14]. EE has been described to be caused by
chronic exposure to enteropathogens which results in a vicious cycle of
constant mucosal inflammation, villous blunting, and a damaged epithelium
[13]. These deficiencies contribute to a markedly reduced nutrient absorption
and thus under-nutrition and stunting [13]. Interestingly, CD, a common cause
of stunting in the United States, with an estimated $1$% prevalence, is an
autoimmune disorder caused by a gluten sensitivity [15] and has many shared
histological features with EE (such as increased inflammatory cells and
villous blunting) [13]. This resemblance has led to the major challenge of
differentiating clinical biopsy images for these similar but distinct
diseases. Therefore, there is a major clinical interest towards developing
new, innovative methods to automate and enhance the detection of morphological
features of EE versus CD, and to differentiate between diseased and healthy
small intestinal tissue [16].

We propose a CNN-based model for the classification of biopsy images. In
recent years, Deep Learning architectures have received great attention after
achieving state-of-the-art results in a wide variety of fundamental tasks such
as classification [17, 18, 19, 20, 21, 10, 22, 23, 24] or other medical
domains [25, 26]. CNNs in particular have proven to be very effective in
medical image processing. CNNs preserve local image relations, while reducing
dimensionality and for this reason are the most popular machine learning
algorithm in image recognition and visual learning tasks [27]. CNNs have been
widely used for classification and segmentation in various types of medical
applications such as histopathological images of breast tissues, lung images,
MRI images, medical X-Ray images, etc. [9, 10]. Researchers produced advanced
results on duodenal biopsies classification using CNNs [11], but those models
are only robust to a single type of image stain or color distribution. Many
researchers apply a stain normalization technique as part of the image pre-
processing stage to both the training and validation datasets [28]. In this
section, varying levels of color balancing were applied during image pre-
processing in order to account for multiple stain variations.

#### 2.1.1 Automated Detection of Celiac Disease using Deep Learning

Wei et. al [1] introduced an automated detection technique for Celiac Disease
on duodenal biopsy. As shown in Figure 2.1, they convert their biopsy to
patches and use deep residual network (ResNet-50) model to classify each
patch, and they filtered out noise using thresholding to discard predictions
of low confidence.

Figure 2.1: Wei et. al [1] works on celiac disease on whole‑slide biopsy
images

S. Syed et. al [2, 29], use Convolutional Neural Network (CNN) on duodenal
biopsies of patients with environmental enteropathy, celiac disease, and
histologically normal controls which automatically learned microlevel features
in duodenal tissue, such as alterations in secretory cell populations. As
shown in Figure 2.2, this model uses 4 Convolutional layers and followed by a
pooling layer and connected to the fully connected Neural Network layer.

Figure 2.2: S. Syed et. al [2] works on Celiac Disease and Environmental
Enteropathy on biopsy images using CNN

#### 2.1.2 Automated Detection of Celiac Disease Using Traditional Technique

A. Ermarth et. al [30] works on the Identification of Pediatric Patients With
Celiac Disease based on a classification and regression tree Analysis (CART).
CART The CART is a predictive model that can be represented as a decision
tree, and it recursively partitions the dataset by doing binary splits to find
the optimal tree [31].

K. Kowal [32] presents an automatic computer system a set of 84 features
extracted from the nuclei is used in the classification by the k-nearest
neighbor (kNN) classifier. In this research, this technique was tested on 450
microscopic images from biopsies, and the proposed computer-aided diagnosis
(CAD) system is able to classify with reasonable accuracy.

E. Radiya et. al. [33], works on Active Feature Extraction (CAFE) consisting
of two logistic regression algorithms on biopsy images from 392 biomarkers,
and they get acceptable AUC when trained on data from one hospital and tested
on samples of the other.

### 2.2 Image Classification Techniques

Many techniques are used in the past decades to diagnosis celiac disease such
as SVM [34], Random Forest [35], ensemble learning [36], decision-tree [37],
Logistic Regression [38], etc. These days, deep learning to diagnosis this
disease from Normal and Environmental Enteropathy (EE) is very popular [2, 7,
39, 6]. In this sub-section, we describe the popular image classification
methods within medical domain. Over the last 50 years, many supervised
learning such as image classification techniques have been developed and
implemented in software to accurately label data. For example, the
researchers, K. Murphy in 2006 [40] and I. Rish in 2001 [41] introduced the
Naïve Bayes Classifier (NBC) as a simple approach to the more general
representation of the supervised learning classification problem and this
technique was developed for image classification task [42]. This approach has
provided a useful technique for data classification and information retrieval
applications. As with most supervised learning classification techniques, NBC
takes an input vector of numeric or categorical data values and produce the
probability for each possible output labels. Another popular classification
technique in 1980s and 1990s was Support Vector Machines (SVM), which has
proven quite accurate over a wide variety of data. This technique constructs a
set of hyper-planes in a transformed feature space. This transformation is not
performed explicitly but rather through the kernal trick which allows the SVM
classifier to perform well with highly nonlinear relationships between the
predictor and response variables in the data. A variety of approaches have
been developed to further extend the basic methodology and obtain greater
accuracy. C. Yu et. al. in 2009 [43] introduced latent variables into the
discriminative model as a new structure for SVM, and S. Tong et. al. in 2001
[44] added active learning using SVM for data classification. For a large
volume of data and datasets with a huge number of features (such as image or
any kind of datasets), SVM implementations are computationally complex.
Another technique that helps mediate the computational complexity of the SVM
for classification tasks is stochastic gradient descent classifier
(SGDClassifier) [45] which has been widely used in image classification.
SGDClassifier is an iterative model for large datasets. The model is trained
based on the SGD optimizer iteratively.

#### 2.2.1 Logistic Regression

One of the earliest methods of classification is logistic regression (LR) that
is used for diagnosis CD [38]. LR was introduced and developed by statistician
David Cox in 1958 [46]. LR is a linear classifier with decision boundary of
$\theta^{T}x=0$. LR predicts probabilities rather than classes [47, 48]. Many
medical image classification application was developed by using LR [49].

##### 2.2.1.1 Basic Framework

The goal of LR is to trained from the probability of variable $Y$ being 0 or 1
given $x$. Let’s have data data which is $X\in\mathbb{R}^{n\times d}$. If we
have binary classification problems, the Bernoulli mixture models function
should be used [50].

##### Combining Instance-based Learning and LR:

The LR model specifies the probability of binary output $y_{i}=\\{0,1\\}$
given the input $x_{i}$. To obey the basic principle underlying instance-based
learning (IBL) [51], the classifier should be a function of the distance
$\delta_{i}$. $p$ will be large if $\delta_{i}\rightarrow 0$ then $y_{i}=+1$,
and small for $y_{i}=-1$. $p$ should be close to $1$ if
$\delta_{i}\rightarrow\infty$; then, neither in favor of $y_{0}=+1$ nor in
favor of $y_{0}=-1$. Multinomial (or multilabeled) logistic classification
[52] uses the probability of $x$ belonging to class $i$. In a classification
task as supervised learning context, the component of $\theta$ is calculated
from the subset of the training data $D$ which belongs to class $i$ where
$i\in\\{1,\ldots,n\\}$. To perform maximum likelihood (ML) estimation of
$\theta$, we need to maximize the log-likelihood function [21].

#### 2.2.2 K-Nearest Neighbor

The k-nearest-neighbors algorithm (k-NN) is a non-parametric technique used
for classification. This method is used for data classification applications
such as medical image classification in many research domains [53] in past
decades.

##### 2.2.2.1 Basic Concept of k-NN

Given a test set $x$, the k-NN algorithm finds the $k$ nearest neighbors of
$x$ among all training set, and scores the category candidates based the class
of $k$ neighbors. The similarity of $x$ and each neighbor’s images could be
the score of the category of the neighbor images. Multiple k-NN images may
belong to the same category; in this case, the summation of these scores would
be the similarity score of the class $k$ with respect to the test image $x$.
After sorting the score values, the algorithm assigns the candidate to the
class with the highest score from the test image $x$ [53]. The Figure 2.3
illustrates the k-NN architecture, but for simplicity, this figure is designed
by 2D dataset (similar with higher dimensional space). The decision rule of
k-NN is:

$\begin{split}f(x)=&arg\max_{j}S(x,C_{j})\\\ =&\sum_{d_{i}\in
k-NN}sim(x,d_{i})y(d_{i},C_{j})\end{split}$ (2.1)

where S refers to score value with respect to $S(x,C_{j})$, the score value of
candidate $i$ to class of $j$, and output of $f(x)$ is a label to the test set
image.

Figure 2.3: A architecture of k-NN model for 2D dataset and three classes

#### 2.2.3 Support Vector Machine (SVM)

The original version of SVM was developed by Vapnik and Chervonenkis [54] in
1963. BE. Boser et al. [55] adapted this version into a nonlinear formulation
in the early 1990s. SVM was originally designed for binary classification
tasks. However, many researchers work on multi-class problems using this
dominate technique [56].

In 2015, Z. Camlica [57] designed a medical image classification using SVM
which this technique using using local binary patterns (LBP) features from
saliency-based folded data.

Figure 2.4: This figure shows the linear and non-linear SVM for 2D dataset
(for text data we have thousands of dimitions )

In the context of image classification, let $x_{1},x_{2},...,x_{l}$ be
training examples belonging to one class $X$, where X is a compact subset of
$R^{N}$ [58]. Since SVMs are traditionally used for the binary classification,
we need to generate a Multiple-SVM (MSVM) [59] for multi-class problems. One-
vs-One is a technique for multi-class SVM that builds N(N-1) classifiers. The
natural way to solve the k-class problem is to construct a decision function
of all $k$ classes at once [60, 61]. Another technique of multi-class
classification using SVM is All-vs-One.

#### 2.2.4 Decision Tree

One earlier classification algorithm for image and data mining is decision
tree [62]. This techniques is used for analysis the risk of celiac disease in
children [37]. Decision tree classifiers (DTCs) are used successfully in many
diverse areas for classification [63]. The structure of this technique is a
hierarchical decomposition of the data space [62, 64]. Decision tree as
classification task was introduced by D. Morgan [65] and developed by J.R.
Quinlan [66]. The main idea is creating a tree based on the attribute for
categorized data points, but the main challenge of a decision tree is which
attribute or feature could be in parents’ level and which one should be in
child level. To solve this problem, De Mántaras [67] introduced statistical
modeling for feature selection in tree.

#### 2.2.5 Ensemble techniques

Voting classification techniques, such as Bagging, Boosting, Random Forest,
and RMDL have been successfully developed for classification [68] tasks. While
Boosting adaptively changes the distribution of the training set based on the
performance of previous classifiers, Bagging does not look at the previous
classifier[69].

##### 2.2.5.1 Random Forest

Random Forest is used to diagnosis of Crohn’s disease and celiac disease using
nuclear magnetic resonance spectroscopy [35]. This technique is used in many
way for medical image classification which Random forests or random decision
forests technique is an ensemble learning method for medical image
classification [70]. This method, which used $t$ tree as parallel, was
introduced by T. Kam Ho [71] in 1995. As shown in Figure 2.5, main idea of RF
is generating random decision trees. In Chapter 4, we use very similar
technique but instead of trees we use Random Deep Learning in our forest. This
technique was further developed in 1999 by L. Breiman [72], who found
convergence for RF as margin measures ($mg(X,Y)$).

Figure 2.5: Random Forest

###### Voting

After training all trees as forest, predictions are assigned based on voting
[73] as follows:

$\delta_{V}=arg\max_{i}\sum_{j:j\neq j}I_{\\{r_{ij}>r_{ji}\\}}$ (2.2)

such that

$r_{ij}+r_{ji}=1$ (2.3)

Algorithm 1 A1

input :

training set S of size m, inducer $\tau$, integer $N$

for  _ $i=1$ to $N$_ do

$C_{i}=\tau(S^{\prime})$ $\displaystyle\epsilon_{i}=\frac{1}{m}\sum_{x_{j}\in
S^{\prime};C_{i}(x_{j})\notin y_{i}}weight(x)$ if  _$\epsilon_{i}
>\frac{1}{2}$_ then

set S’ to a bootstrap sample from S with weight 1 for all instance and go top

endif

$\beta_{i}=\frac{\epsilon_{i}}{1-\epsilon_{i}}$ for  _ $x_{i}\in S^{\prime}$_
do

if  _$C_{i}(x_{j})=y_{i}$_ then

$weight(x_{j})=weight(x_{j}).\beta_{i}$

endif

endfor

Normalize weights of instances

endfor

$\displaystyle C^{*}(x)=arg\max_{y\in
Y}\sum_{i,C_{i}(x)=y}\log\frac{1}{\beta_{i}}$ output : Classifier $C^{*}$

Algorithm 2 The AdaBoost method Figure 2.6: This figure is the Boosting
technique architecture

##### 2.2.5.2 Boosting

The Boosting algorithm was first introduced by RE. Schapire [74] in 1990 as a
technique for boosting the performance of a weak learning algorithm. This
technique was further developed by Freund [75, 76].

The Figure 2.6 shows how boosting algorithm works for 2D datasets, as it shown
we have labeled data then we trained by multi-model architectures (ensemble
learning). These developments resulted in the AdaBoost (Adaptive Boosting)
[77].

input :

training set S, Inducer $\tau$, integer $N$

for  _ $i=1$ to $N$_ do

$S^{\prime}$ = bootstrap sample from S $C_{i}$ = $\tau(S^{\prime})$

endfor

$\displaystyle C^{*}(x)=arg\max_{y\in Y}\sum_{i,C_{i}=y}1$ output : Classifier
$C^{*}$

Algorithm 3 Bagging

##### 2.2.5.3 Bagging

The Bagging algorithm was introduced by L. Breiman [78] in 1996 as a voting
classifier method. The algorithm is generated by different bootstrap
samples[69]. A bootstrap generates a uniform sample from the training set. If
$N$ bootstrap samples $B_{1},B_{2},\ldots,B_{N}$ have been generated, then we
have N classifiers ($C$) which $C_{i}$ is built from each bootstrap sample
$B_{i}$. Finally, our classifier $C$ contain or generated from
$C_{1},C_{2},...,C_{N}$ whose output is the class predicted most often by its
sub-classifiers, with ties broken arbitrarily[69, 78]. Figure 2.7 shows a
simple Bagging algorithm which trained $N$ models.

Figure 2.7: This figure shows a simple model of Bagging technique

#### 2.2.6 Deep Learning:

Neural networks derive their architecture as a relatively simple
representation of the neurons in the human’s brain. They are essentially
weighted combinations of inputs that pass through multiple non-linear
functions. Neural networks use an iterative learning method known as back-
propagation and an optimizer (such as stochastic gradient descent (SGD)).

Multilayer perceptron or Deep Neural Networks (DNN) are based on simple neural
networks architectures but they contain multiple hidden layers. These networks
have been widely used for classification. For example, D. CireşAn et. al. in
2012 [79] used multi-column deep neural networks for classification tasks,
where multi-column deep neural networks use Multilayer perceptron
architectures. Convolutional Neural Networks (CNN) provide a different
architectural approach to learning with neural networks. The main idea of CNN
is to use feed-forward networks with convolutional layers that include local
and global pooling layers. A. Krizhevsky in 2012 [80] used CNN, but they have
used $2D$ convolutional layers combined with the $2D$ feature space of the
image. Another example of CNN in [81] showed excellent accuracy for image
classification. The final type of deep learning architecture is Recurrent
Neural Networks (RNN) where outputs from the neurons are fed back into the
network as inputs for the next step. Some recent extensions to this
architecture uses Gated Recurrent Units (GRUs) [82] or Long Short-Term Memory
(LSTM) units [83]. These new units help control for instability problems in
the original network architecure. RNN have been successfully used for natural
language processing [84]. Recently, In [85] in 2016 developed A hierarchical
convolutional neural network for mitosis detection in phase-contrast
microscopy images. These networks have two important characteristics:
hierarchical structure and an attention mechanism.

New work has combined these three basic models of the deep learning structure
and developed a novel technique for enhancing accuracy and robustness. The
work of M. Turan et. al. in 2017 [86] and M. Liang et. al.in 2015 [87]
implemented innovative combinations of CNN and RNN called A Recurrent
Convolutional Neural Network (RCNN).

### 2.3 Autoencoder

An autoencoder is a type of neural network that is trained to attempt to copy
its input to its output [88]. In this dissertation, we use autoencoder in all
chapters to select only useful images for training and discard useless
information (see Figure 3.2). The autoencoder has achieved great success as a
dimensionality reduction, abnormally detection, etc. method via the powerful
reprehensibility of neural networks [89]. The first version of autoencoder was
introduced by D.E. Rumelhart et al. [90] in 1985. The main idea is that one
hidden layer between input and output layers has fewer units [91] and could
thus be used to reduce the dimensions of a feature space. Especially for
texts, documents, and sequences that contain many features, using an
autoencoder could help allow for faster, more efficient data processsing.

#### 2.3.1 General Framework

As shown in Figure 2.8, the input and output layers of an autoencoder contain
$n$ units where $x=\mathbb{R}^{n}$, and hidden layer $Z$ contains $p$ units
with respect to $p<n$ [92]. For this technique of dimensionality reduction,
the dimensions of the final feature space are reduced from $n\rightarrow p$.
The encoder representation involves a sum of the representation of all words
(for bag-of-words), reflecting the relative frequency of each word [93]:

$a(x)=c+\sum_{i=1}^{|x|}W_{.,x_{i}},\phi(x)=h(a(x))$ (2.4)

where $h(.)$ is an element-wise non-linearity such as the sigmoid (see
Equation (3.12)).

Figure 2.8: This figure shows how a simple autoencoder works. The model
depicted contains the following layers: $Z$ is code and two hidden layers are
used for encoding and two are used for decoding.

#### 2.3.2 Conventional Autoencoder Architecture

As shown in Figure 3.2, we used Conventional Autoencoder Architecture in this
study. A Convolutional Neural Network (CNN)-based autoencoder can be divided
into two main steps [94] (encoding and decoding).

$O_{m}(i,j)=a\bigg{(}\sum_{d=1}^{D}\sum_{u=-2k-1}^{2k+1}\sum_{v=-2k-1}^{2k+1}F^{(1)}_{m_{d}}(u,v)I_{d}(i-u,j-v)\bigg{)}~{}~{}\quad\forall
m=1,\cdots,n$ (2.5)

where $F\in\\{F^{(1)}_{1},F^{(1)}_{2},\ldots,F^{(1)}_{n},\\}$ which is a
convolutional filter, with convolution among an input volume defined by
$I=\left\\{I_{1},\cdots,I_{D}\right\\}$ which learns to represent input
combining non-linear functions:

$z_{m}=O_{m}=a(I*F^{(1)}_{m}+b^{(1)}_{m})\quad m=1,\cdots,m$ (2.6)

where $b^{(1)}_{m}$ is the bias, and the number of zeros we want to pad the
input with is such that: dim(I) = dim(decode(encode(I))). Finally, the
encoding convolution is equal to:

$\begin{split}O_{w}=O_{h}&=(I_{w}+2(2k+1)-2)-(2k+1)+1\\\
&=I_{w}+(2k+1)-1\end{split}$ (2.7)

The decoding convolution step produces $n$ feature maps $z_{m=1,\ldots,n}$.
The reconstructed results $\hat{I}$ is the result of the convolution between
the volume of feature maps $Z=\\{z_{i=1}\\}^{n}$ and this convolutional
filters volume $F^{(2)}$ [95, 96, 94].

$\tilde{I}=a(Z*F^{(2)}_{m}+b^{(2)})$ (2.8)
$\begin{split}O_{w}=O_{h}=&(I_{w}+(2k+1)-1)-\\\
&(2k+1)+1=I_{w}=I_{h}\end{split}$ (2.9)

where Equation (2.9) shows the decoding convolution with $I$ dimensions.
Input’s dimensions are equal to the output’s dimensions.

#### 2.3.3 Recurrent Autoencoder Architecture

A recurrent neural network (RNN) is a natural generalization of feedforward
neural networks to feature sequences [97]. A standard RNN compute the encoding
as a feature sequences of output by iteration:

$\displaystyle h_{t}$ $\displaystyle=sigm(W^{hx}x_{t}+W^{hh}h_{t-1})$ (2.10)
$\displaystyle y_{t}$ $\displaystyle=W^{y^{h}}h_{t}$ (2.11)

where x is inputs $(x_{1},...,x_{T})$ and $y$ refers to output
($y_{1},...,y_{T}$). A multinomial distribution (1-of-K coding) can be output
using a softmax activation function [98]:

$p(x_{t,j}=1\>|\>x_{t-1,...,x_{1}})=\frac{\exp(w_{j}h_{t})}{\sum_{j^{\prime}=1}^{K}\exp(w_{j^{\prime}}h_{t})}$
(2.12)

By combining these probabilities, we can compute the probability of the
sequence $x$ as:

$p(x)=\prod_{t=1}^{T}p(x_{t}\>|\>x_{t-1},...,x_{1})$ (2.13)

### 2.4 Medical Image Staining Problem for Machine Learning Algorithms

In this section, we explain the medical image staining problem for machine
learning algorithms which include medical image staining; then, we discuss
staining problems for machine learning, and finally, we explain possible
solutions including color normalization and color balancing.

#### 2.4.1 Medical Image Staining

Hematoxylin and eosin (H&E) stains have been used for at least a century and
are still essential for recognizing various tissue types and the morphologic
changes that form the basis of contemporary CD, EE, and cancer diagnosis [99].
H&E is used routinely in histopathology laboratories as it provides the
pathologist/researcher a very detailed view of the tissue[100].

Figure 2.9: Hematoxylin and eosin (H&E) staining ith 0 (control, a and b), 50
mg/L (b and e) and 200 mg/L HgCl2 (c and f). Panels (a-c) are shown in low
magnification (400×) while (d-f) in high magnification (1000×). [3].

#### 2.4.2 Staining Problem for Machine Learning Algorithm

Color variation has been a very important problem in histopathology based on
light microscopy. A range of factors makes this problem even more complex such
as the use of different scanners, variable chemical coloring/reactivity from
different manufacturers/batches of stains, coloring being dependent on
staining procedure (timing, concentrations, etc.), and light transmission
being a function of section thickness [101]. This different H&E staining
standard (as shown in Figure 2.10) can mislead the machine learning algorithms
by color; for example, if we have all $"A"$ images from center $One$ and all
labeled $"B"$ images from center $"Two"$ and they used different H&E standard
which makes these based color slightly different with each other; thus, the
training machine learning model based on the original images (without any pre-
processing) misled by the color of these images instead of characteristic of
images such as Crypt Epithelium, Goblet cells, Inflammatory cells, etc.

Figure 2.10: Different H&E Color staining

#### 2.4.3 Color Normalization and Color Balancing

In color normalization, Images are first deconvolved into their principal
staining constituents then normalization is performed on each staining channel
separately, which involves a mapping between reference and source image [102].
However, this technique is limited to one standard color staining from one
center which means non-linear machine learning algorithms such as deep
learning could detect these color staining normalization. Many researchers try
to solve this problem by modifying these algorithm [103, 104, 105, 106, 107,
108].

The another problem in this part is the computational complexity of these
models which the original color normalization could not remove all intensity
values from the image while preserving color values, so it needs to be a more
complex model which increase computation time in these algorithms. Although
many researchers proposed different color normalization techniques, but still
these algorithms cannot remove all intensity values from the images.

The other way to solve this problem is the charging classification model
instead of pre-processing. Many researchers using color balancing techniques
to remove the effectiveness of color in the machine learning model [109, 6] In
this approach, the model is trained with different range of color balanced
images.

### 2.5 Evaluation Techniques

In the research community, having shared and comparable performance measures
to evaluate algorithms is preferable. However, in reality such measures may
only exist for a handful of methods. The major problem when evaluating image
classification methods is the absence of standard data collection protocols.
Even if a common collection method existed (e.g. Reuters news corpus), simply
choosing different training and test sets can introduce inconsistencies in
model performance [110]. Another challenge with respect to method evaluation
is being able to compare different performance measures used in separate
experiments. Performance measures generally evaluate specific aspects of
classification task performance, and thus do not always present identical
information. In this section, we discuss evaluation metrics and performance
measures and highlight ways in which the performance of classifiers can be
compared.

Figure 2.11: Confusion matrix

Since the underlying mechanics of different evaluation metrics may vary,
understanding what exactly each of these metrics represents and what kind of
information they are trying to convey is crucial for comparability. Some
examples of these metrics include recall, precision, accuracy, F-measure,
micro-average, and macro average. These metrics are based on a “confusion
matrix” (shown in Figure 2.11) that comprises true positives (TP), false
positives (FP), false negatives (FN) and true negatives (TN) [111]. The
significance of these four elements may vary based on the classification
application. The fraction of correct predictions over all predictions is
called accuracy (Eq. 2.14). The fraction of known positives that are correctly
predicted is called sensitivity i.e. true positive rate or recall (Eq. 2.15).
The ratio of correctly predicted negatives is called specificity (Eq. 2.16).
The proportion of correctly predicted positives to all positives is called
precision, i.e. positive predictive value (Eq. 2.17).

$\displaystyle accuracy$ $\displaystyle=\frac{(TP+TN)}{(TP+FP+FN+TN)}$ (2.14)
$\displaystyle sensitivity$ $\displaystyle=\frac{TP}{(TP+FN)}$ (2.15)
$\displaystyle specificity$ $\displaystyle=\frac{TN}{(TN+FP)}$ (2.16)
$\displaystyle precision$
$\displaystyle=\frac{\sum_{l=1}^{L}TP_{l}}{\sum_{l=1}^{L}TP_{l}+FP_{l}}$
(2.17) $\displaystyle recall$
$\displaystyle=\frac{\sum_{l=1}^{L}TP_{l}}{\sum_{l=1}^{L}TP_{l}+FN_{l}}$
(2.18) $\displaystyle F1-Score$
$\displaystyle=\frac{\sum_{l=1}^{L}2TP_{l}}{\sum_{l=1}^{L}2TP_{l}+FP_{l}+FN_{l}}$
(2.19)

The pitfalls of each of the above-mentioned metrics are listed in Table 2.1.

Table 2.1: Metrics Pitfalls | Limitation
---|---
Accuracy |  gives us no information on FN and FP
Sensitivity |  does not evaluate TN and FP and any classifier that predicts data points as positives considered to have high sensitivity
Specificity |  similar to sensitivity and does not account for FN and TP
Precision |  does not evaluate TN and FN and considered to be very conservative and goes for a case which is most certain to be positive

#### 2.5.1 Macro-Averaging and Micro-Averaging

A single aggregate measure is required when several two-class classifiers are
being used to process a collection. Macro-averaging gives a simple average
over classes while micro-averaging combines per-datapoint decisions across
classes and then outputs an effective measure on the pooled contingency table
[112]. Macro-averaged results can be computed as follows:

$B_{macro}=\frac{1}{q}\sum_{\lambda=1}^{q}B(TP_{\lambda}+FP_{\lambda}+TN_{\lambda}+FN_{\lambda})$
(2.20)

where $B$ is a binary evaluation measure calculated based on true positives
(TP), false positives (FP), false negatives (FN) and true negatives (TN), and
$L=\\{\lambda_{j}:j=1...q\\}$ is the set of all labels.

Micro-averaged results [113, 114] can be computed as follows:

$\begin{split}B_{macro}=B\bigg{(}&\sum_{\lambda=1}^{q}TP_{\lambda},\sum_{\lambda=1}^{q}FP_{\lambda},\sum_{\lambda=1}^{q}TN_{\lambda},\sum_{\lambda=1}^{q}FN_{\lambda}\bigg{)}\end{split}$
(2.21)

Micro-average score assigns equal weights to every images and as a
consequence, and it is considered to be a per-image average. On the other
hand, macro-average score assigns equal weights to each category without
accounting for frequency and therefore it is a per-category average.

#### 2.5.2 Fβ Score

$F_{\beta}$ is one of the most popular aggregated evaluation metrics for
classifier evaluation [111]. The parameter $\beta$ is used to balance recall
and precision and defined as follows:

${}F_{\beta}=\frac{(1+\beta^{2})(precision\times recall)}{\beta^{2}\times
precision+recall}$ (2.22)

For commonly used $\beta=1$ i.e. $F_{1}$, recall and precision are given equal
weights and Eq. 2.22 can be simplified to:

${}F_{1}=\frac{2TP}{2TP+FP+FN}$ (2.23)

Since $F_{\beta}$ is based on recall and precision, it does not represent the
confusion matrix fully.

#### 2.5.3 Matthews Correlation Coefficient (MCC)

The Matthews correlation coefficient (MCC) [115] captures all the data in a
confusion matrix and measures the quality of binary classification methods.
MCC can be used for problems with uneven class sizes and is still considered a
balanced measure. MCC ranges from $-1$ to $0$ (i.e. the classification is
always wrong and always true respectively). MCC can be calculated as follows:

${}MCC=\frac{TP\times TN-FP\times
FN}{\sqrt{\displaystyle\begin{multlined}(TP+FP)\times(TP+FN)\times\\\
(TN+FP)\times(TN+FN)\end{multlined}(TP+FP)\times(TP+FN)\times\\\
(TN+FP)\times(TN+FN)}}$ (2.24)

While comparing two classifiers, one may have a higher score using MCC and the
other one has a higher score using $F_{1}$ and as a result one specific metric
cannot captures all the strengths and weaknesses of a classifier [111].

#### 2.5.4 Receiver Operating Characteristics (ROC)

Receiver operating characteristics (ROC) [116] curves are valuable graphical
tools for evaluating classifiers. However, class imbalances (i.e. differences
in prior class probabilities [117]) can cause ROC curves to poorly represent
the classifier performance. ROC curve plots true positive rate (TPR) and false
positive rate (FPR):

$TPR=\frac{TP}{TP+FN}$ (2.25) $FPR=\frac{FP}{FP+TN}$ (2.26)

#### 2.5.5 Area Under ROC Curve (AUC)

The area under ROC curve (AUC) [118, 119] measures the entire area underneath
the ROC curve. AUC leverages helpful properties such as increased sensitivity
in the analysis of variance (ANOVA) tests, independence from decision
threshold, invariance to a priori class probabilities, and indication of how
well negative and positive classes are regarding decision index [120].

For binary classification tasks, AUC can be formulated as:

$\begin{split}AUC&=\int_{-\infty}^{\infty}TPR(T)FPR^{\prime}(T)dT\\\
&=\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}I(T^{\prime}>T)f_{1}(T^{\prime})f_{0}(T)dTdT^{\prime}\\\
&=P(X_{1}>X_{0})\end{split}$ (2.27)

For multi-class AUC, an average AUC can be defined [121] as follows:

$AUC=\frac{2}{|C|(|C|-1)}\sum_{i=1}^{|C|}AUC_{i}$ (2.28)

where C is the number of the classes.
Yang [110] evaluated statistical approaches for classification which following
important factors that should be considered when comparing classifier
algorithms:

  * •

Comparative evaluation across methods and experiments which gives insight
about factors underlying performance variations and will lead to better
evaluation methodology in the future

  * •

Impact of collection variability such as including unlabelled images in
training or test set and treat them as negative instances can be a serious
problem.

  * •

Category ranking evaluation and binary classification evaluation show the
usefulness of classifier in interactive applications and emphasize their use
in a batch mode respectively. Having both types of performance measurements to
rank classifiers is helpful in detecting the effects of thresholding
strategies.

  * •

Evaluation of the scalability of classifiers in large category spaces is
rarely investigated area.

### 2.6 Summary

The classification task is one of the most indispensable problems in machine
learning. As Medical image and image data sets proliferate, the development
and documentation of supervised machine learning algorithms becomes an
imperative issue, especially for medical image classification. Having a better
categorization system for these images require discerning these algorithms.
However, the existing image classification algorithms work more efficiently if
we have a better understanding of pre-processing methods and how to evaluate
them correctly. Existing classification algorithms, such as the Rocchio
algorithm, bagging and boosting, logistic regression (LR), k-nearest Neighbor
(KNN), Support Vector Machine (SVM), random forest, and deep learning, are the
primary focus of this part. Evaluation methods, such as accuracy, $F_{\beta}$,
Matthew correlation coefficient (MCC), receiver operating characteristics
(ROC), and area under curve (AUC), are explained. With these metrics, the
image classification algorithm can be evaluated.

##  Chapter 3 Diagnosis of Celiac Disease and Environmental Enteropathy on
Biopsy Images Using Color Balancing on Convolutional Neural Networks

Celiac Disease (CD) and Environmental Enteropathy (EE) are common causes of
malnutrition and adversely impact normal childhood development. CD is an
autoimmune disorder that is prevalent worldwide and is caused by an increased
sensitivity to gluten. Gluten exposure destructs the small intestinal
epithelial barrier, resulting in nutrient mal-absorption and childhood under-
nutrition. EE also results in barrier dysfunction but is thought to be caused
by an increased vulnerability to infections. EE has been implicated as the
predominant cause of under-nutrition, oral vaccine failure, and impaired
cognitive development in low-and-middle-income countries. Both conditions
require a tissue biopsy for diagnosis, and a major challenge of interpreting
clinical biopsy images to differentiate between these gastrointestinal
diseases is striking histopathologic overlap between them. In the current
study, we propose a convolutional neural network (CNN) to classify duodenal
biopsy images from subjects with CD, EE, and healthy controls. We evaluated
the performance of our proposed model using a large cohort containing 1000
biopsy images. Our evaluations show that the proposed model achieves an area
under ROC of 0.99, 1.00, and 0.97 for CD, EE, and healthy controls,
respectively. These results demonstrate the discriminative power of the
proposed model in duodenal biopsies classification.

### 3.1 Introduction and Related Works

In this section, we propose a CNN-based model for the classification of biopsy
images. In recent years, Deep Learning architectures have received great
attention after achieving state-of-the-art results in a wide variety of
fundamental tasks such classification [17, 18, 19, 21, 10, 22, 23] or other
medical domains [25, 26]. CNNs, in particular, have proven to be very
effective in medical image processing. CNNs preserve local image relations
while reducing dimensionality and for this reason are the most popular machine
learning algorithm in image recognition and visual learning tasks [27]. CNNs
have been widely used for classification and segmentation in various types of
medical applications such as histopathological images of breast tissues, lung
images, MRI images, medical X-Ray images, etc. [9, 10]. Researchers produced
advanced results on duodenal biopsies classification using CNNs [11], but
those models are only robust to a single type of image stain or color
distribution. Many researchers apply a stain normalization technique as part
of the image pre-processing stage to both the training and validation datasets
[28, 122, 123]. In this dissertation section, varying levels of color
balancing were applied during image pre-processing in order to account for
multiple stain variations.

Figure 3.1: Overview of methodology

The rest of this Section is organized as follows: In Section 3.2, we describe
the different data sets used in this work, as well as, the required pre-
processing steps. The architecture of the model is explained in Section 3.4.
Empirical results are elaborated in Section 3.5. Finally, Section 3.6
concludes the dissertation section along with outlining future directions.

### 3.2 Data Source

For this project, $121$ Hematoxylin and Eosin (H&E) stained duodenal biopsy
glass slides were retrieved from $102$ patients. The slides were converted
into $3118$ whole slide images and labeled as either EE, CD, or normal. The
biopsy slides for EE patients were from the Aga Khan University Hospital
(AKUH) in Karachi, Pakistan ($n=29$ slides from $10$ patients) and the
University of Zambia Medical Center in Lusaka, Zambia ($n=16$). The slides for
CD patients ($n=34$) and normal ($n=42$) were retrieved from archives at the
University of Virginia (UVa). The CD and normal slides were converted into
whole slide images at $40$x magnification using the Leica SCN $400$ slide
scanner (Meyer Instruments, Houston, TX) at UVa, and the digitized EE slides
were of 20x magnification and shared via the Environmental Enteric Dysfunction
Biopsy Investigators (EEDBI) Consortium shared WUPAX server. Characteristics
of our patient population are as follows: the median ($Q1$, $Q3$) age of our
entire study population was $31$ ($20.25$, $75.5$) months, and we had a
roughly equal distribution of males ($52$%, $n=53$) and females ($48$%,
$n=49$). The majority of our study population were histologically normal
controls $(41.2\%)$, followed by CD patients $(33.3\%)$, and EE patients
$(25.5\%)$.

### 3.3 Pre-Processing

In this section, we cover all of the pre-processing steps which include image
patching, image clustering, and color balancing. The biopsy images are
unstructured (varying image sizes) and too large to process with deep neural
networks; thus, requiring that images are split into multiple smaller images.
After executing the split, some of the images do not contain much useful
information. For instance, some only contain the mostly blank border region of
the original image. In the image clustering section, the process to select
useful images is described. Finally, color balancing is used to correct for
varying color stains which is a common issue in histological image processing.

#### 3.3.1 Image Patching

Although the effectiveness of CNNs in image classification has been shown in
various studies in different domains, training on high-resolution Whole Slide
Tissue Images (WSI) is not commonly preferred due to a high computational
cost. However, applying CNNs on WSI enables losing a large amount of
discriminative information due to extensive downsampling [124]. Due to a
cellular level difference between Celiac, Environmental Enteropathy and normal
cases, a trained classifier on image patches are likely to perform as well as
or even better than a trained WSI-level classifier. Many researchers in
pathology image analysis have considered classification or feature extraction
on image patches [124]. In this project, after generating patches from each
image, labels were applied to each patch according to its associated original
image. CNN was trained to generate predictions on each individual patch.

Figure 3.2: Structure of clustering model with autoencoder and K-means
combination

#### 3.3.2 Clustering

##### 3.3.2.1 Autoencoder

In this study, after image patching, some of the created patches do not
contain any useful information regarding biopsies and should be removed from
the data. These patches have been created from mostly background parts of
WSIs. A two-step clustering process was applied to identify the unimportant
patches. For the first step, a convolutional autoencoder was used to learn
embedded features of each patch and in the second step, we used k-means to
cluster embedded features into two clusters: useful and not useful. In Figure
3.2, the pipeline of our clustering technique is shown which contains both the
autoencoder and k-mean clustering.

An autoencoder is a type of neural network that is designed to match the
model’s inputs to the outputs [88]. The autoencoder has achieved great success
as a dimensionality reduction method via the powerful reprehensibility of
neural networks [89]. The first version of the autoencoder was introduced by
DE. Rumelhart el at. [90] in 1985. The main idea is that one hidden layer
between input and output layers has much fewer units [91] and can be used to
reduce the dimensions of feature space. For medical images which typically
contain many features, using an autoencoder can help allow for faster, more
efficient data processing.

A CNN-based autoencoder can be divided into two main steps [94] : encoding and
decoding.

$\begin{split}O_{m}(i,j)=a\bigg{(}&\sum_{d=1}^{D}\sum_{u=-2k-1}^{2k+1}\sum_{v=-2k-1}^{2k+1}F^{(1)}_{m_{d}}(u,v)I_{d}(i-u,j-v)\bigg{)}\\\
&\quad m=1,\cdots,n\end{split}$ (3.1)

Where $F\in\\{F^{(1)}_{1},F^{(1)}_{2},\ldots,F^{(1)}_{n},\\}$ is a
convolutional filter, with convolution among an input volume defined by
$I=\left\\{I_{1},\cdots,I_{D}\right\\}$ which it learns to represent the input
by combining non-linear functions:

$z_{m}=O_{m}=a(I*F^{(1)}_{m}+b^{(1)}_{m})\quad m=1,\cdots,m$ (3.2)

where $b^{(1)}_{m}$ is the bias, and the number of zeros we want to pad the
input with is such that: dim(I) = dim(decode(encode(I))) Finally, the encoding
convolution is equal to:

$\begin{split}O_{w}=O_{h}&=(I_{w}+2(2k+1)-2)-(2k+1)+1\\\
&=I_{w}+(2k+1)-1\end{split}$ (3.3)

The decoding convolution step produces $n$ feature maps $z_{m=1,\ldots,n}$.
The reconstructed results $\hat{I}$ is the result of the convolution between
the volume of feature maps $Z=\\{z_{i=1}\\}^{n}$ and this convolutional
filters volume $F^{(2)}$ [95, 96].

$\tilde{I}=a(Z*F^{(2)}_{m}+b^{(2)})$ (3.4)
$\begin{split}O_{w}=O_{h}=(I_{w}+(2k+1)-1)-(2k+1)+1=I_{w}=I_{h}\end{split}$
(3.5)

Where Equation 3.5 shows the decoding convolution with $I$ dimensions. The
input’s dimensions are equal to the output’s dimensions.

##### 3.3.2.2 K-Means

K-means clustering is one of the easiest methods of clustering algorithm [125,
126] which data is given $D\in\\{x_{1},x_{2},...,x_{n}\\}$ in $d$ dimensional
vectors which is $x\in f^{d}$. The aim is to identify groups of data points
and assign each point to one of the groups. What is good clustering? There is
no universal approach, but we will try the K-means approach, but it has a lot
of limitations that we could not calculate the error rate and error percentage
although the loss function is available [127].

If we want to find the error rate the time complexity is equal to exponential.
One measure of how good clustering is would be the sum of distances to the
center. Therefore, K-means wants to minimize this $\xi$ (quantity), choosing
$\mu$ and a’s to minimize but it is difficult to do analytically because a’s
are binary assignments. So the K-means algorithm tries to iteratively solve
the minimization (sort of greedy algorithm) [128, 129].

Minimize $\xi$ with respect to $a$ and $\mu$ by:

$\displaystyle\xi=\sum_{j=1}^{k}\sum_{x_{i}}||x_{i}-\mu_{j}||^{2}=\sum_{j=1}^{k}\sum_{i=1}^{n}A_{ij}||x_{i}-\mu_{j}||$
(3.6)

where $\xi$ stand for quantity of dataset, and $\mu$ is centroid of each
cluster

  1. 1.

Initialize $\mu_{1}$ to $\mu_{k}$ arbitrarily.

  2. 2.

Choose the optimal assignment “a” for given centers $\mu$. [Fix µ optimize a].

  3. 3.

Choose optimal µ for fixed “a” [Fix a optimize $\mu$].

  4. 4.

Repeat (iteratively) 2 and 3 until convergence.

Algorithm 4 K-means $n$ images for $K$ Clusters (in our expriment k=2)

Input:
$D=\\{\overrightarrow{x_{1}},\overrightarrow{x_{2}},...,\overrightarrow{x_{n}}\\}$

Output:
$\mu=\overrightarrow{\mu_{1}},\overrightarrow{\mu_{2}},...,\overrightarrow{\mu_{k}}\\}$

$\\{\overrightarrow{s_{1}},\overrightarrow{s_{2}},...,\overrightarrow{s_{k}}\\}$
set random seeds
$(\\{\overrightarrow{x_{1}},\overrightarrow{x_{2}},...,\overrightarrow{x_{n}}\\},K)$

for  _$k\leftarrow 1$ to K_ do

$\overrightarrow{\mu_{k}}\leftarrow\overrightarrow{s_{k}}$

endfor

while  _Criterion has not been met_ do

for  _$k\leftarrow 1$ to K_ do

$w_{k}\leftarrow\\{\\}$

endfor

for  _$n\leftarrow 1$ to $N$_ do

$j\leftarrow
arg\min_{j^{\prime}}|\overrightarrow{\mu_{j^{\prime}}}-\overrightarrow{\mu_{x_{n}}}|$
$w_{j}\leftarrow w_{j}\bigcup~{}\\{\overrightarrow{x_{n}}\\}$

endfor

for  _$k\leftarrow 1$ to K_ do

$\mu_{k}\leftarrow\frac{1}{|w_{k}|}\sum_{\overrightarrow{x}\in
w_{k}}\overrightarrow{x}$

endfor

endwhile

K-Means is not guaranteed to converge to the global minimum of this function.
In fact, finding a global minimum is a NP-hard problem, but it is reasonable
to some extent.

  1. 1.

Length of vectors is M (point and a centroid) so computing the distance
between two points is O (M).

  2. 2.

Reassign clusters: k = number of clusters, and N is number of points =
$O(KN)$. So computing distance = $O(MKN)$

  3. 3.

Computing centroids: Where each point assigned once to a certain centroid =
$O(NM)$ in that cluster k.

  4. 4.

Assuming reassigning of clusters and computing centroids each done at least
once for I iterations = $O(IKNM)$.

So the time complexity of K-means is equal to if K value which the number of
clusters and I value which the number of the iteration will be available;
although, if the number of the iteration value is not available, time
complexity can be exponential time. K-means also is used for image and data
clustering especially broadly used in information retrieval [125, 130]. data-
points (image representation) have the same cluster behave similarly with
respect to relevance to datapoint’s information which is extracted as feature
space. The centroid $\mu$ of a datapoint is calculated as follows:

$\mu(w)=\frac{1}{|w|}\sum_{\bar{x}\in w}\bar{x}$ (3.7)

Results of patch clustering have been summarized in Table 3.1. Obviously,
patches in cluster $1$, which were deemed useful, are used for the analysis in
this section.

Figure 3.3: Some samples of clustering results - cluster 1 includes patches with useful information and cluster 2 includes patches without useful information (mostly created from background parts of WSIs) Table 3.1: The clustering results for all patches into two clusters | Total | Cluster 1 | Cluster 2
---|---|---|---
Celiac Disease (CD) | $16,832$ | $7,742~{}(46\%)$ | $9,090~{}(54\%)$
Normal | $15,983$ | $8,953~{}(56\%)$ | $7,030~{}(44\%)$
Environmental Enteropathy (EE) | $22,625$ | $2,034~{}(9\%)$ | $20,591~{}(91\%)$
Total | $55,440$ | $18,729~{}(34\%)$ | $36,711~{}(66\%)$
Figure 3.4: Color Balancing samples for the three classes

#### 3.3.3 Color Balancing

The concept of color balancing for this dissertation section is to convert all
images to the same color space to account for variations in H&E staining. The
images can be represented with the illuminant spectral power distribution
$I(\lambda)$, the surface spectral reflectance $S(\lambda)$, and the sensor
spectral sensitivities $C(\lambda)$ [109, 131]. Using this notation [131], the
sensor responses at the pixel with coordinates $(x,y)$ can be thus described
as:

$p(x,y)=\int_{w}I(x,y,\lambda)S(x,y,\lambda)C(\lambda)d\lambda$ (3.8)

where $w$ is the wavelength range of the visible light spectrum, p and
$C(\lambda)$ are three-component vectors.

$\displaystyle\left[\begin{array}[]{c}R\\\ G\\\ B\\\
\end{array}\right]_{out}=$
$\displaystyle\left(\alpha\left[\begin{array}[]{c@{\quad}c@{\quad}c}a_{11}&a_{12}&a_{13}\\\
a_{21}&a_{22}&a_{23}\\\ a_{31}&a_{32}&a_{33}\\\
\end{array}\right]\right.{}\times\left.\left[\begin{array}[]{c@{\quad}c@{\quad}c}r_{awb}&0&0\\\
0&g_{awb}&0\\\ 0&0&b_{awb}\\\ \end{array}\right]\left[\begin{array}[]{c}R\\\
G\\\ B\\\ \end{array}\right]_{in}\right)^{\gamma}$ (3.9)

where $RGB_{in}$ is raw images from biopsy and  $RGB_{out}$ is results for CNN
input. In the following, a more compact version of Equation 3.9 is used:

$RGB_{out}=(\alpha AI_{w}.RGB_{in})^{\gamma}$ (3.10)

where $\alpha$ is exposure compensation gain, $I_{w}$ refers the diagonal
matrix for the illuminant compensation and $A$ indicates the color matrix
transformation.

Figure 3.4 shows the results of color balancing for three classes (Celiac
Disease (CD), Normal and Environmental Enteropathy (EE)) with different color
balancing percentages between $0.01$ and $50$.

### 3.4 Method

In this section, we describe Convolutional Neural Networks (CNN) including the
convolutional layers, pooling layers, activation functions, and optimizer.
Then, we discuss our network architecture for the diagnosis of Celiac Disease
and Environmental Enteropathy. As shown in figure 3.5, the input layers starts
with image patches ($1000\times 1000$) and is connected to the convolutional
layer (Conv $1$). Conv $1$ is connected to the pooling layer (MaxPooling), and
then connected to Conv $2$. Finally, the last convolutional layer (Conv $3$)
is flattened and connected to a fully connected perception layer. The output
layer contains three nodes in which each node represents one class.

#### 3.4.1 Convolutional Layer

CNN is a deep learning architecture that can be employed for hierarchical
image classification. Originally, CNNs were built for image processing with an
architecture similar to the visual cortex. CNNs have been used effectively for
medical image processing. In a basic CNN used for image processing, an image
tensor is convolved with a set of kernels of size $d\times d$. These
convolution layers are called feature maps and can be stacked to provide
multiple filters on the input. The element (feature) of input and output
matrices can be different [132]. The process to compute a single output matrix
is defined as follows:

$A_{j}=f\left(\sum_{i=1}^{N}I_{i}\ast K_{i,j}+B_{j}\right)$ (3.11)

Each input matrix $I-i$ is convolved with a corresponding kernel matrix
$K_{i,j}$, and summed with a bias value $B_{j}$ at each element. Finally, a
non-linear activation function (See Section 3.4.3) is applied to each element
[132].

In general, during the back propagation step of a CNN, the weights and biases
are adjusted to create effective feature detection filters . The filters in
the convolution layer are applied across all three ’channels’ or $\Sigma$
(size of the color space) [17].

Figure 3.5: Structure of Convolutional Neural Net using multiple 2D feature
detectors and 2D max-pooling

#### 3.4.2 Pooling Layer

To reduce the computational complexity, CNNs utilize the concept of pooling to
reduce the size of the output from one layer to the next in the network.
Different pooling techniques are used to reduce outputs while preserving
important features [133]. The most common pooling method is max pooling where
the maximum element is selected in the pooling window.

In order to feed the pooled output from stacked featured maps to the next
layer, the maps are flattened into one column. The final layers in a CNN are
typically fully connected [19].

#### 3.4.3 Neuron Activation

The implementation of CNN is a discriminative trained model that uses standard
back-propagation algorithm using a sigmoid (Equation 3.12), (Rectified Linear
Units (ReLU) [134] (Equation 3.13) as activation function. The output layer
for multi-class classification includes a $Softmax$ function (as shown in
Equation 3.14).

$\displaystyle f(x)$ $\displaystyle=\frac{1}{1+e^{-x}}\in(0,1)$ (3.12)
$\displaystyle f(x)$ $\displaystyle=\max(0,x)$ (3.13)
$\displaystyle\sigma(z)_{j}$
$\displaystyle=\frac{e^{z_{j}}}{\sum_{k=1}^{K}e^{z_{k}}}$ (3.14)
$\displaystyle\forall~{}j\in\\{1,\ldots,K\\}$

#### 3.4.4 Optimizer

For this CNN architecture, the $Adam$ Optimizer [135] which is a stochastic
gradient optimizer that uses only the average of the first two moments of
gradient ($v$ and $m$, shown in Equation 3.15, 3.16, 3.17, and 3.18). It can
handle non-stationary of the objective function as in RMSProp, while
overcoming the sparse gradient issue limitation of RMSProp [135].

$\theta\leftarrow\theta-\frac{\alpha}{\sqrt{\hat{v}}+\epsilon}\hat{m}$ (3.15)
$g_{i,t}=\nabla_{\theta}J(\theta_{i},x_{i},y_{i})$ (3.16)
$m_{t}=\beta_{1}m_{t-1}+(1-\beta_{1})g_{i,t}$ (3.17)
$m_{t}=\beta_{2}v_{t-1}+(1-\beta_{2})g_{i,t}^{2}$ (3.18)

where $m_{t}$ is the first moment and $v_{t}$ indicates second moment that
both are estimated. $\hat{m_{t}}=\frac{m_{t}}{1-\beta_{1}^{t}}$ and
$\hat{v_{t}}=\frac{v_{t}}{1-\beta_{2}^{t}}$.

#### 3.4.5 Network Architecture

As shown in Table 3.2 and Figure 3.6, our CNN architecture consists of three
convolution layer each followed by a pooling layer. This model receives RGB
image patches with dimensions of $(1000\times 1000)$ as input.

The first convolutional layer has $32$ filters with kernel size of $(3,3)$.
Then we have Pooling layer with size of $(5,5)$ which reduce the feature maps
from $(1000\times 1000)$ to $(200\times 200)$.

The second convolutional layers with $32$ filters with kernel size of $(3,3)$.
Then Pooling layer (MaxPooling $2D$) with size of $(5,5)$ reduces the feature
maps from $(200\times 200)$ to $(40\times 40)$. The third convolutional layer
has $64$ filters with kernel size of $(3,3)$, and final pooling layer
(MaxPooling $2D$) is scaled down to $(8\times 8)$. The feature maps as shown
in Table 3.2 is flatten and connected to fully connected layer with $128$
nodes. The output layer with three nodes to represent the three classes:
(Environmental Enteropathy, Celiac Disease, and Normal).

The optimizer used is Adam (See Section 3.4.4) with a learning rate of
$0.001$, $\beta_{1}=0.9$, $\beta_{2}=0.999$ and the loss considered is sparse
categorical crossentropy [136]. Also for all layers, we use Rectified linear
unit (ReLU) as activation function except output layer which we use $Softmax$
(See Section 3.4.3).

Figure 3.6: Our Convolutional Neural Networks’ Architecture Table 3.2: CNN Architecture for Diagnosis of Diseased Duodenal on Biopsy Images | Layer (type) | Output Shape |  | Trainable
---
Parameters
1 | Convolutional Layer | $(1000,1000,32)$ | $869$
2 | Max Pooling | $(200,200,32)$ | $0$
3 | Convolutional Layer | $(200,200,32)$ | $9,248$
4 | Max Pooling | $(40,40,32)$ | $0$
5 | Convolutional Layer | $(40,40,64)$ | $18,496$
6 | Max Pooling | $(8,8,64)$ | $0$
7 | dense | $128$ | $524,416$
8 | Output | $3$ | $387$

### 3.5 Empirical Results

#### 3.5.1 Evaluation Setup

In the research community, comparable and shareable performance measures to
evaluate algorithms are preferable. However, in reality such measures may only
exist for a handful of methods. The major problem when evaluating image
classification methods is the absence of standard data collection protocols.
Even if a common collection method existed, simply choosing different training
and test sets can introduce inconsistencies in model performance [110].
Another challenge with respect to method evaluation is being able to compare
different performance measures used in separate experiments. Performance
measures generally evaluate specific aspects of classification task
performance, and thus do not always present identical information. In this
section, we discuss evaluation metrics and performance measures and highlight
ways in which the performance of classifiers can be compared.

#### 3.5.2 Experimental Setup

The following results were obtained using a combination of central processing
units (CPUs) and graphical processing units (GPUs). The processing was done on
a $Xeon~{}E5-2640~{}(2.6GHz)$ with $32$ cores and $64GB$ memory, and the GPU
cards were two $Nvidia~{}Titan~{}Xp$ and a $Nvidia~{}Tesla~{}K20c$. We
implemented our approaches in Python using the Compute Unified Device
Architecture (CUDA), which is a parallel computing platform and Application
Programming Interface (API) model created by $Nvidia$. We also used Keras and
TensorFlow libraries for creating the neural networks [137, 136].

#### 3.5.3 Experimental Results

In this section we show that CNN with color balancing can improve the
robustness of medical image classification. The results for the model trained
on $4$ different color balancing values are shown in Table 3.3. The results
shown in Table 3.4 are also based on the trained model using the same color
balancing values. Although in Table 3.4, the test set is based on a different
set of color balancing values: $0.5,1.0,1.5$ and $2.0$. By testing on a
different set of color balancing, these results show that this technique can
solve the issue of multiple stain variations during histological image
analysis.

As shown in Table 3.3, the f1-score of three classes (Environmental
Enteropathy (EE), Celiac Disease (CD), and Normal) are $0.98$, $0.94$, and
$0.91$ respectively. In Table 3.4, the f1-score is reduced, but not by a
significant amount. The three classes (Environmental Enteropathy (EE), Celiac
Disease (CD), and Normal) f1-scores are $0.94$, $0.92$, and $0.87$
respectively. The result is very similar to MA. Boni et.al [11] which achieved
90.59% of accuracy in their mode, but without using the color balancing
technique to allow differently stained images.

Table 3.3: F1-score for train on a set with color balancing of 0.001, 0.01, 0.1, and 1.0. Then, we evaluate test set with same color balancing | precision | recall | f1-score | support
---|---|---|---|---
Celiac Disease (CD) | $0.89$ | $0.99$ | $0.94$ | $22,196$
Normal | $0.99$ | $0.83$ | $0.91$ | $22,194$
| Environmental Enteropathy
---
(EE)
$0.96$ | $1.00$ | $0.98$ | $22,198$
Table 3.4: F1-score for train with color balancing of 0.001, 0.01, 0.1, and 1.0 and test with color balancing of 0.5, 1.0, 1.5 and 2.0 | precision | recall | f1-score | support
---|---|---|---|---
Celiac Disease (CD) | $0.90$ | $0.94$ | $0.92$ | $22,196$
Normal | $0.96$ | $0.80$ | $0.87$ | $22,194$
| Environmental Enteropathy
---
(EE)
$0.89$ | $1.00$ | $0.94$ | $22,198$

In Figure 3.7, Receiver operating characteristics (ROC) curves are valuable
graphical tools for evaluating classifiers. However, class imbalances (i.e.
differences in prior class probabilities) can cause ROC curves to poorly
represent the classifier performance. ROC curve plots true positive rate (TPR)
and false positive rate (FPR). The ROC shows that AUC of Environmental
Enteropathy (EE) is $1.00$, Celiac Disease (CD) is 0.99, and Normal is 0.97.

Figure 3.7: Receiver operating characteristics (ROC) curves for three classes also the figure shows micro-average and macro-average of our classifier Table 3.5: Comparison accuracy with different baseline methods Method |  | Solve Color
---
Staining Problem
| Model
---
Architecture
Accuracy
Shifting and Reflections [11] | No | CNN | 85.13%
Gamma [11] | No | CNN | 90.59%
CLAHE [11] | No | CNN | 86.79%
Gamma-CLAHE [11] | No | CNN | 86.72%
Fine-tuned ALEXNET [28] | Yes | ALEXNET | 89.95%
Ours | Yes | CNN | 93.39%

As shown in Table 3.5, our model performs better compared to some other models
in terms of accuracy. Among the compared models, only the fine-tuned ALEXNET
[28] has considered the color staining problem. This model proposes a transfer
learning based approach for the classification of stained histology images.
They also applied stain normalization before using images for fine tuning the
model.

### 3.6 Conclusion

In this dissertation section, we proposed a data driven model for diagnosis of
diseased duodenal architecture on biopsy images using color balancing on
convolutional neural networks. Validation results show that this model is very
robust and accurate in comparing with our baselines. Furthermore, color
consistency is an issue in digital histology images and different imaging
systems reproduced the colors of a histological slide differently. Our results
demonstrate that application of the color balancing technique can attenuate
effect of this issue in image classification.

The methods described here can be improved in multiple ways. Additional
training and testing with other color balancing techniques on data sets will
continue to identify architectures that work best for these problems. Also, it
is possible to extend the model to more than four different color balance
percentages to capture more of the complexity in the medical image
classification.

##  Chapter 4 Random Multimodel Deep Learning (RMDL) for Medical Image
Classification

The continually increasing number of complex datasets each year necessitates
ever improving machine learning methods for robust and accurate categorization
of these data. This paper introduces Random Multimodel Deep Learning (RMDL): a
new ensemble, deep learning approach for classification. Deep learning models
have achieved state-of-the-art results across many domains. RMDL solves the
problem of finding the best deep learning structure and architecture while
simultaneously improving robustness and accuracy through ensembles of deep
learning architectures. RDML can accept as input a variety data to include
text, video, images, and symbolic. This chapter describes RMDL and shows test
results for medical images. For testing RMDL, we resize all images to
$100\times 100$ due to time complexity of this algorithm is very high.

### 4.1 Introduction

Categorization and classification with complex data such as medical images,
images, and video are central challenges in the data science community.
Recently, there has been an increasing body of work using deep learning
structures and architectures for such problems. However, the majority of these
deep architectures are designed for a specific type of data or domain. There
is a need to develop more general information processing methods for
classification and categorization across a broad range of data types.

While many researchers have successfully used deep learning for classification
problems (e.g., see [18, 81, 138, 82, 86, 139]), the central problem remains
as to which deep learning architecture (Multilayer perceptron, CNN, or RNN)
and structure (how many nodes (units) and hidden layers) is more efficient for
different types of data and applications. The favored approach to this problem
is trial and error for the specific application and dataset. This section
describes an approach to this challenge using ensembles of deep learning
architectures. This approach, called Random Multimodel Deep Learning (RMDL),
uses three different deep learning architectures: Deep Neural Networks (DNN or
Multilayer Perceptron), Convolutional Neural Networks (CNN), and Recurrent
Neural Networks (RNN). Test results with a variety of data types demonstrate
that this new approach is highly accurate, robust and efficient. The three
basic deep learning architectures use different feature space methods as input
layers. RDML searches across randomly generated hyperparameters for the number
of hidden layers and nodes (density) in each hidden layer in the Multilayer
Perceptron. CNN has been well designed for image classification. RMDL finds
choices for hyperparameters in CNN using random feature maps and random
numbers of hidden layers. CNN can be used for more than image data. The
structures for CNN used by RMDL contains 2D for images. RNN architectures are
mostly used primarily for text classification, but it could be used for image
classification [140]. RMDL uses two specific RNN structures: Gated Recurrent
Units (GRUs) and Long Short-Term Memory (LSTM). The number of GRU or LSTM
units and hidden layers used by the RDML are also the results of search over
randomly generated hyperparameters.

The main contributions of this work are as follows: I) Description of an
ensemble approach to deep learning which makes the final model more robust and
accurate. II) Use of different optimization techniques in training the models
to stabilize the classification task. III) Use of dropout in each individual
RDL to address over-fitting. IV) Use of majority voting among the $n$ RDL
models. This majority vote from the ensemble of RDL models improves the
accuracy and robustness of results. Specifically, if $k$ number of RDL models
produce inaccuracies or over-fitting classifications and $n>k$, the overall
system is robust and accurate V) Finally, the RMDL has ability to process a
variety of data types such as images and videos.

### 4.2 Method

The novelty of this work is in using multi random deep learning models
including Multilayer Perceptron, RNN, and CNN techniques for text and image
classification. The method section of this paper is organized as follows:
first we describe RMDL and we discuss three techniques of deep learning
architectures (Multilayer Perceptron, RNN, and CNN) which are trained in
parallel. Next, we talk about multi optimizer techniques that are used in
different random models.

#### 4.2.1 Feature Extraction and Data Pre-processing

The feature extraction is divided into two main parts for RMDL (Text and
image). Text and sequential datasets are unstructured data, while the feature
space is structured for image datasets.

##### 4.2.1.1 Image and 3D Object Feature Extraction

Image features are the followings: $h\times~{}w\times~{}c$ where $h$ denotes
the height of the image, $w$ represents the width of image, and $c$ is the
color that has 3 dimensions (RGB). For gray scale datasets such as $MNIST$
dataset, the feature space is $h\times~{}w$. A 3D object in space contains $n$
cloud points in space and each cloud point has $6$ features which are (x, y,
z, R, G, and B). The 3D object is unstructured due to number of cloud points
since one object could be different with others. However, we could use simple
instance down/up sampling to generate the structured datasets.

Figure 4.1: Overview of RDML: Random Multimodel Deep Learning for
classification that includes $n$ Random models which are $d$ random model of
the Multilayer Perceptron classifiers, $c$ models of CNN classifiers, and $r$
RNN classifiers where $r+c+d=n$.

#### 4.2.2 Random Multimodel Deep Learning

Random Multimodel Deep Learning is a novel technique that we can use in any
kind of dataset for classification. An overview of this technique is shown in
Figure 4.1 which contains multi Deep Neural Networks (DNN or Multilayer
Perceptron), Deep Convolutional Neural Networks (CNN), and Deep Recurrent
Neural Networks (RNN). The number of layers and nodes for all of these Deep
learning multi models are generated randomly (e.g. 9 Random Models in RMDL
constructed of $3$ CNNs, $3$ RNNs, and $3$ Multilayer Perceptrons, all of them
are unique due to randomly creation).

$\displaystyle M(y_{i1},y_{i2},...,y_{in})=$
$\displaystyle\bigg{\lfloor}\frac{1}{2}+\frac{(\sum_{j=1}^{n}y_{ij})-\frac{1}{2}}{n}\bigg{\rfloor}$
(4.1)

Where $n$ is the number of random models, and $y_{ij}$ is the output
prediction of model for data point $i$ in model $j$ (Equation 4.1 is used for
binary classification, $k\in\\{0~{}\text{or}~{}1\\}$). Output space uses
majority vote for final $\hat{y_{i}}$. Therefore, $\hat{y_{i}}$ is given as
follows:

$\hat{y_{i}}=\begin{bmatrix}\hat{y}_{i1}~{}\ldots~{}\hat{y}_{ij}~{}\ldots~{}\hat{y}_{in}~{}\end{bmatrix}^{T}$
(4.2)

Where $n$ is number of random model, and $\hat{y}_{ij}$ shows the prediction
of label of document or data point of $D_{i}\in\\{x_{i},y_{i}\\}$ for model
$j$ and $\hat{y}_{i,j}$ is defined as follows:

$\hat{y}_{i,j}=arg\max_{k}[softmax(y_{i,j}^{*})]$ (4.3)

After all RDL models (RMDL) are trained, the final prediction is calculated
using majority vote of these models.

#### 4.2.3 Deep Learning in RMDL

The RMDL model structure (section 4.2.2) includes three basic architectures of
deep learning in parallel. We describe each individual model separately. The
final model contains $d$ random Multilayer Perceptrons (Section 4.2.3.1), $r$
RNNs (Section 4.2.3.2), and $c$ CNNs models (Section 4.2.3.3).

Figure 4.2: Random Multimodel Deep Learning (RDML) architecture for
classification which includes $3$ Random models, a Multilayer Perceptron
classifier at left, a Deep CNN classifier at middle, and a Deep RNN classifier
at right (each unit could be LSTM or GRU).

##### 4.2.3.1 Deep Neural Networks

Deep Neural Networks’ structure is designed to learn by multi connection of
layers that each layer only receives connection from previous and provides
connections only to the next layer in hidden part. The input is a connection
of feature space with first hidden layer for all random models. The output
layer is number of classes for multi-class classification and only one output
for binary classification. But our main contribution of this paper is that we
have many training Multilayer Perceptrons for different purposes. In our
techniques, we have multi-classes Multilayer Perceptrons where each learning
models is generated randomly (number of nodes in each layer and also number of
layers are completely random assigned). Our implementation of Deep Neural
Networks (DNN) is discriminative trained model that uses standard back-
propagation algorithm using sigmoid (equation 3.12), ReLU [134] (equation
3.13) as activation function. The output layer for multi-class classification,
should use $Softmax$ equation 3.14.

##### 4.2.3.2 Recurrent Neural Networks (RNN)

Another neural network architecture that contributes in RMDL is Recurrent
Neural Networks (RNN). RNN assigns more weights to the previous data points of
sequence. Therefore, this technique is a powerful method for text, string and
sequential data classification but also could be used for image classification
as we did in this work. In RNN the neural net considers the information of
previous nodes in a very sophisticated method which allows for better semantic
analysis of structures of dataset. General formulation of this concept is
given in Equation 4.4 where $x_{t}$ is the state at time $t$ and
$\boldsymbol{u_{t}}$ refers to the input at step t.

$x_{t}=F(x_{t-1},\boldsymbol{u_{t}},\theta)$ (4.4)

More specifically, we can use weights to formulate the Equation 4.4 with
specified parameters in Equation 4.5

$x_{t}=\mathbf{W_{rec}}\sigma(x_{t-1})+\mathbf{W_{in}}\mathbf{u_{t}}+\mathbf{b}$
(4.5)

Where $\mathbf{W_{rec}}$ refers to recurrent matrix weight, $\mathbf{W_{in}}$
refers to input weights, $\mathbf{b}$ is the bias and $\sigma$ denotes an
element-wise function.

Again, we have modified the basic architecture for use RMDL. Figure 4.1 left
side shows this extended RNN architecture. Several problems arise from RNN
when the error of the gradient descent algorithm is back propagated through
the network: vanishing gradient and exploding gradient [141].

###### 4.2.3.2.1 Long Short-Term Memory (LSTM)

To deal with these problems Long Short-Term Memory (LSTM) is a special type of
RNN that preserve long term dependency in a more effective way in comparison
to the basic RNN. This is particularly useful to overcome vanishing gradient
problem [142]. Although LSTM has a chain-like structure similar to RNN, LSTM
uses multiple gates to carefully regulate the amount of information that will
be allowed into each node state. Figure 4.3 shows the basic cell of a LSTM
model. A step by step explanation of a LSTM cell is as following:

$\displaystyle i_{t}=$ $\displaystyle\sigma(W_{i}[x_{t},h_{t-1}]+b_{i}),$
(4.6) $\displaystyle\tilde{C_{t}}=$
$\displaystyle\tanh(W_{c}[x_{t},h_{t-1}]+b_{c}),$ (4.7) $\displaystyle f_{t}=$
$\displaystyle\sigma(W_{f}[x_{t},h_{t-1}]+b_{f}),$ (4.8) $\displaystyle
C_{t}=$ $\displaystyle i_{t}*\tilde{C_{t}}+f_{t}C_{t-1},$ (4.9) $\displaystyle
o_{t}=$ $\displaystyle\sigma(W_{o}[x_{t},h_{t-1}]+b_{o}),$ (4.10)
$\displaystyle h_{t}=$ $\displaystyle o_{t}\tanh(C_{t}),$ (4.11)

Where equation 4.6 is input gate, Equation 4.7 shows candid memory cell value,
Equation 4.8 is forget gate activation, Equation 4.9 is new memory cell value,
and Equation 4.10 and 4.11 show output gate value. In the above description
all $b$ represents bias vectors and all $W$ represent weight matrices and
$x_{t}$ is used as input to the memory cell at time $t$. Also, $i,c,f,o$
indices refer to input, cell memory, forget and output gates respectively.
Figure 4.3 shows the structure of these gates with a graphical representation.
An RNN can be biased when later words are more influential than the earlier
ones. To overcome this bias Convolutional Neural Network (CNN) models
(discussed in Subsection 4.2.3.3 were introduced which deploys a max-pooling
layer to determine discriminative phrases in a text [143].

###### 4.2.3.2.2 Gated Recurrent Unit (GRU):

Gated Recurrent Unit (GRU) is a gating mechanism for RNN which was introduced
by [82] and [98]. GRU is a simplified variant of the LSTM architecture, but
there are differences as follows: GRU contains two gates, a GRU does not
possess internal memory (the $C_{t-1}$ in Figure 4.3); and finally, a second
non-linearity is not applied (tanh in Figure 4.3). A step by step explanation
of a GRU cell is as following:

$z_{t}=\sigma_{g}(W_{z}x_{t}+U_{z}h_{t-1}+b_{z}),$ (4.12)

Where $z_{t}$ refers to update gate vector of $t$, $x_{t}$ stands for input
vector, $W$, $U$ and $b$ are parameter matrices and vector, $\sigma_{g}$ is
activation function that could be sigmoid or ReLU.

$\tilde{r_{t}}=\sigma_{g}(W_{r}x_{t}+U_{r}h_{t-1}+b_{r}),$ (4.13)
$h_{t}=z_{t}\circ h_{t-1}+(1-z_{t})\circ\sigma_{h}(W_{h}x_{t}+U_{h}(r_{t}\circ
h_{t-1})+b_{h})$ (4.14)

Where $h_{t}$ is output vector of $t$, $r_{t}$ stands for reset gate vector of
$t$, $z_{t}$ is update gate vector of $t$, $\sigma_{h}$ indicates the
hyperbolic tangent function.

Figure 4.3: left Figure is a cell of GRU, and right Figure is a cell of LSTM

##### 4.2.3.3 Convolutional Neural Networks (CNN)

The final deep learning approach which contributes in RMDL is Convolutional
Neural Networks (CNN) that is employed for document or image classification.
Although originally built for image processing with architecture similar to
the visual cortex, CNN have also been effectively used for other datasets
classification [144]; thus, in RMDL, this technique is used in all datasets.
In the basic CNN for image processing an image tensor is convolved with a set
of kernels of size $d\times d$. These convolution layers are called feature
maps and can be stacked to provide multiple filters on the input. To reduce
the computational complexity CNN use pooling which reduces the size of the
output from one layer to the next in the network. Different pooling techniques
are used to reduce outputs while preserving important features [133]. The most
common pooling method is max pooling where the maximum element is selected in
the pooling window.
In order to feed the pooled output from stacked featured maps to the final
layer, the maps are flattened into one column. The final layers in a CNN are
typically fully connected.
In general, during the back propagation step of a convolutional neural network
not only the weights are adjusted but also the feature detector filters. A
potential problem of CNN used for other datasets are the number of ’channels’,
$\Sigma$ (size of the feature space). This might be very large (e.g. 10K) but
for images this is less of a problem (e.g. only 3 channels of RGB) [145].

Figure 4.4: This figure Shows multi SGD optimizer

#### 4.2.4 Optimization

In this paper we use two types of stochastic gradient optimizer in our neural
networks implementation which are RMSProp and Adam optimizer:

##### 4.2.4.1 Stochastic Gradient Descent (SGD) Optimizer

SGD has been used as one of our optimizers that is shown in equation 4.15. It
uses a momentum on re-scaled gradient which is shown in equation 4.16 for
updating parameters. The other technique of optimizer that is used is RMSProp
which does not do bias correction. This will be a significant problem while
dealing with sparse gradient.

$\displaystyle\theta$
$\displaystyle\leftarrow\theta-\alpha\nabla_{\theta}J(\theta,x_{i},y_{i})$
(4.15) $\displaystyle\theta$
$\displaystyle\leftarrow\theta-\big{(}\gamma\theta+\alpha\nabla_{\theta}J(\theta,x_{i},y_{i})\big{)}$
(4.16)

##### 4.2.4.2 Adam Optimizer

Adam is another stochastic gradient optimizer which uses only the first two
moments of gradient ($v$ and $m$ that are shown in equation 3.15, 3.16, 3.17,
and 3.18) and average over them. It can handle non-stationary of objective
function as in RMSProp while overcoming the sparse gradient issue that was a
drawback in RMSProp [135].

##### 4.2.4.3 Multi Optimization rule

The main idea of using multi model with different optimizers is that if one
optimizer does not provide a good fit for a specific datasets, the RMDL model
with $n$ random models (some of them might use different optimizers) could
ignore $k$ models which are not efficient if and only if $n>k$. The Figure 4.4
provides a visual insight on how three optimizers work better in the concept
of majority voting. Using multi techniques of optimizers such as SGD, adam,
RMSProp, Adagrad, Adamax, and so on helps the RMDL model to be more stable for
any type of datasets. In this research, we only used two optimizers (Adam and
RMSProp) for evaluating our model, but the RMDL model has the capability to
use any kind of optimizer.

### 4.3 Limitation and Discussion

Model interpretability of deep learning (DL), especially RMDL, has always been
a limiting factor for use cases requiring explanations of the features
involved in modelling and such is the case for many healthcare problems. This
problem is due to scientists preferring to use traditional techniques such as
linear models, SVM, decision trees, etc. for their works. The weights in a
neural network are a measure of how strong each connection is between each
neuron to find the important feature space. The more accurate model, the
interpretability is lower which means the complex algorithms such as deep
learning is hard to understand. This problem is even worse if we used ensemble
deep learning which is used in this chapter. Deep learning (DL) is one of the
most powerful techniques in artificial intelligence (AI), and many researchers
and scientists focus on deep learning architectures to improve the robustness
and computational power of this tool. However, deep learning architectures
also have some disadvantages and limitations when applied to classification
tasks. One of the main problems of this model is that DL does not facilitate a
comprehensive theoretical understanding of learning [146]. A well-known
disadvantage of DL methods is their “black box” nature [147, 148]. That is,
the method by which DL methods come up with the convolved output is not
readily understandable. Another limitation of DL is that it usually requires
much more data than traditional machine learning algorithms, which means that
this technique cannot be applied to classification tasks over small data sets
[149, 150]. Additionally, the massive amount of data needed for DL
classification algorithms further exacerbates the computational complexity
during the training step [151]. RMDL for medical image classification takes
more time in comparing with other techniques which for training RMDL with 15
model (we already resize our dataset) takes 78 hours to be trained in two
GPUs.

Table 4.1: Results of RMDL per Class for 3, 9 and 15 Random models models RMDL | Class | Accuracy | Precision | Recall | F1-Score
---|---|---|---|---|---
3 RDLs | N | 82.87$\pm$1.20 | 83.85$\pm$1.17 | 82.87$\pm$1.20 | 83.36$\pm$1.18
EE | 94.71$\pm$0.72 | 98.33$\pm$0.41 | 92.94$\pm$0.82 | 93.82$\pm$0.77
C | 82.91$\pm$1.2 | 82.95$\pm$1.21 | 82.36$\pm$1.23 | 82.66$\pm$1.22
9 RDLs | N | 90.84$\pm$0.92 | 89.51$\pm$ 0.97 | 90.84$\pm$0.92 | 90.17$\pm$0.95
EE | 94.71$\pm$0.71 | 95.49$\pm$0.67 | 94.71$\pm$0.72 | 95.10$\pm$0.69
C | 87.80$\pm$1.05 | 88.42$\pm$1.03 | 87.80$\pm$1.05 | 88.11$\pm$1.04
15 RDLs | N | 88.32 $\pm$ 1.02 | 90.50$\pm$ 0.93 | 88.32$\pm$1.02 | 89.40$\pm$0.98
EE | 97.88$\pm$0.46 | 95.65$\pm$0.65 | 97.88$\pm$0.46 | 96.76$\pm$0.57
C | 83.72$\pm$1.19 | 88.41$\pm$1.03 | 88.53$\pm$1.02 | 88.47$\pm$1.03

### 4.4 Results of RMDL for Medical Images

#### 4.4.1 Data

##### 4.4.1.1 Data Source

As shown in Table 6.1, The total population of this dataset is 150 children
participated in this study with a median (interquartile range) age of 37.5
(19.0 to 121.5) months and a roughly equal sex distribution; 77 males (51.3%).
The slides were converted into $461$ whole slide images, and labeled as either
EE, CD, or normal. The biopsy slides for EE patients were from the Aga Khan
University Hospital (AKUH) in Karachi, Pakistan ($n=29$ slides from $10$
patients) and the University of Zambia Medical Center in Lusaka, Zambia
($n=16$). The slides for CD patients ($n=34$) and normal ($n=63$) were
retrieved from archives at the University of Virginia (UVa). The CD and normal
slides were converted into whole slide images at $40$x magnification using the
Leica SCN $400$ slide scanner (Meyer Instruments, Houston, TX) at UVa, and the
digitized EE slides were of 20x magnification and shared via the Environmental
Enteric Dysfunction Biopsy Investigators (EEDBI) Consortium shared WUPAX
server. Characteristics of our patient population are as follows: the median
($Q1$, $Q3$) age of our entire study population was $37.5$ ($19.0$, $121.5$)
months, and we had a roughly equal distribution of males ($52$%, $n=53$) and
females ($48$%, $n=49$). The majority of our study population were
histologically normal controls $(37.7\%)$, followed by CD patients $(51.8\%)$,
and EE patients $(10.05\%)$.

##### 4.4.1.2 Pre-Processing

In this section, we cover all of the pre-processing steps which include image
patching, image clustering, and resizing images. The biopsy images are
unstructured (varying image sizes) and too large to process with deep neural
networks; thus, requiring that images are split into multiple smaller images.
After executing the split, some of the images do not contain much useful
information. For instance, some only contain the mostly blank border region of
the original image. In the image clustering section, the process to select
useful images is described. Finally, color balancing is used to correct for
varying color stains which is a common issue in histological image processing.

###### 4.4.1.2.1 Image Patching

Although the effectiveness of CNNs in image classification has been shown in
various studies in different domains, training on high-resolution Whole Slide
Tissue Images (WSI) is not commonly preferred due to a high computational
cost. However, applying CNNs on WSI enables losing a large amount of
discriminative information due to extensive down-sampling [124]. Due to a
cellular level difference between Celiac, Environmental Enteropathy and normal
cases, a trained classifier on image patches are likely to perform as well as
or even better than a trained WSI-level classifier. Many researchers in
pathology image analysis have considered classification or feature extraction
on image patches [124]. In this project, after generating patches from each
image, labels were applied to each patch according to its associated original
image. CNN was trained to generate predictions on each individual patch. As
you have shown in Figure 6.2, each biopsy whole image is divided into many
patches.

###### 4.4.1.2.2 Patch Clustering

As we discussed in Section 3.3.2, Clustering is organizing objects in a such
way that objects within a group or cluster in some way are more similar to
each other compared to objects in other groups. There is a wide variety of
algorithms for data clustering and K-means clustering is one of the easiest
ones [125].

#### 4.4.2 Empirical results

As shown in Table 6.5, results per class is represent that if the more RDLs is
used, that accuracy, precision, recall and F1-score are improved. Based on
this table accuracy of Normal for RMDL with 3, 9, ad 15 RDLs is 82.87$\pm$1.20
and 90.84$\pm$0.92 and 88.32$\pm$1.02 respectively. The F1-Score of Normal for
RMDL with 3, 9, ad 15 RDLs is 83.36$\pm$1.18 and 90.17$\pm$0.95 and
89.40$\pm$0.98 respectively. The results per class is represent that if the
more RDLs is used, that accuracy, precision, recall and F1-score are improved.
Based on this table accuracy of CD for RMDL with 3, 9, ad 15 RDLs is
82.91$\pm$1.2 and 87.80$\pm$0.1.05 and 83.72$\pm$1.19 respectively. The
F1-Score of CD for RMDL with 3, 9, ad 15 RDLs is 82.66$\pm$0.77 and
88.11$\pm$1.04 and 88.47$\pm$1.03 respectively. Based on this table accuracy
of EE for RMDL with 3, 9, ad 15 RDLs is 94.71$\pm$0.72 and 94.71$\pm$0.71 and
97.88$\pm$0.46 respectively. The F1-Score of CD for RMDL with 3, 9, ad 15 RDLs
is 93.82$\pm$0.77 and 95.1$\pm$0.69 and 96.86$\pm$0.57 respectively.

Figure 4.5: This Figure indicates loss function and accuracy for 3 Random Deep
Learning (RDL) model for 100 epochs.

As shown in Table 4.2, the overall results is represent that if the more RDLs
is used, that accuracy, precision, recall and F1-score are improved. We used
three baseline which are CNN that is shallow CNN is used with 3 Convolutional
Layer with kernel size of $3\times 3$ and 64 filter. this CNN contains
Maxpooling with size of $2$ and dropout of 0.25. The other Baseline we used is
Deep neural network or Deep multi-layer perceptron algorithm which is very
popular these days. In a Multilayer Perceptron, 4 hidden layers are used which
each layer contains 1000 nodes. And third Baseline is DCNN which contains 16
Convolutional Layer with kernel size of $3\times 3$ and 64 filter it has only
3 Maxpooling layer with size of $2$ and used dropout of 0.25. The result shown
as for a Multilayer Perceptron as Accuracy is 75.74$\pm$ 0.79 Precision is
equal to 88.67$\pm$0.58 Recall is 83.85$\pm$0.68 and F1-Score is
86.19$\pm$0.64. For CNN Accuracy is 83.01$\pm$0.69 Precision is equal to
91.62$\pm$0.51 Recall is 89.83$\pm$0.56 F1-Score is 90.72$\pm$0.53. The result
of DCNN is as follows Accuracy is 89.71$\pm$0.56 Precision is equal to
96.44$\pm$0.34 Recall is 92.79$\pm$0.48 and finally F1-Score is
94.58$\pm$0.42. The interesting part of this study shows that 3 RDLs in this
experiment is not good as DCNN; Thus, if we want to get a robust results we
should have more than 3 RDLs, especially, when we used random deep learning
architectures.

The RMDL results with using three RDLs Accuracy is 86.62$\pm$0.63 Precision is
equal to 93.01$\pm$0.47 Recall is 92.66$\pm$0.48 F1-Score is 92.83$\pm$0.47 .
The RMDL using 9 RDLs is as follows Accuracy is 91.12$\pm$0.52 Precision is
equal to 95.08$\pm$0.40 Recall is 95.63$\pm$0.37 F1-Score is 95.35$\pm$0.39
and finally, the results of RMDL with 15 RDLs is as follows: Accuracy is
91.61$\pm$0.51 Precision is equal to 95.85$\pm$0.37 Recall is 95.40$\pm$0.39
F1-Score is 95.62$\pm$0.38 .

Table 4.2: Results of RMDL in comparison with our baselines Model | Accuracy | Precision | Recall | F1-Score
---|---|---|---|---
Baseline | DNN | 75.74$\pm$ 0.79 | 88.67$\pm$0.58 | 83.85$\pm$0.68 | 86.19$\pm$0.64
CNN | 83.01$\pm$0.69 | 91.62$\pm$0.51 | 89.83$\pm$0.56 | 90.72$\pm$0.53
DCNN | 89.71$\pm$0.56 | 96.44$\pm$0.34 | 92.79$\pm$0.48 | 94.58$\pm$0.42
RMDL | 3 RDLs | 86.62$\pm$0.63 | 93.01$\pm$0.47 | 92.66$\pm$0.48 | 92.83$\pm$0.47
9 RDLs | 91.12$\pm$0.52 | 95.08$\pm$0.40 | 95.63$\pm$0.37 | 95.35$\pm$0.39
15 RDLs | 91.61$\pm$0.51 | 95.85$\pm$0.37 | 95.40$\pm$0.39 | 95.62$\pm$0.38
Figure 4.6: This Figure indicates loss function and accuracy for 9 Random Deep
Learning (RDL) model for 100 epoch

Figure 4.5 indicates loss function and accuracy for 3 Random Deep Learning
(RDL) model for 100 epochs which as it clear one of the RDL is converged much
faster that two other models. Figure 4.6 indicates loss function and accuracy
for 9 Random Deep Learning (RDL) model for 100 epochs which as it clear two of
the RDL is never converged but due to overall results is based on majority
votes this RDLs not effect on the final results. Figure 4.7 indicates loss
function and accuracy for 15 Random Deep Learning (RDL) model for 100 epochs
which as the 2 RDL are converged but their accuracy about 30 percent and one
model after 70 epochs the accuracy is drooped to less than 40 percent and it
backs to around 80 percent; but finally these models are not effect to overall
results is based on majority votes this RDLs not effect on the final results.

Figure 4.7: This Figure indicates loss function and accuracy for 15 Random
Deep Learning (RDL) model for 100 epoch

#### 4.4.3 Hardware and Framework

The processing units that has been used through this experiment was intel on
Xeon E5-2640 (2.6 GHz) with 12 cores and 64 GB memory (DDR3). Also, graphical
card on our machine is Nvidia Quadro K620 and Nvidia Tesla K20c. This work is
implemented in Python using Compute Unified Device Architecture (CUDA) which
is a parallel computing platform and Application Programming Interface (API)
model created by $Nvidia$. We used $TensorFelow$ [137] and $Keras$ [136]
library for creating the neural networks.

### 4.5 Conclusion

The classification task is an important problem to address in machine
learning, given the growing number and size of datasets that need
sophisticated classification. We propose a novel technique to solve the
problem of choosing best technique and method out of many possible structures
and architectures in deep learning. This paper introduces a new approach
called RMDL (Random Multimodel Deep Learning) for the medical image
classification that combines multi deep learning approaches to produce random
classification models. Our evaluation on datasets obtained from the biopsy
images shows that combinations of Multilayer Perceptrons, RNNs and CNNs with
the parallel learning architecture, has consistently higher accuracy. These
results show that deep learning methods can provide improvements for
classification and that they provide flexibility to classify datasets by using
majority vote. The proposed approach has the ability to improve accuracy and
efficiency of models and can be use across a wide range of other data types
and applications.

##  Chapter 5 Celiac Disease Severity Diagnosis on Duodenal Histopathological
Images Using Deep Learning

Celiac Disease (CD) is a chronic autoimmune disease that affects the small
intestine in genetically predisposed children and adults. Gluten exposure
triggers an inflammatory cascade which leads to compromised intestinal barrier
function. If this enteropathy is unrecognized, this can lead to anemia,
decreased bone density, and, in longstanding cases, intestinal cancer. The
prevalence of the disorder is $1\%$ in the United States. An intestinal
(duodenal) biopsy is considered the “gold standard” for diagnosis. The mild CD
might go unnoticed due to non-specific clinical symptoms or mild histologic
features. In our current work, we trained two model based on shallow
convolutional neural network (CNN) and deep residual networks to diagnose CD
severity using a histological scoring system called the modified Marsh score.
The proposed model was evaluated using an independent set of $120$ whole slide
images from $15$ CD patients and achieved an AUC greater than $0.96$ for
ResNet and in all classes. These results demonstrate the capability of the
proposed model for CD severity classification using histopathological images.

### 5.1 Introduction

Celiac disease (CD) is an inability to normally process dietary gluten
(present in foods such as wheat, rye, and barley) and is present in $1$% of
the US population. Gluten consumption by people with CD can cause diarrhea,
abdominal pain, bloating, and weight loss. If unrecognized, it can lead to
anemia, decreased bone density, and, in longstanding cases, intestinal cancer
[152, 153]. An intestinal (duodenal) biopsy, obtained via endoscopic
evaluation, is considered the “gold standard” for diagnosis of CD. Due to
unclear clinical symptoms and/or obscure histopathological features (based on
biopsy images), CD is often undiagnosed [154]. There has been major clinical
interest towards developing new and innovative methods to automate and enhance
the detection of morphological features of CD on biopsy images.

Among various architectures of CNNs, Residual Networks (ResNet) have received
special attention due to their considerably superior performance in the
analysis of histopathological images for disease detection, diagnosis and
prognosis prediction to complement the opinion of a human pathologist.
Multiple groups have published on the use of the ResNet architecture for
classification of Hematoxylin and Eosin (H&E) stained biopsy images including
breast and prostate cancer [155, 156, 157, 158, 159] and colorectal polyps
[160]. Similarly, impressive results for CD diagnosis based on whole slide
biopsy images have been noted in published literature [1]. Herein we explore
the performance of deep residual networks in severity diagnosis of CD on
duodenal biopsy images.

This dissertation section is organized as follows: In Section 5.2, disease
severity classes of CD are presented. In Section 5.3, we describe the data
used in this study. Section 5.4 presents the data pre-processing steps. The
methodology is explained in Section 5.5. Empirical results are elaborated in
Section 5.6. Finally, Section 5.7 concludes the dissertation section along
with outlining future directions.

### 5.2 Severity Classes of Celiac Disease

Modified Marsh Score Classification was developed to classify the severity of
CD based on microscopic histological morphological features (Figure 5.1). It
takes into account the architecture of the duodenum as having finger-like
projections (called “villi”) which are lined by cells called epithelial cells.
Between the villi are crevices called crypts that contain regenerating
epithelial cells. The normal ratio of the length of a typical healthy villus
to the depth of a representative health crypt should be between 3:1 and 5:1.
In the normal, healthy duodenum (first part of the small intestine), there
should be no more than 30 immune cells known as lymphocytes interspersed per
100 epithelial cells in the top layer of the villus. Marsh I histology
comprises of normal villus architecture with an increase in the number of
intraepithelial lymphocytes. Marsh II includes increased intraepithelial
lymphocytes along with a finding known as crypt hypertrophy in which the
crypts appear enlarged. This is usually rare since patients typically rapidly
progress from Marsh I to IIIa. Marsh III is sub-divided into IIIa (partial
villus atrophy), Marsh IIIb (subtotal villus atrophy) and Marsh IIIc (total
villus atrophy) to explain the spectrum of villus atrophy along with crypt
hypertrophy and increased intra-epithelial lymphocytes. Finally, in Marsh IV,
villi are completely atrophied. This is called “hypoplastic” or complete
villus atrophy and describes the microscopic histology of duodenal tissue from
patients at the extreme end of gluten sensitivity.

Figure 5.1: CD severity classification based on modified Marsh score [4]

### 5.3 Data Source

$162$ H&E stained duodenal biopsy slides were obtained from the archival
biopsies of $34$ CD patients from the University of Virginia (UVa) in
Charlottesville, VA, United States. Each slide contained multiple biopsies per
patient resulting in $336$ whole slide images at 40x magnification using the
Leica SCN 400 slide scanner (Meyer Instruments, Houston, TX) at the
Biorepository and Tissue Research Facility at UVa. Characteristics of our
patient population were as follows: the median $(Q1,Q3)$ age was
$130~{}(92.5,175.5)$ months. we had a roughly equal distribution of males
$(47.1\%,n=16)$ and females $(52.9\%,n=18)$. Biopsy images for our study
population were scored by two medical professionals and validated with reads
from a pathologist specialized in gastroenterology. Our biopsy image dataset
ranged from Marsh I to IIIc with no biopsy images present in Marsh II.

### 5.4 Data Pre-processing

Since whole slide images (WSIs) were digitized at high resolutions, these were
large files with notable color variability apparent on visual inspection.
Therefore, we pre-processed these before any computational analyses were
conducted. This section describes all pre-processing steps including image
patching, patch clustering and color normalization.

#### 5.4.1 Image Patching

The effectiveness of CNNs in image classification has been shown in various
studies across different domains [19, 161, 17]. However, the training of a CNN
on high resolution WSIs that are at a gigapixel level is not often feasible
due to high computational cost. Also, the application of CNNs on WSIs further
contributes to the loss of a large part of discriminatory information due to
extensive down-sampling which is needed in such images [124]. We hypothesized
that since there were cellular level morphological differences between
different CD severity classes given the spectrum of pathology, a trained
classifier on image patches would likely perform as well or better than a
trained WSI-level classifier. A sliding window method was applied to each
high-resolution WSI to generate patches of size $500\times 500$ pixels with
$50\%$ overlapping area. After generating patches from each image, we labelled
each patch based on its associated image.

Figure 5.2: Color normalization artifacts when using the method proposed by
Vahadane et al. [5]. Images in the first row represent the target image and
some source images. Their associated normalized images are in second row

#### 5.4.2 Patch Clustering

Clustering is organizing objects in a such way that objects within a group or
cluster in some way are more similar to each other compared to objects in
other groups. There is a wide variety of algorithms for data clustering and
K-means clustering is one of the easiest ones [125]. Finding the optimal
solution to the k-means clustering problem is NP-hard in general Euclidean
space even for 2 clusters. Clustering of $n$ $d$-dimension entities in $k$
clusters can be exactly solved in time of $O(n^{dk+1})$ [162]. Obviously,
reduction of dimension $d$ will result in significant improvement of the
K-means clustering algorithm in term of time complexity. To address the
problem of dimensionality reduction, a convolutional auto-encoder [88] was
used to learn embedded features of each patch. These auto-encoders have been
reported in the literature as having had great success as a dimensionality
reduction method via the powerful reprehensibility of neural networks [89].

In our work, a two-step clustering process was applied to identify useless
patches which had mostly been created from the background of the WSIs. All or
a large part of these patches were blank or did not contain any useful biopsy
information. Through the first step, a convolutional autoencoder was used to
learn the embedded features of each patch and in the second step k-means
clustering algorithm was applied to cluster embedded features into two
clusters: useful and not useful. Some results of patch clustering have been
shown in Figure 3.3.

#### 5.4.3 Stain Normalization

Histological images have substantial color variation that adds bias while
training the model [7]. This arises due to a wide variety of factors such as
differences in raw materials and manufacturing techniques of stain vendors,
staining protocols of labs, and color responses of digital scanners [5, 7]. To
avoid any bias, unwanted color variations are neutralized by conducting color
normalization as an essential pre-processing step prior to any analyses.
Various color normalization approaches have been proposed in the published
literature. In this study, we used the approach proposed by Vahadane et al.
[5] for child level due to that all CD images are collected from one center.
This approach preserves biological structure information by basing color
mixture modeling on sparse non-negative matrix factorization. Figure 5.2 shows
an example of the result of applying this technique on representative biopsy
patches.

### 5.5 Methodology

#### 5.5.1 Model Development

CNNs have demonstrated promising performance in image classification tasks.
There are many different architectures of CNNs in the literature, with
associated advantages and drawbacks. In the current study, we used the deep
residual network (ResNet)[163], a model which has shown great performance in
image classification problems including medical image analysis[155, 164].
Although it has been shown that CNNs with more convolutional layers achieve
the most accurate results, simply stacking more convolutional layers will not
lead to better performance. When the deep network reaches a certain depth, its
performance tends to be saturated and even begins to rapidly decline. In such
cases, the models involve a large number of parameters and are computationally
expensive to train through whole parameters. This is called the degradation
problem and ResNet was originally proposed to tackle this issue. The core idea
of ResNet is introducing a skip connection that skips one or more layers and
bypasses the input from the previous layer to the next layer without any
modification. Since these added shortcut connections perform identity mapping,
extra parameters are not added to the model. Such architecture enables
deployment of deeper networks without problem of degeneracy. The building
block of the ResNet is compared to the building block of the traditional
network in Figure 5.3. In the traditional networks, the mapping from input to
output can be represented by the nonlinear function $H(x)$. In residual
learning blocks, $F(x)=H(x)-x$ is used as mapping function [163]. In essence,
as part of traditional CNNs, the input $x$ is mapped to $F(x)$ which is a
completely new representation that does not keep any information about the
original input, while ResNet blocks compute a slight change to the original
input $x$ to get a slightly altered representation. ResNet was the Winner of
the ILSVRC 2015 in image classification, detection, and localization, as well
as the Winner of the MS COCO 2015 detection and segmentation.

Figure 5.3: Building blocks of (left) a traditional CNN, (right) a ResNet  Table 5.1: Architecture of the model Class | Layer Type | Output Shape | Number of Parameters
---|---|---|---
1 | Model | $(7,7,2048)$ | $2,3587,712$
2 | Flatten | $100352$ | $0$
3 | Dense | $1024$ | $102,761,472$
4 | Dropout | $1024$ | $0$
5 | Dense | $4$ | $4,100$

Different variants of ResNet models such as ResNet50, ResNet101, and ResNet152
were trained on the ImageNet dataset [165]. We customized the Resnet50 by
removing fully connected layers and keeping only the ResNet backbone as a
feature extractor. Then we added one fully connected layer with $1024$ neurons
that received the flattened output of the feature extractor. Finally, the
output layer was added such that it represented a prediction probability for
each of the four Marsh score categories: I, IIIa, IIIb and IIIc. We used
dropout on the fully-connected layers with $p=0.5$ as the regulizer. This
model has been summarized in Table 5.1.

Table 5.2: Patch-level performance of model for celiac disease severity diagnosis Based on the two Deep learning techniques which are ResNet and CNN. The model performances for I and IIIa ResNet is more robust but for IIIb and IIIc | Precision (%) | Recall (%) | F1-measure (%)
---|---|---|---
ResNet |  | I
---
(n = 6988)
| 93.30
---
(92.71, 93.89)
| 89.54
---
(88.82, 90.26)
| 91.38
---
(90.72, 92.04)
| IIIa
---
(n = 6615)
| 94.16
---
(93.59, 94.73)
| 84.75
---
(83.88, 85.62)
| 89.20
---
(88.45, 89.95)
| IIIb
---
(n = 7695)
| 83.94
---
(83.12, 84.76)
| 89.45
---
(88.76, 90.14)
| 86.61
---
(85.85, 87.37)
| IIIc
---
(n = 7369)
| 85.53
---
(84.73, 86.33)
| 90.61
---
(89.94, 91.28)
| 87.99
---
(87.25, 88.73)
CNN |  | I
---
(n = 2137)
| 88.73
---
(88.10 , 89.36)
| 85.07
---
(84.36, 85.78)
| 86.86
---
(86.19, 87.53 )
| IIIa
---
(n = 2052)
| 81.19
---
(80.37, 82.01)
| 83.72
---
(82.95, 84.49)
| 82.44
---
(81.64, 83.23)
| IIIb
---
(n = 2436)
| 90.51
---
(90.18, 90.81)
| 90.48
---
(89.31, 91.64)
| 90.49
---
(89.25, 91.74)
| IIIc
---
(n = 2433)
| 89.26
---
(88.03, 90.49)
| 90.18
---
(88.99, 91.36)
| 89.72
---
(88.51, 90.92)

We resized pre-processed patches into $224\times 224$ pixels and used them to
train our model. Both horizontal and vertical random rotations were performed
as part of our data augmentation. The model was trained on around $50,000$
patches for each of four classes. Optimization was performed using RMSprop
optimization with no momentum, a base learning rate of
$1\text{\times}{10}^{-5}$ and a multiclass cross entropy loss function. As
shown in Table 5.2, for classes I and IIIa, ResNet with mean of F1-measure of
91.38 and 89.20 is more accurate than CNN. But as regards to IIIb and IIIc
classes, CNN is more robust with mean F1-measure of 90.49 and 89.72. The
overall F1-measure of CNN is 87.37 and ResNet is 88.59. on T -test [166] which
the The overall F1-measure of ResNet with 1.22 is higher than CNN with
standard error of 0.294, and $p=0.00134$. As shown in Figure 5.4, the result
of ResNet for class I and IIIa is absolutely better than CNN. The result of
CNN in class IIIb is better than ResNet. The result for class IIIc shows that
none of these models is absolutely better than the other but the mean of CNN
model is higher than ResNet.

Figure 5.4: Comparison of F1-score results for four classes between ResNet and